[Solved] How to use strtok in string?

Question

The cause of the problem:

The function strtok() has many problems, due to the fact that subsequent calls depend on previous calls, and this dependency is managed in an unsafe manner:

it’s not thread safe (see Robert’s comment, and C++ standard section 21.8 pt 14)
if one function you call would use strtok() without you knowing, your next call to strtok() would return a lot of surprises.

Now your problem comes from the input string part: ...400 BB 2;;FQ;..., and the definition of strtok() : In subsequent calls, the function (…) uses the position right after the end of last token as the new starting location for scanning. To determine the beginning and the end of a token, the function first scans from the starting location for the first character not contained in delimiters (which becomes the beginning of the token)

So everything works well until it returns “400 BB 2”. The next “;” will according to this algorithm be skipped and your code will jump over the empty field (;;) as if it didn’t exist. Not ony do you have a shift in the following fields, but your last call to strtok() may even cause segmentation fault.

Solution:

Best avoid strtok(). If you like c-style, you may consider instead the use of strpbrk() with some adaptation in your code. For example:

char* get_field(char*p, char*& next, const char* s) // by ref as it's c++
{
    if ((next = strpbrk(p, s)) != NULL)
        *next++ = '\0';
    return p; 
}

with the following usage to replace strtok():

char* next;
char *chars_array = get_field(FQ_array, next, seps);
... 
chars_array = get_field(next, next, seps); // instead of strtok(NULL, seps)
...

My personal recommendation, with C++, would be to consider regex expressions provided in the standard (or in boost), which would also allow for consistency check on you input data.

The full code would then look like:

regex fmt("([0-9]*,[0-9]*);(.*);(.*);(.*);(.*);(.*);(.*);(.*);(.*);([0-9]*\.*[0-9]*)");

for (int FQ_i = 0; ...)
{
    smatch sm;
    printf("FQ_ID_line[FQ_i]=%u\n", FQ_ID_line[FQ_i]);  // ok, a cout would be better
    if (regex_match(FQ_ID_line[FQ_i], sm, fmt)) {
        DataLine[FQ_i].analog_comp_id = sm[2];
        DataLine[FQ_i].RTU_abbr = sm[3]; 
        DataLine[FQ_i].analog_scc_fep_group = sm[8];
        DataLine[FQ_i].analog_bsc_fep_group = sm[9]; 
        DataLine[FQ_i].RTU_number = sm[10]; 
        DataLine[FQ_i].float_RTU_number = stof(sm[10]); 
    }
    else
        cout << " ** Non matching line ignored !!\n";
}

By fine tuning the regex, you could then check even more for consistency before assigning (Here I just did the minimum for the sake of the example).

Accepted Answer