Thread: strtok question

Hello,

I am trying to tokenize an input txt file by spaces.

My code for it is:

Code:

while ( fgets (input, LENGTH, inputTxt))
{
   token = strtok(input, " ");
   while(token)
   {
      printf("@%s@\n", token);
      token = strtok(NULL, " ");   
   }

}

As shown I have printed out the tokens encapsulating them with @ symbols.

The issue is that if the input text line is "Our joy advance,"
My output is

Code:

@Our@
@joy@
@advance,
@

My problem is with the last token "advance," the @ symbol shows that the token include a new line \n or maybe it's \0.

I want to send my tokens to another function, but i'm worried the last word of a line includes the \n in it's token.

Is there a way to just get the word by itself?

Originally Posted by seal308

Is there a way to just get the word by itself?

fgets leaves the newline at the end of the buffer.
The easiest way for you to deal with it is simply to add newline to your strtok delimiters.
You might want to add tab and a comma and perhaps other punctuation as well.

Code:

const char *delims = " \t\n,.;:'\"";
// ...
token = strtok(input, delims);

Thank you, this worked perfectly.
I also had to made an change in the while token loop.

Code:

while (token)
{
...
token = strtok(NULL, delims);
}

Originally Posted by algorism

fgets leaves the newline at the end of the buffer.
The easiest way for you to deal with it is simply to add newline to your strtok delimiters.
You might want to add tab and a comma and perhaps other punctuation as well.

Code:

const char *delims = " \t\n,.;:'\"";
// ...
token = strtok(input, delims);

The delimiters argument is a string that specifies a set of delimiters that may surround the token being extracted.
All the initial bytes that are members of this set are discarded.
The first byte that is not a member of this set of delimiters marks the beginning of the next token.
The end of the token is found by looking for the next byte that is a member of the delimiter set.
This byte in the original string newstring is overwritten by a null byte, and the pointer to the beginning of the token in newstring is returned.