Thread: Question about strtok function.

No. It scans OVER the chars which are delimiters. You should try it however, and see what it does. That's the beauty of a computer program. Small checks, are easily made and tested.

For your case, strtok will scan past the three asterisks, remember that location, then scan until it finds another delimiter (or '\0'), null-terminate at that point and return the remembered location.

If you want (at least) some of your delimiters to also be tokens (as is the case in C syntax, for example) then you can't use strtok, since it overwrites some of the delimiters.

For this reason strtok is kind of annoying to use. Depending on what you need you probably should make some utility funtions for calling it. For example this utility function will print the tokens returned by strtok along with the delimiters which were removed in between

Code:

void chop_string(const char *input_string, const char *delim)
{
    char *token;
    char *str;
    char *temp_string = strdup(input_string);

    for (str = temp_string;
         (token = strtok(str, delim)) != NULL;
         str = NULL)
    {
        // print delimiter if applicable
        size_t pos;
        if ((pos = token - temp_string) > 0) {
            printf("delimiter: '%c'\n", input_string[pos-1]);
        }
        // print token
        printf("%s\n", token);
    }
    free(temp_string);
}

int main(void)
{
    chop_string("one,two.three!four:five", ".,:!");
    return 0;
}

@c99tut,
Why not just write your own strtok that returns the token-ending delimiter?

Code:

#include <stdio.h>
#include <string.h>

char *
my_strtok(
    char * str,
    const char * delims,
    char * del)
{
    static char * start = NULL, * end = NULL;

    if (str)
        start = end = str;
    else if (end == NULL)
        return NULL;
    else
        start = end + 1;

    start += strspn(start, delims);

    end = strpbrk(start, delims);
    if (end) {
        *del = *end;
        *end = '\0';
    }else
        *del = '\0';

    return start;
}


int
main()
{
    char str[] = "***NO&SIR**I&DON'T&&LIKE*IT!";
    char *p, del;

    p = my_strtok(str, "*&", &del);

    while (p) {
        printf("%s\t", p);
        if (del)
            printf("{%c}\n", del);
        else
            printf("{null}\n");
        p = my_strtok(NULL, "*&", &del);
    }

    return 0;
}

Output:

Code:

NO      {&}
SIR     {*}
I       {&}
DON'T   {&}
LIKE    {*}
IT!     {null}

Originally Posted by oogabooga

@c99tut,
Why not just write your own strtok that returns the token-ending delimiter?

Personally I don't find strtok intuitive or clear for the user code. Look at the main function and see what is trying to do. Is it clear what is happening? I would like to replace the printf statements from my previous version with a return statement to return the code. Then, by using the header file here (Coroutines in C) for coroutines, I can substitute the printf calls for return statements

Code:

#include "coroutine.h"
char *next_token(ccrContParam, const char *input_string, const char *delim)
{
    ccrBeginContext;
    char *token;
    char *str;
    char *temp_string;
    char *returnstr;
    ccrEndContext(cc);
    
    ccrBegin(cc);
    cc->temp_string = strdup(input_string);
    for (cc->str = cc->temp_string;
        (cc->token = strtok(cc->str, delim)) != NULL;
        cc->str = NULL)
    {
        // output delimiter if applicable
        size_t pos;
        if ((pos = cc->token - cc->temp_string) > 0) {
            char c[2];
            c[0] = input_string[pos-1];
            c[1] = '\0';
            cc->returnstr = strdup(c);
            ccrReturn(cc->returnstr);
            free(cc->returnstr);
        }
        // output token
        cc->returnstr = strdup(cc->token);
        ccrReturn(cc->returnstr);
        free(cc->returnstr);
    }
    free(cc->temp_string);
    ccrFinish(NULL);
}

It looks strange at first but notice the only difference is the printf is replaced by ccrReturn. I find it an advantage that main can just get tokens without having to care about a NULL parameter or whatever such nonsense

Code:

int main(void)
{
	const char str[] = "***NO&SIR**I&DON'T&&LIKE*IT!";
	const char delim[] = "*&";
	ccrContext cc = 0;
	do {
		char *token = next_token(&cc, str, delim);
		printf("%s\n", token);
	} while (cc);
	
	return 0;
}

Originally Posted by Adak

scan over means
According to Google translation:

Code:

戰國策·秦策三 范雎至秦 一段漏字
范睢至秦，王庭迎，謂范睢曰：「寡人宜以身受令久矣。今者義渠之事急，寡人日自請太后。今義渠之事已，寡人

Which I assure you, is Greek to me! <ROFL!>

And Chinese to him! ;p

Some real Greek here, in case the OP hasn't got it yet.
Η strtok() θα διατρέξει ένα-ένα τα στοιχεία της συμβολοσειράς. Αυτό εννοεί ο Adak scan over.