strtok_r()

This is a discussion on strtok_r() within the C Programming forums, part of the General Programming Boards category; anyone have the implementation of the strtok_r() function? i've been searching on google, but it only returns how to use ...

  1. #1
    Registered User
    Join Date
    Jul 2004
    Posts
    91

    strtok_r()

    anyone have the implementation of the strtok_r() function?
    i've been searching on google, but it only returns how to use the function, but not how it actually is implemented..

  2. #2
    Software Developer jverkoey's Avatar
    Join Date
    Feb 2003
    Location
    University of Waterloo
    Posts
    1,904
    here's the strtok function, couldn't find strtok_r though

    Code:
     /***
     *strtok.c - tokenize a string with given delimiters
     *
     *	   Copyright (c) Microsoft Corporation. All rights reserved.
     *
     *Purpose:
     *	   defines strtok() - breaks string into series of token
     *	   via repeated calls.
     *
     *******************************************************************************/
     
     #include <cruntime.h>
     #include <string.h>
     #ifdef _MT
     #include <mtdll.h>
     #endif  /* _MT */
     
     /***
     *char *strtok(string, control) - tokenize string with delimiter in control
     *
     *Purpose:
     *	   strtok considers the string to consist of a sequence of zero or more
     *	   text tokens separated by spans of one or more control chars. the first
     *	   call, with string specified, returns a pointer to the first char of the
     *	   first token, and will write a null char into string immediately
     *	   following the returned token. subsequent calls with zero for the first
     *	   argument (string) will work thru the string until no tokens remain. the
     *	   control string may be different from call to call. when no tokens remain
     *	   in string a NULL pointer is returned. remember the control chars with a
     *	   bit map, one bit per ascii char. the null char is always a control char.
     *
     *Entry:
     *	   char *string - string to tokenize, or NULL to get next token
     *	   char *control - string of characters to use as delimiters
     *
     *Exit:
     *	   returns pointer to first token in string, or if string
     *	   was NULL, to next token
     *	   returns NULL when no more tokens remain.
     *
     *Uses:
     *
     *Exceptions:
     *
     *******************************************************************************/
     
     char * __cdecl strtok (
     		char * string,
     		const char * control
     		)
     {
     		unsigned char *str;
     		const unsigned char *ctrl = control;
     
     		unsigned char map[32];
     		int count;
     
     #ifdef _MT
     		_ptiddata ptd = _getptd();
     #else  /* _MT */
     		static char *nextoken;
     #endif  /* _MT */
     
     		/* Clear control map */
     		for (count = 0; count < 32; count++)
     				map[count] = 0;
     
     		/* Set bits in delimiter table */
     		do {
     			    map[*ctrl >> 3] |= (1 << (*ctrl & 7));
     		} while (*ctrl++);
     
     		/* Initialize str. If string is NULL, set str to the saved
     		 * pointer (i.e., continue breaking tokens out of the string
     		 * from the last strtok call) */
     		if (string)
     				str = string;
     		else
     #ifdef _MT
     				str = ptd->_token;
     #else  /* _MT */
     				str = nextoken;
     #endif  /* _MT */
     
     		/* Find beginning of token (skip over leading delimiters). Note that
     		 * there is no token iff this loop sets str to point to the terminal
     		 * null (*str == '\0') */
     		while ( (map[*str >> 3] & (1 << (*str & 7))) && *str )
     				str++;
     
     		string = str;
     
     		/* Find the end of the token. If it is not the end of the string,
     		 * put a null there. */
     		for ( ; *str ; str++ )
     			    if ( map[*str >> 3] & (1 << (*str & 7)) ) {
     					    *str++ = '\0';
     					    break;
     				}
     
     		/* Update nextoken (or the corresponding field in the per-thread data
     		 * structure */
     #ifdef _MT
     		ptd->_token = str;
     #else  /* _MT */
     		nextoken = str;
     #endif  /* _MT */
     
     		/* Determine if a token has been found. */
     		if ( string == str )
     				return NULL;
     		else
     				return string;
     }

  3. #3
    Registered User
    Join Date
    Jul 2004
    Posts
    101
    The only difference between strtok and strtok_r is that strtok_r is reentrant. That means that the function does not maintain an internal state between calls. Every implementation of strtok that I know of uses a static variable to save the state so that the next call will begin at the proper location. strtok_r is not a standard function so it can be implemented in different ways as the interface is not set in stone if I recall correctly.

    An example of the interface that strtok_r might use is as follows.
    Code:
    char *strtok_r(char *str, const char *sep, char **last);
    Then instead of saving the position in a static variable, it is saved in last and last is then used as the first argument for subsequent calls.
    Code:
    char *last = string;
    
    while ((t = strtok_r(last, ":", &last)) != NULL) {
      /* Do something with t */
    }

  4. #4
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    Posts
    6,231
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

Popular pages Recent additions subscribe to a feed

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21