Thread: About strtok

  1. #1
    Registered User
    Join Date
    Jan 2008
    Posts
    66

    About strtok

    We all know, that strtok puts a NULL character next to the last character of the first token.
    Any subsequent calls will then use NULL as the string, searching for remaining delimeters.

    My question is, why should we use NULL as the string for those subsequent calls?
    To which part of the string does NULL refer to?

  2. #2
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    A series of char's is just a series of char's - unless and until a NULL '/0' char elevates the char's before it, to the status of a string.

    Then C will accept a series of char's, as a string.

  3. #3
    Hurry Slowly vart's Avatar
    Join Date
    Oct 2006
    Location
    Rishon LeZion, Israel
    Posts
    6,788
    when we call strtok with not NULL argument - it saves the pointer to the string in the static var
    the subsequent call with the NULL argument just informs strtok that it should use the previously saved pointer to continue parsing the string

    If you give the new string - the old pointer is overwritten and work starts from the beginning of the new string
    All problems in computer science can be solved by another level of indirection,
    except for the problem of too many layers of indirection.
    – David J. Wheeler

  4. #4
    Code Goddess Prelude's Avatar
    Join Date
    Sep 2001
    Posts
    9,897
    >We all know, that strtok puts a NULL character next to the last character of the first token.
    Minor nit: you shouldn't use NULL to refer to '\0'. NULL is a standard macro that may have a pointer type. Some people call it NUL instead, but I prefer to say "null character" or '\0' to avoid any confusion.
    My best code is written with the delete key.

  5. #5
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    As a consequence of what vart is saying, you can't use strtok() to tokenise two strings at the same time.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  6. #6
    Registered User
    Join Date
    Jan 2008
    Posts
    66
    Oh, I see thanks for making it clear.

  7. #7
    Registered User
    Join Date
    Jan 2007
    Posts
    330
    basically, strtok sucks roll your own

  8. #8
    Registered User
    Join Date
    Jan 2008
    Posts
    69

    Thumbs down

    Quote Originally Posted by vart View Post
    when we call strtok with not NULL argument - it saves the pointer to the string in the static var
    the subsequent call with the NULL argument just informs strtok that it should use the previously saved pointer to continue parsing the string
    Exactly. Here's an implementation (not thread-safe) that I wrote, which should give you a general idea of what happens internally when you use strtok.

    Code:
    char * 
    str_split(char *str, char *delimiters) {
    
      static char *f;
      static unsigned char reg_delims[32];
    
      if (!f || str) {
        f = str;
      }
    
      memset(reg_delims, 0, 32);
    
      while (*delimiters) {
    
        reg_delims[(*delimiters / 8)] |= (1 << (*delimiters % 8));
        delimiters++;
    
      }
    
      str = f;
    
      if (f) {
    
        while(*f && !(reg_delims[(*f / 8)] &= (1 << (*f % 8)))) {
          f++;
        }
    
        if (!*f) {
          f = NULL;
        } else {
          *(f++) = '\0';
        }  
    
      }
    
      return str;
    
    }

  9. #9
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,057
    delimiters could be const, just like in the real strtok(). http://www.cplusplus.com/reference/c...ng/strtok.html

    unless and until a NULL '/0' char
    Another minor nit: '\0', not '/0'.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. 20q game problems
    By Nexus-ZERO in forum C Programming
    Replies: 24
    Last Post: 12-17-2008, 05:48 PM
  2. strtok is causing segmentation fault
    By yougene in forum C Programming
    Replies: 11
    Last Post: 03-08-2008, 10:32 AM
  3. trying to use strtok() function to parse CL
    By ohaqqi in forum C Programming
    Replies: 15
    Last Post: 07-01-2007, 09:38 PM
  4. Help debugging my program
    By shoobsie in forum C Programming
    Replies: 4
    Last Post: 07-05-2005, 07:14 AM
  5. Trouble with strtok()
    By BianConiglio in forum C Programming
    Replies: 2
    Last Post: 05-08-2004, 06:56 PM