Strtok problem

This is a discussion on Strtok problem within the C++ Programming forums, part of the General Programming Boards category; Hello experts , Need your help for strtok() function it is showing some strange answers to me, mainly i am ...

  1. #1
    Registered User
    Join Date
    Nov 2007
    Posts
    99

    Strtok problem

    Hello experts ,

    Need your help for strtok() function it is showing some strange answers to me,
    mainly i am reading the contents of a file and storing in buffer, and when i do strtok on indivudual line many lines data getting corrupted,

    means i am not getting all the data , many lines are getting corrupted in between what shall i do for i tried from all the way at the end i am seeking your help please help me, and also i have used this code

    Code:
    for (j = 1, str1 = buf; ; j++, str1 = NULL) {
            token = strtok_r(str1, "\n",&saveptr1);//fprintf(fp1,token);
            if (token == NULL)
                break;
            printf("%d: %s\n", j, token);
          for (str2 = token; ; str2 = NULL) {
                subtoken = strtok_r(str2, ",",&saveptr2);//fprintf(fp2,subtoken);
                if (subtoken == NULL)
                    break;
                printf("t --> %s\n", subtoken);
            }
        }
    which is working for linux and unix but not with windows,
    please help me solve this issue.

  2. #2
    and the hat of sweating
    Join Date
    Aug 2007
    Location
    Toronto, ON
    Posts
    3,545
    What exactly is being corrupted?
    You do know that strtok() modifies the string you pass in to it right?
    "I am probably the laziest programmer on the planet, a fact with which anyone who has ever seen my code will agree." - esbo, 11/15/2008

    "the internet is a scary place to be thats why i dont use it much." - billet, 03/17/2010

  3. #3
    Registered User valaris's Avatar
    Join Date
    Jun 2008
    Location
    RING 0
    Posts
    507
    After you give strtok a string it will keep that string until you change it out. You pass it NULL for subseuent calls for tokens from that string until it has reaches the end and itself returns NULL.

    http://www.cppreference.com/wiki/c/string/strtok

  4. #4
    Banned master5001's Avatar
    Join Date
    Aug 2001
    Location
    Visalia, CA, USA
    Posts
    3,685
    Awe man... Elysia took her link to her strtok() functions down. Perhaps you can use the search tool to find them again. Her versions do not alter the internal data.

  5. #5
    Registered User
    Join Date
    Nov 2007
    Posts
    99

    Strtok problem

    thank you experts ,
    for your inputs,

    but the data is getting corrupted means,
    what iam trying to do is opening a file in the erad mode and reading all the contents into the buffer and i am trying tokenize based on newline("\n") where i will get whole row which again is the input for second strtok which tokenizes based on comma(,) , know in this case the data is getting corrupted the file that i was having the contents with output is giving me lot of empty spaces,

    how solve this problem please help me.

  6. #6
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,185
    The first call to strtok and all the other calls to strtok are different (you must use NULL for the first argument if you want to keep tokenizing the same string). So you must adjust your for loops accordingly.

    EDIT: Yes, you have that right (I was not reading your for loops correctly).
    Last edited by tabstop; 10-03-2008 at 11:37 PM.

  7. #7
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,335
    Perhaps the problem is in how you read the file (post this code).
    The tokeniser loops seem OK so far.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  8. #8
    Registered User
    Join Date
    Nov 2007
    Posts
    99

    strtok problem

    hello Experts,

    i am using
    Code:
    fp=fopen(filename,"rb");
    fread(buf, 1, 57700 , fpFile);
    know even i am using array which tokenises on "\n" and gives me array with all lines,
    Code:
    for(int i=0;i<count-1;i++)
     {
      char *sd=buf3[i];
      printf("\nbuffer contents=%s",sd);
      token = strtok(str1, ",");
      while(token)
     {
           printf("t --> %s\n", token);
            token = strtok(NULL,",");
     }  
    }
    know iam tokenizing on comma(,) but in this loop also everything is working fine but data is getting corrupted in between what is the reason

    in this loop only always it is corrupting the data rest of syntax evrything is fine
    so please help me from that context.

  9. #9
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,335
    Unless you have
    char buf[57700]
    or
    char *buff = malloc( 57700 );
    you're going nowhere.

    Also, how did count get set up, is it 57700 as well?

    Use the return result of fread() to tell you how successful you were.
    Remember that if you converted your unix (\n) files to DOS (\r\n), the file is now bigger. So if you're expecting the same amount of data, your program is wrong.

    A better question is why you're not using fgets() to read each line, because that's all you're doing (or at least trying to, badly).
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  10. #10
    Registered User
    Join Date
    Nov 2007
    Posts
    99

    strtok problem

    i am getting that number 57700 simply i havementioned there iam getting the file size which acts as argument of buf[]

    for ex:
    Code:
    filesize=filename.size();
    buf[filesize]
    this is the actual one i have implemented some code which gives me file size.
    so where iam going wrong please help me.

  11. #11
    Registered User C_ntua's Avatar
    Join Date
    Jun 2008
    Posts
    1,853
    Hmmmm

  12. #12
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,335
    1. Use filesize everywhere, instead of that magic number.
    2. Add 1 for a \0, which I guess you don't add, and will cause strtok to barf
    3. Declaring a dynamic array is illegal in C++ (though I guess this may be where you say it's C, and just can't click the right forum).

    Eg.
    Code:
    char *buff = new char [filesize+1];
    n = fread( buff, 1, filesize, fp );
    if ( n != filesize ) {
      // a problem?
    } else {
      buff[n] = '\0'; // make a string out of it
    }
    // now tokenise
    
    // all done
    delete [ ] buff;
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  13. #13
    Registered User C_ntua's Avatar
    Join Date
    Jun 2008
    Posts
    1,853
    Your code works fine for me. Your initial code should read one line and tokenize it. I did this:
    Code:
    #include <stdio.h>
    #include <string.h>
    
    int main( int argc, char* argv[] )
    {
    char *str1, *str2, *token, *saveptr1, *saveptr2, *subtoken;
    int j;
    char buf[50] = ",Hey,Dude,dfa,\n,Hey,Dude,dfa,\n";
    for (j = 1, str1 = buf; ; j++, str1 = NULL) {
            token = strtok(str1, "\n");
            if (token == NULL)
                break;
            printf("&#37;d: %s\n", j, token);
          for (str2 = token; ; str2 = NULL) {
                subtoken = strtok(str2, ",");//fprintf(fp2,subtoken);
                if (subtoken == NULL)
                    break;
                printf("t --> %s\n", subtoken);
            }
        }
    }
    It reads only one line. Thus, it stops on the first \n. I guess this is not what you want eh? You would like it to print everything in buf...
    If you don't do the subtokenize thing it works fine. I don't think you can use strtok() the way you do. Maybe with temporary char[] ?
    Last edited by C_ntua; 10-04-2008 at 06:04 AM.

  14. #14
    Registered User C_ntua's Avatar
    Join Date
    Jun 2008
    Posts
    1,853
    Dunno if the problem was found. But the problem is that strtok() probably enters a '\0' when it finds the delimeter. So, lets say you have "Hey, dud\n".
    1) token will point at the start, '\n' will be '\0'
    2) subtoken will point at the start and ',' will be '\0'
    3) A the second iteration you will actually have "Hey'\0'dud'\0", which is actually "Hey". At the second iteration it won't start at one char after token. It will start after the first '\0'. Of course the reason it does that is because you are supposed to use strtok in a very certain way. So the second time it will ready "dud'\0".

    Anyway, strtok() kind of sucks I guess :P. You should make a copy each time of the string. Then subtokenize the copy.
    It is a C++ though. You could use std::string.find() to find a delimiter. After you find it then just print/save everything until that point. That is your token.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. WS_POPUP, continuation of old problem
    By blurrymadness in forum Windows Programming
    Replies: 1
    Last Post: 04-20-2007, 06:54 PM
  2. Replies: 6
    Last Post: 04-28-2006, 12:06 PM
  3. Strange problem - strtok
    By AngKar in forum C Programming
    Replies: 7
    Last Post: 04-23-2006, 07:36 PM
  4. Laptop Problem
    By Boomba in forum Tech Board
    Replies: 1
    Last Post: 03-07-2006, 05:24 PM
  5. Replies: 5
    Last Post: 11-07-2005, 10:34 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21