Strtok problem

**vin_pll** · 10-03-2008

Hello experts ,

Need your help for strtok() function it is showing some strange answers to me,
mainly i am reading the contents of a file and storing in buffer, and when i do strtok on indivudual line many lines data getting corrupted,

means i am not getting all the data , many lines are getting corrupted in between what shall i do for i tried from all the way at the end i am seeking your help please help me, and also i have used this code

Code:

for (j = 1, str1 = buf; ; j++, str1 = NULL) {
        token = strtok_r(str1, "\n",&saveptr1);//fprintf(fp1,token);
        if (token == NULL)
            break;
        printf("%d: %s\n", j, token);
      for (str2 = token; ; str2 = NULL) {
            subtoken = strtok_r(str2, ",",&saveptr2);//fprintf(fp2,subtoken);
            if (subtoken == NULL)
                break;
            printf("t --> %s\n", subtoken);
        }
    }

which is working for linux and unix but not with windows,
please help me solve this issue.

**cpjust** · 10-03-2008

What exactly is being corrupted?
You do know that strtok() modifies the string you pass in to it right?

**valaris** · 10-03-2008

After you give strtok a string it will keep that string until you change it out. You pass it NULL for subseuent calls for tokens from that string until it has reaches the end and itself returns NULL.

http://www.cppreference.com/wiki/c/string/strtok

**~~master5001~~** · 10-03-2008

Awe man... Elysia took her link to her strtok() functions down. Perhaps you can use the search tool to find them again. Her versions do not alter the internal data.

**vin_pll** · 10-03-2008

thank you experts ,
for your inputs,

but the data is getting corrupted means,
what iam trying to do is opening a file in the erad mode and reading all the contents into the buffer and i am trying tokenize based on newline("\n") where i will get whole row which again is the input for second strtok which tokenizes based on comma(,) , know in this case the data is getting corrupted the file that i was having the contents with output is giving me lot of empty spaces,

how solve this problem please help me.

**tabstop** · 10-03-2008

The first call to strtok and all the other calls to strtok are different (you must use NULL for the first argument if you want to keep tokenizing the same string). So you must adjust your for loops accordingly.

EDIT: Yes, you have that right (I was not reading your for loops correctly).

**Salem** · 10-03-2008

Perhaps the problem is in how you read the file (post this code).
The tokeniser loops seem OK so far.

**vin_pll** · 10-04-2008

hello Experts,

i am using

Code:

fp=fopen(filename,"rb");
fread(buf, 1, 57700 , fpFile);

know even i am using array which tokenises on "\n" and gives me array with all lines,

Code:

for(int i=0;i<count-1;i++)
 {
  char *sd=buf3[i];
  printf("\nbuffer contents=%s",sd);
  token = strtok(str1, ",");
  while(token)
 {
       printf("t --> %s\n", token);
        token = strtok(NULL,",");
 }  
}

know iam tokenizing on comma(,) but in this loop also everything is working fine but data is getting corrupted in between what is the reason

in this loop only always it is corrupting the data rest of syntax evrything is fine
so please help me from that context.

**Salem** · 10-04-2008

Unless you have
char buf[57700]
or
char *buff = malloc( 57700 );
you're going nowhere.

Also, how did count get set up, is it 57700 as well?

Use the return result of fread() to tell you how successful you were.
Remember that if you converted your unix (\n) files to DOS (\r\n), the file is now bigger. So if you're expecting the same amount of data, your program is wrong.

A better question is why you're not using fgets() to read each line, because that's all you're doing (or at least trying to, badly).

**vin_pll** · 10-04-2008

i am getting that number 57700 simply i havementioned there iam getting the file size which acts as argument of buf[]

for ex:

Code:

filesize=filename.size();
buf[filesize]

this is the actual one i have implemented some code which gives me file size.
so where iam going wrong please help me.

**C_ntua** · 10-04-2008

Hmmmm

**Salem** · 10-04-2008

1. Use filesize everywhere, instead of that magic number.
2. Add 1 for a \0, which I guess you don't add, and will cause strtok to barf
3. Declaring a dynamic array is illegal in C++ (though I guess this may be where you say it's C, and just can't click the right forum).

Eg.

Code:

char *buff = new char [filesize+1];
n = fread( buff, 1, filesize, fp );
if ( n != filesize ) {
  // a problem?
} else {
  buff[n] = '\0'; // make a string out of it
}
// now tokenise

// all done
delete [ ] buff;

**C_ntua** · 10-04-2008

Your code works fine for me. Your initial code should read one line and tokenize it. I did this:

Code:

#include <stdio.h>
#include <string.h>

int main( int argc, char* argv[] )
{
char *str1, *str2, *token, *saveptr1, *saveptr2, *subtoken;
int j;
char buf[50] = ",Hey,Dude,dfa,\n,Hey,Dude,dfa,\n";
for (j = 1, str1 = buf; ; j++, str1 = NULL) {
        token = strtok(str1, "\n");
        if (token == NULL)
            break;
        printf("&#37;d: %s\n", j, token);
      for (str2 = token; ; str2 = NULL) {
            subtoken = strtok(str2, ",");//fprintf(fp2,subtoken);
            if (subtoken == NULL)
                break;
            printf("t --> %s\n", subtoken);
        }
    }
}

It reads only one line. Thus, it stops on the first \n. I guess this is not what you want eh? You would like it to print everything in buf...
If you don't do the subtokenize thing it works fine. I don't think you can use strtok() the way you do. Maybe with temporary char[] ?

**C_ntua** · 10-05-2008

Dunno if the problem was found. But the problem is that strtok() probably enters a '\0' when it finds the delimeter. So, lets say you have "Hey, dud\n".
1) token will point at the start, '\n' will be '\0'
2) subtoken will point at the start and ',' will be '\0'
3) A the second iteration you will actually have "Hey'\0'dud'\0", which is actually "Hey". At the second iteration it won't start at one char after token. It will start after the first '\0'. Of course the reason it does that is because you are supposed to use strtok in a very certain way. So the second time it will ready "dud'\0".

Anyway, strtok() kind of sucks I guess :P. You should make a copy each time of the string. Then subtokenize the copy.
It is a C++ though. You could use std::string.find() to find a delimiter. After you find it then just print/save everything until that point. That is your token.

Thread: Strtok problem

Thread Tools

Search Thread

Display

Strtok problem

Strtok problem

strtok problem

strtok problem

Similar Threads

WS_POPUP, continuation of old problem

A bit of a specific problem with reading a file and line

Strange problem - strtok

Laptop Problem

Sorting problem.. well actually more of a string problem