String Tokenizing

I currently have a string delimited with pipes:
i.e. : pMsg = str1|str2|str3|str4

This string is tokenized using strtok. As long as all the strings are not empty, this works fine. After strtok is complete, I have the following:

token1 = str1
token2 = str2
token3 = str3
token4 = str4

BUT, I want to make it so I can have an empty string as one of the tokens in the string.
i.e. : pMsg = |str2|str3|str4

strtok seems to skip over the first pipe in this instance and assigns str2 to token1. Everything is getting bumped up one spot.

token1 = str2
token2 = str3
token3 = str4
token4 =

My goal is to still read token1 in as an empty string, and thus resulting in the following:

token1 =
token2 = str2
token3 = str3
token4 = str4


Any ideas????????

Use boost::string_tokenizer (I think it does that) or my split_string function.

It is recommended that you use something as an alternate to strtok() since it destroys the string... Try writing something on your own : ) it might be a good excercise for you.

strtok() doesn't destroy the string, but it isn't properly reentrant and not thread-safe unless programmed that way.

Yes, strtok() doesn't destroy the string!
and it's a good solution for his simple case...

>>irncty99
I think the problem is in your code, you might not be using strtok() right, maybe if you post some code it will help!

Quote:

---

Originally posted by CornedBee
strtok() doesn't destroy the string, but it isn't properly reentrant and not thread-safe unless programmed that way.

---

Perhaps it's a matter of opinion and what your definition of "destroy" is, but strtok() replaces characters in your string with NULL bytes and that fits my description..

Pardon me, but what do you mean by "it isn't properly reentrant?"

>>strtok() doesn't destroy the string
Err.. yeah, it does.

>>what do you mean by "it isn't properly reentrant?"
Readme

Quote:

---

Originally posted by Hammer
>>strtok() doesn't destroy the string
Err.. yeah, it does.

---

Thank you.. Seeing two people in a row stating otherwise made me start feeling like I'd just entered the twilight zone...

Thanks for the link.

The evidence (and documentation) speaks for itself:

Code:

---

#include <iostream>
using namespace std;

int main(void)
{
  char buf[] = "This is a string";
  char *p;
 
  cout <<"buf is >" <<buf <<"<" <<endl;
  p = strtok (buf, " ");
  cout <<"p is >" <<p <<"<" <<endl;
  cout <<"buf is >" <<buf <<"<" <<endl;
}

/*
 * Program output:
buf is >This is a string<
p is >This<
buf is >This<
*/

---

Hmmm, I was sure it stored a copy internally.

Apparently it's only a pointer. Well, I never cared about strtok anyway.

Quote:

---

Originally posted by Hammer
[B>>what do you mean by "it isn't properly reentrant?"
Readme [/B]

---

I mean that you can only tokenize one string at a time.

Quote:

---

Originally posted by CornedBee
Hmmm, I was sure it stored a copy internally.

Apparently it's only a pointer. Well, I never cared about strtok anyway.

I mean that you can only tokenize one string at a time.

---

I was going to point out, but I'm sure that you have already realized that you can in fact tokenize more than one string at a time (although I don't know why you would intersperse the tokenizing of two strings).

FYI, the way strtok() works is it starts from the given address and moves up the array until a token is located. Once it is, it is replaced with a NULL character and a pointer to the character after the token is returned so that you can continue with your next call... Thus, the destroying of your source string.

Here's a string tokenizer function. I didn't test it, but i think it should work..

Code:

---

/*****************************\
 * String tokenizer          *
 * Input:                    *
 *  inStr                ->Input string  *
 *  p                                ->Position array *
 *  chDelim        ->Delimiter      *
\*****************************/
int fixInStr(char *inStr, short *p, char chDelim){
        char *p1=inStr;
        int i(-1);

        if(*p1==chDelim){        // Handle delimiter in pos. 0
                *p1=0;
                p[++i]=0;
        }
        while((p1=strchr(p1+1, chDelim))){        // Find all tokens
                *p1=0;
                p[++i]=(p1-inStr)+1;
        }
        p1=0;

        return(i+1);
}

// Test
int main(){
        char achTest[] = "Test|of|string|tokenizer";
        short posArray[10], i(0), j(0);
        if((i=strtok_1(achTest, &posArray, '|'))){
                cout<<"Result\n";
                while(j<i)
                        cout<<achTest[j++]<<endl;
        }
        return(0);
}

---

This is what I'm doing:

/*CODE

char* pToken = strtok(pMsg, "|");
if (pToken)
token1 = pToken;
else
token1 = "";

pToken = strtok(NULL, "|");
if (pToken)
token2 = pToken;
else
token2 = "";

pToken = strtok(NULL, "|");
if (pToken)
token3 = pToken;
else
token3 = "";

pToken = strtok(NULL, "|");
if (pToken)
token4 = pToken;
else
token4 = "";

*/CODE

My goal his is to allow for blank fields. When I have a pMsg missing one of its fields (i.e.: pMsg = |field2|field3|field4), I want token1 to get assigned the empty string instead of field2. strtok is skipping over that first delimiter and assigning field2 to token1 and field3 to token2 and so on. It's shifting everything up one. Not recognizing the blank field.

Thanks to all who have replied to this point. I appreciate it.

Then do it this way:

Code:

---

if(*pMsg=='|'){        // Handle blank #1 strings
        strtok(pMsg, "|");
        token1 = "";
}else
        token1 = strtok(pMsg, "|");
       
// Next
pToken = strtok(NULL, "|");
token2 = ((pToken-token1)>1)?pToken:"";
etc..

---

That last solution works fine for my first case of reading in a blank 1st field, but now if I have something like:

pMsg = field1|field2||field4

with a blank field in the middle somewhere, then everything is still hosing up after that first strtok.

token1 is now being assigned the empty string like I want it
token2 is getting field2
token3 is getting field4
token4 is NULL

I would think you could just do the same thing 4 times, but I believe the way strtok is working, pMsg isn't getting moved over. I'm terrible with words. If anyone understands what I'm saying, help is greatly appreciated. If not, I'll keep fighting until I get it figured out. Thanks

Troi oi...
Here, I just wrote this real quick. Maybe it'll work...

Code:

---

const char* pipeTokenize(const char *str, char *tokenized, int bufSize) {
  *tokenized = NULL;
  int size = 0;

  while (*str != NULL && size < bufSize) {

    if (*str == '|') {
      ++str;
      break;
    }

    ++size;
    *tokenized = *str;
    *++tokenized = NULL;

    ++str;
  }

  return *str == NULL ? NULL : str;
}

int main() {
  string delimited = "|field2||field4";
  char buffer[256];

  const  char *p  = delimited.c_str();
  while (p != NULL) {
    p = pipeTokenize(p, buffer, 256);
   
    cout << "Hey dummy, here's a field: " << buffer << endl;
  }

  return 0;
}

---

What's wrong with my function splitting to a vector of strings? After all we're using C++ here, right?

This was on the AP Computer Science exam I took today, sorry if it's already been said.

Quote:

---

Originally posted by CheesyMoo
This was on the AP Computer Science exam I took today, sorry if it's already been said.

---

Yeah, several people have come on the board telling us about your AP computer science exam... It's funny you didn't notice.

Can I have a hug?

I had the same problem because I was reading in from a tab delimited file.... Here is what I did... I used the CStrings replace function to replace "\t\t" with "\t \t" so that there would be at least one space in between tabs and therefore the toke would be picked up as a blank space. Later if I found a blank space I would just copy an empty sting into that token. Of course this means that you have to move the buffer into a string and then back into the buffer. I am not quite sure what the equivalent functions of CString are in string to accomplish this but I am sure that they exist.

Code:

---


while( cFile.ReadString( strInline ) == TRUE)
{
      //Trime all spaces to the right in case of empty string
      strInline.TrimRight();
                       
    if( strInline <= " ")
        continue;

    //Remove all back to back tabs with space in between them
    while( strInline.Find("\t\t") > -1)
            strInline.Replace("\t\t", "\t \t");

    char *buffer;
    char *pToken;

    //Create a new buffer
    buffer = new char[ strInline.GetLength()+1];

    strcpy(buffer, strInline.GetBuffer( strInline.GetLength()+1 ));

    pToken = strtok(buffer, "\t");
    int i = 0;
    CString strTokens[31];

    strTokens[i++] = pToken;
    if(strTokens[i-1] <= " ")
    strTokens[i-1] = _T("");

  while( pToken != NULL && i < 30)
  {
          /* Get next token: */
          pToken = strtok( NULL, "\t");
          strTokens[i++] = pToken;

          if( strTokens[i-1] <= " ")
        strTokens[i-1] = _T("");
  }

---

I guess i'll have to write the whole thing:

Code:

---

if(*pMsg=='|'){        // Handle blank #1 strings
        strtok(pMsg, "|");
        token1 = "";
}else
        token1 = strtok(pMsg, "|");
       
// Next
pToken = strtok(NULL, "|");
token2 = ((pToken-token1)>1)?pToken:"";
// And next
pToken = strtok(NULL, "|");
token3 = ((pToken-token2)>1)?pToken:"";
// And next
pToken = strtok(NULL, "|");
token4 = ((pToken-token3)>1)?pToken:"";

---

This should handle empty strings too. I thought you'd see how to do continue the function..