Thread: Problem with STRTOK function

  1. #1
    Registered User
    Join Date
    Oct 2010
    Posts
    4

    Problem with STRTOK function

    i have a problem using strtok.
    i'm trying to write a c program that counts the number of words in a sentence.
    i'm using strtok , but for some reason i get an exception while the call for strtok is being provoked.

    here is my code:

    Code:
    int countWords(char str[]){
        int counter;
        char * tmp;
    
        counter=0;
        printf("the string is: %s\n", str);
        tmp = strtok(str," ");
        while (tmp != NULL){
            counter++;
            printf("%s\n", tmp);
            tmp = strtok(NULL," ");
        }
        return counter;
    }
    Code:
    int main(){
        int s;
    
        s = countWords2("amir is bla bla bla ");
        printf("%d", s);
        system("pause");
        return 0;
    }
    when i define the string in the fumction itself the problem doesent occur.
    so i'm guessing the problemis with the pointer i'm sending to the function, but i cant really understand how it is a problem and how do i solve it.

    waiting for your wise replies

  2. #2
    Registered User ssharish2005's Avatar
    Join Date
    Sep 2005
    Location
    Cambridge, UK
    Posts
    1,732
    Your trying to tokensie the Read only string. strtok will modify the string. Which mean the string memeory allocation shoudl have have write access as well. When you do this

    Code:
    s = countWords2("amir is bla bla bla ");
    The string is constant here. When strtok modifes a constant string you get ane seg fault.

    Try this instead.
    Code:
    char str[] = "amir is bla bla bla";
    s = countWords2(str);
    And also I would try avoid using strtok. As it modifies the original string. Unless you have a backup of the orginal string, you will never be able retrive the orignal string back.

    ssharish
    Life is like riding a bicycle. To keep your balance you must keep moving - Einstein

  3. #3
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    strtok() MODIFIES the string.

    A "string like this" is a constant, in read-only memory. Being read-only, you can't change it.

    Do something like
    Code:
    char msg[] = "this is a string";
    myfunc( msg );
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  4. #4
    Registered User
    Join Date
    Oct 2010
    Posts
    107
    You could also have countWords make a copy of the string internally:

    Code:
    int countWords(char str[]){
        char* _str = strdup(str); // Allocates a new string and copies str into it.
    
    
        int counter;
        char * tmp;
    
        counter=0;
        printf("the string is: %s\n", _str);
        tmp = strtok(_str," ");
        while (tmp != NULL){
            counter++;
            printf("%s\n", tmp);
            tmp = strtok(NULL," ");
        }
    
        free(_str); // Free the string we allocated; it was just temporary.
    
        return counter;
    }
    In general, there are problems with using strtok, as it has been pointed out it modifies the original string which can be bad in some single-threaded applications. It's also not reentrant, so it can't be used when multiple threads access a string.

  5. #5
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,413
    QuadraticFighte's suggestion is valid, but be warned that strdup is not part of the C standard library.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  6. #6
    Registered User
    Join Date
    Oct 2010
    Posts
    4

    mmmm....

    noob such c am i (as yoda would say.....)

    thank you for the help!

  7. #7
    Registered User
    Join Date
    Oct 2010
    Posts
    107
    Quote Originally Posted by laserlight View Post
    QuadraticFighte's suggestion is valid, but be warned that strdup is not part of the C standard library.
    Oh, I didn't know that. Why not, is there some kind of problem with it? Or they just didn't see a need for it? If we care about using only things in the standard library, we can try:

    Code:
        int len = strlen(str);
        char* _str = calloc(len + 1, sizeof(char));
        assert(_str != NULL); // Let's assume it works.
        strncpy(_str, str, len);

  8. #8
    Registered User
    Join Date
    Oct 2010
    Posts
    4

    strdup worked for me.....

    i tried the code u wrote with strdup and it worked just fine.

    but i'm really not sure what compiler runs it.
    i am using VS 2010 with a c++ project but my file are all *.c and i defined the project to compile as a c program.

    not sure if it's c ot c++ , but i', only doing this for practice so i doesnt matter to me

    anyway, thx again for the help guys!

  9. #9
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,413
    Quote Originally Posted by QuadraticFighte
    Why not, is there some kind of problem with it?
    Not that I know of. The reason is probably just a matter of historical oversight.

    Quote Originally Posted by dardar
    i am using VS 2010 with a c++ project but my file are all *.c and i defined the project to compile as a c program.
    That is good: the compiler should be compiling as C.

    Quote Originally Posted by dardar
    not sure if it's c ot c++ , but i', only doing this for practice so i doesnt matter to me
    C and C++ are two different languages. If you use C++ specific constructs for practice, you may find that when you really need to write a C program, it does not compile.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  10. #10
    Registered User Char*Pntr's Avatar
    Join Date
    Sep 2007
    Location
    Lathrop, CA
    Posts
    198
    Quote Originally Posted by dardar View Post
    i am using VS 2010 with a c++ project but my file are all *.c and i defined the project to compile as a c program.
    darder so am I. However earlier today on another thread, I found out that there could
    be possibly an error in which we have our compilers set up. ( we are both using MS VS 2010 )
    It was pointed out earlier due to some errors I was getting on legitimate C code, that it may
    be configured (still) to compile C++. I'm checking into this.

    not sure if it's c ot c++ , but i', only doing this for practice so i doesnt matter to me
    EDIT: It should matter to you. And it definitely matters to the rest of the board members here, who read our posts.

    Don't make the same mistake that I may have made.
    Last edited by Char*Pntr; 10-16-2010 at 01:30 PM. Reason: clarification

  11. #11
    Registered User
    Join Date
    Oct 2010
    Posts
    4
    darder so am I. However earlier today on another thread, I found out that there could
    be possibly an error in which we have our compilers set up. ( we are both using MS VS 2010 )
    It was pointed out earlier due to some errors I was getting on legitimate C code, that it may
    be configured (still) to compile C++. I'm checking into this.
    can u post the link to that thread?

  12. #12
    Registered User
    Join Date
    Oct 2010
    Posts
    107
    Quote Originally Posted by Char*Pntr View Post
    EDIT: It should matter to you. And it definitely matters to the rest of the board.

    Don't make the same mistake that I may have made.
    QFT

    For one thing, we have a C++ section of the board, so C++ questions should go there first. That way, moderators don't have to move the thread (and so you won't be like WTF happened to my thread?) And that way, you don't get warned or whatever.

    For another, they are like laserlight said, fundamentally different languages. We have no classes in C. Also, as I discovered to my chagrin a couple days ago, in C++ we can overload functions, but in C we can't. Those are just two fundamental differences.

  13. #13
    Registered User
    Join Date
    Sep 2007
    Posts
    1,012
    Quote Originally Posted by QuadraticFighte View Post
    Oh, I didn't know that. Why not, is there some kind of problem with it? Or they just didn't see a need for it? If we care about using only things in the standard library, we can try:

    Code:
        int len = strlen(str);
        char* _str = calloc(len + 1, sizeof(char));
        assert(_str != NULL); // Let's assume it works.
        strncpy(_str, str, len);
    I would recommend avoiding strncpy() as much as possible. It gains you nothing here, because you know you've allocated enough space, so the strcpy() will not overflow. In cases where you don't know whether there's enough space, you have to remember to null terminate the resulting string, because strncpy() won't necessarily do it. It's a function designed with a specific use in mind, and creating strings is not it. It's unfortunately named and, really, should not be in the standard as defined.

  14. #14
    Registered User
    Join Date
    Oct 2010
    Posts
    107
    Quote Originally Posted by cas View Post
    I would recommend avoiding strncpy() as much as possible. It gains you nothing here, because you know you've allocated enough space, so the strcpy() will not overflow. In cases where you don't know whether there's enough space, you have to remember to null terminate the resulting string, because strncpy() won't necessarily do it. It's a function designed with a specific use in mind, and creating strings is not it. It's unfortunately named and, really, should not be in the standard as defined.
    What would you recommend I use instead? Besides, there's nothing wrong with what I wrote, because I alloced len+1 spaced (which is initalized to zero) and then I copy at most len bytes, guaranteeing NULL termination (Because of how calloc works).

  15. #15
    Registered User
    Join Date
    Sep 2007
    Posts
    1,012
    Quote Originally Posted by QuadraticFighte View Post
    What would you recommend I use instead? Besides, there's nothing wrong with what I wrote, because I alloced len+1 spaced (which is initalized to zero) and then I copy at most len bytes, guaranteeing NULL termination (Because of how calloc works).
    As I said, strcpy() is fine here.

    I didn't say that strncpy() wouldn't work here. I said it's pointless: it gains you nothing over strcpy() in this example. Moreover, it's probably slightly slower because it has to both count and zero bytes (plus the memory here was zeroed beforehand).

    The problem with strncpy() is not that it can't be used properly; it's that it tends to be treated as a "safe strcpy()" function, which it is not. It requires extra work to get right. You have to manually null terminate the resulting string. If you already know the target is large enough, though, why would you bother with strncpy()?

    I'm not arguing that strncpy() should be banned from use (although it probably should), but that it should be avoided unless it's really necessary, in large part because it's defective for how it's used in modern times. Something like OpenBSD's strlcpy() would be a better choice, and as it's under the BSD license, you can easily incorporate it into your code (just rename it...)

    strncpy() was not designed to create strings, and the fact that it can be part of a string-creating solution doesn't necessarily mean it's the best option.

    At the very least, I'd recommend a wrapper that does something like:
    Code:
    #define COPYSTR(t, s, n) (strncpy((t), (s), (n) - 1)[(n) - 1] = 0)
    At least now when COPYSTR() is called, it's an actual string-creation “function” in and of itself. Assuming you don't pass 0 as the buffer size...

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Tough problem with function pointers
    By Montejo in forum C Programming
    Replies: 10
    Last Post: 12-22-2009, 01:17 AM
  2. Please Help - Problem with Compilers
    By toonlover in forum C++ Programming
    Replies: 5
    Last Post: 07-23-2005, 10:03 AM
  3. Problem with function pointers
    By vNvNation in forum C++ Programming
    Replies: 4
    Last Post: 06-13-2004, 06:49 AM
  4. Replies: 5
    Last Post: 02-08-2003, 07:42 PM
  5. I need help with passing pointers in function calls
    By vien_mti in forum C Programming
    Replies: 3
    Last Post: 04-24-2002, 10:00 AM