Really weird stack smashing problem

This is a discussion on Really weird stack smashing problem within the C Programming forums, part of the General Programming Boards category; $uname -a Linux localhost 2.6.28-18-generic #60-Ubuntu SMP Fri Mar 12 04:40:52 UTC 2010 i686 GNU/Linux $$ gcc --version gcc (Ubuntu ...

  1. #1
    Registered User
    Join Date
    Jul 2010
    Posts
    20

    Question Really weird stack smashing problem

    $uname -a
    Linux localhost 2.6.28-18-generic #60-Ubuntu SMP Fri Mar 12 04:40:52 UTC 2010 i686 GNU/Linux
    $$ gcc --version
    gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3


    Feel free to experiment with the commented line.
    If you set rotated[32]=0x01 for example, it triggers stack smashing detection.
    Functions and main are at the bottom...

    CODE DELETED
    http://cboard.cprogramming.com/c-pro...e-posting.html
    READ THEM
    Last edited by Salem; 07-23-2010 at 10:26 PM. Reason: Deleted hacking code

  2. #2
    Registered User
    Join Date
    Oct 2008
    Location
    TX
    Posts
    2,047
    Doesn't make sense - as the storage for *rotated is obtained from the heap not stack.

  3. #3
    Registered User
    Join Date
    Jul 2010
    Posts
    20
    Am I the only one having this problem?

    Please try to compile and run using:
    rotated[32] = 0x01;
    for example.

    I can't understand how can stack smashing detection get triggered by that byte of data...

  4. #4
    cas
    cas is offline
    Registered User
    Join Date
    Sep 2007
    Posts
    957
    I didn't look at all the code. But this:
    Code:
    str = malloc(strlen(enc) - 2);
    strncpy(str, (enc + 2), (strlen(enc) - 2));
    for (i=0; i<strlen(str); i++)
    is wrong.

    strncpy() will not necessarily create a string. If there is no null byte in the first n bytes of the source, where n is the third argument, no null byte will be placed in the target. Without a null byte, you haven't got a string. Thus calling strlen(str) is invalid.

    You'll want to allocate one extra byte, and then manually place the null character.

    Incidentally, string literals already contain a null character, so unless you want two at the end of your strings, you needn't add one yourself.

  5. #5
    Registered User
    Join Date
    Jul 2010
    Posts
    20

    Thumbs up

    Quote Originally Posted by cas View Post
    Incidentally, string literals already contain a null character, so unless you want two at the end of your strings, you needn't add one yourself.
    strlen(enc) wouldn't work either...

    NULL byte was added on purpose at the end of those string literals.

    Indeed replacing (strlen(enc) - 2) and strlen(str) with 128 DOES solve the problem!

    But I still can't figure out why does that exact byte at rotated[32] cause all the trouble...

  6. #6
    cas
    cas is offline
    Registered User
    Join Date
    Sep 2007
    Posts
    957
    strlen(enc) wouldn't work either...
    It will work if you pass in a string as the first argument to decKey(), and it appears that you are. You're passing eKeys[i], where i is in the range 0 <= i < 256; since eKeys has 256 entries, there are no problems with the array indexing. Each element of the array is (a pointer to) a string; each has two null characters, in fact (only the first is needed, as I implied above).

    So inside of decKey(), you're doing strlen(eKeys[0]), [1], [2], ... This is fine.
    But I still can't figure out why does that exact byte at rotated[32] cause all the trouble...
    You could probably figure it out if you had the desire and time. When you have undefined behavior (as you did after using the strncpy() call), anything is possible. For what it's worth, setting rotated[32] to 1 didn't invoke any stack smashing protection for me (I built with gcc -fstack-protector-all). But you can try to trace what is being overwritten in decKey(); the “for” loop in there will go beyond the end of the string (perhaps) and thus you'll potentially be writing further into “dec” than you expected. You might, in fact, overflow “dec”, causing weird problems elsewhere.

    Undefined behavior can manifest itself in odd ways, causing problems you may never have expected. In short, since you now know the root of the problem, there's no real need to find out why the stack smasher was getting involved. It was probably some cascading problem from a buffer overrun, and is not especially important, except for academic reasons.

  7. #7
    Registered User
    Join Date
    Jul 2010
    Posts
    20
    Thank you all for your help!

    Changed: strncpy(str, (enc + 2), (strlen(enc) - 2));
    To: strcpy(str, (enc + 2));

    ...now works as desired.


    P.S.: Couldn't imagine strncpy would cause such a problem...

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. *** stack smashing detected ***
    By chakra in forum C Programming
    Replies: 2
    Last Post: 06-09-2009, 09:12 PM
  2. *** stack smashing detected ***
    By Martin_HS in forum C Programming
    Replies: 9
    Last Post: 05-29-2009, 04:01 AM
  3. Weird access violation error with stack
    By ryeguy in forum C++ Programming
    Replies: 2
    Last Post: 02-29-2008, 01:53 AM
  4. Stack Program Here
    By Troll_King in forum C Programming
    Replies: 7
    Last Post: 10-15-2001, 05:36 PM
  5. Array Stack Problem
    By Drew in forum C++ Programming
    Replies: 3
    Last Post: 09-04-2001, 06:58 PM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21