Thread: Really weird stack smashing problem

Doesn't make sense - as the storage for *rotated is obtained from the heap not stack.

Am I the only one having this problem?

Please try to compile and run using:
rotated[32] = 0x01;
for example.

I can't understand how can stack smashing detection get triggered by that byte of data...

I didn't look at all the code. But this:

Code:

str = malloc(strlen(enc) - 2);
strncpy(str, (enc + 2), (strlen(enc) - 2));
for (i=0; i<strlen(str); i++)

is wrong.

strncpy() will not necessarily create a string. If there is no null byte in the first n bytes of the source, where n is the third argument, no null byte will be placed in the target. Without a null byte, you haven't got a string. Thus calling strlen(str) is invalid.

You'll want to allocate one extra byte, and then manually place the null character.

Incidentally, string literals already contain a null character, so unless you want two at the end of your strings, you needn't add one yourself.

Originally Posted by cas

Incidentally, string literals already contain a null character, so unless you want two at the end of your strings, you needn't add one yourself.

strlen(enc) wouldn't work either...

NULL byte was added on purpose at the end of those string literals.

Indeed replacing (strlen(enc) - 2) and strlen(str) with 128 DOES solve the problem!

But I still can't figure out why does that exact byte at rotated[32] cause all the trouble...

strlen(enc) wouldn't work either...

It will work if you pass in a string as the first argument to decKey(), and it appears that you are. You're passing eKeys[i], where i is in the range 0 <= i < 256; since eKeys has 256 entries, there are no problems with the array indexing. Each element of the array is (a pointer to) a string; each has two null characters, in fact (only the first is needed, as I implied above).

So inside of decKey(), you're doing strlen(eKeys[0]), [1], [2], ... This is fine.

But I still can't figure out why does that exact byte at rotated[32] cause all the trouble...

You could probably figure it out if you had the desire and time. When you have undefined behavior (as you did after using the strncpy() call), anything is possible. For what it's worth, setting rotated[32] to 1 didn't invoke any stack smashing protection for me (I built with gcc -fstack-protector-all). But you can try to trace what is being overwritten in decKey(); the “for” loop in there will go beyond the end of the string (perhaps) and thus you'll potentially be writing further into “dec” than you expected. You might, in fact, overflow “dec”, causing weird problems elsewhere.

Undefined behavior can manifest itself in odd ways, causing problems you may never have expected. In short, since you now know the root of the problem, there's no real need to find out why the stack smasher was getting involved. It was probably some cascading problem from a buffer overrun, and is not especially important, except for academic reasons.

Thank you all for your help!

Changed: strncpy(str, (enc + 2), (strlen(enc) - 2));
To: strcpy(str, (enc + 2));

...now works as desired.


P.S.: Couldn't imagine strncpy would cause such a problem...