Thread: Bizarre and Frightening Bug

This bug only pops up occasionally. It was very hard for me to track down and verify that it was happening at all, and then reproduce it.

This is complicated by the fact that this is a CGI (web-based) program, so I can only test it through a web browser.


I have a function that does some work on variables passed by address. Somehow, between the time the function returns, and the time those variables are used, the data is changing.

This is basically what's happening:

Code:

void my_function( char *data, int *length ) {

  // various stuff is done to data
  // length may be changed and is set 

  printf( "before return: %ld characters: %s\n", *length, data );
}

int other_function() {
  char *data;
  int length;

  // stuff, the data pointer gets malloc'd, etc

  my_function( data, &length );
  printf( "after return: %ld characters: %s\n", length, data );

}

At execution, one would expect the "before" and "after" printed lines to be the same ... and usually they are.

However, rarely, they aren't. After some painful work, I discovered that the data pointer is getting changed to point at a different address.

How on earth might that be happening, and what sort of bug should I be looking for?

My first thought was that I have a buffer overrun someplace, but I don't know where to begin to track that down. My understanding is that when a function returns, nothing gets done except the stack variables associated with it get released ... but I can't for the life of me imagine why that would corrupt the value of a pointer declared elsewhere.

The program is not multithreaded.

Help ...

Originally Posted by Salem

Do you have access to the machine the code is running on?

Sort of.

The program runs on a server I have no control over and no access to, except to upload the program. No shell access. Sadly, it's a Windows IIS machine.

It was originally written to run on Linux / Apache. Never had trouble. It's fairly long and complex (from my amateurish perspective anyway) so to ease porting to Windows, I'm compiling it in a cywgin environment on a Windows workstation. The workstation has IIS running, so during testing I'm calling the program through a web browser pointed at 127.0.0.1/the_program.cgi

Turned out to be pretty easy.

I have no idea how to get gdb in the cygwin environment attached to a CGI program being executed (for a couple hundred milliseconds) by the underlying Windows system as the IUSR user.

Debugging to get this far has been a painful, tedious process of adding a bunch of printf statements to see what variables are doing at various points in the program, and then hitting the program with the web browser, and reading the printed statements in the browser.

Originally Posted by Salem

A couple of points.
1. You can use gdb to attach to a running process.

2. You can set up data breakpoints to watch when a memory location is modified.

3. You could try linking against "electric fence", which will trap at the moment any abuse of malloc'ed memory happens.

That looks like an interesting tool. I know buffer overrun errors can cause insidiously weird behavior, but I still can't wrap my brain around how a variable can have one value before a function returns, and the very next instant, be something different. No matter what the program is doing elsewhere.


I also have to confess that I'm an amateur. I took a few lower and upper division CS courses in college out of interest, so I like to think I have mostly good habits and basic understanding of how to write good code and how to stay out of trouble, but I'm certainly not a pro. It has probably been 15 years since I've written something complicated enough that I needed a debugger to fix it. I sort of dread learning how to use gdb again. :-)

Right now I'm going to take the web server out of the equation, and compile a version that doesn't need user input, just goes down the same path that is causing the problems. Once I've got something running from the command line that duplicates the bug, maybe I'll spend some time re-learning the ins and outs of gdb.

Thank you for the help.

Gah. Solved.

A 3rd party library I'm using had a call to realloc in it, that only got called on the very rare occasion when data needed to grow. And sometimes realloc returns a pointer to the same spot, and sometimes it doesn't. So sometimes data ends up pointing to a different block of memory, work gets done on that, everything's cool, but when the function returns, the pointer in the calling function is still aimed at the original block of memory. Because I passed the value of the pointer to my_function(), not a pointer to that pointer.

Man, sometimes I wish I was smarter. There's 7 hours of my life I'll never get back. But I learned something. :-)

Thanks for the reply. Still going to check out that efence thing.