(Used up all of that header...)
First off, a pre-emptive apology. This question *could* fit in the C, C++, Game, and Windows programming forums. I'm not sure which would be most appropriate here, so I chose the more generic 'C'. If it would be better somewhere else, by all means move it.
And secondly, I am not a Windows programmer, and this problem only affects Windows builds of my project. Because of that, it's entirely possible I've missed something common, or I've incorrectly implemented something I thought would fix it (I learned for instance that __declspec(dllexport) is picky about its placement, but not so picky that it'll throw up a warning if misplaced). I'm asking for help, and I'll look into anything, even if I'm pretty sure I already tried it.
I'm working on an engine. Not a big deal I guess; there are plenty of engines to go around. The engine however was originally built on VS6 by others, almost certainly with Frame Pointer Omission (since that's a default option for maximum optimization there). In addition, mods were written for the engine, and for all but the most determined of writers would also have been compiled under VS6, with Frame Pointer Omission.
Fast forward to today. The engine has changed and been upgraded, and VS6 is no longer an option for building. I'd like to get it to build with MingW, but on mingw, it tends to crash horribly when mods are accessed. If I build my OWN mod withOUT frame pointer omission (using VS6 even), everything works perfectly--it's only with that option that the whole program seems to drive itself into the weeds the instant something complicated is called. I also tried compiling using gcc's -fomit-frame-pointer flag. This resulted in a crash when calling LoadLibrary(), and I don't know what to make of that...
Things get a little bit complicated when you consider how the engine and the DLLs call each other. Only one function is actually extracted from the DLL--an init function. The engine passes the DLL two struct *s--one filled with pointers to functions in the engine, and one that the engine expects the mod to copy its own function pointers into for the engine to call. And they do intertwine; mods may call the engine who'll call the mod who'll call the engine...
I hope the description gets the idea across. The code related to setting up the mods is just shy of 2000 lines, about 3/4ths of which exists just to make sure various structs aren't misaligned by any tinkering. Naturally, that probably shouldn't be pasted in.
I don't know what I need to do to make the newer engine safely call the older code. Do I need to somehow tell my compiler 'these functions don't store stack pointers, fix that'? Or 'this function is called by a function that doesn't store stack pointers normally'? Can that even be done? Or will I have to use -fomit-frame-pointer, and figure out why mingw doesn't like that? Although not pleasant to think about, could manually pushing and popping registers help? Or will I just have to declare mingw a lost cause?
For that matter, just how common is this?
