Stack issues when calling a COM library
Hi, I'm trying to call some functions in an external COM library, but for some reason after the functions are called, the stack is screwed up and my program crashes.
The way the SDK library is set up, you first do a LoadLibrary(), then a GetProcAddress() on a particular function. You then pass that function a pointer to a vtable, which is filled with a bunch of pointers to functions within the DLL.
Well, the vtable is getting filled with the correct function pointers... but when I call the library functions, they both a) don't work properly and b ) mess up the stack, causing my program to crash.
I've never worked dealt with COM libraries before, so I don't know if there's something special I need to do. I tried modifying the function pointer definitions to be either __stdcall or __cdecl, but neither fix the problem.
I tried compiling the program both in gcc and VC++, it works on neither. VC++ gives me Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention even though I have verified that the calling convention is correct.
Here's a short example of what I'm doing:
Code:
#include <stdio.h>
#include <windows.h>
#include "tgsdkx.h"
typedef HRESULT ( WINAPI *GetInterfaceProc )(LPSTR, LPSTR, LPVOID*);
struct textCapLib {
HINSTANCE lib;
ITextGRABSDK *sdk;
} textCapLib;
int main(int argc, char **argv) {
BSTR text = NULL;
HINSTANCE lib = textCapLib.lib = LoadLibrary("tgsdk.dll");
if (lib == NULL) {
return 1;
}
/* Now get the address of the GetInterface() function. */
GetInterfaceProc GetInterface;
if ((GetInterface = (GetInterfaceProc)GetProcAddress(lib, "GetInterface")) == NULL) {
goto ERROR_LIB;
}
/* Now call GetInterface(). It'll fill up our SDK structure with function pointers and stuff. */
if (FAILED(GetInterface("", "", (LPVOID *)&(textCapLib.sdk)))) {
goto ERROR_LIB;
}
/* Call the functions. The function pointers are set properly, as the proper functions
* are being called, but the functions appear to be looking for arguments in the wrong
* location, and the stack is messed up after calling them as well. */
(textCapLib.sdk)->lpVtbl->CaptureFromHWND((textCapLib.sdk), (INT_PTR)GetDesktopWindow(), &text);
return 0;
ERROR_LIB:
FreeLibrary(lib);
textCapLib.lib = NULL;
return 1;
}
And here is the tgsdkx.h file. See the code starting at line 810 in particular for the vtable definition.
One last thing: I ran my program through a debugger; it appears that the library functions are looking for arguments in the wrong position on the stack.