Stack issues when calling a COM library

Hi, I'm trying to call some functions in an external COM library, but for some reason after the functions are called, the stack is screwed up and my program crashes.

The way the SDK library is set up, you first do a LoadLibrary(), then a GetProcAddress() on a particular function. You then pass that function a pointer to a vtable, which is filled with a bunch of pointers to functions within the DLL.

Well, the vtable is getting filled with the correct function pointers... but when I call the library functions, they both a) don't work properly and b ) mess up the stack, causing my program to crash.

I've never worked dealt with COM libraries before, so I don't know if there's something special I need to do. I tried modifying the function pointer definitions to be either __stdcall or __cdecl, but neither fix the problem.

I tried compiling the program both in gcc and VC++, it works on neither. VC++ gives me Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention even though I have verified that the calling convention is correct.

Here's a short example of what I'm doing:

Code:

---

#include <stdio.h>
#include <windows.h>
#include "tgsdkx.h"

typedef HRESULT ( WINAPI *GetInterfaceProc )(LPSTR, LPSTR, LPVOID*);

struct textCapLib {

    HINSTANCE lib;
    ITextGRABSDK *sdk;
} textCapLib;

int main(int argc, char **argv) {

    BSTR text = NULL;
    HINSTANCE lib = textCapLib.lib = LoadLibrary("tgsdk.dll");
    if (lib == NULL) {
      return 1;
    }

    /* Now get the address of the GetInterface() function. */
    GetInterfaceProc GetInterface;
    if ((GetInterface = (GetInterfaceProc)GetProcAddress(lib, "GetInterface")) == NULL) {
      goto ERROR_LIB;
    }

    /* Now call GetInterface(). It'll fill up our SDK structure with function pointers and stuff. */
    if (FAILED(GetInterface("", "", (LPVOID *)&(textCapLib.sdk)))) {
      goto ERROR_LIB;
    }

    /* Call the functions. The function pointers are set properly, as the proper functions
      * are being called, but the functions appear to be looking for arguments in the wrong
      * location, and the stack is messed up after calling them as well. */
    (textCapLib.sdk)->lpVtbl->CaptureFromHWND((textCapLib.sdk), (INT_PTR)GetDesktopWindow(), &text);
 
    return 0;

 ERROR_LIB:
    FreeLibrary(lib);
    textCapLib.lib = NULL;
    return 1;
}

---

And here is the tgsdkx.h file. See the code starting at line 810 in particular for the vtable definition.

One last thing: I ran my program through a debugger; it appears that the library functions are looking for arguments in the wrong position on the stack.

You're probably using the wrong calling convention. Try:

typedef HRESULT ( STDMETHODCALLTYPE *GetInterfaceProc )(LPSTR, LPSTR, LPVOID*);

EDIT:
Sorry, I thought this was happening after calling the GetInterfaceProc (even so, being a com object, it might actually have to use the calling convention you posted, anyway, so my suggestion probably won't help any).

But in short, the only way that CaptureFromHWND would expect the arguments differently is if the header declares the wrong calling convention (unlikely).

That's not right. You not supposed to access the v-table member ever - under any circumstances. That is internal C++ compiler specific functionality that can vary from compiler to compiler and even between versions. (Frankly I'm amazed that any compiler gives it such a legible name that doesn't begin with underscores)
COM is language agnostic; It doesn't have any knowledge of the fact that C++ uses v-tables, and hence your calling of this function has nothing to do with v-tables directly.

The GetInterface function, which I assume is some kind of wrapper around CoCreateInstance would give you an interface pointer (that would seem to be what an ITextGRABSDK is), which by the way will have been AddRef'd (you should call Release on it when you're done with it).

You should simply be calling it like this:

Code:

---

textCapLib.sdk->CaptureFromHWND((INT_PTR)GetDesktopWindow(), &text);

---

It only has two parameters, as I see here: TextGRAB SDK Documentation | ITextGRABSDK::CaptureFromHWND - Renovation Software

Don't forget to SysFreeString that BSTR as well, when you're done with that.

I see your code in a pastebin as well.

Oh hey but wait a minute, this is C! So this lpVtbl really is what you call the interface functions through?!?!? So the first parameter is the "this" pointer...

Man I've never done COM through C and would never want to! (Though I've done it for many years through C++)
Any reason you can't use C++? It's a lot easier!

Well heck, just make sure you call Release and SysFreeString afterwards and maybe someone else who's done this in C can help you. I feel dirty now - having had to think about C.