Thread: Stack issues when calling a COM library

  1. #1
    Registered User
    Join Date
    May 2009
    Posts
    1

    Question Stack issues when calling a COM library

    Hi, I'm trying to call some functions in an external COM library, but for some reason after the functions are called, the stack is screwed up and my program crashes.

    The way the SDK library is set up, you first do a LoadLibrary(), then a GetProcAddress() on a particular function. You then pass that function a pointer to a vtable, which is filled with a bunch of pointers to functions within the DLL.

    Well, the vtable is getting filled with the correct function pointers... but when I call the library functions, they both a) don't work properly and b ) mess up the stack, causing my program to crash.

    I've never worked dealt with COM libraries before, so I don't know if there's something special I need to do. I tried modifying the function pointer definitions to be either __stdcall or __cdecl, but neither fix the problem.

    I tried compiling the program both in gcc and VC++, it works on neither. VC++ gives me Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention even though I have verified that the calling convention is correct.

    Here's a short example of what I'm doing:

    Code:
    #include <stdio.h>
    #include <windows.h>
    #include "tgsdkx.h"
    
    typedef HRESULT ( WINAPI *GetInterfaceProc )(LPSTR, LPSTR, LPVOID*);
    
    struct textCapLib {
    
         HINSTANCE lib;
         ITextGRABSDK *sdk;
    } textCapLib;
    
    int main(int argc, char **argv) {
    
         BSTR text = NULL;
         HINSTANCE lib = textCapLib.lib = LoadLibrary("tgsdk.dll");
         if (lib == NULL) {
          return 1;
         }
    
         /* Now get the address of the GetInterface() function. */
         GetInterfaceProc GetInterface;
         if ((GetInterface = (GetInterfaceProc)GetProcAddress(lib, "GetInterface")) == NULL) {
          goto ERROR_LIB;
         }
    
         /* Now call GetInterface(). It'll fill up our SDK structure with function pointers and stuff. */
         if (FAILED(GetInterface("", "", (LPVOID *)&(textCapLib.sdk)))) {
          goto ERROR_LIB;
         }
    
         /* Call the functions. The function pointers are set properly, as the proper functions
          * are being called, but the functions appear to be looking for arguments in the wrong
          * location, and the stack is messed up after calling them as well. */
         (textCapLib.sdk)->lpVtbl->CaptureFromHWND((textCapLib.sdk), (INT_PTR)GetDesktopWindow(), &text);
       
         return 0;
    
     ERROR_LIB:
         FreeLibrary(lib);
         textCapLib.lib = NULL;
         return 1;
    }
    And here is the tgsdkx.h file. See the code starting at line 810 in particular for the vtable definition.

    One last thing: I ran my program through a debugger; it appears that the library functions are looking for arguments in the wrong position on the stack.

  2. #2
    Guest Sebastiani's Avatar
    Join Date
    Aug 2001
    Location
    Waterloo, Texas
    Posts
    5,708
    You're probably using the wrong calling convention. Try:

    typedef HRESULT ( STDMETHODCALLTYPE *GetInterfaceProc )(LPSTR, LPSTR, LPVOID*);

    EDIT:
    Sorry, I thought this was happening after calling the GetInterfaceProc (even so, being a com object, it might actually have to use the calling convention you posted, anyway, so my suggestion probably won't help any).

    But in short, the only way that CaptureFromHWND would expect the arguments differently is if the header declares the wrong calling convention (unlikely).
    Last edited by Sebastiani; 05-31-2009 at 08:54 PM.
    Code:
    #include <cmath>
    #include <complex>
    bool euler_flip(bool value)
    {
        return std::pow
        (
            std::complex<float>(std::exp(1.0)), 
            std::complex<float>(0, 1) 
            * std::complex<float>(std::atan(1.0)
            *(1 << (value + 2)))
        ).real() < 0;
    }

  3. #3
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,318
    That's not right. You not supposed to access the v-table member ever - under any circumstances. That is internal C++ compiler specific functionality that can vary from compiler to compiler and even between versions. (Frankly I'm amazed that any compiler gives it such a legible name that doesn't begin with underscores)
    COM is language agnostic; It doesn't have any knowledge of the fact that C++ uses v-tables, and hence your calling of this function has nothing to do with v-tables directly.

    The GetInterface function, which I assume is some kind of wrapper around CoCreateInstance would give you an interface pointer (that would seem to be what an ITextGRABSDK is), which by the way will have been AddRef'd (you should call Release on it when you're done with it).

    You should simply be calling it like this:
    Code:
    textCapLib.sdk->CaptureFromHWND((INT_PTR)GetDesktopWindow(), &text);
    It only has two parameters, as I see here: TextGRAB SDK Documentation | ITextGRABSDK::CaptureFromHWND - Renovation Software

    Don't forget to SysFreeString that BSTR as well, when you're done with that.

    I see your code in a pastebin as well.
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

  4. #4
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,318
    Oh hey but wait a minute, this is C! So this lpVtbl really is what you call the interface functions through?!?!? So the first parameter is the "this" pointer...

    Man I've never done COM through C and would never want to! (Though I've done it for many years through C++)
    Any reason you can't use C++? It's a lot easier!

    Well heck, just make sure you call Release and SysFreeString afterwards and maybe someone else who's done this in C can help you. I feel dirty now - having had to think about C.
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Confused about Memory
    By gL_nEwB in forum C++ Programming
    Replies: 22
    Last Post: 06-20-2006, 07:32 PM
  2. Finished Stack
    By the pooper in forum C Programming
    Replies: 11
    Last Post: 02-02-2005, 10:52 AM
  3. Stacks
    By Cmuppet in forum C Programming
    Replies: 19
    Last Post: 10-13-2004, 02:32 PM
  4. better c string functions
    By samps005 in forum C Programming
    Replies: 8
    Last Post: 11-04-2003, 01:28 PM
  5. What am I doing wrong, stack?
    By TeenyTig in forum C Programming
    Replies: 2
    Last Post: 05-27-2002, 02:12 PM