Thread: MSI custom action dll function question

I am currently developing a MSI package and after doing some research on custom action dll, I found the following link.

http://msdn.microsoft.com/en-us/libr...38(VS.85).aspx

It says :

Note that any called functions, including custom actions in DLLs, must specify the __stdcall calling convention. For example, to call CustomAction use the following.

Code:

#include <windows.h>
#include <msi.h>
#include <Msiquery.h>

UINT __stdcall CustomAction(MSIHANDLE hInstall)

So I understand that I need UINT as my function return type and __stdcall as a calling convention. For those of you who have worked with MSI before, do you know if putting the argument "MSIHANDLE hInstall" is required as well?

I've tried running my MSI with the argument removed from the function, and it ran well, but I was afraid if might unexpectedly crash one day because of that.

If you're not using the handle, i.e. to get properties or some other data out of the installation process itself, then I imagine it's not a big issue.

Remember that with stdcall the callee cleans the stack. If you change the number of arguments that the function takes you'll end up unbalancing it for the calling function.

The code that calls the MSI wouldn't know that your function didn't take the handle. MS declared a prototype to follow and, since their code has no knowledge of yours beyond that it exists, it will always call it the same way pushing the handle and expecting the custom function to pop it off and clean the stack.

If you've got a recent-ish version of Visual Studio, you can compile the following with the stack frames run time check enabled (Solution Properties-> Configuration Properties->C/C++->Code Generation->Basic Runtime Checks), and that should throw up an error. Alternatively, compile without and check the stackpointer value in the last cout with the rest.

Code:

#include <windows.h>
#include <iostream>
#include <msi.h>

// the function pointer the function should use
typedef UINT (__stdcall*CustomFunc)(MSIHANDLE);

// test functions
UINT __stdcall CustomAction(MSIHANDLE arg)
{
    return 0;
}

UINT __stdcall CustomActionNoArg()
{
    return 0;
}

int main()
{
    void* stackptr = 0;

    MSIHANDLE mHandle = NULL;
    CustomFunc cust = CustomAction;
    _asm mov stackptr, esp; // save the stack pointer

    std::cout << "Stack pointer before good call: " << stackptr << '\n';
    cust(mHandle);
    _asm mov stackptr, esp;

    std::cout << "Stack pointer after good call: " << stackptr << '\n';

    // this is the equivalent of what happens in the msi code
    // only the result will be returned from GetProcAddress
    cust = (CustomFunc)CustomActionNoArg;

    _asm mov stackptr, esp;

    std::cout << "Stack pointer before bad call: " << stackptr << '\n';
    cust(mHandle);
    _asm mov stackptr, esp;

    std::cout << "Stack pointer after bad call: " << stackptr << '\n';
}

I get the following

Stack pointer before good call: 0012FF70
Stack pointer after good call: 0012FF70
Stack pointer before bad call: 0012FF70
Stack pointer after bad call: 0012FF6C

I don't know which code calls CustomAction so I can't tell if it will cause any damage or negative effects in the installer but safe to say, it's not good practice.

Originally Posted by donglee

I am currently developing a MSI package and after doing some research on custom action dll, I found the following link.

http://msdn.microsoft.com/en-us/libr...38(VS.85).aspx

It says :



So I understand that I need UINT as my function return type and __stdcall as a calling convention. For those of you who have worked with MSI before, do you know if putting the argument "MSIHANDLE hInstall" is required as well?

I've tried running my MSI with the argument removed from the function, and it ran well, but I was afraid if might unexpectedly crash one day because of that.

Post your MSI concerns on the MS MSI (installer) discussion newsgroup. I'm sure Alex31 will agree.