The code that calls the MSI wouldn't know that your function didn't take the handle. MS declared a prototype to follow and, since their code has no knowledge of yours beyond that it exists, it will always call it the same way pushing the handle and expecting the custom function to pop it off and clean the stack.
If you've got a recent-ish version of Visual Studio, you can compile the following with the stack frames run time check enabled (Solution Properties-> Configuration Properties->C/C++->Code Generation->Basic Runtime Checks), and that should throw up an error. Alternatively, compile without and check the stackpointer value in the last cout with the rest.
Code:
#include <windows.h>
#include <iostream>
#include <msi.h>
// the function pointer the function should use
typedef UINT (__stdcall*CustomFunc)(MSIHANDLE);
// test functions
UINT __stdcall CustomAction(MSIHANDLE arg)
{
return 0;
}
UINT __stdcall CustomActionNoArg()
{
return 0;
}
int main()
{
void* stackptr = 0;
MSIHANDLE mHandle = NULL;
CustomFunc cust = CustomAction;
_asm mov stackptr, esp; // save the stack pointer
std::cout << "Stack pointer before good call: " << stackptr << '\n';
cust(mHandle);
_asm mov stackptr, esp;
std::cout << "Stack pointer after good call: " << stackptr << '\n';
// this is the equivalent of what happens in the msi code
// only the result will be returned from GetProcAddress
cust = (CustomFunc)CustomActionNoArg;
_asm mov stackptr, esp;
std::cout << "Stack pointer before bad call: " << stackptr << '\n';
cust(mHandle);
_asm mov stackptr, esp;
std::cout << "Stack pointer after bad call: " << stackptr << '\n';
}
I get the following
Stack pointer before good call: 0012FF70
Stack pointer after good call: 0012FF70
Stack pointer before bad call: 0012FF70
Stack pointer after bad call: 0012FF6C
I don't know which code calls CustomAction so I can't tell if it will cause any damage or negative effects in the installer but safe to say, it's not good practice.