The official documention explicitly forbids you from calling code from USER32.dll in DllMain but that is where RegisterClass/UnregisterClass happen to reside. It also doesn't give you a good way to keep track of how many times the DLL has been loaded/unloaded since the variables in DllMain are not shared across processes. I thought about using shared memory to keep track of how many processes are currently using the DLL but it doesn't sound like something I should be doing. Is there no standard way to keep track of when the first process loads your DLL and also when the last process unloads it?
The special thing about calling GetModuleHandle(NULL) inside the DLL code is that it doesn't give you the HINSTANCE of the DLL but rather that of calling process. That means your DLL is injecting its window class name into the namespace of the calling process when you call RegisterClass with that HINSTANCE. Normally that doesn't lead to any issues provided that the class name isn't a common one but it's not something a DLL is supposed to do. I've seen a lot of different open souce code do this and it works well because Windows automatically cleans up window classes upon process exit that don't belong to DLLs so you don't need to worry about calling UnregisterClass. Unfortunately it's not the correct way to do things.
This is the code I'm using to get the instance of the DLL (it works regardless of the how the code is linked):
Code:
bool get_current_module(HMODULE *result) {
const DWORD flags = GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT;
if (GetModuleHandleEx(flags, (LPCWSTR)get_current_module, result))
return true;
else
return false;
}