setwindowshookex returns null

Hi All,

I want to use setwindowshookex for hooking a dll into one of my own applications. setwindowshookex returns null setting error code 87 (invalid parameter), which I believe is the threadid parameter. This is a cut down version of my code to show what im trying to do.

Code:

STARTUPINFO startInfo;
PROCESS_INFORMATION procInfo;

memset(&startInfo, 0, sizeof(startInfo));
memset(&procInfo, 0, sizeof(procInfo));

startInfo.cb = sizeof(startInfo);
				
CreateProcess( "c:/myapp.exe", NULL, NULL, NULL, false, DETACHED_PROCESS, 0, 0, 
&startInfo, &procInfo ) ;

HINSTANCE dll = LoadLibrary(TEXT("c:/mydll.dll")) ;
if (dll == NULL) return false ;

DllHookProc hookProc = (DllHookProc)GetProcAddress(dll,"HookProc");

if (hookProc == NULL) return false ; // will also free dll.

HHOOK hook = SetWindowsHookEx( WH_CALLWNDPROC,(HOOKPROC)hookProc,
		dll, procInfo.dwThreadId );

the hook variable is set to null at this point, however LoadLibrary and GetProcAddress all work correctly. Also CreateProcess works ok as I can see the application load and the procInfo.dwThreadId is none 0

Any ideas what i am doing incorrectly?

Thanks for any help

Originally Posted by cloudy

Code:

DllHookProc hookProc = (DllHookProc)GetProcAddress(dll,"HookProc");

What is a 'DllHookProc'?

I can't find 'DllHookProc' on MSDN or google (or am I just not looking in the right spots?).

Can you cast it to a HOOKPROC successfully?

Hi,

Thanks for the reply.

That is a typedef of the hook function signature that is being exported by my dll via a .def file. I.E.

Code:

typedef LRESULT (CALLBACK *DllHookProc)(int nCode, WPARAM wParam, LPARAM lParam);

The GetProcAddress function returns happily with the address of the HookProc function.

Thanks again

Is myapp.exe an executable that immediately returns? If so, then the threadID would no longer be valid.

Thanks for the reply,

it doesn't immediately return I can still see running.

Does it work if you set the threadID to 0?
Are you attempting to inject a 32 bit DLL into a 64 bit process (or vice-versa)?

Does it work if you set the threadID to 0?
Are you attempting to inject a 32 bit DLL into a 64 bit process (or vice-versa)?

This may be a dumb question but is the SetWindowsHookEX call in the DLL or the EXE?

Most of what that function does only works when you call it from a DLL... any of the parameters in it's description marked "Global" will only work in DLLs.

Originally Posted by CommonTater

This may be a dumb question but is the SetWindowsHookEX call in the DLL or the EXE?

Most of what that function does only works when you call it from a DLL... any of the parameters in it's description marked "Global" will only work in DLLs.

I don't think the SetWindowHookEx call needs to be in the DLL. Just the hook procedure needs to be in the DLL.

Originally Posted by bithub

I don't think the SetWindowHookEx call needs to be in the DLL. Just the hook procedure needs to be in the DLL.

Yeah, it does... the call to SetWindowsHookEx() and the DLLHookProc() have to be in the same scope otherwise the function pointer is invalid.

Here's an example that returns the window handle in a message to the main window proc in an EXE and exports a call that lets you retrieve the program's name....

The DLL...

Code:

/// self
#include "Helper.h"

/////////////////////////////////////////////////////////////////
// Shared data segment

#pragma data_seg("Hook")
HINSTANCE   hdll    = NULL;               // dll handle
HWND        HWind   = NULL;               // handle of Server window
HHOOK       Hook  = NULL;                 // message hook handle
LAUNCHINFO  LInfo   = {0};                // info about program launch
#pragma data_seg()
#pragma comment(linker,"/SECTION:Hook,S")

/////////////////////////////////////////////////////////////////////////////////////////
// Export
//

// return launch info to server
MMAPI VOID HelperGetInfo(pLAUNCHINFO Info)
  { memcpy(Info,&LInfo,sizeof(LInfo)); }



/////////////////////////////////////////////////////////////////////////////////////////
// gather information
//

// get information about launched program
VOID HookGetInfo(HWND wParm)
  { TCHAR   cmdl[MAX_PATH] = {0};   // command line
    TCHAR   pgm[MAX_PATH] = {0};    // program name
    PTCHAR  args;                   // arguments
    // purge old data
    memset(&LInfo,0,sizeof(LAUNCHINFO));
    // set program window handle
    LInfo.Handle = wParm;    
    // get program info
    GetModuleFileName(NULL,pgm,MAX_PATH);
    _wsplitpath(pgm,NULL,NULL,LInfo.Name,LInfo.Ext);
    // get launched file info
    wcsncpy(cmdl,GetCommandLine(),MAX_PATH);
    args = PathGetArgs(cmdl);
    if(wcslen(args) < 1)
      return;
    args = wcsrchr(args,L'.');
    if (args)
      { wcscpy(LInfo.Type,args);
        // fix trailing quote
        args = wcsrchr(LInfo.Type,L'\"');
        if (args)
          *args = 0; } }



/////////////////////////////////////////////////////////////////////////////////////////
// Handle Shell Hook 
//

// hook tosser
MMAPI LRESULT CALLBACK AppCatcher(INT Code, WPARAM wParm, LPARAM lParm)
  { switch (Code)
      { case HSHELL_WINDOWCREATED   :         // new window
          // gather launch info
          HookGetInfo((HWND) wParm);
          // send message to Server, wparam = Handle               
          PostMessage(HWind,UM_HELPERRUN,wParm,0); 
          break;
        case HSHELL_WINDOWDESTROYED :         // window closing
          // send messate to Server
          PostMessage(HWind,UM_HELPEREXIT,wParm,0);
          break; }
    return CallNextHookEx(RMHook,Code,wParm,lParm); } 



/////////////////////////////////////////////////////////////////
// Set the shell hook
// Returns hook status 1 = active, 0 = not
//

// apply the hook to launched programs
MMAPI BOOL WINAPI SetHook(HWND hWind)
  { HWind = hWind;   // handle of window to receive message
    Hook = SetWindowsHookEx(WH_SHELL,&AppCatcher,hdll,0);
    return (Hook != NULL); }      



// remove the hook from launched programs
MMAPI BOOL WINAPI FreeHook(void)
  { return UnhookWindowsHookEx(Hook); }



/////////////////////////////////////////////////////////////////
//  DLL entry point
//
BOOL APIENTRY DllMain(HINSTANCE hinst, DWORD reason, LPVOID reserved)
  { if (reason == DLL_PROCESS_ATTACH)  // save instance handle
          hdll = hinst;
    return 1; }

And the header file....

Code:

#ifndef HELPER_H
#define HELPER_H

// for the compiler
#define UNICODE
#define _UNICODE
#define WIN32_DEFAULT_LIBS
#define WIN32_LEAN_AND_MEAN
#define _WIN32_WINNT 0x0502
#define _X86_

// project global headers
#include <windows.h>
#include <shlwapi.h>
#include <wchar.h>

#include <globaldefs.h>

#pragma lib "Helper.lib"

#define MMAPI __declspec(dllexport)

// messages sent by helper dll
#define UM_HELPERRUN  WM_APP + 444
#define UM_HELPEREXIT WM_APP + 445 

// launch information
#pragma pack(1)
typedef struct tLAUNCHINFO
  { TCHAR Name[MAX_PROGRAMNAME]; // short program name
    TCHAR Ext[MAX_TYPENAME];     // program type
    TCHAR Type[MAX_TYPENAME];    // file type
    HWND  Handle; }              // program window handle  
  LAUNCHINFO, *pLAUNCHINFO;
#pragma pack()


// retrieve the command line
MMAPI VOID HelperGetInfo(pLAUNCHINFO Info);

// set the hook
MMAPI BOOL WINAPI SetHook(HWND hWind);

// release the hook
MMAPI BOOL WINAPI FreeHook(void);


#endif // HELPER_H

This is extracted from working code, some things were changed to not advertise which application ... so any errors are not my fault.

Originally Posted by CommonTater

Yeah, it does... the call to SetWindowsHookEx() and the DLLHookProc() have to be in the same scope otherwise the function pointer is invalid.

No, it doesn't. If it worked like that you'd never be able to call dll functions in the first place.

Originally Posted by adeyblue

No, it doesn't. If it worked like that you'd never be able to call dll functions in the first place.

Don't be silly... we call DLL functions by exporting them... Look at the header file there are three exported functions listed there.

The function needs the address of the hook proc...

Code:

HHOOK WINAPI SetWindowsHookEx(
  __in  int idHook,
  __in  HOOKPROC lpfn,                              <---- it won't work with out this!
  __in  HINSTANCE hMod,                            
  __in  DWORD dwThreadId
);

How is it going to get that from a completely different module?

Thanks everyone for their help.

It is a 32bit dll into a 32 bit process.

I am calling setwindowshookex in my exe not dll.

If i change the threadId to 0 (i assume you mean do a global system hook), then setwindowshookex does not return null but only injects the hook into some processes not all of them.

To test this I changed the threadid in setwindowshookex to my running visual studio exe and everything worked fine (meaning setwindowshookex did not return null) . But as soon as I changed it back to myapp.exe threadid setwindowshookex returns null again. I also tried changing my threadid to google chrome and setwindowshookex still returns null. It seems to work with some processes like visual studio but not others like chrome.

I assume the reason why it wont inject into myapp is the same as why it wont inject into chrome. Whatever that may be.

Thanks

Originally Posted by cloudy

Thanks everyone for their help.

It is a 32bit dll into a 32 bit process.

I am calling setwindowshookex in my exe not dll.

And that's why it's not working... It can't find the *address* of the hookproc in the dll from the exe file.

Hooks Overview (Windows)

Thanks for the reply.

Why does it work injecting into visual studio then?

I just tried the following from within the Dll:

Code:

LRESULT CALLBACK HookProc (int nCode, WPARAM wParam, LPARAM lParam )
{
	return CallNextHookEx( 0, nCode, wParam, lParam);
}

void Test( unsigned long threadId )
{
	HHOOK hook = SetWindowsHookEx(WH_CALLWNDPROC,(HOOKPROC)HookProc,
		NULL, threadId );

	std::ofstream os ("c:/SetWindowsHookEx.txt") ;
	if (hook == NULL)
	os << "IsNull" ;
	else
	os << "IsNotNull" ;
	os.close() ;

}

Test is a dll exported function. ThreadId is the thread Id of myapp.exe (also tried again with chrome.exe and got the same result). c:/SetWindowsHookEx.txt contains the line "IsNull".

Thanks