Thread: setwindowshookex returns null

  1. #1
    Registered User
    Join Date
    Oct 2004
    Posts
    100

    setwindowshookex returns null

    Hi All,

    I want to use setwindowshookex for hooking a dll into one of my own applications. setwindowshookex returns null setting error code 87 (invalid parameter), which I believe is the threadid parameter. This is a cut down version of my code to show what im trying to do.
    Code:
    STARTUPINFO startInfo;
    PROCESS_INFORMATION procInfo;
    
    memset(&startInfo, 0, sizeof(startInfo));
    memset(&procInfo, 0, sizeof(procInfo));
    
    startInfo.cb = sizeof(startInfo);
    				
    CreateProcess( "c:/myapp.exe", NULL, NULL, NULL, false, DETACHED_PROCESS, 0, 0, 
    &startInfo, &procInfo ) ;
    
    HINSTANCE dll = LoadLibrary(TEXT("c:/mydll.dll")) ;
    if (dll == NULL) return false ;
    
    DllHookProc hookProc = (DllHookProc)GetProcAddress(dll,"HookProc");
    
    if (hookProc == NULL) return false ; // will also free dll.
    
    HHOOK hook = SetWindowsHookEx( WH_CALLWNDPROC,(HOOKPROC)hookProc,
    		dll, procInfo.dwThreadId );
    the hook variable is set to null at this point, however LoadLibrary and GetProcAddress all work correctly. Also CreateProcess works ok as I can see the application load and the procInfo.dwThreadId is none 0

    Any ideas what i am doing incorrectly?

    Thanks for any help

  2. #2
    train spotter
    Join Date
    Aug 2001
    Location
    near a computer
    Posts
    3,868
    Quote Originally Posted by cloudy View Post
    Code:
    DllHookProc hookProc = (DllHookProc)GetProcAddress(dll,"HookProc");
    What is a 'DllHookProc'?

    I can't find 'DllHookProc' on MSDN or google (or am I just not looking in the right spots?).

    Can you cast it to a HOOKPROC successfully?
    "Man alone suffers so excruciatingly in the world that he was compelled to invent laughter."
    Friedrich Nietzsche

    "I spent a lot of my money on booze, birds and fast cars......the rest I squandered."
    George Best

    "If you are going through hell....keep going."
    Winston Churchill

  3. #3
    Registered User
    Join Date
    Oct 2004
    Posts
    100
    Hi,

    Thanks for the reply.

    That is a typedef of the hook function signature that is being exported by my dll via a .def file. I.E.

    Code:
    typedef LRESULT (CALLBACK *DllHookProc)(int nCode, WPARAM wParam, LPARAM lParam);
    The GetProcAddress function returns happily with the address of the HookProc function.

    Thanks again

  4. #4
    Registered User
    Join Date
    Sep 2004
    Location
    California
    Posts
    3,268
    Is myapp.exe an executable that immediately returns? If so, then the threadID would no longer be valid.
    bit∙hub [bit-huhb] n. A source and destination for information.

  5. #5
    Registered User
    Join Date
    Oct 2004
    Posts
    100
    Thanks for the reply,

    it doesn't immediately return I can still see running.

  6. #6
    Registered User
    Join Date
    Sep 2004
    Location
    California
    Posts
    3,268
    Does it work if you set the threadID to 0?
    Are you attempting to inject a 32 bit DLL into a 64 bit process (or vice-versa)?
    bit∙hub [bit-huhb] n. A source and destination for information.

  7. #7
    Registered User
    Join Date
    Sep 2004
    Location
    California
    Posts
    3,268
    Does it work if you set the threadID to 0?
    Are you attempting to inject a 32 bit DLL into a 64 bit process (or vice-versa)?
    bit∙hub [bit-huhb] n. A source and destination for information.

  8. #8
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    This may be a dumb question but is the SetWindowsHookEX call in the DLL or the EXE?

    Most of what that function does only works when you call it from a DLL... any of the parameters in it's description marked "Global" will only work in DLLs.

  9. #9
    Registered User
    Join Date
    Sep 2004
    Location
    California
    Posts
    3,268
    Quote Originally Posted by CommonTater View Post
    This may be a dumb question but is the SetWindowsHookEX call in the DLL or the EXE?

    Most of what that function does only works when you call it from a DLL... any of the parameters in it's description marked "Global" will only work in DLLs.
    I don't think the SetWindowHookEx call needs to be in the DLL. Just the hook procedure needs to be in the DLL.
    bit∙hub [bit-huhb] n. A source and destination for information.

  10. #10
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by bithub View Post
    I don't think the SetWindowHookEx call needs to be in the DLL. Just the hook procedure needs to be in the DLL.
    Yeah, it does... the call to SetWindowsHookEx() and the DLLHookProc() have to be in the same scope otherwise the function pointer is invalid.

    Here's an example that returns the window handle in a message to the main window proc in an EXE and exports a call that lets you retrieve the program's name....

    The DLL...
    Code:
    /// self
    #include "Helper.h"
    
    /////////////////////////////////////////////////////////////////
    // Shared data segment
    
    #pragma data_seg("Hook")
    HINSTANCE   hdll    = NULL;               // dll handle
    HWND        HWind   = NULL;               // handle of Server window
    HHOOK       Hook  = NULL;                 // message hook handle
    LAUNCHINFO  LInfo   = {0};                // info about program launch
    #pragma data_seg()
    #pragma comment(linker,"/SECTION:Hook,S")
    
    /////////////////////////////////////////////////////////////////////////////////////////
    // Export
    //
    
    // return launch info to server
    MMAPI VOID HelperGetInfo(pLAUNCHINFO Info)
      { memcpy(Info,&LInfo,sizeof(LInfo)); }
    
    
    
    /////////////////////////////////////////////////////////////////////////////////////////
    // gather information
    //
    
    // get information about launched program
    VOID HookGetInfo(HWND wParm)
      { TCHAR   cmdl[MAX_PATH] = {0};   // command line
        TCHAR   pgm[MAX_PATH] = {0};    // program name
        PTCHAR  args;                   // arguments
        // purge old data
        memset(&LInfo,0,sizeof(LAUNCHINFO));
        // set program window handle
        LInfo.Handle = wParm;    
        // get program info
        GetModuleFileName(NULL,pgm,MAX_PATH);
        _wsplitpath(pgm,NULL,NULL,LInfo.Name,LInfo.Ext);
        // get launched file info
        wcsncpy(cmdl,GetCommandLine(),MAX_PATH);
        args = PathGetArgs(cmdl);
        if(wcslen(args) < 1)
          return;
        args = wcsrchr(args,L'.');
        if (args)
          { wcscpy(LInfo.Type,args);
            // fix trailing quote
            args = wcsrchr(LInfo.Type,L'\"');
            if (args)
              *args = 0; } }
    
    
    
    /////////////////////////////////////////////////////////////////////////////////////////
    // Handle Shell Hook 
    //
    
    // hook tosser
    MMAPI LRESULT CALLBACK AppCatcher(INT Code, WPARAM wParm, LPARAM lParm)
      { switch (Code)
          { case HSHELL_WINDOWCREATED   :         // new window
              // gather launch info
              HookGetInfo((HWND) wParm);
              // send message to Server, wparam = Handle               
              PostMessage(HWind,UM_HELPERRUN,wParm,0); 
              break;
            case HSHELL_WINDOWDESTROYED :         // window closing
              // send messate to Server
              PostMessage(HWind,UM_HELPEREXIT,wParm,0);
              break; }
        return CallNextHookEx(RMHook,Code,wParm,lParm); } 
    
    
    
    /////////////////////////////////////////////////////////////////
    // Set the shell hook
    // Returns hook status 1 = active, 0 = not
    //
    
    // apply the hook to launched programs
    MMAPI BOOL WINAPI SetHook(HWND hWind)
      { HWind = hWind;   // handle of window to receive message
        Hook = SetWindowsHookEx(WH_SHELL,&AppCatcher,hdll,0);
        return (Hook != NULL); }      
    
    
    
    // remove the hook from launched programs
    MMAPI BOOL WINAPI FreeHook(void)
      { return UnhookWindowsHookEx(Hook); }
    
    
    
    /////////////////////////////////////////////////////////////////
    //  DLL entry point
    //
    BOOL APIENTRY DllMain(HINSTANCE hinst, DWORD reason, LPVOID reserved)
      { if (reason == DLL_PROCESS_ATTACH)  // save instance handle
              hdll = hinst;
        return 1; }
    And the header file....
    Code:
    #ifndef HELPER_H
    #define HELPER_H
    
    // for the compiler
    #define UNICODE
    #define _UNICODE
    #define WIN32_DEFAULT_LIBS
    #define WIN32_LEAN_AND_MEAN
    #define _WIN32_WINNT 0x0502
    #define _X86_
    
    // project global headers
    #include <windows.h>
    #include <shlwapi.h>
    #include <wchar.h>
    
    #include <globaldefs.h>
    
    #pragma lib "Helper.lib"
    
    #define MMAPI __declspec(dllexport)
    
    // messages sent by helper dll
    #define UM_HELPERRUN  WM_APP + 444
    #define UM_HELPEREXIT WM_APP + 445 
    
    // launch information
    #pragma pack(1)
    typedef struct tLAUNCHINFO
      { TCHAR Name[MAX_PROGRAMNAME]; // short program name
        TCHAR Ext[MAX_TYPENAME];     // program type
        TCHAR Type[MAX_TYPENAME];    // file type
        HWND  Handle; }              // program window handle  
      LAUNCHINFO, *pLAUNCHINFO;
    #pragma pack()
    
    
    // retrieve the command line
    MMAPI VOID HelperGetInfo(pLAUNCHINFO Info);
    
    // set the hook
    MMAPI BOOL WINAPI SetHook(HWND hWind);
    
    // release the hook
    MMAPI BOOL WINAPI FreeHook(void);
    
    
    #endif // HELPER_H
    This is extracted from working code, some things were changed to not advertise which application ... so any errors are not my fault.

  11. #11
    'Allo, 'Allo, Allo
    Join Date
    Apr 2008
    Posts
    639
    Quote Originally Posted by CommonTater View Post
    Yeah, it does... the call to SetWindowsHookEx() and the DLLHookProc() have to be in the same scope otherwise the function pointer is invalid.
    No, it doesn't. If it worked like that you'd never be able to call dll functions in the first place.

  12. #12
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by adeyblue View Post
    No, it doesn't. If it worked like that you'd never be able to call dll functions in the first place.
    Don't be silly... we call DLL functions by exporting them... Look at the header file there are three exported functions listed there.

    The function needs the address of the hook proc...
    Code:
    HHOOK WINAPI SetWindowsHookEx(
      __in  int idHook,
      __in  HOOKPROC lpfn,                              <---- it won't work with out this!
      __in  HINSTANCE hMod,                            
      __in  DWORD dwThreadId
    );
    How is it going to get that from a completely different module?

  13. #13
    Registered User
    Join Date
    Oct 2004
    Posts
    100
    Thanks everyone for their help.

    It is a 32bit dll into a 32 bit process.

    I am calling setwindowshookex in my exe not dll.

    If i change the threadId to 0 (i assume you mean do a global system hook), then setwindowshookex does not return null but only injects the hook into some processes not all of them.

    To test this I changed the threadid in setwindowshookex to my running visual studio exe and everything worked fine (meaning setwindowshookex did not return null) . But as soon as I changed it back to myapp.exe threadid setwindowshookex returns null again. I also tried changing my threadid to google chrome and setwindowshookex still returns null. It seems to work with some processes like visual studio but not others like chrome.

    I assume the reason why it wont inject into myapp is the same as why it wont inject into chrome. Whatever that may be.

    Thanks

  14. #14
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by cloudy View Post
    Thanks everyone for their help.

    It is a 32bit dll into a 32 bit process.

    I am calling setwindowshookex in my exe not dll.
    And that's why it's not working... It can't find the *address* of the hookproc in the dll from the exe file.

    Hooks Overview (Windows)

  15. #15
    Registered User
    Join Date
    Oct 2004
    Posts
    100
    Thanks for the reply.

    Why does it work injecting into visual studio then?

    I just tried the following from within the Dll:

    Code:
    LRESULT CALLBACK HookProc (int nCode, WPARAM wParam, LPARAM lParam )
    {
    	return CallNextHookEx( 0, nCode, wParam, lParam);
    }
    
    void Test( unsigned long threadId )
    {
    	HHOOK hook = SetWindowsHookEx(WH_CALLWNDPROC,(HOOKPROC)HookProc,
    		NULL, threadId );
    
    	std::ofstream os ("c:/SetWindowsHookEx.txt") ;
    	if (hook == NULL)
    	os << "IsNull" ;
    	else
    	os << "IsNotNull" ;
    	os.close() ;
    
    }
    Test is a dll exported function. ThreadId is the thread Id of myapp.exe (also tried again with chrome.exe and got the same result). c:/SetWindowsHookEx.txt contains the line "IsNull".

    Thanks

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. crashing program, help
    By xniinja in forum Windows Programming
    Replies: 1
    Last Post: 07-07-2010, 03:57 PM
  2. Linked List Not Saving Value as Int
    By bar338 in forum C Programming
    Replies: 4
    Last Post: 05-04-2009, 07:53 PM
  3. Button handler
    By Nephiroth in forum Windows Programming
    Replies: 8
    Last Post: 03-12-2006, 06:23 AM
  4. Help with yacc/compiler design/seg fault
    By trippeer in forum C Programming
    Replies: 1
    Last Post: 04-08-2005, 03:43 AM
  5. button 'message'
    By psychopath in forum Windows Programming
    Replies: 12
    Last Post: 04-18-2004, 09:57 AM