A bit more info for you.
After the CreateProcess call but before SetWindowsHookEx I now do GetGUIThreadInfo and pass the threadId as the first parameter and a pointer to a memset GUITHREADINFO structure (with the size set correctly) as the second paramerter. This returns false with the same error 87 "Invalid parameter". So it must be the threadId which is incorrect causing both GetGUIThreadInfo and SetWindowsHookEx to fail. The threadId I get from CreateProcess (see my first example). The process is still running when I call both GetGUIThreadInfo and SetWindowsHookEx. I also sleep 10 seconds after I do CreateProcess to give the process time to initialize before calling GetGUIThreadInfo (this is only temporary).
I see the same behaviour for both myapp and google chrome.
I believe this to be the cause of the problem