Steam overlay - how?
For those of us that use Steam we know you can hit shift-tab and it will bring up its overaly and you can see which friends are on, chat to them, and perform other operations as well.
My question is how do you think Steam is actually rendering its overlay inside of D3D and OpenGL games? I know how to render GDI to a DC and since an IDirect3DTexture<x> is at it core a DC and all OpenGL textures are also Windows DCs it is pretty simple to create and offscreen DC and render windows GDI to it. However it is not simple to get Windows message handling to windows you create within full-screen D3D and OpenGL games because those are by default top-layer windows. Top layer windows take priority over any window you create. In other words you could create a window, blit it to the DC which just happens to be your texture and draw that as a quad in D3D or OGL. What I am baffled on is how does Steam actually receive Windows messages when the mouse is over their overlay windows? Seems to me the messages would go to the game window and not the Steam windows/controls.
Any ideas? I'm quite intrigued by all this b/c if I can get it working it could mean the end of my struggles to create an in-game Direct3D GUI system. If I can use Windows API like I can with normal apps then I can simply create custom paint functions to change the look and yet use the system just as I would in a normal Windows application.
Do you want me to dig into it with a decompiler for you?
I don't know how Steam does it, but in general this can be accomplished via SetWindowLong() passing GWL_WNDPROC as nIndex. Now Steam can have access to all the Window messages. This assumes that the game runs in the same process as Steam. Since the game is launched via Steam, I'm assuming this is possible.
There's also an interesting article about Fraps and how it works that should give you some pointers, I think: Ring3 Circus » Case study: Fraps
Meanwhile I think Steam (and pretty much every other overlay out there) are DirectDraw overlays. There's this library that seems to have been abandoned. But may have something interesting in its source code.