All GUI's are in 2D and in screen space. They are not or should not be in camera or view space. Don't run the vertices that make up the GUI objects through perspective transform since they are not 3D objects. They are simply quads that are textured to look like a cool GUI.
When the mouse clicks on the screen the WM_MOUSEMOVE or WM_LBUTTONDOWN, etc, will still give you screen space coordinates since the mouse cursor is a global resource shared by all in Windows. But I would recommend firing up DirectInput and linking with it to make it easier to deal with input. Use OpenGL for graphics and use DirectInput for control. You only need to pass the handle of the window needing to use DirectInput. It does not require any IDirect3DDevice9 interface pointer and it should not conflict with OpenGL, and it will be faster than responding to Windows messages.
You can create your own floating point rectangle structure or class similar to the one in Win32 called RECT.
Code:
class fRect
{
float left,top,right,bottom;
public:
fRect(void):left(0.0f),top(0.0f),right(0.0f),bottom(0.0f) {}
fRect(const fRect &rect)
{
left=rect.left;
top=rect.top;
right=rect.right;
bottom=rect.bottom;
}
fRect(float l,float t,float r,float b):left(l),top(t),right(r),bottom(b) {}
float Width(void) {return right-left;};
float Height(void) {return bottom-top;};
void Normalize(void)
{
float temp=0.0f;
//Check width
if (left>right)
{
temp=right;
right=left;
left=right;
}
if (top>bottom)
{
temp=top;
top=bottom;
bottom=top;
}
}
bool PointInRect(float x,float y)
{
if ((x>left && x<right) && (y>top && y<bottom))
{
return true;
} else return false;
}
};
Then in your Window class you might have this:
Code:
class CWindow
{
fRect PlacementInfo;
...
...
};
class CButton::public CWindow
{
};
Button inherits from Window which means now the button has an fRect describing it's placement information.
So button hit-testing is now as simple as calling:
Code:
Object->PlacementInfo.PointInRect(float mousex,float mousey)
Where Object is either a CWindow or a class derived from CWindow.
If it returns true, the user clicked inside this button, else the user clicked outside of this button.