I'm not sure exactly what your code is trying to do, so I've written a quick snippet showing how I'd do it (this would be repeated each time the scene is rendered).
Originally posted by Silvercord
MiddleX and MiddleY are the screen width and height divided by two. it is the center of the screen in screen coords. note that when using getcursorpos and setcursorpos those screen coordinates are setup like a regular cartesian system. the point 0,0 is actually in the middle of the screen. for everything else I think the positive y goes down the screen and 0, 0 is the top left corner.
SetCursorPos(MiddleX, MiddleY); //sets cursor to exact middle of screen
SCREEN_WIDTH and SCREEN_HEIGHT are #define'd compiler directives passed into the function that sets up the resolution.
static float MiddleX = SCREEN_WIDTH / 2;
static float MiddleY = SCREEN_HEIGHT / 2;
If there is anything else that I didn't explain good enough please ask me.
#define SCREEN_WIDTH 1024
#define SCREEN_HEIGHT 768
I'm assuming as you said that the mouse cursor co-ordinate system is cartesian in nature with the +y-axis running up the screen, and the +x-axis running to the right.
The value 'c' in the above code is a constant, which you will need to experiment with. It depends on many things such as screen size and your personal feeling as to how fast the rotation should be. Try something like 0.003 to start. You should then maybe form a formula for 'c', which uses the screen size, so as to account for running this application on different size screens. This should all be possible by trial and error.
float yaw = 0.0f, pitch = 0.0f;
float oldxpos = 0.0f, oldypos = 0.0f;
float xdisp, ydisp;
//all we are concerned with is the mouse's displacement since last render
xdisp = MousePos.x - oldxpos;
ydisp = MousePos.y - oldypos;
yaw += -(xdisp * c); //use -ve as rotations are anti-clockwise
pitch += -(ydisp * c);
//you may want to add a couple of simple loops
//here to make the yaw and the pitch modulus 2*pi.
//This would just prevent the values of yaw and pitch from
//getting too large (or too small) for the float type.
//store positions of mouse for next render
oldxpos = MousePos.x;
oldypos = MousePos.y;
//You then need to transform the position of all objects (rotate them) in your scene through the following:
object.x = object.x * cos(yaw) + object.y * sin(yaw) * sin(pitch) + object.z * sin(yaw) * cos(pitch);
object.y = object.y * cos(pitch) - object.z * sin(pitch);
object.z = -object.x * sin(yaw) + object.y * cos(yaw) * sin(pitch) + object.z * cos(yaw) * cos(pitch);
The last part of the code, which refers to some structure 'object' should naturally be repeated for all objects in your scene.
You must also remember that you need to implement a projection transformation to map the 3D scene on to your 2D screen, and you would also need another translation transform if you intend to move the camera.