I dont' really know what your problem is, in order for gluLookAt to work, you give it:
a point to look at
an up orientation
your math library looks really good, and I'm surprised you can get all of that other math and not get gluLookAt working, but you'll get it eventually.
grady's advice is good, but I found that when you add the projection of the vector onto the axis back into the reflected vector before returning it you *almost* preserve length, so you can very well have two different versions of the mirror function: one that approximates the mirrored velocity for routines that don't need 100% accuracy (which may be more than you think) and the one that gives you 100% accuracy but is forced to call sqrt whenever you use it.
I also noticed some redundancy with your code, and subsequently in mine:
you flat out call A * Dot(A,P) multiple times, you save some speed by not recomputing it, but it's not really a huge deal (however if you profile your code, you'll end up finding dotproduct is the single most used function, which is why so many people try to optimzie it using SSE and 3D now and stuff)
DllExport Vector3d RotateAroundAAxis(Vector3d A, Vector3d P, float degrees)
degrees = (3.14159 * degrees) / 180; //convert to radians
Vector3d vRotated = Vector3d(0,0,0);
float mag = Magnitude(P);
vRotated = (P - (A * Dot(A,P)))*costheta + (Cross(A,P) * sintheta) + (A * Dot(A,P));