Code:
void X3DCamera::Yaw(float angle)
{
D3DXMATRIX T;
if (CameraType == LANDOBJECT)
{
D3DXMatrixRotationY(&T,angle);
}
if (CameraType == AIRCRAFT)
{
D3DXMatrixRotationAxis(&T,&Up,angle);
}
D3DXVec3TransformCoord(&Right,&Right,&T);
D3DXVec3TransformCoord(&Look,&Look,&T);
}
This is DirectX but it is doing the same thing you need to do. First I create an axis rotation matrix about the vector (axis) I need to rotate around. Then I transform my other two remaning vectors (axes) by the matrix computed in the previous step.
Prior to spitting out a view matrix in the case of a camera and model matrix in the case of an object I make sure the 3 vectors are orthogonal to each other.
TransformCoord is basically:
result.x = vector.x * matrix(0,0) + vector.y * matrix(0,1) + vector.z * matrix(0,2);
result.y = vector.x * matrix(1,0) + vector.y * matrix(1,1) + vector.z * matrix(1,2);
result.z = vector.x * matrix(2,0) + vector.y * matrix(2,1) + vector.z * matrix(2,2);
For OpenGL or RH systems use the transpose of matrix().
Pseudocode for orthogonalizing object vectors and ensuring they are orthonormal.
Normalize(Look)
Up = Normalize(Cross(Right,Look))
Right = Normalize(Cross(Up,Look))