if i get what your saying correctly, it's because at 90 degrees you should be rotating on the z axis to go up and down... the way you are attempting to do this is fairly complicated,
you could try
and/or look into vector math and trigonomotry...
eyex, eyey, eyez
The position of the eye point.
centerx, centery, centerz
The position of the reference point.
upx, upy, upz
The direction of the up vector.