I used Silvers rotatevector for my upvector. That worked well. But I also tested what you describe and yes, it does not work. After half a turn things get really really messy. I assume thats what you mean with gimpal lock (not sure though, I didn't yet have the chance to learn a bit more about that problem).
Originally posted by blackrat364
How many times do you call this? Once, or once for each axis (x,y,z)? If you call it once, you more or less already are using quaternions, but I don't think you would be able to pull it off without making the total conversion over to quaternions. So, I'm thinking that you call it three times, with the second getting the modified vector from the first call, and the third call getting the modified vector from the second. If you're indeed calling it three times, it can indeed cause gimbal lock. Assuming you are doing it X,Y, then Z, rotate 90 degrees (.5pi radians) on Y, then try rotation on X and Z. It might gimbal lock. Test it out.