That was the next thing I was going to say. Are you using the normals of the planes?
Hmm. If the normals all point outwards, I'm not sure why the algo is failing when you look down. Quite odd.
So you have surrounded your characters with planes that approximate their shape? Very good idea. In fact, I might steal it.
I guess it would not make sense to try to detect every collision with every poly nor would it be efficient.
I'm really lost as to why the camera or view vector would cause collisions to not register. Are you also rotating the planes that you are using for collision detection? Perhaps you are rotating them too far which would cause the normal test to fail. Give me some more info on exactly how you are testing for hits on the walls - I think I understand but I want to make sure.