DirectX 9 D3DXIntersect with first person camera

I have a first person camera setup and working fine, i am shooting a ray from the camera pos and the direction the camera is facing, when the app first starts the intersect works fine , but when i move around the object to the otherside the is no intersect anymore, why can i only intersect the pologons head on and not all around?
heres a pic to show what i mean , the space the ray is in is object space inverse of (world*view)

Imageshack - fpsintersect.jpg

thanks if you can help

Are the bounding volumes of the objects you are testing also in local space? If not then your calculations will not work. You can test the collisions in any space you want as long as both objects being tested are in the same space.

I suspect your calculations work when the object is at the origin and fail in all other positions.