This isn't the most elegant implementation, but it does work. What I do to alleviate the problem presented in the picture above is to only test for a point, rather than a sphere. I do this by moving the test point from the center of the circle inward radius units towards the center. Yes, I have to do a vector normalize to accomplish this, but in my testing using a sqrt approximation only gives results that are 5% different (and subsequently my tolerance values for the fast sqrt implementation of this are about 6% bigger than the floating point tolerances for the regular sqrt implementation).
Code:
BOOL PointInBSPFace(BSP *pBSP,int faceindex,Vector3 & Pos,float Radius)
{
tBSPFace & face_ref = pBSP->mpFaces[faceindex];
Vector3 dir = face_ref.center - Pos;
float startd_sq = dir.BasicLength();
float rad_sq = Radius * Radius;
if(startd_sq <= rad_sq)
return TRUE; //intersects with the center
dir.Normalize(Faster_Sqrtf(startd_sq));
Vector3 testpoint = Pos + (dir * Radius);
Plane3 *pPlane = NULL;
float d(0);
for(int plane = 0; plane < face_ref.numInPlanes; plane++)
{
pPlane = &face_ref.mpInplanes[plane];
d = DotProduct(&pPlane->mNormal,&testpoint) - pPlane->mDist;
if(d < -FastBSPFacetolerance)
return FALSE;
}
return TRUE;
}