The return value of PathFindExtension is:

"Returns the address of the "." that precedes the extension within pPath if an extension is found, or the address of the terminating null character otherwise."

How to check against the terminating null character?
It never enters if().

Code:
int main()
{
    char * str = "";

    if((str = PathFindExtension( "C:\\filetxt" )) == '\0')
        cout<< "Error" << endl;
    else
        cout<< str << endl;

    return 0;
}