That I agree. I wasn't referring to driver problems. The only binary driver I use is the nvidia driver, and they make 64-bit releases. I was referring to things like browser plug-in compatibility, input method (64-bit SCIM won't work in 32-bit firefox for some reason), etc. Just little but annoying problems.
Whilst I agree in principle, I would say that the drivers problems are MUCH more of a "Windows 64-bit problem" than a "64-bit problem". I think you'd have to hunt for quite some time to find a driver that doesn't work on 64-bit in Linux, as all the standard drivers are regularly compiled for Linux in both 32- and 64-bit since about 2002 when 64-bit came into the x86 world.