I kind of have this problem too. At the end of last year I bought this huge book on openGL, and a huge NVIDIA card* thinking I was gonna get into 3D programming, since it seems very sexy, etc. Then it was a series of challanges, like dealing with lighting, textures, quadratics, quarterions, etc. 3D programming can be pretty consuming, meaning it may leave you with little time for anything else.
Which is why I've pretty much stopped. Occasionally it was interesting, but now I think I had myself obsessed ("Must make this work...properly"), because 3D programming is IMO pretty tedious and boring if you are not also very interested in math.
I went thru a similar thing with device drivers. The tedious part started when I realized that a lot of information is difficult or impossible to get from manufacturers, meaning I was getting myself into some time consuming reverse engineering and testing stuff that probably has very limited practical application. Like, if someone paid me to write a driver, I could deal with the headache, but in that case I would most likely be able to get the specs anyway.