You put it perfectly -- you have to
If you want to program games, your programming language is going to be your damn best friend. You will want it badly. More than your girlfriend. If you learn to love to program, if you really want to learn how to program, if you really study your programming language and try to master every nut and bolt, inspect every dark corner, then yes; You are in your way to become a damn good games programmer.
Programming is programming, whether you're a kernel junkie, an OpenGL guru, or somebody who writes software mainly for networking purposes.
Why do I know this when I'm not even a games programmer, never was or ever will be? Because games programming is no different from any other type of programming. Same rules. Don't trust me? That's cool. We'll talk in a few years.