Generally, game programmers are consumers of math, not mathematicians.
The larger point, though, is that there are many layers in a team in larger firms. The question becomes where within that team do you want to be, or do you want to become one of the 'new' style independent developers - one person doing the whole thing.
Posit for a moment a large firm, highly competitive and at the forefront of new technologies. Artists gain greater flexibility to create detailed environments, and generate massive volumes of graphical data, some of it descriptive of motion (like the motion of hair in the wind), some of it descriptive of special effects processing (complex materials), etc.
New physics modeling may be required, and one or more high priests of math may be employed to design that math, then hand the specifications over to the programming team to implement it.
Similarly, there could be many such design-programmer handoffs, where the programmers may exist in tiers of various levels of competence and training, all paid accordingly. Naturally, the lesser trained are more expendable, less appreciated, but if they survive that stage and gain experience, they may either ascend to higher tiers or into management.
If that image sounds a bit bleak, take it as a warning. Like any other treadmill, it can burn people out. The better prepared you are to excel in that environment, the better your experience will be, but make not mistake of the simple truth - it's about money to the company that employs even the best of them, and unless you are a part of the business, your skills are a commodity they purchase. The company may earn millions, but your salary will be a function of your negotiating skills, and the kind of 'game' the company plays in keeping their costs low.
This works the same way in all fields. There are hundreds of thousands of musicians that toil in obscurity, many of great skill. A few are wealthy, and some of them aren't even all that talented, but are packaged and sold well.
There are personality types that function best in an employment scenario. They lack the incentive and drive to make their own business (or the luck, which is part of it). There are some that simply can't be content as an employee, but must be a creator - an entrepreneur. Depending on what you really want, and what you really are capable of, you should figure out where you are best suited.
Underneath your original question, though, is a larger philosophical point. You are asking, almost literally, if you really have to work all that hard at something you're not that comfortable with, math namely. You asked, too, which is better.
First, you really need to learn the basic point that no matter what you do, you'll get out of it what you put into it. Either decide, and dedicate your entire mental energy toward that, or don't - and benefit accordingly. It's a bit harsh, but then it's also like asking if you have to really run all that fast to get into the Olympics.
Second, physics is more about math than graphics. Physics engines are so plentiful now that you're either relegated to using an existing engine, in which case you're not really a physics engine programmer - or you're inventing new ways to portray detailed physical phenomena, and thus heavily steeped in the mathematical means of portraying it.
