I think the better your math is the better your programs will be. Smaller simplier and faster.
It's all about logic. I guess you could still have a logically thinking mind and do poorly in mathematics, but it would seem to go against the grain. Who really cares if you can't count to ten... Just as long as you can spell stdio.h right. I would really think that learning to program would boost your ability to do math and vise versa. Your not really going to be able to do much with computers without numbers are you? If you like programming, just stick with it. If you run across a math problem chances are the formula for the solution is a google away.
There is *some* math you should be well versed in if you intend to program, and that is Boolean Algebra. If you can't get your head around that, I would think you would also struggle with programming.