Will you take it easy? It isn't as black and white as that.
Some text should really be smaller, some larger.
Text menu items and things like that are suited for smaller font.
But body text is not.
So even when I set the minimum font size to 12, sites get messed up because they can't show 11px in some places where it would fit, so it not a solution.
The solution is, like you said, to use relative size.