Wearing a hajib or other religious symbols should not be banned for the sole reason it is freedom of speech. If some, I am using this as an example, gay person in France wore a huge rainbow colored T-Shirt with the words "I am gay and proud of it" would they be punished? If not, then how is a Crucifix(which is not just a christian symbol, but an ancient Egyption symbol) or the Hajib(which can be viewed as a religiously based hat(which I presume are not banned)) be viewed as a inflamatory symbol anywhere close to my example?
Well freedom of speech is an important point, but many countries (I don't know about the US) do not consider freedom of speech to be absolute per say, for example here in the UK it's against the law to incite violence, so there are some limitations on what i can say in public.

I think France is worried about rising extremism, they have presumeably decided that banning religious symbols in school is worth the sacrifice in freedom of speech.