I think you're on the verge of turning your tokenizer into a parser. It does not need to be more complicated than separating tokens from white space. Separating tokens like "j=+3" or "j+3" into parts of your language, like expression, operator, expression, is what the parser does.
I can not think of a better time to remove excess white space than during tokenizing, either. You want parsing to be as easy as it can be so that it's correct.
