Originally Posted by
laserlight
Ah yes, but just as a function should ideally do something and do it well, a class should ideally model one thing and model it well.
From what I gather, you are trying to parse some kind of assembly program and possibly simulate running the generated machine code. Clearly, your class does more than just read input ;)
One approach would be to create say, a Statement class. You can overload operator>> for std::istream in order to read a line of program text from an input stream and produce the corresponding Statement class object. This might use a private member function that parses the line of text and sets the statement type.
A variation of this approach would eliminate the type code by using a hierarchy of Statement classes. Statement would then be an abstract base class. You might have a virtual function that performs the part that is currently described as "Tokenizer () uses this number to do the operation on OurString". The advantage of this is that it follows the Open-Closed principle: you can add code to handle new kinds of operations without having to modify (much of) the existing code that handles the existing kinds of operations.