Just thought this may be useful to someone.
This is a generic scanner that can be used to tokenize any (I think!) programming language. The interface is straightforward, easy to use, and also compatible with other STL code.
Please let me know what you think!