Thread: String/token parsing

I'm attempting to write some code that will extract certain tokens from a string of variable length. The code I have now is long and messy (loaded with switch statements, if/thens, etc), and I'm sure there's a more elegant solution. For the sake of brevity, say this is the list of valid tokens:

Code:

A
AZ
@
:...:        //String between colons to be read as a string and not tokenized
{0-9}    //Any single-digit integer)

So, valid strings could be:

Code:

A@5AAZ
@01:Hello, world!:@A
AAAZ:AZ:A1
@7@7AZ

For the most part, the strings will be >255 characters in length, and read from a plain text file. The number of valid tokens is more likeley to be several dozen. Some tokens will be followed by a string literal. The code also needs to be able to ignore anything that isn't a valid token. Order counts.

Are there any tricks to make this easier? Or do I really need to brute force my way through the string? As it stands now, my code is pages of this:

Code:

switch(input_char)
{
case 'A':
    if(next_char == 'Z')
    {
        //do stuff
    }
    else
    {
        //do some other stuff
    }
    break;
//etc.
}

It's functional, but it's pretty ugly. And anytime I need to add/change a token, it's not easy. Any help would be greatly appreciated.

Is this an academic or a practical question?

If 2.,

Have a look at boost tokenizer. It does the dirty work for you.
http://www.boost.org/libs/tokenizer/index.html

If you need more advanced parsing, you could try boost spirit
http://spirit.sourceforge.net/

Originally Posted by pheres

Is this an academic or a practical question?

It's a little bit of each. It's academic in that I like to know how/why things are done (I'm constantly reinventing the wheel). It's practical in that my code is becoming impractical, and I need a better solution (that, and I generally find that my wheel isn't as good as whatever else is out there).

FWIW, this is for a silly project of mine. It's not a homework question.