Well, what you asked for in your first post is done fairly simply with scanf, like so:

char token[64 + 1];

while (scanf("%64s", token) == 1) /* handle token */

The downside of this method is...