How to extract variables from between html tags?

Suppose something like this:

Code:

<span class="gugugug">1</span>

<span class="krakaka">enchanted</span><span class="gagaga">chocolate bar</span>

<span class="gugugug">2</span>

<span class="krakaka">very remarkable</span><span class="gagaga">flavored cookies</span>

<span class="gugugug">3</span>

<span class="krakaka">fascinating</span><span class="gagaga">strawberries</span>

(...)

<span class="gugugug">254</span>

<span class="krakaka">amazing</span><span class="gagaga">pineapples</span>

(...)

And it goes on.

I need to extract the data between the <span> tags and put it in a bidimensional array. But what piece of code do I need in order to extract the words between the tags?

look at libxml. it can also be used with html, and I've heard it works quite well.

The 'piece of code' is called an HTML parser.

Well, I don't really want to do anything other than extract the words between the tags. I want "1", "enchanted" and "chocolate bar" to be on the first vertical row of the bidimensional array, and "2", "very remarkable" and "flavored cookies" on the second vertical row of the bidimensional array, and so on.

"gugugug", "krakaka" and "gagaga" are not important. So basically, it's not important that it's html.

I know that if I use this:

Code:

char lol[] = { '<', 's', 'p', 'a', 'n', ' ', 'c', 'l', 'a', 's', 's', '=', '"', 'g', 'u', 'g', 'u', 'g', 'u', 'g', '"', '>', '\0' };

I can treat the html bit as a simple string that I want to ignore.

But I don't know how to get what is between the <span> and </span>.

Is there a way to tell C++ to go to the end of <span class="gugugug">, add what is between the > and the </span> to an array, and then go to <span class="krakaka">, get what is between the > and < over there, and then continue in that style?

I'm thinking it's simple code, so can you help me out?

And what if you get:

Code:

<span    class  =   'gugugug'         

>

>> I can treat the html bit as a simple string that I want to ignore.

Sure, if you want to produce a fragile, broken piece of software.

>> Is there a way to tell C++ to go to the end of <span class="gugugug">, add what is between the > and the </span> to an array, and then go to <span class="krakaka">, get what is between the > and < over there, and then continue in that style?

You can 'tell C++' whatever you like, but the bottom line is you're going to either need to parse the the data byte-by-byte, or find a library/API that can do it for you.

Sounds like you need the data in an easier format, though, like CSV or such.

Well, it's not a lot of data. I want to be able to get the important bits from a few small html files. Parsing byte-by-byte is fine.

It's as though, instead of html, I had something like this:

Code:

>1<

>enchanted<>chocolate bar<

>2<

>very remarkable<>flavored cookies<

>3<

>fascinating<>strawberries<

(...)

>254<

>amazing<>pineapples<

(...)

I'm new, but you're really good, so you can help me, please? If you show me how to get what's between the > and <, I can build a loop with that.

Haha purpleturple, inventing your own format? Just use XML or JSON, and use a library to parse it. I had issues with JSON, even though it's great, so I recommend XML (it's just very simple/structured HTML). I use Boost.Serialize (the xml parser, which probably just wraps libxml)

You *could* also use Boost.Regex easily if the format is simple (which yours seems to be). You just need to figure out the correct regular expression.

Originally Posted by Sebastiani

And what if you get:

Code:

<span    class  =   'gugugug'         

>

Strip/format the code before parsing, or have procedures to ignore whitespace at certain stages in the tree. I don't think that's an issue since he seems to be in control of his own format and files. However he should probably use a library just in case.

Actually, I'm completely new to programming. I was just hoping there was a really simple, short solution, something about getting to the end of the <span class="whatever">, and then using getc until it gets to the <, and then just adding what's in-between the brackets to its appropriate position in the array, and then continuing the loop. (Which I don't know how to do.)

Boost.Regex looks very complicated to me at this point, but I suppose I'll have to find a way to get my head around it if there's no other way. Though it almost looks like using a Rube Goldberg machine.

There is such a function as find, if that's what you're looking for. (Or alternatively, you can tell getline where to stop reading.)

[edit] Drat, that "completely new to programming" post wasn't there when I started this. Sorry about that. I expect this is a bit too complicated. [/edit]

It might be an interesting exercise for purpleturple to try to create a simple parser, nevertheless. I should note right here that creating your own parser is quite a bit of work, and it probably won't work perfectly, and so on; using someone else's code will make your life easier. But if you want to try it, or just wonder how such a library might go about the parsing (note that it's almost certainly more sophisticated than what I present here . . .), read on.

The first thing to note is that your program needs to tell the difference between being "inside" a tag, and not. A proper parser would need to tell whether it's inside a start tag or an end tag, and remember the names of the tags to ensure that opening tags are matched by proper closing tags, and a lot of other information. But if you're willing to assume that your input is valid, and you are only interested in looking at certain parts of the file, you can get away with less.

Let's start with figuring out whether you're inside a tag or not. It's really quite simple. Maintain a variable which either holds the value "inside" or "outside". When you see a '<', set it to "inside"; when you see a '>', set it to outside. I think you can probably figure that part out. Something like this would do the trick:

Code:

#include <iostream>
#include <fstream>
#include <string>

void parse(std::istream &stream);

int main(int argc, char *argv[]) {
    if(argc < 1) {
        std::cerr << "Usage: " << argv[0] << " <file.xml>\n";
        return 1;
    }
    
    std::ifstream stream(argv[1]);
    if(!stream.is_open()) {
        std::cerr << "Can't open \"" << argv[1] << "\"\n";
        return 1;
    }
    
    parse(stream);
    
    return 0;
}

void parse(std::istream &stream) {
    bool inside_tag = false;
    
    std::string line;
    while(std::getline(stream, line)) {
        for(std::string::size_type x = 0; x < line.length(); x ++) {
            if(line[x] == '<') {
                inside_tag = true;
            }
            else if(line[x] == '>') {
                inside_tag = false;
            }
            else {
                // it's not a special character
                if(!inside_tag) {
                    std::cout << line[x];
                }
            }
        }
        
        std::cout << std::endl;
    }
}

That just prints any data which lies outside any XML (or HTML) tags. I'm using as my input

Code:

<?xml version="1.0"?>
<root>
    <something>Greetings.</something>
    <something>Hello.</something>
</root>

Now you need to do some more processing when you're inside a tag, to find its name (if you care about it), attributes, and so on. It helps when you're doing this to have a get_token() function or equivalent, which skips whitespace and reads the next word in your input. But you have to be careful that as you're doing so, you don't miss any '<'s or '>'s which would change your state.

Then once you get that working, you have to consider the first token inside a tag to be its name. Subsequent tokens should be followed by the token "=", and then some sort of double-quote token (or a single quote, if you allow those). And so on. Note that these are different types of tokens here; while looking for attribute names etc you want a token of alphanumeric characters; while looking for '=' or '"' you want a single, non-whitespace character token.

It gets complicated, as you see. Maybe you might want to look into some XML parsing library now . . . ?

Yeah, this is an "oh, programming must be easy" post. Not quite, cupcake. It's why some of us get paid the big bucks

Stop using char pointers and use string and stringstream's. They have all the functions you need to do whatever you want. Lookup std::string, std::stringstream and their respective built-in functions. Believe me, it's far easier to use those than to try to do it manually.

Dae's suggestion about using Boost is good if you're able to get the libraries installed in your compiler paths but if you're going for something simple stick with the standard STL containers.