Extracting data from a webpage

[h3ro]

Hey.

I am looking to extract some data from a few websites for the purpose of some statistics work. I know I could do that by looking that the websites in question and then copy past the information I want into a file, but its not the most interesting thing in the world to do, so I though this would be a great chance for me to bring out my programming skills.

But I have never done anything like this before so I was wondering if anyone knew here to start? I started fooling around using the JavaScript console in Chrome, but I quickly found out that your not able to save data to a text file, which is essential to me .

So I was wondering if there is anyway I can write a script (language not very important, like learning new things) and just give it a start webpage and it will do what I want it to. All I need is being able to scan through the source of the webpage looking for keywords and save them to an text file. It would also be nice if I could open new links and have it running as a recursive processes. And the last thing which I would like, but it is not that important is the ability to save images as well.

Any ideas where I should start? Just to be clear, I not asking for anyone to write any code, I just asking for a push in the right direction when it comes to choosing what do use. I would also like to do it myself, even though I am sure there are lots of programs that already does this for you.

Regards,

[zacs7]

Well, something I'd do in Perl, Bash or Python. You may want to look into (any of) `wget`, curl and/or some sort of string parser (perhaps `awk` or `sed`) if you're not using Perl or Python that is...

Personally I'd pick Python

[h3ro]

Always wanted to use python, guess this is the time. Thanks

Which version on python is the one with best support, Or should I just get the latest one?
And, what is a good IDE for it? If you dont have anyone you really recommend, feel free to ignor the last two questions, im sure google will bring up a few good and long threads arguing about which is best.

[maxorator]

I've usually done this in PHP + string functions/regex.

[Fordy]

Which version on python is the one with best support, Or should I just get the latest one?
And, what is a good IDE for it? If you dont have anyone you really recommend, feel free to ignor the last two questions, im sure google will bring up a few good and long threads arguing about which is best.

I'd go with 2.5 - it has great OS support. I use 2.6, but it means I end up recompiling some stuff every now and again, and if you can get away with 2.5 then that's the one to use.

As far as an IDE, there are loads. I like Komodo Edit - it's free and pretty lightweight

[cyberfish]

Sounds like wget will do everything you want, including recursive downloading (following links), which would get ugly if you try to implement it yourself.

[mike_g]

Perhaps this is a bit late, but if you're using Python then its definitely worth checking out Beautiful Soup for scraping content.