Howdy
I'm considering writing a simple program which grabs a specific piece of information from a webpage online, well, multiple pages online. The page is dynamically created by polling a database through variables passed in the web address through the "get" method, i.e.
http://www.thispage.com/index.aspx?variable=1234
The variable increments by a specified amount which can be accounted for in the program. As far as I'm thinking, the best way to go about this would be using a loop to handle the increments for the variable that's changing. Create a new string containing "wget http://www.thispage.com/index.aspx?variable=%s", variable.
Then system(string) where string is that combined string. To extract what I'm looking for, then create a new string with something such as "grep value *.aspx* > output_file", system(secondstring) and then simply open that file and parse the single string (which was returned from grep and stored in a file named output_file). Finally, unlink the file before the beginning of each subsequent loop (for purposes of identifying if a page wasn't downloaded).
But I'm wondering if this is the best way to go about it? I'm comfortable with the code behind what I've described, but are there libraries available where I wouldn't have to use Unix for the wget/grep aspect of this?
