Thread: Parsing multiple pages of dynamically created Websites

IMO you do not actually make much sense. I would also guess that

I'm comfortable with the code behind what I've described

does not refer your (obviously non-existent) knowledge of the C programming language. Please be honest, life is much easier that way.

Anyway, w/r/t options instead of wget: you will need to use the socket API, and process tcp/ip packets, which is far from simple. Just to grab a page is like 50-100 lines of code.

w/r/t options instead of grep: there is a regexp library for C (regex.h) but you don't need to do that, once you have the page (you don't have to save anything to a file, you can keep it in memory) this would be a reasonably basic parsing exercise, difficult but not impossible for a beginner.

tip: if this is your area of interest, you will be better off pursuing bash scripting or *perl*. To do this completely in C without any system calls will be several hundred lines, in perl <50.

I'm thick skinned, no harm done in your opinion. Though I was surprised to read your attitude towards new posters.

Code:

int main(int argc, char **argv) {

	char http_reference[100];
	int i = 0;

	while (i < 100) 
		{ // grab and parse 100 pages
		sprintf(http_reference, "wget http://www.somedomain.com/index.aspx?variable=%d", i * 12);
		system(http_reference);
		sprintf(http_reference, "grep mystring *.aspx* > my_output_file");
		system(http_reference);
		parseMyFile();
		sprintf(http_reference, "index.aspx?variable=%d", i * 12);
		unlink(http_reference);
		unlink("my_output_file");
		i++; 
	}

return 0;
}

void parseMyFile() {

	FILE *mof; // my_output_file
	char c;

	if ((mof = fopen("my_output_file", "r")) == NULL) 
	{
		exit(0); 
	}

	while ((c = getc(mof)) != 'Z') {	}

	// Do something meaningful with c and the next n characters.

return;
}

I'd like some alternatives or a reference to a C library to simplify this. (And I wouldn't be surprised if there were a few problems with the above code as it's something thrown together as a concept. Trying to offer it as a quick idea to what I'm aiming for.)

Your skin ain't thick, you're paranoid if you think I'm even a little bit mean. Or else I am turning into exactly the kind of insensitive bastard I despised when I was I new poster Maybe you thot this was the "Rainbow Brite" forum ?

That does clarify your intent, well done. Take a crack at sockets if you want; they are challanging. Here is a thing I wrote last year when learning; it downloads an (eg jpg) image off the net as raw data and writes it back into a file, if you want some idea of what's involved in such an apparently simple task (after that, find a tutorial). You will notice the GET HTTP request in main() at the bottom*.

But first, work on taking the grep out. That will mean expanding parseMyFile() to open the file, locate the string and do what you will. As I mentioned before, there is a regex.h, if you have some knowledge or experience of regular expressions (grep = global regular expression). That is not necessary, however. It can be done fairly easily by processing one line at a time, in a loop, using general string functions like strcmp() and strstr().

* Here is the "mine.h" if you actually want to compile and try that, but you don't really need to bother -- this is just a clue as to what you are getting yourself into, sans wget.

Originally Posted by pete142

Don'cha love to learn new stuff? Thanks a million MK27!

I hope you are not being sarcastic or something pete Wow was I cranky on 5/25/09.

Anyway, I did try to introduce that with the caveat that it is something I wrote while learning last year. I put stuff like that on-line as a reference mostly for myself, but occasionally it seems pertinent to others -- it's nice to see someone made some use of it! One day in the near future I may go thru those and update them; for example I now never never put strlen in a for() condition, something that you caught. I still believe it is probably a decent demo of how to use the socket API functions, eg, the syntax is correct and it does work. Unfortunately I was obviously not interested in /* adding comments */ much at the time.

I'm also not using that header anymore; almost certainly I did write "pretrunc" for use there, but made it a generalized function (hence it's in that header). So my expectation was that "pretrunc'ing" a string was something I might do in various programs.

Time has revealed, however, that I don't really need a library of "generalized functions" like that much, since I now think C functions are better the more they are specialized. Which might explain why there are surprisingly few C libraries for anything, compared to many languages where half the programming can be done by including the right::modules or whatever.

Thanks btw pete for the x ^= 1 toggle.

Originally Posted by linuxdude

Meh. I'd use python. Are you required to use C?

No, I think I recommended perl to the OP. Or just a bash script, since he/she likes wget.

Keep in mind tho, the people who wrote python and perl for your convenience used C to do it. So behind your ".url" functions there is probably some hundreds of lines of code much like that.

Wow! Guess what I did this afternoon: I re-wrote that "grabimage" program and added more comments, since a few different people seemed interested, and I have recieved pm's about it, etc.

grabimage.c

It is much more comprehensible now and doesn't require a non-standard header. Plus I found a buffer overflow in the old one, and my method for determining the end of the HTTP header was flawed.

Originally Posted by bithub

There are still some things you may want to look into fixing:
- You assume that HTTP headers are case sensitive. They are not.

Good point.

- What happens if no Content-Length header is sent back?
- You don't check the return values of fopen() and malloc().

I believe Content-Length is required, but in any case the header prints out so the user will notice this. It's not really intended to be a program anyone would actually use, more just to demonstrate some things that IMO slipped thru the cracks of a lot of socket tutorials. For that, I ain't adding clutter checking fopen() and malloc().

- while (buffer[0]!='\r') { The first time through this loop, you are accessing undefined memory. The contents of buffer[0] are undefined.

Yep! Will fix.

Originally Posted by bithub

No, it is not. If a server is using Chunked Encoding, then the size of the response is embedded in the response data.

Maybe I will add a little note about that then.

Thanks bithub!