Thread: Query multiple servers quickly

Hello,

I am trying to determine the best possible solution to accomplish a simple, yet complex task.

I've been working on a new project for quite some time and have written a custom framework from the ground up. It's more or less a (Client --> Server --> Server <-- Client) system.

So far I have the basics done. Here's the process:

1. I expect to have one client connect to my listening server.
2. That client will periodically send me a string of IP Addresses (10 or more at a time).
3. When I receive the data, I split it up accordingly and query the IPs just received.

All of that I can handle and have completed the code for up to 95%. The string of IP Addresses I expect to receive are of other servers in which I will attempt to query and gather data from.

However, the problem is I may need to query up to 100 or 1000 servers and I don't want to have to wait for one to finish, in case it's unresponsive and gets delayed the full timeout length before moving on.

With that said, I want to query as many servers as fast as possible. And I don't care which order they come back in. As soon as I receive the data I just need to save it to a file (no further communication to the original client is required).

I am writing this in C and in the Linux environment -- so is there a way to even do that? Would I need to use the fork() command or write my own custom Socket Pool to handle multiple outgoing connections at once?


- Stack Overflow

Welcome back, you've been away far too long.

You could look at the way nmap does this.

Originally Posted by Salem

Welcome back, you've been away far too long.

Thanks. It's good to be back! 8)

Originally Posted by Salem

You could look at the way nmap does this.

That suggestion may be of some help down the line, however I'd like to show you an example of what I'm trying to do in code format:

Code:

void queryServers(int n, char data[][64]) {
	/*** n = amount of servers to query ***/
	/*** data[x] is the string holding IP and Port info ***/
	/*** Example: 123.45.67.89:10110 */

	/* local variables ... */
	int i, port;

	/* loop through all servers needed to be queried */
	for (i = 0; i < n; i++) {
		/* TODO: parse port from data[] and convert to integer */
		port = atoi( [i] );

		/* function to query server */
		queryData(data[i], port);
	}

	/* end */
	return;
}

Keep in mind that I have already written the query code. I know exactly what to query and what data I should receive. Very much like querying a game server -- you query a specific IP & Port and it dumps a string back to you (like map name, etc). I just need to save that dump information to a file.

What I'm struggling with is developing an efficient way to complete a lot of server queries without losing time if one them times out or hangs.

My code above is very incomplete, as you can probably tell, but it's the basic idea. The for loop is very bad because it only processes one query at a time and I want to find a good way to tackle a lot of them even if I have to open multiple sockets and threads to get it done.

So I'd find a way to open 10 different connections to 10 different servers and begin querying. As soon as one is done, I'd close that connection and begin querying the 11th. So on and so forth... If one of them takes fifteen seconds to complete the query and the other 9 finish sooner than that, it then frees up more slots to begin querying more servers while that slow one doesn't affect the rest of the group.

I just don't know where to begin and what the best concept would be. And I don't know if it's even possible, but I think it should be.

Excellent suggestion, Codeplug.

I've made a lot of progress since your post and have decided to implement non-blocking sockets into my program. I have one question though:

How do you send out data and then catch the response using non-blocking processing?

So far I have a non-blocking server that does what you explained in your post -- it waits to see if there are any incoming connections or data, but doesn't block. However I can't seem to find out how to query another server and have my non-blocking server pick up the response. I think it may be related to the sendto() or recvfrom() command, but I can't get my head around it.

A diagram:

Code:

     Client
	\
	 \
	  \
	Server		Non-Blocking Server
	    \		  /
	     \		 /
	      \		/
	  	Server

So to clarify, the blue and teal colored objects are my listening servers. I use the fork() command to give my Non-Blocking Server full control when it's time to start querying the list of servers sent by the original client. I initialize the NB server only when it's time to start querying and close it down when I'm completely done.

Layout: One Client connects to my basic Server. That Client sends a string of IP Addresses to me. I parse the received data into a readable list and start querying the Server's.

Lastly, which is where I'm stuck, I need the received data from the Server's to send its data to my Non-Blocking Server -- so it doesn't block per server query in case I have 1000 servers to poll.

I have the non-blocking server configured 100&#37;. I just need to know if it's possible to receive that data to a specific socket or port. Also, as a piece of information, my two servers operate on two different ports. For example my basic server listens on port 10110 and my NB server listens on 10111. I need to poll results the send to port 10111 so my NB server can pick it up properly.

I hope that makes sense. Thanks again for any assistance you may be able to provide.

Thank you very much for all of your help and suggestions, Codeplug.

I haven't had much time to code this week nor have I thoroughly read your post, but it looks very insightful and I will begin implementation on Monday.

Thanks again and I will keep you informed.