read() frozen....

[Kleid-0]

Hello! This works about 85% of the time, but the other 15% fails. It's not a random fail, it's dependent on the website...You see I'm creating this HTTP scanner, I'm getting their HTTP header and creating statistics on web server information. So here's the problem:

static int xread(int fd, char *buf, int len) {
  int o,l,r;
  o=0;
  l=len;
  for(;;){
  
  	 puts( "Reading sentence.." );	// <-- Looking good
    r=read(fd, buf+o, l);
	 puts( "DONE" );  // <-- This never happens....
    if(r<0){
      if(errno==EAGAIN || errno==EINTR)
        continue;
      else
        return -1;
    }
    if(r==0)
      return o;
    o+=r;
    l-=r;
    if(l)
      continue;
    return len;
  }
}

fd is a socket that is connected to some stranger's website. I know it exists because I connected to it successfully. It's just the reading freezes. :[.... does anyone know a good solution?

------
UPDATE

http://216.248.62.204/ <-- That's the bad one
It's funny though, because http://216.248.62.203/ is basically identical on the browser, yet it gets read just fine.

[nkhambal]

Hi,

read() is a blocking function (which is what freeze means in your post). i.e. it will not return untill some data is received from server. Till that it will not return any value to "r". If an error occures it will return -1 to "r", it will return 0 when connection is closed by the server otherwise it will simply block till some data is received. (freezes)

The return value, EAGAIN will not occure untill your you have made your socket as NONBLOCKING which is not adviced for an HTTP client program.

To receive any data from HTTP server you need to send HTTP GET request to the server first. Since you have not posted the full code, I assume you have that part right, Otherwise I suggest you to read this small presentation.

[Kleid-0]

Hey thank you nkhambal. I tried to setting the socket to NONBLOCKING. I scanned these IPs:
216.248.62.200 to 216.248.62.204

They all had the same errors (EAGAIN), yet when I switched back to BLOCKING (because I just used NONBLOCKING for validation to see if there is something to read), 216.248.62.204 froze...because it passed as a valid NONBLOCKING source.

I don't know what to do :[

// Make sure the socket has something to read by validating it with NONBLOCKING capabilites
static Bool check_read(int fd,char *buf, int len, struct timeval *timeout, int *err)
{
	int saveflags,ret,back_err;
	unsigned int ulen;
	fd_set fd_w;

	
	/* Save flags from socket */
	saveflags=fcntl(fd,F_GETFL,0);
	if(saveflags<0) { comment( "fcntl1" ); *err = errno; return FALSE; }

	puts( "Check 1" );
	
	/* Set socket to nonblocking */
	if(fcntl(fd,F_SETFL,saveflags|O_NONBLOCK)<0) { comment("fcntl2"); *err=errno; return FALSE; }

	puts( "Check 2" );
	
	/* Nonblocking returns immediately */
	*err=read(fd, buf, len);
	back_err=errno;

	puts( "Check 3" );
	
	/* Restore flags */
	if(fcntl(fd,F_SETFL,saveflags)<0) { comment("fcntl3"); *err=errno; return FALSE; }

	puts( "Check 4" );

	printf( "back_err: %ld\n*err: %ld\n", ( long )back_err, (long)*err );
	if( back_err != EAGAIN )
		return FALSE;
	/* Keep going unless the connection was a failure AND not in progress 
	if(*err<0 && back_err!=EINPROGRESS) { comment("connect"); *err=errno; return FALSE; }*/
	FD_ZERO( &fd_w );
	FD_SET( fd, &fd_w );

	puts( "Check 5" );
	
	/* Select... */
	*err = select( FD_SETSIZE, NULL, &fd_w, NULL, timeout );
	if(*err<0) { comment("select"); *err=errno; return FALSE; }
	
	puts( "Check 6" );

	/* 0 means it timeout out & no fds changed */
	if(*err==0) { close(fd); *err=ETIMEDOUT; return FALSE; }

	puts( "Check 7" );
	
	/* Get the return code from the connect */
	len=sizeof(ret);
	ulen = (unsigned int) len;
	*err=getsockopt(fd,SOL_SOCKET,SO_ERROR,&ret,&ulen);
	if(*err<0) { comment("getsockopt"); *err=errno; return FALSE; }

	puts( "Check 8" );
	
	/* ret=0 means success, otherwise it contains the errno */
	if( ret ) { *err=ret; return FALSE; }
	
	puts( "Check 9" );

	*err=0;
	return TRUE;
}

// Do reading here
static int xread(int fd, char *buf, int len) {
  int o=0,l=len,r;
  
  struct timeval to;
  int err;
  to.tv_sec = 5;
  to.tv_usec = 0;
  
  if( !check_read( fd, buf, len, &to, &err ) )
	 	return -1;
  memset(buf,0,sizeof buf);
  
  for(;;){
  
  	 puts( "Reading sentence.." );	
	 
    r=read(fd, buf+o, l);
	 puts( "DONE" );
    if(r<0){
      if(errno==EAGAIN || errno==EINTR)
        continue;
      else
        return -1;
    }
    if(r==0)
      return o;
    o+=r;
    l-=r;
    if(l)
      continue;
    return len;
  }
}

[Kleid-0]

Alright, I cheated and used some global variables and a POSIX thread. At least now if read() freezes, I'll have my main thread shred the read() worker into pieces and grab what it can from what was discovered. Give me some ideas if you guys have better ideas :-].

[nkhambal]

Are you sending the HTTP GET command properly ? Cause right now that seems to be your problem. Making a socket nonblocking or thread will not solve your problem, unless you send your request properly. For an HTTP client program your socket has to be blocking. I am not sure why you are creating the new thread to receive the data.

Also, I would suggest using recv() function in place of read(). recv() is usually used with sockets for sending and receiving the data.

[Kleid-0]

I'm using HTTP HEAD to get data. I have no need for the whole web page content (as would happen with HTTP GET). The read() freeze happens only rarely, so the HTTP HEAD shouldn't be the problem, I believe, because all sites I send to are all sent the same HTTP HEAD information.

recv() sounds like an awesome idea :]. I'll use it if my POSIX thread/read() combo fails in the future.