Wininet is making me cry

This is a discussion on Wininet is making me cry within the Windows Programming forums, part of the Platform Specific Boards category; I need to download certain files for my application. HttpQueryInfo() fails on two finals with 12150 (ERROR_HTTP_HEADER_NOT_FOUND). I can still ...

  1. #1
    A10
    A10 is offline
    Registered User
    Join Date
    Nov 2006
    Posts
    85

    Unhappy Wininet is making me cry

    I need to download certain files for my application. HttpQueryInfo() fails on two finals with 12150 (ERROR_HTTP_HEADER_NOT_FOUND).

    I can still download the file, but InternetReadFile() doesn't always download the entire file if I don't have the EXACT filesize. It's very strange. There must be something I'm doing wrone.

    Here's the function that I use to grab the files:
    Code:
    std::string Internet::getURL(const char *URL)
    {
         //get size of file
         std::string temp = "";
         hInternet = InternetOpen( "GINA: Version 0.1", INTERNET_OPEN_TYPE_DIRECT, NULL, 0,0);
         hFile = InternetOpenUrl(hInternet, URL, NULL, 0, 0, 0);
         DWORD sizeBuffer;
         DWORD length = sizeof(sizeBuffer);
         char *buffer = NULL;
         
         bool succeeds = HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, (LPVOID)&sizeBuffer, &length, NULL);
         if (!succeeds)
         {//query fails at /maps/
            int i = GetLastError();
            char buf[25];
            MessageBox(NULL,itoa(i,buf,10),"Error",0);
            //MessageBox(NULL,URL,"URL Failed Query",0);
            sizeBuffer = 65536; //read everything damn it
         }
         buffer = new char[sizeBuffer];
              
         //get file     
         DWORD bytesRead = 0;
         DWORD bytesToRead = sizeBuffer;
         if (hFile)
         {
                   do
                   { 
                     InternetReadFile(hFile, buffer, bytesToRead, &bytesRead);
                   }while (bytesRead != 0);
         }
         else
             throw "Unable to grab File";
         temp = buffer;
         InternetCloseHandle(hFile);
         InternetCloseHandle(hInternet);
         return temp;
    }
    The part where I set sizeBuffer to 65536 is my workaround that lets me download the files that fail with error code 12150.

    The URLS that fail (all others work fine with this function) are:
    Code:
    http://www.imperialconflict.com/maps/
    and: http://www.imperialconflict.com/rankings.php?type=topfamilies_score&g=11
    They are only partially retrieved with the above function. Sometimes they are not even retrieved continuously.

    hInternet,hFile are members of my class Internet


    Thanks for any help

  2. #2
    Sweet
    Join Date
    Aug 2002
    Location
    Tucson, Arizona
    Posts
    1,806
    The issue mainly is the use of your buffer.

    You create the buffer but you assume the buffer is big enough to hold the contents of the download. Which is not the case when you don't get the content length(Which can happen).

    I re-wrote your code not to be dependent on the content size and used a fixed buffer to download the file.
    Code:
    std::string GetUrl(const char *URL)
    {
    	//Constant to hold the read size(1k).
    	const int DownloadBufferSize = 1024;
    
    	//Constant to hold an empty string.
    	const std::string errorString = "ERROR";	
    
    	//
    	//Attempt to open a internet connection. Check for success.
    	//
    	HINTERNET hInternet = InternetOpen("GINA: Version 0.1", INTERNET_OPEN_TYPE_DIRECT, NULL, 0,0);
    	if(hInternet == NULL){
    		return errorString;
    	}//if
    
    	//
    	//Attempt to open the url. Check for success
    	//
    	HINTERNET hFile = InternetOpenUrl(hInternet, URL, NULL, 0, 0, 0);
    	if(hFile == NULL){
    		return errorString;
    	}//if
    
    	//
    	//These will be used to hold the content size(Potentially)
    	//
    	DWORD sizeBuffer;
    	DWORD length = sizeof(sizeBuffer);	
    
    	//Attempt to grab the content length of the request.
    	bool succeeds = HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, &sizeBuffer, &length, NULL) == TRUE;
    
    	//Special Case: If the query fails handle failure case, Else output the content length.
    	if (!succeeds){
    		//Grab the last error that occured.
    		DWORD lastError = GetLastError();
    
    		//Special Case: Header not found is expected. Not all pages output their content length.
    		if(lastError != ERROR_HTTP_HEADER_NOT_FOUND){
    			std::stringstream outputStream;
    			outputStream<<"Error: "<<lastError;
    
    			MessageBox(NULL, outputStream.str().c_str(), "Query Info Error", MB_OK);						
    		}//if
    	}//if
    	else{
    		std::stringstream outputStream;
    		outputStream<<"Content Length: "<<sizeBuffer;
    
    		MessageBox(NULL, outputStream.str().c_str(), "Query Info Error", MB_OK);
    	}//else
    	
    	//Will be used to store all of the downloaded data.
    	std::string downloadedContents = "";	
    
    	//This will be used to hold the buffer to download.
    	char *downloadBuffer = new char[DownloadBufferSize];
    
    	//Will be used to hold the number of bytes read.
    	DWORD bytesRead = 0;	
    	do{
    		//Read the current bytes from the internet file.
    		InternetReadFile(hFile, downloadBuffer, DownloadBufferSize, &bytesRead);
    
    		//Append the buffer to the download contents
    		downloadedContents.append(downloadBuffer, DownloadBufferSize);
    	}while(bytesRead != 0);
    	
    	//
    	//Close the handles.
    	//
    	InternetCloseHandle(hFile);
    	InternetCloseHandle(hInternet);
    
    	//Return the downloaded contents to the caller.
    	return downloadedContents;
    }
    Notice I don't really use the value of the content size. Sure you could but most of the time operations like this its best to use a fixed size buffer for downloading.

    You could however use the content size to do a progress bar or something if you wanted.
    Woop?

  3. #3
    A10
    A10 is offline
    Registered User
    Join Date
    Nov 2006
    Posts
    85
    Thanks for your response. Your rewrite was very very helpful.

    Unfortunately It wasn't 100% successful. When I took a look at the file the function downloaded, It was chewed up and not necessarily continuous. After a little bit of thinking I came up with a theory and a solution.

    InternetReadFile() only downloads as much data as is available. Depending on the connection, there isn't always 1k of data available. The file would appear to be chewed up because old data still in the buffer wouldn't have been written over.

    I've rewritten your function to download and append only as much data is available. I also increased the buffer size to 4096 and commented out the size query as I don't yet have a use for it. The code works flawlessly. Thanks for your help
    Code:
    #include <winInet.h>
    #include <string>
    #include <windows.h>
    //#include <sstream>
    
    std::string getURL(const char *URL)
    {
    	//Constant to hold the read size(4k).
    	const int downloadBufferSize = 4096;
    
        //Constant to hold an empty string.
    	const std::string errorString = "ERROR";
    	//
    	//Attempt to open a internet connection. Check for success.
    	//
    	HINTERNET hInternet = InternetOpen("GINA: Version 0.1", INTERNET_OPEN_TYPE_DIRECT, NULL, 0,0);
    	if(hInternet == NULL){
    		return errorString;
    	}//if
    
    	//
    	//Attempt to open the url. Check for success
    	//
    	HINTERNET hFile = InternetOpenUrl(hInternet, URL, NULL, 0, 0, 0);
    	if(hFile == NULL){
    		return errorString;
    	}//if
    
        /*
    	//
    	//These will be used to hold the content size(Potentially)
    	//
    	DWORD sizeBuffer;
    	DWORD length = sizeof(sizeBuffer);	
    
    	//Attempt to grab the content length of the request.
    	bool succeeds = HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, &sizeBuffer, &length, NULL) == TRUE;
    
    	//Special Case: If the query fails handle failure case, Else output the content length.
    	if (!succeeds){
    		//Grab the last error that occured.
    		DWORD lastError = GetLastError();
    
    		//Special Case: Header not found is expected. Not all pages output their content length.
    		if(lastError != ERROR_HTTP_HEADER_NOT_FOUND){
    			std::stringstream outputStream;
    			outputStream<<"Error: "<<lastError;
    
    			MessageBox(NULL, outputStream.str().c_str(), "Query Info Error", MB_OK);						
    		}//if
    	}//if
    	*/
    	
    	//Will be used to store all of the downloaded data.
    	std::string downloadedContents = "";	
    
    	//This will be used to hold the buffer to download.
    	char *downloadBuffer = new char[downloadBufferSize];
        DWORD availableSize = 0;
    	//Will be used to hold the number of bytes read.
    	DWORD bytesRead = 0;	
    	do{
              //query how much data is available
    		  InternetQueryDataAvailable(hFile,&availableSize,0,0);
    		  //read only as much as is available or downloadBufferSize
    		  if (availableSize > downloadBufferSize)
    		     availableSize = downloadBufferSize;
    		  //Read the current bytes from the internet file.
    		  InternetReadFile(hFile, downloadBuffer, availableSize, &bytesRead);
              
    		  //Append the buffer to the download contents
    		  downloadedContents.append(downloadBuffer, availableSize);
    	}while(bytesRead != 0);
    	
    	//
    	//Close the handles.
    	//
    	InternetCloseHandle(hFile);
    	InternetCloseHandle(hInternet);
    	//Return the downloaded contents to the caller.
    	return downloadedContents;
    }

  4. #4
    Sweet
    Join Date
    Aug 2002
    Location
    Tucson, Arizona
    Posts
    1,806
    Perfect. I forgot about that.

    Looks good, glad I could help.
    Woop?

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. making sprites
    By DavidP in forum Game Programming
    Replies: 9
    Last Post: 02-20-2010, 06:00 AM
  2. wininet
    By jopp3 in forum C Programming
    Replies: 0
    Last Post: 04-29-2007, 09:27 AM
  3. Making great graphics
    By MadCow257 in forum Game Programming
    Replies: 1
    Last Post: 02-20-2006, 10:59 PM
  4. Verbal abuse from school teachers -- how to get them caught or busted?
    By deltabird in forum A Brief History of Cprogramming.com
    Replies: 17
    Last Post: 02-13-2003, 06:26 AM
  5. About Unix Programming - Making a career desision
    By null in forum C Programming
    Replies: 0
    Last Post: 10-14-2001, 07:37 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21