Wininet is making me cry

I need to download certain files for my application. HttpQueryInfo() fails on two finals with 12150 (ERROR_HTTP_HEADER_NOT_FOUND).

I can still download the file, but InternetReadFile() doesn't always download the entire file if I don't have the EXACT filesize. It's very strange. There must be something I'm doing wrone.

Here's the function that I use to grab the files:

Code:

---

std::string Internet::getURL(const char *URL)
{
    //get size of file
    std::string temp = "";
    hInternet = InternetOpen( "GINA: Version 0.1", INTERNET_OPEN_TYPE_DIRECT, NULL, 0,0);
    hFile = InternetOpenUrl(hInternet, URL, NULL, 0, 0, 0);
    DWORD sizeBuffer;
    DWORD length = sizeof(sizeBuffer);
    char *buffer = NULL;
   
    bool succeeds = HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, (LPVOID)&sizeBuffer, &length, NULL);
    if (!succeeds)
    {//query fails at /maps/
        int i = GetLastError();
        char buf[25];
        MessageBox(NULL,itoa(i,buf,10),"Error",0);
        //MessageBox(NULL,URL,"URL Failed Query",0);
        sizeBuffer = 65536; //read everything damn it
    }
    buffer = new char[sizeBuffer];
         
    //get file   
    DWORD bytesRead = 0;
    DWORD bytesToRead = sizeBuffer;
    if (hFile)
    {
              do
              {
                InternetReadFile(hFile, buffer, bytesToRead, &bytesRead);
              }while (bytesRead != 0);
    }
    else
        throw "Unable to grab File";
    temp = buffer;
    InternetCloseHandle(hFile);
    InternetCloseHandle(hInternet);
    return temp;
}

---

The part where I set sizeBuffer to 65536 is my workaround that lets me download the files that fail with error code 12150.

The URLS that fail (all others work fine with this function) are:

Code:

---

http://www.imperialconflict.com/maps/
and: http://www.imperialconflict.com/rankings.php?type=topfamilies_score&g=11

---

They are only partially retrieved with the above function. Sometimes they are not even retrieved continuously.

hInternet,hFile are members of my class Internet


Thanks for any help

The issue mainly is the use of your buffer.

You create the buffer but you assume the buffer is big enough to hold the contents of the download. Which is not the case when you don't get the content length(Which can happen).

I re-wrote your code not to be dependent on the content size and used a fixed buffer to download the file.

Code:

---

std::string GetUrl(const char *URL)
{
        //Constant to hold the read size(1k).
        const int DownloadBufferSize = 1024;

        //Constant to hold an empty string.
        const std::string errorString = "ERROR";       

        //
        //Attempt to open a internet connection. Check for success.
        //
        HINTERNET hInternet = InternetOpen("GINA: Version 0.1", INTERNET_OPEN_TYPE_DIRECT, NULL, 0,0);
        if(hInternet == NULL){
                return errorString;
        }//if

        //
        //Attempt to open the url. Check for success
        //
        HINTERNET hFile = InternetOpenUrl(hInternet, URL, NULL, 0, 0, 0);
        if(hFile == NULL){
                return errorString;
        }//if

        //
        //These will be used to hold the content size(Potentially)
        //
        DWORD sizeBuffer;
        DWORD length = sizeof(sizeBuffer);       

        //Attempt to grab the content length of the request.
        bool succeeds = HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, &sizeBuffer, &length, NULL) == TRUE;

        //Special Case: If the query fails handle failure case, Else output the content length.
        if (!succeeds){
                //Grab the last error that occured.
                DWORD lastError = GetLastError();

                //Special Case: Header not found is expected. Not all pages output their content length.
                if(lastError != ERROR_HTTP_HEADER_NOT_FOUND){
                        std::stringstream outputStream;
                        outputStream<<"Error: "<<lastError;

                        MessageBox(NULL, outputStream.str().c_str(), "Query Info Error", MB_OK);                                               
                }//if
        }//if
        else{
                std::stringstream outputStream;
                outputStream<<"Content Length: "<<sizeBuffer;

                MessageBox(NULL, outputStream.str().c_str(), "Query Info Error", MB_OK);
        }//else
       
        //Will be used to store all of the downloaded data.
        std::string downloadedContents = "";       

        //This will be used to hold the buffer to download.
        char *downloadBuffer = new char[DownloadBufferSize];

        //Will be used to hold the number of bytes read.
        DWORD bytesRead = 0;       
        do{
                //Read the current bytes from the internet file.
                InternetReadFile(hFile, downloadBuffer, DownloadBufferSize, &bytesRead);

                //Append the buffer to the download contents
                downloadedContents.append(downloadBuffer, DownloadBufferSize);
        }while(bytesRead != 0);
       
        //
        //Close the handles.
        //
        InternetCloseHandle(hFile);
        InternetCloseHandle(hInternet);

        //Return the downloaded contents to the caller.
        return downloadedContents;
}

---

Notice I don't really use the value of the content size. Sure you could but most of the time operations like this its best to use a fixed size buffer for downloading.

You could however use the content size to do a progress bar or something if you wanted.

Thanks for your response. Your rewrite was very very helpful.

Unfortunately It wasn't 100% successful. When I took a look at the file the function downloaded, It was chewed up and not necessarily continuous. After a little bit of thinking I came up with a theory and a solution.

InternetReadFile() only downloads as much data as is available. Depending on the connection, there isn't always 1k of data available. The file would appear to be chewed up because old data still in the buffer wouldn't have been written over.

I've rewritten your function to download and append only as much data is available. I also increased the buffer size to 4096 and commented out the size query as I don't yet have a use for it. The code works flawlessly. Thanks for your help :)

Code:

---

#include <winInet.h>
#include <string>
#include <windows.h>
//#include <sstream>

std::string getURL(const char *URL)
{
        //Constant to hold the read size(4k).
        const int downloadBufferSize = 4096;

    //Constant to hold an empty string.
        const std::string errorString = "ERROR";
        //
        //Attempt to open a internet connection. Check for success.
        //
        HINTERNET hInternet = InternetOpen("GINA: Version 0.1", INTERNET_OPEN_TYPE_DIRECT, NULL, 0,0);
        if(hInternet == NULL){
                return errorString;
        }//if

        //
        //Attempt to open the url. Check for success
        //
        HINTERNET hFile = InternetOpenUrl(hInternet, URL, NULL, 0, 0, 0);
        if(hFile == NULL){
                return errorString;
        }//if

    /*
        //
        //These will be used to hold the content size(Potentially)
        //
        DWORD sizeBuffer;
        DWORD length = sizeof(sizeBuffer);       

        //Attempt to grab the content length of the request.
        bool succeeds = HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, &sizeBuffer, &length, NULL) == TRUE;

        //Special Case: If the query fails handle failure case, Else output the content length.
        if (!succeeds){
                //Grab the last error that occured.
                DWORD lastError = GetLastError();

                //Special Case: Header not found is expected. Not all pages output their content length.
                if(lastError != ERROR_HTTP_HEADER_NOT_FOUND){
                        std::stringstream outputStream;
                        outputStream<<"Error: "<<lastError;

                        MessageBox(NULL, outputStream.str().c_str(), "Query Info Error", MB_OK);                                               
                }//if
        }//if
        */
       
        //Will be used to store all of the downloaded data.
        std::string downloadedContents = "";       

        //This will be used to hold the buffer to download.
        char *downloadBuffer = new char[downloadBufferSize];
    DWORD availableSize = 0;
        //Will be used to hold the number of bytes read.
        DWORD bytesRead = 0;       
        do{
          //query how much data is available
                  InternetQueryDataAvailable(hFile,&availableSize,0,0);
                  //read only as much as is available or downloadBufferSize
                  if (availableSize > downloadBufferSize)
                    availableSize = downloadBufferSize;
                  //Read the current bytes from the internet file.
                  InternetReadFile(hFile, downloadBuffer, availableSize, &bytesRead);
         
                  //Append the buffer to the download contents
                  downloadedContents.append(downloadBuffer, availableSize);
        }while(bytesRead != 0);
       
        //
        //Close the handles.
        //
        InternetCloseHandle(hFile);
        InternetCloseHandle(hInternet);
        //Return the downloaded contents to the caller.
        return downloadedContents;
}

---

Perfect. I forgot about that.

Looks good, glad I could help.