Thread: Socket Programming: Retrieve file from HTTP

  1. #1
    Registered User Tommo's Avatar
    Join Date
    Jun 2007
    Location
    Scotland
    Posts
    101

    Socket Programming: Retrieve file from HTTP

    Hi. I am writing a program that simply downloads and saves a file over HTTP. See attached code.

    I was wondering if anyone had any hints or tips on this matter or perhaps my code in general and point out any glaring problems.

    Thanks for any advice.
    Last edited by Tommo; 09-03-2007 at 11:58 AM.

  2. #2
    Registered User
    Join Date
    Apr 2007
    Location
    Sydney, Australia
    Posts
    217
    I had a quick look. If you want you can look at some http stuff i made a few days ago (it can break apart a url and download stuff) and compare yours to mine if yours isnt working.

    http://host-a.net/39ster/httpstuff.rar

    The files are .cpp but they should compile in C also.

  3. #3
    Registered User Tommo's Avatar
    Join Date
    Jun 2007
    Location
    Scotland
    Posts
    101
    cool thanks. I'll have a look. I've gotten a bit further with mine. I've just got to use memcpy to save the file as binary rather than just text.

  4. #4
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Why do you need memcpy()? Just use fwrite() - or are you trying to build a larger block of data before you save it to file?

    --
    Mats

  5. #5
    Registered User Tommo's Avatar
    Join Date
    Jun 2007
    Location
    Scotland
    Posts
    101
    No I need to skip the header, then write the block of data to a file. Sorry, what I said above sounds odd; I do use fwrite. My GetRidOfHeaders function used strcpy which I now need to change.

  6. #6
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Why not just figure out where the headers end, and then fwrite that?

    something like:
    Code:
        char buf[...];
        size_t size;
        size_t totalsize;
        totalsize = getstuff(buf);
        size = headersize(buf);
        fwrite(&buf[size], totalsize - size, 1, file);
    --
    Mats

  7. #7
    Registered User Tommo's Avatar
    Join Date
    Jun 2007
    Location
    Scotland
    Posts
    101
    Yeah that's pretty much what I've been trying to do. I've attached my code, as it's a bit different now. Thanks for the suggestion, what you've got there looks sensible.

    Edit: Finished!
    Last edited by Tommo; 09-04-2007 at 12:51 PM.

  8. #8
    int x = *((int *) NULL); Cactus_Hugger's Avatar
    Join Date
    Jul 2003
    Location
    Banks of the River Styx
    Posts
    902
    What happens if the double \r\n is split between recv() calls?

    Also, chunked encoding... when I finally got my first HTTP thing working, chunked encoding bit me in the butt. I now use libcurl... >.>

    It's also been my experience that some servers send a "Continue" (namely IIS) before sending the actual data. (I think if the headers get separated on their way there...) Again, libcurl ftw.
    long time; /* know C? */
    Unprecedented performance: Nothing ever ran this slow before.
    Any sufficiently advanced bug is indistinguishable from a feature.
    Real Programmers confuse Halloween and Christmas, because dec 25 == oct 31.
    The best way to accelerate an IBM is at 9.8 m/s/s.
    recursion (re - cur' - zhun) n. 1. (see recursion)

  9. #9
    Registered User
    Join Date
    Apr 2007
    Location
    Sydney, Australia
    Posts
    217
    If you look at mine, i have a function called recvLine(). This function goes into a loop and receives 1 byte at a time in the recv() function until it reaches a '\n' or until it has reached the buffer size limit. Once you have called this function, trim the end of the buffer (to remove the \n, if there is one) and if the strlen() is 0 than its a blank line....end of the header.

    Code:
    int recvLine(int pSockId, char* pOut, int pOutSize)
    {
        int received = 0;
        char letter;
        memset(pOut, 0, pOutSize);
        for(; received < pOutSize-1; received++)
        {
            if(recv(pSockId, (char*)&letter, 1, 0) > 0)
            {
                pOut[received] = letter;
                if(letter == '\n')
                    break;
            } else break;
        }
        return received;
    }
    Code:
        while((length = recvLine(sockId, buffer, sizeof(buffer))) > 0)
        {
            //trim line
            for(int i = length-1; i >= 0; i--)
            {
                if(buffer[i] < ' ')
                    buffer[i] = 0;
                else break;
            }
    
            //if end of header
            if(!strlen(buffer))
                break;
    	
            //Process header
        }
        //receive file data
    Last edited by 39ster; 09-05-2007 at 01:44 AM.

  10. #10
    Registered User Tommo's Avatar
    Join Date
    Jun 2007
    Location
    Scotland
    Posts
    101
    You mean, what if the header doesn't all come out in the first recv call? Yeah you're right, that is something I need to look at. As for libcurl, I wanted to write my own rather than use someone elses library.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Formatting a text file...
    By dagorsul in forum C Programming
    Replies: 12
    Last Post: 05-02-2008, 03:53 AM
  2. Need Help Fixing My C Program. Deals with File I/O
    By Matus in forum C Programming
    Replies: 7
    Last Post: 04-29-2008, 07:51 PM
  3. help with text input
    By Alphawaves in forum C Programming
    Replies: 8
    Last Post: 04-08-2007, 04:54 PM
  4. Encryption program
    By zeiffelz in forum C Programming
    Replies: 1
    Last Post: 06-15-2005, 03:39 AM
  5. simulate Grep command in Unix using C
    By laxmi in forum C Programming
    Replies: 6
    Last Post: 05-10-2002, 04:10 PM