Thread: HTTP File Upload Libcurl

  1. #1
    Registered User
    Join Date
    Mar 2011
    Posts
    216

    HTTP File Upload Libcurl

    Hey,

    My program isn't uploading the file to the directory it is supposed to. Here's the code.
    Code:
    #include <stdio.h>
    
    #include <curl/curl.h> 
    #include <curl/types.h> 
    #include <curl/easy.h> 
    
    #include <sys/stat.h>
    #include <fcntl.h>
    
    size_t read_data (char *bufptr, size_t size, size_t nitems, void *userp) 
    {
           size_t read;
           read = fread(bufptr, size, nitems, userp);
           return read;
    }
     
    int main(void)
    {
      CURL *curl;
      CURLcode res;
      struct stat file_info;
      double speed_upload, total_time;
      FILE *fd;
      
      char *url = "http://iss.netii.net/projects/message";
      char outfilename[FILENAME_MAX] = "feed.zip";
     
      fd = fopen("feed.txt", "rb"); /* open file to upload */ 
      if(!fd) 
      {
     
        return 1; /* can't continue */ 
      }
     
      /* to get the file size */ 
      if(fstat(fileno(fd), &file_info) != 0) 
      {
     
        return 1; /* can't continue */ 
      }
     
      curl = curl_easy_init();
      if(curl) 
      {
        /* upload to this place */ 
        curl_easy_setopt(curl, CURLOPT_URL, url);
     
        /* tell it to "upload" to the URL */ 
        curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
        
        curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_data); 
     
        /* set where to read from (on Windows you need to use READFUNCTION too) */ 
        curl_easy_setopt(curl, CURLOPT_READDATA, fd);
     
        /* and give the size of the upload (optional) */ 
        curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t)file_info.st_size);
     
        /* enable verbose for easier tracing */ 
        curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
     
        res = curl_easy_perform(curl);
     
        /* now extract transfer info */ 
        curl_easy_getinfo(curl, CURLINFO_SPEED_UPLOAD, &speed_upload);
        curl_easy_getinfo(curl, CURLINFO_TOTAL_TIME, &total_time);
     
        printf("Speed: %.3f bytes/sec during %.3f seconds\n", speed_upload, total_time);
     
        /* always cleanup */ 
        curl_easy_cleanup(curl);
      }
      fflush(stdin);
      getchar();
      return 0;
    }

  2. #2
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    First, fflush(stdin) is wrong, and results in undefined behavior. Read the FAQ: Cprogramming.com FAQ > Why fflush(stdin) is wrong.

    Second, you didn't say anything about why or how it doesn't work. Clear, specific questions usually get clear, specific answers. It could be a network issue, authentication/permissions, bad options, etc. You need to put in some diagnostic code other than just returning 1 in a handful of cases. Use perror for the standard C functions (fopen and fstat), and check the libcurl site for API references and error handling for the libcurl functions.

  3. #3
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    Here is the log for it. IP address is blocked, but you get the gist.

    HTTP File Upload Libcurl-log-jpg

    It seems everything is fine, however it says there is not upload speed...

  4. #4
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    I found this StackOverflow article that suggests your upload may not yet have been complete, hence the option not working. It seems supported by this thread from the libcurl mailing list: Curl: Upload speed and remaining time. You may also want to search the libcurl mailing lists for other info. You can always try sleeping for a few seconds between your perform and getinfo calls, as a quick hack to see if it needs time to complete.

  5. #5
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    So I looked into the documention, extensively, and searched on google, but yielded no results. I don't know what you mean by it's not complete, you mean like I'm missing a step?

    I tried sleeping for a few seconds, before and after the perform calls, but it doesn't seem to make a difference. (except it pauses, of course)

  6. #6
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    New information, after doing some more searching, I found some documentation where it said the url had to include the output file name, so I updated my source, and now it's working. However, it seems "PUTS" is not allowed..? Here's the log.

    HTTP File Upload Libcurl-log2-jpg

    Could someone help decipher this?

    EDIT: So, the method PUT is not allowed via HTTP, so does this mean I have to rewrite my code, and do it through FTP?
    Last edited by binks; 07-20-2011 at 08:03 AM.

  7. #7
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    Doh! I actually noticed the file name thing and thought I mentioned it in one of my posts, but I guess not. Sorry. I actually caught it because I compiled with my warnings cranked up to the max, so I got a message like "outfilename declared but never used". Hint: always compile with your warnings cranked up to the max, and make sure you resolve them all.

    As for why PUT doesn't work, it would probably depend on the web server you're using. If you administer it, you should be able to change it however you want. If it's a hosted site, the admins might have disallowed PUT requests. There should also be some information on how to upload content to the web site (as in how you would do. If they only list FTP as a way to get your content up there, then you'll have to change your code. You could try emailing support or admin for help too.

  8. #8
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    How would it work through FTP? because the site is HTTP.

    libcurl - ftpupload.c

    This is the libcurl example of ftp uploads, however, I don't quite get how it makes it through into the FTP account, without the information.

    Code:
    /***************************************************************************
     *                                  _   _ ____  _
     *  Project                     ___| | | |  _ \| |
     *                             / __| | | | |_) | |
     *                            | (__| |_| |  _ <| |___
     *                             \___|\___/|_| \_\_____|
     *
     * Copyright (C) 1998 - 2011, Daniel Stenberg, <[email protected]>, et al.
     *
     * This software is licensed as described in the file COPYING, which
     * you should have received as part of this distribution. The terms
     * are also available at http://curl.haxx.se/docs/copyright.html.
     *
     * You may opt to use, copy, modify, merge, publish, distribute and/or sell
     * copies of the Software, and permit persons to whom the Software is
     * furnished to do so, under the terms of the COPYING file.
     *
     * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
     * KIND, either express or implied.
     *
     ***************************************************************************/ 
    #include <stdio.h>
    #include <string.h>
     
    #include <curl/curl.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    #include <fcntl.h>
    #include <errno.h>
    #ifdef WIN32
    #include <io.h>
    #else
    #include <unistd.h>
    #endif
     
    /*
     * This example shows an FTP upload, with a rename of the file just after
     * a successful upload.
     *
     * Example based on source code provided by Erick Nuwendam. Thanks!
     */ 
     
    #define LOCAL_FILE      "/tmp/uploadthis.txt"
    #define UPLOAD_FILE_AS  "while-uploading.txt"
    #define REMOTE_URL      "ftp://example.com/"  UPLOAD_FILE_AS
    #define RENAME_FILE_TO  "renamed-and-fine.txt"
     
    /* NOTE: if you want this example to work on Windows with libcurl as a
       DLL, you MUST also provide a read callback with CURLOPT_READFUNCTION.
       Failing to do so will give you a crash since a DLL may not use the
       variable's memory when passed in to it from an app like this. */ 
    static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
    {
      /* in real-world cases, this would probably get this data differently
         as this fread() stuff is exactly what the library already would do
         by default internally */ 
      size_t retcode = fread(ptr, size, nmemb, stream);
     
      fprintf(stderr, "*** We read %d bytes from file\n", retcode);
      return retcode;
    }
     
    int main(void)
    {
      CURL *curl;
      CURLcode res;
      FILE *hd_src;
      struct stat file_info;
      curl_off_t fsize;
     
      struct curl_slist *headerlist=NULL;
      static const char buf_1 [] = "RNFR " UPLOAD_FILE_AS;
      static const char buf_2 [] = "RNTO " RENAME_FILE_TO;
     
      /* get the file size of the local file */ 
      if(stat(LOCAL_FILE, &file_info)) {
        printf("Couldnt open '%s': %s\n", LOCAL_FILE, strerror(errno));
        return 1;
      }
      fsize = (curl_off_t)file_info.st_size;
     
      printf("Local file size: %" CURL_FORMAT_CURL_OFF_T " bytes.\n", fsize);
     
      /* get a FILE * of the same file */ 
      hd_src = fopen(LOCAL_FILE, "rb");
     
      /* In windows, this will init the winsock stuff */ 
      curl_global_init(CURL_GLOBAL_ALL);
     
      /* get a curl handle */ 
      curl = curl_easy_init();
      if(curl) {
        /* build a list of commands to pass to libcurl */ 
        headerlist = curl_slist_append(headerlist, buf_1);
        headerlist = curl_slist_append(headerlist, buf_2);
     
        /* we want to use our own read function */ 
        curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
     
        /* enable uploading */ 
        curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
     
        /* specify target */ 
        curl_easy_setopt(curl,CURLOPT_URL, REMOTE_URL);
     
        /* pass in that last of FTP commands to run after the transfer */ 
        curl_easy_setopt(curl, CURLOPT_POSTQUOTE, headerlist);
     
        /* now specify which file to upload */ 
        curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);
     
        /* Set the size of the file to upload (optional).  If you give a *_LARGE
           option you MUST make sure that the type of the passed-in argument is a
           curl_off_t. If you use CURLOPT_INFILESIZE (without _LARGE) you must
           make sure that to pass in a type 'long' argument. */ 
        curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
                         (curl_off_t)fsize);
     
        /* Now run off and do what you've been told! */ 
        res = curl_easy_perform(curl);
     
        /* clean up the FTP commands list */ 
        curl_slist_free_all (headerlist);
     
        /* always cleanup */ 
        curl_easy_cleanup(curl);
      }
      fclose(hd_src); /* close the local file */ 
     
      curl_global_cleanup();
      return 0;
    }
    It says on the example that the url is ftp://example.com, but this is an HTTP site..?

  9. #9
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    I have made the necessary modifications to apply to my scenerio, here's the code followed by the output.

    Code:
    #include <stdio.h>
    #include <string.h>
     
    #include <curl/curl.h>
    #include <curl/types.h> 
    #include <curl/easy.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    #include <fcntl.h>
    #include <errno.h>
    #ifdef WIN32
    #include <io.h>
    #else
    #include <unistd.h>
    #endif
     
    /*
     * This example shows an FTP upload, with a rename of the file just after
     * a successful upload.
     *
     * Example based on source code provided by Erick Nuwendam. Thanks!
     */ 
     
    #define LOCAL_FILE      "feed.txt"
    #define RENAME_FILE_TO  "feed.zip"
     
    /* NOTE: if you want this example to work on Windows with libcurl as a
       DLL, you MUST also provide a read callback with CURLOPT_READFUNCTION.
       Failing to do so will give you a crash since a DLL may not use the
       variable's memory when passed in to it from an app like this. */ 
    static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
    {
      /* in real-world cases, this would probably get this data differently
         as this fread() stuff is exactly what the library already would do
         by default internally */ 
      size_t retcode = fread(ptr, size, nmemb, stream);
     
      printf("*** We read %d bytes from file\n", retcode);
      return retcode;
    }
     
    int main(void)
    {
      CURL *curl;
      CURLcode res;
      FILE *hd_src;
      struct stat file_info;
      curl_off_t fsize;
      
      char *REMOTE_URL = "http://iss.netii.net/projects/message/";
     
      struct curl_slist *headerlist=NULL;
      static const char buf_2 [] = "RNTO " RENAME_FILE_TO;
     
      /* get the file size of the local file */ 
      if(stat(LOCAL_FILE, &file_info)) {
        printf("Couldnt open '%s': %s\n", LOCAL_FILE, strerror(errno));
        return 1;
      }
      fsize = (curl_off_t)file_info.st_size;
     
      printf("Local file size: %" CURL_FORMAT_CURL_OFF_T " bytes.\n", fsize);
     
      /* get a FILE * of the same file */ 
      hd_src = fopen(LOCAL_FILE, "rb");
     
      /* In windows, this will init the winsock stuff */ 
      curl_global_init(CURL_GLOBAL_ALL);
     
      /* get a curl handle */ 
      curl = curl_easy_init();
      if(curl) {
        /* build a list of commands to pass to libcurl */ 
        headerlist = curl_slist_append(headerlist, buf_2);
     
        /* we want to use our own read function */ 
        curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
     
        /* enable uploading */ 
        curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
     
        /* specify target */ 
        curl_easy_setopt(curl,CURLOPT_URL, REMOTE_URL);
     
        /* pass in that last of FTP commands to run after the transfer */ 
        curl_easy_setopt(curl, CURLOPT_POSTQUOTE, headerlist);
     
        /* now specify which file to upload */ 
        curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);
     
        /* Set the size of the file to upload (optional).  If you give a *_LARGE
           option you MUST make sure that the type of the passed-in argument is a
           curl_off_t. If you use CURLOPT_INFILESIZE (without _LARGE) you must
           make sure that to pass in a type 'long' argument. */ 
        curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
                         (curl_off_t)fsize);
     
        /* Now run off and do what you've been told! */ 
        res = curl_easy_perform(curl);
     
        /* clean up the FTP commands list */ 
        curl_slist_free_all (headerlist);
     
        /* always cleanup */ 
        curl_easy_cleanup(curl);
      }
      fclose(hd_src); /* close the local file */ 
      getchar();
      curl_global_cleanup();
      return 0;
    }
    HTTP File Upload Libcurl-log3-jpg

    Another PUT error...

  10. #10
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    Quote Originally Posted by binks View Post
    How would it work through FTP? because the site is HTTP.
    This is a bit simplified, but...

    HTTP is just a protocol. Think of it like a language. There are certain words and a certain grammar that must be followed. Somebody can speak English and Spanish, right? Similarly a computer can speak HTTP and FTP if it has the right programs. Also, the computer must figure out what language you want to talk in. Unlike people, who can just recognize a language by hearing a few words, the computer needs a well defined method for determining this. That is handled by TCP ports. There are conventions for the most well-known and widely used ports. HTTP is 80, FTP is 21 (it also uses 20 for the file transfer itself). The server listens on those ports. If somebody talks on port 21, the server expects to talk FTP, and if it doesn't hear FTP, it won't talk to that computer. You can configure the computer to listen on non-standard ports, but you have to tell the program that is trying to talk to it to use that non-standard port as well.


    The programs that listen for these conversations may be referred to as "servers" since they serve up data or files, a "daemon" (origin here), or a "service", presumably since it performs some service. Those programs sit in the background and listen on their assigned port. When somebody starts talking, they start handling requests and talking back. Usually those programs are separate (Apache and ftpd on Linux), but sometimes, they are combined, like Microsoft's IIS. IIS can speak multiple languages and listen on multiple ports if configured to do so.


    Often, in a "get a free website here" approach, each user is given a directory on the server that acts as the root of their web page. The HTTP daemon will know that if you ask for jon.example.com, to look in Jon's web root, or maybe it's configure to use something like IANA &mdash; Example domains. But that HTTP daemon may also be configured to only respond to some HTTP commands, like the GET command for fetching files (how a web browser gets pages). The PUT command may be turned off. That's all independent of the FTP daemon, that may allow sending and receiving of files. That FTP command, when you log in as jon, would be configured to only allow you to send/receive data from jon's directory. Hope that clears things up a bit on that front.


    Quote Originally Posted by binks View Post
    Another PUT error...
    Before you spend any more time programming this, figure out how to do it by hand. You will likely need a user name and password to upload here. Then, try it using an FTP client (I recommand a command line one, Windows and Linux both come with one by default). Google for examples on FTP from command line.

    Assuming you can connect via FTP, your code is mostly there, but you didn't follow the example closely enough. Your URL still specifies HTTP:
    Code:
    char *REMOTE_URL = "http://iss.netii.net/projects/message/";
    That red text should be "ftp", so cURL knows what language to try and speak. Also, you are only sending a RNTO command (rename to), but you don't specify renaming what to what. You will also need a RNFR like the example. Study it a little more closely, and try to mimic it more exactly. Lastly, you may need to set authentication info (user/password) so cURL knows how to log you in to the FTP server.

  11. #11
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    Thanks for clearing some things up, I truly appreciate it.

    How do you set authentication info through libcurl? Like so?
    Code:
    curl_easy_setopt(curl, CURLOPT_USERNAME, "username");
    curl_easy_setopt(curl, CURLOPT_PASSWORD, "password");

  12. #12
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    I don't know, I haven't used curl in years. Are CURLOPT_USERNAME and CURLOPT_PASSWORD valid curl options? I provided the link to the libcurl site which has examples and API docs covering all the functions and options. You'll have to do some digging and experimenting yourself.

    Do you know how to upload content manually (i.e. not using this program)? Can you put up an index.html file that has some random text like "This is a test"? When you figure out how to do that, then you can make your curl program mimic the manual process.

  13. #13
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    libcurl - curl_easy_setopt()

    Under Names and Password Options, it states these
    CURLOPT_USERNAME

    Pass a char * as parameter, which should be pointing to the zero terminated user name to use for the transfer.

    CURLOPT_USERNAME sets the user name to be used in protocol authentication. You should not use this option together with the (older) CURLOPT_USERPWD option.

    In order to specify the password to be used in conjunction with the user name use the CURLOPT_PASSWORD option. (Added in 7.19.1)

    CURLOPT_PASSWORD

    Pass a char * as parameter, which should be pointing to the zero terminated password to use for the transfer.

    The CURLOPT_PASSWORD option should be used in conjunction with the CURLOPT_USERNAME option. (Added in 7.19.1)
    Do you know how to upload content manually (i.e. not using this program)? Can you put up an index.html file that has some random text like "This is a test"? When you figure out how to do that, then you can make your curl program mimic the manual process.
    Not with the command line, but through an ftp manager, yes.

  14. #14
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    Well, pay attention to all the options you have set or unset in your FTP manager. Look at things like port, active/passive mode, etc. Your FTP manager may automatically try both active and passive mode, while curl probably doesn't. Go through all the general options and FTP options and make sure everything is just write. I think now you're stuck because you don't know the connection details of your server, not because there's some problem in your use of libcurl or your C code in general.

  15. #15
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    I don't know where to find that information.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 2
    Last Post: 03-30-2011, 10:39 AM
  2. libcurl to file
    By starternewb in forum C Programming
    Replies: 1
    Last Post: 03-15-2011, 07:45 AM
  3. file upload form in c++
    By Yarin in forum A Brief History of Cprogramming.com
    Replies: 1
    Last Post: 10-17-2008, 01:13 PM
  4. File Upload with libcurl
    By Tonto in forum Networking/Device Communication
    Replies: 4
    Last Post: 03-15-2006, 06:11 PM
  5. Anyone know a good file upload routine.
    By bjdea1 in forum C Programming
    Replies: 2
    Last Post: 12-08-2001, 12:04 PM

Tags for this Thread