Thread: Using splice() to copy 1 file to another

  1. #1
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115

    Using splice() to copy 1 file to another

    According to http://en.wikipedia.org/wiki/Splice_(system_call) the proper way to use splice() to copy one file to another involves clever use of the pipe() system call. (This is one way to get around the requirement that one of the file descriptors given to splice() is a pipe). What I find strange is that the code example on that page has a while loop writing the entire file to the pipe, and then having another loop read the entire file from the pipe. I don't exactly know what operations are going on in the kernel. Perhaps all the data of the file is being written to the pipe, thus filling up as much kernel memory as the file has data. Or, perhaps the pipe buffer just contains a series of references to the appropriate data in the file, just making the pipe buffer an unmanageably large number of references to a file, rather than a ridiculously unmanageable amount of raw data.

    In any case, wouldn't it be better that each splice() call that writes to the pipe be followed by a call that reads from the pipe, rather than packaging each type of splice call to its own while loop?

    Another question I have on the side is, why is the writing splice() call not handling a return value of -1 followed by an errno value of ENOMEM ("Out of memory")--something that seems a very likely event?

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,659
    The example code would appear to bail out when the pipe is full.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    This is the wrong use of splice(). As you've realized, splice() only works with a pipe. You don't want to create a pipe just to copy a file.

    To copy bytes from one open fd to another, use the sendfile() function.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  4. #4
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115
    Then output file descriptor to sendfile() must be a socket.

  5. #5
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Angus View Post
    Then output file descriptor to sendfile() must be a socket.
    I do see what you are talking about on the man page. But that isn't by design. The kernel is apparently buggy, or somebody is too lazy to make it work.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  6. #6
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115
    Quote Originally Posted by brewbuck View Post
    I do see what you are talking about on the man page. But that isn't by design. The kernel is apparently buggy, or somebody is too lazy to make it work.
    http://ilia.ws/archives/13-sendfile-...el-sucks!.html gives some slightly different reasons for that. What about memory-mapping the input and output files then memcpy()ing between them?

  7. #7
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by Angus View Post
    http://ilia.ws/archives/13-sendfile-...el-sucks!.html gives some slightly different reasons for that. What about memory-mapping the input and output files then memcpy()ing between them?
    In previous testing, it was found that simply reading the file with fread (or read) and writing with fwrite (or write) is faster than mmap'ing the file.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  8. #8
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115
    Quote Originally Posted by matsp View Post
    In previous testing, it was found that simply reading the file with fread (or read) and writing with fwrite (or write) is faster than mmap'ing the file.
    What about using the pipe as a buffer for the splice calls, as in the Wikipedia code example?

  9. #9
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by Angus View Post
    What I find strange is that the code example on that page has a while loop writing the entire file to the pipe, and then having another loop read the entire file from the pipe.

    In any case, wouldn't it be better that each splice() call that writes to the pipe be followed by a call that reads from the pipe, rather than packaging each type of splice call to its own while loop?
    Have you actually tried this? Chances are each loop only iterates once anyway.

    Quote Originally Posted by Angus View Post
    Another question I have on the side is, why is the writing splice() call not handling a return value of -1 followed by an errno value of ENOMEM ("Out of memory")--something that seems a very likely event?
    It is:
    Code:
    if (ret < 0)
                    goto pipe;
    pipe:
                    close (filedes [0]);
                    close (filedes [1]);
    out:
                    if (ret < 0)
                            return -errno;
                    return 0;
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Need Help Fixing My C Program. Deals with File I/O
    By Matus in forum C Programming
    Replies: 7
    Last Post: 04-29-2008, 07:51 PM
  2. Game Pointer Trouble?
    By Drahcir in forum C Programming
    Replies: 8
    Last Post: 02-04-2006, 02:53 AM
  3. Encryption program
    By zeiffelz in forum C Programming
    Replies: 1
    Last Post: 06-15-2005, 03:39 AM
  4. Simple File encryption
    By caroundw5h in forum C Programming
    Replies: 2
    Last Post: 10-13-2004, 10:51 PM
  5. System
    By drdroid in forum C++ Programming
    Replies: 3
    Last Post: 06-28-2002, 10:12 PM

Tags for this Thread