how to copy binary files using Unix API's

Hello everyone,
As a part of a data backup project, I need to use Unix API's for performing file copy from source to destination.I have used the API's open(),read(),write() and close() for performing this.It works fine for regular files but not for binary files(like audio or video files).The newly created binary file gets partially corrupted(I got to know that the file is corrupted when VLC media player said : "the AVI file is broken".How should I use these API's for copying binary files?Also I dont want to copy the files byte by byte for efficiency reasons.I am using a buffer of size 1MB.

Thank You,
Rohan

Unless I'm missing something not-so-obvious, if you are opening both input and output in binary mode, and copying character by character, nothing should change, therefore nothing should break.

Todd

For the sake of efficiency I don't want to do a byte by byte copy.It woud be better to do a block read and block write.

The sendfile() call can be used to copy bytes directly from one descriptor to another, avoiding all user-space buffering. The only caveat is needing to know how many bytes will be copied up front. If you can seek on the input, there's no problem:

Code:

---

int input, output;
off_t size;
off_t offset;

input = open(input_file, O_RDONLY);
output = open(output_file, O_WRONLY | O_CREAT, 0666);
size = lseek(input, 0, SEEK_END);
offset = 0;
sendfile(output, input, &offset, (size_t)size);
close(output);
close(input);

---

This is the most efficient way to copy bytes from one descriptor to another on UNIX^H^H^H^HLinux, because the data never enters userspace. (Oops -- thought sendfile() was POSIX for some reason. It's widely available, though)

Quote:

---

Originally Posted by rohan_ak1

For the sake of efficiency I don't want to do a byte by byte copy.It woud be better to do a block read and block write.

---

I apologize - you didn't say you were copying byte by byte, and I implied you were. tit-for-tat - if you are reading a block and writing the same sized block, in binary, no alteration should occur.


Todd

Are you writing the size of the buffer, or the answer returned by read ?

while ( (n=read(buff,BUFSIZ)) ) write( buff, n);
kinda thing

This is the code I have written for copying files from source to destination:

Code:

---

fps=open(source,O_RDONLY);
                //printf("dest = %s\n\n",dest);
                fpd=open(dest,O_WRONLY|O_CREAT,0777);
                if(fps==-1)
                {
                        printf("source = %s\n",source);
                        printf("error : fps\n");
                        break;
                }
                if(fpd==-1)
                        printf("error : fpd\n");
                fstat(fps,&buf);
                bzero(buffer,BSIZE);
                filesize=buf.st_size;
                while(filesize)
                {
                        if(filesize>BSIZE-1)
                        {
                                filesize=filesize-BSIZE-1;
                                read(fps,buffer,BSIZE-1);
                                write(fpd,buffer,BSIZE-1);
                                bzero(buffer,BSIZE);
                        }
                        else
                        {
                                read(fps,buffer,filesize);
                                write(fpd,buffer,filesize);
                                filesize=0;
                                bzero(buffer,BSIZE);
                        }
                }

---

This works fine for text files but problems for binary files.
Can anyone suggest changes??

I don't see anything directly wrong with it, but it can be made simpler, less repeats of code by adding a variable for the block-size:

Code:

---

          int blocksize;
          ...
                while(filesize)
                {
                        if(filesize>BSIZE)
                        {
                                blocksize = BSIZE;
                        }
                        else
                        {
                                blocksize = filesize;
                        }
                        filesize -= blocksize;
                        read(fps,buffer,blocksize);
                        write(fpd,buffer,blocksize);
                }

---

I removed the bzero - there should be no need for that.
Also no need to use BSIZE-1 [assuming your buffer is defined as char buffer[BSIZE] - otherwise your bzero is wrong and overwrites one byte outside buffer].

--
Mats

In this code,what should I set as the size of the buffer used for read/write.Currently am using 1MB buffer size.This code works well for text files but not for binary files.Using this to copy audio/video files makes the new copy broken(VLC player shows the video file to be broken).How should I make it work for binary files????

Quote:

---

Originally Posted by rohan_ak1

In this code,what should I set as the size of the buffer used for read/write.Currently am using 1MB buffer size.Also can I use mmap and memcpy to improve performance?If so,how can I use them for large files.

---

The buffer size of 1MB should be "reasonable". I don't know what give the best performance on any given system - never mind a generic solution that gives good average performance on a wide range of systems. It's safe to assume that it should be smaller than large disk-caches (they are often several megabytes), and must be larger than 4KB to make it reasonably efficient for the overhead of the kernel mapping the page(s) that the data is in, etc. Something in the order of 64KB to a few MB would be the range I'd "try out" if given a task of finding the best one.

The biggest factor in performance would be the actual reading of the data, and since you are reading sequentially, mmap/memcpy shouldn't make any difference - since that's essentially the same as what you're already doing, using the read (which memcpy's the data to you) and fwrite() which memcpy's the data into the kernel to be written. mmap() is much better if you are trying to read/write in many different locations (scarsely) in a large file.

Not calling bzero would have some impact.

--
Mats

I have now set the buffer size to 1MB and removed the unwanted bzero statements...but what should I do with the problem of improper copying of binary files ( they are broken).

Quote:

---

Originally Posted by rohan_ak1

I have now set the buffer size to 1MB and removed the unwanted bzero statements...but what should I do with the problem of improper copying of binary files ( they are broken).

---

Well, I would figure that out using something like "od" to see what the binary file looks like in an a more human readable form [Note: I didn't say easy readable!], and then use diff to tell see the difference.

Are the files the same size? Where does the differences start?

--
Mats

On changing the code to :

Code:

---

while(filesize)
                {
                        if(filesize>BSIZE)
                        {
                                blocksize = BSIZE;
                        }
                        else
                        {
                                blocksize = filesize;
                        }
                        filesize -= blocksize;
                        read(fps,buffer,blocksize);
                        write(fpd,buffer,blocksize);
                }

---

and removing all bzero's I am able to copy binary files perfectly.The destination file is no longer broken.I tried with an AVI file.However,when I try to backup a set of 56 mp3 files of total size 478.1 MB having 3 sub-directories the destination backup directory has all the 56 files and 3 sub-folders.But its size is 478MB.One of the directories is 0.1MB lesser than the original directory.I am copying from a FAT32 filesystem to an ext2 filesystem.

You really should check EACH file's size, not the total size. Different OS's count file-sizes differently.

--
Mats

I have checked file sizes on fedora 8 for both source and destination directories.I checked every file-size and it is the same.But the total file size is 0.1MB less.