Thread: Binary Reader Buffer size

  1. #1
    Registered User
    Join Date
    Sep 2011
    Posts
    5

    Binary Reader Buffer size

    Hello,

    I woud like to read a very big binary file in C#. Say, 256 Mega Bytes of data. Actually I should read from one big 256Mbytes file and append to another file.
    How do I append to my original file?
    can i use some thing like this?

    Code:
    public static byte[] ReadFully (Stream stream)
    {
    byte[] buffer = new byte[268435456];
    using (MemoryStream ms = new MemoryStream())
    {
    while (true)
    {
    int read = stream.Read (buffer, 0, buffer.Length);
    if (read <= 0)
    return ms.ToArray();
    ms.Write (buffer, 0, read);
    }
    }
    }
    any help, suggestions is appreciated.


    Thanks,
    Last edited by krischad; 09-28-2011 at 12:17 PM.

  2. #2
    Registered User
    Join Date
    Mar 2011
    Posts
    41
    As long as you have memory for it, you sure can. Just remember that the actual read will take a bit as I believe Streams default to 4k buffers (so it will have to do multiple disk reads to get the entire thing).

  3. #3
    Registered User
    Join Date
    Sep 2011
    Posts
    5
    @Momerath: Thanks,
    If I'm reading from a file and appending to another file, is there any simple way to do it? How do I append what I read to another file?

  4. #4
    Confused Magos's Avatar
    Join Date
    Sep 2001
    Location
    Sweden
    Posts
    3,145
    If you're always reading from a file the simplest way would be to use:
    Code:
    System.IO.File.ReadAllBytes("File.ext");
    If not I'd rather use the following than implementing my own low-level reader:
    Code:
    public byte[] ReadFully(System.IO.Stream Stream)
    {
    	using(var MemoryStream = new System.IO.MemoryStream())
    	{
    		Stream.CopyTo(MemoryStream);
    		return MemoryStream.ToArray();
    	}
    }
    (CopyTo has an overload where you can specify the buffer size, for )
    MagosX.com

    Give a man a fish and you feed him for a day.
    Teach a man to fish and you feed him for a lifetime.

  5. #5
    Confused Magos's Avatar
    Join Date
    Sep 2001
    Location
    Sweden
    Posts
    3,145
    Reading your post in more detail I see the actual question was appending.
    The simplest way of this, assuming it's a textfile (most common when appening I guess), is this:
    Code:
    System.IO.File.AppendAllText("AppendedFile.txt", System.IO.File.ReadAllText("FileToAppend.txt"));
    MagosX.com

    Give a man a fish and you feed him for a day.
    Teach a man to fish and you feed him for a lifetime.

  6. #6
    Registered User
    Join Date
    Sep 2011
    Posts
    5
    @Magos: Thanks.
    I liked using this,
    Code:
    System.IO.File.ReadAllBytes("File.ext")
    Can it read 256Mega Bytes?

    How do I append it to another file? can i do it same tiem i read?

  7. #7
    Registered User
    Join Date
    Sep 2011
    Posts
    5
    Its a .bin file, binary file. I shud use appendallbytes right?

    Thanks a ton for the info

  8. #8
    Registered User
    Join Date
    Sep 2011
    Posts
    5
    I tried this
    Code:
     System.IO.
    Code:
    File.AppendAllBytes("tom.mpa", System.IO.File.ReadAllBytes("xyz.bin")); 
    
    

    error CS0117: 'System.IO.File' does not contain a
    definition for 'ReadAllBytes'
    error CS0117: 'System.IO.File' does not contain a
    definition for 'AppendAllBytes'

  9. #9
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    Hope is the first step on the road to disappointment.

  10. #10
    Registered User
    Join Date
    Mar 2009
    Location
    england
    Posts
    209
    Code:
    public bool AppendFile(String file_to_read, String file_to_append)
    {
        using (FileStream fs1 = new FileStream(file_to_read, FileMode.Open, FileAccess.Read))
        {
            using (FileStream fs2 = new FileStream(file_to_append, FileMode.Append, FileAccess.ReadWrite))
            {
                byte[] buf = new byte[4096];
                int size = 0;
    
                while ((size = fs1.Read(buf, 0, 4096)) > 0)
                    fs2.Write(buf, 0, size);
            }
        }
    
        return true;
    }
    Here's a way of appending the contents of one file onto the end of another. Performance-wise it might be slower than reading the full files as demonstrated of previous answers - I've not tested it on 200+mb files, however, because it is using file streams it means that your application's memory requirement won't suddenly spike at 200+mb.

    Just pointing out another option available for you.

  11. #11
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    You don't really need to return anything there, since you don't have a false case, where you return false on error. (Just a nit pick.)


    Quzah.
    Hope is the first step on the road to disappointment.

  12. #12
    Registered User
    Join Date
    May 2003
    Posts
    1,619
    Quote Originally Posted by theoobe View Post
    Code:
    public bool AppendFile(String file_to_read, String file_to_append)
    {
        using (FileStream fs1 = new FileStream(file_to_read, FileMode.Open, FileAccess.Read))
        {
            using (FileStream fs2 = new FileStream(file_to_append, FileMode.Append, FileAccess.ReadWrite))
            {
                byte[] buf = new byte[4096];
                int size = 0;
    
                while ((size = fs1.Read(buf, 0, 4096)) > 0)
                    fs2.Write(buf, 0, size);
            }
        }
    
        return true;
    }
    Here's a way of appending the contents of one file onto the end of another. Performance-wise it might be slower than reading the full files as demonstrated of previous answers - I've not tested it on 200+mb files, however, because it is using file streams it means that your application's memory requirement won't suddenly spike at 200+mb.

    Just pointing out another option available for you.
    Or even simpler:

    Code:
    public void AppendFile(String file_to_read, String file_to_append)
    {
        using (FileStream fs1 = new FileStream(file_to_read, FileMode.Open, FileAccess.Read))
        {
            using (FileStream fs2 = new FileStream(file_to_append, FileMode.Append, FileAccess.ReadWrite))
            {
                fs1.CopyTo(fs2);
            }
        }
     }
    You ever try a pink golf ball, Wally? Why, the wind shear on a pink ball alone can take the head clean off a 90 pound midget at 300 yards.

  13. #13
    Confused Magos's Avatar
    Join Date
    Sep 2001
    Location
    Sweden
    Posts
    3,145
    Or, if you hate nesting too much: :P
    Code:
    public void AppendFile(String file_to_read, String file_to_append)
    {
      using(var fs1 = new FileStream(file_to_read, FileMode.Open, FileAccess.Read))
      using(var fs2 = new FileStream(file_to_append, FileMode.Append, FileAccess.ReadWrite))
      {
        fs1.CopyTo(fs2);
      }
    }
    MagosX.com

    Give a man a fish and you feed him for a day.
    Teach a man to fish and you feed him for a lifetime.

  14. #14
    Gawking at stupidity
    Join Date
    Jul 2004
    Location
    Oregon, USA
    Posts
    3,218
    Quote Originally Posted by Magos View Post
    Or, if you hate nesting too much: :P
    Code:
    public void AppendFile(String file_to_read, String file_to_append)
    {
      using(var fs1 = new FileStream(file_to_read, FileMode.Open, FileAccess.Read))
      using(var fs2 = new FileStream(file_to_append, FileMode.Append, FileAccess.ReadWrite))
      {
        fs1.CopyTo(fs2);
      }
    }
    How is that less nesting? Messing with indentation != not nesting. All you did was get rid of braces around a single nested statement, akin to changing:
    Code:
    if(true)
    {
      a = b;
    }
    to...
    Code:
    if(true)
      a = b;
    If you understand what you're doing, you're not learning anything.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Buffer size question.
    By Swerve in forum C++ Programming
    Replies: 6
    Last Post: 11-07-2009, 08:45 AM
  2. size of buffer
    By silentintek in forum C Programming
    Replies: 10
    Last Post: 10-29-2008, 07:21 PM
  3. Get file size when in buffer
    By Siphon in forum C++ Programming
    Replies: 10
    Last Post: 04-10-2008, 10:52 AM
  4. getchar buffer size
    By oncemyway in forum C Programming
    Replies: 3
    Last Post: 08-02-2005, 12:49 AM
  5. Sockets and buffer size
    By Joe Monti in forum Linux Programming
    Replies: 1
    Last Post: 04-04-2003, 09:44 AM

Tags for this Thread