How to write files more faster in C

This is a discussion on How to write files more faster in C within the C Programming forums, part of the General Programming Boards category; Originally Posted by Elysia No, it's bloat. Reading one GB into memory and then writing it with one GB buffer ...

  1. #46
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by Elysia View Post
    No, it's bloat. Reading one GB into memory and then writing it with one GB buffer is slower than reading chunks. Sorry.
    No were are working on the basis he has already has 1 gig to write.

    You won't be able to write a program which writes it faster than mine.

  2. #47
    The larch
    Join Date
    May 2006
    Posts
    3,573
    Mine is an example of programming of the very highest standard possible infact.
    Come on! A utility function that has the file name hard-coded? A utility function that is ready to simply throw away 4 GB of data if some error opening the file happens? Hard-coding the name doesn't protect you here, what if the file exists and is read-only?

    You won't be able to write a program which writes it faster than mine.
    May-be there should be a competition? But rather than just writing 1 GB of meaningless data, lets open a 1 GB file, do some simple math and write the results out?
    Last edited by anon; 01-07-2008 at 05:47 PM.
    I might be wrong.

    Thank you, anon. You sure know how to recognize different types of trees from quite a long way away.
    Quoted more than 1000 times (I hope).

  3. #48
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by Elysia View Post

    NO! Again, you should what little you really know. Get your facts straight - read up on how processors work before you start accusing me!
    A 32-bit pointer simply CANNOT hold any values higher than 4 GB.

    My processor is a sempron with a 40 bit address bus which can address 1TB (1000GB) of memory.
    Get *your* facts straight!!

  4. #49
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,824
    > I was therefore giving an example of how to write the data at 'top speed'.
    Unless you really have an OS which can cope with it, and that much real RAM in your machine, your "fast" data is going to wind up spending most of it's time in the swap file.

    The smallest common multiple of BUFSIZ and the cluster size of your file system would be my guess. As soon as the disk seeks to another cluster, all pretence at "maximum" performance is gone.

    Don't worry too much Elysia, esbo has a history of spew. Personally, I thought he'd gone for good, or at least learnt something.
    How many bits are '1' in an integer variable?
    do {} while (0); ??
    array declared 5 in size but it allows 5 and beyond...
    stop an executing C program and finish executing additonal line of code
    And there are plenty more examples. Nearly every senior member has at some time been in some kind of slanging match trying to hammer a point home.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  5. #50
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by anon View Post
    Come on! A utility function that has the file name hard-coded? A utility function that is ready to simply throw away 4 GB of data if some error opening the file happens? Hard-coding the name doesn't protect you here, what if the file exists and is read-only?



    May-be there should be a competition? But rather than just writing 1 GB of meaningless data, lets open a 1 GB file, do some simple math and write the results out?
    Trouble is I only have 1.25 gig of memory.
    A 100 meg file would be a fairer test.

    And I never said it was a utility function just part of a program I wrote.
    A program which works blindly well, as it happens.
    Reads and writes 6 meg in about a second or less.
    Last edited by esbo; 01-07-2008 at 06:12 PM.

  6. #51
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    I smell some benchmarks.

  7. #52
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Posts
    23,012
    It's indeed faster to read once and write once, IF you have the memory, that is.
    Otherwise it will all end up in the swap file and you're literally reading and writing the file twice.
    Well, anyway, let's ignore esbo and the spew of someone who obviously do not know the spew he/she/it is spamming.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  8. #53
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by Salem View Post
    > I was therefore giving an example of how to write the data at 'top speed'.
    Unless you really have an OS which can cope with it, and that much real RAM in your machine, your "fast" data is going to wind up spending most of it's time in the swap file.

    The smallest common multiple of BUFSIZ and the cluster size of your file system would be my guess. As soon as the disk seeks to another cluster, all pretence at "maximum" performance is gone.

    Don't worry too much Elysia, esbo has a history of spew. Personally, I thought he'd gone for good, or at least learnt something.
    How many bits are '1' in an integer variable?
    do {} while (0); ??
    array declared 5 in size but it allows 5 and beyond...
    stop an executing C program and finish executing additonal line of code
    And there are plenty more examples. Nearly every senior member has at some time been in some kind of slanging match trying to hammer a point home.
    And all been proved wrong no doubt.
    My implimentation leaves it up to the operating system so it should be at
    top speed, however you never know with microsoft.

  9. #54
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    You should not put exit in that function because you might want to use it in another program later. If you didn't, you might as well not use a function and put it all in main.

    You should not use a buffer that large because it takes up a lot of memory that other programs might want to use.

    If you use a large buffer, it should be dynamically allocated so that it doesn't crash the stack.

  10. #55
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by robwhit View Post
    You should not put exit in that function because you might want to use it in another program later. If you didn't, you might as well not use a function and put it all in main.

    You should not use a buffer that large because it takes up a lot of memory that other programs might want to use.

    If you use a large buffer, it should be dynamically allocated so that it doesn't crash the stack.
    Exit is in the function because it is a sensible place to put it.
    If wanted to use it in another program I would also want the exit in there too
    so it would be pointless taking it out to then have to but the same frigging
    code back in again would it not? Which also increase the chance of making
    a coding error rewriting it.

    The buffer is large because I am assuming a large buffer is required.
    The code is originally from a program which requires to access a large amount
    of data randomly at top speed. Thus it must all be in memory.

    How can it be 'dynamically' allocated when it is required all the time?
    How can it crash the stack?
    It has never crashed the stack and it never will.

  11. #56
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    >Exit is in the function because it is a sensible place to put it.
    I disagree. Why do you think it is a sensible place to put it?

    >If wanted to use it in another program I would also want the exit in there too
    so it would be pointless taking it out to then have to but the same frigging
    code back in again would it not? Which also increase the chance of making
    a coding error rewriting it.

    why not just:
    Code:
    function();
    exit();
    ?

    >The buffer is large because I am assuming a large buffer is required.
    Why is a large buffer required? Have you tried a smaller buffer, say one of BUFSIZ type? What was the speed difference? Why was that difference not suitable to your application?

    >The code is originally from a program which requires to access a large amount
    of data randomly at top speed. Thus it must all be in memory.
    Does it all need to be in memory /at one time/? Why?

    >How can it be 'dynamically' allocated when it is required all the time?
    I don't understand the question.

    >How can it crash the stack?
    If it was a local variable like it is supposed to be, then it would be allocated on the stack, and since the stack is usually small in size, it would not be able to contain your variable.

    >It has never crashed the stack and it never will.
    Actually, you are right on that part... I didn't notice that it wasn't actually on the stack. However, consider that the computer does not have the memory for that. Then the program would abort before it started, but what would the return code be?

  12. #57
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by robwhit View Post
    >Exit is in the function because it is a sensible place to put it.
    I disagree. Why do you think it is a sensible place to put it?

    If it cannot open the file it cannot run.
    The only logical thing it can do is to exit, which is exactly what it does.
    It also makes the main program look nicer, uncluttered with error handling, easier to read.

    The same would be true in if it was used in other programs, also you only have
    to write the error function once, not several times - less work, less chance of making
    a coding error.

  13. #58
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by robwhit View Post
    >

    >The buffer is large because I am assuming a large buffer is required.
    Why is a large buffer required? Have you tried a smaller buffer, say one of BUFSIZ type? What was the speed difference? Why was that difference not suitable to your application?

    >The code is originally from a program which requires to access a large amount
    of data randomly at top speed. Thus it must all be in memory.
    Does it all need to be in memory /at one time/? Why?
    It all needs to be in memory because fast access is required.
    It would be very slow if disk I/O needed to take place to access a part of the
    buffer.
    It also makes the code much simpler. The entire buffer is read or written in one line of code.
    No loop required. Hence no loop counter. It is smple and efficient.

  14. #59
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    QUOTE=robwhit;704298]
    >>How can it be 'dynamically' allocated when it is required all the time?
    >I don't understand the question.

    Not sure I understood yours either.

    >>How can it crash the stack?
    >If it was a local variable like it is supposed to be, then it would be allocated on the stack, and since the stack is usually small in size, it would not be able to contain your variable.

    I don't want it to be local because there is no benefit in making global data local.

    >>It has never crashed the stack and it never will.
    >Actually, you are right on that part... I didn't notice that it wasn't actually on the stack. However, consider that the computer does not have the memory for that. Then the program would abort before it started, but what would the return code be?

    I don't create large amounts of storage of the stack hence I don't get exotic memory faults
    so I don't need to worry about such things.
    I avoid creating variables on the stack like the plague is just asking for trouble.
    I will use a few local loop counter and sometime some other variable which I know
    will only ever be used locally but other that that I make stuff global.
    It makes things so much easier. It also makes debugging easier, a program can cope better
    with corrupt data than with a corrupt stack.

  15. #60
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    Quote Originally Posted by esbo View Post
    If it cannot open the file it cannot run.
    The only logical thing it can do is to exit, which is exactly what it does.
    It also makes the main program look nicer, uncluttered with error handling, easier to read.
    But it would make the function cleaner.
    The same would be true in if it was used in other programs, also you only have
    to write the error function once, not several times - less work, less chance of making
    a coding error.
    but suppose that I wanted to make 5 files that had that content? How would I do that? the function would exit every time.

    And what error would the compiler not be able to catch with a snippet that small? Would you really not be able to see it with just a casual glance over the code?

    Quote Originally Posted by esbo View Post
    It all needs to be in memory because fast access is required.
    Fast access from what to what?
    It would be very slow if disk I/O needed to take place to access a part of the
    buffer.
    since the buffer is in memory, the only way that you would need disk I/O to access the buffer is if the buffer was in swap space, which would be more likely with a larger buffer.
    It also makes the code much simpler. The entire buffer is read or written in one line of code.
    No loop required. Hence no loop counter.
    The time it takes to increment a counter is microscopic compared to the time it takes for disk i/o.
    It is smple and efficient.
    I think that if a function does one thing, it's simpler than if it does two.

Page 4 of 6 FirstFirst 123456 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. read / write binary files
    By Dark_Phoenix in forum C++ Programming
    Replies: 5
    Last Post: 06-21-2009, 08:56 AM
  2. Create Copies of Files
    By Kanshu in forum C++ Programming
    Replies: 13
    Last Post: 05-09-2009, 08:53 AM
  3. added start menu crashes game
    By avgprogamerjoe in forum Game Programming
    Replies: 6
    Last Post: 08-29-2007, 02:30 PM
  4. Linking header files, Source files and main program(Accel. C++)
    By Daniel Primed in forum C++ Programming
    Replies: 3
    Last Post: 01-17-2006, 11:46 AM
  5. Using c++ standards
    By subdene in forum C++ Programming
    Replies: 4
    Last Post: 06-06-2002, 10:15 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21