What's the fastest way to write a file?

This is a discussion on What's the fastest way to write a file? within the C++ Programming forums, part of the General Programming Boards category; what's the fastest way to write data into a file? cout or fprintf? Is there any even faster way to ...

  1. #1
    Registered User
    Join Date
    Jun 2007
    Posts
    12

    Arrow What's the fastest way to write a file?

    what's the fastest way to write data into a file?
    cout or fprintf?

    Is there any even faster way to do this?

  2. #2
    Registered User
    Join Date
    May 2007
    Posts
    88
    The fastest I/O stuff is pretty much always OS-specific. For example, write() on Unix-based systems.

  3. #3
    Registered User
    Join Date
    Jun 2007
    Posts
    12
    I am using WinXP

  4. #4
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,048
    I imagine that functions which do not do any type parsing would be faster than those which do, such as fprintf(). So fputs() might be a good idea. Though cout.operator<<(const char *) might be just as fast.

    Those are just the standard C and C++ functions. I haven't considered the Windows-specific ones.

    How important is this speed to your program? Are you willing to sacrifice portablity for a few milliseconds' execution speed? Why do you need fast I/O?
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  5. #5
    Deathray Engineer MacGyver's Avatar
    Join Date
    Mar 2007
    Posts
    3,211
    As already said, the fastest way is to use system calls.

    WriteFile() for Windows. write() for Unix/Linux.

  6. #6
    Registered User
    Join Date
    Jun 2007
    Posts
    12
    Quote Originally Posted by dwks View Post
    How important is this speed to your program? Are you willing to sacrifice portablity for a few milliseconds' execution speed? Why do you need fast I/O?
    Well, I used fprintf in my current program. The program opens a file frequently, write something into it, and then close it. I just wanted to see if I can improve speed by using another output function.

    If there is not such a good function, I have to consider about printing the data into a string, and printing the string out when the program ends. Will that save a lot of time?

  7. #7
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,048
    If there is not such a good function, I have to consider about printing the data into a string, and printing the string out when the program ends. Will that save a lot of time?
    It might save some time. Would it be difficult to program?

    It sounds like your program might be taking a lot of time to open files over and over again. You might be able to save lots of time by only opening the file once. Assuming that you're only writing to one file.

    How much data are you writing, and how long does your program take to run?
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  8. #8
    Registered User
    Join Date
    Jun 2007
    Posts
    12
    Sorry for late response, I have just been too absorbed in programming.

    I need to run the program for thousands of times, and one run may takes around 40 mins. The output file size, as I could estimate now, is roughly 20~200k.

    so any suggestions for optimization?

  9. #9
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,048
    Sorry for late response, I have just been too absorbed in programming.
    It happens to all of us.

    I need to run the program for thousands of times, and one run may takes around 40 mins. The output file size, as I could estimate now, is roughly 20~200k.

    so any suggestions for optimization?
    Wow, yes, then definitely go for whatever speed increases you can get away with.

    But one run, generating a 20-200KB file, takes 40 minutes? You must either have a very slow computer, or the program must do a lot of extra work. How much of that time is actually spent on file I/O?

    I suggest using a profiler to see which portions of your code are taking excessive amounts of time. With GCC, you can use gprof.
    Code:
    $ gcc -pg hello.c -o hello.exe
    $ ./hello.exe
    $ gprof hello.exe > gprof_output1
    You should be able to figure out the output from gprof. If not, do some searching or post it here.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  10. #10
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,308
    Quote Originally Posted by pingpangpang View Post
    Sorry for late response, I have just been too absorbed in programming.

    I need to run the program for thousands of times, and one run may takes around 40 mins. The output file size, as I could estimate now, is roughly 20~200k.

    so any suggestions for optimization?
    Taking that long has nothing to do with the method used to write to file.
    The reality is, you have no idea why it takes so long and are just picking one thing and asking how to speed up that one thing. That wont get you anywhere.

    You would have to make available a lot of the code that is so slow. Right now we can't do squat to help you.
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

  11. #11
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,854
    > The output file size, as I could estimate now, is roughly 20~200k.
    If it takes 40 minutes to produce 200K of output, then I/O is NOT your problem.

    Try writing a program which just generates 200K of output using cout or printf, and see how long it takes. You'll barely be able to get your finger off the return key before it's done.

    Which compiler are you using?

    If it's gcc based, then look into something called 'gprof', which is a profiler.
    This will tell you exactly where the code is spending the most time, and those are the areas you need to focus on.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  12. #12
    Tropical Coder Darryl's Avatar
    Join Date
    Mar 2005
    Location
    Cayman Islands
    Posts
    503
    I need to run the program for thousands of times, and one run may takes around 40 mins.
    hmm thousands = at least 2

    2000 *40min / 60min/hr / 24h/day = wtf! 55.55 days, hope you got a snickers :-)

  13. #13
    Malum in se abachler's Avatar
    Join Date
    Apr 2007
    Posts
    3,189
    Quote Originally Posted by Darryl View Post
    2000 *40min / 60min/hr / 24h/day = wtf! 55.55 days, hope you got a snickers :-)
    This sounds like a job for [big booming voice] Super Multithreadalicious Man [/big booming voice] and probably his faithful sidekick Beowulf Boy.
    Until you can build a working general purpose reprogrammable computer out of basic components from radio shack, you are not fit to call yourself a programmer in my presence. This is cwhizard, signing off.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. A development process
    By Noir in forum C Programming
    Replies: 37
    Last Post: 07-10-2011, 11:39 PM
  2. Basic text file encoder
    By Abda92 in forum C Programming
    Replies: 15
    Last Post: 05-22-2007, 02:19 PM
  3. help with text input
    By Alphawaves in forum C Programming
    Replies: 8
    Last Post: 04-08-2007, 05:54 PM
  4. Encryption program
    By zeiffelz in forum C Programming
    Replies: 1
    Last Post: 06-15-2005, 04:39 AM
  5. archive format
    By Nor in forum A Brief History of Cprogramming.com
    Replies: 0
    Last Post: 08-05-2003, 08:01 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21