How to write files more faster in C

This is a discussion on How to write files more faster in C within the C Programming forums, part of the General Programming Boards category; Just a thing i want to toy with: Code: #define ONE_GIGA 0x40000000 #define ONE_MEGA 0x100000 #include<stdio.h> #include<stdint.h> #include<stdlib.h> int main() ...

  1. #1
    Registered User
    Join Date
    Dec 2005
    Posts
    167

    How to write files more faster in C

    Just a thing i want to toy with:

    Code:
    #define ONE_GIGA	0x40000000
    #define ONE_MEGA	0x100000
    
    #include<stdio.h>
    #include<stdint.h>
    #include<stdlib.h>
    
    int main()
    {
        FILE *f=fopen("big_file.dat","wt");
        uint64_t i;
        
        if(f==NULL)
        {
                   exit(-1);
        }
        
        for(i=0 ; i < ONE_GIGA ; i++)
        {
    		if(0 == i % ONE_MEGA)
    		{
    			printf("Created %ld.", (long)i / ONE_MEGA);
    			fflush(stdout);
    			printf("\r");
    		}
            fprintf(f,"x");
        }
    	
        printf("...Done\n\n");
        fclose(f);
        
        return 0;
    }
    this application writes one gig of 'x' in big_file.dat. The problem is that it writes them quite slowly. How can I improve my timing ? using fwrite would improve it?
    I just want some ideas on how can I make file write more faster. Using sys calls will help ?

  2. #2
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,048
    fprintf() is slow, because it does formatting. Writing one character at a time is slow in any case.

    Get rid of both problems, say by fwrite()ing 512 bytes at a time or something, and you'll probably get better performance.

    Direct operating-specific system calls might be slightly faster, but you'd lose portability. You could try them.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  3. #3
    Registered User ssharish2005's Avatar
    Join Date
    Sep 2005
    Location
    Cambridge, UK
    Posts
    1,682
    use system call, such as open, read, write. Its more specifc to your OS though. Which OS are you working on.

    ssharish

  4. #4
    Code Goddess Prelude's Avatar
    Join Date
    Sep 2001
    Posts
    9,796
    >Writing one character at a time is slow in any case.
    Except the usual case where all I/O is buffered for performance. If you take a loop using fputc, a loop using fprintf, and a loop using fwrite, the number of actual device writes will probably be comparable, if not identical for a buffered stream.
    My best code is written with the delete key.

  5. #5
    Chinese pâté foxman's Avatar
    Join Date
    Jul 2007
    Location
    Canada
    Posts
    404
    Writing to the hard drive will always be slow, except if you have some really specialized hardware. I don't have the exact number, but in the best case scenario, you won't be able to get more than something like 25 MB/s...

    Like dwks said, try writing more than 1 character at a time (indeed, you should try writing BUFSIZE characters at a time, which is a constant defined in stdio.h). You will at least save from the overhead of numerous functions calls. Also, you should use the fputs function (if you want to stay with text mode). This will speed up the whole thing.

  6. #6
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,048
    Like dwks said, try writing more than 1 character at a time (indeed, you should try writing BUFSIZE characters at a time, which is a constant defined in stdio.h). You will at least save from the overhead of numerous functions calls. Also, you should use the fputs function (if you want to stay with text mode). This will speed up the whole thing.
    BUFSIZ, actually; no E.

    Empirical evidence:
    Code:
    $ time ./write1 
    
    real	0m0.054s
    user	0m0.048s
    sys	0m0.008s
    $ time ./writeX 
    
    real	0m0.012s
    user	0m0.000s
    sys	0m0.012s
    $ time ./write1 
    
    real	0m0.049s
    user	0m0.036s
    sys	0m0.012s
    $ time ./writeX 
    
    real	0m0.019s
    user	0m0.000s
    sys	0m0.016s
    $ time ./write1 
    
    real	0m0.068s
    user	0m0.040s
    sys	0m0.012s
    $ time ./writeX 
    
    real	0m0.012s
    user	0m0.000s
    sys	0m0.012s
    $ cat write1.c
    #include <stdio.h>
    
    int main() {
        FILE *fp = fopen("write1.txt", "w");
        int x;
    
        for(x = 0; x < 1000000; x ++) {
            putc(0, fp);
        }
    
        fclose(fp);
        return 0;
    }
    
    $ cat writeX.c
    #include <stdio.h>
    
    int main() {
        FILE *fp = fopen("writeX.txt", "w");
        char buffer[BUFSIZ] = {0};
        int x;
    
        for(x = 0; x < 1000000 / BUFSIZ; x ++) {
            fwrite(buffer, sizeof buffer, 1, fp);
        }
    
        fclose(fp);
        return 0;
    }
    
    $
    Actually, all that shows is that the binary function fwrite() is faster than the text function putc(). Though, since this is a Linux system, it shouldn't matter.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  7. #7
    Registered User
    Join Date
    Dec 2005
    Posts
    167
    I believe the constant is BUFSIZ. I'm writing on windows an compiling with DevC++.

    I don't understand what you want to say with

    Quote Originally Posted by foxman View Post
    ...
    Also, you should use the fputs function (if you want to stay with text mode). This will speed up the whole thing.
    What syscalls are available with DevC++? Can I use open() write() read() like on unix ?

    Thanx!

  8. #8
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,048
    I believe the constant is BUFSIZ.
    Yes, it is. That's how large the buffer is supposed to be that files are buffered with, so writing data in chunks that size is a good idea.

    How large this is varies. I think it might have a minimum of 512, and that's certainly the smallest I've ever seen it. This computer has 8192, but it's a 64-bit one, so you might expect larger values.

    What syscalls are available with DevC++? Can I use open() write() read() like on unix ?
    Doubtful -- and if you did, they'd just be wrappers around the DOS code.

    You could try the Win API, but I don't think that would be much faster. Try it if you like.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  9. #9
    Chinese pâté foxman's Avatar
    Join Date
    Jul 2007
    Location
    Canada
    Posts
    404
    Code:
    BUFSIZ, actually; no E.
    Yes. My mistake.

    spank: There's basically two way to write a string to a text file (there's more, but let's say there's only two). You can use either fprintf(yourFileP, "%s", yourString) or fputs(yourString, yourFileP). The first one is usually slower since (beside the extra argument) fprintf is more complex than fputs, since it has to check what you want it to print. The difference is rather small, but maybe you could save a few microseconds if you make a lots of these functions calls.

  10. #10
    and the hat of sweating
    Join Date
    Aug 2007
    Location
    Toronto, ON
    Posts
    3,545
    Quote Originally Posted by spank View Post
    I believe the constant is BUFSIZ. I'm writing on windows an compiling with DevC++.

    I don't understand what you want to say with



    What syscalls are available with DevC++? Can I use open() write() read() like on unix ?

    Thanx!
    I've never used DevC++, but Visual Studio has all those functions (although they put a '_' infront of them to annoy people). When I look at the source code for them, they of course just call the WinAPI functions like CreateFile()...

  11. #11
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Posts
    22,918
    It's likely the actual writing is slower than the overhead of the function calls.
    Here's my tips: use caching. There are many options for fopen. You might want to use "wtS" for optimizing sequential access to disk. This can make a big impact, believe me (especially in Windows).
    Don't call fflush. That will just flush everything in the buffer which causes degradation. Buffering is there for a reason. The stream will automatically flush when the buffer is full. You don't need to do it manually.
    Last edited by Elysia; 12-28-2007 at 05:13 AM.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  12. #12
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Katy, Texas
    Posts
    2,309
    Elysia, is "wtS" a Microsoft thing? I had never seen that before.

    http://msdn2.microsoft.com/en-us/lib...cb(VS.80).aspx

    Todd

  13. #13
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Posts
    22,918
    It's not listed as Microsoft-specific, though. If there's any other documentation available that you can check, it should be easy to determine whether it's Microsoft-specific.
    If it is, then I guess you just have to use API.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  14. #14
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,308
    Obviously you don't simply intend to create gigabyte files of x's. Knowing what you really want to do would mean we can help far more.

    Please explain briefly what you intend to do that involves lots of IO. You may be surprised how helpful an answer you might then get.
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

  15. #15
    and the hat of sweating
    Join Date
    Aug 2007
    Location
    Toronto, ON
    Posts
    3,545
    Quote Originally Posted by Elysia View Post
    It's not listed as Microsoft-specific, though. If there's any other documentation available that you can check, it should be easy to determine whether it's Microsoft-specific.
    If it is, then I guess you just have to use API.
    Quote Originally Posted by MSDN
    The c, n, and t mode options are Microsoft extensions for fopen and _fdopen and should not be used where ANSI portability is desired.
    I don't see a "S" mode specified in the Apr. 2000 MSDN library, and it's not in Linux: http://man.he.net/?topic=fopen&section=all

Page 1 of 6 123456 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. read / write binary files
    By Dark_Phoenix in forum C++ Programming
    Replies: 5
    Last Post: 06-21-2009, 08:56 AM
  2. Create Copies of Files
    By Kanshu in forum C++ Programming
    Replies: 13
    Last Post: 05-09-2009, 08:53 AM
  3. added start menu crashes game
    By avgprogamerjoe in forum Game Programming
    Replies: 6
    Last Post: 08-29-2007, 02:30 PM
  4. Linking header files, Source files and main program(Accel. C++)
    By Daniel Primed in forum C++ Programming
    Replies: 3
    Last Post: 01-17-2006, 11:46 AM
  5. Using c++ standards
    By subdene in forum C++ Programming
    Replies: 4
    Last Post: 06-06-2002, 10:15 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21