Saving large files faster than light!

This is a discussion on Saving large files faster than light! within the C Programming forums, part of the General Programming Boards category; Hi everyone! This is my first post. I hope I'm not saying things that go against this forum's way of ...

  1. #1
    Registered User
    Join Date
    Jan 2005
    Posts
    12

    Saving large files faster than light!

    Hi everyone! This is my first post. I hope I'm not saying things that go against this forum's way of thinking or violating any rule you may have. If I do, I'm sorry.

    Here is the problem: I have to save hundreds of large files (about 3 MB each), each consisting of 1.5 millions integer numbers. I thought that a million and a half "fprintf"s was the slowest way and tried to create a large string containing all my data with:

    for (i=0; i<imagewidth*imageheight; i++) /*they are images!*/
    sprinft(bigstring, "%s%d ", bigstring, integernumber[i]);

    but to my surprise this is much slower!!!!!

    Is there a way to compose strings much faster than sprintf?

    The same problem is when I open such files: is there anything faster than 1.5 millions of "fscanf"???

    Another thing. I was thinking: if I give up a little bit of precision, I might use bytes instead of integer and save the image in bmp format or png format in black and white. But is there a library that allows me to do this?
    I tried many already, but none of them worked. (p.e. ReadAndSavePng did'nt work)

    The compilers I use are Borland Turbo C 2.01 on DOS and LabWindows CVI on Windows.

    Thanks a lot to everyone who will answer.
    By!

  2. #2
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    Is there some reason you need strings? You could save yourself a lot of trouble by using a binary file, and using fwrite or fread calls to read or write blocks at a time.

    Quzah.
    Hope is the first step on the road to disappointment.

  3. #3
    Registered User
    Join Date
    Jan 2005
    Posts
    12

    Ok!

    Ok, I know nothing about binary files and fread and fwrite. You say it is the fastest way, then I know it is what I need. I'll learn that stuff on a tutorial!

    Thank you really really much.

    ZapoTex

  4. #4
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    Basicly, you open the file for writing / reading in binary mode. You can then use the afore mentioned functions to write entire arrays or what not in one shot:
    Code:
    int array[SOMESIZE];
    FILE *fp;
    
    fp = fopen( "filename.foo", "wb" ); /* look up fopen for more info */
    
    ...fill array...
    
    fwrite( array, sizeof( int ), SOMESIZE, fp );
    
    fclose( fp );
    Something like that will work. Look up fread and fwrite for more info.

    Quzah.
    Hope is the first step on the road to disappointment.

  5. #5
    Gawking at stupidity
    Join Date
    Jul 2004
    Location
    Oregon, USA
    Posts
    3,175
    Writing your data to /dev/null instead of something like "filename.foo" also speeds things up tremendously

    The reason your sprintf() attempt was slower than the several fprintf() calls is because fprintf() doesn't actually write to the disk (or even call the syscall write()) every time it's called. Instead, fprintf() dumps the data into the stream's buffer. After you've filled that buffer with standard I/O writing routines (such as fprintf(), fputs(), fputc(), etc.) it will then call write() to dump the whole buffer at once to disk.

    So basically, your sprintf() attempt was just duplicating the effort that the standard I/O system was already performing. That's why it was slower.
    If you understand what you're doing, you're not learning anything.

  6. #6
    Registered User caroundw5h's Avatar
    Join Date
    Oct 2003
    Posts
    751
    You might also want to take a look at setbuf and setvbuf to allocate a larger buffer area.

  7. #7
    Registered User
    Join Date
    Jan 2005
    Posts
    12

    Thank you!

    Wow! Terrific way to give me your welcome.
    This is a very friendly forum with lots of nice people!

    @caroundw5h: I'll try soon, this might be the simplest way to achieve my target

    @itsme86: I didn't understand what you said completely, but you're saying that the time it takes to save the file depends on the operations needed to fill the buffer more than on those needed to acces the hard disk? I thought that because memory is much faster than mass storage unities, by limiting the number of accesses to hard disk I would speed things up. Evidently Windows's smart drive is smarter than I am!

    @quzah: it sounds really fast. I'll code that as soon as I finish writing this post. Thank you again. This way I couldn't read the images I've saved up to now in the dumb way, could I? No problem, I'll write something to convert them all and let the computer work overnight.

    Thanks a lot everybody.
    Bye!

  8. #8
    Gawking at stupidity
    Join Date
    Jul 2004
    Location
    Oregon, USA
    Posts
    3,175
    No, I'm not saying accessing the hard disk is faster than filling a buffer. What I meant was, the standard I/O functions (fprintf() is one of those) already utilize a buffer. So you're filling your buffer and then fprintf() is using its own buffer. So the amount of hard disk access is the same whether or not you use your buffer, but you're duplicating the buffering effort by using your own.

    Also, the printf() family of functions are rather slow compared to non-formatted I/O functions such as fwrite() like quzah suggested.
    If you understand what you're doing, you're not learning anything.

  9. #9
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,855
    > The compilers I use are Borland Turbo C 2.01
    This compiler isn't going to do millions of anything, at least not without a great deal of difficulty.
    Even when you manange it, the results will be less than optimal for your win9x/2K/XP operating system which you're probably running it on.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  10. #10
    /*enjoy*/
    Join Date
    Apr 2004
    Posts
    159
    you can use <fstream.h> if you compile with visual c/c++

    Code:
    char *file="file.txt";
    char inputdata="anythings";
    fstream  data(file, ios::in|ios::out|ios::nocreate);
    if(!data){printf("file.txt does not exist !\n");exit(0);}
    /* if you want  write */
    
    data<<inputdata; /*you can repeate as you want*/
    
    /* if you want read */
    
    /* use while(!data.eof())  so */
    
    data.get(tmp);
    puts(tmp);
    
    data.close();
    printf("end of file\n");
    cin.get();

  11. #11
    Gawking at stupidity
    Join Date
    Jul 2004
    Location
    Oregon, USA
    Posts
    3,175
    This is the C board, not the C++ one.
    If you understand what you're doing, you're not learning anything.

  12. #12
    /*enjoy*/
    Join Date
    Apr 2004
    Posts
    159
    i know
    but i post this
    Code:
    
    if you compile with visual c/c++
    

  13. #13
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    So why didn't you post a Java solution with the line "if you compile with java"? Or how about a Pascal solution? See where this is going? See why we have forums for seperate languages?

    Quzah.
    Hope is the first step on the road to disappointment.

  14. #14
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,855
    > if you compile with visual c/c++
    It's no wonder that you remain clueless with your programming efforts then...
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  15. #15
    /*enjoy*/
    Join Date
    Apr 2004
    Posts
    159
    ok ok ok ....

Page 1 of 2 12 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Saving numbers to files
    By cpudaman in forum C++ Programming
    Replies: 5
    Last Post: 01-10-2008, 01:05 AM
  2. How to write files more faster in C
    By spank in forum C Programming
    Replies: 75
    Last Post: 01-08-2008, 02:15 PM
  3. Builder C++ large files support for simple C program
    By Murfury in forum C Programming
    Replies: 3
    Last Post: 11-23-2007, 03:47 PM
  4. Replies: 8
    Last Post: 12-27-2003, 02:30 PM
  5. Reading Large Files!!!
    By jon in forum Windows Programming
    Replies: 1
    Last Post: 09-10-2001, 12:20 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21