Saving large files faster than light!

**ZapoTex** · 01-06-2005

Hi everyone! This is my first post. I hope I'm not saying things that go against this forum's way of thinking or violating any rule you may have. If I do, I'm sorry.

Here is the problem: I have to save hundreds of large files (about 3 MB each), each consisting of 1.5 millions integer numbers. I thought that a million and a half "fprintf"s was the slowest way and tried to create a large string containing all my data with:

for (i=0; i<imagewidth*imageheight; i++) /*they are images!*/
sprinft(bigstring, "%s%d ", bigstring, integernumber[i]);

but to my surprise this is much slower!!!!!

Is there a way to compose strings much faster than sprintf?

The same problem is when I open such files: is there anything faster than 1.5 millions of "fscanf"???

Another thing. I was thinking: if I give up a little bit of precision, I might use bytes instead of integer and save the image in bmp format or png format in black and white. But is there a library that allows me to do this?
I tried many already, but none of them worked. (p.e. ReadAndSavePng did'nt work)

The compilers I use are Borland Turbo C 2.01 on DOS and LabWindows CVI on Windows.

Thanks a lot to everyone who will answer.
By!

**quzah** · 01-06-2005

Is there some reason you need strings? You could save yourself a lot of trouble by using a binary file, and using fwrite or fread calls to read or write blocks at a time.

Quzah.

**ZapoTex** · 01-06-2005

Ok, I know nothing about binary files and fread and fwrite. You say it is the fastest way, then I know it is what I need. I'll learn that stuff on a tutorial!

Thank you really really much.

ZapoTex

**quzah** · 01-06-2005

Basicly, you open the file for writing / reading in binary mode. You can then use the afore mentioned functions to write entire arrays or what not in one shot:

Code:

int array[SOMESIZE];
FILE *fp;

fp = fopen( "filename.foo", "wb" ); /* look up fopen for more info */

...fill array...

fwrite( array, sizeof( int ), SOMESIZE, fp );

fclose( fp );

Something like that will work. Look up fread and fwrite for more info.

Quzah.

**itsme86** · 01-06-2005

Writing your data to /dev/null instead of something like "filename.foo" also speeds things up tremendously

The reason your sprintf() attempt was slower than the several fprintf() calls is because fprintf() doesn't actually write to the disk (or even call the syscall write()) every time it's called. Instead, fprintf() dumps the data into the stream's buffer. After you've filled that buffer with standard I/O writing routines (such as fprintf(), fputs(), fputc(), etc.) it will then call write() to dump the whole buffer at once to disk.

So basically, your sprintf() attempt was just duplicating the effort that the standard I/O system was already performing. That's why it was slower.

**caroundw5h** · 01-06-2005

You might also want to take a look at setbuf and setvbuf to allocate a larger buffer area.

**ZapoTex** · 01-07-2005

Wow! Terrific way to give me your welcome.
This is a very friendly forum with lots of nice people!

@caroundw5h: I'll try soon, this might be the simplest way to achieve my target

@itsme86: I didn't understand what you said completely, but you're saying that the time it takes to save the file depends on the operations needed to fill the buffer more than on those needed to acces the hard disk? I thought that because memory is much faster than mass storage unities, by limiting the number of accesses to hard disk I would speed things up. Evidently Windows's smart drive is smarter than I am!

@quzah: it sounds really fast. I'll code that as soon as I finish writing this post. Thank you again. This way I couldn't read the images I've saved up to now in the dumb way, could I? No problem, I'll write something to convert them all and let the computer work overnight.

Thanks a lot everybody.
Bye!

**itsme86** · 01-07-2005

No, I'm not saying accessing the hard disk is faster than filling a buffer. What I meant was, the standard I/O functions (fprintf() is one of those) already utilize a buffer. So you're filling your buffer and then fprintf() is using its own buffer. So the amount of hard disk access is the same whether or not you use your buffer, but you're duplicating the buffering effort by using your own.

Also, the printf() family of functions are rather slow compared to non-formatted I/O functions such as fwrite() like quzah suggested.

**Salem** · 01-07-2005

> The compilers I use are Borland Turbo C 2.01
This compiler isn't going to do millions of anything, at least not without a great deal of difficulty.
Even when you manange it, the results will be less than optimal for your win9x/2K/XP operating system which you're probably running it on.

**enjoy** · 01-07-2005

you can use <fstream.h> if you compile with visual c/c++

Code:

char *file="file.txt";
char inputdata="anythings";
fstream  data(file, ios::in|ios::out|ios::nocreate);
if(!data){printf("file.txt does not exist !\n");exit(0);}
/* if you want  write */

data<<inputdata; /*you can repeate as you want*/

/* if you want read */

/* use while(!data.eof())  so */

data.get(tmp);
puts(tmp);

data.close();
printf("end of file\n");
cin.get();

**itsme86** · 01-07-2005

This is the C board, not the C++ one.

**enjoy** · 01-07-2005

i know
but i post this

Code:


if you compile with visual c/c++

**quzah** · 01-07-2005

So why didn't you post a Java solution with the line "if you compile with java"? Or how about a Pascal solution? See where this is going? See why we have forums for seperate languages?

Quzah.

**Salem** · 01-07-2005

> if you compile with visual c/c++
It's no wonder that you remain clueless with your programming efforts then...

**enjoy** · 01-07-2005

ok ok ok ....

Thread: Saving large files faster than light!

Thread Tools

Search Thread

Display

Saving large files faster than light!

Ok!

Thank you!

Similar Threads

Saving numbers to files

How to write files more faster in C

Builder C++ large files support for simple C program

Is there a faster way to open and read files, this seems too slow.

Reading Large Files!!!