writing to binary

is there a difference between writing to a binary file, 1 number at a time, as opposed to lets say 1000 numbers at a time?

is there time used up just by using the fwrite command?

I have a big file to write and at the moment it takes 20 days to write. With a preliminary code and writing one file at a time with no multi threading.

Of course there's a difference.

Each call to fwrite() will have an overhead, so if you write a number at a time or 1000 numbers will make a noticable difference. If you have a huge amount of data, you may want to use an even bigger number than 1000.

--
Mats

ok so the number or numbers in this array that i will write will be limited by my RAM correct.

Im using integers so 16 bits = 2 bytes per number.

How much extra space do i have to allow to keep my computer from crashing?
(Windows XP Pro)

I mean, i have 2GB RAM lets say, so i need to leave X MB for windows

A standard integer is 4 bytes. A short int only uses 2. If your program is crashing and you are using arrays, then are you sure you arent blowing the stack?

How much memory Windows itself uses depends on so many things that it's completely impossible to give you a fixed number that will work at all times on different systems and under different circumstances. As a rough guideline, you will need to leave something like

Code:

---

64 * 1024 * 1024 + (8 * memorysize_in_bytes / 4096) * number_of_apps_running

---

That is in bytes.

The long calculation is to figure out how much memory the page-tables take up, the 64MB is just a guestimate of how much memory kernel, a few drivers, etc is using up.

Note also that the amount of data you can write with fwrite is definitely not limited by the amount of ram in your machine, I could easily fill any size disk with this:

Code:

---

int main(void)
{
  FILE *f = fopen("something.bin", "wb");
  __int64 buf[1000];
  __int64 n = 0;
  for(;;) {
      for(i = 0; i < 1000; i++)
          buf[i] = n+i;
      n += 1000;
      if (fwrite(f, buf, sizeof(buf), 1) <= 0)
        break;
  }
  fclose(f);
}

---

[No, it isn't complete, nor does it good error checking]

--
Mats

my program isnt crashing it just runs to slowly.

oh and i meant i will have to make an array to be written by fwrite.
So Is the size of the array is limited by my ram?

Let's get one thing clear, as you increase the size of data you send to fwrite, the overhead for calling fwrite is reduced. Say we do this:
fwrite(f, b, 1, s); overhead = 1
fwrite(f, b, 10, s); overhead = 0.1
fwrite(f, b, 100, s); overhead = 0.01

As you can see, the overhead per item written reduces quite quickly.

Eventuallly, the difference between calling fwrite with one number and another larger number will be so small that you can't tell the difference. At which point making your storage array larger will not help much.

I'd say something like 32 or 64KB is large enough to not make any difference in overhead - but by all means, make some measurement of the performance for various sizes of data [if you use a named constant for the number of items per fwrite, you can quite easily change the size from one mumber to another]

--
Mats

hmmm just by counting with my metronome i can see:
significant improvements going from 1000 to 10 000 (x2 speed)
negligible improvements going from 10 000 to 100 000 (x12/11 = x1.091)
and the program crashes at 1 000 000

so ill stop at 10 000 elements per column, with 20 columns.

sweet thanks man.

Quote:

---

Originally Posted by chico1st

is there a difference between writing to a binary file, 1 number at a time, as opposed to lets say 1000 numbers at a time?

---

It doesn't matter much. The entire point of a buffered I/O layer like stdio is to make it efficient to do small reads and writes. The library buffering does take time, but not nearly as much as if you'd called the operating system directly a bunch of times with a bunch of small chunks of data.

Suppose you wanted to write 1000 objects at a time instead of 1. So you write some code which buffers 1000 objects in a buffer and then calls fwrite() once. But what you've done is basically duplicated what fwrite() would have done anyway.

Trust the stdio buffering. Do your writes however is most convenient. If it's still too slow, there may be problems elsewhere in the code.

:S i duno... it takes about 10x as long for me to use 20 000 000 fwrites with one number as opposed to 2000 fwrites with 10 000 numbers

i duno completely understand why but i was just testing it roughly.

Quote:

---

Originally Posted by chico1st

:S i duno... it takes about 10x as long for me to use 20 000 000 fwrites with one number as opposed to 2000 fwrites with 10 000 numbers

---

That's probably due to cache considerations, not anything intrinsically wrong with fwrite(). I'd be curious to see a profile...

Quote:

---

Originally Posted by brewbuck

It doesn't matter much. The entire point of a buffered I/O layer like stdio is to make it efficient to do small reads and writes. The library buffering does take time, but not nearly as much as if you'd called the operating system directly a bunch of times with a bunch of small chunks of data.

Suppose you wanted to write 1000 objects at a time instead of 1. So you write some code which buffers 1000 objects in a buffer and then calls fwrite() once. But what you've done is basically duplicated what fwrite() would have done anyway.

Trust the stdio buffering. Do your writes however is most convenient. If it's still too slow, there may be problems elsewhere in the code.

---

It is indeed buffered, but in this case, we're talking about writing two bytes (1 short integer) versus writing a large number of bytes - the time it takes to put 2 bytes into a the buffer is most likely very small, compared to the time it takes to push 4 parameters to the stack [fwrite(f, buffer, size, count) is four parameters] and calling the function itself, along with whatever goes into the "fwrite" function itself.

--
Mats

Code:

---

for(;;)

---

gross!

Quote:

---

Originally Posted by simpleid

Code:

---

for(;;)

---

gross!

---

That's how I always write an unconditional loop. It's typical.

Quote:

---

Originally Posted by simpleid

Code:

---

for(;;)

---

gross!

---

And how do you propose making "infinit" loops?


Yes, I learnt that a long time ago for use in the "forever" loop in the process in a real-time OS (a process in this RTOS isn't meant to end itself, in fact if it ever returns, it will throw an error about "process ended unexpectedly"), and whenever I want to do something "forever", I use that. It's no longer to write than "while(1)" and shorter than "do ... while(1);"

Of course, I could have written:

Code:

---

....
a:
...
goto a;

---

It wold have been slightly longer, and clearly not "good style".

--
Mats

Quote:

---

Originally Posted by simpleid

Code:

---

for(;;)

---

gross!

---

Code:

---

while (1)

---

isn't beautiful either, but every loop type in C/C++ requires some type of loop condition, even if it's always true (presumably it's not worth having a separate type for an infinite loop since the compiler can recognize one when it sees it).

I'm not saying anyone is wrong? Or that you should do it any other way... I still find the syntax of "for(;; )" very visually awkward. I'm only expressing distaste in the style, despite the fact that you deem it preferred.

Do what you want, but if complaints never arise, then change won't occur. Maybe in future languages the aspect will be addressed because someone like me said "damn that's ugly lookin fosho."

:-)

I agree though, while(1) has a very... odd appearance, the 1 is just as unnecessary as the ';;' should be... why can't 'true' be assumed?

What I find as ironic, is that the two most popular looping methods provide a mechanism to break out, yet we ignore it and provide the condition to break free within... that seems funny to me.

Quote:

---

Originally Posted by simpleid

I agree though, while(1) has a very... odd appearance, the 1 is just as unnecessary as the ';;' should be... why can't 'true' be assumed?

---

It is assumed on for-loops, hence the "for(;;)" works. I've seen people us a macro (e.g. "forever" to do "for(;;)".

Quote:

---

What I find as ironic, is that the two most popular looping methods provide a mechanism to break out, yet we ignore it and provide it within.

---

I usually try to use a condition in my while/for-loops - and in the above case, I suppose I should have used a "do ... while(fwrite(...) > 0);" instead.

--
Mats

Quote:

---

Originally Posted by simpleid

I'm only expressing distaste in the style, despite the fact that you deem it preferred.

---

I didn't say it was preferred -- I said it was typical. I've seen and used both.

Heck, if you want:

Code:

---

#define forever for(;;)

forever
{
    ...
}

---

Quote:

---

What I find as ironic, is that the two most popular looping methods provide a mechanism to break out, yet we ignore it and provide the condition to break free within... that seems funny to me.

---

Sometimes the natural place to break the loop is not at the beginning or end. Contorting your code so that the break is triggered by the loop condition can sometimes result in a big mess that's hard to figure out. I'm not making any judgement either way about this PARTICULAR case, since I haven't bothered looking at it in detail.

oh i know, i just wanted to try and make a joke out of it. heh.

however, the 'forever' really does look clean... might have to start using that...

> however, the 'forever' really does look clean... might have to start using that...
I wouldn't do that. It's only slightly shorter, and evil, being a macro. If forever is ever made a built-in keyword, that would be different.

Edit: The exact same length, actually, assuming no white space in for(;;). But someone seeing the latter doesn't have to hunt around to verify that it means what they think it does.

Quote:

---

I'm not making any judgement either way about this PARTICULAR case, since I haven't bothered looking at it in detail.

---

Without looking at it much in detail, I daresay the example in question:

Code:

---

for(;;) {
  for(i = 0; i < 1000; i++)
      buf[i] = n+i;
  n += 1000;
  if (fwrite(f, buf, sizeof(buf), 1) <= 0)
    break;
}

---

Can be just as easily expressed as:

Code:

---

do {
  for(i = 0; i < 1000; i++)
      buf[i] = n+i;
  n += 1000;
} while (fwrite(f, buf, sizeof(buf), 1) > 0);

---

Thus sidestepping this "which deliberate infinite loop is better?" argument, but creating a "is a do while loop better than a deliberate infinite loop where both will suffice?" argument :p