fwrite slowdown problem

This is a discussion on fwrite slowdown problem within the C Programming forums, part of the General Programming Boards category; Hi, I have a probelm with fwrite function. I use it to write large chunk of binary data to 24-bit ...

  1. #1
    Registered User
    Join Date
    Jan 2011
    Posts
    3

    fwrite slowdown problem

    Hi,

    I have a probelm with fwrite function. I use it to write large chunk of binary data to 24-bit bitmap(.bmp) files of resolution 2400x1500.
    In the program below, I read an image 1700.bmp and write the same to 1.bmp. I perform this infinite number of times. At the beginning, the writting is fast. But after about 25 images, the writting becomes too slow. When I remove the fwrite statement, the program executes very fast. So fwrite is the culprit. But I have to make the program run fast and at the same speed. Why is this problem caused and how to rectify it? I have copied the program below:
    Code:
    #include <stdio.h>
    #include<conio.h>
    #include<stdint.h>
    struct MagicNumber
    {
    	unsigned char type[2]; 
    }BmpMN;
    
    struct INFOHEADER
    {
    	unsigned int sizeByte; 
    	unsigned short reserved1; 
    	unsigned short reserved2; 
    	unsigned int offsetbits; 
    	unsigned int sizeHeader; 
    	int width;
    	int height;
    	unsigned short planes; 
    	unsigned short bitcount; 
    	unsigned int compression; 
    	unsigned int sizeImage; 
    	int xpelspermeter;
    	int ypelspermeter;
    	unsigned int colorsused;
    	unsigned int colorsimportant;
    } BitmapInfoHeader; 
    
    int main()
    {
    	FILE *ip,*op;
    	int i=1;	
    	uint8_t buff[1500][2400*3];
    	while(1)
    	{
    		printf("Reading %d\n",i);
    		ip=fopen("op/1700.bmp","rb");
    		op=fopen("op/1.bmp","wb");
    		fread(&BmpMN,sizeof(BmpMN),1,ip); 	 
       		fread(&BitmapInfoHeader,sizeof(BitmapInfoHeader),1,ip);
    		fread(&buff,sizeof(buff),1,ip);
    		fwrite(&BmpMN,sizeof(BmpMN),1,op); 	 
       		fwrite(&BitmapInfoHeader,sizeof(BitmapInfoHeader),1,op);
    		//This statement is executed slower from 26th image onwards
    		fwrite(&buff,sizeof(buff),1,op);		
    		fclose(ip);		
    		fclose(op);
    		i++;
    		fflush(op);
    	}
    	return 0;
    }
    Thank you
    Last edited by ganeshkumar_198; 01-02-2011 at 07:52 AM. Reason: mistake in code

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,821
    What about your completely bogus call to fflush?

    $ gcc foo.c
    foo.c: In function ‘main’:
    foo.c:48: warning: passing argument 1 of ‘fflush’ from incompatible pointer type
    /usr/include/stdio.h:219: note: expected ‘struct FILE *’ but argument is of type ‘uint8_t (*)[1500][7200]’

    Do you get a similar kind of warning?
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #3
    Registered User
    Join Date
    Jan 2011
    Posts
    3
    Sorry. I meant to flush the output file op. That did not increase the speed. So I had changed it to flush buff i.e. the buffer that holds the data in an effort to see whether it speeds up the writing. But that too did not work.
    Yes I got that error that you had mentioned when I used
    Code:
    fflush(buff);
    which disappeared when I changed it to
    Code:
    fflush(&buff);
    Sorry for posting that incorrect modified code and thank you for pointing it out.
    Can you please tell me the reason for this slowdown and how to rectify it?

  4. #4
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by ganeshkumar_198 View Post
    Sorry. I meant to flush the output file op. That did not increase the speed. So I had changed it to flush buff i.e. the buffer that holds the data in an effort to see whether it speeds up the writing. But that too did not work.
    Yes I got that error that you had mentioned when I used
    Code:
    fflush(buff);
    which disappeared when I changed it to
    Code:
    fflush(&buff);
    Sorry for posting that incorrect modified code and thank you for pointing it out.
    Can you please tell me the reason for this slowdown and how to rectify it?
    fflush wants a file handle not a buffer pointer... should be fflush(op); and it should be before fclose(op);

    You might also want to give some thought to not creating an array of 10,800,000 bytes on the stack.
    Last edited by CommonTater; 01-02-2011 at 08:07 AM.

  5. #5
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,821
    Have you removed the fflush() call altogether?

    Because flushing buff (errors or not) is plain wrong.

    Also, which OS/Compiler are you using (I know the conio.h pretty much gives it away).
    But without the fflush(), I ran this on Ubuntu up to 50 iterations before getting bored with the repetitiveness of it all.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  6. #6
    Registered User
    Join Date
    Jan 2011
    Posts
    3
    I have written fflush(op) bfore fclose(op).
    Still, the writting is slow.
    I am using Msys/MinGW with gcc compiler in windows7 32 bit.

  7. #7
    Registered User
    Join Date
    Sep 2008
    Location
    Toronto, Canada
    Posts
    1,832
    Since you're always recreating the destination file, I see no reason that the writes would slow down. It's not under your control. In other words, I think the system may be buffering some amount of "written" file, and at some point the buffering catches up with you.

  8. #8
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Both the BIOS and Windows are getting pretty aggressive about power saving options. I know I was surprised to see a "power option" in Windows XP, that was not what I had set in the BIOS, and in Windows XP before some upgrades from Microsoft.

    After about 20 to 30 seconds of no key presses, my system now slows right down - regardless of what it's doing. I probably wouldn't have noticed it except I was displaying updates to a permutation program, at the time. Then I was hit by a virus "Win32 Malware-gen", and had to go into the registry quite a bit.

    Anyway, try the same program, but keep pressing a key every 10 seconds, and see if that changes anything. If it does, then it's a system slow down to save power.

    Linux has the same power saving options, just found in a different place, of course.

  9. #9
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by Adak View Post
    Both the BIOS and Windows are getting pretty aggressive about power saving options. I know I was surprised to see a "power option" in Windows XP, that was not what I had set in the BIOS, and in Windows XP before some upgrades from Microsoft.

    After about 20 to 30 seconds of no key presses, my system now slows right down - regardless of what it's doing. I probably wouldn't have noticed it except I was displaying updates to a permutation program, at the time. Then I was hit by a virus "Win32 Malware-gen", and had to go into the registry quite a bit.

    Anyway, try the same program, but keep pressing a key every 10 seconds, and see if that changes anything. If it does, then it's a system slow down to save power.

    Linux has the same power saving options, just found in a different place, of course.
    Adak... if you're thinking the hard disk is spinning down, that's very unlikely while it's being accessed. Windows keeps a number of timers --Screensaver, Monitor, Standby, DrivePark, etc.-- and will not shut things down if they are in use.

    It's more likely the rapid chain of open-write-close on the exact same file is making a total mess of the directory and disk buffering, invoking all kinds of cleanup and self-correction activity both in the buffers and on the disk itself.

  10. #10
    Registered User
    Join Date
    Sep 2004
    Location
    California
    Posts
    3,255
    My guess is that for the first 25 iterations, you are not actually writing to disk, you are just writing to the operating system's write cache. After the 25th time, the write cache is full of dirty pages and needs to start writing to disk before anything new can be written into the cache. Since writing to disk is a lot slower than just writing to memory, that is why everything starts moving slower.

    Also, please get rid of that call to fflush(). Not only are you doing it incorrectly, but a call to fclose() will automatically flush the output buffer, so there is no need to explicitly flush it yourself.
    bit∙hub [bit-huhb] n. A source and destination for information.

  11. #11
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    The hard drive doesn't spin down, (these systems never hibernate), but the system does s-l-o-w down. I really see it when it's writing a lot of lines of data to the screen. When the program starts, the lines of data on the screen fly by so fast, it's impossible to read it. After about 30 seconds with no key presses, it slows down so much, I can read each line. Press any key, and the speed immediately jumps up again, for another 20-30 seconds.

    This is a power saving feature that happens without screen saving, or hibernation or suspension, being implemented. Those features are all shut off.

    I've now tested this on several systems, running Windows XP, Windows 2000, and two versions of Ubuntu Linux. They all exhibited this slow down behavior, with default installation. (The Windows 2000 OS was on a laptop, so not unexpected. The rest were either Core2Duo's (3 of them), a Quad core (i7), or a double Quad core (Xeon's.)

    I also believe that data buffers are being filled, and must now be emptied by actual writing of the disk, as CT and Bithub have mentioned.

  12. #12
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by Adak View Post
    The hard drive doesn't spin down, (these systems never hibernate), but the system does s-l-o-w down. I really see it when it's writing a lot of lines of data to the screen. When the program starts, the lines of data on the screen fly by so fast, it's impossible to read it. After about 30 seconds with no key presses, it slows down so much, I can read each line. Press any key, and the speed immediately jumps up again, for another 20-30 seconds.

    This is a power saving feature that happens without screen saving, or hibernation or suspension, being implemented. Those features are all shut off.

    I've now tested this on several systems, running Windows XP, Windows 2000, and two versions of Ubuntu Linux. They all exhibited this slow down behavior, with default installation. (The Windows 2000 OS was on a laptop, so not unexpected. The rest were either Core2Duo's (3 of them), a Quad core (i7), or a double Quad core (Xeon's.)

    I also believe that data buffers are being filled, and must now be emptied by actual writing of the disk, as CT and Bithub have mentioned.
    I should think the latter is causing the former, in this case.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Memory problem with Borland C 3.1
    By AZ1699 in forum C Programming
    Replies: 16
    Last Post: 11-16-2007, 11:22 AM
  2. Someone having same problem with Code Block?
    By ofayto in forum C++ Programming
    Replies: 1
    Last Post: 07-12-2007, 09:38 AM
  3. A question related to strcmp
    By meili100 in forum C++ Programming
    Replies: 6
    Last Post: 07-07-2007, 03:51 PM
  4. WS_POPUP, continuation of old problem
    By blurrymadness in forum Windows Programming
    Replies: 1
    Last Post: 04-20-2007, 07:54 PM
  5. Laptop Problem
    By Boomba in forum Tech Board
    Replies: 1
    Last Post: 03-07-2006, 06:24 PM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21