fread / fwrite

This is a discussion on fread / fwrite within the C Programming forums, part of the General Programming Boards category; hi, i'm trying to do some fast text processing on files, and it seems that the bottleneck in terms of ...

  1. #1
    Registered User
    Join Date
    Sep 2001
    Posts
    32

    Question fread / fwrite

    hi,

    i'm trying to do some fast text processing on files, and it seems that the bottleneck in terms of speed is reading and writting to the file.

    the processing is very fast. is there any way to speed-up feading and writting to files? i increased the read/write buffer size but it didnt make much difference. the H/D light flashes in the same rate as well.

    does the buffer-size on fread/fwrite make any difference? or does it keep it's own internal buffers anyway and only when they are full it flushes them to disk?

    any ideas?

    one more thing...the c++ fstream way of dealing with files, is it any faster?

    thanks a lot

    null

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,852
    > and it seems that the bottleneck in terms of speed is reading and writting to the file.
    Since most disks quote seek and access times in milliseconds, its not surprising. Assuming best case, the upper limit of transfer rates from HD to processor is still comparitively slow. These are physical problems which you won't get past, even if your code ran in zero time.

    > is there any way to speed-up feading and writting to files?
    There is a constant (BUFSIZ) in stdio.h, which is supposed to be an optimal size (for average use), so your fread/fwrite calls should use this.

    You can define your own buffers like so
      int setvbuf(FILE *file, char *buffer, int type, int length);
    Experimenting with large buffers here might help.

    > one more thing...the c++ fstream way of dealing with files, is it any faster?
    Probably not - but it might be worth measuring

  3. #3
    Registered User
    Join Date
    Sep 2001
    Posts
    32

    Thumbs up

    Salem, you are DA MAN!!!

    what a great answer! just says it all. To tell you the truth, when i posted i didn't think there'd be a fix to the problem.

    found the function you said in msdn. I'll do it with that function and a global buffer.

    also, found the BUFSIZ. it seems to be set on 512 (that explains it...).

    i'm using visual studio 6, there seem to be two stdio.h in it. One is in:

    ...\VC98\Include\STDIO.H

    and the other is:

    ...\VC98\CRT\SRC\STDIO.H

    the first contains the lines:
    -------------------------------------------------------------------------
    /* Buffered I/O macros */

    #if defined(_M_MPPC)
    #define BUFSIZ 4096
    #else /* defined (_M_MPPC) */
    #define BUFSIZ 512
    #endif /* defined (_M_MPPC) */
    -------------------------------------------------------------------------




    the second one contains the lines
    --------------------------------------------------------------------------
    #if defined (_M_MPPC)
    #define BUFSIZ 4096
    #else /* defined (_M_MPPC) */
    #define BUFSIZ 512
    #endif /* defined (_M_MPPC) */

    #ifndef _INTERNAL_IFSTRIP_
    /*
    * Real default size for stdio buffers
    */
    #define _INTERNAL_BUFSIZ 4096
    #define _SMALL_BUFSIZ 512
    #endif /* _INTERNAL_IFSTRIP_ */
    ---------------------------------------------------------------------------


    not sure but _M_MPPC looks like powerPC to me...so it's probably the 512.

    just out of curiosity, do you happen to know why the second file has these additional lines.

    what i make out of it is that the second file is for the msvcrt.dll which is the c runtime library (if i remember). on the other hand, the msvcrt.dll is already compiled isn't it?

    Don't know if you use vc++ at all, but thought not to crosspost to the win board yet...

    Anyway, thank you again for your great help. I'll be trying it all tonight. I'll let you know tomorow the difference it made...

    take care

    null

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. 2d array and fwrite and fread
    By totalnewbie in forum C Programming
    Replies: 14
    Last Post: 01-10-2009, 03:45 PM
  2. fwrite / fread question
    By Takteek in forum C Programming
    Replies: 5
    Last Post: 11-27-2008, 04:18 AM
  3. Weird problem with fwrite() and fread()
    By piote in forum C Programming
    Replies: 2
    Last Post: 11-13-2004, 03:07 PM
  4. buffer type for fread & fwrite
    By daluu in forum C Programming
    Replies: 5
    Last Post: 05-08-2003, 07:57 PM
  5. fread() and fwrite() ?
    By Limblet in forum C Programming
    Replies: 4
    Last Post: 09-25-2001, 08:36 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21