Thread: fclose slow

  1. #1
    Registered User
    Join Date
    Mar 2009
    Posts
    3

    fclose slow

    Hello,

    I've made a file synchronization application that copies files from one machine to another across the network. For this I use the standard fopen, fread, fwrite and fclose calls (see code below). The copying itself works a treat but when I call fclose (specially for big files) it takes a long time to close the file (even longer than it took to copy!). I've disabled buffering with a setbuf call but this does not seem to have any effect. I even throw in a fflush call during the copy process. It's really annoying - the user sees a progress bar when copying the file which runs fine up to 100% and then hangs for a long time while it's closing the file. Is there anything I can do to make the fclose return quicker?
    I'm copying from a Linux PC to a Windows PC across my home network.
    Also, other sync programs that I have do not seem to have this problem.

    I'm using GNU C on a Linux PC (Xubuntu)
    IDE = Anjuta + GTK+ and Libglade

    Ruud

    ==========================================

    Code:
    gboolean copy ( gchar *src_name,
                    gchar *dst_name,
                    int size )
    {
      gchar buf [ BUFSIZ ];
      FILE *src;
      FILE *dst;
      int r,w;
      int done = 0;
      struct stat src_sb;
      struct utimbuf file_times;
      gboolean rtn = TRUE;
    
      set_progress ( done, size );
      if ( stat ( src_name, &src_sb ) < 0 )
      {
        g_warning ( "Could not stat %s\n", src_name );
        return FALSE;
      }
      if ( ( src = fopen ( src_name, "rb" ) ) == NULL )
      {
        g_warning ( "Error opening %s", src_name );
        return FALSE;
      }
      setbuf ( src, NULL );
      if ( ( dst = fopen ( dst_name, "wb" ) ) == NULL ) // consider using tmpfile
      {
        g_warning ( "Error opening %s", dst_name );
        rtn = FALSE;
      }
      else
      {
        setbuf ( dst, NULL );
        rtn = in_progress ();
        while ( rtn && ( r = fread ( &buf, sizeof ( gchar ), BUFSIZ, src ) ) > 0 )
        {
          rtn = in_progress ();
          if ( rtn )
          {
            w = fwrite ( &buf, sizeof ( gchar ), r, dst );
            fflush ( dst );
            done = done + w;
            set_progress ( done, size );
            if ( w != r )
            {
              g_warning ( "Error writing to %s", dst_name );
              rtn = FALSE;
            }
          }
          else
            g_warning ( "Aborted...." );
        }
        fclose ( dst );
      }
      fclose ( src );
      if ( rtn )
      {
        file_times.actime = src_sb.st_atime;
        file_times.modtime = src_sb.st_mtime;
        if ( utime ( dst_name, &file_times ) != 0 )
        {
          g_warning ( "Error preserving times for %s", dst_name );
          rtn = FALSE;
        }
      }
      return rtn;
    }

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    That is likely because the fclose causes the data to be ACTUALLY written to disk - up until the fclose, the OS just keeps it all in RAM, but once you close the file, it says "Ok, that's it, lets store this into the file to get rid of it".

    Using the low-level function open() with O_DIRECT will make each write happen as it is issued, rather than being delayed. I'm far from sure this will ACTUALLY HELP, but it may do. You then have to either use open, read, write and close instead for the fopen, etc, or use fdopen() to make a FILE pointer from a file-descriptor from open().

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    Registered User
    Join Date
    Mar 2009
    Posts
    3
    You must think me so very rude.
    I never got a notification for your reply (or have but forgot about it).
    Anyway my apologies and a (very) belated thank you.
    Your suggestion worked perfectly :-)

  4. #4
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    You have to set your "Default Thread Subscription Mode" under User CP->edit options to receive email notification.

    Kind of interesting setbuf() and fflush() did not work. Altho setbuf() does not actually turn buffering off, for that you need setvbuf(). setbuf() sets the buffer to a default value (eg, on my normative linux system it is 8kb).
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  5. #5
    Registered User
    Join Date
    Mar 2009
    Posts
    3
    Thanks for the Default Thread Subscription Mode tip.
    It's set to e-mail notifications now (I would have thought that would be the default).

    Thanks also for the tip on setvbuf. however. I've reverted back to simple low-level functions open, close, read and write. This gives me more control and is more transparent.
    I open the source file with src = open ( src_name, O_RDONLY | O_SYNC )
    and the destination file with dst = open ( dst_name, O_WRONLY | O_CREAT | O_SYNC )
    O_DIRECT did not work for me.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Can someone clarify what a 'slow event' is.
    By Overworked_PhD in forum Networking/Device Communication
    Replies: 2
    Last Post: 01-13-2008, 05:11 PM
  2. slow typing ad highlighting
    By axon in forum Tech Board
    Replies: 10
    Last Post: 01-13-2004, 08:27 PM
  3. slow game
    By lambs4 in forum Game Programming
    Replies: 2
    Last Post: 08-21-2003, 02:08 PM
  4. fclose()
    By GanglyLamb in forum C Programming
    Replies: 4
    Last Post: 11-11-2002, 08:45 AM
  5. Solutions for slow performance?
    By Hunter2 in forum Game Programming
    Replies: 52
    Last Post: 08-24-2002, 10:04 AM

Tags for this Thread