Check if file is opened for reading (unix)

This is a discussion on Check if file is opened for reading (unix) within the C Programming forums, part of the General Programming Boards category; My program is to scan a directory for new files, and then forward them along. However, I cannot forward a ...

  1. #1
    Registered User
    Join Date
    Jan 2007
    Posts
    40

    Question Check if file is opened for writing (unix)

    My program is to scan a directory for new files, and then forward them along. However, I cannot forward a file until it is complete, and as far as I'm aware, the only way to tell if the file is complete is if no other process is accessing it.

    I tried checking for write locks using fcntl, but either I'm using it wrong or the process doesn't put a write lock on the file.

    So basically, I'm looking for something similar to fuser <file> or lsof <file>. What command/library am I looking for here? Don't say "exec", because there's no sense getting a full wheel when all you need is a spoke.

    Thanks in advance,
    -IsmAvatar

    (If you want me to post the code I tried to check for locks, just say so, but I figured there's nothing further to be said there)
    Last edited by IsmAvatar2; 06-24-2009 at 10:44 AM.

  2. #2
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by IsmAvatar2 View Post
    My program is to scan a directory for new files, and then forward them along. However, I cannot forward a file until it is complete, and as far as I'm aware, the only way to tell if the file is complete is if no other process is accessing it.
    Since you do not need to open a file when scanning a directory, it is hard to tell what you mean here -- are you worried about other processes, not part of your program running on the system that could be accessing the file?
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  3. #3
    Registered User
    Join Date
    Jan 2007
    Posts
    40
    Yes. My program is to scan the directory for *new*, *completed* files created by another process.

    Program 1 (not in my control) opens file "out1.txt" for writing.
    P1 writes some bytes to file.
    P1 closes file.
    P1 opens file "out2.txt" for writing.
    P1 writes some bytes to file, but doesn't finish.
    Program 2 (my program) opens directory, and finds "out1.txt"
    P2 finds out that the file is closed and completed, so transmits its information.
    P2 next finds "out2.txt"
    P2 finds out that the file is still open, so it ignores it.
    P2 finds no more files, closes directory, and terminates (until it's run again)
    P1 finishes writing "out2.txt" and closes it.

  4. #4
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by IsmAvatar2 View Post
    Yes. My program is to scan the directory for *new*, *completed* files created by another process.
    I've never had to deal with this so I won't be much help -- my first thought was to try something with fcntl(), but I see you've already done that.

    Most people would consider this a task for a shell script, where you could use lsof, etc "natively". My hunch is you will have to use a system/exec/popen to access the shell anyway.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  5. #5
    Registered User
    Join Date
    Jan 2007
    Posts
    40
    My concern is that my program may not have access to lsof or fuser, as they oftentimes require root privileges. I was also hoping that I might just be able to use their lower level routines without having to use the whole program and parse the output. But of course skimming through lsof's source code can take hours.

  6. #6
    Registered User
    Join Date
    Oct 2008
    Location
    TX
    Posts
    2,057
    Post the code snippet that calls fcntl(); P2 needs to open "out2.txt" for writing in exclusive mode and methinks you are trying to get a shared lock instead.

  7. #7
    pwning noobs Zlatko's Avatar
    Join Date
    Jun 2009
    Location
    The Great White North
    Posts
    132
    Hello.
    You probably didn't see the bottom of the fuser man page. There's an fuser function in man section 2. I never used it myself.

    Edit:
    Sorry, there is no fuser function on linux, I was looking on my Tru64 system. On linux, check /proc/pid/fd for a list of files in use.
    Last edited by Zlatko; 06-24-2009 at 11:25 AM.

  8. #8
    Registered User
    Join Date
    Jan 2007
    Posts
    40
    itCbitC:
    Code:
    int main() {
     char *fn = "/tmp/test.txt"; //the test file
     FILE *F = fopen(fn,"w"); //this simulates another process writing to a file but not closing it yet
    
     //We now attempt to detect that the file is still open
     int fd = open(fn,O_RDONLY);
     struct flock fl;
      fl.l_type   = F_WRLCK;  /* F_RDLCK, F_WRLCK, F_UNLCK */
      fl.l_whence = SEEK_SET; /* SEEK_SET, SEEK_CUR, SEEK_END */
      fl.l_start  = 0; /* Offset from l_whence */
      fl.l_len    = 0; /* length, 0 = to EOF */
      fl.l_pid    = getpid();
    
     printf("%d %d %d %d %d\n",fl.l_type,fl.l_whence,fl.l_start,fl.l_len,fl.l_pid);
     //1 0 0 0 19865
     int ret = fcntl(fd,F_GETLK,&fl);
     printf("%d %d %d %d %d\n",fl.l_type,fl.l_whence,fl.l_start,fl.l_len,fl.l_pid);
     //2 0 0 0 19865
    
     sleep(3); //this gives me time to manually run fuser/lsof to observe that the file is indeed open
    
     //finalize
     fclose(F);
     return 0;
    }
    According to the fcntl documentation, if the lock is not detected, it will keep the struct the same but replace the mode with UNLCK, which is precisely what it does here.


    Zlatko, are you referring to this?
    fuser cannot report on any processes that it doesn’t have permission to
    look at the file descriptor table for. The most common time this prob-
    lem occurs is when looking for TCP or UDP sockets when running fuser as
    a non-root user. In this case fuser will report no access
    Because "man 2 fuser" doesn't contain anything: "No entry for fuser in section 2 of the manual"
    Regardless, when I try to run it without root, I get the following:
    $ fuser
    bash: fuser: command not found
    $
    Last edited by IsmAvatar2; 06-24-2009 at 11:28 AM.

  9. #9
    pwning noobs Zlatko's Avatar
    Join Date
    Jun 2009
    Location
    The Great White North
    Posts
    132
    Quote Originally Posted by IsmAvatar2 View Post

    Zlatko, are you referring to this?

    Because "man 2 fuser" doesn't contain anything: "No entry for fuser in section 2 of the manual"
    Regardless, when I try to run it without root, I get the following:
    $ fuser
    bash: fuser: command not found
    $
    Yes, what I meant in my edit was that there appears to be no fuser C function on linux. I assume you're using linux. I find that I can look at the /proc/pid/fd directory for a process I own, but not for those of other users. If the process creating for the file is owned by the same user as the process waiting for it, the one waiting should have no trouble reading the fd directory.

    If you have access to the source of the creator, then a file lock like lockf() is probably easier.
    Last edited by Zlatko; 06-24-2009 at 11:46 AM.

  10. #10
    Registered User
    Join Date
    Oct 2008
    Location
    TX
    Posts
    2,057
    Request for exclusive lock will fail if file descriptor is not opened with write access.
    Instead of trying to get info about a lock on the file try to set a lock on it.
    The return value of fcntl() indicates if a lock already exists on that file.
    Code:
    struct flock fl;
    int fd = open(fn, O_RDWR);
    fl.l_type   = F_WRLCK;  /* F_RDLCK, F_WRLCK, F_UNLCK */
    fl.l_whence = SEEK_SET; /* SEEK_SET, SEEK_CUR, SEEK_END */
    fl.l_start  = 0; /* Offset from l_whence */
    fl.l_len    = 0; /* length, 0 = to EOF */
    
    if (fcntl(fd, F_SETLK, &fl) == -1)
        printf("%s\n", errno == EACCESS||errno == EAGAIN ? "In use by another process":"another error occurred");
    else
        /* lock granted so do I/O on the file */

  11. #11
    Registered User
    Join Date
    Jan 2007
    Posts
    40
    Zlatko: I cannot edit the source of the process modifying these files. /proc/pid/fd sounds like a good idea, but how do I know which ones refer to the file that I'm interested in?

    itCbitC: In order to better test your idea, I created 2 processes.
    The first one opens a file for writing, waits 5 seconds, and then closes it and terminates.
    The second one simply attempts to put a write lock on the same file (as per your code), and reports the return value and any changes to the struct.

    I ran the first one, and during those 5 seconds, I ran the second one. The second one reports return value 0 (lock granted) and no change to the structure (mode: 1, everything else is 0).
    Unless I'm mistaken, this confirms that the file may be opened for writing without having a write lock on it, which further confirms that I need another way to detect if the file is opened for writing or not.

  12. #12
    pwning noobs Zlatko's Avatar
    Join Date
    Jun 2009
    Location
    The Great White North
    Posts
    132
    Quote Originally Posted by IsmAvatar2 View Post
    Zlatko: I cannot edit the source of the process modifying these files. /proc/pid/fd sounds like a good idea, but how do I know which ones refer to the file that I'm interested in?
    Well, you need to do some experimenting. Here is how I would start.
    The /proc/pid/fd directory has links to all the files in use by the program.
    Assuming you know the pid of the producer, try going through the /proc/pid/fd directory with the opendir/readdir/closedir functions. Each dirent, as returned by readdir, will have a filename. That filename might be the name of the link or it might be the name of the actual file being used. I'm not sure. However, try checking the dirent.d_ino number against the inode number of the file you're interested in and see if they match. You can get the inode of the file you're interested in with the stat function.

  13. #13
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,270
    Quote Originally Posted by IsmAvatar2 View Post
    My program is to scan a directory for new files, and then forward them along. However, I cannot forward a file until it is complete, and as far as I'm aware, the only way to tell if the file is complete is if no other process is accessing it.
    Whatever program is depositing the files in this directory is broken. The files should be created elsewhere, then atomically moved into the depository using the rename() syscall. At the moment the file becomes visible in the directory it is already complete.

    This is a fairly common problem which is solved in the above manner, not by using gross hacks.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  14. #14
    pwning noobs Zlatko's Avatar
    Join Date
    Jun 2009
    Location
    The Great White North
    Posts
    132
    Since it is not possible to modify the producer program, you may need to resort to unsavoury methods. Here is a quick and dirty program that might help. I'll leave the error checking and variable naming up to you.
    Code:
    #include <sys/types.h>
    #include <dirent.h>
    #include <sys/stat.h>
    #include <unistd.h>
    #include <stdio.h>
    #include <errno.h>
    #include <stdlib.h>
    
    int main(void)
    {
    	struct stat statbuf;
    	DIR* dir = opendir("/proc/14867/fd");
    	struct dirent* ent;
    
    	// foo.txt is the file that pid 14867 is producing
    	if (stat("/usr/home/foo.txt",  &statbuf) < 0)
    	{
    		perror("Cannot stat");
    		exit(1);
    	}
    
    	while ((ent = readdir(dir)) != NULL)
    	{
    		char buf[128];
    		struct stat statbuf2;
    		sprintf(buf, "/proc/14867/fd/%s", ent->d_name);
    		stat(buf, &statbuf2);
    
    		if (statbuf2.st_ino == statbuf.st_ino)
    		{
    			// This means that pid 14867 still has foo.txt open
    			fprintf(stderr, "YOU GOT IT\n");
    			break;
    		}
    	}
    
    	closedir(dir);
    }

  15. #15
    Registered User
    Join Date
    Jan 2007
    Posts
    40
    brewbuck: The program that is depositing the files is ftp. If you want to say FTP is broken, be my guest.

    Zlatko: Wouldn't it be more efficient to just get the d_ino from the dirent, rather than trying to stat it from the filename? Or is there a reason you chose to do it this way?

Page 1 of 2 12 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. C Delete opened file
    By gidion in forum C Programming
    Replies: 25
    Last Post: 12-04-2008, 10:35 AM
  2. Formatting a text file...
    By dagorsul in forum C Programming
    Replies: 12
    Last Post: 05-02-2008, 04:53 AM
  3. Need Help Fixing My C Program. Deals with File I/O
    By Matus in forum C Programming
    Replies: 7
    Last Post: 04-29-2008, 08:51 PM
  4. spell check in C using a dictionary file
    By goron350 in forum C Programming
    Replies: 10
    Last Post: 11-25-2004, 06:44 PM
  5. Replies: 4
    Last Post: 10-21-2003, 05:28 PM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21