C Board  

Go Back   C Board > General Programming Boards > C Programming

Reply
 
LinkBack Thread Tools Display Modes
Old 06-24-2009, 09:27 AM   #1
Registered User
 
Join Date: Jan 2007
Posts: 40
Question Check if file is opened for writing (unix)

My program is to scan a directory for new files, and then forward them along. However, I cannot forward a file until it is complete, and as far as I'm aware, the only way to tell if the file is complete is if no other process is accessing it.

I tried checking for write locks using fcntl, but either I'm using it wrong or the process doesn't put a write lock on the file.

So basically, I'm looking for something similar to fuser <file> or lsof <file>. What command/library am I looking for here? Don't say "exec", because there's no sense getting a full wheel when all you need is a spoke.

Thanks in advance,
-IsmAvatar

(If you want me to post the code I tried to check for locks, just say so, but I figured there's nothing further to be said there)

Last edited by IsmAvatar2; 06-24-2009 at 09:44 AM.
IsmAvatar2 is offline   Reply With Quote
Old 06-24-2009, 09:34 AM   #2
subminimalist
 
MK27's Avatar
 
Join Date: Jul 2008
Location: NYC
Posts: 3,946
Quote:
Originally Posted by IsmAvatar2 View Post
My program is to scan a directory for new files, and then forward them along. However, I cannot forward a file until it is complete, and as far as I'm aware, the only way to tell if the file is complete is if no other process is accessing it.
Since you do not need to open a file when scanning a directory, it is hard to tell what you mean here -- are you worried about other processes, not part of your program running on the system that could be accessing the file?
__________________

Accuracy and integrity mean nothing if you don't make it past the censors...PYTHAGORAS
MK27 is offline   Reply With Quote
Old 06-24-2009, 09:39 AM   #3
Registered User
 
Join Date: Jan 2007
Posts: 40
Yes. My program is to scan the directory for *new*, *completed* files created by another process.

Program 1 (not in my control) opens file "out1.txt" for writing.
P1 writes some bytes to file.
P1 closes file.
P1 opens file "out2.txt" for writing.
P1 writes some bytes to file, but doesn't finish.
Program 2 (my program) opens directory, and finds "out1.txt"
P2 finds out that the file is closed and completed, so transmits its information.
P2 next finds "out2.txt"
P2 finds out that the file is still open, so it ignores it.
P2 finds no more files, closes directory, and terminates (until it's run again)
P1 finishes writing "out2.txt" and closes it.
IsmAvatar2 is offline   Reply With Quote
Old 06-24-2009, 09:50 AM   #4
subminimalist
 
MK27's Avatar
 
Join Date: Jul 2008
Location: NYC
Posts: 3,946
Quote:
Originally Posted by IsmAvatar2 View Post
Yes. My program is to scan the directory for *new*, *completed* files created by another process.
I've never had to deal with this so I won't be much help -- my first thought was to try something with fcntl(), but I see you've already done that.

Most people would consider this a task for a shell script, where you could use lsof, etc "natively". My hunch is you will have to use a system/exec/popen to access the shell anyway.
__________________

Accuracy and integrity mean nothing if you don't make it past the censors...PYTHAGORAS
MK27 is offline   Reply With Quote
Old 06-24-2009, 10:03 AM   #5
Registered User
 
Join Date: Jan 2007
Posts: 40
My concern is that my program may not have access to lsof or fuser, as they oftentimes require root privileges. I was also hoping that I might just be able to use their lower level routines without having to use the whole program and parse the output. But of course skimming through lsof's source code can take hours.
IsmAvatar2 is offline   Reply With Quote
Old 06-24-2009, 10:13 AM   #6
Registered User
 
Join Date: Oct 2008
Location: TX
Posts: 1,262
Post the code snippet that calls fcntl(); P2 needs to open "out2.txt" for writing in exclusive mode and methinks you are trying to get a shared lock instead.
itCbitC is offline   Reply With Quote
Old 06-24-2009, 10:14 AM   #7
pwning noobs
 
Zlatko's Avatar
 
Join Date: Jun 2009
Location: The Great White North
Posts: 125
Hello.
You probably didn't see the bottom of the fuser man page. There's an fuser function in man section 2. I never used it myself.

Edit:
Sorry, there is no fuser function on linux, I was looking on my Tru64 system. On linux, check /proc/pid/fd for a list of files in use.

Last edited by Zlatko; 06-24-2009 at 10:25 AM.
Zlatko is offline   Reply With Quote
Old 06-24-2009, 10:25 AM   #8
Registered User
 
Join Date: Jan 2007
Posts: 40
itCbitC:
Code:
int main() {
 char *fn = "/tmp/test.txt"; //the test file
 FILE *F = fopen(fn,"w"); //this simulates another process writing to a file but not closing it yet

 //We now attempt to detect that the file is still open
 int fd = open(fn,O_RDONLY);
 struct flock fl;
  fl.l_type   = F_WRLCK;  /* F_RDLCK, F_WRLCK, F_UNLCK */
  fl.l_whence = SEEK_SET; /* SEEK_SET, SEEK_CUR, SEEK_END */
  fl.l_start  = 0; /* Offset from l_whence */
  fl.l_len    = 0; /* length, 0 = to EOF */
  fl.l_pid    = getpid();

 printf("%d %d %d %d %d\n",fl.l_type,fl.l_whence,fl.l_start,fl.l_len,fl.l_pid);
 //1 0 0 0 19865
 int ret = fcntl(fd,F_GETLK,&fl);
 printf("%d %d %d %d %d\n",fl.l_type,fl.l_whence,fl.l_start,fl.l_len,fl.l_pid);
 //2 0 0 0 19865

 sleep(3); //this gives me time to manually run fuser/lsof to observe that the file is indeed open

 //finalize
 fclose(F);
 return 0;
}
According to the fcntl documentation, if the lock is not detected, it will keep the struct the same but replace the mode with UNLCK, which is precisely what it does here.


Zlatko, are you referring to this?
Quote:
fuser cannot report on any processes that it doesn’t have permission to
look at the file descriptor table for. The most common time this prob-
lem occurs is when looking for TCP or UDP sockets when running fuser as
a non-root user. In this case fuser will report no access
Because "man 2 fuser" doesn't contain anything: "No entry for fuser in section 2 of the manual"
Regardless, when I try to run it without root, I get the following:
$ fuser
bash: fuser: command not found
$

Last edited by IsmAvatar2; 06-24-2009 at 10:28 AM.
IsmAvatar2 is offline   Reply With Quote
Old 06-24-2009, 10:39 AM   #9
pwning noobs
 
Zlatko's Avatar
 
Join Date: Jun 2009
Location: The Great White North
Posts: 125
Quote:
Originally Posted by IsmAvatar2 View Post

Zlatko, are you referring to this?

Because "man 2 fuser" doesn't contain anything: "No entry for fuser in section 2 of the manual"
Regardless, when I try to run it without root, I get the following:
$ fuser
bash: fuser: command not found
$
Yes, what I meant in my edit was that there appears to be no fuser C function on linux. I assume you're using linux. I find that I can look at the /proc/pid/fd directory for a process I own, but not for those of other users. If the process creating for the file is owned by the same user as the process waiting for it, the one waiting should have no trouble reading the fd directory.

If you have access to the source of the creator, then a file lock like lockf() is probably easier.

Last edited by Zlatko; 06-24-2009 at 10:46 AM.
Zlatko is offline   Reply With Quote
Old 06-24-2009, 10:56 AM   #10
Registered User
 
Join Date: Oct 2008
Location: TX
Posts: 1,262
Request for exclusive lock will fail if file descriptor is not opened with write access.
Instead of trying to get info about a lock on the file try to set a lock on it.
The return value of fcntl() indicates if a lock already exists on that file.
Code:
struct flock fl;
int fd = open(fn, O_RDWR);
fl.l_type   = F_WRLCK;  /* F_RDLCK, F_WRLCK, F_UNLCK */
fl.l_whence = SEEK_SET; /* SEEK_SET, SEEK_CUR, SEEK_END */
fl.l_start  = 0; /* Offset from l_whence */
fl.l_len    = 0; /* length, 0 = to EOF */

if (fcntl(fd, F_SETLK, &fl) == -1)
    printf("%s\n", errno == EACCESS||errno == EAGAIN ? "In use by another process":"another error occurred");
else
    /* lock granted so do I/O on the file */
itCbitC is offline   Reply With Quote
Old 06-24-2009, 11:24 AM   #11
Registered User
 
Join Date: Jan 2007
Posts: 40
Zlatko: I cannot edit the source of the process modifying these files. /proc/pid/fd sounds like a good idea, but how do I know which ones refer to the file that I'm interested in?

itCbitC: In order to better test your idea, I created 2 processes.
The first one opens a file for writing, waits 5 seconds, and then closes it and terminates.
The second one simply attempts to put a write lock on the same file (as per your code), and reports the return value and any changes to the struct.

I ran the first one, and during those 5 seconds, I ran the second one. The second one reports return value 0 (lock granted) and no change to the structure (mode: 1, everything else is 0).
Unless I'm mistaken, this confirms that the file may be opened for writing without having a write lock on it, which further confirms that I need another way to detect if the file is opened for writing or not.
IsmAvatar2 is offline   Reply With Quote
Old 06-24-2009, 12:14 PM   #12
pwning noobs
 
Zlatko's Avatar
 
Join Date: Jun 2009
Location: The Great White North
Posts: 125
Quote:
Originally Posted by IsmAvatar2 View Post
Zlatko: I cannot edit the source of the process modifying these files. /proc/pid/fd sounds like a good idea, but how do I know which ones refer to the file that I'm interested in?
Well, you need to do some experimenting. Here is how I would start.
The /proc/pid/fd directory has links to all the files in use by the program.
Assuming you know the pid of the producer, try going through the /proc/pid/fd directory with the opendir/readdir/closedir functions. Each dirent, as returned by readdir, will have a filename. That filename might be the name of the link or it might be the name of the actual file being used. I'm not sure. However, try checking the dirent.d_ino number against the inode number of the file you're interested in and see if they match. You can get the inode of the file you're interested in with the stat function.
Zlatko is offline   Reply With Quote
Old 06-24-2009, 12:35 PM   #13
Senior software engineer
 
brewbuck's Avatar
 
Join Date: Mar 2007
Location: Portland, OR
Posts: 5,381
Quote:
Originally Posted by IsmAvatar2 View Post
My program is to scan a directory for new files, and then forward them along. However, I cannot forward a file until it is complete, and as far as I'm aware, the only way to tell if the file is complete is if no other process is accessing it.
Whatever program is depositing the files in this directory is broken. The files should be created elsewhere, then atomically moved into the depository using the rename() syscall. At the moment the file becomes visible in the directory it is already complete.

This is a fairly common problem which is solved in the above manner, not by using gross hacks.
__________________
"Congratulations on your purchase. To begin using your quantum computer, set the power switch to both off and on simultaneously." -- raftpeople@slashdot
brewbuck is offline   Reply With Quote
Old 06-24-2009, 12:44 PM   #14
pwning noobs
 
Zlatko's Avatar
 
Join Date: Jun 2009
Location: The Great White North
Posts: 125
Since it is not possible to modify the producer program, you may need to resort to unsavoury methods. Here is a quick and dirty program that might help. I'll leave the error checking and variable naming up to you.
Code:
#include <sys/types.h>
#include <dirent.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>

int main(void)
{
	struct stat statbuf;
	DIR* dir = opendir("/proc/14867/fd");
	struct dirent* ent;

	// foo.txt is the file that pid 14867 is producing
	if (stat("/usr/home/foo.txt",  &statbuf) < 0)
	{
		perror("Cannot stat");
		exit(1);
	}

	while ((ent = readdir(dir)) != NULL)
	{
		char buf[128];
		struct stat statbuf2;
		sprintf(buf, "/proc/14867/fd/%s", ent->d_name);
		stat(buf, &statbuf2);

		if (statbuf2.st_ino == statbuf.st_ino)
		{
			// This means that pid 14867 still has foo.txt open
			fprintf(stderr, "YOU GOT IT\n");
			break;
		}
	}

	closedir(dir);
}
Zlatko is offline   Reply With Quote
Old 06-24-2009, 01:38 PM   #15
Registered User
 
Join Date: Jan 2007
Posts: 40
brewbuck: The program that is depositing the files is ftp. If you want to say FTP is broken, be my guest.

Zlatko: Wouldn't it be more efficient to just get the d_ino from the dirent, rather than trying to stat it from the filename? Or is there a reason you chose to do it this way?
IsmAvatar2 is offline   Reply With Quote
Reply

Tags
file, fuser, lock, lsof, open

Thread Tools
Display Modes

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
C Delete opened file gidion C Programming 25 12-04-2008 10:35 AM
Formatting a text file... dagorsul C Programming 12 05-02-2008 03:53 AM
Need Help Fixing My C Program. Deals with File I/O Matus C Programming 7 04-29-2008 07:51 PM
spell check in C using a dictionary file goron350 C Programming 10 11-25-2004 06:44 PM
C programming with Unix functions for manipulation of a virtual file system zagelle C Programming 4 10-21-2003 04:28 PM


All times are GMT -6. The time now is 09:32 AM.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.3.0 RC2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22