Print a File on Multiple Processors

Hey guys,

I am using MPI to send data between multiple processors. I have one file that I run on about 12 processors, all of which send data back to a single file running on a main node.

I am trying to have the file which is running on the 12 processors spawn a text file with some information regarding the data and time sent to the main node.

However, in the one file running on 12 processors, the code I am using to open the file and print to it is not working. I have a strong feeling this is because the way the code is written now, it is trying to open 12 of the same file and there is a conflict.

I am trying to think of a way to print this information to a file without doing it the cheesy, obvious way, I.E - replicating the fprintf() statements 12 times and have 12 fopen()s, f close()s, etc.

Does anyone have any ideas?

Thanks.

Any chance you could describe how it doesn't work?

Also a snippet of the code attempting to print to the file would perhaps be useful.

--
Mats

Well absolutely no files are created. How I'm printing to the file is standard, I think:

Code:

---

int main(etc){

FILE *data_one, *data_two;

  while statement{

  <some various code>

    data_one = fopen("data_one", "w");

  MPI_Recv();

    fprintf(data_one, "info %f, time %f", info, time);

    data_two = fopen("data_two", "w");

  MPI_Send();

    fprintf(data_two, "info %f, time %f", info, time);

  fclose(data_one);
  fclose(data_two);

} end while

---

Something like that. The above code is replicated 12 times. I do not see data_one or data_two being created at all.

I think there is some sort of conflict creating these files. However, I'd expect them to be created and just over-written - but they are not created at all.

Use a DBMS instead a regular file. (FireBird, by example).
One procesor own the DBMS, and the others CONNECT to a Data Base via network or locally via localhost. Using SQL calls you can read/ from all the proccesors including the LOCALHOST that own DBMS.
If the file have no PRIMARY KEY, use a TimeStamp + proc ID por secuence the entries in tables.
In addition, a process in one processor can write as regular file (appending in file) using a select from table X where ID > last processed. Is possible in INSERT left by default the attribute that contains the ID. Also, you must include a field that constains the processor number, to identify the origin of data and generate non repeated primary keys on destination.

Best Regards.

Well, if you have the same name on the same filesystem, you probably need to make sure that you either:
1. Open the file in exclusive mode to prevent others from opening the file at the same time.
2. Use some interprocess locks to prevent the file from being created/written to at the same time.
3. Use specific filenames for each process, and let a main process merge them together at the end.

Using a database may also solve the problem, but I think for simple stuff like this, it would be a bit of a sledgehammer to crack a nut (but I'm almost certain it will solve the problem).

--
Mats

When all the processors are in same computer, obviusly use of lock tecnics is preferable, but if processors resides in differents computer, I think that the stuff is not a stuff and the db solution is the best.

Best Regards.
Oteros.

I'm assuming there's a common NFS for all 12 processors?

>> 3. Use specific filenames for each process, and let a main process merge them together at the end.
For example, incorporate the MPI_Comm_rank() into the filename. Keep in mind that the performance of the NFS may not be nearly as efficient as an MPI gather operation. In other words, it may take more bandwidth and time.

gg