Print a File on Multiple Processors

This is a discussion on Print a File on Multiple Processors within the Linux Programming forums, part of the Platform Specific Boards category; Hey guys, I am using MPI to send data between multiple processors. I have one file that I run on ...

  1. #1
    PhysicistTurnedProgrammer Cell's Avatar
    Join Date
    Jan 2009
    Location
    New Jersey
    Posts
    72

    Print a File on Multiple Processors

    Hey guys,

    I am using MPI to send data between multiple processors. I have one file that I run on about 12 processors, all of which send data back to a single file running on a main node.

    I am trying to have the file which is running on the 12 processors spawn a text file with some information regarding the data and time sent to the main node.

    However, in the one file running on 12 processors, the code I am using to open the file and print to it is not working. I have a strong feeling this is because the way the code is written now, it is trying to open 12 of the same file and there is a conflict.

    I am trying to think of a way to print this information to a file without doing it the cheesy, obvious way, I.E - replicating the fprintf() statements 12 times and have 12 fopen()s, f close()s, etc.

    Does anyone have any ideas?

    Thanks.

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Any chance you could describe how it doesn't work?

    Also a snippet of the code attempting to print to the file would perhaps be useful.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    PhysicistTurnedProgrammer Cell's Avatar
    Join Date
    Jan 2009
    Location
    New Jersey
    Posts
    72
    Well absolutely no files are created. How I'm printing to the file is standard, I think:

    Code:
    int main(etc){
    
    FILE *data_one, *data_two;
    
      while statement{
    
       <some various code>
    
        data_one = fopen("data_one", "w");
    
       MPI_Recv();
    
        fprintf(data_one, "info %f, time %f", info, time); 
    
        data_two = fopen("data_two", "w");
    
       MPI_Send();
    
        fprintf(data_two, "info %f, time %f", info, time); 
    
       fclose(data_one);
       fclose(data_two);
    
    } end while
    Something like that. The above code is replicated 12 times. I do not see data_one or data_two being created at all.

    I think there is some sort of conflict creating these files. However, I'd expect them to be created and just over-written - but they are not created at all.

  4. #4
    Registered User oteros's Avatar
    Join Date
    Mar 2009
    Location
    Madrid, Spain
    Posts
    7

    Use a DBMS

    Use a DBMS instead a regular file. (FireBird, by example).
    One procesor own the DBMS, and the others CONNECT to a Data Base via network or locally via localhost. Using SQL calls you can read/ from all the proccesors including the LOCALHOST that own DBMS.
    If the file have no PRIMARY KEY, use a TimeStamp + proc ID por secuence the entries in tables.
    In addition, a process in one processor can write as regular file (appending in file) using a select from table X where ID > last processed. Is possible in INSERT left by default the attribute that contains the ID. Also, you must include a field that constains the processor number, to identify the origin of data and generate non repeated primary keys on destination.

    Best Regards.

  5. #5
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Well, if you have the same name on the same filesystem, you probably need to make sure that you either:
    1. Open the file in exclusive mode to prevent others from opening the file at the same time.
    2. Use some interprocess locks to prevent the file from being created/written to at the same time.
    3. Use specific filenames for each process, and let a main process merge them together at the end.

    Using a database may also solve the problem, but I think for simple stuff like this, it would be a bit of a sledgehammer to crack a nut (but I'm almost certain it will solve the problem).

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  6. #6
    Registered User oteros's Avatar
    Join Date
    Mar 2009
    Location
    Madrid, Spain
    Posts
    7

    Depends

    When all the processors are in same computer, obviusly use of lock tecnics is preferable, but if processors resides in differents computer, I think that the stuff is not a stuff and the db solution is the best.

    Best Regards.
    Oteros.

  7. #7
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,669
    I'm assuming there's a common NFS for all 12 processors?

    >> 3. Use specific filenames for each process, and let a main process merge them together at the end.
    For example, incorporate the MPI_Comm_rank() into the filename. Keep in mind that the performance of the NFS may not be nearly as efficient as an MPI gather operation. In other words, it may take more bandwidth and time.

    gg

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Newbie homework help
    By fossage in forum C Programming
    Replies: 3
    Last Post: 04-30-2009, 04:27 PM
  2. File transfer- the file sometimes not full transferred
    By shu_fei86 in forum C# Programming
    Replies: 13
    Last Post: 03-13-2009, 12:44 PM
  3. Unknown Memory Leak in Init() Function
    By CodeHacker in forum Windows Programming
    Replies: 3
    Last Post: 07-09-2004, 09:54 AM
  4. archive format
    By Nor in forum A Brief History of Cprogramming.com
    Replies: 0
    Last Post: 08-05-2003, 07:01 PM
  5. Need a suggestion on a school project..
    By Screwz Luse in forum C Programming
    Replies: 5
    Last Post: 11-27-2001, 01:58 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21