Difficulty with program which has to open hundreds of files at a time.

This is a discussion on Difficulty with program which has to open hundreds of files at a time. within the C++ Programming forums, part of the General Programming Boards category; Hello, at the moment I am writing a program which needs to open, close and modify many different files at ...

  1. #1
    Registered User Swarvy's Avatar
    Join Date
    Apr 2008
    Location
    United Kingdom
    Posts
    195

    Difficulty with program which has to open hundreds of files at a time.

    Hello, at the moment I am writing a program which needs to open, close and modify many different files at the same time (typically over 100 at a given time). Because of this requirement of the project, I decided to make a class which contains a vector of pointers to the different fstream instances for the files. That way I have a simple and 'clean' way to handle this many files. The fact that I am using vectors also means I can easily add new fstream instances (open new files) just by using the push_back method etc.

    For some reason, I'm having a bit of difficulty getting it to open a new file (it shouldn't be this hard). The program compiles it just doesn't do what I want it to, and I can't work out where the problem is.

    Code:
    #include <fstream>
    #include <vector>
    #include <string.h>
    
    using namespace std;
    
    class CFileManage
    {
          private:
                  CFileManage();
                  virtual ~CFileManage();
                  
                  vector<fstream *> m_file;
                  static CFileManage * hcInstance;
                  
          public:
                 /* Make sure only one instance of this class is generated */
                 static CFileManage * GetClassInstance();
                 static void GenerateClassInstance();
                 
                 void DeleteClassInstance();
                 
                 /* Make use of the i/o filestream vectors */
                 bool CreateNewFileInstance(char * filename);
                 void CreateNewFileInstance(string filename);
                 
                 void CloseFileInstance(unsigned int vec_idx);
                 void CloseAllFileInstances();
                 
                 unsigned int GetNumberOfFileInstances();
                 
                 fstream * GetFileInstance(unsigned int vec_idx);
    };
    Code:
    bool CFileManage::CreateNewFileInstance(char * filename)
    {
         m_file.push_back( new fstream );
    
         m_file.back()->open(filename);
         
         if( m_file.back()->is_open() )
         {
             return false;
         }
         
         return true;
    }
    
    void CFileManage::CloseAllFileInstances()
    {
         int idx;
         
         for(idx=0; idx < m_file.size(); idx++)
         {
                    m_file[idx]->close();
                    delete m_file[idx];
         }
         
         m_file.clear();
         
         return;
    }
    For some reason, the code compiles, but it doesn't work as expected when it comes to making a new file.

    At a later date I plan to change the CreateFile Methods so that it allows support for binary files aswell as different modes, e.g. app, ate, trunc etc.
    Last edited by Swarvy; 08-23-2009 at 09:25 AM.

  2. #2
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,761
    Many systems limit the number of open files a process is allowed to have open at one time.
    "hundreds" is pushing (or exceeded) such limits.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #3
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Posts
    22,915
    Also, shouldn't you make sure that the fstream is opened successfully before pushing it into the vector? What good will it do in the vector if it isn't actually open?
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  4. #4
    Dae
    Dae is offline
    Deprecated Dae's Avatar
    Join Date
    Oct 2004
    Location
    Canada
    Posts
    1,034
    If your program is also creating these files, you should think of archiving them (.pak, .zip, .tar, .7z, etc.) into one file. It's faster and you shouldn't have this problem.
    Warning: Have doubt in anything I post.

    GCC 4.5, Boost 1.40, Code::Blocks 8.02, Ubuntu 9.10 010001000110000101100101

  5. #5
    Super Moderator VirtualAce's Avatar
    Join Date
    Aug 2001
    Posts
    9,598
    I'm going to echo what Salem stated b/c it was my understanding that file handles were precious resources in most operating systems. I certainly doubt the OS would let you open hundreds of files at one time. You will have to open a few at a time, do your operations on them, close them, wash, rinse, repeat for the number of files you need to work on.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Program Plan
    By Programmer_P in forum C++ Programming
    Replies: 0
    Last Post: 05-11-2009, 02:42 AM
  2. Get program to copy itself into program files, then start on startup.
    By guitarist809 in forum Windows Programming
    Replies: 6
    Last Post: 03-03-2008, 09:42 AM
  3. help with basic program
    By JOlszewski in forum C Programming
    Replies: 3
    Last Post: 02-01-2006, 04:19 PM
  4. Replies: 26
    Last Post: 06-15-2005, 03:38 PM
  5. Multi - File Program not finding files
    By johnh444 in forum C++ Programming
    Replies: 2
    Last Post: 07-03-2004, 02:48 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21