Thread: Reading data from volumes

  1. #1
    Registered User
    Join Date
    Mar 2011
    Posts
    216

    Reading data from volumes

    Hi,

    I'm having a problem switching to another file once a certain amount has been read. Once volSize has been read, I want to open the next file (in a volume set) and continue reading data from it. Here is the code.

    Code:
    void Volume(char *fileToOpen)
    {
        FILE* fp;
        FILE* out;
        
        size_t result;
        size_t totalRead = 0;
        size_t bytesToRead = 0;
        size_t GlobRead = 0;
        
        int numFiles;
        int volume = 1;
        int volSize;
        int i;
        
        int finishVol = 0;
        
        int bufsize = 4096;
        
        char buffer[bufsize]; // 4 kb
        
        char volName[256];
        
        sprintf( volName, "%i-%s", volume, fileToOpen);
        
        fp = fopen(volName, "rb");
        if(fp == NULL) Error("No file handle to bin");
        
        // Read in general data
        ReadInfo(fp, &numFiles, &volSize);
        
        printf("Number of files: %i\nVolume: %i MB\n\n", numFiles, volSize/1024/1024);
    
        struct file file[numFiles];
        
        // Read in file specific data for all files
        for(i = 0; i < numFiles; i++)
              ReadHeader(fp, &file[i].fileName, &file[i].fileNameLen, &file[i].fileSize);
        
        printf("Gathered all file information\n\n");
        
        /* Now begin writing out each file */
        
        for(i = 0; i < numFiles; i++)
        {
              out = fopen(file[i].fileName, "wb");
              if(out == NULL) Error("No file handle");
              
              printf("File %i: %.2f MB\n", i + 1, (double)file[i].fileSize / 1024 / 1024);
              
              totalRead = 0;
              bytesToRead = 0;
              
              do
              {
                    bytesToRead = file[i].fileSize - totalRead;
                    if(bytesToRead > sizeof buffer)
                       bytesToRead = sizeof buffer;
                       
                    result = fread(buffer, 1, bytesToRead, fp);
                    if(result <= 0) Error("No bytes read\n");
                    
                    totalRead += result;
                    GlobRead += totalRead;
                    
                    WriteChunkBin(out, buffer, result);
                    
                    if(GlobRead + bufsize >= volSize) // Code to switch volume
                    {
                            finishVol = (GlobRead + bufsize) - volSize;
                       
                            result = fread(buffer, 1, finishVol, fp);
                            if(result <= 0) Error("No bytes read\n");
                    
                            totalRead += result;
                    
                            WriteChunkBin(out, buffer, result);
                            
                            fclose(fp);
                            free(volName);
                            
                            finishVol = 0;
                            GlobRead = 0;
                            volume++;
                            
                            sprintf( volName, "%i-%s", volume, fileToOpen);
                            
                            fp = fopen(volName, "rb");
                            if(fp == NULL)  Error("No file handle on volume\n");
                            
                            totalRead = 0;
                            
                    }
                    
              }
              while(totalRead < file[i].fileSize);
              
              printf("Wrote file %i: %s\n\n", i + 1, file[i].fileName);
              
              // Repeat for all files
              fclose(out);
              
        }
        
        fclose(fp);
        
        printf("Success!");
        
    }
    For example, let's say the volume size is 50 mb. When 50 mb has been read in total, regardless of however many files have been read, it should switch to the next file and continue reading the data. This should continue until all the data has been read from a multi-volume set.

    Thank you for you time and help.

    EDIT: Here is the code for splitting files into volumes in case it matters.

    Code:
        out = fopen(outName, "ab");
        if(out == NULL)  Error("No file handle\n");
        
        for(i = 0; i < numFiles; i++)
        {
              // Open file
              fp = fopen(argv[i + minArg - 1], "rb");
              if(fp == NULL)  Error("Error opening file\n");
              
              fileSize = GetFileSize(fp);
              
              printf("File %i: %.2f MB\n", i + 1, (double)fileSize / 1024 / 1024);
              
              do
              {
                    result = fread(buffer, 1, sizeof buffer, fp);
                    if(result <= 0) break;
                    
                    WriteChunkBin(out, buffer, result);
                    
                    written = written + (int)result;
                    
                    if(written >= volSize)
                    {
                               fclose(out);
                               free(outName);
                               
                               volume++;
                               
                               sprintf( outName, "%i-%s", volume, argv[2]);
                               
                               out = fopen(outName, "wb");
                               if(out == NULL)  Error("No file handle\n");
                               
                               written = 0;
                               
                    }
                    
              }
              while(result == sizeof buffer);
              
              printf("Wrote file %i: %s\n\n", i + 1, argv[i + minArg - 1]);
              
              // Repeat for all files
              fclose(fp);
              
        }
        
        fclose(out);
    Last edited by binks; 05-02-2012 at 05:24 PM.

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    > totalRead += result;
    > GlobRead += totalRead;
    Why are you adding totalRead each time here?

    > free(volName);
    This is just plain stupid.
    It's a local array, not something you malloc'ed.

    Have you ever tried to use a debugger?
    Because now is a really good time to start.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    I doesn't crash though, it simply stops 30 mb into the next file and exits with 'No bytes read' error. Judging by the fact that my first file is 20.4 mb, and the second file is 163 mb (which only 30 mb approx. is read from) then it's safe to say the 'No bytes read' error means it's at the end of the first volume, which is 50 mb.

    Is my code to transition to the next volume good, or are there flaws?

    totalRead only keeps track of the total read from a single file. GlobRead is supposed to keep track of everything read, and once the limit has been reached (in this case 50 mb) then open the next volume, reset GlobRead to 0, and begin the process again.

  4. #4
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    I told you some of the things that were wrong.

    If you're just going to say "well it doesn't crash" without fixing anything, then you're on your own.

    Use a debugger, watch the values of variables and figure it out.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  5. #5
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    I'm going to debug it once I get home. I was just wondering if the code I had written for changing to the next volume was correct or not.

  6. #6
    - - - - - - - - oogabooga's Avatar
    Join Date
    Jan 2008
    Posts
    2,808
    There's so much wrong with both the pack and unpack routines that I don't feel like describing it in words, so here's some untested code for the pack routine.

    It's probably worth picking some better variable names at this point to make things clearer.
    Code:
    FILE  * fArchive        = NULL;
    FILE  * fData;
    size_t  fileBytesLeft;
    size_t  volBytesLeft    = 0;   // force new volume first time
    size_t  bytesToRead;           // number of bytes to attempt to read
    size_t  bytesRead;             // number of bytes actually read
    size_t  headerSize;            // TODO: set this to the size of the header
    int     volume          = 0;
    In the code below, min3 is assumed to return the minimum of the 3 size_t inputs. I switched from counting up to counting down for the number of bytes left in a file or volume. I'm assuming that freopen can be called with fArchive as NULL the first time (I think that's the case).
    Code:
        for(i = 0; i < numFiles; i++)
        {
            // Open data file
            fData = fopen(argv[i + minArg - 1], "rb");
            if(fData == NULL)  Error("Error opening file\n");
    
            printf("File %i: %.2f MB\n", i + 1, (double)fileSize / 1024 / 1024);
    
            fileBytesLeft = file[i].fileSize;
    
            do
            {
                if(nVolBytesLeft == 0)
                {
                    volume++;
                    sprintf(archiveName, "%s-%d", argv[2], volume);
    
                    fArchive = freopen(archiveName, "wb", fArchive);
                    if(fArchive == NULL)  Error("No file handle\n");
    
                    volBytesLeft = volSize - headerSize;
                    headerSize = 0; // no header after the first one
                }
    
                bytesToRead = min3(sizeof buffer, fileBytesLeft, volBytesLeft);
    
                bytesRead = fread(buffer, 1, bytesToRead, fData);
                if(bytesRead < bytesToRead) Error("fread failure");
    
                WriteChunkBin(fArchive, buffer, bytesRead);
    
                fileBytesLeft -= bytesRead;
                volBytesLeft -= bytesRead;
    
            }while(fileBytesLeft > 0);
    
            printf("Wrote file %i: %s\n\n", i + 1, argv[i + minArg - 1]);
    
            fclose(fData);
        }
    
        fclose(fArchive);
    Try to get that code to work, study it, and use the same ideas to implement the unpack routine.
    The cost of software maintenance increases with the square of the programmer's creativity. - Robert D. Bliss

  7. #7
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    For getting the header size, does this not work?

    Code:
    headerSize = sizeof numFiles + sizeof volSize + (sizeof file * numFiles);
    I have added your code to writing the volumes, and besides the headerSize, everything works well. For extracting of the archive, this code will not work and exits with an fread error. What have I missed?

    Code:
    int min3(size_t one, size_t two, size_t three)
    {
        if(one < two && three)
               return one;
        if(two < one && three)
               return two;
        if(three < one && two)
               return three;
               
    }
    
    void Volume(char *fileToOpen)
    {
        FILE* fArchive = NULL;
        FILE* fData = NULL;
        
        size_t result;
        size_t fileBytesLeft;
        size_t volBytesLeft = 0;
        size_t bytesRead = 0;
        size_t bytesToRead = 0;
        
        int numFiles;
        int volume = 0;
        int volSize;
        int i;
        
        
        int bufsize = 4096;
        
        char buffer[bufsize]; // 4 kb
        
        char archiveName[256];
        
        sprintf( archiveName, "%i-%s", volume, fileToOpen);
        
        fArchive = fopen(archiveName, "rb");
        if(fArchive == NULL) Error("No file handle to bin");
        
        // Read in general data
        ReadInfo(fArchive, &numFiles, &volSize);
        
        volBytesLeft = volSize;
        
        printf("Number of files: %i\nVolume Size: %i MB\n\n", numFiles, volSize/1024/1024);
    
        struct file file[numFiles];
        
        // Read in file specific data for all files
        for(i = 0; i < numFiles; i++)
              ReadHeader(fArchive, &file[i].fileName, &file[i].fileNameLen, &file[i].fileSize);
        
        printf("Gathered all file information\n\n");
        
        /* Now begin writing out each file */
        
        for(i = 0; i < numFiles; i++) 
        {     
              // Open data file     
              fData = fopen(file[i].fileName, "wb");     
              if(fData == NULL)  Error("Error opening file\n");      
              
              printf("File %i: %.2f MB\n", i + 1, (double)file[i].fileSize / 1024 / 1024);      
              
              fileBytesLeft = file[i].fileSize;      
              
              do    
              {         
                        if(volBytesLeft == 0)         
                        {             
                                      volume++;             
                                      sprintf(archiveName, "%i-%s", volume, fileToOpen);              
                                      
                                      fArchive = freopen(archiveName, "rb", fArchive);             
                                      if(fArchive == NULL)  Error("No file handle\n");              
                                      
                                      volBytesLeft = volSize; /* - headerSize*/            
                                      //headerSize = 0; // no header after the first one         
                        }          
                        
                        bytesToRead = min3(sizeof buffer, fileBytesLeft, volBytesLeft);          
                        
                        bytesRead = fread(buffer, 1, bytesToRead, fData);         
                        if(bytesRead < bytesToRead) Error("fread failure");          
                        
                        WriteChunkBin(fData, buffer, bytesRead);          
                        
                        fileBytesLeft -= bytesRead;         
                        volBytesLeft -= bytesRead;      
                        
              }
              while(fileBytesLeft > 0);      
              
              printf("Wrote file %i: %s\n\n", i + 1, file[i].fileName);      
              
              fclose(fData); 
        }  
        
        fclose(fArchive);
        
        printf("Success!");
        
    }
    Have I written the min3 code correctly?
    Last edited by binks; 05-03-2012 at 07:13 PM.

  8. #8
    - - - - - - - - oogabooga's Avatar
    Join Date
    Jan 2008
    Posts
    2,808
    If I remember correctly, struct file contains a couple of ints and a char*. In that case, simply using sizeof file is not going to work, and is not really correct anyway do to possible (though unlikely in this case) padding. Better to sum up the individual members:
    Code:
    headerSize = sizeof numFiles + sizeof volSize
               + numFile * (sizeof file[0].fileSize + sizeof file[0].nameLen);
    for (i = 0; i < numFiles; i++)
        headerSize += strlen(fileName[i].fileName)
    (I may have gotten some of the member names wrong.)

    And your min3 is wrong. You can't say
    if (one < two && three)
    instead it should be
    if (one < two && one < three)
    Or better yet
    Code:
    int min3(size_t a, size_t b, size_t c) {
        if (a < b)
            return a < c ? a : c;
        else
            return b < c ? b : c;
    }
    The cost of software maintenance increases with the square of the programmer's creativity. - Robert D. Bliss

  9. #9
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    Using this code to remove the header, I still get extra bytes in the first arcive (0-pack.bin). It's not exactly 50 mb, it's 44 bytes larger.

    Reading data from volumes-file-jpg

    Code:
        headerSize = sizeof numFiles + sizeof volSize + 
                     numFiles * (sizeof file[0].fileSize + sizeof file[0].fileNameLen); 
                     
        for (i = 0; i < numFiles; i++)    
            headerSize += strlen(argv[i + minArg - 1]);
    
    // code in main loop
        for(i = 0; i < numFiles; i++) 
        {     
              // Open data file     
              fData = fopen(argv[i + minArg - 1], "rb");     
              if(fData == NULL)  Error("Error opening file\n");      
              
              printf("File %i: %.2f MB\n", i + 1, (double)file[i].fileSize / 1024 / 1024);      
              
              fileBytesLeft = file[i].fileSize;      
              
              do    
              {         
                        if(volBytesLeft == 0)         
                        {             
                                      volume++;             
                                      sprintf(archiveName, "%i-%s", volume, argv[2]);              
                                      
                                      fArchive = freopen(archiveName, "wb", fArchive);             
                                      if(fArchive == NULL)  Error("No file handle\n");              
                                      
                                      volBytesLeft = volSize - headerSize;           
                                      headerSize = 0; // no header after the first one         
                        }          
                        
                        bytesToRead = min3(sizeof buffer, fileBytesLeft, volBytesLeft);          
                        
                        bytesRead = fread(buffer, 1, bytesToRead, fData);         
                        if(bytesRead < bytesToRead) Error("fread failure");          
                        
                        WriteChunkBin(fArchive, buffer, bytesRead);          
                        
                        fileBytesLeft -= bytesRead;         
                        volBytesLeft -= bytesRead;      
                        
              }
              while(fileBytesLeft > 0);      
              
              printf("Wrote file %i: %s\n\n", i + 1, argv[i + minArg - 1]);      
              
              fclose(fData); 
        }
    Is it doing something wrong?

  10. #10
    - - - - - - - - oogabooga's Avatar
    Join Date
    Jan 2008
    Posts
    2,808
    Post the whole program so I can try it out.
    The cost of software maintenance increases with the square of the programmer's creativity. - Robert D. Bliss

  11. #11
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    This is for packer.exe

    Code:
    int min3(size_t a, size_t b, size_t c) 
    {     
          if (a < b)         
             return a < c ? a : c;     
          else        
             return b < c ? b : c; 
    }
    
    void Volume( int argc, char *argv[])
    {
        FILE* fArchive = NULL;
        FILE* fData  = NULL;
        
        size_t result;
        
        size_t  fileBytesLeft; 
        size_t  volBytesLeft    = 0;   // force new volume first time 
        size_t  bytesToRead;           // number of bytes to attempt to read 
        size_t  bytesRead;             // number of bytes actually read 
        size_t  headerSize = 0;
        
        int minArg = 4; // Counts filename, 0, pack.bin, filetopack.txt
        int numFiles = argc - minArg + 1; 
        int fileSize;
        int i;
        
        int volume = 0;
        int volSize = atoi(argv[1]) * 1024 * 1024;
        volBytesLeft = volSize;
    
        int bufsize = 4096;
        
        char buffer[bufsize]; // 4 kb
        
        char archiveName[256];
        
        struct file file[numFiles]; 
        
        for(i = 0; i < numFiles; i++)
        {
              // Gather file info
              GetFileInfo(argv[i + minArg - 1], &file[i].fileNameLen, &file[i].fileSize);
              
        }
        
        headerSize = sizeof numFiles + sizeof volSize + 
                     numFiles * (sizeof file[0].fileSize + sizeof file[0].fileNameLen); 
                     
        for (i = 0; i < numFiles; i++)    
            headerSize += strlen(argv[i + minArg - 1]);
        
        printf("Gathered all file information -> Header Size: %i\n\n", headerSize);
        
        sprintf( archiveName, "%i-%s", volume, argv[2]);
        
        // Open outfile for writing
        fArchive = fopen(archiveName, "wb");
        if(fArchive == NULL)  Error("No file handle\n");
        
        // Write number of files
        if( fwrite(&numFiles, sizeof(numFiles), 1, fArchive) != 1)  Error("Error writing number of files to bin\n");
        
        // Write volume size to bin
        if( fwrite(&volSize, sizeof(volSize), 1, fArchive) != 1)  Error("Error writing volume size to bin\n");
        
        // Write the header of information regarding the files
        for(i = 0; i < numFiles; i++)
        {
              WriteHeader(fArchive, argv[i + minArg - 1], file[i].fileNameLen, file[i].fileSize);
              
        }
        fclose(fArchive);
        
        printf("Wrote all file header information\n\n");
        
        fArchive = fopen(archiveName, "ab");
        if(fArchive == NULL)  Error("No file handle\n");
        
        
        for(i = 0; i < numFiles; i++) 
        {     
              // Open data file     
              fData = fopen(argv[i + minArg - 1], "rb");     
              if(fData == NULL)  Error("Error opening file\n");      
              
              printf("File %i: %.2f MB\n", i + 1, (double)file[i].fileSize / 1024 / 1024);      
              
              fileBytesLeft = file[i].fileSize;      
              
              do    
              {         
                        if(volBytesLeft == 0)         
                        {             
                                      volume++;             
                                      sprintf(archiveName, "%i-%s", volume, argv[2]);              
                                      
                                      fArchive = freopen(archiveName, "wb", fArchive);             
                                      if(fArchive == NULL)  Error("No file handle\n");              
                                      
                                      volBytesLeft = volSize - headerSize; /* - headerSize*/            
                                      headerSize = 0; // no header after the first one         
                        }          
                        
                        bytesToRead = min3(sizeof buffer, fileBytesLeft, volBytesLeft);          
                        
                        bytesRead = fread(buffer, 1, bytesToRead, fData);         
                        if(bytesRead < bytesToRead) Error("fread failure");          
                        
                        WriteChunkBin(fArchive, buffer, bytesRead);          
                        
                        fileBytesLeft -= bytesRead;         
                        volBytesLeft -= bytesRead;      
                        
              }
              while(fileBytesLeft > 0);      
              
              printf("Wrote file %i: %s\n\n", i + 1, argv[i + minArg - 1]);      
              
              fclose(fData); 
        }  
        
        fclose(fArchive);
    
        printf("Success!");
        
    }

  12. #12
    - - - - - - - - oogabooga's Avatar
    Join Date
    Jan 2008
    Posts
    2,808
    I was away for the weekend.

    The problem seems to be that you've set volBytesLeft = volSize;
    but it should be set to volBytesLeft = volSize - headerSize; (after the header size is known).

    Closing the archive and opening it again (with append mode) is unnecessary. Take those lines out.

    I may do a rewrite of your code, with some variable name changes and perhaps one or two extra functions. But I've been assuming that this isn't homework, so I've got to ask: is it homework?

    Also, since you didn't post the WHOLE program, I couldn't actually run it.
    The cost of software maintenance increases with the square of the programmer's creativity. - Robert D. Bliss

  13. #13
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    No, this isn't homework. I had actually just commented out - headersize; because it was not working with it.

  14. #14
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    > No, this isn't homework.
    Your job perhaps?
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  15. #15
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    Not that either, supposively I intended to do it "for fun" but it wasn't.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Read size of data array when reading .txt data
    By Taquito in forum C Programming
    Replies: 13
    Last Post: 04-29-2007, 01:52 AM
  2. Replies: 1
    Last Post: 10-22-2005, 05:28 AM
  3. windows volumes
    By valis in forum Windows Programming
    Replies: 0
    Last Post: 08-18-2005, 11:23 AM
  4. Replies: 2
    Last Post: 06-16-2005, 10:03 AM
  5. volumes and areas
    By oscuro in forum C++ Programming
    Replies: 1
    Last Post: 04-02-2002, 10:27 PM