Thread: file - memory operations

  1. #1
    Registered User
    Join Date
    Jun 2007
    Posts
    11

    file - memory operations

    Hello to all!

    I've got a question. You surely all know about packed files, very often used in games. In these big files there are many others, smaller files (images, sounds etc.). So the question is, how can I read the content into the memory and then - using pointers - access these different files?
    For example we have data.dat with 3 files in it (file1.bmp, file2.bmp, file3.bmp). I open data.dat like that:
    Code:
    ifstream file("data.dat", ios::binary);
    Now I have some pointers, like
    Code:
    // This structure is from Allegro if someone happen to know that
    BITMAP** bmp = new BITMAP*[3]; 
    //Normally it's used like that:
    BITMAP* img = load_bitmap("file.bmp", NULL);
    What I would like to do now is to let all the 3 pointers to show to the beginning of each file in memory, without creating any temporary files. I know that it works pretty fine for integers for example or some user defined structures using
    Code:
    int* x = 0;
    myStruct* s = new myStruct;
    file.read((char*)x, 4);
    file.read((char*)s, sizeof(myStruct));
    ...
    delete x;
    delete s;
    file.close();
    But how can I access bigger files, about 100 and more KBytes? I want to have bmp[0], bmp[1], bmp[2] be ready to use with file1.bmp, file2.bmp, file3.bmp, just if I would write:
    Code:
    // BITMAP *load_bitmap(const char *filename, RGB *pal); 
    BITMAP* bmp1 = load_bitmap("file1.bmp", NULL);
    BITMAP* bmp2 = load_bitmap("file2.bmp", NULL);
    BITMAP* bmp3 = load_bitmap("file3.bmp", NULL);
    
    // I tried it this way but it doesn't work
    for(int i = 0; i < 3; i++)
        file.read((char*)bmp[i], 12345); // assuming each bitmap has the same size
    
    /* I tried out different possibilities, don't remember them all but sometimes
    I got compiler errors, another time the program crashes like it doe's because
    of memory access violation ...*/
    Does anyone have any suggestions? I would appreciate any help I can get!
    Thanks in advance

  2. #2
    Deathray Engineer MacGyver's Avatar
    Join Date
    Mar 2007
    Posts
    3,210
    "Packed files" are usually custom formats specific to the game. They may not store bitmaps at all. You have to know what you're dealing with. I think you're assuming too much about how bitmaps might be stored in such a file.

    Code:
    int* x = 0;
    myStruct* s = new myStruct;
    file.read((char*)x, 4);
    file.read((char*)s, sizeof(myStruct));
    ...
    delete x;
    delete s;
    file.close();
    That's one way to attempt a crash.

  3. #3
    Ethernal Noob
    Join Date
    Nov 2001
    Posts
    1,901
    Do you have the specification of the file. usually non-raw files have headers which indicate a way to read certain file-types. As for having a pointer to a particular point (byte) in a file I'm not sure you can without say creating three ifstreams and having them seek to the particular point in a file and working from there. But then again how do you know where the first file ends and the next begins?

  4. #4
    Registered User
    Join Date
    Jun 2007
    Posts
    11
    Quote Originally Posted by MacGyver
    "Packed files" are usually custom formats specific to the game. They may not store bitmaps at all. You have to know what you're dealing with. I think you're assuming too much about how bitmaps might be stored in such a file.
    I know about custom format in games .... and there are a lot of examples where bitmaps are stored as they are, without compression or cryption ...
    And in this example it will be my own format, where all the bitmaps will be right after each other ....

    Quote Originally Posted by indigo0086
    usually non-raw files have headers which indicate a way to read certain file-types ...
    But then again how do you know where the first file ends and the next begins?
    Yes, also my packed file will have a header and I know where every file is and how long it is, that's everything not the point to discuss. The point is how I can get the files associated with the pointers? Before I only worked with some structures, allocated memory and pushed data to there. But here ...

  5. #5
    Deathray Engineer MacGyver's Avatar
    Join Date
    Mar 2007
    Posts
    3,210
    If it's your file format, why are you asking us how to read from it?

  6. #6
    Ethernal Noob
    Join Date
    Nov 2001
    Posts
    1,901
    I don't know, It would be better if your file had some sort of header to have a specifier indicating the size of each "file", and you could just ignore the file you don't want. A file isn't in memory so in order for you to do it you'd need to load the file into memory, otherwise you'd just have to skip unwanted information through the stream.

  7. #7
    Registered User
    Join Date
    Jun 2007
    Posts
    11
    Quote Originally Posted by MacGyver View Post
    If it's your file format, why are you asking us how to read from it?
    The problem doesn't lie in format, in fact the format doesn't play a role. Maybe I should facilitate the question ...

    I have a BIGFILE with DATA1, DATA2 and DATA3 (just a weird sum of bits and bytes with a lenght = 10kb every). I have got 3 pointers
    Code:
    void* p1;
    void* p2;
    void* p3;
    The jackpot-question is: How to make p1 show to DATA1, p2 to DATA2 and p3 to DATA3?

  8. #8
    Registered User
    Join Date
    Jun 2007
    Posts
    11
    Quote Originally Posted by indigo0086 View Post
    A file isn't in memory so in order for you to do it you'd need to load the file into memory
    That's the question, how can I do that?

  9. #9
    Ethernal Noob
    Join Date
    Nov 2001
    Posts
    1,901
    You could load the file into an unsigned char array or vector and use pointer/iterator arithmetic to point it to each file location specified by the header. I don't know exactly how streams are handled in memory but if the file is large you're better off using ifstream::ignore() function for doing that. But if you truly need 3 pointers working simultaneously I'd go with the array option.

    Code:
    fstream file(filename, ios::binary);
    vector<unsigned char> data;
    unsigned char tmp;
    
    while(file >> tmp)
        data.push_back(tmp);

  10. #10
    Registered User
    Join Date
    Jun 2007
    Posts
    11
    Quote Originally Posted by indigo0086 View Post
    You could load the file into an unsigned char array or vector and use pointer/iterator arithmetic to point it to each file location specified by the header. I don't know exactly how streams are handled in memory but if the file is large you're better off using ifstream::ignore() function for doing that. But if you truly need 3 pointers working simultaneously I'd go with the array option.

    Code:
    fstream file(filename, ios::binary);
    vector<unsigned char> data;
    unsigned char tmp;
    
    while(file >> tmp)
        data.push_back(tmp);
    Ok, that's sound good, I will try it. Thanks.

  11. #11
    Ethernal Noob
    Join Date
    Nov 2001
    Posts
    1,901
    The thing with that is from the getgo it reads everything, the header, the files, etc. If you have data longer than a byte like a short or a float you have to compile them yourself

    Code:
    short int num = 0x0000;
    unsigned char a = data[index1];
    unsigned char b = data[index2];
    
    num = num | a;
    num = (num << 8) | b;
    Though if it's intel ordered/litte endian, you have to switch a and b. If the number is longer you have to add more steps. fread I believe by default reads it little endian.

  12. #12
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,895
    A very, very inefficient way of reading a file. Since the file is binary, a better way would be (unsure about the exact parameters):
    Code:
    file.seekg(ios_base::end, 0);
    size_t size = file.tellg();
    data.resize(size);
    file.seekg(ios_base::beg, 0);
    file.read(&data[0], size);
    The basic rule is, the more you read at once, the better.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  13. #13
    Ethernal Noob
    Join Date
    Nov 2001
    Posts
    1,901
    Thanks for that tip CornedBee, it helped me with another program I was working on that did pretty much the same thing with uneccesary operations. I was wondering if I have a struct with varying sizes like

    Code:
    struct a
    {
      char data1;
      int data2;
      float data3;
      char data4;
    };
    And I know that in the file the data is sequential and is to be read to the struct in that order, is there a way to read the struct all at once?

  14. #14
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,895
    Is it in binary? Then you could just do a read() too.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  15. #15
    Ethernal Noob
    Join Date
    Nov 2001
    Posts
    1,901
    yeah, it's a binary file.

    so it would be just
    Code:
    file.read((char*)&a, sizeof(a))

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. A development process
    By Noir in forum C Programming
    Replies: 37
    Last Post: 07-10-2011, 10:39 PM
  2. File Writing Problem
    By polskash in forum C Programming
    Replies: 3
    Last Post: 02-13-2009, 10:47 AM
  3. Game Pointer Trouble?
    By Drahcir in forum C Programming
    Replies: 8
    Last Post: 02-04-2006, 02:53 AM
  4. Post...
    By maxorator in forum C++ Programming
    Replies: 12
    Last Post: 10-11-2005, 08:39 AM
  5. Dikumud
    By maxorator in forum C++ Programming
    Replies: 1
    Last Post: 10-01-2005, 06:39 AM