Storing resources into a single file

This is not a new idea or anything, but while I'm google'ing for it I'm thinking someone might have some knowledge on the subject. Instead of leaving the dozens of resources (bmp's, wav's, mid's) laying around my game's directory and whatnot, I'm going to opt in to dumping all of them into a single file (like pak and pk3 files) and load them during initialization. My question:
Should I load the resource in memory or extract them all to a temporary directory, load, and delete? I would prefer the former (unless there are good reasons not to) because I don't like the idea of creating/deleting such temp files every run.

What do you think? Please point me at any good resources that might assist in my quest. Thanks.

[edit]
If you know of any resources detailing how to go about loading resources (BMP, WAV, MID) from memory, let me know.
[/edit]

I would suggest loading them into memory instead of temporary files, simply because they're gonna have to get in there eventually anyway. However, you would only load up the ones you need, and the way you go about that depends on how you organize your resource files (if it's in a giant one, you would only extract that which you need, or you can make a resource file for each level or area and load everything in that file).

isnt this like a virtual file system?
try and google for that.

You should just load them into memory. With most computers shipping with 256 to 512mb of memory available it isn't too likely you'll have problems.

I have a class I wrote a while back that provides an interface for PAK file creation/manipulation, so that will serve as my VFS. The bigger issue is that I would like to (as we all agree) load my files into memory then into the application, but I have no idea how that could be done. I know that with a little research loading a bitmap from memory won't be too difficult, but WAVs and MIDIs are a whole other story...
Without any idea on how to do such a thing I will probably end up just extracting from the PAK to the machine's temp directory, loading them, then deleting them.

instead of loading the entire blob file from disk and then worrying about creating these types from memory, why dont you build the types as they are loaded from disk. I dont know if this is possible with the pak format, but you can just open the file with a stream or whatever, then pass the stream around to the functions which load the bmp's, wavs, and midi's in the standard ways. This way each component is built as the data is loaded, rather than loading all the data into memory and trying to bend other functions to build from memory.

just a thought, this would be my approach.

[quote = Perspective]I dont know if this is possible with the pak format[/quote]
As long as you extract and decompress as necessary, it's the same as loading from a normal file. loading .wavs and midis is different depending on what you're working with. DirectSound is ALMOST as simple as saying "play this file," and DirectPlay is just as easy(you have to work with some com objects and such). I haven't worked with OpenAL, but I hear it's not much different.

If you're using Win32 API , I suggest you look up mapping files to memory functions.

This will get you started.

1) CreateFile
2) CreateFileMapping
3) MapViewOfFile

4) Perform operations

5) UnmapViewOfFile
6) CloseHandle [mapping]
7) CloseHandle [file]

This will let you treat your file as a pointer to a large chunk of memory. You can pass around the pointer wherever you want. This is probably a good route to take. If you have questions with anything specific let me know, I use this in my projects at work all the time.

That is an enormous help. It looks like that will allow me to do exactly what I've been thinking about.

DirectMusic8 provides the capability to load sound files in memory and I just successfully ran a little test that actually worked. I'm on the right track and lovin it.

Thanks a lot Mr Wizard.

I've got one question for you, MrWizard. In MapViewOfFile(), I am attempting to set the low-order DWORD to the offset of the logical file within the physical file, but it fails. GetLastError() says "The parameter is incorrect." What I did was insert 10 bytes at the start of the file then try to map the file changing the last two args:

Code:

---

m_pMappedFile = MapViewOfFile(
      m_hFileMapping,      // handle to the file mapping object
      FILE_MAP_READ,        // desired access to the map
      0,                    // high order DWORD offset where mapping begins
      10,                    // low order DWORD offset where mapping begins
      m_dwFileSize-10                    // number of bytes of the file to map
    );

---

I am clearly misunderstanding something about how file mapping works. Do you have any ideas?

The other alternative that does work, but may be less desireable (but then again may not be), is to just map the entire file into view then use the pointer to read the header info and such and then just move the pointer to the start of each file accordingly.

[edit]
After thinking about it, I don't really see a need for what I'm asking here. All of my resources will be in a single file, so mapping the entire thing makes complete sense.
[/edit]

Your use of MapViewOfFile is failing because the offset parameter must be a multiple of the system allocation size. See this blog post for a MapViewOfFile sample.

There is a function to map a complete file in this post.

You could also consider using windows .RC resource files, with FindResource, LoadResource, etc.

Thanks for the resources.

Now, one final question (with any luck)... I am having nothing but success loading audio with the mapped file (using DirectX, btw) but nothing but trouble trying to load bitmaps. I have traversed google for several hours and tried fiddling around with things myself with no luck. I am perhaps mistakenly attempting to load the bitmap with CreateDIBSection(), with no success and cannot find with my own efforts an alternative method to perform the task. Any ideas, tips, or hints?

It shouldn't be a problem so long as your parameters to CreateDIBSection are correct. I'd really have to see some code to make sure. Perhaps also I'd even have to see the values in memory from your file. There shouldn't be a problem though.

Ok, so I am right to use CreateDIBSection(). What I'm doing, in brief, is (for just a plain bitmap file for testing) use the pointer from MapViewOfFile() and copy sizeof(BITMAPINFO) bytes into a BITMAPINFO struct.

Code:

---

    HDC dc = CreateCompatibleDC(NULL);
    m_hBmp = CreateDIBSection(dc, &bmi, DIB_PAL_COLORS, NULL, m_hFileMapping, sizeof(BITMAPINFO));

---

It returns NULL but GetLastError() returns success (zero). Does this seem like I'm using it properly?

You want DIB_PAL_COLORS ? I'll assume yes.

Well I'm not sure exactly what all your values are. However, I'll quote MSDN here with regards to the 'offset' parameter (last param) of CreateDIBSection:

"If hSection is not NULL, the CreateDIBSection function locates the bitmap's bit values at offset dwOffset in the file-mapping object referred to by hSection. An application can later retrieve the hSection handle by calling the GetObject function with the HBITMAP returned by CreateDIBSection. "

You specify sizeof(BITMAPINFO) which may not yield the correct results from the beginning of the mapping.

Does that make sense?

I think one typically uses DIB_RGB_COLORS for a 24, 16 or 32bpp image. Have you tried it?

Does your file mapping meet this requirement?

Quote:

---

Originally Posted by MSDN

If hSection is not NULL, it must be a handle to a file-mapping object created by calling the CreateFileMapping function with the PAGE_READWRITE or PAGE_WRITECOPY flag. Read-only DIB sections are not supported. Handles created by other means will cause CreateDIBSection to fail.

---

[edit]Who says that you can pass NULL for ppvBits?[/edit]
[edit2]Got to take into account BITMAPFILEHEADER which is before BITMAPINFO in file.[/edit]

I read that on MSDN as well and did that. I'm calling CreateFile() with GENERIC_READ | GENERIC_WRITE, CreateFileMapping() with PAGE_READWRITE, and MapViewOfFile() with FILE_MAP_READ | FILE_MAP_WRITE.

>>[edit]Who says that you can pass NULL for ppvBits?[/edit]
I just realized that MSDN said nothing about it, so I passed an arg to it.

>>[edit2]Got to take into account BITMAPFILEHEADER which is before BITMAPINFO in file.[/edit]
I'm doing this now:

Code:

---

    LPBITMAPFILEHEADER pBFH = (LPBITMAPFILEHEADER)m_pMappedFile;
    LPBITMAPINFOHEADER lpBMIH = LPBITMAPINFOHEADER(LPBYTE(m_pMappedFile) + sizeof(BITMAPFILEHEADER));
    DWORD dwOffset = pBFH->bfOffBits;
    LPVOID pBits;

    HDC dc = CreateCompatibleDC(NULL);
    m_hBmp = CreateDIBSection(dc, (LPBITMAPINFO)lpBMIH, DIB_RGB_COLORS, &pBits, m_hFileMapping, dwOffset);

---

But it is still returning NULL but with GetLastError() returning success... What am I doing wrong?

I do appreciate all the help. This one thorn in my eye is a killer. I can't continue with my project until I resolve this issue.

[edit]
>> You want DIB_PAL_COLORS ? I'll assume yes.
I've been going back and forth just trying to see if anything would change. It hasn't. It seems to me that I'm doing things properly, but apparently I'm not. What can cause GetLastError() to return success upon failure?
[/edit]

Do you check with a debugger that the values pointed to by pBFH and lpBMIH are correct? You should really do that.

Also make sure dc is valid.

OK, you want the good news or the bad news?

The good news:
Change:

Code:

---

    m_hBmp = CreateDIBSection(dc, (LPBITMAPINFO)lpBMIH, DIB_RGB_COLORS, &pBits, m_hFileMapping, dwOffset);

---

to:

Code:

---

    m_hBmp = CreateDIBSection(dc, (LPBITMAPINFO)lpBMIH, DIB_RGB_COLORS, &pBits, m_hFileMapping, 52);

---

and it will "work". This is to do with DWORD alignment. I'll post back in a minute.

And now the bad news...

The problem is that the standard bitmap header, without extra color information, is 54 bytes. However:

Quote:

---

Originally Posted by MSDN

The bitmap bit values are aligned on doubleword boundaries, so dwOffset must be a multiple of the size of a DWORD.

---

Unfortunately, the value 54 is not DWORD aligned (a multiple of 4). Therefore, the CreateDibSection and file mapping solution does not work with most bitmap images!

You can find more information on this issue:
http://groups.google.com/groups?q=CreateDibSection%2054
http://www.eggheadcafe.com/ng/micros...post309964.asp

There doesn't seem to be a simple elegant solution but some ideas:

- Use an incorrect offset like I did in the previous post. Your image will be a couple of bytes off, but the user might just think that their monitor is shoddy!

- Patch the input bitmaps before distributing with your program. This is not a bad solution, especially since you are already working with the input bitmap when you add it to your "resource file". You could change the offset to 56.

- Ditch the file mapping with CreateDibSection. Just create a normal dib section and copy the bits into it from the file mapping. This is a little less efficient but requires no change to the input bitmap.

Out of curiosity, let me know what solution you go with.

Holy cow. You are awesome anontymouse. I just started thinking about the DWORD alignment after rereading MSDN for the nth time. That is a beautiful thing. Now that it's actually loading...

I changed the assignment of dwOffset to:

Code:

---

DWORD dwOffset = pBFH->bfOffBits - (pBFH->bfOffBits % sizeof(DWORD));

---

Son of a gun, who'da thunk it would be that simple? Thanks a lot anontymouse and MrWizard. Finally the aspirin for my headache...

[edit]
As for the approach I think I'll be taking, I think padding the bitmap header info is what I'll go with since (as you mentioned) the resources in my file will be made for a specific purpose. Again, thanks.
[/edit]

[edit2]
By the way, I love that first solution "the user might just think that their monitor is shoddy!" Heheh.
[/edit2]