Have you read the Game Programming Gems books? Some of the articles are about resource management systems such as this. GPG1, for example, contains the article "A Generic Handle-Based Resource Manager". GPG3 has "Handle-Based Smart Pointers", which you can use to simplify usage of the thing.

Bubba, I think you should replace 0xFFFFFFFF by some named constant.