Using XML? My own binary system? A paint by number scheme?

Somehow, I have no idea how, I have to read structured data from a file. I've been pointed to XML, I can understand how this would work, but another poster suggested I make my own hierarchy, and stated:

Quote:

---

It could be as simple as assigning a number to each node, and then when you store the nodes in the file you reference pointers to children by using their number. Then when reading the nodes from the file, you restore all the pointers after the nodes are loaded.

---

This seems really easy to do, but that doesn't mean I even know where to begin code wise. I'd rather do this over xml because if I used xml then I'd need something like Tiny-XML to parse it and that I feel would be overdoing it. My questions seem simple, but really they are overly complex in my opinion.

1. What do my data files need to look like?
2. How do I create them?
3. How do I load them?

The key to making it simple and to avoid the pointer mess is to avoid the pointers all together.

Instead of maintaining pointers to objects on disk which is a mess, you can do this. When you create the BSP tree you will A) know how many objects are in the level/game, B) how many of those objects are in a node, and C) the ID of those objects.

So you have an object file which contains the objects. If you use the ID of the object as its location in the file, then object with ID of 0 will be the first object in the file. Now to store the BSP tree, simply store the object ID's. This will also work for the actual game engine too since every object is ran through some type of resource manager that's either based on ID's or some other identification system. Also give each node an ID in the tree, a way to store it's parent's ID, and a way to store it's children's ID. You can store the final information for the tree (how many nodes, children, etc, etc) in a structure and use that structure as a header for the file, followed by the actual BSP data.

For terrain quad-tree's you only need to store the world min,max bounds and the node type. The header of the file would store the cellsize for the given world, world width, depth, etc, etc. The pointers to children would be converted to ID's. As long as you use a depth-first method this should work. This means that in the file you go as deep as possible on one node until you reach a leaf node. Then you know the next item in the file belongs in another node. When you load this into memory you can now use your quadtree min,max and type information to construct the actual vertices.

Quote:

---

So you have an object file which contains the objects.

---

In what format? How do you accomplish this? Are you using XML? Are you using your own file types? If you are, how are you doing this?

If you're using XML, what do I need to learn to accomplish the task at hand, granted I don't want to learn an entire markup language to produce 300 lines of functional code, if that.

I remember not 3 months ago I was learning about stl containers... Now this?!

XML is always always a slow solution.
While it can be handy for creating configuration files I'd never use it to hold the actual data.

So you might have an XML file holding the index table to a binary file containing the actual data, but not the actual data itself.

I've decided against making my own binary file format due to my lack of, hmm, wanting to write my own binary file format? XML will be perfectly fine for my needs.

so instead of your own binary file format you are having to make your own DTD or XML Schema document.
Instead of code to read and write that binary format you have to write a parser and writer for that XML Schema or DTD.

And you're ending up with a solution that's slower and takes more memory in the process.

Parser is covered, TinyXML will handle that. And I'm pretty sure I don't need a schema or DTD as long as I'm not displaying the information in an html file.. Thats how I understood it..

you'll still need a means to interpret the data that your XML parser spits out...

that's the real parser, all TinyXML (or Xerces which I prefer) does is give it to you in more easily digestible bites.

A DTD will be crucial as documentation if nothing else, and also allows your parser to check whether input files are syntactically correct.
Prevents a lot of headbanging later when you spent 3 days debugging your code only to notice a typo in your XML file at 3:14:32 sunday morning (trust me, I've had it happen, though not necessarilly at that exact time and date).

So enlighten me oh enlightened one, how does one even begin to write a binary file system?

Binary is base 2, because a computer only needs two possible answers. Yes and no.

Normaly, people use base 10. Ten possible answers, 0-9.

So, the easiest way is to establish a different base. But I dont see why you want to make a new binary system.

Let me re word my statement

A binary file type, which holds a binary ized tree structure?

You mean like open a file for "binary writing" except your version of binary? I dont think you can without making your base computer understand the system. Dunno if I got the question, but I'd have to say its not possible without tons of work.

Nono! I don't wanna create my own binary version!

I just wanna create a binary file structure.. Like a .x file or something!

I need to figure out what I need and stuff...

I know how to do I/O in binary, I'm just unsure exactly what I need in my file to get my job done :D

so you figure out what you want to write to it.
Define how you're going to write and read that information, then possibly (advisable) define a file header containing information about the file (type, maybe a checksum, etc.).

If you don't know what information you need in the file, figuring that out is the first step :)

Binary file I/O is actually quite simple no matter what method you use.

You can use C block array-based/character-based I/O, C++ streams, Win32 file I/O, or a combination of them. Don't be fooled into thinking one is more elegant than the other. I've used them all and have found that good old fashioned _open,_read, and _write get the job done quite nicely.

There are newer versions of these in MSVC .NET but they do the same thing.

Array-based I/O is really what you need since you won't be looking for text strings, etc, etc.
Remember you cannot store pointers on disk because a pointer is just an address in memory that contains another address in memory. So the program would have to load at the exact same place in memory and the OS would have to stick all the data in the exact same place in order for the pointer to be valid. Not gonna happen. You could use offsets into the segment, but my advice is don't use pointers on disk. You can de-reference a pointer and write to disk what it points to (IE: a string, some data, vertices, etc, etc.) but writing the pointer won't accomplish anything. A pointer is just a DWORD or a 32-bit variable that contains an address in memory. When you de-reference the pointer, the compiler looks at the address of the pointer, reads the value at that address, and uses that value as an address to another portion of memory.

This is why I said use IDs in your tree's instead of pointers.

But here is how I normally use file I/O for my game code. The editor in MFC uses nearly the same process except it does so through CArchive and CFile. CArchive's know how to write CString's to disk correctly so no pointer problems. Quite handy.

This is only ONE of many, many methods available to you. Use what you like.
Open file in binary and write to it

Code:

---

struct TestStructure
{
  DWORD dwValue1;
  int iValue2;
  BYTE uValue3;
};

int handle=_open("MyFile.dat",_O_BINARY | O_CREAT, _S_IWRITE);
if (handle==-1)
{
  //error
}

//Read structure from file
TestStructure Object;
_write(handle,&Object,sizeof(TestStructure));

//close file
_close(handle);

---

Stream based I/O can also accomplish this because they also support array-based reading and writing using certain functions.

Take for instance this CArchive from MFC.

Code:

---


//Create CFile and CArchive
CFile theFile("MyFile.dat",CFile::modeCreate | CFile::modeWrite);
CArchive theArchive(theFile,CArchive::store);

//Set some variables for testing

//Integer
int iX=5;

//DWORD
DWORD dwY=0xFFFFFFFF;

//CString
CString testString;
testString.Format("Testing: %i %u",iX,dwY);

//DWORD array
DWORD *m_pData;
m_pData=new DWORD[dwY];
memset(m_pData,0xFFFFFFFF,dwY * sizeof(DWORD));

//Write integer and CString object to disk
theArchive << iX  << testString;

//Write literal string to disk
theArchive.WriteString("Hello");

//Write m_pData array to disk
theArchive.Write(m_pData,dwY*sizeof(DWORD));

//Close archive
theArchive.Close();

//Close the file
theFile.Close();

---

For more info concerning the extremely robust C/C++ I/O system, I recommend consulting a book on C/C++.

Also Win32 provides functions for this as well. So there are about a billion ways to do this.

Will a root node structure or class and a vector of children survive through a file? all that is done by value right? So therefore I can put something in a vector, write it to a file, then retrieve the vector with no problems right?

The only thing I really know, is that I need to represent a tree structure in a binary file.

I can probably do it in the same way I do it in my C++ code, with a few minor modifications.

Instead of a vector of the children themselves, I need a struct Node, with an ID according to its type, Transformation is 0, geometry is 1, etc etc...

It also needs to hold a vector of children, again based on ID's...

Soooo, when I tell my program, after I load the file..

To create a root node of type 0, with a 1 as a child, it knows to create a DOF node, then create a new geometry node, and set the DOF node as the geometry nodes parent, then I can just add that to the tree.

Simple :d

no, you're going to have to write your own serialisation and deserialisation routines.
But that should be no problem, when writing iterate over the vector and write out the content, when reading read from the file and populate a new vector with what you read.

Creating an interface (abstract base class in C++) that defines the serialisation and deserialisation functions and having everything you need to serialise derive from that is the logical option.
That's in fact exactly what Java does for you ;)

So you'd have something like

Code:

---

class Serialisable {
  public:
    std::ostream& serialise(std::ostream& os) const = 0;
    std::istream& de_serialise(std::istream& is) const = 0;
}

---

Serialization what?

What is this serialization you speak of?

serialisation is the process of writing out objects to disk and reading them back in.

oh you mean simple binary I/O?

like fout << obj (with a bit more syntax)

yes, it's the process of streaming objects to whereever and recreating them afterwards.

When you get further along you might also want to create a system to ascertain that you have the correct version of the class definitions to recreate the version you're trying to read, but first get to read and write at all ;)

Sweet, so does my idea of doing this sound valid? Let me re-iterate:

Quote:

---

So, my idea is this, I can have ID's (integers) representing the various types of nodes we can have..

For instance

0 for Transformation
1 for Geometry
2 for Particles
3 etc etc...

I can have a structure that I write to the file, that contains basically, a tree.

First we have the root, in the root structure we can have a vector of children

Lets say the root is a 0, and it has a 1 as a child (in the vector..

The 1 is a geometry node, which also has a particle emitter attached to it, which is a 2 in the geometry's vector of children

So we have something that looks like this:

a 0 with a 1 for a child
a 1 with a 2 for a child

Does this sound like a good way to go about it? I figure with this much information I can create a hard coded tree structure out of the information given.

Step 1: get the root node from the file and all its children, recursively..
create the nodes according to the ID's

step 2: Link them together according to the information in the structure, again the ID's

Does this sound right?

---

I also thought I might need two sets of id's for the structs, perhaps one vector for the parent, or just a variable for the parent, and then a vector for the children...

You may want to add some more information, either in the header or at the individual records.
A table of content with offsets for each entry is one, makes quickly reading a specific entry faster.
Then per record type an entry showing the size in bytes of that entry type (if possible, but I'd strongly advise a fixed length format for each type, as it makes reading and writing a lot easier).
Then as part of each record you write write the index in the index of all its children.
So effectively you're getting a file header containing an index of all records and types.
Then you have for each record a record header containing the record type and index number as well as a list of all child records, and after that record header the record data.
You might want to throw in some checksums (per record maybe, or on a file level) as well to prevent tampering and detect corruption.

I see, a header file with like, an index to the file? Or another file as a table of contents..

Label each object with an ID?

Should this index be hard coded? Or should this be a file too?

The index idea sounds cool..

Then I could delete an object judging by its ID from the scene graph, wow..

Imagine..

Just being able to say to the scene graph... Don't draw torches! Don't draw lights! Don't do this, or that, blahblahlah

yup, that's the thing.
More important still, you no longer have to rewrite the entire file to add a child somewhere.
Just update the index and put the new child at the end of the file (or in some place where you previously removed something, leaving a hole that's large enough).

This is serious coding, that quite frankly, isn't rewarding unless you spend a good month on it, specially if you've never done it..

So for now I'm going to play with terrain and stuff :D