The double role a comma has in your spec -- it is both a delimiter between node name, and a list delimiter between children -- hides a glaring problem in your spec.
Please, let me show how I'd go about this. I'll use ABNF, which is what is used in internet RFC's, and definitely useful to know if you work with internet stuff.
Let's start with a simple recursive definition similar to brewbuck and others suggested above, but with a colon instead of the first comma, just to make it a bit more readable.
node := "<" value [ ":" [ node ] *( "," node ) ] ">"
value := 1*digit
digit := "0" / "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9"
In other words, a digit is a single digit, a value is one or more digits, and a node begins with a < followed by a value. If the node has child nodes, a : follows, and then the comma-separated list of child nodes. Finally, a > closes the node.
(I'm not assuming binary trees here; the above spec allows any number of leaves per node.)
A leaf node with a value 5 would be
<5> or
<5:>
If we have node 1 with child nodes 2 and 3, we could specify that with
<1:<2>,<3>>
To describe this tree,
Code:
│ 3
│ ╱ ╲
│ ╱ ╲
│ 2 5
│ ╱ ╱ ╲
│ 1 4 6
we could use
Code:
<3:<2:<1>>,<5:<4>,<6>>
A simple recursive definition of a tree will always use pre-order tree traversal. This means that starting at a given node, you emit the node value, then descend into each child starting at the leftmost. Above, we start at 3, descend into 2, then into 1. Since 1 is a leaf node, and 2 only had the left child, we've done the first subtree of node 3. Then we descend to node 5, and from there to leaf node 4. Having completed 4, we only have the second subtree of node 5 to do, and that's node 6.
The above paragraph is the key. If you don't understand it, compare the tree and its specification, and how the specification builds the tree or vice versa, until you grok it.
I only used colons above, because it makes it easier to read (and write) the spec correctly. We can further augment the syntax, to not require angle brackets for a leaf node, and to make commas and colons optional and interchangeable:
node := value / "<" value *( [ ":" / "," ] node ) ">"
value := 1*digit
digit := "0" / "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9"
Using this augmented syntax, we could describe the above tree using
<3:<2,1>,<5:4,6>> or
<3,<2,1>,<5,4,6>> or
<3<2,1>,<5,4,6>> or
<3<2<1>><5<4><6>>>
Although the separators make a big difference for human readability, it does not matter for a machine parser. So, which separators and where to use and require or leave optional, should be determined by what makes it easiest for humans to understand the format correctly.
(Note: Programmers are humans, too. If you thought that "well, since this format is going to be read and written by computers only, I don't need to worry about how easy it is for humans to understand", you'd be wrong: the programmers have to understand the format to be able to implement it. So, human understanding of the format definitely matters, even if none of the users are human.)
The original post describes this tree:
Code:
│ 1
│ ╱ ╲
│ ╱ ╲
│ 2 3
│ ╱ ╲
│ 4 5
Using the syntaxes defined above, we could describe it using
<1:<2>,<3:<4>,<5>>> or
<1,<2>,<3,<4>,<5>>> or
<1,2,<3,4,5>>
At this point it should be clear that the syntax OP (nickmenphis) wishes to use is either a logic bug, or requires something other than simply recursive specification: the < and > are used very differently than a simple recursive spec would.
Indeed, comparing the text <1,<2,3,<4,5>>> to the tree above shows that in OP's spec, < and > are used to denote a level in the tree, filling nodes from right to left.
Nicmenphis, your spec is not going to be easy to implement, and even if you implement it, it will be frustrating to your users. Can you rethink your spec?