I'm not stuck on the engine...I've been reading about different VSDs both mid-level and high-level. Also I've been reading a lot about the mathematics and linear algebra behind the engine. A better understanding of the math will eventually lead to a better implementation in code - IMO.
So far I've only purchased 2 books on actual math and one of them incorporates code into it. My next book will be simply about the math and perhaps physics - actually will probably be two books.
I've found some very good ways to do trees and other foliage w/o using billboarding and without storing tons of vertexes. Actually bushes will still be simple planes but trees will be stacked triangle fans with the 'point' of each fan having a positive y vector (with postive y being down in my arrangement of axes).
As for visibility determination I've decided to implement a combination of techniques. For outdoor scenes it will simpy be quadtrees and/or sectorization. For indoor scenes it will be portals and perhaps BSPs...if I can find a very good algo to populate the BSP w/o it breaking down to a linear search - I'm still working on this - part of it has to do with level design. As to how the engine knows if you are in an open area or an indoor area I will use portals...but in a different way. Essentially if the player has passed through one portal from outdoors to indoors then the VSD algo will be switched...althought the quadtree must still be used in case there are windows - aka more portals in the building.
The quadtree algo is extremely fast and the terrain algo that I'm still sketching out should allow for some very long distances to be rendered. I've looked at ROAM and another variation of it and I'm probably going to combine the best of both worlds. Combine this with some frustrum culling as well as object culling (determining if an object lies completely inside of, completely outside of or partially inside of the view frustrum.
I'm still hashing out the portal system but I like its advantages as opposed to a BSP. A good BSP would simply take me ages to compute - basically I must use each line or edge as the root and create a tree from it. Then I recurse the tree and track the amount of time it would theoretically take to render it. If it takes more time than previous ones...I discard that option and start over. This has to be repeated for every edge or wall in the scene or level. Eventually it will come up with the optimal BSP and write it to disk....but the design process is extremely slow. This is one way I've found to get an optimal BSP....but its very slow.
So I'm not stuck...I'm just researching so that I don't code myself into a corner later.
And on a side note I think that having each one of use code a portion of a much larger engine would be a better idea. For the actual game design and graphics we would defer that to artists and designers. I for one suck at game design but I love to code the engine portion. In any one game engine there are several sub-engines at work. Here are just some I can think of and each of these probably have sub-categories in them.
3D Graphics
Input
Sound
AI
Scripts
Storyline and/or flow (movies..cutscenes...etc)
Physics
Multiplayer
Storage (memory management)
Compression (if any)
You might think a storage engine sounds stupid but its not. The storage portion will manage the memory. Most APIs try to place everything in hardware...but there is not enough onboard RAM on sound or video cards to do that.
Another idea is to actually create COM interfaces for the engine. This would require programming the engine and its counterparts as COM objects. This would be the ideal case...but certainly not the easiest. There are books out there on COM and how to create COM objects.