I'm writing a simple 3d engine. It renders triangles and quads. However, now I'm having a hard time figuring out how to do my depth testing. No polygons will ever intersect. I originally thought I could simply get the average z coordinate of all the polygon's vertices (the center of the polygon) and sort based on that. However this doesn't always work. For example, if I place two identical quads on top of each other, the bottom one has the same z coordinate even though it should be rendered behind the top quad.
If someone could give me some advice I'd be very grateful.
It seems like you're doing this from a pretty low level, so I'd also suggest reading up on the Painter's algorithm. The Painter's algorithm is basically an archaic alternative to a depth buffer (but a depth buffer is somewhat more straightforward to implement).