I am currently working on a fractal based game engine. This engine generates a fractal world (yes it's actually roundish) from a set of functions and refines it and unrefines it in real time as you run across the surface. Currently I have the world generation, collision detection, and camera working so I can move around the surface. However I haven't started shading it yet. I have face normals and I more or less know how to generate vertex normal's from them. I intend to use procedural shading.
However there is a problem. Sometimes the fractal algorithm generates sharp peaks and stuff like that. I want to be able to set a threshold so that sharp peaks (i.e. drastic changes in normals from one face to the next) are not smooth shaded so they look realistic. I know how to handle this in DirectX but I need a good algorithm to figure out how many normals are needed per vertex and to which triangles they apply. Does anyone know of a good algorithm that will smooth shade up to a point but allows for non smooth shading for sharp corners? I have a couple possible ideas I but I wanted to find out if there was some standard way to handle this.