you would be best seeing if you can increase the number of vertexes and triangles and see if it fixes the issue, other than that i think unity and unreal are using voxel based terrains, they have all updated their source code to take advantage of geometry shaders aswell and use more levels of tessellation than torque can because of the whole dx9 thing