So I'm back at improving the terrain texture blending visuals. As some of you may remember, I worked on this technique a long time ago and got stuck because I lost the height information after each terrain layer.
So, first a bit of background information:
Torque's Terrain Texture blending
In general, Torque's terrain is pretty well done for what it seeks to accomplish. It lacks dynamic tesselation and what not, but it gets the job done for most games. T3D has a "terrain" object, which has a huge diffuse map which is used for coloring the terrain. Up-close you can mix it up with detail textures to allow detail into the over-all coloring of the terrain.
The terrain can have 32 different texture layers, which consists of:
- A Diffuse Texture - Which contributes to the aforementioned diffuse map
- A Detail Texture - Which provides up-close detail
- A Macro Texture - Which provides medium-distance detail
- A Normal Texture - Which provides normal and height information for the detail texture
T3D currently blends these linearly, using a layer-texture which holds the information about what texture is painted to what layer. This is similar to using a lerp function to blend between two variables. The result of blending a red, a green, a yellow and a black texture can be seen in the following image:

Which would typically provide results similar to the following image, when used on actual textures:

And to if we lerp-blend left-to-right between two textures it looks like this:

The sampler limit
When rendering each layer, we need to send 4 textures per layer to the shader. There is an inherent limit in how many registers each shader can receive, which varies from shader model to shader model. I believe it is 32 for D3D11, however I don't feel like finding a source on that again. However, assuming it is 32, we can easily see how we might hit that cap quickly:
$$\frac{32}{4}=8$$
I do believe the cap was even lower for D3D9, so at some point, someone decided that we would send just 3 textures down to each pass over the terrain, meaning a terrain block can be rendered over 11 draw-calls.
My first attempt
As mentioned, I worked on this a long time ago. Back then I encountered an issue with these multiple passes made over the terrain (the 11 draw-calls), namely that I lost the height information between each pass. Now this is bad, since the approach I took was a height-based blending of the textures, so I looked at flip-flopping a height-map render target.
What does flip-flopping mean? The concept is pretty simple actually. Due to how the GPU works, we can't read and write to the same texture each pass. If we need to both read and write the height information to a texture, then we need to create 2 textures, and swap them around for each pass. So in pass 1 we read from texture 1 and write to texture 2, in pass 2 we read from texture 2 and write to texture 1, in pass 3 we read from texture 1 and write to texture 2, etc. This process is called flip-flopping.
However, I never really got flip-flopping to work, for some reason it just wouldn't work.
But what now!?
It works! I dunno how or why, possibly the change to D3D11 changed something, but nevertheless it works!
Just to recap, my work was based on Andrey Mishkins' Gamasutra article on texture blending (http://www.gamasutra.com/blogs/AndreyMi ... atting.php). It's a really short and nice article, I recommend you to read it, so I don't have to repeat the details here.
Using his technique on the left-to-right example showed earlier, we get the following result:

And it can be easily customized to change the results, by changing the "blendDepth" variable:

And the in-game example looks like this:

What's left?
Confirming that it works. There's a bit of spaghetti code, especially in the generated shaders, I will have to clean them up, and try some more use-cases. Furthermore, some performance tests are necessary to see how much the new shader code and flip-flopping of textures impacts the performance of the terrain rendering system.
So I can't promise you that this will see the light of day or be usable, but I think it is pretty cool, and I hope it can help us take some steps in the direction of more nuanced and realistic texture blending in our terrain rendering.