Custom Materials are "weird". They can work with deferred lighting, as Duion mentioned, but they do a lot of hoop-jumping in the backend that can sometimes cause problems, and EVERYTHING has to be implemented manually on them. So if you want to apply them to an animated character mesh, you have to implement the hardware skinning logic yourself for it to work, otherwise it's unanimated.
This cropped back up again recently for a few people, and has been a personal bugbear that's slightly bothered me for a while. And, given that I want to expand the material system to support more functionality, as well as a visual editor for art types so they don't have to learn coding or the like, I took some time during the week to start hashing a system that lets you implement custom shader logic, but hooks back through the normal material system and ShaderGen.
Currently it's pretty basic, but it should get quite fleshed out in the near future and doesn't impact regular materials in any negative way.
As you see in the example above, it's going to pretty much be as simple as making a new instance of a new class, CustomShaderFeatureData, like so:
//Special logic setup happens here
Then you refer to that on a material with the new CustomShaderFeature field, like so:
mapTo = "GridMaterial";
CustomShaderFeature = FlatColorFeature;
Finally, you implement some functions that Shadergen calls back into when our cube_GridMaterial is generated, using our CustomShaderFeature's name as the namespace:
//Nothing to do here
%this.addVariable("bobsyeruncle", "float", 1.1);
%this.writeLine(" float testing = 15.915;");
That ultimately will add in a little chunk of code to the material's resulting shaders automatically as per shadergen, which will look like this:
float bobsyeruncle = 1.1;
float testing = 15.915;
The plan is to next support binding of resources(passing in uniforms, texture samplers, etc) and dependencies and feature overrides.
Ultimately, you could either have it implement a small piece of additional logic, or you could have it disable all normal features and write a fully custom shader that goes through the main Shadergen system, meaning it'll be consistent to the rest of the materials the engine uses instead of the sorta-special-snowflake that is CustomMaterials.
Because the CustomShaderFeature is it's own object that is referenced by a material, you could, say, write a ToonShadedFeature, and then use it in all your materials and only have to write the code once, and just refer to it on all your materials and poof, all your materials are toon shaded, etc, etc.
This should be a LOT easier to work with than juggling through the current custom material system, and behave far more consistently.
So with that explained, if there's anything you think would make things better/easier to utilize with this system, lemme know