[UE4] Adding a custom shading model (Part I)

CustomShadingHeader-1

This was written in February 2016 which means it was written for an old version of the Unreal Engine.
There were a few changes with the newer versions so this is not up to date (mostly changes in which lines the code can be found)
In the comment section of the second part someone mentioned a few useful hints for changes to the 4.14 version so be sure to give that a look if something isn’t working as expected.

For my first three posts I want to write about how to add a custom shading model for the Material-Editor of the Unreal Engine 4. This is something that isn’t very well documented and quite a few people on the forums were struggling with it so I would like to share my results so far.


CustomShadingExamples

Rows: 1. roughness & specular 2. shadow smoothness 3. light support 4. normal map examples

First of here’s a few pictures on what the result looks like. This model will give us some control over the shadows from within the Material-Editor to achieve a toony look. It also offers support for metallic, translucency etc. Some of the effects like the outline, the fresnel or the colored shadow are handled in the Material-Editor.

If you feel interested in how this was implemented please continue reading. Even if you try to implement something else or simply want to change the BRDF you can continue reading as the groundwork is basically the same.
Before we get to the implementation there’s a few things worth of notice:

  • You should be familiar with at least one shading language. Unreals shading files (.usf) are pretty similar to HLSL.
  • This is just a guide about doing some changes to the deferred pipeline so rendering on mobile devices might not be affected (I haven’t made any changes to the forward lighting). I don’t plan to look into that at the moment but if you have any information on the subject you can add it in the comments. ūüėõ
  • I won’t upload the whole engine code so don’t ask for it.
  • This shading model is mostly based on the information found in an old Thread on the Unreal Forums. I also found out later that there was another attempt making a custom toon shading model which is based on the preview version so you might look up line numbers there if you have problems finding something.

 


Let’s get started.

First we need to get our hands on a source code version of Unreal4 since the normal launcher installation won’t let us modify the code. Either grab a fresh source copy of the Engine from Epics GitHub or if you already have one make a backup of all files you are editing and then open the solution in the editor of your choice. I used version 4.10 of the Engine. If you are using a really old version you will notice differences in the base structure of the code so this probably won’t be compatible. On the newer versions (current preview builds) Unreal added a few more shading models and the code has changed a bit so keep that in mind (mostly line changes and new names for the custom variables).

 

EngineTypes.h
First we need to add our new shading model so that the engine actually knows about it. Open up EngineTypes.h and scroll to line 203. You should find the enum EMaterialShadingModel. Before MSM_Max  add a line:

MSM_StylizedShadow UMETA(DisplayName = "Stylized Shadow"),

This will add a new model to our dropdown inside the material editor. You can of course name  it however you want just remember the name of the enum.

 

MaterialShared.cpp
Next we need to tell the shader about our newly added model. At line 1257 add a line below the last shading model.

case MSM_StylizedShadow: OutEnvironment.SetDefine(TEXT("MATERIAL_SHADINGMODEL_STYLIZED_SHADOW"), TEXT("1")); break;

 

Material.cpp
The last thing we want to do is enable a few pins for additional customizing inside of our shader. Scroll down to around line 490. This part of the code defines which pins will be unlocked for the different shading models. Instead of going through the process of creating new pins we will hijack some of the existing ones which aren’t in use. Search for the cases MP_Clearcoat and MP_ClearcoatRoughness and simply change the line below to this:

Active = ShadingModel == MSM_ClearCoat || ShadingModel == MSM_StylizedShadow;

In the Preview Versions those variables should be called something like MP_Custom. This is because they get reused a for different shading models. I decided against adding a new input here as I don’t think it’s good practice to sent more information to the graphics card as necessary. If you want to add another input instead refer to the code changes made by the other toon shading model mentioned above.

 


CustomShadingModel

Our new shading model

And this should be all. If you would compile now the Material-Editor should show our newly added shading model inside the dropdown menu.

 

In part 2 we will go over the shader side of things.

6 thoughts on “[UE4] Adding a custom shading model (Part I)

  1. juangea

    Doies this require the full reompilation of the engine or can it be done in the default engine?

    Thanks for the tutorial!!!

    Reply
    1. FelixK Post author

      Hey, sorry for the late answer I usually only have a ton of spam bots here since I never linked it anywhere so I didn’t notice until now.

      I don’t think it will recompile the whole engine but at least the shader related parts have to be compiled again avery time you make a change. (Which will still takes a lot of time)
      This was written for an old version of the Unreal Engine 4 so a lot of things might have changed (code wise and regarding how it is compiled) but I usually would recommend to prepare the code for the basic shading / light calculation in a faster compiling environment so you don’t waste too much time with recompiling.

      Reply
  2. Keiko

    Hi, thanks you for this tutorial, it is very useful for my needs.
    You said that you didn’t want to add pins to the material input, but I want to do it. Do you know the way, or can you give me some hints ?
    I found some references to materials attributes in SceneTypes.h and MaterialShared.cpp, and I think it is related to FPixelMaterialInputs in the shaders.

    Your help is welcomed.

    Reply
    1. FelixK Post author

      Sorry can’t help you with that one. It’s been a while and I currently have no active source code based UE4 installed.

      There was another guy on the Unreal Forums that tried something similar around the time I wrote this post. As far as I remember he had custom pins in his model so maybe you can find some information regarding that on his GitHub.
      https://forums.unrealengine.com/showthread.php?70150-Toon-shading-model

      Edit:
      Tried to look into their current release branch again.
      – Material.cpp: Lines 4381+ (seem to be relevant to activate pins depending on which shading model is active)
      – MaterialShared.cpp: Lines 2774+ (Definition of the Pin names and input types?)
      I don’t think SceneType.h matters because it doesn’t mention clearcoat or custom inputs at all.

      Reply
      1. Keiko

        Thank you I didn’t find this thread on Unreal Engine forum, I’ll look into it and leave a reply here if I find a solution that works.

        Reply
  3. Steven

    Hi Felix! Awesome tutorial! Just want to share how to customize a custom pin’s name so it can say something like “Smoothness” instead of “Clear Coat”. It’s under GetCustomDataPinName() in MaterialGraph.cpp.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *