[UE4] Adding a custom shading model (Part II)

This is the second part of the shading model implementation. In this part we are going to edit a few shader files. Unreal shader files end with a .usf but if you are familiar with hlsl you shouldn’t have any problems. You should also at least understand what deferred rendering means. If you are unfamiliar with that subject take a look at Wikipedia. As this part is mostly about coding there won’t be many pictures. Sorry about that. :<

Alright time to move on.
Look for the Shaders folder directly nested inside the project root of the engine. Editing shader files can be done in any text editor since the compiling will happen once you open the Unreal Editor. On Windows using VisualStudio is a good choice since the compiler tells you were the parser found an error and sometimes even information about what it encountered.


Let’s start were our input gets written into the GBuffer. For this open up BasePassPixelShader.usf.
This shader writes all the values acquired from pins inside the Material-Editor into the GBuffer. Keep in mind that this shader will render the Geometry itself.
First let’s take a look at how the GBuffer is build. For now we only need to worry about GBuffer A – D.

  • GBuffer A – float3: normals, float: packed alpha
  • GBuffer B – float: metallic, specular, roughness, decalMask
  • GBuffer C – float3: baseColor, float: ao
  • GBuffer D – float: opacity, float3: customData

CustomData in GBuffer D has different uses depending on the shading model.
We will also make use of it later.
You might have noticed that the emissive isn’t listed inside one of the buffers. Unreal calculates lighting by starting from zero and then adding the light.
The emissive value gets written into the basic output instead of a buffer so it will already be there before the light is added.

First go to line 774. Our target is directly under the comment saying “Volume lighting for lit translucency”.
Change the next line to:


We need to do this so that we actually see something if our blending is set to translucent or additive.

After this jump over to line 879. This is were we specify shading model specific data. Let’s add our own model here:


    float smoothness =  GetMaterialClearCoat(MaterialParameters);

    GBuffer.CustomData.x = smoothness;

We use the CustomData to add our hijacked pins into the GBuffer. We still have the GetMaterialClearCoatRoughness() pin unlocked so if you need another custom input write that to CustomData.y. This is also were you might do changes to all the other GBuffers like baseColor etc. if your shader requires it.
Again in the preview version of the engine the GetMaterialClearCoat() should be called something different like GetMaterialCustomData0().

And that’s all for this shader. Time to move on.


The next step is optional so skip it if you want. What we do here is changing the color with which our shading model is displayed in the Shading Model Buffer Visualization. Go to line 1010 and you will find a list of colors used for the different shading models. Add this to line 1022:

 else if (GBuffer.ShadingModelID == SHADINGMODELID_STYLIZED_SHADOW) return float3(0.4f, 0.0f, 0.8f); // Purple

and this to line 1034:

 case SHADINGMODELID_STYLIZED_SHADOW: return float3(0.4f, 0.0f, 0.8f); // Purple


Next up is setting the definition for our shading model. In the shader at line 227 add this and replace the line saying max number with our new maximum.



Now that we have our basic setup out of the way let’s get to the interesting part. This file contains information about the shading itself. Take a look at the already defined ones and you will get an idea what’s going on here. The BRDF’s are part of a different file called BRDF.usf so if you want to add a new function for a BRDF you can do that there. If you are unfamiliar with BRDF’s it’s basically how light information get’s handled in this case mainly specular and fresnel. Unreal has a few unused snippets there as well so you could try out replacing the lambert model with the oren nayar one in one of the shading functions in this file etc.

For this shading model I haven’t added anything BRDF related. Instead I made a copy of the SimpleShading() function and made changes to that. The reason behind it is that we only need the specular calculation. In line 69 after the StandardShading() function we add our own:

// Stylized shadow shading model for manipulating the gradient of the specular
float3 StylizedShadowShading( FGBufferData GBuffer, float Roughness, float3 L, float3 V, half3 N, float2 DiffSpecMask )
	float range = GBuffer.CustomData.x * 0.5;

	float3 H = normalize(V + L);
	float NoH = saturate( dot(N, H) );

	return GBuffer.DiffuseColor + saturate(smoothstep(0.5 - range, 0.5 + range, D_GGX( Roughness, NoH )) * GBuffer.SpecularColor);

This is a very basic shading model that takes the base color and adds the specular on top of it. For this we use the already in BRDF.usf defined D_GGX() function. Here we will use our CustomData for the first time to manipulate how sharp the edges of the specular highlight will be. Again if you are unfamiliar with the smoothstep() function take a look here.  A smoothstep of (0, 1, x) would give us the default while one at (0.5, 0.5, x) would be the same as a round(). Going up or down on both sides would shift the resulting gradient in that direction.

After we defined our function we need to tell the shader to actually use it. For this move to line 270 and add a case for our shading model inside the SurfaceShading() function.

    return StylizedShadowShading( GBuffer, LobeRoughness.y, L, V, N, DiffSpecMask);

We now have added our own custom shading. Let’s get to the last part of this guide: The lighting function. If you just wanted to make changes to the BRDF you can stop now. The next section is only for getting the toony shadow look.


The actual lighting for the different light types takes place in DeferredLightPixelShaders.usf but as all the lights use the functions defined here we only need to change this one. We will only make changes to a single big function here with the name GetDynamicLighting(). This one calculates our light and shadow in the deferred rendering pipeline. So what do we do here? It’s simple: We want to remove the NdotL calculation from our models shadows and sharpen the edges (after the engine went through all this work of smoothing them :>).

In line 352 we add a new branch.

float3 Attenuation = 0;

// In case we want to stylize the shadow changing it's falloff
    float range = ScreenSpaceData.GBuffer.CustomData.x * 0.5;
    Attenuation = LightColor * ((DistanceAttenuation * LightRadiusMask * SpotFalloff) * smoothstep(0.5 - range, 0.5 + range, SurfaceShadow) * 0.1);
    Attenuation = LightColor * (NoL * SurfaceAttenuation);

we also need to need to make changes to the default calculation to make use of our newly added attenuation variable so in lines 372 and 376 change

 LightColor * (NoL * SurfaceAttenuation)




That’s it. We have everything together for our custom shading model. In case you haven’t done it yet recompile the changed C++ classes and start the engine.
The startup will take some time now as it needs to recompile the shaders. Once all shaders are compiled and your project is opened create a new Material in the Content-Browser and change it’s shading model to our newly created one. Add in a few constants and it should look like this.


Our new shading model

The last and final part will be about adding stuff to the Material and making a Post-Process for a colored outline.

8 thoughts on “[UE4] Adding a custom shading model (Part II)

  1. Rawkul

    Hello FelixK !

    First of all, I should say you “a HUGE thank you” for the tutorial! I didn’t find any tutorial for writing my own shaders in Unreal, but now I’m able to write my own shaders for UE4! Thank you 🙂

    Apart from that, the second main reason why I’m writing is because I encountered several errors during the process and I was able to fix them, so I would like to point them out. The errors are in this second part of the process and are referred for anyone trying to reproduce this shader, in general terms the tutorial explains pretty well which files you have to modify. I suppose most of these errors are due to this was thought for past versions, but for the new one you have to change a few things, here they go (In chronological order):

    -) FIRST ONE: In the last version that I’m using (4.14 I suppose, just downloaded the source yesterday) you can’t change the colors for the Shading Model Buffer Visualization in the file “MaterialTemplate.usf”, instead you should do it in the file “DeferredShadingCommon.usf”, the same where you define the ID for your shader. Just scroll a few lines below from the ID definition, or simple search for the comment ” Cyan ” in order to find the lists with the others colors (there’s one with cyan, that’s why I said that).

    -)SECOND ONE: (AND MOST IMPORTANT ONE) When you’re defining your own shading function, in your case is StylizedShadowShading, in the file “ShadingModels.usf” you’re calling a sixth argument, a 2D vector called DiffSpecMask. This parameter hasn’t been used in the function at all and it doesn’t exists (or at least it’s NOT declared in the document), so you MUST get rid of it since that parameter no longer exists (or at least it isn’t declared), and otherwise the UE4 built will fail. Once you’re done, you should have a function like MyShadingFunction( FGBufferData GBuffer, float Roughness, float3 L, float3 V, half3 N ). If you’re declaring a different one not meant for cell shading, then this may not be necessary.

    -)THIRD ONE: I don’t know pretty well all the classes and objects in Unreal yet, but when you’re defining the attenuation for the cell lighting model in “DeferredLightingCommon.usf”, you’re using ScreenSpaceData.GBuffer.CustomData.x in the definition of the range. If you try to debug, your compiler will tell you there is no such type like ScreenSpaceData. And, indeed, if you search for it, it doesn’t exists. So get rid of it, and you should have a range definition like this one: float range = GBuffer.CustomData.x * 0.5; Try to debug/build and it will do without problems. You’ll be done at this point :D.

    I will like also to remember again that for the latest version the lines are different since there are new shaders! Also remember to define the attenuation just after the declaration of the LightColor variable! And, finally, you should add your pin to MP_CustomData0 or MP_CustomData1 and change the function GetMaterialClearCoat() for GetCustomData0() or GetCustomData1() depending on which one you’ve used your pin (or both!), just like FelixK said.

    Thank you again and I hope this is useful for someone 😉

    1. eyewitness

      Thank you both very very much for the awesome crash course through the rendering pipeline!
      It does take some minor cognition to work out what goes where with all the code-having-been-moved-around, but it can (and should) be figured out if someone is seriously attempting stuff like this.
      As a minor side-note, I’d like to add that, while the first article does say to grab a fresh copy of the editor, this ( in hindsight quite obviously ) means that you do have to get the engine source, because changes made to the files of a project created with the EpicLauncher (even though the engine seems to be there and editable) will not work, as it uses precompiled binaries and doesn’t actually build new ones. Admittedly this caused some confusion for me, as I’ve been on the platform for < 1 month.
      Again, cheers for the excellent guide!

  2. Rawkul

    I did a mistake. I said that deleting ScreenSpaceData fix it, but not. You can compile correctly the editor, but when you try to build lighting it gives an error and I can’t build. Any suggestion on how to fix that? (If I keep it the compiler says there is no identifier called ScreenSpaceData).

    Thank you!

    1. FelixK Post author

      Thank you very much for the replies. (And sorry for the late reply I don’t check the blog that often nowadays)
      I haven’t touched Unreal4 in a while so this is sadly a bit outdated now.

      I tried taking a quick glance at the release branch on Epics Github regarding the ScreenSpaceData.
      In DeferredLightningCommon.usf I could still find the “FScreenSpaceData ScreenSpaceData” parameter in the “GetDynamicLighting()” method. so it’s kind of weird that you get any errors on that. The FScreenSpaceData is from the “DeferredShadingCommon.usf” and uses another struct called FGBufferData to store the gbuffers which is also from the same file.
      This one still contains the “float3 CustomData;”. Maybe the problem lies somewhere else?

      Also if you don’t mind I would add a small comment on the first part that will point to your reply here in case someone else tries this with a newer Unreal version.

      1. Rawkul

        Hey, don’t worry about the late response, I’m also sorry for my late response haha.

        I don’t mind for you quoting my comment, in fact, I posted it hoping It’ll be useful !!

        And in the end I could fix my problem of baking light, by simply recompiling the lighting module of Unreal.

        Thank you again for this wonderful tutorial!

        And, can I ask you something? In your material, is anyway of making the specular light stronger (without changinf the code) ? I mean, for doing metal It’ll be nice.

        1. FelixK Post author

          Specular is part of the 2nd Gbuffer which (most likely) doesn’t use HDR so it’s range will only go from 0 to 1.
          Inputting a higher value will only result in 1 so you can’t do much about it without touching the code.

          One possible way to solve it would be to add a multiplier in the line where the D_GGX() specular calculation method is called.
          For example multiply the inputed specular by 2 to treat a 0.5 value as full specular in the material and a 1.0 value as double specular etc.
          The tradeoff is that the byte used for specular in the RenderTexture would be used for a wider range of values which makes it less precise so it might be best to keep the multiplier at a reasonable level.

  3. CG_Bull

    Thank you blogger, I follow your steps in unreal 4.13 version, found that the number of rows are not right, and the final result is wrong.. Can you update the label?

  4. Steven

    I’m using 4.15 and in order to actually change CustomData in GBuffer, you need to go to BasePassCommon.usf, and add MATERIAL_SHADINGMODEL_STYLIZED_SHADOW to WRITES_CUSTOMDATA_TO_GBUFFER. Otherwise, CustomData is always 0 no matter what you write to it.


Leave a Reply

Your email address will not be published. Required fields are marked *