Overview to “Toon/Cel shading”

CelShadingTitle

For the last couple of weeks I often had discussions about toon / anime esque shading.
In this post I want to list a few approaches I tried out.


 

Shading Methods


Flat shading / Unlit

ExampleFlat
This is not directly shading but it should still be mentioned since this is obviously the cheapest way and is often used for weak performance platforms like mobile.
If you go for this approach you should also paint in the shadows.

 

MatCap

ExampleMatCap
You might have heared of this before if you have looked into ZBrush or Mudbox.
By sampling a round spherical texture we can fake light distribution from a locked perspective (turning the object will move the “lightsource” / view too.
It is useful if you have a locked perspective and lightsource as it is pretty cheap but that is the limit on what it can do.
For this approach we only want a light distribution grayscale map instead of the full material so we can later use it together with our main textures:

MatCapMap
The calculation for getting the coordinates for sampling the map simply require us to calculate the view normal in the vertex shader:

float2 matCapCoord = ((mul((float3x3) Matrix_Inverse_WorldView, normal) + 1) * 0.5).xy; // "normal" is the unmodified input normal
matCapCoord.y = 1 - matCapCoord.y; // Depending on our environment we need to invert the y coordinate

 

Step shading

ExampleSteps
A really simple approach.
We start of by doing a simple nDotL calculation to get the basic light information. Casted shadows can also be multiplied into it.
From here on we use a simple calculation to remap the light/shadow gradient into multiple steps.

float nDotL = dot(wNormal, lightVector);
float light = floor(nDotL * _StepCount) / stepcount; // _Stepcount is an input value that depends on how many color tones we want

 

Gradient shading

ExampleGradient
The last approach I list here uses a gradient texture to define our steps.
We simply use our NdotL calculation as x coordinate input for sampling the gradient texture.

GradientMap

float nDotL = dot(wNormal, lightVector);
float light = tex2D(_GradientSampler, float2(NdotL, 0.5));

 

Fresnel
This is not used for the light calculation itself but can add a nice touch depending on what kind of style we want to go for.
It can either be used together with the basic shading or be added ontop afterwards.
The example below is just for getting the basic outside glow. Fresnels mostly use a pow() node to manipulate the falloff but a smoothstep could also be used.

float nDotL = dot(wNormal, lightVector);
float nDotV = dot(wNormal, viewVector); // The basic view dot normal calculation
float fresnel = 1 - nDotV; // We want to invert our fresnel here so that the glow goes to the outside

 

Applying color


Multiply

ExampleMultiply
The probably most common approach for coloring.
By multiplying our color / texture with one of the light calculations of the previous section we can get a basic cel shading effect.
There are a few options here as to multiply the texture by itself to colorize darker areas a bit more.

 

Shift hue

ExampleHueShift
An alteration to the multiply. By calculating a hue shift we try bringing more variation into our shadows.
It might look more interesting but is not really a practical approach as we need to make a lot of calcluations to get the color to hsv space and then back to rgb after the hue shift which is a bit expensive.

 

Gradient

ExampleColorGradient
This one is a bit special and can’t be used in all kind of situations.
It is similar to the gradient shading process I described above but can use a palette of multiple gradients. (Not in this example since it would require some work to make the index texture)

ColorPaletteMap
At the time we are sampling our gradient texture the y coordinate will be used to decide which palette we want to use.
This gives us the ultimate freedom over how our shadows should look but at the same time limits us in many ways.
Blending multiple gradients is a bit problematic so we can either limit it to hard edges only or use another “outline” mask to hide borders between colors.
Ways to assign the palette index could be either by directly vertex painting it into the red channel of our mesh or having a grayscale lookup texture. (No bilinear filtering  for that texture since it would blend borders)

float index = tex2D(_MainTex, texcoord); // Or vertexcolor.r depending on our setup
float color = tex2D(_GradientSampler, float2(light, _Index));

 

Shadow textures

ExampleShadowMap
A really simple but effective way to solve our problem and the approach I used for this character.
By providing a second texture that holds a fully shadowed version of our main texture we can gain full control over our shadows.
This obviously only works when we have a clear shadow/light situation and comes at the cost of that second texture.
A clear advantage of it is that we can already pre-paint shadow areas into our main texture and blend them together with the calculated ones without getting any visible borders.

float mainTex = tex2D(_MainTex, texcoord);
float shadowTex = tex2D(_ShadowTex, texcoord);
float color = lerp(shadowTex, mainTex, light);

 

Outlines


Sobel

The sobel part is a bit blurry since I had to use an AntiAliasing image effect to get rid of jagged borders.

Inverted mesh
An often used method for creating outlines. By using an extra pass we calculate a copy of our geometry that has it’s cull mode inverted so that the faces look inside.
Then we need to push our vertices out inside the vertex shader.
We can either return 0 for a black outline in our fragment/pixel shader or sample our texture again and make it darker to have outlines similar to the “inside” colors.
This method requires us to have smooth normals or holes will appear when we push the vertices out. Another problem is that certain details like mouth or eyes can get outlines we don’t actually want.

float4 pos = mul( MVP_Matrix, vertexPos ); // Calculate our basic rendering position like usual
pos = pos + lineThickness * mul( MVP_Matrix, normalize(normal)); // Make the outline by pushing it into the normals direction; We can also multiply the distance to the camera here to render a thicker line the farther we zoom out

 

Sobel
A post process way to do outlines.
This works by sampling a scene texture (either colors, normals, depth) multiple times with an offset into different directions.
Then it compares the overlying pixels and if the difference is bigger then a threshold it is considered to be an outline.
If we want only certain objects to have an outline we have to provide some kind of mask beforehand.
I won’t include a code example in here but if you are using Unity you can look into the standard ImageEffects package to see their edge detection effect implementation for it.
For an Unreal implementation (that also takes colors into account) you can take a look at the node network in the last part of the Unreal shading model blog post.

 

Other useful hints


Prepaint shadow areas
If we provide a grayscale texture or vertexpaint beforehand we can manipulate the light calculation a bit.
This texture would hold informations about how much the NdotL calculation would affect certain areas and can guide the shadows a bit.

 

Check your geometry
The hard borders between light and shadow in most toon shading approaches make hard angle switches in the geometry really apparent.
If you have a character try to model it so that the mesh flow is really smootlh and try to avoid abrupt changes in the angle.
Usually more vertices here mean you get a better looking lighting.

 

Normal maps
Another really nice addition. Using normal maps for toon shading works fine but is often less apparent.
It can’t completly replace geometry but can still be used to add a bit of detail.

 

Multiple lightsources (multi pass)
This might be a bit tricky because we can’t rely on additive blending. (It would break our “handpicked” colors)
One way to do it is by using a hard zero/one alpha blending to only add light areas ontop of the previous passes.
Another a bit difficult way is to calculate a grayscale light situation first and apply our colors in a post process.
Using a deferred pipeline here could greatly help us with the second one.

 

Hatching

ExampleHatching
Sometimes you want to use hatching textures instead of shadows. This is pretty easy.
We need to input a hatching texture (grayscale) into our shader and scale it not with the texture coordinates of the object but the screen coordinates.
We also need to repeat it depending on the resolution. The last step is simply to use our shadow as a mask for the hatching texture.

 

Smoothstep
A really handy function to remap a gradient. You can use this to bring the gradient down to a really thin line to avoid too hard edges at the light / shadow borders.

 

That’s it for now. I will try adding more information to it at a later date if I find something interesting.

4 thoughts on “Overview to “Toon/Cel shading”

    1. FelixK Post author

      Sorry kind of forgot to answer until now.

      The question is kind of hard for me to answer since all of the mentioned features would be added as normal.
      There is nothing really special about those in Toon Shading.

      – Shadow casting depends on what engine you use. Unreal should usually do that for you. In Unity you need to add another pass to your shader (which is called per light). You can find a lot of information about that pass online since it’s the same as for non toon shaders.

      – Indirect light is something I haven’t bothered with too much since it would most likely only be calculated for static geometry. If you want to include lightmaps or similar you can do that as you would normaly do in the first pass of the shader.

      – As for reflections you could either sample a cubemap or planar reflection. Or if you are using an advanced deferred rendering system you can go for SSR. If you want to stylze the reflection itself you would do that after sampling it’s texture.

      Reply
      1. Alexander

        Thanks for your answer.
        – indirect lighting is the biggest problem in my opinion.

        Example: an open window, and a room without lights, and a character inside the room. For a static environment, this is not very important and you can use the engine tools, bake GI or dynamic GI solutions.

        But for the characters, I have not found any acceptable way to dynamically stylize this light. In particular, different dynamic GI solutions for UE4 do not really help with this.

        I understand that dynamic GI is a common problem for game graphics, and the number of solutions is quite small.
        Maybe there are some tricks to dynamically highlight a character, for example, from window openings? Perhaps there is some rough imitation? Are there relatively quick NoL solutions for rectangular light sources?
        What are the solutions?

        Reply
        1. FelixK Post author

          Sorry for the late answer. Not sure what to write since (as previously mentioned) I didn’t look too much into this topic.

          I guess if you mention rectangular lights you are talking about area lights? Those are usualy a pre calculated feature since they are too expensive for runtime calculation.

          Three things I could think of for a plain rectangular light imitation:
          – Try to imitate directional lights but instead of including everything the player can see use the directional lights “orthographic camera” to only include the rectangular lights path.
          – Use the decal system or a projector (in Unitys case) to project a fake light decal.
          – Use Unreals LightFunction to manipulate a spotlight so look like a rect. There is still the problem that the spot light becomes bigger over distance but maybe it can work in some cases.

          Reply

Leave a Reply

Your email address will not be published. Required fields are marked *