This is normal 3: types of normal maps
This is the third part of a tutorial series I did on Artstation doing about normal mapping. You can find the part one here, but its not required to understand this part.
As many things in this industry, normal maps have evolved throughout the years and there are several types of normal maps and they may look different. Heres a compilation of the ones I can remember from memory, but there might be others as well.
Tangent space normal map: this is the most common normal map nowadays, and the one we have been talking about during this tutorial series. It modifies the normal direction of the model, based on the normal direction of its vertices (so we must control the vertex normals of our lowpoly).
Mitkk tangent space normal map: Not every program calculates the average of the vertex normals the same way. This can lead to differences in how a normal map looks in different engines, so we should bake the normal map using the same method as the rendering program will use (this is called "using a synched workflow")
Mikk is a proposed way to calculate vertex normals aimed to be universal, so that every program calculates them in the same way. Workflow-wise, this means that we can use a lowpoly with all its vertex normals averaged (one smoothing group, or all edges smooth), bake a normal map using the Mikk tangent space and it will look just like the highpoly, without having to deal with smoothing errors or separating the hard edges in the UVs. I will upload a tutorial on how to do this in the future.
Keep in mind that this is still a tangent space normal map, but the normals of the models are calculated in a way that is universal and interchangeable between programs.
2-channel tangent space normal map: turns out that using the information stored on two of the three channels of a normal map, the computer can calculate the third one, reducing memory usage but slightly increasing processing usage. Since memory is usually a bigger concern, this optimization is commonly used and some engines do it automatically (ie Unreal engine when we set a texture normal compression to "normal map"). Freeing up a channel on our normal map allows us to reduce the texture size or use the channel for metalness/roughness/opacity...
Usually, the discarded normal map channel is the blue one, so these textures look yellow. This optimization is sometimes done automatically by some engines, so you might see these textures from time to time in your project.
World space normal map: this normal map instead of modifying the normal direction of the vertex normals, ignores them completely and changes how the lowpoly bounces light in the world space (considers the normals of the vertices as aligned with the world when baking).
Think of a tangent space normal map as "you should reflect light to your right" and a world space normal map as "you should bounce light to the east".
These normal maps are more colorful and have more prominent gradients; they were used because you didn´t have to worry about the lowpoly vertex normals, but they have the drawback that you can´t move the model as it will look strange (we are setting a face so it always bounces light to the east. If you rotate it, the face will keep bouncing light to the east.).
World space normal maps are very rarely used in games nowadays, but they can be used to create some nice textures, ie: the blue channel shows how your model should bounce light that comes from the top of the model so you can use this to add a painted light to the texture.
Keep in mind that the world coordinates change between applications: in Unreal, 3D Studio Max, Blender the Z axis is up, while in Maya, Modo, and Cinema4D, Y is up. This means that world space normal maps can break when changing between different applications.
Object space normal map: this is an upgraded version of the world space normal map, and very similar. The idea is that when moving the model in the world, its world space normal map would change to reorient itself relative to the object.
Think of it as "this face will bounce light to the right of the model". If you rotate the model in the world, the normal map would change to reflect this change. However, this doesn´t work with deforming meshes, as it only takes into account the object transform. This is the reason that tangent space normal maps are more widely used today.
Bent normal maps: They basically combine the information of an AO and a normal map, bending the normal directions so that light tends to bounce towards the parts of the model that are exposed to the light.
These are basically used for improving Ambient Oclussion and avoid an effect called "light leaking", where a model could bounce light from parts that light shouldn´t reach. I never personally used them, but I would investigate them if I had a noticeable "light leak". You can find more info here, here and here
16 bit normal maps: sometimes, if we a have a very smooth gradient in our normal map, we can see some banding. This banding comes from the lack of enough colors to represent the smooth gradient, usually from texture compression.
Even then, sometimes we have a large and smooth surface and these problems appear even with an uncompressed texture. In this case, we can use 16-bit normal maps, usually as .tga files, which have more colors and are larger in size then the usual 8-bit normal maps.
You can learn much more about 16 bit normal maps from the god of tutorials himself, Earthquake.
Keep in mind that there are other techniques that can be used to mitigate this problem, such as removing the normal map altogether (use only geometry to represent this smooth surface), making the lowpoly more similar to the highpoly so the gradients are less noticeable, or using dithering.
So, which one should we use?
Mikk Tangent space normal maps are 90% of the time the best option. Unlike world and object space normal maps, your model will be able to deform and the normal direction will remain correct.
You should bake your normal map using the same tangent space as the rendering program. The most used tangent space is Mikk, so you should use it when possible.
And, if your normal map is showing some pixellation, consider using 16 bit normal maps or one of the solutions mentioned above.
Those are basically all the normal maps I have encountered that I can remember. If you know about some other types of normal maps, let me know so I can include them in this tutorial!
Thank you for your time, and I hope you found this useful. Thanks as well to Shnya for his feedback and help.