This is Normal 4: Normal map troubleshooting
This is the fourth part of a series of normal map tutorials I made in Artstation. You can find the first part here, but its not needed to understand this tutorial.
Here´s a compilation of normal map problems I have seen throught the years, and some of the solutions I know to fix them.
Problem: there are "black lines" or "insets" at the edges of my model.
This happens when you have hard edges in your model, because the vertices of your model have normals completely perpendicular to the polygon surface, and this can cause the baker to generate those black lines in your model.
Solution: normal map bakers take this problem in consideration when creating normal maps, and try to mitigate it by calculating a little bit extra information beyond the vertex normals, but in order to store it, they need a gap between the polygon UVs.
Here´s a more detailed explanation, but the rule of thumb is very simple: whenever you have a hard edge on your model, separate the faces connected by it in your UVs.
Problem: my normal map looks VERY wrong, especially from some angles.
This problem can appear for multiple reasons, let´s discuss some of them:
You are using the wrong tangent space:
The normals on your lowpoly, that we are trying to bend using a normal map, can be calculated differently in the baking program from the program you are using to render the model. If these calculation differ, your normal map can look very strange, specially from some angles.
Solution: always try to use the Mikk tangent space basis to calculate your normal maps. This is a standarized way of calculating normals, that was made to avoid these kind of problems. If your normal map baking program can´t use Mikk, try using a program such as handplane to switch between one tangent space and another.
You are using gamma correction on your normal map:
Normal maps are not regular images with color information. They carry surface normal information, and don´t behave as color images. Gamma correction is an adjustment of the colors of images, and can change the color of your normal map in unwanted ways. To remove the gamma correction in your normal map, change the color space of your normal map to linear/linear color/raw, or untick the sRGB option in Unreal engine.
You are not using a tangent space normal map as a tangent space normal map:
Make sure your engine is not using your tangent space normal map as an object space normal map, a bump map, displacement map, etc...
Your lowpoly normals are different in your baking program from the lowpoly normals in your rendering program:
This can happen if you lose smoothing groups/hard edge information during the export/import processess, if you are using custom/weighted normals and your rendering program doesn´t support them or discards this information.
In this case, compare the lowpoly in both apps and if they look different, try changing the import/export settings, the file formats you are using (obj files lose normal information) and the compatibility of your program with custom normals.
Problem: how do I make a normal map of a spiky cone?
Solution: you... don´t. You don´t need a normal map for everything.
The spiky cone is a classic example of this, but there are many other places where normal mapping just isnt the solution.
We use normal maps to change the normal direction of our lowpoly normals. Sometimes, the direction of our normals is perfectly fine and doesnt need any adjustments and sometimes, the normals of our lowpoly are extremely bended (such as in the case of a spike) and details from the highpoly don´t align with the lowpoly surface properly. In these cases, I simply erase the normal map details using this color:
This color is 50% red, 50% green and 100% blue, and doesn´t change the normal direction of a tangent space normal map, so you can use it to erase details where the projection isnt good.
The spiky cone is just an example of one case where normal mapping might not solve your problems. Whats important is that we remember that there are some cases where a normal map is not the best solution, and that they are limited and we can´t expect them to do what we need for every situation. Sometimes we spend a lot of time trying to make a normal map work when we could just add the details to the diffuse texture or the lowpoly, and not rely on the normal map for that specific detail.
Problem: the details on my model look inverted.
This is a very common problem, and can be seen in a lot of video games, even AAA.
As we saw in this tutorial, normal maps are textures that use the green, red and blue channels of a texture to change how light reflects from the surface of the model when it comes from the side, top and front of our model, respectively (keep in mind this is a simplified explanation and not 100% correct).
The problem is that some apps consider that the green channel should show the model as lit from below while some apps consider it should show the model lit from above. This is sometimes refered to as "normal map right-handiness":
- OpenGL apps (right handed, positive green channel): Blender, Maya, Modo, Toolbag, Unity.
- DirectX apps (left handed, negative green channel): 3DStudio Max, CryEngine, Source Engine, Unreal Engine.
- Substance Painter can work with both and export both types of normal maps.
Solution: invert the green channel of your normal map. Most game engines will have the option in the textures to invert the normal map, or you can manually invert the green channel of your texture in Photoshop (navigate to the channels tab, select the green channel and press Ctrl+i).
Problem: some parts appear flat/missing some detail
When baking normal maps, imagine that the baking program casts rays from the surface of your lowpoly following your lowpoly normals until the rays hit the highpoly and bend. Then, the baking program takes this information and stores it into a normal map.
These rays that have been casted can´t travel forever, because they could hit a far away part of your highpoly and bend incorrectly, so the baking program limits how far away can these rays be casted and, sometimes, the rays could be stopped before they even hit the highpoly. In this case, we lose details and our normal map has zones of flat color.
Solution: depends on how your baking program lets you control the baking distance:
-Some programs will only look for details outside your lowpoly and ignore whats "inside" your lowpoly (though most modern bakers will look in both directions). In this case, adjust your models so the lowpoly completely fits inside your highpoly.
-Other programs such as Max will use cage, a "extruded" version of your lowpoly that you can modify to control with precision the limit of the baking process.
-Other programs let you set a baking distance using a number (max frontal and rear distance in substance Painter).
You can also try to make the lowpoly and/or the highpoly more similar to eachother, so the rays can get detail everywhere in your model. Another option is to bake two normal maps using different cage distances and mix them in different parts of your textures. Some normal map purists might scream at you, so tighten your headphones.
Problem: my normal map has distorted details.
This is a very typical problem. It happens when our lowpoly normals don´t align properly with the highpoly details, so they appear bended (in reality they are perfectly aligned if you look from the vertex normal direction), This usually happens because you have some faces forming an extreme angle.
Solution: I wrote more extensively about this topic in a different tutorial, but the general solutions are:
-Soften your extreme angle by adding a bevel.
-Convert the edge at your extreme angle into a hard edge/separate the faces into different smoothing groups.
-Use custom normals/use weighted normals.
Problem: my normal map looks pixelated or has bands
Earthquake (AKA the god of normal maps) wrote a very good explanation of this problem here.
If your lowpoly and highpoly are very similar, most of your normal map will have the base normal map color, with different color where your lowpoly differs from your highpoly.
If we have the opposite situation, and your lowpoly and highpoly are very different, the normal map will have much higher color variety, and gradients will start to appear:
These soft gradients are troublesome because we need a lot of colors to represent them, and the most common ways of compressing textures are based on reducing the total number of colors.
Solutions:
-Make your lowpoly more similar to the highpoly: this way, the normal map has to do less work, and it will look more similar to the first image, avoiding these large and soft gradients. Modifying the normals of your lowpoly so they align better with the highpoly could also help.
-Use 16bit normal maps: by default, most images use 8 bit color depth. This means that each color channel of your texture can use 8 different values between 0 and 1. When you consider all 3 color channels, this gives us 256 possible colors. When we have soft gradients we might see bands in our model, because the image doesn´t have enough colors to represent such a small change of color.
16 bit images can use 16 different values for each channel, which means up to 65536 possible colors. This means a lot more range for soft gradients. Be aware that 16 bit images are larger in size than 8 bit (because they carry more information). Also, sometimes 16 bit images have alpha channels, and are referred to as 24 bit images.
There are also images with higher bit depth, but they are not used for normal maps as 16 bit is more than enough.
-Use dithering: Lack of colors in our textures is a problem that has been around for decades, and one solution that appeared decades ago was to use dithering: the idea is that we alternate pixels in our texture to represent the gradient, and it works fine when you zoom out the image. You can usually activate it when exporting your texture.
-Make sure your normal map is correctly compressed: when textures are compressed, the computer takes zones of a similar color and merges them to create a "patch" of color, reducing the amount of colors in your image. This is usually fine for regular images, but terrible for normal maps: not only destroys your gradients but can also merge the information in your color channels. There are special compression algorithms designed for normal maps. Make sure your game engine is interpreting the image as a normal map (usually by selecting an option in your texture asset to mark it as a normal map) and the compression settings will be configured automatically.
Problem: there are some visible pixels on some parts of my model
The obvious solution would be to increase the size of your UV island for that part of your model, or use larger textures, but lets take a look at some less obvious solutions:
-Bake at double your final normal map resolution and then reduce the size of your image: If you are using a 512x512 texture, bake your normal map at 1024x1024 resolution and then convert the image to 512x512. This way, each pixel of your final texture will take information from 4 pixels, making a sort of "antialiasing" and reducing the pixellation. This is also true for other baked images as well, and you will also keep a high-res version of your textures in case you need to increase the detail in some zones later.
Notice how in this image the normal maps have the same resolution, but the one we baked at 1024 resolution looks more rounded and similar to the highpoly because it stored some extra information during the reduction process.
-You can stack your UV islands on top of eachother, so they use the same normal map information on different parts of your model. Just make sure you move one side of the model 1 unit outside the UV space so the baker doesn´t try to get details from both sides at the same time.
You can go even further and use trim textures or decals for some details to optimize your texture usage.
-Textures use a pixel grid, and pixels are square. If you have some details that form a line, try to align this line vertically or horizontally. This way, the pixel grid and your texture details will align:
Problem: my model is symmetrical, but the normal map looks different depending on the side.
When applying symmetry to your model, the normal directions can change because the way that faces are connected has changed. Sometimes, this means that you can see a seam right at the center of your model. To avoid it, make sure your lowpoly normals right at the center are aligned and adjust the smoothing if needed.
Another possible cause is triangulation: when importing models to a game engine, they are always triangulated and sometimes, this process can change the lowpoly normals and some artifacts will appear at the diagonal of your lowpoly faces. To void this, triangulate the model before baking, bake the normal map and then apply the symmetry modifier.
Finally, here´s a small tutorial by Earthquake that helped me understand a little bit more about vertex normals and normal mapping. I talked about the same topics throught this tutorial series, but I wanted to include it here.
Also, check the polycount wiki for more information about normal mapping.