This is normal 2: Baking normal maps
This is the second part of a tutorial series I did on Artstation doing about normal mapping. You can find the part one here, but its not required to understand this part.
The general idea of baking a normal map is relatively simple: you have a lowpoly with UVs and a highpoly model; and you transfer the normal information from the highpoly to the lowpoly. This way, the lowpoly will bounce light as the highpoly would.
During this process, the baking program will basically cast rays from the lowpoly, following the vertex normals and searching for the higpoly. This is the most important aspect of normal mapping, and most problems people have when working with normal maps are related to this.
If you don´t control the vertex normals of your lowpoly model, you will lose control over your normal map.
Bad normal map correlation
In order to control the smoothing of our lowpoly model, we can have split vertex normals (to create hard edges) or averaged vertex normals (to create soft edges).
Turns out that not all 3D programs use the same calculations to average the vertex normals. This means that your lowpoly will look different and have its vertex normals pointing at slightly different locations depending on your 3D program. This isn´t usually a big problem, since these deviations are very small, but can affect how your model looks; and these differences are exagerated when using normal maps, since your normal maps are modifying the lowpoly normals that are changing between applications.
The 3D industry is working to fix this problem, and a standard has recently appeared, called Mikk space. This is a method of calculating vertex normals that all 3D apps could use, so vertex normal don´t change between 3D programs. Keep in mind that not all 3d apps use it yet.
Another way to reduce this is effect is not to rely too much on normal maps when baking. Try to match your lowpoly more closely to the highpoly and use more hard edges on flat surfaces. This way, your normal map wont have to do all the work and these small deviations will be less noticeable.
Normal map detail skewing
When the computer averages the normal direction of your lowpoly vertex normals, big changes in the angles of your surface can "skew" the lowpoly normals and they won´t be perpendicular to the lowpoly surface.
https://i.gyazo.com/50bd7d9f5f977216b2864fbf950c901f.mp4
Since the normal map baker uses the lowpoly normal directions when searching for the highpoly details, if these directions are skewed; they will appear skewed on the normal map:
This is a very common problem, and several solutions have been found. There isn´t a best solution, it really depends on the geometry.
-Some 3D bakers have the option of rebaking these parts modifying the lowpoly normals temporarily, so they are baked without skewing. Marmoset Toolbag has this option. Reddit user Tanagashi kindly explained to me that some programs such as xNormal can tesselate the lowpoly to add new vertices and make the normals perpendicular to the lowpoly surface, bake an object space normal map and then convert it to tangent space using the original lowpoly normals. Using this new normal map, the program can create masks to control where to use the original normal map and the one created from the tesselated lowpoly.
-Adding vertices will make the transition between the vertex normals less skewed, as one 90º angle can be split into smaller degrees, making the second transition less skewed. This obviously increases your polycount and,since you are adding geometry, I reccomend you use this extra geometry to add a more interesting silhouette to your model.
-Split the averaged vertex normals (making the edge hard/use separate smoothing groups): this way, each vertex will have several normals, each one perpendicular to the lowpoly surface. Keep in mind that, when the 3D program has a split vertex normal; it actually creates a duplicate of the vertex, so this will increase your vertex count and slightly decrease performance. Additionally, hard edges will also give you a "black edge" problem, as we will see later.
-We can modify the normals of our models to bend the normals of our lowpoly so they are perpendicular to the highpoly details. Keep in mind that not all programs allow modified normals (Zbrush only has averaged normals, OBJ and older FBX files dont have custom normal information). There are basically 2 ways of modifying the normals:
- Weighted normals: this is an automatic method similar to the average vertex normals. The idea here is that when averaging the vertex normals not all faces will have the same strenght: larger faces will "pull" the vertex normals towards them with more strenght than smaller faces. This way, larger faces which are usually more important will have better detail projection. This works specially well with highpoly panels.
- Custom normals: using tools from your 3D software, you can bend your lowpoly normals. This is a relatively new idea and there aren´t standarized tools for this. Keep in mind that bending the normals can create very weird unintended shading on other parts of your model, so this technique is usually combined with bevels. Some people call this technique "midpoly modelling".
Cage/baking distance
By default,the rays that are cast from the lowpoly surface travel a limited distance, to prevent the lowpoly from recieving normal information from far away parts of the highpoly. This distance is usually called "frontal/rear distance", as the rays can be casted towards the inside, outside of the model, or both. You can see this distance represented in red in the following image:
Some 3D apps (3ds Max for instance) also allow us to use a cage. A cage is a "copy" of our lowpoly model that we can modify so that it encapsulates the highpoly perfectly. And, in some cases (not all), also allow us to change the direction of the rays, without changing your original lowpoly vertex normals. This can help get the best baking extremes and avoid skews, but keep in mind that you are not baking using the normal direction of your vertex normals, but in the end you will use a normal map to modify the actual lowpoly normals, so the result could look strange.
Edge seams
As we have seen, if you have a model with hard edges (some faces have more than 1 smoothing group, or some edges are set to "hard"), the baking program will split the normal of the vertices between 2 faces. This has good and bad effects.
The good effect is that the normals are not averaged, so there are less normal map distortions: the vertex normals are completely perpendicular to the surface of the lowpoly. It can also help make your lowpoly look better, if it has faces on extreme angles, more appropiate for hard edges.
The bad effect is that there is now a gap between the normals, and this could mean that you lose information if your lowpoly has a gap in the normals where it can´t get highpoly details. Furthermore, some parts of your lowpoly projection could be intersecting, and will compete for the same UV space. Both effects leave a seam along the edge, thats more or less noticeable depending on your engine.
This, however, can be easily avoided using a simple trick: if you have two faces separated by a hard edge, put each face on a different UV island, with some space between them.
If both faces are connected in the UVs, there are drastic changes in color from one face to the other, and color can bleed from one face to the other (because of the processes that occur when rendering faces) which is extremely noticeable in the normal map. By separating the faces in the UVs, the baker can add padding between your faces, and avoid this color bleeding. This video might help you understand this process.
In conclusion:
Once I have my lowpoly model ready and adjusted to the highpoly model as close as possible, I start working on the smoothing before UVs.
I set my smoothing for the lowpoly (if its organic, I start with a completely smooth model, if its hard surface I start with a set angle smooth of 30-60º; and tweak the model smoothing until it looks good).
Once I have a set smoothing for the model, I work on the UVs, making sure that all hard edges are split into separate UV islands (to avoid edge seams).
If I have skewing errors, I add additional edges (usually bevels, to keep a more rounded silhouette). This works for most of my models, but I could also fix the skewing errors if I used Marmoset Toolbag for baking, or by using custom/weighted normals.
If there are projection errors, I modify the baking distance/ cage, modify the lopoly/highpoly so they are better fit for baking, or erase the normal maps on certain, really hard parts such as the tip of a cone.
Next up, I´ll be making a troubleshoothing guide for normal map baking and discuss some of the most common problems and solutions. If you are enjoying these tutorials, please comment so I have some feedback, even if its negative. I´m doing this so I can learn and improve, but a complete silence can be discouraging. Thank you for your time, and I hope you are enjoying these!