Normal Map Baking Doesn't Flip Back Faces #74553

New Issue

Joseph Eagar · 2020-03-08T22:27:47+01:00

Joseph Eagar commented

2020-03-08 22:27:47 +01:00

When baking with "selected to active", Cycles fails to detect when it's hit a back face and needs to flip the normal. This is important since it's the nature of baking low-poly objects from high-poly ones that polygon intersections happen, which leads to flipped normals in the final map. Offsetting the cage vertices along their normals (the "cage" option") to avoid intersections gives visual errors on fine concave features like mouth corners, noses, etc.

I wrote an OSL shader to bake out normals maps with properly flipped normals. Here is a comparison of a normal map baked with a tweaked cage offset (top) and the shader (bottom).

On a related note, it might be a good idea to flip all baked normals greater than 90 degrees to the original surface normal for tangent space baking.

Example file: NormalMapExample.blend

When baking with "selected to active", Cycles fails to detect when it's hit a back face and needs to flip the normal. This is important since it's the nature of baking low-poly objects from high-poly ones that polygon intersections happen, which leads to flipped normals in the final map. Offsetting the cage vertices along their normals (the "cage" option") to avoid intersections gives visual errors on fine concave features like mouth corners, noses, etc. I wrote an OSL shader to bake out normals maps with properly flipped normals. Here is a comparison of a normal map baked with a tweaked cage offset (top) and the shader (bottom). ![image.png](https://archive.blender.org/developer/F8395753/image.png) On a related note, it might be a good idea to flip all baked normals greater than 90 degrees to the original surface normal for tangent space baking. Example file: [NormalMapExample.blend](https://archive.blender.org/developer/F8398363/NormalMapExample.blend)

Joseph Eagar commented

2020-03-08 22:27:47 +01:00

Added subscriber: @JosephEagar

Joseph Eagar commented

2020-03-08 22:28:41 +01:00

Here's the normal map osl shader.

bake_normals.osl

Here's the normal map osl shader. [bake_normals.osl](https://archive.blender.org/developer/F8395764/bake_normals.osl)

Joseph Eagar commented

2020-03-08 22:33:30 +01:00

For anyone who tries the shader, you have to bake in "Combined" with "View Transform" under Color Management set to "Raw."

Germano Cavalcante commented

2020-03-09 15:45:47 +01:00

Added subscriber: @mano-wii

Germano Cavalcante commented

2020-03-09 15:45:47 +01:00

I'm not sure if this is a bug or a request for modified/improved behavior.

For user requests and feedback, please use other channels: https://wiki.blender.org/wiki/Communication/Contact#User_Feedback_and_Requests

For more information on what is considered a bug, visit: https://wiki.blender.org/wiki/Reference/Not_a_bug

I'm not sure if this is a bug or a request for modified/improved behavior. For user requests and feedback, please use other channels: https://wiki.blender.org/wiki/Communication/Contact#User_Feedback_and_Requests For more information on what is considered a bug, visit: https://wiki.blender.org/wiki/Reference/Not_a_bug

Joseph Eagar commented

2020-03-09 16:55:04 +01:00

It's a bug.

Germano Cavalcante commented

2020-03-09 18:56:56 +01:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Germano Cavalcante commented

2020-03-09 18:56:56 +01:00

In this case we need a simple .blend file showing the problem (as explained in our submission template and guidelines).
Also explained here: https://www.youtube.com/watch?v=JTD0OJq_rF4

(But I'm still not sure if inverted normal should be seen as a bug)

In this case we need a simple .blend file showing the problem (as explained in our [submission template and guidelines](https://developer.blender.org/maniphest/task/edit/form/1/)). Also explained here: https://www.youtube.com/watch?v=JTD0OJq_rF4 (But I'm still not sure if inverted normal should be seen as a bug)

Joseph Eagar commented

2020-03-10 02:21:26 +01:00

It's definitely a bug, here's an example image.

And here's a file. Just press "bake"

NormalMapExample.blend

Here's a file with the OSL shader to demonstrate proper behavior (same as before, just press bake):

NormalMapExampleOSL.blend

Thanks.

It's definitely a bug, here's an example image. ![image.png](https://archive.blender.org/developer/F8398354/image.png) And here's a file. Just press "bake" [NormalMapExample.blend](https://archive.blender.org/developer/F8398363/NormalMapExample.blend) Here's a file with the OSL shader to demonstrate proper behavior (same as before, just press bake): [NormalMapExampleOSL.blend](https://archive.blender.org/developer/F8398364/NormalMapExampleOSL.blend) Thanks.

Germano Cavalcante commented

2020-03-17 20:29:41 +01:00

Changed status from 'Needs User Info' to: 'Needs Triage'

Ankit Meel commented

2020-03-18 20:35:08 +01:00

Added subscriber: @ankitm

Ankit Meel commented

2020-03-18 20:35:08 +01:00

I get

Blender.app/Contents/Resources/2.83/scripts/addons/cycles/ui.py:1912
rna_uiItemR: property not found: BakeSettings.use_pass_subsurface

and

Circular dependency for image "Head2B_Normals.png" from object "Topology"
Uninitialized image "Head2B_Normals.png" from object "Topology"

I get ``` Blender.app/Contents/Resources/2.83/scripts/addons/cycles/ui.py:1912 rna_uiItemR: property not found: BakeSettings.use_pass_subsurface ``` and > Circular dependency for image "Head2B_Normals.png" from object "Topology" > Uninitialized image "Head2B_Normals.png" from object "Topology"

Germano Cavalcante commented

2020-03-19 16:08:11 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Germano Cavalcante commented

2020-03-19 16:08:11 +01:00

Marking it as a bug.
Needs a closer look to see if flipping would break other things but is worth investigating.

Marking it as a bug. Needs a closer look to see if flipping would break other things but is worth investigating.

Andy Davies commented

2022-01-24 14:36:56 +01:00

Added subscribers: @dfelinto, @AndyDavies-3

Andy Davies commented

2022-01-24 14:36:56 +01:00

Hey everyone,

This 100% isn't a bug. The cage just isn't extruded enough to fully encompass the high poly mesh. There will usually be intersections in areas of geometry that are very close together, such as the corners of the mouth and in between fingers, but in most cases these are not noticeable when applied on the mesh. The correct fix for this would be to have the mouth and fingers opened wider when baking, or make a custom cage mesh that has no intersections. I just want to note that all bakers have this issue. It is absolutely not specific to Blender. :)

In any case, we really shouldn't be considering flipping normals when baking under any circumstances, either in the shader or on the mesh during baking. There are legitimate reasons why someone may want to bake with inverted faces, and it's a very bad idea for the vertex normals of the mesh to not match the baked normal map.

Rather than flipping the normals we should instead, discard back face hits in situations where the lowpoly and highpoly normal are opposite and use the closest ray hit if there is no ray collision between high and low poly meshes (assuming we don't do these already...it's been a while and I cant remember :P).

@dfelinto @mano-wii If further confirmation is needed I can prove the "not a bug" status, but I'm a little short on time right now, so I can't post images just yet. Please let me know.

Hey everyone, This 100% isn't a bug. The cage just isn't extruded enough to fully encompass the high poly mesh. There will usually be intersections in areas of geometry that are very close together, such as the corners of the mouth and in between fingers, but in most cases these are not noticeable when applied on the mesh. The correct fix for this would be to have the mouth and fingers opened wider when baking, or make a custom cage mesh that has no intersections. I just want to note that all bakers have this issue. It is absolutely not specific to Blender. :) In any case, we really shouldn't be considering flipping normals when baking under any circumstances, either in the shader or on the mesh during baking. There are legitimate reasons why someone may want to bake with inverted faces, and it's a very bad idea for the vertex normals of the mesh to not match the baked normal map. Rather than flipping the normals we should instead, discard back face hits in situations where the lowpoly and highpoly normal are opposite and use the closest ray hit if there is no ray collision between high and low poly meshes (assuming we don't do these already...it's been a while and I cant remember :P). @dfelinto @mano-wii If further confirmation is needed I can prove the "not a bug" status, but I'm a little short on time right now, so I can't post images just yet. Please let me know.

Germano Cavalcante commented

2022-01-24 21:55:01 +01:00

Changed status from 'Confirmed' to: 'Archived'

Germano Cavalcante closed this issue

2022-01-24 21:55:01 +01:00

Germano Cavalcante commented

2022-01-24 21:55:01 +01:00

I can't think of a use for inverted face normals, but since "all bakers have this issue", this seems a strong argument for not considering this a bug.

In fact, apparently a so-called "Green Normal Map" is used in some cases. See https://www.reddit.com/r/blender/comments/ibjg6d/what_is_the_difference_between_green_and_blue/

So it doesn't seem right to remove this "green normal map" in Blender because the faces are inverted.

The ideal would be to add options for baking normals. These options would be something like:

Flip negative normals
Flip positive normals
Discard negative normals
Discard positive normals

Adding options involves deep code changes and is not intended to fix a bug.

I agree that the lack of these options can make the baking workflow limiting, but this kind of change needs to be in a baking ToDo task and not a Bug Report.
See #68925 (Baking workflow revamp).

So I'm closing the report.

I can't think of a use for inverted face normals, but since "all bakers have this issue", this seems a strong argument for not considering this a bug. In fact, apparently a so-called "Green Normal Map" is used in some cases. See https://www.reddit.com/r/blender/comments/ibjg6d/what_is_the_difference_between_green_and_blue/ So it doesn't seem right to remove this "green normal map" in Blender because the faces are inverted. The ideal would be to add options for baking normals. These options would be something like: - Flip negative normals - Flip positive normals - Discard negative normals - Discard positive normals Adding options involves deep code changes and is not intended to fix a bug. I agree that the lack of these options can make the baking workflow limiting, but this kind of change needs to be in a baking ToDo task and not a Bug Report. See #68925 (Baking workflow revamp). So I'm closing the report.

Andy Davies commented

2022-01-24 22:48:48 +01:00

@mano-wii Inverted faces normals can be used to fake SSS in real time.

The "green normal map" is one of two things. It's either a compressed normal map using BC5 in the game engine, which removes the blue channel and then recreates it in the shader, or it's a derivative map (a coarse version of which can be made simply by inverted the red and green channels of a tangent space normal map and filling the blue channel with black. Regular tangent space normals cannot contain inverted colours under usual circumstances without custom support in the engine.

I would strongly recommend against adding any options to flip normals in the normal map. If the vertex normals in the mesh do not match the mesh then it will break the sync between baker, normal map and engine and the tangent basis will no longer match. Discarding the normals in the way I mentioned above is the correct way forward. We really don't want to give people more opportunity to break their normals unknowingly. Maintaining tangent basis sync is enough of a headache already :)

@mano-wii Inverted faces normals can be used to fake SSS in real time. The "green normal map" is one of two things. It's either a compressed normal map using BC5 in the game engine, which removes the blue channel and then recreates it in the shader, or it's a derivative map (a coarse version of which can be made simply by inverted the red and green channels of a tangent space normal map and filling the blue channel with black. Regular tangent space normals cannot contain inverted colours under usual circumstances without custom support in the engine. I would strongly recommend against adding any options to flip normals in the normal map. If the vertex normals in the mesh do not match the mesh then it will break the sync between baker, normal map and engine and the tangent basis will no longer match. Discarding the normals in the way I mentioned above is the correct way forward. We really don't want to give people more opportunity to break their normals unknowingly. Maintaining tangent basis sync is enough of a headache already :)

Dalai Felinto commented

2022-01-25 17:02:31 +01:00

Thanks Andy. Germano for the records, I agree to what Andy say (he is my go-to artist when it comes to baking). Daniel Bystedt also agrees with Andy's point of view:

"Rather than flipping the normals we should instead, discard back face hits in situations where the lowpoly and highpoly normal are opposite and use the closest ray hit if there is no ray collision between high and low poly meshes"

Thanks Andy. Germano for the records, I agree to what Andy say (he is my go-to artist when it comes to baking). Daniel Bystedt also agrees with Andy's point of view: "Rather than flipping the normals we should instead, discard back face hits in situations where the lowpoly and highpoly normal are opposite and use the closest ray hit if there is no ray collision between high and low poly meshes"

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Normal Map Baking Doesn't Flip Back Faces #74553