Page MenuHome

OBJ: support edge-split option for compatibility with software that has incomplete support for normals
Confirmed, NormalPublicTO DO

Description

System Information
Operating system: Windows-10-10.0.18362-SP0 64 Bits
Graphics card: GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 452.06

Blender Version
Broken: version: 2.91.0 Alpha, branch: master, commit date: 2020-08-24 10:28, hash: rBcb8da6efce80
Worked: (newest version of Blender that worked as expected)

Hi, I've just set up a new website 3d-illusions.co.uk to start selling some of my Blender work, and I've run into a bit of a problem trying to export the files to different formats so I can sell to a wider audience. FBX files seem to have completely mangled normals and badly formed ngons when opening in 3ds max 2017, and obj files have the same problem unless choosing to import into max as a single object, which is not an option as the end users wont be able to separate without generating more problems.

I've taken the obj file from blender into Houdini and then after amending some of the primitive path attributes to point to the correct material in the mtl file, saved it straight back out, and that seems to correct the problem. I compared the two obj files in VS CODE, and I think the problem with Blenders output is an insufficient number of floating points, or perhaps it's the different order?

It's a bit problematic because I can't save the file to other formats in Houdini unless I upgrade my licence, which I'd prefer not to do just for this functionality. Is this something that you're able to fix, or is this just an insurmountable limitation of Blender at it's core?

.blend, and both obj files below:

(ignore the name of the file, its the same file I used for a different bug report)

ps USD seems to be exporting to cm rather than metres even if units are set correctly in Blender's scene/units tab......although this could be an issue with Houdini (i'll investigate that further).

Cheers :)

Event Timeline

Seems like it might be a bug in the other software, Blender appears to be generating valid .obj's at-least..

The normals are slightly different but not corrupted, doubt floating point precision is the issue given the ngons. If you look in the face section you will notice that Houdini forces all "vertices" (face corners in blender) to be a unique entry. While blender is making more efficient use of storage by differentiating between "vertex corners" (vertices) and "vertices" (points) in the file contents. It seems that might be causing confusion in that version of 3DSMax (which might not know how to deal with that).

Example:
Blender:

f 284//1 279//2 15//3 9//4

Houdini:

f 284//284 279//279 15//15 9//9

See face elements:
https://en.wikipedia.org/wiki/Wavefront_.obj_file/Face

In the meantime if necessary you might be able to fix that in MeshLab.

Thanks Alex. Could you modify blender's exporter to provide the same data as Houdini's? I'm assuming it's possible as Houdini is generating that additional data from the obj file provided by blender.

Tried meshlab but it appears it has the same problems with the normals as 3ds max does. Perhaps Blender's more efficient approach is actually problematic when it comes to cross application compatibility? I'd definitely rather have a slightly larger obj file than worry the files i'm selling aren't going to work in various applications.

Not very familiar with that area of blender code, just looking to help out the export issues to see if it can be addressed. (Edit, just found myself fixing up the obj .mtl code :P)

I was able to successfully import the model you supplied in the description into the latest version of MeshLab with no visible errors. MeshLab's obj exporter generated similar face elements to Houdini. But interestingly enough it seems that it is the only one that's ngons isn't mangled in Windows' 3D Viewer. For normal issues in the FBX exporter you can try different smoothing group options (normals only/edge/face) and see if that solves normal issues, but if the ngons are giving you issues you can also provide a triangulated/quads and tri version of your model as well if needed. Different software have different algorithms for triangulating ngons, you might be running up against that issue.

This kind of asset distribution is not an unusual practice as game pipelines prefer pre-triangulated meshes for flawless engine import while VFX prefers quaded meshes for deformation and sculpt, ngons can be dangerous when multiple apps are involved.

Hi @Alex Strand (astrand130), great news that you're fixing up the .mtl code :)

Regarding meshlab, here's a different part of the model that shows the shading issues more clearly in meshlab. Be sure to turn on smooth shading in meshlab to see the problem.

Regarding FBX, I've tried every concievable combination of settings to no avail, so i'm guessing fbx is even more susceptible to blender's more efficient but less robust geometry storage.

OBJ, It is very strange that in 3ds max the problem can be solved by importing all obj objects as a single mesh, perhaps that would be a clue for someone with more knowledge of the obj format? Users can't really import as single objects though, because it's not possible to keep the normals in max when detaching elements (that I know of).

Ah, Thats a rather interesting series of artifacts... Did not see that when inspecting the handle in MeshLab. Almost looks like tangent space mismatch but without the tangent normal map. I will take a look at it later. My PC is in maintainence for a bit. (HDD replacement likely coming up) Are the shading issues fixed when re-calculating the normals in 3DSMax? What about triangulating before export? I have a couple more theories on what that shading error might be caused by.

Since I started adding PBR support to mtl I could probably just fix this task if it turns out to be a bug on Blender's end.

Ouch, good excuse to get an m.2 (they're very cheap at the moment apparently). Retriangulation makes the issue even worse, recalculating the normals doesnt help either. The only thing that helps is to import the obj as a single object rather than keeping all the individual pieces. FBX just breaks completely where there are ngons.

Alright. Let me try to separate the two issues.

FBX is broken with Ngons: In my experience that is usually a mismatch between different software's handling of ngon tessellation. (Might be mistaken but I would need more information/screenshots of shading, triangulated wireframes comparing Blender with 3DSMax).

Weird Obj faceting... I have a feeling that (if it is a blender issue) it could be related to either non-unit normal vectors or the loading of that vertex data. If I could have access to the blend files (or parts of them) that you used to make "problems with normals after import.obj" that will be helpful. I will try loading that into meshlab or (a trial of max if necessary) to replicate the issue.

I cannot guarantee any fixes soon, I need to work on my PC and submit other code before 2.91 goes to bug fix only on Sep 16. But I will try my best when in the clear.

In the meantime keep searching for any potential viable workarounds or fixes in your pipeline. I know a lot of posts on forums like Polycount and Art Station have good information on normal/tangent space/topology issues when going between applications. Best wishes.

This seems more like a bug in 3DS MAX, did you report a bug to them?

How can it be a bug in 3ds max if it’s happening in multiple softwares, but not if the obj is recreated in Houdini?

It's not out of the question that the geometry is perfectly valid but there is a bug in the importers by other software. I will try to verify whether or not that is the case in the coming weeks.

Alex Strand (astrand130) renamed this task from exported obj files can't be opened in 3ds max to Obj: Exported normals compatibility issues.Thu, Sep 3, 12:47 AM
Alex Strand (astrand130) changed the task status from Needs Triage to Confirmed.Mon, Sep 7, 11:12 AM

After going between software packages a bit I can confirm that the issues with mangled ngons stems from a mismatch in tessellation algorithms and can be fixed with the "triangulate faces" option.

As for the normals I was able to replicate the faceting issue with the model and in a couple of tests in MeshLab and TopoGun3. Windows 3D Viewer and Blender were both able to load the Obj with perfect normals. The obj file is completely valid, the other importers are the issue.

However when deleting the normal information from the file the results pointed to something more clear... The results in MeshLab and TopoGun3 remained the same... Blender loaded the model with flat shading and Windows 3D Viewer said the file had been corrupted. This indicates that some software when loading an Obj file does not load normals and instead generates them using their own algorithms. This creates broken results with auto-smooth (which works on a continuous set of vertices with normal discontinuities between splits stored in face corners). Any software that bothers to actually load the normal data will produce correct results while software that doesn't will have inevitable artifacts. Production 3D models often don't have edge splits or the need of auto-smoothing as they primarily focus on creating quad control surfaces for subdivision surfaces, while often most models for 3D Design are created in packages that primarily use smoothing groups, low poly assets are still a nieche category as such it makes sense why many 3D packages would opt not to load normals from Obj (or other formats)... (although it would certainly help to ask them on their forums)

Solution: Substitute Auto-Smooth for the Edge Split Modifier to create vertex discontinuities to help software that generates normals better understand and resolve edge splits. This could be done by the user or by adding such an option to the exporter.
Todo: Add Split Edges option to Obj Exporter

Alex Strand (astrand130) renamed this task from Obj: Exported normals compatibility issues to Obj: Export needs edge split option for compatibility.Mon, Sep 7, 11:15 AM

ps USD seems to be exporting to cm rather than metres even if units are set correctly in Blender's scene/units tab......although this could be an issue with Houdini (i'll investigate that further).

Please open a new issue for this if you can confirm it. Thank you!

@Alex Strand (astrand130)

  • Is it a correct statement that Blender is writing valid OBJ files that other software fails to load?
  • Does any software load Blender's normal data correctly?
  • Any software that ignores custom normals should load the geometry still, although it seems some applications don't?
  • Is there a list of applications that don't support OBJ normals & give problems when normals are written? (besides 3DS-Max 2017)

@Alex Strand (astrand130) I'm getting the normals issue when importing and choosing to use the files normals (not recalculating). Triangulating resolves the broken ngons, but makes shading broken (I believe even if loading the file back into blender).

Don't worry if you can't find the problem though, it's no problem to just keep regenerating the obj in Houdini to ensure maximum compatibility across softwares.

@Alex Strand (astrand130)

  • Is it a correct statement that Blender is writing valid OBJ files that other software fails to load?
  • Does any software load Blender's normal data correctly?
  • Any software that ignores custom normals should load the geometry still, although it seems some applications don't?
  • Is there a list of applications that don't support OBJ normals & give problems when normals are written? (besides 3DS-Max 2017)
  • Yes, Blender is creating valid OBJ files. OBJ however is a format with a rather non-standardized ecosystem of files/exporters/importers.
  • Yes, there is software that loads normal information from OBJ, but there are others that don't and generate normals from OBJ smoothing groups.
  • Yes, software that doesn't load normals loads the geometry. However auto-smoothing has a tendency to break their calculation of normals. (wasn't able to figure out why, but edge splits worked as a more correct substitute in this scenerio)
  • In addition to 3DSMax other software (I tested) that has this issue includes MeshLab and Topogun3.

I was initially considering marking this invalid but upon reflection this seems like it should be a compatibility option to add to the exporter, thus I strongly suggest adding the ability to apply edge splits replicating auto-smooth at export time to maximize compatibility with other software.

@Alex Strand (astrand130) I'm getting the normals issue when importing and choosing to use the files normals (not recalculating). Triangulating resolves the broken ngons, but makes shading broken (I believe even if loading the file back into blender).

Don't worry if you can't find the problem though, it's no problem to just keep regenerating the obj in Houdini to ensure maximum compatibility across softwares.

Okay, so when you sent me the picture of the normal faceting in MeshLab I assumed that is the error you are referring to having in Max as well. When testing in a variety of software I isolated that issue to what I previously described. But it appears you are having more than one issue in your workflow (uniquely specific to Max?) that you are describing in this bug report?

Triangulating resolves the broken ngons, but makes shading broken (I believe even if loading the file back into blender).

Triangulating NGONs with different algorithms will usually result in different barycentric interpolation patterns, this is likely the issue you are experiencing. This is the reason that most production environments avoid NGONS. (I'm going to drop the NGON subject as there is not much you can do about that).

I am trying to isolate this issue to a specific cause, it would help if we could narrow this down to a specific issue.

Hi Alex. the visual appearance of the normals in max and meshlab appeared to be identically incorrect. Max is able to read the split normal data because if the option to import as a single object is checked, then the file normals appear fine (i'm selecting to use the file normals and not to recalculate them at import). If I resave the obj file in Houdini then max can import as multiple objects without the normal issue, so it seems that Houdini is producing more widely compatible obj files. I think you mentioned it was because Houdini adds additional vertex normal data whereas blender does it in a more efficient way?

It's no biggy anyway, close it down if you like. This is probably something I can code myself when things quiet down with client work.

Campbell Barton (campbellbarton) changed the subtype of this task from "Report" to "To Do".Wed, Sep 9, 8:00 AM
Campbell Barton (campbellbarton) added a project: Restricted Project.

@Alex Strand (astrand130), thanks for the feedback, setting as a paper-cut.

Campbell Barton (campbellbarton) renamed this task from Obj: Export needs edge split option for compatibility to OBJ: support edge-split option for compatibility with software that has incomplete support for normals.Wed, Sep 9, 8:00 AM

Hi Alex. the visual appearance of the normals in max and meshlab appeared to be identically incorrect. Max is able to read the split normal data because if the option to import as a single object is checked, then the file normals appear fine (i'm selecting to use the file normals and not to recalculate them at import). If I resave the obj file in Houdini then max can import as multiple objects without the normal issue, so it seems that Houdini is producing more widely compatible obj files. I think you mentioned it was because Houdini adds additional vertex normal data whereas blender does it in a more efficient way?

It's no biggy anyway, close it down if you like. This is probably something I can code myself when things quiet down with client work.

Don't worry about it. Your' report actually highlighted an important issue that needs to be addressed in obj I/O.

If you want to investigate further: Try deleting the normal data in the file in a text editor and re-importing it in MAX. If the issue persists it should validate my assumption, but If it fails to load in MAX or produces different results that indicates that there is something else going on. If so edge split modifier in Blender will create similar normals/topology continuity as in the houdini example, no need to buy an expensive license to another software.

Awesome, thanks dude.

in my models I'm generating custom normals from the data transfer modifier to correct normals after boolean or knife project. Are you saying if I set the edge angle to something really low in the edge split modifier, like 0.00001, then all edges will be split and the custom normals will be re-assigned to each of the corner vertices...so still appearing smooth where necessary even though they're split?

@Alex Strand (astrand130) I decided to have a go at the test you mentioned. stripping the normals resulted in the model importing without the shading artefacts that were being caused by Blender's obj file but not Houdini's obj file, this is because max was forced to recalculate it's own normals rather than use those Blender had created in the file. Here's the file if you want to give it a try:

If i'm understanding correctly, blender writes out point normals (in houdini language) because there only exists one vertex at corners unless they're purposefully split either using mark as sharp, edge split modifier, or weighed normal modifier, whereas houdini writes out vertex normals? So basically I just need to use Bmesh to create one corner vertex per connected face, and transfer the value of the point normal to each of the new vertices? sorry, haven't really looked into Bmesh and how blender stores geometry yet, I think it uses just one corner vertex for all connected faces?

blender writes out point normals (in houdini language)

No, it writes face corner normals (vertex in houdini language)

or weighed normal modifier

I don't think the weighted normal modifier creates vertex discontinuities (might be mistaken) but you can tell by applying the modifier and seeing if there are "doubles". (we need to create those double vertices for compatibility)

houdini writes out vertex normals

"point" (vertex in blender language) normals (I believe? looked that way in the files)

Whatever code you write should replicate the results of the applied edge split modifier. (haven't worked much with bmesh, I can't help you there)

Ah ok, i'll just try splitting the entire model up into seperate faces and then using the data transfer modifier to project the original normals back onto the new vertices.

That will make every face faceted as other programs generate their normals based on topology continuity, by doing that you just made every face discontinuous. You need to substitute the edge split modifier for whenever you use use auto-smooth. Normals of any kind will not be loaded in these applications.