Page MenuHome

Import/Export: Wavefront Obj Mtl Overhaul (Improved Compatibility and PBR Extension Support)
Changes PlannedPublic

Authored by Alex Strand (astrand130) on Fri, Sep 11, 12:21 PM.
Tokens
"Love" token, awarded by Edmis."Love" token, awarded by donmccurdy."Burninate" token, awarded by xdanic."Love" token, awarded by ajohnson223."Love" token, awarded by filibis."Love" token, awarded by lcs_cavalheiro."Like" token, awarded by Fracture128.

Details

Summary

This patch overhauls the reading and writing of Obj's Mtl files to improve compatibility and use an existing PBR extension better support the Principled BSDF Shader.

Depends on: D4971 D8777

Issues with previous Exporter:

  • Shoved Roughness in Gloss Map
  • Shoved Normal Map in Bump Map
  • Shoved Metalic in Environment Map
  • Did not specify alpha from color maps

Issues with previous Importer:

  • Generated Metal from Ambient Light Color
  • Incorrect Unpacking of Legacy Specular Exponent to Roughness
  • Relied too much on Illumination Model
  • Loaded Normal Map from Bump Map (this gets to a broader ecosystem problem)
  • Cannot load alpha from color maps

This version of the importer/exporter:

  • Adds support for D4971
  • Improves the handling of legacy mtls (PBR extension not supported) to give more acceptable results in a PBR environment
  • Improves export compatibility to legacy rendering environments
  • Implements alpha channel selection "-imfchan m" to better support alpha masked textures
  • Supports Disney PBR with the following extension: http://exocortex.com/blog/extending_wavefront_mtl_to_support_pbr (currently used in TinyObj)
  • Makes a distinction between "bump" and "normal" using the PBR normal map extension (this prevents loading height bump maps as normal maps)
  • Makes a distinction between "dissolve" and "transmission" opacity
  • Improves set up of transparency/refraction in eevee
  • Drops legacy faked reflection mapping
  • Adds context switching to properly load alpha/transmission/translucency
  • Improves conformance to 1995 alias/wavefront specification while accommodating common community usage
  • Compatibility options for loading non-conformant files
  • Other fixes

This not only aims to address issues in blender's loading of obj files but also aims to improve the quality of material data in the obj ecosystem and push the adoption in PBR modernization of the MTL format.

As a note to artists: This should greatly improve importing in general but there are a great many MTLs that are fundamentally broken. Detecting these issues automatically is rather difficult and requires human intervention by editing the ASCII file. If you experience any of the following issues it is likely the fault of the file and not the importer:

  • Normal map is not being loaded:
    • Solution: Replace "bump", "map_Bump", "disp", "map_Disp" with "norm" or import with "Bump/Displacement is Normal Map" enabled
    • Explanation: The specification does not detail how to load normal maps, only greyscale bump maps. As a result many obj files misuse the bump/displacement attribute to specify normal maps... This is really bad, bump/disp can no longer be trusted. Many files/packages however have opted to extend MTL with a proper normal map property. This updated importer/exporter makes the decision to switch to using this specifier to attempt to undo some of the damage to the ecosystem.
  • Alpha mapped textures are not being loaded correctly (color):
    • Solution: Add "-imfchan m" after every "map_d" attribute which is not a greyscale texture or import with "Alpha from Diffuse Texture" enabled
    • Explanation: The obj specification provides the "-imfchan" texture option to sample scalar information from a specific image channel. The initial version of the importer/exporter update currently only supports "m" which stands for the term used in compositing "matte"
  • Everything is invisible:
    • Solution: For every "d" option flip (newvalue = 1.0 - value) every number (or just set it to 1.0) or import with "Invert Alpha" enabled
    • Explanation: The specification defines "d: 1.0" to be fully opaque and "d: 0.0" to be fully transparent... Some people don't read the manual.
  • Legacy materials don't have glass:
  • Materials that are supposed to be shiny are diffuse and vice versa:
    • Solution: For every "Ns" flip the value (newvalue = 1000.0 - value) or import with "Invert Specular Exponent" enabled
    • Explanation: The specification defines "Ns" to be the specular exponent, 0 is diffuse, 1000 is shiny (blender stops at 900). Some hand-written files might again... have this reversed. The formula we used flips the specular exponent. (The manual formula described doesn't do a propper exponent conversion so this is not technically correct but neither is the exporter that wrote the file.)
  • Glass appears as black:
    • Solution: Edit all glass materials to have "Kd" set to the value of "Tf" and set "Tf" to "1.0 1.0 1.0" or import with "Transmittance Compatibility" enabled
    • Explanation: Some other software doesn't apply diffuse color as the color of glass and instead uses transmittance as the glass color.
  • Every material has subsurface scattering:
    • Solution: Delete all non-glass entries of "Tf" or import with "Transmittance Compatibility" enabled
    • Explanation: "Tf" defines a surfaces ability to let light through, often this is used for glass absorption, other times software will define this as translucency. So in the case of a solid surface the importer deems this to be subsurface translucency, this is a settlement between software that uses Tf as absorption and Tf as translucency. Some software (unwisely) writes transmittance of "1.0 1.0 1.0" for solid materials.
  • Glass materials are transparent:
    • Solution: For every glass material find "d" replace it with "Tr", flip (newvalue = 1.0 - value) for every number
    • Explanation: Many applications don't distinguish between "dissolve" (d) and "transmission" (Tr). Dissolve is available in every shading model (as per-spec), transmission is a component of glass.
  • Transparent materials are made of glass (rare):
    • Solution: For every "Tr" replace it with "d" option flip (newvalue = 1.0 - value) every number (or just delete "Tr")
    • Explanation: Same as above just the other way around. This really should never happen but with the state of files available... you never know.
  • Random materials are glass/metallic/missing reflections:
    • Solution: Set "illum" to 2 or import with "Basic Illumination" enabled
    • Explanation: The importer sees any opaque illumination model without fresnel to likely be metallic, anything with refraction is transmission, specular is disabled in illumination models below 2. Some files have very incorrect illumination models.
  • Color tint is missing from textures on import:
    • Solution: Requires manual setup: unplug the texture, copy the "default value" of the socket (should have been set to the multiplier), create a multiply node with the value and texture as needed.
    • Explanation: Currently texture modulation is not yet supported. This requires extra complicated setups behind the scenes to get working correctly for importing and exporting outside of the Principled BSDF wrangler. This should be a more rare use case for older files where texture memory was scarce and had to be re-used with tinting. If your file needs this however, sorry! You will have to set it up manually.

Another note to artists: Exporter limitations (Questions/Answers):

  • Question: Why isn't my custom shader exporting?
    • Answer: The exporter only exports the Principled BSDF
  • Question: Why isn't my node graph texture exporting?
    • Answer: The exporter only exports image textures which are directly plugged into an acceptable output socket. No procedurals/blending is handled. Consider authoring your textures in a workflow which is baked to an image.
  • Question: Why isn't my subsurface+glass+alpha material exporting with every property?
    • Answer: Not every property can be exported together, the importer/exporter uses context switching to encode properties in the most compatible way possible, as a result you should try to author your materials with only one of these properties. For best results subsurface should also always be "1.0" and base color texture in "subsurface color" for subsurface materials.
  • Question: I have the texture plugged directly into the socket, why isn't it working?
    • Answer: Likely exporting this as a texture is not supported the following section should address that.

Supported Properties (Scalars):

  • Base Color
  • Subsurface Radius * Subsurface Intensity
  • Metallic
  • Specular
  • Roughness
  • Anisotropic
  • Anisotropic Rotation
  • Sheen
  • Clearcoat
  • Clearcoat Roughness
  • IOR
  • Transmission
  • Emission * Emission Strength
  • Alpha
  • Normal Map Intensity

Supported Properties (Textures):

  • Base Color
  • Metallic
  • Specular
  • Roughness
  • Sheen
  • Transmission
  • Emission
  • Alpha
  • Normal Map

Testing:
Multiple files were tested, legacy files, hand authored files, files written from the updated importer, etc.
In addition another tool was written to test the resulting files against the PBR implementation in TinyObj (a popular obj library)
The following code can be found here: https://github.com/astrand130/ObjTest

TinyObj test results: https://developer.blender.org/P1644
Test file:

Future work on MTL (left to time constraints):

  • RGB channel masking for texture packing
  • Texture value modulation/tinting (optional)
  • Texture clamping/filtering
  • Glass volumetric absorption
  • Subdivision and Displacement mapping
  • Bump mapping (needs way to deal with broken ecosystem)
  • Decal texturing
  • "Halo" (facing) transparency
  • DirectXMesh RMA packing
  • Legacy gloss map loading

Possible future considerations (yes, 1995 MTL supports these):

  • Spectral support
  • Compiled procedural textures/graphs (find suitable modern file format to replace .cx*)

Images:


Flawless Sponza Import


Subsurface Test


Material Property Test Map


San Miguel Model (Using new compatibility options)

Diff Detail

Event Timeline

Alex Strand (astrand130) requested review of this revision.Fri, Sep 11, 12:21 PM
Alex Strand (astrand130) created this revision.
Alex Strand (astrand130) edited the summary of this revision. (Show Details)
Alex Strand (astrand130) retitled this revision from Import/Export: Wavefront Obj: Mtl Refactor to Import/Export: Wavefront Obj Mtl Overhaul (compatibility + PBR extension Support).
Alex Strand (astrand130) retitled this revision from Import/Export: Wavefront Obj Mtl Overhaul (compatibility + PBR extension Support) to Import/Export: Wavefront Obj Mtl Overhaul (Improved Compatibility and PBR Extension Support).Fri, Sep 11, 7:45 PM
Alex Strand (astrand130) planned changes to this revision.EditedMon, Sep 14, 5:50 AM

Need to add additional compatibility options for import... Just found another place to download mtl files... they are all broken in the worst ways imaginable.

Added code to solve common compatibility issues with broken files and older software

Alex Strand (astrand130) edited the summary of this revision. (Show Details)
Alex Strand (astrand130) planned changes to this revision.EditedFri, Sep 18, 10:28 AM

Need to add proper Emission Strength extraction via RGBM encoding in the importer.
(See: https://graphicrants.blogspot.com/2009/04/rgbm-color-encoding.html)

Moved to RGBM emission strength import

Alex Strand (astrand130) planned changes to this revision.EditedFri, Sep 18, 1:14 PM

Should look into adding "-imfchan l" to every scalar texture socket to ensure compatibility. Blender by default uses a luminance conversion but not all software does. TinyObj defaults to looking in the matte, so that compatibility issue would need to be fixed. Also need to treat values being affected by textures as multipliers as per spec.

Compatibility improvements: Enforced luminance texture sampling on exported scalar textures and all textured values properties are treated as multipliers to better comply with the specification.

Upon digging around I discovered that we likely need to write base color to ambient because apparently Maya and Max is broken (Lightwave does this). Usually I ignore non-conformance as a bug in someone else's software but this is a big one to overlook. I will try to verify if this is still the case in modern versions and if so I might need another compatibility option. Shouldn't be a blocking issue though for the initial version.

Alex Strand (astrand130) planned changes to this revision.EditedMon, Sep 21, 9:51 AM

Upon investigating it seems that I need to accommodate Autodesk's weird ambient/diffuse bug. More software than theirs has intentionally replicated this bug... See: T78126

Furthermore I may have made a wrong assumption about the extension a little regarding translucency defined as transmittance. To remedy this I might introduce a new parameter explicitly for subsurface profile to further ensure compatibility.

The plan is likely to just always write the diffuse to the ambient... Loading ambient will still be ignored though unless a compatibility option is enabled.

I am planning on making a lite version of this update just to add emission if necessary, as it's gotten a bit of a snowball effect going so it might not make it to 2.91 if this is considered too big for Bcon2.

It appears modern versions of Maya don't have this issue with ambient/diffuse being swapped anymore. I am going to add compatibility code to handle this on import for older files which may be affected. I would encourage users to report bugs to software that still has this issue: it's a them problem, not a blender problem.

Campbell Barton (campbellbarton) requested changes to this revision.Thu, Sep 24, 3:30 AM

This is mostly a basic review of Python code.

I'm assuming the options added are needed and MTL spec is correctly followed.

io_scene_obj/export_obj.py
86

lerp is a method.

Vector.lerp(Vector(mat_wrap.base_color[:3]), Vector(mat_wrap.subsurface_color[:3]), mat_wrap.subsurface)

Can be written as:

Vector(mat_wrap.base_color[:3]).lerp(mat_wrap.subsurface_color[:3], mat_wrap.subsurface)

103–106

This looks like it will modify the color of the node (as the value returned us from the node).

It should be copied first.

io_scene_obj/import_obj.py
40

Only Vector seems to be used.

613

Should be expanded or reference a longer comment elsewhere, explain why it's needed, which packages require this for compatibility ... etc.

1091–1097

Since there are already many options to these functions, suggest a common prefix for material options.

use_mtl_bump_as_normal, use_mtl_invert_dissolve... etc.