Page MenuHome

USD: Introducing USD support
ClosedPublic

Authored by Sybren A. Stüvel (sybren) on Thu, Nov 21, 5:58 PM.
Tokens
"Love" token, awarded by m__x."Like" token, awarded by threedslider."Pirate Logo" token, awarded by wahooney."Pterodactyl" token, awarded by Grische."Love" token, awarded by brilliant_ape."Love" token, awarded by Brandon777."Love" token, awarded by ZohaibAli."Love" token, awarded by harsan81."Love" token, awarded by amonpaike."Love" token, awarded by martinium."Love" token, awarded by SomeAB."Love" token, awarded by RC12."Love" token, awarded by Kronk."Burninate" token, awarded by franMarz."Love" token, awarded by Alrob."Love" token, awarded by Yegor."Love" token, awarded by amyspark."Party Time" token, awarded by Zino."Love" token, awarded by dgsantana."Love" token, awarded by swerner."Yellow Medal" token, awarded by Way."Love" token, awarded by xrg."Love" token, awarded by Kramon."Love" token, awarded by koloved.

Details

Summary

This commit introduces an exporter to Pixar's Universal Scene Description (USD) format.

  • The USD libraries are built by make deps, not yet by install_deps.sh.
  • Only experimental support for instancing; by default all duplicated objects are made real in the USD file. This is fine for exporting a linked-in posed character, not so much for thousands of pebbles etc.
  • This patch contains LazyDodo's fixes for building on Windows in D5359.
  • This patch assumes D6246: Blendfile-loading test class is already applied.
  • To run the unittest, place the following file in ../lib/tests/usd/usd_hierarchy_export_test.blend

For understanding how the depsgraph is iterated to produce the exported data, I would recommend starting at source/blender/usd/intern/abstract_hierarchy_iterator.h and source/blender/usd/intern/abstract_hierarchy_iterator.cc, and then looking at the USD-specific subclasses.

For ease of reviewing, the contents of this patch can be found in the temp-sybren-usd-patch-02 branch. This branch also includes the required changes in D6246.

Building USD manually is quite simple:

  1. Download and extract the sources
  2. Make sure /opt/usd exists and is writable for your user.
  3. python2 build_scripts/build_usd.py --no-python /opt/usd

Meshes

USD seems to support neither per-material nor per-face-group
double-sidedness, so we just use the flag from the first non-empty
material slot. If there is no material we default to double-sidedness.

Each UV map is stored on the mesh in a separate primvar. Materials can
refer to these UV maps, but this is not yet exported by Blender. The
primvar name is the same as the UV Map name. This is to allow the
standard name "st" for texture coordinates by naming the UV Map as such,
without having to guess which UV Map is the "standard" one.

Face-varying mesh normals are written to USD. When the mesh has custom
loop normals those are written. Otherwise the poly flag ME_SMOOTH is
inspected to determine the normals.

The UV maps and mesh normals take up a significant amount of space, so
exporting them is optional. They're still enabled by default, though.
For comparison: a shot of Spring (03_035_A) is 1.2 GiB when exported
with UVs and normals, and 262 MiB without. We probably have room for
optimisation of written UVs and normals.

The mesh subdivision scheme isn't using the default value 'Catmull
Clark', but uses 'None', indicating we're exporting a polygonal mesh.
This is necessary for USD to understand our normals; otherwise the mesh
is always rendered smooth. In the future we may want to expose this
choice of subdivision scheme to the user, or auto-detect it when we
actually support exporting pre-subdivision meshes.

A possible optimisation could be to inspect whether all polygons are
smooth or flat, and mark the USD mesh as such. This can be added when
needed.

Animation

Mesh and transform animation are now written when passing
animation=True to the export operator. There is no inspection of
whether an object is actually animated or not; USD can handle
deduplication of static values for us.

The administration of which timecode to use for the export is left to
the file-format-specific concrete subclasses of
AbstractHierarchyIterator; the abstract iterator itself doesn't know
anything about the passage of time. This will allow subclasses for the
frame-based USD format and time-based Alembic format.

support for simple preview materials

Very simple versions of the materials are now exported, using only the
viewport diffuse RGB, metallic, and roughness.

When there are multiple materials, the mesh faces are stored as geometry
subset and each material is assigned to the appropriate subset. If there
is only one material this is skipped.

The first material if any) is always applied to the mesh itself
(regardless of the existence of geometry subsets), because the Hydra
viewport doesn't support materials on subsets. See
https://github.com/PixarAnimationStudios/USD/issues/542 for more info.

Note that the geometry subsets are not yet time-sampled, so it may break
when an animated mesh changes topology.

Hair

Only the parent strands are exported, and only with a constant colour.
No UV coordinates, no information about the normals.

Camera

Only perspective cameras are supported for now.

Particles

Particles are only written when they are alive, which means that they
are always visible (there is currently no code that deals with marking
them as invisible outside their lifespan).

Particle-system-instanced objects are exported by suffixing the object
name with the particle's persistent ID, giving each particle XForm a
unique name.

Instancing/referencing

This exporter has experimental support for instancing/referencing.

Dupli-object meshes are now written to USD as references to the original
mesh. This is still very limited in correctness, as there are issues
referencing to materials from a referenced mesh.

I am still committing this, as it gives us a place to start when
continuing the quest for proper instancing in USD.

Lights

USD does not directly support spot lights, so those aren't exported yet.
It's possible to add this in the future via the UsdLuxShapingAPI. The
units used for the light intensity are also still a bit of a mystery.

Fluid vertex velocities

Currently only fluid simulations (not meshes in general) have explicit
vertex velocities. This is the most important case for exporting
velocities, though, as the baked mesh changes topology all the time, and
thus computing the velocities at import time in a post-processing step
is hard.

JSON files

USD needs some JSON files at runtime. These are copied from ${LIBDIR}/usd/lib/usd by make install to a usd directory next to the blender executable. I extended our USD patch to make it find those files (USD itself only allows build-time configuration of the paths, or requires an environment variable to be set before Blender even starts).

Diff Detail

Repository
rB Blender

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

updated windows libs are available in svn ( rBL62323 )

@Sybren A. Stüvel (sybren) is it possible to have USB libraries committed so it's possible to easily compile and test the patch other than reading the code? :)

Yes, they are in lib/linux_centos7_x86_64/usd.

  • Cleanup: USD: remove unused header files
  • USD: Check location of USD JSON files in unit tests. I've been bitten enough times by USD not finding its JSON files that I added a unit test to help. Furthermore, I've added a check for the PXR_PATH_DEBUG environment variable, which when set causes the USD library to log which paths are searched for those files.
  • USD: fixed paths in usd.diff
  • USD: Fixed include path for unit tests, now TBB headers can be found
  • USD: Allow running unit tests without 'make install' step
superfunc added inline comments.Fri, Nov 29, 4:04 PM
build_files/build_environment/patches/usd.diff
64

This should probably go through the TfDebug system so it can be controlled like the rest of env-var debugging in USD. For a quick background, here is an example usage of TfDebug (https://github.com/PixarAnimationStudios/USD/blob/master/pxr/usd/lib/usd/clip.cpp#L187). TfDebug allows users to set a space delimited, wildcard accepting env var (TF_DEBUG), for example, if I wanted information about asset resolution behavior in the Ar library, as well as information on layer operations from Sdf (Scene Description Foundation sublib), I could set TF_DEBUG="AR_* SDF_LAYER" and rerun my program to get that information. Each library defines its debug codes in a debugCodes.h/cpp file (https://github.com/PixarAnimationStudios/USD/blob/master/pxr/base/lib/plug/debugCodes.h); this one could either use one of the existing from libPlug such as PLUG_INFO_SEARCH or add another.

superfunc added inline comments.Fri, Nov 29, 4:22 PM
build_files/build_environment/cmake/usd.cmake
40

If possible, we should consider setting the internal USD namespace here (something like blender_v_xxx). This will prevent symbol collision issues if users have other plugins which require a different version of USD to be running inside the same process space. Your code can keep using the stable, external pxr:: namespace prefix when referring to USD code.

Documentation link: https://github.com/PixarAnimationStudios/USD/blob/master/BUILDING.md#usd-developer-options

release/scripts/startup/bl_ui/space_topbar.py
444

This should also have .usd, which can be either binary or ascii (defaults to binary). The .usd extension is the one commonly used in VFX pipelines, as it allows the underlying representation of a file to be changed without breaking all composition arcs which refer to it. There is a function which can tell you all supported file format extensions .usd knows about (https://github.com/PixarAnimationStudios/USD/commit/85060875cac89c925f49f3ee27db0868c5ed3326) but that would also return .usdz (and .abc or .obj if you had the respective plugins built), so it would require some additional filtering as .usdz export can't be handled like the other three extensions.

Sybren A. Stüvel (sybren) marked an inline comment as done.Fri, Nov 29, 4:57 PM
Sybren A. Stüvel (sybren) added inline comments.
build_files/build_environment/cmake/usd.cmake
40

What would the v_xxx indicate?

build_files/build_environment/patches/usd.diff
64

I had to avoid writing to std::out, because that would segfault, so I made it as simple as possible to work for my purpose. Even though I want to do the Right Thing, there are more pressing matters to work on for me. Of course a patch would be very welcome ;-)

superfunc added inline comments.Fri, Nov 29, 5:00 PM
source/blender/usd/intern/usd_capi.cc
93

Probably worth noting; there are a couple reasons stage creation could fail. TfErrorMark and UsdUtilsCoalescingDiagnosticDelegate can be used to capture errors and warnings respectively to relay to users.

tests/gtests/usd/usd_stage_creation_test.cc
40

Same note as above wrt stage creation.

superfunc added inline comments.Fri, Nov 29, 5:09 PM
build_files/build_environment/cmake/usd.cmake
40

Ah sorry that wasn't clear, by v_xxx I was trying to indicate some versioning scheme such as blender_v_282 for blender 2.8.2.

build_files/build_environment/patches/usd.diff
64

Sure thing, I'd be happy to add this once your patch goes through

Sybren A. Stüvel (sybren) marked an inline comment as done.
  • USD: Support '.usd' as export extension (thanks @superfunc) and proper filtering in the file browser
  • USD: Fixed export of animated meshes + using sparse value writer
Sergey Sharybin (sergey) requested changes to this revision.Tue, Dec 10, 2:19 PM

USD needs some JSON files at runtime. These are copied from ${LIBDIR}/usd/lib/usd by make install to a usd directory next to the blender executable.

Canonical place for this within Blender's package structure would be ${TARGETDIR_VER}/datafiles/usd.

Having those files next to executable is not possible within macOS bundle.

I extended our USD patch to make it find those files (USD itself only allows build-time configuration of the paths, or requires an environment variable to be set before Blender even starts).

More correct is to provide an explicit path where to find those JSON files.

There doesn't seem to be a way to import USD back to Blender, so how can one review the generated file?

build_files/build_environment/cmake/harvest.cmake
203

I can not really follow this comment, needs more context about what your goal is and what is the limitation.

204

Put relevant information in the comment itself. Requiring to click, switch to a browser, read the entire discussion to see what is it exactly relevant in this case is just extra and unnecessary steps.

Not even mentioning that one must be a member of USD group to read the discussion you've linked.
Going beyond this case, i don't think we should be promoting any of such private groups.

205–207

Not sure why everything is harvest and this one is install.
Is it because harvest is sourcing to a different location? Should such install be in usd.cmake ?

build_files/build_environment/cmake/usd.cmake
40

Not sure how this is applicable to us. We don't have plugin system and recompiling libraries for every Blender release isn't really great.

We can just mark all USD related symbols as local (blender.map, osx_locals.map) to ease burden of keeping track of those namespaces.

68–81

Why not to have libusd_m.a handled here as well?

build_files/build_environment/patches/usd.diff
64

had to avoid writing to std::out, because that would segfault

Why would std::out segfault?

87–88

Such things are not something you usually do in CMake.

build_files/cmake/Modules/FindUSD.cmake
56

No space after if.
Is also not how you check for not-found symbol in CMake.

57

This must not happen here. The job if Find* module is to set FOO_FOUND to False in such cases and NOT to die with a fatal error.

source/blender/depsgraph/DEG_depsgraph_query.h
29–30 ↗(On Diff #20102)

This can go to the master already.

source/blender/editors/io/io_usd.c
71

Why not to have a typed structure?

77

Store BKE_main_blendfile_path(bmain) in a variable, reuse it in this case an in the case 2 lines below.

92

If function argument is unused in all codepaths:

  • In C use void foo(int UNUSED(bar))
  • In C++ use void foo(int /*bar*/)
130

I wouldn't use Alembic as a source of truth.

See operators like RENDER_OT_render, OBJECT_OT_bake_image, CLIP_OT_solve_camera and many many others.

If this approach is no longer desired, please let me know which other operators are using the desired approach.

I don't know it was ever desired. Just more like got out of hands in Alembic together with other design violations there =\

source/blender/usd/CMakeLists.txt
24–25

Have "they" been notified?

26

Not needed with MSVC? XCode?

source/blender/usd/intern/abstract_hierarchy_iterator.cc
176

Think i've misread the code.

195

LISTBASE_FOREACH

source/blender/usd/intern/abstract_hierarchy_iterator.h
56

Such rather generic name in a global namespace.

I would say USD integration is to be in its own namespace.

203

I still don't know xform is superior name over tfm.

The latter one is a historical term in Blender and in Cycles. The former one is some newcomer.

After deeper look it actually seems that XFORM started to spread into other areas with the Gizmo work.

While i don't really agree this is a great move i guess is better to STFU and let the cancer spread.. Ugh.

source/blender/usd/intern/usd_capi.cc
136

I don't think such guesstimate makes things any more clear, and i don't think we should be adding it.

source/blender/usd/intern/usd_hierarchy_iterator.h
38–40

Private indications.

source/blender/usd/intern/usd_writer_abstract.cc
41

No semicolon.

source/blender/usd/intern/usd_writer_abstract.h
45

Private indicator.

This revision now requires changes to proceed.Tue, Dec 10, 2:19 PM
Sybren A. Stüvel (sybren) marked 33 inline comments as done.Wed, Dec 11, 4:40 PM

Canonical place for this within Blender's package structure would be ${TARGETDIR_VER}/datafiles/usd.

As discussed face-to-face, I've hacked around the USD static constructor and turned it into a function that's called from Blender. This now explicitly gets the absolute path of ${TARGETDIR_VER}/datafiles/usd.

I have also removed the setting of the PXR_PLUGINPATH_NAME environment variable from the unit tests' CMakeLists.txt. As a result, the USD unit tests only succeed when you've installed everything or explicitly made changes (set PXR_PLUGINPATH_NAME, add a symlink, whatever) to make things run without installing.

There doesn't seem to be a way to import USD back to Blender, so how can one review the generated file?

I've used the usdview tool included with USD. This uses PyQt and Python 2.7, though, so we can't include it in the Blender-build of USD. From the USD sources you can run this command to get the tooling:

python2 build_scripts/build_usd.py /opt/usd-with-python --generator Ninja

Alternatively you can load the USD files into Gaffer, but I find that less friendly, and it doesn't provide the same inspection possibilities of USD properties.

build_files/build_environment/cmake/usd.cmake
40

Not sure what the use would be to include the version there, as even just blender or usd_blender would make it unique from the default USD internal namespace.

build_files/build_environment/patches/usd.diff
64

Because it isn't initialised yet when running from a static constructor.

87–88

I know, but it's done with the other PXR_BUILD_xxx options in their CMake file as well, so I'm just being consistent.

source/blender/depsgraph/DEG_depsgraph_query.h
29–30 ↗(On Diff #20102)

Done (rB7683641a7f9e1c44c260b74144dbf0bb215cccad), but I've kept it in this patch so that I don't have to merge/rebranch again. Will be removed when committed to master.

source/blender/editors/io/io_usd.c
130

Because the operator also uses the file browser, it's hard to make a custom modal function that does the appropriate thing.

source/blender/usd/CMakeLists.txt
24–25

yes: https://github.com/PixarAnimationStudios/USD/issues/1057

(and I also updated the comment)

26

Nope, just on Linux. I've updated the comment to reflect this.

source/blender/usd/intern/usd_capi.cc
93

I'll leave that for later, I've added a note about this in T72204: Universal Scene Description

136

What's the alternative? No progress report at all, simply because it can't be perfect?

build_files/build_environment/patches/usd.diff
87–88

If it's something common to how USD itself is doing i'm fine.

source/blender/editors/io/io_usd.c
130

That is a fair point.

I am not sure how to cleanly address this. This is something we need for audio baking as well (although, in there the execution is always blocking).

Since there are some other areas doing similar approach think it's more than acceptable to use it here as well, with possibly re-iterating in the future.

source/blender/usd/CMakeLists.txt
26

Why not to make if:

if(UNIX AND NOT APPLE)
  add_definitions(-D_GLIBCXX_PERMIT_BACKWARD_HASH)
endif()
source/blender/usd/intern/usd_capi.cc
136

Just set progress based on current frame and frame range, without attempts to try be more monotonous.

In a more ideal world i would think job system should be able to report status of steps which don't have clear progress.

Sybren A. Stüvel (sybren) marked 11 inline comments as done.Wed, Dec 11, 5:21 PM

I've used the usdview tool included with USD.

PS: when using usdview, choose View → Camera → Some Camera. That'll show the view from the exported camera, which can save you from trying to navigate the scene with unknown controls.

  • patch USD to allow Blender to construct a path to JSON files
  • remove PXR_PLUGINPATH_NAME when testing
  • install libusd_m.a from usd.cmake instead of from harvesting step
  • Avoid potential conflict with 2nd USD library (this required rebuilding the USD libraries, which I've done for CentOS7 and committed to SVN).
  • added note about upstream pull request for PXR_BUILD_USD_TOOLS
  • no more fatal message in FindUSD.cmake
  • Store BKE_main_blendfile_path(bmain) in a variable
  • properly mark unused variable
  • pass data from invoke() to execute() in a struct
  • clarified a comment
  • Fix exporting of meshes of dupli-objects
  • clarified comment about defining _GLIBCXX_PERMIT_BACKWARD_HASH
  • moved all code into C++ namespace 'USD'
  • renamed 'xform' to 'transform'
  • use LISTBASE_FOREACH
  • private indicators
  • set Linux-specific #define only on Linux
  • Report progress on a frame basis and with less guesswork
  • mark the exporter as Experimental feature

How exactly the 'Experimental' tab in the user preferences is going to behave is subject to change (@Dalai Felinto (dfelinto) is working on that right now), so don't include that in the review for now.

Minor think about find/found discussed here in person. Not really a stopper i guess.

The experimental discussion i don't know. There are many interface things i don't agree with, but i'm not even in the UI module. Will just keep quietly being grumpy ;)

LazyDodo (LazyDodo) requested changes to this revision.Thu, Dec 12, 7:15 PM

I have one build error in usd_test and one failing test also the usd export option does not show in the UI (i assume due to it being unable to find the json files but haven't looked into it)

fix for the build error (yeah.... i know... this again...)

diff --git a/tests/gtests/usd/CMakeLists.txt b/tests/gtests/usd/CMakeLists.txt
index 020122bad1f..a1b7fd77310 100644
--- a/tests/gtests/usd/CMakeLists.txt
+++ b/tests/gtests/usd/CMakeLists.txt
@@ -25,6 +25,9 @@
 if(UNIX AND NOT APPLE)
   add_definitions(-D_GLIBCXX_PERMIT_BACKWARD_HASH)
 endif()
+if(WIN32)
+    add_definitions(-DNOMINMAX)
+endif()

 set(INC
   .

Don't have a fix for the failing test but it seems pretty straight forward

116: [----------] 1 test from USDStageCreationTest
116: [ RUN      ] USDStageCreationTest.JSONFileLoadingTest
116: Runtime Error: in _ReadPlugInfoWithWildcards at line 374 of E:\db16\build\S\VS1564R\build\usd\src\external_usd\pxr\base\lib\plug\info.cpp -- Plugin info file 2.82\datafiles\usd/ is not absolute
116: Coding Error: in _CreateNew at line 456 of E:\db16\build\S\VS1564R\build\usd\src\external_usd\pxr\usd\lib\sdf\layer.cpp -- Failed verification: ' fileFormat '
116: K:\BlenderGit\blender\tests\gtests\usd\usd_stage_creation_test.cc(46): error: Value of: usd_stage
116:   Actual: false
116: Expected: true
116: unable to find suitable USD plugin to write usd-stage-creation-test.usdc
116: [  FAILED  ] USDStageCreationTest.JSONFileLoadingTest (6 ms)
This revision now requires changes to proceed.Thu, Dec 12, 7:15 PM

I have one build error in usd_test and one failing test also the usd export option does not show in the UI (i assume due to it being unable to find the json files but haven't looked into it)

The Export to USD menu option is hidden behind the new Experimental prefs: User Preferences, enable Developer Extras, Experimental tab, enable USD Exporter.

Don't have a fix for the failing test but it seems pretty straight forward

It requires running an install step, that should make it work. I changed the unit test so that it doesn't find the build-time JSON files any more, because that'll make it actually test the proper runtime conditions.

looks like there was an issue with the copy of the json files they ended up in 2.82/datafiles/usd/usd

diff --git a/source/creator/CMakeLists.txt b/source/creator/CMakeLists.txt
index 9e9f7be90f8..2b465a26db1 100644
--- a/source/creator/CMakeLists.txt
+++ b/source/creator/CMakeLists.txt
@@ -1069,7 +1069,7 @@ if (WITH_USD)
   add_definitions(-DWITH_USD)
   install(DIRECTORY
     ${LIBDIR}/usd/lib/usd
-    DESTINATION "${TARGETDIR_VER}/datafiles/usd"
+    DESTINATION "${TARGETDIR_VER}/datafiles/"
   )
 endif()

fixes it for me, but you may want to check that on other platforms.

with the fix above the exporter now works from blender, test still fails though but one failing test with minor path issues should not keep this from missing the bcon2 window imho.

other minor nitpicking is the 'feedback' button in the experimental page goes to page that doesn't exist ( https://devtalk.blender.org/usd-feedback-thread )

This revision is now accepted and ready to land.Fri, Dec 13, 1:01 AM
  • Rebased on top of master
  • Made the unit test that checks for the JSON files use absolute paths, and make it build only on Linux. Other platforms are to be tested as well, of course, but I won't merge a failing unit test to master.
This revision was automatically updated to reflect the committed changes.