Included USD libraries cause symbol conflict with addons using USD binding #76490

New Issue

Brian Savery (AMD) · 2020-05-07T04:03:59+02:00

2020-05-07 04:03:59 +02:00

System Information
Operating system: macOS
Graphics card: N/A

Blender Version
Broken: 2.83 beta
Worked: Never

Included USD libraries cause symbol conflict with addons using USD python bindings. That is, if an addon wants to use USD in it (as a python c-extension most likely), there is an error:

"FATAL ERROR: [TF_DEBUG_ENVIRONMENT_SYMBOL] multiple symbol definitions.  This is usually due to software misconfiguration.  Contact the build team for assistance. (duplicate 'TF_SCRIPT_MODULE_LOADER')
in _Add at line 96 of /Users/brecht/dev/build_darwin/deps/build/usd/src/external_usd/pxr/base/lib/tf/debug.cpp"

Exact steps for others to reproduce the error

Download and extract a build of USD. These ones have python3 bindings (I built my own). https://developer.nvidia.com/usd#bin
Open python editor in blender.
Run commands:

import sys
sys.path.append("PATH_TO_USD/lib/python")   # this is to get the pythonpath right
from pxr import Usd

So I can see two possible solutions to this.

include python bindings with Blender
somehow fix the symbol names in the blender side or in the external USD build so they don't clash. I tried to do this with specifying a different namespace name than "pxr" in the USD build but this didn't seem to help.
3, link python bindings of USD against blender.exe (Not sure how/if this would work)

Willing to work on the solution, but wanted input from Blender developers.

**System Information** Operating system: macOS Graphics card: N/A **Blender Version** Broken: 2.83 beta Worked: Never Included USD libraries cause symbol conflict with addons using USD python bindings. That is, if an addon wants to use USD in it (as a python c-extension most likely), there is an error: ``` "FATAL ERROR: [TF_DEBUG_ENVIRONMENT_SYMBOL] multiple symbol definitions. This is usually due to software misconfiguration. Contact the build team for assistance. (duplicate 'TF_SCRIPT_MODULE_LOADER') in _Add at line 96 of /Users/brecht/dev/build_darwin/deps/build/usd/src/external_usd/pxr/base/lib/tf/debug.cpp" ``` **Exact steps for others to reproduce the error** 1. Download and extract a build of USD. These ones have python3 bindings (I built my own). https://developer.nvidia.com/usd#bin 2. Open python editor in blender. 3. Run commands: ``` import sys sys.path.append("PATH_TO_USD/lib/python") # this is to get the pythonpath right from pxr import Usd ``` So I can see two possible solutions to this. 1. include python bindings with Blender 2. somehow fix the symbol names in the blender side or in the external USD build so they don't clash. I tried to do this with specifying a different namespace name than "pxr" in the USD build but this didn't seem to help. 3, link python bindings of USD against blender.exe (Not sure how/if this would work) Willing to work on the solution, but wanted input from Blender developers.

Brian Savery (AMD) commented

2020-05-07 04:03:59 +02:00

Added subscriber: @BrianSavery

Sybren A. Stüvel commented

2020-05-07 10:02:51 +02:00

Added subscriber: @dr.sybren

Sybren A. Stüvel commented

2020-05-07 10:02:51 +02:00

Changed status from 'Needs Triage' to: 'Archived'

Sybren A. Stüvel closed this issue

2020-05-07 10:02:51 +02:00

Sybren A. Stüvel self-assigned this 2020-05-07 10:02:51 +02:00

Sybren A. Stüvel commented

2020-05-07 10:02:51 +02:00

Hi Brian!

Broken: 2.83 beta

For future reference, unless a version is an official release, please include the Git hash in reports. The easiest to get this from Blender itself is to click Help → Report a Bug.

include python bindings with Blender

For this the USD library needs to support Python 3.7 first. It's still only compatible with Python 2.7. I don't think Blender should switch to a third party version of the library either.

somehow fix the symbol names in the blender side or in the external USD build so they don't clash. I tried to do this with specifying a different namespace name than "pxr" in the USD build but this didn't seem to help.

This is done in Blender as well (see -DPXR_SET_INTERNAL_NAMESPACE=usdBlender in usd.cmake), so the USD library should not conflict. If this namespace is not used in certain parts of the USD library, that would defeat the purpose of that option. This would be a bug in USD, though, and not in Blender.

Hi Brian! > Broken: 2.83 beta For future reference, unless a version is an official release, please include the Git hash in reports. The easiest to get this from Blender itself is to click Help → Report a Bug. > include python bindings with Blender For this the USD library needs to support Python 3.7 first. It's still [only compatible with Python 2.7](https://github.com/PixarAnimationStudios/USD/blob/master/VERSIONS.md). I don't think Blender should switch to a third party version of the library either. > somehow fix the symbol names in the blender side or in the external USD build so they don't clash. I tried to do this with specifying a different namespace name than "pxr" in the USD build but this didn't seem to help. This is done in Blender as well (see `-DPXR_SET_INTERNAL_NAMESPACE=usdBlender` in [usd.cmake](https://developer.blender.org/diffusion/B/browse/master/build_files/build_environment/cmake/usd.cmake)), so the USD library should not conflict. If this namespace is not used in certain parts of the USD library, that would defeat the purpose of that option. This would be a bug in USD, though, and not in Blender.

Brian Savery (AMD) commented

2020-05-07 14:58:45 +02:00

Changed status from 'Archived' to: 'Resolved'

Brian Savery (AMD) commented

2020-05-07 14:58:45 +02:00

@dr.sybren

You're mistaken about the python3 compatibility:
https://github.com/PixarAnimationStudios/USD/blob/master/CHANGELOG.md#build
USD 20.05 supports Python 3:
"Added support for Python 3. This may be enabled by specifying PXR_USE_PYTHON_3=TRUE when running cmake or by running build_usd.py with Python 3."

The nvidia ones have 3.6 support but you can build master with 3.7: https://developer.nvidia.com/usd-20.02-binary-windows-vs2017-python-3.6

Regardless, if I was using a USD builds with python 2 bindings I would not have gotten to this point.

@dr.sybren You're mistaken about the python3 compatibility: https://github.com/PixarAnimationStudios/USD/blob/master/CHANGELOG.md#build USD 20.05 supports Python 3: "Added support for Python 3. This may be enabled by specifying PXR_USE_PYTHON_3=TRUE when running cmake or by running build_usd.py with Python 3." The nvidia ones have 3.6 support but you can build master with 3.7: https://developer.nvidia.com/usd-20.02-binary-windows-vs2017-python-3.6 Regardless, if I was using a USD builds with python 2 bindings I would not have gotten to this point.

Brian Savery (AMD) commented

2020-05-07 15:02:44 +02:00

Furthermore. After filing this I am able to work around this by building blender from source without USD support (which obviously removes the symbol conflict), but I would claim that's not a valid workaround for most users

Brian Savery (AMD) commented

2020-05-07 15:04:03 +02:00

Changed status from 'Resolved' to: 'Archived'

Brian Savery (AMD) commented

2020-05-07 15:04:03 +02:00

Not sure about status how that got changed. Sybren marked it in valid, which I think is erroneous.

Brian Savery (AMD) commented

2020-05-08 04:13:19 +02:00

Might this be related to https://developer.blender.org/T76442 ?

Sybren A. Stüvel commented

2020-05-08 10:39:09 +02:00

Changed status from 'Archived' to: 'Needs User Info'

Sybren A. Stüvel reopened this issue

2020-05-08 10:39:09 +02:00

Sybren A. Stüvel commented

2020-05-08 10:39:09 +02:00

In #76490#926093, @BrianSavery wrote:
You're mistaken about the python3 compatibility:
https://github.com/PixarAnimationStudios/USD/blob/master/CHANGELOG.md#build
USD 20.05 supports Python 3:

I'm very happy to be wrong here :) Still, on Linux they only test with Python 3.6, whereas both the VFX Reference Platform and Blender are on 3.7. That's an infintesemally small difference compared to 2.7 though!

If it is indeed related to #76442, @BrianSavery could you maybe test what happens when you build Blender with the conflicting symbols added to source/creator/blender.map? That should hide those symbols; if that is enough I'd be happy to include that in Blender and get things moving forward.

> In #76490#926093, @BrianSavery wrote: > You're mistaken about the python3 compatibility: > https://github.com/PixarAnimationStudios/USD/blob/master/CHANGELOG.md#build > USD 20.05 supports Python 3: I'm very happy to be wrong here :) Still, on Linux they only test with Python 3.6, whereas both the VFX Reference Platform and Blender are on 3.7. That's an infintesemally small difference compared to 2.7 though! If it is indeed related to #76442, @BrianSavery could you maybe test what happens when you build Blender with the conflicting symbols added to `source/creator/blender.map`? That should hide those symbols; if that is enough I'd be happy to include that in Blender and get things moving forward.

Brian Savery (AMD) commented

2020-05-08 15:34:32 +02:00

Yep will test. Here's what I've done so far:

macOS with standard blender build - Error above
macOS build with USD disabled - Ok
ubuntu build with 3.7 build of USD - Ok.

Yep will test. Here's what I've done so far: 1. macOS with standard blender build - Error above 2. macOS build with USD disabled - Ok 3. ubuntu build with 3.7 build of USD - Ok.

JT Nelson commented

2020-05-15 06:45:14 +02:00

Added subscriber: @jta

Ankit Meel commented

2020-05-22 16:55:39 +02:00

Added subscriber: @ankitm

Ankit Meel commented

2020-05-22 16:55:39 +02:00

Changed status from 'Needs User Info' to: 'Confirmed'

Ankit Meel commented

2020-05-22 16:55:39 +02:00

(since it's a todo)

lvxejay commented

2020-06-26 18:00:09 +02:00

Added subscriber: @lvxejay

Mel Massadian commented

2020-12-10 00:11:28 +01:00

Added subscriber: @mel

Yegor commented

2020-12-10 00:34:35 +01:00

Added subscriber: @YegorSmirnov

Mel Massadian commented

2021-08-09 18:36:18 +02:00

Is there any advancement on this issue?
I think that adding a custom namespace is the solution, like specified here:

https://github.com/PixarAnimationStudios/USD/blob/release/BUILDING.md#usd-developer-options

Is there any advancement on this issue? I think that adding a custom namespace is the solution, like specified here: https://github.com/PixarAnimationStudios/USD/blob/release/BUILDING.md#usd-developer-options

Mel Massadian commented

2021-08-23 05:14:56 +02:00

This is also an issue for projects like https://github.com/GPUOpen-LibrariesAndSDKs/BlenderUSDHydraAddon

But apparently only on macOS?

This is also an issue for projects like https://github.com/GPUOpen-LibrariesAndSDKs/BlenderUSDHydraAddon But apparently only on macOS?

Jason schleifer commented

2021-09-08 03:31:45 +02:00

Added subscriber: @JasonSchleifer

Sybren A. Stüvel commented

2022-01-31 11:28:32 +01:00

Blender already builds USD with a custom namespace: https://developer.blender.org/diffusion/B/browse/master/build_files/build_environment/cmake/usd.cmake$36

I'm certainly open to suggestions on how to solve this issue.

Blender already builds USD with a custom namespace: https://developer.blender.org/diffusion/B/browse/master/build_files/build_environment/cmake/usd.cmake$36 I'm certainly open to suggestions on how to solve this issue.

Karl Hendrikse commented

2022-07-11 06:51:43 +02:00

Added subscriber: @karlhendrikse

Karl Hendrikse commented

2022-07-11 06:51:43 +02:00

Using a custom namespace doesn't fix this because the problem isn't a symbol collision.

The problem is that when any shared library is loaded into the process, the USD library looks in the library's Mach-O headers and pulls out function pointers for all special "constructors" and calls them. The USD library contains a macro one can use to declare such "constructors", and they are stored in ".pxrctor" sections in the binary. Some of the constructors declared in the USD library cause the application to abort when run multiple times, which is what happens when there are multiple instances of the library loaded, both searching for ".pxrctor" sections and calling all "constructors" when libraries are loaded.

One solution I found is to change the name used for the sections in one instance of the USD library so the instances aren't trying to run each other's "constructors". Changing all occurrences of pxrctor into fooctor (and likewise pxrdtor into foodtor) in the USD library fixed the problem for me.

(I had this problem in a separate application, but I found this page when searching for the solution and after figuring it out for myself I thought posting here might be helpful.)

Using a custom namespace doesn't fix this because the problem isn't a symbol collision. The problem is that when any shared library is loaded into the process, the USD library looks in the library's Mach-O headers and pulls out function pointers for all special "constructors" and calls them. The USD library contains a macro one can use to declare such "constructors", and they are stored in "`.pxrctor`" sections in the binary. Some of the constructors declared in the USD library cause the application to abort when run multiple times, which is what happens when there are multiple instances of the library loaded, both searching for "`.pxrctor`" sections and calling all "constructors" when libraries are loaded. One solution I found is to change the name used for the sections in one instance of the USD library so the instances aren't trying to run each other's "constructors". Changing all occurrences of `pxrctor` into `fooctor` (and likewise `pxrdtor` into `foodtor`) in the USD library fixed the problem for me. (I had this problem in a separate application, but I found this page when searching for the solution and after figuring it out for myself I thought posting here might be helpful.)

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Included USD libraries cause symbol conflict with addons using USD binding #76490