Hide Blender symbols by default on all platforms #76442

New Issue

Brecht Van Lommel · 2020-05-05T12:22:15+02:00

Brecht Van Lommel commented

2020-05-05 12:22:15 +02:00

The purpose of this task is to hide Blender symbols by default on all platforms, where previously it was only done on Windows.

Related reports that are solved by hiding symbols, and there have been various others in the past.

#74262 (Cycles render hangs on Linux and AMD Radeon RX 5700 XT)
blender/blender-addons#76414 (Blender crashing when trying to use Cupy)
#68052 (Segfault when accessing Cycles Render Devices preferences with two OpenCL platforms, AMD Mesa Clover and Intel Gen9 NEO)
#99900 (Segfault on Linux when running third party Python library function with multithreading enabled)

One solution would be compiling all external libraries with -fvisibility=hidden, however the simpler solution seems to be to use a symbol map to hide all symbols except a few required ones.

Status:

Windows
Linux (D14971)
macOS

The purpose of this task is to hide Blender symbols by default on all platforms, where previously it was only done on Windows. Related reports that are solved by hiding symbols, and there have been various others in the past. * #74262 (Cycles render hangs on Linux and AMD Radeon RX 5700 XT) * blender/blender-addons#76414 (Blender crashing when trying to use Cupy) * #68052 (Segfault when accessing Cycles Render Devices preferences with two OpenCL platforms, AMD Mesa Clover and Intel Gen9 NEO) * #99900 (Segfault on Linux when running third party Python library function with multithreading enabled) One solution would be compiling all external libraries with `-fvisibility=hidden`, however the simpler solution seems to be to use a symbol map to hide all symbols except a few required ones. Status: - [x] Windows - [x] Linux ([D14971](https://archive.blender.org/developer/D14971)) - [ ] macOS

Brecht Van Lommel commented

2020-05-05 12:22:15 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Brecht Van Lommel commented

2020-05-05 12:22:15 +02:00

Added subscribers: @brecht, @Sergey

Brecht Van Lommel commented

2020-05-06 17:48:20 +02:00

What might also help here is to use RLTD_DEEPBIND when loading libraries, so that those libraries use their own symbols over global symbols from Blender.

This could be done in clew and Python (with sys.setdlopenflags).

What might also help here is to use `RLTD_DEEPBIND` when loading libraries, so that those libraries use their own symbols over global symbols from Blender. This could be done in clew and Python (with `sys.setdlopenflags`).

Sergey Sharybin commented

2020-05-07 09:50:14 +02:00

Will this break ctypes usage in Python scripts?

RLTD_DEEPBIND seems fine to just go ahead and do it. Just make sure this is done in upstream, covering the generator templates. Think you have commit access to all wranglers we use.

What is also not clear to me is whether anything special is to be done with Python modules (_ssl.so for example). Are they not pulling symbols to global scope already? Is it controlled with sys.setdlopenflags? Or do we need to compile all dependencies with hidden visibility?

Will this break `ctypes` usage in Python scripts? `RLTD_DEEPBIND` seems fine to just go ahead and do it. Just make sure this is done in upstream, covering the generator templates. Think you have commit access to all wranglers we use. What is also not clear to me is whether anything special is to be done with Python modules (`_ssl.so` for example). Are they not pulling symbols to global scope already? Is it controlled with `sys.setdlopenflags`? Or do we need to compile all dependencies with hidden visibility?

Brecht Van Lommel commented

2020-05-07 21:36:15 +02:00

If ctypes is being used to access Blender symbols, that will indeed break. However that already would not have worked on Windows, so I guess none of the commonly used add-ons/plugins rely on it?

RTLD_DEEPBIND didn't seem to help in #68052. But probably it's the right thing to do regardless.

From my understanding RTLD_DEEPBIND is about preferring local symbols over globals ones when binding symbols. So for example if a Luxrender add-on Python module uses OpenImageDenoise, it would use its own version rather than Blender's.

sys.setdlopenflags sets the default flags used when loading Python modules. I don't think this matters much for bundled Python modules like ssl, it's for add-ons that ship binary Python modules.

If `ctypes` is being used to access Blender symbols, that will indeed break. However that already would not have worked on Windows, so I guess none of the commonly used add-ons/plugins rely on it? `RTLD_DEEPBIND` didn't seem to help in #68052. But probably it's the right thing to do regardless. From my understanding `RTLD_DEEPBIND` is about preferring local symbols over globals ones when binding symbols. So for example if a Luxrender add-on Python module uses OpenImageDenoise, it would use its own version rather than Blender's. `sys.setdlopenflags` sets the default flags used when loading Python modules. I don't think this matters much for bundled Python modules like ssl, it's for add-ons that ship binary Python modules.

Sergey Sharybin commented

2020-05-08 10:24:19 +02:00

Did you use RTLD_DEEPBIND , RTLD_NOW | RTLD_DEEPBIND or RTLD_LAZY | RTLD_DEEPBIND ?

The reason our tweaks might not have helped is because the driver might be doing library load, and then you have collision of symbols in the whatever ICD is loading.

In Python I remember having all symbols which are not Py*, _Py*, _py* be hidden. The reason of this was conflict with some of the addon which was loading another .so. Apparently, this mapping file got lost when we've switched to make deps for the release environment. So we either need to restore this mapping file, or use hidden visibility for Python as well. Do you have strong opinion on this topic?

Did you use `RTLD_DEEPBIND` , `RTLD_NOW | RTLD_DEEPBIND` or `RTLD_LAZY | RTLD_DEEPBIND` ? The reason our tweaks might not have helped is because the driver might be doing library load, and then you have collision of symbols in the whatever ICD is loading. In Python I remember having all symbols which are not `Py*`, `_Py*`, `_py*` be hidden. The reason of this was conflict with some of the addon which was loading another .so. Apparently, this mapping file got lost when we've switched to `make deps` for the release environment. So we either need to restore this mapping file, or use hidden visibility for Python as well. Do you have strong opinion on this topic?

blender-admin commented

2022-05-18 11:24:08 +02:00

This issue was referenced by f937c186de

This issue was referenced by f937c186dedeec4ead1ead9c16646c4aaebafd1b

Xavier Hallade commented

2022-07-06 17:09:00 +02:00

Added subscriber: @xavierh

blender-admin commented

2022-07-29 17:31:52 +02:00

This issue was referenced by cfd16c04f8

This issue was referenced by cfd16c04f8316270b67c73775a5885d6bc4dc4e2

Brecht Van Lommel changed title from ~~Build with -fvisibility=hidden on Linux and macOS~~ to Hide Blender symbols by default on all platforms

2022-07-29 18:04:36 +02:00

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Hide Blender symbols by default on all platforms #76442