Page MenuHome

CMake: Refactor external dependencies handling
ClosedPublic

Authored by Sergey Sharybin (sergey) on Jan 21 2020, 2:22 PM.

Details

Summary

This is a more correct fix to the issue Brecht was fixing in D6600.

While the fix in that patch worked fine for linking it broke ASAN
runtime under some circumstances.
For example, make full debug developer would compile, but trying
to start blender will cause assert failure in ASAN (related on check
that ASAN is not running already).

Top-level idea: leave it to CMake to keep track of dependency graph.

The root of the issue comes to the fact that target like "blender" is
configured to use a lot of static libraries coming from Blender sources
and to use external static libraries. There is nothing which ensures
order between blender's and external libraries. Only order of blender
libraries is guaranteed.

It was possible that due to a cycle or other circumstances some of
blender libraries would have been passed to linker after libraries
it uses, causing linker errors.

For example, this order will likely fail:

libbf_blenfont.a libfreetype6.a libbf_blenfont.a

This change makes it so blender libraries are explicitly provided
their dependencies to an external libraries, which allows CMake to
ensure they are always linked against them.

General rule here: if bf_foo depends on an external library it is
to be provided to LIBS for bf_foo.
For example, if bf_blenkernel depends on opensubdiv then LIBS in
blenkernel's CMakeLists.txt is to include OPENSUBDIB_LIBRARIES.

The change is made based on searching for used include folders
such as OPENSUBDIV_INCLUDE_DIRS and adding corresponding libraries
to LIBS ion that CMakeLists.txt. Transitive dependencies are not
simplified by this approach, but I am not aware of any downside of
this: CMake should be smart enough to simplify them on its side.
And even if not, this shouldn't affect linking time.

Benefit of not relying on transitive dependencies is that build
system is more robust towards future changes. For example, if
bf_intern_opensubiv is no longer depends on OPENSUBDIV_LIBRARIES
and all such code is moved to bf_blenkernel this will not break
linking.

The not-so-trivial part is change to blender_add_lib (and its
version in Cycles). The complexity is caused by libraries being
provided as a single list argument which doesn't allow to use
different release and debug libraries on Windows. The idea is:

  • Have every library prefixed as "optimized" or "debug" if separation is needed (non-prefixed libraries will be considered "generic").
  • Loop through libraries passed to function and do simple parsing which will look for "optimized" and "debug" words and specify following library to corresponding category.

This isn't something particularly great. Alternative would be to
use target_link_libraries() directly, which sounds like more code
but which is more explicit and allows to have more flexibility
and control comparing to wrapper approach.

Tested the following configurations on Linux, macOS and Windows:

  • make full debug developer
  • make full release developer
  • make lite debug developer
  • make lite release developer
NOTE: Linux libraries needs to be compiled with D6641 applied, otherwise, depending on configuration, it's possible to run into duplicated zlib symbols error.

Diff Detail

Repository
rB Blender

Event Timeline

Brecht Van Lommel (brecht) requested changes to this revision.EditedJan 21 2020, 6:16 PM

Much better solution than my hack, just some minor notes.

CMakeLists.txt
783–788

Is it intentional that the asan linker flags are left out as well if there is no library to link?

If would think this endif() needs to move up two lines.

build_files/cmake/macros.cmake
290–306

This could use a comment explaining what it does.

extern/mantaflow/CMakeLists.txt
96

OpenVDB depends on boost, regardless if blsoc is used.

intern/cycles/kernel/osl/CMakeLists.txt
37–38

This should be guarded by if(WITH_OPENCOLORIO).

We have a direct dependency on OpenColorIO in intern/cycles/render/CMakeLists.txt, so it should be added there.

OSL does not use OpenColorIO so I'm not sure it's needed here. OpenImageIO can but we don't do it for make deps.

This revision now requires changes to proceed.Jan 21 2020, 6:16 PM

seems ok, in the future transitioning to modern cmake targets rather than sprinkling includes all over the place may not be the worst idea though, however that will be a rather significant undertaking and shouldn't stop this diff.

Indeed nice improvement.

FWIW, tested here on my machine (Debian64 testing, using install_deps-generated libs environment), and both full debug+ASAN and full release builds worked fine.

extern/mantaflow/CMakeLists.txt
93

missing indentation ;)

Addressed majority of feedback.

Missing parts:

  • ASAN library. See inlined comment.
  • OpenColorIO in kernel_osl.

    Need more time to investigate. Removing dependency all together seems to pass some of compile tests. Need to run more configurations and platforms to verify. But agree with Brecht this is something odd to have.

Removed OpenColorIO from cycles_kernel_sol

Seems to work on all platform with all configurations listed in the
original description.

CMakeLists.txt
783–788

I don't really know what you mean here.

All these three lines are referencing COMPILER_ASAN_LIBRARY which is not defined on macOS.

Brecht Van Lommel (brecht) added inline comments.
CMakeLists.txt
783–788

I misunderstand what was happening here, the code is fine as is.

This revision is now accepted and ready to land.Jan 22 2020, 3:46 PM