Page MenuHome

CMake: Linux, Hard-code all paths to pre-compiled libraries
AbandonedPublic

Authored by Sergey Sharybin (sergey) on Feb 19 2020, 11:22 AM.

Details

Summary

Due to lack of standard in how FindFOO modules work, and, possibly, how
CMake's find helpers work as well, it was possible that system-wide
libraries are used.

This change ensures that all libraries are used from their pre-compiled
location.

There might be some trick to utilize some of CMake's functionality which
will work reliable for all libraries without splitting configuration
code paths, but things I've tried didn't really work for me and nobody
is else is looking into them. I'd rather have stupid-looking solution
which works for users rather than spend a fortune looking for an ideal
solution. This can be changed any time anyway.

Diff Detail

Repository
rB Blender
Branch
arcpatch-D6888 (branched from master)
Build Status
Buildable 6723
Build 6723: arc lint + arc unit

Event Timeline

I'd prefer to avoid this, but having looked into this issue as well I can't find another reliable solution.

For example, even FindPNG.cmake has problems, it will prefer a system libpng.a over our libpng16.a and there seems to be no way to specify a path to change that.

For some background:
https://devtalk.blender.org/t/linking-errors-with-prebuild-libs-on-linux/11797

I really would like to fix this quickly so things are not broken for users, and if we can come up with a better solution we can do it later.

This revision is now accepted and ready to land.Feb 19 2020, 11:37 AM

Hi,
libpng.a is given as "missing". In the svn centos lib it's called libpng16.a.

EDIT: Ok, just it's the same concern has brecht.

Missing libLLVMX86AsmPrinter.a.

build_files/cmake/platform/platform_unix_precompiled.cmake
38

change to libpng16.a?

178

Missing on linux_centos_x86_64 svn libs.

I only have WSL/Centos to test on but i needed the following tweaks for it to build

diff --git a/build_files/cmake/platform/platform_unix_precompiled.cmake b/build_files/cmake/platform/platform_unix_precompiled.cmake
index dad8268ab1d..f5be12edb7f 100644
--- a/build_files/cmake/platform/platform_unix_precompiled.cmake
+++ b/build_files/cmake/platform/platform_unix_precompiled.cmake
@@ -34,7 +34,7 @@ set(ZLIB_LIBRARIES ${ZLIB_LIBRARY})
 set(PNG_FOUND TRUE)
 set(PNG_ROOT ${LIBDIR}/png)
 set(PNG_INCLUDE_DIR ${PNG_ROOT}/include ${ZLIB_INCLUDE_DIR})
-set(PNG_LIBRARY ${PNG_ROOT}/lib/libpng.a ${ZLIB_LIBRARY})
+set(PNG_LIBRARY ${PNG_ROOT}/lib/libpng16.a ${ZLIB_LIBRARY})
 set(PNG_INCLUDE_DIRS ${PNG_INCLUDE_DIR})
 set(PNG_LIBRARIES ${PNG_LIBRARY})
 
@@ -174,7 +174,6 @@ set(LLVM_LIBRARY
   ${LLVM_ROOT}/lib/libLLVMX86Desc.a
   ${LLVM_ROOT}/lib/libLLVMMCDisassembler.a
   ${LLVM_ROOT}/lib/libLLVMX86Info.a
-  ${LLVM_ROOT}/lib/libLLVMX86AsmPrinter.a
   ${LLVM_ROOT}/lib/libLLVMX86Utils.a
   ${LLVM_ROOT}/lib/libLLVMMCJIT.a
   ${LLVM_ROOT}/lib/libLLVMInterpreter.a
@@ -203,6 +202,10 @@ set(LLVM_LIBRARY
   ${LLVM_ROOT}/lib/libLLVMBinaryFormat.a
   ${LLVM_ROOT}/lib/libLLVMSupport.a
   ${LLVM_ROOT}/lib/libLLVMDemangle.a
+  ${LLVM_ROOT}/lib/libLLVMRemarks.a
+  ${LLVM_ROOT}/lib/libLLVMDebugInfoDWARF.a
+  ${LLVM_ROOT}/lib/libLLVMAggressiveInstCombine.a
+  ${LLVM_ROOT}/lib/libLLVMBitstreamReader.a
 )
 
 # PCRE
@@ -277,6 +280,7 @@ set(OPENIMAGEDENOISE_LIBRARIES
 )
 
 # OPENIMAGEIO
+set(OPENIMAGEIO_PUGIXML_FOUND TRUE)
 set(OPENIMAGEIO_FOUND TRUE)
 set(OPENIMAGEIO_ROOT ${LIBDIR}/openimageio)
 set(OPENIMAGEIO_INCLUDE_DIRS ${OPENIMAGEIO_ROOT}/include)
@@ -381,6 +385,7 @@ set(USD_INCLUDE_DIR ${USD_ROOT}/include)
 set(USD_LIBRARY ${USD_ROOT}/lib/libusd_m.a)
 set(USD_INCLUDE_DIRS ${USD_INCLUDE_DIR})
 set(USD_LIBRARIES ${USD_LIBRARY})
+set(USD_LIBRARY_DIR ${LIBDIR}/usd/lib/usd)
 
 # XML2
 set(XML2_FOUND TRUE)

@natecraddock has a branch which avoids the need for this patch: temp-precompiled-cmake.

I tested this branch and the system libraries aren't being included.

While everything compiles it looks like the libraries from CentOS aren't compatible with ArchLinux & Suse. It's looking for symbols which aren't available (scanned all *.so and *.a files under /usr/ to be sure I wasn't missing something, also checked these symbols aren't defined under lib/linux_centos7_x86_64).

For references, here are the errors we're running into P1268.


Could someone currently using pre-compiled libraries check the temp-precompiled-cmake branch?

@Campbell Barton (campbellbarton) If by precompiled libraries you mean the lib folder from SVN, then yes, I tested compiling using those libraries with the temp-precompiled-cmake, and on Arch Linux I have the same results as you pasted in your comment, namely undefined references to __pow_finite and similar functions.

I'm fine using the temp-precompiled-cmake approach as well. It will need to be added for all libraries though. At least libpng is a case that is giving issues which is not fixed by that branch.

While everything compiles it looks like the libraries from CentOS aren't compatible with ArchLinux & Suse. It's looking for symbols which aren't available (scanned all *.so and *.a files under /usr/ to be sure I wasn't missing something, also checked these symbols aren't defined under lib/linux_centos7_x86_64).

For references, here are the errors we're running into P1268.

Have you tried adding libm explicitly for linking? And maybe as dependencies for the libraries that give problems, so it's in the right order? I know you didn't find the symbols, but maybe there something unusual about those symbols.

Presumably the Blender release works on these systems? I guess that means it is linking these symbols statically. Maybe there is a way include the centos libm.a as part of our libraries, but that sounds tricky.

Yes, I downloaded the libm from centos7 and replacing the -lm in the linker command with it.

The static library fails to link - saying it needs to be built with -fPIC.
The dynamic library didn't work either, although it didn't give any useful error.

The official blender.org binaries work, so statically linking libm should work.

  • small changes to make it link on WSL/Centos6

We can't easily provide libm with other libraries: it's part of libc and trying to match different parts of it from different versions will be a very wonderful experience.

I think the reason those symbols are "problematic" is because they started to fade away at around libc 2.29-2.30, and and they are removed in 2.31 (can no longer see any aliases for them in 2.31). Or, maybe, Arch never included those aliases.

One possible solution is to have libc-compat library, similar to FFmpeg, where we will implement those symbols.

Another possible solution is to somehow forbid CentOS from using them. Not sure how to do this though, those are related on fast-math, which we do not enable explicitly but some of libraries are using it.

Out of curiosity, what gcc -O2 --fast-math -c main.c && nm main.o for

#include <stdio.h>
#include <math.h>

static double foo(double input) {
  return input * pow(input, input);
}

int main(int argc, char** argv) {
  printf("%f\n", foo(argc));
  return 0;
}

shows on Arch?

This outputs:

0000000000000000 r .LC0
                 U _GLOBAL_OFFSET_TABLE_
0000000000000000 T main
                 U pow
                 U printf

With clang -O2 -ffast-math -c main.c && nm main.o

0000000000000000 r .L.str
                 U __powidf2
0000000000000000 T main
                 U printf

With centos6/WSL and the temp-precompiled-cmake branch the only things in /usr being picked up are

JACK_LIBRARY:FILEPATH=/usr/lib64/libjack.so
OPENGLES_EGL_LIBRARY:FILEPATH=/usr/lib64/libEGL.so
OPENGLES_LIBRARY:FILEPATH=/usr/lib64/libGLESv2.so
OPENGL_egl_LIBRARY:FILEPATH=/usr/lib64/libEGL.so
OPENGL_glx_LIBRARY:FILEPATH=/usr/lib64/libGLX.so
OPENGL_opengl_LIBRARY:FILEPATH=/usr/lib64/libOpenGL.so
OpenMP_LIBRARIES:FILEPATH=/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/libgomp.a
OpenMP_gomp_LIBRARY:FILEPATH=/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/libgomp.a
X11_ICE_LIB:FILEPATH=/usr/lib64/libICE.so
X11_SM_LIB:FILEPATH=/usr/lib64/libSM.so
X11_X11_LIB:FILEPATH=/usr/lib64/libX11.so
X11_Xau_LIB:FILEPATH=/usr/lib64/libXau.so
X11_Xcursor_LIB:FILEPATH=/usr/lib64/libXcursor.so
X11_Xdamage_LIB:FILEPATH=/usr/lib64/libXdamage.so
X11_Xext_LIB:FILEPATH=/usr/lib64/libXext.so
X11_Xfixes_LIB:FILEPATH=/usr/lib64/libXfixes.so
X11_Xi_LIB:FILEPATH=/usr/lib64/libXi.so
X11_Xinerama_LIB:FILEPATH=/usr/lib64/libXinerama.so
X11_Xrandr_LIB:FILEPATH=/usr/lib64/libXrandr.so
X11_Xrender_LIB:FILEPATH=/usr/lib64/libXrender.so
X11_Xt_LIB:FILEPATH=/usr/lib64/libXt.so
X11_Xxf86vm_LIB:FILEPATH=/usr/lib64/libXxf86vm.so
X11_Xxf86vmode_LIB:FILEPATH=/usr/lib64/libXxf86vm.so

which all seems expected, so yeah seems fine.

I may actually give this a whirl on windows as well.

So there are two problems to solve: