install_deps.sh issues on arch linux
Closed, ResolvedPublic

Description

System Information
Arch

Blender Version
All

Short description of error
install_deps.sh references packages "llvm35" and "clang35" which no longer exist in the Arch "Extra" repo - only the "llvm" and "clang" packages are there now. Python in the Arch repo is 3.7.0 which also now causes build issues.

Further info
install_deps.sh cannot install the nonexistant llvm35 and clang35 packages so tries to build LLVM3.4 which fails with the error below:
[ 7%] Building CXX object lib/Object/CMakeFiles/LLVMObject.dir/Object.cpp.o
[ 7%] Building CXX object lib/Object/CMakeFiles/LLVMObject.dir/ObjectFile.cpp.o
[ 7%] Building CXX object lib/Object/CMakeFiles/LLVMObject.dir/YAML.cpp.o
[ 7%] Linking CXX static library ../libLLVMObject.a
[ 7%] Built target LLVMObject
make: *** [Makefile:152: all] Error 2
ERROR! LLVM-3.4 failed to compile, exiting

Furthermore, it checks for a specific version of LLVM rather than using -ge like some of the other packages.

Modifying the package names in install_deps.sh and trying to force 3.4.2 with --ver-llvm=3.4.2 fails with the error below:
[ 7%] Linking CXX static library ../libLLVMTableGen.a
[ 7%] Built target LLVMTableGen
make: *** [Makefile:152: all] Error 2
ERROR! LLVM-3.4.2 failed to compile, exiting

releases.llvm.org changed their archive format from tar,gz to tar.xz after version 3.4.2 so it's necessary to modify LLVM_SOURCE and LLVM_CLANG_SOURCE to use anything newer. Changing these and passing --ver-llvm=6.0.1 (and also modifying install_deps.sh to use -ge version matching for LLVM) works.

install_deps.sh emits PYTHON_VER_MIN as the installed python version to cmake rather than the installed or compiled version. I guess there is a reason for this so I added a PYTHON_VER_MAX and modified install_deps.sh to use the -ge-lt version check so that it compiles 3.6.2 instead of trying to use 3.7 from the repo.

Exact steps for others to reproduce the error
Run install_deps.sh on a machine running an Arch-based distro (e.g. Manjaro)

Fix
I'm pretty terrible with Git so here is a modified version of install_deps.sh instead: https://github.com/johnakki/blender-misc-scripts/blob/master/install_deps.sh

Details

Type
Bug
Bastien Montagne (mont29) triaged this task as Normal priority.

Thanks for the report. However, we are in the process of heavy update of our libs, including llvm/clang/OSL and python, so I’d rather not touch that script for now. Will keep the report open though, for when I tackle the install_deps update.

Updated script to latest libs (in sync with windows/osx/linux official builds too). Can you please check if it works as-is for you now?

Note that this script atempts to use as much as possible distro dev packages, that’s why it is as little as possible picky on versions. E.g. about py version, 3.6 is (well, was) the minimal version, using 3.7 instead should not have been a big issue?

As for llvm, it was mandatory to stick to 3.4 until a few weeks ago, other libs (OSL) were not able to work correctly with newer ones.

Was a long due update of the whole Cycles-related libraries ;)

Bastien Montagne (mont29) lowered the priority of this task from Normal to Incomplete.Sep 6 2018, 4:08 PM

I'm a bit rushed at the moment but should have time to test properly on a clean system later today.

Just a couple of things I noticed:
*Could you change the package names on lines 3832 to 3836 from llvm35/clang35 to llvm and clang so that the repo packages are used if they're of high enough version?
*Lines 3863-3870 can presumably be uncommented now so that openshadinglanguage can be installed from the repo. This might also negate the need to install/compile llvm since I think it's only a build dependency I believe.

I unfortunately didn't make a note of the exact issue that python37 was giving me and I've just been using python36 since. I'll give it another test later.

Thanks for looking into this!

Done for the llvm/clang package names, thanks.

As for trying to use ARCH OSL package… I’d rather see that tested first, before I change it on the repo. If that works, indeed we can get rid of llvm/clang explicit installation, OSL package will handle itself the required dependencies.

Sorry for the delay.
In a clean chroot using the master branch, install_deps.sh --with-all runs successfully and just compiles osl and alembic.

Ran with:
    install_deps.sh --with-all


If you're using CMake add this to your configuration flags:
  -D WITH_CODEC_SNDFILE=ON
  -D PYTHON_VERSION=3.7
  -D WITH_OPENCOLORIO=ON
  -D WITH_CYCLES_OSL=ON
  -D WITH_LLVM=ON
  -D LLVM_VERSION=6.0.1
  -D OSL_ROOT_DIR=/opt/lib/osl
  -D WITH_OPENSUBDIV=ON
  -D WITH_OPENVDB=ON
  -D WITH_OPENVDB_BLOSC=ON
  -D WITH_OPENCOLLADA=ON
  -D WITH_JACK=ON
  -D WITH_JACK_DYNLOAD=ON
  -D WITH_ALEMBIC=ON
  -D ALEMBIC_ROOT_DIR=/opt/lib/alembic
  -D WITH_CODEC_FFMPEG=ON
  -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;xvidcore;vpx;mp3lame;x264;openjpeg'

Or even simpler, just run (in your blender-source dir):
  make -j4 BUILD_CMAKE_ARGS="-U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0.1 -D OSL_ROOT_DIR=/opt/lib/osl -D WITH_OPENSUBDIV=ON -D WITH_OPENVDB=ON -D WITH_OPENVDB_BLOSC=ON -D WITH_OPENCOLLADA=ON -D WITH_JACK=ON -D WITH_JACK_DYNLOAD=ON -D WITH_ALEMBIC=ON -D ALEMBIC_ROOT_DIR=/opt/lib/alembic -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;xvidcore;vpx;mp3lame;x264;openjpeg'"

Or in all your build directories:
  cmake -U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0.1 -D OSL_ROOT_DIR=/opt/lib/osl -D WITH_OPENSUBDIV=ON -D WITH_OPENVDB=ON -D WITH_OPENVDB_BLOSC=ON -D WITH_OPENCOLLADA=ON -D WITH_JACK=ON -D WITH_JACK_DYNLOAD=ON -D WITH_ALEMBIC=ON -D ALEMBIC_ROOT_DIR=/opt/lib/alembic -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;xvidcore;vpx;mp3lame;x264;openjpeg' .

But...
If I then just run make on it's own with no CMakeCache.txt, it doesn't locate the python libs

Configuring Blender in "/root/blender/master/build_linux" ...
-- The C compiler identification is GNU 8.2.1
-- The CXX compiler identification is GNU 8.2.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test SUPPORT_SSE_BUILD
-- Performing Test SUPPORT_SSE_BUILD - Success
-- SSE Support: detected.
-- Performing Test SUPPORT_SSE2_BUILD
-- Performing Test SUPPORT_SSE2_BUILD - Success
-- SSE2 Support: detected.
-- Found Git: /usr/bin/git (found version "2.18.0") 
CMake Warning at CMakeLists.txt:798 (message):
  Translation path '/root/blender/master/blender/release/datafiles/locale' is
  missing, This is a 'git submodule', which are known not to work with
  bridges to other version control systems, disabling 'WITH_INTERNATIONAL'.


CMake Warning at CMakeLists.txt:820 (message):
  Addons path '/root/blender/master/blender/release/scripts/addons' is
  missing, This is a 'git submodule', which are known not to work with
  bridges to other version control systems: * CONTINUING WITHOUT ADDONS *


-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found X11: /usr/lib/libX11.so
-- Found JPEG: /usr/lib/libjpeg.so (found version "80") 
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.11") 
-- Found PNG: /usr/lib/libpng.so (found version "1.6.34") 
-- Found Freetype: /usr/lib/libfreetype.so (found version "2.9.1") 
CMake Error at /usr/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonLibsUnix (missing: PYTHON_LIBRARY PYTHON_LIBPATH
  PYTHON_INCLUDE_DIR PYTHON_INCLUDE_CONFIG_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  build_files/cmake/Modules/FindPythonLibsUnix.cmake:181 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  build_files/cmake/platform/platform_unix.cmake:82 (find_package)
  CMakeLists.txt:933 (include)


-- Configuring incomplete, errors occurred!
See also "/root/blender/master/build_linux/CMakeFiles/CMakeOutput.log".
make: *** [GNUmakefile:140: all] Error 1

This can be worked around by just adding the -D PYTHON_VERSION=3.7 cmake arg mentioned in the output of install_deps.sh, but I guess FindPackageHandleStandardArgs.cmake should be able to locate it without?

If I clear my CMakeCache.txt again and run make -j4 BUILD_CMAKE_ARGS="-U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0.1 -D OSL_ROOT_DIR=/opt/lib/osl -D WITH_OPENSUBDIV=ON -D WITH_OPENVDB=ON -D WITH_OPENVDB_BLOSC=ON -D WITH_OPENCOLLADA=ON -D WITH_JACK=ON -D WITH_JACK_DYNLOAD=ON -D WITH_ALEMBIC=ON -D ALEMBIC_ROOT_DIR=/opt/lib/alembic -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;xvidcore;vpx;mp3lame;x264;openjpeg'", everything is successful until linking openvdb to the blender binary. I changed WITH_OPENVDB_BLOSC to OFF but got the same linker error. Here's the relevant make output

and CMakeCache.txt

Blender builds fine if I set WITH_OPENVDB=OFF but I've not been able to figure out the right CMake flags to get it to compile with it enabled. Here is a listing of files owned by the openvdb package - hopefully this is of some use.

To test out using the repo versions of osl and alembic, I renamed /opt/lib, deleted CMakeCache.txt and ran
make -j4 BUILD_CMAKE_ARGS="-U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0.1 -D WITH_OPENSUBDIV=ON -D WITH_OPENVDB=OFF -D WITH_OPENVDB_BLOSC=OFF -D WITH_OPENCOLLADA=ON -D WITH_JACK=ON -D WITH_JACK_DYNLOAD=ON -D WITH_ALEMBIC=ON -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;xvidcore;vpx;mp3lame;x264;openjpeg'" (note removed OSL_ROOT_DIR and ALEMBIC_ROOT_DIR).
The compile completed but blender segfaulted when I tried to add an OSL shader. Log is here

Do you want me to do anything to troubleshoot this further?

I compiled Blender again using the compiled osl package and OSL shaders rendered correctly. I couldn't remember if alembic was for particles/fluid/smoke so I added and baked all three (still using alembic from the repo). This all worked fine.

Thanks

Eeek sorry, forgot to come back and check here…

About OpenVDB, that’s probably because blender tries to link against your ARCH package… If you add something like -D OPENVDB_ROOT_DIR=/opt/lib/openvdb -D BLOSC_ROOT_DIR=/opt/lib/blosc as CMake options, should work as expected?

About OSL, if it works with install_deps-compiled libs, then let's just stick to that, LLVM/OSL is a very complicated nightmare, let’s just stick to exact versions we officially support for now.

Alembic is only used for explicit Import/Export to that format, and mesh cache/mesh sequence cache modifiers afaik…

OK, think we can assume this is fixed :)