Buildbot is ignoring important CMake settings in buildbot/config/blender_linux.cmake #88431

Closed
opened 2021-05-20 16:02:43 +02:00 by Sybren A. Stüvel · 28 comments

Blender Version
Broken: all builds made with the new buildbot
Worked: old buildbot

Short description of error
The new buildbot infrastructure ignores the settings in buildbot/config/blender_linux.cmake.

Note that this affects other builds as well: 2.83, 2.93, and master.

Exact steps for others to reproduce the error
Run ldd bin/blender on a recent buildbot build, and see that it links to libpulse.so.0:

$ ldd blender-3.0.0-alpha+master.47e88345a154-linux.x86_64-release/blender
        linux-vdso.so.1 (0x00007ffe468fc000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5e27021000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5e26e02000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5e26bfe000)
        libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f5e268c6000)
        libXi.so.6 => /usr/lib/x86_64-linux-gnu/libXi.so.6 (0x00007f5e266b6000)
        libXxf86vm.so.1 => /usr/lib/x86_64-linux-gnu/libXxf86vm.so.1 (0x00007f5e264b0000)
        libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f5e262aa000)
        libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f5e260a0000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5e25d02000)
        libGL.so.1 => /usr/lib/x86_64-linux-gnu/libGL.so.1 (0x00007f5e25a76000)
        libpulse.so.0 => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e25685000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5e27224000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5e2546d000)
        libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f5e25245000)
        libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f5e25033000)
        libGLX.so.0 => /usr/lib/x86_64-linux-gnu/libGLX.so.0 (0x00007f5e24e02000)
        libGLdispatch.so.0 => /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0 (0x00007f5e24b4c000)
        libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f5e24948000)
        libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f5e24742000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f5e2452d000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5e24325000)

This library (and others) should be dynloaded, as per the config in blender_linux.cmake. Furthermore, that file contains some important linker options (CMAKE_EXE_LINKER_FLAGS).

Suggested short-term solution: update the buildbot to also load that configuration file.

Suggested longer-term solution: remove the build_files/buildbot directory altogether, and make sure the regular CMake config build_files/cmake/config/blender_release.cmake is updated so that it has the correct options for Blender release versions.

**Blender Version** Broken: all builds made with the [new buildbot](https://wiki.blender.org/wiki/Infrastructure/BuildBot) Worked: old buildbot **Short description of error** The new buildbot infrastructure ignores the settings in [buildbot/config/blender_linux.cmake](https://developer.blender.org/diffusion/B/browse/master/build_files/buildbot/config/blender_linux.cmake). Note that this affects other builds as well: 2.83, 2.93, and `master`. **Exact steps for others to reproduce the error** Run `ldd bin/blender` on a recent buildbot build, and see that it links to `libpulse.so.0`: ``` $ ldd blender-3.0.0-alpha+master.47e88345a154-linux.x86_64-release/blender linux-vdso.so.1 (0x00007ffe468fc000) libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5e27021000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5e26e02000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5e26bfe000) libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f5e268c6000) libXi.so.6 => /usr/lib/x86_64-linux-gnu/libXi.so.6 (0x00007f5e266b6000) libXxf86vm.so.1 => /usr/lib/x86_64-linux-gnu/libXxf86vm.so.1 (0x00007f5e264b0000) libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f5e262aa000) libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f5e260a0000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5e25d02000) libGL.so.1 => /usr/lib/x86_64-linux-gnu/libGL.so.1 (0x00007f5e25a76000) libpulse.so.0 => not found libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e25685000) /lib64/ld-linux-x86-64.so.2 (0x00007f5e27224000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5e2546d000) libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f5e25245000) libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f5e25033000) libGLX.so.0 => /usr/lib/x86_64-linux-gnu/libGLX.so.0 (0x00007f5e24e02000) libGLdispatch.so.0 => /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0 (0x00007f5e24b4c000) libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f5e24948000) libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f5e24742000) libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f5e2452d000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5e24325000) ``` This library (and others) should be dynloaded, as per the config in `blender_linux.cmake`. Furthermore, that file contains some important linker options (`CMAKE_EXE_LINKER_FLAGS`). **Suggested short-term solution:** update the buildbot to also load that configuration file. **Suggested longer-term solution:** remove the `build_files/buildbot` directory altogether, and make sure the regular CMake config `build_files/cmake/config/blender_release.cmake` is updated so that it has the correct options for Blender release versions.

Added subscriber: @dr.sybren

Added subscriber: @dr.sybren

#88387 was marked as duplicate of this issue

#88387 was marked as duplicate of this issue
James Monteath was assigned by Sybren A. Stüvel 2021-05-20 16:04:16 +02:00

I'm classifying this as "Unbreak Now!" as 2.93 is in BCON4 now, and this really must be resolved before a release is released into the wild.

I'm classifying this as "Unbreak Now!" as 2.93 is in BCON4 now, and this really must be resolved before a release is released into the wild.

Buildbot always used the build_files/cmake/config/blender_release.cmake file.
I don't think we can remove all of build_files/buildbot just yet.

I would need to have a chat to get more details.

Buildbot always used the `build_files/cmake/config/blender_release.cmake` file. I don't think we can remove all of `build_files/buildbot` just yet. I would need to have a chat to get more details.

Added subscriber: @Sergey

Added subscriber: @Sergey

In #88431#1163442, @ponderz wrote:
Buildbot always used the build_files/cmake/config/blender_release.cmake file.

Yes, but indirectly on Linux. It would use buildbot/config/blender_linux.cmake, which first runs include("${CMAKE_CURRENT_LIST_DIR}/../../cmake/config/blender_release.cmake") and then sets some extra settings. Source: @Sergey is sitting here next to me saying "yeah, yeah" while I type this :)

In the end, the previous buildbot produced files that don't directly link libpulse.so.0 and the current buildbot produces files that do link there. Since PulseAudio is not installed on the render farm workers, Blender can't run there. Even though it's started with the -noaudio CLI option.

> In #88431#1163442, @ponderz wrote: > Buildbot always used the `build_files/cmake/config/blender_release.cmake` file. Yes, but indirectly on Linux. It would use `buildbot/config/blender_linux.cmake`, which first runs `include("${CMAKE_CURRENT_LIST_DIR}/../../cmake/config/blender_release.cmake")` and then sets some extra settings. Source: @Sergey is sitting here next to me saying "yeah, yeah" while I type this :) In the end, the previous buildbot produced files that don't directly link `libpulse.so.0` and the current buildbot produces files that do link there. Since PulseAudio is not installed on the render farm workers, Blender can't run there. Even though it's started with the `-noaudio` CLI option.

Ok, I can change the pipeline to always use the correct cmake file.
This file would be different per tracked version ?

Ok, I can change the pipeline to always use the correct cmake file. This file would be different per tracked version ?

Changed status from 'Needs Triage' to: 'Confirmed'

Changed status from 'Needs Triage' to: 'Confirmed'

In #88431#1163473, @ponderz wrote:
Ok, I can change the pipeline to always use the correct cmake file.

👍

This file would be different per tracked version ?

What does that mean?

> In #88431#1163473, @ponderz wrote: > Ok, I can change the pipeline to always use the correct cmake file. :+1: > This file would be different per tracked version ? What does that mean?

Added subscriber: @brecht

Added subscriber: @brecht

As far as I can tell this is just an error in the work_compile.py in the bdr-devops-core repo.

It sets config_file_path to blender_linux.cmake too late.

As far as I can tell this is just an error in the `work_compile.py` in the `bdr-devops-core` repo. It sets `config_file_path` to `blender_linux.cmake` too late.

I could install PulseAudio if we need it ?

I could install PulseAudio if we need it ?

From the logs, I think it's already there.
https://builder.blender.org/admin/#/builders/10/builds/257/steps/6/logs/stdio

However it seems to not find OpenMP and Jack, which is an issue.

From the logs, I think it's already there. https://builder.blender.org/admin/#/builders/10/builds/257/steps/6/logs/stdio However it seems to not find OpenMP and Jack, which is an issue.

Ah easy fix this, will check

Ah easy fix this, will check

Do we need to install missing packages on the machines ?

Do we need to install missing packages on the machines ?

Added subscriber: @Memento

Added subscriber: @Memento

In #88431#1163528, @brecht wrote:
From the logs, I think it's already there.
https://builder.blender.org/admin/#/builders/10/builds/257/steps/6/logs/stdio

However it seems to not find OpenMP and Jack, which is an issue.

And PipeWire, as it is used in the latest Fedora 34 Workstation?

> In #88431#1163528, @brecht wrote: > From the logs, I think it's already there. > https://builder.blender.org/admin/#/builders/10/builds/257/steps/6/logs/stdio > > However it seems to not find OpenMP and Jack, which is an issue. And PipeWire, as it is used in the latest Fedora 34 Workstation?

Installed pulseaudio on all Centos 7 machines

[blender@pvet-lvm-centos-7-x64-07 ~]$ ldconfig -p | grep -i "gomp"
        libgomp.so.1 (libc6,x86-64) => /lib64/libgomp.so.1

[blender@pvet-lvm-centos-7-x64-07 ~]$ ldconfig -p | grep -i "pulse"
        libpulse.so.0 (libc6,x86-64) => /lib64/libpulse.so.0
        libpulse.so (libc6,x86-64) => /lib64/libpulse.so
        libpulse-simple.so.0 (libc6,x86-64) => /lib64/libpulse-simple.so.0
        libpulse-simple.so (libc6,x86-64) => /lib64/libpulse-simple.so
        libpulse-mainloop-glib.so.0 (libc6,x86-64) => /lib64/libpulse-mainloop-glib.so.0
        libpulse-mainloop-glib.so (libc6,x86-64) => /lib64/libpulse-mainloop-glib.so

[blender@pvet-lvm-centos-7-x64-07 ~]$ ldconfig -p | grep -i "jack"
        libjackserver.so.0 (libc6,x86-64) => /lib64/libjackserver.so.0
        libjacknet.so.0 (libc6,x86-64) => /lib64/libjacknet.so.0
        libjack.so.0 (libc6,x86-64) => /lib64/libjack.so.0
Installed pulseaudio on all Centos 7 machines ``` [blender@pvet-lvm-centos-7-x64-07 ~]$ ldconfig -p | grep -i "gomp" libgomp.so.1 (libc6,x86-64) => /lib64/libgomp.so.1 [blender@pvet-lvm-centos-7-x64-07 ~]$ ldconfig -p | grep -i "pulse" libpulse.so.0 (libc6,x86-64) => /lib64/libpulse.so.0 libpulse.so (libc6,x86-64) => /lib64/libpulse.so libpulse-simple.so.0 (libc6,x86-64) => /lib64/libpulse-simple.so.0 libpulse-simple.so (libc6,x86-64) => /lib64/libpulse-simple.so libpulse-mainloop-glib.so.0 (libc6,x86-64) => /lib64/libpulse-mainloop-glib.so.0 libpulse-mainloop-glib.so (libc6,x86-64) => /lib64/libpulse-mainloop-glib.so [blender@pvet-lvm-centos-7-x64-07 ~]$ ldconfig -p | grep -i "jack" libjackserver.so.0 (libc6,x86-64) => /lib64/libjackserver.so.0 libjacknet.so.0 (libc6,x86-64) => /lib64/libjacknet.so.0 libjack.so.0 (libc6,x86-64) => /lib64/libjack.so.0 ```

Starting linux builds on all branches.
See if that helps for some of the issues.

Starting linux builds on all branches. See if that helps for some of the issues.

Issues ?
OpenMP

-- Could NOT find OpenMP_C (missing: OpenMP_pthread_LIBRARY) (found version "4.5")
-- Could NOT find OpenMP_CXX (missing: OpenMP_pthread_LIBRARY) (found version "4.5")
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND) 

Jack

-- Could NOT find Jack (missing: JACK_LIBRARY JACK_INCLUDE_DIR) 

Anyone know the package names for those on Centos 7 ?

Issues ? OpenMP ``` -- Could NOT find OpenMP_C (missing: OpenMP_pthread_LIBRARY) (found version "4.5") -- Could NOT find OpenMP_CXX (missing: OpenMP_pthread_LIBRARY) (found version "4.5") -- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND) ``` Jack ``` -- Could NOT find Jack (missing: JACK_LIBRARY JACK_INCLUDE_DIR) ``` Anyone know the package names for those on Centos 7 ?
-- Found Jack: /usr/lib64/libjack.so  
-- Found Pulse: /usr/lib64/libpulse.so  
``` -- Found Jack: /usr/lib64/libjack.so -- Found Pulse: /usr/lib64/libpulse.so ```

I added a filter to show missing options into warnings

{F10133532,size=full}

I added a filter to show missing options into warnings {[F10133532](https://archive.blender.org/developer/F10133532/image.png),size=full}

https://builder.blender.org/admin/#/builders/10/builds/268/steps/6/logs/stdio

-- Could NOT find Jack (missing: JACK_LIBRARY JACK_INCLUDE_DIR) 
-- Found Pulse: /usr/lib64/libpulse.so  
-- Found X11: /usr/include   

Will check the sequence of of the cmake calls in script.

https://builder.blender.org/admin/#/builders/10/builds/268/steps/6/logs/stdio ``` -- Could NOT find Jack (missing: JACK_LIBRARY JACK_INCLUDE_DIR) -- Found Pulse: /usr/lib64/libpulse.so -- Found X11: /usr/include ``` Will check the sequence of of the cmake calls in script.

The config file path was updated after the option was added to command line.

Looking at the logs is best way to find out what happened in the build.
https://builder.blender.org/admin/#/builders/10/builds/268/steps/6/logs/stdio
Line 91-92

INFO:root:scl enable devtoolset-9 -- cmake /home/blender/git/blender-vdev/blender.git -C/home/blender/git/blender-vdev/blender.git/build_files/cmake/config/blender_release.cmake -DCMAKE_BUILD_TYPE:STRING=release -DCMAKE_INSTALL_PREFIX=/home/blender/git/blender-vdev/install_release -DWITH_GTESTS=ON -DCUDA_BUILD_CUBIN=OFF -DWITH_CYCLES_CUDA_BINARIES=OFF -DWITH_CYCLES_CUDA_BUILD_SERIAL=OFF -DWITH_CYCLES_NATIVE_ONLY=OFF -DOPTIX_ROOT_DIR:PATH=/home/blender/git/NVIDIA-Optix-SDK-7.1
loading initial cache file /home/blender/git/blender-vdev/blender.git/build_files/cmake/config/blender_release.cmake

Testing it then will pushing to buildbot.
Will start jobs.

The config file path was updated after the option was added to command line. Looking at the logs is best way to find out what happened in the build. https://builder.blender.org/admin/#/builders/10/builds/268/steps/6/logs/stdio Line 91-92 ``` INFO:root:scl enable devtoolset-9 -- cmake /home/blender/git/blender-vdev/blender.git -C/home/blender/git/blender-vdev/blender.git/build_files/cmake/config/blender_release.cmake -DCMAKE_BUILD_TYPE:STRING=release -DCMAKE_INSTALL_PREFIX=/home/blender/git/blender-vdev/install_release -DWITH_GTESTS=ON -DCUDA_BUILD_CUBIN=OFF -DWITH_CYCLES_CUDA_BINARIES=OFF -DWITH_CYCLES_CUDA_BUILD_SERIAL=OFF -DWITH_CYCLES_NATIVE_ONLY=OFF -DOPTIX_ROOT_DIR:PATH=/home/blender/git/NVIDIA-Optix-SDK-7.1 loading initial cache file /home/blender/git/blender-vdev/blender.git/build_files/cmake/config/blender_release.cmake ``` Testing it then will pushing to buildbot. Will start jobs.

v283

INFO:root:scl enable devtoolset-9 -- cmake /home/blender/git/blender-v283/blender.git -C/home/blender/git/blender-v283/blender.git/build_files/buildbot/config/blender_linux.cmake -DCMAKE_BUILD_TYPE:STRING=release -DCMAKE_INSTALL_PREFIX=/home/blender/git/blender-v283/install_release -DWITH_GTESTS=ON -DCUDA_BUILD_CUBIN=ON -DWITH_CYCLES_CUDA_BINARIES=ON -DWITH_CYCLES_CUDA_BUILD_SERIAL=OFF -DWITH_CYCLES_NATIVE_ONLY=OFF -DOPTIX_ROOT_DIR:PATH=/home/blender/git/NVIDIA-Optix-SDK-7.0 -DCUDA10_TOOLKIT_ROOT_DIR:PATH=/usr/local/cuda-10.1 -DCUDA10_NVCC_EXECUTABLE:STRING=/usr/local/cuda-10.1/bin/nvcc;-ccbin;/opt/rh/devtoolset-8/root/usr/bin/gcc -DCUDA_TOOLKIT_ROOT_DIR:PATH=/usr/local/cuda-11.3 -DCUDA_NVCC_EXECUTABLE:FILEPATH=/usr/local/cuda-11.3/bin/nvcc.exe
loading initial cache file /home/blender/git/blender-v283/blender.git/build_files/buildbot/config/blender_linux.cmake
v283 ``` INFO:root:scl enable devtoolset-9 -- cmake /home/blender/git/blender-v283/blender.git -C/home/blender/git/blender-v283/blender.git/build_files/buildbot/config/blender_linux.cmake -DCMAKE_BUILD_TYPE:STRING=release -DCMAKE_INSTALL_PREFIX=/home/blender/git/blender-v283/install_release -DWITH_GTESTS=ON -DCUDA_BUILD_CUBIN=ON -DWITH_CYCLES_CUDA_BINARIES=ON -DWITH_CYCLES_CUDA_BUILD_SERIAL=OFF -DWITH_CYCLES_NATIVE_ONLY=OFF -DOPTIX_ROOT_DIR:PATH=/home/blender/git/NVIDIA-Optix-SDK-7.0 -DCUDA10_TOOLKIT_ROOT_DIR:PATH=/usr/local/cuda-10.1 -DCUDA10_NVCC_EXECUTABLE:STRING=/usr/local/cuda-10.1/bin/nvcc;-ccbin;/opt/rh/devtoolset-8/root/usr/bin/gcc -DCUDA_TOOLKIT_ROOT_DIR:PATH=/usr/local/cuda-11.3 -DCUDA_NVCC_EXECUTABLE:FILEPATH=/usr/local/cuda-11.3/bin/nvcc.exe loading initial cache file /home/blender/git/blender-v283/blender.git/build_files/buildbot/config/blender_linux.cmake ```

I looked on the old Linux worker machine, OpenMP with pthread was not setup there either.

{F10133817,size=full}

This is not relevant to the current issue.

I looked on the old Linux worker machine, OpenMP with pthread was not setup there either. {[F10133817](https://archive.blender.org/developer/F10133817/image.png),size=full} This is not relevant to the current issue.

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Confirmed' to: 'Resolved'

Added subscribers: @mcq, @scurest, @lichtwerk, @ponderz

Added subscribers: @mcq, @scurest, @lichtwerk, @ponderz
This repo is archived. You cannot comment on issues.
No Milestone
No project
No Assignees
7 Participants
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: archive/blender-buildbot#88431
No description provided.