buildbot codesign fails due to presence of dylibs in binary folder (probably) #90418

New Issue

Closed

opened 2021-08-04 06:06:06 +02:00 by Ankit Meel · 30 comments

Ankit Meel commented 2021-08-04 06:06:06 +02:00

Copy Link

System Information
Buildbot macOS worker

Blender Version
Worked: blender/blender@a25a1f39aa
First bad commit: blender/blender@652fbc2005

Short description of error

https://builder.blender.org/admin/#/builders/9/builds/1202/steps/11/logs/stdio

Loading workspace settings [/Users/blender/.devops/workspace-settings.yaml]
Collecting files to process in [/Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS]
security 
codesign --remove-signature /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender
codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender
/Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender: code object is not signed at all
In subcomponent: /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/libomp.dylib

Exact steps for others to reproduce the error

**System Information** Buildbot macOS worker **Blender Version** Worked: blender/blender@a25a1f39aa First bad commit: blender/blender@652fbc2005 **Short description of error** https://builder.blender.org/admin/#/builders/9/builds/1202/steps/11/logs/stdio ``` Loading workspace settings [/Users/blender/.devops/workspace-settings.yaml] Collecting files to process in [/Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS] security codesign --remove-signature /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender: code object is not signed at all In subcomponent: /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/libomp.dylib ``` **Exact steps for others to reproduce the error**

Ankit Meel commented 2021-08-04 06:06:06 +02:00

Author

Copy Link

Changed status from 'Needs Triage' to: 'Confirmed'

Changed status from 'Needs Triage' to: 'Confirmed'

Ankit Meel commented 2021-08-04 06:06:06 +02:00

Author

Copy Link

Added subscriber: @ankitm

Added subscriber: @ankitm

blender-admin commented 2021-08-04 06:06:06 +02:00

Owner

Copy Link

blender/blender#90517 was marked as duplicate of this issue

blender/blender#90517 was marked as duplicate of this issue

blender-admin commented 2021-08-04 06:06:06 +02:00

Owner

Copy Link

blender/blender#90505 was marked as duplicate of this issue

blender/blender#90505 was marked as duplicate of this issue

James Monteath was assigned by Ankit Meel 2021-08-04 06:06:17 +02:00

Ankit Meel commented 2021-08-04 06:08:55 +02:00

Author

Copy Link

@ponderz We can probably mitigate it by changing the library location from Blender.app/Contents/MacOS/libomp.dylib to Blender.app/Contents/Resources/lib/libomp.dylib if it cannot be fixed on buildbot side.
That's the fallback.

@ponderz We can probably mitigate it by changing the library location from `Blender.app/Contents/MacOS/libomp.dylib` to `Blender.app/Contents/Resources/lib/libomp.dylib` if it cannot be fixed on buildbot side. That's the fallback.

Ankit Meel commented 2021-08-04 07:19:23 +02:00

Author

Copy Link

https://builder.blender.org/admin/#/builders/58/builds/401/steps/11/logs/stdio
Okay yeah we can do that. I don't have a preference eitherway. Just two variable changes.
https://developer.blender.org/rB3985822f48d3c48f5bc9ddfa1bedf620f8df4895

https://builder.blender.org/admin/#/builders/58/builds/401/steps/11/logs/stdio Okay yeah we can do that. I don't have a preference eitherway. Just two variable changes. https://developer.blender.org/rB3985822f48d3c48f5bc9ddfa1bedf620f8df4895

Ankit Meel commented 2021-08-07 18:42:00 +02:00

Author

Copy Link

Added subscriber: @Pyer

Added subscriber: @Pyer

Germano Cavalcante commented 2021-08-07 21:08:38 +02:00

Copy Link

Added subscriber: @mano-wii

Added subscriber: @mano-wii

Germano Cavalcante commented 2021-08-09 19:40:57 +02:00

Copy Link

Added subscriber: @Nurb2Kea

Added subscriber: @Nurb2Kea

Germano Cavalcante commented 2021-08-09 19:43:53 +02:00

Copy Link

Is this still an issue?
blender/blender@ff594715b8 fixed some things in the libraries and may have fixed this problem too.

Is this still an issue? blender/blender@ff594715b8 fixed some things in the libraries and may have fixed this problem too.

Ankit Meel commented 2021-08-09 19:44:48 +02:00

Author

Copy Link

yes
that commit only changes library version, not its location or codesign code on buildbot.

yes that commit only changes library version, not its location or codesign code on buildbot.

Guido commented 2021-08-09 19:55:50 +02:00

Copy Link

master and cycles-x still won't start at all on macos 11.5.1 on different machines!

master and cycles-x still won't start at all on macos 11.5.1 on different machines!

Brecht Van Lommel commented 2021-08-09 22:09:49 +02:00

Copy Link

Added subscriber: @brecht

Added subscriber: @brecht

Brecht Van Lommel commented 2021-08-09 22:09:49 +02:00

Copy Link

With a test build with the Arm libraries, I did not see this issue.
https://builder.blender.org/admin/#/builders/58/builds/401/steps/11/logs/stdio

However if moving the library back to the Resources folder fixes it and tests keep working and finding the correct libomp.dylib (and not another one found on the system), then it's fine to commit that fix I think.

With a test build with the Arm libraries, I did not see this issue. https://builder.blender.org/admin/#/builders/58/builds/401/steps/11/logs/stdio However if moving the library back to the `Resources` folder fixes it and tests keep working and finding the correct `libomp.dylib` (and not another one found on the system), then it's fine to commit that fix I think.

Guido commented 2021-08-09 22:46:03 +02:00

Copy Link

macOS 11.5.1 Intel i7 8 core
Tried to put back the Library from the last working version into the package contens of the versions (see below), but didn't work.

blender-3.0.0-alpha+cycles-x.499008939a48-darwin.x86_64-release
blender-3.0.0-alpha+master.57281b73c451-darwin.x86_64-release

Couldn't find an error log because it isn't starting at all on two differnent machines, but I could get the problem details from apple error message. see below!

macOS 11.5.1 Intel i7 8 core Tried to put back the Library from the last working version into the package contens of the versions (see below), but didn't work. blender-3.0.0-alpha+cycles-x.499008939a48-darwin.x86_64-release blender-3.0.0-alpha+master.57281b73c451-darwin.x86_64-release Couldn't find an error log because it isn't starting at all on two differnent machines, but I could get the problem details from apple error message. see below!   

Brecht Van Lommel commented 2021-08-09 22:58:57 +02:00

Copy Link

@Nurb2Kea, it's best to wait until this task is solved, and then check if you still have the problem. The current build is from August 3, we first have to get a new build working and then we can if it works correctly for everyone or not. What you found may be unrelated to this task.

@Nurb2Kea, it's best to wait until this task is solved, and then check if you still have the problem. The current build is from August 3, we first have to get a new build working and then we can if it works correctly for everyone or not. What you found may be unrelated to this task.

blender-admin commented 2021-08-10 07:00:55 +02:00

Owner

Copy Link

This issue was referenced by blender/blender@b83ee724a4

This issue was referenced by blender/blender@b83ee724a418bee98f4e81e0c6854d03bc2e1fa6

Ankit Meel commented 2021-08-10 07:03:03 +02:00

Author

Copy Link

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Confirmed' to: 'Resolved'

Ankit Meel closed this issue 2021-08-10 07:03:03 +02:00

Brecht Van Lommel commented 2021-08-10 17:30:43 +02:00

Copy Link

Changed status from 'Resolved' to: 'Confirmed'

Changed status from 'Resolved' to: 'Confirmed'

Brecht Van Lommel reopened this issue 2021-08-10 17:30:43 +02:00

Brecht Van Lommel commented 2021-08-10 17:30:43 +02:00

Copy Link

I still see the same error on buildbot for x86_64, but not Arm.
https://builder.blender.org/admin/#/builders/9/builds/1297

And libomp.dylib is now getting installed in both MacOS and Resources/lib, is that intentional?

I still see the same error on buildbot for x86_64, but not Arm. https://builder.blender.org/admin/#/builders/9/builds/1297 And `libomp.dylib` is now getting installed in both `MacOS` and `Resources/lib`, is that intentional?

Ankit Meel commented 2021-08-10 17:51:26 +02:00

Author

Copy Link

And libomp.dylib is now getting installed in both MacOS and Resources/lib, is that intentional?

No

Suggest cleaning the build folder

As I also commented on the commit, the logs show the codesign to be fine. Any build before this one had the codesign issue.
https://builder.blender.org/admin/#/builders/58/builds/460

> And libomp.dylib is now getting installed in both MacOS and Resources/lib, is that intentional? No Suggest cleaning the build folder As I also commented on the commit, the logs show the codesign to be fine. Any build before this one had the codesign issue. https://builder.blender.org/admin/#/builders/58/builds/460

Brecht Van Lommel commented 2021-08-10 18:30:00 +02:00

Copy Link

It seems indeed that a clean build solves both issues.
https://builder.blender.org/admin/#/builders/9/builds/1299/steps/11/logs/stdio

However a subsequent automatic build still fails, since presumably that runs on a different machine.
https://builder.blender.org/admin/#/builders/9/builds/1300/steps/11/logs/stdio

@ponderz, is there a way to clear the macOS build folder on all machines for daily and experimental builds?

In principle this should not be needed, but for some there is something in the build folder that sticks around when it shouldn't.

It seems indeed that a clean build solves both issues. https://builder.blender.org/admin/#/builders/9/builds/1299/steps/11/logs/stdio However a subsequent automatic build still fails, since presumably that runs on a different machine. https://builder.blender.org/admin/#/builders/9/builds/1300/steps/11/logs/stdio @ponderz, is there a way to clear the macOS build folder on all machines for daily and experimental builds? In principle this should not be needed, but for some there is something in the build folder that sticks around when it shouldn't.

James Monteath commented 2021-08-11 08:04:05 +02:00

Copy Link

Few ways to do this:

• Start the number of builds matching number of machines you need to clean. Full clean no delivery.
• Use script that clears the build folders on the specific machines with filter and track id. That I have a way to do this already.
• The previous way could be added to a management coordinator with some choices to run the script with choice of machine etc...

I started 4 clean builds on the PROD Buildbot.

Few ways to do this: - Start the number of builds matching number of machines you need to clean. Full clean no delivery. - Use script that clears the build folders on the specific machines with filter and track id. That I have a way to do this already. - The previous way could be added to a management coordinator with some choices to run the script with choice of machine etc... I started 4 clean builds on the PROD Buildbot. 

James Monteath commented 2021-08-11 08:55:27 +02:00

Copy Link

All good now.

All good now. 

James Monteath commented 2021-08-11 09:00:07 +02:00

Copy Link

Only issue is if any of the experimental branches that did not update from master yet, and continue to build will re-introduce the problem on the machine.

Only issue is if any of the experimental branches that did not update from master yet, and continue to build will re-introduce the problem on the machine.

Ankit Meel commented 2021-08-11 09:49:06 +02:00

Author

Copy Link

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Confirmed' to: 'Resolved'

Ankit Meel closed this issue 2021-08-11 09:49:06 +02:00

Ankit Meel commented 2021-08-11 09:49:06 +02:00

Author

Copy Link

thanks!

thanks!

Ankit Meel commented 2021-08-17 04:40:05 +02:00

Author

Copy Link

d36cf98aa8/security/mac/hardenedruntime/codesign.bash (L115)
A fix on the server side could be signing dylibs before Blender executable.

https://github.com/mozilla/gecko-dev/blob/d36cf98aa85f24ceefd07521b3d16b9edd2abcb7/security/mac/hardenedruntime/codesign.bash#L115 A fix on the server side could be signing dylibs before Blender executable.

James Monteath commented 2021-08-17 11:31:32 +02:00

Copy Link

We already do that.

Any reason why we have the libomp.dylib in 2 different places ?

codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender

codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/libomp.dylib

Collecting files to process in [/Users/blender/git/blender-vdev/install_release/Blender.app/Contents/Resources]
codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/Resources/3.0/lib/libomp.dylib

We already do that. Any reason why we have the `libomp.dylib` in 2 different places ? ``` codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/Blender codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/MacOS/libomp.dylib Collecting files to process in [/Users/blender/git/blender-vdev/install_release/Blender.app/Contents/Resources] codesign --timestamp --options runtime --entitlements=/Users/blender/git/blender-vdev/blender.git/release/darwin/entitlements.plist --sign Stichting Blender Foundation /Users/blender/git/blender-vdev/install_release/Blender.app/Contents/Resources/3.0/lib/libomp.dylib ```

Ankit Meel commented 2021-08-17 11:59:55 +02:00

Author

Copy Link

Any reason why we have the libomp.dylib in 2 different places ?

That is a leftover of stale cmake scripts.
Recent builds don't do that
https://builder.blender.org/admin/#/builders/9/builds/1376/steps/8/logs/stdio

>Any reason why we have the libomp.dylib in 2 different places ? That is a leftover of stale cmake scripts. Recent builds don't do that https://builder.blender.org/admin/#/builders/9/builds/1376/steps/8/logs/stdio

This repo is archived. You cannot comment on issues.

No Branch/Tag Specified

Branches Tags

master

tmp-release-builds

Labels

Clear labels   

legacy module

Development Management

  

legacy module

Platforms, Builds, Tests & Devices

  

legacy module

Python API

  

legacy module

Rendering & Cycles

  

legacy project

3.3

  

legacy project

Development Management

  

legacy project

Documentation

  

legacy project

Infrastructure: Blender Buildbot

  

legacy project

Infrastructure: Websites

  

legacy project

Platform: macOS

  

legacy project

Platforms, Builds, Tests & Devices

  

legacy project

Platform: Windows

  

legacy project

Python API

  

legacy project

Render & Cycles

  

Priority::Low

  

Priority::Normal

  

Priority::Unbreak Now!

  

Status::Archived

  

Status::Confirmed

  

Status::Duplicate

  

Status::Needs Information from Developers

  

Status::Needs Triage

  

Status::Resolved

  

Type::Bug

  

Type::Report

  

Type::To Do

No Label

legacy module

Development Management

legacy module

Platforms, Builds, Tests & Devices

legacy module

Python API

legacy module

Rendering & Cycles

legacy project

3.3

legacy project

Development Management

legacy project

Documentation

legacy project

Infrastructure: Blender Buildbot

legacy project

Infrastructure: Websites

legacy project

Platform: macOS

legacy project

Platforms, Builds, Tests & Devices

legacy project

Platform: Windows

legacy project

Python API

legacy project

Render & Cycles

Priority::Low

Priority::Normal

Priority::Unbreak Now!

Status::Archived

Status::Confirmed

Status::Duplicate

Status::Needs Information from Developers

Status::Needs Triage

Status::Resolved

Type::Bug

Type::Report

Type::To Do

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

James Monteath

6 Participants

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: archive/blender-buildbot#90418

No description provided.