Code sign Windows build on buildbot #67054

New Issue

Brecht Van Lommel · 2019-07-16T13:50:03+02:00

Brecht Van Lommel commented

2019-07-16 13:50:03 +02:00

We want to sign builds on the buildbot so users can run them without scary warnings, and to catch potential bugs in bundling and code signing earlier.

Obtain a code signing certificate. We are in the process of obtaining an OV code signing certificate from SSL.com. An EV certificate was deemed unnecessary and we expect trust will build up quickly given the amount of users we have (details ).
Configure separate buildbot virtual machine with code signing certificate.
Update buildbot scripts to include code signing with time stamping, remotely on the separate virtual machine .

We want to sign builds on the buildbot so users can run them without scary warnings, and to catch potential bugs in bundling and code signing earlier. - [x] Obtain a code signing certificate. We are in the process of obtaining an OV code signing certificate from SSL.com. An EV certificate was deemed unnecessary and we expect trust will build up quickly given the amount of users we have ([details ](https://blogs.msdn.microsoft.com/ie/2012/08/14/microsoft-smartscreen-extended-validation-ev-code-signing-certificates)). - [x] Configure separate buildbot virtual machine with code signing certificate. - [x] Update buildbot scripts to include code signing with time stamping, remotely on the separate virtual machine .

Brecht Van Lommel commented

2019-07-16 13:50:03 +02:00

Added subscriber: @brecht

Danny McGrath commented

2019-07-16 14:38:35 +02:00

Added subscriber: @dmcgrath

Danny McGrath commented

2019-07-16 14:44:02 +02:00

In general, you don't want to trust your certificates online, in case of hack. But I think if you take reasonable precautions with your signing, you should be okay. For example, have a signing VM that is isolated and only able to mount the filesystem of the buildbot to sign the exe (NFS, etc.).

Might be worth checking what requirements Apple has for key signing though. They may forbid automation, or have specific requirements on how you can keep keys online, or something.

For the non Apple/Windows stuff, might I suggest that you also generate a gpg key to sign a checksum file, to distribute with our Linux files on the mirrors so that people can at least trust the mirrors aren't compromised?

In general, you don't want to trust your certificates online, in case of hack. But I think if you take reasonable precautions with your signing, you should be okay. For example, have a signing VM that is isolated and only able to mount the filesystem of the buildbot to sign the exe (NFS, etc.). Might be worth checking what requirements Apple has for key signing though. They may forbid automation, or have specific requirements on how you can keep keys online, or something. For the non Apple/Windows stuff, might I suggest that you also generate a gpg key to sign a checksum file, to distribute with our Linux files on the mirrors so that people can at least trust the mirrors aren't compromised?

Ray molenkamp commented

2019-07-16 15:08:33 +02:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2019-07-16 15:08:33 +02:00

Windows codesign is rather straight forward, just use signtool to sign the blender.exe binary and the msi installer (official releases only). There is some support in our cmake scripts for codesign, however if we do not end up using this (if the codesign is not on the build bot but remote like @dmcgrath suggests) we should clean that up and remove it.

Gavin Scott commented

2019-07-16 18:40:47 +02:00

Added subscriber: @GavinScott

Gavin Scott commented

2019-07-16 18:40:47 +02:00

The modern Apple notarization process is nice because if you have a bad build (malicious commit discovered after you ship say) you can get Apple to revoke that specific notarization and their Gatekeeper will warn users about that specific version if they try to install it, but it does not affect other builds notarized by the same signing process and developer key.

With Windows codesign, I think you could potentially have to revoke the signing key in the event of a similar serious issue and I assume that would potentially invalidate every build previously signed by that key? Or is there also a way to revoke a specific signature?

The modern Apple notarization process is nice because if you have a bad build (malicious commit discovered after you ship say) you can get Apple to revoke that specific notarization and their Gatekeeper will warn users about that specific version if they try to install it, but it does not affect other builds notarized by the same signing process and developer key. With Windows codesign, I think you could potentially have to revoke the signing key in the event of a similar serious issue and I assume that would potentially invalidate every build previously signed by that key? Or is there also a way to revoke a specific signature?

Ray molenkamp commented

2019-07-16 18:56:58 +02:00

On windows codesign is a trust issue, it validates the code has been signed by a certain entity, at a certain time. nothing more than that. If your certificate got compromised there is an option to revoke it before its expiry date, Anything signed before the expiry date in the revocation list will still properly load (given you timestamped the signature).

However codesign is not meant as a patch mechanism for failing QA practices, it's not meant as way to recall a buggy build and should not be used as such.

Also this is our bug tracker / task list, not quite the appropriate place to strike up a casual conversation on the subject on hand.

On windows codesign is a trust issue, it validates the code has been signed by a certain entity, at a certain time. nothing more than that. If your certificate got compromised there is an option to revoke it before its expiry date, Anything signed before the expiry date in the revocation list will still properly load (given you timestamped the signature). However codesign is not meant as a patch mechanism for failing QA practices, it's not meant as way to recall a buggy build and should not be used as such. Also this is our bug tracker / task list, not quite the appropriate place to strike up a casual conversation on the subject on hand.

Brecht Van Lommel commented

2019-09-24 13:55:51 +02:00

Added subscriber: @jesterking

Nathan Letwory commented

2019-09-25 09:55:12 +02:00

Just as a FYI, I've been discussing this with @LazyDodo yesterday, and work will continue in the coming days.

Nathan Letwory self-assigned this 2019-09-25 09:55:24 +02:00

Brecht Van Lommel commented

2019-09-25 12:30:59 +02:00

Ok, we are buying the certificate to install here at the Blender Institute. For testing the implementation I believe you can generate a test certificate, and then we can use the real one once the signing code is deployed on the buildbot here.

Ray molenkamp commented

2019-09-25 15:12:11 +02:00

if you want to test with self sign, there's a pretty good guide for making your own cert over here

it's mentioned in the article but i cannot stress this enough : once you are done testing remove the cert from the certificate store!!

if you want to test with self sign, there's a pretty good guide for making your own cert [over here ](https://docs.microsoft.com/en-us/windows/msix/package/create-certificate-package-signing?redirectedfrom=MSDN) it's mentioned in the article but i cannot stress this enough : once you are done testing remove the cert from the certificate store!!

Nathan Letwory commented

2019-10-08 21:42:26 +02:00

It took a while to prepare the patches to form a more cohesive whole. No doubt it can be improved upon.

These two have the following:

buildbot_signing.patch

This is the patch for buildbot. This essentially piggy-backs on the existing setup. On the machine that is going to do the signing create a virtualenv from a Python 3.7 install, then install all necessary buildbot bits and pieces, along with pywin32. Get the buildbot repo and apply this patch. Then essentially copy buildbot_signer.tac over buildbot.tac and use this repo as the basis for the master.

Then set up a worker for this as well. Both master and worker will work with localhost.

This machine needs to have scp installed. Further on some machine a SSH git repository should be hosted containing the contents of the attached .tar.gz.

The zipname file in that repository contains the name to the zipfile that will be scp'ied to the worker, extracted, then EXE and DLL files that we built get signed. After signing it all gets zipped up again and uploaded back to where it was originally loaded from. The .tar.gz:

blendersigner.tar.gz

This setup was tested with a self-signed certificate. I basically followed the outdated makecert path. The PFX file I ended up with I but in the build folder that gets created in the worker.

A sidenote: it appears signing blender.exe increases its size quite dramatically. I haven't further investigated that part, because rezipping with ZIP_BZIP2 as compression type gives still a slightly smaller zip than when downloaded originally.

It took a while to prepare the patches to form a more cohesive whole. No doubt it can be improved upon. These two have the following: [buildbot_signing.patch](https://archive.blender.org/developer/F7797249/buildbot_signing.patch) This is the patch for buildbot. This essentially piggy-backs on the existing setup. On the machine that is going to do the signing create a virtualenv from a Python 3.7 install, then install all necessary buildbot bits and pieces, along with pywin32. Get the buildbot repo and apply this patch. Then essentially copy buildbot_signer.tac over buildbot.tac and use this repo as the basis for the master. Then set up a worker for this as well. Both master and worker will work with localhost. This machine needs to have scp installed. Further on some machine a SSH git repository should be hosted containing the contents of the attached .tar.gz. The `zipname` file in that repository contains the name to the zipfile that will be scp'ied to the worker, extracted, then EXE and DLL files that we built get signed. After signing it all gets zipped up again and uploaded back to where it was originally loaded from. The .tar.gz: [blendersigner.tar.gz](https://archive.blender.org/developer/F7797243/blendersigner.tar.gz) This setup was tested with a self-signed certificate. I basically followed the outdated makecert path. The PFX file I ended up with I but in the `build` folder that gets created in the worker. A sidenote: it appears signing `blender.exe` increases its size quite dramatically. I haven't further investigated that part, because rezipping with ZIP_BZIP2 as compression type gives still a slightly smaller zip than when downloaded originally.

Ray molenkamp commented

2019-10-08 21:52:53 +02:00

Executable shouldn't increase dramatically, i ran a quick local test with a self signed cert

unsigned           82.028.032 
signed             82.029.488 (+1456)
signed+timestamped 82.033.608 (+4120)

which seems to match my experience with proper certs.

perhaps you are comparing a debug and release biuld?

Executable shouldn't increase dramatically, i ran a quick local test with a self signed cert ``` unsigned 82.028.032 signed 82.029.488 (+1456) signed+timestamped 82.033.608 (+4120) ``` which seems to match my experience with proper certs. perhaps you are comparing a debug and release biuld?

Nathan Letwory commented

2019-10-08 22:03:25 +02:00

nvm about the size increase of blender.exe - I was looking in the wrong places and comparing the wrong numbers. Size increase of a few kb is what actually happens and is expected.

nvm about the size increase of `blender.exe` - I was looking in the wrong places and comparing the wrong numbers. Size increase of a few kb is what actually happens and is expected.

Brecht Van Lommel commented

2019-10-08 22:34:07 +02:00

Can you explain how these pieces fit together? It's not clear to me how starting a build on the main buildbot would trigger work on the signing buildbot, or how files get passed around.

To be honest I was expecting something simpler. Additions to slave_pack.py to connect to a server that signs files, but still doing the packing itself. Also because files need to be signed before and after packing for DMG bundles and the MSI installer, and that packing code uses the Blender source repository and build system.

Can you explain how these pieces fit together? It's not clear to me how starting a build on the main buildbot would trigger work on the signing buildbot, or how files get passed around. To be honest I was expecting something simpler. Additions to `slave_pack.py` to connect to a server that signs files, but still doing the packing itself. Also because files need to be signed before and after packing for DMG bundles and the MSI installer, and that packing code uses the Blender source repository and build system.

Nathan Letwory commented

2019-10-09 10:35:15 +02:00

After receiving the uploaded build zip blender-buildbot master writes the contained zipfile package name to the file zipname of the blendersigner.git repository and pushes that change.

On the signing machine, which is supposed to be as off-line as possible a the signer buildbot master is running, looking for changes in blendersigner.git together with a scheduler that will fire off a new task running winsigner.py.

This script does three things:

fetch the zipfile pointed to by the latest commit in the zipname file
unpack the zipfile
sign all files we want to sign
repacks the zipfile
copies the zipfile to the designated location

This setup means only the SSH port needs to be open for scp. And outgoing HTTP(S?) connection to the timestamp server.

No further interaction with any kind of network environment should be necessary here.

I did not yet bother thinking about MSI and DMG, but I think the signing machine(s) should eventually do the installer or image creation, separating it from the actual building. For now the focus is on ensuring signing of binaries in the Windows zip is handled.

Further I wanted to use buildbot since it already contains all necessary tools to do polling and scheduling that I don't have to write myself.

After receiving the uploaded build zip blender-buildbot master writes the contained zipfile package name to the file `zipname` of the blendersigner.git repository and pushes that change. On the signing machine, which is supposed to be as off-line as possible a the signer buildbot master is running, looking for changes in blendersigner.git together with a scheduler that will fire off a new task running winsigner.py. This script does three things: * fetch the zipfile pointed to by the latest commit in the zipname file * unpack the zipfile * sign all files we want to sign * repacks the zipfile * copies the zipfile to the designated location This setup means only the SSH port needs to be open for scp. And outgoing HTTP(S?) connection to the timestamp server. No further interaction with any kind of network environment should be necessary here. I did not yet bother thinking about MSI and DMG, but I think the signing machine(s) should eventually do the installer or image creation, separating it from the actual building. For now the focus is on ensuring signing of binaries in the Windows zip is handled. Further I wanted to use buildbot since it already contains all necessary tools to do polling and scheduling that I don't have to write myself.

Nathan Letwory commented

2019-10-09 11:26:07 +02:00

Oh, I see. the patch I uploaded doesn't have all necessary changes, the scheduler bit is missing at least. Let me upload an updated patch in a bit.

Brecht Van Lommel commented

2019-10-09 12:39:42 +02:00

slave_pack.py can copy files to some folder, and the signing VM can poll that folder for any files to be signed. Accessing that shared folder can be done through SSH, or whatever is the most convenient way to share files between two virtual machines running on the same host.

Adding a buildbot and blendersigner.git seems like adding more steps in between, and it's not clear to me which problem they are meant to solve.

I think we should keep the MSI and DMG creating on the same machine as building. Splitting it off makes maintenance more complicated, I don't see which benefits it would have.

`slave_pack.py` can copy files to some folder, and the signing VM can poll that folder for any files to be signed. Accessing that shared folder can be done through SSH, or whatever is the most convenient way to share files between two virtual machines running on the same host. Adding a buildbot and blendersigner.git seems like adding more steps in between, and it's not clear to me which problem they are meant to solve. I think we should keep the MSI and DMG creating on the same machine as building. Splitting it off makes maintenance more complicated, I don't see which benefits it would have.

Nathan Letwory commented

2019-10-09 14:31:22 +02:00

Well, setting up all that polling and messaging between slave_pack.py and the signing machine is what then needs to be set up and tested. With the buildbot we have already something that is tested and working with certainty, without having to write custom code that can fail.

I don't think maintenance would be an issue here, but I'll make a pass on doing it through slave_pack.py instead.

Well, setting up all that polling and messaging between `slave_pack.py` and the signing machine is what then needs to be set up and tested. With the buildbot we have already something that is tested and working with certainty, without having to write custom code that can fail. I don't think maintenance would be an issue here, but I'll make a pass on doing it through `slave_pack.py` instead.

Nathan Letwory commented

2019-10-10 12:36:09 +02:00

New patch at D6036 together with P1137.

In short:

slave_pack.py for Windows will collect blender.exe and BlendThumb.dll and pack to zipfile to a shared location, then wait for files to appear in second location and unpack the signed files from it before continuing.

P1137 is the signing script that runs on the signing machine.

New patch at [D6036](https://archive.blender.org/developer/D6036) together with [P1137](https://archive.blender.org/developer/P1137.txt). In short: slave_pack.py for Windows will collect blender.exe and BlendThumb.dll and pack to zipfile to a shared location, then wait for files to appear in second location and unpack the signed files from it before continuing. [P1137](https://archive.blender.org/developer/P1137.txt) is the signing script that runs on the signing machine.

Nathan Letwory removed their assignment 2019-11-04 22:20:14 +01:00

Sergey Sharybin was assigned by Nathan Letwory

2019-11-04 22:20:14 +01:00

Nathan Letwory commented

2019-11-04 22:20:14 +01:00

I added some steps for testing to D6036. You should be able to start with some testing partly decoupled from the actual process, a separate test buildbot worker is probably a good idea.

When done with testing remember to remove the test certificate completely.

I added some steps for testing to [D6036](https://archive.blender.org/developer/D6036). You should be able to start with some testing partly decoupled from the actual process, a separate test buildbot worker is probably a good idea. When done with testing remember to remove the test certificate completely.

Brecht Van Lommel commented

2019-11-25 17:17:36 +01:00

Changed status from 'Open' to: 'Resolved'

Brecht Van Lommel closed this issue

2019-11-25 17:17:36 +01:00

Brecht Van Lommel commented

2019-11-25 17:17:36 +01:00

Assuming this has been resolved, can reopen or create a new task if there are further things to complete.

Thomas Dinges added this to the 2.81 milestone 2023-02-08 16:48:10 +01:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Code sign Windows build on buildbot #67054