Page MenuHome

Code sign Windows build on buildbot
Open, NormalPublic

Description

We want to sign builds on the buildbot so users can run them without scary warnings, and to catch potential bugs in bundling and code signing earlier.

  • Obtain a code signing certificate. We are in the process of obtaining an OV code signing certificate from SSL.com. An EV certificate was deemed unnecessary and we expect trust will build up quickly given the amount of users we have (details).
  • Configure separate buildbot virtual machine with code signing certificate.
  • Update buildbot scripts to include code signing with time stamping, remotely on the separate virtual machine .

Details

Type
To Do

Event Timeline

Brecht Van Lommel (brecht) lowered the priority of this task from Needs Triage by Developer to Normal.Jul 16 2019, 1:50 PM
Brecht Van Lommel (brecht) created this task.

In general, you don't want to trust your certificates online, in case of hack. But I think if you take reasonable precautions with your signing, you should be okay. For example, have a signing VM that is isolated and only able to mount the filesystem of the buildbot to sign the exe (NFS, etc.).

Might be worth checking what requirements Apple has for key signing though. They may forbid automation, or have specific requirements on how you can keep keys online, or something.

For the non Apple/Windows stuff, might I suggest that you also generate a gpg key to sign a checksum file, to distribute with our Linux files on the mirrors so that people can at least trust the mirrors aren't compromised?

Windows codesign is rather straight forward, just use signtool to sign the blender.exe binary and the msi installer (official releases only). There is some support in our cmake scripts for codesign, however if we do not end up using this (if the codesign is not on the build bot but remote like @Dan McGrath (dmcgrath) suggests) we should clean that up and remove it.

The modern Apple notarization process is nice because if you have a bad build (malicious commit discovered after you ship say) you can get Apple to revoke that specific notarization and their Gatekeeper will warn users about that specific version if they try to install it, but it does not affect other builds notarized by the same signing process and developer key.

With Windows codesign, I think you could potentially have to revoke the signing key in the event of a similar serious issue and I assume that would potentially invalidate every build previously signed by that key? Or is there also a way to revoke a specific signature?

LazyDodo (LazyDodo) added a comment.EditedJul 16 2019, 6:56 PM

On windows codesign is a trust issue, it validates the code has been signed by a certain entity, at a certain time. nothing more than that. If your certificate got compromised there is an option to revoke it before its expiry date, Anything signed before the expiry date in the revocation list will still properly load (given you timestamped the signature).

However codesign is not meant as a patch mechanism for failing QA practices, it's not meant as way to recall a buggy build and should not be used as such.

Also this is our bug tracker / task list, not quite the appropriate place to strike up a casual conversation on the subject on hand.

Just as a FYI, I've been discussing this with @LazyDodo (LazyDodo) yesterday, and work will continue in the coming days.

Ok, we are buying the certificate to install here at the Blender Institute. For testing the implementation I believe you can generate a test certificate, and then we can use the real one once the signing code is deployed on the buildbot here.

if you want to test with self sign, there's a pretty good guide for making your own cert over here

it's mentioned in the article but i cannot stress this enough : once you are done testing remove the cert from the certificate store!!

It took a while to prepare the patches to form a more cohesive whole. No doubt it can be improved upon.

These two have the following:

This is the patch for buildbot. This essentially piggy-backs on the existing setup. On the machine that is going to do the signing create a virtualenv from a Python 3.7 install, then install all necessary buildbot bits and pieces, along with pywin32. Get the buildbot repo and apply this patch. Then essentially copy buildbot_signer.tac over buildbot.tac and use this repo as the basis for the master.

Then set up a worker for this as well. Both master and worker will work with localhost.

This machine needs to have scp installed. Further on some machine a SSH git repository should be hosted containing the contents of the attached .tar.gz.

The zipname file in that repository contains the name to the zipfile that will be scp'ied to the worker, extracted, then EXE and DLL files that we built get signed. After signing it all gets zipped up again and uploaded back to where it was originally loaded from. The .tar.gz:

This setup was tested with a self-signed certificate. I basically followed the outdated makecert path. The PFX file I ended up with I but in the build folder that gets created in the worker.

A sidenote: it appears signing blender.exe increases its size quite dramatically. I haven't further investigated that part, because rezipping with ZIP_BZIP2 as compression type gives still a slightly smaller zip than when downloaded originally.

Executable shouldn't increase dramatically, i ran a quick local test with a self signed cert

unsigned           82.028.032 
signed             82.029.488 (+1456)
signed+timestamped 82.033.608 (+4120)

which seems to match my experience with proper certs.

perhaps you are comparing a debug and release biuld?

nvm about the size increase of blender.exe - I was looking in the wrong places and comparing the wrong numbers. Size increase of a few kb is what actually happens and is expected.

Can you explain how these pieces fit together? It's not clear to me how starting a build on the main buildbot would trigger work on the signing buildbot, or how files get passed around.

To be honest I was expecting something simpler. Additions to slave_pack.py to connect to a server that signs files, but still doing the packing itself. Also because files need to be signed before and after packing for DMG bundles and the MSI installer, and that packing code uses the Blender source repository and build system.

After receiving the uploaded build zip blender-buildbot master writes the contained zipfile package name to the file zipname of the blendersigner.git repository and pushes that change.

On the signing machine, which is supposed to be as off-line as possible a the signer buildbot master is running, looking for changes in blendersigner.git together with a scheduler that will fire off a new task running winsigner.py.

This script does three things:

  • fetch the zipfile pointed to by the latest commit in the zipname file
  • unpack the zipfile
  • sign all files we want to sign
  • repacks the zipfile
  • copies the zipfile to the designated location

This setup means only the SSH port needs to be open for scp. And outgoing HTTP(S?) connection to the timestamp server.

No further interaction with any kind of network environment should be necessary here.

I did not yet bother thinking about MSI and DMG, but I think the signing machine(s) should eventually do the installer or image creation, separating it from the actual building. For now the focus is on ensuring signing of binaries in the Windows zip is handled.

Further I wanted to use buildbot since it already contains all necessary tools to do polling and scheduling that I don't have to write myself.

Oh, I see. the patch I uploaded doesn't have all necessary changes, the scheduler bit is missing at least. Let me upload an updated patch in a bit.

slave_pack.py can copy files to some folder, and the signing VM can poll that folder for any files to be signed. Accessing that shared folder can be done through SSH, or whatever is the most convenient way to share files between two virtual machines running on the same host.

Adding a buildbot and blendersigner.git seems like adding more steps in between, and it's not clear to me which problem they are meant to solve.

I think we should keep the MSI and DMG creating on the same machine as building. Splitting it off makes maintenance more complicated, I don't see which benefits it would have.

Well, setting up all that polling and messaging between slave_pack.py and the signing machine is what then needs to be set up and tested. With the buildbot we have already something that is tested and working with certainty, without having to write custom code that can fail.

I don't think maintenance would be an issue here, but I'll make a pass on doing it through slave_pack.py instead.

New patch at D6036 together with P1137.

In short:

slave_pack.py for Windows will collect blender.exe and BlendThumb.dll and pack to zipfile to a shared location, then wait for files to appear in second location and unpack the signed files from it before continuing.

P1137 is the signing script that runs on the signing machine.