Incident: bcon3 branching didn't happen in nice time to make builds #84674

New Issue

Sergey Sharybin · 2021-01-13T17:36:01+01:00

Sergey Sharybin commented

2021-01-13 17:36:01 +01:00

Ideally, after branching we will check buildbots are happy to compile the branch. In the past it was some obscure issue with build system which was only visible in a fresh build. The build takes a long time (slow compiler, builder VMs not being super fast), and coudl easily take 50min or even more than an hour.

The 2.92 is the second release when at the end of the day the branch was not done yet, forcing either baby-sitting builds past a day full of weird and wonderful adventures or forcing to postpone branching/announcement for the next day.

Seems that a lot of time was waiting on some tweaks in the interface to happen prior to branching. Why is it so? How can such tweaks be avoided at the branching and happen sooner? Or, maybe, we should allow such changes after the branch? Can they be considered fixes?

Or just relax expectations on builds ready immediately.

Ideally, after branching we will check buildbots are happy to compile the branch. In the past it was some obscure issue with build system which was only visible in a fresh build. The build takes a long time (slow compiler, builder VMs not being super fast), and coudl easily take 50min or even more than an hour. The 2.92 is the second release when at the end of the day the branch was not done yet, forcing either baby-sitting builds past a day full of weird and wonderful adventures or forcing to postpone branching/announcement for the next day. Seems that a lot of time was waiting on some tweaks in the interface to happen prior to branching. Why is it so? How can such tweaks be avoided at the branching and happen sooner? Or, maybe, we should allow such changes after the branch? Can they be considered fixes? Or just relax expectations on builds ready immediately.

Sergey Sharybin commented

2021-01-13 17:36:01 +01:00

Added subscriber: @Sergey

Dalai Felinto commented

2021-01-13 17:55:45 +01:00

Added subscriber: @dfelinto

Dalai Felinto commented

2021-01-13 18:38:41 +01:00

In a hindsight the bcon3 should have been postponed to Thursday I think. And I would go as far as saying that all commits should be in before morning CET otherwise we automatically move bcon3 to the next day.

Sergey Sharybin commented

2021-01-14 09:58:43 +01:00

Would phrase it as "all 'stopper' commits". Don't really see a reason to apply any restrictions on bugfixes, for example. Just don't do them during the branching to avoid merge hell.

Which brings us to the question is what is it what is considered stopper for bcon3. Lets try assemble a checklist:

Do we have splash screen?
...

But now i'm not even sure what do we consider a stopper commit? Is there really a difference between those two scenarios:

Scenario A: Postpone bcon3 for until commit FOO is done, move to bcon3 right after

Scenario B: Move to bcon3, and consider commit FOO as a bugfix

What do you think made the commits we were waiting for yesterday not being considered a bugfix?

Would phrase it as "all 'stopper' commits". Don't really see a reason to apply any restrictions on bugfixes, for example. Just don't do them during the branching to avoid merge hell. Which brings us to the question is what is it what is considered stopper for bcon3. Lets try assemble a checklist: - Do we have splash screen? - ... But now i'm not even sure what do we consider a stopper commit? Is there really a difference between those two scenarios: Scenario A: Postpone bcon3 for until commit FOO is done, move to bcon3 right after Scenario B: Move to bcon3, and consider commit FOO as a bugfix What do you think made the commits we were waiting for yesterday not being considered a bugfix?

Dalai Felinto commented

2021-01-14 15:39:51 +01:00

A small tangent here, but I think it is easy to find a criteria to what a stopper would be. Basically anything that should not be committed during bcon3. So indeed bugfixes are fine. But UI changes, new features are 'stopper' commits. I also agree that the splashscreen file should be ready before the day.

Sergey Sharybin commented

2021-01-15 10:10:15 +01:00

Let's make it 2nd order of tangent for a bit. To me it seems better acceptable if the release team and their cheerleaders/babysitters do not need to stress out or work long day by accepting the UI changes next day. We do know those changes are to be committed anyway, and those can be called fixes. Basically, to me it would be better if bcon3 branching happened earlier that day. and the socket colors and such were committed when ready as a fix.

Now, going back from tangents.

New features needing to be committed at the day of branching is very worrying. If it is something minor, can as well consider holding off on it, allowing release team do their job without extra dependencies. If it is something major, the feature will likely need some UI polish, which is forbidden at bcon3.

How about this: all the stopper changes release/coordination team wants to be in bcon3 should have task on phabricator tagged with the release tag, and marked as high priority.

So then the checklist can look something like:

Do we have splash screen? Yes: go next in the checklist; no: move bcon3 to the next day.
At the noon check high priority tasks on phabricator. If there are any: poke responsible people to tackle them.
At 4pm if there are still high priority tasks, move bcon3 to the next day.

Let's make it 2nd order of tangent for a bit. To me it seems better acceptable if the release team and their cheerleaders/babysitters do not need to stress out or work long day by accepting the UI changes next day. We do know those changes are to be committed anyway, and those can be called fixes. Basically, to me it would be better if bcon3 branching happened earlier that day. and the socket colors and such were committed when ready as a fix. Now, going back from tangents. New features needing to be committed at the day of branching is very worrying. If it is something minor, can as well consider holding off on it, allowing release team do their job without extra dependencies. If it is something major, the feature will likely need some UI polish, which is forbidden at bcon3. How about this: all the stopper changes release/coordination team wants to be in bcon3 should have task on phabricator tagged with the release tag, and marked as high priority. So then the checklist can look something like: - Do we have splash screen? Yes: go next in the checklist; no: move bcon3 to the next day. - At the noon check high priority tasks on phabricator. If there are any: poke responsible people to tackle them. - At 4pm if there are still high priority tasks, move bcon3 to the next day.

Ankit Meel commented

2021-04-15 20:09:11 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Thomas Dinges added this to the 2.92 milestone 2023-02-08 16:13:46 +01:00

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Incident: bcon3 branching didn't happen in nice time to make builds #84674