Page MenuHome

Library Override: Investigate Resync Improvements.
Closed, ResolvedPublicDESIGN

Description

Having to manually deal with resync of overrides is a growing problem in production context. Further more, not all possible cases are yet covered by current resync code.

We have to find a way to detect all cases where a resync is needed. Then either warn user about it, or do it as a new, automatic step after .blend file has been read?

IMPORTANT: Current idea is to automatically and silently resync when user did not create any explicit overrides in replaced IDs. otherwise, we should rather tag out-of-sync IDs, and inform user about the issue.

Here is a list of cases (changes in library data) that affect ID pointers:

Note: Status reflects current code in override-refactor-tmp branch, not yet in master.

Step I

1) New ID

A new ID is added in the hierarchy/dependency chain of linked data that is already overridden, and that new ID should also be overridden (because e.g. it has dependencies over already overridden IDs). Or an existing linked ID starts using another one that is overridden, and therefor would also need to become overridden.

Detection
  • This can be easily detected using same code we currently use to re-sync/rebuild the overrides.
Handling
  • Current re-sync code is already working fine in that case.

2) ID re-assignment - Already Overridden

An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets re-assigned to a different ID that is also already overridden.

Detection
  • This needs some minimal extra/new work in override properties, to tag operations over ID pointer properties as 'matching' original linked data, or not, when generating override operations. We can then detect a mismatch when applying those rules (on file loading e.g.).
Handling
  • Current re-sync code is already working fine in that case.

3) ID re-assignment - Not Yet Overridden

An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets re-assigned to a different ID that would now need to be overridden.

Detection
  • Same case as 2) above.
Handling
  • Will require a full resync, not always properly handled by current resync code (when the previous assigned ID remains a valid one with an existing override).

4) ID unassignment

An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets unassigned.

Detection
  • Same case as 2) above.
Handling
  • Current re-sync code is already working fine in that case.

5) Removed ID

A currently overridden ID in the hierarchy/dependency chain of linked data that is already overridden, gets deleted (or not used anymore) in the linked data.

Note that this is close to 4) unassignment case above, but we may also need to clear the overrides of the removed ID? Although regular 'not used anymore' cleanup process on save & reload should also do the trick here.

Detection
  • Same case as 2) above.
Handling
  • Current re-sync code is already working fine in that case.

Step II

Report detected 'need resync' cases in UI, and/or perform needed resync automatically.

The later solution would be ideal, however it is not yet clear how heavy that process will be (and hence how much time it would add to loading production .blend files).

  • Try to only re-override needed IDs (in theory partial re-sync of only affected IDs should be doable?).
  • Add post-loading step to re-generate needed overrides.

Event Timeline

Bastien Montagne (mont29) changed the task status from Needs Triage to Confirmed.Dec 15 2020, 6:43 PM
Bastien Montagne (mont29) created this task.
Bastien Montagne (mont29) changed the subtype of this task from "Report" to "Design".Dec 15 2020, 8:18 PM

I've been testing overrides a bit, automatic re-sync at file loading seems an important feature. On some cases you just want to change something and re-render a bunch (100's) of shots without having to manually re-sync them. This can append several times during a production.
One solution (not the best) could be to have python operators that allow someone to implement automatic resync if needed . The resync can take times on some heavy assets (eg sets) which can make opening shots longer, but to me it's better than having to fix manually everything.
I'm really looking forward using overrides seems really cool and a great improvements toward proxies.

Bastien Montagne (mont29) closed this task as Resolved.Mar 11 2021, 2:36 PM
Bastien Montagne (mont29) claimed this task.

Happy to see this task.

Autoresync on load I think is very important for production. When assets get into the hundreds of linked data and overrides, you generally trust linked data would just update and then the set override apply to the updated linked data. If it can't apply on a resync (removed bone, eg), then I should get a warning and get an option to remove de-linked override.

Hey Bastien!

So far Auto-Resync left us with unusable files if the underlying library hierarchy changed and we had to disable it.
E.g. from layout rig to final rig we have massive re-structures in terms of collection hierarchy and Auto-Resync leaves us with partly loaded files and a runtime error:

RuntimeError: Error: Impossible to resync data-block GEO-body and its dependencies, as its linked reference is missing
Error: Impossible to resync data-block GEO-glasses and its dependencies, as its linked reference is missing

I would expect it syncs as much as it can and keeps inactive overrides around for anything that cannot be addressed; throwing an error and stopping is not acceptable.

@Tristan (Plyro) I would not expect resync to be able to properly handle massive changes in hierarchies (and most likely data-blocks themselves) indeed...

Not sure what you mean by 'stopping' though, this error is not a stopping one, it just tells you that this override does not have any matching linked reference anymore, and therefore cannot do anything with it, and just goes to the next one?

I've another question: What will happen with the Exprimental checkbox in the preferences related to this?
We can use older files from Blender 3.0.1 only by switching the Override Auto resynch OFF, else Blender 3.3.2 RC will crash.

Switching the option off is the only way we can use our older files, when continuing the work in the new version.