Library Override: Investigate Resync Improvements. #83811

New Issue

Bastien Montagne · 2020-12-15T18:43:10+01:00

Bastien Montagne commented

2020-12-15 18:43:10 +01:00

Having to manually deal with resync of overrides is a growing problem in production context. Further more, not all possible cases are yet covered by current resync code.

We have to find a way to detect all cases where a resync is needed. Then either warn user about it, or do it as a new, automatic step after .blend file has been read?

IMPORTANT: Current idea is to automatically and silently resync when user did not create any explicit overrides in replaced IDs. otherwise, we should rather tag out-of-sync IDs, and inform user about the issue.

Here is a list of cases (changes in library data) that affect ID pointers:

*Note: Status reflects current code in override-refactor-tmp branch, not yet in master. //

Step I

1) New ID

A new ID is added in the hierarchy/dependency chain of linked data that is already overridden, and that new ID should also be overridden (because e.g. it has dependencies over already overridden IDs). Or an existing linked ID starts using another one that is overridden, and therefor would also need to become overridden.

Detection

This can be easily detected using same code we currently use to re-sync/rebuild the overrides.

Handling

Current re-sync code is already working fine in that case.

2) ID re-assignment - Already Overridden

An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets re-assigned to a different ID that is also already overridden.

Detection

This needs some minimal extra/new work in override properties, to tag operations over ID pointer properties as 'matching' original linked data, or not, when generating override operations. We can then detect a mismatch when applying those rules (on file loading e.g.).

Handling

Current re-sync code is already working fine in that case.

3) ID re-assignment - Not Yet Overridden

An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets re-assigned to a different ID that would now need to be overridden.

Detection

Same case as 2) above.

Handling

Will require a full resync, not always properly handled by current resync code (when the previous assigned ID remains a valid one with an existing override).

4) ID unassignment

An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets unassigned.

Detection

Same case as 2) above.

Handling

Current re-sync code is already working fine in that case.

5) Removed ID

A currently overridden ID in the hierarchy/dependency chain of linked data that is already overridden, gets deleted (or not used anymore) in the linked data.

Note that this is close to 4) unassignment case above, but we may also need to clear the overrides of the removed ID? Although regular 'not used anymore' cleanup process on save & reload should also do the trick here.

Detection

Same case as 2) above.

Handling

Current re-sync code is already working fine in that case.

Step II

Report detected 'need resync' cases in UI, and/or perform needed resync automatically.

The later solution would be ideal, however it is not yet clear how heavy that process will be (and hence how much time it would add to loading production .blend files).

Try to only re-override needed IDs (in theory partial re-sync of only affected IDs should be doable?).
Add post-loading step to re-generate needed overrides.

Having to manually deal with resync of overrides is a growing problem in production context. Further more, not all possible cases are yet covered by current resync code. We have to find a way to detect all cases where a resync is needed. Then either warn user about it, or do it as a new, automatic step **after** .blend file has been read? IMPORTANT: Current idea is to automatically and silently resync *when user did not create any explicit overrides* in replaced IDs. otherwise, we should rather tag out-of-sync IDs, and inform user about the issue. Here is a list of cases (changes in library data) that affect ID pointers: *Note: Status reflects current code in [override-refactor-tmp](https:*developer.blender.org/diffusion/B/history/override-refactor-tmp/) branch, not yet in master. // # Step I ### 1) New ID A new ID is added in the hierarchy/dependency chain of linked data that is already overridden, and that new ID should also be overridden (because e.g. it has dependencies over already overridden IDs). Or an existing linked ID starts using another one that is overridden, and therefor would also need to become overridden. **Detection** - [ ] This can be easily detected using same code we currently use to re-sync/rebuild the overrides. **Handling** - [x] Current re-sync code is already working fine in that case. ### 2) ID re-assignment - Already Overridden An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets re-assigned to a different ID that is also already overridden. **Detection** - [x] This needs some minimal extra/new work in override properties, to tag operations over ID pointer properties as 'matching' original linked data, or not, when generating override operations. We can then detect a mismatch when applying those rules (on file loading e.g.). **Handling** - [x] Current re-sync code is already working fine in that case. ### 3) ID re-assignment - Not Yet Overridden An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets re-assigned to a different ID that would now need to be overridden. **Detection** - [x] Same case as 2) above. **Handling** - [x] Will require a full resync, not always properly handled by current resync code (when the previous assigned ID remains a valid one with an existing override). ### 4) ID unassignment An ID usage in the hierarchy/dependency chain of linked data that is already overridden, gets unassigned. **Detection** - [x] Same case as 2) above. **Handling** - [x] Current re-sync code is already working fine in that case. ### 5) Removed ID A currently overridden ID in the hierarchy/dependency chain of linked data that is already overridden, gets deleted (or not used anymore) in the linked data. *Note that this is close to 4) unassignment case above, but we may also need to clear the overrides of the removed ID? Although regular 'not used anymore' cleanup process on save & reload should also do the trick here.* **Detection** - [x] Same case as 2) above. **Handling** - [x] Current re-sync code is already working fine in that case. # Step II Report detected 'need resync' cases in UI, and/or perform needed resync automatically. The later solution would be ideal, however it is not yet clear how heavy that process will be (and hence how much time it would add to loading production .blend files). - [ ] Try to only re-override needed IDs (in theory partial re-sync of only affected IDs should be doable?). - [ ] Add post-loading step to re-generate needed overrides.

Bastien Montagne commented

2020-12-15 18:43:10 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Bastien Montagne commented

2020-12-15 18:43:10 +01:00

Added subscribers: @mont29, @dfelinto, @eyecandy

Paul Szajner commented

2021-01-09 15:20:07 +01:00

Added subscriber: @sozap

Paul Szajner commented

2021-01-09 15:20:07 +01:00

I've been testing overrides a bit, automatic re-sync at file loading seems an important feature. On some cases you just want to change something and re-render a bunch (100's) of shots without having to manually re-sync them. This can append several times during a production.
One solution (not the best) could be to have python operators that allow someone to implement automatic resync if needed . The resync can take times on some heavy assets (eg sets) which can make opening shots longer, but to me it's better than having to fix manually everything.
I'm really looking forward using overrides seems really cool and a great improvements toward proxies.

I've been testing overrides a bit, automatic re-sync at file loading seems an important feature. On some cases you just want to change something and re-render a bunch (100's) of shots without having to manually re-sync them. This can append several times during a production. One solution (not the best) could be to have python operators that allow someone to implement automatic resync if needed . The resync can take times on some heavy assets (eg sets) which can make opening shots longer, but to me it's better than having to fix manually everything. I'm really looking forward using overrides seems really cool and a great improvements toward proxies.

T.R.O. Nunes commented

2021-01-24 01:53:00 +01:00

Added subscriber: @T.R.O.Nunes

Bastien Montagne commented

2021-03-11 14:36:48 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Bastien Montagne closed this issue

2021-03-11 14:36:48 +01:00

Bastien Montagne self-assigned this 2021-03-11 14:36:48 +01:00

Andres Stephens commented

2021-05-07 23:43:59 +02:00

Added subscriber: @AndresStephens

Andres Stephens commented

2021-05-07 23:43:59 +02:00

Happy to see this task.

Autoresync on load I think is very important for production. When assets get into the hundreds of linked data and overrides, you generally trust linked data would just update and then the set override apply to the updated linked data. If it can't apply on a resync (removed bone, eg), then I should get a warning and get an option to remove de-linked override.

Happy to see this task. Autoresync on load I think is very important for production. When assets get into the hundreds of linked data and overrides, you generally trust linked data would just update and then the set override apply to the updated linked data. If it can't apply on a resync (removed bone, eg), then I should get a warning and get an option to remove de-linked override.

Satish Goda commented

2022-02-15 01:57:09 +01:00

Added subscriber: @satishgoda1

apon commented

2022-07-25 05:04:47 +02:00

Added subscriber: @apon

Tristan commented

2022-10-10 09:25:18 +02:00

Added subscriber: @Plyro

Tristan commented

2022-10-10 09:25:18 +02:00

Hey Bastien!

So far Auto-Resync left us with unusable files if the underlying library hierarchy changed and we had to disable it.
E.g. from layout rig to final rig we have massive re-structures in terms of collection hierarchy and Auto-Resync leaves us with partly loaded files and a runtime error:

RuntimeError: Error: Impossible to resync data-block GEO-body and its dependencies, as its linked reference is missing
Error: Impossible to resync data-block GEO-glasses and its dependencies, as its linked reference is missing

I would expect it syncs as much as it can and keeps inactive overrides around for anything that cannot be addressed; throwing an error and stopping is not acceptable.

Hey Bastien! So far Auto-Resync left us with unusable files if the underlying library hierarchy changed and we had to disable it. E.g. from layout rig to final rig we have massive re-structures in terms of collection hierarchy and Auto-Resync leaves us with partly loaded files and a runtime error: ``` RuntimeError: Error: Impossible to resync data-block GEO-body and its dependencies, as its linked reference is missing Error: Impossible to resync data-block GEO-glasses and its dependencies, as its linked reference is missing ``` I would expect it syncs as much as it can and keeps inactive overrides around for anything that cannot be addressed; throwing an error and stopping is not acceptable.

Bastien Montagne commented

2022-10-10 10:57:11 +02:00

@Plyro I would not expect resync to be able to properly handle massive changes in hierarchies (and most likely data-blocks themselves) indeed...

Not sure what you mean by 'stopping' though, this error is not a stopping one, it just tells you that this override does not have any matching linked reference anymore, and therefore cannot do anything with it, and just goes to the next one?

@Plyro I would not expect resync to be able to properly handle massive changes in hierarchies (and most likely data-blocks themselves) indeed... Not sure what you mean by 'stopping' though, this error is not a stopping one, it just tells you that this override does not have any matching linked reference anymore, and therefore cannot do anything with it, and just goes to the next one?

Christoph Werner commented

2022-11-23 13:39:52 +01:00

Added subscriber: @ChristophWerner

Christoph Werner commented

2022-11-23 13:39:52 +01:00

I've another question: What will happen with the Exprimental checkbox in the preferences related to this?
We can use older files from Blender 3.0.1 only by switching the Override Auto resynch OFF, else Blender 3.3.2 RC will crash.

Switching the option off is the only way we can use our older files, when continuing the work in the new version.

I've another question: What will happen with the Exprimental checkbox in the preferences related to this? We can use older files from Blender 3.0.1 only by switching the Override Auto resynch OFF, else Blender 3.3.2 RC will crash. Switching the option off is the only way we can use our older files, when continuing the work in the new version. ![image.png](https://archive.blender.org/developer/F13963085/image.png)

Thomas Dinges added this to the 2.93 LTS milestone 2023-02-07 18:46:38 +01:00

Richard Antalik referenced this issue

2023-08-04 01:34:01 +02:00

Library overrides #73318

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Library Override: Investigate Resync Improvements. #83811

Step I

1) New ID

2) ID re-assignment - Already Overridden

3) ID re-assignment - Not Yet Overridden

4) ID unassignment

5) Removed ID

Step II