Library Overrides - Override Group proposal for better character overrides. #72062

New Issue

Alexander Gavrilov · 2019-11-30T10:58:08+01:00

2019-11-30 10:58:08 +01:00

Problem Description

As currently designed and implemented, the override system stores and handles overrides purely per datablock, without any internal support for systems that are composed of multiple inter-related datablocks, like character rigs.

This means that the inter-related nature of the character is only handled by an operator that is run once when the character is initially overriden, and this intent information is then discarded, because there is no way to store it.

This results in issues dealing with the character override going forward:

Redundancy

Since the override system is not aware that the set of datablock overrides comprising the character override forms a consistent inter-related system, the operator has to decompose this intent into many individual property overrides for links between datablocks. This produces a lot of data, which can't be good for performance. It also freezes the state of the links, so changing them in the original object will no longer affect the property values in the overrides.

Failure to handle new links

Another consequence of discarding the inter-related system intent is that new links appearing between datablocks (new drivers, etc) will not be overridden either. This is a huge issue, as it can lead to very confusing 'bugs' in the overridden rigs, and is not as immediately obvious as the case of a completely new object added to the rig.

Proposal: Override Groups

As described above, the root of the problem is that there is no way to represent the intent that a set of datablock overrides in fact forms an integrated inter-related system to the low-level override engine in blender. Solving this requires adding a new low-level concept to the override data.

This proposes introducing a concept of an Override Group, which represents an instance of a character or other inter-related system, and follows these rules:

One group per override

Every datablock override can belong to at most one group, which is set at the time of creation and cannot be changed.

A datablock that doesn't belong to a real group should be considered to be in a group with itself when processing override rules. I.e. a datablock is always in group with itself, whether in a real group or not.

Links within the group are implicitly overridden

Every group represents an implicit mapping to the group member overrides from their originals. This mapping should be applied to all datablock pointer properties of the member overrides before evaluating the rules stored in the datablock itself. The remapped value should also be considered to be the 'original' when recomputing the rules from the values of the datablock, so the implicit group rules aren't redundantly duplicated as per-datablock rules.

Thus, datablock pointer properties can now have three override states: fully original, group override (new), local override. Other properties still have only two states. The local override rules have priority over the group override mapping.

Since every override is always in a group with itself, internal links are always in the group override state, even if the property owner doesn't belong to a real group.

UI: Override into group

When a datablock override belongs to an override group, the UI should allow overriding outgoing references into the same group, in addition to overriding just one reference, or all references, as is currently possible. Arguably, overriding into group should be the default in this case too.

Tie to collections?

To avoid the need to create a new datablock type to represent the groups, it seems to make sense to use one of the members as the representative of the group, and simply store datablock pointers in the override data. It makes most sense to use the root collection of the character for this.

Effects

Introducing override groups that behave as described above will solve following issues:

The override system will be aware of the intent that a set of overrides forms an integrated system, and will be able to automatically handle new links between originals within the group.
Keeping the intra-group link overrides implicit means that the link targets can be changed in the original file, and the overrides will update to follow, unless explicitly changed by the user.
UI support for adding overrides for outgoing links to the group makes it very easy to manually fix new objects added to the character, or extra datablocks that weren't handled by the operator, like in #70319.

Technical

From very cursory understanding of the override code, implementing groups should be as simple as doing an extra scan of overrides to build up the mappings for the groups, and then using those in a pointer replace pass, and in the rule generator. It seems to me that supporting the new override state in the UI might be the more difficult part.

Self references already seem to effectively be handled 'implicitly' by the existing copy datablock code, but the rule generator is not aware of that and creates new rules afterwards. I think that's a design bug, which I adressed in the above rule via self-grouping.

Future Ideas

Group Nesting

It might make sense to support a hierarchy of groups, implemented by including the mapping of the 'parent' into the final mapping of the group, with group members having priority. However it's not clear what the use may be. One idea may be to have a 'root' group which is the final parent of all groups, including the imaginary one-member groups for non-grouped overrides. That group can be used to truly override 'all' references to an original object.

Deep Collection Overrides

To fully automate updating of overrides to changes in a character it is necessary to also automatically handle addition of new objects. This can be done by supporting some kind of 'deep' overrides for collections, which will automatically spawn a group, and then fill it with overrides created for anything contained in the collection and its child collections. Basically, converting the create character override operator into a built-in feature.

However, after addition of groups, handling new objects should become quite easy and foolproof (click on it and choose to 'override into group'), so this part is not that critical compared to the current state of affairs.

# Problem Description As currently designed and implemented, the override system stores and handles overrides purely per datablock, without any internal support for systems that are composed of multiple inter-related datablocks, like character rigs. This means that the inter-related nature of the character is only handled by an operator that is run once when the character is initially overriden, and this intent information is then discarded, because there is no way to store it. This results in issues dealing with the character override going forward: ## Redundancy Since the override system is not aware that the set of datablock overrides comprising the character override forms a consistent inter-related system, the operator has to decompose this intent into many individual property overrides for links between datablocks. This produces a lot of data, which can't be good for performance. It also freezes the state of the links, so changing them in the original object will no longer affect the property values in the overrides. ## Failure to handle new links Another consequence of discarding the inter-related system intent is that new links appearing between datablocks (new drivers, etc) will not be overridden either. This is a huge issue, as it can lead to very confusing 'bugs' in the overridden rigs, and is not as immediately obvious as the case of a completely new object added to the rig. --- # Proposal: Override Groups As described above, the root of the problem is that there is no way to represent the intent that a set of datablock overrides in fact forms an integrated inter-related system to the low-level override engine in blender. Solving this requires adding a new low-level concept to the override data. This proposes introducing a concept of an Override Group, which represents an instance of a character or other inter-related system, and follows these rules: ## One group per override Every datablock override can belong to at most one group, which is set at the time of creation and cannot be changed. A datablock that doesn't belong to a real group should be considered to be in a group with itself when processing override rules. I.e. a datablock is *always in group with itself*, whether in a real group or not. ## Links within the group are implicitly overridden Every group represents an implicit mapping to the group member overrides from their originals. This mapping should be applied to all datablock pointer properties of the member overrides before evaluating the rules stored in the datablock itself. The remapped value should also be considered to be the 'original' when recomputing the rules from the values of the datablock, so the implicit group rules aren't redundantly duplicated as per-datablock rules. Thus, datablock pointer properties can now have three override states: fully original, group override (new), local override. Other properties still have only two states. The local override rules have priority over the group override mapping. Since every override is always in a group with itself, internal links are always in the group override state, even if the property owner doesn't belong to a real group. ## UI: Override into group When a datablock override belongs to an override group, the UI should allow overriding outgoing references into the same group, in addition to overriding just one reference, or all references, as is currently possible. Arguably, overriding into group should be the default in this case too. ## Tie to collections? To avoid the need to create a new datablock type to represent the groups, it seems to make sense to use one of the members as the representative of the group, and simply store datablock pointers in the override data. It makes most sense to use the root collection of the character for this. --- # Effects Introducing override groups that behave as described above will solve following issues: * The override system will be aware of the intent that a set of overrides forms an integrated system, and will be able to automatically handle new links between originals within the group. * Keeping the intra-group link overrides implicit means that the link targets can be changed in the original file, and the overrides will update to follow, unless explicitly changed by the user. * UI support for adding overrides for outgoing links to the group makes it very easy to manually fix new objects added to the character, or extra datablocks that weren't handled by the operator, like in #70319. --- # Technical From very cursory understanding of the override code, implementing groups should be as simple as doing an extra scan of overrides to build up the mappings for the groups, and then using those in a pointer replace pass, and in the rule generator. It seems to me that supporting the new override state in the UI might be the more difficult part. Self references already seem to effectively be handled 'implicitly' by the existing copy datablock code, but the rule generator is not aware of that and creates new rules afterwards. I think that's a design bug, which I adressed in the above rule via self-grouping. --- # Future Ideas ## Group Nesting It might make sense to support a hierarchy of groups, implemented by including the mapping of the 'parent' into the final mapping of the group, with group members having priority. However it's not clear what the use may be. One idea may be to have a 'root' group which is the final parent of all groups, including the imaginary one-member groups for non-grouped overrides. That group can be used to truly override 'all' references to an original object. ## Deep Collection Overrides To fully automate updating of overrides to changes in a character it is necessary to also automatically handle addition of new objects. This can be done by supporting some kind of 'deep' overrides for collections, which will automatically spawn a group, and then fill it with overrides created for anything contained in the collection and its child collections. Basically, converting the create character override operator into a built-in feature. However, after addition of groups, handling new objects should become quite easy and foolproof (click on it and choose to 'override into group'), so this part is not that critical compared to the current state of affairs.

Bastien Montagne was assigned by Alexander Gavrilov

2019-11-30 10:58:08 +01:00

Alexander Gavrilov commented

2019-11-30 10:58:08 +01:00

Added subscribers: @angavrilov, @icappiello, @Mets, @mont29, @jpbouza-4

ronan ducluzeau commented

2019-12-03 11:09:08 +01:00

Added subscriber: @zeauro

ronan ducluzeau commented

2019-12-03 11:09:08 +01:00

Every datablock override can belong to at most one group, which is set at the time of creation and cannot be changed.

Do you mean creation of the rig or creation of the override ?

IMO, it will be simpler to let rigger deal with creation of a datablock group and avoid checks from animator, at moment, he needs to create an override.

If necessary, we could image a popup warning asking user if animator wants to override the group or just specific datablock knowing that drivers will be lost.

Overrides are there to simplify life of animator.
If user have to think of how character is rigged, for each instance he is dealing with, in a complex scene ; that may be exhausting.

>Every datablock override can belong to at most one group, which is set at the time of creation and cannot be changed. Do you mean creation of the rig or creation of the override ? IMO, it will be simpler to let rigger deal with creation of a datablock group and avoid checks from animator, at moment, he needs to create an override. If necessary, we could image a popup warning asking user if animator wants to override the group or just specific datablock knowing that drivers will be lost. Overrides are there to simplify life of animator. If user have to think of how character is rigged, for each instance he is dealing with, in a complex scene ; that may be exhausting.

Hadrien Brissaud commented

2019-12-03 11:14:32 +01:00

Added subscriber: @hadrien

Hadrien Brissaud commented

2019-12-03 11:14:32 +01:00

The technicalities may fly over my head, the importance of being able to deeply modify a linked character while it's being animated in another file (add/remove/rename objects) is not lost on me - not being able to do this would be quite the limitation. However I don't understand why there has to be something else on the UI side ? Shouldn't this all be mostly transparent ? How are these override groups relevant to the user ?

Thanks

The technicalities may fly over my head, the importance of being able to deeply modify a linked character while it's being animated in another file (add/remove/rename objects) is not lost on me - not being able to do this would be quite the limitation. However I don't understand why there has to be something else on the UI side ? Shouldn't this all be mostly transparent ? How are these override groups relevant to the user ? Thanks

Alexander Gavrilov commented

2019-12-04 17:45:09 +01:00

Groups are relevant because they represent instances of the character. The biggest complaint about the old proxies is that you can only have one copy of a character in your scene, unless you literally make multiple copies of the library file. Overrides are supposed to address this by allowing multiple override instances for the same original object, which means they need to know which of the other object overrides form the same instance of the character as a whole.

Overrides are basically a middle ground between linking and appending (or Make Local) - you get a new separate datablock like with appending, but when you save the file only fields you specifically changed are stored, while the rest is reloaded from the original file like with linking. Groups discussed here is a proposed low-level mechanism to efficiently store the fact that you 'appended' a bunch of objects as a unit, so all references within the group should be automatically updated to stay within the 'appended' group. Currently that is done as if you 'appended' all those objects separately, and then manually went in and 'modified' all those references - meaning that the system can't handle any new references that may have appeared when you reload.

Groups are relevant because they represent instances of the character. The biggest complaint about the old proxies is that you can only have one copy of a character in your scene, unless you literally make multiple copies of the library file. Overrides are supposed to address this by allowing multiple override instances for the same original object, which means they need to know which of the other object overrides form the same instance of the character as a whole. Overrides are basically a middle ground between linking and appending (or Make Local) - you get a new separate datablock like with appending, but when you save the file only fields you specifically changed are stored, while the rest is reloaded from the original file like with linking. Groups discussed here is a proposed low-level mechanism to efficiently store the fact that you 'appended' a bunch of objects as a unit, so all references within the group should be automatically updated to stay within the 'appended' group. Currently that is done as if you 'appended' all those objects separately, and then manually went in and 'modified' all those references - meaning that the system can't handle any new references that may have appeared when you reload.

Luc Revardel commented

2019-12-04 21:01:05 +01:00

Added subscriber: @lrevardel

Paul Szajner commented

2019-12-04 23:28:42 +01:00

Added subscriber: @sozap

Brecht Van Lommel commented

2020-01-15 15:07:01 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2020-01-15 15:07:01 +01:00

Untagging design tasks from specific releases, in general these should only be tagged with the relevant modules and listed on the module page.

Bastien Montagne changed title from ~~Static Overrides - Override Group proposal for better character overrides.~~ to Library Overrides - Override Group proposal for better character overrides.

2020-01-16 10:02:19 +01:00

Nico Zevallos commented

2020-08-15 03:12:25 +02:00

Added subscriber: @gnastacast

Damien Picard commented

2021-01-25 11:58:50 +01:00

Added subscriber: @pioverfour

Bastien Montagne removed their assignment 2021-02-16 18:12:10 +01:00

Bastien Montagne commented

2021-11-15 15:11:10 +01:00

Changed status from 'Confirmed' to: 'Archived'

Bastien Montagne closed this issue

2021-11-15 15:11:10 +01:00

Bastien Montagne commented

2021-11-15 15:11:10 +01:00

Archiving, since this proposal implies huge changes in Blender, and there is currently no practical plans about it. The main issues regarding liboverrides described here have also been addressed since this proposal was made.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Library Overrides - Override Group proposal for better character overrides. #72062

Problem Description

Redundancy

Failure to handle new links

Proposal: Override Groups

One group per override

Links within the group are implicitly overridden

UI: Override into group

Tie to collections?

Effects

Technical

Future Ideas

Group Nesting

Deep Collection Overrides