Initial ID static override proposal, for merging into blender2.8
Needs ReviewPublic

Authored by Bastien Montagne (mont29) on Dec 14 2016, 4:11 PM.

Details

Summary

Reminder: “static override” only allows overriding linked data. Since those do not change, we can evaluate static overrides once (when loading .blend, or re-loading a library). This could also be named 'partially localized data-blocks'. On the other hand, “dynamic override” would allow overriding local data, and would be handled by the depsgraph evaluation to generate final data.

This is not finalized project, but most (all) core concepts, API and code are expected to be final. Lots remain to be done mostly in adding overridable status to many more properties, and… stress-test all kind of complex corner cases!

Right now, only a few properties are overridable, among which object's and bone’s loc/rot/scale, you can test the code by:

  • linking an object;
  • make an override of it (through operator, type 'override' in space-bar search menu);
  • change e.g. location, rotation and/or scale of the override;
  • save file;
  • open file again, and see that changed location/rotation/scale were kept.

You may also note that non-overridable properties remain grayed-out in UI.

Further more, to test & demonstrate differential override, scale is currently stored as a multiplication factor, which means that if you scale up or down your object in library file, it will scale accordingly in its overriden copy (once file is reloaded of course).

Finally, at least basic support of animation override is done and working (i.e. being able to add, or replace, animation of the local overriding ID).

Technical Details

Here is a summary of how static overrides work.

  1. Overrides are handled at RNA level That means we use RNA paths to control which property are overridden (much like animation in fact).
  2. Overrides are 'rooted' at ID level That means that a given ID stores all its overriding, including those in “sub-data” structures.
  3. Each overridden, local ID keeps a pointer to its (linked) reference ID.
  4. Overrides behave exactly like any other local data-blocks, for 99% of Blender code. Only places that need to be aware of their status are:
    1. UI (through RNA, to give feedback to the user).
    2. Load and save code.
    3. Some way to hook the auto-override creation when someone changes an override's property...

RNA changes.

The way RNA properties are compared was heavily rewritten to allow much more complex behavior. Aside from override-specific topics, this brings real comparison (and not only-equality one), and more importantly, the ability to have a custom callback for custom comparison process.

Override Creation

This is merely making a local copy of the linked datablock, and generate an empty override structure for it.

To be addressed : Currently there is only one basic operator handling a single ID, and some RNA API, we most certainly want some more advanced and complex behavior to be possible here (especially thinking about rigged chars, where you need to override armature and mesh objects, and relink the relations between them...).

Auto-generating override operations

User can define/control overriding of properties by hand, but we also generate them automatically, by making some RNA-based comparison between reference and local IDs. RNA comparison code has been extensively enhanced to support that.

To be addressed : We still only fully support basic types, handling of collections remains mostly TODO (that will allow to add/remove modifiers or constraints e.g., in the future).

Override at .blend saving

We save local override 'as is', along with its override operations list. In case some operations are differential (add, sub, multiply), we compute and store the other operand in another datablock (storage one). This simplifies/sanitize write code (and most importantly, avoid having to temporarily modify real ID data during save process!), and also allows to get the 'overridden state' of the ID even when opening in an older Blender unaware of overriding process.

To be addressed : Right now we make full usual copy of the local overriding ID, which is not acceptable for huge datablocks (thinking especially about mesh data here!), we'll need a way to trim useless data (probably using a new copy flag).

Override at .blend loading

Once we have loaded the whole .blend file and all its libraries, we go over whole Main database and apply overrides. To do that without having to recreate, and remap, and rename whole IDs, we edit a copy of linked (reference) ID, and then swap its content with local one.

Known TODOs, Issues, etc.

This is not feature complete! By far… Only basic operations are implemented, several areas remain fuzzy, etc. However, core concepts and code should be mature enough to be merged in blender2.8, where ongoing development can go on. Also, current state is enough and needed for Asset Engine project.

Auto-generation of override operations (i.e. edit to DEG code).

Those changes are only used to quickly give user feedback in UI that a property has been overridden. They to work nice, but looks like that node is not evaluated often anymore (when one edit some of its data)?

This is main opened issue afaik, not critical, but would be nice to solve properly sooner than later.

Another way to do this could be that, every time an ID is tagged for update in DEG, we also tag it for static override update.
Then, on every run of the DEG evaluation, we also start a low-priority, background task that would merely go over whole Main set of IDs, check them for that 'auto-update static override rules' tag, and do the job if needed?

Need some advice from DEG dev here. ;)

IDProperties

Even though they were taken into account to some extent into new code, those are not supported currently. This needs to be addressed, but don’t think it's super-high priority for now.

Non-trivial cases

While some tests were done, and primary issues fixed, current code should not be considered ready for things like 'replacing proxies'. Much more testing-and-fixing work will be needed for that.

Diff Detail

Repository
rB Blender
Branch
arcpatch-D2417
Build Status
Buildable 981
Build 981: arc lint + arc unit
Bastien Montagne (mont29) retitled this revision from to Quick and dirty hack to get overridable RNA prop flag..Dec 14 2016, 4:11 PM
Bastien Montagne (mont29) updated this object.
Bastien Montagne (mont29) retitled this revision from Quick and dirty hack to get overridable RNA prop flag. to First draft, proof-of-concept code of ID static override.Dec 14 2016, 4:22 PM
Bastien Montagne (mont29) updated this object.
  • Merge branch 'master' into override_static
  • Include "static override rules generator" operation into new depsgraph.
  • Add ways to forbid override/auto-override.
  • Merge branch 'master' into id_override_static
  • ID override static: Various minor fixes and tweaks.
  • Add first basic working auto-override.
  • More work and refactor in RNA override part mostly.
  • Add basic code to prevent editing non-overridable properties.
  • Merge branch 'master' into id_override_static
  • Merge branch 'master' into id_override_static
  • More WIP work towards supporting override differentila operations.
  • Merge branch 'master' into id_override_static
  • Working proof-of-concept of differential override operations.
  • Merge branch 'master' into id_override_static

Merge branch 'master' into id_override_static

  • Merge branch 'master' into id_override_static
  • Merge blender2.8 branch.
  • Merge branch 'blender2.8' into id_override_static
  • Merge branch 'blender2.8' into id_override_static
  • Auto-override detection: ignore animated properties.
  • Cleanup while attempting to fix auto-override detection in DEG.
  • Merge branch 'blender2.8' into id_override_static
  • Some cleanup, fix auto-override handling in NULL pointer case.
  • Heavy refactor of new RNA override/comparison code.
  • Merge branch 'blender2.8' into id_override_static
  • Add custom override_apply func for animdata.
  • Rename destination and source parameters in override_apply.
  • Some cleanup/renaming, and move 'self remapping' into new id_swap function.
  • Merge branch 'blender2.8' into id_override_static
  • Hide override timing stuff behind some debug define.
Bastien Montagne (mont29) retitled this revision from First draft, proof-of-concept code of ID static override to Initial ID static override proposal, for merging into blender2.8.Thu, Nov 16, 5:16 PM
Bastien Montagne (mont29) edited the summary of this revision. (Show Details)
Bastien Montagne (mont29) edited projects, added BF Blender: 2.8; removed BF Blender.

Rename main -> bmain (avoid -Wmain)

@Campbell Barton (campbellbarton) mind pushing your main -> bmain change to the branch itself? ;)

First pass review, went over the patch, tested with animating simple pose, works well.

Overall design seems fine, I don't have much to add to your existing notes, think this just needs testing with production files to check it works.

source/blender/blenkernel/intern/library_override.c
460

Using a timer feels a bit weak, wonder if this could be done on a higher level, when UI finishes dragging a button for eg, when transform ends - or before an undo push.

source/blender/makesrna/intern/rna_rna.c
1143

could use PYRNA_STACK_ARRAY for these fixed sizes.

1407

alloca in a loop, best move into static function.

1501–1516

feel like I'm missing something here, why is this check needed - shouldnt the override values that fall outside the range be clamped instead of changing the operation type - even as a default?