Static Override (parent task)
Open, LowPublic

Description

This task is here to explain current design behind static overrides in BF Blender: 2.8, and to keep track of known limitations and TODOs.

Reminder: “static override” only allows overriding linked data. Since those do not change, we can evaluate static overrides once (when loading .blend, or re-loading a library). This could also be named 'partially localized data-blocks'. On the other hand, “dynamic override” would allow overriding local data, and would be handled by the depsgraph evaluation to generate final data.

This is not finalized project, but most (all) core concepts, API and code are expected to be final. Lots remain to be done mostly in adding overridable status to many more properties, and… stress-test all kind of complex corner cases!

Right now, only a few properties are overridable, among which object's and bone’s loc/rot/scale, you can test the code by:

  • linking an object;
  • make an override of it (through operator, type 'override' in space-bar search menu);
  • change e.g. location, rotation and/or scale of the override;
  • save file;
  • open file again, and see that changed location/rotation/scale were kept.

You may also note that non-overridable properties remain grayed-out in UI.

Finally, at least basic support of animation override is done and working (i.e. being able to add, or replace, animation of the local overriding ID).

Technical Details

Here is a summary of how static overrides work.

  1. Overrides are handled at RNA level That means we use RNA paths to control which property are overridden (much like animation in fact).
  2. Overrides are 'rooted' at ID level That means that a given ID stores all its overriding, including those in “sub-data” structures.
  3. Each overridden, local ID keeps a pointer to its (linked) reference ID.
  4. Overrides behave exactly like any other local data-blocks, for 99% of Blender code. Only places that need to be aware of their status are:
    1. UI (through RNA, to give feedback to the user).
    2. Load and save code.
    3. Some way to hook the auto-override creation when someone changes an override's property (done as part of undo step creation currently).

RNA changes.

The way RNA properties are compared was heavily rewritten to allow much more complex behavior. Aside from override-specific topics, this brings real comparison (and not only-equality one), and more importantly, the ability to have a custom callback for custom comparison process.

Override Creation

This is merely making a local copy of the linked datablock, and generate an empty override structure for it.

  • We need some advanced and complex behavior to be possible here (especially thinking about rigged chars, where you need to override armature and mesh objects, and relink the relations between them…).
  • Hook up override creation into UI (add menu entries, etc.). Easy, to be done once all known issues are fixed.

Auto-generating override operations

User can define/control overriding of properties by hand, but we also generate them automatically, by making some RNA-based comparison between reference and local IDs. RNA comparison code has been extensively enhanced to support that. Auto-generation is currently performed each time an undo step is generated.

  • Support for some collections (modifiers or constraints e.g.).
  • Refined handling of pointers. Pointers can be to other IDs (then we only want to compare pointer itself), or to some sub-data we own (in which case, we want to go further down the path), or to some other we do not own (we only want to compare pointer itself here too).
  • Try to improve performances (T54762) (for big production characters, this can add several hundreds of milliseconds to every undo step... we can live with it, but this could be improved). A temp workaround could be to add an option to not do that processing (and give a way to do it manually)?

Override at .blend saving

We save local override 'as is', along with its override operations list. In case some operations are differential (add, sub, multiply), we compute and store the other operand in another datablock (storage one). This simplifies/sanitize write code (and most importantly, avoid having to temporarily modify real ID data during save process!), and also allows to get the 'overridden state' of the ID even when opening in an older Blender unaware of overriding process.

  • Right now we make full usual copy of the local overriding ID, which is not acceptable for huge datablocks (thinking especially about mesh data here!), we'll need a way to trim useless data (probably using a new copy flag). Not critical though, since Mesh (and other data) IDs are not overridden currently. Unless you need to override their material pointers e.g. :(((

Override at .blend loading

Once we have loaded the whole .blend file and all its libraries, we go over whole Main database and apply overrides. To do that without having to recreate, and remap, and rename whole IDs, we edit a copy of linked (reference) ID, and then swap its content with local one.

  • Would be nice to investigate improving a bit code efficiency here, if possible. E.g. we could make out-of-main copy, but this would then probably break ID refcounting… Rather low priority anyway, current code does not seem to be a serious bottleneck.

Known TODOs, Issues, etc.

This is not feature complete! By far… Only basic operations are implemented, several areas remain fuzzy, etc. However, core concepts and code should be mature enough to be merged in blender2.8, where ongoing development can go on. Also, current state is enough and needed for Asset Engine project.

Code Working on DNA

Mainly operators, but can be also a few buttons using low-level access to data…

This is a bit of a hairy issue, since overriding data-blocks are essentially local ones, any operator can go playing on their data with any knowledge about what they should be allowed to do!

A first half of the solution is done, which is adding a post-operation check to try to restore data from reference when needed (added as part of the auto-override generation in undo step).

This does not solve everything though, we cannot undo all changes from RNA. So second part of the solution will unfortunately be to add checks to operators. We probably can limit this to a general check over a datablock (at least for a start), to forbid some operators to work on overriding ones completely.

But reaching a full and perfect control on the situation here is likely to be very difficult and long, if possible at all… So question is, to what point can we accept to have some changed data in local overrides that shall not be different from their reference? Knowing that a save & reload of the .blend file (or a manual 'resync' from linked references) will restore things to their correct status?

Override Templates & Locked Override

Those features are partially implemented, but require more work to be usable by users. They are not high-priority currently, and are not being actively worked on.

  • Differential overrides (add/subtract, or multiplication factor, also binary operation for flags, etc.). Mostly implemented, but need some UI/UX design work.
  • Override templates: those are overrides defined in the library file. They give a default override “pattern” when creating a new override in final files (e.g. for a rig, it will allow to restrict which bones are editable in the overrides, and which will remain 'locked' for the end user - at least from UI, PyRNA API will allow to do everything of course).
  • Locked override: by defining a NO-OP (no operation) override operation for a given property, you can effectively prevent any overriding of that property (ensure it will always remain at same value as the linked overridden data-block).

IDProperties

T54652

  • Even though they were taken into account to some extent into new code, those are not supported currently. This needs to be addressed, but don’t think it's super-high priority for now.

Non-trivial cases - Rigged chars

While a lot of tests were done, and a lot of issues fixed, current code should not yet be considered fully ready to replace proxies.

  • Support complex rigs with lots of constraints, drivers and other nifty features.
  • Support production chars like Autumn (a.k.a uber-complex rigs, with sub-rigs etc.).

Non-trivial cases - Materials

  • The way materials are linked to object or obdata make them especially tricky to handle in override case.

Non-trivial cases - Nodes

  • Not yet checked, but fairly sure that the way nodetrees can be real or fake IDs (root trees of materials/compositing/etc.) will also be an issue…

Details

Type
Design
Bastien Montagne (mont29) updated the task description. (Show Details)
Bastien Montagne (mont29) changed Type from Bug to Design.
Bastien Montagne (mont29) updated the task description. (Show Details)
Sergey Sharybin (sergey) triaged this task as Normal priority.Dec 7 2017, 2:56 PM
Brecht Van Lommel (brecht) renamed this task from Static Override to Static Override (parent task).Aug 13 2018, 11:51 AM
Brecht Van Lommel (brecht) lowered the priority of this task from Normal to Low.