Page MenuHome

Library Override (parent task)
Confirmed, NormalPublicDESIGN

Tokens
"Love" token, awarded by Brandon777."Love" token, awarded by mistajuliax."Burninate" token, awarded by mazigh."Love" token, awarded by duarteframos."Love" token, awarded by bintang."Love" token, awarded by Scaredyfish."Love" token, awarded by amonpaike."Love" token, awarded by bnzs."Yellow Medal" token, awarded by aliasguru.
Authored By

Description

This task is here to explain current design behind lybrary overrides in BF Blender: 2.8, and to keep track of known limitations and TODOs.

Reminder: “library override” only allows overriding linked data. Since those do not change, we can evaluate library overrides once (when loading .blend, or re-loading a library). This could also be named 'partially localized data-blocks'. On the other hand, “dynamic override” would allow overriding local data, and would be handled by the depsgraph evaluation to generate final data.

This is not finalized project, but most (all) core concepts, API and code are expected to be final. Lots remain to be done mostly in adding overridable status to many more properties, and… stress-test all kind of complex corner cases!

Right now, only a few properties are overridable, among which object's and bone’s loc/rot/scale, you can test the code by:

  • linking an object;
  • make an override of it (through operator, type 'override' in space-bar search menu);
  • change e.g. location, rotation and/or scale of the override;
  • save file;
  • open file again, and see that changed location/rotation/scale were kept.

You may also note that non-overridable properties remain grayed-out in UI.

Finally, at least basic support of animation override is done and working (i.e. being able to add, or replace, animation of the local overriding ID).

TODO's for 2.81

Those are known tasks that should be addressed before we go into 2.81 beta phase (i.e. we create the release branch).

TODO's for 2.82:

  • Define much more RNA properties as overridable (this is easy-monkey work mostly).
  • T73154: Library Overrides - UI: Add access to the 'auto override' flag of IDs
  • Add/extend operator to override chains of dependencies (you select an ID to override, and it automatically also overrides all/some of its linked users up to the first local data-block) - mostly for Outliner?
  • Add an operator to fully revert an override to its linked data-block (it's basically ID remapping to linked data + deletion of the override).
  • Add an operator to fully reset an override to its linked data-block state. Warning: likely more complicated than it looks, as this should not break the overridden pointers to other IDs in the same 'overriding family'.
  • Add an operator to convert from existing proxies to new lib overrides.

TODO's for 2.83:

  • Define much more RNA properties as overridable (this is easy-monkey work mostly).
  • T73156: ID search menu issue with linked data-blocks
  • Solve/define a proper way to handle materials overrides?
  • Add a setting similar to auto-keyframe to automatically generate an override when you edit a property? NOTE: this is likely very complex to get right, as by default linked IDs are fully non-editable...

TODO's for later:

Technical Details

Here is a summary of how library overrides work.

  1. Overrides are handled at RNA level That means we use RNA paths to control which property are overridden (much like animation in fact).
  2. Overrides are 'rooted' at ID level That means that a given ID stores all its overriding, including those in “sub-data” structures.
  3. Each overridden, local ID keeps a pointer to its (linked) reference ID.
  4. Overrides behave exactly like any other local data-blocks, for 99% of Blender code. Only places that need to be aware of their status are:
    1. UI (through RNA, to give feedback to the user).
    2. Load and save code.
    3. Some way to hook the auto-override creation when someone changes an override's property (done as part of undo step creation currently).

RNA changes.

The way RNA properties are compared was heavily rewritten to allow much more complex behavior. Aside from override-specific topics, this brings real comparison (and not only-equality one), and more importantly, the ability to have a custom callback for custom comparison process.

Override Creation

This is merely making a local copy of the linked datablock, and generate an empty override structure for it.

  • We need some advanced and complex behavior to be possible here (especially thinking about rigged chars, where you need to override armature and mesh objects, and relink the relations between them…).
  • Hook up override creation into UI (add menu entries, etc.). Easy, to be done once all known issues are fixed.

Auto-generating override operations

User can define/control overriding of properties by hand, but we also generate them automatically, by making some RNA-based comparison between reference and local IDs. RNA comparison code has been extensively enhanced to support that. Auto-generation is currently performed each time an undo step is generated.

  • Support for some collections (modifiers or constraints e.g.).
  • Refined handling of pointers. Pointers can be to other IDs (then we only want to compare pointer itself), or to some sub-data we own (in which case, we want to go further down the path), or to some other we do not own (we only want to compare pointer itself here too).
  • Try to improve performances (T54762) (for big production characters, this can add several hundreds of milliseconds to every undo step... we can live with it, but this could be improved). Partially addressed already, but it should be possible to improve things further still.

Override at .blend saving

We save local override 'as is', along with its override operations list. In case some operations are differential (add, sub, multiply), we compute and store the other operand in another datablock (storage one). This simplifies/sanitize write code (and most importantly, avoid having to temporarily modify real ID data during save process!), and also allows to get the 'overridden state' of the ID even when opening in an older Blender unaware of overriding process.

  • Right now we make full usual copy of the local overriding ID, which is not acceptable for huge datablocks (thinking especially about mesh data here!), we'll need a way to trim useless data (probably using a new copy flag). Not critical though, since Mesh (and other data) IDs are not overridden currently. Note however that removing actual geometry data will make file fully broken in case lib data is lost, or if loaded in an older blender version... Not sure how critical this is in either way.

Override at .blend loading

Once we have loaded the whole .blend file and all its libraries, we go over whole Main database and apply overrides. To do that without having to recreate, and remap, and rename whole IDs, we edit a copy of linked (reference) ID, and then swap its content with local one.

  • Would be nice to investigate improving a bit code efficiency here, if possible. E.g. we could make out-of-main copy, but this would then probably break ID refcounting… Rather low priority anyway, current code does not seem to be a serious bottleneck.

Known TODOs, Issues, etc.

This is not feature complete! By far… Only basic operations are implemented, several areas remain fuzzy, etc. However, core concepts and code should be mature enough to be merged in blender2.8, where ongoing development can go on. Also, current state is enough and needed for Asset Engine project.

Code Working on DNA

Mainly operators, but can be also a few buttons using low-level access to data…

This is a bit of a hairy issue, since overriding data-blocks are essentially local ones, any operator can go playing on their data with any knowledge about what they should be allowed to do!

A first half of the solution is done, which is adding a post-operation check to try to restore data from reference when needed (added as part of the auto-override generation in undo step).

This does not solve everything though, we cannot undo all changes from RNA. So second part of the solution will unfortunately be to add checks to operators. We probably can limit this to a general check over a datablock (at least for a start), to forbid some operators to work on overriding ones completely.

But reaching a full and perfect control on the situation here is likely to be very difficult and long, if possible at all… So question is, to what point can we accept to have some changed data in local overrides that shall not be different from their reference? Knowing that a save & reload of the .blend file (or a manual 'resync' from linked references) will restore things to their correct status?

Override Templates & Locked Override

Those features are partially implemented, but require more work to be usable by users. They are not high-priority currently, and are not being actively worked on.

  • Differential overrides (add/subtract, or multiplication factor, also binary operation for flags, etc.). Mostly implemented, but need some UI/UX design work.
  • Override templates: those are overrides defined in the library file. They give a default override “pattern” when creating a new override in final files (e.g. for a rig, it will allow to restrict which bones are editable in the overrides, and which will remain 'locked' for the end user - at least from UI, PyRNA API will allow to do everything of course).
  • Locked override: by defining a NO-OP (no operation) override operation for a given property, you can effectively prevent any overriding of that property (ensure it will always remain at same value as the linked overridden data-block).

IDProperties

T54652

  • Even though they were taken into account to some extent into new code, those are not supported currently. This needs to be addressed, but don’t think it's super-high priority for now.

Non-trivial cases - Rigged chars

While a lot of tests were done, and a lot of issues fixed, current code should not yet be considered fully ready to replace proxies.

  • Support complex rigs with lots of constraints, drivers and other nifty features.
  • Support production chars like Autumn (a.k.a uber-complex rigs, with sub-rigs etc.).

Non-trivial cases - Materials

  • The way materials are linked to object or obdata make them especially tricky to handle in override case.

Non-trivial cases - Nodes

  • Not yet checked, but fairly sure that the way nodetrees can be real or fake IDs (root trees of materials/compositing/etc.) will also be an issue…

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Brecht Van Lommel (brecht) renamed this task from Static Override to Static Override (parent task).Aug 13 2018, 11:51 AM
Brecht Van Lommel (brecht) triaged this task as Low priority.Oct 2 2018, 12:49 PM
Brecht Van Lommel (brecht) moved this task from Doing to Release Targets on the Code Quest board.

I've been testing this a bit, and it works great! I've noticed shape keys don't seem to be overridable. Is this a limitation, or just something not implemented yet?

@Andrew Charlton (Scaredyfish) Thanks for testing! Shape keys are planned, yes, but not for 2.81, they are likely to give some "interesting" issues. ;)

That's good to know. Thanks for the info!

I didn't want to submit a bug report for this(but let me know if I should), since I know this is work in progress, but it seems that the new Armature constraints don't play along well. I see it is indicated in the UI when a target ID is overriden, by a turqoise color. This doesn't seem to be the case for Armature constraint, so I guess that's meaningful to you.

I'm super excited for this system and hope you can find time to fix the above thing, and make custom properties work for 2.82. I think that's basically all we would need to let this replace linking. And then if you figure out how to add shape key support... we will throw you a party at the studio.

I've been using Library Overrides on a game character Armature to mostly great effect (some crashing, see below).

One issue I've had is when I needed to rename an attachment bone. When the rig updates in the animation scenes the animation still has the old name - how would it know to change I guess? Is there or would there be some smart way to re-target to the new name?

Also when exporting to FBX if the curve properties can't be found the animation export is broken completely.

So after deleting the old properties and saving the scene, the scene crashes on opening. I think it's dues to the library linking as I can move the blend file so it can't see the linked filed and it opens without crashing, but with errors about the missing link file. I've reported it as a bug here https://developer.blender.org/T70695

Lots of other things to add to this system I know, I hope it gets some more love soon as it has the potential to be really great.
One thing I found with the current capabilities is that whilst I can set TRS keys as an override, in the Graph Editor I can't set the value of a key in the properties panel.

@Demeter Dzadik (Mets) please make report(s), it’s waaaayyyyy easier to keep track of issues that way. ;)

@Malcolm Reed (mal_reed) Indeed I would not expect miracles if you rename some bones in your rig, when doing this on a regular local data-block the releated code already has to go in a rather complex 'check every reference to old name and remap it to new name' process, but this is essentially impossible to do on an override… So things like anim curves, drivers etc. will need manual fixing in those cases am afraid.

And I expect many fixing work, and do have some more improvements ideas for library overrides in the future. :)

Hey Bastien, understand on the name change issue.

Maybe an area for someone to write a tool at some point to find anim curves for missing bones and ask what to do with them (hmm maybe it exists somewhere - I'll have a hunt). Fix path by renaming or delete for example. Changes to rigs can happen quite often during game dev - often gets locked down when you finish the project!

Anyway Overrides/Referencing is great and something I used heavily back in my dark days as a Maya user, so I'm backing Library Overrides all the way. Look forward to future additions.

So, after playing with overrides for a trivial character (bouncing ball with a rig), I noticed a big issue that in my view makes the current implementation not very good for complex characters that are not 100% perfect and finalized.

The way overrides are applied for a character is by using a run-once operator that recursively overrides a collection with all its child collections and contained objects, patching references within the override set to refer to overridden objects. The initial result of this is perfectly fine, but it cannot cope with any new references between objects added to the library file after the override setup was already created, as well as new objects. This is because the handling of references between the objects was applied by the operator, and after that the override system itself has no idea that these objects are supposed to be inter-related.

Basically, in practical terms, after an override of a character is created in a scene, it is no longer possible to add new child objects to the character in the library, new constraints/modifiers to the child objects (that reference the armature), new drivers to child objects (referencing the armature, e.g. for new shape keys). All these references will require fixing in every scene that already has an override for the character.

Fixing this requires supporting deep overriding of collections at the override system level, rather than an operator. Something like:

  1. Internally the override system has to support override groups: every override in a group implicitly overrides all references to its original ID in all overrides of the group (including itself).
    • Implicit means that no explicit property override records are created at ID level - being in the group is sufficient. An override rule on the ID itself would only be created if the reference is changed by the user to something different, because per-property overrides prevent propagating reference changes in the library.
    • Any override ID is effectively always in a group with itself. That is, self-references are also patched implicitly. Currently they already seem to be automatically patched, but override rules are also created, which is redundant.
    • When an ID override belongs to a group, it should be possible to override a reference into the same group, in addition to the existing global and one reference only options.
    • It's probably enough to support one group per override instance, and no hierarchy between groups; although maybe some clever use of that kind of things could be found in the future.
  2. Collection overrides get a 'deep override' mode, in which case the override system will automatically create overrides for all objects and sub-collections of the original collection. Sub-collections become deep overrides themselves.
    • A root deep collection override creates and owns an override group. All overrides inside the collection and its children are assigned to the group.
    • The operator proposed to be the replacement to Make Proxy simply creates a root deep collection override, and populates it, without any complex one-time logic.
    • It probably should be impossible to unlink an override from such collections just in case.
Bastien Montagne (mont29) raised the priority of this task from Low to Normal.Jan 8 2020, 7:50 PM

I don't think this is a bug per-se, but it would be very handy for library overrides to support NLA strips. I feel like this is a good use case for library overrides since the main advantage currently is the ability to link the same character multiple times, and the ability to mix a set of existing animations for those characters is a good feature.

Let me know if you want this logged as a bug, or if this is a harder problem that is not a priority.