Port operators to use COW (Parent Task) #54810

New Issue

Joshua Leung · 2018-04-25T13:13:16+02:00

Joshua Leung commented

2018-04-25 13:13:16 +02:00

This is the parent task for the copy-on-write operator porting taskforce.

With copy-on-write in place we have clear separation between original data and evaluated state: dependency graph gets original data, goes through all the animation systems, drivers, constraints and modifiers stack and gets evaluated state of that data within the current context.
This means that now operators should read from an evaluated state and write to original data, and to inform dependency graph about changes.
Practical example: if an operator needs to get an object matrix, it should read it from the evaluated version. But if the operator needs to update the object’s location, it should write it to the original object and tag it as updated.

For further information read the dependency graph design .

How this works:

Find an operator you want to port in the linked sub-tasks below.
Submit patches for review using arcanist - tagging @Sergey and @JoshuaLeung as reviewers.
Once the patch is reviewed and committed, the operator in the task will be marked as "done".

Note: Not all operators will actually need fixing (many should be fine already, without needing further changes). In that case, instead of submitting a patch, mark the operator as done directly.

What to do:

The code work is straightforward. But there are things which will help migrating operators:

To go from original object to evaluated, use DEG_get_evaluated_object().

To go from original ID to evaluated use `DEG_get_evaluated_id()`.
 Keep in mind, these functions do hash lookup based on ID, so if multiple lookups of same object is needed try to store result of this lookup.

To go from evaluated to original object use DEG_get_original_object().

 To go from evaluated to original ID use `DEG_get_original_id()`.
 These functions are cheap.

There is also a DEG_get_evaluated_rna_pointer() method that can be used in exceptional cases. Unlike the other functions, it is slow - other methods should be used where possible.
If some operator changes interface-only data (example: scene’s 3d cursor location) apart form notifier also use DEG_id_tag_update(id, DEG_TAG_COPY_ON_WRITE); (e.g., 6a75a1a669).

Other examples:

40199c1d10 - Manipulator: Use evaluated object's matrix to get manipulator position.
eb521b22b2 - Make View Selected to be aware of copy-on-write.
3e26b84397 - Camera manipulator: Make it aware of evaluated version of object.

Testing:

Run Blender normally.

If the operator works as well as it did in 2.7, it works!
If there are problems (e.g. things won't update, invalid results, etc.) the operator (still) needs some more fixes applied.

We also found the following tips useful:

Add the --debug-depsgraph-tag to check if the correct tagging updates are happening
Use the --disable-copy-on-write flag to check if something else in 2.8 (e.g. Message Bus system) may be causing refresh issues
Try checking what happens when using the operators in an animated setup. Some tools only break during animation
Sometimes other object data properties may be broken/requiring updates (e.g. armature settings, constraint influence, or shape keys).

Design Notes:

Context (bContext) contains original non-evaluated data. This means CTX_data_scene will give original scene, not evaluated.

  This is also valid for context in Python.

Main structure is always holding original datablocks.
Naming is very important to keep things clear. Agreed convention is: for evaluated versions of data we add _eval suffix to variables.
Where possible, mark evaluated data variables as const

Sub-tasks for each module's operators:

Stuff you can do now:

Major Changes Required (High Priority - for Blender Studio Migration to 2.8):

View3D - #54829
Transform - #54831
Object - #54817
~~Pose Tools~~ - #54812
Physics - #54818
Sculpt - #54821
Render - #54819
~~Animation Editors/Module~~ - [Required changes here are being tackled by @JoshuaLeung Some things are broken (e.g. recalc after editing keyframes)]

Major Changes Required (Lower Priority):

~~Armature Edit Mode~~ - #54811
IO - #54813
Mask - #54814
Paint - #54820
Sound - #54822
Movie Clip Editor - #54823
Image Editor - #54824
UV Editor - #54830

Likely Only Minor Changes Needed:

User Interface Stuff - #54832
Mesh - ~~#54815~~ (all but blend_from_shape)
Node Editor - #54825
Outliner - #54827
Sequencer - #54828

Postponed (Blocking/Pending Issues need resolving first)

Waiting for Sergey to test/check first

Curve/Surface (CURVE_OT_*)
Font (FONT_OT_*)
Metaballs (META_OT_*)
Lattices (LATTICE_OT_*)

Waiting for Grease Pencil branch merge

Grease Pencil module (editors/gpencil)

COW Task - #55392
GP Branch Merge Task - #54893

Should be fine without needing fixes

Console
View2D
Text Editor

May need fixes at some point (but aren't operators/for taskforce work)

editors/space_view3d/view3d_manipulator_*.c

This is the parent task for the **copy-on-write operator** porting taskforce. With copy-on-write in place we have clear separation between original data and evaluated state: dependency graph gets original data, goes through all the animation systems, drivers, constraints and modifiers stack and gets evaluated state of that data within the current context. This means that now operators should read from an evaluated state and write to original data, and to inform dependency graph about changes. Practical example: if an operator needs to get an object matrix, it should read it from the evaluated version. But if the operator needs to update the object’s location, it should write it to the original object and tag it as updated. For further information read the [dependency graph design ](https://wiki.blender.org/wiki/Source/Depsgraph). # How this works: 1) Find an operator you want to port in the linked sub-tasks below. 2) Submit patches for review using [arcanist ](https://wiki.blender.org/wiki/Tools/CodeReview#Use_Arcanist) - tagging @Sergey and @JoshuaLeung as reviewers. 3) Once the patch is reviewed and committed, the operator in the task will be marked as "done". **Note**: Not all operators will actually need fixing (many should be fine already, without needing further changes). In that case, instead of submitting a patch, mark the operator as done directly. ## What to do: The code work is straightforward. But there are things which will help migrating operators: * To go from original object to evaluated, use `DEG_get_evaluated_object()`. ``` To go from original ID to evaluated use `DEG_get_evaluated_id()`. Keep in mind, these functions do hash lookup based on ID, so if multiple lookups of same object is needed try to store result of this lookup. ``` * To go from evaluated to original object use `DEG_get_original_object()`. ``` To go from evaluated to original ID use `DEG_get_original_id()`. These functions are cheap. ``` * There is also a `DEG_get_evaluated_rna_pointer()` method that can be used in exceptional cases. Unlike the other functions, it is slow - other methods should be used where possible. * If some operator changes interface-only data (example: scene’s 3d cursor location) apart form notifier also use `DEG_id_tag_update(id, DEG_TAG_COPY_ON_WRITE);` (e.g., 6a75a1a669). Other examples: * 40199c1d10 - Manipulator: Use evaluated object's matrix to get manipulator position. * eb521b22b2 - Make View Selected to be aware of copy-on-write. * 3e26b84397 - Camera manipulator: Make it aware of evaluated version of object. ## Testing: Run Blender normally. * If the operator works as well as it did in 2.7, it works! * If there are problems (e.g. things won't update, invalid results, etc.) the operator (still) needs some more fixes applied. We also found the following tips useful: * Add the `--debug-depsgraph-tag` to check if the correct tagging updates are happening * Use the `--disable-copy-on-write` flag to check if something else in 2.8 (e.g. Message Bus system) may be causing refresh issues * Try checking what happens when using the operators in an animated setup. Some tools only break during animation * Sometimes other object data properties may be broken/requiring updates (e.g. armature settings, constraint influence, or shape keys). ## Design Notes: 1) Context (`bContext`) contains original non-evaluated data. This means `CTX_data_scene` will give original scene, not evaluated. ``` This is also valid for context in Python. ``` 2) Main structure is always holding original datablocks. 3) Naming is very important to keep things clear. Agreed convention is: for evaluated versions of data we add `_eval` suffix to variables. 4) Where possible, mark evaluated data variables as `const` # Sub-tasks for each module's operators: ## Stuff you can do now: **Major Changes Required (High Priority - for Blender Studio Migration to 2.8):** - View3D - #54829 - Transform - #54831 - Object - #54817 - ~~Pose Tools~~ - #54812 - Physics - #54818 - Sculpt - #54821 - Render - #54819 - ~~Animation Editors/Module~~ - [**Required changes here are being tackled by @JoshuaLeung** Some things are broken (e.g. recalc after editing keyframes)] **Major Changes Required (Lower Priority):** - ~~Armature Edit Mode~~ - #54811 - IO - #54813 - Mask - #54814 - Paint - #54820 - Sound - #54822 - Movie Clip Editor - #54823 - Image Editor - #54824 - UV Editor - #54830 **Likely Only Minor Changes Needed:** - User Interface Stuff - #54832 - Mesh - ~~#54815~~ (all but blend_from_shape) - Node Editor - #54825 - Outliner - #54827 - Sequencer - #54828 ## Postponed (Blocking/Pending Issues need resolving first) **Waiting for Sergey to test/check first** - Curve/Surface (`CURVE_OT_*`) - Font (`FONT_OT_*`) - Metaballs (`META_OT_*`) - Lattices (`LATTICE_OT_*`) **Waiting for Grease Pencil branch merge** Grease Pencil module (`editors/gpencil`) * COW Task - #55392 * GP Branch Merge Task - #54893 **Should be fine without needing fixes** - Console - View2D - Text Editor **May need fixes at some point (but aren't operators/for taskforce work)** - `editors/space_view3d/view3d_manipulator_*.c`

Joshua Leung self-assigned this 2018-04-25 13:13:16 +02:00

Joshua Leung commented

2018-04-25 13:13:16 +02:00

Added subscribers: @Sergey, @JoshuaLeung

Joshua Leung changed title from ~~Port operators to use COW~~ to Port operators to use COW (Parent Task)

2018-04-25 13:13:58 +02:00

Dalai Felinto commented

2018-04-25 18:49:35 +02:00

Added subscriber: @dfelinto

Andrew Charlton commented

2018-04-30 22:44:04 +02:00

Added subscriber: @Scaredyfish

JT Nelson commented

2018-05-01 17:59:20 +02:00

Added subscriber: @jta

Sybren A. Stüvel commented

2018-05-01 18:02:32 +02:00

Added subscriber: @dr.sybren

Inês Almeida commented

2018-05-03 18:30:16 +02:00

Added subscriber: @brita

Jaed Al Zaber commented

2018-05-09 08:39:02 +02:00

Added subscriber: @codekhor

Inês Almeida commented

2018-05-19 16:29:30 +02:00

I made this quick patch to show if Blender is running with or without COW enabled as I kept getting confused with multiple Blenders while testing.
It has an include for depsgraph, so double check your includes if editing that file.

P691

I made this quick patch to show if Blender is running with or without COW enabled as I kept getting confused with multiple Blenders while testing. It has an include for depsgraph, so double check your includes if editing that file. [P691](https://archive.blender.org/developer/P691.txt)

Tuomo Keskitalo commented

2018-05-20 12:31:35 +02:00

Added subscriber: @TuomoKeskitalo

Ray molenkamp commented

2018-08-11 00:31:36 +02:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2018-08-11 00:31:36 +02:00

fixed the wiki links, @Sergey can you transfer dependency graph design to the new wiki? (had to place a en.blender.org link for now)

fixed the wiki links, @Sergey can you transfer [dependency graph design ](https://en.blender.org/index.php/Dev:2.8/Source/Depsgraph) to the new wiki? (had to place a en.blender.org link for now)

Ray molenkamp commented

2018-08-16 16:53:40 +02:00

Added subscriber: @nBurn