Context in Timers #79690

New Issue

Jacques Lucke · 2020-08-10T14:16:06+02:00

Jacques Lucke commented

2020-08-10 14:16:06 +02:00

Calling operators in [timers ]] is difficult currently, because very few attributes of bpy.context are initialized (which is expected and not a bug). The issue is, even when you [ https:*docs.blender.org/api/blender2.8/bpy.ops.html#overriding-context | override the context , problems remain. This task exists to keep track of these problems and to discuss solutions.

Problem 1: Incomplete operator poll functions

Many operators have poll functions that are supposed to check if the operator can run in a given context. The issue is that many poll functions are incorrect, because they assume that CTX_wm_window(C) and CTX_wm_screen(C) don't return NULL (maybe some more like CTX_wm_workspace(C) are missing here). Those context variables are almost always initialized, but not in timers. This is the main source of crashes when dealing with operators in timers.

To reproduce the issue, run this script:

import bpy

def add_object():

bpy.ops.mesh.primitive_uv_sphere_add()


bpy.app.timers.register(add_object, first_interval=1)

This results in a crash, but it should throw an exception due to incorrect context. The issue is that this operator internally calls ED_object_editmode_exit in make_prim_finish, which calls CTX_data_edit_object which returns NULL, because CTX_data_screen(C) is NULL.

The script can be fixed as follows:

import bpy

def get_base_context():
    '''Create a minimum context override that is necessary to avoid crashes.'''
    wm = bpy.data.window_managers[0]
    window = wm.windows[0]
    context = bpy.context.copy()
    context["window"] = window
    context["screen"] = window.screen
    context["workspace"] = window.workspace
    return context

def add_object():
    context = get_base_context()
    bpy.ops.mesh.primitive_uv_sphere_add(context)

bpy.app.timers.register(add_object, first_interval=1)

Solution 1

Try to find all the incorrect poll functions and fix them individually. This will definitely solve the issue, but might require a lot of effort (maybe it's also easy, I haven't tried yet).

Solution 2

Define some "minimal context" that is required to run operators. For example, we could say that in order to run an operator the context must have a window and a screen, otherwise the poll function of the operatoris not even checked. It's fairly easy to implement such a check in WM_operator_poll. The downside is that there might be some operators that can run without any context. To mitigate this downside, we could add a flag that can be added to operators that don't require any context.

Problem 2: Context Override

We do support context overrides when calling operators from Python already. However, there is one severe limitation when calling a operator that is defined in Python that itself calls other operators. The issues is that the context override only applies to the immediately called operator, but not to the indirectly called operators.

To see this issue in action, run the following script and check the terminal output:

import bpy

def get_base_context():

'''Create a minimum context override that is necessary to avoid crashes.'''
wm = bpy.data.window_managers[0]
window = wm.windows[0]
context = bpy.context.copy()
context["window"] = window
context["screen"] = window.screen
context["workspace"] = window.workspace
return context

def custom_operator():

context = get_base_context()
bpy.ops.object.my_operator1(context)

class MyOperator1(bpy.types.Operator):

bl_idname = "object.my_operator1"
bl_label = "My Operator 1"

def execute(self, context):
print(1, context.screen)
bpy.ops.object.my_operator2()
return {"FINISHED"}

class MyOperator2(bpy.types.Operator):

bl_idname = "object.my_operator2"
bl_label = "My Operator 2"

def execute(self, context):
print(2, context.screen)
return {"FINISHED"}

bpy.utils.register_class(MyOperator1)
bpy.utils.register_class(MyOperator2)

bpy.app.timers.register(custom_operator, first_interval=1)

In my case it prints:

1 <bpy_struct, Screen("Scripting")>
2 None

Solution 1

Apply this patch:

diff --git a/source/blender/python/intern/bpy_operator.c b/source/blender/python/intern/bpy_operator.c
index 4b2b5f129a7..b599242219c 100644
    - a/source/blender/python/intern/bpy_operator.c
+++ b/source/blender/python/intern/bpy_operator.c
@@ -227,7 +227,9 @@ static PyObject *pyop_call(PyObject *UNUSED(self), PyObject *args)

context_dict_back = CTX_py_dict_get(C);

 
- CTX_py_dict_set(C, (void *)context_dict);
+  if (context_dict != NULL) {
+    CTX_py_dict_set(C, (void *)context_dict);
+  }

Py_XINCREF(context_dict); /* so we done loose it */

if (WM_operator_poll_context((bContext *)C, ot, context) == false) {

With this change the python context override is not reset when an operator is called without an explicit context override. This fixes the example above, and might even work in general. One issue is when someone overwrites the context without calling context.copy() first, because then the underlying context might suddenly change in unexpected ways.

Solution 2

Refactor bContext to properly support "context override stacking". The pattern that parts of the context are temporarily overridden is quite common in Blender. We could change bContext so that it supports multiple layers of overrides natively. The API could look like so:

CTX_override_push("area", area);
/* ... */
CTX_override_pop();

An "override" could be a single overridden property, or a python dictionary or something else, that we do not know yet.

While this refactor is a larger project, it seems doable to me, because bContext is fairly well encapsulated.

With this system, we could also support doing context overrides with a https://book.pythontips.com/en/latest/context_managers.html in python, instead of copying the entire context and passing it to the operator call.

Calling operators in [timers ]] is difficult currently, because very few attributes of `bpy.context` are initialized (which is expected and not a bug). The issue is, even when you [[ https:*docs.blender.org/api/blender2.8/bpy.ops.html#overriding-context | override the context ](https:*docs.blender.org/api/blender2.8/bpy.app.timers.html), problems remain. This task exists to keep track of these problems and to discuss solutions. Problem 1: Incomplete operator `poll` functions **** Many operators have `poll` functions that are supposed to check if the operator can run in a given context. The issue is that many poll functions are incorrect, because they assume that `CTX_wm_window(C)` and `CTX_wm_screen(C)` don't return `NULL` (maybe some more like `CTX_wm_workspace(C)` are missing here). Those context variables are almost always initialized, but not in timers. This is the main source of crashes when dealing with operators in timers. To reproduce the issue, run this script: ```lang=python import bpy def add_object(): ``` bpy.ops.mesh.primitive_uv_sphere_add() ``` bpy.app.timers.register(add_object, first_interval=1) ``` This results in a crash, but it should throw an exception due to incorrect context. The issue is that this operator internally calls `ED_object_editmode_exit` in `make_prim_finish`, which calls `CTX_data_edit_object` which returns `NULL`, because `CTX_data_screen(C)` is `NULL`. The script can be fixed as follows: ```lang=python import bpy def get_base_context(): '''Create a minimum context override that is necessary to avoid crashes.''' wm = bpy.data.window_managers[0] window = wm.windows[0] context = bpy.context.copy() context["window"] = window context["screen"] = window.screen context["workspace"] = window.workspace return context def add_object(): context = get_base_context() bpy.ops.mesh.primitive_uv_sphere_add(context) bpy.app.timers.register(add_object, first_interval=1) ``` Solution 1 --------- Try to find all the incorrect `poll` functions and fix them individually. This will definitely solve the issue, but might require a lot of effort (maybe it's also easy, I haven't tried yet). Solution 2 ---------- Define some "minimal context" that is required to run operators. For example, we could say that in order to run an operator the context must have a window and a screen, otherwise the `poll` function of the operatoris not even checked. It's fairly easy to implement such a check in `WM_operator_poll`. The downside is that there might be some operators that can run without any context. To mitigate this downside, we could add a flag that can be added to operators that don't require any context. Problem 2: Context Override **** We do support context overrides when calling operators from Python already. However, there is one severe limitation when calling a operator that is defined in Python that itself calls other operators. The issues is that the context override only applies to the immediately called operator, but not to the indirectly called operators. To see this issue in action, run the following script and check the terminal output: ```lang=python import bpy def get_base_context(): ``` '''Create a minimum context override that is necessary to avoid crashes.''' wm = bpy.data.window_managers[0] window = wm.windows[0] context = bpy.context.copy() context["window"] = window context["screen"] = window.screen context["workspace"] = window.workspace return context ``` def custom_operator(): ``` context = get_base_context() bpy.ops.object.my_operator1(context) ``` class MyOperator1(bpy.types.Operator): ``` bl_idname = "object.my_operator1" bl_label = "My Operator 1" def execute(self, context): print(1, context.screen) bpy.ops.object.my_operator2() return {"FINISHED"} ``` class MyOperator2(bpy.types.Operator): ``` bl_idname = "object.my_operator2" bl_label = "My Operator 2" def execute(self, context): print(2, context.screen) return {"FINISHED"} ``` bpy.utils.register_class(MyOperator1) bpy.utils.register_class(MyOperator2) ``` ``` bpy.app.timers.register(custom_operator, first_interval=1) ``` In my case it prints: ``` 1 <bpy_struct, Screen("Scripting")> 2 None ``` Solution 1 --------- Apply this patch: ``` diff --git a/source/blender/python/intern/bpy_operator.c b/source/blender/python/intern/bpy_operator.c index 4b2b5f129a7..b599242219c 100644 - a/source/blender/python/intern/bpy_operator.c +++ b/source/blender/python/intern/bpy_operator.c @@ -227,7 +227,9 @@ static PyObject *pyop_call(PyObject *UNUSED(self), PyObject *args) ``` context_dict_back = CTX_py_dict_get(C); ``` - CTX_py_dict_set(C, (void *)context_dict); + if (context_dict != NULL) { + CTX_py_dict_set(C, (void *)context_dict); + } ``` Py_XINCREF(context_dict); /* so we done loose it */ ``` ``` if (WM_operator_poll_context((bContext *)C, ot, context) == false) { ``` ``` With this change the python context override is not reset when an operator is called without an explicit context override. This fixes the example above, and might even work in general. One issue is when someone overwrites the context without calling `context.copy()` first, because then the underlying context might suddenly change in unexpected ways. Solution 2 ------------ Refactor `bContext` to properly support "context override stacking". The pattern that parts of the context are temporarily overridden is quite common in Blender. We could change `bContext` so that it supports multiple layers of overrides natively. The API could look like so: ```lang=c CTX_override_push("area", area); /* ... */ CTX_override_pop(); ``` An "override" could be a single overridden property, or a python dictionary or something else, that we do not know yet. While this refactor is a larger project, it seems doable to me, because `bContext` is fairly well encapsulated. With this system, we could also support doing context overrides with a [[ https://book.pythontips.com/en/latest/context_managers.html | context manager ]] in python, instead of copying the entire context and passing it to the operator call.

Jacques Lucke commented

2020-08-10 14:16:06 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Jacques Lucke commented

2020-08-10 14:16:06 +02:00

Added subscriber: @JacquesLucke

Jacques Lucke commented

2020-08-10 14:17:22 +02:00

Added subscribers: @brecht, @ideasman42

Brecht Van Lommel commented

2020-08-10 17:12:06 +02:00

Incomplete operator poll functions: perhaps we can just set the first window as context? Background mode is the same really. It has a window in its context which I guess makes most operators work there.
Content override: I think we should support context override stacking. I think it would be enough to define a function that can merge two contexts to create a new one? I don't think push/pop functions are needed.

* Incomplete operator poll functions: perhaps we can just set the first window as context? Background mode is the same really. It has a window in its context which I guess makes most operators work there. * Content override: I think we should support context override stacking. I think it would be enough to define a function that can merge two contexts to create a new one? I don't think push/pop functions are needed.

Jacques Lucke commented

2020-08-10 18:42:22 +02:00

In #79690#993999, @brecht wrote:

Incomplete operator poll functions: perhaps we can just set the first window as context? Background mode is the same really. It has a window in its context which I guess makes most operators work there.

That sounds like it could work. Didn't know if this was acceptable. Is there always at least one window? Will try this tomorrow.

In #79690#993999, @brecht wrote:

Content override: I think we should support context override stacking. I think it would be enough to define a function that can merge two contexts to create a new one? I don't think push/pop functions are needed.

That could work if you mean it the way I interpret it. Did you mean to make individual bContext instances more immutable? Can you show an example how one would override the area of a context with that approach (just some pseudocode)?

> In #79690#993999, @brecht wrote: > * Incomplete operator poll functions: perhaps we can just set the first window as context? Background mode is the same really. It has a window in its context which I guess makes most operators work there. That sounds like it could work. Didn't know if this was acceptable. Is there always at least one window? Will try this tomorrow. > In #79690#993999, @brecht wrote: > * Content override: I think we should support context override stacking. I think it would be enough to define a function that can merge two contexts to create a new one? I don't think push/pop functions are needed. That could work if you mean it the way I interpret it. Did you mean to make individual `bContext` instances more immutable? Can you show an example how one would override the area of a context with that approach (just some pseudocode)?

Brecht Van Lommel commented

2020-08-10 18:54:21 +02:00

In #79690#994067, @JacquesLucke wrote:
That could work if you mean it the way I interpret it. Did you mean to make individual bContext instances more immutable? Can you show an example how one would override the area of a context with that approach (just some pseudocode)?

I was thinking that, but actually I'm not sure it will work. It means bpy.context can pointer to a different context, which might break some Python code.

Still, we could copy the entire context for backup, overwrite the context with the merged result, then restore the backup. No need to do push/pop on the level of individual context members, performance will not be a problem.

> In #79690#994067, @JacquesLucke wrote: > That could work if you mean it the way I interpret it. Did you mean to make individual `bContext` instances more immutable? Can you show an example how one would override the area of a context with that approach (just some pseudocode)? I was thinking that, but actually I'm not sure it will work. It means `bpy.context` can pointer to a different context, which might break some Python code. Still, we could copy the entire context for backup, overwrite the context with the merged result, then restore the backup. No need to do push/pop on the level of individual context members, performance will not be a problem.

RedMser commented

2020-08-12 18:06:32 +02:00

Added subscriber: @RedMser

Matthias Thurau commented

2020-08-15 21:07:54 +02:00

Added subscriber: @Zweistein

Matthias Thurau commented

2020-08-15 21:07:54 +02:00

I ran into this exact problem today. I have an expensive python operator. Therefore I dont run it on every mesh change, but I wait till the last mesh change of the users is >2s ago. something like an lazy update.

My timer starts the operator, but it gives the error that the context is not correct for editmode toggle.

I ran into this exact problem today. I have an expensive python operator. Therefore I dont run it on every mesh change, but I wait till the last mesh change of the users is >2s ago. something like an lazy update. My timer starts the operator, but it gives the error that the context is not correct for editmode toggle.

Oliver Weissbarth commented

2020-09-11 09:56:35 +02:00

Added subscriber: @oweissbarth

Campbell Barton commented

2020-09-26 05:49:11 +02:00

Regarding context override, suggest to stack the override dictionaries, so an operator that's called from Python will merge it's dictionary with the existing override dictionary (creating a new override dictionary).

There are some things to consider with data becoming invalid though, currently when context data is set (CTX_data_scene_set, CTX_wm_window_set... etc) it creates an editable copy of the override dictionary and removes those context members. So the context-override doesn't prevent the operator modifying the context as part of it's own logic. Ideally this would act on all context overrides up the call stack.

Regarding context override, suggest to stack the override dictionaries, so an operator that's called from Python will merge it's dictionary with the existing override dictionary (creating a new override dictionary). There are some things to consider with data becoming invalid though, currently when context data is set (`CTX_data_scene_set`, `CTX_wm_window_set`... etc) it creates an editable copy of the override dictionary and removes those context members. So the context-override doesn't prevent the operator modifying the context as part of it's own logic. Ideally this would act on all context overrides up the call stack.

Pauan commented

2020-10-07 13:43:28 +02:00

Added subscriber: @pauanyu_blender

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Context in Timers #79690

Solution 1

Solution 2

Solution 1

Solution 2