RenderEngine - use of python threads with sleep(0)/wait(0) crash #50308

New Issue

Kerim Borchaev · 2016-12-25T20:47:56+01:00

Kerim Borchaev commented

2016-12-25 20:47:56 +01:00

System Information
Windows 7/x64, W8100

Blender Version
Broken: 2.78a, e8299c8
Worked: 2.77a

Usage of time.sleep(0)/Event.wait(0) with a RenderEngine leads to crash

Load the test addon from addon.7z . Enable 'Test' RenderEngine. Repeat pressing Shift-Z in a View3D twice(to enable RENDERED mode and to turn if off). Blender crashes(eventually, for me it's very often immediately after second Shift-Z).
Cause of this - is use of time.sleep(0) within the thread proc. Changing '0' to even '0.000001' seems to prevent crash.
Crash message varies, could be

Fatal Python error: PyEval_SaveThread: NULL tstate

or

Error: EXCEPTION_ACCESS_VIOLATION"

**System Information** Windows 7/x64, W8100 **Blender Version** Broken: 2.78a, e8299c8 Worked: 2.77a Usage of time.sleep(0)/Event.wait(0) with a RenderEngine leads to crash Load the test addon from [addon.7z](https://archive.blender.org/developer/F422886/addon.7z) . Enable 'Test' RenderEngine. Repeat pressing Shift-Z in a View3D twice(to enable RENDERED mode and to turn if off). Blender crashes(eventually, for me it's very often immediately after second Shift-Z). Cause of this - is use of time.sleep(0) within the thread proc. Changing '0' to even '0.000001' seems to prevent crash. Crash message varies, could be > Fatal Python error: PyEval_SaveThread: NULL tstate or > Error: EXCEPTION_ACCESS_VIOLATION"

Kerim Borchaev commented

2016-12-25 20:47:56 +01:00

Changed status to: 'Open'

Kerim Borchaev commented

2016-12-25 20:47:56 +01:00

Added subscriber: @KerimBorchaev

Bastien Montagne commented

2016-12-27 15:43:12 +01:00

Added subscriber: @mont29

Bastien Montagne commented

2016-12-27 15:43:12 +01:00

Not sure to understand the point of using sleep(0)… In any case, this totally looks like python issue to me, Blender does absolutely not interact with low-level modules like time or thread.

Not sure what to do anyway, unless you can show us that Blender is responsible for that. Did you try doing that in exact same python directly (without using Blender's interpreter)?

Not sure to understand the point of using `sleep(0)`… In any case, this totally looks like python issue to me, Blender does absolutely not interact with low-level modules like `time` or `thread`. Not sure what to do anyway, unless you can show us that Blender is responsible for that. Did you try doing that in exact same python directly (without using Blender's interpreter)?

Kerim Borchaev commented

2016-12-27 17:36:47 +01:00

Not sure what to do anyway, unless you can show us that Blender is responsible for that.

It is crashing immediately after RENDERED mode is turned off. No addon code is executed on this event - therefore it's something in Blender?

Did you try doing that in exact same python directly (without using Blender's interpreter)?

Sure, threads with sleep(0) works fine in pure python.

Not sure to understand the point of using sleep(0)

Doesn't matter for me, personally, as anything non-zero passed to sleep seemingly fixes the issue.
But there's something wrong, this code should not crash.

> Not sure what to do anyway, unless you can show us that Blender is responsible for that. It is crashing immediately after RENDERED mode is turned off. No addon code is executed on this event - therefore it's something in Blender? > Did you try doing that in exact same python directly (without using Blender's interpreter)? Sure, threads with sleep(0) works fine in pure python. > Not sure to understand the point of using sleep(0) Doesn't matter for me, personally, as anything non-zero passed to sleep seemingly fixes the issue. But there's something wrong, this code should not crash.

C.G.Vedant commented

2016-12-31 13:43:50 +01:00

Added subscriber: @Chipe1

Sergey Sharybin was assigned by Bastien Montagne

2017-01-13 15:21:26 +01:00

Bastien Montagne commented

2017-01-13 15:21:26 +01:00

Added subscriber: @Sergey

Bastien Montagne commented

2017-01-13 15:21:26 +01:00

Bisection shows that crash appears in blender/blender@0cc514ec84. @Sergey, Maybe something missing in new toggle code?

Sergey Sharybin commented

2017-01-13 17:36:16 +01:00

This is not related to my change actually, but rather specific to how threading works in Python.

Basically, before my change the shading mode was changing via python script, so it was a stacktrace like:

C code
Python code
C code

From the C side we do use BPy_BEGIN_ALLOW_THREADS /BPy_END_ALLOW_THREADS to ensure we don't run into threading issues with render engine API. This works fine for Cycles (which doesn't use Python threads), but gives issues in this particular case. Seems call of BPy_BEGIN_ALLOW_THREADS causes race condition with the thread which is still spinning from the python side.

Those begin/end blocks were added in blender/blender@5036ac6. Not yet sure whether it's specific to our BPY_thread_save()/BPY_thread_restore() implementation or not.

With the old code it was still possible to crash blender by changing shading using menu instead of the shortcut.

This is not related to my change actually, but rather specific to how threading works in Python. Basically, before my change the shading mode was changing via python script, so it was a stacktrace like: - C code - Python code - C code From the C side we do use `BPy_BEGIN_ALLOW_THREADS `/`BPy_END_ALLOW_THREADS ` to ensure we don't run into threading issues with render engine API. This works fine for Cycles (which doesn't use Python threads), but gives issues in this particular case. Seems call of `BPy_BEGIN_ALLOW_THREADS` causes race condition with the thread which is still spinning from the python side. Those begin/end blocks were added in blender/blender@5036ac6. Not yet sure whether it's specific to our `BPY_thread_save()`/`BPY_thread_restore()` implementation or not. With the old code it was still possible to crash blender by changing shading using menu instead of the shortcut.

Kerim Borchaev commented

2017-01-16 15:38:14 +01:00

Looks like python threads are just doomed to crash when Shading mode is toggled. It's not just about sleep/wait.
E.g. replacing sleep(0) in the example code with simple with lock: pass makes it crash immediately on second Shift-Z.

Looks like python threads are just doomed to crash when Shading mode is toggled. It's not just about sleep/wait. E.g. replacing `sleep(0)` in the example code with simple `with lock: pass` makes it crash immediately on second Shift-Z.

Kerim Borchaev commented

2017-01-16 16:51:45 +01:00

Is there a possible workaround? Does addon code have any chance to shutdown it's threads before shading mode is toggled? Or defer the toggle so it doesn't happens immediately when Shift-Z is pressed?

Bastien Montagne commented

2017-01-16 17:03:46 +01:00

Hmpff… the main issue is that using python threads inside of Blender is risky at best, it’s only reasonably safe when py threads are fully outside of Blender context (i.e. do not use Blender data, do not call Blender API, are not called by Blender, etc.). I think the issue here is that renderengine callabacks can be called from a job, i.e. from non-main Blender thread, if that’s the case those callbacks should definitively never use any python threading.

Further more, due to main lock in python, pythreads are usually not interesting for performances-driven parallelization, one should rather use processes for that.

Am tempted to consider this as known limitation for now, do you have any reason to actually use pythreads here, and not e.g. concurrent.future with a process pool? On the other hand, if you only want some kind of async update, then I would recommend looking at asyncio, even though its usage remains a bit dodgy in Blender currently…

Hmpff… the main issue is that using python threads inside of Blender is risky at best, it’s only reasonably safe when py threads are fully outside of Blender context (i.e. do not use Blender data, do not call Blender API, are not called by Blender, etc.). I *think* the issue here is that renderengine callabacks can be called from a job, i.e. from non-main Blender thread, if that’s the case those callbacks should definitively *never* use any python threading. Further more, due to main lock in python, pythreads are usually not interesting for performances-driven parallelization, one should rather use processes for that. Am tempted to consider this as known limitation for now, do you have any reason to actually use pythreads here, and not e.g. [concurrent.future](https:*docs.python.org/3.5/library/concurrent.futures.html) with a process pool? On the other hand, if you only want some kind of async update, then I would recommend looking at [asyncio](https:*docs.python.org/3.5/library/asyncio.html), even though its usage remains a bit dodgy in Blender currently…

Kerim Borchaev commented

2017-01-16 20:34:03 +01:00

Yes, I'm aware that Blender context should not be used in a thread. But my examples I think are completely avoiding it. If I'm not mistaken ;)

The reason we are using python threads - is convenience. We have a third-party dll that has rendering capabilities and thread is essentially needed to call a (blocking) 'render' function asynchronously. Your guess is right. This requires a thread. Everything else - is suboptimal. Native thread will add substantial complexity(C++ code), same with interprocess communication. And asyncio won't help - threads will need to be used there too for a blocking task(which we have here).

Am tempted to consider this as known limitation for now,

Worked fine in 2.77!!! From my perspective)

Yes, I'm aware that Blender context should not be used in a thread. But my examples I think are completely avoiding it. If I'm not mistaken ;) The reason we are using python threads - is convenience. We have a third-party dll that has rendering capabilities and thread is essentially needed to call a (blocking) 'render' function asynchronously. Your guess is right. This requires a thread. Everything else - is suboptimal. Native thread will add substantial complexity(C++ code), same with interprocess communication. And asyncio won't help - threads will need to be used there too for a blocking task(which we have here). > Am tempted to consider this as known limitation for now, Worked fine in 2.77!!! From my perspective)

Kerim Borchaev commented

2017-01-17 00:09:58 +01:00

I think the issue here is that renderengine callabacks can be called from a job, i.e. from non-main Blender thread, if that’s the case those callbacks should definitively never use any python threading.

Are you saying that the problem might be that the thread is started from an engine callback?
Is it a viable workaround then - call a custom operator while in view_update and start that thread from the operator's execute?

> I *think* the issue here is that renderengine callabacks can be called from a job, i.e. from non-main Blender thread, if that’s the case those callbacks should definitively *never* use any python threading. Are you saying that the problem might be that the thread is started from an engine callback? Is it a viable workaround then - call a custom operator while in `view_update` and start that thread from the operator's `execute`?

Kerim Borchaev commented

2017-01-21 22:00:57 +01:00

Seems like ANY python thread might crash when RENDERED is being turned off. In my test I've simply added an operator that spawns a thread. Not using a renderengine callback(callbacks are empty). It crashes.
addon_thread_from_operator_crashes_on_rendered_off.7z Just execute the 'Test Start Thread' operator and start pressing Shift-Z(using Test engine or Blender doesn't matter) - it crashes after a few iterations.

Seems like ANY python thread might crash when RENDERED is being turned off. In my test I've simply added an operator that spawns a thread. Not using a renderengine callback(callbacks are empty). It crashes. [addon_thread_from_operator_crashes_on_rendered_off.7z](https://archive.blender.org/developer/F438033/addon_thread_from_operator_crashes_on_rendered_off.7z) Just execute the 'Test Start Thread' operator and start pressing Shift-Z(using Test engine or Blender doesn't matter) - it crashes after a few iterations.

Sergey Sharybin commented

2017-01-25 15:11:13 +01:00

Changed status from 'Open' to: 'Archived'

Sergey Sharybin closed this issue

2017-01-25 15:11:13 +01:00

Sergey Sharybin commented

2017-01-25 15:11:14 +01:00

Here is some conclusion here:

The only thing which worked before is Shit-Z shortcut because it was invoked via Python interpreter.
All Blender versions would have crashed in other cases, such as selecting shading mode from the menu.
The crash is also actually all about timing, meaning depending on where exactly the render thread is, hitting Shit-Z will or will not crash blender.
We never considered python's usable for anything, those are only giving issues.

Surely supporting more cases will be cool, but Blender is not a framework and such tasks gets really low priority. Would also be nice to have help figuring the threading issues from the guys who actually use such threads.

For until then considering this a TODO.

Here is some conclusion here: - The only thing which worked before is Shit-Z shortcut because it was invoked via Python interpreter. - All Blender versions would have crashed in other cases, such as selecting shading mode from the menu. - The crash is also actually all about timing, meaning depending on where exactly the render thread is, hitting Shit-Z will or will not crash blender. - We never considered python's usable for anything, those are only giving issues. Surely supporting more cases will be cool, but Blender is not a framework and such tasks gets really low priority. Would also be nice to have help figuring the threading issues from the guys who actually use such threads. For until then considering this a TODO.

Sign in to join this conversation.