About asyncio loop handling in BlenderCloud addon #49275

New Issue

Bastien Montagne · 2016-09-06T18:21:22+02:00

Bastien Montagne commented

2016-09-06 18:21:22 +02:00

So, have been banging my head against the issue of running asyncio's loop as a sub-loop of Blender one, trying to understand how the kick_async_loop() can actually work - and failing so far.

As far as I understand, asyncio loop is run by the addon that way:

Create a dummy modal operator, that execute a function very (very!) often, every 0.01ms!
That operator calls a 'loop kicker' (kick_async_loop()) to “make advance” the asyncio loop.

My problem is with that loop kicker: outside of some 'break' checks, it basically calls loop.stop(), and immediately after loop.run_forever(). But loop.run_forever() is blocking, right? It will only return once loop.stop() has been called. In other words, we are blocking the main thread (i.e. the whole Blender itself!) waiting for asyncio loop to suspend itself.

Which leads me to think that this cannot (should not!) actually work, unless maybe it does currently because you limit number of simultaneous pillar connections to 3?

Anyway, @dr.sybren, can you confirm that, or explain me how you stop the loop once launched? ;) get back to the

For own work (asset engines in asset-engine branch), after a lot of hours banging my head against the issue, I think I found the (correct?) solution: you have to schedule the call to loop.stop() before calling loop.run_forever() (with either call_soon() or, if you want to leave some more processing time to your loop, call_later() with reasonably small delay (below 1ms for sure)). That way, you can safely run a step or a small bit of the loop, and quickly get back the hand over main thread. ;)

So, have been banging my head against the issue of running asyncio's loop as a sub-loop of Blender one, trying to understand how the `kick_async_loop()` can actually work - and failing so far. As far as I understand, asyncio loop is run by the addon that way: - Create a dummy modal operator, that execute a function very (very!) often, every 0.01ms! - That operator calls a 'loop kicker' (`kick_async_loop()`) to “make advance” the asyncio loop. My problem is with that loop kicker: outside of some 'break' checks, it basically calls `loop.stop()`, and immediately after `loop.run_forever()`. But `loop.run_forever()` is blocking, right? It will only return once `loop.stop()` has been called. In other words, we are blocking the main thread (i.e. the whole Blender itself!) waiting for asyncio loop to suspend itself. Which leads me to think that this cannot (should not!) actually work, unless maybe it does currently because you limit number of simultaneous pillar connections to 3? Anyway, @dr.sybren, can you confirm that, or explain me *how* you stop the loop once launched? ;) get back to the For own work (asset engines in asset-engine branch), after a lot of hours banging my head against the issue, I think I found the (correct?) solution: you have to schedule the call to `loop.stop()` before calling `loop.run_forever()` (with either `call_soon()` or, if you want to leave some more processing time to your loop, `call_later()` with reasonably small delay (below 1ms for sure)). That way, you can safely run a step or a small bit of the loop, and quickly get back the hand over main thread. ;)

Bastien Montagne commented

2016-09-06 18:21:23 +02:00

Changed status to: 'Open'

Bastien Montagne commented

2016-09-06 18:21:23 +02:00

Added subscribers: @dr.sybren, @mont29

Sybren A. Stüvel commented

2016-09-07 09:07:37 +02:00

The select() calls in asyncio are pretty smart, so it doesn't wait around for data to appear. This is why it can very quickly relinquish execution to Blender.

The call to loop.stop() is there because asyncio doesn't have a loop.run_one_iteration() method. However, from what I've seen, this simulates it quite well. The loop.stop() call sets a flag that's checked at the end of a loop iteration. Effectively it makes asyncio do whatever can be done now, then stops it again.

We could run the asyncio loop in a separate thread, or even using a ThreadPoolExecutor in multiple parallel threads. That would have the advantage that it can keep running, and doesn't need any "kicking". However, it also introduces all the difficulty associated with multithreaded code. As a result, it prevents any async code from calling Blender functions or modifying Blender state, which is IMO a very big downside. With the code in the current state, there is little difference between what you can do in an async function and a regular one. Async-running operators can simply use self.report() to notify the user of events. If this ability disappears, we should have a well thought-out way to communicate with the user that's thread-safe.

The `select()` calls in asyncio are pretty smart, so it doesn't wait around for data to appear. This is why it can very quickly relinquish execution to Blender. The call to `loop.stop()` is there because asyncio doesn't have a `loop.run_one_iteration() ` method. However, from what I've seen, this simulates it quite well. The `loop.stop()` call sets a flag that's checked at the end of a loop iteration. Effectively it makes asyncio do whatever can be done *now*, then stops it again. We could run the asyncio loop in a separate thread, or even using a `ThreadPoolExecutor` in multiple parallel threads. That would have the advantage that it can keep running, and doesn't need any "kicking". However, it also introduces all the difficulty associated with multithreaded code. As a result, it prevents any async code from calling Blender functions or modifying Blender state, which is IMO a very big downside. With the code in the current state, there is little difference between what you can do in an async function and a regular one. Async-running operators can simply use `self.report()` to notify the user of events. If this ability disappears, we should have a well thought-out way to communicate with the user that's thread-safe.

Bastien Montagne commented

2016-09-07 09:43:57 +02:00

Ah OK, I see know… so basically, calling (assuming loop is currently stopped of course):

lang=python
loop.stop()
loop.run_forever()

is essentially the same thing as

lang=python
loop.call_soon(loop.stop)
loop.run_forever()

…right? So now things make sense, thanks! :)

I still remain unconvinced about the non-threadibility of the thing though, imho it would allow much more flexibility, and enforce more sane code by precisely preventing ability to call Blender stuff from everywhere. ;) But I do see how it can make things more easy on the other hand, so… can live with it!

Ah OK, I see know… so basically, calling (assuming loop is currently stopped of course): ``` lang=python loop.stop() loop.run_forever() ``` is essentially the same thing as ``` lang=python loop.call_soon(loop.stop) loop.run_forever() ``` …right? So now things make sense, thanks! :) I still remain unconvinced about the non-threadibility of the thing though, imho it would allow much more flexibility, and enforce more sane code by precisely preventing ability to call Blender stuff from everywhere. ;) But I do see how it can make things more easy on the other hand, so… can live with it!

Sybren A. Stüvel commented

2016-09-13 15:36:25 +02:00

Be sure to check whether loop.call_soon(loop.stop) doesn't starve other loop.call_soon(...) calls. Those are the tricky undocumented behaviours we might want to check in a unit test or something.

Be sure to check whether `loop.call_soon(loop.stop)` doesn't starve other `loop.call_soon(...)` calls. Those are the tricky undocumented behaviours we might want to check in a unit test or something.

Bastien Montagne commented

2016-10-07 14:14:54 +02:00

Changed status from 'Open' to: 'Archived'

Bastien Montagne closed this issue

2016-10-07 14:14:54 +02:00

Bastien Montagne self-assigned this 2016-10-07 14:14:54 +02:00

Bastien Montagne commented

2016-10-07 14:14:54 +02:00

Eeeh, think we can close that one for now :)

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

About asyncio loop handling in BlenderCloud addon #49275