Using bpy module with multiprocessing causes ModuleNotFoundError: No module named '_bpy' #98534

New Issue

Emad Aghajani · 2022-06-01T10:04:20+02:00

Emad Aghajani commented

2022-06-01 10:04:20 +02:00

System Information
Operating system: macOS Monterey (12.4)
Graphics card: M1 Pro

Blender Version
Broken: 3.1
Worked: -

Hello,

I'm using Blender through the bpy module and I noticed using bpy module with multiprocessing causes a ModuleNotFoundError error. I just wanted to draw Blender developers attention to this bug.

How to reproduce the crash

Create bug.py with this content, and run python3 bug.py:

import bpy
import multiprocessing as mp
if __name__ == "__main__":
      manager = mp.Manager()

You will get the following traceback:

Traceback (most recent call last):

File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
  exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
  prepare(preparation_data)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
  _fixup_main_from_path(data['init_main_from_path'])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
  main_content = runpy.run_path(main_path,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 269, in run_path
  return _run_module_code(code, init_globals, run_name,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 96, in _run_module_code
  _run_code(code, mod_globals, init_globals,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
  exec(code, run_globals)
File "/Users/emadpres/Downloads/bug.py", line 1, in <module>
  import bpy
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/Resources/3.1/scripts/modules/bpy/__init__.py", line 38, in <module>
  from _bpy import (

ModuleNotFoundError: No module named '_bpy'
Traceback (most recent call last):

File "/Users/emadpres/Downloads/bug.py", line 4, in <module>
  manager = mp.Manager()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 57, in Manager
  m.start()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/managers.py", line 566, in start
  self._address = reader.recv()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 255, in recv
  buf = self._recv_bytes()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes
  buf = self._recv(4)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 388, in _recv
  raise EOFError

EOFError

Why does it happen?

A few discussion/threads have already touched on this issue and here is a summary from here nicely describing why this error happens in essence:

We import bpy in the main process

This brings in the C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.832\Scripts\2.79\scripts\modules to the top of sys.path because the Blender runtime needs it

The multiprocessing.Pool instantiation grabs our bad sys.path

Since modules aren't portable across the multiprocessing.Queue our new processes are going to try to import the modules again

Search along sys.path for bpy: I found one at C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules!

Error: can't locate or import its dependency _bpy (mostly because that is a Blender runtime convention that does not apply in Python)

Hacky Workarounds

I also found a couple of hacky workarounds:

backup sys.path before import bpy. And restore it right before instantiating the multiprocessing object. Once done, revert your change. Code, or
guarding the import bpy to avoid re-importing it:

from multiprocessing import current_process
import multiprocessing as mp

if current_process().name == 'MainProcess':
    import bpy


if __name__ == "__main__":
    manager = mp.Manager()

**System Information** Operating system: macOS Monterey (12.4) Graphics card: M1 Pro **Blender Version** Broken: 3.1 Worked: - Hello, I'm using Blender through the `bpy` module and I noticed using `bpy` module with `multiprocessing` causes a ModuleNotFoundError error. I just wanted to draw Blender developers attention to this bug. ## How to reproduce the crash Create `bug.py` with this content, and run `python3 bug.py`: ```Py import bpy import multiprocessing as mp if __name__ == "__main__": manager = mp.Manager() ``` You will get the following traceback: ```traceback Traceback (most recent call last): File "<string>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 269, in run_path return _run_module_code(code, init_globals, run_name, File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 96, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/Users/emadpres/Downloads/bug.py", line 1, in <module> import bpy File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/Resources/3.1/scripts/modules/bpy/__init__.py", line 38, in <module> from _bpy import ( ModuleNotFoundError: No module named '_bpy' Traceback (most recent call last): File "/Users/emadpres/Downloads/bug.py", line 4, in <module> manager = mp.Manager() File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 57, in Manager m.start() File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/managers.py", line 566, in start self._address = reader.recv() File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError EOFError ``` ## Why does it happen? A few discussion/threads have already touched on this issue and here is a summary from [here](https://github.com/TylerGubala/blenderpy/issues/23#issuecomment-514826760) nicely describing why this error happens in essence: > 1. We import bpy in the main process > 2. This brings in the `C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.832\Scripts\2.79\scripts\modules` to the top of sys.path because the Blender runtime needs it > 3. The `multiprocessing.Pool` instantiation grabs our bad `sys.path` > 4. Since modules aren't portable across the `multiprocessing.Queue` our new processes are going to try to import the modules again > 5. Search along `sys.path` for `bpy`: I found one at `C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules`! > 6. Error: can't locate or import its dependency `_bpy` (mostly because that is a Blender runtime convention that does not apply in Python) ## Hacky Workarounds I also found a couple of hacky workarounds: 1. backup `sys.path` before `import bpy`. And restore it right before instantiating the multiprocessing object. Once done, revert your change. [Code](https://github.com/TylerGubala/blenderpy/issues/23#issuecomment-514826760), or 2. guarding the `import bpy` to avoid re-importing it: ```python from multiprocessing import current_process import multiprocessing as mp if current_process().name == 'MainProcess': import bpy if __name__ == "__main__": manager = mp.Manager() ```

👀 1

Emad Aghajani commented

2022-06-01 10:04:20 +02:00

Added subscriber: @emadpres

Campbell Barton commented

2022-06-02 08:36:27 +02:00

Added subscriber: @ideasman42

Campbell Barton commented

2022-06-02 08:36:27 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Campbell Barton commented

2022-06-02 08:36:27 +02:00

Thanks for the detailed description of the error. If I understand this correctly the error is caused by there being two bpy modules.

./bpy.so
./3.3/scripts/modules/bpy/__init__.py

Once C/Python module is loaded, spawned subprocesses find the second one which has not loaded the C/Python module _bpy which bpy.so defines.

There are probably a few ways to resolve this, although I'm currently unable to reproduce the bug, the example code provided loads the Blender module and exists, tested with a build from running: make bpy lite.

Could you show an example that spawns processes that attempt to use the bpy module?

Thanks for the detailed description of the error. If I understand this correctly the error is caused by there being two `bpy` modules. - `./bpy.so` - `./3.3/scripts/modules/bpy/__init__.py` Once C/Python module is loaded, spawned subprocesses find the second one which has not loaded the C/Python module `_bpy` which `bpy.so` defines. ---- There are probably a few ways to resolve this, although I'm currently unable to reproduce the bug, the example code provided loads the Blender module and exists, tested with a build from running: `make bpy lite`. Could you show an example that spawns processes that attempt to use the bpy module?

Emad Aghajani commented

2022-06-20 13:32:40 +02:00

Hi Campbell and thanks for your following up. I used make bpy to build the Blender module. The error happens on my mac (arm64), but not on my ubuntu machine. Are you on macOS or other operating systems?

Hi Campbell and thanks for your following up. I used `make bpy` to build the Blender module. The error happens on my mac (arm64), but not on my ubuntu machine. Are you on macOS or other operating systems?

Campbell Barton commented

2022-06-21 00:45:02 +02:00

@emadpres I'm using Arch Linux, it seems this may be spesific to macOS.

Emad Aghajani commented

2022-11-03 10:18:50 +01:00

@ideasman42 Yes, it is specific to macOS. It doesn't occur in Ubuntu for me.
Any update on the issue?

@ideasman42 Yes, it is specific to macOS. It doesn't occur in Ubuntu for me. Any update on the issue?

Alexander Hartmann commented

2022-11-03 13:08:43 +01:00

Added subscriber: @Alexander-Hartmann

Alexander Hartmann commented

2022-11-03 23:29:27 +01:00

This comment was removed by @Alexander-Hartmann

*This comment was removed by @Alexander-Hartmann*

Campbell Barton commented

2022-11-04 00:50:43 +01:00

@Alexander-Hartmann, as this isn't related to multiprocessing module, best make a new report.

@Alexander-Hartmann, as this isn't related to `multiprocessing` module, best make a new report.

Alexander Hartmann commented

2022-11-04 02:30:51 +01:00

ok deleted, until ##multiprocessing## I could not run yet

import bpy
import multiprocessing as mp
if __name__ == "__main__":
       manager = mp.Manager()

ok deleted, until ##multiprocessing## I could not run yet ``` import bpy import multiprocessing as mp if __name__ == "__main__": manager = mp.Manager()

Moritz Brückner commented

2022-12-10 20:45:19 +01:00

Added subscriber: @timodriaan

Thomas Dinges commented

2022-12-15 16:56:24 +01:00

Changed status from 'Needs User Info' to: 'Needs Triage'

Alexander Hartmann commented

2022-12-17 15:01:50 +01:00

Removed subscriber: @Alexander-Hartmann

Atamert Ölçgen commented

2023-01-02 18:58:25 +01:00

Added subscriber: @muhuk

Tobiasz Karoń commented

2023-01-02 19:04:37 +01:00

Added subscriber: @TobiaszunfaKaron

Tobiasz Karoń commented

2023-01-02 19:04:37 +01:00

It seems I am having the same problem on Windows 10.

When using Grid UV Packer in Blender 3.4.1 it seems to freeze blender with this console output:

Seed being used is: 2094157608
batch size is 30.
Process SpawnProcess-1:
Traceback (most recent call last):
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker
    call_item = call_queue.get(block=True)
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\unfa\AppData\Roaming\Blender Foundation\Blender\3.4\scripts\addons\guvp\__init__.py", line 42, in <module>
    import bpy                                                  # type: ignore
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\scripts\modules\bpy\__init__.py", line 20, in <module>
    from _bpy import (
Process SpawnProcess-4:
Traceback (most recent call last):
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker
    call_item = call_queue.get(block=True)
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get
    return _ForkingPickler.loads(res)
ModuleNotFoundError: No module named '_bpy'
  File "C:\Users\unfa\AppData\Roaming\Blender Foundation\Blender\3.4\scripts\addons\guvp\__init__.py", line 42, in <module>
    import bpy                                                  # type: ignore
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\scripts\modules\bpy\__init__.py", line 20, in <module>
    from _bpy import (
ModuleNotFoundError: No module named '_bpy'
Process SpawnProcess-9:
Traceback (most recent call last):
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker
    call_item = call_queue.get(block=True)
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\unfa\AppData\Roaming\Blender Foundation\Blender\3.4\scripts\addons\guvp\__init__.py", line 42, in <module>
    import bpy                                                  # type: ignore
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\scripts\modules\bpy\__init__.py", line 20, in <module>
    from _bpy import (
Process SpawnProcess-7:
ModuleNotFoundError: No module named '_bpy'
Process SpawnProcess-8:
Traceback (most recent call last):
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker
    call_item = call_queue.get(block=True)
Process SpawnProcess-5:
Process SpawnProcess-3:
Process SpawnProcess-2:
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get
    return _ForkingPickler.loads(res)
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)

It seems I am having the same problem on Windows 10. When using Grid UV Packer in Blender 3.4.1 it seems to freeze blender with this console output: ``` Seed being used is: 2094157608 batch size is 30. Process SpawnProcess-1: Traceback (most recent call last): File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker call_item = call_queue.get(block=True) File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get return _ForkingPickler.loads(res) File "C:\Users\unfa\AppData\Roaming\Blender Foundation\Blender\3.4\scripts\addons\guvp\__init__.py", line 42, in <module> import bpy # type: ignore File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\scripts\modules\bpy\__init__.py", line 20, in <module> from _bpy import ( Process SpawnProcess-4: Traceback (most recent call last): File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker call_item = call_queue.get(block=True) File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get return _ForkingPickler.loads(res) ModuleNotFoundError: No module named '_bpy' File "C:\Users\unfa\AppData\Roaming\Blender Foundation\Blender\3.4\scripts\addons\guvp\__init__.py", line 42, in <module> import bpy # type: ignore File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\scripts\modules\bpy\__init__.py", line 20, in <module> from _bpy import ( ModuleNotFoundError: No module named '_bpy' Process SpawnProcess-9: Traceback (most recent call last): File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker call_item = call_queue.get(block=True) File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get return _ForkingPickler.loads(res) File "C:\Users\unfa\AppData\Roaming\Blender Foundation\Blender\3.4\scripts\addons\guvp\__init__.py", line 42, in <module> import bpy # type: ignore File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\scripts\modules\bpy\__init__.py", line 20, in <module> from _bpy import ( Process SpawnProcess-7: ModuleNotFoundError: No module named '_bpy' Process SpawnProcess-8: Traceback (most recent call last): File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) Traceback (most recent call last): File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\concurrent\futures\process.py", line 240, in _process_worker call_item = call_queue.get(block=True) Process SpawnProcess-5: Process SpawnProcess-3: Process SpawnProcess-2: File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\queues.py", line 122, in get return _ForkingPickler.loads(res) Traceback (most recent call last): Traceback (most recent call last): File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) ```

Iliya Katushenock commented

2023-05-31 15:03:50 +02:00

No description provided.

py stack.txt

1.1 KiB

Stshalson commented

2023-06-19 11:56:23 +02:00

@emadpres I'm using Arch Linux, it seems this may be specific to macOS.

It seems OS-specific because multiprocessing uses the "spawn" start method on MacOS and Windows, and "fork" as default on Linux. This problem occurs only if the start method is set to "spawn". Although everything seems to work nicely with the fork, I need the spawn method to initialize the CUDA context in subprocesses, as of Blender 3.6 it still doesn't work.
Note that you cant set the "fork" start method in non-Linux operating systems as it is not supported.

> @emadpres I'm using Arch Linux, it seems this may be specific to macOS. It seems OS-specific because multiprocessing uses the "spawn" start method on MacOS and Windows, and "fork" as default on Linux. This problem occurs only if the start method is set to "spawn". Although everything seems to work nicely with the fork, I need the spawn method to initialize the CUDA context in subprocesses, as of Blender 3.6 it still doesn't work. Note that you cant set the "fork" start method in non-Linux operating systems as it is not supported.

Iliya Katushenock removed the

Status

Needs Triage

label 2023-08-24 17:38:31 +02:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Using bpy module with multiprocessing causes ModuleNotFoundError: No module named '_bpy' #98534

How to reproduce the crash

Why does it happen?

Hacky Workarounds

Using `bpy` module with `multiprocessing` causes `ModuleNotFoundError: No module named '_bpy'` #98534