Unable to export/import EXR files after doing `import bpy` and when using the multiprocessing in Python (Blender freezes)
Open, LowPublic

Description

I am trying to render depth maps and surface Normals using Cycles and store the results in OpenEXR format using nodes. I set up the nodes as follow using Blender's Python API:

#blenderClass.py
import importlib

class Blender(object):
    def __init__(self):
        self.bpy = importlib.import_module("bpy")
        self.scene = self.bpy.context.scene
        self.scene.render.use_sequencer = False
        self.scene.display_settings.display_device = 'sRGB'
	self.scene.view_settings.view_transform = 'Raw'
	self.scene.sequencer_colorspace_settings.name = 'Raw'
        self.scene.render.engine = 'CYCLES'
        self.scene.use_nodes = True
	self.setupRenderNodes()

     def setupRenderNodes(self):
		
	for node in self.scene.node_tree.nodes:
		self.scene.node_tree.nodes.remove(node)
	renderNode = self.scene.node_tree.nodes.new('CompositorNodeRLayers')

	# Depth map
	self.depthOutputNode = self.scene.node_tree.nodes.new('CompositorNodeOutputFile')
	self.depthOutputNode.format.file_format = 'OPEN_EXR'
	self.depthOutputNode.format.color_depth = '32'
	self.depthOutputNode.format.color_mode = 'RGB'
	self.depthOutputNode.format.exr_codec = 'ZIP'
	self.depthOutputNode.base_path = 'somePath/'
	self.depthOutputNode.file_slots[0].path = 'fileNameDepth#'

	# Link
	self.scene.node_tree.links.new(renderNode.outputs[2], self.depthOutputNode.inputs[0])

    def render(self, objPath):
        self.bpy.ops.import_scene.obj(filepath=objPath)
        self.bpy.ops.render.render(write_still=True)
#main.py
from multiprocessing import Process
import blenderClass import Blender
blender = Blender()

objPaths = ['obj1.obj', 'obj2.obj', 'obj3.obj', 'obj4.obj']

procList = []
for path in objPaths:
    proc = Process(target=blender.render, kwargs={'objPath': path})
    procList.append(proc)
    proc.start()

for job in procList:
    job.join()

The problem is when I do bpy.ops.render.render(write_still=True) Blender freezes and the results are not saved on disk. However, if I change OPEN_EXR_MULTILAYER and OPEN_EXR to PNG and change the color_depth to '16' everything works and I get the rendering results on disk. Note that I do not change any of my rendering engine settings like tile size, resolution etc. Does anyone know why Blender freezes and the rendering results are not on disk when using OpenEXR?

The strange thing is everything works normally when running Blender with GUI and setup the nodes and rendering engine the same way. It would be also useful if someone can try this and let me know if things work fine on their side.

I'm not sure if this has something to do with the problem that I'm facing but I have compiled Blender manually and import it as a module in the Python installed on my machine. Here are the CMake settings I used to compile Blender 2.79b from source:

cmake blender \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/python3/dist-packages \
    -DWITH_INSTALL_PORTABLE=OFF \
    -DWITH_PYTHON_INSTALL=OFF \
    -DWITH_PYTHON_MODULE=ON \
    -DPYTHON_SITE_PACKAGES=/usr/lib/python3/dist-packages \
    -DPYTHON_VERSION=3.5 \
    -DWITH_OPENAL=ON \
    -DWITH_CODEC_AVI=ON \
    -DWITH_MOD_OCEANSIM=ON \
    -DWITH_CODEC_FFMPEG=ON \
    -DWITH_SYSTEM_GLEW=ON \
    -DWITH_FFTW3=ON \
    -DWITH_OPENCOLORIO=ON \
    -DWITH_GAMEENGINE=OFF \
    -DWITH_PLAYER=OFF

Update: I experience the same behavior with Blender 2.78 and 2.79, compiled with the same exact settings above.

Details

Type
Bug
Amir (Warrior) added a project: Python.
Amir (Warrior) updated the task description. (Show Details)
Brecht Van Lommel (brecht) triaged this task as Incomplete priority.EditedSun, Apr 1, 7:23 AM

I can't reproduce this issue. I tried putting the above code in a test.py script with bpy.ops.render.render(write_still=True) at the end, and then running ./blender -b -P test.py.

  • Please try with official builds from blender.org, to check if it's caused by any difference in your own build.
  • Mention which operating system you are testing this on.
  • Trying run with blender with --factory-startup to check if any user preferences are influencing this.
  • If you're using Linux, you can run gdb --args ./blender -b -P test.py, then type run, press ctrl+C while it hangs and type thread apply bt all. The output of that should give a clue about where it's hanging, you can attach that here.

@Brecht Van Lommel (brecht) Sorry I should have been more clear. The problem is I am compiling Blender as Python module in Ubuntu 16.04. I don't think it is possible to import the Blender module given those arguments you mentioned. Could you please double-check this as well?

Amir (Warrior) raised the priority of this task from Incomplete to Needs Triage.Mon, Apr 2, 3:53 PM
Amir (Warrior) added a comment.EditedMon, Apr 2, 5:32 PM

@Brecht Van Lommel (brecht) Sorry for the missing information. I just realized that the problem is being caused if I call the rendering function using multiprocessing.Process and updated the script in my bug report that shows how exactly I do rendering in my project, with all other irrelevant details removed. The strange thing is I can easily render my meshes if I do not want the rendering result to be stored in EXR files. So, if I replace OPEN_EXR with PNG and replace 32 with 8/16 I will definitely get the rendering results stored on disk. This does not happen when trying to store the result in OpenEXR format and I did not know that the multiprocessing package might be causing it. What should I do now? Why I cannot store EXR files when using multiprocessing.Process?

Also, another person has issues storing multi-channel EXR files as he/she mentioned here. Maybe you can try fixing both my issue and his/her? Or I can post another bug report for that.

Can you try using the spawn start method?
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

It would not surprise me if there is some part of Blender or the OpenEXR library code that does not work with forking. I'm also not inclined to spend much time investigating that, I don't think this is something we can guarantee to work well without a lot of effort verifying lots of code.

Brecht Van Lommel (brecht) triaged this task as Incomplete priority.Mon, Apr 2, 6:30 PM

If there's another bug with OpenEXR saving unrelated to multiprocessing, another bug report can be created for that,

Amir (Warrior) added a comment.EditedMon, Apr 2, 8:59 PM

@Brecht Van Lommel (brecht) I tried with spawn but I keep getting weird errors and things do not work. It seems that I have to somehow make the process spawnable and I don't know how to do that precisely since changing my pipeline is not as straight-forward. I hope you can find a solution for fixing the issue if it's coming from Blender side. It's so annoying for me since I have to do millions of renderings and I didn't expect to do it sequentially :(

Amir (Warrior) added a comment.EditedTue, Apr 3, 5:50 AM

@Brecht Van Lommel (brecht) You said it wouldn't surprise you if some parts of Blender of OpenEXr does not work well with forking. So I wonder, which one do you think is more likely to be the case given that I can store my renderings in other formats? Would you think it's more likely that OpenEXR is the one to blame here?

@Brecht Van Lommel (brecht) I also posted an issue on OpenEXR's GitHub repo (link here). Hopefully they can also make some clarification from their side.

Amir (Warrior) renamed this task from Unable to export rendering results in OpenEXR format using Python (Blender freezes) to Unable to export renderings in OpenEXR when using the multiprocessing in Python (Blender freezes).Tue, Apr 3, 6:13 AM

I would not make a bug report to the OpenEXR project, it's not clear at all there is a bug in their library and I wouldn't expect them to investigate a vague report like that. It's a possibility but too early to tell.

To work around the problem you can just launch multiple Python scripts that each do part of the work, or import the Blender module after calling multiprocessing.Process so that it is not forked.

@Brecht Van Lommel (brecht) I'm not sure if this could be very relevant but I used OpenEXR Python bindings package to do some tests. Here's what I found out: if I have a function that simply loads an exr file and separates its channels into (R, G,B) as shown below and I execute this function via multiprocessing.Process everything works fine. However, when I just do import bpy and execute the same function via multiprocessing.Process things do not work and Python freezes. I would say this is a bug in Blender. Could you please look into this? I would really appreciate it.

import OpenEXR, Imath, array
import numpy as np
from multiprocessing import Process

def readEXR(exrPath):
    exrFile = OpenEXR.InputFile(exrPath)
    print ('1')
    FLOAT = Imath.PixelType(Imath.PixelType.FLOAT)
    print ('2')
    (R,G,B) = [array.array('f', exrFile.channel(Chan, FLOAT)).tolist() for Chan in ("R", "G", "B") ]
    print ('3')

path = 'someExrFile.exr'
proc = Process(target=readEXR, kwargs={'exrPath': path})
proc.start() # This will correctly print 3
proc.join() 

import bpy
proc = Process(target=readEXR, kwargs={'exrPath': path})
proc.start() # This will never print 3. It will print 1 and 2
proc.join()
Amir (Warrior) raised the priority of this task from Incomplete to Needs Triage.Wed, Apr 4, 4:53 AM
Amir (Warrior) renamed this task from Unable to export renderings in OpenEXR when using the multiprocessing in Python (Blender freezes) to Unable to export/import EXR files after doing `import bpy` and when using the multiprocessing in Python (Blender freezes).
Brecht Van Lommel (brecht) triaged this task as Low priority.Wed, Apr 4, 5:01 AM

This is a very low priority issue for me, since it should be quite easy to work around in the ways I suggested and the python module is not something we officially release. Given that there are many more important bugs to solve I'm not going to spend time on this now.

@Brecht Van Lommel (brecht) I hope you will prioritize this in the future. Many people like me need to use Blender as a module for their research work. Anyways, I used gdb to get you the back trace after Python freezes. Here's what I get (I had to CTRL+C at the end):

gdb python3
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/python3 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import OpenEXR, Imath, array
>>> import numpy as np
[New Thread 0x2aaab098f700 (LWP 37861)]
[New Thread 0x2aaab0b90700 (LWP 37862)]
[New Thread 0x2aaab4d91700 (LWP 37863)]
[New Thread 0x2aaab6f92700 (LWP 37864)]
[New Thread 0x2aaab9193700 (LWP 37865)]
[New Thread 0x2aaabb394700 (LWP 37866)]
[New Thread 0x2aaabd595700 (LWP 37867)]
[New Thread 0x2aaabf796700 (LWP 37868)]
[New Thread 0x2aaac1997700 (LWP 37869)]
[New Thread 0x2aaac3b98700 (LWP 37870)]
[New Thread 0x2aaac5d99700 (LWP 37871)]
[New Thread 0x2aaac7f9a700 (LWP 37872)]
[New Thread 0x2aaac819b700 (LWP 37873)]
[New Thread 0x2aaacc39c700 (LWP 37874)]
[New Thread 0x2aaace59d700 (LWP 37875)]
[New Thread 0x2aaad079e700 (LWP 37876)]
[New Thread 0x2aaad299f700 (LWP 37877)]
[New Thread 0x2aaad4ba0700 (LWP 37878)]
[New Thread 0x2aaad6da1700 (LWP 37879)]
[New Thread 0x2aaad8fa2700 (LWP 37880)]
[New Thread 0x2aaadb1a3700 (LWP 37881)]
[New Thread 0x2aaadd3a4700 (LWP 37882)]
[New Thread 0x2aaadf5a5700 (LWP 37883)]
[New Thread 0x2aaae17a6700 (LWP 37884)]
[New Thread 0x2aaae39a7700 (LWP 37885)]
[New Thread 0x2aaae5ba8700 (LWP 37886)]
[New Thread 0x2aaae7da9700 (LWP 37887)]
[New Thread 0x2aaae9faa700 (LWP 37888)]
[New Thread 0x2aaaec1ab700 (LWP 37889)]
[New Thread 0x2aaaee3ac700 (LWP 37890)]
[New Thread 0x2aaaf05ad700 (LWP 37891)]
[New Thread 0x2aaaf27ae700 (LWP 37892)]
[New Thread 0x2aaaf49af700 (LWP 37893)]
[New Thread 0x2aaaf6bb0700 (LWP 37894)]
[New Thread 0x2aaaf8db1700 (LWP 37895)]
[New Thread 0x2aaafafb2700 (LWP 37896)]
[New Thread 0x2aaafd1b3700 (LWP 37897)]
[New Thread 0x2aaafd3b4700 (LWP 37898)]
[New Thread 0x2aab015b5700 (LWP 37899)]
[New Thread 0x2aab017b6700 (LWP 37900)]
[New Thread 0x2aab039b7700 (LWP 37901)]
[New Thread 0x2aab07bb8700 (LWP 37902)]
[New Thread 0x2aab07db9700 (LWP 37903)]
[New Thread 0x2aab09fba700 (LWP 37904)]
[New Thread 0x2aab0e1bb700 (LWP 37905)]
[New Thread 0x2aab0e3bc700 (LWP 37906)]
[New Thread 0x2aab105bd700 (LWP 37907)]
>>> from multiprocessing import Process
>>> 
>>> def readEXR(exrPath):
...     exrFile = OpenEXR.InputFile(exrPath)
...     print ('1')
...     FLOAT = Imath.PixelType(Imath.PixelType.FLOAT)
...     print ('2')
...     (R,G,B) = [array.array('f', exrFile.channel(Chan, FLOAT)).tolist() for Chan in ("R", "G", "B") ]
...     print ('3')
... 
>>> path = 'data/ShapeNetCore.v2/rawRenderings/train/airplane/airplane-3048/256/depth/cam01.exr'
>>> proc = Process(target=readEXR, kwargs={'exrPath': path})
>>> proc.start() # This will correctly print 3
[Thread 0x2aaaf6bb0700 (LWP 37894) exited]
[Thread 0x2aab105bd700 (LWP 37907) exited]
[Thread 0x2aab0e3bc700 (LWP 37906) exited]
[Thread 0x2aab0e1bb700 (LWP 37905) exited]
[Thread 0x2aab09fba700 (LWP 37904) exited]
[Thread 0x2aab07db9700 (LWP 37903) exited]
[Thread 0x2aab07bb8700 (LWP 37902) exited]
[Thread 0x2aab039b7700 (LWP 37901) exited]
[Thread 0x2aab017b6700 (LWP 37900) exited]
[Thread 0x2aab015b5700 (LWP 37899) exited]
[Thread 0x2aaafd3b4700 (LWP 37898) exited]
[Thread 0x2aaafd1b3700 (LWP 37897) exited]
[Thread 0x2aaafafb2700 (LWP 37896) exited]
[Thread 0x2aaaf8db1700 (LWP 37895) exited]
[Thread 0x2aaaf49af700 (LWP 37893) exited]
[Thread 0x2aaaf27ae700 (LWP 37892) exited]
[Thread 0x2aaaf05ad700 (LWP 37891) exited]
[Thread 0x2aaaee3ac700 (LWP 37890) exited]
[Thread 0x2aaaec1ab700 (LWP 37889) exited]
[Thread 0x2aaae9faa700 (LWP 37888) exited]
[Thread 0x2aaae7da9700 (LWP 37887) exited]
[Thread 0x2aaae5ba8700 (LWP 37886) exited]
[Thread 0x2aaae39a7700 (LWP 37885) exited]
[Thread 0x2aaae17a6700 (LWP 37884) exited]
[Thread 0x2aaadf5a5700 (LWP 37883) exited]
[Thread 0x2aaadd3a4700 (LWP 37882) exited]
[Thread 0x2aaadb1a3700 (LWP 37881) exited]
[Thread 0x2aaad8fa2700 (LWP 37880) exited]
[Thread 0x2aaad6da1700 (LWP 37879) exited]
[Thread 0x2aaad4ba0700 (LWP 37878) exited]
[Thread 0x2aaad299f700 (LWP 37877) exited]
[Thread 0x2aaad079e700 (LWP 37876) exited]
[Thread 0x2aaace59d700 (LWP 37875) exited]
[Thread 0x2aaacc39c700 (LWP 37874) exited]
[Thread 0x2aaac819b700 (LWP 37873) exited]
[Thread 0x2aaac7f9a700 (LWP 37872) exited]
[Thread 0x2aaac5d99700 (LWP 37871) exited]
[Thread 0x2aaac3b98700 (LWP 37870) exited]
[Thread 0x2aaac1997700 (LWP 37869) exited]
[Thread 0x2aaabf796700 (LWP 37868) exited]
[Thread 0x2aaabd595700 (LWP 37867) exited]
[Thread 0x2aaabb394700 (LWP 37866) exited]
[Thread 0x2aaab9193700 (LWP 37865) exited]
[Thread 0x2aaab6f92700 (LWP 37864) exited]
[Thread 0x2aaab4d91700 (LWP 37863) exited]
[Thread 0x2aaab0b90700 (LWP 37862) exited]
[Thread 0x2aaab098f700 (LWP 37861) exited]
>>> proc.join() 
1
2
3
>>> 
>>> import bpy
[New Thread 0x2aab105bd700 (LWP 37909)]
[New Thread 0x2aab0e3bc700 (LWP 37910)]
[New Thread 0x2aab0e1bb700 (LWP 37911)]
[New Thread 0x2aab09fba700 (LWP 37912)]
[New Thread 0x2aab07db9700 (LWP 37913)]
[New Thread 0x2aab07bb8700 (LWP 37914)]
[New Thread 0x2aab039b7700 (LWP 37915)]
[New Thread 0x2aab017b6700 (LWP 37916)]
[New Thread 0x2aab015b5700 (LWP 37917)]
[New Thread 0x2aaafd3b4700 (LWP 37918)]
[New Thread 0x2aaafd1b3700 (LWP 37919)]
[New Thread 0x2aaafafb2700 (LWP 37920)]
[New Thread 0x2aaaf8db1700 (LWP 37921)]
[New Thread 0x2aaaf6bb0700 (LWP 37922)]
[New Thread 0x2aaaf49af700 (LWP 37923)]
[New Thread 0x2aaaf27ae700 (LWP 37924)]
[New Thread 0x2aaaf05ad700 (LWP 37925)]
[New Thread 0x2aaaee3ac700 (LWP 37926)]
[New Thread 0x2aaaec1ab700 (LWP 37927)]
[New Thread 0x2aaab4d91700 (LWP 37928)]
[New Thread 0x2aaab6f92700 (LWP 37929)]
[New Thread 0x2aaab9193700 (LWP 37930)]
[New Thread 0x2aaabb394700 (LWP 37931)]
[New Thread 0x2aaabd595700 (LWP 37932)]
[New Thread 0x2aaabf796700 (LWP 37933)]
[New Thread 0x2aaac1997700 (LWP 37934)]
[New Thread 0x2aaac3b98700 (LWP 37935)]
[New Thread 0x2aaac5d99700 (LWP 37936)]
[New Thread 0x2aaacc39c700 (LWP 37937)]
[New Thread 0x2aaace59d700 (LWP 37938)]
[New Thread 0x2aaad079e700 (LWP 37939)]
[New Thread 0x2aaad299f700 (LWP 37940)]
[New Thread 0x2aaad4ba0700 (LWP 37941)]
[New Thread 0x2aaad6da1700 (LWP 37942)]
[New Thread 0x2aaad8fa2700 (LWP 37943)]
[New Thread 0x2aaadb1a3700 (LWP 37944)]
[New Thread 0x2aaadd3a4700 (LWP 37945)]
[New Thread 0x2aaadf5a5700 (LWP 37946)]
[New Thread 0x2aaae17a6700 (LWP 37947)]
[New Thread 0x2aaae39a7700 (LWP 37948)]
[New Thread 0x2aaae5ba8700 (LWP 37949)]
[New Thread 0x2aaae7da9700 (LWP 37950)]
[New Thread 0x2aaae9faa700 (LWP 37951)]
[New Thread 0x2aab34361700 (LWP 37952)]
[New Thread 0x2aab34a00700 (LWP 37953)]
[New Thread 0x2aab34c01700 (LWP 37954)]
[New Thread 0x2aab34e02700 (LWP 37955)]
[New Thread 0x2aab35003700 (LWP 37956)]
[New Thread 0x2aab3e563700 (LWP 37957)]
>>> proc = Process(target=readEXR, kwargs={'exrPath': path})
>>> proc.start() # This will never print 3. It will print 1 and 2
>>> proc.join() 1
2

Thread 1 "python3" received signal SIGINT, Interrupt.
0x00002aaaaafed5d3 in select () at ../sysdeps/unix/syscall-template.S:84
84	../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x00002aaaaafed5d3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x00002aaaac102378 in ?? () from /usr/lib/python3.5/lib-dynload/readline.cpython-35m-x86_64-linux-gnu.so
#2  0x0000000000442526 in PyOS_Readline ()
#3  0x0000000000442a87 in ?? ()
#4  0x00000000004feb89 in PyTokenizer_Get ()
#5  0x00000000005ebee2 in PyParser_ParseFileObject ()
#6  0x000000000060e5f4 in PyParser_ASTFromFileObject ()
#7  0x000000000046b7f9 in PyRun_InteractiveOneObject ()
#8  0x000000000046ba48 in PyRun_InteractiveLoopFlags ()
#9  0x000000000046cfa0 in ?? ()
#10 0x00000000004cf2bd in ?? ()
#11 0x00000000004cfeb1 in main ()
Amir (Warrior) added a comment.EditedWed, Apr 11, 5:24 PM

@Brecht Van Lommel (brecht) Just an update: I can get somehow around the forking issue using subprocess and doing one rendering at the time. However, the problem with using subprocess is I have to literally import everything I need from scratch. This makes the rendering much more time-consuming than doing the rendering normally.

As far as I know, there is little difference between spawning a process and using subprocess. They literally both require to import all required packages within the same process. So I wonder, what did you mean by "easier ways"? were you thinking of doing something else?

Sorry if it's too obvious that I'm just trying to convince you to prioritize this :) I hope that happens :)