Unable to export/import EXR files after doing import bpy
and when using the multiprocessing in Python (Blender freezes) #54461
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#54461
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I am trying to render depth maps and surface Normals using Cycles and store the results in OpenEXR format using nodes. I set up the nodes as follow using Blender's Python API:
The problem is when I do bpy.ops.render.render(write_still=True) Blender freezes and the results are not saved on disk. However, if I change OPEN_EXR_MULTILAYER and OPEN_EXR to PNG and change the color_depth to '16' everything works and I get the rendering results on disk. Note that I do not change any of my rendering engine settings like tile size, resolution etc. Does anyone know why Blender freezes and the rendering results are not on disk when using OpenEXR?
The strange thing is everything works normally when running Blender with GUI and setup the nodes and rendering engine the same way. It would be also useful if someone can try this and let me know if things work fine on their side.
I'm not sure if this has something to do with the problem that I'm facing but I have compiled Blender manually and import it as a module in the Python installed on my machine. Here are the CMake settings I used to compile Blender 2.79b from source:
Update: I experience the same behavior with Blender 2.78 and 2.79, compiled with the same exact settings above.
Added subscriber: @AmirS
Added subscriber: @brecht
I can't reproduce this issue. I tried putting the above code in a
test.py
script withbpy.ops.render.render(write_still=True)
at the end, and then running./blender -b -P test.py
.--factory-startup
to check if any user preferences are influencing this.gdb --args ./blender -b -P test.py
, then typerun
, press ctrl+C while it hangs and typethread apply bt all
. The output of that should give a clue about where it's hanging, you can attach that here.@brecht Sorry I should have been more clear. The problem is I am compiling Blender as Python module in Ubuntu 16.04. I don't think it is possible to import the Blender module given those arguments you mentioned. Could you please double-check this as well?
@brecht Sorry for the missing information. I just realized that the problem is being caused if I call the rendering function using
multiprocessing.Process
and updated the script in my bug report that shows how exactly I do rendering in my project, with all other irrelevant details removed. The strange thing is I can easily render my meshes if I do not want the rendering result to be stored in EXR files. So, if I replaceOPEN_EXR
withPNG
and replace32
with8/16
I will definitely get the rendering results stored on disk. This does not happen when trying to store the result in OpenEXR format and I did not know that themultiprocessing
package might be causing it. What should I do now? Why I cannot storeEXR
files when usingmultiprocessing.Process
?Also, another person has issues storing multi-channel EXR files as he/she mentioned here . Maybe you can try fixing both my issue and his/her? Or I can post another bug report for that.
Can you try using the
spawn
start method?https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
It would not surprise me if there is some part of Blender or the OpenEXR library code that does not work with forking. I'm also not inclined to spend much time investigating that, I don't think this is something we can guarantee to work well without a lot of effort verifying lots of code.
If there's another bug with OpenEXR saving unrelated to multiprocessing, another bug report can be created for that,
@brecht I tried with
spawn
but I keep getting weird errors and things do not work. It seems that I have to somehow make the processspawn
able and I don't know how to do that precisely since changing my pipeline is not as straight-forward. I hope you can find a solution for fixing the issue if it's coming from Blender side. It's so annoying for me since I have to do millions of renderings and I didn't expect to do it sequentially :(@brecht You said it wouldn't surprise you if some parts of Blender of OpenEXr does not work well with forking. So I wonder, which one do you think is more likely to be the case given that I can store my renderings in other formats? Would you think it's more likely that OpenEXR is the one to blame here?
@brecht I also posted an issue on OpenEXR's GitHub repo (link here ). Hopefully they can also make some clarification from their side.
Unable to export rendering results in OpenEXR format using Python (Blender freezes)to Unable to export renderings in OpenEXR when using the multiprocessing in Python (Blender freezes)I would not make a bug report to the OpenEXR project, it's not clear at all there is a bug in their library and I wouldn't expect them to investigate a vague report like that. It's a possibility but too early to tell.
To work around the problem you can just launch multiple Python scripts that each do part of the work, or import the Blender module after calling
multiprocessing.Process
so that it is not forked.@brecht I'm not sure if this could be very relevant but I used OpenEXR Python bindings package to do some tests. Here's what I found out: if I have a function that simply loads an
exr
file and separates its channels into (R, G,B) as shown below and I execute this function viamultiprocessing.Process
everything works fine. However, when I just doimport bpy
and execute the same function viamultiprocessing.Process
things do not work and Python freezes. I would say this is a bug in Blender. Could you please look into this? I would really appreciate it.Unable to export renderings in OpenEXR when using the multiprocessing in Python (Blender freezes)to Unable to export/import EXR files after doing `import bpy` and when using the multiprocessing in Python (Blender freezes)This is a very low priority issue for me, since it should be quite easy to work around in the ways I suggested and the python module is not something we officially release. Given that there are many more important bugs to solve I'm not going to spend time on this now.
@brecht I hope you will prioritize this in the future. Many people like me need to use Blender as a module for their research work. Anyways, I used
gdb
to get you the back trace after Python freezes. Here's what I get (I had toCTRL+C
at the end):@brecht Just an update: I can get somehow around the forking issue using
subprocess
and doing one rendering at the time. However, the problem with usingsubprocess
is I have to literally import everything I need from scratch. This makes the rendering much more time-consuming than doing the rendering normally.As far as I know, there is little difference between
spawn
ing a process and usingsubprocess
. They literally both require to import all required packages within the same process. So I wonder, what did you mean by "easier ways"? were you thinking of doing something else?Sorry if it's too obvious that I'm just trying to convince you to prioritize this :) I hope that happens :)
I hope this will have some higher priority soon :)