I posted a devtalk thread regarding this as well --> https://devtalk.blender.org/t/is-it-possible-to-store-keep-the-rendering-result-in-memory-only-and-avoid-doing-i-o/11852

@Shane Ambler (sambler) I haven't tried this in 2.8x. Did you try it with both 2.79x and 2.8x?

@Jeroen Bakker (jbakker) Does this fix have anything to do with this issue?

@Brecht Van Lommel (brecht) But can you also try this on your side as well?

@Brecht Van Lommel (brecht) I just compiled Blender from source as Python module, with CUDA enabled, and noticed that doing import bpy will still initialize CUDA without doing anything else. Are you sure this should delay CUDA initialization?

@Sergey Sharybin (sergey) Would you also do this for 2.7?

@Brecht Van Lommel (brecht) Is it possible to have this in Blender 2.8 as well?

It looks like the changes here resolve this issue and Blender also has lazy initialization for CUDA now.

@Brecht Van Lommel (brecht) @Sergey Sharybin (sergey) Do the changes here resolve this as well --> https://developer.blender.org/T57813 ?

@Brecht Van Lommel (brecht) Also, I don't this issue and the import order issue are duplicates ... . I would appreciate if the developers can be more patient with issues that have the sentences like "I compiled Blender as Python module" and don't close/ignore them immediately. In the world of researchers using Blender as Python module is an amazing feature.

@Brecht Van Lommel (brecht) But I don't think it's necessarily about the bpy module. As far as I know you can also compile Blender so that it uses the system's Python internally. If there is any goal to make Blender usable by researchers I think this is definitely one thing many researchers who do AI research need in.

@Brecht Van Lommel (brecht) Sorry but I forgot to remove that line where I say compiling Blender without CUDA would also be okay. However, this is not the solution I was looking for here (I was looking for such solution on devtalk). Could you please reopen this and look into it?

Any updates on this?

@Brecht Van Lommel (brecht) I think I fixed it actually but this is not a permanent solution and I'm not sure if Armory is going to work properly this way.

@Brecht Van Lommel (brecht) No it shows me the following error although I ran the code above to install it.

@Brecht Van Lommel (brecht) I think it's totally understandable to assume that people who compile Blender as Python module would ONLY need to use the "factory setting" functionalities, but it's not a good idea not to provide an option for others to be able to install addons :(

@Bastien Montagne (mont29) I don't have an issue running Blender with the -b flag but the reason that I'm compiling Blender as Python module is that I have many other packages that I'm using in my project and things are integrated. I do import pytorch, import cv2 etc and it's not possible to disentangle things. On the other hand, I don't think it is possible to compile a heavy code base like PyTorch and add its module into the internal Python that Blender comes with.

@Campbell Barton (campbellbarton) @Brecht Van Lommel (brecht) But it doesn't make sense not to be able to install addons if you're using the Python module. The Python module should be Blender with no GUI in my opinion and there shouldn't be any difference. Would you agree with that?
But still, is there a way to manually load a userpref.blend file with the Python module? How can one install addons when using the Python module?

I hope this will have some higher priority soon :)

This issue still persists ...

@Bastien Montagne (mont29) Is this bug/issue relevant to what I had posted earlier here?

@Philipp Oeser (lichtwerk) @Jeroen Bakker (jbakker) Thanks for investigating this more. I also remember at some point I was accessing the pixels directly through "Viewer Node" image block and stored them on disk. What I realized however was the pixels were sort of rotated. I remember the depth map rendering that I got was shown 90 degrees rotated. If possible, please take a look at this too.

@Brecht Van Lommel (brecht) Just an update: I can get somehow around the forking issue using subprocess and doing one rendering at the time. However, the problem with using subprocess is I have to literally import everything I need from scratch. This makes the rendering much more time-consuming than doing the rendering normally.

This issue on PyTorch GitHub page might be helpful for the developers: https://github.com/pytorch/pytorch/issues/6194

@Brecht Van Lommel (brecht) I hope you will prioritize this in the future. Many people like me need to use Blender as a module for their research work. Anyways, I used gdb to get you the back trace after Python freezes. Here's what I get (I had to CTRL+C at the end):

@Brecht Van Lommel (brecht) I'm not sure if this could be very relevant but I used OpenEXR Python bindings package to do some tests. Here's what I found out: if I have a function that simply loads an exr file and separates its channels into (R, G,B) as shown below and I execute this function via multiprocessing.Process everything works fine. However, when I just do import bpy and execute the same function via multiprocessing.Process things do not work and Python freezes. I would say this is a bug in Blender. Could you please look into this? I would really appreciate it.

@Brecht Van Lommel (brecht) I also posted an issue on OpenEXR's GitHub repo (link here). Hopefully they can also make some clarification from their side.

@Brecht Van Lommel (brecht) You said it wouldn't surprise you if some parts of Blender of OpenEXr does not work well with forking. So I wonder, which one do you think is more likely to be the case given that I can store my renderings in other formats? Would you think it's more likely that OpenEXR is the one to blame here?

@Brecht Van Lommel (brecht) I tried with spawn but I keep getting weird errors and things do not work. It seems that I have to somehow make the process spawnable and I don't know how to do that precisely since changing my pipeline is not as straight-forward. I hope you can find a solution for fixing the issue if it's coming from Blender side. It's so annoying for me since I have to do millions of renderings and I didn't expect to do it sequentially :(

@Brecht Van Lommel (brecht) Sorry for the missing information. I just realized that the problem is being caused if I call the rendering function using multiprocessing.Process and updated the script in my bug report that shows how exactly I do rendering in my project, with all other irrelevant details removed. The strange thing is I can easily render my meshes if I do not want the rendering result to be stored in EXR files. So, if I replace OPEN_EXR with PNG and replace 32 with 8/16 I will definitely get the rendering results stored on disk. This does not happen when trying to store the result in OpenEXR format and I did not know that the multiprocessing package might be causing it. What should I do now? Why I cannot store EXR files when using multiprocessing.Process?

@Brecht Van Lommel (brecht) Sorry I should have been more clear. The problem is I am compiling Blender as Python module in Ubuntu 16.04. I don't think it is possible to import the Blender module given those arguments you mentioned. Could you please double-check this as well?