Page MenuHome

import bpy initializes CUDA drivers and crashes forked processes for transferring data to GPU
Closed, ArchivedPublic

Description

System Information
OS: All Unix-based OSes
Python version: 3.x
Graphics card: All NVIDIA graphics cards

Blender Version
I guess this should be applicable to any version of Blender with CUDA support

Short description of error
I recently ran into an issue when trying to move my data to GPU using PyTorch's Python API. By reading a couple of threads[1][2][3] I noticed that CUDA is not fork-safe unfortunately. The only way the problem can get resolved is by not calling any cuInit() driver before calling a forked process (it looks like you can do whatever you want in the forked process without causing this issue).

After some trials and errors I realized that doing import bpy is causing this issue for me as I guess the import process is calling the cuInit() function to initialize CUDA drivers somewhere while getting loaded. In PyTorch, they avoided any call that causes CUDA initialization until such a call is actually needed and they called this fix Lazy Init. I'm not guessing what is happening exactly when doing import bpy but I guess this problem can get resolved if the call to cuInit() is only done when people have to change any settings related to the GPU or they explicitly click on GPU rendering in Cycles or basically do anything that clearly points that people are going to use GPU for something (including clicking on EEVEE to start realtime rendering)

Here is a potentially helpful comment from someone in PyTorch's Slack channel who has a better idea of what's happening under the hood:

CUDA, as a complex, multithreaded set of libraries, is totally and permanently incompatible with a fork() not immediately followed by exec(). That means the multiprocessing method fork cannot work, unless the fork is done before CUDA is initialized (by direct or indirect call to cuInit()). Once a process goes multithreaded or initializes CUDA, it’s usually too late.Second, torch.cuda.manual_seed() is lazy, meaning it will not initialize CUDA if it hasn’t been done already. That’s a good thing, for the reasons above.

P.S. I'm not entirely sure but this might also be relevant to this bug or this bug that I reported earlier this year.

Exact steps for others to reproduce the error

python3 -m pip install torch
or
conda install pytorch

Compile Blender (master branch) as Python module with the following CMake flags:

-DCMAKE_INSTALL_PREFIX=/usr/local/lib/python3.6/dist-packages \
-DWITH_PYTHON_INSTALL=OFF \
-DWITH_PYTHON_MODULE=ON \
-DPYTHON_ROOT_DIR=/usr/local \
-DPYTHON_SITE_PACKAGES=/usr/local/lib/python3.6/dist-packages \
-DPYTHON_INCLUDE=/usr/include/python3.6/ \
-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m \
-DPYTHON_LIBRARY=/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so \
-DPYTHON_VERSION=3.6 \
-DWITH_OPENAL=OFF \
-DWITH_OPENCOLORIO=ON \
-DWITH_GAMEENGINE=OFF \
-DWITH_PLAYER=OFF \
-DWITH_INTERNATIONAL=OFF \
-DCMAKE_BUILD_TYPE:STRING=Release

Note that I manually change PYTHON_VERSION_MIN="3.7" in install_deps.sh to `PYTHON_VERSION_MIN="3.6".

Then in Python:

#main.py
from multiprocessing import Process
import torch
import numpy as np

def moveDataToGPU(procID, importBpy=False):
	if importBpy:
		# doing import bpy inside the forked process or outside of it (before moveDataToGPU) 
                #would still cause things to crash
		import bpy

	# Doing the next two lines before moveDataToGPU would cause this function to crash
	# but having Lazy Initialization of CUDA drivers makes it fork-safe 
	# Look at the followings for a better idea of what's going on:
	# https://github.com/pytorch/pytorch/blob/master/torch/cuda/random.py
	# https://github.com/pytorch/pytorch/blob/master/torch/cuda/__init__.py
	torch.cuda.manual_seed(1)
	print(torch.cuda.get_rng_state().sum())


	data = np.random.uniform(0, 1, (5, 5))
	print ('data created for procID ' + str(procID))
	torchData = torch.from_numpy(data)
	torchData = torchData.cuda()
	print ('successfully moved the data to GPU for procID ' + str(procID))
	print ('')

forkedProcess = Process(target=moveDataToGPU, kwargs={'procID': 0, 'importBpy': False})
forkedProcess.start()
forkedProcess.join()

forkedProcess = Process(target=moveDataToGPU, kwargs={'procID': 1, 'importBpy': False})
forkedProcess.start()
forkedProcess.join()

Details

Type
Bug

Event Timeline

Amir (Warrior) updated the task description. (Show Details)
Brecht Van Lommel (brecht) claimed this task.

You can compile with WITH_CYCLES_DEVICE_CUDA=OFF or WITH_CYCLES=OFF.

@Brecht Van Lommel (brecht) Sorry but I forgot to remove that line where I say compiling Blender without CUDA would also be okay. However, this is not the solution I was looking for here (I was looking for such solution on devtalk). Could you please reopen this and look into it?

The bpy module is not an officially supported feature, and you can compile it in such a way to disable CUDA. So I do not consider this a bug to be handled in the tracker.

@Brecht Van Lommel (brecht) But I don't think it's necessarily about the bpy module. As far as I know you can also compile Blender so that it uses the system's Python internally. If there is any goal to make Blender usable by researchers I think this is definitely one thing many researchers who do AI research need in.

@Brecht Van Lommel (brecht) Also, I don't this issue and the import order issue are duplicates ... . I would appreciate if the developers can be more patient with issues that have the sentences like "I compiled Blender as Python module" and don't close/ignore them immediately. In the world of researchers using Blender as Python module is an amazing feature.

This issue is the same as the other one, it's a conflict with both Cycles and PyTorch using CUDA.

If you want to use the unsupported Python module, or if you want to use Blender for AI research, then you are free to do so. But Blender is primarily a tool for artists, and that means we set priorities and can consider issues like this outside of what we spend time supporting.

It looks like the changes here resolve this issue and Blender also has lazy initialization for CUDA now.