Multi-GPU rendering crashing #101758

New Issue

Nathan Huntoon · 2022-10-12T00:11:21+02:00

Nathan Huntoon commented

2022-10-12 00:11:21 +02:00

System Information
Operating system: Ubuntu 22.04
Graphics card: nVidia A10 x 4

Blender Version
Broken: 3.1.1
Worked: N/A

Short description of error
Doing command line rendering with --cycles-device CUDA on a multi-GPU machine results in a core dump and blender crash. Running the same command with a single GPU machine results in successful renders.

Exact steps for others to reproduce the error
I have developed a remote rendering system to perform rendering on cloud infrastructure. The heart of the system is Blender, taking advantage of the command line rendering capability.
I have a multi-GPU system with 4 nVdidia A10 graphics cards, CUDA 11.7 fully setup and the latest drivers. I perform the rendering in a docker image where Blender 3.3.1 is installed at /usr/local/blender. The script that’s running inside calls the command line:

/usr/local/blender/blender -b blendFile.blend -o imgFilename# -f frame -- --cycles-device=CUDA

If I run this docker passing only a single GPU into the image, it runs perfectly and outputs a rendered image. If I pass it multiple GPUs though, the rendering appears to run, but when Blender goes to save the output file I get an error:

pure virtual method called
terminate called without an active exception
Aborted (core dumped)

I’ve verified that other CUDA workloads work with multiple GPUs both on the base machine and inside the docker image. This is the only things that’s breaking.

I ran the command with --debug-cycles enabled and the full output is attached as log.txt
log.txt

**System Information** Operating system: Ubuntu 22.04 Graphics card: nVidia A10 x 4 **Blender Version** Broken: 3.1.1 Worked: N/A **Short description of error** Doing command line rendering with --cycles-device CUDA on a multi-GPU machine results in a core dump and blender crash. Running the same command with a single GPU machine results in successful renders. **Exact steps for others to reproduce the error** I have developed a remote rendering system to perform rendering on cloud infrastructure. The heart of the system is Blender, taking advantage of the command line rendering capability. I have a multi-GPU system with 4 nVdidia A10 graphics cards, CUDA 11.7 fully setup and the latest drivers. I perform the rendering in a docker image where Blender 3.3.1 is installed at /usr/local/blender. The script that’s running inside calls the command line: ``` /usr/local/blender/blender -b blendFile.blend -o imgFilename# -f frame -- --cycles-device=CUDA ``` If I run this docker passing only a single GPU into the image, it runs perfectly and outputs a rendered image. If I pass it multiple GPUs though, the rendering appears to run, but when Blender goes to save the output file I get an error: ``` pure virtual method called terminate called without an active exception Aborted (core dumped) ``` I’ve verified that other CUDA workloads work with multiple GPUs both on the base machine and inside the docker image. This is the only things that’s breaking. I ran the command with --debug-cycles enabled and the full output is attached as log.txt [log.txt](https://archive.blender.org/developer/F13662121/log.txt)

Nathan Huntoon commented

2022-10-12 00:11:21 +02:00

Added subscriber: @Nathan-Huntoon

Pratik Borhade commented

2022-10-12 03:52:03 +02:00

Added subscriber: @PratikPB2123

Pratik Borhade commented

2022-10-12 05:08:26 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Pratik Borhade commented

2022-10-12 05:08:26 +02:00

Hi, thanks for the report. Uploaded Log file doesn't contain any relevant information of the crash.
I'm not sure if this is an issue in Blender or the docker container you're running on remote machine. But rendering in multi-GPU device has some work scheduling issues I guess. see: #89833 (Cycles: multi-device rendering performance) and #95687 (Cycle CPU + GPU hybrid path tracer)
Does this happen with "any" blend file and when Optix is used for GPU acceleration.

Hi, thanks for the report. Uploaded Log file doesn't contain any relevant information of the crash. I'm not sure if this is an issue in Blender or the docker container you're running on remote machine. But rendering in multi-GPU device has some work scheduling issues I guess. see: #89833 (Cycles: multi-device rendering performance) and #95687 (Cycle CPU + GPU hybrid path tracer) Does this happen with "any" blend file and when Optix is used for GPU acceleration.

Pratik Borhade commented

2022-11-11 07:15:41 +01:00

Changed status from 'Needs User Info' to: 'Archived'

Pratik Borhade closed this issue

2022-11-11 07:15:41 +01:00

Pratik Borhade commented

2022-11-11 07:15:41 +01:00

No activity for more than a week. As per the tracker policy we assume the issue is gone and can be closed.

Thanks again for the report. If the problem persists please open a new report with the required information.

No activity for more than a week. As per the tracker policy we assume the issue is gone and can be closed. Thanks again for the report. If the problem persists please open a new report with the required information.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Multi-GPU rendering crashing #101758