Page MenuHome

Cycles crashes with Windows GPU Scheduling and 2 or more GPUs [driver bug]
Confirmed, NormalPublicKNOWN ISSUE

Description

System Information
Operating system: Windows-10-10.0.17763-SP0 64 Bits
Graphics card: GeForce GTX 1080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 442.19

Blender Version
Broken: version: 2.82 (sub 7), branch: master, commit date: 2020-02-12 16:20, hash: rB77d23b0bd76f
Worked: (optional)

Short description of error

The problem occurs if you use 2 or more GPUs.
If you use cycles for rendering (in GPU mode), then when the viewport rendered mode is on, pressing f12 causes the program to crash.

This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released.

Event Timeline

Ankit Meel (ankitm) renamed this task from Blender falls when you press f12 to Blender crashes on pressing f12.Mar 18 2020, 4:08 PM
Ankit Meel (ankitm) updated the task description. (Show Details)
Ankit Meel (ankitm) renamed this task from Blender crashes on pressing f12 to Blender crashes on pressing f12 with 2 or more GPUs.Mar 18 2020, 4:17 PM

Please run blender_debug_log.cmd (in blender directory ) and upload the two files it mentions

Ankit Meel (ankitm) changed the task status from Needs Triage to Needs Information from User.Mar 19 2020, 11:38 AM

Ankit Meel (ankitm) changed the task status from Needs Information from User to Confirmed.Mar 19 2020, 11:50 AM

seems problematic

CUDA error: Launch failed in cuGraphicsResourceGetMappedPointer(&buffer, &bytes, pmem.cuPBOresource), line 2000

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

CUDA error: Launch failed in cuModuleGetFunction(&cuFilmConvert, cuModule, "kernel_cuda_convert_to_half_float"), line 1865
Brecht Van Lommel (brecht) changed the task status from Confirmed to Needs Developer to Reproduce.Mar 20 2020, 2:36 PM
Brecht Van Lommel (brecht) changed the subtype of this task from "Report" to "Bug".
ver (verrafa) claimed this task.EditedAug 3 2020, 2:07 PM
ver (verrafa) added a subscriber: ver (verrafa).

#MY SOLUTION#
Hey.
I had the same issue on my win 10 after the latest win update.

I searched for 2 month for this issue. Today i found a setting in win 10 that was causing this issue from one day to another.

I found a setting that was called "GPU Scheduling" in your display settings under graphics options.

Turned it off and after a restart everthing was working fine. My RTX 2070 and my 980TI are working like a charm together.

My output with blender --debug-cycles:

CUDA error: Launch failed in cuGraphicsResourceGetMappedPointer(&buffer, &bytes, pmem.cuPBOresource), line 2000

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

CUDA error: Launch failed in cuModuleGetFunction(&cuFilmConvert, cuModule, "kernel_cuda_convert_to_half_float"), line 1865

Someone had the right idea back in June 30th
https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-190

I don't see why Moving the scheduler to the GPU would make blender crash or time out. While interesting, while would I want to move the scheduler back to my CPU instead?

Nice work around.

This is odd - I can't find that GPU Scheduling setting, or Hardware Accelerated GPU Scheduling mentioned under the Graphics/Display settings screens in Windows 10.. I've looked up step by step guides and I'm missing that part completely!

This is odd - I can't find that GPU Scheduling setting, or Hardware Accelerated GPU Scheduling mentioned under the Graphics/Display settings screens in Windows 10.. I've looked up step by step guides and I'm missing that part completely!

You won't see it unless you are running the latest nvidia driver and Windows 10 Version 2004 at the very least.

Be aware that this windows 10 update may not show up for your device if you have any device in your computer (or drivers) that have not passed the 2004 verification.

That is the reason why it suprised me. I didn't change anything. The latest windows update was the reason.
As i have seen in the debug logs / sources i think the problem is that the multithreading timing is broken in this case.

The error message and the point it crashed was inconsistent. Sometimes at building the bhv sometimes at rendering the first tile.

Looks for me like a multthreading / async issue. The process is started correctly but the end of the multithreading operations seems not to be caught correctly.

Someone had the right idea back in June 30th
https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-190

I don't see why Moving the scheduler to the GPU would make blender crash or time out. While interesting, while would I want to move the scheduler back to my CPU instead?

Nice work around.

Thanks for the devblog article. Added a comment to.

Took me a while to find it out. Included source code reading;) But what bothers me is that microsoft does a major change in the GPU Architecture and does not inform you that this has been activated. Especially when they know that only rescent GPUs are supported.

Also in the comments:
https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-194

It seems this feature requres an onboard hardware scheduler. So my 980 ti does not support it. Thats the reason why:

See:
https://www.reddit.com/r/nvidia/comments/grwe8y/will_900_series_support_hardwareaccelerated_gpu/

So i dont think it is a bug in blender, maybe the error message should be better in this case

actually they do, I had to manually set this setting up because microsoft will not enable it directly, you have to enable it manually.

Also, I am seeing something odd with this now, I have disabled the gpu scheduling, but if you try to enable optix at all it will freeze and crash the exact same way as if the scheduling was enabled in the gpu.

ver (verrafa) added a comment.EditedAug 5 2020, 12:07 PM

Interesting. I dont remember that i activated it, but i cannot preclude that i did that.

With disabled gpu scheduling optix is running fine. Do you have the latest drivers installed? I also deinstalled nvidia completly and removed geforce experience before installing the new driver. As i read in another article. So maybe this helps.

Denoising with nodes and viewport optix is working fine.

just tested it.

Also your 1080 ti should support HAGS. (Hardware accelerated gpu scheduling)

Brecht Van Lommel (brecht) renamed this task from Blender crashes on pressing f12 with 2 or more GPUs to Cycles crashes with Windows GPU Scheduling and 2 or more GPUs.Aug 5 2020, 2:07 PM

@Patrick Mours (pmoursnv), could you look into this perhaps?

Sorry for the late response. I'll check this out as soon as I can!

I am going over old reports. Can you still reproduce this issue in latest build form https://builder.blender.org/download/?

Yes, HAGS (hardware accelerated gpu scheduling) still does not work on blender 2.90.1

HW info:
Windows 10 Pro v2004 19041.630
Nvidia driver 457.30 game ready driver
GTX 1080 Ti and RTX 2060 both showing this issue.

After enabling the feature in windows and a restart windows
take the default scene, switch to cycles, switch to GPU rendering.
The kernel loads, then after some the display driver restarts, killing blender.

This is going to hurt blender's growth in new users, as HAGS will be on more in the future because it benefits gamers.

Forgot to update this, sorry. This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released.

Philipp Oeser (lichtwerk) changed the subtype of this task from "Bug" to "Known Issue".

Forgot to update this, sorry. This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released.

OK, thx getting back.
Will close though (since there is nothing to be done on the blender side) and leave a note in the task description

Philipp Oeser (lichtwerk) renamed this task from Cycles crashes with Windows GPU Scheduling and 2 or more GPUs to Cycles crashes with Windows GPU Scheduling and 2 or more GPUs [driver bug].Nov 30 2020, 11:31 AM
Philipp Oeser (lichtwerk) updated the task description. (Show Details)
Brecht Van Lommel (brecht) reopened this task as Confirmed.Mon, Feb 8, 12:49 PM

Keeping the report open since it is an active issue, and it's easier to find duplicates this way.

@Patrick Mours (pmoursnv), any update on this?

Is still being worked on, turns out to be a more complex problem, so no fix is available yet. I continue to have this on my radar though, so will update when that changes.