Cycles crashes with Windows GPU Scheduling and 2 or more GPUs [driver bug] #74883

New Issue

Svyatoslav · 2020-03-18T15:24:44+01:00

Svyatoslav commented

2020-03-18 15:24:44 +01:00

System Information
Operating system: Windows-10-10.0.17763-SP0 64 Bits
Graphics card: GeForce GTX 1080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 442.19

Blender Version
Broken: version: 2.82 (sub 7), branch: master, commit date: 2020-02-12 16:20, hash: 77d23b0bd7
Worked: (optional)

Short description of error

The problem occurs if you use 2 or more GPUs.
If you use cycles for rendering (in GPU mode), then when the viewport rendered mode is on, pressing f12 causes the program to crash.

In #74883#1064903, @pmoursnv wrote:
This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released.

**System Information** Operating system: Windows-10-10.0.17763-SP0 64 Bits Graphics card: GeForce GTX 1080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 442.19 **Blender Version** Broken: version: 2.82 (sub 7), branch: master, commit date: 2020-02-12 16:20, hash: `77d23b0bd7` Worked: (optional) **Short description of error** The problem occurs if you use 2 or more GPUs. If you use cycles for rendering (in GPU mode), then when the viewport rendered mode is on, pressing f12 causes the program to crash. > In #74883#1064903, @pmoursnv wrote: > This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released.

Svyatoslav commented

2020-03-18 15:24:44 +01:00

Added subscriber: @Granduca

blender-admin commented

2020-03-18 15:24:44 +01:00

#87121 was marked as duplicate of this issue

blender-admin commented

2020-03-18 15:24:44 +01:00

#85317 was marked as duplicate of this issue

blender-admin commented

2020-03-18 15:24:44 +01:00

#85421 was marked as duplicate of this issue

Ankit Meel changed title from ~~Blender falls when you press f12~~ to Blender crashes on pressing f12

2020-03-18 16:08:15 +01:00

Ankit Meel changed title from ~~Blender crashes on pressing f12~~ to Blender crashes on pressing f12 with 2 or more GPUs

2020-03-18 16:17:23 +01:00

Ankit Meel commented

2020-03-19 11:37:53 +01:00

Added subscriber: @ankitm

Ankit Meel commented

2020-03-19 11:37:53 +01:00

Please run blender_debug_log.cmd (in blender directory ) and upload the two files it mentions

Ankit Meel commented

2020-03-19 11:38:02 +01:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Svyatoslav commented

2020-03-19 11:42:55 +01:00

blender_system_info.txt

blender_debug_output.txt

[blender_system_info.txt](https://archive.blender.org/developer/F8415553/blender_system_info.txt) [blender_debug_output.txt](https://archive.blender.org/developer/F8415552/blender_debug_output.txt)

Ankit Meel commented

2020-03-19 11:50:29 +01:00

Changed status from 'Needs User Info' to: 'Confirmed'

Ankit Meel commented

2020-03-19 11:50:29 +01:00

seems problematic

CUDA error: Launch failed in cuGraphicsResourceGetMappedPointer(&buffer, &bytes, pmem.cuPBOresource), line 2000

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

CUDA error: Launch failed in cuModuleGetFunction(&cuFilmConvert, cuModule, "kernel_cuda_convert_to_half_float"), line 1865

seems problematic ``` CUDA error: Launch failed in cuGraphicsResourceGetMappedPointer(&buffer, &bytes, pmem.cuPBOresource), line 2000 Refer to the Cycles GPU rendering documentation for possible solutions: https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html CUDA error: Launch failed in cuModuleGetFunction(&cuFilmConvert, cuModule, "kernel_cuda_convert_to_half_float"), line 1865 ```

Brecht Van Lommel commented

2020-03-20 14:36:41 +01:00

Changed status from 'Confirmed' to: 'Needs Developer To Reproduce'

ver commented

2020-08-03 14:07:12 +02:00

Added subscriber: @verrafa

ver self-assigned this 2020-08-03 14:07:12 +02:00

ver commented

2020-08-03 14:07:12 +02:00

MY SOLUTION

Hey.
I had the same issue on my win 10 after the latest win update.

I searched for 2 month for this issue. Today i found a setting in win 10 that was causing this issue from one day to another.

I found a setting that was called "GPU Scheduling" in your display settings under graphics options.

Turned it off and after a restart everthing was working fine. My RTX 2070 and my 980TI are working like a charm together.

My output with blender --debug-cycles:

CUDA error: Launch failed in cuGraphicsResourceGetMappedPointer(&buffer, &bytes, pmem.cuPBOresource), line 2000

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

CUDA error: Launch failed in cuModuleGetFunction(&cuFilmConvert, cuModule, "kernel_cuda_convert_to_half_float"), line 1865

# MY SOLUTION Hey. I had the same issue on my win 10 after the latest win update. I searched for 2 month for this issue. Today i found a setting in win 10 that was causing this issue from one day to another. I found a setting that was called "GPU Scheduling" in your display settings under graphics options. Turned it off and after a restart everthing was working fine. My RTX 2070 and my 980TI are working like a charm together. My output with blender --debug-cycles: CUDA error: Launch failed in cuGraphicsResourceGetMappedPointer(&buffer, &bytes, pmem.cuPBOresource), line 2000 Refer to the Cycles GPU rendering documentation for possible solutions: https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html CUDA error: Launch failed in cuModuleGetFunction(&cuFilmConvert, cuModule, "kernel_cuda_convert_to_half_float"), line 1865

Cristobal Escamilla commented

2020-08-04 04:27:25 +02:00

Added subscriber: @Kryztoval

Cristobal Escamilla commented

2020-08-04 04:27:25 +02:00

Someone had the right idea back in June 30th
https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-190

I don't see why Moving the scheduler to the GPU would make blender crash or time out. While interesting, while would I want to move the scheduler back to my CPU instead?

Nice work around.

Someone had the right idea back in June 30th https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-190 I don't see why Moving the scheduler to the GPU would make blender crash or time out. While interesting, while would I want to move the scheduler back to my CPU instead? Nice work around.

Peter commented

2020-08-04 11:20:29 +02:00

Added subscriber: @highonsnow

Peter commented

2020-08-04 11:20:29 +02:00

This is odd - I can't find that GPU Scheduling setting, or Hardware Accelerated GPU Scheduling mentioned under the Graphics/Display settings screens in Windows 10.. I've looked up step by step guides and I'm missing that part completely!

Cristobal Escamilla commented

2020-08-05 06:44:38 +02:00

In #74883#990175, @highonsnow wrote:
This is odd - I can't find that GPU Scheduling setting, or Hardware Accelerated GPU Scheduling mentioned under the Graphics/Display settings screens in Windows 10.. I've looked up step by step guides and I'm missing that part completely!

You won't see it unless you are running the latest nvidia driver and Windows 10 Version 2004 at the very least.

Be aware that this windows 10 update may not show up for your device if you have any device in your computer (or drivers) that have not passed the 2004 verification.

> In #74883#990175, @highonsnow wrote: > This is odd - I can't find that GPU Scheduling setting, or Hardware Accelerated GPU Scheduling mentioned under the Graphics/Display settings screens in Windows 10.. I've looked up step by step guides and I'm missing that part completely! You won't see it unless you are running the latest nvidia driver and Windows 10 Version 2004 at the very least. Be aware that this windows 10 update may not show up for your device if you have any device in your computer (or drivers) that have not passed the 2004 verification.

ver commented

2020-08-05 10:18:37 +02:00

That is the reason why it suprised me. I didn't change anything. The latest windows update was the reason.
As i have seen in the debug logs / sources i think the problem is that the multithreading timing is broken in this case.

The error message and the point it crashed was inconsistent. Sometimes at building the bhv sometimes at rendering the first tile.

Looks for me like a multthreading / async issue. The process is started correctly but the end of the multithreading operations seems not to be caught correctly.

That is the reason why it suprised me. I didn't change anything. The latest windows update was the reason. As i have seen in the debug logs / sources i think the problem is that the multithreading timing is broken in this case. The error message and the point it crashed was inconsistent. Sometimes at building the bhv sometimes at rendering the first tile. Looks for me like a multthreading / async issue. The process is started correctly but the end of the multithreading operations seems not to be caught correctly.

ver commented

2020-08-05 10:30:11 +02:00

In #74883#989912, @Kryztoval wrote:
Someone had the right idea back in June 30th
https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-190

I don't see why Moving the scheduler to the GPU would make blender crash or time out. While interesting, while would I want to move the scheduler back to my CPU instead?

Nice work around.

Thanks for the devblog article. Added a comment to.

Took me a while to find it out. Included source code reading;) But what bothers me is that microsoft does a major change in the GPU Architecture and does not inform you that this has been activated. Especially when they know that only rescent GPUs are supported.

> In #74883#989912, @Kryztoval wrote: > Someone had the right idea back in June 30th > https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-190 > > I don't see why Moving the scheduler to the GPU would make blender crash or time out. While interesting, while would I want to move the scheduler back to my CPU instead? > > Nice work around. Thanks for the devblog article. Added a comment to. Took me a while to find it out. Included source code reading;) But what bothers me is that microsoft does a major change in the GPU Architecture and does not inform you that this has been activated. Especially when they know that only rescent GPUs are supported.

ver commented

2020-08-05 10:37:00 +02:00

Also in the comments:
https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-194

It seems this feature requres an onboard hardware scheduler. So my 980 ti does not support it. Thats the reason why:

See:
https://www.reddit.com/r/nvidia/comments/grwe8y/will_900_series_support_hardwareaccelerated_gpu/

So i dont think it is a bug in blender, maybe the error message should be better in this case

Also in the comments: https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/#comment-194 It seems this feature requres an onboard hardware scheduler. So my 980 ti does not support it. Thats the reason why: See: https://www.reddit.com/r/nvidia/comments/grwe8y/will_900_series_support_hardwareaccelerated_gpu/ So i dont think it is a bug in blender, maybe the error message should be better in this case

Cristobal Escamilla commented

2020-08-05 11:20:37 +02:00

actually they do, I had to manually set this setting up because microsoft will not enable it directly, you have to enable it manually.

Also, I am seeing something odd with this now, I have disabled the gpu scheduling, but if you try to enable optix at all it will freeze and crash the exact same way as if the scheduling was enabled in the gpu.

actually they do, I had to manually set this setting up because microsoft will not enable it directly, you have to enable it manually. Also, I am seeing something odd with this now, I have disabled the gpu scheduling, but if you try to enable optix at all it will freeze and crash the exact same way as if the scheduling was enabled in the gpu.

ver commented

2020-08-05 12:07:24 +02:00

Interesting. I dont remember that i activated it, but i cannot preclude that i did that.

With disabled gpu scheduling optix is running fine. Do you have the latest drivers installed? I also deinstalled nvidia completly and removed geforce experience before installing the new driver. As i read in another article. So maybe this helps.

Denoising with nodes and viewport optix is working fine.

just tested it.

Interesting. I dont remember that i activated it, but i cannot preclude that i did that. With disabled gpu scheduling optix is running fine. Do you have the latest drivers installed? I also deinstalled nvidia completly and removed geforce experience before installing the new driver. As i read in another article. So maybe this helps. Denoising with nodes and viewport optix is working fine. just tested it.

ver commented

2020-08-05 12:10:32 +02:00

Also your 1080 ti should support HAGS. (Hardware accelerated gpu scheduling)

Brecht Van Lommel changed title from ~~Blender crashes on pressing f12 with 2 or more GPUs~~ to Cycles crashes with Windows GPU Scheduling and 2 or more GPUs

2020-08-05 14:07:29 +02:00

Brecht Van Lommel commented

2020-08-05 14:07:54 +02:00

Added subscribers: @pmoursnv, @brecht

Brecht Van Lommel commented

2020-08-05 14:07:54 +02:00

@pmoursnv, could you look into this perhaps?

Patrick Mours commented

2020-08-17 13:15:09 +02:00

Sorry for the late response. I'll check this out as soon as I can!

Richard Antalik commented

2020-11-26 18:16:35 +01:00

Added subscriber: @iss

Richard Antalik commented

2020-11-26 18:16:35 +01:00

I am going over old reports. Can you still reproduce this issue in latest build form https://builder.blender.org/download/?

JvG commented

2020-11-29 21:16:41 +01:00

Added subscriber: @HexerPsy

JvG commented

2020-11-29 21:16:41 +01:00

Yes, HAGS (hardware accelerated gpu scheduling) still does not work on blender 2.90.1

HW info:
Windows 10 Pro v2004 19041.630
Nvidia driver 457.30 game ready driver
GTX 1080 Ti and RTX 2060 both showing this issue.

After enabling the feature in windows and a restart windows
take the default scene, switch to cycles, switch to GPU rendering.
The kernel loads, then after some the display driver restarts, killing blender.

This is going to hurt blender's growth in new users, as HAGS will be on more in the future because it benefits gamers.

Yes, HAGS (hardware accelerated gpu scheduling) still does not work on blender 2.90.1 HW info: Windows 10 Pro v2004 19041.630 Nvidia driver 457.30 game ready driver GTX 1080 Ti and RTX 2060 both showing this issue. After enabling the feature in windows and a restart windows take the default scene, switch to cycles, switch to GPU rendering. The kernel loads, then after some the display driver restarts, killing blender. This is going to hurt blender's growth in new users, as HAGS will be on more in the future because it benefits gamers.

Patrick Mours commented

2020-11-30 11:22:42 +01:00

Forgot to update this, sorry. This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released.

Philipp Oeser commented

2020-11-30 11:30:53 +01:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2020-11-30 11:30:53 +01:00

Changed status from 'Needs Developer To Reproduce' to: 'Archived'

Philipp Oeser closed this issue

2020-11-30 11:30:53 +01:00

Philipp Oeser commented

2020-11-30 11:30:53 +01:00

In #74883#1064903, @pmoursnv wrote:
Forgot to update this, sorry. This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released.

OK, thx getting back.
Will close though (since there is nothing to be done on the blender side) and leave a note in the task description

> In #74883#1064903, @pmoursnv wrote: > Forgot to update this, sorry. This is a driver bug and not a Blender issue. I'll post an update once a new driver with a fix was released. OK, thx getting back. Will close though (since there is nothing to be done on the blender side) and leave a note in the task description

Philipp Oeser changed title from ~~Cycles crashes with Windows GPU Scheduling and 2 or more GPUs~~ to Cycles crashes with Windows GPU Scheduling and 2 or more GPUs [driver bug]

2020-11-30 11:31:28 +01:00

Brecht Van Lommel commented

2021-02-08 12:47:06 +01:00

Added subscribers: @Renderbicks, @deadpin

Brecht Van Lommel commented

2021-02-08 12:48:28 +01:00

Added subscribers: @chelaru, @kaek

Brecht Van Lommel commented

2021-02-08 12:49:44 +01:00

Changed status from 'Archived' to: 'Confirmed'

Brecht Van Lommel reopened this issue

2021-02-08 12:49:44 +01:00

Brecht Van Lommel commented

2021-02-08 12:49:44 +01:00

Keeping the report open since it is an active issue, and it's easier to find duplicates this way.

@pmoursnv, any update on this?

Keeping the report open since it is an active issue, and it's easier to find duplicates this way. @pmoursnv, any update on this?

Patrick Mours commented

2021-02-08 13:51:12 +01:00

Is still being worked on, turns out to be a more complex problem, so no fix is available yet. I continue to have this on my radar though, so will update when that changes.

David Delgado commented

2021-03-30 00:44:09 +02:00

Added subscriber: @DND

David Delgado commented

2021-03-30 00:44:09 +02:00

@brecht Thank you for keeping the one thread open... this problem has been here for quite a while now, i really hope theres a fix..

Michael Klein commented

2021-03-30 03:24:25 +02:00

In #74883#1138146, @DND wrote:
@brecht Thank you for keeping the one thread open... this problem has been here for quite a while now, i really hope theres a fix..

It‘s fixed in 2.93 Alpha but not in 2.92 yet.

> In #74883#1138146, @DND wrote: > @brecht Thank you for keeping the one thread open... this problem has been here for quite a while now, i really hope theres a fix.. It‘s fixed in 2.93 Alpha but not in 2.92 yet.

Chelaru commented

2021-03-30 18:04:45 +02:00

This is my initial bug : https://developer.blender.org/T85317 , the issue is present on 2.93 alpha with the error :
Illegal address in cuCtxSynchronize() (device_cuda_impl.cpp:2049)Illegal address in cuGraphicsMapResources(1,$&pmem.cuPBOresource,0) (device_cuda_impl.cpp:2135)

This is my initial bug : https://developer.blender.org/T85317 , the issue is present on 2.93 alpha with the error : Illegal address in cuCtxSynchronize() (device_cuda_impl.cpp:2049)Illegal address in cuGraphicsMapResources(1,$&pmem.cuPBOresource,0) (device_cuda_impl.cpp:2135)

Michael Klein commented

2021-03-30 18:18:42 +02:00

In #74883#1138570, @chelaru wrote:
This is my initial bug : https://developer.blender.org/T85317 , the issue is present on 2.93 alpha with the error :
Illegal address in cuCtxSynchronize() (device_cuda_impl.cpp:2049)Illegal address in cuGraphicsMapResources(1,$&pmem.cuPBOresource,0) (device_cuda_impl.cpp:2135)

My apologies. It was confusion with another bug.

> In #74883#1138570, @chelaru wrote: > This is my initial bug : https://developer.blender.org/T85317 , the issue is present on 2.93 alpha with the error : > Illegal address in cuCtxSynchronize() (device_cuda_impl.cpp:2049)Illegal address in cuGraphicsMapResources(1,$&pmem.cuPBOresource,0) (device_cuda_impl.cpp:2135) My apologies. It was confusion with another bug.

Cristobal Escamilla commented

2021-03-30 19:20:17 +02:00

Removed subscriber: @Kryztoval

James commented

2021-04-11 01:43:56 +02:00

Added subscriber: @stuckpixel

James commented

2021-04-11 01:43:56 +02:00

I also had spent a few hours trying to figure out why cycles was hanging when I tried to use both my 1080ti GPUs. Just before I found this thread I did stumble into turning off my hardware scheduling and things started working.

One thing I noticed is that when scheduling was on. It only worked for the GPU that was connected to a monitor. My other GPU was headless and hung every time when set to be active. I switched which GPU had the monitors on it and the issue moved over the card that was now headless.

My main app is C4D/Redshift and it was not having noticeable issues with hardware scheduling and 2 GPUs. So I have had the setting on for a while. I thought Blender was working fine with it on a few months ago. I will try keeping it off until there is a fix in blender. At least turning it off avoids the issue.

I also had spent a few hours trying to figure out why cycles was hanging when I tried to use both my 1080ti GPUs. Just before I found this thread I did stumble into turning off my hardware scheduling and things started working. One thing I noticed is that when scheduling was on. It only worked for the GPU that was connected to a monitor. My other GPU was headless and hung every time when set to be active. I switched which GPU had the monitors on it and the issue moved over the card that was now headless. My main app is C4D/Redshift and it was not having noticeable issues with hardware scheduling and 2 GPUs. So I have had the setting on for a while. I thought Blender was working fine with it on a few months ago. I will try keeping it off until there is a fix in blender. At least turning it off avoids the issue.

Jesse Yurkovich commented

2021-06-24 10:33:46 +02:00

Added subscriber: @Cjhosken

Patrick Mours commented

2021-06-24 12:31:12 +02:00

Changed status from 'Confirmed' to: 'Resolved'

Patrick Mours closed this issue

2021-06-24 12:31:12 +02:00

Patrick Mours commented

2021-06-24 12:31:12 +02:00

Fixed starting with r470 drivers (471.11 was released 06/23).

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Cycles crashes with Windows GPU Scheduling and 2 or more GPUs [driver bug] #74883

MY SOLUTION