Cycles CUDA error: Illegal address in cuCtxSynchronize() #76526

Closed
opened 2020-05-07 23:54:27 +02:00 by Alexander Majewski · 23 comments

System Information
Operating system: Windows 10
Graphics card: MSI GeForce GTX 1650 Ventus XS OC 4GB GDDR5

Blender Version
Broken: v 2.82a & v 2.80
Worked: n/a

Short description of error
I just bought a brand new GPU and installed it in my old PC (2014) to turn it into a rendering machine. When rendering with Cycles using GPU, it throws an error

CUDA error: Illegal address in cuCtxSynchronize() line 1835

The console log says:

CUDA error: illegal address in mem_alloc_result, line 855
CUDA error: illegal address in cuModuleGetGlobal(&cumem, &cubytes, cuModule, bind_name.c_str()), line 1103
CUDA error: illegal address in cuMemcpyHtoD(cumem, (void *)&ptr, cubytes), line 1113
  • I tried setting TdrDelay and TdrDdiDelay to 60, didn't help.
  • Error happens even in a super simple test scene with like 30 cubes and a few area lights.
  • Error happens even in an archival 2.80 blender version I had lying around
  • Error appears at 256 tile size as well as 64x64.
  • I reinstalled the Nvidia drivers with GeForce Experience and performed a "clean install".
  • This happens with nvidia's game-ready drivers as well as studio drivers.
  • The error sometimes appears right off the bat, sometimes in the middle of a render.
  • LuxCore also crashes when approaching 800~ samples.

Nothing has ever been overclocked in my PC.

Here's a gpu debug file (not sure if it helps)
blender_debug_output.txt
blender_system_info.txt

The render goes really well before the error. Doesn't freeze or anything. When it happens, the currently rendered tile starts going semi-transparent.
Please help!

edit: additional info -- in Performance tab of task manager GPU seems to be at 2-5% activity, while CPU is at 50%. The CPU is AMD FX(tm)-6300 six-core processor.

**System Information** Operating system: Windows 10 Graphics card: MSI GeForce GTX 1650 Ventus XS OC 4GB GDDR5 **Blender Version** Broken: v 2.82a & v 2.80 Worked: n/a **Short description of error** I just bought a brand new GPU and installed it in my old PC (2014) to turn it into a rendering machine. When rendering with Cycles using GPU, it throws an error ``` CUDA error: Illegal address in cuCtxSynchronize() line 1835 ``` The console log says: ``` CUDA error: illegal address in mem_alloc_result, line 855 CUDA error: illegal address in cuModuleGetGlobal(&cumem, &cubytes, cuModule, bind_name.c_str()), line 1103 CUDA error: illegal address in cuMemcpyHtoD(cumem, (void *)&ptr, cubytes), line 1113 ``` - I tried setting TdrDelay and TdrDdiDelay to 60, didn't help. - Error happens even in a super simple test scene with like 30 cubes and a few area lights. - Error happens even in an archival 2.80 blender version I had lying around - Error appears at 256 tile size as well as 64x64. - I reinstalled the Nvidia drivers with GeForce Experience and performed a "clean install". - This happens with nvidia's game-ready drivers as well as studio drivers. - The error sometimes appears right off the bat, sometimes in the middle of a render. - LuxCore also crashes when approaching 800~ samples. # Nothing has ever been overclocked in my PC. Here's a gpu debug file (not sure if it helps) [blender_debug_output.txt](https://archive.blender.org/developer/F8519125/blender_debug_output.txt) [blender_system_info.txt](https://archive.blender.org/developer/F8519132/blender_system_info.txt) The render goes really well before the error. Doesn't freeze or anything. When it happens, the currently rendered tile starts going semi-transparent. Please help! edit: additional info -- in Performance tab of task manager GPU seems to be at 2-5% activity, while CPU is at 50%. The CPU is AMD FX(tm)-6300 six-core processor.

Added subscriber: @xander91

Added subscriber: @xander91
Member

Added subscriber: @Alaska

Added subscriber: @Alaska
Member

Two things I can suggest:

  1. The task manager reporting 2-5% usage is a issue with task manager. Task manager reports 3D performance while Blender uses CUDA and/or compute. You can change some settings in task manager to report utilization in the CUDA or compute of your GPU if you want.
  2. When replacing your graphics card, you should do a clean install of Windows to avoid issues.

Now, I understand if you don't want to do a clean install of Windows, and maybe some other people on here can suggest other things to try. But this is the best way to eliminate a software issue.

Two things I can suggest: 1. The task manager reporting 2-5% usage is a issue with task manager. Task manager reports 3D performance while Blender uses CUDA and/or compute. You can change some settings in task manager to report utilization in the CUDA or compute of your GPU if you want. 2. When replacing your graphics card, you should do a clean install of Windows to avoid issues. Now, I understand if you don't want to do a clean install of Windows, and maybe some other people on here can suggest other things to try. But this is the best way to eliminate a software issue.

Added subscriber: @brecht

Added subscriber: @brecht

It's also possible to test the latest daily build in case that fixes the issue:
https://builder.blender.org/download/

For a useful debug log, run blender_debug_log.cmd and then make it produce the CUDA error.

It's also possible to test the latest daily build in case that fixes the issue: https://builder.blender.org/download/ For a useful debug log, run `blender_debug_log.cmd` and then make it produce the CUDA error.

I don't mind reinstalling it as long as it really seems like the most probable cause & solution. My Windows is very unkept and twisted. I think the GPU indeed actually works, because the renders are blazing fast compared to that old CPU.

I'll reinstall it tomorrow and let you know if it resolved the issue. Thank you!

I don't mind reinstalling it as long as it really seems like the most probable cause & solution. My Windows is very unkept and twisted. I think the GPU indeed actually works, because the renders are blazing fast compared to that old CPU. I'll reinstall it tomorrow and let you know if it resolved the issue. Thank you!

Changed status from 'Needs Triage' to: 'Needs User Info'

Changed status from 'Needs Triage' to: 'Needs User Info'

Unfortunately, I reinstalled Windows (Settings > Update & Security > Recovery > Reset this PC with all data erasing) and the problem persists.

I tried downloading blender v 2.83 and 2.90 and the error is also present there. I'm starting to think this might be an issue with my setup consisting of old parts with old drivers.

I don't know how to produce a debug log that contains the CUDA error. The last one I uploaded was by running the blender_debug_log.cmd then producing the error and turning it off.

In the back of my head I'm already considering purchasing a new motherboard, cpu & ram.

Unfortunately, I reinstalled Windows (Settings > Update & Security > Recovery > Reset this PC with all data erasing) and the problem persists. I tried downloading blender v 2.83 and 2.90 and the error is also present there. I'm starting to think this might be an issue with my setup consisting of old parts with old drivers. I don't know how to produce a debug log that contains the CUDA error. The last one I uploaded was by running the `blender_debug_log.cmd` then producing the error and turning it off. In the back of my head I'm already considering purchasing a new motherboard, cpu & ram.

Changed status from 'Needs User Info' to: 'Needs Triage'

Changed status from 'Needs User Info' to: 'Needs Triage'

I think the last log was created with blender_debug_gpu.cmd rather than blender_debug_log.cmd (it's confusing).

I think the last log was created with `blender_debug_gpu.cmd` rather than `blender_debug_log.cmd` (it's confusing).

Added subscriber: @dark999

Added subscriber: @dark999

@xander91 you forgot to upload log file produced with blender_debug_log.cmd, check and disable any type of overclock related of PCIe speed

@xander91 you forgot to upload log file produced with `blender_debug_log.cmd`, check and disable any type of overclock related of PCIe speed

Oh whoops! You guys are right, thanks for being patient with me. Here are the actual blender_debug_log.cmd files.
blender_debug_output.txt
blender_system_info.txt

In the meantime I read a blog post saying the method I used for Windows reinstall may not help with system corruption issues and I can try reinstalling it with a usb stick.

@dark999 I don't know how to check/disable that yet, but speaking of PCIe, my motherboard has PCIe 2.0 slots and my card has PCIe 3.0, could this be the issue?

Oh whoops! You guys are right, thanks for being patient with me. Here are the actual `blender_debug_log.cmd` files. [blender_debug_output.txt](https://archive.blender.org/developer/F8520663/blender_debug_output.txt) [blender_system_info.txt](https://archive.blender.org/developer/F8520667/blender_system_info.txt) In the meantime I read a blog post saying the method I used for Windows reinstall may not help with system corruption issues and I can try reinstalling it with a usb stick. @dark999 I don't know how to check/disable that yet, but speaking of PCIe, my motherboard has PCIe 2.0 slots and my card has PCIe 3.0, could this be the issue?
Member

Removed subscriber: @Alaska

Removed subscriber: @Alaska

Changed status from 'Needs Triage' to: 'Resolved'

Changed status from 'Needs Triage' to: 'Resolved'
Alexander Majewski self-assigned this 2020-05-12 13:48:28 +02:00

I fixed it. dark999 was correct. I found another thread where a guy with the same error decreased the GPUs clock speed by -10 and I did the same with an MSI Afterburner and I finally got a full render. Do you know why blender is this way?

It still froze once (without an error), but in general it's becoming clear it's about clocking settings.

I fixed it. dark999 was correct. I found another thread where a guy with the same error decreased the GPUs clock speed by -10 and I did the same with an MSI Afterburner and I finally got a full render. Do you know why blender is this way? It still froze once (without an error), but in general it's becoming clear it's about clocking settings.

Added subscriber: @toxsickcity

Added subscriber: @toxsickcity

I'd also like to add a PSA, gpu's dont like to be overclocked in blender workloads. it's slightly more sensitive to overclocking, So if you receive memory errors or cuda stopping, try checking your gpu speeds and maybe clock back a little bit.

this worked for me as I was scratching my head that the 8GB on the 1070s were running out but only using about 3.5GB!!

Anyways Good I found this.

I'd also like to add a PSA, gpu's dont like to be overclocked in blender workloads. it's slightly more sensitive to overclocking, So if you receive memory errors or cuda stopping, try checking your gpu speeds and maybe clock back a little bit. this worked for me as I was scratching my head that the 8GB on the 1070s were running out but only using about 3.5GB!! Anyways Good I found this.

Added subscriber: @WilliamSchilthuis

Added subscriber: @WilliamSchilthuis

Adding information in case anyone else comes across this issues and ends up here.

Windows 10 "Hardware-accelerated GPU scheduling" causes CUDA errors for me and prevents GPU rendering.
https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/

It can be disabled under Windows Settings > Graphics Settings. Disabling the feature and rebooting fixed GPU rendering for me

Adding information in case anyone else comes across this issues and ends up here. Windows 10 "Hardware-accelerated GPU scheduling" causes CUDA errors for me and prevents GPU rendering. https://devblogs.microsoft.com/directx/hardware-accelerated-gpu-scheduling/ It can be disabled under Windows Settings > Graphics Settings. Disabling the feature and rebooting fixed GPU rendering for me

Added subscriber: @Rob-Betz

Added subscriber: @Rob-Betz

I'm running into this error and so far none of the above has worked, including disabling the windows hardware settings.

i can get a scene to render once or twice but then get the CUDA error. I noticed when the error appears, mem numbers skyrocket to like 15000.

System Information
Operating system: Win10
Graphics card: GeForce RTX 2070 SUPER

This happens in any version of blender.

I'm running into this error and so far none of the above has worked, including disabling the windows hardware settings. i can get a scene to render once or twice but then get the CUDA error. I noticed when the error appears, mem numbers skyrocket to like 15000. **System Information** Operating system: Win10 Graphics card: GeForce RTX 2070 SUPER This happens in any version of blender.

I solved the CUDA error on my machine by switching my render output to "slot 2". I have NO idea why that worked, but I've been churning out full 8K renders for an interactive VR environment using my GPU. Renders of these scenes before took about 1.5-2 hours. Now they take 15 minutes.

I solved the CUDA error on my machine by switching my render output to "slot 2". I have NO idea why that worked, but I've been churning out full 8K renders for an interactive VR environment using my GPU. Renders of these scenes before took about 1.5-2 hours. Now they take 15 minutes.
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
7 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#76526
No description provided.