CUDA errors #48765

New Issue

Nicolas LABORDE · 2016-06-29T19:10:26+02:00

Nicolas LABORDE commented

2016-06-29 19:10:26 +02:00

System Information
Windows 10, GTX 670

Blender Version
Broken: 2.76, 2.77a

Short description of error
Crash during render. Various CUDA errors
CUDA error at cuCtxCreate: ILLEGAL_ADDRESS or CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

Exact steps for others to reproduce the error
Press render and wait. I have a blend file to send but I packed the textures and it's 120Mb now... I can Wetransfer it if you give me an address. Memory usage is in the 200Mb range... And I have 16 Gb of Ram so it shouldn't be that...

**System Information** Windows 10, GTX 670 **Blender Version** Broken: 2.76, 2.77a **Short description of error** Crash during render. Various CUDA errors CUDA error at cuCtxCreate: ILLEGAL_ADDRESS or CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize() **Exact steps for others to reproduce the error** Press render and wait. I have a blend file to send but I packed the textures and it's 120Mb now... I can Wetransfer it if you give me an address. Memory usage is in the 200Mb range... And I have 16 Gb of Ram so it shouldn't be that...

Nicolas LABORDE commented

2016-06-29 19:10:26 +02:00

Changed status to: 'Open'

Nicolas LABORDE commented

2016-06-29 19:10:26 +02:00

Added subscriber: @JulesetMariusLABORDE

Nicolas LABORDE commented

2016-06-29 19:11:14 +02:00

Let me know if I can help you!

Sergey Sharybin commented

2016-06-30 11:24:01 +02:00

Added subscriber: @Sergey

Sergey Sharybin commented

2016-06-30 11:24:01 +02:00

Please simplify your file as much as possible so we can investigate it. We really need something which demonstrates the issue in order to fix it.

You can also use ftp://ftp.blender.org/incoming to share .blend file.

Please simplify your file as much as possible so we can investigate it. We really need something which demonstrates the issue in order to fix it. You can also use ftp://ftp.blender.org/incoming to share .blend file.

Sergey Sharybin commented

2016-06-30 11:24:24 +02:00

Sorry, forgot to mention, please test latest builds from builder.blender.org to see if the issue was fixed already :)

Nicolas LABORDE commented

2016-06-30 11:57:32 +02:00

I downloaded the latest build. Opened the file. Some objects got upscaled. I checked these objects and their scale is 1 to 1 to 1. What can I do?

Sergey Sharybin commented

2016-06-30 12:11:59 +02:00

For each of the issue you've got:

Prepare test file which demonstrates the issue, keep it as small as possible.
Submit a bug report. Make sure you're following bug report guidelines.

Important note: do not mix different issues in the same report, one issue -- one report. Helps keeping things under control here.

For each of the issue you've got: - Prepare test file which demonstrates the issue, keep it as small as possible. - Submit a bug report. Make sure you're following bug report guidelines. Important note: do not mix different issues in the same report, one issue -- one report. Helps keeping things under control here.

Nicolas LABORDE commented

2016-06-30 13:14:25 +02:00

Hi,

The file can be downloaded here: https://we.tl/MvQ4kxXnoV

Sorry, it's quite heavy but I couldn't do better than that...

Hi, The file can be downloaded here: https://we.tl/MvQ4kxXnoV Sorry, it's quite heavy but I couldn't do better than that...

Sergey Sharybin commented

2016-06-30 13:55:50 +02:00

I can not reproduce any issue with the file you've attached.

Some notes to check on:

You're using GPU rendering, so it is important your scene fits into VRAM, which would be much lower than your main RAM.
If your GPU is alos used for display, then it'll have some additional memory usage on it due to OS and applications, so VRAM available for Blender is lower than your overall amount of VRAM
Additionally, if your GPU is used for display it'll have additional time limits on kernels. See - how to solve/workaround this issue.
Blender requires some additional VRAM just to get render kernel working. This memory requirement became much lower in latest Blender builds. So 2.77a requires almost 700MB of VRAM here to render while latest builds requires only about 470MB.

So at this point, please check:

Make sure you've got latest drivers installed from NVidia site (it is important to use NVidia since drivers provided by Windows Update are known to be glitchy).
Latest Blender builds manages to render the scene (ignoring the fact that some objects are wrong size for you).
Check if smaller tile size / increased TDR time makes your scene to render.
Check VRAM consumption using GPU-Z to see if you're itting memory limit issue of some sort.
Start Blender from the command line with --debug-cycles argument and attach log after the issue occurred.
https://www.blender.org/manual/render/cycles/gpu_rendering.html#the-nvidia-opengl-driver-lost-connection-with-the-display-driver

I can not reproduce any issue with the file you've attached. Some notes to check on: - You're using GPU rendering, so it is important your scene fits into VRAM, which would be much lower than your main RAM. - If your GPU is alos used for display, then it'll have some additional memory usage on it due to OS and applications, so VRAM available for Blender is lower than your overall amount of VRAM - Additionally, if your GPU is used for display it'll have additional time limits on kernels. See - [x] how to solve/workaround this issue. - Blender requires some additional VRAM just to get render kernel working. This memory requirement became much lower in latest Blender builds. So 2.77a requires almost 700MB of VRAM here to render while latest builds requires only about 470MB. So at this point, please check: - Make sure you've got latest drivers installed from NVidia site (it is important to use NVidia since drivers provided by Windows Update are known to be glitchy). - Latest Blender builds manages to render the scene (ignoring the fact that some objects are wrong size for you). - Check if smaller tile size / increased TDR time makes your scene to render. - Check VRAM consumption using GPU-Z to see if you're itting memory limit issue of some sort. - Start Blender from the command line with --debug-cycles argument and attach log after the issue occurred. - [x] https://www.blender.org/manual/render/cycles/gpu_rendering.html#the-nvidia-opengl-driver-lost-connection-with-the-display-driver

Wolfgang Faehnle commented

2016-06-30 14:22:31 +02:00

Added subscriber: @mib2berlin

Wolfgang Faehnle commented

2016-06-30 14:22:31 +02:00

Hi, cant get errors too on my GTX 670 2 GB.
The file need 760 MB VRAM.
I had to reduce samples to 200, no time for a 60 minutes render but should not make a difference.

Opensuse Leap 42.1 x86_64
Intel i5 3570K
RAM 16 GB
GTX 760 4 GB /Display card
GTX 670 2 GB
Driver 367.27

I checked with Blender 2.77a and GTX 670 only.

Hi, cant get errors too on my GTX 670 2 GB. The file need 760 MB VRAM. I had to reduce samples to 200, no time for a 60 minutes render but should not make a difference. Opensuse Leap 42.1 x86_64 Intel i5 3570K RAM 16 GB GTX 760 4 GB /Display card GTX 670 2 GB Driver 367.27 I checked with Blender 2.77a and GTX 670 only.

Nicolas LABORDE commented

2016-06-30 15:43:21 +02:00

Hi,

I have created a Registry key for TDR and set it to 8. Render started and crashed.

This is a GPU-Z log (not sure it's useful):
GPU-Z Sensor Log.txt

Hi, I have created a Registry key for TDR and set it to 8. Render started and crashed. This is a GPU-Z log (not sure it's useful): [GPU-Z Sensor Log.txt](https://archive.blender.org/developer/F319112/GPU-Z_Sensor_Log.txt)

Nicolas LABORDE commented

2016-06-30 15:48:49 +02:00

I don't know how to do that... I started Blender using the Command Prompt but I don't know how to do this :
"...with --debug-cycles argument and attach log after the issue occurred."

I don't know how to do that... I started Blender using the Command Prompt but I don't know how to do this : "...with --debug-cycles argument and attach log after the issue occurred."

Sergey Sharybin commented

2016-06-30 16:05:40 +02:00

What exactly you mean by "crash" ? (technically, crash is when application encounters unrecoverable error and being terminated by the OS).

In order to get debug log, run this: blender.exe --debug-cycles > log.txt 2>&1 and attach log.txt here.

What exactly you mean by "crash" ? (technically, crash is when application encounters unrecoverable error and being terminated by the OS). In order to get debug log, run this: `blender.exe --debug-cycles > log.txt 2>&1` and attach log.txt here.

Nicolas LABORDE commented

2016-06-30 16:34:19 +02:00

Sorry for being unprecise.

I started Blender using the command Promt, opened the file from Blender, pressed the render button.
Render starts, then stops and I get the Illegal address in cuCtxSynchronize() error message.

Command prompt shows this:
command log.txt

Sorry for being unprecise. I started Blender using the command Promt, opened the file from Blender, pressed the render button. Render starts, then stops and I get the Illegal address in cuCtxSynchronize() error message. Command prompt shows this: [command log.txt](https://archive.blender.org/developer/F319113/command_log.txt)

Thomas Dinges commented

2016-07-01 21:41:25 +02:00

Added subscriber: @ThomasDinges

Thomas Dinges commented

2016-07-01 21:41:25 +02:00

It's not clear to me if you got the issue with a new build as well? The attached log is from an older blender (not current master).

Nicolas LABORDE commented

2016-07-01 23:24:37 +02:00

Hi Thomas,
No, I had the issue with 2.77a. I'm testing the latest build tonight. I'll let you know.

Hi Thomas, No, I had the issue with 2.77a. I'm testing the latest build tonight. I'll let you know.

Nicolas LABORDE commented

2016-07-02 08:06:21 +02:00

Same thing. : (
Here's the log.
command log-2.77.1.txt

Same thing. : ( Here's the log. [command log-2.77.1.txt](https://archive.blender.org/developer/F319272/command_log-2.77.1.txt)

Nicolas LABORDE commented

2016-07-04 18:49:09 +02:00

Do you have enough infos with my latest log?

Sergey Sharybin commented

2016-07-05 14:51:32 +02:00

There is unfortunately not so much in the logs, just for some reason CUDA fails.

On a fortunate side i'm back to the studio and have some NVidia hardware around and Windows 10 machine. Will try to reproduce the issue on a closer configuration to yours.

At the meantime, did anyone else confirm the issue?

@JulesetMariusLABORDE, did you make sure you've got latest drivers installed from nvidia.com site?

There is unfortunately not so much in the logs, just for some reason CUDA fails. On a fortunate side i'm back to the studio and have some NVidia hardware around and Windows 10 machine. Will try to reproduce the issue on a closer configuration to yours. At the meantime, did anyone else confirm the issue? @JulesetMariusLABORDE, did you make sure you've got latest drivers installed from nvidia.com site?

Nicolas LABORDE commented

2016-07-05 15:55:40 +02:00

Hi Sergey,
Yes, I have installed the latest drivers (368.39).

Hi Sergey, Yes, I have installed the latest drivers (368.39).

Nicolas LABORDE commented

2016-07-12 19:59:48 +02:00

I just tried to launch a render of the same scene on another computer.

I had a render error:
CUDA error: Launch exceeded timeout in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)

I just tried to launch a render of the same scene on another computer. I had a render error: CUDA error: Launch exceeded timeout in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)

Nicolas LABORDE commented

2016-07-12 20:01:26 +02:00

And yet another:
CUDA error at cuCtxCreate: Launch exceeded timeout

And yet another: CUDA error at cuCtxCreate: Launch exceeded timeout

Sergey Sharybin commented

2016-07-12 20:08:22 +02:00

Changed status from 'Open' to: 'Archived'

Sergey Sharybin closed this issue

2016-07-12 20:08:22 +02:00

Sergey Sharybin self-assigned this 2016-07-12 20:08:22 +02:00

Sergey Sharybin commented

2016-07-12 20:08:22 +02:00

Timeout is caused by specific of how CUDA works -- it'll limit amount of time kernel can do calculations on the display card. Additionally, on Windows you've got extra timeout which is applied on driver itself and will re-set it if it did not respond within a given timeout.

See - [x] for a bit details and possible solutions.

So it all seems you're just hitting limitations of GPGPU, nothing we can really solve from blender side.

https://www.blender.org/manual/render/cycles/gpu_rendering.html#the-nvidia-opengl-driver-lost-connection-with-the-display-driver

Timeout is caused by specific of how CUDA works -- it'll limit amount of time kernel can do calculations on the display card. Additionally, on Windows you've got extra timeout which is applied on driver itself and will re-set it if it did not respond within a given timeout. See - [x] for a bit details and possible solutions. So it all seems you're just hitting limitations of GPGPU, nothing we can really solve from blender side. - [x] https://www.blender.org/manual/render/cycles/gpu_rendering.html#the-nvidia-opengl-driver-lost-connection-with-the-display-driver

Nicolas LABORDE commented

2016-07-12 22:17:26 +02:00

How come this is not working on this brand new computer:
https://www.grosbill.com/4-msi_gt72s_6qe_483fr_i7_ssd_gtx980m_gsync-671328-ordinateurs-ordi_portable

How come this is not working on this brand new computer: https://www.grosbill.com/4-msi_gt72s_6qe_483fr_i7_ssd_gtx980m_gsync-671328-ordinateurs-ordi_portable

Nicolas LABORDE commented

2016-07-12 22:19:14 +02:00

Still, thank you for your time. I will read your link tomorrow.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

CUDA errors #48765