CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
Closed, ResolvedPublic

Description

System Information
Ubuntu 14.04 64bit; Nvidia GTX 550 Ti (sm21)

Blender Version
Broken: 2.72 47b8bf5

Short description of error
Cuda crashes/stops working (but can be toggled off/on again) if multiple importance is disabled after being set to 512.

Exact steps for others to reproduce the error

  • Enable cycles in viewport
  • Switch to world settings and enable Multiple Importance
  • Change 256 to 512
  • Disable Multiple Importance

Output of the console:

Dan McGrath (dmcgrath) updated the task description. (Show Details)
Dan McGrath (dmcgrath) raised the priority of this task from to Needs Triage.
Dan McGrath (dmcgrath) added a project: Cycles.
Dan McGrath (dmcgrath) set Type to Bug.
Bastien Montagne (mont29) triaged this task as Normal priority.Oct 10 2014, 9:35 AM

Hi, I cant verify this error on my system.
Blender 45ce901

Opensuse 13.1/64
Intel i5 3770K
GTX 760 4 GB (Display)
GTX 560Ti 1.28 GB 448 Cores
Driver 343.19

Tested on both cards and with GTX 760 also with experimental kernel.

Cheers, mib

Ok, I just upgraded Blender to c2d6de8e20 and it still happens.

Can confirm the error on my system with Blender 2.72

Windows 7 x64
Twin GTX 580 3GB

Tested Supported and Experimental cubin
Blender 2.72 hash 95182d1 (official release)
Blender 2.72 hash ee5936c

Blender 2.70a and 2.71 does not cause a CUDA error while performing the steps outlined in first post.

After further tests, Multiple Importance Map Resolution of 511 and below does not cause a CUDA error with 2.72, just settings of 512 and above.

David

I have had this problem in 2.69, 2.71, 2.72a and now 2.73a.

Everytime I come to this sight, it is listed as fixed. This is not fixed.

OUTPUT START

CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

Refer to the Cycles GPU rendering documentation for possible solutions:
http://www.blender.org/manual/render/cycles/gpu_rendering.html

CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemAlloc(&device_pointer, size)
CUDA error: ILLEGAL_ADDRESS in cuModuleGetGlobal(&cumem, &cubytes, cuModule, name)
CUDA error: ILLEGAL_ADDRESS in cuMemcpyHtoD(cumem, (void*)&ptr, cubytes)
CUDA error: ILLEGAL_ADDRESS in cuMemAlloc(&device_pointer, size)
CUDA error: ILLEGAL_ADDRESS in cuModuleGetGlobal(&cumem, &cubytes, cuModule, name)
CUDA error: ILLEGAL_ADDRESS in cuMemcpyHtoD(cumem, (void*)&ptr, cubytes)
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
Error: CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

OUTPUT END
DIAGNOSIS BEGIN

karim@Phenuntu:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 340.29 Thu Jul 31 20:23:19 PDT 2014
GCC version: gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1)
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$ gcc --version
gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$ lsmod | grep nv
nvidia_uvm 35066 2
nvidia 10540162 68 nvidia_uvm
drm 303102 2 nvidia
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$ cd /usr/local/cuda/samples/1_Utilities/deviceQuery
karim@Phenuntu:/usr/local/cuda/samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 760"

CUDA Driver Version / Runtime Version          6.5 / 6.5
CUDA Capability Major/Minor version number:    3.0
Total amount of global memory:                 4096 MBytes (4294639616 bytes)
( 6) Multiprocessors, (192) CUDA Cores/MP:     1152 CUDA Cores
GPU Clock rate:                                1150 MHz (1.15 GHz)
Memory Clock rate:                             3004 Mhz
Memory Bus Width:                              256-bit
L2 Cache Size:                                 524288 bytes
Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
Total amount of constant memory:               65536 bytes
Total amount of shared memory per block:       49152 bytes
Total number of registers available per block: 65536
Warp size:                                     32
Maximum number of threads per multiprocessor:  2048
Maximum number of threads per block:           1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch:                          2147483647 bytes
Texture alignment:                             512 bytes
Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
Run time limit on kernels:                     Yes
Integrated GPU sharing Host Memory:            No
Support host page-locked memory mapping:       Yes
Alignment requirement for Surfaces:            Yes
Device has ECC support:                        Disabled
Device supports Unified Addressing (UVA):      Yes
Device PCI Bus ID / PCI location ID:           1 / 0
Compute Mode:
   < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 6.5, CUDA Runtime Version = 6.5, NumDevs = 1, Device0 = GeForce GTX 760
Result = PASS
karim@Phenuntu:/usr/local/cuda/samples/1_Utilities/deviceQuery$

DIAGNOSIS ENDS

Ubuntu 14.04

It was fixed for Dan, so you might be experiencing another issue. So please do a separate report then with all the info requested in the template (example .blend, steps to reproduce etc). It's also better to attach system info as a file -- makes things easier to read :)

I am getting this exact error in 2.74 (from Debian Sid repository). The output looks like this:

write exr tmp file, 1920x1080, /home/sofus/blender_temp/blender_EzOX9n/first_blend_2_(Logo_FX)_RenderLayer_exr
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

Before launching into this:

CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemAlloc(&device_pointer, size)
//...x31...//
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
//..x24...//
read exr tmp file: /home/sofus/blender_temp/blender_EzOX9n/first_blend_2_(Logo_FX)_RenderLayer_exr
Error: CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

I am using a 560 Ti 1GB in tandem with a 780 Ti 3GB. My scene includes a 500MB fluid simulation, and a dynamic paint cache. The memory usage supposedly peaks at 600MB (4GB during compositing; I had buffer groups and OpenCL on). There is only a subdivided plane and some mesh text in terms of geometry. It managed to render 100 frames (~3 minutes each) before throwing this error. I had Save Buffers on using 256x256 tiles.

Please always make a *new* report, do not attach these infos to old and already fixed reports.

And please test with official builds from blender.org only and not some third party builds.