Page MenuHome

CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
Closed, ResolvedPublic

Description

System Information
Ubuntu 14.04 64bit; Nvidia GTX 550 Ti (sm21)

Blender Version
Broken: 2.72 47b8bf5

Short description of error
Cuda crashes/stops working (but can be toggled off/on again) if multiple importance is disabled after being set to 512.

Exact steps for others to reproduce the error

  • Enable cycles in viewport
  • Switch to world settings and enable Multiple Importance
  • Change 256 to 512
  • Disable Multiple Importance

Output of the console:

Event Timeline

Dan McGrath (dmcgrath) added a project: Cycles.
Dan McGrath (dmcgrath) set Type to Bug.
Dan McGrath (dmcgrath) created this task.
Dan McGrath (dmcgrath) raised the priority of this task from to Needs Triage by Developer.
Bastien Montagne (mont29) triaged this task as Normal priority.Oct 10 2014, 9:35 AM

Hi, I cant verify this error on my system.
Blender 45ce901

Opensuse 13.1/64
Intel i5 3770K
GTX 760 4 GB (Display)
GTX 560Ti 1.28 GB 448 Cores
Driver 343.19

Tested on both cards and with GTX 760 also with experimental kernel.

Cheers, mib

Ok, I just upgraded Blender to c2d6de8e20 and it still happens.

Can confirm the error on my system with Blender 2.72

Windows 7 x64
Twin GTX 580 3GB

Tested Supported and Experimental cubin
Blender 2.72 hash 95182d1 (official release)
Blender 2.72 hash ee5936c

Blender 2.70a and 2.71 does not cause a CUDA error while performing the steps outlined in first post.

After further tests, Multiple Importance Map Resolution of 511 and below does not cause a CUDA error with 2.72, just settings of 512 and above.

David

I have had this problem in 2.69, 2.71, 2.72a and now 2.73a.

Everytime I come to this sight, it is listed as fixed. This is not fixed.

OUTPUT START

CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

Refer to the Cycles GPU rendering documentation for possible solutions:
http://www.blender.org/manual/render/cycles/gpu_rendering.html

CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()
CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemAlloc(&device_pointer, size)
CUDA error: ILLEGAL_ADDRESS in cuModuleGetGlobal(&cumem, &cubytes, cuModule, name)
CUDA error: ILLEGAL_ADDRESS in cuMemcpyHtoD(cumem, (void*)&ptr, cubytes)
CUDA error: ILLEGAL_ADDRESS in cuMemAlloc(&device_pointer, size)
CUDA error: ILLEGAL_ADDRESS in cuModuleGetGlobal(&cumem, &cubytes, cuModule, name)
CUDA error: ILLEGAL_ADDRESS in cuMemcpyHtoD(cumem, (void*)&ptr, cubytes)
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
Error: CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

OUTPUT END
DIAGNOSIS BEGIN

karim@Phenuntu:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 340.29 Thu Jul 31 20:23:19 PDT 2014
GCC version: gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1)
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$ gcc --version
gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$ lsmod | grep nv
nvidia_uvm 35066 2
nvidia 10540162 68 nvidia_uvm
drm 303102 2 nvidia
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$
karim@Phenuntu:~$ cd /usr/local/cuda/samples/1_Utilities/deviceQuery
karim@Phenuntu:/usr/local/cuda/samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 760"

CUDA Driver Version / Runtime Version          6.5 / 6.5
CUDA Capability Major/Minor version number:    3.0
Total amount of global memory:                 4096 MBytes (4294639616 bytes)
( 6) Multiprocessors, (192) CUDA Cores/MP:     1152 CUDA Cores
GPU Clock rate:                                1150 MHz (1.15 GHz)
Memory Clock rate:                             3004 Mhz
Memory Bus Width:                              256-bit
L2 Cache Size:                                 524288 bytes
Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
Total amount of constant memory:               65536 bytes
Total amount of shared memory per block:       49152 bytes
Total number of registers available per block: 65536
Warp size:                                     32
Maximum number of threads per multiprocessor:  2048
Maximum number of threads per block:           1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch:                          2147483647 bytes
Texture alignment:                             512 bytes
Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
Run time limit on kernels:                     Yes
Integrated GPU sharing Host Memory:            No
Support host page-locked memory mapping:       Yes
Alignment requirement for Surfaces:            Yes
Device has ECC support:                        Disabled
Device supports Unified Addressing (UVA):      Yes
Device PCI Bus ID / PCI location ID:           1 / 0
Compute Mode:
   < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 6.5, CUDA Runtime Version = 6.5, NumDevs = 1, Device0 = GeForce GTX 760
Result = PASS
karim@Phenuntu:/usr/local/cuda/samples/1_Utilities/deviceQuery$

DIAGNOSIS ENDS

Ubuntu 14.04

It was fixed for Dan, so you might be experiencing another issue. So please do a separate report then with all the info requested in the template (example .blend, steps to reproduce etc). It's also better to attach system info as a file -- makes things easier to read :)

I am getting this exact error in 2.74 (from Debian Sid repository). The output looks like this:

write exr tmp file, 1920x1080, /home/sofus/blender_temp/blender_EzOX9n/first_blend_2_(Logo_FX)_RenderLayer_exr
CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

Before launching into this:

CUDA error: ILLEGAL_ADDRESS in cuMemcpyDtoH((uchar*)mem.data_pointer + offset, (CUdeviceptr)(mem.device_pointer + offset), size)
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: ILLEGAL_ADDRESS in cuMemAlloc(&device_pointer, size)
//...x31...//
CUDA error: ILLEGAL_ADDRESS in cuMemFree(cuda_device_ptr(mem.device_pointer))
//..x24...//
read exr tmp file: /home/sofus/blender_temp/blender_EzOX9n/first_blend_2_(Logo_FX)_RenderLayer_exr
Error: CUDA error: ILLEGAL_ADDRESS in cuCtxSynchronize()

I am using a 560 Ti 1GB in tandem with a 780 Ti 3GB. My scene includes a 500MB fluid simulation, and a dynamic paint cache. The memory usage supposedly peaks at 600MB (4GB during compositing; I had buffer groups and OpenCL on). There is only a subdivided plane and some mesh text in terms of geometry. It managed to render 100 frames (~3 minutes each) before throwing this error. I had Save Buffers on using 256x256 tiles.

Please always make a *new* report, do not attach these infos to old and already fixed reports.

And please test with official builds from blender.org only and not some third party builds.