Page MenuHome

Blender can't find GPU
Closed, ArchivedPublic

Description

System Information
Operating system: Ubuntu 18.04.2 LTS
Graphics card: NVidia K80

Blender Version
Broken: ( 2.80, master)
Worked: (2.80, master)

I was able to run to run Blender 2.79 and 2.80 on Google Colab just fine for last two months but it stopped working few days ago. While every tool (nvidia-smi/Keras/Tensorflow) does find the GPU and works fine Blender just doesn't find the gpu at all. Running

blender --debug-cycles

with xvfb didn't yield any usefull error information either.

I do use the following python script with the CLI to setup CUDA etc. automatically, which should output all activated devices:

import re
import bpy
scene = bpy.context.scene
scene.cycles.device = 'GPU'
prefs = bpy.context.preferences
prefs.addons['cycles'].preferences.get_devices()
cprefs = prefs.addons['cycles'].preferences
print(cprefs)
# Attempt to set GPU device types if available
for compute_device_type in ('CUDA', 'OPENCL', 'NONE'):
    try:
        cprefs.compute_device_type = compute_device_type
        break
    except TypeError:
        pass
# Enable all CPU and GPU devices
for device in cprefs.devices:
    print('Activating',device)
    device.use = True

For a finished python notebook take a look at: https://gist.github.com/donmahallem/a05100077ec1327268f28f0b2bd8da60 You don't need a working .blend file as the script runs before loading the blend file. If a GPU is being found the script should output its name.

Driver Information etc. output from nvidia-smi:

Sat Aug 24 20:52:39 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.40       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   32C    P8    26W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Running the command with --debug-cycles yields:

Blender 2.80 (sub 75) (hash f6cb5f54494e built 2019-07-29 17:17:04)
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5007:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM default
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory
found bundled python: /content/blender280/2.80/python
I0824 21:30:00.756532   262 blender_python.cpp:180] Debug flags initialized to:
CPU flags:
  AVX2       : True
  AVX        : True
  SSE4.1     : True
  SSE3       : True
  SSE2       : True
  BVH layout : BVH8
  Split      : False
CUDA flags:
 Adaptive Compile: False
OpenCL flags:
  Device type    : ALL
  Debug          : False
  Memory limit   : 0
I0824 21:30:00.785424   262 device_opencl.cpp:48] CLEW initialization succeeded.
I0824 21:30:00.790717   262 device_cuda.cpp:2552] CUEW initialization succeeded
I0824 21:30:00.790748   262 device_cuda.cpp:2554] Found precompiled kernels
<bpy_struct, CyclesPreferences at 0x7f2fe922ede8>
Device found CUDA
Error: Cannot read file './test.blend': No such file or directory

Blender quit

Details

Type
Bug

Event Timeline

LazyDodo (LazyDodo) triaged this task as Needs Information from User priority.

Script seems to work just fine here (although it does use CPU+GPU rendering, not sure if that is what you were going for) but the prints appear at the end of the render for me, probably has something to do with python buffering stdout (wild guess)

Saved: 'e:\builds\build_soc-2019-openxr\bin\bmw27_cpu0001.jpg'
 Time: 01:35.21 (Saving: 00:00.13)

<bpy_struct, CyclesPreferences at 0x0000024EB77CB6F8>
Activating <bpy_struct, CyclesDeviceSettings("GeForce GTX 1660")>
Activating <bpy_struct, CyclesDeviceSettings("Intel Core i7-3770 CPU @ 3.40GHz")>

Blender quit

Blenders command line arguments are processed in the order given, in your python notebook, first your script runs setting up the gpu, then it loads the blend file, overwriting any settings you just have done, which is probably causing the issues you are seeing.

Don (DonMahallem) added a comment.EditedSun, Aug 25, 3:02 AM

If I do run:

!sudo ./blender280/blender -P setgpu.py -b 'bmw27/bmw27_gpu.blend' -P setgpu.py -o 'cast.png' -f 1

The output starts with:

Blender 2.80 (sub 75) (hash f6cb5f54494e built 2019-07-29 17:17:04)
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5007:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM default
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory
found bundled python: /content/blender280/2.80/python
<bpy_struct, CyclesPreferences at 0x7f34aace3de8>
Activating <bpy_struct, CyclesDeviceSettings("Intel Xeon CPU @ 2.30GHz")>
Read blend: /content/bmw27/bmw27_gpu.blend
<bpy_struct, CyclesPreferences at 0x7f34aace3de8>
Activating <bpy_struct, CyclesDeviceSettings("Intel Xeon CPU @ 2.30GHz")>
Fra:1 Mem:68.61M (0.00M, Peak 69.35M) | Time:00:00.54 | Mem:0.00M, Peak:0.00M | Scene, RenderLayer | Synchronizing object | Light
Fra:1 Mem:68.61M (0.00M, Peak 69.35M) | Time:00:00.55 | Mem:0.00M, Peak:0.00M | Scene, RenderLayer | Synchronizing object | Floor
............................................

So the run order shouldn't be the issue. My problem is that as can be seen, that only the CPU is detected by blender.

Another thing I did notice which did work flawlessly before is running sheep it on collab:

!wget https://www.sheepit-renderfarm.com/media/applet/sheepit-client-5.1621.3003.jar
!java -jar ./sheepit-client-5.1621.3003.jar --show-gpu
#Id        : CUDA_Tesla K80_0000:00:04
#Model     : Tesla K80
#Memory, MB: 11441
!sudo java -jar ./sheepit-client-5.1621.3003.jar -ui text -login LOGIN -password PASSWORD -compute-method GPU -gpu 'CUDA_Tesla K80_0000:00:04'
#Nvidia::getGpus cuInit failed (ret: 100)
#GPU unknown, list of available gpus can be display with --show-gpu

throws a cuInit error with code 100 which correspondends to CUDA_ERROR_NO_DEVICE according to: https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__TYPES.html#group__CUDA__TYPES_1ggc6c391505e117393cc2558fff6bfc2e990696c86fcee1f536a1ec7d25867feeb

CUDA_ERROR_NO_DEVICE = 100
This indicates that no CUDA-capable devices were detected by the installed CUDA driver.

I'd look into verifying the cuda installation at this point.

Don (DonMahallem) added a comment.EditedSun, Aug 25, 8:13 AM

I do can make the CUDA Samples just fine:

!cat /proc/driver/nvidia/version
!nvcc -V
%cd ~/
!sudo apt-get install cuda-samples-10-0 -y
%cd /usr/local/cuda-10.0/samples
!sudo make

and I do can run the samples and it detects the GPU with:

!./bin/x86_64/linux/release/deviceQuery

./bin/x86_64/linux/release/deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Tesla K80"

CUDA Driver Version / Runtime Version          10.1 / 10.0
CUDA Capability Major/Minor version number:    3.7
Total amount of global memory:                 11441 MBytes (11996954624 bytes)
(13) Multiprocessors, (192) CUDA Cores/MP:     2496 CUDA Cores
GPU Max Clock rate:                            824 MHz (0.82 GHz)
Memory Clock rate:                             2505 Mhz
Memory Bus Width:                              384-bit
L2 Cache Size:                                 1572864 bytes
Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
Total amount of constant memory:               65536 bytes
Total amount of shared memory per block:       49152 bytes
Total number of registers available per block: 65536
Warp size:                                     32
Maximum number of threads per multiprocessor:  2048
Maximum number of threads per block:           1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch:                          2147483647 bytes
Texture alignment:                             512 bytes
Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
Run time limit on kernels:                     No
Integrated GPU sharing Host Memory:            No
Support host page-locked memory mapping:       Yes
Alignment requirement for Surfaces:            Yes
Device has ECC support:                        Enabled
Device supports Unified Addressing (UVA):      Yes
Device supports Compute Preemption:            No
Supports Cooperative Kernel Launch:            No
Supports MultiDevice Co-op Kernel Launch:      No
Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 4
Compute Mode:
   < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.0, NumDevs = 1
Result = PASS

So I do believe that my installation is not to blame imho.

EDIT:

Just tried samples for 10.1 too and it passes just fine too

can u try to test the 2.79? again? to see is it just the blender or maybe google messed up something.

Same issue with Blender 2.79b, the GPU isn't recognized.

so it is something with your VM(google) not blender it self. maybe contact google support it might be that they banned useing blender on their cloud for free

Robert Guetzkow (rjg) closed this task as Archived.Sun, Aug 25, 4:53 PM
Robert Guetzkow (rjg) claimed this task.

Closing this tickets for now since this doesn't seem to be caused by Blender. If this turns out to be an issue in the way Blender detects CUDA devices, then we can re-open the ticket.

The issue seems to be caused by libtcmalloc-minimal4 installed on Google Colab. Without sudo blender will segfault with an error in tcmalloc.cc. Uninstalling and reinstalling the libtcmalloc-minimal4 from the Ubuntu repository and adjusting LD_PRELOAD to /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4.3.0 fixes the problem. Surprisingly the ppa by Thomas Schiex (thomas-schiex/blender) works out of the box. Does this have anything to do with how Blender uses its allocator? Seems like the version of TCmalloc that Google uses has a bug.

The valgrind output (relevant parts unfortunately not with debug symbols) in case anybody is interested in that:

Mounted at /content/drive
==529== Memcheck, a memory error detector
==529== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==529== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==529== Command: ./blender-2.80-linux-glibc217-x86_64/blender -b test.blend -P enable_gpu_v3.py -noaudio -E CYCLES -o ./drive/My\ Drive/gpu_test_ -f 1
==529== 
==529== Mismatched free() / delete / delete []
==529==    at 0xAC16D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==529==    by 0xAE34E04: ??? (in /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.3.0)
==529==    by 0x9FF6732: call_init (dl-init.c:72)
==529==    by 0x9FF6732: _dl_init (dl-init.c:119)
==529==    by 0x9FE70C9: ??? (in /lib/x86_64-linux-gnu/ld-2.27.so)
==529==    by 0xB: ???
==529==    by 0x1FFF000992: ???
==529==    by 0x1FFF0009BF: ???
==529==    by 0x1FFF0009C2: ???
==529==    by 0x1FFF0009CD: ???
==529==    by 0x1FFF0009D0: ???
==529==    by 0x1FFF0009E1: ???
==529==    by 0x1FFF0009EA: ???
==529==  Address 0xe7a7210 is 0 bytes inside a block of size 4 alloc'd
==529==    at 0xAC1689F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==529==    by 0xAE34DF7: ??? (in /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.3.0)
==529==    by 0x9FF6732: call_init (dl-init.c:72)
==529==    by 0x9FF6732: _dl_init (dl-init.c:119)
==529==    by 0x9FE70C9: ??? (in /lib/x86_64-linux-gnu/ld-2.27.so)
==529==    by 0xB: ???
==529==    by 0x1FFF000992: ???
==529==    by 0x1FFF0009BF: ???
==529==    by 0x1FFF0009C2: ???
==529==    by 0x1FFF0009CD: ???
==529==    by 0x1FFF0009D0: ???
==529==    by 0x1FFF0009E1: ???
==529==    by 0x1FFF0009EA: ???
==529== 
==529== Mismatched free() / delete / delete []
==529==    at 0xAC16D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==529==    by 0xAE4A71F: MallocExtension::Initialize() (in /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.3.0)
==529==    by 0xAE34E09: ??? (in /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.3.0)
==529==    by 0x9FF6732: call_init (dl-init.c:72)
==529==    by 0x9FF6732: _dl_init (dl-init.c:119)
==529==    by 0x9FE70C9: ??? (in /lib/x86_64-linux-gnu/ld-2.27.so)
==529==    by 0xB: ???
==529==    by 0x1FFF000992: ???
==529==    by 0x1FFF0009BF: ???
==529==    by 0x1FFF0009C2: ???
==529==    by 0x1FFF0009CD: ???
==529==    by 0x1FFF0009D0: ???
==529==    by 0x1FFF0009E1: ???
==529==  Address 0xe7a72c0 is 0 bytes inside a block of size 45 alloc'd
==529==    at 0xAC1617F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==529==    by 0x64F7FD8: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) (in /content/blender-2.80-linux-glibc217-x86_64/blender)
==529==    by 0x64F9572: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long) (in /content/blender-2.80-linux-glibc217-x86_64/blender)
==529==    by 0xAE4A708: MallocExtension::Initialize() (in /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.3.0)
==529==    by 0xAE34E09: ??? (in /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.3.0)
==529==    by 0x9FF6732: call_init (dl-init.c:72)
==529==    by 0x9FF6732: _dl_init (dl-init.c:119)
==529==    by 0x9FE70C9: ??? (in /lib/x86_64-linux-gnu/ld-2.27.so)
==529==    by 0xB: ???
==529==    by 0x1FFF000992: ???
==529==    by 0x1FFF0009BF: ???
==529==    by 0x1FFF0009C2: ???
==529==    by 0x1FFF0009CD: ???
==529==