Cycles uses only 4 GPUs instead of 5 available #36137

New Issue

Jozef Zajac · 2013-07-14T15:06:08+02:00

Jozef Zajac commented

2013-07-14 15:06:08 +02:00

%%%--- Operating System, Graphics card ---
Visible in attached printscreens, specs of MB in the post on blenderartist

PCI_E1 - 580 - here comes the display
PCI_E4 - 590
PCI_E5 - 590

- Blender version with error, and version that worked ---

2.67.1 but tested on several releases and builds, from graphicall, always used only 4 GPUs

- Short description of error ---

Cycles uses only 4 GPUs instead o 5

- Further description in this post ---

http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why

- Steps for others to reproduce the error (preferably based on attached .blend file) ---

I have used the http://mikepan.com/files/BMW1M-MikePan.blend
the test HW should be the same :( or have at least 5 GPUs to try to reproduce this issue

with friendly regards

Jozef

%%%

%%%--- Operating System, Graphics card --- Visible in attached printscreens, specs of MB in the post on blenderartist PCI_E1 - 580 - here comes the display PCI_E4 - 590 PCI_E5 - 590 - Blender version with error, and version that worked --- 2.67.1 but tested on several releases and builds, from graphicall, always used only 4 GPUs - Short description of error --- Cycles uses only 4 GPUs instead o 5 - Further description in this post --- http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why - Steps for others to reproduce the error (preferably based on attached .blend file) --- I have used the http://mikepan.com/files/BMW1M-MikePan.blend the test HW should be the same :( or have at least 5 GPUs to try to reproduce this issue with friendly regards Jozef %%%

Jozef Zajac commented

2013-07-14 15:06:08 +02:00

Changed status to: 'Open'

Brecht Van Lommel commented

2013-07-15 17:12:36 +02:00

%%%In the screenshot you post, it's indeed only working on 4 tiles at a time. Is that after setting the user preferences to use all 5 cards as in the other screenshot? Or does it show 5 tiles sometimes but one GPU isn't actually doing anything? What happens when you set the tile size smaller, does it show more than 4 tiles as active then?%%%

Jozef Zajac commented

2013-07-15 20:32:58 +02:00

%%%Yes I have setted up first to use all devices, and it always shows only 4tiles at time, independently on tile size. I also uses only 580+3x590,
since the workload on the last 590core, is off load. I am using the last Techpower UP GPU-Z, sensors to control what hapens on the GPUs.
I can post a screenshot, in an hour or so.%%%

%%%Yes I have setted up first to use all devices, and it always shows only 4tiles at time, independently on tile size. I also uses only 580+3x590, since the workload on the last 590core, is off load. I am using the last Techpower UP GPU-Z, sensors to control what hapens on the GPUs. I can post a screenshot, in an hour or so.%%%

Jozef Zajac commented

2013-07-15 20:59:53 +02:00

%%%Here is the screenshot I was unable to upload it to Attachments, so it is on the forum ...
http://blenderartists.org/forum/attachment.php?attachmentid=246573&d=1373914621%%%

%%%Here is the screenshot I was unable to upload it to Attachments, so it is on the forum ... http://blenderartists.org/forum/attachment.php?attachmentid=246573&d=1373914621%%%

Jozef Zajac commented

2013-07-15 21:11:06 +02:00

%%%I have did the same test in 2.64a, using OpenCL but Blender Crashed, just after uploading the data and BVH to the graphic card, at the moment of the start of render.%%%

Jozef Zajac commented

2013-07-15 21:17:51 +02:00

%%%Errata: It also uses only 580+3x590 !!!%%%

Jozef Zajac commented

2013-07-17 12:24:00 +02:00

%%%Further things are appearing in the thread, worth to take a look on it, before further advance in solving this issue.%%%

Doug Gale commented

2013-07-17 13:14:58 +02:00

%%%- 5 GTX 580-type GPUs would consume an awful lot of power. I would suspect a power issue. What wattage is your power supply?

At https://devtalk.nvidia.com/default/topic/517620/maximum-number-of-cuda-devices/ it is mentioned that some motherboards have BIOS initialization issues with more than 4 GPUs. I spoke on IRC and heard that you have used 5 GPUs in other renderers. How sure are you that all five worked?
I ask the previous question because in the BlenderArtists thread at http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why your screenshot shows a screenshot with 5 devices, and one screenshot with 4. Why the screenshot of just 4?
To the best of my knowledge, it is impossible to have all 3 devices run at x16 (even if one card is tested to work at x16, in each slot). The CPU literally has separate pins for every slot, so the CPU can communicate with all of the slots concurrently, and there are not enough "lanes" to run 3x16 on any CPU that I am aware of. It may be possible for them to run 3 at x8. What motherboard do you have?
What operating system do you run? Are you a developer? (I ask because I'm curious about whether you would be able to apply a an experimental patch and build blender).
%%%

%%%- 5 GTX 580-type GPUs would consume an awful lot of power. I would suspect a power issue. What wattage is your power supply? - At https://devtalk.nvidia.com/default/topic/517620/maximum-number-of-cuda-devices/ it is mentioned that some motherboards have BIOS initialization issues with more than 4 GPUs. I spoke on IRC and heard that you have used 5 GPUs in other renderers. How sure are you that all five worked? - I ask the previous question because in the BlenderArtists thread at http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why your screenshot shows a screenshot with 5 devices, and one screenshot with 4. Why the screenshot of just 4? - To the best of my knowledge, it is impossible to have all 3 devices run at x16 (even if one card is tested to work at x16, in each slot). The CPU literally has separate pins for every slot, so the CPU can communicate with all of the slots concurrently, and there are not enough "lanes" to run 3x16 on any CPU that I am aware of. It may be possible for them to run 3 at x8. What motherboard do you have? - What operating system do you run? Are you a developer? (I ask because I'm curious about whether you would be able to apply a an experimental patch and build blender). %%%

Bao 2 commented

2013-07-17 13:20:21 +02:00

%%%Doug Gale, in the blenderartist link you can see Jozef run Octane and Arion without problems showing 5 gpus were used. So is a cycles issue not a hardware issue.

I think perhaps the second gpu in each of the two 590 receive same ID perhaps?
%%%

%%%Doug Gale, in the blenderartist link you can see Jozef run Octane and Arion without problems showing 5 gpus were used. So is a cycles issue not a hardware issue. I think perhaps the second gpu in each of the two 590 receive same ID perhaps? %%%

Jozef Zajac commented

2013-07-17 23:46:17 +02:00

%%%Doug:

no power issue, on full load it consumes about 1000W, I have an 1500W Enermax Platinum - PSU
as bao2 wrote, it works in other renderers
one GPU I have disabled in NVidia control panel to see the difference
MB as posted in threat ... http://www.msi.com/product/mb/P7N-Diamond.html#?div=Detail
Win 7, x64

I am not a developer, but I can read some code, and paste some code ;) ... but I am not a programmer
long time ago I have done some code in basic, C++, OpenGL, php, html, js, and tried some lines in maxscript, mel, OSL ...

But If you paste me the code, path to the file and point me to the line number where it should go I can handle it for sure :)
and test it on my computer

Bao 2

the ID stuff, I am not sure about this, if I disable the 580 in NVidia control panel ( as in Arion) it can handle all four 590 GPUs.
If it would be an ID thing, it would show only 3 GPUs then, wouldn't ?? but maybe I am wrong%%%

%%%Doug: 1) no power issue, on full load it consumes about 1000W, I have an 1500W Enermax Platinum - PSU 2) as bao2 wrote, it works in other renderers 3) one GPU I have disabled in NVidia control panel to see the difference 4) MB as posted in threat ... http://www.msi.com/product/mb/P7N-Diamond.html#?div=Detail 5) Win 7, x64 I am not a developer, but I can read some code, and paste some code ;) ... but I am not a programmer long time ago I have done some code in basic, C++, OpenGL, php, html, js, and tried some lines in maxscript, mel, OSL ... But If you paste me the code, path to the file and point me to the line number where it should go I can handle it for sure :) and test it on my computer Bao 2 the ID stuff, I am not sure about this, if I disable the 580 in NVidia control panel ( as in Arion) it can handle all four 590 GPUs. If it would be an ID thing, it would show only 3 GPUs then, wouldn't ?? but maybe I am wrong%%%

Jozef Zajac commented

2013-07-17 23:51:00 +02:00

%%%the ID stuff, I am not sure about this, if I disable the 580 in NVidia control panel ( as in Arion) it can handle all four 590 GPUs.
If it would be an ID thing, it would show only 3 GPUs then, wouldn't ?? but maybe I am wrong

just to be exact I was talking about CYCLES, that it utilizes all four GPUs in the two 590, when the 580 is disabled in NVidia control panel %%%

%%%the ID stuff, I am not sure about this, if I disable the 580 in NVidia control panel ( as in Arion) it can handle all four 590 GPUs. If it would be an ID thing, it would show only 3 GPUs then, wouldn't ?? but maybe I am wrong just to be exact I was talking about CYCLES, that it utilizes all four GPUs in the two 590, when the 580 is disabled in NVidia control panel %%%

Jozef Zajac commented

2013-07-17 23:56:50 +02:00

%%%You can find me on Skype: blenderslovakia or should I make an ICQ account to get in touch with you gyus ? We can make the communication faster
even we can do a Skypecall and use TeamViewer to make a meeting and you can see my desktop, and test it together%%%

%%%You can find me on Skype: blenderslovakia or should I make an ICQ account to get in touch with you gyus ? We can make the communication faster even we can do a Skypecall and use TeamViewer to make a meeting and you can see my desktop, and test it together%%%

Jozef Zajac commented

2013-07-18 10:04:12 +02:00

%%%Other strange thing that came out from my todays test. Utilizing the GPUs in 3D, Rendered viewport shading, in CYCLES. It seems in viewport it uses all GPUs, in comparison to Render. The loads of GPUs are unequally underutilized (probably because of progressive refine of whole viewport) and loaded as follow.

Rendering Mike Pan's BMW scene in official 2.68 RC1. GPU, Load an Memory usage, measured using GPU-Z

In rendering inside viewport:
580 - 77-83% - 792MB
590 - 11-13% - 446MB
590 - 37-43% - 446MB
590 - 54-63% - 446MB
590 - 56-67% - 446MB

While in F12 rendering with 256x256 tiles
580 - up to 99% - 675MB
590 - 0% - 207MB
590 - up to 99% - 426MB
590 - up to 99% - 426MB
590 - up to 99% - 426MB %%%

%%%Other strange thing that came out from my todays test. Utilizing the GPUs in 3D, Rendered viewport shading, in CYCLES. It seems in viewport it uses all GPUs, in comparison to Render. The loads of GPUs are unequally underutilized (probably because of progressive refine of whole viewport) and loaded as follow. Rendering Mike Pan's BMW scene in official 2.68 RC1. GPU, Load an Memory usage, measured using GPU-Z In rendering inside viewport: 580 - 77-83% - 792MB 590 - 11-13% - 446MB 590 - 37-43% - 446MB 590 - 54-63% - 446MB 590 - 56-67% - 446MB While in F12 rendering with 256x256 tiles 580 - up to 99% - 675MB 590 - 0% - 207MB 590 - up to 99% - 426MB 590 - up to 99% - 426MB 590 - up to 99% - 426MB %%%

Jozef Zajac commented

2013-07-19 08:35:14 +02:00

%%%Great job with 2.68, the improvements in Cycles is really awesome.
I am really curious, how we will make it even faster ;)%%%

%%%Great job with 2.68, the improvements in Cycles is really awesome. I am really curious, how we will make it even faster ;)%%%

Jozef Zajac commented

2013-07-20 10:41:09 +02:00

%%%Hi guys, so the drive is in, linux installed Ubuntu 13.04, NVidia drivers 304.

I had pretty hard times with unstability of fresh install, untill i installed the drivers exactly this way:
http://falkvinge.net/2013/02/15/how-to-install-nvidia-drivers-in-ubuntu-12-10-quantal/

now it is stable, running Blender 2.68, and the GPU problem is the same
Blender finds all 5GPUs, but renders tiles only with 4 of them%%%

%%%Hi guys, so the drive is in, linux installed Ubuntu 13.04, NVidia drivers 304. I had pretty hard times with unstability of fresh install, untill i installed the drivers exactly this way: http://falkvinge.net/2013/02/15/how-to-install-nvidia-drivers-in-ubuntu-12-10-quantal/ now it is stable, running Blender 2.68, and the GPU problem is the same Blender finds all 5GPUs, but renders tiles only with 4 of them%%%

Jozef Zajac commented

2013-07-20 17:44:35 +02:00

%%%A little upgrade again, 3x 590, from 6GPUs Cycles utilizes only 4. Arion uses all of them.
http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why/page3

looking for feedback and progress :)%%%

%%%A little upgrade again, 3x 590, from 6GPUs Cycles utilizes only 4. Arion uses all of them. http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why/page3 looking for feedback and progress :)%%%

Jozef Zajac commented

2013-07-20 22:46:55 +02:00

%%%It would be great to have a checkbox list on the available devices, in Blender Preferences / Compute device dropdown
as it is in NVIDIA Control Panel / Manage 3D Settings / Global Settings / CUDA - GPUs / in All dropdown list
to have the full ability to assign various combination of devices to be used for rendering.%%%

%%%It would be great to have a checkbox list on the available devices, in Blender Preferences / Compute device dropdown as it is in NVIDIA Control Panel / Manage 3D Settings / Global Settings / CUDA - GPUs / in All dropdown list to have the full ability to assign various combination of devices to be used for rendering.%%%

Bao 2 commented

2013-07-22 13:48:19 +02:00

%%%Brecht, take a look to my post 63 here:
http://www.blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why&p=2428638&viewfull=1#post2428638%%%

%%%Brecht, take a look to my post 63 here: http://www.blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why&p=2428638&viewfull=1#post2428638%%%

Jozef Zajac commented

2013-07-23 23:54:17 +02:00

%%%Bao 2 ... suggested a workaround, that worked out ... take a look on the my post #68
http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why&p=2429679&viewfull=1#post2429679

It is only a workaround, it sill needs to be bug-fixed in the automatic detection

%%%

%%%Bao 2 ... suggested a workaround, that worked out ... take a look on the my post #68 http://blenderartists.org/forum/showthread.php?300318-CYCLES-NVidia-580-GTX-2-x-590-GTX-uses-only-4-GPUs-why&p=2429679&viewfull=1#post2429679 It is only a workaround, it sill needs to be bug-fixed in the automatic detection %%%

Brecht Van Lommel commented

2013-07-24 00:28:17 +02:00

%%%Ah, this indeed explains the issue very well, I wasn't see the issue because I have 8 threads here by default. Should be easy to fix then.%%%