Cycles GPU Performance Regression on Titan #38679

New Issue

Gabriel Caraballo · 2014-02-17T14:26:26+01:00

Gabriel Caraballo commented

2014-02-17 14:26:26 +01:00

System Information
Ubuntu 12.04, kernel 3.2.0-58-generic
Intel® Xeon(R) CPU E5620 @ 2.40GHz × 16
7,7 GiB RAM
GeForce GT 430
NVIDIA Corporation GK110 [GeForce GTX Titan] (rev a1)

NVIDIA driver version: 331.20
X.Org: 1.11.3

Blender Version
Broken: fdcdd5e
Worked: 2.69

Short description of error
Rendering a project with GeForce GTX Titan (only) on Blender fdcdd5e takes double time than on Blender 2.69.

Exact steps for others to reproduce the error

Open Blender 2.69
Change Render Engine to Cycles
Set Render Samples to 100
Render time: 2 seconds
Open Blender fdcdd5e
Change Render Engine to Cycles
Set Render Samples to 100
Render time: 4 seconds

**System Information** Ubuntu 12.04, kernel 3.2.0-58-generic Intel® Xeon(R) CPU E5620 @ 2.40GHz × 16 7,7 GiB RAM GeForce GT 430 NVIDIA Corporation GK110 [GeForce GTX Titan] (rev a1) NVIDIA driver version: 331.20 X.Org: 1.11.3 **Blender Version** Broken: fdcdd5e Worked: 2.69 **Short description of error** Rendering a project with GeForce GTX Titan (only) on Blender fdcdd5e takes double time than on Blender 2.69. **Exact steps for others to reproduce the error** * Open Blender 2.69 * Change Render Engine to Cycles * Set Render Samples to 100 * Render time: 2 seconds * Open Blender fdcdd5e * Change Render Engine to Cycles * Set Render Samples to 100 * Render time: 4 seconds

Gabriel Caraballo commented

2014-02-17 14:26:26 +01:00

Changed status to: 'Open'

Gabriel Caraballo commented

2014-02-17 14:26:26 +01:00

Added subscriber: @GabrielCaraballo

Gabriel Caraballo commented

2014-02-17 14:31:12 +01:00

On GPU.

The name of the bug must be: Cycles GPU Performance Regression (I can't change it now)
The regression is only rendering on GPU.

On GPU. The name of the bug must be: Cycles GPU Performance Regression (I can't change it now) The regression is only rendering on GPU.

Brecht Van Lommel changed title from ~~Cycles Performance Regression~~ to Cycles GPU Performance Regression

2014-02-17 14:34:46 +01:00

Brecht Van Lommel commented

2014-02-17 14:34:46 +01:00

Added subscriber: @MartijnBerger

Brecht Van Lommel commented

2014-02-17 14:37:47 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2014-02-17 14:37:47 +01:00

There was a change to avoid CPU load at 100% during GPU render, which may have a big impact on very simple renders like the default cube. However for more complex scenes the impact should be be a lot less.

Can you test what kind of impact there is on more complex scenes?

There was a change to avoid CPU load at 100% during GPU render, which may have a big impact on very simple renders like the default cube. However for more complex scenes the impact should be be a lot less. Can you test what kind of impact there is on more complex scenes?

Gabriel Caraballo commented

2014-02-17 14:45:55 +01:00

Sure Brecht, a Kiribati frame takes 5 minutes with fdcdd5e, and 2:30 minutes on 2.69

Sure Brecht, a Kiribati frame takes **5** minutes with fdcdd5e, and **2:30** minutes on **2.69**

Brecht Van Lommel commented

2014-02-17 14:51:22 +01:00

Thanks, that definitely needs to be fixed then.

Sergei Smyslov commented

2014-02-17 14:52:21 +01:00

Added subscriber: @Luarvik

Sergei Smyslov commented

2014-02-17 14:52:21 +01:00

@GabrielCaraballo
How about more complex scene with longer rendering time? Is it still double rate time difference? I have GTX 560Ti 2Gb. Comparing Blender 2.69.0 and today's buildbot version I got "only" 10% regression.

@GabrielCaraballo How about more complex scene with longer rendering time? Is it still double rate time difference? I have GTX 560Ti 2Gb. Comparing Blender 2.69.0 and today's buildbot version I got "only" 10% regression.

Martijn Berger commented

2014-02-17 15:19:18 +01:00

For me the regression depends very much on the OS and or drivers and the average time a tile takes.

The time we lose should be per tile. We could en-queue the next tile for single GPU case to take away most of the pain.
But proper a-syncify-ing device_cuda takes a non trivial effort and it on my todo-list.

@brecht Ill be on irc tonight if you want to talk about this. but we can always revert to busy waiting or make it user definable

For me the regression depends very much on the OS and or drivers and the average time a tile takes. The time we lose should be per tile. We could en-queue the next tile for single GPU case to take away most of the pain. But proper a-syncify-ing device_cuda takes a non trivial effort and it on my todo-list. @brecht Ill be on irc tonight if you want to talk about this. but we can always revert to busy waiting or make it user definable

Brecht Van Lommel commented

2014-02-17 15:53:44 +01:00

If we can't find a fix quickly we should indeed revert this change and move it to the next release.

Martijn Berger commented

2014-02-17 18:09:41 +01:00

This is a possible fix

Brecht Van Lommel commented

2014-02-17 18:52:30 +01:00

This patch seems to work well for me on OS X. I tried rendering the default cube with 100 samples and the default (small) tile size, and it does make a big performance difference there. For more complex scenes I didn't notice that much difference but it worked fine.

Viewport rendering seems to be stable here as well, so it might be good to commit this and get feedback from more people.

This patch seems to work well for me on OS X. I tried rendering the default cube with 100 samples and the default (small) tile size, and it does make a big performance difference there. For more complex scenes I didn't notice that much difference but it worked fine. Viewport rendering seems to be stable here as well, so it might be good to commit this and get feedback from more people.

Martijn Berger commented

2014-02-17 19:13:42 +01:00

This fix adds some extra pandemic sync() for the case where we cancel the task.

@GabrielCaraballo do you have an option to compile with a patch or should I make a build for you to test with ?

This fix adds some extra pandemic sync() for the case where we cancel the task. @GabrielCaraballo do you have an option to compile with a patch or should I make a build for you to test with ?

Gabriel Caraballo commented

2014-02-17 19:24:26 +01:00

If you can make a build will be great @juicyfruit.

Martijn Berger commented

2014-02-17 20:17:39 +01:00

@GabrielCaraballo after consulting with @brecht I committed a posible fix f1aeb2ccf4

tomorrows http://builder.blender.org/ builds should be good for testing this.

@GabrielCaraballo after consulting with @brecht I committed a posible fix f1aeb2ccf4 tomorrows http://builder.blender.org/ builds should be good for testing this.

Brecht Van Lommel commented

2014-02-17 20:31:49 +01:00

The latest 64 bit linux build here now has the fix:
http://builder.blender.org/download/

The latest 64 bit linux build here now has the fix: http://builder.blender.org/download/

Gabriel Caraballo commented

2014-02-17 22:13:37 +01:00

Thanks @Brecht. New Render Times:

Blender	2k half	2k
`f1aeb2c`	2:31	9:25
2.69	1.29	5:35

Tile Size: 256x215

Two Render Layers, Foreground and Background. Some compositing for color correction. (Compositing time is the same, don't affect)

Thanks @Brecht. New Render Times: | Blender | 2k half | 2k | | -- | -- | -- | | f1aeb2c| 2:31 | 9:25 | | **2.69** | 1.29 | 5:35 | Tile Size: 256x215 Two Render Layers, Foreground and Background. Some compositing for color correction. (Compositing time is the same, don't affect)

Martijn Berger commented

2014-02-17 22:19:37 +01:00

wow . that is bad...

I have a problem that I cannot reproduce a case where I get such bad number.

wow . that is bad... I have a problem that I cannot reproduce a case where I get such bad number.

Martijn Berger commented

2014-02-17 22:38:57 +01:00

The closest I have to your hardware is a GTX 780 (the plain one) and it is windows

But there in BMW M1 and Barcelona I have almost no regression even without the proposed fix.
I do use larger tiles ~90k pixels per tile tiles are all same size. (h/4 , w/4)

The closest I have to your hardware is a GTX 780 (the plain one) and it is windows But there in BMW M1 and Barcelona I have almost no regression even without the proposed fix. I do use larger tiles ~90k pixels per tile tiles are all same size. (h/4 , w/4)

Thomas Dinges commented

2014-02-17 22:43:41 +01:00

Added subscriber: @ThomasDinges

Thomas Dinges commented

2014-02-17 22:43:41 +01:00

That is weird, both the Titan and the GTX 780 are sm_35 and should therefore use the same code...

Windows/Linux issue? We should be able to find other Titan users, who can confirm/disconfirm the issue.

That is weird, both the Titan and the GTX 780 are sm_35 and should therefore use the same code... Windows/Linux issue? We should be able to find other Titan users, who can confirm/disconfirm the issue.

Martijn Berger commented

2014-02-17 22:51:39 +01:00

After the fix it should be at most a few ms per tile. say there are 50 tiles per frame and 2 layers. then 1 sec is the absolute max regression there should be.

I would want to work towards a scene / case where I can reproduce and fix the problem. If sharing the problem scene is not possible maybe @GabrielCaraballo can also try BMW or Barcelona so we can bisect the issue some more.

After the fix it should be at most a few ms per tile. say there are 50 tiles per frame and 2 layers. then 1 sec is the absolute max regression there should be. I would want to work towards a scene / case where I can reproduce and fix the problem. If sharing the problem scene is not possible maybe @GabrielCaraballo can also try BMW or Barcelona so we can bisect the issue some more.

Wolfgang Faehnle commented

2014-02-17 23:25:33 +01:00

Added subscriber: @mib2berlin

Wolfgang Faehnle commented

2014-02-17 23:25:33 +01:00

Hi, I also cant reproduce the regression with BMW and Barcelona files.

Opensuse 13.1/64
Intel i5 3770K
GTX 760
GTX 560Ti 448 Cores
Driver 331.20

BMW, set tiles to 256x256:

http://www.blenderartists.org/forum/showthread.php?239480-2-6x-Cycles-render-benchmark

Barcelona is not available from orginal BA thread, uploaded on pastall.
Please switch scene to Benchmark1 GPU.

http://www.pasteall.org/blend/26908

@GabrielCaraballo, Hi, I think it is not possible for you to upload sample file, maybe a simplified one?

Cheers, mib.

Hi, I also cant reproduce the regression with BMW and Barcelona files. Opensuse 13.1/64 Intel i5 3770K GTX 760 GTX 560Ti 448 Cores Driver 331.20 BMW, set tiles to 256x256: http://www.blenderartists.org/forum/showthread.php?239480-2-6x-Cycles-render-benchmark Barcelona is not available from orginal BA thread, uploaded on pastall. Please switch scene to Benchmark1 GPU. http://www.pasteall.org/blend/26908 @GabrielCaraballo, Hi, I think it is not possible for you to upload sample file, maybe a simplified one? Cheers, mib.

Martijn Berger commented

2014-02-17 23:39:19 +01:00

System windows 7 x64.
GTX 780 ( nvidia latest forgot to check exact number)

Default cube 1920x1080 * 1/4
500 samples tiles 16x16 -> 510 tiles

2.69 release	trunk without fix	trunk + fix
2:28:90	3:13:71	2:04:07

System windows 7 x64. GTX 780 ( nvidia latest forgot to check exact number) Default cube 1920x1080 * 1/4 500 samples tiles 16x16 -> 510 tiles | 2.69 release | trunk without fix | trunk + fix| | -- | -- | -- | | 2:28:90| 3:13:71 | 2:04:07 |

Gabriel Caraballo commented

2014-02-17 23:50:59 +01:00

Added subscriber: @jpbouza-4

Gabriel Caraballo commented

2014-02-17 23:50:59 +01:00

@jpbouza-4, can you reproduce this issue?

Martijn Berger commented

2014-02-18 09:06:15 +01:00

@brecht I am beginning to suspect that it might not be the changes to device_cuda.

Those are not the only thing that changed. I don't have a ubuntu 12.04 machine to build a few build but it would be good to bisect this.

@brecht I am beginning to suspect that it might not be the changes to device_cuda. Those are not the only thing that changed. I don't have a ubuntu 12.04 machine to build a few build but it would be good to bisect this.

Juan Pablo Bouza commented

2014-02-18 14:33:12 +01:00

OK

System Specs:

i7 3930k
32G RAM
Geforce TITAN
Kubuntu 13.04

Cube with 100 samples and just 1 tile (1920x1080)

2.69 ----- 2.73 secs
f91368d ------ 3.47 secs
today's build 688d098 ----- 2.51

So, apparently the fix was merged into trunk and now the performance is better than in 2.69.

The only peculiar thing I can mention is that perormance seems to be the opposite with 64x64 tiles, as in the latest build it takes 7 sec, while in the other build and in 2.69 it takes 5...

OK System Specs: i7 3930k 32G RAM Geforce TITAN Kubuntu 13.04 Cube with 100 samples and just 1 tile (1920x1080) 2.69 ----- 2.73 secs f91368d ------ 3.47 secs today's build 688d098 ----- 2.51 So, apparently the fix was merged into trunk and now the performance is better than in 2.69. The only peculiar thing I can mention is that perormance seems to be the opposite with 64x64 tiles, as in the latest build it takes 7 sec, while in the other build and in 2.69 it takes 5...

Martijn Berger commented

2014-02-18 14:55:18 +01:00

@jpbouza-4 thx for these numbers.

@GabrielCaraballo it would be good to know what it is in your scene that causes this or what change to cycles causes this for you. Can you share any details on the scene/ shaders anything

Does it also happen if you override your materials with a diffuse 80 % grey shader for example ?

@jpbouza-4 thx for these numbers. @GabrielCaraballo it would be good to know what it is in your scene that causes this or what change to cycles causes this for you. Can you share any details on the scene/ shaders anything Does it also happen if you override your materials with a diffuse 80 % grey shader for example ?

Brecht Van Lommel commented

2014-02-18 15:23:53 +01:00

So, it seems that at the moment we have no way to reproduce the problem, with different scenes on different graphics cards.

@GabrielCaraballo, would it be possible to send me a .blend file with this problem privately to brecht@blender.org?

So, it seems that at the moment we have no way to reproduce the problem, with different scenes on different graphics cards. @GabrielCaraballo, would it be possible to send me a .blend file with this problem privately to brecht@blender.org?

Gabriel Caraballo commented

2014-02-18 16:21:43 +01:00

This comment was removed by @GabrielCaraballo

*This comment was removed by @GabrielCaraballo*

blender-admin commented

2014-02-19 16:08:08 +01:00

This issue was referenced by blender/blender-addons-contrib@6b1a4fc66e

This issue was referenced by blender/blender-addons-contrib@6b1a4fc66ef4e3197601318ce4c36db2c8359b98

blender-admin commented

2014-02-19 16:08:08 +01:00

This issue was referenced by 6b1a4fc66e

This issue was referenced by 6b1a4fc66ef4e3197601318ce4c36db2c8359b98

Brecht Van Lommel commented

2014-02-19 16:21:05 +01:00

I got a production file now from @GabrielCaraballo to test this, but couldn't confirm any major impact testing with 460 GTX and 650M GT on Ubuntu Linux.

It may be a different issue, but given that #38712 (Cycles render is not updating) also is a problem, I reverted these busywait changes for now. I rather postpone it than trying to find fixes at the last moment. If you test the next build, then we will at least know if this is the cause of the problem.

I got a production file now from @GabrielCaraballo to test this, but couldn't confirm any major impact testing with 460 GTX and 650M GT on Ubuntu Linux. It may be a different issue, but given that #38712 (Cycles render is not updating) also is a problem, I reverted these busywait changes for now. I rather postpone it than trying to find fixes at the last moment. If you test the next build, then we will at least know if this is the cause of the problem.

Gabriel Caraballo commented

2014-02-19 16:27:43 +01:00

More times with 688d098:

Resolution: 2048x858 100%
Tile Size: 256x215

688d098= 5:07 minutes
2.69= 4:03 minutes

Resolution: 2048x858 100%
Tile Size: 2048x858

688d098= 4:07 minutes
2.69= 4:21 minutes

Same times for me and @jpbouza-4 .

jpbouza will try 6b1a4fc66e .

More times with 688d098: Resolution: 2048x858 100% Tile Size: 256x215 - 688d098= **5:07** minutes - 2.69= **4:03** minutes Resolution: 2048x858 100% Tile Size: 2048x858 - 688d098= **4:07** minutes - 2.69= **4:21** minutes Same times for me and @jpbouza-4 . jpbouza will try 6b1a4fc66e .

Thomas Dinges commented

2014-02-20 21:32:48 +01:00

Is the regression fixed for you guys?

Gabriel Caraballo commented

2014-02-20 22:01:35 +01:00

Render times with e7f3424 :

Resolution: 2048x858 100%
Tile Size: 256x215

e7f3424 = 5:22 minutes
2.69 = 4:03 minutes

Resolution: 2048x858 100%
Tile Size: 2048x858

e7f3424 = 5:45 minutes
2.69 = 4:21 minutes

Not fixed :(

Render times with e7f3424 : Resolution: 2048x858 100% Tile Size: 256x215 - e7f3424 = **5:22** minutes - 2.69 = **4:03** minutes Resolution: 2048x858 100% Tile Size: 2048x858 - e7f3424 = **5:45** minutes - 2.69 = **4:21** minutes Not fixed :(

Thomas Dinges commented

2014-02-21 16:16:47 +01:00

Then I run out of ideas here.
We need someone else who can confirm the issue (maybe it's a Titan only issue).

@GabrielCaraballo You mention that you have a Geforce 430 as well, can you run some tests there? This way we would know if its a Titan only issue or maybe something special on your system.

Then I run out of ideas here. We need someone else who can confirm the issue (maybe it's a Titan only issue). @GabrielCaraballo You mention that you have a Geforce 430 as well, can you run some tests there? This way we would know if its a Titan only issue or maybe something special on your system.

Gabriel Caraballo commented

2014-02-21 19:53:05 +01:00

@ThomasDinges , with the GeForce 430:

Dimensions: 2048x858, Tile size: 256x215

e7f3424 = 13:44 minutes
2.69 = 13:25 minutes

Apparently is a TITAN specific issue.

@ThomasDinges , with the GeForce 430: Dimensions: 2048x858, Tile size: 256x215 - e7f3424 = **13:44** minutes - **2.69** = **13:25** minutes Apparently is a TITAN specific issue.

Brecht Van Lommel changed title from ~~Cycles GPU Performance Regression~~ to Cycles GPU Performance Regression on Titan

2014-03-03 18:32:28 +01:00

Thomas Dinges commented

2014-03-22 01:09:29 +01:00

Maybe this is the same issue as here? https://developer.blender.org/T39089

Do you have any nodes connected to the Volume Output in those files? If so, remove the link.

Maybe this is the same issue as here? https://developer.blender.org/T39089 Do you have any nodes connected to the Volume Output in those files? If so, remove the link.

Brecht Van Lommel was assigned by Sergey Sharybin

2014-04-10 18:43:14 +02:00

Sergey Sharybin commented

2014-04-10 18:43:14 +02:00

Added subscriber: @Sergey

Sergey Sharybin commented

2014-04-10 18:43:14 +02:00

@brecht, i would ask you to keep track of this report. Currently it looks like being abandoned.

Maverick commented

2014-04-16 22:04:11 +02:00

Added subscriber: @dragostanasie

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Cycles GPU Performance Regression on Titan #38679