Page MenuHome

Blender Open DataExperimental
ActivePublic

Members (6)

Details

Description

Blender Open Data is a platform to collect and display the results of hardware and software performance tests, analytics, telemetry and more. The infrastructure is made of two components:

  • Open Data: public portal where all data samples are available to view and download
  • My Data: portal where user can manage the samples they submit to the Open Data portal

The first purpose of the Blender Open Data project is to host Blender Benchmark results. We are currently in Beta phase and we would love to hear your feedback!

Reporting an issue?

Please fill in the form. Thanks!

Feedback and ideas?

Join the discussion on devtalk.blender.org.

Recent Activity

Fri, Feb 14

ronsn (ronsn) added a comment to T73787: Security issue.

@Brecht Van Lommel (brecht) uff, sorry, I didn't realize that.
@rob (rob_on_earth) you should remove your last comment, because it is still visible in the edit history.

Fri, Feb 14, 12:57 AM · Blender Open Data
rob (rob_on_earth) added a comment to T73787: Security issue.
Fri, Feb 14, 12:46 AM · Blender Open Data
Brecht Van Lommel (brecht) closed T73787: Security issue as Resolved.
Fri, Feb 14, 12:45 AM · Blender Open Data
Brecht Van Lommel (brecht) added a comment to T73787: Security issue.

@ ronsn (ronsn), the point is to report the security issue privately.

Fri, Feb 14, 12:41 AM · Blender Open Data
ronsn (ronsn) added a comment to T73787: Security issue.
  • Where,
  • when,
  • what exactly,
  • what are your system specs,
  • which version(s),
  • screenscots,
  • exact steps for other to reproduce this issue
Fri, Feb 14, 12:18 AM · Blender Open Data
rob (rob_on_earth) created T73787: Security issue.
Fri, Feb 14, 12:02 AM · Blender Open Data
rob (rob_on_earth) added a comment to T73648: Benchmark fails for scene Victor.

I tried the Victor scene with an unrelated Windows laptop with 16GB ram and it had the same result.

Fri, Feb 14, 12:00 AM · Blender Benchmark, Blender Open Data

Thu, Feb 13

Patrick Mours (pmoursnv) added a comment to T73322: Suggestion to improve "Top 50 devices" list.

@Sem Mulder (SemMulder) How about this for the Top 50 query:

SELECT rank() OVER (ORDER BY sum(per_benchmark.median_render_time)) AS rank,
       per_benchmark.device_name                                    AS device_name,
       json_object_agg(per_benchmark.benchmark,
                       per_benchmark.median_render_time)            AS median_render_times,
       sum(per_benchmark.number_of_samples)                         AS number_of_samples
FROM (
         SELECT rank() OVER (PARTITION BY per_device_type.device_name, per_device_type.benchmark ORDER BY per_device_type.median_render_time) as rank,*
         FROM (
                  SELECT percentile_cont(0.5)
                         WITHIN GROUP (ORDER BY opendata_main_benchmark.render_time) AS median_render_time,
                         opendata_main_benchmark.device_name                         AS device_name,
                         opendata_main_benchmark.device_type                         AS device_type,
                         opendata_main_benchmark.benchmark                           AS benchmark,
                         count(*)                                                    AS number_of_samples
                  FROM opendata_main_benchmark
                  WHERE opendata_main_benchmark.benchmark IN %(benchmarks)s AND opendata_main_benchmark.device_name NOT IN %(device_name_blacklist)s
                  GROUP BY opendata_main_benchmark.device_name, opendata_main_benchmark.device_type, opendata_main_benchmark.benchmark
                  HAVING count(*) >= %(minimum_number_of_samples_per_benchmark)s
              ) per_device_type
     ) per_benchmark
WHERE per_benchmark.rank = 1
GROUP BY per_benchmark.device_name
HAVING count(per_benchmark.benchmark) = %(number_of_benchmarks)s
ORDER BY rank
LIMIT 50;

This adds one additional step to the query which selects the better device type for a device (see the extra SELECT rank() in there, the rest is pretty much identical to the old query). It will only have an effect on RTX GPUs currently, which support two device types (CUDA and OptiX). But it ensures the median render time is only calculated for the better option instead of calculating it over all device types.

Thu, Feb 13, 6:35 PM · Blender Open Data

Wed, Feb 12

Sem Mulder (SemMulder) changed the status of T73657: Benchmark GUI launcher does not run on Quadro cards from Needs Triage to Confirmed.

Hmm, this is strange. I am able to reproduce this on a Quadro RTX 5000. I will look into attaching a debugger on Windows.

Wed, Feb 12, 9:41 AM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73322: Suggestion to improve "Top 50 devices" list.

Yet on the home page, the GeForce RTX 2080 Ti shows 38.63 for the bmw27 scene (which is close to the CUDA number) and the GeForce RTX 2080 SUPER shows 28.94 (which is close to the OptiX number).

Wed, Feb 12, 9:36 AM · Blender Open Data
Sem Mulder (SemMulder) closed T73648: Benchmark fails for scene Victor as Resolved.

I believe @Ray molenkamp (LazyDodo) is correct here. We are currently still investigating how to detect this properly (detecting the amount of GPU memory available across all brands is a bit complicated). Closing this for now as this is not an issue with the launcher per se.

Wed, Feb 12, 9:31 AM · Blender Benchmark, Blender Open Data

Mon, Feb 10

Francesco Siddi (fsiddi) added a comment to T73322: Suggestion to improve "Top 50 devices" list.

@Sem Mulder (SemMulder) mind having a look at the last remark here?

Mon, Feb 10, 10:44 AM · Blender Open Data

Fri, Feb 7

Patrick Mours (pmoursnv) created T73657: Benchmark GUI launcher does not run on Quadro cards.
Fri, Feb 7, 2:42 PM · Blender Open Data
Patrick Mours (pmoursnv) added a comment to T73322: Suggestion to improve "Top 50 devices" list.

There is some weird behavior with the results currently displayed on http://opendata.blender.org:


It shows the GeForce RTX 2080 SUPER as being the fastest GPU, even though e.g. the GeForce RTX 2080 Ti has better results. Looks like this is because it uses the OptiX results for the SUPER, but the CUDA results for all other RTX GPUs to determine rank.

Fri, Feb 7, 1:59 PM · Blender Open Data

Thu, Feb 6

Ray molenkamp (LazyDodo) added a comment to T73648: Benchmark fails for scene Victor.

peak memory for victor on my box is 16774.97M that 16GB of ram may be just a little short there.

Thu, Feb 6, 10:08 PM · Blender Benchmark, Blender Open Data
rob (rob_on_earth) updated the task description for T73648: Benchmark fails for scene Victor.
Thu, Feb 6, 9:49 PM · Blender Benchmark, Blender Open Data
rob (rob_on_earth) created T73648: Benchmark fails for scene Victor.
Thu, Feb 6, 9:24 PM · Blender Benchmark, Blender Open Data

Wed, Jan 29

Francesco Siddi (fsiddi) added a comment to T73452: Document location of scene caches for benchmark runner.

Yes that would be preferable!

Wed, Jan 29, 6:14 PM · Blender Open Data
Patrick Mours (pmoursnv) added a comment to T73322: Suggestion to improve "Top 50 devices" list.

You can still send in results for CUDA on RTX GPUs right now though, in addition to OptiX. It's only that those ideally should not affect the overall rank, since it's the worse option.

Wed, Jan 29, 12:07 PM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73322: Suggestion to improve "Top 50 devices" list.

Another approach might be to limit the results on the homepage to recent results (e.g. one year).

Wed, Jan 29, 12:01 PM · Blender Open Data
Sem Mulder (SemMulder) closed T73454: Do not include barbershop_interior in benchmark as Resolved.

Implemented in rBODb9b62db1ed307fdbcabcd2ef82d23940589d76c8 and deployed.

Wed, Jan 29, 10:56 AM · Blender Open Data
Sem Mulder (SemMulder) closed T73455: Tweaks to "Top 50 devices" table as Resolved.

This is now live.

Wed, Jan 29, 10:55 AM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73452: Document location of scene caches for benchmark runner.

Would it not be better to include something in the launcher for this? It would not be that hard to add a button that allows the user to clear the cache.

Wed, Jan 29, 10:34 AM · Blender Open Data

Tue, Jan 28

Francesco Siddi (fsiddi) changed the status of T73455: Tweaks to "Top 50 devices" table from Needs Triage to Confirmed.
Tue, Jan 28, 3:21 PM · Blender Open Data
Francesco Siddi (fsiddi) changed the status of T73454: Do not include barbershop_interior in benchmark from Needs Triage to Confirmed.
Tue, Jan 28, 3:08 PM · Blender Open Data
Francesco Siddi (fsiddi) changed the status of T73452: Document location of scene caches for benchmark runner from Needs Triage to Confirmed.
Tue, Jan 28, 9:32 AM · Blender Open Data

Mon, Jan 27

Sem Mulder (SemMulder) closed T66415: All GPU tests crash on Nvidia RTX 2070 windows64 (1.0beta2) as Resolved.

Closing this for now, feel free to open a new ticket if the problem occurs with the new launcher as well.

Mon, Jan 27, 3:18 PM · Blender Open Data
Sem Mulder (SemMulder) closed T73291: Benchmark does not run as Resolved.

I have no idea why am I missing the path variable if it's supposed to be there by default. I haven't removed any from the defaults. This is also a clean Windows 1909 installed on 24/12/2019 and set-up "as new" without any Microsoft account, back-ups or etc.

Mon, Jan 27, 2:37 PM · Blender Open Data
Boyan (siegmour) added a comment to T73291: Benchmark does not run.

@Sem Mulder (SemMulder) Hi Sem! Thank you very much for looking into this. I added "C:\windows\system32\wbem\" and "%SystemRoot%\System32\Wbem" to path and it did indeed resolve the issue. The old blender benchmark wasn't working either (version 1.0 beta 2, missing "Start" buttons) and it fixed that one as well!

Mon, Jan 27, 2:29 PM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73291: Benchmark does not run.

Looking deeper into this it is likely that a missing %SystemRoot%\System32\Wbem entry in the PATH will break various other stuff on Windows. @Boyan (siegmour) can you add this entry back into you PATH and try again?

Mon, Jan 27, 2:18 PM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73291: Benchmark does not run.

It seems to be an issue with the PATH environment variable. Probably hardcoding the path C:\windows\system32\wbem\wmic.exe will resolve this.

Mon, Jan 27, 2:13 PM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73291: Benchmark does not run.

Which version of Windows are you using?

Mon, Jan 27, 2:08 PM · Blender Open Data
Patrick Mours (pmoursnv) added a comment to T73325: Improve device selection in benchmark.

Restricting to GPUs of the same kind makes for cleaner benchmark results I think. Otherwise the data is not likely to be all that useful, because in heterogeneous systems you can no longer figure out which device is really driving the results (e.g. in a system with a Turing GPU and a Maxwell GPU, the former will be heavily penalized by the latter). And it would avoid having to change the presentation of the data much, since results can just be collected under new "2/4/...x GPU Name" devices, similar to the CPUs.

Mon, Jan 27, 1:35 PM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73325: Improve device selection in benchmark.

I'm open to allowing multiple GPUs. We do need to figure out some stuff though:

  • Do we only allow GPUs of the same kind, as you mentioned?
  • Do we allow the user to pick the number of cards? (if so, how?)
Mon, Jan 27, 11:38 AM · Blender Open Data
Sem Mulder (SemMulder) closed T73313: Include OptiX in fastest GPU chart in frontpage as Resolved.

This is now live.

Mon, Jan 27, 11:35 AM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73291: Benchmark does not run.

Looking at the logs corresponding to the error ids it seems that it cannot find the wmic utility (which we use to query the CPU information). Which version of Windows are you using?

Mon, Jan 27, 9:50 AM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T73322: Suggestion to improve "Top 50 devices" list.

The simplest solution would be to query and display the fastest result for a device, instead of taking the median over all the results. But that may not be accurate enough, depending on how stable the results are.

Mon, Jan 27, 9:44 AM · Blender Open Data
Campbell Barton (campbellbarton) edited projects for T73421: Blender Benchmark: CLI only works in batch if you run it once interactively, added: Blender Open Data; removed BF Blender.
Mon, Jan 27, 5:37 AM · Blender Benchmark

Fri, Jan 24

Aaron Carlisle (Blendify) closed T71129: Could not submit results (unable to connect) as Resolved.
Fri, Jan 24, 6:48 AM · Blender Open Data
Aaron Carlisle (Blendify) closed T70380: Benchmark doesn't run, quick or full, displays "collecting system information" only. as Resolved.
Fri, Jan 24, 6:47 AM · Blender Open Data
Aaron Carlisle (Blendify) closed T56392: barbershop_interior test crashes on GeForce GTX 1080 Ti as Resolved.
Fri, Jan 24, 6:43 AM · Blender Open Data

Wed, Jan 22

Patrick Mours (pmoursnv) created T73325: Improve device selection in benchmark.
Wed, Jan 22, 6:57 PM · Blender Open Data
Francesco Siddi (fsiddi) changed the status of T73322: Suggestion to improve "Top 50 devices" list from Needs Triage to Confirmed.
Wed, Jan 22, 6:38 PM · Blender Open Data
Francesco Siddi (fsiddi) changed the status of T73313: Include OptiX in fastest GPU chart in frontpage from Needs Triage to Confirmed.
Wed, Jan 22, 12:51 PM · Blender Open Data

Tue, Jan 21

Boyan (siegmour) created T73291: Benchmark does not run.
Tue, Jan 21, 12:04 PM · Blender Open Data

Jan 14 2020

Francesco Siddi (fsiddi) renamed T71129: Could not submit results (unable to connect) from Could nog submit results (unable to connect) to Could not submit results (unable to connect).
Jan 14 2020, 11:59 AM · Blender Open Data
Francesco Siddi (fsiddi) closed T56306: Customizable dashboards as Invalid.
Jan 14 2020, 11:58 AM · Blender Open Data
Sem Mulder (SemMulder) closed T56321: Verified Results as Resolved.

This is implemented in the new website.

Jan 14 2020, 11:01 AM · Blender Open Data
Sem Mulder (SemMulder) added a comment to T56304: Benchmarks should be Samples.

more generic data structure

Jan 14 2020, 10:58 AM · Blender Open Data
Sem Mulder (SemMulder) closed T56730: Improve error reporting when deleting samples in My Data as Resolved.

Closing this as I think this is no longer relevant for the new website. Feel free to reopen if it is.

Jan 14 2020, 10:52 AM · Blender Open Data