Cycles: Adaptive rendering based on a norm of rendered pixels
Needs ReviewPublic

Authored by Milan Jaros (jar091) on May 9 2017, 5:18 PM.

Details

Summary

This patch contains very simple modification for adaptive rendering. Now it works only for CPUDevice, but it is expandable for other types of devices.

The settings is placed under Sampling Category and contains this values:

Norm - || rgb_current - rgb_previous || - distance between two colors of same pixels after "Step" samples
Step - the step of checking in loop over samples
Confirm - the count of repeated checks
Log - it helps for finding "Norm" value

BPT - AA_Samples - maximum samples
PT - Samples - maximum samples

This patch can save lot of rendering times.

Tests: http://blender.it4i.cz/research/adaptive-rendering-based-on-a-norm-of-rendered-pixels/

I used this patch for the rendering of Agent327 in the some scenes: http://blender.it4i.cz/rendering/agent-327-operation-barbershop/

Diff Detail

Repository
rB Blender

There was a bad type of adaptive_sampling_use. I changed from int to bool.

Any chance you can update this to current master to test?

Adaptive sampling is certainly a feature we want to have in Cycles. I think we need a more advanced implementation than this though, with fewer users parameters, but mainly something is more robust. Looking only within a single pixel with a simple metric like this is not very reliable and will give artifacts in common cases. Maybe this is good enough, but then we'd really need to see some renders to prove that.

The recent denoiser infrastructure might help to get a better variance estimate, and to access pixels from neighbouring tiles by keeping them around.

Adaptive sampling is certainly a feature we want to have in Cycles. I think we need a more advanced implementation than this though, with fewer users parameters, but mainly something is more robust. Looking only within a single pixel with a simple metric like this is not very reliable and will give artifacts in common cases. Maybe this is good enough, but then we'd really need to see some renders to prove that.

The recent denoiser infrastructure might help to get a better variance estimate, and to access pixels from neighbouring tiles by keeping them around.

Hi,

you are right. It is very simple with minimum parameters. But I think it works very well. For example: you can check scene 04_01_H from Agent327.

I think this is very good start with adaptive rendering because you can compare new features with this function.

I am preparing more tests.

So I understand that Norm here is a sort of threshold, right? Then, setting Norm, is like setting the "main level of smoothness" of the image.
Could this include a time-halt condition?

Milan Jaros (jar091) added a comment.EditedMay 22 2017, 6:24 PM

So I understand that Norm here is a sort of threshold, right? Then, setting Norm, is like setting the "main level of smoothness" of the image.
Could this include a time-halt condition?

Hi,

all parameters describe a stopping criteria for monte carlo simulation. This criteria checks the convergence of the calculation of the rendering equation. The norm value and the step value describe a noise/smoothness. The confirm value increases the propability of the correct calculation of the norm.

I work on examples now. It will be more clear.

Right, some example renders + timings would be great, so it's easy for all of us to see how well this works.

For such a simplistic approach, it works surprisingly well, I have to admit. However, it is very dependent on tile size: if neighbouring tiles have very different numbers of samples, you can end up with visible squares in your render, see the attached example:

Here some tiles found the caustics within the first 200 samples and added more and more samples to refine them. Other tiles missed the caustics in the first 200 samples and therefore stopped sampling with an (incorrect) tile without any caustics.

And equal time render without adaptive sampling illustrates the caustics that were missed:

As an improvement, this could be extended to work in progressive refine mode - then it could look at neighbouring tiles to determine whether or not to stop sampling or to continue. Another improvement could be to not look at the final pixel value but at other measures of variance, see the extra passes that the denoising feature uses. In this test scene, the adaptive sampler never recognised the tile containing the (very sharp and noise-free) specular reflection of the triangle as finished, most likely due to the high intensity of the emission.

For those who care about the academics, note that most adaptive sampling methods (and at first glance, I'd guess this one too) are biased. (see https://pdfs.semanticscholar.org/65c0/b6bfa930b887244069899505fd5166abd036.pdf) I doubt that anyone would notice in practice though.

One thing I didn't understand: is the check done at pixel level or at the whole tile?

If the user could paint areas needing more sample and continue render, it would avoid the limitations of computed versions for still images and remove the overhead.

Right, some example renders + timings would be great, so it's easy for all of us to see how well this works.

Hi,

more tests you can find here:
http://blender.it4i.cz/research/adaptive-rendering-based-on-a-norm-of-rendered-pixels/

The site is still under the reconstruction.

One thing I didn't understand: is the check done at pixel level or at the whole tile?

Hi,

one thread checks each pixel over the current tile. It finds the maximum change of the color in each pixel over the tile.

Milan

If the user could paint areas needing more sample and continue render, it would avoid the limitations of computed versions for still images and remove the overhead.

Hi,

you can check the headlight in the table:
http://blender.it4i.cz/research/adaptive-rendering-based-on-a-norm-of-rendered-pixels/bmw27/

I think the algorithm detects the noise in the headlight very well.

Milan

Frankly, all the tests look absolutely gorgeous, not just the headlights :)
What I miss is a comparison at same render time: plain vs adaptive.
The original render takes hours but deciding what is 100% quality sounds arbitrary here.

one thread checks each pixel over the current tile. It finds the maximum change of the color in each pixel over the tile.

And then that maximum change is tested against the norm value? and if lower then it stops the tile?

In order to not "pollute" here with trivial questions, would you mind if I open a dedicated thread at blenderartists.org and discuss there?

If the user could paint areas needing more sample and continue render, it would avoid the limitations of computed versions for still images and remove the overhead.

Hi,

you can check the headlight in the table:
http://blender.it4i.cz/research/adaptive-rendering-based-on-a-norm-of-rendered-pixels/bmw27/

I think the algorithm detects the noise in the headlight very well.

Milan

You did a great job, however, to be really usable and save human time (the most important as it's what cost the most money today), it must be very robust. As swerner shows here https://developer.blender.org/D2662#64166, caustics give artifacts on tile borders. As he said, reflections of strong lights, that are perfectly uniform to human eye, also keep the adaptive rendering busy it seems.
Shouting in the dark here, but maybe checking mean variance of a tile between sample x and x-1 of the tonemapped image could help here to take color management and it's final dynamic range into account. If it's already the case, sorry, I don't know the color management part of the code.

Do you have a recent version of the patch that applies on latest master? I couldn't apply it, even with arcanist.

I'm also finding it difficult to interpret the results, there's so much data :). I guess the renders with norm 1 are effectively renders with no adaptive sampling and using step samples? So then it's possible to compare renders with norm 1 to others with a similar render time.

For example BMW no adaptive vs. adaptive with similar render time. There it's doing basically what you'd expect, distributing noise more evenly.

Being able to set a norm value instead of a number of samples is by itself a great feature I think, regardless of the speedup. In an ideal world you would be able to say, give me a low/medium/final quality image, and no other sampling parameters to tweak. A long way to go before we get there, but this is a step towards that.

The tile issue is tricky, you almost have to require progressive rendering of the whole image for that, along with some fairly complex sync between multiple threads and devices. Perhaps in practice the cases where you get tile artifacts are also typically the cases where the image is unacceptably noisy anyway, and still better than adaptive sampling per pixel?

Shouting in the dark here, but maybe checking mean variance of a tile between sample x and x-1 of the tonemapped image could help here to take color management and it's final dynamic range into account. If it's already the case, sorry, I don't know the color management part of the code.

Ideally the adaptive sampling should take color management into account indeed, though in general it's unknown what kind of image transforms might be applied in compositing. A simple color_scene_linear_to_srgb() or sqrt() might already see a noticeable improvement.

Ideally the adaptive sampling should take color management into account indeed, though in general it's unknown what kind of image transforms might be applied in compositing. A simple color_scene_linear_to_srgb() or sqrt() might already see a noticeable improvement.

or maybe in a "logarithmic space" as in Log view?

Ideally the adaptive sampling should take color management into account indeed, though in general it's unknown what kind of image transforms might be applied in compositing. A simple color_scene_linear_to_srgb() or sqrt() might already see a noticeable improvement.

Correct me if I'm wrong, but the tile updates already create/update an srgb color managed buffer. The new filmic made color dynamic changes in post rather an exception, at least in our workflow. In fact, we don't do any change in dynamic range in compositing anymore. Everything is done with film, curves and gamma. So to not add any overhead, analyzing this buffer, which is already there and which is very near from end version dynamic range would be fast and relatively reliable?

Correct me if I'm wrong, but the tile updates already create/update an srgb color managed buffer. The new filmic made color dynamic changes in post rather an exception, at least in our workflow. In fact, we don't do any change in dynamic range in compositing anymore. Everything is done with film, curves and gamma. So to not add any overhead, analyzing this buffer, which is already there and which is very near from end version dynamic range would be fast and relatively reliable?

No, this buffer does not exist normally. The color space transform happens in a GLSL shader while drawing, and for background renders it happens when saving the file at the very end, or not at all when writing to EXR.

Milan Jaros (jar091) updated this revision to Diff 8993.EditedThu, Jul 6, 11:55 AM

I updated code to the latest master (4d124418b7e471b314a19c142a54b6a6bb0088f0).

The best solution for the number of samples will be to create the vector of samples (one sample for one pixel). But it needs more changes.

Milan

thanks for the update. For some reason, when used with denoising, it is partly transparent.
Here, the sky in Barcelona:

Milan Jaros (jar091) edited the summary of this revision. (Show Details)Thu, Jul 6, 3:59 PM
Milan Jaros (jar091) added a comment.EditedThu, Jul 6, 4:16 PM

thanks for the update. For some reason, when used with denoising, it is partly transparent.
Here, the sky in Barcelona:

Hi, I look at on the combination with the denoising (which will be good idea), but unfortunately it is not easy to fix it. I would have to completely rewrite the getting of the num_samples value. Milan

In my tests, noise is even, which is good and overhead of analysis seem to be low. But setting a norm of 0.005 and steps of 4 gives the same result as norm of 0.01 and steps of 16. A norm of 0.01 and a step of 4 halves the time compared to same norm but step of 16, although in theory, the max difference should be of 12 samples between those 2 settings. As I rendered with max 1000 samples and a norm of 0.01 compares in quality with 200 samples, a step of 4 could stop in the strongest discrepancy case at 200 samples, while the 16 step case would stop at 212samples, which wouldn't make a 2x longer render time. Plus the higher step value should reduce the overhead of image comparison.
norm 0.01 step 16
render time: 4minutes


norm 0.01 step 4
render time: 2min 10

norm 0.005 step 4
render time: 3min54

Ideally, same norm should give about same noise level whatever the step is. Higher step may make noise distribution less even, but not really impact overall noise level.
From a user point of view, I would replace the norm by a quality slider between 0 and 100 like for JPEG and all lossy algorithm. 100 would offer a picture with an imperceptible noise level and 0 would be a 1 sample render. 99 would allow a variance of 1%, 20 would allow a variance of 80%, etc... But @Lukas Stockner (lukasstockner97) may have some good experience here on how to best map those values for the user.