Page MenuHome

Cycles: Implement animation denoising support
ClosedPublic

Authored by Lukas Stockner (lukasstockner97) on Nov 2 2018, 1:57 PM.

Details

Summary

This patch extends the denoiser to support using multiple input frames in order to denoise one frame. This can significantly reduce noise and flickering in animations.

Since future frames are required before a frame can be denoised, animation denoising can not be performed during rendering.
Instead, the sequence is rendered to Multilayer EXR files and then denoised using a function of the Cycles Python API.

In order to generate the neccessary information, the option to output denoising passes is now indepent from whether the frame will be denoised at render time.
When the output passes are enabled, Cycles will perform prefiltering of the feature passes during rendering (since that step is performed individually on each frame).

In the future, an integration into the user interface or, even better, the animation rendering workflow is planned.

This feature was originally developed for Tangent Animation, thanks for the support!

Diff Detail

Branch
animation-denoising
Build Status
Buildable 2426
Build 2426: arc lint + arc unit

Event Timeline

Brecht Van Lommel (brecht) requested changes to this revision.Nov 4 2018, 7:17 PM
Brecht Van Lommel (brecht) added inline comments.
intern/cycles/blender/addon/engine.py
271–277

Why did these passes change? Does this commit include a change to the still denoising algorithm, and if so can you briefly explain what and why?

intern/cycles/blender/blender_python.cpp
621–623

I'm not sure why we don't just get most of this information from the current scene in Blender. Is there a specific need for passing these as parameters to the function rather than reading them from the scene?

Seems like when we add a Denoise button in the UI there would be less code duplication.

intern/cycles/render/denoising.cpp
2

Add the license header.

19–25

Can we have this #ifdef only inside the system_console_shape?

30

Code style: always use {}, in various functions in this file.

84–89

Code style: don't use this many empty lines.

493

We should use array<> instead of new/delete.

This revision now requires changes to proceed.Nov 4 2018, 7:17 PM
Lukas Stockner (lukasstockner97) marked 5 inline comments as done.Nov 7 2018, 4:50 AM
Lukas Stockner (lukasstockner97) added inline comments.
intern/cycles/blender/addon/engine.py
271–277

The denoising process essentially consists of three steps - prefilter the feature passes, compute the reduced feature space transform and run the fitting code. The current code in master does all three in one go, so the input data are the passes coming out of the rendering code and the output is the final image.

For animation denoising, that would also be possible, but of course all input frames would have to be prefiltered. However, that step is done for each frame individually, so each frame would get prefiltered multiple times. To avoid that, I've implemented it so that Cycles outputs prefiltered passes if you enable the option in the pass menu - that way, the denoising tool can just read the prefiltered version and skip that step.

In practice, this means that tiles still get scheduled for denoising even if you only select the pass output, but the DenoisingTask will only perform the denoising step. If you enable both direct denoising and the pass output, it will save the intermediate and the final output

intern/cycles/blender/blender_python.cpp
621–623

Good point, the Python interface was pretty much an afterthought and the original version had a standalone binary so I didn't think of that.

I think it would be useful to keep the optional arguments for convenient use from the command line, but the default values should indeed be loaded from the current scene/renderlayer.

However, I'm not really sure how to get the current scene from here since there's no straightforward way of accessing bpy.context, would be great if you could help here.

Rebased to current master.

Also, I increased the radius of the box blur on the intensity normalization pass. Its intention is to compensate large-scale illumination differences, but it caused problems at edges - due to the normalization, the NLM difference term ended up quite small and so the pixels form the other side of the edge made it into the fitting process.

This is actually only a symptom of a larger issue with the underlying algorithm: This kind of situation should be resolved by the feature passes, but since we use a linear fit while the intensity along the time dimension looks more like a step function, the fit can cause the color of neighboring frames to "leak through". Usually the NLM weighting gets rid of those, but when that also fails (as explained above), we get artifacts on edges. Even with the fix (or without any intensity normalization), those artifacts pop up along low-contrast edges.

I have some ideas for fixing this (for example, also applying the normalization term to the pixel value that is used for fitting and/or using some simple form of optical flow to align the search patch across neighboring frames), but I still need to look into those.

Anyways, with the fix this is only visible with highly noisy inputs and right along low-contrast edges, so the impact is small enough to be mostly a theoretical annoyance, not a real issue in practical use.

Rebase to current 2.8 master.

We need this for Spring, so I'm reviewing and merging this now.

I'll commit this in pieces as I review them.

intern/cycles/blender/blender_python.cpp
621–623

I'll implement this.

I've been refactoring the current implementation to work better for UI integration
and ensure the input and output EXR match files match more closely.

Some control and minor optimizations got lost, but I think this is a more robust
state to build on.

  • Add cycles.denoise_animation operator which denoises an animation sequence.
  • Filepath, frame range and settings come from the current scene and view layer.
  • Files can be denoised in place, or to a separate file path.
  • Still missing: good place in the UI, support for cancel and progress reports.
  • Improve error handling so we catch more errors and always abort in case of failures instead of possible writing invalid image data.
  • Save to temporary file and only at the end put image in output location.
  • Always use exact same channel layout for input and output file. Simplifies code and is a bit more predictable for users.
  • _cycles.denoise() now takes a list of input and output file paths, so we can be sure they match Blender filepaths or other custom locations, rather than relying on OIIO pattern matching.

I'd like to commit this soon, maybe at first without the UI so we can test
this for Spring immediately. And then add into the UI soon after once cancel
and progress reports are there.

This revision was not accepted when it landed; it landed in state Needs Review.Feb 11 2019, 1:40 PM
This revision was automatically updated to reflect the committed changes.