Details

Reviewers

Brecht Van Lommel (brecht)
Sergey Sharybin (sergey)

Commits

rBSe379a9ba918d: Cycles: add animation denoising Python operator.
rC3c4ce44ef15f: Add animation denoising python operator
rBe379a9ba918d: Cycles: add animation denoising Python operator.
rBb26cc7ce2e58: Squashed commit of D3889

Summary

This patch extends the denoiser to support using multiple input frames in order to denoise one frame. This can significantly reduce noise and flickering in animations.

Since future frames are required before a frame can be denoised, animation denoising can not be performed during rendering.
Instead, the sequence is rendered to Multilayer EXR files and then denoised using a function of the Cycles Python API.

In order to generate the neccessary information, the option to output denoising passes is now indepent from whether the frame will be denoised at render time.
When the output passes are enabled, Cycles will perform prefiltering of the feature passes during rendering (since that step is performed individually on each frame).

In the future, an integration into the user interface or, even better, the animation rendering workflow is planned.

This feature was originally developed for Tangent Animation, thanks for the support!

Diff Detail

Branch

animation-denoising

Build Status

Buildable 2426
Build 2426: arc lint + arc unit

Event Timeline

Lukas Stockner (lukasstockner97) created this revision.Nov 2 2018, 1:57 PM

Julian Perez (julperado) awarded a token.Nov 2 2018, 3:49 PM

Brecht Van Lommel (brecht) requested changes to this revision.Nov 4 2018, 7:17 PM

Brecht Van Lommel (brecht) added inline comments.

intern/cycles/blender/addon/engine.py
271–277	Why did these passes change? Does this commit include a change to the still denoising algorithm, and if so can you briefly explain what and why?
intern/cycles/blender/blender_python.cpp
621–623	I'm not sure why we don't just get most of this information from the current scene in Blender. Is there a specific need for passing these as parameters to the function rather than reading them from the scene? Seems like when we add a Denoise button in the UI there would be less code duplication.
intern/cycles/render/denoising.cpp
2	Add the license header.
19–25	Can we have this #ifdef only inside the `system_console_shape`?
30	Code style: always use `{}`, in various functions in this file.
84–89	Code style: don't use this many empty lines.
493	We should use `array<>` instead of new/delete.

This revision now requires changes to proceed.Nov 4 2018, 7:17 PM

Lukas Stockner (lukasstockner97) marked 5 inline comments as done.Nov 7 2018, 4:50 AM

Lukas Stockner (lukasstockner97) added inline comments.

intern/cycles/blender/addon/engine.py
271–277	The denoising process essentially consists of three steps - prefilter the feature passes, compute the reduced feature space transform and run the fitting code. The current code in master does all three in one go, so the input data are the passes coming out of the rendering code and the output is the final image. For animation denoising, that would also be possible, but of course all input frames would have to be prefiltered. However, that step is done for each frame individually, so each frame would get prefiltered multiple times. To avoid that, I've implemented it so that Cycles outputs prefiltered passes if you enable the option in the pass menu - that way, the denoising tool can just read the prefiltered version and skip that step. In practice, this means that tiles still get scheduled for denoising even if you only select the pass output, but the DenoisingTask will only perform the denoising step. If you enable both direct denoising and the pass output, it will save the intermediate and the final output
intern/cycles/blender/blender_python.cpp
621–623	Good point, the Python interface was pretty much an afterthought and the original version had a standalone binary so I didn't think of that. I think it would be useful to keep the optional arguments for convenient use from the command line, but the default values should indeed be loaded from the current scene/renderlayer. However, I'm not really sure how to get the current scene from here since there's no straightforward way of accessing `bpy.context`, would be great if you could help here.

Addressed feedback.

Rebased to current master.

Also, I increased the radius of the box blur on the intensity normalization pass. Its intention is to compensate large-scale illumination differences, but it caused problems at edges - due to the normalization, the NLM difference term ended up quite small and so the pixels form the other side of the edge made it into the fitting process.

This is actually only a symptom of a larger issue with the underlying algorithm: This kind of situation should be resolved by the feature passes, but since we use a linear fit while the intensity along the time dimension looks more like a step function, the fit can cause the color of neighboring frames to "leak through". Usually the NLM weighting gets rid of those, but when that also fails (as explained above), we get artifacts on edges. Even with the fix (or without any intensity normalization), those artifacts pop up along low-contrast edges.

I have some ideas for fixing this (for example, also applying the normalization term to the pixel value that is used for fitting and/or using some simple form of optical flow to align the search patch across neighboring frames), but I still need to look into those.

Anyways, with the fix this is only visible with highly noisy inputs and right along low-contrast edges, so the impact is small enough to be mostly a theoretical annoyance, not a real issue in practical use.

Stefan Werner (swerner) added a subscriber: Stefan Werner (swerner).Nov 22 2018, 9:39 AM

Rebase to current 2.8 master.

Harbormaster completed remote builds in B2777: Diff 13348.Jan 25 2019, 3:22 PM

Lukas Stockner (lukasstockner97) mentioned this in rBb26cc7ce2e58: Squashed commit of D3889.Jan 25 2019, 3:32 PM

Lukas Stockner (lukasstockner97) mentioned this in rB7bda09e55d63: Revert "Squashed commit of D3889".

Adam Friesen (ace_dragon) awarded a token.Jan 25 2019, 4:19 PM

We need this for Spring, so I'm reviewing and merging this now.

I'll commit this in pieces as I review them.

intern/cycles/blender/blender_python.cpp
621–623	I'll implement this.

Brecht Van Lommel (brecht) mentioned this in rB405cacd4cd95: Cycles: prefilter feature passes separate from denoising..Feb 6 2019, 3:20 PM

Brecht Van Lommel (brecht) mentioned this in rBc183ac73dcfd: Cycles: tweak outlier detection, preparing for animation denoising..

Brecht Van Lommel (brecht) mentioned this in rBfccf506ed7fd: Cycles: animation denoising support in the kernel..

I've been refactoring the current implementation to work better for UI integration
and ensure the input and output EXR match files match more closely.

Some control and minor optimizations got lost, but I think this is a more robust
state to build on.

Add cycles.denoise_animation operator which denoises an animation sequence.
Filepath, frame range and settings come from the current scene and view layer.
Files can be denoised in place, or to a separate file path.
Still missing: good place in the UI, support for cancel and progress reports.

Improve error handling so we catch more errors and always abort in case of failures instead of possible writing invalid image data.
Save to temporary file and only at the end put image in output location.
Always use exact same channel layout for input and output file. Simplifies code and is a bit more predictable for users.
_cycles.denoise() now takes a list of input and output file paths, so we can be sure they match Blender filepaths or other custom locations, rather than relying on OIIO pattern matching.

I'd like to commit this soon, maybe at first without the UI so we can test
this for Spring immediately. And then add into the UI soon after once cancel
and progress reports are there.

Harbormaster completed remote builds in B2862: Diff 13567.Feb 8 2019, 9:42 PM

This revision was not accepted when it landed; it landed in state Needs Review.Feb 11 2019, 1:40 PM

Closed by commit rBe379a9ba918d: Cycles: add animation denoising Python operator. (authored by Lukas Stockner (lukasstockner97), committed by Brecht Van Lommel (brecht)). · Explain Why

This revision was automatically updated to reflect the committed changes.

Brecht Van Lommel (brecht) mentioned this in rCb945b6f10331: Prefilter feature passes separate from denoising.Feb 11 2019, 7:29 PM

Brecht Van Lommel (brecht) mentioned this in rCd1753aa6c66a: Tweak outlier detection, preparing for animation denoising..

Brecht Van Lommel (brecht) mentioned this in rCab59dde7227a: Animation denoising support in the kernel.

Don (DonCoyote) added a subscriber: Don (DonCoyote).Jul 9 2019, 8:01 PM

		Path
M		intern/cycles/blender/addon/engine.py (9 lines)
M		intern/cycles/blender/blender_python.cpp (95 lines)
M		intern/cycles/blender/blender_session.cpp (29 lines)
M		intern/cycles/blender/blender_sync.cpp (35 lines)
M		intern/cycles/device/device_cpu.cpp (90 lines)
M		intern/cycles/device/device_cuda.cpp (182 lines)
M		intern/cycles/device/device_denoising.h (27 lines)
M		intern/cycles/device/device_denoising.cpp (119 lines)
M		intern/cycles/device/device_task.h (8 lines)
M		intern/cycles/device/opencl/opencl.h (16 lines)
M		intern/cycles/device/opencl/opencl_base.cpp (100 lines)
M		intern/cycles/kernel/filter/filter_defines.h (7 lines)
M		intern/cycles/kernel/filter/filter_features.h (77 lines)
M		intern/cycles/kernel/filter/filter_features_sse.h (53 lines)
M		intern/cycles/kernel/filter/filter_nlm_cpu.h (32 lines)
M		intern/cycles/kernel/filter/filter_nlm_gpu.h (45 lines)
M		intern/cycles/kernel/filter/filter_prefilter.h (82 lines)
M		intern/cycles/kernel/filter/filter_reconstruction.h (24 lines)
M		intern/cycles/kernel/filter/filter_transform.h (59 lines)
M		intern/cycles/kernel/filter/filter_transform_gpu.h (54 lines)
M		intern/cycles/kernel/filter/filter_transform_sse.h (60 lines)
M		intern/cycles/kernel/kernel_types.h (9 lines)
M		intern/cycles/kernel/kernels/cpu/filter_cpu.h (23 lines)
M		intern/cycles/kernel/kernels/cpu/filter_cpu_impl.h (71 lines)
M		intern/cycles/kernel/kernels/cuda/filter.cu (61 lines)
M		intern/cycles/kernel/kernels/opencl/filter.cl (54 lines)
M		intern/cycles/render/CMakeLists.txt (2 lines)
M		intern/cycles/render/buffers.h (5 lines)
M		intern/cycles/render/buffers.cpp (122 lines)
A	M	intern/cycles/render/denoising.h (239 lines)
A	M	intern/cycles/render/denoising.cpp (857 lines)
M		intern/cycles/render/film.h (1 line)
M		intern/cycles/render/film.cpp (4 lines)
M		intern/cycles/render/session.h (10 lines)
M		intern/cycles/render/session.cpp (11 lines)
M		intern/cycles/util/util_system.h (3 lines)
M		intern/cycles/util/util_system.cpp (17 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	12362	1b97456		Nov 2 2018, 1:57 PM	★	★
Diff 2	12430	1b97456	Addressed feedback.	Nov 7 2018, 5:05 AM	★	★
Diff 3	12603	968bf0d	Rebased to current master.	Nov 22 2018, 3:43 AM	★	★
Diff 4	13348	35f0a59	Rebase to current 2.8 master.	Jan 25 2019, 3:22 PM	★	★
Diff 5	13567	e21ae0b	I've been refactoring the current implementation to work better for UI…	Feb 8 2019, 9:42 PM	★	★
Diff 6	13601	382fe85	rBe379a9ba918dbf27e81fd289e98314197109b54c	Feb 11 2019, 1:39 PM	★	★

Cycles: Implement animation denoising supportClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 12430

intern/cycles/blender/addon/engine.py

intern/cycles/blender/blender_python.cpp

intern/cycles/blender/blender_session.cpp

intern/cycles/blender/blender_sync.cpp

intern/cycles/device/device_cpu.cpp

intern/cycles/device/device_cuda.cpp

intern/cycles/device/device_denoising.h

intern/cycles/device/device_denoising.cpp

intern/cycles/device/device_task.h

intern/cycles/device/opencl/opencl.h

intern/cycles/device/opencl/opencl_base.cpp

intern/cycles/kernel/filter/filter_defines.h

intern/cycles/kernel/filter/filter_features.h

intern/cycles/kernel/filter/filter_features_sse.h

intern/cycles/kernel/filter/filter_nlm_cpu.h

intern/cycles/kernel/filter/filter_nlm_gpu.h

intern/cycles/kernel/filter/filter_prefilter.h

intern/cycles/kernel/filter/filter_reconstruction.h

intern/cycles/kernel/filter/filter_transform.h

intern/cycles/kernel/filter/filter_transform_gpu.h

intern/cycles/kernel/filter/filter_transform_sse.h

intern/cycles/kernel/kernel_types.h

intern/cycles/kernel/kernels/cpu/filter_cpu.h

intern/cycles/kernel/kernels/cpu/filter_cpu_impl.h

intern/cycles/kernel/kernels/cuda/filter.cu

intern/cycles/kernel/kernels/opencl/filter.cl

intern/cycles/render/CMakeLists.txt

intern/cycles/render/buffers.h

intern/cycles/render/buffers.cpp

intern/cycles/render/denoising.h

intern/cycles/render/denoising.cpp

intern/cycles/render/film.h

intern/cycles/render/film.cpp

intern/cycles/render/session.h

intern/cycles/render/session.cpp

intern/cycles/util/util_system.h

intern/cycles/util/util_system.cpp

Cycles: Implement animation denoising support
ClosedPublic
Actions

Revision Contents
Changeset List