Design: EEVEE Compositing #72007

New Issue

Jeroen Bakker · 2019-11-28T11:30:54+01:00

Jeroen Bakker commented

2019-11-28 11:30:54 +01:00

EEVEE currently has render passes (Combined, Z, Mist, Depth, Normal, Subsurface Light, Subsurface Color, Ambient Occlusion). These render passes were added as they could be extracted easily from the buffers and shaders EEVEE already used.

The current set of passes makes it very hard to do proper compositing as they're not a complete set. You're not able to construct the Combined render pass by mixing the other passes. The goal of this project is to create a good foundation for supporting render passes in EEVEE and have a useful set to do common compositing operations.

Target situation

In the target situation we must at least support Diffuse Color, Diffuse Light, Glossy Color, Glossy Light, Emission, World Environment, Volume and Bloom.

AOVs, Denoising Data and Cryptomatte passes have specific implementation and workflow needs and therefore will need a specific design. In the future we will add other render passes based on needs and technical consequences/feasibility.

Feasibility

D6331: EEVEE: Render Passes demonstrates a proof of concept where we were able to extract useful data during rendering and construct a similar image in the compositor. It is a proof of concept to test the basic implementation and find the limitations and difficulties when using EEVEE inside the compositor. The proof of concept has some flaws and is limited.
0001-2028.mp4

The solution adds booleans to the material fragment shader that tells the shader what part of the data needs to be extracted.
Per render pass a texture is created. This texture will store the accumulated result of all samples for a specific render pass.
When the material shader is created. For all the selected render passes a shading group is created. When an object is added to the normal material shader it is also added to the shading groups for the passes.
Bloom is an additive effect. This patch stores the addition to a accumulate texture.

Alternatives

This section will explain the alternative options for implementation.

Multi target rendering

This option would have added multi target rendering to the GPU material by in stead of the current energy and transmittance we would have stored all the different material passes. and at the end of the shader we would store the data into the different passes.

PRO: single draw call could fill several buffers.
CON: compilation time increases a lot due to higher registry/local data requirements and unrolling.
CON: viewport rendering will slow down.

Multi pass rendering

This option (current PoC implementation) will create a DRWPass per render pass and have a generic GPU material that can render any pass.

PRO: No noticeable performance impact when rendering viewport.
CON: per renderpass a draw call is needed. Performance noticable during image rendering

Decisions, Consequences and Limitations

To construct these render passes we need to allocate memory on the GPU during the whole render time. This is a float RGB texture when rendering with many samples or a half RGB texture when rendering with fewer samples. The exact needed resources depends on the the implementation of the OpenGL Driver. It could be that scenes would not be able to render on low-end cards due to memory limitations and how advanced the resource management inside the driver is. When enabling 20 passes for a HD render with many samples 500MB additional GPU Memory is allocated to store the results. In the future we could optimize this by re-rendering the full frame with different passes enabled.
Ambient Occlusion is mixed in the light passes. Inside these passes different and not constant mix factors are used. It will not be possible to composite the final result from the individual passes when we only have a separate AO pass.
Screen space Depth of Field will not be available as a pass. The concern is that the pass isn't useful. At this time you could use the defocus node in the compositor.
The accumulate buffers have more precision and extract the data in the right moment. This can lead to small (but visible) differences between the Combined pass and the composited individual passes.

Revisited Approach
Note that during implementation our ideas have shifted a bit. The first implementation will use multi pass rendering. But eventually we will move to multi target rendering. The reason is that extracting the glossy/diffuse light passes needs upto 3 different passes. During the actual implementation we found a way how to overcome the complexity that we expected for the multi target rendering. The implementation of the multi target rendering will be done in a separate ticket, as it isn't clear when we will realizing that project.

Next Steps

- Get feedback on this design from users and developers.
- Get approval of this design from at least GPU, Compositor and Rendering module.
- Plan when we want start implementing.
- Finalize multi pass implementation D6331: EEVEE: Render Passes

EEVEE currently has render passes (`Combined`, `Z`, `Mist`, `Depth`, `Normal`, `Subsurface Light`, `Subsurface Color`, `Ambient Occlusion`). These render passes were added as they could be extracted easily from the buffers and shaders EEVEE already used. The current set of passes makes it very hard to do proper compositing as they're not a complete set. You're not able to construct the Combined render pass by mixing the other passes. The goal of this project is to create a good foundation for supporting render passes in EEVEE and have a useful set to do common compositing operations. **Target situation** In the target situation we must at least support `Diffuse Color`, `Diffuse Light`, `Glossy Color`, `Glossy Light`, `Emission`, World `Environment`, `Volume` and `Bloom`. AOVs, Denoising Data and Cryptomatte passes have specific implementation and workflow needs and therefore will need a specific design. In the future we will add other render passes based on needs and technical consequences/feasibility. **Feasibility** [D6331: EEVEE: Render Passes](https://archive.blender.org/developer/D6331) demonstrates a proof of concept where we were able to extract useful data during rendering and construct a similar image in the compositor. It is a proof of concept to test the basic implementation and find the limitations and difficulties when using EEVEE inside the compositor. The proof of concept has some flaws and is limited. [0001-2028.mp4](https://archive.blender.org/developer/F8216242/0001-2028.mp4) * The solution adds booleans to the material fragment shader that tells the shader what part of the data needs to be extracted. * Per render pass a texture is created. This texture will store the accumulated result of all samples for a specific render pass. * When the material shader is created. For all the selected render passes a shading group is created. When an object is added to the normal material shader it is also added to the shading groups for the passes. * Bloom is an additive effect. This patch stores the addition to a accumulate texture. **Alternatives** This section will explain the alternative options for implementation. __Multi target rendering__ This option would have added multi target rendering to the GPU material by in stead of the current energy and transmittance we would have stored all the different material passes. and at the end of the shader we would store the data into the different passes. * PRO: single draw call could fill several buffers. * CON: compilation time increases a lot due to higher registry/local data requirements and unrolling. * CON: viewport rendering will slow down. __Multi pass rendering__ This option (current PoC implementation) will create a DRWPass per render pass and have a generic GPU material that can render any pass. * PRO: No noticeable performance impact when rendering viewport. * CON: per renderpass a draw call is needed. Performance noticable during image rendering **Decisions, Consequences and Limitations** * To construct these render passes we need to allocate memory on the GPU during the whole render time. This is a float RGB texture when rendering with many samples or a half RGB texture when rendering with fewer samples. The exact needed resources depends on the the implementation of the OpenGL Driver. It could be that scenes would not be able to render on low-end cards due to memory limitations and how advanced the resource management inside the driver is. When enabling 20 passes for a HD render with many samples 500MB additional GPU Memory is allocated to store the results. In the future we could optimize this by re-rendering the full frame with different passes enabled. * `Ambient Occlusion` is mixed in the light passes. Inside these passes different and not constant mix factors are used. It will not be possible to composite the final result from the individual passes when we only have a separate AO pass. * Screen space `Depth of Field` will not be available as a pass. The concern is that the pass isn't useful. At this time you could use the defocus node in the compositor. * The accumulate buffers have more precision and extract the data in the right moment. This can lead to small (but visible) differences between the Combined pass and the composited individual passes. **Revisited Approach** Note that during implementation our ideas have shifted a bit. The first implementation will use multi pass rendering. But eventually we will move to multi target rendering. The reason is that extracting the glossy/diffuse light passes needs upto 3 different passes. During the actual implementation we found a way how to overcome the complexity that we expected for the multi target rendering. The implementation of the multi target rendering will be done in a separate ticket, as it isn't clear when we will realizing that project. **Next Steps** * - [x] Get feedback on this design from users and developers. * - [x] Get approval of this design from at least GPU, Compositor and Rendering module. * - [x] Plan when we want start implementing. * - [x] Finalize multi pass implementation [D6331: EEVEE: Render Passes](https://archive.blender.org/developer/D6331)

Jeroen Bakker self-assigned this 2019-11-28 11:30:54 +01:00

Jeroen Bakker commented

2019-11-28 11:30:54 +01:00

Added subscriber: @Jeroen-Bakker

Evan Wilson commented

2019-11-28 15:56:35 +01:00

Added subscriber: @EAW

Jeroen Bakker changed title from ~~EEVEE: Material based Renderpasses~~ to EEVEE: Compositing

2019-12-13 12:22:12 +01:00

Jeroen Bakker changed title from ~~EEVEE: Compositing~~ to Design: EEVEE Compositing

2019-12-13 12:30:15 +01:00

Pipeliner commented

2019-12-13 17:16:05 +01:00

Added subscriber: @Pipeliner

Chris Clyne (Lateasusual) commented

2019-12-14 02:55:26 +01:00

Added subscriber: @lateasusual

Julien Kaspar commented

2019-12-14 13:48:34 +01:00

Added subscriber: @JulienKaspar

Aditia A. Pratama commented

2019-12-16 10:36:54 +01:00

Added subscriber: @AditiaA.Pratama

Duarte Farrajota Ramos commented

2019-12-16 14:19:09 +01:00

Added subscriber: @DuarteRamos

RedMser commented

2019-12-17 17:54:29 +01:00

Added subscriber: @RedMser

Stephen Bailey commented

2019-12-17 20:46:58 +01:00

Added subscriber: @Crumpley

Stephen Bailey commented

2019-12-17 20:46:58 +01:00

Could this task possibly include cryptomatte support from EEVEE renders?

Adriano Oliveira commented

2019-12-18 05:47:09 +01:00

Added subscriber: @AdrianoOliveira

Adriano Oliveira commented

2019-12-18 05:47:09 +01:00

Excelent news! I would also add SPEED PASS, so we could get better motion blur in post.
Of course cryptomatte support for Eevee is a must.

Excelent news! I would also add SPEED PASS, so we could get better motion blur in post. Of course cryptomatte support for Eevee is a must.

Jeroen Bakker commented

2019-12-18 08:29:59 +01:00

@Crumpley, please read the task description.

 AOVs, Denoising Data and Cryptomatte passes have specific implementation and workflow needs and therefore will need a specific design. In the future we will add other render passes based on needs and technical consequences/feasibility.

So no, it is not in scope for this task as it needs a specific approach. It is defined as a future target.

@AdrianoOliveira in technical perspective EEVEE already has a speed pass, this is only limited to the camera movement and not useful to add as a pass right now. The best way forward is to first implement the Object Motion Blur so it will also be available in the speed pass. I hear you, just a different project needs to be solved first.

@Crumpley, please read the task description. ``` AOVs, Denoising Data and Cryptomatte passes have specific implementation and workflow needs and therefore will need a specific design. In the future we will add other render passes based on needs and technical consequences/feasibility. ``` So no, it is not in scope for this task as it needs a specific approach. It is defined as a future target. @AdrianoOliveira in technical perspective EEVEE already has a speed pass, this is only limited to the camera movement and not useful to add as a pass right now. The best way forward is to first implement the Object Motion Blur so it will also be available in the speed pass. I hear you, just a different project needs to be solved first.

Stephen Bailey commented

2019-12-18 18:05:12 +01:00

Apologies @Jeroen-Bakker I did not notice that (and, I thought I did a page search for the word Cryptomatte that came up blank). My mistake. Thanks for the response.