Shadow Catcher bug with transparency and Branched Path Tracing
Closed, ResolvedPublic

Description

System Information

Ubuntu Linux 16.04, CPU rendering

Blender Version
Broken: (example: 2.78.5 ba256b32ee5)

Short description of error

When a transparent or semi-transparent material is used over a Shadow Catcher plane, the plane gets rendered in front of the transparent object when using Branched Path Tracing sampling.

Exact steps for others to reproduce the error

Just add a transparent object in front of a plane with Shadow Catcher enabled and use Branched Path Tracing sampling.

For example, this is the expected render obtained with Path Tracing sampling:

This is the broken one obtained with Branched Path Tracing:

This is the material and sampling settings used with the broken plane:

Setup for the example above:

Sergey Sharybin (sergey) triaged this task as "Confirmed" priority.Jul 11 2017, 9:36 AM
Sergey Sharybin (sergey) claimed this task.

Thanks for confirming the issue.

Is the fix for this likely to be in 2.79? The branched tracing greatly improves the rendering time in cases where a significant portion of the image is empty, which turns out to be a common case for the Shadow Catcher since it's integrating into existing footage.

Thanks again for looking into this.

The fix will be committed soon. Currently working on some unit tests to help testing shadow catcher in the future.

For the record, PT result was not correct either: if you render on a transparent film and alpha-over on an actual background you'll have artifacts. So both PT and BPT will be changed in render results.

Is this supposed to be fixed in the latest builds? I'm still seeing artifacts in the nightly of 2.79 (more specifically, 2.78.5 1bd9531fda7).

I've reproduced by just putting a fully transparent object in front of a shadow catcher plane, and traces of the shadow remain on it.

I can try a new reproducer if that's useful.

The .blend file attached to this report seems to render correctly for me. If there's a different case that renders wrong, please do attach a .blend for that.

With that same example .blend file, just turn the factor all the way down to zero. The shadow (or AO, more likely) remain:

Here is a more clear example showing the issue and including the more noisy artifacts, in white background so it's easier to spot.

Rendered with 0.3 opacity it looks plausible (2.78.5 1bd9531fda7):

Going all the way down to zero shows the issue:

And this composites the phantom shadow and the extra noise on top of the prior scene:

Here is the blend file for the scene:

@Gustavo Niemeyer (niemeyer), thanks for the detailed files. I am currently at siggraph, so really limited on time for bugfixing. But definitely looking into figuring this out.

What you might want to do is to copy the details to a new report dedicated to this particular problem. This way it'll make it easier to reference the issue from code and track changes back in the future when needed.

Here is a potential fix:

1diff --git a/intern/cycles/kernel/kernel_accumulate.h b/intern/cycles/kernel/kernel_accumulate.h
2index 9ed16aceb55..acfe1a5a63b 100644
3--- a/intern/cycles/kernel/kernel_accumulate.h
4+++ b/intern/cycles/kernel/kernel_accumulate.h
5@@ -223,6 +223,7 @@ ccl_device_inline void path_radiance_init(PathRadiance *L, int use_light_pass)
6​ L->shadow_background_color = make_float3(0.0f, 0.0f, 0.0f);
7​ L->shadow_radiance_sum = make_float3(0.0f, 0.0f, 0.0f);
8​ L->shadow_throughput = 0.0f;
9+ L->shadow_transparency = 1.0f;
10​ #endif
11
12​ #ifdef __DENOISING_FEATURES__
13@@ -398,10 +399,11 @@ ccl_device_inline void path_radiance_accum_total_light(
14​ #endif
15​ }
16
17-ccl_device_inline void path_radiance_accum_background(PathRadiance *L,
18- ccl_addr_space PathState *state,
19- float3 throughput,
20- float3 value)
21+ccl_device_inline void path_radiance_accum_background(
22+ PathRadiance *L,
23+ ccl_addr_space PathState *state,
24+ float3 throughput,
25+ float3 value)
26​ {
27​ #ifdef __PASSES__
28​ if(L->use_light_pass) {
29@@ -421,9 +423,7 @@ ccl_device_inline void path_radiance_accum_background(PathRadiance *L,
30​ #ifdef __SHADOW_TRICKS__
31​ if(state->flag & PATH_RAY_STORE_SHADOW_INFO) {
32​ L->path_total += throughput * value;
33- if(state->flag & PATH_RAY_SHADOW_CATCHER_ONLY) {
34- L->path_total_shaded += throughput * value;
35- }
36+ L->path_total_shaded += throughput * value * L->shadow_transparency;
37​ }
38​ #endif
39
40@@ -671,7 +671,7 @@ ccl_device_inline float path_radiance_sum_shadow(const PathRadiance *L)
41​ if(path_total != 0.0f) {
42​ return path_total_shaded / path_total;
43​ }
44- return 1.0f;
45+ return L->shadow_transparency;
46​ }
47
48​ /* Calculate final light sum and transparency for shadow catcher object. */
49diff --git a/intern/cycles/kernel/kernel_path.h b/intern/cycles/kernel/kernel_path.h
50index c340b3bc968..f2f59034bbb 100644
51--- a/intern/cycles/kernel/kernel_path.h
52+++ b/intern/cycles/kernel/kernel_path.h
53@@ -319,7 +319,8 @@ ccl_device void kernel_path_indirect(KernelGlobals *kg,
54
55​ #ifdef __SHADOW_TRICKS__
56​ if(!(sd->object_flag & SD_OBJECT_SHADOW_CATCHER)) {
57- state->flag &= ~PATH_RAY_SHADOW_CATCHER_ONLY;
58+ L->shadow_transparency *=
59+ average(shader_bsdf_transparency(kg, sd));
60​ }
61​ #endif /* __SHADOW_TRICKS__ */
62
63@@ -644,7 +645,6 @@ ccl_device_inline float kernel_path_integrate(KernelGlobals *kg,
64​ if((sd.object_flag & SD_OBJECT_SHADOW_CATCHER)) {
65​ if(state.flag & PATH_RAY_CAMERA) {
66​ state.flag |= (PATH_RAY_SHADOW_CATCHER |
67- PATH_RAY_SHADOW_CATCHER_ONLY |
68​ PATH_RAY_STORE_SHADOW_INFO);
69​ state.catcher_object = sd.object;
70​ if(!kernel_data.background.transparent) {
71@@ -656,7 +656,8 @@ ccl_device_inline float kernel_path_integrate(KernelGlobals *kg,
72​ }
73​ }
74​ else {
75- state.flag &= ~PATH_RAY_SHADOW_CATCHER_ONLY;
76+ L->shadow_transparency *=
77+ average(shader_bsdf_transparency(kg, &sd));
78​ }
79​ #endif /* __SHADOW_TRICKS__ */
80
81diff --git a/intern/cycles/kernel/kernel_path_branched.h b/intern/cycles/kernel/kernel_path_branched.h
82index 77d4f1df447..48124b36095 100644
83--- a/intern/cycles/kernel/kernel_path_branched.h
84+++ b/intern/cycles/kernel/kernel_path_branched.h
85@@ -500,7 +500,6 @@ ccl_device float kernel_branched_path_integrate(KernelGlobals *kg,
86​ #ifdef __SHADOW_TRICKS__
87​ if((sd.object_flag & SD_OBJECT_SHADOW_CATCHER)) {
88​ state.flag |= (PATH_RAY_SHADOW_CATCHER |
89- PATH_RAY_SHADOW_CATCHER_ONLY |
90​ PATH_RAY_STORE_SHADOW_INFO);
91​ state.catcher_object = sd.object;
92​ if(!kernel_data.background.transparent) {
93@@ -511,7 +510,8 @@ ccl_device float kernel_branched_path_integrate(KernelGlobals *kg,
94​ L->shadow_throughput = average(throughput);
95​ }
96​ else {
97- state.flag &= ~PATH_RAY_SHADOW_CATCHER_ONLY;
98+ L->shadow_transparency *=
99+ average(shader_bsdf_transparency(kg, &sd));
100​ }
101​ #endif /* __SHADOW_TRICKS__ */
102
103diff --git a/intern/cycles/kernel/kernel_types.h b/intern/cycles/kernel/kernel_types.h
104index e6a62c42a38..e12557fc5b2 100644
105--- a/intern/cycles/kernel/kernel_types.h
106+++ b/intern/cycles/kernel/kernel_types.h
107@@ -346,8 +346,7 @@ enum PathRayFlag {
108​ PATH_RAY_DIFFUSE_ANCESTOR = (1 << 13),
109​ PATH_RAY_SINGLE_PASS_DONE = (1 << 14),
110​ PATH_RAY_SHADOW_CATCHER = (1 << 15),
111- PATH_RAY_SHADOW_CATCHER_ONLY = (1 << 16),
112- PATH_RAY_STORE_SHADOW_INFO = (1 << 17),
113+ PATH_RAY_STORE_SHADOW_INFO = (1 << 16),
114​ };
115
116​ /* Closure Label */
117@@ -524,6 +523,7 @@ typedef ccl_addr_space struct PathRadiance {
118​ */
119​ float3 shadow_radiance_sum;
120​ float shadow_throughput;
121+ float shadow_transparency;
122​ #endif
123
124​ #ifdef __DENOISING_FEATURES__
125diff --git a/intern/cycles/kernel/split/kernel_holdout_emission_blurring_pathtermination_ao.h b/intern/cycles/kernel/split/kernel_holdout_emission_blurring_pathtermination_ao.h
126index fec671be016..ad7e5d1cbee 100644
127--- a/intern/cycles/kernel/split/kernel_holdout_emission_blurring_pathtermination_ao.h
128+++ b/intern/cycles/kernel/split/kernel_holdout_emission_blurring_pathtermination_ao.h
129@@ -127,7 +127,6 @@ ccl_device void kernel_holdout_emission_blurring_pathtermination_ao(
130​ if(state->flag & PATH_RAY_CAMERA) {
131​ PathRadiance *L = &kernel_split_state.path_radiance[ray_index];
132​ state->flag |= (PATH_RAY_SHADOW_CATCHER |
133- PATH_RAY_SHADOW_CATCHER_ONLY |
134​ PATH_RAY_STORE_SHADOW_INFO);
135​ state->catcher_object = sd->object;
136​ if(!kernel_data.background.transparent) {
137@@ -143,7 +142,8 @@ ccl_device void kernel_holdout_emission_blurring_pathtermination_ao(
138​ }
139​ }
140​ else {
141- state->flag &= ~PATH_RAY_SHADOW_CATCHER_ONLY;
142+ PathRadiance *L = &kernel_split_state.path_radiance[ray_index];
143+ L->shadow_transparency *= average(shader_bsdf_transparency(kg, sd));
144​ }
145​ #endif /* __SHADOW_TRICKS__ */
146

There are still some artifacts with less trivial topology than a box and AO-only lit scenes. Will need to have a closer look to see if that's something easily to solve.

P.S. If you don't mind, it's still handy to have separate report for this particular issue.

@Sergey Sharybin (sergey) Thanks again for your attention on this.

I've create a more appropriate task as requested: T52229