Grease pencil : masked layers bring their mask dependencies in in their own view layer #88202

New Issue

Samuel Bernou · 2021-05-11T16:55:10+02:00

Samuel Bernou commented

2021-05-11 16:55:10 +02:00

System Information
Operating system: Linux-5.4.0-58-generic-x86_64-with-debian-bullseye-sid 64 Bits
Graphics card: GeForce RTX 2060 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 460.32.03

Blender Version
Broken: version: 2.92.0, branch: master, commit date: 2021-02-24 16:25, hash: 02948a2cab
Worked: (newest version of Blender that worked as expected)

Short description of error
The layers that have masks active bring their "mask dependencies" with them in the render view layer, where we want only whats in the layer.

Exact steps for others to reproduce the error

In provided example file, hit render image then check the compositing nodes.
The shadow layer is on it's own on the shadow view_layer, so we expect to get only this layer content without any other appearing (and preferably unmasked).
But the color layer also appear on it.

In some cases, I suppose user you would want it to be masked (meaning, mask "applyed"). BUT, in a production logic, it's way better to have just the plain, unmasked, layer content (just hiding other layer at view_layer render time) to be able to tweak things in internal/external compositor afterwards.
In any case, user never want to see strokes of layers that aren't on the view layer, as it does now.

Blend:
view_layer_masking_issue.blend

Issue

expected result

**System Information** Operating system: Linux-5.4.0-58-generic-x86_64-with-debian-bullseye-sid 64 Bits Graphics card: GeForce RTX 2060 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 460.32.03 **Blender Version** Broken: version: 2.92.0, branch: master, commit date: 2021-02-24 16:25, hash: `02948a2cab` Worked: (newest version of Blender that worked as expected) **Short description of error** The layers that have masks active bring their "mask dependencies" with them in the render view layer, where we want only whats in the layer. **Exact steps for others to reproduce the error** In provided example file, hit render image then check the compositing nodes. The shadow layer is on it's own on the `shadow` view_layer, so we expect to get only this layer content without any other appearing (and preferably unmasked). But the `color` layer also appear on it. In some cases, I suppose user you would want it to be masked (meaning, mask "applyed"). BUT, in a production logic, it's way better to have just the plain, unmasked, layer content (just hiding other layer at view_layer render time) to be able to tweak things in internal/external compositor afterwards. In any case, user never want to see strokes of layers that aren't on the view layer, as it does now. Blend: [view_layer_masking_issue.blend](https://archive.blender.org/developer/F10079119/view_layer_masking_issue.blend) Issue ![Viewlayer_mask_issue.png](https://archive.blender.org/developer/F10079124/Viewlayer_mask_issue.png) expected result ![expected_viewlayer_with_masks.png](https://archive.blender.org/developer/F10079127/expected_viewlayer_with_masks.png)

Samuel Bernou commented

2021-05-11 16:55:11 +02:00

Added subscriber: @Pullup

Falk David commented

2021-05-11 17:03:40 +02:00

Added subscriber: @filedescriptor

Falk David commented

2021-05-11 17:03:40 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Falk David commented

2021-05-11 17:03:40 +02:00

I can confirm this issue in the latest 3.0.0 Alpha.

Richard Antalik commented

2021-05-11 17:42:13 +02:00

Added subscriber: @iss

Richard Antalik commented

2021-05-11 17:42:13 +02:00

I would say that this works as documented - see https://docs.blender.org/manual/en/dev/grease_pencil/properties/masks.html

For comment that suggests layers to be rendered without masks, this would imply that compositing would be done as final stage of rendering and not part of layer itself. Looking at code it seems that part of layer drawing is applied mask. So to me this raher sounds like feature request.

This worked as @Pullup expects in 2.82 though.

I would say that this works as documented - see https://docs.blender.org/manual/en/dev/grease_pencil/properties/masks.html For comment that suggests layers to be rendered without masks, this would imply that compositing would be done as final stage of rendering and not part of layer itself. Looking at code it seems that part of layer drawing is applied mask. So to me this raher sounds like feature request. This worked as @Pullup expects in 2.82 though.

Falk David commented

2021-05-11 18:24:15 +02:00

@iss Not sure what explains the results seen in the report in the documentation you linked?

In BKE_gpencil_visible_stroke_iter after checking if a viewlayer is used, we include mask layers deliberately (even if they shouldn't be, since they are a different layer). The comment states:

/* If the layer is used as mask, cannot be filtered or the masking system
 * will crash because needs the mask layer in the draw pipeline. */

Which is correct, the renderer will try to mask the layer using the mask, which it can't find if it is skipped - and then crash. But this means we render the mask, even if it is not part of the layer. IMO this should not be the case.

@iss Not sure what explains the results seen in the report in the documentation you linked? In `BKE_gpencil_visible_stroke_iter` after checking if a viewlayer is used, we include mask layers deliberately (even if they shouldn't be, since they are a different layer). The comment states: ``` /* If the layer is used as mask, cannot be filtered or the masking system * will crash because needs the mask layer in the draw pipeline. */ ``` Which is correct, the renderer will try to mask the layer using the mask, which it can't find if it is skipped - and then crash. But this means we render the mask, even if it is not part of the layer. IMO this should not be the case.

Richard Antalik commented

2021-05-11 18:34:53 +02:00

In #88202#1159356, @filedescriptor wrote:
@iss Not sure what explains the results seen in the report in the documentation you linked?

It says If you want to make a full transparent masking you will have to set the mask layer(s) opacity to 0. which implies that mask becomes part of layer, that uses this mask.

In any case I would deffer this to developers that know how this should work at this point.

> In #88202#1159356, @filedescriptor wrote: > @iss Not sure what explains the results seen in the report in the documentation you linked? It says `If you want to make a full transparent masking you will have to set the mask layer(s) opacity to 0.` which implies that mask becomes part of layer, that uses this mask. In any case I would deffer this to developers that know how this should work at this point.

Falk David commented

2021-05-11 18:45:02 +02:00

@iss I see what you are saying now. I think this is still an issue because you cannot render that layer if it's used as a mask and you have to turn the opacity off. But then it's also debatable if we consider this a bug or not.

Antonio Vazquez commented

2021-05-11 20:27:25 +02:00

Added subscriber: @antoniov

Antonio Vazquez commented

2021-05-11 20:27:25 +02:00

We are assuming here the Samuel request is the way to go....and I'm not sure...maybe for some users it's the way to go, but for others no and they expect to filter only the main layer and consider masking list as attributes of the current layer.

Edit: But I'm not against the change...just want to be sure we do the right one.

We are assuming here the Samuel request is the way to go....and I'm not sure...maybe for some users it's the way to go, but for others no and they expect to filter only the main layer and consider masking list as attributes of the current layer. Edit: But I'm not against the change...just want to be sure we do the right one.

Richard Antalik commented

2021-05-11 20:49:33 +02:00

The described expected behavior differs from patch. By expected behavior I understand render primitive shapes and let compositor take over actual compositing. What can go wrong is that you want to use complex layer with 100 masks applied in total. You don't want to reconstruct that in compositor once again. On the other hand you may want to use some layers in compositor clean without masks, but you don't want to duplicate them and maintain them separately over long time.

So IMO it would be best if you could choose layer behavior for rendering.

The described expected behavior differs from patch. By expected behavior I understand render primitive shapes and let compositor take over actual compositing. What can go wrong is that you want to use complex layer with 100 masks applied in total. You don't want to reconstruct that in compositor once again. On the other hand you may want to use some layers in compositor clean without masks, but you don't want to duplicate them and maintain them separately over long time. So IMO it would be best if you could choose layer behavior for rendering.

Samuel Bernou commented

2021-05-11 21:01:00 +02:00

I agree with @iss, having the choice would be the best solution since both cases can actually happen.
A user might want to have his multiples masks applied on the layer directly at render, and more advanced users would want to have control over the compositing of the layers.

But in both cases, you just never want to see the "only used for masking" layers (except if they are deliberately added in this view layer), so in this regard the patch is already a blessing :).

I agree with @iss, having the choice would be the best solution since both cases can actually happen. A user might want to have his multiples masks applied on the layer directly at render, and more advanced users would want to have control over the compositing of the layers. But in both cases, you just never want to see the "only used for masking" layers (except if they are deliberately added in this view layer), so in this regard the patch is already a blessing :).

Antonio Vazquez commented

2021-05-12 10:52:15 +02:00

Maybe we could add an option to enable new filtering mode and keep both methods:

Maybe we could add an option to enable new filtering mode and keep both methods: ![image.png](https://archive.blender.org/developer/F10087538/image.png)

Falk David commented

2021-05-12 10:57:09 +02:00

@antoniov I thought about that too. Maybe a checkbox "Disable Masks in Viewlayer"? And it would be a setting at the layer level? (Grayed-out when Viewlayer is empty)

Antonio Vazquez commented

2021-05-12 11:01:43 +02:00

Yes, by layer and the name is OK...we can change it later.

Falk David self-assigned this 2021-05-12 11:03:56 +02:00

Falk David commented

2021-05-12 11:03:56 +02:00

Ok, I think I can make that work.

Falk David commented

2021-05-12 12:39:50 +02:00

Updated the "Fix" to a patch since it adds a new option to the Relations panel in the Layer properties.
Now you can disable the masking for the view layer render.

Updated the "Fix" to a patch since it adds a new option to the Relations panel in the Layer properties. Now you can disable the masking for the view layer render.

blender-admin commented

2021-05-26 16:07:03 +02:00

This issue was referenced by d5a5575685

This issue was referenced by d5a5575685b650d297b40ace925042ce1525ecae

Falk David commented

2021-05-26 16:37:01 +02:00

Changed status from 'Confirmed' to: 'Resolved'

Falk David closed this issue

2021-05-26 16:37:01 +02:00

Falk David commented

2021-05-26 16:37:01 +02:00

Going to mark this as resolved now. The commit d5a5575685 fixes the problem of the mask showing in the view layer render (2.93). And the patch D11234 will add the option to disable masks in the render entirely (3.0.0).

Going to mark this as resolved now. The commit d5a5575685 fixes the problem of the mask showing in the view layer render (2.93). And the patch [D11234](https://archive.blender.org/developer/D11234) will add the option to disable masks in the render entirely (3.0.0).

blender-admin commented

2021-05-26 16:46:00 +02:00

This issue was referenced by e459a25e6c

This issue was referenced by e459a25e6cbe9321ad25f87843e2fe5a8a2306f9

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Grease pencil : masked layers bring their mask dependencies in in their own view layer #88202