Freestyle: Rendered preview crashes when non-primary render layer is selected and disabled. #39941

New Issue

Shinsuke Irie · 2014-04-28T22:17:13+02:00

Shinsuke Irie commented

2014-04-28 22:17:13 +02:00

System Information

Ubuntu 14.04 + Geforce GTX670

Blender Version

Official release builds 2.68a, 2.69, 2.70a
Own build 3d9b415 (2014-04-28 16:55)

Short description of error

The summary says it all...

Exact steps for others to reproduce the error

Step to reproduce:

Start Blender
Enable Freestyle
Start rendered preview
Add a render layer and select it
Disable the selected render layer

Then Blender crashes immediately.

**System Information** Ubuntu 14.04 + Geforce GTX670 **Blender Version** Official release builds 2.68a, 2.69, 2.70a Own build 3d9b415 (2014-04-28 16:55) **Short description of error** The summary says it all... **Exact steps for others to reproduce the error** Step to reproduce: 1. Start Blender 2. Enable Freestyle 3. Start rendered preview 4. Add a render layer and select it 5. Disable the selected render layer Then Blender crashes immediately.

Shinsuke Irie commented

2014-04-28 22:17:13 +02:00

Changed status to: 'Open'

Tamito Kajiyama was assigned by Shinsuke Irie

2014-04-28 22:17:13 +02:00

Shinsuke Irie commented

2014-04-28 22:17:13 +02:00

Added subscriber: @IRIEShinsuke

Bastien Montagne commented

2014-04-29 08:40:45 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2014-04-29 08:40:45 +02:00

I cannot reproduce this here… Please attach a crashing .blend file, will save us some time.

Shinsuke Irie commented

2014-04-29 09:11:07 +02:00

That's curious, I confirmed this on both Ubuntu and MS-Windows...

Anyway, attached a .blend file (freestyle_crash_deleting_nonprimary_renderlayer.blend).
Pressing Shift+Z on 3D viewport and clicking a checkbox of "RenderLayer.001" should crash Blender.

That's curious, I confirmed this on both Ubuntu and MS-Windows... Anyway, attached a .blend file ([freestyle_crash_deleting_nonprimary_renderlayer.blend](https://archive.blender.org/developer/F86345/freestyle_crash_deleting_nonprimary_renderlayer.blend)). Pressing Shift+Z on 3D viewport and clicking a checkbox of "RenderLayer.001" should crash Blender.

Bastien Montagne commented

2014-04-29 09:16:58 +02:00

Thanks for the file, I can confirm it now. :)

For what is worth, here is the bt:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffca57e700 (LWP 15498)]

0x0000000002c75e6a in prepare (bmain=0x7fffca57c780, re=0x606e0001e108, srl=0x60200020fd48)

    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:461

461 for (RenderPass *rpass = (RenderPass *)rl->passes.first; rpass; rpass = rpass->next) {

(gdb) print rl
$1 = (RenderLayer *) 0x0
(gdb) bt

#0 0x0000000002c75e6a in prepare (bmain=0x7fffca57c780, re=0x606e0001e108, srl=0x60200020fd48)

    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:461

#1 0x0000000002c76d97 in FRS_do_stroke_rendering (re=0x606e0001e108, srl=0x60200020fd48, render=1)

    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:607

2 0x00000000029c9e93 in add_freestyle (re=0x606e0001e108, render=1) at /home/i7deb64/blender-2.5-svn/work/src/source/blender/render/intern/source/pipeline.c:1847
3 0x00000000029c45a7 in RE_TileProcessor (re=0x606e0001e108) at /home/i7deb64/blender-2.5-svn/work/src/source/blender/render/intern/source/pipeline.c:1171
#4 0x0000000002743046 in render_view3d_startjob (customdata=0x601c0004b7c8, stop=0x602800077dbc, do_update=0x602800077dba, UNUSED_progress=0x602800077dc0)

    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/editors/render/render_internal.c:1222

5 0x0000000001d17ce2 in do_job_thread (job_v=0x602800077d48) at /home/i7deb64/blender-2.5-svn/work/src/source/blender/windowmanager/intern/wm_jobs.c:335
6 0x0000000003bc4b21 in tslot_thread_start (tslot_p=0x600c006feb48) at /home/i7deb64/blender-2.5-svn/work/src/source/blender/blenlib/intern/threads.c:266
7 0x00007ffff4e67a58 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.0
8 0x00007ffff2aa7062 in start_thread (arg=0x7fffca57e700) at pthread_create.c:312
9 0x00007fffec2d3a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thanks for the file, I can confirm it now. :) For what is worth, here is the bt: ``` Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffca57e700 (LWP 15498)] ``` 0x0000000002c75e6a in prepare (bmain=0x7fffca57c780, re=0x606e0001e108, srl=0x60200020fd48) ``` at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:461 ``` 461 for (RenderPass *rpass = (RenderPass *)rl->passes.first; rpass; rpass = rpass->next) { ``` (gdb) print rl $1 = (RenderLayer *) 0x0 (gdb) bt ``` #0 0x0000000002c75e6a in prepare (bmain=0x7fffca57c780, re=0x606e0001e108, srl=0x60200020fd48) ``` at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:461 ``` #1 0x0000000002c76d97 in FRS_do_stroke_rendering (re=0x606e0001e108, srl=0x60200020fd48, render=1) ``` at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:607 ``` - 2 0x00000000029c9e93 in add_freestyle (re=0x606e0001e108, render=1) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/render/intern/source/pipeline.c:1847 - 3 0x00000000029c45a7 in RE_TileProcessor (re=0x606e0001e108) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/render/intern/source/pipeline.c:1171 #4 0x0000000002743046 in render_view3d_startjob (customdata=0x601c0004b7c8, stop=0x602800077dbc, do_update=0x602800077dba, UNUSED_progress=0x602800077dc0) ``` at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/editors/render/render_internal.c:1222 ``` - 5 0x0000000001d17ce2 in do_job_thread (job_v=0x602800077d48) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:335 - 6 0x0000000003bc4b21 in tslot_thread_start (tslot_p=0x600c006feb48) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/blenlib/intern/threads.c:266 - 7 0x00007ffff4e67a58 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.0 - 8 0x00007ffff2aa7062 in start_thread (arg=0x7fffca57e700) at pthread_create.c:312 - 9 0x00007fffec2d3a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Tamito Kajiyama commented

2014-04-29 10:56:04 +02:00

Thanks all, I confirmed the crash as well on my side.
I believe I found a fix which I need to test a bit more.
I will update the task soon.

Thanks all, I confirmed the crash as well on my side. I believe I found a fix which I need to test a bit more. I will update the task soon.

blender-admin commented

2014-04-29 13:41:48 +02:00

This issue was referenced by blender/blender-addons-contrib@182e97a2cd

This issue was referenced by blender/blender-addons-contrib@182e97a2cd4bdca9709dbbd1a4e6c175aed448a6

blender-admin commented

2014-04-29 13:41:48 +02:00

This issue was referenced by 182e97a2cd

This issue was referenced by 182e97a2cd4bdca9709dbbd1a4e6c175aed448a6

Tamito Kajiyama commented

2014-04-29 13:42:29 +02:00

Changed status from 'Open' to: 'Resolved'

Tamito Kajiyama closed this issue

2014-04-29 13:42:29 +02:00

Tamito Kajiyama commented

2014-04-29 13:42:29 +02:00

Closed by commit 182e97a2cd.

Closed by commit 182e97a2cd.

Shinsuke Irie commented

2014-04-29 20:41:55 +02:00

Thanks for the fix, the crash doesn't occur anymore.

However, Freestyle's behavior is still odd.
If non-primary render layer is selected and disabled, Freestyle draws outlines using primary layer's settings. If the primary one is selected and disabled, no outlines are drawn.

Thanks for the fix, the crash doesn't occur anymore. However, Freestyle's behavior is still odd. If non-primary render layer is selected and disabled, Freestyle draws outlines using primary layer's settings. If the primary one is selected and disabled, no outlines are drawn.

Tamito Kajiyama commented

2014-04-30 03:00:51 +02:00

Yes, I do see the point. I also felt like the fixed behavior looked odd. But after some thoughts I concluded that everything is consistent.

If non-primary render layer is selected and disabled, Freestyle draws outlines using primary layer's settings.

This is because in that case Blender internally resets the active scene render layer index to zero (i.e., the first in the list of scene render layers; see line 606 of render_result.c in 182e97a). Freestyle then uses the line settings of the first scene render layer.

If the primary one is selected and disabled, no outlines are drawn.

In that case, because of the aforementioned resetting of the active scene render layer index to zero, Freestyle tries to use the line settings of the first scene render layer and finds it disabled. That is why no outlines are drawn. Even Blender renders everything in the scene no matter how object visibility is specified in scene render layers.

Let us visually document the observations:

Panels (a) and (b) of the attached screen captures show two scene render layers displaying a cube and a sphere, respectively. The outlines of these objects are drawn in different colors specified by individual line settings as shown in the panels.

Panel (c) shows a preview render in the case that the first scene render layer (Cube) is active but disabled. Note that all objects in the scene are rendered regardless of object visibility settings by the scene render layers. Freestyle does not render outlines because the active scene render layer is disabled (no way to choose a line style).

Panel (d) shows the case where the second scene render layer (Sphere) is active but disabled. In this case, Blender uses the first scene render layer as if it is active. Now this active (first) scene render layer is enabled, so that Freestyle uses the line settings specified by the layer.

I found it strange that Blender internally (and silently) resets the active scene render layer when the true active scene render layer is disabled. All the oddity comes from this behavior of Blender, although we don't have much choice in that case (I would say this behavior is a compromise).

It is also remarked that Freestyle rendering is consistent with what is drawn by preview rendering.

Yes, I do see the point. I also felt like the fixed behavior looked odd. But after some thoughts I concluded that everything is consistent. > If non-primary render layer is selected and disabled, Freestyle draws outlines using primary layer's settings. This is because in that case Blender internally resets the active scene render layer index to zero (i.e., the first in the list of scene render layers; see line 606 of render_result.c in 182e97a). Freestyle then uses the line settings of the first scene render layer. > If the primary one is selected and disabled, no outlines are drawn. In that case, because of the aforementioned resetting of the active scene render layer index to zero, Freestyle tries to use the line settings of the first scene render layer and finds it disabled. That is why no outlines are drawn. Even Blender renders everything in the scene no matter how object visibility is specified in scene render layers. Let us visually document the observations: ![preview_render.png](https://archive.blender.org/developer/F86423/preview_render.png) Panels (a) and (b) of the attached screen captures show two scene render layers displaying a cube and a sphere, respectively. The outlines of these objects are drawn in different colors specified by individual line settings as shown in the panels. Panel (c) shows a preview render in the case that the first scene render layer (Cube) is active but disabled. Note that all objects in the scene are rendered regardless of object visibility settings by the scene render layers. Freestyle does not render outlines because the active scene render layer is disabled (no way to choose a line style). Panel (d) shows the case where the second scene render layer (Sphere) is active but disabled. In this case, Blender uses the first scene render layer as if it is active. Now this active (first) scene render layer is enabled, so that Freestyle uses the line settings specified by the layer. I found it strange that Blender internally (and silently) resets the active scene render layer when the true active scene render layer is disabled. All the oddity comes from this behavior of Blender, although we don't have much choice in that case (I would say this behavior is a compromise). It is also remarked that Freestyle rendering is consistent with what is drawn by preview rendering.

Shinsuke Irie commented

2014-04-30 07:51:03 +02:00

Added subscriber: @brecht

Shinsuke Irie commented

2014-04-30 07:51:03 +02:00

Ah, got it. Thanks for the investigation.

I'm now testing an alternative solution as follows:

diff --git a/source/blender/render/intern/source/render_result.c b/source/blender/render/intern/source/render_result.c
index 9056305..792b3ba 100644
--- a/source/blender/render/intern/source/render_result.c
+++ b/source/blender/render/intern/source/render_result.c
@@ -483,9 +483,7 @@ RenderResult *render_result_new(Render *re, rcti *partrct, int crop, int savebuf
                        if (strcmp(srl->name, layername) != 0)
                                continue;
 
-               if ((re->r.scemode & R_SINGLE_LAYER) && nr != re->r.actlay)
-                       continue;
-               if (srl->layflag & SCE_LAY_DISABLE)
+               if ((re->r.scemode & R_SINGLE_LAYER) ? (nr != re->r.actlay) : (srl->layflag & SCE_LAY_DISABLE))
                        continue;
                
                rl = MEM_callocN(sizeof(RenderLayer), "new render layer");
@@ -604,8 +602,6 @@ RenderResult *render_result_new(Render *re, rcti *partrct, int crop, int savebuf
                rl->passflag = SCE_PASS_COMBINED;
                
                re->r.actlay = 0;
-               srl = BLI_findlink(&re->r.layers, re->r.actlay);
-               BLI_strncpy(rl->name, srl->name, sizeof(rl->name));
        }
        
        /* border render; calculate offset for use in compositor. compo is centralized coords */

SCE_LAY_DISABLE flag is ignored if R_SINGLE_LAYER is true. The default render layer is never used for rendered preview, so the previous fix 182e97a2cd is no longer needed.
So far it seems to work but I'm still not sure if further changes are needed...

Ah, got it. Thanks for the investigation. I'm now testing an alternative solution as follows: ``` diff --git a/source/blender/render/intern/source/render_result.c b/source/blender/render/intern/source/render_result.c index 9056305..792b3ba 100644 --- a/source/blender/render/intern/source/render_result.c +++ b/source/blender/render/intern/source/render_result.c @@ -483,9 +483,7 @@ RenderResult *render_result_new(Render *re, rcti *partrct, int crop, int savebuf if (strcmp(srl->name, layername) != 0) continue; - if ((re->r.scemode & R_SINGLE_LAYER) && nr != re->r.actlay) - continue; - if (srl->layflag & SCE_LAY_DISABLE) + if ((re->r.scemode & R_SINGLE_LAYER) ? (nr != re->r.actlay) : (srl->layflag & SCE_LAY_DISABLE)) continue; rl = MEM_callocN(sizeof(RenderLayer), "new render layer"); @@ -604,8 +602,6 @@ RenderResult *render_result_new(Render *re, rcti *partrct, int crop, int savebuf rl->passflag = SCE_PASS_COMBINED; re->r.actlay = 0; - srl = BLI_findlink(&re->r.layers, re->r.actlay); - BLI_strncpy(rl->name, srl->name, sizeof(rl->name)); } /* border render; calculate offset for use in compositor. compo is centralized coords */ ``` SCE_LAY_DISABLE flag is ignored if R_SINGLE_LAYER is true. The default render layer is never used for rendered preview, so the previous fix 182e97a2cd is no longer needed. So far it seems to work but I'm still not sure if further changes are needed...

Tamito Kajiyama commented

2014-04-30 17:59:59 +02:00

Changed status from 'Resolved' to: 'Open'

Tamito Kajiyama reopened this issue

2014-04-30 17:59:59 +02:00

Tamito Kajiyama commented

2014-04-30 17:59:59 +02:00

Thanks for the alternative solution, which looks okay to me.
Since this task has already been closed, the new fix may get unattended and lost.
I reopen the task to draw attention of other core Blender developer to it.

Thanks for the alternative solution, which looks okay to me. Since this task has already been closed, the new fix may get unattended and lost. I reopen the task to draw attention of other core Blender developer to it.

Tamito Kajiyama commented

2014-04-30 18:05:04 +02:00

Added subscriber: @ideasman42

Tamito Kajiyama commented

2014-04-30 18:05:04 +02:00

Hi Brecht and Campbell,

I would appreciate it if either of you could take a look at the alternative fix by @IRIEShinsuke for this task.

Hi Brecht and Campbell, I would appreciate it if either of you could take a look at the alternative fix by @IRIEShinsuke for this task.

Brecht Van Lommel commented

2014-05-01 14:21:26 +02:00

This changes behavior, previously it would render the active layer regardless if it is enabled when use single layer is on.

Can't freestyle just follow the same logic as this function? Checking the same flags etc, not sure why the render result code needs to change.

This changes behavior, previously it would render the active layer regardless if it is enabled when use single layer is on. Can't freestyle just follow the same logic as this function? Checking the same flags etc, not sure why the render result code needs to change.

Tamito Kajiyama commented

2014-05-01 15:20:48 +02:00

Changed status from 'Open' to: 'Resolved'

Tamito Kajiyama closed this issue

2014-05-01 15:20:48 +02:00

Tamito Kajiyama commented

2014-05-01 15:20:48 +02:00

Freestyle can follow the same logic thanks to the commit 182e97a2cd.
It looks like that just keeping the logic unchanged is the way to go.

Thank you Brecht for the timely response. And thank you Irie-san for the report and alternative solution.
I re-close the task now.

Freestyle can follow the same logic thanks to the commit 182e97a2cd. It looks like that just keeping the logic unchanged is the way to go. Thank you Brecht for the timely response. And thank you Irie-san for the report and alternative solution. I re-close the task now.

Shinsuke Irie commented

2014-05-01 22:03:05 +02:00

The behavior change is intentional. I'd like to fix the existing behavior that's confusing and a bit buggy. Currently, if active layer is disabled, the rendered viewport shows all layers for actlay=0, and shows primary layer (or isn't updated) for actlay>0.

If applying my patch, the viewport always shows an active layer regardless of whether it's enabled or disabled.

The behavior change is intentional. I'd like to fix the existing behavior that's confusing and a bit buggy. Currently, if active layer is disabled, the rendered viewport shows all layers for actlay=0, and shows primary layer (or isn't updated) for actlay>0. If applying my patch, the viewport always shows an active layer regardless of whether it's enabled or disabled.

Brecht Van Lommel commented

2014-05-02 18:26:12 +02:00

Ok, I see what you mean. Can you write it like this though, it's quite unusual to use ?: inside an if():

if (re->r.scemode & R_SINGLE_LAYER) {
    if(nr != re->r.actlay)
        continue;
}
else {
    if(srl->layflag & SCE_LAY_DISABLE))
        continue;
}

Ok, I see what you mean. Can you write it like this though, it's quite unusual to use `?:` inside an `if()`: ``` if (re->r.scemode & R_SINGLE_LAYER) { if(nr != re->r.actlay) continue; } else { if(srl->layflag & SCE_LAY_DISABLE)) continue; }

Shinsuke Irie commented

2014-05-03 11:34:07 +02:00

Committed 6ec2d72eca. Thanks!

Committed 6ec2d72eca. Thanks!

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Freestyle: Rendered preview crashes when non-primary render layer is selected and disabled. #39941