Page MenuHome

Freestyle: Rendered preview crashes when non-primary render layer is selected and disabled.
Closed, ResolvedPublic

Description

System Information

Ubuntu 14.04 + Geforce GTX670

Blender Version

Official release builds 2.68a, 2.69, 2.70a
Own build 3d9b415 (2014-04-28 16:55)

Short description of error

The summary says it all...

Exact steps for others to reproduce the error

Step to reproduce:

  1. Start Blender
  2. Enable Freestyle
  3. Start rendered preview
  4. Add a render layer and select it
  5. Disable the selected render layer

Then Blender crashes immediately.

Event Timeline

Shinsuke Irie (irie) raised the priority of this task from to 90.
Shinsuke Irie (irie) updated the task description. (Show Details)
Shinsuke Irie (irie) edited a custom field.
Bastien Montagne (mont29) lowered the priority of this task from 90 to 30.Apr 29 2014, 8:40 AM

I cannot reproduce this here… Please attach a crashing .blend file, will save us some time.

That's curious, I confirmed this on both Ubuntu and MS-Windows...

Anyway, attached a .blend file (

).
Pressing Shift+Z on 3D viewport and clicking a checkbox of "RenderLayer.001" should crash Blender.

Bastien Montagne (mont29) raised the priority of this task from 30 to 50.Apr 29 2014, 9:16 AM

Thanks for the file, I can confirm it now. :)

For what is worth, here is the bt:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffca57e700 (LWP 15498)]
0x0000000002c75e6a in prepare (bmain=0x7fffca57c780, re=0x606e0001e108, srl=0x60200020fd48)
    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:461
461		for (RenderPass *rpass = (RenderPass *)rl->passes.first; rpass; rpass = rpass->next) {
(gdb) print rl
$1 = (RenderLayer *) 0x0
(gdb) bt
#0  0x0000000002c75e6a in prepare (bmain=0x7fffca57c780, re=0x606e0001e108, srl=0x60200020fd48)
    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:461
#1  0x0000000002c76d97 in FRS_do_stroke_rendering (re=0x606e0001e108, srl=0x60200020fd48, render=1)
    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/freestyle/intern/blender_interface/FRS_freestyle.cpp:607
#2  0x00000000029c9e93 in add_freestyle (re=0x606e0001e108, render=1) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/render/intern/source/pipeline.c:1847
#3  0x00000000029c45a7 in RE_TileProcessor (re=0x606e0001e108) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/render/intern/source/pipeline.c:1171
#4  0x0000000002743046 in render_view3d_startjob (customdata=0x601c0004b7c8, stop=0x602800077dbc, do_update=0x602800077dba, UNUSED_progress=0x602800077dc0)
    at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/editors/render/render_internal.c:1222
#5  0x0000000001d17ce2 in do_job_thread (job_v=0x602800077d48) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:335
#6  0x0000000003bc4b21 in tslot_thread_start (tslot_p=0x600c006feb48) at /home/i7deb64/blender-2.5-svn/__work__/src/source/blender/blenlib/intern/threads.c:266
#7  0x00007ffff4e67a58 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.0
#8  0x00007ffff2aa7062 in start_thread (arg=0x7fffca57e700) at pthread_create.c:312
#9  0x00007fffec2d3a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thanks all, I confirmed the crash as well on my side.
I believe I found a fix which I need to test a bit more.
I will update the task soon.

Tamito Kajiyama (kjym3) changed the task status from Unknown Status to Resolved.Apr 29 2014, 1:42 PM

Closed by commit rB182e97a2cd4b.

Thanks for the fix, the crash doesn't occur anymore.

However, Freestyle's behavior is still odd.
If non-primary render layer is selected and disabled, Freestyle draws outlines using primary layer's settings. If the primary one is selected and disabled, no outlines are drawn.

Yes, I do see the point. I also felt like the fixed behavior looked odd. But after some thoughts I concluded that everything is consistent.

If non-primary render layer is selected and disabled, Freestyle draws outlines using primary layer's settings.

This is because in that case Blender internally resets the active scene render layer index to zero (i.e., the first in the list of scene render layers; see line 606 of render_result.c in rB182e97a). Freestyle then uses the line settings of the first scene render layer.

If the primary one is selected and disabled, no outlines are drawn.

In that case, because of the aforementioned resetting of the active scene render layer index to zero, Freestyle tries to use the line settings of the first scene render layer and finds it disabled. That is why no outlines are drawn. Even Blender renders everything in the scene no matter how object visibility is specified in scene render layers.

Let us visually document the observations:

Panels (a) and (b) of the attached screen captures show two scene render layers displaying a cube and a sphere, respectively. The outlines of these objects are drawn in different colors specified by individual line settings as shown in the panels.

Panel (c) shows a preview render in the case that the first scene render layer (Cube) is active but disabled. Note that all objects in the scene are rendered regardless of object visibility settings by the scene render layers. Freestyle does not render outlines because the active scene render layer is disabled (no way to choose a line style).

Panel (d) shows the case where the second scene render layer (Sphere) is active but disabled. In this case, Blender uses the first scene render layer as if it is active. Now this active (first) scene render layer is enabled, so that Freestyle uses the line settings specified by the layer.

I found it strange that Blender internally (and silently) resets the active scene render layer when the true active scene render layer is disabled. All the oddity comes from this behavior of Blender, although we don't have much choice in that case (I would say this behavior is a compromise).

It is also remarked that Freestyle rendering is consistent with what is drawn by preview rendering.

Ah, got it. Thanks for the investigation.

I'm now testing an alternative solution as follows:

diff --git a/source/blender/render/intern/source/render_result.c b/source/blender/render/intern/source/render_result.c
index 9056305..792b3ba 100644
--- a/source/blender/render/intern/source/render_result.c
+++ b/source/blender/render/intern/source/render_result.c
@@ -483,9 +483,7 @@ RenderResult *render_result_new(Render *re, rcti *partrct, int crop, int savebuf
                        if (strcmp(srl->name, layername) != 0)
                                continue;
 
-               if ((re->r.scemode & R_SINGLE_LAYER) && nr != re->r.actlay)
-                       continue;
-               if (srl->layflag & SCE_LAY_DISABLE)
+               if ((re->r.scemode & R_SINGLE_LAYER) ? (nr != re->r.actlay) : (srl->layflag & SCE_LAY_DISABLE))
                        continue;
                
                rl = MEM_callocN(sizeof(RenderLayer), "new render layer");
@@ -604,8 +602,6 @@ RenderResult *render_result_new(Render *re, rcti *partrct, int crop, int savebuf
                rl->passflag = SCE_PASS_COMBINED;
                
                re->r.actlay = 0;
-               srl = BLI_findlink(&re->r.layers, re->r.actlay);
-               BLI_strncpy(rl->name, srl->name, sizeof(rl->name));
        }
        
        /* border render; calculate offset for use in compositor. compo is centralized coords */

SCE_LAY_DISABLE flag is ignored if R_SINGLE_LAYER is true. The default render layer is never used for rendered preview, so the previous fix rB182e97a2cd4b is no longer needed.
So far it seems to work but I'm still not sure if further changes are needed...

Tamito Kajiyama (kjym3) changed the task status from Resolved to Unknown Status.Apr 30 2014, 5:59 PM

Thanks for the alternative solution, which looks okay to me.
Since this task has already been closed, the new fix may get unattended and lost.
I reopen the task to draw attention of other core Blender developer to it.

Hi Brecht and Campbell,

I would appreciate it if either of you could take a look at the alternative fix by @Shinsuke Irie (irie) for this task.

This changes behavior, previously it would render the active layer regardless if it is enabled when use single layer is on.

Can't freestyle just follow the same logic as this function? Checking the same flags etc, not sure why the render result code needs to change.

Tamito Kajiyama (kjym3) changed the task status from Unknown Status to Resolved.May 1 2014, 3:20 PM

Freestyle can follow the same logic thanks to the commit rB182e97a2cd4b.
It looks like that just keeping the logic unchanged is the way to go.

Thank you Brecht for the timely response. And thank you Irie-san for the report and alternative solution.
I re-close the task now.

The behavior change is intentional. I'd like to fix the existing behavior that's confusing and a bit buggy. Currently, if active layer is disabled, the rendered viewport shows all layers for actlay=0, and shows primary layer (or isn't updated) for actlay>0.

If applying my patch, the viewport always shows an active layer regardless of whether it's enabled or disabled.

Ok, I see what you mean. Can you write it like this though, it's quite unusual to use ?: inside an if():

if (re->r.scemode & R_SINGLE_LAYER) {
    if(nr != re->r.actlay)
        continue;
}
else {
    if(srl->layflag & SCE_LAY_DISABLE))
        continue;
}