Collections: Exclude From View toggle causes segment violation. #72490

New Issue

Garry R. Osgood · 2019-12-16T22:22:42+01:00

Garry R. Osgood commented

2019-12-16 22:22:42 +01:00

System Information
Operating system: Linux-5.4.0-gentoo-bertha-x86_64-Intel-R-_Xeon-R-CPU_E5-2630_v4@_2.20GHz-with-gentoo-2.6 64 Bits
Graphics card: Radeon Pro WX 9100 (VEGA10, DRM 3.35.0, 5.4.0-gentoo-bertha, LLVM 8.0.1) X.Org 4.5 (Core Profile) Mesa 19.1.8

Blender Version
Broken: version: 2.82 (sub 5), branch: master, commit date: 2019-12-16 11:53, hash: 8d16dc029e
Worked: version: 2.81 (sub 16), branch: master, commit date: 2019-11-20 14:27, hash: 26bd5ebd42, type: Release
Bisection point - breaking commit: 9516921c05
Author: Clément Foucault foucault.clem@gmail.com
Date: Mon Dec 2 01:40:58 2019 +0100 Overlay Engine: Refactor & Cleanup

Short description of error
Unchecking, then rechecking a collection's Exclude from View checkbox, containing one mesh object in initially in Edit Mode, crashes Blender. However, to reliably reproduce the crash, follow the exact steps here. Deselecting the mesh object, or switching to object mode (or any other mode) seems to bypass the issue.
[Edit: note critical typo in original]

Exact steps for others to reproduce the error

Open colselect.blend, attached.
- Observe: One collection - "Mutt Collection" with one mesh object - "Mutt"
- Viewport Shading - Wireframe X-Ray toggled on.
- Collection Filters. Object visibility (eye), is enabled for both the object, "Mutt", and the collection, "Mutt_Collection". The "Exclude from View Layer" is unchecked.
- "Mutt" is the active object. Mutt is in edit mode. Some faces are selected. (picture 1)
Uncheck the "Exclude from View Layer" for "Mutt Collection". "Mutt" disappears from the viewport. Observe also, in the upper left corner edit mode selector, that the edit mode switches to Object Mode.
Check on the "Exclude from View Layer" for "Mutt Collection". "Mutt" reappears in the viewport, but is presented as only a faint outline. Still in object mode. Here, the viewport has the appearance shown in picture 2
Click on this faded representation of Mutt. Blender crashes. See Backtrace, GDB Notes.

picture 1: 1D

picture 2: 3

colselect.blend
colselect.blend

Backtrace

Thread 1 "blender" received signal SIGSEGV, Segmentation fault.

0x0000555578321039 in drw_command_create (shgroup=0x0, type=DRW_CMD_SELECTID) at

/home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:609

609 DRWCommandChunk *chunk = shgroup->cmd.last;

(gdb) bt 7

#0 0x0000555578321039 in drw_command_create (shgroup=0x0, type=DRW_CMD_SELECTID) at

/home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:609

#1 0x0000555578322230 in drw_command_set_select_id (shgroup=0x0, buf=0x0, select_id=1) at

/home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:678

#2 0x0000555578322ac0 in DRW_shgroup_call_ex (shgroup=0x0, ob=0x61b0003185b8, obmat=0x0, geom=0x6110007d8138, bypass_culling=true,

user_data=0x0) at /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:728

#3 0x00005555784ada10 in overlay_edit_mesh_add_ob_to_pass (pd=0x61c00026d8b8, ob=0x61b0003185b8, in_front=false) at

/home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_edit_mesh.c:259

#4 0x00005555784ae749 in OVERLAY_edit_mesh_cache_populate (vedata=0x61700029ddb8, ob=0x61b0003185b8) at

/home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_edit_mesh.c:328

#5 0x00005555784a0ede in OVERLAY_cache_populate (vedata=0x61700029ddb8, ob=0x61b0003185b8) at

/home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_engine.c:230

#6 0x00005555783083b8 in drw_engines_cache_populate (ob=0x61b0003185b8) at

/home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager.c:1145
(More stack frames follow...)

GDB Notes

The immediate cause of the crash is that ##DRWShadingGroup *shgroup## is null, but ##drw_command_create()## seems coded with the presumption that this should never be the case.
A few frames up at ##5## OVERLAY_cache_populate () , ## OVERLAY_PrivateData *pd = data->stl->pd## exhibits interesting characteristics:

  (gdb) p *pd
  $1 = {
  armature_bone_select_act_grp = 0x0, 
  armature_bone_select_grp = 0x0, 
  edit_curve_normal_grp = {0x0, 0x0}, 
  edit_curve_wire_grp = {0x0, 0x0}, 
  edit_curve_handle_grp = 0x0, 
  edit_curve_points_grp = 0x0, 
  edit_lattice_points_grp = 0x0, 
  edit_lattice_wires_grp = 0x0, 
  edit_mesh_depth_grp = {0x0, 0x0}, 
  edit_mesh_faces_grp = {0x0, 0x0}, 
  edit_mesh_faces_cage_grp = {0x0, 0x0}, 
  edit_mesh_verts_grp = {0x0, 0x0}, 
  edit_mesh_edges_grp = {0x0, 0x0}, 
  edit_mesh_facedots_grp = {0x0, 0x0}, 
  edit_mesh_skin_roots_grp = {0x0, 0x0}, 
  edit_mesh_normals_grp = 0x0, 
  edit_mesh_analysis_grp = 0x0, 
  edit_mesh_weight_grp = 0x0, 
  edit_particle_strand_grp = 0x0, 
  edit_particle_point_grp = 0x0, 
  edit_text_overlay_grp = 0x0, 
  edit_text_wire_grp = {0x0, 0x0}, 
**(remainder omitted for brevity)** ...}

In the usual case, most of these pointers are populated - in particular, the ##edit_mesh_*_grp## always seem pre-populated. Two frames up at ##3## in overlay_edit_mesh_add_ob_to_pass() ##pd->edit_mesh_edges_grp- [x]## is set to ##edge_shgrp## and is passed to ##DRW_shgroup_call_no_cull(edge_shgrp, geom_edges, ob)##, leading directly to the observed segment violation.

[Edit: omission]
Also at frame ##5## in OVERLAY_cache_populate() sets the ##in_edit_mode## flag to ##TRUE##, suggesting that the object being passed in, ##"OBMutt"## is itself still in edit mode.

(gdb) p in_edit_mode
$3 = true
(gdb) p ob->id.name
$4 = "OBMutt", '\000' <repeats 59 times>

As noted above, this behavior was not present in 2.81, but appears in builds after the Refactor & Cleanup commit. Stopped to do this report. Evening is still young here; plan to poke around some more.

system-info.txt
system info for stack trace

**System Information** Operating system: Linux-5.4.0-gentoo-bertha-x86_64-Intel-R-_Xeon-R-_CPU_E5-2630_v4_@_2.20GHz-with-gentoo-2.6 64 Bits Graphics card: Radeon Pro WX 9100 (VEGA10, DRM 3.35.0, 5.4.0-gentoo-bertha, LLVM 8.0.1) X.Org 4.5 (Core Profile) Mesa 19.1.8 **Blender Version** Broken: version: 2.82 (sub 5), branch: master, commit date: 2019-12-16 11:53, hash: 8d16dc029e Worked: version: 2.81 (sub 16), branch: master, commit date: 2019-11-20 14:27, hash: 26bd5ebd42, type: Release Bisection point - breaking commit: 9516921c05 Author: Clément Foucault <foucault.clem@gmail.com> Date: Mon Dec 2 01:40:58 2019 +0100 **Overlay Engine: Refactor & Cleanup** **Short description of error** Unchecking, then rechecking a collection's Exclude from View checkbox, containing one mesh object in initially in Edit Mode, crashes Blender. However, to reliably reproduce the crash, follow the exact steps here. Deselecting the mesh object, or switching to object mode (or any other mode) seems to bypass the issue. [Edit: note critical typo in original] ### Exact steps for others to reproduce the error - Open **colselect.blend**, attached. - Observe: One collection - "Mutt Collection" with one mesh object - "Mutt" - Viewport Shading - Wireframe X-Ray toggled on. - Collection Filters. Object visibility (eye), is enabled for both the object, "Mutt", and the collection, "Mutt_Collection". The "Exclude from View Layer" is ~~un~~checked. - "Mutt" is the active object. Mutt is in edit mode. Some faces are selected. (**picture 1**) - Uncheck the "Exclude from View Layer" for "Mutt Collection". "Mutt" disappears from the viewport. Observe also, in the upper left corner edit mode selector, that the edit mode switches to Object Mode. - Check on the "Exclude from View Layer" for "Mutt Collection". "Mutt" reappears in the viewport, but is presented as only a faint outline. Still in object mode. Here, the viewport has the appearance shown in **picture 2** - Click on this faded representation of Mutt. Blender crashes. See **Backtrace**, **GDB Notes**. ![Screenshot at 2019-12-16 16-08-26.png](https://archive.blender.org/developer/F8227134/Screenshot_at_2019-12-16_16-08-26.png) **picture 1: 1D** ![Screenshot at 2019-12-16 16-09-05_2.png](https://archive.blender.org/developer/F8227138/Screenshot_at_2019-12-16_16-09-05_2.png) **picture 2: 3** [colselect.blend](https://archive.blender.org/developer/F8226836/colselect.blend) **colselect.blend** ## Backtrace ``` Thread 1 "blender" received signal SIGSEGV, Segmentation fault. ``` 0x0000555578321039 in drw_command_create (shgroup=0x0, type=DRW_CMD_SELECTID) at ``` /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:609 ``` 609 DRWCommandChunk *chunk = shgroup->cmd.last; ``` (gdb) bt 7 ``` #0 0x0000555578321039 in drw_command_create (shgroup=0x0, type=DRW_CMD_SELECTID) at ``` /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:609 ``` #1 0x0000555578322230 in drw_command_set_select_id (shgroup=0x0, buf=0x0, select_id=1) at ``` /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:678 ``` #2 0x0000555578322ac0 in DRW_shgroup_call_ex (shgroup=0x0, ob=0x61b0003185b8, obmat=0x0, geom=0x6110007d8138, bypass_culling=true, ``` user_data=0x0) at /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager_data.c:728 ``` #3 0x00005555784ada10 in overlay_edit_mesh_add_ob_to_pass (pd=0x61c00026d8b8, ob=0x61b0003185b8, in_front=false) at ``` /home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_edit_mesh.c:259 ``` #4 0x00005555784ae749 in OVERLAY_edit_mesh_cache_populate (vedata=0x61700029ddb8, ob=0x61b0003185b8) at ``` /home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_edit_mesh.c:328 ``` #5 0x00005555784a0ede in OVERLAY_cache_populate (vedata=0x61700029ddb8, ob=0x61b0003185b8) at ``` /home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_engine.c:230 ``` #6 0x00005555783083b8 in drw_engines_cache_populate (ob=0x61b0003185b8) at ``` /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager.c:1145 (More stack frames follow...) ``` ## GDB Notes - The immediate cause of the crash is that ##DRWShadingGroup *shgroup## is null, but ##drw_command_create()## seems coded with the presumption that this should never be the case. - A few frames up at ##5## [OVERLAY_cache_populate () ](https://developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$180), ## OVERLAY_PrivateData *pd = data->stl->pd## exhibits interesting characteristics: ``` (gdb) p *pd $1 = { armature_bone_select_act_grp = 0x0, armature_bone_select_grp = 0x0, edit_curve_normal_grp = {0x0, 0x0}, edit_curve_wire_grp = {0x0, 0x0}, edit_curve_handle_grp = 0x0, edit_curve_points_grp = 0x0, edit_lattice_points_grp = 0x0, edit_lattice_wires_grp = 0x0, edit_mesh_depth_grp = {0x0, 0x0}, edit_mesh_faces_grp = {0x0, 0x0}, edit_mesh_faces_cage_grp = {0x0, 0x0}, edit_mesh_verts_grp = {0x0, 0x0}, edit_mesh_edges_grp = {0x0, 0x0}, edit_mesh_facedots_grp = {0x0, 0x0}, edit_mesh_skin_roots_grp = {0x0, 0x0}, edit_mesh_normals_grp = 0x0, edit_mesh_analysis_grp = 0x0, edit_mesh_weight_grp = 0x0, edit_particle_strand_grp = 0x0, edit_particle_point_grp = 0x0, edit_text_overlay_grp = 0x0, edit_text_wire_grp = {0x0, 0x0}, **(remainder omitted for brevity)** ...} ``` In the usual case, most of these pointers are populated - in particular, the ##edit_mesh_*_grp## always seem pre-populated. Two frames up at ##3## in [overlay_edit_mesh_add_ob_to_pass() ](https://developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_edit_mesh.c$259) ##pd->edit_mesh_edges_grp- [x]## is set to ##edge_shgrp## and is passed to ##DRW_shgroup_call_no_cull(edge_shgrp, geom_edges, ob)##, leading directly to the observed segment violation. [Edit: omission] Also at frame ##5## in [OVERLAY_cache_populate() ](https://developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$189) sets the ##in_edit_mode## flag to ##TRUE##, suggesting that the object being passed in, ##"OBMutt"## is itself still in edit mode. ``` (gdb) p in_edit_mode $3 = true (gdb) p ob->id.name $4 = "OBMutt", '\000' <repeats 59 times> ``` As noted above, this behavior was not present in 2.81, but appears in builds after the Refactor & Cleanup commit. Stopped to do this report. Evening is still young here; plan to poke around some more. [system-info.txt](https://archive.blender.org/developer/F8227132/system-info.txt) **system info for stack trace**

Garry R. Osgood commented

2019-12-16 22:22:42 +01:00

Added subscriber: @grosgood

Dalai Felinto commented

2019-12-16 22:50:11 +01:00

Added subscribers: @dfelinto, @fclem

Dalai Felinto commented

2019-12-16 22:50:11 +01:00

Yet to go over the steps to reproduce. But adding the interested parties as subscribed meanwhile.

Garry R. Osgood commented

2019-12-22 23:33:06 +01:00

Debug notes. I won't revisit this until after the 25th, and, with a head full of grog, will need notes to recover context.

Here is how ##OVERLAY_PrivateData *pd = data->stl->pd## in OVERLAY_cache_populate() winds up 'exhibiting interesting characteristics', as illustrated in the previous comment. It seems that ##OVERLAY_cache_populate()## is unaware of various drawing mode contexts within which callers operate, and how callers can set up ##OVERLAY_PrivateData *pd## in different ways. That's the bug in a nutshell. Gory details follow.

On the mouse click over "Mutt" ##OVERLAY_cache_populate()## executes:

- 0  0x000055557a3b59b0 in OVERLAY_cache_populate (vedata=0x6170002b3838, ob=0x61b0002f86b8) at /home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_engine.c:189
- 1  0x000055557a21d3b9 in drw_engines_cache_populate (ob=0x61b0002f86b8) at /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager.c:1145
- 2  0x000055557a227b2c in DRW_draw_select_loop (depsgraph=0x6130003ab8f8, ar=0x6150001c5038, v3d=0x61a0001c08b8, UNUSED_use_obedit_skip=false, draw_surface=false, UNUSED_use_nearest=false, rect=0x7fffffff2830, select_pass_fn=0x55557d1d7576 <drw_select_loop_pass>, select_pass_user_data=0x7fffffff28c0, object_filter_fn=0x0, object_filter_user_data=0x0) at /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager.c:2286
- 3  0x000055557d1daa1d in view3d_opengl_select (vc=0x7fffffff2fb0, buffer=0x7fffffff3040, bufsize=10000, input=0x7fffffff2d60, select_mode=VIEW3D_SELECT_PICK_ALL, select_filter=VIEW3D_SELECT_FILTER_OBJECT_MODE_LOCK) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_view.c:1149
- 4  0x000055557d18a0a7 in mixed_bones_object_selectbuffer (vc=0x7fffffff2fb0, buffer=0x7fffffff3040, mval=0x7fffffffce40, select_filter=VIEW3D_SELECT_FILTER_OBJECT_MODE_LOCK, do_nearest=false) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:1684
- 5  0x000055557d18af94 in mixed_bones_object_selectbuffer_extended (vc=0x7fffffff2fb0, buffer=0x7fffffff3040, mval=0x7fffffffce40, select_filter=VIEW3D_SELECT_FILTER_OBJECT_MODE_LOCK, use_cycle=true, enumerate=false, r_do_nearest=0x7fffffff2f40) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:1783
- 6  0x000055557d18e171 in ed_object_select_pick (C=0x610000000c78, mval=0x7fffffffce40, extend=false, deselect=false, toggle=false, obcenter=false, enumerate=false, object=false) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:2061
- 7  0x000055557d191f5e in view3d_select_exec (C=0x610000000c78, op=0x611000809d78) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:2421
#8  0x000055557d192252 in view3d_select_invoke (C=0x610000000c78, op=0x611000809d78, event=0x6100008e2c78) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:2460
(remainder omitted for brevity)

##OVERLAY_cache_populate()##, depends on a prior invocation of OVERLAY_edit_mesh_cache_init() , which, among other things, populates the block of ##DRWShadingGroup## references in ##OVERLAY_PrivateData pd = data->stl->pd## (the ##edit_mesh__grp## and friends), and this prior invocation*almost always// takes place -- but not here, where Exclude from View Layer has been toggled.

##OVERLAY_edit_mesh_cache_init()##, upon which ##OVERLAY_cache_populate()## depends, is called upstream at [drw_engines_cache_init() ]], which iterates over a list of 'engine' objects of ##DrawEngineType##; the variable ##engine## at line #1099 successively assumes references to all active engines during the course of iteration, including the overlay engine, defined at https:*developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$466. The member ##cache_init## contains a reference to an engine-specific cache initializer. ##engine## eventually assumes a reference to the overlay engine and its ##cache_init## member points to the overlay implementation [ https://developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$99|OVERLAY_cache_init(void *vedata) . This function conditionally calls ##OVERLAY_edit_mesh_cache_init()## predicated on the draw mode state ##pd->ctx_mode == CTX_MODE_EDIT_MESH##.

However, with the click on "Mutt", the viewport is not in edit mode; it is in object mode, a condition arising from toggling Exclude from View Layer. See the screen shots in the previous comment. The draw mode ##pd->ctx_mode == CTX_MODE_OBJECT## prevails instead so ##OVERLAY_edit_mesh_cache_init()## does not execute and leaves ##OVERLAY_PrivateData *pd## in ##OVERLAY_cache_populate()## without the block of ##DRWShadingGroup## references set up. Having established that ##Mutt## is to be drawn as an object, ##OVERLAY_cache_init()## completes the rest of the initialization of what will be ##OVERLAY_PrivateData pd## in ##OVERLAY_cache_populate()## and returns to ##drw_engines_cache_init()##, which, in due course, returns to its caller ##DRW_draw_select_loop().##

With cache initialization complete, ##DRW_draw_select_loop()## invokes [drw_engines_cache_populate(Object *ob) ]] , which performs a similar iteration over engine objects seen in ##drw_engines_cache_init()##, but invoking the engines' ##engine->cache_populate(data, ob)## implementations instead. For the 'Overlay' engine, that implementation is [ https:*developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$180|OVERLAY_cache_populate(void *vedata, Object *ob) , which extracts private data ##OVERLAY_PrivateData *pd## from parameter ##ViewportEngineData vedata## (recast in the function call to ##OVERLAY_data##). As noted, ##pd## lacks the benefit of ##OVERLAY_edit_mesh_cache_init()##, as ##OVERLAY_cache_init()## calls that function only in a draw mode of ##CTX_MODE_EDIT_MESH##, not the currently prevailing one: ##CTX_MODE_OBJECT##. Put another way, ##OVERLAY_cache_populate()## lives in draw mode context ##CTX_MODE_EDIT_MESH##, in that it assumes its prerequisite, ##OVERLAY_edit_mesh_cache_init(),## always runs, but, alas, it missed the memo announcing it must also populate in ##CTX_MODE_OBJECT## too. Having missed the memo, it proceeds as if ##DRWShadingGroup## references live in ##pd## when they do not. To sustain the illusion, ## BKE_object_is_in_editmode(ob)## returns true, assigning that to ##in_edit_mode## and setting the stage for calling ##OVERLAY_edit_mesh_cache_populate(vedata, ob)## and that, in turn, calling ##DRW_shgroup_call_ex (shgroup=0x0, ob=0x61b0003185b8, obmat=0x0, geom=0x6110007d8138, bypass_culling=true, user_data=0x0)## with a null ##shgroup## pointer.

Debug notes. I won't revisit this until after the 25th, and, with a head full of grog, will need notes to recover context. Here is how ##OVERLAY_PrivateData *pd = data->stl->pd## in [OVERLAY_cache_populate() ](https://developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$180) winds up 'exhibiting interesting characteristics', as illustrated in the previous comment. It seems that ##OVERLAY_cache_populate()## is unaware of various drawing mode contexts within which callers operate, and how callers can set up ##OVERLAY_PrivateData *pd## in different ways. That's the bug in a nutshell. Gory details follow. On the mouse click over "Mutt" ##OVERLAY_cache_populate()## executes: ``` - 0 0x000055557a3b59b0 in OVERLAY_cache_populate (vedata=0x6170002b3838, ob=0x61b0002f86b8) at /home/gosgood/git_repositories/blender/source/blender/draw/engines/overlay/overlay_engine.c:189 - 1 0x000055557a21d3b9 in drw_engines_cache_populate (ob=0x61b0002f86b8) at /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager.c:1145 - 2 0x000055557a227b2c in DRW_draw_select_loop (depsgraph=0x6130003ab8f8, ar=0x6150001c5038, v3d=0x61a0001c08b8, UNUSED_use_obedit_skip=false, draw_surface=false, UNUSED_use_nearest=false, rect=0x7fffffff2830, select_pass_fn=0x55557d1d7576 <drw_select_loop_pass>, select_pass_user_data=0x7fffffff28c0, object_filter_fn=0x0, object_filter_user_data=0x0) at /home/gosgood/git_repositories/blender/source/blender/draw/intern/draw_manager.c:2286 - 3 0x000055557d1daa1d in view3d_opengl_select (vc=0x7fffffff2fb0, buffer=0x7fffffff3040, bufsize=10000, input=0x7fffffff2d60, select_mode=VIEW3D_SELECT_PICK_ALL, select_filter=VIEW3D_SELECT_FILTER_OBJECT_MODE_LOCK) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_view.c:1149 - 4 0x000055557d18a0a7 in mixed_bones_object_selectbuffer (vc=0x7fffffff2fb0, buffer=0x7fffffff3040, mval=0x7fffffffce40, select_filter=VIEW3D_SELECT_FILTER_OBJECT_MODE_LOCK, do_nearest=false) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:1684 - 5 0x000055557d18af94 in mixed_bones_object_selectbuffer_extended (vc=0x7fffffff2fb0, buffer=0x7fffffff3040, mval=0x7fffffffce40, select_filter=VIEW3D_SELECT_FILTER_OBJECT_MODE_LOCK, use_cycle=true, enumerate=false, r_do_nearest=0x7fffffff2f40) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:1783 - 6 0x000055557d18e171 in ed_object_select_pick (C=0x610000000c78, mval=0x7fffffffce40, extend=false, deselect=false, toggle=false, obcenter=false, enumerate=false, object=false) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:2061 - 7 0x000055557d191f5e in view3d_select_exec (C=0x610000000c78, op=0x611000809d78) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:2421 #8 0x000055557d192252 in view3d_select_invoke (C=0x610000000c78, op=0x611000809d78, event=0x6100008e2c78) at /home/gosgood/git_repositories/blender/source/blender/editors/space_view3d/view3d_select.c:2460 (remainder omitted for brevity) ``` ##OVERLAY_cache_populate()##, depends on a prior invocation of [OVERLAY_edit_mesh_cache_init() ](https:*developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_edit_mesh.c$55), which, among other things, populates the block of ##DRWShadingGroup## references in ##OVERLAY_PrivateData *pd = data->stl->pd## (the ##edit_mesh_*_grp## and friends), and this prior invocation*almost always// takes place -- but not here, where **Exclude from View Layer** has been toggled. ##OVERLAY_edit_mesh_cache_init()##, upon which ##OVERLAY_cache_populate()## depends, is called upstream at [drw_engines_cache_init() ]], which iterates over a list of 'engine' objects of ##DrawEngineType##; the variable ##engine## at line #1099 successively assumes references to all active engines during the course of iteration, including the overlay engine, defined at [[ https:*developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$466|overlay_engine.c:466 ]]. The member ##cache_init## contains a reference to an engine-specific cache initializer. ##engine## eventually assumes a reference to the overlay engine and its ##cache_init## member points to the overlay implementation [[ https://developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$99|OVERLAY_cache_init(void *vedata) ](https:*developer.blender.org/diffusion/B/browse/master/source/blender/draw/intern/draw_manager.c$1099). This function conditionally calls ##OVERLAY_edit_mesh_cache_init()## predicated on the draw mode state ##pd->ctx_mode == CTX_MODE_EDIT_MESH##. However, with the click on "Mutt", the viewport is not in edit mode; it is in object mode, a condition arising from toggling **Exclude from View Layer.** See the screen shots in the previous comment. The draw mode ##pd->ctx_mode == CTX_MODE_OBJECT## prevails instead so ##OVERLAY_edit_mesh_cache_init()## does not execute and leaves ##OVERLAY_PrivateData *pd## in ##OVERLAY_cache_populate()## without the block of ##DRWShadingGroup## references set up. Having established that ##Mutt## is to be drawn as an object, ##OVERLAY_cache_init()## completes the rest of the initialization of what will be ##OVERLAY_PrivateData pd## in ##OVERLAY_cache_populate()## and returns to ##drw_engines_cache_init()##, which, in due course, returns to its caller ##DRW_draw_select_loop().## With cache initialization complete, ##DRW_draw_select_loop()## invokes [drw_engines_cache_populate(Object *ob) ]] , which performs a similar iteration over engine objects seen in ##drw_engines_cache_init()##, but invoking the engines' ##engine->cache_populate(data, ob)## implementations instead. For the 'Overlay' engine, that implementation is [[ https:*developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/overlay/overlay_engine.c$180|OVERLAY_cache_populate(void *vedata, Object *ob) ](https:*developer.blender.org/diffusion/B/browse/master/source/blender/draw/intern/draw_manager.c$1120), which extracts private data ##OVERLAY_PrivateData *pd## from parameter ##ViewportEngineData vedata## (recast in the function call to ##OVERLAY_data##). As noted, ##pd## lacks the benefit of ##OVERLAY_edit_mesh_cache_init()##, as ##OVERLAY_cache_init()## calls that function only in a draw mode of ##CTX_MODE_EDIT_MESH##, not the currently prevailing one: ##CTX_MODE_OBJECT##. Put another way, ##OVERLAY_cache_populate()## lives in draw mode context ##CTX_MODE_EDIT_MESH##, in that it assumes its prerequisite, ##OVERLAY_edit_mesh_cache_init(),## always runs, but, alas, it missed the memo announcing it must also populate in ##CTX_MODE_OBJECT## too. Having missed the memo, it proceeds as if ##DRWShadingGroup## references live in ##pd## when they do not. To sustain the illusion, ## BKE_object_is_in_editmode(ob)## returns true, assigning that to ##in_edit_mode## and setting the stage for calling ##OVERLAY_edit_mesh_cache_populate(vedata, ob)## and that, in turn, calling ##DRW_shgroup_call_ex (shgroup=0x0, ob=0x61b0003185b8, obmat=0x0, geom=0x6110007d8138, bypass_culling=true, user_data=0x0)## with a null ##shgroup## pointer.

blender-admin commented

2020-01-14 17:59:32 +01:00

This issue was referenced by e2724abc22

This issue was referenced by e2724abc22d53f3a1f1552a6f40af3d47f7512e4

Clément Foucault commented

2020-01-14 18:00:07 +01:00

Changed status from 'Needs Triage' to: 'Resolved'

Clément Foucault closed this issue

2020-01-14 18:00:07 +01:00

Clément Foucault self-assigned this 2020-01-14 18:00:07 +01:00

Clément Foucault commented

2020-01-14 18:07:33 +01:00

@grosgood Thanks a lot for all the detailed informations and even then patch! I've commited an equivalent patch which is just a change of codestyle. The bug was caused by a false assumption so I think it's safe to treat is as solved. I just hope no other areas of blender make the same assumption.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Collections: Exclude From View toggle causes segment violation. #72490

Exact steps for others to reproduce the error

Backtrace

GDB Notes