This is a patch forwarded to address T72490. The specific trigger of this bug is a corner case somewhat removed from daily workflow. The user has to:
- Create a mesh, surface, curve, text or lattice object (a mesh object named "Mutt" in the example colselect.blend file serves).
- Put "Mutt" in a collection branched off the root Scene Collection. The example uses Mutt_Collection.
- Put "Mutt" in edit mode. Perform any number of edits on "Mutt", including none at all.
- Hide Mutt_Collection using the Exclude from View Layer checkbox. This is key: the user does NOT place the object, "Mutt," in any other mode. When being hidden via the collection, "Mutt" is still in edit mode. What is also key is the implicit "tab key action" which arises from hiding the collection. The viewport switches from edit mode to object mode, presumably because the object being edited is now hidden and nothing visible has been selected for editing. There is now something of a mismatch between the object, "Mutt", still in edit mode, and the viewport in object mode. "Mutt" is hidden via the collection it is in, however, so the mismatch is not obvious.
- Do any number of other things, including nothing at all.
- Unhide Mutt Collection. ""Mutt"" reappears, but not with the appearance of an object under edit. However, should one switch to the Data API view and look up "Mutt", it is marked as still being in edit mode.
- Click on "Mutt", perhaps to resume work on it.
- Blender crashes, as described in T72490. See also the included video clip.
Functions DRW_draw_select_loop() or DRW_draw_render_loop_ex() orchestrate the execution of viewport refresh in response to mouse clicks, where one or the other inventories active viewport drawing engines, initializes them and ensures that caches and buffers are set up. They do not embody direct knowledge on how to do these things, but iterate over the active viewport engines and call function references on the engines themselves to perform actions given the exigencies of particular drawing engines.
Of concern here are the cache initialization and populate phases of the overlay engine, and the implementations provided by the overlay drawing engine, which is responsible for rendering various visual cues for the user: the grid plane, outlining of selected objects and the like. It provides implementations OVERLAY_cache_init() and OVERLAY_cache_populate() for the cache initialization and populate phases.
These two implementations coordinate their behavior in part through transient state, OVERLAY_PrivateData. Of interest is the enumeration field, eContextObjectMode ctx_mode. This drawing context mode enumeration encodes 'view operation' (object, edit...) on an object type (mesh, surface...). Thus drawing context mode CTX_MODE_EDIT_SURFACE reflects an edit mode context on a surface object. CTX_data_mode_enum_ex() determines this mode during the course of OVERLAY_engine_init().
Function OVERLAY_cache_init() uses the eContextObjectMode enumeration ctx_mode to choose an object-type-draw-context-operation specific initializer. Depending on these, OVERLAY_cache_init() chooses, say OVERLAY_edit_mesh_cache_init() for mesh objects being edited. Among other actions, these initializers populate OVERLAY_PrivateData for use by follow-on functions such as OVERLAY_edit_mesh_cache_populate(). There is potential coupling between function pairs such as OVERLAY_*_cache_init() and OVERLAY_edit_*_cache_populate() in that the populator can behave badly if data sets like OVERLAY_PrivateData have not been set up by the appropriate initializer, the root cause of this particular bug. In the example colselect.blend file, containing "Mutt", a mesh object, OVERLAY_cache_populate() chooses OVERLAY_edit_mesh_cache_populate() as if the prevailing drawing context mode was that "Mutt" was being edited (CTX_MODE_EDIT_MESH). But that is not the conclusion that OVERLAY_cache_init() draws. Finding a eContextObjectMode of CTX_MODE_OBJECT instead, it does not call OVERLAY_edit_mesh_cache_init() to initialize OVERLAY_PrivateData in the manner that OVERLAY_edit_mesh_cache_populate() expects, setting up the immediate circumstances of the crash. T72490 reviews these gory details.
The motivation of this patch is to harness eContextObjectMode ctx_mode to ensure that OVERLAY_cache_populate() chooses a cache populator appropriate for the prevailing drawing context mode. Absent this patch, OVERLAY_cache_populate() depends on BKE_object_is_in_editmode(ob) to set a flag (in_edit_mode). In common cases, the drawing context mode aligns with that of the object, but the peculiar action of Exclude from View Layer checkbox on a collection leaves the internal state of member objects undisturbed. BKE_object_is_in_editmode(ob) finds that "Mutt" has a mesh and is content to report to its caller that "Mutt" is in edit mode, and OVERLAY_cache_populate() is equally content to take that to mean that CTX_MODE_EDIT_MESH is the prevailing drawing mode context, without explicitly referencing ctx_mode. Pity.
This patch is an immediate and local approach to the bug. One could say it is a defensive measure against the lazy management of objects when collections are hidden. The more appropriate approach to this bug may lie in the direction of having the behavior of Exclude from View Layer to be more transparent to the rest of the application - given the viewport mode change that this toggle engenders, when a collection is toggled back into visibility, the contained objects should be mode-set appropriately.