Page MenuHome

Crash in sculpt mode at trying to rotate after a cancelled rotate
Closed, ResolvedPublicBUG

Description

System Information
Operating system: Darwin-18.7.0-x86_64-i386-64bit 64 Bits
Graphics card: Intel(R) HD Graphics 6000 Intel Inc. 4.1 INTEL-12.10.12

Blender Version
Broken: version: 2.83 (sub 5), branch: master, commit date: 2020-02-25 17:16, hash: rB9cc5af64c65e

Exact steps for others to reproduce the error

  • Go to "sculpting" mode on the default cube,
  • Hit R, move the mouse but don't click anywhere yet. Instead hit esc.
  • Hit R again, and crash.
blender-build/build/bin/Debug/Blender.app/Contents/Resources/2.83/datafiles/icons/brush.sculpt.clay_thumb.dat Unable to load from file
0   Blender                             0x000000010b990684 BLI_system_backtrace + 52
1   Blender                             0x0000000100483e08 BKE_undosys_step_push_init_with_type + 168
2   Blender                             0x00000001011971db sculpt_undo_push_begin + 59
3   Blender                             0x000000010115ffe4 ED_sculpt_init_transform + 148
4   Blender                             0x00000001011a5671 initTransform + 1873
5   Blender                             0x0000000101228ddd transformops_data + 221
6   Blender                             0x00000001012287b5 transform_invoke + 37
7   Blender                             0x00000001004bc324 wm_operator_invoke + 708
8   Blender                             0x00000001004befd5 wm_handler_operator_call + 1525
9   Blender                             0x00000001004be171 wm_handlers_do_keymap_with_keymap_handler + 417
10  Blender                             0x00000001004bd1ef wm_handlers_do_intern + 463
11  Blender                             0x00000001004b6048 wm_handlers_do + 40
12  Blender                             0x00000001004b59a9 wm_event_do_handlers + 2985
13  Blender                             0x00000001004abe37 WM_main + 39
14  Blender                             0x0000000100140b9b main + 1243
15  libdyld.dylib                       0x00007fff60bce3d5 start + 1
BLI_assert failed: source/blender/blenkernel/intern/undo_system.c:457, BKE_undosys_step_push_init_with_type(), at 'ustack->step_init == ((void*)0)'
(UndoStep *) step_init = 0x00006000032157a8
next	UndoStep *	NULL	0x0000000000000000
prev	UndoStep *	NULL	0x0000000000000000
name	unsigned char [64]	"Transform"	
type	const UndoType *	0x60000260e4c8	0x000060000260e4c8
data_size	size_t	0
skip	bool	false
use_memfile_step	bool	false
is_applied	bool	false

Event Timeline

Ankit (ankitm) updated the task description. (Show Details)
Robert Guetzkow (rjg) changed the task status from Needs Triage to Confirmed.Feb 25 2020, 7:23 PM
Robert Guetzkow (rjg) triaged this task as High priority.Feb 25 2020, 7:32 PM
Ankit (ankitm) added a comment.EditedFeb 25 2020, 7:41 PM
  • umm but the state I posted variables in shows some memory address: 0x00006000032157a8 , not NULL. I did printf("%d", ustack->step_init == NULL);, it gives 1 and 0 and then crash.
  • I also posted the trace, though it doesn't have function line numbers. Should I put the one like yours next time ^_^'?
This comment was removed by Robert Guetzkow (rjg).

@Ankit (ankitm) Sorry, I misread the assert statement. The assert was triggered because it wasn't NULL. The state that you've reported is the one that causes the crash. I removed my previous comments in order to avoid any confusion.

I think I can add some info. For starters, this happens when any transform is applied, not just rotate. The other operations (translate, scale, etc.) on the left toolbar (where all the sculpting tools are) all the way down also cause this problem. You can manually click and drag to eg. translate, then without releasing drag, hit escape, and then drag again (or any other operation) for the assertion to fail. Now, for the diagnosis, I believe the culprit is in this callstack, whose sequence is only called when the rotate/translate/scale commands are cancelled with escape:

recalcData_sculpt(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1164)
recalcData(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1188)
restoreTransObjects(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1976)
transformEnd(bContext * C, TransInfo * t) (blender/source/blender/editors/transform/transform.c:2266)
transform_modal(bContext * C, wmOperator * op, const wmEvent * event) (blender/source/blender/editors/transform/transform_ops.c:434)

recalcData is called when the sculpting operations are done, but when they are cancelled, only the rotate etc. operations go through the stack above. The other sculpt options go through one like this:

sculpt_undo_push_end() (blender/source/blender/editors/sculpt_paint/sculpt_undo.c:1126)
sculpt_stroke_done(const bContext * C, struct PaintStroke * UNUSED_stroke) (blender/source/blender/editors/sculpt_paint/sculpt.c:8243)
stroke_done(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:992)
paint_stroke_cancel(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:1552)
sculpt_brush_stroke_cancel(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/sculpt.c:8330)
paint_stroke_modal(bContext * C, wmOperator * op, const wmEvent * event) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:1424)

The last one uses sculpt_undo_push_end, which I think is the key to the issue. Probably need a call to ED_sculpt_undo_geometry_end(Object *ob) somewhere, but I don't know how/where to put it. There is also ED_sculpt_init_transform(C) that might be a solution.

Ok, it looks like there is a matching call to ED_sculpt_end_transform(C); at line 1843 in blender/source/blender/editors/transform/transform.c (saveTrasform), but it does not reset the step_init because wm->op_undo_depth == 0 has to be true (line 1143 blender/source/blender/editors/sculpt_paint/sculpt_undo.c in sculpt_undo_push_end(void)) but in this case it is equal to 1. Forcefully setting it to 0 does not work (segfault later in code). Looks like it is set 0 after the call in line 2004 in blender/source/blender/windowmanager/intern/wm_event_system.c

I have a hack that seems to work, maybe someone who knows more can make it better. File blender/source/blender/editors/transform/transform.c in saveTransform lines 1842-1851:

if ((t->options & CTX_SCULPT) && !(t->options & CTX_PAINT_CURVE)) {
  wmWindowManager *wm = CTX_wm_manager(C);
  if (t->state == TRANS_CANCEL) {
    wm->op_undo_depth--;
  }
  ED_sculpt_end_transform(C);
  if (t->state == TRANS_CANCEL) {
    wm->op_undo_depth++;
  }
}

The core issue is that wm->op_undo_depth is supposed to be 0 here, but it isn't, so I just force it. The reason it is not zero is because of line 1988 of blender/source/blender/windowmanager/intern/wm_event_system.c (wm_handler_operator_call) which is:

if (ot->flag & OPTYPE_UNDO) {
        wm->op_undo_depth++;
}

For whatever reason, canceling while doing a transform has OPTYPE_UNDO. This (I think) is the true cause of the issue.

you could take some inspiration from the fact that the (hit R, move mouse ,hit escape, hit R again) <- this doesn't cause crash in layout mode. examine the trace maybe ?

also submitting a patch increases chances of "better implementation" comments. (:

Haha, that is true, I just wanted to get at least a slightly less hacky solution before I submitted anything. Also, I'm not familiar with layout mode, but the way I found this solution is by first looking at what happens in Object mode, and then realizing that sculpt mode does undos in a completely different way. Object mode cancels never touch stack_init, whereas sculpt does for all operations. There are also many other differences and also comments that say sculpt mode does undos differently. Essentially sculpt mode does all its ops differently (which is fine because it handles it own stuff) but for rotates etc. it borrows the other ones, which all set OPTYPE_UNDO, which it explicitly doesn't:

/* Flags (sculpt does own undo? (ton)). */
ot->flag = OPTYPE_BLOCKING;

line 8355 blender/source/blender/editors/sculpt_paint/sculpt.c SCULPT_OT_brush_stroke
Adding OPTYPE_UNDO and removing condition sculpt_undo_push_end(void) is another solution, but then you have to hit Ctrl+Z a few times before it actually undoes anything :/
Should I bother submitting either of these?

Ahh I'll submit my initial solution, alternative is just too glitchy having to hit Ctrl+z multiple times to undo. Revision here: https://developer.blender.org/D7018

Jacques Lucke (JacquesLucke) changed the subtype of this task from "Report" to "Bug".Tue, Mar 17, 3:32 PM