Crash in sculpt mode at trying to rotate after a cancelled rotate #74205

New Issue

Ankit Meel · 2020-02-25T19:19:47+01:00

Ankit Meel commented

2020-02-25 19:19:47 +01:00

System Information
Operating system: Darwin-18.7.0-x86_64-i386-64bit 64 Bits
Graphics card: Intel(R) HD Graphics 6000 Intel Inc. 4.1 INTEL-12.10.12

Blender Version
Broken: version: 2.83 (sub 5), branch: master, commit date: 2020-02-25 17:16, hash: 9cc5af64c6

Exact steps for others to reproduce the error

Go to "sculpting" mode on the default cube,
Hit R, move the mouse but don't click anywhere yet. Instead hit esc.
Hit R again, and crash.

blender-build/build/bin/Debug/Blender.app/Contents/Resources/2.83/datafiles/icons/brush.sculpt.clay_thumb.dat Unable to load from file
0   Blender                             0x000000010b990684 BLI_system_backtrace + 52
1   Blender                             0x0000000100483e08 BKE_undosys_step_push_init_with_type + 168
2   Blender                             0x00000001011971db sculpt_undo_push_begin + 59
3   Blender                             0x000000010115ffe4 ED_sculpt_init_transform + 148
4   Blender                             0x00000001011a5671 initTransform + 1873
5   Blender                             0x0000000101228ddd transformops_data + 221
6   Blender                             0x00000001012287b5 transform_invoke + 37
7   Blender                             0x00000001004bc324 wm_operator_invoke + 708
8   Blender                             0x00000001004befd5 wm_handler_operator_call + 1525
9   Blender                             0x00000001004be171 wm_handlers_do_keymap_with_keymap_handler + 417
10  Blender                             0x00000001004bd1ef wm_handlers_do_intern + 463
11  Blender                             0x00000001004b6048 wm_handlers_do + 40
12  Blender                             0x00000001004b59a9 wm_event_do_handlers + 2985
13  Blender                             0x00000001004abe37 WM_main + 39
14  Blender                             0x0000000100140b9b main + 1243
15  libdyld.dylib                       0x00007fff60bce3d5 start + 1
BLI_assert failed: source/blender/blenkernel/intern/undo_system.c:457, BKE_undosys_step_push_init_with_type(), at 'ustack->step_init == ((void*)0)'

(UndoStep *) step_init = 0x00006000032157a8
next	UndoStep *	NULL	0x0000000000000000
prev	UndoStep *	NULL	0x0000000000000000
name	unsigned char [64]	"Transform"	
type	const UndoType *	0x60000260e4c8	0x000060000260e4c8
data_size	size_t	0
skip	bool	false
use_memfile_step	bool	false
is_applied	bool	false

**System Information** Operating system: Darwin-18.7.0-x86_64-i386-64bit 64 Bits Graphics card: Intel(R) HD Graphics 6000 Intel Inc. 4.1 INTEL-12.10.12 **Blender Version** Broken: version: 2.83 (sub 5), branch: master, commit date: 2020-02-25 17:16, hash: `9cc5af64c6` **Exact steps for others to reproduce the error** - Go to "sculpting" mode on the default cube, - Hit R, move the mouse but don't click anywhere yet. Instead hit esc. - Hit R again, and crash. ``` blender-build/build/bin/Debug/Blender.app/Contents/Resources/2.83/datafiles/icons/brush.sculpt.clay_thumb.dat Unable to load from file 0 Blender 0x000000010b990684 BLI_system_backtrace + 52 1 Blender 0x0000000100483e08 BKE_undosys_step_push_init_with_type + 168 2 Blender 0x00000001011971db sculpt_undo_push_begin + 59 3 Blender 0x000000010115ffe4 ED_sculpt_init_transform + 148 4 Blender 0x00000001011a5671 initTransform + 1873 5 Blender 0x0000000101228ddd transformops_data + 221 6 Blender 0x00000001012287b5 transform_invoke + 37 7 Blender 0x00000001004bc324 wm_operator_invoke + 708 8 Blender 0x00000001004befd5 wm_handler_operator_call + 1525 9 Blender 0x00000001004be171 wm_handlers_do_keymap_with_keymap_handler + 417 10 Blender 0x00000001004bd1ef wm_handlers_do_intern + 463 11 Blender 0x00000001004b6048 wm_handlers_do + 40 12 Blender 0x00000001004b59a9 wm_event_do_handlers + 2985 13 Blender 0x00000001004abe37 WM_main + 39 14 Blender 0x0000000100140b9b main + 1243 15 libdyld.dylib 0x00007fff60bce3d5 start + 1 BLI_assert failed: source/blender/blenkernel/intern/undo_system.c:457, BKE_undosys_step_push_init_with_type(), at 'ustack->step_init == ((void*)0)' ``` ``` (UndoStep *) step_init = 0x00006000032157a8 next UndoStep * NULL 0x0000000000000000 prev UndoStep * NULL 0x0000000000000000 name unsigned char [64] "Transform" type const UndoType * 0x60000260e4c8 0x000060000260e4c8 data_size size_t 0 skip bool false use_memfile_step bool false is_applied bool false ```

Ankit Meel commented

2020-02-25 19:19:47 +01:00

Added subscriber: @ankitm

Robert Guetzkow commented

2020-02-25 19:23:49 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Robert Guetzkow commented

2020-02-25 19:26:29 +01:00

Added subscriber: @rjg

Robert Guetzkow commented

2020-02-25 19:26:29 +01:00

This comment was removed by @rjg

*This comment was removed by @rjg*

Brecht Van Lommel commented

2020-02-25 19:33:32 +01:00

Added subscriber: @PabloDobarro

Ankit Meel commented

2020-02-25 19:41:36 +01:00

umm but the state I posted variables in shows some memory address: 0x00006000032157a8 , not NULL. I did printf("%d", ustack->step_init == NULL);, it gives 1 and 0 and then crash.
I also posted the trace, though it doesn't have function line numbers. Should I put the one like yours next time ^_^'?

- umm but the state I posted variables in shows some memory address: 0x00006000032157a8 , not NULL. I did `printf("%d", ustack->step_init == NULL);`, it gives `1` and `0` and then crash. - I also posted the trace, though it doesn't have function line numbers. Should I put the one like yours next time `^_^'`?

Robert Guetzkow commented

2020-02-25 21:15:09 +01:00

This comment was removed by @rjg

*This comment was removed by @rjg*

Robert Guetzkow commented

2020-02-25 21:28:22 +01:00

@ankitm Sorry, I misread the assert statement. The assert was triggered because it wasn't NULL. The state that you've reported is the one that causes the crash. I removed my previous comments in order to avoid any confusion.

@ankitm Sorry, I misread the assert statement. The assert was triggered because it *wasn't* `NULL`. The state that you've reported is the one that causes the crash. I removed my previous comments in order to avoid any confusion.

Asad-ullah Khan commented

2020-03-02 11:57:13 +01:00

Added subscriber: @Asad-ullahKhan

Asad-ullah Khan commented

2020-03-02 11:57:13 +01:00

I think I can add some info. For starters, this happens when any transform is applied, not just rotate. The other operations (translate, scale, etc.) on the left toolbar (where all the sculpting tools are) all the way down also cause this problem. You can manually click and drag to eg. translate, then without releasing drag, hit escape, and then drag again (or any other operation) for the assertion to fail. Now, for the diagnosis, I believe the culprit is in this callstack, whose sequence is only called when the rotate/translate/scale commands are cancelled with escape:

recalcData_sculpt(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1164)
recalcData(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1188)
restoreTransObjects(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1976)
transformEnd(bContext * C, TransInfo * t) (blender/source/blender/editors/transform/transform.c:2266)
transform_modal(bContext * C, wmOperator * op, const wmEvent * event) (blender/source/blender/editors/transform/transform_ops.c:434)

recalcData is called when the sculpting operations are done, but when they are cancelled, only the rotate etc. operations go through the stack above. The other sculpt options go through one like this:

sculpt_undo_push_end() (blender/source/blender/editors/sculpt_paint/sculpt_undo.c:1126)
sculpt_stroke_done(const bContext * C, struct PaintStroke * UNUSED_stroke) (blender/source/blender/editors/sculpt_paint/sculpt.c:8243)
stroke_done(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:992)
paint_stroke_cancel(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:1552)
sculpt_brush_stroke_cancel(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/sculpt.c:8330)
paint_stroke_modal(bContext * C, wmOperator * op, const wmEvent * event) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:1424)

The last one uses sculpt_undo_push_end, which I think is the key to the issue. Probably need a call to ED_sculpt_undo_geometry_end(Object *ob) somewhere, but I don't know how/where to put it. There is also ED_sculpt_init_transform(C) that might be a solution.

I think I can add some info. For starters, this happens when any transform is applied, not just rotate. The other operations (translate, scale, etc.) on the left toolbar (where all the sculpting tools are) all the way down also cause this problem. You can manually click and drag to eg. translate, then without releasing drag, hit escape, and then drag again (or any other operation) for the assertion to fail. Now, for the diagnosis, I believe the culprit is in this callstack, whose sequence is only called when the rotate/translate/scale commands are cancelled with escape: ``` recalcData_sculpt(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1164) recalcData(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1188) restoreTransObjects(TransInfo * t) (blender/source/blender/editors/transform/transform_generics.c:1976) transformEnd(bContext * C, TransInfo * t) (blender/source/blender/editors/transform/transform.c:2266) transform_modal(bContext * C, wmOperator * op, const wmEvent * event) (blender/source/blender/editors/transform/transform_ops.c:434) ``` `recalcData` is called when the sculpting operations are done, but when they are cancelled, only the rotate etc. operations go through the stack above. The other sculpt options go through one like this: ``` sculpt_undo_push_end() (blender/source/blender/editors/sculpt_paint/sculpt_undo.c:1126) sculpt_stroke_done(const bContext * C, struct PaintStroke * UNUSED_stroke) (blender/source/blender/editors/sculpt_paint/sculpt.c:8243) stroke_done(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:992) paint_stroke_cancel(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:1552) sculpt_brush_stroke_cancel(bContext * C, wmOperator * op) (blender/source/blender/editors/sculpt_paint/sculpt.c:8330) paint_stroke_modal(bContext * C, wmOperator * op, const wmEvent * event) (blender/source/blender/editors/sculpt_paint/paint_stroke.c:1424) ``` The last one uses `sculpt_undo_push_end`, which I think is the key to the issue. Probably need a call to `ED_sculpt_undo_geometry_end(Object *ob)` somewhere, but I don't know how/where to put it. There is also `ED_sculpt_init_transform(C)` that might be a solution.

Asad-ullah Khan commented

2020-03-02 21:53:15 +01:00

Ok, it looks like there is a matching call to ED_sculpt_end_transform(C); at line 1843 in blender/source/blender/editors/transform/transform.c (saveTrasform), but it does not reset the step_init because wm->op_undo_depth == 0 has to be true (line 1143 blender/source/blender/editors/sculpt_paint/sculpt_undo.c in sculpt_undo_push_end(void)) but in this case it is equal to 1. Forcefully setting it to 0 does not work (segfault later in code). Looks like it is set 0 after the call in line 2004 in blender/source/blender/windowmanager/intern/wm_event_system.c

Ok, it looks like there is a matching call to `ED_sculpt_end_transform(C);` at line 1843 in `blender/source/blender/editors/transform/transform.c` (`saveTrasform`), but it does not reset the `step_init` because `wm->op_undo_depth == 0` has to be true (line 1143 `blender/source/blender/editors/sculpt_paint/sculpt_undo.c` in `sculpt_undo_push_end(void)`) but in this case it is equal to 1. Forcefully setting it to 0 does not work (segfault later in code). Looks like it is set 0 after the call in line 2004 in `blender/source/blender/windowmanager/intern/wm_event_system.c`

Asad-ullah Khan commented

2020-03-03 09:05:55 +01:00

I have a hack that seems to work, maybe someone who knows more can make it better. File blender/source/blender/editors/transform/transform.c in saveTransform lines 1842-1851:

  if ((t->options & CTX_SCULPT) && !(t->options & CTX_PAINT_CURVE)) {
    wmWindowManager *wm = CTX_wm_manager(C);
    if (t->state == TRANS_CANCEL) {
      wm->op_undo_depth--;
    }
    ED_sculpt_end_transform(C);
    if (t->state == TRANS_CANCEL) {
      wm->op_undo_depth++;
    }
  }

The core issue is that wm->op_undo_depth is supposed to be 0 here, but it isn't, so I just force it. The reason it is not zero is because of line 1988 of blender/source/blender/windowmanager/intern/wm_event_system.c (wm_handler_operator_call) which is:

if (ot->flag & OPTYPE_UNDO) {
        wm->op_undo_depth++;
}

For whatever reason, canceling while doing a transform has OPTYPE_UNDO. This (I think) is the true cause of the issue.

I have a hack that seems to work, maybe someone who knows more can make it better. File `blender/source/blender/editors/transform/transform.c` in `saveTransform` lines 1842-1851: ``` if ((t->options & CTX_SCULPT) && !(t->options & CTX_PAINT_CURVE)) { wmWindowManager *wm = CTX_wm_manager(C); if (t->state == TRANS_CANCEL) { wm->op_undo_depth--; } ED_sculpt_end_transform(C); if (t->state == TRANS_CANCEL) { wm->op_undo_depth++; } } ``` The core issue is that `wm->op_undo_depth` is supposed to be 0 here, but it isn't, so I just force it. The reason it is not zero is because of line 1988 of `blender/source/blender/windowmanager/intern/wm_event_system.c` (`wm_handler_operator_call`) which is: ``` if (ot->flag & OPTYPE_UNDO) { wm->op_undo_depth++; } ``` For whatever reason, canceling while doing a transform has `OPTYPE_UNDO`. This (I think) is the true cause of the issue.

Ankit Meel commented

2020-03-03 11:35:59 +01:00

you could take some inspiration from the fact that the (hit R, move mouse ,hit escape, hit R again) <- this doesn't cause crash in layout mode. examine the trace maybe ?

Ankit Meel commented

2020-03-03 11:40:02 +01:00

also submitting a patch increases chances of "better implementation" comments. (:

Asad-ullah Khan commented

2020-03-04 05:05:41 +01:00

Haha, that is true, I just wanted to get at least a slightly less hacky solution before I submitted anything. Also, I'm not familiar with layout mode, but the way I found this solution is by first looking at what happens in Object mode, and then realizing that sculpt mode does undos in a completely different way. Object mode cancels never touch stack_init, whereas sculpt does for all operations. There are also many other differences and also comments that say sculpt mode does undos differently. Essentially sculpt mode does all its ops differently (which is fine because it handles it own stuff) but for rotates etc. it borrows the other ones, which all set OPTYPE_UNDO, which it explicitly doesn't:

/* Flags (sculpt does own undo? (ton)). */
ot->flag = OPTYPE_BLOCKING;

line 8355 blender/source/blender/editors/sculpt_paint/sculpt.c SCULPT_OT_brush_stroke
Adding OPTYPE_UNDO and removing condition sculpt_undo_push_end(void) is another solution, but then you have to hit Ctrl+Z a few times before it actually undoes anything :/
Should I bother submitting either of these?

Haha, that is true, I just wanted to get at least a slightly less hacky solution before I submitted anything. Also, I'm not familiar with layout mode, but the way I found this solution is by first looking at what happens in Object mode, and then realizing that sculpt mode does undos in a completely different way. Object mode cancels never touch `stack_init`, whereas sculpt does for all operations. There are also many other differences and also comments that say sculpt mode does undos differently. Essentially sculpt mode does all its ops differently (which is fine because it handles it own stuff) but for rotates etc. it borrows the other ones, which all set `OPTYPE_UNDO`, which it explicitly doesn't: ``` /* Flags (sculpt does own undo? (ton)). */ ot->flag = OPTYPE_BLOCKING; ``` line 8355 `blender/source/blender/editors/sculpt_paint/sculpt.c` `SCULPT_OT_brush_stroke` Adding `OPTYPE_UNDO` and removing condition `sculpt_undo_push_end(void)` is another solution, but then you have to hit Ctrl+Z a few times before it actually undoes anything :/ Should I bother submitting either of these?

Asad-ullah Khan commented

2020-03-04 06:22:23 +01:00

Ahh I'll submit my initial solution, alternative is just too glitchy having to hit Ctrl+z multiple times to undo. Revision here: https://developer.blender.org/D7018