Ah thanks for the report, actually discovered that yesterday, and have been trying to understand what exactly is going on with it (part of the issue also comes from rB1cb07530a956, but reverting this does not fully fix it either).

Note that another symptom of the same issue is that from startup file, if you switch to Sculpt mode, undo will stay in sculpt mode (instead of going back to Object mode).

This is actually fairly involved to fix... Summoning some more brains here.

Root of the issue: Since initial memfile undo step is the only one existing, none of its IDs are flagged as 'changed', so when undoing we just fully re-use current IDs, hence the lack of undoing of the first sculpt action. Other modes like Paint ones are not affected because they do store a memfile undo step on entering, but for sculpt this was changed to a sculpt undo step in rB1cb07530a956.

Only quick solution I can see is to revert rB4c7b1766a7f1, but this will break most of the work done for 'preserving tool settings across undos' (since it relies on re-using existing data-blocks). So I'm not sure which issue would be considered the more critical right now?

For the more long term solution, we can try to be smart, and e.g 'remove' modes that are not supported by memfile undo when reading one of those steps. We already do it for Edit mode, this snippets adds Sculpt to the party. It solves part of the issue, but not all (especially when already starting in Sculpt mode, we still have a memfile undo step at the beginning in sculpt mode, so this is not working here).

We can also make all those 'editing' modes to actually go into last memfile undo step, find the memchunks for the data-block(s) they will be editing, and set their is_identical_future to false. I think this is the 'proper' solution, but not sure how involved it will be to implement it.

In T82388#1047720, @Philipp Oeser (lichtwerk) wrote:

renamed this task from Sculpt mode: Unexpected undo behavior to Sculpt mode: Unexpected undo behavior in Sculpting Template.

Actually, even not using the sculpting template, you can still replicate this erratic behavior in sculpt mode, if you load a texture/perform a brush stroke and try to undo... the undo won't work, but the Redo will undo the brush stroke...

True ^^

@Bastien Montagne (mont29), I don't really understand how the bug report and your description relate. Why would undoing the first stroke be affected by memfile undo or mode switching undo at all? Why doesn't it just undo the stroke with sculpt mode undo and nothing else?

I can imagine that if switching sculpt mode bypasses memfile undo for performance, some additional code would be needed to ensure the sculpt session is initialized properly. But I guess the steps to reproduce that issue are different than what is in the bug report description.

When a file is opened in sculpt mode as in the sculpting template, I would not expect there to be an undo step to go from sculpt to object mode that the user can actually do.

rB1cb07530a956: Fix T71564: Undo stroke lags after entering sculpt mode is also strange. It calls SCULPT_undo_push_begin, but that is supposed to go along with a call to SCULPT_undo_push_end once the operator is done. I don't know what it means to call just begin, maybe that indirectly does something useful, but it's not how this is supposed to be used.

In T82388#1048177, @Brecht Van Lommel (brecht) wrote:

@Bastien Montagne (mont29), I don't really understand how the bug report and your description relate. Why would undoing the first stroke be affected by memfile undo or mode switching undo at all? Why doesn't it just undo the stroke with sculpt mode undo and nothing else?

I can imagine that if switching sculpt mode bypasses memfile undo for performance, some additional code would be needed to ensure the sculpt session is initialized properly. But I guess the steps to reproduce that issue are different than what is in the bug report description.

When a file is opened in sculpt mode as in the sculpting template, I would not expect there to be an undo step to go from sculpt to object mode that the user can actually do.

Those both have the same cause, symptoms are different because when doing new -> sculpting, the initial memfile undo step stored is already in sculpt mode, so we do not have an initial, unchanged sculpt mode undo step, unlike when starting default blender and then switching to sculpt mode.

Here is the undo stack after opening default blender and switching to sculpt and doing one stroke:

Undo 3 Steps (*: active, #=applied, M=memfile-active, S=skip)
[*#M ]   0 {0x60c000319c08} type='Global Undo', name='Original'
[    ]   1 {0x60d000636218} type='Sculpt', name='Sculpt Mode'
[    ]   2 {0x60d00066e588} type='Sculpt', name='Draw Brush'

And here is the undo stack after doing new -> sculpting and doing one stroke:

Undo 2 Steps (*: active, #=applied, M=memfile-active, S=skip)
[*#M ]   0 {0x60c000468288} type='Global Undo', name='Original'
[    ]   1 {0x60d0002c71b8} type='Sculpt', name='Draw Brush'

rB1cb07530a956: Fix T71564: Undo stroke lags after entering sculpt mode is also strange. It calls SCULPT_undo_push_begin, but that is supposed to go along with a call to SCULPT_undo_push_end once the operator is done. I don't know what it means to call just begin, maybe that indirectly does something useful, but it's not how this is supposed to be used.

Yes, it is strange, and I'm not sure I understand it properly either, but think that that SCULPT_undo_push_begin simply ensures that the undo step stored at the end of this sculpt_mode_toggle_exec operator (by generic code in WM area) is actually a SCULPT undo step, and not a global, memfile one?

In T82388#1048179, @Bastien Montagne (mont29) wrote:

Those both have the same cause, symptoms are different because when doing new -> sculpting, the initial memfile undo step stored is already in sculpt mode, so we do not have an initial, unchanged sculpt mode undo step, unlike when starting default blender and then switching to sculpt mode.

Right, but I'm not sure why such an initial, unchanged sculpt mode undo step would be needed. As far as I know the 'Draw Brush' undo step should contain the old vertex coordinates to be restored, and that should be sufficient to perform undo.

In T82388#1048240, @Brecht Van Lommel (brecht) wrote:

In T82388#1048179, @Bastien Montagne (mont29) wrote:

Those both have the same cause, symptoms are different because when doing new -> sculpting, the initial memfile undo step stored is already in sculpt mode, so we do not have an initial, unchanged sculpt mode undo step, unlike when starting default blender and then switching to sculpt mode.

Right, but I'm not sure why such an initial, unchanged sculpt mode undo step would be needed. As far as I know the 'Draw Brush' undo step should contain the old vertex coordinates to be restored, and that should be sufficient to perform undo.

That would be the case, if the mesh ID's memchunks were properly tagged with is_identical_future being set to True.

But currently, that flag is only set when writing the next memfile undo step, and a difference is detected. So we'd need the first Sculpt undo step to do that as well I think.

It seems we are talking past each other. I don't know why you are talking about memchunks when my point was that for this bug report, no memfile or memchunks should be relevant. If they are relevant, please explain why.

If there is some other sequence of steps that you are doing a bugfix for, please explain what those steps are.

As already said, memfile/memchunks are the issue, since in both cases they are the first initial undo step, the ones that fail to apply properly:

Those both have the same cause, symptoms are different because when doing new -> sculpting, the initial memfile undo step stored is already in sculpt mode, so we do not have an initial, unchanged sculpt mode undo step, unlike when starting default blender and then switching to sculpt mode.

Here is the undo stack after opening default blender and switching to sculpt and doing one stroke:

Undo 3 Steps (*: active, #=applied, M=memfile-active, S=skip)
[*#M ]   0 {0x60c000319c08} type='Global Undo', name='Original'
[    ]   1 {0x60d000636218} type='Sculpt', name='Sculpt Mode'
[    ]   2 {0x60d00066e588} type='Sculpt', name='Draw Brush'

And here is the undo stack after doing new -> sculpting and doing one stroke:

Undo 2 Steps (*: active, #=applied, M=memfile-active, S=skip)
[*#M ]   0 {0x60c000468288} type='Global Undo', name='Original'
[    ]   1 {0x60d0002c71b8} type='Sculpt', name='Draw Brush'

Anyway, will be at the office tomorrow, easier to sort this in person (and also hope to have a working proper fix by then to review)

Hi @Bastien Montagne (mont29) not sure where to post this, but I just found another situation where this undo issue is still happening , this time involving the multires.... But it's easy to reproduce....

• File/New/Sculpting
• Add a multires modifier
• Subdivide once
• Perform a brush stroke
• Try to undo (nothing happens)
• Now try to redo (it will undo the brush stroke)

(blender-2.92.0-3e325c35f870-windows64)