Page MenuHome

Fix assertion failure when converting BMesh to Mesh with shape keys
ClosedPublic

Authored by Sybren A. Stüvel (sybren) on Fri, May 22, 1:42 PM.

Details

Summary

The BM_mesh_bm_to_me() function copies shape keys from the BMesh to the Mesh. However, it tries to copy the same number of shape keys as are defined on the target mesh. Since the target mesh does not necessarily have the same number of shape keys as the BMesh, this would crash if the target Mesh has more.

Found while performing some tests for D7785: Fix T65148: Can't use shape key properties in driver when modifiers generate a new mesh.

To reproduce the issue:

  • Open the attached blend file in a debug build of Blender.
  • Run the code in the text editor.
  • Assertion failure:
blender: …/blender/source/blender/bmesh/intern/bmesh_mesh_conv.c:941: BM_mesh_bm_to_me: Assertion `cd_shape_offset != -1' failed.

Note that even with this fix, the remaining shapekeys on the mesh are invalid. It might be desirable to delete the shapekeys before or after the operation, but I feel that this is out of the scope of the BM_mesh_bm_to_me() function.

Diff Detail

Repository
rB Blender

Event Timeline

Sybren A. Stüvel (sybren) requested review of this revision.Fri, May 22, 1:42 PM
Sybren A. Stüvel (sybren) retitled this revision from Fix crash when converting BMesh to Mesh with shape keys to Fix assertion failure when converting BMesh to Mesh with shape keys.Fri, May 22, 1:45 PM
Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)
Brecht Van Lommel (brecht) requested changes to this revision.Fri, May 22, 2:24 PM
Brecht Van Lommel (brecht) added inline comments.
source/blender/bmesh/intern/bmesh_mesh_conv.c
898

Use continue instead.

This revision now requires changes to proceed.Fri, May 22, 2:24 PM
source/blender/bmesh/intern/bmesh_mesh_conv.c
898

Why? Is it possible for a mesh to not have CD_SHAPEKEY with index j but to do have one with index j+1? Otherwise I don't see anything in this for-loop that should be called for every CD_SHAPEKEY layer in the target mesh.

source/blender/bmesh/intern/bmesh_mesh_conv.c
898

j is computed by bm_to_mesh_shape_layer_index_from_kb. It's not obvious that it is monotonically increasing. Even if it is, relying on such assumptions is fragile.

Sybren A. Stüvel (sybren) marked 2 inline comments as done.
Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)
  • Feedback from Brecht
This revision is now accepted and ready to land.Fri, May 22, 6:31 PM