Fix T88756: crash when baking with autosmooth

When baking some data, we create a new Mesh with edits and modifiers applied.
However, in some cases (e.g. when there is no modifier), the returned Mesh is
actually referencing the original one and its data layers. When autosmooth is
enabled we also split the Mesh. However, since the new Mesh is referencing the
original one, although `BKE_mesh_split_faces` is creating new vertices and edges,
the reallocation of the custom data layers is preempted because of the
reference, so adding the new vertices and edges overwrites valid data

To fix this we duplicate referenced layers before splitting the faces.

Reviewed By: brecht

Differential Revision: https://developer.blender.org/D11703
This commit is contained in:
Kévin Dietrich 2021-06-25 12:56:01 +02:00
parent 3558bb8eae
commit 5f9677fe0c
Notes: blender-bot 2023-02-13 18:31:54 +01:00
Referenced by commit 2dbb492268, Fix T89455: Cycles crash when rendering a Mesh with autosmooth
Referenced by issue #88449, Blender LTS: Maintenance Task 2.93
Referenced by issue #89842, Crash when baking vertex colors when Auto Smooth is enabled
Referenced by issue #89477, Crash and/or stall with some geometry
Referenced by issue #89455, Crash when using Cycles preview or render -  custom data layer / auto smooth
Referenced by issue #88756, Crash when baking to vertex colors while "Normals-> Auto Smooth" is enabled.
3 changed files with 19 additions and 0 deletions

View File

@ -233,6 +233,9 @@ void *CustomData_duplicate_referenced_layer_named(struct CustomData *data,
const int totelem);
bool CustomData_is_referenced_layer(struct CustomData *data, int type);
/* Duplicate all the layers with flag NOFREE, and remove the flag from duplicated layers. */
void CustomData_duplicate_referenced_layers(CustomData *data, int totelem);
/* set the CD_FLAG_NOCOPY flag in custom data layers where the mask is
* zero for the layer type, so only layer types specified by the mask
* will be copied

View File

@ -2813,6 +2813,14 @@ void *CustomData_duplicate_referenced_layer_named(CustomData *data,
return customData_duplicate_referenced_layer_index(data, layer_index, totelem);
}
void CustomData_duplicate_referenced_layers(CustomData *data, int totelem)
{
for (int i = 0; i < data->totlayer; i++) {
CustomDataLayer *layer = &data->layers[i];
layer->data = CustomData_duplicate_referenced_layer(data, layer->type, totelem);
}
}
bool CustomData_is_referenced_layer(struct CustomData *data, int type)
{
/* get the layer index of the first layer of type */

View File

@ -2087,6 +2087,14 @@ void BKE_mesh_split_faces(Mesh *mesh, bool free_loop_normals)
SplitFaceNewVert *new_verts = NULL;
SplitFaceNewEdge *new_edges = NULL;
/* Ensure we own the layers, we need to do this before split_faces_prepare_new_verts as it will
* directly assign new indices to existing edges and loops. */
CustomData_duplicate_referenced_layers(&mesh->vdata, mesh->totvert);
CustomData_duplicate_referenced_layers(&mesh->edata, mesh->totedge);
CustomData_duplicate_referenced_layers(&mesh->ldata, mesh->totloop);
/* Update pointers in case we duplicated referenced layers. */
BKE_mesh_update_customdata_pointers(mesh, false);
/* Detect loop normal spaces (a.k.a. smooth fans) that will need a new vert. */
const int num_new_verts = split_faces_prepare_new_verts(
mesh, &lnors_spacearr, &new_verts, memarena);