Page MenuHome

Crash related to custom data draw (Blender with ASAN)
Open, Confirmed, MediumPublic

Description

Exact steps for others to reproduce the error

Open

in Blender compiled with ASAN and see crash

Details

Type
Bug

Event Timeline

Bastien Montagne (mont29) triaged this task as Confirmed, Medium priority.Sep 9 2016, 10:50 AM

The root of the issue is that active render index for UVs is wrong. which should not be happening. So question is: how did you run into such situation? This is to be fixed..

But since such situation was possible, should we apply something like this?

1diff --git a/source/blender/blenkernel/intern/DerivedMesh.c b/source/blender/blenkernel/intern/DerivedMesh.c
2index 5f759c6..8168817 100644
3--- a/source/blender/blenkernel/intern/DerivedMesh.c
4+++ b/source/blender/blenkernel/intern/DerivedMesh.c
5@@ -3474,13 +3474,17 @@ void DM_calc_loop_tangents(
6
7 /* Update active layer index */
8 uv_index = CustomData_get_layer_index_n(&dm->loopData, CD_MLOOPUV, act_uv_n);
9- tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
10- CustomData_set_layer_active_index(&dm->loopData, CD_TANGENT, tan_index);
11+ if (uv_index != -1) {
12+ tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
13+ CustomData_set_layer_active_index(&dm->loopData, CD_TANGENT, tan_index);
14+ }
15
16 /* Update render layer index */
17 uv_index = CustomData_get_layer_index_n(&dm->loopData, CD_MLOOPUV, ren_uv_n);
18- tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
19- CustomData_set_layer_render_index(&dm->loopData, CD_TANGENT, tan_index);
20+ if (uv_index != -1) {
21+ tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
22+ CustomData_set_layer_render_index(&dm->loopData, CD_TANGENT, tan_index);
23+ }
24 }
25 }
26

Don't know how to reproduce this from scratch. I'm testing all files from Blend4Web SDK and the original file with this bug is from the end of 2014.
I've just started to investigate this bug.

But the good news is that this is the last crash that I met)

I think we should check the type of layer where we expect UV. Crash occurs because of broken logic there.

1diff --git a/source/blender/blenkernel/intern/DerivedMesh.c b/source/blender/blenkernel/intern/DerivedMesh.c
2index 5f759c6..ceb3ff0 100644
3--- a/source/blender/blenkernel/intern/DerivedMesh.c
4+++ b/source/blender/blenkernel/intern/DerivedMesh.c
5@@ -3306,14 +3306,25 @@ void DM_calc_loop_tangents_step_0(
6 *ract_uv_n = CustomData_get_active_layer(loopData, CD_MLOOPUV);
7 ract_uv_name[0] = 0;
8 if (*ract_uv_n != -1) {
9- strcpy(ract_uv_name, loopData->layers[*ract_uv_n + layer_index].name);
10+ int index = *ract_uv_n + layer_index;
11+ if (index < loopData->totlayer && loopData->layers[index].type == CD_MLOOPUV)
12+ strcpy(ract_uv_name, loopData->layers[index].name);
13+ else
14+ *ract_uv_n = -1;
15 }
16
17 /* Active tangent in render */
18 *rren_uv_n = CustomData_get_render_layer(loopData, CD_MLOOPUV);
19 rren_uv_name[0] = 0;
20 if (*rren_uv_n != -1) {
21- strcpy(rren_uv_name, loopData->layers[*rren_uv_n + layer_index].name);
22+ int index = *rren_uv_n + layer_index;
23+ /* It is not known why, but layer.active_rnd can have wrong value,
24+ * therefore we should check the layer type
25+ * The same is done above for layer.active see T49297 */
26+ if (index < loopData->totlayer && loopData->layers[index].type == CD_MLOOPUV)
27+ strcpy(rren_uv_name, loopData->layers[index].name);
28+ else
29+ *rren_uv_n = -1;
30 }
31
32 /* If active tangent not in tangent_names we take it into account */
33@@ -3473,14 +3484,18 @@ void DM_calc_loop_tangents(
34 int uv_index, tan_index;
35
36 /* Update active layer index */
37- uv_index = CustomData_get_layer_index_n(&dm->loopData, CD_MLOOPUV, act_uv_n);
38- tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
39- CustomData_set_layer_active_index(&dm->loopData, CD_TANGENT, tan_index);
40+ if (act_uv_n != -1) {
41+ uv_index = CustomData_get_layer_index_n(&dm->loopData, CD_MLOOPUV, act_uv_n);
42+ tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
43+ CustomData_set_layer_active_index(&dm->loopData, CD_TANGENT, tan_index);
44+ }
45
46 /* Update render layer index */
47- uv_index = CustomData_get_layer_index_n(&dm->loopData, CD_MLOOPUV, ren_uv_n);
48- tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
49- CustomData_set_layer_render_index(&dm->loopData, CD_TANGENT, tan_index);
50+ if (ren_uv_n != -1) {
51+ uv_index = CustomData_get_layer_index_n(&dm->loopData, CD_MLOOPUV, ren_uv_n);
52+ tan_index = CustomData_get_named_layer_index(&dm->loopData, CD_TANGENT, dm->loopData.layers[uv_index].name);
53+ CustomData_set_layer_render_index(&dm->loopData, CD_TANGENT, tan_index);
54+ }
55 }
56 }
57

Checking type earlier is same as checking whether CustomData_get_layer_index_n() gave proper index for a given layer type or do i miss something?

Checking type earlier is same as checking whether CustomData_get_layer_index_n() gave proper index for a given layer type or do i miss something?

Another attempt. Return 0 index because DM_calc_loop_tangents_step_0 will not be invoked without CD_MLOOPUV

1diff --git a/source/blender/blenkernel/intern/DerivedMesh.c b/source/blender/blenkernel/intern/DerivedMesh.c
2index 5f759c6..8c40d27 100644
3--- a/source/blender/blenkernel/intern/DerivedMesh.c
4+++ b/source/blender/blenkernel/intern/DerivedMesh.c
5@@ -3306,14 +3306,20 @@ void DM_calc_loop_tangents_step_0(
6 *ract_uv_n = CustomData_get_active_layer(loopData, CD_MLOOPUV);
7 ract_uv_name[0] = 0;
8 if (*ract_uv_n != -1) {
9- strcpy(ract_uv_name, loopData->layers[*ract_uv_n + layer_index].name);
10+ if (CustomData_get_layer_index_n(loopData, CD_MLOOPUV, *ract_uv_n) >= 0)
11+ strcpy(ract_uv_name, loopData->layers[*ract_uv_n + layer_index].name);
12+ else
13+ *ract_uv_n = 0;
14 }
15
16 /* Active tangent in render */
17 *rren_uv_n = CustomData_get_render_layer(loopData, CD_MLOOPUV);
18 rren_uv_name[0] = 0;
19 if (*rren_uv_n != -1) {
20- strcpy(rren_uv_name, loopData->layers[*rren_uv_n + layer_index].name);
21+ if (CustomData_get_layer_index_n(loopData, CD_MLOOPUV, *rren_uv_n) >= 0)
22+ strcpy(rren_uv_name, loopData->layers[*rren_uv_n + layer_index].name);
23+ else
24+ *rren_uv_n = 0;
25 }
26
27 /* If active tangent not in tangent_names we take it into account */