Non-deterministic geometry statistics when baking (race condition?) #48061

New Issue

Steffen Ohrendorf · 2016-04-05T19:02:36+02:00

Steffen Ohrendorf commented

2016-04-05 19:02:36 +02:00

System Information
Windows 10 x64, 2x Radeon R9 270X

Blender Version
Broken: 2.77

Short description of error
When baking the supplied .blend, the geometry counts (vertices, faces, edges) jump around. From looking at the values, I think the counts vary from three duplicated objects, which are grouped into different groups for selective smoke simulation, but where the domains and the smoke emitters share the same mesh (not object).

So the setup is: 3 Objects are in 3 distinct groups, but share 1 mesh; the objects are children of a base object, and have different animations.
The problem is: Geometry counts jump between a few hard values, from ~7k vertices (which seems to be the basic scene geometry) and ~80k vertices (which is the more than the whole scene should contain).

Additionally, the 3 emitters sometimes do not emit smoke, or have "gaps" when emitting; also, when using left/right arrow keys to step through the animations, the objects' positions are occassionally not synchronized, meaning an object is lagging behind.

I have a crashdump (Windows) lying around and can submit it when needed.

Other CPU- and RAM-heavy applications run fine, and memtest86 didn't report any errors.

Exact steps for others to reproduce the error
Open the .blend, select a domain, and hit "Update to current frame" in the physics tab. Occassionally blender also crashes when doing this.

2nd_reality.blend

**System Information** Windows 10 x64, 2x Radeon R9 270X **Blender Version** Broken: 2.77 **Short description of error** When baking the supplied .blend, the geometry counts (vertices, faces, edges) jump around. From looking at the values, I think the counts vary from three duplicated objects, which are grouped into different groups for selective smoke simulation, but where the domains and the smoke emitters share the same mesh (not object). So the setup is: 3 Objects are in 3 distinct groups, but share 1 mesh; the objects are children of a base object, and have different animations. The problem is: Geometry counts jump between a few hard values, from ~7k vertices (which seems to be the basic scene geometry) and ~80k vertices (which is the more than the whole scene should contain). Additionally, the 3 emitters sometimes do not emit smoke, or have "gaps" when emitting; also, when using left/right arrow keys to step through the animations, the objects' positions are occassionally not synchronized, meaning an object is lagging behind. I have a crashdump (Windows) lying around and can submit it when needed. Other CPU- and RAM-heavy applications run fine, and memtest86 didn't report any errors. **Exact steps for others to reproduce the error** Open the .blend, select a domain, and hit "Update to current frame" in the physics tab. Occassionally blender also crashes when doing this. [2nd_reality.blend](https://archive.blender.org/developer/F300911/2nd_reality.blend)

Steffen Ohrendorf commented

2016-04-05 19:02:36 +02:00

Changed status to: 'Open'

Steffen Ohrendorf commented

2016-04-05 19:02:36 +02:00

Added subscriber: @stohrendorf

jack commented

2016-04-06 01:57:23 +02:00

Added subscriber: @burton

Sergey Sharybin commented

2016-04-12 11:27:36 +02:00

Added subscriber: @Sergey

Sergey Sharybin commented

2016-04-12 11:27:36 +02:00

Jumping frames is not really a bug. Baking could have some run-time created geometry for intermediate calculation. You don't see it in the viewport because interface is actually all locked.

Crash shouldn't happen tho, but it i can not reproduce here. Please test with 2.77a and latest builds from builder.blender.org, there were some fixes in related areas.

If you'll still have crash, please try isolating it and make more obvious to happen.

Jumping frames is not really a bug. Baking could have some run-time created geometry for intermediate calculation. You don't see it in the viewport because interface is actually all locked. Crash shouldn't happen tho, but it i can not reproduce here. Please test with 2.77a and latest builds from builder.blender.org, there were some fixes in related areas. If you'll still have crash, please try isolating it and make more obvious to happen.

Steffen Ohrendorf commented

2016-04-14 08:26:34 +02:00

First results with 2.77a, now relatively reproducable with the above blend:

Open the blend, select the right domain, click "free all bakes" and "update all to frame", I get the following result:

Only one domain is showing smoke/flames, but all three should. This isn't a display error, as it's also not rendered.

Again, click "free all bakes" and "update all to frame", immediate crash follows in most cases. In seldom cases, another domain partially shows smoke flames (looks like it's starting to simulate somewhere in the midst of the specified frame range), or the progress bar at the top immediately starts at 20%, not at 0%.

BTW, I'm running the RenderSheep client in parallel, which also renders using 2.77a, and there are no problems as far as I can tell.

I will continue testing nightly builds, and also do some debugging to narrow the problem down.

First results with 2.77a, now relatively reproducable with the above blend: 1) Open the blend, select the right domain, click "free all bakes" and "update all to frame", I get the following result: ![update_1.jpg](https://archive.blender.org/developer/F303315/update_1.jpg) Only one domain is showing smoke/flames, but all three should. This isn't a display error, as it's also not rendered. 2) Again, click "free all bakes" and "update all to frame", immediate crash follows in most cases. In seldom cases, another domain *partially* shows smoke flames (looks like it's starting to simulate somewhere in the midst of the specified frame range), or the progress bar at the top immediately starts at 20%, not at 0%. BTW, I'm running the RenderSheep client in parallel, which also renders using 2.77a, and there are no problems as far as I can tell. I will continue testing nightly builds, and also do some debugging to narrow the problem down.

Bastien Montagne commented

2016-04-22 19:53:51 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2016-04-22 19:53:51 +02:00

Actually, here with latest master and asan debug build got a crash nearly immediately on first baking of right domain, here is TB (note that I also tried with -t 1 mono-threaded option, got same result but from non-main thread too…):

read blend: /home/i74700deb64/Téléchargements/2nd_reality.blend
loadTile: No noise tile '/tmp/blender_0L86nM/noise.wavelets' found.
Generating new 3d noise tile size=128^3 
saveTile: Noise tile file '/tmp/blender_0L86nM/noise.wavelets' saved.
Generating new 3d noise done
Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded.
Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded.
Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded.
Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded.
=================================================================
==7197==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000188bb4 at pc 0x000005149aaf bp 0x7f40f2a6d650 sp 0x7f40f2a6d648
READ of size 4 at 0x602000188bb4 thread T32
    - 0 0x5149aae in FLUID_3D::copyBorderX(float*, BasicVector::Vector3Dim<int>, int, int) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/FLUID_3D_STATIC.cpp:241
    - 1 0x5159338 in WTURBULENCE::advectTextureCoordinates(float, float*, float*, float*, float*, float*) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/WTURBULENCE.cpp:354
    - 2 0x515ed5d in WTURBULENCE::stepTurbulenceFull(float, float*, float*, float*, unsigned char*) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/WTURBULENCE.cpp:848
    - 3 0x512385a in smoke_turbulence_step /home/i74700deb64/blender/__work__/src/intern/smoke/intern/smoke_API.cpp:95
    - 4 0x4313f35 in smokeModifier_process /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2777
    - 5 0x4314144 in smokeModifier_do /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2797
    - 6 0x3688a19 in applyModifier /home/i74700deb64/blender/__work__/src/source/blender/modifiers/intern/MOD_smoke.c:112
    - 7 0x41157eb in modwrap_applyModifier /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/modifier.c:744
    - 8 0x3dbab15 in mesh_calc_modifiers /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2035
    - 9 0x3dbf4d7 in mesh_build_data /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2575
    - 10 0x3dbff77 in makeDerivedMesh /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2672
    - 11 0x419e23b in BKE_object_handle_data_update /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object_update.c:201
    - 12 0x4184173 in BKE_object_handle_update_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object.c:3163
    - 13 0x428ed6d in scene_update_object_func /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1526
    - 14 0x4b724e4 in BLI_task_pool_work_and_wait /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/task.c:481
    - 15 0x428f6c8 in scene_update_objects /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1684
    - 16 0x428f918 in scene_update_tagged_recursive /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1726
    - 17 0x4290775 in BKE_scene_update_for_newframe_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1995
    - 18 0x4290482 in BKE_scene_update_for_newframe /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1898
    - 19 0x426bec7 in BKE_ptcache_bake /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/pointcache.c:3653
    - 20 0x2df5d3c in ptcache_job_startjob /home/i74700deb64/blender/__work__/src/source/blender/editors/physics/physics_pointcache.c:128
    - 21 0x213469e in do_job_thread /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:337
    - 22 0x4b73daa in tslot_thread_start /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/threads.c:254
    - 23 0x7f4133667453 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7453)
    #24 0x7f4129680ecc in clone (/lib/x86_64-linux-gnu/libc.so.6+0xe8ecc)

0x602000188bb4 is located 0 bytes to the right of 4-byte region [0x602000188bb0,0x602000188bb4)
allocated by thread T32 here:
    - 0 0x7f4135231d5a in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x94d5a)
    - 1 0x51564f5 in WTURBULENCE::WTURBULENCE(int, int, int, int, int, char const*, int, int) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/WTURBULENCE.cpp:112
    - 2 0x512304f in smoke_turbulence_init /home/i74700deb64/blender/__work__/src/intern/smoke/intern/smoke_API.cpp:49
    - 3 0x42eee06 in smoke_reallocate_highres_fluid /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:173
    - 4 0x4305e48 in adjustDomainResolution /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:1842
    - 5 0x430964c in update_flowsfluids /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2196
    - 6 0x4310caf in step /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2548
    - 7 0x4313e9b in smokeModifier_process /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2769
    - 8 0x4314144 in smokeModifier_do /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2797
    - 9 0x3688a19 in applyModifier /home/i74700deb64/blender/__work__/src/source/blender/modifiers/intern/MOD_smoke.c:112
    - 10 0x41157eb in modwrap_applyModifier /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/modifier.c:744
    - 11 0x3dbab15 in mesh_calc_modifiers /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2035
    - 12 0x3dbf4d7 in mesh_build_data /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2575
    - 13 0x3dbff77 in makeDerivedMesh /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2672
    - 14 0x419e23b in BKE_object_handle_data_update /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object_update.c:201
    - 15 0x4184173 in BKE_object_handle_update_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object.c:3163
    - 16 0x428ed6d in scene_update_object_func /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1526
    - 17 0x4b724e4 in BLI_task_pool_work_and_wait /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/task.c:481
    - 18 0x428f6c8 in scene_update_objects /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1684
    - 19 0x428f918 in scene_update_tagged_recursive /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1726
    - 20 0x4290775 in BKE_scene_update_for_newframe_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1995
    - 21 0x4290482 in BKE_scene_update_for_newframe /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1898
    - 22 0x426bec7 in BKE_ptcache_bake /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/pointcache.c:3653
    - 23 0x2df5d3c in ptcache_job_startjob /home/i74700deb64/blender/__work__/src/source/blender/editors/physics/physics_pointcache.c:128
    - 24 0x213469e in do_job_thread /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:337
    - 25 0x4b73daa in tslot_thread_start /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/threads.c:254
    #26 0x7f4133667453 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7453)

Thread T32 created by T0 here:
    - 0 0x7f41351d2f04 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x35f04)
    - 1 0x4b73eae in BLI_insert_thread /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/threads.c:270
    - 2 0x2134df3 in WM_jobs_start /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:422
    - 3 0x2df693b in ptcache_bake_invoke /home/i74700deb64/blender/__work__/src/source/blender/editors/physics/physics_pointcache.c:223
    - 4 0x211705f in wm_operator_invoke /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:1095
    - 5 0x2118085 in wm_operator_call_internal /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:1290
    - 6 0x21182e6 in WM_operator_name_call_ptr /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:1338
    - 7 0x291a98e in ui_apply_but_funcs_after /home/i74700deb64/blender/__work__/src/source/blender/editors/interface/interface_handlers.c:768
    - 8 0x2960367 in ui_handler_region_menu /home/i74700deb64/blender/__work__/src/source/blender/editors/interface/interface_handlers.c:9944
    - 9 0x2112ec2 in wm_handler_ui_call /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:453
    - 10 0x211c238 in wm_handlers_do_intern /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:2053
    - 11 0x211c822 in wm_handlers_do /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:2137
    - 12 0x211e0d0 in wm_event_do_handlers /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:2420
    - 13 0x2100ff2 in WM_main /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm.c:489
    - 14 0x20f7636 in main /home/i74700deb64/blender/__work__/src/source/creator/creator.c:479
    - 15 0x7f41295b860f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2060f)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/i74700deb64/blender/__work__/src/intern/smoke/intern/FLUID_3D_STATIC.cpp:241 FLUID_3D::copyBorderX(float*, BasicVector::Vector3Dim<int>, int, int)
Shadow bytes around the buggy address:
  0x0c0480029120: fa fa fd fd fa fa fa fa fa fa fa fa fa fa fd fd
  0x0c0480029130: fa fa fd fa fa fa fa fa fa fa fd fd fa fa fa fa
  0x0c0480029140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0480029150: fa fa fa fa fa fa fa fa fa fa 04 fa fa fa fa fa
  0x0c0480029160: fa fa fd fd fa fa fa fa fa fa fd fd fa fa fa fa
=>0x0c0480029170: fa fa fd fd fa fa[04]fa fa fa fa fa fa fa fd fd
  0x0c0480029180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0480029190: fa fa fd fd fa fa fa fa fa fa fd fd fa fa fa fa
  0x0c04800291a0: fa fa fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c04800291b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c04800291c0: fa fa fa fa fa fa fd fd fa fa 00 00 fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==7197==ABORTING

Actually, here with latest master and asan debug build got a crash nearly immediately on first baking of right domain, here is TB (note that I also tried with `-t 1` mono-threaded option, got same result but from non-main thread too…): ``` read blend: /home/i74700deb64/Téléchargements/2nd_reality.blend loadTile: No noise tile '/tmp/blender_0L86nM/noise.wavelets' found. Generating new 3d noise tile size=128^3 saveTile: Noise tile file '/tmp/blender_0L86nM/noise.wavelets' saved. Generating new 3d noise done Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded. Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded. Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded. Noise tile file '/tmp/blender_0L86nM/noise.wavelets' loaded. ================================================================= ==7197==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000188bb4 at pc 0x000005149aaf bp 0x7f40f2a6d650 sp 0x7f40f2a6d648 READ of size 4 at 0x602000188bb4 thread T32 - 0 0x5149aae in FLUID_3D::copyBorderX(float*, BasicVector::Vector3Dim<int>, int, int) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/FLUID_3D_STATIC.cpp:241 - 1 0x5159338 in WTURBULENCE::advectTextureCoordinates(float, float*, float*, float*, float*, float*) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/WTURBULENCE.cpp:354 - 2 0x515ed5d in WTURBULENCE::stepTurbulenceFull(float, float*, float*, float*, unsigned char*) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/WTURBULENCE.cpp:848 - 3 0x512385a in smoke_turbulence_step /home/i74700deb64/blender/__work__/src/intern/smoke/intern/smoke_API.cpp:95 - 4 0x4313f35 in smokeModifier_process /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2777 - 5 0x4314144 in smokeModifier_do /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2797 - 6 0x3688a19 in applyModifier /home/i74700deb64/blender/__work__/src/source/blender/modifiers/intern/MOD_smoke.c:112 - 7 0x41157eb in modwrap_applyModifier /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/modifier.c:744 - 8 0x3dbab15 in mesh_calc_modifiers /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2035 - 9 0x3dbf4d7 in mesh_build_data /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2575 - 10 0x3dbff77 in makeDerivedMesh /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2672 - 11 0x419e23b in BKE_object_handle_data_update /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object_update.c:201 - 12 0x4184173 in BKE_object_handle_update_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object.c:3163 - 13 0x428ed6d in scene_update_object_func /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1526 - 14 0x4b724e4 in BLI_task_pool_work_and_wait /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/task.c:481 - 15 0x428f6c8 in scene_update_objects /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1684 - 16 0x428f918 in scene_update_tagged_recursive /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1726 - 17 0x4290775 in BKE_scene_update_for_newframe_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1995 - 18 0x4290482 in BKE_scene_update_for_newframe /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1898 - 19 0x426bec7 in BKE_ptcache_bake /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/pointcache.c:3653 - 20 0x2df5d3c in ptcache_job_startjob /home/i74700deb64/blender/__work__/src/source/blender/editors/physics/physics_pointcache.c:128 - 21 0x213469e in do_job_thread /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:337 - 22 0x4b73daa in tslot_thread_start /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/threads.c:254 - 23 0x7f4133667453 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7453) #24 0x7f4129680ecc in clone (/lib/x86_64-linux-gnu/libc.so.6+0xe8ecc) 0x602000188bb4 is located 0 bytes to the right of 4-byte region [0x602000188bb0,0x602000188bb4) allocated by thread T32 here: - 0 0x7f4135231d5a in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x94d5a) - 1 0x51564f5 in WTURBULENCE::WTURBULENCE(int, int, int, int, int, char const*, int, int) /home/i74700deb64/blender/__work__/src/intern/smoke/intern/WTURBULENCE.cpp:112 - 2 0x512304f in smoke_turbulence_init /home/i74700deb64/blender/__work__/src/intern/smoke/intern/smoke_API.cpp:49 - 3 0x42eee06 in smoke_reallocate_highres_fluid /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:173 - 4 0x4305e48 in adjustDomainResolution /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:1842 - 5 0x430964c in update_flowsfluids /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2196 - 6 0x4310caf in step /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2548 - 7 0x4313e9b in smokeModifier_process /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2769 - 8 0x4314144 in smokeModifier_do /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/smoke.c:2797 - 9 0x3688a19 in applyModifier /home/i74700deb64/blender/__work__/src/source/blender/modifiers/intern/MOD_smoke.c:112 - 10 0x41157eb in modwrap_applyModifier /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/modifier.c:744 - 11 0x3dbab15 in mesh_calc_modifiers /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2035 - 12 0x3dbf4d7 in mesh_build_data /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2575 - 13 0x3dbff77 in makeDerivedMesh /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/DerivedMesh.c:2672 - 14 0x419e23b in BKE_object_handle_data_update /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object_update.c:201 - 15 0x4184173 in BKE_object_handle_update_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/object.c:3163 - 16 0x428ed6d in scene_update_object_func /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1526 - 17 0x4b724e4 in BLI_task_pool_work_and_wait /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/task.c:481 - 18 0x428f6c8 in scene_update_objects /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1684 - 19 0x428f918 in scene_update_tagged_recursive /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1726 - 20 0x4290775 in BKE_scene_update_for_newframe_ex /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1995 - 21 0x4290482 in BKE_scene_update_for_newframe /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/scene.c:1898 - 22 0x426bec7 in BKE_ptcache_bake /home/i74700deb64/blender/__work__/src/source/blender/blenkernel/intern/pointcache.c:3653 - 23 0x2df5d3c in ptcache_job_startjob /home/i74700deb64/blender/__work__/src/source/blender/editors/physics/physics_pointcache.c:128 - 24 0x213469e in do_job_thread /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:337 - 25 0x4b73daa in tslot_thread_start /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/threads.c:254 #26 0x7f4133667453 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7453) Thread T32 created by T0 here: - 0 0x7f41351d2f04 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x35f04) - 1 0x4b73eae in BLI_insert_thread /home/i74700deb64/blender/__work__/src/source/blender/blenlib/intern/threads.c:270 - 2 0x2134df3 in WM_jobs_start /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_jobs.c:422 - 3 0x2df693b in ptcache_bake_invoke /home/i74700deb64/blender/__work__/src/source/blender/editors/physics/physics_pointcache.c:223 - 4 0x211705f in wm_operator_invoke /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:1095 - 5 0x2118085 in wm_operator_call_internal /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:1290 - 6 0x21182e6 in WM_operator_name_call_ptr /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:1338 - 7 0x291a98e in ui_apply_but_funcs_after /home/i74700deb64/blender/__work__/src/source/blender/editors/interface/interface_handlers.c:768 - 8 0x2960367 in ui_handler_region_menu /home/i74700deb64/blender/__work__/src/source/blender/editors/interface/interface_handlers.c:9944 - 9 0x2112ec2 in wm_handler_ui_call /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:453 - 10 0x211c238 in wm_handlers_do_intern /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:2053 - 11 0x211c822 in wm_handlers_do /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:2137 - 12 0x211e0d0 in wm_event_do_handlers /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm_event_system.c:2420 - 13 0x2100ff2 in WM_main /home/i74700deb64/blender/__work__/src/source/blender/windowmanager/intern/wm.c:489 - 14 0x20f7636 in main /home/i74700deb64/blender/__work__/src/source/creator/creator.c:479 - 15 0x7f41295b860f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2060f) SUMMARY: AddressSanitizer: heap-buffer-overflow /home/i74700deb64/blender/__work__/src/intern/smoke/intern/FLUID_3D_STATIC.cpp:241 FLUID_3D::copyBorderX(float*, BasicVector::Vector3Dim<int>, int, int) Shadow bytes around the buggy address: 0x0c0480029120: fa fa fd fd fa fa fa fa fa fa fa fa fa fa fd fd 0x0c0480029130: fa fa fd fa fa fa fa fa fa fa fd fd fa fa fa fa 0x0c0480029140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0480029150: fa fa fa fa fa fa fa fa fa fa 04 fa fa fa fa fa 0x0c0480029160: fa fa fd fd fa fa fa fa fa fa fd fd fa fa fa fa =>0x0c0480029170: fa fa fd fd fa fa[04]fa fa fa fa fa fa fa fd fd 0x0c0480029180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0480029190: fa fa fd fd fa fa fa fa fa fa fd fd fa fa fa fa 0x0c04800291a0: fa fa fd fd fa fa fa fa fa fa fa fa fa fa fa fa 0x0c04800291b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c04800291c0: fa fa fa fa fa fa fd fd fa fa 00 00 fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe ==7197==ABORTING ```

Steffen Ohrendorf commented

2016-04-25 08:16:01 +02:00

From a quick glance at the code, I'd say that in FLUID_3D_STATIC.cpp:241 the array values are overwritten with uninitialized data unless the array has been allocated with a size that contains the (probably virtual) borders.

From a quick glance at the code, I'd say that in `FLUID_3D_STATIC.cpp:241` the array values are overwritten with uninitialized data unless the array has been allocated with a size that contains the (probably virtual) borders.

Steffen Ohrendorf commented

2016-04-30 05:37:24 +02:00

In this specific case, WTURBULENCE::_resSm is (1,1,1), so the float array passed to FLUID_3D::copyBorderX has a size of 1; res is here (1,1,1), zBegin is 0 and zEnd is 1. This leads to the first index value of index = y * res- [x] + z * slabSize = 0 * 1 + 0 * 1 = 0, which results in a out-of-bounds access at field[index] = field[index + 1]. Putting a few ifs around the field accesses resolved the crashes, but there are still some some troubles with multiple emitters.

Here, again only one domain shows some smoke, and it also starts within the midst of the simulation:
And in this blend (icetunnel.blend), the right sphere's smoke should be much more like the left one's (which is cloned from the right one, without changes to the smoke physics):

In this specific case, `WTURBULENCE::_resSm` is (1,1,1), so the float array passed to `FLUID_3D::copyBorderX` has a size of 1; `res` is here (1,1,1), `zBegin` is 0 and `zEnd` is 1. This leads to the first index value of `index = y * res- [x] + z * slabSize = 0 * 1 + 0 * 1 = 0`, which results in a out-of-bounds access at `field[index] = field[index + 1]`. Putting a few `if`s around the field accesses resolved the crashes, but there are still some some troubles with multiple emitters. 1. Here, again only one domain shows some smoke, and it also starts within the midst of the simulation: ![blender_partial_smoke.png](https://archive.blender.org/developer/F308154/blender_partial_smoke.png) 2. And in this blend ([icetunnel.blend](https://archive.blender.org/developer/F308156/icetunnel.blend)), the right sphere's smoke should be much more like the left one's (which is cloned from the right one, without changes to the smoke physics): ![blender_smoke_inconsistency.png](https://archive.blender.org/developer/F308157/blender_smoke_inconsistency.png)

Steffen Ohrendorf commented

2016-05-01 03:21:27 +02:00

After doing a larger refactoring on the smoke code (results here, containing the whole blender/intern/smoke folder: smoke.7z), the crashes disappeared.

The reason for the strange smoke behaviour above seems to be that the influence maps aren't properly used, leading to the fact that the smoke's temperature isn't applied to the cells' heat values. In fact, there are non-zero influence values calculated, but in apply_inflow_fields, the emission_value parameter is always zero for the right sphere. Deleting the left sphere didn't change anything.

Also (which is also strange), there's no noise applied to the smoke emitted from the right sphere, and the low-resolution smoke isn't visible at all.

I double-checked the emitter smoke settings, and they are all the same for both spheres.

After doing a larger refactoring on the smoke code (results here, containing the whole `blender/intern/smoke` folder: [smoke.7z](https://archive.blender.org/developer/F308260/smoke.7z)), the crashes disappeared. The reason for the strange smoke behaviour above seems to be that the influence maps aren't properly used, leading to the fact that the smoke's temperature isn't applied to the cells' heat values. In fact, there *are* non-zero influence values calculated, but in `apply_inflow_fields`, the `emission_value` parameter is always zero for the right sphere. Deleting the left sphere didn't change anything. Also (which is also strange), there's no noise applied to the smoke emitted from the right sphere, and the low-resolution smoke isn't visible at all. I double-checked the emitter smoke settings, and they are all the same for both spheres.

Steffen Ohrendorf commented

2016-05-03 19:17:37 +02:00

I think I finally found the problem: Smoke is emitted to both the low-res and the high-res maps, but because it only gets successfully emitted in the high-res map and not in the low-res maps due to the low maximum surface distance, it doesn't get processed further, except for dissolving effects. Increasing the domain's resolution or the emitters' maximum surface distance solves problem.

I think I finally found the problem: Smoke is emitted to both the low-res and the high-res maps, but because it only gets *successfully* emitted in the high-res map and not in the low-res maps due to the low maximum surface distance, it doesn't get processed further, except for dissolving effects. Increasing the domain's resolution or the emitters' maximum surface distance solves problem.

Steffen Ohrendorf commented

2016-05-18 08:06:16 +02:00

Although I presume that the whole smoke simulation code will be revamped with the current GSOC project aiming the consolidation of fluid and smoke simulation, or at least for 2.8, I'd really like to see a fix for the out-of-bounds accesses in the next release. Still, there's the problem I mentioned in my previous comment, which also looks a bit like a memory corruption, because when I use the patched code with the 2nd_reality blend I initially posted, emission of smoke is still non-deterministic, and totally changes behaviour when selecting different domains to do a full scene bake (although that could simply be the case because I messed up the code in some other place).

Although I presume that the whole smoke simulation code will be revamped with the current GSOC project aiming the consolidation of fluid and smoke simulation, or at least for 2.8, I'd really like to see a fix for the out-of-bounds accesses in the next release. Still, there's the problem I mentioned in my previous comment, which also looks a bit like a memory corruption, because when I use the patched code with the `2nd_reality` blend I initially posted, emission of smoke is still non-deterministic, and totally changes behaviour when selecting different domains to do a full scene bake (although *that* could simply be the case because I messed up the code in some other place).

Alexander Gavrilov commented

2016-07-23 22:05:30 +02:00

Added subscriber: @angavrilov

Alexander Gavrilov commented

2016-07-23 22:05:30 +02:00

The reason it crashes seems to be that somebody simply forgot to add a check for an empty smoke domain for the highres step. Doing the same check as can be found in the lowres case fixes it:

--- a/source/blender/blenkernel/intern/smoke.c
+++ b/source/blender/blenkernel/intern/smoke.c
@@ -2772,7 +2772,7 @@ static void smokeModifier_process(SmokeModifierData *smd, Scene *scene, Object *
                // create shadows before writing cache so they get stored
                smoke_calc_transparency(sds, scene);
 
-               if (sds->wt)
+               if (sds->wt && sds->total_cells > 1)
                {
                        smoke_turbulence_step(sds->wt, sds->fluid);
                }

Now, the more interesting question is why there is so little fire and smoke that the adaptive domain collapses to nothing. It turns out that it may be a bad idea to use the vertex group option to separate emitting and totally non-emitting faces, especially if the low res domain cell size is bigger or comparable to the size of the emitting area on the object. This is because the emitting code first finds the closest face to the center of the domain cell, and only then checks the vertex group. If that face turns out to be non-emitting - tough luck, even if there are emitting faces within range.

If you separate the emitting faces of those ships into a separate object, suddenly they look like they are on fire and about to explode.

The reason it crashes seems to be that somebody simply forgot to add a check for an empty smoke domain for the highres step. Doing the same check as can be found in the lowres case fixes it: ``` --- a/source/blender/blenkernel/intern/smoke.c +++ b/source/blender/blenkernel/intern/smoke.c @@ -2772,7 +2772,7 @@ static void smokeModifier_process(SmokeModifierData *smd, Scene *scene, Object * // create shadows before writing cache so they get stored smoke_calc_transparency(sds, scene); - if (sds->wt) + if (sds->wt && sds->total_cells > 1) { smoke_turbulence_step(sds->wt, sds->fluid); } ``` Now, the more interesting question is why there is so little fire and smoke that the adaptive domain collapses to nothing. It turns out that it may be a bad idea to use the vertex group option to separate emitting and totally non-emitting faces, especially if the low res domain cell size is bigger or comparable to the size of the emitting area on the object. This is because the emitting code first finds the closest face to the center of the domain cell, and only then checks the vertex group. If that face turns out to be non-emitting - tough luck, even if there are emitting faces within range. If you separate the emitting faces of those ships into a separate object, suddenly they look like they are on fire and about to explode.

blender-admin commented

2016-07-27 15:40:32 +02:00

This issue was referenced by d7bd64df5d

This issue was referenced by d7bd64df5d88ed135c57e99ab3c2240943e4c01f

Sergey Sharybin commented

2016-07-27 15:43:12 +02:00

Changed status from 'Open' to: 'Resolved'

Sergey Sharybin closed this issue

2016-07-27 15:43:12 +02:00

Steffen Ohrendorf commented

2016-08-25 21:25:16 +02:00

Many, many thanks for resolving this!

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Non-deterministic geometry statistics when baking (race condition?) #48061