Page MenuHome

Crashing with animation rendering and sound
Closed, ResolvedPublic

Description

System Information
Operating system: MacOS 10.14
Graphics card: Radeon Pro 580 8192 MB

Blender Version
Broken: 2.80, 3600bb79d05d-x86_64, 2019-03-16

Short description of error
Blender exits while rendering this project as an animation (fails to render). Usually seems to render first few tens of frames, but exits before getting to frame 100.

Exact steps for others to reproduce the error
Open file. Edit animation output location if necessary. Render animation.

Details

Type
Bug

Event Timeline

Tomasz Kaye (bitbutter) closed this task as Invalid.Mar 16 2019, 4:01 PM
Antonio Vazquez (antoniov) triaged this task as Confirmed, Medium priority.

The crash is related to sound. Here the call stack:

>	blender.exe!std::_Tree<std::_Tset_traits<void *,std::less<void *>,std::allocator<void *>,0> >::_Orphan_ptr(std::_Tree_node<void *,void *> * _Ptr) Line 2097	C++
 	blender.exe!std::_Tree<std::_Tset_traits<void *,std::less<void *>,std::allocator<void *>,0> >::clear() Line 1410	C++
 	blender.exe!std::_Tree<std::_Tset_traits<void *,std::less<void *>,std::allocator<void *>,0> >::erase(std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<void *> > > _First, std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<void *> > > _Last) Line 1389	C++
 	blender.exe!std::_Tree<std::_Tset_traits<void *,std::less<void *>,std::allocator<void *>,0> >::_Tidy() Line 2113	C++
 	blender.exe!std::_Tree<std::_Tset_traits<void *,std::less<void *>,std::allocator<void *>,0> >::~_Tree<std::_Tset_traits<void *,std::less<void *>,std::allocator<void *>,0> >() Line 1168	C++
 	blender.exe!std::set<void *,std::less<void *>,std::allocator<void *> >::~set<void *,std::less<void *>,std::allocator<void *> >()	C++
 	blender.exe!std::set<void *,std::less<void *>,std::allocator<void *> >::`scalar deleting destructor'(unsigned int)	C++
 	blender.exe!AUD_destroySet(void * set) Line 36	C++
 	blender.exe!BKE_sound_update_scene(Main * bmain, Scene * scene) Line 912	C
 	blender.exe!BKE_scene_graph_update_for_newframe(Depsgraph * depsgraph, Main * bmain) Line 1562	C
 	blender.exe!engine_depsgraph_init(RenderEngine * engine, ViewLayer * view_layer) Line 497	C
 	blender.exe!RE_engine_render(Render * re, int do_all) Line 727	C
 	blender.exe!do_render_3d(Render * re) Line 1081	C
 	blender.exe!do_render(Render * re) Line 1160	C
 	blender.exe!do_render_composite(Render * re) Line 1366	C
 	blender.exe!do_render_all_options(Render * re) Line 1635	C
 	blender.exe!RE_BlenderAnim(Render * re, Main * bmain, Scene * scene, ViewLayer * single_layer, Object * camera_override, int sfra, int efra, int tfra) Line 2487	C
 	blender.exe!render_startjob(void * rjv, short * stop, short * do_update, float * progress) Line 612	C
 	blender.exe!do_job_thread(void * job_v) Line 331	C
 	blender.exe!tslot_thread_start(void * tslot_p) Line 252	C

I don't know who is the person to fix this.

Checking console log showed that it was a problem with a 3rd party add-on (snapscrubber). Removing the add on fixed the problem.

@Tomasz Kaye (bitbutter) I got an error with sound rendering your file. Are you sure we can close it?

@Antonio Vazquez (antoniov) will comment to confirm after this render is done, But on frame 1000 without problems so far.

Ah seems there's still a problem. Crashed again with this complaint:

Append frame 3764
 Time: 00:00.68 (Saving: 00:00.06)
blender(57290,0x11544c5c0) malloc: *** error for object 0x7fe864f1b870: pointer being freed was not allocated
blender(57290,0x11544c5c0) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
Brecht Van Lommel (brecht) raised the priority of this task from Confirmed, Medium to Confirmed, High.Mar 18 2019, 1:50 PM

I have not been able to reproduce this crash on Linux. There may be random memory corruption involved, or something with thread timing.

It's likely possible to further pin down the issue.

  • Render with an empty scene, or with Cycles. This may be unrelated to grease pencil.
  • Render to jpg file instead of ffmpeg video, as this may be audio related.
  • Disable audaspace in the build and render (for developers).

Can still confirm on windows, different trace than @Antonio Vazquez (antoniov) but still coming out of audaspace though, @Joerg Mueller (nexyon) mind taking a peek here?

>	blender.exe!std::_Tree<std::_Tset_traits<void *,std::less<void *>,std::allocator<void *>,0> >::begin() Line 1187	C++
 	blender.exe!AUD_getSet(void * set=0x000000018b292fe0) Line 58	C++
 	blender.exe!BKE_sound_update_scene(Main * bmain=0x000000008223b958, Scene * scene=0x00000000822716f8) Line 901	C
 	blender.exe!BKE_scene_graph_update_tagged(Depsgraph * depsgraph=0x0000000038e2cef8, Main * bmain=0x000000008223b958) Line 1512	C
 	blender.exe!wm_event_do_depsgraph(bContext * C=0x0000000013b8af78) Line 343	C
 	blender.exe!wm_event_do_refresh_wm_and_depsgraph(bContext * C=0x0000000013b8af78) Line 369	C
 	blender.exe!wm_event_do_notifiers(bContext * C=0x0000000013b8af78) Line 528	C
 	blender.exe!WM_main(bContext * C=0x0000000013b8af78) Line 424	C
 	blender.exe!main(int argc=1, const unsigned char * * UNUSED_argv_c=0x0000000008e5afa0) Line 507	C
 	[External Code]

I tried to reproduce the crash, but it also didn't happen on my linux. Given the stack traces though, it looks like the issue is BKE_sound_update_scene being called at the same time in different threads. As such a mutex in this function should solve the problem?

The problem seams to have been introduced in 48f9e24f0c1e Enable dependency graph update while rendering.

During rendering scene->speaker_handles gets manipulated from both the render thread and the window event thread. This leads to a race condition, even when not using audio, as both threads attempt to delete and replace the scene->speaker_handles. I'm not sure whether it even makes sense to add/remove sound strips to/from the same scene->sound_scene from two places, even in a thread safe manner. Viewport and renderer would probably need each its own instance. I guess this could be solved by (TODO, move to depsgraph).

This would be a solution to revert the problem in the meantime:

diff --git a/source/blender/blenkernel/intern/scene.c b/source/blender/blenkernel/intern/scene.c
index 644eb704655..e48d1ba2d82 100644
--- a/source/blender/blenkernel/intern/scene.c
+++ b/source/blender/blenkernel/intern/scene.c
@@ -65,6 +65,7 @@
 #include "BKE_editmesh.h"
 #include "BKE_fcurve.h"
 #include "BKE_freestyle.h"
+#include "BKE_global.h"
 #include "BKE_gpencil.h"
 #include "BKE_icons.h"
 #include "BKE_idprop.h"
@@ -1506,7 +1507,9 @@ void BKE_scene_graph_update_tagged(Depsgraph *depsgraph,
 	 */
 	DEG_evaluate_on_refresh(depsgraph);
 	/* Update sound system animation (TODO, move to depsgraph). */
-	BKE_sound_update_scene(bmain, scene);
+	if (!G.is_rendering) {
+		BKE_sound_update_scene(bmain, scene);
+	}
 
 	/* Notify python about depsgraph update */
 	if (run_callbacks) {

Nice find on where the problem was introduced! I'd be fine with the temporary fix. Are you sure that this doesn't prevent both calls of the two threads during rendering? One should still be done. I'm not sure what moving to the dependency graph will change here, does the dependency graph get updated from two threads as well? Or is it two different instances and thus as you said viewport and renderer each need their own sound_scene instance?

The two calls to BKE_sound_update_scene both come from a depencygraph. One holds the copies for the viewport and the other one for the renderer. Both are running in their own thread. BKE_scene_graph_update_tagged happens to not be called for the render thread. But BKE_scene_graph_update_for_newframe is called for both, which is a problem.

Apparently each dependency graph already creates a copy of sound_scene and speaker_handles. So this should be working (only tested without audio):

diff --git a/source/blender/blenkernel/intern/scene.c b/source/blender/blenkernel/intern/scene.c
index 644eb704655..77ee3731766 100644
--- a/source/blender/blenkernel/intern/scene.c
+++ b/source/blender/blenkernel/intern/scene.c
@@ -1506,7 +1506,7 @@ void BKE_scene_graph_update_tagged(Depsgraph *depsgraph,
 	 */
 	DEG_evaluate_on_refresh(depsgraph);
 	/* Update sound system animation (TODO, move to depsgraph). */
-	BKE_sound_update_scene(bmain, scene);
+	BKE_sound_update_scene(bmain, DEG_get_evaluated_scene(depsgraph));
 
 	/* Notify python about depsgraph update */
 	if (run_callbacks) {
@@ -1551,7 +1551,7 @@ void BKE_scene_graph_update_for_newframe(Depsgraph *depsgraph,
 	 */
 	DEG_evaluate_on_framechange(bmain, depsgraph, ctime);
 	/* Update sound system animation (TODO, move to depsgraph). */
-	BKE_sound_update_scene(bmain, scene);
+	BKE_sound_update_scene(bmain, DEG_get_evaluated_scene(depsgraph));
 	/* Notify editors and python about recalc. */
 	BLI_callback_exec(bmain, &scene->id, BLI_CB_EVT_FRAME_CHANGE_POST);
 	/* Inform editors about possible changes. */

I added an audio track in the sequencer to do some testing. Everything seams to work fine. Only when adding another track while rendering, the added track is part of the remaining render. But this seems to be a general problem, as moving the cube in the default scene while rendering does show up in the render too. I assume this is limited to animation rendering.

I also tested one scene with audio as a track in another scene with audio. Which worked out too.

Brecht Van Lommel (brecht) renamed this task from Crashing while rendering animation. Grease pencil. to Crashing with animation rendering and sound.EditedApr 15 2019, 5:45 PM

So this fix goes in the right direction, but it's not complete. All the sound handles and state should only exist and get updated on the evaluated scene, not the original scene. That means saving audio and playback should also get data from the evaluated scene.

Assigning to @Sergey Sharybin (sergey) to look into.

Sergey Sharybin (sergey) closed this task as Resolved.May 3 2019, 3:59 PM

I did a series of changes, which should have made the audio system compatible with copy-on-write concept. The latest change is in rBa460e972123.

Give it a test, and if there are more issues discovered make a new report.

Re-opening. Need to re-iterate over the fix.

I can still trigger this bug fairly frequently just by rendering the default .blend file at a low resolution.

Primarily crashing in AUD_destroySet and occasionally in AUD_getSet.

Sergey Sharybin (sergey) closed this task as Resolved.Jun 5 2019, 2:25 PM

Committed reworked patch at rBbbaa1bffe9d. The issue is now expected to be solved.

Thanks for the report, closing.