Page MenuHome

Mantaflow rendering unstable (when using multiple fluid domains)
Confirmed, NormalPublic

Description

System Information
Operating system: Ubuntu 18.04
Graphics card: Radeon R9 FURY

Blender Version
Broken: 2.83.5
Worked: none

Short description of error
Mantaflow seems unstable during rendering of 6 simultaneous domains and it relates to the cycles AA samples.
The error it shows:

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33

Exact steps for others to reproduce the error

  1. Open the example file
  2. Bake the domains one-by-one
  3. Run the Render Animation
  4. It's quite a random, so could take some time
  5. on ~160 frame the blender will fail and close with the next error:
Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0161.png'
 Time: 00:03.10 (Saving: 00:00.08)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0051.png'
 Time: 00:01.78 (Saving: 00:00.09)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33

Event Timeline

Raimund Klink (Raimund58) changed the task status from Needs Triage to Needs Information from User.Thu, Aug 27, 10:46 AM
Rabit (Rabit) added a comment.EditedThu, Aug 27, 10:00 PM

Looks like it's happening during animation rendering once per ~150 frames, still trying to figure out the minimal configuration.

Ok, just reproduced the issue on a simple blend file and default blender 2.83.5 . Will reproduce one more time from nothing and will attach the steps to reproduce and the file.

Simple test blend file:

Steps to reproduce:

  1. Open the example file
  2. Bake the domains one-by-one
  3. Run the Render Animation
  4. It's quite a random, so could take some time
  5. on ~160 frame the blender will fail and close with the next error:
Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0161.png'
 Time: 00:03.10 (Saving: 00:00.08)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0051.png'
 Time: 00:01.78 (Saving: 00:00.09)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
Raimund Klink (Raimund58) changed the task status from Needs Information from User to Needs Triage.Mon, Aug 31, 10:16 PM

@Rabit (Rabit) Are you able to get a crash with the latest daily build too?

Blender 2.83.5 just crashed for me. Where do I find the logs?

Rabit (Rabit) added a comment.EditedTue, Sep 1, 9:55 PM

So, I checked recently released 2.90 - even worse it crashed on 2th step "baking" (5th domain bake, 215 frame):

$ blender
found bundled python: /home/user/local/blender-2.90.0-linux64/2.90/python
Read blend: /home/user/tmp/mantaflow/mantaflow-render-test.blend
terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_290/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  blender

Restarted blender and restarted 5th domain baking, it's going significantly (2-3 times) slower from 4th domain bake - and failed again on 180 frame. So I can't even reproduce the previous 2.83.5 steps.

On 2.83.5 (left domain to right):

  • First baked slowly
  • Second quickly
  • Third slowly
  • Fourth slowly
  • Fifth slowly (but at least baked)
  • Sixth quickly

Next I switched to 2.90 to render with baked caches from 2.83.5:

  • First render animation - passed
  • Second render animation - crashed:
Saved: '/home/user/tmp/mantaflow/out/0099.png'
 Time: 00:03.66 (Saving: 00:00.06)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_290/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  blender

So in 2.90 we have another issue with baking the example blend file, as well as with the rendering.

@Raimund Klink (Raimund58) The logs I posted is right in the stderr/stdout of blender - just check it's system console (on windows) or run it directly from the console.

@Rabit (Rabit) Just had my first crash - it took some time as you mentioned already. Will look into this now.

@Rabit (Rabit) Could you still try and see if you can get a crash with a daily build: https://builder.blender.org/download/

I had my crash with 2.90 too. It would be interesting to know if the issue is present in 2.91 as well.

Sebastián Barschkis (sebbas) changed the task status from Needs Triage to Confirmed.Wed, Sep 2, 11:47 AM

I just tested the latest blender-2.91.0-ba188e721899-linux64 build:

  • Prebuilt 2.83.5 caches render:
    • First animation render - passed
    • Second animation render - crashed
Saved: '/home/user/tmp/mantaflow/out/0007.png'
 Time: 00:02.80 (Saving: 00:00.06)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_master/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  ~/local/blender-2.91.0-ba188e721899-linux64/blender
  • Baking domains 1st pass (left to right):
    • 1st - slow
    • 2nd - quick
    • 3rd - quick
    • 4th - slow
    • 5th - slow
    • 6th - slow

Warnings during baking:

Traceback (most recent call last):
  File "<string>", line 47, in <module>
KeyError: 'reactIn_s1130'
Traceback (most recent call last):
  File "<string>", line 47, in <module>
KeyError: 'reactIn_s1200'
  • Baking domains 2nd pass (left to right):
    • 1st - slow
    • 2nd - quick
    • 3rd - quick
    • 4th - slow
    • 5th - slow
    • 6th - slow
  • Fresh baked caches render:
    • First animation render - passed
    • Second animation render - stopped:
Executing command:
  /home/user/local/blender-2.91.0-ba188e721899-linux64/blender -a -f 24 1.0 -s 1 -e 250 -j 1 /home/user/tmp/mantaflow-2/out/0001.png
  • Third animation - crashed:
Saved: '/home/user/tmp/mantaflow-2/out/0015.png'
 Time: 00:04.20 (Saving: 00:00.06)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_master/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  ~/local/blender-2.91.0-ba188e721899-linux64/blender

So it looks quite the same as for 2.83.5 - but I did not catch the failure during 2 passes of baking as for 2.90 version.

@Rabit (Rabit) Thanks a lot for all the tests! I also got some more crashes in my tests, it looks like this is an issue in general when having multiple domains in the scene.
Investigating this ..

Sebastián Barschkis (sebbas) renamed this task from Mantaflow rendering unstable to Mantaflow rendering unstable (when using multiple fluid domains).Fri, Sep 11, 10:54 AM

Just found another thing with baking on 2.90 - if I'm changing the initial fuel level (0-1.3 bezier) for example and not starting from 1.3 - it's dissolving before reaching top of the domain, than it probably will not fail. Not sure how it's related, but maybe it's a workaround for now - just make sure the fire particles not reaching the domain limits.