Mantaflow rendering unstable (when using multiple fluid domains) #80154

New Issue

Rabit · 2020-08-27T08:23:15+02:00

Rabit commented

2020-08-27 08:23:15 +02:00

System Information
Operating system: Ubuntu 18.04
Graphics card: Radeon R9 FURY

Blender Version
Broken: 2.83.5
Worked: none

Short description of error
Mantaflow seems unstable during rendering of 6 simultaneous domains and it relates to the cycles AA samples.
The error it shows:

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33

Exact steps for others to reproduce the error

Open the example file mantaflow-render-test.blend
Bake the domains one-by-one
Run the Render Animation
It's quite a random, so could take some time
on ~160 frame the blender will fail and close with the next error:

Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0161.png'
 Time: 00:03.10 (Saving: 00:00.08)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33

Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0051.png'
 Time: 00:01.78 (Saving: 00:00.09)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33

**System Information** Operating system: Ubuntu 18.04 Graphics card: Radeon R9 FURY **Blender Version** Broken: 2.83.5 Worked: none **Short description of error** Mantaflow seems unstable during rendering of 6 simultaneous domains and it relates to the cycles AA samples. The error it shows: ``` terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 ``` **Exact steps for others to reproduce the error** 1. Open the example file [mantaflow-render-test.blend](https://archive.blender.org/developer/F8819438/mantaflow-render-test.blend) 2. Bake the domains one-by-one 3. Run the `Render Animation` 4. It's quite a random, so could take some time 5. on ~160 frame the blender will fail and close with the next error: ``` Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0161.png' Time: 00:03.10 (Saving: 00:00.08) terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 ``` ``` Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0051.png' Time: 00:01.78 (Saving: 00:00.09) terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 ```

Rabit commented

2020-08-27 08:23:15 +02:00

Added subscriber: @Rabit

Raimund Klink commented

2020-08-27 10:46:49 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Rabit commented

2020-08-27 22:00:07 +02:00

Looks like it's happening during animation rendering once per ~150 frames, still trying to figure out the minimal configuration.

Rabit commented

2020-08-28 03:26:50 +02:00

Ok, just reproduced the issue on a simple blend file and default blender 2.83.5 . Will reproduce one more time from nothing and will attach the steps to reproduce and the file.

Rabit commented

2020-08-28 03:37:57 +02:00

Simple test blend file: mantaflow-render-test.blend

Steps to reproduce:

Open the example file
Bake the domains one-by-one
Run the Render Animation
It's quite a random, so could take some time
on ~160 frame the blender will fail and close with the next error:

Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0161.png'
 Time: 00:03.10 (Saving: 00:00.08)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33

Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0051.png'
 Time: 00:01.78 (Saving: 00:00.09)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33

Simple test blend file: [mantaflow-render-test.blend](https://archive.blender.org/developer/F8816327/mantaflow-render-test.blend) Steps to reproduce: 1. Open the example file 2. Bake the domains one-by-one 3. Run the `Render Animation` 4. It's quite a random, so could take some time 5. on ~160 frame the blender will fail and close with the next error: ``` Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0161.png' Time: 00:03.10 (Saving: 00:00.08) terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 ``` ``` Saved: '/home/user/tmp/mantaflow-render-test-scene/out/0051.png' Time: 00:01.78 (Saving: 00:00.09) terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_lts_283/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 ```

Rabit commented

2020-08-31 19:53:52 +02:00

Added subscriber: @Raimund58

Rabit commented

2020-08-31 19:53:52 +02:00

Hello Guys, anyone could reproduce? @Raimund58

Raimund Klink commented

2020-08-31 22:16:00 +02:00

Changed status from 'Needs User Info' to: 'Needs Triage'

Sebastián Barschkis commented

2020-09-01 12:45:13 +02:00

Added subscriber: @sebbas

Sebastián Barschkis commented

2020-09-01 12:45:13 +02:00

@Rabit Are you able to get a crash with the latest daily build too?

Raimund Klink commented

2020-09-01 13:43:51 +02:00

Blender 2.83.5 just crashed for me. Where do I find the logs?

Rabit commented

2020-09-01 21:55:07 +02:00

So, I checked recently released 2.90 - even worse it crashed on 2th step "baking" (5th domain bake, 215 frame):

$ blender
found bundled python: /home/user/local/blender-2.90.0-linux64/2.90/python
Read blend: /home/user/tmp/mantaflow/mantaflow-render-test.blend
terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_290/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  blender

Restarted blender and restarted 5th domain baking, it's going significantly (2-3 times) slower from 4th domain bake - and failed again on 180 frame. So I can't even reproduce the previous 2.83.5 steps.

On 2.83.5 (left domain to right):

First baked slowly
Second quickly
Third slowly
Fourth slowly
Fifth slowly (but at least baked)
Sixth quickly

Next I switched to 2.90 to render with baked caches from 2.83.5:

First render animation - passed
Second render animation - crashed:

Saved: '/home/user/tmp/mantaflow/out/0099.png'
 Time: 00:03.66 (Saving: 00:00.06)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_290/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  blender

So in 2.90 we have another issue with baking the example blend file, as well as with the rendering.

@Raimund58 The logs I posted is right in the stderr/stdout of blender - just check it's system console (on windows) or run it directly from the console.

So, I checked recently released 2.90 - even worse it crashed on 2th step "baking" (5th domain bake, 215 frame): ``` $ blender found bundled python: /home/user/local/blender-2.90.0-linux64/2.90/python Read blend: /home/user/tmp/mantaflow/mantaflow-render-test.blend terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_290/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 zsh: abort (core dumped) blender ``` Restarted blender and restarted 5th domain baking, it's going significantly (2-3 times) slower from 4th domain bake - and failed again on 180 frame. So I can't even reproduce the previous 2.83.5 steps. On 2.83.5 (left domain to right): * First baked slowly * Second quickly * Third slowly * Fourth slowly * Fifth slowly (but at least baked) * Sixth quickly Next I switched to 2.90 to render with baked caches from 2.83.5: * First render animation - passed * Second render animation - crashed: ``` Saved: '/home/user/tmp/mantaflow/out/0099.png' Time: 00:03.66 (Saving: 00:00.06) terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_290/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 zsh: abort (core dumped) blender ``` So in 2.90 we have another issue with baking the example blend file, as well as with the rendering. @Raimund58 The logs I posted is right in the stderr/stdout of blender - just check it's system console (on windows) or run it directly from the console.

Sebastián Barschkis commented

2020-09-02 11:30:12 +02:00

@Rabit Just had my first crash - it took some time as you mentioned already. Will look into this now.

Sebastián Barschkis commented

2020-09-02 11:47:41 +02:00

@Rabit Could you still try and see if you can get a crash with a daily build: https://builder.blender.org/download/

I had my crash with 2.90 too. It would be interesting to know if the issue is present in 2.91 as well.

@Rabit Could you still try and see if you can get a crash with a daily build: https://builder.blender.org/download/ I had my crash with 2.90 too. It would be interesting to know if the issue is present in 2.91 as well.

Sebastián Barschkis commented

2020-09-02 11:47:59 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Rabit commented

2020-09-03 01:03:43 +02:00

I just tested the latest blender-2.91.0-ba188e721899-linux64 build:

Prebuilt 2.83.5 caches render:
- First animation render - passed
- Second animation render - crashed

Saved: '/home/user/tmp/mantaflow/out/0007.png'
 Time: 00:02.80 (Saving: 00:00.06)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_master/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  ~/local/blender-2.91.0-ba188e721899-linux64/blender

Baking domains 1st pass (left to right):
- 1st - slow
- 2nd - quick
- 3rd - quick
- 4th - slow
- 5th - slow
- 6th - slow
  Warnings during baking:

Traceback (most recent call last):
  File "<string>", line 47, in <module>
KeyError: 'reactIn_s1130'
Traceback (most recent call last):
  File "<string>", line 47, in <module>
KeyError: 'reactIn_s1200'

Baking domains 2nd pass (left to right):
- 1st - slow
- 2nd - quick
- 3rd - quick
- 4th - slow
- 5th - slow
- 6th - slow
Fresh baked caches render:
- First animation render - passed
- Second animation render - stopped:

Executing command:
  /home/user/local/blender-2.91.0-ba188e721899-linux64/blender -a -f 24 1.0 -s 1 -e 250 -j 1 /home/user/tmp/mantaflow-2/out/0001.png

Third animation - crashed:

Saved: '/home/user/tmp/mantaflow-2/out/0015.png'
 Time: 00:04.20 (Saving: 00:00.06)

terminate called after throwing an instance of 'Manta::Error'
  what():  can't clean grid cache, some grids are still in use
Error raised in /home/sources/buildbot-worker-linux_centos7/linux_master/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33
zsh: abort (core dumped)  ~/local/blender-2.91.0-ba188e721899-linux64/blender

So it looks quite the same as for 2.83.5 - but I did not catch the failure during 2 passes of baking as for 2.90 version.

I just tested the latest blender-2.91.0-ba188e721899-linux64 build: * Prebuilt 2.83.5 caches render: * First animation render - passed * Second animation render - crashed ``` Saved: '/home/user/tmp/mantaflow/out/0007.png' Time: 00:02.80 (Saving: 00:00.06) terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_master/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 zsh: abort (core dumped) ~/local/blender-2.91.0-ba188e721899-linux64/blender ``` * Baking domains 1st pass (left to right): * 1st - slow * 2nd - quick * 3rd - quick * 4th - slow * 5th - slow * 6th - slow Warnings during baking: ``` Traceback (most recent call last): File "<string>", line 47, in <module> KeyError: 'reactIn_s1130' Traceback (most recent call last): File "<string>", line 47, in <module> KeyError: 'reactIn_s1200' ``` * Baking domains 2nd pass (left to right): * 1st - slow * 2nd - quick * 3rd - quick * 4th - slow * 5th - slow * 6th - slow * Fresh baked caches render: * First animation render - passed * Second animation render - stopped: ``` Executing command: /home/user/local/blender-2.91.0-ba188e721899-linux64/blender -a -f 24 1.0 -s 1 -e 250 -j 1 /home/user/tmp/mantaflow-2/out/0001.png ``` * Third animation - crashed: ``` Saved: '/home/user/tmp/mantaflow-2/out/0015.png' Time: 00:04.20 (Saving: 00:00.06) terminate called after throwing an instance of 'Manta::Error' what(): can't clean grid cache, some grids are still in use Error raised in /home/sources/buildbot-worker-linux_centos7/linux_master/blender.git/extern/mantaflow/preprocessed/fluidsolver.cpp:33 zsh: abort (core dumped) ~/local/blender-2.91.0-ba188e721899-linux64/blender ``` So it looks quite the same as for 2.83.5 - but I did not catch the failure during 2 passes of baking as for 2.90 version.

Sebastián Barschkis commented

2020-09-03 10:30:00 +02:00

@Rabit Thanks a lot for all the tests! I also got some more crashes in my tests, it looks like this is an issue in general when having multiple domains in the scene.
Investigating this ..

@Rabit Thanks a lot for all the tests! I also got some more crashes in my tests, it looks like this is an issue in general when having multiple domains in the scene. Investigating this ..

Sebastián Barschkis changed title from ~~Mantaflow rendering unstable~~ to Mantaflow rendering unstable (when using multiple fluid domains)

2020-09-11 10:54:20 +02:00

Rabit commented

2020-09-11 19:52:13 +02:00

Just found another thing with baking on 2.90 - if I'm changing the initial fuel level (0-1.3 bezier) for example and not starting from 1.3 - it's dissolving before reaching top of the domain, than it probably will not fail. Not sure how it's related, but maybe it's a workaround for now - just make sure the fire particles not reaching the domain limits.

Alexandre Comeau commented

2021-02-25 05:31:00 +01:00

Added subscriber: @VGSymphony

Alexandre Comeau commented

2021-02-25 05:31:00 +01:00

I still have an issue with baking in 2.92 with multiple domains. Any update on the bug?

Ankit Meel commented

2021-06-24 14:58:49 +02:00

Added subscriber: @ankitm

Ankit Meel commented

2021-06-24 14:58:49 +02:00

I confirmed a report with same error with simpler steps and reliable crash in #87828 (Mantaflow error when using blender in background mode). There's also an assert hit there.
Keeping this open since steps to redo are very different.

I confirmed a report with same error with simpler steps and reliable crash in #87828 (Mantaflow error when using blender in background mode). There's also an assert hit there. Keeping this open since steps to redo are very different.

Evan Ryan commented

2021-12-24 08:29:33 +01:00

Added subscriber: @evantryan

Dalai Felinto commented

2022-05-09 11:01:37 +02:00

Mantaflow (fluid simulator) has no active developer at the moment. Moving those bugs as known issues so at least other users can find their issues already reported.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Mantaflow rendering unstable (when using multiple fluid domains) #80154