Blender consistently deleting Bake files #56083

New Issue

Mike · 2018-07-22T23:20:42+02:00

Mike commented

2018-07-22 23:20:42 +02:00

System Information
Experienced this on two systems.
A. MacPro 2103,8-core, 64GB RAM, Dual D700 GPUs running MacOS Sierra 12.12.16
B. 16-Core AMD, 64GB RAM, 1080 Ti and 980 Ti GPUs, running Windows 10

Blender Version
Broken: Offical 2.79b, and 2.79b Released on July 19th, 2018
Worked: None

Short description of error
The blend projects on the Mac and PC are two completely different projects for different films.

On the Mac: Baked a large smoke domain that took two days to bake. 90% or so through the bake, the bake files were gone, and the only bake files that remained where those from the 90% onward. So the bake files went missing in the middle of the bake!

On the PC: I have 60 smoke domains for torches. Half way through the 250 frame bake using internal bake files, Blender crashed and all the bake files were lost. It was recommended I put the smoke domains in a separate scene and divide them up into layers. I did this, and baked 4-5 domains at a time. After two days of baking these domains (external bakes this time) the bake files went missing. No apparent reason whatsoever. So I started over, now baking up the bake files after the bake is completed. I baked 30 domains (took 24+ hours) and then backed up the bake files. I discovered however, than frames 146-250 on almost all of those bakes were missing! In fact, I would bake 4 domains and immediately perform a backup. The program found all 250 bake files and started copying them to the backup drive, and even before it could finish, it reported errors saying files were missing. So even before the backup could finish, Blender had somehow erased the latter half of the bake files! What in the world is going on!? I cannot get anything baked at all! This is completely broken -- broken on two platforms with all Blender 2.79b builds. I have not tried others but have had problems like this along time ago. This needs to get fixed ASAP. If I cant bake, the software is broken.

Exact steps for others to reproduce the error
Described above.

**System Information** Experienced this on two systems. A. MacPro 2103,8-core, 64GB RAM, Dual [D700](https://archive.blender.org/developer/D700) GPUs running MacOS Sierra 12.12.16 B. 16-Core AMD, 64GB RAM, 1080 Ti and 980 Ti GPUs, running Windows 10 **Blender Version** Broken: Offical 2.79b, and 2.79b Released on July 19th, 2018 Worked: None **Short description of error** The blend projects on the Mac and PC are two completely different projects for different films. On the Mac: Baked a large smoke domain that took two days to bake. 90% or so through the bake, the bake files were gone, and the only bake files that remained where those from the 90% onward. So the bake files went missing in the middle of the bake! On the PC: I have 60 smoke domains for torches. Half way through the 250 frame bake using internal bake files, Blender crashed and all the bake files were lost. It was recommended I put the smoke domains in a separate scene and divide them up into layers. I did this, and baked 4-5 domains at a time. After two days of baking these domains (external bakes this time) the bake files went missing. No apparent reason whatsoever. So I started over, now baking up the bake files after the bake is completed. I baked 30 domains (took 24+ hours) and then backed up the bake files. I discovered however, than frames 146-250 on almost all of those bakes were missing! In fact, I would bake 4 domains and immediately perform a backup. The program found all 250 bake files and started copying them to the backup drive, and even before it could finish, it reported errors saying files were missing. So even before the backup could finish, Blender had somehow erased the latter half of the bake files! What in the world is going on!? I cannot get anything baked at all! This is completely broken -- broken on two platforms with all Blender 2.79b builds. I have not tried others but have had problems like this along time ago. This needs to get fixed ASAP. If I cant bake, the software is broken. **Exact steps for others to reproduce the error** Described above.

Mike commented

2018-07-22 23:20:42 +02:00

Added subscriber: @LMProductions-1

Mike changed title from ~~Blender consistently deleting cached Bake files~~ to Blender consistently deleting Bake files

2018-07-22 23:21:05 +02:00

Brecht Van Lommel commented

2018-07-24 11:48:39 +02:00

Added subscriber: @brecht

Brecht Van Lommel commented

2018-07-24 11:48:39 +02:00

It sounds like a bad bug, but we require a .blend file and steps to reproduce it in bug reports. It should be possible to create an example .blend with a very simple smoke sim that bakes quickly and exhibits the same problem.

Chris commented

2018-07-25 00:36:16 +02:00

Added subscriber: @ElDirector

Chris commented

2018-07-25 00:36:16 +02:00

I'm having this issue too. I've tried both OpenVDB and Point Cache. Low Divisions and High Divisions. External Path and Lib Path. In the past, I've had issues where it wouldn't bake all the way and I would have to run it again. Usually the cause was being on a frame other than the first frame of the bake, which seems odd. If I went back to frame one and started the bake again, it would usually work the second time and I'd never have any issues. Medium Smoke Trail-Chris.blend

I've attached the most recent project that has has the bake files vanish. This was sent to me by a guy helping to make the smoke assets we need for some VFX

I'm having this issue too. I've tried both OpenVDB and Point Cache. Low Divisions and High Divisions. External Path and Lib Path. In the past, I've had issues where it wouldn't bake all the way and I would have to run it again. Usually the cause was being on a frame other than the first frame of the bake, which seems odd. If I went back to frame one and started the bake again, it would usually work the second time and I'd never have any issues. [Medium Smoke Trail-Chris.blend](https://archive.blender.org/developer/F4033724/Medium_Smoke_Trail-Chris.blend) I've attached the most recent project that has has the bake files vanish. This was sent to me by a guy helping to make the smoke assets we need for some VFX

Mike commented

2018-07-30 05:42:19 +02:00

here is a link to a dramatically simplified .blend with only 5 smoke domains. (Imagine 4-5 more smoke domains on 10 more layers and that is my scene for the torches). Bake these domains using the external settings, only maybe changing the folders, and then save the project. Close it. Restart the computer or just wait awhile and then make move the playhead to some different frames, and eventually the bake files will disappear. I dont even know what's causing this (thats the problem), or else I would give more specific steps to reproduce it, but again, the behaviour is consistent across 3 machines and different .blend projects on different platforms (mac & windows). We REALLY need some help with this. Cant get any bakes to stay put... and with large bakes (over 200gb collectively and 12+ hours into bakes) this is a HUGE HUGE problem.
Bake sample can be downloaded here: bake_sample.zip

here is a link to a dramatically simplified .blend with only 5 smoke domains. (Imagine 4-5 more smoke domains on 10 more layers and that is my scene for the torches). Bake these domains using the external settings, only maybe changing the folders, and then save the project. Close it. Restart the computer or just wait awhile and then make move the playhead to some different frames, and eventually the bake files will disappear. I dont even know what's causing this (thats the problem), or else I would give more specific steps to reproduce it, but again, the behaviour is consistent across 3 machines and different .blend projects on different platforms (mac & windows). We REALLY need some help with this. Cant get any bakes to stay put... and with large bakes (over 200gb collectively and 12+ hours into bakes) this is a HUGE HUGE problem. Bake sample can be downloaded here: [bake_sample.zip ](http://selflesslove.domain.com/temp/bake_sample.zip)

Chris commented

2018-08-03 00:13:06 +02:00

More files being deleted. Started a smoke bake that ranged from frame 1-500. . The attached image shows the only files left.

More files being deleted. Started a smoke bake that ranged from frame 1-500. ![Screen Shot 2018-08-02 at 3.11.21 PM.png](https://archive.blender.org/developer/F4110719/Screen_Shot_2018-08-02_at_3.11.21_PM.png) . The attached image shows the only files left.

Mike commented

2018-08-03 03:00:05 +02:00

Did a bake with 5 new domains a couple days ago for torches. Started at frame 1. Baked the domains. Then moved the playhead to a half-way point through the animation to check the fire and they were all there. Then I did a F12 render and 2 of the torches were missing. Checked the bake files and the files for those two domains were gone. Guys, this happens over and over and over again. It’s killing me. I’m literally not able to do anything that requires a bake. We are losing bake files at work that take days to bake! Please help us find a fix or a work-around or something.

Mike commented

2018-09-02 20:50:59 +02:00

Over the three weeks I think Ive managed to eliminate the deleting of bake files. I did this by making all the bake files READ ONLY after they were done baking. Today I viewed the process output from one render job and I noticed a ton of errors where blender was reporting that it could not delete a file. Could it be, that blender is trying to delete my bake files, but cant because I made them read-only??

Over the three weeks I think Ive managed to eliminate the deleting of bake files. I did this by making all the bake files READ ONLY after they were done baking. Today I viewed the process output from one render job and I noticed a ton of errors where blender was reporting that it could not delete a file. Could it be, that blender is trying to delete my bake files, but cant because I made them read-only??![deleting bakes.PNG](https://archive.blender.org/developer/F4525558/deleting_bakes.PNG)

Hermann Voßeler commented

2019-03-23 16:14:37 +01:00

Added subscriber: @Ichthyostega

Hermann Voßeler commented

2019-03-23 16:14:37 +01:00

Hi Mike,
can't access your sample file -- but I kind of know that problem. I encounter it as unfortunate combination of circumstances.

As a starting point: whenever a simulation is not yet marked as baked in blender, then on any move of the playhead, Blender seemingly marks its internal cache as dirty; and when this especially touches some smoke sim, it attempts to calculate the current frame. Now, there is a special twist: a smoke sim can only be calculated, when you enter it from the first frame, since it needs to be calculated consecutively. For that reason, blender discards any "stale" cache data behind the current position, which also results in deleting the disk cache.

The only way to prevent that from happening is to have the smoke sim marked as "baked". You can see this in the property pane of the smoke domain, on physics tab: when baked, most settings are grayed (can't be edited), and there is a "free bake" button in the cache area. (you can also see this state when you look at the data blocks of the blend file)

Usually, if you just bake interactively, by pressing the "bake" button, blender freezes during the bake process, and when successfully done, immediately marks this simulation as baked. Then you must save the file to capture this state. If you just close, the files are in the cache, but next time blender considers them as stale, since it doesn't recognise the simulation as baked.

And now comes the twist: if you bake through a python script, e.g. on some external machine with many cores, then this script must actively save the blend file and you need to retrieve not only the baked frames, but also the blend file.

There is a second workaround (and this is what I am using in practice): Once you have calculated the simulation on said external machine, you retrieve the cache frames and place them into another directory (not the blendcache_* directory). Make sure the playhead is somewhere in the middle of the smoke sim's area. (It should not show any smoke at that point, since the internal cache is still stale). Then you toggle the "external" checkbox below the cache settings and enter the path to the external directory. Important: now wait until the smoke sim shows up in the 3D viewport. Never move the playhead, don't touch anything. And when it has showed up, which requires a considerable amount of time where your CPU goes 100%, immediately click the "current cache to bake" button. And then immediately save the blend file. This gets the simulation into "baked" state, and you're safe after that point.

The fun fact is, once a smoke sim is in "baked" state, you can do quite some manipulations without problems. E.g. you can exchange the cache files, or you can even switch the location for external cache files to another directory (e.g. with a newer, augmented build of the cache files). And you can jump around arbitrarily and inspect the state at any frame (I wouldn't call that "playback", since it is quite slow, once the smoke as a reasonable resolution).

And to state the obvious: you need a backup of your precious cache files... I know this is a cost factor, but you just need to price that in. Allocating storage is way cheaper than allocating calculation and work time for re-creating lost cache files.

Hi Mike, can't access your sample file -- but I kind of know that problem. I encounter it as unfortunate combination of circumstances. As a starting point: whenever a simulation *is not yet marked as baked* in blender, then on *any move of the playhead,* Blender seemingly marks its internal cache as dirty; and when this especially touches some smoke sim, it attempts to calculate the current frame. Now, there is a special twist: a smoke sim can only be calculated, when you enter it from the first frame, since it needs to be calculated consecutively. For that reason, blender discards any "stale" cache data behind the current position, which also results in **deleting** the disk cache. The only way to prevent that from happening is to **have the smoke sim marked as "baked"**. You can see this in the property pane of the smoke domain, on physics tab: when *baked*, most settings are grayed (can't be edited), and there is a "free bake" button in the cache area. (you can also see this state when you look at the data blocks of the blend file) Usually, if you just bake interactively, by pressing the "bake" button, blender freezes during the bake process, and *when successfully done,* immediately marks this simulation as baked. Then you **must save the file** to capture this state. If you just close, the files are in the cache, but next time blender considers them as stale, since it doesn't recognise the simulation as baked. And now comes the twist: if you bake through a python script, e.g. on some external machine with many cores, then this script **must actively save the blend file** and you need to retrieve not only the baked frames, but also the blend file. There is a second workaround (and this is what I am using in practice): Once you have calculated the simulation on said external machine, you retrieve the cache frames and place them into another directory (not the blendcache_* directory). Make sure *the playhead is somewhere in the middle of the smoke sim's area.* (It should *not* show any smoke at that point, since the internal cache is still stale). Then you toggle the "external" checkbox below the cache settings and enter the path to the external directory. Important: now wait until the smoke sim shows up in the 3D viewport. *Never move the playhead, don't touch anything*. And when it has showed up, which requires a considerable amount of time where your CPU goes 100%, **immediately click the "current cache to bake"** button. And then **immediately save the blend file**. This gets the simulation into "baked" state, and you're safe after that point. The fun fact is, once a smoke sim is in "baked" state, you can do quite some manipulations without problems. E.g. you can exchange the cache files, or you can even switch the location for external cache files to another directory (e.g. with a newer, augmented build of the cache files). And you can jump around arbitrarily and inspect the state at any frame (I wouldn't call that "playback", since it is quite slow, once the smoke as a reasonable resolution). And to state the obvious: you need **a backup** of your precious cache files... I know this is a cost factor, but you just need to price that in. Allocating storage is way cheaper than allocating calculation and work time for re-creating lost cache files.

Sybren A. Stüvel commented

2020-02-04 16:33:28 +01:00

Added subscriber: @dr.sybren

Sybren A. Stüvel commented

2020-02-04 16:33:28 +01:00

Changed status from 'Confirmed' to: 'Archived'

Sybren A. Stüvel closed this issue

2020-02-04 16:33:28 +01:00

Sybren A. Stüvel self-assigned this 2020-02-04 16:33:28 +01:00

Sybren A. Stüvel commented

2020-02-04 16:33:29 +01:00

This report does not contain all the requested information, which is required for us to investigate the issue. The example blend file from @ElDirector is too complex for us to diagnose (with 35 hidden objects that may or may not be influencing this issue), and there are no clearly described steps to reproduce the issue. The link provided by @LMProductions-1 is dead.

If this problem still occurs with a recent daily build of Blender (https://builder.blender.org/), please submit a new report and carefully follow the instructions. Be sure to provide system information, Blender version, the last Blender version which worked, and a minimal .blend file with exact steps to reproduce the problem.

A guideline for making a good bug report can be found at https://wiki.blender.org/wiki/Process/Bug_Reports

This report does not contain all the requested information, which is required for us to investigate the issue. The example blend file from @ElDirector is too complex for us to diagnose (with 35 hidden objects that may or may not be influencing this issue), and there are no clearly described steps to reproduce the issue. The link provided by @LMProductions-1 is dead. If this problem still occurs with a recent daily build of Blender (https://builder.blender.org/), please submit a new report and carefully follow the instructions. Be sure to provide system information, Blender version, the last Blender version which worked, and a minimal .blend file with exact steps to reproduce the problem. A guideline for making a good bug report can be found at https://wiki.blender.org/wiki/Process/Bug_Reports

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Blender consistently deleting Bake files #56083