Better handling of temp files #39690

New Issue

Bastien Montagne · 2014-04-12T16:42:22+02:00

Bastien Montagne commented

2014-04-12 16:42:22 +02:00

Right now, Blender just stores its temp files in temp directory. In some cases, it even uses PID to avoid several instances of it using the same files (at least, for point caches of unsaved .blends, and added that behavior to fix #39521, see 6cd717e0a3).

The issue is that Blender never removes those files, and some OSs (Windows…) never clean up their temp dir, which can leads to a quick waste of space. So here is the proposition (also from discussion with Campbell):

At startup, Blender creates a (PID-tagged) folder in temp dir, where it stores all files that are really temp ones.
When closing, Blender removes that folder and all its content.

For the very few cases where we need a 'cross-session' persistent 'temp' files (only render result afaik?), we could create temp file directly in main tmp dir (not in Blender session’s PID-tagged one).

The issue here is how to avoid several instances to use the same 'persistent' file? We could use locks, but I can’t find a nice way to handle them (main issue being to handle crashes, which are one key reason to have those persistent render results). Or maybe we just decide this is a corner case we do not want to support (i.e. in situations like #39521, we simply accept both processes write the same temp render result file - intermediate samples being saved in PID temp folder anyway)?

Right now, Blender just stores its temp files in temp directory. In some cases, it even uses PID to avoid several instances of it using the same files (at least, for point caches of unsaved .blends, and added that behavior to fix #39521, see 6cd717e0a3). The issue is that Blender never removes those files, and some OSs (Windows…) never clean up their temp dir, which can leads to a quick waste of space. So here is the proposition (also from discussion with Campbell): * At startup, Blender creates a (PID-tagged) folder in temp dir, where it stores all files that are really temp ones. * When closing, Blender removes that folder and all its content. For the very few cases where we need a 'cross-session' persistent 'temp' files (only render result afaik?), we could create temp file directly in main tmp dir (**not** in Blender session’s PID-tagged one). The issue here is how to avoid several instances to use the same 'persistent' file? We could use locks, but I can’t find a nice way to handle them (main issue being to handle crashes, which are one key reason to have those persistent render results). Or maybe we just decide this is a corner case we do not want to support (i.e. in situations like #39521, we simply accept both processes write the same temp render result file - intermediate samples being saved in PID temp folder anyway)?

Bastien Montagne commented

2014-04-12 16:42:22 +02:00

Changed status to: 'Open'

Bastien Montagne commented

2014-04-12 16:42:22 +02:00

Added subscribers: @mont29, @ideasman42, @brecht, @Sergey

Brecht Van Lommel commented

2014-04-13 14:07:26 +02:00

I suggest to look at what other applications do, this must be a common problem. There also exist operating system functions like mktemp to get unique filenames, which I think we should use when possible rather than creating our own system with PIDs.

One case to think of as well is crashes, you don't want all the temp files to stick around when Blender crashes.

For the render result we should make the caching an option I think, that is disabled by default. Not sure about the best location for the file, maybe the output file path. It can still create a temp file if the caching is disabled.

I suggest to look at what other applications do, this must be a common problem. There also exist operating system functions like `mktemp` to get unique filenames, which I think we should use when possible rather than creating our own system with PIDs. One case to think of as well is crashes, you don't want all the temp files to stick around when Blender crashes. For the render result we should make the caching an option I think, that is disabled by default. Not sure about the best location for the file, maybe the output file path. It can still create a temp file if the caching is disabled.

Sergey Sharybin commented

2014-04-13 16:25:05 +02:00

I'm not so much fun of this PID and mktemp things actually.

Not sure what else apart from the saved buffers gives issues, but for saved buffers it'll be better idea to NOT use PID but use full .blend file path converted to a proper file name (i.e. /home/sergey/blah.blend becomes /tmp/home_sergey_blah.blend). This way it'll be still possible to restart blender and use render buffers from the previous instance.

Just an idea, if we wouldn't find it's suitable then i'll fine with the PID salt or mktemp. Maybe mktemp is the better approach in fact.

I'm not so much fun of this PID and mktemp things actually. Not sure what else apart from the saved buffers gives issues, but for saved buffers it'll be better idea to NOT use PID but use full .blend file path converted to a proper file name (i.e. /home/sergey/blah.blend becomes /tmp/home_sergey_blah.blend). This way it'll be still possible to restart blender and use render buffers from the previous instance. Just an idea, if we wouldn't find it's suitable then i'll fine with the PID salt or mktemp. Maybe mktemp is the better approach in fact.

Bastien Montagne self-assigned this 2014-04-17 14:15:44 +02:00

Bastien Montagne commented

2014-04-17 14:15:44 +02:00

Not sure what else apart from the saved buffers gives issues, but for saved buffers it'll be better idea to NOT use PID but use full .blend file path converted to a proper file name (i.e. /home/sergey/blah.blend becomes /tmp/home_sergey_blah.blend). This way it'll be still possible to restart blender and use render buffers from the previous instance.

That’s already what is done, but this fails when you render the same frames of the same file in two different processes (like in org BT, where the only diff was the used cameras).

For the render result we should make the caching an option I think, that is disabled by default. Not sure about the best location for the file, maybe the output file path. It can still create a temp file if the caching is disabled.

Agree, current situation is a bit like bug-that-becomes-a-feature…

One case to think of as well is crashes, you don't want all the temp files to stick around when Blender crashes.

Yes, this is the main issue, yet have to check whether our crash handler (the one writing crash log) would be able to also delete temp files (hopefully it will ;) ).

> Not sure what else apart from the saved buffers gives issues, but for saved buffers it'll be better idea to NOT use PID but use full .blend file path converted to a proper file name (i.e. /home/sergey/blah.blend becomes /tmp/home_sergey_blah.blend). This way it'll be still possible to restart blender and use render buffers from the previous instance. That’s already what is done, but this fails when you render *the same frames of the same file* in two different processes (like in org BT, where the only diff was the used cameras). > For the render result we should make the caching an option I think, that is disabled by default. Not sure about the best location for the file, maybe the output file path. It can still create a temp file if the caching is disabled. Agree, current situation is a bit like bug-that-becomes-a-feature… > One case to think of as well is crashes, you don't want all the temp files to stick around when Blender crashes. Yes, this is the main issue, yet have to check whether our crash handler (the one writing crash log) would be able to also delete temp files (hopefully it will ;) ).

Derrick Lambert commented

2014-04-19 08:07:37 +02:00

Added subscriber: @drekryan