Add File Locking Support to BLI_fileops #103935

New Issue

Bastien Montagne · 2023-01-17T12:03:41+01:00

Bastien Montagne commented

2023-01-17 12:03:41 +01:00

UPDATE:

After talking with @brecht, it seems that writing in temp file + atomically moving it to the actual draft.blend file would match the requirements for the draft project:

On UNIX, this works 'out of the box', processes having 'old' draft.blend open will keep using it, new openings will have the new version of the file.
On Windows, the atomic move of the temp file to its final name will fail if the current draft.blend is used by other processes. This is acceptable and can be worked around (e.g. by looping a reasonable amount of times over the move call).
The only change from the Draft handling PoV is that it needs to have a way to report failures to write to the user, in the rare case this should still happen. Not a bad thing to have anyway.

ORIGINAL Proposal:

The Draft project will require several instances of Blender being able to safely access a same draft .blend file, either for reading or writing.

This could be achieved in two ways:

Implement our own locking mechanism, e.g. using an additional draft.blend.lock with some extra info in it if we need R/W locks.
Use systems solutions, flock on UNIX, and LockFileEx on Windows.

Either solution would be implemented as new calls in BLI_fileops, e.g.

bool BLI_flock_shared(cons int file_descriptor, const bool is_blocking), bool BLI_flock_exclusive(cons int file_descriptor, const bool is_blocking), bool BLI_flock_release(cons int file_descriptor).

These will work on file descriptors as returned by e.g. BLI_open().

For the records, here is a partial implementation of using system's file locking feature (Windows code is not working, Linux and OSX implementation seems fine): D17022.

UPDATE: After talking with @brecht, it seems that writing in temp file + atomically moving it to the actual `draft.blend` file would match the requirements for the draft project: * On UNIX, this works 'out of the box', processes having 'old' draft.blend open will keep using it, new openings will have the new version of the file. * On Windows, the atomic move of the temp file to its final name will fail if the current `draft.blend` is used by other processes. This is acceptable and can be worked around (e.g. by looping a reasonable amount of times over the move call). The only change from the Draft handling PoV is that it needs to have a way to report failures to write to the user, in the rare case this should still happen. Not a bad thing to have anyway. -------------------------- ORIGINAL Proposal: The Draft project will require several instances of Blender being able to safely access a same draft .blend file, either for reading or writing. This could be achieved in two ways: - Implement our own locking mechanism, e.g. using an additional `draft.blend.lock` with some extra info in it if we need R/W locks. - Use systems solutions, `flock` on UNIX, and `LockFileEx` on Windows. Either solution would be implemented as new calls in `BLI_fileops`, e.g. `bool BLI_flock_shared(cons int file_descriptor, const bool is_blocking)`, `bool BLI_flock_exclusive(cons int file_descriptor, const bool is_blocking)`, `bool BLI_flock_release(cons int file_descriptor)`. These will work on file descriptors as returned by e.g. `BLI_open()`. For the records, here is a partial implementation of using system's file locking feature (Windows code is not working, Linux and OSX implementation seems fine): [D17022](https://archive.blender.org/developer/D17022).

Bastien Montagne commented

2023-01-17 12:03:41 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Bastien Montagne commented

2023-01-17 12:03:41 +01:00

Added subscribers: @mont29, @dfelinto, @JulianEisel, @brecht

Brecht Van Lommel commented

2023-01-17 15:33:24 +01:00

I'm not sure we should be using file locking. System specific solutions only work in some cases, not for network drives for example, so at a minimum we'd need our own solution. But then you need to deal with cases like a lock file that hangs around after a crash.

Personally I would save every brush in its own .blend file in a draft/ folder rather than a draft.blend, and maybe check file modified dates to prevent conflicts.

I'm not sure we should be using file locking. System specific solutions only work in some cases, not for network drives for example, so at a minimum we'd need our own solution. But then you need to deal with cases like a lock file that hangs around after a crash. Personally I would save every brush in its own .blend file in a draft/ folder rather than a draft.blend, and maybe check file modified dates to prevent conflicts.

Brecht Van Lommel commented

2023-01-17 18:14:12 +01:00

So there was some confusion here. I assumed that the purpose of file locking was to somehow show the asset as locked in the Blender UI, to prevent a user from editing it in the multiple Blender instances. However this is not planned, and it seems acceptable to not prevent such conflicts.

The purpose here was to atomically overwrite draft.blend while another Blender application has it open for reading, which can instead by done by moving another file over the existing file. The one caveat is that on Windows this may fail when another application has the file open, which could be solved by retries of move command for the rare cases this happens.

So there was some confusion here. I assumed that the purpose of file locking was to somehow show the asset as locked in the Blender UI, to prevent a user from editing it in the multiple Blender instances. However this is not planned, and it seems acceptable to not prevent such conflicts. The purpose here was to atomically overwrite `draft.blend` while another Blender application has it open for reading, which can instead by done by moving another file over the existing file. The one caveat is that on Windows this may fail when another application has the file open, which could be solved by retries of move command for the rare cases this happens.

Bastien Montagne commented

2023-01-17 18:24:52 +01:00

@brecht thanks, updated the task, will keep it open as part of the Draft project for the time being.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Add File Locking Support to BLI_fileops #103935