BAM support for Compressed blendfiles #46996

New Issue

Gabriel Caraballo · 2015-12-16T15:16:28+01:00

Gabriel Caraballo commented

2015-12-16 15:16:28 +01:00

The current design is the following according to @ideasman42

Looked into this and having compressed files locally - which need to be uncompressed to make a few edits - then recompress,
Its going to slow down local operations too much.
Instead, on the server side, bam can just not compress the files in the first place.

The packer can ensure all files are uncompressed before sending.

I found some problems with that approach:

Problem
If I save a compressed Blend file ¿How should BAM respond when I try to commit it? ¿Should BAM prevent such operation, or should uncompress the file directly, despite the user decision?

If we want to keep the file compressed for the user but rather uncompress it only for file transference the workflow is the following:

Commit new Compressed Blend
- BAM marks the .blend as compressed
- BAM uncompress it
- BAM sends the file to BAM Server, with the "compress" mark
- BAM Server compress the file again and stores it on SVN
Checkout a Compressed Blend
- BAM Server uncompress the file from SVN
- BAM Server sends the uncompressed file to BAM, with the "compressed" mark
- BAM compress the file?
Status of a compressed file
- BAM uncompress the file?
- BAM check the UUID/Hash

Using uncompressed blends on local BAM session but compressed on BAM Server

Commit new Compressed Blend
- BAM return Error
Commit new Uncompressed Blend
- BAM sends the file to BAM Server
- BAM Server stores it on SVN
Checkout a Compressed Blend
- BAM Server uncompress the file from SVN
- BAM Server sends the uncompressed file to BAM
Status of a compressed file
- BAM check the UUID/Hash
Commit modified compressed (on server) Blend
- BAM sends the (uncompressed) file to BAM Server
- BAM Server knows somehow that the file should be compressed
- BAM Server stores the file compressed on SVN

The problem with this approach is that you cant compress (nor uncompress) the Blend file from a BAM session.

Workflow of BAM overriding user compressed option:

Commit new Compressed Blend
- BAM uncompress it
- BAM Server stores it on SVN
Checkout a Compressed Blend
- BAM Server uncompress the file from SVN if compressed
- BAM Server sends the uncompressed file to BAM

Since I can access the SVN repo directly, if my SVN already have compressed BAM files, every time I checkout and commit them the compress setting will be lost.

As you see you can't go around the compression without overriding user decisions.

With this approach a user will be commiting a compressed file to SVN and if someone edited it using BAM the file (and all linked files) will be suddenly uncompressed. That is dangerous because after some time a big project (like Kiribati) will grow from 100Gb to 1Tb of size, and will not fit on the standard 1TB workstations discs.

Proposal

What I propose is to let the user decide what is the best for her/him. If there is a problem with BAM taking too much to commit the user will just stop using compression (if he/she can effort it).
BAM pack could print warnings every time it founds a compressed file, to give the user a clue about why it takes too much time.

Commit new Compressed Blend
- BAM uncompress it, edit it, compress again
- BAM Server stores it on SVN
Checkout a Compressed Blend
- BAM Server sends the file
- BAM uncompress it, edit it, compress again

This way the server don't know and don't care if the file is compressed or not, the Blend file is consistent for BAM and SVN users, and more importantly, the user can choose con use compression or not according her/his needs.

The current design is the following according to @ideasman42 > Looked into this and having compressed files locally - which need to be uncompressed to make a few edits - then recompress, > Its going to slow down local operations too much. > Instead, on the server side, bam can just not compress the files in the first place. The packer can ensure all files are uncompressed before sending. I found some problems with that approach: **Problem** If I save a compressed Blend file ¿How should BAM respond when I try to commit it? ¿Should BAM prevent such operation, or should uncompress the file directly, despite the user decision? If we want to keep the file compressed for the user but rather uncompress it only for file transference the workflow is the following: - Commit new Compressed Blend - BAM marks the .blend as compressed - BAM uncompress it - BAM sends the file to BAM Server, with the "compress" mark - BAM Server compress the file again and stores it on SVN - Checkout a Compressed Blend - BAM Server uncompress the file from SVN - BAM Server sends the uncompressed file to BAM, with the "compressed" mark - BAM compress the file? - Status of a compressed file - BAM uncompress the file? - BAM check the UUID/Hash Using uncompressed blends on local BAM session but compressed on BAM Server - Commit new Compressed Blend - BAM return Error - Commit new Uncompressed Blend - BAM sends the file to BAM Server - BAM Server stores it on SVN - Checkout a Compressed Blend - BAM Server uncompress the file from SVN - BAM Server sends the uncompressed file to BAM - Status of a compressed file - BAM check the UUID/Hash - Commit modified compressed (on server) Blend - BAM sends the (uncompressed) file to BAM Server - BAM Server knows somehow that the file should be compressed - BAM Server stores the file compressed on SVN The problem with this approach is that you cant compress (nor uncompress) the Blend file from a BAM session. Workflow of BAM overriding user compressed option: - Commit new Compressed Blend - BAM uncompress it - BAM Server stores it on SVN - Checkout a Compressed Blend - BAM Server uncompress the file from SVN if compressed - BAM Server sends the uncompressed file to BAM Since I can access the SVN repo directly, if my SVN already have compressed BAM files, every time I checkout and commit them the compress setting will be lost. As you see you can't go around the compression without overriding user decisions. With this approach a user will be commiting a compressed file to SVN and if someone edited it using BAM the file (and all linked files) will be suddenly uncompressed. That is dangerous because after some time a big project (like Kiribati) will grow from 100Gb to 1Tb of size, and will not fit on the standard 1TB workstations discs. **Proposal** What I propose is to let the user decide what is the best for her/him. If there is a problem with BAM taking too much to commit the user will just stop using compression (if he/she can effort it). BAM pack could print warnings every time it founds a compressed file, to give the user a clue about why it takes too much time. - Commit new Compressed Blend - BAM uncompress it, edit it, compress again - BAM Server stores it on SVN - Checkout a Compressed Blend - BAM Server sends the file - BAM uncompress it, edit it, compress again This way the server don't know and don't care if the file is compressed or not, the Blend file is consistent for BAM and SVN users, and more importantly, the user can choose con use compression or not according her/his needs.

Gabriel Caraballo commented

2015-12-16 15:16:28 +01:00

Changed status to: 'Open'

Gabriel Caraballo commented

2015-12-16 15:16:28 +01:00

Added subscribers: @GabrielCaraballo, @ideasman42

Campbell Barton commented

2015-12-16 15:25:14 +01:00

I think it doesn't have to be so complicated.

Bam has local cache - this is a directory to store data so you dont need to re-download the same blend files every time you checkout files.

I think the local cache should always be uncompressed, otherwise doing a local checkout will be very slow.

If the user saves a compressed Blend file, we can silently uncompress when storing in the local-cache (when they commit it). But when they check it out again, it wont be compressed.

On the other hand - we could support compressed (track which files are compressed, re-compress based on this info). I just don't see this as very valuable since it adds quite some overhead to performing edits on the blend file - which is needed for Bam's typical usage.

I realize it can be nice to let the user choose - but in practice users will have this set and forget about it, them BAM will run very slow and they won't want to use it.

I think it doesn't have to be so complicated. Bam has `local cache` - this is a directory to store data so you dont need to re-download the same blend files every time you checkout files. I think the local cache should always be uncompressed, otherwise doing a local checkout will be very slow. If the user saves a compressed Blend file, we can silently uncompress when storing in the local-cache (when they commit it). But when they check it out again, it wont be compressed. On the other hand - we _could_ support compressed (track which files are compressed, re-compress based on this info). I just don't see this as very valuable since it adds quite some overhead to performing edits on the blend file - which is needed for Bam's typical usage. ---- I realize it can be nice to let the user choose - but in practice users will have this set and forget about it, them BAM will run very slow and they won't want to use it.

Gabriel Caraballo commented

2015-12-16 16:34:48 +01:00

Maybe we can make a deal, adding a setting on BAM Client called "ALLOW_COMPRESSED_BLENDS" with a default on False.

If ALLOW_COMPRESSED_BLENDS == False
BAM client will uncompress downloaded files, but will not compress it again.

If ALLOW_COMPRESSED_BLENDS == True
BAM client will uncompress downloaded files, and will compress it again.

Something like (from https://developer.blender.org/diffusion/BAM/browse/master/bam/blend/blendfile.py;28429f297bc7abc3a9e94f8a179ab9df3b25b9d4$166 ):

    def close(self):
        """
        Close the blend file
        writes the blend file to disk if changes has happened
        """
        if not self.is_modified:
            self.handle.close()
        else:
            handle = self.handle
            if self.is_compressed:
                if config['ALLOW_COMPRESSED_BLENDS']:                     ######## If allowed, recompress it
                    log.debug("close compressed blend file")
                    handle.seek(os.SEEK_SET, 0)
                    log.debug("compressing started")
                    fs = gzip.open(self.filepath_orig, "wb")
                    data = handle.read(FILE_BUFFER_SIZE)
                    while data:
                        fs.write(data)
                        data = handle.read(FILE_BUFFER_SIZE)
                    fs.close()
                    log.debug("compressing finished")
                else:                                                      ####### If not allowed just unset as compressed
                    self.is_compressed = False
                

         handle.close()

If SVN already have compressed files BAM will be able to handle them, but will not compress the again.
If the user adds a compressed file BAM will uncompress it on cache, and send it to the Server uncompressed, the user file will remain compressed (will slow down some local checks a little bit, we can Warn the user about that), but next time the file is downloaded from server will be uncompressed.

Maybe we can make a deal, adding a setting on BAM Client called "ALLOW_COMPRESSED_BLENDS" with a default on False. If ALLOW_COMPRESSED_BLENDS == False BAM client will uncompress downloaded files, but will not compress it again. If ALLOW_COMPRESSED_BLENDS == True BAM client will uncompress downloaded files, and will compress it again. Something like (from https://developer.blender.org/diffusion/BAM/browse/master/bam/blend/blendfile.py;28429f297bc7abc3a9e94f8a179ab9df3b25b9d4$166 ): ``` def close(self): """ Close the blend file writes the blend file to disk if changes has happened """ if not self.is_modified: self.handle.close() else: handle = self.handle if self.is_compressed: if config['ALLOW_COMPRESSED_BLENDS']: ######## If allowed, recompress it log.debug("close compressed blend file") handle.seek(os.SEEK_SET, 0) log.debug("compressing started") fs = gzip.open(self.filepath_orig, "wb") data = handle.read(FILE_BUFFER_SIZE) while data: fs.write(data) data = handle.read(FILE_BUFFER_SIZE) fs.close() log.debug("compressing finished") else: ####### If not allowed just unset as compressed self.is_compressed = False handle.close() ``` If SVN already have compressed files BAM will be able to handle them, but will not compress the again. If the user adds a compressed file BAM will uncompress it on cache, and send it to the Server uncompressed, the user file will remain compressed (will slow down some local checks a little bit, we can Warn the user about that), but next time the file is downloaded from server will be uncompressed.

Bastien Montagne commented

2015-12-16 20:57:08 +01:00

Added subscriber: @mont29

Sybren A. Stüvel commented

2019-04-16 10:26:51 +02:00

Added subscriber: @dr.sybren

Sybren A. Stüvel commented

2019-04-16 10:26:51 +02:00

Changed status from 'Open' to: 'Archived'

Sybren A. Stüvel closed this issue

2019-04-16 10:26:51 +02:00

Sybren A. Stüvel self-assigned this 2019-04-16 10:26:51 +02:00

Sybren A. Stüvel commented

2019-04-16 10:26:51 +02:00

Superseeded by BAT?

BAM has been superseded by Blender Asset Tracer (BAT). BAT supports many of the features of BAM, and is actively being developed. All open tasks regarding BAM are closed, and the project will be set to read-only. If you use BAM in your own project, please consider migrating to BAT.

# Superseeded by BAT? BAM has been superseded by [Blender Asset Tracer (BAT)](https://developer.blender.org/project/profile/79/). BAT supports many of the features of BAM, and is actively being developed. All open tasks regarding BAM are closed, and the project will be set to read-only. If you use BAM in your own project, please consider migrating to BAT.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

BAM support for Compressed blendfiles #46996

Superseeded by BAT?