Page MenuHome

BAM support for Compressed blendfiles
Closed, ArchivedPublic

Description

The current design is the following according to @Campbell Barton (campbellbarton)

Looked into this and having compressed files locally - which need to be uncompressed to make a few edits - then recompress,
Its going to slow down local operations too much.
Instead, on the server side, bam can just not compress the files in the first place.

The packer can ensure all files are uncompressed before sending.

I found some problems with that approach:

Problem
If I save a compressed Blend file ¿How should BAM respond when I try to commit it? ¿Should BAM prevent such operation, or should uncompress the file directly, despite the user decision?

If we want to keep the file compressed for the user but rather uncompress it only for file transference the workflow is the following:

  • Commit new Compressed Blend
    • BAM marks the .blend as compressed
    • BAM uncompress it
    • BAM sends the file to BAM Server, with the "compress" mark
    • BAM Server compress the file again and stores it on SVN
  • Checkout a Compressed Blend
    • BAM Server uncompress the file from SVN
    • BAM Server sends the uncompressed file to BAM, with the "compressed" mark
    • BAM compress the file?
  • Status of a compressed file
    • BAM uncompress the file?
    • BAM check the UUID/Hash

Using uncompressed blends on local BAM session but compressed on BAM Server

  • Commit new Compressed Blend
    • BAM return Error
  • Commit new Uncompressed Blend
    • BAM sends the file to BAM Server
    • BAM Server stores it on SVN
  • Checkout a Compressed Blend
    • BAM Server uncompress the file from SVN
    • BAM Server sends the uncompressed file to BAM
  • Status of a compressed file
    • BAM check the UUID/Hash
  • Commit modified compressed (on server) Blend
    • BAM sends the (uncompressed) file to BAM Server
    • BAM Server knows somehow that the file should be compressed
    • BAM Server stores the file compressed on SVN

The problem with this approach is that you cant compress (nor uncompress) the Blend file from a BAM session.

Workflow of BAM overriding user compressed option:

  • Commit new Compressed Blend
    • BAM uncompress it
    • BAM Server stores it on SVN
    • Checkout a Compressed Blend
    • BAM Server uncompress the file from SVN if compressed
    • BAM Server sends the uncompressed file to BAM

Since I can access the SVN repo directly, if my SVN already have compressed BAM files, every time I checkout and commit them the compress setting will be lost.

As you see you can't go around the compression without overriding user decisions.

With this approach a user will be commiting a compressed file to SVN and if someone edited it using BAM the file (and all linked files) will be suddenly uncompressed. That is dangerous because after some time a big project (like Kiribati) will grow from 100Gb to 1Tb of size, and will not fit on the standard 1TB workstations discs.

Proposal

What I propose is to let the user decide what is the best for her/him. If there is a problem with BAM taking too much to commit the user will just stop using compression (if he/she can effort it).
BAM pack could print warnings every time it founds a compressed file, to give the user a clue about why it takes too much time.

  • Commit new Compressed Blend
    • BAM uncompress it, edit it, compress again
    • BAM Server stores it on SVN
  • Checkout a Compressed Blend
    • BAM Server sends the file
    • BAM uncompress it, edit it, compress again

This way the server don't know and don't care if the file is compressed or not, the Blend file is consistent for BAM and SVN users, and more importantly, the user can choose con use compression or not according her/his needs.

Details

Type
Design

Event Timeline

I think it doesn't have to be so complicated.

Bam has local cache - this is a directory to store data so you dont need to re-download the same blend files every time you checkout files.

I think the local cache should always be uncompressed, otherwise doing a local checkout will be very slow.

If the user saves a compressed Blend file, we can silently uncompress when storing in the local-cache (when they commit it). But when they check it out again, it wont be compressed.

On the other hand - we _could_ support compressed (track which files are compressed, re-compress based on this info). I just don't see this as very valuable since it adds quite some overhead to performing edits on the blend file - which is needed for Bam's typical usage.


I realize it can be nice to let the user choose - but in practice users will have this set and forget about it, them BAM will run very slow and they won't want to use it.

Maybe we can make a deal, adding a setting on BAM Client called "ALLOW_COMPRESSED_BLENDS" with a default on False.

If ALLOW_COMPRESSED_BLENDS == False
BAM client will uncompress downloaded files, but will not compress it again.

If ALLOW_COMPRESSED_BLENDS == True
BAM client will uncompress downloaded files, and will compress it again.

Something like (from https://developer.blender.org/diffusion/BAM/browse/master/bam/blend/blendfile.py;28429f297bc7abc3a9e94f8a179ab9df3b25b9d4$166 ):

def close(self):
    """
    Close the blend file
    writes the blend file to disk if changes has happened
    """
    if not self.is_modified:
        self.handle.close()
    else:
        handle = self.handle
        if self.is_compressed:
            if config['ALLOW_COMPRESSED_BLENDS']:                     ######## If allowed, recompress it
                log.debug("close compressed blend file")
                handle.seek(os.SEEK_SET, 0)
                log.debug("compressing started")
                fs = gzip.open(self.filepath_orig, "wb")
                data = handle.read(FILE_BUFFER_SIZE)
                while data:
                    fs.write(data)
                    data = handle.read(FILE_BUFFER_SIZE)
                fs.close()
                log.debug("compressing finished")
            else:                                                      ####### If not allowed just unset as compressed
                self.is_compressed = False
            

     handle.close()

If SVN already have compressed files BAM will be able to handle them, but will not compress the again.
If the user adds a compressed file BAM will uncompress it on cache, and send it to the Server uncompressed, the user file will remain compressed (will slow down some local checks a little bit, we can Warn the user about that), but next time the file is downloaded from server will be uncompressed.

Bastien Montagne (mont29) changed Type from Bug to Design.Dec 16 2015, 8:57 PM
Bastien Montagne (mont29) triaged this task as Normal priority.
Sybren A. Stüvel (sybren) closed this task as Archived.
Sybren A. Stüvel (sybren) claimed this task.

Superseeded by BAT🦇

BAM has been superseded by Blender Asset Tracer (BAT). BAT supports many of the features of BAM, and is actively being developed. All open tasks regarding BAM are closed, and the project will be set to read-only. If you use BAM in your own project, please consider migrating to BAT.