Compressing blendfiles can help save a lot of disk space, but the slowdown while loading and saving is a major annoyance.
Currently Blender uses Zlib (aka gzip aka Deflate) for compression, but there are now several more modern algorithms that outperform it in every way.
In this patch, I decided for Zstandard aka Zstd for several reasons:
- It is widely supported, both in other programs and libraries as well as in general-purpose compression utilities on Unix
- It is extremely flexible - spanning several orders of magnitude of compression speeds depending on the level setting.
- It is pretty much on the Pareto frontier for all of its configurations (meaning that no other algorithm is both faster and more efficient).
One downside of course is that older versions of Blender will not be able to read these files, but one can always just re-save them without compression or run the file through unzstd.
- Versioning when loading older files: Currently, the Compression Method setting will be set to None when loading a gzipped file from an unpatched Blender (the reverse works).
- Compression Level setting: I don't know if we want to support this, but I can see uses (e.g. turning up the compression before sending the file to someone). As a workaround, right now you can just run the uncompressed blendfile though the standalone zstd at any setting and it will load.
- MAYBE: Other algorithms? Something like LZ4 might even be interesting for in-memory undos.
- Most of the build_files changes are untested - it builds on my Arch machine, that's all I can say so far
- CMake scripts for manually/statically building libzstd are missing.
Some benchmarks on a random massive file I had lying around:
|Method||Size on Disk||Time to load||Time to save|
|None||863MB||1.449 sec||0.333 sec|
|Zlib||476MB||3.426 sec||14.46 sec|
|Zstd||455MB||1.505 sec||1.630 sec|
All I/O is to/from ramdisk. The time to load also includes a few linked files iirc, so just look at the differences instead of the absolute times.
To validate that the implementation isn't bottlenecked somewhere, I ran the standalone zstd on the uncompressed file and it took 1.365sec, which matches almost exactly with the measurement difference above.