This implements T56162, to add a fast alternative compression for blend file io.
- LZ4 (latest stable from https://lz4.github.io/lz4) is included in extern/ or the system LZ4 can be used.
- Compression has been changed from a boolean to an enum.
Initial Timing Tests
Tests with a file containing subdivided cubes (23 cubes, subdivided 8 times ~ 293216 faces), reading writing to a HDD (dropping caches to test for cold reads).
|Compression||Read (cold)||Read (warm)||Write||Size|
|NONE||19.62 sec||9.20 sec||13.97 sec||2364M|
|ZLIB||15.40 sec||14.37 sec||33.98 sec||624M|
|LZ4||11.28 sec||6.30 sec||6.42 sec||1166M|
Note: peak memory usage isn't significantly different for read and write.
- Using G.fileflags for two compression types is odd, should be a separate member however this is a bigger refactor of file read/writing code.
We could move these settings into a struct.
- Wrap LZ4 w/ an API that hides details, similar to how zlib works.
- Thumbnail extraction utility.
It might be best to have a single implementation shared between Linux & Windows since Python doesn't support LZ4.