Page MenuHome

LZ4 Compression for blend file IO
Needs ReviewPublic

Authored by Campbell Barton (campbellbarton) on Feb 24 2019, 12:05 PM.

Details

Summary

This implements T56162, to add a fast alternative compression for blend file io.

  • LZ4 (latest stable from https://lz4.github.io/lz4) is included in extern/ or the system LZ4 can be used.
  • Compression has been changed from a boolean to an enum.

Initial Timing Tests

Tests with a file containing subdivided cubes (23 cubes, subdivided 8 times ~ 293216 faces), reading writing to a HDD (dropping caches to test for cold reads).

CompressionRead (cold)Read (warm)WriteSize
NONE19.62 sec9.20 sec13.97 sec2364M
ZLIB15.40 sec14.37 sec33.98 sec624M
LZ411.28 sec6.30 sec6.42 sec1166M

Note: peak memory usage isn't significantly different for read and write.


Possible Changes

  • Using G.fileflags for two compression types is odd, should be a separate member however this is a bigger refactor of file read/writing code.

    We could move these settings into a struct.
  • Wrap LZ4 w/ an API that hides details, similar to how zlib works.

TODO

  • Thumbnail extraction utility.

    It might be best to have a single implementation shared between Linux & Windows since Python doesn't support LZ4.
    • release/bin/blender-thumbnailer.py
    • release/windows/blendthumb/src/BlenderThumb.cpp.

Diff Detail

Repository
rBS Blender Staging
Branch
temp-blend-lz4
Build Status
Buildable 4677
Build 4677: arc lint + arc unit

Event Timeline

  • Only call BLI_open once, (reuse the file handle)
  • Use 4mb source buffer size (avoids calling decompress so much).
Campbell Barton (campbellbarton) edited the summary of this revision. (Show Details)
  • Add timing prints for file load/save
Harbormaster completed remote builds in B2984: Diff 13823.
  • Use size hint as recommended in LZ4 docs, gives minor speedup

release\windows\blendthumb\src\BlenderThumb.cpp will need to be updated as well, this code is peculiar, since it lives in the blender source tree, but it doesn't get compiled during the blender build (it requires both a 32 and 64 bit build which cmake doesn't like doing) so we keep binaries of this code in SVN.

  • Tune block size for compression level and remove redundant buffering stage.
  • Sync w/ master (applied some writefile.c changes to master).
  • readfile: tune block size for improved performance.
Campbell Barton (campbellbarton) edited the summary of this revision. (Show Details)

@LazyDodo (LazyDodo) thanks, included in the TODO.

Campbell Barton (campbellbarton) edited the summary of this revision. (Show Details)
Campbell Barton (campbellbarton) edited the summary of this revision. (Show Details)
Campbell Barton (campbellbarton) edited the summary of this revision. (Show Details)
Campbell Barton (campbellbarton) edited the summary of this revision. (Show Details)

From a first glance it seems your timings are bound on the speed of your disk. To get more accurate comparison use either m.2 SSD if you have it, or use ramdisk.
On a second glance i'm not sure why warm reading of uncompressed file will be slower than LZ4 compressed. Is either you don't have enough RAM to fit the entire uncompressed file, or uncompressed reading codepath is doping funky things.

CMakeLists.txt
350–353

I don't think this is a good thing to allow building blender which can't read own files.
Also, wording "best compression" is somewhat misleading.

source/blender/windowmanager/intern/wm_files.c
1325

Development-only thing?

2098

Guess this is just for development?

Harbormaster completed remote builds in B2990: Diff 13841.
  • correct typo
CMakeLists.txt
350–353

Yep, the option will probably go from the final patch. best was copypaste error from best LZMA line above.

source/blender/windowmanager/intern/wm_files.c
2098

Yes, will remove from final patch.

  • Merge 'master' into 'temp-blend-lz4'
  • Merge branch 'master' into temp-blend-lz4 a771fdb5dc4b868cda4a778bfc4f2670111815bf