The goal of this project is to implement a new Point Cache system based on the Alembic library and unify the various caching methods used in Blender.
This is a prerequisite for further work on particle systems. Having a working Alembic exporter/importer will also greatly benefit integration of Blender into larger pipelines.
- Point cache is used for most physics simulations, but does not include mesh caching. For this there is a separate Mesh Cache modifier, using mdd or pc2 file formats, with its own limiations. Using a unified point cache system would help scene management.
- Alembic, unlike the current point cache binary format, supports arbitrary data attributes. This is very important for flexible simulations and user-defined data in particles and mesh data (vertex groups, custom data layers, etc.).
- Point cache is limited to fixed-size data sets. This is sufficient for many simulations which only deform geometry (cloth, hair), but for particles and constructive modifier caching variable point data sets are important.
- Compression in the point cache file format leaves a lot to be desired. Alembic is specifically designed to have a small memory/disk space footprint and make good use of threading for fast export/import.
- Point cache binary format is unique to Blender. While it is fairly simple to interpret, it does not facilitate import/export and integration into larger pipeline workflows. Alembic on the other hand is a commonly used tool and can leverage a large developer community.
- Settings for caches are object-centric, each simulation writes its own separate cache files. Baking all caches is simply a loop over all objects, which is not very efficient. Having scene-level settings and operators would make Alembic export/import easier.
- Keep basic cache settings on Scene level:
- "Disk Cache" vs. "Memory Cache": This could be reframed as having cache file output by default and use ''packing'' in .blend files like we do with images.
- Output directory setting for cache files (with nice defaults)
- Frame ranges
- Automatic cache recording settings (see below)
- Optional overrides on object/sim level If deviating settings are needed for individual simulation caches. By default it should not be necessary to tweak each cache individually.
- Question: How do we handle automatic caching? Current approach is to automatically record point cache when stepping through frames, unless the cache is explicitly baked. This breaks easily when jumping over frames and or scrubbing in the timeline, because simulations need a contiguous ordered frame range for correct results. Does this work well enough to justify keeping it? Are there alternatives to this behavior? One suggestion was to make simulations independent from the animation playback by default, but this makes it difficult to sync simulation results to keyframe-animated objects and to record valid caches on-the-fly.
- Question: Do we really need multiple caches for each object? This feature adds a list of point caches for every object, only the active cache is used, the others are basically a version control system. Could this be moved to the scene level? The problem with this way of "versioning" is that there is really no way to roll back to a previous version, other than comparing the results without any way to revert settings.
Directory and File Names
How should directories and file naming work?
Current naming scheme works per object and uses a combination of output directory + filename:
- Directory is chosen in this order:
- explicit path if specified by the user
- or constructed as //blendcache_<blend file name>/ (if .blend is saved and thus relative paths work)
- or using a temp folder like <tmp folder>/blendcache/
- File name is generated as <basename>_<frame number>_<cache index>.bphys
- <basename> can be specified by the user or otherwise the ID datablock name (usually Object) is used
- <cache index> is generated automatically by default (-1), but needed if multiple caches exist in the same ID block.
Frames, Times and Sample Indices
There are 3 different modes of interpreting "time" that need to be managed in a clean way:
- Blender usually defines data in terms of frame. Any notion of "time" is secondary, defined by the frames-per-second (fps) settings in the scene and additional start/end frames. Some simulations also add their own interpretation of time on top of this, by using time scale factors (e.g. smoke sim time scale options), although these factors are mostly for the internal simulation time step and don't have much consequence for mapping sample times.
- Alembic samples can be accessed
While it would be possible to largely ignore time sampling for Blender cache files and simply read each cached sample as a frame, it is probably more reliable and flexible to calculate time values as defined by the scene render range and fps settings. This would allow using externally generated Alembic files which use a different sampling rate than the current scene and map them into the scenes frame range.
Ultimately it could be very useful to integrate cached data into the NLA editor as a track. At that point the direct sample<->frame relationship that might still exist for cached data from the same scene is useless anyway.
- Output directory is by default specified on scene level, using the same rules as before.
- Alembic cache files don't store files for each individual frame, instead one file for the whole cached frame range is used
- 2 possible behaviors:
- Each cached simulation/modifier creates its own file in the cache folder
- Combine caches in a single scene cache file. This is more in line with standard Alembic use, but makes it more difficult to cache objects separately
- In both cases identifier names for cache data are a problem. This is already the case with current point caches: Renaming an object will sever the mapping to file names, leading to unused files on the disk and making re-bake necessary.
Almost all cacheable data is located somewhere in modifier stacks, so these require some special attention. While caching the final result of mesh modifier evaluation (final_dm) is obviously useful, we also want to support partial caching and layered evaluation of mesh stages and simulation.
- Add a "Point Cache" modifier for defining intermediate stages that can be cached. It would have almost no settings and just let users define fixed points, rather than caching every modifier (too much data) or relying on some heuristic of modifier "cost" to determine suitable stages.
- By default there is one virtual Cache modifier instance shown at the end of the stack. This does not have to be a real modifier, but just reflect caching of the final_dm result. Caching can be disabled entirely by disabling this modifier. [Is it preferable to leave this completely to the user and not do any modifier caching at all by default?]
|Base modifier stack with cache||Modifier stack with intermediate caching|
- How to treat multiple successive cache modifiers? Can/Should this be prevented, or just ignore redundant instances? Similarly for caches at the beginning of the stack where they are useless.
- Should these caches also handle internal data of simulation modifiers (smoke/fluid voxels, particles, hair, ...). Or do they only handle DerivedMesh caching and leave sim data to explicit cache instances elsewhere (physics tab) [I would prefer the latter to keep design clean and avoid nasty corner cases]
Caching vs. Export
These two cases may need to have somewhat different requirements in terms of necessary data attributes, sampling, etc. (while running the same backend input/output functions).
- Caching is an internal procedure. The data being written only has to be interpreted by Blender itself (although the files would largely follow Alembic schemas where applicable). Some data attributes (e.g. normals) might be omitted here, since reconstructing them is fairly cheap. [confirm?]
- Export is a one-way procedure to describe Blender data in a generic format for other software. All possible data attributes should be included to ensure compatible results.
- Scene stores global settings for caching
- Simulations and modifiers can have own settings if needed
- Cache data is separate from other DNA by default, but can be packed into blend files like images
- Point Cache is a thin API, using Alembic as a backend
- Different schema implementations (writer/reader) are created for the various DNA types and to handle interpolation (comparable to current PTCacheID)
- Scene writer/reader ties everything together for exporting the scene as a whole, possibly using object hierarchy (flattening scene structure for export would also be possible)
- Build instructions for Alembic and the development branch: http://wiki.blender.org/index.php/User:Phonybone/PointCache/BuildNotes
- Unsorted list of thoughts on Alembic implementation details: http://wiki.blender.org/index.php/User:Phonybone/PointCache/InitialThoughts
- Original dia file for the diagram: