Fracture Modifier 2.8 port
Open, NormalPublic

Description

This is a summary of my design thoughts for porting the Fracture Modifier to 2.8
based on https://lists.blender.org/pipermail/bf-committers/2018-February/049150.html and following.

It is intended to open a hopefully fruitful discussion about what to consider during the Fracture modifier port and even the cache design.

For now, I assume Alembic will be used as cache backend, like described in https://developer.blender.org/T54690
There duplis are mentioned to represent streamed geometry in blender.

The Fracture Modifier should hold internally some representation of streamed geometry too, but return a regular mesh datablock, so it can be used as a regular modifier in the stack.

Using duplis here on object level would fix the geometry streaming to the base object thus being like always the first modifier.
But it is useful to have other modifiers before the FM, to be able to fracture this content and use it nondestructively.
Generally the geometry cache just shall avoid continouous refractures which are computationally expensive.
For example this is bmesh boolean and bisect for less suitable / nonmanifold geometry.

Modularization of Geometry Generation

Maybe it is wise decouple the actual "cutting" method ? For example just feed in raw voronoi cells, "process" them and write result to alembic geometry cache.
Process is boolean or bisect or maybe something else.
Even more generally spoken, decouple geometry generation from storage.
The generation process can be nonvoronoi too, like island decomposition or even vdb-based volume fracturing.
(just to make sure design is modular and extendible here)

Base / previous modifier Geometry → geometry generator operator (voronoi shatter (+intersect provider), island decompose, packing separate objects) → FM

Modularization of Cache

It is important in case we decide to have different backends / or backend has major update.
Suggestion would be to split the cache logically to a Geometry Cache and a Transform Cache.

Geometry Cache

The storage engine should be encapsulated in a "geometry storage provider".
(can be abc, can be different format for static geometry atleast, just to decouple it)
And how to keep geometry "parts" apart ? natural mesh islands, user selectable facemaps, generated facemaps
vertexgroups (something which isnt awfully slow, since it will be needed to advance the simulation)

storage provider needs to be dynamic, as in must be able to store more geometry (or less) per frame, without doubling the existing geometry, which is there on subsequent frame ranges
as in change contents at change events, but keep indifferent pieces only once. kinda smart geometry storage (vs Alembic dumping all in each frame)
Thats why each cache item should have an own frame range, but additionally we quickly need to query what is on frame x (lookup structure, update during simulation, slower at sim, faster playback)
for faster sim maybe option for "dumb/ big storage" ? and "smartify" in an additional post step to reduce persistent file footprint

Transform Cache

The simulation cache should be a "transformation provider", which can transform the geometry parts. for rigidbody this has been the traditional blender pointcache,
but each geometry item could get an own loc/rot or even transform matrix (and even more custom data, velocity, acceleration, etc)
But as transform changes over time, this needs to be a sequence per item and its valid frame range.
For the dynamic fracture case, make sure to take over last old transform to new "child" shards, setting a suitable start transform.
As backend we could have alembic here too, transformation provider(item) would write xform property data to the abc file.

So the abc file has as seen from FM a geometry provider part(store geometry once per update step, nothing inbetween...
prefracture is 1 initial step, dynamic are multiple) and a transform (sequence) provider

FM → geometry cache (untransformed) → abc

FM → transform sequence cache(per item, for frame range) → abc

FM → transformed geometry provider (cachestream) → Mesh

disabled FM, empty FM(before first fracture) → Mesh passthrough

Fitting in the simulation provider

Here we have the rigidbody bullet wrapper, decouple this from "Object" and object group in world.
The rigidbody world maybe needs to know contained objects, or even only the rigidbody parts....
since those are the cached items... shape per item is generated from geometry provider,
bullet updates loc/rot and feeds into transform provider (making matrix transform, writing to abc)
item could have as customdata an object name/ index + local object shard index.

FM → simulation provider → shape geometry, transform → rigidbody wrapper → bullet
bullet → rigidbody wrapper → transform update, rigidbody object → FM
FM → transformed geometry provider (mesh output),
FM → transform provider (item) (write to sequence); dynamic (new initial transform)
FM → geometry provider (item) (dynamic, new items)

Update (27.05.2018)

(just a rough outline)

typical fracturing case: pointcloud → voronoi (bbox) → bisect / boolean → shard
shard holds an own mesh, own customdata

(other cases, cutter plane based fracturing for example, but we get shards too there)

we only need indexes to polys for quick access (instead of storing parts of the mesh elsewhere like in mesh islands → vertices)
so:

struct MIsland {
	int polystart;
	int polycount;
	//it is a contiguous area of polys... well except for split shards !!!
}

so better keep an array of polys → can we use facemaps for it ? is Lookup quick enough ? like... get faces of map x... vertexgroup lookup was awfully slow before (2.7x) for example
would like to avoid a custom structure

or... just an int cd layer per poly... meshisland_index.... extract Meshisland structure from it at load time and store as runtime structure in FM ?
another layer (or 2) per island... rigidbody index in world ←→ which object / island is this rigidbody ?
to avoid searches... a lookup map ... now we have cache_index_map / cache_offset_map in the world
would rather give each rigidbody that "address" info directly (ob / island) and each island the rb index (another cd poly int layer)

The FM, holds a Mesh representing the current state of geometry (with current transforms, and maybe the start transform too)

this is a case for Alembic, Islands should be stored like objects are, + transforms over time... start transform is transform on start frame then

each island should have a defined range of validity, like frame 10 to 50 or so, so each shard is stored only once and is has valid transforms in that range

(each island could have some kind of ID, too.. a Name, an unique number, or both) → useful for later python exposure.. this is island X, this is island Y... etc

struct MeshIsland {
	int ID;
	int startframe, endframe;
	int polycount;
	int* poly;
	int rigidbody_index; //in the world
	//should be stored when the rigidbody is created, and the rigidbody gets current object index (from collection) and island index it belongs to
	//rigidbody will hold loc,rot
	//mi should make a matrix for transforming its verts from it (more easily)
	float [4][4] transform;

	float last_loc[3];
	float last_rot[4];
}
FractureModifierData {
	//ListBase islands; // maybe an array... since mostly this is a fixed number, but, it might grow in dynamic, but lookup still is faster for stuff to be removed

	MeshIsland **islands;
	int island_count;

	AbcWriter, AbcReader....
	Abcfilename...

	//if not cached yet... need some logic to check whether frame has been simulated
	writeIslands(frame) → do we have duplicates, if valid, do not write again, only write transform... if not valid anymore... delete island from current frame, if new, write geometry assign new id
	update islands listbase (append), need quick way to lookup islands to remove

	readIslands(frame) → if not valid, try to simulate this frame !!!
	else "stream" from Abc, transforming the existing islands (so store geometry in prefracture case only on startframe, with validity till endframe, also take care of start / endframe updates
	update start and end accordingly...

	if island_count different, rebuild mesh (better on validity events / changes), else just re-calc transform, in the modifier (un-apply old trans, re-apply new)

	Fracture operator will invoke the modifier once (or just trigger the actual fracturing code, the modifier should just update transforms + the mesh)
	Dynamic fracture events will trigger the fracture code too
	Fracturing will update the meshislands array and re-initialize the writer (to write new islands, new start transforms)
}

FM "queries" the rigidbody each frame and updates the matrix if necessary (keep lastloc, lastrot for quick comparision, to avoid matrix ops)
currently the FM geometry flickers / looks broken because the rb sim loop updates the geometry (a bad approach)

Proposal: remove the old Rigidbody system... replace by FM, each classical rigidbody could be an FM with 1 island
also handle constraints properly... cases: 1:1 1:n n:n... FM should handle this, but empties cant carry modifiers.... → use "empty mesh" here, hidden / not rendered, maybe with the axis shape of empty
in this case we need to be able to edit individual constraints (even if FM holds multiple, they just must not be objects !!!!, but pickable somehow

alternative, keep objectbased constraints, but its duplicate code...
further considerations: shard-space texture space... like duplis but for islands (since the object stays in place, only the mesh moves... for correct displacement, where the texture moves with each shard)

Details

Type
Design

whoops, this was automagically added to the codequest workboard... just wanted to tag it properly with "code quest"...

Using Alembic as a cache format has a big downside: it can't be updated. Alembic performs certain compression and deduplication operations while writing to the file, which disallows partial updates to an already-existing file. If we want to be able to read frames 1-100 from the cache and then write 101-200 to it, we need either a way to split those ranges into separate Alembic files, or not use Alembic.

hi,teacher!
The Fracture Modifier will support blender2.8?

we need either a way to split those ranges into separate Alembic files, or not use Alembic.

Well, or we just don't care and for baking always write the entire frame range. That's fine by me too.

Sybren A. Stüvel (sybren) triaged this task as Normal priority.

New proposal is to split the old Fracture Modifier into a geometry and simulation component.
Optionally the live postprocessing part (autohide, automerge) could be a separate modifier, too.

Fracture Modifier (FM):
It has a runtime Mesh Cache with poly customdata Layer which marks the shard index per poly.
An Object can have multiple FMs, where each FM will have a Refresh Operator. This operator will refresh the FM and all other FMs above this FM in the stack. For this purpose each FM needs to store its input mesh, and pass its output down to the next FM or the RBM, which will store it as their respective input.
Each FM will have own fracturing settings (pointsource, algorithm etc) and have optionally an ability to restrict the fracturing to certain pieces, for instance by specifying the indexes. Ideally those
could be picked from a visualization in the viewport. (Assign vertex colors, materials maybe ?)

Rigid Body Modifier (RBM):
It has a runtime geometry cache and a runtime motion data cache.
It will also have a Cache File with a reference to an Alembic file.
A Bake Operator will write the full geometry and the motiondata to the abc file, including "invalid" shards.
The concept of valid and invalid shards is explained below.
To handle dynamic fracture, the geometry runtime cache will just grow and contain a history, where each shard will appear once.
Each shard will have a validity range in frames. If the current frame is outside the range, the respective shard will not be simulated and not be rendered / displayed, which means it will not be part of the output mesh either.
If a fracture event happens, either manually invoked or during the sim, the affected shards will be fractured.
The end frame of those shards will be set to the current frame, and the start frame of their "children" will be also set to the current frame.
If the blend file is loaded or after a bake, if there are abc files present and valid for the RBM, their geometry and motion data will be used.
It needs to be ensured FMs above this baked RBM do not refresh their geometry. Can be disabled manually, or could be ensured automatically, by suppressing their execution internally with a refresh flag.
Constraint handling will happen in the RBM too (same way as its being done in the old FM, currently).
The rigidbody world group will only contain objects and there will be one pointcache read / write call per Object. Inside that call for each found RBM there will be an iteration over all valid shards. The motion data for them will be updated or read back. So the pointcache will only serve as facade for this RBM motion data cache.

Advantages:
Having a separate cache per RBM makes the concept of the cache index map in the world unnecessary. The order of the shards was subject to be corrected after loading a baked file if multiple FM objects and regular rigidbodies were involved. This would have implied to store the order inside the DNA or the ABC file either.
The writing to alembic will happen similarly to the export operator, but specifically for that particular object only. So it should be ensured the user can only scrub in either the ram cache or the fully written abc file.
Scrubbing in halfly-written abc files can be avoided this way.

Disadvantages:
Except for more rewrite work none at the moment :)