Page MenuHome

Blenloader Decentralization
Confirmed, NormalPublicDESIGN

Description

This is part of a larger refactoring towards a more extensible architecture in Blender (T75724).

A work in progress implementation of the proposal is in the blenloader-decentralization branch.

Steps that can be done now:

  • Commit API to master.
  • Update writefile.c to use the API (except for very few special cases, search for writer->wd).
  • Update direct linking in readfile.c to use the API (except for a few cases, search for reader->fd).
  • Implement blendRead and blendWrite callbacks for modifiers (the fluid modifier still has some special handling; haven't changed pointcache reading yet, to avoid merge conflicts with another patch of mine).
  • Update lib linking in readfile.c to use the API.
  • Update expand in readfile.c to use the API.

API

First, there needs to be an API that allows specifying how .blend file I/O works for a particular piece of data (e.g. a data block, a node, a modifier, ...). This API is proposed in BLO_read_write.h (source code). Please read the documentation in this header to see how the API works.

Partial copy of the documentation:

This file contains an API that allows different parts of Blender to define what data is stored
in .blend files.

Four callbacks have to be provided to fully implement .blend I/O for a piece of data. One of
those is related to file writing and three for file reading. Reading requires multiple
callbacks, due to the way linking between files works.

Brief description of the individual callbacks:
- Blend Write: Define which structs and memory buffers are saved.
- Blend Read Data: Loads structs and memory buffers from file and updates pointers them.
- Blend Read Lib: Updates pointers to ID data blocks.
- Blend Expand: Defines which other data blocks should be loaded (possibly from other files).

Each of these callbacks uses a different API functions.
Using the API

Fortunately, we don't have to start using the API everywhere at the same time. Instead we can gradually move functions from writefile.c and readfile.c to e.g. blenkernel. Furthermore, callbacks can be added to ModifierTypeInfo and IDTypeInfo (and later bNodeType).

Modifiers

/* Is called when the modifier is written to a file. The modifier data struct itself is written
  * already.
  *
  * This method should write any additional arrays and referenced structs that should be
  * stored in the file.
  */
void (*blendWrite)(struct BlendWriter *writer, const struct ModifierData *md);

/* Is called when the modifier is read from a file.
  *
  * It can be used to update pointers to arrays and other structs. Furthermore, fields that have
  * not been written (e.g. runtime data) can be reset.
  */
void (*blendRead)(struct BlendDataReader *reader, struct ModifierData *md);

In the branch I implemented these callbacks for almost all modifiers (e.g. hook modifier).

Data Blocks


typedef void (*IDTypeBlendWriteFunction)(struct BlendWriter *writer, struct ID *id, const void *id_address);
typedef void (*IDTypeBlendReadDataFunction)(struct BlendDataReader *reader, struct ID *id);
typedef void (*IDTypeBlendReadLibFunction)(struct BlendLibReader *reader, struct ID *id);
typedef void (*IDTypeBlendExpandFunction)(struct BlendExpander *expander, struct ID *id);

/**
 * Write all structs that should be saved in a .blend file.
 */
IDTypeBlendWriteFunction blend_write;

/**
 * Update pointers for all structs directly owned by this data block.
 */
IDTypeBlendReadDataFunction blend_read_data;

/**
 * Update pointers to other id data blocks.
 */
IDTypeBlendReadLibFunction blend_read_lib;

/**
 * Specify which other id data blocks should be loaded when the current one is loaded.
 */
IDTypeBlendExpandFunction blend_expand;

In the branch I implemented these callbacks in action.cand key.c already (action, key).

Misc

In order to implement the callbacks mentioned above, I also had to move the I/O code of some other structures out of blenloader. Namely I already moved AnimData, CurveMapping, CurveProfile, IDProperty and some more. It takes quite some time, but it's mostly pattern matching. Usually, I just copy the code from e.g. readfile.c and then update the code line by line.

Event Timeline

Jacques Lucke (JacquesLucke) changed the task status from Needs Triage to Confirmed.May 3 2020, 4:51 PM
Jacques Lucke (JacquesLucke) created this task.
Jacques Lucke (JacquesLucke) changed the subtype of this task from "Report" to "Design".

Generally fine with this proposal.

I’m missing a more/better structured planning though, with some clear milestones etc.

Another question worse investigating is regarding the lib linking and expand processes. Those two could be entirely abstracted and performed through generic lib_query “foreach_id” system, which would save us adding two of the three callbacks currently needed for read code. Main issue/work here would be to add handling of deprecated DNA ID Pointers to lib_query, but that should not be a huge work (we already support conditional handling of some pointers for UI data-block pointers, doing the same thing for deprecated ones should be fairly straightforward).

I agree that lib linking and expanding can and should be done using the generic foreach_id system. However, I don't think that we should do both refactorings at the same time. If we decide to do both refactorings separately, the order probably does not matter too much. The difference is that I could start with the decentralization immediately, while you'd probably need to spend some time to make lib linking and expand work with the foreach_id system.

We can also decide that we only decentralize file writing and direct data reading at first.

As for milestones, I'd say that a first milestone would be to get all modifier specific code out of readfile.c.

Once the API is committed, I could also start using it in readfile.c and writefile.c directly. This would simplify the actual decentralization later on.

If we agree on the API, this should be fairly straight forward to do.

My main point is that if we use libquery for linking and expanding, we do not need those two callbacks in IDTypeInfo then. Feels a bit useless and very noisy to add, then remove two callbacks from such a struct.

I do not know how urgent it is for you to move forward on this topic, if needed i can spend a few hours to check how feasible using libquery for this is…

For the next code quality day, I'd like to follow the checklist I added to the main post. Is that ok?

I think updating lib linking and expand to use the new API is a code quality improvement by itself, independent on how we decide to implement it in the future.

I read in the weekly report that @Bastien Montagne (mont29) did some initial investigation to check if it is possible to use libquery for lib linking and expand. It seems to be not straight forward (which is also what I found). Let's discuss how to proceed with this topic, when I'm done with the tasks listed in the main post.

Yeah, am not super happy to have to go through those intermediate steps (since the IDTypeInfo liblink/expand callbacks will likely be removed soon-ish), but I don’t really plan to work on this usage of libquery for those in immediate future, so I'm fine with your plan.

Going ahead with this for the next code quality day sounds good then.