Asset System (Back End)¶
The asset system back end implements the core runtime asset types and functionality: Assets representations, asset libraries, catalogs, asset storage management, asset library loading, catalog loading, etc. It is the technical foundation for the goal of making assets first class citizens in Blender. Public APIs make assets and their functionality available to the UI (and later on scripts).
It should at some point become possible to expand the asset system functionality. For example scripts could add custom asset library loading (e.g. to load libraries from own servers), new asset types, or own preview rendering pipelines.
Why is the Asset System Needed?¶
The asset project initially focused on delivering an Asset Browser to deal with ID data-blocks that were marked as assets. The File Browser was used as the base for this, but it started showing more and more issues, and the requirements changed: Now assets should become available anywhere in the UI, for example in search menus, add menus, or asset shelves that can be accessed within 3D Views and other editors. So global access to assets with efficient storage management was needed, the per-editor storage of the Asset/File Browser wasn't enough anymore.
Out of this, the asset system design was born. It would be a globally accessible system, optimized for what's needed for the asset functionality wanted in Blender. The asset system provides asset functionality and a global asset database that is separate from the Main database in Blender (the data-base representing .blend file data). Note that (unlike Main), the asset system is not a direct database, since the asset system can hold multiple asset libraries. Assets are also not meant to be written to files by the asset system, unlike the data-blocks in Main (the asset system does write asset information into an index though, for fast loading of asset libraries).
From File Browser Back-End to the Asset System¶
As described, the current Asset Browser is essentially the File Browser in camouflage. So initially, the file browser back-end played the role of the asset system back-end. We are slowly transitioning away from this, and there are a number of hacks and sources of confusion related to this transition.
Asset UIs and functionality should be completely detached from the file browser and its back-end. Instead the asset system should be developed further as basis for all asset functionality and UIs.
An asset representation holds the necessary information about an asset so that the asset system can work with it, and the user interact with it as per design, without requiring the represented entity (like the represented object or material) to be available. It holds information like the asset's name, traits and other asset metadata. Asset representations are meant to be organized into asset libraries.
In code you will find the
AssetRepresentation class. It is one of
the key entities of the asset system.
AssetRepresentationwas only recently introduced and should replace the
AssetHandletype, which is a temporary solution wrapping a file of the File Browser backend.
- Asset libraries contain storage for asset representations but loading
the asset library doesn't load the asset representations itself.
Currently the File Browser backend or the
AssetListAPI need to be used to perform the loading. They use the
AssetLibrary.add_local_id_asset()functions to construct an asset representation that will be owned by an asset library, and as such becomes available via the asset system.
- Storage and lifetime management for assets needs to be improved still.
Meanwhile, parts of Blender that load asset libraries should also
unload assets when they are no longer needed using
AssetLibrary.remove_asset(). Assets are currently not de-duplicated (there may be more than one representation in a library's storage for the same underlying asset), so for as long as the code can assume that no other part of Blender added an asset representation in question, it can safely remove it.
Information to uniquely identify and locate an asset.
AssetIdentifier combines a custom asset library location with a
custom relative asset identifier (identifier for the asset within the
asset library), so that an asset can be uniquely identified and located.
Currently the asset library location and the relative asset identifier
need to add up to an absolute ID library path. For example the asset
library root path could be
/home/assets/ and the relative asset path
subdir/asset.blend/Materials/Material.001, adding up to
Asset identifier paths locate the asset, not the asset library
Just from looking at the full path generated from an asset
identifier, it's not clear which asset library an asset representation
belongs to. In the example above,
/home/assets/ may both
be different asset libraries.
This is a runtime only identifier and should not be stored on disk in any way. Otherwise the identifier breaks when asset data is shared with another machine or user.
Currently, all assets must use Blender ID data-blocks (Blender objects, collections, materials, etc.) as their underlying entity. While the asset system design should allow other kinds of assets in future, support for this kind of asset is a core feature of the asset system. They are referred to as ID assets or data-block assets wherever this characteristic matters.
- ID data-blocks already include a name, and some already include a
preview image (e.g.
Object.preview_image). So this data is not included in the current
AssetMetaDatastruct, and the asset system gets this information from the ID data directly, instead of the asset specific data.
- There's an important differentiation between local, and external ID assets, as explained in the following.
Local ID Assets¶
As a basic principle, Blender is not allowed to edit external .blend files. This means only assets that are in the currently edited file can be edited. The asset system gives them some special treatment, and works together with general ID management code to support them and the editing features.
- When a local ID asset is "imported" (as in, dragged in) in from an asset library, this means the local ID data-block is reused, so no data is actually imported. There could be an option to duplicate instead, similar to the Append and Append (Reuse Data) choice.
External ID Assets¶
- External ID Assets use the existing link and append functionality in Blender for importing.
AssetLibraryinstances are owned by the asset system (
AssetLibraryServiceto be precise) when created through
AssetLibraryService). It is destructed when loading a different .blend file or closing Blender.
AssetCataloginstances and related data are owned by the asset system (Through
AssetRepresentationis owned by
AssetLibrary. However, it's lifetime must mostly be managed by the code using the asset library! It is responsible for creating the asset (
add_local_id_asset()) and removing it once not needed anymore (
remove_asset()). Each user gets its own representation of each asset, so the asset system doesn't have to keep track of how often an asset is used and when it can be freed.
AssetListis owned by the static
AssetListStorage. It is destructed when loading a different .blend file or closing Blender. It manages own asset representations so it keeps pointers into asset library data and frees asset representations as it gets destructed. Therefore, care has to be taken that its lifetime doesn't surpass any of its asset libraries.
AssetCatalogsame as above.
- Let the UI request views on (or call it partitions?) asset libraries,
which user count the asset representations so these can be freed when
no longer in use. May want to use
std::shared_ptrfor this, or a simple garbage collector.
Asset List API¶
The asset list API (see
a more or less temporary design for an asset system like, global storage
of assets, while much of the code still relies on the File Browser back
end. It is needed anywhere where assets need to be available outside of
the Asset Browser.
How asset lists relate to the actual asset system and types like
AssetLibrary is a bit confusing at this point. Essentially, asset
libraries can either be loaded by the Asset/File Browser, or through the
asset list API. Both use the File Browser back end internally, but also
load a matching
AssetLibrary and store the loaded asset
representations into it. So the asset representations are already owned
by the asset system (through asset libraries), but the loading and
access is still managed through the asset list API. In future the asset
list API may be replaced by some kind of asset library loading API.
Entry point and container for asset representations and asset catalogs.
An asset library contains:
- A unique identifier for itself. [Not yet the case in master]
- Asset catalog storage and access.
- Asset representation storage and access.
While catalogs are loaded when loading the asset library, the asset representations have to be added to the library by external code. Asset libraries are meant to be "unopinionated" containers for asset representations: An asset library will accept any asset representation being fed to it, it's not its responsibility to decide what it should contain.
An asset representation that is not added to an asset library may not work well with some asset system functionality. So while it may be useful to hold orphan asset representations in some cases, this should be done carefully in small scope.
Pain point: Nested/overlapping asset libraries
Asset libraries are not strictly separate entities, there may be overlap between them:
- If an asset library directory includes the currently open file, the assets from the "Current File" asset library are added instead of the file on disk.
- The "All" asset library (not yet in master) merges all asset libraries into one.
- A custom asset library may be nested within another custom asset library.
This is one reason why it's important to keep asset libraries "unopinionated".
Since there may be multiple representations of the same underlying asset, this does not create an ownership problem currently.
Consider possible implications in further design work!
Asset libraries may hold a root path, which makes it possible to locate them and the assets inside on disk.
An asset library reference is a handle to uniquely identify an asset library. That may be either one of the builtin libraries (Current File, Current Project, ...) or a custom asset library as defined in the user's Preferences.
It references an asset library through two values:
eAssetLibraryType type: Identifies the builtin library, or that a custom library is referenced (
int custom_library_index: If a custom library is referenced, this is set to the index of the custom library in the
U.asset_librarieslist. Should be -1 otherwise, for easy debugging.
By referencing custom libraries by index, it's ensured that the library reference stays valid even if the library is renamed. Downside is that asset libraries can't easily be reordered. And a bit of care has to be taken when storing references: After removing/adding custom libraries in the Preferences, the library referred to by the index may be a different one or may not exist anymore.