Design Catalog system #90066

New Issue

Sybren A. Stüvel · 2021-07-23T12:09:42+02:00

Sybren A. Stüvel commented

2021-07-23 12:09:42 +02:00

The Catalog system (as described in Asset Browser Workshop Outcomes) needs to be designed, and concrete implementation tasks created.

Catalogs Plan of Attack

Context: 2021-08-20 Asset Browser and 3.0 meeting.

This is a draft planning for tasks regarding the Catalogs. Unknowns will be split out into separate tasks, as not every question has to be answered now.

Design Considerations

C vs C++

The asset catalog system will be C++ with a C interface where necessary.

Since all the catalog information is runtime-only, and storage happens in separate plain-text files, there is no need for DNA.

File Format

A catalog definition file (CDF) is a plain text (UTF-8) file.

Empty lines are ignored.
Lines starting with # are ignored.
The file should start with a declaration of what it is (human-readable), and a version number indicating the file format revision (human- and machine-readable).

Any other line should contain a catalog definition:

CATALOG_ID posix/style/path

The CATALOG_ID may not contain spaces, as a space is used as separator between it and the path.
The path can contain spaces.
Path components are separated by a slash (also on Windows).
Leading and traling whitespace and slashes are meaningless, and are stripped on loading the file.

Multiple definitions

A library could have zero, one, or multiple catalog definition files.

One CDF: simple configuration. Any newly created catalogs will stored here.
Zero CDFs: unexpected configuration. Should not break Blender.
- Maybe show assets under a ${CATALOGID} (unknown catalog) label? Renaming that catalog should create its path. This could also work for other cases where unknown catalog IDs are used.
- #90373 (Handling of unknown catalogs) created for this.
Multiple CDFs: complex configuration. Renames could influence multiple CDFs simultaneously. Changed catalogs should be written back to the file they came from.
- #90194 (Parse & Show Catalogs From Multiple Definition Files)
- #90195 (Write Changes to Multiple Definition Files)

Deduplication

Multiple definitions of a CATALOG_ID in one file should be avoided. A warning could be logged (on the terminal, for example) and the first occurrence of the catalog wins.

Implemented in {1f0d6f763573}.

Defining the same catalog in multiple files is fine, as long as they don't confict. In case of conflict, the one defined in the top-most file wins, and the rest is synchronized when it changes (i.e. when a write is triggered).

Could be implemented as part of #90194 (Parse & Show Catalogs From Multiple Definition Files).

File Locations

Multiple locations are supported:

Top level: ${ASSET_LIBRARY_ROOT}/blender-asset-library.cats.txt
Per blend file: blendfile-path-without-extension.cats.txt
Maaaaybe arbitrary locations? Like a blender-asset-library.cats.txt in each subdirectory? Will become relevant when subdirs are possible with the Asset Browser.

Big chunks to do

Decide on line endings

What happens when writing the file? Probably best to keep the line ending that's in use (to stay friendly with versioning systems).

First step: use platform-native line endings when writing.
Second step: remember line endings used, and use those when writing.

Decide how to keep track of which catalog is defined in which CDF

A catalog can be defined in multiple files, and when updated should be written to all those files.
This can be done by keeping track of a set<AssetCatalogDefinitionFile *> in each catalog.

Decide UX for editing catalog IDs

The catalog paths are what users will be primarily interacting with. The catalog IDs allow for nifty tricks (like moving assets without updating their blend file), but do make things harder to explain/use. This should be clear in the UI.

Decide UX for creating new Catalogs

Users have to choose a catalog ID, which could default to the slugified last element of the catalog path.

Something (user or Blender) has to choose where to save the catalog info:

If there is a CDF for the current blend file, store there, otherwise
if there is a top-level CDF, store there, otherwise
if the current blend file is in an asset library, create a top-level CDF, otherwise
create a CDF for the current blend file.

Add Catalog ID to `AssetMetaData`

A symbolic identifier of which catalog path this asset is stored in.

typedef struct AssetMetaData {

...
char catalog_id[64];

TBD: the max length. 64 was just chosen arbitrarily.

Add `AssetLibraryCatalogs` class

Manages catalogs of a single asset library.

Given a Catalog ID, can look up the path.
Can give the Catalog ID of a given path (f.e. when moving assets)
Can rename paths (keeping track of other paths to change as well, when the changed part is shared by other paths).
Can read CDFs.
Can write CDFs.
Maybe: can work as back-end for tree view.

TBD: this needs to be split up into smaller TODOs.

Reading, writing, tracking


using CatalogID = std::string;
using CatalogPath = std::string;
using CatalogPathComponent = std::string;
using CatalogFilePath = std::string;

class AssetCatalogDefinitionFile;

class AssetCatalogTreeNode {

  /* Pointer to the parent tree node. */
  AssetCatalogTreeNode *parent = nullptr;

  /* Child nodes, ordered by their name. */
  std::set<AssetCatalogTreeNode *> children;
  
  /* Catalogs that have this path. */
  Map<CatalogID, AssetCatalog *> catalogs;

  /* The path component that is associated with this catalog. Note that this may contain separators
   * when ancestors don't exist. This can happen when there are catalog definitions like this:
   *
   * PARENT       parent                        -> name = "parent"
   * THIS_CATALOG parent/nonexistant/thisone    -> name = "nonexistant/thisone"
   */
  CatalogPathComponent name;
  
  /* Set the catalog path to `parent->catalog_path() + separator + node_name` and update all child catalogs. */
  void set_catalog_name(const CatalogPathComponent &node_name);

  /* Return the full catalog path, defined as the name of this catalog prefixed by the full
   * catalog path of its parent and a separator. */
  std::string catalog_path() const;

};

class AssetCatalog {
  /* Symbolic ID, "THIS_CATALOG" in the above example. */
  CatalogID catalog_id;

  /* Return the full catalog path, defined as the name of this catalog prefixed by the full
   * catalog path of its parent and a separator. */
  std::string catalog_path

  /* Which definition files this catalog was defined in. Can be multiple, which means that multiple
   * files will have to be updated when this catalog is edited. */
  Set<AssetCatalogDefinitionFile *> defined_in;
};


class AssetCatalogDefinitionFile {
  CatalogFilePath file_path;
  Map<CatalogID, AssetCatalog *> catalogs;
  VectorSet<AssetCatalog *> root_catalogs;

  void write_to_disk() const;

  *Defined here instead of on AssetCatalog itself for reasons:* - Caller doesn't have to know whether this catalog is even defined in this file.
  *- Allows for keeping track of dirty files (i.e. where write_to_disk() has to*   actually do something).
  *- Allows for keeping the internal bookkeeping up to date.* - Allows for path changes to be reflected in more than one catalog.
  // No-op if this file doesn't have this catalog.
  void change_catalog_path(const CatalogID &catalog_id, const CatalogPath &new_catalog_path);
  void change_catalog_id(const CatalogID &catalog_id, const CatalogID &new_catalog_id);

  // Return nullptr if not found.
  AssetCatalog *find_catalog(const CatalogID &catalog_id);
};

TBD: internal data structure. What is written above is just one simple idea. It's probably better to have the internal structure stored in a tree, such that a rename of a path just requires moving some elements of the tree around. Might be over-engineered though, as the catalogs are meant to be human-manageable and thus there won't be thousands of them anyway.

Add Tree view of available catalogs

Make things collapsible, selectable, determines which assets are shown in the asset browser.

Allows renaming and moving of catalogs. These operations are the same from the CDF perspective, but are different operations for the user interface.

Add current catalog (ID or path) to asset browser

The asset browser should be able to show assets from a single catalog path + its children.

The Catalog system (as described in [Asset Browser Workshop Outcomes](https://code.blender.org/2021/06/asset-browser-workshop-outcomes/)) needs to be designed, and concrete implementation tasks created. ------- # Catalogs Plan of Attack **Context:** [2021-08-20 Asset Browser and 3.0](https://devtalk.blender.org/t/2021-08-20-asset-browser-and-3-0/19730) meeting. This is a draft planning for tasks regarding the Catalogs. Unknowns will be split out into separate tasks, as not every question has to be answered now. ## Design Considerations ### C vs C++ The asset catalog system will be C++ with a C interface where necessary. Since all the catalog information is runtime-only, and storage happens in separate plain-text files, there is no need for DNA. ### File Format A **catalog definition file (CDF)** is a plain text (UTF-8) file. - Empty lines are ignored. - Lines starting with `#` are ignored. - The file should start with a declaration of what it is (human-readable), and a version number indicating the file format revision (human- and machine-readable). Any other line should contain a catalog definition: ``` CATALOG_ID posix/style/path ``` - The `CATALOG_ID` may not contain spaces, as a space is used as separator between it and the path. - The path can contain spaces. - Path components are separated by a slash (also on Windows). - Leading and traling whitespace and slashes are meaningless, and are stripped on loading the file. ### Multiple definitions A library could have zero, one, or multiple catalog definition files. - **One CDF: simple configuration.** Any newly created catalogs will stored here. - **Zero CDFs: unexpected configuration.** Should not break Blender. - Maybe show assets under a *${CATALOGID} (unknown catalog)* label? Renaming that catalog should create its path. This could also work for other cases where unknown catalog IDs are used. - #90373 (Handling of unknown catalogs) created for this. - **Multiple CDFs: complex configuration.** Renames could influence multiple CDFs simultaneously. Changed catalogs should be written back to the file they came from. - #90194 (Parse & Show Catalogs From Multiple Definition Files) - #90195 (Write Changes to Multiple Definition Files) ### Deduplication Multiple definitions of a `CATALOG_ID` in one file should be avoided. A warning could be logged (on the terminal, for example) and the first occurrence of the catalog wins. - Implemented in {1f0d6f763573}. Defining the same catalog in multiple files is fine, as long as they don't confict. In case of conflict, the one defined in the top-most file wins, and the rest is synchronized when it changes (i.e. when a write is triggered). - Could be implemented as part of #90194 (Parse & Show Catalogs From Multiple Definition Files). ### File Locations Multiple locations are supported: - Top level: `${ASSET_LIBRARY_ROOT}/blender-asset-library.cats.txt` - Per blend file: `blendfile-path-without-extension.cats.txt` - Maaaaybe arbitrary locations? Like a `blender-asset-library.cats.txt` in each subdirectory? Will become relevant when subdirs are possible with the Asset Browser. ## Big chunks to do ### Decide on line endings What happens when writing the file? Probably best to keep the line ending that's in use (to stay friendly with versioning systems). - First step: use platform-native line endings when writing. - Second step: remember line endings used, and use those when writing. ### Decide how to keep track of which catalog is defined in which CDF A catalog can be defined in multiple files, and when updated should be written to all those files. This can be done by keeping track of a `set<AssetCatalogDefinitionFile *>` in each catalog. ### Decide UX for editing catalog IDs The *catalog paths* are what users will be primarily interacting with. The catalog IDs allow for nifty tricks (like moving assets without updating their blend file), but do make things harder to explain/use. This should be clear in the UI. ### Decide UX for creating new Catalogs Users have to choose a *catalog ID*, which could default to the slugified last element of the catalog path. Something (user or Blender) has to choose where to save the catalog info: - If there is a CDF for the current blend file, store there, otherwise - if there is a top-level CDF, store there, otherwise - if the current blend file is in an asset library, create a top-level CDF, otherwise - create a CDF for the current blend file. ### Add Catalog ID to `AssetMetaData` A symbolic identifier of which catalog path this asset is stored in. ```lang=c typedef struct AssetMetaData { ``` ... char catalog_id[64]; ``` } ``` **TBD: the max length.** 64 was just chosen arbitrarily. ### Add `AssetLibraryCatalogs` class Manages catalogs of a single asset library. - Given a Catalog ID, can look up the path. - Can give the Catalog ID of a given path (f.e. when moving assets) - Can rename paths (keeping track of other paths to change as well, when the changed part is shared by other paths). - Can read CDFs. - Can write CDFs. - Maybe: can work as back-end for tree view. **TBD: this needs to be split up into smaller TODOs.** #### Reading, writing, tracking ```lang=cpp using CatalogID = std::string; using CatalogPath = std::string; using CatalogPathComponent = std::string; using CatalogFilePath = std::string; class AssetCatalogDefinitionFile; class AssetCatalogTreeNode { /* Pointer to the parent tree node. */ AssetCatalogTreeNode *parent = nullptr; /* Child nodes, ordered by their name. */ std::set<AssetCatalogTreeNode *> children; /* Catalogs that have this path. */ Map<CatalogID, AssetCatalog *> catalogs; /* The path component that is associated with this catalog. Note that this may contain separators * when ancestors don't exist. This can happen when there are catalog definitions like this: * * PARENT parent -> name = "parent" * THIS_CATALOG parent/nonexistant/thisone -> name = "nonexistant/thisone" */ CatalogPathComponent name; /* Set the catalog path to `parent->catalog_path() + separator + node_name` and update all child catalogs. */ void set_catalog_name(const CatalogPathComponent &node_name); /* Return the full catalog path, defined as the name of this catalog prefixed by the full * catalog path of its parent and a separator. */ std::string catalog_path() const; }; class AssetCatalog { /* Symbolic ID, "THIS_CATALOG" in the above example. */ CatalogID catalog_id; /* Return the full catalog path, defined as the name of this catalog prefixed by the full * catalog path of its parent and a separator. */ std::string catalog_path /* Which definition files this catalog was defined in. Can be multiple, which means that multiple * files will have to be updated when this catalog is edited. */ Set<AssetCatalogDefinitionFile *> defined_in; }; class AssetCatalogDefinitionFile { CatalogFilePath file_path; Map<CatalogID, AssetCatalog *> catalogs; VectorSet<AssetCatalog *> root_catalogs; void write_to_disk() const; *Defined here instead of on AssetCatalog itself for reasons:* - Caller doesn't have to know whether this catalog is even defined in this file. *- Allows for keeping track of dirty files (i.e. where write_to_disk() has to* actually do something). *- Allows for keeping the internal bookkeeping up to date.* - Allows for path changes to be reflected in more than one catalog. // No-op if this file doesn't have this catalog. void change_catalog_path(const CatalogID &catalog_id, const CatalogPath &new_catalog_path); void change_catalog_id(const CatalogID &catalog_id, const CatalogID &new_catalog_id); // Return nullptr if not found. AssetCatalog *find_catalog(const CatalogID &catalog_id); }; ``` **TBD: internal data structure.** What is written above is just one simple idea. It's probably better to have the internal structure stored in a tree, such that a rename of a path just requires moving some elements of the tree around. Might be over-engineered though, as the catalogs are meant to be human-manageable and thus there won't be thousands of them anyway. ### Add Tree view of available catalogs Make things collapsible, selectable, determines which assets are shown in the asset browser. Allows renaming and moving of catalogs. These operations are the same from the CDF perspective, but are different operations for the user interface. ### Add current catalog (ID or path) to asset browser The asset browser should be able to show assets from a single catalog path + its children.

Sybren A. Stüvel self-assigned this 2021-07-23 12:09:42 +02:00

Sybren A. Stüvel commented

2021-07-23 12:09:42 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Sybren A. Stüvel commented

2021-07-23 12:09:42 +02:00

Added subscriber: @dr.sybren

Paul Larson commented

2021-07-23 18:52:52 +02:00

Added subscriber: @GeorgiaPacific

Sybren A. Stüvel was unassigned by Julian Eisel

2021-10-19 19:05:19 +02:00

Julian Eisel commented

2021-12-09 16:03:18 +01:00

Added subscriber: @JulianEisel

Julian Eisel commented

2021-12-09 16:03:18 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Julian Eisel closed this issue

2021-12-09 16:03:18 +01:00

Julian Eisel self-assigned this 2021-12-09 16:03:18 +01:00

Julian Eisel commented

2021-12-09 16:03:18 +01:00

Bulk closing all tasks in the "Done" column. Sybren and I agreed that we don't need to keep this history anymore at this point.
Thanks all :)

Bulk closing all tasks in the "Done" column. Sybren and I agreed that we don't need to keep this history anymore at this point. Thanks all :)

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Design Catalog system #90066

Catalogs Plan of Attack

Design Considerations

C vs C++

File Format

Multiple definitions

Deduplication

File Locations

Big chunks to do

Decide on line endings

Decide how to keep track of which catalog is defined in which CDF

Decide UX for editing catalog IDs

Decide UX for creating new Catalogs

Add Catalog ID to AssetMetaData

Add AssetLibraryCatalogs class

Reading, writing, tracking

Add Tree view of available catalogs

Add current catalog (ID or path) to asset browser

Add Catalog ID to `AssetMetaData`

Add `AssetLibraryCatalogs` class