Collections for Import/Export #68933

Open
opened 2019-08-20 22:42:52 +02:00 by Dalai Felinto · 1 comment

Problem Statement

Currently importing and exporting files like glTF, FBX, Alembic or USD is done through operators. This is fine for a one time export, but when re-importing or re-exporting the same asset many times it becomes cumbersome. We need to store relations to such files and settings permanently, so re-import and re-export is quick, and may even be done automatically on changes.

Example Use Cases

  • Creating assets for a game, where the same asset is tweaked and exported many times.
  • Production pipeline loading animated characters from Alembic file and using Blender for rendering.
  • Production pipeline step for an animator, import USD layer, animate character and export a modified USD layer.
  • Realtime collaboration through USD / Omniverse.

Design

Collection Settings

The basic idea is that Collection datablock would get settings for import and export.

For export, a collection could have one or more associated exporters. Blender would natively support USD and Alembic exporters. However this could be an API that Python add-ons can plug into, to also support file formats like FBX and glTF. The settings would be defined by the exporter, and could be exactly the same as existing export operator properties. Once export settings are set up, re-exporting one (or all) collections would be a single click, much like rendering.

For import, the simplest case would be similar to export. A collection would have one associated importer and settings. Re-importing would be done with a single click as well. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost.

Libraries and Overrides

For a production pipeline, we could treat USD or Alembic files like libraries and linked datablocks. They would be loaded on .blend file load, be non-editable by default and then support overrides and make local just like linked datablocks.

To support such a system, the existing Library datablock would be used. It would contain the filepath and other importer settings. Objects and other datablocks coming from USD or Alembic collections would be associated with a library much like linked datablocks.

That system would also replace the current Alembic importer system. That system adds local objects to the Blender scene, and then dynamically loads in geometry through modifiers. In that system only geometry can change dynamically, while we want for example object or material properties to update on changes also. If the override system is used, the choice of which properties to modify locally is up to the user rather than being a predefined subset. Implementing #69046 would no longer be necessary.

USD / Hydra

USD Layers and Blender Collections have similar purpose. Evolving Blender Collections to match USD Layers more closely would help integrate well with USD.

When using Hydra or another renderer that can directly load USD, there is no need to convert to Blender data structures. In such cases the importer may create only a bounding box or other stand-in object for Blender. This feature would be useful also when not using Hydra, to only partially load a scene for performance reasons.

Nodes

For additional flexibility, a USD importer or exporter may want to do some processing on the data. The design is compatible with an importer or exporter having a node graph to do such processing. Also without importers/exporters, it may be useful to be able to have a collection that can be generated by some node graph based on the contents of other collections.

To keep the project scope under control, it seems unlikely that the initial implementation would have support for nodes.

Realtime Collaboration

A system like Omniverse for realtime collaboration could use some of the same infrastructure. Collections could have settings associating them with e.g. a URI instead of a file path.

Such a system might support an advanced production pipeline use case where there is a USD layer coming in, overrides are applied and the result is then exported as another USD layer. However the initial or main target would probably be cases where datablocks are simply edited without Blender overrides. Rather the datablocks would be local and editable for the duration of the session, and any changes would be synchronized with the server.

### Problem Statement Currently importing and exporting files like glTF, FBX, Alembic or USD is done through operators. This is fine for a one time export, but when re-importing or re-exporting the same asset many times it becomes cumbersome. We need to store relations to such files and settings permanently, so re-import and re-export is quick, and may even be done automatically on changes. ### Example Use Cases * Creating assets for a game, where the same asset is tweaked and exported many times. * Production pipeline loading animated characters from Alembic file and using Blender for rendering. * Production pipeline step for an animator, import USD layer, animate character and export a modified USD layer. * Realtime collaboration through USD / Omniverse. ### Design **Collection Settings** The basic idea is that Collection datablock would get settings for import and export. For export, a collection could have one or more associated exporters. Blender would natively support USD and Alembic exporters. However this could be an API that Python add-ons can plug into, to also support file formats like FBX and glTF. The settings would be defined by the exporter, and could be exactly the same as existing export operator properties. Once export settings are set up, re-exporting one (or all) collections would be a single click, much like rendering. For import, the simplest case would be similar to export. A collection would have one associated importer and settings. Re-importing would be done with a single click as well. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost. **Libraries and Overrides** For a production pipeline, we could treat USD or Alembic files like libraries and linked datablocks. They would be loaded on .blend file load, be non-editable by default and then support overrides and make local just like linked datablocks. To support such a system, the existing Library datablock would be used. It would contain the filepath and other importer settings. Objects and other datablocks coming from USD or Alembic collections would be associated with a library much like linked datablocks. That system would also replace the current Alembic importer system. That system adds local objects to the Blender scene, and then dynamically loads in geometry through modifiers. In that system only geometry can change dynamically, while we want for example object or material properties to update on changes also. If the override system is used, the choice of which properties to modify locally is up to the user rather than being a predefined subset. Implementing #69046 would no longer be necessary. **USD / Hydra** USD Layers and Blender Collections have similar purpose. Evolving Blender Collections to match USD Layers more closely would help integrate well with USD. When using Hydra or another renderer that can directly load USD, there is no need to convert to Blender data structures. In such cases the importer may create only a bounding box or other stand-in object for Blender. This feature would be useful also when not using Hydra, to only partially load a scene for performance reasons. **Nodes** For additional flexibility, a USD importer or exporter may want to do some processing on the data. The design is compatible with an importer or exporter having a node graph to do such processing. Also without importers/exporters, it may be useful to be able to have a collection that can be generated by some node graph based on the contents of other collections. To keep the project scope under control, it seems unlikely that the initial implementation would have support for nodes. **Realtime Collaboration** A system like Omniverse for realtime collaboration could use some of the same infrastructure. Collections could have settings associating them with e.g. a URI instead of a file path. Such a system might support an advanced production pipeline use case where there is a USD layer coming in, overrides are applied and the result is then exported as another USD layer. However the initial or main target would probably be cases where datablocks are simply edited without Blender overrides. Rather the datablocks would be local and editable for the duration of the session, and any changes would be synchronized with the server.
Brecht Van Lommel changed title from Per-Collection I/O data for re-import/export to Collections for Import/Export 2020-08-07 15:47:30 +02:00

The other way around, for importing it would similarly be a welcome improvement if you could link an FBX, OBJ, Alembic etc into Blender and once the associated file is updated, you would be able to update the contents of the file in Blender. Many popular rendering programs are also able to memorise material overrides (i.e. they propagate the 'old' material assignments upon updating the geometry, so the materials stay the same). I hope this will also be considered at a later stage of development.

The other way around, for importing it would similarly be a welcome improvement if you could link an FBX, OBJ, Alembic etc into Blender and once the associated file is updated, you would be able to update the contents of the file in Blender. Many popular rendering programs are also able to memorise material overrides (i.e. they propagate the 'old' material assignments upon updating the geometry, so the materials stay the same). I hope this will also be considered at a later stage of development.
Philipp Oeser removed the
Interest
Pipeline, Assets & IO
label 2023-02-10 08:54:22 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#68933
No description provided.