Handle Data Dependencies #82653

New Issue

Julian Eisel · 2020-11-12T15:11:21+01:00

Julian Eisel commented

2020-11-12 15:11:21 +01:00

When dragging in an asset that depends on other data-blocks or files, these dependencies should be handled in a clear way.

- Indirect datablocks get appended. -- This is done now, including re-using of heavy indirect data-blocks (794c2828af) and proper scene instantiation (df9120b365)
- External files can use pack, append, then unpack (into e.g. textures/ folder next to .blend) -- In progress, D12423.

When dragging in an asset that depends on other data-blocks or files, these dependencies should be handled in a clear way. * - [x] Indirect datablocks get appended. *-- This is done now, including re-using of heavy indirect data-blocks (794c2828af) and proper scene instantiation (df9120b365)* * - [ ] External files can use pack, append, then unpack (into e.g. textures/ folder next to .blend) *-- In progress, [D12423](https://archive.blender.org/developer/D12423).*

Julian Eisel commented

2020-11-12 15:11:21 +01:00

Added subscriber: @JulianEisel

blender-admin commented

2020-11-12 15:11:21 +01:00

#89986 was marked as duplicate of this issue

Julian Eisel commented

2020-11-12 15:12:01 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Julian Eisel changed title from ~~Handle Data Dependencies (Pack Data)~~ to Handle Data Dependencies

2020-11-12 15:12:28 +01:00

Vilem Duha commented

2020-11-25 10:16:19 +01:00

Added subscriber: @VilemDuha

Vilem Duha commented

2020-11-25 10:16:20 +01:00

It's a question on how to move a file with it's dependencies, at least if these are in relative subfolders. In the Python API, things work strangely. E.g. to change the path where the image will be unpacked, changing image.filepath doesn't work. You need to change image.packed_file.filepath to have the unpack working where you want it. It would be great to be able to pack/unpack without opening the file, and possibly also read unpacked file paths to be able to move them with the .blend.

Büke Beyond commented

2021-01-02 15:55:43 +01:00

Added subscriber: @BukeBeyond

Büke Beyond commented

2021-01-02 15:55:43 +01:00

I am happy to hear you are thinking of the essential "deduplication" function. I have seen this problem with duplicated materials on various forums reviewing your new Asset Browser.

Imagine a high quality city library, with about a 100 buildings and city detail meshes, sharing about a 100 textures. Upon dragging one building asset into the user project, almost all the 100 textures of about 4GB get uploaded to the GPU. If you drag the same (or a similar) building again, the textures get uploaded again, totaling to 8GB, and waiting the same time (5-10 seconds) again. By the time you are on the 4th asset, you have used up more GPU memory (and waited more) than the entire source library file.

Link importing by default would solve this problem of course. But if you were to put an option into the Asset Browser UI to change the dragging behavior to Append/Copy instead, the duplication of the materials and textures would still make a mess of the scene. Imagine changing the glass reflectivity on all the buildings, now you have to go through them all manually. There is no basic dedup function in Blender either. The current Asset Browser viewport option after dragging to change the import to a Link is currently broken with an "Object Not Found" error. It is also inefficient changing this after the drag and all that uploading to the GPU already.

As idiolistic as it is to embed everything (assets, textures, videos) into a single blender file for sharing with peers, it is also not practical in the industry for serious use. I think the new Asset, Linking and Library Override functions getting built into Blender are paving the way for more complex and higher quality projects. While the Library mentality is very useful for kickstarting new scenes, we must also consider these functions as basic as the "#include" in C++, to break down a scene into manageable modular pieces. Just like textures, it would make sense to include and search for asset blender files within the same directory (say the original link location was not found), not necessarily used as Libraries, but just dependencies. In that case, peer sharing linked data would be just as convenient as including those files in the same directory (or subdirectory), just like textures, without embedding everything into a single grand file.

Currently, the manual Link function of Blender works well in our pipeline, and your new filter by Asset feature is very useful. I am sure you are putting more thought into a robust and flexible API backend than the current UI. We are looking forward trying the Asset Browser with thumbnails when the Link importing becomes functional.

I am happy to hear you are thinking of the essential "deduplication" function. I have seen this problem with duplicated materials on various forums reviewing your new Asset Browser. Imagine a high quality city library, with about a 100 buildings and city detail meshes, sharing about a 100 textures. Upon dragging one building asset into the user project, almost all the 100 textures of about 4GB get uploaded to the GPU. If you drag the same (or a similar) building again, the textures get uploaded again, totaling to 8GB, and waiting the same time (5-10 seconds) again. By the time you are on the 4th asset, you have used up more GPU memory (and waited more) than the entire source library file. Link importing by default would solve this problem of course. But if you were to put an option into the Asset Browser UI to change the dragging behavior to Append/Copy instead, the duplication of the materials and textures would still make a mess of the scene. Imagine changing the glass reflectivity on all the buildings, now you have to go through them all manually. There is no basic dedup function in Blender either. The current Asset Browser viewport option after dragging to change the import to a Link is currently broken with an "Object Not Found" error. It is also inefficient changing this after the drag and all that uploading to the GPU already. As idiolistic as it is to embed everything (assets, textures, videos) into a single blender file for sharing with peers, it is also not practical in the industry for serious use. I think the new Asset, Linking and Library Override functions getting built into Blender are paving the way for more complex and higher quality projects. While the Library mentality is very useful for kickstarting new scenes, we must also consider these functions as basic as the "#include" in C++, to break down a scene into manageable modular pieces. Just like textures, it would make sense to include and search for asset blender files within the same directory (say the original link location was not found), not necessarily used as Libraries, but just dependencies. In that case, peer sharing linked data would be just as convenient as including those files in the same directory (or subdirectory), just like textures, without embedding everything into a single grand file. Currently, the manual Link function of Blender works well in our pipeline, and your new filter by Asset feature is very useful. I am sure you are putting more thought into a robust and flexible API backend than the current UI. We are looking forward trying the Asset Browser with thumbnails when the Link importing becomes functional.

Jeroen Bakker commented

2021-08-25 07:41:10 +02:00

Added subscriber: @Jeroen-Bakker

Jeroen Bakker commented

2021-08-25 07:41:10 +02:00

When 2 assets share different textures with the same relative paths there can be conflicts. This needs a more in depth design/discussion.
I assume that the first part of this task is already done in master.

When 2 assets share different textures with the same relative paths there can be conflicts. This needs a more in depth design/discussion. I assume that the first part of this task is already done in master.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Handle Data Dependencies #82653