BAT: Blender Asset Tracer ? #54125

New Issue

Sybren A. Stüvel · 2018-02-22T10:47:55+01:00

Sybren A. Stüvel commented

2018-02-22 10:47:55 +01:00

We need a replacement for BAM, as it's bloated (webserver, just to name one thing) and written in an overly complex way. In this ticket I propose BAT ?, the Blender Asset Tracer.

Requirements

Layered structure, so that tracing dependencies, copying files, rewriting links, and changing variations (which are all done in a single function in BAM) should be in separate bits of code. The caller should be able to use just one of those.
Callable from Python. The current BAM Pack functionality is hard to call directly from Python, so in the Blender Cloud add-on I actually call the CLI interface as a subprocess.
Use pathlib.Path to represent paths, rather than byte strings.
CLI entry points only as callables defined as entry points in setup.py, no directy executing of Python scripts. Current BAM has a mixture of a top-level Python file, files in subpackages that can be run as CLI when they are executed directly, and __main__.py files that allow python -m bam.pack.
Unit tested using py.test, rather than hand-writing a custom test runner.

Features

Trace & report dependencies: for each dependency, report which datablock in which file requires it.
Generator-based approach. This makes it possible to report dependencies without having to maintain an in-memory list of all of them.
Filter reported dependencies: for example, skip Alembic files that are on /shared.
"Action Collector", which collects descriptions of copy actions that should be performed on assets:
- While tracing the dependencies we can store the copy-actions ("copy file A to location B and update reverse dependencies").
- When tracing is done, the Action Collector can go over the assets to be copied, and either point you to the original asset (if it doesn't require rewriting) or a temporary copy (if it did require rewriting).
- It's up to the caller of the Action Collector to determine what should happen with those files (copy to a directory on the local filesystem, upload to S3 storage, FTP somewhere, etc.)
Data obtained by the Tracer might be shared with the Action Collector (uncompressed blend files, data block information of the blend file, etc.) to prevent analyzing the same blend file multiple times.

End-user features

maintained or added

BAT? Pack: copying a blend file with all of its dependencies to another location. Paths to dependencies are updated to point to the correct location. This is a core feature of BAM that's used a lot, and thus will be implemented in BAT too.
List dependencies: similar as creating the BAT Pack, but without copying of files.
Path remapping. For those who don't know this part of BAM, it's where you say "remap start", then move some assets around, then say "remap end", and BAM will detect which assets moved where, and update any blend file that refers to it. Apparently this has saved a few productions in the past ;-)

removed (until proven necessary)

The entire webservice part, both client and server.
SVN support. At the Blender Animation Studio we do use SVN, but as a tool next to BAM. BAM is never used to directly create SVN checkouts. We want to add SVN support to Flamenco at some point so that we can create render jobs with SVN checkouts, but I don't think this belongs in BAT.
Overrides. At this moment we don't use them at the BAM level. I think an override / asset switching system would be nice, but (at least from what I understand of the functionality) would make more sense inside Blender itself so that the artists see what they're doing.

We need a replacement for [BAM](https://developer.blender.org/diffusion/BAM/), as it's bloated (webserver, just to name one thing) and written in an overly complex way. In this ticket I propose BAT ?, the Blender Asset Tracer. ## Requirements - Layered structure, so that tracing dependencies, copying files, rewriting links, and changing variations (which are all done in a single function in BAM) should be in separate bits of code. The caller should be able to use just one of those. - Callable from Python. The current BAM Pack functionality is hard to call directly from Python, so in the Blender Cloud add-on I actually call the CLI interface as a subprocess. - Use `pathlib.Path` to represent paths, rather than byte strings. - CLI entry points only as callables defined as entry points in `setup.py`, no directy executing of Python scripts. Current BAM has a mixture of a top-level Python file, files in subpackages that can be run as CLI when they are executed directly, and `__main__.py` files that allow `python -m bam.pack`. - Unit tested using `py.test`, rather than hand-writing a custom test runner. ## Features - Trace & report dependencies: for each dependency, report which datablock in which file requires it. - Generator-based approach. This makes it possible to report dependencies without having to maintain an in-memory list of all of them. - Filter reported dependencies: for example, skip Alembic files that are on `/shared`. - "Action Collector", which collects descriptions of copy actions that should be performed on assets: - While tracing the dependencies we can store the copy-actions ("copy file A to location B and update reverse dependencies"). - When tracing is done, the Action Collector can go over the assets to be copied, and either point you to the original asset (if it doesn't require rewriting) or a temporary copy (if it did require rewriting). - It's up to the caller of the Action Collector to determine what should happen with those files (copy to a directory on the local filesystem, upload to S3 storage, FTP somewhere, etc.) - Data obtained by the Tracer might be shared with the Action Collector (uncompressed blend files, data block information of the blend file, etc.) to prevent analyzing the same blend file multiple times. ## End-user features ### maintained or added - **BAT? Pack**: copying a blend file with all of its dependencies to another location. Paths to dependencies are updated to point to the correct location. This is a core feature of BAM that's used a lot, and thus will be implemented in BAT too. - **List dependencies**: similar as creating the BAT Pack, but without copying of files. - **Path remapping**. For those who don't know this part of BAM, it's where you say "remap start", then move some assets around, then say "remap end", and BAM will detect which assets moved where, and update any blend file that refers to it. Apparently this has saved a few productions in the past ;-) ### removed (until proven necessary) - The entire webservice part, both client and server. - SVN support. At the Blender Animation Studio we do use SVN, but as a tool next to BAM. BAM is never used to directly create SVN checkouts. We want to add SVN support to Flamenco at some point so that we can create render jobs with SVN checkouts, but I don't think this belongs in BAT. - Overrides. At this moment we don't use them at the BAM level. I think an override / asset switching system would be nice, but (at least from what I understand of the functionality) would make more sense inside Blender itself so that the artists see what they're doing.

Sybren A. Stüvel self-assigned this 2018-02-22 10:47:55 +01:00

Sybren A. Stüvel commented

2018-02-22 10:47:55 +01:00

Added subscribers: @dr.sybren, @fsiddi

Francesco Siddi commented

2018-02-22 10:59:11 +01:00

Looks good! Just want to mention here that we have a package called blender-file https://developer.blender.org/source/blender-file/ that should be probably used for some of the blendfile inspection.

Looks good! Just want to mention here that we have a package called `blender-file` https://developer.blender.org/source/blender-file/ that should be probably used for some of the blendfile inspection.

Sybren A. Stüvel commented

2018-02-22 11:01:01 +01:00

This is a clone of what is in BAM at the moment. Ideally BAT? should replace both.

This is a clone of [what is in BAM at the moment](https://developer.blender.org/diffusion/BAM/browse/master/bam/blend/blendfile.py). Ideally BAT? should replace both.

Sybren A. Stüvel commented

2018-02-22 11:18:22 +01:00

Added subscriber: @dfelinto

Oliver Weissbarth commented

2018-02-22 16:51:59 +01:00

Added subscriber: @oweissbarth

Arto Kitula commented

2018-02-23 03:16:57 +01:00

Added subscriber: @ArtoKitula

Campbell Barton commented

2018-02-25 13:33:13 +01:00

Added subscriber: @ideasman42

Campbell Barton commented

2018-02-25 13:33:13 +01:00

Hey, good to see a rewrite that kicks out features that were added speculatively and didn't end up getting used, the feature list is fine but a bit technical and doesn't mention things BAT will drop.

It would be good have a list of features (from user perspective) and anti-features (assume this includes BAM's goals of being an SVN replacement, or being a tool to checkout from a remote server with de-duplication from local data-store).
Would there be the concept of a BAT repository? Would there be the ability to apply overrides? Or is this left up to Python developers to write this using BAT modules?

Hey, good to see a rewrite that kicks out features that were added speculatively and didn't end up getting used, the feature list is fine but a bit technical and doesn't mention things BAT will drop. It would be good have a list of features (from user perspective) and anti-features (assume this includes BAM's goals of being an SVN replacement, or being a tool to checkout from a remote server with de-duplication from local data-store). Would there be the concept of a BAT repository? Would there be the ability to apply overrides? Or is this left up to Python developers to write this using BAT modules?

Sybren A. Stüvel commented

2018-02-26 17:55:57 +01:00

In #54125#485948, @ideasman42 wrote:
It would be good have a list of features (from user perspective) and anti-features (assume this includes BAM's goals of being an SVN replacement, or being a tool to checkout from a remote server with de-duplication from local data-store).
Would there be the concept of a BAT repository? Would there be the ability to apply overrides? Or is this left up to Python developers to write this using BAT modules?

Good points, I've updated the task details.

> In #54125#485948, @ideasman42 wrote: > It would be good have a list of features (from user perspective) and anti-features (assume this includes BAM's goals of being an SVN replacement, or being a tool to checkout from a remote server with de-duplication from local data-store). > Would there be the concept of a BAT repository? Would there be the ability to apply overrides? Or is this left up to Python developers to write this using BAT modules? Good points, I've updated the task details.

Isaac Weaver commented

2018-02-27 02:13:48 +01:00

Added subscriber: @IsaacWeaver

Campbell Barton commented

2018-02-28 05:21:44 +01:00

That's better, note that the statement "We need a replacement for BAM, as it's bloated (webserver, just to name one thing)" is not really true.

BAM was written so the webserver was only needed for the SVN wrapper, and is not needed for pack/remap/listing-deps (all features BAT is keeping).
The web-server dependency can be trivially removed from BAM without impacting any of these sub-commands.

That's better, note that the statement *"We need a replacement for BAM, as it's bloated (webserver, just to name one thing)"* is not really true. BAM was written so the webserver was only needed for the SVN wrapper, and is not needed for pack/remap/listing-deps (all features BAT is keeping). The web-server dependency can be trivially removed from BAM without impacting any of these sub-commands.

Sybren A. Stüvel commented

2018-02-28 11:11:19 +01:00

The webserver isn't the only thing that I'll address. The current code of BAM is quite a mess, and it's very hard to figure out what's going on. I'm talking about conditionally defined inner functions with duplicate names, which are even undocumented too. There is a tuple[index] depending on the endianness for every read on a blend file, even though the endianness of that file will not change. It's also the inconsistent ways in which reporting is done throughout the project (logging module, print statements, reporting callback functions, conditionally-defined functions, yielding strings). Cyclomatic complexity of the BAM code is also very high, whereas there is no way to easily measure unit test code coverage due to the non-standard way of doing unit testing.

By the way, I'm not blaming anybody for the current state, I understand it's been written rapidly and during pressure of movie production. The code issues need to be addressed, though, to make sure we can keep the functionality functional.

The webserver isn't the only thing that I'll address. The current code of BAM is quite a mess, and it's very hard to figure out what's going on. I'm talking about [conditionally defined inner functions with duplicate names](https://developer.blender.org/diffusion/BAM/browse/master/bam/blend/blendfile_path_walker.py;7afb1fe606b9c9e9e735653ab61859ee8a62dc22$375), which are even undocumented too. There is a tuple[index] depending on the endianness for every read on a blend file, even though the endianness of that file will not change. It's also the inconsistent ways in which reporting is done throughout the project (logging module, print statements, reporting callback functions, conditionally-defined functions, yielding strings). Cyclomatic complexity of the BAM code is also very high, whereas there is no way to easily measure unit test code coverage due to the non-standard way of doing unit testing. By the way, I'm not blaming anybody for the current state, I understand it's been written rapidly and during pressure of movie production. The code issues need to be addressed, though, to make sure we can keep the functionality functional.

Campbell Barton commented

2018-03-01 00:19:36 +01:00

I was only commenting on your statement about the web-server, other critiques you make - I generally agree with.

I don't think its too valuable to justify details of the decisions made here, but one thing that made the code quite complicated was needing the server to be able communicate with the client while it was archiving blend files (so the client could avoid downloading data it already had locally).

The pressure of movie production was not so much of a factor. We were writing something to replace SVN - we had a list of tasks which we managed to get working, with tests. The fact artists where not working remotely on large repositories meant the primary use case for such a tool wasn't a problem for us, artists were happy using SVN.

Said different, if I was making BAM without it being an SVN replacement, I would rewrite with less strange code too :)

I was only commenting on your statement about the web-server, other critiques you make - I generally agree with. I don't think its too valuable to justify details of the decisions made here, but one thing that made the code quite complicated was needing the server to be able communicate with the client while it was archiving blend files (so the client could avoid downloading data it already had locally). The pressure of movie production was not so much of a factor. We were writing something to replace SVN - we had a list of tasks which we managed to get working, with tests. The fact artists where not working remotely on large repositories meant the primary use case for such a tool wasn't a problem for us, artists were happy using SVN. Said different, if I was making BAM without it being an SVN replacement, I would rewrite with less strange code too :)

John Roper commented

2018-07-12 23:08:22 +02:00

Added subscriber: @JohnRoper

John Roper commented

2018-07-12 23:08:22 +02:00

Can this be used to check for missing assets in blend files? I'm looking for a way to check a blend file and look for missing assets (images, etc.) that should have been packed into the blend.