Page MenuHome

[Blender Asset Tracer] Files are not closed on Windows after using BAT to trace .blend file in Add-on
Closed, ArchivedPublic

Description

System Information
Operating system: Windows 10

Short description of error
Importing BAT as a module into a blender add-on and performing a trace or pack operation on the currently open .blend file will result in the following error upon trying to save the .blend file.

Exact steps for others to reproduce the error
I have attached a simple add-on for the purposes of demonstrating this behavior. It adds a panel to blender with a single button called "Trace" which will trace the currently open blend file if it has been saved.


To reproduce:

  1. Open blender
  2. Install provided add-on
  3. Save a scene so that that the file exists on disk
  4. Click 'trace'
  5. The add-on should print all external assets that the file references to the console (This works fine - as far as I can tell BAT correctly traces all assets)
  6. Attempt to save the .blend file again and you should see the 'Unable to make version backup' error.

I believe the issue is that BAT is not correctly closing files on Windows. I have tried manually calling the blender_asset_tracer.blendfile.close_all_cached() method but nothing changes.

Details

Type
Bug

Event Timeline

Oliver Dawes (haiku) renamed this task from [Blender Asset Tracer] to [Blender Asset Tracer] Files are not closed on Windows after using BAT to trace .blend file in Add-on.
Oliver Dawes (haiku) triaged this task as Waiting for Developer to Reproduce priority.

Process explorer show that once the 'trace' method is called Blender keeps open file handles for all .blend files found by BAT during the trace (This includes linked .blend files).

I can manually close these files as administrator using process explorer which resolves the saving issue but if I then try and trace again using the Add-on I get the following error:

Traceback (most recent call last):

File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\core.py", line 28, in execute
  for usage in block_usage_generator:
File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\__init__.py", line 59, in deps
  for block_usage in blocks2assets.iter_assets(block):
File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\blocks2assets.py", line 53, in iter_assets
  yield from block_reader(block)
File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\blocks2assets.py", line 159, in scene
  block_ed = block.get_pointer(b'ed')
File "S:\blender-addons\addons\gridmarkets_blender_addon\lib\blender_asset_tracer\blendfile\__init__.py", line 668, in get_pointer
  result = self.get(path, default=default)
File "S:\blender-addons\addons\gridmarkets_blender_addon\lib\blender_asset_tracer\blendfile\__init__.py", line 582, in get
  self.bfile.fileobj.seek(self.file_offset, os.SEEK_SET)

OSError: [Errno 9] Bad file descriptor

I have been able to solve this issue by adding the following code:

For the deps() method in trace\__init__.py just add a bfile.close() to the end of the method. This will close the main .blend file being traced after iterating through all the blocks.

def deps(bfilepath: pathlib.Path, progress_cb: typing.Optional[progress.Callback] = None) \
        -> typing.Iterator[result.BlockUsage]:
    """Open the blend file and report its dependencies.

    :param bfilepath: File to open.
    :param progress_cb: Progress callback object.
    """

    log.info('opening: %s', bfilepath)
    bfile = blendfile.open_cached(bfilepath)

    bi = file2blocks.BlockIterator()
    if progress_cb:
        bi.progress_cb = progress_cb

    # Remember which block usages we've reported already, without keeping the
    # blocks themselves in memory.
    seen_hashes = set()  # type: typing.Set[int]

    for block in asset_holding_blocks(bi.iter_blocks(bfile)):
        for block_usage in blocks2assets.iter_assets(block):

            usage_hash = hash(block_usage)
            if usage_hash in seen_hashes:
                continue
            seen_hashes.add(usage_hash)
            yield block_usage

    bfile.close()

All libraries also need closing (although not necessary to fix the original saving bug mentioned in the original post) since they also never get closed.
To do this just add a libfile.close() to the end of the _visit_linked_blocks() method in trace/file2blocks.py

def _visit_linked_blocks(self, blocks_per_lib):
    # We've gone through all the blocks in this file, now open the libraries
    # and iterate over the blocks referred there.
    for lib_bpath, idblocks in blocks_per_lib.items():
        lib_path = pathlib.Path(lib_bpath.to_path())
        try:
            lib_path = lib_path.resolve()
        except FileNotFoundError:
            log.warning('Library %s does not exist', lib_path)
            continue

        log.debug('Expanding %d blocks in %s', len(idblocks), lib_path)
        libfile = blendfile.open_cached(lib_path)
        
        yield from self.iter_blocks(libfile, idblocks)
        libfile.close()
Sybren A. Stüvel (sybren) closed this task as Archived.
Sybren A. Stüvel (sybren) claimed this task.

Calling libfile.close() like that is not a good idea. Opening a blend file is the most expensive part of the tracing process, which is why it's cached in the first place by calling blendfile.open_cached(). Also in this example it's not called when an exception occurs, which means it can still keep files open.

These cached files should be closed by calling blender_asset_tracer.blendfile.close_all_cached(). For me this works fine:

  1. Replay the steps as described by @Oliver Dawes (haiku) (saving the file indeed shows the described error)
  2. Open the Python console, and type import blender_asset_tracer.blendfile; blender_asset_tracer.blendfile.close_all_cached()
  3. Save the file. This time it is succesful.

The problem is not with BAT, but with the way it's loaded in your add-on. You add the lib path to sys.path so that BAT can import its own submodules from blender_asset_tracer, but you also import BAT as from .lib.blender_asset_tracer import trace. This means that BAT is now imported two times; once as bat_bug_addon.lib.blender_asset_tracer and once as blender_asset_tracer. The cached files are in blender_asset_tracer.blendfile._cached_bfiles, but I'm guessing you're calling bat_bug_addon.lib.blender_asset_tracer.blendfile.close_cached().

The solution is to do one of the two, but not both:

  • Add your lib directory to sys.path and import everything in there as if it's top-level.
  • Do not add lib to sys.path and import everything from bat_bug_addon.lib (and adjust whatever is in there to only use relative imports).

Thank you, that makes a lot of sense.