[Blender Asset Tracer] Files are not closed on Windows after using BAT to trace .blend file in Add-on #65933

New Issue

Oliver Dawes · 2019-06-19T18:22:01+02:00

Oliver Dawes commented

2019-06-19 18:22:01 +02:00

System Information
Operating system: Windows 10

Short description of error
Importing BAT as a module into a blender add-on and performing a trace or pack operation on the currently open .blend file will result in the following error upon trying to save the .blend file.

Exact steps for others to reproduce the error
I have attached a simple add-on for the purposes of demonstrating this behavior. It adds a panel to blender with a single button called "Trace" which will trace the currently open blend file if it has been saved.
BAT_bug_addon.zip

To reproduce:

Open blender
Install provided add-on
Save a scene so that that the file exists on disk
Click 'trace'
The add-on should print all external assets that the file references to the console (This works fine - as far as I can tell BAT correctly traces all assets)
Attempt to save the .blend file again and you should see the 'Unable to make version backup' error.

I believe the issue is that BAT is not correctly closing files on Windows. I have tried manually calling the blender_asset_tracer.blendfile.close_all_cached() method but nothing changes.

**System Information** Operating system: Windows 10 **Short description of error** Importing BAT as a module into a blender add-on and performing a *trace* or *pack* operation on the currently open .blend file will result in the following error upon trying to save the .blend file. ![unnamed.png](https://archive.blender.org/developer/F7272476/unnamed.png) **Exact steps for others to reproduce the error** I have attached a simple add-on for the purposes of demonstrating this behavior. It adds a panel to blender with a single button called "Trace" which will trace the currently open blend file if it has been saved. [BAT_bug_addon.zip](https://archive.blender.org/developer/F7272931/BAT_bug_addon.zip) ![unnamed.png](https://archive.blender.org/developer/F7273187/unnamed.png) To reproduce: - Open blender - Install provided add-on - Save a scene so that that the file exists on disk - Click 'trace' - The add-on should print all external assets that the file references to the console (This works fine - as far as I can tell BAT correctly traces all assets) - Attempt to save the .blend file again and you should see the '*Unable to make version backup*' error. I believe the issue is that BAT is not correctly closing files on Windows. I have tried manually calling the blender_asset_tracer.blendfile.close_all_cached() method but nothing changes.

Oliver Dawes commented

2019-06-19 18:22:02 +02:00

Added subscriber: @haiku

Oliver Dawes changed title from ~~[Blender Asset Tracer]~~ to [Blender Asset Tracer] Files are not closed on Windows after using BAT to trace .blend file in Add-on

2019-06-19 18:23:52 +02:00

Oliver Dawes commented

2019-06-19 18:28:01 +02:00

Process explorer show that once the 'trace' method is called Blender keeps open file handles for all .blend files found by BAT during the trace (This includes linked .blend files).

I can manually close these files as administrator using process explorer which resolves the saving issue but if I then try and trace again using the Add-on I get the following error:

Traceback (most recent call last):

File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\core.py", line 28, in execute
  for usage in block_usage_generator:
File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\__init__.py", line 59, in deps
  for block_usage in blocks2assets.iter_assets(block):
File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\blocks2assets.py", line 53, in iter_assets
  yield from block_reader(block)
File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\blocks2assets.py", line 159, in scene
  block_ed = block.get_pointer(b'ed')
File "S:\blender-addons\addons\gridmarkets_blender_addon\lib\blender_asset_tracer\blendfile\__init__.py", line 668, in get_pointer
  result = self.get(path, default=default)
File "S:\blender-addons\addons\gridmarkets_blender_addon\lib\blender_asset_tracer\blendfile\__init__.py", line 582, in get
  self.bfile.fileobj.seek(self.file_offset, os.SEEK_SET)

OSError: [Errno 9] Bad file descriptor

Process explorer show that once the 'trace' method is called Blender keeps open file handles for all .blend files found by BAT during the trace (This includes linked .blend files). I can manually close these files as administrator using process explorer which resolves the saving issue but if I then try and trace again using the Add-on I get the following error: Traceback (most recent call last): ``` File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\core.py", line 28, in execute for usage in block_usage_generator: File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\__init__.py", line 59, in deps for block_usage in blocks2assets.iter_assets(block): File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\blocks2assets.py", line 53, in iter_assets yield from block_reader(block) File "C:\Users\01\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\BAT_bug_addon\lib\blender_asset_tracer\trace\blocks2assets.py", line 159, in scene block_ed = block.get_pointer(b'ed') File "S:\blender-addons\addons\gridmarkets_blender_addon\lib\blender_asset_tracer\blendfile\__init__.py", line 668, in get_pointer result = self.get(path, default=default) File "S:\blender-addons\addons\gridmarkets_blender_addon\lib\blender_asset_tracer\blendfile\__init__.py", line 582, in get self.bfile.fileobj.seek(self.file_offset, os.SEEK_SET) ``` OSError: [Errno 9] Bad file descriptor

Oliver Dawes commented

2019-06-24 12:24:26 +02:00

I have been able to solve this issue by adding the following code:

For the deps() method intrace_init_.py just add a bfile.close() to the end of the method. This will close the main .blend file being traced after iterating through all the blocks.

def deps(bfilepath: pathlib.Path, progress_cb: typing.Optional[progress.Callback] = None) \
        -> typing.Iterator[result.BlockUsage]:
    """Open the blend file and report its dependencies.

    :param bfilepath: File to open.
    :param progress_cb: Progress callback object.
    """

    log.info('opening: %s', bfilepath)
    bfile = blendfile.open_cached(bfilepath)

    bi = file2blocks.BlockIterator()
    if progress_cb:
        bi.progress_cb = progress_cb

    - Remember which block usages we've reported already, without keeping the
    - blocks themselves in memory.
    seen_hashes = set()  # type: typing.Set[int]

    for block in asset_holding_blocks(bi.iter_blocks(bfile)):
        for block_usage in blocks2assets.iter_assets(block):

            usage_hash = hash(block_usage)
            if usage_hash in seen_hashes:
                continue
            seen_hashes.add(usage_hash)
            yield block_usage

    bfile.close()

All libraries also need closing (although not necessary to fix the original saving bug mentioned in the original post) since they also never get closed.
To do this just add a libfile.close() to the end of the _visit_linked_blocks() method in trace/file2blocks.py

    def _visit_linked_blocks(self, blocks_per_lib):
        - We've gone through all the blocks in this file, now open the libraries
        - and iterate over the blocks referred there.
        for lib_bpath, idblocks in blocks_per_lib.items():
            lib_path = pathlib.Path(lib_bpath.to_path())
            try:
                lib_path = lib_path.resolve()
            except FileNotFoundError:
                log.warning('Library %s does not exist', lib_path)
                continue

            log.debug('Expanding %d blocks in %s', len(idblocks), lib_path)
            libfile = blendfile.open_cached(lib_path)
            
            yield from self.iter_blocks(libfile, idblocks)
            libfile.close()

I have been able to solve this issue by adding the following code: For the *deps()* method in*trace\__init__.py* just add a *bfile.close()* to the end of the method. This will close the main .blend file being traced after iterating through all the blocks. ``` def deps(bfilepath: pathlib.Path, progress_cb: typing.Optional[progress.Callback] = None) \ -> typing.Iterator[result.BlockUsage]: """Open the blend file and report its dependencies. :param bfilepath: File to open. :param progress_cb: Progress callback object. """ log.info('opening: %s', bfilepath) bfile = blendfile.open_cached(bfilepath) bi = file2blocks.BlockIterator() if progress_cb: bi.progress_cb = progress_cb - Remember which block usages we've reported already, without keeping the - blocks themselves in memory. seen_hashes = set() # type: typing.Set[int] for block in asset_holding_blocks(bi.iter_blocks(bfile)): for block_usage in blocks2assets.iter_assets(block): usage_hash = hash(block_usage) if usage_hash in seen_hashes: continue seen_hashes.add(usage_hash) yield block_usage bfile.close() ``` All libraries also need closing (although not necessary to fix the original saving bug mentioned in the original post) since they also never get closed. To do this just add a *libfile.close()* to the end of the *_visit_linked_blocks()* method in *trace/file2blocks.py* ``` def _visit_linked_blocks(self, blocks_per_lib): - We've gone through all the blocks in this file, now open the libraries - and iterate over the blocks referred there. for lib_bpath, idblocks in blocks_per_lib.items(): lib_path = pathlib.Path(lib_bpath.to_path()) try: lib_path = lib_path.resolve() except FileNotFoundError: log.warning('Library %s does not exist', lib_path) continue log.debug('Expanding %d blocks in %s', len(idblocks), lib_path) libfile = blendfile.open_cached(lib_path) yield from self.iter_blocks(libfile, idblocks) libfile.close() ```

Sybren A. Stüvel commented

2019-06-24 15:54:02 +02:00

Added subscriber: @dr.sybren

Sybren A. Stüvel commented

2019-06-24 15:54:02 +02:00

Changed status from 'Open' to: 'Archived'

Sybren A. Stüvel closed this issue

2019-06-24 15:54:02 +02:00

Sybren A. Stüvel self-assigned this 2019-06-24 15:54:02 +02:00

Sybren A. Stüvel commented

2019-06-24 15:54:02 +02:00

Calling libfile.close() like that is not a good idea. Opening a blend file is the most expensive part of the tracing process, which is why it's cached in the first place by calling blendfile.open_cached(). Also in this example it's not called when an exception occurs, which means it can still keep files open.

These cached files should be closed by calling blender_asset_tracer.blendfile.close_all_cached(). For me this works fine:

Replay the steps as described by @haiku (saving the file indeed shows the described error)
Open the Python console, and type import blender_asset_tracer.blendfile; blender_asset_tracer.blendfile.close_all_cached()
Save the file. This time it is succesful.

The problem is not with BAT, but with the way it's loaded in your add-on. You add the lib path to sys.path so that BAT can import its own submodules from blender_asset_tracer, but you also import BAT as from .lib.blender_asset_tracer import trace. This means that BAT is now imported two times; once as bat_bug_addon.lib.blender_asset_tracer and once as blender_asset_tracer. The cached files are in blender_asset_tracer.blendfile._cached_bfiles, but I'm guessing you're calling bat_bug_addon.lib.blender_asset_tracer.blendfile.close_cached().

The solution is to do one of the two, but not both:

Add your lib directory to sys.path and import everything in there as if it's top-level.
Do not add lib to sys.path and import everything from bat_bug_addon.lib (and adjust whatever is in there to only use relative imports).

Calling `libfile.close()` like that is not a good idea. Opening a blend file is the most expensive part of the tracing process, which is why it's cached in the first place by calling `blendfile.open_cached()`. Also in this example it's not called when an exception occurs, which means it can still keep files open. These cached files should be closed by calling `blender_asset_tracer.blendfile.close_all_cached()`. For me this works fine: 1. Replay the steps as described by @haiku (saving the file indeed shows the described error) 2. Open the Python console, and type `import blender_asset_tracer.blendfile; blender_asset_tracer.blendfile.close_all_cached()` 3. Save the file. This time it is succesful. The problem is not with BAT, but with the way it's loaded in your add-on. You add the `lib` path to `sys.path` so that BAT can import its own submodules from `blender_asset_tracer`, but you also import BAT as `from .lib.blender_asset_tracer import trace`. This means that BAT is now imported two times; once as `bat_bug_addon.lib.blender_asset_tracer` and once as `blender_asset_tracer`. The cached files are in `blender_asset_tracer.blendfile._cached_bfiles`, but I'm guessing you're calling `bat_bug_addon.lib.blender_asset_tracer.blendfile.close_cached()`. The solution is to do one of the two, but not both: * Add your `lib` directory to `sys.path` and import everything in there as if it's top-level. * Do not add `lib` to `sys.path` and import everything from `bat_bug_addon.lib` (and adjust whatever is in there to only use relative imports).

Oliver Dawes commented

2019-06-24 17:42:36 +02:00

Thank you, that makes a lot of sense.

Sign in to join this conversation.