Flamenco render job submission fails #65726

Closed
opened 2019-06-11 15:58:19 +02:00 by Joel Howe · 18 comments

System Information
Operating system: Windows 10
Graphics card: AMD Radeon (TM) R9 Fury Series

Blender Version
Broken: version: 2.79 (sub 7), branch: blender2.7, commit date: 2019-04-09 16:45, hash: 10f724cec5e3

Submitting certain .blend files to the Flamenco manager results in an error, apparently occurring when copying associated assets from the .blend file to the Flamenco render job folder. Trying multiple files, I found that I can generate this error with an audio file in the video sequencer or when I have an external Alembic file referenced absolutely in a Mesh Sequence Cache modifier, even if that file is in an accessible location.

Exception while running task:
Exception from asynchronous task
Traceback (most recent call last):
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 280, in _stop_async_task
    self.async_task.result()  # This re-raises any exception of the task.
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 369, in async_execute
    outdir, outfile, missing_sources = await self.bat_pack(job_id, filepath)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 616, in bat_pack
    relative_only=relative_only)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\bat_interface.py", line 168, in copy
    await loop.run_in_executor(None, packer.execute)
  File "C:\blender-2.79-latest\blender-2.79.0-git.10f724cec5e3-active\2.79\python\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.1.2-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 375, in execute
    self._rewrite_paths()
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.1.2-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 462, in _rewrite_paths
    assert bfile_pp is not None
AssertionError

Sample file with WAV sound in the video sequencer and an Alembic file reference.
DevTest.blend

**System Information** Operating system: Windows 10 Graphics card: AMD Radeon (TM) R9 Fury Series **Blender Version** Broken: version: 2.79 (sub 7), branch: blender2.7, commit date: 2019-04-09 16:45, hash: 10f724cec5e3 Submitting certain .blend files to the Flamenco manager results in an error, apparently occurring when copying associated assets from the .blend file to the Flamenco render job folder. Trying multiple files, I found that I can generate this error with an audio file in the video sequencer or when I have an external Alembic file referenced absolutely in a Mesh Sequence Cache modifier, even if that file is in an accessible location. ``` Exception while running task: Exception from asynchronous task Traceback (most recent call last): File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 280, in _stop_async_task self.async_task.result() # This re-raises any exception of the task. File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 369, in async_execute outdir, outfile, missing_sources = await self.bat_pack(job_id, filepath) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 616, in bat_pack relative_only=relative_only) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\bat_interface.py", line 168, in copy await loop.run_in_executor(None, packer.execute) File "C:\blender-2.79-latest\blender-2.79.0-git.10f724cec5e3-active\2.79\python\lib\concurrent\futures\thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.1.2-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 375, in execute self._rewrite_paths() File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.1.2-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 462, in _rewrite_paths assert bfile_pp is not None AssertionError ``` Sample file with WAV sound in the video sequencer and an Alembic file reference. [DevTest.blend](https://archive.blender.org/developer/F7098695/DevTest.blend)
Author

Added subscriber: @JoelHowe

Added subscriber: @JoelHowe
Sybren A. Stüvel was assigned by Brecht Van Lommel 2019-06-11 16:22:25 +02:00

Which version of the Blender Cloud add-on was used?

Which version of the Blender Cloud add-on was used?
Author

In #65726#699528, @dr.sybren wrote:
Which version of the Blender Cloud add-on was used?

Latest version as of yesterday (1.13.4) but with the wheel file you had sent me (attached here)
blender_asset_tracer-1.1.2-py3-none-any.whl.

> In #65726#699528, @dr.sybren wrote: > Which version of the Blender Cloud add-on was used? Latest version as of yesterday (1.13.4) but with the wheel file you had sent me (attached here) [blender_asset_tracer-1.1.2-py3-none-any.whl](https://archive.blender.org/developer/F7098908/blender_asset_tracer-1.1.2-py3-none-any.whl).

Please give us files we can use to reproduce the issue. The blend file you uploaded packs just fine, albeit with warnings about missing files.

It would also help to know the layout of your files. What directory is set as the project directory? You're using drive R:, is that a network drive (if so, which protocol?) or a local disk?

Please give us files we can use to reproduce the issue. The blend file you uploaded packs just fine, albeit with warnings about missing files. It would also help to know the layout of your files. What directory is set as the project directory? You're using drive R:, is that a network drive (if so, which protocol?) or a local disk?
Author

Adding a sound file to the sequencer in an empty .blend file was enough to reproduce the error for me, but I am attaching this actual file. This file has the proprietary elements removed, including the WAV file in the sequencer. To get this to submit to Flamenco without error, I have to delete the WAV audio and both the Alembic fluid surface and the Ocean modifier fluid surface.
Flamenco-Error.blend

Adding a sound file to the sequencer in an empty .blend file was enough to reproduce the error for me, but I am attaching this actual file. This file has the proprietary elements removed, including the WAV file in the sequencer. To get this to submit to Flamenco without error, I have to delete the WAV audio and both the Alembic fluid surface and the Ocean modifier fluid surface. [Flamenco-Error.blend](https://archive.blender.org/developer/F7100246/Flamenco-Error.blend)
Author

I have been able to successfully submit to Flamenco a version of the file that has the object with an Ocean modifier, but only when the Cache path for that modifier was empty.

UPDATE: It looks like just changing Cache path helped submit this, as I wasn't able to submit today with an empty value but then changing it to "//" a relative reference to the current folder, it submitted.

I have been able to successfully submit to Flamenco a version of the file that has the object with an Ocean modifier, but only when the Cache path for that modifier was empty. UPDATE: It looks like just changing Cache path helped submit this, as I wasn't able to submit today with an empty value but then changing it to "//" a relative reference to the current folder, it submitted.

In #65726#699782, @dr.sybren wrote:
It would also help to know the layout of your files. What directory is set as the project directory? You're using drive R:, is that a network drive (if so, which protocol?) or a local disk?

As an addition to these not-yet-answered questions: what have you set as job storage and job output directories? With the Flamenco-Error.blend file you're using the D: drive, what kind of drive is that?

> In #65726#699782, @dr.sybren wrote: > It would also help to know the layout of your files. What directory is set as the project directory? You're using drive R:, is that a network drive (if so, which protocol?) or a local disk? As an addition to these not-yet-answered questions: what have you set as job storage and job output directories? With the Flamenco-Error.blend file you're using the D: drive, what kind of drive is that?
Author

D:/ is a local physical data drive on my main Win 10 workstation.
R:/ is the Flamenco project.
R:/ is a network mapped folder location to D:/Flamenco, so in the case of submitting the job, R:/ points to the same hard drive. R:/ was created using Windows "Map network drive", which I don't believe allows you to specify a protocol.
All Flamenco workers have an R:/ mapped back to my workstation, which is also running Flamenco manager.
Inside R:/ are three folders:

  • JobStorage - Flamenco writes its files to these folders
  • SourceFiles - I save a copy of .blend files to be rendered here, as I have to be in the Flamenco project (R:) to submit.
  • TempImages - Folder for the standard JPEG images being written. This is essentially for display on the Flamenco manager screen last rendered image. I purge this folder a lot, as I use the compositing node editor to generate multi-EXR files.
D:/ is a local physical data drive on my main Win 10 workstation. R:/ is the Flamenco project. R:/ is a network mapped folder location to D:/Flamenco, so in the case of submitting the job, R:/ points to the same hard drive. R:/ was created using Windows "Map network drive", which I don't believe allows you to specify a protocol. All Flamenco workers have an R:/ mapped back to my workstation, which is also running Flamenco manager. Inside R:/ are three folders: - JobStorage - Flamenco writes its files to these folders - SourceFiles - I save a copy of .blend files to be rendered here, as I have to be in the Flamenco project (R:) to submit. - TempImages - Folder for the standard JPEG images being written. This is essentially for display on the Flamenco manager screen last rendered image. I purge this folder a lot, as I use the compositing node editor to generate multi-EXR files.

@JoelHowe can you try with this BAT wheel file? Just remove any exist blender_asset_tracer*.whl file from the wheels directory of the Blender Cloud add-on, then put this one in place and restart Blender.

blender_asset_tracer-1.2.dev1-py3-none-any.whl

@JoelHowe can you try with this BAT wheel file? Just remove any exist `blender_asset_tracer*.whl` file from the `wheels` directory of the Blender Cloud add-on, then put this one in place and restart Blender. [blender_asset_tracer-1.2.dev1-py3-none-any.whl](https://archive.blender.org/developer/F7773226/blender_asset_tracer-1.2.dev1-py3-none-any.whl)
Author

Error is still occurring for both 2.79 and 2.80, but only when a WAV file is in the video sequencer and I try to submit to Flamenco. The .blend files don't submit and get an error as below, with the display in the Flamenco Render tab reading "Found asset Narration-Audio.wav" when it stops packing. All files submit properly once the audio track has been removed, which is an acceptable workaround for me.


Exception while running task:
Exception from asynchronous task
Traceback (most recent call last):
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 280, in _stop_async_task
    self.async_task.result()  # This re-raises any exception of the task.
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 369, in async_execute
    outdir, outfile, missing_sources = await self.bat_pack(job_id, filepath)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 616, in bat_pack
    relative_only=relative_only)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\bat_interface.py", line 168, in copy
    await loop.run_in_executor(None, packer.execute)
  File "C:\Blender\blender-2.79.0-git.10f724cec5e3-active\2.79\python\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 383, in execute
    self._rewrite_paths()
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 470, in _rewrite_paths
    assert bfile_pp is not None
AssertionError
<Task finished coro=<FLAMENCO_OT_render.async_execute() done, defined at C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py:255> exception=AssertionError()>: resulted in exception
Traceback (most recent call last):
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 95, in kick_async_loop
    res = task.result()
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 280, in _stop_async_task
    self.async_task.result()  # This re-raises any exception of the task.
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 369, in async_execute
    outdir, outfile, missing_sources = await self.bat_pack(job_id, filepath)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 616, in bat_pack
    relative_only=relative_only)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\bat_interface.py", line 168, in copy
    await loop.run_in_executor(None, packer.execute)
  File "C:\Blender\blender-2.79.0-git.10f724cec5e3-active\2.79\python\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 383, in execute
    self._rewrite_paths()
  File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 470, in _rewrite_paths
    assert bfile_pp is not None
AssertionError
Error is still occurring for both 2.79 and 2.80, but only when a WAV file is in the video sequencer and I try to submit to Flamenco. The .blend files don't submit and get an error as below, with the display in the Flamenco Render tab reading "Found asset Narration-Audio.wav" when it stops packing. All files submit properly once the audio track has been removed, which is an acceptable workaround for me. ``` Exception while running task: Exception from asynchronous task Traceback (most recent call last): File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 280, in _stop_async_task self.async_task.result() # This re-raises any exception of the task. File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 369, in async_execute outdir, outfile, missing_sources = await self.bat_pack(job_id, filepath) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 616, in bat_pack relative_only=relative_only) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\bat_interface.py", line 168, in copy await loop.run_in_executor(None, packer.execute) File "C:\Blender\blender-2.79.0-git.10f724cec5e3-active\2.79\python\lib\concurrent\futures\thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 383, in execute self._rewrite_paths() File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 470, in _rewrite_paths assert bfile_pp is not None AssertionError <Task finished coro=<FLAMENCO_OT_render.async_execute() done, defined at C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py:255> exception=AssertionError()>: resulted in exception Traceback (most recent call last): File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 95, in kick_async_loop res = task.result() File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\async_loop.py", line 280, in _stop_async_task self.async_task.result() # This re-raises any exception of the task. File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 369, in async_execute outdir, outfile, missing_sources = await self.bat_pack(job_id, filepath) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\__init__.py", line 616, in bat_pack relative_only=relative_only) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\flamenco\bat_interface.py", line 168, in copy await loop.run_in_executor(None, packer.execute) File "C:\Blender\blender-2.79.0-git.10f724cec5e3-active\2.79\python\lib\concurrent\futures\thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 383, in execute self._rewrite_paths() File "C:\Users\joel\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\blender_cloud\wheels\blender_asset_tracer-1.2.dev1-py3-none-any.whl\blender_asset_tracer\pack\__init__.py", line 470, in _rewrite_paths assert bfile_pp is not None AssertionError ```
Author

I have discovered that the error during the job submission can be prevented if "Relative paths only" is turned on in the Blender Cloud settings. I am not sure if that will introduce other issues, or if this is on by default, but it appears that it addresses the issue.
Capture.PNG

I have discovered that the error during the job submission can be prevented if "Relative paths only" is turned on in the Blender Cloud settings. I am not sure if that will introduce other issues, or if this is on by default, but it appears that it addresses the issue. ![Capture.PNG](https://archive.blender.org/developer/F7785201/Capture.PNG)

I suspect it has to do with the behaviour of Path.resolve() (doc). AFAIK on Windows it also "resolves" mapped drive letters to their \\SERVER\share\path\to\file notation, which is a pain in the rear end.

@JoelHowe can you install Python 3.7, run pip install -U blender-asset-tracer and give us the output of bat list R:/project/thefile.blend ? You can also try bat pack -p R:/project R:/project/thefile.blend testpack; this will create a directory testpack in the current working directory, and put the BAT pack in there.

Testing with BAT directly will probably be easier than trying to use Blender with the Cloud add-on, and can provide us with more detailed info. If you replace bat in the above commands with bat -d you should get debug-level messages.

I suspect it has to do with the behaviour of `Path.resolve()` ([doc](https://docs.python.org/3/library/pathlib.html#pathlib.Path.resolve)). AFAIK on Windows it also "resolves" mapped drive letters to their `\\SERVER\share\path\to\file` notation, which is a pain in the rear end. @JoelHowe can you install Python 3.7, run `pip install -U blender-asset-tracer` and give us the output of `bat list R:/project/thefile.blend` ? You can also try `bat pack -p R:/project R:/project/thefile.blend testpack`; this will create a directory `testpack` in the current working directory, and put the BAT pack in there. Testing with BAT directly will probably be easier than trying to use Blender with the Cloud add-on, and can provide us with more detailed info. If you replace `bat` in the above commands with `bat -d` you should get debug-level messages.
Author

Unfortunately I won't have time to do this, but on a positive note I am unable to replicate this issue in Blender 2.81 with latest Blender Cloud plugin.

Unfortunately I won't have time to do this, but on a positive note I am unable to replicate this issue in Blender 2.81 with latest Blender Cloud plugin.

Added subscriber: @MaciejJutrzenka

Added subscriber: @MaciejJutrzenka

so u are saying it all works fine with 2.81? correct? that would means that we can close this as fixed/.

so u are saying it all works fine with 2.81? correct? that would means that we can close this as fixed/.
Author

It is not 100% fixed, but I can confirm that I have certain .blend files with audio tracks in them that submit properly now in 2.81. I am not sure why some others don't submit to Flamenco, but there is the easy workaround of deleting the audio track before submitting the render or turning on "Relative paths only" in the Cloud plugin settings.

It is not 100% fixed, but I can confirm that I have certain .blend files with audio tracks in them that submit properly now in 2.81. I am not sure why some others don't submit to Flamenco, but there is the easy workaround of deleting the audio track before submitting the render or turning on "Relative paths only" in the Cloud plugin settings.

Changed status from 'Open' to: 'Archived'

Changed status from 'Open' to: 'Archived'

I'll close this as 'Archived' then; if you know of a way for us to reproduce the issue we can always reopen.

I'll close this as 'Archived' then; if you know of a way for us to reproduce the issue we can always reopen.
Sign in to join this conversation.
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: archive/blender-cloud#65726
No description provided.