Page MenuHome

Add proxy support for all sequences
AbandonedPublic

Authored by Richard Antalik (ISS) on Aug 12 2018, 10:48 AM.
Tokens
"Like" token, awarded by mathers."Mountain of Wealth" token, awarded by jaker."Like" token, awarded by tintwotin."Like" token, awarded by mon.

Details

Summary

This patch will enable generating proxies for all picture generating sequences using established interface.
This applies to fallback AVI engine as well as FFmpeg engine.
Support for PNG AVI output is added to sequencer.

Output of non-movie and non-image(seq's without input file) sequences is always bound to setting "per project".
To this output path, path <scene-name>/<seq-name>/ will be appended.
good idea is to set "per project" path to correspond to .blend file name.

Sequences will not keep track of proxies. If you rename/copy scene/sequence, either rename proxy path or rebuild proxies.

big TODO's:

  • There should be a way to track health status of proxies - if parameters of sequence change, proxy should not be used anymore, rebuild is required, user should be notified.
  • I don't really know how to use animation system, so that part of code should be reviewed
  • thread mutex I am using is quick and dirty hack. it is still possible to crash blender due to this, but it should be very rare.

minor TODO's:

  • Filenames are created from sequence / scene names. This has to be sanitized.
  • there is 1 unfreed mem block. I don't know where...
  • I would move indexer.c from Imbuf module to space sequencer

Diff Detail

Event Timeline

Hi @Richard Antalik (ISS) I would like you to join a small slack workspace with myself and a few others who are interested in upgrading the VSE.
If you can share with me an email address I would be happy to send an invite to you.

Hi, it's richardantalik at google mail

Great, thanks! invite sent.

fix minor issues, that slipped

Richard Antalik (ISS) edited the summary of this revision. (Show Details)Aug 15 2018, 12:23 AM
Richard Antalik (ISS) updated this revision to Diff 11584.

Fix HUGE memory leak (still some bytes left, but not gigs of them at least :)
Add PNG / MJPEG codec selection to UI
Render smaller imbufs instead of post-scaling
Render everything to fixed size - so 25% proxy no longer means 25% of source resolution, but it is 25% of "project" resolution (scene->r.xsch)

Richard Antalik (ISS) edited the summary of this revision. (Show Details)Aug 17 2018, 8:56 PM
Richard Antalik (ISS) updated this revision to Diff 11612.

Fix issue with not deleting original proxy file
Add dirty mutex, so it is very unlikely to crash building proxies by human interaction, but I have to look deeper, how it is usualy done in blender.
For multithreading I am using my wm_jobs "hack" but I think, that it may be useful for this kind of job manager to be able to run more than one jobs.

Richard Antalik (ISS) edited the summary of this revision. (Show Details)Aug 23 2018, 10:21 PM
Richard Antalik (ISS) updated this revision to Diff 11667.

Refactor proxy API, 1st pass

Richard Antalik (ISS) edited the summary of this revision. (Show Details)Aug 26 2018, 8:12 PM
Richard Antalik (ISS) updated this revision to Diff 11691.

finished refactoring, movieclip still uses its own job manager, but could maybe use unified...
I don't know if there was multiview support, but now is.

Patch is pretty usable.
It may be annoying, when you want to change parameters of sequences, because you have to delete proxies to see effect of action. Which will probably push me to correct that, but it isn't matter of quick fix.

That being said, I will leave this patch as is for some time.

Add codec selection to set selected strip proxies operator

When disabling proxy, cache for sequence is invalidated. This is instead of delete proxy "button" until we can work with proxy health.

I try to compile this, but the diff file no longer works (can't patch). Can anyone guide me? Is it for the master branch? Is there any chance that this can work in Blender 2.80?

I try to compile this, but the diff file no longer works (can't patch). Can anyone guide me? Is it for the master branch? Is there any chance that this can work in Blender 2.80?

Hi, my plan is to maintain this in my own branch, so once beta is out, I will rebase this.
I am too busy now so unfortunately there may be some delay.

if you want win32 binary, you can download it there https://drive.google.com/open?id=112jLW3yjXIP0RHMiw6K6GUwtinX3QYpA

@Richard Antalik (ISS)
I've been playing around with your 'test patch D3597'

A few remarks:

  • The progress bar on the audio strip visually suggests that a proxy is being generated for the audio, even though it's not. (I would remove it)
  • Overwrite does not seem to work. (When set to disabled)

I think you'd have to enable the panel for the clip strips, though I'm not sure if this is outside of you plan (supporting clip strip proxies with and without distortion in the VSE)

Feature request:

Please... please... can we have the option to render image sequences.

They can save so much time by just deleting the frames you need re-rendered and then rendering with overwrite disabled.

Misc:

This is probably an issue with Blender itself, but if you select a location that is read only Blender will crash.
Same if the drive is full.

In your build I can't seem to add Image sequence strips.

@Richard Antalik (ISS)
I've been playing around with your 'test patch D3597'

  • The progress bar on the audio strip visually suggests that a proxy is being generated for the audio, even though it's not. (I would remove it)

This would be easy to solve.

  • Overwrite does not seem to work. (When set to disabled)

Will look at this.

I think you'd have to enable the panel for the clip strips, though I'm not sure if this is outside of you plan (supporting clip strip proxies with and without distortion in the VSE)

Did you build install target? I see file release/scripts/startup/bl_ui/space_sequencer.py included in patch.

Feature request:
Please... please... can we have the option to render image sequences.
They can save so much time by just deleting the frames you need re-rendered and then rendering with overwrite disabled.

Sure, we can pass ibufs to different encoding engine. Dunno if it's better to use older custom solution, or if ffmpeg can be configured to do such task.
As there are some TODO's still I would push this down the priority list.

Render mutex and invalidation may conflict with cache "remake" I am working on, therefore I want to wait for result of that patch before implementing.

Misc:
This is probably an issue with Blender itself, but if you select a location that is read only Blender will crash.
Same if the drive is full.

I am pretty sure, that blender has some functions built in to deal with this. It didn't come to my mind even that this would be an issue.

In your build I can't seem to add Image sequence strips.

Must look into this - it was not my intention.

Hi @Richard Antalik (ISS) , I'm trying to test your patch, but I am unable to apply it to the master branch. What is the base commit for this patch ?

Once I can build it, I think I can tidy this patch up a bit, remove warnings etc.

Hi @Richard Antalik (ISS) , I'm trying to test your patch, but I am unable to apply it to the master branch. What is the base commit for this patch ?
Once I can build it, I think I can tidy this patch up a bit, remove warnings etc.

You would have to look by date, but this patch needs to be reworked:

  • to keep things simple, code cleanup done in this patch should be dropped
  • multithreading code should be rewritten in similar fashion to D3934. Also it should be consulted with other developper, with more experience with FFMPEG, probably even with FFMPEG developper. Approach used here worked however...
    • use one proxy job, as controller for sub jobs
    • work on scene copy
    • there should be lock on strips, that are being processed)

Only strips with video file input should be supported. Instead of using proxy system to handle all strip types, cache with disk dumping will be used.
reasons are:

  • multiple processing steps in pipeline (proxies are ideally 1:1 copy, but optimized for scrubbing or scaled to increase performance)
  • changes in timeline will affect cached result, sometimes partially
    • invalidation is easier using cache
    • image export instead of ffmpeg must be used

If you want to help me with this, I will be happy to help / guide you in process.

You would have to look by date, but this patch needs to be reworked:

  • to keep things simple, code cleanup done in this patch should be dropped

You mean this patch should be re-written from scratch ?

  • multithreading code should be rewritten in similar fashion to D3934. Also it should be consulted with other developper, with more experience with FFMPEG, probably even with FFMPEG developper. Approach used here worked however...
    • use one proxy job, as controller for sub jobs
    • work on scene copy
    • there should be lock on strips, that are being processed)

I don't think we should generate several proxies in parallel :

  • when encoding a movie proxy, ffmpeg should already be multi-threaded when working on a single file (I'm not sure if it is implemented in blender, but it certainly works with ffmpeg cli)
  • when encoding a non-vse scene proxy (3d renders), the rendering process is already multi-threaded

By the way, when generating several proxies (25%...100%) for a non-vse scene (3d render), the fastest way would be to generate the highest-res proxy first, then downscale it to obtain the other proxy resolutions, right ? How feasible would this be ?

Only strips with video file input should be supported. Instead of using proxy system to handle all strip types, cache with disk dumping will be used.
reasons are:

  • multiple processing steps in pipeline (proxies are ideally 1:1 copy, but optimized for scrubbing or scaled to increase performance)
  • changes in timeline will affect cached result, sometimes partially
    • invalidation is easier using cache
    • image export instead of ffmpeg must be used

I'm not sure what you mean by that. I think it's important to separate proxies and pre-fetching. Actually I'm still not sure why we are switching from image sequences to avi files for proxies, I feel image files were more flexible and allow for easier scrubbing and pre-fetching.

If you want to help me with this, I will be happy to help / guide you in process.

Yes I'd like to spend a bit of time on these issues, but I feel this patch is getting way too big and hard to merge/apply. I think it's important to separate proxy generation, pre-fetching and interface changes in several incremental patches (trying to keep their size to <100 lines).

I'm not sure how the blender git tree is organized, but from my perspective it would make sense to develop these features in a separate branch, and to make minimal (and documented) changes to the API, so it is easier to merge in the master branch later.

You would have to look by date, but this patch needs to be reworked:

  • to keep things simple, code cleanup done in this patch should be dropped

You mean this patch should be re-written from scratch ?

Contributor has to decide ultimately. I don't have means to force this :)

I don't think we should generate several proxies in parallel :

  • when encoding a movie proxy, ffmpeg should already be multi-threaded when working on a single file (I'm not sure if it is implemented in blender, but it certainly works with ffmpeg cli)

Good - code will be cleaner. ultimately we should check if multithreading works properly.

  • when encoding a non-vse scene proxy (3d renders), the rendering process is already multi-threaded

By the way, when generating several proxies (25%...100%) for a non-vse scene (3d render), the fastest way would be to generate the highest-res proxy first, then downscale it to obtain the other proxy resolutions, right ? How feasible would this be ?

Not applicable - as I said this will be done by cache management. *Only* movies will have proxies.

Only strips with video file input should be supported. Instead of using proxy system to handle all strip types, cache with disk dumping will be used.
reasons are:

  • multiple processing steps in pipeline (proxies are ideally 1:1 copy, but optimized for scrubbing or scaled to increase performance)
  • changes in timeline will affect cached result, sometimes partially
    • invalidation is easier using cache
    • image export instead of ffmpeg must be used

I'm not sure what you mean by that. I think it's important to separate proxies and pre-fetching.

First let's clear definitions: Proxy is close enough representation of original file, pre-processed to have required properties(fast scrubbing, scaled), so working with it is more time efficient.

Prefetching has nothing to do with dumping cache.
As cache is generated, it can be stored on disk. When cache is polled for frame, it will look first in RAM, if frame is not found, is can look at disk - still faster then re-render in most cases.
this dumped data however *have to be* stored individually, as it may be requested to be deleted/flagged as out-of-date (properties of strip has changed -> invalidate cache).

Actually I'm still not sure why we are switching from image sequences to avi files for proxies, I feel image files were more flexible and allow for easier scrubbing and pre-fetching.

FFMPEG recoding is supposed to be faster then saving images.
As for speed of loading data, both methods have very similar performance.

If you want to help me with this, I will be happy to help / guide you in process.

Yes I'd like to spend a bit of time on these issues, but I feel this patch is getting way too big and hard to merge/apply. I think it's important to separate proxy generation, pre-fetching and interface changes in several incremental patches (trying to keep their size to <100 lines).
I'm not sure how the blender git tree is organized, but from my perspective it would make sense to develop these features in a separate branch, and to make minimal (and documented) changes to the API, so it is easier to merge in the master branch later.

If FFMPEG is keeping CPU busy even with 1 thread, there is nothing much to be done performancewise.

Some cleaning of old code, refactoring and updating UI to reflect reality
Small changes are good approach, but end goal should be discussed. With prefetching I changed same code number of times, because I was not familiar with basics, and therefore couldn't plan changes in detail.

By the way, when generating several proxies (25%...100%) for a non-vse scene (3d render), the fastest way would be to generate the highest-res proxy first, then downscale it to obtain the other proxy resolutions, right ? How feasible would this be ?

Not applicable - as I said this will be done by cache management. *Only* movies will have proxies.

I'm not sure what you mean by that. I think it's important to separate proxies and pre-fetching.

First let's clear definitions: Proxy is close enough representation of original file, pre-processed to have required properties(fast scrubbing, scaled), so working with it is more time efficient.
Prefetching has nothing to do with dumping cache.
As cache is generated, it can be stored on disk. When cache is polled for frame, it will look first in RAM, if frame is not found, is can look at disk - still faster then re-render in most cases.
this dumped data however *have to be* stored individually, as it may be requested to be deleted/flagged as out-of-date (properties of strip has changed -> invalidate cache).

Yes I'd like to spend a bit of time on these issues, but I feel this patch is getting way too big and hard to merge/apply. I think it's important to separate proxy generation, pre-fetching and interface changes in several incremental patches (trying to keep their size to <100 lines).
I'm not sure how the blender git tree is organized, but from my perspective it would make sense to develop these features in a separate branch, and to make minimal (and documented) changes to the API, so it is easier to merge in the master branch later.

Some cleaning of old code, refactoring and updating UI to reflect reality
Small changes are good approach, but end goal should be discussed. With prefetching I changed same code number of times, because I was not familiar with basics, and therefore couldn't plan changes in detail.

If I understand correctly, the plan is to :

  • Drop proxy support for all sequences except movies (it would also be dropped for clips and image sequences right ?). All proxies would be ffmpeg-encoded avi files, no image sequences.
  • Improve the RAM-cache already present in blender with a second-level disk-cache for all sequences

If that is correct, I think this should be split in two patches, one focusing on proxies, and a second one focusing on implementing disk cache (this should probably be in a new file ?)

I also think the title of this diff is pretty misleading since it actually removes proxy support for all sequences (except movies) and adds disk cache.

Since proxies are only for movies, I think ot would make sense to add a "batch process" feature so the user can generate proxies for all his footage before even importing them in blender. I find it frustrating to wait for proxy generation each time I import a clip.

The reason I mention this now is because this would require knowing in advance where the proxy files should be generated, but for now it is linked to each clip.

Should we make this "proxy path" a configuration a project-wide param ? Do we need a per-clip override ?

If I understand correctly, the plan is to :

  • Drop proxy support for all sequences except movies (it would also be dropped for clips and image sequences right ?). All proxies would be ffmpeg-encoded avi files, no image sequences.

Not quite. Movieclips do have "own" proxy management. Builder is shared, but you can not build proxies for movieclip from sequencer

Images are already loaded fast. If scaled proxy of image has to be used, we can smaller image. FFMPEG builder can be used, it would make consistent naming scheme, but other than that there are only disadvantages.

As I said, image sequences are arguable...

  • Improve the RAM-cache already present in blender with a second-level disk-cache for all sequences

If that is correct, I think this should be split in two patches, one focusing on proxies, and a second one focusing on implementing disk cache (this should probably be in a new file ?)

FFMPEG builder can be found in imbuf module file indexer.c. In sequencer proxy building is fragmented and all over sequencer code. this patch did clean it up, however since both sequencer and movieclip is using this API this turned out to be a little bit tricky.

Cache dumping should be in imbuf module in 'moviecache.c' as this cache is "universal" cache for storing images and it's used across blender. This way not only sequencer, but other projects can benefit also.

I also think the title of this diff is pretty misleading since it actually removes proxy support for all sequences (except movies) and adds disk cache.

Maybe usable, but very impractical disk cache...
This patch was done a lot earlier, then I was thinking about cache...

Since proxies are only for movies, I think ot would make sense to add a "batch process" feature so the user can generate proxies for all his footage before even importing them in blender. I find it frustrating to wait for proxy generation each time I import a clip.

How would you implement that?
There are scripts, addons and a lot of info how to create proxies outside(or even inside using non-standard methods) of Blender

The reason I mention this now is because this would require knowing in advance where the proxy files should be generated, but for now it is linked to each clip.
Should we make this "proxy path" a configuration a project-wide param ? Do we need a per-clip override ?

You can set 3 settings:

  • path per strip
    • Per strip is good option, if you want to move / archive data with proxies pre-built, or multiple projects are using same video files - no need to generate proxy per project
  • path per project
    • Per project setting is good, if you want to have only proxies groupped at one place.
  • mannual override
    • mannual override is necessary if you have 2 instances of one sequence, and for some reason you need to change proxy for another file, while other instance must use normal proxy file.

I don't see these settings as a complication for development or anything

If I understand correctly, the plan is to :

  • Drop proxy support for all sequences except movies (it would also be dropped for clips and image sequences right ?). All proxies would be ffmpeg-encoded avi files, no image sequences.

Not quite. Movieclips do have "own" proxy management. Builder is shared, but you can not build proxies for movieclip from sequencer

I see, so dropping proxies for all sequences except movie and movieclip

Images are already loaded fast. If scaled proxy of image has to be used, we can smaller image. FFMPEG builder can be used, it would make consistent naming scheme, but other than that there are only disadvantages.

Ok, let's say no image sequence proxies for now

  • Improve the RAM-cache already present in blender with a second-level disk-cache for all sequences

If that is correct, I think this should be split in two patches, one focusing on proxies, and a second one focusing on implementing disk cache (this should probably be in a new file ?)

FFMPEG builder can be found in imbuf module file indexer.c. In sequencer proxy building is fragmented and all over sequencer code. this patch did clean it up, however since both sequencer and movieclip is using this API this turned out to be a little bit tricky.

I'm not sure why we need to change this API ? I'll take a deeper look at your patch.

Cache dumping should be in imbuf module in 'moviecache.c' as this cache is "universal" cache for storing images and it's used across blender. This way not only sequencer, but other projects can benefit also.

Sure. I suppose we should use image sequences for this cache ?

I also think the title of this diff is pretty misleading since it actually removes proxy support for all sequences (except movies) and adds disk cache.

Maybe usable, but very impractical disk cache...
This patch was done a lot earlier, then I was thinking about cache...

There are scripts, addons and a lot of info how to create proxies outside(or even inside using non-standard methods) of Blender

In the long run, I think "asset preparation" should be an workspace of its own for the VSE, but let's keep this as a future project.

The reason I mention this now is because this would require knowing in advance where the proxy files should be generated, but for now it is linked to each clip.

Ok, I hadn't realized yet that proxy_dir was shared across all strips

Generally I find the proxy controls are extremely confusing in blender, especially for beginners :

  • In the VSE
    • The "proxy storage" option (project or strip) is a project-wide option, but it is available only when selecting a strip. It should be moved to user preferences, or left in the N-menu with its own header, always visible (same as "Annotations" is now)
    • The "proxy custom directory" and "proxy custom file" options for "per strip" proxies are confusing : is blender going to overwrite the file, or just going to read it ?
    • "Set Selected Strip Proxies" and "Rebuild Proxy and Timecode Indices" should be moved to a global toolbar menu, same as "proxy storage"
  • In the movie clip editor
    • It's not obvious that "proxy size" is related to the current display, and not the proxy building process

If that's ok with you I'll start cleaning up the code in small incremental patches, starting from the current master branch, 25772c9e1d2d5239e1f7f83f6ea0c4d7d1f1df4a

I'm not sure why we need to change this API ? I'll take a deeper look at your patch.

What can be done to proxies now is

  • cleanup in sequencer part
  • PNG codec support was not a bad idea. It has a bit worse performance, but it has transparency.
  • Even more demanding lossless codec can be implemented - FFv1, huffyuv...
  • it may be useful to introduce scale to specific size - VSE should support this as-is I think. Also See last section.

Cache dumping should be in imbuf module in 'moviecache.c' as this cache is "universal" cache for storing images and it's used across blender. This way not only sequencer, but other projects can benefit also.

Sure. I suppose we should use image sequences for this cache ?

Yes.

I also think the title of this diff is pretty misleading since it actually removes proxy support for all sequences (except movies) and adds disk cache.

Maybe usable, but very impractical disk cache...
This patch was done a lot earlier, then I was thinking about cache...
There are scripts, addons and a lot of info how to create proxies outside(or even inside using non-standard methods) of Blender

In the long run, I think "asset preparation" should be an workspace of its own for the VSE, but let's keep this as a future project.

Not sure, what do you mean by "asset preparation", but it sounds scary :)

Generally I find the proxy controls are extremely confusing in blender, especially for beginners :

Yes, UI is not the greatest. it is being worked on, but there are hotter issues. Mainly this prefetching... It should have be done by now, so it can be tested

  • In the movie clip editor
    • It's not obvious that "proxy size" is related to the current display, and not the proxy building process

When you choose size of 50% this will reduce movie resolution by 50%, then during rendering proxy is scaled up by 50%
This can be annoying, when you have files with more than one resolution.
For example When I use (""slow" motion") 320 x 240 movie along with FHD movie, make proxies for all movies, I end up with 160 x 120 file...
I think, that most users would want to have all proxies of the same resolution, as that is much closer to final render.

If that's ok with you I'll start cleaning up the code in small incremental patches, starting from the current master branch, 25772c9e1d2d5239e1f7f83f6ea0c4d7d1f1df4a

K, I will try to allocate some time for review.
Sequencer is not worked on as much as other modules, so there is little chance of conflicts with your patches.

@Richard Antalik (ISS) Reading through; I'm seeing a lot of conflicting communication here.
A design task clearly detailing the goals, timeframes, scopes and functionality would be desirable, so other developers and uses can see what you have planned and what you intend to fix and work on. (Something like T46049: Assets Integration in Blender with T59540#583476)

A design task clearly detailing the goals, timeframes, scopes and functionality would be desirable

Got this.
I intended to create some roadmap, I can define goals and abstract functionality.
Currently I should be working on prefetching, with aim to get it up and running in 2.8 release, which should be doable as code is already working.
I delved into a lot of stuff lately, maybe unnecessarily, well...

Ok so I will try to do some writing probably next week when I will have nothing better to do in work. I will probably state "obvious" things, which are broken, with solution being "it has to be dealt with". So I won't be able to provide precise timeframe, but it will be roadmap nonetheless...

A design task clearly detailing the goals, timeframes, scopes and functionality would be desirable, so other developers and uses can see what you have planned and what you intend to fix and work on. (Something like T46049: Assets Integration in Blender with T59540#583476)

Done : T60912

For now I'm trying to focus on small changes that improve the user experience with maximum chances of being ready for the 2.8 release