Page MenuHome

Sequencer proxies and cache
Open, NormalPublic

Description

This design task is a WIP detailing our work on the Sequencer proxy system. For now, we are mostly interested in features that most improve the sequencer UX with minimal work. Feel free to add anything.

Timeframe :
This should ideally be done for the 2.8 release (I don't exactly know the deadline)

Goals (in loose priority order ?) :

  • Improve the sequencer proxy UI to make it easier for beginners to generate and use proxies
  • Now that movie proxies only use avi files, we should remove all dead code associated with the generation of png proxies
  • Prefetching with basic disk cache (@Richard Antalik (ISS)) ?
  • Allow to pre-render a complex strip (think scene clip with a 3d render, or meta-strip) to see a smooth playback (1)

Scope :

  • Caching/prefetching/proxying in the Sequencer, with minimal changes to the API
  • Small incremental patches (<500 LOC each ?) that each deal with a specific issue

Notes :
(1) This was actually very easy to do before proxy support was removed for scene strips. Generating a proxy for a scene strip would simply render the scene and generate png's in the proxy dir. I think we should bring this pack (with another name, since proxy is reserved for movie strips), probably as part of disk cache or prefetching, or as a standalone option.

Details

Type
Design

Event Timeline

Improve the sequencer proxy UI : D4262

Excellent!

So, the way I see it is the following (for terms of functionality, in order of priority):

  1. Movie strip Proxy file
    • As video proxies are standard, and expected, in pretty much all NLEs.
  2. Image strip Proxy file
    • Depending on the compression and codec, image files can be slow to decode.
  3. Scene strip Proxy file
    • Essentially normal rendering but for proxies. see UX section below (This is a special case, as it's unique to Blender.)
  4. All other strip types* are to use a cache
    • As this data is generated on the fly and is not an asset outside of Blender.

*Excluding Sound and Movie Clip.


In terms of the User Experience I see:

  • The option to select between a real time proxy cache and a rendered proxy file (we have this already, just not in plain language).
    • An example would be for a Scene strip. If you were using cycles a rendered proxy file would be the choice. If EEVEE is used then real time proxy cache can be used.
  • Rendering a rendered proxy file will always be at the user's discretion. A prefetch cache, or real time proxy cache when applicable, will be on by default.

Hello,
I'm not sure to understand , do you plan to remove proxy made of image sequence ?
I'm using VSE a lot to edit 3D renders, converting .png image sequences (the actual renders) to .jpeg allow to have a very smooth playback.
I find it quite convenient also because there is a direct relation between the render and the proxy file.
Also it's great for scrubbing in the timeline because frame aren't dependent of other ( unlike h264 ).

In the other hand, having movie proxy for movie file make sense.

Hello,
I'm not sure to understand , do you plan to remove proxy made of image sequence ?

Not now, not until equivalent or better system will be in place
Playback of images should be fast without proxies. Proxies may make sense to be used for 4K or bigger resolution Image sequences(sets of images instead movie file) with a lot of processing, or some demanding formats.

As for proxy Vs disk cache - effect will be the *same*, just user will have to click on different buttons. Proxy for images will be duplicated feature...

@paul szajner (szap) Proxy support for scenes has been temporarily removed in the master branch, because the proxy function was originally designed only for movie strips.

We are trying to bring this back, with a better interface.

Apparently FFMPEG avi proxies encodes and decodes faster than a sequence of jpegs, and is generally cleaner. Scrubbing shouldn't be a problem because inter-frame compression is (or should be, I haven't investigated yet) disabled for proxy files. There is also an overhead cost when opening many files (in the worst case they may be on a network drive even if that's a terrible idea).


@Richard Antalik (ISS) maybe we should generatlize "Scene strip Proxy file" to meta strips too, this would be similar to After Effects' precomposition tool, and could be extremely useful to make heavily-composed scenes with real-time preview.

Pros :

  • Make a meta strip with heavy effects (time mapping, blurs, many layers), put it in a meta strip, pre-compose it, then add color correction with real-time preview
  • This system could be re-used for "scene strip proxies"

Cons :

  • Heavy compositing is usually done in the compositor, and would be included as a scene, not a meta strip

What do you think ? Should we have "precomposition" only for scene strips or meta strips too ? FWIW it used to be possible to hack this with proxies, so it shouldn't be too hard to implement (see my patch in the comments of T54048)

@Richard Antalik (ISS) also we should clearly define the difference between disk cache, pre-composition (or pre-render) and proxy files. I propose the following :

  • Proxy files are generated from movie files, there is a 1-to-1 correspondence between an external video file and a proxy (no effects, no transparency, etc.). The proxy may have a different resolution, and is only generated when the user explicitly asks for it. The option to use proxy is attached to the viewer (same as now)
  • Pre-composition files are image sequences generated from meta strips or scene strips, that have the output resolution (1). They are generated when the user asks for it, and always used when available. The option to use the precomp is attached to the strip (2).
  • Disk cache files are image sequences generated from the output of the sequencer (3). This cache is initially stored in RAM, then dumped to disk when the memory limit is exceeded. These images are the result of the full composition (4), and are invalidated when a strip is moved, or anything is changed in the sequencer. This allows for real-time preview and scrubbing, even in scenes with heavy compositing. Pre-fetching is used.

Points where I am unsure :

  • (1) maybe we should generate proxies of precomps ? A pre-composed meta or scene strip essentially behaves as a movie file, and high-resolution playback will surely be an issue. This is a hard problem, if the user sets proxies for a scene, should we regenerate proxies when a pre-comp (render) is triggered ?
  • (2) I think this is best, because you may want to temporarily disable precomp for a meta that takes 1s/frame to generate, but keep it for another meta that takes 30s/frame. But this is debatable, and may also apply to proxies (maybe you want 25% resolution for 4K footage, but 100% res for low-res footage in the same scene)
  • (3) What happens when you set a viewer window to another sequencer layer ? Should we generate cache for all layers ? Only the active one ? Only the "0" (final output) layer ?
  • (4) I think this is best, because individual strips that are neither meta nor movie shoud be very fast to generate. The thing we really need to cache (because it is slow) is effects (time control, blur, etc.)

Thoughts on these points ?

There are some misconceptions here.

  • Disk cache files are image sequences generated from the output of the sequencer (3). This cache is initially stored in RAM, then dumped to disk when the memory limit is exceeded. These images are the result of the full composition (4), and are invalidated when a strip is moved, or anything is changed in the sequencer. This allows for real-time preview and scrubbing, even in scenes with heavy compositing. Pre-fetching is used.

This is the main one. As I said I will try to write, how sequencer cache works(or rather will work). It can be added to documentation for reference maybe.
tl;dr: 4 steps of composing image are cached. 2 or 3 for individual strips(raw image(ffmpeg output), preprocessed(scaling, color stuff, filters,...), composite(after alpha over with another strip))
output of sequencer is the last item, that is cached. Currently this is color corrected and cached again, but in some different cache - this has to be changed yet so last item can be post-color correction

dumping raw image to disk has identical effect as building image based proxy.
however cache *can* detect changes in content, invalidate sections and rebuild them. That is, why using proxies will not make much sense.

@Richard Antalik (ISS) Hmm okay, so there will be caches on several levels when compositing. A RAM cache is already implemented, right ? Is it a strip-level or global-level cache ?

Thanks a lot for the answer !
disk cache as a replacement for proxy seems very promising !

As a side note :

maybe we should generatlize "Scene strip Proxy file" to meta strips too, this would be similar to After Effects' precomposition tool, and could be extremely useful to make heavily-composed scenes with real-time preview.

What would be handy too is proxy for adjustment layers as they're very useful for color grading and (in 2.79) we quickly loose realtime playback even with proxified sequences underneath.
But if I understand correctly disk cache solve all this in a better way.

@Richard Antalik (ISS) Disk cache would be implemented at the level of IMB right ? Is this going to be enough to cache everything we need ? Do we need to implement cache invalidation code or is it going to work natively through IMB's reference counting ? I did a simple test with a blur effect and it looks like cache invalidation works fine as it is.

If that is the case, these are the only things we need to do for the cache side of this task :

  • Cleanup the code ?
  • Add option to disable disk cache in user prefs
  • Implement prefetching
  • Implement disk cache

Note : do you concur that there is no hope of keeping the disk cache when we close and re-open blender ?

I implemented some pre-fetching using ImBuf's own caching system (which takes care of invalidation etc.) : D4288

There are still a few issues when the available memory is not enough to pre-fetch enough frames.

I'll issue a cleanup diff first, then try to fix this pre-fetching memory issue.

I cleaned up an old unused threading api : D4289