VSE: Thumbnails are exposing the content of the source material and not the trimmed strip #91591

New Issue

tintwotin · 2021-09-22T09:18:05+02:00

tintwotin commented

2021-09-22 09:18:05 +02:00

System Information
Operating system:
Graphics card:

Blender Version
Broken: (example: 2.80, edbf15d3c0, master, 2018-11-28, as found on the splash screen)

Long description of error
Now the strategy chosen for the thumbnails has reached its end and has been committed, I think that it is necessary to speak out loud about the downsides to this strategy. As I understand it, it is using a frame from every n frame of the source material and drawing it on the strip and apparently it needs to do this for every strip even though its from the same source, which makes thumbnails very slow at showing up at each strip:
{F10539393,size=full}
(The one file Tears of Steel has been shot detected, so each split point is a new shot)

As shown here doesn't the thumbnail represent the content of the actual strip, because numerous strips gets the same thumbnail, as if all of them were the same shot:
{F10539374,size=full}

This can also lead to cases where the thumbnail is not showing the content of the strip at all:
{F10539466,size=full}

This can be quite confusing for users.

In devtalk and pm's, early on I have discussed the strategy for the thumbnails and also referenced different sources revolving around the purpose of showing the thumbnails. In most softwares it is the accurate first and last frame which have the highest priority, so it can be used for trimming.

Ex. Kdenlive:
{F10539559,size=full}

You could argue that "Preview during transform" fills this need, but it isn't on by default and no one knows about it.

The other purpose of showing thumbnails on strips is to id them, but as shown above also here is the current strategy not really solving the problem as it may not show the thumbnail of the actual strip.

If ID'ing and accurate start end end thumbnails was chosen as strategy it would look like this in a very early(and slow) iteration of the thumbnail patch:
{F10539630,size=full}

This way the users will actually see thumbnails representing the content of the strip and not potentially some thumbnail from the content before it. I know the challenge is speed, but I feel it is necessary to make you guys @fsiddi @Sergey @pablovazquez aware that the current strategy can produce wrong thumbnails before bug reports on this are parked as known issues.

Example file showing wrong thumbnails(link tears of steel to the strips manually):
thumbs_wrong.blend

**System Information** Operating system: Graphics card: **Blender Version** Broken: (example: 2.80, edbf15d3c044, master, 2018-11-28, as found on the splash screen) **Long description of error** Now the strategy chosen for the thumbnails has reached its end and has been committed, I think that it is necessary to speak out loud about the downsides to this strategy. As I understand it, it is using a frame from every n frame of the source material and drawing it on the strip and apparently it needs to do this for every strip even though its from the same source, which makes thumbnails very slow at showing up at each strip: {[F10539393](https://archive.blender.org/developer/F10539393/thumb_update2.gif),size=full} (The one file Tears of Steel has been shot detected, so each split point is a new shot) As shown here doesn't the thumbnail represent the content of the actual strip, because numerous strips gets the same thumbnail, as if all of them were the same shot: {[F10539374](https://archive.blender.org/developer/F10539374/thumb_update_wide.gif),size=full} This can also lead to cases where the thumbnail is not showing the content of the strip at all: {[F10539466](https://archive.blender.org/developer/F10539466/thumb_update_off.gif),size=full} This can be quite confusing for users. In devtalk and pm's, early on I have discussed the strategy for the thumbnails and also referenced different sources revolving around the purpose of showing the thumbnails. In most softwares it is the accurate first and last frame which have the highest priority, so it can be used for trimming. Ex. Kdenlive: {[F10539559](https://archive.blender.org/developer/F10539559/fb5e301ba18eaaf1e7c433056d2a50e7b0333b91.gif),size=full} You could argue that "Preview during transform" fills this need, but it isn't on by default and no one knows about it. The other purpose of showing thumbnails on strips is to id them, but as shown above also here is the current strategy not really solving the problem as it may not show the thumbnail of the actual strip. If ID'ing and accurate start end end thumbnails was chosen as strategy it would look like this in a very early(and slow) iteration of the thumbnail patch: {[F10539630](https://archive.blender.org/developer/F10539630/5ccc637b7333dc07830c3d87186fff0e077ad9a3.gif),size=full} This way the users will actually see thumbnails representing the content of the strip and not potentially some thumbnail from the content before it. I know the challenge is speed, but I feel it is necessary to make you guys @fsiddi @Sergey @pablovazquez aware that the current strategy can produce wrong thumbnails before bug reports on this are parked as known issues. Example file showing wrong thumbnails(link tears of steel to the strips manually): [thumbs_wrong.blend](https://archive.blender.org/developer/F10539660/thumbs_wrong.blend)

tintwotin commented

2021-09-22 09:18:05 +02:00

Added subscribers: @Sergey, @fsiddi, @pablovazquez, @tintwotin

Philipp Oeser commented

2021-09-22 09:32:28 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2021-09-22 09:32:28 +02:00

Changed status from 'Needs Triage' to: 'Needs Developer To Reproduce'

Philipp Oeser commented

2021-09-22 09:32:28 +02:00

I can confirm the behavior, but since I havent followed to discussion about the strategy closely, I would kindly ask the aforementioned devs to classify this

Aditya Y Jeppu commented

2021-09-22 11:23:32 +02:00

Added subscriber: @quantimoney

Aditya Y Jeppu commented

2021-09-22 11:23:32 +02:00

Ok so two things - even with the same source file for different strips we do have the issue of reloading, but this is because caching is done per seq, or per strip. We would have to match the source to a strip, and then have a quick lookup to get the seq with the same source which has already cached the images. Thinking about it now, it is weird that the caching is done per strip, so maybe I am wrong here and there are ways to cache per source also.

the second is the correct display of thumbnails for the content in the source footage - Even if we did use some sort of averaging to get the approximate point where the thumbnail will correctly represent the frames that are spanned by the thumbnail image, it will not be perfect and you will get cases where the frame just before a new movie scene is taken for the thumbnail frame.

Keeping this in mind :

we could find an easy enough fix for caching images based on source files and not based on strips (again not sure if this doesn't exist yet).
For the correct thumbnail -> we could have the first frame's image (where the left handle of the strip is) be taken for the thumbnail, keeping the position of the thumbnail same (i.e. not this (as it might get confusing) but in this way

The initial versions long ago had the real time updation of the left most frame but it was slow and will be very slow now with caching also involved, so rather than real time display, we can update it when the user stops moving the handles. (have something like V2D_IS_HANDLE_MOVING flag? )

Ok so two things - even with the same source file for different strips we do have the issue of reloading, but this is because caching is done per seq, or per strip. We would have to match the source to a strip, and then have a quick lookup to get the seq with the same source which has already cached the images. Thinking about it now, it is weird that the caching is done per strip, so maybe I am wrong here and there are ways to cache per source also. the second is the correct display of thumbnails for the content in the source footage - Even if we did use some sort of averaging to get the approximate point where the thumbnail will correctly represent the frames that are spanned by the thumbnail image, it will not be perfect and you will get cases where the frame just before a new movie scene is taken for the thumbnail frame. Keeping this in mind : * we could find an easy enough fix for caching images based on source files and not based on strips (again not sure if this doesn't exist yet). * For the correct thumbnail -> we could have the first frame's image (where the left handle of the strip is) be taken for the thumbnail, keeping the position of the thumbnail same (i.e. not this (as it might get confusing) ![moving_handle_with_thumb.gif](https://archive.blender.org/developer/F10540806/moving_handle_with_thumb.gif) but in this way ![moving_handle_without_thumb.gif](https://archive.blender.org/developer/F10540809/moving_handle_without_thumb.gif) The initial versions long ago had the real time updation of the left most frame but it was slow and will be very slow now with caching also involved, so rather than real time display, we can update it when the user stops moving the handles. (have something like `V2D_IS_HANDLE_MOVING` flag? )

Richard Antalik commented

2021-09-22 11:51:13 +02:00

Added subscriber: @iss

Richard Antalik commented

2021-09-22 11:51:13 +02:00

I agree, that first and maybe last thumbnail should be always presented, so this behavior should be improved. Then all other thumbnails should follow in regular frequency as is done now. I was looking into changing that before comitting, but way it was implemented, I wasn't sure if I would be able to implement it well enough.

In #91591#1223698, @quantimoney wrote:
Ok so two things - even with the same source file for different strips we do have the issue of reloading, but this is because caching is done per seq, or per strip. We would have to match the source to a strip, and then have a quick lookup to get the seq with the same source which has already cached the images. Thinking about it now, it is weird that the caching is done per strip, so maybe I am wrong here and there are ways to cache per source also.

Changing cache design wouldn't help with performance too much, but it would help reduce memory needed, which is good thing too. I have mentioned this during development, that it is aspect should be changed in (ideally near) future.

Implementing preview during transform as feature for thumbnails isn't bad idea. I would insist on this being done asynchronously in same way as thumbnail job works right now though.

I agree, that first and maybe last thumbnail should be always presented, so this behavior should be improved. Then all other thumbnails should follow in regular frequency as is done now. I was looking into changing that before comitting, but way it was implemented, I wasn't sure if I would be able to implement it well enough. > In #91591#1223698, @quantimoney wrote: > Ok so two things - even with the same source file for different strips we do have the issue of reloading, but this is because caching is done per seq, or per strip. We would have to match the source to a strip, and then have a quick lookup to get the seq with the same source which has already cached the images. Thinking about it now, it is weird that the caching is done per strip, so maybe I am wrong here and there are ways to cache per source also. Changing cache design wouldn't help with performance too much, but it would help reduce memory needed, which is good thing too. I have mentioned this during development, that it is aspect should be changed in (ideally near) future. Implementing preview during transform as feature for thumbnails isn't bad idea. I would insist on this being done asynchronously in same way as thumbnail job works right now though.

Francesco Siddi commented

2021-09-22 23:36:02 +02:00

I suggest to implement thumbnail update using this method:

making sure that the thumbnail itself reflects the frame at the cut point. This way the interface will require minimal updates (only one thumbnail).

I suggest to implement thumbnail update using this method: ![moving_handle_without_thumb.gif](https://archive.blender.org/developer/F10540809/moving_handle_without_thumb.gif) making sure that the thumbnail itself reflects the frame at the cut point. This way the interface will require minimal updates (only one thumbnail).

Richard Antalik commented

2021-09-22 23:48:30 +02:00

In #91591#1224126, @fsiddi wrote:
I suggest to implement thumbnail update using this method:

making sure that the thumbnail itself reflects the frame at the cut point. This way the interface will require minimal updates (only one thumbnail).

I don't quite understand what you mean by this. Is it that only thumbnail should change and not it's position? Then do you think this is better, than recalculating subsequent thumbnails after moving handle? Because for implementation there is very little difference.

> In #91591#1224126, @fsiddi wrote: > I suggest to implement thumbnail update using this method: > > ![moving_handle_without_thumb.gif](https://archive.blender.org/developer/F10540809/moving_handle_without_thumb.gif) > > making sure that the thumbnail itself reflects the frame at the cut point. This way the interface will require minimal updates (only one thumbnail). I don't quite understand what you mean by this. Is it that only thumbnail should change and not it's position? Then do you think this is better, than recalculating subsequent thumbnails after moving handle? Because for implementation there is very little difference.

Francesco Siddi commented

2021-09-23 00:03:01 +02:00

Yes, the thumbnail stays in place, as well as all subsequent thumbnails. The only thing changing is the content of the thumbnail. The update should happen as you transform the end of the strip.

tintwotin commented

2021-09-23 00:20:15 +02:00

@fsiddi

In #91591#1224126, @fsiddi wrote:
I suggest to implement thumbnail update using this method:

making sure that the thumbnail itself reflects the frame at the cut point. This way the interface will require minimal updates (only one thumbnail).

Back in the day Adytia and I went over the various options. ^In this solution you'll have to be aware that unless you see the full first thumbnail, you may miss the part which is moving. Then you can try to tinker with ex. letting the next thumbnail start moving when ex. 70% is covered of the first thumbnail:
{F10551111,size=full}

Slide:
{F10551178,size=full}

Or keep the first frame below locked to the handle position(resulting in the same problem):
{F10551126,size=full}

But in the end of the day the only solution where the user can see the first and last thumbnail clearly is the first example with the first and last thumbnail position locked to the handles and the other images slide under them.
{F10539630,size=full}
(If the first and last frame can't be shown in full, they should divide the strip space between them)

@fsiddi > In #91591#1224126, @fsiddi wrote: > I suggest to implement thumbnail update using this method: > > ![moving_handle_without_thumb.gif](https://archive.blender.org/developer/F10540809/moving_handle_without_thumb.gif) > > making sure that the thumbnail itself reflects the frame at the cut point. This way the interface will require minimal updates (only one thumbnail). Back in the day Adytia and I went over the various options. ^In this solution you'll have to be aware that unless you see the full first thumbnail, you may miss the part which is moving. Then you can try to tinker with ex. letting the next thumbnail start moving when ex. 70% is covered of the first thumbnail: {[F10551111](https://archive.blender.org/developer/F10551111/In_trim_50_b.gif),size=full} Slide: {[F10551178](https://archive.blender.org/developer/F10551178/In_trim_slide.gif),size=full} Or keep the first frame below locked to the handle position(resulting in the same problem): {[F10551126](https://archive.blender.org/developer/F10551126/slip_weird2.gif),size=full} But in the end of the day the only solution where the user can see the first and last thumbnail clearly is the first example with the first and last thumbnail position locked to the handles and the other images slide under them. {[F10539630](https://archive.blender.org/developer/F10539630/5ccc637b7333dc07830c3d87186fff0e077ad9a3.gif),size=full} (If the first and last frame can't be shown in full, they should divide the strip space between them)

Richard Antalik commented

2021-09-23 00:27:35 +02:00

In #91591#1224132, @fsiddi wrote:
The update should happen as you transform the end of the strip.

I think this is best to evaluate in practice, because there are 2 ways this can work:

Blocks UI, but is always correct. This can cause choppy interface with slow files. - OK for precision work
Asynchronous, thumbnail is updated when it is ready, which may feel weird with slow files. - Not great for precision work

I have noticed especially with debug build in SF edit, even starting the job is causing UI to freeze, I assume this is because list of strips is built and not reused. We have tested this during development, but not with 1K strips though... So This should be looked at first probably.

I will summarize this into task that is bit easier to follow

> In #91591#1224132, @fsiddi wrote: > The update should happen as you transform the end of the strip. I think this is best to evaluate in practice, because there are 2 ways this can work: - Blocks UI, but is always correct. This can cause choppy interface with slow files. - OK for precision work - Asynchronous, thumbnail is updated when it is ready, which may feel weird with slow files. - Not great for precision work I have noticed especially with debug build in SF edit, even starting the job is causing UI to freeze, I assume this is because list of strips is built and not reused. We have tested this during development, but not with 1K strips though... So This should be looked at first probably. I will summarize this into task that is bit easier to follow

Francesco Siddi commented

2021-09-23 00:40:01 +02:00

As a final remark: thumbnails are meant to be a visual aid. Precision trimming and sliding should be performed by looking at the Preview window (this functionality is not there currently, and that is worth spending time on). What @tintwotin shows in the first clip is acceptable, although I would limit it to one thumbnail only and not affect the next thumbnail.

Richard Antalik commented

2021-09-23 01:02:05 +02:00

Changed status from 'Needs Developer To Reproduce' to: 'Archived'

Richard Antalik closed this issue

2021-09-23 01:02:05 +02:00

Richard Antalik commented

2021-09-23 01:02:05 +02:00

Right, I have created #91618, if I have missed something please remind me.

Now I will close this post.

Right, I have created #91618, if I have missed something please remind me. Now I will close this post.

tintwotin commented

2021-09-23 06:24:00 +02:00

In #91591#1224161, @iss wrote:
Right, I have created #91618, if I have missed something please remind me.

How about a mock-up of how you envision the look and behaviour, as mentioned there are several options.

In #91591#1224148, @fsiddi wrote:
Precision trimming and sliding should be performed by looking at the Preview window (this functionality is not there currently, and that is worth spending time on).

Maybe Preview During Transform in the Preview's View menu should be on by default and maybe not hidden away in the Preview menu?
{F10553963,size=full}

{F9805804, size=full}

Patch: https://developer.blender.org/D10424

> In #91591#1224161, @iss wrote: > Right, I have created #91618, if I have missed something please remind me. How about a mock-up of how you envision the look and behaviour, as mentioned there are several options. > In #91591#1224148, @fsiddi wrote: > Precision trimming and sliding should be performed by looking at the Preview window (this functionality is not there currently, and that is worth spending time on). Maybe `Preview During Transform` in the Preview's View menu should be on by default and maybe not hidden away in the Preview menu? {[F10553963](https://archive.blender.org/developer/F10553963/image.png),size=full} {[F9805804](https://archive.blender.org/developer/F9805804/transform_preview3.gif), size=full} Patch: https://developer.blender.org/D10424

Francesco Siddi commented

2021-09-28 14:36:28 +02:00

In #91591#1224235, @tintwotin wrote:
How about a mock-up of how you envision the look and behaviour, as mentioned there are several options.

The idea is to refresh only the first thumbnail, on transform. Easier to show through implementation I think.

In #91591#1224148, @fsiddi wrote:
Precision trimming and sliding should be performed by looking at the Preview window (this functionality is not there currently, and that is worth spending time on).

Maybe Preview During Transform in the Preview's View menu should be on by default and maybe not hidden away in the Preview menu?

I agree this should be on by default, and it probably touches upon the VSE workspace startup template and the default one. Is this something you @tintwotin or Richard could look into?

> In #91591#1224235, @tintwotin wrote: > How about a mock-up of how you envision the look and behaviour, as mentioned there are several options. > The idea is to refresh only the first thumbnail, on transform. Easier to show through implementation I think. >> In #91591#1224148, @fsiddi wrote: >> Precision trimming and sliding should be performed by looking at the Preview window (this functionality is not there currently, and that is worth spending time on). > Maybe `Preview During Transform` in the Preview's View menu should be on by default and maybe not hidden away in the Preview menu? I agree this should be on by default, and it probably touches upon the VSE workspace startup template and the default one. Is this something you @tintwotin or Richard could look into?

tintwotin commented

2021-09-30 06:57:11 +02:00

In #91591#1227318, @fsiddi wrote:

In #91591#1224235, @tintwotin wrote:
How about a mock-up of how you envision the look and behaviour, as mentioned there are several options.

The idea is to refresh only the first thumbnail, on transform. Easier to show through implementation I think.

The industry standard is to refresh the first AND the last thumbnail. No other software is only refreshing the first thumbnail?

In #91591#1224148, @fsiddi wrote:
Precision trimming and sliding should be performed by looking at the Preview window (this functionality is not there currently, and that is worth spending time on).

Maybe Preview During Transform in the Preview's View menu should be on by default and maybe not hidden away in the Preview menu?

I agree this should be on by default, and it probably touches upon the VSE workspace startup template and the default one. Is this something you @tintwotin or Richard could look into?

If you want this done, someone who gets patches committed, should do it.

> In #91591#1227318, @fsiddi wrote: >> In #91591#1224235, @tintwotin wrote: >> How about a mock-up of how you envision the look and behaviour, as mentioned there are several options. >> > > The idea is to refresh only the first thumbnail, on transform. Easier to show through implementation I think. The industry standard is to refresh the first AND the last thumbnail. No other software is only refreshing the first thumbnail? > >>> In #91591#1224148, @fsiddi wrote: >>> Precision trimming and sliding should be performed by looking at the Preview window (this functionality is not there currently, and that is worth spending time on). >> Maybe `Preview During Transform` in the Preview's View menu should be on by default and maybe not hidden away in the Preview menu? > > I agree this should be on by default, and it probably touches upon the VSE workspace startup template and the default one. Is this something you @tintwotin or Richard could look into? If you want this done, someone who gets patches committed, should do it.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VSE: Thumbnails are exposing the content of the source material and not the trimmed strip #91591