Page MenuHome

Sequencer problem w/ variable rate MJPEG
Closed, ArchivedPublic

Description

System Information
Windows 8

Blender Version
Broken: tested on 2.75 and 2.76-rc3

Short description of error
When effect strips are put over video strips, the timing and everything seems to screw up more than usual. It's erratic. Sometimes the video will look one way, sometimes another. Try rendering it. Usually the output video file does not match the timeline preview.

Exact steps for others to reproduce the error

  1. Watch the sequence by pressing Alt-A. Sometimes it displays the proper way and sometimes it doesn't. Restarting the program may change the displayed video. It's haunted!
  2. Press Ctrl-F12 to render out the video. Usually the video corresponds to the improper way, not matching the intention of the edits.

Details

Type
Bug

Event Timeline

Fluffy Rabbit (fluffrabbit) created this task.
Fluffy Rabbit (fluffrabbit) raised the priority of this task from to Needs Triage by Developer.
Campbell Barton (campbellbarton) triaged this task as Confirmed, Medium priority.

Confirmed. It looks like there is frame-skipping in the cache, which is worse when effects strips are used. (since playback ends up skipping more frames).

For me, if I change the playback option from AV-Sync to No-Sync (save and reload).
Then the problem goes away. Can you confirm this?

This may be some issue with random access & ffmpeg.

I have not confirmed the problem going away by doing that. I don't see a No-Sync option in 2.76-rc3 though I disabled AV-Sync. I'm still seeing the middle clip playing too fast, which I guess differs from the gremlin manifestation of before. Video also renders out the same incorrect way as it did before.

I'm going to go drown my sorrows. At least somebody acknowledged the problem as when I was talking about it in #blendervse some filmmaker was trying to impress upon me a philosophy about codecs and how ffmpeg uses playback codecs thereby with some differing interest (ffmpeg codecs proclaimed by him to be for real-time playback and not rendering).

With no sync playback seems to match rendering but the middle clip is still glitched out. It's too fast. That was not how I cut it.

Also, now that I have the ball rolling, I think proxy media may be broken as well.

@Fluffy Rabbit (fluffrabbit), could you try a simpler coded (one that doesn't use keyframes - MJPEG for example). to see if thats the issue?

Actually Campbell, all of the clips are indeed MJPEG.

A frame sequence might take us a step closer. I'll see if I can figure out how to do that.

I made a test case with frame sequences. Based on the results, here is what I think is happening: Video coming off of this particular low-end camera is recorded in a variable-framerate optimized stills format, which is commonly used in cases where subsequent frames in an intra-only format are identical, for example when "auto slow shutter" kicks in and the format is MJPEG. turning the AVIs into frame sequences resulted in a strip that speeds up and slows down according to available light, an interesting phenomenon that is consistent through subsequent playbacks and rendering. So yes, it appears that effect strips somehow exacerbate the problem, but (depending on who you want to blame) the problem is that Blender is incorrectly or inconsistently handling variable-framerate videos, at least in the VSE.

Blender uses ffmpeg for decoding - AFAIK it supposed to manage format/codec details for us,
assigning to sergey

Campbell Barton (campbellbarton) renamed this task from [VSE] timing screws up when effect strips are used to Sequencer problem w/ variable rate MJPEG.Oct 8 2015, 12:17 AM

With ffmpeg you typically have to do a lot of stuff manually to get it right. For instance those video clips included in the first test case were re-encoded with ffmpeg from the camera source files, with ffmpeg maintaining the original frame pattern exactly as it was. In order to get 25 actual frames per second to match the reported framerate when transcoding, something else would have to be typed into the command line. I'm not exactly sure how this particular thing applies to Blender and how they dance though.

Does Sergey even exist, or is he just some black hole you assign unsolvable problems?

Sergey Sharybin (sergey) lowered the priority of this task from Confirmed, Medium to Needs Information from User.Oct 8 2015, 10:21 AM

No, i obviously don't exist.

Please try generating timecodes for the video strips and use something like "Record Run". It'll make random seek in the video streams more reliable.

Thank you, Sergey. That did not fix the problem. Behavior is the same after applying timecodes. Did you try it yourself in this specific case? When it renders out we've got those damn glitches again.

Regarding timecode, Record Run is sometimes not enough with some files, please also try with Record Run No Gaps.

I tried doing it with Record Run No Gaps and it did not solve the problem. What did solve the problem was transcoding the clips using the ffmpeg video filter -vf "fps=25". Since it's a video filter I'm sure it would be a simple matter of having Blender apply it during playback and rendering to enforce the reported framerate of the file. The nature of these things is such that it wouldn't affect how Blender handles clip timing with project framerates. I'm attaching a zip containing these transcoded files so you can see the effect.

bump

Mark this as "confirmed".

Please put away forum-ish "bump" approach. Reports are being reviewed as soon as developers has time for this.

As for the files you're attaching here as demonstration of issue -- it's all really undefinitive, because:

  • Original file has AV-sync enabled
  • Image sequence test file has no AV-sync, additionally it uses different frame range of the video
  • Third video also has no AV-sync, but seems to have proper frame range.

So it's hard to tell what exact timing you're considering correct. But that being said i don't see difference in behavior between original testfile with timecode generated and testfile3. Did you rebuild timecodes for all the strips (with all strips selected Strip -> Rebuild Proxy and Timecode indices)?

@Sergey-

The only difference between the original and testscene3 is the video clips themselves, not the blendfile. I observe a huge difference in behavior. When you "render it out" (which is when Blender creates a video file from the contents of the timeline), the timing is correct and the speed of movement is constant. In the blendfile with image sequences, the sequence was (ignorantly) edited to match the intended timing as closely as possible before I realized that simply outputting an image sequence from the video file is incorrect in this case, because in this case the video files have a variable framerate, which is not something Blender has ever been equipped to handle. I have suggested how this problem might be solved, and to illustrate what could be I had to take the unorthodox step of changing the video files, not the blendfile, to reflect how Blender should treat the original clips, as the new clips have their longer frames repeated so that there are exactly 25 frames per second rather than a variable number, which is what a variable framerate is. I really don't have a better way of explaining this.

To reiterate my previous comment, ffmpeg, which is the library that Blender has been declared to use, has a video filter called fps, which stands for "frames per second". An example of use:

Let's say we have a video clip of Sergey taking two swigs of vodka. Let's say we are editing a video in which we only want to show him taking one swig. Let's pretend for a minute that our name is Blender. Maybe our brain has been put in a blender, because sometimes the number of frames we see indicate the video being cut to show him taking one swig, sometimes two. We have an extra arm called ffmpeg, and ffmpeg has the muscle memory to show to us the single-swig clip. Maybe the problem isn't our brain but the film we are cutting. We look closer at the strip and see that some of the frames are stretched out to fill the length of multiple frames, and some of the frames are of normal length. Maybe the camera that exposed those frames is the one who has been drinking too much, because of course the proper protocol would be to simply repeat a frame rather than stretching it out. Good thing our extra arm ffmpeg knows the protocols, because it can do a cool trick called "fps", which shows us multiple identical frames rather than one stretched out frame, thus we see Sergey taking one swig of vodka rather than two.

My knowledge of this does not extend as far as the ffmpeg library, but I know that the ffmpeg executables include the fps video filter and it is very fast. I also know that variable-framerate video files indicate what their correct framerate is supposed to be regardless of how many frames they actually include, so for example if our video file indicates 25 FPS, we tell ffmpeg fps=25.

If the video stream has proper i-frames applying video filter will not change behavior of how this video appears in blender. This is because we don't do time-based frame mapping but doing frame number mapping.

What makes sense for robust use of video files in blender is properly set i-frames in the video stream. There are two ways to make sure no bad i-frames are used:

  • Transcode video externally, which will correct all headers and i-frames. This is basically what you did by transcoding with video filter (transcoding without filter applied will likely also solve the issue).
  • Generate timecode from blender, which roughly speaking stores mapping between frame number and according packet in the stream for that frame. What is being done by enabling Proxy/Timecode for all strips, choosing Record Run as a timecode, and hitting Strip -> Rebuild Proxy and Timecode.

Here's a render of testscene.blend with proper timecodes generated and used:


And here's render of testscene3.blend without any extra changes:

There's slight difference in certain areas of video (1 frame difference timing), which could be caused by difference in how blender and your transcode application repairs i-frames. But all in all i don't see issues with the way how blender rendered the file.

I do not get that output file when using Record Run. Maybe you confused which video files you were using. I am attaching a new file which includes the original video clips and the original blendfile using "Record Run" on all of the clips.

To illustrate how much the problem is NOT solved, I am attaching a new test case which follows your instrustions with the original video clips and the resulting output file. Please tell me if I am using Record Run correctly.

And no, simply transcoding with ffmpeg would not fix the issue. You have to use fps=25 otherwise this happens. I know this because I already transcoded the clips to a lower resolution before making these tests. FFMPEG normally likes to preserve frame configuration. Blender uses FFMPEG. Timecodes don't solve the problem, and the frames are not "broken", they merely use variable length, and the video editing situation with Blender would be just as robust if it dealt with long frames as repeating frames. In that way let's think of it in terms of 3 framerates: the framerate of the project, the framerate of the clip, and the framerate of the frame. Hopefully ffmpeg's realtime FPS video filter abstracts away the framerate of the frame so that you only have the framerate of the project and the framerate of the clip.

Did you deliberately didn't include BL_proxy folder into the archive or you forgot to actually generate proxy/timecodes for your videos?

Not sure why transcoding with fps=25 will have something visible, the input files are already 25fps.

Timecodes seem to somehow depend on proxies; one can't exist without the other. Somehow building actual proxies with Record Run seemed, at least for now, to fix the problem. I'm attaching a fifth test case to illustrate this.

Not sure why transcoding with fps=25 will have something visible, the input files are already 25fps.

Without timecodes, variable framerate video files are not handled properly by Blender. Transcoding with fps=25 makes FFMPEG create an actual 25 frames per second rather than unknown frames per second. Fun fact: Standards-compliant MJPEG AVIs can tell you they have any framerate in the world, but players are supposed to look at the length of each frame to play back a file properly rather than going through at a fixed rate, and different frames can have different lengths, so by their very nature variable framerate files are problematic if a program assumes that the reported "25fps" is accurate, since only sometimes will it be 25. Other times it could be 12.5, 20, 3, et cetera within a "25fps" video stream. A "25fps" 3-second long video is theoretically supposed to have 75 frames, but if it's variable-framerate it may only have 20 frames, and maybe for only 1/10th of a second it's 25fps. This is a standard, as screwy as it is. It's a way of efficiently storing identical frames or long pauses.

Sergey Sharybin (sergey) closed this task as Archived.Nov 3 2015, 12:33 PM

Without timecodes, variable framerate video files are not handled properly by Blender.

This is more a correct statement and that's why timecodes were added in the first place. Sure enough making timecode automatically is something we should do and in fact it's in our TODO. But not really considered a bug for now.

Thanks for the report anyway.