Audio will not keep sync with video at 20 FPS.
Closed, InvalidPublic

Description

System Information
Microsoft Windows 8.1 (6.3.9600), 6GB RAM, x64 AMD A6-5200 APU w/ Radeon 8400 R3 series (512mb) Driver ver: 7.1.542.0

Blender Version
2.76b f337fea

Short description of error
When using the Video Editing to rejoin a 20 FPS WMV to its (modified) sound, it syncs up during playback in the editor but always goes out of sync in the final output (H.264/AAC, OggTheora/Vorbis, mpeg, etc.) after a short time (if it was ever in sync to begin with).

I have tried everything... And by that I mean different combinations of settings & codecs, all at the 20 FPS... including importing the video and audio before/after frame rate change... And yes, also the various sync settings (I have zero problem syncing in the editor, only final output).

The only thing that seemed to get around the final output's A/V sync problem was pre-converting my source video to 29.97 FPS. Only then did the issue (completely) disappear. If nothing else, a simple warning informing the user that they are about to blow an entire day, by trying to save a few frames, would be nice =]

Exact steps for others to reproduce the error

  1. Open Blender 2.76b and choose "Video Editing" for "Screen Layout".
  2. Set FPS to Custom -> 20.
  3. Add a video file and an audio file that are to remain in sync, together, for the duration (~1min.)
  4. Play the project in the editor and notice that the audio & video remain in sync for the entire duration.
  5. [CTRL]+[F12] to "Render Animation" (or just use the big button in Render properties). -> Out of sync!

Thank you!

Details

Type
Bug
vegan (vegAnaize) updated the task description. (Show Details)
vegan (vegAnaize) raised the priority of this task from to Needs Triage.
vegan (vegAnaize) added a project: BF Blender.
vegan (vegAnaize) set Type to Bug.

I've tried it with & without AV-sync. I also tried Frame Drop.

do you have an example wmv & blend file?

Bastien Montagne (mont29) triaged this task as Incomplete priority.Feb 29 2016, 7:55 AM

Yes, we need example data here. As well as precise ffmpeg settings used. Encoding is complex topic, many things can go wrong there.

Also, please try upcoming 2.77 release (or the latest build from our buildbot), it uses updated ffmpeg lib.

The particular workflow, that I am using, doesn't utilize a .blend file... unless you would like me to somehow export one that includes my specific settings. Others seem to be experiencing a similar/exact problem, as well, with no mention of WMV:
http://blender.stackexchange.com/a/13679/22331

Here is a link to my specific test video (it was a test video to begin with; nothing impressive):
https://www.dropbox.com/s/dv2kvks8emsdesm/ScreenCapture_2-28-2016%2012.45.57%20PM.wmv?dl=1

Here is a link to the modified audio (not required unless problem won't reproduce without it):
https://www.dropbox.com/s/n7xwvl4nar74xca/ScreenCapture_2-28-2016%2012.45.57%20PM.output.wav?dl=1

Please remember that the source video is @ 20 FPS and that I would (preferably) like to keep it at that rate.
Thank you

I'm experiencing the same exact problem in version 2.77 RC1.

I don't know how to discover the options that FFMPEG is using. If someone could enlighten me, I'd be happy to share.

I've created a "wmv.blend" project file that should reflect the (mostly default) settings that I am trying to use to produce this 20 FPS video...

.blend --> https://www.dropbox.com/s/85en6muaucu1xsh/wmv.blend?dl=1

And the .WMV video...
https://www.dropbox.com/s/dv2kvks8emsdesm/ScreenCapture_2-28-2016%2012.45.57%20PM.wmv?dl=1

^ The .wmv and .blend file are referenced from my C:\temp\ folder (in case you need to duplicate that structure)

Bastien Montagne (mont29) raised the priority of this task from Incomplete to Confirmed.Mar 11 2016, 4:47 PM

OK, problem here is with audio sampling.

  • Original wmv file is 44.1kHz
  • Blender internal settings is 48kHz (by default).
  • Output file claims to be 44.1kHz again, and audio is clearly too long compared to video.

@Joerg Mueller (nexyon), @Sergey Sharybin (sergey), you should know better what's going on here?

Cool. Nice work! That doesn't sound like a bug, per se, because it makes sense (logically).

The fact that the SCENE tab is set @ 44.1 kHz, but rendering 48 kHz, obviously is the bug.

When this issue is corrected, it would be nice to have some preset choices for popular/common frequency rates. Like those here:
http://manual.audacityteam.org/index.php?title=Sample_rates

and/or here:
https://en.wikipedia.org/wiki/Sampling_%28signal_processing%29#Sampling_rate

Thank you very much! Blender & team rock!

I'm sorry, but you are wrong. Audio works fine in the original file and in the rendered output, the problematic part is the video which in the rendered result seems to run at a faster speed than the original. I tried this by playing back both videos at the same time.

So what's up with this? Bug or not?

It's definitely not an audio bug. I think it's a video bug, blender plays correctly, but the rendered output is wrong.

Rendering out the audio separately as a flac is the same (though when aligning it it is 1 frame off)

Rendering out video + audio via ffmpeg produces audio that doesnt stay in sync.

To me this sounds like a ffmpeg exporting problem

All of these work fine; Audio/Video stays in sync and framerate remains correct (bps varies);
They all keep the correct length (9m:51s):

$ ffmpeg -i infile.wmv -vcodec copy -acodec copy outfile.wmv
$ ffmpeg -i infile.wmv -vcodec copy -acodec pcm_s16le -ac 1 -ar 44100 outfile.mkv
#WARNING: The following command will produce a huge file...
$ ffmpeg -i infile.wmv -vcodes huffyuv -acodec pcm_s16le -ac 1 -ar 44100 outfile.mkv
$ ffmpeg -i infile.wmv -vcodec libx264 -ar 44100 outfile.mkv
$ ffmpeg -i infile.wmv -vcodec libx264 outfile.mkv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec aac -b:a 192k outfile.mkv
$ ffmpeg -i infile.wmv -vcodec libx264 -ar 48000 outfile.mkv

Blender 2.76b
Render:
FPS --> 20
Output --> H.264 RGB
Encoding --> AVI H.264
Audio Codec --> MP3 192
^ DOESN"T SYNC; VIDEO IS AHEAD OF AUDIO (and took forever to render vs. command-line ffmpeg)

Maybe Blender is using buggy ffmpeg library? My tests were done with (same system bug report is filed against and) ffmpeg:
ffmpeg version N-78758-g5156578 Copyright (c) 2000-2016 the FFmpeg developers

built with gcc 5.3.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av

isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --
enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-l
ibilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enab
le-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --en
able-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --ena
ble-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --
enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-lib
x265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-d
ecklink --enable-zlib

libavutil      55. 19.100 / 55. 19.100
libavcodec     57. 27.100 / 57. 27.100
libavformat    57. 26.100 / 57. 26.100
libavdevice    57.  0.101 / 57.  0.101
libavfilter     6. 37.100 /  6. 37.100
libswscale      4.  0.100 /  4.  0.100
libswresample   2.  0.101 /  2.  0.101
libpostproc    54.  0.100 / 54.  0.100

Hyper fast Audio and Video encoder

Sorry, forgot to test the identical (from Blender) on the command-line:

$ ffmpeg -i infile.wmv -vcodec libx264 -acodec mp3 -b:a 192k outfile,avi

^ Works, but causes Media Player Classic Home Cinema (MPC-HC) v1.7.10 to report total length as 9m:52s (one extra second). However, MediaInfo v0.7.83 still reports the file as being 9m:51s long.

SIDENOTE: When rendered in Blender (with any codec combo) the resulting file is obviously out of sync by the 40 second mark.

Blender uses FFmpeg 2.8.4. Don't know if it's buggy or not without doing tests tho. You can try building blender with system's FFmpeg or test if the issue could be reproduced with CLI of FFmpeg 2.8.4.

It looks like the original video might be missing frames. Converting the MKV directly with ffmpeg (2.8.4) cli gives some useful info - *lots* of PTS/DTS warnings:

ffmpeg -i "ScreenCapture_2-28-2016 12.45.57 PM.wmv" images/%05d.png
errors:
Past duration 0.619987 too large
Last message repeated 2 times
Past duration 0.639992 too large
Last message repeated 5 times

...

^ this eventually outputs 11828 frames

Settings '-vsync 0' (without this, ffmpeg inserts duplicate frames to try and maintain 20fps CFR):
ffmpeg -i "ScreenCapture_2-28-2016 12.45.57 PM.wmv" -vsync 0 images_sync0/%05d.png
errors:
[image2 @ 0x2642100] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3173 >= 3173
[image2 @ 0x2642100] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3199 >= 3199
[image2 @ 0x2642100] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3322 >= 3322

...

^ outputs 11656 frames

So 11828 to 11656 would be a difference of 8 or 9 seconds @ 20fps

ffmpeg 2.8.4 (win64 static):

Working...
$ ffmpeg -i infile.wmv -vcodec copy -acodec copy outfile.wmv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec pcm_s16le outfile.wmv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec adpcm_ms outfile.wmv

^ I only tested this last (adpcm_ms) line in v3.0

NOT working... (tested versions: 2.8.4; 2.8.6; 3.0)
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy outfile.wmv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -ar 44100 outfile.wmv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -ar 48000 outfile.wmv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -vsync 1 outfile.wmv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -vsync 0 outfile.wmv
$ ffmpeg -re -i infile.wmv -vcodec libx264 -acodec copy outfile.wmv
$ ffmpeg -r 20 -i infile.wmv -vcodec libx264 -acodec copy outfile.wmv
$ ffmpeg -r 20 -i infile.wmv -vcodec libx264 -acodec copy -vframes 11828 outfile.wmv
$ ffmpeg -r 20 -i infile.wmv -vcodec libx264 -acodec copy -framerate 20 outfile.wmv

The ffmpeg FAQ ( https://www.ffmpeg.org/faq.html#Which-codecs-are-supported-by-Windows_003f )
states that the following are the only audio codecs that should be used and that almost anything else is (obviously) unsupported and Micro$oft will eat us for exposing the fact that they don't even understand their own codec...

adpcm_ima_wav
adpcm_ms
pcm_s16le (always works)
libmp3lame

My personal conclusion == [1] Proprietary software still sucks. [2] Blender (and/or ffmpeg) should either issue a warning when attempting to convert the audio, from a WMV input, into anything other than the above (unless "-acodec copy" is paired with .WMV output) [3] Proprietary software still sucks (ad infinitum).

TYPO: All of the above "outfile.wmv" were actually "outfile.avi"... except for the very first one.

Can you clarify what you mean by working/not working?

I tried the first from your working list and get a noticeable a/v drift within the first minute:
ffmpeg -i infile.wmv -vcodec copy -acodec copy outfile.wmv

But the first from your not-working list seems to maintain sync, at least for the first minute:
ffmpeg -i infile.wmv -vcodec libx264 -acodec copy outfile.wmv

When converting the second command, ffmpeg cli will output a lot of errors since it's running through a corrupted file (and fixing it enough for Blender to use).

The absence of errors from the first command doesn't mean the stream is ok, just that ffmpeg hasn't identified any issues since it's only re-muxing the streams.

I *think* that's what might be happening, anyway.

I get noticeable drift, when rendering (with NLE exclusively) the 20 FPS WMV sample from above, to any output via Blender.

I can easiily get it to work (no noticeable drift), with just about any codec combo, using ffmpeg cli... Except for an ffmpeg-specific (.wmv input -> .aac output) bug that I've filed here:
https://trac.ffmpeg.org/ticket/5340

It seems that this problem is pretty much Blender-specific. I'm currently looking at/around https://wiki.blender.org/index.php/Dev:Ref/GoogleSummerOfCode/2005/FFMpeg_Integration for insight.

If I can sort it all out, I plan to change the UI interface for Encoding to allow a funneled selection of compatible parameters (for a given container; ie. no VBR choices for .AVI container, etc.):

Container
... Compatible video codec
... Compatible audio codec

But, at this point, I am suffering from some kind of sleep-deprivation-induced delirium that bars me from all sense of logic. (whatever that means)

I understand Python (on a basic level), I can program in C, and I can compile most open source software at the command line. I just need to comprehend the process (video/render pipeline) better; The documentation (and current Encoding UI) use improper terminology... ie. "Format" should be "Container" or "Container Format"... etc...
https://www.blender.org/manual/render/output/video.html#encoding-panel

This bug should probably be closed as "Author is insane". And re-opened as "VSE renders pre-fab video/audio extremely slow, using bogus presets, and has corner-case a/v sync issues"

Thank you.

Just based on the comments it seems like the bug is caused by the missing frames in the source file. During playback blender respects the PTS (presentation timestamps) in the video and is fine, but during rendering the output, blender just uses the next frame every time, ignoring the PTS and thus time compressing the video. To test this, you could simply use ffmpeg to reencode the video, to fix the missing frame issue and then try again to use this video in blender.

I tried re-encoding before blederizing and the result is the same. Audio and/or video drifting. (after blender's VSE conversion)

Paul R (intracube) added a comment.EditedMar 19 2016, 8:33 PM

I tried re-encoding before blederizing and the result is the same. Audio and/or video drifting. (after blender's VSE conversion)

I can't reproduce this on Linux. Blender sometimes uses the host OS's encoders which might explain the difference, so this needs to be checked by a Windows 8 user.

converted using:
ffmpeg -i "ScreenCapture_2-28-2016 12.45.57 PM.wmv" -c:v libx264 -c:a copy output.wmv

  • set new Blender project to 20fps @ 1068x640 (but 50% resolution)
  • imported "output.wmv" to the VSE
  • rendered the full length to AVI with H.264 + MP3 audio

The file plays in Mplayer and VLC without any visible a/v drift.

There's a slight misalignment between video and audio of about 100ms (2 frames) but it's consistent from start to end. The VSE has done this for a long time with some codecs and it's usually no more than 2 or 3 frames. Enough to be a fault, but it's not the issue you're describing.

Just based on the comments it seems like the bug is caused by the missing frames in the source file. During playback blender respects the PTS (presentation timestamps) in the video and is fine, but during rendering the output, blender just uses the next frame every time, ignoring the PTS and thus time compressing the video.

^ Yep, that would be my guess too.

The best/quickest/easiest way to duplicate the bug is

  1. Open Blender
  2. Choose Screen layout --> Video Editing
  3. Switch the "Graph Editor" window to a "Properties" window
  4. VSE --> Add --> Movie --> "ScreenCapture_2-28-2016 12.45.57 PM.wmv"
  5. VSE --> Strip --> Set Render Size
  6. Properties --> Dimensions --> Scale resolution --> 100%
  7. Properties --> Output --> File format --> H.264
  8. Properties --> Encoding --> Format: Matroska
  9. Properties --> Encoding --> Audio Codec: MP3
  10. Properties --> Dimensions --> End Frame: 11828
  11. Properties --> Render --> "Animation" button
  • It should be noticeable by ~15 second mark (the [ENTER] key smack for TAR command lags behind video)

The best/quickest/easiest way to duplicate the bug is

  1. Open Blender
  2. Choose Screen layout --> Video Editing
  3. Switch the "Graph Editor" window to a "Properties" window
  4. VSE --> Add --> Movie --> "ScreenCapture_2-28-2016 12.45.57 PM.wmv"

That's using the original (likely corrupted) file?

You said you tried re-encoding the original before importing to Blender, but still had the same issue. That's the part I can't reproduce.

Re-encoding with these formats were the only that gave issue with ffmpeg via CLI:

$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy outfile.wmv
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -ar 44100 outfile.avi
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -ar 48000 outfile.avi
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -vsync 1 outfile.avi
$ ffmpeg -i infile.wmv -vcodec libx264 -acodec copy -vsync 0 outfile.avi
$ ffmpeg -re -i infile.wmv -vcodec libx264 -acodec copy outfile.avi
$ ffmpeg -r 20 -i infile.wmv -vcodec libx264 -acodec copy outfile.avi
$ ffmpeg -r 20 -i infile.wmv -vcodec libx264 -acodec copy -vframes 11828 outfile.avi
$ ffmpeg -r 20 -i infile.wmv -vcodec libx264 -acodec copy -framerate 20 outfile.avi

The original .WMV file was created with microsoft's expression encoder. Despite (or because of) being "from the source", their codec apparently produces somewhat broken output. FFmpeg CLI seems to handle most of these issues gracefully. Blender must be struggling with them.

The expression encoder screen capture tool was (previously) my primary method for capturing decent quality desktop recordings. However, I've finally figured out to drop in FFmpeg for that task and now I can get bug-free, high-quality desktop recordings sans microsoft.

Sergey Sharybin (sergey) lowered the priority of this task from Confirmed to Incomplete.Mar 22 2016, 4:34 PM

Does the drift happens when you're using TimeCode for the source video files?

Paul R (intracube) added a comment.EditedMar 22 2016, 5:05 PM

Generating Record Run timecode is enough to fix this.

  • tested with "ScreenCapture_2-28-2016 12.45.57 PM.wmv"
  • Blender encoding settings: AVI with H.264 + MP3
  • checked in Mplayer and VLC on linux

and sorry, I should've suggested trying TC - I remember being told long ago that it was often needed for accurate editing.

Aaron Carlisle (Blendify) claimed this task.

Seems like no bug then.

It would be nice if Blender could detect the situation and notify the user (maybe offer to carry out the suggested solution).