Audio/Video Sync Test Preliminary Results

[Quu]

Full details at http://quu.livejournal.com/821424.html
Problem
When encoding a video using modern encoding tools, the idea that the tools would change the audio and video sync of the source material is rarely considered. When initially investigating this, I found that this could not be counted on. MP3 and AAC audio encoders need to pad the beginning of the stream with empty packets, and the amount of empty packets is not always formalized, leading to different vendors using different padding, confusing the decoders. Video encoding tools have a simpler job, since a frame is well defined, but even then, some video encoders automatically change or adjust the frames with default settings, trying to compensate for interlaced source footage.

After initially doing a very broad investigation into audio and video sync, I decoded to narrow my focus. The decoder used has as much to do with the sync of any videos as the encoders. I selected two popular video players that supported an easy export feature, VLC and MPlayer. The idea is to take a known "Baseline" uncompressed AVI file, encode it with various tools, then export it with the players, and test how much the audio and video has gone out of sync as compared to the original AVI exported/played the same way.

My testing is not looking for audio or video quality of the various encoders, but just the ability to maintain the original audio and video sync when played with VLC and MPlayer.

Tools and Version
Stand Alone Full Audio and Video Encoders
Handbrake - 0.9.4 (CLI version used)
x264 JEEB's build with L-SMASH, QTAAC and LAME - x264 0.104.0+1943 19e73ec built on Sep 4 2010, gcc: 4.4.4 (x86.generic.Komisar)

Audio Encoders
FAAC - 1.28
FLAC - 1.2.1
LAME - 3.98.4
Nero AAC Encoder - 1.5.1
QTAACEnc - version 20100725 by tmkk

Video Encoder
x264 (from x264.nl) x264 0.104.1713 c276662 built on Sep 4 2010, gcc: 4.4.4

Muxers
mkvmerge - v4.2.0 ('No Talking') built on Jul 28 2010 18:38:23
mp4box - MP4Box - GPAC version 0.4.5 (build 33 - Dec 11 2008) - compiled by Kurtnoise
mp4creator - mpeg4ip version 1.6.1d
tsmuxer - Version 1.10.6

Results
As far as video sync goes, the default "High Profile" settings in Handbrake breaks the video sync as it tries to compensate for interlaced footage, even when given progressive footage. When the default profile is modified, Handbrake is able to produce in sync video with no lost frames. No matter how it is muxed, x264 footage is always in sync.

For audio sync, things were a little different. Both of the stand alone audio/video encoders where able to produce audio that stayed in sync, even when Handbrake's default setting dropped video frames, the audio was perfectly in sync when played back. Of the stand alone audio encoders, only FAAC and FLAC were able to stay in sync with the encoded video, no matter how they were muxed. Uncompressed PCM audio was also tested, but only VLC was able to play it, and it was in sync when tested. What is strange is that JEEB's build of x264, which included QTAAC and LAME was able to maintain perfect audio sync, but when both of those tools were tested externally, they failed the sync test. Even when JEEB's x264 build imported the externally rendered files they were out of sync, so it is not purely because of the L-SMASH mp4 muxer in JEEB's build.

LAME, Nero, and QTAAC all produced audio files, that when muxed and played back in the two test players, that were delayed when compared to the original audio and out of sync with the video. It did not matter what muxer was used, the audio created by these three stand alone encoders where not played properly by VLC or MPlayer, while FAAC, FLAC, and the stand alone audio/video encoders where.

[Ghitulescu]

I think what you tested here (and there) is the ability of the muxers to keep the streams in synch (no encoder I know stretches or shrinks the duration of an A/V stream). Besides, playing in VLC and MP is not at all relevant, normally it should be used a reference decoder/player, very close to these yet cheap are only the hardware players of a certain quality (;)). The asynch issue is known to industry since Edison times and various measures are enforced in order to deliver asynch-free content.

[Quu]

I guess I am approaching this as a "normal consumer" if there is such a thing on the internet.
I am looking at a simple scenario. I have a Video X, I want to encode that video and then play it back, and have the audio and video sync be identical to the original Video X. My testing is based around that assumption. I am not done with my testing, this was just the first iteration, an initial run of my testing tool to see if ti worked.

Also... i did find this article on the Apple website about their own AAC encoder, details the empty samples before and after the encoded audio, and how that can lead to a delay
http://developer.apple.com/library/ios/#technotes/tn2009/tn2258.html
Allot of what I am testing is based on the details in that tech note.

[yetanotherid]

If it's just s simple delay of the audio track inside the original file most encoders should handle this when muxing the encoded video and audio, however I doubt it's always that simple.

I've battled with sync issues many times and if you find a perfect solution I'd be eager to know what it is.

It seems to me.... and I'm guessing here..... that there's often other factors involved such as breaks in one stream that aren't present in another, an audio stream which may have been time stretched when muxed into the original video.... most encoders seem to demux the audio and convert it separately so are oblivious to what may be effecting it inside the original video container. I've even converted video running at 23.976 fps and found that to get the audio in sync for the converted file I've had to change it's frame rate slightly.... say to 23.975 fps. That'd seem impossible when I've compared the two as the original video has the same number of frames as the converted video and both the original and converted audio tracks are exactly the same length, so I guess there's factors at play I don't understand.

My standard solution for fixing audio which seems to do the impossible is to re-encode it via directshow. I usually try to do it after muxing the original audio and video into an MKV file as it seems to work best, then I use an encoder which will play back the mkv via directshow and convert the audio that way (rather than demux it and convert it). That method seems most likely to produce an encoded audio track which will be in synch with both the original and encoded video.
I haven't done any experimenting with it yet, but MeGUI gives the option to use directshow for decoding audio, however I don't know if it has to demux the audio first. If it doesn't, it might be a solution some of the time.

And of course every time I think I've found a program which will maintain the sync when converting both audio and video in difficult files I then find a file for which it can't and have to try another program or the directshow method. If you find the perfect solution please post back!

What happens when you try to encode video which has a variable frame rate?

[Ghitulescu]

One more reason to stick with industry-enforced standards: MPEG-PS and -TS. At the expense of space, of course. Both implement a very good synch policy. To give an example, the private broadcasters in Germany, Sat1, Pro7 and especially RTL have a variable audio delay, like one program has a -673ms, the next has a -1205ms, the next has a -120ms and so on (I'm exaggerating a bit). Yet even the most chinese SAT-receiver can keep the A/V in synch.

If I may add something slightly off-topic: some 10 years ago (AFAIK) some came with the idea of using VBR-MP3, to gain an extra 10MB storage space. The result (then): playable only on PC, most software refused to accept them or generated asynch. Now the situation improved a lot but still there are "scars" from those times....

[yetanotherid]

I've been encoding using VBR MP3 for quite a while now but I can't say it's everf given me an issue when played by either a software or hardware player. I do tend to use CBR often, but that's because I use VirtualDubMod for muxing AVIs a bit and it gets along better with CBR MP3.

There was some theory that VBR MP3 inside an AVI was a hack, and while I can't remember the details it was a no, no for a lot of people for a long time because apparently one day the Windows AVI parser might change and they won't play any more...... or something like that.

Me..... I just figured I'd use VBR when I wanted to and if the sky ever started falling I'd open the problem AVIs with MKVToolnix, remux them as MKV files and go on my merry way.

[Quu]

I've battled with sync issues many times and if you find a perfect solution I'd be eager to know what it is.

I don't have a perfect solution... but what I am working on is an automated (mostly automated) way to test X amount of encoding programs with Y amount of sources against Z amount of decoders, to see the results. It is meant to be a work in progress, as I can easily add new tools and new versions of tools as I discover them. I can add more test cases, videos that break encoders, easily. and finally, I can add more "output targets" or decoders/players if they provide a way to test the output.


It seems to me.... and I'm guessing here..... that there's often other factors involved such as breaks in one stream that aren't present in another, an audio stream which may have been time stretched when muxed into the original video.... most encoders seem to demux the audio and convert it separately so are oblivious to what may be effecting it inside the original video container. I've even converted video running at 23.976 fps and found that to get the audio in sync for the converted file I've had to change it's frame rate slightly.... say to 23.975 fps. That'd seem impossible when I've compared the two as the original video has the same number of frames as the converted video and both the original and converted audio tracks are exactly the same length, so I guess there's factors at play I don't understand.

My primary focus is the lifecycle of "Uncompressed AVI" => "Encode for distrobution" => "Playback on consumer's computer".

I am trying to minimize the amount of variables in my testing. I start with an uncomressed AVI file, encode it using command line tools, then decode that video file with VLC, MPlayer, or (soon) ffmpeg and then compare the sync to the original AVI file. I don't do anything fancy right now, as I am getting disturbing results before I even try to get tricky.

Seriously... one of my examples is
test.avi (uncompressed AVI)
test.avs (avs file with sole line is AVISource("test.avi")
test.wav (uncompressed wav file exported from test.avi using VirtualDub)
(here is all three of the above files as wave form rendered in AVISynth to confirm they are all in sync)
http://pics.livejournal.com/quu/pic/00061f03

I then run these commands (as an example)
x264\x264.exe --crf 18 --profile high --preset veryslow --tune fastdecode -o test-vx264.mp4 test.avs
NeroAAC-1.5.1\neroAacEnc.exe -q 0.6 -if test.wav -of test-anero.mp4
mkvtoolnix\mkvmerge.exe -o test-vx264-anero-mmkv.mkv test-vx264.mp4 test-anero.mp4
mplayer\mplayer.exe -vo null -ao pcm:waveheader:fast:file=test-vx264-anero-mmkv-dmpl.wav test-vx264-anero-mmkv.mkv
mplayer\mplayer.exe -vo null -ao pcm:waveheader:fast:file=test-dmpl.wav test.avi

I then compare the sync of test-dmpl.wav and test-vx264-anero-mmkv-dmpl.wav (I use AVISynth and wave form drawing) and I find that the nero encoded audio is delayed when compared to the original... but if i check the video sync, it has not also been delayed, so the audio and video is now out of sync when compared to the original.... This is confirmed with mp4box, mp4creator, mkvmerge, and tsmuxer... all four muxing tools provide the exact same desync, so my suspistion is the audio encoder, or the command line I am running


My standard solution for fixing audio which seems to do the impossible is to re-encode it via directshow. I usually try to do it after muxing the original audio and video into an MKV file as it seems to work best, then I use an encoder which will play back the mkv via directshow and convert the audio that way (rather than demux it and convert it). That method seems most likely to produce an encoded audio track which will be in synch with both the original and encoded video.
I haven't done any experimenting with it yet, but MeGUI gives the option to use directshow for decoding audio, however I don't know if it has to demux the audio first. If it doesn't, it might be a solution some of the time.

I want to write a command line tool that will create a direct show filter graph using specific filters that exports to an file, so I can "simulate" the playback in MPC-HC and MediaPlayer using certian filters... but my time is limited.


And of course every time I think I've found a program which will maintain the sync when converting both audio and video in difficult files I then find a file for which it can't and have to try another program or the directshow method. If you find the perfect solution please post back!

I have not found a perfect solution... but I have found that with my testing, given the tools listed in my report... x264+FAAC+AnyMuxer works... x264+FLAC+MKV works... JEEB's build of x264 with the internal audio encoding works with QTAAC and LAME as the audio encoder, and will probably work with all of the internal coders. Handbrake, if you turn off decomb and detelecine, works. When I find problem videos, I will add them to my test harness and provide feedback on that.


What happens when you try to encode video which has a variable frame rate?

I do not have a test video with variable frame rate... sorry

One more reason to stick with industry-enforced standards: MPEG-PS and -TS. At the expense of space, of course. Both implement a very good synch policy. To give an example, the private broadcasters in Germany, Sat1, Pro7 and especially RTL have a variable audio delay, like one program has a -673ms, the next has a -1205ms, the next has a -120ms and so on (I'm exaggerating a bit). Yet even the most chinese SAT-receiver can keep the A/V in synch.

I tested with MPEG-2 TS... one of the four muxing tools I used was TSMuxer version 1.10.6 and it provided no difference on the audio sync when compared to mkvmerge, mp4box and mp4creator. All four external muxers provided identical results each time to ALL of the audio encoded... so I suspect that the problem is not the muxers, but the output from the stand alone audio encoders.

I did not test with MPEG-2 PS becasue it could not handle the H.264 video which I had limited my testing to (time, sorry)... when I expand my testing out to MPEG-2 I will re include the MPEG-2 PS muxers.

[GodofaGap]

all four muxing tools provide the exact same desync, so my suspistion is the audio encoder, or the command line I am running
The Apple link suggest that lossy audio encoders often use a primer delay, which should be removed on decoding. The page mentioned a value of around 2100 samples IIRC (although this is not fixed) which is a bit more than 1/24 of a second for 48 kHz, so this coincides nicely with you results of a delay of around a frame.

This is interesting theory. Are you intending to test Vorbis as well?

[Midzuki]

I did not test with MPEG-2 PS because it could not handle the H.264 video which I had limited my testing to

If you mean that the .MPG container does not support H.264 (not to mention VC-1), then I guess you are wrong.

[Quu]

If you mean that the .MPG container does not support H.264 (not to mention VC-1), then I guess you are wrong.

OK... from what I read MPEG-2 PS did not support H.264... only MPEG-1 video, MPEG-2 video, and MPEG-4 ASP
http://en.wikipedia.org/wiki/MPEG_program_stream
the list of registered private streams included VC-1 but not H.264 (Or MPEG-4 AVC Part 10)
http://smpte-ra.org/mpegreg/mpegreg.html

I did not find any MPEG-2 muxers that supported H.264... thought I did not look hard.

MPEG-2 TS supports it, after all, Bluray uses MPEG-2 TS as its video container format. My thought was that if the PS stream was designed for disc based storage (like a bluray) and TS for broadcast usage (IE satellite, IPTV), then Sony would have gone with PS if it supported H.264 properly...

but you are right, I should have investigated more.

The Apple link suggest that lossy audio encoders often use a primer delay, which should be removed on decoding. The page mentioned a value of around 2100 samples IIRC (although this is not fixed) which is a bit more than 1/24 of a second for 48 kHz, so this coincides nicely with you results of a delay of around a frame.

This is interesting theory. Are you intending to test Vorbis as well?

I am considering it... If you look at my test XML document, I have allot of tools commented out (MPEG-2 video, MPEG-1 Layer 2 audio)... I should probably add Vorbis... but I guess where do you stop then. Should I add VP-8 or Dirac? Is there a good command line MPEG-2 ASP encoder (all the XVid I can find is VFW filters only... nothing command line so i can automate it)

[Midzuki]

en.wikipedia.org/wiki/MPEG_program_stream says:

Program streams are used on DVD-Video discs and HD-DVD video discs, but with some restrictions and extensions.

Quu wrote:

Sony would have gone with PS if it supported H.264 properly...

No, 'twas because PS was the container used by the "enemy". :D

[Quu]

No, 'twas because PS was the container used by the "enemy". :D

http://en.wikipedia.org/wiki/Enhanced_VOB
Yea... kind of... =^_^=
meh... who knows why they went with MPEG-2 TS...

either way, I can adapt my testing rig to include any windows based tools. I prefer command line tools, since the GUI tools I have to do manually right now instead of letting the batch files handle it.

[GodofaGap]

I am considering it... If you look at my test XML document, I have allot of tools commented out (MPEG-2 video, MPEG-1 Layer 2 audio)... I should probably add Vorbis... but I guess where do you stop then.
I would mainly focus on audio formats. A video encoder that drops/adds frames is really broken, while this primer delay thing actually seems to have a valid reason and audio decoders should apparently deal with it.

Should I add VP-8 or Dirac? Is there a good command line MPEG-2 ASP encoder (all the XVid I can find is VFW filters only... nothing command line so i can automate it)
xvid_encraw builds by squid80 might help you: http://forum.doom9.org/showthread.php?t=98469&page=35

[Quu]

I would mainly focus on audio formats. A video encoder that drops/adds frames is really broken, while this primer delay thing actually seems to have a valid reason and audio decoders should apparently deal with it.


xvid_encraw builds by squid80 might help you: http://forum.doom9.org/showthread.php?t=98469&page=35

thank you very much... and yea... for now I am going to focus on the audio side of things... you are right... when the video fails... it is very obvious