Log in

View Full Version : Questions about H.264 sync


Quu
12th August 2010, 20:06
I want to check the sync and timings of h.264 encoding. For right now, I am not worried about quality, just sync. To test for sync, I want to see if any encoding solution losses or adds frames, and if the audio and video get out of sync with each other. I am focusing on MPEG-4 h.264 as the video source, and AAC and MP3 as the audio source. I am planning on using the MPEG-4 container format.

Depending on the results, MPEG-4 ASP and MPEG-2 video can also be tested, along with the potential of trying the MKV container format.

The Problem:
A video was brought to my attention which had been encoded from an MPEG-2 source to a h.264 MPEG-4 file using Handbrake. The sync on the resulting video was different than the original, so I decided to investigate the cause. Initial testing using the source video was problematic, so a synthetic video was created to refine the testing system. Once I am satisfied with the testing methodology and results, I can repeat the testing with the problem videos instead of the synthetic video.

My initial testing used the MSU VQMT application to detect differences in the encoded video files. There was problems with sync with this testing application (I later discovered it was DirectShow), and I had to use custom AVISynth scripts to fix the sync issues. As I started messing with AVISynth more, I discovered that I could test for the video sync and audio sync inside of a single script, with out needing the MSU VQMT application.

The testing involved the following encoding tools: Handbrake, x264, FAAC, NeroAACEncode, LAME, and mp4box.
The comparison tools used: VLC, AVISynth, VirtualDub, AudioGraph (AVS Plugin), FFmpegSource2 (AVS Plugin), Haali splitter (DirectShow) (http://otakuvideo.com/~quu/H264Test/DirectShow%20splitter%20info.png), and ffdshow tryouts (DirectShow) (http://otakuvideo.com/~quu/H264Test/DirectShow%20video%20decoder%20info.png).
I have only used freely available tools for this testing, nothing requires a license for personal use.

the synthetic video is an AVS script of the following (my master AVS)


# original mp3 from http://freemusicarchive.org
# vlc 1.1.2 used to convert to wav file
audio = WAVSource("Fabrizio_Paterlini_-_01_-_Forever_blue_March_Rosetta_re-imagined_version.wav")
return ShowSMPTE(AudioGraph(AudioDub(BlankClip(length=8500, width=720, height=480, fps=24), audio), 10))



Overview of Results:
The best sync seams to be x264 video with LAME encoded MP3 for the audio.



Handbrake "High Profile" has a video sync issue, with missing frames in the resulting encoded video, no other video encoding method showed any frame loss
Handbrake adds extra frames to the end of my testing, it repeats the last couple of frames a second time
Handbrake audio is out of sync, not matter how it is rendered
The DirectShow playback of H.264 video was horrible, with sync problems becoming progressively worse as the video is played
Inside of AVISynth, FFAudioSource induces a ~1 frame shift in the audio. This needs to be tested in ffmpeg based stand alone players (VLC, etc)
NeroAACEncode adds a ~1 frame skew in audio, where FAAC does not. This is in addition to the FFAudioSource skew



I need to have my results reviewed, and see if there is additional things I need to adjust for. The timing problem with Nero is surprising, since I have been told that it was the superior encoder to FAAC. This is just my first pass at the timing results, I expect there to be things to fix and adjust. I know that AVISynth is not known for being the most accurate as far as audio goes, I just need a way to check for drift.

:EDIT:
It also seams that the timing problems with FFAudioSource go away if I use the MKV container. I need to test with VLC native, as it is either my usage of mp4box or the "player" of AVISynth

The Testing Methodology
All of the scripts and the source WAV file used can be found in this .zip file (http://otakuvideo.com/~quu/H264Test/src.zip) so my results can be verified and my scripts confirmed. I am including the WAV file instead of the mp3, just for consistency. All encoding steps were repeated three times and the resulting files compared. There was no inconsistencies in encoding, the results where binary identical for each iteration.

An uncompressed AVI and an uncompressed WAV file was created with VirtualDub using the master AVS file as the source. The sync from all three of these baselines (AVS, AVI, WAV) were checked and found to be identical.

Using Handbrake (http://otakuvideo.com/~quu/H264Test/handbrake%20version.png) and the baseline AVI, a "Normal" and "High Profile" encode was made. Other than selecting the preset, all other settings are default.

(command line notes) (http://otakuvideo.com/~quu/H264Test/test%20notes.txt)
Using the master AVS file, x264 (32 bit version) was used to encode two versions (main profile and high profile) of the video to raw H.264 files. Other than "--profile main" and "--profile high", no other optional command line settings where used.

Using LAME, the baseline WAV file was encoded to MP3. The "-b 192", 192 bit rate, and "-h", high quality encode, command line options were used.

Using FAAC, the baseline WAV was encoded to an AAC file (actually an .mp4 file with the AAC stream inside of it). No optional command line parameters where used.

Using NeroAACEncode, the baseline WAV was encoded to a different AAC file (actually an .mp4 file with the AAC stream inside of it). No optional command line parameters where used.

Using mp4box, both of the x264 encoded video streams were combined with the three different audio streams, resulting in 6 MPEG-4 files in total. Other than setting the frame rate at 24 fps (since the H.264 video was raw, not in a container), no optional command line parameters where used.

At this point I had 8 Audio/Video and 3 Audio files to compare to the baseline. For each test file, two AVS scripts where made for comparison.

The first was a simple comparison, with FFVideoSource/FFAudioSource used to make a simple display. For the audio only tests, the AudioGraph was rebuilt from the encoded audio. While this is good for testing the video initial sync, it could not tell if the audio and video were in sync with each other.

A second comparison script was built for each test file. Inside of this script, the video was decoded using FFVideoSource and DirectShowSource, and the audio was also decoded using FFAudioSource and DirectShowSource. Two additional "video sources" where made using the audio decoded both ways back through the AudioGraph plugin. Using this comparison script we can tell if the video is drifting out of sync with the audio, either way it is decoded.

Detailed Results
The video sync results, the most important is the time code in the bottom. Frame 4,000 was picked because it was easy to remember, and had an easy to see wave form.
The original master AVS (http://otakuvideo.com/~quu/H264Test/Original.png)
The baseline AVI (http://otakuvideo.com/~quu/H264Test/Original%20avi.png)
The baseline WAV (http://otakuvideo.com/~quu/H264Test/Original%20wav.png)
The baselines are identical to each other. This is just a confirmation that no errors are creeping in before the encoding occurs.

The encoded audio file test scripts are basically copies of the master AVS, but with FFAudioSource used instead of WAVSource.
The LAME file (http://otakuvideo.com/~quu/H264Test/LAME%20mp3.png)
The FAAC file (http://otakuvideo.com/~quu/H264Test/FAAC%20aac.png)
The NeroAACEncoder file (http://otakuvideo.com/~quu/H264Test/Nero%20aac.png)
All three files are out of sync with the baseline. The Nero encode more than the other two, but only by a little bit. This shows potential issues with the future results, since the audio is already coming out of the gate out of sync. (Please note, later testing used DirectShowSource for audio, and no sync issues found with FAAC and MP3)

Now the two Handbrake encodes. I was most interested in this, because Handbrake is easy and convenient to use.
Handbrake, Normal preset (main profile) (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29.png)
Handbrake, High Profile preset (high profile) (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29.png)
Well, the normal preset looks dead on, but the high profile has already lost a frame, and is running a frame faster. I reran the encode additional times, and each time it was binary identical. Also, both the normal (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20comparison%20-%20at%20end.png) and the high (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20comparison%20-%20at%20end.png) profile added additional frames to the end of the file. When I inspected the end, it seams that Handbrake just went back a couple of frames and replayed them again.

Finally the x264 encodes. the initial scrip only showcases the video sync, not the audio sync, so only the FAAC versions need to be shown.
x264 main profile (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC.png)
x264 high profile (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC.png)
The encoding is spot on, and no frame drift, loss, or extra frames with the x264 encoding.

The video comparison was only half of my testing, the second half was the audio and video sync. To me, this was the critical testing, If the video and audio get out of sync, then the viewing experience can be spoiled.

For the three audio comparisons, a script was made to compare the timings based on how it was decoded, and then also compared to the baseline WAV.
The LAME file comparison (http://otakuvideo.com/~quu/H264Test/LAME%20mp3%20comparison.png)
The FAAC file comparison (http://otakuvideo.com/~quu/H264Test/FAAC%20aac%20comparison.png)
The NeroAACEncoder file comparison (http://otakuvideo.com/~quu/H264Test/Nero%20aac%20comparison.png)
When using the FFAudioSource there is a obvious skew of about 1 frame across the board that is not seen in the DirectShowSource. The LAME MP3 source file, with DirectShow is spot on perfect in its timing. The FAAC timing is slow, but less than a frame slow. the Nero encoder is running a complete frame slow, even through the DirectShowSource.

For the audio/video comparisons, the file is decoded with DirectShow and ffmpeg and the compared. The audio is extracted and a new AudioGraph is rendered from it to show any potential audio skew that the mp4box muxing might have added. In total there is four graphs.

First is the two Handbrake videos. This will also be the first time the audio from Handbrake is checked for sync, so reference this image (http://otakuvideo.com/~quu/H264Test/Original.png) to see if it is running fast or not.
Handbrake, Normal preset (main profile) comparison (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20comparison.png)
Handbrake, High Profile preset (high profile) comparison (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20comparison.png)
the normal profile has audio that is running about half a frame slow, but the video is spot on when rendered from ffmpeg. with direct show the video is starting to drift. Strangely, the audio is close between ffmpeg and direct show, which was not seen in the stand alone encodes. The high profile comparison is a mess. the Direct Show is running almost 7 seconds slow, and the ffmpeg audio is showing the same 1 frame skew we have seen before.

The command line encodes are actually very similar. Since we have already shown that, video wise, there is no sync difference between x264 main and high profile, and we have seen the sync difference between the three audio formats, the results here should be predictable, we should only see problems if mp4box is broken.
x264 Main LAME (http://otakuvideo.com/~quu/H264Test/x264-main%20LAME%20comparison.png)
x264 Main FAAC (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC%20comparison.png)
x264 Main NeroAACEncoder (http://otakuvideo.com/~quu/H264Test/x264-main%20Nero%20comparison.png)
x264 High LAME (http://otakuvideo.com/~quu/H264Test/x264-high%20LAME%20comparison.png)
x264 High FAAC (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC%20comparison.png)
x264 High NeroAACEncoder (http://otakuvideo.com/~quu/H264Test/x264-high%20Nero%20comparison.png)
And the results are exactly as predicted. Everything run through the ffmpeg AVISynth decoder runs about a frame fast. The DirectShow video decoder is about 3-4 frames to slow. The LAME mp3 audio when decoded with DirectShow seams to be the best synced of all the audio, though FAAC is really close.

:Update:
I was reminded that VLC can export/encode to H.264/AAC, so I quickly gave that a try (most of the settings are deep in the config, but I tried with the defaults)
The video was on track, but it was the comparison that amazed me.
VLC H.264 + AAC MP4 Default settings (http://otakuvideo.com/~quu/H264Test/vlc%20-%20comparison.png)
The audio tracks were lined up, even the FFAudioSource. Of course the direct show video was all messed up, but that is normal.

I also decided to give the Matroska (MKV) container format a try, after all I still had the raw streams as files, using MKVMerge to make mkv files should be easy. I used the GUI to make the mkv files, only having to specify the frame rate. I then had to go in via the command line afterwords and remux them, removing the compression for stream -1.

But the results were startling
x264 Main LAME MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20LAME%20MKV%20comparison.png)
x264 Main FAAC MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC%20MKV%20comparison.png)
x264 Main NeroAACEncoder MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20Nero%20MKV%20comparison.png)
x264 High LAME MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20LAME%20MKV%20comparison.png)
x264 High FAAC MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC%20MKV%20comparison.png)
x264 High NeroAACEncoder MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20Nero%20MKV%20comparison.png)
The FAAC audio was synced, even with the FFAudioSource. Changing the container was not supposed to change the sync.

As a last bit of an experiment, I decided to remux the Normal profile handbrake video (Remember, the high profile was broken) into an MKV. I used the GUI to make a base MKV file, then the command line to remove the stream -1 compression. Also, I attempted to use the native MKV output to see what would happen.
Handbrake Normal (main profile) Remuxed with MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20MKV%20repack%20comparison.png)
Handbrake Normal (Main Profile) native as MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20MKV%20comparison.png)
Handbrake High Profile (High) native as MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20MKV%20comparison.png)
Well, that was unxpected, all the audio sync problems went away with MKV. High Profile is still broken.

Update Summary
Changing the container changed the sync. It could be that I am not using mp4box properly, or that the "reader" of AVISynth and the FFMS2 is doing that one frame "miss" in the FFAudioSource. Either way, there is more to test.

Guest
12th August 2010, 20:40
Moved thread to General Disucssion as it ranges much wider than simply MPEG4 AVC.

Quu
12th August 2010, 20:56
ahh, true.

my original investigations on it lead me to think it was possible vfr with h.264 for my timing messups, now I mainly know it was becasue of Directshow drivers.

NeroAACEnc still worries me. I have been told it is the best AAC encoder out there, but it's timing, when compared to FAAC and MP3, and even the original WAV, is off. Maybe I am just measuring or testing it wrong?

Guest
12th August 2010, 21:47
I think looking for a processing chain that will give perfect AV sync for all sources and containers and scripts is a Holy Grail. You'll never attain it. You may eliminate some common hose-ups, but always you'll have special cases to deal with.

Quu
13th August 2010, 00:10
Yea, and I can consider this the due diligence on the path. Any issues with my testing methodology or things I missed or should try?

so far I am loving x264, and not so much Handbrake. I wanted Handbrake so much to be able to "work" since it was cross platform and so easy to use.

Zarxrax
13th August 2010, 00:23
A few years ago I also noticed occurrences of 1-2 frame desync on my videos, but I couldn't quite track down the problem. Good to see that it wasn't just me.

Dark Shikari
13th August 2010, 00:36
The single-audio-frame desyncs you're encounter likely relate to inconsistent MDCT priming behavior, both in encoders and decoders.

You will not encounter them with audio formats that don't use MDCTs.

Quu
13th August 2010, 03:32
The single-audio-frame desyncs you're encounter likely relate to inconsistent MDCT priming behavior, both in encoders and decoders.

You will not encounter them with audio formats that don't use MDCTs.

Well, I am currently working on a custom version of VLC as my playback target (hopefully once I have it working, the code changes will become part of the trunk, leaving only the skin and html interface). My old playback target was a Netstream2000 card and custom software for it.

As long as i can get an encoder chain that plays well with VLC, i will be happy

Quu
13th August 2010, 05:01
I just updated the report with my testing using MKV as a container format, and VLC as an "Encoder"

I thik I really need to figure out who to get the timings using VLC as the rending engine.

Quu
14th August 2010, 06:07
Before i got much further with testing the various encoding methods for sync... i wanted to make sure I could use my "target" of VLC and test sync that way
http://forum.videolan.org/viewtopic.php?f=2&t=81005&p=268012

I ended up pointing a camcorder at a monitor to test for audio/video sync

what i found... VLC can adjust for environmental issues causing a/v desync (like sound processors and the like) by globally skewing the audio vs the video, handy when tuning and configuring playback stations

the "results" from my synthetic test fit with the VLC playback (when accounting for the audio skew caused by the env)
Nero is "early" when compared to mp3/faac/qtaacenc/world+dog as far as audio timing (I don't care how pretty it sounds, it is out)
mp4box muxed mpeg-4 files cause their audio to also be early when played by an ffmpeg based player, mp4box nero audio is double early
VLC when it exports an mpeg-4 container file is not early like mp4box... this means either I am using mp4box wrong, or VLC is doing something that mp4box is not
MKV files have perfect audio timings, even given the exact same streams as the mp4... given this playback target

and I now need to go to sleep

Zarxrax
14th August 2010, 16:49
If you encode the audio with nero multiple times, does it become progressively more and more out of sync?
If you decode/reencode through ffmpeg multiple times, does it become progressively more and more out of sync?

Quu
14th August 2010, 16:56
I did not try that...

Quu
14th August 2010, 19:16
EDIT
blah, i was wrong read this - http://forum.doom9.org/showthread.php?p=1426169
If i go beyond just three iterations, and use a more accurate method of testing the wav form, even the FAAC encoded audio goes out of sync. that being said... Nero's sync is VERY much worse

---------------------------------------------------------------------
(This is not valid anymore, refer to the above link)

ok, I tried reencoding the audio using faac and neroaacenc 3 times (four aac files in total) then compared the timings for each one

you can get all of my scripts and the original wav file here
http://otakuvideo.com/~quu/aac%20timings/
all you need to do is download BePipe, FAAC, NeroAACEnc, and VirtualDub to repeate this test

So the first thign I tested was the delay that FFAudioSource included in the synthetic testing. I created an AAC file from the reference wav file and then compared its timings
http://otakuvideo.com/~quu/aac%20timings/faac%20original.png
by the look of things, the audio is about half a frame slow

I then re encoded this aac file, using BePipe and FFAudioSource (see the batch file and the AVS scripts in the directory for details) three iterations. I would expect the delay we see in the FFAudioSource to be larger at this point
http://otakuvideo.com/~quu/aac%20timings/faac%20reencode%203.png
And we see pretty much the exact amount of delay we say in the original encode. The delay that we see with FFAudioSource is not a real delay, but most likely a perception or an interaction between FFAudiosource and AudioGraph, or even if WAVSource runs fast? We need to not trust our synthetic results with out providing other tests like this. Testing results multiple ways is for the best.

So the "Delay" that the FFAudioSource provides is not cumulative, re encoding the same audio through the FFAudioSource again and again does not increase the delay. What this means is that, most likely, the delay that we see in the synthetic testing is an artifact of the AVISynth plugins.

The second audio delay we saw in the NeroAACEnc. I had high hopes for this encode, since I was told multiple times that this is the best audio quality of the free encoders.
So the first, "original", aac file created by Nero.
http://otakuvideo.com/~quu/aac%20timings/nero%20original.png
this is a full frame (maybe 95% of a frame) slower than the WAV file. So while the FFACC audio was around half a frame, Nero is almost a full frame slower.

But, as we learned with the FFAudioSource, it might be an interaction between other things, so we want to check to see if the delay is cumulative, if it is, then we know that Nero is responsible. Three re-encodes later...
http://otakuvideo.com/~quu/aac%20timings/nero%20reencode%203.png
and Nero is almost two and a half frames slower than the original WAV file. We know this is not the delay from FFAudioSource as we tested that first.

Nero AAC Encoder appears to be artificially adding an audio sync problem into streams encoded with it. I can not trust and audio file to be in sync when encoded with NeroAACEnc. While a frame off might be imperceptible to most people, for some applications, a single frame difference can matter.

Quu
15th August 2010, 07:07
updated the last post with the correct audio sync information with AVISynth

Quu
15th August 2010, 18:00
After working with the Audio Sync stuff, i need to get to the root of things, I need to evaluate the tools to test for the audio video sync. I feel like I am in a house of mirrors with this stuff. I can test all the combination of encoders and muxers I want, but until I figure out how to check if something is "right" i can't know which of the outputs is correct.

My target is VLC... maybe I just need to make a test video that can work with the camcorder test? or figure out how to test the sync of the video and audio through VLC and use that as the guideline?

Zarxrax
17th August 2010, 17:36
Regarding this whole issue, you mentioned that vlc can adjust the sync to compensate for everything. In that case, does it really even matter if your encoding process is out of sync at all? Just decide on a certain encoding workflow, and then figure out the known desync and compensate it.

Quu
17th August 2010, 17:41
Regarding this whole issue, you mentioned that vlc can adjust the sync to compensate for everything. In that case, does it really even matter if your encoding process is out of sync at all? Just decide on a certain encoding workflow, and then figure out the known desync and compensate it.

That is exactly what I am currently working on
http://quu.livejournal.com/819616.html

my goal is to find a encode process/chain/whatever that will, given my target, provide the same sync that the original does.

I would loved to have made my testing more broad, but it is primarily the audio decoding that has hampered me. As it was pointed out, all encoders put some delay at the beginning, it is up to the decoder to compensate for it... sadly the different tools don't agree with each other (hell, LAME does not agree with itself)

I still have to support all of the legacy MPEG-2 files that I already have

Zarxrax
17th August 2010, 19:26
If matroska seems to keep sync with everything, what about just remuxing everything to that? You could even remux the old mpeg-2 files to matroska, if need be.

Quu
17th August 2010, 19:32
If matroska seems to keep sync with everything, what about just remuxing everything to that? You could even remux the old mpeg-2 files to matroska, if need be.


that is something i will be testing, part of my plan so to speak... if the existing files play with out any issues, my plan is to leave them alone.

I also found that I may have been sabotaging the mp4 playback by scrubbing... apparently the sync is much better if i let the videos play to frame 4000 instead of scrubbing to there

like i said before... my biggest problem seams to be the audio encoder vs decoder... the encoder has to add some empty samples to the front, and it is up to the decoders to account for that... i just wish it was part of the official spec, so that the encoders and decoders would have to use the same amount of padding and you would not have these issues