Quu
12th August 2010, 20:06
I want to check the sync and timings of h.264 encoding. For right now, I am not worried about quality, just sync. To test for sync, I want to see if any encoding solution losses or adds frames, and if the audio and video get out of sync with each other. I am focusing on MPEG-4 h.264 as the video source, and AAC and MP3 as the audio source. I am planning on using the MPEG-4 container format.
Depending on the results, MPEG-4 ASP and MPEG-2 video can also be tested, along with the potential of trying the MKV container format.
The Problem:
A video was brought to my attention which had been encoded from an MPEG-2 source to a h.264 MPEG-4 file using Handbrake. The sync on the resulting video was different than the original, so I decided to investigate the cause. Initial testing using the source video was problematic, so a synthetic video was created to refine the testing system. Once I am satisfied with the testing methodology and results, I can repeat the testing with the problem videos instead of the synthetic video.
My initial testing used the MSU VQMT application to detect differences in the encoded video files. There was problems with sync with this testing application (I later discovered it was DirectShow), and I had to use custom AVISynth scripts to fix the sync issues. As I started messing with AVISynth more, I discovered that I could test for the video sync and audio sync inside of a single script, with out needing the MSU VQMT application.
The testing involved the following encoding tools: Handbrake, x264, FAAC, NeroAACEncode, LAME, and mp4box.
The comparison tools used: VLC, AVISynth, VirtualDub, AudioGraph (AVS Plugin), FFmpegSource2 (AVS Plugin), Haali splitter (DirectShow) (http://otakuvideo.com/~quu/H264Test/DirectShow%20splitter%20info.png), and ffdshow tryouts (DirectShow) (http://otakuvideo.com/~quu/H264Test/DirectShow%20video%20decoder%20info.png).
I have only used freely available tools for this testing, nothing requires a license for personal use.
the synthetic video is an AVS script of the following (my master AVS)
# original mp3 from http://freemusicarchive.org
# vlc 1.1.2 used to convert to wav file
audio = WAVSource("Fabrizio_Paterlini_-_01_-_Forever_blue_March_Rosetta_re-imagined_version.wav")
return ShowSMPTE(AudioGraph(AudioDub(BlankClip(length=8500, width=720, height=480, fps=24), audio), 10))
Overview of Results:
The best sync seams to be x264 video with LAME encoded MP3 for the audio.
Handbrake "High Profile" has a video sync issue, with missing frames in the resulting encoded video, no other video encoding method showed any frame loss
Handbrake adds extra frames to the end of my testing, it repeats the last couple of frames a second time
Handbrake audio is out of sync, not matter how it is rendered
The DirectShow playback of H.264 video was horrible, with sync problems becoming progressively worse as the video is played
Inside of AVISynth, FFAudioSource induces a ~1 frame shift in the audio. This needs to be tested in ffmpeg based stand alone players (VLC, etc)
NeroAACEncode adds a ~1 frame skew in audio, where FAAC does not. This is in addition to the FFAudioSource skew
I need to have my results reviewed, and see if there is additional things I need to adjust for. The timing problem with Nero is surprising, since I have been told that it was the superior encoder to FAAC. This is just my first pass at the timing results, I expect there to be things to fix and adjust. I know that AVISynth is not known for being the most accurate as far as audio goes, I just need a way to check for drift.
:EDIT:
It also seams that the timing problems with FFAudioSource go away if I use the MKV container. I need to test with VLC native, as it is either my usage of mp4box or the "player" of AVISynth
The Testing Methodology
All of the scripts and the source WAV file used can be found in this .zip file (http://otakuvideo.com/~quu/H264Test/src.zip) so my results can be verified and my scripts confirmed. I am including the WAV file instead of the mp3, just for consistency. All encoding steps were repeated three times and the resulting files compared. There was no inconsistencies in encoding, the results where binary identical for each iteration.
An uncompressed AVI and an uncompressed WAV file was created with VirtualDub using the master AVS file as the source. The sync from all three of these baselines (AVS, AVI, WAV) were checked and found to be identical.
Using Handbrake (http://otakuvideo.com/~quu/H264Test/handbrake%20version.png) and the baseline AVI, a "Normal" and "High Profile" encode was made. Other than selecting the preset, all other settings are default.
(command line notes) (http://otakuvideo.com/~quu/H264Test/test%20notes.txt)
Using the master AVS file, x264 (32 bit version) was used to encode two versions (main profile and high profile) of the video to raw H.264 files. Other than "--profile main" and "--profile high", no other optional command line settings where used.
Using LAME, the baseline WAV file was encoded to MP3. The "-b 192", 192 bit rate, and "-h", high quality encode, command line options were used.
Using FAAC, the baseline WAV was encoded to an AAC file (actually an .mp4 file with the AAC stream inside of it). No optional command line parameters where used.
Using NeroAACEncode, the baseline WAV was encoded to a different AAC file (actually an .mp4 file with the AAC stream inside of it). No optional command line parameters where used.
Using mp4box, both of the x264 encoded video streams were combined with the three different audio streams, resulting in 6 MPEG-4 files in total. Other than setting the frame rate at 24 fps (since the H.264 video was raw, not in a container), no optional command line parameters where used.
At this point I had 8 Audio/Video and 3 Audio files to compare to the baseline. For each test file, two AVS scripts where made for comparison.
The first was a simple comparison, with FFVideoSource/FFAudioSource used to make a simple display. For the audio only tests, the AudioGraph was rebuilt from the encoded audio. While this is good for testing the video initial sync, it could not tell if the audio and video were in sync with each other.
A second comparison script was built for each test file. Inside of this script, the video was decoded using FFVideoSource and DirectShowSource, and the audio was also decoded using FFAudioSource and DirectShowSource. Two additional "video sources" where made using the audio decoded both ways back through the AudioGraph plugin. Using this comparison script we can tell if the video is drifting out of sync with the audio, either way it is decoded.
Detailed Results
The video sync results, the most important is the time code in the bottom. Frame 4,000 was picked because it was easy to remember, and had an easy to see wave form.
The original master AVS (http://otakuvideo.com/~quu/H264Test/Original.png)
The baseline AVI (http://otakuvideo.com/~quu/H264Test/Original%20avi.png)
The baseline WAV (http://otakuvideo.com/~quu/H264Test/Original%20wav.png)
The baselines are identical to each other. This is just a confirmation that no errors are creeping in before the encoding occurs.
The encoded audio file test scripts are basically copies of the master AVS, but with FFAudioSource used instead of WAVSource.
The LAME file (http://otakuvideo.com/~quu/H264Test/LAME%20mp3.png)
The FAAC file (http://otakuvideo.com/~quu/H264Test/FAAC%20aac.png)
The NeroAACEncoder file (http://otakuvideo.com/~quu/H264Test/Nero%20aac.png)
All three files are out of sync with the baseline. The Nero encode more than the other two, but only by a little bit. This shows potential issues with the future results, since the audio is already coming out of the gate out of sync. (Please note, later testing used DirectShowSource for audio, and no sync issues found with FAAC and MP3)
Now the two Handbrake encodes. I was most interested in this, because Handbrake is easy and convenient to use.
Handbrake, Normal preset (main profile) (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29.png)
Handbrake, High Profile preset (high profile) (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29.png)
Well, the normal preset looks dead on, but the high profile has already lost a frame, and is running a frame faster. I reran the encode additional times, and each time it was binary identical. Also, both the normal (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20comparison%20-%20at%20end.png) and the high (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20comparison%20-%20at%20end.png) profile added additional frames to the end of the file. When I inspected the end, it seams that Handbrake just went back a couple of frames and replayed them again.
Finally the x264 encodes. the initial scrip only showcases the video sync, not the audio sync, so only the FAAC versions need to be shown.
x264 main profile (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC.png)
x264 high profile (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC.png)
The encoding is spot on, and no frame drift, loss, or extra frames with the x264 encoding.
The video comparison was only half of my testing, the second half was the audio and video sync. To me, this was the critical testing, If the video and audio get out of sync, then the viewing experience can be spoiled.
For the three audio comparisons, a script was made to compare the timings based on how it was decoded, and then also compared to the baseline WAV.
The LAME file comparison (http://otakuvideo.com/~quu/H264Test/LAME%20mp3%20comparison.png)
The FAAC file comparison (http://otakuvideo.com/~quu/H264Test/FAAC%20aac%20comparison.png)
The NeroAACEncoder file comparison (http://otakuvideo.com/~quu/H264Test/Nero%20aac%20comparison.png)
When using the FFAudioSource there is a obvious skew of about 1 frame across the board that is not seen in the DirectShowSource. The LAME MP3 source file, with DirectShow is spot on perfect in its timing. The FAAC timing is slow, but less than a frame slow. the Nero encoder is running a complete frame slow, even through the DirectShowSource.
For the audio/video comparisons, the file is decoded with DirectShow and ffmpeg and the compared. The audio is extracted and a new AudioGraph is rendered from it to show any potential audio skew that the mp4box muxing might have added. In total there is four graphs.
First is the two Handbrake videos. This will also be the first time the audio from Handbrake is checked for sync, so reference this image (http://otakuvideo.com/~quu/H264Test/Original.png) to see if it is running fast or not.
Handbrake, Normal preset (main profile) comparison (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20comparison.png)
Handbrake, High Profile preset (high profile) comparison (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20comparison.png)
the normal profile has audio that is running about half a frame slow, but the video is spot on when rendered from ffmpeg. with direct show the video is starting to drift. Strangely, the audio is close between ffmpeg and direct show, which was not seen in the stand alone encodes. The high profile comparison is a mess. the Direct Show is running almost 7 seconds slow, and the ffmpeg audio is showing the same 1 frame skew we have seen before.
The command line encodes are actually very similar. Since we have already shown that, video wise, there is no sync difference between x264 main and high profile, and we have seen the sync difference between the three audio formats, the results here should be predictable, we should only see problems if mp4box is broken.
x264 Main LAME (http://otakuvideo.com/~quu/H264Test/x264-main%20LAME%20comparison.png)
x264 Main FAAC (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC%20comparison.png)
x264 Main NeroAACEncoder (http://otakuvideo.com/~quu/H264Test/x264-main%20Nero%20comparison.png)
x264 High LAME (http://otakuvideo.com/~quu/H264Test/x264-high%20LAME%20comparison.png)
x264 High FAAC (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC%20comparison.png)
x264 High NeroAACEncoder (http://otakuvideo.com/~quu/H264Test/x264-high%20Nero%20comparison.png)
And the results are exactly as predicted. Everything run through the ffmpeg AVISynth decoder runs about a frame fast. The DirectShow video decoder is about 3-4 frames to slow. The LAME mp3 audio when decoded with DirectShow seams to be the best synced of all the audio, though FAAC is really close.
:Update:
I was reminded that VLC can export/encode to H.264/AAC, so I quickly gave that a try (most of the settings are deep in the config, but I tried with the defaults)
The video was on track, but it was the comparison that amazed me.
VLC H.264 + AAC MP4 Default settings (http://otakuvideo.com/~quu/H264Test/vlc%20-%20comparison.png)
The audio tracks were lined up, even the FFAudioSource. Of course the direct show video was all messed up, but that is normal.
I also decided to give the Matroska (MKV) container format a try, after all I still had the raw streams as files, using MKVMerge to make mkv files should be easy. I used the GUI to make the mkv files, only having to specify the frame rate. I then had to go in via the command line afterwords and remux them, removing the compression for stream -1.
But the results were startling
x264 Main LAME MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20LAME%20MKV%20comparison.png)
x264 Main FAAC MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC%20MKV%20comparison.png)
x264 Main NeroAACEncoder MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20Nero%20MKV%20comparison.png)
x264 High LAME MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20LAME%20MKV%20comparison.png)
x264 High FAAC MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC%20MKV%20comparison.png)
x264 High NeroAACEncoder MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20Nero%20MKV%20comparison.png)
The FAAC audio was synced, even with the FFAudioSource. Changing the container was not supposed to change the sync.
As a last bit of an experiment, I decided to remux the Normal profile handbrake video (Remember, the high profile was broken) into an MKV. I used the GUI to make a base MKV file, then the command line to remove the stream -1 compression. Also, I attempted to use the native MKV output to see what would happen.
Handbrake Normal (main profile) Remuxed with MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20MKV%20repack%20comparison.png)
Handbrake Normal (Main Profile) native as MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20MKV%20comparison.png)
Handbrake High Profile (High) native as MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20MKV%20comparison.png)
Well, that was unxpected, all the audio sync problems went away with MKV. High Profile is still broken.
Update Summary
Changing the container changed the sync. It could be that I am not using mp4box properly, or that the "reader" of AVISynth and the FFMS2 is doing that one frame "miss" in the FFAudioSource. Either way, there is more to test.
Depending on the results, MPEG-4 ASP and MPEG-2 video can also be tested, along with the potential of trying the MKV container format.
The Problem:
A video was brought to my attention which had been encoded from an MPEG-2 source to a h.264 MPEG-4 file using Handbrake. The sync on the resulting video was different than the original, so I decided to investigate the cause. Initial testing using the source video was problematic, so a synthetic video was created to refine the testing system. Once I am satisfied with the testing methodology and results, I can repeat the testing with the problem videos instead of the synthetic video.
My initial testing used the MSU VQMT application to detect differences in the encoded video files. There was problems with sync with this testing application (I later discovered it was DirectShow), and I had to use custom AVISynth scripts to fix the sync issues. As I started messing with AVISynth more, I discovered that I could test for the video sync and audio sync inside of a single script, with out needing the MSU VQMT application.
The testing involved the following encoding tools: Handbrake, x264, FAAC, NeroAACEncode, LAME, and mp4box.
The comparison tools used: VLC, AVISynth, VirtualDub, AudioGraph (AVS Plugin), FFmpegSource2 (AVS Plugin), Haali splitter (DirectShow) (http://otakuvideo.com/~quu/H264Test/DirectShow%20splitter%20info.png), and ffdshow tryouts (DirectShow) (http://otakuvideo.com/~quu/H264Test/DirectShow%20video%20decoder%20info.png).
I have only used freely available tools for this testing, nothing requires a license for personal use.
the synthetic video is an AVS script of the following (my master AVS)
# original mp3 from http://freemusicarchive.org
# vlc 1.1.2 used to convert to wav file
audio = WAVSource("Fabrizio_Paterlini_-_01_-_Forever_blue_March_Rosetta_re-imagined_version.wav")
return ShowSMPTE(AudioGraph(AudioDub(BlankClip(length=8500, width=720, height=480, fps=24), audio), 10))
Overview of Results:
The best sync seams to be x264 video with LAME encoded MP3 for the audio.
Handbrake "High Profile" has a video sync issue, with missing frames in the resulting encoded video, no other video encoding method showed any frame loss
Handbrake adds extra frames to the end of my testing, it repeats the last couple of frames a second time
Handbrake audio is out of sync, not matter how it is rendered
The DirectShow playback of H.264 video was horrible, with sync problems becoming progressively worse as the video is played
Inside of AVISynth, FFAudioSource induces a ~1 frame shift in the audio. This needs to be tested in ffmpeg based stand alone players (VLC, etc)
NeroAACEncode adds a ~1 frame skew in audio, where FAAC does not. This is in addition to the FFAudioSource skew
I need to have my results reviewed, and see if there is additional things I need to adjust for. The timing problem with Nero is surprising, since I have been told that it was the superior encoder to FAAC. This is just my first pass at the timing results, I expect there to be things to fix and adjust. I know that AVISynth is not known for being the most accurate as far as audio goes, I just need a way to check for drift.
:EDIT:
It also seams that the timing problems with FFAudioSource go away if I use the MKV container. I need to test with VLC native, as it is either my usage of mp4box or the "player" of AVISynth
The Testing Methodology
All of the scripts and the source WAV file used can be found in this .zip file (http://otakuvideo.com/~quu/H264Test/src.zip) so my results can be verified and my scripts confirmed. I am including the WAV file instead of the mp3, just for consistency. All encoding steps were repeated three times and the resulting files compared. There was no inconsistencies in encoding, the results where binary identical for each iteration.
An uncompressed AVI and an uncompressed WAV file was created with VirtualDub using the master AVS file as the source. The sync from all three of these baselines (AVS, AVI, WAV) were checked and found to be identical.
Using Handbrake (http://otakuvideo.com/~quu/H264Test/handbrake%20version.png) and the baseline AVI, a "Normal" and "High Profile" encode was made. Other than selecting the preset, all other settings are default.
(command line notes) (http://otakuvideo.com/~quu/H264Test/test%20notes.txt)
Using the master AVS file, x264 (32 bit version) was used to encode two versions (main profile and high profile) of the video to raw H.264 files. Other than "--profile main" and "--profile high", no other optional command line settings where used.
Using LAME, the baseline WAV file was encoded to MP3. The "-b 192", 192 bit rate, and "-h", high quality encode, command line options were used.
Using FAAC, the baseline WAV was encoded to an AAC file (actually an .mp4 file with the AAC stream inside of it). No optional command line parameters where used.
Using NeroAACEncode, the baseline WAV was encoded to a different AAC file (actually an .mp4 file with the AAC stream inside of it). No optional command line parameters where used.
Using mp4box, both of the x264 encoded video streams were combined with the three different audio streams, resulting in 6 MPEG-4 files in total. Other than setting the frame rate at 24 fps (since the H.264 video was raw, not in a container), no optional command line parameters where used.
At this point I had 8 Audio/Video and 3 Audio files to compare to the baseline. For each test file, two AVS scripts where made for comparison.
The first was a simple comparison, with FFVideoSource/FFAudioSource used to make a simple display. For the audio only tests, the AudioGraph was rebuilt from the encoded audio. While this is good for testing the video initial sync, it could not tell if the audio and video were in sync with each other.
A second comparison script was built for each test file. Inside of this script, the video was decoded using FFVideoSource and DirectShowSource, and the audio was also decoded using FFAudioSource and DirectShowSource. Two additional "video sources" where made using the audio decoded both ways back through the AudioGraph plugin. Using this comparison script we can tell if the video is drifting out of sync with the audio, either way it is decoded.
Detailed Results
The video sync results, the most important is the time code in the bottom. Frame 4,000 was picked because it was easy to remember, and had an easy to see wave form.
The original master AVS (http://otakuvideo.com/~quu/H264Test/Original.png)
The baseline AVI (http://otakuvideo.com/~quu/H264Test/Original%20avi.png)
The baseline WAV (http://otakuvideo.com/~quu/H264Test/Original%20wav.png)
The baselines are identical to each other. This is just a confirmation that no errors are creeping in before the encoding occurs.
The encoded audio file test scripts are basically copies of the master AVS, but with FFAudioSource used instead of WAVSource.
The LAME file (http://otakuvideo.com/~quu/H264Test/LAME%20mp3.png)
The FAAC file (http://otakuvideo.com/~quu/H264Test/FAAC%20aac.png)
The NeroAACEncoder file (http://otakuvideo.com/~quu/H264Test/Nero%20aac.png)
All three files are out of sync with the baseline. The Nero encode more than the other two, but only by a little bit. This shows potential issues with the future results, since the audio is already coming out of the gate out of sync. (Please note, later testing used DirectShowSource for audio, and no sync issues found with FAAC and MP3)
Now the two Handbrake encodes. I was most interested in this, because Handbrake is easy and convenient to use.
Handbrake, Normal preset (main profile) (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29.png)
Handbrake, High Profile preset (high profile) (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29.png)
Well, the normal preset looks dead on, but the high profile has already lost a frame, and is running a frame faster. I reran the encode additional times, and each time it was binary identical. Also, both the normal (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20comparison%20-%20at%20end.png) and the high (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20comparison%20-%20at%20end.png) profile added additional frames to the end of the file. When I inspected the end, it seams that Handbrake just went back a couple of frames and replayed them again.
Finally the x264 encodes. the initial scrip only showcases the video sync, not the audio sync, so only the FAAC versions need to be shown.
x264 main profile (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC.png)
x264 high profile (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC.png)
The encoding is spot on, and no frame drift, loss, or extra frames with the x264 encoding.
The video comparison was only half of my testing, the second half was the audio and video sync. To me, this was the critical testing, If the video and audio get out of sync, then the viewing experience can be spoiled.
For the three audio comparisons, a script was made to compare the timings based on how it was decoded, and then also compared to the baseline WAV.
The LAME file comparison (http://otakuvideo.com/~quu/H264Test/LAME%20mp3%20comparison.png)
The FAAC file comparison (http://otakuvideo.com/~quu/H264Test/FAAC%20aac%20comparison.png)
The NeroAACEncoder file comparison (http://otakuvideo.com/~quu/H264Test/Nero%20aac%20comparison.png)
When using the FFAudioSource there is a obvious skew of about 1 frame across the board that is not seen in the DirectShowSource. The LAME MP3 source file, with DirectShow is spot on perfect in its timing. The FAAC timing is slow, but less than a frame slow. the Nero encoder is running a complete frame slow, even through the DirectShowSource.
For the audio/video comparisons, the file is decoded with DirectShow and ffmpeg and the compared. The audio is extracted and a new AudioGraph is rendered from it to show any potential audio skew that the mp4box muxing might have added. In total there is four graphs.
First is the two Handbrake videos. This will also be the first time the audio from Handbrake is checked for sync, so reference this image (http://otakuvideo.com/~quu/H264Test/Original.png) to see if it is running fast or not.
Handbrake, Normal preset (main profile) comparison (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20comparison.png)
Handbrake, High Profile preset (high profile) comparison (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20comparison.png)
the normal profile has audio that is running about half a frame slow, but the video is spot on when rendered from ffmpeg. with direct show the video is starting to drift. Strangely, the audio is close between ffmpeg and direct show, which was not seen in the stand alone encodes. The high profile comparison is a mess. the Direct Show is running almost 7 seconds slow, and the ffmpeg audio is showing the same 1 frame skew we have seen before.
The command line encodes are actually very similar. Since we have already shown that, video wise, there is no sync difference between x264 main and high profile, and we have seen the sync difference between the three audio formats, the results here should be predictable, we should only see problems if mp4box is broken.
x264 Main LAME (http://otakuvideo.com/~quu/H264Test/x264-main%20LAME%20comparison.png)
x264 Main FAAC (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC%20comparison.png)
x264 Main NeroAACEncoder (http://otakuvideo.com/~quu/H264Test/x264-main%20Nero%20comparison.png)
x264 High LAME (http://otakuvideo.com/~quu/H264Test/x264-high%20LAME%20comparison.png)
x264 High FAAC (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC%20comparison.png)
x264 High NeroAACEncoder (http://otakuvideo.com/~quu/H264Test/x264-high%20Nero%20comparison.png)
And the results are exactly as predicted. Everything run through the ffmpeg AVISynth decoder runs about a frame fast. The DirectShow video decoder is about 3-4 frames to slow. The LAME mp3 audio when decoded with DirectShow seams to be the best synced of all the audio, though FAAC is really close.
:Update:
I was reminded that VLC can export/encode to H.264/AAC, so I quickly gave that a try (most of the settings are deep in the config, but I tried with the defaults)
The video was on track, but it was the comparison that amazed me.
VLC H.264 + AAC MP4 Default settings (http://otakuvideo.com/~quu/H264Test/vlc%20-%20comparison.png)
The audio tracks were lined up, even the FFAudioSource. Of course the direct show video was all messed up, but that is normal.
I also decided to give the Matroska (MKV) container format a try, after all I still had the raw streams as files, using MKVMerge to make mkv files should be easy. I used the GUI to make the mkv files, only having to specify the frame rate. I then had to go in via the command line afterwords and remux them, removing the compression for stream -1.
But the results were startling
x264 Main LAME MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20LAME%20MKV%20comparison.png)
x264 Main FAAC MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20FAAC%20MKV%20comparison.png)
x264 Main NeroAACEncoder MKV (http://otakuvideo.com/~quu/H264Test/x264-main%20Nero%20MKV%20comparison.png)
x264 High LAME MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20LAME%20MKV%20comparison.png)
x264 High FAAC MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20FAAC%20MKV%20comparison.png)
x264 High NeroAACEncoder MKV (http://otakuvideo.com/~quu/H264Test/x264-high%20Nero%20MKV%20comparison.png)
The FAAC audio was synced, even with the FFAudioSource. Changing the container was not supposed to change the sync.
As a last bit of an experiment, I decided to remux the Normal profile handbrake video (Remember, the high profile was broken) into an MKV. I used the GUI to make a base MKV file, then the command line to remove the stream -1 compression. Also, I attempted to use the native MKV output to see what would happen.
Handbrake Normal (main profile) Remuxed with MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20MKV%20repack%20comparison.png)
Handbrake Normal (Main Profile) native as MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20normal%20%28main%29%20MKV%20comparison.png)
Handbrake High Profile (High) native as MKV (http://otakuvideo.com/~quu/H264Test/Handbrake%20High%20Profile%20%28high%29%20MKV%20comparison.png)
Well, that was unxpected, all the audio sync problems went away with MKV. High Profile is still broken.
Update Summary
Changing the container changed the sync. It could be that I am not using mp4box properly, or that the "reader" of AVISynth and the FFMS2 is doing that one frame "miss" in the FFAudioSource. Either way, there is more to test.