A/V Sync issue on reencoding MKV w/ AC3 to MP4 w/ AAC

[Soichiro]

I wasn't entirely sure where to put this but this seemed like the best place.

I have multiple sets of MKV videos that are H.264 with AC3 5.1 audio tracks. With each of these, I am going through the following process:

Demux AC3 audio track with mkvextract
Reencode video track and output to raw .264 file
Decode AC3 audio track to WAV with eac3to
Encode WAV audio track to AAC with qaac
Mux new H.264 video track and AAC audio track into MP4 with mp4box

However, this seems to result in an A/V desync that grows the further along the video goes (to a max of 1-2 seconds desync in worst cases, .5 seconds in favorable cases). Some things I've tried are:

Use valdec instead of eac3to as the decoder (issue persists)
Mux new video and audio into MKV (issue persists)
Mux original video and new audio into MKV (issue is fixed)
Mux new video and original audio (issue persists)
Mux new video and audio into MP4 using ffmpeg (video becomes choppy)
Encode the intermediate video stream to mp4 instead of .264 before muxing new video and audio to mp4 (issue persists)

For what it's worth, if I follow the original process for a video with DTS audio (instead of AC3), there is no A/V sync issue.

[tebasuna51]

Mux original video and new audio into MKV (issue is fixed)

Then the problem is in this pass:

"Reencode video track and output to raw .264 file"

Check if original and recoded video have the same number of frames and fps.

BTW use always eac3to to extract audio tracks (if there are some delay in audio track is corrected), here you can recode with qaac directly.
If AC3 is track 2:

"EAC3PATH/eac3to" "INPUT.mkv" 2: stdout.wav | "QAACPATH/qaac" -V 99 --ignorelength -o "OUTPUT.m4a" -

Replace upercase with exact values and desired parameters of qaac.

[Soichiro]

Good tip on extracting with eac3to, I didn't know it could do that. (Tip for anyone else reading: In eac3to, the track numbers start at 1, whereas in mkvextract they start at 0.)

Unfortunately it didn't help with the issue. I checked the frames and framerate as well. The source and output are the same (in this case, 35260 frames at 24000/1001 fps). There doesn't appear to be any delay on the audio track in the source file either.

[sneaker_ger]

There could be a delay on the video track (or even worse: variable framerate). Or the constant framerate is misdetected/not correctly chosen during conversion. I don't know if eac3to syncs to timecode 0ms or to first video frame when extracting.

[Soichiro]

MediaInfo reports that the source video is CFR 23.976 fps. I'm not sure if it would tell me if there's a delay on the video track but it doesn't seem to indicate that there is one (there are 4 tracks on this source, the only one that reports a delay is the 4th (commentary audio track) of 9ms).

I'm not sure if it will help but here is the output of eac3to, qaac, and mp4box (I didn't rerun the video encode because it would be more time consuming):

MKV, 1 video track, 3 audio tracks, 4 subtitle tracks, 0:24:31, 24p /1.001
1: h264/AVC, English, 720p24 /1.001
2: FLAC, Japanese, 2.0 channels, 0:24:30, 16 bits, 48kHz
3: AC3, English, 5.1 channels, 448kbps, 48kHz
4: AAC, English, 2.0 channels, 48kHz, 9ms
"Commentary"
5: Subtitle (ASS), English
6: Subtitle (SRT), English
7: Subtitle (ASS), English, "Signs"
8: Subtitle (SRT), English, "Signs"
a03 ac3, 48000, 5.1
a03 Extracting audio track number 3...
a03 Decoding with libav/ffmpeg...
a03 Reducing depth from 64 to 24 bits...
a03 Writing WAV...
a03 Creating file "D:\Encoding\Videos\TestFile\TestFile.wav"...
Video track 1 contains 35260 frames.
eac3to processing took 28 seconds.
Done.
qaac 2.52, CoreAudioToolbox 7.9.9.6

TestFile.m4a
AAC-LC Encoder, ABR 448kbps, Quality 96
[100.0%] 24:29.663/24:29.663 (29.9x), ETA 0:00.000
70543872/70543872 samples processed in 0:49.219
Overall bitrate: 441.728kbps
Optimizing...done
IsoMedia import TestFile.mp4 - track ID 1 - Video (size 1280 x 720)
IsoMedia import TestFile.m4a - track ID 1 - Audio (SR 48000 - 6 channels)
Saving D:\Users\User\Videos\TestFile.mp4: 0.500 secs Interleaving

[tebasuna51]

(Tip for anyone else reading: In eac3to, the track numbers start at 1, whereas in mkvextract they start at 0.)
Correct, also eac3to sort tracks to show video tracks first, after audio tracks and subs at last.
That is not always true using mkvextract, then using eac3to you need use the track number than eac3to inform.

I checked the frames and framerate as well. The source and output are the same (in this case, 35260 frames at 24000/1001 fps). There doesn't appear to be any delay on the audio track in the source file either.

If that is true I can't understand how the AAC is in sync with ORIGINAL video and desync with RECODED video.

BTW, to avoid wrong MediaInfo data (sometimes not exact), you can try (if track 0 is the video track):

"MKVTOOLNIXPATH\mkvextract.exe" timecodes_v2 "ORIGINAL.mkv" 0:"timecodes.txt"

And load the video timecodes.txt in the RECODED video track (MkvMergeGUI -> Select video track -> General track options tab -> Timecodes/Browse)

That must solve the video delay or variable fps problems than sneaker_ger say.

[kuchikirukia]

Why are you reencoding 448k AC3 to 448k AAC?

[Soichiro]

Why are you reencoding 448k AC3 to 448k AAC?

I have my reasons. Mostly I'm OCD and want all my files to be in the same format. This isn't for distribution or anything like that, it's just for personal storage. I have reasons why it needs to be in an MP4 container as well (because I would much rather use MKV) but unfortunately keeping AC3 and remuxing to MKV hasn't helped with the issue at hand.

[Soichiro]

Correct, also eac3to sort tracks to show video tracks first, after audio tracks and subs at last.
That is not always true using mkvextract, then using eac3to you need use the track number than eac3to inform.



If that is true I can't understand how the AAC is in sync with ORIGINAL video and desync with RECODED video.

BTW, to avoid wrong MediaInfo data (sometimes not exact), you can try (if track 0 is the video track):

"MKVTOOLNIXPATH\mkvextract.exe" timecodes_v2 "ORIGINAL.mkv" 0:"timecodes.txt"

And load the video timecodes.txt in the RECODED video track (MkvMergeGUI -> Select video track -> General track options tab -> Timecodes/Browse)

That must solve the video delay or variable fps problems than sneaker_ger say.

I tried this process of muxing the timecodes into the recoded video track and the sync issue still occurs.

[tebasuna51]

For me all your test discard problems with audio conversion or muxer issues.

That and:
Mux new video and audio into MP4 using ffmpeg (video becomes choppy)
point to a problem in video conversion.

Maybe the recoded video have invalid frames than players skip "(video becomes choppy)" and produce async with the correct audio.

Please describe your method to recode video and players used than show async.

[Soichiro]

Please describe your method to recode video and players used than show async.

The video encode loads the original through Avisynth using ffms2, then uses vfw4x264 to pipe from 32-bit avisynth to 64-bit x264. I'm on the latest avisynth and x264. My x264 options look like this:

vfw4x264 --crf 18 --keyint 400 --min-keyint 12 --ref 8 --mixed-refs --no-fast-pskip --b-adapt 2 --bframes 8
--b-pyramid normal --weightb --direct spatial --subme 10 --trellis 2 --partitions all --psy-rd 0.7:0.0 --deblock -1:-1
--aq-strength 0.7 --me umh --merange 24 --fade-compensate 0.5 --fgo 5 --rc-lookahead 60
--vbv-maxrate 40000 --vbv-bufsize 30000 --colormatrix bt709 --colorprim bt709 --transfer bt709 --output "%~dpn1.264" %1

The output of x264 looks like this:

raw [info]: 1280x720p 0:0 @ 24000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
x264 [info]: profile High, level 4.1
x264 [info]: cabac=1 ref=8 deblock=1:-1:-1 analyse=0x3:0x133 me=umh subme=10 psy=1 fade_compensate=0
.50 psy_rd=0.70:0.00 mixed_ref=1 me_range=24 chroma_me=1 trellis=2 8x8dct=1 cqm=0 deadzone=21,11 fas
t_pskip=0 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interl
aced=0 bluray_compat=0 constrained_intra=0 fgo=5 bframes=8 b_pyramid=2 b_adapt=2 b_bias=0 direct=1 w
eightb=1 open_gop=0 weightp=2 keyint=400 keyint_min=12 scenecut=40 intra_refresh=0 rc_lookahead=60 r
c=crf mbtree=1 crf=18.0000 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=40000 vbv_bufsize=30000
crf_max=0.0 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:0.70
frames fps kb/s elapsed remain size est.size
[100.0%] 35260/35260 25.07 1748.55 0:23:26 0:00:00 306.54 MB 306.54 MB
x264 [info]: frame I:303 Avg QP:10.94 size:154394
x264 [info]: frame P:7934 Avg QP:16.41 size: 21494
x264 [info]: frame B:27023 Avg QP:20.67 size: 3853
x264 [info]: consecutive B-frames: 4.0% 4.9% 7.5% 20.5% 8.8% 16.9% 5.9% 24.6% 6.7%
x264 [info]: mb I I16..4: 8.0% 63.2% 28.8%
x264 [info]: mb P I16..4: 1.2% 6.3% 2.2% P16..4: 29.8% 10.0% 14.8% 0.4% 0.2% skip:35.2%
x264 [info]: mb B I16..4: 0.1% 0.4% 0.1% B16..8: 23.2% 2.6% 0.9% direct: 2.5% skip:70.2% L
0:45.6% L1:50.9% BI: 3.5%
x264 [info]: 8x8 transform intra:64.5% inter:63.9%
x264 [info]: coded y,uvDC,uvAC intra: 76.3% 76.0% 51.8% inter: 7.5% 9.4% 1.4%
x264 [info]: i16 v,h,dc,p: 21% 16% 12% 51%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 10% 10% 10% 12% 10% 11% 10% 13%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 8% 7% 11% 14% 12% 12% 11% 13%
x264 [info]: i8c dc,h,v,p: 44% 21% 20% 15%
x264 [info]: Weighted P-Frames: Y:6.7% UV:6.1%
x264 [info]: ref P L0: 53.4% 5.7% 15.7% 7.6% 5.2% 4.7% 3.9% 2.8% 0.9% 0.1%
x264 [info]: ref B L0: 67.9% 13.7% 7.8% 4.2% 3.3% 2.1% 1.0%
x264 [info]: ref B L1: 88.9% 11.1%
x264 [info]: kb/s:1748.55

encoded 35260 frames, 25.07 fps, 1748.55 kb/s, duration 0:23:26.50

I am going to try running an encoding pass with verbose mode to see if that tells me anything.

I've attempted playback for this particular file in MPC-HC and VLC, both producing the sync issue. I've also experienced the same issue with another set of files played back through Chrome's HTML5 player, but have not tested that with this specific file.

Edit: I didn't find anything unusual in the output from the verbose encode.

[tebasuna51]

All seems ok.

And the AviSynth script?

EDIT:
Try this simple script to reject problems:

FFVideoSource("ORIGINAL.mkv", threads=1)
AssumeFps(24000,1001)
Spline36Resize(1280,720) # if necessary, also some Crop()

[Soichiro]

That didn't appear to make a difference. I also tried forcing x264 to 1 thread which also didn't fix the issue.

I'm going to try with 64-bit avisynth directly to see if the pipe through vfw4x264 is causing the issue.

[sneaker_ger]

Piping should not be creating such issues.
Either:
- problem with your script
- problem with ffms2 (try latest RC (http://forum.doom9.org/showpost.php?p=1734855&postcount=2136))
- problem with muxing

[Soichiro]

Well, I tried:

64-bit Avisynth+
Latest RC of FFMS2

Neither of which changed anything. And the script isn't doing anything but sourcing with FFVideoSource.

I was thinking it was potentially the muxing but if I mux the original video with the new audio to MP4 there's no sync issue. But the original video and the new video both appear to have the same framerate and number of frames. Plus it happens whether I mux to MKV or MP4. I'm fairly certain the issue is in the video encoding step.

[kuchikirukia]

How are you counting frames? I hope you're not using FFMS2 and expecting it to come out with anything other than the borked output it fed into x264.

Just encode VFR with Handbrake.

[Soichiro]

How are you counting frames? I hope you're not using FFMS2 and expecting it to come out with anything other than the borked output it fed into x264.

Just encode VFR with Handbrake.

I've used multiple methods to check the frame count and they all return the same number (for source and output). I'm also 99% certain that this is CFR material.

[kuchikirukia]

Encode it VFR with Handbrake.

[tebasuna51]

Neither of which changed anything. And the script isn't doing anything but sourcing with FFVideoSource.

Only this exactly?

FFVideoSource("ORIGINAL.mkv", threads=1)
AssumeFps(24000,1001)

Try also:

LWLibavVideoSource("ORIGINAL.mkv", threads=1)
AssumeFps(24000,1001)

[Soichiro]

No luck with LWLibavVideosource either. I've now tried the following input methods:

FFVideoSource through Avisynth on Windows
LWLibavVideoSource through Avisynth on Windows
Direct input to x264 on Windows (internal ffms)
FFVideoSource through wine Avisynth on Linux
Direct input to x264 on Linux (internal ffms)
DGAvcDec to AVCSource through Avisynth on Windows

None of which work. However, I did find something interesting. If I use MKVToolnix to mux and manually set the FPS for the video to 24000/1001p, the sync issue is fixed. It also works if I manually specify the FPS in MP4Box. More interestingly, sneaker_ger took a look at the file and didn't have any sync issues transcoding with FFVideoSource and muxing to MKV with (I'm assuming) default settings.

For now I can manually specify the FPS in the muxer as a workaround but it's still strange that sneaker_ger doesn't experience the issue with the same source file.

[sneaker_ger]

His input sample was totally regular. No delays or VFR at all. His output file had 10 extra frames at the beginning (and 10 frames missing at the end so total frame count was exactly like source). Since the beginning and ending were just black anyways you can't see any weird effects (looping etc.).
But that issue is impossible to fix by manually setting fps in mkvmerge so we are now talking about different issues. You must be mixing things up now. If you want further help I suggest you alway post exactly and fully what you are doing. Complete AviSynth script, complete command lines and logs etc. Always make sure fps is corretcly set. And please look for those extra frames I mentioned every time. Just open the script in VirtualDub and see where the first non-black picture starts. In the original it was at frame 24, in your output it was at frame 34. That way we don't have to rely on your hearing or false fps choice.

[Soichiro]

So the strangest thing has happened. I am now no longer able to replicate the issue I was having previously. It's particularly strange because I didn't change anything between then and now that should have fixed it.

I suppose we can consider this closed and say it was caused by ghosts.