VFR MKV Sync Problems

[TheBashar]

I'm having trouble with the audio getting out of sync in merged vfr mkv files. I put together a simple test case to illustrate this. Can someone more informed than I please clear up what I missing?

All the files to reproduce my test case can be downloaded from:
http://www.gigasize.com/get.php/481342/VFRSyncTest.7z

Basically it goes something like this:

Have AVS file which generates blank video and constant tone
Call to DeDup's DupeMC records the duplicate frame statistics
Another AVS (almost same) adds Info() and uses DeDup to remove what would have been duplicate frames.
DeDup outputs timecode file
Another set of AVS files are almost identical but with different color and tone.
I use MeGUI to encode the video to MKV
I use MeGUI to encode the audio with Nero AAC
I use MKVMergeGUI to merge each video MKV, audio MP4, and corresponding timecode file
I use MKVMergeGUI to repeatedly merge the two clips together in an A-B-A-B-A-B pattern


Problem: When you watch the ABABAB.mkv, you see that the audio tone switches in time with the color changes. But as the video progresses (actually as the merge points are passed) the color changes lag farther and farther behind the audio.

Observations that I don't understand:

If I do the alternating in one video clip, the video dooesn't get out of sync; only with the merges.
When I watch the A (or B) clip, in MPC I see only 2 distinct frames, though there should be 4.
When I watch the merged ABABAB clip, I see 4 distinct frames per color, except the last one which shows only 2 distinct frames.


I could use the audio delay parameter in MKVMergeGUI to fix this if I could figure out some pattern to how much the audio is getting advanced (or chopped I think). While the pattern seems predictable with this simple test, I've had trouble with real video sometime getting a little out of sync and sometimes a lot.

In case you don't want to download the 84kB archive, here are the AVS files so you can see how simple this test should be:

a1.avs:
vid = BlankClip(length=75, width=320, height=240, pixel_type="YV12", fps=15.0, audio_rate=22050, stereo=false, color=color_firebrick)
aud = Tone(length=6, frequency=440, samplerate=22050, channels=1, type="sine")
CLIP = AudioDubEx(vid, aud)
Trim(CLIP, 0, -CLIP.framecount, true)
DupMC(log="a.dup")


a2.avs:
vid = BlankClip(length=75, width=320, height=240, pixel_type="YV12", fps=15.0, audio_rate=22050, stereo=false, color=color_firebrick)
aud = Tone(length=6, frequency=440, samplerate=22050, channels=1, type="sine")
CLIP = AudioDubEx(vid, aud)
Trim(CLIP, 0, -CLIP.framecount, true)
Info()
DeDup(log="a.dup", times="a.tmc", maxcopies=20, maxdrops=20, decwhich=0)


b1.avs:
vid = BlankClip(length=75, width=320, height=240, pixel_type="YV12", fps=15.0, audio_rate=22050, stereo=false, color=color_turquoise)
aud = Tone(length=6, frequency=660, samplerate=22050, channels=1, type="sine")
CLIP = AudioDubEx(vid, aud)
Trim(CLIP, 0, -CLIP.framecount, true)
DupMC(log="b.dup")


b2.avs:
vid = BlankClip(length=75, width=320, height=240, pixel_type="YV12", fps=15.0, audio_rate=22050, stereo=false, color=color_turquoise)
aud = Tone(length=6, frequency=660, samplerate=22050, channels=1, type="sine")
CLIP = AudioDubEx(vid, aud)
Trim(CLIP, 0, -CLIP.framecount, true)
Info()
DeDup(log="b.dup", times="b.tmc", maxcopies=20, maxdrops=20, decwhich=0)


And this is one of the timecode files (the other is identical):
# timecode format v2
0.000000
1333.333333
2666.666667
4000.000000


Finally a note about the archive. It contains all the files to recreate the test videos as well as the videos themselves. I just want to note that it contains a "ababab.tmc" file which is a timecode file _extracted_ from the ababab.mkv file, NOT used to create the video file.

[foxyshadis]

I suspected when I read, but I confirmed when I made a test.

a is exactly 5000 ms long, and the last frame is 4966. But in the timecodes, note that the first frame b starts at 5333 ms, not 5000. This is a failure of the timecode v2 format: There is no duration for the last frame. You end up with mkvmerge having to make up a number - and it picks the length of the frame prior to the last. (That's just a guess, I'm still investigating the exact method.) Doing anything like this, you'll basically have to be absolutely sure you have two of the same length frames on the end.

Tacking the start time of the next segment onto the timecode file doesn't work, sadly. Perhaps Mosu can be convinced to support that.

[TheBashar]

a is exactly 5000 ms long, and the last frame is 4966. But in the timecodes, note that the first frame b starts at 5333 ms, not 5000.

Thanks for looking into the problem foxyshadis. Where did you get the 4966 number from? I don't know how you were able to dig that out.

I've done some investigating of my own, and continue to be perplexed by what I find. I re-mkvmerged the individual a.mkv and b.mkv with artificial timecode files. I constructed these so each successive frame's duration is 100ms more than the previous.

[Note: I added a DeDup override to keep the first frame so you'll see an extra frame from my previous post's tests - now 5 frames per clip]

inca.tmc: (pseudo syntax to save space here)
0.000000, 100, 300, 600, 1000
incb.tmc: (pseduo syntax to save space here)
0.000000, 500, 1100, 1800, 2600

I then mkvmerge these two timecoded videos together (ababab) and extract the merged video's timecode file:

mergeinc.tmc: (pseduo syntax etc etc)
0.000000, 100.000000, 300.000000, 600.000000, 1000.000000,
1100.000000, 1600.000000, 2200.000000, 2900.000000, 3700.000000,
4200.000000, 4300.000000, 4500.000000, 4800.000000, 5200.000000,
5300.000000, 5800.000000, 6400.000000, 7100.000000, 7900.000000,
8400.000000, 8500.000000, 8700.000000, 9000.000000, 9400.000000,
9500.000000, 10000.000000, 10600.000000, 11300.000000, 12100.000000

Notice here, at each merge point the duration of the previous clips last frame is set to the duration of said clips first frame. This would be an understandable rule IF it held true!

Here's the kicker, I got the "real" a.tmc file and I manually changed the duration of the first frame so if I add that duration to the last frame it would be exactly 5000ms - what the clip length is. So here's my timecode file:

a.tmc:
0.000000, 933.333333, 1400.000000, 2733.333333, 4066.666667
933.333333 + 4066.666667 = 5000 right?

b.tmc:
0.000000, 66.666667, 1400.000000, 2733.333333,
4066.666667

Now, after mkmerging these into the individual video files, mkvmerging the two videos together, and extracting the merged video's timecode file I, unfathomably find:

merged.tmc:
0.000000, 933.000000, 1400.000000, 2733.000000, 4067.000000,
5400.000000, 5467.000000, 6800.000000, 8133.000000, 9467.000000,
10801.000000, 11734.000000, 12201.000000, 13534.000000, 14868.000000,
16201.000000, 16268.000000, 17601.000000, 18934.000000, 20268.000000,
21601.000000, 22534.000000, 23001.000000, 24334.000000, 25668.000000,
27002.000000, 27069.000000, 28402.000000, 29735.000000, 31069.000000

Now mysteriously the merge points all have durations of 1333ms - which is a totally different pattern than the previous test. Is it now looking at the duration of the 2nd to last frame?

IF I could figure out how it was determining the inter-merge duration, I could write a utility to patch the timecode file(s) to make it fit the actual video length. But, even if I could figure that out, my trouble doesn't end there.

I made an artificial timecode file which I knew would match the actual merged ababab and used mkvmerge to replace the timecodes on the merged video:

replace.tmc:
0.000000, 1000.000000, 2000.000000, 3000.000000, 4000.000000
5000.000000, 6000.000000, 7000.000000, 8000.000000, 9000.000000
10000.000000, 11000.000000, 12000.000000, 13000.000000
14000.000000, etc, etc, etc

Now, when I view this new video, the audio slowly falls behind the video at each successive merge point! Now I now the timecodes for the video match the exact clip lengths. What is causing the audio to be elongated or delayed?

Current Conclusions:

Audio sync problems are caused (in part) by mkvmerge assigning final frame durations based on some algorithm other than looking at audio length.
Audio sync problems are also caused by a yet unknown secondary source related to the merge points.


Though you still need some of the files from the link in the first post, you can get the updated scripts and files to reproduce these tests at:
http://www.gigasize.com/get.php/483970/Sync_Test_Updated_Files.7z

[foxyshadis]

Bizarre. (I just meant the last frame pre-decimation with dedup, sorry.)

Since the behavior is unpredictable, the only way to work with it will be to fix mkvmerge. Adding the capability of reading a final end-duration timecode would probably be the most correct solution. I'll point Mosu to the thread. Once that's done, it'll be much simpler to find out if there are any further audio merge problems.