Matching audio streams from two movies

[nji]

(This is a re-post of a thread I started in a forum at a different site.
I seems that my question is too technical, so maybe I'm at the right place here :) )

From time to time it happens that I have a movie in two different versions;
with different languages each.
I want to have the two audio streams in one container of the video.
Extracting (losslessly) and re-muxing the audio streams I do with AviDemux.

Sometimes it happens that the audio streams don't match by time.
In that case I take an audio editor to match their start,
and after that eventually stretch/squeeze in time the to-be-inserted.
Then save in wav-format, to load it into the video container.
That works fine for most cases.

But sometimes... it doesn't:
Although having done a "perfect match" near start and end,
in-between there is desync to the movie.
(Both movies have constant framerate mode).

My actual example:

The master movie:
(It is partwise in Swiss German; very hard to understand for me.)
https://archive.org/download/Der-Richter-und-sein-Henker_Krimi-von-Friedrich-Duerrenmatt_MFG-TRAC_1978/Der-Richter-und-sein-Henker_Krimi-von-Friedrich-Duerrenmatt_MFG-TRAC_1978.mp4

The additional audio is from a DVD I own of that movie
(German audio, but worse video quality).
To-be-added audio:
https://c.gmx.net/@1155842887640945833/bXHQAygfegQunqa1mK-rHw

MediaInfo of the DVD (where audio above is taken from):

General
Complete name : D:\B.mkv
Format : Matroska
Format version : Version 2
File size : 3.53 GiB
Duration : 1 h 28 min
Overall bit rate mode : Variable
Overall bit rate : 5 719 kb/s
Encoded date : UTC 2019-12-17 19:49:08
Writing application : MakeMKV v1.14.4 win(x64-release)
Writing library : libmakemkv v1.14.4 (1.3.5/1.4.7) win(x64-release)

Video
ID : 1
ID in the original source m : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings : CustomMatrix / BVOP
Format settings, BVOP : Yes
Format settings, Matrix : Custom
Format settings, GOP : M=3, N=15
Format settings, picture st : Frame
Codec ID : V_MPEG2
Codec ID/Info : MPEG 1 or 2 Video
Duration : 1 h 28 min
Bit rate mode : Variable
Bit rate : 5 522 kb/s
Maximum bit rate : 9 800 kb/s
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 4:3
Frame rate mode : Constant
Frame rate : 25.000 FPS
Standard : PAL
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.533
Time code of first frame : 10:00:00:00
Time code source : Group of pictures header
GOP, Open/Closed : Closed
Stream size : 3.41 GiB (97%)
Language : English
Default : No
Forced : No
Original source medium : DVD-Video

Audio
ID : 2
ID in the original source m : 189 (0xBD)128 (0x80)
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Codec ID : A_AC3
Duration : 1 h 28 min
Bit rate mode : Constant
Bit rate : 192 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 31.250 FPS (1536 SPF)
Compression mode : Lossy
Stream size : 121 MiB (3%)
Title : Stereo
Language : German
Service kind : Complete Main
Default : Yes
Forced : No
Original source medium : DVD-Video

First step should always be to understand the cause.

What on earth causes this strange unregulary desync?

It can't be two different cuts of the movie, as the cuts would be
much more different than some 100ms here and there.
Or is it something about the timestamps... kind of association of
the containers with the streams? (I don't know much about that).
I hope somebody can tell.

OK, I did without understanding then... :)
Took both audios in an audio editor (ancient Audition 3.0),
synced their start (DVD -0.58s) and compared at some points.
There seem to be MANY small (100ms etc.) insertions/ deletions
here and there. Repairing hundreds of them manually...? Impossible!
(Some of the expensive audio editors seem to be capable of automatic syncing...)

Next I visualized some timestamps' differences:

https://c.gmx.net/@1155842887640945833/fo8p1tlGeHjQz2IXiWzvmQ

Getting ever stranger...

The only practical way to sync I came to was to partwise
linearize the diff graph (squeeze/quench) these audio parts,
and repair the steps (cutting/ including).
This is what I did:

https://c.gmx.net/@1155842887640945833/VEsOPZf_c0Arl1nGudXX_g

The result is not so bad... but I'm not too satisfied.
(The result is not optimal, and the lineariziation is wrong by principle).

So my question to the professionals for comments on my thinking. :)

[Emulgator]

This is to be expected. Different releases may (and do) have different edits.
Open them in AviSynth side by side and you will spot them easily.
Often single dirty frames had been trimmed.
My 3 discs of "Capricorn" (a DVD release, a BD-AVC release and a BD-MPEG-2 release)
show a frame bilance of MPEG-2 minus AVC = 19 f, DVD minus MPEG-2BD = -475f
I had to go the same route, make an Excel spreadsheet and hand match per cut.

[tebasuna51]

As you've already discovered, when both versions come from the same source with delays and frame rates, they can usually be easily adjusted.
The problem arises when the sources are different.
Let's suppose one or both come from a TV capture with ads inserted. When the ads are removed, the cuts are never perfect, and discrepancies can occur in several places.
Or in some countries, a scene has been censored.
In these cases, only tedious manual editing can produce the best approximation.

[nji]

First of all: Thank you both for the feedback :)
Now I'm a bit assured not being on the wrong track.

I attach the LO calc document I built for convenience - take it if you like.

However I'm thinking about automatic identifying the desyncs.
That would be 1. automatic and 2. complete.
When done I'll post it in sub-forum "devolopment" and post a link from here.

Do you mind a follow-up question?
...
(I knew you wouldn't :) )

What would be the workflow if one or both movies are vfr?

[tebasuna51]

If one, or both, movies have VFR and you want preserve the video VFR you just need to consider the video as the sum of the video stream plus the timecode file as a CFR video at the average fps (last timecode/num of frames).
And after do the mux with the same timecode file.

[nji]

And that's exactly the point where I've been worried from beginning:
The lack of some knowledge about some connection between
container, video and audio in some special constellation!

OK, now I learned, that I've to take care on that only for vfr. (Right?)

"Preserving VFR"?
Yes, of course.
Converting to CFR would'nt be without image quality degradation.

If I get you right, then first place I've to build the timecode file(s) from vfr(s)?

Having exported losslessy both audio streams...
They won't be "vfr" I assume?
How do they associate with the timecode file?
Not at all? I.e. when playing muxed vfr video and (cfr) audio "someone"
takes care to take the right audio samples?

However:
How do I go on with my task syncronizing the to-be-added audio stream
to the master audio stream?
Just simple, as before?
And then just insert/ mux the manipulated audio (e.g. with ADM)?

Is my thinking correct?

[tebasuna51]

If I get you right, then first place I've to build the timecode file(s) from vfr(s)?
Always you need first extract the timecodes from the container with MkvExtract, Mp4Box... or using AviSynth decoders.

Having exported losslessy both audio streams...
They won't be "vfr" I assume?
Video frames can be displayed for a variable time but there is no equivalence with audio samples that must be played back for a fixed time depending on the global samplerate of 44100, 48000,...

How do I go on with my task syncronizing the to-be-added audio stream to the master audio stream?
You can make twitches/stretches depending on the framerates of both sources and add delays or cuts comparing the two audios (common noises since not dialogues).

And then just insert/ mux the manipulated audio (e.g. with ADM)?
ADM? The proper tool to mux Mp4Box, MkvMerge,...

[nji]

Ad automatic syncing - just some information I came to meanwhile:

Premiere Pro, Davinci Resolve and (probably) Adobe Audition seem to be capable of it.

DIY idea:
Identify isolated, short (but not too short) peaks in both audios as candidates for syncing.
Use dynamic programming alg to find a matching with minimized
- shifting of the lower row and
- number of left-out points

[marsoupilami]

Hi!
You might need exactly this:
https://forum.doom9.org/showthread.php?p=2025938#post2025938
:)