RubberDub released - NEW audio import & syncing tool for mkv

[marsoupilami]

Hi!
Long time ago I tried to get the audio track from a VHS tape in sync with a DVD - a really painful job!
Now I finished writing a tool for doing this - "transplanting" a desired language track from a "bad" into a HD movie.

Main concept is to "tack" characteristic audio patterns with "syncpoints" together.
The imported audio is then "rubberbanded" (stretched) between those syncpoints.
Workflow should be intuitive - please grab into the included "readme" for further explanations.

Current restrictions:
The imported track is always converted to 16bit stereo, resampled & recoded to ac3 (or aac?).
Output is always a .mkv
Optional text subtitle adding is provided. Some fundamental edit fuctions for .srt format only.

I couldn't manage to create a .deb package - only a AppImage (created Ubuntu22.04) and a zip folder Win10_64 (should run standalone) is available.
Not tested for other os versions.

FFmpeg installed & inside execution path is required.

Binaries for Win10_64 and Linux-x86_64 & readme are here:
https://github.com/marsoupilamigit/RubberDub

[nji]

Finally!
Finally I have the time to check your app.

But... first of all... thank you for providing it!

I checked it on a new movie case I stumbled upon:
An US movie from the 40ies I own as DVD with orig and German audio.
Just recently I get the hands on a restored version (abt. HD), with German audio.
As the orig audio is much livelier (the German sync seems to be overduped upon
the orig audio, so the background noise is nearly gone. Sounds like a chamber play.)
I wanted to use the orig audio from DVD for the HD.
Sadly at the DVD there seem lot of unregular miss out frames all-over, it sums up to abt. 20s in 2h.
A very scattered CONVOY

My impression using AppImage @ OpenSUSE Leap 15.6:

Its big advantage:
you can see right away if your repairs are sufficient so far,
or if you are missing "unexpected" desyncs (so have insert more sync points).
With every refinement the matching gets better.
Keep in mind that the actual cause of desync won't be time stretch issues,
but cuts/ inserts; so stretching is just a workaround.
But refining the interval will get you automatically to the points
where the cuts/ inserts are obvious and can be mended by this tool too.

Its big drawback:
As the program can be used already, it seems at alpha state.
Lot's of things should be repaired/ improved so that it's really usable.
This holds especially for its main ideal.
But these seem to be only simple changes.

Sadly the author seems having to quit dev;
https://forum.doom9.org/showthread.php?t=186740
Still it's open source:
https://github.com/marsoupilamigit/RubberDub
Maybe someone will fork?

The two waveforms are shown with respect to the matched sync points
only with realtime playback,
or by clicking "ReSync" for master and adjustee waveform (again & again).
The 4 (fast) forward/ backward buttons, and also scrubbing
with the 3rd time line (= set both) don't do.
I tried "Stretch&Post" settings, without any success.
the same with timeline scrubbing.

In other words:
When having set 2 sync points in both movies
(say one at 10min, the other at 50min)
it would be most useful to know if the adjusting of that interval
was sufficient not only for its borders (10min and 50min),
but also in-between.
So I would move the "both movies" time slider and see
if the amplitudes match in-between.
If they don't, I have to set more sync points.
But:
I found no way to have RD show the to-be-adjusted waveform
stretched/ squeezed to the master interval.
Moving from first sync point towards the second,
when reaching the right interval end, the second sync point's
lines are not matched.
Do I miss something?
Is it possible to set sliding both waveforms in a way that they reflect
the stretching/ squeezing/ inserting etc.?
E.g. changing "Pre&Post mode" doesn't help.

For every new sync point you have to check "Strategy" - otherwise ... (?)

Saving settings
Only (!!) by "Ctrl-S".
Closing the app without Ctrl-S: No message, no saving!

First read-in the movies:
It takes some time (abt. 1min/movie) to have the audios ready;
in this time the app can be used - but the audios are not shown
(without any message).

Another idea to this: It could help to show a graph:
the cummulative dubs (+ and -) along the to-be-adjusted timeline (x-axis).
By that you see where a refinement of sync points is necessary,
because the interval is very large, or because the changing of dubs is big.

List of sync points: Cursor positioning by keyboard doesn't show in timeline.

No icon in taskbar.

Should allow one or both files to be audios only (i.e. no video)

Waveform should show larger (there is much room. Maybe video side-by-side; audios beyond it from left to right window border?)
If different loudness: missing remark about RD handles that.
RD seems to calculate and show it in the sliders (what are they for? Just showing or changing loudness? Not in the waveform.
Moreover the calculation of the peaks doesn't seem to use a loudness correction/ matching? Effect: worse identification.

In "Strategy" I don't see "Spline resampling".

Button "DelSync": Does nothing. What is it supposed to do?

Stretch Pre&Post modes (Keep 1/1, Largest Block, Video Framerate):
What do they stand for? I don't see an effect.

Button "Cleanup": What is it for?

The dubbed audio's format (and bitrate) should be selectable.

Double-click for deleting (a sync point) is anything but intuitive/ usual.
Better: e.g. Del key etc.

"Automatic mode" (for future releases)
As it is quite possible there are unexpected inserts/cuts time intervals
in the desync audio, you have to carefully check the adjusted waveform
against the orig one for them to occur.
For that it would be convenient if RD would offer an automatic mode
that proposes in-between sync points the user can accept or deny.

(Automatic) Identifying peaks (for future releases)
could be more specific
if only (1) kind of "isolated" ones are taken,
that (2) are not too long and
have (3) similiar lenght.
By that technical (background) sound will be taken
(in case of different languages). But might be
problematic, if new languages has been over-spoken
(i.e. background very low).

All in all: it seems worthwhile to repair/ improve.

[PatchWorKs]

Hi there,
very interesting I'll add to the HyMPS resources collection ASAP.

Meanwhile check out other interesting open source "synchers" projects under:

AUDIO section \ Treatments page \ Alignment/synch

My suggestion is to check both AudioAlign (and its Aurio library backend) and FFmatch.

Hope that inspires !