What tools do I need for an uncut project?

[Adam Cole]

Not sure if this is the right forum to ask. I didn't find a forum where this question appears to be suited for, so pls move/delete how you see fit @mods.

I have a litte project for myself going to make an uncut version of a popular anime in german dub. In germany that anime was produced in a cut form, with many scenes edited and ending, next episode preview and eyecatcher missing. I obtained the original, uncut, japanese episodes and want to dub them with my german dub, with japanese audio filling the parts that were cut.

So basically my goal is to mix german and japanese audio together to form a german uncut track with japanese playing on those parts that are missing from the original dub.

So far I used Audacity for this and while it does the job, it's horribly time consuming cuz everything has to be done by hand.

Are there any tools available that can help me get the job done faster? For example a tool that lets me compare cut and uncut video frame by frame at the same time including both audio tracks and cut and paste tracks from one place to another.

[Emulgator]

Vegas.

[johnmeyer]

+1 Emulgator. I've been using Vegas for twenty years, and what you describe would be trivially easy using this tool.

[lansing]

It's not going to make a difference than loading the two tracks into the audio program while using playing back the video to find the cut points, because at the end you can't rely on the frame timing for the audio cutting anyway.

[FranceBB]

As other people have already pointed out, it's actually way easier (and faster) to do these kind of things in a "normal" NLE rather than in a frameserver like Avisynth.
I personally use AVID Media Composer on a daily basis and if you wanna compare two clips you can use the "gang" function when you make your sequence.
Whenever I'm not using AVID Media Composer, my second choice is Davinci Resolve.
There are many non linear editors that can do this (AVID, Adobe Premiere, Final Cut, EDIUS, Sony Vegas etc), but if you wanna use something free (at least partially) you can use Davinci Resolve.

Cheers

[Frank62]

Does someone know if there is some function in some / or application that can assist this process (of synchronizing two clips with multiple cut points, missing scenes etc. to dub two audio tracks) to something (semi-)automatical? What strategies do you use for this purpose?

[johnmeyer]

If you want to synchronize multiple clips based on audio, the "go to" package is PluralEyes (https://www.redgiant.com/products/shooter-pluraleyes/). They have plugin versions that work inside various NLEs. I use it any time I am trying to sync multiple cameras. While I am not a vidoegrapher by trade, I have shot quite a few weddings, and shot many stage shows. I've done four-camera wedding shoots, operating all four cameras myself. During long uninterrupted stretches in the proceedings (like during a reading), I'll attend to the other cameras and often have to turn them off and back on again. I end up with lots of interrupted time codes. PluralEyes syncs everything up, even with these gaps. I just dump all the footage from the four cameras onto the Vegas timeline, push a button, and 5-10 minutes later all the footage from the ceremony, reception, and toasts are all lined up and ready to edit

BTW, getting an NLE that supports "multi-camera" editing makes it really easy to cut between these different cameras, as you watch the playback, much like a TV director would do with a live broadcast.

[lansing]

Does someone know if there is some function in some / or application that can assist this process (of synchronizing two clips with multiple cut points, missing scenes etc. to dub two audio tracks) to something (semi-)automatical? What strategies do you use for this purpose?

There is none, you'll have to do it manually.

[StainlessS]

How do the NLE cope with eg different FrameRate, OP says German and Japanese, so assuming differeing FPS,
Probably different size clips and perhaps non aligned frame edges.

EDIT:
If ALL characteristics of both clips are the same, then could try SirPlant,
its not fool proof, but sometimes works.
https://forum.doom9.org/showthread.php?p=1809957#post1809957

EDIT: Below not included in the linked post, only present in RT_Stats v2.0 Beta something, zip.


Function SirPlant(clip src,clip rep,
\ string srcARR,string srcNxtARR, string repARR,string repNxtARR,
\ Float "Th"=0.90,Float "Th2"=0.75,Float "LumaTol"=8.0,Int "MatchLen"=60,Int "SearchStep"=MatchLen/4,
\ Float "CW"=1.0/3.0,Int "BlkW"=64,Int "BlkH"=BlkW,Int "OLapX"=BlkW/2,int "OLapY"=BlkH/2,
\ Bool "Strict"=True,Int "InsMode"=0,String "Project"="Default",Bool "Debug"=True,Int "Verbosity"=1)

Req:-
RT_Stats v2.0 Beta 4, http://forum.doom9.org/showthread.php?p=1787898#post1787898 (c) StainlessS
Prune, http://forum.doom9.org/showthread.php?t=162446 (c) StainlessS
GScript, http://forum.doom9.org/showthread.php?t=147846 (c) Gavino
DebugView, https://technet.microsoft.com/en-us/sysinternals/debugview.aspx (c) SysInternals (M$).

Function to generate a Prune() Command file to substitute/supplant sequences in src source clip with similar sequences from rep replacement clip.
Both clips must have scenes similarly ordered (same size, Framerate, Colorspace etc). Can Also insert unique ranges in rep clip, into
the src source clip at approximately the correct position (or both replace and insert, with all unique ranges in the single result clip,
perhaps easier to hand edit into correct ordering).

NOTE, RT_QwikScanCreate() which is used to create the srcARR, srcNxtARR, and repARR, and repNxtARR Arrays, sets the Matrix (RGB to Luma-Y)
and X, Y, W, and H args used for RT_LumaCorrelation differencing. This function acquires those values from the ARR array ID's.
The X, Y, W, and H args provided to RT_QwikScanCreate() will be used to avoid eg Logos on one clip that do not exist on other (as for Crop).

For every rep replacement frame, it searches the entire src source clip (from current src clip position) for a frame that best matches (initially
via RT_QwikScan FM mode to locate BEST MATCH, and then via RT_LumaCorrelation Th arg to see if match is good enough), this is Sync1 frame, if
Sync1+1 ie Sync2 frames (also via Th) also match, then tries to match remainder of frames (both forwards and backwards using Th2) and the
resulting range length is compared to MatchLen. If sufficiently long, then that range will be replaced. That is basically it. The function creates
a Prune() Command file which is used to splice the sequences together. Note, Prune will Trim/Splice BOTH video and Audio.
(It also alternately does a reverse scan to find current position src frame in rep clip, in between above rep scan failures, doing about same as above).

Args:-
src, source clip.

rep, replacement/substitute clip (probably better quality clip of similar dimensions/colorspace and spatially in sync with src).

srcArr, RT_Stats Array filename, for array as created by RT_QwikScanCreate(), eg
RT_QwikScanCreate(src,"MySrc.ARR","","MySrcNxt.ARR",Matrix=Matrix,x=X,y=Y,w=W,h=H)
where
Matrix defaults (src.Width>1100||src.Height>600)?3:2) # 3=PC709, 2=PC601
X,Y,W,H are crop style coords (ignore eg logo).

srcNxtARR, RT_Stats Array filename, for array as created by RT_QwikScanCreate() as in above example.
Used for fast forward searching and locating of frames using RT_QwikScan().

repArr, RT_Stats Array filename, for array as created by RT_QwikScanCreate(), eg
RT_QwikScanCreate(rep,"MySrc.ARR","","MySrcNxt.ARR",Matrix=Matrix,x=X,y=Y,w=W,h=H)
where
Matrix defaults (src.Width>1100||src.Height>600)?3:2) # 3=PC709, 2=PC601
X,Y,W,H are crop style coords (ignore eg logo).

repNxtARR, RT_Stats Array filename, for array as created by RT_QwikScanCreate() as in above example.
Used for fast forward searching and locating of frames using RT_QwikScan().

Th, Default 0.90. LumaCorrelation threshold, greater or equal to this are considered similar frames. Th is used to find 'Sync' frames, and
Th should be greater than Th2. Range 0.0 < Th <= 1.0.
0.0 = not correlated, 1.0 = full correlation.

Th2, Default 0.75. LumaCorrelation threshold, greater or equal to this are considered similar frames, used for all remaining frames that are not
'Sync' frames. Range 0.0 < Th2 <= Th.

LumaTol, Default 8.0. Range 0.0->16.0. See RT_Stats RT_QwikScan docs. Suggest 16.0 if significant differences in brightness/contrast between clips,
lower is faster but may not find some frames if significant difference in brightness between src and rep clips.
Small increase in LumaTol results in significant speed decrease.
If you end up with duplicate ranges in result clip, then LumaTol is likely set too low, it cannot find the frames via RT_QwikScan fingerprint,
and so sees the frames as not matching.
When Verbosity=2, and Sync1 and Sync2 are shown high (In DebugView), but below Th, then Th is set too high to match frames.

MatchLen, Default = 60. Minimum number of frames (of source clip, when Strict=False) that must match, otherwise not replaced. Range 16 or more.

SearchStep, Default MatchLen/4 (1 -> MatchLen). When frame search fails, the search frame number is incremented by this, and search retried. When matching frame
is found, will scan both forwards and backwards, re-doing frames that were stepped over. Higher faster, but possibility of missing some
matches.

CW Default 1.0/3.0 (0.0 -> 1.0),
ChromaWeight, 0.0 = Luma differencing only, else weighting for pair of chroma channels.
If RGB, 0.0 converts to Luma-Y using Matrix arg as used in creating Scan Arrays. If 0.0 < CW, then instead it differences using
(AvergeRedPixelDifference + AvergeGreenPixelDifference + AvergeBluePixelDifference) / 3.0.
The CW ChromaWeight arg is used during the (new in RT_Stats v2.0 Beta 4) RT_QwikScan FM FrameMovement mode, See Blk args below.
CW==0.0 will be a little faster for YUV, and perhaps a little slower for RGB, suggest keep at default or close, maybe 0.2 or 0.1.

BlkW, Default 64 (8 <= BlkW, even only, Silently limited to frame dimensions).
Used in the new FM FrameMovement mode of RT_QwikScan.
RT_FrameMovement(), splits a frame into BlkW x BlkH blocks (repeated with overlaps of OLapX and OLapY), the result of RT_FrameMovement
is the greatest average pixel difference between similar positioned blocks in each frame, and so is more sensitive to localised movement
like lips moving, where the rest of the frame stays almost exactly the same, allows us to find the correct Sync frames whilst ignoring
most of the noise between clips.
Once we find a Sync Frame using RT_QwikScan FM mode (and LumaTol arg), we compare found frames with RT_LumaCorrelation Th arg, if greater
or equal, then Sync Frame + 1 also has to match greater or equal to Th. When the pair of Sync Frames match then we do a bi-directional
scan to establish the extent of matching frames, these additional frames have to be greater or equal to Th2 to be considered part of a
matching range. Lower BlkW will be a bit slower. See Also RT_QwikScan.

BlkH, Default BklW. (8 <= BlkH, even only, Silently limited to frame dimensions). See Above BlkW.

OLapX, Default BklW/2. (0 <= BlkW/2, Silently limited to frame dimensions). Block Overlap, See Above BlkW.

OLapY, Default BlkH/2. (0 <= BlkH/2, Silently limited to frame dimensions). Block Overlap, See Above BlkW.

Strict, Default True. If True, then compared ranges must match 1:1, if replacement rep clip has extra frames or fewer frames in places,
then matched ranges will break at those points.
If False, then will try to see if match will be re-established if allowing for fewer or more frames in the replacement clip.
NOTE, if set false, then audio must use audio from the 'Prune'ed clips to maintain audio sync, if the src and rep clips are NOT
of similar volume, then the effect will not be good (change in volume where ranges are replaced, the alternative is out of sync audio).
Suggest audio pre-normalizing of both clips, if Strict=False (Can just use original source audio if Strict=True).

InsMode, Default 0 (0->2).
0) Replace matching ranges.
1) Insert unique extra ranges in rep clip into src, at approximate correct postition (matching ranges not replaced).
2) Replace and Insert extras mode.
When inserting extra ranges into the source clip, we can never be always correct in the ordering (but we do try to predict correct order),
when we have a unique sequence in both clips, we do not know which should be inserted first, indeed one of the sequences may have to
be inserted midway in the other to be correct, we have no way of telling, but do try to match end of sequence frames to foretell
the correct ordering.

Project, Default "Default". Empty string "" will be converted to the default.
Project is the name prefix given to output command and log files. (Below output files using default project).
"Default_Command.txt" # Prune Command file to create result clip.
"Default_UnUsed.txt" # Prune Command file to create clip of 'unused' frames from rep replacement clip.
"Default_SirPlant.LOG" # The log file.

Debug, Default True. Send info to DebugView (Google).

Verbosity, Default 1. (0 -> 2) higher more info when Debug=true.


EDIT: Further to above:- https://forum.doom9.org/showthread.php?p=1787899#post1787899

[Frank62]

@johnmeyer
Thanks a lot for your explanation. I fear, what I tried to say, does not exactly fit this scenario. So let me be more precise:

-You have f. e. an US-HD-Master, very good picture, English sound.
-You have an old DigiBeta with old 4:3 cropped picture and German sound
-There are a lot of small differences, at some scenchanges a few frames more, at others a few frames less, some missing scenes in one, later in the other master.

The goal is to:
-First bring everything to the same (progressive) framerate (if necessary by also IVTC), that's not the problem.

-Then put everything in a timeline of a good NLE, let some plugin do some magic, and get a result where corresponding frames are aligned. (Or maybe there is also a solution for all this in AviSynth?)

-Then do the rest (fill some sound gaps etc.) by hand.

Can PluralEyes help also in this context? Then it could be what I search all the time.

@lansing
That's what we do since years and years, but there's always hope.

@StainlessS
Again the hardest problem to solve seems to ALIGN the two clips in size and frame edges. I tried this in multiple ways meanwhile - nothing did work really reliable up to now. Would be some Holy Grale to me. And would save many, many hours of manual work...

[StainlessS]

There is this, Auto Overlay:- https://forum.doom9.org/showthread.php?t=175247
But its not easy to use (I aint figured it out yet), perhaps of use to get croppings only for entire set (using small section from each [or even two single identical synced frames]),
but may lose some frame area (assumes cropping dont change scene to scene).

EDIT:
Auto Overlay, main intention to get Union of both clips, whereas you want probably Intersection ie common area of clips.
I think that it may be able to get Intersection instead.

EDIT: SirPlant(), worst case scenario settings:- https://forum.doom9.org/showthread.php?p=1809957#post1809957

[lansing]

Again the hardest problem to solve seems to ALIGN the two clips in size and frame edges. I tried this in multiple ways meanwhile - nothing did work really reliable up to now. Would be some Holy Grale to me. And would save many, many hours of manual work...

You should not be relying on the frames to align the two audios because the audios itself may not even be in sync for the same frame.

You need to do this all in the wave form. The speed of the process depends on the efficiency of your workflow and the number of cuts there are in the episode. If there're only a few, you should be able to finish it in less than 10 minutes.

[Frank62]

@StainlessS
AutoOverlay does not work in our scenario. It is good in overlaying clips that are nearly perfect similar - apart from the two different parametres - but I tried everything to make it useful for our problem - no way. A newly restored HD master is much to different from a sometimes very old scan of an old cinema copy used for tv airing decades ago.
The problem is how to find the similarity - and even better the alignment parametres - for these kind of different clips.

@lansing
We do and did dubbing audio like this for about 10 years for hundreds of series and films, by aligning the clips by hand and taking the picture as reference. Due to the fact that the (in our case) German dubs had been made to the (now low-quality-)picture by the studios once, the German audio IS in sync to this picture, that's not the problem.
The time you need for an average episode is between 10-20 minutes to hours (with a lot of practice).
By the way: The worst candidate I ever dubbed was an old classic Italian film that had been restored brilliantly, but the German dub had been made on a copy that had been "restored" before by cutting two to seven frames on EACH scene change. This took quite some time.
To take the sound wave as reference would be even more complicated - not much, but I decided long ago to always take the two pictures for alignment.

[lansing]

By the way: The worst candidate I ever dubbed was an old classic Italian film that had been restored brilliantly, but the German dub had been made on a copy that had been "restored" before by cutting two to seven frames on EACH scene change. This took quite some time.
To take the sound wave as reference would be even more complicated - not much, but I decided long ago to always take the two pictures for alignment.

What you are doing is trying to trim the source video according to the cuts in the dubbed version, that's not what the op is asking.

[StainlessS]

What you are doing is trying to trim the source video according to the cuts in the dubbed version, that's not what the op is asking.

If so, then think Gavino has a script function somewhere to do just that, but would require same framesize etc.

Called something like "CutClipsSimilar". (Not thread name, is isolated function in some thread).

EDIT: Was maybe thinking of this thread (CutFrames, EDIT: Is not as required, ie wrong thread.):- https://forum.doom9.org/showthread.php?t=135423

[StainlessS]

Found it, "FindCuts" @ VideoHelp:- https://forum.videohelp.com/threads/310044-video-censor-detection#post1912566

[johnmeyer]

My PluralEyes idea still might work if most of the audio track is foley and music, and only occasional dialog. However, the more dialog they have, with one being English and the other German, the less well PluralEyes would work. At some point it will give up (there are settings to have it ignore some level of mismatch).

If this were my project, after I adjusted each video to a common frame rate, I'd first find a good optical scene detection program. StainlessS has written at least one, and there are many others. Use that to create (most of) the cut points. This saves you a LOT of time scrolling back and forth to find an exact point at which to re-sync the video. I assume that most of your cuts between different sources will be at these hard scene cuts.

Then, in my NLE (Vegas), I would put the scene-cut capture that I think will be used the most in the top (preferred) position. I'd use the shortcut key to go to the next scene on that top line, and then look at the scene in the capture on the track below below (with Multicamera enabled, you can see them both at the same time in the preview window). If the edit below didn't match (i.e., come from the same moment in time), I'd move it forward (or move the top scene forward) to the next scene and then ripple just that track. It will be a little tedious, but once you get a rhythm, it might not take too long. The audio from each video will move with these edits. You'll then have to decide how you want to mix those.

You'll also have to do a lot of color matching and other adjustments so that the cuts between sources isn't too jarring.

Finally, I wouldn't get hung up on trying to be frame accurate. You are going to lose or duplicate a few frames. Also, the audio probably will be a little off after you do a rough cut. What I'd do at that point is not do anything until there is a close up of dialog, where lip sync is important. Do the usual tricks to get lip sync at that point and then, on the non-speaking sections before and after that sync point, overlap the audio slightly by doing a minor time stretch. You'll never hear that transition, and the minor speed up or slow down (usually 0.2% or less) will never be noticed. Vegas has something called Elastique which is an audio stretching technology that is amazingly good: you can change audio by several percent without getting any audible artifacts.

[Frank62]

What you are doing is trying to trim the source video according to the cuts in the dubbed version
Of course, not. We would not cut a new restored version to an old shorter dub. Are my formulations that misleading? :confused: Sorry, if so.

@StainlessS
Thank you so much as always. This function is useful, but only if you have exactly the same source - just with different cuts. Gavino just compares lumadifferences with a certain threshold. But with completely different sources with completely different color grading, levelling, and last but not least - alignment, you won't find a working threshold.
That's why I look first for a way to find a SIMILARITY between such two clips, which does NOT depend on all this. One could come nearer to a solution if you could align the two clips perfectly. As a first step.

@johnmeyer
Thank you for all these hints! What you describe is about 90% as also we do these jobs. :)
We are very used to it and use as much tricks to save as much hand-working as possible, but it is still very time-consuming anyway, if you do this for two or three series and some feature films every month by hand.

About scene detection: As you pointed out somewhere else (I hope I remember correctly), there is no 100% correct way to find scene cuts, any approach can be fooled by something. I am convinced this is right (at least, as long we won't have an approach based on A.I. with big databases as with automatic colorization).
For auto-aligning based on automatic scene detection for the two clips, you would need 100%, because if only some scenes are detected different, you get more problems to re-align the mess than to do everything by hand from the beginning.
I once wrote a scene detection script with some new approach (I think), that was better than anything else I tried - especially for an old B&W film with a lot of dark scenes and short lights (cars at night). But I didn't get 100% (of course), you always find some scene changes in one clip only, no matter what thresholds you use.

[lansing]

Of course, not. We would not cut a new restored version to an old shorter dub. Are my formulations that misleading? :confused: Sorry, if so.


German dub had been made on a copy that had been "restored" before by cutting two to seven frames on EACH scene change

If this is true, then in order to correct the shorter dubbed audio to match the new original, you'll have to somehow fill in all those missing gap in the dub for each scene as well, meaning hundreds of audio edits if not into the thousand.

This doesn't sound right to me, that why I'm guessing you're meaning to do it the other way around.

[Frank62]

Yes, and this was a lot of work...
But we mixed filling up, overlapping, and time-stretching (as johnmeyer also proposed), so it was doable and in the result you do not notice gaps.
In this special case it was much more work to fill the gaps, but for most projects 70-80% of the work is timeline-alignment. So my goal would be:
-align (left - up - size)
-test similarity somehow, in spite of different levelling
-cut autmatically like in gavinos FindCuts by using this similarity

@johnmeyer
Does Vegas (that I never used before, but sounds interesting) have something like script-import? XML or something similar non-proprietary?

[StainlessS]

test similarity somehow, in spite of different levelling
Immune-ish to differing levels (EDIT: At least rough alignment, better results)

RT_LumaCorrelation(clip c,clip c2,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,int "n2"=current_frame,
int "delta2"=0,int "x2"=x,int "y2"=y,bool "interlaced"=false,int "Matrix"=(Width>1100||Height>600||clip2.Width>1100||clip2.Height>600?3:2))
Returns FLOAT value luma correlation (-1.0 -> 1.0) between clip c frame (n+delta) area x,y,w,h, and clip c2 frame (n2+delta2) area x2,y2,w,h.
Note, 'x2' and 'y2' default to 'x' and 'y' respectively.
v2.0, c and c2 need not be same dimensions but BEWARE, x2 and y2 default to x and y also w and h default to 0
which is converted to c.width-x and c.height-y, may be best to provide x2, y2, w and h where c2 not same dimensions as c.

Pearson's Sample Correlation Coefficient.
http://en.wikipedia.org/wiki/Correlation_and_dependence
http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
Uses equivalent routine to the JMac698's JCorr plugin here:
http://forum.doom9.org/showthread.php?t=165386
and here:
http://forum.doom9.org/showthread.php?p=1495098#post1495098

[Frank62]

Wow! This looks interesting! :thanks:

[StainlessS]

By the way, SirPlant() uses RT_LumaCorrelation() a lot (but not exclusively).
Has to find an initial frame via RT_FrameDifference (luma + optional weighted chroma difference),
from initial matched frame, scans both forwards and backwards using Luma Correlation to establish extent of matching range.
During forwards/backwards scan, (when STRICT=False) allows for single missing or extra frame so long as then next along
frame matches ok.

[Frank62]

After re-thinking I found that it will be a bit more difficult: The alignment (up, left, width, height) of two very different clips often changes every few scenes. This ruins my 3-points-approach (in spe).
-I would align automatically - ok.
-The new "FindCuts" then could find the first gap - ok, but maybe not - maybe it would find only a new wrong alignment.

So I would have to nest alignment and matching. I fear I can not do this in avisynth, will see if it's possible to log somehow and write something outside - and that maybe also can export some script format that can be used in Premiere (or Vegas?).
But this correlation-thing is an important step on the way, could work!

[StainlessS]

The alignment (up, left, width, height) of two very different clips often changes every few scenes.
I'm real glad I dont got that there clip.

[Frank62]

:D
This is quite common...

[johnmeyer]

I'm real glad I dont got that there clip.I ran into this with a very similar project where we wanted to add hard-coded English subtitles from a very poor VHS version of a film to a high-quality DVD version in the original French language. Each transfer was scanned with something that didn't go to the edges of the film, and the portions of the film that were missed was different for each scan. As a result, I couldn't line them up perfectly, and the alignment changed from scene to scene. Also, there were one or two short scenes missing from one print.

I did what I described earlier, and once I had them aligned, used a mask to allow the subtitles from the lousy print to burn into the good print. The subtitles were yellow so I was able to construct a mask that only transferred the pixels in that color and brightness range, and didn't transfer much else. This wasn't perfect, so I used a "garbage mask" to block out everything outside the lower third area where the subtitles normally appear. I briefly tried to OCR them to make real DVD subtitles, but the tools I found online didn't work well enough for that. The bad VHS print also was very dark, and the colors were different from the good DVD print, so I had to do color and gamma correction before I merged in the subtitles.

It worked, and I ended up with a usable film, but I am not tempted to do it again. It was just too much tedious, repetitive work.

So, you may not be able to automate this much more than what you are doing, and what has been described.

[Frank62]

Yes, I fear so, too - I tried to automate this for several years with no success. But hope dies last.

By the way: I remember something similar to your subtitle replacement. I had a new HD-scan and an old but not-too-bad SD scan. The problem was that the blue layer of the HD scan had been destroyed - looked like clouds sometimes with, sometimes without blue. No way to repair, tried everything. So finally I decided to align both clips exactly and use the color of the SD clip. Of course, every few scenes x-y-alignment changed.
In the end I did this primarily by interleaving the two (time-aligned) clips, then used Deshaker. Was not too bad in the end. Maybe this trick also can help someone somewhen. :)

[ChaosKing]

There's also a https://github.com/introspected/AutoOverlay plugin for avisynth.

p.s. click on the "> Portrait image + landscape image -> script output" text

[Frank62]

Yes, as I said: I tested this, but in the context it is useless. You have to align 95% before, otherwise it fails. And the sources have to be very similar. Then it seems to do a very nice job, but only then.