AutoOverlay: auto-alignment and color matching

[Frank62]

I am not sure. SSIM is mostly used for checking quality differences of one and the same picture. But when I find the time (can last a while...) I will test it, thanks for the idea.

[Goggen240]

FFMatch does 100% exactly what you want!
http://fsinapsi.altervista.org/code/ffmatch/index.html

It's kinda tricky to use, since the user manual is only available in Italian.
EDIT: And I was going to write it hadn't been updated since 2011, but it was updated last month! Happy days! But that means my guide may be out-of-date. Also, no screenshots, because I have to be at work in an hour...

To use it, you need both files encoded in XVID (some varieties of h.264 and x.264 *may* work), and both soundtracks as separate .MP3 files.
Top left, select movies 1 and 2 (doesn't matter which is which)
Top middle: Turn on supervisor mode (pencil-and-paper button)
Top middle: start (gear button)
This will run for a real long while, first it analyzes both movies, and then it starts matching them. It's cool to watch once it gets to that point.
Then, once it "finishes", you'll have matched maybe 80% of the frames to each other; to actually finish, you go in manually...
Bottom right you have video manuevering controls; for the sections that don't match, you line up two matching frames for both clips, and press the green arrow-down button.
Top right text is info of matching and unmatching ranges; green matches in frame count and content, red ranges do not match.
The goal is to make all the red ranges green, which means weeding out the missing frames:
A "red" range might say
[040684-040856] -/-> ??? {173-172}
Which means that movie 1 frames 40684->40856 is 173 frames, but the same region in movie 2 is 173 frames, and doesn't fit.
How to match manually:
Bottom right:
Left number is frame movie 1, right number is frame movie 2, and the frame-forward/back apply to both movies.
The drop-down menu is key; it lets you pick matching frames for a frame range: [Number is first frame in a matching range, Number] is last frame, and (Number) is first frame *after* it loses matching.
In this case, I select (40684), the first unmatched movie 1 frame in that region, and it automatically picks movie 2 frame 21755, the first unmatched frame in movie 2's region.
I then arrow right a little (both movies) until I find a frame that is easy to match; there's a big-ol' hair on movie 1 frame 40694, and it matches one frame "early" to movie 2 frame 21764.
(Between movie 1 frames 40684 and 40694, there's one frame missing compared to movie 2.)
I press green arrow down button (bottom right), linking 40694 to 21764, and top right text now reads:
[040684-040693] -/-> ??? {10-9}
[040694-040694] ---> [021764-021764] {000001} (diff=-18930)
[040695-040856] -/-> ??? {162-162}
First range is misaligned, second range is 1 matched frame, third range is aligned, but has not been matched yet, which you do by pressing the start gear button again. But first:
To get rid of the unmatched region entirely, I arrow left until I am at 40685-21755, and press the green arrow button. The range now reads:
[040685-040685] ---> [021755-021755] {000001} (diff=-18930)
[040686-040693] -/-> ??? {8-8}
[040694-040694] ---> [021764-021764] {000001} (diff=-18930)
[040695-040856] -/-> ??? {162-162}
The one extra frame is now "gone", but listed top right here:
1st movie frame(s) not found in the 2nd one:
[040684-040684] ---> [021754] {1}
If you press the gear button, it will "green" out the range, tidying it up. Keep doing this until the whole thing is green.

And now for the important part for you two:
Top middle there's a headphone button, "audio tracks".
If you load audio 1 and audio 2, MP3 tracks for movies 1 and 2, you can output either audio 2 synced to movie 1, or vice versa.
That gives you a single MP3 file where it uses audio 2 for movie 1, and audio 1 for parts missing from audio 2.

So, in short:
Convert both animes to XVID as .avi
Make both soundtracks into MP3
Load both .avis (top left)
Run automatic alignment (gear button)
Finish manual alignment (bottom right, green arrow button and such)
Select both .mp3s (headphone button)
And finally "save track as..." and you're done!


P.S.: I've done this for several full-length movies, such as 1966's "Is Paris Burning?" so I can get rid of dubbing: Instead of everyone being dubbed to English, or French, or German, now everyone speaks their original languages, after I combined the soundtracks.
I have a project going on with the Soviet "Blockade" movies (1973-1977), where the Russian dub is pan-and-scan 4:3 and the German dub is full 2,2:1 but lousy quality, so I'll need to learn AutoOverlay, and I'm hoping to use the FFMatch "top text" for that, since I'll align them anyway for the soundtrack.
So if FFMatch turns out useful, good thing I came here checking up on AutoOverlay! And sorry for going off topic!

[Frank62]

This looks really interesting! The question will be, how exact matches are, when the two sources are really different in resolution, position, etc. I will check this, thank you very much!

[poisondeathray]

If somebody out there has a perfect solution for this particular problem (full automization of soundtrack alignment of two slightly different sources)...? I would even pay a nice amount of money for this.

Auto sync clips by similar audio, slightly different cuts or cameras, already exists with software/plugins like plural eyes.

Adam wants sync clips by video similarity (he has different audio, different language).

Maybe temporal alignment iterations, and avisynth optimizer (zopti) using temporal iterations
https://forum.doom9.org/showthread.php?t=175723

[Frank62]

Auto sync clips by similar audio, slightly different cuts or cameras, already exists with software/plugins like plural eyes.


Yes, I know. For mixing shots from different angles, but with very similar audio.


Adam wants sync clips by video similarity (he has different audio, different language).

Yes, that's my purpose, too. And the tool, Goggen240 recommended is very promising. I will test this with different scenarios. If the automatic matching algorithm was doing well in different situations it would be a big progress in work and time saving for me. Maybe I can use the protocol to do the final sound mixing myself, I'll see.


Maybe temporal alignment iterations, and avisynth optimizer (zopti) using temporal iterations
https://forum.doom9.org/showthread.php?t=175723
Will not be sufficient, I fear. I tried a lot in the last few years. The way that is used by the above tool, to split in scenes before even trying to find 100% matches, seems mandatory. Very difficult to automate this process in pure avisynth.

And I still have no really good idea of how to compare two sources that may be different in
position (even changing)
width/height (even changing)
sharpness

AND to get results that are nearly 100% good matches. A really tough problem, I fear.

[Goggen240]

This looks really interesting! The question will be, how exact matches are, when the two sources are really different in resolution, position, etc. I will check this, thank you very much!


And I still have no really good idea of how to compare two sources that may be different in
position (even changing)
width/height (even changing)
sharpness

I've been doing comparative tests on a Soviet movie that is available in 4:3 Russian pan-and-scan (cropped sides) and 2.35:1 German (cropped top/bottom). Both have been cropped from something probably 2.20:1; the full height of the Russian version (1.33:1.00) and the width of the German version (closer to 2.20:0.94). Also, the position of the 4:3 version jumps around side-to-side inside the 2.35:1 version, as the pan part of pan-and-scan implies.

I checked old and new versions against this material:
The 2011 FFMatch version worked fine, but slowly, and the 2020 version would only get going once I disabled Preferences->Checks->Scaling, which was the only setting that changed between the two versions.
But yes, it will match a tall image moving around inside a wide image, with wildly different framing, colour, contrast, and edit.
First run with 2020 version it got about 95k out of 157k frames matched, where the 2011 version got 138k, so some more settings have to change, but it works.

In the past I've matched Blu-rays to VHS rips, so it is reeeal flexible if you're fine with some manual labour...

[Frank62]

But yes, it will match a tall image moving around inside a wide image, with wildly different framing, colour, contrast, and edit.

Wow! Fascinating! Do you have any idea how comparing is done?

And: From where did you get the older version? I would like to compare with the newer, too.

[Goggen240]

And: From where did you get the older version? I would like to compare with the newer, too.

I got it from that site, but it seems gone now. I couldn't seem to attach it to this post, but I can send it to you by some reasonable conveyance, it's 11MB.

Wow! Fascinating! Do you have any idea how comparing is done?

It compares frames using SIFT. As far as I understand it, that's an algorithm that finds patterns (say, purple corner on blue background) and assigns points to them, and then it looks for similar points in both files, and if enough points line up it assigns it as a match. It deals with rotation, scale and warp really well; it treats the point patterns as "constellations".
The same technique is used in photogrammetry, a 3D-modelling method where you take dozens (or tens of thousands) of pictures of an object from different angles, and it SIFT matches them to map out the position of cameras, and work out the parallax between them to find depth, and map the point depths to 3D space. I've made 3D models of a dead rat and a tractor, among others. But those are really the high points.

[StainlessS]

I couldn't seem to attach it to this post, but I can send it to you by some reasonable conveyance, it's 11MB.
You can post a file without account on SendSpace.com, file deleted 30 days after most recent download.
[If you have an account, it will email you a warning 7 days (I think) before file deleted]

Max file size for D9 attachment is about 200KB.

[Frank62]

It compares frames using SIFT.

Learned something. Really new to me, and very, very promising!

Does any filter for avisynth exist that uses this technique?

[Goggen240]

Here you go, the 2011 version of FFMatch:

https://www.sendspace.com/file/ghzfj7

Learned something. Really new to me, and very, very promising!

Does any filter for avisynth exist that uses this technique?

I'm not sure, but phones use it for panorama stitching, and possibly camera deshake methods. There are some camera deshake filters that use very similar methods; I found somebody's FFMPEG project that uses BRIEF, whatever that is. But it seems to be a similar same point-finding constellation-matching technique.
"Deshaker" for VirtualDub does not use SIFT though, that matches blocks for motion.

[Frank62]

Great, thank you.

Besides I found a .net library ( http://www.nowozin.net/sebastian/tu-berlin-2006/libsift/ , seems a bit older) and a C# project ( https://www.codeproject.com/Articles/95453/Automatic-Image-Stitching-with-Accord-NET ).
If I only had the time, but I will try everything as soon as I will find some, ffmatch first, of course. Very curious.

[AdamCarter]

Goggen!!!!

This looks amazing.

What do we need to do with the github files
I have no programming skillz

Thanks

[StainlessS]

What do we need to do with the github files
I have no programming skillz


I got
503 Service Temporarily Unavailable
Think I got that the day it was posted too.
[It might be a "region restricted" thing, ie not available in your/my country].

[AdamCarter]

Stainless i was able to download the files

Try using a vpn maybe to connect.

Would love to see this as an aviscript that could work without converting to xvid.

[StainlessS]

without converting to xvid.
OK, link is working now, was as it said "temp unavailable".

I remember ffmatch from some time back, it requires XVid to work its magic, it uses compression data
(DCT, Discrete Cosine Transform stuff I think) and wil not work with decompressed video (as in avisynth).
Its also the reason that it is so fast.

[AdamCarter]

Well at least it tells me which frames are good or mismatched, for avisynth i can use the trim function on the real video. It’s a step in the right direction.

Wonder if something similar could be programmed for avisynth from scratch.

It would go a long way for restoration.

[introspected]

v0.4.0 with warp transformations released: https://github.com/introspected/AutoOverlay/releases/tag/0.4.0

[wonkey_monkey]

with warp filter by wonkey_monkey

Yay! I'm useful!

[introspected]

Yay! I'm useful!
Thanks again for your support

[wonkey_monkey]

I notice it says "Only 8 bit clips support" on that line, but Warp should support all clips. Is it a limitation of AutoOverlay?

[introspected]

I get this message for the YUV420P16 color space:
warp/quad: High bit depths not yet supported
sample: ColorBars(pixel_type="YUV420P16").Warp(0,0,0,0, 2,2,2,2, 4,4,4,4)
Note that I use v0.1b

[StainlessS]

"Warp (includes quad)" here shows v0.1:- http://horman.net/avisynth/
Granted its a little non obvious what any of Wonkey's stuff is, no version info in zip name (could all be exact same version),
and no version resource on dll.
Its no real surprise that he sometimes posts the wrong version, sad but there you go.

[introspected]

"Warp (includes quad)" here shows v0.1:- http://horman.net/avisynth/
Granted its a little non obvious what any of Wonkey's stuff is, no version info in zip name (could all be exact same version),
and no version resource on dll.
Its no real surprise that he sometimes posts the wrong version, sad but there you go.
v0.1b is the latest without this (https://forum.doom9.org/showthread.php?p=1923288#post1923288) problem. So that's why it's included to the AutoOverlay package.

[StainlessS]

OK, so Wonkey does his beta version after the stable version, novel choice, to each his own. :)

[wonkey_monkey]

It's true, my ways are inscrutable and mysterious.

[StainlessS]

Anybody any success getting this to do anything at all [v0.4.0] ?
I've given up trying to get it loading plugs directly, [no idea what to do with MathNet.Numerics.dll and simd.dll]
and just dumped the lot in plugins [with ImageSeq.dll for jpg loading].
Only loading image, and display, crash on close Vdub2, Access Violation [x86 OR x64]. [ie no use of AutoOverlay, except during plugin autoload from plugins]

Also, for x64, using AvsMeter64 -avsinfo, get this [32 bit dll's ???]


C:\Users\root>avsmeter64 -avsinfo

AVSMeter 3.0.8.0 (x64), (c) Groucho2004, 2012-2021

VersionString: AviSynth+ 3.7.0 (r3382, 3.7, x86_64)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 21:36:39 (UTC)
PluginDir2_5 (HKLM, x64): C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plug
ins
PluginDir+ (HKLM, x64): C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plug
ins


[C++ 2.6 Plugins (64 Bit)] [Version, Time stamp]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\AvsFilterNet.dll [1.0.4
.0, 2020-06-28]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\ImageSeq.dll [n/a, 2021
-01-11]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\warp.dll [n/a, 2020-09-
13]

[Uncategorized DLLs (32 Bit)] [Version, Time stamp]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\AutoOverlay_netautoload.
dll [0.4.0.0, 2021-01-17]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\MathNet.Numerics.dll [4
.12.0.0, 2020-08-02]

[Uncategorized DLLs (64 Bit)] [Version, Time stamp]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\AutoOverlayNative.dll [
n/a, 2021-01-07]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\Simd.dll [0.3.0.0, 2020
-08-28]

C:\Users\root>

Then AvsMeter64 crashes.

Avsmeter -avsinfo
32 bit, just hangs [seemingly forever].

[StvG]

I have no issues. x64 version used.
For example:OM=Colorbars(pixel_type="yv12").ColorRangeMask(20,60)
WS=Rect(50,60,70,80)
OverlayEngine(OM.ConvertToY8(), WS.ConvertToY8(), configs=OverlayConfig(subpixel=2))
OverlayRender(OM.ConvertToYV24(), WS.ConvertToYV24(), debug=true)
https://images2.imgbox.com/95/72/EGv7pQNf_o.png

[StainlessS]

Thanks for responding StvG,
maybe my machine [W7 Pro] is misconfigured or something.
I'm not seeing any DotNet 4.8 installed, although I do see 3 updates for it installed.
I'm gonna forget this again for a while.

EDIT: Guess I'm gonna re-install W7 at some point soon, already got the ISO.

[introspected]

Thanks for responding StvG,
maybe my machine [W7 Pro] is misconfigured or something.
I'm not seeing any DotNet 4.8 installed, although I do see 3 updates for it installed.
I'm gonna forget this again for a while.

EDIT: Guess I'm gonna re-install W7 at some point soon, already got the ISO.
Copy x86/x64 DLL's to AviSynth plugins folder.
DLL's may be blocked after downloading. Open file properties to unblock them.
All dll's from x64 folder should be unpacked to the x64 plugins folder.
I don't know how AvsMeter64 works but I guess it marks AutoOverlay dll's as uncategorized because they are not avisynth plugins themself. Only AvsFilterNet is the native plugin.

[StainlessS]

Thanks for your answer Introspected,
Most people object to non avs plugins/scripts in the bittage specific Plugins directories, so eg
32 bit system or 64 bit system with x64 avs, standard OS dll's go in system32.
64 bit system and x86 avs, standard OS dll's go in SysWOW64.

It seems strange that there should be 32 bit AutoOverlay supplied dll's in x64 plugins directory, and is rightly flagged as probable problem by AvsMeter.

[Uncategorized DLLs (32 Bit)] [Version, Time stamp]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\AutoOverlay_netautoload.
dll [0.4.0.0, 2021-01-17]
C:\VideoTools\AvisynthRepository\AVSPLUS370_x64\plugins\MathNet.Numerics.dll [4
.12.0.0, 2020-08-02]


Can you please outline how to load your plugins that are using manual load and NOT using AVS AutoLoad plugins directory.
(LoadDLL (http://avisynth.nl/index.php/LoadDLL) can be used to load SYSTEM dll's not in system32 or SysWOW64)
Many of us dont like dumping everything in plugins directory, and would prefer to manual load where necessary.

[introspected]

.NET dll's are cross-platform so they are the same both for x86 and x64 :)
Explicit plugin loading is described in the readme file:
LoadPlugin("%plugin folder%\AvsFilterNet.dll")
LoadNetPlugin("%plugin folder %\AutoOverlay_netautoload.dll")

v0.4.1 released: https://github.com/introspected/AutoOverlay/releases/tag/0.4.1

[Tlen]

Hi,
i'm trying to use the plugin but no luck.
Virtual dub just crashes on opening.

I've tried loading the dll from plugin folder

LoadPlugin("D:\Programs\AviSynthPlugins\AutoOverlay\x64\AvsFilterNet.dll")
LoadNetPlugin("D:\Programs\AviSynthPlugins\AutoOverlay\x64\AutoOverlay_netautoload.dll")

or copying the x64 files in C:\Program Files (x86)\AviSynth+\plugins64+

Same situation.

I've not found in the readme the section on how to load manually all the needed .dll

Could you help?

Thanks

[torturesauce]

Hello, all! I am bad at AviSynth and would like your help. I can use AutoOverlay without any problems, I want to use ColorAdjust by itself in order to color match two video frames. On AutoOverlay's Github, these are the instructions:

ColorAdjust(clip sample, clip reference, clip sampleMask, clip referenceMask, float intensity,
int adjacentFramesCount, float adjacentFramesDiff,
bool limitedRange, string channels, float dither, float exclude, string interpolation,
bool extrapolation, bool dynamicNoise, bool simd, bool debug)

So, this is my script. I want to make mpv-shot0002.png look like mpv-shot0001.png and I used all the defaults mentioned here (https://github.com/introspected/AutoOverlay#coloradjust):

video = ImageSource("c:\mpv-shot0002.png").ConvertToYV12()
video = ImageSource("c:\mpv-shot0001.png").ConvertToYV12()

ColorAdjust(clip="c:\mpv-shot0002.png", sample="c:\mpv-shot0002.png", reference="c:\mpv-shot0001.png", samplemask=0, referencemask=0, intensity=1,
adjacentframescount=0, adjacentframesdiff=1,
limitedrange=true, channels="yuv", dither=0.95, exclude=0, interpolation="linear",
extrapolation=false, dynamicnoise=true, simd=true, debug=false)
)

This script doesn't work. I'm sure I've made tons of mistakes, but there are some mind-boggling things about this that prevent me from fixing it:

What is the correct way to leave both sampleMask and referenceMask empty? Is zero the correct value?
Apparently the latest version of ColorAdjust has a new, undocumented parameter called "CacheId". What should I add there?
Do I need to call MathNet.Numerics.dll, or is it automatically loaded?


I use AutoOverlay 0.4.2, AvsPmod x64, AviSynth+ x64 (both latest versions) on Win 10 x64.

[introspected]

v0.4.3 released: https://github.com/introspected/AutoOverlay/releases/tag/0.4.3

[StainlessS]

Thanks very Much :)

[introspected]

v0.5 released: https://github.com/introspected/AutoOverlay/releases/tag/0.5.0
Many user functions added. Most interesting one is for color level fixing at frame borders.

[StainlessS]

Thanks again :)

[introspected]

v0.5.1 released: https://github.com/introspected/AutoOverlay/releases/tag/0.5.1

[Dogway]

I'm having a hard time trying to make this work.
I want to align two manga pages in different languages, so I create a mask for only the four corners (so it doesn't pick the text bubbles).
a=ImageSource("S:\a.jpg",start=0,end=0).ConvertToY8("PC709")
b=ImageSource("S:\b.jpg",start=0,end=0).ConvertToY8("PC709")

# masking everything except four corners
msk = a.BoxMask(0,250,0,250).PadResize(a.width()/2,a.height()/2,biasw=b.width()/4,biash=b.height()/4)
msk = stackhorizontal(msk,msk.FlipHorizontal())
msk = stackvertical (msk,msk.FlipVertical())

cfg = OverlayConfig(branches=3,acceptableDiff=20,correction=1,angle1=-5,angle2=5,debug=false)
OverlayEngine(a,b, configs=cfg,backwardFrames=0,forwardFrames=0,sourceMask=msk,overlayMask=msk,editor=false)
OverlayRender(a,b)

It doesn't seem to do anyhing. Is there also a way to enable autoassisted distort or warping?

[Frank62]

For me it works, but I have another big problem with the result:
I have two clips, one in HD 1920*1080 in 16:9 from a BluRay and the other in SD 720*480 letterboxed 1.66:1. The goal was to extend the cropped 1.77:1 HD at top and bottom with the 1.66:1 source. This works quite well, when I first

-crop the SD clip so that there are no black borders left
-resize the SD clip so that it nearly fits the HD clip

With the correct parameters, especially minoverlayarea and minsourcearea, the result seems quite good at first sight.
But in several scenes the whole picture shakes a lot, which it doesn't at the original sources (in none of the two). It seems that resizing and fitting to each other does not work 100% but only 99%, so that there are small errors all the time that cause random misaligning and a shaking over the whole result.

This would be acceptable, if only the overlayed clip would shake, but the problem ist that BOTH clips shake! No matter what I declare as overlayed and source clip, the clips are touched BOTH by the filter.

So the question: Does anyone know if there are settings that let the source clip 100% alone, without any changing, by only adapting the overlay clip, in my example the SD-clip? So that only the extended area shakes a little bit, but the rest keeps stable?

It seems that @introspected didn't visit us a longer time, I hope he reads this.

:thanks:

[Frank62]

If anyone interested:
1. It seems that the shaking that is not in the source and thus generated by AutoOverlay is mostly horizontal, but not 100% sure.

2. The parameter "backwardframes" seems to be sensitive to it, so when you turn it to 4 (instead of 0) most(?) of the instability is gone. Seems to refer temporally somehow and so calm down the wobbling. Strange: higher values like 10 seem to be rather counterproductive.

[PoeBear]

It might be worth also checking on the sister thread at FanRes (https://forum.fanres.com/thread-1974.html). There's been more discussion there in the past, and there could be some useful ears that aren't over here

[Frank62]

Thanks for the hint, but for me not necessary any more. With the above parameter plus a bit of stabilisation afterwards my result looks very good meanwhile.

[BilboFett]

I've spent probably about 3-4 hours now testing, retesting, trying every possible setting.. I cannot get 2 identical scenes (same rez, similar color grading, same fps, etc.) to overlay correctly. Is it because one is PAL widescreen and the other is PAL fullscreen? I want to overlay the higher quality widescreen footage on top of the lower quality fullscreen, so it becomes like an "open matte". Would it help if I artificially forced one or the other's aspect ratio? I also cropped the widescreen one.. didn't make any difference. Res of both are 720x576 (widescreen one gets down to 544 if I crop). Here's my script, which is prob full of mistakes:
----
widescreen=directshowsource("V:\PAL SCENE 16x9-from premiere.mov")
openmatte=directshowsource("V:\PAL SCENE Full Frame-from premiere.mov")
OverlayConfig(aspectRatio1=1.78, subpixel=3)
OverlayEngine(openmatte, widescreen, editor=true)
#ComplexityOverlay(widescreen, openmatte)
OverlayRender(openmatte, widescreen, colorAdjust=1, mode=3, gradient=20)
#OverlayCompare(widescreen, openmatte)

[Frank62]

Did you try to resize first both to (nearly) quadratic pixels?
4:3 source to 768*576.
1.77 source to 1024*576, then crop.

[BilboFett]

Did you try to resize first both to (nearly) quadratic pixels?
4:3 source to 768*576.
1.77 source to 1024*576, then crop.

I will try that, thank you

[BilboFett]

Did you try to resize first both to (nearly) quadratic pixels?
4:3 source to 768*576.
1.77 source to 1024*576, then crop.

Thank you, this was a good tip!
That indeed lined them up the best they've ever been, but, for some reason, the two images warp and shift around (so they match?) and its just really artificial and fake looking, almost like a CG effect w/ the camera.
Wish I knew what was going on.
Its the identical scene, frame for frame, same rez, same fps, similar color grade, but one is full-frame and the other is letterbox..I don't know why they won't line up 100% and meld/merge into one.

[Frank62]

I didn't use AutoOverlay for quite a time, I will look at what I used, and what worked. But I also remember, that one source did the same as yours, and I couldn't fix it... Will look into this tomorrow.

[Frank62]

This is my code snippet (vu and vd were the two clips):

config=overlayconfig(subpixel=2,minoverlayarea=80,minsourcearea=80)
o=overlayengine(vu.converttoy8(),vd.converttoy8(),configs=config,backwardframes=0,forwardframes=0,mode="update",stabilize=false)
vd=overlayrender(o,vu,vd,debug=false,opacity=1,colorAdjust=1)

I don't remember much, but stabilize did never do anything.
Very important were the numbers minoverlayarea and minsourcearea. (You should look in the readme)

Good luck!