stabilisation / image registration

[jmac698]

Hi,

I wrote a script 10 years ago to match images, but now I forget what plugins I used.

The problem:
match two frames by translation and unequal scale, with cropping.
For example, find the transformation from 4:3 version to 16:9 version - telling me the cropping, scaling, and even stretching (fattening).
This would be considered an affine transform.
There is no lens distortion. I need the actual parameters, besides to transform one to the other.
It's pan and scan with linear stretch.

I think there was one that used an FFT.

Thanks.

[poisondeathray]

AutoOverlay is probably different than what you used
https://forum.doom9.org/showthread.php?t=175247
https://github.com/introspected/AutoOverlay

It has a debug editor pop up option (editor=true) - a GUI of sorts that displays the crop, overlay values

[Frank62]

If it was not AutoOverlay it would be extremely interesting what else you had used!

[jmac698]

Thanks. It's not providing me with a correct result and I'm not clear if there's a way to solve this. I'm having trouble understanding how to adjust it. Does anyone have experience with this?

As for what I used before, it was probably DePan or MVTools.

Here's the pics I want to match and the result:
https://imgur.com/a/J8UStlS


LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\AvsFilterNet.dll")

source=ImageSource("raw.png").ConvertToRGB24()
overlay=ImageSource("scene 1 zoom 1.png").ConvertToRGB24().BilinearResize(960, 540)
OverlayConfig(angle1=0, angle2=0)
OverlayEngine(source, overlay, debug=True, editor=True)
OverlayRender(source, overlay, colorAdjust=1, mode=2, gradient=20, width=500, height=500)

[jmac698]

This plugin just won't work for me. I gave it a perfect crop, but it insists on ignoring MaxDiff and MinX.


LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\AvsFilterNet.dll")

# Create a single frame video from an image
source=ImageSource("raw.png", use_DevIL=true).Trim(0, -1) #start frame 0 for -n frames

# Compute a 16:9 crop of the source
left = 0
vertical_space = source.Height() - source.Height() * 9/16
vertical_pan = 419 - 224
#vertical_pan = 0
top = vertical_space/2 + vertical_pan
width = source.Width()
height = source.Height() * 9/16
overlay=source.Crop(left, top, width, height)
OverlayConfig(angle1=0, angle2=0, aspectRatio1=16/9, aspectRatio2=16/9, branches = 5, correction = 5, minX = 0, minY = 224)
OverlayEngine(source, overlay, debug=True, editor=True, maxDiff = 0)
OverlayRender(source, overlay, colorAdjust=1, mode=2, gradient=20, width=1024, height=1024)

result is x=-4, y=415, diff = 2.7, size = 1036x582
it should be 0, 419, 0, 1024x576

[jmac698]

I've decided to do a brute force search starting with an estimate from a paint program. I can use Compare to find PSNR and animate crop, zoom etc. in clips
Too bad this has to be so hard.

[poisondeathray]

You posted obviously different frames, maybe AutoOverlay getting confused? Did you try using the same frame ?

If you roughly align the ground in avisynth, it's obvious the clouds are moving in the z-axis - but you don't even know that from 2 frames - it's just a bunch of assumptions I'm making. Maybe camera is moving too between those frames

This is apng, it should animate in most browsers
https://i.postimg.cc/tTgXc4KR/alignground-apng.png (https://postimages.org/)

rough ground alignment

ImageSource("1_9q9ndot.png")
crop(67,430,-66,-27,true)
levels(15,1.5,222,0,255,false)
lanczosresize(1024,576)
subtitle("align ground")


reference @ 1024x576

ImageSource("2_9aeb6lU.png")
lanczosresize(1024,576)
subtitle("reference 1024x576")


During the time difference between those frames - the clouds look like they have moved, people have obviously moved, - that will confuse everything, including commercial trackers/aligners/warpers, unless you guide the inclusion / exclusion features. They will try to align the clouds as "features" too, and that will obviously mess things up. Not sure what you expected from different frames ?

Is the alignment different per frame, or per scene ? ie. is this something you have to recalculate per frame or per scene ?

[jmac698]

Hello,

Thank you for your help and advice. Strangely enough, I had never looked at the two frames in sequence, and I agree with your assessment that the clouds have moved and would confuse the registration.

I made a crop of just the ground, however I am still getting nonsensical results.


LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\AvsFilterNet.dll")

source = ImageSource("raw.png", use_DevIL=true).Trim(0, -1)
overlay = ImageSource("scene 1 zoom 1.png", use_DevIL=true).Trim(0, -1).BilinearResize(960, 540)
w = overlay.Width()
h = overlay.Height()
ground_height = Round(146/1080.*h) # ground height in HD pixels
#Return MessageClip(String(ground_height))
ground = overlay.Crop(0, h - ground_height, w, ground_height)
#return ground
OverlayConfig(angle1=0, angle2=0, minX = 0)
OverlayEngine(source, ground, debug=True, editor=True)
OverlayRender(source, ground, colorAdjust = 1, mode = 2, gradient = 20, width = 1024, height = 1024)


minX is still being ignored. The results I get are:
X: -3203 Y:905 Angle: 0.00
Diff.: 15.4

I don't know how it can match something that is completely offscreen.

Your rough match is very good. How did you make that?

I only need to match 2 frames with the reference.

Let me try again, comparing ground to ground.

[jmac698]

I tried ground to ground with matching levels, it's not working.


LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\AvsFilterNet.dll")

source = ImageSource("raw.png", use_DevIL=true).Trim(0, -1).Levels(15, 1.5, 222, 0, 255, false)
source_ground = source.Crop(0, 920, 1024, 104)
source_w = source_ground.Width()
source_h = source_ground.Height()
#return source_ground
overlay = ImageSource("scene 1 zoom 1.png", use_DevIL=true).Trim(0, -1).BilinearResize(960, 540)
w = overlay.Width()
h = overlay.Height()
ground_height = Round(146/1080.*h) # ground height in HD pixels
#Return MessageClip(String(ground_height))
ground = overlay.Crop(0, h - ground_height, w, ground_height)
#return ground
OverlayConfig(angle1=0, angle2=0, minX = 0)
OverlayEngine(source_ground, ground, debug=True, editor=True)
OverlayRender(source_ground, ground, colorAdjust = 1, mode = 2, gradient = 20, width = source_w, height = source_h)


Results:
X: -37 Y: -1 Angle: 0.00
Diff: 11.6

[jmac698]

I think there's a bug in the X direction matching. On a hunch, I rotated the videos. The results are much more sensible now.


LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\AvsFilterNet.dll")

source = ImageSource("raw.png", use_DevIL=true).Trim(0, -1).Levels(15, 1.5, 222, 0, 255, false)
source_ground = source.Crop(0, 920, 1024, 104).TurnRight
source_w = source_ground.Width()
source_h = source_ground.Height()
#return source_ground
overlay = ImageSource("scene 1 zoom 1.png", use_DevIL=true).Trim(0, -1).BilinearResize(960, 540)
w = overlay.Width()
h = overlay.Height()
ground_height = Round(146/1080.*h) # ground height in HD pixels
#Return MessageClip(String(ground_height))
ground = overlay.Crop(0, h - ground_height, w, ground_height).TurnRight
#return ground
OverlayConfig(angle1=0, angle2=0, minX = 0)
OverlayEngine(source_ground, ground, debug=True, editor=True)
OverlayRender(source_ground, ground, colorAdjust = 1, mode = 2, gradient = 20, width = source_w, height = source_h)


Results:
Size: 92x889
X: 14 Y: 67 (Angle can't be read)
Diff: 10.9

That would translate to ground going on top of source_ground at
X=67 and resized to 889, and moved up from the bottom by 14 so that the two tops match (the ground is also squished vertically)

[poisondeathray]

Your rough match is very good. How did you make that?


Manually , with tabs in avspmod. I cropped until it had similar features and resized until the ground matched, checking back and forth between tabs. Because tabs are spatially (and temporally aligned), it makes seeing difference between reference easy, and the hot swap is fast enough (number keys) that you don't get frustrated . It's like tracks or layers in a NLE, but faster


I think there's a bug in the X direction matching. On a hunch, I rotated the videos. The results are much more sensible now.

Consider reporting the issue to the author in the github issues tracker, or the AutoOverlay thread

[StainlessS]

In the past, anything posted in AutoOverlay thread had response witin a day or two.