Dynamically resize/crop based on Overlay's results

[Snowknight26]

I have a 720p OAR HDTV cap and a 1080p non-OAR Blu-ray of a short video that I'd like to encode. The Blu-ray is higher bitrate than the HDTV cap so I'm going to encode that solely for the purpose of better archiving quality. The only issue is the mismatched AR.

What I'm trying to do is somehow find a way to dynamically/programatically resize my 720p HDTV cap so that when I Overlay() it ontop of the Blu-ray, it matches. From there, I want to crop the Blu-ray's video so that it becomes OAR and the finally resize down to 720p. This should make the final output look better than my HDTV cap and also have the same framing.

What I'm thinking is make a function that first resizes the video A (HDTV cap) up to match the size of video B (Blu-ray). It would take several parameters, such as best guess width, best guess height, best guess x-position, best guess y-position, and Overlay subtract tolerance. From there, some kind of loop resizes video A 1 pixel at a time using the parameters as a starting point, somehow calculate black levels of Overlay(mode=subtract), loop till the resulting overlay output returns the most black frame (not taking into consideration the parts of video B that haven't been covered by video A), then output that frame accordingly cropped and resized. From there, continue to the next frame. After its all done, resize to 720p. From what I can best determine, this should return video B cropped and resized to match the framing of video A - or so I'd like to hope.

I know it seems very cumbersome (at least for me), but I'm positive that it would be much faster than manually checking each frame, resizing in photoshop, finding the necessary resize/positions, etc.

If anyone has any helps on how to get started with this, a filter or combinations of filters that can already achieve this, or a completely different way at tackling this, I'd be very greatful. Any help at all is appreciated.

[Comatose]

Samples? (screenshots)

[Snowknight26]

http://stfcc.org/misc/se7en.framing.png
http://stfcc.org/misc/se7en.framing2.png

Inside the red outline is video A resized to match the unmodified video B. I'd have to take a guess and say it changes every scene, which is why I'm trying to make it automated and go through every frame.

[mikeytown2]

can you explain in great detail how you came up with the red box. Right now it looks like you just drew it, we need to know why it's in that location/shape. Give more pictures of your process.

In short you want to recreate the pan and scan of a low res source with your high res source, correct?

[Snowknight26]

can you explain in great detail how you came up with the red box. Right now it looks like you just drew it, we need to know why it's in that location/shape. Give more pictures of your process.
I took a screenshot of the same frame of video A and video B, then placed one ontop of the other in Photoshop with reduced opacity. I then resized the smaller picture till the two screenshots were in the same position, had the same orientation, matched, etc (not sure how else to describe it). I added the red outline to make it more clear where each frame was positioned.

In short you want to recreate the pan and scan of a low res source with your high res source, correct?
Essentially, yes - Pan and scan the high res source to match the framing of the low res one.

[mikeytown2]

Once the calculations are done (the hard part) the easy part is to script the resizing/crop. You can use Ken Burns Effect to do all the easy stuff http://forum.doom9.org/showthread.php?t=135776

If you want to script the hard part, take the center block of like 48x48 of the low res shot. match that up with the high res. Since it's centered, you now have the bounding box; plus your only matching a smaller portion. If a perfect match can't be found, use http://avisynth.org/mediawiki/Compare (PSNR) to get it good enough. If you do this right, at the start of each scene it does a full search; and the script tracks the rest using a smaller search area. Set some max ranges, for the smaller search, based on your source.

How long is the clip going to be, how many frames?

[Snowknight26]

I'm not sure if I'm understanding this correctly, but you're suggesting I use the Ken Burns Effect? Isn't that for stills only? Maybe I should have clarified that it's actual motion film footage that needs to be pan and scan'd, not stills... unless I'm mistaken. ;)

My initial test clip was a couple minutes, but once I've gotten that pretty much perfected, I'd like to do the whole film (of which the test clip is a part) - it's a little under 2 hours.



Here's a perfect example of what I'm trying to accomplish.
http://en.wikipedia.org/wiki/File:PanScan7BridesPan.gif

Imagine the 320x140 image being the the high-def version that I have, and the (for the purpose of this example) 100x100 red rectangle being an enlargement of the 50x50 pixel low quality version that I have.

[mikeytown2]

My KBE script works on film :) it's what you want, once you know the motion of the pans. If doing this by hand then Mode 1 (Align, PanX, PanY, & ZoomFactor) is what you want. If you automate this then Mode 2 (define 3/4 points of box) is what you want. If you upload a short section (2 pans), I can show you the various ways to use KBE. I'll be using what I call mode 1 to do this.

[Snowknight26]

Samples will be hard to upload; might have to work without them for now. :\

[Mug Funky]

if this is for archiving, there might be some merit in getting it scripted, then just saving the original source and the avs file...

[Snowknight26]

I've thought about that - either way, still need to write the script.

So I've tried getting started but I can't seem to find a way to make a plain old loop. Does the avisynth syntax allow for it? Either a while or a for loop will do.

[mikeytown2]

recursion is the answer... function that calls its self.

[Didée]

Generally, it should be possible to match the frames up using MVTools. A problem is, though, that full-HD is quite big, and MVTools' max blocksize of 16x16 is pretty small in comparison. Perhaps you'd have to match up half-sized frames first, and proceed from there. MVIncrease would come handy, if it only supported subpel precision (afaik it doesn't).