FXpanse plugin for creating a single frame from panned video

[vcmohan]

FXpanse (http://avisynth.org/vcmohan/FXpanse/FXpanse.html): This plugin creates an expanded frame out of panned video frames. The panning can be towards, left, right, up or down or any combination of these.
A full scene by joining these frames at appropriate places is created. Rotation and Zooming are not handled.
A maximum of 1/4 frame dimension shift between adjacent frames can be handled(Unfurl plugin can handle shifts to one side only).
A fuller description is here (http://avisynth.org/vcmohan/FXpanse/FXpanse.html).
The plugin estimates the frame to frame shifts by cross correlation (in freq domain) and requires fftw dll. To minimise the memory requirement in constructing the full scene, parameters may be specified as tight as possible.
The examples given are out of NTSC format videos I recorded. Some joints seen are due to rotation and some due to persons moving infront close to camera.

To correctly specify parameters a prior run with another plugin Corr2D (http://avisynth.org/vcmohan/Corr2D/Corr2D.html) will be helpful.

All processing is done by the constructor. As such it will be bothersome if this process line is included in the script without using the outpput of this plugin. Comment out.

Even though YUY2 and YV12 video can also be input, incase of odd pixels of shift between frames some inaccuracy will be introduced. On the other hand RGB format will be accurate to a pixel but requires more memory.
The joints seen on the reconstructed scene
http://img81.imageshack.us/img81/3734/flowergardens0av9.th.jpg (http://img81.imageshack.us/my.php?image=flowergardens0av9.jpg)

produced from a video are not seen from scene produced from a synthetic video generated from a large image by cropping and joining.

wd = 800
ht = 240
nw = 12000
nh= 50
nf = 100
imagereader("D:\images\parisbynight.jpg", end = 3)
crop(0,0,14900,500)
trim(1,1)
r = crop(nw,nh,wd,ht)
#return (r)
cr = canvasZigZag(last,r,nf,wd,ht,nw,nh)

fx = FXpanse(cr,wmax = 22, wmin = -22, hmax = 18, hmin = -18)

return (fx)
function canvasZigZag(clip c,clip r,int nframes, int wd,int ht, int nw, int nh)
{
dx = 20 - rand(40)
dy = 2
r = r + crop(c,nw+dx,nh+dy,wd,ht)
nframes <= 1 ? r: canvaszigzag(c,r,nframes - 1,wd,ht, nw+dx, nh+dy)
}

for example
http://img206.imageshack.us/img206/1368/paris0uz7.th.jpg (http://img206.imageshack.us/my.php?image=paris0uz7.jpg)http://img206.imageshack.us/images/thpix.gif (http://g.imageshack.us/thpix.php)

[vcmohan]

On further investigation I found the reason for the joints being seen as black lines. My input video has garbage in the last vertical line and in the last couple of horizontal lines. After cropping them I get a better output as below.
http://img158.imageshack.us/img158/9643/flowergardens0mh6.th.jpg (http://img158.imageshack.us/my.php?image=flowergardens0mh6.jpg)

I have problem with YUY2 format. In case of odd number of pixel shift between frames how to deal. Do I need to internally maintain a 4x4x4 color space and only at output stage process appropriately or is there a better way?

[Sagekilla]

There still seams to be some seams in the output image.. Still, that looks much better than before. Do you do any interpolation on the boundary of two images? I'd imagine it would help smooth out those seams.

[vcmohan]

Do you do any interpolation on the boundary of two images? I'd imagine it would help smooth out those seams.
No. I do not do any interpolation. The seams are seen because the brightness of the scene changes as it is being panned probably due to auto exposure setting. May be a little blurring can help. However if the shift between frames is only one or two pixels then this will mar the scene. I am not sure which way to go about. The seams seen on the bench are probably because the camera was not perfectly horizontal or panning was not. There appears to be some angle between them. This results in that stair case effect.

[mikeytown2]

You could use http://avisynth.org.ru/exinpaint/exinpaint.html to fill in the holes

[vcmohan]

You could use http://avisynth.org.ru/exinpaint/exinpaint.html to fill in the holes
Yes. As a post process one can do that. The unfilled holes are all black. Adding that code to include in this plugin will make it unweildy.