very unusual request for filter or avisynth help: slit scan reconstruction

[snellwilco]

OK...this is really out there. I've searched on vain on doom9 and I don't think it has come up yet.

There is a television series whose title sequence consists of a rostrum camera shot. This means a moving shot over a stationary, non-moving object. In this case the object is a long painting or mural, probably about 1 ft. tall X 25 ft. long, I'd guess. The camera dollies to the right, depicting scenes from the series painted on a long canvas. The titles, of course, are superimposed over this. Actually I suspect the way it really happened was the canvas was wound onto some kind of reel-to-reel and the camera was stationary. You get the picture by now.

What I want to do is to use the title sequence, to reconstruct as a bitmap image, the entire painting. This is somewhat similar, conceptually, to what this person did to reconstruct the 2001 slit-scan sequence sources.
http://www.seriss.com/people/erco/2001/
But to me it should be less complicated, because the rostrum and camera movement is simpler than a slit scan special effects sequence.


As I said I realize this is way out there. Any help on where to begin would be hugely appreciated! BTW I'm a pretty experienced user of Avisynth and Virtualdub, not a newbie. But I don't think the answer to this would be obvious to anyone but those with a plug-in developer level of knowledge! BTW I know there is a brute force approach of taking many screencaps and trying to use a panoramic stitching program to put them together, but it would be really difficult and time consuming. A lot of the time the titles are in the way and you'd have to work around that. OTOH the slit-scan reconstructor could just work with the rightmost 20 pixels and never be bothered by the titles. I know avisynth was not designed for writing bmps, except as single screencaps, but I've seen people do incredible things just by programming in avisynth and not having to even write an external filter. I really hope someone has experimented with something like this before.

Since the curiosity to see what I'm trying to capture is probably insatiable, here's what it looks like. Obviously the DVD I have is higher quality.
http://www.youtube.com/watch?v=9j4jJVyU-Nk

[mikeytown2]

Can you post a clip?

[snellwilco]

I added a link, thanks.

[mikeytown2]

IMHO the easiest way is to do it manually.
http://hugin.sourceforge.net/tutorials/scans/en.shtml

Otherwise a bunch of Trim's & Crop and a huge StackHorizontal would be the way to do it. I don't know of any plugin/script that will do it automatically.

[Gavino]

Otherwise a bunch of Trim's & Crop and a huge StackHorizontal would be the way to do it. I don't know of any plugin/script that will do it automatically.
It wouldn't be too difficult if you could assume that the camera is panning right (or the picture moving left) at a constant rate, say x pixels/frame (or frames/pixel) where x is a simple rational number (supplied by the user). But maybe that's not a realistic assumption?

[mikeytown2]

FFMpegSource("Mapp and Lucia Painting.flv").Trim(19,484)
SelectEvery(32,0)

StackHorizontal(Trim(0,-1),Trim(-1,-2),Trim(-2,-3),Trim(-3,-4),Trim(-4,-5),
\ Trim(-5,-6),Trim(-6,-7),Trim(-7,-8),Trim(-8,-9),Trim(-9,-10),
\ Trim(-10,-11),Trim(-11,-12),Trim(-12,-13),Trim(-13,-14),Trim(-14,-15))
Trim(14,0)


Took about 5 min
http://img258.imageshack.us/img258/6696/testcz0.th.jpg (http://img258.imageshack.us/img258/6696/testcz0.jpg)

It's not perfect, but other then the slight edge misalignments and text, it does work. Using Hugin should fix the color imbalances between the left and right sides of the frame.

[JanBing]

There's a filter for VirtualDub which does exactly what you need, named Panorama Generator (http://www.fml-home.de/videoamateur/filter/panorama.html).

[snellwilco]

thanks to both of you who posted solutions. I plan to try the panorama filter later today, and it's good to know the backup method is feasible as well.

[Gavino]

Here's a (recursive) AVS function that does what I was getting at in post #5 above.
function DeMorse(clip c, int p, int f) {
DeMorse2(Crop(c, 0, 0, p, 0), c.Trim(f, 0), p, f)
# add remainder of last frame:
end = c.Framecount-1
StackHorizontal(c.Trim(end, 0).Crop(p-(end%f)*p/f, 0, 0, 0))
}

function DeMorse2(clip left, clip right, int p, int f) {
left = StackHorizontal(left, right.Crop(0, 0, p, 0))
right.Framecount <= f ? left : DeMorse2(left, right.Trim(f, 0), p, f)
}

It stitches together the leftmost p pixels every f frames, thus avoiding the credits.
I've called it DeMorse in honour of renowned rostrum cameraman Ken Morse (http://en.wikipedia.org/wiki/Ken_Morse).

Using it like this
DirectShowSource("C:\folder\mapp and lucia.flv", fps=10)
Trim(19, 484) # select panning sequence
ConvertToRGB() # avoid cropping restrictions
DeMorse(30,3)
ImageWriter("C:\folder\mapp and lucia", 0, 0, "jpg")
I was able to generate the following image
http://img155.imageshack.us/img155/3302/mappandlucia000000re6.th.jpg (http://img155.imageshack.us/my.php?image=mappandlucia000000re6.jpg)

I've chosen jpg for its smaller filesize but you can generate bmp just by changing the ImageWriter call.

It's not perfect as you can just make out the joins. This is because the scan rate is not exactly 30 pixels in 3 frames - you should be able to improve with the higher resolution original as you can use bigger numbers to get closer to the exact ratio.

Great to see that Avisynth copes with over 150 levels of recursion without breaking sweat.
OK, StackHorizontal and Crop are pretty cheap as filters go, but I was impressed.

[hanfrunz]

try the resizer-ability to use fractional values for cropping, so you could compensate the scrolling speed to something exact (30pixels/sec) But don't ask me how this is done exactly :-)

you could track one point an measure it's speed and then go from here...

hanfrunz

[Gavino]

try the resizer-ability to use fractional values for cropping, so you could compensate the scrolling speed to something exact (30pixels/sec) But don't ask me how this is done exactly :-)
Here's a version that uses your suggestion, taking a float for the number of pixels where the previous version used an int.
function DeMorseF(clip c, float p, int f) {
DemorseF2(GetSlice(c, p), c.Trim(f, 0), p, f)
# add remainder of last frame:
end = c.Framecount-1
StackHorizontal(c.Trim(end, 0).Crop(floor(p-(end%f)*p/f), 0, 0, 0))
}

function DeMorseF2(clip left, clip right, float p, int f) {
left = StackHorizontal(left, GetSlice(right, p))
right.Framecount <= f ? left : DeMorseF2(left, right.Trim(f, 0), p, f)
}

function GetSlice(clip c, float p) {
BilinearResize(c, ceil(p), c.height, 0, 0, p, 0) # or another resizer
}

I got subjectively best results using DeMorseF(29.4, 3), but to be honest I can scarcely see much difference from the earlier attempt. Perhaps some sort of debanding filter could make the joins look smoother.

[vcmohan]

I have a new plugin for this problem. Please see thread Unfurl (http://forum.doom9.org/showthread.php?t=140854)in the Avisynth usage section.

Please comment on its usefulness etc.;

[snellwilco]

vcmohan, hanfrunz, Gavino...THANK YOU! I'm returning to this problem after a month or so of working on other projects. (outdoor ones, have to do it while the weather is nice LOL)

You guys are great! I plan to try these new functions ASAP.