Project to separate film telerecordings into their original fields

[alexh110]

I have recently performed this new experiment:

http://colour-recovery.wikispaces.com/Extrapolating+A+More+Complete+Field+Separation

Until now it was thought impossible to separate film telerecordings into their original interlaced fields; however my experiment shows that this is not the case!

Would anyone be interested in developing this process into an AviSynth script, as I don't have the expertise to write the code?

[jmac698]

Hi, welcome to doom9!
Feel free to state any ongoing problems here, even things that you assumed were impossible. It turns out that your particular problem is easily turned into script, and there's plugins already developed that can make it fast.
Also to note, I've used qbasic for experiments myself, it's great for prototyping, but you can use http://fbide.freebasic.net/ it's completely compatible and much faster.
I can work on your script a bit later.

There's a plugin that can calculate the scale factor you're looking for; it has to do with a triangle identity.

[pbristow]

Hooray!!! It's great to see the CRWG becoming active again! :)



Until now it was thought impossible to separate film telerecordings into their original interlaced fields; however my experiment shows that this is not the case!


Assumed by whom? *I* never assumed that. :) Indeed I was disappointed that the CRWG seemed to go quiet just when they were on the verge of being able to do this. To me, it had always seemed the main obstacle was getting a high enough resolution scan of the film to be able to reliably pick out the original lines, given that (a) they would no longer be linear (due to image distortions), and (b) they would tend to shift around from frame to frame.

Based on the results you're showing here (I haven't dared look at the code yet) I think the technique you've come up with, in combination with motion-estimation technology as used in the VidFire technique (or in our very own MVTools2), raises the possibility of some very authentic and "clean" restorations in future. For example, the problem with losing sync in the middle of your demo sequence could be detected by the sudden change in apparent motion, and corrected either by re-tuning the detection of the lines or, if that failed, by using a mixture of motion estimation and image subtraction to remove the "wrong field" information from the target output. Run the result of that process through motion-compensated de-noising, and you could get a pretty solid looking result even if, say, 20% of fields were wrongly detected on the first analysis, as long as the good field-detections happened close enough together to provide the necessary clues as to what *should* be going on...

Those Doctor Who "Revisitations" boxsets might become a bit more attractive to me in the near future! :)

Would anyone be interested in developing this process into an AviSynth script, as I don't have the expertise to write the code?

That's very tempting... I'm not an expert scripter and I don't get a lot of free time to work on stuff, so I'm very hesitant make any promises. If you get another offer, take it; but I'd love to be involved in this in some way if I can.

Step one, I'll need to sober up a bit. :) Ping me tomorrow evening.

[alexh110]

Thanks very much for your replies.

These are the stages I envisaged for a fully developed process:

1) Take an undistorted raw 1920X1080 Y8 video-stream as input (with file-extension .y).
2) Multiply each source frame by my field filters (the two VERTFILT bmps published at the bottom of my experimental write-up), with suitable pixel-scaling factor to ensure no white-crushing occurs. This produces the partially separated fields.
3) Downscale to 720X576.
4) Process the partially separated fields with a script equivalent to Fldsep3.bas to produce fully separated fields.
5) Combine the fields to make an interlaced output stream at 576i.
6) Denoise the stream.

This can then be saved within an avi container using Virtual Dub, with or without compression as required.

N.B. The downscaling needs to be based on a very high quality resampling algorithm for the process to work, but I think the Lanczosplusv3 Avisynth script would probably be adequate.

Fldsep3.bas now assumes 576 lines instead of 573, and writes output at 576p (simply by writing each field-line out twice).
This is just to make the output clearer and brighter so I can see what's going on.
Obviously for a proper 576i version the line-doubling should be removed from the code.

This is the denoising script I used:

AviSource("F:\Project\Feb 2012\Fldsep3.bas\Denoise\Raw file.avi")
ConvertToYV12
loadplugin("fft3dfilter.dll")
loadplugin("FQuiver.dll")
loadplugin("deflicker.dll")

F1Quiver(last,"bh",304,15,degree=32,gamma=0.1)
fft3dfilter(sigma=10, interlaced=false)
DeFlicker()


F1Quiver removes the chroma dot-crawl; though this would be better done with F2Quiver once I work out the correct parameters.
It may well be better to apply the FQuiver filtering immediately after the downscaling to 720X576, prior to processing with Fldsep3.bas.

[pbristow]

Hmmm... Your technique seems to assume that the "undistorted" image is *perfectly* undistorted, which I'm pretty sure it isn't. I would be more inclined to treat it as a good first approximation. Rather than using a static mask to isolate the fields and then downsizing to 576 lines, I would use something like your .bas script to examine adjacent pairs of lines in each frame of the 1080p image, looking at shortish segments (say, 64 pixels wide) to see if the two lines correlate strongly with each other at that point or not. Where they do correlate, you can assume both samples came from the same original line of the 576i image; where they don't, you'll need some way to figure out whether the line has drifted upward or downward (possibly by checking the apparent motion with previous and next frames/fields), and adjust your "tracking" accordingly for that line segment. In this way you can build a 576 line image that has the two fields isolated as accurately as possible, given the (non-ideal) 1080 lines source.

The ideal source, of course, would have something over 1152 lines, to ensure there were at least two samples available from each original scan line. If you had 1728 or more lines to work from (3 samples for each original scanline) the problem becomes trivial: Pick and average the best two samples out of three, and check as you go along which sample is disagreeing the most with the other two. If it's the top sample that starts to disagree, then you need to shift your window of examination downward by one line; if it's the bottom one, then upward.

Once you've got a working .bas script for that process, the natural thing to convert it to would be an avisynth plug-in in C/C++, rather than a script. Avisynth scripting isn't naturally suited to treating frames line-by-line: its natural mode of processing is "do this operation to the entire field or frame". Line-by-line or pixel-by-pixel detail is much better addressed within a plug-in, both for ease of getting your head around the coding and for performance of the end result.

All IMHO, of course. :)

[alexh110]

Thanks.
What you're touching on here is the really difficult part of the project! I hadn't envisaged attempting that part until the first stage had been completed as described above.

What I've found is that my undistorted version of the TOTP footage is accurate enough to allow partial field separation using static filters; but after 200 frames or so the geometric distortion has drifted so much that the process breaks down completely.
It's easy to detect when this happens, because the pixel differences between the fields will tend towards zero.

However correcting the problem is really hard. You would want to apply a geometric adjustment every so often to re-straighten the scan-lines, before applying the field-filters. Or track and isolate them in some manner as you suggest.
Coding for this is way beyond my ability; but perhaps not beyond yours :-)

Perhaps it might be better though to do the easy part first, along the lines I suggested, and worry about the tracking later on.
There's no guarantee this process will work on anything other than TOTP. The spot-wobble could be more severe on other film recordings, and if so, the scan-lines would be completely blended, and my method would not work.
What I had in mind was to develop a relatively simple AviSynth process that I could offer the BBC for testing on other film recordings.
If it was found to work, then a more advanced process could be developed.

N.B. Richard Russell uses the chroma frequencies to track vertical distortion. He mixes the signals at 72 and 216 vertical cycles to produce a signal at 288 cycles. This would have zero horizontal frequency with respect to the original geometry (before it was distorted by the film recorder). So that's also a possible method of tracking the scan-lines; though the vertical phase of the signal at 288 cycles is unstable.

[alexh110]

So I think my script would look something roughly like this:


loadplugin("rawsource.dll")
rawinput = RawSource("F:\totp_ch03.y", 1920, 1080, "Y8")
f1mask = ImageSource("F:\VERTFILT field 1.bmp")
f2mask = ImageSource("F:\VERTFILT field 2.bmp")

f1partial = mt_lutxy(rawinput, f1mask, expr = "x y * 255 /")
f2partial = mt_lutxy(rawinput, f2mask, expr = "x y * 255 /")

lanczosplus(f1partial, 720, 576)
lanczosplus(f2partial, 720, 576)

(Remove the blank lines from f1partial and f2partial, leaving only the active lines. Don't know how to do this part!)

f1full = mt_lutxy(f1partial, f2partial, expr = "x y - 2 * x +)
f2full = mt_lutxy(f1partial, f2partial, expr = "y x - 2 * y +)

(May need to clamp output pixel values between 0 and 255 using mt_clamp?)

Interleave(f1full, f2full)
SeparateFields()
SelectOdd()
Weave()

ConvertToYV12
loadplugin("fft3dfilter.dll")
loadplugin("FQuiver.dll")
loadplugin("deflicker.dll")

F1Quiver(last,"bh",304,15,degree=32,gamma=0.1)
fft3dfilter(sigma=10, interlaced=true)
DeFlicker()


I know I've probably made some schoolboy errors, as I've only ever written simple scripts in the past!
Comments welcome.

[alexh110]

Here's a revised version of my script:

loadplugin("rawsource.dll")
rawinput = RawSource("F:\totp_ch03.y", 1920, 1080, "Y8")
f1mask = ImageSource("F:\VERTFILT field 1.bmp")
f2mask = ImageSource("F:\VERTFILT field 2.bmp")

f1partial = mt_lutxy(rawinput, f1mask, expr = "x y * 255 /")
f2partial = mt_lutxy(rawinput, f2mask, expr = "x y * 255 /")

lanczosplus(f1partial, 720, 576)
SeparateFields()
f1partial720 = SelectOdd()
lanczosplus(f2partial, 720, 576)
SeparateFields()
f2partial720 = SelectEven()

f1full = mt_lutxy(f1partial720, f2partial720, expr = "x y - 2 * x +")
f2full = mt_lutxy(f1partial720, f2partial720, expr = "y x - 2 * y +")

(May need to clamp output pixel values between 0 and 255 using mt_clamp?)

Interleave(f1full, f2full)
SeparateFields()
SelectOdd()
Weave()

(Think I got that last bit wrong: it needs to treat f1full and f2full as fields (they're both 288 lines) and interlace them to form 576i. Then may need to set the framerate at 25fps.)

ConvertToYV12
loadplugin("fft3dfilter.dll")
loadplugin("FQuiver.dll")
loadplugin("deflicker.dll")

F1Quiver(last,"bh",304,15,degree=32,gamma=0.1)
fft3dfilter(sigma=10, interlaced=false)
DeFlicker()

[Gavino]

May need to clamp output pixel values between 0 and 255 using mt_clamp?
Output of mt_lut is already clamped to [0,255].

Interleave(f1full, f2full)
SeparateFields()
SelectOdd()
Weave()

(Think I got that last bit wrong: it needs to treat f1full and f2full as fields (they're both 288 lines) and interlace them to form 576i.
Then you want SelectEvery(4,0,3) instead of SelectOdd().

Then may need to set the framerate at 25fps.)
Resulting frame rate will be the same as the original raw input.
You can easily add AssumeFPS(25) if necessary.

[alexh110]

Many thanks!
I spotted several other errors too, so my new script is:



loadplugin("rawsource.dll")
rawinput = RawSource("F:\totp_ch03.y", 1920, 1080, "Y8")
f1mask = RawSource("F:\VERTFILT field 1.y", 1920, 1080, "Y8")
f2mask = RawSource("F:\VERTFILT field 2.y", 1920, 1080, "Y8")

f1partial = mt_lutxy(rawinput, f1mask, expr = "x y * 255 /")
f2partial = mt_lutxy(rawinput, f2mask, expr = "x y * 255 /")

LanczosResize(f1partial, 720, 576)
SeparateFields()
f1partial720 = SelectOdd()
LanczosResize(f2partial, 720, 576)
SeparateFields()
f2partial720 = SelectEven()

f1full = mt_lutxy(f1partial720, f2partial720, expr = "x y - 2 * x +")
f2full = mt_lutxy(f1partial720, f2partial720, expr = "y x - 2 * y +")



Interleave(f1full, f2full)
AssumeFrameBased()
SeparateFields()
SelectEvery(4, 0, 3)
Weave()



ConvertToYV12
loadplugin("fft3dfilter.dll")
loadplugin("FQuiver.dll")
loadplugin("deflicker.dll")

F1Quiver(last,"bh",304,15,degree=32,gamma=0.1)
fft3dfilter(sigma=10, interlaced=false)
DeFlicker()

[alexh110]

Output is being produced at 288i, so something's wrong.
However this seems to produce the correct output (at least in terms of frame-rate and raster size):

Interleave(f1full, f2full)
Weave()

[pbristow]

Inserting some comments that may help you spot the problems.



loadplugin("rawsource.dll")
rawinput = RawSource("F:\totp_ch03.y", 1920, 1080, "Y8")
f1mask = RawSource("F:\VERTFILT field 1.y", 1920, 1080, "Y8")
f2mask = RawSource("F:\VERTFILT field 2.y", 1920, 1080, "Y8")

f1partial = mt_lutxy(rawinput, f1mask, expr = "x y * 255 /")
f2partial = mt_lutxy(rawinput, f2mask, expr = "x y * 255 /")

LanczosResize(f1partial, 720, 576) # Result is 576 high, 25 fps
SeparateFields() # Result is 288 high, 50 fps
f1partial720 = SelectOdd() # Result is 288 high, 25 fps
LanczosResize(f2partial, 720, 576) # Result is 576 high, 25 fps
SeparateFields() # Result is 288 high, 50 fps
f2partial720 = SelectEven() # Result is 288 high, 25 fps

f1full = mt_lutxy(f1partial720, f2partial720, expr = "x y - 2 * x +")
f2full = mt_lutxy(f1partial720, f2partial720, expr = "y x - 2 * y +")



Interleave(f1full, f2full) # Result is 288 high, 50fps
AssumeFrameBased()
SeparateFields() # Result is 144 high, 100fps - This step is redundant, you are already working with separated fields.
SelectEvery(4, 0, 3) # Result is 144 high, 50fps
Weave() # Result is 288 high, 25fps


( - - SNIP! - - )



Instead of:

Interleave(f1full, f2full) # Result is 288 high, 50fps
AssumeFrameBased()
SeparateFields() # Result is 144 high, 100fps - This step is redundant, you are already working with separated fields.
SelectEvery(4, 0, 3) # Result is 144 high, 50fps
Weave() # Result is 288 high, 25fps



Try:

Interleave(f1full, f2full) # Result is 288 high, 50fps
AssumeFieldBased() # This allows "weave" to work.
Weave() # Result is 576 high, 25fps

[pbristow]

Heh, and now I see that that's exactly what you've done. :)

[alexh110]

I didn't use AssumeFieldBased()
Seems Weave works without it; but should I perhaps put it in anyway?

[Gavino]

The clip is already fieldbased at that point, since f1full and f2full are derived from the separated fields of f1partial and f2partial.

[alexh110]

Thanks.
Think I'm starting to get the hang of it now!

Have tested the script, and it works!
Produces similar results to the QBASIC program.
Though some frames separated better; others were worse.

Now I just need to test a longer sequence.

[alexh110]

Have tested a 361 frame clip, and was pleased to find the separation continues to work over the whole clip! However performance degrades intermittently, and the field-order reverses several times due to misalignment of the field-filters with the source scan-lines.

I think film-stabilisation processing may fix the problem.
Otherwise would have to develop dynamic filters that track the scan-lines by locking onto the chroma-patterning. The process would have to be PAL-aware, to allow for the PAL 8-field sequence.

My current script is:


loadplugin("FQuiver.dll")
loadplugin("rawsource.dll")

rawinput = RawSource("F:\totp_ch03(undist).y", 1920, 1080, "Y8")
f1mask = RawSource("F:\VERTFILT field 1.y", 1920, 1080, "Y8")
f1mask2 = F2Quiver(f1mask,"bhh",70,15,degree=32)
f2mask = RawSource("F:\VERTFILT field 2.y", 1920, 1080, "Y8")
f2mask2 = F2Quiver(f2mask,"bhh",70,15,degree=32)


f1partial = mt_lutxy(rawinput, f1mask2, expr = "x y * 255 /")
f2partial = mt_lutxy(rawinput, f2mask2, expr = "x y * 255 /")


LanczosResize(f1partial, 720, 576)
SeparateFields()
f1partial720 = SelectEven()

LanczosResize(f2partial, 720, 576)
SeparateFields()
f2partial720 = SelectOdd()


f1full = mt_lutxy(f1partial720, f2partial720, expr = "x y - 2 * x +")
f1full2 = ConvertToYUY2(f1full)
f1fullshift = AddBorders(f1full2,0,1,0,0)
f1fullshift2 = Crop(f1fullshift,0,0,0,-1)
f1fullshift3 = ConvertToYV12(f1fullshift2)

f2full = mt_lutxy(f1partial720, f2partial720, expr = "y x - 2 * y +")


Interleave(f1fullshift3, f2full)
Weave()


loadplugin("fft3dfilter.dll")

F1Quiver(last,"bh",304,15,degree=32)
fft3dfilter(sigma=10)



Not sure whether I need all those stream-labels? As an AviSynth novice, I'm not 100% sure when you need them and when you don't!

The F2Quiver filtering is designed to remove the horizontal banding at 72 c/aph. This is a beat product of 1080 with the 1152 harmonic of 576 (or rather it's alias at 1008).
Ideally would want to low-pass filter the fields before downsampling; but this slows down the process too much. So as a compromise I am pre-filtering the field-filters instead.
(Obviously I will do this as a separate process with a secondary script; I've only left it in the main script for illustrative purposes.)

[alexh110]

Tried film stabilisation; but it doesn't solve the problem.
Some way will have to be found to automatically determine field-order, and re-sort the fields accordingly. Perhaps by motion-tracking?

[jmac698]

Hey,
any progress on this? I can write plugins now so I could handle most of that stuff.

[alexh110]

None I'm afraid.
I've discovered that the process will only work on about 20% of film recordings, as they need to have unusually low cross-talk between the fields.

If you could write a plugin that would re-sort mis-sequenced fields into the correct order that would certainly help!