Restore24: Script to recover FILM from field-blended PAL

[Didée]

Hello all.

Beware, this post is LONG.

The beginning of this effort lies here. I couldn't live with simple deinterlacing, and I couldn't live with SmartBobbed video, either. So I started just another attempt - probably the X-thousandth one in digital video history - to somehow restore 24p content of such mistreated sources.
The task seems easy: don't use the blended fields, but only the good ones, while avoiding to drop or duplicate any of the good fields. Well, it would be easy, if computers weren't dumb like grass ...
Now, I scripted something together that, although being far from perfect yet, already works pretty well. As of now, it's hardly more than proof-of-concept, and lots of improvement should be possible. But since already the very first and *dumbest possible* version worked like a charm, I really *do believe* in this concept, and strongly suggest you people to step in and help improving!

The basic principle is in no way spectacular:

1. Bob the video to 50fps
2. Detect fields that are blends
3. Replace those blended fields with unblended neighbors
4. Decimate the cleaned stream to 24fps
5. Smile and enjoy

Step 1 is obvious.
Step 2 is working quite nice, although I cannot analyze the frames in an optimal way through an AviSynth script only. This would need additional commands not yet available. In the end, the best solution would be to make a filter plugin out of all of this.
Step 3 & 4 are directly related, and these are the really tricky ones! Avoiding the blends is not so hard, but if we get a jerky stream, like "go-stop-go-go-stop-jump-go-jump-go-..." as an exchange, then that would be even more distracting. And exactly these two steps are the first target to optimise. Currently, my solution still has jerks every now and then - it depends also a lot on scene content. Straight pannings work really good, and so do most sequences with noticable motion in it. It gets harder in still scenes with little motion, e.g. mouth-movement only, and in scenes with too low contrast or blurry picture.

Now let me explain the mehod. The script comes at the very end.


1. Bobbing: Doubling the frame rate

This is the first step: separate the fields, and convert them to fullsize frames, so that we have something to work with.
For analyzing, I use an Unsmart bob, to retain the fields as pure as they are. In fact, "TomsBob" is used with line averaging acivated ( TomsMoComp(x,-1,1) ) as this seems to work out best with the given concept.
However, it is perfectly possible to use any other method of (Smart)Bobbing to finally render the stream, as long as the output of the bob filter used for rendering is exactly in-phase with TomsBob.


2. Detecting blended fields

At first, I tried the plugins "deblend" and "unblend" for this step. Most probably, it was me being too dumb to use them correctly Quite some blends were detected, but lots of blends sneaked through. Changing the parameters seemed to have almost no effect at all, and all the filter's work was too much of a "black box" to me: I simply don't understand their working principle. Then, I would have liked to try the PFR plugin, but this one in all versions crashes for me with Access Violation after about 3 to 5 frames - sorry, Simon. So, I started my own approach of detecting blends.)

Look at a clean frame and a blended frame side-by-side. The clean frame has well-defined content. The blended frame's content is not so well defined: the detail in it has lower contrast, just because the blending has dimished the detail.
Well, that's it: if we calculate an edge mask of these two frames, the edge mask of the clean frame will appear noticabely brighter than that of the blended frame. Or, in contrary, the edge mask of the blended frame will appear darker.
This, simple as it is, is my way to detect the blended fields:
- create an edge mask on the bobbed stream
- for every frame, check if the current frame's edgemask is darker than that of both the preceeding and proceeding frame. If yes, it's a blend, on to step 3. If no, it's clean, leave it as it is.
This works wonders, detecting almost every blend!
BUT, one problem is left here. See "Possible improvements" below.


3. Replacing the blends

Replacing a blend is quite easy, as soon as we have detected one. The bigger question is: replace with what?
The obvious answer seems to be: replace it with the neighbor that is more similar. However, in practice, this is not necessarily true. My first attempts did use the most similar neighbor, but the end result was quite jerky in places. After fiddling a long time without noticeable improvement, I lost nerves and simply replaced with the preceeding frame *always*. To my surprise, the result was much, much smoother - but still not satisfying. The goal is to achieve an evenly distributed amount of 2-frame-dups and 3-frame-dups in the bobbed source, and by all means avoid to let stay single frames, because these are in danger to be missed by the final decimation step. As for now, I have some simple checks in the script that try to do exactly this, but it's not working out good enough - yet.


4. Decimating the cleaned stream

After replacing the blends, we need to decimate the 50fps stream down to 24fps (or maybe to 25fps, in case the NTSC stream was created from PAL, not FILM).
To make a long story short: after lots of trying, waiting, swearing and tearing-out-my-hair, I stepped away from Donald's decimate for this particular task. I don't want at all to make Donalds work small somehow - what would we do without his great decomb package? But in the given case, I couldn't get from it what I wanted. (But keep in mind, as a PAL user, I am no deinterlacing expert at all ). However, with decimate(), I'd need to decimate in at least two steps to get to 24fps: "decimate(2).decimate(25)". Firstly, decimating 1-of-25 is somewhat slow, making previewing difficult, and decimate(2) is not smart enough for the stream created so far. It got better with "decimate(4, mode=2).decimate(3, mode=2).decimate(25)", but this is even slower, and still was somehow not satisfying.
Thanks go to Kevin Atkinson and his filter "SmartDecimate", which comes to a rescue here. In fact, I used SmartDecimate for the very first time. Since this filter does field matching + decimating in one step, it is much better suited for the given problem. Or, at least, it was easier for me to get it working like it should.
Briefly, now the cleaned stream is UnBobbed again, and fed to SmartDecimate (while providing the cleaned bobbed stream as bob source to the filter, but that's detail). SmartDecimate then fiddles out a given number of unique fields (according to decimation ratio, here: 24/50), searches for matching fields for these, and uses the bob source for those fields without match.

Et voilà: it works.


So far, so good. The results are quite promising, but there are several

Possible improvements

1. Detecting blends

As stated above, the edge mask of a blended frame appears darker, but lets have a closer look.
What I am doing is simply taking "AverageLuma" through Conditional Filtering, and compare that. It works quite well, but it is not correct:
- a clean frame has higher contrast edges, leading to brighter lines in the edgemask, but
- a blended frame has lower contrast on edges, but it also might have *more* edges than a corresponding clean frame, because of the overlaying of a second frame.
Therefore, it is possible that the sum of "darker, but more" edge lines is equal, or even higher than the sum of "fewer, but brighter" edge lines, which obviously makes detection by AverageLuma fail. What would be needed is something like "average distance from overall average", or a similar stochastic measurement. Comparing the distribution of lightness of the edge masks, a blended frame will have more samples in medium range, less in the higher range. A clean frame will produce more samples in the higher range, and less in the medium range.

I don't see an easy solution to do this with AviSynth's current instruction set - or maybe I just forgot about something?
Moreover, I think it could be quite beneficial not to compare the whole frames at once, but to implement a "windowed" comparison. Correcly done, this should help in ignoring static parts of the frames, and to detect blended frames even if the blending occurs only in small areas like mouth movement.
This would even be possible through scripting somehow, but it would get awfully slow for sure.


2. Replacing blends

The current procedure is really dumb. The problems to solve are
- preferably not letting stay single frames alone
- by all means, avoid the creation of 4-frame-dups.

Imagine the bobbed source has a sequence
...- A - AB - B - B - BC - C - CD - ...
If the algorithm should decide to replace "AB" with "next", and "BC" and "CD" with "previous", we would get
...- A - B - B - B - B - C - C -...
This is very bad, since four dup's in a row are very likely to come out as "B-B" after decimation. A jerk.
We should replace either "AB" with "previous", or "BC" with "next". But which is better ... ?

Or imagine this sequence:
...- A - AB - B - C - CD - D -...
If (worst case!) "AB" gets "previous" and "CD" gets "next", we get
...- A - A - B - C - D - D -...
Also very, very bad. We should avoid to let stay even one 'single' frame, because it would be in danger to be dropped by decimation.
The last example shows it can easily happen to produce two 'singles' in a row, where one of them will be dropped by decimation almost for sure. Another jerk.
Although SmartDecimate seems to handle the stream with very reasonable intelligence, we should definetely care to create an even distribution through the replacing process!

Interlacing experts - Coding experts - Hello !?! - I'm neither, and my brain is smoking ...


So far the explanation.

You will notice that several things in the script's could be improved immediately. For example, the script would run much quicker if Kurosu's MaskTools were used for all the edge related stuff ... - should be not too hard, as long as there's no nifty little memory leak in them
There are also outcommented fragments left of some more things I played with. I hope it's readable, comments are a little sparse ...

Try it, play with it, tell me something.

Thanks for your time


- Didée