Motion filter : +1st MC FPS-change script for real usage

[Joel Cairo]

castellandw:

With regard to the Doctor Who restorations, there should be no need to for the Restoration Team to recover frames from interlaced content... the episodes that they treat with motion conversion are derived from 25 fps film telerecordings, so it's my understanding that they just have special video transfers made, clean up the image, and work directly from those.

-Kevin

[castellandw]

Kevin, what are you talking about? I'm talking about regular broadcast standards conversion process (aka PAL-to-NTSC or NTSC-to-PAL) in general with the Snell Converter that the Restoration Team uses. This is nothing to do with recovering the original interlaced frames from film telerecordings like the actual VIDFIRE process.

[Joel Cairo]

Ahh-- my mistake... I lost track of the conversation.

Sorry for the confusion!

-Kevin

[mg262]

Oh yeah, the Doctor Who restoration team video engineers said the motion compensation with phase correlation was done on a full frame meaning all the 50-60 fps full progressive frames were deinterlaced from the 25-30 fps interlaced frames.

Thank you, that is worth knowing...

So you have to make the frames progressive for motion compensation.

... but that doesn't follow from it. It just means that they/the equipment they had did it that way.

[castellandw]

Well, I hope I'm explaining this right. They did mention interpolating to full frame if I remember correctly (God, I wish they didn't delete their old message threads.), but you can ask them too.

[mg262]

castellandw,

Don't worry, that's clear enough. As I said, I had the impression that that was the standard method. I just want to suggest a different way of doing it...

[castellandw]

scharfis_brain mentioned about bobbing before motion compensation took place, and it got me wondering, shouldn't we use better bobbers than just smart bob such as leakkernelbob?

[Mug Funky]

it's a chicken and egg thing.

to do a motion-compensated deinterlace (good bob) we need to start with a good bob.

[scharfis_brain]

That's why I cannot wait tritical releasing his new Tdeint2!

[mg262]

Mug Funky, scharfis,

Why do you need to start with a normal bob?

The only thing that comes to my mind is the case where you have vertical motion of 1 (or 3 or 5, etc) original scan lines per field, i.e. the case where field 0 and field 1 match pixel for pixel. In this case, you either have to look further afield temporally (which may not help and increases the risk from incorrect motion prediction), or (safer) use pure spatial upconversion.

Cf again the paper Fizick pointed us to:
http://www.ics.ele.tue.nl/~dehaan/pdf/111_IVCP_ZHAO

There is a motion-compensated deinterlace which starts ab initio.

[castellandw]

@msg262, has anyone actually implemented the GST algorithm as a plugin for AVIsynth or at least tested out the GST algorithm to see if it actually has good results? Also, do you think I should go to the Doctor Who restoration team to see if they know what kind of deinterlace technique is used before motion compensation starts in the Snell converters they use?

[MfA]

Originally Posted by mg262

if you are applying half pixel compensation, direct interlaced->interlaced FPS conversion is almost the same problem as progressive->progressive conversion, and the methods extend directly.

Performing interpolation for inherently aliased fields works even worse than for progressive frames though.

[mg262]

do you think I should go to the Doctor Who restoration team to see if they know what kind of deinterlace technique is used before motion compensation starts in the Snell converters they use?

If you are curious about it, go ahead... but it's not necessary for this thread. The strength of AVISynth is that tools can be mixed and matched -- you are free to try any deinterlacer with this or any any motion compensation. If you find convincing evidence that a particular deinterlacer is very effective (for a particular task), you can present it clearly in a new thread and hope someone is interested enough to build one.

It is certainly worthwhile to know what methods are used in hardware -- if the explanation is sufficiently detailed to be replicable. But I think you should not assume that the methods used in hardware, even high-end hardware, are the "best" methods around. (The same caveat should be placed on academic material.) In the end, to choose an approach for a given task, you need to immerse yourself in the material until you understand the strengths and weaknesses of each method well enough to pick -- or construct -- the appropriate one for the task.

@msg262, has anyone actually implemented the GST algorithm as a plugin for AVIsynth or at least tested out the GST algorithm to see if it actually has good results?

I don't think so, but I couldn't tell you for sure... you could

On this, on phase correlation, things are only implemented when someone has an interest in implementing them -- which typically means an interest in the specific tasks for which they are useful. I'm going to steal some well chosen words:

Originally Posted by Fizick

6. Some time ago i also waited when somebody will wrote filter with advanced algo.
... take C compiler, and become the talented progammer !

[mg262]

MfA, inherently aliased fields: do you mean aliased in the sense that they are aliases of each other or aliased in the sense of anti-aliasing? (I think you mean the former, but if it's the latter I'll have to think some more...)

Update on progress: I put in the half-pixel assembly, and then spent a fair while profiling, optimising, trying a lot of different things to squeeze out more speed. I will probably release a new version after putting in FindReverseMotion and ReverseCompensate functions. After that, real life will soak up most of my time, but I'm thinking of implementing one of these two (in C to start with):
-- simple (bilinear) mesh warping
-- simple (no clever occlusion handling) FPS conversion.

Simple because I like to have a baseline to test more advanced features against. Preferences?

[MfA]

The frequency of the underlying signal is unchanged compared to progressive, but the vertical sampling frequency is halfed ... so more aliasing.

[castellandw]

Originally Posted by msg262

On this, on phase correlation, things are only implemented when someone has an interest in implementing them -- which typically means an interest in the specific tasks for which they are useful.

@msg262, so does this mean phase correlation really isn't important toward making a better motion compensation engine for right now? I'm not trying to go away from the fact that I don't want to code a plugin that does phase correlation because I'm a lesser-experienced programmer, but for now, you guys pretty much seem to have more experience in this and are much more advanced programmers in this than me. Of course, the reason I wanted to ask about what deinterlacer that the Snell converter uses for motion compensation was because I thought which deinterlacer to use did matter when motion compensation is applied to the frame.

[mg262]

Got it. I would certainly agree that

50p --MC-FPS-> 60p --discard fields-> 30i

Beats

50p --discard fields-> 25i --MC-FPS-> 30i

But our case is slightly different. Our underlying material is 25i , and the only way we can get 50p back out of it is by bobbing, i.e. by interpolating values. Now IMO the MC-FPS unit is just as capable of interpolating values as a bobber. In fact, I would argue that it is better placed because:

a) Suppose we are interpolating midway between fields B and C in this configuration:
A B C D
Now a bobber will use parts of C to fill in bits of B, and then the FPS unit will use those filled in bits to create the intermediate frame -- but really this is giving us no more information than it already gets from considering C directly.

b) The bobber will also use A to fill in B -- but the FPS unit can do this if it chooses to, simply by expanding the temporal radius. You could argue that this introduces extra inaccuracy (pulling information across 1.5 fields distance rather than 1 field distance), but
i) the information is being pulled across 1.5 fields anyway; it's just happening in two hops
ii) the bobber itself is always pulling information across 1 field's distance, whereas the MC-FPS unit can get away with looking across 0.5 fields distance.

Also note that at least some motion compensated deinterlacing methods, like the GST, can be directly applied to FPS change -- we obtain motion compensated samples from previous and next fields, apply the generalised sampling theorem to reconstruct a curve, and read the values we want off the curve. (I think there are better methods available, which allow things like e.g. lower weighting of information pulled across a larger temporal distance... but that's another matter.)

[mg262]

castellandw,

does this mean phase correlation really isn't important toward making a better motion compensation engine

I honestly don't know. I'm not confident enough to call this one. If you want my gut feeling... I have a tendency to distrust maths-based methods unless I can see (theoretically or empirically) that they model the problem domain well. (IMO, there is a visible tendency for academics who are not mathematicians per se to be impressed by maths just because it looks hard.) A priori, you can also see that phase correlation will tend to mess up on small objects.

The only way to really to which method is best is to implement both methods and refine them with all the tweaks everyone has thought of and then compare them on real applications like FPS conversion. So, having implemented one method, we need to refine it and include applications before any comparison is even possible.

[scharfis_brain]

60p --MC-FPS-> 60p --discard fields-> 30i

is it meant this way or is it just a typo?

Did you mean:
50p --MC-FPS-> 60p --discard fields-> 30i

or

60p --MC-FPS-> 50p --discard fields-> 50i

[mg262]

Oops... completely right, scharfis. I fixed it. Thank you.