Apply a deinterlacer on a moving segment

[sirt]

Hi,

Well I have an "educative" problem. Unfortunately, I don't have any source to illustrate my problem. So let's have a theoretical talk.

Imagine I have some video flux I would like to deinterlace with Tdeint (let's say Tdeint() would be the simple command line to apply it). My video would be formed of a black rectangular window and inside let's have a rectangular window with some people dancing for example. But I insist this window must be included in the black one and it moves when playing the video.

So here is my question : how could I only deinterlace the moving part without touching the rest (black pixels around) ? If it would be a static screen, I could have set up some mask so as to deinterlace only the part needed in the video but there this part is also moving from a frame to another. Then, I would like to know if there would be a way to detect black pixels around while moving from a frame to another, for example from frames 10 to 20. In C, one may use a for loop in order to apply the deinterlacer for pixels different from 0 (black) but how would something like that be done in avisynth language ?

[poisondeathray]

Practically speaking - if the large rectangle is pure black, it won't matter if it's deinterlaced

I don't know if I am understanding your description correctly, but you can still use a mask with masktools - eg. a luma mask - to exclude the black pixels (or a given range of "dark" pixels, say 0-30) as a garbage matte

Presumably there will be some overlap "dark" or black pixels in the small rectangle region of interest, so maybe you can use some mask refinement and combining techniques like mt_hysteresis to exclude those

[LoRd_MuldeR]

What is the purpose of deinterlacing only the "moving" part?

Applying the deinterlacer also on the black broders might be waste of CPU cycles, but it's certainly much easier to do. Also, if you wish to apply the deinterlacer only on the "moving" part, you will have to detect the location of the "moving" part, crop away the black borders, apply the deinterlacer only on the remaining part and finally re-add the black borders. In the end, this might cost more CPU cycles overall, unless you use a very slow deinterlcer, maybe.

Last but not least, a "smart" deinterlacer uses detection of "static" areas and areas with "motion". So, with such deinterlacer, the "static" areas will be passed trough unmodified anyway.

[sirt]

Lord, there is no particular purpose intended in so far as I don't want to do that. I'm just wondering how to do it. Then how would you detect the moving part ? Would you have an example of code you may use for an avi flux test.avi for example ?

poisondeathray, thank you and, indeed, you have understood what I asked for. The problem will be the overlapping at boarders, especially if I don't deinterlace each consecutive frame - I mean a difference at boarders with black pixels will be certainly be noticeable from a deinterlaced to a non deinterlaced frame. Unfortunately I am not an expert at all with masks, so how would write such a code ?

[LoRd_MuldeR]

It's pretty much impossible to give an advice as long as your actual goal isn't clear.

As said before, a modern "smart" deinterlacer will probably detect the black borders as "static" area anyway and thus pass trough that area unmodified.

Not that it would matter with a solid black area, like you were talking about...

[poisondeathray]

poisondeathray, thank you and, indeed, you have understood what I asked for. The problem will be the overlapping at boarders, especially if I don't deinterlace each consecutive frame - I mean a difference at boarders with black pixels will be certainly be noticeable from a deinterlaced to a non deinterlaced frame. Unfortunately I am not an expert at all with masks, so how would write such a code ?


The most simplest example of a luma mask is using mt_binarize

Areas in white are affected through the mask ,areas in black are unaffected

mt_binarize(30) will make 0-30 "black", 31-255 "white" . You can use upper=true to invert the mask 0-30 "white", 31-255 "black"

Often you need to blend the mask borders, other wise too sharp of a transition makes it look out of place

You can use mt_merge() or overlay() so the filtered section is only appled through the mask (areas in 100% white) . Variable shades of grey are partially affected proportional to their grey scale values

eg.




original = WhateverSource()

original
mt_binarize(30) #0-30 black, 31-255 white
mymask=last

original
yourfilter()
filtered=last

mt_merge(original, filtered, mymask, luma=true)



replace "yourfilter() " with whatever filter or deinterlacer you choose to use

There is another function called Lumamask() (search for it) , where you can set black and white points directly b=, w=

There is a lot more to manipulating masks than that, but that's the basic overlay using a luma mask. It's difficult to make better suggestions without a concrete example or better description

If you need to apply to specific frames, you can combine with Trim() or ReplaceFramesSimple() with a filtered/non filtered version

You can refine and combine masks on other parameters besides Y' . e.g. Green screen is really a hue/saturation mask, there are edge derived masks, motion masks, etc... because invariably the problem is getting an accurate mask (in your example, there might be some "overlap" pixels between 0-30 in the region of interest)




Lord Mulder already mentioned it, but the good deinterlacers already use motion compensation, and some use comb detection (so only combed areas are filtered). eg. You can also use Tdeint+TMM (the TMM is a motion mask) and use QTGMC as the deinterlacer. So you could even combine one of those with a user defined mask

[sirt]

Thank you for those detailed explanations. I'm aware there are smart deinterlacers. I may have some example of source but it is way more complicated : there is a ballon that moves from a frame to another. You can iVTC this video and try to work on the problematic frames afterwards.

So the goal would be to track the ballon and to apply some deinterlacer - let's say QTGMC - but just on the ball. Of course, you would say it is smart enough to deinterlace almost what's needed but as I said it's purely for educative purpose.

Precision : this sample comes from a place I worked two years ago.

http://www.sendspace.com/file/3a9ggh

[Guest]

In that clip the video appears to be telecined as you say but the ball is rendered progressively on top at 29.97. After IVTC for the background, the ball gets matched wrong. There is no good solution for this that I know of. Postprocessing with a deinterlacer will make the ball a blurred mess, but that may be the best you can do.

You can forget about trying to track the ball and deinterlace just that. That would be ridiculously difficult and would give you nothing better than a good post-processor could give you.

And even if you could seperate out the ball movement and keep it progressive, then IVTC the background with the idea of recombining them, you would have the problem of differing frame rates and would have to accept making either the background or the ball jerky.

[poisondeathray]

1) neuron is right, content is 23.976p, but graphics are 29.97p

But even if you addressed that issue:

2) The blurred edges of the "ball" during deformation and bouncing makes it difficult to separate effectively with any method, even non-avisynth methods

3) Crossing over objects (foreground vs. background) always poses a problem to achieve complete separation . (e.g. a motion mask will not define the boundaries of runners legs vs. the "ball" when it crosses over)

4) The "usual" methods to achieve separation wont work here e.g. "brown skin" and "brown ball" are too close for hue or satuation methods

The only "real" solution is rotoscoping it manually, after you address issue #1 somehow

[sirt]

neuron2 and poisondeathray, thanks for your answer. In fact, what I would like to do (but I insist, it's for my education) is to detect the ball or at least a rectangle that fits it. Then apply the deinterlacer on this rectangle. So, I could choose a frame then try to crop so as to find the rectangle manually. But, after that, the rectangle (with the ball) will move from a frame to the next one, so the idea would be to apply (automatically) the deinterlacer on the same rectangle in the next frame. Do you think there a way to do that e.g. to "track" the rectangle from a frame to the next one and apply the deinterlacer ? To sum up, that is the problem I would like to solve.

Of course, if the ball could be detected perfectly, it would be better.

[Guest]

IMHO, what you ask for makes no sense, for the reasons you've been given, and therefore, you will not motivate anyone to get interested in your approach. But who knows, maybe Merlin the Magician will drop in. :)

[poisondeathray]

neuron2 and poisondeathray, thanks for your answer. In fact, what I would like to do (but I insist, it's for my education) is to detect the ball or at least a rectangle that fits it. Then apply the deinterlacer on this rectangle. So, I could choose a frame then try to crop so as to find the rectangle manually. But, after that, the rectangle (with the ball) will move from a frame to the next one, so the idea would be to apply (automatically) the deinterlacer on the same rectangle in the next frame. Do you think there a way to do that e.g. to "track" the rectangle from a frame to the next one and apply the deinterlacer ? To sum up, that is the problem I would like to solve.

Of course, if the ball could be detected perfectly, it would be better.




There are ways, with non avisynth tools - but there is manual work and correction involved . There are motion trackers commonly used in the FX industry e.g. mocha pro, nukex, after effects. They will partially fail on "automatic" mode because of the reasons above - especially because of the blurring and deformation and rotation (it's no longer registered as the same "object" because the properties have changed) . And when objects cross over another, object borders are no longer clear - there is no way to distinguish what is what. So you will need some manual intervention to assist the motion tracking

You can get a very "rough" mask (with contamination, missing some spots) using mmask (motionmask) from mvtools2 . As mentioned above, the eternal problem is always getting complete perfect separation with masks

Maybe this will get you started, and when "Merlin" :) pops in , he might suggest other ways to refine the mask with other filters and techniques . eg You might be able to play with the mvtools2 vectors settings

I did not address the 23.976p vs. 29.97p issue here, just left it as is, and overlayed a "red" clip to visualize areas where the filter would be applied



orig=MPEG2Source("aaa.d2v", cpu=0)

orig
vectors = MSuper().MAnalyse(isb = false)
MMask(vectors, kind=0, ml=10)
mymask=last

blankclip(orig, color=color_red)
ovr=last

mt_merge(orig, ovr, mymask, luma=true)
red=last

orig
crop(0,0,0,-214,true)
cropped=last

overlay(red, cropped)