TDeint and TIVTC

[manono]

Originally Posted by Terranigma

I was wonderin' if i should deinterlace first, then decimate, or decimate then deinterlace.

Neither; field match followed by decimation (TFM().TDecimate()), just as Leak explained.

[Leak]

Originally Posted by Terranigma

Actually, it's ntsc video 30000/1001 that I ivtc'ed and decimated to film ensuring that each frame is different. I was wonderin' if i should deinterlace first, then decimate, or decimate then deinterlace.

Well, what you *shouldn't* do in any case is to unconditionally deinterlace the result of fieldmatching and decimation, as that will only worsen the image quality

That's why TFM has postprocessing options to actually deinterlace combed frames that resisted field matching. Try to tweak those options to eliminate any interlaced remains.

[Terranigma]

Thanks Leak, manono.

[ChiDragon]

tritical, I've found another weird bug...

Try this:

BlankClip(pixel_type="YV12",length=11,fps_denominator=1001,fps=30000)
TFM(input="test1.txt")
TDecimate(input="test2.txt",display=true)

With test1.txt:

#TFM v1.0.1 by tritical
5 p - [60]
6 c - [118]
7 p - [6]
8 c - [17]
9 p - [3]

test2.txt:

#TDecimate v1.0.1 by tritical
4 21320 546816
5 7194 169921
6 23761 599224
7 5737 181105
8 13369 452464
9 2606 170827

Frame 5, with a metric of 2.12 is dropped instead of frame 9 (0.77). If TDecimate is set to mode = 1 instead of 0, or hint = false instead of true, the correct frame is dropped!

[tritical]

@ChiDragon
It happens due to one of the 'special handling' functions mode 0 uses over plain "drop the frame with the lowest metric mode." When match information from tfm is available, tdecimate will detect match duplicates and one of the things it then looks for are standalone duplicates (a frame that is a match duplicate and neither of its neighbor frames are). It then goes through the cycle and determines which of these standalone duplicates to decimate (assuming there is more than 1) based on the sum of the next frame's difference metric and the previous frame's difference metric... whichever standalone has the largest sum is chosen. The only extra check is that the selected frame has at least the n'th lowest metric in the cycle, where n is the number of standalone duplicates. In this case, there are 3 standalone duplicates, frame 5 has the largest sum, and it has the 3rd lowest metric in the cycle... so it gets decimated.

There should probably be a check related to the selected frames metric (either hard, or relative to the lowest metric in the cycle) before it can be chosen for decimation. Including the current frames metric in the decision about which standalone duplicate to choose would probably also be a good idea. Looking at the code, it seems I had included a variable to keep track of the selected frames metric, but it never gets used for anything . Anways, I'll take a look at this later this week when I should have some free time again.

[Leak]

Hi tritical,

in case you're interested - I've written a patch for ffdshow's AviSynth filter (my latest build is here, more info here) that allows you to use TFM/TDecimate (and of course a slew of other filters) on the fly during playback. Maybe you (or whoever else reads this, of course) would like to give it a try?

[tritical]

Awesome, I quickly tried it out on my laptop (the ffdshow version on here was from 2005 ) and it worked great. If anyone has problems with tivtc using too much cpu, changing the following settings should speed things up without sacrificing much:

tfm(slow=0,mchroma=false,pp=2 or 3)
tdecimate(chroma=false)

While I was testing, the cpu load went to 100% briefly on a few complex scenes (my laptop has a 1.6Ghz pentium M) and averaged ~60-65%. With the changes mentioned above it never broke 85 and averaged ~50-55%.

Also, there might be the smallest bit of jitter due to tdecimate. Currently, while it is returning frames for the current cycle it computes the metrics for the next cycle one frame at a time (it does that so that it doesn't have to request and compute metrics for all 5 frames at the beginning of each cycle). However, that only takes care of 4 frames and at the beginning of each cycle 2 frames will have to be requested and computed. I couldn't notice it though. Probably tdecimate is fast enough that it doesn't matter (even on my laptop it runs > 1000fps, assuming no blending is required).

[Leak]

Originally Posted by tritical

Awesome, I quickly tried it out on my laptop (the ffdshow version on here was from 2005 ) and it worked great.

It never really had a chance of working before as the old AviSynth filter implementation in ffdshow always returned the same frame no matter which frame got requested by AviSynth - so I set out to fix that...

While I was testing, the cpu load went to 100% briefly on a few complex scenes (my laptop has a 1.6Ghz pentium M) and averaged ~60-65%. With the changes mentioned above it never broke 85 and averaged ~50-55%.

It hardly goes above 30% on my Athlon 64 X2 4400+, but since that's a dual-core machine it means about 60% of one core is used. But hey - at least I can now watch my anime DVDs like they were meant to be watched, and using my tiny projector (don't laugh) to boot...

Now my patch only needs to get checked into the ffdshow-tryouts SVN...

[Chainmax]

tritical, here's a sample of a scene that's been giving me trouble:

http://www.bestsharing.com/files/HL7...muxed.m2v.html

The deinterlacing portion of the script is this:

AssumeTFF()
Interp = SeparateFields().SelectEven().EEDI2(field=1)
Deinted=TDeint(order=1,field=1,type=1,edeint=Interp)
TFM(d2v="C:\SimpS7\e3\Epi3.d2v",order=1,mode=6,PP=7,slow=2,mChroma=false,Clip2=Deinted,micmatching=3)
TDecimate(mode=1)

which I think leaves some residual combing but I'm not sure wether it's that or the picture itself is just fux0red. Could you please take a look at it?

[Terranigma]

Could someone tell me, what are the advantages as to using/disabling mchroma?

[Leak]

Originally Posted by Terranigma

Could someone tell me, what are the advantages as to using/disabling mchroma?

Well, this controls whether TFM takes the color information into account when matching fields, in addition to the luma information.

There's material out there where the chroma has been botched (rainbows, color delayed a field or three vs. luma, plus other stuff I don't even want to imagine) so disabling it for matching will probably give better results in those cases. That, and it of course takes some extra time for the extra checks.

Of course, if you have a scene where the luma stays the same and only the chroma changes the outcome will be pretty random - but that's rather unusual for normal footage.

[foxyshadis]

Originally Posted by Chainmax

tritical, here's a sample of a scene that's been giving me trouble:

http://www.bestsharing.com/files/HL7...muxed.m2v.html

I think it's pretty safe to say it's not TIVTC/TDeint's fault... output of separatefields.

BTW, tritical, did you have a chance to test out my newer compile? I think it's worth replacing the old one, just because apparently it crashes for older cpus.

[Chainmax]

Originally Posted by foxyshadis

I think it's pretty safe to say it's not TIVTC/TDeint's fault... output of separatefields.
...

I had already seen those scenes, but take a look at this:


Result:


Corresponding frame with AssumeTFF().SeparateFields():


Previous frame:


So maybe some parts could be slightly improved.

[rottweily]

I have serious trouble in deinterlacing this tennis sequence:
LineNastyness-Interlaced.mkv : http://www.mediafire.com/?5nkmnk2nzyd
LineNastyness-TDeint.mkv : http://www.mediafire.com/?ci2jiitbbmi
LineNastyness-TDeint+EEDI2.mkv : http://www.mediafire.com/?amdxqmymie2
LineNastyness-TDeint-Type4.mkv : http://www.mediafire.com/?0dyzdq3nv2n
LineNastyness-TDeint-Type5.mkv : http://www.mediafire.com/?djcyjd0enm3

You can't miss the artifacts.
I've always used TDeint with all the matches I've encoded and till now never noticed any obvious artifacts.
As you can see I've also tried using EEDI2, but it didn't help.
Any simple solution?

Edit: I already see that I have an outdated version of TDeint, time to upgrade. Already tested it, has the same problem.
The new blend interpolation already works a lot better type=4 or 5, but still not ideal.

[tritical]

Sorry for the lack of replies, I just haven't had much free time lately. Next week is spring break though (Yes, we have spring break 2 weeks after everyone else).

@rottweily
The links wouldn't work for me, but if you set mthreshL=0,mthreshC=0 do the artifacts go away? If so (I am guessing they will), then it's just standard motion-adaptive artifacts, and there isn't much you can really tweak as far as TDeint to make it go away other than mthreshL/mthreshC/mtnmode. You might try running vinverse() afterwards, but that might not help much if the artifacts are a lot stronger than normal residual combing. It might be worth giving securebob() a try.

@Chainmax
I must be blind, what exactly am I suppose to be looking at in the images you posted?

@foxyshadis
I haven't, but I'll try it soon. The link to your version that I have on my website just links to Eedi2mt.zip on your webspace so anyone who gets it from there should get the latest version. I need to put a link to it in the first page of this thread also.

[foxyshadis]

Oops, I never noticed that. Hah!

I think chainmax refers to this:

But that comes from:


There's a lot of large and small artifacts just in that short piece posted, whomever transferred it really did a great job of mangling it.

[Didée]

@ rottweily (& tritical not having seen the clips)

The problem here is just the usual threshold disease ("up to x is static, bigger than x is motion").
Remember the look of the scene when a tennis match is filmed from the usual bird's view camera position. When the camera is perfectly static, then the "deinterlaced" lines are 100% static. But heaven forbid that the camera starts moving even so slightly ... then hell breaks loose. (The "deinterlacing hell".)
In that regard, it's a worst-case scenario ... the lines are of very high contrast (white lines with a small black halo). Because of the high contrast, even with mini-minimal motion the usual motion thresholds are exceeded immediately. And because of the high white-to-black contrast, the wobbling seems so extreme and annoying.

SecureDeint won't help here ... it's more "anxious" about motion, hence it will produce even more wobbling. (Probably, didn't try it.)

Bigger thresholds won't help either, you'll loose the ball to blending land.

What would (and will) help is MCBob's smart thresholding, which is adaptive to local complexity.
(Have been wondering that there were no requests to combine this smart-thresholding with some not-mo'comped deinterlacing, à la MVBob-->SecureDeint ...)

***

BTW, the idea behind and justification of the smart thresholding in MCBob is very easy: at each pixel's location, there is a certain min-max value of the neighborhood - it's small for flat areas, big for detail areas. Now: If we consider up to 1/4 pel motion as "static", and 1/2 pel motion as "motion", then changes up to (max-min)/4 (+N) for that pixel are considered static, pixel changes bigger than (max-min)/2 (+N) are considered "motion", and inbetween is considered inbetween. (N is a small tolerance value to compensate for general flicker.)

One example with numbers: on a black-white transition of some high-contrast detail, the min-max-span is, say, (max-min)=200. Now, if there is some minor motion of only 1/4 pel (which is neglectable for the matter of deinterlacing), the pixel might change by already +/- 50. Fifty! With (fixed) motion thresholds being usable mostly/only in the range 2~10, it's no wonder that deinterlacing usually produces shimmering, flickering and wobbling ...

Hard thresholding obviously fails miserably here.

Smart thresholding is a simple idea, works out, and free to use.

[Chainmax]

Originally Posted by Didée

...
What would (and will) help is MCBob's smart thresholding, which is adaptive to local complexity.
(Have been wondering that there were no requests to combine this smart-thresholding with some not-mo'comped deinterlacing, à la MVBob-->SecureDeint ...)
...

Well, McBob seems to be the best available deinterlacer/bobber we have right now, and people who use it are willing to sacrifice speed for quality. Therefore, I don't see much use in trying to enhance MVBob as it's already very slow and apparently not as good. SecureDeint might be a better candidate for this enhancement, but my favorite candidate would be Tdeint as it's extremely fast in comparison and according to WorBry's tests it's pretty close to the big three.

[rottweily]

Originally Posted by Didée

What would (and will) help is MCBob's smart thresholding, which is adaptive to local complexity.
(Have been wondering that there were no requests to combine this smart-thresholding with some not-mo'comped deinterlacing, à la MVBob-->SecureDeint ...)

Didn't help enough, I will keep the interlaced version.

[tritical]

I might add the smart thresholding idea to TDeint. When you talk about the min/max of the neighborhood do you mean within the same field or both fields? say the x's are the top field pixels, the o's are the bottom field pixels, and we're trying to create the missing pixel 'c'. The neighborhood for determining the min/max for the motion threshold for 'c' would consist of which pixels?

  frame 0       frame 1      frame 2
 xxxxx         xxxxx        xxxxx
       ooooo        ooooo        ooooo
 xxxxx         xxxxx        xxxxx
       ooooo     c  ooooo        ooooo
 xxxxx         xxxxx        xxxxx
       ooooo        ooooo        ooooo
 xxxxx         xxxxx        xxxxx

Maybe you could just copy the above diagram and indicate the needed pixels by making them bold. I was thinking it would be the 3 x's above and below 'c', but want to make sure.

EDIT: I've been looking at this a little more and I don't quite understand why the threshold is based on the range (max-min) of the neighborhood. Let's say we have the following 9 pixel neighborhood from a single field:

abc
def
ghi

if we allow 1/4 pel motion for pixel 'e' and assume linear interpolation will give an accurate result for this motion. Then the possible values for the new 'e', assuming only vertical or horizontal motion for the moment, would be:

0.75*e+0.25*b
0.75*e+0.25*f
0.75*e+0.25*h
0.75*e+0.25*d

Which would mean the threshold for 'e' should be the max of:

abs(e-(0.75*e+0.25*b))
abs(e-(0.75*e+0.25*f))
abs(e-(0.75*e+0.25*h))
abs(e-(0.75*e+0.25*d))

i.e.

max(abs(e-b),abs(e-h),abs(e-d),abs(e-f))*0.25

and for half pixel motion it would be:

max(abs(e-b),abs(e-h),abs(e-d),abs(e-f))*0.5

That reasoning can then be extended to simultaneous motion in both vertical/horizontal directions.