TDeint and TIVTC

[tritical]

You can upload the clip to my ftp if you haven't already found someplace else:

ip: 68.119.245.113
port: 17252
user/pass: upload/upload

as long as it's not larger than 2 or 3 GB's, any size is fine . I don't think my email will take attachments larger than about 5 MB's.

[rig_veda]

Done uploading

[berrinam]

I've done a fair bit of searching and been unable to find anything said on this topic. I hope I'm not hijacking this thread by asking this here.

1. Is there a big difference between DeComb's combed frame detection and TIVTC's one?
2. Is TIVTC's more accurate?
3. Are the thresholds normalized to be consistent with each other?

I am asking because I am working interlace detection for MeGUI, and at the moment I'm using DeComb's IsCombed() function. I wonder if it would perhaps give better results to change to IsCombedTIVTC() for this.

[tritical]

@rig_veda
I looked at the clip and I was able to reproduce your results if I seeked to frame 268. If I ran up to frame 268 linearly from the beginning both scripts produce the same result.

Involved explanation of what happens
The difference is w/o ttempsmooth seeking to frame 268 causes only the cycle starting at frame 335, plus the previous (330) and next (335) cycles to be evaluated. However, with ttempsmooth in there to request extra previous frames, the cycle at 325 is also evaluated. The blend decision in mode 0 with hybrid=0 is based on seeing match dups + lowest metric'd frames at the right interval that it looks like an ivtc pattern change that produced two dups in one cycle. In this case the match dup in the previous cycle appears at pos 0 (frame 330), but without evaluating the cycle prior to that (starting at frame 225) it won't know the match used for frame 329 and therefore can't tell that frame 330 is a match dup. So with ttempsmooth it can tell frame 330 is match dup which triggers the detection and without it it can't so it doesn't trigger.

So the solutions are:
1.) don't seek
2.) if you don't want the blending for such cases when hybrid=0 then use "noblend=true" in tdecimate()
3.) Use "d2v=source.d2v" in tfm. In this case your source is almost entirely flagged (94% reported by tfm). With the d2v option enabled it doesn't try to blend in that cycle and picks up the correct ivtc pattern... plus it is faster .

A problem though is that the following script should support seeking (assuming you use the output parameters on a previous pass to generate input files):

mpeg2source()
tfm(input="tfm.txt")
tdecimate(input="tdec.txt",tfmIn="tfm.txt")

but I seem to have made it so you can't use a tfmIn file with hybrid=0 for some reason... I'll have to fix that. I'm also considering making noblend default to false since most people would rather just keep the extra dup than see a blend in such cases.

[tritical]

@berrinam
They are mostly the same except that tivtc's function uses overlapping windows (the normal set plus 3 others offset at 0.5*blockx in the x direction, 0.5*blocky in the y direction, and 0.5*blockx in the x + 0.5*blocky in the y, the window size is variable (blockx, blocky) (fixed at 4x8 in iscombed), the window threshold amount is variable (MI) (fixed at 15 in iscombed), chroma parameter is adjustable (iscombed uses true), and it uses a different per-pixel combing metric.

Assume we have 5 pixels vertically and we want to know if 'c' is combed or not:

a
b
c
d
e

IsCombed's per-pixel metric:

val = (b - c) * (d - c);
if (val > threshold*threshold) it's combed;

IsCombedTIVTC's per-pixel metric:

d1 = c - b;
d2 = c - d;
if ((d1 > cthresh && d2 > cthresh) || (d1 < -cthresh && d2 < -cthresh))
{
   if (abs(a+4*c+e-3*(b+d)) > cthresh*6) it's combed;
}

That pseudo code is not exactly how the metrics are implemented code wise, but is equivalent to what they do. So tivtc's cthresh and the threshold of iscombed() are both on a 0-255 scale, but will give differing results. IsCombed's threshold defaults to 20 and IsCombedTIVTC's cthresh defaults to 9... so it would seem to be more sensitive. However, tivtc's metric is tougher to trigger and it uses a larger default window size (16x16 vs 4x8), has to see more combed pixels (80 vs 15), and chroma defaults to false.

[scharfis_brain]

tritical, I have a question (or suggestion) regarding fieldmatching principles.

I want to discuss the low-motion (also no-motion) case.

The most fieldmatchers I know of will prefer a blind match on them even if the underlying video is true interlaced stuff thus risking the well known mouth combing.

How about introducing a threshold to identify these low/no motion scenes and the apply a motion adaptive deinterlacer on them?

The problem is pretty obvious with tdeint(mode=1,tryweave=true, full=false). -low motion interlaced scenes always are fieldmatched instead of being deinterlaced.

EDIT: IMO smartdecimate does something like this, in past I always wondered, why it started bobbing in low/no-motion scenes...

I mean fieldmatching should only be applied, if the algorithm is pretty sure that two fields are derived from one frame.
This is only the case with a reasonable amount of motion.

Of course, on can continue a running pattern (let the fieldmatcher adapt to a pattern), if the motion gets low or stops, but after a scene change it needs to reset this immediatelly.

[tritical]

@scharfis
Just a warning, I did a lot of rambling and thinking while writing this reply so it kind of jumps arounds. Also I was not sure whether you were specifically talking about TDeint or also TIVTC.

I mean fieldmatching should only be applied, if the algorithm is pretty sure that two fields are derived from one frame.
This is only the case with a reasonable amount of motion.

I think it depends on the material type. If you know the source is primarily telecined then I don't think having it use a motion adaptive deinterlacer in low motion areas is preferable because even the best ones aren't perfect and will give inferior results compared to pure field matching. However, if the source is primarily interlaced then it would probably give better results. I see the problem as one of combed frame detection and not field matching (which are two completely separate processes in TIVTC/TDeint). In fact, TIVTC's field matching has no notion of a frame being combed or not, it just decides which of two fields fits best to another field.

TIVTC's field matcher can handle low motion scenes (mouthes moving) pretty well (assuming a good match exists) because it uses temporal differencing in combination with field differencing (if it wants to compare matching two top fields A/B with a bottom field C, it first compares A/B to find where they are different and then only applies the spatial metric to the created A/C and B/C frames in those areas). That said, the field matching built into TDeint is much simpler than that of TIVTC's (though at this point it probably runs slower than tfm w/ slow=2). It doesn't use temporal differencing at all and just runs the spatial metric at every point and sums the result... so I would expect it to be much more inaccurate in low motion areas. Too check if the chosen match is combed the created frame is then passed off to the combed frame detection routine.

The reason the combed frame detection routine has such a hard time detecting frames which have very little combing or 'weak' combing (falls under the cthresh level) is simply that it is a tough problem. The comb metric isn't perfect and causes false detections due to noise and high frequency detail. Because of this, it cannot be made that sensitive (low cthresh and low MI values) without causing a lot of false detections.

In TIVTC, which works under the assumption that a large majority of the clip should be matchable, but that you will occasionally get a frame that isn't, it is usually better to let a few combed frames slip through every now and than to have a lot of false detections. However, in TDeint, which w/ tryWeave=true should work under the assumption that a majority of the clip is interlaced, it should be better to have a some false detections than to let a lot of combed frames slip through. So the low-motion idea might help. Or it might help to simply adjust the combed frame detection based on the motion level (the defaults for cthresh/MI/blockx/blocky are 6/64/16/16, so maybe use 6/15/4/8 in low motion areas). The filter could then vary the settings between those two ranges based on the measured motion level.

[tritical]

More thinking... Source type detection in areas without adequate motion is almost impossible. Therefore, the type of such areas must be determined based off assumption. TIVTC works under the assumption that the clip is mostly matchable. Therefore, it should assume that low motion areas are matchable. TDeint works under the assumption that the clip is mostly interlaced (not matchable). Therefore, it should assume that low motion areas are interlaced. Roughly speaking, the number of combed pixels present in frames from an interlaced clip is directly proportional to the amount of motion. Based on that, the amount of combing that the combed frame detection routine looks for (controlled by MI, blockx, blocky) when tryWeave=true should be based on the amount of motion, and it should be requiring less combing as the amount of motion decreases.

[MfA]

Originally Posted by tritical

The directions are found using a 4 scan line vector matching method (which is a custom method that combines ideas from 3 or 4 papers I've read).

You know it's funny, Ive only ever been able to find one other written account (admittedly a while ago) of somoene using vector matching instead of pel matching with an ELA type approach ... and that was in medical imaging to reconstruct dropped scanlines.

[Zarxrax]

I have a question about TDecimate modes 0 and 1. Do they both support random access, or do either of them expect the frames to come in a linear order?

[tritical]

Originally Posted by Mfa

You know it's funny, Ive only ever been able to find one other written account (admittedly a while ago) of somoene using vector matching instead of pel matching with an ELA type approach ... and that was in medical imaging to reconstruct dropped scanlines.

I'm not sure if my use of vector here is what you are referring to or not. By vector I just mean that instead of doing:

abcdefghi
    o
jklmnopqr

diff = abs(h-k)

to test the +3 direction you do:

abcdefghi
    o
jklmnopqr

diff = abs(j-g)+abs(k-h)+abs(l-i)

Or you can use 5, 7, 9, etc... pixels. Of the papers I read only one was using that method. However, that paper was still only using two vectors (one 1 line above and one 1 line below) and sliding them in the standard x+x tap ela style. One of the other papers was using single pel differences but kept points anchored above and below and added 2 extra taps (one 2 lines above and one 2 lines below) so that you then have 2 stationary points and 4 moving points. That helps quite a bit in preventing the line crossover tendency. My method just combines the vectors with the anchor idea and extended it to include additional anchors another scanline above and below the current anchors. So in total there are 10 vectors... 4 stationary (bolded below) and 6 moving.

abcdefghi

jklmnopqr
    o
stuvwxyzA

BCDEFGHIJ

Of course, that alone still isn't accurate enough to be used without a sanity threshold. The processing/filtering of the resulting direction map helps eliminate most of the incorrect directions but is far from perfect.

Originally Posted by Zarxrax

I have a question about TDecimate modes 0 and 1. Do they both support random access, or do either of them expect the frames to come in a linear order?

Assuming that information from tfm (match info, d2v info, etc...) is not present in the stream:

mode 1 does not support random access. mode 0 does support random access.

Assuming information from tfm is present:

Currently, neither mode truely (i.e. 100% reliably) supports random access. Though once the next version of tivtc is out (so that tdecimate allows using a tfmIn file when hybrid=0) it will be supported in mode 0 by using:

tdecimate(input="metrics.txt",tfmIn="matches.txt")

Mode 1 can never 100% support random access since its decision's on the current cycle can be effected by what frame was dropped from the previous cycle.

Next version will also have an option to stop tfm from putting any information into the stream which is not possible atm (it will always put at least the match used and combed/not combed), and an option in tdecimate to ignore the extra information and go purely on metrics (also not possible atm). Unfortunately, I haven't had the time to work on avisynth stuff lately... should change soon.

[tritical]

I sat down last night and did some work on tivtc... I gave up on the having a separate 1.1 branch and only doing bug fixes to 1.0 because I'm too lazy to maintain both. So there might be quite a few more release candidates before a final 1.0. Current working changes:

- fixed cthresh < 0 problems (was broken for both assembly and c routines)
- allow tfmIn file when hybrid=0 in tdecimate
- if the field order of a d2v file does not match the user specified order an error is thrown
- noblend defaults to true
- added new spatial comb metric to tfm and is/showcombedtivtc (from decomb's iscombed/fielddeinterlace)
- added hint option to tfm to disable putting hints in the stream
- added hint option to tdecimate to disable reading of hint information
- replace all frame copying with env->makewritable() where possible
- seeking is fully supported for both modes 0/1 of tdecimate when a metrics input file and a tfmIn file (if hints=true) are used
- added clip2 option to tdecimate (metrics are calculated off the input clip but frames are returned from clip2, all modes supported)
- added hclip option to tdecimate (hybrid=1/3 take frames from hclip instead of doing blend conversion, modes 0/1)
- added RequestLinear filter (makes sure all requests for frames are linear and that all frames are requested)

There is one problem though, the hclip option works perfectly well for hybrid=1 by doing:

mpeg2source()
tfm()
mc = last.mcconvertfps()
tdecimate(hybrid=1,hclip=mc)

However, it is not possible to handle hybrid=3 in the same manner since the clip fed to the mc filter would first need to have the duplicates removed. That causes another problem because the resulting clip (w/ dups removed in film sections) after being fps converted would not have a constant relation to the frames in the input clip. Atm, it looks like for hybrid=3 it will take two passes and the following second pass setup:

mpeg2source()
tfm()
mc = last.tecimate(input="",tfmIn="",mode=8).mcconvertfps()
tdecimate(input="",tfmIn="",hybrid=3,hclip=mc)

where mode=8 would be a special mode for the purpose of dropping the correct frames. If anyone has any other ideas I'm all ears.

[tritical]

[link removed], changes:

+ Added mode 2, and blim parameter

After talking to Zarxrax, I decided to add a same frame rate mode to tdeint that is similar to Blendbob(). How it works is it bobs the input stream like normal mode=1. When asked to deliver frame n it grabs frames n*2-1, n*2, and n*2+1 from the bobbed stream (could be n*2, n*2+1, n*2+2 if field != order). It then differences the middle frame with the previous and next frames. If both differences are over the blim value then it delivers the source frame as is, otherwise it blends the source frame with the frame that produced the smallest difference and returns that frame.

The difference to blendbob is that it will never try to deliver a p/n blend (which is what makes blendbob not support seeking since the decision to blend p/n is based on previous differences) and that it has the max difference value at which point it will not blend the source frame with prev or next.

The main reason for mode 2 is pping for ivtc. Namely:

deinted = tdeint(mode=2)
tfm(clip2=deinted)

mode 2 can work with tdeint's edeint option perfectly fine (the edeint clip has to be as though it is being used in mode=1), so something like:

interp = separatefields().eedi2(field=-2)
deinted = tdeint(mode=2,edeint=interp)
tfm(clip2=deinted)

is also possible.

[Mr. Brown]

@tritical
i wanted to test Tdeint(mode=2) but it crashed always (avisynth 2.5.6a)

setmemorymax(384)
### Plugins ###
LoadPlugin("C:\dgmpgdec146\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\Tdeint.dll")

###  Source ###
mpeg2source("C:\fliegen\flying.d2v")

crop(4,0,-12,-0)

tdeint(mode=2,order=1)

Here is the source
http://rapidshare.de/files/15891931/flying.m2v.html

[tritical]

What type of cpu do you have? Also, when it crashes is it a hard exit or does it give you an error message?

[Mr. Brown]

1. Athlon XP 3000+ (Barton) look at my signature for more information
2. hard exit

my OS is wxp sp2

[tritical]

[link removed], changes:

   + output MIC values in debug info when tryweave=true or full=false
   + added value 70 to emask input
   + added mmx versions of isse/sse2 compare/blend routines for mode=2
   - refactored/rewrote a lot of the code to clean up and simply things, 
       no changes that effect output... should give a slight speed up

@Mr. Brown
Unfortunately, I wasn't able to reproduce your problem. Does it crash if you use tdeint(mode=1,order=1)? Could you also put debug=true in the tdeint() line and use debugview to capture whatever it outputs before crashing?

[berrinam]

tritical, I just wanted to say that I tried out using IsCombedTIVTC instead of IsCombed for my Source Detection, and it performed much better. MeGUI now uses IsCombedTIVTC for its comb detection.

Thank you.

[DarkNite]

I see what you're talking about Mr. Brown. I'm using RC4 in AviSynth 2.5.6a with DGIndex 1.4.6 and VirtualDub 1.6.14 on an AthlonXP 3200+, Windows XP SP2.

mpeg2source()
deinted = tdeint(mode=2,debug=true)
tfm(clip2=deinted)


Here's what DebugView has to say:

[964] TDeint:  v1.0 RC4 (03/19/2006) by tritical
[964] TDeint:  mode = 1 (bob - double rate)
[964] TDeint:  frame 1066:  accumP = 1464846  accumN = 10990531
[964] TDeint:  frame 1066:  field = interp top (0)  order = tff (1)
[964] TDeint:  frame 1066:  mthreshL = 6  mthreshC = 6  type = 2
[964] TDeint:  frame 1067:  accumP = 11032302  accumN = 1905505
[964] TDeint:  frame 1067:  field = interp bottom (1)  order = tff (1)
[964] TDeint:  frame 1067:  mthreshL = 6  mthreshC = 6  type = 2
[964] TDeint:  frame 1067:  accumP = 1308674  accumN = 1368299
[964] TDeint:  frame 1067:  field = interp top (0)  order = tff (1)
[964] TDeint:  frame 1067:  mthreshL = 6  mthreshC = 6  type = 2

Does it crash if you use tdeint(mode=1,order=1)?

Not here, or at least not within the 40544 frames my test source consisted of.

[Mr. Brown]

I wasn't able to make debug file because vdub(mod) close directly after loading the skript

Does it crash if you use tdeint(mode=1,order=1)?

No it works fine

and i try this

deinted=tdeint(mode=1,order=1,debug=true)
tfm(order=1,clip2=deinted)

now it load the skript but it crash after starting the encoding an no debug file was made