TDeint and TIVTC [Archive] - Page 25

tritical

25th January 2010, 21:59

It's possible, but I don't have the time or desire to do it. It isn't a simple modification, at least an easy method isn't coming to mind. The code for tfm/tdecimate is a big mess. It might actually be easier to create a separate filter/program to do it based off tfm stat file and tdecimate stat files from tfm output and bobbed output.

I don't know why manually editing the timecodes file wouldn't work... assuming you splice sections (add the 60p video in the right place)/modify frame numbers as needed.

thetoof

26th January 2010, 17:13

Alrighty, thanks for the answer.

Gser

15th February 2010, 21:06

Interested in a plan to update the gorgeous plug -TFM!? I take them a long time and would like to know whether in the near future, its updating and some "delicious"))) because there is no limit to perfection, and should always be striving towards the ideal ))

Like support for dga files. :'(

johnmeyer

16th February 2010, 04:07

Is there a way to force TDecimate to decimate all combed frames? I have a source where TFM flags really bad frames as combed frames. I want to decimate these first, and then remove the remaining duplicates to achieve a given frame rate. The MIC metric in TFM shows 255 on these bad frames, so TFM is providing the information I need. I don't mind doing a two-pass, but I cannot figure out how to pass a file back into TDecimate that will force it to decimate these "255" frames first, and then use the metrics from TFM to further decimate by removing the true duplicates.

[edit]
I figured it out. This does what I want:tfm(display=false,mode=1,pp=1,cthresh=25,micmatching=3,mmsco=false)
tdecimate(mode=7,display=false, rate=23.976)

ikarad

22nd February 2010, 14:17

Some options that could speed up processing are:

tfm:
slow = 0
micmatching = 0 or 2

tdecimate:
chroma = false

Of course, changing those could change the output, but in most cases it probably wont be noticeable. I would try slow=0 in tfm and chroma=false in tdecimate, and see if it is fast enough.
TVITC support rgb or only YUY2 and YV 12 colorspace?

turbojet

1st March 2010, 23:55

Hi, is there any chance adding support for reading frame type (film or ntsc) from dgi files to TFM like the d2v function?

Here's an example (http://www.mediafire.com/?n1yl2w5t2n1) of a dgi file that has both ntsc and film frames.

D3C0D3R

7th May 2010, 11:35

i've use TIVTC couple of years. it amazing filter pack.

I've found serious memory leak (about 1MB-2MB per second) when use tdecimate mode=5 (2nd pass of hybrid)
with serious filter chain - couple of them writted by tritical too )) *BIG RESPECT*

i've placed this chain before and fter tDecimate, but leak is still appear I even remove some memory & cpu most hungry filters but it seems doesnt depend from type of filters and it position in script.

my settings
tdecimate(mode=5, hybrid=2, vfrDec=1,mkvOut=mkvTimeCodeOut,input=metricStats, vidThresh=15,dupThresh=10, vidDetect=1, cycleR = 1, cycle=2)
encoded by VDub+x264vfw one of latest versions/revisions

I didnt use TFM to do output="matches.txt" at first pass.
I haven't use TFM at all, only TDecimate - to remove duplicates in progressive stream.

In mode=3 all works fine, but mode=3 produces blackness at the end of stream and dont change count of frames.

But I wanna use 2-pass (mode=4 stats -> mode=5 work) because i run batch and dont want to crop end of every file manually

Maybe exists some option that I havn't set or i've found a bug?

tritical

7th May 2010, 15:55

Could you provide an exact full script that causes the issue? I'm not sure how tdecimate could be leaking memory since in mode 5 it doesn't allocate any memory during frame requests... only during script initialization. Does the memory issue happen if you add setmemorymax(64) to the script (i.e. is this 1-2MB per second just due to avisynth's cache filling up at the beginning).

D3C0D3R

11th May 2010, 11:50

>>setmemorymax(64)
yes, i did it at immediately when i found this leak. didnt help.

>>> (i.e. is this 1-2MB per second just due to avisynth's cache filling up at the beginning).
no, it grows all time during encode and finally eat all 3Gb :( and throw Exception

here my a bit simplified script that causes leak

common init part

src = "s02e01.avi"
metricStats = "metrics_01.txt"
mkvTimeCodeOut = "mkv-timecodesfile_01.txt"
SetMemoryMax(80)
LoadVirtualDubPlugin("MSU_cartoon_restore.vdf","MSUCartoonRestore", 0)
AviSource(src,audio=false)
Lanczos4Resize(width*2, height*2)
ConvertToRGB32()
MSUCartoonRestore("bilateral", 1, 56, 36, 1)
spline64resize(width/2, height/2)
ConvertToYUY2()

1st pass

tdecimate(mode=4, output=metricStats)
crop(0,0,8,8)

2nd pass

tdecimate(mode=5, hybrid=2, vfrDec=1,input=metricStats, mkvOut=mkvTimeCodeOut, vidThresh=15,dupThresh=10, vidDetect=1, cycleR = 1, cycle=2)

Zep

24th September 2010, 18:28

1280 x 720p source
Set MT build 2.5.8

=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)

116 FPS

=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

58 FPS

=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

TDecimate(mode=0,cycleR=1,cycle=5)

39 FPS

=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

BilinearResize(320,180)
TDecimate(mode=0,cycleR=1,cycle=5)

28 FPS

More time used to resize down than decimate at full size. interesting!
Though 47 FPS if I MT() the resize but that is for another thread.

=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

ResizedVideo=BilinearResize(320,180)
TDecimate(ResizedVideo,mode=0,cycleR=1,cycle=5,clip2=last)

3.1 FPS

I thought it would be around 28 FPS like the previous example.
Is this a bug? Did i do something wrong?

Thanks

tritical

24th September 2010, 21:40

Without looking into it in depth, I'd guess it's a cache issue. Try:

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

ResizedVideo=BilinearResize(320,180)
TDecimate(ResizedVideo,mode=0,cycleR=1,cycle=5,clip2=last.blur(1.0,0))

Or replace blur(1.0,0) with some other fast filter.

Zep

25th September 2010, 14:19

Zep

25th September 2010, 14:28

ok just now i tried

TDecimate(ResizedVideo,mode=0,cycleR=1,cycle=5,clip2=ChangeFPS(last,last,true))

and FPS is at 9 FPS

that is the only filter I have passed thus far that has helped.
Since that filter grabs 10 frames and caches them it appears it is
as you said a cache issue.

mastrboy

8th July 2011, 11:37

Any chance we will see a updated version of TFM with support for .dgi files (DGDecNV) similar to the support already implemented for d2v files?

Lyle_JP

8th July 2011, 19:39

Any chance we will see a updated version of TFM with support for .dgi files (DGDecNV) similar to the support already implemented for d2v files?

I have no trouble using TFM with DGDecodeNV as my source filter. :confused: You just need to set field order manually.

mastrboy

9th July 2011, 11:41

Not meaning as a source filter, off course that works. TFM has the parameter called d2v="filename", which aids in the IVTC process, currently it only seems to support the format from DVD2Avi and DGIndex d2v files.

From the manual:

d2v -

This option is intended to be used if you are using an mpeg2source() with a d2v file.
It sets the name and path to a d2v file, which TFM will analyze to see if there are any
illegal field order changes and optionally set the order parameter using the field
order of the d2v file. If the d2v file is found to have illegal field order transitions,
TFM will create a fixed d2v file with the string "-FIXED" attached to the end of the file
name. The new file will be located in the same directory as the original. You can then
use this fixed d2v file for processing. If the order parameter is set to "-1" then TFM
will detect the field order from the d2v file and set the order parameter to match. Depending
on the value of the "flags" parameter, TFM will also use the d2v info for field matching
and will pass info from the d2v on to tdecimate to help aid duplicate detection and hybrid
detection.

*NOTE: This option currently supports all d2v formats that I am aware of... which
include: dvd2avi 1.76, 1.77.3 and its variants, all dvd2avidg versions, and
all dgindex versions.

example => TFM(d2v="myd2v.d2v")

Default: "" (String)

flags -

Controls how much of the info from the d2v file is used when the "d2v" parameter is set.
Possible options:

0 - Check the d2v file for illegal transitions and set the order parameter if it
is not already manually set. Also, pass on rff flag duplicate info to
tdecimate.
1 - Same as 0, plus use the trf flags for field matching in film sections (sections
where the trf flags follow the 012301... pattern)
2 - Same as 1, but use the trf flags for field matching in all areas (doesn't have
to be in 0123 pattern) (very much not recommended!)
3 - Same as 0, but don't pass on any info to tdecimate (i.e. only set order and check
for illegal transitions)
4 - Same as 1, but d2v matches are checked for being combed. If a d2v match is detected
as combed then tfm uses its own matching routine for that frame.
5 - Same as 4, but d2v matches are only checked for being combed around scenechanges.

VERY IMPORTANT (MUST READ): For options 0, 1, 2, 4, and 5 to work correctly, tfm/tdecimate
must either immediately follow the mpeg2source() command or
any filters inbetween them must not modify the order or number
of fields in the stream in any way. Or use a trimIn file to
tell TFM what frames have been discarded.

Default: 4 (int)

Stereodude

27th November 2011, 23:58

I just wanted to post a thank you here for tritical for the great work on TIVC (tfm / tdecimate). The ability to use one clip as the basis for another's decimation with TDecimate ( clip2= ) really helped out my workflow on a few clips I was working on getting back to p24.

Thanks! :cool:

manolito

28th November 2011, 00:38

I.E. optimum quality vs. speed solution

Just want to find out if this line (posted by tritical a few years ago) is still state of the art:

yadifmod(edeint=tdeint(mthreshL=0,mthreshC=0))

A simple TomsMoComp or Yadif has too many quality issues, and all those ultra high quality (and ultra slow) deinterlacing scripts are just unusable for my rather slow machine. I always liked the quality and the speed I got from the above command. Is there anything new out there which surpasses it in terms of speed vs. quality?

Cheers
manolito

tritical

28th November 2011, 19:55

A simple TomsMoComp or Yadif has too many quality issues, and all those ultra high quality (and ultra slow) deinterlacing scripts are just unusable for my rather slow machine. I always liked the quality and the speed I got from the above command. Is there anything new out there which surpasses it in terms of speed vs. quality?

My recommendation would be QTGMC (http://forum.doom9.org/showthread.php?t=156028) with the slowest preset that meets your speed requirements. I don't think there is much question that QTGMC is the way to go for deinterlacing in Avisynth nowadays. Getting it setup can seem difficult, but Vit has made it very easy if you read the first post. If you can't find a preset that betters your current script in speed/quality report back :).

manolito

28th November 2011, 21:57

Thanks a lot for the tip, I will give it a thorough test run tonight...

Cheers
manolito

manolito

29th November 2011, 01:58

Alright, here's what I've come up with:

First of all, setting up QTGMC really was a little difficult, because my CPU (P3 Coppermine) does not support SSE2. Some of the plugins Vit provides are SSE2 plugins, and of course I got a floating point error. So I had to replace the SSE2 plugins with different versions while still trying to not brake LSFMod or LimitedSharpenFaster. Took me some time, but it works now.

Now for the speed tests:

I took an interlaced TFF DVD source, no filters except the deinterlacer, feeding it to QuEnc. These are the speeds for the different deinterlacers:

TomsMoComp: 11.77 fps
Yadif: 11.02 fps
YadifMod(edeint=tdeint(mthreshL=0,mthreshC=0)): 6.34 fps
QTGMC very fast: less than 1 fps
QTGMC draft: 4.65 fps

QTGMC is only usable with Preset="Draft" on my machine, and I do have my doubts if the resulting quality can match the quality of "yadifmod(edeint=tdeint(mthreshL=0,mthreshC=0))". I don't have the knowledge to analyze the QTGMC script enough to understand what the "Draft" preset really does, but it seems to use Yadif with Yadif's own spatial prediction, and I have the feeling that YadifMod using TDeint's prediction might do a better job....

Cheers
manolito

-Vit-

29th November 2011, 02:32

"Draft" is an extremely cut-down version of the core TGMC algorithm. It doesn't use Yadif, it performs a motion-compensated temporal binomial smooth to remove shimmer, but it only uses a bob as an interpolate, does not mask against creating blur and does no resharpening. As it's name implies it's not intended for final usage. QTGMC only really becomes useful at "Super Fast" and above IMO.

No offense intended, but you can't really ask for the "state of the art" if you intend to use a Pentium 3.

OvejaNegra

4th August 2012, 10:46

Hi: I'm trying to create a tool for simplify the process of creating VFR with 59.97, 29.97 and 23.97 fps using the metrics output of tivtc.
Can anyone explain to me what the metrics of TDecimate means:
0 0 0
1 1824 470069
2 1296 322363
3 1328 316633
4 1563 299634
5 2136 447795
6 2617 397772
7 3621 471805
8 3872 508060
9 5221 487823
10 5963 546645

Also, what the h matching means on TFM metrics?

And one more question:

(sorry the quote button does not works on my explorer, probably the antivirus is blocking scripts on the page):

This is from tritical:
"It might actually be easier to create a separate filter/program to do it based off tfm stat file and tdecimate stat files from tfm output and bobbed output."

"tfm output and bobbed output" mmmm why? and how can i give two outputs to tdecimate (or is actually about apply bob to the tfm output and give it to tdecimate? that is a mess i think? can anyone elaborate a little more please.

english is not my language, maybe i'm not undertanding properly.
thanks

cretindesalpes

4th August 2012, 11:52

Can anyone explain to me what the metrics of TDecimate means
"Each line will include a frame number, plus the difference metric, and the scene change metric." IIRC the metrics are not scaled, they are more or less proportional to the picture area (they are sums of absolute differences or something related).

Also, what the h matching means on TFM metrics?
From the TDecimate source code:

#define ISDB 0x00000005 // l = (deinterlaced c bottom field)
#define ISDT 0x00000006 // h = (deinterlaced c top field)

OvejaNegra

4th August 2012, 17:44

And how TDecimate goes from something like:
1 1824 470069

To calculate the video threshold or dup threshold (wich is something like 1.1 , 2.8, etc).

I just want to parse a tdm metrics file to get the video and film sections by myself. tfm metrics are clear to me but not the tdm.

Another question please:
why tfm has out of order matching modes like:
5 = 3-way match + 4th/5th matches on combed (p/c/n + u/b)

because hard telecine sources have field order changes because bad edits or cuts?
but such changes wont produce jumpy movement when displaying on tv set?
or to reformulate my question: Why mode 0 is not enough?

And another one:

Why tdm detects 17 fps ranges? (i'm talking about mode 5 with timecodes output).
Thanks.

j7n

2nd October 2012, 13:29

I just got a recurring error in Avisynth v2.5.8 and TIVTC 1.0.5:

TDecimate: mode 2 internal error (ret less than 0). Please report this to tritical ASAP!

I was trying to trim and further process the output of TDecimate(mode=2,rate=25.0,maxndl=30,m2PA=true,chroma=false). The error occurred after the end of the decimated clip.

Whops. I was trying to request more frames than there actually were after decimation. Scaling back the trim range by a few frames did away with the error.

I could reproduce the same message just by having a script with two lines and playing it from start to finish. The source was progressive with dupe frames, running at 29.97 fps.

SubJunk

1st April 2013, 00:01

I'm trying to get rid of some residual combing. Here's the result after just doing TFM + TDedimate:

http://i.imgur.com/evbdoDn.png

I tried default settings and mode=1 but no change, the only thing that fixed it was cthresh=-1.
Other than taking longer to process, are there any problems with using cthresh=-1? Could it reduce quality? Any suggestions?

DeathAngelBR

1st April 2013, 04:10

I used to use 2-pass encodes with retarded awful sources. Something like this:

# 1st pass
tfm(d2v=_d2v,order=-1,slow=2,mode=4,pp=1,cthresh=5,mi=20,blockx=128,blocky=128,output="tfm.txt")
tdeint(slow=2,mi=20,cthresh=5,blockx=128,blocky=128)
tdecimate(hybrid=2,mode=4,expp=true,output="decimate.txt")

# 2nd pass
tfm(d2v=_d2v,order=-1,slow=2,mode=4,pp=1,cthresh=5,mi=20,blockx=128,blocky=128,input="tfm.txt",ovr="over.txt")
tdeint(slow=2,mi=20,cthresh=5,blockx=128,blocky=128)
tdecimate(hybrid=2,mode=5,expp=true,input="decimate.txt",mkvout="mkv.txt")

OvejaNegra

3rd April 2013, 18:57

Usually, using cthresh=1 is enough for me to detect all the residual combing (on anime sources).
Using cthresh=-1 sets all teh frames as interlaced and applys deinterlacing on all the frames.

SubJunk: ARE YOU USING COLOR MATRIX?
If you are usign it, please set the interlaced parameter to true, because if dont, it creates those artifacts if the source is interlaced.

Try this:
#this is a deinterlacer that gives very good results, even on false positive detection
interp = full.separatefields().eedi2(field=-2)
deint = full.tdeint(mode=2,edeint=interp)
TFM(display=false,micout=0,mode=6,pp=2,slow=2,mChroma=true,chroma=false,cthresh=1,micmatching=0,clip2=deint)

#set chroma=false is your source has many chroma problems like rainbow or dot crawl

Good luck

SubJunk

6th April 2013, 05:36

Thanks guys :) Neither of them worked though. Anyway the rest of the clip is fine with the regular defaults, it's just that one part which only lasts for less than a second so it will be OK. Thanks anyway!

osgZach

12th April 2013, 20:40

If you set cthres=-1 then it treats all frames (pixels really) as combed and would be deinterlacing every frame in the clip. This can certainly lead to quality loss if you are throwing out good data.

The proper thing to do is to use an overrides file and specify the frame numbers (or frame range) in question for deinterlacing.

sikboy1029

14th April 2013, 08:07

Hey guys, you mind helping my out on the script for this one? The interlacing is pretty significant & I can't get rid of it.

http://www.mediafire.com/?u2buiev9awsgsdt

LigH

14th April 2013, 08:23

One of your mistakes was uploading not the original video, but the result of a failed filtering; yet, a frame-by-frame forward stepping still reveals a 3-2 pattern; and it has NTSC framerate. Therefore I would expect it being telecined (3:2 pulldown). In this case, "Deinterlacing" is wrong, "Inverse Telecine" (IVTC) is right.

sikboy1029

14th April 2013, 08:44

It's actually a cap of a netfix stream, been noticing the same issue with a lot of the animation they have on the service & it was bugging me. You would think they would have some sort of QA...it is visible with the naked eye watching it on TV & very distracting...I did try this

Telecide(guide=1)
Decimate(cycle=5)

Any recommended change?

johnmeyer

15th April 2013, 00:14

I have found that captures from Netflix often contain dropped frames, so you may need to deal with that. What surprises me is that the original was sent as 29.97 rather than 24 or 23.976. Did you force your capture card to capture 29.97? If the original was normal NTSC film rate (23.976), I have always found that this is what Netflix sends. Therefore, perhaps you have the wrong capture settings.

sikboy1029

15th April 2013, 20:13

I also capped in 23.976 & it looks the same, when you say I need to deal with the dropped frames what do you mean?

johnmeyer

15th April 2013, 23:37

I also capped in 23.976 & it looks the same, when you say I need to deal with the dropped frames what do you mean?I said you may need to deal with dropped frames. I have observed dropped frames in video I have captured from the analog port on my XBox. They do not occur at a regular interval, and probably depend on many things, including my local connection speed as well as issues at the Netflix server. Some services, like Hulu, can be even worse.

I don't know what more clarification I can provide as to what I mean: a dropped frame usually results in a discontinuity in the video (a "jump") and is usually followed by a duplicate frame in order to keep the sound from drifting out of sync.

Netflix may also be using variable frame rate encoders that respond to connection speed changes, in which case I don't know what to recommend.

Lenchik

18th October 2013, 21:30

Can anyone create build of TFM and TDecimate (or whole TIVTC package) that supports YV24, please? Or maybe there is one already?

turbojet

20th November 2013, 09:20

Currently there doesn't seem to be any way to multithread tdecimate. Would anyone have interest in trying to implement it? Tritical has said in the past that openmp might work.

Also some combed frames slip through with tfm default mode=1, while they don't with mode=3 with no noticeable drop in speed. Any consideration on changing the default mode?

ChiDragon

21st November 2013, 00:52

Sure you have fewer combed frames, but the frames you get are just extra duplicates that can't be decimated normally, no?

johnmeyer

21st November 2013, 01:12

Currently there doesn't seem to be any way to multithread tdecimate. Would anyone have interest in trying to implement it? Tritical has said in the past that openmp might work.I don't think that TDecimate is particularly CPU intensive and therefore I don't think multi-threading would help much. The only time I've found TDecimate to be slow is when using really large Cycle/CycleR values, but that is a different problem, I think.

turbojet

21st November 2013, 01:58

Tritical mentioning openmp (http://forum.doom9.org/showthread.php?p=1405365#post1405365) there's also another good idea in that message that's been unrealized.

Sure you have fewer combed frames, but the frames you get are just extra duplicates that can't be decimated normally, no?

Nope, all clean, no dupes. I'll hunt for a sample, with any luck at all will have one by the end of the week.

I don't think that TDecimate is particularly CPU intensive and therefore I don't think multi-threading would help much. The only time I've found TDecimate to be slow is when using really large Cycle/CycleR values, but that is a different problem, I think.

When it's encoding HD on a typical 2/4/6/8 core cpu it can keep the cpu loaded but when encoding smaller resolutions or with more cpus (untested) there's a lot to be desired.

Here's some 1080i to 720x404 on FX8320 results:
tfm() 51.15 fps, 50-60% cpu
tfm().tdecimate() 36.64 fps 40-50% cpu
setmtmode(2,4).tfm() 66.29 fps 95-100% cpu
setmtmode(2,4).tfm().tdecimate() 29.79 fps 30-40% cpu
nnedi3() 59.51 fps 95-100% cpu

Sorathread/threadrequest used between tfm() and tdecimate() can give ~5% fps and cpu increase that's the only increase I've been able to achieve. Openmp was a huge increase in nnedi3, if it gave half the returns with tivtc it would be a much welcome increase. If tfm().tdecimate() used 100% cpu in this case is 70-80 fps a good estimate?

turbojet

3rd December 2013, 23:04

I haven't been able to find another source where tfm(mode=3) helps yet but I did find one where it causes missing frames.

Also after some testing it doesn't seem like tdecimate is the bottleneck but rather tfm(). Using telecide() or uncomb() with tdecimate() greatly increases fps and cpu usage.

Stereodude

26th January 2014, 20:27

I realize the TIVTC package hasn't been updated in a long time, but I'm going to make a feature request anyhow...

It would be nice when if TDecimate mode=5, hybrid=2 (2nd VFR pass) was more verbose with display=true. Something similar to what TDecimate does for mode=0, hybrid=3. I do the first pass with TFM and TDecimate and make the necessary output files and feed them into the 2nd pass. Easy enough, but on the 2nd pass I have no easy, displayed for me on the video, visibility into the metrics calculated in the first pass that determine what frames are dropped or when it switches in or out of video/film mode. As a result it's hard to visually tune the vidThresh or dupThresh parameters.

johnmeyer

27th January 2014, 00:07

What I do is to use the "output" parameter. It provides a file in Pass1 that gives you all the metrics.

You might also have a look at the "debug" parameter.

Stereodude

27th January 2014, 01:20

What I do is to use the "output" parameter. It provides a file in Pass1 that gives you all the metrics.I'm aware the information is ultimately in the file, but it's a huge pain in the butt to find the frames you're interested in and look at a file of numbers without the context of seeing the video frames at the same time.

You might also have a look at the "debug" parameter.FWIW, in TDecimate mode=5 The information that comes via the debug parameter is as useless as in the information displayed on the screen.

johnmeyer

28th January 2014, 00:29

I'm aware the information is ultimately in the file, but it's a huge pain in the butt to find the frames you're interested in and look at a file of numbers without the context of seeing the video frames at the same time.That is most definitely true. I just wanted to make sure you knew that the information was available.

I wonder ... is there an "overlay" plugin that could that this output file, which has the frame number as the first parameter, read it, and then display the parameters onto each frame?