TDeint and TIVTC

[ChiDragon]

Well I use TDecimate's display mode with TFM's hints to see what both filters are doing, but I guess I understand if keeping them in would interfere with the second decimation.

[foxyshadis]

Tritical, would you mind updating your page with the smartdecimate compile? Just to have someplace to reference besides a lonely post when telling people where to get the latest. ;)

[Chainmax]

I am going to use TDeint+TMMv1 (this one (http://bengal.missouri.edu/~kes25c/mmask.dll), right?) as Clip2 for an IVTC, would this:

AssumeTFF()
Deinted=TDeint(order=1,field=1,type=1,emask=TMM(order=1,field=1))
TFM(mode=6,order=1,PP=7,slow=2,mChroma=true,Clip2=Deinted)
TDecimate(mode=1)


be the correct way to do so?

[Wilbert]

Tritical, would you mind updating your page with the smartdecimate compile? Just to have someplace to reference besides a lonely post when telling people where to get the latest.
Tritical? I can't find that version in this thread.

[Leak]

Tritical? I can't find that version in this thread.
Try this post (http://forum.doom9.org/showthread.php?p=897598#post897598)... :)

np: Gui Boratto - Beautiful Life (Chromophobia)

[Zarxrax]

I'm having some trouble fully understanding the d2v parameter that tfm takes.

If I set the d2v parameter, then tfm will check the d2v for illegal field order, and create a new d2v if necessary. Also, tfm will use hints from the d2v to help its field matching process? Will this theoretically give more accurate results, or will be be faster?

Next, is it really necessary to specify the d2v parameter manually? Can't dgdecode pass the data to tfm as a hint or something? And also, is there anyway that tfm could simply fix the d2v file on the fly, rather than making a new one and requiring you to go back and change your script?

[tritical]

@Chainmax
The script looks right, but you'll need to use TMMv1 (http://bengal.missouri.edu/~kes25c/TMMv1.zip) instead of mmask.dll for that syntax to work.

@foxyshadis
I'll do that.

@Zarxrax
Next, is it really necessary to specify the d2v parameter manually? Can't dgdecode pass the data to tfm as a hint or something? And also, is there anyway that tfm could simply fix the d2v file on the fly, rather than making a new one and requiring you to go back and change your script?
It would be possible for dgdecode to pass the information to tfm, but dgdecode doesn't currently have this ability. Also, it is slightly easier reading from the d2v and it's faster. TFM could fix the d2v on the fly (meaning it could simply overwrite the existing file instead of creating a new one), but the dgdecode object would still need to be recreated (script closed/reopened) for the changes to effect decoding. My main reason for not overwritting the existing file is I sometimes like to examine the two side by side.

If I set the d2v parameter, then tfm will check the d2v for illegal field order, and create a new d2v if necessary. Also, tfm will use hints from the d2v to help its field matching process? Will this theoretically give more accurate results, or will be be faster?
What exactly happens depends on the 'flags' parameter. With the default setting of 4, tfm will check the d2v for illegal transitions and create a new d2v if necessary. It will also pass rff information to tdecimate. The rff information is only used to help aid hybrid detection (detection of 30p parts), so unless you are using hybrid > 0 it won't effect anything (and if the d2v is just all 2's or 0's it won't effect anything either). tfm will also use the trf flags for field matching in film sections (areas of the d2v with the 012301230123 pattern). That means it will determine what match to use based on the d2v flags. It will then check each of these d2v based match decisions to see if the resulting match is combed or not. If it is, then tfm will go back and run its usual match algorithm. Otherwise, it will use the d2v match. This can give a significant speed up if the video has a lot of sections that follow the 01230123 trf pattern, since it means tfm wont have to run its match algorithm. It can also improve things since tfm isn't always correct in its match decisions.

I will say that if you work mostly with r1 anime, the d2v parameter is usually not that helpful since most r1 anime never has 01230123 flagging (the d2v will just have 0's and 2's, 100% NTSC reported by dgindex). In that case, all that setting the d2v parameter accomplishes is checking for illegal transitions.

[Zarxrax]

Alright, thanks for the clarification.

[ChiDragon]

tritical, can you check this out?

Metrics file ("cap9_TDec.txt") (http://chidragon.thedessie.com/cap9_TDec.txt)
BlankClip(length=80061,pixel_type="YV12",fps=30000,fps_denominator=1001)
TDecimate(mode=2,rate=23.483,input="cap9_TDec.txt",batch=true,display=true)

This gives me a "System exception - Access Violation." Other rate values also cause it. 23.482 works fine, as do these rates with mode=7 instead of 2.

I've also got a request... would it be possible to add standalone "+" to the override options to force a decimated frame to be kept? From what I've gathered through use and TDecimate's doc, "+" is currently only used to separate kept and dropped frames for override cycles (which are hard to write out for hundreds of frames at a time) and doesn't actually override anything itself. Currently I usually just edit my TDecimate metrics files and give frames I want to keep huge values, but it's not really practical, especially when I mess up and want to undo something.

[chipzoller]

Would someone mind taking a look at this sample (http://www.megaupload.com/?d=ZJ2I3WHE). I'm trying to figure out how to setup tfm and tdecimate to restore progressive frames from this apparent 2p4i pattern clip but after reading the help files for both and trying samples/experimenting around with decimation cycles am not sure how to restore this. It actually looks like blended fields, but I know this series (Sherlock Holmes with Jeremy Brett) was shot in film, and all the DVDs in the series except the one from which this episode was taken were just soft pull-downs. Or is this not a candidate for tfm+tdecimate but rather some function like restore24 or the likes? Thanks much for the advice.

[Boulder]

The series were originally shot as progressive 25fps I think. Does RePAL do any good?

[Livesms]

@All
Can anybody suggest good Bob deinterlace
I tried Tdeint as a bob deinterlace for my TFF PAL video
topf = separatefields().selecteven().createMM()
botf = separatefields().selectodd().createMM()
tdeint(mode=1,order=1,emask=buildMM(topf,botf,mode=1,order=1,dis=2)) and got this
http://img5.imagevenue.com/loc385/th_50064_TDeint_122_385lo.jpg (http://img5.imagevenue.com/img.php?image=50064_TDeint_122_385lo.jpg)
Yadif(mode=1) give me better result with straight lines but has lots of artifacts
http://img175.imagevenue.com/loc379/th_50068_Yadif_122_379lo.jpg (http://img175.imagevenue.com/img.php?image=50068_Yadif_122_379lo.jpg)

[chipzoller]

@Boulder,
Yes I think you may be right. It's just that with the majority of the releases in the "Adventures of..." series they did a proper telecine job, but with this one it seems they went the lazy route and just did a blend-conversion. I'm really not "up" on the methods and processes to get PAL to NTSC so I can't speak from a knowledgeable position...In any case, RePal seemed to do a good job, although I had to use convertfps(25) at the end as it was outputting a 24.75 frame rate otherwise. I really wish there were some tutorials out there that gave examples of various field blendings and avisynth scripts/plugins to try and tackle them.

[Boulder]

Instead of ConvertFPS, use AssumeFPS.. then again, Restore24 has a RePAL equivalent which outputs 25fps IIRC. You might want to post in the R24 thread or create a new one in the Avisynth usage forum so it will get noticed by the pros :)

[ChiDragon]

You should probably use AssumeFPS(25,1,true) followed by an audio resample instead of ConvertFPS. Bump up the video framerate and alter the audio pitch slightly instead of adding blends to your unblended video...

EDIT: Sorry, I see Boulder already said that. I had a cached version open.

[chipzoller]

Thanks ChiDragon. I'll try AssumeFPS with those params as AssumeFPS(25) produces out of sync audio. However since I'm dealing with a demuxed DVD source, how would, once you demux the audio track, you get AssumeFPS and avisynth to consider and alter the AC3 file?

[manono]

@Boulder,
In any case, RePal seemed to do a good job, although I had to use convertfps(25) at the end as it was outputting a 24.75 frame rate otherwise.
What's the matter with 24.975fps? Is it any more peculiar than 23.976? If for AVI, keep it there. If for DVD, run the MPV through DGPulldown with the Custom box ticked and 24.975->29.97 filled in. That way you can keep the audio unchanged. ConvertFPS just reblends what RePAL unblended, and AssumeFPS, as you discovered, requires stretching the audio.

If you are going to use AssumeFPS and stretch the audio, you may as well make it for 23.976fps if, as you say, it was shot on film.

[chipzoller]

Manono, these are destined for my iPod and it needs a frame rate of PAL, film, or ntsc to comply with spec.

[manono]

OK, but if I'm not mistaken, I believe that's the first time you've mentioned requiring specific framerates.

Me, I'd make it 23.976fps (that's what you mean by film, isn't it?), and stretch the audio to do away with the PAL speedup and higher pitched audio. If you want to keep the same audio, then you can do a ChangeFPS(25) after RePAL, which will add one duplicate frame every 1000 frames, or 1 every 40 seconds, to produce a slight stutter which may or may not be noticeable.

[ChiDragon]

However since I'm dealing with a demuxed DVD source, how would, once you demux the audio track, you get AssumeFPS and avisynth to consider and alter the AC3 file?

I use NicAC3Source, part of NicAudio (check http://avisynth.org/warpenterprises/). You could also use AC3Source or DirectShowSource with an AC3 codec/filter installed. The AC3 wll be decompressed to uncompressed PCM of course.

[laserfan]

I know this is a Purist thread, in a (mostly) Purist forum, but I'm looking for some simple advice in handling conversions of my HDTV captures to Xvid. I routinely capture ABC TV shows that have been converted by the local affiliate station from the network's original 720p to 1080i, and then sometimes they are also time-compressed. Bottom line is they started life as Film and along the way get messed with--they're mostly 3-2 but then sometimes 3-2-2-2-2-1-3-2-3-2 etc w/no repeating pattern. 95% of the time my conversions look great, or at least "good enough for me" using this:

tfm().tdecimate(hybrid=1)

On rare occasions the above doesn't do quite as good a job as I might've hoped (the 23.976 result is more herky-jerky than I'd like) and I wonder if there is a new-or-better script I can or should use for these captures.

Bear in mind please that I don't want to analyze/count frames for each & every conversion I do--I'm not looking for perfection; I just want to choose a "good enough" method (a compromise) and stick with it if possible. TIA for any advice...

[Mug Funky]

@ chipzoller:

if something was shot on film for TV in PAL land, the cameras will have been run at 25fps as there is no reason to run them at 24. so i'd assumefps, and be assured the audio pitch will be correct as it was shot at that speed to begin with (and nobody would bother with a timestretch for a 1/1000 difference in speed).

it's possible this episode was either never aired in NTSC land, or a tape could not be found and thus a conversion was done off a PAL master.

[laserfan]

tfm().tdecimate(hybrid=1)
...is [there] a new-or-better script I can or should use for these captures.Hmmm, I wonder why no one has an opinion about this? I didn't think my question was too off-the-wall?

[ChiDragon]

laserfan,

EDIT: Nevermind, I tried my suggestion and it doesn't work... your script is the best I can think of, of the solutions that don't require manual intervention.

[Atak_Snajpera]

TDeint() generates strange artefacts.
Source is 1440x1080i (Canon HV20)
http://www.archive.org/download/jvc_hd7/hv20_05.m2t.mpg

After TDeint() look what happens

http://img524.imageshack.us/img524/3817/tdeintfx5.th.png (http://img524.imageshack.us/my.php?image=tdeintfx5.png)

I've also uploaded sample after conversion to 1280x720 (x264)

http://www.megaupload.com/?d=71Z12QI0


Another example:
Source: http://www.archive.org/download/jvc_hd7/hv20_07.m2t.mpg

Once again strange spots show up
http://img517.imageshack.us/img517/4987/tdeint2cd3.th.png (http://img517.imageshack.us/my.php?image=tdeint2cd3.png)

I used very simple script

LoadPlugin("C:\Users\Dawidos\Documents\Delphi_Projects\RipBot264\Tools\dgindex\DGDecode.dll")
MPEG2Source("c:\TEMP\job1\job1.d2v")

tdeint()
LanczosResize(1280,720)

[laserfan]

...your script is the best I can think of, of the solutions that don't require manual intervention.Thanks for that--I just had to ask, given the many, many pages of enhancements discussed here, most of which I just don't understand. :(

But thanks for responding! :)

[tritical]

Almost ready to start working on tivtc/tdeint again. I read back through the last few pages and this is the list of current issues:

tdecimate mode 2 problem non-standard framerates - Xesdeeni/ChiDragon
tdecimate problem with mode 0 decimation - ChiDragon
add tfm hints to tdeint - ChiDragon
tdecimate '+' override - ChiDragon
tdecimate problem with video section decimation in mode 5 (starting in version 0.9.12.7) - Asrial

If there are any other items I've forgotten please let me know :).

[JuanC]

Well, now that you mention other forgotten items, there's this old request (http://forum.doom9.org/showthread.php?p=821397#post821397), for an exclusion band for Tdecimate. Not really a "current issue" but it will certainly help people like me dealing with 30fps interlaced subtitles blended on top of 24fps hard-telecined material. Thanks ;)

[tritical]

An exclusion band in tdecimate could be accomplished by using the clip2 parameter:

mpeg2source()
tfm()
saved = last
stackvertical(crop(),crop()) # set crops to remove lines containing subtitles
tdecimate(clip2 = saved)

In this case, tdecimate will calculate everything based on the input clip, but will output frames from clip2. The clip2 clip only has to have the same number of frames as the input. All other properties: height, width, and colorspace can be different. You could replace cropping/stackvertical with another method of removing the subtitles (set the subtitle area to black using overlay, etc...).

It would be possible to add y0/y1 parameters to tdecimate and have it do the above internally, but I don't think it's worth it. Also, actually having y0/y1 parameters that are taken into account during metric calculation would be too much of a pain to add atm. However, it is on the list of things to add in the tdeint/tivtc rewrite.

[DarkNite]

However, it is on the list of things to add in the tdeint/tivtc rewrite.

Did I miss something, or have you been working on a whole new code base in your secret other life while cackling like a mad scientist who just built two tesla towers with just enough space for an operating table between them? ^_^

tdecimate problem with video section decimation in mode 5 (starting in version 0.9.12.7) - Asrial

Thank you for adding that to your current to do list.

[max24]

How is one this SOURCE Deinterlacen?
I despair of it :helpful:

Sampler:
http://rapidshare.com/files/31967235/Sampler.m2v.html
http://simpleupload.net/download/88293/Sampler.m2v.html

Thanks for your Help

[JarrettH]

http://forum.doom9.org/showthread.php?t=125990

Maybe I should ask this here instead.....

I'm trying to encode Pan's Labyrinth and MeGUI detects it as Partially Film (correct) with Top Field First. What's the difference between choosing TIVTC alone vs TIVTC+TDEINT?

:thanks:

[ChiDragon]

TFM by default will use PP=6 which is motion-adaptive cubic interpolation on frames detected as combed. TFM+TDeint by default will use TDeint's different motion masking and kernel interpolation instead of cubic.

Maybe you should post of a sample of the unprocessed VOB in a section where you say TIVTC creates jerkiness.

[JarrettH]

I think I figured it out. So apparently clicking analyze will make a different avs script than if I were to choose Partially Film + Top Field + TIVTC manually.

If I select myself:
tfm(order=1).tdecimate(hybrid=3)

If I click analyze
tfm(order=1).tdecimate(hybrid=1)

:devil:

[ChiDragon]

hybrid=1 is for mostly film (blends 30p->24p), hybrid=3 is for mostly video (blends 24p->30p).

[JarrettH]

It definitely did fix it though. The stream is 23.97 fps and looks proper. So TIVTC+TDEINT should produce better quality?

Thanks Chi:)

[ChiDragon]

@max24, except for 2 shots of the plane in the opening (before and after the kangaroo at the end), that is just plain old easy field shifting. Use "TFM()" or even "TFM(mode=0,PP=0)". The two plane shots mentioned have some duplicated fields which will cause the video to be a bit jerky there, but they're short sequences anyway.

[ChiDragon]

JarrettH, I believe tritical would agree that TDeint's deinterlacing is superior to what's built into TFM yes.

[JarrettH]

It probably is, but I was bored and made some captures :devil:

I think this movie only begins to have interlacing problems near the end. Three shots, one TIVTC alone the other TIVTC+TDEINT

If anyone wants to compare:

http://rapidshare.com/files/32249715/TIVTC_and_TDEINT.rar.html

Thanks!

[JarrettH]

http://img503.imageshack.us/img503/5196/tivtc1px1.png

http://img128.imageshack.us/img128/2616/tivtctdeint1kg5.png

http://img264.imageshack.us/img264/2695/tivtc2qk7.png

http://img503.imageshack.us/img503/2765/tivtctdeint2ue0.png

http://img264.imageshack.us/img264/46/tivtc3hp1.png

http://img264.imageshack.us/img264/4259/tivtctdeint3sh4.png

Top one in each set is TIVTC alone, bottom is TIVTC+TDEINT

[foxyshadis]

I don't think the TFM clip2 parameter is working as it should. I've been using:

clipt=TDeint(field=1,emask=TMM(field=1),edeint=separatefields.selecteven.eedi2(field=1))
clipb=TDeint(field=0,emask=TMM(field=0),edeint=separatefields.selectodd.eedi2(field=0))
TFM(clip2=clipt,pp=4)

But the frames I get back are exactly the same as with TFM(), and very different from clipt. I haven't had any luck figuring out why.

[tritical]

@DarkNite
You wouldn't happen to have a clip that results in different mode 5 output (latest version vs. older versions) would you? I've been trying to duplicate the issue but haven't been successful. I pm'd Asrial a couple days ago about getting a clip, but I don't think he visits very often.

@foxyshadis
Thanks for the report. I'll try to duplicate it.

[foxyshadis]

Never mind, I just didn't understand the docs well enough. I didn't realize that it only uses clip2 for combed frames, so I was checking the wrong frames, but using a much lower cthresh for testing forced it to work. Sorry!

[DarkNite]

I don't recall exactly what portion of which source it was atm (I have 6 projects going on now), but for you I will attempt to hunt it down in my free time.

[tritical]

@DarkNite
Thanks, though if you don't remember where it was don't worry about it too much... I'm sure you have better things to do with your free time :).

@All
I have managed to fix the mode 2 problems. The drifting on long sequences that Xesdeeni reported was caused by having a '>' instead of a '>='. The crash ChiDragon reported was caused by a bug in the sorting algorithm that was used for very large cycles (could only happen when using mode 2 with an input file and non-standard framerates). I also made some minor improvements.

I'm still trying to duplicate the mode 5 problem. I've been comparing TIVTC v1.0 RC1 vs v1.0.1 on a few hybrid sources and have yet to see any major differences. Although, I did find a bug in tfm's cubic pping. The handling of the very last line (for field=1) and very first line (for field=0) has been slightly messed up since ~ April, 06.

[Terranigma]

tritical, any new news on EEDI (http://forum.doom9.org/showthread.php?p=966991#post966991)'s successor? How is it progressing? :cool:

[Livesms]

I don't think the TFM clip2 parameter is working as it should. I've been using:

clipt=TDeint(field=1,emask=TMM(field=1),edeint=separatefields.selecteven.eedi2(field=1))
clipb=TDeint(field=0,emask=TMM(field=0),edeint=separatefields.selectodd.eedi2(field=0))
TFM(clip2=clipt,pp=4)

But the frames I get back are exactly the same as with TFM(), and very different from clipt. I haven't had any luck figuring out why.

Can you explain this script :) Please :)

[tritical]

@Livesms
clipt/clipb are just tdeint deinterlaced versions of the original source... clipt keeping top fields and clipb keeping bottom fields. In both cases TMM() is being used to build the motion mask instead of tdeint's built in motion masking, and eedi2 is being used to interpolate pixels instead of one of tdeint's built in interpolation methods. One of those two clips is then being used by tfm's clip2 parameter (in this example it is clipt). That means that when tfm detects a combed frame combing out of field matching it requests that frame from clipt instead of deinterlacing it itself. When pp is set < 5, the frame from clip 2 will be delivered as is. If pp is >= 5 then tfm will still apply its own motion masking.

@Terranigma
It's doing quite well. I haven't commented on it for a while because I'm planning on using the idea as part of my master's thesis, which means that a public binary is still a ways off.

[foxyshadis]

Maaaaaaaaan, that would be nice to see indeed. ;_;

The reason I used two different tdeints, if you're curious, was because I had a filter in between that would swap in frames from the other (a quick hack of fizick's badframes). It was an old VHS capture and had an annoying amount of partially smeared lines that could be mostly fixed by interpolating over the other field. I got sick of finding bad frames pretty quickly, though.

[Terranigma]

tritical, thanks for keeping me informed. I can only imagine the possibilities this new filter will be able to achieve (Specifically when it comes to aliasing/deinterlacing). :D

[audyovydeo]

@tritical

This is just a word of appreciation as I really like TDeint. I've read around the forums and seen some speed comparisons with other deinterlacers. I dont do comparative testing as it's precious time taken off real encoding.
I find TDeint is a real workhorse : I have two avs scripts each TDeinting two DV_PAL.avi files and layering them, both scripts fed into Adobe Premiere for yet another transparency, and it works like a beauty (on a T5200 cpu). That means very good workload scaling, which to me is more important than absolute speed.

So, for me, every minute you spend on multithread optimization is wecome.

cheers
audyovydeo

[zdark]

I am having certain lag in slow scenes. If I to use...telecide() for But its filter it is better, I changed setting.

source original: DVD NTSC 720x480
my script for desintrelaced

interp = separatefields().eedi2(field=-2)
deinted = tdeint(mode=2,edeint=interp,type=3)
TFM(clip2=deinted,order=-1,mode=5,pp=7,field=-1,slow=2)

now its fine?

is for anime Vision of escaflowne

You can say me because she is giving lag?

http://rapidshare.com/files/33066521/teste2.mp4.html

[Leak]

source original: DVD NTSC 720x480
[...]
interp = separatefields().eedi2(field=-2)
deinted = tdeint(mode=2,edeint=interp,type=3)
TFM(clip2=deinted,order=-1,mode=5,pp=7,field=-1,slow=2)
Since I very much doubt Escaflowne was animated at 30 FPS - where's your call to TDecimate?

Your current script will result in eveyr fifth frame being a duplicate.

Also, what's the deal with "fansubbing" something that's been released all over the world (at least Japan, America, Canada, UK, Germany and France) for years? :mad:

np: Legiac - Dide Skin (Mings Feaner)

[zdark]

I taste of you liven up old. I go to test adding the TDecimate!

[zdark]

leak my test.

no filter: http://rapidshare.com/files/33178778/escaflowne_no_filter.mp4.html
filtered: http://rapidshare.com/files/33175940/escaflowne_filter.mp4.html

now its fixed.

I used

edeintted = last.AssumeTFF().SeparateFields().SelectEven().EEDI2(field=-1)
deinted = TDeint(order=1,edeint=edeintted)
TFM(clip2=deinted,order=-1,mode=5,pp=7,field=-1,slow=2).tdecimate(hybrid=3)



thanks for help!

[DarkNite]

Also, what's the deal with "fansubbing" something that's been released all over the world (at least Japan, America, Canada, UK, Germany and France) for years?

Well, it may be that the translation was weak and erroneous (if it's anything like the English translation) or that it may not be available in his native language.

Or, it may just be that karaoke scripts are indeed that important to some people. :)

[tritical]

Haven't been able to reproduce the mode 5 problem, so I'm just gonna go ahead and release a new version with the fixes so far. [link removed], changes:
tdecimate:
- Lots of fixes for mode 2 and non-standard framerates
- Fixed problem in mode 0 decimation when using tfm hints and looking for
singleton match duplicates

tfm:
- Fixed incorrect handling of the top/bottom lines in cubic post-processing

[Terranigma]

You know the drill ;) :goodpost:

[zdark]

tritical thanks, good job!

[plugh]

An exclusion band in tdecimate could be accomplished by using the clip2 parameter:

mpeg2source()
tfm()
saved = last
stackvertical(crop(),crop()) # set crops to remove lines containing subtitles
tdecimate(clip2 = saved)

In this case, tdecimate will calculate everything based on the input clip, but will output frames from clip2. The clip2 clip only has to have the same number of frames as the input. All other properties: height, width, and colorspace can be different. You could replace cropping/stackvertical with another method of removing the subtitles (set the subtitle area to black using overlay, etc...)..

If I may inject a question:

In a thread in the avisynth usage forum, I'm trying to use tdecimate on an animated film that was sped up slightly for broadcast. One of the problems I'm having is (I believe) related to vertical film jitter; the animation has its own dups, but frame to frame film jitter means they don't always align perfectly leading to higher difference metrics.

How can I get the dup matching code to search up/down a few lines to find the min difference metric? That is, given current frame "C" and previous frame "P" calc metrics between

lines C 1 thru N vs lines P 1 thru N as usual,
but also
lines C 1 thru N-1 vs lines P 2 thru N
lines C 1 thru N-2 vs lines P 3 thru N
and
lines C 2 thru N vs lines P 1 thru N-1
lines C 3 thru N vs lines P 1 thru N-2

and use the min metric for the dup check.

BTW, I also encountered a mode 2 rate dependant accvio with this movie. I see your latest version addresses this - I'll give it a whirl.

Thanks!

[ChiDragon]

plugh, I don't get why you'd want it to detect these animation duplicates as dups and decimate them. If anything the higher metrics caused by the jitter should help you only remove the telecine dups.

[plugh]

The jitter isn't consistent. Sometimes it is there, sometimes it isn't. If they aren't detected as dups, then it seems as though mode 1 can't properly apply 'longest run' (and with 720p source at 60fps, there are long runs of dups).

Thank you for your interest. Do you have a suggestion on how to do this type of matching? I was reading about the FrameDiff function, and the post I quoted gave me the idea that there might be some way to tackle this. But 1) I'm not proficient at avisynth scripting, and 2) perhaps tcritical knows some trick in his code that will do this (or could)...

Thanks again.

[plugh]

In an effort to understand and verify the impact of the film jitter on the dup matching, after much RTFMing and experiments I came up with the following (probably horribly inefficient) script which uses tcriticals' CFrameDiff function to evaluate the current frame against vertically offset previous framessep=" "
file = "i:\work5\dupinfo.log"
c=mpeg2source("lk1.d2v",cpu=0)
c=converttoyuy2(c)
c=WriteFile(c, file, "current_frame","sep","u2","sep","u1","sep","at","sep","d1","sep","d2")
c=FrameEvaluate(c, "crop(0,0,0,-4).trim(0,current_frame-1)+crop(0,2,0,-2).trim(current_frame,0)"+chr(13)+"global u2 = CFrameDiff(norm=true)")
c=FrameEvaluate(c, "crop(0,1,0,-3).trim(0,current_frame-1)+crop(0,2,0,-2).trim(current_frame,0)"+chr(13)+"global u1 = CFrameDiff(norm=true)")
c=FrameEvaluate(c, "crop(0,2,0,-2).trim(0,current_frame-1)+crop(0,2,0,-2).trim(current_frame,0)"+chr(13)+"global at = CFrameDiff(norm=true)")
c=FrameEvaluate(c, "crop(0,3,0,-1).trim(0,current_frame-1)+crop(0,2,0,-2).trim(current_frame,0)"+chr(13)+"global d1 = CFrameDiff(norm=true)")
c=FrameEvaluate(c, "crop(0,4,0, 0).trim(0,current_frame-1)+crop(0,2,0,-2).trim(current_frame,0)"+chr(13)+"global d2 = CFrameDiff(norm=true)")
return(c)

which I ran on the opening sequence of my 'problem' film. The opening consists of a black background and red stationary letters which fade in and back out (no scroll, pan, or zoom). The following (editted for formatting) is a portion of the script output, which clearly shows the jitter and it's impact on the dup metric (middle column is the non-shifted metric) 29 17.001106 9.122610 0.000000 9.624715 18.472639
30 17.005119 9.123948 0.305900 9.626945 18.470409
31 8.893853 1.430508 9.107002 17.618704 25.334885
32 16.609142 8.854167 0.000000 9.027629 17.556274
33 16.600225 8.830979 0.531535 9.004441 17.549141
34 23.565033 16.905233 9.010684 0.497200 9.523937
35 16.892748 9.003550 0.366991 9.539098 18.390589
36 16.899437 9.008454 0.141356 9.540436 18.392818
37 16.897207 9.009792 0.472674 9.546233 18.392372
38 16.891855 9.009792 0.520387 9.545341 18.389698
39 16.891855 9.009792 0.131100 9.545341 18.389698
40 16.891855 9.009792 0.663973 9.545341 18.389698
41 16.891855 9.009792 0.051727 9.545341 18.389698
42 16.891855 9.009792 0.223851 9.545341 18.389698
43 16.885612 9.008009 0.367437 9.543557 18.387022
44 16.882936 9.002658 0.085616 9.538206 18.380779
45 8.827412 0.139127 9.002658 17.533087 25.325075
46 16.597103 8.827412 0.140910 9.002658 17.533087
47 23.467377 16.760756 8.859072 1.454142 9.607770
48 16.890518 8.935324 0.127087 9.490047 18.414669
49 16.896315 8.933540 0.127087 9.491831 18.411993
50 16.896315 8.937553 0.059307 9.493169 18.415115
51 16.896315 8.937553 0.057078 9.493169 18.415115
52 16.896315 8.937553 0.021404 9.493169 18.415115
53 16.896315 8.937553 0.000000 9.493169 18.415115
54 16.896315 8.937553 0.000000 9.493169 18.415115
55 16.896315 8.937553 0.000000 9.493169 18.415115
56 16.896315 8.937553 0.000000 9.493169 18.415115
57 16.896315 8.937553 0.000000 9.493169 18.415115
58 16.890518 8.932649 0.296982 9.484696 18.396832
59 16.884722 8.934878 0.000000 9.488709 18.390144
60 16.884722 8.934878 0.074468 9.488709 18.390144
61 8.926851 1.258383 8.865761 17.437660 25.249269
62 16.601563 8.854167 0.000000 9.027629 17.555384
63 16.602455 8.850599 0.572114 9.026737 17.558950
64 23.569492 16.901220 9.017373 0.302779 9.555597
65 16.890965 9.003996 0.000000 9.546233 18.386129
66 16.886505 9.003103 0.074914 9.553368 18.386129
67 16.890965 9.006671 0.117277 9.542665 18.381670
68 16.890965 8.997752 0.000000 9.539990 18.386129
69 16.890965 8.997752 0.111034 9.539990 18.386129
70 16.886505 8.991509 0.084725 9.534639 18.384346
71 16.888288 8.987943 0.000000 9.530180 18.383455
72 16.888288 8.987943 0.049051 9.530180 18.383455
73 16.885612 8.992401 0.070009 9.535531 18.382563
74 16.888288 8.997752 0.000000 9.539990 18.383455
75 16.888288 8.997752 0.041025 9.539990 18.383455
76 16.890072 8.997752 0.058861 9.544449 18.384346
77 16.890965 9.002212 0.000000 9.544449 18.386129
78 16.890965 9.002212 0.028093 9.544449 18.386129
79 8.972335 1.400185 8.910353 17.450592 25.309467
80 16.681828 8.883597 0.000000 9.057952 17.646797
81 16.678707 8.892962 0.112372 9.062857 17.645458
82 23.489672 16.706354 8.810467 1.412225 9.757599
83 16.869560 8.952269 0.000000 9.469089 18.372753
84 16.852169 8.947364 0.178814 9.475778 18.369631
85 16.857965 8.955836 0.325967 9.457495 18.347780
86 16.857965 8.949594 0.000000 9.466413 18.361158
87 16.859749 8.951377 0.114155 9.468197 18.357592
88 16.868668 8.949594 0.233216 9.461954 18.372753
89 16.867777 8.948702 0.000000 9.465522 18.370970
90 16.867777 8.948702 0.101224 9.465522 18.370970
91 16.933773 9.023616 0.807559 9.457495 18.341537
92 8.798427 0.397314 8.938891 17.521938 25.271564
93 16.578375 8.795752 0.345141 8.941566 17.532640
94 23.573059 16.888288 8.977241 0.266660 9.506992
95 16.885612 8.963417 0.000000 9.511451 18.387468
96 16.883829 8.962525 0.047267 9.510559 18.385685
97 16.883383 8.960296 0.131992 9.506546 18.389698
98 16.889181 8.962525 0.000000 9.510559 18.391035
99 16.889181 8.962525 0.049943 9.510559 18.391035
100 16.890965 8.964309 0.083387 9.515018 18.391928
101 16.891855 8.968768 0.000000 9.516802 18.393711
102 16.891855 8.968768 0.019620 9.516802 18.393711
103 16.893639 8.972335 0.060645 9.520370 18.393711
104 16.893639 8.973228 0.000000 9.521261 18.395494
105 16.893639 8.973228 0.037457 9.521261 18.395494
106 8.951823 1.404199 8.871112 17.413136 25.279591
107 16.641695 8.820277 0.000000 8.976794 17.601313
108 16.645262 8.826520 0.063320 8.980362 17.606665
109 16.646601 8.822061 0.104345 8.983037 17.607109
110 23.477633 16.704124 8.805116 1.358269 9.680009
111 16.858858 8.943796 0.155180 9.464184 18.346889
112 16.864208 8.942904 0.277808 9.463292 18.343323
113 16.860195 8.941566 0.000000 9.461954 18.345552
114 16.860195 8.941566 0.182381 9.461954 18.345552
115 16.860195 8.941566 0.285834 9.461954 18.345552
116 16.860195 8.941566 0.000000 9.461954 18.345552
117 16.860195 8.941566 0.111926 9.461954 18.345552
118 16.858412 8.940675 0.073131 9.457495 18.341093
119 16.855736 8.938000 0.000000 9.458386 18.341093
120 16.855736 8.938000 0.050389 9.458386 18.341093
121 16.904787 8.992401 0.869988 9.332638 18.212667
122 16.764769 8.830087 0.000000 9.356717 18.253693
123 16.754066 8.819385 0.838328 9.339772 18.286690
124 16.777700 8.822952 0.162315 9.373662 18.305418
125 16.782606 8.844802 0.000000 9.385702 18.307648
126 16.782606 8.844802 0.045484 9.385702 18.307648
127 16.781713 8.847478 0.109696 9.397296 18.315676
128 16.791525 8.852829 0.000000 9.396404 18.318350
129 16.807577 8.865314 0.045484 9.397296 18.314783
130 8.812250 1.475546 8.881814 17.469320 25.242580
131 16.695652 8.853721 0.000000 9.028075 17.658836
132 16.694759 8.857288 0.221622 9.021832 17.658836
133 16.691193 8.849261 0.185948 9.030750 17.663296
134 16.695652 8.853721 0.000000 9.028075 17.658836
135 23.490564 16.802671 8.834546 1.364066 9.674212
136 16.913260 8.971443 0.074023 9.495399 18.441423
137 16.913260 8.971443 0.000000 9.495399 18.441423
138 16.913260 8.971443 0.035674 9.495399 18.441423
139 16.916826 8.966092 0.177030 9.486480 18.437857
140 16.911476 8.966092 0.000000 9.490939 18.437857
141 16.913260 8.963417 0.049943 9.490047 18.437857
142 16.908800 8.965200 0.109250 9.492723 18.440533
143 16.907909 8.966092 0.000000 9.490939 18.434290
144 16.907909 8.966092 0.075806 9.490939 18.434290
145 16.909246 8.967430 0.075806 9.491385 18.431168
146 16.907017 8.966092 0.000000 9.489155 18.429831
147 16.907017 8.966092 0.028539 9.489155 18.429831
148 16.907017 8.966092 0.020512 9.489155 18.429831
149 16.907017 8.966092 0.000000 9.489155 18.429831
150 16.907017 8.966092 0.034782 9.489155 18.429831
151 16.907017 8.966092 0.818261 9.489155 18.429831
152 8.831425 1.478221 8.898759 17.474672 25.225636
153 16.578821 8.812250 0.465985 8.980807 17.546465
154 16.578375 8.816263 0.243918 8.975456 17.537546
155 23.445526 16.752729 8.821169 1.395280 9.590825
156 16.902557 8.950485 0.075360 9.480682 18.391481
157 16.903450 8.951377 0.155626 9.480682 18.391481
158 16.900774 8.948702 0.000000 9.478899 18.390589
159 16.899881 8.948702 0.101670 9.478007 18.389698
160 16.899881 8.947809 0.071347 9.478007 18.389698
161 16.899881 8.947809 0.000000 9.478007 18.389698
162 16.899881 8.947809 0.028093 9.478007 18.389698
163 16.899881 8.947809 0.061091 9.478007 18.389698
164 16.899881 8.947809 0.000000 9.478007 18.389698
165 16.899881 8.947809 0.026755 9.478007 18.389698
166 16.897654 8.942904 0.132884 9.467752 18.384792
167 16.889181 8.935324 0.000000 9.465522 18.378996
168 16.889181 8.935324 0.063320 9.465522 18.378996
169 16.897207 8.949594 0.308130 9.445901 18.353132
170 16.887842 8.925068 0.000000 9.452144 18.378103
171 8.778360 0.121290 8.932649 17.531303 25.325521
172 16.603792 8.785941 0.099440 8.922392 17.517481
173 23.601599 16.879816 8.924176 0.089184 9.446793
174 16.881599 8.922392 0.089184 9.449469 18.371861
175 16.880262 8.926406 0.065104 9.448131 18.375874
176 16.884275 8.926851 0.000000 9.453928 18.374537
177 16.886505 8.926406 0.070901 9.454373 18.374983
178 16.886951 8.926851 0.035228 9.453928 18.377213
179 16.886951 8.926851 0.000000 9.453928 18.377213
180 16.886951 8.926851 0.021404 9.453928 18.377213
181 16.894531 8.980807 1.240992 9.342002 18.245667
182 16.768335 8.822061 0.000000 9.369649 18.282677
183 16.808023 8.876908 0.541791 9.366528 18.268854
184 16.777700 8.829641 0.197096 9.382581 18.296501
185 8.801994 1.439872 8.876908 17.434093 25.210920
186 16.674248 8.835884 0.132884 9.004441 17.635649
187 16.678707 8.834101 0.102115 9.007117 17.632973
188 23.485214 16.761648 8.834101 1.318136 9.659050
189 16.888735 8.941121 0.072685 9.481575 18.413332
190 16.893194 8.952715 0.061091 9.473548 18.407089
191 16.886951 8.944688 0.000000 9.479791 18.412439
192 16.886059 8.942904 0.042808 9.479791 18.410656
193 16.886505 8.945134 0.152504 9.472210 18.407534
194 16.882492 8.941121 0.000000 9.475331 18.405304
195 16.879816 8.938445 0.092751 9.475331 18.404413
196 16.879816 8.938445 0.105683 9.473548 18.405304
197 16.879816 8.938445 0.000000 9.473548 18.405304
198 16.879816 8.938445 0.054402 9.473548 18.405304
199 16.879816 8.938445 0.042808 9.473548 18.405304
200 16.879816 8.938445 0.000000 9.473548 18.405304
201 16.879816 8.938445 0.021404 9.473548 18.405304
202 8.798427 0.161869 8.917933 17.520155 25.292522
203 23.536940 16.863317 8.924176 0.157855 9.488709
204 16.878923 8.942013 0.034782 9.477116 18.404413
205 16.878923 8.942013 0.034782 9.477116 18.404413
206 16.878923 8.942013 0.000000 9.477116 18.404413
207 16.878923 8.942013 0.028539 9.477116 18.404413
208 16.879370 8.942458 0.028539 9.477561 18.403967
209 16.879370 8.942904 0.000000 9.478454 18.403521
210 16.878923 8.942458 0.018729 9.478007 18.403967
211 16.865992 8.942458 0.513253 9.470427 18.378103
212 16.839684 8.918379 0.000000 9.456158 18.359821
213 16.872681 8.943351 0.648366 9.388377 18.300068
214 16.797321 8.855059 0.108804 9.396404 18.313000
215 16.792416 8.867990 0.000000 9.406660 18.310770
216 16.792416 8.867990 0.047713 9.406660 18.310770
217 16.798658 8.871557 0.253728 9.408443 18.316120
218 16.804010 8.873342 0.000000 9.412011 18.322363
219 8.829195 1.440764 8.871112 17.425175 25.205568
220 16.671572 8.841681 0.329088 9.001766 17.619595
221 16.662207 8.830533 0.000000 8.995077 17.610231
222 16.664883 8.830533 0.135559 8.995969 17.612907
223 23.627907 16.955622 8.971443 0.342466 9.528842
224 16.968554 8.987050 0.000000 9.507884 18.472193
225 16.972120 8.989726 0.105237 9.506100 18.470409
226 16.966770 8.976349 0.125303 9.508776 18.480219
227 16.965878 8.989726 0.000000 9.510559 18.469517
228 16.965878 8.989726 0.050835 9.510559 18.469517
229 16.964987 8.990618 0.151167 9.512343 18.470409
230 16.965878 8.987943 0.000000 9.508776 18.469517
231 16.965878 8.987943 0.034782 9.508776 18.469517
232 16.965878 8.987943 0.051727 9.508776 18.469517
233 16.965878 8.987943 0.000000 9.508776 18.469517
234 16.965878 8.987943 0.025863 9.508776 18.469517
235 16.965878 8.987943 0.047713 9.508776 18.469517
236 16.965878 8.987943 0.000000 9.508776 18.469517
237 16.965878 8.987943 0.018729 9.508776 18.469517
238 16.964987 8.987050 0.168557 9.511451 18.466841
239 8.920609 1.451020 8.909461 17.604879 25.417826
240 16.713488 8.907231 0.101670 9.061519 17.660173
241 16.715271 8.906340 0.717484 9.063303 17.657499
242 23.522671 16.802671 8.948702 1.434075 9.559610
243 16.783052 8.868436 0.312589 9.417362 18.320580
244 16.791969 8.863976 0.120398 9.406660 18.302744
245 16.789295 8.872004 0.000000 9.411119 18.308094

[plugh]

buglet report:

In the process of working with CFrameDiff, I discovered the following:

As expected, (mode=0,norm=false) and (mode=1,norm=false) return radically differant values.
However, (mode=0,norm=true) and (mode=1,norm=true) give IDENTICAL results.

Appears you can't get the normalized 'mode=min' values.


And FWIW, building upon my previous post, I put together a 'vertical jitter correction' script, which I have posted in the other forum here (http://forum.doom9.org/showthread.php?p=1009932#post1009932)

[WorBry]

I actually find the idea behind yadif quite interesting, but the major drawback IMO is the edi method it is uses is overly prone to artifacts. It uses the typical two sliding window method which is highly likely to choose the wrong direction around anything more than a single, thick, lone edge. Every other filter/program that I know of that uses this method caps the output value to be within +-2 or 3 of the min/max of the vertical neighbors to prevent major artifacts. Just to see what would happen, I created a c only version with such capping and to me it looked much better. I also made it possible to take spatial predictions from an external clip. I have some results here: deinterlace_comparison.txt (http://bengal.missouri.edu/~kes25c/deinterlace_comparison.txt) from a comparison I am working on, which I think show the benefits.


Hi Tritical,

The metrics obtained with your modified version of Yadif are very interesting.

Is it possible that the potential interpolation errors introduced by not 'capping' the output might explain the 'vertical jittering' (shimmering) that Yadif seems (by my observation at least) quite susceptible to, or is that a quite different phenomenon?

http://forum.doom9.org/showthread.php?p=1010873#post1010873

[tritical]

@plugh
Yep, it is indeed a bug (actually a missing return statement). As for using the minimum of metrics calculated with different vertical offsets, the method you came up with is probably the simplest. There isn't an easy way to modify tdecimate to do this internally.

@WorBry
Is it possible that the potential interpolation errors introduced by not 'capping' the output might explain the 'vertical jittering' (shimmering) that Yadif seems (by my observation at least) quite susceptible to, or is that a quite different phenomenon?
I think they are slightly different. The artifacts I'm talking about stem from running yadif's type of edi interpolation on something like:

006 202 004 200 004 199 004
o
003 200 005 204 005 198 003

which is an extreme case. However, it would choose to replace 'o' with 4, while it should simply use vertical interpolation. This causes problems mainly around fine lines and details (thick edges are usually fine).

Could you describe more what you mean by 'vertical jittering' or 'vertical shimmering'. I think I know what you are talking about but just want to be sure before writing a lengthy reply :p.

[WorBry]

Could you describe more what you mean by 'vertical jittering' or 'vertical shimmering'. I think I know what you are talking about but just want to be sure before writing a lengthy reply :p.

Sure. I'm referring to a slight jumping up and down ('over-bobbing' you might say) of relatively static objects from frame to frame, resulting in a vertical jitter or shimmering type effect. Most noticable with fine detail - patterns, lettering etc.

There are good examples in the src reference clips that you have been using in your comparison tests, for example the pattern on the shirt of the piano player in the 'Musicians' Clip' and in the 'Toy Train' clip, the sheep on the wall-paper and the dates on the calender.

I'm also posting a short home DV clip that shows the effect quite well

http://rapidshare.com/files/35553085/Shimmer_example_PAL_DV_Type_II.avi

If you run it through Yadif (Mode=1, Order=0) you'll note the marked 'shimmering' of the toy car mat. With MCBob, it is effectively 'calmed-down'.

[plugh]

the method you came up with is probably the simplest. There isn't an easy way to modify tdecimate to do this internally.
OK, thanks.

The mode4 / mode2 combo with the de-jittered stream has produced the best decimation results so far, but it still isn't quite right. I *have* identified the telecining pattern used in this 720p broadcast - 576 film frames into 1409 video frames, which works out to 59.94*(576/1409) = 24.503506032, which yeilds
rate = 24.503506 actual rate = 24.503515
mode2_num = 1 mode2_den = 2 numCycles = -20 clength = 12
mode2_cfs 0 = 12
mode2_cfs 1 = 132
mode2_cfs 2 = 3564
mode2_cfs 3 = 99792
or (60000/1001)*(576/1409) = 24.503530536, which yeilds
TDecimate: mode 2 error, number of frames after decimation doesn't match!
Specifically - the dup counts are
212121212121212121 45->18
112121212121212121 44->18
112121212121212121 44->18
202121212121212121 44->18
202121212121212121 44->18
211121212121212121 44->18
211121212121212121 44->18
212021212121212121 44->18
212111212121212121 44->18
212111212121212121 44->18
212120212121212121 44->18
212121112121212121 44->18
212121112121212121 44->18
212121112121212121 44->18
212121202121212121 44->18
212121211121212121 44->18
212121211121212121 44->18
212121212021212121 44->18
212121212111212121 44->18
212121212111212121 44->18
212121212120212121 44->18
212121212120212121 44->18
212121212121112121 44->18
212121212121112121 44->18
212121212121202121 44->18
212121212121211121 44->18
212121212121211121 44->18
212121212121212021 44->18
212121212121212021 44->18
212121212121212111 44->18
212121212121212111 44->18
212121212121212120 44->18
The most notable 'feature' of the resulting decimation (using 24.503506032) is that sometimes tdecimate eliminates three frames in a row (you can see runs of three telecine dups never occur in the above pattern) and thus ends up including dups elsewhere (see attached images - you can clearly see where dups are getting through). Thus, screen motion is jerky in places.

So my question is, how can I get this 'right', short of hand crafting an ovr file? Are the decimation cycles being derived by mode2 not quite correct? Is there any way to tell it 'max dup length=2' (if that would help)? Or am I doing something fundamentally wrong :confused:

Thanks in advance

[tritical]

@plugh
Could you post the metrics file? Or just send it to my ftp:

12.216.251.99:17262
upload/upload

[tritical]

@WorBry
Here are my results on the clip you posted:

yadif_test.zip (http://bengal.missouri.edu/~kes25c/yadif_test.zip)

The four clips included are:

yadif-type0 - same as yadif(mode=1,order=0)
yadif-type1 - w/ capped edi
yadif-edeint0 - w/ tdeint's temporally switched kernel interpolation for spatial prediction
yadif-edeint1 - w/ eedi2 for spatial prediction

[plugh]

@plugh
Could you post the metrics file? Or just send it to my ftp:


You got it - lk2-try5-tdout.txt
...

So, I thought I'd try something simpler...

Noting the basic blocks of 44 pattern, I decided to try to eliminate just the '2's - ie remove 8 from 44.

Thus:
tdecimate(mode=1,cycleR=8,cycle=44,sdlim=-2,hint=false,blockx=64,blocky=64,nt=1,input=base+tag+"-tdout.txt",batch=true,debug=true)

No joy. For example
TDecimate: 2376: 6.07 0.14 0.10 11.20 0.34 20.93 0.83 23.67 0.15 23.06 0.82 0.13 20.12 0.44 17.25 0.15 0.08 20.86 0.15 18.62 0.14 0.15 27.64 0.34 25.47 0.81 0.39 26.08 0.51 31.60 0.81 0.08 30.27 0.28 25.64 0.19 0.22 16.07 0.41 20.31 0.12 0.30 29.53 0.50
TDecimate: 2376: 0 1 1 0 1 0 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1
TDecimate: 2376: x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
TDecimate: 2420: 28.05 0.44 0.19 30.24 1.34 29.10 0.21 25.14 0.66 19.43 0.36 0.06 20.21 0.14 39.44 0.20 0.17 59.02 0.17 54.47 0.06 0.23 56.35 0.87 32.10 0.21 0.08 31.95 0.22 36.14 0.18 0.32 38.80 0.36 54.67 0.12 0.39 57.76 0.49 54.34 0.51 0.51 58.11 0.49
TDecimate: 2420: 0 1 1 0 0 0 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1
TDecimate: 2420: x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
TDecimate: 2464: 0.49 0.12 0.13 50.22 0.12 1.44 0.02 0.05 3.34 1.04 0.07 0.10 3.36 0.05 0.13 0.03 0.05 3.41 0.74 1.42 0.07 0.03 3.03 0.17 1.15 0.06 0.26 3.27 0.19 1.30 0.02 0.04 2.93 0.06 1.30 0.10 0.04 3.01 0.13 1.08 0.05 0.04 2.92 0.06
TDecimate: 2464: 1 1 1 0 1 0 1 1 0 1 1 1 0 1 1 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 1 1 1 0 1
TDecimate: 2464: x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
TDecimate: 1980: Dropping Frames: 2422 2431 2436 2440 2446 2450 2455 2463
to do what I wanted, it should have dropped 2461, not 2463

I'm guessing it got confused by 2462 being followed by 4 dups - a telecine dup, a FILM dup, then two telecine dups.
But that is why I specified sdlim. Oh, sdlim only kicks in for mode1 if not enough dups found.
Uh, wouldn't it make more sense for it to kick in if too many dups are found, so it could enforce a spacing?

EDIT: hmmm. no, this really needs pattern guidance.
Note the seventh and eighth lines in the pattern above
211121212121212121 44->18
212021212121212121 44->18
Here is my breakdown of the above debug output
TDecimate: 2420: 0 1 1 0 0 0 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1
F d d F d F d F d F d d F d F d d F d F d d F d F d d F d F d d F d F d d F d F d d F d
2 1 1 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1

TDecimate: 2464: 1 1 1 0 1 0 1 1 0 1 1 1 0 1 1 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 1 1 1 0 1
D d d F d D d d F D d d F d D d d F d D d d F d D d d F d D d d F d D d d F d D d d F d
2 1 2 0 2 1 2 1 2 1 2 1 2 1 2 1 2 1
('F' is Film, 'D' is film dup, and 'd' is telecine dup)
threshhold dup longest run based algo, even with a full blown sdlim constraint probably can't cope with those film dups on top of the telecine dups.

EDIT (again): OOPS - it is '8 out of 44' only in the blocks with '111' in the pattern. All others have 9 dup pairs. Back to the drawing board...

[plugh]

a stream of consciousness / thought experiment...
-----------------------------------------------------
Given a telecine pattern input, and a set of frame metrics:

assume pattern is expressed as say, 0,1,1,0,... (0=frame, 1=dup)

convert metrics to pattern form using threshold test
(eg less than threshold = -1, more = 0)

pattern match

pat_end = pat_lng-1
for i = 0 to pat_end
for j= 0 to pat_end
correlation(i)= pattern( mod( i+j, pat_lng) ) + metric(j)

This tests every possible rotation of the pattern against the metric.
A perfectly aligned match would give a correlation value of zero.

hmmm. but so would an 'off by one' alignment
pattern = 1 0 1 0 1 0
metric = 0-1 0-1 0-1

perhaps metric should be tri-state (like mode7)
-1 = definite dup, 1 = definite non-dup, 0 = indeterminate?

then pattern is expressed same way
-1 = dup, 1 = non-dup (no zeroes?)

[perhaps 'pattern zeroes' allows fuzziness later?]
[[hmmm... 'fuzzy logic' & pattern matching - where is that text from school?]]


[thoughts -
other choices of 1/0/-1 mapping?
choice of correlation computation? (SAD? SSD?, stats 'correlation coefficient'?)
perhaps binary representation and XOR? ]

Continuing---

With 'best' correlation established, and rotation of pattern aligned,
start decimation process, 'rolling' the pattern along the input.

hmmm. What about pattern breaks? (commercial edits, etc)?

OK, instead of simply rolling the pattern along the input,
at each frame you find the 'best' correlation.

If there was no pattern break, the current rotation should
provide the best correlation.

Yuck - but as you approach the break from upstream, your current 'correct'
rotation of the pattern produces increasingly worse correlation.

So we need a way to recognize discontinuities: hmmm -

-Behind the discontinuity, for the preceding pat_len frames,
we are 'highly' correlated with the 'old' pattern alignment.
-In front of the discontinuity, for the following pat_len frames,
we are 'highly' correlated with the 'new' pattern alignment.
-The discontinuity causes a decrease in correlation as you
approach it (from either side) with a given pattern alignment.
-As you approach the discontinuity, NO (?) pattern alignment will
give a 'good' correlation (what is 'good'?)

Hmmm. perhaps a 'lookahead'?

Given a correlated pattern alignment at frame/metric(n),
ie pattern matches metric(n) through metric(n+pat_lng)
at each frame we check the current alignment against
metric(n + pat_lng) through metric(n +pat_lng +pat_lng)

ie Does the current pattern alignment correlate for two cycles?

If it matches metric n thru n+pat_lng but not n+pat_lng thru n+2*patlng
you have found a discontinuity / pattern break . You need to re-sync
the pattern alignment when you get to that point !


SO - the algo becomes

at the start, you align / correlate pattern to metric.

At each frame, you check to see if the current alignment is
still 'best choice' one full cycle ahead of your current position.

If so, you continue rolling the pattern along the frames.

If not, you finish the current cycle up to the discontinuity,
and start over with new 'best choice' at the new position.

Cool! I think that would work!

Requires input of guidance pattern,
and ability to look ahead two full cycles.

Just leaves choice of 'pattern matching' / correlation algo TBD

Where is that fuzzy logic textbook?
Should also review the stats textbook.
The key is the pattern matcher!!!

[plugh]

576 film frames into 1409 video frames, which works out to 59.94*(576/1409) = 24.503506032, which yeildsrate = 24.503506 actual rate = 24.503515
mode2_num = 1 mode2_den = 2 numCycles = -20 clength = 12
mode2_cfs 0 = 12
mode2_cfs 1 = 132
mode2_cfs 2 = 3564
mode2_cfs 3 = 99792
or (60000/1001)*(576/1409) = 24.503530536, which yeilds TDecimate: mode 2 error, number of frames after decimation doesn't match!

I trim()'d a few frames off the end, deleted the same number from end of metrics file, and second case now yeildsdrop count = 174539 expected = 174539
rate = 24.503531 actual rate = 24.503535
mode2_num = 1 mode2_den = 2 numCycles = 24603 clength = 12
mode2_cfs 0 = 12
mode2_cfs 1 = 132
mode2_cfs 2 = 3564
mode2_cfs 3 = 103356

Which is interesting in several ways.

1) That's how to 'fix' that error condition

2) numCycles is a positive number this time (?)

3) an extra line of debug output that wasn't present before

Seems obvious to me now, but the correct way to calculate the tdecimate target frame rate is in terms of the rate passed in from mpeg2source - in this case 60000/1001 = 59.94005994...

Don't expect it will make a big difference, but we'll see...

[plugh]

I noticed some puzzling results from a script that uses cframediff.

Dug into it, and eventually distilled it down to the following which illustrates the behaviour.

Version().converttoyuy2()

frameevaluate("NOP()")

Writefile("avstst1.txt"\
, "((current_frame==3) && (cframediff(prevf=false) >= 0))"\
, "current_frame"\
)


This generates the following output:
false0
false1
false2
true4
false4
false5
false6
false7

If prevf is set true, you get correct output (ie 'true3').
If the frameevaluate is deleted, you get correct output.

a cframediff buglet?
avisynth weirdness??
stupidity on my part???

Is this cosmetic, or is something getting fundamentally borked?

My application script uses both prevf=true and prevf=false, uses writefileIF rather than writefile, and has a number of scriptclips rather than the frameevaluate. It also has a fair number of references to 'current_frame' other than in the output statement.

I am using avisynth 2.5.6 and tivtc 1.0.2

EDIT: Oh yeah, forgot to ask...

When I look at the debug output of, for example, tdecimate(mode=0), part of the display are some number that I assume are 'normalized framediff metrics'. I also assume that the results from cframediff should be the same as those.

They are not. With identical args (blockx, blocky, chroma, nt) I get somewhat similar, but not identical results.

Are one of my assumptions incorrect?

For example:
TDecimate: 968: 0.02 0.03 0.03 0.01 2.54 0.04 0.09 0.04 1.09 0.08 0.12 0.09 0.02 2.55 0.00 0.03 0.14 0.01 0.76 0.08 0.00 0.08 0.05 1.46 0.02 0.03 0.06 0.01 0.62 0.00 0.02 0.12 0.01 2.71 0.05 0.01 0.16 0.05 1.56 0.03 0.04 0.16 0.01 1.77
TDecimate: 968: x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
TDecimate: 792: Dropping Frames: 969 974 978 983 987 993 997 1003
output from a script (related to above) -
first value is prevf=true, second is prevf=false
(I think; as you can see 'true' output is borked,
so I'm not sure what I'm *really* getting, but
the first column is similar to the above)
true969 - 0.137566 0.115047
false969 - 0.137566 0.115047
false970 - 0.115047 0.063989
false971 - 0.063989 2.397595
false972 - 2.397595 0.086843
true974 - 0.208690 0.134556
false974 - 0.208690 0.134556
false975 - 0.134556 1.227280
false976 - 1.227280 0.184499
false977 - 0.184499 0.239236
true979 - 0.178479 0.062206
false979 - 0.178479 0.062206
false980 - 0.062206 2.563254
false981 - 2.563254 0.001561
false982 - 0.001561 0.115493
true984 - 0.266325 0.045261
false984 - 0.266325 0.045261
false985 - 0.045261 0.952706
false986 - 0.952706 0.173351
false987 - 0.173351 0.000000
true989 - 0.185837 0.134333
false989 - 0.185837 0.134333
false990 - 0.134333 1.654136
false991 - 1.654136 0.104568
false992 - 0.104568 0.128536
true994 - 0.180597 0.069563
false994 - 0.180597 0.069563
false995 - 0.069563 0.793401
false996 - 0.793401 0.000000
false997 - 0.000000 0.088180
true999 - 0.270673 0.060756
false999 - 0.270673 0.060756
false1000 - 0.060756 3.016084
false1001 - 3.016084 0.183273
false1002 - 0.183273 0.029654
true1004 - 0.237340 0.162092
false1004 - 0.237340 0.162092
false1005 - 0.162092 1.813664
false1006 - 1.813664 0.117277
false1007 - 0.117277 0.149383
true1009 - 0.245924 0.060756
false1009 - 0.245924 0.060756
false1010 - 0.060756 2.123355
false1011 - 2.123355 0.041470
false1012 - 0.041470 0.095538

[dude051]

I got a few questions about the two pass vfr function of TIVTC, and have searched the entire day but found no concrete answers. I am interested in how the vfr function exactly outputs its frames, I am used to Decomb512VFR so bare with me on this. My source is an anime DVD just for reference.

I realize that in using the two pass method, it makes the video seekable by use of the metric files, and the output uses non-padded frames at a cfr output (24fps?). When using the timecodes file this adapts the framerate of the listed frame ranges (correct if I am wrong please) to, in my case, either 23.976024(FILM) or 17.982018(?) with 29.970030(VIDEO) as the rest. This is what creates the variableness of vfr?

My question though is does the process actually remove or insert any frames? I was under the assumption that the process worked mainly by removing frames and using the timecode file to change the frame rate of the range of frames that were removed to make it run at full speed in each cycle. I've noticed that when I have always run Decomb512VFR that the output video was always shorter, and I noticed the parts that sped up from the resulting missing frames. In my source this results in a video with 33474 frames at 18:36.

But when I ran TIVTC's vfr, the resulting number of frames was shorter but also 31 frames larger than the cfr 23.976 counterpart using tdecimate(mode=1,hybrid=1). Resulting in a video with 33381 frames at 23:10 (equal time). So is the vfr output sped up to true FILM 24fps? Does the vfr routine differ from the two, and that accounts for the frame difference? The only thing I can think of, since the time/frame ratio is so drastically different (18:36 more frames to 23:10 less frames), is that Decomb512VFR outputs 29.976fps material and TIVTC outputs 24fps material in their vfr routines. Unfortunately, development has ceased and none of the documentation of Decomd512VFR addresses this for me.

One last thing, could it ever be possible to implement a similar thing in TIVTC where the filter in itself will run the first pass for metrics on the video and create the time codes file? Decomb512VFR is able to run the first pass somehow where ever the avisynth script has been called, so I know its possible, not that I am wining and complaining; but I'm not a programmer and this would just be a nicer feature. Thanks to MeGUI I had to dust off VirtualDub to get the first pass done with TIVTC. On a lighter note, TFM is the best around combined with eeid2 and tdeint, it tears through anime like swiss cheese.

As always, any help appreciated, please correct me if I'm wrong. Thanks.

[plugh]

An OVR file in conjunction with tdecimate mode 7 does not seem to work.

Example:
extract from ovr file
6793 -
6803 -
6808 -

debug output
TDecimate: inframe = 5563 useframe = 6799 chosen = 2
TDecimate: prev = 6798 curr1 = 6799 curr2 = 6800 next = 6800
TDecimate: 6796: 18.79 276299 (N) *
TDecimate: 6797: 0.13 1909 (D) *
TDecimate: 6798: 14.07 206950 (N) *
TDecimate: 6799: 3.37 49609 (S) *
TDecimate: 6800: 0.36 5306 (D)
TDecimate: 6801: 14.50 213178 (N)
TDecimate: 6802: 0.71 10417
TDecimate: ------------------------------------------
TDecimate: inframe = 5564 useframe = 6801 chosen = 1
TDecimate: prev = 6799 curr1 = 6800 curr2 = 6801 next = 6801
TDecimate: 6798: 14.07 206950 (N) *
TDecimate: 6799: 3.37 49609 (S) *
TDecimate: 6800: 0.36 5306 (D)
TDecimate: 6801: 14.50 213178 (N) *
TDecimate: 6802: 0.71 10417
TDecimate: 6803: 14.78 217278 (N)
TDecimate: 6804: 1254484575869219.00 18446744073709551615 (N)
TDecimate: ------------------------------------------
TDecimate: inframe = 5565 useframe = 6802 chosen = 2
TDecimate: prev = 6801 curr1 = 6801 curr2 = 6802 next = 6802
TDecimate: 6799: 3.37 49609 (S) *
TDecimate: 6800: 0.36 5306 (D)
TDecimate: 6801: 14.50 213178 (N) *
TDecimate: 6802: 0.71 10417 (D) *
TDecimate: 6803: 14.78 217278 (N)
TDecimate: 6804: 0.30 4364 (D)
TDecimate: 6805: 1254484575869219.00 18446744073709551615 (N)
TDecimate: ------------------------------------------
TDecimate: inframe = 5566 useframe = 6803 chosen = 1
TDecimate: prev = 6802 curr1 = 6802 curr2 = 6803 next = 6804
TDecimate: 6800: 0.36 5306 (D)
TDecimate: 6801: 14.50 213178 (N) *
TDecimate: 6802: 0.71 10417 (D) *
TDecimate: 6803: 14.78 217278 (N) *
TDecimate: 6804: 0.30 4364 (D)
TDecimate: 6805: 0.07 979 (D)
TDecimate: 6806: 13.25 194905 (N)

As you can see, it chose to use 6803 in spite of the ovr file.

[Terranigma]

As you can see, it chose to use 6803 in spite of the ovr file.
Maybe to retain synchronization?

[juhu]

Hi
I'm currently trying to remove dupes from a dvb-t capture (silent flick). A basic tdecimate(cycler=9,cycle=32) gets rid of the large majority of the dupes...but there are still some

So I played a bit with tdecimate mode2 and maxndl parameter.
As it seems there are never more than 3 real consecutive frames, I used maxndl=3, but to my surprise it doesn't seem to work : with debug (display) mode, I can clearly see almost each time a dupe is still there, it's in a SIX consecutive frames sequence (ie let's say displayed useframe= 1000 to 1005, dupe @ 1003.)
I shouldn't have 6 consecutive frame numbers with maxndl set to 3... What did I miss ?

[tritical]

@plugh
An OVR file in conjunction with tdecimate mode 7 does not seem to work.
Yeah, overrides in mode 7 of tdecimate are not currently possible. I will add that to the readme.

I noticed some puzzling results from a script that uses cframediff.

Dug into it, and eventually distilled it down to the following which illustrates the behaviour.

prevf is set true, you get correct output (ie 'true3').
If the frameevaluate is deleted, you get correct output.

a cframediff buglet?
avisynth weirdness??
stupidity on my part???
I can confirm this happens, but am not sure 100% sure why. The code in cframediff is correct in that it calculates the difference between frames n and n+1. What I think happens is that when cframediff requests frame n+1 it increments the "current_frame" value. As a test I changed cframediff to request frame n+2 instead of n+1 and I got true5. I'm not sure how to change cframediff so this works correctly, or if it is possible to.

When I look at the debug output of, for example, tdecimate(mode=0), part of the display are some number that I assume are 'normalized framediff metrics'. I also assume that the results from cframediff should be the same as those.

They are not. With identical args (blockx, blocky, chroma, nt) I get somewhat similar, but not identical results.
Can you post the script for this. I tried it here and get identical results.

[tritical]

@dude051

My question though is does the process actually remove or insert any frames? I was under the assumption that the process worked mainly by removing frames and using the timecode file to change the frame rate of the range of frames that were removed to make it run at full speed in each cycle. I've noticed that when I have always run Decomb512VFR that the output video was always shorter, and I noticed the parts that sped up from the resulting missing frames. In my source this results in a video with 33474 frames at 18:36. Yes the process only removes frames from film sections. Video sections are left undecimated. It never inserts extra frames. The timecode file is then used to set the appropriate display length for each frame.

But when I ran TIVTC's vfr, the resulting number of frames was shorter but also 31 frames larger than the cfr 23.976 counterpart using tdecimate(mode=1,hybrid=1). Resulting in a video with 33381 frames at 23:10 (equal time). So is the vfr output sped up to true FILM 24fps? Does the vfr routine differ from the two, and that accounts for the frame difference? The only thing I can think of, since the time/frame ratio is so drastically different (18:36 more frames to 23:10 less frames), is that Decomb512VFR outputs 29.976fps material and TIVTC outputs 24fps material in their vfr routines. Unfortunately, development has ceased and none of the documentation of Decomd512VFR addresses this for me.
tdecimate sets the framerate of mode 3/5 output such that the duration of the output is the same length as the input (some gui's with bitrate calculators need this). If you actually play it at that rate without using the timecode file it will be out of sync. I am pretty sure DecombVFR just sets the frame rate to 29.970 or leaves it the same as the input.

One last thing, could it ever be possible to implement a similar thing in TIVTC where the filter in itself will run the first pass for metrics on the video and create the time codes file? Decomb512VFR is able to run the first pass somehow where ever the avisynth script has been called, so I know its possible, not that I am wining and complaining; but I'm not a programmer and this would just be a nicer feature. Thanks to MeGUI I had to dust off VirtualDub to get the first pass done with TIVTC. It is possible, but I'm not going to go the trouble of adding in a gui window that pops up. So it would simply be an option to request every frame in the constructor.

[tritical]

[link removed], changes:
FrameDiff/CFrameDiff:
+ added display modes 3/4 to framediff
+ added ability to return block position to cframediff
- fixed bug in cframediff which caused mode=0 to return the highest value instead of
the lowest when norm was set to true

TFM:
+ added relative indexing from end frame number for trimIn files
+ added ability to set trimIn to a string containing a single frame range

TDecimate:
- fixed "number of frames after decimation doesn't match" error being incorrectly
thrown in mode 2

I plan to look into the reported problems and shortcomings of tdecimate's mode 7 and mode 2 when I have some more time.

[MOmonster]

:eek:
tritical you are great!

Thanks so much for adding these features so fast. I´ll try out as soon as possible.

:thanks: for all your expertise.

[zdark]

thanks for your job tritical!!!

[tritical]

[link removed], changes:
- fixed CFrameDiff returning invalid positions when rpos=true and there
was no motion
- added elim parameter to RequestLinear

Thanks to MOmonster for the CFrameDiff bug report and RequestLinear idea.

[kilik]

hello, i have a source with 5 frame interlaced and 1 frame progressive, is NTSC source, music dvd. I thing is called combed?

In dgindex i got NTSC Interlaced.

How to filter this source? only Tivtc or tivtc + tdeint?

Thank in advance.. bye :thanks::scared:

[manono]

Neither. You use some sort of an unblender most likely, RePAL or some such. It's hard to be positive without a sample, though.

[Vesi]

Need help for using TFM with tdeint.

I analyse my source with megui's analyse and show it film partially and suggest this.

DGDecode_mpeg2source("C:\...)
ColorMatrix(hints=true,interlaced=true)
tfm(order=1).tdecimate(hybrid=1)
crop( 2, 48, -2, -50).

Most of my source are 29 fbps and i want to bring them to 23.976 fbps.

as i am new to this filter, i want to know how to get good quality for my outputs.

[manono]

Indian DVD? Got a small sample you can upload somewhere so we can have a look? I'd trust the MeGUI analysis about as far as I can throw it.

[Vesi]

sample http://maxupload.com/7670C9B1

[manono]

Hi-

The next time you upload a sample, please try and have more movement. There's not much that can be learned from a sample where 2 people are sitting and talking. Anyway, you asked via PM for a full tutorial, something I have no intention of doing. By keeping it public, others can learn as well, and others who know more about the subject than I can also add to the discussion.

It's from an Indian DVD, as I suspected. However, this one is just hard telecined, and based solely on the sample, a simple:

TFM()
TDecimate()

will work fine. The reason I said I wouldn't trust MeGUI's analysis (or any automatic analysis), especially for NTSC Indian DVDs, was because over 90% of the classic film DVDs and probably over half of modern film DVDs are field blended, and no analysis is going to pick up on that. In such cases you use the RePAL filter, Restore24, MRestore, CDeblend, or some other unblender.

One way to determine what you have is to separate the fields (SeparateFields()), or put on a bobber (Bob()), and examine the fields carefully. In this case, after seeing that DGIndex confirmed it to be NTSC and Interlaced, and making the D2V with Honor Pulldown Flags, followed by bobbing it, I saw there were 2 fields the same followed by 3 fields the same, a sure indication of an already telecined source. For field blended sources, after separating the fields or bobbing the source, an examination of the fields will show a good percentage of them being blended, or ghosted, or having double images. In such cases an IVTC such as TIVTC won't work well, and I prefer to use rePAL. But sometimes they're even worse, being field blended 23.976->29.97fps, or even double blended, and those are especially difficult to work with.

To learn more I would suggest reading at the AviSynth site, reading the documentation included with TIVTC, reading the documentation included in the DGMPGDec package, and reading the DVD2AVI tutorial, outdated as it is:

http://www.doom9.org/ivtc-tut.htm

[Vesi]

Thanks alot manono for your replay.

I will do more reading on this subject as you suggest me. and will upload a sample with more details next time.

o.Since i am dealing with xvid and new to riping dvd to xvid, what is the good way or which way you recommend to analysis my source and chose the right deinterlacing type?

[manono]

In the case of your movie, the MeGUI analysis worked fine. There may be some real interlaced stuff elsewhere - studio logo, maybe - which might be why it called it hybrid, but mostly film.

The only good analysis is done with your eyes, in my opinion, again, especially true where Indian DVDs are concerned. And the knowledge of how to interpret what you see can only come with experience. Make a basic Avisynth script, separate the fields, and open it in VDub(Mod). Then scroll to a place with movement and start advancing a frame at a time. Since this is the TIVTC and TDeint thread, one or the other can handle most types of video, but not blended fields since I don't consider just deinterlacing a field blended video an option.

I have maybe 250 Indian DVDs, and more than 200 of them have blended fields. I prefer the classics, but the situation is only marginally better with DVDs of newer films. As long as you remember that the DVDs produced by the Indian DVD production companies are the worst in the world, that whatever can go wrong will go wrong, and that working with them requires a great deal of knowledge and experience, then you'll be better prepared for what you're going to encounter.

[Vesi]

Edit:
I made this simple script with out any deinterlacing line. And put the (SeparateFields()) line at the end of script and load it in VDM, now I see the image change to half size. Nothing else

DGDecode_mpeg2source("C:\Documents and Settings\vesi\Desktop\Test\Test.d2v",cpu=4)
crop( 2, 62, -2, -62)

Spline36Resize(640,272)
(SeparateFields())

I just download the DGMPGDec package and reading the documents on it, i am trying to understand what each things means, a bit hard for me since i have never done such thing before.I am some how lost in this, and trying to learn it
manono told me to do that, but it is hard for me to get him what he relay meant.

[laserfan]

how i separate the fields? should i remove the SeparateFields line when my script is ready for rip?You need to understand what manono has already told you: you use SeparateFields in Vdub in order to step-thru your video field-by-field to see what you have, and thus what you need to do.

As he has said already too, especially the DGMPGDec package contains excellent information/guides for examining video.

[tyee]

Can I use TDeint on HD-DVD to ivtc a movie?

I'm backing up to divx and H.264 but I get jerkyness every couple of seconds when I set my encoder to 23.976 output. AFAIK all HD-DVDs are flagged for 29.97fps but are 23.976fps on the disk.

Am I heading in the right direction to solve this?

[Chainmax]

While preparing a filterchain for ripping Robot Chicken based on advice MOmonster gave me a long time ago, I came across a section that has issues with modes 3 and 5 in TFM. Here's the current filterchain:

Mpeg2source("C:\Robot Chicken\Eps\RC_S1E1.d2v")

Interp = nnedi()
Deint = TDeint(type=1,cthresh=8,mi=66,edeint=Interp,emask=TMM())
TFM(mode=6,PP=7,slow=2,ubsco=false,cthresh=8,mi=66,clip2=deint)

Here's the problem (source) frame:
http://img165.imageshack.us/img165/8099/sourcesq9.png (http://imageshack.us)

Here's mode = 3:
http://img247.imageshack.us/img247/2712/mode3oo3.png (http://imageshack.us)

Here's mode = 5:
http://img292.imageshack.us/img292/6558/mode5eb0.png (http://imageshack.us)

All the other modes handle this frame perfectly. [link removed] a short clip with the frame in question.

[tedkunich]

@tritical

Is there any special trick/dlls needed for the output feature of TFM to actually generate an output file? Playing the script in Vdub, a file is created, but it is blank/empty. I am using TFM v1.0.3


AVISource("C:\test.avi")
AssumeBFF()
converttoYUY2(interlaced=false)
tfm(,output="C:\tfm.txt")

[LigH]

1) From my experience with clip comparing functions and plugins, which are able to output files, they held the output file exclusively open until you close the script or the application. So, for example: Loading a script in VirtualDub, and running it once, gives you an open file, which gets finalised only when you close the script or exit VirtualDub.

2) It may not harm, but the comma before "output" is strange. Possibly unnecessary.

[tritical]

TFM creates the file in the constructor, but it doesn't write anything to the output file until the destructor is run so either go to "file->close video file" after you've requested all the frames or just quit vdub and it will write everything to the file.

LigH beat me to it.

@Chainmax
I tried to download the clip, but the download link on the page you linked to wont work for me.

[tedkunich]

1) From my experience with clip comparing functions and plugins, which are able to output files, they held the output file exclusively open until you close the script or the application. So, for example: Loading a script in VirtualDub, and running it once, gives you an open file, which gets finalised only when you close the script or exit VirtualDub.

2) It may not harm, but the comma before "output" is strange. Possibly unnecessary.

Thanks, that was the problem.

[Chainmax]

...
@Chainmax
I tried to download the clip, but the download link on the page you linked to wont work for me.

It doesn't for me either. Please try this link (http://rapidshare.com/files/83086326/sample.demuxed.m2v.html) instead.

[tritical]

It's caused by the combination of tfm's internal motion masking (PP>4) and the external deinterlacing. If you set PP=4 instead of PP=7 (or use only nnedi as the clip2 input) the problem goes away. It's a good idea to disable tfm's internal motion masking, which is run post field matching, when using TDeint (or another motion-adaptive deinterlacer) for post-processing since TDeint's/TMM's motion masking is run on the original stream.

[Fizick]

Recently I saw some thread about comparing various deinterlacers (and its modes) by speed and quality. Some mode of TDeint was a winner (optimal). I want point some user to it, but I can not find it. Anybody can pont me to this thread? Or it was deleted?

[jeffy]

Recently I saw some thread about comparing various deinterlacers (and its modes) by speed and quality. Some mode of TDeint was a winner (optimal). I want point some user to it, but I can not find it. Anybody can pont me to this thread? Or it was deleted?
Some hints:
http://forum.doom9.org/showthread.php?p=1059016#post1059016

http://heptium.sh.cvut.cz/~integra/deint/
linked from:
http://forum.doom9.org/showthread.php?p=900011#post900011

[Chainmax]

It's caused by the combination of tfm's internal motion masking (PP>4) and the external deinterlacing. If you set PP=4 instead of PP=7 (or use only nnedi as the clip2 input) the problem goes away. It's a good idea to disable tfm's internal motion masking, which is run post field matching, when using TDeint (or another motion-adaptive deinterlacer) for post-processing since TDeint's/TMM's motion masking is run on the original stream.

Are you sure? Like I said in the post with the pictures, only modes 3 and 5 caused it, all other modes worked fine.

[tritical]

@Chainmax
Well, I was sure until I started to write an explanation of why it occurs, and it didn't actually make sense. Turns out there was a bug in modes 3/5 with micmatching > 0, which would cause tfm to sometimes return a u or b match when it thought it was returning a p, c, or n match. That error could only happen if all 5 matches were combed. I'll put up a fixed version later tonight or tommorrow.

@Fizick
I can't recall reading a thread comparing deinterlacers by both speed and quality.

EDIT:

Maybe it was this thread: http://forum.doom9.org/showthread.php?t=117025 ? It has some speed/filesize numbers plus some quality (ssim/psnr) numbers.

I did a comparison back in April 07 that I never finished, but I think that when taking both speed and quality (ssim/psnr wise) into account that it is difficult to beat:

yadifmod(mode=1,edeint=tdeint(mthreshL=0,mthreshC=0))

If only quality is considered than mcbob/mvbob are the clear winners (usually mcbob is best). If only speed is considered than the winners are leakkerneldeint/bob, yadif, or tomsmocomp (leaving out avisynth's internal bob() of course).

[Fizick]

Thanks for responds, but it was some other thread...
I found it ! :)

http://forum.doom9.org/showthread.php?t=133078

[tritical]

TIVTC v1.0.5 (http://bengal.missouri.edu/~kes25c/TIVTCv105.zip), changes:
- Fixed a bug that would cause tfm to return a 'u' or 'b' match when it thought it was
returning a 'p', 'c', or 'n' match. This problem only occured if mode was set to
3 or 5, micmatching was enabled (not set to 0), and all five matches were combed.

[Mottodj]

Hi everybody! I have question about results field maching by TFM. I have DVD with NTSC(film but without pulldown flags) anime and i pass it through TFM using avisynth. Field maching is nice but there is artifact in picture that was already mached(on fade to dark and in other way). Lines on picture have different brightness and it changing each other(ex. http://mini.100-games.ru/4super/artifact.png ). Have you ever seen that thing, what to do with this? Passing through tomsmocomp don't produce such artifact but tomsmocomp blur picture and it isn't solution...

[Dreassica]

That's not missed interlace, but common in fades. I usually just applyrange(x,x,"blur",0,0.7) it, works perfectly.

[Leak]

That's not missed interlace, but common in fades.
To be totally precise, that's a fade that has been applied AFTER the telecine was applied, so you have a 30 frames (or maybe even 60 fields) per second fade over pulled down 24 FPS material, so you can have either of them cleaned up, but not both - but then again, in a fade it's usually not noticeable UNLESS of course you take a screenshot... :)

A similar thing would be 30p/60i end credits (for extra smoothness that's easily produced) scrolling over 24p animation, which also happens often.

[foxyshadis]

Use Vinverse for mild residual combing, not a destructive blur. Works wonders with some post-telecine editing, unless it's a very fast fade.

[NorthPole]

I've been using the following sort-of generic catch-all script for re-encoding ATSC digital broadcast caps to 23.976. I had a couple of questions..

Do I need to determine the order or does DGDecode_mpeg2source set the order correctly?

When converting from 59.94 fps progressive to 23.976 fps, sometimes I get some jerkiness in using the tdecimate(cycler=3, cycle=5). Is there anything I could do to tweak this?

I'm kind of lazy about counting frames in VirtualDub....Sorry

I have used the tdecimate(hybrid=1) setting on 29.97 fps to 23.976 fps and seem to get good results but I don't think I can use that for 59.94 fps conversions.

Any input would be appreciated.

PS.. Thanks tritical for you work!


DGDecode_mpeg2source("__vid__",info=3)
LoadPlugin("C:\Program Files\Media\AviSynth\plugins\tivtc.dll")
LoadPlugin("C:\Program Files\Media\AviSynth\plugins\eedi2.dll")
LoadPlugin("C:\Program Files\Media\AviSynth\plugins\tdeint.dll")
(last.height == 480)? SD(last): HD(last)
last.letterbox(4,2,2,2) # to mask screen edge noise

Function SD(video) {
ord = video.getparity() ? 1 : 0
ord == 1 ? video.AssumeTFF() : video.AssumeBFF()
edeintted = video.SeparateFields().SelectEven().EEDI2(field=ord)
tdeintted = video.TDeint(edeint=edeintted,order=ord,hints=false)
video.tfm(order=ord,clip2=tdeintted).tdecimate(hybrid=1)
#last.Lanczos4Resize(720,480,0,0,704,480)
}

Function HD(video) {
(video.height == 720)?
\ video.tfm().tdecimate(cycler=3,cycle=5):
\ video.tfm().tdecimate(hybrid=1)
}

[foxyshadis]

Do I need to determine the order or does DGDecode_mpeg2source set the order correctly?
mpeg2source will do that.

When converting from 59.94 fps progressive to 23.976 fps, sometimes I get some jerkiness in using the tdecimate(cycler=3, cycle=5). Is there anything I could do to tweak this?
Well, not everything is telecined. You're sure these are, rather than true 30 or even 60 fps segments? You can try tweaking chroma, nt, and dupthresh parameters to see if they help your stuff.

I have used the tdecimate(hybrid=1) setting on 29.97 fps to 23.976 fps and seem to get good results but I don't think I can use that for 59.94 fps conversions.
You'll get better results out of mvflowfps if you must force an fps change.

[NorthPole]

@foxyshadis
Thank you for the input.

mpeg2source will do that.

Good, I'll take it out.


Well, not everything is telecined. You're sure these are, rather than true 30 or even 60 fps segments? You can try tweaking chroma, nt, and dupthresh parameters to see if they help your stuff.

I believe most, if not all, is just 24p slowed down and telecined... Mostly movies and sitcoms.

If I understand the hybrid mode, it does normal 1 out of 5 decimation for 30 fps video segments provided it has a match. If it doesn't then it does blended decimation to get to 24 fps. I was kind of looking for something like that for 59.94 fps or 60 fps.

like say this...
tfm().tdecimate(cycle=2).tdecimate(hybrid=1)

I will try the dupThresh, chroma and nt options but then I think I have to use mode 1. So perhaps say something like..

tfm().tdecimate(mode=1, dupThresh=0.9, chroma=true, nt=1, cycler=3,cycle=5)
or
tfm().tdecimate(mode=1, dupThresh=1.2, chroma=false, nt=1, cycler=3,cycle=5)


You'll get better results out of mvflowfps if you must force an fps change.

I'm not sure what this is?

[NorthPole]

I did a couple of test encodes on a 1280 x 720p ATSC cap with:

tdecimate(cycler=3,cycle=5)
and
tdecimate(cycle=2).tdecimate(hybrid=1)

I wasn't thinking earlier but I wouldn't need the field matcher, tfm, since the source is all progressive.

the frames pattern go as follows: 1 0 0 1 0 1 0 0 1 0 1 0 0 1 0 etc.

1 = new frame, 0 = duplicate frame

I get the correct resulting frame selections with both methods and the encoding time is very similar. I believe that tdecimate(cycle=2).tdecimate(hybrid=1) might be a better way to go because it could correctly handle true 24 fps and 30 fps sources. The 24 fps source would be returned to the original frame rate and the 30 fps source would be blend decimated to 24 fps and hopefully avoid any jerkiness.

I tried the dupThresh, chroma and nt options with no noticably difference however, unfortunately, I deleted the original 720p video clip that was giving me jerkiness. I will have more in the next week or two and will report whether those tweaks helped.

[rebkell]

ABC is notorious for dropping frames, one of their favorite sequences is:
3:2
3:2
3:2
3:2
3:2
3:2
3:2
3:2
3:1
3:2
3:2
3:2
3:2
3:2
3:2
3:2
2:2

and then they repeat, it will cause the jerkiness when trying to get to the 23.9675 FPS. It works out to actually be 24.553735 FPS. I've seen it on several shows, LOST, Men In Trees, and some others, occasionally they will add an extra frame.

[Adub]

Ok, so how do you deal with it?

[rebkell]

Ok, so how do you deal with it?

Check out this thread, I'm not sure why it's in the DVD2AVI / DGIndex section, but it is.

http://forum.doom9.org/showthread.php?t=122472

I use FDecimate, I use megui and I actually built a profile to step through the frames, normally you can determine the cadence, basically you use the metrics=true to try to determine the best threshold value to use and of course after stepping through you can determine the FPS. The cadence I posted is the most frequent one I've run into, it's not perfect, but if you can figure out a good threshold value it works pretty well. All the above will make more sense when you read the thread I linked to above. I've also run into 10 consecutive 3:2 sequences and 1 2:2, that repeats every 11 cadences(I guess that's the correct word)

I've seen some cadences on the Fox Network that I never could determine any pattern for.

[Adub]

Thank you for that. I have to learn how to count fields. Anyone have a "guide" for doing it?

[NorthPole]

@rebkill,

Thanks for your input... I haven't counted as many frames as you have (not fun). I will check out the thread you suggested.

@merlin777,

I just count frames in virtualdub as I go through the video manually. I don't know of any specific method or program that does it automatically.

[Adub]

Wait, I thought they were counting fields, not frames. Am I mistaken?

[rebkell]

Wait, I thought they were counting fields, not frames. Am I mistaken?

Frames, you probably got confused by the conversation of our Texas friends that have the ABC 720p that is broadcast in 1080i, it's a fairly unique situation. If your ABC and/or Fox broadcasts in 720p, then we're only dealing with frames.

[Adub]

Okay,
Originally Posted by scharfis_brain
Actually I did excatly this: wirting down the pattern of the first frames until a pattern was visible. I counted 160 fields. Then I estimated the other pattern breaks with the simple math (every 67 fields a break) to be sure that the pattern stays constant.
That took about 5 minutes concentrated work. Not more.

I am confused by this however.

[laserfan]

Here is my last "Inspect for Motion.avs"

MPEG2Source("Cane - Brotherhood.d2v")
assumetff()
separatefields()

Then I open this w/VDub and count. So it's fields that are counted when it's a 1080i program.

[Adub]

Okay, thanks for that clarification. Now when you are counting fields, what are you looking for?

[laserfan]

Okay, thanks for that clarification. Now when you are counting fields, what are you looking for?Looking for the cadence. If you find 3 alike then 2 alike then 3 again then 2 again and it never changes, you're golden. You can do a simple

Telecide(guide=1).Decimate()

and proceed to re-encode from there. If you have an odd cadence then it gets an order-of-magnitude more complicated! ;)

If you're new to this stuff, my advice is to find neuron2's website (Donald Graft) and look at the Readmes & tutorials he's created for the above, and his tools--I'd start with Decomb and study the DecombTutorial.html therein.

[NorthPole]

Don't know if this is correct, but I find it easier to look at frames. For 480i and 1080i, I look for a repeating pattern of 3 progressive frames followed by 2 interlaced frames. If the pattern stays consistant for say 50 frames, I just go with a tfm().decimate(). If the pattern is not consistant, then I try tfm().decimate(hybrid=1). 720p would be just frames.

[Adub]

See now some of the reasons I get confused is that the 3:2 pattern applies to both fields and frames.

Ex. 3 progressive frames, 2 interlaced.
AND
Ex. 3 fields that "current", 2 fields that are "next".

I just need to tie that together in my head.

[foxyshadis]

Don't know if this is correct, but I find it easier to look at frames. For 480i and 1080i, I look for a repeating pattern of 3 progressive frames followed by 2 interlaced frames. If the pattern stays consistant for say 50 frames, I just go with a tfm().decimate(). If the pattern is not consistant, then I try tfm().decimate(hybrid=1). 720p would be just frames.

You can pull your hair out trying to figure out field-shifted video by doing that, though, or just make the wrong choice. All interlaced video is naturally fields, so you might as well start at that level unless the pattern is instantly obvious.

[NorthPole]

All interlaced video is naturally fields, so you might as well start at that level unless the pattern is instantly obvious.

OK, I will take your advise and go the fields route.

Had the best luck to date at getting smooth playback @ 23.976 fps on 1280 x 720p ABC ATSC OTA cap of movie meet the fockers with:

tdecimate(mode=7,nt=1,dupthresh=1,vidthresh=3.5)

pattern was 3:2:3:2:3:2:3:2:3:1:1:2 got lost (gave up) at that point...Not good at that.

[NorthPole]

Spoke to soon... Still not happy with the results.

Tried a script by plugh from this thread to get and idea of where the dupthresh might be.

http://forum.doom9.org/showthread.php?t=122472&page=2

Going to try mode 7 again with different dupthresh and vidthresh values. Had one question....

Assuming 720p 59.94 fps material as source.

Is this correct in stating that the frames with a difference value that falls between the dupthresh and the vidthresh are the frames that are subject to blending during decimation?

Sorry, I just realized that this is in the development forum. Should have posted in the usage forum.

Thanks.

[thetoof]

Dunno if it's the right place to post this, but I've been told TIVTC can be helpful for interlaced footage encoded as progressive. However, everything I've tried can't produce any good results.

Source = NTSC DVD (anime)
DGIndex tells me: 99.98% FILM, Progressive, frame-based

Honor pulldown flag: no pattern at all, about 50% of the frames have interlacing artefacts

Here's the first chapter ripped with DGIndex (decrypted vob)

http://www.megaupload.com/fr/?d=TVCZAWUB

For those who'd like to help, but not download a 156 MB file, here's a m2v generated using demux in DGIndex with "honor pulldown flags"
http://www.megaupload.com/fr/?d=87FM0ENK

[Chainmax]

I have a trouble sample as well. It's from the X-Men 1992 series' intro. For some reason, on certain spots peppered throughout it, a pretty messed-up frame is chosen instead of a considerably cleaner version of it. The IVTCing portion of the script is this one:

AssumeTFF()
Interp = nnedi(field=1)
Deinted=TDeint(order=1,field=1,type=1,edeint=Interp,emask=TMM(order=1,field=1))
TFM(d2v="X:\wherever\Sanctuary1.d2v",mode=6,order=1,PP=7,slow=2,mChroma=false,Clip2=Deinted)
TDecimate(mode=1)

You can download a sample with the intro here:
http://rapidshare.com/files/92404386/Intro.demuxed.m2v.html

[tritical]

@thetoof
The main problem with your clip is that it has blended fields. Worse than that they seem to be causing jerkiness, as though the field order is incorrect, but it isn't. Maybe someone else can recommended something better, but

mpeg2source()
tdeint(mode=2)
tdecimate()

seems to give ok results.

@Chainmax
I downloaded your clip, and ran it through dgindex to generate a d2v file. However, when I try to load that d2v file with mpeg2source() in virtualdub it just hangs. I'm using dgindex/dgecode v1.5 RC2. The downloaded file has 59768148 bytes and the crc is 9F0C270B, is that correct?

[Chainmax]

Unfortunately, I deleted the file as soon as it was uploaded. I'll make another cut with the intro and make sure Mpeg2Source is able to pick it up before uploading it. I apologize for the inconvenience.


[edit]I get the same eror when trying to open a d2v generated (from a freshly cut intro) with DGIndex 1rc2 on VDubMod 1.5.4.1 via MPEG2Source. The m2v opens fine by itself in VDubMod though. I'll post this in the DGIndex thread, maybe I hit a bug.

[thetoof]

Not perfect, but those settings give better results than everything I'd tried. Thanks!

'course, if someone finds a way to completely remove the temporal blending (not sure that this is how it's called... I'm talking about the frames that are present as "almost transparant" in previous or next ones), it'd be even better... but heh, I guess there are some things that just can't be fixed.

Anyways, thanks again!

[plugh]

Usage Q:
Can I use an override file to specify 'keep' frames? The doc only refers to using '+' "as place holders" in a "decimation pattern", not as an override directive unto itself.

Doc Q:
The write up on 'maxndl' is confusing and seems to have errors wrt the examples.
For example, say we have the following pattern in a video:

5 5 7 7 2

where the numbers indicate how many frames there are between duplicates. In this case we want to remove 5 in every 26 frames.
If the numbers indicate the number of frames between duplicates then the total number of frames in the sequence is 31, not 26. This affects the rest of the discussion of the example.
Also note this sentence:This decimation ratio (5/26) would indicate that there is one duplicate in every 5.2 frames (5/26 = 1/5.2).
The numerator is your 'number of dups to be removed'
The denominator is -- well it's not clear what it is (see above)
Let's look at the next example:
Another example would be decimating a 59.94fps video to 23.976. This ratio would indicate one duplicate in every 1.667 (5/3) frames.
Hmmm. In this case
the numerator is the total number of frames in a cycle
the denominator is the number of dups to be removed

But in both examples the ratio is stated to represent the same quantity - "one duplicate in every N frames"

I am very :confused:

[rebkell]

I posted this in another thread, but this probably a better place to discuss it:

Ran into another weird cadence on Jericho the other night:

21112111211121112111211121112111211111
21112111211121112111211121112111211111

Looks like it's keeping 38(dropping 9) out of every 47 frames, which would work out to 24.231 FPS. I only examined one section of it, but that section seemed to be consistent .... I wanted to cry, CBS has been so great, I think it's the first strange pattern I've ever run into on CBS broadcasts.

would the maxndl=5 or would it = 6 in the above cadence.

The 2 in the above pattern equals two frames that are the same. At the end of the first row I have 5 nonduplicates in a row and then the 6th frame on the next row would also be different and the seventh would be a duplicate of the 6th, so I would assume that maxndl would = 6?

If the pattern stays consistent I assume I would use:

tdecimate(mode=2, rate=24.231, maxndl=6)

[K0zi]

I have a Chinese movie released on NTSC DVD.
I tried both tfm().tdecimate and tfm().tdecimate(cycle=6), but the results didn't seem to be what I was looking for.

Sample: http://rapidshare.com/files/89421694/sample.rar.html

[rebkell]

I have a Chinese movie released on NTSC DVD.
I tried both tfm().tdecimate and tfm().tdecimate(cycle=6), but the results didn't seem to be what I was looking for.

Sample: http://rapidshare.com/files/89421694/sample.rar.html

tfm(order=1).tdecimate(hybrid=1,cycle=6) seemed to work well, the hybrid was only applicable to one section of 6 frames, the rest were definitely one duplicate in 6. It appears it was 25.0(24.975) FPS.

[K0zi]

What if I wanna get some "proper" fps instead? I mean, the source video was apparently PAL, as it's also a Chinese standard these days. Is it possible to get the original framerarte back without any seriuos issues (like audio track sync for instance)?

[rebkell]

What if I wanna get some "proper" fps instead? I mean, the source video was apparently PAL, as it's also a Chinese standard these days. Is it possible to get the original framerarte back without any seriuos issues (like audio track sync for instance)?

I don't really know, you might try tfm().TDecimate(rate=25.0)

[tritical]

@plugh
Yep, the maxndl explanation explanation is not very clear. In the first example with (5 5 7 7 2) the numbers aren't the number of frames inbetween each duplicate frame, but the number of frames from the current duplicate to the next duplicate. So the sequence is really:

dnnnndnnnndnnnnnndnnnnnndn

Here, you would want to remove 5 in every 26 or 1 in every 5.2.

In the 59.94 to 23.976 case there are more duplicates in the stream then there are new frames, which means you want to remove more than 1 out of every 2 frames and you can't state that in 1/N terms using only whole numbers. That's why I used 5/3. However, 1/0.6 specifies the desired value just as well.

What the fraction actually means is that in every 5 frames you want to remove 3 frames (or in every 1 frame you want to remove 0.6 frames). The top number in the fraction is just some arbitrary cycle length, and the number in the bottom is the number of frames you want to remove within that cycle length. The actual values you use don't matter as long as the decimal resulting from the fraction is the same.

On the keep frame overrides, I can't remember for sure. I want to say that it isn't possible.

@rebkell
On the maxndl value for mode 2 I think you will just need to experiment and see what works best. In all honesty, I've never really spent a lot of time working on mode 2 to get it to a solid/stable state, so sometimes it produces good results and sometimes it just sucks horribly.

[rebkell]

I have a question about TDecimate and using the ovr (Override file)

I tried using it and it wasn't doing anything, when I looked at it. Just want to make sure I'm using it correctly.

I'm hopelessly looking for ways to cut out those weird ABC-HD captures and their strange cadences. Anyway.

If I include a line in my avs script like:
TDecimate(ovr="C:\test.ovr", display=true)

and have a file named test.ovr in the C:\ directory
with one line like so:

0,130 +-------+

this should from my understanding keep 1 frame starting at frame 0 and then drop the next 7 and keep the next one and then repeat this through the first 130 frames, then it would revert back to dropping one frame in every five as the normal TDecimate without any parameters does. When I step through an avs script using display, it doesn't appear to be using the .ovr file at all, it just seems to be using the normal 1 drop in every 5 frames.

Am I doing something wrong? the above test.ovr file is just for a test, but it should definitely be easy to tell if it's working.

[Zarxrax]

I'm having some difficulty understanding the d2v fixing.

I indexed a d2v file with dgindex, and dgindex said it had illegal field order transitions, so I let dgindex fix it. Then when I was trying to use TFM on the file, it was just utterly failing everywhere, it wasn't matching hardly any of the combed fields.

I then tried the original "broken" d2v file, and everything worked great!

Ok, so then I thought maybe I'll let tfm try fixing the file. So I use the d2v parameter to let tfm fix the file, and then I tried the one that it output. Once again, the fields aren't being matched.

Now, I can use the "bad" d2v and its giving me awesome results, but it's just really bugging me that I don't understand what's going on here. I'm being told that my original d2v has some bad stuff going on in it, so I should use a fixed one... but the fixed ones are the ones that end up not working right!

[NorthPole]

I'm having some difficulty understanding the d2v fixing

What is the type of your source material. If it is 720p, then the field fix would not apply. If it is 480i or 1080i then you have to make sure you do not have frame repeats. (not sure if you have to preview the whole clip to get that info).

Another option is to use the "bad" file and use tfm(mode=4) which I believe can handle illegal field order because of the 3 way matching.

[Zarxrax]

It's just a regular dvd of old telecined anime material.

[reepa]

I'm not sure if this is a bug or if I'm misinterpreting the documentation, but TDeint averages pixels that are woven even when mtnmode is set to 2 ("no averages, replace with most similar field"). I'm using TDeint(mode=1, mtnmode=2, map=1) and nearly all woven pixels are either 153 ("use average of curr/next") or 204 ("use average of curr/prev"). Mtnmode=3 doesn't have this problem (i.e. all woven pixels are either 51 or 102).

[plugh]

I'm trying to process a TV cap, and tfm + tdecimate(hybrid=1) are doing ok with it EXCEPT in areas where an insert fades in/out or slides on/off the bottom of the screen.

If I do something like this
mpeg2source(d2v=)
cropbottoim(80)
tfm(d2v=)
tdecimate(hybrid=1)
everything works correctly (except I've lost the bottom of the picture). Smooth motion, no unnecessary blends, nice clean field match and decimation...

Great, I sayz, I'll use an exclusion band...
oops - tfm has y0/y1 args, but tdecimate doesn't! grrr...

REQUEST: exclusion band input for tdecimate

OK, so do it the 'hard way'
mpeg2source(d2v=)
tfm(d2v=,y0=400,y1=480)
orig=last
cropbottom(80)
mergehints(hintclip=orig)
tdecimate(hybrid=1,clip2=orig)

Only problem is, it don't work. I'm back to frame blends again. As far as I can figure, the above two scripts should be yielding the same ivtc decisions, but they aren't.

What am I missing / doing wrong?

Thanks!

PS - are Y0/Y1 zero-based or one-based? Do they count from top of frame?

[thetoof]

Sorry for the noobish questions, but I don't have any hybrid source atm, so I can't run any tests to be sure that my scripts are done the right way.

I found some info floating around about creating a VFR clip for hybrid sources, but I just want to be sure... Leaving post-processing aside, what would be the settings to use for 1st and 2nd pass of TIVTC?

Also, what is the format of a timecode file? I'd like to create one manually to select which part of the movie will run at which framerate.

[Adub]

It's a little dated, but check this out:
http://avisynth.org/mediawiki/VFR#encoding_to_vfr_.28mkv.29

[thetoof]

Yeah, I read it... but I thought I had done something wrong, because nothing was being written in the "tfm.txt" and "stats.txt".

I've tried running a preview, an analysis pass and a lossless pass in vdub without getting anything... any ideas why it doesn't work?

And, do you know the format of the timecode?

Thx

[Adub]

You did let the passes finish fully, correct? Because no information is written until the script is done. I was doing the same thing and I realized that I had to wait until the end before the info was written.

as for the timecodes, I would have to look at them to tell you. Or you could do it yourself: Check which of these the file looks like:
http://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html#EXTERNAL%20TIMECODE%20FILES

[thetoof]

That's exactly what I needed, thanks! Everything's working perfectly fine now.

[Adub]

Glad to help.

[shadell]

I have tried using TDeint with the blend interpolation option as a replacement for fielddeinterlace(), which I find produces pleasing results in some situations, despite the ghosting. The call was TDeint(type=4). This produced a lot of jagged artefacts, such as the following:

http://i297.photobucket.com/albums/mm223/shadelll/artifact013crop.png

When I look at the map=2 output, I see that the effected areas have (mostly) not been marked as 'interpolate pixel'. When I look at the interlaced input, the area is combed.

Reducing mthreshL and mthreshC down to 1 reduces the number of artefacts, but there are still quite a few.

Since there is a blend interpolation option, I thought it should work OK. Are there other options I need to use to get it to work without producing these artefacts?

[Nikos]

a. The denoise option of TDecimate is for internal metric calculations only or not?

denoise -

Sets whether or not to denoise frames prior to doing difference metric calculations.
This can greatly improve metrics for noisy sources (i.e. stabilize duplicate frame
metrics and make it easier to tell the difference between duplicates and non-duplicates).
It also works very well for sources with lots of dot-crawl because the denoising
effectively cancels all dot-crawl, whereas the dot-crawl would usually inflate difference
values of duplicates. Using denoising will slow things down somewhat, but it is MMX
optimized and pretty fast.

true - use denoising
false - don't


b. May i crop before IVTC with tfm and TDecimate?

mpeg2source("Black&White-source.d2v")
Greyscale()
crop(32,16,-32,-24)
tfm(d2v="source.d2v", mode=1, mChroma=false)
Tdecimate(chroma=false)


c. The greyscale place is right?
The source is Black&White telecine 29.97 fps film.

[Adub]

Since your clip is already black and white, why add the greyscale parameter? Greyscale() is usually used when converting color clips to greyscale.

Also, I believe that it is never good to crop before an ivtc, or whenever dealing with interlaced material. I thought I heard there was a way some time ago, but I have always found it safer to crop afterward.

[Nikos]

Thanks Merlin7777 for the answer.
Althought the clip is black and white, sometimes there is noise in the uv planes.
For interlaced or telecined material i know that it's safe to resize horizontal before IVTC but i am interesting if it's safe to crop before IVTC or deinterlace.

[Adub]

Althought the clip is black and white, sometimes there is noise in the uv planes.

Ah, I understand then.

And like I said before, cropping before IVTC or Deinterlacing is never a safe thing to do. I have never gotten it to work correctly at least.

[Boulder]

I'd expect that cropping horizontally is OK, and cropping vertically mod-4 also to be OK.

[Mystery Keeper]

I'd expect that cropping horizontally is OK, and cropping vertically mod-4 also to be OK.

Only cropping mod16 is safe. One should crop mod16, do what needs to be done and then crop the rest.

[Didée]

Only cropping mod16 is safe. One should crop mod16, do what needs to be done and then crop the rest.
Ah, a statement. Please explain why it should be so.

(I don't see why mod16 should be required for purposes of IVTC or deinterlacing in general. mod4 vertically is a technical requirement for YV12 sources, since that's the "luma distance" between same-parity chroma samples. That's all.)

[Mystery Keeper]

Ah, a statement. Please explain why it should be so.

(I don't see why mod16 should be required for purposes of IVTC or deinterlacing in general. mod4 vertically is a technical requirement for YV12 sources, since that's the "luma distance" between same-parity chroma samples. That's all.)

I tried cropping before deinterlacing with TempGaussMC and it gave me ghosting ^_^'

[Didée]

During testing TGMC, I'm often cropping wildly around on the input, only caring for mod-4, not -16, and never got any unusual ghosting. Feel free to post script + material that makes your observation reproducable.

Again, mod4 is the only technical requirement in this case.

[Adub]

Ah, so that was the trick to it. it's okay to crop horizontally. but when cropping vertically, always respect mod 4.

[thetoof]

Hi,
I've got something fairly complicated to do via vfr decimation, so I've got a help request if it's doable and a feature request if it isn't :p

Let's say I have these framerates :
a=23.976
b=25
c=29.97
d=59.94
They were then spliced together with
a.changefps(d)+b.changefps(d)+c.changefps(d)+d
So, I have a CFR clip @ 59.94fps with tons of duplicates (that I want to remove via vfr decimation). How can I do that???

Also, there are many duplicates in the 23.976 clip and it could be decimated to very low framerates for a couple of frames (even below 8 fps) So, I was wondering if it would be possible to have a vfr decimation that adapts to the amount of motion and uses framerates other than the standards ones to display the same motion to encode a lot less frames with no visual difference (in anime you can even have the same frame for more than 1 second after all the noise has been removed).

Here is a test clip I have created for the occasion that replicates the situation I just described: http://rapidshare.com/files/141496007/VFR_testclip.7z.html

[sander815]

how do i call the multithreaded version of this plugin? EEDI2_imp.dll

[foxyshadis]

You replace EEDI2.dll with that one. That's it.

[aand]

Question about TDecimate:
I have a series of black frames; instead of cutting some of them, it removes all of them and leaves more 'visible' dups behind.
Is there a way of setting <max nr of dups in a row> ? If not, I'm making an official feature request :D.
Thanks.

[r00t61]

Is there any way I can set the location of the debug output information provided by TFM/TDecimate (i.e, when display=true)? Currently, the output is always fixed to the upper left-hand corner, so if I have display=true on for both TFM/TDecimate, the debug output from TDecimate overlays on top, obscuring the debug output for TFM.

I'd like to be able to see both sets of debug data at the same time.

[thetoof]

I didn't check the doc, but a quick and dirty way of doing it is by using stackvertical/horizontal to see the same frame with different info, or by cropping the part of the frame where the info is to overlay it on another corner.
source=your source
tfminfo=source.tfm(settings).crop(select the part of the frame containing the info here)
tdecimateinfo=source.tdecimate(settings)
overlay(tdecimateinfo,tfminfo,coordinates where you want to overlay)
#uncomment the next line to tweak the crop settings
#return tfminfo

[r00t61]

Well, thetoof, your suggestion didn't quite work out the way I wanted to, but I'll keep experimenting. Thanks.

In the meantime, is there any to alter TDecimate's behavior for vfr output (i.e., when mode=5), for handling 30p video sections? In particular, I would like to make TDecimate perform a bob (59.94 fps) instead of a blend decimation for those sections designated as video, while still IVTCing/decimating the film portions to 23.976 fps.

[thetoof]

You may be interested in AnimeIVTC, as it bobs interlaced sections & IVTC telecined ones to create a VFR clip. See mode=3, 4 or 6 with omode=2.

[canuckerfan]

i have a clip here which i'm trying to deinterlace correctly... it's hard telecined without any blends so an ivtc should work. however, I'm having a hard time determining the field order. I've tried the separatefields().assumetff test but both tff and bff show no backward motion. weird. so far I've thrown 2 scripts at it:

interp = nnedi(field=1)
deint = tdeint(mode=0,order=1,field=1,edeint=interp,slow=2,emask=TMM(mode=0,order=1,field=1))
tfm(mode=3,order=1,clip2=deint,slow=2)
tdecimate()

interp = nnedi(field=1)
deint = yadifmod(edeint=interp)
tfm(mode=3,order=1,clip2=deint,slow=2)
tdecimate()

both produce pretty much the same result to my eyes and the ivtc looks pretty decent. but I'm wondering if this is right with the field order discrepancy. here's the clip(23 mb): http://www.sendspace.com/file/upa8m7

[AVIL]

IMHO your clip is progressive. No temporal difference between fields. Only spatial one. There is some residual combing that you can fight with plugin vinverse. Internal function Info() says that the clip is bff .In fact, when assumed as bff, after separatefields() the first field has the top line. But deinterlace this progressive clip is unnecesary (again IMHO). In scenes with motion there is blending. Try to deblend first.

[canuckerfan]

IMHO your clip is progressive. No temporal difference between fields. Only spatial one. There is some residual combing that you can fight with plugin vinverse. Internal function Info() says that the clip is bff .In fact, when assumed as bff, after separatefields() the first field has the top line. But deinterlace this progressive clip is unnecesary (again IMHO). In scenes with motion there is blending. Try to deblend first.
thanks for your input. why i seem to think it's partially interlaced is because of the horizontal lines evident throughout the clip. classic sign of interlacing. in fact, it seems to follow a 3p/2i pattern. as for blending... i separated the fields and didn't see any myself. during high motion there is plenty of blurriness, however.

[manono]

AssumeTFF().SeparateFields()#plays smoothly
AssumeBFF().SeparateFields()#plays jerky

The sample is TFF and any simple IVTC seems to work. It's not field-blended, although there does seem to be some sort of motion blur or similar going on.

[canuckerfan]

AssumeTFF().SeparateFields()#plays smoothly
AssumeBFF().SeparateFields()#plays jerky

The sample is TFF and any simple IVTC seems to work. It's not field-blended, although there does seem to be some sort of motion blur or similar going on.
yea, I don't know what the hell that stuff is. thanks. your input has confirmed my thoughts.

[AVIL]

I have repeated my test but using dgindex/dgdecode. Formerly I've used directshowsource (an enourmous mistake). My new results agree totally with manono's ones.

[@Tron@]

Interested in a plan to update the gorgeous plug -TFM!? I take them a long time and would like to know whether in the near future, its updating and some "delicious"))) because there is no limit to perfection, and should always be striving towards the ideal ))

And what with TDecimate and MergeHints???

[ikarad]

TIVTC is multithread or not?

[tritical]

@ikarad
No, tivtc isn't multithreaded.

@@Tron@
Atm, I don't have any plans to update tivtc further. My original motivation for coding/working on tivtc was the fact that I actually used it regularly, but as I haven't encoded anything in 2-3 years that motivation is pretty much gone. Now I tend to stick to filters that interest me academically.

[rebkell]

Is there anyway to get tdecimate to ignore the bottom part of the clip, when making decisions on which frames to drop, all the animated logos, and popups on all the network shows that I capture really plays havoc with trying to get the right frames dropped.

[tritical]

I think 'clip2' parameter of tdecimate can accomplish what you want. Something like:

saved = last
crop()
tdecimate(clip2=saved)

It will make decisions based on the cropped clip, but return frames from clip2. If you only crop the bottom of frames it shouldn't effect hint passing.

[rebkell]

I think 'clip2' parameter of tdecimate can accomplish what you want. Something like:

saved = last
crop()
tdecimate(clip2=saved)

It will make decisions based on the cropped clip, but return frames from clip2. If you only crop the bottom of frames it shouldn't effect hint passing.

Thanks, I'll give it a try.

[rebkell]

I think 'clip2' parameter of tdecimate can accomplish what you want. Something like:

saved = last
crop()
tdecimate(clip2=saved)

It will make decisions based on the cropped clip, but return frames from clip2. If you only crop the bottom of frames it shouldn't effect hint passing.

That seems to have worked perfectly. Thanks again.

[ikarad]

@ikarad
No, tivtc isn't multithreaded.

@@Tron@
Atm, I don't have any plans to update tivtc further. My original motivation for coding/working on tivtc was the fact that I actually used it regularly, but as I haven't encoded anything in 2-3 years that motivation is pretty much gone. Now I tend to stick to filters that interest me academically.

thanks for your answer.
Is it possible to hope a multithreading version of TIVTC or is it an utopia?

[tritical]

Is it possible to hope a multithreading version of TIVTC or is it an utopia?
It is very unlikely. With the way the code is, it would be a major task to multithread tivtc interally.

[ikarad]

I have a problem with TIVTC and ffdshow.
If I use TIVTC with libavcodec (MPeg 2 codec), I d'ont have 24 fps but 28 fps.
If I use libmpeg2 instead of libavcodec, I have 24 fps

with libmepg2
http://nsa07.casimages.com/img/2009/05/10/mini_090510040236829569.jpg (http://www.casimages.com/img.php?i=090510040236829569.jpg)

with libavcodec
http://nsa08.casimages.com/img/2009/05/10/mini_090510040442784974.jpg (http://www.casimages.com/img.php?i=090510040442784974.jpg)


without TIVTC
http://nsa08.casimages.com/img/2009/05/10/mini_090510040600831936.jpg (http://www.casimages.com/img.php?i=090510040600831936.jpg)


If I use decomb there isn't any problem with libavcodec or libmpeg2

I use MPC-HC and ffdshow-mt 2940

It is very unlikely. With the way the code is, it would be a major task to multithread tivtc interally.

The problem is that with blu-ray movie at 30 fps, even my cpu Q6600@3.0ghz isn't enough powerfull. I must use Decomb but Decomb is less good than TIVTC

[tritical]

Some options that could speed up processing are:

tfm:
slow = 0
micmatching = 0 or 2

tdecimate:
chroma = false

Of course, changing those could change the output, but in most cases it probably wont be noticeable. I would try slow=0 in tfm and chroma=false in tdecimate, and see if it is fast enough.

[PzSniper]

***removed because my crossposting was a rule #8 infringement***

Sorry neuron2 :(

but...

why i keep gettig this error?
http://i36.tinypic.com/2d7de1k.jpg

[DivxBr]

Sorry for a (maybe) stupid question, but when the TFM plugin sends a popup message telling me that it created an xxxx-FIXED.d2v file, to use, this file (FIXED) must be used on TFM command only or in the mpeg2source command too? This FIX increases/decreases the movie's total frame count?

[tritical]

In both. It could increase or decrease the count... it tries to keep the frame count as close as possible to the original while fixing the field order changes. dgmpgdec also has a tool to do this fixing.

[DivxBr]

I tried the Fix d2v option (in DGindex), but it show no errors to correct(!).
My concern is that this increase/decrease or frames could lead to audio synch errors.

[tritical]

Can you post the d2v file? Usually, the change in frame count is only 1 frame so not noticeable. If you are concerned then you could always ignore it, and use a matching mode that can deal with changing field order.

[B.F.]

Can you post the d2v file? Usually, the change in frame count is only 1 frame so not noticeable. If you are concerned then you could always ignore it, and use a matching mode that can deal with changing field order.
Also has this problem.

Here my d2v file.
Hope it help.

[DivxBr]

Can you post the d2v file? Usually, the change in frame count is only 1 frame so not noticeable. If you are concerned then you could always ignore it, and use a matching mode that can deal with changing field order.

I dont have this file anymore (If it helps, I can generate it again). But I checked the corrected file against the uncorrected, and found one difference only in the last frames, so no problem.

My concern, is if there's a possibility with bad encoded material that the frame count difference accumulates (within corrected file), so we get "a lot of" frames of difference?

[thetoof]

Hi,

I can't find how to detect the small 30p vertical motion of this clip to create good VFR timecodes. http://forum.doom9.org/showthread.php?p=1361082#post1361082

Script:
i=source
match=i.tfm(slow=2,clip2=i.tdeint(2,edeint=i.nnedi2(-2),emask=i.tmm(1)))
bob=i.tgmc.selecteven() #use whatever tgmc version you like to test
match.trim(0,454)+bob.trim(455,503)+match.trim(504,0)
TDecimate(4, output="stats.txt",denoise=true)
#TDecimate(5,hybrid=2,input="stats.txt",mkvout="timecodes.txt")

Any help please?

On a side note, it would be nice to have the option of creating a vfr clip that'd mix 23.976, 29.97 and 59.94 for situations like these (where full motion is actually 60p and is mixed with 24p content).

[tritical]

In mode 5 with vidDetect=3 and no tmfIn file, tdecimate assumes all matches are c. So the only reason a cycle (every 5 frame group, so 0-4,5-9,10-14,etc...) wouldn't be marked as 30 fps is if one or more of the frame difference metrics in that cycle is < vidThresh. Thus, try lowering vidthresh (you can look at the debug/display output in mode 4 to find the lowest difference value in that range of frames). I replaced tgmc with bob() for quick testing, but it looks like with denoise=true you need vidThresh=0.8 for that section. However, in a few cycles I can't tell if there is or is not a duplicate... sometimes the difference metrics go down to 0.3x, but those look like dups to be me. Alternatively, you could use manual overrides to force video detection or manually edit the tdecimate stats file (increase the difference values).


On a side note, it would be nice to have the option of creating a vfr clip that'd mix 23.976, 29.97 and 59.94 for situations like these (where full motion is actually 60p and is mixed with 24p content).
I agree :), but at the time I wrote the hybrid handling in tdecimate I was only concerned with mixes of 24p and 30p content.

[thetoof]

Could you add a feature in tdecimate to handle 23.976, 29.97 and 59.94?

Hm, just thought about this:

Can I manually edit the timecodes to look like this - I mean, will it work? (I know I could test, but it's faster to get an answer here :p)

# timecode format v1
Assume 29.970030
0,39,23.976024
775,778,59,940060
994,997,23.976024

[tritical]

It's possible, but I don't have the time or desire to do it. It isn't a simple modification, at least an easy method isn't coming to mind. The code for tfm/tdecimate is a big mess. It might actually be easier to create a separate filter/program to do it based off tfm stat file and tdecimate stat files from tfm output and bobbed output.

I don't know why manually editing the timecodes file wouldn't work... assuming you splice sections (add the 60p video in the right place)/modify frame numbers as needed.

[thetoof]

Alrighty, thanks for the answer.

[Gser]

Interested in a plan to update the gorgeous plug -TFM!? I take them a long time and would like to know whether in the near future, its updating and some "delicious"))) because there is no limit to perfection, and should always be striving towards the ideal ))

Like support for dga files. :'(

[johnmeyer]

Is there a way to force TDecimate to decimate all combed frames? I have a source where TFM flags really bad frames as combed frames. I want to decimate these first, and then remove the remaining duplicates to achieve a given frame rate. The MIC metric in TFM shows 255 on these bad frames, so TFM is providing the information I need. I don't mind doing a two-pass, but I cannot figure out how to pass a file back into TDecimate that will force it to decimate these "255" frames first, and then use the metrics from TFM to further decimate by removing the true duplicates.

[edit]
I figured it out. This does what I want:tfm(display=false,mode=1,pp=1,cthresh=25,micmatching=3,mmsco=false)
tdecimate(mode=7,display=false, rate=23.976)

[ikarad]

Some options that could speed up processing are:

tfm:
slow = 0
micmatching = 0 or 2

tdecimate:
chroma = false

Of course, changing those could change the output, but in most cases it probably wont be noticeable. I would try slow=0 in tfm and chroma=false in tdecimate, and see if it is fast enough.
TVITC support rgb or only YUY2 and YV 12 colorspace?

[turbojet]

Hi, is there any chance adding support for reading frame type (film or ntsc) from dgi files to TFM like the d2v function?

Here's an example (http://www.mediafire.com/?n1yl2w5t2n1) of a dgi file that has both ntsc and film frames.

[D3C0D3R]

i've use TIVTC couple of years. it amazing filter pack.

I've found serious memory leak (about 1MB-2MB per second) when use tdecimate mode=5 (2nd pass of hybrid)
with serious filter chain - couple of them writted by tritical too )) *BIG RESPECT*

i've placed this chain before and fter tDecimate, but leak is still appear I even remove some memory & cpu most hungry filters but it seems doesnt depend from type of filters and it position in script.

my settings
tdecimate(mode=5, hybrid=2, vfrDec=1,mkvOut=mkvTimeCodeOut,input=metricStats, vidThresh=15,dupThresh=10, vidDetect=1, cycleR = 1, cycle=2)
encoded by VDub+x264vfw one of latest versions/revisions

I didnt use TFM to do output="matches.txt" at first pass.
I haven't use TFM at all, only TDecimate - to remove duplicates in progressive stream.

In mode=3 all works fine, but mode=3 produces blackness at the end of stream and dont change count of frames.

But I wanna use 2-pass (mode=4 stats -> mode=5 work) because i run batch and dont want to crop end of every file manually

Maybe exists some option that I havn't set or i've found a bug?

[tritical]

Could you provide an exact full script that causes the issue? I'm not sure how tdecimate could be leaking memory since in mode 5 it doesn't allocate any memory during frame requests... only during script initialization. Does the memory issue happen if you add setmemorymax(64) to the script (i.e. is this 1-2MB per second just due to avisynth's cache filling up at the beginning).

[D3C0D3R]

>>setmemorymax(64)
yes, i did it at immediately when i found this leak. didnt help.

>>> (i.e. is this 1-2MB per second just due to avisynth's cache filling up at the beginning).
no, it grows all time during encode and finally eat all 3Gb :( and throw Exception

here my a bit simplified script that causes leak

common init part

src = "s02e01.avi"
metricStats = "metrics_01.txt"
mkvTimeCodeOut = "mkv-timecodesfile_01.txt"
SetMemoryMax(80)
LoadVirtualDubPlugin("MSU_cartoon_restore.vdf","MSUCartoonRestore", 0)
AviSource(src,audio=false)
Lanczos4Resize(width*2, height*2)
ConvertToRGB32()
MSUCartoonRestore("bilateral", 1, 56, 36, 1)
spline64resize(width/2, height/2)
ConvertToYUY2()


1st pass

tdecimate(mode=4, output=metricStats)
crop(0,0,8,8)


2nd pass

tdecimate(mode=5, hybrid=2, vfrDec=1,input=metricStats, mkvOut=mkvTimeCodeOut, vidThresh=15,dupThresh=10, vidDetect=1, cycleR = 1, cycle=2)

[Zep]

1280 x 720p source
Set MT build 2.5.8

=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)


116 FPS


=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

58 FPS

=================================

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

TDecimate(mode=0,cycleR=1,cycle=5)

39 FPS


=================================


mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

BilinearResize(320,180)
TDecimate(mode=0,cycleR=1,cycle=5)

28 FPS

More time used to resize down than decimate at full size. interesting!
Though 47 FPS if I MT() the resize but that is for another thread.

=================================


mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

ResizedVideo=BilinearResize(320,180)
TDecimate(ResizedVideo,mode=0,cycleR=1,cycle=5,clip2=last)


3.1 FPS


I thought it would be around 28 FPS like the previous example.
Is this a bug? Did i do something wrong?



Thanks

[tritical]

Without looking into it in depth, I'd guess it's a cache issue. Try:

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

ResizedVideo=BilinearResize(320,180)
TDecimate(ResizedVideo,mode=0,cycleR=1,cycle=5,clip2=last.blur(1.0,0))

Or replace blur(1.0,0) with some other fast filter.

[Zep]

Without looking into it in depth, I'd guess it's a cache issue. Try:

mpeg2source("D:\whatever.d2v",idct=5)
ChangeFPS(last,last,true)
Selecteven()

ResizedVideo=BilinearResize(320,180)
TDecimate(ResizedVideo,mode=0,cycleR=1,cycle=5,clip2=last.blur(1.0,0))

Or replace blur(1.0,0) with some other fast filter.


I tried your suggestion and a few other filters and no improvement :(


thanks

[Zep]

ok just now i tried

TDecimate(ResizedVideo,mode=0,cycleR=1,cycle=5,clip2=ChangeFPS(last,last,true))

and FPS is at 9 FPS


that is the only filter I have passed thus far that has helped.
Since that filter grabs 10 frames and caches them it appears it is
as you said a cache issue.

[mastrboy]

Any chance we will see a updated version of TFM with support for .dgi files (DGDecNV) similar to the support already implemented for d2v files?

[Lyle_JP]

Any chance we will see a updated version of TFM with support for .dgi files (DGDecNV) similar to the support already implemented for d2v files?

I have no trouble using TFM with DGDecodeNV as my source filter. :confused: You just need to set field order manually.

[mastrboy]

Not meaning as a source filter, off course that works. TFM has the parameter called d2v="filename", which aids in the IVTC process, currently it only seems to support the format from DVD2Avi and DGIndex d2v files.

From the manual:

d2v -

This option is intended to be used if you are using an mpeg2source() with a d2v file.
It sets the name and path to a d2v file, which TFM will analyze to see if there are any
illegal field order changes and optionally set the order parameter using the field
order of the d2v file. If the d2v file is found to have illegal field order transitions,
TFM will create a fixed d2v file with the string "-FIXED" attached to the end of the file
name. The new file will be located in the same directory as the original. You can then
use this fixed d2v file for processing. If the order parameter is set to "-1" then TFM
will detect the field order from the d2v file and set the order parameter to match. Depending
on the value of the "flags" parameter, TFM will also use the d2v info for field matching
and will pass info from the d2v on to tdecimate to help aid duplicate detection and hybrid
detection.

*NOTE: This option currently supports all d2v formats that I am aware of... which
include: dvd2avi 1.76, 1.77.3 and its variants, all dvd2avidg versions, and
all dgindex versions.

example => TFM(d2v="myd2v.d2v")

Default: "" (String)

flags -

Controls how much of the info from the d2v file is used when the "d2v" parameter is set.
Possible options:

0 - Check the d2v file for illegal transitions and set the order parameter if it
is not already manually set. Also, pass on rff flag duplicate info to
tdecimate.
1 - Same as 0, plus use the trf flags for field matching in film sections (sections
where the trf flags follow the 012301... pattern)
2 - Same as 1, but use the trf flags for field matching in all areas (doesn't have
to be in 0123 pattern) (very much not recommended!)
3 - Same as 0, but don't pass on any info to tdecimate (i.e. only set order and check
for illegal transitions)
4 - Same as 1, but d2v matches are checked for being combed. If a d2v match is detected
as combed then tfm uses its own matching routine for that frame.
5 - Same as 4, but d2v matches are only checked for being combed around scenechanges.

VERY IMPORTANT (MUST READ): For options 0, 1, 2, 4, and 5 to work correctly, tfm/tdecimate
must either immediately follow the mpeg2source() command or
any filters inbetween them must not modify the order or number
of fields in the stream in any way. Or use a trimIn file to
tell TFM what frames have been discarded.

Default: 4 (int)

[Stereodude]

I just wanted to post a thank you here for tritical for the great work on TIVC (tfm / tdecimate). The ability to use one clip as the basis for another's decimation with TDecimate ( clip2= ) really helped out my workflow on a few clips I was working on getting back to p24.

Thanks! :cool:

[manolito]

I.E. optimum quality vs. speed solution

Just want to find out if this line (posted by tritical a few years ago) is still state of the art:

yadifmod(edeint=tdeint(mthreshL=0,mthreshC=0))

A simple TomsMoComp or Yadif has too many quality issues, and all those ultra high quality (and ultra slow) deinterlacing scripts are just unusable for my rather slow machine. I always liked the quality and the speed I got from the above command. Is there anything new out there which surpasses it in terms of speed vs. quality?


Cheers
manolito

[tritical]

A simple TomsMoComp or Yadif has too many quality issues, and all those ultra high quality (and ultra slow) deinterlacing scripts are just unusable for my rather slow machine. I always liked the quality and the speed I got from the above command. Is there anything new out there which surpasses it in terms of speed vs. quality?

My recommendation would be QTGMC (http://forum.doom9.org/showthread.php?t=156028) with the slowest preset that meets your speed requirements. I don't think there is much question that QTGMC is the way to go for deinterlacing in Avisynth nowadays. Getting it setup can seem difficult, but Vit has made it very easy if you read the first post. If you can't find a preset that betters your current script in speed/quality report back :).

[manolito]

Thanks a lot for the tip, I will give it a thorough test run tonight...

Cheers
manolito

[manolito]

Alright, here's what I've come up with:

First of all, setting up QTGMC really was a little difficult, because my CPU (P3 Coppermine) does not support SSE2. Some of the plugins Vit provides are SSE2 plugins, and of course I got a floating point error. So I had to replace the SSE2 plugins with different versions while still trying to not brake LSFMod or LimitedSharpenFaster. Took me some time, but it works now.

Now for the speed tests:

I took an interlaced TFF DVD source, no filters except the deinterlacer, feeding it to QuEnc. These are the speeds for the different deinterlacers:

TomsMoComp: 11.77 fps
Yadif: 11.02 fps
YadifMod(edeint=tdeint(mthreshL=0,mthreshC=0)): 6.34 fps
QTGMC very fast: less than 1 fps
QTGMC draft: 4.65 fps

QTGMC is only usable with Preset="Draft" on my machine, and I do have my doubts if the resulting quality can match the quality of "yadifmod(edeint=tdeint(mthreshL=0,mthreshC=0))". I don't have the knowledge to analyze the QTGMC script enough to understand what the "Draft" preset really does, but it seems to use Yadif with Yadif's own spatial prediction, and I have the feeling that YadifMod using TDeint's prediction might do a better job....


Cheers
manolito

[-Vit-]

"Draft" is an extremely cut-down version of the core TGMC algorithm. It doesn't use Yadif, it performs a motion-compensated temporal binomial smooth to remove shimmer, but it only uses a bob as an interpolate, does not mask against creating blur and does no resharpening. As it's name implies it's not intended for final usage. QTGMC only really becomes useful at "Super Fast" and above IMO.

No offense intended, but you can't really ask for the "state of the art" if you intend to use a Pentium 3.