TDeint and TIVTC [Archive] - Page 19

Isochroma

24th January 2007, 19:38

Just noticed that b13 of DGIndex came out. One of the changes listed is:

* The Correct Field Order option is removed and the field order correction function is now available through the Tools menu: Fix D2V. I did this because this correction should not be applied to streams with frame repeats, and trying to prevent it via a code check runs into technical complications.

Will a wrong field order have any effect on TIVTC?

ChiDragon

25th January 2007, 02:39

Oh, I didn't know I had to use tfmIn for it to work! Sorry about that. It makes sense considering it would need to grab every frame to get the hints otherwise. The only mention I see of that in the TDecimate readme is in the Changes List section. Maybe you should add it to the tfmIn description for us dummies. :p (right now it says that it's only useful for hybrid > 0)

I get an invalid specifier error when trying to use tfmIn for files output with PP=0 in TFM though. :(

tritical

25th January 2007, 19:00

@Isochroma
If you use the d2v="" parameter in tfm, then it checks for illegal field order changes in the d2v (wont work with frame repeats, but tfm shouldn't be needed on those streams)... so it wont matter there. If you don't use the d2v parameter, use a matching mode that can't handle field order changes (only modes 4 and 5 can), and the stream does have field order changes, then there will be a problem.

@ChiDragon
An explanation in the tdecimate readme is probably a good idea. Another way to get around the need for the tfmIn file is to set hints=false in tdecimate. However, then tdecimate wont get to use any info from tfm. The basic chain of events when tdecimate runs is:

startup:
0.) If hints isn't manually set by the user, then grab frame 0 and detect if there are hints in it. If there are, set hints=true, else, set hints=false.

during operation:
1.) If there's a tfmIn file, don't read hints for frames that the tfmIn file has info for.
2.) else if hints=true, reads hints for frames that don't have info from a tfmIn file.
3.) else if hints=false, never read hints

I'll check out the problem with the pp=0 files.

tritical

29th January 2007, 09:46

Found and fixed the problem with tdecimate reading tfm output files created with pp=0. I'll put up a new version soon (a work around would be to just use pp=1 in tfm, though it will be slightly slower than pp=0 depending on the matching mode). I am also considering changing the sdlim default in tdecimate so that when cycleR > 1 it defaults to the largest possible value such that there would never be any cases where it couldn't mark a full cycleR worth of frames. I'm thinking that when someone uses cycleR>1 they are generally looking for the decimation to be spread throughout the cycle as sdlim allows instead of actually removing the cycleR frames with the lowest metric (which could all be next to each other).

ChiDragon

30th January 2007, 07:28

Thanks! Yeah, I've been using PP=1 in the meantime.

I'm not sure defaulting sdlim to the largest possible value is a good idea. With the videos I'm encoding now that need sdlim=2, setting sdlim to 3 (or -3) makes TDecimate drop non-dups often. Actually, it drops wrong frames every cycle, but most of them don't matter as it's simply dropping the first of two same-frames (they're only "wrong" since you'd normally want to drop the 2nd frame, with the low metric). The trouble comes at the start and end of the cycles, where it may drop an actual dup at the end of a cycle and then drop a new frame at the start of the next cycle.

ChiDragon

1st February 2007, 04:27

I've got a weird crash happening now... I'm using this ovr line: 80194,80303 ++-++-++++-++++-++++-++++-++-++++-++++-++++-++++-++-++++-++++-++++-++++-++-++++-++++-++++-++++-++-++++-++++-++++-++++-++-++++-++++-++++-+++-+++-++++-++++-++++-+++-+++-++++-++++-++++-++-++++-++++-++++-++
and whenever I try to seek to any of the frames in that cycle, VirtualDub crashes. It's the last cycle of the video, and it doesn't crash without that line. It also doesn't crash if the same line of specifiers is used for earlier frames.

cacepi

1st February 2007, 04:51

I've got a weird crash happening now... I'm using this ovr line and whenever I try to seek to any of the frames in that cycle, VirtualDub crashes.
Just a stab in the dark, but why is your ovr line so long? You have almost twice as many overrides as frames in the range.

ChiDragon

1st February 2007, 11:22

Well, I took the ovr line for a full cycle and just copied and pasted it to this partial cycle (partial because the video ends before it completes). I haven't had problems with redundant overrides before, and using that full line for the range "79992,80002" doesn't cause a crash.

Terranigma

6th February 2007, 00:46

tritical, I know this is offtopic, but I was wonderin' if you could perhaps compile speedier versions of Didée's filters such as SeeSaw, Deblock_QED, LimitedSharpenFaster, Mcbob, Soothe, Spresso like how you did for Vinverse?

Probably not gonna happen, but I thought i'd ask since it usually don't hurt to do so :D

btw, keep up the nice work. :)

Edit:
tritical, if you or someone else who's interested in porting these filters to .dlls need the scripts, (the latest versions), let me know.

tritical

7th February 2007, 22:36

The biggest problem is that I don't know the inner workings of a number of the scripts you metion, some of which are very complex, so it would take me some time to go through them before I could even start turning them into filters. Some of the simpler ones, like Soothe and Spresso, I could do pretty easily, but I don't think there would be much of a speed up since they are mainly just lutxy calls. Though in memory intensive scripts it might help since it would cut down on the number of filters/clips being created and thus cache instances.

Could you post a link to the latest versions of Soothe/Spresso, just to make sure I'm looking at the right ones? I might take a stab at them sometime.

Terranigma

7th February 2007, 23:55

Sure. Here (http://www.savefile.com/files/474238)'s a link for both soothe & spresso :)

Chainmax

17th February 2007, 01:15

I'm having a weird issue: when using the folloeing IVTC lines:

Deinted=TDeint(order=1,field=1,type=1)
TFM(d2v="C:\SimpS7\Epi1.d2v",order=1,mode=6,PP=7,slow=2,mChroma=false,Clip2=Deinted)
TDecimate(mode=1)

I receive the following error message:

http://img403.imageshack.us/img403/8568/errmsgba3.png (http://imageshack.us)

If I remove the "d2v=..." part in the TFM line, the .AVS loads without an issue. What could be causing this?

Alain2

17th February 2007, 03:15

It's written.. apparently you set order=1 and the d2v info probably says order=0...

tritical

17th February 2007, 03:34

Chainmax

17th February 2007, 03:56

Alain2: the video is TFF for sure and I even put an AssumeTFF() before the code snippet I posted. Alas, I upgraded to dgmpgdecv1.49b14 right before trying to load the scripts, whereas previous attempts used b13, so tritical is right on the money :).
I patiently await for news, master :D.

foxyshadis

17th February 2007, 05:57

Alain2 is correct... the error literally means that the field order value you gave to tfm isn't the same as the field order the d2v file indicates. Also, the latest dgindex/dgdecode (v1.4.9 beta 14) will not work with tfm's d2v option because of a new field that was added to the data lines (this may be what is causing your problem since the extra field means tfm never gets to the trf flags). I'll add support for d2v format v16 in the next release.

It was punishment for your hybris in removing the version check. ;p

Perhaps it's time to create a well-defined format and some smarter shareable parser code to go with it? (Not to be an xml-solves-all-things sort, but it would make the parsing easier. But then you'd have to include expat, meh.)

Terranigma

17th February 2007, 16:57

I was gonna post about this other day, but I did'nt wanna bother tritical too much; but support for the latest beta version would be great.:p

tritical

17th February 2007, 22:26

@ChiDragon
Could you post the entire script you were using that caused the crashes? I tried to reproduce with:

blankclip(pixel_type="YV12",length=80304,fps_denominator=1001,fps=30000)
tdecimate(ovr="ovr.txt")

where ovr.txt had the single line you posted, but it doesn't crash for me.

@foxyshadis
Yeah, I thought it was quite funny that the very next version after I removed the check wouldn't work. However, adding support for new d2v format versions isn't actually much of a problem atm (it took only 4 lines of code to support v16).

When you say create a well defined format and sharable parser code are you talking about creating a parser separate from tivtc and having it create a file that tfm could read instead of tfm parsing the d2v itself?

Terranigma

18th February 2007, 16:50

How are you guys actually using an ovr file with tdecimate? I'd try everything to specify a dropped frame or pattern, and it seems to be of no effect. The frame that i've specified to drop would still be there.

I'd use tfm with tdecimate sorta like this
tfm(mode=4,pp=1,slow=2,flags=1,hint=true)
tdecimate(mode=1,expp=true,ovr="C:\Myovr.txt")

The "ovr" file would read something like this
217 -

tritical

18th February 2007, 18:31

Terranigma

18th February 2007, 18:44

@Terranigma
ovr file works fine for me. Could you add display=true to the tdecimate() line, go to the cycle that includes frame 217 of the input (should start at frame 172 of the output), grab a pic of one of the frames, and post it here? It will show '**' next to whichever frame in the cycle has been dropped.
OK. when I entered 217 -, frame 172 was dropped.
I thought the way it works is, it'd drop whatever frame(s) I entered in the ovr file. Thanks for the help! :)

ChiDragon

19th February 2007, 23:44

tritical

20th February 2007, 01:17

Thanks, when getting the ovr info it wasn't adjusting the maximum number of frames that could be dropped if the last cycle wasn't full length.

TIVTC v1.0.1 (http://bengal.missouri.edu/~kes25c/TIVTCv101.zip), changes:
tfm:
+ Support d2v file format v16
- fixed a bug in tfmpp destroying hints
- fixed generating ovr help info about combed frames in output files when PP=0

tdecimate:
- fixed reading in tfm output files created with PP=0
- fixed a bug in destroying hints on frame output
- fixed a bug in drop/keep frame overrides in the last cycle of a video if the cycle
was not full length

Terranigma

20th February 2007, 01:20

tritical. what can I say.... other than thanks?
Thanks for the updated release! :D

Terranigma

20th February 2007, 16:40

tritical, if you ever decide to release another update, could you perhaps add another mode for BFF material? Matches for only c & n frames? :)

tritical

21st February 2007, 08:44

Not sure I understand your request, since there shouldn't be any need for a field order specific matching mode. With bff material it only makes sense to match c/n if matching off the top field... just like it only makes sense to match c/n for tff material if matching off the bottom field.

If you have a bff clip and want to match off the top field use:

tfm(order=0,field=1,mode=0)

in which case tfm will only use c/n matches.

Terranigma

21st February 2007, 15:43

ok thanks. I figured it out yesterday. :)

radar

21st February 2007, 20:11

hi,im trying to clean up this clip...http://rapidshare.com/files/16554591/VTS_01_1.VOB.html.

im a noob to using filters and writing scripts.would TDeint and TIVTC work on this clip and if so how would the script be written.thanks

foxyshadis

23rd February 2007, 16:59

I posted another compile of MT EEDI2, exact same source, but the latest version of ICL9 works with VC2k5 SP1 now. (That's what the i in imp stood for. :p) I patched it so hopefully it works on Athlons, I'd like some feedback, whether it works and if it's any faster than the VC one.

http://foxyshadis.slightlydark.com/random/Eedi2mt.zip

Xesdeeni

1st March 2007, 03:56

Using tivtc, I have a video that ends up slightly stretched in time. The video is the correct duration, but the last frame is not at the end of the source. As a result, my audio is out of sync with the video.

Details are in this thread (http://forum.doom9.org/showthread.php?p=956612#post956612), but the synopsis is:

I'm converting an OTA HD TS captured from ABC to HD DivX (our local ABC affiliate converts the 720p to 1080i, hence the need for tivtc). I used DGIndex 1.4.9 beta 14 to extract the AC3 and create the D2V file. I used AVISynth 2.57 and with some help, I was able to determine that the video was telecined at something like 23.2603 fps. For the best results, I'm creating a 23.2603 fps progressive video and remuxing it with the original AC3 using VirtualDubMod and the latest DivX codec.

I used the following script:MPEG2Source("Toy Story (WFAA).d2v")
AssumeBFF()
Crop(0, 0, 0, -8)
tfm() #mode=5
tdecimate(mode=2, rate=23.2603)
LanczosResize(1280, 720)When I view the video (in VirtualDubMod, but I had already encoded the video to DivX when I noticed the audio sync issue on the PC and on my Zensonic Z500), I see that although the duration of the video is correct, the video is not finished on the last frame. The credits fade to black in the source, but the fade is not there when viewing the telecined version. On this 1:19:37.96 video, a little more than 1 second is cut off. This is the cause, but the effect I noticed was that the audio was out of sync by a bit over a second at the end of the movie (it was fine at the beginning).

This is with your latest release, version 1.0 final.

I noticed in playing around, that if I use 23.976, it doesn't seem to end early.

FDecimate doesn't seem to do this, but it has serious problems picking the right frames to discard in this noisy (fairly significant pre-processing and MPEG-2 artifacts) source.

Xesdeeni

Eggroll

2nd March 2007, 06:18

Hello, I noticed how trimin allows the trimming of the d2v prior tfm operation. Is there a way to add frames to the d2v before tfm instruction to do the exact opposite? Allowing for duplicateframe for instance.

Thank you in advance.

tritical

5th March 2007, 01:12

@Eggroll
There isn't, but it could be added as part of the trimIn parameter... could you give the exact details of what you are wanting to do?

@Xesdeeni
I'll try to figure out what's wrong with mode 2. It was never tested much with non-standard frame rates.

@radar
It's pure interlaced, so a basic script would be:

mpeg2source()
tdeint()

I would recommend that you search the avisynth usage forum and try out some other deinterlacers to see which is best to you in terms of speed/quality. Some common deinterlacers are: leakkerneldeint, tdeint, tdeint+eedi2, securebob, mvbob, mcbob. You'll also have to decide whether or not you want double framerate (bobbed) output or not. Sports footage with fast action and sharp lines/edges is probably the toughest type of video to get good deinterlacing results on.

Terranigma

5th March 2007, 22:40

Guys. what would be the correct way deinterlace ntsc film material? Doing a field blend or interpolation? When I do a interpolation deinterlace, the deinterlaced frame becomes a clone of the next frame causing jerky playback, so field blending seems to be the best way to go. Perhaps there's another way I should go about doing this? :rolleyes:

radar

6th March 2007, 00:13

hi tritical,thank you for your reply.

i have another question and sorry if its a dumb one.
i can see why you put tdeint() in the script but what is mpeg2source() for.again sorry if this is a dumb question.

thanks

i think i answered my question.im using dvd rb with the filter,and rb puts mpeg2source() in for me.if im wrong please correct me.thanks

jeffy

6th March 2007, 00:37

hi tritical,thank you for your reply.

i have another question and sorry if its a dumb one.
i can see why you put tdeint() in the script but what is mpeg2source() for.again sorry if this is a dumb question.

thanks
For loading your MPEG-2 file, after you indexed it with DGIndex:
http://www.neuron2.net/dgmpgdec/DGIndexManual.html#WhatIsDGIndex

http://neuron2.net/dgmpgdec/QuickStart.html
http://www.neuron2.net/dgmpgdec/DGDecodeManual.html#MPEG2Source

Leak

6th March 2007, 10:04

Guys. what would be the correct way deinterlace ntsc film material?[...]Perhaps there's another way I should go about doing this? :rolleyes:
If it really is film, as opposed to video, you need to do field matching followed by decimation to get back to the original 24FPS framerate, i.e. you need TIVTC.

TFM(order=0 or 1)

followed by

TDecimate(mode=1,hybrid=0 or 1)

should be a good starting point...

Eggroll

6th March 2007, 15:35

@tritical

I wanted to "edit" a d2v so the frames would match the ones from another d2v (r1 vs r2 in this case) prior tfm operation. Since to match r2 with r1 I need to trim some frames here and duplicates some frames there, it was impossible in this situation.

I used another approach though. I removed the d2v parameter and simply trim or duplicate frames between mpeg2source and tfm instruction. Works like a charm and I don't have to use a trimin file anymore.

I guess it could be a nice feature to add if you've got some time to waste... If you can trim so why not be allow to add duplicate frames? But since I seem to be the first to ask for such a thing and that I found a workaround, I guess there really is no need for such a feature at this time.

Anyway, thank you very much.

Terranigma

6th March 2007, 15:41

If it really is film, as opposed to video, you need to do field matching followed by decimation to get back to the original 24FPS framerate, i.e. you need TIVTC.

TFM(order=0 or 1)

followed by

TDecimate(mode=1,hybrid=0 or 1)

should be a good starting point...

Actually, it's ntsc video 30000/1001 that I ivtc'ed and decimated to film ensuring that each frame is different. I was wonderin' if i should deinterlace first, then decimate, or decimate then deinterlace.

radar

6th March 2007, 18:03

jeffy

thanks for your reply,and thank you very much for the links.

manono

7th March 2007, 03:56

I was wonderin' if i should deinterlace first, then decimate, or decimate then deinterlace.
Neither; field match followed by decimation (TFM().TDecimate()), just as Leak explained.

Leak

7th March 2007, 14:28

Actually, it's ntsc video 30000/1001 that I ivtc'ed and decimated to film ensuring that each frame is different. I was wonderin' if i should deinterlace first, then decimate, or decimate then deinterlace.
Well, what you *shouldn't* do in any case is to unconditionally deinterlace the result of fieldmatching and decimation, as that will only worsen the image quality

That's why TFM has postprocessing options to actually deinterlace combed frames that resisted field matching. Try to tweak those options to eliminate any interlaced remains.

Terranigma

7th March 2007, 16:29

Thanks Leak, manono. :)

ChiDragon

12th March 2007, 06:58

tritical, I've found another weird bug... :confused:

Try this:

BlankClip(pixel_type="YV12",length=11,fps_denominator=1001,fps=30000)
TFM(input="test1.txt")
TDecimate(input="test2.txt",display=true)

With test1.txt:
#TFM v1.0.1 by tritical
5 p - [60]
6 c - [118]
7 p - [6]
8 c - [17]
9 p - [3]

test2.txt:
#TDecimate v1.0.1 by tritical
4 21320 546816
5 7194 169921
6 23761 599224
7 5737 181105
8 13369 452464
9 2606 170827

Frame 5, with a metric of 2.12 is dropped instead of frame 9 (0.77). If TDecimate is set to mode = 1 instead of 0, or hint = false instead of true, the correct frame is dropped!

tritical

12th March 2007, 08:29

@ChiDragon
It happens due to one of the 'special handling' functions mode 0 uses over plain "drop the frame with the lowest metric mode." When match information from tfm is available, tdecimate will detect match duplicates and one of the things it then looks for are standalone duplicates (a frame that is a match duplicate and neither of its neighbor frames are). It then goes through the cycle and determines which of these standalone duplicates to decimate (assuming there is more than 1) based on the sum of the next frame's difference metric and the previous frame's difference metric... whichever standalone has the largest sum is chosen. The only extra check is that the selected frame has at least the n'th lowest metric in the cycle, where n is the number of standalone duplicates. In this case, there are 3 standalone duplicates, frame 5 has the largest sum, and it has the 3rd lowest metric in the cycle... so it gets decimated.

There should probably be a check related to the selected frames metric (either hard, or relative to the lowest metric in the cycle) before it can be chosen for decimation. Including the current frames metric in the decision about which standalone duplicate to choose would probably also be a good idea. Looking at the code, it seems I had included a variable to keep track of the selected frames metric, but it never gets used for anything :rolleyes:. Anways, I'll take a look at this later this week when I should have some free time again.

Leak

12th March 2007, 09:18

Hi tritical,

in case you're interested - I've written a patch for ffdshow's AviSynth filter (my latest build is here (http://forum.doom9.org/showthread.php?p=968844#post968844), more info here (http://forum.doom9.org/showthread.php?p=968561#post968561)) that allows you to use TFM/TDecimate (and of course a slew of other filters) on the fly during playback. Maybe you (or whoever else reads this, of course) would like to give it a try? :)

tritical

12th March 2007, 09:59

Awesome, I quickly tried it out on my laptop (the ffdshow version on here was from 2005 :eek:) and it worked great. If anyone has problems with tivtc using too much cpu, changing the following settings should speed things up without sacrificing much:

tfm(slow=0,mchroma=false,pp=2 or 3)
tdecimate(chroma=false)

While I was testing, the cpu load went to 100% briefly on a few complex scenes (my laptop has a 1.6Ghz pentium M) and averaged ~60-65%. With the changes mentioned above it never broke 85 and averaged ~50-55%.

Also, there might be the smallest bit of jitter due to tdecimate. Currently, while it is returning frames for the current cycle it computes the metrics for the next cycle one frame at a time (it does that so that it doesn't have to request and compute metrics for all 5 frames at the beginning of each cycle). However, that only takes care of 4 frames and at the beginning of each cycle 2 frames will have to be requested and computed. I couldn't notice it though. Probably tdecimate is fast enough that it doesn't matter (even on my laptop it runs > 1000fps, assuming no blending is required).

Leak

12th March 2007, 10:35

Awesome, I quickly tried it out on my laptop (the ffdshow version on here was from 2005 :eek:) and it worked great.
:D

It never really had a chance of working before as the old AviSynth filter implementation in ffdshow always returned the same frame no matter which frame got requested by AviSynth - so I set out to fix that... :)
While I was testing, the cpu load went to 100% briefly on a few complex scenes (my laptop has a 1.6Ghz pentium M) and averaged ~60-65%. With the changes mentioned above it never broke 85 and averaged ~50-55%.
It hardly goes above 30% on my Athlon 64 X2 4400+, but since that's a dual-core machine it means about 60% of one core is used. But hey - at least I can now watch my anime DVDs like they were meant to be watched, and using my tiny projector (http://www.samsung.com/uk/products/projectors/mobileprojector/spp300memxedc.asp) (don't laugh) to boot... :)

Now my patch only needs to get checked into the ffdshow-tryouts SVN...

Chainmax

12th March 2007, 13:41

tritical, here's a sample of a scene that's been giving me trouble:

http://www.bestsharing.com/files/HL7M0Yw241160/Sample.demuxed.m2v.html

The deinterlacing portion of the script is this:

AssumeTFF()
Interp = SeparateFields().SelectEven().EEDI2(field=1)
Deinted=TDeint(order=1,field=1,type=1,edeint=Interp)
TFM(d2v="C:\SimpS7\e3\Epi3.d2v",order=1,mode=6,PP=7,slow=2,mChroma=false,Clip2=Deinted,micmatching=3)
TDecimate(mode=1)

which I think leaves some residual combing but I'm not sure wether it's that or the picture itself is just fux0red. Could you please take a look at it?