Nasty duplicate-frames

[Jedd]

Hello everybody.
I'm trying to make my first DVD-rip from the NTSC DVD disk, so I've put new avsynth 2.5 and decomb.dll on and started to play with it.
Movie material is hybrid telecined - you never can tell when the next interlaced frame will be, but anyway I telecided it back and what happens I'm getting duplicates all over the place (well as it should be). But the problem is - when I try to decimate them to bring it to 23.97fps it works vey weird.
When I do decimate(mode=0) it cuts off not only duplicated frame but original frame too - so I'm getting nice jerks here and there, because two adjacent frames were removed.
If I do Decimate(mode=3,threshold=1.0) it leaves the duplicates untouched, if I put treshhold to 50.0 and to 99.0 - it's still has no affect - although these two frames looks absolutely the same to my eyes.
Decimate (mode=2) tends to leave duplicate frames intact, but kills good ones.
Mode=1 leave some of the duplicate frames and blend another (but it leaves me with 29.9fps)
:confused:

[DJ Bobo]

This is the correct way to do Inverse Telecine with decomb:

Telecide()
Decimate(cycle=5)

[Richard Berg]

If you're sure it's an NTSC pattern, adding a "guide=" parameter to Telecide can help.

[neuron2]

Post an URL to a short clip that shows your problem and I will be happy to have a look at it.

[Jedd]

DJ Bobo
Telecide()
Decimate(cycle=5)
Somehow it doesnt work well. Decimate kills frame N5(original) and N6(duplicate) and leaves me with a nice "jerk"

neuron2
I appreciate it.
No Decimate (http://jedd.web1000.com/pass2_d0.avi)
Check frames N10 and N11 - they are our dups.
Decimate mode=0 (http://jedd.web1000.com/pass2_m0.avi)
Here both N10 and N11 got decimated. I don't understand how it could be.
Decimate mode=3 (http://jedd.web1000.com/pass2_m3.avi)
Some other frames got decimated and blended here, but N10 and N11 (now they became N8 and N9) persist.

[MrBunny]

@Jedd

Quick question first, the non-decimated sample is using telecide(), but no decimate, correct?

If that's so, based on that clip, the dupe frame pattern seems to vary, but does seem to be longer than the standard 3:2 pattern. As far as I can tell, between the first and second dupes are 5 frames and there are 6 frames between the 2nd pair and 3rd. A change in interval is strange, but could be just a case of a bad sample choice (pattern change at that instant or something).

My recommendation is to check that telecide only sample and see if you can determine if there is a pattern, whether it be a pair of dupes every 6 frames or more. If there is one, then choose the decimate cycle appropriately.
My offhand guess is that you have a PAL -> NTSC conversion, which would mean that there should be a pair of duplicate frames in each set of 6 frames. You would then use telecide(some args).decimate(cycle=6,mode=0,some args) to get it back to (almost) the native 25fps framerate.

I think that a longer (lower rez would be sufficient) sample with only telecide() might be helpful for us to look at further if you can't determine the problem.

If the pattern actually runs longer than the cycle of 5 that's been chosen, it's most likely that it's not actually that both dupe frames have been dropped (as you had suspected), but instead that the correct dupe has been dropped and somewhere down the line an incorrect one (between frames 9 and 10 in the mode=0 sample) was removed since decimate couldn't find a proper dupe to drop within the cycle of 5.
And FYI, mode=3 decimate is experimental...and in the shop?

Mr. B

[neuron2]

The clip appears to have been telecined from 25fps (not 24fps) to 30fps. This is not unusual. Such clips require Decimate(mode=6). There is a pattern guidance mode for this kind of telecining.

If this really is a cycle=6 clip, then anything you do with cycle=5 will create jerkiness.

Decimate(mode=3) is neither experimental nor in the shop. It just doesn't work quite as well as I had hoped (not due to bugs but simply the idea itself).

[Jedd]

I've just explored the longer sample and dups comes as follow:
1,2..7,8..14,15..19,20..25,26..32,33..37,38..43,44..50,51..55,56..61,62..68,69..73,74
So the range is 5,6,4,5,6,4,5,6,4,5,6,4 and so far
There is a pattern but how should I solve it?
I don't think this paticular movie has a lot of intended duplicates. Is there a way to kill 1 frame from every sequence of duplicates?

Aside question - how comes that decimate(mode=0) kills two frames in a row? I thought it should never do it.

[waka]

Try:

decimate(cycle=18).decimate(cycle=17).decimate(cycle=16)

It might have some quirks here and there, but I suspect it will work fairly well.

[neuron2]

Originally posted by Jedd
Aside question - how comes that decimate(mode=0) kills two frames in a row? I thought it should never do it. Decimate must remove one frame per cycle of frames. All follows from that.

This is the first time I have seen a clip like this. Fortunately there is a solution, as waka has so ingeniously pointed out.

@waka

My hat is off to you!

[Jedd]

Worked like a charm. :)
Thanks, guys!

[waka]

Glad to hear it worked ok.

@neuron2

Ahh, but it is you who deserves the taking off of hats. Afterall, workers are only as good as their tools and you created the tool...

On a more serious note, I wonder if the thinking behind the chaining of decimates together warrants some attention. Last time I looked(version 3.9x), you compared a group of cycle+1 frames and discarded the frame most similar to its predecessor. Obviously you can have problems when you end up with two duplicates in a group and none in the next, as what happened here.

If you were to compare cycle*N+1 frames and then discard the N frames with the lowest difference, decomb might handle these odd situations better. You could chain decimates together and get the same effect, but doing it in one go should(?) eliminate redundant checking and perhaps be faster.

Of course, false duplicates could be more harmful with larger cycles and you would have to figure a way to gracefully exit a clip that ends in the middle of a large cycle. Just something that the decimate chain made me think about.

[Jedd]

Actually another one..
decimate(cycle=18).decimate(cycle=17).decimate(cycle=16)
It leave us with 25fps, should it be converted or slowed down to 24fps?

[neuron2]

Originally posted by Jedd
It leave us with 25fps, should it be converted or slowed down to 24fps? No, because the duplicate data you gave indicates an average decimation of 1 in 6. So:

5/6 * 30 = 25fps

Leaving it at 25fps is correct and will not be an issue. For example, MPEG encoders accept that frame rate.

@waka

I agree that a "decimate n frames out m" syntax would be a useful improvement. I am considering other enhancements as well, such as decimating all (and only) duplicates, preferentially decimating combed frames, etc.

[Jedd]

Originally posted by neuron2
Leaving it at 25fps is correct and will not be an issue. For example, MPEG encoders accept that frame rate.

I'd like to restore this thread with yet another question.
I'm backing up the very same movie, but now it's a DVD era, so my script has to deal with that and simply reducing number of the frames is not very good any more.

The basic script I'm dealing with (intended for DVD Rebuilder) does some cleaning and removes black borders:

mpeg2source("H:\NEWS\MTMP\D2VAVS\V01.D2V",idct=7)
trim(0,2969)

Telecide(1,blend=false)
decimate(cycle=18).decimate(cycle=17).decimate(cycle=16)

before=Crop(2,2,716,476, align=true).Lanczos4Resize(720,480)

firstpass=RemoveDirt(before)
RemoveDirt(before, neighbour=firstpass)

ConvertToYUY2(interlaced=false)


If I leave it as is, decimated video yields 15% time difference with the audio.
AssumeFPS(25) doesn't do enything, ChangeFPS(25) inserts all dups back and ConvertFPS(25) blurs it quite nasty. Decimate(mode=1) takes a very long time and mixes up RemoveDirt.
Are there any elegant ways to solve this problem?