59.94p to 23.976p - Removing blocky frames (a Q with an A, leading to another Q)

[DoctorM]

Sooo. I'm trying something... different.
I have an HDTV broadcast, 720p @ 59.94fps. The source is 23.976 so every frame has 1 to 2 duplicates (a 2,3,... pattern). The problem is after MANY of the cuts the first frame is pixellated and the second frame is clean.

A normal selectodd().decimate(5) gives 23.976 but it preferentially removes the duplicate frame, I want the crappier one removed.

I thought pixellated=more edges=restore24's algorithm.

I thought maybe Restore24 can handle it (if configured properly), except I can't get it to work. With 24deint set to 0 I thought I could use a progressive source, except: Error = Invalid arguments to function "duplicateframe".

Mrestore gave inconsistent results.

REALLY what I need is some way to use BlindPP (or other deblocker) to report the blockiest frames without deblocking so those frames can be decimated... but I wouldn't know where to start.

Just for the heck of it I set up a script with FDecimate(rate=23.976) (for comparison) and amazingly it seems to work. In THEORY FDecimate removes duplicate frames but it would seem that it decimates the FIRST frame not the subsequent duplicate frame. (And in cases of 3 frames it removes the first and third). As a result the WORST frames are dropped.

What's even stranger is the default threshold is 1, but in the case of the bad sections the metrics show: 58.332471 and 2.569145. Again theoretically neither frame should be removed, but the the 58.33 one is definitely gone and the 2.569145 remains.

I'm going experiment with this some more, but does anyone have an explanation (or suggestion)?

[rebkell]

I'd also give TDecimate(Cycle=5, CycleR=3) a try also, it's the one I use almost exclusively for my 720P HD captures, it's always served me well.

[DoctorM]

Nope, TDecimate leaves the blockiest frames behind.
It must be a quirk with how FDecimate was coded. I can't imagine the author planned for this sort of thing... but you never know because I'm starting to believe Donald Graft is a frickin' genius.

So far out of a dozen cases of bad frames FDecimate has removed 100% of them... I'll keep checking the video, but I'm about ready to declare this a success.

[Guest]

Thanks for the kind words. It wasn't a design goal. If you'd like to post an unprocessed source stream sample, I'll be able to look it over and tell you why FDecimate() is succeeding with it and whether it is something that can be relied on versus just a coincidence.

[DoctorM]

Ok here's a clip: http://rapidshare.com/files/126421039/HP2Clip.ts.html
This is a bad section. I see 5 instances of the problem in these 49 seconds.
Like I said above, the issue shows up right when there is a cut from one camera angle to the next; the very first frame after the cut is blocky.

What's working for me is a simple fdecimate(rate=23.976), and even though the "good" duplicate frames don't always fall below the default threshold of 1, the results are beautiful.
Accident or not... don't you dare touch that filter. :)

I also tried adding deblocking: mpeg2source("video.d2v",cpu=1). It doesn't seem to effect fdecimate's success, there's little impact on the picture detail, but does improve cases where the second frame isn't 100% clean.

[IanB]

I believe it is a coincidence.

Consider the codedec_data[] = {0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, ...} // For 29.97 -> 23.976

// Decimate to achieve configured frame rate.
// Avoiding delivering a duplicate if possible.
if (last < 0 || inframe == last || CalculateMetrics(last, dec_data[inframe], env) > threshold)
use = dec_data[inframe];
else
use = dec_data[inframe] + 1;
GETFRAME(use, src);
last = use;Assuming 2 3, 7 8, 12 13, 17 18 are duplicates then use will follow this pattern :-
0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 12, 14, 15, ...
Which is using the 1st of a pair of duplicates.

However if duplicates occur thus 4 5, 9 10, 14, 15 then use will follow this pattern :-
0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, ...
Which is using the 2nd of a pair of duplicates.

And if the blockey duplicates are different enough to exceed threshold then fdecimate will always follow the blind pattern from dec_data[].



And Don, some minor bugs:-
1. the inframe == last test always fails (and it trips on specific keyed accesses). I believe it should be dec_data[inframe] == last or similar.
2. The use of last in the metrics call assumes linear frame access. Random access will always deliver the blind pattern from dec_data[]. In the 1st case above if I ask for frame 2, then some other frames, then frame 3, I will get back input frames 2 and 3, whereas if I ask for frames 2 and 3 consecutively I will get back input frames 2 and 4. I believe you should be referencing dec_data[inframe-1].

[DoctorM]

Hmm. I'll have to check deeper into the film to see if the pattern changes leaving the blocky frames behind.

Edit:
If I understand correctly then if the duplicates are below the threshold, the duplicate frame(s) (second and subsequent) are removed.
But since in my case they always exceed the threshold, Fdecimate is using a blind decimation pattern and coincidentally (in the first chunk anyway), all the bad frames fall within the blind pattern's decimation.

My test seems to confirm that. Jumping to later in the film the blocks aren't removed.
What I really need is to be able to set a correct threshold for this video, but force fdecimate to remove the frame immediately BEFORE the duplicate.

Does that sound right? Any suggestions on how to make that happen (I haven't the slightest ability to code that myself)? Still, it would make an awesome filter for low bitrate HD broadcasts like this.

Edit2: Re-reading a 4th time, it sounds like if a frame falls below the threshold fdecimate may choose either the first OR second frame to remove depending on the relative position (I'm using IanB's interpretation since I don't understand the code)... that's an odd thing to do. Is there any reason for it? Again being able to force removal of the 1st frame 100% of the time would be a nice.

[IanB]

The difficulty is, if the frames are duplicates then it does not matter which one you use, i.e. duplicates means they are the same.

In your case the frames concerned are not duplicates. What you want is to establish a local pattern based on frames that are good duplicates and then when faced with a pair of frames, that temporally should be duplicates, but are not, you need to use the existing local pattern to override and force choose the 2nd frame.

In the past I have used Neuron2's older 2 pass multidecimate and Excel to graph the frame differences and then hand butcher the cells that were out of place.

[DoctorM]

Well, yes and no. The metrics, for the "duplicate" frames where one is blocky, are remarkably low. Low enough that you could set a threshold for the whole film.
In THEORY they aren't duplicates, but whatever algorithm calculates the difference finds them to be fairly similar compared to true new frames.

That's why I was trying something like MRestore and Restore24, since they are looking for the frame/field that has a bigger difference (yet MRestore had a 75% or better success rate when set to KEEP all blended fields (bf=-1)).

In the clip I posted (cropping the 4 right lines because of noise(?)) we get 3.45, 2.55, 3.30, etc. About half of the metric of lower motion scenes.

Thinking further about this, it would almost be a good idea to have a decimate filter with 2 thresholds.
One low one (defaulting around 1) for easily identifiable duplicates, and a slightly higher one (around 2 or 3) for cases where any noise, dirt or misalignment of the frame exists.
Then decimate based on <low threshold: decimate duplicate, between low and high threshold: decimate based on the lowest calculated metric of that group of frames, and then above the high threshold: blind decimate.
That way in cases where it's too close to tell, the metrics help decide, but low/no motion scenes will have better discrimination than if you have a single threshold that you've raised too high.

You could in theory make a filter always calculate and choose frames based on lowest metric(s) of a group of frames (with a single high threshold), but I don't know how slow that would become.

Just some random thoughts. You guys are smarter than I am on this stuff.

Mind you while I'm writing my nonexistent dream decimation filter, I would definitely add an integer parameter "decimation_frame" (or whatever). Default 1 to delete the second of the duplicates and allowing you to set 0 for deleting the first of the pair.

[IanB]

Thinking a little outside the box. The problem here is that the "duplicate" frames are not really duplicates. So how about making them true exact duplicates before decimating.

Don Graft has a filter, dup230.zip (http://neuron2.net/dup/dup230.zip), that does exactly this, and for bonus points the code always return the last in a sequence (or a blend of all the matching frames).

[DoctorM]

OMG, that's EXACTLY what I was talking about:
The last frame is used instead of the first because often the first frame after a scene change has more blocking artifacts, etc.

I'll give that a try later. I say it again: DG is a genius.

[DoctorM]

Nice, brilliant, fantastic. Absolutely what I was looking for. This filter was designed for exactly the problem I'm experiencing.

2 concerns with it...
First, the max block size is limited to 32. That's a bit small for working with 720p content, but it's a good trade-off for speed. I can work around that by tweaking the threshold.

Second, maxcopies seems like a bogus parameter.
maxcopies (1-20, default 20): Determines the maximum number of copies that can be emitted due to duplicate declaration. However, if the input stream has more than the maximum number of duplicates in a string, then, of course, they will all be output. This limit just forces the current duplicate sequence to end; the comparisons restart with the nextframe frame.

In other words you can try to limit it, but it'll still make as many duplicates as it wants. That's unfortunate since I know there is a 2/3 pattern to my video so there should never be more than 2 duplicate frames. Setting maxcopies to 2 (I'm assuming that means 1 original and 2 copies) I still end up with runs of 5-6 duplicate frames in low motion scenes (with unique frames that fall below the threshold being discarded). I can't lower the threshold any more, or I don't remove the frames I really want gone. It would be nice if the program used maxcopies as a cut off and jumped to the next group of duplicates.

Still, it is better than NOT having this filter, and after fdecimate thrashes through it, you are talking 1-2 duplicate frames/sec in the final output once in awhile. Hardly a deal breaker.

[Guest]

Stand by, guys. I've just enhanced FDecimate to give you just what you need. On your test clip it automatically and reliably skips the bad frames. Just putting together a release package now...

I did two things. The first one doesn't specifically address your issue but improves the decimation significantly by doing a better job of avoiding delivering duplicates. I now look at the next two frames after the blind decimate frame to find the first one that differs above threshold; if neither does, I deliver the blind frame. I used to just deliver the next frame if the blind frame was a duplicate. Second, and this is the one that helps you, I added an option that enables this: after calculating the frame to return, if the following frame is a duplicate of it, use that one instead. It means I'll never deliver the first frame of a string of duplicates. Note that the threshold must be properly set as described in the manual.

Regarding the reported bugs, the first one I agree with, and I deleted the test. The second I disagree with, because we are trying to prevent duplicates in the *output* stream, so comparisons have to made againt the last frame delivered to the output.

[Guest]

OK, here it is:

http://neuron2.net/fdecimate/fdecimate110.zip

For the sample clip linked above, I used this:

fdecimate(rate=23.976,threshold=8,protect=true)

[DoctorM]

Will definitely give this a spin.
That'll let me do what I want with one filter instead of two.
There is no way to thank you enough for the tweak.

Edit: And it most excellently delivers all unique frames in the low motion section.
My new favorite filter. :)

Edit2: I was finding some instances of duplicate frames being delivered with a unique one dropped (even though the metrics didn't support it). Turns out it's because about a dozen frames earlier there were duplicate frames misaligned vertically. I had to raise the threshold a bit more to keep it from screwing up the decimation pattern. (Which is then a trade off against the lower motion scenes. Sigh).

[jmac698]

I've run into this before. The problem can be even worse; some encoders interpolate between film frames creating non-existant frames. I figured out the problem was with the pattern of I,B,P frames. Never take a P after an I and one other such rule.
Anyhow, I know how to solve this problem. If you could make a decimate that takes at least two rules based on surrounding patterns of I,B,P I can write a pattern to keep the best frames.
Of course, this information is not available to script. So I'm hoping the amazing Neuron can serve frames this way; or expose them somehow.

Doc, step through in dgdecode and write down the frame type I,B,P surrounding the worst frames - you should be able to recreate my work.
IPB - the last B is almost always a good frame, for example.

A propsed rule string, "nABCD" where n is the good frame out of this pattern, ABCD are the frame types of the previous 4 frames. Each pattern is matched in turn to see if a rule covers it. The rules are evaluated at the time that you've decided what you *think* should be output when decimating. The rules override the default decision. Make sense?
eg "3IPB" would keep the current frame if the last 3 frames were I,P,B. If they weren't, it would check the next rule. When no more rules, just output this frame.

[DoctorM]

Can you recommend software that's good for identifying the frames of .ts files?

[DoctorM]

You should theoretically be able to get better accuracy on sub-threshold frames by having the filter guesstimate instead of blind decimating.
For example if your threshold is 6 and you are converting 59.94 to 23.976, you'd expect a 3/2 pattern.
If you then find a run like this: 23, .8, 1, 15, 2, 4.5, 1, 2.3, 3.7, 0, ...
You could extrapolate that the unique frames are 23, 15, 4.5, 3.7. Even though 4.5 and 3.7 are below threshold, they are the largest of a seemingly overlong string of duplicates and would match where the blind decimate should expect a new frame to be.

That is if a string of duplicates is longer than anticipated for the pattern, perhaps the larger-non-threshold frame should be picked.

Next-morning-more-awake edit: To elaborate, I AM getting a pattern similar to the above. Whereas you'd think blind decimation would see the pattern WITH the frames above threshold and then continue from there once they fall below threshold, but it doesn't. (Even though the actually pattern of duplicates in the film hasn't changed.)

Instead it's dropping a unique frame and putting in a duplicate. I tried turning protect to false, but that has no effect.
Can anyone explain how fdecimate switches from threshold decimation to blind decimation?
It looks a bit like it just sort of kicks in and starts dropping frames without regard to the pattern that went before it. Is that true?

[jmac698]

ya, dgindex.

[Guest]

Can anyone explain how fdecimate switches from threshold decimation to blind decimation?
It looks a bit like it just sort of kicks in and starts dropping frames without regard to the pattern that went before it. Is that true? No. FDecimate works by first making a mapping of the input frame number to the output frame number. This mapping is calculated just as a function of the two frame rates and without looking at the frames at all, so this is the "blind" mapping. Now, when a frame number is requested by Avisynth, FDecimate first looks up the required output frame from the blind mapping, call it frame X. Then it compares frame X to the last frame delivered, X+1 to the last frame delivered, and X+2 to the last frame delivered. The first one that differs above threshold is chosen. Then if protect is not enabled, the chosen frame is returned. If protect is enabled, the frame after chosen is returned if it is a duplicate of chosen.

I'm not aware of any decimation filter that implements pattern guidance. Maybe it's time to think about making one.

[DoctorM]

What I was thinking was for something like how Telecide works.
IIRC it finds the pattern to inverse telecine based on the guide and/or combing threshold. Then frames are declared in-pattern and out-of-pattern where out-of-pattern can reset the IVTC guide.
Something similar in a decimate would be slick. While things are above threshold and everything appears in patterm FDecimate would work as normal, but below threshold it would just continue with the pattern as it was last seen.
The pattern could also be reset as needed when above threshold metrics indicate it.

Honestly I sort of thought that was how most decimate filters worked. Just shows what I know.

[Guest]

Honestly I sort of thought that was how most decimate filters worked. How could that be if you have not told the filter what kind of pattern it is dealing with (compare Telecide's guide parameter)?

[DoctorM]

Good point.

Edit: Still it would be nice to see a decimate filter that DID have a guide.

Anyway thanks for the info and help.

[DoctorM]

I've found a bug in fdecimate 1.1.0.
It happens quite often in low motion scenes. It doesn't seem to be effected by Protect.)
Frames are returned in duplicate and out of order.

For example (in a case of 29.97 to 23.976 with low motion):

a=below threshold, b=below threshold, c=below threshold, d=below threshold, e=above threshold, f=below threshold, g=below threshold, h=below threshold, i=below threshold, j=below threshold.

Outputs as:
b,e,d,e,f,h,i...

Which, with a through d being frame 1 and f through j being frame 2, it then looks like: 1,1,2,1,2,2,2 and causes flickering.

Version 1.0.2 decimates better:
a,c,d,e,f,g,i,j

I can't tell you what's causing it, but something that was changed did it, I've had to go back to 1.0.2.

[Guest]

Please post a stream fragment that I can use to deuplicate the issue. Thank you.

[DoctorM]

Oops, sorry, I didn't have time... and then forgot about this.

The problem is easily reproduced in any case of opening credits on a still background... that is several seconds of a still title card that's then swapped for the next still title card.

I could throw a stick and hit 5 cases where this occurs, but if you want I'll snip something to show you.

[Guest]

Please post a stream fragment that I can use to duplicate the issue. Thank you.

[Dominiko]

FDecimate(protect=true) is useful. But sometimes it's impossible to detect all dublicates due to higher threshold, that also detects as dublicates good frames too. I think some improvement on protect function can handle this. Instead keeping frame n, and throwing frame n+1 (both are detected as non-dublicates, and metrics on n+1 frame are very low), it can do the inverse thing. I think there could be parameter threshold2 for this job. Please, neuron2, if you can, do something to handle this.