Script for reliably detecting dropped frames

[johnmeyer]

Dropped frames are common in streamed video and also in screen captures. Many people ask how to fix them. For reference to prior art, here are a few discussions about how to do this:

Dropped frames

Automatically fix dups followed (eventually) by drops

I created a script which I posted in those threads which sometimes does a good job fixing the problem (yes, there is still an issue with TDecimate, but I think I can fix it).

However, even if I come up with a fix to the TDecimate problem it needs a MUCH better method for detecting jumps.

It is clear that motion estimation should do a much better job at finding temporal gaps in video than the YDifference method I have been using. To test my first attempt at a motion estimation detection method, I decided to create my own test clip, similar to what poisondeathray did in the first thread linked to above. I wanted a variety of different drop/dup situations, so that's why I'm not using his clip. Here is a link to that tiny 100k video file:

https://www.mediafire.com/file/cof5y..._test.mp4/file

Here is my first attempt at doing detection using motion estimation:

source = AVISource("e:\fs.avi").ConvertToYV12 
super = MSuper(source)
backward_vectors = MAnalyse(super, blksize=32, isb = true, delta=1)
forward_vectors  = MAnalyse(super, blksize=32, isb = false, delta=1)
inter = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70) 
diff1 = overlay(source,inter, mode="subtract") 
diff1 = ScriptClip(diff1, "Subtitle(String(Averageluma()))")
return diff1

It simply estimates where the next frame should be and compares it to the actual frame. It spikes when a drop happens and then goes to zero when the dup happens. It also spikes on the frame after a dup, but that false spike can easily be ignored (although if there is another drop immediately after the dup, it won't find that -- a second pass with a slightly different script would be required).

I need help on two things.

First, I can't get the estimation to line up. I really don't want to estimate halfway between two frames, but instead want to get the estimate of what the next frame should look like. Using time=100 doesn't do that, so I'm stuck with time=50.

Second, I'm not sure Overlay is the best method to compare (subtract) the two frames. I tried mt_makediff, but it didn't do as well.

Hopefully someone out there with better skills with MVTools2 can suggest a few changes. I use this tool a lot, but even after fifteen years, and even after many discussions in this forum, I still haven't quite wrapped my head around it.

[zorr]

Originally Posted by johnmeyer

To test my first attempt at a motion estimation detection method, I decided to create my own test clip, similar to what poisondeathray did in the first thread linked to above.

Your test clip is good for getting started on the problem but eventually the solutions should be tested with more realistic scenarios. The good thing is that it's easy to create test clips where some frames are removed and some are duplicated. With such a clip it would be possible to let Zopti find the optimal settings for whatever scripts anyone comes up with. That's why the test clip(s) need to be as realistic as possible, if there's any way to get a good result by cheating Zopti will find it.

Originally Posted by johnmeyer

First, I can't get the estimation to line up. I really don't want to estimate halfway between two frames, but instead want to get the estimate of what the next frame should look like. Using time=100 doesn't do that, so I'm stuck with time=50.

The time=50 is ok but you need to use delta=2 and when comparing the original clip has to be forwarded by one frame. Or you could use MCompensate instead and use delta=1 (time not needed).

Originally Posted by johnmeyer

Second, I'm not sure Overlay is the best method to compare (subtract) the two frames. I tried mt_makediff, but it didn't do as well.

I once started working on a scene detection method which had almost the same idea, using MCompensate to recreate the next frame from the current frame. If it couldn't be done well that would indicate a scene change. I decided to use a real similarity metric to compare the recreated and original frame, after all those algorithms are the best we have for comparing frames for similarity. So something like SSIM, GMSD or MDSI would probably work well here.

It's also possible to get the SAD (sum of absolute differences) of the matched blocks using MVTools MMask kind=1. Those could be compared by getting the average brightness. Brighter than usual could mean a dropped frame.

[johnmeyer]

Originally Posted by zorr

Your test clip is good for getting started on the problem but eventually the solutions should be tested with more realistic scenarios.

I totally agree and that is what I repeatedly said in the other thread that you've been following. This test clip is simply being used to make sure the script is functioning, but it doesn't guarantee it will work on real video.

Originally Posted by zorr

The time=50 is ok but you need to use delta=2 and when comparing the original clip has to be forwarded by one frame. Or you could use MCompensate instead and use delta=1 (time not needed).

I didn't mention in my original post, but that is what I started with before I posted. Here is my earlier version:

source = AVISource("e:\fs.avi").ConvertToYV12 
super = MSuper(source)
backward_vectors = MAnalyse(super, blksize=32, isb = true, delta=2)
forward_vectors  = MAnalyse(super, blksize=32, isb = false, delta=2)
inter = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70) 
diff1 = overlay(selectevery(source,1,1),inter, mode="subtract") 
diff1 = ScriptClip(diff1, "Subtitle(String(Averageluma()))")
return diff1

Unfortunately, because it is looking two frames ahead and behind, it fails to provide a sharp single-frame metric. Instead, you get two frames on either side of the drop where the metric increases. For this test video I can simply take the largest value in the center, but as you point out, with real-world video, this is not likely to work.

Originally Posted by zorr

It's also possible to get the SAD (sum of absolute differences) of the matched blocks using MVTools MMask kind=1. Those could be compared by getting the average brightness. Brighter than usual could mean a dropped frame.

I never paid any attention to that function. I just spent ten minutes trying to get something useful out of it, but failed. I did set it to use SAD.

source = AVISource("e:\fs.avi").ConvertToYV12 
super = MSuper(source)
backward_vectors = MAnalyse(super, blksize=32, isb = false, delta=1)
inter = MMask(source,backward_vectors,kind=1)
diff1 = overlay(selectevery(source,1,1),inter, mode="subtract") 
diff1 = ScriptClip(diff1, "Subtitle(String(Averageluma()))")
return diff1

Just to make sure everyone understands, the code I posted in #1 actually does work, but I think it could work a LOT better with some hints from someone who really understands MVTools2.

[poisondeathray]

Originally Posted by johnmeyer

Unfortunately, because it is looking two frames ahead and behind, it fails to provide a sharp single-frame metric.

delta=2 is not the explanation - you can demonstrate that the results are exactly the same with delta=1 , by using 2 interpolation groups with delta=1 and selecteven, selectodd then interleaving them

WhateverSource()
source=last
super = MSuper(source)
backward_vectors = MAnalyse(super, blksize=32, isb = true, delta=2)
forward_vectors  = MAnalyse(super, blksize=32, isb = false, delta=2)
inter = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70)


WhateverSource()
source2=last

source_even = source2.selecteven()
source_odd = source2.selectodd()

super_even = MSuper(source_even)
backward_vectors = MAnalyse(super_even, blksize=32, isb = true, delta=1)
forward_vectors  = MAnalyse(super_even, blksize=32, isb = false, delta=1)
inter_even = MFlowInter(source_even,super_even, backward_vectors, forward_vectors, time=50, ml=70)

super_odd = MSuper(source_odd)
backward_vectors = MAnalyse(super_odd, blksize=32, isb = true, delta=1)
forward_vectors  = MAnalyse(super_odd, blksize=32, isb = false, delta=1)
inter_odd= MFlowInter(source_odd,super_odd, backward_vectors, forward_vectors, time=50, ml=70)

inter2=interleave(inter_even, inter_odd)

subtract(inter, inter2)
Levels(127, 1, 129, 0, 255)

When the interpolation references are "not good" - that will "contaminate" the n+1_est frame - ie. if n or n+2 is one of the drops or duplicates , or some other "bad" frame

[StainlessS]

Maybe Pel=1 [think default=2, not sure], and/or BlkSize=16.
EDIT: yep, from Old MvTools: "Default is 2 since v1.4.10." so assume mvtools2 is same. Yep, same for mvtools2.

[Frank62]

Originally Posted by poisondeathray

delta=2 is not the explanation - you can demonstrate
...

So delta=1 with compensate and delta=2 with FlowInter, time=50 will give the exactly same result? Did I get that right?

By the way: I would add truemotion=false, if not default.

[johnmeyer]

Originally Posted by poisondeathray

delta=2 is not the explanation - you can demonstrate that the results are exactly the same with delta=1 , by using 2 interpolation groups with delta=1 and selecteven, selectodd then interleaving them.

When the interpolation references are "not good" - that will "contaminate" the n+1_est frame - ie. if n or n+2 is one of the drops or duplicates , or some other "bad" frame

I disagree with your first statement, but agree with the second. The problem only happens when I change the two deltas from 1 to 2. I make no other change. Instead of getting a single, clean "spike," I get a half-spike on either side of the bad spot. The reasons, I think, are what you describe in your second statement.

So, it may not be the explanation, but changing only delta, and nothing else, does break it.

I had a long interchange in this forum about six years ago trying to get to the bottom of how the vectors work and what frames are really being used when estimating intermediate frames. I'm going to go back and re-read that entire thread, and then I'm going to carefully read the new MVTools2 documentation. Someone spent quite a bit of time adding information to it, and it looks like there may be a few clues there.

I might glance at the source code, as I did yesterday for TDecimate (I've never seen so many If statements in my life). I have written a little C, but not enough to be proficient. However, if the code has a few comments, I may be able to glean a few things.

[poisondeathray]

To clarify, the n+1 frame is not being interpolated mflowinter when delta=1

A delta of 2 for mflowinter means the n+1 frame is being interpolated. Mvtools2 needs to look at n+2 frame for the foward vector reference in order to interpolate the n+1 frame. A delta of 1 means you're looking at the n+1 frame - the same frame cannot be used as interpolation reference for itself


How do you propose to get around "bad" references? They might be placed anywhere in a real-life bad video that has drops/dupes

[johnmeyer]

Originally Posted by poisondeathray

How do you propose to get around "bad" references? They might be placed anywhere in a real-life bad video that has drops/dupes

I'm not sure what you mean by "bad references." Perhaps you mean references to the dups, which can also make the detection logic spike and would also fool the motion estimation. That part is easy because dups are almost trivial to detect (YDiff works quite well), so once detected, I'll simply exclude them from the detection logic. However, I first have to get the basics worked out. That step comes later.

If you look at my existing detection logic, I did exactly what I just described (excluding the dup numbers from the comparison logic), and also took care of the other problem, namely that most detection schemes have to be relative to the local video. It is almost never a good idea to just use a fixed, unvarying threshold or detection number.

[poisondeathray]

Originally Posted by johnmeyer

I'm not sure what you mean by "bad references." Perhaps you mean references to the dups, which can also make the detection logic spike and would also fool the motion estimation. That part is easy because dups are almost trivial to detect (YDiff works quite well), so once detected, I'll simply exclude them from the detection logic. However, I first have to get the basics worked out. That step comes later.

If you look at my existing detection logic, I did exactly what I just described (excluding the dup numbers from the comparison logic), and also took care of the other problem, namely that most detection schemes have to be relative to the local video. It is almost never a good idea to just use a fixed, unvarying threshold or detection number.

Yes, I meant any reference that messes up expected interpolation result - If one or both are flawed (eg. interpolating from wrong frame), you don't get the expected result, and that could result in false positive. But it sounds like you're way past that already, so ignore that and carry on...

[johnmeyer]

Originally Posted by poisondeathray

But it sounds like you're way past that already, so ignore that and carry on...

I'll take all the help I can get.

[johnmeyer]

Well, the idea didn't work. YDiff produces better results. I'm going to try to fix the TDecimate problem and then post the drop/dup fixer script.