Dropped frames

[stephen22]

I have a video with dropped frames. There is always a duplicated frame adjacent but unfortunately it is sometimes before and sometimes after.

Filldrops or a slight modification thereof can repair either type but not both. Using YDifferenceFromPrevious or ToNext from current and previous frames, it's easy to distinguish between the two types. Is it possible to get this information for use in a runtime filter?

Didee and John Meyer did some nifty scripting using SelectEvery (clip,1,x) which seemed to address the problem but the script (I think called "dropped") doesn't work here and I think I may have copied it wrong and I can't find it again in this forum.

This must be quite a common problem, so before I start to attack it, has it already been done? Any other suggestions?

[johnmeyer]

This link will take you directly to the best version of the script that I posted in that thread. You can go to the first post to read what the script does and how it works.

Automatically fix dups followed (eventually) by drops

Even though the title says that the drops are "followed" by dups, the way the script works it will fix the video whether the dups are before or after the drops, so don't get hung up on that wording.

Depending on your video, you may want to keep the detection threshold pretty high, as I did in the script I posted. It has to be <1, and I had it at 0.8.

The only issue with the script is that it still has to make some attempt to figure out where to fill in the drop (finding dups is relatively easy; finding the drop location is tricky). StainlessS or someone else who has spent a lot of time comparing frames and looking for difference metrics might be able to suggest 1-2 lines of different code that would be more sensitive to a big discontinuity (a "drop") than my code.

The code that might be improved is all in the GenerateMask function. All that function does is to compare the current frame YDifference metric to the YDifference metrics from the preceeding two frames and the two frames which follow. If there is a local "spike" then a mask is generated which, given how the script works, will offer that frame as a potential for filling in with a motion estimated frame.

Fortunately, the way the code works, if you end replacing the wrong frame, it usually isn't too noticeable.

I am not aware of any other script which attempts to do this, so I think you are better off starting with this and then trying to perfect it. I included quite a few comments in the script, so hopefully you can understand what it does and how to tweak it.

If you post 5-10 seconds of video that is really bad, others may be willing to see if they can improve my code.

[StainlessS]

You might like to try out DropDeadGorgeous() [Part of (and requires) Duplicity2]
It works something like this

Originally Posted by StainlessS

Duplicity2(), is a dupe Detector, Dupe frame list creator, Dupes replaced with exact dupes, Dupes replaced with interpolated frames + DBase of dupes creator.
DropDeadGorgeous2() uses Duplicity2 DBase to do the almost impossible, smoothing out jerking clips (where source has jerks and dupes, much better than simple interpolation of dupes).
DropDeadGorgeous2 can also [EDIT: help] up framerate (eg 25.0 -> 29.97) keeping all source frames verbatim, and creating smooth result
[squeezes out dupes, where are replaced by interpolated frames between frames of most movement - non dupe frames are relocated between jerk interpolations and dupes to squeeze out dupes (hard to explain it)].

EDIT: Perhaps this explains better

abcXdefgDDhi       Src, Two dupe [D] frames preceded (within a set range) by a jerk frame. Jerk frame X, has greater motion to its predecessor than do d, e, f, or g.
abc12Xdefghi       DropDeadGorgeous dupe removal and Jerk interpolation

Lower case non dupe non Jerk
X     = Frame of greatest motion from predecessor (Jerk)
D     = Dupe Frames
Digit = Interpolated frames between X and its predecessor c (Same Count as D dupes)

Duplicity2:- https://forum.doom9.org/showthread.php?t=175357

[johnmeyer]

I haven't used DropDeadGorgeous (although I've seen the movie), but the way you describe it, I don't think it will help with the OP's video. His problem -- and the problem most people have with captures which drop frames -- is NOT the duplicate that is inserted sometime after the drop in order to keep the audio in sync, but is the drop itself. Detecting the exact duplicates that are inserted is trivial, but figuring out where there is a gap in the motion is dreadfully difficult, even though it is (unfortunately) very easy for the eye to see.

To do a REALLY good job at detecting temporal gaps would require access to the internals of a motion estimation DLL, like MVTools2. That tool keeps track of small blocks of video and attempts to track their movement. I would expect that there is some set of numbers which describe the vector of each block of pixels. If those vectors could be exposed, then I think they would "blow up" on any frame where the contents have moved too much, relative to nearby frames.

That makes me think of something: maybe one of the motion estimated scene detection metrics, like MVTools2 MSCDetection, could be used instead of my comparison metric.

[StainlessS]

In above quote, the X Jerk frame is the frame following a missing dropped frame, and DD, one or more dupes.

EDIT:
During Capture, when a frame is dropped, subsequent frames in the clip are temporally advanced, up until a duplicate is produced, after that it is back in sync.
What we do [as in my prev post] is to reverse that temporal advance, by inserting [in descibed case] 2 interpolated frames before the X jerk frame, and relocate X->DD [excluding DD] later in clip, squeezing out the dupes.

abcXdefgDDhi       Src, Two dupe [D] frames preceded (within a set range) by a jerk frame. Jerk frame X, has greater motion to its predecessor than do d, e, f, or g.
abc12Xdefghi       DropDeadGorgeous dupe removal and Jerk interpolation

Lower case non dupe non Jerk
X     = Frame of greatest motion from predecessor (Jerk)
D     = Dupe Frames
Digit = Interpolated frames between X and its predecessor c (Same Count as D dupes)

# below assumes that 2 frames both dropped and duped, similar where 1 frame dropped/duped. [2 frames dropped is extreme and more difficult case, we describe the more difficult]

abc12Xdefghi  # How it should have been in capture, 1,2 will both be dropped, X is jerk frame following frames that will be dropped.
abcXdefgDDhi  # when 1,2 dropped by CAP device, and DD dupes inserted.
abc12Xdefghi  # DropDeadGorgeous() fixed. (1,2 are both interpolated frames [c<=>X], originals GONE $4E4 [forever])

EDIT: Also, all non-dupe source frames are kept verbatim.

[MysteryX]

Take a look at this: InterpolateDoubles

"Replace double frames with interpolated frames using FrameRateConverter"

[StainlessS]

Thanks MX, I'm sure that we do something similar-ish already.
[Although we temporally relocate frames between X jerk frame [inclusive] and dupe frames [exclusive] to squeeze out the dupes,
and interpolate dropped double frames between the new relocated X jerk frame and the frame that was immediately prior to the X jerk frame].

EDIT: And see prev post edits.

[johnmeyer]

Originally Posted by StainlessS

what we do [as in my prev post] is to reverse that temporal advance, by inserting [in descibed case] 2 interpolated frames.

abcXdefgDDhi       Src, Two dupe [D] frames preceded (within a set range) by a jerk frame. 
                   Jerk frame X, has greater motion to its predecessor than do d, e, f, or g.
abc12Xdefghi       DropDeadGorgeous dupe removal and Jerk interpolation

Yup, that is what is needed.

By contrast, InterpolateDoubles seems to only be fixing the duplicate problem. MugFunky developed FillDrops() years ago, doing pretty much the same thing. I refined that a little by extending it to interlaced footage.

It would be great if the OP could post some footage so we could each try out our toys and see what the results look like.

[StainlessS]

Yep, I thought that MX doubles whotis was same as MugFuncky thingy, below is better, in-case you did not see it.

# below assumes that 2 frames both dropped and duped, similar where 1 frame dropped/duped. [2 frames dropped is extreme and more difficult case, we describe the more difficult]

abc12Xdefghi  # How it should have been in capture, 1,2 will both be dropped, X is jerk frame following frames that will be dropped.
abcXdefgDDhi  # when 1,2 dropped by CAP device, [EDIT: X->g are temporally advanced] and DD dupes inserted.
abc12Xdefghi  # DropDeadGorgeous() fixed. [EDIT: X->g are temporally Delayed and dupes squeezed out]. 1,2 are both interpolated frames [c<=>X], originals GONE $4E4 [forever].

In bottom line, 1,2 are interpolated from c,X.

[poisondeathray]

There is always a duplicated frame adjacent but unfortunately it is sometimes before and sometimes after.

I'm assuming it's something like this, where c is the dropped frame. "adjacent" implies right next to , at least to me, not a few frames later or before

abbdef - if dup occurs before, replace 2nd dup
abddef - if dup occurs after, replace 1st dup

Here is a test clip with both cases
https://www.mediafire.com/file/cvawf...tclip.mp4/file

I played with DropDeadGorgeous2(), and it caught the 2nd, but not the 1st. But I just used default settings. SS probably tweak it correctly

[StainlessS]

But I just used default settings. SS probably tweak it correctly

Probably not, I did not anticipate dupe before drop, and to try do both at once is a big ask,
is bloody difficult doing just the implemented one.

EDIT: Might be do-able if is KNOWN that dupes are IMMEDIATELY ADJACENT to drop.

EDIT: Got the clip thanks PDR.
EDIT: Nice test clip.

[poisondeathray]

Smoothskip caught both in that testvid . I don't know if they are immediately adjacent for the OP, but that's how I read it

I chose 65, because that's roughly the distance between the blips

orig=LSmashVideoSource("drops_and_dups_testclip.mp4")

s=orig.SmoothMotion(cycle=65, skips=1, dupes=1)

stackvertical(orig,s)


function SmoothMotion(clip last, int "cycle", int "dupes", int "skips", int "debug" ) {
  cycle  = default(cycle,  4)
  dupes  = default(dupes,  1)
  skips  = default(skips,  1)
  debug  = default(debug,  0)
  TDecimate(cycle=cycle, cycleR=dupes, display=(debug == 1))
  super = MSuper()
  bv = MAnalyse(super, overlap=4, isb = true, search=3)
  fv = MAnalyse(super, overlap=4, isb = false, search=3)
  inter = MFlowInter(super, bv, fv, time=50, ml=70)
  SmoothSkip(inter, cycle=(cycle-dupes), create=skips, offset=-1, debug=(debug == 2))
}

https://forum.doom9.org/showthread.php?t=172377
https://github.com/jojje/SmoothSkip

[poisondeathray]

Also, the OP says "Filldrops or a slight modification thereof can repair either type but not both. Using YDifferenceFromPrevious or ToNext from current and previous frames, it's easy to distinguish between the two types." - so that has to be directly adjacent n-1 or n+1

[Frank62]

What would you think about simply:

Filldrops()

Reverse()
Filldrops()
Reverse()

?

[StainlessS]

What would you think

Maybe, so long as it did not stuff up the other ones on first pass.
[Not sure if it would get slower and slower though - could try I suppose, but not me - I'm off to Ballet practice soon.]

[poisondeathray]

Originally Posted by Frank62

What would you think about simply:

Filldrops()

Reverse()
Filldrops()
Reverse()

?

It's what SS said - it messed up on that test clip because it should have left one alone for the 2nd pass Filldrops. But there is a duplicate there and it so it "blindly" interpolates it. Since the interpolated frame is in the "wrong" place, the jump remains

[StainlessS]

Anyways, to get you all in the mood to best solve this, see this VERY enlightening post from the big G himself, Gavino:- https://forum.doom9.org/showthread.p...78#post1409378
Its only 3 pages of posts, best read the lot, very interesting, Didée also makes an appearance.

I''m gonna see if I can get some sleep, went to bed at about 06:30, up again about 09:00 and about 10 mins kip in between,
had a couple of orange juices at ballet practice, maybe that helps a little. [Althougn I might have a large glass of apple juice too before the feathers].

[johnmeyer]

My script posted above fixed it perfectly (this is WAY too easy a test case). I did, however, have to change the double fps from 59.974 to 48, since the test case is 24 fps.

If this is going to be a "thing," then I should probably re-work the script to put the parameters that need to be changed at the front of the script. Here is the one change to fix the trivially easy test case:

Change from this:

double = source.MFlowFps(super, bvec, fvec, num=60000, den=1001, blend=false)

to this:

double = source.MFlowFps(super, bvec, fvec, num=48, den=1, blend=false)

P.S. I'm not knocking poisondeathray and the test he manufactured, but only pointing out that, since it is so trivially easy to fix, it won't really provide much information on which approach is most likely to work for real-world cases.

[poisondeathray]

Originally Posted by johnmeyer

My script posted above fixed it perfectly (this is WAY too easy a test case). I did, however, have to change the double fps from 59.974 to 48, since the test case is 24 fps.

If this is going to be a "thing," then I should probably re-work the script to put the parameters that need to be changed at the front of the script. Here is the one change to fix the trivially easy test case:

Change from this:

double = source.MFlowFps(super, bvec, fvec, num=60000, den=1001, blend=false)

to this:

double = source.MFlowFps(super, bvec, fvec, num=48, den=1, blend=false)

P.S. I'm not knocking poisondeathray and the test he manufactured, but only pointing out that, since it is so trivially easy to fix, it won't really provide much information on which approach is most likely to work for real-world cases.

Did you change anything else other than source filter and that "double" line? Your script does not work for the 2nd blip when I tried it earlier (I already made those changes when trying earlier). Check around frame 119,120, there is a jump in motion

Smoothskip works on this "trivial" test case

[poisondeathray]

here is the comparison video (top and bottom borders cropped, I labelled it "john" because that script function did not have name)
https://www.mediafire.com/file/c2qo3...mpare.mp4/file