View Single Post
Old 29th June 2011, 23:30   #1  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,685
Automatically fix dups followed (eventually) by drops

Overview

A friend wanted a portion of this video of his son placing third in the state 3200 M track race:

2011 CIF Track & Field Championships

Keepvid wasn’t an option; WMRecorder and similar tools didn’t work. So, I had to use Camtasia (or SnagIt) which records screen video. Unfortunately, despite reading all the hints, I didn’t get smooth video.

Here’s a link to a really bad section of the video:

10 Second Problem Clip

If you want to try the script, download this clip. If you want to suggest a better way to capture the video, click on the first link above.

The first sixty frames (of the downloadable clip) has a huge number of problems, and then things settle down.


Problem I Tried To Solve

In a nutshell, the video randomly (i.e., not periodically) drops frames and then, to restore sync, duplicates a frame, usually within about five frames of the dup. So, here's the statement of the problem: I want to delete the dups and simultaneously synthesize a frame to insert at the point where the video jumps.


Prior art

This has been discussed before, both here:

Filter to remove duplicate frames

and here:

Inverse of Decimate

Also, MugFunky, a long time ago wrote a script for detecting and replacing exact duplicates:

FillDrops

However, his problem only included duplicates: apparently the capture mechanism did not drop frames to make up for the dups.


Solution

Didée created a nifty script for interpolating missing frames. This can be found in the "Inverse of Decimate" link above. However, there were three things missing in that script that I needed for my situation:
  1. I couldn’t find any way to easily and reliably tweak the mask settings in order to change the threshold for determining the size of the "jump" between frames needed to trigger the logic for creating a motion-estimated replacement.
  2. As a corollary to #1, I couldn’t figure out a way to put meaningful metrics onto the screen or into debugview so I could see, frame-by-frame, what sort of metrics correlated to the amount of visual discontinuity between frames. It is hard to tweak a threshold setting without metrics.
  3. I needed different decimation than did the OP in that thread, both because I didn’t need to go between 30 and 24 fps, but also because my duplicate and drop distribution is quite different (much more random).
  4. Despite the power of masking, I actually found that I could achieve far more "accurate" and reliable detection of major movement by doing a simple moving average of the YDiffPrevious values from surrounding frames, and then taking a ratio of that average with the value for the current frame. This approach largely eliminates the problem with Ydiff, namely that it can be very large for some parts of the video, and very small in other parts, depending on the amount of motion, contrast, etc.
So, with that as background, I created the script you see below. It seems to work well for this particular problem. However, if anyone wants to use it, here are a few issues that I didn’t quite solve.
  1. The TDecimate logic is pretty slow. I had to use rather large Cycle parameters because sometimes the dups/drops clustered in bunches. I tried to do a 2-pass TDecimate, which is supposed to be designed for exactly this sort of thing, and I have included that code below (commented out) to show what I tried. I tried various maxndl settings for pass-2, but never got results that were better (compared to the 1-pass solution shown below) by a big enough amount to warrant the extra grief of doing a two-pass decimation.
  2. Decimation still leaves some jumps. This seems to be a decimation problem and not a problem in the logic which detects jumps.
  3. The motion estimation is pretty ugly for some frames. I didn’t spend any time on this, and simply used the vectors that Didée created in that other thread. If this video were more important, I might try to tweak the settings to get better results.
  4. SetMTMode doesn’t seem to work reliably. I’m pretty sure, based on past posts by Didée and others that this is due to TDecimate. I was able to get the script to work for enough frames before it crashed to determine that SetMTMode wouldn’t improve the performance much unless I could come up with a better approach to decimation than TDecimate (i.e., because of the large cycle size, TDecimate was causing most of the slowness).

[edit]Click on the following link to go to updated script later in this thread:

Later version of script

Code:
# Based on script created by Didée
# Modified by John Meyer on June 29, 2011
#
# Script overview
#
# Create interpolated frames a 2x original frame rate using MVTools2
# Detect jumps 
# Create white mask at each jump point; black mask for all other frames
# Repeat each frame of original video and use mask to "choose" between original video, or motion estimated video
# Decimate exactly 50% to get back to original frame rate. 
# This decimation removes the dup frames from the original video and also the dups created by repeating each frame of original video.
# However, at each point where motion-estimated frame is inserted, no decimation occurs. Thus, if dups=drops, and the drop happens 
# within < "cycle" (TDecimate parameter) of the dup, the dup will be removed and the drop will be filled. 
# If no drops or dups occur within "cycle," then no motion estimation happens, and decimation merely gets back to original, 
# unchanged video.

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

#Threshold for detecting jumps. Increase to catch more jumps. Should always be less than 1.0
JumpThresh = 0.74
showdot = true # true for troubleshooting; otherwise, false

#SetMTMode(5,4)
global source=AVISource( "E:\CIF Track\3200-Complete (edited).avi" ).ConvertToYV12
#SetMTMode(2)

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )

super = showdot ? source.subtitle("***").MSuper(pel=2) : source.MSuper(pel=2) 
bvec  = MAnalyse(super, overlap=4, isb = true, search=4, dct=5) 
fvec  = MAnalyse(super, overlap=4, isb = false, search=4, dct=5) 
double = source.MFlowFps(super, bvec, fvec, num=60, den=1, blend=false)

#Remove comment from ShowMetrics, and change "return final" to "return test" to look at metrics in order to determine proper JumpThresh
#test=ShowMetrics(source)

#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)

interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate
RequestLinear
final=tdecimate(display=false,mode=1,cycleR=10,cycle=20)  # Decimate half of all frames (set to twice the length of "normal" dup/drop cycle)

#---------------------
#Alternate two-pass approach to decimation 

#Pass 1
#RequestLinear(debug=false)
#final=tdecimate(display=false,mode=4,output="e:\metrics.txt")

#Pass 2 (remember to un-comment "requestlinear")
#RequestLinear
#final=tdecimate(display=false,mode=2,rate=30,input="e:\metrics.txt",maxndl=20)
#---------------------

return final
#return stackvertical(source,final)

#----------------
#This function displays the YDiff value that will be used for detecting big jumps

function ShowMetrics (clip c) 
{
  fixed=source.ScriptClip("Subtitle(String(
    \ ((YDifferenceFromPrevious(selectevery(source, 1, 2)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1, 1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-2)) ) / 4 ) / 
    \ (YDifferenceFromPrevious(source) + 0.01)
    \ ))")
  return fixed
}

#----------------
#This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned

function GenerateMask (clip c)
{
  MyMask=c.ScriptClip("""
    \ ((YDifferenceFromPrevious(selectevery(source, 1, 2)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1, 1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-2)) ) / 4 ) / 
    \ (YDifferenceFromPrevious(source) + 0.01) <= JumpThresh 
    \ ? WhiteFrame : BlackFrame """)
  return MyMask
}

Last edited by johnmeyer; 30th June 2011 at 08:28. Reason: Fixed wrong explanation of dups & drops (under "Problem ..." heading); Added link to later script
johnmeyer is offline   Reply With Quote