Automatically fix dups followed (eventually) by drops

[johnmeyer]

Overview

A friend wanted a portion of this video of his son placing third in the state 3200 M track race:

2011 CIF Track & Field Championships

Keepvid wasn’t an option; WMRecorder and similar tools didn’t work. So, I had to use Camtasia (or SnagIt) which records screen video. Unfortunately, despite reading all the hints, I didn’t get smooth video.

Here’s a link to a really bad section of the video:

10 Second Problem Clip

If you want to try the script, download this clip. If you want to suggest a better way to capture the video, click on the first link above.

The first sixty frames (of the downloadable clip) has a huge number of problems, and then things settle down.


Problem I Tried To Solve

In a nutshell, the video randomly (i.e., not periodically) drops frames and then, to restore sync, duplicates a frame, usually within about five frames of the dup. So, here's the statement of the problem: I want to delete the dups and simultaneously synthesize a frame to insert at the point where the video jumps.


Prior art

This has been discussed before, both here:

Filter to remove duplicate frames

and here:

Inverse of Decimate

Also, MugFunky, a long time ago wrote a script for detecting and replacing exact duplicates:

FillDrops

However, his problem only included duplicates: apparently the capture mechanism did not drop frames to make up for the dups.


Solution

Didée created a nifty script for interpolating missing frames. This can be found in the "Inverse of Decimate" link above. However, there were three things missing in that script that I needed for my situation:

1. I couldn’t find any way to easily and reliably tweak the mask settings in order to change the threshold for determining the size of the "jump" between frames needed to trigger the logic for creating a motion-estimated replacement.
   
2. As a corollary to #1, I couldn’t figure out a way to put meaningful metrics onto the screen or into debugview so I could see, frame-by-frame, what sort of metrics correlated to the amount of visual discontinuity between frames. It is hard to tweak a threshold setting without metrics.
   
3. I needed different decimation than did the OP in that thread, both because I didn’t need to go between 30 and 24 fps, but also because my duplicate and drop distribution is quite different (much more random).
   
4. Despite the power of masking, I actually found that I could achieve far more "accurate" and reliable detection of major movement by doing a simple moving average of the YDiffPrevious values from surrounding frames, and then taking a ratio of that average with the value for the current frame. This approach largely eliminates the problem with Ydiff, namely that it can be very large for some parts of the video, and very small in other parts, depending on the amount of motion, contrast, etc.

So, with that as background, I created the script you see below. It seems to work well for this particular problem. However, if anyone wants to use it, here are a few issues that I didn’t quite solve.

1. The TDecimate logic is pretty slow. I had to use rather large Cycle parameters because sometimes the dups/drops clustered in bunches. I tried to do a 2-pass TDecimate, which is supposed to be designed for exactly this sort of thing, and I have included that code below (commented out) to show what I tried. I tried various maxndl settings for pass-2, but never got results that were better (compared to the 1-pass solution shown below) by a big enough amount to warrant the extra grief of doing a two-pass decimation.
   
2. Decimation still leaves some jumps. This seems to be a decimation problem and not a problem in the logic which detects jumps.
   
3. The motion estimation is pretty ugly for some frames. I didn’t spend any time on this, and simply used the vectors that Didée created in that other thread. If this video were more important, I might try to tweak the settings to get better results.
   
4. SetMTMode doesn’t seem to work reliably. I’m pretty sure, based on past posts by Didée and others that this is due to TDecimate. I was able to get the script to work for enough frames before it crashed to determine that SetMTMode wouldn’t improve the performance much unless I could come up with a better approach to decimation than TDecimate (i.e., because of the large cycle size, TDecimate was causing most of the slowness).

[edit]Click on the following link to go to updated script later in this thread:

Later version of script

# Based on script created by Didée
# Modified by John Meyer on June 29, 2011
#
# Script overview
#
# Create interpolated frames a 2x original frame rate using MVTools2
# Detect jumps 
# Create white mask at each jump point; black mask for all other frames
# Repeat each frame of original video and use mask to "choose" between original video, or motion estimated video
# Decimate exactly 50% to get back to original frame rate. 
# This decimation removes the dup frames from the original video and also the dups created by repeating each frame of original video.
# However, at each point where motion-estimated frame is inserted, no decimation occurs. Thus, if dups=drops, and the drop happens 
# within < "cycle" (TDecimate parameter) of the dup, the dup will be removed and the drop will be filled. 
# If no drops or dups occur within "cycle," then no motion estimation happens, and decimation merely gets back to original, 
# unchanged video.

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

#Threshold for detecting jumps. Increase to catch more jumps. Should always be less than 1.0
JumpThresh = 0.74
showdot = true # true for troubleshooting; otherwise, false

#SetMTMode(5,4)
global source=AVISource( "E:\CIF Track\3200-Complete (edited).avi" ).ConvertToYV12
#SetMTMode(2)

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )

super = showdot ? source.subtitle("***").MSuper(pel=2) : source.MSuper(pel=2) 
bvec  = MAnalyse(super, overlap=4, isb = true, search=4, dct=5) 
fvec  = MAnalyse(super, overlap=4, isb = false, search=4, dct=5) 
double = source.MFlowFps(super, bvec, fvec, num=60, den=1, blend=false)

#Remove comment from ShowMetrics, and change "return final" to "return test" to look at metrics in order to determine proper JumpThresh
#test=ShowMetrics(source)

#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)

interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate
RequestLinear
final=tdecimate(display=false,mode=1,cycleR=10,cycle=20)  # Decimate half of all frames (set to twice the length of "normal" dup/drop cycle)

#---------------------
#Alternate two-pass approach to decimation 

#Pass 1
#RequestLinear(debug=false)
#final=tdecimate(display=false,mode=4,output="e:\metrics.txt")

#Pass 2 (remember to un-comment "requestlinear")
#RequestLinear
#final=tdecimate(display=false,mode=2,rate=30,input="e:\metrics.txt",maxndl=20)
#---------------------

return final
#return stackvertical(source,final)

#----------------
#This function displays the YDiff value that will be used for detecting big jumps

function ShowMetrics (clip c) 
{
  fixed=source.ScriptClip("Subtitle(String(
    \ ((YDifferenceFromPrevious(selectevery(source, 1, 2)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1, 1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-2)) ) / 4 ) / 
    \ (YDifferenceFromPrevious(source) + 0.01)
    \ ))")
  return fixed
}

#----------------
#This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned

function GenerateMask (clip c)
{
  MyMask=c.ScriptClip("""
    \ ((YDifferenceFromPrevious(selectevery(source, 1, 2)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1, 1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-1)) + 
    \ YDifferenceFromPrevious(selectevery(source, 1,-2)) ) / 4 ) / 
    \ (YDifferenceFromPrevious(source) + 0.01) <= JumpThresh 
    \ ? WhiteFrame : BlackFrame """)
  return MyMask
}

[Gavino]

Just a quick comment on the use of ScriptClip, for future reference.

Originally Posted by johnmeyer

...
global source=AVISource( "E:\CIF Track\3200-Complete (edited).avi" ).ConvertToYV12
...
#test=ShowMetrics(source)
...
BWMask=GenerateMask(source)
...
function ShowMetrics (clip c) 
{
  fixed=source.ScriptClip("Subtitle(String(
    \ ((YDifferenceFromPrevious(selectevery(source, 1, 2)) + 
  ...
}

function GenerateMask (clip c)
{
  MyMask=c.ScriptClip("""
    \ ((YDifferenceFromPrevious(selectevery(source, 1, 2)) + 
  ...
}

The input clip to ScriptClip is (initially) available as 'last' inside the run-time script, so you don't need to refer there to 'source' (which consequently need not be global).

function ShowMetrics (clip c) 
{
  fixed=c.ScriptClip("Subtitle(String(
    \ ((YDifferenceFromPrevious(selectevery(1, 2)) + 
  ... # etc
}

function GenerateMask (clip c)
{
  MyMask=c.ScriptClip("""
    \ ((YDifferenceFromPrevious(selectevery(1, 2)) + 
  ... # etc
}

[Robert Martens]

You can grab the stream with some minor detective work, if you want.

At that Fox Sports West page, view the source and search for the video ID ( _SeMs5xdJTNhA5dHUbpNK_WG9UvYWUtW ). The fourth match, under "// Select a release for playback", gives you the URL the player contacts to choose the appropriate stream. Visit that address and you'll be presented with a SMIL file named "content.select". Open it in a text editor, and you'll see the RTMP addresses for the three available versions of the video.

I'm not a streaming expert, so as far as getting through that URL to simply nab the MP4 from the server, I'm lost. I was, however, able to use RTMPDump to drop the output into an FLV file. Different container, same content (as far as I know). There are Windows binaries in the download directory; I grabbed version 2.3, at the bottom.

Keep in mind that at an hour and forty-eight minutes, even the low quality version of this video is over 520MB, so be prepared to wait a bit for the dump to finish.

[poisondeathray]

Thanks for the script

If you want an easy to use GUI to download that video, streamtransport appears to work (I didn't download the entire video, but it looks to be working fine)

[johnmeyer]

First of all, I am going to post (tonight or tomorrow) an update to the script I posted above. I feel pretty stupid about this, but my moving average includes the frames which are duplicates. These frames yield a Ydiff of exactly zero, and this invalidates the moving average. So, I wrote some tortured code (given how conditionals work) to use an adjacent clip's values whenever zero is detected. I'm testing this now, and I think it is going to completely eliminate the residual jumps.

Originally Posted by Gavino

Just a quick comment on the use of ScriptClip, for future reference ...

The input clip to ScriptClip is (initially) available as 'last' inside the run-time script, so you don't need to refer there to 'source' (which consequently need not be global).

I'll see if that works. I am sure you are correct, and this is how I originally wrote the code. However, as I modified things, some of my attempts involved using outside variables inside the conditional which, as you know, requires them to be declared as global.

I have some vague understanding that "global = bad", but I'm not quite sure what calamity awaits me if I do that.

Originally Posted by Robert Martens

You can grab the stream with some minor detective work, if you want ... I was, however, able to use RTMPDump to drop the output into an FLV file ...

I'll definitely look into this after dinner. While I have enjoyed the challenge of creating this script, the original objective (other than draining the swamp ...) was to give this guy the video of his kid, and make it look as good as possible. A better capture is obviously the right way to go.

Originally Posted by poisondeathray

If you want an easy to use GUI to download that video, streamtransport appears to work (I didn't download the entire video, but it looks to be working fine)

Ditto my last comment. I just Googled that name and it looks like a site similar to Keepvid. I didn't know about this one. I think I'll try this first.

Thanks to everyone!

[johnmeyer]

The updated script with the conditionals that block diff values = 0 from being included in the moving average is shown below.

Also, I tried the two methods for downloading the video. Streamtransport was straightforward, but it grabbed the low-res version of the video. I couldn't find any documentation as to how to specify the higher res version, nor was I able to find any menu or setting in the program.

The RTMPDump was a little more convoluted (reading source code, using the DOS command line, etc.), but ultimately gave me exactly what I wanted.

Finally, I figured out how to use AVISynth and VFAPIConv to serve the resulting FLV files back into Sony Vegas (which can't read FLV) without re-encoding.

So, even though the script posted below ultimately may not be used for this project, I know I'll be using it on various things that clients send to me.

Once again, many thanks for the help!

# Based on script created by Didée
# Modified by John Meyer on June 29, 2011
#
# Create interpolated frames a 2x original frame rate using MVTools2
# Detect jumps 
# Create white mask at each jump point; black mask for all other frames
# Repeat each frame of original video and use mask to "choose" between original video, or motion estimated video
# Decimate exactly 50% to get back to original frame rate. 
# This decimation removes the dup frames from the original video and also the dups created by repeating each frame of original video
# However, at each point where motion-estimated frame was inserted, no decimation occurs. Thus, if dups=drops, and the drop happens 
# within < "cycle" (TDecimate parameter) of the dup, the dup will be removed and the drop will be filled. 
# If no drops or dups occur within "cycle," then no motion estimation happens, and decimation merely gets back to original, 
# unchanged video.

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

#Threshold for detecting jumps. Increase to catch more jumps. Should always be less than 1.0
JumpThresh = 0.8
showdot = false # true for troubleshooting; otherwise, false

#SetMTMode(5,4)
global source=AVISource( "E:\CIF Track\3200-Complete (edited).avi" ).ConvertToYV12
#SetMTMode(2)

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )

super = showdot ? source.subtitle("***").MSuper(pel=2) : source.MSuper(pel=2) 
bvec  = MAnalyse(super, overlap=4, isb = true, search=4, dct=5) 
fvec  = MAnalyse(super, overlap=4, isb = false, search=4, dct=5) 
double = source.MFlowFps(super, bvec, fvec, num=60, den=1, blend=false)

#Remove comment from ShowMetrics, and change "return final" to "return test" to look at metrics in order to determine proper JumpThresh
#test=ShowMetrics(source)

#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)

interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate
RequestLinear
final=tdecimate(display=false,mode=1,cycleR=10,cycle=20)  # Decimate half of all frames (set to twice the length of "normal" dup/drop cycle)

#---------------------
#Alternate two-pass approach to decimation 

#Pass 1
#RequestLinear(debug=false)
#final=tdecimate(display=false,mode=4,output="e:\metrics.txt")

#Pass 2 (remember to un-comment "requestlinear")
#RequestLinear
#final=tdecimate(display=false,mode=2,rate=30,input="e:\metrics.txt",maxndl=20)
#---------------------

return final
#return stackvertical(source,final)

#----------------
#This function displays the YDiff value that will be used for detecting big jumps
#Each YDiff must eliminate Ydiff=0 (duplicate) from moving average

function ShowMetrics (clip c) 
{
  fixed=source.ScriptClip("Subtitle(String(
    \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.2 ? 
    \       YDifferenceFromPrevious(selectevery(source, 1, 3))  :
    \       YDifferenceFromPrevious(selectevery(source, 1, 2)) )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, 2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, 1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -3))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  )
    \     )/4) / 
    \    (YDifferenceFromPrevious(source) + 0.01)
    \ ))")
  return fixed
}



#----------------
#This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned
#Each YDiff must eliminate Ydiff=0 (duplicate) from moving average

function GenerateMask (clip c)
{
  MyMask=c.ScriptClip("""
    \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.2 ? 
    \       YDifferenceFromPrevious(selectevery(source, 1, 3))  :
    \       YDifferenceFromPrevious(selectevery(source, 1, 2)) )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, 2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, 1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -3))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  )
    \     )/4) / 
    \    (YDifferenceFromPrevious(source) + 0.01) <= JumpThresh 
    \ ? WhiteFrame : BlackFrame """)
  return MyMask
}

[Gavino]

Originally Posted by johnmeyer

However, as I modified things, some of my attempts involved using outside variables inside the conditional which, as you know, requires them to be declared as global.

Not if you use GRunT.
GRunT will also allow you to replace things like
YDifferenceFromPrevious(selectevery(source, 1, -2))
by YDifferenceFromPrevious(-2) and, as I said, 'source' is unnecessary even without GRunT.

I have some vague understanding that "global = bad", but I'm not quite sure what calamity awaits me if I do that.

It's not really a problem here, since your globals are effectively constants and your functions are specific to this particular script. But in more general functions (especially in conjunction with ScriptClip, etc), it can lead to bugs such as the function not working if called more than once. And generally speaking, it's easier to test and understand functions that act as 'black boxes' (the modularity principle).

[johnmeyer]

Originally Posted by Gavino

Not if you use GRunT.

I just quickly skimmed the GRunT and GScript threads, and downloaded both extensions. I see you did them several years ago. I wish I'd seen them sooner because they are going to make my life much easier.

Thanks!

[bizz & buzz]

I'm trying to use this script but it generates a wired "shadow" frame on top of most of the frames.This shadow frame is actually several frames apart from the base frame. There is also some crazy jumping back and forth whenever there's a scene change

source: http://www.mediafire.com/?hmjzf86w1gfd0m6

processed: http://www.mediafire.com/?dfo56cueeix83xk

Script is exactly as in post #6.

Thanks for any help

Pic:

[johnmeyer]

The script won't work with your video because most, if not all, of your duplicates occur exactly at the beginning of each scene change. The script replaces the first instance of the duplicated frames with a motion estimated version from the previous frame, which is from another scene. The problem isn't so much scene detection as it is the fact that the script fails when the duplicate is at the beginning of a scene change. This could be change by estimating the second instance of the dup from the following frame. Of course that approach will fail when the duplicate happens exactly at the end of a scene.

Also, you have to change the Mflowfps parameter to num=48000, den=1001 because your source is 23.976. The Tdecimate CycleR and Cycle parameters will probably have to be changed.

The script worked perfectly on my source clip and was meant as a starting point for others to solve similar problems. However, I never generalized it to all frame rates, and your duplicates exactly at the beginning of each scene is a case that the script as written will not handle.

[bizz & buzz]

OK... I'll try to modify the parameters according to your suggestions and also to find a way to make the script behave differently according to whether the dup is in the end or the beggining of a scence.
Thanks a lot.