Need help "perfecting" script to delete drops and dups

[johnmeyer]

I have been trying for a decade to perfect a script which can reliably and automatically detect and remove dropped frames, and also automatically fix the duplicates which usually follow the drops. This drop/dup pattern happens at some point in almost every streaming video, and is also often seen in screen captures. It is an incredibly common problem.

I need help with three things:

1. TDecimate is not reliable in this application (this is the main problem)
2. A minor problem: mt_merge does not provide identical video to the original when merged with pure white. The script works despite this flaw, but I'd like to understand how to fix it.
3. I cannot multi-thread the script because of TDecimate

Any help would be appreciated. At this point, I have given up. The script is still useful as written, but it will fail to fix quite a few drops and dups, all because of TDecimate's failures.

However, it is tantalizingly close to being an incredibly useful script.

--------------------------
For those interested in helping, here is the background, followed by my current script.

I first posted my script in 2011: Automatically fix dups followed (eventually) by drops , and then again a few weeks ago: Dropped frames

When I answered the OP a few weeks ago I thought initially that the main problem was that I wasn't detecting drops very well, but after posting about this (Script for reliably detecting dropped frames) I found out the my technique is actually quite good. There is no need to revisit my detection scheme.

The REAL problem is TDecimate. If a frame drop happen on a Cycle boundary (where "Cycle" is the group of frames TDecimate works on at one time), it fails. The reason for this is that Didée's original brilliant idea requires that the drop and the dup happen within a TDecimate "Cycle."

Perhaps I need to come up with a different approach, or perhaps there is a way to tweak Didée's idea.

The script is also very slow, but does not play nice with MT (also a TDecimate problem), so I can't use that technique to make it play faster. Because it is so slow, the idea of running TDecimate three times with different parameters and then "voting" on the best result is not going to work because the script speed will be measure in seconds per frame instead of frames per second.

BTW, I used my "really good" MVTools2 technique for creating the intermediate frames, so when a replacement is made it is usually undetectable. That is why there are so many lines of code preceding the MFlowFPS call.

I also tried SVP, and it does work and it is faster, even without MT, so I might go with that if and when I figure out a replacement or a fix for TDecimate. I have left that commented out code in the script.

Things I have tried

I have tried FDecimate and it doesn't work. I have tried TDecimate using its 2-pass approach which I thought would work for sure, but it does not. I have also tried TDecimate's mode 0, mode 1, mode 2, and mode 7.

I am thinking about playing around with Requestlinear settings, and also am wondering if I can use the metrics from FrameDiff to create my own decimation.

I am running AVISynth+ 3.7.0 and am using pinterf's 2/22/2021 compile of TIVTC (which includes TDecimate).

The comments at the beginning of the script describe how it works. If anyone has any ideas about how to fix it, let me know. The script is still partially hard-wired for 59.94 fps, but I will clean that up and turn it into an AVSI if I ever get the problems solved.

Here are links to two really tiny test videos. The first is artificial video (just a title scrolling horizontally) and the other is real video with multiple short scenes chosen for their different types of motion. I used my NLE to create a drop, followed by a dup within 1 to 12 frames later. The real video link also includes a text file which gives you the frame numbers where I created a drop, and also the frame numbers for the dups I inserted.

"Trivial" manufactured video test
https://www.mediafire.com/folder/yjf...d3d4mlo/shared

Real Video Test
https://www.mediafire.com/file/yjffj..._Test.zip/file

Thanks in advance to anyone who has read this far and might have some ideas.

# Based on script created by Didée
# Modified by John Meyer on June 29, 2011
# Further modification on June 8, 2013
# Tdecimate modifications on October 26, 2021
#
# 1. Create interpolated frames a 2x original frame rate using MVTools2.
# 2. Detect jumps in the original 1x video.
# 3. Create white mask at each jump point; black mask for all other frames.
# 4. Repeat each frame of original video.
# 5. For each frame in double-frame video, use a mask to "choose" between that original video, or the motion estimated video
# 6. Decimate exactly 50% to get back to original frame rate. 
#
# This decimation removes the dup frames that existed in the original video and also removes the dups created by 
# repeating each frame of original video.
# However, at each point where motion-estimated frame was inserted, no decimation occurs. 
# Thus, if dups=drops, all dups in the original video will be removed and the drop will be filled. 
# On the other hand, if no drops or dups occur, then no motion estimation appears in the final result, and the decimation 
# merely gets back to original, unchanged video.
#*********************************************

#*********************************************
#User-settable paramaters

#Puts dot on screen during debugging. True for troubleshooting; otherwise, false.
showdot = true 

#Threshold for detecting jumps. Increase to catch more jumps. Should always be less than 1.0
JumpThresh = 0.8

#Frame rate of your video 
#PAL: Vidnum=25; VidDen=1;   NTSC: Vidnum=30000; VidDen=1001;  etc.
VidNum = 60000
VidDen = 1001

#Detection window size. Try changing if you find residual dups or drops. Script gets slow if you use big numbers
DupCycle = 18

threads = 1  #TDecimate does not like multithreading
#*********************************************

loadplugin("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\MVTools2 (latest)\mvtools2.dll")
Loadplugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools-26.dll") 

#These were used when trying to get AVISynth+ multi-threading to work
#SetFilterMTMode("TDecimate", MT_SERIALIZED)   
#SetFilterMTMode("TDecimate", MT_MULTI_INSTANCE)

global source=AVISource("E:\fs.avi").ConvertToYV12.killaudio()

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )

prefiltered = RemoveGrain(source,22)
super = MSuper(source,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=4,search=3,dct=0)
forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=4,search=3,dct=0)
forward_re = MRecalculate(super, forward, blksize=8, overlap=2, thSAD=100)
backward_re = MRecalculate(super, backward, blksize=8, overlap=2, thSAD=100)

#This is alternative way to double the number of frames. 
#double = SmoothFPS2 (source,threads)

double = source.MFlowFps(super, backward_re, forward_re, num=VidNum*2, den=VidDen, blend=false)
double = showdot ? double.subtitle("***") : double

test=ShowMetrics(source)

#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)

# *** NOTE *** mt_merge does NOT produce a clean merge, and the dups are slightly altered from the originals -- unable to fix

interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate half of all frames 
RequestLinear
tdecimate(display=false,debug=false,mode=1,Cycle=DupCycle*2,CycleR=DupCycle,vfrdec=1,sdlim=0)  

#Alternate decimation -- should work, but doesn't
#tdecimate(display=false,debug=false,mode=2,rate=59.94006,maxndl=19,m2PA=true)  

#Alternate ways to end the script. Must assign variable to tdecimate output instead of using implied "last."
#prefetch(threads)	
#return test
#Stacked view, for debugging
#return stackvertical(source,decimated)

#End Main Program

#----------------
#Begin Functions
#----------------

#This function displays the YDiff value that will be used for detecting big jumps
#Each YDiff must eliminate Ydiff~=0 (i.e., a duplicate) from moving average
#This is why there are three lines for each YDiff. Each block uses the next or preceeding
#frame, if the current YDiff is near zero, indicating a dup.
#Note: This function works just fine

function ShowMetrics (clip c) 
{
  fixed=source.ScriptClip("Subtitle(String(
    \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.3 ? 
    \       YDifferenceFromPrevious(selectevery(source, 1, 3))  :
    \       YDifferenceFromPrevious(selectevery(source, 1, 2)) )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.3 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, 2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, 1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.3 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.3 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -3))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  )
    \     )/4) / 
    \    (YDifferenceFromPrevious(source) + 0.01)
    \ ))")
  return fixed
}


#----------------
#This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned
#Each YDiff must eliminate Ydiff=0 (duplicate) from moving average.
#Since the first two frames of the clip have no preceding frame, don't return a mask
#for those two frames (which means there will be no fix for the first two frames)

#This function works just fine (i.e., really great detection)

function GenerateMask (clip c)
{
  MyMask=c.ScriptClip("""
    \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.3 ? 
    \       YDifferenceFromPrevious(selectevery(source, 1, 3))  :
    \       YDifferenceFromPrevious(selectevery(source, 1, 2)) )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.3 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, 2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, 1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.3 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.3 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -3))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  )
    \     )/4) / 
    \    (YDifferenceFromPrevious(source) + 0.01) <= JumpThresh 
    \  && current_frame > 2
    \ ? WhiteFrame : BlackFrame """)
  return MyMask
}

#This function is hard-wired for 60 fps. Not worth the time to generalize it, since SVP
#didn't seem to provide a huge benefit over MVTools2, although it should be faster.

function SmoothFPS2(clip source, threads) { 
  super_params="{pel:2,gpu:1}"
  analyse_params="""{
        block:{w:16,h:16}, 
	main:{search:{coarse:{distance:0}}},
	refine:[{thsad:200}]
        }"""
  smoothfps_params="{rate:{num:120,den:60,abs:false},scene:{mode:0,limits:{scene:8500}},algo:21,cubic:1}"

  super =   SVSuper(source,super_params)
  vectors = SVAnalyse(super, analyse_params)
  SVSmoothFps(source,super,  vectors,  smoothfps_params,  url="www.svp-team.com",  mt=threads)
}