Need help "perfecting" script to delete drops and dups

[johnmeyer]

I have been trying for a decade to perfect a script which can reliably and automatically detect and remove dropped frames, and also automatically fix the duplicates which usually follow the drops. This drop/dup pattern happens at some point in almost every streaming video, and is also often seen in screen captures. It is an incredibly common problem.

I need help with three things:

1. TDecimate is not reliable in this application (this is the main problem)
2. A minor problem: mt_merge does not provide identical video to the original when merged with pure white. The script works despite this flaw, but I'd like to understand how to fix it.
3. I cannot multi-thread the script because of TDecimate

Any help would be appreciated. At this point, I have given up. The script is still useful as written, but it will fail to fix quite a few drops and dups, all because of TDecimate's failures.

However, it is tantalizingly close to being an incredibly useful script.

--------------------------
For those interested in helping, here is the background, followed by my current script.

I first posted my script in 2011: Automatically fix dups followed (eventually) by drops (http://forum.doom9.org/showthread.php?p=1510813#post1510813), and then again a few weeks ago: Dropped frames (https://forum.doom9.org/showthread.php?p=1954728#post1954728)

When I answered the OP a few weeks ago I thought initially that the main problem was that I wasn't detecting drops very well, but after posting about this (Script for reliably detecting dropped frames (https://forum.doom9.org/showthread.php?t=183313)) I found out the my technique is actually quite good. There is no need to revisit my detection scheme.

The REAL problem is TDecimate. If a frame drop happen on a Cycle boundary (where "Cycle" is the group of frames TDecimate works on at one time), it fails. The reason for this is that Didée's original brilliant idea requires that the drop and the dup happen within a TDecimate "Cycle."

Perhaps I need to come up with a different approach, or perhaps there is a way to tweak Didée's idea.

The script is also very slow, but does not play nice with MT (also a TDecimate problem), so I can't use that technique to make it play faster. Because it is so slow, the idea of running TDecimate three times with different parameters and then "voting" on the best result is not going to work because the script speed will be measure in seconds per frame instead of frames per second.

BTW, I used my "really good" MVTools2 technique for creating the intermediate frames, so when a replacement is made it is usually undetectable. That is why there are so many lines of code preceding the MFlowFPS call.

I also tried SVP, and it does work and it is faster, even without MT, so I might go with that if and when I figure out a replacement or a fix for TDecimate. I have left that commented out code in the script.

Things I have tried

I have tried FDecimate and it doesn't work. I have tried TDecimate using its 2-pass approach which I thought would work for sure, but it does not. I have also tried TDecimate's mode 0, mode 1, mode 2, and mode 7.

I am thinking about playing around with Requestlinear settings, and also am wondering if I can use the metrics from FrameDiff to create my own decimation.

I am running AVISynth+ 3.7.0 and am using pinterf's 2/22/2021 compile of TIVTC (which includes TDecimate).

The comments at the beginning of the script describe how it works. If anyone has any ideas about how to fix it, let me know. The script is still partially hard-wired for 59.94 fps, but I will clean that up and turn it into an AVSI if I ever get the problems solved.

Here are links to two really tiny test videos. The first is artificial video (just a title scrolling horizontally) and the other is real video with multiple short scenes chosen for their different types of motion. I used my NLE to create a drop, followed by a dup within 1 to 12 frames later. The real video link also includes a text file which gives you the frame numbers where I created a drop, and also the frame numbers for the dups I inserted.

"Trivial" manufactured video test
https://www.mediafire.com/folder/yjffjz3ydgxhvxj,cof5ytpid3d4mlo/shared

Real Video Test
https://www.mediafire.com/file/yjffjz3ydgxhvxj/Dropped_Frame_Test.zip/file

Thanks in advance to anyone who has read this far and might have some ideas.

# Based on script created by Didée
# Modified by John Meyer on June 29, 2011
# Further modification on June 8, 2013
# Tdecimate modifications on October 26, 2021
#
# 1. Create interpolated frames a 2x original frame rate using MVTools2.
# 2. Detect jumps in the original 1x video.
# 3. Create white mask at each jump point; black mask for all other frames.
# 4. Repeat each frame of original video.
# 5. For each frame in double-frame video, use a mask to "choose" between that original video, or the motion estimated video
# 6. Decimate exactly 50% to get back to original frame rate.
#
# This decimation removes the dup frames that existed in the original video and also removes the dups created by
# repeating each frame of original video.
# However, at each point where motion-estimated frame was inserted, no decimation occurs.
# Thus, if dups=drops, all dups in the original video will be removed and the drop will be filled.
# On the other hand, if no drops or dups occur, then no motion estimation appears in the final result, and the decimation
# merely gets back to original, unchanged video.
#*********************************************

#*********************************************
#User-settable paramaters

#Puts dot on screen during debugging. True for troubleshooting; otherwise, false.
showdot = true

#Threshold for detecting jumps. Increase to catch more jumps. Should always be less than 1.0
JumpThresh = 0.8

#Frame rate of your video
#PAL: Vidnum=25; VidDen=1; NTSC: Vidnum=30000; VidDen=1001; etc.
VidNum = 60000
VidDen = 1001

#Detection window size. Try changing if you find residual dups or drops. Script gets slow if you use big numbers
DupCycle = 18

threads = 1 #TDecimate does not like multithreading
#*********************************************

loadplugin("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\MVTools2 (latest)\mvtools2.dll")
Loadplugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools-26.dll")

#These were used when trying to get AVISynth+ multi-threading to work
#SetFilterMTMode("TDecimate", MT_SERIALIZED)
#SetFilterMTMode("TDecimate", MT_MULTI_INSTANCE)

global source=AVISource("E:\fs.avi").ConvertToYV12.killaudio()

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )

prefiltered = RemoveGrain(source,22)
super = MSuper(source,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=4,search=3,dct=0)
forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=4,search=3,dct=0)
forward_re = MRecalculate(super, forward, blksize=8, overlap=2, thSAD=100)
backward_re = MRecalculate(super, backward, blksize=8, overlap=2, thSAD=100)

#This is alternative way to double the number of frames.
#double = SmoothFPS2 (source,threads)

double = source.MFlowFps(super, backward_re, forward_re, num=VidNum*2, den=VidDen, blend=false)
double = showdot ? double.subtitle("***") : double

test=ShowMetrics(source)

#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)

# *** NOTE *** mt_merge does NOT produce a clean merge, and the dups are slightly altered from the originals -- unable to fix

interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate half of all frames
RequestLinear
tdecimate(display=false,debug=false,mode=1,Cycle=DupCycle*2,CycleR=DupCycle,vfrdec=1,sdlim=0)

#Alternate decimation -- should work, but doesn't
#tdecimate(display=false,debug=false,mode=2,rate=59.94006,maxndl=19,m2PA=true)

#Alternate ways to end the script. Must assign variable to tdecimate output instead of using implied "last."
#prefetch(threads)
#return test
#Stacked view, for debugging
#return stackvertical(source,decimated)

#End Main Program

#----------------
#Begin Functions
#----------------

#This function displays the YDiff value that will be used for detecting big jumps
#Each YDiff must eliminate Ydiff~=0 (i.e., a duplicate) from moving average
#This is why there are three lines for each YDiff. Each block uses the next or preceeding
#frame, if the current YDiff is near zero, indicating a dup.
#Note: This function works just fine

function ShowMetrics (clip c)
{
fixed=source.ScriptClip("Subtitle(String(
\ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, 3)) :
\ YDifferenceFromPrevious(selectevery(source, 1, 2)) )
\ +
\ (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, 2)) :
\ YDifferenceFromPrevious(selectevery(source, 1, 1)) )
\ +
\ (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, -2)) :
\ YDifferenceFromPrevious(selectevery(source, 1, -1)) )
\ +
\ (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, -3)) :
\ YDifferenceFromPrevious(selectevery(source, 1, -2)) )
\ )/4) /
\ (YDifferenceFromPrevious(source) + 0.01)
\ ))")
return fixed
}


#----------------
#This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned
#Each YDiff must eliminate Ydiff=0 (duplicate) from moving average.
#Since the first two frames of the clip have no preceding frame, don't return a mask
#for those two frames (which means there will be no fix for the first two frames)

#This function works just fine (i.e., really great detection)

function GenerateMask (clip c)
{
MyMask=c.ScriptClip("""
\ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, 3)) :
\ YDifferenceFromPrevious(selectevery(source, 1, 2)) )
\ +
\ (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, 2)) :
\ YDifferenceFromPrevious(selectevery(source, 1, 1)) )
\ +
\ (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, -2)) :
\ YDifferenceFromPrevious(selectevery(source, 1, -1)) )
\ +
\ (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.3 ?
\ YDifferenceFromPrevious(selectevery(source, 1, -3)) :
\ YDifferenceFromPrevious(selectevery(source, 1, -2)) )
\ )/4) /
\ (YDifferenceFromPrevious(source) + 0.01) <= JumpThresh
\ && current_frame > 2
\ ? WhiteFrame : BlackFrame """)
return MyMask
}

#This function is hard-wired for 60 fps. Not worth the time to generalize it, since SVP
#didn't seem to provide a huge benefit over MVTools2, although it should be faster.

function SmoothFPS2(clip source, threads) {
super_params="{pel:2,gpu:1}"
analyse_params="""{
block:{w:16,h:16},
main:{search:{coarse:{distance:0}}},
refine:[{thsad:200}]
}"""
smoothfps_params="{rate:{num:120,den:60,abs:false},scene:{mode:0,limits:{scene:8500}},algo:21,cubic:1}"

super = SVSuper(source,super_params)
vectors = SVAnalyse(super, analyse_params)
SVSmoothFps(source,super, vectors, smoothfps_params, url="www.svp-team.com", mt=threads)
}

[poisondeathray]

2. A minor problem: mt_merge does not provide identical video to the original when merged with pure white. The script works despite this flaw, but I'd like to understand how to fix it.


It looks like "video" white is used, ie., Y=235; not Y=255
Same for "black" . Y=16, not Y=255


global source=AVISource("E:\fs.avi").ConvertToYV12.killaudio()

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )


You can use expr("255", "128", "128") to make it full white in 8bit YUV or coloryuv(preset="TV->PC") , not sure which is faster, probably expr .

[johnmeyer]

Thanks for that. I started to try out your suggestion by adding a levels statement (e.g., levels(255,1,255,255,255)) when all of a sudden the script wouldn't work. After ten wasted minutes I decided to re-boot (Win XP 32-bit). The script started working again (with the problems reported above, of course).

This means I am chasing memory leak problems as well.

I like AVISynth and I appreciate open source, but it sure can be fragile at times.

Oh yes, the levels(255,1,255,255,255) and levels(0,1,0,0,0) statements did not fix the problem.

global BlackFrame = BlankClip( source, Color=$000000 ).levels(0,1,0,0,0)
global WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(255,1,255,255,255)

TDecimate, with display=true still had non-zero metrics for the dups created by theinterleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3) statement. If I change that to just interleave(source,source), without the mt_merge, TDecimate shows 0.00 for the dups, as it should.

If I end up having to create my own decimation workflow outside of AVISynth, exporting metrics and then importing a decimation override file, it will be much more important to have these metrics be zero.

[videoh]

The reason for this is that Didée's original brilliant idea requires that the drop and the dup happen within a TDecimate "Cycle." Is there a link to a description of Didée's idea?

[poisondeathray]

Oh yes, the levels(255,1,255,255,255) and levels(0,1,0,0,0) statements did not fix the problem.

global BlackFrame = BlankClip( source, Color=$000000 ).levels(0,1,0,0,0)
global WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(255,1,255,255,255)



For the mask part, if you wanted to use levels, it would be this for 8bit YUV


BlackFrame = BlankClip( source, Color=$000000 ).levels(16,1,235,0,255,false)
WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(16,1,235,16,235,false)

[johnmeyer]

Is there a link to a description of Didée's idea?Yes.

https://forum.doom9.org/showthread.php?p=1409324#post1409324

I completely changed the detection logic not only because his wasn't very good, but also because there was no way to display the detection metrics. If you can't display them, you can't tune the script.

[videoh]

Thank you.

[johnmeyer]

For the mask part, if you wanted to use levels, it would be this for 8bit YUV


BlackFrame = BlankClip( source, Color=$000000 ).levels(16,1,235,0,255,false)
WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(16,1,235,16,235,false)
I've shut down for the day, but I'll try that tomorrow. I obviously took your advice the wrong way around. (255 instead of 235, etc.)

[StainlessS]

Yes.

https://forum.doom9.org/showthread.php?p=1409324#post1409324

I completely changed the detection logic not only because his wasn't very good, but also because there was no way to display the detection metrics. If you can't display them, you can't tune the script.

I was playing with this (above Didee link mod) a few weeks back (I think I left it in working order, but not sure),


# DeDecimateNtsc24BackTo30.avsi

Function DeDecimateNtsc24BackTo30(clip c,Bool "ShowDot") {
/*
DeDecimateNtsc24BackTo30.avsi : Mod to Script function by StainlessS. Original script by Didee @ https://forum.doom9.org/showthread.php?p=1409324#post1409324
required, AVS v2.60/+, Pinterf mt_masktools, Pinterf mvtools2, Pinterf tivtc.

Source should be NTSC_FILM(23.976p) that has been decimated from original PROGRESSIVE_NTSC_STANDARD(29.97p).
*/
c myName="DeDecimateNtsc24BackTo30: "
ShowDot = Default(ShowDot,False) # shows a "dot" on the interpolated frames of the result
o = assumefps(1.0) # Avoid frame doubling rounding issues, due to weird NTSC framerates.
ox = o.width() oy=o.height()
Assert(!c.IsRGB && c.IsPlanar,myname+"Planar YUV only")
try { bpc=c.BitsPerComponent } catch(msg) { bpc=8 }
Assert(8 <= bpc <= 16,myname+"8 to 16 bit only")

super = (showdot ? o.subtitle(".") : o).MSuper(pel=2,hpad=16,vpad=16)
bvec = MAnalyse(super, overlap=4, isb = true, search=4, dct=5)
fvec = MAnalyse(super, overlap=4, isb = false, search=4, dct=5)
double = o.MFlowFps(super, bvec, fvec, num=2, den=1, blend=false) # Result will be 2.0 FPS, and FrameCount=(c.FrameCount*2)-1. Dont Blend at SceneChange.

shftn = BitLShift(1, bpc-8)
cs256 = BitLShift(256,shftn)
cs255 = cs256-1
cs128 = BitLShift(128,shftn)
cs32 = BitLShift(32,shftn)

d2n_yexpr="x " + String(cs128) + " - abs " + String(cs32) + " / 1 2.0 / ^ " + String(cs128) + " *" # [8-Bit]:- rpn="x 128 - abs 32 / 1 2.0 / ^ 128 *" : Infix="( (abs(x-128)/32) ^ (1/2.0) ) * 128"

diff2next = mt_makediff(o,o.selectevery(1,1)).mt_lut(d2n_yexpr,U=-(cs128),V=-(cs128))
diff2next = mt_lutf(diff2next,diff2next,yexpr="x",mode="average").pointresize(32,32)
diff2next = interleave(diff2next.selectevery(4,0).tsg(2),diff2next.selectevery(4,1).tsg(2),
\ diff2next.selectevery(4,2).tsg(2),diff2next.selectevery(4,3).tsg(2))


maximum = diff2next.mt_logic(diff2next.selectevery(1,-3),"max")
\ .mt_logic(diff2next.selectevery(1,-2),"max")
\ .mt_logic(diff2next.selectevery(1,-1),"max")
\ .mt_logic(diff2next.selectevery(1, 1),"max")
\ .mt_logic(diff2next.selectevery(1, 2),"max")
\ .mt_logic(diff2next.selectevery(1, 3),"max")

ismax_yexpr = "x y < 0 " + String(cs255) + " ?" # [8-Bit]:- rpn = "x y < 0 255 ?" : Infix = "(x < y) ? 0 : 255"

ismax = mt_lutxy(diff2next,maximum,ismax_yexpr,U=-(cs128),V=-(cs128)).pointresize(ox,oy)
themask = interleave(o.mt_lut("0"),ismax)
interleave(o,o).mt_merge(double,themask,luma=true,U=3,V=3) # Process U and V

tdecimate(mode=1,cycleR=3,cycle=8) # Decimate 3 out of 8 = 18 out of 48 (leaving 24 from 48)
assumefps(30000,1001) # Back to NTSC_Video
return last
}

#===========================

# tsg by Didee @ https://forum.doom9.org/showthread.php?p=1409324#post1409324
# ssS: Looks like some kind of limited Gausian Temporal Soften.
# TemporalSoften (clip, int radius, int luma_threshold, int chroma_threshold, int "scenechange", int "mode")
# Radius=1=Adjacent_frames, Luma_threshold=255=no limit, chroma_threshold=0=dont process, scenechange=255=ignore, mode=2=ISSE
function tsg(clip c, int t) { c
t<5?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<4?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<3?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<2?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<1?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25) }

EDITED: [there were two instances where I had not changed csLev128 to cs128]

EDIT: 8 -> 16 bit depth.

[VoodooFX]

I had problems too with TDecimate. To avoid drops I do real-fps *3 video, and TDecimate just sometimes drops non-duplicate frame when duplicate is just next one.

[StainlessS]

There is also this Doggie-Doo mod of Didee / JohnM script[FillMissing]:- https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/FillMissing.avsi

As well as JKyle mod [with wrong black/white frame colors]:- https://forum.doom9.org/showthread.php?p=1947624#post1947624

EDIT: Doggy uses Color_YUV=$000000, and Color_YUV=$FFFFFF which are also a bit wrong,
should be (for 8 bit ONLY)
Black = BlankClip( source, Color_yuv=$008080 )
White = BlankClip( source, Color_yuv=$FF8080)

[color_YUV AVS+ only]

Maybe use (where masktools 2 used already)
Black=source.mt_lut("0")
White=black.mt_Invert() # OK for HBD, eg 16 bit Y becomes $FFFF, not $FF00 (FULL WHITE required for Mt_merge mask).

[poisondeathray]

Black = BlankClip( source, Color_yuv=$008080 )
White = BlankClip( source, Color_yuv=$FF8080)


This is the fastest, least overhead method to generate a black or white YUV "blankclip" with the "source" characteristics - compared to expr, levels, coloryuv, mt_lut

[StainlessS]

May be for 8-bit only, but did you see this,
Maybe use (where masktools 2 used already)
Black=source.mt_lut("0")
White=black.mt_Invert() # OK for HBD, eg 16 bit Y becomes $FFFF, not $FF00 (FULL WHITE required for Mt_merge mask).

And its only really done the once, not at every frame so speed hardly matters at all.

[Dogway]

If your mask is $FFFFFF you can set luma=false which might be faster

[StainlessS]

Cheers Doggggy, did not know what that one was for.

EDIT: Luma=false = default.
mt_merge
bool luma = false

luma is a special mode, where only the the first plane (luma or red plane) of the mask is used to process all three channels.
Note: when luma=true, it forces chroma processing (chroma="process" or u=3, v=3).

[StainlessS]

Dog,
this defo dont look right to me,

#This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned
#Each YDiff must eliminate Ydiff=0 (duplicate) from moving average
BWMask = ScriptClip("
YDif1 = YDifferenceFromPrevious(src8,2)
YDif1 = YDif1 < 0.2 ? YDifferenceFromPrevious(src8,3) : YDif1

YDif2 = YDifferenceFromPrevious(src8,1)
YDif2 = YDif2 < 0.2 ? YDifferenceFromPrevious(src8,2) : YDif2

YDif3 = YDifferenceFromPrevious(src8,-1)
YDif3 = YDif3 < 0.2 ? YDifferenceFromPrevious(src8,-2) : YDif3

YDif4 = YDifferenceFromPrevious(src8,-2)
YDif4 = YDif4 < 0.2 ? YDifferenceFromPrevious(src8,-3) : YDif4

YDiff = ((YDif1 + YDif2 + YDif3 + YDif4) / 4.0 ) / (YDifferenceFromPrevious(src8,) + 0.01) <= JumpThresh ? \
source.White : \
source.Black \
",args="src8,JumpThresh,Black,White")

I'm gonna steal some of that, but with some changes.
what is source.White and Source.Black, Source not visible inside scriptclip, and source.White dont make a lot of sense (to me).
What is assign to YDiff supposed to do, and why do you just return Last, ie src8 of input.

[Dogway]

I noticed it. Fixed already!!

[StainlessS]

Good dog, now that is worth stealing.

[zorr]

2. A minor problem: mt_merge does not provide identical video to the original when merged with pure white. The script works despite this flaw, but I'd like to understand how to fix it.

You need Pinterf's version of MVTools, v2.2.7 or later.

**v2.2.7 (20170421)
- fix: mt_edge 10,12,14 bits: clamp mask value from 65535 to 1023 (10 bits), 4095 (12 bits) and 16383 (14 bits)
- fix: mt_merge 10-16 bits + non mod-16 width + luma=true + 4:2:2 colorspace, correct right side pixels
- fix: mt_merge 8 bit clips: keep original pixels from clip1/2 when mask is exactly 0 or 255

[poisondeathray]

May be for 8-bit only, but did you see this,


Maybe use (where masktools 2 used already)
Black=source.mt_lut("0")
White=black.mt_Invert() # OK for HBD, eg 16 bit Y becomes $FFFF, not $FF00 (FULL WHITE required for Mt_merge mask).


And its only really done the once, not at every frame so speed hardly matters at all.

For different bit depths blackclip(souce).expr should be faster than source.mt_lut; both will "flexible" in terms of bitdepth and correct on inverting
expr("0", "x", "x") for copy u,v and expr("0", "range_half", "range_half") for half uv at source bitdepth



source=("random 1080p source")

black = source.mt_lut("0").killaudio()
#black = blankclip(source).expr("0", "range_half", "range_half"").killaudio()
#black = blankclip(source).expr("0", "x", "x").killaudio()
#black = blankclip(source).mt_lut("0").killaudio()

black


8bit source benchmarks , fps, cpu%, no prefetch

(I believe expr is optimized with avx2 instructions, but it didn't make much of a difference on these benchmarks with SetMaxCPU("sse4.1") for this operation )

ffmpeg -i mt_lut.avs -f null NUL
285fps, 60%

ffmpeg -i blankclip_expr_uv_range_half.avs -f null NUL
1060fps, 25%

ffmpeg -i blankclip_expr_uv_copy.avs -f null NUL
1010fps, 25%

ffmpeg -i blankclip_mt_lut.avs -f null NUL
490fps, 22%

Eitherway, you're right it should have limited impact speedwise on John's specific script (it's not going to be the part that is the "bottleneck")

Do you need to copy the chroma channels for the mask ? Really you only need "Y" plane 100% white or 100% black. It might be slightly faster to do it in Y or Gray for the mask part, instead of YUV (discard instead of dragging the chroma channels along)

[johnmeyer]

You need Pinterf's version of MVTools, v2.2.7 or later.I'm definitely using his compile, but I'll have to wait until I'm back on the office computer to check the version number. I'm pretty sure I updated it to something fairly recent.

However, I still think the problem with the merge with white not producing an identical frame lies with the mt_merge plugin, not with MVTools2.

I've been tied up with other things today, but during driving time I thought about this script and am slowly coming to the conclusion that as good as Didée's idea is, it may actually be at the heart of the problem.

The issue is that you have to keep the video the same length, which his idea brilliantly handles. Put another way, for each drop you fill, you have to find a duplicate (or near duplicate) to delete.

Dups removed must equal new frames added at drop points.

I have really good detection logic (he said modestly) that finds both drops and dups. The trick of the whole thing is to have an approach which 100% guarantees that the number of drops filled exactly equals the number of dups removed. If the drops and dups are sparse, and if the Cycle size is large enough, this Didée'-inspired script is going to work most of the time, but if you have lots of drops and if the Cycle size is small, it will fail a lot.

So, I am contemplating rethinking the whole idea and coming up with something that eliminates TDecimate from the script. Unfortunately, this is likely to lead to a convoluted workflow, where I export my dup and drop metrics, dump them into a spreadsheet, and then use spreadsheet formulas and VBA to come up with a list of dups and drops which I'll feed back into an AVISynth script. Unfortunately, I don't know if there is any plugin which will take such a list because I don't think you can just arbitrarily add a few frames here or subtract a few frames somewhere else. That isn't how AVISynth works.

Anyway, I'm going to keep trying for a few more days. There needs to be a solution to this problem.

[StainlessS]

For different bit depths blackclip(souce).expr should be faster than source.mt_lut; both will "flexible" in terms of bitdepth and correct on inverting
expr("0", "x", "x") for copy u,v and expr("0", "range_half", "range_half") for half uv at source bitdepth


I'm kinda reluctant to use Expr() for anything in my scripts directory, If I need to test something under 2.60 std, then I gotta remove all of those scripts so as not to cause multiple problems.

[stephen22]

mt_merge does not provide identical video to the original when merged with pure white.

Never was quite sure why you need to use a mask at all. Why can't you just get your drop seeking routine to choose between Source and a (single) MFlowInter interpolation clip? Interleave with Source and do your TDecimate.

[StainlessS]

However, I still think the problem with the merge with white not producing an identical frame lies with the mt_merge plugin, not with MVTools2.
mt_merge does not provide identical video to the original when merged with pure white.
The problem is that it aint PURE White, rgb $FFFFFF arrives at Y = 235dec $EB hex, and RGB black $000000 at 16 dec, $10 hex,
so you dont get all of it when you want it all, and do get some of it when you want none.
Nothing wrong with mt_merge, problem is in script [as pointed out (I think) by several people in several threads].

[zorr]

However, I still think the problem with the merge with white not producing an identical frame lies with the mt_merge plugin, not with MVTools2.

Sorry, meant to say MaskTools, not MVTools. Pinterf makes too many plugins. :)

[johnmeyer]

Never was quite sure why you need to use a mask at all. Why can't you just get your drop seeking routine to choose between Source and a (single) MFlowInter interpolation clip? Interleave with Source and do your TDecimate.I have had the same question a few times as I've been working on this. Didée's code from the "Inverse of Decimate" thread used a mask to choose the clip. If I'd coded this from scratch I would have used ConditionalFilter, or something similar. Perhaps I will make that change now because it would make it simpler to get to the bottom of my problems if the exact dups created by doubling the number of rames source video would cause TDecimate to return a dup metric of 0.00. This would make it simpler to do the right thing with the "almost" dups in the actual video that need to be decimated.

[StainlessS]

Pinterf makes too many plugins. :)
Yep, he just wants to be popular, very needy, sad really.

[johnmeyer]

The problem is that it aint PURE White, rgb $FFFFFF arrives at Y = 235dec $EB hex, and RGB black $000000 at 16 dec, $10 hex,
so you dont get all of it when you want it all, and do get some of it when you want none.
Nothing wrong with mt_merge, problem is in script [as pointed out (I think) by several people in several threads].Yes, poinsondeathray and others have pointed this out, but I still haven't been able to get the proper level. I posted earlier my attempt, using Levels(), to get the right result.

[StainlessS]

Levels(16,1.0,235,0,255,coring=false)

should do it I think. [Must have the coring thingy]

[johnmeyer]

For the mask part, if you wanted to use levels, it would be this for 8bit YUV


BlackFrame = BlankClip( source, Color=$000000 ).levels(16,1,235,0,255,false)
WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(16,1,235,16,235,false)
As I speculated in my earlier post, I had it backwards. This fixes the problem. Thanks! (Thanks to StainlessS also).

[stephen22]

Perhaps I will make that change now
Doesn't it just mean replacing MFlowfps with MFlowInter, Blackframe with Source, WhiteFrame with the MFlowInter clip, juggling the interleave and forgetting the merge_mt?

It seems to me there's no way to get round the cycle difficulty, and that the problem is intrinsic in the method, not down to TDecimate, which is just doing what it's paid for. Whatever method you use for removing the 50% of frames with the least change, it will have to be regional, otherwise (apart from the time taken!) sequences with lots of movement will inevitably steal frames from sequences with very little movement. Regions have boundaries and dupe/drop pairs will inevitably straddle those boundaries at times.

You'd have to devise a very sophisticated algorithm which would seek to identify the dupe nearest to a drop and link the two in some way, marking the dupe so it's no longer available - not too hard in a normal programming environment but I suspect way outside the constraints imposed by Avisynth.

Simplest perhaps to just repeat the whole procedure as necessary - lengthy perhaps, but getting rid of the mask must save a bit of time. It's a very fine script as it is.

[johnmeyer]

Doesn't it just mean replacing MFlowfps with MFlowInter, Blackframe with Source, WhiteFrame with the MFlowInter clip, juggling the interleave and forgetting the merge_mt?

It seems to me there's no way to get round the cycle difficulty, and that the problem is intrinsic in the method, not down to TDecimate, which is just doing what it's paid for. Whatever method you use for removing the 50% of frames with the least change, it will have to be regional, otherwise (apart from the time taken!) sequences with lots of movement will inevitably steal frames from sequences with very little movement. Regions have boundaries and dupe/drop pairs will inevitably straddle those boundaries at times.

You'd have to devise a very sophisticated algorithm which would seek to identify the dupe nearest to a drop and link the two in some way, marking the dupe so it's no longer available - not too hard in a normal programming environment but I suspect way outside the constraints imposed by Avisynth.

Simplest perhaps to just repeat the whole procedure as necessary - lengthy perhaps, but getting rid of the mask must save a bit of time. It's a very fine script as it is.You are MUCH smarter than I am because it took me a few years to realize what you just wrote. Your analysis is 100.0% correct, and very clearly written.

Which brings me to my current idea, but one which I can't quite figure out to do. Here it is.

My next approach will be to export the drop matching metrics from my detection function and also export dup matching metrics, either from TDecimate or from my own matching logic. I'll then put those metrics into an Excel spreadsheet. In that sheet I can use a combination of formulas and VBA (much easier than coding a plugin in C), to look backwards and forwards to match the nearest dup with its corresponding drop and use those matches to create a column which shows at which frames a motion-estimated frame must be inserted, and also which frames must be decimated because they are dups. The logic will ensure that dups=drops.

Then, in AVISynth, I'll create the intermediate frames by doubling the frame rate and then using SelectOdd() to choose only the intermediate frames. This will give me a stream from which I can insert frames to bridge the drop.

However, this is where I hit a dead end. I need to have some way in which I can insert frames arbitrarily and can decimate frames arbitrarily. I don't think AVISynth provides a mechanism for this even though the total number of frames will, in the end, be the same.

I can write a script in my NLE, Vegas Pro, that will take the original video along with the "in-between frames" stream (which is the same length as the original video, but shifted in time by half a frame) and perform the substitutions and decimation, but very few people reading this will be able to use such a thing because most people don't own Sony Vegas Pro.

I know that I can do arbitrary decimation using MultiDecimate because I do that all the time with my shutterless projector film transfer system that I invented fifteen years ago and which I have posted about several times in this forum. However, while it can easily do arbitrary decimation from a list of frame numbers, I don't think it permits me to arbitrarily insert frames.

So, bottom line, the approach I adopted from Didée years ago is fundamentally flawed, and I am now thinking that the way in which AVISynth works will make it impossible for me to do this task completely from within the AVISynth environment.

To do so will require me to

Figure out how to make the number of dups removed exactly equal the number of drops inserted;
Insert synthesized video frames at arbitrary locations in the video;
Delete a "duplicate" frame within some set distance from each drop;

To make this work, the drops must be the "controlling" metric, and they must drive the dup decimation.

I need to build up the desire to keep going on this. My head hurts right now from banging it against the wall. I may re-visit the idea Zorr had in your Dropped frames (http://forum.doom9.org/showthread.php?t=183300) thread where I run TDecimate three times with Cycle, 2*Cycle, and 3*Cycle and then "vote" by using Median() to make the decision on decimation based on which two provides the same result. However, as I think through that, if there are lots of drops and dups, you get overlap between the drops and the dups, and the voting may get confused.

[VoodooFX]

As I speculated in my earlier post, I had it backwards. This fixes the problem. Thanks! (Thanks to StainlessS also).

This should be faster than poisondeathray's example:


BlackFrame = BlankClip( source, Color_yuv=$000000, pixel_type="Y8" )
WhiteFrame = BlankClip( source, Color_yuv=$FFFFFF, pixel_type="Y8" )

[stephen22]

The writer of that script weren't no slouch.

I'm very inexperienced with Avisynth so be kind if I'm talking nonsense.

If you export your data (how do you do this?) and can externally generate a text file that has a code for each frame (normal/delete/insert_after) that can be read by ConditionalReader, might it be possible to build up the output file in one pass? You would have to have at least one variable available from one frame to the next (is this possible?) which kept tabs on how out of step you were when choosing a frame (increment after delete, decrement after insert) from the clip (normal or interpolated) specified by the imported code. Something like

Code
normal__________selectevery(source,1,stepcode)

delete__________increment stepcode; selectevery(source,1,stepcode)

insert_after______decrement stepcode; selectevery(interp,1,stepcode)

I've probably got the logic wrong but is the general idea viable?

[johnmeyer]

If you export your data (how do you do this?) and can externally generate a text file that has a code for each frame (normal/delete/insert_after) that can be read by ConditionalReader, might it be possible to build up the output file in one pass? The key limitation of AVISynth is that it has to know, at every step in the script, the number of frames in the video file. You can use functions which, as an example, double the frames (like MFlowFPS), but I don't think there is a way to insert a frame here, and then a frame there, at arbitrary points.

[stephen22]

Don't follow. The number of frames in the output is the same as the input and doesn't change. There's no insertion or deletion: each output frame is specified according to the algorithm based on information from the text file which your clever spreadsheet program has compiled.

It might look something like
Outputfile=Scriptclip(source,"stepcode=stepcode+framecode"+ chr(13) + \
"framecode < 1 ? selectevery(source,1,stepcode) : selectevery(interp,1,stepcode)")

ConditionalReader("spreadsheet.txt","framecode",false)

The framecode would be -1 = insert_after, 0 = normal, 1 = delete (dupe). Stepcode (how many frames ahead the required frame will be) would have been initialised as 0.

Your spreadsheet text file would look something like
Type Int
Default 0
1 0
2 0
3 -1
4 0
5 1
etc

I don't know whether I've got the syntax right, or indeed whether this would be allowed. If you can't use a variable like this, the "stepcode" value could be computed by your spreadsheet program and buried in the codes in the text.

And of course your spreadsheet would have to provide the text file in the right format.....with the right values!

[VoodooFX]

Why you need Excel for dupe detection?

[johnmeyer]

Why you need Excel for dupe detection?I don't need Excel for detection; AVISynth YDifference can do that quite well. What I need it for is to solve the problem of deciding which "dup" to delete.

Let me go back to the statement of the problem:

The script must first detect the location of where a frame has been dropped;
It must then look for a duplicate that has been added to keep audio in sync. This usually happens within a few frames of the drop
It must do both these things so that dups=drops in order that the video length stay the same.
It must then insert a motion estimated frame at each drop location and must also delete all the duplicate frames

My script works really well at detecting dups and drops, but it fails to match some dups with the drops because any decimation algorithm based on cycles will inevitably have a drop that happens in one cycle, but the dup doesn't happen until the next cycle.

By putting the metrics into an Excel spreadsheet, the decimation can be done without using cycles. The idea is to start with the first drop and then scan the next "n" frames for a metric that indicates a dup, and mark that for deletion. That will take a little work, but I know I can do it because I've done something very similar before.

What I don't know is how to use the metrics I create with the spreadsheet to reassemble the video in AVISynth, with motion estimated frames inserted at the drop points, and duplicates removed at the locations with identical frames. I can script it in Vegas Pro, but that won't help people here.

I have a long plane ride at the end of the week and I may use that time to see how far I can get with this. However, I'm stuck at the moment doing it all with AVISynth until and unless I get a few hints or ideas.

BTW, one reason I stuck with the original approach for so long is not only does it work for many drop/dup pairs, but also if the detection logic flags a "drop" that really isn't a drop, the visual penalty is very small. Any approach must gracefully handle false detection, given that perfect detection of every drop is not possible.

[stephen22]

There 'tis done. Finally cracked the diabolical ConditionalReader syntax (phew!)

source=AVISource( "D:\Video\wares-28-09-2021-17-35-08.avi" ).trim(5000,6000)

stepcode=0

super = MSuper(source)
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)
interp = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70).subtitle("**")

final=Scriptclip(source,"stepcode=stepcode+framecode"+ chr(13) + \
"framecode < 1 ? selectevery(source,1,stepcode) : selectevery(interp,1,stepcode)")
final2=ConditionalReader(final, "spreadsheet.txt", "framecode", true)

return final2

It'll need tweaking when you sort out your spreadsheet program - for instance I think Insert_after should be insert_before and the interp file may need a nudge.

With the text file example in my previous post, the output would be


Frame 0=source frame 0, stepcode=0
Frame 1=source frame 1
Frame 2=source frame 2
Frame 3=interp frame 2, stepcode= -1
Frame 4=source frame 3, stepcode= -1
Frame 5=source frame 5, stepcode=0
etc

(Must be reset with F2 if you fiddle with the timeline or the value of stepcode is corrupted)

[stephen22]

That does work but only if dupes and drops are guaranteed to alternate, which doesn't usually happen. Better approach is to generate a text file which specifies the actual frame number required, indicating whether it's from the original clip or an interpolation. A good way of doing this is by by making interpolated frames negative, so 127 means frame 127 from the source clip and -127 means frame 127 from the interpolation clip.

So this is the method, based on John's brilliant insights - it works pretty well!

1. Generate a file of the ydiff data: the mfile from Multidecimate will do for this.
loadplugin ("D:\Videos\Avisynth\plugins\MultiDecimate.dll")
avisource ("D:\Video\SONY_DVD_RECORDER_VOLUME\ambass test.avi").converttoyuy2
multidecimate (pass=1)

#***run video analysis pass then exit VD immediately***

2. Feed the resulting mfile.txt data into a program which will generate a text file as described above (I have called the generated file "spreadsheet.txt")

3. Put the file in the same folder as this avs script, and run the script.
loadplugin("C:\program files (x86)\avisynth\plugins\mvtools2.dll")

source=avisource ("D:\Video\SONY_DVD_RECORDER_VOLUME\ambass test.avi")
source2=scriptclip(source,"subtitle (string(current_frame))")

super = MSuper(source)
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)
interp = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70).duplicateframe(0)
interp2=scriptclip(interp,"subtitle (string(current_frame) + chr(42))")

final=Scriptclip(source2,"step = (framecode>0)? framecode - current_frame : 0 - (framecode + current_frame)" + chr(13) + \
"framecode >= 0 ? selectevery(source2,1,step) : selectevery(interp2,1,step)")

return stackvertical(ConditionalReader(final, "spreadsheet.txt", "framecode", true),source2)

Writing the program for step 2 is the key, and free from the shackles of Avisynth scripting, the world is your oyster! My own version works as follows:

1. Set thresholds for dupe, drop and scene change.

2. Divide the clip into scenes, which will be processed individually

3. In each scene, score each frame by comparing with the average of the 16 frames surrounding it.

4. Identify and mark the drops

5. For each drop, mark the first dupe among the 8 surrounding frames. If none is found, mark the frame which has the lowest score.

6. Build up the output file, adding frame numbers to the list sequentially starting from 0. If a frame is marked as a dupe, skip its frame number. Put in frames marked as drops twice, the first time negative (to specify an interpolated frame). Save as a text file, in the format required by ConditionalReader (frame number followed by data.)

Here is a ragged clip captured by an HDMI to PAL converter, and the "spreadsheet" text file generated from its mfile data by my program. The Avisynth script above will display the clip before and after processing for comparison, with the frame number, aesterisked if interpolated, in the top left hand corner.
https://drive.google.com/file/d/1kzyrFB7sZwGEKO08xmQIozesrFdTEqws/view?usp=sharing

There's enormous scope here for skilful programming - for instance a sudden increase in movement within a scene can be a problem, creating runs of spurious "drops" which will in turn generate spurious "dupes" which will be removed and introduce new dropped frames. I thought I might try to identify this by comparing the 8 framescores before a frame with the 8 after, and perhaps starting a new "scene" if there is a large increase. It might also be useful to automatically adjust the dupe and drop thresholds for individual scenes according to the amount of movement. There's all sorts of problem-solving to be done. All of which is easy in a normal programming environment.

Unfortunately my version is programmed on an ancient computer - the Atari ST (in fact an ST emulator for Windows called Steem.) If anybody's interested I could supply the necessary files to run the program, which I would have to make a bit more user-friendly. But maybe if there's sufficient interest and demand, programmers more skilful than I could write exe files that everyone could use easily.

[VoodooFX]

Better approach is to generate a text file...
Can't you do everything in one go from one script, like using RT_Stats to parse clips and write/read from its DB files?

And why not deal with the missing frames issue at the capture stage instead of interpolating them after?

[johnmeyer]

That does work but only if dupes and drops are guaranteed to alternate, which doesn't usually happen. Better approach is to generate a text file which specifies the actual frame number required, indicating whether it's from the original clip or an interpolation. A good way of doing this is by by making interpolated frames negative, so 127 means frame 127 from the source clip and -127 means frame 127 from the interpolation clip.

So this is the method, based on John's brilliant insights - it works pretty well! <snip>Wow, thanks for all that. I put this project aside, since I had hit a wall, and also because I flew cross country to the east coast where I a wedding reception. I mention this because the actual wedding occurred a year ago, during lockdown, and the only video if the actual ceremony is something recorded on Zoom. Needless to say, that video is filled with the exact problems this script is supposed to fix.

I hope to get to your code in a few days and see if I can make it work. Thanks for all the help.

[StainlessS]

By the way, I've just noticed this
For the mask part, if you wanted to use levels, it would be this for 8bit YUV


BlackFrame = BlankClip( source, Color=$000000 ).levels(16,1,235,0,255,false)
WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(16,1,235,16,235,false)

thats wrong, should be same [levels] for both black and white [as above, white comes out at 235 not 255]

BlackFrame = BlankClip( source, Color=$000000 ).levels(16,1,235,0,255,false) # Must Haves in Blue
WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(16,1,235,0,255,false)

[VoodooFX]

[color_YUV AVS+ only]

Are you sure? Wiki says that it's added from v2.55.

Without levels should be faster:
BlackFrame = BlankClip( source, Color_yuv=$000000, pixel_type="Y8" )
WhiteFrame = BlankClip( source, Color_yuv=$FFFFFF, pixel_type="Y8" )

[stephen22]

the actual wedding occurred a year ago, during lockdown, and the only video if the actual ceremony is something recorded on Zoom. Needless to say, that video is filled with the exact problems this script is supposed to fix.

I'm having a celebration myself this weekend - my 80th!

If you let me have some of your wedding footage, I'll put it through my program and generate a textfile for the Avisynth script so we can see how well it works. The constraints of my vintage computer mean I can only process up to 16,000 frames at a time, but of course I can do it in sections.

[StainlessS]

Are you sure? Wiki says that it's added from v2.55.
Oops, yes, just checked in v2.58 and it is implemented.
Guess I got mixed up with Color_YUV for AddBorders and Letterbox in AVS+.
[ I even did a script to convert Colors_RGB.avsi to a Colors_YUV.avsi, so go figure :( ]

I'm having a celebration myself this weekend - my 80th!
What Wedding Aniversary, ... Respect.

[johnmeyer]

I'm having a celebration myself this weekend - my 80th!Congratulations. I've got 70 coming up next year.If you let me have some of your wedding footage, I'll put it through my program and generate a textfile for the Avisynth script so we can see how well it works. The constraints of my vintage computer mean I can only process up to 16,000 frames at a time, but of course I can do it in sections.Let me see if I can get it up and running myself because I'll need to do this again in the future.

I keep getting interrupted with more projects and things that need fixing (our microwave just quit, and I'd rather repair it than get a new one because it is built in). I could go on, but I'm about three interrupts deep and will need to pop the stack several times before I get back to this.

[Emulgator]

OT
Microwave oven: If it pops fuses you may go for a search to inrush current limiter.
Could be a timed relay bridging a resistor, the latter being burnt or bad contact, or a NTC.
I drew a partial schematic of a Siemens HE0867 recently. PM me if you like.
/OT

[johnmeyer]

OT
Microwave oven: If it pops fuses you may go for a search to inrush current limiter.
Could be a timed relay bridging a resistor, the latter being burnt or bad contact, or a NTC.
I drew a partial schematic of a Siemens HE0867 recently. PM me if you like.
/OTI've repaired several microwaves. I'd done this one twice before. It is the interlock switch. The contacts pit over time (this was built in 1994) and then it either doesn't work, or blows the house circuit breaker each time it starts. The last two times I removed the switch and then sawed through the ultrasonic welds on the switch, filed down the contacts with an ignition file, and then glued it back together. (It is physically large and therefore easy to repair.) I got 4-5 years from those patches.

This time I decided that method had run its course, so I poked around and, to my great amazement, found an NIB exact replacement. I installed it, the part works perfectly, and I expect the microwave will probably outlast me (something, I now realize, is probably true of a lot of things around here).

[Emulgator]

Congrats ! Yes, hardswitching that 1,5kW transformer demands its toll from the contacts...