Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
|
27th October 2021, 23:44 | #1 | Link |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,710
|
Need help "perfecting" script to delete drops and dups
I have been trying for a decade to perfect a script which can reliably and automatically detect and remove dropped frames, and also automatically fix the duplicates which usually follow the drops. This drop/dup pattern happens at some point in almost every streaming video, and is also often seen in screen captures. It is an incredibly common problem.
I need help with three things: 1. TDecimate is not reliable in this application (this is the main problem) 2. A minor problem: mt_merge does not provide identical video to the original when merged with pure white. The script works despite this flaw, but I'd like to understand how to fix it. 3. I cannot multi-thread the script because of TDecimate Any help would be appreciated. At this point, I have given up. The script is still useful as written, but it will fail to fix quite a few drops and dups, all because of TDecimate's failures. However, it is tantalizingly close to being an incredibly useful script. -------------------------- For those interested in helping, here is the background, followed by my current script. I first posted my script in 2011: Automatically fix dups followed (eventually) by drops , and then again a few weeks ago: Dropped frames When I answered the OP a few weeks ago I thought initially that the main problem was that I wasn't detecting drops very well, but after posting about this (Script for reliably detecting dropped frames) I found out the my technique is actually quite good. There is no need to revisit my detection scheme. The REAL problem is TDecimate. If a frame drop happen on a Cycle boundary (where "Cycle" is the group of frames TDecimate works on at one time), it fails. The reason for this is that Didée's original brilliant idea requires that the drop and the dup happen within a TDecimate "Cycle." Perhaps I need to come up with a different approach, or perhaps there is a way to tweak Didée's idea. The script is also very slow, but does not play nice with MT (also a TDecimate problem), so I can't use that technique to make it play faster. Because it is so slow, the idea of running TDecimate three times with different parameters and then "voting" on the best result is not going to work because the script speed will be measure in seconds per frame instead of frames per second. BTW, I used my "really good" MVTools2 technique for creating the intermediate frames, so when a replacement is made it is usually undetectable. That is why there are so many lines of code preceding the MFlowFPS call. I also tried SVP, and it does work and it is faster, even without MT, so I might go with that if and when I figure out a replacement or a fix for TDecimate. I have left that commented out code in the script. Things I have tried I have tried FDecimate and it doesn't work. I have tried TDecimate using its 2-pass approach which I thought would work for sure, but it does not. I have also tried TDecimate's mode 0, mode 1, mode 2, and mode 7. I am thinking about playing around with Requestlinear settings, and also am wondering if I can use the metrics from FrameDiff to create my own decimation. I am running AVISynth+ 3.7.0 and am using pinterf's 2/22/2021 compile of TIVTC (which includes TDecimate). The comments at the beginning of the script describe how it works. If anyone has any ideas about how to fix it, let me know. The script is still partially hard-wired for 59.94 fps, but I will clean that up and turn it into an AVSI if I ever get the problems solved. Here are links to two really tiny test videos. The first is artificial video (just a title scrolling horizontally) and the other is real video with multiple short scenes chosen for their different types of motion. I used my NLE to create a drop, followed by a dup within 1 to 12 frames later. The real video link also includes a text file which gives you the frame numbers where I created a drop, and also the frame numbers for the dups I inserted. "Trivial" manufactured video test https://www.mediafire.com/folder/yjf...d3d4mlo/shared Real Video Test https://www.mediafire.com/file/yjffj..._Test.zip/file Thanks in advance to anyone who has read this far and might have some ideas. Code:
# Based on script created by Didée # Modified by John Meyer on June 29, 2011 # Further modification on June 8, 2013 # Tdecimate modifications on October 26, 2021 # # 1. Create interpolated frames a 2x original frame rate using MVTools2. # 2. Detect jumps in the original 1x video. # 3. Create white mask at each jump point; black mask for all other frames. # 4. Repeat each frame of original video. # 5. For each frame in double-frame video, use a mask to "choose" between that original video, or the motion estimated video # 6. Decimate exactly 50% to get back to original frame rate. # # This decimation removes the dup frames that existed in the original video and also removes the dups created by # repeating each frame of original video. # However, at each point where motion-estimated frame was inserted, no decimation occurs. # Thus, if dups=drops, all dups in the original video will be removed and the drop will be filled. # On the other hand, if no drops or dups occur, then no motion estimation appears in the final result, and the decimation # merely gets back to original, unchanged video. #********************************************* #********************************************* #User-settable paramaters #Puts dot on screen during debugging. True for troubleshooting; otherwise, false. showdot = true #Threshold for detecting jumps. Increase to catch more jumps. Should always be less than 1.0 JumpThresh = 0.8 #Frame rate of your video #PAL: Vidnum=25; VidDen=1; NTSC: Vidnum=30000; VidDen=1001; etc. VidNum = 60000 VidDen = 1001 #Detection window size. Try changing if you find residual dups or drops. Script gets slow if you use big numbers DupCycle = 18 threads = 1 #TDecimate does not like multithreading #********************************************* loadplugin("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\MVTools2 (latest)\mvtools2.dll") Loadplugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools-26.dll") #These were used when trying to get AVISynth+ multi-threading to work #SetFilterMTMode("TDecimate", MT_SERIALIZED) #SetFilterMTMode("TDecimate", MT_MULTI_INSTANCE) global source=AVISource("E:\fs.avi").ConvertToYV12.killaudio() global BlackFrame = BlankClip( source, Color=$000000 ) global WhiteFrame = BlankClip( source, Color=$FFFFFF ) prefiltered = RemoveGrain(source,22) super = MSuper(source,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=4,search=3,dct=0) forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=4,search=3,dct=0) forward_re = MRecalculate(super, forward, blksize=8, overlap=2, thSAD=100) backward_re = MRecalculate(super, backward, blksize=8, overlap=2, thSAD=100) #This is alternative way to double the number of frames. #double = SmoothFPS2 (source,threads) double = source.MFlowFps(super, backward_re, forward_re, num=VidNum*2, den=VidDen, blend=false) double = showdot ? double.subtitle("***") : double test=ShowMetrics(source) #Generate a white or black frame, depending on frame difference BWMask=GenerateMask(source) #Generate the 2x framerate mask needed to choose the motion-estimated frames themask = interleave(BlackFrame,trim(BWMask,1,0)) #Merge double framerate from original with motion-esimated frames, but only where there are jumps #(i.e., original frames are used except at jump points) # *** NOTE *** mt_merge does NOT produce a clean merge, and the dups are slightly altered from the originals -- unable to fix interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3) #Decimate half of all frames RequestLinear tdecimate(display=false,debug=false,mode=1,Cycle=DupCycle*2,CycleR=DupCycle,vfrdec=1,sdlim=0) #Alternate decimation -- should work, but doesn't #tdecimate(display=false,debug=false,mode=2,rate=59.94006,maxndl=19,m2PA=true) #Alternate ways to end the script. Must assign variable to tdecimate output instead of using implied "last." #prefetch(threads) #return test #Stacked view, for debugging #return stackvertical(source,decimated) #End Main Program #---------------- #Begin Functions #---------------- #This function displays the YDiff value that will be used for detecting big jumps #Each YDiff must eliminate Ydiff~=0 (i.e., a duplicate) from moving average #This is why there are three lines for each YDiff. Each block uses the next or preceeding #frame, if the current YDiff is near zero, indicating a dup. #Note: This function works just fine function ShowMetrics (clip c) { fixed=source.ScriptClip("Subtitle(String( \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, 3)) : \ YDifferenceFromPrevious(selectevery(source, 1, 2)) ) \ + \ (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, 2)) : \ YDifferenceFromPrevious(selectevery(source, 1, 1)) ) \ + \ (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, -2)) : \ YDifferenceFromPrevious(selectevery(source, 1, -1)) ) \ + \ (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, -3)) : \ YDifferenceFromPrevious(selectevery(source, 1, -2)) ) \ )/4) / \ (YDifferenceFromPrevious(source) + 0.01) \ ))") return fixed } #---------------- #This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned #Each YDiff must eliminate Ydiff=0 (duplicate) from moving average. #Since the first two frames of the clip have no preceding frame, don't return a mask #for those two frames (which means there will be no fix for the first two frames) #This function works just fine (i.e., really great detection) function GenerateMask (clip c) { MyMask=c.ScriptClip(""" \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, 3)) : \ YDifferenceFromPrevious(selectevery(source, 1, 2)) ) \ + \ (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, 2)) : \ YDifferenceFromPrevious(selectevery(source, 1, 1)) ) \ + \ (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, -2)) : \ YDifferenceFromPrevious(selectevery(source, 1, -1)) ) \ + \ (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.3 ? \ YDifferenceFromPrevious(selectevery(source, 1, -3)) : \ YDifferenceFromPrevious(selectevery(source, 1, -2)) ) \ )/4) / \ (YDifferenceFromPrevious(source) + 0.01) <= JumpThresh \ && current_frame > 2 \ ? WhiteFrame : BlackFrame """) return MyMask } #This function is hard-wired for 60 fps. Not worth the time to generalize it, since SVP #didn't seem to provide a huge benefit over MVTools2, although it should be faster. function SmoothFPS2(clip source, threads) { super_params="{pel:2,gpu:1}" analyse_params="""{ block:{w:16,h:16}, main:{search:{coarse:{distance:0}}}, refine:[{thsad:200}] }""" smoothfps_params="{rate:{num:120,den:60,abs:false},scene:{mode:0,limits:{scene:8500}},algo:21,cubic:1}" super = SVSuper(source,super_params) vectors = SVAnalyse(super, analyse_params) SVSmoothFps(source,super, vectors, smoothfps_params, url="www.svp-team.com", mt=threads) } |
27th October 2021, 23:55 | #2 | Link | |
Registered User
Join Date: Sep 2007
Posts: 5,434
|
Quote:
Same for "black" . Y=16, not Y=255 Code:
global source=AVISource("E:\fs.avi").ConvertToYV12.killaudio() global BlackFrame = BlankClip( source, Color=$000000 ) global WhiteFrame = BlankClip( source, Color=$FFFFFF ) |
|
28th October 2021, 02:31 | #3 | Link |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,710
|
Thanks for that. I started to try out your suggestion by adding a levels statement (e.g., levels(255,1,255,255,255)) when all of a sudden the script wouldn't work. After ten wasted minutes I decided to re-boot (Win XP 32-bit). The script started working again (with the problems reported above, of course).
This means I am chasing memory leak problems as well. I like AVISynth and I appreciate open source, but it sure can be fragile at times. Oh yes, the levels(255,1,255,255,255) and levels(0,1,0,0,0) statements did not fix the problem. Code:
global BlackFrame = BlankClip( source, Color=$000000 ).levels(0,1,0,0,0) global WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(255,1,255,255,255) Code:
interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3) If I end up having to create my own decimation workflow outside of AVISynth, exporting metrics and then importing a decimation override file, it will be much more important to have these metrics be zero. |
28th October 2021, 02:38 | #4 | Link | |
Registered User
Join Date: Sep 2007
Posts: 5,434
|
Quote:
Code:
BlackFrame = BlankClip( source, Color=$000000 ).levels(16,1,235,0,255,false) WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(16,1,235,16,235,false) |
|
31st October 2021, 19:51 | #7 | Link | |
Banana User
Join Date: Sep 2008
Posts: 1,010
|
Quote:
Code:
BlackFrame = BlankClip( source, Color_yuv=$000000, pixel_type="Y8" ) WhiteFrame = BlankClip( source, Color_yuv=$FFFFFF, pixel_type="Y8" )
__________________
InpaintDelogo, DoomDelogo, JerkyWEB Fixer, Standalone Faster-Whisper - AI subtitling |
|
28th October 2021, 02:53 | #9 | Link |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,710
|
Yes.
https://forum.doom9.org/showthread.p...24#post1409324 I completely changed the detection logic not only because his wasn't very good, but also because there was no way to display the detection metrics. If you can't display them, you can't tune the script. |
28th October 2021, 10:18 | #11 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,992
|
Quote:
Code:
# DeDecimateNtsc24BackTo30.avsi Function DeDecimateNtsc24BackTo30(clip c,Bool "ShowDot") { /* DeDecimateNtsc24BackTo30.avsi : Mod to Script function by StainlessS. Original script by Didee @ https://forum.doom9.org/showthread.php?p=1409324#post1409324 required, AVS v2.60/+, Pinterf mt_masktools, Pinterf mvtools2, Pinterf tivtc. Source should be NTSC_FILM(23.976p) that has been decimated from original PROGRESSIVE_NTSC_STANDARD(29.97p). */ c myName="DeDecimateNtsc24BackTo30: " ShowDot = Default(ShowDot,False) # shows a "dot" on the interpolated frames of the result o = assumefps(1.0) # Avoid frame doubling rounding issues, due to weird NTSC framerates. ox = o.width() oy=o.height() Assert(!c.IsRGB && c.IsPlanar,myname+"Planar YUV only") try { bpc=c.BitsPerComponent } catch(msg) { bpc=8 } Assert(8 <= bpc <= 16,myname+"8 to 16 bit only") super = (showdot ? o.subtitle(".") : o).MSuper(pel=2,hpad=16,vpad=16) bvec = MAnalyse(super, overlap=4, isb = true, search=4, dct=5) fvec = MAnalyse(super, overlap=4, isb = false, search=4, dct=5) double = o.MFlowFps(super, bvec, fvec, num=2, den=1, blend=false) # Result will be 2.0 FPS, and FrameCount=(c.FrameCount*2)-1. Dont Blend at SceneChange. shftn = BitLShift(1, bpc-8) cs256 = BitLShift(256,shftn) cs255 = cs256-1 cs128 = BitLShift(128,shftn) cs32 = BitLShift(32,shftn) d2n_yexpr="x " + String(cs128) + " - abs " + String(cs32) + " / 1 2.0 / ^ " + String(cs128) + " *" # [8-Bit]:- rpn="x 128 - abs 32 / 1 2.0 / ^ 128 *" : Infix="( (abs(x-128)/32) ^ (1/2.0) ) * 128" diff2next = mt_makediff(o,o.selectevery(1,1)).mt_lut(d2n_yexpr,U=-(cs128),V=-(cs128)) diff2next = mt_lutf(diff2next,diff2next,yexpr="x",mode="average").pointresize(32,32) diff2next = interleave(diff2next.selectevery(4,0).tsg(2),diff2next.selectevery(4,1).tsg(2), \ diff2next.selectevery(4,2).tsg(2),diff2next.selectevery(4,3).tsg(2)) maximum = diff2next.mt_logic(diff2next.selectevery(1,-3),"max") \ .mt_logic(diff2next.selectevery(1,-2),"max") \ .mt_logic(diff2next.selectevery(1,-1),"max") \ .mt_logic(diff2next.selectevery(1, 1),"max") \ .mt_logic(diff2next.selectevery(1, 2),"max") \ .mt_logic(diff2next.selectevery(1, 3),"max") ismax_yexpr = "x y < 0 " + String(cs255) + " ?" # [8-Bit]:- rpn = "x y < 0 255 ?" : Infix = "(x < y) ? 0 : 255" ismax = mt_lutxy(diff2next,maximum,ismax_yexpr,U=-(cs128),V=-(cs128)).pointresize(ox,oy) themask = interleave(o.mt_lut("0"),ismax) interleave(o,o).mt_merge(double,themask,luma=true,U=3,V=3) # Process U and V tdecimate(mode=1,cycleR=3,cycle=8) # Decimate 3 out of 8 = 18 out of 48 (leaving 24 from 48) assumefps(30000,1001) # Back to NTSC_Video return last } #=========================== # tsg by Didee @ https://forum.doom9.org/showthread.php?p=1409324#post1409324 # ssS: Looks like some kind of limited Gausian Temporal Soften. # TemporalSoften (clip, int radius, int luma_threshold, int chroma_threshold, int "scenechange", int "mode") # Radius=1=Adjacent_frames, Luma_threshold=255=no limit, chroma_threshold=0=dont process, scenechange=255=ignore, mode=2=ISSE function tsg(clip c, int t) { c t<5?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25) t<4?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25) t<3?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25) t<2?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25) t<1?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25) } EDIT: 8 -> 16 bit depth.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 28th October 2021 at 18:34. |
|
28th October 2021, 16:19 | #12 | Link |
Banana User
Join Date: Sep 2008
Posts: 1,010
|
I had problems too with TDecimate. To avoid drops I do real-fps *3 video, and TDecimate just sometimes drops non-duplicate frame when duplicate is just next one.
__________________
InpaintDelogo, DoomDelogo, JerkyWEB Fixer, Standalone Faster-Whisper - AI subtitling |
28th October 2021, 18:20 | #13 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,992
|
There is also this Doggie-Doo mod of Didee / JohnM script[FillMissing]:- https://github.com/Dogway/Avisynth-S...llMissing.avsi
As well as JKyle mod [with wrong black/white frame colors]:- https://forum.doom9.org/showthread.p...24#post1947624 EDIT: Doggy uses Color_YUV=$000000, and Color_YUV=$FFFFFF which are also a bit wrong, should be (for 8 bit ONLY) Black = BlankClip( source, Color_yuv=$008080 ) White = BlankClip( source, Color_yuv=$FF8080) [color_YUV AVS+ only] Maybe use (where masktools 2 used already) Black=source.mt_lut("0") White=black.mt_Invert() # OK for HBD, eg 16 bit Y becomes $FFFF, not $FF00 (FULL WHITE required for Mt_merge mask).
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 28th October 2021 at 18:41. |
17th November 2021, 18:06 | #15 | Link |
Banana User
Join Date: Sep 2008
Posts: 1,010
|
Are you sure? Wiki says that it's added from v2.55.
Without levels should be faster: Code:
BlackFrame = BlankClip( source, Color_yuv=$000000, pixel_type="Y8" ) WhiteFrame = BlankClip( source, Color_yuv=$FFFFFF, pixel_type="Y8" )
__________________
InpaintDelogo, DoomDelogo, JerkyWEB Fixer, Standalone Faster-Whisper - AI subtitling Last edited by VoodooFX; 17th November 2021 at 18:08. |
28th October 2021, 18:45 | #16 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,992
|
May be for 8-bit only, but did you see this,
Quote:
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 28th October 2021 at 18:48. |
|
28th October 2021, 20:59 | #17 | Link | ||
Registered User
Join Date: Sep 2007
Posts: 5,434
|
Quote:
expr("0", "x", "x") for copy u,v and expr("0", "range_half", "range_half") for half uv at source bitdepth Code:
source=("random 1080p source") black = source.mt_lut("0").killaudio() #black = blankclip(source).expr("0", "range_half", "range_half"").killaudio() #black = blankclip(source).expr("0", "x", "x").killaudio() #black = blankclip(source).mt_lut("0").killaudio() black (I believe expr is optimized with avx2 instructions, but it didn't make much of a difference on these benchmarks with SetMaxCPU("sse4.1") for this operation ) ffmpeg -i mt_lut.avs -f null NUL 285fps, 60% ffmpeg -i blankclip_expr_uv_range_half.avs -f null NUL 1060fps, 25% ffmpeg -i blankclip_expr_uv_copy.avs -f null NUL 1010fps, 25% ffmpeg -i blankclip_mt_lut.avs -f null NUL 490fps, 22% Eitherway, you're right it should have limited impact speedwise on John's specific script (it's not going to be the part that is the "bottleneck") Do you need to copy the chroma channels for the mask ? Really you only need "Y" plane 100% white or 100% black. It might be slightly faster to do it in Y or Gray for the mask part, instead of YUV (discard instead of dragging the chroma channels along) |
||
29th October 2021, 03:10 | #18 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,992
|
I'm kinda reluctant to use Expr() for anything in my scripts directory, If I need to test something under 2.60 std, then I gotta remove all of those scripts so as not to cause multiple problems.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
28th October 2021, 18:52 | #19 | Link |
Registered User
Join Date: Nov 2009
Posts: 2,367
|
If your mask is $FFFFFF you can set luma=false which might be faster
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread |
28th October 2021, 19:08 | #20 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,992
|
Cheers Doggggy, did not know what that one was for.
EDIT: Luma=false = default. Quote:
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 28th October 2021 at 19:13. |
|
Tags |
drops, duplicates, tdecimate |
Thread Tools | Search this Thread |
Display Modes | |
|
|