Another frame duplication function

[Great Dragon]

neuron2, It's no difference with exactly named agrument or without. I have only Avisource() filter before trim section so "last" is explicitly provided I think. Anyway nothing changed, an error is still present.

[Guest]

Quite right, sorry for the noise.

[bxyhxyh]

Sorry for reviving ancient thread.
I think
global lastnewframe = 0

can fix "i don't know lastnewframe means".

And i think those assert() don't need. And stats=true mod is now become faster.

# Dupped() by corran.  Feel free to modify as you see fit. 
# Attribution not required.
#
# This filter requires a YUV source. Use converttoyv12 if needed
# before calling dupped()
#
## Parameters
# thresh = Used to determine when to delare a frame as new
# Use stats=true to help determine the best value for your source
# Lower number = more sensitive. (Default=16)
#
# panthresh=1.7 This is used to determine when to consider a frame
# part of a low motion scene. (Default=1.65)
#
# stats = Enable/disable display of stats. (Default=false | Not shown)
# (Last = last frame, this = this frame, next = next frame)
#
# showme = Enable/disable display of video that most stats are
# derived from. (Default=false | Not shown)
#
## Notes
# Do not use with SetMTMode(). This function requires the frames to be
# accessed sequentially. Instead, use MT() to mulit-thread on a per-filter basis.
#
# Example:
# dupped(thresh=20,panthresh=1.7,stats=true,showme=true) 

#Function to return the requested frame
function getFrame(clip c, int frameNum)
{
    default(c, last)
    c = trim(frameNum,frameNum)
    return c
}

function dupped(clip c, int "thresh", float "panThresh", bool "stats", bool "showme")
{
    default(c, last)
    global dupThresh = default(thresh, 16)
    global panThresh = default(panThresh, 1.65)
    global dupStats = default(stats, false)
    global frame = 0
    global lastnewframe = 0
    showme = default(showme, false)
    a = c
    c = scriptclip(last,"""
        c = last
    
        finalframe = c.framecount-1
        prev_frame = current_frame != 0 ? current_frame-1 : 0
        next_frame = current_frame != finalframe ? current_frame+1 : finalframe
            
        #Get Y related stats as needed
        prevCurrentYMinMax = subtract(getframe(prev_frame),c).YPlaneMinMaxDifference
        currentNextYMinMax = prevCurrentYMinMax < dupThresh ? subtract(c,getframe(next_frame)).YPlaneMinMaxDifference : 256
        prevNextYMinMax = currentNextYMinMax < dupThresh ? subtract(getframe(prev_frame),getframe(next_frame)).YPlaneMinMaxDifference : 0
    
        #calculate dynamic pan threshold.   Set to 256 if looking of a pan is pointless
        neighborAvgMinMax = (prevCurrentYMinMax + currentNextYMinMax) / 2
        dynPanThresh = prevNextYMinMax != 0 ? Abs(neighborAvgMinMax * panthresh) : 256

        #determine if the current frame is new
        newframe = prevCurrentYMinMax >= dupThresh || prevNextYMinMax >= dynPanThresh ? true : false
        lastNewFrame = newframe ? current_frame : lastNewFrame
    
        c = getFrame(lastNewFrame)
        c = dupStats ? c.subtitle(\
"prevframe: "+string(prev_frame)+"  currentframe: "+string(current_frame)+"  nextframe: "+string(next_frame) + \
"\nLast new frame: "+string(lastNewFrame)+" Newframe: "+string(newframe) + \
"\nprevCurrentYMinMax: "+string(prevCurrentYMinMax)+" (Threshold:"+string(dupThresh)+")", \
lsp=1) : c
        c = dupStats && currentNextYMinMax != 256 ? c.subtitle("currentNextYMinMax: "+string(currentNextYMinMax),y=55) : c
        c = dupStats && prevNextYMinMax >= dynPanThresh ? c.subtitle(\
"prevNextYMinMax: "+string(prevNextYMinMax) + \
"\nPan Threshold: "+string(dynPanThresh)+" ("+string(neighborAvgMinMax)+" * "+string(panThresh)+")" + \
"\nSlow pan detected", \
lsp=1, y = 90) : c
        return c
    """)
    c = showme ? stackvertical(c,subtract(a.duplicateframe(0),a)) : c
    
    return c
}

[StainlessS]

Original Dupped, assignments to lastnewframe were missing Global prefix (ie were actually setting a Local Var).

Moved here from another thread:- http://forum.doom9.org/showthread.ph...84#post1716484

Multi-Instance Version Dupped for v2.58+ .

Dupped_MI.avs

Function Dupped_MI(clip c, Int "Thresh",Float "Ignore",Float "Override",Float "ChromaWeight",
        \ bool "Show", bool "Stack",String "Fn",Bool "FnComments") {
/*
    Dupped_MI.avs v1.01 
    Dupped, Multi-Instance version for Avisynth v2.58+ --- Planar, YUY2, RGB24/32 --- by StainlessS on Doom9
    http://forum.doom9.org/showthread.php?p=1716968#post1716968
        Dupped() original author, Corran, thanks very much.

    Note, Thresh, and Override defaults are set for YUV TV levels, would likely need to increase both if using RGB
          (maybe up to about 33% higher, use Show=True,Stack=True to tune args for source).
    
    ##############

    Thresh (Default 16, 0 -> 254).
            If MinMaxDifference(Subtract(Anchor_Frame,current_frame)) <= Thresh, then is Dupe, Else is new non-dupe frame.
            The detector is a bit 'back-to-front', it detects non-dupes rather than dupes. (when mnmxdif > Thresh, is non-dupe).
            Where Thresh=0, every frame that is not EXACTLY the same (mnmxdif!=0) is a non-dupe.
             
    Ignore (Default 0.01, Range 0.0 -> 100.0) Percentage, As for Threshold in YPlaneMinMaxDifference (avoids a few extreme pixels ie noise)
            Suggested range for anime about 0.001 -> 0.01, (0.0 = OFF, dont ignore any noisy pixels).
            0.001 would ignore up to 1 extreme pixel in every 100,000 and 0.01 would ignore up to 1 extreme pixel in every 10,000.
            From Avisynth Docs for eg YPlaneMin/Max/YPlaneMinMaxDifference:-
                Threshold is a percentage, stating how many percent of the pixels are allowed above or below minimum.
            ColorYUV ignores 1 pixel in every 256 when showing Loose Minimimum and Loose Maximimum values (about 0.4%) when Analyze=True. 
            From Avisynth Docs for ColorYUV Analyze mode,                
                There is a "loose maximum" and "loose minimum". The "loose" values are made to filter out very bright or very dark
                noise creating an artificially low or high minimum / maximum.
            To detect Dupes, we first Subtract current_frame from the Anchor Frame (anchor frame is the previous non-dupe frame),
            after this we do the equivalent of YPlaneMinMaxDifference(Threshold=Ignore) on the subtracted result frame and if
            that is <= Thresh then is treated as a dupe, otherwise as a new non-dupe frame. YPlaneMinMaxDifference(Threshold=0.0) would
            detect a single pixel difference between Anchor and current_frame that is > Thresh as a non-dupe frame, so a very small
            amount of noise could throw off dupe detection. The 'Ignore' setting, allows the detector to ignore a few noisy pixels when
            finding both minimum and maximum pixels values in the subtracted frame and so also affects the MinMaxDifference.
               
    Override (Default 0.5).  Suggest about 0.5 -> 1.5. 255.0 switches Override completely OFF.
            Metrics 'FD' is the average pixel difference between the Anchor Frame and current_frame.
            Thresh detected Dupes are Overridden to non-dupe if metrics 'FD' (RT_FrameDifference) is greater than Override.
            If there are frames in sample that should not be dupes then set Override just lower than the shown FD for the 
            worst frame (frame with lowest FD that should be overridden to 'not a dupe').
            Override is just a safety measure so that you can set quite ferocious THRESH and IGNORE without fear of making big mistakes. 
            
    ChromaWeight (Default 1.0/3.0, 0.0 -> 1.0) ChromaWeight arg to RT_FrameDifference, weight for combined U + V, use 0.0 for 
            Grayscale source or to use Luma Differences only. RGB not used. Suggest about 1.0/3.0 -> 1.0/2.0.
                          
    Show   (Default False) Shows Metrics if True
            Metrics shown on top line are, character flags 'T','O', and 'D'. Where Hilited, 'T' means Thresh detected dupe, 'O' means
            Dupe Overridden to non-dupe, and 'D' means using Dupe. So 'T' means was detected as dupe by Thresh, however if 'O' is also
            hil-lited, then was Overridden to be NOT dupe. The 'D' flag gives the final decision, will be hi-lited if 'T' is hi-lited
            but 'O' is not hi-lited. 
            Following the character flags, is the current dupe number ie 1 is first in a run of dupes, 2 is 2nd etc. and Following
            that is the currently greatest count of consecutive dupes in any single run.
                
    Stack  (Default False) Shows difference between Current frame and Anchor Frame (Using Subtract).
    
    Fn     (Default "")    Filename, if set then write command text file for FrameSel() plugin.
    
    FnComments (Default False). Write Comments to Fn File if True.

    ##############
    
    If not detecting enough dupes, then raise Ignore and/or Thresh.
    Override should be set AFTER BOTH Thresh and Ignore (can set Override to 255.0 to temporarily disable Override).
    To show EXACT dupes only, use Thresh=0(Exact dupes only),Ignore=0.0(ignoring no noise),Override=255.0(dont override anything).
       
    Must play through without jumping about, if jumping about then will not get back into sync until a new non-dupe anchor frame is found.
        
    Script for use with FrameSel plugin, can use result of below FrameSel script where using some kind of non linear access script.
    >>>>>>>>>>>>>>>>>>>>>>>>>>>>
        Avisource("LowBitrateAnime.mkv.AVI").trim(0,0)  # Trim or Pad audio to length
        Audio = Last                                    # Save Audio, FrameSel does not return audio
        FrameSel(Cmd="FrameSelCmd.txt",Ordered=False)   # where "FrameSelCmd.txt" is Fn in Dupped_MI()
        Return AudioDub(Last,Audio)                     # Restore audio
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<
    !!! NOTE !!!
    To REMOVE (rather than replace) detected duplicates change above FrameSel arg 'Ordered=False' to 'Ordered=True'.
    Obviously the audio will be out of sync if removing dupes so do not restore audio via AudioDub (eg security camera caps).                
*/
    myName="Dupped_MI: "
    Assert(RT_FunctionExist("GScriptClip"),myName+"Essential GRunT plugin installed, http://forum.doom9.org/showthread.php?t=139337")
    Assert(RT_FunctionExist("GScript"),myName+"Essential GScript plugin installed, http://forum.doom9.org/showthread.php?t=147846")
    IsV26 = VersionNumber >= 2.6                                # Avisynth Version v2.6 ?
    Thresh = Default(Thresh, 16)                                # Smaller or equal to this is Dupe, Greater than this is non-dupe.
    Ignore = Float(Default(Ignore,0.01))                        # Ignore up to 1 pixel in every 10,000 (eg 100 x 100 pixel square)
    Override=Float(Default(Override,0.5))                       # Suggest about 0.5 -> 1.5 (perhaps higher if noisy clip)
    CW      =Float(Default(ChromaWeight,1.0/3.0))               # Suggest 1.0/3.0 -> 1.0/2.0
    Show   = Default(Show, false)
    Stack  = Default(Stack, false)
    Fn     = Default(Fn, "")
    FnCom  = Default(FnComments,False)
    Assert(Thresh>=0 && Thresh<255,RT_String("%sThresh 0 -> 254(%d)",myName,Thresh))
    Assert(Ignore>=0.0 && Ignore<=100.0,RT_String("%sIgnore 0.0 -> 100.0(%f)",myName,Ignore))    
    Assert(Override>=0.0 && Override<=255.0,RT_String("%sOverride 0.0 -> 255.0(%f)",myName,Override))    
    Assert(CW>=0.0 && CW<=1.0,RT_String("%sChromaWeight 0.0 -> 1.0(%f)",myName,CW))    
    (Fn != "" && Exist(Fn)) ? RT_FileDelete(Fn) : NOP
    (Fn!="" && FnCom) ? RT_WriteFile(Fn,"#\n# Thresh=%d Ignore=%f Override=%f\n#",Thresh,Ignore,Override) : NOP
    c = (Stack) ? StackVertical(c,c) : c
    Fmt = "%6d] Using=%d : \a%cT\a%cO\a%cD\a- : %d : %d\n" +
            \ "MnMxDif=%3d FD=%-7.3f\n" +
            \ "Anchor=%d OvrCnt=%d DupCnt=%d\n" +
            \ "\n\aR{Thresh=%d Ignore=%.3f Override=%.3f CW=%.3f}"
    FuncS="""
        Function Fn@@@(clip clp,Float Thresh,Float Ignore,Float Override,Float CW,Bool Show,Bool Stack,
                    \ String Fmt,String Fn,Bool FnCom,Bool IsV26) {
            Last = (Stack) ? clp.Crop(0,0,-0,clp.Height/2): clp
            n = current_frame
            InSeq = (Prev@@@ == n-1)
            OldAnchor =  (InSeq) ? Anchor@@@ : max(n-1,0)                   # Remember on entry Anchor FrameNo for metrics
            AFrm=Trim(OldAnchor,-1)
            BFrm=Trim(n,-1)
            SubFrm = Subtract(AFrm,BFrm)
            fd = RT_FrameDifference(AFrm,BFrm,n=0,n2=0,ChromaWeight=CW)     # Average pixel difference
            if(IsYUV) {
                MnMxDif = SubFrm.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
                if(!IsV26) {
                    MnMxDifU = SubFrm.UToY.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
                    MnMxDifV = SubFrm.VToY.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
                    MnMxDif=Max(MnMxDif,MnMxDifU,MnMxDifV)
                } Else if(!IsY8()) {
                    MnMxDifU = SubFrm.UToY8.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
                    MnMxDifV = SubFrm.VToY8.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
                    MnMxDif=Max(MnMxDif,MnMxDifU,MnMxDifV)
                }                    
            } Else {
                SubFrm.RT_RgbChanStats(n=0,Chan=-1,Flgs=$04,Prefix="DUP_RCS@@@_",Threshold=Ignore)
                MnMxDif=Max(DUP_RCS@@@_MinMaxDiff_0,DUP_RCS@@@_MinMaxDiff_1,DUP_RCS@@@_MinMaxDiff_2)
            }
            ThreshDetect = (MnMxDif <= Thresh)
            Overridden  = (ThreshDetect && fd > Override) 
            IsNewFrame = (!ThreshDetect || Overridden || !InSeq)
            Global Anchor@@@ = IsNewFrame ? n : Anchor@@@       # Using this FrameNo, Next time will be Anchor FrameNo
            Trim(Anchor@@@,-1)                                  # Get Using Frame
            if(Show) {            
                Global DupCnt@@@ = IsNewFrame ? DupCnt@@@ : DupCnt@@@ + 1
                Global OvrCnt@@@ = Overridden ? OvrCnt@@@ + 1 : OvrCnt@@@
                DupNum=IsNewFrame?0:n-OldAnchor
                Global DupMax@@@ = Max(DupMax@@@,DupNum)
                Last = RT_SubTitle(Fmt,n,Anchor@@@,(ThreshDetect)?54:76,(Overridden)?54:76,(!IsNewFrame)?54:76,DupNum,DupMax@@@,
                        \ MnMxDif,fd,
                        \ OldAnchor,OvrCnt@@@,DupCnt@@@,
                        \ Thresh,Ignore,Override,CW
                        \ )
            }
            If(Fn != "") {
                if(!FnCom) {
                    RT_WriteFile(FN,"%d",Anchor@@@,Append=True)
                } Else {
                    (Overridden)      ? RT_WriteFile(FN,"%d # *** OVERRIDE: MnMxDif=%d FD=%f",Anchor@@@,MnMxDif,fd,Append=True)
                    \ : (!IsNewFrame) ? RT_WriteFile(FN,"%d # [%d] !!! DUPE: MnMxDif=%d FD=%f",Anchor@@@,n,MnMxDif,fd,Append=True)
                    \ :                 RT_WriteFile(FN,"%d # NON-DUPE: MnMxDif=%d FD=%f",Anchor@@@,MnMxDif,fd,Append=True)                  
                }
            }
            Global Prev@@@=n
            (Stack) ? StackVertical(Last,SubFrm) : Last
            Return Last         
        }
        #######################################
        # Unique Global Variables Initialization
        #######################################
        Global Anchor@@@=0 Global DupCnt@@@=0 Global DupMax@@@=0 Global OvrCnt@@@=0 Global Prev@@@ = -666
        #######################################
        # Unique Runtime Call, GScriptClip must be a one-liner:
        #######################################
        ARGS = "Thresh,Ignore,Override,CW,Show,Stack,Fmt,Fn,FnCom,IsV26"
        c.GScriptClip("Fn@@@(last, "+ARGS+")", local=true, args=ARGS)
    """
    #######################################
    # Unique Identifier Definition
    #######################################
    RT_IncrGlobal("Dupped_MI_InstanceNumber")
    InstS = RT_StrReplace(FuncS,"@@@","_Dupped_MI_"+String(Dupped_MI_InstanceNumber))
#   RT_WriteFile("DEBUG_Dupped_MI_"+String(Dupped_MI_InstanceNumber)+".TXT","%s",InstS)
    Return GScript(InstS)
}

Multi-Instance Wrapper

Function Dupped_MI_X(clip c, Int "Thresh",Float "Ignore",Float "Override",Float "ChromaWeight",
    \ bool "Show", bool "Stack",String "Fn",Bool "FnComments",
    \ Bool "Pair", Int "Thresh2",Float "Ignore2",Float "Override2",Float "ChromaWeight2") {
    /*
        Wrapper for convenience.
        Works same as Dupped_MI() but auto StackHorizontal of 2nd instance if Pair=True.
        Defaults for 2nd Instance are for EXACT dupes only, Change args to 2nd clip using Thresh2, Ignore2, Override2, and ChromaWeight2.
    */
    Pair = Default(Pair, false)
    Th2  = Default(Thresh2, 0)
    Ig2  = Default(Ignore2,0.0)
    Ov2  = Default(Override2,255.0)
    Cw2  = Default(ChromaWeight2,ChromaWeight)
    A = c.Dupped_MI(Thresh=Thresh,Ignore=Ignore,Override=Override,ChromaWeight=ChromaWeight,Show=Show,Stack=Stack,Fn=Fn,FnComments=FnComments)    
    B = (Pair) ? c.Dupped_MI(Thresh=Th2,Ignore=Ig2,Override=Ov2,ChromaWeight=Cw2,Show=Show,Stack=Stack,Fn="",FnComments=False) : 0
    (B.IsClip) ? StackHorizontal(A,B) : A                 
    Return Last
}

Client Script for Multi-instance wrapper

FN="WebCam3.AVI"

Avisource(FN)

SHOW     = True             # True=Show metrics.
STACK    = True             # Show StackVertical Subtract frame
COMMENTS = True
PAIR     = True             # True shows StackHorizontal 2 or 4 windows dependant upon Stack
THRESH   = 16               # DEFAULTS
IGNORE   = 0.01             #
OVERRIDE = 0.5              #

#ConvertToRGB24
#ConvertToRGB32
#ConvertToYUY2
#ConvertToYV12
#ConvertToY8                # V2.6 ONLY
#ConvertToYV16
#ConvertToYV24
#ConvertToYV411

Dupped_MI_X(Thresh=THRESH,Ignore=IGNORE,Override=OVERRIDE,Show=SHOW,Stack=STACK,Fn="FrameSelCmd.txt",FnComments=COMMENTS,Pair=PAIR)

#ConvertToRGB32              # For v2.6 Colorspace              
Return Last

ReplaceDupes.avs (EDIT: Replace all detected dupes with previous non-dupe)

FN="WebCam.AVI"

AvisourcE(FN).Trim(0,0)
Audio = (HasAudio) ? KillVideo : 0
FrameSel(Cmd="FrameSelCmd.txt",Ordered=False)
(Audio.IsClip) ? AudioDub(Audio) : NOP
Return Last

NoDupes.avs (remove/delete all duplicates)

FN="WebCam.AVI"
Avisource(FN)
FrameSel(Cmd="FrameSelCmd.txt",Ordered=True)
Return Last

[StainlessS]

Lagarith File sizes Sorted in order for "LowBitrateAnime.mkv.AVI"
(I think may have come from Avisynth Usage/Consolidated list of test video clip resources as MKV)
EDIT: Had a look for this clip, found instance of it here: http://forum.doom9.org/showthread.ph...97#post1468097
1.77MB, 3:28 secs. (I converted to AVI lossless using ffmpeg)

Lagarith Nul Frames ONLY                     729,092,096 bytes
Dupped  Thresh=8                             563,208,192 bytes
Dupped  Thresh=16(Default)                   541,859,840 bytes
Dupped_MI Thresh=8, Ignore=0.0               538,779,648 bytes
Dupped_MI (Default)Thresh=8, Ignore=0.001    519,776,256 bytes)
Dupped_MI Thresh=16, Ignore=0.0              501,577,728 bytes
Dupped_MI Thresh=16, Ignore=0.001            476,020,736 bytes
Dupped_MI Thresh=8,  Ignore=0.01             455,946,240 bytes
Dupped_MI Thresh=16, Ignore=0.01             395,296,768 bytes

I was very surprised that Lagarith with NUL frames does quite a lot better (smaller) than UT_Video(fast decompress mode) or MagicUV.
(UTVideo/MagicUV figures not shown)

EDIT: The fixed version of original Dupped here:- http://forum.doom9.org/showthread.ph...84#post1716484

[Great Dragon]

Here is a short sample for you to test. Lags 34 mb.
https://dl.dropboxusercontent.com/u/...6_ep1_trim.zip

There are moving skies on a background where all "dupe" filters usually fails.

[StainlessS]

That is a difficult clip. Seems that what should be dupes are horizontally shifted a little bit (as can be seen when Stack=True)

Lagarith NUL frames                     34,048,000 bytes
Dupped Thresh=8                         34,048,000 bytes
Dupped (Def)Thresh=16                   34,048,000 bytes       
Dupped_MI (Def)Thresh=8 Ignore=0.001    34,048,000 bytes
Dupped_MI Thresh=8 Ignore=0.01          34,048,000 bytes
Dupped_MI Thresh=16 Ignore=0.001        34,048,000 bytes
Dupped_MI Thresh=16 Ignore=0.01         33,693,696 bytes
Dupped_MI Thresh=16 Ignore=0.1          33,001,472 bytes     
Dupped_MI Thresh=32 Ignore=0.4          23,226,368 bytes
Dupped_MI Thresh=50 Ignore=0.4          18,665,472 bytes

Almost no effect until quite savage settings used
(50,0.4 seems to work quite well, missed a couple of dupes, no mistake dupes detected, I think).

EDIT: Am quite impressed with the dupped detection method (well the fixed version) although the panThresh thing was
not terribly good, misses quite a few dupes that are easily detected using MI version 'Anchor frame'.
The MI version however requires strict linear access and play all the way though, The FrameSel option should allow
use with later temporal processing - non linear access.

You can use two instances of MI version to compare settings.

[Great Dragon]

You can use two instances of MI version to compare settings.

Yep, that's what I've used.
s1=AVISource("g:\_podw\pass6_ep1.avi").Dupped_MI(show=true,stack=true,Thresh=35,Ignore=0.1)
s2=AVISource("g:\_podw\pass6_ep1.avi").Dupped_MI(show=true,stack=true,Thresh=30,Ignore=0.001)
StackHorizontal(s1,s2)

[StainlessS]

Great Dragon,
I should have taken my own recommendation and viewed two instances at once.
I did not actually notice the 'moving sky' in your clip and thought Dupped_MI(50,0.4 ) was working pretty good, it was not.
Have updated Dupped_MI (not v2.58 version yet), added an Override that allows setting savage Thresh and Ignore without much danger.
Give it a try, I think it works pretty good now even with your moving sky's (clouds).

EDIT: I kept the 'savage' settings to show Override working.
EDIT: Maybe Override=0.95 instead of 1.0 for frames 111, 185 and 189.

[StainlessS]

Post #44, Dupped_MI() Updated, now for both v2.58 and v2.6 Avisynth, working great.
Changed defaults.

WebCam cap I420 codec about 11GB (640x480 25FPS, 25490 Frames, 16:59.6)
Compress to Lagarith with Null Frames about 1.97GB (EDIT: Only 10 exact dupes ie NULL frames).
After Dupped_MI(), 365 MB (EDIT: with NULL frames).
After NoDupes.avs, reduced to 4586 frames (EDIT: about same size ie 365MB), 3:03.44 (constant motion throughout).

[Great Dragon]

StainlessS, could you please give some explanation show to use 'Override' option properly? It seems I don't understand how to use those debug values correctly especially the FD parameter.

[StainlessS]

Using below

FN="pass6_ep1_trim.avi"     # Great Dragon Clip
THRESH   = 35               # Great Dragon clip ONLY
IGNORE   = 0.4              # Great Dragon clip ONLY
OVERRIDE = 0.95             # Great Dragon clip ONLY

IGNORE=0.4 ignores about 1 in 250 pixels.
There seems to be horizontal movement between what should (I think) be dupes, also reason for high Thresh.
If you step through, frame 5 should show MnMxDif=17 and FD=1.150, so this has detected dupe (MnMxDif=17)
but, should not be a dupe due to the moving clouds. We want to override this frame and so Override should be below
the FD=1.15.

Worst frames in your sample are 111(FD=0.986), 185(FD=1.011) and 189(FD=1.016).
So as the worst frame (111) is FD=0.986 you need an Override which is lower than FD, I chose 0.95.
If there are other frames in full sample that should not be dupes then set Override just lower than the shown
FD for that worst frame (frame with lowest FD that should be overridden to 'not a dupe').

Override is just a safety measure so that you can set quite ferocious THRESH and IGNORE without fear of making big mistakes.
I'm guessing that not many dupe detectors would work very well (if at all) with that clip,
thank you I shall keep a copy of it.

EDIT: If you keep your eye on the Character flags on top line as clip plays, you will see the Flag sequences
change as it changes from scene to scene (ie the dupe sequences are different in each scene making it very hard for most dupe detectors).

EDIT: If not detecting enough dupes, then raise Ignore and/or Thresh.
EDIT: Override should be set AFTER BOTH Thresh and Ignore.
EDIT: Can raise Thresh to about 50 (with same other settings).

[Great Dragon]

Thanks and I have a few more questions.
In the debug first line there is the abbreviation TOD. As far as I understand O stands for "Override", D stands for "Duplicate". But what is T stands for?
Sometimes I have a situation when picture MnMxDif is far lower than Thresh and Override too but it still doesn't detecting as ''dupe" and "T" just turns yellow.

Is it worth to stabilize clip before using dupe detector?
Is crop helps to better detect dupes?

[StainlessS]

Show (Default False) Shows Metrics if True
Metrics shown on top line are, character flags 'T','O', and 'D'. Where Hilited, 'T' means Thresh detected dupe, 'O' means
Dupe Overridden to non-dupe, and 'D' means using Dupe. Following the character flags, is the current dupe number ie 1
is first in a run of dupes, 2 is 2nd etc. and Following that is the currently greatest count of consecutive dupes in
any single run.

So 'T' means was detected as dupe by Thresh, however if 'O' is also hil-lited, then was Overridden to be NOT dupe.
The 'D' flag gives the final decision, will be hi-lited if 'T' but not 'O' is set hi-lited.

Sometimes I have a situation when picture MnMxDif is far lower than Thresh and Override too but it still doesn't detecting as ''dupe" and "T" just turns yellow.

Not sure what you meant there, was 'O' hilited as well, if so then was Overriden to not dupe.

EDIT:

Override (Default 1.0). Thresh detected Dupes are Overridden to non-dupe if RT_FrameDifference is greater than Override.
RT_FrameDifference is the average pixel difference between the Anchor Frame and current_frame. Suggest about 0.5 -> 1.5.

If overriding too many frames then Override is set too low, if not enough then Override is set too high (Override need to be just below
frame with moving clouds with lowest FD [RT_FrameDifference] as shown in metrics).

Give Thresh, Ignore and Override args and also displayed MnMxDif and FD.
I actually chose 50 as Thresh due to what I think are horizontally displaced frames (which may or may not be intended to be a dupe),
if you decide that these shifted frames are not dupes then probably a lower thresh would suffice, likewise if you do decide that they should be dupes,
you might want to stabilize (and crop) beforehand (but a higher Thresh eg 50 would probably produce exactly same result).

Hope that helps.

EDIT: 50 was just my first guess for Thresh, perhaps somewhere between 35 and 50 is probably more appropriate.

EDIT: It is also possible that setting ChromaWeight to 0.0 could produce better results where only luma difference would be used by
RT_FrameDifference when comparing with Override.

[Great Dragon]

Thanks, maybe I'll make another sample when I get home to show you what I've meant.
For me I've found that high Thresh with low Ignore works better. It just gives me more freedom. Because when Ignore is to high then MnMxDif changes in to low values and it's easy to get wrong dupes. Especially when moving objects are small.

[StainlessS]

It probably needs a little bit of practice to use for best results (for me too), I dont usually touch anime as I just don't have any real interest in it.

Just a little bit of extra info:-

The Subtracted frame is actually half scaled result Int((Anchor-current_frame)/2)+128 (I think), this is how Subtract works. Reason is that eg
255-0=255 and 0-255=-255 ie max possible range=510. The /2 + 128 takes into range 0->255 but does impose a little error in the resultant frame.

As far as YPlaneMin is concerned (and also simililar for YPlaneMax and YPlaneMinMaxDifference),
The function calculates the number of pixels that corresponds with percentage Threshold arg, we will call this pixlim.
It samples every pixel and updates a count array holding a pixel count for each luma level.
It then iterates through the array from 0 -> 255.
Something like:

    cnt=0
    result = 255
    for(i = 0 to 255){
        cnt=cnt + count_array[i]
#       if(cnt>=pixlim) {     # WRONG, Edited to below
        if(cnt>pixlim) {
            result=i
            exit i  # exit the loop with result
        }    
    }
    return result

The above is same as Ignore thing, it allows it to ignore (EDIT: up to) an amount of extreme noise pixels.
YPlaneMedian is actually implemented as YPlaneMin(Threshold=50.0)

EDITED above mistake, here is exact implementation in C (from conditional_functions.cpp)

  // Find the value we need.
  if (mode == MIN) {
    unsigned int counted=0;
    for (int i = 0; i< 256;i++) {
      counted += accum[i];
      if (counted>tpixels)
        return AVSValue(i);
    }
    return AVSValue(255);
  }

[StainlessS]

Additional to earlier posted NoDupes.avs.
Can cut out audio along with Dupes, if required (EDIT: eg remove static scenes from security cam footage that includes audio).

Supplied (along with SelectRanges.avs) in FrameSel.zip.

Function RejectRanges(clip c,String "SCmd",String "Cmd",Bool "TrimAudio",Float "FadeMS") {
    # RejectRanges() by StainlessS. Required:- FrameSel, Prune, RT_Stats
    # Wrapper to delete frames/ranges along with audio, can supply frames/ranges in SCmd string And/Or Cmd file.
    #   The wrapper makes for easier usage of Prune() which supports up to 256 input clips, but requires a clip index,
    #   eg '3, 100,200' would specify clip 3, range 100 to 200. The wrapper does away with the necessity for the clip index as we
    #   are only using a single clip here. Prune also does not have a 'reject' arg to delete specified frames rather than select them,
    #   this wrapper also converts a list of frames to delete into a list of frames to select so that we can use Prune and its audio
    #   capability.
    #
    # SCmd: Frames/Ranges specified in String (Frames/Ranges either Chr(10) or ';' separated, infix ',' specifies range, eg 'start,end').
    # Cmd:  Frames/Ranges specified in file (one frame/range per line, comments also allowed, see FrameSel for Further info).
    # TrimAudio:
    #   True(default), deletes audio belonging to deleted frames
    #   False, returns original audio, probably out of sync.
    # FadeMS: (default 1.0 millisec). Linear Audio Fade duration at splices when TrimAudio==true, 0 = dont fade (might result in audio 'clicks/cracks').
    c
    TrimAudio=Default(TrimAudio,True)   # default true trims audio, false returns original audio (audiodubbed, as Framesel returns no audio)
    FadeMS=Float(Default(FadeMS,1.0))   # 1 millisecond linear fadeout/fadein at splices
    PruneCmd = (TrimAudio) ? "~Prune_"+RT_LocalTimeString+".txt" : ""
    (!TrimAudio)
        \ ? FrameSel(scmd=SCmd,cmd=Cmd,reject=true)
        \ : FrameSel_CmdReWrite(PruneCmd,scmd=SCmd,cmd=Cmd,reject=true,Prune=True,range=true)
    (TrimAudio) ? Prune(Cmd=PruneCmd,FadeIn=True,FadeSplice=True,FadeOut=True,Fade=FadeMS) : NOP
    # If TrimAudio==true then delete Prune temp file, Else restore original Audio to the now audio-less clip
    (TrimAudio)
        \ ? RT_FileDelete(PruneCmd)
        \ : (c.HasAudio) ? AudioDub(c) : NOP
    Return Last
}

Client

FN="WebCam.AVI"
Avisource(FN)
RejectRanges(Cmd="FrameSelCmd.txt")
Return Last

[Great Dragon]

At last I have some time to work a little bit more with the script to find out how good it is. And it is really good. None of dupe detectors works as good as Dupped_MI. Especially with fades and Override feature helps a lot with it.

As I've promised early here there are files with examples where the script could fails a little bit more. Of course it's only in my case. It can be tuned to catch dupes almost in any situations. But in a video I work with most of dupes can be removed with this settings

Dupped_MI(Thresh=95, Ignore=0.01, Override=0.95)

And to catch dupes with small movement like blinking, small objects, moving clouds on a background etc. I have to lower override to around 0.5. But in this case all other dupes which don't need such low value would be skipped.

1. Blinking https://www.dropbox.com/s/fmi87fngdz...nking.avi?dl=0
2. More moving clouds https://www.dropbox.com/s/ngfwju9q66...louds.avi?dl=0
3. Small objects https://www.dropbox.com/s/9jsx0vtzp9...jects.avi?dl=0
4. More small objects https://www.dropbox.com/s/ekern8ysci...ects2.avi?dl=0

Of course it couldn't be universal settings but at least all those example could help to improve script accuracy.
For now to fix problem areas I isolate them with trim()

[StainlessS]

Glad you found a use for it.
Saw original author Corran on-site about 4:00AM this morning, I looked previously and he had not been on-site for some time (I think).
I shall implement a optional 'windowed' override (and maybe Thresh detect) at some point so that may assist with blinking eyes type probs.
EDIT:
Might also implement equivalent of Subtract(anchor,current).YPlaneMinMaxDifference but avoiding the Subtract() halfscale clip along with
its precision losses.


EDIT: I shall pick up your samples some time in next day or two (maybe today), thanx.

EDIT: Corran has updated links and first script today.
@Corran, If you come back again soon, suggest remove GetFrame() function [no need for it ie Trim(n,-1) trims a single frame
without all the other nonsense checks or function call], as shown in modded script here:- http://forum.doom9.org/showthread.ph...84#post1716484
Replace your script (or parts) with the mod, it is still your script as far as I'm concerned, attribution not required.

[Corran]

Thank you for the contributions StainlessS. I've updated the first post with the changes. Sorry you had to put up with my amateurish code. I'm glad to see this idea take new forms though. I tried using your new script (Dupped_MI) in Avisynth 2.5.8 but encountered an error with the RT_ColorSpace functions. AviSynth 2.6 appears to work fine however.