View Single Post
Old 1st April 2018, 21:08   #1  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Duplicity2/DropDeadGorgeous v2.13 - Dupe Tool - 17 Nov 2018

Post #1 of 2.
Code:
/*
    Duplicity2(), v2.13, by StainlessS @ Doom9:- https://forum.doom9.org/showthread.php?t=175357

    A two faced scheming and double dealing [on your behalf] dupe tool.

    Three modes of operation,
      0) Write duplicate Frames file only (as single frames or ranges).
      1) Replace duplicates with exact duplicates. (Can also write Frames file).
      2) Replace duplicates with Interpolated or Blended frames (control via MaxDuplen and MaxInterp). (Can also write Frames file).

    Req:-
      RT_Stats v2.00_Beta_11 + CallCmd (c) StainlessS,
      GSCript, Grunt, (c) Gavino,
      MaskTools2, MvTools2, (c) Manoa/Fizick/Pinterf & Others (Pinterf Versions).
      RemoveGrain, (c) Kassandro.

      GScript not needed if Avs+ [(c) Ultim, Pinterf and others.].
      If CallCmd available then will AutoDelete DBase on clip closure, otherwise requires manual deletion.

    ColorSpace:- All v2.6 standard PLANAR colorspaces except YV411, no support for YUY2, RGB24 or RGB32.

    Function Duplicity2(clip c,"Mode"=0,Float "ThG"=1.0,Float "ThB"=ThG*2.0, Int "ThP"=64,Float "IgP"=0.0,Int "ThK"=7,
        \ Int "MaxDupLen"=9,Int "LagMax"=MaxDupLen,Int "MaxInterp"=Min(9,MaxDupLen),
        \ int "BlkW"=64,int "BlkH"=BlkW,int "OLapX"=BlkW/2,Int "OLapY=BlkH/2",
        \ Int "X"=0,Int "Y"=0,Int"W"=0,Int "H"=0,Float "CW"=0.0,Bool "ChromaI"=False,Bool "AltScan"=False,
        \ Int "SPad"=16, Int "SSharp"=1, Int "SRFilter"=4,                             [* MSuper          *]
        \ Int "ABlkSize"=16, Int "AOverlap"=4,   Int "ASearch"=3,Int "ADct"=0,         [* MAnalyse        *]
        \ Int "RBlkSize"=8,  Int "ROverlap"=2,   Int "RthSAD"=100,                     [* MRecalculate    *]
        \ Float "Iml"=200.0, Bool "IBlend"=True, Int "IthSCD1"=400, Int "IthSCD2"=130, [* MFlowInter      *]
        \ Int "OthSCD1"=400,Int "OthSCD2"=130,                                         [* Dupe Detect Override @ Scene Change *]
        \ String "OverRide"="",String "Frames"="Frames.txt",Bool "Ranges"=True,Bool "FrameAsRange"=False,
        \ Bool "Show"=True,Bool "Verb"=True,Bool "ShowBlk"=True,Bool "Dim"=True,Bool "ShowDot"=False,
        \ Bool "InterpFast"=True,String "DBase"=""
        \ )

    Duplicity2() uses 3 RT_Stats measurement functions, RT_FrameDifference(), RT_FrameMovement(), and RT_LumaPixelsDifferent().
        the first two may difference using either luma only, or with a combined weighted chroma difference. Also see CW,
    and ChromaI args. RT_LumaPixelsDifferent() differences luma only.

        DeDuppers use some method to determine if a frame is a duplicate of its predecessor, a typical measuring method might be
    RT_FrameDifference (average absolute difference per pixel), unfortunately the result of such measure may be due to a small
    frame global change (lots and lots of pixels change by a small amount), or a smaller number of pixels change by a
    larger amount, both these cases could produce the exact same result. Another case may be that of a very small number of
    pixels change by a very large amount (eg a person blinks an eye with remainder of frame static) and such measure might seem
    to be a duplicate as it could produce much smaller numbers than a frame global change of a small amount.

        Duplicity2() uses a triple measurement system (maybe it should be called Triplicity), Primary measurement (RT_FrameDifference)
    to detect frames 'Unique' to their immediate predecessor (large or 'frame Global' changes), and second and third methods to
    determine if 'Non-Unique' frames are either 'Low Motion' or 'Duplicate' frames. The second method is a block differencing
    method (RT_FrameMovement) where it splits the frame up into BlkW by BlkH blocks and finds the maximum average pixel difference
    of any one block. The third measurement method is RT_LumaPixelsDifferent (result 0.0->255.0 where 255.0 is equivalent to 100.0%,
    user args for this method actually require Duplicity2 arg 'IgP' in range 0.0->100.0%, and also show results in range 0->100.0%).
        RT_LumaPixelsDifferent, compares frames and shows percent of pixels whose corresponding absolute luma pixel difference is
    greater than Int arg 'ThP', if this percent difference is greater than 'IgP', then it is deemed to be a 'Low motion' frame and if
    less or equal to 'IgP' then is a duplicate ('IgP' can be 0.0, meaning any pixels at all with difference greater than 'ThP' is a
    'Low Motion' Frame). Where both 'ThP' and 'IgP' are 0 and 0.0 respectively, will show percentage of pixels that are not Exactly the same.
    'IgP' could be set to 0.0 (Default) in which case the function is governed solely by any pixel differences exceeding 'ThP', or can set
    'IgP'= eg 0.001% meaning the it will disregard as noise 1 pixel difference exceeding 'ThP' in every 100,000 pixels
    [eg 1 pixel in every 316x316 region]. It is probably easiest to always use IgP of 0.0 and only adjust ThP.

    The function examines the DBase to determine if status for current frame is already known, if not already known then,
    the function determines if the current frame is a 'Unique' frame or not (via 'ThG' RT_FrameDifference to predecessor), if Unique,
    then the job is done and is considered a Non-Duplicate, these Unique frames are marked in a DBase as Unique. If a frame is
    'Non-Unique', then the function scans both backwards and forwards looking for 'Unique Start' and 'Unique End' frames (and if
    necessary, marking them in DBase). This Backwards/Forwards scanning permits jumping around in the clip for perusal and getting
    exact same results as if travelling forwards only (even backwards traversal only, should work just fine). When the
    'Unique Start' and 'Unique End' frames are established, we then know the extent of the unknown (unvisited) frames, and can scan
    and measure those with the second and third measurement methods (RT_FrameMovement and RT_LumaPixelsDifferent) to determine
    Low Motion/Dupe status. When determining Low Motion/Dupe status, we compare with an 'Anchor Frame', which at the beginning of a
    'Non-Unique' sequence is the 'Unique Start' frame. When scanning a non-unique sequence the 'Anchor Frame' is the predecessor frame
    to the very first frame in the current sub sequence of the 'Non-Unique' sequence, so potential duplicates are not necessarily
    compared with their immediate predecessor. When a frame is determined to be 'Low Motion' (not a dupe), any detected duplicates are
    written to the DBase and also to the output Frames file, and it switches to detecting sequences of 'Low Motion' frames within the
    'Non-Unique' sequence. On each detected 'Low Motion' frame, the Anchor frame for the next frame (whether it be another
    'Low Motion' frame or a new 'Dupe' frame) will be switched to that newly detected 'Low Motion' frame. With this method,
    all frames BETWEEN (but excluding) the 'Unique Start' and 'Unique End' in the 'Non-Unique' sequence are detected as perhaps
    multiple 'Low Motion' and 'Duplicate' sub sequences.
      Setting LagMax arg to some low number (min 1), will limit distance of the Anchor frame prior to the current frame, eg setting
    LagMax=1, would always do LoMo/Dupe compare with the current frame and its predecessor. A LagMax greater than 1 allows better detection
    of Low-Motion frames where there is only very slow creeping movement in the sequence.

      The above method allows for reasonably fast processing, where a 'unique' frame is encountered [where it is clearly different from
    its predecessor] but where not so clear, will examine the LoMo/dupe suspect frames more closely. ThG sets the boundary between
    'Unique' frames and non unique, and so has some effect on speed, setting too high for your source will slow processing speed as many
    non dupes will be examined more closely, and setting ThG too low for your source, may increase speed but may miss duplicates in noisy
    sources (probably better to set too high than too low).

      Although you might be able to scan backwards and get same detections, results are written to the Frames file in visited sequence and
    you should NOT jump about whilst writing the final frames file. Also, this is a Multi-Instance script, can use multiple instances
    at the same time (view side by side metrics) but you should really choose a unique 'Frames' filename for each instance.

    Args:-
        Mode,         Default 0, (Range 0->2)
                        0) Write Frames file only (where Frames file="" then not written so only Show=true does anything).
                        1) Make duplicate sequences Exact Dupes.
                        2) Interpolate/Blend Duplicates (depends upon MaxDupLen and MaxInterp).

        ThG,          Default 1.0 (0.0 < ThG <= 255.0) Suggest 1.0, never below 0.5.
                      Primary Global Unique detect threshold. [current_frame-1 <-> current_frame].
                      If using 255.0 will likely scan all of the way to both Start and End frames, and then
                      detect LoMo/Dupe frames of entire clip using the 2nd & 3rd detection methods, will seem to pause for a long time,
                      dont do it (no error if using daft numbers).

        ThB,          Default Float ThG * 2.0, (Range ThG <= ThB < 255.0, Suggested about 1.5->3.5).
                      Secondary Local LoMo/Dupe Block detect threshold, [Anchor<->current_frame]
                      Tune for source and Block Size. (For smaller block size, just by random chance there may be several noisy pixels
                      in duplicate frame block, that produce considerably higher metric than most other blocks).

        ThP,          Default Int 64. (Advised Range ThB < ThP <= 64, or 255=OFF)
                      Ternary Local [single pixel] LoMo/Dupe detect threshold,  [Anchor<->current_frame].
                        On a test with 256 artificial generated 640x480 identical frames of random coloured pixels, encoded at
                      MeGUI AVC CRF 23.0(default), RT_LumaPixelsDifferent(Thresh=1) found no frames where any single pixel was
                      different by more than 1 luma level when compared with the first encoded frame from that sequence. Mpeg2
                      and flv might not fare as well. This test was on a totally clean artificial source without artefacts
                      and such, you will likely need considerably higher values of ThP than 1.
                        I have seen multiple clips exhibiting some kind of minute (perhaps sub pixel) frame shift (both horizontal and
                      vertical) where frame should have been an exact duplicate, no idea to the cause.
                        This setting is an additional knob to twiddle, and you could turn it off and rely on ThG and ThB
                      completely [ThP=255, turns off]. The default of 64 and the below setting IgP=0.0, will consider any single pixel
                      with a luma difference of 64 or more, to be a non dupe, unfortunately any clip having above mentioned sub pixel
                      shifting at dupe frames, may need a high setting to not trigger a lomo detection, I have not I think
                      seen so far any clip that needed ThP more than about 48. 64 might seem ridiculously high but if frames are shifted,
                      then a little preposterous-ness is likely necessary, in a true non-dupe, there will most usually be at least
                      one pixel that differs in luma by more than 64.
                      We have thusly decided to make the default an emergency fallback detect value of 64, the user may decide to lower
                      this value or indeed turn it off completely[ThP=255].

        IgP,          Default 0.0, (Range 0.0 <= IgP < 100.0). Percent noise threshold for Ternary Local [pixel] LoMo/Dupe Detect,
                      Note, default 0.0% means that any frame with any single pixel with abs luma difference greater than ThP will
                      be detected as LoMotion.

        ThK,          Default = 7, (0 <= ThK <= 255). 0 = K Overrides off.
                      If RT_YPlaneMinMaxDifference(Threshold=0.2) of current_frame is less ThK then will Override any detection,
                      ie black (or narrow luma range) frames are ignored as not duplicates.
                      [Threshold=0.2 ignores as noise, 1 in 500 extreme pixels when establishing loose minimum and loose maximum & luma range].

        MaxDupLen,    Default 9. Maximum permitted detection length of duplicate run, If detected dupe run exceeds this setting,
                      then it is totally ignored and presumed to be a static scene.

        LagMax,       Default MaxDupLen. Setting to a low number (min 1) will limit the distance of the Anchor frame prior to
                      the current frame, eg setting LagMax=1, would always do LoMo/Dupe compare with the current frame and its
                      predecessor. Maybe best left alone.

        MaxInterp,    Default Min(MaxDupLen,9), (Range 0->9). For Mode=2(Interpolate/Blend mode) only. Max length of interpolated frames.
                        Sequences longer than this but less or equal to MaxDupLen will be blended instead.

        BlkW,         Default 64, 8 <= BlkW, Mod 2, Best Mod 4. Width of Blocks for Secondary Metric.
        BlkH,         Default BlkW, 8 <= BlkH, Mod 2, Best Mod 4. Height of Blocks for Secondary Metric.
        OLapX,        Default BlkW/2, 0 <= OLapX <= BlkW/2, Horizontal block overlap for Secondary Block metric.
        OLapY,        Default BlkH/2, 0 <= OLapY <= BlkH/2, Vertical block overlap for Secondary Block metric.
                         Where CW==0.0 and AltScan==True, might try eg OLapY=(BlkH/2)-1, where might be faster due to AltScan and,
                      also in overlapped 'Y_Block' scan will then scan alternate scanlines that were not scanned in previous non overlapped
                      scan (if that makes any sense, ie even scanlines sampled in first block, odd lines sampled in overlapped block scan).

        X,Y,W,H,      All Default 0, Coordinates of test area, as in crop (all 0 = full frame).
                      The x,y coords will be rounded up mod4, and w,h coords rounded down mod 4.
                      Ignore eg animated logo.

        CW            [ChromaWeight] Default 0.0. (0.0 <= CW <=1.0).
                      For Primary Frame Global and Secondary Block Mode difference detection, weighting of combined chroma diff to luma diff.
                        If required, recommend about 0.1, where difference would be weighted as
                        TotDif = ((1.0 - CW) * YDif) + (CW * ((UDif+VDif)/2.0))
                      Suggest always CW=0.0 where Night Capture.
See Post #22 for current script

Zip Includes DropDeadGorgeous()

For Zip, See Mediafire in sig below this post, or SendSpace.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 17th November 2018 at 15:26. Reason: Update
StainlessS is offline   Reply With Quote