HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Duplicity2/DropDeadGorgeous v2.13 - Dupe Tool - 17 Nov 2018
Post #1 of 2.
Code:
/*
Duplicity2(), v2.13, by StainlessS @ Doom9:- https://forum.doom9.org/showthread.php?t=175357
A two faced scheming and double dealing [on your behalf] dupe tool.
Three modes of operation,
0) Write duplicate Frames file only (as single frames or ranges).
1) Replace duplicates with exact duplicates. (Can also write Frames file).
2) Replace duplicates with Interpolated or Blended frames (control via MaxDuplen and MaxInterp). (Can also write Frames file).
Req:-
RT_Stats v2.00_Beta_11 + CallCmd (c) StainlessS,
GSCript, Grunt, (c) Gavino,
MaskTools2, MvTools2, (c) Manoa/Fizick/Pinterf & Others (Pinterf Versions).
RemoveGrain, (c) Kassandro.
GScript not needed if Avs+ [(c) Ultim, Pinterf and others.].
If CallCmd available then will AutoDelete DBase on clip closure, otherwise requires manual deletion.
ColorSpace:- All v2.6 standard PLANAR colorspaces except YV411, no support for YUY2, RGB24 or RGB32.
Function Duplicity2(clip c,"Mode"=0,Float "ThG"=1.0,Float "ThB"=ThG*2.0, Int "ThP"=64,Float "IgP"=0.0,Int "ThK"=7,
\ Int "MaxDupLen"=9,Int "LagMax"=MaxDupLen,Int "MaxInterp"=Min(9,MaxDupLen),
\ int "BlkW"=64,int "BlkH"=BlkW,int "OLapX"=BlkW/2,Int "OLapY=BlkH/2",
\ Int "X"=0,Int "Y"=0,Int"W"=0,Int "H"=0,Float "CW"=0.0,Bool "ChromaI"=False,Bool "AltScan"=False,
\ Int "SPad"=16, Int "SSharp"=1, Int "SRFilter"=4, [* MSuper *]
\ Int "ABlkSize"=16, Int "AOverlap"=4, Int "ASearch"=3,Int "ADct"=0, [* MAnalyse *]
\ Int "RBlkSize"=8, Int "ROverlap"=2, Int "RthSAD"=100, [* MRecalculate *]
\ Float "Iml"=200.0, Bool "IBlend"=True, Int "IthSCD1"=400, Int "IthSCD2"=130, [* MFlowInter *]
\ Int "OthSCD1"=400,Int "OthSCD2"=130, [* Dupe Detect Override @ Scene Change *]
\ String "OverRide"="",String "Frames"="Frames.txt",Bool "Ranges"=True,Bool "FrameAsRange"=False,
\ Bool "Show"=True,Bool "Verb"=True,Bool "ShowBlk"=True,Bool "Dim"=True,Bool "ShowDot"=False,
\ Bool "InterpFast"=True,String "DBase"=""
\ )
Duplicity2() uses 3 RT_Stats measurement functions, RT_FrameDifference(), RT_FrameMovement(), and RT_LumaPixelsDifferent().
the first two may difference using either luma only, or with a combined weighted chroma difference. Also see CW,
and ChromaI args. RT_LumaPixelsDifferent() differences luma only.
DeDuppers use some method to determine if a frame is a duplicate of its predecessor, a typical measuring method might be
RT_FrameDifference (average absolute difference per pixel), unfortunately the result of such measure may be due to a small
frame global change (lots and lots of pixels change by a small amount), or a smaller number of pixels change by a
larger amount, both these cases could produce the exact same result. Another case may be that of a very small number of
pixels change by a very large amount (eg a person blinks an eye with remainder of frame static) and such measure might seem
to be a duplicate as it could produce much smaller numbers than a frame global change of a small amount.
Duplicity2() uses a triple measurement system (maybe it should be called Triplicity), Primary measurement (RT_FrameDifference)
to detect frames 'Unique' to their immediate predecessor (large or 'frame Global' changes), and second and third methods to
determine if 'Non-Unique' frames are either 'Low Motion' or 'Duplicate' frames. The second method is a block differencing
method (RT_FrameMovement) where it splits the frame up into BlkW by BlkH blocks and finds the maximum average pixel difference
of any one block. The third measurement method is RT_LumaPixelsDifferent (result 0.0->255.0 where 255.0 is equivalent to 100.0%,
user args for this method actually require Duplicity2 arg 'IgP' in range 0.0->100.0%, and also show results in range 0->100.0%).
RT_LumaPixelsDifferent, compares frames and shows percent of pixels whose corresponding absolute luma pixel difference is
greater than Int arg 'ThP', if this percent difference is greater than 'IgP', then it is deemed to be a 'Low motion' frame and if
less or equal to 'IgP' then is a duplicate ('IgP' can be 0.0, meaning any pixels at all with difference greater than 'ThP' is a
'Low Motion' Frame). Where both 'ThP' and 'IgP' are 0 and 0.0 respectively, will show percentage of pixels that are not Exactly the same.
'IgP' could be set to 0.0 (Default) in which case the function is governed solely by any pixel differences exceeding 'ThP', or can set
'IgP'= eg 0.001% meaning the it will disregard as noise 1 pixel difference exceeding 'ThP' in every 100,000 pixels
[eg 1 pixel in every 316x316 region]. It is probably easiest to always use IgP of 0.0 and only adjust ThP.
The function examines the DBase to determine if status for current frame is already known, if not already known then,
the function determines if the current frame is a 'Unique' frame or not (via 'ThG' RT_FrameDifference to predecessor), if Unique,
then the job is done and is considered a Non-Duplicate, these Unique frames are marked in a DBase as Unique. If a frame is
'Non-Unique', then the function scans both backwards and forwards looking for 'Unique Start' and 'Unique End' frames (and if
necessary, marking them in DBase). This Backwards/Forwards scanning permits jumping around in the clip for perusal and getting
exact same results as if travelling forwards only (even backwards traversal only, should work just fine). When the
'Unique Start' and 'Unique End' frames are established, we then know the extent of the unknown (unvisited) frames, and can scan
and measure those with the second and third measurement methods (RT_FrameMovement and RT_LumaPixelsDifferent) to determine
Low Motion/Dupe status. When determining Low Motion/Dupe status, we compare with an 'Anchor Frame', which at the beginning of a
'Non-Unique' sequence is the 'Unique Start' frame. When scanning a non-unique sequence the 'Anchor Frame' is the predecessor frame
to the very first frame in the current sub sequence of the 'Non-Unique' sequence, so potential duplicates are not necessarily
compared with their immediate predecessor. When a frame is determined to be 'Low Motion' (not a dupe), any detected duplicates are
written to the DBase and also to the output Frames file, and it switches to detecting sequences of 'Low Motion' frames within the
'Non-Unique' sequence. On each detected 'Low Motion' frame, the Anchor frame for the next frame (whether it be another
'Low Motion' frame or a new 'Dupe' frame) will be switched to that newly detected 'Low Motion' frame. With this method,
all frames BETWEEN (but excluding) the 'Unique Start' and 'Unique End' in the 'Non-Unique' sequence are detected as perhaps
multiple 'Low Motion' and 'Duplicate' sub sequences.
Setting LagMax arg to some low number (min 1), will limit distance of the Anchor frame prior to the current frame, eg setting
LagMax=1, would always do LoMo/Dupe compare with the current frame and its predecessor. A LagMax greater than 1 allows better detection
of Low-Motion frames where there is only very slow creeping movement in the sequence.
The above method allows for reasonably fast processing, where a 'unique' frame is encountered [where it is clearly different from
its predecessor] but where not so clear, will examine the LoMo/dupe suspect frames more closely. ThG sets the boundary between
'Unique' frames and non unique, and so has some effect on speed, setting too high for your source will slow processing speed as many
non dupes will be examined more closely, and setting ThG too low for your source, may increase speed but may miss duplicates in noisy
sources (probably better to set too high than too low).
Although you might be able to scan backwards and get same detections, results are written to the Frames file in visited sequence and
you should NOT jump about whilst writing the final frames file. Also, this is a Multi-Instance script, can use multiple instances
at the same time (view side by side metrics) but you should really choose a unique 'Frames' filename for each instance.
Args:-
Mode, Default 0, (Range 0->2)
0) Write Frames file only (where Frames file="" then not written so only Show=true does anything).
1) Make duplicate sequences Exact Dupes.
2) Interpolate/Blend Duplicates (depends upon MaxDupLen and MaxInterp).
ThG, Default 1.0 (0.0 < ThG <= 255.0) Suggest 1.0, never below 0.5.
Primary Global Unique detect threshold. [current_frame-1 <-> current_frame].
If using 255.0 will likely scan all of the way to both Start and End frames, and then
detect LoMo/Dupe frames of entire clip using the 2nd & 3rd detection methods, will seem to pause for a long time,
dont do it (no error if using daft numbers).
ThB, Default Float ThG * 2.0, (Range ThG <= ThB < 255.0, Suggested about 1.5->3.5).
Secondary Local LoMo/Dupe Block detect threshold, [Anchor<->current_frame]
Tune for source and Block Size. (For smaller block size, just by random chance there may be several noisy pixels
in duplicate frame block, that produce considerably higher metric than most other blocks).
ThP, Default Int 64. (Advised Range ThB < ThP <= 64, or 255=OFF)
Ternary Local [single pixel] LoMo/Dupe detect threshold, [Anchor<->current_frame].
On a test with 256 artificial generated 640x480 identical frames of random coloured pixels, encoded at
MeGUI AVC CRF 23.0(default), RT_LumaPixelsDifferent(Thresh=1) found no frames where any single pixel was
different by more than 1 luma level when compared with the first encoded frame from that sequence. Mpeg2
and flv might not fare as well. This test was on a totally clean artificial source without artefacts
and such, you will likely need considerably higher values of ThP than 1.
I have seen multiple clips exhibiting some kind of minute (perhaps sub pixel) frame shift (both horizontal and
vertical) where frame should have been an exact duplicate, no idea to the cause.
This setting is an additional knob to twiddle, and you could turn it off and rely on ThG and ThB
completely [ThP=255, turns off]. The default of 64 and the below setting IgP=0.0, will consider any single pixel
with a luma difference of 64 or more, to be a non dupe, unfortunately any clip having above mentioned sub pixel
shifting at dupe frames, may need a high setting to not trigger a lomo detection, I have not I think
seen so far any clip that needed ThP more than about 48. 64 might seem ridiculously high but if frames are shifted,
then a little preposterous-ness is likely necessary, in a true non-dupe, there will most usually be at least
one pixel that differs in luma by more than 64.
We have thusly decided to make the default an emergency fallback detect value of 64, the user may decide to lower
this value or indeed turn it off completely[ThP=255].
IgP, Default 0.0, (Range 0.0 <= IgP < 100.0). Percent noise threshold for Ternary Local [pixel] LoMo/Dupe Detect,
Note, default 0.0% means that any frame with any single pixel with abs luma difference greater than ThP will
be detected as LoMotion.
ThK, Default = 7, (0 <= ThK <= 255). 0 = K Overrides off.
If RT_YPlaneMinMaxDifference(Threshold=0.2) of current_frame is less ThK then will Override any detection,
ie black (or narrow luma range) frames are ignored as not duplicates.
[Threshold=0.2 ignores as noise, 1 in 500 extreme pixels when establishing loose minimum and loose maximum & luma range].
MaxDupLen, Default 9. Maximum permitted detection length of duplicate run, If detected dupe run exceeds this setting,
then it is totally ignored and presumed to be a static scene.
LagMax, Default MaxDupLen. Setting to a low number (min 1) will limit the distance of the Anchor frame prior to
the current frame, eg setting LagMax=1, would always do LoMo/Dupe compare with the current frame and its
predecessor. Maybe best left alone.
MaxInterp, Default Min(MaxDupLen,9), (Range 0->9). For Mode=2(Interpolate/Blend mode) only. Max length of interpolated frames.
Sequences longer than this but less or equal to MaxDupLen will be blended instead.
BlkW, Default 64, 8 <= BlkW, Mod 2, Best Mod 4. Width of Blocks for Secondary Metric.
BlkH, Default BlkW, 8 <= BlkH, Mod 2, Best Mod 4. Height of Blocks for Secondary Metric.
OLapX, Default BlkW/2, 0 <= OLapX <= BlkW/2, Horizontal block overlap for Secondary Block metric.
OLapY, Default BlkH/2, 0 <= OLapY <= BlkH/2, Vertical block overlap for Secondary Block metric.
Where CW==0.0 and AltScan==True, might try eg OLapY=(BlkH/2)-1, where might be faster due to AltScan and,
also in overlapped 'Y_Block' scan will then scan alternate scanlines that were not scanned in previous non overlapped
scan (if that makes any sense, ie even scanlines sampled in first block, odd lines sampled in overlapped block scan).
X,Y,W,H, All Default 0, Coordinates of test area, as in crop (all 0 = full frame).
The x,y coords will be rounded up mod4, and w,h coords rounded down mod 4.
Ignore eg animated logo.
CW [ChromaWeight] Default 0.0. (0.0 <= CW <=1.0).
For Primary Frame Global and Secondary Block Mode difference detection, weighting of combined chroma diff to luma diff.
If required, recommend about 0.1, where difference would be weighted as
TotDif = ((1.0 - CW) * YDif) + (CW * ((UDif+VDif)/2.0))
Suggest always CW=0.0 where Night Capture.
See Post #22 for current script
Zip Includes DropDeadGorgeous()
For Zip, See Mediafire in sig below this post, or SendSpace.
Last edited by StainlessS; 17th November 2018 at 15:26.
Reason: Update
|