MDec2 - v1.02 - 26 Apr 2018

[StainlessS]

dll's for Avisynth v2.58, Avs+v2.6 x86 and x64.
Requires CPP runtimes for VS 2008.


MDec2 by StainlessS

MDec2, MultiDecimate like decimating filter.

Planar, YUY2, RGB

MDEC2(clip c,float "rate"=24.0,bool "show"=false,bool "create"=false,string "dir"="C:\",bool "chroma"=true,bool "ver"=false)

MDec2 is a 2 pass decimating filter, acting much like the Multidecimate filter by Donald A. Graft.
Requires MultiDecimate.Exe & ProcessMD.Exe from MultiDecimate package.

Args:-

c, clip. No default

Rate=24.0, final output frame rate, used only on 2nd pass.

show=false. 1st pass, shows metrics on frame, 2nd pass shows frame being used.

create=false. Set true for 1st pass, false on 2nd pass after using MultiDecimate.Exe.

dir="C:\". Path to data files (mfile.txt, cfile.txt, dfile.txt).

chroma=true. If false then only use luma for metrics. (RGB not used).

ver=false. True shows version on frame.

See MultiDecimate docs for further info.


Metrics:-
MDec2, modified metrics based on FDecimate by Donald Graft.
Splits frames into blocks and returns metric of block with greatest difference.
YUV Metrics weight combined chroma with same weight as luma.

NOTE, if wanting to eg decimate 29.97 to 25.0, then numbers dont work and will throw an error because of
mismatch to what it is expecting. In such circomstance, use something like:-
>>>>>>>>>>>
SHOW=True
ORG=Last
AssumeFPS(30.0,sync_audio=True)
# ResampleAudio(ORG.AudioRate)
Mdec2(Rate=25.0,create=True,Show=True,Dir="") # Pass 1 [Dir="", files in current directory]
###
Use MultiDecimate.Exe, eg drop 1 frame in every 6.
######
SHOW=False
ORG=Last
AssumeFPS(30.0,sync_audio=True)
ResampleAudio(ORG.AudioRate)
Mdec2(Rate=25.0,create=False,Show=SHOW,Dir="") # Pass 2, output 25 fps.
<<<<<<<<<<<

StainlessS


Script included in zip, MDec2_MetricsTest.avs

########################################################################
# Script to test metrics of MultiDecimate, FDecimate and MDec2. #
# MultiDecimate v1.07 & FDecimate v1.02/v1.10 metrics are both bugged. #
########################################################################

#Loadplugin("MDec2_x86.dll")
Loadplugin("Multidecimate.dll")
#Loadplugin("FDecimate.dll")

MODE=1 # Testing Mode, 0 = MDec2 : 1=MultiDecimate : 2=FDecimate (All three should show metrics as 100.0%)
# 0) MDec2, any valid 8 bit ColorSpace. Metrics shown on frame. (only 100.0% @ PC Levels)
# 1) MultiDecimate YUY2 ONLY, Metrics stored script directory MFile.txt. (YUY2, Shows 108.51% Qual=2, 217.02% Qual=3)
# 2) FDecimate YV12 and YUY2, Metrics shown on frame. (Shows 187.68% for YV12, 112.61% YUY2)

C="YUY2" # Choose Test Colorspace (must be valid for the called dll)

MDQUAL=2 # MultiDecimate Quality setting default 2. 2=Luma Only : 3=Luma+Chroma

IsRGBClip=FindStr(C,"RGB")!=0

W=(IsRGBClip) ? BlankClip(Pixel_Type=C,Color=$FFFFFF) : BlankClip(Pixel_Type=C,Color_YUV=$FFFFFF)
K=(IsRGBClip) ? BlankClip(Pixel_Type=C,Color=$000000) : BlankClip(Pixel_Type=C,Color_YUV=$000000)
Interleave(W,K) # White/Black(RGB) or Pink/Green(YUV) alternate frames

MODE==0 ? MDec2(rate=25.0,create=true,Show=true,dir="",Ver=True,Chroma=true) : \
MODE==1 ? MultiDecimate(Pass=1,Show=True,quality=MDQUAL) : \
FDecimate(Rate=25.0,Metrics=true)
Return Last

EDIT: Above script modified on-line.

Zip contains dll's + source + FDecimate() original source. (metrics modified from FDecimate metrics).
Zip ~ 120KB.

See MediaFire or SendSpace below this post in my sig.

EDIT: Original MultiDecimate page gone, so also available here:- http://www.mediafire.com/file/bw4i71vf9h9wbmv/multidecimate107.zip
(maybe of use for docs, and MultiDecimate.exe and ProcessMD.exe).

[Marsu42]

See MediaFire in sig.

MediaFire link seems to be dead (404) - can you please re-upload? Thanks!

[StainlessS]

404 is a MediaFire problem, I have reported it. this should work: LINK REMOVED

EDIT: Link in sig now working OK.

[jmac698]

Can this be used to keep the 2nd copy of a set of repeating frames? There's a few cases for this. For one thing, imagine 24fps film encoded to 720p60. There's up to 3 repeated frames, but because of the encoding, the first in the sequence is low quality, and it's better to get the frame after, where the P frame is correcting the last I frame.

Another case is pulldown. I'd like to average the two repeating fields for better quality. Another use is dot crawl; if you average the same image on two frames it perfectly removes dot crawl, as if it were s-video. There was one other use I can't remember.

[StainlessS]

Sorry no. It is MultiDecimate.Exe that selects which frames to keep, MDEC2 only provides the metrics.
Source for MutiDecimate.Exe/Process.Exe was (I think) never published.

EDIT: Source for the dll was never published either.

[johnmeyer]

I am not sure I understand what problem this is solving. I use TFM all the time to create metrics in pass 1 which I then pass through Multidecimate.exe. In pass 2 I use "MultiDecimate(pass=2)" to read the metric files created by Multidecimate.exe (mfile.txt, etc.) and actually decimate the file by some arbitrary amount (i.e., non repeating patterns).

So what part of this workflow is MDec2 designed to replace, and what advantages does it have over existing solutions?

[StainlessS]

Hi john, I wrote it for myself for the YUY2 and YV12 and RGB capability, someone in some thread asked where to get source for Multidecimate,
so I (cleaned up and extended) posted what I had done. Multidecimate even in pass 2 does not support YV12 (EDIT: Or was it YUY2), even though all that is needed is return the next
non decimated frame without actually needing any knowledge of the colorspace (unless metrics required).

EDIT: You could continue to use TFM and Multidecimate.exe and just use MDEC2 for the second pass if you like (if YV12 or RGB).
I've no idea offhand what colorspaces TFM supports to provide metrics.

EDIT: Forgot, v2.6 planar compatible.

[johnmeyer]

Hi john, I wrote it for myself for the YV12 and RGB capability.Perfect. That explains it. I didn't see the thread that prompted this work. Thanks!

[StainlessS]

Posting prompted this: http://forum.doom9.org/showthread.php?p=1577816#post1577816

A bit more here: http://forum.doom9.org/showthread.php?p=1644669#post1644669

[StainlessS]

MDec2 v1.01, 30 Mar 2015. Recompile v2.6 dll with Avisynth Version 6 Header.
See 1st post.

[StainlessS]

MDec2() v1.02, 26 Apr 2018, new version.
Dll's for avs v2.58, avs+ v2.60 x86 and x64, see first post.

EDIT:

Modified by StainlessS, metrics from FDecimate by Donald A. Graft (Neuron2).

v1.00, 9 Aug 2013. Early release.
v1.01, 30 Mar 2015. Recompile v2.6 dll with Avisynth Version 6 Header.
v1.02, 26 Apr 2018. Fix pitch can change. Add x64 dll. Recompile VS2008.


Added MDec2_MetricsTest.avs showing bugs in Multidecimate and FDecimate metrics.
FDecimate metrics bug adds chroma difference to the wrong luma difference block.

Here FDEcimate on YV12 (v1.02, v1.10 same bug).
https://s20.postimg.cc/4qk5vhyr1/MDec2_Metrics_Test.jpg (https://postimages.org/)

MultiDecimate is also bugged but dont seem as bad. (no source to check)

[bruno321]

First, thanks for the work! But I can't figure out how to run it :confused:

I do:

tfm()
MDec2(create=true, dir="D:\myfolder")

Then I run the avs through megui and only the encoded video file comes out.... no txt files.

Perhaps this serves as a nudge to write an example or two here http://avisynth.nl/index.php/MDec2 :D

[StainlessS]

Dont use MeGUI on first pass (it does all kinds of weird multiple scan stuff).
You need scan complete clip in eg VDub2 [EDIT: Run Video Analysis Pass], on closure should produce result mfile.txt text file (same as Multidecimate).
Use the Multidecimate.Exe file to produce cfile.txt and dfile.txt files.
Run 2nd pass with create=false (and rate arg set, can use 2nd pass in MeGUI).
It is assumed that you have used multidecimate before [and have the MultiDecimate.Exe and ProcessMD.exe from Multidecimate].

EDIT: It also puts out error message to DebugView if cannot write file for some reason, see Debugview (Google).

EDIT: Multidecimate:- https://avisynth.org.ru/docs/english/externalfilters/multidecimate.htm

EDIT: DebugView:- https://docs.microsoft.com/en-us/sysinternals/downloads/debugview

[bruno321]

Thanks, that was useful. I've the three text files now (they look legit, with many lines of data), but when I try to preview the second pass I get (in avspmod):

MDEC2: incompatible number of frames.

script:
MPEGSource("file.d2v",cpu=0)
tfm()
MDec2(create=false, rate=20,dir="mypath")


(I used Decimation mode: remove duplicates, global, protect static scenes, threshold 1.0, run length 5)

FWIW I want to get rid of dupes in this file which are distributed haphazardly, in some sections I think a TDecimate(0,7,12) would do, but in others it has another repetition pattern which I suspect to be TDecimte(0,33,199) but I'm only 95% sure.

[StainlessS]

And what was the rest of the output to debugview ?


sprintf(buf,"MDEC2: mode=%d cycle=%d drop=%d num_iframes=%d num_oframes=%d\n",mode,cycle,drop,num_iframes,num_oframes);
OutputDebugString(buf);
sprintf(buf,"MDEC2: num_frames_hi=%d num_frames_lo=%d\n",num_frames_hi,num_frames_lo);
OutputDebugString(buf);
sprintf(buf,"MDEC2: Incompatible Number of frames\n");


EDIT: And what was framecount AFTER TFM, and FrameRate
Insert "Return Info" before Mdec2

EDIT: Oh, and by the way, I've only ever used with the Naive Multidecimate.exe option, no idea if works with the option you chose. [its actually multidecimate.exe that makes decision what to drop]

EDIT: It will throw a wobbly, if framecount after decimation is not exactly as expected for output rate.

[bruno321]

Framecount before or after tfm (no difference): 13960

Here's the output of debugview. First with the "Protect static scenes" option, then without:


[10324] MDEC2: Opening file D:\dfile-protect.txt
[10324] MDEC2: mode=1 cycle=5 drop=1 num_iframes=13960 num_oframes=7167
[10324] MDEC2: num_frames_hi=13960 num_frames_lo=6986
[10324] MDEC2: Incompatible Number of frames

[10324] MDEC2: Opening file D:\dfile.txt
[10324] MDEC2: mode=0 cycle=5 drop=1 num_iframes=13960 num_oframes=6025
[10324] MDEC2: num_frames_hi=13960 num_frames_lo=6986
[10324] MDEC2: Incompatible Number of frames


How should I choose the framerate? I don't know how to guess. Perhaps this is not the program I should be using? I want to remove the dupes that I talked about at the end of the previous post.

[StainlessS]

How should I choose the framerate? I don't know how to guess.
Me neither, you still did not say what input framerate is.

Sorry, I meant that I've only ever used it with "Cycle based Naive" (3rd one in list) not "Global Naive" [with threshold].
If say input framecount is 3000 @ 30.0 FPS, and remove 1 from cycle of 6 frames, so,
output framecount = 3000 * ((6-1) / 6) = 2500 : in_frame_count * ((cycle-1)/cycle)
output framerate = 30.0 * ((6-1) / 6) = 25.0 FPS

The check is basically so it can ensure that no cockup has been made, and is probably not necessary, but I prefer as is, as it has told me of my error more than once.
I dont like the other modes of Multidecimate, I only ever used the one mentioned for N in Cycle decimation, I actually wrote it (Mdec2) for myself as Multidecimate
supported only YUY2. Somebody asked about YV12 support for Multidecimate, so I posted my personal use tool, MDec2.

If you need arbitrary decimation where output rate or framecount is not predictable and audio sync is not required, then I guess you are stuck with Multidecimate's other modes or some other dedupper, sorry.

Suggestions in addition to TDecimate,
ExactDeDup(), where dupes are bit exact duplicates, single bit different is not a dupe, this should be very fast where many identical dupes.
Duplicity2(), good where difficult to detect duplicates (where will interpolate dupes, or create a dupes file of dupe frame numbers).

EDIT: Maybe I try find time to see if I can skip the check, "Incompatible number of Frames" (or whatever is was).

[StainlessS]

Bruno,
You wanna give this a go [MDEC2 v1.03Beta1] :- http://www.mediafire.com/file/cb6i7m3yscnjujc/MDec2_x86_x64_dll_v1.03Beta1_20200609.zip/file

Should now work with all four decimate modes of Multidecimate.
Rate arg now does nothing.
Dir arg default no longer 'C:\', dont work on Vista+, also '.\' current directory broken on vista+, both fixed with new default of current script directory, ie '.\'


Modified by StainlessS, metrics from FDecimate by Donald A. Graft (Neuron2).

v1.00, 9 Aug 2013. Early release.
v1.01, 30 Mar 2015. Recompile v2.6 dll with Avisynth Version 6 Header.
v1.02, 26 Apr 2018. Fix pitch can change. Add x64 dll. Recompile VS2008.
v1.03, 09 June 2020.
Changed default directory Dir=".\", ie current directory, old default "C:\" not good under Vista+.
Fix current directory relative. (Broken/changed under Vista/W7+).
Ignore Rate arg, calc new rate using vi.MulDivFPS(output_frame_count,input_frame_count).



MDec2 by StainlessS

MDec2, MultiDecimate like decimating filter.

Planar, YUY2, RGB, no change to audio.

MDEC2(clip c,float "rate"=24.0,bool "show"=false,bool "create"=false,string "dir"="C:\",bool "chroma"=true,bool "ver"=false)

MDec2 is a 2 pass decimating filter, acting much like the Multidecimate filter by Donald A. Graft.
Requires MultiDecimate.Exe & ProcessMD.Exe from MultiDecimate package.

Args:-

c, clip. No default

Rate=24.0, final output frame rate, used only on 2nd pass. ### v1.03, IGNORED, auto calculated from input framerate, input & output framecount.

show=false. 1st pass, shows metrics on frame, 2nd pass shows frame being used.

create=false. Set true for 1st pass, false on 2nd pass after using MultiDecimate.Exe.

dir=".\". Path to data files (mfile.txt, cfile.txt, dfile.txt). ### v1.03, Now '.\' current directory, old 'C:\' broken by Vista+, cant write to 'C:\'.

chroma=true. If false then only use luma for metrics. (RGB not used).

ver=false. True shows version on frame.

See MultiDecimate docs for further info.


V2.60 std colorspaces only. [Multidecimate YUY2 only]
Give it a whirl please.

EDIT: Also note,
Could use with Avs+ additional PLANAR colorspaces in Pass 2 (I think, untested, so long as SHOW=FALSE, ie no metrics on frame), and
use something like ConvertToYV12 for Pass 1.

EDIT:
PASS 1: (Create=true)

AviSource("D:\Parade.avi")
Assumefps(30.0,sync_audio=true)
Mdec2(create=true,show=true) # Show onframe metrics not necessary


Debugview

00000077 0.16522254 MDEC2: Dir FullPath=D:\DUPES\mfile.txt
00000078 0.16527906 MDEC2: create=true, Pass 1, create multidecimate file 'D:\DUPES\mfile.txt'
00000079 0.16536927 MDEC2: Pass 1 Alloc arrays OK
00000080 23.46486092 MDEC2: File write OK # After full clip scan
00000081 23.46486092 'D:\DUPES\mfile.txt'


Choosing Multidecimate.Exe Mode 2, 1 in 5 decimate.

PASS 2: (Create=false)

AviSource("D:\Parade.avi")
Assumefps(30.0,sync_audio=true)
Mdec2(show=true) # Show onframe metrics not necessary


Debugview, all from filter constructor

00000238 0.16752903 MDEC2: Dir FullPath=D:\DUPES\dfile.txt
00000239 0.16759570 MDEC2: create=false, Pass 2, reading decimation file 'D:\DUPES\dfile.txt'
00000240 0.16772106 MDEC2: From File: mode=2 cycle=5 remove=1 num_iframes=7373 num_oframes=5898
00000241 0.16777469 MDEC2: From Clip: num_frames=7373
00000242 0.16940221 MDEC2: Output 5898 frames at 23.998 FPS # input was not an exact multiple of Cycle(5) frames (might ideally be 24.0 FPS).
00000243 0.16945148 MDEC2: Pass 2 Read decimation file OK


EDIT:
23.998 FPS # input was not an exact multiple of Cycle(5) frames (might ideally be 24.0 FPS).
With longer clip (source was about 4 mins, 7373 frames), variation from ideal FPS would be lesser where source not multiple of cycle.

[bruno321]

Hey, thanks a lot for all this! Haven't yet looked at it in detail but the new version worked like a charm.

Input framerate: 25fps. Most of it had a 7-in-12 repetition pattern, so framerate would have been 12.488fps, but some short sections had a 33-in-199 patern, so with MDec2 I get a 12.935fps framerate which sounds sensible (in that it's slightly higher than 12.488) and looks good (some dupes remain but that's up to me to play with the threshold).

I also had a similar problem with another file but where I couldn't discern any pattern. Input framerate: 25fps. MDec2 outputs a 20.864fps framerate which looks good.

Weird framerates, I know, but these two sources are messed up.


One question: I don't understand when should I use the "cycle based naive" option from Multidecimate. If I know the pattern, then why use Multidecimate when I can do m-in-n decimation with TDecimate? My motivation for using MDec2 was precisely these cases when I can't easily discern a global pattern of duplication.

[StainlessS]

Main benefit of MDec2/Multidecimate (that I see), is that you can do Pass 1 scan once early on in complex restoration script (before adding any denoise/despot/descratch or whatever)
and later tinkering with script does not involve decimation scan every time you tweak a setting.
Prior to MDec2 Pass 1, almost no filters before and DEFO NONE AFTERWARDS, I guess acceptable filters before are eg
Tfm, AssumeFPS, and similar.
Can add additional filters following MDec2 where Pass = 2 (Create=false).

MDec2/Multidecimate Pass=2, is very light-weight, very little overhead in script.

EDIT: I guess that the best reasons to use MDec2 may reside in Multidecimate docs (I have not looked at them recently), also examples are likely therein.

[bruno321]

I'm back to tinkering with MDec2. My new question is, how do I generate from this usage of MDec2 a timecodes file to use when muxing to mkv so as to generate a VFR file and not lose audio synchronicity?

[StainlessS]

Sorry, I have no idea, I dont touch VFR, and esp dislike NTSC weird framerates.
Perhaps look into Multidecimate, Mdec2 uses exact same files mfie.txt, dfile.txt and cfile.txt (I think).
I have pretty much no spare time for next few months. (just looking in on the odd posts).

[poisondeathray]

dedup can decimate and generate VFR timecodes

There is only x86 version, but it works in avs+ x64 with mp_pipeline