ExactDeDup Update v0.06 - 15 Feb 2019

[StainlessS]

Here mod of original ExactDeDup,

posting in new thread just so is easier to find, and if (and that is a big long if) I do another update, then is in same place as this one.

Original thread on another forum here:- http://tasvideos.org/forum/viewtopic.php?t=12065

updated for avs 2.58, avs/+ v2.60 x86 and x64, req VS 2008 runtimes.

doc in brief,

ExactDedup v0.06
Copyright (c) 2011 Steve Melenchuk, Arick Chan
==============================================

0. Table of Contents
--------------------
  1. Introduction
  2. Usage
  3. Licensing

1. Introduction
---------------
ExactDedup is an AVISynth 2.58/v2.60/+ x86 & x64 plugin providing a filter intended to remove
frames that are EXACT duplicates of each other, leaving only the first and
(optionally) last frames of a run intact, and generates a Matroska v2
timecodes file with timing information for the ensuing stream.
NOTE, 'exact duplicate' means exactly that, ie completely and bit for bit identical.

Due to the particulars of the AVISynth API, ExactDedup requires two passes
to function efficiently. It could theoretically be designed to run with
one pass, but this would result in a very long startup delay while the
plugin precomputes all of the duplicate frame information and timecodes.

ExactDedup supports any standard v2.60 input colorspace.

You will need the Visual C++ 2008 runtimes (i.e. MSVCR90.dll) to be able
to run this plugin.

2. Usage
--------

ExactDedup(clip [, bool firstpass] [, string dupinfo], [, string times]
           [, int maxdupcount] [, bool keeplastframe] [,bool show])

Options:
- firstpass    : Default true. Controls which pass the plugin is running.

                 The first pass generates a mapping of output frames to
                 input frames (stored in the file specified by dupinfo) and
                 Matroska v2 timecodes (stored in the file specified by
                 times); seeking during this pass is impossible.
                 If not completely linear access from beginning to end of clip, then
                 will abandon processing, you cannot jump around the clip, and should not
                 use any multi processing (MT) methods.

                 The second pass generates a video stream containing only
                 the frames specified in the dupinfo file. (Seeking is
                 permissible during this pass.)

- dupinfo      : Default "dupinfo.txt". The name of a temporary file
                 generated by the first pass and read by the second pass;
                 each line corresponds to a frame in the output video
                 stream, and specifies the frame number of the frame in
                 the input stream that will be used as that output frame.
                 Note, Dupinfo/Times files deleted before pass one commences,
                 sends a message to the debug stream when dupinfo successfully written (Google DebugView). EDIT: On last frame.

- times        : Default "times.txt". The name of the Matroska v2 timecodes
                 file that will be output by the first pass.

- maxdupcount  : Default 20. The number of maximum consecutive duplicate
                 frames corresponding to a single frame of the output; i.e.
                 a maximum of (maxdupcount - 1) frames will be removed from
                 a run of duplicates before another frame is kept.

- keeplastframe: Default false. Keeps the last frame of a run of duplicates.
                 If a run of duplicates is larger than maxdupcount, this
                 keeps only the last frame of the entire run, not of the
                 individual maxdupcount-size blocks.

- Show         : Default false. If True, shows a little info on second pass.



v0.06 (15 Feb 2019) (StainlessS)
    Moved to VS2008, Added Version resource + x64. Req VS2008 Runtimes. Fixed M$ broken relative pathnames.

EDIT: Above bug from original version, bool _keeplastframe, should read bool keeplastframe, fixed above but not in zip Readme.txt.

See Mediafire or SendSpace below this post.

EDIT: Link to previous version on Wiki is now invalid. [To Wiki editor, I'll leave in MediaFire StainlessS folder, point link at folder].

[Reel.Deel]

Originally Posted by StainlessS

EDIT: Link to previous version on Wiki is now invalid. [To Wiki editor, I'll leave in MediaFire StainlessS folder, point link at folder].

Fixed

[StainlessS]

Cheers RD

[goorawin]

Hi StainlessS
I would like to render many files with duplicate frames.
Is there a simple way to name the temp files (dupinfo="clip2.txt", times="times2.txt") using the clip name instead, so that more than one file can be rendered at the same time?

This is an example of the script I have used.
SetMemoryMax(2048)
Clip=("D:\Testing\Part01.mp4")
LWLibavVideoSource(clip)
A1=LWLibavAudioSource(Clip)
SCRIPT1=ExactDedup(firstpass=true, dupinfo="clip2.txt", times="times2.txt", maxdupcount=2, show=false)
RT_ForceProcess(SCRIPT1)
SCRIPT1 = 0
ExactDedup(firstpass=false, dupinfo="clip2.txt", times="times2.txt", maxdupcount=2, show=false)
BicubicResize(1920,1080)
AudioDub(Last, A1)
AssumeFPS(25)

Thanks in advance

[StainlessS]

Use this as an Avisynthesizer_MOD template

#ASYNTHER BATCH_Goorawin
########################### Above uses name "BATCH_Goorawin" in Avisynthesizer Dialog box.

FN = "___FILE___"               # No square brackets, is Batch Mode Avisynthesizer template (Avisynthesizer_Mod inserts filename here)

#FN= ".\Parade.avi"             # Use this only for my testing, then comment out

SetMemoryMax(2048)              # This is probably too Much, 1GB usually more than enough or just let it default.

FN = RT_GetFullPathName(FN)    # EDIT: Probably not needed, should (I think) insert full pathname already.
INFO_FN  = FN + "_INFO.Txt"
TIMES_FN = FN + "_TIMES.Txt"

LWLibavVideoSource(FN)
Audio=LWLibavAudioSource(FN)

PASS1=ExactDedup(firstpass=true, dupinfo=INFO_FN, times=TIMES_FN, maxdupcount=2, show=false)
RT_ForceProcess(PASS1)
PASS1 = 0
ExactDedup(firstpass=false, dupinfo=INFO_FN, times=TIMES_FN, maxdupcount=2, show=false)
BicubicResize(1920,1080)
AudioDubEx(Audio)
AssumeFPS(25)

Right click select many source files, "SendTo Avisynthesizer", and then select "BATCH_Goorawin", will auto create an instance of avs script for each selected source file.

Avisynthesizer_MOD
https://forum.doom9.org/showthread.p...ynthesizer_Mod

EDIT: Above script needs to have an *.avst extension in Avisynthesizer_Mod template directory.

[goorawin]

Thank you so much StainlessS.
It works really well, so will certainly make life easier.

[mbcd]

Silly question, but does it need deinterlaced input, or can it handle interlaced stuff?

I am asking, because I deinterlace first and it removes some frames, except those ones which are duplicates, they are kept.
I have a max of two identical frames (most around scenchanges), the last of each group has to be removed.
The duplicated are not 100% exact, some pixel vary al very little bit (brighter, darker), because of compression, but it is hard to see a visble difference betewwn them.

Cant find out if there is a little range that dups are allowed to be different.

[StainlessS]

ExactDedup will only detect Exact dupes (hence the name), indeed if it finds a single pixel different from prev frame, it immediately ends its frame scan, and returns with a not dupe status.
Interlacing makes no difference to result.

EDIT: ExactDedup probably works best/fastest with Lagarith lossless codec (which can compress to nul frames[fast] where exact dupes).
If you have clip with many exact dupes, Lagarith does surprisingly well at compressing them, better than UT_Video and also MagicUV (at least for older versions, I use only XP compatible versions).

[AdamCarter]

This script is awesome.

I wonder.....
Could you make a script similar to this
Where instead of looking for dupes within the source
It looks for dupes within another source and replaces them with the second source.

Let me give you an example

I have a cartoon which was encoded at a lower quality but has english audio
I have the same cartoon which has a dvd version which i can encode at higher quality, but only has japanese or french audio.

The audio doesnt exactly match up, as the videos have very slightly different cuts, and different intermission bumpers for the different regions.

I thought about grabbing the frame by hand and playing the matching game, but you are talking about 30k frames per episode x 52 episodes.


And for exporting, If the script cant find a match, if i export as frames maybe an option would be cool to leave the original frame with no matches and perhaps have an asterisk in the file name so as to do any cleanup or frame fixing later on or a log file could be created noting all the frames with no higher quality match.. And if i export to video then again just leave the original frame if no match could be found from the higher quality source, and again have a log.

[nji]

Hi StainlessS.

Thankyou for your work.

I don't want to miss something so I've got to ask:
I still don't understand the meaning of _keeplastframe.

Does it mean that - in addition to the first frame of a exact run -
it keeps also the last one?
Which would mean it keeps 1 dupe. But what for?
And if the run is larger than maxdupcount size,
then its behaviour is different again by "deactivating" maxdupcount
and keeping only one (= the last) frame of the entire run?
I can't imagine what this could be for...
Can you give an example please?

[StainlessS]

Hi nji
Note from 1st post

EDIT: Above bug from original version, bool _keeplastframe, should read bool keeplastframe, fixed above but not in zip Readme.txt.

Sorry, I dont really use ExactDeDup, and only updated for avs v2.60/Avs+ and x64 versions,
not sure, think somebody requested x64 ver$, and so I just did recompile with a few fixes.

Perhaps someone who is familiar with ExactDeDup usage could advise nji, if not then I guess I'll havta figure it out.
Thanks.

EDIT: I do remember reading myself about keeplastframe and not knowing what it was meaning.

[StainlessS]

ExactDedup v0.7BETA, x86 v2.58, v2.60 avs/+, x64 Avs+.
https://www.mediafire.com/file/xoqhb...10604.zip/file

v0.07 (04 June 2021) (StainlessS)
    Switched to DDigit v2.0 Metrics renderer for Show=true.
    Added HBD for Avs+ [all colorspaces, Avs+ frame Properties NOT supported nor passed to next filter]

v0.06 (15 Feb 2019) (StainlessS)
	Moved to VS2008, Added Version resource + x64. Req VS2008 Runtimes. Fixed M$ broken relative pathnames.

Need some testers. [I've never touched any VFR stuff].

EDIT:

FN="D:\Parade.avi" # JohnMeyer Parade.avi
AviSource(FN)
Assumefps(30)
ChangeFPS(60)   # Every 2nd frame an EXACT dupe
ExactDedup(firstpass=true)  # first pass scan detect EXACT dupes

AVSMeter 3.0.8.0 (x86), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.0 (r3382, 3.7, i386) (3.7.0.0)

Number of frames:                    14746
Length (hh:mm:ss.ms):         00:04:05.767
Frame width:                           488
Frame height:                          360
Framerate:                          60.000 (60/1)
Colorspace:                           YV12
Audio channels:                          2
Audio bits/sample:                      16
Audio sample rate:                   44100
Audio samples:                    10849148

Frames processed:                   14746 (0 - 14745)
FPS (min | max | average):          855.6 | 2051 | 1597
Process memory usage (max):         40 MiB
Thread count:                       9
CPU usage (average):                28.5%

Time (elapsed):                     00:00:09.233

Nothing exotic, just Win7 x64, and Core 2 Quad Q9550 2.83 GHz. SATA 3 physical drive.

EDIT:

v0.07 (04 June 2021) (StainlessS)
Switched to DDigit v2.0 Metrics renderer for Show=true.
Added HBD for Avs+ [all colorspaces, Avs+ frame Properties NOT supported nor passed to next filter]

Actually, Pass 1 or pass 2, we dont create a new frame so frame properties should be passed on untouched.
[Even if Pass 2 and Show=True, we use dont make new frame only MakeWritable, which I think will also pass on frame properties].

[kedautinh12]

Thanks

[Hotte]

StainlessS.

I have just tested it with YUV422P10 which is native 10-bit HBD (AVS+ 64bit) and it worked perfectly with Full-HD and 4K formats.

You really saved my day (more than the day to be honest)!

Just to remember that you can run ExactDedup virtually in a one-pass mode if you code like this:

DUPTXT  = "C:\Temp\dupinfo.txt"
DUPTXT  = RT_GetFullPathName(DUPTXT)
TIMETXT = "C:\Temp\times.txt"
TIMETXT = RT_GetFullPathName(TIMETXT)
AudioDub(LSMASHVideoSource("C:\myclip.mov",format="YUV422P10"),LSMASHAudioSource("C:\myclip.mov"))
ExactDedup(firstpass=true, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)
RT_ForceProcess()
Last=0

ExactDedup(firstpass=false, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)
Return(last)

Have fun!

[kedautinh12]

Wow, thank Hotte

[StainlessS]

Hotte,

RT_ForceProcess()
Last=0

I dont think the Last=0 is necessary. Only necessary for TWriteAVI which returns a clip, RT_ForceProcess()
does not return a clip and so no need to close the returned clip, B'cos there aint one.

EDIT: TWriteAVI's ForceProcessAVI() filter.

EDIT: Also hotte,
By nji

I still don't understand the meaning of keeplastframe.

Can you answer his query, also for maxdupcount.
[I looked at the code and thought, "I just dont care", not sure what its supposed to achieve,].
Docs could be clearer.

EDIT:

ExactDedup(firstpass=false, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)

maxdupcount=0 also switches it OFF. [it was added as note in BETA doc].

[Hotte]

Sorry, I did not have the time to check your questions. But now I tried it all out:

1. Yes, the virtual one-pass runs fine without the "Last=0" statement

2. "maxdupcount" is a parameter to limit the number of dups of an individual frame. I never really understood this and I never really needed it, so I worked around.

My testclip had exactly 1.392 frames. It consists of 348 individual frames stretched to 25% speed by inserting exactly 3 copies of each frame. So we have 4 occurrences (occs) of each frame. maxdupcount=0 cuts the clip down to its original which is 348 frames - so no limits. This is what I was looking for and I will use 0 in the future.

Results with "keeplastframe=false" (your new version):
maxdupcount = 0 => 348 frames (1 occurrence per frame - all dups deleted)
maxdupcount = 1 => 1.392 frames (all 4 occs - no deletions)
maxdupcount = 2 => 696 frames (2 occs)
maxdupcount = 3 => 696 frames (2 occs)
maxdupcount = 4 => 348 frames (1 occurrence per frame - all dups deleted)
maxdupcount = 5 => 348 frames (1 occurrence per frame - all dups deleted)

These are strange results. I'd have more understanding if the result of maxdupcount = 2 would have been 1.044 frames (but only by thinking "inverse").

3. "keeplastframe". In my tests this parameter always keeps at least 1 copy of each frame, independant of what the value of maxdupcount is.

so keeplastframe=true results in:
maxdupcount = 0 => 696 frames (2 occs)
maxdupcount = 1 => 1.392 frames (all 4 occs - no deletions)
maxdupcount = 2 => 1.044 frames (3 occs)
maxdupcount = 3 => 696 frames (2 occs)
maxdupcount = 4 => 696 frames (2 occs)
maxdupcount = 5 => 696 frames (2 occs)

I have no idea what these parameters could be useful for. But if sbdy would want to keep a minimum of frames dups I'd rather expect a "mindupcount" to represent the number of dups of the original to be kept, which would be (number of occurrences) + 1.

[StainlessS]

Thanks for that Hotte. [results for new version should be no different to original, I did not touch that stuff].

Maybe I initally looked at it and could not find any sense to it, so decided that I did not care what it was for.

[AdamCarter]

Question.
Animation uses a ton of duplicates and if mastered improperly will have blends.
Srestore can remove blends, but not the dupes.

I want to use a frame interpolation ai like DAIN to replace only the images that have blends. And leave the others untouched. Unfortunately i dont think dain has that specific function, it only allows for 2x and 4x increased in frames.

Wondering if i could use dedup and srestore to remove blends and dups and get it to 23.976 to 11.988. Then use Dain to make 2x.

[StainlessS]

ExactDedup is unlikely to be of any use to you,
if a single pixel of a single channel is not exactly the same as for previous frame, then it is not a dupe.