ExactDeDup

[Danette]

Hopefully, someone here knows how to use ExactDeDup. I am trying it, for the first time, and am running into the error message:
"Error requesting frame 239
ExactDeDup: Trying to compute frames out of order. Did you try to seek during processing?"

I've tried many different source videos and it's always the same frame # 239 cited in the message.

The script for the first pass is:
ExactDedup(firstpass=true, dupinfo="dupinfo.txt", times="times.txt", maxdupcount=20, keeplastframe=false)

[StainlessS]

Works for me, however its best to delete both times.txt and dupinfo.txt before you start (if they already exist).
There is at least one bug on the source,


if (dupfile)
{
fclose(dupfile);
timesfile = NULL; // should be dupfile=NULL;
}
if (timesfile)
{
fclose(timesfile);
timesfile = NULL;
}
if (framemap)
{
delete[] framemap;
framemap = NULL;
}

Will leave timesfile open.

What version of Avisynth are you using, and how are you playing file ?

Also, zip has been deleted from the TasVideo thread as linked on wiki, perhaps there is a reason for deletion.
http://avisynth.nl/index.php/ExactDedup

[Danette]

Thanks, StainlessS.

Neither of the txt files exist because it doesn't get that far. I can’t change the code, unless you can give me an idea as to how to do that, since I am simply loading the ExactDedup.dll plugin.

What version of Avisynth are you using, and how are you playing file ?

I use Avisynth 2.6 and play it through AvsPmod.

Further help would be appreciated.

[StainlessS]

I'm doing mod for 32 bit only.
Fixed several more issues, and will not write file until fully scanned (so you dont have to open in text editor to change mode, would start writing file again).
Also support for v2.6 planar (would not work with YV12, at least would not scan U, and V).

[StainlessS]

Ok Danette, see SendSpace link below this post in my sig. (not on MediaFire, having problems uploading anything).


ExactDedup v0.03
Copyright (c) 2011 Steve Melenchuk, Arick Chan
==============================================

0. Table of Contents
--------------------
1. Introduction
2. Usage
3. Licensing

1. Introduction
---------------
ExactDedup is an AVISynth 2.5 plugin providing a filter intended to remove
frames that are exact duplicates of each other, leaving only the first and
(optionally) last frames of a run intact, and generates a Matroska v2
timecodes file with timing information for the ensuing stream.

Due to the particulars of the AVISynth API, ExactDedup requires two passes
to function efficiently. It could theoretically be designed to run with
one pass, but this would result in a very long startup delay while the
plugin precomputes all of the duplicate frame information and timecodes.

ExactDedup theoretically supports any input colorspace, but has been
tested only with RGB colorspaces.

You will need the Visual C++ 2008 runtimes (i.e. MSVCR90.dll) to be able
to run this plugin.

2. Usage
--------

ExactDedup(clip [, bool firstpass] [, string dupinfo], [, string times]
[, int maxdupcount] [, bool _keeplastframe] [,bool show])

Options:
- firstpass : Default true. Controls which pass the plugin is running.

The first pass generates a mapping of output frames to
input frames (stored in the file specified by dupinfo) and
Matroska v2 timecodes (stored in the file specified by
times); seeking during this pass is impossible.

The second pass generates a video stream containing only
the frames specified in the dupinfo file. (Seeking is
permissible during this pass.)

- dupinfo : Default "dupinfo.txt". The name of a temporary file
generated by the first pass and read by the second pass;
each line corresponds to a frame in the output video
stream, and specifies the frame number of the frame in
the input stream that will be used as that output frame.

- times : Default "times.txt". The name of the Matroska v2 timecodes
file that will be output by the first pass.

- maxdupcount : Default 20. The number of maximum consecutive duplicate
frames corresponding to a single frame of the output; i.e.
a maximum of (maxdupcount - 1) frames will be removed from
a run of duplicates before another frame is kept.

- keeplastframe: Default false. Keeps the last frame of a run of duplicates.
If a run of duplicates is larger than maxdupcount, this
keeps only the last frame of the entire run, not of the
individual maxdupcount-size blocks.

- Show : Default false. If True, shows a little info on second pass.

3. Licensing
------------
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

#####################################################################

v0.4 Mod by StainlessS, 16 Feb 2017.
Avisynth v2.6 plugin.
Fixed a number of bugs.
Added Support for v2.6 Planar (also added U & V chroma scan for YV12, only scanned luma).
Added Show Arg.
Write times and dupinfo files at end of scan,so you dont have to edit script in text editor before second pass.
No runtime dependency.

ssS


EDIT: Avisynth v2.6 32 bit only dll, incl source + original v0.3 source.

[Danette]

Wow! It looks like you did some good work to bring this up to speed.

I tried your link, but can only see four files there, all from 2016.

[StainlessS]

Oops sorry. Was outside of the Public StainlessS folder, its there now. (still having Mediafire problems).

[Danette]

Thanks: downloaded fine.

Running it, the same error message appears, but with two differences now:

1. Instead of frame 279 showing, it is now frame 195 (for any video source - I've tried dgi, directshow and AVI for various videos).

2. Once I receive the error message, above, if I click anywhere in the AvsPmod window, the video appears, without the error message, but there is no processing by ExactDeDup that I can see.

I'm guessing that ExactDeDup is stumbling during the first pass attempt (no dupinfo or times txt files are being generated) and is bypassed when I click after the error message. Before trying your update, it could not be bypassed and I had to shutdown AvsPmod to escape.

[StainlessS]

How does it fair if you use eg VirtualDub ? (perhaps AvsPMod related).

Are you using any kind of Multi-threaded version of Avisynth v2.6 ?

ExactDeDup demands total linear access, if MT accessing frames in scrambled order, it will produce the described error message.

The text files are now only produced after completed successful scan (was a bit of a problem before, as if you load into eg AVSPMod
for second pass [without first editing the avs file for 2nd pass], it would immediately start writing the pass 1 files again).

EDIT: Also, can anyone else reproduce error with AvsPMod ?

[StainlessS]

Danette. Can you try out "TEST_1" version on SendSpace, I've added some debug information
(will require DebugView to see debugging stuff:- https://technet.microsoft.com/en-us/sysinternals/debugview.aspx )

First few lines of GetFrame()

PVideoFrame __stdcall ExactDedup::GetFrame(int n, IScriptEnvironment* env) {
dprintf("GetFrame(%d)",n);
n=(n<0)?0:(n>=vi.num_frames)?vi.num_frames-1:n; // Range limit n frame to valid range
if (firstpass) {
if(framemap!=NULL) { // FrameMap is still allocated and not yet written to files
if(n==0&&prevframe==0) {
framemap[n]=0; // Special case frame 0
} else {
if (n != prevframe+1) { // Must be strictly linear
dprintf(" *** PrevFrame=%d n=%d *** n is not PrevFrame + 1",prevframe,n);
delete [] framemap;
framemap=NULL;
env->ThrowError("ExactDedup: Trying to compute frames out of order. Did you try to seek during processing?");
}
prevframe = n;



Debug output in blue

Post last few lines of debug info.

EDIT: And you are not trying to use any filters after ExactDeDup, pass==1, are you ?

[Danette]

While you were typing, I tried your first suggestions. I'll try your new download, but thought you'd be interested in the results from your earlier suggestion:

How does it fair if you use eg VirtualDub ? (perhaps AvsPMod related).

Just tried pushing the script through Vdub. Vdub locked-up. So, I trimmed it to a 5-10 second segment to see if it would process. Then, after the 2 minutes, Vdub generated the error message:
“ffmpeg: decode video took longer than 2 minutes reading frame 0, aborted”

Script used was:
DGSource("C:\Users\Main\Desktop\Videos\MD.dgi")
Trim(4386,4695)
ExactDeDup()

Are you using any kind of Multi-threaded version of Avisynth v2.6 ?

No, just the standard 2.6, build: Mar 31 2015

[StainlessS]

What colorspace is it ?, wondering why ffmpeg was mentioned (and what audio).

EDIT: Also, make sure that old plugin is gone from your plugs directory.

[Danette]

Will have to postpone this, but will work on the debug later and report. In the meantime here is info on colorspace and more:

[Danette]

Ahhhh ...you gave me a clue with:

Also, make sure that old plugin is gone from your plugs directory.

I must not have copied it successfully. Deleted the old one from the plugins folder and copied the one from your link and now it seems to work fine (your original work is good). Txt files are generated and no error messages.

All I have to do now is to figure out how to work with it, which I'll do tomorrow.

Much thanks for staying with me through this and for working the new code.

[StainlessS]

Okay Dokey.

I added a whole load more debugging stuff now on SendSpace as Test_2, dont download.
I'll update on SendSpace in about an hour (bit busy).

[StainlessS]

OK, update zip on SendSpace, left all debugging in but disabled (uncomment Compiler.h #define BUG to switch back on).
Made a few small mods but not worth a version bump.

[Danette]

Well, now I’m into usage issues.

I have a dgi file with, clearly, many frames that are duplicates. I run ExactDeDup and the DupInfo.txt file is created. When I open that txt file, every single frame is listed (0-309). There are telecined frames in the file, but there are also duplicate frames. Not a single gap in the frame sequence exists so, I assume, that indicates that ExactDeDup found none of those duplicates.

I also created an AVI progessive file that has intentional duplicates in it (detelecined but not decimated), but get the same error message as reported, above, whenever AVI’s are sourced. So, it seems, that AvI’s cannot be used.

Also, when sourcing the dgi file, I loaded TFM and Tdecimate ahead of ExactDeDup, but that generates the error message as well. Is that a no-no?

[ndjamena]

??

If it's anything like DeDup metrics are recorded for every frame, it calculates which frames to discard later using the info from the txt file.

If you were talking about the time.txt file then that would be different...

[StainlessS]

ExactDedup calculates no metrics whatsoever. It decides based puely on exact frame contents, ie exits scan early if finds a single
pixel of any channel that is different where it is taken as not a dup, otherwise a dupe.

Can you post you problem avi, I'll have a look at it, but works fine for me on my test clip.
I got exact same results from original plug as modded one, but I had no problems getting original to work ok.
On my AVI test clip anime, it discarded about 1000, frames out of 4000.

It outputs a file of frames to keep, and also Matroska version 2 times for each of those frames.

First few entries for my test file, frames to keep.

0
6
7
8
9
11
12
13
14
16
17
18
19
21
22
23
24
26
27
28
29
31


And some times

# timecode format v2
0.000000000
200.200000000
233.566666667
266.933333333
300.300000000
367.033333333
400.400000000
433.766666667
467.133333333
533.866666667
567.233333333
600.600000000
633.966666667
700.700000000
734.066666667
767.433333333
800.800000000
867.533333333
900.900000000
934.266666667
967.633333333
1034.366666667
1067.733333333


Source file was 1920x1080 @ 29.97
EDIT: Ok my numbers were a bit off, test file had 4042 frames and kept 3371 of them.

[ndjamena]

I just tested it on an animated sequence sped up from 25fps to 50fps using changefps.

Although it successfully removed all the frames changefps added to the clip, I can't find a single instance of any of the original frames being removed from the clip, despite the fact that there are quite a few static scenes where each of the frames are almost visually indistinguishable from each other.

Given that's how it works, if he's really run a video through TFM it's unlikely ANY of the frames would be detected as duplicates.

I think he needs a different filter.

This was my attempt at re-writing DeDup:

http://www.mediafire.com/file/mfs7y8w7npv80mf/ReDup.zip

It's called ReDup, its purpose was to be run on animation to remove duplicates before processing, then add them back again once processing was complete. It was for when you're using something like Remation or another combination of filters that take forever to run.

Unfortunately my PTSD got in the way and I could never finished it... it was supposed to do much more than it does. I'm pissed off that I never managed to finish it.

It's based on the old DeDup code so it's got the same bugs, it contains the same functions as Dedup, but other than the ability to add the frames back in in the end it can also take an alternate clip for detection and even a second alternate clip to be compared to the first.

*sigh*

He probably just needs DeDup though, there's a version featuring fixed code floating on the forum somewhere.

[Danette]

You can find (hopefully) the mpg file at:

https://www.sendspace.com/file/j4w9y1

It has every 5th frame duplicated (I IVTC’d an NTSC file, but did not decimate for purposes of the test). This is not anime.

When I run the first pass through Vdub, I get the attached dupinfo and times data. As you can see, each frame is listed in the sequence, i.e.; no duplicates found by ExactDeDub. Same thing happens when I run the source AVI file. I have also found that I need to re-load the ExactDeDub filter into AvsPmod each time I make a script change to avoid an error message, if that’s meaningful.

I have not tried removing the duplicates with ExactDeDup, as yet, since the dupinfo data seems wrong. I assume that I would change the firstpass variable to “False” and re-run the file(?).

ndjamena: I didn't try your mod, as this is not anime.

[ndjamena]

ExactDeDup removes frames that are EXACTLY the same. IVTC’d frames are NOT exactly the same, the MPEG encoding process plus the IVTC’d leaves the "duplicate" frames slightly different so ExactDeDup doesn't touch them.

ReDup IS DeDup, you can use it to create a VFR MKV just like you can with DeDup. The difference for your purposes would be the ability to provide a separate clip for detection purposes or a second one if desired. The point of that is that you can provide a denoised clip for detection purposes without needing the denoising applied to the output. The secondary detection clip is so you can use a different denoising process for the comparison...

something like
fft3dfilter(bt=2)
for the first detection clip and
reverse.fft3dfilter(bt=2).reverse
for the second

I would have added more options if I'd been able to continue.

All of which is completely besides the point. The point being, EXACT DEDUP IS THE WRONG FILTER FOR YOUR PURPOSES.

DeDup or even just the decimation features of TDecimate would be more appropriate.

If it's simple NTSC with every 5th frame duplicated I'm not sure why you'd need any of these filters anyway, you'd be better of simply changing the frame rate by removing the 5th frames. ExactDeDup would be completely unnecessary for that, as would be any flavour of DeDup.

[StainlessS]

Ndjamena,
Check out framesel, framerep.
Framesel can extract a list of frames, framerep can put them back from whence they came.
All your repdupe thing needs to do create the frames file.

Mobile.

[Danette]

Thanks to you both. It sounds like non-anime duplicates are not exact and, so I can see where ExactDeDup won’t work. I tried ReDup, ndjamena, but stumble on what to enter for the “counts” variable (no instructions in the download). I get an error message, in AvsPmod, saying “the named argument “counts” to ReDup had the wrong type.” What should be entered here?

I also tried DeDup, but get the error message, in AvsPmod, that “DeDup: failed to open logfile” with the script written as: DeDup(log=test.dup.txt), which is patterned on the authors instructions for version .17. I tried looking around for a newer release, but couldn’t locate one.

Regarding your question:
I'm not sure why you'd need any of these filters anyway, you'd be better of simply changing the frame rate by removing the 5th frames.

You’re right, for this example I can remove that 5th frame with Tdecimate, but I created that simply to verify functionality for the different functions you two have offered. My intent is to use something that works on a badly mangled file that has random duplicates, sometimes 3-4 sequential duplicates that Tdecimate won’t address.

[ndjamena]

TDecimate Mode 3 and Mode 5 both can create VFR Matroska files.

DeDup(log=test.dup.txt)

Do you mean DeDup(log="test.dup.txt")?

I hope so. DeDup only throws that error if "log" equals NULL.

It shouldn't be hard to modify DeDup to have a default log file name if one isn't provided.

ReDup is as buggy as hell, "counts" is only used for actual ReDup when you're returning frames to the video.

These are the functions:

env->AddFunction("DeDup", "c[threshold]f[threshold2]f[range2]i[trigger2]f[show]b[dec]b[maxcopies]i[maxdrops]i[decwhich]i[log]s[times]s[counts]s[timesin]s[ovr]s[debug]s", Create_DeDup, 0);
env->AddFunction("ReDup", "c[counts]s", Create_ReDup, 0);
env->AddFunction("DupMC", "c[chroma]b[log]s[search]i[alt]c[alt2]c[maxaltthresh]f[minaltthresh]f", Create_DupMetric, 0);

Decimation function is still called DeDup, the ReDup function reverses it and "counts" is the file that stores which frames were removed. It's optional, you can provide "counts" or "times" or both...

Anyway, I believe your problems with DeDup is are being cause by you providing a null variable as an argument.

This is the TDecimate instructions:

https://avisynth.org.ru/docs/english/externalfilters/tivtc_tdecimate.htm

DeDup:
http://akuvian.org/src/avisynth/dedup/dedup.txt

The problem with DeDup is that it tends to be impossible to find a value that removes the dups but doesn't remove non-dup frames as well, since the values overlap. I was trying to correct that in ReDup but I think the dll is useless as it is.

[Danette]

Well, with the help of StainlessS and ndjamena, I found the solution to convert my badly telecined film into progressive that eliminates all duplicates, provides smooth video play with synced audio. I thought I’d post the results in case others find it useful.

This was an NTSC film that had several telecined patterns (3:2 and 2:1) as well as duplicate frames between some of the telecined frames. Although there may be better ways to do this (and I would be interested if there are any), here is what worked:

1. Before any IVTC, I used DeDup on the source to remove all duplicates between the telecined frames
2. On the resulting file, I then applied TFM and Tdecimate(cycler=2) - the default cycle=5 takes care of both telecine patterns).

My next step is to figure out how to do this in a batch process, but that should be simple at this point.

[johnmeyer]

I only skimmed the thread so I don't know if you publicly posted a sample of the video you were trying to IVTC. However, with patterns that don't repeat, or which have patterns that change, you can often get a good result by dramatically increasing the cycle size (to 20 or more) and then changing the TDecimate paramters to remove the most similar from each cycle.

[Yanak]

Hello,

Would like to know if by any chance there is a x64 version of the modified .dll file done by StainlessS ?
There is a x64 for the original .dll in version.03 but for this one i have to run it using MP_Pipeline and it probably hurts the performances on the first pass :/ .

Another question is how the "maxdupcount" settings works exactly, probably my English defaults me to understand properly this :
"Default 20. The number of maximum consecutive duplicate frames corresponding to a single frame of the output; i.e. a maximum of (maxdupcount - 1) frames will be removed from a run of duplicates before another frame is kept."

If i set it to let's say 10 and i have a sequence with 50 exact same frames it will keep one frame then delete (10-1) identical frames then keep one frame and again delete (10-1) identical frames and keep another frame etc ... and in the end will leave only 5 identical frames in the video sequence output, or am i understanding incorrectly ?

If i want no duplicate frames at all i just need to set this value really high or maybe something like "-1" will be more efficient ?

( Also does this parameters have any sort of impact on the performances during the first pass please ? if i set it really high like 500 or more will it hurts ? )

i done only a few tests for now as it takes really a long time to do the first pass on the footage i tested so i would like to be sure before starting to apply this on my projects ( and sorry if the question sounds dumb )

Thanks in advance.

[StainlessS]

ill try knock up 64bit within coupla days.
mobile.

[Yanak]

It will be awesome if you can do it ,

thanks in advance .

[StainlessS]

Yanak, see MediaFire or SendSpace below in my sig.
Set maxdupcount to a big number eg 1000000 to switch it off.


ExactDedup 0.05 (2017-05-17) (StainlessS)
Using Avs+ Header.
Convert to VS2008, x64 compile.


EDIT: Oops, shoulda been (2017-10-05)
EDIT: Avisynth v2.6 only (did not do v2.58 plug), and Avs+ x64.

[Yanak]

Thanks a lot for this,
as expected it goes a bit faster than using it in Mp_Pipeline, that's great :) .

Only thing different i noticed is that previously using Mp_Pipline in my script launched from Staxrip program it was creating the text files and using them in my Temp destination folder directly, now it creates the txt files directly inside the Staxrip folder, but during the second pass it cannot find them and i get an error message telling me it can't read the files.

But this is fixed if indicate a full path for the txt files creation on 1st pass and use the same full path for the .txt files during the second pass so not a problem at all.
Not sure what is the behavior using the 0.4 version with avisynth x86, outside MP_Pipeline i mean.

Thanks again a lot for this, you can't imagine how useful it will be for me.

[StainlessS]

Functionality did not change at all from v0.4, simple recompile (although v2.6 and using avs+ header now).

I did not play with paths at all, paths used to open files are exactly as provided, in all versions.

You can use in script (req RT_Stats, 64bit beta in your case), RT_GetFullPathName(filename) in any script to change relative path to full path.
If you use RT_Stats 64bit, and have any probs, let me know.

EDIT: I'll probably have XP64 set up to compile/test 64 bit stuff, within a week or two. (initially set up now, but no service packs or
other additions).

[Yanak]

Thanks,

Probably it worked differently when I was using it in Mp_Pipeline compared to when used directly, never tried in a x86 the 0.4 version but i'll give a try for the 0.3 native x64 version later to compare behaviors, anyways not a problem in this case, just had to setup a full path for my filter and it works now very nicely with a speed boost on the files i tested so it's perfect.

Thanks again for all, this thing will serve me nicely :)

[Yanak]

Just run a couple of tests using the 0.3 x64 version and it is the same indeed, so it is Mp_Pipleine that acts differently when creating/storing or reading the txt files, apparently creating them where the avs script is called and reading them there too, will keep this in mind for the next times i'll need similar things in Mp_Pipeline,

Thanks again.

[Ben_Nicholls]

Rather than outputting the duplicate frames to a timecode file to merge with an MKV creating a VFR video would it be possible to set it to use SVPflow to interpolate them to a specified CFR, thus creating smooth (non-ghosted) interpolated new frames?

It could also be expanded upon in order to allow creation of video at whatever desired rate you want without having to use SVP twice (once to remove duplicates, then again to change the FPS to the desired output).

So for example if you have a video that was shot at 24fps, but has been encoded at 25fps (creating 1 duplicate every second) but want to create 30fps CRF content for YouTube, you could set it to interpolate the detected duplicate frames, and then interpolate all the frames until the desired frame rate was achieved.

[StainlessS]

@Ben_Nicholls,
Assuming that above is aimed at me, sorry but no, It's not my plug and I dont really wanna adopt it (I just spotted several bugs and fixed them).
I did not touch times code stuff and have no idea how it works, nor do I use SPV, and likewise know nothing of its working.
Got way too much on my plate already and am actually trying to get away from Avisynth stuff.
Again sorry.

EDIT: This any good:- https://forum.doom9.org/showthread.php?p=1764865#post1764865

[JoyBell]

I have been reading this thread for a few days. I have come to the conclusion that I am missing the basic information as to where to start.

I have no idea where to attempt to use this code.

Let me back up.

I use StaxRip and would like to use ExactDeDup to take in a CFR source file and output a VFR file. I am thinking on Animatics or perhaps Cartoons as many frames are the same.

I don't see how ExactDeDup can be use as an Avisynth filter in Staxrip.

Is this a Script I run as a separate tool to just remux the source and then use that in Staxrip? If so, how do I do that?
I have been asking in my groups discord channel for a few days now and have gotten no where. Thanks for help!

[Yanak]

Hello,

Will try to explain wiht all steps, hopefully it will do the work.

Previously in Staxrip i was using the event commands combined with a template to make the 1st pass and scan file for dupes, but since yesterday it is possible to now execute the filter in a single pass with the help of a little new plugin found here : https://forum.doom9.org/showthread.php?t=175616 ( Thx again StainlessS :) )

Download the file here : http://www.mediafire.com/file/xyvpruldqjyugb4/ForceProcess_x86_x64_dll_v1-00_20180724.zip/file
Inside the zip file Avisynth+_x64 folder you find the ForceProcess_x64.dll file, extract it and put it inside avisynth+ plugin64+ folder, located here :
"C:\Program Files\AviSynth+\plugins64+"

This done we will create a filter inside Staxrip to use ExactDeDup, in the list of filters make a right click and then on the dropdown menu click on profiles, like here :
https://i.imgur.com/Kw7XYiF.png


On the new window scroll a bit until you see the [Misc] category , and there you can copy paste the following :
Remove Dup Frames =
if (Exist("%working_dir%times.txt")) {
ExactDedup(firstpass=false, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
maxdupcount=1000000, keeplastframe=false, show=false)
}
else {
ExactDedup(firstpass=true, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
maxdupcount=1000000, keeplastframe=false, show=false)
ForceProcess()
ExactDedup(firstpass=false, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
maxdupcount=1000000, keeplastframe=false, show=false)
}
It should then look like this :
https://i.imgur.com/tYIlLcG.png
If needed adjust a bit the code for things like keeplasframe setting etc.
( Filter can be put in another category, it's up to you, just found Misc one to fit :p )

Once this done you have now a new filter added into StaxRip and can now select the new created filter and add it to the list :
https://i.imgur.com/HLpCnu2.png

When adding this filter for the first time into a project it may look like Staxrip is freezing a bit and it can take a bit of time before it recovers, in fact immediately when you add the filter the new little plugin "ForceProcess" starts his work and analyze the video file to find the duplicate frames and create the data text files, so just be a bit patient in case it takes too long, but the filter code i provided is made so that if the timestamps "times.txt" file already exist it will not scan it again, if the scan was done and the file is present on the temp folder it will just use the tiemstamp file directly.

Last thing to do is on the MKV container to set up the path for the timestamps, click on "Container Options" :
https://i.imgur.com/9FLXkEZ.png

And then go to the "Options" Tab and select the times.txt file, by default clicking the little button like in the pic below should send you directly to the folder where the file is located :
https://i.imgur.com/9L1oKRl.png
just select the times.txt file there.

I did not found a way to automate this part yet, adding additional custom switches in a dedicated MKV profile doesn't seems to work for this particular one so you will have to select the timestamps manually each time, unless i find a solution for this one of those days...


After this you are done, filter is created, when needed just add the "Remove Dup Frames" filter, wait a little for video to be scanned for duplicate frames and finally go to the mkv container options and select the times.txt file.

From there the rest is up to you for encoding parameters etc, once encoded the MKV should be a variable framerate file where all duplicate frames have been eliminated and the container having the mkv timestamps.

Hope it helps a bit :)

PS : forgot to say but of course you also need to download ExactDeDup file and put the win64 dll in the same plugin64+ folder as the dll in the top of this post ;)

[StainlessS]

Nice walkthrough Yanak (I would not have had a clue how to answer that question). :thanks:

[Yanak]

The least i can do, if it can help a bit :)

I am now trying to have a feedback message from Staxrip program when it finished the scan of the video but no success for now, just so it does not look like freezing when processing the entire video to get duplicate frames data and get a small popup notification when it's done, previously i used a template dedicated for this associated with event commands :
- When i needed this plugin i selected the dedicated template
- The videos loaded in this template triggered an event command , launching avs2pipemod64 -benchmark and processing the analyze, and Staxrip window was showing the progress of the benchmark,
- Once analyzed another event command was replacing the first pass filter by the second pass filter automatically

Might post this method too if needed, at least you have a feedback on what Staxrip is doing and not like it is drawing and freezing, i don't mind myself but might be less scary :p


BTW i did not had time to start benchmarking things yet plus it's way too hot those last days to run the PC in a room for a long time, room goes up to 30°c inside after barely one hour using the PC with light tasks... other rooms stays at 26-27 which is too high already, but will do soon tho no worries.

[Yanak]

For this other method you don't need ForceProcess.dll plugin, only ExactDedup x64 dll is needed.

First let's create a template dedicated for this task :
- Select your preferred encoding template ( or the one who starts wiht Staxrip usually )
- Go to Project > Save as Template > and give it a dedicated name like "Remove Duplicate Frames":
https://i.imgur.com/s4xIpHq.png

On this template we will directly add the filter code, same as previously but lighter code, in the list of filters make a right click and then on the dropdown menu click on profiles, and copy paste this on the Misc Section :
https://i.imgur.com/fZ27M13.png
Remove Duplicate Frames =
ExactDedup(firstpass=True, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
maxdupcount=1000000, keeplastframe=false, show=false)
Note that you can't have both this one and previous one at the same time, well it is possible but need to change the filter name... Best is to stay with one or the other method :p


Once the filter created we will add it to our project and to our new created template :
Right click on the list of filters Add > Misc > Remove Duplicate Frames

Then we save our template with this filter added:
Project, Save as Template > we leave the name we put earlier "Remove Duplicate Frames" and just click OK


Now we will create event commands triggers :
go to tools > Advanced > Event Commands > click on "+ Add"
On this first one we will set it like this :
https://i.imgur.com/K9nyefe.png

The command line is :
"%app:avs2pipemod%" -benchmark "%script_file%"
The rest is easy to copy from the picture :p

Once done we will again create another event command, same as before click on +Add, and this time we will set it like this :
https://i.imgur.com/GO6mXgO.png
Unlike on this pic for the name above the code line it needs to be Remove Duplicate Frames and not "Remove Dup Frames", forgot to change it sorry :p

Notice the little button on the orange arrow, that's where you click to bring the editor, easier than trying to type in the tiny field, in the editor you paste this :
https://i.imgur.com/SZxq8Su.png
ExactDedup(firstpass=false, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
maxdupcount=1000000, keeplastframe=false, show=false)

Lastly check the 2 little boxes to make them active :
https://i.imgur.com/xv5XamU.png


That's it, The next time you will need to use this filter go to templates, select the template named "Remove Duplicate Frames" , and then the rest it automated,when you will browse or drag and drop a video into staxrip the first pass will start and you will see something like this while the video is being analyzed for duplicate frames :
https://i.imgur.com/R7w4dOO.png

Once finished the second event command we created will be triggered and automatically replace the first pass filter by the second pass one in the list of filters and video from there have all the duplicate frames removed, you still have like in the previous method to go to the MKV container options and select the times.txt file for the timestamps but that's all.

( It is possible using the template method to save the times.txt file to a specific location that will always be the same, not in the temp folder, and then save the template with the MKV setting for the timestamp path saved too, this will be a full auto process, if needed just ask, or in both the filters codes of this page set a static path for the times.txt file , and in MKV options set this same path too then save the template, again ask if needed or interested )

At least here you see what Staxrip is doing, like i said i don't mind myself but maybe it's better and less scary idk

;)

[johnmeyer]

I probably should have posted this back when this thread first started.

The following is the simple script I use to detect duplicates, and near-duplicates. I haven't used Dedup, ExactDeDup, or StanlessS' mods, but this certainly can and does detect exact duplicates perfectly, and does a very nice job with near-duplicates. I usually use this code in conjunction with some sort of replacement logic, either by taking the frame number list and operating on that in a second pass, or by simply adding some conditional logic to the function. I've posted these frame replacement scripts in other doom9 threads.

The script is based entirely on YDifference, a built-in AVISynth function that compares the luma of every pixel in the current frame with every pixel in the previous frame and then does a sum of absolute differences to obtain a metric. This is a very useful function for scene change detection, motion detection, and any other situation where you want to get a metric that relates to the amount of visual change between frames.

YDifference sure ain't perfect, but it is amazing how often it gets the job done.

Here's my very simple-minded approach to finding duplicates. It is obviously hard-wired and not written as a proper function, so you have to go through and change things as needed (e.g., assumeBFF). I never could be bothered to clean it up.#This script finds either exact duplicate frames (blankthreshold=0) or near duplicates
#and then outputs the frame numbers of the dups.

global blankthreshold=3

filename = "e:\output_duplicate_frames.txt"
AVISource("e:\fs.avi").killaudio()
i=AssumeBFF.ConvertToYV12

#This line below will output EVERY frame that is below threshold, which results in LOTS of frames
#Normally you don't do this, but it is included for those who want this capability.
#WriteFileIf(last, filename, "(YDifferenceFromPrevious(i)<=blankthreshold)", "current_frame+1", append = false)

#The line below writes the FIRST frame that falls below the threshold
WriteFileIf(last, filename, "(YDifferenceFromPrevious(i)>blankthreshold)&&YDifferenceToNext(i)<=blankthreshold", "current_frame", append = false)

#Use this instead of WriteFile in order to determine blankthreshold
#ScriptClip("Subtitle(String(YDifferenceFromPrevious(i)))")

[Yanak]

Glad to see you get it working,

About the error reported , what version of ExactDeDup are your using please ? The "show" parameter was added in version 0.04 apparently :

v0.4 Mod by StainlessS, 16 Feb 2017.
Avisynth v2.6 plugin.
Fixed a number of bugs.
Added Support for v2.6 Planar (also added U & V chroma scan for YV12, only scanned luma).
Added Show Arg.
Write times and dupinfo files at end of scan,so you dont have to edit script in text editor before second pass.
No runtime dependency.

Latest version of this plugin is v0.05, direct link to StainlessS last version :
http://www.mediafire.com/file/8184da6dp4hsdcq/

And for :
%working_dir%\times.txt it should be %working_dir%times.txt without the \ in the middle, not sure if it's just a typo you made while posting on the forum or if you entered this in the code, i checked again and in my code samples posted here it seems to be correct without the /, unless i am not seeing it.

You can quickly test this :
Just open a video in Staxrip no matter the video, template or filters, just open one video so it creates temp files,
Then go to the filters editor and just for the test add a line :
%working_dir%
then go to the preview tab :
https://i.imgur.com/tbp8KSW.png
does it show the correct path there ? i mean is the path indicated the location where Staxrip just created some .AVS and a .LOG file for the video you opened in it ?
( think about removing this %working_dir% line from the profiles editor created just for the test after it's done ^^ )

I'm using Staxrip v 1.7.0.6, the latest one released by Stax76 and it works there, not sure what version of the program you are using.
You can check it in Help > Info

As for the duplicates frames removed it depends of the sources and the quality, this plugin removes exact duplicates, sometimes if your source material is already an altered source, already encoded i mean, there can be frame that were originally the same but during the encoding some little artifacts were added so it ends up in 2 different frames for the plugin, but for things like gaming capture unaltered/raw footage there is often a lot of duplicate frames.


@johnmeyer, i'll give it a try one of those days, might serve in some occasions , thank you

[Wilbert]

@JoyBell, Please respect the forum rules. No talk about illegal downloaded stuff.

[Yanak]

? I have been out for a bit and see this now, cannot remember exactly the last posts but i don't recall any piracy discussion, i see that my last post was removed in the process too but i'm sure at 100% that i never spoke of piracy related things...

Anyways...

Came to continue the conversation started here (http://forum.doom9.org/showthread.php?t=175043&page=3), makes more sense to post it here on ExactDedup dedicated thread.

Took me way longer to experiment things than expected as being doing some other stuff irl, plus really hot weather all those days did not push me to make the PC run a lot when i had some free time to burn, but as promised i recompiled it, on some files it gains nicely during the 1st pass, the 2nd pass is always more or less the same, on some others video files there is close to no gain, probably because i reach the hardware limit before the software part.

A part of the tests results :

***********************************
#### Script ###
ColorBars(width = 1920, height = 1080, pixel_type = "YV12", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999)
***********************************
########### 1st pass Original dll ##########
Frames processed: 10000 (0 - 9999)
FPS 263.7
Time (elapsed): 00:00:37.920
########### 1st pass Compiled dll ##########
Frames processed: 10000 (0 - 9999)
FPS 745.6
Time (elapsed): 00:00:13.411

########### 2nd pass Original dll ##########
Too fast for metrics
########### 2nd pass Compiled dll ##########
Too fast for metrics

***********************************
#### Script ###
ColorBars(width = 1920, height = 1080, pixel_type = "RGB32", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999)
***********************************
########### 1st pass Original dll ##########
Frames processed: 10000 (0 - 9999)
FPS 66.34
Time (elapsed): 00:02:30.736
########### 1st pass Compiled dll ##########
Frames processed: 10000 (0 - 9999)
FPS 187.2
Time (elapsed): 00:00:53.432

########### 2nd pass Original dll ##########
Too fast for metrics
########### 2nd pass Compiled dll ##########
Too fast for metrics

***********************************
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 6.10 GiB
Duration : 2 min 36 s
Overall bit rate : 336 Mb/s
Format : Fraps
Codec ID : FPS1
Duration : 2 min 36 s
Bit rate : 334 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 2.685
Stream size : 6.07 GiB (100%)
***********************************
#### Script ###
FFVideoSource("video.avi", colorspace = "YUV420P8", \
cachefile = "video.ffindex")
***********************************
########### 1st pass Original dll ##########
Frames processed: 9366 (0 - 9365)
FPS 373.1
Time (elapsed): 00:00:25.102
########### 1st pass Compiled dll ##########
Frames processed: 9366 (0 - 9365)
FPS 426.9
Time (elapsed): 00:00:21.939

########### 2nd pass Original dll ##########
Frames processed: 5647 (0 - 5646)
FPS 271.0
Time (elapsed): 00:00:20.836
########### 2nd pass Compiled dll ##########
Frames processed: 5647 (0 - 5646)
FPS 272.1
Time (elapsed): 00:00:20.753

***********************************
#### Script ###
DSS2("video.avi")
***********************************
########### 1st pass Original dll ##########
Frames processed: 9365 (0 - 9364)
FPS 92.93
Time (elapsed): 00:01:40.779
########### 1st pass Compiled dll ##########
Frames processed: 9365 (0 - 9364)
FPS 116.5
Time (elapsed): 00:01:20.355

########### 2nd pass Original dll ##########
Frames processed: 5647 (0 - 5646)
FPS 73.85
Time (elapsed): 00:01:16.464
########### 2nd pass Compiled dll ##########
Frames processed: 5647 (0 - 5646)
FPS 73.65
Time (elapsed): 00:01:16.672

***********************************
#### Script ###
AviSource("video.avi", Audio = False)
***********************************
########### 1st pass Original dll ##########
Frames processed: 9366 (0 - 9365)
FPS 73.12
Time (elapsed): 00:02:08.086
########### 1st pass Compiled dll ##########
Frames processed: 9366 (0 - 9365)
FPS 101.8
Time (elapsed): 00:01:31.967

########### 2nd pass Original dll ##########
Frames processed: 5647 (0 - 5646)
FPS 83.01
Time (elapsed): 00:01:08.028
########### 2nd pass Compiled dll ##########
Frames processed: 5647 (0 - 5646)
FPS 83.06
Time (elapsed): 00:01:07.991

***********************************
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 3.88 GiB
Duration : 55 s 933 ms
Overall bit rate : 596 Mb/s
Format : Fraps
Codec ID : FPS1
Duration : 55 s 933 ms
Bit rate : 594 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 4.778
Stream size : 3.87 GiB (100%)
***********************************
#### Script ###
FFVideoSource("video.avi", colorspace = "YUV420P8", \
cachefile = "video.ffindex")
***********************************
########### 1st pass Original dll ##########
Frames processed: 3356 (0 - 3355)
FPS 357.9
Time (elapsed): 00:00:09.378
########### 1st pass Compiled dll ##########
Frames processed: 3356 (0 - 3355)
FPS 387.7
Time (elapsed): 00:00:08.657

########### 2nd pass Original dll ##########
Frames processed: 2455 (0 - 2454)
FPS 298.9
Time (elapsed): 00:00:08.214
########### 2nd pass Compiled dll ##########
Frames processed: 2455 (0 - 2454)
FPS 298.8
Time (elapsed): 00:00:08.217

***********************************
#### Script ###
DSS2("video.avi")
***********************************
########### 1st pass Original dll ##########
Frames processed: 3355 (0 - 3354)
FPS 78.28
Time (elapsed): 00:00:42.858
########### 1st pass Compiled dll ##########
Frames processed: 3355 (0 - 3354)
FPS 84.04
Time (elapsed): 00:00:39.923

########### 2nd pass Original dll ##########
Frames processed: 2454 (0 - 2453)
FPS 62.36
Time (elapsed): 00:00:39.352
########### 2nd pass Compiled dll ##########
Frames processed: 2454 (0 - 2453)
FPS 62.50
Time (elapsed): 00:00:39.264

***********************************
#### Script ###
AviSource("video.avi", Audio = False)
***********************************
########### 1st pass Original dll ##########
Frames processed: 3356 (0 - 3355)
FPS 64.34
Time (elapsed): 00:00:52.159
########### 1st pass Compiled dll ##########
Frames processed: 3356 (0 - 3355)
FPS 77.35
Time (elapsed): 00:00:43.387

########### 2nd pass Original dll ##########
Frames processed: 2455 (0 - 2454)
FPS 65.33
Time (elapsed): 00:00:37.579
########### 2nd pass Compiled dll ##########
Frames processed: 2455 (0 - 2454)
FPS 65.33
Time (elapsed): 00:00:37.576

***********************************
Format : AVI
Format/Info : Audio Video Interleave
File size : 839 MiB
Duration : 50 s 0 ms
Overall bit rate : 141 Mb/s
Format : Fraps
Codec ID : FPS1
Duration : 50 s 0 ms
Bit rate : 139 Mb/s
Width : 1 280 pixels
Height : 1 024 pixels
Display aspect ratio : 5:4
Frame rate : 30.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 3.545
Stream size : 831 MiB (99%)
***********************************
#### Script ###
FFVideoSource("video.avi", colorspace = "YUV420P8", \
cachefile = "video.ffindex")
***********************************
########### 1st pass Original dll ##########
Frames processed: 1500 (0 - 1499)
FPS 474.5
Time (elapsed): 00:00:03.161
########### 1st pass Compiled dll ##########
Frames processed: 1500 (0 - 1499)
FPS 492.5
Time (elapsed): 00:00:03.046

########### 2nd pass Original dll ##########
Frames processed: 1500 (0 - 1499)
FPS 492.3
Time (elapsed): 00:00:03.047
########### 2nd pass Compiled dll ##########
Frames processed: 1500 (0 - 1499)
FPS (min | max | average): 54.79 | 6789 | 501.7
Time (elapsed): 00:00:02.990

***********************************
#### Script ###
DSS2("video.avi")
***********************************
########### 1st pass Original dll ##########
Frames processed: 1500 (0 - 1499)
FPS 161.7
Time (elapsed): 00:00:09.274
########### 1st pass Compiled dll ##########
Frames processed: 1500 (0 - 1499)
FPS 162.0
Time (elapsed): 00:00:09.259

########### 2nd pass Original dll ##########
Frames processed: 1500 (0 - 1499)
FPS 161.5
Time (elapsed): 00:00:09.290
########### 2nd pass Compiled dll ##########
Frames processed: 1500 (0 - 1499)
FPS 161.6
Time (elapsed): 00:00:09.284

***********************************
#### Script ###
AviSource("video.avi", Audio = False)
***********************************
########### 1st pass Original dll ##########
Frames processed: 1500 (0 - 1499)
FPS 139.5
Time (elapsed): 00:00:10.749
########### 1st pass Compiled dll ##########
Frames processed: 1500 (0 - 1499)
FPS 154.1
Time (elapsed): 00:00:09.734

########### 2nd pass Original dll ##########
Frames processed: 1500 (0 - 1499)
FPS 161.5
Time (elapsed): 00:00:09.288
########### 2nd pass Compiled dll ##########
Frames processed: 1500 (0 - 1499)
FPS 161.5
Time (elapsed): 00:00:09.286

***********************************
Format : Matroska
Format version : Version 4 / Version 2
File size : 492 MiB
Duration : 2 min 28 s
Overall bit rate : 27.8 Mb/s
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L8.5@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 2 min 28 s
Width : 960 pixels
Height : 540 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 50.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
***********************************
#### Script ###
DSS2("video.mkv")
***********************************
########### 1st pass Original dll ##########
Frames processed: 7415 (0 - 7414)
FPS 348.4
Time (elapsed): 00:00:21.286
########### 1st pass Compiled dll ##########
Frames processed: 7415 (0 - 7414)
FPS 354.5
Time (elapsed): 00:00:20.915

########### 2nd pass Original dll ##########
Frames processed: 6962 (0 - 6961)
FPS 334.5
Time (elapsed): 00:00:20.815
########### 2nd pass Compiled dll ##########
Frames processed: 6962 (0 - 6961)
FPS 334.8
Time (elapsed): 00:00:20.793

***********************************


Hosting it here for the moment until attached x64 dll is approved : https://www84.zippyshare.com/v/CdNTNTbC/file.html

[StainlessS]

? I have been out for a bit and see this now, cannot remember exactly the last posts but i don't recall any piracy discussion

I too was puzzled by post #46 at the time it was posted.