Looking for "radical" 1-pass duplicate frame removal

[Hotte]

Hi,

I need to post-process special avi output from my NLE with avisynth.

The footage contains longer or shorter sequences of exactly identical progressive frames that need to be removed.

The duplication pattern varies according to the situation. In other words: It is unknown, so frame to frame duplication detection is needed.

I do not care for audio-sync because I know the underlying framerate and audio will be restored separately.

A one-pass solution is an absolute must and is has to be part of the avisynth post-processing script, no external solution.

All the filters,scripts and plugins I tried either leave some duplicates in the output
or they want to replace the deleted frames with something
or they are 2-pass solutions (as far as I tried them out they all need two separate runs of avisynth which does not work for my setup).

Any ideas ? Thankx.

[johnmeyer]

Duplicate frame detection is easy if your video contains perfect duplicates. You can look for my "filldropsI()" function that I adapted from something MugFunky wrote many, many years ago to see how to detect duplicates and then take action. That function contains a variable that you can change if you also want to include near-duplicates as well as those which are perfectly identical.

It is also easy to replace those frames with a motion estimated frame. That is actually what that filldrops script does.

However, doing decimation in one pass may not be possible because you don't know in advance how many frames are going to be removed. I suggest you use the forum search for either the word "decimate" or "decimation" and restrict that search to only look at the title of posts, not the content of the posts. I just did a quick look, and there are lots of questions similar to yours. Perhaps someone came up with a clever solution.

Here's a link to one of several places where I posted the filldropsI code. It is not the total solution to your current problem, but it is part of it:

FilldropsI() (https://forum.doom9.org/showthread.php?p=1794809#post1794809)

You will find lots of other information in that thread which might be useful.

BTW, as you will see if you read that thread I linked to, I mention something there that should be mentioned here: most videos which contain lots of duplicates at random, non-repeating places, have that anomaly because the capture software or the streaming software dropped a frame. This is extremely common and happens in almost all streaming video, and can also happen with lousy capture software (most capture software is highly flawed). Once a frame is dropped/skipped/missed, something must be done in order to keep the audio in sync, and the obvious thing is to add a duplicate frame. This is usually done within a few frames of the drop, but the frame offset from the drop is not the same number of frames each time, so you can't simply find the duplicate (easy) and use it as the way to find where the drop happened (hard).

As you step through the video, one frame at a time, the duplicates are obvious, but the dropped frames don't always reveal themselves in as obvious a way. However, if someone is walking across the frame, to take one example, and a frame is dropped, that drop is extremely obvious and annoying. So, before you decide that you simply want to decimate frames and throw away the audio, you should first verify that you don't also have dropped frames. If you do, I developed some software that can dramatically reduce the problem. However, while it is possible to flawlessly identify perfect duplicates, finding the place where a frame is dropped is very difficult to do, and the results will not be flawless. Fortunately, with this script, because of the way it works, when it inserts a frame at the wrong place, that mistake does not call attention to itself.

If you want more information on how to detect both duplicates and drops, and automatically do something about it, you might get some help from this thread I started many years ago:

Automatically fix dups followed (eventually) by drops (https://forum.doom9.org/showthread.php?t=161758)

.

[Hotte]

thanks johnmeyer for your detailed answer.

Maybe it is helpful to tell the full story. The NLE allows me to use a slow-mo effect. For this the NLE will either duplicate frames or blend them over. I dislike both. So I would like to read out the slow-mo factor of the clip on the timeline, export the next-neighbour duplicated version, discard the dups and feed the clip into Interframe from SVP to interpolate the slow-mo sequence.

I just tried out "filldrops(). It replaces the dups with MVTools generated frames - which is a good idea! However although the output is nicely slow and synced, motion comes in periodical waves which doesn´t look very nice. The reason is probably, that the dup frame pattern with long dup sequences is not ideal for interpolation. However Interpolation is very smooth when the exported clip has it original frame sequence i.e. all the dups have been removed and Interframe or MVTools do the full interpolation. This is why I cannot use a replacement filter.

Maybe you could help me with this: I found a two pass filter which drops the frames exactly the way I want. Is it possible to have the two passes run within in one avisynth script i.e. run pass1 until it has come to its end and written the log file and then start pass2 ? I am not a programmer and as far as I understood the concept of avisynth it works on a frame base so this is probably not feasible. Anyway, there might be a solution ?

[StainlessS]

ForceProcessAVI(clip c,bool "Debug"=True)
Force Process clip c, ie read from first to last frame, for TWriteAVI writes the AVI file (Video + Audio) without having to play clip.
Debug, default True, sends some progress info to DebugView (google).



Two Pass:

WhateverSource(...)
A_Two_Pass_Function(Pass=1) # Create some kind of deshaker log or whatever.
ForceProcessAVI() # Force Pass 1, creating log (A function From TWriteAVI plug)
A_Two_Pass_Function(Pass=2) # Use created log
return Last

The ForceProcessAVI function forcibly processes both video and audio streams (ie reads each frame/sample in sequence),
so that any eg log files produced by either video and/or audio processing filter will be forcibly written to disk.
ForceProcessAVI/WAV are runtime functions not filters and so only return on completion.

Simultaneous play and save AVI

WhateverSource(...)
FN="Output.AVI"
FOURCC="ULY0" # Ut_Video
TWriteAVI(FN,Overwrite=true,fourcc=FOURCC).
return Last


Do some kind of processing and save AVI, Then play Lossless file.

WhateverSource(...)
FN="Output.AVI"
FOURCC="ULY0" # Ut_Video
Sharpen(0.3) # Whatever
TWriteAVI(FN,Overwrite=true,fourcc=FOURCC). # Write Sharp AVI whilst clip is played
ForceProcessAVI() # Force Write above TWriteAVI file without playing clip.
AVISource(FN) # Load Saved AVI
return Last # Play Sharpened AVI lossless clip


TWriteAVI v2.0:- http://forum.doom9.org/showthread.php?t=172837&highlight=TWriteAVI

EDIT: Not for MT processing.

[johnmeyer]

So I would like to read out the slow-mo factor of the clip on the timeline, export the next-neighbour duplicated version, discard the dups and feed the clip into Interframe from SVP to interpolate the slow-mo sequence.If the goal is to get motion-estimated slow-mo instead of the usual NLE frame-blended version (or duplicated version), then the way I've always done that is to frameserve each section into a MVTools2 MFlowFPS script (or Interframe, etc.). I then use a high-quality intermediate, like Cineform, to store that slow-mo and then import it onto the timeline (I use Vegas Pro), directly above the field-blended slow mo version.

I am able to automate a lot of tasks using the scripting that is part of Vegas. In fact, many years ago, I wrote a suite of scripts to completely automate the two-pass Deshaker process. It works like this:

1. You select all the events you want to stabilize.

2. You run my Vegas script which goes through each selected event on the selected track and for each one, it exports it to a temporary file, using either uncompressed or, if Vegas supports smart-rendering (like for DV files), it will use smart rendering.

3. My Vegas script then launches a VirtualDub script. That script opens the file I just exported and runs the first Deshaker pass.

4. That VirtualDub script then runs the second Deshaker pass.

5. The VD script then renders the resulting video to an intermediate file.

6. The VD script then returns control to the Vegas script which takes the newly rendered file and adds it as a take to the current event.

There are lots of other things that I had to do to make this work, but it actually works well, and a lot of people used it over the years.

A similar thing could certainly be done for your situation, although it is more work than I would want to take on, even though I do occasionally need to do something similar.

Having said that, it would be pretty cool to add automatic motion-estimated slo-mo to Vegas.

BTW, I don't know how much experience you have using the results of MVTools2, SVPFlow (the GPU version, optimized for real time frame rate conversion), and Interframe, the front-end for SVPFlow that makes its weird syntax a little easier to use. However, if you haven't used it much, I warn you in advance that you are going to end up with lots of weird artifacts. Even with the big project to add masking to this mix, it still fails a lot. The solution is to still do the field blended version and, when the motion estimation fails, but to that version for the 1-2 frames during which the artifact shows up. If you really want to get into it, and if you are using Vegas, you can create a simple mask over the artifact, and only use the field-blended version to replace that one section of the frame. This is lots of work, but it produces really amazing results.

[Hotte]

Johnmeyer, funny, how - at the end of day - people are coming to similar solutions. I have got EDIUS. Unfortunately there is no built-in scripting in EDIUS so I am using Autohotkey. I built a comfortable filtering interface within Edius which allows me to customize each clip with individual avisynth filtering.

Then with the help of Autohotkey I export a high-quality intermediate AVI (Canopus Hqx) of the timeline, and all the filtering is automatically applied in a scheduled multi-parallel processing of vdup/avisynth, which could easily run the night through at full processing power. The timeline is coming back as individual processed intermediate clips and can be easily dragged into or over the timeline. Its so nice to empower Edius with all that wonderful avisynth filters like interframe, qtgmc or mdegrain.

Concerning Interframe I recently made an intense and detailed comparison of MVTools2, FrameRateConverter (based on MVTools but handles things slightly different), Edius Optical Flow (yeah they have something built in) and SVPFlow. My results were:

1. Interframe script (SVPFlow), gpu=true tuning=Film, OA=23, blksizes=8 (exactly these parameters) delivers the most convincing object-morphing of all (continuously flowing object movement). If object movement gets too wild, push up the blksize to 16x16 or 32x32 and it will automatically switch to object-blending where movement is too strong. There is no need for masking which only introduces blur. This behaviour works perfectly for me. I adjust the blksize according to the character of the scene and I am pretty happy with it. I do not understand why so many people put Interframe down. However other parameters deliver worse to horrible results.

2. Edius Optcial Flow is the very best choice, if the camera pans slowly across the scene and there is only little object movement. Whatever reasonable slow-mo factor is applied such a scene capture stays cristal-clear without noticeable artifacts on the frame edges. However it will easily create really big halos around moving objects and blends objects far too early.

3. MVTools2 delivers decent allround quality but will not deliver that smooth morphing I found with interframe with the parameters above. It has a bit less artifacts on the frame-edge (right side when you move right and vice versa) with slow camera panning than interframe and the object halos are a tiny, tiny bit weaker (which is not really relevant).

4. FRC is sowewhat similar to MVTools2, but it tends to switch to object blending too early.

It's also a question of a never ending combination of parameters and, of course, taste....

[Hotte]

Wow, Stainless... I tried this out immediately. My filter - you kow it very well - is ExactDeDup. But I have difficulties with this code, which runs absolutely fine with two passes.
This is the complete script:


AviSource("E:\blabla.avi")
ExactDedup(firstpass=true, dupinfo="Z:\Temp\dupinfo.txt", times="Z:\Temp\times.txt", maxdupcount=1000000, keeplastframe=false)
ForceProcessAVI()
ExactDedup(firstpass=false, dupinfo="Z:\Temp\dupinfo.txt", times="Z:\Temp\times.txt", maxdupcount=1000000, keeplastframe=false)
AssumeFPS(2500, 100, true)


After loading the Script into VDub I am getting a pretty long pause (where it obviously does pass 1) and I do find times.txt and dupinfo.txt filled with the right data. However after the pause it says:

ExactDedup: Failed to open dupinfo file for reading.

(Also I have the impression that pass 1 is not faster, maybe even slower than an Analyse Pass 1 in VDub. I read in some post that there should be a significant speed improvement).

Could you help ? Thanx!

[johnmeyer]

Wow, Stainless... I tried this out immediately. My filter - you kow it very well is ExactDeDup. But I have difficulties with this code, which runs absolutely fine with two passes.
This is the complete script:


AviSource("E:\blabla.avi")
ExactDedup(firstpass=true, dupinfo="Z:\Temp\dupinfo.txt", times="Z:\Temp\times.txt", maxdupcount=1000000, keeplastframe=false)
ForceProcessAVI()
ExactDedup(firstpass=false, dupinfo="Z:\Temp\dupinfo.txt", times="Z:\Temp\times.txt", maxdupcount=1000000, keeplastframe=false)
AssumeFPS(2500, 100, true)


After loading the Script into VDub I am getting a pretty long pause (where it obviously does pass 1) and I do find times.txt and dupinfo.txt filled with the right data. However after the pause it says:

ExactDedup: Failed to open dupinfo file for reading.

(Also I have the impression that pass 1 is not faster, maybe even slower than an Analyse Pass 1 in VDub. I read in some post that there should be a significant speed improvement).

Could you help ? Thanx!Sounds like a path temp variable that needs to be set, or something similar.

[Hotte]

mmhhh, does not work without the Z:\Temp\ either
and does work with two separate script runs...

[StainlessS]

Not sure of the cause of problem, but below should work ok (untested).


FN = "E:\blabla.avi"
DUPTXT = "dupinfo.txt"
TIMETXT = "times.txt"
###
DUPTXT = RT_GetFullPathName(DUPTXT)
TIMETXT = RT_GetFullPathName(TIMETXT)
###
AviSource(FN)
ExactDedup(firstpass=true, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)
ForceProcessAVI()
Last=0 # Force close source [EDIT: Probably not necessary due to reopen]
AviSource(FN) # Re-open [EDIT: As per John's suggestion. EDIT: I thought John suggested this, apparently not]
ExactDedup(firstpass=false, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)
AssumeFPS(2500, 100, true)


does not work without the Z:\Temp\ either
No idea what that means unless needs fullpath, so added the RT_GetFullPathName stuff.
Good luck.

EDIT: In my tests was faster than VDub analysis pass, (probably) due to not displaying any frames [EDIT: at all].

[Hotte]

Yeah! That`s it!
Thank you so much.

[StainlessS]

I'll have to look at the not closing thing, think maybe the "keeplastframe=false" is related, ForceWriteAVI does something to
close things (cant remember exactly what it does) when last frame processed, perhaps keeplastframe=false prevented that closure.

[StainlessS]

I think that Avisynth forgets all paths once frame serving begins, so local path might work ok during first pass runtime processing of ForceWriteAVI,
but on 2nd pass, it has forgotten all paths (they no longer exist for avisynth), that probably explains your previously noted problem.
Maybe something like that.
EDIT: Local paths dont seem to work as well as they used to, no idea why but noticed it for about 6 months or more,
maybe I'm imagining things.

[Hotte]

I just fiddled around this code to try to find out what makes a difference.

This also works:

PluginPath = "C:\..."
LoadPlugin(PluginPath+"RT_Stats26.dll")
FN = "E:\sth\blabla.avi"
AviSource(FN)
ExactDedup(firstpass=true, dupinfo="Z:\Temp\dup7.txt", times="Z:\Temp\tim7.txt", maxdupcount=1000000, keeplastframe=false)
ForceProcessAVI()
Last=0 # Force close source
AviSource(FN) # Re-open
ExactDedup(firstpass=false, dupinfo="Z:\Temp\dup7.txt", times="Z:\Temp\tim7.txt", maxdupcount=1000000, keeplastframe=false)
AssumeFPS(2500, 100, true)


And it stops working exactly when you only leave away the Load Plugin RT_Stats26.dll.
Even though I do not use the Plugin`s command anymore, it does some magic...

[StainlessS]

And it stops working exactly when you only leave away the Load Plugin RT_Stats26.dll.

What does that mean ?
Just loading the plugin should not make any difference.

EDIT: As I said earlier, some scripts that used to work some time ago, seem to have problems with local paths,
absolutely no idea why (a source filter for VDub using ffmpeg code, used to produce weird problems that interfered
with VDub and Avisynth, some years ago, I stopped using that source filter, wonder if VirtualDubFilterMod has
re-introduced a similar problem source filter).
EDIT: I think it changed the search order of paths.

[Hotte]

It does. It exactly makes the difference. Don´t know why but it is reproducable.

[Hotte]

Just to mention: I am not using a path variable or something for my plugins (don`t know how that autoload stuff works reelly). I always load the plugins I refer to at the beginning of my scripts.
Need to stop for today. Thanks again.

[StainlessS]

Yep, but still, "And it stops working" what does that mean, any error message ???

EDIT: You just put plugins in the Plugins directory (Avs+ has several I think), no other requirements for autoload.

[Hotte]

If I load the plugin, as you can see in my code example, it will work fine.
If comment-out the loadPlugin-Line it yields the ExactDedup: Failed to open dupinfo file for reading message from above right after pass1.

Btw: Pass1 is faster than Vdub. Had a wrong impression. It's fast and fine.

[StainlessS]

No idea.
Go get some sleep :)

[Hotte]

StainlessS, if you are listening out there....
I am about to move to Avisynth+ 64bit
The brilliant solution you found to make ExactDedup operate as a one-pass solution seems to need the plugin "TWriteAVI.dll" to be able to use the command "ForceProcessAVI()"
However it pretty much looks like TWriteAVI.dll does not exits as x64 plugin.
Do you have an idea what could be done about it or if there is a workaround?
Thanks!

[StainlessS]

From Here:- https://forum.doom9.org/showthread.php?p=1846838#post1846838

As time (& capability) permits I am converting all to x64 (unfortunately, me gots bout 100 projects currently active :( )
Currently, I think only CallCmd not implemented in x64, and also TWriteAVI v2.0, CallCmd no real problem, in fact probably
just needs changed headers and a bit of fiddling, and should be OK.
TWriteAVI however, is beyond me, I identified about 4 problems in source, (1 fixed here locally), but other 3 I have no idea
how to approach, (2 I think were CPP related, I aint a CPP programmer), and 1 was the fact that there is a lot of x86
ASM used, me dont talk dat lingo, so no go. Maybe someone else more at home with CPP and x86 ASM could have a bash.

ASM the main problem. Source is in the zip on Mediafire.

EDIT: Although RT_Stats v2.0 Beta has this (an almost drop in replacement).

RT_ForceProcess(clip, bool "Video"=True,bool "Audio"=false,Bool "Debug"=True)
If Video (default true), Force read every frame of clip from 0 to FrameCount-1.
If Audio (default false), Force read every sample of clip from 0 to NumberOfSamples-1.
Useful where a clip outputs some kind of file for use in a second filter, this function would in such a case
forcibly read and therefore write the file, so that it may be available to other filters or to a second pass of the filter that
initially wrote the file.
Debug, default true. Write Progress to DebugView.
Returns only after completed force reading of video frames and/or audio samples.
Returns 0.


EDIT: Here tis (RT_Stats v2.0 Beta11, or on MediaFire below in my sig):- http://www.mediafire.com/file/dhrrngn3g4iov1r/RT_Stats_25%2626_dll_v2.00Beta11_20170910.zip
For avs v2.58, v2.6/+, v2.6 x64.


EDIT:
solution you found to make ExactDedup operate as a one-pass solution
Is this the thread you speak of ?:- https://forum.doom9.org/showthread.php?p=1768936#post1768936
RT_ForceProcess can take place of ForceProcessAVI, but cannot write AVI files.
Post a bit of script that you want it for.

[Hotte]

This is basically the code you suggested earlier and it works just simply great with Avisynth 32bit!

Now I need a 64-bit solution:

The idea is: The file is being deduped, before other filters are being applied. The whole thing runs in only one pass although ExactDeDup is a two pass filter:


PluginPath = "C:\Portables\AvisynthPlugins\"
LoadPlugin(PluginPath+"ExactDedup.dll")
LoadPlugin(PluginPath+"TWriteAVI.dll")
LoadPlugin(PluginPath+"RT_Stats26.dll")


FILNAM = "F:\Ausgabe\Export.avi"
DUPTXT = "Z:\Temp\~dupinfo1.txt"
DUPTXT = RT_GetFullPathName(DUPTXT)
TIMETXT = "Z:\Temp\~times1.txt"
TIMETXT = RT_GetFullPathName(TIMETXT)

Avisource(FILNAM)
ExactDedup(firstpass=true, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)
ForceProcessAVI()
Last=0

Avisource(FILNAM)
LanczosResize(1920,1080)
#etc

[StainlessS]

Just try dropping RT_ForceProcess in place of ForceProcessAVI() and see what happens (think should be ok).

EDIT:
Here, ExectDeDup x86 and x64, and RT_Stats v2.0 beta 11 in my prev post.
http://www.mediafire.com/file/8184da6dp4hsdcq/ExactDeDup-0.05_20171005.zip/file

Remove TWriteAVI dll load, and you should be good to go (assuming it all works ok in 32 bit).

[Hotte]

Works perfectly! As I always said: StainlessS is the man!!!

:thanks:

Full Example for anyone interested:
A 25p clip was part of a 29.97p NLE-Editor project and exported with duplicate frames to match 29.97p.
The following script takes the 29.97p clip as input, removes all duplicate frames and applies Interframe to interpolate up from 25p to 29.97p.
Thanks to StainlessS ExactDeDup prescan is fully incorporated and the whole script runs in ONE GO.

FILNAM = "Z:\Temp\Ausgabe\Export.avi"
DUPTXT = "Z:\Temp\~dupinfo1.txt"
DUPTXT = RT_GetFullPathName(DUPTXT)
TIMETXT = "Z:\Temp\~times1.txt"
TIMETXT = RT_GetFullPathName(TIMETXT)

Avisource(FILNAM)
Trim(0,134)
ExactDedup(firstpass=true, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)
RT_ForceProcess()
Last=0

Avisource(FILNAM)
clp = Trim(0,134)
clp = ExactDedup(clp, firstpass=false, dupinfo=DUPTXT, times=TIMETXT, maxdupcount=1000000, keeplastframe=false)
clp = ConvertToYV12(clp)
clp = AssumeFPS(clp,25,false)
clp = InterFrame(clp, GPU=true, Tuning="Film", OverrideAlgo=23, blksize=08, blksizeH=08, NewNum=299700, NewDen=10000, Cores=1)
clp = AssumeFPS(clp,29.97,false)

Return(clp)

[StainlessS]

On 2nd pass, in Interframe might be able to set Cores=whatever, probably required 1 if using TWriteAVI to write an AVI
but as you aint doing that, probably OK to up it a bit for a little more speed, give it a bash.

[Hotte]

Just tried cores=4, but it made no difference.

[StainlessS]

Of well, nothing ventured ...

[Yanak]

Nice solution for a 1 pass run, i use avsmeter ( same with avs2pipemod64 -benchmark script.avs) to execute the first pass and collect the data, ( task is automated in Staxrip "events commands" when a specific template is selected ) but i have to try this RT_ForceProcess() solution and will probably adopt it in my workflow, speed to make the first pass using this instead of one of the exe's should be the same i guess, but will be more versatile and easier :p

And i can also give a try a recompiling ExectDeDup_x64 when I'll have a bit of time to kill and see if there is any gain of FPS, if you are OK with it StainlessS, don't want to step on your toes :)

Thanks for this elegant 1 pass solution, you always come with brilliant ideas to make our work easier

[StainlessS]

ForceProcessAVI, scanned clip faster than VirtualDub's "Run video Analysis Pass" (about 70.0% of VD time), and as RT_ForceProcess is very much the
same, I expect that it to will compare favourably. (test on VD might have been with video frames being shown about once every second, as would be
usual, although I might have switched off all video preview, not sure, if anyone does another test, post results).

[Yanak]

Tried in Vdub2 a small duration but heavy video, 2mn36, 1080p 60 fps, 6.09GB, recorded wiht Fraps = very high bitrate, 334mb/s reported by mediainfo, time to analyze it took 1mn25 with previews off.


Quick run of a normal first pass of ExectDeDup in AVSmeter took 2mn08 , then tested with RT_ForceProcess and had to use a stopwatch as avsmeter during this first pass only displays Loading Script and start to run metrics only on the second pass, but it was more or less the same given one or 2 seconds difference, probably due to the reaction time on the stopwatch :p

Then tried a script with the video as avisource and RT_ForceProces() only in avsmeter ( with new -o option to not run the starting tests so it starts immediately ),
avisource=("clip.avi")
RT_ForceProcess()
Last

again it says "Loading Script..." while executing RT_ForceProcess and starts metrics after that when the script arrives at the line with "Last", timed with stopwatch from the launch until it start the metrics and it took 1mn16 there.

Seems like it's faster by a a little bit only but probably due to the nature of those particular high bitrates videos tested, it might be the getting old SSD drive speed not following, will have to try this placing the videos in my Ramdisk if they can fit there, and test with lighter bitrate videos too...

You didn't answered but I'll try to compile it and see if there is any gain at all, who knows, just need a bit of free time later today or in the next days to run some more tests on different type of videos and I'll report back here.

[StainlessS]

# See DebugView, with DebugView options 'Clock Time', and 'Show Millisecs' ticked.

AviSource(".\G.avi")
#ColorBars(Pixel_Type="YV12").ShowFrameNumber
Trim(0,-10000) # 10,000 frames
RT_ForceProcess
return MessageClip("ALL DONE").Trim(0,-1)



00000005 12:42:07.125 [3404] RT_ForceProcess: Commencing Forced process
00000006 12:42:07.437 [3404] RT_ForceProcess: 187] 4.98% nFrms=188 T=0.313sec : 600.64FpS 0.001665SpF
00000007 12:42:07.828 [3404] RT_ForceProcess: 376] 9.99% nFrms=189 T=0.390sec : 484.62FpS 0.002063SpF
00000008 12:42:08.140 [3404] RT_ForceProcess: 565] 14.99% nFrms=189 T=0.313sec : 603.83FpS 0.001656SpF
00000009 12:42:08.484 [3404] RT_ForceProcess: 754] 20.00% nFrms=189 T=0.343sec : 551.02FpS 0.001815SpF
00000010 12:42:08.781 [3404] RT_ForceProcess: 942] 24.98% nFrms=188 T=0.297sec : 633.00FpS 0.001580SpF
00000011 12:42:09.093 [3404] RT_ForceProcess: 1131] 29.99% nFrms=189 T=0.313sec : 603.83FpS 0.001656SpF
00000012 12:42:09.437 [3404] RT_ForceProcess: 1320] 34.99% nFrms=189 T=0.344sec : 549.42FpS 0.001820SpF
00000013 12:42:09.750 [3404] RT_ForceProcess: 1509] 40.00% nFrms=189 T=0.312sec : 605.77FpS 0.001651SpF
00000014 12:42:10.093 [3404] RT_ForceProcess: 1697] 44.98% nFrms=188 T=0.344sec : 546.51FpS 0.001830SpF
00000015 12:42:10.468 [3404] RT_ForceProcess: 1886] 49.99% nFrms=189 T=0.375sec : 504.00FpS 0.001984SpF
00000016 12:42:10.875 [3404] RT_ForceProcess: 2075] 54.99% nFrms=189 T=0.406sec : 465.52FpS 0.002148SpF
00000017 12:42:11.265 [3404] RT_ForceProcess: 2264] 60.00% nFrms=189 T=0.391sec : 483.38FpS 0.002069SpF
00000018 12:42:11.625 [3404] RT_ForceProcess: 2452] 64.98% nFrms=188 T=0.359sec : 523.68FpS 0.001910SpF
00000019 12:42:11.953 [3404] RT_ForceProcess: 2641] 69.99% nFrms=189 T=0.328sec : 576.22FpS 0.001735SpF
00000020 12:42:12.250 [3404] RT_ForceProcess: 2830] 74.99% nFrms=189 T=0.297sec : 636.36FpS 0.001571SpF
00000021 12:42:12.515 [3404] RT_ForceProcess: 3019] 80.00% nFrms=189 T=0.266sec : 710.53FpS 0.001407SpF
00000022 12:42:12.781 [3404] RT_ForceProcess: 3207] 84.98% nFrms=188 T=0.265sec : 709.43FpS 0.001410SpF
00000023 12:42:13.093 [3404] RT_ForceProcess: 3396] 89.99% nFrms=189 T=0.313sec : 603.83FpS 0.001656SpF
00000024 12:42:13.453 [3404] RT_ForceProcess: 3585] 94.99% nFrms=189 T=0.359sec : 526.46FpS 0.001899SpF
00000025 12:42:13.765 [3404] RT_ForceProcess: 3774] 100.00% nFrms=189 T=0.313sec : 603.83FpS 0.001656SpF
00000026 12:42:13.765 [3404] RT_ForceProcess: Time=6.641secs (0.111mins) : Avg 568.438FpS 0.001759SpF

EDIT: Audio=False is default in RT_ForceProcess (so not force scanning audio).
EDIT: Debug=True is default in RT_ForceProcess as above, (cant change your mind and switch on, half way through).
EDIT: RT_ForceProcess Debug==true, outputs progress every ~ 5.0%.
EDIT: Maybe try without including timing for ExactDeDup or whatever (would just add a fixed amount to all timings probably)
EDIT: All RT_ForceProcess does is request frames sequentially (same as player), but without and/or resize/colorspace_convert/display.

EDIT: G.avi only had 3775 frames (not 10000 frames), did mod to script after did timings.

[Yanak]

Tested on my clip, it have only 9366 frames :
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 6.10 GiB
Duration : 2 min 36 s
Overall bit rate : 336 Mb/s

Video
ID : 0
Format : Fraps
Codec ID : FPS1
Duration : 2 min 36 s
Bit rate : 334 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 2.685
Stream size : 6.07 GiB (100%)
00000001 14:08:14.722 [7540] RT_ForceProcess: Commencing Forced process
00000002 14:08:18.894 [7540] RT_ForceProcess: 467] 5.00% nFrms=468 T=4.172sec : 112.18FpS 0.008915SpF
00000003 14:08:23.167 [7540] RT_ForceProcess: 935] 9.99% nFrms=468 T=4.273sec : 109.52FpS 0.009130SpF
00000004 14:08:27.652 [7540] RT_ForceProcess: 1403] 14.99% nFrms=468 T=4.485sec : 104.35FpS 0.009583SpF
00000005 14:08:31.919 [7540] RT_ForceProcess: 1872] 20.00% nFrms=469 T=4.266sec : 109.94FpS 0.009096SpF
00000006 14:08:36.067 [7540] RT_ForceProcess: 2340] 24.99% nFrms=468 T=4.149sec : 112.80FpS 0.008865SpF
00000007 14:08:39.883 [7540] RT_ForceProcess: 2808] 29.99% nFrms=468 T=3.816sec : 122.64FpS 0.008154SpF
00000008 14:08:43.239 [7540] RT_ForceProcess: 3277] 35.00% nFrms=469 T=3.356sec : 139.75FpS 0.007156SpF
00000009 14:08:46.919 [7540] RT_ForceProcess: 3745] 40.00% nFrms=468 T=3.680sec : 127.17FpS 0.007863SpF
00000010 14:08:50.438 [7540] RT_ForceProcess: 4213] 44.99% nFrms=468 T=3.519sec : 132.99FpS 0.007519SpF
00000011 14:08:53.684 [7540] RT_ForceProcess: 4682] 50.00% nFrms=469 T=3.246sec : 144.49FpS 0.006921SpF
00000012 14:08:57.674 [7540] RT_ForceProcess: 5150] 55.00% nFrms=468 T=3.990sec : 117.29FpS 0.008526SpF
00000013 14:09:01.440 [7540] RT_ForceProcess: 5618] 59.99% nFrms=468 T=3.765sec : 124.30FpS 0.008045SpF
00000014 14:09:04.858 [7540] RT_ForceProcess: 6086] 64.99% nFrms=468 T=3.419sec : 136.88FpS 0.007306SpF
00000015 14:09:08.440 [7540] RT_ForceProcess: 6555] 70.00% nFrms=469 T=3.582sec : 130.93FpS 0.007638SpF
00000016 14:09:12.074 [7540] RT_ForceProcess: 7023] 74.99% nFrms=468 T=3.634sec : 128.78FpS 0.007765SpF
00000017 14:09:15.581 [7540] RT_ForceProcess: 7491] 79.99% nFrms=468 T=3.507sec : 133.45FpS 0.007494SpF
00000018 14:09:18.986 [7540] RT_ForceProcess: 7960] 85.00% nFrms=469 T=3.405sec : 137.74FpS 0.007260SpF
00000019 14:09:23.648 [7540] RT_ForceProcess: 8428] 90.00% nFrms=468 T=4.662sec : 100.39FpS 0.009962SpF
00000020 14:09:27.958 [7540] RT_ForceProcess: 8896] 94.99% nFrms=468 T=4.310sec : 108.58FpS 0.009209SpF
00000021 14:09:31.082 [7540] RT_ForceProcess: 9365] 100.00% nFrms=469 T=3.124sec : 150.13FpS 0.006661SpF
00000022 14:09:31.082 [7540] RT_ForceProcess: Time=76.360secs (1.273mins) : Avg 122.656FpS 0.008153SpF

Pretty close to what vdub2 does in the analyze pass

and then used an avi having almost the double bitrate than this & 1mn less length, 5433 frames :
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 6.54 GiB
Duration : 1 min 30 s
Overall bit rate : 621 Mb/s

Video
ID : 0
Format : Fraps
Codec ID : FPS1
Duration : 1 min 30 s
Bit rate : 619 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 4.977
Stream size : 6.53 GiB (100%)
00000001 14:10:48.179 [6688] RT_ForceProcess: Commencing Forced process
00000002 14:10:51.657 [6688] RT_ForceProcess: 270] 4.99% nFrms=271 T=3.477sec : 77.94FpS 0.012830SpF
00000003 14:10:55.768 [6688] RT_ForceProcess: 542] 9.99% nFrms=272 T=4.111sec : 66.16FpS 0.015114SpF
00000004 14:10:59.669 [6688] RT_ForceProcess: 813] 14.98% nFrms=271 T=3.901sec : 69.47FpS 0.014395SpF
00000005 14:11:03.161 [6688] RT_ForceProcess: 1085] 19.99% nFrms=272 T=3.492sec : 77.89FpS 0.012838SpF
00000006 14:11:07.179 [6688] RT_ForceProcess: 1357] 25.00% nFrms=272 T=4.019sec : 67.68FpS 0.014776SpF
00000007 14:11:11.235 [6688] RT_ForceProcess: 1628] 29.98% nFrms=271 T=4.055sec : 66.83FpS 0.014963SpF
00000008 14:11:15.090 [6688] RT_ForceProcess: 1900] 34.99% nFrms=272 T=3.855sec : 70.56FpS 0.014173SpF
00000009 14:11:19.223 [6688] RT_ForceProcess: 2172] 40.00% nFrms=272 T=4.132sec : 65.83FpS 0.015191SpF
00000010 14:11:22.854 [6688] RT_ForceProcess: 2443] 44.98% nFrms=271 T=3.632sec : 74.61FpS 0.013402SpF
00000011 14:11:26.418 [6688] RT_ForceProcess: 2715] 49.99% nFrms=272 T=3.563sec : 76.34FpS 0.013099SpF
00000012 14:11:30.278 [6688] RT_ForceProcess: 2987] 55.00% nFrms=272 T=3.861sec : 70.45FpS 0.014195SpF
00000013 14:11:33.123 [6688] RT_ForceProcess: 3258] 59.99% nFrms=271 T=2.845sec : 95.25FpS 0.010498SpF
00000014 14:11:34.774 [6688] RT_ForceProcess: 3530] 64.99% nFrms=272 T=1.650sec : 164.85FpS 0.006066SpF
00000015 14:11:37.649 [6688] RT_ForceProcess: 3802] 70.00% nFrms=272 T=2.876sec : 94.58FpS 0.010574SpF
00000016 14:11:40.830 [6688] RT_ForceProcess: 4073] 74.99% nFrms=271 T=3.182sec : 85.17FpS 0.011742SpF
00000017 14:11:43.950 [6688] RT_ForceProcess: 4345] 79.99% nFrms=272 T=3.119sec : 87.21FpS 0.011467SpF
00000018 14:11:46.963 [6688] RT_ForceProcess: 4617] 85.00% nFrms=272 T=3.013sec : 90.28FpS 0.011077SpF
00000019 14:11:50.063 [6688] RT_ForceProcess: 4888] 89.99% nFrms=271 T=3.100sec : 87.42FpS 0.011439SpF
00000020 14:11:53.242 [6688] RT_ForceProcess: 5160] 94.99% nFrms=272 T=3.179sec : 85.56FpS 0.011687SpF
00000021 14:11:56.164 [6688] RT_ForceProcess: 5432] 100.00% nFrms=272 T=2.922sec : 93.09FpS 0.010743SpF
00000022 14:11:56.164 [6688] RT_ForceProcess: Time=67.984secs (1.133mins) : Avg 79.916FpS 0.012513SpF


I'll have to dig but i probably don't have much avi's with more reasonable bitrates on my PC, at least no 1080p ones to compare for sure, maybe some low resolution ones stored somewhere, will need ot go search for samples on the net i guess.

I can use other containers but i'd like to keep trying the avisource and not trigger an external plugin needed for other containers that could itself be a limiter, but will do later when testing ExactDedup tho.

Will need a bit of time to run some of the other tests now :p

I will need a bit of time and see what i get.

[StainlessS]

Only (real I think) diff between ForceProcessAVI (from TWriteAVI v2.0) and RT_ForceProcess is later returns 0, former returns original clip.
RUSBBISH, SEE BELOW

Am posting stand alone ForceProcess dll now, x86/x64

EDIT: Ignore above, ForceProcessAVI also returns 0, not a clip (just checked). Think it originally returned a clip but
caused problems during development, and so changed behaviour. So, assigning Last=0 after call to ForceProcessAVI
is doing nothing useful (It used to call filter destructor, it is no longer a filter, just a function).

EDIT: Very first instance of ForceProcess was a GScript function which just sampled a single pixel at 0,0 from each frame.

This would work (I think) with only GScript/avs+

Function FP(clip c) {
# GSCript""" # Uncomment for non AVS+
c2=c.Crop(0,0,16,16)
for(i=0,c2.FrameCount-1) {
dummy = c2.Trim(i,-1).AverageLuma # "dummy =" Not necessary
}
# """ # Uncomment for non AVS+
return 0
}

.

[Yanak]

Thanks a lot,

I will download ForceProcess and run some tests to see if there is any difference, but i think the problem is those fat bitrate videos and hardware limitations, that's why i need to run tests on lot of different sources, then once done and data accumulated I'll compile ExactDedup_x64 and will do the tests again to compare, but probably not today tho, it's way too hot inside and outside already to make some benchs and stay here while the computer is loaded and adding more heat to a room :p But be sure that i will do this soon :)

Thanks again for all you do StainlessS.

[StainlessS]

Not sure, I may have added this bit after timing, dont know.


if((n & 0x7F)==0) {
Sleep(0); // yield process for a little bit
}

Also, I would be timing with either SD UT_Video "ULY0" or maybe even the colorbars.ShowFrameNumber thing.

[Yanak]

Just tried it, results are mostly the same, but while at it decided to test something else and i might have found something :

ColorBars(width = 1920, height = 1080, pixel_type = "RGB32", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999)
ForceProcess()
Last
Gives this :
00000023 18:12:25.637 [5760] ForceProcess: Commencing Forced process
00000024 18:12:25.918 [5760] ForceProcess: 499] 5.00% nFrms=500 T=0.280sec : 1785.71FpS 0.000560SpF
00000025 18:12:26.183 [5760] ForceProcess: 999] 10.00% nFrms=500 T=0.266sec : 1879.70FpS 0.000532SpF
00000026 18:12:26.464 [5760] ForceProcess: 1499] 15.00% nFrms=500 T=0.280sec : 1785.71FpS 0.000560SpF
00000027 18:12:26.744 [5760] ForceProcess: 1999] 20.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000028 18:12:27.010 [5760] ForceProcess: 2499] 25.00% nFrms=500 T=0.265sec : 1886.79FpS 0.000530SpF
00000029 18:12:27.290 [5760] ForceProcess: 2999] 30.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000030 18:12:27.556 [5760] ForceProcess: 3499] 35.00% nFrms=500 T=0.265sec : 1886.79FpS 0.000530SpF
00000031 18:12:27.836 [5760] ForceProcess: 3999] 40.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000032 18:12:28.102 [5760] ForceProcess: 4499] 45.00% nFrms=500 T=0.265sec : 1886.79FpS 0.000530SpF
00000033 18:12:28.382 [5760] ForceProcess: 4999] 50.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000034 18:12:28.663 [5760] ForceProcess: 5499] 55.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000035 18:12:28.928 [5760] ForceProcess: 5999] 60.00% nFrms=500 T=0.265sec : 1886.79FpS 0.000530SpF
00000036 18:12:29.209 [5760] ForceProcess: 6499] 65.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000037 18:12:29.474 [5760] ForceProcess: 6999] 70.00% nFrms=500 T=0.265sec : 1886.79FpS 0.000530SpF
00000038 18:12:29.755 [5760] ForceProcess: 7499] 75.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000039 18:12:30.020 [5760] ForceProcess: 7999] 80.00% nFrms=500 T=0.265sec : 1886.79FpS 0.000530SpF
00000040 18:12:30.301 [5760] ForceProcess: 8499] 85.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000041 18:12:30.582 [5760] ForceProcess: 8999] 90.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000042 18:12:30.847 [5760] ForceProcess: 9499] 95.00% nFrms=500 T=0.265sec : 1886.79FpS 0.000530SpF
00000043 18:12:31.128 [5760] ForceProcess: 9999] 100.00% nFrms=500 T=0.281sec : 1779.36FpS 0.000562SpF
00000044 18:12:31.128 [5760] ForceProcess: Time=5.491secs (0.092mins) : Avg 1821.162FpS 0.000549SpF
And this :
ColorBars(width = 1920, height = 1080, pixel_type = "YV12", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999)
ForceProcess()
Last
gives this :
00000045 18:13:01.738 [5096] ForceProcess: Commencing Forced process
00000046 18:13:01.800 [5096] ForceProcess: 499] 5.00% nFrms=500 T=0.063sec : 7936.51FpS 0.000126SpF
00000047 18:13:01.878 [5096] ForceProcess: 999] 10.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000048 18:13:01.941 [5096] ForceProcess: 1499] 15.00% nFrms=500 T=0.062sec : 8064.52FpS 0.000124SpF
00000049 18:13:02.019 [5096] ForceProcess: 1999] 20.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000050 18:13:02.081 [5096] ForceProcess: 2499] 25.00% nFrms=500 T=0.062sec : 8064.52FpS 0.000124SpF
00000051 18:13:02.159 [5096] ForceProcess: 2999] 30.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000052 18:13:02.221 [5096] ForceProcess: 3499] 35.00% nFrms=500 T=0.063sec : 7936.51FpS 0.000126SpF
00000053 18:13:02.299 [5096] ForceProcess: 3999] 40.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000054 18:13:02.362 [5096] ForceProcess: 4499] 45.00% nFrms=500 T=0.062sec : 8064.52FpS 0.000124SpF
00000055 18:13:02.440 [5096] ForceProcess: 4999] 50.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000056 18:13:02.502 [5096] ForceProcess: 5499] 55.00% nFrms=500 T=0.063sec : 7936.51FpS 0.000126SpF
00000057 18:13:02.580 [5096] ForceProcess: 5999] 60.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000058 18:13:02.643 [5096] ForceProcess: 6499] 65.00% nFrms=500 T=0.062sec : 8064.52FpS 0.000124SpF
00000059 18:13:02.721 [5096] ForceProcess: 6999] 70.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000060 18:13:02.783 [5096] ForceProcess: 7499] 75.00% nFrms=500 T=0.062sec : 8064.52FpS 0.000124SpF
00000061 18:13:02.861 [5096] ForceProcess: 7999] 80.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000062 18:13:02.923 [5096] ForceProcess: 8499] 85.00% nFrms=500 T=0.063sec : 7936.51FpS 0.000126SpF
00000063 18:13:03.001 [5096] ForceProcess: 8999] 90.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000064 18:13:03.064 [5096] ForceProcess: 9499] 95.00% nFrms=500 T=0.062sec : 8064.52FpS 0.000124SpF
00000065 18:13:03.142 [5096] ForceProcess: 9999] 100.00% nFrms=500 T=0.078sec : 6410.26FpS 0.000156SpF
00000066 18:13:03.142 [5096] ForceProcess: Time=1.404secs (0.023mins) : Avg 7122.507FpS 0.000140SpF
YV12 is way faster compared to RGB32, Fraps videos have 2 modes for recording one is full RGB other is compressed a tiny bit and detected as yuvj420p by ffmpeg if i recall correctly, not sure what option i used when recording those videos and how avisource acts with those, if it treats them differently or not, i have no idea...

PS: i know i don't need the killaudio in the script, just a copy paste from and older test script ^^

Edit: just seeing your reply above after posting this:p

[StainlessS]

ColorBars ( [ int width, int height, string pixel_type, bool staticframes ] ) AVS+

bool staticframes = true
AVS+ If set to false, generate all frames. Default true (one static frame is served)


http://avisynth.nl/index.php/ColorBars

Thats handy to have, was not aware. (Better than Colorbars.ShowFrameNumber)

You do same test with Vdub ?

EDIT:
In ClipBlend plugin, when doing trim on last frame eg

return Last.ClipBlend.Trim(FrameCount-1,-1)

Force reads every frame returning only a single clip global blended frame, was quite a lot faster than using
VDub Analysis pass. (and it was not just eating ColorBars, but VD analysis would be forcing intermediate frame results)

EDIT: Well, above Clipblend thing probably not a fair comparison, we skipped creating intermediate result frame when not necessary.

[Yanak]

I'm happy to have made you discover something ^^

In vdub i just fed the avi video itself, not the avs script with avisource= , and had not tried the colorbas before, need to try it too.

Btw tried same tests as above but with .ShowFrameNumber at the end of colorbars line:

YV12:
ForceProcess: Time=243.111secs (4.052mins) : Avg 41.133FpS

RGB32:
ForceProcess: Time=247.541secs (4.126mins) : Avg 40.397FpS

only a slight difference but repeated the tests twice and it's constant.

Edit : in vdub2 the colorbars run at 265fps and take 38s for the RGB32, without the .ShowFrameNumber at the end of course, in comparison to 5.491s so yeah forceprocess is not only a bit faster here :p
and around 36FPS with .ShowFrameNumber at the end and 4mn32 to analyze

and for the YV12 it brings me an error message when opening the script and tells me it could not locate a decompressor for YV12... idk...

Will test with different sources in the next days, then test ExactDedup itself and a version self compiled in case there is some gains like some other things i tried before, but for now i think ForceProcess won't be any faster, seems like i reach the hardware limitations and it proved it can run really fast.

Thanks a lot for all will report back when i'll have done some more deep testings, life will already be easier for me with ForceProcess when using ExactDedup

[StainlessS]

could not locate a decompressor for YV12

Same with Standard VDub ?


EDIT:
Clearly faster than my crap m/c (I just used RT_ version)


RGB32: RT_ForceProcess: Time=31.172secs (0.520mins) : Avg 320.801FpS 0.003117SpF

YV12: RT_ForceProcess: Time=7.563secs (0.126mins) : Avg 1322.227FpS 0.000756SpF

[Yanak]

Had not come to my mind to try it, thanks, it runs in old vdub x64 at 435-450 FPS and takes 23s , again a big difference between YV12 and RGB32 not sure why.

Thank you.

Edit: the RGB32 in old vdub is even slower than on vdub2 : 135 FPS and 1mn13 to finish o_O vdub2 run it at 265 fps...

[StainlessS]

When I did timing stuff was certainly on old VD, so that explains a lot.

[Yanak]

Yes it certainly can explain a lot of things, either earlier with Rt_Stats or in vdub2 there was basically no differences on the fat avi's i tested, and almost no differences between Rt_Stats and ForceProcess either, but i bet those videos reached hardware limitations due to the very high bitrate they have.

I will have to spend time and test with more reasonable content, probably using DDS2 that usually is faster than FFVideoSource or LSMASHVideoSource /LSMASHVideoSource but i'll probably test a few of those anyways, then i'll focus on ExactDedup to see if it can gain a bit once recompiled.

But don't get me wrong, ForceProcess is a perfect new tool to play with, will certainly help a lot in automating things when it needs multiples passes to collect data, a big thanks for this new toy :)

[StainlessS]

Yep well, I just thought that TWriteAVI aint gonna be an option for x64 anytime soon, and RT_Stats v2.0 beta might not be acceptable to some,
as beta (although I use nothing but that usually).
13KB for x64 version is also nice size.

[Yanak]

I like to have a plugin dedicated to one task :p RT_Stats is like a Swiss army knife but each time i start to read the doc i end up lost into it due to the too numerous possibilities this thing offers ;)

Will take a little break for today and test all this stuff in the next days then post here when i'll be done,
Thanks.

Edit: just saw your post #40 edited, yup it's faster but getting old now too, just a 3770k slightly clocked, wish i could afford one of those last beasty CPU's and build a new machine... encoding times are often a pain in the rear, hence why i try to re-compile some plugins and get some speed boosts here and there... and let's not even talk about x265 encoding, in slow mode i could probably draw paintings and make a comic book with them before it ends :p ( and it got worse with last win updates for mitigating the CPU attacks... )

[Yanak]

Started to test a bit on very heavy videos like the examples i posted before where the hardware seems to be the limitation, where there was not much differences ,if any, between vdub2, Rt_Stats or ForceProcess during the analyze pass, what was limiting me there wasn't the methods and programs used i mean.

Then made a few tests using ExactDedup and compared to a quickly version i compiled, interestingly enough the results obtained come closer to the hardware or my system limitations seen when only running the video in analyze mode in vdub2 or like in previous page in a avs script using ForceProcess but without ExactDedup involved.

#### Script ###
AviSource("video.avi", Audio = False)
#1st Pass
ExactDedup(firstpass=true, dupinfo="dupinfo.txt", times="times.txt", maxdupcount=1000000, keeplastframe=false, show=false)
******************************************************************
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 6.10 GiB
Duration : 2 min 36 s
Overall bit rate : 336 Mb/s
Format : Fraps
Codec ID : FPS1
Duration : 2 min 36 s
Bit rate : 334 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 2.685
Stream size : 6.07 GiB (100%)
******************************************************************
########### 1st pass Original dll ##########
Frames processed: 9366 (0 - 9365)
FPS 73.12
Time (elapsed): 00:02:08.086
########### 1st pass Compiled dll ##########
Frames processed: 9366 (0 - 9365)
FPS 101.9
Time (elapsed): 00:01:31.931

########### 2nd pass Original dll ##########
Frames processed: 5647 (0 - 5646)
FPS 83.01
Time (elapsed): 00:01:08.028
########### 2nd pass Compiled dll ##########
Frames processed: 5647 (0 - 5646)
FPS 83.05
Time (elapsed): 00:01:07.999
******************************************************************

Let's compare with results from the previous page using only ForceProcess, without ExactDedup in script :
RT_ForceProcess: Time=76.360secs (1.273mins) : Avg 122.656FpS 0.008153SpF

Another Example but here gains are smaller :
******************************************************************
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 6.54 GiB
Duration : 1 min 30 s
Overall bit rate : 621 Mb/s
Format : Fraps
Codec ID : FPS1
Duration : 1 min 30 s
Bit rate : 619 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 4.977
Stream size : 6.53 GiB (100%)
******************************************************************
########### 1st pass Original dll ##########
Frames processed: 5433 (0 - 5432)
FPS 72.39
Time (elapsed): 00:01:15.048
########### 1st pass Compiled dll ##########
Frames processed: 5433 (0 - 5432)
FPS 77.38
Time (elapsed): 00:01:10.208

########### 2nd pass Original dll ##########
Frames processed: 4934 (0 - 4933)
FPS 74.83
Time (elapsed): 00:01:05.93
########### 2nd pass Compiled dll ##########
Frames processed: 4934 (0 - 4933)
FPS 74.82
Time (elapsed): 00:01:05.948
******************************************************************
From previous page, again without ExactDedup involved :
RT_ForceProcess: Time=67.984secs (1.133mins) : Avg 79.916FpS 0.012513SpF

And a last one :
******************************************************************
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 3.88 GiB
Duration : 55 s 933 ms
Overall bit rate : 596 Mb/s
Format : Fraps
Codec ID : FPS1
Duration : 55 s 933 ms
Bit rate : 594 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 60.000 FPS
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 4.778
Stream size : 3.87 GiB (100%)
******************************************************************
########### 1st pass Original dll ##########
Frames processed: 3356 (0 - 3355)
FPS 64.34
Time (elapsed): 00:00:52.159
########### 1st pass Compiled dll ##########
Frames processed: 3356 (0 - 3355)
FPS 77.30
Time (elapsed): 00:00:43.414

########### 2nd pass Original dll ##########
Frames processed: 2455 (0 - 2454)
FPS 65.33
Time (elapsed): 00:00:37.579
########### 2nd pass Compiled dll ##########
Frames processed: 2455 (0 - 2454)
FPS 65.32
Time (elapsed): 00:00:37.582
******************************************************************
vs :
ForceProcess: Time=39.608secs (0.660mins) : Avg 84.730FpS 0.011802SpF


A you can see there is gains ( more or less important ) only for the 1st pass, there is no gain at all during the second pass on those fat videos.
Tried also to place one of those in a fast Ramdisk but no differences if analyzed from there vs the SSD or a HDD...

Also quickly tried on less heavy and smaller resolution videos, only a couple of runs and there i cannot see any noticeable difference between the 2 dll's, probably because i don't hit any dll limitations like when using those heavy bitrate ones so the result is the best my system can get with either dll used.

Still a lot of things to test in the next days, then will also try to train a bit the dll with PGO to see if there is more to gain in those heavy conditions, but it's too hot today to continue this right now :p

[Yanak]

Continued and posted the results in Exactdedup dedicated thread here : http://forum.doom9.org/showthread.php?p=1848413#post1848413

Will be easier to find :)