Another frame duplication function

[Corran]

Edit: Other members have since fixed some problems with this script. Check out their contributions here: http://forum.doom9.org/showthread.php?p=1589995#post1589995

--------------

I posted about this at animemusicvdieos.org but the only person I really know that frequents that place from doom9 is Zarxrax... I'm not sure how useful it really is to amv editors, but perhaps someone here may find better a use for it.

This function acts much like Donald Graft's dup() in that it attempts to find frames that are exactly the same as the last frame that changed, a "new frame" so to speak, and then overwrite them with that "new frame".

In other words it works like this:
(Assume captial letters represent frames with actual changes)
AbcdeFGhiJkLmnop

This filter will produce:
AAAAAFGGGJJLLLLL

When the adjusted video is played back it should look exactly the same as the original. This works because most anime does not have new movement in every single frame.

What this ultimately means is that when the filtered footage is sent to the encoder, it is much easier to compress because a lot of the frames are mathematically identical. The noise that occurs between each frame is now limited to the frames that actually change.

Why not just use dup()? Well, firstly I wanted to give myself a challenge and only learned about his plugin until after I had started work. However, I found that his did not accurately identify very small changes in the video. If I adjusted the threshold to identify very slow pans, zooms, or other motion, it would start picking up on the noise in my source. (Honestly I thought this after only a minimal amount of testing, so there may be a better way to use it than I did. In other words, I'm not trying to say that the filter is no good or that it sucks... <3 DG >_>) Therefore my goal with this was to create a function that required little effort to configure assuming a relatively clean source, and the ability to detect small changes accurately.

So here are the details that probably matter to anyone reading this...

How can I use this practically?
The biggest benefit will probably be realized if you are the kind of person that uses avisynth to create lossless filtered AVI files and then use those, not the scripts themselves. The Lagarith codec has a nifty option called null frames.
[When null frames is enabled,] if the previous frame is mathematically identical to the current, the current frame is discarded and the decoder will simply use the previous frame again.
I took The Girl Who Leapt Through Time at 640x368, cleaned it up, and encoded it with lagarith twice using the null frames option. The first time was without dupped() and the second time was with.
The first encode was 10.9GB
The second one was 6.3GB. For a 98 minute lossless encode...

It may also prove useful for fansubbers that use softsubs I guess...

To use this function:

Right Click, Save as, and place in your avisynth plugins directory
dupped.avsi (https://raw.githubusercontent.com/eparsons/AviSynthStuff/master/Scripts/dupped.avsi)

Currently there are only three arguments.
Thresh=20 will determine that a new frame exists when YMinMax(a,b) is >= 20. The value of YMinMax(a,b) can be found by setting stats=true. I've currently set the default for this parameter to 16. I've only used this on one clean source so it probably needs a little tweaking.
panthresh=1.65 is used in calculating the dynamic threshold of low motion scenes. The first two YMinMax values shown in the stats are averaged and then multiplied by panthresh. If the third YMinMax is equal to or greater than this calculation then the current frame is considered a new frame. The dynamic threshold is now displayed in the stats to the right of YMinMax(last,next).
stats=True will display the stats used to determine if there was a change in the frame.
Showme=True will show the video that the stats are created from.

I suggest using this function at the end of your script and only after you have cleaned up your source. Also, the video needs to be in a YUV colorspace. In otherwords, if get an error saying the "Image must be planar" then you need to put converttoyv12() right before the dupped() function.

Here is an example of such usage if needed.
AVISource("Videofile.avi")
converttoyv12()
dupped()
Also, don't use SetMTMode() if you are using a multi-threaded version of avisynth. (If you don't know if you are, then you aren't running it.) Instead, perform multi-threading on a per-filter basis using MT() like this:
AVISource("Videofile.avi")
MT("ttempsmooth()",4)
fft3dgpu()
dupped()

Also, it may be obvious... but make sure stats and showme are not enabled during your actual encodes otherwise the frames won't be identical where they should be. >_>

Here are some screenshots of the stats and showme parameters in use.
dupped(stats=true)
http://www.randomdestination.net/members/corran/misc/dupped/dupped_pan_with_debugging.jpg

dupped(stats=true,showme=true)
http://www.randomdestination.net/members/corran/misc/dupped/dupped_with_debugging.jpg


# Dupped() attempts to make duplicate frames mathematically identical.
#
# This filter requires a YUV source. Use converttoyv12 if needed
# before calling dupped()
#
## Parameters
# thresh = Used to determine when to delare a frame as new
# Use stats=true to help determine the best value for your source
# Lower number = more sensitive. (Default=16)
#
# panthresh=1.65 This is used to determine when to consider a frame
# part of a low motion scene. (Default=1.65)
#
# stats = Enable/disable display of stats. (Default=false | Not shown)
# (Last = last frame, this = this frame, next = next frame)
#
# showme = Enable/disable display of video that most stats are
# derived from. (Default=false | Not shown)
#
## Notes
# Do not use with SetMTMode(). This function requires the frames to be
# accessed sequentially. Instead, use MT() to mulit-thread on a per-filter basis.
#
# Example:
# dupped(thresh=20,panthresh=1.65,stats=true,showme=true)

Function Dupped(clip c, int "thresh", float "panThresh", bool "stats", bool "showme") {
c
Global dupThresh = default(thresh, 16)
Global panThresh = Float(default(panThresh, 1.65))
Global dupStats = default(stats, false)
Showme = default(showme, false)
Global lastNewFrame=0
scriptclip("""
c = Last
n=current_frame
FC = FrameCount

prev_frame = (n > 0) ? n-1 : 0
next_frame = (n < FC-1) ? n+1 : FC-1

#Get Y related stats as needed
prevCurrentYMinMax = Subtract(Trim(prev_frame,-1),c).YPlaneMinMaxDifference
currentNextYMinMax = prevCurrentYMinMax < dupThresh ? Subtract(Trim(next_frame,-1),c).YPlaneMinMaxDifference : 256
prevNextYMinMax = currentNextYMinMax < dupThresh ? Subtract(Trim(prev_frame,-1),Trim(next_frame,-1)).YPlaneMinMaxDifference : 0

#calculate dynamic pan threshold. Set to 256 if looking for a pan is pointless
neighborAvgMinMax = (prevCurrentYMinMax + currentNextYMinMax) / 2.0
dynPanThresh = prevNextYMinMax != 0 ? Abs(neighborAvgMinMax * panthresh) : 256.0

#determine if the current frame is new
IsNewFrame = (prevCurrentYMinMax >= dupThresh || prevNextYMinMax >= dynPanThresh)
Global lastNewFrame = IsNewFrame ? n : lastNewFrame

c = Trim(lastNewFrame,-1)
c = dupStats ? c.subtitle("prevframe: "+string(prev_frame)+" currentframe: "+string(n)+" nextframe: "+string(next_frame)) : c
c = dupStats ? c.subtitle("Last new frame: "+string(lastNewFrame)+" IsNewframe: "+string(IsNewFrame),y=15) : c
c = dupStats ? c.subtitle("prevCurrentYMinMax: "+string(prevCurrentYMinMax)+" (Threshold:"+string(dupThresh)+")",y=45) : c
c = dupStats && currentNextYMinMax != 256 ? c.subtitle("currentNextYMinMax: "+string(currentNextYMinMax),y=60) : c
c = dupStats && prevNextYMinMax >= dynPanThresh ? c.subtitle("prevNextYMinMax: "+string(prevNextYMinMax),y=90) : c
c = dupStats && prevNextYMinMax >= dynPanThresh ? c.subtitle("Pan Threshold: "+string(dynPanThresh)+
\ " ("+string(neighborAvgMinMax)+" * "+string(panThresh)+")",y=105) : c
c = dupStats && prevNextYMinMax >= dynPanThresh ? c.subtitle("Slow pan detected",y=120) : c

return c
""")
Return showme ? stackvertical(Last,subtract(c.duplicateframe(0),c)) : Last
}

[stickboy]

I didn't look at your script too carefully, but some comments:

Nested functions are misleading. AviSynth scoping is pretty basic, and all functions are in the global namespace.

Beware of using Trim when its arguments are variable. (http://forum.doom9.org/showthread.php?s=&threadid=58358)

Expressions like:
booleanExpression ? true : falseare unnecessarily complex. Just use booleanExpression by itself.

[Corran]

I didn't look at your script too carefully, but some comments:

Nested functions are misleading. AviSynth scoping is pretty basic, and all functions are in the global namespace.

Beware of using Trim when its arguments are variable. (http://forum.doom9.org/showthread.php?s=&threadid=58358)

Expressions like:
booleanExpression ? true : falseare unnecessarily complex. Just use booleanExpression by itself.

I am aware of the issues with trim (in fact it was you who last made me aware of it (http://forum.doom9.org/showthread.php?p=580528#post580528) a few years ago), but as far as I've noticed in testing, the videos have not been adversely affected by the way I've used it. As for the boolean, is it really not fast to do it this way than repeating the mathematical expression in multiple conditions? I did it mostly to cut down on the number of times calculations had to be repeated. Otherwise, the script would become much more complex to read and calcuations that I needed to reference multiple times would have to be computed several times before the script was fully completed.

[Adub]

Have you considered making it into a C plugin instead?

[Kumo]

i only get this message on the video preview:"i don't know what newframe means([scriptclip], line 16)"
is anything wrong in the script?

[stickboy]

but as far as I've noticed in testing, the videos have not been adversely affected by the way I've used it.Only because it happens that the videos you've tested it with haven't required replacing frames near the beginning and end, but can you guarantee that you'll never need to replace those?

Maybe you can make that guarantee. I haven't looked at your script carefully enough to tell. Personally I'd rather use my safer versions of Trim() and not have to bother thinking about that stuff.

But if you can make that guarantee, you should enforce it with Assert(). A hard failure at the beginning is much better than allowing people to waste hours encoding videos that they later realize don't contain the correct frames.
As for the boolean, is it really not fast to do it this way than repeating the mathematical expression in multiple conditions?

I did it mostly to cut down on the number of times calculations had to be repeated. Otherwise, the script would become much more complex to read and calcuations that I needed to reference multiple times would have to be computed several times before the script was fully completed.That's not what I'm talking about at all.

If you want to save a boolean expression into a variable, that's perfectly fine. The "? true : false" part is completely useless.
foo = booleanExpression
foo = booleanExpression ? true : false
foo = (booleanExpression ? true : false) ? true : false
foo = ((booleanExpression ? true : false) ? true : false) ? true : falseAre all equivalent.

[Corran]

Have you considered making it into a C plugin instead?

If I knew how I would have definitely gone that route... Unfortunately, my only real programming experience is with using php. I'm self-taught and still rely heavily on the php.net references so I don't think jumping straight into C plugin development would be very fruitful...


i only get this message on the video preview:"i don't know what newframe means([scriptclip], line 16)"
is anything wrong in the script?
Could you please post your script? It sounds like your script is not feeding the video to dupped sequentially. I've noticed this can happen if you use SetMTMode. If this is the case, please see the original post for details on this..

Maybe you can make that guarantee. I haven't looked at your script carefully enough to tell. Personally I'd rather use my safer versions of Trim() and not have to bother thinking about that stuff.
I just made a couple changes to my function that will prevent the replacement of the first and last frames. Previously, I had been doing a check for the first frame to make sure the newframe variable was set to something, but I hadn't actually told it to not replace the frame... ...in fact every frame was previously getting replaced whether it needed to or not which was a rather large waste of cpu time. -_-;; Needless to say, the function runs much faster now...

If you want to save a boolean expression into a variable, that's perfectly fine. The "? true : false" part is completely useless.
foo = booleanExpression
foo = booleanExpression ? true : false
foo = (booleanExpression ? true : false) ? true : false
foo = ((booleanExpression ? true : false) ? true : false) ? true : falseAre all equivalent.

Ah, I understand now... thanks for explaining. : )

[Kumo]

here's my scriptDGDecode_mpeg2source("D:\dario\VTS_01rc2.d2v",info=3)
ColorMatrix(hints=true)

source = last
backward_vec3 = source.MVAnalyse(isb = true, delta = 3, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec3 = source.MVAnalyse(isb = false, delta = 3, pel = 2, overlap=4, sharp=1, idx = 1)
source.MVDegrain3(backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400,idx=1)

dfttest(sigma= 0.8)

crop(4,4,-4,-0)

LimitedSharpenFaster(ss_x=1.1, ss_y=1.1, smode=4, strength=200,undershoot= 10,dest_x=720,dest_y=480)

fastlinedarkenmod()

ColorYUV(levels="tv->pc")
there's a way to let dupped run on it?i'd like to test it with my encoding

[Corran]

Well... everything seems to work fine for me. Your script is extremely slow but I'm not getting any errors about newframe not being defined when I tag dupped() to the end of it.


At this point, I'd try moving out any plugins and avsi scripts you aren't using and see if there is something auto-loading that is causing your problem.

[Caroliano]

I get the following error:
I don't know what "newframe" means
([ScriptClip], line 16

Printed on top of the top frame. What is it? This script depends on any non-core avisynth filter?

And what are the main diferences between yours motion-detection algorithm and dup's one? It uses more than two frames to make decisions? I also don't see the small blocks seen in dup subdivision of the frame (I use blksize=8 for clean sources). If you don't subdivide like this, how you detect small moviments?

[Corran]

I get the following error:
I don't know what "newframe" means
([ScriptClip], line 16

Printed on top of the top frame. What is it? This script depends on any non-core avisynth filter?
All filters used are internal to avisynth. The only time I get that error is when the frames are pulled to the function out of sequence. This most often happens as a result of using SetMTMode() on TSP's multi-thread avisynth (http://avisynth.org/mediawiki/index.php?title=MT_support_page#Differences_between_MT.28.29_and_SetMTMode.28.29). I.e. One thread does the even frames while the second does the odd frames. If you don't have that in your script, it may be possible that there is an avsi file in your plugin directory that does. This isn't to say you can't run filters multi-threaded, you have to do it per filter.

AVISource("Videofile.avi")
MT("ttempsmooth()",4)
fft3dgpu()
dupped()

And what are the main diferences between yours motion-detection algorithm and dup's one? It uses more than two frames to make decisions?

Yes. If needed it will check more than two frames. Beyond that, I'm not sure what the differences are. I'm not aware of how dup works internally.

I also don't see the small blocks seen in dup subdivision of the frame (I use blksize=8 for clean sources). If you don't subdivide like this, how you detect small moviments?

First, I'm taking a subtract() of the current and previous frame. If the frames were mathematically identical, such a subtract would result in a pure gray frame with a luma of about 128. When a change occurs, the luma of this subtract will also. Some areas will darken while others will brighten. With YPlaneMinMaxDifference() I can calculate the difference between the lowest and highest values of the luma. If this value passes a certain threshold, the frame is considered new. If not, then similar checks will be performed on the current and next frame as well as the previous and next frame as needed. This use of subtract() essentially makes small changes stand out like night and day. Tiny lip flap on a tv in the background is enough to make a huge difference in the MinMax values of the subtract's luma, whereas mosquito noise is still too subtle to be a major influence.

[Ranguvar]

Hey,

Would you please consider adding a parameter, that, when enabled, would make the filter create images (PNG, BMP, GIF, JPG, etc.) of any frames that were classified as duplicates but were close to the threshold by a user-inputed amount, and the file they were classified as duplicates of, with filenames that were recognizable as duplicates of a certain other frame? And then possibly, another parameter that would load a *.txt and exclude any frames listed in there from being duplicates?

Big request, but if you get bored :D

Thanks very much! Gonna have some fun with this one!

[Guest]

It's not a windowed algorithm so it will not be as discriminating to signal versus noise as is Dup().

[Guest]

It's not a windowed algorithm so it will not be as discriminating to signal versus noise as is Dup(). That tends to defeat its purpose.

[Caroliano]

I removed everything from avisynth plugins folder to C:/ leaving there only the DirectShowSource.dll and the dupped.avsi. Then I used the folowing script:

directshowsource("testfile.wmv", fps=23.976, convertfps=true)
Dupped()

This still give the same error on top of the frame. I'm runing an MT enabled avisynth, but I'm not using those capabilities ATM. I tried also using an Avisource("test.avi") but no joy.

And yours change detection algorithm is realy clever. I was afraid of an frame-wide averaging because there was no subdivision in the picture. I think that dup makes an simple (or not so simple) averaging in each block of the picture, and if any of them is higher than the threshold it considers the frame diferent. At least, that is what I deducted from dup's show=true.

[foxyshadis]

You should be able to fix that by adding a global newframe = a in the function definition (but not the scriptclip) somewhere. Definitely don't use MT mode, conditionals either don't work at all with it or burn tons of extra cpu time.

[Caroliano]

Yeah, that fixed here. Thanks.

Now to feature requests:
- Add the option to show the "showme screen" in the right of the picture. Useful especialy for editing non-widescreen content in widescreen monitors, but also fits better in AvsP in most cases.
- Add the blend=true option from dup() for help in denoising.
- Your funcion seems to not process chroma. Adding it would make the detection more precise, isn't? I think it would be especialy useful for detecting moviment in CG. Toki wo Kakeru Shoujo don't have explicit CG, but many others animes do. Or it don't matter and only slows down?

I'm now comparing dup and dupped with the folowing method:
dup1 = last.Dup(threshold=0.85, blksize=8, copy=false, show=true) # visual analize mode
dup2 = last.dupped(thresh=28,panthresh=1.7,stats=true,showme=false)
merge(dup1,dup2)
It is a bit anoying that dup compares (this,next) and dupped (last, this), but it's manageable. I preffer the dup one, because I know when to look harder for diferences before-hand.

Latter I will do an test encode with lagarith. In moviments with lines, both are good enought, with your script reporting the highest diferences when in moviment. The problem is with slow clouds moviment or pans, zooms and fading...

[Zarxrax]

neuron2: regarding signal to noise, since anime these days is produced digitally, there doesn't seem to be much problem with noise in many sources. On a nice clean source, your dup filter failed pretty badly on simple things like slow pans, detecting many duplicates that were not really duplicates (I should note that this was in my usage, which may or may not have been performed correctly; I don't mean to say bad things about your filter). However, I'm sure your filter would perform much better on sources with more noise in them, as it seems that this was one of your primary goals in designing your filter.

However it seems that this function of Corran's, although simple in it's approach, may be more useful than dup in some situations. Perhaps though, you could add functionality like this into your dup filter, as an extra check to avoid marking frames incorrectly as duplicates?

Personally, I am very wary of using either of these filters on my encodes, as I don't particularly trust it to make the best decision when it comes to something as major as completely dropping frames from the source. But I think a combination of these 2 filters would provide a very good safeguard against false positives, and would be something that I feel I could rely on. What do you think?

Also, I had an idea of using a strong denoiser as a preprocessor for the duplicate detection. Couldn't something like this improve the accuracy of detecting duplicates?

[Ranguvar]

I'm tending to agree with Zarxrax here. My intended use for a function like this would be to increase compressibility with x264, so I would put it near the end of a script, certainly after denoising.

Perhaps this would help, too:

[All filtering, including normal denoising]
beforeSDenoise = last
foo = somedenoiserhere(parametersstrongerthanusualusage)
return beforeSDenoise.dupped(foo, ......)

After all, I think it'd be extremely rare the case when a denoiser actually removed enough small details to create erroneous duplicate frames when it is dupped().

[thetoof]

Is this thread dead? IMO a "motion-adaptative" dup function would be very nice... especially with a denoised dclip. A hard threshold like dup() has is simply destructive for low motion scenes, so is there any further development considered?

[McCauley]

Added to the wiki (http://avisynth.org/mediawiki/External_filters#Duplicate_Frame_Detectors).

Regards
McCauley

[MadRat]

Thanks McCauley

[lansing]

I wonder if a flicker detector could be added to this, since older anime still flicker quite a bit even though they were stabilized. It will surely help save A LOT of time of some hardcore anime fans that I know who spends hours and hours replacing duplicated frame manually.

[Corran]

I wonder if a flicker detector could be added to this, since older anime still flicker quite a bit even though they were stabilized. It will surely help save A LOT of time of some hardcore anime fans that I know who spends hours and hours replacing duplicated frame manually.

Is this thread dead? IMO a "motion-adaptative" dup function would be very nice... especially with a denoised dclip. A hard threshold like dup() has is simply destructive for low motion scenes, so is there any further development considered?
I apologize for not working on this in the year and a half since I posted it. I don't have any immediate plans to continue work on it in the future. AVISynth's scripting language seems rather restrictive for something like this and it might be better off as a plugin. Additionally, my goal in all of this was to test out an idea I had using the internal subtract filter to detect motion. To that end I accomplished my goal. If any one wants, they can take what I've posted and tweak/re-write/remove attribution/etc as desired. Consider it public domain if it hasn't been considered such already. If I need to specify this in the script itself, let me know how I should do so.



@Kumo, If you see this post, I think I finally figured out why you were having problems. (maybe) I was recently trying out AvsP with this script and no mater where I would seek, I would get the newframe error message. When I opened the script in an external player (i.e. virtualdub) the problem did not occur. Not sure how frame access differs from AvsP and other tools. Back when I posted this I did all of my testing in VirtualDubMod.

Edit: Posted cleaner version of function. Should hopefully be easier to understand if anyone wants to tackle the requests in this thread later down the road.

[andy_blah]

I don't know why but no matter on what kind of a clip I get this error at the very first duplicated frame:
I don't know what "last NewFrame" means
([Script Clip], line 19)

I've recently got this problem after I've reinstalled Windows and had to place everything back where it was (including Avisynth and it's plugins)
Is there any solution for that message to be removed? It isn't that visible but it's quite annoying if there are lots of duplicate frames at the beginning of the clip.

Thank you in advance~

EDIT: Sorry, forgot to post in the script file

LoadPlugin("D:\untitled folder\DGIndex 1-5-8\DGDecode.dll")
sectionAvid = MPEG2source("sectionA.d2v").Crop(6, 0, -8, 0).Lanczos4resize(640, 480).Levels(0, 1.3, 255, 0, 255).ChangeFPS(29.970).Trim(196,0)
sectionAimg = ImageSource("sectionA.png", fps = 29.970).ConvertToYV12().Trim(0, 196)
secAvid = sectionAimg + sectionAvid
secAaud = NicAC3Source("sectionA.ac3")
secA = AudioDub(secAvid, secAaud).Trim(15,0)
return secA

[Gavino]

Sorry, forgot to post in the script file
I don't see any call to 'dupped' in that script.

[andy_blah]

Mistake, forgot that I've removed the .Dupped() part from the code to be able to encode that avs script but the script is still the same.

[Gavino]

But where is/was it called?
This is vital information to diagnose the problem!

[andy_blah]

LoadPlugin("D:\untitled folder\DGIndex 1-5-8\DGDecode.dll")
sectionAvid = MPEG2source("sectionA.d2v").Crop(6, 0, -8, 0).Lanczos4resize(640, 480).Levels(0, 1.3, 255, 0, 255).Tweak(cont = 1, sat=1.25).ChangeFPS(29.970).Trim(196,0)
sectionAimg = ImageSource("sectionA.png", fps = 29.970).ConvertToYV12().Trim(0, 196)
secAvid = sectionAimg + sectionAvid
secAaud = NicAC3Source("sectionA.ac3")
secA = AudioDub(secAvid, secAaud).Trim(15,0).Dupped()
return secA

[Gavino]

Looking at the code of the 'dupped' function, I can see that the "I don't know what 'lastNewFrame' means" error is due to a bug which will show up when the first two frames of the clip are the same (as in andy_blah's case where the clip starts with an image sequence).

The line
newframe = prevCurrentYMinMax >= dupThresh || prevNextYMinMax >= dynPanThresh ? true : false
should be
newframe = prevCurrentYMinMax >= dupThresh || prevNextYMinMax >= dynPanThresh || current_frame == 0

There is also a separate bug in the function getFrame which can cause wrong results when frameNum=0
c = trim(frameNum,frameNum)
should be
c = trim(frameNum, -1)
Obviously, the author did not fully apply stickboy's advice in post #2. :(

Edit:
@andy_blah
A workaround for your script would be to move the application of 'dupped' to apply only to the non-static part of your output, ie use
secAvid = sectionAimg + sectionAvid.dupped()
You lose nothing by doing this since applying dupped to a static clip leaves it unchanged anyway.

[AlfOng]

This is my first post and I thought that I would give something back for all the help I've gleaned from the site.

I post this "bad frame" fix that I have been using.
It takes one good frame on each side of the bad frames range and then does a "tweening effect" and replaces the bad frame range.

It uses frame numbers starting from zero. So if the clip is trimmed beforehand then 'startf' would equal the trim start frame number.

It works extremely well for a handfull of frames, but not too well if there is an object with high motion between the 2 good frames.

(I find delta=1 gives a sharper picture)

Use and modify as you like. I hope it helps someone.

function FrameMorph2(clip c, int startf, int in, int "out", int "blksize")
{

Function fill_loop(string s, int stop, int count)
{
return(stop==0) ? s :

string("tofill.mflowinter(super,fill_vb,fill_vf,time="+string(100*(count-stop+1)/float(count+1)))+").selectevery(numfr,0

),"+fill_loop(s,stop-1,count)
}

out =default(out, 0)
blksize =default(blksize,16)

tofill =c.trim(in-startf,out-startf)
numfr =tofill.framecount

dx =c.trim(in-startf-1,-1)++c.trim(out-startf+1,-1)
super =dx.MSuper(pel=2)
fill_vf =MAnalyse(super,search=3,isb=false,delta=1)
fill_vb =MAnalyse(super,search=3,isb=true,delta=1)

filled =eval("interleave("+fill_loop("",numfr,numfr)+"tofill.selectevery(numfr,numfr)).assumefps(c.framerate())")

return c.trim(0,in-startf-1)++filled.trim(0,numfr-1)++c.trim(out-startf+1,0)
}

Edit: There are some obvious line split errors in the fill-loop function which should not be there! I don't seem to be able to edit this post to fix it?

[Gavino]

There are some obvious line split errors in the fill-loop function which should not be there! I don't seem to be able to edit this post to fix it?
Put code between code and /code tags to avoid automatic line break. If you are logged in, you should get the option to edit any of your posts.

I'm having trouble getting my head around your fill_loop function. Can you explain what it does please?

Also what do 'startf', 'in' and 'out' mean from the user's point of view?
Some usage examples would be helpful.

Finally, shouldn't this really be a new thread? What is the connection with 'dupped'?

[AlfOng]

Put code between code and /code tags to avoid automatic line break. If you are logged in, you should get the option to edit any of your posts.

I'm having trouble getting my head around your fill_loop function. Can you explain what it does please?

Also what do 'startf', 'in' and 'out' mean from the user's point of view?
Some usage examples would be helpful.

Finally, shouldn't this really be a new thread? What is the connection with 'dupped'?

I'm no expert on coding.. but let me try and explain.

The code is taken from ideas posted in the forum and modified to fit my specific needs. I started out looking for "bad frame fixes" for several home videos I want to try and repair.

If it is posted here in the wrong part of the forum, then sorry.

The function takes "in" and "out" frame numbers, ie in=first bad frame, out=last bad frame. (They can be the same). Frame numbers are based on untrimmed total video length.

(So for example if you trim a scene, then startf would be the scene 'in' frame number)

The fill_loop simply generates new frames to replace the old frames eg 110to113

eg
Trim out a scene frames 100 to 500
Bad frames are 110-113 inclusive, based on original untrimmed video.

(Note you can also use startf=0 and bad frame numbers relative to the trim point, it really depends on how you want to identify the bad frame numbers)

sourcevideo
....
startf=100
trim(startf,500)
FrameMorph2(startf, 110,113)
...

Hope this makes sense and helps to clarify.

[Gavino]

Thanks for the clarification, I see now how these parameters are meant to be used. Clearly it is a necessary precondition that in>=startf+1 and out-startf<c.Framecount-1 (otherwise you have no good frames to use). Perhaps your function should check for this and report an error if it is not satisfied (using Assert()).

Unfortunately, there is at least one bug in this line:
return c.trim(0,in-startf-1)++filled.trim(0,numfr-1)++c.trim(out-startf+1,0)

The first trim will not work if in equals startf+1 - use Trim(0, -(in-startf)) instead.
The second will not work if numfr equals 1, ie in=out (unless filled has only a single frame in this case - I can't easily see if this is so).

Also, declaring one function (fill_loop) inside another is permitted by a quirk of the parser, but it's not recommended since all functions are global in Avisynth - it is not actually in a separate scope and can confuse the reader.

And I'm still trying to figure out how fill_loop works - any hints? :)

[Gavino]

Right, I see how fill_loop works now, and 'filled' will have 1 frame when in=out, so that second Trim is OK. (The first is still wrong.)

However, why use repeated applications of MFlowInter when you could use a single call to MFlowFps?
filled = dx.MFlowFPS(super, fill_vb, fill_vf, num=(numfr+1)*c.FrameRateNumerator, den=c.FrameRateDenominator).AssumeFPS(c)
return c.trim(0,-(in-startf))++filled.trim(1,-numfr)++c.trim(out-startf+1,0)
This also makes the fill_loop function unnecessary.

Other observations:
- default for 'out' is 0 which is never a valid value - a better default would be the value of 'in'
- parameter 'blksize' is not used.

[AlfOng]

Thanks Gavino, your comments appreciated.

I need to have a look at this when I get a bit more time.

I'm learning all the time with how to do the programming.

MFlowInter is being used for the tweening effect, ie the variable "time" based on the number of bad frames.

I'll look into MFlowFPS again... I looked at it earlier but could not get it to do the job for some reason. (probably just my poor understanding!)

Maybe you can advise me on how to do something.?

The way I currently use the function is by repeated calls to it in ano scripts. This is far from ideal. A better solution would be to use a text file listing the bad frames, and then call the function one time only, using that list.
I'm having trouble understanding how to read in a text file and then use those pairs of numbers in the function.

[Gavino]

MFlowInter is being used for the tweening effect, ie the variable "time" based on the number of bad frames.
I'll look into MFlowFPS again... I looked at it earlier but could not get it to do the job for some reason.
With MFlowInter, you have to generate the tweening frames individually and put them together using your very complicated fill_loop function. MFlowFPS does exactly the same job in one call, as I have shown.
The way I currently use the function is by repeated calls to it in ano scripts. This is far from ideal. A better solution would be to use a text file listing the bad frames, and then call the function one time only, using that list.
I'm having trouble understanding how to read in a text file and then use those pairs of numbers in the function.
Avisynth does not provide a facility to read a text file (other than ConditionalReader, which doesn't help here). I don't see a problem with calling the function several times, but a way to do what you want would be to write a wrapper function that takes a number of pairs and calls the original function for each pair.

[Great Dragon]

I still have the "I don't know what 'lastNewFrame' means" error with trim() function. Even with latest changes from Gavino.

[Great Dragon]

Sorry, guess I've mislabeled an error.
Actually I have the "GetFrame(): FrameNumber supplied is larger than video's framecount." error on frame 1789 with this script:

s1=trim(0,1790).dupped(thresh=70,panthresh=0.90,stats=false,Showme=false)
s2=trim(1791,1937)
s3=trim(1938,0).dupped(thresh=70,panthresh=0.90,stats=false,Showme=false)
s1++s2++s3
I'm using dupped script with latest recommended changes from Gavino.

[Guest]

vid=last
s1=vid.trim(0,1790).dupped(thresh=70,panthresh=0.90,stats=false,Showme=false)
s2=vid.trim(1791,1937)
s3=vid.trim(1938,0).dupped(thresh=70,panthresh=0.90,stats=false,Showme=false)
s1++s2++s3

[Great Dragon]

neuron2, It's no difference with exactly named agrument or without. I have only Avisource() filter before trim section so "last" is explicitly provided I think. Anyway nothing changed, an error is still present.

[Guest]

Quite right, sorry for the noise.

[bxyhxyh]

Sorry for reviving ancient thread.
I think
global lastnewframe = 0

can fix "i don't know lastnewframe means".

And i think those assert() don't need. And stats=true mod is now become faster.


# Dupped() by corran. Feel free to modify as you see fit.
# Attribution not required.
#
# This filter requires a YUV source. Use converttoyv12 if needed
# before calling dupped()
#
## Parameters
# thresh = Used to determine when to delare a frame as new
# Use stats=true to help determine the best value for your source
# Lower number = more sensitive. (Default=16)
#
# panthresh=1.7 This is used to determine when to consider a frame
# part of a low motion scene. (Default=1.65)
#
# stats = Enable/disable display of stats. (Default=false | Not shown)
# (Last = last frame, this = this frame, next = next frame)
#
# showme = Enable/disable display of video that most stats are
# derived from. (Default=false | Not shown)
#
## Notes
# Do not use with SetMTMode(). This function requires the frames to be
# accessed sequentially. Instead, use MT() to mulit-thread on a per-filter basis.
#
# Example:
# dupped(thresh=20,panthresh=1.7,stats=true,showme=true)

#Function to return the requested frame
function getFrame(clip c, int frameNum)
{
default(c, last)
c = trim(frameNum,frameNum)
return c
}

function dupped(clip c, int "thresh", float "panThresh", bool "stats", bool "showme")
{
default(c, last)
global dupThresh = default(thresh, 16)
global panThresh = default(panThresh, 1.65)
global dupStats = default(stats, false)
global frame = 0
global lastnewframe = 0
showme = default(showme, false)
a = c
c = scriptclip(last,"""
c = last

finalframe = c.framecount-1
prev_frame = current_frame != 0 ? current_frame-1 : 0
next_frame = current_frame != finalframe ? current_frame+1 : finalframe

#Get Y related stats as needed
prevCurrentYMinMax = subtract(getframe(prev_frame),c).YPlaneMinMaxDifference
currentNextYMinMax = prevCurrentYMinMax < dupThresh ? subtract(c,getframe(next_frame)).YPlaneMinMaxDifference : 256
prevNextYMinMax = currentNextYMinMax < dupThresh ? subtract(getframe(prev_frame),getframe(next_frame)).YPlaneMinMaxDifference : 0

#calculate dynamic pan threshold. Set to 256 if looking of a pan is pointless
neighborAvgMinMax = (prevCurrentYMinMax + currentNextYMinMax) / 2
dynPanThresh = prevNextYMinMax != 0 ? Abs(neighborAvgMinMax * panthresh) : 256

#determine if the current frame is new
newframe = prevCurrentYMinMax >= dupThresh || prevNextYMinMax >= dynPanThresh ? true : false
lastNewFrame = newframe ? current_frame : lastNewFrame

c = getFrame(lastNewFrame)
c = dupStats ? c.subtitle(\
"prevframe: "+string(prev_frame)+" currentframe: "+string(current_frame)+" nextframe: "+string(next_frame) + \
"\nLast new frame: "+string(lastNewFrame)+" Newframe: "+string(newframe) + \
"\nprevCurrentYMinMax: "+string(prevCurrentYMinMax)+" (Threshold:"+string(dupThresh)+")", \
lsp=1) : c
c = dupStats && currentNextYMinMax != 256 ? c.subtitle("currentNextYMinMax: "+string(currentNextYMinMax),y=55) : c
c = dupStats && prevNextYMinMax >= dynPanThresh ? c.subtitle(\
"prevNextYMinMax: "+string(prevNextYMinMax) + \
"\nPan Threshold: "+string(dynPanThresh)+" ("+string(neighborAvgMinMax)+" * "+string(panThresh)+")" + \
"\nSlow pan detected", \
lsp=1, y = 90) : c
return c
""")
c = showme ? stackvertical(c,subtract(a.duplicateframe(0),a)) : c

return c
}

[StainlessS]

Original Dupped, assignments to lastnewframe were missing Global prefix (ie were actually setting a Local Var).

Moved here from another thread:- http://forum.doom9.org/showthread.php?p=1716484#post1716484

Multi-Instance Version Dupped for v2.58+ .

Dupped_MI.avs

Function Dupped_MI(clip c, Int "Thresh",Float "Ignore",Float "Override",Float "ChromaWeight",
\ bool "Show", bool "Stack",String "Fn",Bool "FnComments") {
/*
Dupped_MI.avs v1.01
Dupped, Multi-Instance version for Avisynth v2.58+ --- Planar, YUY2, RGB24/32 --- by StainlessS on Doom9
http://forum.doom9.org/showthread.php?p=1716968#post1716968
Dupped() original author, Corran, thanks very much.

Note, Thresh, and Override defaults are set for YUV TV levels, would likely need to increase both if using RGB
(maybe up to about 33% higher, use Show=True,Stack=True to tune args for source).

##############

Thresh (Default 16, 0 -> 254).
If MinMaxDifference(Subtract(Anchor_Frame,current_frame)) <= Thresh, then is Dupe, Else is new non-dupe frame.
The detector is a bit 'back-to-front', it detects non-dupes rather than dupes. (when mnmxdif > Thresh, is non-dupe).
Where Thresh=0, every frame that is not EXACTLY the same (mnmxdif!=0) is a non-dupe.

Ignore (Default 0.01, Range 0.0 -> 100.0) Percentage, As for Threshold in YPlaneMinMaxDifference (avoids a few extreme pixels ie noise)
Suggested range for anime about 0.001 -> 0.01, (0.0 = OFF, dont ignore any noisy pixels).
0.001 would ignore up to 1 extreme pixel in every 100,000 and 0.01 would ignore up to 1 extreme pixel in every 10,000.
From Avisynth Docs for eg YPlaneMin/Max/YPlaneMinMaxDifference:-
Threshold is a percentage, stating how many percent of the pixels are allowed above or below minimum.
ColorYUV ignores 1 pixel in every 256 when showing Loose Minimimum and Loose Maximimum values (about 0.4%) when Analyze=True.
From Avisynth Docs for ColorYUV Analyze mode,
There is a "loose maximum" and "loose minimum". The "loose" values are made to filter out very bright or very dark
noise creating an artificially low or high minimum / maximum.
To detect Dupes, we first Subtract current_frame from the Anchor Frame (anchor frame is the previous non-dupe frame),
after this we do the equivalent of YPlaneMinMaxDifference(Threshold=Ignore) on the subtracted result frame and if
that is <= Thresh then is treated as a dupe, otherwise as a new non-dupe frame. YPlaneMinMaxDifference(Threshold=0.0) would
detect a single pixel difference between Anchor and current_frame that is > Thresh as a non-dupe frame, so a very small
amount of noise could throw off dupe detection. The 'Ignore' setting, allows the detector to ignore a few noisy pixels when
finding both minimum and maximum pixels values in the subtracted frame and so also affects the MinMaxDifference.

Override (Default 0.5). Suggest about 0.5 -> 1.5. 255.0 switches Override completely OFF.
Metrics 'FD' is the average pixel difference between the Anchor Frame and current_frame.
Thresh detected Dupes are Overridden to non-dupe if metrics 'FD' (RT_FrameDifference) is greater than Override.
If there are frames in sample that should not be dupes then set Override just lower than the shown FD for the
worst frame (frame with lowest FD that should be overridden to 'not a dupe').
Override is just a safety measure so that you can set quite ferocious THRESH and IGNORE without fear of making big mistakes.

ChromaWeight (Default 1.0/3.0, 0.0 -> 1.0) ChromaWeight arg to RT_FrameDifference, weight for combined U + V, use 0.0 for
Grayscale source or to use Luma Differences only. RGB not used. Suggest about 1.0/3.0 -> 1.0/2.0.

Show (Default False) Shows Metrics if True
Metrics shown on top line are, character flags 'T','O', and 'D'. Where Hilited, 'T' means Thresh detected dupe, 'O' means
Dupe Overridden to non-dupe, and 'D' means using Dupe. So 'T' means was detected as dupe by Thresh, however if 'O' is also
hil-lited, then was Overridden to be NOT dupe. The 'D' flag gives the final decision, will be hi-lited if 'T' is hi-lited
but 'O' is not hi-lited.
Following the character flags, is the current dupe number ie 1 is first in a run of dupes, 2 is 2nd etc. and Following
that is the currently greatest count of consecutive dupes in any single run.

Stack (Default False) Shows difference between Current frame and Anchor Frame (Using Subtract).

Fn (Default "") Filename, if set then write command text file for FrameSel() plugin.

FnComments (Default False). Write Comments to Fn File if True.

##############

If not detecting enough dupes, then raise Ignore and/or Thresh.
Override should be set AFTER BOTH Thresh and Ignore (can set Override to 255.0 to temporarily disable Override).
To show EXACT dupes only, use Thresh=0(Exact dupes only),Ignore=0.0(ignoring no noise),Override=255.0(dont override anything).

Must play through without jumping about, if jumping about then will not get back into sync until a new non-dupe anchor frame is found.

Script for use with FrameSel plugin, can use result of below FrameSel script where using some kind of non linear access script.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Avisource("LowBitrateAnime.mkv.AVI").trim(0,0) # Trim or Pad audio to length
Audio = Last # Save Audio, FrameSel does not return audio
FrameSel(Cmd="FrameSelCmd.txt",Ordered=False) # where "FrameSelCmd.txt" is Fn in Dupped_MI()
Return AudioDub(Last,Audio) # Restore audio
<<<<<<<<<<<<<<<<<<<<<<<<<<<<
!!! NOTE !!!
To REMOVE (rather than replace) detected duplicates change above FrameSel arg 'Ordered=False' to 'Ordered=True'.
Obviously the audio will be out of sync if removing dupes so do not restore audio via AudioDub (eg security camera caps).
*/
myName="Dupped_MI: "
Assert(RT_FunctionExist("GScriptClip"),myName+"Essential GRunT plugin installed, http://forum.doom9.org/showthread.php?t=139337")
Assert(RT_FunctionExist("GScript"),myName+"Essential GScript plugin installed, http://forum.doom9.org/showthread.php?t=147846")
IsV26 = VersionNumber >= 2.6 # Avisynth Version v2.6 ?
Thresh = Default(Thresh, 16) # Smaller or equal to this is Dupe, Greater than this is non-dupe.
Ignore = Float(Default(Ignore,0.01)) # Ignore up to 1 pixel in every 10,000 (eg 100 x 100 pixel square)
Override=Float(Default(Override,0.5)) # Suggest about 0.5 -> 1.5 (perhaps higher if noisy clip)
CW =Float(Default(ChromaWeight,1.0/3.0)) # Suggest 1.0/3.0 -> 1.0/2.0
Show = Default(Show, false)
Stack = Default(Stack, false)
Fn = Default(Fn, "")
FnCom = Default(FnComments,False)
Assert(Thresh>=0 && Thresh<255,RT_String("%sThresh 0 -> 254(%d)",myName,Thresh))
Assert(Ignore>=0.0 && Ignore<=100.0,RT_String("%sIgnore 0.0 -> 100.0(%f)",myName,Ignore))
Assert(Override>=0.0 && Override<=255.0,RT_String("%sOverride 0.0 -> 255.0(%f)",myName,Override))
Assert(CW>=0.0 && CW<=1.0,RT_String("%sChromaWeight 0.0 -> 1.0(%f)",myName,CW))
(Fn != "" && Exist(Fn)) ? RT_FileDelete(Fn) : NOP
(Fn!="" && FnCom) ? RT_WriteFile(Fn,"#\n# Thresh=%d Ignore=%f Override=%f\n#",Thresh,Ignore,Override) : NOP
c = (Stack) ? StackVertical(c,c) : c
Fmt = "%6d] Using=%d : \a%cT\a%cO\a%cD\a- : %d : %d\n" +
\ "MnMxDif=%3d FD=%-7.3f\n" +
\ "Anchor=%d OvrCnt=%d DupCnt=%d\n" +
\ "\n\aR{Thresh=%d Ignore=%.3f Override=%.3f CW=%.3f}"
FuncS="""
Function Fn@@@(clip clp,Float Thresh,Float Ignore,Float Override,Float CW,Bool Show,Bool Stack,
\ String Fmt,String Fn,Bool FnCom,Bool IsV26) {
Last = (Stack) ? clp.Crop(0,0,-0,clp.Height/2): clp
n = current_frame
InSeq = (Prev@@@ == n-1)
OldAnchor = (InSeq) ? Anchor@@@ : max(n-1,0) # Remember on entry Anchor FrameNo for metrics
AFrm=Trim(OldAnchor,-1)
BFrm=Trim(n,-1)
SubFrm = Subtract(AFrm,BFrm)
fd = RT_FrameDifference(AFrm,BFrm,n=0,n2=0,ChromaWeight=CW) # Average pixel difference
if(IsYUV) {
MnMxDif = SubFrm.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
if(!IsV26) {
MnMxDifU = SubFrm.UToY.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
MnMxDifV = SubFrm.VToY.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
MnMxDif=Max(MnMxDif,MnMxDifU,MnMxDifV)
} Else if(!IsY8()) {
MnMxDifU = SubFrm.UToY8.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
MnMxDifV = SubFrm.VToY8.RT_YPlaneMinMaxDifference(n=0,Threshold=Ignore)
MnMxDif=Max(MnMxDif,MnMxDifU,MnMxDifV)
}
} Else {
SubFrm.RT_RgbChanStats(n=0,Chan=-1,Flgs=$04,Prefix="DUP_RCS@@@_",Threshold=Ignore)
MnMxDif=Max(DUP_RCS@@@_MinMaxDiff_0,DUP_RCS@@@_MinMaxDiff_1,DUP_RCS@@@_MinMaxDiff_2)
}
ThreshDetect = (MnMxDif <= Thresh)
Overridden = (ThreshDetect && fd > Override)
IsNewFrame = (!ThreshDetect || Overridden || !InSeq)
Global Anchor@@@ = IsNewFrame ? n : Anchor@@@ # Using this FrameNo, Next time will be Anchor FrameNo
Trim(Anchor@@@,-1) # Get Using Frame
if(Show) {
Global DupCnt@@@ = IsNewFrame ? DupCnt@@@ : DupCnt@@@ + 1
Global OvrCnt@@@ = Overridden ? OvrCnt@@@ + 1 : OvrCnt@@@
DupNum=IsNewFrame?0:n-OldAnchor
Global DupMax@@@ = Max(DupMax@@@,DupNum)
Last = RT_SubTitle(Fmt,n,Anchor@@@,(ThreshDetect)?54:76,(Overridden)?54:76,(!IsNewFrame)?54:76,DupNum,DupMax@@@,
\ MnMxDif,fd,
\ OldAnchor,OvrCnt@@@,DupCnt@@@,
\ Thresh,Ignore,Override,CW
\ )
}
If(Fn != "") {
if(!FnCom) {
RT_WriteFile(FN,"%d",Anchor@@@,Append=True)
} Else {
(Overridden) ? RT_WriteFile(FN,"%d # *** OVERRIDE: MnMxDif=%d FD=%f",Anchor@@@,MnMxDif,fd,Append=True)
\ : (!IsNewFrame) ? RT_WriteFile(FN,"%d # [%d] !!! DUPE: MnMxDif=%d FD=%f",Anchor@@@,n,MnMxDif,fd,Append=True)
\ : RT_WriteFile(FN,"%d # NON-DUPE: MnMxDif=%d FD=%f",Anchor@@@,MnMxDif,fd,Append=True)
}
}
Global Prev@@@=n
(Stack) ? StackVertical(Last,SubFrm) : Last
Return Last
}
#######################################
# Unique Global Variables Initialization
#######################################
Global Anchor@@@=0 Global DupCnt@@@=0 Global DupMax@@@=0 Global OvrCnt@@@=0 Global Prev@@@ = -666
#######################################
# Unique Runtime Call, GScriptClip must be a one-liner:
#######################################
ARGS = "Thresh,Ignore,Override,CW,Show,Stack,Fmt,Fn,FnCom,IsV26"
c.GScriptClip("Fn@@@(last, "+ARGS+")", local=true, args=ARGS)
"""
#######################################
# Unique Identifier Definition
#######################################
RT_IncrGlobal("Dupped_MI_InstanceNumber")
InstS = RT_StrReplace(FuncS,"@@@","_Dupped_MI_"+String(Dupped_MI_InstanceNumber))
# RT_WriteFile("DEBUG_Dupped_MI_"+String(Dupped_MI_InstanceNumber)+".TXT","%s",InstS)
Return GScript(InstS)
}


Multi-Instance Wrapper

Function Dupped_MI_X(clip c, Int "Thresh",Float "Ignore",Float "Override",Float "ChromaWeight",
\ bool "Show", bool "Stack",String "Fn",Bool "FnComments",
\ Bool "Pair", Int "Thresh2",Float "Ignore2",Float "Override2",Float "ChromaWeight2") {
/*
Wrapper for convenience.
Works same as Dupped_MI() but auto StackHorizontal of 2nd instance if Pair=True.
Defaults for 2nd Instance are for EXACT dupes only, Change args to 2nd clip using Thresh2, Ignore2, Override2, and ChromaWeight2.
*/
Pair = Default(Pair, false)
Th2 = Default(Thresh2, 0)
Ig2 = Default(Ignore2,0.0)
Ov2 = Default(Override2,255.0)
Cw2 = Default(ChromaWeight2,ChromaWeight)
A = c.Dupped_MI(Thresh=Thresh,Ignore=Ignore,Override=Override,ChromaWeight=ChromaWeight,Show=Show,Stack=Stack,Fn=Fn,FnComments=FnComments)
B = (Pair) ? c.Dupped_MI(Thresh=Th2,Ignore=Ig2,Override=Ov2,ChromaWeight=Cw2,Show=Show,Stack=Stack,Fn="",FnComments=False) : 0
(B.IsClip) ? StackHorizontal(A,B) : A
Return Last
}


Client Script for Multi-instance wrapper

FN="WebCam3.AVI"

Avisource(FN)

SHOW = True # True=Show metrics.
STACK = True # Show StackVertical Subtract frame
COMMENTS = True
PAIR = True # True shows StackHorizontal 2 or 4 windows dependant upon Stack
THRESH = 16 # DEFAULTS
IGNORE = 0.01 #
OVERRIDE = 0.5 #

#ConvertToRGB24
#ConvertToRGB32
#ConvertToYUY2
#ConvertToYV12
#ConvertToY8 # V2.6 ONLY
#ConvertToYV16
#ConvertToYV24
#ConvertToYV411

Dupped_MI_X(Thresh=THRESH,Ignore=IGNORE,Override=OVERRIDE,Show=SHOW,Stack=STACK,Fn="FrameSelCmd.txt",FnComments=COMMENTS,Pair=PAIR)

#ConvertToRGB32 # For v2.6 Colorspace
Return Last


ReplaceDupes.avs (EDIT: Replace all detected dupes with previous non-dupe)

FN="WebCam.AVI"

AvisourcE(FN).Trim(0,0)
Audio = (HasAudio) ? KillVideo : 0
FrameSel(Cmd="FrameSelCmd.txt",Ordered=False)
(Audio.IsClip) ? AudioDub(Audio) : NOP
Return Last


NoDupes.avs (remove/delete all duplicates)

FN="WebCam.AVI"
Avisource(FN)
FrameSel(Cmd="FrameSelCmd.txt",Ordered=True)
Return Last

[StainlessS]

Lagarith File sizes Sorted in order for "LowBitrateAnime.mkv.AVI"
(I think may have come from Avisynth Usage/Consolidated list of test video clip resources as MKV)
EDIT: Had a look for this clip, found instance of it here: http://forum.doom9.org/showthread.php?p=1468097#post1468097
1.77MB, 3:28 secs. (I converted to AVI lossless using ffmpeg)


Lagarith Nul Frames ONLY 729,092,096 bytes
Dupped Thresh=8 563,208,192 bytes
Dupped Thresh=16(Default) 541,859,840 bytes
Dupped_MI Thresh=8, Ignore=0.0 538,779,648 bytes
Dupped_MI (Default)Thresh=8, Ignore=0.001 519,776,256 bytes)
Dupped_MI Thresh=16, Ignore=0.0 501,577,728 bytes
Dupped_MI Thresh=16, Ignore=0.001 476,020,736 bytes
Dupped_MI Thresh=8, Ignore=0.01 455,946,240 bytes
Dupped_MI Thresh=16, Ignore=0.01 395,296,768 bytes


I was very surprised that Lagarith with NUL frames does quite a lot better (smaller) than UT_Video(fast decompress mode) or MagicUV.
(UTVideo/MagicUV figures not shown)

EDIT: The fixed version of original Dupped here:- http://forum.doom9.org/showthread.php?p=1716484#post1716484

[Great Dragon]

Here is a short sample for you to test. Lags 34 mb.
https://dl.dropboxusercontent.com/u/30473146/pass6_ep1_trim.zip

There are moving skies on a background where all "dupe" filters usually fails.

[StainlessS]

That is a difficult clip. Seems that what should be dupes are horizontally shifted a little bit (as can be seen when Stack=True)

Lagarith NUL frames 34,048,000 bytes
Dupped Thresh=8 34,048,000 bytes
Dupped (Def)Thresh=16 34,048,000 bytes
Dupped_MI (Def)Thresh=8 Ignore=0.001 34,048,000 bytes
Dupped_MI Thresh=8 Ignore=0.01 34,048,000 bytes
Dupped_MI Thresh=16 Ignore=0.001 34,048,000 bytes
Dupped_MI Thresh=16 Ignore=0.01 33,693,696 bytes
Dupped_MI Thresh=16 Ignore=0.1 33,001,472 bytes
Dupped_MI Thresh=32 Ignore=0.4 23,226,368 bytes
Dupped_MI Thresh=50 Ignore=0.4 18,665,472 bytes


Almost no effect until quite savage settings used
(50,0.4 seems to work quite well, missed a couple of dupes, no mistake dupes detected, I think).

EDIT: Am quite impressed with the dupped detection method (well the fixed version) although the panThresh thing was
not terribly good, misses quite a few dupes that are easily detected using MI version 'Anchor frame'.
The MI version however requires strict linear access and play all the way though, The FrameSel option should allow
use with later temporal processing - non linear access.

You can use two instances of MI version to compare settings.

[Great Dragon]

You can use two instances of MI version to compare settings.
Yep, that's what I've used.
s1=AVISource("g:\_podw\pass6_ep1.avi").Dupped_MI(show=true,stack=true,Thresh=35,Ignore=0.1)
s2=AVISource("g:\_podw\pass6_ep1.avi").Dupped_MI(show=true,stack=true,Thresh=30,Ignore=0.001)
StackHorizontal(s1,s2)

[StainlessS]

Great Dragon,
I should have taken my own recommendation and viewed two instances at once.
I did not actually notice the 'moving sky' in your clip and thought Dupped_MI(50,0.4 ) was working pretty good, it was not.
Have updated Dupped_MI (not v2.58 version yet), added an Override that allows setting savage Thresh and Ignore without much danger.
Give it a try, I think it works pretty good now even with your moving sky's (clouds).

EDIT: I kept the 'savage' settings to show Override working.
EDIT: Maybe Override=0.95 instead of 1.0 for frames 111, 185 and 189.

[StainlessS]

Post #44, Dupped_MI() Updated, now for both v2.58 and v2.6 Avisynth, working great.
Changed defaults.

WebCam cap I420 codec about 11GB (640x480 25FPS, 25490 Frames, 16:59.6)
Compress to Lagarith with Null Frames about 1.97GB (EDIT: Only 10 exact dupes ie NULL frames).
After Dupped_MI(), 365 MB (EDIT: with NULL frames).
After NoDupes.avs, reduced to 4586 frames (EDIT: about same size ie 365MB), 3:03.44 (constant motion throughout).