LRemoveDust in AVS+ x64

[ABurns]

I'm trying to make the move to 64-bit AVS+ and meGUI for the better memory management (and in the case of meGUI, stability). I'm using it for a lot of things now, particularly 1080 material, but one filter is holding me up from being able to make the move completely.

I still find LRemoveDust useful for some purposes, even though it's fairly destructive compared to other denoisers. It requires RemoveGrain and SSETools with the LimitChange() function, and it seems that neither RGTools nor the 1.0pre x64 build linked here (http://avisynth.nl/index.php/RemoveGrain) provide that function, and I can't find an x64 build of SSETools at all. I believe LimitChange is part of SSETools, but I'm not sure.

Is there an x64 build somewhere out there that I'm missing that provides this? Or is there a workaround to get this filter working in x64?

TIA

[StainlessS]

Generated by an RT_Stats script


SSE2Tools_ORDERED_Function_List


There follows a list of all function names together with CPP style argument specifiers that inform
Avisynth the argument types and optional names. Optional arguments have square brackets surrounding
their name as in [name] and are followed by a type specifier character that gives the type.
Unnamed arguments are not optional. eg "cc[arg1]b[arg2]i" would be two compulsory unnamed clip args,
followed by optional 'arg1' of type bool and optional 'arg2' of type int.

# Argument type specifier strings.
c - Video Clip
i - Integer number
f - Float number
s - String
b - boolean
. - Any type (dot)
# Array Specifiers
i* - Integer Array, zero or more
i+ - Integer Array, one or more
.* - Any type Array, zero or more
.+ - Any type Array, one or more
# Etc
###################################


AmplifyDifference "cc[factor]f[factorU]f[factorV]f[grey]b"
Difference "cc[frames]i[tolerance]i[above]b[yuv]b"
Interleaved2Planar "c[grey]b"
LimitChange "cci[limitU]i[limitV]i"
Planar2Interleaved "c[grey]b"
Quadruple "c[grey]b"
ReduceFluctuations "c[limit]i[limitU]i[limitV]i[reduceflicker]b"
Shrinkby2 "c[grey]b"
TransformColor "cffffffffff"
Truncate "cii[lowerU]i[upperU]i[lowerV]i[upperV]i"

I think some people decided that it was not needed (SSETools).

[StainlessS]

Sorry, I dont got SSETools source, and it dont seem to be available anywhere.
Kassandro site now offline a few years (both old and later site).
K's plugs were always a bit peculiar, undocumented secret functions and the like, and no source in case of SSETools.

If you come across a source (or if anyone else has it) then post.

EDIT: There is even a thread on-site about missing source for SSETools:- https://forum.doom9.org/showthread.php?t=144871

This is problably the best (or only) docs for SSETools:-
http://web.archive.org/web/20160324211647/http://videoprocessing.fr.yuku.com/sreply/256/Basic-question-about-SSETools

[Groucho2004]

There's a comment/description about LimitChange:

"LimitChange" can also be implemented with one instance of Lutxy and as a lutmaniac you will probably prefer that, Didee. It is not much slower either, especially if you don't have SSE2.

Anyone know how to do that?

Also, why do some people not post the source for their plugs?

[StainlessS]

Also, why do some people not post the source for their plugs?

K's plugs were always a bit peculiar
I guess that K was a bit peculiar too :)

LimitChange - This can be implemented with one instance of Lutxy
I was about to go have 40 winks, I'm knackered, but then noticed the above (in the linked post).
I guess it just means supply original clip, + altered clip, and limit the max difference by the unnamed int (Lets call it LimitY),
and LimitU Default(LimitY), and LimitV Default(LimitU). [think that is how K described defaults to be implemented in all his plugs]

I would though suggest using Avs+ Expr() instead of mt_lutxy, I recently tried but could not get UExpr or VExpr to do anything,
worked first time in Expr().

Gonna have a nap.

[Reel.Deel]

There's a comment/description about LimitChange:



Anyone know how to do that?

Also, why do some people not post the source for their plugs?

Perhaps this might be of some help:

Example LimitChange(): (from the other thread)

Sure, for simple "clipping-the-effect" one could use LimitChange instead of building a YV12Lutxy expression. Again, LimitChange supposely is a tad faster.
Now, I could argue that IMHO LimitChange puts the two clip arguments backwards - but that's not an argument. ;)

But what I do very often is not to do a "hard clipping", but clipping with some modifications. An action often taken is, say, "apply effect by 50%, but not more than +/- 3". With LUT, this is:

a=last
b=blur(1.58)
yv12lutxy(a,b,"x 5 + y < x 3 + x 5 - y > x 3 - x 50 * y 50 * + 100 /")

With LimitChange, this is not possible to do directly, but would require another additional filtering step.



yv12lutxy(beforeFLD,afterFLD,"x 2 + y < x 2 + x 2 - y > x 2 - y ? ?",U=2,V=2) It would be more elegant with LimitChange from SSETools instead of YV12Lutxy ... but in LimitChange, I tend to always swap the clip arguments, somehow. :o





Seriously - implementing TPresso is almost trivial. See:

... this is SPresso, basically:
source = last
spat = source.SomeSpatialDenoiser()

source.LimitChange( source.Merge(spat,bias), limit )


... this is TPresso, basically:
source = last
temp = source.SomeTemporalDenoiser()

source.LimitChange( source.Merge(temp,bias), limit )

STPresso (https://forum.doom9.org/showpost.php?p=1551871&postcount=2) has mt_lutxy expressions for spatial and temporal limits. Someone who understands this can surely figure out LimitChange and how to replicate it using mt_lutxy.

[Groucho2004]

I guess that K was a bit peculiar too :)Ja, diese Germans sind ein weird bunch.

[ABurns]

Ugh, that's what I was afraid of. Kind of like Steady and the dust family back in the day. I knew some of Kassandro's stuff was closed source, but I didn't remember which ones.

If there is a reasonably straightforward way to implement limitchange() with yv12lutxy (I'm not a programmer, but I assume we are talking about the masktools function) I imagine that would probably be worthwhile, considering the popularity of Kassandro's filters and all of their derivatives like this. If that were so, I assume limitchange() would just become a script function as opposed to a plugin? However, if it's going to be very difficult, it's probably not worth investing a lot of time in trying to figure it out, since there are a ton of other lightweight denoisers out there that will probably do just as well. In a worst case, I can just keep using x32 for that function, I just don't want to have to maintain 2 sets of apps and plugins on my system if I don't have to.

[StainlessS]

DONT USE THESE FUNCTIONS : SEE POST #14

Can you try this out (dont know if I got the Src and Denoised around the wrong way, just change order in function def)
[Seems more sensible the other way around, but as Didee said the same I did it what I thought was wrong way around]

Implemented as Didee described "Hard Clipping".

Non working function using masktools

Function MT_LimitChange(clip Src,clip Denoised,int LimitY,int "LimitU",int "LimitV") {
# THIS FUNCTION DONT WORK FOR U AND V CHANNELS
LimitU=Default(LimitU,LimitY) LimitV=Default(LimitV,LimitU)
Lim_Y = String(LimitY,"%.0f") Lim_U = String(LimitU,"%.0f") Lim_V = String(LimitV,"%.0f")
# "min(max(Denoised,Src-LimitY),Src+LimitY)"
# Infix = "min(max(y,x-LimitY),x+LimitY)" # where x is Src and y = Denoised
# RPN = "y x LimitY - max x LimitY + min"
RPN_1 = "y x "
RPN_2 = " - max x "
RPN_3 = " + min"
Expr_Y = RPN_1 + Lim_Y + RPN_2 + Lim_Y + RPN_3
Expr_U = RPN_1 + Lim_U + RPN_2 + Lim_U + RPN_3
Expr_V = RPN_1 + Lim_V + RPN_2 + Lim_V + RPN_3
Return mt_lutxy(Src,Denoised,YExpr=Expr_Y,UExpr=Expr_U,VExpr=Expr_V) # Only this line different to below function
}


Working function using Avs+ Expr()

Function LimitChange(clip Src,clip Denoised,int LimitY,int "LimitU",int "LimitV") {
LimitU=Default(LimitU,LimitY) LimitV=Default(LimitV,LimitU)
Lim_Y = String(LimitY,"%.0f") Lim_U = String(LimitU,"%.0f") Lim_V = String(LimitV,"%.0f")
# "min(max(Denoised,Src-LimitY),Src+LimitY)"
# Infix = "min(max(y,x-LimitY),x+LimitY)" # where x is Src and y = Denoised
# RPN = "y x LimitY - max x LimitY + min"
RPN_1 = "y x "
RPN_2 = " - max x "
RPN_3 = " + min"
Expr_Y = RPN_1 + Lim_Y + RPN_2 + Lim_Y + RPN_3
Expr_U = RPN_1 + Lim_U + RPN_2 + Lim_U + RPN_3
Expr_V = RPN_1 + Lim_V + RPN_2 + Lim_V + RPN_3
Return Expr(Src,Denoised,Expr_Y,Expr_U,Expr_V)
}


real basic test

Denoised=ColorBars(Pixel_Type="YV12").BilinearResize(480,240)
Src=Denoised.BlankClip(Color_YUV=$808080)

A=Denoised.ShowChannels
B=LimitChange(Src,Denoised,64) # Limit change (ie colorbars) to 64 from grey (all channels)
C=Mt_LimitChange(Src,Denoised,64)
B=B.ShowChannels
C=C.ShowChannels
StackVertical(A,B,C)

https://i.postimg.cc/mD0PG5br/Limit-Change.jpg (https://postimages.org/)
1st clip, Colorbars channel values.
2nd clip, Limitchange Y,U, and V limited to 128 - 64 and 128 + 64.
3rd clip, NON WORKING mt_Limitchange Y,limited to 128 - 64 and 128 + 64, what happened to U and V ? [have previously had probs with mt_lutxy U & V]

EDIT:
Ja, diese Germans sind ein weird bunch.
Arh, that explains it, so sad really, I guess we just have to be happy with what the good lord gave us.

Above Script fixed and extended a bit.

NOTE, in Masktools docs, min() and max() supported RPN, but maybe dont work proper going back quite some time
(I have at some earlier point [a few weeks ago] tested I think back to Fizick version Masktools, but forgot).
The supported operators are : "+", "-", "*", "/", "%" ( modulo ) and "^" ( power )
The supported functions are : "sin", "cos", "tan", "asin", "acos", "atan", "exp", "log", "abs", "round", "clip", "min", "max".

[StainlessS]

Prev post script fixed and updated.

[ABurns]

Can you try this out (dont know if I got the Src and Denoised around the wrong way, just change order in function def)
[Seems more sensible the other way around, but as Didee said the same I did it what I thought was wrong way around]

Implemented as Didee described "Hard Clipping".

StainlessS my man, you are an absolute genius. I think we are in business. Using x264 -crf 20, I encoded an 800 frame sample of Spartacus using AVS32 with SSETools and AVS64 with the function above, and the MD5 hashes are identical. Both encodes came out at 2.89MB, while the control encode with no noise reduction came out at 3.66MB.

(That's one of the reasons I like this function - it's moderately destructive, but the compression gains are huge if file size is important in the project. It's a very decent tradeoff.)

Thanks a million.

[StainlessS]

Magic.
Are you also interested in the Didee mod as posted by Reel.Deel ?
An action often taken is, say, "apply effect by 50%, but not more than +/- 3". With LUT, this is:
If so, I'll try get around to it tomorrow.

[ABurns]

Magic.
Are you also interested in the Didee mod as posted by Reel.Deel ?

If so, I'll try get around to it tomorrow.

Maybe, but what would be the practical advantage of doing it that way? I'm really not sure what I'm looking at with that post. It's a little over my head.

[StainlessS]

Here, Decided to change order in which clips given to Expr() and mt_lutxy()
[as seemed wrong way around as per Didee observation, I had just used same order as the LimitChange() function]
In changing order, I got different results from mt_lutxy(), still not correct but different.
This prompted me to again suspect that I had done something wrong, and found the answer.

Return mt_lutxy(Denoised,Src,YExpr=Expr_Y,UExpr=Expr_U,VExpr=Expr_V,U=3,V=3) # MUST SET U=3 and V=3 (Even when UExpr and VEXpr explicitly supplied, ELSE dont work proper).

Guess I just assumed that it would be automatically taken as true that U and V should be auto set to 3. (I have made the exact same mistake before).
x=3 : the plane will be processed with the processing the filter is designed to do.

Anyways, here corrected pair of script functions, both in working order for Expr() or Mt_lutxy().

Working for Avs v2.58, Avs v2.60 [Maybe Remove the 'MT_' bit from start of function name to use in existing script]

Function MT_LimitChange(clip Src,clip Denoised,int LimitY,int "LimitU",int "LimitV") { # https://forum.doom9.org/showthread.php?p=1870697#post1870697
LimitU=Default(LimitU,LimitY) LimitV=Default(LimitV,LimitU)
Lim_Y = String(LimitY,"%.0f") Lim_U = String(LimitU,"%.0f") Lim_V = String(LimitV,"%.0f")
# "min(max(Denoised,Src-LimitY),Src+LimitY)"
# Infix = "min(max(x,y-LimitY),x+LimitY)" # where x is Denoised and y = Src
# RPN = "x y LimitY - max y LimitY + min"
RPN_1 = "x y "
RPN_2 = " - max y "
RPN_3 = " + min"
Expr_Y = RPN_1 + Lim_Y + RPN_2 + Lim_Y + RPN_3
Expr_U = RPN_1 + Lim_U + RPN_2 + Lim_U + RPN_3
Expr_V = RPN_1 + Lim_V + RPN_2 + Lim_V + RPN_3
Return mt_lutxy(Denoised,Src,YExpr=Expr_Y,UExpr=Expr_U,VExpr=Expr_V,U=3,V=3) # MUST SET U=3 and V=3 (Even when UExpr and VEXpr explicitly supplied, ELSE dont work proper).
}


Working for Avs+ Only

Function LimitChange(clip Src,clip Denoised,int LimitY,int "LimitU",int "LimitV") { # https://forum.doom9.org/showthread.php?p=1870697#post1870697
LimitU=Default(LimitU,LimitY) LimitV=Default(LimitV,LimitU)
Lim_Y = String(LimitY,"%.0f") Lim_U = String(LimitU,"%.0f") Lim_V = String(LimitV,"%.0f")
# "min(max(Denoised,Src-LimitY),Src+LimitY)"
# Infix = "min(max(x,y-LimitY),x+LimitY)" # where x is Denoised and y = Src
# RPN = "x y LimitY - max y LimitY + min"
RPN_1 = "x y "
RPN_2 = " - max y "
RPN_3 = " + min"
Expr_Y = RPN_1 + Lim_Y + RPN_2 + Lim_Y + RPN_3
Expr_U = RPN_1 + Lim_U + RPN_2 + Lim_U + RPN_3
Expr_V = RPN_1 + Lim_V + RPN_2 + Lim_V + RPN_3
Return Expr(Denoised,Src,Expr_Y,Expr_U,Expr_V)
}


Client script

YLIM=64 # Limit change (ie colorbars) to 64 from grey (all channels)
ULIM=YLIM
VLIM=ULIM

Denoised=ColorBars(Pixel_Type="YV12").BilinearResize(480,240)
Src=Denoised.BlankClip(Color_YUV=$808080)

A=Denoised.ShowChannels
B=LimitChange(Src,Denoised,YLIM,ULIM,VLIM)
C=Mt_LimitChange(Src,Denoised,YLIM,ULIM,VLIM)
B=B.ShowChannels
C=C.ShowChannels
StackVertical(A,B,C)


https://i.postimg.cc/QM1z9npZ/Limit-Change-Fixed.jpg (https://postimages.org/)

EDIT: Small mod to client script.

[ABurns]

Sweet!

I benchmarked both with AVSMeter64, and on a 1080p source LRemoveDust with MT_LimitChange() averaged around 53fps, while LimitChange() came in at around 42fps, so there appears to be a significant speed advantage in the mt_lutxy approach. I'm sticking with that one for now.

Thanks again.

[StainlessS]

Not surprising, mt_lutxy() is lut, whereas Expr() calc'ed in real time.
mt_lutxyz(), huge lut so Expr() would perhaps be favourable in that circumstance.