Dogway's Filters Packs

[anton_foy]

Originally Posted by Dogway

Oh shii... the AVS filter is so undocumented it looks like shady business haha

block_step, that is: https://github.com/WolframRhodium/Va.../kernel.cu#L49

Tried out BM3DCUDA again two days ago and also lowered block_step to 2 which rid the blocking.
I usually go here

[ChaosKing]

Undocumented? It's in the paramaters list here https://github.com/WolframRhodium/Va...UDA#parameters
It is mentioned 3 times in the read me (ok, note2 says blocking_step instead of block_step)

[Dogway]

There's also bm_range and ps_range so I made a preset matrix to conform to original profiles. For preset "fast" and "normal" I will lower the array by 1, I had to make an ArrayOp() function, just edited the old MatrixMul() in TransformsPack so now it's multipurpose.

[gispos]

Originally Posted by Dogway

Check this:

function HighlightLimiter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method") {

    bi  = BitsPerComponent(v)
    fs  = propNumElements (v,"_ColorRange")  > 0 ? \
          propGetInt      (v,"_ColorRange") == 0 : false

    grad    = Default (gradient,true)
    gblur   = Default (gblur, grad ? 100 : 5)
    thres   = Default (threshold,160)          #The lower the value, the more sensitive the filter will be.
    twopass = Default (twopass,false)
    amount  = Default (amount,20)
    method  = Default (method, 0) # 0 or 1

    amount  = ex_bs(abs(amount), 8, bi, true)

    darken  = v.ex_binarize(thres,hi=-1)
    blurred = grad ? darken.ex_gaussianblur(0.073*gblur+17.78,UV=1) : \
                     darken.ex_gaussianblur(0.064*gblur+ 3.75,UV=1)

    multiply = (method==0) ? v.ex_lut("x dup * range_max /", fulls=fs) : v.ex_lut(Format("x {amount} + ")+(!fs?"ymax min":""), fulls=fs)
    multiply = !twopass    ? multiply : \
               (method==0) ? multiply.ex_lutxy(v,"x y * range_max /", fulls=fs) : \
                             multiply.ex_contrast(0.15, 1, 163,   tv_range=!fs)
    return mt_merge(v,multiply,blurred) }


function HighlightLimiterMod(clip v, int "threshold", int "dark", float "rad",int "mode", float "gamma1_2") {

    w   = Width (v)
    h   = Height(v)
    bi  = BitsPerComponent(v)
    fs  = propNumElements (v,"_ColorRange")  > 0 ? \
          propGetInt      (v,"_ColorRange") == 0 : false

    thres   = Default (threshold,160)
    dark    = Default (dark, 255)
    rad     = Default (rad,    64)
    mode    = Default (mode, 0)       # 0,1,2
    gamma1_2= Default (gamma1_2, 0.6) # for mode 1,2

    m    = min(w,h)
    dark = ex_bs(dark, 8, bi, true)
    rad  = round(m/rad/4) < 1 ? max(round(m/4.0),16) : rad

    darken  = v.ex_binarize(thres,hi=-1)
    blurred = darken.ex_GaussianBlur(rad*0.83825-0.0631,pad=true,UV=1)

    multiply = (mode==0) ? v.ex_lut(Format("x dup * {dark} /"), fulls=fs) : \
               (mode==1) ? v.ex_levels(gamma=gamma1_2)                    : \
                           v.ex_lut(Format("x dup * {dark} /"), fulls=fs).ex_levels(gamma=gamma1_2)

    return mt_merge(v,multiply,blurred) }

Thanks Dogway. Results:
For HighlightLimiter, your version 'ex_HighlightLimiter' wins. Almost the same result but 10 times faster.

I can't decide about HighlightLimiterMod.
Your version is 8% faster but there are small differences. Sometimes darker areas are also slightly darkened.
Which parameters could I play around with?

On these?
blurred = darken.ex_GaussianBlur(rad*0.83825-0.0631,pad=true,UV=1)

Edit: Everything Ok. I just have to make the radius a little smaller to get almost the same result.

[Dogway]

8% or 8 times ?

Yes, mapping bicubic blur to gaussian blur is not exact but the later gives better results gaussian like, I couldn't decide if this or (rad-1) so tweak visually.


EDIT: By the way, I made a bandpass replacement filter to replace mid frequencies from one clip to another, please have a look if you find something odd. I think gaussian is not ideal here (might need mean blur) but anyway here it is:

# Bandpass replacement
# Replace Mid Frequencies of first clip from the second clip
#
# Bandpass workflow diagram (shaded '/' means deleted by lowpassing):
#
#     A               A                  A            A                 A
#  _______         _______            _______      _______           _______
# |       |       |/ / / /|          |       |    |/ / / /|         |       |
# |       |       | / / / |          |       |    | / / / |         |       |
# |       |       |-------|-> HI      -------     |/ / / /|          -------
# |       |       |       |                       | / / / |
# |       |   -   |       |     =               + |/ / / /|     =
# |       |       |       |                       |-------|-> LO     _______
# |       |       |       |                       |       |         |       |
# |       |       |       |                       |       |         |       |
#  -------         -------                         -------           -------
#                                                           A
#     B               B                  B                 /           A+B
#  _______         _______                                /          _______
# |/ / / /|       |/ / / /|                              /\ +       |       |
# | / / / |       | / / / |                             /  \        |   A   |
# |-------|-> HI  |/ / / /|           _______          /    \       |-------|
# |       |       | / / / |          |       |        B      -----> |       |
# |       |   -   |/ / / /|     =    |       |                      |   B   |
# |       |       |-------|-> LO      -------                       |-------|
# |       |       |       |                                         |       |
# |       |       |       |                                         |   A   |
#  -------         -------                                           -------
#
# Example:
#     ex_MFR(a,b,1700,700) # Replace mid frequencies from 700 to 1700 from 'b' to 'a'
#
#
function ex_MFR (clip c, clip ref, float "hi", float "lo", int "UV") {

    rgb  = isRGB (c)
    is42 = is420 (c)
    w    = width (c)
    h    = height(c)
    mx   = max(w,h)
    bi   = BitsPerComponent(c)
    fulls= propNumElements (c,"_ColorRange")  > 0 ? \
           propGetInt      (c,"_ColorRange") == 0 : rgb

    HI   = Default( HI, 1200*(w/1920.)) # Default is 1200 for 1080p
    LO   = Default( LO,         HI/2. ) # Range of Mid Frequency to restore
    UV   = Default( UV,              3) # Set UV to 4 to copy the chroma from the 'ref' clip
    UV2  = UV==4?2:UV

    LO   = max(LO, 4)

    Fs   = max(w,h) * 2                 # Frequency sample rate is resolution * 2 (for Nyquist)
    k    = sqrt(log(2)/2) * HI          # Constant for -3dB
    HI   = Fs / ( k * 2 * pi )          # Frequency Cutoff for Gaussian Sigma

    k    = sqrt(log(2)/2) * LO
    LO   = Fs / ( k * 2 * pi )

    LOUV = c.is422() || is42 ? LO/2. : c.isYV411() ? LO/4. : LO
    LOUVv= is42              ? LO/2. :                       LO
    HIUV = c.is422() || is42 ? HI/2. : c.isYV411() ? HI/4. : HI
    HIUVv= is42              ? HI/2. :                       HI

    mpc   =   c.vsTCanny(HI, HIUV, HIUVv, mode=-1, u=UV2, v=UV2)
    mpc2  =   c.vsTCanny(LO, LOUV, LOUVv, mode=-1, u=UV2, v=UV2)
    mpr   = ref.vsTCanny(HI, HIUV, HIUVv, mode=-1, u=UV2, v=UV2)
    mpr2  = ref.vsTCanny(LO, LOUV, LOUVv, mode=-1, u=UV2, v=UV2)

    Expr(c,   mpc, mpc2, mpr, mpr2, "x y - z + a b - +", isy(c) ? Undefined() : ex_UVexpr("x y - z + a b - +", UV2, bi, rgb, fulls, "none") ) }

[gispos]

Originally Posted by Dogway

8% or 8 times ?

Not a typo, 8 to 10% is already correct.

The normal version used:
blurred=darken.gaussianblur(gblur).gaussianblur(gblur+100).gaussianblur(gblur+200)

and that is slower than my Mod version:
blurred=darken.bicubicresize(round(clp.width()/rad/4)*4, round(clp.height()/rad/4)*4).bicubicresize(clp.width(),clp.height(),1,0)

therefore the difference to your variant is not so big:
blurred = darken.ex_GaussianBlur(rad*0.83825-0.0631,pad=true,UV=1)

Originally Posted by Dogway

EDIT: By the way, I made a bandpass replacement filter to replace mid frequencies from one clip to another, please have a look if you find something odd.

Um, I do not know what to do with it? There are no reference clips for my clips. Surely this must be the same content?

Edit:
I think the topic of lightening dark areas or reducing light areas interests many users here. One could prepare with it here many a joy.

Edit2:
The normal version you changed 'ex_HighlightLimiter' is the best, same results but 8 to 10 times faster. Thanks again.

[Dogway]

Yes I noticed, you were already using bicubic for blurring which is inherently fast, you can also try vsTCanny for blurring, for mid sigmas sometimes it's faster than ex_gaussianblur().
I edited the scripts because I missed 'tv_range' in ex_levels. Well 10% is already very good in my book. These filters look to me like mini Retinex filters, the concept works fine, I will have to compare to ex_retinex() for speed and quality benchmarks.

No, ex_MFR() is totally unrelated, I was testing on improving UVRecon in TransformsPack finally used ex_LFR though.

Also for anyone interested, I'm making a window kernel list in Desmos for study (edited from here) and include in TransformsPack, still investigating on how to derive 'taps'.

[gispos]

Dogway, one more question:
You use the properties for TV_Range, I don't really like that. At least I still use filters that don't pass properties.
I have added tv_range as parameter and and also changed the 'fs' assignments in the code. Do you see any disadvantages?

function ex_Highlighter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method", bool "tv_range") {

    bi  = BitsPerComponent(v)
    fs = default(tv_range, True)

    grad    = Default (gradient,true)
    gblur   = Default (gblur, grad ? 100 : 5)
    thres   = Default (threshold,100)          #The lower the value, the more sensitive the filter will be.
    twopass = Default (twopass,false)
    amount  = Default (amount,10)
    method  = Default (method, 1) # 0, 1 or 2

    amount  = ex_bs(abs(amount), 8, bi, true)
    thres   = ex_bs(thres,       8, bi, true)

    lighten = v.ex_lut(Format("x {thres} > 0 x ?"), UV=128)
    blurred = grad ? lighten.ex_gaussianblur(0.073*gblur+17.78,UV=1) : \
                     lighten.ex_gaussianblur(0.064*gblur+ 3.75,UV=1)

    photoshop_overlay = (method==1) ? ex_lutxy(v,blurred,"x range_half > range_max y - Y@ range_max x - X@ range_half / * range_max + Y x range_half / * ? ",  fulls=!fs) : \
                        (method==2) ? ex_lutxy(v,blurred,"range_max range_max x - y * range_max / -",fulls=!fs) : v.ex_lut(Format("x {amount} + ")+(fs?"ymax min":""), fulls=!fs)

    photoshop_overlay = twopass ? photoshop_overlay.ex_contrast(-0.8,1,167,tv_range=fs) : photoshop_overlay
    return mt_merge(v,photoshop_overlay,blurred) 
}

[Dogway]

'fulls' should have the opposite sign of 'tv_range' so I think it would make better sense to write "fs = !Default(tv_range, True)", then change according in following calls (whether the filter expects 'fulls' or 'tv_range').
But what I normally do is "tv = Default(tv_range, True)" and use 'tv' or '!tv' upon requirements.

Yes, filters that don't pass frameprops is a problem, maybe we should make a list of them. I have been thinking for some time as a header filter to define initial frameprops aside those assigned by the loader, so you can add _ColorRange, or override other frameprops. I have also been thinking on a color space detection but I will leave that for the future.

[gispos]

Hello Dogway,
I just downloaded your latest version.

Very nice that you have already added the AvsPmod function definetion to many functions.
Also the histogram curves are very nice (especially with 'Preview Filter ).

A wish from me to ex_contrast:
Is it possible to add another pivot slider? Or something like a 'soft limiter'.
So that you can delimit dark, middle and light areas more.

What I noticed is that the contrast value in the middle range does not allow any 0.01 changes, only from 0.1 can visible changes of 0.01 be made.
This is probably due to mathematical facts and cannot be improved?

I just don't like setting parameters by reading the frame properties. This should not be done in hidden functions
Why not set 'tv_range' manually? But this is only my opinion.

[Dogway]

Yes, with the preview histograms and your preview filters this is starting to look like a decent alternative to more modern packages. Just tested Preview Filters and it's awesome, one problem though, I have to add 'last' to my script after '**/' or AvsPmod fails to render output (in case my preview filter is the last one in the script).

I didn't understand your question. You can play with 'pivot' to have dark or light areas affect more, for middle you have gamma in ex_levels (not exactly unbiased but mostly).

ex_contrast function definition is "by 0.1", does it work if you change it to "0.01"? I see changes in the histogram curve going from 0.0 to 0.01 and all the way up to 0.1.

I had a few wishes for function definition, one is max stepping for int types is '4', I don't know why. The other is string types doesn't understand anything that is not alphanumeric, so "string preset="[fast]" ("[slow]" / "[fast]") " fails.

As for frameprops, in my functions they are only read as a fallback (Defaults) if you explicitly don't define the argument, so no problem with that.

[gispos]

Originally Posted by Dogway

I didn't understand your question. You can play with 'pivot' to have dark or light areas affect more, for middle you have gamma in ex_levels (not exactly unbiased but mostly).

ex_contrast function definition is "by 0.1", does it work if you change it to "0.01"? I see changes in the histogram curve going from 0.0 to 0.01 and all the way up to 0.1.

There is a difference whether I can adjust the curve with only one or two controllers and thus have, for example, a control value that can be adjusted from the dark to only the middle areas. SmoothLevels, for example, has extra controls for dark and light areas and the mode parameter can be used to adjust the behavior.

It depends on how you specify the slider values (I do not use your examples). Values from 0.01 are set visible, but only after the value has at least 0.1
From 0.00 to 0.01 there is a jump as if 0.1 was set.

Originally Posted by Dogway

I had a few wishes for function definition, one is max stepping for int types is '4', I don't know why.

I don't know that either. But it should depend on the maximum value of the slider. Must see if I can find that.

Originally Posted by Dogway

The other is string types doesn't understand anything that is not alphanumeric, so "string preset="[fast]" ("[slow]" / "[fast]") " fails.

I think it's because there are valid rules for self-created sliders. And [ ] is for example responsible as a character for slider start and end.
At that time there were e.g. also no arrays in avisynth. The whole mesh around the slider is far-reaching and requires time, effort and desire to tackle.
Edit: In the function definitions [ ] are a flag for hidden parameters. Have a look at the filterdb.dat. The whole thing must also be readable and implementable, so there are certain rules.

Originally Posted by Dogway

As for frameprops, in my functions they are only read as a fallback (Defaults) if you explicitly don't define the argument, so no problem with that.

Ok.

AvsPmod fails to render output (in case my preview filter is the last one in the script).
Sometimes I also have to use a return last in the script. These are the requirements avisynth has for its syntax.
I can't check with every parameter change if a return last is present and also at a valid position. You will manage that!

[gispos]

Originally Posted by Dogway

I had a few wishes for function definition, one is max stepping for int types is '4', I don't know why.

It works for me. max and min and default values must of course be divisible by e.g. 5 or must be 0

There is even a warning if you do not meet these requirements.
I just defined a slider with the values: min 0 default 100 max 200 step by 50

[ENunn]

Does anyone know why Avisynth has been more unstable since I installed the scripts? I'm trying to use QTGMC and the moment I try to load the preview Avspmod insta-crashes. Ffmpeg does nothing and crashes as well. Before I kept getting access violation errors with ZS_RF_Shared.avsi, either that or it would load fine when it feels like it. Now it's not loading at all. I have the latest version of the script and the dependencies so I'm not sure what's up.

[gispos]

Originally Posted by ENunn

I try to load the preview Avspmod insta-crashes.

What version of Avisynth are you using?

[ENunn]

Originally Posted by gispos

What version of Avisynth are you using?

3.7.1.

I cleared out my plugins folder and installed them all back in, looks like neo_fft3d.dll might be the culprit for the crashing, which kinda sucks because I use tempdegrain2 a lot. But now I'm getting random access violations with ExTools.avsi.

So my BM3D woe's continue.

I have several older PC's that aren't AVX2, BUT they have reasonable nVidia GPU's for CUDA / OpenCL support.

Naturally BM3D CPU doesn't work, BUT I just can't get BM3D CUDA to work on them. I really thought I got it to work before...but can't get it to, anymore

KNLMeansCL & DGDenoise work fine.

So do I just forget about using BM3D pre-filters ???, what's the next best thing ??

I'm tearing my hair out (and I haven't got much of that anyway) !!

Help...

[Dogway]

@TDS: I think your issue would probably be one of resources distribution through the server. Maybe CUDA instructions are not passed, you can try first outside avisynth to check if it's a server or an AVS problem, for example with a GPU benchmark tool. As a fallback KNLMeansCL is a nice alternative as it's also temporal.

@ENunn: Try updating to 3.7.2 test3, some critical bugs were fixed there, just in case to rule out other possibilities.

[kedautinh12]

@TDS: you can try change radius=1 to preset="low" here
https://github.com/Dogway/Avisynth-S...4.7d.avsi#L248

Originally Posted by kedautinh12

@TDS: you can try change radius=1 to preset="low" here
https://github.com/Dogway/Avisynth-S...4.7d.avsi#L248

Hi,

Is this what you mean :-

(prefilter== 6) ?  pref.ex_BM3D(sigma=10,preset="low",CUDA=true,UV=Chr,tv_range=tv)