Dogway's Filters Packs

[Dogway]

#ex_boxblur(5,mode="mean")
Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()

but I can see that they are not similar

They are the same, the only difference is that Dither_box_filter16 is frequency domain based so it's faster for higher sigma, when above 5 in ex_boxblur().
a=Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()
ex_boxblur(4,mode="mean")
ex_makediff(a, aug=true, UV=128)

So if the question is if we need a DFT box filter, probably the answer is yes, same with binomial blur. Of course with HBD support and asymmetric blur option. I would also vouch for porting CTM (Constant Time Median) from VapourSynth, so doing a 5x5 or 7x7 median isn't as slow as it is now.

ex_bilateral() gives same output as vsTBilateral(), speed is similar as well. Dither_bilateral16() employs shortcuts to speed up the process but in my opinion the output doesn't look nice.

Excerpt from docs.
"Actually, this implementation is an approximation of a bilateral filter, like many other image processing tools do. The spatial weighting is not a gaussian curve but a box, and the value weighting is a trapezoid."

Thanks for LimitFilter link, I will have a look when I got time, I've been busy porting *.mp4 sharpeners

[real.finder]

a=Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()
ex_boxblur(4,mode="mean")
ex_makediff(a, aug=true, UV=128)

so Dither_box_filter16(x) = ex_boxblur(x-1, mode="mean")

CTM (Constant Time Median) from VapourSynth

https://github.com/pinterf/MedianBlur2

Dither_bilateral16() employs shortcuts to speed up the process but in my opinion the output doesn't look nice.

but there are people who like the Dither_bilateral16 more than others

[Dogway]

MedianBlur2 didn't perform well on my benchmarks, something is going on there, occasionally it would perform "fine" but must of the time it crawls so I don't know.

I will try to post comparison shots for ex_bilateral() and Dither_bilateral16() when I got some time.

[Dogway]

Here's the comparison (brighten to see better):

Converttostacked().Dither_bilateral16(radius=2,thr=10,flat=1.0,u=1,v=1).Convertfromstacked()
https://i.imgur.com/hoy2mTk.png

ex_bilateral(1)
https://i.imgur.com/qHznfYB.png

As you can see Dither_bilateral16() output is very dirty/grainy, and artifacts similar to combing.



By the way, updated SharpenersPack with new sharpeners; SSSharp, ReCon, blah, MedSharp, ex_unsharp and SharpenComplex2

[real.finder]

Here's the comparison (brighten to see better):

Converttostacked().Dither_bilateral16(radius=2,thr=10,flat=1.0,u=1,v=1).Convertfromstacked()
https://i.imgur.com/hoy2mTk.png

ex_bilateral(1)
https://i.imgur.com/qHznfYB.png

As you can see Dither_bilateral16() output is very dirty/grainy, and artifacts similar to combing.



By the way, updated SharpenersPack with new sharpeners; SSSharp, ReCon, blah, MedSharp, ex_unsharp and SharpenComplex2

I think it depends on the settings and the source

here a friend words

https://i.postimg.cc/Kc5D2Zh0/1.png (https://postimages.org/)

https://i.postimg.cc/wTTcxS6g/2.png (https://postimages.org/)

[guest]

Well, I have tried a lot of different things to diagnose this, but it seems to be on only 4 servers that I use for the Distributed Encoding function of RipBot.

So they are simply connected via a LAN, and don't need the complete program to encode (very clever, actually), and yet other pc's on the LAN don't have the problem,

So I think my only recourse is a fresh setup of Windows.

Update:- It turned out that I needed to update these:-VC_redist.x86.exe & VC_redist.x64.exe

There must be some subtle change in Dog's ExTools & SMDegrain.

I've finally figured out what the problem is....

Even tho it was good to update VC, it's an OpenCL problem when using KNLMeansCL, as a pre-filter in Doggy's SMDegrain !!!

The pc's that have this error haven't got a compatible GPU, and I'm not sure if you can get a "driver" for older Xeon Intel CPU's for OpenCL Support.

[kedautinh12]

Wait a minute, i remember you said you had Ryzen cpu??

[guest]

Wait a minute, i remember you said you had Ryzen cpu??

Good memory, I actually have 2 :)

I also have a few I7 Intels, some E5 Xeons, And a LOT of older X series Xeons.

And the older Xeons have no "proper" Gpu's....

Its a RipBot264 Distributed Encoding set up...

[kedautinh12]

Good memory, I actually have 2 :)

I also have a few I7 Intels, some E5 Xeons, And a LOT of older X series Xeons.

And the older Xeons have no "proper" Gpu's....

Its a RipBot264 Distributed Encoding set up...

Rich oldman. When you want discard any old amd ryzen 5000 series, can you give me :D :D :D

[Emulgator]

Be kind, ask for grannys. These are good too ;-)

[guest]

You welcome

;)

I'll see you @ the other Forum.

[Dogway]

ExTools (https://github.com/Dogway/Avisynth-Scripts/blob/master/ExTools.avsi) updated to v5.4
-ex_vinverse() - small optimization
-ex_blur() - 'bifit' arg to toggle between gaussian or binomial distribution
-ex_guidedblur(), ex_ANguidedblur() - moved to yugefunc.avsi
-ex_GaussianBlur() - New. Optimized true gaussian blur approximation for high sigma
-ex_luts() - bugfix when using two clips. 'exprf' arg to append code to the end.
-ex_shape() - add anisotropy for shapes (except 'cross')
-sel_med() - more medians, fill up to n=49

Just updated ExTools to v5.4, required for my recent updates (from today updates, and before for SharpenersPack). This feels like the last version although I will fix any reported bugs or add things if it feels convenient, but my update rate on ExTools will be much slower from now on.
I also uploaded today v1.0 of SharpenersPack (https://github.com/Dogway/Avisynth-Scripts/blob/master/SharpenersPack.avsi), because it feels very solid. There are possible additions like Oyster, qUINT Sharp, NLM Sharp and other kind of sharpeners, but for now it will stay like this.
Also updated SMDegrain (https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain%20v.3.2.8d/SMDegrain%20v3.2.8d.avsi)to refine ex_retinex() with the new ex_GaussianBlur() function.

Now I will focus on yugefunc (https://github.com/Dogway/Avisynth-Scripts/blob/master/yugefunc.avsi)and port worthy functions from VapourSynth, I added a list in the header for the filters I wish to port.
I will also have a look at QTGMC to optimize it more, port Didée's LTSMC, remake ex_vibrance() and whatever arises.

[guest]

ExTools (https://github.com/Dogway/Avisynth-Scripts/blob/master/ExTools.avsi) updated to v5.4
-ex_vinverse() - small optimization
-ex_blur() - 'bifit' arg to toggle between gaussian or binomial distribution
-ex_guidedblur(), ex_ANguidedblur() - moved to yugefunc.avsi
-ex_GaussianBlur() - New. Optimized true gaussian blur approximation for high sigma
-ex_luts() - bugfix when using two clips. 'exprf' arg to append code to the end.
-ex_shape() - add anisotropy for shapes (except 'cross')
-sel_med() - more medians, fill up to n=49

Just updated ExTools to v5.4, required for my recent updates (from today updates, and before for SharpenersPack). This feels like the last version although I will fix any reported bugs or add things if it feels convenient, but my update rate on ExTools will be much slower from now on.
I also uploaded today v1.0 of SharpenersPack (https://github.com/Dogway/Avisynth-Scripts/blob/master/SharpenersPack.avsi), because it feels very solid. There are possible additions like Oyster, qUINT Sharp, NLM Sharp and other kind of sharpeners, but for now it will stay like this.
Also updated SMDegrain (https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain%20v.3.2.8d/SMDegrain%20v3.2.8d.avsi)to refine ex_retinex() with the new ex_GaussianBlur() function.

Now I will focus on yugefunc (https://github.com/Dogway/Avisynth-Scripts/blob/master/yugefunc.avsi)and port worthy functions from VapourSynth, I added a list in the header for the filters I wish to port.
I will also have a look at QTGMC to optimize it more, port Didée's LTSMC, remake ex_vibrance() and whatever arises.

Busy, busy, busy...I'm having a hard time just keeping up :)

:thanks:

Edit :- Unfortunately, since changing up to SMDegrain 3.2.8, I'm getting another Line 161 error...go back to 3.2.7, it's good :), when using BM3D in CPU mode.

On further investigation, the difference was :-

function ex_BM3D(clip a, float "sigma", int "radius", int "UV") {

rgb = isRGB(a)
isy = isy(a)
bi = BitsPerComponent(a)

s = Default(sigma, 3)
r = Default(radius, 1)
UV = Default(UV, rgb ? 3 : 1)

ch = !(isy || UV!=3)
ch ? ConverttoYUV444(a,chromaresample="spline16") : a
ConvertBits(bits=32, fulls=fs)

BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)
BM3D_VAggregate(radius=r)

if (bi < 32) {
ConvertBits(bits=16, dither=-1, fulls=fs)
ConvertBits(bits=bi, dither= 1, fulls=fs)
}
ch ? MatchColorFormat(a) : mergechroma(a) }

full=fs was the only change I could see, so whatever that's for, didn't work for me. :(

Also note other change in "red" (thanks to kedautinh12)

[tormento]

Unfortunately, since changing up to SMDegrain 3.2.8, I'm getting another Line 161 error...go back to 3.2.7, it's good :), when using BM3D in CPU mode.
Updated to last BM3D of a few days ago?

[tormento]

I will also have a look at QTGMC to optimize it more, port Didée's LTSMC, remake ex_vibrance() and whatever arises.
Is Zs_RF_Shared still required to use SMDegrain?

[guest]

Updated to last BM3D of a few days ago?

I only have a problem when using it in CPU mode..

[Dogway]

@Pauly Dunne: Thanks for the report! I updated SMDegrain, last minute changes are never good. Also added CPU mode but it's not callable from SMDegrain args, just change to CUDA=false in line 159.

@tormento: I think it's needed only for sh_GetUserGlobalIMTint(), but when I used it it didn't report my threads. I still don't know how AviSynth doesn't have a thread reporting internal function...

[real.finder]

@tormento: I think it's needed only for sh_GetUserGlobalIMTint(), but when I used it it didn't report my threads. I still don't know how AviSynth doesn't have a thread reporting internal function...

there are https://forum.doom9.org/showthread.php?p=1925253#post1925253 but seems no one use it

[Dogway]

Can avstp report cores/thread as string?

Here an image for ex_shapes(), still need to work out cross anisotropy (EDIT: fixed):
https://i.imgur.com/VAu8ahP.gif

[real.finder]

Can avstp report cores/thread as string?

don't think so, I didn't mean avstp but https://forum.doom9.org/showthread.php?p=1777021#post1777021 and https://forum.doom9.org/showthread.php?p=1778346#post1778346

[Dogway]

Yes, the MT resizers. I used them some time ago, then I benchmarked recently and wasn't as fast as internal. Maybe you need several resizer calls to see the improvement, I will test that but IMO these kind of things should be dealt directly in AVS+, I for one am aware of the issue and try to merge redundant Expr() calls into one to reduce the call penalty (used registers).

By the way, big update day. SMDegrain, QTGMC, ExTools, GradePack, etc
SMDegrain: reworked ex_retinex() to don't cause flicker in new 'lvl' mode 2 while keeping performance, 'lvl' 1 uses fixed levels (Default)
ExTools: basically adding 'tv_in', 'tv_out' args to ex_bs() so you can convert values from PC levels to TV levels and opposite.
QTGMC: Performance update, some cleanup, and a few small bugs. Still needs 32-bit support.
GradePack: reworked chroma in ex_contrast() and ex_levels() to make more sense. Actually ex_levels() reaches parity with Levels() regarding chroma. Also added more dither modes to ex_posterize() by using fmtc_bitdepth(). Still need to update ex_vibrance().
Other: Small updates to ResizersPack, DeStripe, FastLineDarkenMOD, LSFMod, FrameRateConverter and GrainFactory3mod.

Please report if you find any issue.

[guest]

Yes, the MT resizers. I used them some time ago, then I benchmarked recently and wasn't as fast as internal. Maybe you need several resizer calls to see the improvement, I will test that but IMO these kind of things should be dealt directly in AVS+, I for one am aware of the issue and try to merge redundant Expr() calls into one to reduce the call penalty (used registers).

By the way, big update day. SMDegrain, QTGMC, ExTools, GradePack, etc
SMDegrain: reworked ex_retinex() to don't cause flicker in new 'lvl' mode 2 while keeping performance, 'lvl' 1 uses fixed levels (Default)
ExTools: basically adding 'tv_in', 'tv_out' args to ex_bs() so you can convert values from PC levels to TV levels and opposite.
QTGMC: Performance update, some cleanup, and a few small bugs. Still needs 32-bit support.
GradePack: reworked chroma in ex_contrast() and ex_levels() to make more sense. Actually ex_levels() reaches parity with Levels() regarding chroma. Also added more dither modes to ex_posterize() by using fmtc_bitdepth(). Still need to update ex_vibrance().
Other: Small updates to ResizersPack, DeStripe, FastLineDarkenMOD, LSFMod, FrameRateConverter and GrainFactory3mod.

Please report if you find any issue.

I thought you'd finished modding these....:eek:

No problems for me :)

[Julek]

I don't know if this has been ported to AVS yet, but chroma recontructing would be a good function for you.
And since the AVS's Expr is more complete than the VS's Expr you might get an even better result.

Func:
https://github.com/Irrational-Encoding-Wizardry/lvsfunc/blob/master/lvsfunc/recon.py#L70-L119
https://en.wikipedia.org/wiki/Simple_linear_regression

Usage:
https://github.com/Ichunjo/encode-scripts/blob/b016e0b1e64e627051086a6e54555e9b14e7a11b/Raws-Maji/Lapis%20ReLiGHTs%20%5BBD%5D/lapis_bdmenu03.py#L72-L93
https://github.com/Ichunjo/encode-scripts/blob/b016e0b1e64e627051086a6e54555e9b14e7a11b/Other%20groups/MTBB/Sirius%20%5BBD%5D/sirius_03.py#L182-L207
https://github.com/Ichunjo/encode-scripts/blob/b016e0b1e64e627051086a6e54555e9b14e7a11b/Raws-Maji/Rifle%20is%20Beautiful%20%5BBD%5D/rifle_bdmenu_02.py#L68-L90

Result: https://slow.pics/c/MfqBYVId

[Dogway]

I thought you'd finished modding these....:eek:

ex_retinex() always had problems with the autolevels, so it was prone to change. Yesterday I also tested gamma aware blurring for retinex and results were worse even after normalization, so I'm always testing how to improve things or make them faster. The core of SMDegrain though is pretty much done with the current concept. MatteCrop needs an update for the lookback loop, I will replace it with frameProps but didn't have time yet to implement. GrainFactory3mod is slow, I want to replace it with something new. It's a bit of a cleanup before I move to yugefunc.


@Julek: Looks good! anything that is math clean and fast is welcome. I will have a look at linear regression and see how it fits although the chroma reconstruction function shown includes some over the top filtering with MaskedDHA, QTGMC, etc. I already had a chroma reconstruction (https://github.com/Dogway/Avisynth-Scripts/blob/205307cba48b581634b91875496aebf7906981ae/TransformsPack.v1.0.RC18.avsi#L376) function (https://github.com/Dogway/Avisynth-Scripts/blob/205307cba48b581634b91875496aebf7906981ae/MIX%20mods/ChromaReconstructor%20v3mod.avsi#L92)ported from feisty2.
By the way yesterday I ported adaptive threshold successfully, I'm still on optimization stage. Do you have performance numbers for 1080p@16-bit (without bitdepth down) and CPU? I'm managing 3 parameters, threshold, blur scale, and integral size.

[anton_foy]

Tried using ex_median() and I get error: Script error: Expr: Failed to convert 'max to float' (extools.avsi, line 2511)

Am I missing something?

[Dogway]

"DGM0" ? Works fine here. Are you in avs+ 3.7.1?

[anton_foy]

"DGM0" ? Works fine here. Are you in avs+ 3.7.1?

I sure hope so, because it gave me another error message "need to upgrade to avisynth 3.7.1+" before so I upgraded and now this error message turned up (I replaced the old avisynth.dll with the new x86 3.7.1 one in sysWOW64 and system32 folder)

[Dogway]

In system32 goes the one with CRC32: D12B9040 and in SysWOW64 the one: 4B48E737 (size is bigger here)

EDIT: you need version test9, check here (https://forum.doom9.org/showthread.php?p=1946928#post1946928).

[anton_foy]

In system32 goes the one with CRC32: D12B9040 and in SysWOW64 the one: 4B48E737 (size is bigger here)

Sorry for the stupid off topic questions. I use 32-bit Avisynth+.
Which ones are CRC32: D12B9040 and 4B48E737?
Two different avisynth.dll's?

[Dogway]

Yes, x64 version goes to system32 and x86 goes to sysWOW64 if I reckon correctly. Check the link in my edit above. Then with avsmeter you can check your install is fine: "avsmeter64 avsinfo -lf"

@Julek: ex_athres(th=40,sigma=6) running here at 194fps for my CPU with Prefetch(8). It can be sped up slightly but I need this (https://forum.doom9.org/showthread.php?p=1951860#post1951860) avs+ bug fixed.

[anton_foy]

Yes, x64 version goes to system32 and x86 goes to sysWOW64 if I reckon correctly. Check the link in my edit above. Then with avsmeter you can check your install is fine: "avsmeter64 avsinfo -lf"


:thanks: it is working now!

[Julek]

@Julek: ex_athres(th=40,sigma=6) running here at 194fps for my CPU with Prefetch(8). It can be sped up slightly but I need this (https://forum.doom9.org/showthread.php?p=1951860#post1951860) avs+ bug fixed.

3001 frames 1080p (AMD 3700X)
src: Time elapsed: 0:06.013 - 499.10511998696642876894 FPS
VS + OpenCV adaptiveThresh (luma only) : Time elapsed: 0:11.184 - 268.33492243462092119444 FPS

I can only test in 8 bits, because that's how VS+CV works

[Dogway]

Then more or less the same if it was processed in 16-bit, will upload tomorrow if possible with multi-otsu or adaptive histogram.

[guest]

In system32 goes the one with CRC32: D12B9040 and in SysWOW64 the one: 4B48E737 (size is bigger here)

EDIT: you need version test9, check here (https://forum.doom9.org/showthread.php?p=1946928#post1946928).

Hey Dogway,

After clicking on the link, I found test 11 & test 12 of 3.7.1, so naturally I grabbed them, unfortunately both test 11 & 12 give me a Line 161 error in SMDegrain 3.2.9d :(

So I had to return to test 9 :) or :(

[Dogway]

Yeah, it's because this commit (https://github.com/AviSynth/AviSynthPlus/commit/f89f473ffb62f3a96457e085aad305318b37e7a6) yesterday, guess my trick doesn't work now, will have to ask WolframRhodium to make sigma arg array type. I will update it.

@Julek: Just uploaded ex_athres(). It's not mathematically exact to the OpenCV implementation because I didn't check its source code but comparing with your test clips and other samples I found it works wonders. Now it looks to me more like an edge detection filter than a threshold per se. You need AVS+ 3.7.1 test12 released yesterday for the integral optimizations. It's basically a DoBtG (Difference of Box to Gaussian) but the integral helps to clean the output a bit more.

[real.finder]

Yeah, it's because this commit (https://github.com/AviSynth/AviSynthPlus/commit/f89f473ffb62f3a96457e085aad305318b37e7a6) yesterday, guess my trick doesn't work now, will have to ask WolframRhodium to make sigma arg array type. I will update it.


https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/actions/runs/1215772307

[guest]

Yeah, it's because this commit (https://github.com/AviSynth/AviSynthPlus/commit/f89f473ffb62f3a96457e085aad305318b37e7a6) yesterday, guess my trick doesn't work now, will have to ask WolframRhodium to make sigma arg array type. I will update it.

Hey Dogway,

Have been updating things again, with Extools 5.6, etc. and I grabbed this :-

https://github.com/WolframRhodium/Va...uns/1215772307 (thanks real.finder)

Wolfram changed quite a bit in the latest build :)

re-addressed my scripts, and it now seems to be working with AVS test 12 :)

[Julek]

@Julek: Just uploaded ex_athres()

It's not working here.
Expr: Failed to convert 'B^' to float
Same thing with 3.7.1 test12

[Dogway]

Are you on linux? It certainly looks like it's not returning the next string line correctly, this was implemented a few test versions earlier but at least it works for me on Windows.

EDIT: Can you run the next?
Expr(last," x X^
range_max X - abs","")

The line carriage in ExTools should be Line Feed -> "LF"

Test other examples like ex_median(mode="median5")

[Julek]

I am on windows 10, the problem was the line break, I edited it to one line and it worked.
I'm not very familiar with AVS and may not have installed test12 correctly.
I will do more tests tonight.

[Julek]

I did some tests now and the result is awesome, maybe even better than OpenCV for my use,
and the best thing is that it gets this result without the luma equalization that I used with OpenCV,
so the speed is much higher if you take that into account, great job.

Now I have to wait for Akarin to update his plugin, because I cannot use B^ in VS Expr :(
https://github.com/AkarinVS/vapoursynth-plugin

#EDIT: I think I learned how to use Akarin's "var!" and "var@", it can replace the "^".

[Dogway]

Good to hear. I don't know about the luma equalization. If you installed avs+ test 12 correctly you shouldn't have issues with the line carriage.
By the way my gaussian blur values are true gaussian sigma approximations not binomial so it might differ depending on what you use on VS.

[tormento]

Good to hear.
I am having problems with the "cosmetical enhanced" version of ExTools 5.6. The same version old one works ok.

Error:

(D:/Programmi/Media/AviSynth+/plugins64/ExTools-5.6new~Dogway.avsi, line 514, column 14)

[Dogway]

Thank you, fixed now. I always get the order wrong of ">="

[wonkey_monkey]

Thank you, fixed now. I always get the order wrong of ">="

The crocodile ( > ) always wants to eat the bigger number: 8 > 3

https://img.ifunny.co/images/a0775459ee5b597246cf16c5f1225356a3e7c1fb6347f2179d7e0969e717de72_1.webp

[Dogway]

Hehe, it's actually whether >= or =>. In language normally is said "equal or better/greater"

[kedautinh12]

Hehe, it's actually whether >= or =>. In language normally is said "equal or better/greater"

In my country use "=>" mean arrow :D

[tormento]

Hehe, it's actually whether >= or =>. In language normally is said "equal or better/greater"
Can I suggest you to use Notepad3 (https://github.com/rizonesoft/Notepad3) with a glyph enabled font, such as Cascadia or Iosevka?

If you get the correct order, it will transform the >= to ≥ but only visually.

There is AVS syntax highlight already present.

[Dogway]

Can I suggest you to use Notepad3 (https://github.com/rizonesoft/Notepad3) with a glyph enabled font, such as Cascadia or Iosevka?

Oh my gosh, thank you! I already use Notepad2, didn't have a clue about a third version, thought it was abandoned.

It might be simple for some who work on full fledged IDEs, but this is my most heavily used program on the PC. Hopefully it has improved regex and fixed crashes with "Mark Occurrences"

[tormento]

Oh my gosh, thank you!
You are the most welcome.

I warmly suggest to join the GitHub community. The AVS syntax needs a refresh.

this is my most heavily used program on the PC

Here too. It had a Notepad debug integration, so that every program calling for Notepad would have used Notepad 3 but the latest builds of Windows removed that feature.