Dogway's Filters Packs

[guest]

Megui tools up to date if you change sever to development
https://forum.doom9.org/showthread.php?p=1946568#post1946568

Fair enough, but I don't think I need to learn how to use another app.

But have you had any experience with RipBot ??

[kedautinh12]

I'm now getting a Line 160 error, when I change BM3D, CUDA to CPU, when using prefilter=5.

But 3.2.6d works on the other scripts / filters. :)

You changed command in this line or just change LoadPlugin??
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

[kedautinh12]

Fair enough, but I don't think I need to learn how to use another app.

But have you had any experience with RipBot ??

No, i'm just experience with use scripts in avs+ and i think 2 app (ripbot & megui) work same scripts structure with avs+

[guest]

You changed command in this line or just change LoadPlugin??
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

I just changed the whole thing !!!

[guest]

No, i'm just experience with use scripts in avs+ and i think 2 app (ripbot & megui) work same scripts structure with avs+

Well, I only use RipBot, and I have only just recently started to figure out how scripts work...a LOT of trial & error, and help from guy's like you, & Dogway, etc.

The main reason I started was RipBot wasn't getting any regular updates, like it use to get, not sure why, maybe the dev "Atak" has lost interest.

It's rather out of date, now :(...still works well, especially the Distributed Encoding function.

Cheers

[guest]

If you want use BM3D_CPU just change line above from BM3D_CUDA(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch, fast=true, extractor_exp=6) to BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)


Hi, I read somewhere that if you want to use the BM3D_CPU option, you don't need BM3D_VAggregate, is that correct ??? (haven't had a chance to test it for myself)

[kedautinh12]

Hi, I read somewhere that if you want to use the BM3D_CPU option, you don't need BM3D_VAggregate, is that correct ??? (haven't had a chance to test it for myself)

No, bm3d.VAggregate should be called after temporal filtering
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA#notes

[Dogway]

Very good news, advancements in ex_luts(), finally managed to port "*.mp4 guy" DeStripe() filter, I post the benchmarks.
Normally with plain pixel fetching Prefetch(4) (cores in my case) is faster, when expressions are involved I get more speed with Prefetch(6)

# P(4) 29 P(6) 36.5 (8-bit)
# P(4) 244 P(6) 271 (8-bit)
# P(4) 179 P(6) 184 (16-bit)

Original DeStripe() in 16-bits doesn't even start at all. With this I finally can port more filters that make use of ex_luts() like some other sharpeners. Will upload ExTools on Monday most likely. Just sharing the excitement : P


By the way, is there any way to retrieve Cores/Threads so I can automatically set correct Prefetch() for some functions/modes?

[kedautinh12]

Your destripe() ported need cullresize.dll same destripe in vs (.py), avs (.avsi) or don't need anymore??
https://github.com/YomikoR/VapourSynth-Destripe#related

[Reel.Deel]

Your destripe() ported need cullresize.dll same destripe in vs (.py), avs (.avsi) or don't need anymore??
https://github.com/YomikoR/VapourSynth-Destripe#related

*.mp4 guy's DeStripe (https://forum.doom9.org/showthread.php?t=154863&page=2#post1408276) has never required CullResize. The other Destripe script is a scaling script that uses CullResize and is used for fixing cross-conversion.

PS: there's an edit button that helps avoid posting back to back :p

[guest]

Hello again,

Can someone please tell me whats missing with this script :-

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCPU_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\dfttest\dfttest.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain326d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)

It has a error with Line 160 of SMDegrain3.2.6d.


But if I run this, there's no error :-

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCUDA_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\dfttest\dfttest.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain326d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)

BTW, I'm using either a 3950X or a 5950X.

:thanks:

[kedautinh12]

Hello again,

Can someone please tell me whats missing with this script :-



It has a error with Line 160 of SMDegrain3.2.6d.


But if I run this, there's no error :-



BTW, I'm using either a 3950X or a 5950X.

:thanks:

In BM3D_CPU don't have fast=true, extractor_exp=6, just deleted it when you change BM3D_CUDA to BM3D_CPU
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

[guest]

In BM3D_CPU don't have fast=true, extractor_exp=6, just deleted it when you change BM3D_CUDA to BM3D_CPU
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

You're awesome, I could make a "custom" SMDegrain with those deleted, and get the script to call the modded one.

I'll give it a shot, and let you know :)

Reply:- OK, that didn't work at first, but when I changed something else, it did, but it's SLOW !!!!

BM3D_CUDA(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch, fast=true, extractor_exp=6)
to
BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)

Maybe that could be tweaked....

[kedautinh12]

Try this script i shared with BM3D_CPU & BM3D_CUDA. After benchmark between them
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)
Prefetch(16) #or try any number can make a good speed fps

[Boulder]

I personally wouldn't put too much effort on the prefiltering part. I use this kind of prefiltering method that was snatched from QTGMC (where it is used for prefiltering as well).

w = clip.Width()
h = clip.Height()
srchClip = clip.RemoveGrain(12,12).GaussResize(w,h, 0,0, w+0.0001,h+0.0001, p=2).MergeLuma(clip, 0.1)

[StainlessS]

Yo Doggy,
does anything in your Extools [or other] whotsits, measure distance of current pixel to nearest eg black[or white] pixel ? [maybe result pixel 0->255 for 8 bit]

Thanks in advance.

EDIT: Or suggestion. I'm not really expecting any +ve reply, but thought I'de ask.
Good Doggy :)

[Dogway]

Maybe you can try this. Binarize first, then expand (to calculate distance) and calculate output value.

ex_binarize(254)
aug = true # to see the mask

aug = aug ? "0.250001 *" : "range_max / "
ex2 = ex_expand(2)

Expr(last,ex_expand(1),ex2,ex2.ex_expand(1),Format("x y z a + + + "+aug),"")

[StainlessS]

Woah, I really was not expecting anything at all, I'll give it a go, thanks very much :)

[Dogway]

Just uploaded SMDegrain v3.2.7d, it includes a new function, ex_retinex(), its use is similar to ex_luma_rebuild() but more robust and a about half the speed although still fast.
I didn't like the implementations in VapourSynth or other sources so I directly ported from a paper and it works great. You can use it with SMDegrain argument "Str" and value 1.0, with value 0 it fallsback to frameprops to check if clip is TV levels and do a range conversion.
Also added a small optimization update to ex_minblur().

I also uploaded DeStripe() although either this or ex_retinex() make use of updates in ExTools that I still haven't uploaded, maybe this evening since I still want to do a quality check on ex_luts(), should be working fine but want to be sure.
I also ported from VapourSynth to ExTools Guided Blur and Anisotropic Guided Filtering, they are both slow. Will think for optimizations at another time.

[Julek]

Just uploaded SMDegrain v3.2.7d, it includes a new function, ex_retinex(), its use is similar to ex_luma_rebuild() but more robust and a about half the speed although still fast.

I get an error with ex_lutxyza, did you forget to add it or did I do something wrong?

[Dogway]

I also uploaded DeStripe() although either this or ex_retinex() make use of updates in ExTools that I still haven't uploaded, maybe this evening since I still want to do a quality check on ex_luts(), should be working fine but want to be sure.

I had a birthday tonight, will upload tomorrow ExTools update after some quality checks. I also think that ex_retinex() should better make use of AutoLevels instead of my runtime levels for better performance, will have a look tomorrow.

[Julek]

Oh, my bad.

[kedautinh12]

It's strange. Your ex_retinex use int sigma but MSRCP use float sigma[...] and your don't have upper_thr

[StainlessS]

Here something for Doggies Birthday[225 KB]:- https://www.mediafire.com/file/4fw6cgrzcds41dp/DogwayBirthday.7z/file

[Dogway]

Here something for Doggies Birthday[225 KB]:- https://www.mediafire.com/file/4fw6cgrzcds41dp/DogwayBirthday.7z/file

Haha thank you, did I misscommunicate? I didn't mean my birthday lol. Thanks anyway :rolleyes:

Just uploaded ExTools update. Didn't do a deep QC for ex_luts() today but should work fine for most things. If something odd arises I will fix on the go. I also had to update GradePack to allow floats in ex_levels() so ex_retinex() works fine.

Please test and if you see any issue let me know to fix it.

@kedautinh12: Just changed sigma to float. It wasn't an issue but I didn't port from anywhere. MSRCP is for RGB (chromaticity preservation), YUV doesn't need to preserve chroma as it is already detached from luma, one problem though is that YUV chroma is not entirely decorrelated, so one option I can add in the future is to preconvert to a better decorrelated color space. For SMDegrain prefiltering though this is not important.

@Julek: Test now with updated ExTools. Also if you let me know what type of Adaptive Threshold you use I can give it another stab. "Adaptive" is a generic family, some use integral other not. In your case it seemed to also use an edge mask. Here is a list of most kind of thresholds I could find.

Adaptive Mean
Adaptive Median
Adaptive Gaussian
Otsu Thresholding
Maximum Entropy Thresholding
Mixture Model Thresholding
Binary Clustering
Metric
Moment-Preserving Thresholding
Inner-class Variance
Pun Thresholding
Niblack Thresholding
Bernsen Thresholding
Abutaleb Thresholding
Sauvola Thresholding
Phansalkar Thresholding
PirahanSiah Adaptive Single thresholding based on PSNR ( www.pirahansiah.com )
Lloyd Thresholding
Ridler-Calvard Thresholding // Iterative Selection Thresholding
Johannsen Entropy Thresholding
Yen, Chang Thresholding
Sahoo, Wilkins, Yeager Thresholding // Renyi's Entropy Thresholding
Triangle Thresholding
Kittler-Illingworth Thresholding // Minimum Error Thresholding
Kapur, Sahoo, Wong Thresholding

[kedautinh12]

Thanks, can i useable ex_retinex(sigma=[50, 200, 350])??

[Dogway]

I haven't implemented custom sigmas, my reference paper did careful measurements to retrieve the optimal sigmas and weighted sums, so it is [1.5%, 5%, global mean] of width dimension and with weights [0.2, 0.1, 0.7]. 'sigma' arg is just a multiplier.
Do you have an example where this doesn't work fine?

[kedautinh12]

Oh, i try use your ex_retinex for port retinex_edgemask to avs+
https://github.com/Irrational-Encoding-Wizardry/kagefunc/blob/ab4f79a4d04a70bc83ef3e37d68c01fb4995a902/kagefunc.py#L217

[Dogway]

I don't have VS installed to compare results, so maybe that MSRCP call is not optimized enough, for example the sigmas there are fixed. I think you might be safe by replacing it entirely with ex_retinex(), if any play with the percentile value in my function to roughly match the dynamic range.

[tormento]

SMDegrain
Which filter do you suggest to lower HBD output from 16/12/10 bit SMDegrain to 8 bit? I am really not satisfied from the internal dithering used by x264 as it sometimes gives really visible banding.

I prefer modern filter ones. :)

[Dogway]

I'm yet to try fmtconv's Void & Cluster dithering, it's an ordered dither so maybe it compresses better than error diffusion, in my opinion it also distributes better so less banding.
Anyway my preference for the last years have been to (denoise), add grain and increase bitrate, I like texture in the images.

[guest]

I would like to ask a real dumb, noobie question about SMDegrain & pre-filter=4, KNLMeansCL, if I may.

How do you change the "strength" settings for KNLMeansCL ???, is it Line ***, or somewhere else ??

Example :-
#After_Prefetch_Denoise
LoadPlugin("%AVISYNTHPLUGINS%\KNLMeansCL\KNLMeansCL.dll")
video=KNLMeansCL(video,d=1, a=2, s=4, h=2, device_type="GPU", device_id=0)

And while I'm at it, does BM3D have optional settings within SMDegrain ??

TIA

[kedautinh12]

Here for document of KNLMeansCL
http://avisynth.nl/index.php/KNLMeansCL

BM3D don't have optional settings in SMDegrain. If you want change settings suitable for you. Can change here
https://github.com/Dogway/Avisynth-Scripts/blob/ec795d2ef5b515fa2e54427c4279541ee50fc438/SMDegrain%20v.3.2.7d/SMDegrain%20v3.2.7d.avsi#L159
Documents of BM3D:
http://avisynth.nl/index.php/BM3DCUDA

[Julek]

@Julek: Test now with updated ExTools. Also if you let me know what type of Adaptive Threshold you use I can give it another stab. "Adaptive" is a generic family, some use integral other not. In your case it seemed to also use an edge mask. Here is a list of most kind of thresholds I could find.

Adaptive Mean
Adaptive Median
Adaptive Gaussian
Otsu Thresholding
Maximum Entropy Thresholding
Mixture Model Thresholding
Binary Clustering
Metric
Moment-Preserving Thresholding
Inner-class Variance
Pun Thresholding
Niblack Thresholding
Bernsen Thresholding
Abutaleb Thresholding
Sauvola Thresholding
Phansalkar Thresholding
PirahanSiah Adaptive Single thresholding based on PSNR ( www.pirahansiah.com )
Lloyd Thresholding
Ridler-Calvard Thresholding // Iterative Selection Thresholding
Johannsen Entropy Thresholding
Yen, Chang Thresholding
Sahoo, Wilkins, Yeager Thresholding // Renyi's Entropy Thresholding
Triangle Thresholding
Kittler-Illingworth Thresholding // Minimum Error Thresholding
Kapur, Sahoo, Wong Thresholding

Here what I use:
https://docs.opencv.org/4.5.1/d7/d4d/tutorial_py_thresholding.html
cv.ADAPTIVE_THRESH_MEAN_C and cv.ADAPTIVE_THRESH_GAUSSIAN_C, using OpenCV + VS is already quite fast, so I don't know if via Expr would be faster.

[guest]

Here for document of KNLMeansCL
http://avisynth.nl/index.php/KNLMeansCL

BM3D don't have optional settings in SMDegrain. If you want change settings suitable for you. Can change here
https://github.com/Dogway/Avisynth-Scripts/blob/ec795d2ef5b515fa2e54427c4279541ee50fc438/SMDegrain%20v.3.2.7d/SMDegrain%20v3.2.7d.avsi#L159
Documents of BM3D:
http://avisynth.nl/index.php/BM3DCUDA

Thanks again, I thought it might be a "line" in SMDegrain somewhere :)

Not sure what's happened, but I am now getting x264 errors, and I can't seem to diagnose them :-

y4m [error]: bad sequence header magic
x264 [error]: could not open input file `-'

[kedautinh12]

Maybe your name of video file contain special characters or characters don't belong to English alphabet. Others your source video (L-SMASH) can't open that video format

[guest]

Maybe your name of video file contain special characters or characters don't belong to English alphabet. Others your source video (L-SMASH) can't open that video format

Yes, I wonder if it is the "name", thanks for the prompt.

Now that you've mentioned L-Smash....

I was going to post a comment on what ppl thought the best decoder was out of FFMS2 or L-Smash ???

What do you think is the "best" (non GPU based) ?

[kedautinh12]

Yes, I wonder if it is the "name", thanks for the prompt.

Now that you've mentioned L-Smash....

I was going to post a comment on what ppl thought the best decoder was out of FFMS2 or L-Smash ???

What do you think is the "best" (non GPU based) ?

Don't have best Video Source plugin, choose video source plugin base your video format

[tormento]

I'm yet to try fmtconv's Void & Cluster
Thanks, I will try it too.

It would be nice to have 2 parameters in SMDegrain where you can decide the internal engine bitdepth and the output one.

[Dogway]

I have always been reticent of adding too many options if they are not well justified, the reason, humans don't deal well when faced with too many options, they stress out and paralyze (FOBO (theurbanengine.com/blog//the-fear-of-better-options)).

FOBO (Fear of Better Options) is a subjective, first-person emotion of indecision, regret and frustration which is associated with being unable to identify a path of action due to an abundance of options.

I think that ConvertBits() is well and nice in its own, visible on the uppermost layer of the script so you can follow the bitdepth conversion flow.

[guest]

Maybe your name of video file contain special characters or characters don't belong to English alphabet. Others your source video (L-SMASH) can't open that video format

Well, I have tried a lot of different things to diagnose this, but it seems to be on only 4 servers that I use for the Distributed Encoding function of RipBot.

So they are simply connected via a LAN, and don't need the complete program to encode (very clever, actually), and yet other pc's on the LAN don't have the problem,

So I think my only recourse is a fresh setup of Windows.

Update:- It turned out that I needed to update these:-VC_redist.x86.exe & VC_redist.x64.exe

There must be some subtle change in Dog's ExTools & SMDegrain.

[tormento]

I think that ConvertBits() is well and nice in its own, visible on the uppermost layer of the script so you can follow the bitdepth conversion flow.
I have tried Void & Cluster and I can't really see lot of improvements over x264 internal Sierra dithering. Perhaps I need to touch knobs that I don't really know, such as pattern size and so.

[Dogway]

Well, there's another option, to use SmoothGrad() it uses Dither Tools, maybe there's an equivalent in fmtconv but haven't checked. In any case I have plans of porting it to ExTools from VS ports in a new pack called yugefunc.avsi

[StainlessS]

@Pauly Dunne [or anyone else]

Here is abbodi1406 vcredist AIO (All In One) version 52, 1st Aug 2021.
[27MB]:- https://github.com/abbodi1406/vcredist/releases

All runtimes from 2005 to current for x86 and x64 combined, all un-necessary stuff stripped out and much smaller than M$ download of all separately.

Good idea to keep the link, updated fairly frequently [about once per month, when M$ makes updates].

[guest]

@Pauly Dunne [or anyone else]

Here is abbodi1406 vcredist AIO (All In One) version 52, 1st Aug 2021.
[27MB]:- https://github.com/abbodi1406/vcredist/releases

All runtimes from 2005 to current for x86 and x64 combined, all un-necessary stuff stripped out and much smaller than M$ download of all separately.

Good idea to keep the link, updated fairly frequently [about once per month, when M$ makes updates].

Nice, StainlessS, can't remember where I found mine, but it's the latest build 14.30.30423.0.

Also need to mention I think it's wise to install both x86 & x64, as I had a fairly recent build of x64, installed, but rather old x86, but after installing, everything works again :)

[real.finder]

Well, there's another option, to use SmoothGrad() it uses Dither Tools, maybe there's an equivalent in fmtconv but haven't checked. In any case I have plans of porting it to ExTools from VS ports in a new pack called yugefunc.avsi

SmoothGrad still missing, along with Dither_box_filter16 and dither_bilateral16 https://forum.doom9.org/showpost.php?p=1943559&postcount=1131

[Dogway]

Dither_box_filter16 is ex_boxblur(mode="mean"), for Dither_bilateral16() we have ex_bilateral() or vsTBilateral(), I still need to port LimitFilter (https://github.com/HomeOfVapourSynthEvolution/mvsfunc/blob/7948c8be129bc9cb282cf24e25b3c4b77328a9e0/mvsfunc.py#L1776)and SmoothGrad (https://github.com/WolframRhodium/muvsfunc/blob/80e99100c78b96b1c925a5989259ef22c1bc6173/muvsfunc.py#L1899).

[kedautinh12]

Here for new ver of mvsfunc. If you want port something from vs. Check here:
https://github.com/AmusementClub/mvsfunc

[real.finder]

Dither_box_filter16 is ex_boxblur(mode="mean"), for Dither_bilateral16() we have ex_bilateral() or vsTBilateral(), I still need to port LimitFilter (https://github.com/HomeOfVapourSynthEvolution/mvsfunc/blob/7948c8be129bc9cb282cf24e25b3c4b77328a9e0/mvsfunc.py#L1776)and SmoothGrad (https://github.com/WolframRhodium/muvsfunc/blob/80e99100c78b96b1c925a5989259ef22c1bc6173/muvsfunc.py#L1899).

ex_bilateral give same output as Dither_bilateral16?

LimitFilter = https://github.com/realfinder/AVS-Stuff/blob/b71a487e3f9619a5a23162215412abcad024aa56/avs%202.5%20and%20up/Zs_RF_Shared.avsi#L476

[real.finder]

Dither_box_filter16 is ex_boxblur(mode="mean")

#ex_boxblur(5,mode="mean")
Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()

but I can see that they are not similar