Dogway's Filters Packs

[Boulder]

I've tried testing the deblock pack, but Deblock_QED gives me an error that there is no function named Deblock_deblock. I do have the Deblock plugin installed, and I recall that calling the function like that should definitely work (removing Deblock_ fixes the issue).

[StainlessS]

Is Deblock.dll called exactly that, ie without version number or other mod ?

[Boulder]

Is Deblock.dll called exactly that, ie without version number or other mod ?

Ah, that's it. The 64-bit build is named Deblock-x64.dll so it won't work directly without either renaming or editing the function.

[StainlessS]

Not a good idea to have hyphen in dll file name, advise only "_" or alpha for 1st char, and thereafter digits too, then ".dll".

[Dogway]

I saw you released ExTools 6.0. Can I consider ex_smooth as definitive version to play with? :)

Yes sure. Also I'm the process of updating GradFun3 to ExTools, I will keep it its own script though.


By the way I'm wondering if there's a scaling option in Expr() for evaluating the expression as float instead of input bitdepth. I know the opposite is possible (evaluate as 8-bit).

[real.finder]

By the way I'm wondering if there's a scaling option in Expr() for evaluating the expression as float instead of input bitdepth. I know the opposite is possible (evaluate as 8-bit).

f32 keyword should do it

[Dogway]

Didn't work for me (https://forum.doom9.org/showthread.php?p=1953427#post1953427). I think it's not supported but pinterf has to confirm.


There are two ways currently to do this:

with ConvertBits()
ConvertBits(32)
Expr("x 0.5 - abs","",scale_inputs="none") # 187
ConvertBits(16)

And the annoying way of dividing by range_max if input is int:
Expr("x range_max / 0.5 - abs range_max *","",scale_inputs="none") # 458

[real.finder]

Expr("f32 x 0.5 scalef +","") seems work, but I see no reason for not make it work with scale_inputs, so we will wait pinterf

[tormento]

Call like ex_smooth(1 or 2, sharp=true, limit=true)
I tried to unsharpen Transformers 5, to lower encoding bitrate, but with not much success.

I remember you that this (https://send.cm/d/4ybo) was the clip.

My first attempt was

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\transformers5\transformers5.dgi",ct=64,cb=64,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

with the x264 switches

x264.exe --crf 20 --preset slow --aq-mode 2 --colorprim bt709 --colormatrix bt709 --transfer bt709

and the result have been

encoded 5053 frames, 5.58 fps, 8195.16 kb/s, duration 0:15:05.53

As I told in my previous post, I usually get from 3000 to 4000 kbps with those settings, depending on the type of movie. So more than 8k is just a bit too much and notice that there is almost no grain or noise at all in the original video.

So I have tried to use

Blur(0,1)

just after the SMDegrain call, with same parameters, obtaining

encoded 5053 frames, 5.31 fps, 7412.87 kb/s, duration 0:15:52.00

As last attempt, I have used your new ex_smooth, just after SMDegrain call too

ex_smooth(0,1,sharp=true,limit=true)

obtaining

encoded 5053 frames, 5.16 fps, 7990.14 kb/s, duration 0:16:19.11

So, is there a better way to unsharpen and calm down that rabid movie?

[Dogway]

That was only my first suggestion, why don't you try the other (https://forum.doom9.org/showthread.php?p=1953074#post1953074) ones?
Anyway you have to consider that is an action packed snippet, maybe it's not representative of the whole movie.
For a clearer picture of the final bitrate encode chunks of it with SelectRangeEvery(), for example 2 minutes every 10 minutes or so.

[tormento]

why don't you try the other ones?
I gave the consecutive clip to the community but here I am testing with CompTest.

Where can I find SantiagMod, deconv_h and mt_convolution?

[Dogway]

This is SantiagMod, my go-to antialiaser, I simply replaced GScript with internal syntax:

###
### SantiagMod
###
### Antialiasing function by Archimedes,
### mod from original cretindesalpes's santiag.
###
###
### Requires:
###
### LoadPlugin("plugins\EEDI3\eedi3.dll")
### LoadPlugin("plugins\NNEDI2\nnedi2.dll")
### LoadPlugin("plugins\NNEDI3\nnedi3.dll")
###
### I've changed the posted function santiag a little bit.
###
### Santiag has now three parameters: type, strh and strv. Type means the antialiasing type.
### Possible values are "EEDI3", "NNEDI2" and "NNEDI3". strh is the strength for the horizontal
### anti aliasing and strv is the strength for the vertical anti aliasing. Default values are:
### type="nnedi3", strh=1 and strv=1.
###
### Contrary to the first posted function, santiag(strh=0, strv=0) now means no anti aliasing.
### I've also corected the center shift. With NNEDI3 as type, the center shift now works correct
### for all possible input clips (YV12, YUY2 and RGB24).


function SantiagMod(clip input, string "type", int "strh", int "strv") {
type = Default(type, "NNEDI3")
strh = Default(strh, 1)
strv = Default(strv, strh)

input

strh > 0 ? AntiAliasing1(type=type, strength=strh) : NOP()
TurnLeft()
strv > 0 ? AntiAliasing1(type=type, strength=strv) : NOP()
TurnRight()

function AntiAliasing1(clip input, string "type", int "strength") {
input

if (type == "EEDI3") {
EEDI3(dh=True, field=0)
for (i = 2, strength) {
EEDI3(dh=False, field=(i + 1) % 2)
}
}
else if (type == "NNEDI2") {
NNEDI2(dh=True, field=0)
for (i = 2, strength) {
NNEDI2(dh=False, field=(i + 1) % 2)
}
}
else {
NNEDI3(dh=True, field=0,qual=2)
for (i = 2, strength) {
NNEDI3(qual=2,dh=False, field=(i + 1) % 2)
}
}

Spline36Resize(input.Width(), input.Height(), 0, 0.5, input.Width(), input.Height() * 2)
}
}

mt_convolution() is from masktools2, I was too lazy to port to Expr() but if it works for you I can craft it.


By the way I have to ask if anyone is able to compile this program (https://github.com/bertdobbelaere/SorterHunter) for Windows, I need it to optimize the latest medians from 35 and up. For example "median7" has the potential to increase from current 7.8% to 12%.
median7 (48 inputs) can be composed from two 24 sorting networks and finally a batcher's odd-even merge, but before I spend more days on it I want to test if the program can find an optimized sorting network.

On another note I have been reading about Julia, a new programming (scripting?) language. As fast as C, and as "easy" as Python. I might start to learn it.


EDIT: Also I want to comment that I have been testing GradFun3 and a DFT box filter is unbeatable for large radii, so I might just keep it (if I go for back compatibility). Dither_bilateral16() can't also be replaced, I tested with vsTBilateral() at large radii but something is going on, it appears as it denoises the same as Dither_bilateral16() but when passed dithered down it doesn't deband as much, this might have to do with particular settings like flat or wmin. So rather than a revolution I might do an in-place adaptation for HBD.

[tormento]

This is SantiagMod
I have tried with the other filters.

SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
deconv_h = "0 -5 25 100 25 -5 0 "
mt_convolution (deconv_h, deconv_h, Y=3, U=2, V=2)

encoded 5053 frames, 4.98 fps, 7831.54 kb/s, duration 0:16:54.68

SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
SantiagMod(strh=2,strv=2)
a=last
ex_luts(last,last.removegrain(12,0),mode="clamp",UV=1)
ex_lutxy(last,a,"x y - abs 7 > y x y min ?")

encoded 5053 frames, 4.50 fps, 8190.77 kb/s, duration 0:18:43.92

SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
SantiagMod(strh=2,strv=2)
a=last
ex_luts(last,last.removegrain(12,0),mode="clamp",UV=1)
ex_lutxy(last,a,"x y - abs 7 > y x y min ?")
ex_boxblur(0.15,mode="weighted")

encoded 5053 frames, 4.36 fps, 7953.44 kb/s, duration 0:19:18.57

As you can see, not many changes at all.

Very hard video to compress.

[Dogway]

Put SMDegrain last, and as I commented those 5000 frames are not representative of the movie pace. Use SelectRangeEvery() to get evenly spaced chunks and get a better estimate of final bitrate. Or simply encode with target bitrate 2 passes.

[kedautinh12]

New ver santiag
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Santiag.avsi

[Dogway]

New ver santiag
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Santiag.avsi

Oh my god, who said to keep it simple. :p

[tormento]

I am having a problem with the following script

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_48 Fantozzi\fantozzi.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
Spresso(10,30).STpresso(10,30,4,22,8,49,1,mc=true) #last updates
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

with error

mt_lutxy : Error at position 3: cannot convert to number: 1.000000
(D:/Programmi/Media/AviSynth+/plugins64/STPresso-2020_0507.avsi, line 40)

Could you please convert STPresso too to use your ex_tools?

[Dogway]

I made STTWM() (https://forum.doom9.org/showthread.php?p=1950095#post1950095) 2 months ago. Did you try it?

[kedautinh12]

You don't upload that to github. I think it's only test build

[Dogway]

It was only an example script I crafted for tormento. I don't know to what degree it works or people find it useful so I can't vouch for that.


EDIT: By the way, compiled (https://github.com/Dogway/SorterHunter)SorterHunter just to know it doesn't work with inputs larger than 32, oh well...

[tormento]

I made STTWM
Sorry, it jumped out of my mind. Have I to replace into

Spresso(10,30).STpresso(10,30,4,22,8,49,1,mc=true) #last updates

or use per se?

[Dogway]

No, it's not a port it's an alternative, it uses the same concept, thresholded temporal, thresholded spatial then mix with another threshold. It's way faster. When I tested I liked the results but I normally don't use STPresso anyway. These are impulse noise denoisers (salt'n'pepper, etc) which I don't use much.

I just tested STPresso and worked for me with latest masktools and AVS+

[tormento]

No, it's not a port it's an alternative, it uses the same concept, thresholded temporal, thresholded spatial then mix with another threshold.
I am just looking for an alternative for very grainy and oddly noised video, where SMDegrain simply surrender, even with 9/900.
I just tested STPresso and worked for me with latest masktools and AVS+
Any suggested parameter? The defaults are ok?

[Dogway]

Ah for heavy grain I implemented temporalsoften into SMDegrain, it works great so far as I ported it from TSMC() mod and some bits from LTSMC() from Didée. Yes, there's some low frequency (motion) blur like, I'm working on it.

I tested STPresso with default values, maybe it might be something on your end. What line/call is giving you issues?

[tormento]

Ah for heavy grain I implemented temporalsoften into SMDegrain
Please explain me how to use it, as I can't see in the documentation.
I tested STPresso with default values, maybe it might be something on your end. What line/call is giving you issues?
I saw a new release of MaskTools is out, allowing TAB, CR and LF characters as symbol separators in expression strings.

Will give it a try.

[Dogway]

Please explain me how to use it, as I can't see in the documentation.

I explained it to you in this post (https://forum.doom9.org/showthread.php?p=1953018#post1953018) 2 weeks back.

[tormento]

I explained it to you
Oops.

I saw you mentioned ex_retinex too.

I am trying to recover some color cast, probably from film degradation, as you can see in the Retinex thread.

So, is it possible to integrate my separate line for color restoration inside SMDegrain? As you can see there, I am having some problems with too much red restoration.

[Dogway]

My version of retinex is not for color balance but for contrast enhancement.
I think there are better tools to remove the color cast like Gammac or RGBadapt. See here (https://forum.doom9.org/showthread.php?p=1632812#post1632812).

Also check here (https://youtu.be/irbZewfz-SA?t=544)from videoFred

[tormento]

My version of retinex is not for color balance but for contrast enhancement.
Thanks for your links, I will read them.

The issue with that film is that every component aged in a different manner and it's really hard to balance the hues of each one.

[guest]

What is the actual function on the "pre-filters" in SMDegrain ? eg:- Dfttest, KNLMeansCL, & BM3D.

Are they to compliment the main SMDegrain function ?, 'cause SMDegrain works pretty well, by itself !

And can the "strengths" be easily changed ?

[kedautinh12]

What is the actual function on the "pre-filters" in SMDegrain ? eg:- Dfttest, KNLMeansCL, & BM3D.

Are they to compliment the main SMDegrain function ?, 'cause SMDegrain works pretty well, by itself !

And can the "strengths" be easily changed ?

I seen default prefilter=-1 so SMDegrain works by itself
https://github.com/Dogway/Avisynth-Scripts/blob/d8634fcf489d6eaf7dee179d15dd6cb7f39240b4/SMDegrain%20v3.3.2d/SMDegrain%20v3.3.2d.avsi#L164

[guest]

I seen default prefilter=-1 so SMDegrain works by itself
https://github.com/Dogway/Avisynth-Scripts/blob/d8634fcf489d6eaf7dee179d15dd6cb7f39240b4/SMDegrain%20v3.3.2d/SMDegrain%20v3.3.2d.avsi#L164

Hi kedautinh12,

That really doesn't answer my question, though.

What is the point or the benefit of using those pre-filters ?

How to use BM3D by itself, for example.

[kedautinh12]

Hi kedautinh12,

That really doesn't answer my question, though.

What is the point or the benefit of using those pre-filters ?

How to use BM3D by itself, for example.

I seen when you prefilter some case result better in details video

You use SMDegrain(prefilter=5) to work with BM3D. You can try SMDegrain(prefilter=4) to work with KNLMeansCL, that make beter result too

[guest]

I seen when you prefilter some case result better in details video

You use SMDegrain(prefilter=5) to work with BM3D. You can try SMDegrain(prefilter=4) to work with KNLMeansCL, that make beter result too

I have found that BM3D (either CPU or CUDA) is VERY slow encoding 4K footage.

Can you change the default settings in the pre-filters ??

When I update to latest SMDegrain 3.3.2, I get errors, can only use 3.3.0.

[kedautinh12]

I have found that BM3D (either CPU or CUDA) is VERY slow encoding 4K footage.

Can you change the default settings in the pre-filters ??

When I update to latest SMDegrain 3.3.2, I get errors, can only use 3.3.0.

That cost when you use BM3D but result very well and KNLMeansCL too. You need updated all Dogway's scripts to make SMDegrain work: LSFmod, ExTools, SharpenersPack. Others like Utils-r41 and Zs_RF_Shared
https://github.com/raffriff42/AvisynthPlusUtilities/blob/master/Utils-r41.avsi
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Zs_RF_Shared.avsi

[guest]

That cost when you use BM3D but result very well and KNLMeansCL too. You need updated all Dogway's scripts to make SMDegrain work: LSFmod, ExTools, SharpenersPack. Others like Utils-r41 and Zs_RF_Shared
https://github.com/raffriff42/AvisynthPlusUtilities/blob/master/Utils-r41.avsi
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Zs_RF_Shared.avsi

It's hard to tell which one out of BM3D & KNLMeansCL does the better job..

The problem with BM3D is it is rather limited on hardware & software requirements, where as KNL is a little more versatile.

I'm pretty sure I have ALL of Dogway's latest scripts, but there are so many, and some variant's, it's hard to know which one's go with which one's.

And the other problem is there are other dependencies required, as well...depending what filters & pre-filters you choose :(

[kedautinh12]

I think it's dependencies required
https://forum.doom9.org/showthread.php?p=1953024#post1953024

[Dogway]

What is the actual function on the "pre-filters" in SMDegrain ? eg:- Dfttest, KNLMeansCL, & BM3D.

Are they to compliment the main SMDegrain function ?, 'cause SMDegrain works pretty well, by itself !

And can the "strengths" be easily changed ?
It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

[tormento]

smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=900,contrasharp=false,refinemotion=true)
smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=300,mfilter=ex_blur(1),contrasharp=true,refinemotion=true)
Are those supposed to run alternatively or consecutively?

[Dogway]

Consecutively. I reworked the mfilter logic for modes other than MDegrain, I will release it later today.

[guest]

It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

So when you say:- "You can define your own prefilter in the "prefilter" argument", do you mean in these lines, or somewhere else ?.

(prefilter== 4) ? inputP.ex_KNLMeansCL(a=1,d=1,h=7.0,chroma=chroma,device_type="GPU",device_id=device_id) : \ how come there's no s= option ?
(prefilter== 5) ? inputP.ex_BM3D(sigma=5,radius=1,CUDA=true,UV=Chr,fulls=fs)

I am running a 5950X with a GTX 1080, and I just find that 4K is a bit of a chore, I know it's 4x "bigger than 1080p...but :(

I also know that BM3D can be used in CPU or CUDA, BUT only on AVX2 CPU's :(.

I will get the new SMDegrain, and check all the other dependencies, update my scripts, and see if I have any problems.

Thanks

[videoh]

Consecutively. Of course. It's like deinterlacing twice because the first time wasn't good enough. :p

[guest]

It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

I compiled a reply to this post, and when I went to submit it, some "warning" flashed up about moderator clearance, and it disappeared :(

If it doesn't show up in the next day or so, I will try and remember what I wrote, and try again.

And yet, this one worked !!! :confused:

[MysteryX]

I seen when you prefilter some case result better in details video

You use SMDegrain(prefilter=5) to work with BM3D. You can try SMDegrain(prefilter=4) to work with KNLMeansCL, that make beter result too

I've done extensive testing of BM3D, KNLMeansCL and MVTools with prefilters while working on xClean (https://forum.doom9.org/showthread.php?t=183245).

What I found is that each of the 3 behave differently when it comes to prefilters. Here's a summary of the strengths and weaknesses of each, and the effect of prefilter.

MVTools
Strength: Removes a lot of noise, good at removing temporal noise.
Weakness: Can remove too much, especially with delicate textures like water.
Ref: Impacts vectors analysis but low impact on outcome

BM3D
Strength: Good at preserving fine details!
Weakness: Doesn't remove much grain. Poor temporal stability.
Ref: Moderate impact on outcome. A blurry ref will remove more grain while BM3D puts back a lot of details.
radius=1 provides nearly no benefit at huge performance cost since MVTools already does temporal analysis

KNLMeansCL
Strength: Best general-purpose denoiser
Weakness: Can blur out details and give an artificial plastic effect
Ref: Highly impacts the outcome. All prefilters benefit from running KNLMeans over it.

MVTools + BM3D
Strength: Keeps a lot of details with good denoising of fine details as well as larger grain. Single frames can look great.
Weakness: Poor noise temporal stability. The video doesn't look as good as single frames.

MVTools + KNLMeans
Strength: KNLMeans with extra denoising. Works best in most circumstances.
Weakness: Delicate textures like water or fog will suffer.

MVTools + BM3D + KNLMeans
Strength: Like MvTools+KNLMeans but preserves details with delicate textures. Works best for any kind of content tested.
Weakness: Performance and memory usage.

[guest]

It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

So despite getting all the latest updates, and updating my scripts, I still can't get them all to work happily together.

Also, have to say a big thanks for listing ALL the dependencies, for SMDegrain :)

[kedautinh12]

So despite getting all the latest updates, and updating my scripts, I still can't get them all to work happily together.

Also, have to say a big thanks for listing ALL the dependencies, for SMDegrain :)

You need say exactly your problem

[guest]

You need say exactly your problem

I need to have a bit of a cleanup...I've got so many different script builds, might need to start from scratch.

But I am a little puzzled that it all worked several builds back, but then things get shuffled around, and it produces errors.

[tormento]

MVTools + BM3D + KNLMeans
Strength: Like MvTools+KNLMeans but preserves details with delicate textures. Works best for any kind of content tested.
Weakness: Performance and memory usage.
How do you combine and apply them? Can post a sample script?

[Dogway]

I've done extensive testing of BM3D, KNLMeansCL and MVTools with prefilters while working on xClean (https://forum.doom9.org/showthread.php?t=183245).

I found BM3D to be a bit soft on edges, I think because it does some spatial filtering (3D). For video I normally avoid denoising directly in spatial domain unless it's something very specific or masked. For guiding I think it's good, in my brief comparisons motion estimation was better than KNLMeansCL, but it usually hanged when trying higher radius.

@TDS: Post the error message so I can have a look, otherwise can't help. If you want a cleanup tip, just keep core filters and move out old ones or used sparsely, then try to update them to latest version, including AviSynth.

[StainlessS]

I compiled a reply to this post, and when I went to submit it, some "warning" flashed up about moderator clearance, and it disappeared :(

If it doesn't show up in the next day or so, I will try and remember what I wrote, and try again.




TDS, you need this [in the 1st post, TextArea Cache ].:- https://forum.doom9.org/showthread.php?t=177066


First time I've had to use it and saved me about 20->30 minutes of repeated searching to re-write post after accidental deletion.

I had several different threads open on D9, and accidentally went back to previous page on the wrong tab, result: about 20 to 30 minutes of searching/writing lost.
I clicked on the TextArea Cache tool icon, and it was there just like it was before deletion, this is defo gonna be staying on all my machines, it was there just when I needed it.

It saves copy of text edit boxes of fireFox. [you can give list of websites not to cache eg your on-line banking]
Saved my bacon, perhaps do same for you one day.