Dogway's Filters Packs

[pinterf]

Originally Posted by Dogway

With the new mvtools2 version DCT is now much faster, so I defaulted DCT to 5, if everyone agrees.

"dct = 5: SATD instead of SAD for luma."
So for this mode you noticed a speed increase? There was a fix indeed, but I couldn't tell the magnitude of this change, did not perform a before-after test.

[tormento]

Originally Posted by Dogway

only 8-bit is used for motion vectors, MDegrain uses a MSuper of original bitdepth, and everything else is processed in original bitdepth.

Even if I feed a HBD source? So some dithering/truncating occurs?

[Dogway]

SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted. Also optimized ex_luma_rebuild() and some overall cleanup. In the future I want to add some frequency restoration system, this is something that I also looked into 6 years ago.

OLD DCT = 0   7.600fps
OLD DCT = 5   6.300fps
NEW DCT = 5   6.300fps
NEW DCT = 0   7.700fps

@Tormento, yes, the performance/gain ratio is very bad for HBD motion search. ex_luma_rebuild() makes up for an extra 16% precision by converting to PC levels which is very fast with the latest update. You can run tests to see the difference, check below example.

@pinterf, I was confused about bitdepth handling with motion vectors, is the next valid code or do I need MScaleVect()?

super_search = MSuper(rfilter=2)

bv3 = super_search.MAnalyse(isb = true,  delta = 3, overlap= 4)
bv2 = super_search.MAnalyse(isb = true,  delta = 2, overlap= 4)
bv1 = super_search.MAnalyse(isb = true,  delta = 1, overlap= 4)
fv1 = super_search.MAnalyse(isb = false, delta = 1, overlap= 4)
fv2 = super_search.MAnalyse(isb = false, delta = 2, overlap= 4)
fv3 = super_search.MAnalyse(isb = false, delta = 3, overlap= 4)

convertbits(16)
MDegrain3(MSuper(levels=1), bv1, fv1, bv2, fv2 ,bv3, fv3, thSAD=300, thSADC=150)

[pinterf]

Originally Posted by Dogway

SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted.

dct>0 can be quicker indeed, except for 8x8 block sizes at 8 bit videos. Other than that special format (which is computed by integer dct code) the others are using fftw3, which remained unchanged but the epilog code got a fix in the SSE2 code.

[pinterf]

Originally Posted by Dogway

SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted. Also optimized ex_luma_rebuild() and some overall cleanup. In the future I want to add some frequency restoration system, this is something that I also looked into 6 years ago.

OLD DCT = 0   7.600fps
OLD DCT = 5   6.300fps
NEW DCT = 5   6.300fps
NEW DCT = 0   7.700fps

@Tormento, yes, the performance/gain ratio is very bad for HBD motion search. ex_luma_rebuild() makes up for an extra 16% precision by converting to PC levels which is very fast with the latest update. You can run tests to see the difference, check below example.

@pinterf, I was confused about bitdepth handling with motion vectors, is the next valid code or do I need MScaleVect()?

super_search = MSuper(rfilter=2)

bv3 = super_search.MAnalyse(isb = true,  delta = 3, overlap= 4)
bv2 = super_search.MAnalyse(isb = true,  delta = 2, overlap= 4)
bv1 = super_search.MAnalyse(isb = true,  delta = 1, overlap= 4)
fv1 = super_search.MAnalyse(isb = false, delta = 1, overlap= 4)
fv2 = super_search.MAnalyse(isb = false, delta = 2, overlap= 4)
fv3 = super_search.MAnalyse(isb = false, delta = 3, overlap= 4)

convertbits(16)
MDegrain3(MSuper(levels=1), bv1, fv1, bv2, fv2 ,bv3, fv3, thSAD=300, thSADC=150)

The code is valid.
The super clip provided to MDegrain must be originated from the same format as the video to denoise.
It can even be a 4:4:4 32 bit float as well, while the vectors can be computed from a plain 8 bit YV12.

I still do not understand what is luma_rebuild for.
"makes up for an extra 16% precision". If this extra range would help with anything then using a 16 bit clip for motion vector search would yield an extra 25600% precision increase. In real life: when you compare the vectors from a 8 bit clip or its ConvertBits(16) counterpart, there will be basically no difference.
What I can imagine that you stretch the Y range and thus it will have a relatively larger weight when chroma is involved.
Btw. you can adjust luma/chroma weights in SAD search with MAnalyze "scaleCSAD" parameter which fine tunes chroma SAD weight relative to luma SAD.

[Dogway]

Bitdepth has more precision, but is not feature bound. Take for example the "opposite" of a TV->PC conversion, a source in log space (or a very low contrast processed source), the SAD will be misleading even in float precision because it doesn't represent human vision.

luma_rebuild() also has a brightening option for the dark areas for the same reason. Human vision is very adaptive to dark (more so in dim/dark surround) so it makes sense to brighten this up to give it more SAD weight and avoid smearing. If this wasn't true we would just try to find motion vectors in linear light and not gamma encoded sources.

I'm using scaleCSAD (MAnalyze), but only for chroma subsampling as written in the docs. Does it have an effect on thSADC of MDegrain? Meaning are they mutually exclusive?

[pinterf]

I see. So it is for lifting the shadows. thsadc can be used independendly from scaleCSAD

[tormento]

Originally Posted by Dogway

yes, the performance/gain ratio is very bad for HBD motion search

I am a bit confused now about the differences between the parameters in your SMDegrain and real.finder's one.

I am looking at your html file, that perhaps you have not updated yet looking at the changelog, and I can see the lsb_in and lsb_out parameters. Weren't they the parameters used when AVS did not support HBD?

Let's have some example:

1. I have a 8 bit input, I want to internally work in 8 bit and output 8 bit
2. I have a 8 bit input, I want to internally work in 16/32 bit and output 8/10/12/16 bit
3. I have a 10/12/16 bit input, I want to internally work in 10/16/32 bit and output 8/10/12/16 bit

Could you please write the correct parameters to be used for each case?

Thanks in advance.

[Dogway]

The documentation hasn't been updated, not even real.finder updated it so it is very old by current terms. I will update when I finish Transforms Pack which will be very soon.

My updated SMDegrain tries to be as simple as possible so I removed redundant arguments like HBD in/out and things like that.
For example 2 precede it with ConvertBits(bits), others need nothing.

[Boulder]

Originally Posted by Dogway

luma_rebuild() also has a brightening option for the dark areas for the same reason. Human vision is very adaptive to dark (more so in dim/dark surround) so it makes sense to brighten this up to give it more SAD weight and avoid smearing. If this wasn't true we would just try to find motion vectors in linear light and not gamma encoded sources.

I've always wondered a bit about luma_rebuild. Using s0=1.0 makes the image generally much darker compared to the original clip than cretindesalpes's original version which has s0=2.0. I thought it would make the clip have less dark areas.

[Dogway]

I don't remember what was the default before my recent edits but I usually try to leave defaults in "safe" settings otherwise people may complain too much, but if there's a consensus I can set it to 2.0, it's still safe (I've gone to 6.0 on a recent project). s0=1.0 by default just converts to PC levels so it appears darker when seen on TV levels preview (ie. AvsPmod)

[tormento]

Originally Posted by Dogway

ExTools

I just updated to 1.7 version and I get:

Script error: Invalid arguments to function 'ex_Yexpr'.
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.0d~Dogway.avsi, line 618)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.0d~Dogway.avsi, line 161)
(F:\In\1_30 Febbre da cavallo\febbre_4.avs, line 20)
(F:\In\1_30 Febbre da cavallo\febbre_4_temp\febbre_4_source.avs, line 1)

My script is simple:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

1.6 is working fine, SMDegrain is the last one.

[Dogway]

Yes, ExTools is a bit developed ahead as I'm making time to solve this issue and rebase the other filters.

For the moment use v1.6. I think I will doing other things until the issue which is critical is solved.

[tormento]

Originally Posted by Dogway

For the moment use v1.6. I think I will doing other things until the issue which is critical is solved.

Do you plan to abandon dependencies on Zs_RF_Shared?

[kedautinh12]

Originally Posted by tormento

Do you plan to abandon dependencies on Zs_RF_Shared?

SMDegrain of Dogway don't dependencies on Zs_RF_Shared

[tormento]

Originally Posted by kedautinh12

SMDegrain of Dogway don't dependencies on Zs_RF_Shared

Strange. If I recall well, I tried on one of the first beta and it needed.

I will try again with latest one.

[real.finder]

it's still needed for https://github.com/Dogway/Avisynth-S....2.0d.avsi#L39

edit: and also https://github.com/Dogway/Avisynth-S...2.0d.avsi#L145

also there are bug that case error in this line https://github.com/Dogway/Avisynth-S...2.0d.avsi#L170 with subpixel=3 that fixed in https://github.com/Dogway/Avisynth-Scripts/pull/11

[Dogway]

Originally Posted by real.finder

also there are bug that case error in this line https://github.com/Dogway/Avisynth-S...2.0d.avsi#L170 with subpixel=3 that fixed in https://github.com/Dogway/Avisynth-Scripts/pull/11

You mean ex_luma_rebuild()? Expr() has issues dealing with clip luma range as explained in this post. So development is pretty much halted until solved.

[real.finder]

Originally Posted by Dogway

You mean ex_luma_rebuild()?

it's another one with subpixel=3

[tormento]

Tried pure speed comparison between SMDegrain and NotSMDegrain on an old movie. Latest versions of AviSynth+, AVSI and dll filters.

Script for SMDegrain

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

Script for NotSMDegrain

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
NotSMDegrain (tr=6, thSAD=600, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

AVSMeter 3.0.8.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.1 (r3413, master, x86_64) (3.7.1.0)

Number of frames: 2712
Length (hh:mm:ss.ms): 00:01:48.480
Frame width: 1920
Frame height: 1032
Framerate: 25.000 (25/1)
Colorspace: YUV420P16

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.918 | 959.8 | 5.764
Process memory usage (max): 2162 MiB
Thread count: 75
CPU usage (average): 72.4%

Time (elapsed): 00:07:50.498

NotSMDegrain (tr=6, thSAD=600, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.956 | 125000 | 6.124
Process memory usage (max): 1961 MiB
Thread count: 76
CPU usage (average): 73.0%

Time (elapsed): 00:07:22.854

SMDegrain (tr=7, thSAD=700, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.788 | 1055 | 4.498
Process memory usage (max): 2020 MiB
Thread count: 75
CPU usage (average): 73.2%

NotSMDegrain (tr=7, thSAD=700, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.477 | 243903 | 5.029
Process memory usage (max): 1783 MiB
Thread count: 74
CPU usage (average): 72.5%

SMDegrain (tr=9, thSAD=900, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.683 | 714286 | 3.412
Process memory usage (max): 2278 MiB
Thread count: 75
CPU usage (average): 73.1%

NotSMDegrain (tr=9, thSAD=900, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.835 | 625000 | 3.909
Process memory usage (max): 1964 MiB
Thread count: 75
CPU usage (average): 72.7%