Dogway's Filters Packs

[LeXXuz]

That is very high. The result would most likely show quite some ghosting/smearing in moving objects. Wouldn't it?

[Dogway]

Originally Posted by tormento

There are really bad movies outta there

Sometimes I had to use thsad=900 up to 1200

Then I learned about double pass denoising.

I always thought that increasing the frame span, besides denoising better, increases its precision. Am I wrong?

Theoretically yes, maybe for true static like anime, but for live action half a pixel is the difference between sharp and blurry, MAnalyse might have a hard time picking half a pixel change more so in noisy clips and the higher the 'tr' the blurrier the output. MAnalyse is lacking rotational and zoom predictors, so the stakes are high.

Double pass is more effective as it performs a binomial weighting to the pixel motion estimation, the furthest frames having less weight.
Also for larger 'tr' all the truemotion and vector penalty settings lose their meaning and turn basically to noop. Read that on a DTL post but can't track it. So for my next Zopti tests I'm going to optimize for tr=1.

[VoodooFX]

What that means: "Resize: height must be greater than 0.", when I add to script "FlatMask()"?

ExTools.avsi line 1793
ExTools.avsi line 1820
SMDegrain v3.5.0d.avsi line 893
MasksPack.avsi line 161

AvS+ 3.7.2, and latest scripts from your github.

PS:
Could it do magic on sample there? -> https://forum.doom9.org/showthread.p...72#post1968672

[Dogway]

Yes thanks, looks like an ex_retinex() issue with small clips (actually ex_gaussianblur() ). I'm having a look, for the time being you can disable that line (161 ln on MasksPack), as it's doesn't seem essential for this task even maybe harmful as it wasn't designed for halos...

[LeXXuz]

I did a few more encodes with SMDegrain in the meantime.

I think there is a slight visible improvement over the old version when it comes to detail retention, movement and the ghosting/smearing problem. Which is very nice.

But the differences in file size are kinda worrying me, because in comparison to the slight visual improvements they are 'huge'.

I already raised thSAD and prefiltering values beyond what I was used to from the old version. And yet there is still quite a big difference in filesize.

To give an example. With the old version and these settings:

LoadPlugin("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\neo-dfttest_rf-fix\64\neo-dfttest.dll") #fixed version by real.finder from 05/29/21
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\neo-f3kdb_r6\clang-x64\neo-f3kdb.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\masktools2-v2.2.26\x64_clang\masktools2.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\mvtools-2.7.44\x64\mvtools2.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\RgTools-v1.1\x64_LLVM_build\RgTools.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\MedianBlur2_v1.1\x64_Clang\MedianBlur2.dll")
#
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\LSFmod.avsi") #v2.193
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\NotSMDegrain.avsi") #v3.1.2.116s mod for v3.1.2d
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\Zs_RF_Shared.avsi") #v1.147
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\mtmodes.avsi")
#
ConvertBits(16)
PRE=(sneo_dfttest(Y=3, U=3, V=3, tbsize=1, slocation="0.0:0 0.5:2 0.9:8 1.0:16", chrslocation="0.0:0 0.5:0.5 0.9:2 1.0:4"))
NotSMDegrain(tr=3, thSAD=200, thSADC=50, contrasharp=100, refinemotion=true, truemotion=false, Str=1.0, pel=1, subpixel=2, prefilter=PRE, chroma=true, plane=4)
neo_f3kdb(range=30, Y=64, Cb=32, Cr=32, grainY=32, grainC=16, sample_mode=4)
Prefetch(12,12)
Return(Last)

I ended up with an HEVC file of 8,04GB.

With the new version and these settings:

LoadPlugin("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\neo-dfttest_rf-fix\64\neo-dfttest.dll") #fixed version by real.finder from 05/29/21
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\neo-f3kdb_r6\clang-x64\neo-f3kdb.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\masktools2-v2.2.26\x64_clang\masktools2.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\mvtools-2.7.44\x64\mvtools2.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\RgTools-v1.1\x64_LLVM_build\RgTools.dll")
LoadPlugin("c:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\MedianBlur2_v1.1\x64_Clang\MedianBlur2.dll")
#
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\GradePack.avsi") #v7.1
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\ResizersPack.avsi") #v9.2
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\SharpenersPack.avsi") #v4.2
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\ExTools.avsi") #v8.4
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\LSFplus.avsi") #v5.3
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\SMDegrain.avsi") #v3.5.0.d
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\Zs_RF_Shared.avsi") #v1.147
Import("C:\Video Editing\MeGUI (x64)\tools\avisynth_plugin\mtmodes.avsi")
#
ConvertBits(16)
PRE=(sneo_dfttest(Y=3, U=3, V=3, tbsize=1, slocation="0.0:0 0.5:8 0.9:16 1.0:32", chrslocation="0.0:0 0.5:4 0.9:8 1.0:16"))
SMDegrain(tr=3, thSAD=400, thSADC=200, contrasharp=20, refinemotion=true, truemotion=false, Str=5.0, pel=1, subpixel=2, prefilter=PRE, chroma=true, plane=4)
neo_f3kdb(range=30, Y=64, Cb=32, Cr=32, grainY=32, grainC=16, sample_mode=4)
Prefetch(12,12)
Return(Last)

I ended up with an HEVC file of 10,86GB.
Encoder settings are exactly the same for both files of course.

That's a 35% gain in file size. I just wonder if I may do something wrong or something is not working right. (Or was not working right before)

[VoodooFX]

Originally Posted by Dogway

disable that line (161 ln on MasksPack)

That made it work. Thanks.

[Dogway]

Originally Posted by LeXXuz

That's a 35% gain in file size. I just wonder if I may do something wrong or something is not working right. (Or was not working right before)

Let me have a deeper look as it seems LSFplus smode=5 had issues with secure implementation. secure=false retrieves what seems LSFmod parity, but this is not good for flat areas where banding might occur. I need some time and concentration so will check the expression when I have time.

EDIT: Looks like it was a simple swap of clip variables. I'd swear I had fixed that before, but anyways check now, updated LSFplus and ExTools.

[LeXXuz]

Originally Posted by Dogway

Let me have a deeper look as it seems LSFplus smode=5 had issues with secure implementation. secure=false retrieves what seems LSFmod parity, but this is not good for flat areas where banding might occur. I need some time and concentration so will check the expression when I have time.

I'm very grateful guys like you share their vast knowledge about digital video with the community. Giving normal 'users' like me tools at hand that even professional studios don't have (judging by the quality of many commercial sources out there...)

So take all the time you need. In the meantime I'll do some double encodes without contrasharp at all to see how the differences between old and new SMDegrain play out.

[Julek]

Originally Posted by Dogway

Finally ported. It acts more like a chroma deblocker so added into DeblockPack. I added a few optimizations and features so you might want to give it a look.

I finally updated my version, I made the matrix size more settable.

[anton_foy]

I try to find ex_occlusion() but I cannot find it, in which pack is it?

[Julek]

Originally Posted by anton_foy

I try to find ex_occlusion() but I cannot find it, in which pack is it?

rebranded as MotionMask()

[anton_foy]

Originally Posted by Julek

rebranded as MotionMask()

Oh I see, thanks!

[LeXXuz]

Originally Posted by Dogway

EDIT: Looks like it was a simple swap of clip variables. I'd swear I had fixed that before, but anyways check now, updated LSFplus and ExTools.

Oops. Totally missed your edit in that post.

Just wanted to tell my findings with the (now obsolete) old version. But I'll do it anyway.

Did a couple of tests with Blu-ray and DVD sources.
- there was not much of a difference between cs=50 and cs=100 in filesize
- cs=5 however resulted in a notably bigger file than cs=100/50 (setting too low or out of range maybe?)
- LSFplus with CS=50 & limitS=false produced filesizes about equal to cs=true setting
- limitS=true produced much bigger files than limitS=false (that really confused me, thought the limiter should reduce sharpen strength? Or is this because of the swap error in the code?)

However, I'll get the new versions and redo those encodes. Let's see how filesizes will behave.

[Dogway]

Originally Posted by LeXXuz

Just wanted to tell my findings with the (now obsolete) old version. But I'll do it anyway.

Yes sorry, retest if you can as the output made no sense. I think I had been working with another 'secure' optimization with that order and got them mixed up.
Now it should behave as expected (I hope). At the time I recall I preferred contrasharp=true rather than LSF, but now I don't know, specially with the limiters.

[LeXXuz]

Originally Posted by Dogway

Yes sorry, retest if you can as the output made no sense. I think I had been working with another 'secure' optimization with that order and got them mixed up.
Now it should behave as expected (I hope). At the time I recall I preferred contrasharp=true rather than LSF, but now I don't know, specially with the limiters.

No worries.

Can't run the new LSFplus version though. Get an error upon avs import. See pic:


My script has not been changed. Just replaced LSFPlus and Extools with newer versions.

[Dogway]

Thanks, you are using EX mod right? sometimes some bug might sneak in there as I don't hammer test them as much as MIX mods. Did a 'mode' remap in ExTools so it should work now. Update also LSFplus EX.

[LeXXuz]

Once again shame on me.
Just realized there are 2 different mods.
Yes, I downloaded the Ex mods.
But if you recommend Mix mods for max. speed I'd better switch to those.

[Dogway]

Yes no prob, just depends what suits you best. EX mods is there for legacy reasons and also max support for those out of Windows.

By the way I was playing a bit and came up with a faster method for corner masks.

removegrain(12)
a=removegrain(17)
removegrain(26)

ex_makediff(a, metric="none", aug=5, dif=true, show=3)

Yes, it's a bit hacky and not as good as CornerMask() but it's much faster.
I cobbled it up into one call for a 18% speed increase:

removegrain(12)
ex_lut("
 x[-1,1] A@ x[1,-1]  H@  dup1 dup1 min swap2 max
 x[0,1]  B@ x[0,-1]  G@  dup1 dup1 min swap2 max
 x[1,1]  C@ x[-1,-1] F@  dup1 dup1 min swap2 max
 x[-1,0] D@ x[1,0]   E@  dup1 dup1 min swap2 max
 swap3                             max swap2 min
 swap3 swap1 swap5                 max swap3 min
                                   min swap2 max
                         dup1 dup1 max swap2 min
 x swap2 clip
 
 
 A B            dup1 dup1 min swap2 max
 B C            dup1 dup1 min swap2 max
 C E            dup1 dup1 min swap2 max
 E H            dup1 dup1 min swap2 max
 swap7 swap1 swap5 max swap1 swap4 swap2 max max LO^
 swap1 swap2 min min min                         HI^
 
 G H            dup1 dup1 min swap2 max
 F G            dup1 dup1 min swap2 max
 D F            dup1 dup1 min swap2 max
 A D            dup1 dup1 min swap2 max
 swap7 swap1 swap5 max swap1 swap4 swap2 max max LO max LO^
 swap1 swap2 min min min                         HI min HI@ LO

 x swap2 min LO HI max clip

- abs 5 *")

# ex_luts(mode="max",pixels=ex_shape(3,mode="disk"),exprf=" ymin 2 * - 5 *  0 max") # (augmenting the dots)

Compared to CornerMask() (36fps) it's about 3 times faster, including the dot augmentation (101fps).

[LeXXuz]

Okay I did a few encodes with the new version. Source was an old movie on DVD in black & white.

This is roughly the filtering I used:

#
AssumeFPS(25)
CTelecine()
CPostProcessing()
crop(4, 0, 0, 0)
ConvertBits(16)
PRE=(sneo_dfttest(Y=3, U=1, V=1, tbsize=1, slocation="0.0:2 0.5:12 0.9:24 1.0:48"))
SMDegrain(tr=3, thSAD=600, thSADC=0, LFR=false, contrasharp=25, limitS=true, refinemotion=true, truemotion=true, DCTFlicker=true, Str=5.0, pel=2, subpixel=2, prefilter=PRE, chroma=false, plane=0)
neo_f3kdb(range=30, Y=64, Cb=0, Cr=0, grainY=32, grainC=0, sample_mode=4)
Greyscale()
Prefetch(16,16)
Return(Last)

Results (old faulty version in brackets):
contrasharp=25, limitS=false: 1.42GB (/)
contrasharp=25, limitS=true: 1.66GB (/)

contrasharp=50, limitS=false: 1.42GB (1.62GB)
contrasharp=50, limitS=true: 1.68GB (2.07GB)

contrasharp=100, limitS=false: 1.41GB (1.61GB)
contrasharp=100, limitS=true: 1.70GB (2.07GB)

contrasharp=true, limitS=false: 1.73GB
contrasharp=true, limitS=true: 1.65GB

Encoding speed with limitS=true was slightly faster than with limitS=false.


Conclusion:
- with limitS=false there is no difference in filesize between contrasharp=25/50/100
- with limitS=true the difference in filesize is quite small
- files with limitS=true are bigger than with false.

==>

If all this is correct I clearly misunderstand the 'limit' in limitS-parameter. I expected it to be the other way round. Maybe you can explain a little.

The value for contrasharp (strength of LSFplus) has little to no effect.

Is this what is to be expected with these settings or is there still something wrong maybe?

[Dogway]

That makes total sense to me, with limitS=false you are using temporal limiting, therefore having more temporal coherence and easier for the codec to compress interframe.

Also I understand you are using limitS=true with contrasharp=true right?