porting LSFmod for HBD in avs+

[real.finder]

edit: all old talk that was here moved to the 2nd post now

the porting is done https://github.com/realfinder/AVS-Stuff/raw/master/avs%202.5%20and%20up/LSFmod.avsi

[real.finder]

days ago start port LSFmod, I didn't care for it before cuz I don't use sharp most of times but since it's used by SMDegrain so I plan for port it

but there are some lut lines that can't make it give same result in another bit depth, but in 32 bit with clamp_f_i8 it's ok

https://pastebin.com/NnjaqCa8

the problem when use smode 4 and 5


\ : Smode==4 ? VersionNumber() < 2.6 ? mt_lutxy(tmp, method, yexpr="x y == x x x y - abs "+string(Szrp)+" / 1 "+string(Spwr)+" / ^ "+string(Szrp)+" * "+string(str)+" * x y - 2 ^ x y - 2 ^ "
\ +string(SdmpLo)+" + / * x y - x y - abs / * 1 "+string(SdmpHi)+" 0 == 0 x y - abs "+string(SdmpHi)+" / 4 ^ ? + / + ?",U=1,V=1) \
: mt_lutxy(tmp, method, yexpr="clamp_f_i8 x y == x x x y - abs "+string(Szrp)+" / 1 "+string(Spwr)+" / ^ "+string(Szrp)+" * "+string(str)+" * x y - 2 ^ x y - 2 ^ "
\ +string(SdmpLo)+" scalef + / * x y - x y - abs / * 1 "+string(SdmpHi)+" 0 == 0 x y - abs "+string(SdmpHi)+" / 4 ^ ? + / + ?",U=1,V=1)
\ : VersionNumber() < 2.6 ? mt_lutxy(tmp, method, yexpr="x y == x x x y - abs "+string(Szrp)+" / 1 "+string(Spwr)+" / ^ "+string(Szrp)+" * "+string(str)+" * x y - x y - abs / * x y - 2 ^ "
\ +string(Szrp)+" 2 ^ "+string(SdmpLo)+" + * x y - 2 ^ "+string(SdmpLo)+" + "+string(Szrp)+" 2 ^ * / * 1 "
\ +string(SdmpHi)+" 0 == 0 "+string(Szrp)+" "+string(SdmpHi)+" / 4 ^ ? + 1 "+string(SdmpHi)+" 0 == 0 x y - abs "+string(SdmpHi)+" / 4 ^ ? + / * + ?",U=1,V=1) \
: mt_lutxy(tmp, method, yexpr="clamp_f_i8 x y == x x x y - abs "+string(Szrp)+" / 1 "+string(Spwr)+" / ^ "+string(Szrp)+" * "+string(str)+" * x y - x y - abs / * x y - 2 ^ "
\ +string(Szrp)+" 2 ^ "+string(SdmpLo)+" scalef + * x y - 2 ^ "+string(SdmpLo)+" scalef + "+string(Szrp)+" 2 ^ * / * 1 scalef "
\ +string(SdmpHi)+" 0 == 0 "+string(Szrp)+" scalef "+string(SdmpHi)+" scalef / 4 ^ ? + 1 scalef "+string(SdmpHi)+" scalef 0 == 0 x y - abs "+string(SdmpHi)+" scalef / 4 ^ ? + / * + ?",U=1,V=1)


some test code for lut only (smode=4), Formula in non RPN https://s11.postimg.org/ej1ht7bo3/Untitled.png


#video source here
convertbits(16)
tmp=last
method=Blur(1)

szrp=16
spwr=4
str=100/100.0
sdmplo=4
sdmphi=48

mt_lutxy(tmp, method, yexpr="clamp_f_i8 x y == x x x y - abs "+string(Szrp)+" / 1 "+string(Spwr)+" / ^ "+string(Szrp)+" * "+string(str)+" * x y - 2 ^ x y - 2 ^ "
\ +string(SdmpLo)+" scalef + / * x y - x y - abs / * 1 "+string(SdmpHi)+" 0 == 0 x y - abs "+string(SdmpHi)+" / 4 ^ ? + / + ?",U=1,V=1)
convertbits(8)


maybe we need pinterf to make an option to scale the inputs (x, y, etc...) in non floats sources too (like scale_input=true, and it's better to make it true by default so any old script that no one port yet may work in HBD without any edit), and keep only clamp_f out of another clamp_f_XX

edit: scale_inputs="int" or "float" or "both" (default) or "none" will be better than bool

I forgot to mention that there are other changes, like fix https://forum.doom9.org/showthread.php?p=1807637#post1807637

and remove yv12 limit, and some speed optimization in avs2.6 and avs+ (thanks to Y8/Y)

[FranceBB]

Thanks! :)
Any chance in getting it to work in 16bit with Dither Tool in Avisynth 2.6 (not just AVS+)?

[pinterf]

days ago start port LSFmod, I didn't care for it before cuz I don't use sharp most of times but since it's used by SMDegrain so I plane for port it

but there are some lut lines that can't make it give same result in another bit depth, but in 32 bit with clamp_f_i8 it's ok

https://pastebin.com/NnjaqCa8

some test code for lut only (smode=4), Formula in non RPN https://s11.postimg.org/ej1ht7bo3/Untitled.png


#video source here
convertbits(16)
tmp=last
method=Blur(1)

szrp=16
spwr=4
str=100/100.0
sdmplo=4
sdmphi=48

mt_lutxy(tmp, method, yexpr="clamp_f_i8 x y == x x x y - abs "+string(Szrp)+" / 1 "+string(Spwr)+" / ^ "+string(Szrp)+" * "+string(str)+" * x y - 2 ^ x y - 2 ^ "
\+string(SdmpLo)+" scalef + / * x y - x y - abs / * 1 "+string(SdmpHi)+" 0 == 0 x y - abs "+string(SdmpHi)+" / 4 ^ ? + / + ?",U=1,V=1)
convertbits(8)


maybe we need pinterf to make an option to scale the inputs (x, y, etc...) in non floats sources too (like scale_input=true, and it's better to make it true by default so any old script that no one port yet may work in HBD without any edit), and keep only clamp_f out of another clamp_f_XX

edit: scale_inputs="int" or "float" or "both" (default) or "none" will be better than bool

Finally got time and had a look at the formula itself.

szrp, sdmphi is in the 8-bit (256) magnitude
sdmplo appears in a square context so it should be in the 256^2 magnitude

Thus for 10-16 bits variables szrp and sdmphi must be scaled by 4* (10 bits) or 256* (16 bits).
Or they have to be divided by 255.0 for 32 bit float.

Same rule applies on sdmplo, but the scaling is 4*4 (10 bits) and 256*256 (16 bits). And for 32 bit float we should divide its original value by (255.0*255.0).

So this expression won’t work properly even for 10-16 bits.

Good news (obviously) that there is no need of extra pre-scaling magic for float, we cannot divide apples with oranges. The constants are for 8 bits data, they have to be scaled accordingly. The scaling can also be done with scaleb inside the expression string.

This solution gave me nearly identical results for all bit depths.

(Script contains tests for the new avs+ filter Expr, ignore it at the moment)

(source filter)
Spline64Resize(480,240) #resize, result is a multistacked image
src=last

# expr
c8 = CalcTest(src,8, False)
c16 = CalcTest(src,16, False)
c32 = CalcTest(src,32, False)

# lutxy
c8e = CalcTest(src,8, True)
c16e = CalcTest(src,16, True)
c32e = CalcTest(src,32, True)

res8=Diff(c8,src)
res16=Diff(c16,src)
res32=Diff(c32,src)

res8e=Diff(c8e,src)
res16e=Diff(c16e,src)
res32e=Diff(c32e,src)

col1=StackVertical(c8,c16.convertbits(8),c32.convertbits(8))
col2=StackVertical(res8, res16, res32)
col3=StackVertical(c8e,c16e.convertbits(8),c32e.convertbits(8))
col4=StackVertical(res8e, res16e, res32e)
StackHorizontal(col1, col2, col3, col4)

Function Diff(clip src1, clip src2)
{
return Subtract(src1.ConvertBits(8),src2.ConvertBits(8)).Levels(120, 1, 255-120, 0, 255, coring=false)
}

Function CalcTest(clip src, int bits, bool lut)
{
src
convertbits(bits)
tmp=last
method=Blur(1)

szrp=16
spwr=4
str=100/100.0
sdmplo=4
sdmphi=48
expr = "x y == x x x y - abs "+string(Szrp) +" scaleb / 1 "+string(Spwr)+" / ^ "+string(Szrp) +" scaleb * "+string(str)+" * x y - 2 ^ x y - 2 ^ "
\+string(SdmpLo)+" scaleb scaleb + / * x y - x y - abs / * 1 "+string(SdmpHi)+" scaleb 0 == 0 x y - abs "+string(SdmpHi)+" scaleb / 4 ^ ? + / + ?"
ret=lut ? mt_lutxy(tmp,method, yexpr=expr, U=1,V=1 ) : Expr(tmp,method,expr,"","")
return ret
}

[real.finder]

seems ok. good scale, but what about the other expr? you only test with smode 4 there are another one and more complex (smode 5)

well even if your scale is better than using any "extra pre-scaling magic" and even faster (theoretically), I don't think you can do same thing with other expr's especially for float

and like I said here (https://github.com/pinterf/masktools/issues/1#issuecomment-340470392) "extra pre-scaling magic" it's better than edit hundreds of scripts just for high bit :) you can make it auto (https://github.com/pinterf/masktools/issues/1#issuecomment-337638835) and none by default in new Expr() since there are no old scripts made with it

and thank you for your time for look at this issue, and for developing avs things

[pinterf]

Now I had only time for this, smode 5 follows. At least it turned out that expr used pow for ^ and = for ==. So I made available the mt_lut syntax in Expr to avoid using different expression strings.

[real.finder]

finally! here is the last "working" one (need last masktools and avs+ (https://forum.doom9.org/showthread.php?p=1844527#post1844527))

https://pastebin.com/uG1jcPKt

some expr still use lut since there are some differences in output and make image darker or brighter, don't know if it bug in avs+ expr or something else, let see what pinterf said

even if I test smode from 3 to 5 in 16 and 32 bit, other options need to test by any volunteer

[pinterf]

Investigating.

EDIT:
Problem fixed in avs+ r2721 test. See avs+ topic

Evaluating the expression with Expr is a must have.
Checked the speeds for a 16 bit clip: for smode=3 the speed factor is 4x. For smode=4 the speed gain is even higher: 10x (on i7-7700, it has AVX2, Expr benefits from it)

[real.finder]

the last edit now, I think you can use it for real encode now! https://pastebin.com/J5CRtqjj (not the last one now, see the post below)


Evaluating the expression with Expr is a must have.
Checked the speeds for a 16 bit clip: for smode=3 the speed factor is 4x. For smode=4 the speed gain is even higher: 10x (on i7-7700, it has AVX2, Expr benefits from it)

yes it's very fast now in HBD, maybe in the future all new expr's in new functions will not use lut anymore especially if avs+ continuously evolving

[real.finder]

fix a bug that happen in the last edit and high optimization ver.

test done using 2012 pentium laptop cpu (don't has avx) and the tests done with every lut line separately

stable new LSFmod https://pastebin.com/pzACBAX1

another scripts updates will come soon

[FranceBB]

fix a bug that happen in the last edit and high optimization ver.

test done using 2012 pentium laptop cpu (don't has avx) and the tests done with every lut line separately

stable new lsfmod https://pastebin.com/WVmCWifm

another scripts updates will come soon

Thank you for your efforts. Is there any change in the code for 8bit processing or can I stick with the old one if I have Avisynth 2.6.1?

Last but not least, I'm gonna quote myself, asking the same question I asked a year ago:

Thanks! :)
Any chance in getting it to work in 16bit stacked or interleaved (HDR Core/Dither Tools) in Avisynth 2.6 (not just AVS+)?

[real.finder]

Thank you for your efforts. Is there any change in the code for 8bit processing or can I stick with the old one if I have Avisynth 2.6.1?

there are some

like fix https://forum.doom9.org/showthread.php?p=1807637#post1807637

and remove yv12 limit, and some speed optimization in avs2.6 and avs+ (thanks to Y8/Y)

Last but not least, I'm gonna quote myself, asking the same question I asked a year ago:
Any chance in getting it to work in 16bit stacked or interleaved (HDR Core/Dither Tools) in Avisynth 2.6 (not just AVS+)?

not likely

[`Orum]

This doesn't appear to work when using it to upscale, i.e. dest_x or dest_y are larger than the input clip. I'll work on a fix unless real.finder is still around, as he's more familiar with the script than me.

[real.finder]

This doesn't appear to work when using it to upscale, i.e. dest_x or dest_y are larger than the input clip. I'll work on a fix unless real.finder is still around, as he's more familiar with the script than me.

fixed :)

[FranceBB]

fixed :)

Good to know, thanks.
I laughed reading back my post from 2017 in which I was asking for 16bit stacked or interleaved as I was on Avisynth 2.6.1 and I was tied to these two high bit depth options xD
(Yes, it was 2017 and I moved to Avisynth+ one year later) :P

[real.finder]

Good to know, thanks.
I laughed reading back my post from 2017 in which I was asking for 16bit stacked or interleaved as I was on Avisynth 2.6.1 and I was tied to these two high bit depth options xD
(Yes, it was 2017 and I moved to Avisynth+ one year later) :P

I just did another fix for using source clip case and some YUY2 cases codes

when dest_x or dest_y are larger than the input clip there are no YUY2 support and no avs2.6/avs+ speed up ofc

aside form that, 16bit stacked maybe impossible for LSFmod since lut/expr code was hard even for HBD

[Gser]

(Yes, it was 2017 and I moved to Avisynth+ one year later) :P

I remember that. It was a good day.

[Tempter57]

real.finder
Hi, LSFmod v.2.183 clashes with SMDegrain v3.1.2.104s, it concerns lines with sisupscale : "MergeLuma: Images must have same width and height! LSFmod v.2.183.avsi, line 600 \SMDegrain v3.1.2.104s.avsi, line 545"

[real.finder]

real.finder
Hi, LSFmod v.2.183 clashes with SMDegrain v3.1.2.104s, it concerns lines with sisupscale : "MergeLuma: Images must have same width and height! LSFmod v.2.183.avsi, line 600 \SMDegrain v3.1.2.104s.avsi, line 545"

fixed https://pastebin.com/P2TDCB7d

[Tempter57]

fixed https://pastebin.com/P2TDCB7d

Thanks you for script updating, now all are normal, it was necessary to connect only still function
function Padding(clip c, int left, int top, int right, int bottom)
{
w = c.width()
h = c.height()
c.pointresize( w+left+right, h+top+bottom, -left, -top, w+left+right, h+top+bottom )
}

[real.finder]

Thanks you for script updating, now all are normal, it was necessary to connect only still function
function Padding(clip c, int left, int top, int right, int bottom)
{
w = c.width()
h = c.height()
c.pointresize( w+left+right, h+top+bottom, -left, -top, w+left+right, h+top+bottom )
}

I have Padding as part of Advanced Denoising.avsi

anyway I plan to change this soon as I said here https://forum.doom9.org/showthread.php?p=1884872#post1884872

[Tempter57]

I have Padding as part of Advanced Denoising.avsi

anyway I plan to change this soon as I said here https://forum.doom9.org/showthread.php?p=1884872#post1884872

Has understood, but there was a problem of application LSFmod with smode=1 and smode=2

[real.finder]

Has understood, but there was a problem of application LSFmod with smode=1 and smode=2

that because not all filters updated yet to use avs 2.6 Y8

try this https://pastebin.com/eE37iYWZ

LSFmod(smode=2,avs26_optimize=false)

and if all filters updated later then you will not need to set avs26_optimize=false

[Boulder]

Found a little problem with show=true: the AddBorders calls have an incorrect syntax. Fixed by removing ',true' off them.

[real.finder]

Found a little problem with show=true: the AddBorders calls have an incorrect syntax. Fixed by removing ',true' off them.

what settings and what LSFmod is used? since I got no problem with lsfmod(show=true)

[Boulder]

HD source, AVS+ v3.6. The latest version of LSFMod, defaults="slow", show=true right after loading the source.

As far as I know, AddBorders doesn't have any Boolean variables.

[real.finder]

HD source, AVS+ v3.6. The latest version of LSFMod, defaults="slow", show=true right after loading the source.

As far as I know, AddBorders doesn't have any Boolean variables.

yes, it's copy/paste bug when I add it to crop, I will update it soon

[Boulder]

While you're working on the script, is it possible to get a larger kernel to use with (U)HD sources? As far as I remember, the 3x3 kernel suits SD sources but HD would require a larger one for the same effect regarding detail enhancement.

[real.finder]

While you're working on the script, is it possible to get a larger kernel to use with (U)HD sources? As far as I remember, the 3x3 kernel suits SD sources but HD would require a larger one for the same effect regarding detail enhancement.

I think that need RemoveGrainHD

[Boulder]

Ah, that one is still non-hbd :(

[LeXXuz]

I'm playing around with LSFmod a little. I use latest avs+ and LSFmod v2.187.

I use either 10-bit or 16-bit scripts/filterchains and I've noticed the sharpening effect with the default value of 100 is rather subtle, if noticeable at all.
So just increase strength to your liking, right? Well that works fine on LBD. However on HBD I get artefacts.

For a little test I use this line on a 1080p source:
LSFmod(strength=400, defaults="slow")

I chose 400 (which is way too much, of course) to make the difference more visible. ;)

8-bit looks okay, well oversharpened but okay. 10/16-bit looks, well see for yourself:

https://abload.de/thumb/2020-07-04123ckta.png (https://abload.de/image.php?img=2020-07-04123ckta.png)https://abload.de/thumb/2020-07-043x2kul.png (https://abload.de/image.php?img=2020-07-043x2kul.png)

Why is this happening in HBD?

[real.finder]

I'm playing around with LSFmod a little. I use latest avs+ and LSFmod v2.187.

I use either 10-bit or 16-bit scripts/filterchains and I've noticed the sharpening effect with the default value of 100 is rather subtle, if noticeable at all.
So just increase strength to your liking, right? Well that works fine on LBD. However on HBD I get artefacts.

For a little test I use this line on a 1080p source:
LSFmod(strength=400, defaults="slow")

I chose 400 (which is way too much, of course) to make the difference more visible. ;)

8-bit looks okay, well oversharpened but okay. 10/16-bit looks, well see for yourself:

https://abload.de/thumb/2020-07-04123ckta.png (https://abload.de/image.php?img=2020-07-04123ckta.png)https://abload.de/thumb/2020-07-043x2kul.png (https://abload.de/image.php?img=2020-07-043x2kul.png)

Why is this happening in HBD?

https://i.postimg.cc/PP9SLY5c/2020-07-04123ckta.png (https://postimg.cc/PP9SLY5c)

with

ImageSource("2020-07-04123ckta.png")
converttoyv24
convertbits(16)
LSFmod(strength=400, defaults="slow")
convertbits(8)

https://i.postimg.cc/5Qjmy2db/test-Copy-3-Copy-Copy000120.png (https://postimg.cc/5Qjmy2db)

[LeXXuz]

Thanks for the quick response. So something seems to be wrong on my system? :o But I wonder what. :(

[real.finder]

Thanks for the quick response. So something seems to be wrong on my system? :o But I wonder what. :(

can you try with the same image I post? maybe the original frame has noise that case this

if it still has that artefacts then make sure that you use last rgtools https://github.com/pinterf/RgTools and masktools https://github.com/pinterf/masktools/

[LeXXuz]

It seems I was NOT running the latest AVS+ version. My bad. :o

I had r3106, 3.5 (x64) installed. Now updated to r3300, 3.6 (x64)

But I can't get LSFmod to work anymore.
Some problem with version check:
https://abload.de/thumb/2020-07-0415snj5d.png (https://abload.de/image.php?img=2020-07-0415snj5d.png)

My AvsPlusVersionNumber.avsi
function IsAvsPlus()
{
FindStr(VersionString, "AviSynth+") != 0
}

function AvsPlusVersionNumber()
{
IsAvsPlus ? eval(MidStr(VersionString(),17,4)) : 0
}

My LSFmod.avsi:
https://pastebin.com/cZFKV0ab

Can't I just remove this version check thingy somehow?

[real.finder]

It seems I was NOT running the latest AVS+ version. My bad. :o

I had r3106, 3.5 (x64) installed. Now updated to r3300, 3.6 (x64)

But I can't get LSFmod to work anymore.
Some problem with version check:
https://abload.de/thumb/2020-07-0415snj5d.png (https://abload.de/image.php?img=2020-07-0415snj5d.png)

My AvsPlusVersionNumber.avsi
function IsAvsPlus()
{
FindStr(VersionString, "AviSynth+") != 0
}

function AvsPlusVersionNumber()
{
IsAvsPlus ? eval(MidStr(VersionString(),17,4)) : 0
}

My LSFmod.avsi:
https://pastebin.com/cZFKV0ab

Can't I just remove this version check thingy somehow?

update scripts from https://github.com/realfinder/AVS-Stuff

[LeXXuz]

What exactly should I update? :o

I can only find this avsi on that page:
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/LSFmod.avsi

Which looks to be the same.

[Boulder]

Zs_RF_Shared.avsi is one to update as well, it contains a lot of helper functions etc.

[LeXXuz]

Zs_RF_Shared.avsi is one to update as well, it contains a lot of helper functions etc.

That's what I was missing. Thank you! :)

can you try with the same image I post? maybe the original frame has noise that case this

if it still has that artefacts then make sure that you use last rgtools https://github.com/pinterf/RgTools and masktools https://github.com/pinterf/masktools/

Well I get the same result with this image as you do.
Then I took a picture of the source frame and ran your little script with it. Again no artefacts.

So, like you already assumed, something must be in the source vid that produces these artefacts. But why only in HBD? And how do I get rid of these? :confused:

I could downconvert to 8-bit, use LSFm and convert back up to 16-bit before using the next filter, but thats not really ideal, isn't it?

[real.finder]

Then I took a picture of the source frame and ran your little script with it. Again no artefacts.


then it should work with video too, did you tried after updating things?

anyway, sharpening should be after denoising

[LeXXuz]

then it should work with video too, did you tried after updating things?

anyway, sharpening should be after denoising

Yeah it's odd. I don't understand it atm. But the result with the source video is the same as before.

I always denoise before using LSFm. And, believe it or not, this picture is already denoised with a 2 stage filtering by dfttest. If I'll do more, I'll lose too much detail...

Usually I use some chain like:

-convert to 10/16bit
-denoise: dfttest (sometimes fft3dgpu, but rarely since it doesn't support HBD)
-sharpen: LSFm
-deband: neo_f3kdb
-encode to i420p10 with x.265

I was quite happy to now have a complete HBD capable filter chain, until I saw these strange artefacts.

However, I still have 0.98 of RGTools (x64-clang) and masktools2-v2.2.21 (x64-clang). Can this be the problem?

I saw you have newer versions on your page but I could only find source codes. (And I'm too dumb to compile stuff myself :()

Would be great if someone has these as compiled x64-dlls (maybe even optimized for Zen2 ?)

[real.finder]

Yeah it's odd. I don't understand it atm. But the result with the source video is the same as before.

I always denoise before using LSFm. And, believe it or not, this picture is already denoised with a 2 stage filtering by dfttest. If I'll do more, I'll lose too much detail...

Usually I use some chain like:

-convert to 10/16bit
-denoise: dfttest (sometimes fft3dgpu, but rarely since it doesn't support HBD)
-sharpen: LSFm
-deband: neo_f3kdb
-encode to i420p10 with x.265

I was quite happy to now have a complete HBD capable filter chain, until I saw these strange artefacts.

However, I still have 0.98 of RGTools (x64-clang) and masktools2-v2.2.21 (x64-clang). Can this be the problem?

I saw you have newer versions on your page but I could only find source codes. (And I'm too dumb to compile stuff myself :()

Would be great if someone has these as compiled x64-dlls (maybe even optimized for Zen2 ?)

can you upload raw video sample?

https://i.postimg.cc/yxzdKPC9/Untitled.png (https://postimages.org/)

https://github.com/pinterf/RgTools/releases has the compiled dlls, same for masktools2

and it's by pinterf not mine

[LeXXuz]

I did some testing with a few titles and I'm quite sure now it's not the movie.

Only noisy material makes those artefacts visible 'earlier'.

These next screenshots are taken from Angry Birds anime movie which is perfectly clean.
So I had to crank strength up to 1000(:eek:) to make the difference appear more noticeable.
I know that's way off daily use, but there's a clear visible difference. And it shouldn't, right?

https://abload.de/thumb/2020-07-0427yik7j.png (https://abload.de/image.php?img=2020-07-0427yik7j.png)https://abload.de/thumb/2020-07-04282ej75.png (https://abload.de/image.php?img=2020-07-04282ej75.png)


Note that strange solarize effect that appears on the HBD version. It seems the limiting does not quite work as it should and as it does in the LBD version.

Anyhow, I will update my RGTools and Masktools to see if it'll make any difference. :)

[feisty2]

the LUT expressions were targeted for 8-bit range (0-255), have you tweaked the expressions accordingly for each bit depth? you have to first scale the pixel range to [0.0, 255.0] before the LUT evaluation then scale it back when the evaluation completes.

[real.finder]

the LUT expressions were targeted for 8-bit range (0-255), have you tweaked the expressions accordingly for each bit depth? you have to first scale the pixel range to [0.0, 255.0] before the LUT evaluation then scale it back when the evaluation completes.

that already done, but I think there are some bug in bit scale (round error), I will see where it is

[real.finder]

I made mini LSFmod for debug


function LSFmoddebug( clip inclip )
{

ox = inclip.width
oy = inclip.height

strength = 400
kernel = 11

Szrp = 16
Spwr = 4
SdmpLo = 4
SdmpHi =48

overshoot = 4
undershoot = overshoot
overshoot2 = overshoot*2
undershoot2 = overshoot2

soft = -2
keep =20

ss_x = 1.50
ss_y = ss_x
dest_x = ox
dest_y = oy

soft = int( (1.0+(2.0/(ss_x+ss_y))) * sqrt(strength) )
soft = soft<=100 ? soft : 100

xxs = round(ox*ss_x/8)*8
yys = round(oy*ss_y/8)*8

str = float(strength)/100.0

input = inclip.converttoy()

### SHARP
tmp = input.spline36resize(xxs,yys)
pre = tmp

dark_limit = pre.mt_inpand(U=1,V=1)
bright_limit = pre.mt_expand(U=1,V=1)
minmaxavg = mt_average(dark_limit,bright_limit,U=1,V=1)

method1 = minmaxavg.removegrain(kernel,-1)

method = method1.mt_lutxy(pre,"x y < x 1 scalef + x y > x 1 scalef - x ? ?",use_expr=2,U=1,V=1)

normsharp = mt_lutxy(tmp, method, yexpr="x y == x x x y - abs "+string(Szrp)+" / 1 "+string(Spwr)+" / ^ "+string(Szrp)+" * "+string(str)+" * x y - x y - abs / * x y - 2 ^ "
\ +string(Szrp)+" 2 ^ "+string(SdmpLo)+" + * x y - 2 ^ "+string(SdmpLo)+" + "+string(Szrp)+" 2 ^ * / * 1 "
\ +string(SdmpHi)+" 0 == 0 "+string(Szrp)+" "+string(SdmpHi)+" / 4 ^ ? + 1 "+string(SdmpHi)+" 0 == 0 x y - abs "+string(SdmpHi)+" / 4 ^ ? + / * + ?",U=1,V=1,use_expr=2,scale_inputs="allf")


### LIMIT
normal = mt_clamp(normsharp, bright_limit, dark_limit, overshoot, undershoot, U=1, V=1)
second = mt_clamp(normsharp, bright_limit, dark_limit, overshoot2, undershoot2, U=1, V=1)

edge1 = mt_logic( tmp.mt_edge(thY1=0,thY2=255,"8 16 8 0 0 0 -8 -16 -8 4",U=1,V=1)
\ ,tmp.mt_edge(thY1=0,thY2=255,"8 0 -8 16 0 -16 8 0 -8 4",U=1,V=1)
\ ,"max",U=1,V=1)

edge = edge1.mt_lut("x range_half / 0.86 ^ range_max *",use_expr=2,scale_inputs="floatf",clamp_float=true,U=1,V=1)

limit1 = mt_merge(second, normal, edge.mt_inflate(U=1,V=1), U=1, V=1)

limit2 = limit1


### SOFT
sharpdiff1 = mt_makediff(tmp,limit2,U=1,V=1)
sharpdiff = mt_lutxy(sharpdiff1,sharpdiff1.removegrain(19,-1),
\ "x range_half - abs y range_half - abs > y "+string(soft)+" * x "+string(100-soft)+" * + 100 / x ?",use_expr=2,U=1,V=1)

PP1 = mt_makediff(tmp,sharpdiff,U=1,V=1)


### SOOTHE
diff = mt_makediff(tmp,PP1,U=1,V=1)
diff2 = diff.temporalsoften(1,255,0,32,2)
diff3 = mt_lutxy(diff,diff2,"x range_half - y range_half - * 0 < x range_half - 100 / "+string(keep)+
\ " * range_half + x range_half - abs y range_half - abs > x "+string(keep)+" * y 100 "+string(keep)+" - * + 100 / x ? ?",use_expr=2,clamp_float=true,U=1,V=1)

PP2 = mt_makediff(tmp,diff3,U=1,V=1)


### OUTPUT
out = PP2.spline36resize(dest_x,dest_y)

in = input

shrpD = mt_makediff(in,out,U=1,V=1)
output = out

output = CombinePlanes(output,inclip,planes="YUV",sample_clip=inclip)

return output
}
Source("")
convertbits(16)
LSFmoddebug()
convertbits(8)

[real.finder]

ok, I think I know why, every function (Especial the cores and the rgtools one) give slightly different output in HBD compare to 8bit and since it's complex script then the final output is kinda big different compare to 8bit one, maybe because these are as old as https://forum.doom9.org/showthread.php?p=1916380#post1916380 so they has no rounding

edit: using https://forum.doom9.org/showthread.php?p=1916545#post1916545 you can check with

ColorBars
converttoyv12
AddGrainC(constant=true)
Interleave(somefilter().luma_histogram(),convertbits(16).somefilter().convertbits(8).luma_histogram())

[real.finder]

I did some testing with a few titles and I'm quite sure now it's not the movie.

Only noisy material makes those artefacts visible 'earlier'.

These next screenshots are taken from Angry Birds anime movie which is perfectly clean.
So I had to crank strength up to 1000(:eek:) to make the difference appear more noticeable.
I know that's way off daily use, but there's a clear visible difference. And it shouldn't, right?

https://abload.de/thumb/2020-07-0427yik7j.png (https://abload.de/image.php?img=2020-07-0427yik7j.png)https://abload.de/thumb/2020-07-04282ej75.png (https://abload.de/image.php?img=2020-07-04282ej75.png)


Note that strange solarize effect that appears on the HBD version. It seems the limiting does not quite work as it should and as it does in the LBD version.

Anyhow, I will update my RGTools and Masktools to see if it'll make any difference. :)

can you try this update? https://github.com/realfinder/AVS-Stuff/raw/Community/avs%202.5%20and%20up/LSFmod.avsi

[Boulder]

Does the problem affect also other ports, like FineSharp?

[real.finder]

Does the problem affect also other ports, like FineSharp?

I think no, LSFmod was more complex, but you can try with many settings and see