little problem porting an 8bpc lut to 16bpc

[feisty2]

I wanna port a lut written by Didee in 2011 to 16bpc precision, so I can use it on 16 bpc clips
original script

mt_lutxy(o,o.removegrain(20),"x x y - abs 4 / 1 4 / ^ 4 * 1.51 * x y - x y - abs 1.001 + / * +",U=2,V=2)
#http://forum.doom9.org/showthread.php?t=163598#

here's my try

#16bpc input clip#
#16bpc sharp clip#
#str=1.51#
Sharpdif = Dither_sub16 (sharp, input, dif=True, y=3, u=3, v=3)
Dif = Dither_lut16 (Sharpdif, "4 "+String (str)+" * x 32768 - abs 4 / 1 4 / ^ * x 32768 - x 32768 - abs / 256.256 + * 32768 +", y=3, u=3, v=3)

input.Dither_add16 (Dif, dif=True, y=3, u=3, v=3)

somehow it didn't work as I expected, things get totally messed up, what did I do wrong? thx in advance
:stupid:

[feisty2]

okay, did a little hack and it works now

str = 1.00

Sharpdif = Dither_sub16 (sharp, input, dif=True, y=3, u=3, v=3)
Dif = Dither_lut16 (Sharpdif, "x 256 / 128 - abs 4 / 1 4 / ^ 4 * "+String (str)+" * x 256 / 128 - x 256 / 128 - abs 1.001 + / * 128 + 256 *", y=3, u=3, v=3)

input.Dither_add16 (Dif, dif=True, y=3, u=3, v=3)

I scaled down the domain from 16bpc to 8bpc and expanded it back to 16bpc at the end so I can use the original lut expr, since dither_lut16 works at float point precision anyways, it won't decrease precision, but still don't know why the earlier code doesn't work

[colours]

The "x y - abs 4 / 1 4 / ^ 4 *" part is nonlinear and you can't just change the scale of the x and y values unless you adjust your str parameter to compensate for that.

"x 32768 - x 32768 - abs / 256.256 +" also has the operations in a different order from the original; in your version x=32768 actually produces a division by zero.

[feisty2]

well, thx, I'm new at lut stuff, so I guess I'll stick to the latter hack version, it avoids so many troubles

[feisty2]

The "x y - abs 4 / 1 4 / ^ 4 *" part is nonlinear and you can't just change the scale of the x and y values unless you adjust your str parameter to compensate for that.

are you referring to the earlier version or the hack version, I think the nonlinear part is correct on the hack version since the coefficients are designed for 8bpc scale and "x" is actually at 8bpc scale in hack version, I tested the hack version, with u=2 v=2 and rounded to 8bpc, it produces the same result as the original 8bpc version

[colours]

I was referring to the earlier version; the "hack" version already scales the domain so there're no issues there.