overlay mask bug : 8bit mask values 129 to 254 in RGB or Y produce wrong value

[poisondeathray]

avs+ x64 r3593

values 129 to 254 in mask produce output expected value -1 for 8bit RGB or YUV overlay (mask value 255 ok)

test png 256x256 gradient, RGB 0-255. Each x coordinate is RGB value. e.g x=30 is RGB 30,30,30

https://i.postimg.cc/X7GxkHN6/greyscale.png
or
https://www.mediafire.com/file/dmktu625gyau5d8/greyscale.png/file


g=ImageSource("greyscale.png")
b=blankclip(g, colors=[0,0,0] )
w=blankclip(g, colors=[255,255,255] )

o=overlay(b,w,mask=g)

o
#subtract(g,o)
#Levels(127, 1, 129, 0, 255, false)



You can check with color picker in avspmod or vsedit ; or use amplified differences (subtract).

Repeating test in Y; with converttoyv24(matrix="pc.709"), (and b=0,128,128 , w=255,128,128) for YUV, same trend of 129 to 254 ; output Y expected value -1



vapoursynth overlay ok


clip = core.imwri.Read(r'greyscale.png')
b = core.std.BlankClip(clip, color=[0, 0, 0])
w = core.std.BlankClip(clip, color=[255,255,255])
ov = haf.Overlay(b,w, mask=clip)

d = core.std.MakeDiff(clip, ov, planes=[0,1,2])
da = core.std.Levels(d, min_in=127, max_in=129, gamma=1, min_out=0, max_out=255, planes=[0,1,2])

ov.set_output()
#da.set_output()



EDIT:
Same trend happens in avs+ with 10bit overlay with 0-1023 grad . Mask values over half (513 to 1022) produce output values -1 . 1023 works ok

[real.finder]

I can see this bug in mt_merge too

g=ImageSource("greyscale.png").ConvertToPlanarRGB
b=blankclip(g, colors=[0,0,0] )
w=blankclip(g, colors=[255,255,255] )

o=mt_merge(b,w,g, u=3,v=3)

o
#subtract(g,o)
#Levels(127, 1, 129, 0, 255, false)

replacing mt_merge with smaskmerge fix it

there was fix for both mt_merge https://github.com/pinterf/masktools/commit/ad143a5693882c6c827dd4ff8dbfe29de843dcb2 and overlay https://github.com/AviSynth/AviSynthPlus/commit/fb74724bf6719b07261c136f212f49d0b648b1d1 because of https://github.com/tp7/masktools/issues/12 but seems these fixes was not enough... now I think it need to fix this case too by maybe "Mask > range_half ? mask + 1 : mask"

[poisondeathray]

mt_merge is ok in planar RGB with that example using luma as the mask (luma=true) . (or in the YUV case is ok too)

m = mt_merge(b,w,g, luma=true)

dogway's ex_merge also ok

ex_merge(b,w,g)

related functions that use the internal Overlay are affected (such as overlayplus , etc...)

[Reel.Deel]

related functions that use the internal Overlay are affected (such as overlayplus , etc...)

OverlayPlus uses mt_merge. I was trying to see the problem myself yesterday but I had trouble understanding what the problem is :o.

[poisondeathray]

OverlayPlus uses mt_merge. I was trying to see the problem myself yesterday but I had trouble understanding what the problem is :o.



Whoops, I should have looked more closely

But , mt_merge is incorrect too... my bad I mixed up the test

[poisondeathray]

Semi related, maybe the same bug, mt_lutspa is supposed to make a gradient


blankclip(width=256, height=256, pixel_type="YV24")
mt_lutspa(mode="relative", expr="x 255 *", u=128,v=128)



And it's the same trend, values over halfway point have -1 expected value. But in the mt_lutspa case, the xpos=255 is still 254, not like the overlay bug where the full mask value is ok

[pinterf]

avs+ x64 r3593

values 129 to 254 in mask produce output expected value -1 for 8bit RGB or YUV overlay (mask value 255 ok)

test png 256x256 gradient, RGB 0-255. Each x coordinate is RGB value. e.g x=30 is RGB 30,30,30

EDIT:
Same trend happens in avs+ with 10bit overlay with 0-1023 grad . Mask values over half (513 to 1022) produce output values -1 . 1023 works ok
Works as expected, at least usual implementations work like this. The expected value is unfortunately not the mask value in between the extremes.

Ideally the formula is the following:
result = p1 * (1-mask) + p2 * mask

For speed considerations (one less multiplication) it is written as
result = p1 + (p2-p1)*mask

where mask is between 0..1.0 inclusive.

Unfortunately we do not want to use float arithmetic, neither slow integer division. Mask is an integer value between 0 and 255 for a 8 bit input (let's call it mask_i)

In integer arithmetic we use the following formula:
result = (p1 * 256 + (p2-p1)*mask_i + 128) / 256

which is implemented as
result = ((p1 << 8) + (p2-p1)*mask_i + 128) >> 8

128 is the rounding before scaling back 8 bits, << is the shift operator. 8 shift right makes division by 2^8, that is 256

Let's check four cases.
ideal:
result = p1 + (p2-p1)*mask
approximate:
result_i = ((p1 << 8) + (p2-p1)*mask_i + 128) >> 8

Inputs are p1=0, p2=255

mask = 0 (0.0)
result = 0 + (255 - 0) * 0.0 => 0
result_i = (0 * 256 + (255 - 0) * 0 + 128) >> 8 => 0


mask = 128 (128.0/256)
result = 0 + (255 - 0) * (128.0/256) => 127.5 => 128
result_i = (0 * 256 + (255 - 0) * 128+ 128) >> 8 => 128
still resembles the the real result

mask = 129 (129.0/256)
result = 0 + (255 - 0) * (129.0/256) => 128.49 => 128
result_i = (0 * 256 + (255 - 0) * 129+ 128) >> 8 => 128

From now on, because mask is not
mask_value_integer/255
but for speed reasons
mask_value_integer/256
The output will differ, until special case correction of mask=255

mask = 255 (1.0)
result = 0 + (255 - 0) * 1.0 => 255
result_i = (0 * 256 + (255 - 0) * 255 + 128) >> 8 => 254
!!! Note that for the maximum mask value of 255 we still get only 254 instead of 255! This is so much unexpected and ugly that this special case is handled: when mask=255, overlay (and mt_merge) returns p2 unaltered! Similarly when mask=0 then p1 is returned unaltered. This is because in our approximate calculation 255 is really not 1.0 but rather 255/256 (0.996), but we specially treat is to behave as it was a real 1.0 value

So the basic code is the following:
if (mask == 0)
return p1;
if (mask == 0xFF)
return p2;
// p1*(1-mask_f) + p2*mask_f -> p1 + (p2-p1)*mask_f
return (BYTE)(((p1 << 8) + (p2 - p1)*mask + 128) >> 8);


I hope this explanation helped a bit, one can turn the calculations into a super exact one for mask values 1-254 but it will be significantly slower, but this ultra special case of this example will definitely work.

[pinterf]

Semi related, maybe the same bug, mt_lutspa is supposed to make a gradient


blankclip(width=256, height=256, pixel_type="YV24")
mt_lutspa(mode="relative", expr="x 255 *", u=128,v=128)



And it's the same trend, values over halfway point have -1 expected value. But in the mt_lutspa case, the xpos=255 is still 254, not like the overlay bug where the full mask value is ok

Same reason, which is special-cased in mt_merge and Overlay:
(255*255 + 128) >> 8 is 254

I guess you can fine tune the range (inclusive/exclusive ending) with 'biased' parameter
http://avisynth.nl/index.php/MaskTools2/mt_lutspa

[Reel.Deel]

Same reason, which is special-cased in mt_merge and Overlay:
(255*255 + 128) >> 8 is 254

I guess you can fine tune the range (inclusive/exclusive ending) with 'biased' parameter
http://avisynth.nl/index.php/MaskTools2/mt_lutspa

Using biased=false did not change anything, the middle value (128) gets repeated twice. Using expr="x 256 *" instead of expr="x 255 *" does produce a correct 0-255 gradient.

Edit:

replacing mt_merge with smaskmerge fix it

Yeah, using the expression Expr(b,w,g, "x 255 z - * y z * + 255 /") from mirkosp's MaskCL (https://pastebin.com/Hh3NutUB) works correctly.

[poisondeathray]

I hope this explanation helped a bit, one can turn the calculations into a super exact one for mask values 1-254 but it will be significantly slower, but this ultra special case of this example will definitely work.

Thanks for the detailed explanations


Using biased=false did not change anything, the middle value (128) gets repeated twice. Using expr="x 256 *" instead of expr="x 255 *" does produce a correct 0-255 gradient.

Nice; do you think the example in the wiki should be edited ?

http://avisynth.nl/index.php/MaskTools2/mt_lutspa

[zorr]

one can turn the calculations into a super exact one for mask values 1-254 but it will be significantly slower

My spider sense says that it should not be significanty slower, the amount of computation compared to memory access is still quite low. And you can get rid of the conditionals when using the exact one (might still not be faster without conditionals when using simple masks as the branch predictor will guess correctly most of the time).

Would it be a lot of work to test the performance of the exact version? I guess it depends on whether it's optimized SIMD code or plain C. In any case these mask operations are some of the fastest routines in Avisynth so even a bit slower version would not hurt a lot.

[Reel.Deel]

Nice; do you think the example in the wiki should be edited ?

http://avisynth.nl/index.php/MaskTools2/mt_lutspa

I'm not sure... It seems that when the width of a horizontal (height for vertical) gradient is greater than 509, expr="x 255 *" produces a correct [0,255] gradient. When you go below those dimensions is when the behavior changes, here's what I found:


width expr to use to get [0,255]
73-101 "x 258 *"
102-170 "x 257 *"
170-509 "x 256 *"
≥510 "x 255 *"


The lower you go, the closer the you encounter a change in the highest value. I tried up to 3840 width and the gradient was still correct.

Edit: after a bit more testing, mt_lutspa(mode="relative closed", expr="x 255 *") or mt_lutspa(relative=true, biased=false, expr="x 255 *") produce correct results for any dimension. Whenever "mode" is set, it overrides the parameters "relative" and "biased", hence why pinterf's suggestion didn't work for me, the mode parameter was set to relative. I completely had forgotten about that.

[pinterf]

My spider sense says that it should not be significanty slower, the amount of computation compared to memory access is still quite low. And you can get rid of the conditionals when using the exact one (might still not be faster without conditionals when using simple masks as the branch predictor will guess correctly most of the time).

Would it be a lot of work to test the performance of the exact version? I guess it depends on whether it's optimized SIMD code or plain C. In any case these mask operations are some of the fastest routines in Avisynth so even a bit slower version would not hurt a lot.
My opinion is that this is a very quick filter and making it accurate does not harm any serious scripts' running time in general. In 2022 we can surely prioritize quality over sparing one or two CPU cycles. I remember on examples when filters were made a tiny bit slower but then they accurately used proper rounding in their internal calculations.

[real.finder]

My opinion is that this is a very quick filter and making it accurate does not harm any serious scripts' running time in general. In 2022 we can surely prioritize quality over sparing one or two CPU cycles. I remember on examples when filters were made a tiny bit slower but then they accurately used proper rounding in their internal calculations.

I agree, maybe as a kind of compatibility adding parameter to switch to the old faster formula (bool "fast" as false by default maybe?) also just in case for use case that don't need full accurate output

[Reel.Deel]

I did some test just to judge the speed of Overlay/mt_merge versus doing it the correct way using mt_lutxyz/Expr. Clearly Overlay and mt_merge are much faster but even then, a 100fps filter wont be the bottle neck once you start doing more complex things like denoising, etc., let alone encoding via x264/5. I assume that with a proper implementation via MaskTools or in the avs+ core would be somewhat faster.

Script:

g = Blankclip(width=1920, height=1080, length=1, pixel_type="YUV444P8"). \
mt_lutspa(mode="relative", expr="x 255 *", u=3, v=3).Loop(1000)
b = blankclip(g, colors=[0,0,0])
w = blankclip(g, colors=[255,255,255])

o1 = Overlay(b, w, mask=g)
o2 = mt_merge(b, w, g, u=3, v=3)
o3 = mt_lutxyz(b, w, g, expr="x 255 z - * y z * + 255 /", u=3, v=3)
o4 = Expr(b, w, g, "x 255 z - * y z * + 255 /")
o5 = Expr(b, w, g, "x dup y - z 0.003921569 * * -")

Results (from a 10yr old i7, avx only):

Overlay
Frames processed: 1000 (0 - 999)
FPS (min | max | average): 189.6 | 351.3 | 336.6
Process memory usage (max): 118 MiB
Thread count: 16
CPU usage (average): 7.0%

Time (elapsed): 00:00:02.971

mt_merge
Frames processed: 1000 (0 - 999)
FPS (min | max | average): 190.8 | 349.5 | 322.5
Process memory usage (max): 118 MiB
Thread count: 16
CPU usage (average): 7.0%

Time (elapsed): 00:00:03.101

mt_lutxyz
Frames processed: 1000 (0 - 999)
FPS (min | max | average): 81.85 | 131.7 | 125.4
Process memory usage (max): 118 MiB
Thread count: 16
CPU usage (average): 7.9%

Time (elapsed): 00:00:07.974

Expr
Frames processed: 1000 (0 - 999)
FPS (min | max | average): 76.97 | 106.6 | 102.3
Process memory usage (max): 118 MiB
Thread count: 16
CPU usage (average): 7.9%

Time (elapsed): 00:00:09.772

Expr (Dogway)
FPS (min | max | average): 73.92 | 139.9 | 129.9
Process memory usage (max): 117 MiB
Thread count: 16
CPU usage (average): 7.8%

Time (elapsed): 00:00:07.699

[Dogway]

I get faster times in Expr with multithreading at least on my CPU (AVX2) but you need to use the lerp formula.
g = Blankclip(width=1920, height=1080, length=1, pixel_type="YUV444P8"). \
mt_lutspa(mode="relative", expr="x 255 *", u=3, v=3).Loop(5000)
b = blankclip(g, colors=[0,0,0])
w = blankclip(g, colors=[255,255,255])

o1 = Overlay(b, w, mask=g) # 545 800
o2 = mt_merge(b, w, g, u=3, v=3) # 540 760
o3 = mt_lutxyz(b, w, g, expr="x 255 z - * y z * + 255 /", u=3, v=3) # 184 615 720 (with use_expr=3)
o4 = Expr(b, w, g, "x 255 z - * y z * + 255 /") # 204 792
o5 = Expr(b, w, g, "x dup y - z 0.003921569 * * -") # 268 1000

# Prefetch(8)