What the dillio with them dots?

[mf]

When doing unsharp masking, I always get this weird side-effect. That's with AVS' Subtract() as well as MaskTools' YV12Subtract().

Script:

input = last

input.Deen("a2d", 4, 20, 20)
unsharp = last

YV12Subtract(input, unsharp)
YV12Subtract(last, unsharp).Invert()
YV12Subtract(last, input)
YV12Subtract(last, unsharp).Invert()

Output:
http://mf.creations.nl/avs/dots.png

As you can see, I get inverted pixels on pure black and on pure white. Am I missing some math knowledge here or is this really strange™ ?

[Mug Funky]

that's really weird... i can't replicate this one.

try it with blur() instead of deen() maybe and see what happens?

[mf]

Originally posted by Mug Funky
that's really weird... i can't replicate this one.

try it with blur() instead of deen() maybe and see what happens?
Same thing. The dots appear rarely, but they're very annoying. Try using it on Subtitle()'d material.

[Mug Funky]

i'll be damned... there they are.

hmm.

btw, couldn't you use:

YV12Subtract(unsharp,last)

instead of:

YV12Subtract(last, unsharp).Invert()

(btw, i don't use masktools, in fact i don't even have it or deen, but with regular subtract this works)

[edit]

WOW! you should see it in RGB32!! that's whack. thanks for pointing this one out...

http://www.sharemation.com/mugfunky/whack.jpg

[scharfis_brain]

it seems to be a levels-overflow-problem..

if out put of subtract is within 0 to 255, everything is fine, but if the subtraction expands this range, every thiing over 255 gets started at 0
256 -> 0
257 -> 1
...

and vice versa:
-1 -> 255
-2 -> 254
...

try to use

levels(0,1,255,16,235) before doing anything
and levels(16,1,235,0,255) after all processing.

this sqeezes the lumarange while processing, avoiding those strange dots.

[mf]

Originally posted by scharfis_brain
levels(0,1,255,16,235) before doing anything
and levels(16,1,235,0,255) after all processing.
Tried that already, no improvement.

[Kurosu]

YV12Subtract/MaskTools should have an option for that where it saturates. Can't remember its name, probably tol=0 or the like. And scharfis_brain is completely right. It was surprising to see that in the original subtract code, so I left it supposing it had some use, but nonetheless added another mode (faster because MMX code doesn't have to unpack bytes)

[mf]

So why is it inverted at all, instead of just rounded down as you'd expect anything that is off-limits?

[Kurosu]

Because it needs branching, and branching cost CPU cycles.
The code that handles it fine is like that in pseudo-C:
if (val>255) out=255
else if (val<0) out=0
else out=val
And maybe because the one that wrote that code had some use of such a behaviour, like the coring option of levels (quite dangerous for masks, necessary to abide ITU standards).

MMX doesn't have this problem because there's a function that automatically saturates to 0 and 255 (paddusb, while paddb has the behaviour of C)

Maybe intuition is something that seems only natural, and a rule by itself, but we are dealing with bitwise circuitry here. Don't expect things to be natural. Ever heard of Little or Big Endian?

[mf]

So it was just not thought of when coded :). So what's the fix?
Edit: so tol=0 it is. Does force me to change the way I subtract though :).

[mf]

Ok, any idea how I go about using tol? tol=0 only gives me dark difference (meaning no white halos - this is of course positive when it comes to things like line darkening, but that's not what I'm doing now!), using tol=128 gives me a dark picture, and tol=255 gives me black. Negative values result in said dots. I was hoping to be using this stuff on an encode I need to do tomorrow.