Transparency/Alpha implementation

[jmac698]

I need to understand exactly how alpha is calculated, but I can't quite get it right. Here's my script:

l=255
v=64
lv=70
blankclip(width=64,height=64,color=l*65536+l*256+l)#test
mask(blankclip,last)
layer(blankclip(color=v*65536+v*256+v),last,"subtract",level=lv,use_chroma=false)
subtitle(string((l*lv+(257-lv)*v)/256.0))

My estimate shows 116.4, but the actual result is 115. Did I find some inaccuracy in avisynth, or is my model wrong somehow?
This is based on 2.58. If you get different results in 2.6, let me know.

The code is the asm at layer.cpp, line 2104. I can't read it.


if (!lstrcmpi(Op, "Subtract"))
{
if (chroma)
{
__asm {
push ebx // stupid compiler forgets to save ebx!!
mov edi, src1p
mov esi, src2p
mov ebx, myy
movd mm1, mylevel
pxor mm0, mm0
pcmpeqd mm3, mm3
pcmpeqb mm4, mm4
psrlq mm3, 63 ;00000000|00000001
punpcklbw mm4, mm0 ;0x00ff00ff00ff00ff

sub32loop:
mov edx, myx
xor ecx, ecx

align 16
sub32xloop:
movd mm6, [esi + ecx*4] ;src2
movd mm7, [edi + ecx*4] ;src1/dest
movq mm2, mm6
punpcklbw mm6, mm0 ;mm6= 00aa|00rr|00gg|00bb [src2]
//----- extract alpha into four channels
psrld mm2, 24 ;mm2= 0000|0000|0000|00aa
pandn mm6, mm4 ;mm6 =~mm6
pmullw mm2, mm1 ;mm2= pixel alpha * script alpha
punpcklbw mm7, mm0 ;mm7= 00aa|00rr|00gg|00bb [src1]
paddd mm2, mm3 ;mm2+=1
psrld mm2, 8 ;mm2= 0000|0000|0000|00aa*
punpcklwd mm2, mm2 ;mm2= 0000|0000|00aa*|00aa*
punpckldq mm2, mm2 ;mm2=00aa*|00aa*|00aa*|00aa*
//----- begin the fun stuff
psubsw mm6, mm7
pmullw mm6, mm2 ;mm6=scaled difference*255
psrlw mm6, 8 ;scale result
mov eax, ecx
paddb mm6, mm7 ;add src1
//----- end the fun stuff...

[Gavino]

(l*lv+(257-lv)*v)/256.0
Did I find some inaccuracy in avisynth, or is my model wrong somehow?
How did you derive your model?
The Avisynth code seems to match the Layer wiki description:
http://avisynth.org/mediawiki/Layer

Formula used :-
RGB32 :: base += ((overlay-base)*(alpha*level+1)>>8)>>8

subtract: the same as add, but the overlay_clip is inverted before.

[jmac698]

Ugh. The built in documentation does not match the wiki. How can I download the updated documentation? My model comes from the definition of alpha plus the apparently bad built-in docs.
All I had to go on was this:

in RGB32 the alpha-channel of the overlay_clip is multiplied with level, so the resulting alpha = (alpha_mask * level + 1) / 256

[Wilbert]

Ugh. The built in documentation does not match the wiki. How can I download the updated documentation?
I'm not sure when it was corrected, but it's correct in the offline documentation of v2.60a3. So you should get that.

[Gavino]

The bit I quoted is also in the offline (installed) docs, since at least v2.57. You have to scroll down to the description of 'add' and 'subtract'.

[jmac698]

aha, I'm blind and once again G to the rescue! Thanks.
final=base*(1-alpha)+alpha*overlay
final=base+alpha*(overlay-base)
base=(alpha*overlay-final)/(alpha-1)
alpha=(final-base)/(overlay-base)
overlay=(final+(alpha-1)*base)/alpha
where alpha=0 is transparent (pure base), alpha=1 is opaque (pure overlay)
overlay is the logo (as a typical use), base and final are before and after overlay
These are the simple formulas where values range from 0 to 1. You'll have to adjust to make it work with integer video levels.
There's a few more things to know, like how to really find a logo and reverse it, and shadow logos, and dealing with color.
You should be able to undo any transparent parts of logos from one still frame of black, or logos with shadow parts with a grey frame. You could detect opaque parts with two frames, I think. There is also a loss of accuracy involved, and also the problem that compressed video with logo isn't accurate.

[jmac698]

Ok I'm confused about a few things.
-If you make a logo on a black background then you should import that as an alpha channel and invert it, so the black parts are fully transparent. layer doesn't allow full transparency because of the "+1". Overlay would always be a solid white, it's not used.
-Is it true that even for a fully opaque logo, the anti-aliased edges are semi-transparent?
-How would you compute a colored logo? It seems you need 3 alpha channels and to apply the formula with each one in RGB.
-How would you handle a logo with drop-shadow? Alpha can only add to the pixels. Maybe this is a use for overlay<>white. But how are real logos stored, as plus and minus layers or a "signed" graphic against a grey background?
-It seems the simplified formula is just final=base+alpha*(1-base)

[Gavino]

layer doesn't allow full transparency because of the "+1".
With full transparency (alpha=0), the +1 has no effect as it disappears after >>8.
I believe the +1 is there to make full opacity (alpha=255, level=257) work correctly.

Actually, there is an error in the docs (both online and offline):

RGB32 :: base += ((overlay-base)*(alpha*level+1)>>8)>>8
should be
RGB32 :: base += ((overlay-base)*((alpha*level+1)>>8))>>8