Layer and Overlay operation giving wrong results with RGBA image source

[pancserzso]

I would like to compose two image sources on each other, using alpha blending.

My problem is that:
1. If I just simply load the image with imagesource, the result is OK (I think it just discards the alpha channel)

2. However as soon as I do an Overlay or Layer operation, Avisynth seems to handle the alpha of the image wrong.

3. I believe the problem is pre-multiplied vs. post-multiplied alpha. I found a filter especially for fixing this, avisynth-unpremultiply (https://code.google.com/p/avisynth-unpremultiply/), however it made a totally crazy buggy colors with rainbow like blends and other bugs.

Is there any way to tell AviSynth to handle the alpha in my foreground images properly?

Here is my bg image:
https://dl.dropbox.com/s/fdq7hfm679hbmh2/composebug_bg.png

My foreground image:
https://dl.dropbox.com/s/t5c8rt0psu0l56o/composebug_fg.png

How it looks in AviSynth:
https://dl.dropbox.com/s/h7412jh6e81t1pl/composebug_avisynth.png

And how it should look, as when composed in After Effect, and selecting "premultiplied" for the foreground's alpha:
https://dl.dropbox.com/s/xb9igdqd8yyby4y/ae_00000.png


This is the Avisynth script I'm trying to use (in 2.60 b4):

background = ImageSource( "e:\render_outside\draft%05d.png", 1, 5600, 24, pixel_type="RGB24" ).ConvertToRGB32()

foreground = ImageSource( "e:\render_mixer\image%05d.png", 1, 5600, 24, pixel_type="RGB32" )

Layer( background, foreground )

[sqrt(9801)]

Funny, I was thinking of making a thread for the same problem.

However, I'm not sure whether it is a matter of pre/post-multiplied alpha. To be honest, I didn't even consider the possibility before you mentioned it, mostly because I was thinking it would be caused by something like a RGB <-> YUV conversion.

For instance, this script is supposed to return the overlay layer (basic ColorBars clip fading out via alpha channel) untouched by overlaying it over a blank, transparent clip. It doesn't. It even looks like it changes the RGB values for some reason.

ColorBars(width=640, height=480, pixel_type="RGB32")
KillAudio
Trim(0, length=255)

MergeARGB(BlankClip(last, color=$FF000000).FadeOut(framecount-1, color=$00000000), 
    \ ShowRed, 
    \ ShowGreen,
    \ ShowBlue)
    
top_layer=last

Layer(BlankClip(last, color=$00000000), last, op="add", level=257)

Compare(top_layer, channels="RGBA", show_graph=false) # try with channels="R", "G", "B" and "A"

I've also tried with Panzerboy's Gimp-style Layer merge plug-in, no dice.

[IanB]

Yes the alpha mask scaling is incompatible.

See this thread, "Alternative to the old Unpremultiply() plugin?", for a fixed version of the unpremultiply plugin and discussion on alternatives (lutxy etc).

[pancserzso]

Thanks IanB,

I've tried the linked version, and while it is indeed better, it's still way off from the real solution.
https://dl.dropbox.com/s/lat7axfw9in...remult_new.png

Actually, this image looks really similar to what After Effects looks when I'm in 8 bit mode. I had to put After Effects to 32 bit floating point mode to get the correct results.

Which is strange to me! Do pre-multiplying really need 32-bit floating point operation?

[poisondeathray]

Originally Posted by pancserzso

Thanks IanB,

I've tried the linked version, and while it is indeed better, it's still way off from the real solution.
https://dl.dropbox.com/s/lat7axfw9in...remult_new.png

Actually, this image looks really similar to what After Effects looks when I'm in 8 bit mode. I had to put After Effects to 32 bit floating point mode to get the correct results.

Which is strange to me! Do pre-multiplying really need 32-bit floating point operation?

Way off ? It think if will be closer if you use showalpha() first, before using unpremultiply , but even then there are some differences in the little specs

 

fg=ImageSource("composebug_fg.png", pixel_type="rgb32")
bg=ImageSource("composebug_bg.png", pixel_type="rgb32")
overlay(bg, fg, mask=fg.showalpha.unpremultiply)

[pancserzso]

Originally Posted by poisondeathray

Way off ? It think if will be closer if you use showalpha() first, before using unpremultiply , but even then there are some differences in the little specs

 

fg=ImageSource("composebug_fg.png", pixel_type="rgb32")
bg=ImageSource("composebug_bg.png", pixel_type="rgb32")
overlay(bg, fg, mask=fg.showalpha.unpremultiply)

Update!

OK, I'm puzzled for many reasons:

1. The method poisondeathray looks good, however it's just a coincidence. Upon looking at the mask, it's clear that we are pretty much discarding all the grayscale information and making a binary mask. Thus, the background is _totally_ not visible behind those areas.
Look:
normal alpha
unpremultiplied's alpha

2. Upon looking at the code, I'm even more confused.

I don't understand 2 things:
1. How could a 32-bit float operation help here. The result I get with Layer seems really similar to AE's 8-bit mode.
2. Upon looking at the code, it seems to me that alpha should stay intact. However fg.showalpha.unpremultiply() clearly shows different results compared to the normal one. How is this possible?

From cpp:

const int srcapix = srcp[w+3]  ;
dstp[w+0] = lookup[ (srcp[w+0]<<8) + srcapix ];
dstp[w+1] = lookup[ (srcp[w+1]<<8) + srcapix ];
dstp[w+2] = lookup[ (srcp[w+2]<<8) + srcapix ];
dstp[w+3] = srcapix;

[Gavino]

Originally Posted by poisondeathray

overlay(bg, fg, mask=fg.showalpha.unpremultiply)

How can that work?
ShowAlpha() copies the alpha channel into R, G and B.
Unpremultiply then divides R, G and B by alpha, so you should end up with a mask with all pixels 255.

Originally Posted by pancserzso

BTW, it's still interesting that it looked the same as when AE was in 8-bit mode. Maybe by using floats for the calculation in Unpremultiply we could have the same effect.

No, the precision has already been lost in the 8-bit RGB32 input.
The result of the premultiplication needs to be stored in at least 16 bits to preserve precision.

[pancserzso]

Originally Posted by Gavino

How can that work?
ShowAlpha() copies the alpha channel into R, G and B.
Unpremultiply then divides R, G and B by alpha, so you should end up with a mask with all pixels 255.


No, the precision has already been lost in the 8-bit RGB32 input.
The result of the premultiplication needs to be stored in at least 16 bits to preserve precision.

Posted at the same time, I've realized the same thing, look at my images linked above.

[Gavino]

Originally Posted by pancserzso

I don't understand 2 things:
1. How could a 32-bit float operation help here. The result I get with Layer seems really similar to AE's 8-bit mode.
2. Upon looking at the code, it seems to me that alpha should stay intact. However fg.showalpha.unpremultiply() clearly shows different results compared to the normal one. How is this possible?

1. As I said, the entire operation needs to be done in 32 bits (or at least 16). Once you have an 8-bit premultiplied output, precision has been lost.
2. unpremultiply preserves alpha, but mask in Overlay() does not use alpha, it uses the RGB as greyscale.

[pancserzso]

Originally Posted by Gavino

1. As I said, the entire operation needs to be done in 32 bits (or at least 16). Once you have an 8-bit premultiplied output, precision has been lost.

Does it mean that the whole Avisynth operation would need to be 32-bit float? Thus it's not possible with Avisynth no matter what filter we use?

For reference, here is how After Effects looks in 8-bit mode and 32-bit float mode (just with blank background):
8-bit
32-bit float
And the Avisynth output: unpremultiply new

[Gavino]

Originally Posted by pancserzso

Does it mean that the whole Avisynth operation would need to be 32-bit float? Thus it's not possible with Avisynth no matter what filter we use?

I meant that the premultiplication itself has to be both performed and stored in at least 16 bits.
Avisynth by itself can process only 8-bit inputs (although perhaps something could be done with dithertools?).

Originally Posted by sqrt(9801)

this script is supposed to return the overlay layer (basic ColorBars clip fading out via alpha channel) untouched by overlaying it over a blank, transparent clip. It doesn't. It even looks like it changes the RGB values for some reason.

Changing the RGB values is correct, since each channel is set according to the formula:
output = fg*alpha + bg*(1-alpha)
However, the same formula is used for output alpha, which appears to be incorrect.

[wonkey_monkey]

background = ImageSource( "composebug_bg.png", 1, 5600, 24, pixel_type="RGB24" ).ConvertToRGB32()

foreground = ImageSource( "composebug_fg.png", 1, 5600, 24, pixel_type="RGB32" )

foreground.showalpha

m_background=overlay(background,foreground.showalpha,mode="multiply",pc_range=true)
overlay(m_background,foreground,mode="add",pc_range=true)

Any help?

David

[pancserzso]

Originally Posted by Gavino

I meant that the premultiplication itself has to be both performed and stored in at least 16 bits.
Avisynth by itself can process only 8-bit inputs (although perhaps something could be done with dithertools?).

premultiplication itself you mean is what was used to generate the image. You're right, it was 16-bit, but I now converted it to 8-bit and the result is still the same in After Effects.

So, the source image is 8-bit, and somehow it's still possible to have the correct output in AE.

[pancserzso]

Originally Posted by davidhorman

background = ImageSource( "composebug_bg.png", 1, 5600, 24, pixel_type="RGB24" ).ConvertToRGB32()

foreground = ImageSource( "composebug_fg.png", 1, 5600, 24, pixel_type="RGB32" )

foreground.showalpha

m_background=overlay(background,foreground.showalpha,mode="multiply",pc_range=true)
overlay(m_background,foreground,mode="add",pc_range=true)

Any help?

David

Thanks! I've tried it, it does the exact opposite, almost like composing with inverted alpha.

[Gavino]

Originally Posted by davidhorman

m_background=overlay(background,foreground.showalpha,mode="multiply",pc_range=true)

The background needs to be multiplied by '1-fg.alpha', so I think it should be

m_background=overlay(background,foreground.showalpha.invert(),mode="multiply",pc_range=true)

[pancserzso]

Wow, thanks, this is finally it! Perfect!

background = ImageSource( "composebug_bg.png", pixel_type="RGB24" ).ConvertToRGB32()
foreground = ImageSource( "composebug_fg.png", pixel_type="RGB32" )

m_background=overlay(background,foreground.showalpha.invert(),mode="multiply",pc_range=true)

overlay(m_background,foreground,mode="add",pc_range=true)

Actually I started to understand, this is the whole point of the premultiplied alpha, that we can compose images without loss of quality.

[wonkey_monkey]

Originally Posted by pancserzso

Thanks! I've tried it, it does the exact opposite, almost like composing with inverted alpha.

Strange - it's an exact copy/paste of my script, which works fine and looks very close to your AE result to me.

[pancserzso]

Originally Posted by davidhorman

Strange - it's an exact copy/paste of my script, which works fine and looks very close to your AE result to me.

Because my background was an edge case, 99.9% black, thus you couldn't see the difference.

To make a crazy illustration, here is on a rainbow background:

with invert:


without invert:

[wonkey_monkey]

Originally Posted by pancserzso

Because my background was an edge case, 99.9% black, thus you couldn't see the difference.

Ah, yes, of course! Now I think of it, I did turn up my brightness when I first viewed the background image... and then tried to brush the specks of dust off my screen!

[creaothceann]

Originally Posted by pancserzso

background = ImageSource( "composebug_bg.png", pixel_type="RGB24" ).ConvertToRGB32()
foreground = ImageSource( "composebug_fg.png", pixel_type="RGB32" )

m_background=overlay(background,foreground.showalpha.invert(),mode="multiply",pc_range=true)

overlay(m_background,foreground,mode="add",pc_range=true)

Wouldn't Layer be better?

Originally Posted by http://avisynth.nl/index.php/Overlay

The input clips are internally converted to a general YUV (with no chroma subsampling) format

No subsampling is good, but the conversion could still make the results not bit-perfect (and maybe slower).