Layer and Overlay operation giving wrong results with RGBA image source

[pancserzso]

I would like to compose two image sources on each other, using alpha blending.

My problem is that:
1. If I just simply load the image with imagesource, the result is OK (I think it just discards the alpha channel)

2. However as soon as I do an Overlay or Layer operation, Avisynth seems to handle the alpha of the image wrong.

3. I believe the problem is pre-multiplied vs. post-multiplied alpha. I found a filter especially for fixing this, avisynth-unpremultiply (https://code.google.com/p/avisynth-unpremultiply/), however it made a totally crazy buggy colors with rainbow like blends and other bugs.

Is there any way to tell AviSynth to handle the alpha in my foreground images properly?

Here is my bg image:
https://dl.dropbox.com/s/fdq7hfm679hbmh2/composebug_bg.png
(https://dl.dropbox.com/s/fdq7hfm679hbmh2/composebug_bg.png)
My foreground image:
https://dl.dropbox.com/s/t5c8rt0psu0l56o/composebug_fg.png
(https://dl.dropbox.com/s/t5c8rt0psu0l56o/composebug_fg.png)
How it looks in AviSynth:
https://dl.dropbox.com/s/h7412jh6e81t1pl/composebug_avisynth.png
(https://dl.dropbox.com/s/h7412jh6e81t1pl/composebug_avisynth.png)
And how it should look, as when composed in After Effect, and selecting "premultiplied" for the foreground's alpha:
https://dl.dropbox.com/s/xb9igdqd8yyby4y/ae_00000.png


This is the Avisynth script I'm trying to use (in 2.60 b4):

background = ImageSource( "e:\render_outside\draft%05d.png", 1, 5600, 24, pixel_type="RGB24" ).ConvertToRGB32()

foreground = ImageSource( "e:\render_mixer\image%05d.png", 1, 5600, 24, pixel_type="RGB32" )

Layer( background, foreground )

[sqrt(9801)]

Funny, I was thinking of making a thread for the same problem.

However, I'm not sure whether it is a matter of pre/post-multiplied alpha. To be honest, I didn't even consider the possibility before you mentioned it, mostly because I was thinking it would be caused by something like a RGB <-> YUV conversion.

For instance, this script is supposed to return the overlay layer (basic ColorBars clip fading out via alpha channel) untouched by overlaying it over a blank, transparent clip. It doesn't. It even looks like it changes the RGB values for some reason.

ColorBars(width=640, height=480, pixel_type="RGB32")
KillAudio
Trim(0, length=255)

MergeARGB(BlankClip(last, color=$FF000000).FadeOut(framecount-1, color=$00000000),
\ ShowRed,
\ ShowGreen,
\ ShowBlue)

top_layer=last

Layer(BlankClip(last, color=$00000000), last, op="add", level=257)

Compare(top_layer, channels="RGBA", show_graph=false) # try with channels="R", "G", "B" and "A"

I've also tried with Panzerboy's Gimp-style Layer merge plug-in (http://sourceforge.net/projects/avisynthgmplyr), no dice. :confused:

[IanB]

Yes the alpha mask scaling is incompatible.

See this thread, "Alternative to the old Unpremultiply() plugin? (http://forum.doom9.org/showthread.php?p=1608283#post1608283)", for a fixed version of the unpremultiply plugin and discussion on alternatives (lutxy etc).

[pancserzso]

Thanks IanB,

I've tried the linked version, and while it is indeed better, it's still way off from the real solution.
https://dl.dropbox.com/s/lat7axfw9ingoh5/composebug_unpremult_new.png

Actually, this image looks really similar to what After Effects looks when I'm in 8 bit mode. I had to put After Effects to 32 bit floating point mode to get the correct results.

Which is strange to me! Do pre-multiplying really need 32-bit floating point operation?

[poisondeathray]

Thanks IanB,

I've tried the linked version, and while it is indeed better, it's still way off from the real solution.
https://dl.dropbox.com/s/lat7axfw9ingoh5/composebug_unpremult_new.png

Actually, this image looks really similar to what After Effects looks when I'm in 8 bit mode. I had to put After Effects to 32 bit floating point mode to get the correct results.

Which is strange to me! Do pre-multiplying really need 32-bit floating point operation?

Way off ? It think if will be closer if you use showalpha() first, before using unpremultiply , but even then there are some differences in the little specs



fg=ImageSource("composebug_fg.png", pixel_type="rgb32")
bg=ImageSource("composebug_bg.png", pixel_type="rgb32")
overlay(bg, fg, mask=fg.showalpha.unpremultiply)

[pancserzso]

Way off ? It think if will be closer if you use showalpha() first, before using unpremultiply , but even then there are some differences in the little specs



fg=ImageSource("composebug_fg.png", pixel_type="rgb32")
bg=ImageSource("composebug_bg.png", pixel_type="rgb32")
overlay(bg, fg, mask=fg.showalpha.unpremultiply)



Update!

OK, I'm puzzled for many reasons:

1. The method poisondeathray looks good, however it's just a coincidence. Upon looking at the mask, it's clear that we are pretty much discarding all the grayscale information and making a binary mask. Thus, the background is _totally_ not visible behind those areas.
Look:
normal alpha (https://dl.dropbox.com/s/f9n10fcmw7vgbmq/composebug%20normal%20alph.png)
unpremultiplied's alpha (https://dl.dropbox.com/s/z4yqbokgyhisiuh/composebug%20unpre%20alpha.png)

2. Upon looking at the code, I'm even more confused.

I don't understand 2 things:
1. How could a 32-bit float operation help here. The result I get with Layer seems really similar to AE's 8-bit mode.
2. Upon looking at the code, it seems to me that alpha should stay intact. However fg.showalpha.unpremultiply() clearly shows different results compared to the normal one. How is this possible?

From cpp:
const int srcapix = srcp[w+3] ;
dstp[w+0] = lookup[ (srcp[w+0]<<8) + srcapix ];
dstp[w+1] = lookup[ (srcp[w+1]<<8) + srcapix ];
dstp[w+2] = lookup[ (srcp[w+2]<<8) + srcapix ];
dstp[w+3] = srcapix;

[Gavino]

overlay(bg, fg, mask=fg.showalpha.unpremultiply)
How can that work? :confused:
ShowAlpha() copies the alpha channel into R, G and B.
Unpremultiply then divides R, G and B by alpha, so you should end up with a mask with all pixels 255.

BTW, it's still interesting that it looked the same as when AE was in 8-bit mode. Maybe by using floats for the calculation in Unpremultiply we could have the same effect.
No, the precision has already been lost in the 8-bit RGB32 input.
The result of the premultiplication needs to be stored in at least 16 bits to preserve precision.

[pancserzso]

How can that work? :confused:
ShowAlpha() copies the alpha channel into R, G and B.
Unpremultiply then divides R, G and B by alpha, so you should end up with a mask with all pixels 255.


No, the precision has already been lost in the 8-bit RGB32 input.
The result of the premultiplication needs to be stored in at least 16 bits to preserve precision.

Posted at the same time, I've realized the same thing, look at my images linked above.

[Gavino]

I don't understand 2 things:
1. How could a 32-bit float operation help here. The result I get with Layer seems really similar to AE's 8-bit mode.
2. Upon looking at the code, it seems to me that alpha should stay intact. However fg.showalpha.unpremultiply() clearly shows different results compared to the normal one. How is this possible?
1. As I said, the entire operation needs to be done in 32 bits (or at least 16). Once you have an 8-bit premultiplied output, precision has been lost.
2. unpremultiply preserves alpha, but mask in Overlay() does not use alpha, it uses the RGB as greyscale.

[pancserzso]

1. As I said, the entire operation needs to be done in 32 bits (or at least 16). Once you have an 8-bit premultiplied output, precision has been lost.


Does it mean that the whole Avisynth operation would need to be 32-bit float? Thus it's not possible with Avisynth no matter what filter we use?

For reference, here is how After Effects looks in 8-bit mode and 32-bit float mode (just with blank background):
8-bit (https://dl.dropbox.com/s/vbv2q6glf7zwfdh/composebug_ae8bit_00000.png)
32-bit float (https://dl.dropbox.com/s/qbkv0io3nrxygpa/composebug_ae32float_00000.png)
And the Avisynth output: unpremultiply new (https://dl.dropbox.com/s/lat7axfw9ingoh5/composebug_unpremult_new.png)

[Gavino]

Does it mean that the whole Avisynth operation would need to be 32-bit float? Thus it's not possible with Avisynth no matter what filter we use?
I meant that the premultiplication itself has to be both performed and stored in at least 16 bits.
Avisynth by itself can process only 8-bit inputs (although perhaps something could be done with dithertools?).

this script is supposed to return the overlay layer (basic ColorBars clip fading out via alpha channel) untouched by overlaying it over a blank, transparent clip. It doesn't. It even looks like it changes the RGB values for some reason.
Changing the RGB values is correct, since each channel is set according to the formula:
output = fg*alpha + bg*(1-alpha)
However, the same formula is used for output alpha, which appears to be incorrect.

[wonkey_monkey]

background = ImageSource( "composebug_bg.png", 1, 5600, 24, pixel_type="RGB24" ).ConvertToRGB32()

foreground = ImageSource( "composebug_fg.png", 1, 5600, 24, pixel_type="RGB32" )

foreground.showalpha

m_background=overlay(background,foreground.showalpha,mode="multiply",pc_range=true)
overlay(m_background,foreground,mode="add",pc_range=true)

Any help?

David

[pancserzso]

I meant that the premultiplication itself has to be both performed and stored in at least 16 bits.
Avisynth by itself can process only 8-bit inputs (although perhaps something could be done with dithertools?).

premultiplication itself you mean is what was used to generate the image. You're right, it was 16-bit, but I now converted it to 8-bit and the result is still the same in After Effects.

So, the source image is 8-bit, and somehow it's still possible to have the correct output in AE.

[pancserzso]

background = ImageSource( "composebug_bg.png", 1, 5600, 24, pixel_type="RGB24" ).ConvertToRGB32()

foreground = ImageSource( "composebug_fg.png", 1, 5600, 24, pixel_type="RGB32" )

foreground.showalpha

m_background=overlay(background,foreground.showalpha,mode="multiply",pc_range=true)
overlay(m_background,foreground,mode="add",pc_range=true)

Any help?

David

Thanks! I've tried it, it does the exact opposite, almost like composing with inverted alpha.

[Gavino]

m_background=overlay(background,foreground.showalpha,mode="multiply",pc_range=true)
The background needs to be multiplied by '1-fg.alpha', so I think it should be
m_background=overlay(background,foreground.showalpha.invert(),mode="multiply",pc_range=true)

[pancserzso]

Wow, thanks, this is finally it! Perfect!

background = ImageSource( "composebug_bg.png", pixel_type="RGB24" ).ConvertToRGB32()
foreground = ImageSource( "composebug_fg.png", pixel_type="RGB32" )

m_background=overlay(background,foreground.showalpha.invert(),mode="multiply",pc_range=true)

overlay(m_background,foreground,mode="add",pc_range=true)

Actually I started to understand, this is the whole point of the premultiplied alpha, that we can compose images without loss of quality.

[wonkey_monkey]

Thanks! I've tried it, it does the exact opposite, almost like composing with inverted alpha.

Strange - it's an exact copy/paste of my script, which works fine and looks very close to your AE result to me.

[pancserzso]

Strange - it's an exact copy/paste of my script, which works fine and looks very close to your AE result to me.

Because my background was an edge case, 99.9% black, thus you couldn't see the difference.

To make a crazy illustration, here is on a rainbow background:

with invert:
http://i.imgur.com/R4uWLjY.jpg

without invert:
http://i.imgur.com/5eidWeu.jpg

[wonkey_monkey]

Because my background was an edge case, 99.9% black, thus you couldn't see the difference.

Ah, yes, of course! Now I think of it, I did turn up my brightness when I first viewed the background image... and then tried to brush the specks of dust off my screen!

[creaothceann]

background = ImageSource( "composebug_bg.png", pixel_type="RGB24" ).ConvertToRGB32()
foreground = ImageSource( "composebug_fg.png", pixel_type="RGB32" )

m_background=overlay(background,foreground.showalpha.invert(),mode="multiply",pc_range=true)

overlay(m_background,foreground,mode="add",pc_range=true)
Wouldn't Layer be better?

http://avisynth.nl/index.php/Overlay[/url]"]The input clips are internally converted to a general YUV (with no chroma subsampling) format
No subsampling is good, but the conversion could still make the results not bit-perfect (and maybe slower).

[Gavino]

Wouldn't Layer be better?

No subsampling is good, but the conversion could still make the results not bit-perfect (and maybe slower).
Layer() is certainly to be preferred in general to Overlay() when operating on RGB clips, as it is faster, more accurate and uses less memory.

But the original problem in this thread was that Layer doesn't deal with premultiplied alpha, hence the search for an alternative solution. In the method you quoted, Overlay() can't be replaced by Layer() because Layer() doesn't have an 'add' mode. (Actually, it has something called 'add', but it's equivalent to Overlay's 'blend' mode.) In addition, the documentation suggests Layer's 'mul'(tiply) mode works only with YUY2.

[wonkey_monkey]

I don't think Layer's add and mul mode works the same way as Overlay's.