Overlay discussion (Was: Happy new year..)

[sh0dan]

Hi everybody!

Thanks for a great year!

I have a small gift for you - something I spend my holiday doing: Overlay.

Overlay enables you to overlay video sequences. You can now overlay video with an optional alpha channel. It supports YUY2 and YV12 and has an optional output format. Further information and documentation can be found at avisynth.org (http://www.avisynth.org/index.php?page=Overlay).

I felt it was better to start from scratch rather than continuing work on Layer / Masktools, as it would require rather latge work to make it do what I wanted. This also means that there might still be some bugs left - but just report them here, and I'll look at it.

As usual the binary can be found at the CVS binary site in my signature.

See you all in the next year!

[Richard Berg]

This looks like a great start! I too planned to write the new alpha/layering code from scratch, but logged into the forums just now to give notice that the upcoming semester will be even more time-consuming. I didn't want to see this functionality wait until I graduate in May, so I'm glad to see it get off the ground. I do have a week or two before things get crazy so hopefully I'll be able to review the code and add a few other things I had in mind.

[mf]

Is it teh rofflechoptar optimized with MMX and stuff? :) Great work and happy new year.

[Richard Berg]

Ok, the only thing I think is necessary for the first stage of operations to be complete is an AlphaToMask function. Basically, we should allow people with RGB32 overlay clips the ability to use both the clip data (via ConvertToYUY2 or YV12) and the alpha information directly. All it would take is to copy the alpha into a new YV12 clip. The colorspace conversion on the RGB channels would be lossy which is generally not good when compositing, but we can assume they're only using Overlay when they need to work with YUV in the first place, else Layer would suffice.

It can be a very simple new filter, or perhaps added as new parameter for the (currently not very useful) ShowAlpha that returns greyscale YUV instead of greyscale RGB. I'll be back at my compiler next Tuesday, but there are better MMX programmers than I here so I won't be disappointed if someone beats me to it.

[Zarxrax]

THANK YOU!!!!!!!

:D

Once the AlphaToMask function is in place I can be even happier :D

[sh0dan]

Originally posted by mf
Is it teh rofflechoptar optimized with MMX and stuff? :) Great work and happy new year.
Actually it's not MMX'ed yet. I'd rather have a working implementation before beginning optimizing.

RGB input/output is still a possibility - even though it would require an RGB->YUV (4:4:4) conversion.

[Mug Funky]

OMG! thankyouthankyouthankyouthankyouthankyouthankyou!

sh0dan, i love you :O

i've been wanting a way to layer in yuy2/yv12 without converting.

expect a nice deinterlacer/smartbob to be posted in the usage forum soon :)

happy new year!

[Dali Lama]

I'd like to say thank you for all of your work Sh0dan. It is very much appreciated by all of us.

I guess this means I can do PIP in AVISynth now! Great.

Happy New Year to all,

Dali

[Richard Berg]

In the CVS: ShowMask now returns RGB, YUY2, or YV12 via the pixel_type argument. The latter two can be used to layer an RGB clip with alpha transparency data onto a YUV clip using the 3-argument form of Overlay.

For example:
Overlay(yuy2clip, rgbclip.ConvertToYUY2, mask = rgbclip.ShowAlpha("yuy2"))

RGB output is kept mainly for compatibility; the display should be used as a reference only since the luminance returned is not completely accurate. I.e., don't say something like Overlay(clip, overlay, mask = rgbclip.ShowAlpha.ConvertToYUY2).

I've done some testing as usual but more feedback (once the next CVS binary is posted) is always nice. Also feel free to suggest more features -- I like the thought of Avisynth having good support for simple compositing.

I'll put up a doc page on avisynth.org in a few days unless someone beats me.

[sh0dan]

New CVS binary up.

Changelist for the new year:

- Overlay now accepts RGB24/32 output.
- Added "pc_range=true/false" parameter to overlay. This will make all RGB<->YUV conversions inside Overlay assume [0->255] YUV range, and not CCIR 601 [16->235] range. Default is false.
- ShowMask now returns RGB, YUY2, or YV12 via the pixel_type argument. The latter two can be used to layer an RGB clip with alpha transparency data onto a YUV clip using the 3-argument form of Overlay.
- Fixed very small sample corruption in SSRC.
- SSRC: "fast" is now true per default. This setting is recommended, unless you are doing a big samplerate adjsutment (not just 48000 -> 44100).
- Updated installer. (Thanks to V_ICE for inspiration).
- Re-Added German documentation.
- Fixed audio corruption problem if audio with start < 0 was requested. (introduced in Dec. 30th binary).
- Added RGB input conversion to overlay.

[Zarxrax]

The function seems to be working just fine for me to overlay an RGBA clip onto a YV12 clip. Thank you guys so much for your work!

I have a question about this usage though:
Overlay(yv12clip, rgbclip.ConvertToYV12, mask = rgbclip.ShowAlpha("yv12"))

I based this off the line that Richard wrote, and theres a few things I don't quite understand. Is the ConvertToYV12 necessary here? It seems to work fine without it. And next, what does the "yv12" parameter in ShowAlpha do?

[Richard Berg]

Is the ConvertToYV12 necessary here? It seems to work fine without it.
Not anymore. When I wrote the example I didn't know sh0dan was adding RGB support so quickly.

And next, what does the "yv12" parameter in ShowAlpha do?
This sets the output colorspace of ShowAlpha. This way (1) Overlay doesn't have to do any extra conversions (2) you avoid using RGB, whose luminance is not accurate. To elaborate on the last point since I glossed over it before, here's what happens:


RGB32 input clip: r1 g1 b1 a1 r2 g2 b2 a2
ShowAlpha("rgb"): a1 a1 a1 a1 a2 a2 a2 a2
ShowAlpha("yuy2"): a1 80 a2 80 ...
ShowAlpha("yv12"): a1 a2 ...


ShowAlpha("rgb") shows up as a slightly different image. To illustrate the point, ShowAlpha("rgb").ConvertToYUY2 or YV12 would have luminance values of (r_coef + b_coef + g_coef) * a1 instead of a1.

[sh0dan]

Adding RGB was merely a question of adding conversion routines.

YV12/YUY2 are completely independant. Inputting RGB gives some further aspects - mainly in the area of how (many) conversions are done. I updated the documentation at avisynth.org (http://www.avisynth.org/index.php?page=Overlay) with some things that could be important.



ShowAlpha("rgb") shows up as a slightly different image. To illustrate the point, ShowAlpha("rgb").ConvertToYUY2 or YV12 would have luminance values of (r_coef + b_coef + g_coef) * a1 instead of a1.

.. But if the conversion is done in full scale, (r_coef + b_coef + g_coef) = 1.0.

ShowAlpha("rgb").ConvertToYUY2().ColorYUV(levels="TV->PC") will maintain alpha levels (though probably slightly rounded).

If the mask clip in Overlay is in RGB, the values are not converted at all, so they are passed straigt through. In fact alpha is equal to the blue component (as R=G=B in greyscale RGB images).

[Wilbert]

You are making it very complicated. I'm updating the docs, and would like to know the following.

Are the following three script equivalent:

# script 1:
Overlay(yuy2clip, rgbclip.ConvertToYUY2, mask = rgbclip.ShowAlpha("yuy2"))

# script 2:
mask = rgbclip.ShowAlpha("rgb").ConvertToYUY2.ColorYUV(levels="TV->PC")
Overlay(yuy2clip, rgbclip.ConvertToYUY2, mask)

# script 3:
mask = rgbclip.ShowAlpha("rgb")
Overlay(yuy2clip, rgbclip.ConvertToYUY2, mask, pc_range=true)

[Wilbert]

Stupid question: I guess this opacity option is the same as level (in Layer)?

[sh0dan]

I haven't used Layer much, but I guess it is. It's the term used in Photoshop and AfterEffects, which is why it seemed like a good term.

Btw, mode = "Luma", "Chroma", "Lighten" & "Darken" implemented for next version. :)

[Wilbert]

Sorry to bug you. If I understand it correcly, your mask must always be [0,255]. If you supply a mask in [16,235] (like script 3), it never leads to good results. Correct?

[sh0dan]

Exactly! Mask should be 0 (transparent) to 255 (full blend), so you need to upscale luma or use an RGB mask.

btw, implemented mode="softlight", "hardlight", "difference" & "exclusion", as you might have noticed at avisynth.org

[Wilbert]

Exactly! Mask should be 0 (transparent) to 255 (full blend), so you need to upscale luma or use an RGB mask.
Doesn't that conflict with this statement:

pc_range: When set to true, this will make all internal RGB -> YUV -> RGB conversions assume that luma range is 0 to 255 instead of default 16->235 range.

By default this setting is false, this means that a rgb-mask is converted to YUV [16,235]?

In general all clips are treated as 0->255 values. This means that numbers will not be clipped at CCIR 601 range.
Final question: The base and overlay clips, should they also be [0,255] instead of [16,235]?

btw, implemented mode="softlight", "hardlight", "difference" & "exclusion", as you might have noticed at avisynth.org
Very nice! I didn't commit the corresponding doc updates to sourceforge yet. Because some things are still not clear to me.

[sh0dan]

Originally posted by Wilbert
Doesn't that conflict with this statement:

pc_range: When set to true, this will make all internal RGB -> YUV -> RGB conversions assume that luma range is 0 to 255 instead of default 16->235 range.

By default this setting is false, this means that a rgb-mask is converted to YUV [16,235]?

From the docs
Inputting RGB for mask clip

The mask clip from RGB may behave a bit different than it could be expected. If you always use a greyscale mask, and don't disable greymask you will get the result you'd expect. You should note that mask clip values are never scaled, so it will automatically be in 0->255 range, directly copied from the RGB values.

So in general no - RGB masks are treated as 0-255, and are not scaled. If you input and RGB, RGB=0,0,0 will be transparent, rgb(255,255,255) will be opague.

Final question: The base and overlay clips, should they also be [0,255] instead of [16,235]?

Not in general, no. However doing an mode="additive" for example, overlay should be scaled to 0->255, as lume 16 will add 16 to all luma values in the overlay area.


Very nice! I didn't commit the corresponding doc updates to sourceforge yet. Because some things are still not clear to me.
I hope my avisynth.org updates are enough - just fire away questions. In my experience it is good to have several eyes on the docs.

[Wilbert]

What would be the equivalent of

Layer(clip1,clip2,"fast") # average of two clips

Is it just

Overlay(clip1,clip2,mode="blend")

[sh0dan]

As I've mentioned, I haven't really used layer, but what you suggest will draw clip2 non-transparent on top of clip1 at 0,0 and down.

If you want a half-half blend, Use:

Overlay(clip1,clip2,mode="blend", opacity=0.5)


btw, looked at your example:

mask = rgbclip.ShowAlpha("rgb")
Overlay(yuy2clip, rgbclip.ConvertToYUY2, mask)


Input clips does not have to be same colorspace, so you can actually leave out the converttoyuy2() stuff also!

Added a section in "RGB considerations" about "Using RGB32 alpha channel" in the Overlay docs.

[Richard Berg]

For maintaining maximum quality it is recommended to extract the alpha as RGB.
Why is that? The YUV spaces will always work, but with RGB you have to make sure the conversion is respecting the correct luma range. Depending on how your internal 444 works, using something that's YUV for a mask should be faster too.

[sh0dan]

@Richard: RGB masks are never scaled, so it actually just copies the Blue-values of a pixel as luma in the 4:4:4 image. No actual conversion is done. RGB-Masks are treated different than the baseclip and overlay clip, which are always converted fully.

Using YUV-masks, you risk than it is 16-235 range - you don't risk that with an RGB mask.

[Richard Berg]

Ok, I missed your comment about using the blue channel. Thank you.

[Asrial]

What would be some reasons for using something like this? I'm confused as to the whole purpose of overlaying.

[Wilbert]

What would be some reasons for using something like this? I'm confused as to the whole purpose of overlaying.
- To put a logo on (a part of) your clip.
- To make a nice animated menu.
- To layer two different broadcastings to denoise your clip.
- To remove a logo.

I'm sure there are more applications :)

[Asrial]

Originally posted by Wilbert

- To layer two different broadcastings to denoise your clip.I follow you on all the parts about removing things and adding things but can you explain this one a bit more? How would something like this work?

I have 2 different sources (each one having their own problems) that I put together to create a 3rd source that pulls the good qualities from source 1 and 2?

[Wilbert]

There is a nice thread about it:

http://forum.doom9.org/showthread.php?s=&threadid=28438

Btw, different broadcastings from the "same" videoclip.

[cweb]

Here's a small example of removing a rectangular logo (replacing it with a black rectangle):

black=BlankClip(video, width=66,height=32)
video=video.overlay(black, x=431,mode="blend",opacity=1)


Thanks for 'Overlay' - it saved me having to use logo/xlogo etc...