Chroma Key for overlaying video?

[zilog jones]

I know about the Layer and ColorKeyMask filters, but don't really understand how to use them. I'm pretty new to AviSynth, and I'm a bit confused by the info on these particular filters.

Basically, I have two AVIs (same sizes/framerates to make things easy) - one of them is just some sort of video content, the other is some simple animations and stuff created with, for example, Flash, with a green background. All I want to do is do a kind of chroma key thing with the green on the second video, so the stuff from Flash is overlayed onto the other video content.

Please could someone help me making a script to do this? Do I have to have to make the green in the second video transparent (in RGB32 of course) somehow?

I'm not too bothered about the mediocre quality of chroma-keying for these kind of purposes.

Also, please don't advise me to use some other program that will easily overlay text onto video - I'm just using Flash as an example of a program that has crap video importing functions - there's other programs I want to be able to source overlaying video from, too.

Thanks for any help!

[zilog jones]

Ohh, I've tried getting something like this to work using things like Overlay and Layer for ages, and all I usually get is either just one of the input sources (either just the background or the overlay video), or just the two added (or whatever mode I tried) together.

Please please please could someone tell me how to do a green-key kind of thing?

I also tried this:

AVIsource("g:\video\overlay.avi")

ConvertToRGB32()

ColorKeyMask($00ff00, 10)

showalpha()

(the video is some red animated text over a green background, which is supposed to be 00ff00). All I get with this is black though. What am I doing wrong?

[rfmmars]

I use a program called MEP2004 by Magix. It is a super breeze to do it. It is a current topic now, here's the link.

http://support.magix.net/boards/magix/index.php?showtopic=12335

richard
www.photorecall.net

[Mug Funky]

bam!


function greenkey(clip fore, clip back, int "low", int "high", float "gamma", bool "interlaced", bool "show")
{
interlaced=default(interlaced,true)
low=default(low,96)
gamma=default(gamma,1)
high=default(high,212)
show=default(show,false)

fore = (interlaced==true)? fore.separatefields() : fore
back = (interlaced==true)? back.separatefields() : back

green1 = overlay(fore.vtoy(),fore.utoy(),mode="multiply").coloryuv(autogain=true)
green = green1.levels(low,gamma,high,0,255,coring=false).bicubicresize(fore.width,fore.height,0,.5)
overlay(back,fore,mask=green)

interlaced==true? last.weave() : last

show==true? green1 : last

}


usage:

greenkey(fore,back,low,high,gamma,interlaced,show)

low, high and gamma are values for the mask's levels. verify these with "show=true", and copy levels values from virtualdub or something else with that function.

defaults are based on a poorly-lit green painted screen with a girl dancing in front of it that i had lying around on my machine... if you have cleaner video, then you'll want to set your own values.

if you're using interlaced video, then use "interlaced=true", or it will b0rk the masking on YV12 sources.

convert to yuy2 or yv12 first! this doesn't support RGB.

[edit]
ouch... typos in the script. fixed now
[edit 2]
changed it a bit - coloryuv(autogain=true) is pretty useful - means you don't need to keep tweaking levels :)


[edit 3]
embarrasing typo fixed. interlaced=false will work now.

changed gamma to a float

[zilog jones]

Ooh, just what I was looking for - well actually a lot more! Thanks!

So I just put the function right into an AVS script and then just use "greenkey" as I would a normal filter? Sorry, I'm still quite new to all this AVISynth stuff...

Well my source (at the moment) is totally digital, so it'll be (well it *should* be unless the colour conversion went weird somewhere) exactly 00ff00 green.

Can I just run ConvertToYUY2 filters before this in the same script? The two main animation programs I use - Cinema4D and Flash - only output in RGB. Huffyuv crashes if I try anything else.

This will be even more useful when one of the college societies I'm in gets a video camera next year! Heheheh - low budget special effects-o-rama! All I need now is a decent size hard disk... and a half-decent PC at that (it took me 3 hours to resize and encode SEVEN MINUTES of SVCD video)...

Thanks again!

MEP 2004 does sound intriguing, but I'm a (smelly) student and this way is $100 cheaper!

[Mug Funky]

C4D supports alpha channels doesn't it? as does huffy.

in this case you'll get better mattes out of C4D if you just render alpha.

if huffy gives you grief with alpha channels, then use corePNG (search... hehehe). this basically saves PNG to an avi - very good for graphics. i'd expect it to outperform huffy for Flash output.

alpha channels can be accessed in avisynth with the "layer" set of functions. the docs are very useful (my memory is so bad i've always got them open while scripting).

the above script is for natural video, but i'm sure you and your student friends will find a use for it :):):)

i love student films. you should see the crap that i've made :devil:

btw, what's Cinema4d's rendering like? last i heard it had an early version of FinalRender in it. saw some impressive pics from it. i use 3dsMAX myself, but haven't the money for fancy renderers just yet (i'd luuurve to get Brazil r/s)

[edit]
wrong use of "your/you're". you'd think i know english by now

oh! to use the function, whack it in your script (usually at the top, but i don't think it matters) and call it's name with arguments. just like a real programming language :)

better still (i'm lazy), you can put it in it's own text file called "greenkey.avsi" and place it in the avisynth plugins directory. this way it'll be loaded on startup, and you can call it from any avs script.

[zilog jones]

I haven't really experimented much with alpha channels yet - I think C4D does have them... somewhere...

C4D's rendering looks good enough - it can do all the fancy things like anti-aliasing, motion blur, refraction, and even cel-shading. I have a pretty old version though - I got it free with some magazine about 3 years ago, so the actual version is at least 4 years old - I think it's 5? It can do field rendering though, which is very nice for making half-decent looking animation for TV use, though it takes a while to render on my crappy PC!

Yeah, Flash's AVI output is dodgy at best - most codecs die if I use any sort of bitmaps in the animation.

I tried your function yesterday, but was most unsucsessful. I had two videos - both 320x240 @ 25fps, Huffyuv YUY2, and of course one with a green background (a and b in the script) - both crappy Flash animations that I knocked up in about 20 minutes (one's just a load of moving squares, the other is just "this is a test" in red, fading in and out on green). The script was something like this:

a= AVISource(<filename>)
b=AVISource(<other filename>)

function greenkey(clip fore, clip back, int "low", int "high", int "gamma", bool "interlaced", bool "show")
{
interlaced=default(interlaced,true)
low=default(low,96)
gamma=default(gamma,1)
high=default(high,212)
show=default(show,false)

fore = (interlaced==true)? fore.separatefields() : fore
back = (interlaced==true)? back.separatefields() : fore

green1 = overlay(fore.vtoy(),fore.utoy(),mode="multiply").coloryuv(autogain=true)
green = green1.levels(low,gamma,high,0,255,coring=false).bicubicresize(fore.width,fore.height,0,.5)
overlay(back,fore,mask=green)

interlaced==true? last.weave() : last

show==true? green1 : last

}

greenkey(a, b, interlaced=false)

I open this with VirtualDubMod, but just get one of the videos as normal - if I'm swap around the videos, it's just the other video displayed normally. Which ones do you mean by fore and back exaclty? My logic's not good in general, so sorry for sounding like a complete idiot, but I assume the "fore" is the one with the green...? I tried both ways round anyway.

I also tried putting low=0 and high=255, but still just one of the videos is displayed. I tried show=true (though I'm not quite sure what this does), but it just displays a white 160x240 video.

I also tried converting them to YV12 video before all this, but in VitrualDub it just displayed green noise with some occasional magenta noise - it wanted to decode it with DivX 5, which I assume is what caused this mess.

What am I doing wrong? It's probably something really stupid, but what?

[Mug Funky]

hmm.... try put the function before the avisource statements. it should work.

oh, and put "converttoyv12()" after the avisources. so:


function greenkey(clip fore, clip back, int "low", int "high", float "gamma", bool "interlaced", bool "show")
{
interlaced=default(interlaced,true)
low=default(low,96)
gamma=default(gamma,1)
high=default(high,212)
show=default(show,false)

fore = (interlaced==true)? fore.separatefields() : fore
back = (interlaced==true)? back.separatefields() : back

green1 = overlay(fore.vtoy(),fore.utoy(),mode="multiply").coloryuv(autogain=true)
green = green1.levels(low,gamma,high,0,255,coring=false).bicubicresize(fore.width,fore.height,0,.5)
overlay(back,fore,mask=green)

interlaced==true? last.weave() : last

show==true? green1 : last

}

a= AVISource(<filename> ).converttoyv12()
b=AVISource(<other filename> ).converttoyv12()

greenkey(a, b, interlaced=false)


[edit]

changed above script... bloody stupid bug meant it was overlaying the foreground onto the foreground when interlaced=false.

also changed gamma to a float... kinda obvious.

[zilog jones]

Oh, I did the converttoYV12 in seperate scripts altogether. But I thought YUY2 works too? That's what I tried when I just got one of the videos as the output.

Surely it doesn't matter where I put the function assignment thing as long as it's before where I actually use it? It would have came up with an error not recognising "greenkey" if the function assignment didn't work?

I'll try it again this evening (I'm in college now, and live at the other side of town which is quite annoying).

Is there anything else I haven't considered? Am I doing the actual opening part right? I just open the file directly with VirtualDubMod ad play it straight from there. I've also tried playing straight from Media Player Classic, but it appears to do the same thing.

I'm also using AviSynth 2.5.4 - anything wrong with that?

[UPDATE]I tried the script *exactly* as you said (well, with the real filenames in it of course), but it still just shows the foreground video with no changes.

[Mug Funky]

argh! you're right.

serves me right for only testing the interlaced mode.

it'll work in yuy2, and interlaced and progressive are the same this way, so until i find an incredibly obvious (but elusive) bug, just use interlaced=true.

i'm kinda embarrassed about that. hehe.

[edit]

CUT AND PASTE ERROR!!!!!

argh. my bad. it was masking the foreground... and overlaying the foreground. 1 word. above scripts edited (wait 30 sec after this post...)

[zilog jones]

That will *probably* fix things then! I'll try it tonight, but if it doesn't work I may have to kill someone! The only other way I've found out to do this without spending an absurd amount of money on an Adobe product is by using Zwei-Stein, the video editor with the most evil interface EVER!

So it should work in progressive too, then? I do like my interlaced video anyways, though. What script do you use for making fields? Someone told me to do:

AVISource("q:\blahhhhhh")
SeparateFields()
SelectEvery(4,1,2)
Weave()
ConvertToRGB()

But I'm not sure if this is the best way to do it. Is it? It's better than my original way of interlacing - mapping a video as a texture onto a flat square in Cinema 4D and just rendering the fields from that!

[smok3]

Mug Funky, 1st sorry for bringing up this old thread...

and this is working pretty fine indeed, but it does miss some features/parameters like:

1. bluring the matte
2. making the matte to look a bit more soft (vectorized), maybe some EDI method could be used somehow?
3. removing the remaining green cast from front layer (some sort of channel specific 'color correction'...of some kind..).
4. filling small black portions of the matte (that would be a cool one!) true/false would do it fine :)
5. using garbage mattes (pretty much impossible via scripting i guess..)
6. basic brightness/contrast correction on the matte.

p.s. i can deliver some testing material (it is CG) encoded with fraps.

[Mug Funky]

that's a fair bit to ask out of a proof-of-concept chroma keyer... :)

here's what i use for correction/chroma key/secondaries, etc:

http://www.assimilateinc.com/

little bit expensive though :)

on a serious note though, all that stuff could be implemented but it'd take a lot of messing around. i'd also like to be able to key based on HSV values. i don't know how to do that with current avisynth, but i've got a couple of ideas.

i might give this a shot when i get some spare time. i like the idea of building a full on colour grading solution in avisynth (then sitting back and waiting for more than 8 bit support).

[smok3]

well, not a request, more like 'loud thinkering', will work on it a bit today and report back.

blur should be easy if there is a stronger bluring filter? This doesn't do much:
mask=green.blur(1.58)

edit: well, besides blur iam getting nowhere, whatever i try to do is changing the matte in weird ways (levels get completely screwed...)

[smok3]

question: how would i trim output to the length of front klip?

nm, this 'patch' seems to work:
...
overlay(back,fore,mask=green.blur(1.58))

interlaced==true? last.weave() : last.Trim(0,framecount(fore))

show==true? green1 : last.Trim(0,framecount(fore))
...

[Mug Funky]

just for fun, here's a couple of functions to return Hue and Saturation. i pulled the maths for these out of my arse, but it seems to work.

saturation here is defined as the distance from grey on a vectorscope. this is found by getting the U and V channels and doing pythagoras on them.

hue is defined as red = 0 degrees, degrees going from 0-255 (well it's the only way i could fit it into 8 bit space...). when i write some secondary colour correction stuff, i'll need to rescale the inputs to degrees obviously. primary colours are 120 degrees apart, but in this case they're 256/3 values apart. there's a hard black-to-white transition in hue, which is a consequence of the maths. think of it as when red flips around the other side to magenta. again when i script some colour correction tools, this shouldn't be a problem as the numbers will wrap around.

luminance is the same as the input clip.greyscale().

function ShowHue (clip c)
{
hues=c.tweak(hue=108.5)
mt_lutxy(hues.utoy(),hues.vtoy(),expr="y 128 - 128 / x 128 - 128 / / atan 81.487 * 128 +")
}

function ShowSat (clip c)
{
hues=c.tweak(hue=108.5)
mt_lutxy(hues.utoy(),hues.vtoy(),expr="x 128 - 128 / 2 ^ y 128 - 128 / 2 ^ + .5 ^ 256 *")
}

[Wilbert]

i'd also like to be able to key based on HSV values. i don't know how to do that with current avisynth, but i've got a couple of ideas.
It's pretty easy with avs 2.6 (Tweak/MaskHS).

[Mug Funky]

d'oh! so i spent a good hour writing the above script and all i had to do was install avs 2.6? hehe.

srlsly though, HSV keying is so much nicer than what you'd get out of AE's normal keying.

[NerdWithNoLife]

I have a way of doing this now, using the free program CineGobs Keyer. For me the output is 320x240 internet video so I just separate the fields, select one of them, resize it, and work with that as a progressive image. The script for CineGobs to key would be simply:

avisource("footage.avi").separatefields().selecteven().bicubicresize(320,240).converttorgb32()

I have CineGobs output an alpha only video (key.avi) and save it with the Lagarith lossless codec. It doesn't take much space because it's greyscale. I configure Lagarith to output RGBA. Then, I put it all together with this script:

#load clips
footage=avisource("footage.avi").separatefields().selecteven().bicubicresize(320,240).converttorgb32()
key=avisource("key.avi")
slug=blankclip(footage)

#you can use a still image or a moving clip; make sure it has the same
#properties as your footage and uncomment either of the next 2 lines:
#bg=imagesource("bg.png",pixel_type="rgb32").loop()
#bg=avisource("bg.avi").loop()

#this step makes sure your clip won't loop forever
bg=layer(slug,bg,"add",255)

#mask the footage with the key created by CineGobs Keyer
footage=mask(footage,key)

#layer and redub the audio
video=layer(bg,footage,"add",255)
audiodub(video,footage)

And that should output your keyed video. You may have to tinker with CineGobs for a while, but it does a good job (I think) and is highly customizable. Also good lighting is key, when filming the footage. However, the difference keyer in CineGobs can compensate for an unevenly lit green screen.

Again, I'm making an internet video so you may have to modify the script to work for your resolution.