avisynth replacement for ColorKeyMask, ResetMask, Overlay, Layer ...

[george84]

In vapoursynth docu it says

Do the adjustment yourself for above functions. After halfway understandig lut I could maybe succeed. And for Overlay I found a replacement.

Maybe somebody has already done it himself? So it would just save me some time if anybody had ResetMask, Layer, and ColorKeyMask functions

[poisondeathray]

For layer and overlay couldn't you just use Merge / MaskedMerge ?

Not sure why anyone would use avisynth or vapoursynth for keying . It's just painful with terrible results. There are much better free open source alternatives on every OS . eg. Blender, Natron . Even the limited free versions of commercial software have much better keying, like hitfilm, nuke . Complete solutions with roto/masking/tracking/spill removal

[poisondeathray]

I suppose some useful functions would be the layer blend modes in overlay and layer and rr42 added some additional photoshop modes

http://avisynth.nl/index.php/UU_mt_blend
https://forum.doom9.org/showthread.php?t=170490

I don't know if anyone has "translated" it to vapoursynth

[Myrsloik]

The main reason it's not included is that there are so many different blend modes.

ResetMask=>BlankClip
ColorKeyMask=>Expr("x thr tol - > x thr tol + <") where thr is the threshold and tol is the tolerance
Layer is just MaskedMerge (add)
MaskedMerge with one input inverted (sub)
And some Expr string of questionable value for the remaining modes like "x y max" and "x y min" for lighten and darken (and some threshold constants).

As usual you're probably better off saying what your input is and what you want your output to look like after all processing is done if you want a better solution.