I'd recommend
rgba_rpn for this kind of thing, but the learning curve is somewhat precipitous and I've pretty much rewritten the entire thing since then so even I may not be of much help...
Actually masktools or even just Avisynth+'s expr might be better.