Idea for speeding up masked processing with AVISynth

[scharfis_brain]

we all know, that if masks are used, a lot of CPU Cycles are wasted for calculating invisible pixels, that are getting dicharged due to masking.

how about a general/global avisynth enhancement, that gives filters at least two input sources.
the input-clip itself and a binary mask, that descibes, which pixels has to be calculated for the output. The remaining parts of the image may stay black or unprocessed(copied input).

I could imagine a syntax like this:

#pseudocode
mask=input.motionmask()
blurred=input.blur(mask=mask)
overlay(input,blurred,mask=mask)

this would only calculate the needed pixels, all other pixels are unprocessed, thus reducing usage of CPU.

this wouldn't need big changs on existing scripts, because the mask is already existant in most scripts before the filtering itself is done.

would this technique be possible to be implemented into avisynth and external plugins?

It would be really great if it could be done.

If the current architectur of avisynth doesn't allow that, would it be possible to implement it into AVISynth3 (is this planned jet?)

[mf]

I don't think many filters have a way of processing that allows for much optimization in this field. What you say would only work with binary masks (= baad) or soft masks that have a lot of pure black, and even then it would stand in the way of processor optimizations.

[scharfis_brain]

and even then it would stand in the way of processor optimizations.

pre: I do not understand anything about optimizations.

but what if the masked areas are blown up to mod4 or mod8 sizes (a near vertical thin diagonal line will become thick and stairsteppy), instead of using the fine detailed must, just process a little bit more but have the optimization used...

[Manao]

scharfis_brain : it would not work.

Firstly, as mf said, it would work only on binary mask, but that's not really an issue for some kind of filter / mask.

But, secondly, you would not be able to use MMX / iSSE optimizations, because such optimizations require to be able to process several pixels in a row.

And finally, you would not be able to use optimized algorithms ( the typical example is the computation of a convolution with a matrix filled with ones. the basic algorithm needs NxN additions by pixels ( where NxN is the size of the matrix ), whereas another algorithm allows you to make only a few additions ( around 4 I guess ). But the second algorithm needs the pixels to be process in a particular order, which would not be possible with your masked filtering ).

That was for the speed part : the conlusion is that you may not gain speed using masked filtering.

Now, for the implemention part, it would need to rewrite all the filters that would take a mask as an argument, but it would be possible in the current architecture ( maskedmerge, overlay are already working, and it's the same principle )

[bill_baroud]

sound like a task for modern GPU :D

[stickboy]

Originally posted by scharfis_brain
this would only calculate the needed pixels, all other pixels are unprocessed, thus reducing usage of CPU.Not necessarily... wouldn't each filter need to branch on every pixel to check whether that pixel is masked or not?

It'd be even worse for any kind of temporal or spatial smoother, since you'd need to check all pixels within the smoothing radius.

[sh0dan]

To sum up:

* Branching. Branching on random data is bad. All modern CPU's suffer greatly on a mispredicted branch. Having a per-pixel branch is very bad. Especially P4's die on this due to the long pipeline, but Athlons are also negatively affected.

Having an "if (mask >0) {}" is usually slower than doing the actual calculation, if mask is > 0 more than 75% of the time. Furthermore branching on MMX data is very hard to do (and also leads to futher stall issues).

* MMX. MMX operates on 8 pixels or more per loop. You need to test all 8 pixels - and again the branching issue or the MMX to GPR (General Purpose Register) problem kick in.

* Out-of-place filters. Some filters require all data to be processed since they need write to a different buffer than they write to. Blur is an example of this IIRC.

* Memory. Most filters are usually memory-limited. This means, the CPU is waiting for data to be read from memory. But memory is read in 64byte blocks, so you need to be able to skip 64 pixels in a row before you get any gain there.

The only way to get speedups (assuming code is already MMX/SSE2) is to lower the amount of data (YV12 vs RGB for instance), or simplifying the alogrithm. YV12 has the ability to be able to skip entire planes, which MaskTools is capable of for instance. I'm actually considering doing the same with Overlay - but since it also supports other colorspaces it's a bit more painful.

[Mug Funky]

hmm. that's a pity.

but i suppose it makes sense - adaptive oversampling in 3d rendering (raytracing etc) is only effective when there's a very small amount that needs to be supersampled.

i couldn't count the number of times i've set a renderer like Brazil r/s to simply antialias the entire image rather than distinguish what needs antialiasing. some sources just require too much AA, and it's quicker to do the lot.

2d processing must have even more problems in this arena i'd imagine. to say nothing of the MMX stuff (kinda goes right over my head).