Idea: YToRGB()

[mf]

With all the mask stuff and colorspace conversions, I've come up with an idea. When I have a greyscale mask (or want to make one), I need to ConvertToRGB32() which is slow and lossy. Now my idea is to simply copy the Y channel into the R, G and B channels (not caring about green weight or clamping or all that stuff), so I have a greyscale RGB image. The logical road back of course is either RToY() (if it's greyscale all RGB channels will have the same value) or RGBToY(), which would just copy the average values of R, G and B into Y (more processor intensive than just copying R to Y). I don't care about color reproduction or anything like that, just internal processing so I can use the alpha channel until AVISynth has AYUV support (will it ever?). Maybe something like CopyToRGB(), which would just double the UV sizes and copy Y U and V into R G and B might also be interesting. A CopyToYV12() would be needed as well then of course :).

Any thoughts?

[geoffwa]

Surely it's simpler/saner to rewrite function to work in YUV space (particularly if we're dealing with greyscale stuff here) than to start 'arbitrary' RGB implementations.

:confused: (missing the point here)

[mf]

Originally posted by geoffwa
Surely it's simpler/saner to rewrite function to work in YUV space (particularly if we're dealing with greyscale stuff here) than to start 'arbitrary' RGB implementations.

:confused: (missing the point here)
How do you want to handle transparency in colorspaces that don't have an alpha channel? You could add the AYUV colorspace to AVISynth, but that seems to be too big of a hassle (or at least not nr.1 priority right now) as I've already suggested it before.

[Kurosu]

That's what I tried to propose to Avisynth community with Masktools, without success. I hadn't time to put up part of the newer versions (for my very own purpose) which consists in a YV12 layer function, but I'll try to take time to do it (considering mfToon is 2 times to 3 times faster without the supersampled sharpening by using it).

[Bidoche]

@mf

YUVA is not a problem, I can do it easily for the 3.0
I can make a GrayScale colorspace too, it should come in handy to handle masks, no ?

In fact anything you want is possible (motions vectors, different precision, planar RGB, interleaved YUV, YUV 4:4:4)...
Just ask for what you want/need.

[morsa]

Well FredThompson and I want YUV 4:4:4. Any body else?

(sorry Fred;) )

[Richard Berg]

Well if we're gonna create a new YUV colorspace, my vote goes to 4:4:4:4 YUV + alpha. Deciding whether to make it planar or packed is another matter -- with 4 "colors" maintaining a pointer to each uses up a lot of registers, but it also allows single-channel manipulations to be very quick. We could even have a hybrid, e.g. YUV in one plane and the alpha in another.

[Shalcker]

Isn't that a lot easier to create "layer" plugin that will accept three clips instead of two with third clip used as alpha-channel?
something like "layer2(clip1, clip2, alphaclip)"

This should work with any colorspace, no matter planar and interleaved - just go through the clip, multiply-add-divide or whatever three values, write result as output... why do you need alpha within clip itself? With alpha as separate clip you can also transform alpha-clip independently with existing filters, no need to add new ones or rewrite old ones...

[mf]

Whoah.. Feedback :D :). I've read about some AYUV colorspace on MSDN, which apparently is a 4:4:4 colorspace with alpha. So that satisfies both of our needs. That's what I was talking about.

@Kurosu: I know mfToon can be faster, I absolutely hate the colorspace conversions I have to do in it. As I was making it, I was seeing it grow slower by the hour. If you're interested in making it any faster, I still have an idea lying around for a "piecemeal supersampler" which should speed up the sharpening by a fewfold (think tenfold or more). The only thing is, AVISynth doesn't support loops so I couldn't script it myself :D.

[Richard Berg]

I'm not too familiar with mfToon. Can you comment on whether a 3-argument Layer would be better/worse/equivalent than AYUV for your needs?

[Bidoche]

@Richard Berg

Maybe we can do both packed and planar, that's not a problem for me.
Adding planar YUV 4:4:4 is just a 5 line modification (just change the reported plane size in the colorspace, videoframe use that info to init themselves), another plane is not much either.
And for packed AYUV, it's just like RGB32.


A few days ago, esby suggested to me to allow plugins to add colorspaces to the core. I replied I didn't see any real interest in that since it was really easy to add new ones, but maybe it was me being short-minded. Any comment ?

[mf]

Originally posted by Richard Berg
I'm not too familiar with mfToon. Can you comment on whether a 3-argument Layer would be better/worse/equivalent than AYUV for your needs?
Hm, probably slightly better, for speed. AYUV would be very nice for other things though (although atm I can't think of them, must be cause I shouldn't be browsing doom9 forums but should be preparing to go on holiday tomorrow). An 8-bit greyscale colorspace could be very fast for mask operations (layer masks, etc). So for now I'd go for the 3-argument Layer that works on all colorspaces.

[Richard Berg]

Now that I've thought about it some, I'm starting to wonder why the alpha channel came about at all. It's convenient at the CPU level, where you can deal with [power of 2] channels at a time; and it's convenient at the GUI application level, where you construct a video out of several layers. In Avisynth, though, we're not afraid of creating dozens of intermediate clips when necessary (which is what GUI apps do too, behind the scenes), so it seems most logical to deal with alpha masks as clips in their own right. Let's consider WarpEnterprises' three Mask operations from the other thread:

#LumaToAlpha: as does Mask
# should be written
mask=CreateMask(clip)

#AlphaToAlpha: copy Alpha
# should be written
mask2=mask

#AlphaToLuma: the same as ShowAlpha
# should be written
clip=ShowMask(mask)

#We'll need some helper functions to deal with "legacy" RGBA
mask=CreateMask(clip, from_alpha=true) # requires RGB32 source
clip=ShowMask(mask, write_alpha=true)

#And lest we forget...
clip=Layer(src1, src2, mask) # src1,src2 = any colorspace, mask = 8-bit

All we need is a single-channel 8-bit colorspace, some easy filters, and you've created the ability to manipulate masks with the scripting language itself, not relying on C code that can parse out the alpha channels. And of course, it works with any colorspace, once we write the above filters. I haven't gone back and read your MaskTools thread, Kurosu, but does this sound similar? (This is just one use of this proposed colorspace; I'm sure there are others).

BTW, I'm not completely happy with the above filter names, feel free to suggest alternatives.

@Bidoche

You can feel free to add any/all of this to the new framework; for now I'm interested in the 2.x codebase even if it's quick-n-dirty.

[Bidoche]

For coherence, the function names should be, supposed this 8 bit mask colospace is called GreyScale :

mask = clip.ConvertToGreyScale() #convert clip Y to a mask

clip = mask.ConvertToXXX() #convert mask into Y

mask = ExtractAlpha(clip) #where clip must be RGB32



And for implementation in 2.x, you can probably just make it as YV12 without UV.

[WarpEnterprises]

1) Please NO MORE COLORSPACES :)
2) Why not only syntactically DEFINE the alpha channel as being the Y channel for YUY2 and YV12, the luma for RGB32 and RGB24?
Then we don't need a new colorspace and the existing syntax can be expanded as written above.
A clip then is a mask if it is USED as a mask.

3) Layer will have three clip arguments, so we could assume:
2 args -> old version, mask clip must be RGB32 with correct A
3 args -> new version, mask clip is separate clip, Y/lum is used
4) The same goes with the Mask or other functions (the number of args switches from old to new syntax)

[sh0dan]

@WarpE: What you are saying is exactly what I'm hearing from many users.

1) No more colorspaces. Use RGB32 if extreme precision is important.
2) Exactly.
3-4) Exactly - I have written a preliminary design doc for "Overlay" (new name instead of "Layer2").

[Richard Berg]

I agree. YV12 should be the preferred format for storing pure masks since you don't have to unpack the Y/alpha information, but it should work with anything. We now have 4 ways of storing masks: as alpha in RGB32, or as luma in RGB/YUY2/YV12. Thus our suite of functions looks something like this (correct me if I misunderstand):

# Getting masks
mask = ConvertToXXX(clip) # if mask = luma
mask = Mask(clip) # if mask = alpha; need to modify Mask to support all colorspaces

# Copying masks
mask2 = mask # if mask = luma
mask2 = CopyAlpha(mask) # if mask = alpha

# Viewing masks
return mask # if mask = luma in YUV
return GreyScale(mask) # if mask = luma in RGB; could also use ConvertToYV12 + BlankClip + YToUV
return GetAlpha(mask, out_colorspace="xxx") # if mask = alpha

# Using masks
Layer(clip1, clip2.CopyAlpha(mask), ...) # if mask = alpha
Layer(clip1, clip2, mask.ConvertToYUY2 or YV12) # if mask = luma; should do conversion to YUV (if necessary) first so Layer code can be clean & fast

[Bidoche]

I think it still make more sense to have a new Colorspace to store those masks.

But for users to not be bothered by the detail of it, Layer and other mask functions could implicitly do the appropriate conversion.
That way, for simple task you don't even have to know about it while retaining possibility to do complex ones with a compact format.

[WarpEnterprises]

@bidoche: can you sketch an easy example how to use built-in colorspace conversion in a plugin?

Some care is missing to assure the color is white when "showing" a YUV/YV12 mask.

Maybe we should create a "complete" document (at the wiki?) where to discuss the whole set before starting to implement. These functions are very important but difficult to understand, too, so they should be as "beautiful" as possible (for the enduser)

[Bidoche]

@WarpE

Invoke won't do ? or maybe it's me who don't understood your question.

[Richard Berg]

I think my previous post is a pretty good outline. Although looking at it, we should fold GetAlpha into CopyAlpha, just adding a parameter to choose whether the output is alpha-based or luma-based.

[Shalcker]

If we will have alpha as separate clip then why alpha channel should be limited to luma and rest of the image space wasted? Wouldn't having separate alpha value for each byte/word within image allow more flexibility in some cases?
It is a bit easier to code as mmx/sse too since all you need to know in this case is image size in memory (as long as each clip has same colorspace and size/pitch).

For example, with 3 YV12 clips it is possible to use luma channels of third clip as alpha for luma and chroma channels as alpha for chroma (allowing MergeLuma+MergeChroma operations with different weights for certain areas).

Otherwise we also have to average 4 luma values for YV12 to obtain alpha for each chroma point, which is probably faster to do while creating mask instead of calculating them with each application of mask.

[WarpEnterprises]

@Bidoche: maybe in principle, but can you point me to a source where it is coded?
e.g. assume you have a rgb32 clip and want to convert it to yuy2, how to place Invoke (in the constructor and getFrame)?
(Maybe this is better suited for the plugin example thread)

[WarpEnterprises]

@Shalcker: you are right in principal, but please also consider that the usage of Layer & co must stay as simple as possible, else it won't be used by many people. It is already quite difficult to "teach".

[Bidoche]

@WarpE

You can do it like it :

class MyFilter : public GenericVideoFilter {

static PClip AdjustChild(IScriptEnvironment * env, AVSValue args)
{
//some work with args (exceptions ?)
//AVSValue modArgs = ...
PClip result = env->Invoke("FunctionIWant", modArgs);
//maybe more ....
return result;
}

public:
MyFilter(ISCriptEnvironment * env, AVSValue args) : GenericVideoFilter(AdjustChild(env, args), env) { }
//VideoInfo arg missing :p

//the regular methods...
};


@Shalcker

What you are talking about is a coding optimisation, which should not be exposed to users.
I am sure tehy will do the averaging separately if it is faster.

And by the way, with the GrayScale ColorSpace design, it would be just an invocation of ReduceBy2.

[Richard Berg]

If there are no further objections I think I'll start coding the new layering tools. I'm not certain exactly how chroma-based alpha would work, but it can always be added later.

[Kurosu]

Originally posted by Richard Berg
I'm not certain exactly how chroma-based alpha would work, but it can always be added later.
The use I made most often was to Reduce by 2 the luma mask used and copy it to chroma planes. In that regard, I've sent some weeks ago a MMX ReduceBy2 for YV12 (during all that time, only a C version was available...) to Sh0dan. Dunno if that was integrated yet.

And while I'm at it, pick up whatever you may need in the last package of masktools (http://kurosu.inforezo.org/avs/MaskTools/MaskTools1.4.zip). There are too many things to list in it. If you ever pick up or feel interested in a part of it and need some decyphering, then please ask. I've been too lazy during the past months to clean up enough things to 'release' the YV12 Layer function in it.

[WarpEnterprises]

Looks great.
Tt seems quite intuitive and useful that we then can use all luma functions to view/process mask information!
For the docs there is some need to check all RGB32 functions if they let the A channel intact. I will look into this.

[Richard Berg]

I think we're agreed on the basic concept: "overload" (in the C++ sense) the existing YV12 colorspace to also represent (Y) a planar mask (UV) downsampled masks; allow easy conversion between these masks and traditional RGBA-based masks; extend Layer to use these masks on all colorspaces.

Interestingly enough, two applications of these masks have simultaneously evolved, both geared toward edge masks, overlays, and similar image processing tools. MaskTools (http://kurosu.inforezo.org/avs/MaskTools/MaskTools1.4.zip) may be familiar to you; FatMan (http://www11.brinkster.com/poptones/newcleus/ow.asp?FatMan) may not. Some of the ancillary tools may be duplicated -- for example, Inflate is similar to clip.Layer(clip.blur,"or") -- but I'll have to go through & see what would be useful for the core. Thanks all.

Edit -- Kurosu, is there a reason you had every filter input x,y,w,h? Using Crop seems more "atomic" :)

Edit #2 -- here is the poptones source (http://avisynth.org/src/fatman.zip)

[Kurosu]

Originally posted by Richard Berg
Some of the ancillary tools may be duplicated -- for example, Inflate is similar to clip.Layer(clip.blur,"or")
I don't remember what does the "or" processing (not a bitwise or, I guess, otherwhise it won't work the same way) but that should be very close anyway.

[quote]
Edit -- Kurosu, is there a reason you had every filter input x,y,w,h? Using Crop seems more "atomic" :)
Yes, those filters do more than atomic operations. But I considered worth adding such parameters so as to only process a part of the picture, starting at point (x,y) and processing the box (w,h) from that point. Layer has such an option but not with the same effect. I considered that if Layer was only processing a part of the picture and possibly with an offset, it could be wise to feed it clips (and eventually mask, if a 3rd clip is used as mask, allowing more dynamic layering, for instance) where only a part was modified.

I didn't keep the scripts that produce such effects, but it had an use for:
- YV12 overlaying of logos (even if there is already filters for that)
- green keying or whatever this is: make a color transparent.

For the later point, I had played (that's the word) a bit with the misnamed filter overlay:
- it builds a mask of what parts are different from another clip (can be seen as a way to track motion if the same clip is fed twice, with a delay for one)
- this mask can be used in maskedmerge, ending up in replacing static background by another clip (same as green keying but you don't need to have a particular color on the background)
This was funny but not that effective, as most videos are noisy or compressed (thus having compression noise).

[Bidoche]

@Kurosu

Making the filters work into a window is just making things more complicated for not much. You lose the possibility to use the extra padding between row_size/pitch to align.
I have not really searched for that but your code may miss that point in some places.

Besides in the input side, working in a window is less efficient than cropping first. The MakeWritable may have to blit the whole frame compared to only a part when cropping first.

On the other hand to recompose the full frame after, quite a number of Crop and Stack calls are needed. But we can create a function to copy a clip into another at the specified coordinates. It's not that hard and cost only one BitBlit. (I did it in 3.0 BufferWindow)

[Richard Berg]

On the other hand to recompose the full frame after, quite a number of Crop and Stack calls are needed.
Why not just Layer(clip, cropped_clip, "add", 255, x, y) ?

[Bidoche]

I was not aware of that solution, as I guess many of the users

So, I think a straight function will be clearer for them.
It can be coded as a Layer fallback in a first time, but the just one blit version must be faster.


Edit: the version I made for 3.0 works even if the second clip is not within the first, copying only the overlapping part.

[Richard Berg]

Layer will also only layer the overlapping parts. I agree it's not as fast as possible, but that's easily solved by making a new op for Layer, say "fastblit." IOW, the best solution would still involve using smaller intermediate clips rather than specifying x,y,w,h parameters for every function in between.

Unless I'm missing something, of course; I've been looking at MaskTools for less than a day.

Meanwhile, regardless of how we implement the advanced features, getting basic support for YV12 masks can be done when I wake up tomorrow.

[Bidoche]

If it's different feature, just let it be a different function.
I don't think you write your scripts with only Eval...

That function, (OverLap ?) should be just at the right of Crop in the editing functions, ensuring that way users find it.

IOW, the best solution would still involve using smaller intermediate clips rather than specifying x,y,w,h parameters for every function in between.I totally agree with that, dragging around x,y,w,h is a bother and you lose the AlignedRowSize possibility.
The coordinates-taking Layer itself should be built that way, thus separing its two features in more atomic functions : the overlapping one and the same size clips Layer.


clip.OverLap(clip.Crop(4,0,2,6).AFilter(...),4,0) is shorter and clearer than a Layer call

[mf]

So, I'm back now, and how are things going? I see some very positive things going on, but are some of them implemented yet, and in which code (2.5 or 3) ?

[Wilbert]

For 2.53:

1) layer isn't extended yet.
2) invert and showalpha are added, see avisynth.org

[mf]

Ah, nice :).

[Bidoche]

I have started an LumaFrame VideoFrame subclass (and PlanarVideoFrame superclass) and the corresponding ColorSpace.

[mf]

Could some work on that any-colorspace-mask/layer stuff PLEASE be done? I'm currently developing a new version of DetailIllusion and framerates are really getting too low :(.

[Richard Berg]

I got about halfway done with the reconceptualize-clips-as-masks framework, but school started before anything beyond ShowAlpha (which has a bug in SF, incidentally) and Invert could be committed. It's on my list to complete, next block of free time I get.

[mf]

Your work is very much appreciated. :)