How does one get the average color of the frame?

[D9JK]

Hello,

I thought this would be a simple task, but it's giving me a bit of trouble. I need a filter that averages the entire (current) frame into a single color, can be grayscale as well. If you've used Photoshop, I'm after an effect that is similar to "Filter/Blur/Average".

Some background (not necessary to read):

First I tried simply general blur filters with obscenely high values, but where doing this seemed to produce a flat color, it was simply gray across all frames, which was wrong because the average brightness should clearly have varied across the video frames. Additionally, applying excessive blurring seems to pick up some blackness from outside the image frame, but this may depend on which blur filter is being used.

Next I tried a slightly hacky way; resizing a frame to 1x1 (in image editing, this works as average blur filter), and then bringing it back to the original size. But the lowest resolution the resize filters allow is 4x4, so that didn't work.

Then I found a mention of a filter called "AverageLuma". As this sounded promising, I gave it a try. Apparently simply entering AverageLuma() causes an error message about invalid parameters for the filter... but I cannot find any helpful documentation about it. The filter is not listed neither in the internal (http://avisynth.org.ru/docs/english/corefilters.htm) nor external (http://avisynth.nl/index.php/External_filters) filters of Avisynth.

Perhaps I'm missing something.

[StainlessS]

Suggest take a look at Showchannels.#
http://forum.doom9.org/showthread.php?t=163829

[D9JK]

Your filter is a useful analyzing tool, but currently I'm looking for a filter that averages the entire frame into a single, flat color. That is, the filter should actually modify the frame contents.

Thank you anyway.

[poisondeathray]

Next I tried a slightly hacky way; resizing a frame to 1x1 (in image editing, this works as average blur filter), and then bringing it back to the original size. But the lowest resolution the resize filters allow is 4x4, so that didn't work.



If you work in RGB, you can use pointresize(1,1)

[Gavino]

If you work in RGB, you can use pointresize(1,1)
But that won't average the pixels, it will just replicate the centre one.
EDIT: Not the centre pixel, but the corner one (bottom left for RGB).

Then I found a mention of a filter called "AverageLuma". As this sounded promising, I gave it a try. Apparently simply entering AverageLuma() causes an error message about invalid parameters for the filter... but I cannot find any helpful documentation about it. The filter is not listed neither in the internal (http://avisynth.org.ru/docs/english/corefilters.htm) nor external (http://avisynth.nl/index.php/External_filters) filters of Avisynth.
AverageLuma() is not a filter - it is a run-time function (http://avisynth.nl/index.php/ScriptClip#Runtime_Functions) that gives a numeric result, and it can only be used inside a run-time filter such as ScriptClip(). This should give you what you want (assuming a YUV source):
ScriptClip("""
y = round(AverageLuma())
u = round(AverageChromaU())
v = round(AverageChromaV())
BlankClip(last, color_yuv=65536*y + 256*u + v)
""")

[D9JK]

Thanks Gavino, and sorry for my ignorance, but how do I use it exactly? I've written a few custom functions, but ScriptClip looks a bit different.

[Gavino]

Just add that code after your source filter, eg

AviSource("mysource.avi") # or whatever you are using
ScriptClip("""
... as shown earlier ...
""")

[D9JK]

Ah, I see. I'm usually using filters this way:

Changed = Original.Grayscale().Invert()

The script is a bit long when I have other filters in there already, but I suppose I can put it in a new custom function, then give it a name short enough.

Thanks a bunch!

[pandy]

Perhaps resize will work for you? (1 pixel probably not).
Bilinear work to 2x2 then 4 - 5 time in series Blur (1.5, 1.5) should solve your problem (at least from my perspective looks like all 4 pixels have same value - obviously this is RGB space, for YCbCr can be else, probably limit is 4x4 and multiple pass Blur (16 times or more) is required).

[StainlessS]

This will also work with RGB & YUY2, Need Gavino's Grunt() plugin for the "After_frame=True" arg.
EDIT: No you dont, see next post.


ShowChannels(SetVar=True,Show=False)
ScriptClip("""
SC_Ave_1 = (SC_Ave_1<0.0) ? 0.0 : SC_Ave_1 # If Avisynth v2.6, convert Y8 channels 1 and 2 to 0 (will be -1.0)
SC_Ave_2 = (SC_Ave_2<0.0) ? 0.0 : SC_Ave_2
z = (256*Round(SC_Ave_0)+Round(SC_Ave_1))*256+Round(SC_Ave_2)
IsRGB ? BlankClip(last, color=z) : BlankClip(last, color_yuv=z)
""",after_frame=True)

EDITED: Added Y8 guard.

You can even select an area of the frame to examine via ShowChannels x,y,w,h args.


Changed = Original.Grayscale().Invert()


will work equally well as here (you dont need to have everything on one line)

Original.Grayscale()
Invert()
ShowChannels(SetVar=True,Show=False)
ScriptClip("""
SC_Ave_1 = (SC_Ave_1<0.0) ? 0.0 : SC_Ave_1 # If Avisynth v2.6, convert Y8 channels 1 and 2 to 0 (will be -1.0)
SC_Ave_2 = (SC_Ave_2<0.0) ? 0.0 : SC_Ave_2
z = (256*Round(SC_Ave_0)+Round(SC_Ave_1))*256+Round(SC_Ave_2)
IsRGB ? BlankClip(last, color=z) : BlankClip(last, color_yuv=z)
""",after_frame=True)
Changed=Last

[Gavino]

Need Gavino's Grunt() plugin for the "After_frame=True" arg.
The built-in ScriptClip() also supports the after_frame arg.

[D9JK]

Although my original intention was something else, I also wrote a crude automatic white balance function as a test:
function AutoWB(clip X, float O)
{
X
ConvertToYV12()
ScriptClip(""" u=round(AverageChromaU()) v=round(AverageChromaV()) BlankClip(last, color_yuv=65536*128 + 256*u + v) """)
Invert()
Overlay(X, mode="softlight")
Overlay(X, mode="chroma", opacity=1-O)
}

I'm probably not going to use it very much, as automatic white balance functions would have to be much more sophisticated to be practical. But out of curiosity: is ScriptClip limited to working on one frame at a time, or would it be possible to somehow take nearby frames into consideration? I'm asking this because the AutoWB function above can of course produce very abrupt adjustments on the WB, and averaging the color of eg. 10-30 neighboring frames would help in making the effect less apparent.

[Gavino]

is ScriptClip limited to working on one frame at a time, or would it be possible to somehow take nearby frames into consideration?
Each output frame can be derived from several input frames, although with the standard ScriptClip(), to get access to previous or future frames you have to construct additional clips via filters such as Trim() or SelectEvery().
It's easier with the GRunT version of ScriptClip() - for example, AverageChromaU(-2) gives the average U value of the frame two before the current one.

[D9JK]

Thanks Gavino, I'll take a look. That's a funny abbreviation, too. :-)

[Dogway]

Gavino, do you know how to access every other frame? I am trying to compare YDifferenceToNext() between current and current+2, but I'm out of ideas, this is what I'm trying:

scriptclip("""Subtitle(c,string(YDifferenceToNext(Fmod(current_frame, 2.0)==0.0?even:odd)),align=7,size=30)""",
\ args="c,even=selectevery(c,2,0,2),odd=selectevery(c,2,1,3)")
}

What I get is a value of 0 every other frame as if they were repeated.

[StainlessS]

RT_YDifference(clip,int "n"=current_frame,int "delta"=1,int "x"=0,int "y"=0,int "w"=0,int "h"=0,int "x2"=x,int "y2"=y,
bool "interlaced"=false,int "Matrix"=(Width>1100||Height>600?3:2))
Returns FLOAT value luma difference (0.0 -> 255.0) between frame n area x,y,w,h, and frame (n+delta) area x2,y2,w,h.
Note, by default it will be equivalent to YDifferenceToNext as delta defaults to 1 and x,y,w,h defaults full frame.
Note, 'x2' and 'y2' default to 'x' and 'y' respectively.

Eg, RT_YDifference(clip,delta=0,y2=1,interlaced=true)
Would difference the even and odd lines of the same frame,



RT_YDifference(delta=2)

[Dogway]

If that means there's no way with built-in functions then thanks for the help.

[StainlessS]

Avisource("E:\v\avatar.avi")
target = Trim(2,0)
# target = SelectEvery(2,2,3) # EDIT: Added


SS="""
x = LumaDifference(Target)
y = RT_YDifference(delta=2)
Subtitle(string(x) + " " + String(y),align=7,size=30)
"""

Last.scriptclip(SS, args="Target")

[Dogway]

Yes, thank you, RT_Stats plugin. If possible I will try to keep using built-in function, if not possible I guess RT_YDifference() is fine.

[StainlessS]

Already given built-in ie LumaDifference, RT_ was just a check for valid.
can also use SelectEvery(2,2,3) instead of trim.

Or more like your original post


return Avisource("E:\v\avatar.avi").scriptclip("""Subtitle(string(LumaDifference(SelectEvery(2,2,3))),align=7,size=30)""")


EDIT: Or probably SelectEvery(1,2) better


return Avisource("E:\v\avatar.avi").scriptclip("""Subtitle(string(LumaDifference(SelectEvery(1,2))),align=7,size=30)""")

[Dogway]

Yes that's right, LumaDifference accepts two inputs, that caught me by surprise.
Thank you StainlessS.

[BakaProxy]

I suggest you take a look at masktools2, more specifically mt_lutf, mt_luts and mt_average.

[Keiyakusha]

Photoshop average? Here. No plugins needed. (as was mentioned, resizing to 1x1 is indeed not possible, so have to be like this)

Source() # must be rgb or yv24, see below
w=Width()
h=Height()
BilinearResize(2,2)
tl=Crop(0,0,-1,-1)
tr=Crop(1,0,0,-1)
bl=Crop(0,1,-1,0)
br=Crop(1,1,0,0)
x=Overlay(tl,tr,opacity=0.5)
y=Overlay(bl,br,opacity=0.5)
Overlay(x,y,opacity=0.5)
PointResize(w,h)
Though it is probably slow as hell compared to any other solutions, I dunno.

[wonkey_monkey]

I think that method needs a converttorgb or a converttoyv24, otherwise it complains that "Source image too small for resize method" (I think it should say destination image).

It also seems to give a slightly different colour result depending which convert you use. Not sure why.

This is marginally faster:


w=Width()
h=Height()
BilinearResize(2,2)
overlay(last,last.flipvertical,opacity=0.5)
overlay(last,last.fliphorizontal,opacity=0.5)
pointresize(w,h)

[Keiyakusha]

davidhorman
Oh yes of course. Converting to RGB or YV24 is a must because YV12 can not be 1-pixel in size. I forgot to mention. But it shouldn't be the case for your version?
Also it is not 100% perfect because color conversion is lossy. There are rounding errors. If you convert to YV24, then it must be converted to RGB later on for preview. Perhaps source->YV24->RGB is somehow different from source->RGB, and the difference comes from here?

Edit:
did some tests. It appears that BilinearResize() can do 2x2 in rgb/yv24 but not in yv12. too bad. Prolly because of chroma.

Edit2: lol, with this script:
Colorbars()
converttoyv12()
BilinearResize(4,4)

I get "error trying to display the clip". I think it is bugged ^^

[StainlessS]

Edit2: lol, with this script:
Colorbars()
converttoyv12()
BilinearResize(4,4)

I get "error trying to display the clip". I think it is bugged ^^

Works for me loading into VirtualDub current or current MPC-HC, Avisynth v2.6 standard.
(fail, on load into MPlayer2 from XP/W2K, 'unable to download an appropriate decompressor').

[Reel.Deel]

Converting to RGB or YV24 is a must because YV12 can not be 1-pixel in size.

For any YUV colorspace you can do something like this, output colorspace is YV24:

Source() # must be yuv
w=Width()
h=Height()

y = renameittowhateveryouwant()
u = UToY8().renameittowhateveryouwant()
v = VToY8().renameittowhateveryouwant()

YToUV(u,v,y)
PointResize(w,h)


Function renameittowhateveryouwant(clip clip)
{
clip.ConvertToY8()
BilinearResize(2,2)
tl=Crop(0,0,-1,-1)
tr=Crop(1,0,0,-1)
bl=Crop(0,1,-1,0)
br=Crop(1,1,0,0)
x=Overlay(tl,tr,opacity=0.5)
y=Overlay(bl,br,opacity=0.5)
Overlay(x,y,opacity=0.5)
}

For RGB you can do this:

r = ShowRed("Y8").renameittowhateveryouwant()
g = ShowGreen("Y8").renameittowhateveryouwant()
b = ShowBlue("Y8").renameittowhateveryouwant()

MergeRGB(r,g,b)

I think that method needs a converttorgb or a converttoyv24... It also seems to give a slightly different colour result depending which convert you use. Not sure why.

Have you tried setting pc_range=true in Overlay when working with RGB?

[cretindesalpes]

Why so complicated?
mt_lutf (last, mode="avg", expr="x", u=3, v=3)