Automatic White Balance (AWB) script function(s)

[martin53]

EDIT: attached script version 27th October 2013 This one may be called more than once from a script, e.g. to compare settings
EDIT: script version 13th November a bit faster and hopefully no more AviSynth string heap issue - but has severe bugs in AWB algorithm, please don't use at this time. Update to come soon.
EDIT: script version 1st December Finalized White Patch (Max RGB) algorithm. Also:
+ uses arrays instead of global vars: full non linear access with 100% reproduceable output
+ multi instance capable
+ unprecedented plugin check

I'll concentrate on a new approach now (after I learned so much about things that are not what they seem in consumer cameras YUV or RGB values)

——————
Original Post

I'd like to announce a script function or maybe script function set for automatic white balance for real world footage, similar to the known function in digital cameras.

Some widely used algorithms implement theories of J. von Kries on the ability of the human visual system to recognize the color of objects under biased illumination, called "color constancy".
They are known as "Gray World", "Max-RGB" and "Shades of Gray".

Maybe I failed in finding an already existing plugin for AviSynth or VirtualDub that implements a useful automatic white balance (I do not count ColorYUV(autowhite=true) as useful). They may be not too fast since they use the runtime environment. But they show once again the power of AviSynth scripting when one refuses to write in C (as I do up to now). Here are the first appetizers, and comments are welcome.

Needs AviSynth V2.6 and dither, GRunt, mt_masktools and RT_Stats plugins.

--> see attached GrayWorldSimple.txt, Max-RGB_Simple.txt, ShadesOfGraySimple.txt

[StainlessS]

Have not tried as yet, but shall.
I have real expectations of this, hope you live up to them.

EDIT: Was the MYStats thing of any use ?

[martin53]

Originally Posted by StainlessS

...have real expectations of this, hope you live up to them.

Same for me! To be honest, the underlying concepts of the three algorithms are quite ungarnished. But there was or is plenty of research on the topic, and this is what publicly evolved from it. To repeat myself: the concepts are not my inventions.

Originally Posted by StainlessS

Was the MYStats thing of any use ?

I am convinced it will! I first tried it on itself on some clips and found that it seemed to work well. The list of challenges on my mind contains averaging over a group of frames, which in turn demands scene detection, an automatic mode which evaluates several algorithms and chooses/balances between them, and, maybe most important, masking of a color range that is valid at all for potential color temperature shift, i.e. lies along the temperature curve and is not a colorful object. I expect much from this to reduce misled corrections.

Up to now, I had to cope with color bleeding, banding etc. from the bad resolution of 8 bits in the neccessary YUV-RGB-YUV processing. Research on how to define the interesting color range will come next.

You can see from the examples that there is a lack of full featured statistics functions for RGB as there is for YUV for the task. I found workarounds though, so that's not a job stopper.

[StainlessS]

Originally Posted by martin53

lack of full featured statistics functions for RGB as there is for YUV

Give me time and I'll do similar funcs for RGB

[martin53]

If you like to do it for fun, go ahead. For the white balance task, it will evolve which statistic functions will be needed.
Among them are, as far as I can judge today, masked versions for AverageChromaU and AverageChromaV. The mask should be taken from the Y plane of the mask clip, but the U and V planes of the main clip be evaluated.

Also, for the "Shades of Gray" algorithm, average functions that do not average the first power of samples, but an arbitrary power are useful.
To date the clip is prepared with 16 bit bitdepth and the samples taken to the power of 6 with Dither_lut8() and then dithered down to 8 bit resolution before measuring. But of course that's a workaround.

[martin53]

I am just wondering about color conversion documentation.
The AviSynth v2.6 doc in Advanced Topics / ColorSpace Conversions looks very reasonable. The constants Kr, Kg and Kb are taken from the standard, and the following equations derived from that.
But at second sight, the (v - 128 = ...) and (u - 128 = ...) equations have factors of 0.5, which seems arbitrary.

The web page http://en.wikipedia.org/wiki/YUV defines Wr=Kr, Wg=Kg and Wb=Kg, but furthermore Umax=0.436 and Vmax=0.615. This of course leads to different RGB->YUV and YUV->RGB transformations.

My question is: who is right, and are the AviSynth V2.6 colorspace conversion routines trustable?

[Gavino]

Originally Posted by martin53

The AviSynth v2.6 doc in Advanced Topics / ColorSpace Conversions looks very reasonable. The constants Kr, Kg and Kb are taken from the standard, and the following equations derived from that.
But at second sight, the (v - 128 = ...) and (u - 128 = ...) equations have factors of 0.5, which seems arbitrary.

The web page http://en.wikipedia.org/wiki/YUV defines Wr=Kr, Wg=Kg and Wb=Kg, but furthermore Umax=0.436 and Vmax=0.615. This of course leads to different RGB->YUV and YUV->RGB transformations.

The difference is simply an arbitrary scale factor in U and V.
The Avisynth wiki page gives the formula to derive U and V in the range [-1, 1], while the Wikipedia article uses the range [-Umax, Umax] and [-Vmax, Vmax].

But when doing RGB<->YUV conversions on 8-bit inputs, you have to rescale anyway to fit the range [16, 240] or [0, 255] (depending on the matrix used). The Avisynth conversion routines do this correctly.

[martin53]

Originally Posted by Gavino

The Avisynth wiki page gives the formula to derive U and V in the range [-1, 1]

Thank you for answering, Gavino.
Alas, I intentionally asked for the lowercase letters u and v because I need the constants outside the builtin conversion routines.
Reading on in the Wikipedia article, I found now that I confused Y'UV with Y'CbCr anyway, and instead of calculating
U = -0.147 * R - 0.289 * G + 0.436 * B
which would only be valid for an analog device, for a digital device it must be
Cb = -0.169 * R - 0.331 * G + 0.499 * B + 128

I take 0.499 as 0.5, and so indeed the two formulas match. -0.169 comes from 0.5*Kr/(1-Kb) while the analog -0.147 comes from 0.436*Kr/(1-Kb) and so on, and it is confirmed that in AviSynth, the u and v scale factors are 0.5
(which btw. makes sense, because Y's [0,1] range is 1 wide while U,V's [-1,1] range has a width of 2. The [0,255] scaled variables share a range ot the same width, so a downscaling with 0.5 for u and v is obvious from that)

[martin53]

As an ancillary product, here is a script to change the 'color temperature' of a clip by a fixed offset.
It uses physically correct values to correct RGB and tries not to change the overall luminance. If that clamps RGB, you may enter a gain factor < 1.0 to make the clip darker with less clamping.

-> see 1st post

EDIT: function is part of complete script in 1st post

[martin53]

More advanced versions of the simple estimators drafted above will restrict estimation to colors in the picture that might be biased grey.
Below is a mask generator that I have used lately and that will find its way into later versions of the AWB function itself.

It can be fed with this test clip, to show the functionality.

Luma statistics functions give much better results if their input is restricted to these pixels, i.e. the MYStats functions with this mask give much better results than the standard functions.

I'll be offline for several days from now.

function testClipYUV() {
    c = BlankClip(240,640,480,"YV12")
    c.mt_lutspa(yexpr="128", uexpr="x 255 *", vexpr="y 255 *", chroma="process")
}

Mask generator

function GrayGamutMask(clip c) {
#
# Function to create a mask that is only white for clip pixels within a defined color and luminance range.
# Expects clip in YUV format
# Returns mask clip in YV24 format
#
# Author		doom9.org/martin53
# Version		1.0
# Date			June 18, 2013
# Uses plugins	GRunT, MaskTools v2.0a48
# Tested with	Avisynth V2.6
#
# Changes		n.a.
#
    TOL_UP = 0.9
    TOL_DN = -0.4
    U_DN = 128-15	# -50 mired, i.e. color temperature change from D65 to 4900K warm day or 7800K to 5600K flash
    V_UP = 128+7	# -50 mired, i.e. color temperature change from D65 to 4900K warm day or 7800K to 5600K flash
    U_UP = 128+15	#  50 mired, i.e. color temperature change from 4900K light to D65
    V_DN = 128-7	#  50 mired, i.e. color temperature change from 4900K light to D65
    YMIN = 100
    YMAX = 235 - 1
    c24 = c.ConvertToYV24()
	#Make a mask according to experimentally found U and V change over color temperature
	#mt_lutxy is fed with the U and V components. x stands for the U component, y for the V component.
    expr = mt_polish("(x-128)/15+(y-128)/7 <= "+string(TOL_UP)
		 \+" & (x-128)/15+(y-128)/7 >= "+string(TOL_DN)
		 \+" & x <= "+string(U_UP)
		 \+" & y >= "+string(V_DN)
		 \+" & x >= "+string(U_DN)
		 \+" & y <= "+string(V_UP)
		 \+" ? 255 : 0")
    exprX = mt_polish("(x-128)/15+(y-128)/7 <= "+string(TOL_UP*2)
		 \+" & (x-128)/15+(y-128)/7 >= "+string(TOL_DN*2)
		 \+" & x <= "+string(U_UP*2)
		 \+" & y >= "+string(V_DN*2)
		 \+" & x >= "+string(U_DN*2)
		 \+" & y <= "+string(V_UP*2)
		 \+" ? 255 : 0")
	#subtitle(expr)
    mChroma1 = mt_lutxy(c24.UToY(), c24.VToY(), expr=expr, chroma="128")
	#Choose this mask if it is not all black, otherwise choose fallback mask with doubled tolerance values
	mChroma = GConditionalFilter(mChroma1, mChroma1, mt_lutxy(c24.UToY(), c24.VToY(), expr=exprX, chroma="128"), "AverageLuma()>0")
	#Complete the mask with minimum and maximum luma restrictions.
	#mt_lutxy is fed with the Y component and the previous mask. x stands for Y (luma), y for the previous mask.
    expr = mt_polish("y > 0"+" & x >= "+string(YMIN)+" & x <= "+string(YMAX)+" ? 255 : 0")
	#subtitle(expr)
    mLuma = mt_lutxy(c24, mChroma, expr=expr, chroma="128")
	#Choose mask mLuma if it is not all black, else choose fallback mask mChroma
	GConditionalFilter(mLuma, mLuma, mChroma, "AverageLuma()>0")
	#Choose this mask if it is not all black, else choose fallback all white mask
	GConditionalFilter(last, last, BlankClip(c24, color_yuv=$f08080), "AverageLuma()>0")
	#debug
	#GScriptClip("""RT_Debug("Mask",string(AverageLuma())) last""")	#"
}

Usage e.g.

TestClipYUV()
merge(last, GrayGamutMask().ConvertToYV12(), weight=0.1)

[Navarre66]

I'm trying to use MaxRGB1, but I keep getting the error:

Script error: ScriptClip does not have a named argument "args"

Apparently inferring it doesn't like the args="RGBinYUV"

I'm using AviSynth 2.6.0 Alpha4 because I get problems with dither.avsi if I use AviSynth 2.5.8 or SEt's version of Avisynth 2.6.

[Gavino]

Originally Posted by Navarre66

I keep getting the error:
Script error: ScriptClip does not have a named argument "args"
Apparently inferring it doesn't like the args="RGBinYUV"

You need to install the GRunT plugin (which - among other things - extends ScriptClip with the 'args' parameter).

[Navarre66]

Originally Posted by Gavino

You need to install the GRunT plugin (which - among other things - extends ScriptClip with the 'args' parameter).

Thanks. That did the trick.

[martin53]

Script is now attached to 1st post. Please read abstract before you try it.
Comments welcome.

[martin53]

This is a call for comments on expectations what an AWB script should serve for.

After I fiddled so much with my lots of images & scenes, I learned that it is not that difficult to average RGB in an image, but that slight adjustments of scenes require a script to restrict itself, while the correction of images taken with sunlight setting in bulb illuminated environment, or very shadowy environment, require strong impact.

So I may optimize it to do subtle enhancements, but sacrifice the ability to always return the same result, no matter how extreme the user may bias the input in advance. For that purpose, I shifted the 'correlated color temperature' of the input clip by +-1000°C, and the output does not stay the same, how I earlier defined the goal for myself. Personally I feel now that it is important to retain the original mood of scenes in most cases, so the color of the result just seems 'right' in an unobstrusive way, i.e. just eliminate annoying color bias e.g. from incandescent light, but not overdo compensation. That might require manual pre-balancing in some cases, however. A script for manual pre-shifting of the correlated color temperature is provided in this thread.

[yup]

Hi martin53!
I find Your script useful, thanks for Your time. My source mainly VHS capture indoor shooting with color distortion (a lot of yellow see https://www.dropbox.com/s/wyc1o6ygvxbuv8a/samplevhs.avi).

AVISource("tape1.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
Crop(16,12,-16,-12)
s=SeparateFields()
s.AutoGain(adjust_mode=1).ShadesOfGray1()
# OR s.AutoGain(adjust_mode=1).GrayWorld1()
StackVertical(s,last)
#AssumeFieldBased().Weave()

work fine shift yellow to blue color and video look good (I know real color because have photo and slide from this place), but script

AVISource("tape1.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
Crop(16,12,-16,-12)
s=SeparateFields()
s.AutoGain(adjust_mode=1).AWB()
StackVertical(s,last)
#AssumeFieldBased().Weave()

give more yellow than at source. At other source script give unstable output atr preview in VirtualDub.
Please advice parameter set for AWB for my source..
yup.

[martin53]

yup,
downloaded the example.
What codec should I use to decode lagarith LAGS video (W7 x86)? my (I admit, probably outdated) AviSource/ffdshow are helpless. VLC plays, but with *heavy* blue flicker.

(Forgive my ignorance, I of course know how to run google, but at the moment don't like to spend time learning about lagarith)

EDIT:[ don't bother, XMediaRecode->HuffYUV worked.

I checked with the current script on my HD. One problem is the date stamp. The "10/" shows good white. I will add xm and ym to the parameters list to allow masking of a rectangle, please have a bit patience.
In between, you might mask that rectangle with a Overlay() line.

But aside from automatic balancing, I found

ShiftCCT(5600)

quite appealing in this case.

If you allow, I'll use the short sequence as additional test case. It is interesting because if flickers more than my other test clips.
]/EDIT

[StainlessS]

M53, pretty sure I've found a bug in RT_Stats RT_RGBChanxxx filters, suspend usage for a few hours, I'll do a fix.
Found it when doing MRGBChanStats() plug.

Lagarith pretty common codec for AVI YV12
http://www.videohelp.com/tools/Lagar...ss-Video-Codec
Just install it, not much to learn.


But being overtaken by UT_Video (RGB,YUY2,YV12)
http://www.videohelp.com/tools/Ut-Video-Codec-Suite
I hardly ever use Lagarith or HuffYUV now.
Just install it, not much to learn.

[yup]

martin53!

I try code

AVISource("tape1.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
Crop(16,12,-16,-12)
s=SeparateFields()
s.AutoGain(adjust_mode=1).ShiftCCT(5600)
StackVertical(s,last)

And see good result. Please confirm need I use Autogain before Your AWB functions?
yup.

[martin53]

Originally Posted by yup

Please confirm need I use Autogain before Your AWB functions?

I was not aware of any influence between the two. When I first checked some minutes ago, the clip with AutoGain was automatically balanced in a much better way than before, but then when I compared histograms I did not find an effect of Autogain on any single pixel of a frame - and I looked very closely, with a difference amplifier. And I can not reproduce the mentioned effect now.

At the moment, I can not identify any difference. Please describe the effect you observe with/-out it.