New grayworld video filter in FFmpeg 5.0

[Arx1meD]

The new version of FFmpeg (News) has a grayworld video filter for changing the white balance using an algorithm based on the gray world hypothesis. Read more about the algorithm here.
Help from FFmpeg is here.

11.102 grayworld
A color constancy filter that applies color correction based on the grayworld assumption
The algorithm uses linear light, so input data should be linearized beforehand (and possibly correctly tagged).

ffmpeg -i INPUT -vf zscale=transfer=linear,grayworld,zscale=transfer=bt709,format=yuv420p OUTPUT

Is it possible to make a filter or script function for AviSynth?

This is a port of the FFmpeg filter grayworld: grayworld

Picture from pdf file:



The vf_grayworld.c file from the source code in the attachment. (change .txt to .c)

[Dogway]

This is a cheap approach in YUV.

function GreyWorld(clip a) {

rgb  = isRGB(a)
isy  = isy(a)
px   = PixelType(a)
bi   = BitsPerComponent(a)
fs   = propNumElements (a,"_ColorRange")  > 0 ? \
       propGetInt      (a,"_ColorRange") == 0 : rgb
       
ScriptClip(a,function[bi,fs,px] () {

    U  = PlaneMinMaxStats(0, 0, 1, false)
    V  = PlaneMinMaxStats(0, 0, 2, false)
    H  = ex_bs(128,  8, bi, true, float=true)
    U  = H - U[4]
    V  = H - V[4]

    Expr(last,SwapUV(),"",Format("x {U} +"),Format("x {V} +"), Format=px)
} ) }

[Reel.Deel]

Originally Posted by Arx1meD

The vf_grayworld.c file from the source code in the attachment. (change .txt to .c)

For better accessibility, source is here: https://github.com/FFmpeg/FFmpeg/blo...vf_grayworld.c

FFmpeg over the past few years has added a lot of interesting audio/video filters. Most of the unique filters for avs/vs have been ported over to FFmpeg. I'd be nice to access to the collection of filters in AviSynth.

Originally Posted by Dogway

This is a cheap approach in YUV.

GreyWorld()

[Dogway]

Yes, that example wasn't so good, the other one was better but I wanted to avoid going through RGB or LAB since that's very slow. Anyway I had the project in my todo list. There was an attempt before.

[Arx1meD]

Dogway,
I know about GrayWorldSimple and ShadesOfGraySimple. They change the white balance quite normally, but they have a big drawback. When the scene changes or when an object with a different color suddenly appears in the frame, the colors flash.

[takla]

Originally Posted by Arx1meD

Dogway,
I know about GrayWorldSimple and ShadesOfGraySimple. They change the white balance quite normally, but they have a big drawback. When the scene changes or when an object with a different color suddenly appears in the frame, the colors flash.

Probably due to no scene detection or too much weight for one frame.

I'd recommend whoever will make an avisynth plugin out of this to take some things from this http://avisynth.nl/index.php/AutoAdjust

[Dogway]

That will happen with mine or the paper algorithm as well because metrics is a different topic. As I said in another thread we need a plugin or something internal that accumulates/averages metrics per scenes. It requires forward look ahead.

EDIT: Improved

function GreyWorld(clip a) {

rgb  = isRGB(a)
isy  = isy(a)
px   = PixelType(a)
bi   = BitsPerComponent(a)

ScriptClip(a,function[bi,px] () {

    U  = PlaneMinMaxStats(1, 0, 1, false)
    V  = PlaneMinMaxStats(1, 0, 2, false)
    H  = ex_bs(128,  8, bi, true, float=true)
    Ux  = H - U[4]
    Vy  = H - V[4]
    Us = abs(Ux) > abs(Vy) ? "range_half - "+string(U[4] / V[4])+" * range_half +" : ""
    Vs = abs(Ux) < abs(Vy) ? "range_half - "+string(V[4] / U[4])+" * range_half +" : ""

    Expr(last,SwapUV(),"",Format("x {Ux} + "+Us),Format("x {Vy} + "+Vs), Format=px)
} ) }

[Arx1meD]

This is a port of the FFmpeg filter grayworld: grayworld. Thank you very much Asd-g.
The description says the input clip must be in RGB(A) 32-bit planar format and in linear light.
I can't understand how to convert the clip to RGB planar format. ConvertToRGB32() and ConvertToPlanarRGBA() filters are not suitable.

[Reel.Deel]

Originally Posted by Arx1meD

The description says the input clip must be in RGB(A) 32-bit planar format and in linear light.
I can't understand how to convert the clip to RGB planar format. ConvertToRGB32() and ConvertToPlanarRGBA() filters are not suitable.

Use avsresize:

z_ConvertFormat(pixel_type="RGBPS", colorspace_op="rgb:srgb:709:full=>rgb:linear:709:full")
grayworld()
z_ConvertFormat(pixel_type="RGBP16", colorspace_op="rgb:linear:709:full=>rgb:srgb:709:full")

I hope that is right, I added this to the examples section of avsresize but did not test it.

Edit: tested it with the example image in the first post:

# load jpeg in native YUV colorspace, convert it to linear RGB, process it with
# grayworld() and finally convert the result back to 8-bit sRGB.

FFImageSource("KdoTPHuv_o.jpg")
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="601:601:470bg:full=>rgb:linear:470bg:full", chromaloc_op="center=>center")
grayworld()
z_ConvertFormat(pixel_type="RGBP8", colorspace_op="rgb:linear:709:full=>rgb:srgb:709:full")

Result:


Hmm, result looks a little different than in the example. I found the original image here: https://i2.wp.com/worldadventuredive...8/img_4477.jpg

Originally Posted by Arx1meD

I can't understand how to convert the clip to RGB planar format. ConvertToRGB32() and ConvertToPlanarRGBA() filters are not suitable.

And to answer your question, only a few of the convert filters have a bitdepth designation, e.g ConvertToRGB64 will convert any clip to 16 bit RGBA (packed). However for the Convert functions that do not have a bitdepth designation, you have to convert the bipdepth yourself (before or after, depending on what your're doing). So lets say that you have an RGB clip and you wanted to convert it to float, first thing is to convert it to planar, if the clip is not already, then comes ConvertBits(32) which converts the clip to float, resulting colorspace is now RGBPS. The wiki does not mention any of this and I find it a little confusing also, and that table there confused me even more when I first was experimenting lol.

[Arx1meD]

Thank you Reel.Deel
I made several tests comparing the result of different filters.

Source clips: source_bubbles, source_carnival, source_cat

My comparison result of grayworld, GamMac and AWB filters:

Video comparison: test_bubbles, test_carnival, test_cat

It is difficult to compare which filter is better. I think, when the colors are heavily distorted, then grayworld gives better results, and when the colors are almost normal, then the result is worse.

[Reel.Deel]

I can't get the same results as the example picture. In the example result it looks better, the highlights are not blown out. I tried FFmpeg also and get similar results to AviSynth.

Not very experienced with FFmpeg but I used this:

ffmpeg -i "img_4477.jpg" -vf zscale=transfer=linear,format=gbrpf32be,grayworld,zscale=transfer=bt709,format=rgba "img_4477.png"

[Arx1meD]

Reel.Deel, I took picture examples from Bianco_etal_ISPRS_CIPA.pdf

It seems to me that vf_grayworld.c has a different algorithm than Bianco_etal_ISPRS_CIPA.pdf (see APPENDIX on the last page).

From vf_grayworld.c:

static const float lms2lab[3][3] = {
    {0.5774, 0.5774, 0.5774},
    {0.40825, 0.40825, -0.816458},
    {0.707, -0.707, 0}
};

static const float lab2lms[3][3] = {
    {0.57735, 0.40825, 0.707},
    {0.57735, 0.40825, -0.707},
    {0.57735, -0.8165, 0}
};

static const float rgb2lms[3][3] = {
    {0.3811, 0.5783, 0.0402},
    {0.1967, 0.7244, 0.0782},
    {0.0241, 0.1288, 0.8444}
};

static const float lms2rgb[3][3] = {
    {4.4679, -3.5873, 0.1193},
    {-1.2186, 2.3809, -0.1624},
    {0.0497, -0.2439, 1.2045}
};

From Bianco_etal_ISPRS_CIPA.pdf:

[Reel.Deel]

Interesting find Arx1meD. Looking around I found another implementation in Python:

• Underwater colour correction - In this project, we studied, implemented and tested an underwater colour correction method from Bianco et al.
• https://github.com/p-ebert/underwater_colour_correction

The color transformations there are consistent with what is used in the pdf.

Edit:

I found this page: Paralenz color correction algorithm and it produces a decent looking result for the example image.



The GitHub page describes the process:

The algorithm

Calculate the average color for image.
Hueshift the colors into the red channel until a minimum red average value of 60.
Create RGB histogram with new red color.
Find low threshold level and high threshold level.
Normalize array so threshold level is equal to 0 and threshold high is equal to 255.
Create color filter matrix based on the new values.

Maybe something that can already be done in AviSynth?

IMO it is worse, gives blue tint.

[poisondeathray]

Originally Posted by Reel.Deel

Interesting find Arx1meD. Looking around I found another implementation in Python:

• Underwater colour correction - In this project, we studied, implemented and tested an underwater colour correction method from Bianco et al.
• https://github.com/p-ebert/underwater_colour_correction

The color transformations there are consistent with what is used in the pdf.

Were you able to reproduce the paper results or those in the 1st post ?

I ran the python script and was not able to reproduce the paper results using "KdoTPHuv_o.jpg" as input

"--lab_constrast_correction equalisation" mode came the closest but there was still channel clipping, and playing with --constrast_cut_off values did not help

parser.add_argument('--colour_correction', default="LAB", help="perform colour correction in [LAB, CIELAB, LUV, YCBC] spaces")
parser.add_argument('--lab_constrast_correction', default="none", help="perform contrast correction with [stretching, CLAHE, equalisation, none] methods")
parser.add_argument('--constrast_cut_off', default=1e-3, help="cut-off level for lab contrast correction")

Originally Posted by richardpl

IMO it is worse, gives blue tint.

paralenz -
Yes, and red channel looks damaged or missing (pinkish splotch below fish becomes blue) .

Maybe some of the settings can be altered to reduce the channel loss ?

[Reel.Deel]

Originally Posted by poisondeathray

Were you able to reproduce the paper results or those in the 1st post ?

I ran the python script and was not able to reproduce the paper results using "KdoTPHuv_o.jpg" as input

"--lab_constrast_correction equalisation" mode came the closest but there was still channel clipping, and playing with --constrast_cut_off values did not help

I did not, I was just point out another implementation that used the same color transformation as the paper. Interesting that you couldn't get the same result as what the paper shows

Originally Posted by poisondeathray

paralenz -
Yes, and red channel looks damaged or missing (pinkish splotch below fish becomes blue) .

Maybe some of the settings can be altered to reduce the channel loss ?

I did not play around with the settings, I just set the gain to the highest level and saved the image. The author mentioned there are some images that are problematic.

https://github.com/nikolajbech/under...ction/issues/1

[anton_foy]

Originally Posted by Reel.Deel

I did not play around with the settings, I just set the gain to the highest level and saved the image. The author mentioned there are some images that are problematic.

https://github.com/nikolajbech/under...ction/issues/1

Actually only one image, the "Lena" image
But seriously I am curious to see how this handles skintones, that is what color analog film was all about.

[ChaosKing]

https://github.com/Asd-g/AviSynthPlus-grayworld

[poisondeathray]

Originally Posted by ChaosKing

https://github.com/Asd-g/AviSynthPlus-grayworld

This was mentioned in post #8

There are some differences between the ffmpeg version, avs port vs. the paper results

[kedautinh12]

Latest ver
https://github.com/Asd-g/AviSynthPlu...world/releases