New grayworld video filter in FFmpeg 5.0

[Arx1meD]

The new version of FFmpeg (News (https://ffmpeg.org/index.html#news)) has a grayworld video filter for changing the white balance using an algorithm based on the gray world hypothesis. Read more about the algorithm here (https://www.researchgate.net/publication/275213614_A_New_Color_Correction_Method_for_Underwater_Imaging).
Help from FFmpeg is here (https://ffmpeg.org/ffmpeg-filters.html#grayworld).

11.102 grayworld
A color constancy filter that applies color correction based on the grayworld assumption
The algorithm uses linear light, so input data should be linearized beforehand (and possibly correctly tagged).

ffmpeg -i INPUT -vf zscale=transfer=linear,grayworld,zscale=transfer=bt709,format=yuv420p OUTPUT


Is it possible to make a filter or script function for AviSynth?

This is a port of the FFmpeg filter grayworld: grayworld (https://github.com/Asd-g/AviSynthPlus-grayworld)

Picture from pdf file:
https://images2.imgbox.com/f4/9d/gwWlEPhF_o.jpghttps://images2.imgbox.com/f1/a3/1QvaAedp_o.jpg
https://images2.imgbox.com/27/33/KdoTPHuv_o.jpghttps://images2.imgbox.com/71/a0/sPi5rPdJ_o.jpg

The vf_grayworld.c file from the source code in the attachment. (change .txt to .c)

[Dogway]

This is a cheap approach in YUV.
function GreyWorld(clip a) {

rgb = isRGB(a)
isy = isy(a)
px = PixelType(a)
bi = BitsPerComponent(a)
fs = propNumElements (a,"_ColorRange") > 0 ? \
propGetInt (a,"_ColorRange") == 0 : rgb

ScriptClip(a,function[bi,fs,px] () {

U = PlaneMinMaxStats(0, 0, 1, false)
V = PlaneMinMaxStats(0, 0, 2, false)
H = ex_bs(128, 8, bi, true, float=true)
U = H - U[4]
V = H - V[4]

Expr(last,SwapUV(),"",Format("x {U} +"),Format("x {V} +"), Format=px)
} ) }

[Reel.Deel]

The vf_grayworld.c file from the source code in the attachment. (change .txt to .c)

For better accessibility, source is here: https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_grayworld.c

FFmpeg over the past few years has added a lot of interesting audio/video filters. Most of the unique filters for avs/vs have been ported over to FFmpeg. I'd be nice to access to the collection of filters in AviSynth.

This is a cheap approach in YUV.

GreyWorld()
https://i.ibb.co/FqwtxKm/greyworld-avs.png

[Dogway]

Yes, that example wasn't so good, the other one was better but I wanted to avoid going through RGB or LAB since that's very slow. Anyway I had the project in my todo list. There was an attempt before (https://forum.doom9.org/showthread.php?t=168062).

https://i.imgur.com/FbPf17p.png

[Arx1meD]

Dogway,
I know about GrayWorldSimple and ShadesOfGraySimple. They change the white balance quite normally, but they have a big drawback. When the scene changes or when an object with a different color suddenly appears in the frame, the colors flash.

[takla]

Dogway,
I know about GrayWorldSimple and ShadesOfGraySimple. They change the white balance quite normally, but they have a big drawback. When the scene changes or when an object with a different color suddenly appears in the frame, the colors flash.

Probably due to no scene detection or too much weight for one frame.

I'd recommend whoever will make an avisynth plugin out of this to take some things from this http://avisynth.nl/index.php/AutoAdjust

[Dogway]

That will happen with mine or the paper algorithm as well because metrics is a different topic. As I said in another thread we need a plugin or something internal that accumulates/averages metrics per scenes. It requires forward look ahead.

EDIT: Improved

function GreyWorld(clip a) {

rgb = isRGB(a)
isy = isy(a)
px = PixelType(a)
bi = BitsPerComponent(a)

ScriptClip(a,function[bi,px] () {

U = PlaneMinMaxStats(1, 0, 1, false)
V = PlaneMinMaxStats(1, 0, 2, false)
H = ex_bs(128, 8, bi, true, float=true)
Ux = H - U[4]
Vy = H - V[4]
Us = abs(Ux) > abs(Vy) ? "range_half - "+string(U[4] / V[4])+" * range_half +" : ""
Vs = abs(Ux) < abs(Vy) ? "range_half - "+string(V[4] / U[4])+" * range_half +" : ""

Expr(last,SwapUV(),"",Format("x {Ux} + "+Us),Format("x {Vy} + "+Vs), Format=px)
} ) }
https://i.imgur.com/FD5N0cP.png

[Arx1meD]

This is a port of the FFmpeg filter grayworld: grayworld (https://github.com/Asd-g/AviSynthPlus-grayworld). Thank you very much Asd-g.
The description says the input clip must be in RGB(A) 32-bit planar format and in linear light.
I can't understand how to convert the clip to RGB planar format. ConvertToRGB32() and ConvertToPlanarRGBA() filters are not suitable.

[Reel.Deel]

The description says the input clip must be in RGB(A) 32-bit planar format and in linear light.
I can't understand how to convert the clip to RGB planar format. ConvertToRGB32() and ConvertToPlanarRGBA() filters are not suitable.

Use avsresize:

z_ConvertFormat(pixel_type="RGBPS", colorspace_op="rgb:srgb:709:full=>rgb:linear:709:full")
grayworld()
z_ConvertFormat(pixel_type="RGBP16", colorspace_op="rgb:linear:709:full=>rgb:srgb:709:full")

I hope that is right, I added this to the examples section of avsresize (http://avisynth.nl/index.php/Avsresize#Linear_Light_Downscaling) but did not test it.

Edit: tested it with the example image in the first post:
# load jpeg in native YUV colorspace, convert it to linear RGB, process it with
# grayworld() and finally convert the result back to 8-bit sRGB.

FFImageSource("KdoTPHuv_o.jpg")
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="601:601:470bg:full=>rgb:linear:470bg:full", chromaloc_op="center=>center")
grayworld()
z_ConvertFormat(pixel_type="RGBP8", colorspace_op="rgb:linear:709:full=>rgb:srgb:709:full")

Result:
https://i.ibb.co/yFJ8yM5/grayworld-avs.png

Hmm, result looks a little different than in the example. I found the original image here: https://i2.wp.com/worldadventuredivers.com/wp-content/uploads/2013/08/img_4477.jpg


I can't understand how to convert the clip to RGB planar format. ConvertToRGB32() and ConvertToPlanarRGBA() filters are not suitable.

And to answer your question, only a few of the convert filters have a bitdepth designation, e.g ConvertToRGB64 will convert any clip to 16 bit RGBA (packed). However for the Convert functions that do not have a bitdepth designation, you have to convert the bipdepth yourself (before or after, depending on what your're doing). So lets say that you have an RGB clip and you wanted to convert it to float, first thing is to convert it to planar, if the clip is not already, then comes ConvertBits (http://avisynth.nl/index.php/ConvertBits)(32) which converts the clip to float, resulting colorspace is now RGBPS. The wiki does not mention any of this and I find it a little confusing also, and that table there confused me even more when I first was experimenting lol.

[Arx1meD]

Thank you Reel.Deel
I made several tests comparing the result of different filters.

Source clips: source_bubbles (https://mega.nz/file/0OIHmaxK#VNrazmFfUBIRTkbA9jpAE4oWbeMJ0dDb_2FMhlTBptU), source_carnival (https://mega.nz/file/sfBFEbwB#n_wA0pNQyMoPjy9BsMfSK3ck0x_FWwDZmtncxHPyO74), source_cat (https://mega.nz/file/pWQAAZpL#XO-vBvHu5-Yax-wXZMvwUbFdZYXTA4WMt1sU0M-uPUs)

My comparison result of grayworld, GamMac (https://forum.doom9.org/showthread.php?t=173695&highlight=AWB) and AWB (https://forum.doom9.org/showthread.php?t=168062) filters:
https://thumbs2.imgbox.com/ff/e1/x7uO6vIt_t.jpg (https://imgbox.com/x7uO6vIt) https://thumbs2.imgbox.com/1d/5b/qdbOwL0D_t.jpg (https://imgbox.com/qdbOwL0D) https://thumbs2.imgbox.com/c6/b8/SdMrSYdC_t.jpg (https://imgbox.com/SdMrSYdC)
Video comparison: test_bubbles (https://mega.nz/file/cOpTxAia#KiJQdve5Ryw_etellNaeO6PJ5TDtAaziK6hjq-EHc3s), test_carnival (https://mega.nz/file/EChWFRYJ#R8P1lPkiGP_yfetk-slViBI-fou-rlLYTtbTQRt-PvM), test_cat (https://mega.nz/file/4GBSjDpD#wR8Lay9QccH-fxAxcVYxQfn_UMLJMs6Vp2IwplpqqiA)

It is difficult to compare which filter is better. I think, when the colors are heavily distorted, then grayworld gives better results, and when the colors are almost normal, then the result is worse.

[Reel.Deel]

I can't get the same results as the example picture. In the example result it looks better, the highlights are not blown out. I tried FFmpeg also and get similar results to AviSynth.

Not very experienced with FFmpeg but I used this:
ffmpeg -i "img_4477.jpg" -vf zscale=transfer=linear,format=gbrpf32be,grayworld,zscale=transfer=bt709,format=rgba "img_4477.png"

[Arx1meD]

Reel.Deel, I took picture examples from Bianco_etal_ISPRS_CIPA.pdf (https://www.researchgate.net/publication/275213614_A_New_Color_Correction_Method_for_Underwater_Imaging)

It seems to me that vf_grayworld.c has a different algorithm than Bianco_etal_ISPRS_CIPA.pdf (see APPENDIX on the last page).

From vf_grayworld.c:
static const float lms2lab[3][3] = {
{0.5774, 0.5774, 0.5774},
{0.40825, 0.40825, -0.816458},
{0.707, -0.707, 0}
};

static const float lab2lms[3][3] = {
{0.57735, 0.40825, 0.707},
{0.57735, 0.40825, -0.707},
{0.57735, -0.8165, 0}
};

static const float rgb2lms[3][3] = {
{0.3811, 0.5783, 0.0402},
{0.1967, 0.7244, 0.0782},
{0.0241, 0.1288, 0.8444}
};

static const float lms2rgb[3][3] = {
{4.4679, -3.5873, 0.1193},
{-1.2186, 2.3809, -0.1624},
{0.0497, -0.2439, 1.2045}
};

From Bianco_etal_ISPRS_CIPA.pdf:
https://images2.imgbox.com/94/40/zM6zvVAt_o.png

[Reel.Deel]

Interesting find Arx1meD. Looking around I found another implementation in Python:

Underwater colour correction - In this project, we studied, implemented and tested an underwater colour correction method from Bianco et al.
https://github.com/p-ebert/underwater_colour_correction

The color transformations there are consistent with what is used in the pdf.

Edit:

I found this page: Paralenz color correction algorithm (https://colorcorrection.firebaseapp.com/) and it produces a decent looking result for the example image.

https://i.ibb.co/nn6Kxn6/img-4477-corrected.png

The GitHub page (https://github.com/nikolajbech/underwater-image-color-correction) describes the process:

The algorithm

Calculate the average color for image.
Hueshift the colors into the red channel until a minimum red average value of 60.
Create RGB histogram with new red color.
Find low threshold level and high threshold level.
Normalize array so threshold level is equal to 0 and threshold high is equal to 255.
Create color filter matrix based on the new values.


Maybe something that can already be done in AviSynth?

[richardpl]

IMO it is worse, gives blue tint.

[poisondeathray]

Interesting find Arx1meD. Looking around I found another implementation in Python:

Underwater colour correction - In this project, we studied, implemented and tested an underwater colour correction method from Bianco et al.
https://github.com/p-ebert/underwater_colour_correction

The color transformations there are consistent with what is used in the pdf.



Were you able to reproduce the paper results or those in the 1st post ?

I ran the python script and was not able to reproduce the paper results using "KdoTPHuv_o.jpg" as input

"--lab_constrast_correction equalisation" mode came the closest but there was still channel clipping, and playing with --constrast_cut_off values did not help


parser.add_argument('--colour_correction', default="LAB", help="perform colour correction in [LAB, CIELAB, LUV, YCBC] spaces")
parser.add_argument('--lab_constrast_correction', default="none", help="perform contrast correction with [stretching, CLAHE, equalisation, none] methods")
parser.add_argument('--constrast_cut_off', default=1e-3, help="cut-off level for lab contrast correction")




IMO it is worse, gives blue tint.
paralenz -
Yes, and red channel looks damaged or missing (pinkish splotch below fish becomes blue) .

Maybe some of the settings can be altered to reduce the channel loss ?

[Reel.Deel]

Were you able to reproduce the paper results or those in the 1st post ?

I ran the python script and was not able to reproduce the paper results using "KdoTPHuv_o.jpg" as input

"--lab_constrast_correction equalisation" mode came the closest but there was still channel clipping, and playing with --constrast_cut_off values did not help


I did not, I was just point out another implementation that used the same color transformation as the paper. Interesting that you couldn't get the same result as what the paper shows :confused:


paralenz -
Yes, and red channel looks damaged or missing (pinkish splotch below fish becomes blue) .

Maybe some of the settings can be altered to reduce the channel loss ?

I did not play around with the settings, I just set the gain to the highest level and saved the image. The author mentioned there are some images that are problematic.

https://github.com/nikolajbech/underwater-image-color-correction/issues/1

[anton_foy]

I did not play around with the settings, I just set the gain to the highest level and saved the image. The author mentioned there are some images that are problematic.

https://github.com/nikolajbech/underwater-image-color-correction/issues/1

Actually only one image, the "Lena" image :D
But seriously I am curious to see how this handles skintones, that is what color analog film was all about.

[ChaosKing]

https://github.com/Asd-g/AviSynthPlus-grayworld

[poisondeathray]

https://github.com/Asd-g/AviSynthPlus-grayworld

This was mentioned in post #8

There are some differences between the ffmpeg version, avs port vs. the paper results

[kedautinh12]

Latest ver
https://github.com/Asd-g/AviSynthPlus-grayworld/releases

[Arx1meD]

I decided to compare two different methods in the Grayworld filter. If you do not look closely, the difference is almost not noticeable. Or I am doing something wrong again.
so = DSS2("C:\test\underwater.mkv", fps=30.000)
#==========================================================
v1 = so.ConvertBits(32)
v1 = v1.z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:f=>rgb:linear:709:f")
v1 = v1.grayworld(cc=0)
v1 = v1.z_ConvertFormat(pixel_type="YUV420", colorspace_op="rgb:linear:709:f=>709:709:709:l")
#==========================================================
v2 = so.ConvertBits(32)
v2 = v2.z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:f=>rgb:linear:709:f")
v2 = v2.grayworld(cc=1)
v2 = v2.z_ConvertFormat(pixel_type="YUV420", colorspace_op="rgb:linear:709:f=>709:709:709:l")
#==========================================================
StackHorizontal(so.AddBorders(0, 0, 0, 20).Subtitle("Source", align=2),
\ v1.AddBorders(0, 0, 0, 20).Subtitle("grayworld(cc=0)", align=2),
\ v2.AddBorders(0, 0, 0, 20).Subtitle("grayworld(cc=1)", align=2))

https://thumbs2.imgbox.com/6a/ae/TJvVVUQe_t.jpg (https://imgbox.com/TJvVVUQe) https://thumbs2.imgbox.com/92/15/ziCwkjmb_t.jpg (https://imgbox.com/ziCwkjmb) https://thumbs2.imgbox.com/04/e6/oMSiLKPQ_t.jpg (https://imgbox.com/oMSiLKPQ)

Test video: underwater (https://mega.nz/file/5G4DBZrY#i4QxHgex0ik7qkZV2TYOpwY4-7eZ8-2rsKcesweFfbU)
Comparison: underwater_test (https://mega.nz/file/cPoznTRa#ntDTWLj0sJLCf-SBYKm8wKoTwrx-aFag03N9HFLaxEo)

[Reel.Deel]

@Arx1meD

I don't think you're doing anything wrong. I tried the 2 modes with the "corals" image from here: https://www.eng.tau.ac.il/~berman/UnderwaterColorRestoration/

https://i.ibb.co/2WTkMg6/grayworld-modes.png

The difference in this image is not day and night but it is visible. Maybe with certain images the difference is very small. For what it's worth, asd updated the docs and added this to the median mode: "This mode is not affected by extreme values in luminance or chrominance."

[cork_OS]

Grayworld effect is unstable on my sources. Is there a way to smooth it temporally or limit its strength?

[Arx1meD]

Cork_os, I think not.
Maybe create a clip with compensation for movement. Something like that:

function MC_Clip(clip clp, int "blksize", int "overlap", bool "chroma", bool "truemotion", bool "global") {
blksize = Default(blksize, clp.Width()<=720?16:32)
overlap = Default(overlap, blksize<=16?2:4)
chroma = Default(chroma, true)
truemotion = Default(truemotion, true)
glob = Default(global, true)

sup = MSuper(clp, hpad=8, vpad=8, pel=1, sharp=2, chroma=chroma)
bvec1 = MAnalyse(sup, isb=true, blksize=blksize, overlap=overlap, divide=0, delta=1, plevel=0, truemotion=truemotion, global=glob, chroma=chroma) # backward vectors
fvec1 = MAnalyse(sup, isb=false, blksize=blksize, overlap=overlap, divide=0, delta=1, plevel=0, truemotion=truemotion, global=glob, chroma=chroma) # forward vectors
backw1 = MFlow(clp, sup, bvec1)
forw1 = MFlow(clp, sup, fvec1)

Interleave(backw1, clp, forw1)
}

MC_Clip(blksize=32, overlap=16, chroma=true, truemotion=true, global=true)
# Your filters are here!
SelectEvery(3, 1)