out of gamut fix

[kolak]

Is it possible to use avisynth as out of gamut filter?
Some YUV signals produce illegal RGB values once converted over Rec.601/Rec.709 inverse matrixes. Could avisynth help?

More about the problem:

http://www.vidcheck.com/support/white-papers.php

[colours]

Slightly inefficient, but one way you can check for YUV exceeding the gamut is to first reduce the range then convert to RGB as usual, after which we can apply some sort of fancy soft limiting for out-of-gamut values. Maybe something like this (entirely untested):

source
Dither_lut8(expr="x 128 - 3 / 128 + 256 *",yexpr="x 125.5 - 3 / 125.5 + 256 *",u=3,v=3)
# replace 128 and 125.5 with 127.5 if working with a full-range source
Dither_convert_yuv_to_rgb(matrix="709",lsb_in=true,output="rgb48y")
r = SelectEvery(3,0)
g = SelectEvery(3,1)
b = SelectEvery(3,2)
undershoot = cl_exprxyz(r,g,b,"21760 x y z min min -",lsb=true).SelectEvery(1,0,0,0)
overshoot = cl_exprxyz(r,g,b,"x y z max max 43520 -",lsb=true).SelectEvery(1,0,0,0)
rgb_fixed = cl_exprxyz(last,undershoot,overshoot,"x 21760 y - - 21760 y z + + / 65280 *",lsb=true)
# the above is not exactly "fancy" soft limiting, but derp.
rgb_fixed.DitherPost(mode=6)
MergeRGB(SelectEvery(3,0),SelectEvery(3,1),SelectEvery(3,2))

Edit: now actually tested; forgot the lsb_in=true earlier.

[kolak]

Will try, thx.

So whole difficultly is to have fancy soft clipping method?

I assume first step (or maybe not necessarily 1st) is to limit luma and chroma to YUV values, which is also part of the whole legalisation process.

Another thing, which seams to be easier is to report out of gamut frames. This has some threasholds, described in EBU R103 standard. This has details:

https://www.google.co.uk/url?sa=t&so...KyKP5DGXFzNgFA

They mention low pass filtering, before checking, is this possible?

[kolak]

Originally Posted by colours

Slightly inefficient, but one way you can check for YUV exceeding the gamut is to first reduce the range then convert to RGB as usual, after which we can apply some sort of fancy soft limiting for out-of-gamut values. Maybe something like this (entirely untested):

source
Dither_lut8(expr="x 128 - 3 / 128 + 256 *",yexpr="x 125.5 - 3 / 125.5 + 256 *",u=3,v=3)
# replace 128 and 125.5 with 127.5 if working with a full-range source
Dither_convert_yuv_to_rgb(matrix="709",output="rgb48y")
r = SelectEvery(3,0)
g = SelectEvery(3,1)
b = SelectEvery(3,2)
undershoot = cl_exprxyz(r,g,b,"21760 x y z min min -",lsb=true).SelectEvery(1,0,0,0)
overshoot = cl_exprxyz(r,g,b,"x y z max max 43520 -",lsb=true).SelectEvery(1,0,0,0)
rgb_fixed = cl_exprxyz(last,undershoot,overshoot,"x 21760 y - - 21760 y z + + / 65280 *",lsb=true)
# the above is not exactly "fancy" soft limiting, but derp.
rgb_fixed.DitherPost(mode=6)
MergeRGB(SelectEvery(3,0),SelectEvery(3,1),SelectEvery(3,2))

Have an error that mt_lut supports only planar color spaces - fixed

but...

cl_exprxyz fails due to my card being old/not supported probably. Is there work around?

[hanfrunz]

a very brutal method is to use converttorgb32() and then converttoyuy2(), but if you need broadcast safe colors better use a plugin or a hardware legalizer (eyeheight is a good brand)

[StainlessS]

I believe Wilbert and Gavino might have been involved with something along those lines a few years ago.

[kolak]

Originally Posted by hanfrunz

a very brutal method is to use converttorgb32() and then converttoyuy2(), but if you need broadcast safe colors better use a plugin or a hardware legalizer (eyeheight is a good brand)

Yes, eyeheight is good, but they have only NLE plugins.

I want something which I can chain, script using pipes etc.
I believe it's way easier to do than some other avisynth plugins, scripts, just not needed by many people.

[Gavino]

Originally Posted by StainlessS

I believe Wilbert and Gavino might have been involved with something along those lines a few years ago.

You're probably thinking of the thread U and V ranges for valid RGB.
In post #14 of that thread, I gave a function ShowBadRGB() that shows the areas of a YUV clip that contain 'invalid' RGB values.
(Note: it takes a long time to load because of the mt_lutxyz function.)

It doesn't correct the problem, but it does show whether the problem exists and which areas need attention.

[StainlessS]

Yes Gavino, that was the thread I was thinking of although was wrong about Wilbert being involved (I think that was another thread).
I mainly remembered the animation which I still find fascinating.

[kolak]

Can anything be developed in this area- maybe sponsored (but probably rather for vapourysnth)?

[raffriff42]

Limiter is the gamut filter you are looking for (I think).
[EDIT wrong, see next post]

Also, subtracting Limiter's output from the source shows the out-of-range pixels.
Something like this:

On the right-hand "Diff" image, luma < 16 shows as black, white > 235 shows as white, etc
(note source contrast is exaggerated to show the effect)

Assert(IsYUV(Last), "YUV source required")

## for demo purposes, exaggerate contrast
#ColorYUV(levels="TV->PC")

return Last.ShowOutOfGamut()

##################################
### Show out-of-gamut pixels. 
##
## @ diff_scale - exaggerate the diff signal (range 1..128, default 32)
##
function ShowOutOfGamut(clip C, int "diff_scale")
{
    diff_scale = Min(Max(1, Default(diff_scale, 32)), 128)

    C ## Last==C
    Diff=Subtract(Last, Limiter())
    \ .ColorYUV(
    \  off_y=2*diff_scale, 
    \  cont_y=f2cuv(diff_scale), 
    \  cont_u=f2cuv(diff_scale), 
    \  cont_v=f2cuv(diff_scale))

    return StackHorizontal(
    \  Last, 
    \  Diff.Subtitle("Diff x "+String(Round(diff_scale)), size=Last.Height/8))
    \ .BilinearResize(Last.Width, Last.Height)
}

## scale "normal" float arguments to suit ColorYUV
function f2cuv(float f) { return Round((f - 1.0) * 256.0) }

[colours]

Limiter only limits the YUV values to [16,235] or [16,240], which is not enough to guarantee that the resulting RGB values are in [0,255].

I fixed the script I posted earlier, and it seems to work exactly as expected now. The test image is a U/V gradient with Y set to 168. The Mach bands are annoying, but I can't think of any obvious way to deal with them at the moment.

It shouldn't be too difficult to port this directly to a C/C++ filter for Avisynth or VapourSynth, though it might not be useful as is.

[kolak]

Thanks a lot. It doesn't look very bad. I think it lacks some soft roll-off.
Do you know what are the requirements for cl_expr plugin in terms of GPU?
Any CPU replacement for it?

I've checked and it looks quite good- in gamut are is practically untouched. Here is a difference (boosted):




I assume we just don't want this hard borders?
How/can it be "smoothed"?

[colours]

Originally Posted by kolak

I think it lacks some soft roll-off.

Probably because it's actually still hard limiting the colours, albeit in a way different from limiting the red/green/blue channels individually. What I'm doing preserves hue while sacrificing luma and saturation accuracy.

Originally Posted by kolak

Do you know what are the requirements for cl_expr plugin in terms of GPU?

Not sure, but it works with my laptop's integrated graphics (Intel).

Originally Posted by kolak

I've checked and it looks quite good- in gamut are is practically untouched. [snip] I assume we just don't want this hard borders?
How/can it be "smoothed"?

The problem is, to implement soft limiting, it's necessary to sacrifice accuracy even for in-gamut YCbCr values. I have a rough idea on how to do this, but CLExpr likes to error out when I use it too many times in a single script (tp7 says this is probably because Intel's OpenCL drivers are bad) so I'm not particularly interested in implementing it at the moment.

[kolak]

I may have more interesting news tomorrow, once I test your image with pro solution. It may end up that your script is actually very good

As far as I know you have to change correct values also in order to have some soft clipping. I think this is exactly the case for pro solutions, so please carry on with your ideas

[kolak]

No idea which one is correct, but pro plugin gives basically no change for 0-100% RGB.
To show some adjustment I've adjusted it to 90% clip (85% knee) settings and than it gives this:



By the logic this is correct. I'm confused now, as looks like avs script is wrong.

[colours]

Originally Posted by kolak

No idea which one is correct, but pro plugin gives basically no change for 0-100% RGB.
To show some adjustment I've adjusted it to 90% clip (85% knee) settings and than it gives this:

By the logic this is correct. I'm confused now, as looks like avs script is wrong.

1: You shouldn't be feeding an RGB image, because, well, the levels are already crushed. I generated my test image with mt_lutspa; something like blankclip(width=256,height=256,pixel_type="yv24").mt_lutspa(mode="absolute",yexpr="168",uexpr="x",vexpr="y",u=3,v=3).pointresize(768,768).

2: Just to be clear, the mouseout pic in the comparison I linked before is Dither_convert_yuv_to_rgb, while the mouseover pic (with the obvious diagonal Mach banding) is the script I wrote.

[kolak]

I'm lost...

I need image which is wrong, so I can compare against pro solutions.

If I generate one as per your command and than save it as yuv uncompressed files will this a "bad source" with areas out of gamut?
(looks like it's way out)

[colours]

Originally Posted by kolak

I'm lost...

I need image which is wrong, so I can compare against pro solutions.

Arguably, anything with out-of-gamut/out-of-range YCbCr values is already wrong, and clamping the converted RGB values to [0,255] is PSNR-optimal.

I uploaded my sample clip to a temporary file hosting site; the luma value changes by frame, while each frame is a 0–255 chroma gradient.

[kolak]

What is the point of whole legalisation in digital world? It will be clipped anyway at some point.
I assume it had more sense for analog transmission.