Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 Doom9's Forum out of gamut fix
 Register FAQ Calendar Search Today's Posts Mark Forums Read

 26th February 2015, 00:39 #1  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 out of gamut fix Is it possible to use avisynth as out of gamut filter? Some YUV signals produce illegal RGB values once converted over Rec.601/Rec.709 inverse matrixes. Could avisynth help? More about the problem: http://www.vidcheck.com/support/white-papers.php
 26th February 2015, 04:55 #2  |  Link colours Registered User     Join Date: Mar 2014 Posts: 312 Slightly inefficient, but one way you can check for YUV exceeding the gamut is to first reduce the range then convert to RGB as usual, after which we can apply some sort of fancy soft limiting for out-of-gamut values. Maybe something like this (entirely untested): Code: ```source Dither_lut8(expr="x 128 - 3 / 128 + 256 *",yexpr="x 125.5 - 3 / 125.5 + 256 *",u=3,v=3) # replace 128 and 125.5 with 127.5 if working with a full-range source Dither_convert_yuv_to_rgb(matrix="709",lsb_in=true,output="rgb48y") r = SelectEvery(3,0) g = SelectEvery(3,1) b = SelectEvery(3,2) undershoot = cl_exprxyz(r,g,b,"21760 x y z min min -",lsb=true).SelectEvery(1,0,0,0) overshoot = cl_exprxyz(r,g,b,"x y z max max 43520 -",lsb=true).SelectEvery(1,0,0,0) rgb_fixed = cl_exprxyz(last,undershoot,overshoot,"x 21760 y - - 21760 y z + + / 65280 *",lsb=true) # the above is not exactly "fancy" soft limiting, but derp. rgb_fixed.DitherPost(mode=6) MergeRGB(SelectEvery(3,0),SelectEvery(3,1),SelectEvery(3,2))``` Edit: now actually tested; forgot the lsb_in=true earlier. __________________ Say no to AviSynth 2.5.8 and DirectShowSource! Last edited by colours; 28th February 2015 at 08:20.
 26th February 2015, 09:19 #3  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 Will try, thx. So whole difficultly is to have fancy soft clipping method? I assume first step (or maybe not necessarily 1st) is to limit luma and chroma to YUV values, which is also part of the whole legalisation process. Another thing, which seams to be easier is to report out of gamut frames. This has some threasholds, described in EBU R103 standard. This has details: https://www.google.co.uk/url?sa=t&so...KyKP5DGXFzNgFA They mention low pass filtering, before checking, is this possible? Last edited by kolak; 26th February 2015 at 09:27.
26th February 2015, 10:42   #4  |  Link
kolak
Registered User

Join Date: Nov 2004
Location: UK
Posts: 2,324
Quote:
 Originally Posted by colours Slightly inefficient, but one way you can check for YUV exceeding the gamut is to first reduce the range then convert to RGB as usual, after which we can apply some sort of fancy soft limiting for out-of-gamut values. Maybe something like this (entirely untested): Code: ```source Dither_lut8(expr="x 128 - 3 / 128 + 256 *",yexpr="x 125.5 - 3 / 125.5 + 256 *",u=3,v=3) # replace 128 and 125.5 with 127.5 if working with a full-range source Dither_convert_yuv_to_rgb(matrix="709",output="rgb48y") r = SelectEvery(3,0) g = SelectEvery(3,1) b = SelectEvery(3,2) undershoot = cl_exprxyz(r,g,b,"21760 x y z min min -",lsb=true).SelectEvery(1,0,0,0) overshoot = cl_exprxyz(r,g,b,"x y z max max 43520 -",lsb=true).SelectEvery(1,0,0,0) rgb_fixed = cl_exprxyz(last,undershoot,overshoot,"x 21760 y - - 21760 y z + + / 65280 *",lsb=true) # the above is not exactly "fancy" soft limiting, but derp. rgb_fixed.DitherPost(mode=6) MergeRGB(SelectEvery(3,0),SelectEvery(3,1),SelectEvery(3,2))```

Have an error that mt_lut supports only planar color spaces - fixed

but...

cl_exprxyz fails due to my card being old/not supported probably. Is there work around?

Last edited by kolak; 26th February 2015 at 13:11.

 26th February 2015, 14:39 #5  |  Link hanfrunz Registered User     Join Date: Feb 2002 Location: Germany Posts: 534 a very brutal method is to use converttorgb32() and then converttoyuy2(), but if you need broadcast safe colors better use a plugin or a hardware legalizer (eyeheight is a good brand)
 26th February 2015, 16:02 #6  |  Link StainlessS HeartlessS Usurer     Join Date: Dec 2009 Location: Over the rainbow Posts: 5,906 I believe Wilbert and Gavino might have been involved with something along those lines a few years ago. __________________ I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
26th February 2015, 16:26   #7  |  Link
kolak
Registered User

Join Date: Nov 2004
Location: UK
Posts: 2,324
Quote:
 Originally Posted by hanfrunz a very brutal method is to use converttorgb32() and then converttoyuy2(), but if you need broadcast safe colors better use a plugin or a hardware legalizer (eyeheight is a good brand)
Yes, eyeheight is good, but they have only NLE plugins.

I want something which I can chain, script using pipes etc.
I believe it's way easier to do than some other avisynth plugins, scripts, just not needed by many people.

27th February 2015, 11:26   #8  |  Link
Gavino
Avisynth language lover

Join Date: Dec 2007
Location: Spain
Posts: 3,380
Quote:
 Originally Posted by StainlessS I believe Wilbert and Gavino might have been involved with something along those lines a few years ago.
You're probably thinking of the thread U and V ranges for valid RGB.
In post #14 of that thread, I gave a function ShowBadRGB() that shows the areas of a YUV clip that contain 'invalid' RGB values.
(Note: it takes a long time to load because of the mt_lutxyz function.)

It doesn't correct the problem, but it does show whether the problem exists and which areas need attention.
__________________
GScript and GRunT - complex Avisynth scripting made easier

 27th February 2015, 12:30 #9  |  Link StainlessS HeartlessS Usurer     Join Date: Dec 2009 Location: Over the rainbow Posts: 5,906 Yes Gavino, that was the thread I was thinking of although was wrong about Wilbert being involved (I think that was another thread). I mainly remembered the animation which I still find fascinating. __________________ I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
 27th February 2015, 13:25 #10  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 Can anything be developed in this area- maybe sponsored (but probably rather for vapourysnth)?
 28th February 2015, 01:46 #11  |  Link raffriff42 Retried Guesser     Join Date: Jun 2012 Posts: 1,377 Limiter is the gamut filter you are looking for (I think). [EDIT wrong, see next post] Also, subtracting Limiter's output from the source shows the out-of-range pixels. Something like this: On the right-hand "Diff" image, luma < 16 shows as black, white > 235 shows as white, etc (note source contrast is exaggerated to show the effect) Code: ```Assert(IsYUV(Last), "YUV source required") ## for demo purposes, exaggerate contrast #ColorYUV(levels="TV->PC") return Last.ShowOutOfGamut() ################################## ### Show out-of-gamut pixels. ## ## @ diff_scale - exaggerate the diff signal (range 1..128, default 32) ## function ShowOutOfGamut(clip C, int "diff_scale") { diff_scale = Min(Max(1, Default(diff_scale, 32)), 128) C ## Last==C Diff=Subtract(Last, Limiter()) \ .ColorYUV( \ off_y=2*diff_scale, \ cont_y=f2cuv(diff_scale), \ cont_u=f2cuv(diff_scale), \ cont_v=f2cuv(diff_scale)) return StackHorizontal( \ Last, \ Diff.Subtitle("Diff x "+String(Round(diff_scale)), size=Last.Height/8)) \ .BilinearResize(Last.Width, Last.Height) } ## scale "normal" float arguments to suit ColorYUV function f2cuv(float f) { return Round((f - 1.0) * 256.0) }``` Last edited by raffriff42; 17th March 2017 at 23:57. Reason: (fixed image link)
 28th February 2015, 09:41 #12  |  Link colours Registered User     Join Date: Mar 2014 Posts: 312 Limiter only limits the YUV values to [16,235] or [16,240], which is not enough to guarantee that the resulting RGB values are in [0,255]. I fixed the script I posted earlier, and it seems to work exactly as expected now. The test image is a U/V gradient with Y set to 168. The Mach bands are annoying, but I can't think of any obvious way to deal with them at the moment. It shouldn't be too difficult to port this directly to a C/C++ filter for Avisynth or VapourSynth, though it might not be useful as is. __________________ Say no to AviSynth 2.5.8 and DirectShowSource!
 28th February 2015, 15:44 #13  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 Thanks a lot. It doesn't look very bad. I think it lacks some soft roll-off. Do you know what are the requirements for cl_expr plugin in terms of GPU? Any CPU replacement for it? I've checked and it looks quite good- in gamut are is practically untouched. Here is a difference (boosted): I assume we just don't want this hard borders? How/can it be "smoothed"? Last edited by kolak; 1st March 2015 at 00:14.
1st March 2015, 12:51   #14  |  Link
colours
Registered User

Join Date: Mar 2014
Posts: 312
Quote:
 Originally Posted by kolak I think it lacks some soft roll-off.
Probably because it's actually still hard limiting the colours, albeit in a way different from limiting the red/green/blue channels individually. What I'm doing preserves hue while sacrificing luma and saturation accuracy.

Quote:
 Originally Posted by kolak Do you know what are the requirements for cl_expr plugin in terms of GPU?
Not sure, but it works with my laptop's integrated graphics (Intel).

Quote:
 Originally Posted by kolak I've checked and it looks quite good- in gamut are is practically untouched. [snip] I assume we just don't want this hard borders? How/can it be "smoothed"?
The problem is, to implement soft limiting, it's necessary to sacrifice accuracy even for in-gamut YCbCr values. I have a rough idea on how to do this, but CLExpr likes to error out when I use it too many times in a single script (tp7 says this is probably because Intel's OpenCL drivers are bad) so I'm not particularly interested in implementing it at the moment.
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!

Last edited by colours; 1st March 2015 at 12:53.

 1st March 2015, 14:04 #15  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 I may have more interesting news tomorrow, once I test your image with pro solution. It may end up that your script is actually very good As far as I know you have to change correct values also in order to have some soft clipping. I think this is exactly the case for pro solutions, so please carry on with your ideas
 2nd March 2015, 18:30 #16  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 No idea which one is correct, but pro plugin gives basically no change for 0-100% RGB. To show some adjustment I've adjusted it to 90% clip (85% knee) settings and than it gives this: By the logic this is correct. I'm confused now, as looks like avs script is wrong. Last edited by kolak; 2nd March 2015 at 18:53.
2nd March 2015, 23:34   #17  |  Link
colours
Registered User

Join Date: Mar 2014
Posts: 312
Quote:
 Originally Posted by kolak No idea which one is correct, but pro plugin gives basically no change for 0-100% RGB. To show some adjustment I've adjusted it to 90% clip (85% knee) settings and than it gives this: By the logic this is correct. I'm confused now, as looks like avs script is wrong.
1: You shouldn't be feeding an RGB image, because, well, the levels are already crushed. I generated my test image with mt_lutspa; something like blankclip(width=256,height=256,pixel_type="yv24").mt_lutspa(mode="absolute",yexpr="168",uexpr="x",vexpr="y",u=3,v=3).pointresize(768,768).

2: Just to be clear, the mouseout pic in the comparison I linked before is Dither_convert_yuv_to_rgb, while the mouseover pic (with the obvious diagonal Mach banding) is the script I wrote.
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!

 2nd March 2015, 23:38 #18  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 I'm lost... I need image which is wrong, so I can compare against pro solutions. If I generate one as per your command and than save it as yuv uncompressed files will this a "bad source" with areas out of gamut? (looks like it's way out) Last edited by kolak; 3rd March 2015 at 00:07.
3rd March 2015, 01:20   #19  |  Link
colours
Registered User

Join Date: Mar 2014
Posts: 312
Quote:
 Originally Posted by kolak I'm lost... I need image which is wrong, so I can compare against pro solutions.
Arguably, anything with out-of-gamut/out-of-range YCbCr values is already wrong, and clamping the converted RGB values to [0,255] is PSNR-optimal.

I uploaded my sample clip to a temporary file hosting site; the luma value changes by frame, while each frame is a 0–255 chroma gradient.
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!

 3rd March 2015, 09:24 #20  |  Link kolak Registered User   Join Date: Nov 2004 Location: UK Posts: 2,324 What is the point of whole legalisation in digital world? It will be clipped anyway at some point. I assume it had more sense for analog transmission.