HDR ColorBars - bug at transitions ?

[DTL]

It looks I find fun bug with HDR but still can not understand how to fix it to get ColorBars test pattern in HDR transfer encoded.


LoadPlugin("plugins_JPSDR.dll")
LoadPlugin("fmtcavs.dll")

ColorBarsHD(9600,1000, pixel_type="YUV444P16")
Crop(0,0,9600,500)
AddBorders(100,100,100,100)
#fmtc_matrix(mats="709", matd="rgb", fulls=false, fulld=false, csp="RGBP16")
#ConvertToRGB48(matrix="PC.709")
#ConvertToPlanarRGB()
#fmtc_transfer(transs="709", transd="linear", bits=16, flt=false, fulls=false, fulld=false)

#ConvertYUVtoLinearRGB(Color=2, OutputMode=2)
ConvertYUVtoLinearRGB(Color=2)

GaussResize(width/10,height/3, p=2)
# Linear RGBfloat planar in out size


#ConvertToPlanarRGB()
#fmtc_transfer(transs="linear", transd="2084",bits=16, flt=false, fulls=false, fulld=false)

#ConvertToYUV444().Tweak(sat=0)
#return ConvertBits(8).ConvertToRGB24()



#ConvertToYUV444()
#ConvertLinearRGBtoYUV(Color=2)
ConvertLinearRGBtoYUV(Color=0, HDRMode=0) # PQ
#ConvertLinearRGBtoYUV(Color=0, HDRMode=2) # HLG
# YUV 444
#Tweak(sat=0)
return ConvertBits(8).ConvertToRGB24()


It looks like appear in both jpsdr's linear to HDR convertor and in fmtconv plugin (with fmtc_transfer()). But I still can not understand how in appear in simple transfer operation in 4:4:4.

The bug looking - white lines at colour transitions of ColorBars pattern. And the width of white line depends on the data transition between colors (if lower GaussResize p-param the transient become wider and bug line too).
And it happens only with HDR transfers and not old bt.709 (or may be simple as low as invisible). In HDR transfers mostly visible at green-magenta transition.

Results:
https://i5.imageban.ru/out/2021/09/16/9f6d1ba7e45b40468b393f1f4280afc1.png

If process only Y black and white - the bug not happens. So I assume it depends on RGB<->YUV calculations and for some reason occur even with 4:4:4 without even going to sub-sampled colour and back.
Maybe something wrong with luma/chroma placement from ColorBarsHD() source and it become amplified when transformed to much more non-linear transfer like PQ (in compare with gamma about 2 at bt.709).

[poisondeathray]

I think it's because you didn't adjust primaries;

zimg (avsresize) works ok, and you get similar artifact if you don't adjust primaries


ColorBarsHD(9600,1000, pixel_type="YUV444P16")
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:709:f")

GaussResize(width/10,height/3, p=2)

z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:709:f=>2020ncl:st2084:2020:l")

#don't adjust primaries for 2020
#z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:709:f=>2020ncl:st2084:709:l")

#tweak(sat=0)

#convert to 8bit srgb preview
z_convertformat(pixel_type="rgbp", colorspace_op="2020ncl:st2084:2020:l=>rgb:srgb:709:f")

#convert to 8bit srgb preview 709 primaries
#z_convertformat(pixel_type="rgbp", colorspace_op="2020ncl:st2084:709:l=>rgb:srgb:709:f")

[DTL]

Oh - you almost save the world. But it looks either my misunderstanding of avsresize conversions or something strange with z_convert format:


LoadPlugin("avsresize.dll")

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

#z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:709:f")
z_convertformat(pixel_type="rgbp", colorspace_op="709:709:709:l=>rgb:linear:2020:l") # attempt to have primaries in 2020 before conditioning - no help

GaussResize(width/10,height/10, p=10)

#testing source
#return Crop(300,0, width-300, height).SincResize(width*4, height*4, taps=16).ConvertToRGB24(matrix="PC.709")

z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l")
#PQ HDR here

# attempt to resize in PQ HDR transfer
Crop(300,0, width-300, height).SincResize(width*4, height*4, taps=16)
return z_convertformat(pixel_type="yuv444p16", colorspace_op="2020ncl:st2084:2020:l=>709:709:709:l").ConvertToRGB24(matrix="PC.709")

# attempt to resize in linear
rgb_lin=z_convertformat(pixel_type="rgbps", colorspace_op="2020ncl:st2084:2020:l=>rgb:linear:2020:f")
rgb_lin=SincResize(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
z_convertformat(rgb_lin, pixel_type="YUV444P16", colorspace_op="rgb:linear:2020:f=>709:709:709:l")

return Crop(300,0, width-300, height).ConvertToRGB24(matrix="PC.709")


When I try to test linear scaling of the produced by z_convertformat PQ HDR output it cause severe vertical ringing (on black border, though colour patches almost clear - that is strange), but the source linear RGB before convert to PQ HDR and back to linear was good enough conditioned:
https://i5.imageban.ru/out/2021/09/16/1de6d40e26b13e34a50e3998f3264e74.png

Also the ringing is colored so I assume in something inside z_convertformat at primaries/matrix/transfer conversion happens. May be need more arguments in conversions to switch z_convertformat logic to convert in 'linear' ? May be avsresize can not completely revert HDR transfer and still leave OOTF part of transform and it distorts data before sinc scaling ? Though OOTF gamma is not very great so I do not think it can cause so significant distortions.

When trying to resize in HDR PQ domain the ringing swtches to colour patches from black border.

[DTL]

Well - the only that works as it expected is combination of avsresize and HDRtools:

For PQ to 709:

rgb_lin=z_convertformat(pixel_type="rgbps", colorspace_op="2020ncl:st2084:2020:l=>rgb:linear:2020:f")
rgb_lin=SincResize(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
yuv_out=ConvertLinearRGBToYUV(rgb_lin,Color=2)


Same is for HLG input with changed colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:f" at z_convertformat. So it looks linear output from avsresize is enough linear for sincresize. But may I miss some options for back convert from linear to bt.709/srgb for monitoring ?

[StvG]

LoadPlugin("avsresize.dll")

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

#z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:709:f")
z_convertformat(pixel_type="rgbp", colorspace_op="709:709:709:l=>rgb:linear:2020:l") # attempt to have primaries in 2020 before conditioning - no help

GaussResize(width/10,height/10, p=10)

#testing source
#return Crop(300,0, width-300, height).SincResize(width*4, height*4, taps=16).ConvertToRGB24(matrix="PC.709")

z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l")
#PQ HDR here

# attempt to resize in PQ HDR transfer
Crop(300,0, width-300, height).SincResize(width*4, height*4, taps=16)
return z_convertformat(pixel_type="yuv444p16", colorspace_op="2020ncl:st2084:2020:l=>709:709:709:l").ConvertToRGB24(matrix="PC.709")

# attempt to resize in linear
rgb_lin=z_convertformat(pixel_type="rgbps", colorspace_op="2020ncl:st2084:2020:l=>rgb:linear:2020:f")
rgb_lin=SincResize(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
z_convertformat(rgb_lin, pixel_type="YUV444P16", colorspace_op="rgb:linear:2020:f=>709:709:709:l")

return Crop(300,0, width-300, height).ConvertToRGB24(matrix="PC.709")


Replacing sinc kernel with another or using lower taps gives good result.

[DTL]

I assume good quality moving pictures display must be based on simple sinc kernel and enough number of taps to restore most of high valid frequencies. Unfortunately using simple sinc without any adjustments works only for luma but not for chroma (it need to be de-ringed at final end / display point with current design of common-use digital video systems).

And 'production' workflow must provide data, which do not cause (significant visible) ringing with (infinite taps) sinc interpolators.

[poisondeathray]

Well - the only that works as it expected is combination of avsresize and HDRtools:

For PQ to 709:

rgb_lin=z_convertformat(pixel_type="rgbps", colorspace_op="2020ncl:st2084:2020:l=>rgb:linear:2020:f")
rgb_lin=SincResize(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
yuv_out=ConvertLinearRGBToYUV(rgb_lin,Color=2)






Not really

1) you're not "test linear scaling of the produced by z_convertformat PQ HDR output it cause severe vertical ringing" - because now you're now performing the sinc scaling step in RGB linear, whereas before you were testing in PQ HDR YUV444P16.

2) The bars are way off in color now ; pay attention to what color they are supposed to be, not just the artifacts

Proper way to do this is perform the scaling steps in RGB linear

[poisondeathray]

1) Is the vapoursynth placebo shader sinc kernel the same as avs internal sinc? or fmtc sinc? There seem to be fewer errors, more accurate colors , even when feeding the YUV444P16 avs input to perform the sinc scaling step in vapoursynth

2) Same with fmtc r22 (r24 has a srgb transfer issue, there was a fix commit, but no new binary). sinc kernel seems more accurate

3) Same with avsresize in YUV/RGB conversion - seems to be less accurate in avs version. But both use zimg library, you'd expect same results

e.g. 8bit sRGB converted "red" should 191,0,0 . AVS version is 188,0,0. VPY version is 191,0,0

Maybe StvG can look into it because he can compile both



AVS, Sinc scaling step in RGB linear float

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:2020:l")

GaussResize(width/10,height/10, p=10)

SincResize(last.width*4, last.height*4, taps=16)

z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l")

#8bit sRGB Preview
z_convertformat(pixel_type="rgbp", colorspace_op="2020ncl:st2084:2020:l=>rgb:srgb:709:f")


note "red" is RGB is 188,0,0, "mg" is 187,0,189 - ie. everything a bit off - but absense of large ringing artifacts


AVS, Sinc scaling step in YUV444P16

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:2020:l")

GaussResize(width/10,height/10, p=10)

z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l")

SincResize(last.width*4, last.height*4, taps=16)

#8bit sRGB Preview
z_convertformat(pixel_type="rgbp", colorspace_op="2020ncl:st2084:2020:l=>rgb:srgb:709:f")


Same, but ringing artifacts


AVS YUV444P16 input script into VPY, to check VPY sinc and RGB conversions

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:2020:l")

GaussResize(width/10,height/10, p=10)

z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l")



VPY, avs input, sinc scaling step in YUV444P16

clip = core.avisource.AVISource(r'avs_input.avs')

#clip = core.placebo.Resample(clip, width=clip.width*4, height=clip.height*4, filter ="sinc", param1=16)
clip = core.fmtc.resample(clip, w=clip.width*4, h=clip.height*4, kernel="sinc", taps=16)

#8bit RGB preview
clip = core.resize.Point(clip, format=vs.RGB24, matrix_in_s="2020ncl", matrix_s="rgb", transfer_in_s="st2084", transfer_s="709", primaries_in_s="2020", primaries_s="709")

clip.set_output()


No large ringing with either placebo.Resample sinc (not sure if taps are correct with param1), or fmtc

Colors are slightly more accurate, e.g. "red" is 191,0,0, "mg" is 191,0,192

[DTL]

"AVS, Sinc scaling step in RGB linear float
z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:2020:l")
GaussResize(width/10,height/10, p=10)
SincResize(last.width*4, last.height*4, taps=16)"
"

Here I got what cause misunderstanding. My work is to create ColourBars HDR in distribution format (that is 4:2:2 SDR/HDR PQ/HLG 10bit) but to test if it is correct I need some 'ideal moving pictures display simulator' - So the sinc-upscaling must be performed with (after receiving) 'distribution' data. The GaussResize is member of production side just to condition transients of the typical awful output of ColorBars(HD). In the actual work I use not gauss and sinc but userdefined2resizemt and sinclin2resizemt but close results may be shown with built-in avisynth resizers and do not introduce more bugs from external to avisynth plugins.

So the workflow to test:

Production:ColorBarsHD->(conditioning_in_linear_domain and generate output size)->convert_to_HDR_PQ(primaries,transfer,matrix)->standard(sharp_cut-off_chromaLPF)_subsample_to_4:2:2->Bitdepth_reduction_to10->Release.

Quality control: Release->Convert_to_4:4:4(monitorLPF_slow_roll-off for UV)->Bitdepth_upto_16->Convert_to_linear(transfer,primaries(?),matrix(?))->4:4:4 Sinc upscale (to some larger size like 4x)->Put_to_display(convert to sRGB/709 (primaries,transfer,matrix) for standard pixel display feed).

Used standard 4:4:4 to 4:2:2 conversion is simple and equal for all versions with sharp sincresize cut-off in UV:

Function ConvertTo422p16(clip c)
{
yuv444=ConvertToYUV444(c)
uc=UToY(yuv444)
vc=VToY(yuv444)
uc=SincLin2ResizeMT(uc,src_left=0, uc.width/2, uc.height, taps=16)
vc=SincLin2ResizeMT(vc,src_left=0, vc.width/2, vc.height, taps=16)
return CombinePlanes(yuv444, uc, vc, planes="YUV", source_planes="YYY", pixel_type="YUV422P16")
}



My current 'quality_control' functions for 709/HDR_PQ/HDR_HLG is

Function Convert422ToRGB24mon_lin_x4_709(clip c)
{
uc=UToY(c)
vc=VToY(c)
# uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=95, c=-10)
# vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=95, c=-10)
uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=125, c=18)
vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=125, c=18)
# uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=80, c=-20)
# vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=80, c=-20)
uc=SincLin2ResizeMT(uc, src_left=0, uc.width*2, uc.height, taps=16)
vc=SincLin2ResizeMT(vc, src_left=0, vc.width*2, vc.height, taps=16)
yuv444=CombinePlanes(c, uc, vc, planes="YUV", source_planes="YYY", pixel_type="YUV444P10")
rgb_lin=ConvertYUVToLinearRGB(yuv444,Color=2)
rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
yuv_out=ConvertLinearRGBToYUV(rgb_lin,Color=2)
return yuv_out.ConvertToRGB24(matrix="PC.709")
}

Function Convert422ToRGB24mon_lin_x4_PQ(clip c)
{
uc=UToY(c)
vc=VToY(c)
# uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=95, c=-10)
# vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=95, c=-10)
uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=125, c=18)
vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=125, c=18)
# uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=80, c=-20)
# vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=80, c=-20)
uc=SincLin2ResizeMT(uc, src_left=0, uc.width*2, uc.height, taps=16)
vc=SincLin2ResizeMT(vc, src_left=0, vc.width*2, vc.height, taps=16)
yuv444=CombinePlanes(c, uc, vc, planes="YUV", source_planes="YYY", pixel_type="YUV444P10")
rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:st2084:2020:l=>rgb:linear:2020:f")
rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
yuv_out=ConvertLinearRGBToYUV(rgb_lin,Color=2)
return yuv_out.ConvertToRGB24(matrix="PC.709")
}

Function Convert422ToRGB24mon_lin_x4_HLG(clip c)
{
uc=UToY(c)
vc=VToY(c)
# uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=95, c=-10)
# vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=95, c=-10)
uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=125, c=18)
vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=125, c=18)
# uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=80, c=-20)
# vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=80, c=-20)
uc=SincLin2ResizeMT(uc, src_left=0, uc.width*2, uc.height, taps=16)
vc=SincLin2ResizeMT(vc, src_left=0, vc.width*2, vc.height, taps=16)
yuv444=CombinePlanes(c, uc, vc, planes="YUV", source_planes="YYY", pixel_type="YUV444P10")
rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:f")
rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
yuv_out=ConvertLinearRGBToYUV(rgb_lin,Color=2)
return yuv_out.ConvertToRGB24(matrix="PC.709")
}


The task of pereserving exact RGB codevalues is not main. The main task is to check presence and possible quality of required colour-difference anti-ringing filter in the control monitor (being feed with 'Release' dataset). So some errors in exact code values of colour patches are acceptable. More important is relative position of Y and UV to make colour transients as great as possible. Currently I not very happy with some black gaps around blue patch for example. My current idea - the build-in ColorBars(HD) datasource produces not very perfect data and may be it is good to start from RGB manually designed in MS-Paint and it allow to adjust position of RGB levels transients in each RGB channel down to 0.1 in-between_samples step independently when using /10 downsampling at production. May it can help to get more perfect transients.

"note "red" is RGB is 188,0,0, "mg" is 187,0,189 - ie. everything a bit off - but absense of large ringing artifacts"

It is just testing of the quality of transients of source at 'Production' side - to proof if it generate good conditioned dataset and we have a chance to got some close quality at end of chain side after we create 4:2:2 HDR PQ or HLG 10bit release dataset (compressed in bitdepth and spatially in colour-difference).

"AVS, Sinc scaling step in YUV444P16
colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l"
Same, but ringing artifacts
"

Here it not only YUV444P16 but also transfer converted from linear to PQ-domain, so it losts conditioning (prepared in linear domain) and starts to ring with sinc-resize.

"VPY, avs input, sinc scaling step in YUV444P16"

It is also not only YUV444P16 but transfer converted to PQ-domain (in the end of avisynth script with z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l")) so sinc-resize in PQ-domain will ring. To simulate correct display-side scaling transform back to linear domain required.

[poisondeathray]

The task of pereserving exact RGB codevalues is not main. The main task is to check presence and possible quality of required colour-difference anti-ringing filter in the control monitor (being feed with 'Release' dataset). So some errors in exact code values of colour patches are acceptable. More important is relative position of Y and UV to make colour transients as great as possible. Currently I not very happy with some black gaps around blue patch for example. My current idea - the build-in ColorBars(HD) datasource produces not very perfect data and may be it is good to start from RGB manually designed in MS-Paint and it allow to adjust position of RGB levels transients in each RGB channel down to 0.1 in-between_samples step independently when using /10 downsampling at production. May it can help to get more perfect transients.


ok... but when the code values are +/- 30 off, it's a big deal (!!!) . +/- 3 in 8bit is acceptable tolerance. The main reason for colorbars in the first place are for... color accuracy

The primaries are not adjusted in ConvertLinearRGBToYUV, I think that is the main reason for the large color errors in post #4. But how do you know the errors that you see are not a cascade of errors effect ? eg. The primaries omission was the cause for the 1st error in the 1st post

The above post demonstrates 16tap sinc works ok in YUV444P16 in vapoursynth fmtc and resize.Shader without the large ringing seen avisynth. So maybe it's an avisynth sinc bug? Or maybe the sinc implmentation is not the same

I point out differences in zimg, because how do you know those errors upstream are not cascading and producing larger errors in avisynth downstream when you use sinc or anything else ? It's the same library, but the same operation (e.g. omit all scaling) produces different results

[DTL]

"The main reason for colorbars in the first place are for... color accuracy"

In the target testing the colour accuracy is handled with separate large number of colour full-screen patches. This colorbar bar is added only to test scaling (in colour mode, not only black and white). The shape of colorbars pattern is just something containing colour transients with high amplitude colour-difference data walking and so if no required filtering applied and depends on the monitor scaler it may shows (small) ringing. Unfortunately at the awfull HDR epoch and with freeware tools it still better in compare with nothing. If there were any table of the HDR colorbars (of some exact type) in PQ and HLG (of some exact white/black and other variables) it can be compared in the release 4:2:2 10bit files for total code value error. But I think non of official tables still exist. Having perfect decoded levels accuracy is sort of 'nice to have' feature for now I think.

"maybe the sinc implmentation is not the same"

Yes - there at least 2 ways possible:
1. anti-ringing pre-filtering before sinc (not clear sinc so it will show fading of frequency responce at highest valid frequencies and a bit less sharpness)
2. it really do not scale in the input HDR-transfer but perform converting to linear inside. Though it need to know the current transfer-domain of the input content to perform converting to linear and back.

" without the large ringing seen avisynth."

Avisynth sinc mostly probably do not perform non-requested conversions to linear domain (and back) and not perform pre-filtering so if input content is not anti-ringing conditioned it will expose full ringing.

[poisondeathray]

If there were any table of the HDR colorbars (of some exact type) in PQ and HLG (of some exact white/black and other variables) it can be compared in the release 4:2:2 10bit files for total code value error. But I think non of official tables still exist. Having perfect decoded levels accuracy is sort of 'nice to have' feature for now I think.

Vapoursynth colorbars are exact, because they are synthetically generated according to SMPTE RP 219-1, 219-2 and ITU-R BT.2111-0 . The diagram and exact 10,12 bit code values are listed in the later for PQ narrow/full and HLG

https://github.com/ifb/vapoursynth-colorbars

You can import vpy scripts in to avs for many pixel types with VSImport("script.vpy")



"maybe the sinc implmentation is not the same"

Yes - there at least 2 ways possible:
1. anti-ringing pre-filtering before sinc (not clear sinc so it will show fading of frequency responce at highest valid frequencies and a bit less sharpness)
2. it really do not scale in the input HDR-transfer but perform converting to linear inside. Though it need to know the current transfer-domain of the input content to perform converting to linear and back.

" without the large ringing seen avisynth."

Avisynth sinc mostly probably do not perform non-requested conversions to linear domain (and back) and not perform pre-filtering so if input content is not anti-ringing conditioned it will expose full ringing.


It does not look like they do, taking a quick look at the code. But I might have missed something

If you replace internal SincResize with

fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)

it does not exhibit the large ringing patterns either


avs internal code I think is this

https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/resample_functions.cpp

]
/***********************
*** Sinc filter ***
***********************/
SincFilter::SincFilter(int _taps) {
taps = (double)clamp(_taps, 1, 20);
}

double SincFilter::f(double value) {
value = fabs(value);

if (value > 0.000001) {
value *= M_PI;
return sin(value)/value;
} else {
return 1.0;
}
}

[DTL]

"ITU-R BT.2111-0 . The diagram and exact 10,12 bit code values are listed in the later for PQ narrow/full and HLG"

Well - it is good to see. But the tables listed only R'G'B' values - not YUV in actual distribution formats. For manual checking in hex-editor in the exported 4:2:2 raw files.
Also to be mentioned - in the more and more degrading world the https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.2111-2-202012-I!!PDF-E.pdf looks like finally missed completely description about transients between levels. The old standards for HD and SD colorbars at least put some words about it. If I have time I will try to make R'G'B' to YUV table to check for levels in the 4:2:2 YUV files.
Also tables list only sustained code values and can not cover the transients because of too much numbers - so anyway producing of real test patterns require transients conditioning with software-processing and it need to be done in RGB-linear (not in R'G'B' even) and later subsampled to typical 4:2:2.

"If you replace internal SincResize with
fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)
it does not exhibit the large ringing patterns either
"

To check quality of sincing I use 1-sample pattern (like 235 code value at 16 field or 200 at 30 or like this to see result at any sRGB-full monitor without additional black-raising).
PNG file is
https://i1.imageban.ru/out/2021/09/17/685005102de17509948c9adf9e7d591d.png

LoadPlugin("fmtcavs.dll")

ImageReader("1-sample.png", pixel_type="RGB24")

#SincResize(width*4, height*4, taps=16)
ConvertToPlanarRGB()
fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)
fmtc_bitdepth(bits=8)

ConvertToRGB24()

Levels(0,1,60,0,255)

I do not see any significant difference in kernel response between avisynth built-in sincresize and fmtc_convert(sinc)
https://i5.imageban.ru/out/2021/09/17/44179dd1a9fde83a39dbcb348c9219d3.png

They looks like match close to bitexact:

avs=SincResize(width*4, height*4, taps=16)
fmtc=ConvertToPlanarRGB()
fmtc=fmtc_resample(fmtc,w=last.width*4, h=last.height*4, kernel="sinc", taps=16)
fmtc=fmtc_bitdepth(fmtc,bits=8)

fmtc=ConvertToRGB24(fmtc)
Subtract(avs,fmtc)

Produces flat grey field.
So may be some other magic work somewhere. Though I made black and white Y-data test. May be with colour data results are different.

[poisondeathray]

"If you replace internal SincResize with
fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)
it does not exhibit the large ringing patterns either
"

They looks like match close to bitexact:


That comment was referring to the earlier example in YUV444P16. Check internal SincResize vs. FMTC. Why such a big difference ?




ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:2020:l")

GaussResize(width/10,height/10, p=10)

#SincResize(last.width*4, last.height*4, taps=16)
fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)

#8bit sRGB Preview
z_convertformat(pixel_type="rgbp", colorspace_op="2020ncl:st2084:2020:l=>rgb:srgb:709:f")

[DTL]

" Check internal SincResize vs. FMTC. Why such a big difference ?"

The provided script do not work for me to control in VirtualDub for some reason. But I made return after sinc proc

LoadPlugin("avsresize.dll")
LoadPlugin("fmtcavs.dll")

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(300,300,300,300) # here enlarge from 100 to 300.

z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:2020:l")

GaussResize(width/10,height/10, p=10)

SincResize(last.width*4, last.height*4, taps=16)
#fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)

return ConvertToRGB24(matrix="PC.709")


and see the difference between sincresize and fmtc at the top/bottom borders and frame left and right ringing. I think it is because of different edge-workarounds in different resamplers.
Current black borders are 100/10=10 and taps=16 > width of border.

When I enlarge border to AddBorders(300,300,300,300) the SincResize gives also non-ringing result. It may be reported as sort of issue to avisynth development - may be as another fix to resampler engine. Though it may allow additional resources and slow-down processing. Typically users do not look at the very edge of frame and center of frame processed well.

As for the lowest size of required black border: A.5 of ARIB-STD-B28 Transient
The number of samples to be used for the transient shall be 6 to 9, in the case of 1920
horizontal samples, although it may depend upon the scale of hardware, process performance
and the so-called “make up”. (It was about 20 years old when degradation of industry was lower in compare with todays)

So I hope 10 is enough to have up to 9 samples symmetrical to useful data.

[StvG]

e.g. 8bit sRGB converted "red" should 191,0,0 . AVS version is 188,0,0. VPY version is 191,0,0

Maybe StvG can look into it because he can compile both

I cannot reproduce the difference.

Are you sure you used the very same conversion for both AVS and VS? You have destination transfer "srgb" in the final step for AVS but for VS it's "709".


When I enlarge border to AddBorders(300,300,300,300) the SincResize gives also non-ringing result. It may be reported as sort of issue to avisynth development - may be as another fix to resampler engine. Though it may allow additional resources and slow-down processing. Typically users do not look at the very edge of frame and center of frame processed well.

The visible difference is not only at the very edge of the frame.

Internal avs sinc:
https://i.slow.pics/kQyCnBCF.png

fmtc_resample(kernel="sinc")
https://i.slow.pics/aKH6Gy3e.png

Click for the full size images. Both examples are from 16-bit processing.

[DTL]

That comment was referring to the earlier example in YUV444P16. Check internal SincResize vs. FMTC. Why such a big difference ?



ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(100,100,100,100)

z_convertformat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:linear:2020:l")

GaussResize(width/10,height/10, p=10)

#SincResize(last.width*4, last.height*4, taps=16)
fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)

#8bit sRGB Preview
z_convertformat(pixel_type="rgbp", colorspace_op="2020ncl:st2084:2020:l=>rgb:srgb:709:f")


At the morning I see the error - the first z_convert if from 709 to linear but last z_convert is from st2084 to srgb. So it produce black or grey field.
After correction I see the SincResize really produced additional ringing from the edges even with 300 initial borders and it is colored. FMTC is clear. Will try to report this issue but I unsure if the resampler of avisynth can be fixed quickly - it is very complex and highly asm-optimized. And FMTC may use slower simple float logic.
The conversion to linear and back only shows the bug better.
It also may be seen without converting to linear and back:

LoadPlugin("fmtcavs.dll")

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(300,300,300,300)

GaussResize(width/10,height/10, p=10)

SincResize(last.width*4, last.height*4, taps=16)
#fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)

ConvertBits(8)

Levels(0,1,20,0,255).Crop(1000,0,width-1000, height).ConvertToRGB24(matrix="PC.709")

[poisondeathray]

You have destination transfer "srgb" in the final step for AVS but for VS it's "709".


Thanks, nice catch - they match

[poisondeathray]

At the morning I see the error - the first z_convert if from 709 to linear but last z_convert is from st2084 to srgb.


Sorry, I copied /pasted incorrectly


There are other differences in the result using placebo shader implementation of sinc also

What is the expected result in other programs, say matlab ?

[DTL]

Because of still existing bugs in avisynth resampler I made temporal workaround function SafeSincResize() . It auto-adds padding and crops after resize. But only works with sources with already having some black borders.

Now it looks output of both fmtc_resample(sinc) and SafeSincResize is identical as I see after Subtract()


LoadPlugin("avsresize.dll")
LoadPlugin("fmtcavs.dll")

Function SafeSincResize(clip c, int width, int height, int taps)
{
xratio = width/c.width
yratio = height/c.height
tt=2*taps
c=AddBorders(c,tt,tt,tt,tt)
c=SincResize(c, c.width*xratio, c.height*yratio, taps=taps)
return Crop(c,tt*xratio,tt*yratio,width,height)
}

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(200,200,200,200)

z_convertformat(pixel_type="rgbp", colorspace_op="709:709:709:l=>rgb:linear:2020:l")

GaussResize(width/10,height/10, p=10)

z_convertformat(pixel_type="yuv444p16", colorspace_op="rgb:linear:2020:l=>2020ncl:st2084:2020:l")
#PQ HDR here

rgb_lin=z_convertformat(pixel_type="rgbps", colorspace_op="2020ncl:st2084:2020:l=>rgb:linear:2020:l")
rgb_lin_ss=SafeSincResize(rgb_lin, rgb_lin.width*4, rgb_lin.height*4, 16)
rgb_lin_fmtc=fmtc_resample(rgb_lin,w=last.width*4, h=last.height*4, kernel="sinc", taps=16)
out_ss=z_convertformat(rgb_lin_ss, pixel_type="YUV444P16", colorspace_op="rgb:linear:2020:f=>709:709:709:l")
out_fmtc=z_convertformat(rgb_lin_fmtc, pixel_type="YUV444P16", colorspace_op="rgb:linear:2020:f=>709:709:709:l")

Subtract(out_ss, out_fmtc)

return ConvertToRGB24(matrix="PC.709")



"There are other differences in the result using placebo shader implementation of sinc also"

Can you show example of that shader output ? I do not have vapoursynth.

Also I found more execution non-stability of SincResize at different CPUs ans OS version - at about Core2 6400 CPU and Win 7 https://i3.imageban.ru/out/2021/09/17/0097b84914ff6256cd2288d58ca4e007.png and at i5-9600 and Win10 https://i4.imageban.ru/out/2021/09/18/07c0cf04d5f8fa3ed7f4026c2093b402.png .

"the expected result in other programs, say matlab ?"

In theory sinc interpolation is very simple sum of weighted by input samples sincs and have to match between different programs with equal size of kernel. May be some difference because of different order of V and H passes at 2D simulated processing as V+H 1D+1D for speed instead of 2D one pass. But 2D one pass will produce significant different results (assuming sinc_2d=sinc(radius)) and will be slower in speed.

"The visible difference is not only at the very edge of the frame."

It looks like poor of high frequences source. Can you process some test pattern like hyperbolic zoneplate ? It linear raw calculator is https://github.com/DTL2020/hpzp or converted to 709 transfer h264 encoding is https://cloud.mail.ru/public/h1kR/o56qALaK5

[poisondeathray]

Can you show example of that shader output ? I do not have vapoursynth.


This is FFV1, YUV444P16 output of placebo.Resample Sinc 16 tap. You can open in AVS with LWLibavVideoSource, or FFVideoSource. It looks worse to me. FMTC was bit identical to avs version
https://www.mediafire.com/file/uy2k1cynkc5325r/test3shader.mkv/file

avs input to vpy script

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(300,300,300,300)

GaussResize(width/10,height/10, p=10)


vpy

clip = core.placebo.Resample(clip, width=clip.width*4, height=clip.height*4, filter ="sinc", param1=16)




Also I found more execution non-stability of SincResize at different CPUs ans OS version
Maybe different path execution, SIMD, instruction sets ? Was there a way to disable ? In vapoursynth there was cpu_type="none" , IIRC there was similar switch in avs for debugging

[DTL]

e.g. 8bit sRGB converted "red" should 191,0,0 . AVS version is 188,0,0. VPY version is 191,0,0


I made calculation of 8bit codevalues for HDR colorbars (that I can design in MS-Paint and save as 8bit BMP/PNG) for 75% (brightness):

HLG narrow: 40% grey - 104, R'G'B' - 180 high, 16 low
PQ narrow: 40% grey - 104, R'G'B' - 143 high, 16 low
PQ full: 40% grey - 102, R'G'B' - 148 high, 0 low

And target 10bit YUV table for output now this (OpenOffice Calc designed)
https://i7.imageban.ru/out/2021/09/18/a47a286a8394e21cba7175a87caeafa0.png

"It looks worse to me."

Yes - it have some ringing on horizontal transients. And it is strange because the data after GaussResize have equal conditioning of vertical and horizontal transients.
May be some codevalues distorition happens when transfer from avisynth to vapoursynth. Anyway it may be task to vapoursynth shader designers to check.
Also difference between avs and vps in subtract operation is also with transients shape and levels (causing coloring also)

vps=FFMpegSource2("test3shader.mkv")
vps=vps.ConvertToRGB24(matrix="PC.709")
ConvertToRGB24(matrix="PC.709")
Subtract(last,vps)

Image: https://i5.imageban.ru/out/2021/09/18/f73f3ab3cef4049cfbbace054da523c4.png
So it looks there are really lots of codevalues changes somwhere.

[DTL]

It looks I understand why it is so hard to get nice color-bars in subsampled form. The typically good looking only 1 of 3 RGB transients like white-yellow (B-only transient) and cyan-green (B-only transient). But with 2 or 3 transients at once it much more awful. It looks like different rise and fall speed of full-band Y/G and half band R and B makes bright and dark shadows on many color transients.
The green-magenta is something like (not very exact/symmethrical created in MS-paint with lines and curves)
https://i4.imageban.ru/out/2021/09/18/9f982bf5a897cfe985dcf6107884aebe.png

So if any component is sub-sample misaligned - we got significant Y-error out of expected shape.

So I think of attempt of limiting band of Y so sort of 2:2:2 (in faked 4:2:2) may be look better - need to test. Current result still no good.

[DTL]

Made 2 tools for manual transients design (in SDR and PQ/HLG HDR transfers/2020-matrix):
Web-based simulator https://github.com/DTL2020/TransientsDesign
Tool for write raw files from text-defined samples: https://github.com/DTL2020/WriteTransients
It have built-in chroma anti-ring filter I hope close to UserDefined2Resize(b=105,c=0).

Green-magenta transient in SDR looks like
https://i1.imageban.ru/out/2021/09/30/7e2cbf8432de85906aa423591a84e339.png

So typical display-simulator script is:

Function Convert422ToRGB24mon_lin_x4_709(clip c)
{
uc=UToY(c)
vc=VToY(c)
uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=105, c=0)
vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=105, c=0)
uc=SincLin2ResizeMT(uc, src_left=0, uc.width*2, uc.height, taps=8)
vc=SincLin2ResizeMT(vc, src_left=0, vc.width*2, vc.height, taps=8)
yuv444=CombinePlanes(c, uc, vc, planes="YUV", source_planes="YYY", pixel_type="YUV444P10")
rgb_lin=ConvertYUVToLinearRGB(yuv444,Color=2)
rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
yuv_out=ConvertLinearRGBToYUV(rgb_lin,Color=2)
return yuv_out.ConvertToRGB24(matrix="PC.709")
}


Chroma position relative to Y' looks like not require 0.5..1 sample shift at conversion to 4:4:4 (not sure if it correct).

[DTL]

Now with ability to put transient sourcing to scripting (instead of loading external raw-file) it is easier to test different HLG to linear convertors (for monitoring).


LoadPlugin("plugins_JPSDR.dll")
LoadPlugin("avsresize.dll")
LoadPlugin("fmtcavs.dll")

Function Convert422ToRGB24mon_lin_x4_HLG(clip c)
{
uc=UToY(c)
vc=VToY(c)
uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=105, c=0)
vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=105, c=0)
uc=SincLin2ResizeMT(uc, src_left=0, uc.width*2, uc.height, taps=8)
vc=SincLin2ResizeMT(vc, src_left=0, vc.width*2, vc.height, taps=8)
yuv444=CombinePlanes(c, uc, vc, planes="YUV", source_planes="YYY", pixel_type="YUV444P10")

# jpsdr_plugin
# rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2)
# rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

# avsresize
# rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", nominal_luminance=300)

#fmtconv
rgb_lin=fmtc_matrix(yuv444,mat="RGB", mats="2020",fulls=true, fulld=true)
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear",bits=32)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
yuv_out=ConvertLinearRGBToYUV(rgb_lin,Color=2)
return yuv_out.ConvertToRGB24(matrix="PC.709")
}

plY=Blankclip(width = 10, height = 1, pixel_type="Y10").Expr("sx 0 == 509 sx 1 == 509 sx 2 == 509 sx 3 == 509 sx 4 == 509 sx 5 = 498 sx 6 = 446 sx 7 == 361 sx 8 == 292 276 ? ? ? ? ? ? ? ? ?")
plU=Blankclip(width = 5, height = 1, pixel_type="Y10").Expr("sx 0 == 270 sx 1 == 263 sx 2 == 330 sx 3 == 761 754 ? ? ? ?")
plV=Blankclip(width = 5, height = 1, pixel_type="Y10").Expr("sx 0 == 203 sx 1 == 224 sx 2 == 271 sx 3 == 832 821 ? ? ? ?")
tr=CombinePlanes(plY, plU, plV, "YUV", "YYY", pixel_type="YUV422P10")

sust_l=Blankclip(width = 4, height = 1, pixel_type="YUV422P10").Expr("sx 0 == 509 sx 1 == 509 sx 2 == 509 509 ? ? ?","sx 0 == 270 270 ?","sx 0 == 203 203 ?").PointResize(50,1)
sust_r=Blankclip(width = 4, height = 1, pixel_type="YUV422P10").Expr("sx 0 == 276 sx 1 == 276 sx 2 == 276 276 ? ? ?","sx 0 == 754 754 ?","sx 0 == 821 821 ?").PointResize(52,1)

StackHorizontal(sust_l,tr)
StackHorizontal(last,sust_r)
PointResize(width, 100)
# source finishes here, 75% green-magenta transient HLG 10bit 4:2:2

#test convert to 4:4:4 and back
#ConvertToYUV444()
#ConvertToYUV422()

Convert422ToRGB24mon_lin_x4_HLG()


Results of decoding same green-magenta 75% transient to linear RGBPS by different tools are a bit different:
https://i2.imageban.ru/out/2021/10/02/26f0640c8947aa252b6398cc24ed1a49.png
jpsdr plugin and fmtc looks close enough and z_convertformat looks like differs most. May be its 'std-b67' is not really linear but also contain OOTF ? Currently the most close to as expected result is from fmtc.
Both jpsdr convert and fmtc require RGB gain 3.0 to bring intensity to something >50%.
z_convertformat have some misterious param 'nominal_luminance' that greatly infuence the output result but no adjustment for me can bring result close to the other 2 tools.

[StvG]

jpsdr plugin and fmtc looks close enough and z_convertformat looks like differs most. May be its 'std-b67' is not really linear but also contain OOTF ?

https://github.com/sekrit-twc/zimg/issues/71#issuecomment-319853981

approximate_gamma also changes the output.

[DTL]

"approximate_gamma also changes the output."

Yes - it significally changes sustained levels and transient. Looks make things better.
https://i2.imageban.ru/out/2021/10/03/2c6c44756add08c58deecdccb2ff6b1d.png

Strange why default is not 'false'. Desciption at http://avisynth.nl/index.php/Avsresize says only about reduced precision. I think it is something about 1 LSB error in 16bit. But it may be slower so default is 'true' for speed ?

Anyway - one HLG-encoded dataset, one processing path and so different results with different conversion HLG to linear tools. It looks HDR workflows is still a big mess.
The colour tone and saturation of magenta field also different a lot. May be de-matrix transforms also not very equal ?

I tried to set available control params for HLG to linear conversion to get both good looking of transient and sustained levels of green and magenta. May be better to get best possible RGB levels of sustained green (R=B=0 G=some_level) and magenta (R=B=same_some_level G=0) and look how transient rendering will be.

[StvG]

Yes, approximate_gamma=false is slow. That's the reason to be true by default.

I suggest to use the filter for yuv_HLG->linear to reverse linear->yuv too.
So for avsresize: ...
# avsresize
# rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", nominal_luminance=1000, approximate_gamma=false)
...
yuv_out=z_ConvertFormat(rgb_lin, pixel_type="yuv444ps", colorspace_op="rgb:linear:2020:l=>709:709:709:l")
...

Now you have different output again.

Something about mixing of filters: I would suggest to not use avsresize/ftmconv (didn't tested the last) to perfrom PQ->linear and then tonemapping with HDRTools. If HDRTools is used for tonemapping, use this filter for everything (yuv->linear->tonemapping). If DGx filters are used for tonemapping, use avsresize for yuv->linear.

[DTL]

"Now you have different output again."

Yes - in this script we not control result in linear RGB but pack it again to rec.709 system to feed close to rec.709 typical display (at 2021 year). So may be additional errors.

"yuv_out=z_ConvertFormat(rgb_lin, pixel_type="yuv444ps", colorspace_op="rgb:linear:2020:l=>709:709:709:l")"

It looks again some misunderstanding. We test colorbars defined as 0..HLG_inverse(0.75) RGB triplets (and transients between this sustained values). Its numbers may define any physical primaries. But when we command system to convert prmaries from 2020 to 709 it creates mess of RGB triplets (also may be also some limited by 709 top values because 2020 colour space > 709 colour space and 0.75 100% saturated HLG is most possibly out of 709 colour space). So all transients will be damaged too.

We are not monitor exact rec.2020 physical spectrum (it is not possible on 709 monitors) but want to check if transients do not have ringing/large over/under shooting, parasitic bright or dark lines etc. So to try it we 'cast' rec.2020 RGB triplets to our physical rec.709 display output. It will display rec.709 spectrum colourbars but I hope the Intensity on transients will more or less follow the output of real rec.2020 primaries display. Though in best case it need to be checked with colorimetry calculation.

So for 'casting' decoded RGB linear primaries to current display's primaries we just do not touch it:
Both
yuv_out=z_ConvertFormat(rgb_lin, pixel_type="yuv444ps", colorspace_op="rgb:linear:2020:l=>709:709:2020:l")
yuv_out=z_ConvertFormat(rgb_lin, pixel_type="yuv444ps", colorspace_op="rgb:linear:709:l=>709:709:709:l")
gives the same result.

Though it also differs from jpsdr's plugin.


Because jpsdr's plugin can not (I do not know) apply only transfer to RGB without matrix convert to YUV it looks no way to test transfer only. We not need to monitor to go down to YUV matriced data and back so better return from monitor function is
return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:l=>rgb:709:709:l").ConvertToRGB() (I do not see how to output interleaved RGB from z_convertformat directly)
Or with fmtc:
return fmtc_transfer(rgb_lin,transs="linear",transd="709",fulls=true, fulld=true).ConvertBits(8).ConvertToRGB()
https://i3.imageban.ru/out/2021/10/03/f199bb4a4b19002cbfd84c93b224abb1.png
It is strange but fmtc again looks the best (almost same as jpsdr's with going to YUV and back). For unknown reason z_convertformat output have lot of low-amplitude ringing (may be it come from x4 upsize or not).

[DTL]

"I suggest to use the filter for yuv_HLG->linear to reverse linear->yuv too."

Why ? When we monitor we do not need to subsample and any more resample - so no additional filtering. The task of processing after getting linear RGB in restored 4x upsampled size is just feed our RGB monitor (that can not still accept linear RGB so we still need to convert RGB to R'G'B' in the transfer of the used monitor that on PC's typically srgb/rec.709).
Also at the last simplification if not use jpsdr's plugin it is no more need YUV stage after linear RGB at all.

[StvG]

Because jpsdr's plugin can not (I do not know) apply only transfer to RGB without matrix convert to YUV it looks no way to test transfer only.

Right, HDRTools cannot convert only transfer.

"I suggest to use the filter for yuv_HLG->linear to reverse linear->yuv too."

Why ?

I meant the same filter should be used for both operations: convert_to_linear and convert_from_linear.

You're using full range for fmtc bul limited for avsresize.
You can try:# avsresize
rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:f=>rgb:linear:2020:f", nominal_luminance=800)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)
...
return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:f=>rgb:709:709:f").ConvertToRGB()
#fmtconv
rgb_lin=fmtc_matrix(yuv444,mat="RGB", mats="2020",fulls=true, fulld=true)
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear",bits=32)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)
...
return fmtc_transfer(rgb_lin,transs="linear",transd="709",fulls=true, fulld=true).ConvertBits(8).ConvertToRGB()

[DTL]

"I meant the same filter should be used for both operations: "

Oh - I think of 'filter' as engineering device/operation for spectrum changing (filtering) and the only applied spatial frequency filter here is for anti-ringing of subsampled UV. And you use Avisynth filter term as any command operation in Avisynth.

Well - the

rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", nominal_luminance=800)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:l=>rgb:709:709:l").ConvertToRGB()


The only 'magic' combination that produce close result to fmtc and jpsdr's plugin (without conversion params adjustments but with RGB gain = 3.0).
I later think nominal_luminance may be same adjustment as RGB gain after z_convertformat, but it looks no.
So the main important and hidden param to z_convertformat for HLG to linear is "nominal_luminance=800".

[StvG]

Why are you using avsresize with limited range but fmtc with full?

[DTL]

Why are you using avsresize with limited range but fmtc with full?

Oh - it looks my error of too bad reading documentation. I think fulls/fulld = true as of 'limited'.
Also somwhere set it, somwhere forgot. jpsdr's plugin uses default=limited, fmtc_transfer uses default=full(true), fmtc_matrix (depends). z_convertformat have no default ?

With sustained R'G'B' output still a big mess:
I hope the encoded HLG in 'limited' range so output low-RGB (limited 8bit) should be 16 (and max_RGB not exactly defined because of some 3.0 multiplier to RGB linear to make image bright enough, though maxR=maxG=maxB).

rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)
.. scale x4
return ConvertLinearRGBToYUV(rgb_lin,Color=2).ConvertToRGB24(matrix="PC.709")

returns green R'G'B' 0, 205,13, magenta R'G'B' 201,11,213 - I think it some matrix and transfer errors ?


rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", nominal_luminance=800)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

.. scale x4

return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:l=>rgb:709:709:l").ConvertToRGB()

returns green R'G'B' 16,211,16 and magenta R'G'B' 212,16,212 that looks most perfect and as expected.


rgb_lin=fmtc_matrix(yuv444,mat="RGB", mats="2020",fulls=false, fulld=false)
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear",bits=32,fulls=false, fulld=false)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

.. scale x4

return fmtc_transfer(rgb_lin,transs="linear",transd="709",fulls=false, fulld=false).ConvertBits(8).ConvertToRGB()

returns green R'G'B' 0,227,0 and magenta R'G'B' 228,0,228 that looks like somewhere scaled to 'full' though I everywhere set full*=false. May I need to report issue to fmtc development thread ? (post a question - https://forum.doom9.org/showthread.php?p=1953894#post1953894 ).

[DTL]

Oh - the fmtc require some (undocumented magic) of Avisynth to work as expected:


return fmtc_transfer(rgb_lin,transs="linear",transd="709",fulls=false, fulld=false).ConvertBits(8,fulls=true,fulld=false).ConvertToRGB()


Now it outputs also
green R'G'B'=16,211,16 and magenta R'G'B' =211,16,211.

Need post request for update docs at http://avisynth.nl/index.php/ConvertBits .

Current HLG monitor function now:

Function Convert422ToRGB24mon_lin_x4_HLG(clip c)
{
uc=UToY(c)
vc=VToY(c)
uc=UserDefined2ResizeMT(uc,src_left=0.001,uc.width, uc.height, b=105, c=0)
vc=UserDefined2ResizeMT(vc,src_left=0.001,vc.width, vc.height, b=105, c=0)
uc=SincLin2ResizeMT(uc, src_left=0, uc.width*2, uc.height, taps=8)
vc=SincLin2ResizeMT(vc, src_left=0, vc.width*2, vc.height, taps=8)
yuv444=CombinePlanes(c, uc, vc, planes="YUV", source_planes="YYY", pixel_type="YUV444P10")

# jpsdr_plugin
# rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2)
# rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

# avsresize
#rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", nominal_luminance=800)
#rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

#fmtconv
rgb_lin=fmtc_matrix(yuv444,mat="RGB", mats="2020",fulls=false, fulld=false)
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear",bits=32,fulls=false, fulld=false)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
#return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:l=>rgb:709:709:l").ConvertToRGB()
return fmtc_transfer(rgb_lin,transs="linear",transd="709",fulls=false, fulld=false).ConvertBits(8,fulls=true,fulld=false).ConvertToRGB()
# return ConvertLinearRGBToYUV(rgb_lin,Color=2).ConvertToRGB24(matrix="PC.709")
}


fmtc again wins in sustained levels precision (very small - 1LSB at 8bit at some RGB channels). z_convertformat (very small) better with less visual intensity falling at the green-magenta transient.
jpsdr's plugin is subject to many questions.

And 2 of 3 full-3 primaries changing at transient examples:

# red-cyan
plY=Blankclip(width = 10, height = 1, pixel_type="Y10").Expr("sx 0 == 240 sx 1 == 252 sx 2 == 283 sx 3 == 359 sx 4 == 418 sx 5 = 495 sx 6 = 545 sx 7 == 548 sx 8 == 548 548 ? ? ? ? ? ? ? ? ?")
plU=Blankclip(width = 5, height = 1, pixel_type="Y10").Expr("sx 0 == 418 sx 1 == 418 sx 2 == 432 sx 3 == 600 606 ? ? ? ?")
plV=Blankclip(width = 5, height = 1, pixel_type="Y10").Expr("sx 0 == 848 sx 1 == 844 sx 2 == 799 sx 3 == 184 176 ? ? ? ?")
tr=CombinePlanes(plY, plU, plV, "YUV", "YYY", pixel_type="YUV422P10")

sust_l=Blankclip(width = 4, height = 1, pixel_type="YUV422P10").Expr("sx 0 == 237 sx 1 == 237 sx 2 == 237 237 ? ? ?","sx 0 == 418 418 ?","sx 0 == 848 848 ?").PointResize(50,1)
sust_r=Blankclip(width = 4, height = 1, pixel_type="YUV422P10").Expr("sx 0 == 548 sx 1 == 548 sx 2 == 548 548 ? ? ?","sx 0 == 606 606 ?","sx 0 == 176 176 ?").PointResize(52,1)

StackHorizontal(sust_l,tr)
StackHorizontal(last,sust_r)
rc=PointResize(width, 100)
# source finishes here, 75% red-cyan transient HLG 10bit 4:2:2

# blue-yellow
plY=Blankclip(width = 10, height = 1, pixel_type="Y10").Expr("sx 0 == 103 sx 1 == 103 sx 2 == 143 sx 3 == 294 sx 4 == 416 sx 5 = 568 sx 6 = 659 sx 7 == 684 sx 8 == 682 682 ? ? ? ? ? ? ? ? ?")
plU=Blankclip(width = 5, height = 1, pixel_type="Y10").Expr("sx 0 == 848 sx 1 == 848 sx 2 == 710 sx 3 == 202 169 ? ? ? ?")
plV=Blankclip(width = 5, height = 1, pixel_type="Y10").Expr("sx 0 == 485 sx 1 == 485 sx 2 == 507 sx 3 == 539 539 ? ? ? ?")
tr=CombinePlanes(plY, plU, plV, "YUV", "YYY", pixel_type="YUV422P10")

sust_l=Blankclip(width = 4, height = 1, pixel_type="YUV422P10").Expr("sx 0 == 103 sx 1 == 103 sx 2 == 103 103 ? ? ?","sx 0 == 848 848 ?","sx 0 == 485 485 ?").PointResize(50,1)
sust_r=Blankclip(width = 4, height = 1, pixel_type="YUV422P10").Expr("sx 0 == 682 sx 1 == 682 sx 2 == 682 682 ? ? ?","sx 0 == 176 176 ?","sx 0 == 539 539 ?").PointResize(52,1)

StackHorizontal(sust_l,tr)
StackHorizontal(last,sust_r)
by=PointResize(width, 100)
# source finishes here, 75% blue-yellow transient HLG 10bit 4:2:2


Looks not very great on simulator but at the monitoring looks better (additionally manually tweaked to look best at monitoring). They are not exist in 'old standard color bars' but possibly useful for testing too.

[StvG]

avsresize has the following default values for range:
- rgb - full
- yuv < 32-bit - limited
- yuv 32-bit - full.

Both avsresize and fmtc treat a clip in float in the same way - always full range.

To avoid range mess and to perform also matrix conversion with fmtc in float:# avsresize
# rgb_lin=ConvertBits(yuv444, 32, fulls=false, fulld=true).z_convertformat(pixel_type="rgbps", colorspace_op="2020ncl:std-b67=>rgb:linear", nominal_luminance=800)
# rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

#fmtconv
rgb_lin=ConvertBits(yuv444, 32, fulls=false, fulld=true).fmtc_matrix(mat="RGB", mats="2020")
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear")
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)
...
# return z_ConvertFormat(rgb_lin, colorspace_op="rgb:linear=>rgb:709").ConvertBits(8, fulls=true, fulld=false).ConvertToRGB()
return fmtc_transfer(rgb_lin,transs="linear",transd="709").ConvertBits(8, fulls=true, fulld=false).ConvertToRGB()
...

[DTL]

"fmtc_matrix(mat="RGB", mats="2020")"

Yes - the most of full*=false function arguments to fmtc are either useless or may be even make things worse. Unfortunately Avisynth API/environment looks like have not ability to display user-side useful warning messages about arguments interpreting, formats and other. Only can throw stop-errors messages.

[DTL]

Some interesting update after I finally was not too patient to wait more years untill official release of EBU Tech 3325 Test Pattern 7 and decide to ask jpsdr in the mid of 2022 about why it outputs not perfectly matched result with fmtconv and avsresize: https://forum.doom9.org/showthread.php?p=1970995#post1970995

He point to some more magic in HDRtools:
With

rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2, OOTF=false, HLGLw=350)
return ConvertLinearRGBToYUV(rgb_lin,Color=2, OOTF=false).ConvertToRGB24(matrix="PC.709")

Conversion to linear and scaling and back to rec.709 transfer the sustained levels are close enough to avsresize and fmtconv plugins but the view of transient looks different (and may be close to the initial design of this transient in simulator - less contrast dark stripe):
https://i6.imageban.ru/out/2022/06/28/760f514040c5803cb74efedf501230ce.png

Though the Lw=350 is completely adjusted by hand value not based on any theory math just to attempt to put green and magenta sustained levels to match other plugins as close as possible.

Compare with old script of 2021 year. So it looks playing around OOTF and Lw values may give different results in the intermediate levels calcultaion. Are there ways in fmtconv and or avsresize to play also with OOTF on/off and Lw adjustment ?

[StvG]

The following two conversions give me close output:
# jpsdr_plugin
rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,OOTF=false, HDRMode=2)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)


rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return ConvertLinearRGBToYUV(rgb_lin,Color=2, OOTF=false).ConvertToRGB24(matrix="PC.709")
# avsresize
rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", nominal_luminance=1500)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return z_ConvertFormat(rgb_lin, pixel_type="yuv444p16", colorspace_op="rgb:linear:709:l=>709:709:709:l").ConvertToRGB(matrix="PC.709")

Also the following two conversions:
# jpsdr_plugin
rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return ConvertLinearRGBToYUV(rgb_lin,Color=2).ConvertToRGB24(matrix="PC.709")
# avsresize
rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", nominal_luminance=1000)
rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return z_ConvertFormat(rgb_lin, pixel_type="yuv444p16", colorspace_op="rgb:linear:709:l=>709:709:709:l").ConvertToRGB(matrix="PC.709")


I cannot get close result with fmtconv. It has sceneref and ambient arg but they didn't change anything.

[DTL]

With updated avsresize r15 plugin https://forum.doom9.org/showthread.php?p=1972716#post1972716 and scene_referred param it looks finally the winner in precision but still need some very magic numbers of additional correction to finally output 180/16 codevalues high/low R'G'B' 8bit in green-magenta for HLG narrow:

# jpsdr_plugin
# rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2, OOTF=false, HLGLw=1000)
# rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

# avsresize
rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", scene_referred=true)
rgb_lin=RGBAdjust(rgb_lin,r=0.157,g=0.157,b=0.157)

#fmtconv
# rgb_lin=fmtc_matrix(yuv444,mat="RGB", mats="2020",fulls=false, fulld=false)
# rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear",bits=32,fulls=false, fulld=false)
# rgb_lin=RGBAdjust(rgb_lin,r=3,g=3,b=3)

rgb_lin=SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:l=>rgb:709:709:l").ConvertToRGB()
#return fmtc_transfer(rgb_lin,transs="linear",transd="709",fulls=false, fulld=false).ConvertBits(8,fulls=true,fulld=false).ConvertToRGB()
#return ConvertLinearRGBToYUV(rgb_lin,Color=2, OOTF=false).ConvertToRGB24(matrix="PC.709")

also the transient looks about good.

jpsdr plugin can only make some unbalanced R'G'B' of Green 16,182,13 and Magenta 171,11,176 when trying to align high value to 180. May be some matrixing is not perfect.

[StvG]

rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2, OOTF=false, HLGLw=1000) - use OutputMode=2 for float output like avsresize. Also OOTF should be true to be scene-referred according to the docs? Also EOTF doesn't have effect for HDRMode=1/2 and Color=0 according to docs but changing between true/false there is noticeable diff. EOTF should be set to false? Also HLGLw should be set to 12 (to match avsresize (https://github.com/sekrit-twc/zimg/blob/797024c67be7ffd959da381adf7b0164029ef3e5/src/zimg/colorspace/gamma.cpp#L351)).

Here what I have without any rgbadjust - avsresize and fmtc are close but hdrtools differs noticeably:

# jpsdr_plugin
rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2, OOTF=true, EOTF=false, outputmode=2, HLGlw=12, hlglb=0)

# avsresize
#rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020=>rgb:linear:2020", scene_referred=true) # nominal_luminance does not have effect when scene_referred=true

#fmtconv
#rgb_lin=fmtc_bitdepth(yuv444, 32).fmtc_matrix(mat="RGB", mats="2020")
#rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear", sceneref=true)

rgb_lin= SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
#return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709=>rgb:709:709").ConvertToRGB()
#return fmtc_transfer(rgb_lin,transs="linear",transd="709").fmtc_bitdepth(8, dmode=1).ConvertToRGB()
return ConvertLinearRGBToYUV(rgb_lin,Color=2, OOTF=false, EOTF=true).ConvertToRGB24(matrix="PC.709")

Edit: Actually it's expected the different result from HDRTools because ConvertLinearRGBToYUV(..., Color=2) converts matrix, transfer and primaries to 709 while with avsresize and fmtc the matrix and primaries aren't converted.

[DTL]

rgb_lin=ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2, OOTF=true, EOTF=false, outputmode=2, HLGlw=12, hlglb=0)
return ConvertLinearRGBToYUV(rgb_lin,Color=2, OOTF=false, EOTF=true).ConvertToRGB24(matrix="PC.709")

It outputs both distorted transient and R'G'B' sustained big about 230+ and unaligned.


rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020=>rgb:linear:2020", scene_referred=true) # nominal_luminance does not have effect when scene_referred=true
return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709=>rgb:709:709").ConvertToRGB()

It makes very distorted transient and R'G'B' clipped to 0 and 255.


rgb_lin=fmtc_bitdepth(yuv444, 32).fmtc_matrix(mat="RGB", mats="2020")
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear", sceneref=true)

return fmtc_transfer(rgb_lin,transs="linear",transd="709").fmtc_bitdepth(8, dmode=1).ConvertToRGB()

Transient is a bit dark, R'G'B' clipped to 0..255 (full range, not narrow ?)

This assumed current 'reference' conversion:

rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", scene_referred=true)
rgb_lin=RGBAdjust(rgb_lin,r=0.157,g=0.157,b=0.157)
..scale..
return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:l=>rgb:709:709:l").ConvertToRGB()

With R'G'B' result of
https://i2.imageban.ru/out/2022/08/06/7c42b6e1c8142dd516e9c4e9b8c7d56c.png

[StvG]

About HDRTools - after HLG->linear conversion the green is shifted. Compare:
# jpsdr_plugin
return ConvertYUVtoLinearRGB(yuv444,Color=0,HDRMode=2, OOTF=true, EOTF=false, outputmode=2, HLGlw=12, hlglb=0)

# avsresize
#return z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", scene_referred=true)

# fmtconv
#rgb_lin=fmtc_bitdepth(yuv444, 32).fmtc_matrix(mat="RGB", mats="2020")
#return fmtc_transfer(rgb_lin,transs="hlg",transd="linear", sceneref=true)

Edit: Here fmtc equivalent of the avsresize 'reference' conversion:

#fmtconv
#rgb_lin=fmtc_bitdepth(yuv444, 32).fmtc_matrix(mat="RGB", mats="2020")
#rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear", sceneref=true)
#rgb_lin=RGBAdjust(rgb_lin, r=0.5, g=0.5, b=0.5)

rgb_lin= SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
#return fmtc_transfer(rgb_lin, transs="linear",transd="1886").fmtc_bitdepth(8, fulls=true, fulld=false).ConvertToRGB()

fmtc transfer "709" is scene-referred - same as z_convertformat(colorspace_op="x:x:x=>x:709:x", scene_referred=true).
fmtc transfer "1886" is display-referred - same as z_convertformat(colorspace_op="x:x:x=>x:709:x").

Also for avsresize add approximate_gamma=false in the latest step - return z_ConvertFormat(rgb_lin, pixel_type="rgbp8", colorspace_op="rgb:linear:709:l=>rgb:709:709:l", approximate_gamma=false).ConvertToRGB()

[DTL]

" fmtc equivalent of the avsresize 'reference' conversion:


#fmtconv
rgb_lin=fmtc_bitdepth(yuv444, 32).fmtc_matrix(mat="RGB", mats="2020")
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear", sceneref=true)
rgb_lin=RGBAdjust(rgb_lin, r=0.5, g=0.5, b=0.5)

rgb_lin= SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return fmtc_transfer(rgb_lin, transs="linear",transd="1886").fmtc_bitdepth(8, fulls=true, fulld=false).ConvertToRGB()

"

It is good too. But only not understand why "fmtc_bitdepth(8, fulls=true, fulld=false)" - it mean RGB linear float always full range (in fmtc) ? Or it is a internal feature of fmtc to convert narrow input range to full at some point ? Also the 'magic' amplitude correction of 0.5 looks less magic.

I test fmtc_bitdepth(8, fulls=false, fulld=false) - it returns the same result. Looks like 'fulls' with float input is ignored in fmtc ?
Same result with simple fmtc_bitdepth(8, fulld=false).

Addition: it looks difference between gamma 2.2 and 2.4 is good visible at the 'look' of transient. Also the real physical output of display depends on its physical transfers in RGB channels, so at LCD the view of transient also depends on viewing angle.

[StvG]

" fmtc equivalent of the avsresize 'reference' conversion:


#fmtconv
rgb_lin=fmtc_bitdepth(yuv444, 32).fmtc_matrix(mat="RGB", mats="2020")
rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear", sceneref=true)
rgb_lin=RGBAdjust(rgb_lin, r=0.5, g=0.5, b=0.5)

rgb_lin= SincLin2ResizeMT(rgb_lin,rgb_lin.width*4, rgb_lin.height*4, taps=16)
return fmtc_transfer(rgb_lin, transs="linear",transd="1886").fmtc_bitdepth(8, fulls=true, fulld=false).ConvertToRGB()

"

It is good too. But only not understand why "fmtc_bitdepth(8, fulls=true, fulld=false)" - it mean RGB linear float always full range (in fmtc) ? Or it is a internal feature of fmtc to convert narrow input range to full at some point ? Also the 'magic' amplitude correction of 0.5 looks less magic.

rgb_lin=fmtc_bitdepth(yuv444, 32) - here the range is converted from tv range to full because input is int (assumed tv range) and output is float (always [0,1]). Then everything is done in full range. fmtc_bitdepth(8, fulls=true, fulld=false) - float->8 but rgb, this makes the output [0,255], so, yes, fulls is not needed, fulld is enough for [0,255]->[16,235].

About the 'magic' amplitude correction:
- fmtconv - after rgb_lin=fmtc_transfer(rgb_lin,transs="hlg",transd="linear", sceneref=true) the ouput is within range [0,~1]. After applying RGBAdjust values of 0.5, the output becomes within range [0, ~0.5].
- avsresize after rgb_lin=z_convertformat(yuv444,pixel_type="rgbps", colorspace_op="2020ncl:std-b67:2020:l=>rgb:linear:2020:l", scene_referred=true) the output is within range [0, ~3.188]. After applying RGBAdjust values of 0.157, the output becomes within range [0, ~0.5].