HDR 10 color matrix setting question

[Selur]

I got a clip which mediainfo reports as:
Color range : Limited
colour_primaries_Original : BT.2020
transfer_characteristics_Original : PQ
matrix_coefficients_Original : BT.2020 non-constant
Mastering display color primaries : Display P3
Mastering display luminance : min: 0.0001 cd/m2, max: 1000 cd/m2
Codec configuration box : hvcC

Using:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'G:\TestClips&Co\files\HDR\HDR10\4K sun HDR test.mp4'
# current color space: YUV420P10, bit depth: 10, resolution: 3840x2160, fps: 25, color matrix: 2020ncl, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\HDR\HDR10\4K sun HDR test.mp4 using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="G:/TestClips&Co/files/HDR/HDR10/4K sun HDR test.mp4")

clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
# Output
clip.set_output()
I can open and preview the file without a problem.
Wanting to make ssure that the color matrix is correctly set to 'BT.2020 non-constant'
I added
clip = core.std.SetFrameProps(clip, _Matrix=10)
and when calling:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'G:\TestClips&Co\files\HDR\HDR10\4K sun HDR test.mp4'
# current color space: YUV420P10, bit depth: 10, resolution: 3840x2160, fps: 25, color matrix: 2020ncl, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\HDR\HDR10\4K sun HDR test.mp4 using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="G:/TestClips&Co/files/HDR/HDR10/4K sun HDR test.mp4")
clip = core.std.SetFrameProps(clip, _Matrix=10)
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
# Output
clip.set_output()
encoding works fine, but preview fails with:
Resize error: Resize error 3074: invalid colorspace definition (10/16/9 => 0/16/9). May need to specify additional colorspace parameters.
using _Matrix=9 works, any idea what I'm missing? ( might also be a bug in the vsviewer and https://github.com/YomikoR/VapourSynth-Editor/releases)

Cu Selur

[_Al_]

original clip is like this, is that right?:
_ColorRange 1=limited
_Matrix 9=2020ncl
_Transfer 16=st2084
_Primaries 9=2020
which is: 9/16/9 if I got vapoursynth order correctly as _Matrix/_Transfer/_Primaries

why to correct Matrix to 10?

I tried setting _Matrix to 10 in props with different previewer and that failed as well, it would fail at any point when a frame is requested

[kolak]

Your original settings are correct.

Industry uses 2020 non-constant luminance, which is value 9.
I don't know any case for use of 2020 constant luminance, which is what you get by using value of 10.
Leave it at default detection, which is correct.

[Selur]

I get that 9 is correct, but shouldn't 10 also work? I mean, does it make sense to even include it in the matrix list?

[kolak]

It should but it's so rare that probably buggy/untested.
It's part of UHD spec, so I assume it's included (even if practically never used).

[Selur]

Okay, thanks. :)

Cu Selur

[JustinTArthur]

tldr: the path from BT.2020 CL (matrix 10) Y′CbCr to R′G′B′ is only possible for a single specific transfer characteristic (′), and it's not PQ.

This is not so much a bug but a bug fix to the traditional way of separating luma and chroma components from mixed RGB light. From the analog era into HDTV, the industry conversion from R, G, and B to intensity/color components had you perform any display gamma-correction or perceptual quantization you wanted to the individual R, G, and B channels before mixing and separating to luma/chroma components like Y/I/Q and Y/Cb/Cr. In digital video terms, this can be summarized as going from R′G′B′ to Y′CbCr. In analog terms, typically R′G′B′ to Y′IQ or Y′UV. The prime symbol here indicates levels adjusted to gamma-correct for a display characteristic or to quantize perceptually-relevant light information.

Charles Poynton and others had been yelling at standards body audiences for years that it makes no sense to pre-correct video for displays or to optimize perceptually-relevant information before you mix R, G, and B light into luma and chroma components. Well, standards bodies finally listened! BT.2020 includes a mix/separation matrix defined for going from (RGB)′ to Y′CbCr, optionally allowing you to start with RGB (no primes!). In this case the "′" is the SMPTE ST 170 OETF calculated to modern accuracy and from vapoursynth's perspective it is applied within the transformation instead of before.

In your case, you were trying to go from Y′CbCr to R′G′B′. vapoursynth could have figured out how to go from Y′CbCr to (RGB)′ to R′G′B′ like you wanted if your "′" was SMPTE ST 170 transfer characteristics because that's the only Y′CbCr mix the standard defines. Instead, you supplied a video with an impossible combination: components adjusted according to ST 2084 transfer characteristics and mixed/separated from RGB with an algorithm that incorporates ST 170 transfer characteristics.

There are luma-ish/chroma separation transformations like that available for PQ like ICtCp and Dolby Vision's mix. Like BT.2020's CL matrix, they incorporate the transfer characteristic in the transformation from RGB to separated components instead of before.

To learn more about why this was done, here's Poynton's abstract to his presentation to the 32nd SMPTE AMIC:
Luminance, luma, and the migration to DTV (1998) (https://poynton.ca/papers/SMPTE_98_YYZ_Luma/index.html)

And here's ITU's research leading to the BT.2020 CL transformation function, including nice block diagrams of the process:
Report ITU-R BT.2246-8: The present state of ultra-high definition television (https://www.itu.int/dms_pub/itu-r/opb/rep/R-REP-BT.2246-8-2023-PDF-E.pdf)
(see Annex 2, section 2.3, "The constant luminance issue" and the section after.)