range_in_s settings on bitdepth conversion ?

[poisondeathray]

What is math or generic explanation for this observation?

Start with YUV420P8 "bars"; Y=0,16,235,255 . Then convert to YUV420P10 , using either range_in_s full or limited

Y values read with vsedit picker, but double checked with planestats (crop to a single color)

Limited
Y=0,64,940,1020

Full
Y=0,64,943,1023

a = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[0, 128, 128])
b = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[16, 128, 128])
c = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[235, 128, 128])
d = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[255, 128, 128])

clip = core.std.StackHorizontal([a,b,c,d])

clip10 = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited")
#clip10 = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="full")

clip10.set_output()

[feisty2]

I always use fmtc for this kinda stuff, I never encountered any problem.
you can specify "fulls" and "fulld" to enforce the assumed source range and the result range.

[poisondeathray]

Originally Posted by feisty2

I always use fmtc for this kinda stuff, I never encountered any problem.
you can specify "fulls" and "fulld" to enforce the assumed source range and the result range.

It's not a "problem" per se ; I'm asking about the underlying explanation

This is not a full=>limited or limited=>full discussion ( it's not clamping or expanding levels. )

To clarify - When you convert YUV 8bit to YUV 10bit - What are the expected values? Why should the range parameter change the results when the setting is same for input and output

range_in_s="limited", range_s="limited"
Y=0,64,940,1020

range_in_s="full", range_s="full"
Y=0,64,943,1023

core.fmtc.bitdepth (clip, bits=10, fulls=False, fulld=False)
Y=0,64,940,1020

core.fmtc.bitdepth (clip, bits=10, fulls=True, fulld=True)
Y=0,64,943,1023

[feisty2]

when the input range is "limited", the bit depth conversion is a bit shift operation, concretely
result = input << (output_depth - input_depth)
and as you can see, 235 << (10-8) = 235 * 4 = 940

for full range input, the pixel range is first normalized to [0, 1], then linearly scaled to the target range
235 / 255 * 1023 = 942.76470588235294117647058823529 ≈ 943

[poisondeathray]

Originally Posted by feisty2

when the input range is "limited", the bit depth conversion is a bit shift operation, concretely
result = input << (output_depth - input_depth)
and as you can see, 235 << (10-8) = 235 * 4 = 940

for full range input, the pixel range is first normalized to [0, 1], then linearly scaled to the target range
235 / 255 * 1023 = 942.76470588235294117647058823529 ≈ 943

Ahhh!
Thanks f2!

[feisty2]

as defined in BT601-7 (page 4), the upper and lower bounds of limited range for bitdepths > 8 themselves are the bit shifted values of 8-bit bounds, and since the conversion is a linear function, the bit shift operation must also apply to everything in between.

and since bit shift fails to map the full range bounds of different bitdepths (apparently 255 << (10-8) = 255 * 4 = 1020 ≠ 1023), we follow the regular "normalize then scale" routine for full range bitdepth conversions

now I dont usually worry about bullshit like this because I only work with normalized fp32 inputs (0.0 = black, 1.0=white), sample types other than fp32 or fp64 should just f**kin' die