Why does x265 lossy encoding have issues with 12bit banding?

[poisondeathray]

Why does x265 lossy encoding have issues with 12bit banding?

Even at low CRF or QP values, like 0 or 1, a simple 12bit ramp exhibits gaps and duplicate values

12bit ramp, 4K DCI 4096x2160, Y= 0 to 4095. Every x= position has the same value in the Y channel. 1 frame.

Gaps in the waveform at regular intervals. eg. x=252, x=253 both have a Y value of 252. The value Y=256 is missing, etc... x=4060, 4061 both have a Y value of 4060, and Y value of 4064 missing, etc...

eg. --crf 0 waveform
https://i.postimg.cc/nhFd7Kvt/x265-yuv444p12-crf0.png

Double checked values with vapoursynth color picker (missing values/gaps, and repeated values), and psnr testing

eg.

vspipe --y4m 4k_greyramp_yuv444p12.vpy - | "x265.exe" --y4m - --input-csp i444 --input-depth 12 --output-depth 12 --qp 1 -o x265_yuv444p12_qp1.hevc


--lossless is ok - so that mostly rules out a script or toolchain issue; a decoder side issue also less likely

eg. --lossless waveform
https://i.postimg.cc/ZnPcv0nR/x265-yuv444p12-lossless.png


But when using --qp 0, --qp 1, or --crf 0, --crf 1 , some variations like --tune psnr... , checked several x265 builds => always gaps, banding without the --lossless switch

A simple gradient ramp should be lossless at low qp values - it is with 10bit ramps; e.g. even at crf16, x264 is lossless on a simple 10bit ramp

Source file ffv1 yuv444p12le for testing (lossless, PSNR infinity compared to original script)
http://www.mediafire.com/file/2h4xrmvug0lm127/4k_greyramp_yuv444p12_ffv1.mkv/file

[benwaggoner]

Have you read https://x265.readthedocs.io/en/default/lossless.html? Lots of good stuff in there.

You might try turning on --tskip and perhaps --cu-lossless. Those will evaluate whether lossy encoding with a transform skip or straight up lossless encoding is more efficient per CU. They can slow things down quite a bit, however.

If --tskip works, you could then try --tskip-fast to see it that preserves the quality improvement with a smaller perf hit.

[Cary Knoop]

Gaps in the waveform at regular intervals. eg. x=252, x=253 both have a Y value of 252. The value Y=256 is missing, etc... x=4060, 4061 both have a Y value of 4060, and Y value of 4064 missing, etc...

Assuming the setup was correct then this bug is as big as a Buick!

[poisondeathray]

Yes, well hopefully it's a simple error on my end, so someone repeat the test

I'll do some more testing, but the point is you shouldn't have to do anything assuming these observations were valid. A lowish CRF or QP should be lossless for a simple ramp. Or it should auto adjust until it is. x264 is lossless at CRF16 on a 10bit ramp (probably higher CRF is too). The regularity of the pattern suggests a deeper issue to me

12bit AV1 has a similar bug, but only in dark shadows


EDIT: adding --tskip --cu-lossless at CRF 1 does not help, same gaps, repeat values

[benwaggoner]

Yes, well hopefully it's a simple error on my end, so someone repeat the test

I'll do some more testing, but the point is you shouldn't have to do anything assuming these observations were valid. A lowish CRF or QP should be lossless for a simple ramp. Or it should auto adjust until it is. x264 is lossless at CRF16 on a 10bit ramp (probably higher CRF is too). The regularity of the pattern suggests a deeper issue to me

12bit AV1 has a similar bug, but only in dark shadows


EDIT: adding --tskip --cu-lossless at CRF 1 does not help, same gaps, repeat values
And since straight-up --lossless fixes the problem, we can rule out issues in the source or analysis. File a bug with MCW!