Jaggy Edges on Bright Colors in 10bit

[EdenElanis]

The Source is 8Bit 4:2:0. Whenever I try to encode to 10bit, the result has these Jaggy Edges wherever there are bright colors, especially bright red. I know it has something to do with chroma subsampling.

https://ibb.co/Gcc1Ntd

When I encode to 8Bit however, those edges are smooth, like the source.

https://ibb.co/5vjd4W3

Is there a way to encode to 10bit and avoid the jaggy edges WITHOUT converting to 4:4:4?

Thanks in advance.

here is my script

______________________________



LoadPlugin ("C:\Program Files (x86)\MeGUI\tools\ffms\ffms2.dll")


FFVideoSource ("E:\00_PROCESSING\ENGLISH_0005.mkv",colorspace="YV12", resizer="Bicubic", width=1280, height=720, threads=1)


trim (9721,9960)

[benwaggoner]

My first guess is that something went wrong in your conversion to 10-bit or in the 10-bit to 8-bit conversion on your display.

There really isn’t any reason to use 10-bit with 8-bit source in HEVC. H.264 did have some bitrate savings for this scenario, but not really in HEVC.

[sneaker_ger]

Why are the colors in your screenshots different? Solve that problem first.

[EdenElanis]

Is that really important to the edges problem? No Idea how to do that

[benwaggoner]

Is that really important to the edges problem? No Idea how to do that
Getting that right is necessary for diagnosis, given the problem is likely in color space conversion pre or post encoding.

[EdenElanis]

I also tried all sorts of colorspace conversions bt709, 2020, 604, YV12,YUY2,YV24 RGB. No success.

[excellentswordfight]

I also tried all sorts of colorspace conversions bt709, 2020, 604, YV12,YUY2,YV24 RGB. No success.
Why do you need any conversion at all? What processing do you want o apply? If it just resize I dont see why this would be an issue.

I would start with just piping the original clip to x265 first to see if the problem still exist. Input depth should auto detect, but if you think something goes wrong you can add --input-depth 8 --output-depth 10 just to make sure. And if source is 709, specify the flags in x265 as well --colorprim bt709 --transfer bt709 --colormatrix bt709 --range limited, shoudlnt be an issue, but better safe then sorry.

[Forteen88]

Maybe try making screenshots using MPC-HC player,
https://github.com/clsid2/mpc-hc/releases

and try using madVR in MPC-HC,

or use VLC-player.

[EdenElanis]

seems to work with MadVR. I guess the problem is with the renderer, not the encoded footage itself. That's a Load off my back.

[benwaggoner]

seems to work with MadVR. I guess the problem is with the renderer, not the encoded footage itself. That's a Load off my back.
Yep. We are only early in the era of the 10-bit Windows desktop. Everything is amazingly better using my Intel NUC attached to my Sony 940c HDR TV, since Windows does is own advanced GPU-accelerated tone mapping and dithering.

Lots of legacy playback scenarios start displaying 10-bit video by truncating off the last two bits, introducing some banding. Doing it right requires a 64-940 Y'CbCr to 0-255 RGB without an intermediate 8-bit stage, with good dithering.