Scaling of YCoCg to 8-bit bytes

[kornelski]

I'd like to add support for YCoCg color space in my AVIF image compressor (https://github.com/kornelski/cavif-rs). I can easily find RGB to YCoCg formulas (including the original YCoCg paper) that are equivalent of a matrix rotation, but I'm missing a second formula that scales the result to actual bytes to be encoded.

All the sources I can find assume it's lossless and/or Co/Cg can be negative, but that means a conversion of 8-bit RGB to YCoCg gives 9- or 10-bit values. How am I supposed to store that in 8-bit unsigned bytes of YUV channels when encoding?

[benwaggoner]

I'd like to add support for YCoCg color space in my AVIF image compressor (https://github.com/kornelski/cavif-rs). I can easily find RGB to YCoCg formulas (including the original YCoCg paper) that are equivalent of a matrix rotation, but I'm missing a second formula that scales the result to actual bytes to be encoded.

All the sources I can find assume it's lossless and/or Co/Cg can be negative, but that means a conversion of 8-bit RGB to YCoCg gives 9- or 10-bit values. How am I supposed to store that in 8-bit unsigned bytes of YUV channels when encoding?
U and V are also signed, with "0" meaning no chroma for that channel and saturation coming from absolute value. It's really only luma or native RGB that are purely unsigned. You should be able to treat other chroma formats identically, and the codec itself doesn't really care.

[nhw_pulsar]

Yes, just for the precision, I used in my codec (NHW) for test the YCoCg matrix from the english wikipedia article.

With these matrixes if you input 8-Bit RGB, it will output 8-Bit YCoCg, just the ranges will be Y: [0-255], Co: [-128,127], Cg: [-128,127], so typically you need to add an offset of +128 for Co and Cg, (and then remove it before converting back to RGB)... This would give a classic YCoCg 8-Bit colorspace...

Just as we are talking about YCoCg, how do you estimate it? Its big advantage is that it is very fast (no multiplies), but visually I find that it is not as good as YCbCR/YUV... When I tested it 3-4 years ago in my codec, I made quite extensive tests at -l9 and -l11 compression settings if I remember correctly, and the results were not visually as pleasant as YUV at the same compression ratio.Did you have same experience?

By the way, I read that your AVIF image compressor: CAVIF is great!

Cheers,
Raphael

[benwaggoner]

Just as we are talking about YCoCg, how do you estimate it? Its big advantage is that it is very fast (no multiplies), but visually I find that it is not as good as YCbCR/YUV... When I tested it 3-4 years ago in my codec, I made quite extensive tests at -l9 and -l11 compression settings if I remember correctly, and the results were not visually as pleasant as YUV at the same compression ratio.Did you have same experience?
Yeah, Y'CbCr was well researched and grounded in science. Math is a little more complex, but it was still worth using it even back in JPEG where available MIPS per pixel were many orders of magnitude less. Y'CtCp also has its psychovisual strengths.

YCoCg doesn't make a lot of practical sense for me to adopt, since the easier math is pretty immaterial in modern systems.

[Balling]

"How am I supposed to store that in 8-bit unsigned bytes"

x264 suports 9 bytes, that is how.