I'd like to add support for YCoCg color space in my
AVIF image compressor. I can easily find RGB to YCoCg formulas (including the original YCoCg paper) that are equivalent of a matrix rotation, but I'm missing a second formula that scales the result to actual bytes to be encoded.
All the sources I can find assume it's lossless and/or Co/Cg can be negative, but that means a conversion of 8-bit RGB to YCoCg gives 9- or 10-bit values. How am I supposed to store that in 8-bit unsigned bytes of YUV channels when encoding?