10-bit input to 8-bit output. What happens?

[benwaggoner]

If I'm using a 10-bit capable version of x264, but encoding to 8-bit, how does x264 handle the 10-bit to 8-bit conversion?

I'd expect it to just truncate the two least significant bits. But it would be lovely if it did some dithering at least.

My pipe dream for years would be for encoders to take the >8-bit information into the frequency transform stage, so the codec can know what's banding and what's an edge, and make the final psychovisual decisions appropriately.

is this anything anyone is doing, or looking at, or would like to have me pitch in more detail :)?

[sneaker_ger]

If I'm using a 10-bit capable version of x264, but encoding to 8-bit

Since bit depth is set at compile time, I don't think that is even possible.

Any current version will always convert to the requested bit depth before passing it on to the actual encoder without making use of any pre-exisiting higher precision AFAIK.

[benwaggoner]

Right. Well, a feature request, then. An encoder should be able to do a better job of 10-bit to 8-bit conversion than any preprocessing technique.

A 10-bit build can't encode 8-bit output at all? Weird; it's not like we need to have different builds for 0-255 range etcetera. I imagine the intent is to merge them at some point.

[MasterNobody]

A 10-bit build can't encode 8-bit output at all?
Yes. If you want to encode to 8-bit output than you should use 8-bit build which also support high bitdepth input but will dither it to 8-bit before sending to libx264. As for encoder implemented dithering (as part of encoding for better dithering keeping) I doubt it can be made effectively (at least I don't have idea how to implement this).

[benwaggoner]

Yes. If you want to encode to 8-bit output than you should use 8-bit build which also support high bitdepth input but will dither it to 8-bit before sending to libx264.
Do we have any documentation on the dithering algorithm it uses?

As for encoder implemented dithering (as part of encoding for better dithering keeping) I doubt it can be made effectively (at least I don't have idea how to implement this).
One way to think about it do the initial frequency conversion from the full precision of the source, and then defer how exactly to quantize to the 8-bit precision until later in the encoding process.

Alternatively, imagine doing normal 10-bit encode, and then modifying the bitstream at the end to 8-bit precision. At that point, the higher frequency data can be used to determine what sort of dithering is appropriate for different parts of the image.

Also, the dithering itself could use some sort of trellis-like approach to find the right dither for optimum compressibility. There's a huge number of ways to dither "well," and being able to pick the one that compresses best could be a big help.

As it is, any 10-bit to 8-bit conversion that happens before the codec is either adding noise or banding, blind as to its impact on compression.

[MasterNobody]

Do we have any documentation on the dithering algorithm it uses?
The dithering algorithm is based on Sierra-2-4A error diffusion (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=filters/video/depth.c;h=9465f177daef38ec07a17358979f1206ed866a8b;hb=HEAD#l68)

[benwaggoner]

The dithering algorithm is based on Sierra-2-4A error diffusion (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=filters/video/depth.c;h=9465f177daef38ec07a17358979f1206ed866a8b;hb=HEAD#l68)
Cool. That's a lot better than just truncating the LSBs, which happens in WAAY too many other encoders...

[akupenguin]

Also, the dithering itself could use some sort of trellis-like approach to find the right dither for optimum compressibility.
QNS, not trellis. Trellis quantization assumes error can be measured independently for each coef, which is incompatible with correct treatment of rounding. Fixing that assumption requires abandoning the dynamic programming approach entirely; it's more similar to matching pursuit.

A 10-bit build can't encode 8-bit output at all? Weird; it's not like we need to have different builds for 0-255 range etcetera. I imagine the intent is to merge them at some point
TV-range vs full-range is a single bit written in the SPS header; the rest of the encoder doesn't know or care.
8-bit vs 10-bit is a different datatype, uint8_t vs uint16_t. To compile both (rather than typedef) you'd have to template every function that touches pixels or coefs. The corresponding change in ffh264 modified 2700 LOC (excluding asm).
I have no plans to merge them; and if I did, the obvious strategy would be to compile two whole copies of libx264 and add a wrapper to select one.