quantization and tolerances in various formats

[vcmohan]

I use a tolerance value x for my 8 bit luma formats. It checks whether my result n is within a+x and a - x. This is to allow for spatial as well as sensor quantization effect. This x is used over entire range 0 to 255, even though n and a may have independantly any value between 0 and 255.
In case of 16 bit fixed point format, actual data (I am assuming) is 10 to 12 bit, and same value of x as in case of 8 bit format can be used. x is independant of the values of n and a. Is this my assumption valid?

In case of single precision float format , I read that the actual sensor resolution is ranged and dependent on the luma value. In that case If my result is n, x will be proportional to n.
May be even for 12 bit data, x is proportional to n.
Are my inferences correct?
Wat is the step(quantization) value in case of single precision float format?

Request advice on this.

[foxyshadis]

For float, it's a bit difficult to define, since precision increases the closer you get to zero. However, just below 1.0, the step precision is 2^-11 for half (1/2048) and 2^-24 for single (1/16million), so that sets a good upper bound.

[vcmohan]

Thanks. Please see at the link (http://www.anyhere.com/gward/hdrenc/hdr_encodings.html) HDR encoding comparison table. It shows different values.

[foxyshadis]

VS uses 32-bit OpenEXR, like most software, although it's not actually specified anywhere; all plugins except rean's use the standard 0.0-1.0 range for basic black to white (what we think of as 0-255 or 16-235). The HDR pixels can be stored outside of that and scaled in any way. The OpenEXR listed on that page only lists half-float, which VS technically supports but almost no actual filters do, including fmtconv. I'm not sure how they calculate a step precision of 0.1%, which would be 1/1024, but the actual half-precision datatype does have 1/2048 steps between 0.5 and 1.0; it's only above 1.0 that they become 1/1024. (And 1/4096 between 0.25 and 0.5, etc on down, which is why float is exceptionally useful for linear-light video: higher precision in darks without needing a gamma curve.) That gives you a minimum equivalent of 11-bit integer, with a maximum equivalent of 24-bit right next to zero. 32-bit float is so much higher precision that step size is essentially meaningless for video.

All of the other formats on that page are on-disk formats (except RGBE used to be processed in directly in the 80's, before float hardware was common). All in-memory formats are now a standard 16-, 32-, or 64-bit float.

[foxyshadis]

I'm re-reading the first part of your question, and I'm not sure about "16 bit fixed point." If you mean real fixed point, like in JPEG-XR, VS has no support for that at all, although I suppose you could implement it internally. (JPEG-XR uses 0-8196 for 0-1.0 16-bit signed fixed point, making the step size 1/8196 or 14-bit int.) If you mean 16-bit integer, the step size is 256 times as much as standard 0-256 int, so you just scale your tolerance by that much.

However, you're correct, linear light is more difficult than gamma light. 16-bit int can be in either linear or gamma-corrected, and there's no property defining this; as long as it's gamma corrected one value would work over the whole range. For linear, you have to reverse the gamma correction on the value first to obtain the tolerance: It would be tiny near 1.0 and huge near 0.0. In floating point, you can approximate gamma with the BT.1889 function: 1/(1.099*pow(YOUR_PIXEL,0.45)-0.099) That should compensate your tolerance. For linear integer, scale the pixel value down first.

[vcmohan]

Thanks. 16 bit I mean vapoursynth uint16_t values which probably were derived from 10 or 12 bit recordings.
I find quite confusing to me and for the present experiment with some images and decide.
Thanks a lot for your time