View Single Post
Old 17th April 2015, 17:58   #9  |  Link
mawen1250
Registered User
 
Join Date: Aug 2011
Posts: 103
Thanks for your reply!
This build fixes the magnitude of ordered dither (to 0.5), which solved lots of problems. The underflow issues is solved, but the overflow issues for 9-15bit still exists.

2. Not only full range YUV can produce this kind of overflow issue, but also limited range RGB/YUV with out of range value, such as reducing 65535 to 9-15bit with dithering, or converting 65535 from limited range 16bit to full range 9-15bit. Thus, IMO this may lead to potential problems in practice, since we may not guarantee the input image is perfect. Acctually converting to 9-15bit is not so often used except in the case of final output, so I prefer safe output rather than performance or intermediate precision. Perhaps an additional option for clamping the result to valid range can be added? Also it can be used for limiting the value to limited range when fullrange_out=False.

3. My mistake, the ordered dither does affect only the least significant bit of the output, maybe I was misled by the image with underflow issues. In the previous build, when the output depth & range are the same as input, the ordered dithering pattern still applies to the image. After fixing the magnitude issue of ordered dither it's also solved. I suppose it will be faster to directly return the src frame pointer on this condition, since the frame data is always unchanged.

4. Yes, I got it.
mawen1250 is offline   Reply With Quote