error diffusion and rounding

[feisty2]

which will you choose if doing ultra high bitdepth -- high bitdepth conversion, like float32 -- int16
I had been picking rounding to do stuff like that cuz int16 simply doesn't have any "banding" if u just use it to store images, and dithering noise, no good, it just makes images harder to compress
but I'm thinking of processing now, error diffusion generally gives less overall noise over rounding, but more noise on each individual sample
what do you think, error diffusion is better here or rounding

[raffriff42]

If going directly to the video display, then error diffusion. If possibly going to an encoder at some point, rounding.
Because error diffusion is less compressible, as you said.

Another issue is you don't want to add error diffusion more than once.

By definition, error diffusion adds noise or complexity, depending on the error pattern. Error diffusion also trades improved precision (less banding) for lower spatial resolution, so there should be plenty of surplus spatial resolution to start with. Something like 4x. Try converting some images in Photoshop(tm) to 4-bit using halftone, and you will see how much the image needs to be scaled up before conversion to preserve all the original detail. That's an extreme example of course.

Naturally converting from float to int should be left to the last possible moment, after all color correction etc.
But I'm not telling you anything you don't already know. Just thinking out loud here.

EDIT I guess error diffusion might* sometimes* help* hide banding due to downstream color correction etc, making the question more complicated.
(* = weasel words FTW - add enough of them and you can never be proven wrong)

[feisty2]

I'm thinking about what should I do with float32 clips if I gotta apply int16 filters like mvtools on them
guess I'm error diffusing them at 32x32 and 16x16 motion estimation, those are large blocks, and minimal local error (error diffusion) should work better
and rounding at 8x8 4x4 ME, small blocks, minimal pixel error (rounding) feels good

[colours]

You don't even need error diffusion for this. Random dither would work just as well when the error is going to be on the order of one part in 100000 anyway—maybe even better, because there wouldn't be spurious high-frequency noise caused by error diffusion. Error diffusion also causes a(n extremely tiny) image shift, which might be counterproductive to what you're doing.

It's probably the case that it doesn't really matter which you choose, though. The inaccuracy in motion estimation itself will most definitely exceed (by a few orders of magnitude) that of whatever scheme you use to convert from float32 to int16.

[feisty2]

image shift, damn, that didn't even flash across my mind, that's toxic, I'm going rounding

[pandy]

what do you think, error diffusion is better here or rounding
My two cents...
Avoid random dither and avoid error diffusion in classical implementations (like Floyd Steinberg).
This method works only on display.
For video processing it is better to go for ordered dither or similar to FS but pattern based Ulichney.

[cretindesalpes]

Error diffusion also causes a(n extremely tiny) image shift
??? Diffusing the quantization error does not mean diffusing the picture data! If low-bitdepth conversion could show some individual pixels or features to be shifted (in any direction), the effect will be much more pronounced without error diffusion than with it.

EDIT: on another channel, colours pointed me to an interesting paper about picture shift in dithering:
http://caca.zoy.org/attachment/wiki/publications/2008-displacement.pdf?format=raw
So yeah there is a shift, but as colours wrote, it would be insignificant for 16-bit dithering, especially when used in combination with motion vector estimation.

This method works only on display.
For video processing it is better to go for ordered dither or similar to FS but pattern based Ulichney.
Have you any pointer for such an assertion? I can’t see any justification for using ordered dithering instead of Floyd-Steinberg for processing. The only advantage of ordered dithering is to be an interesting compromise between dithering efficiency and compressibility (in addition of being fast to compute), so it makes only sense to use it before encoding. For any other purpose, just use the “best” algorithm available, maximizing SNR and feature preservation. And for high bitdepth conversions, dithering quality is much less important than for low bitdepth quantization, so the difference between classic FS and more sophisticated algorithms is likely to be insignificant regarding the actual content.

[pandy]

Have you any pointer for such an assertion? I can’t see any justification for using ordered dithering instead of Floyd-Steinberg for processing. The only advantage of ordered dithering is to be an interesting compromise between dithering efficiency and compressibility (in addition of being fast to compute), so it makes only sense to use it before encoding. For any other purpose, just use the “best” algorithm available, maximizing SNR and feature preservation. And for high bitdepth conversions, dithering quality is much less important than for low bitdepth quantization, so the difference between classic FS and more sophisticated algorithms is likely to be insignificant regarding the actual content.

It is obvious - FS (and similar class dither) provide error propagation trough all pixels (single bit or pixel value change will create something i call pixel storm), ordered dither doesn't propagate and this can be bigger benefit from compression perspective (you can even imagine codec design where particular dither nature is involved in encode process and as such there is no loss of quality - due higher qp required for noise like i.e. higher entropy dither).
IMHO for high bitdepth (low dither level/amplitude in terms bits used for dither vs bits used for quantizer) ordered is same or better in terms of reproduced visual effect (probably bellow perception level anyway on most of LCD displays).
FS class is definitely better for high DPI @ limited bitdepth - with low/medium DPI and relatively high bitdepth (more than 6 bits) ordered is same or better than FS class dither.
(my observation are based on series experiments i made also with help of your tool, i used various displays technologies etc - i spend lot of time doing tests - see my Amiga post - i try lot of things having on mind low color, low resolution video reproduction - this results are directly transposed to higher resolutions and higher bitdepths).
I really regret current situation where dither is not correctly addressed in video processing).

Dither is not trivial thing, correct choice of dither imply knowledge on rendering device which is usually unknown in moment when dither is applied .

btw personally i prefer Sierra lite over FS and i see that sometimes video is/can be beneficial from Atkinson also... (as You see we going to very subjective topics).

From my perspective ANY dither with low entropy (spatial and time) is more beneficial than FS class if it provide acceptable (from subjective perspective) results (and dither is always highly subjective).

On http://caca.zoy.org/study/index.html main dither problems can be seen nicely illustrated - random dither simply don't work on static parts without significant resolution reduction - usually any dither will work better than random dither.
And i never saw plain "Error Feedback Rounding" in video software - not sure why - during my experiments i saw it works kind of reasonable (as error propagation is limited to line only)...

[foxyshadis]

I have a hard time believing that choice of dither can ever affect lossy compression, since the low bits are always going to be munged anyway. In my experience, you can dither any way you want and the codec will cheerfully round it off for you anyway, until you reach absurd near-lossless bitrates.

[Sulik]

+1 to what foxyshadis said -> the transform itself isn't lossless, so most dither will be eliminated even at the lowest quantization levels (it gets worse with larger transform sizes in HEVC). Same reason HEVC has trouble preserving low-amplitude film grain.

[pandy]

I have a hard time believing that choice of dither can ever affect lossy compression, since the low bits are always going to be munged anyway. In my experience, you can dither any way you want and the codec will cheerfully round it off for you anyway, until you reach absurd near-lossless bitrates.

Nope if you design codec well (dither is integral part of system, dither characteristic is known and codec is tuned to keep dither intact - this imply no random or similar dither - consider this as x264 grain 'tune').

Imagine you are capable to inject dither at frequency domain level before quantization...
Pattern based dither can be complex both spatially and temporary without affecting bitrate - concept similar to subtractive dither (used sometimes - for example Tek VM700 have programmable dither generator).

[pandy]

+1 to what foxyshadis said -> the transform itself isn't lossless, so most dither will be eliminated even at the lowest quantization levels (it gets worse with larger transform sizes in HEVC). Same reason HEVC has trouble preserving low-amplitude film grain.


H.265 is immature and i don't expect at this stage good psychovisual tuning - this need more time.

[Biggiesized]

Any pattern based dither will have visual artifacts. And as previously mentioned, you can't preserve a lot of dither detail once the image is quantized and compressed.

[pandy]

Any pattern based dither will have visual artifacts. And as previously mentioned, you can't preserve a lot of dither detail once the image is quantized and compressed.

What kind of details you want to preserve when we talking about dither with level approx 2LSB on 8 - 10 bit quantization range and pixel density usually more than 50DPI? This mean you have falcon (or any similar hunting bird) eyes... humans are not so good...