Avisynth dithering

[fvisagie]

From version 2.6 some Avisynth filters like Levels() and Tweak() support dithered output. What kind of data sizes are used for calculations prior to dithering? Put somewhat differently, how do calculation and dithering in e.g. Levels() compare to those in, say, SmoothLevels()?

Thanks,
François

[feisty2]

Float I guess, dunno, probably the same when dithered down to int8

[cretindesalpes]

These filters use LUTs. When dithering, a 16-bit LUT is used. The input is scaled and a dithering pattern is added before feeding the LUT. Dithering before the LUT is technically wrong (or at least sub-optimal) but it should give a decent result with moderate amounts of amplification.

[feisty2]

Sounds weird... Why not convert to float and do the expression and dither back to int8? That would be better (quality), right?

[cretindesalpes]

Well… is there any issue with the algorithm precision to begin with, leaving apart the bayer-like dithering ? Painting everything in float does not automagically increase the SNR, “quality” or whatever (when you have a float in your hand, every problem looks like a precision issue). But anyway, floating point is used to make the LUTs.

[fvisagie]

Thanks for the feedback.

Cheers,
François

[feisty2]

no, I mean why BEFORE the expression, not AFTER, let's drop the float stuff and I was asking, like, "why not dither after the 16bits LUT?"

[cretindesalpes]

Maybe the author thought it was simpler to implement. On the complexity side before and after are more or less equivalent: shift left, add pattern, 16->8 LUT (before), or 8->16 LUT, add pattern and shift right (after). But as I said, considering the dithering algorithm, the result is almost equivalent when the gain stays close to 1, which is generally the case.

[StainlessS]

I was thinking of maybe producing Stack16 RGB result in RGBAdapt() [so as to use Dither() dithering].
EDIT: RGBAdapt() here:- http://forum.doom9.org/showthread.php?t=170642

I managed to implement Stack16 RGB in Clipblend16() without referring to Dither docs (well just a glance) and it seems that interleaved RGB32/24 Stack16
is not supported (also implemented it for YUY2 Stack16 I think).
I guess conversion with eg ShowRed() etc could be used to make RGB32/24 results compatible with Dither.


Is there a post on-site or in Wiki fully documenting Dither's Stack16 and other bit extended formats, colorspaces and TV/PC/Studio ranges?
Its a lot of work ploughing through Dither docs and avs script and CPP source just to try and figure out what is and is not supported.
(I have never used Dither).

[cretindesalpes]

StainlessS: Dither doesn’t support stack16 clips in RGB, for the main reason that it is not a planar format. However, as you said, you can use separate Y(UV) clips to convey each RGB plane, a format recognized by Dither_convert_rgb_to_yuv().

[StainlessS]

Thank you Cretindesalpes, I think I'll implement RGBAdapt16() as RGB interleaved Stack16 as I did for Clipblend16().
[Did ClipBlend S16 mainly as described by poster requesting Stack16 output rather than dissecting dither source, guess the YUY2 S16 output is
only usable in Avisynth v2.6 where eg UtoY can be used to convert].

I am assuming that RGB S16 range is 0-65535 (or should it be 0xFF00).

EDIT: look like 0 -> $FF00 is correct range.

EDIT: Yep 0->$FF00


Blankclip(color=$FFFFFF)
Interleave(showred("yv12"), showgreen("yv12"), showblue("yv12")).Dither_convert_8_to_16
MSB=RT_YPlaneMin(n=0, x=0,y=0,w=1,h=1)
LSB=RT_YPlaneMin(n=0, x=0,y=height-1,w=1,h=1)
S=RT_String("MSB=$%02X LSB=$%02X",MSB,LSB)
Subtitle(S)
# Shows YV12 top white, bottom green : prints MSB=$FF LSB=$00


Presumably has to be limited at $FF00.

[raffriff42]

Presumably has to be limited at $FF00.
Here's a little worksheet that makes sense of it for me:(8 bit) (16 bit)
0 = 0
16 (0x10) = 4096 (0x1000) ( 16<<8)
128 (0x80)(2^8 / 2) = 32768 (0x8000)(2^16 / 2) (128<<8)
235 (0xEB) = 60160 (0xEB00) (235<<8)
255 (0xFF)(2^8 - 1) = 65280 (0xFF00)(2^16 - 256) (255<<8)

[StainlessS]

Thank you Raff. The other option was * 257 ie 255 * 257 == 65535. [also eg 15 * 17 = 255 : (2 ^ n - 1) * (2 ^ n + 1) == (2 ^ 2n - 1)]

Have posted Beta (should be OK) in RGBAdapt thread if you would like to give it a test drive.

EDIT Removed a rubbish edit.