Color banding and noise removal

[cretindesalpes]

Dither_convert_rgb_to_yuv: Fails on big inputs (greater than 1500px or so for any dimension)
Dither_convert_yuv_to_rgb: Fails on big inputs. Fails on not mod4
Yes I know. Actually it comes from the resizer/convolver wanting to allocate huge temporary data buffers. I'll rework the buffering system to something more efficient for a future update. In the meantime, a working solution would be splitting the picture in tiles or bands, interleaving, converting, deinterleaving and stacking them back to reduce the size pressure on the convolver.

Linear also occasionaly makes some strong black outline/ringing
This is normal. Because linear dark levels have their range reduced relative to gamma corrected levels, ringing caused by bright pixels on dark ones are greatly emphasized and produce super-black artifacts (and very bright secondary ringing). To avoid them, you should use a kernel with very low or no negative lobes at all. This is a drawback of linear light operations.

[Yellow_]

I'm getting very odd discoloration in my first attempt at going 8bit h264AVC to 10bit h264.

http://blendervse.files.wordpress.com/2011/06/8-10-10bit_h264.jpg

I'm using this:

FFmpegSource2("clip.MOV")
Dither_convert_8_to_16 ()
Dither_convey_yuv4xxp16_on_yvxx()

Then avs2yuv:

avs2yuv -raw dither16test.avs -o - | x264-64-10bit --demuxer raw --input-depth 16 --input-res 1920x1088 --fps 25 --fullrange on --output dither16testout.mp4

The source looks like this:

http://blendervse.files.wordpress.com/2011/06/orig8bitmov.jpg

I'm using ffmpeg to decompress the 10bit h264 and with other 10bit sources there is no discolouration so don't think this is an issue with reading the file correctly.

Also if I add --preset ultrafast --qp 0 for lossless h264 encoding (10bit) all I get is a solid mid grey output. Tried both 32 & 64bit current (1995) 10bit x264 builds from http://x264.nl/

[cretindesalpes]

Correct 10-bit H264 decoding is quite fresh in libav/FFmpeg. Get a recent FFmpeg build (http://ffmpeg.zeranoe.com/builds/) and your color problems will be gone.

[Yellow_]

Great, ffplay shows correct output from latest git ffmpeg. Many thanks indeed. :-)

[Yellow_]

Is it possible to use Dither_convert_yuv_to_rgb and then avs2yuv to pipe to 16bit image sequences to a graphics library like Imagemagick (Q16) or Openexr?

[cretindesalpes]

Using output="rgb48yv12", it should work with any program capable of reading raw RGB48 streams. I haven't tried with Imagemagick, but it looks like it can do it.

[Yellow_]

Using output="rgb48yv12", it should work with any program capable of reading raw RGB48 streams. I haven't tried with Imagemagick, but it looks like it can do it.

Ok, I'll pursue that.

**EDIT** removed queries about x264 colormatrix for HD BT601. Plenty of info here if I'd search first. :-)

[Yellow_]

Using output="rgb48yv12", it should work with any program capable of reading raw RGB48 streams. I haven't tried with Imagemagick, but it looks like it can do it.

I'm not familiar with IM so want to be sure I'm feeding it the 'correct' format out of avisynth so, does this script look correct:

LoadPlugins...

FFmpegSource2("MVI_8068.MOV")
Dither_convert_8_to_16 ()
Dither_convert_yuv_to_rgb(matrix="601", tv_range=false, cplace="MPEG2", chromak="bicubic", mode=6, output="rgb48yv12", lsb_in=true)
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)

Dither_convey_rgb48_on_yv12(r,g,b)

My source files are HD Canon DSLR h264AVC, which is BT601 color matrix and full range luma.

[cretindesalpes]

Mode=6 will be ignored because you're not dithering to 8 bits, and given your script, you would have to use output="rgb48y" to get the three RGB components on interleaved luma planes.

But you can do it in a simpler way:
FFmpegSource2 ("MVI_8068.MOV")
Dither_convert_yuv_to_rgb (matrix="601", tv_range=false, cplace="MPEG2",
\ chromak="bicubic", output="rgb48yv12")

[Yellow_]

Mode=6 will be ignored because you're not dithering to 8 bits, and given your script, you would have to use output="rgb48y" to get the three RGB components on interleaved luma planes.

But you can do it in a simpler way:
FFmpegSource2 ("MVI_8068.MOV")
Dither_convert_yuv_to_rgb (matrix="601", tv_range=false, cplace="MPEG2",
\ chromak="bicubic", output="rgb48yv12")

Thanks for the clarification, so I need nothing more than the above, no Dither_convey_rgb48_on_yv12() needed?

I'm making progress with Imagemagick for writing image sequences, recieved help from the IM forum, all sounds encouraging. :-)

**EDIT**

Success, using the above with avs2yuv and IM (Q16) I can export 16bit image sequences to .tif, .exr and .dpx. :-)

Many thanks

[Yellow_]

Is it possible to visualise and export just the additional information generated / extracted from the video source using dither over and above what is normally seen at 8bit playback?

[cretindesalpes]

Success, using the above with avs2yuv and IM (Q16) I can export 16bit image sequences to .tif, .exr and .dpx. :-)
Excellent! Would you like to publish the command line and other specific instructions so I could add them to the documentation?


Is it possible to visualise and export just the additional information generated / extracted from the video source using dither over and above what is normally seen at 8bit playback?
Call export_rgb48() instead of visu() if you want to export RGB48 data.
FFmpegSource2 ("MVI_8068.MOV")
p = Dither_convert_yuv_to_rgb (matrix="601", tv_range=false, cplace="MPEG2",
\ chromak="bicubic", output="rgb48y")
q = ConvertToRGB (matrix="PC.601")
q = Interleave (q.ShowRed ("YV12"), q.ShowGreen ("YV12"), q.ShowBlue ("YV12"))

dm = mt_makediff (p.Dither_get_msb (), q)
StackVertical (dm, p.Dither_get_lsb ())

# Amplification
Dither_lut16 (expr="x 32768 - 64 * 32768 +", y=3, u=1, v=1)

visu ()

Function visu (clip src)
{
src.DitherPost (mode=6, y=3, u=1, v=1)
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)
MergeRGB (r, g, b)
}

Function export_rgb48 (clip src)
{
src
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)
Dither_convey_rgb48_on_yv12 (r, g, b)
}

[Yellow_]

Excellent! Would you like to publish the command line and other specific instructions so I could add them to the documentation?

This is what I'm using for h264 from my Canon DSLR which is 1920x1088. Thanks to help from the IM community at www.imagemagick.org/discourse-server.

avs2yuv.exe -raw dithertest.avs -o - | convert.exe -depth 16 -size 1920x1088 rgb:- dithertest_%d.tif

Cheers for the exportrgb48() & visu() info. :-)

A sample 16bit video frame: (12.5mb) http://www.yellowspace.webspace.virginmedia.com/test_0.tif

And the export from visualise for same frame: (12.5mb) http://www.yellowspace.webspace.virginmedia.com/visualise_0.tif

Should it really look like that? :-)

[cretindesalpes]

Thanks for the command line.

Should it really look like that? :-)
Yeah, unless I misunderstood your request, it's the amplified difference between 16 and 8-bit conversions. You can comment the x64 amplification line in the script, but it will look grey everywhere.

[Yellow_]

Thanks for the command line.


Yeah, unless I misunderstood your request, it's the amplified difference between 16 and 8-bit conversions. You can comment the x64 amplification line in the script, but it will look grey everywhere.

No that's great, thank you for the functions. I don't quite know what I expected to see. :-)

[mandarinka]

I noticed that there's the possibility to do colormatrix conversion in 16bits now, thanks a lot!

Which chroma kernel would you suggest for converting colormatrix (709->601) using Dither_convert_yuv_to_rgb + Dither_convert_rgb_to_yuv?

My script:
Dither_convert_yuv_to_rgb (matrix="709", output="rgb48y", lsb_in=true)
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)
Dither_convert_rgb_to_yuv (r, g, b, matrix="601", lsb=true)


At first I was thinking about using point scaling because it is kind of reversible, but then I realised that's probably silly because of the rgb step :(

BTW, is this route somewhat compromised by the double conversion, or is that not an issue thanks to the 16bit processing, meaning that this workflow is close to optimal quality?
(Please forgive me for my newby questions, I don't know much about how exactly the colormatrices influence the conversion to rgb, or which yuv channels are influenced by colormatrix conversion, etc...)

[cretindesalpes]

Which chroma kernel would you suggest for converting colormatrix (709->601) using Dither_convert_yuv_to_rgb + Dither_convert_rgb_to_yuv?
You should use something that does not ring too much to avoid false colors. A generic "bicubic" (Mitchell-Netravali) or "spline16" should be fine. You can even try "bilinear", but it may be a bit blurry.

BTW, is this route somewhat compromised by the double conversion, or is that not an issue thanks to the 16bit processing, meaning that this workflow is close to optimal quality?
The rounding errors shouldn't be a problem here thanks to the high bitdepth, however spatial shape and bandwidth losses are the main concerns here: the chroma is resampled twice, for up/downscaling to 4:4:4 and sub-pixel shifting. A direct route could use only one resampling per plane, which would be better. I'll probably include a generic 16-bit colormatrix in a future Dither release. Anyway we already have something of high quality right now with the YUV->RGB->YUV conversions.

[mandarinka]

I see, thanks for the info.
BTW, I think the Dither suite would deserve its own thread, it's not even mentioned in the title...
I really like how this made high bitdepth processing possible:thanks:

[Yellow_]

In searching to find information to read on the methods used and similar methods of the dither plugin, stacked msb/lsb etc I came across this which I found interesting and thought I'd post a link to:

http://www.banterle.com/francesco/publications/download/EGstar2009.pdf

Specifically 3.0 LDR to HDR Expansion on page 4.

Comments?

[SilaSurfer]

The most accurate dithering algorithm are the error diffusion ones (6 to 8). Mode=6 is a classic serpentine Floyd-Steinberg and should give good results. It's a good idea to add a slight bit of noise before (ampn=0.5) to avoid artifacts related to this kind of algorithm. Anyway, I don't think gradfun2db dither can really be considered as good! But it does its job well and very quickly.

Hi cretindesalpes. I'm interested in those three dithering algorithms especially in:

mode 7: Stucki error diffusion + noise. Looks sharp and preserve light edges and details well.

Do you recommend using some noise generation through ampn in this case too and what about apmo setting? I'm intersested very much in this. Thanks

Edit:

Well after doing some testing

ampo - is paramater for incresing amplitude of that fancy ;) ordered dithering from your function

ampn- paramater for using dithering + noise, but raising it increases the amount of noise aka similar to gradfun2db.

Anyway I like ampn setting set to 0, ampo setting from 2~3. It plays a lot better with compressibility compared using noise like from Gradfundb. :rolleyes:

[TheProfileth]

Hmmm I don't know why but every time I use gradfun3 (since version 1.5) all it does is output the luma (like I get the generic green from removing chroma) if I leave the mask on. If I set the mask to 0 then this does not happen, I am sort of thinking it has something to do with the introduction of the dither_removegrain but I don't really know. Either way could someone propose a solution to this problem.

[mandarinka]

That seems to happen if you have masktools conflicts.

Assuming you use plugin autoload directory: get rid of masktools.dll (that is masktools v1... or was it mt_masktools.dll?), and make sure you only have installed that Masktools2 a48 version that cretindesalpes links. If you run on avisynth 2.5.x, use the mt_masktools-25.dll file, if you are on avisynth 2.6, use mt_masktools-26.dll. Having both or the wrong compile (xxx-26/xxx-25) seems to give the green screen.

[cretindesalpes]

ampn- paramater for using dithering + noise, but raising it increases the amount of noise aka similar to gradfun2db.
Actually we have two different uses here:

If you dither for direct display or further 8-bit processing, you need the maximum visual quality. Error diffusion algorithms perform much better than simple ordered dithering. Adding noise before can be interesting to add a temporal component to the dithering, and to homogenize flat and gradient parts.
If you dither for efficient video compression, prefer the ordered dithering (Bayer matrix) without noise.
Note that ampo and ampn parameters have an effect on all the dithering modes, ordered and error diffusion. Also, the noise is always added before dithering, not after (like in GradFun2DBmod). This allows it to be part of the noise shaping.

@TheProfileth:

As mandarinka said, it's likely a problem of Masktools version.

[SilaSurfer]

Thanks cretindesalpes for your explenation.

[pereant]

Thanks so much for the dither package - this works really really great and it appears to be very benefiting in terms of compressibility. as far as i understand it, it prevents from strong visible blocking artefacts. in other words they are still there, but not significant. is this right so far?

[cretindesalpes]

Dither 1.9.1 (http://forum.doom9.org/showthread.php?p=1386559#post1386559) released:
Improved memory usage and CPU load for the functions based on the 16-bit resizer.
Simplified a bit the GradFun3() detail masking.
Fixed a few glitches.
Next, I'm planning to add native multithread support to the most CPU-intensive functions. Currently, the Dither functions support all the MT modes, but these aren't available in recent 2.6 Avisynth versions.

as far as i understand it, it prevents from strong visible blocking artefacts.
The target of gradient smoothing is not strong blocking. Strong blocking artifacts are addressed by deblocking and deringing filters. Actually, banding artifacts are not really blocking; the main problem appears in the final colorspace. Of course, too high quantization during compression will cause color banding, the light dithering noise being filtering out. However dithering noise is hard to compress, and if you spend all your bits in the dither, you'll end up with a poorly encoded picture. By changing the dither from noise to repetitive patterns (ordered dithering), you make the dither more predictable, thus much more compressible, at a minor visual cost.

[Dogway]

Thank you! just before my encode! I wanted to request 2 new options for dither if it's possible.
One is implement nnedi3 for resize. I think I can do some masking add difference et al but do you think is possible to do it natively?
The other is when encoding anamorphic, to have also anamorphic dither.

edit: I just noticed that when using 4 taps Dither_resize16 crashes, so spline64, lanczos 4 taps, etc.

[Yellow_]

Many Thanks.

[pereant]

Lovin' it :)

http://imgur.com/a/Oc7Pu

But i guess i won't use Dither seriously until i can effort a new pc. It would take me a week to run this script. Core2Duo T7200 atm. :(

[cretindesalpes]

But i guess i won't use Dither seriously until i can effort a new pc. It would take me a week to run this script. Core2Duo T7200 atm. :(
If you just did Dither2Pre ().DitherPost () as quoted in your screenshots, no wonder it's slow. IIRC the default Dither2Pre() creates something like 7 instances of dfttest! You can achieve the same result just with dfttest (lsb=true).DitherPost (), but much faster.

An if your goal is denoising without creating colorbanding, you'll probably get better visual results by using only the previous line with a carefully tuned motion compensation.

One is implement nnedi3 for resize. I think I can do some masking add difference et al but do you think is possible to do it natively?
I haven't studied the nnedi3 code in detail, but I have the feeling that the masking solution will be simpler! One could just modify nnedi3 to make it output the mask of the pixels processed with the neural networks.

The other is when encoding anamorphic, to have also anamorphic dither.
Why would you want this one? For maximum efficiency, the dither must be related to the pixel grid, not to the aspect ratio. It's not intended to be actually seen.

I just noticed that when using 4 taps Dither_resize16 crashes, so spline64, lanczos 4 taps, etc.
Interesting. I'll check this issue.

Despite this my LinearResize function works pretty much ok, one just to be aware that it must not be optimal for rgb sources, since you will need mod2 input and at least avisynth 2.6.0 to avoid chroma subsampling. Let it be what it is, focused on video sources the same as Dither originally is.
You can bypass the YUV conversion by using ShowRed ("YV12"), ShowGreen and ShowBlue, resizing the luma channel of these 3 clips and merging the components back with MergeRGB().

[pereant]

If you just did Dither2Pre ().DitherPost () as quoted in your screenshots, no wonder it's slow. IIRC the default Dither2Pre() creates something like 7 instances of dfttest! You can achieve the same result just with dfttest (lsb=true).DitherPost (), but much faster.

An if your goal is denoising without creating colorbanding, you'll probably get better visual results by using only the previous line with a carefully tuned motion compensation.

Sorry for my ignorance. I just had a look what dfttest really does/is. It looks like as if it makes the use of the fft3dfilter in my script rather unnecessary i guess. Have to play around w/ it - Thanks!

[cretindesalpes]

Dither 1.9.2 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Fixed crashes and bugs in Dither_resize16().

[Yellow_]

Hi

I was having some odd resizing problems with 1.9.1, the image was getting split over two frames, noise in one, image in other, some green frames and was going to post my queries but you provided 1.9.2 before I got time to comment. :-)

But still having problems with 1.9.2.

First, should this work uprezzing a 1920x1088:

Dither_convert_8_to_16 ()

Dither_resize16 (3480, 2176)

Dither_convert_yuv_to_rgb(matrix="601", tv_range=false, cplace="MPEG2", chromak="bicubic", output="rgb48yv12")

With the above writing out to imagemagick (and specifying the same image resolution) with avs2yuv I get pairs of images for each frame of video, so processing 6 frames of video gives me 12 images, the first image of each pair is 'normal', the second image of each pair is the grey and green noise.

On the CLI Dither tells me: script.avs: 6960x8704, 25fps, 6 frames

[cretindesalpes]

Just add the parameter lsb_in=true to Dither_convert_yuv_to_rgb(). Actually, the functions cannot know the kind of data (8 or 16 bits) they are fed with, so it's important to tell them when multiple cases are allowed.

[Yellow_]

Just add the parameter lsb_in=true to Dither_convert_yuv_to_rgb(). Actually, the functions cannot know the kind of data (8 or 16 bits) they are fed with, so it's important to tell them when multiple cases are allowed.

That's great thanks, also seem to notice quite a speed up. Excellent.

[TheElix]

Hey, cretindesalpes, your script is magic. http://screenshotcomparison.com/comparison/64373
I'm total noob in AviSynth, but I was suggested to use these lines:
SetMTMode(3)
ffdShow_Source()
SetMTMode(2)
Dither_convert_8_to_16 ()
SmoothGrad(thr=1.0, radius=8)
DitherPost ()
It works wonders on such sources as above. But I was wondering if there's any room for improvement here.

[mp3dom]

It works wonders on such sources as above. But I was wondering if there's any room for improvement here.

There's something wrong, I think you should lower the values of smoothgrad or use Gradfun3 with mask because you have a lot of noise around the edges.

[cretindesalpes]

It works wonders on such sources as above. But I was wondering if there's any room for improvement here.

It's hard to give advices just on a single picture. The thr parameter is high and efficient for removing the dirt on the flat areas, but I fear it will affect details too much. And as mp3dom said, it makes halos around the edges (but masking is almost useless for this kind of artifacts). Using a bilateral filter for the main task will improve things a lot, albeit being slower.

FFImageSource ("Source_screenshot_7849102365.jpg")

Dither_convert_8_to_16 ()
o16 = last
r = 8
Dither_bilateral16 (radius=r, flat=0.75, subspl=8)
Dither_limit_dif16 (last, o16, thr=0.7)
SmoothGrad (thr=0.15, radius=r*2)
DitherPost ()

Or you can just use a denoiser with a high spatial range like dfttest. As a side effects, it will greatly reduce the blocking and smooth the edges, but will blur the picture a little bit.

FFImageSource ("Source_screenshot_7849102365.jpg")

o = last
r = 8
dfttest (sigma=r, tbsize=1, sbsize=r*2, sosize=r*3/2, lsb=true)
f = DitherPost (mode=-1)
d = ContraSharpening_dif (f, o)
DitherPost ()
mt_adddiff (last, d, y=3, u=2, v=2)

Function ContraSharpening_dif(clip denoised, clip original)
{
# contra-sharpening: sharpen the denoised clip, but don't add more to
# any pixel than what was removed previously.
# script function from Didee from the VERY GRAINY thread
# Damp down remaining spots of the denoised clip.
s = denoised.minblur(1,1)
# The difference achieved by the denoising.
allD = mt_makediff(original,denoised)
# The difference of a simple kernel blur.
ssD = mt_makediff(s,s.removegrain(11,-1))
# Limit the difference to the max of what the denoising removed locally.
ssDD = ssD.repair(allD,1)
# abs(diff) after limiting may not be bigger than before.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?")
# Apply the limited difference. (Sharpening is just inverse blurring.)
# denoised.mt_adddiff(ssDD,U=2,V=2)
return (ssDD)
}

[TheElix]

With the last script I have this error http://rghost.ru/14201971/image.png
As for the first one, it gives me a slideshow... Strange, even though CPU usage isn't rising above 65%.
As for the quality, here's a comparison between my old script and the new one: http://screenshotcomparison.com/comparison/65664
I have to add that haloing is barely noticable here on my PDP. You may say it's plasma panel's own dithering that prevents to see those 'details' or you may say my panel is good at masking those artifacts :) But yes, I do see haloing in my old variant now. But these new settings are either too much stress on my system or there's something in the them that's making video stutter.

[cretindesalpes]

With the last script I have this error http://rghost.ru/14201971/image.png
Install the modified dfttest version included in the Dither archive.

As for the first one, it gives me a slideshow...
Yeah, I said it's slower (bilateral filters are very slow). Definitely not for realtime.

[TheElix]

I failed to mention I was looking for a real-time version... Could you suggest a setting that's between the two versions that have been compared?

[cretindesalpes]

For realtime, just stick with your SmoothGrad script, dfttest or bilateral-based processing will be overkill. You can lower the thr parameter to something like 0.5-0.7 to reduce the halos, but the original banding artifacts will become more visible. You'll have to find the right compromise according to your taste.

[Rumbah]

I'm using MCTemporalDenoise(settings="very high", edgeclean=true) to denoise a clip of mine. It works great but introduces some banding.
If I use enhance=true it dithers the output with gradfun2dbmod if I understand it correctly (I guess as the last step). It looks good but encoding with x264 uses a lot of bandwidth.
So I thought of using dither for postprocessing with gradfun3. Can I directly use gradfun3 as a substitute for gradfun2db in gradfun2dbmod? Or can you convert the settings from one to the other to get a similar result?
I wasn't able to achieve a similar result by playing with the settings so I'd be grateful for some help.

[cretindesalpes]

You can replace GradFun2DBmod with GradFun3, not just GradFun2DB. The GradFun3 defaults are similar to GF2DBm excepted it doesn't generate additional grain and processes luma and chroma the same way. You'll find below a conversion table for the parameters of equivalent role:
GradFun2DBmod Conversion GradFun3
------------- ---------- --------
thr * 0.3 thr
radius = mask
range = thr_det

Anyway, you can still replace GradFun2DB with GradFun3 in GradFun2DBmod. Use the following settings:
LUM = RDY.GradFun3(thr=thr *0.3, mask=0).Crop(...)
CHR = RDY.GradFun3(thr=thrC*0.3, mask=0).Crop(...)
These lines appear three times in the GF2DBm code, with different Crop() values, make sure you don't forget one or mix the Crop().

[Rumbah]

Thanks for your help, I'll try to "mod" MCTemporalDenoise to use dither instead of gradfun2db.

I hope it gets me some compression gain while still looking good (I don't know if differing between chroma and luma has a visible impact).

[Rumbah]

I just tested it and it's much more compressible now although there's a little more banding.

Just to be sure that I got it right here are the changes (in bold):
\ : (GPU==false&&fixband==true ) ? i.FFT3Dfilter(sigma=sigma*0.8,sigma2=sigma*0.6,sigma3=sigma*0.4,sigma4=sigma*0.2,bw=bwbh,bh=bwbh,ow=owoh,oh=owoh,plane=chroma?4:0,bt=bt,ncpu=ncpu).gradfun2db(thr=1.0+sigma*0.05)
\ : i.FFT3Dgpu(sigma=sigma*0.8,sigma2=sigma*0.6,sigma3=sigma*0.4,sigma4=sigma*0.2,bw=bwbh,bh=bwbh,ow=owoh,oh=owoh,plane=chroma?4:0,bt=bt,precision=precision).gradfun2db(thr=1.0+sigma*0.05)

\ : (GPU==false&&fixband==true ) ? i.FFT3Dfilter(sigma=sigma*0.8,sigma2=sigma*0.6,sigma3=sigma*0.4,sigma4=sigma*0.2,bw=bwbh,bh=bwbh,ow=owoh,oh=owoh,plane=chroma?4:0,bt=bt,ncpu=ncpu).gradfun3(thr=(1.0+sigma*0.05)*0.3, mask=0)
\ : i.FFT3Dgpu(sigma=sigma*0.8,sigma2=sigma*0.6,sigma3=sigma*0.4,sigma4=sigma*0.2,bw=bwbh,bh=bwbh,ow=owoh,oh=owoh,plane=chroma?4:0,bt=bt,precision=precision).gradfun3(thr=(1.0+sigma*0.05)*0.3, mask=0)
\ : mt_makediff(smL,mt_makediff(smL,smP,U=chr31,V=chr31).gradfun2DB(thr=1.0+post*0.5),U=chr31,V=chr31)

\ : mt_makediff(smL,mt_makediff(smL,smP,U=chr31,V=chr31).gradfun3(thr=(1.0+post*0.5)*0.3, mask=0),U=chr31,V=chr31)
GFc = (enhance==true) ? TTc.GradFun2DBmod(thr=GFthr,thrC=chroma?GFthr:1.0,mode=0,str=AGstr,strC=0.0,temp=temp,adapt=bias,mask=false,show=false)

GFc = (enhance==true) ? TTc.GradFun3(thr=GFthr*0.3,mask=0)
I wasn't sure about the complex gradfun2dbmod call, perhaps I did something wrong.

Changing gradfun2dbmod to use gradfun3 didn't result in the desired effect of better compressibility as it seems that it adds noise in addition to gradfun2db.

[cretindesalpes]

Looks OK to me. If you feel you have more banding than before, increase GFthr, or the 0.3 multiplier in the modified parts so it matches your taste.

EDIT: only the last part (enhance==true) needs to be changed. You can leave the other GradFun2DB, or add mode=6 to these first GradFun3 calls.

[Rumbah]

Thanks again.
I only changed the enhanced==true line and it worked as intended.

I noticed some strong haloing so I started tinkering with the values again and noticed that using
GradFun3(thr=1.8,mask=0,smode=2,radius=16)crashed dither.
GradFun3(thr=1.8,mask=0,smode=2)works.
(I need some strong debanding and there aren't many details as it's an 16x upscale ;) )

EDIT: I also noticed that I mistook the halo artifacts for banding so with smode=2 the quality is much better and the high threshold isn't needed.

EDIT2: After some testing I found that using a radius>13 crashes the above example.

[Disabled]

Did you try other ordered dithering matrices like those generated by the void and cluster algorithm (very bottom (http://caca.zoy.org/wiki/libcaca/study/2))?
In my tests they showed a similar compressibility to other ordered dithering algorithms and looked superior to bayer dithering.

[cretindesalpes]

EDIT2: After some testing I found that using a radius>13 crashes the above example.
Oh right, thanks for the feedback.

Dither 1.9.3 (http://forum.doom9.org/showthread.php?p=1386559#post1386559) released:


Fixed crashes in Dither_bilateral16() occuring with subsampling using more than 32 reference pixels. This affected GradFun3(smode=2) with radii greater than 13 and default subsampling settings.
Fixed a mod4 check in Dither_convert_yuv_to_rgb().


Did you try other ordered dithering matrices like those generated by the void and cluster algorithm (very bottom (http://caca.zoy.org/wiki/libcaca/study/2))?
In my tests they showed a similar compressibility to other ordered dithering algorithms and looked superior to bayer dithering.
I was afraid of the compressibility so I implemented only the Bayer matrix, but if you say the V&C algorithm has a good compressibility too, I'll probably add it later.