Color banding and noise removal

[osgZach]

Is it possible to deband a source and also denoise it?

Should I denoise first and then deband?

Using GradFun2DB for instance, it won't get rid of it but it will take the banding down a notch, however if I were to there apply something like LSFMod, or in general some other noise reducers / it will undo the bebanding( as it seems it just adds grain to break it up), and then the banding will come back with even more clarity and well defined borders.

So it ends up looking like an ugly gradient, for lack of a better term, with those wavey line borders and all that fun stuff.

What also puzzles me, is that even though I am doing no color mode conversions, etc to even create this banding (its present on the DVD source) when I watch either the DVD or a straight unfiltered encode of the same episode, you can almost not even tell it is there on the TV screen.

Why is that?

Any suggestions on some strong, but safe settings for GradFun2DB, or other x64 compiled filters for this (degrainmedian, etc?) would be preferred. I'll provide a source section once a test encode I am running is done, if its needed. But this was more of a general question on effective settings for some filters versus a plea for someone to do the work for me.

[OvejaNegra]

After ALOT of battle agains grain VS Banding i can tell you this for sure:
You eat the grain, you ll get banding.
Some sources are encoded without dithering and have banding.
Some sources have no grain but are encoded with dithering to avoid banding (dithering in the end is noise like grain for you)

My actual choices (after MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANY teest with animes and grainy sources)

Opt A: Do a strong denoising using FFT3d or moderate denoising using vaguedenoiser or anything like that -> good compression (not too high bit rate needed, not too big file) -> banding -> enable deband filter on FFDSHOW duting playback and it works excellent.

Opt B: Calm the grain using some temporal grain compressor like TTEMPSMOTH (my favorite) or this:
http://forum.doom9.org/showthread.php?t=139766
or this:
http://forum.doom9.org/showthread.php?t=130611&highlight=grain+optimizer
that will give you a more stable grain
then -> encode using high bit rate (1800 kbps or more) using grain retention settings on X264 or qpel + MPEG matrix on XVID -> Bigger file -> no banding -> looks very nice to me (for anime i'm ok sometimes with the anime preset on x264, i don need the grain preset)


IF your source have banding -> subtle denoiser (there is ALWAYS some noise on DVDs) -> encode normally -> enable deband on ffdshow during playback.

Or

Encode using gradfun or THIS: GradFun2DBmod.v1.3.avsi (google) Your source will look grainy again but it wont have banding. Of course you need high bit rate for this.


But i can tell you this for sure >
DVD sources have grain or dithering (i cant speak for HD yet, ask other person) so Degrain or denoise + no banding is impossible.

Yes on the TV (a common TV) you can see the banding even if is an encode or the original DVD with banding.
When you are on your monitor maybe bacause a bigger luma range you can see the banding. If you have a flat display it will look worst.

It's up to you, keep the grain or remove it and leave the banding. If your DVD has banding, well use some of the solutions i tested.

Good luck


EDIT:
Read this:
http://forum.doom9.org/showthread.php?t=118255&highlight=grain+optimizer

[cretindesalpes]

(Original reply replaced by this Dither presentation.)


Download

>>>> dither-1.28.1.zip <<<< (https://ldesoras.fr/src/avs/dither-1.28.1.zip)
Main scripts and plug-ins from the Dither package, including avstp.dll.
>>>> dfttest 1.9.4 <<<< (https://ldesoras.fr/src/avs/dfttest-1.9.4.zip)
Mod16 version of dfttest.
>>>> MVTools 2.6.1.0 <<<< (https://ldesoras.fr/src/avs/mvtools-2.6.1.0.zip)
Mod16 version of MVTools 2
Masktools 2 a48 (http://manao4.free.fr/) or above is required. Previous Masktools versions will silently fail and cause the greenish screen of death. Please update your Masktools before installing the Dither script.

All these plug-ins include the -Vit- threading fixes.


What is it?

Dither is a package including several scripts and avisynth plug-ins. Some of them are modification of existing ones (dfttest and MVTools 2). I primarily wrote the Dither tools to fight color banding (http://en.wikipedia.org/wiki/Color_banding) during denoising in an unusual way. But Dither can perform various operations :

Recover the high bitdepth data possibly contained in a noisy clip. This can be seen as an "inverse-dithering" operation. In computer graphics, people usually use a gaussian filter to turn halftones into shades of grey. We'll use denoisers here.
Dither a high bitdepth picture into a standard YV12. The picture may have been obtained with the inverse-dithering operation, the combination could be called "transdithering".
Smooth fine gradients to remove colorbanding, resulting in a high bitdepth picture which can be dithered later.
Performing basic operations (masking, curves...) on high bitdepth pictures, as they cannot be manipulated safely with conventional avisynth filters.


Why should I use it?

Because denoising and lossy compression can easily make colorbanding appear. By transdithering, it is possible to recover original gradients better than using a filter + GradFun2DB(mod). Moreover, classic error-diffusion dithering algorithms require a high bitrate during encoding, otherwise color banding appears again. Dither offers an ordered dithering (http://en.wikipedia.org/wiki/Ordered_dithering) (Bayer matrix), which is more resilient to data compression.


How can I use it?

The simplest way to transdither a relatively clean source, using the modified dfttest:
dfttest (sigma=2, lsb=true)
DitherPost ()

You can add a SmoothGrad() call between both statements to remove colorbanding present in the source. And because dfttest is a spatio-temporal filter, it would be better to also use motion compensation (http://forum.doom9.org/showthread.php?t=147676) to increase filter accuracy and prevent ghosting.

For an already denoised source, use
GradFun3 ()

GradFun3 is intended to replace GradFun2DB(mod). Actually it works on the same principle and does almost the same things, but it uses ordered dithering by default, which is better for compressibility.

[osgZach]

Interesting solution you have there.

As for this particular source, I decided to just let it be as it was too much confusion on my part. Its worth noting I don't watch much on my PC, I encode for playback on my WDTV Live on a large screen (sadly I don't know its size haha) HDTV. Compressibility isn't much of a concern as x264 under CRF gives close to DVD quality results in the upper range of sub-1000kbps (on the TV). But I'll probably come back to this for future encodes I know will be pretty noisy as they were poor transfers to DVD.

[Terka]

could you please post same samples?

[osgZach]

Clip #1 (http://www.oldskoolgames.com/videos/clip1.demuxed.m2v) This is not an example clip, I was wondering however if this is an example of Halo artifacts? (the edges around the students portable game thing)

Clip #2 (http://www.oldskoolgames.com/videos/clip2.demuxed.m2v) - This is the color banding (I hope) clip. It's 26MB and spans several scenes which each have varying degrees of banding. I have never been able to successfully filter them, or reduce noise without greatly enhancing the visibility of the bands.

These come from Death Note, Episode 1, R1 DVD

[osgZach]

The only thing I've had the patients to sit around and check was Dither1Pre ().Dither1Post ()


Which did make a minor difference but I think mostly did some denoising.. Under x32 using some of the other examples (which I honestly copied and pasted, I can't even begin to understand them or how they work) I got some pretty nice results that pretty much made it look about as good as watching it on my TV or converting the file to RGB. Although banding was still there (big reduction though)..

It would be really nice if someone could port the modified DFTTEST linked to above, to x64 so I can try this out under JoshyD's MT_x64 build. Maybe some other people would find it useful too.

I recently purchased Last Exile when I saw it at Best Buy.. A really nice show, and very good quality for the most part. However I was disheartened to find the same horrible color banding issues when I popped it into the drive and took a quick peek at it.. Very dismayed :(

Seems to be a rather ugly trend in the industry to not push these shows out the door properly... They have to know this is a huge problem :(

[cretindesalpes]

I think the banding you have comes from the final YV12 to RGB conversion. I tried to process your second clip with the following script :

FFmpegSource2 ("clip2.demuxed.m2v")
TFM ()
TDecimate (mode=1)

zzz_denoise ()
DitherPost (mode=2)

Function zzz_denoise (clip src, float "sigma", int "thr", bool "mask", int "sad")
{
sigma = Default (sigma, 16)
thr = Default (thr, 5)
mask = Default (mask, False)
sad = Default (sad, 200)

w = src.Width ()
h = src.Height ()

# Motion analysis
super = MSuper (src)
super_a = MSuper (src.TTempSmooth ().RemoveGrain (12))

fwd_vect_3 = super_a.MAnalyse (isb=false, delta=3, overlap=4)
fwd_vect_2 = super_a.MAnalyse (isb=false, delta=2, overlap=4)
fwd_vect_1 = super_a.MAnalyse (isb=false, delta=1, overlap=4)
bck_vect_1 = super_a.MAnalyse (isb=true, delta=1, overlap=4)
bck_vect_2 = super_a.MAnalyse (isb=true, delta=2, overlap=4)
bck_vect_3 = super_a.MAnalyse (isb=true, delta=3, overlap=4)

fwd_comp_2 = src.MCompensate (super, fwd_vect_2, thSAD=sad)
fwd_comp_1 = src.MCompensate (super, fwd_vect_1, thSAD=sad)
bck_comp_1 = src.MCompensate (super, bck_vect_1, thSAD=sad)
bck_comp_2 = src.MCompensate (super, bck_vect_2, thSAD=sad)

# Spatio-temporal denoising using modified dfttest
c_dft = Interleave (fwd_comp_2, fwd_comp_1, src, bck_comp_1, bck_comp_2)
c_dft = c_dft.dfttest (sigma=sigma, lsb=true) # Double height
c_dft = c_dft.SelectEvery (5, 2)

# Temporal-only denoising using modified MDegrain
c_deg = src.MDegrain3 (super, bck_vect_1, fwd_vect_1, bck_vect_2, fwd_vect_2, bck_vect_3, fwd_vect_3, thSAD=sad, lsb=true) # Double height

# Spatio-temporal denoising smoothes too much the details,
# therefore we use pure temporal denoising on edges or detailed areas.
edge_src = c_deg.Crop (0, 0, w, h)
edge_mask = edge_src.mt_edge (mode="prewitt", thY1=thr, thY2=thr)
edge_mask = edge_mask.mt_expand ()
edge_mask = StackVertical (edge_mask, edge_mask) # Double height
c_hyb = mt_merge (c_dft, c_deg, edge_mask, luma=true, y=3, u=3, v=3)

return (mask ? edge_mask.GreyScale () : c_hyb)
}

(This script also requires a modified version of MDegrain3. I added a link to my previous post (http://forum.doom9.org/showthread.php?p=1386559#post1386559))

I've got pretty good results regarding denoising and debanding. However, depending on which software I play the processed clip, bands may appear or not, and obviously come from the player's YV12->RGB converter. But I don't know if it's only a matter of codec or if the video card's driver is involved too.

An example :

Original, noisy clip :
http://img24.imageshack.us/img24/2337/noisy.th.png (http://img24.imageshack.us/img24/2337/noisy.png)

Denoised, with bands :
http://img576.imageshack.us/img576/815/denoised.th.png (http://img576.imageshack.us/img576/815/denoised.png)
On this picture, one can see clearly the YV12->RGB conversion problem. Every six bands, there is a bigger jump caused by a step of 1 unit in luma values converted to a step of 2 units in the RGB colorspace instead of 1.

Denoised and dithered :
http://img46.imageshack.us/img46/2183/dithered.th.png (http://img46.imageshack.us/img46/2183/dithered.png)

For the record, GradFun2DBMod :
http://img356.imageshack.us/img356/8567/gf2dbm.th.png (http://img356.imageshack.us/img356/8567/gf2dbm.png)

To show the effect, a magnified part of the picture above (bottom left), "HSV stretched" (with GIMP) :
http://img63.imageshack.us/img63/7898/magnifiedandstretched.png
Original / Denoised
GradFun2DBMod / Dithered

[osgZach]

There shouldn't be ANY YV12 - > RGB conversion going on. I work on my clips in Vdub, so unless its auto color mode detection is borked up and doesn't work, what I am getting -should- be YV12 ? Or is it gonna be RGB at viewing stage regardless, just by nature of PC displays?

My encoding path is off the DVD to a HuffYV12 (ffdshow) file. Not sure if that would have any impact on it though. I will say doing a color conversion to RGB seems to clean it up rather nicely all by itself. So I'm not sure why even at the viewing stage, an RGB conversion would look so horrible.

That being said. I appreciate your efforts. I'm not sure if I have tried Gradfun2db MOD, but I bet I at least played around with the original.. Your GF2DBMod image looks very nice for certain. I may have to install Set's 2.5.8 MT x32 version of Avisynth I guess so I can at least experiment at reasonable speeds.

Would anything during the DGindex process affect this too? i.e there is PC / TV modes for clamping some color stuff IIRC. I don't recall which I used but probably default to PC..

[OvejaNegra]

osgZach:
I havent tested your samples, but there are animes with banding, i dont know why but it happens.
GradFun2DBMod should put grain on those banded areas, yes you need a higgher bit rate or x264 will eat the grain again.

By the way (maybe you dont know this):
VDUB will convert yoyr source to RGB if you use full processing mode and the codec will re-convert to YV12 during encoding (in case you are encoding with xvid / divx / etc on vdub).

Go to the options menu / preferences -> output color depth: is on fastest 16 bit? because if it is, THATS the cause of your banding.
you should put 24 bit or match display depth :)

Maybe you know this but maybe not, but you mention you are using VDUB and maybe you should check this.

And, if you eat your grain with FFTxxx or vaguedenoiser or any good spatial filter, well you always will have banding.
The mpeg2 encoders have dithering for make the color conversions (like CCE, i dont know the modern versions, but the old SP has it).
Try a temporal filter to calm the grain on the static surfaces like ttempsmotth (the only one who gave me a good looking Kanokon).
I can send you samples if you like it, there is one scene on episode one that you can use to test your anti-banding strategy and find the best filter to your needs :)

Good luck
Please check VDUB settings

[osgZach]

My Vdub should be set to "use output settings". Actually not quite sure what that means, but I definitely take it off the 16-bit junk every time I setup a fresh install, etc.. As for my processing mode, I never use full. I use fast recompress, so it pass right through without being touched, except by Avisynth. So in my mind its VOB - > FFDShow (in this case FFDshow's HuffYV12), which I then send through x264 via an AVS script after I am done working on a file.

I do have "Enable dithering for 16-bit displays" turned off as well.. Not sure if that would impact anything. But at any rate.. I probably need a break from all this stuff..

[OvejaNegra]

Output color depth:
"Selects the default color precision for video display -- 24-bit looks better, but 16-bit is generally faster. This does not affect file output"
Enable 16-bit dithering:
"If enabled, 24-bit images are dithered when displaying in 16-bit to reduce banding, at the cost of a little speed. This does not affect file output"


If you set that to "Fastest 16 bit" the videos looks like if your display to 16 bits color. They band a lot. If you enable dithering, well you'll see grain/noise/dither that is not part of the original source.

I always use "Match display depth"

you could install xvid and disable raw video support from the VFW and the DSHOW filter, so xvid will handle YV12 content, to see if that helps. (The divx pack also have a YV12 decoder).

[cretindesalpes]

dither.avsi updated to v1.1, see the original post (http://forum.doom9.org/showthread.php?p=1386559#post1386559).

[Floatingshed]

I get "the named argument clip was passed more than once to blankclip". Line 755, 413, 250.
Please help....

[foxyshadis]

Replace your version of the script with the one in the zip file. Yours might be modified or corrupted.

[dlnm]

Is it possible to use other denoisers instead of MDegrain or dfttest?

[cretindesalpes]

Is it possible to use other denoisers instead of MDegrain or dfttest?Yes. Use Dither1Pre() or Dither2Pre() to generate a 16-bit denoised clip (actually it will have only 9 or 10 significant bits).

Usage :Dither2Pre ("YourFavoriteDenoisingFilter (param=1234)")
DitherPost ()
If your filter has signal amplitude dependent parameters (thresholds, etc.), multiply them by 2 or 4 for Dither1Pre and ~2Pre. The drawback of this function is that the processing time will also be multiplied by 2 or 4.

[andrew_seul]

dither.avsi updated to v1.1, see the original post (http://forum.doom9.org/showthread.php?p=1386559#post1386559).

There's any chances that you can port those plugins you modifed to 64 bit mode (including NoiseGenerator also)? I need a little speed on my encodings thats why Im asking :)

[Jawed]

Just chain stuff together :devil:

e.g. I use this to kill blocking artefacts:



function BlockKiller(clip source)
{
y=source.greyscale()
y=y.ChannelBlockKiller()
u=source.utoy()
u=u.ChannelBlockKiller()
v=source.vtoy()
v=v.ChannelBlockKiller()

return ytouv(u, v, y)
}

function ChannelBlockKiller(clip source)
{
source
""" spline36resize(source.width*16,source.height*16)
gradfunkmirror()
gradfunkmirror()
gradfunkmirror()
spline36resize(source.width*8,source.height*8)
gradfunkmirror()
gradfunkmirror()
gradfunkmirror() """
spline36resize(source.width*4,source.height*4)
gradfunkmirror()
gradfunkmirror()
gradfunkmirror()
spline36resize(source.width*2,source.height*2)
gradfunkmirror()
gradfunkmirror()
gradfunkmirror()
spline36resize(source.width,source.height)
gradfunkmirror()
gradfunkmirror()
gradfunkmirror()

return last
}

notice, in the function ChannelBlockKiller, that the *16 and *8 sections are commented out with the """. As posted, that code only uses *4 and *2 sizes, before returning the picture to its original size.

Basically you can adjust the maximum size, e.g. make it larger for smaller frames by un-commenting the *8 and *16 sections. The basic concept here is that any resizing algorithm creates a gradient on all edges (banding and non-banding). The gradient will be "smooth" but "regular". So then apply gradfunkmirror() to dither that gradient. Then resize down. Resizing down averages the dither.

The quantity of steps and the count of gradfunkmirror() calls per step adjust the result. You adjust for blockiness versus softness. (I'm still experimenting with this function - which is why there are no funky parameters, as I haven't decided which parameters are important and how to graduate the effect.)

In general you can simply use 2 or more instances of gradfunkmirror(), one after the other, to control most banding.

Also, I always use gradfunkmirror() on playback (the DeBand filter in FFDShow).

Jawed

[Gavino]

notice, in the function ChannelBlockKiller, that the *16 and *8 sections are commented out with the """.
An innovative use of quotes.
Why not use a real comment /* ..... */ which can span multiple lines.

[Jawed]

An innovative use of quotes.
Why not use a real comment /* ..... */ which can span multiple lines.
Not on my system.

If Avisynth allows that, now, then :cool:

I'm using 2.57 tsp MT version 5, built March 1 2007 - it works for the things I do - so I have no interest in spending time debugging anything newer.

[Floatingshed]

Replace your version of the script with the one in the zip file. Yours might be modified or corrupted.

That is the version I'm using...

[cretindesalpes]

The dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559) has been updated to version 1.3.

[tormento]

Do you plan to release a x64 aware version?

[cretindesalpes]

I don't know. I will release soon an additional function to smooth the low gradients in the 16-bit world, and a derived function roughly equivalent to gradfun2db(). Then, I'll try to port the whole thing to x64, but only if I can obtain the source code for the x64 ports of dfttest and MVTools2. Currently, I have no news from JoshyD on this topic.

[tormento]

However, thanks ;)

[tormento]

I am trying to use your package but I keep on getting VirtualDub crashing when opening the file. Please, tell me a simple, stupid denoise script you get working. =P

[cretindesalpes]

I am trying to use your package but I keep on getting VirtualDub crashing when opening the file.

"Crashing" is very vague and I cannot do anything with only this bit of information. Please copy the crash error message and the script you tried to open.

Please, tell me a simple, stupid denoise script you get working. =P

This is taken from the dither.avsi documentation:

h = Height ()
w = Width ()
dfttest (lsb=true)
msb = last.Crop (0, 0, w, h)
lsb = last.Crop (0, h, w, h)
DitherPost (msb, lsb)

[tormento]

Please copy the crash error message and the script you tried to open.
I always use the word crash when I have nothing to report, i.e. the error is so severe the program simply hangs or outputs a stupid meaningless hex error.
This is taken from the dither.avsi documentation:
Mm.. the script I tried to wrote was even simpler but took in account both pre and post. Wait a couple of hours, I'll arrive at office and try again, now I'm on a sluggish train..

[tormento]

Ok, I had some spare time to play with dither but I can't see any difference between the results of these 2 scripts:

super = MSuper(pel=2, sharp=1)
bv3 = MAnalyse(super, isb = true, delta = 3, overlap=8, blksize=16)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=8, blksize=16)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=8, blksize=16)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=8, blksize=16)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=8, blksize=16)
fv3 = MAnalyse(super, isb = false, delta = 3, overlap=8, blksize=16)

MDegrain3(super, bv1, fv1, bv2, fv2, bv3, fv3, thSAD=400)


h = Height ()
w = Width ()

super = MSuper(pel=2, sharp=1)
bv3 = MAnalyse(super, isb = true, delta = 3, overlap=8, blksize=16)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=8, blksize=16)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=8, blksize=16)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=8, blksize=16)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=8, blksize=16)
fv3 = MAnalyse(super, isb = false, delta = 3, overlap=8, blksize=16)

MDegrain3(super, bv1, fv1, bv2, fv2, bv3, fv3, thSAD=400, lsb=true)

msb = last.Crop (0, 0, w, h)
lsb = last.Crop (0, h, w, h)
DitherPost (msb, lsb)


Am I doing something wrong?

P.S: The purpose is to degrain without too much banding.

[cretindesalpes]

Am I doing something wrong
This looks correct. Anyway, the result will really depend on your source. If the noise you try to remove has nothing to do with dithering of a high-bitdepth master or is not related to film grain, and if your source already has banding, the MDegrain won't fix anything.

You'll have probably more success with a spatio-temporal denoiser, dfttest for instance. I generally use a combination of both MDegrain for the detailed areas, and dfttest for low gradient and flat areas. See my post #8 on this thread.

[tormento]

Well, a lsb enabled version of fft3dfilter and fft3dgpu (perhaps this is more difficult) would be nice ;)

[james99]

It increases the FPS of progressive Blu ray video from 24fps to 48fps during my test
I do believe it is
Interleave (base, rem) that is doubling the fps in the Dither2Pre function.

can be fixed using
assumefps(24) to change the fps back to 24.

[cretindesalpes]

It increases the FPS of progressive Blu ray video from 24fps to 48fps during my test

You're right. But put the AssumeFps after the DitherPost call. This will be fixed in the next release.

[cretindesalpes]

I released a new version of the Dither tools (1.4) (http://forum.doom9.org/showthread.php?p=1386559#post1386559).

I added SmoothGrad, a smoothing function to get the gradients clean where it low frequency noise or color-banding steps remain after denoising. From this, I built GradFun3, a function intended to smooth sources containing banding and already denoised. This is very similar to GradFun2db.

Here are some examples, comparing several denoising+debanding methods:

http://img835.imageshack.us/img835/6113/comparisonoriginal.th.png (http://img835.imageshack.us/img835/6113/comparisonoriginal.png) http://img818.imageshack.us/img818/1936/comparisonfiltered.th.png (http://img818.imageshack.us/img818/1936/comparisonfiltered.png)
Original / Filtered
http://img38.imageshack.us/img38/206/comparisongradfun2db.th.png (http://img38.imageshack.us/img38/206/comparisongradfun2db.png) http://img411.imageshack.us/img411/6465/comparisongradfun3.th.png (http://img411.imageshack.us/img411/6465/comparisongradfun3.png)
GradFun2db / GradFun3
http://img31.imageshack.us/img31/5200/comparisondither.th.png (http://img31.imageshack.us/img31/5200/comparisondither.png) http://img809.imageshack.us/img809/7963/comparisondithersmoothg.th.png (http://img809.imageshack.us/img809/7963/comparisondithersmoothg.png)
Dither / Dither+SmoothGrad

From the previous pictures, I magnified and amplified several zones in order to show the differences more clearly:
http://img4.imageshack.us/img4/3425/comp1v.th.png (http://img4.imageshack.us/img4/3425/comp1v.png) http://img295.imageshack.us/img295/4491/comp2.th.png (http://img295.imageshack.us/img295/4491/comp2.png) http://img38.imageshack.us/img38/4233/comp3.th.png (http://img38.imageshack.us/img38/4233/comp3.png)
Still the same order:
Original / Filtered
GradFun2db / GradFun3
Dither / Dither+SmoothGrad

One can see that both GradFun function perform poorly near the edges. This is because they are based on massive box filtering and have to stop as soon as the box overlaps an area of a different color. Ideally, these function would use a large bilateral filter, but this would be really overkill and much much slower.

[Floatingshed]

I'm struggling to understand how to use this. Some examples don't use dither2pre others do.
This is what I've been trying, every odd frame is all green noise, and the framerate has doubled:


LoadPlugin("D:\Avisynth scripts\best_dither\dither.dll")
Import("D:\Avisynth scripts\dither.avs")
LoadPlugin("D:\Avisynth scripts\dfttest-1.8mod16a\dfttest.dll")
LoadPlugin("D:\Avisynth scripts\mvtools-2.5.10.1-mod16a\mvtools2.dll")

Dither2pre()

h = Height ()
w = Width ()

super = MSuper(pel=2, sharp=1)
bv3 = MAnalyse(super, isb = true, delta = 3, overlap=8, blksize=16)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=8, blksize=16)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=8, blksize=16)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=8, blksize=16)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=8, blksize=16)
fv3 = MAnalyse(super, isb = false, delta = 3, overlap=8, blksize=16)

MDegrain3(super, bv1, fv1, bv2, fv2, bv3, fv3, thSAD=400, lsb=true)

msb = last.Crop (0, 0, w, h)
lsb = last.Crop (0, h, w, h)
DitherPost (msb, lsb)

[cretindesalpes]

Either use Dither1Pre/Dither2Pre, either the modified MDegrain or dfttest, but not both. The only purpose of DitherXPre is to give the possibility to use a specific denoiser which hasn't been modified to output 16 bit data. For example :

# (Put your source here)
Dither2Pre (flt="TTempSmoothF(lthresh=4*4,cthresh=5*4).frfun7(Lambda=1.1*4)")
DitherPost ()

Without Dither2Pre, your script becomes:

# (Import scripts, plug-ins and input clip here)

h = Height ()
w = Width ()

super = MSuper(pel=2, sharp=1)
bv3 = MAnalyse(super, isb = true, delta = 3, overlap=8, blksize=16)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=8, blksize=16)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=8, blksize=16)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=8, blksize=16)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=8, blksize=16)
fv3 = MAnalyse(super, isb = false, delta = 3, overlap=8, blksize=16)

MDegrain3(super, bv1, fv1, bv2, fv2, bv3, fv3, thSAD=400, lsb=true)

msb = last.Crop (0, 0, w, h)
lsb = last.Crop (0, h, w, h)
DitherPost (msb, lsb)

With the latest version, the three last lines can just be replaced with DitherPost (stacked=true) and you don't need to collect w and h.

[Floatingshed]

Now its making sense and working. Many thanks.

[Dark Shikari]

From the previous pictures, I magnified and amplified several zones in order to show the differences more clearly:
http://img4.imageshack.us/img4/3425/comp1v.th.png (http://img4.imageshack.us/img4/3425/comp1v.png) http://img295.imageshack.us/img295/4491/comp2.th.png (http://img295.imageshack.us/img295/4491/comp2.png) http://img38.imageshack.us/img38/4233/comp3.th.png (http://img38.imageshack.us/img38/4233/comp3.png)
Still the same order:
Original / Filtered
GradFun2db / GradFun3
Dither / Dither+SmoothGrad

One can see that both GradFun function perform poorly near the edges. This is because they are based on massive box filtering and have to stop as soon as the box overlaps an area of a different color. Ideally, these function would use a large bilateral filter, but this would be really overkill and much much slower.Every single one there except gradfun looks utterly terrible. Why? Because they use ordered dither. Use error diffusion, for the sake of your eyes.

[cretindesalpes]

Every single one there except gradfun looks utterly terrible. Why? Because they use ordered dither. Use error diffusion, for the sake of your eyes.

This topic is about noise removal and color banding, not converting a 16M color picture in 256 colors. In actual use, you should never really see the patterns you quoted. I HSV-stretched these excerpts of the previous pictures in GIMP just to show what the plug-in does without having to push the contrast & gamma of your monitor. Also, keep in mind that these patterns have a temporal component.

I used ordered dither instead of a 3D-version of the Floyd-Steinberg algorithm to give a chance to the dither to resist better to the video compression.

Here [dead link, sorry] (http://comparescreenshots.slicx.com/comparison/82872) is a comparison of a standard dither from a high-quality picture (direct bluray rip) compressed @ 6 Mbps and the same movie, re-dithered and compressed with the same settings.

[stpdrgstr]

I get "there is no function named "Max"" when trying to use Gradfun3().

[Didée]

min() and max() functions were added in Avisynth 2.5.8.

Seems you're still using 2.5.6 or 2.5.7. ;)

[stpdrgstr]

So that's why. I was using the 2.5.7 mt version I could find at the avisynth wiki because one of the 2.5.8 versions gave me weird errors with assumefps. What 2.5.8 mt do you recommend?

EDIT: I just noticed there's only one 2.5.8 supporting MT, but it's 64 bits. Isn't there a 32 bits version?

[Didée]

Sure there is (http://forum.doom9.org/showthread.php?t=148117).

[stpdrgstr]

That's weird, I can't see that and other threads listed.

I went with 2.6 at the end, but thanks anyways.

[cretindesalpes]

Dither (http://forum.doom9.org/showthread.php?p=1386559#post1386559) updated to version 1.5. Added several error diffusion algorithms (however ordered dithering still looks better after video compression) and fixed the randomness of the internal noise generator.

[sneaker_ger]

I wanted to try dither out but I always get the error "dfftest: unable to load libfftw3f-3.dll". I'm using AviSynth 2.5.8 (MT) on Windows 7 64bit with a Core i7. I have copied libfftw3f-3.dll to my plugin directory. Any idea?

[Usedocne]

libfftw3f-3.dll goes in the Windows\SysWOW64 folder.

[sneaker_ger]

Thank you, works now.

[Fullmetal Encoder]

I'm having a bit of difficulty getting this to work properly in the absence of a denoiser. I prefer to use TNLMeans to denoise and I have a clip which has been denoised to my satisfaction. From the sample images posted in this thread it appears that the debanding methods that DitherPost employs are superior to GradFun2DBMod. Plus, I would like to see how I can use the mask tool to preserve detail beyond the level I have been able to accomplish with GradFun2DBMod. How do I go about using/testing DitherPost on a previously denoised clip? While I would like to have 16-bit depth output with TNLMeans I'm afraid that the sun would go supernova before Dither2Pre finished processsing the video. Here is what I have attempted:

SetMTMode(2,8)
SetMemoryMax(1400)
AVISource("clip.avi")
dfttest(sigma=0,sigma2=0,lsb=true)
msb = last.Crop (0, 0, 720, 480)
lsb = last.Crop (0, 480, 720, 480)
DitherPost (msb, mode=0, lsb, prot=false)

Since I didn't want DFTTest to be doing any extra denoising my idea was to deactivate any of it's processing and still have the filter output an msb and lsb, which, as I understand it, are absolutely necessary for DitherPost. I also tried setting y,u,v to "false" in dfttest() and using various other modes for DitherPost but in all cases this script either produces nothing (displays the source clip without any apparent processing) or slows the script down (dfttest appears to slow down navigation in VirtualDub but no dithering is taking place). There really isn't any point in doing extra processing with a very slow DFTTest just to get rid of banding. This is somewhat of a hack, however, as invoking DFTTest at all leaves me not knowing whether it may be slowing down processing even if I use sigma=0 so ideally I'd like to be able to use DitherPost by itself. While I have been able to get DitherPost to add dithering by omitting DFTTest and msb and lsb the results are very ugly with very visible spots and band.

BTW, I was the one who created a thread earlier asking about the possibility of sending only portions of a frame through to a filter based on a luma threshold or other criteria. What I was thinking of using that to accomplish appears to be almost exactly what you have done with DitherPost and BuildDitherMask, which is to allow high quality dithering while protecting more detailed areas of the image.

Thanks in advance for any help you can provide.