Color banding and noise removal

[osgZach]

Is it possible to deband a source and also denoise it?

Should I denoise first and then deband?

Using GradFun2DB for instance, it won't get rid of it but it will take the banding down a notch, however if I were to there apply something like LSFMod, or in general some other noise reducers / it will undo the bebanding( as it seems it just adds grain to break it up), and then the banding will come back with even more clarity and well defined borders.

So it ends up looking like an ugly gradient, for lack of a better term, with those wavey line borders and all that fun stuff.

What also puzzles me, is that even though I am doing no color mode conversions, etc to even create this banding (its present on the DVD source) when I watch either the DVD or a straight unfiltered encode of the same episode, you can almost not even tell it is there on the TV screen.

Why is that?

Any suggestions on some strong, but safe settings for GradFun2DB, or other x64 compiled filters for this (degrainmedian, etc?) would be preferred. I'll provide a source section once a test encode I am running is done, if its needed. But this was more of a general question on effective settings for some filters versus a plea for someone to do the work for me.

[OvejaNegra]

After ALOT of battle agains grain VS Banding i can tell you this for sure:
You eat the grain, you ll get banding.
Some sources are encoded without dithering and have banding.
Some sources have no grain but are encoded with dithering to avoid banding (dithering in the end is noise like grain for you)

My actual choices (after MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANY teest with animes and grainy sources)

Opt A: Do a strong denoising using FFT3d or moderate denoising using vaguedenoiser or anything like that -> good compression (not too high bit rate needed, not too big file) -> banding -> enable deband filter on FFDSHOW duting playback and it works excellent.

Opt B: Calm the grain using some temporal grain compressor like TTEMPSMOTH (my favorite) or this:
http://forum.doom9.org/showthread.php?t=139766
or this:
http://forum.doom9.org/showthread.ph...rain+optimizer
that will give you a more stable grain
then -> encode using high bit rate (1800 kbps or more) using grain retention settings on X264 or qpel + MPEG matrix on XVID -> Bigger file -> no banding -> looks very nice to me (for anime i'm ok sometimes with the anime preset on x264, i don need the grain preset)


IF your source have banding -> subtle denoiser (there is ALWAYS some noise on DVDs) -> encode normally -> enable deband on ffdshow during playback.

Or

Encode using gradfun or THIS: GradFun2DBmod.v1.3.avsi (google) Your source will look grainy again but it wont have banding. Of course you need high bit rate for this.


But i can tell you this for sure >
DVD sources have grain or dithering (i cant speak for HD yet, ask other person) so Degrain or denoise + no banding is impossible.

Yes on the TV (a common TV) you can see the banding even if is an encode or the original DVD with banding.
When you are on your monitor maybe bacause a bigger luma range you can see the banding. If you have a flat display it will look worst.

It's up to you, keep the grain or remove it and leave the banding. If your DVD has banding, well use some of the solutions i tested.

Good luck


EDIT:
Read this:
http://forum.doom9.org/showthread.ph...rain+optimizer

[cretindesalpes]

(Original reply replaced by this Dither presentation.)


Download

• >>>> dither-1.28.1.zip <<<<
  Main scripts and plug-ins from the Dither package, including avstp.dll.

• >>>> dfttest 1.9.4 <<<<
  Mod16 version of dfttest.

• >>>> MVTools 2.6.1.0 <<<<
  Mod16 version of MVTools 2

Masktools 2 a48 or above is required. Previous Masktools versions will silently fail and cause the greenish screen of death. Please update your Masktools before installing the Dither script.

All these plug-ins include the -Vit- threading fixes.


What is it?

Dither is a package including several scripts and avisynth plug-ins. Some of them are modification of existing ones (dfttest and MVTools 2). I primarily wrote the Dither tools to fight color banding during denoising in an unusual way. But Dither can perform various operations :

• Recover the high bitdepth data possibly contained in a noisy clip. This can be seen as an "inverse-dithering" operation. In computer graphics, people usually use a gaussian filter to turn halftones into shades of grey. We'll use denoisers here.

• Dither a high bitdepth picture into a standard YV12. The picture may have been obtained with the inverse-dithering operation, the combination could be called "transdithering".

• Smooth fine gradients to remove colorbanding, resulting in a high bitdepth picture which can be dithered later.

• Performing basic operations (masking, curves...) on high bitdepth pictures, as they cannot be manipulated safely with conventional avisynth filters.

Why should I use it?

Because denoising and lossy compression can easily make colorbanding appear. By transdithering, it is possible to recover original gradients better than using a filter + GradFun2DB(mod). Moreover, classic error-diffusion dithering algorithms require a high bitrate during encoding, otherwise color banding appears again. Dither offers an ordered dithering (Bayer matrix), which is more resilient to data compression.


How can I use it?

The simplest way to transdither a relatively clean source, using the modified dfttest:

dfttest (sigma=2, lsb=true)
DitherPost ()

You can add a SmoothGrad() call between both statements to remove colorbanding present in the source. And because dfttest is a spatio-temporal filter, it would be better to also use motion compensation to increase filter accuracy and prevent ghosting.

For an already denoised source, use

GradFun3 ()

GradFun3 is intended to replace GradFun2DB(mod). Actually it works on the same principle and does almost the same things, but it uses ordered dithering by default, which is better for compressibility.

[osgZach]

Interesting solution you have there.

As for this particular source, I decided to just let it be as it was too much confusion on my part. Its worth noting I don't watch much on my PC, I encode for playback on my WDTV Live on a large screen (sadly I don't know its size haha) HDTV. Compressibility isn't much of a concern as x264 under CRF gives close to DVD quality results in the upper range of sub-1000kbps (on the TV). But I'll probably come back to this for future encodes I know will be pretty noisy as they were poor transfers to DVD.

[Terka]

could you please post same samples?

[osgZach]

Clip #1 This is not an example clip, I was wondering however if this is an example of Halo artifacts? (the edges around the students portable game thing)

Clip #2 - This is the color banding (I hope) clip. It's 26MB and spans several scenes which each have varying degrees of banding. I have never been able to successfully filter them, or reduce noise without greatly enhancing the visibility of the bands.

These come from Death Note, Episode 1, R1 DVD

[osgZach]

The only thing I've had the patients to sit around and check was

Dither1Pre ().Dither1Post ()

Which did make a minor difference but I think mostly did some denoising.. Under x32 using some of the other examples (which I honestly copied and pasted, I can't even begin to understand them or how they work) I got some pretty nice results that pretty much made it look about as good as watching it on my TV or converting the file to RGB. Although banding was still there (big reduction though)..

It would be really nice if someone could port the modified DFTTEST linked to above, to x64 so I can try this out under JoshyD's MT_x64 build. Maybe some other people would find it useful too.

I recently purchased Last Exile when I saw it at Best Buy.. A really nice show, and very good quality for the most part. However I was disheartened to find the same horrible color banding issues when I popped it into the drive and took a quick peek at it.. Very dismayed

Seems to be a rather ugly trend in the industry to not push these shows out the door properly... They have to know this is a huge problem

[cretindesalpes]

I think the banding you have comes from the final YV12 to RGB conversion. I tried to process your second clip with the following script :

FFmpegSource2 ("clip2.demuxed.m2v")
TFM ()
TDecimate (mode=1)

zzz_denoise ()
DitherPost (mode=2)

Function zzz_denoise (clip src, float "sigma", int "thr", bool "mask", int "sad")
{
	sigma = Default (sigma,    16)
	thr   = Default (thr,       5)
	mask  = Default (mask,  False)
	sad   = Default (sad,     200)

	w = src.Width ()
	h = src.Height ()

	# Motion analysis
	super   = MSuper (src)
	super_a = MSuper (src.TTempSmooth ().RemoveGrain (12))

	fwd_vect_3 = super_a.MAnalyse (isb=false, delta=3, overlap=4)
	fwd_vect_2 = super_a.MAnalyse (isb=false, delta=2, overlap=4)
	fwd_vect_1 = super_a.MAnalyse (isb=false, delta=1, overlap=4)
	bck_vect_1 = super_a.MAnalyse (isb=true,  delta=1, overlap=4)
	bck_vect_2 = super_a.MAnalyse (isb=true,  delta=2, overlap=4)
	bck_vect_3 = super_a.MAnalyse (isb=true,  delta=3, overlap=4)

	fwd_comp_2 = src.MCompensate (super, fwd_vect_2, thSAD=sad)
	fwd_comp_1 = src.MCompensate (super, fwd_vect_1, thSAD=sad)
	bck_comp_1 = src.MCompensate (super, bck_vect_1, thSAD=sad)
	bck_comp_2 = src.MCompensate (super, bck_vect_2, thSAD=sad)

	# Spatio-temporal denoising using modified dfttest
	c_dft = Interleave (fwd_comp_2, fwd_comp_1, src, bck_comp_1, bck_comp_2)
	c_dft = c_dft.dfttest (sigma=sigma, lsb=true)	# Double height
	c_dft = c_dft.SelectEvery (5, 2)

	# Temporal-only denoising using modified MDegrain
	c_deg = src.MDegrain3 (super, bck_vect_1, fwd_vect_1, bck_vect_2, fwd_vect_2, bck_vect_3, fwd_vect_3, thSAD=sad, lsb=true)	# Double height

	# Spatio-temporal denoising smoothes too much the details,
	# therefore we use pure temporal denoising on edges or detailed areas.
	edge_src = c_deg.Crop (0, 0, w, h)
	edge_mask = edge_src.mt_edge (mode="prewitt", thY1=thr, thY2=thr)
	edge_mask = edge_mask.mt_expand ()
	edge_mask = StackVertical (edge_mask, edge_mask)	# Double height
	c_hyb = mt_merge (c_dft, c_deg, edge_mask, luma=true, y=3, u=3, v=3)

	return (mask ? edge_mask.GreyScale () : c_hyb)
}

(This script also requires a modified version of MDegrain3. I added a link to my previous post)

I've got pretty good results regarding denoising and debanding. However, depending on which software I play the processed clip, bands may appear or not, and obviously come from the player's YV12->RGB converter. But I don't know if it's only a matter of codec or if the video card's driver is involved too.

An example :

Original, noisy clip :


Denoised, with bands :

On this picture, one can see clearly the YV12->RGB conversion problem. Every six bands, there is a bigger jump caused by a step of 1 unit in luma values converted to a step of 2 units in the RGB colorspace instead of 1.

Denoised and dithered :


For the record, GradFun2DBMod :


To show the effect, a magnified part of the picture above (bottom left), "HSV stretched" (with GIMP) :

Original / Denoised
GradFun2DBMod / Dithered

[osgZach]

There shouldn't be ANY YV12 - > RGB conversion going on. I work on my clips in Vdub, so unless its auto color mode detection is borked up and doesn't work, what I am getting -should- be YV12 ? Or is it gonna be RGB at viewing stage regardless, just by nature of PC displays?

My encoding path is off the DVD to a HuffYV12 (ffdshow) file. Not sure if that would have any impact on it though. I will say doing a color conversion to RGB seems to clean it up rather nicely all by itself. So I'm not sure why even at the viewing stage, an RGB conversion would look so horrible.

That being said. I appreciate your efforts. I'm not sure if I have tried Gradfun2db MOD, but I bet I at least played around with the original.. Your GF2DBMod image looks very nice for certain. I may have to install Set's 2.5.8 MT x32 version of Avisynth I guess so I can at least experiment at reasonable speeds.

Would anything during the DGindex process affect this too? i.e there is PC / TV modes for clamping some color stuff IIRC. I don't recall which I used but probably default to PC..

[OvejaNegra]

osgZach:
I havent tested your samples, but there are animes with banding, i dont know why but it happens.
GradFun2DBMod should put grain on those banded areas, yes you need a higgher bit rate or x264 will eat the grain again.

By the way (maybe you dont know this):
VDUB will convert yoyr source to RGB if you use full processing mode and the codec will re-convert to YV12 during encoding (in case you are encoding with xvid / divx / etc on vdub).

Go to the options menu / preferences -> output color depth: is on fastest 16 bit? because if it is, THATS the cause of your banding.
you should put 24 bit or match display depth

Maybe you know this but maybe not, but you mention you are using VDUB and maybe you should check this.

And, if you eat your grain with FFTxxx or vaguedenoiser or any good spatial filter, well you always will have banding.
The mpeg2 encoders have dithering for make the color conversions (like CCE, i dont know the modern versions, but the old SP has it).
Try a temporal filter to calm the grain on the static surfaces like ttempsmotth (the only one who gave me a good looking Kanokon).
I can send you samples if you like it, there is one scene on episode one that you can use to test your anti-banding strategy and find the best filter to your needs

Good luck
Please check VDUB settings

[osgZach]

My Vdub should be set to "use output settings". Actually not quite sure what that means, but I definitely take it off the 16-bit junk every time I setup a fresh install, etc.. As for my processing mode, I never use full. I use fast recompress, so it pass right through without being touched, except by Avisynth. So in my mind its VOB - > FFDShow (in this case FFDshow's HuffYV12), which I then send through x264 via an AVS script after I am done working on a file.

I do have "Enable dithering for 16-bit displays" turned off as well.. Not sure if that would impact anything. But at any rate.. I probably need a break from all this stuff..

[OvejaNegra]

Output color depth:
"Selects the default color precision for video display -- 24-bit looks better, but 16-bit is generally faster. This does not affect file output"
Enable 16-bit dithering:
"If enabled, 24-bit images are dithered when displaying in 16-bit to reduce banding, at the cost of a little speed. This does not affect file output"


If you set that to "Fastest 16 bit" the videos looks like if your display to 16 bits color. They band a lot. If you enable dithering, well you'll see grain/noise/dither that is not part of the original source.

I always use "Match display depth"

you could install xvid and disable raw video support from the VFW and the DSHOW filter, so xvid will handle YV12 content, to see if that helps. (The divx pack also have a YV12 decoder).

[cretindesalpes]

dither.avsi updated to v1.1, see the original post.

[Floatingshed]

I get "the named argument clip was passed more than once to blankclip". Line 755, 413, 250.
Please help....

[foxyshadis]

Replace your version of the script with the one in the zip file. Yours might be modified or corrupted.

[dlnm]

Is it possible to use other denoisers instead of MDegrain or dfttest?

[cretindesalpes]

Originally Posted by dlnm

Is it possible to use other denoisers instead of MDegrain or dfttest?

Yes. Use Dither1Pre() or Dither2Pre() to generate a 16-bit denoised clip (actually it will have only 9 or 10 significant bits).

Usage :

Dither2Pre ("YourFavoriteDenoisingFilter (param=1234)")
DitherPost ()

If your filter has signal amplitude dependent parameters (thresholds, etc.), multiply them by 2 or 4 for Dither1Pre and ~2Pre. The drawback of this function is that the processing time will also be multiplied by 2 or 4.

[andrew_seul]

Originally Posted by cretindesalpes

dither.avsi updated to v1.1, see the original post.

There's any chances that you can port those plugins you modifed to 64 bit mode (including NoiseGenerator also)? I need a little speed on my encodings thats why Im asking

[Jawed]

Just chain stuff together

e.g. I use this to kill blocking artefacts:

function BlockKiller(clip source)
{
    y=source.greyscale()
    y=y.ChannelBlockKiller() 
    u=source.utoy()
    u=u.ChannelBlockKiller() 
    v=source.vtoy()
    v=v.ChannelBlockKiller() 
      
    return ytouv(u, v, y)  
}

function ChannelBlockKiller(clip source)
{
    source
"""   spline36resize(source.width*16,source.height*16)
    gradfunkmirror() 
    gradfunkmirror() 
    gradfunkmirror() 
    spline36resize(source.width*8,source.height*8)
    gradfunkmirror() 
    gradfunkmirror() 
    gradfunkmirror() """
    spline36resize(source.width*4,source.height*4)
    gradfunkmirror() 
    gradfunkmirror() 
    gradfunkmirror() 
    spline36resize(source.width*2,source.height*2)
    gradfunkmirror() 
    gradfunkmirror() 
    gradfunkmirror() 
    spline36resize(source.width,source.height)
    gradfunkmirror() 
    gradfunkmirror() 
    gradfunkmirror() 

    return last
}

notice, in the function ChannelBlockKiller, that the *16 and *8 sections are commented out with the """. As posted, that code only uses *4 and *2 sizes, before returning the picture to its original size.

Basically you can adjust the maximum size, e.g. make it larger for smaller frames by un-commenting the *8 and *16 sections. The basic concept here is that any resizing algorithm creates a gradient on all edges (banding and non-banding). The gradient will be "smooth" but "regular". So then apply gradfunkmirror() to dither that gradient. Then resize down. Resizing down averages the dither.

The quantity of steps and the count of gradfunkmirror() calls per step adjust the result. You adjust for blockiness versus softness. (I'm still experimenting with this function - which is why there are no funky parameters, as I haven't decided which parameters are important and how to graduate the effect.)

In general you can simply use 2 or more instances of gradfunkmirror(), one after the other, to control most banding.

Also, I always use gradfunkmirror() on playback (the DeBand filter in FFDShow).

Jawed

[Gavino]

Originally Posted by Jawed

notice, in the function ChannelBlockKiller, that the *16 and *8 sections are commented out with the """.

An innovative use of quotes.
Why not use a real comment /* ..... */ which can span multiple lines.