Adjusting x264 settings to prevent banding

[x265]

Which all options should i tweak in x264 to prevent banding from appearing?

Sample > http://www.mediafire.com/?y9vqkd98gku7hrs

mpeg2source("C:\Users\Abhijith Nair\Desktop\Anime\Vol1.d2v")
AnimeIVTC(mode=1, aa=0, ifade=true)
trim(8288, 8566)
Crop(2, 4, -2, -0)
dehalo_alpha_mt(ss=3.0)
SMDegrain(lsb=true, refinemotion=true, pel=4, subpixel=3, tr=6, thSAD=300, prefilter=2, contrasharp=true)
f3kdb(range=18, dither_algo=3, input_depth=8, output_depth=10, output_mode=2)

[TheSkiller]

If you use ordered dithering you greatly increase the chances of the dither surviving the lossy encoding.
For ordered dithering use dither_algo=2 in f3kdb.

If you experience banding after encoding you could try tweaking --psy-rd and/or --aq-strength in x264.

[LoRd_MuldeR]

If you don't do yet, try the 10-Bit version of x264.

Other than that, using "--preset noise" with a very low CRF value might be your best bet to retain dither noise...

[tormento]

SMDegrain already has a dither flag inside, why should you use another dither?

[Reel.Deel]

Like tormento has already pointed out, let SMDegrain output 16bits and let f3kdb do the final dithering.

SMDegrain(lsb=true, lsb_out=true, refinemotion=true, pel=4, subpixel=3, tr=6, thSAD=300, prefilter=2, contrasharp=true)
f3kdb(range=18, dither_algo=3, input_depth=16, output_depth=10, output_mode=2)

[x265]

If you don't do yet, try the 10-Bit version of x264.

Other than that, using "--preset noise" with a very low CRF value might be your best bet to retain dither noise...

I'm using x264tmod(10-bit) -ffmpeg

[x265]

If you use ordered dithering you greatly increase the chances of the dither surviving the lossy encoding.
For ordered dithering use dither_algo=2 in f3kdb.

If you experience banding after encoding you could try tweaking --psy-rd and/or --aq-strength in x264.


Do the two dithering methods look the same at high bitrates(1800-2000kbps)?. Is it safe to increase aq-strength from 0.5 to 0.7?.

[Overdrive80]

Like Lord_Mulder says, you must use very level of crf but you will get a big file. Maybe you can determinate the zones (on x264) that need more bitrate (dark scenes, much movement). Example of --zones on x264:

--zones 0,1000,crf=16.5/1001,2000,crf=18/2001,3000,crf=16

And use:

--psy-rd 1.8:0.0 (Second parameter is still experimental)

[x265]

Other than that, using "--preset noise" with a very low CRF value might be your best bet to retain dither noise...

Could you specify a range(crf value) at which x264 will retain most of the noise?.

[x265]

Why does the video flicker(after IVTC) during playback in avspmod? any way i can fix this?

[LoRd_MuldeR]

Could you specify a range(crf value) at which x264 will retain most of the noise?.

The smaller the better, of course. But going too low will also make file size explode. You'll need to find the best trade-off for your specific content and eyes ;)

[x265]

SMDegrain is destroying the details in the dark scene's
Screenshot:
http://screenshotcomparison.com/comparison/19807

[poisondeathray]

SMDegrain is destroying the details in the dark scene's
Screenshot:
http://screenshotcomparison.com/comparison/19807


It's a combination of overdenoising, and a side effect of dithering .

You can try reducing the denoising and different dithering algorithms , or dither and filter more selectively (e.g. using trim() or masks)

[tormento]

Like tormento has already pointed out, let SMDegrain output 16bits and let f3kdb do the final dithering.

What I really meant was:

mode
[int: -1, "0", 1, 2, 3, 4, 5, 6, 7, 8]

If you process in 32bit and lsb_out (16bit output) is not enabled, a dithering method must be chosen for the 32bit->8bit conversion. (Interlaced content is locked to mode=6)

-1: no dither, round to the closest value
0: 8-bit Bayer ordered dither + noise. (Default)
1: 1-bit dither
2: 2-bit dither, light
3: 2-bit dither, medium
4: 2-bit dither, strong
5: 2-bit dither, stronger
6: Serpentine Floyd-Steinberg error diffusion + noise. Well-balanced algorithm.
7: Stucki error diffusion + noise. Looks sharp and preserve light edges and details well.
8: Atkinson error diffusion + noise. Generates distinct patterns but keeps clean the flat areas.

For this to be used you will need the Dither package with its version of MVTools2.

Why should we need debanding functions AFTER SMDegrain?

[mandarinka]

SMDegrain is destroying the details in the dark scene's
Screenshot:
http://screenshotcomparison.com/comparison/19807

Yes, that is an issue with the whole underlying MVTools2 denoising. It hits darker parts much more strongly than brighter parts.

In a way, it is a general issue with smoothing too :)

Are you sure it is a good idea to run a degraining script on Death Note? It is a computer-aniamted show, there is no analogue noise in it that you would want to remove this way...

[x265]

are you saying that i should just use a debanding filter(f3kdb)?

[x265]

Yes, that is an issue with the whole underlying MVTools2 denoising. It hits darker parts much more strongly than brighter parts.

will there be a fix for this in the future releases of mvtool2?

[Overdrive80]

You can try this:

###################################
# FILTRADO #
###################################

Setmtmode(2)

original=last

aa=spline64Resize(width(original)*2,height(original)*2).SangNom(aa=29).spline64Resize(width(original),height(original)).mergechroma(original)

prefiltrado= dfttest(sigma=x,tbsize=1,tmode=0,lsb=true).ditherpost(mode=2)


super=Msuper(prefiltrado)

backward_vec3 = MAnalyse(super, isb = true, delta = 3, overlap=4,search=4)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4,search=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4,search=4)

forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4,search=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4,search=4)
forward_vec3 = MAnalyse(super, isb = false, delta = 3, overlap=4,search=4)

prefiltrado.MDegrain3(super,backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,
\thSAD=y,plane=4,lsb=true).ditherpost(mode=2)

contrasharpening(last,aa)

toon(0.10)

gradfun3()

On dfftest, assign sigma values for reducing spatial noise. On MDegrain3, thSAD for temporal noise. For anime, work fine. You can begin with sigma=4, thsad=300

[mandarinka]

Those are strong thSAD/sigma values! Also, I don't like you line darkening :(

What he wants is to limit denoising in darker parts to smaller strength. Could be achieved this way, although this is a bit of a hacky approach (works though/has been used in the wild, heh):

source = last

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)

degrain = source.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=140)
degrainless = source.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=30)

mask = source.mt_lut("x 3.10 ^ 38000 /").blur(0.3)
mt_merge(degrainless,degrain,mask,luma=true)


Alternatively use mt_merge(source,degrain,mask,luma=true) at end to better protect the dark parts.

[x265]

The SMDegrain readme says :

SMDegrain() (its core: mdegrain in fact) is likely to produce blending artifacts or oversmooth in dark areas specially on cartoons, to tackle this see RefineMotion
or expand the luma range in the darks. For this there are many options, you can use your preferred gamma enhacer and input it as the prefilter,
or use the examples below for linear brightening (masktools2) and input it as a prefilter as well. Or at last and the recommended solution,
to use the built-in Str (strength) and Amp (amplitude) parameters for a curve type dark enhacement.

But i don't know what values to put in for Str and Amp.

[sneaker_ger]

Why should we need debanding functions AFTER SMDegrain?

Though dithering is not the same as debanding, so if the source has banding, you still need it.

[sneaker_ger]

Via PM:
Will dither_algo=3 look better at high bitrates?, like 2000-2200kbps for SD?

I still consider that rather low, so I'm inclined to say: no. Maybe at very hight bit rates like 10 Mbit/s for SD, but it also depends on the source, of course.

[x265]

Will "tune --grain" have any negative effects on animation?

[x265]

Encoded sample http://www.mediafire.com/?jl5q9nb53ux75we

Settings used : --tune animation --crf 14.0 --preset veryslow --no-fast-pskip --aq-strength 0.65 --qcomp 0.72

[x265]

Left : Filtered | Right : Filtered+Encoded

AVS Script
DGSource("C:\Users\Abhijith Nair\Desktop\Anime\Death Note\MainMovie\Episode.dgi")
AnimeIVTC(mode=1, aa=0, ifade=true)
#ClipOne=DGSource("C:\Users\Abhijith Nair\Desktop\Anime\Death Note\1\OP.dgi").AnimeIVTC(mode=1, aa=0, ifade=true).trim(0, 1917)
#ClipTwo=trim(1918,30688)
#ClipThree=DGSource("C:\Users\Abhijith Nair\Desktop\Anime\Death Note\2\ED.dgi").AnimeIVTC(mode=1, aa=0, ifade=true).trim(0, 1678)
#ClipFour=trim(32368,32727)
#AlignedSplice(ClipOne, ClipTwo, ClipThree, ClipFour)
trim(2014, 2151)+trim(8175, 8412)
Crop(2, 4, -2, -0)
Dehalo_Alpha_mt(ss=3.0)
f3kdb(dither_algo=2, range=20, input_depth=8, grainC=25, grainY=25, output_depth=10, output_mode=2)


https://imageshack.us/a/img248/773/banding000804.png

The banding is not present in the encoded on the right

[feisty2]

turn off "prefilter" parameter and you will make the detail loss much lesser

[x265]

I did not use a denoiser(SMDegrain() ) due to the oversmoothing in dark areas.

[osgZach]

Haha... I recognize that show!

I gave up on messing with it a long time ago.

Post #6 - my Color Banding and Noise Removal thread (http://forum.doom9.org/showthread.php?p=1387156#post1387156)

[turbojet]

With natural content negative beta deblocking helps avoid banding with 8bit x264, for example x264 --deblock 0:-1 maybe it helps for 10 bit anime.