X264 artefact problems

[Zelos]

Hi to all,

i have a problem with x264 encoding on some animes.
in some dark area , i have always big differences level between colors, during encoding (the source doesn't have this problem).
I have tested many filters (chroma smoother, mipsmooth,deblock,etc....), without success.
I'm not this kind of professional of scripts, but if someone could help me to find a solution , that will be great :D

i post a pic to see what i'm speaking about (the source is 1280*720)
http://img132.imageshack.us/img132/2924/testyk9.th.jpg (http://img132.imageshack.us/my.php?image=testyk9.jpg)


thanks in advance.

[Dark Shikari]

This is called banding. It isn't a product of x264, but rather a product of the limited precision of luminance in your colorspace.

The only way to avoid the ugly "artifacting" is to add dither, such as with gradfun2db.

x264 can retain dither, but only with the appropriate settings and relatively high bitrates. For example, you probably want AQ, and psy-RD/psy-trellis/no-dct-decimate can help retain dither as well.

[Zelos]

thanks for the answer Dark Shikari.
btw, i tested with AQ , but i used to read not to touch about trellis , i will test in 2 mins and put the results.

[Zelos]

so i have tested with aq and psy strength , that helped just a little , but not enough.
i will test with gradfun2db.

[Zelos]

ok , i used gradfun2b, which work perfectly in my case !

thanks a lot Dark Shikari, now i know what is banding :D

[Sagekilla]

Trellis is not "bad." If you enable psy-trellis (--psy-rd A:B <-- B is the psy-trellis param) with --trellis 2 you can benefit from it pretty well.

[Zelos]

Sagekilla,

i tried --psy-rd2:2 --trellis 2
the result was better on the banding , but was still there enough to see them.
if i mix this parameters with gradfun2db(6) that works pretty well.

btw , is it possible to use the debanding just for some frames ? (i mean not for the entire anime ?)

[kemuri-_9]

psy-rd can be applied via zones in x264
and you can use ApplyRange() in avisynth to apply gradfun2b on frame ranges.

[Zelos]

thanks kemuri, i will try it

[Adub]

Those Psy-RD settings are actually pretty high. I usually stick under the 1.0:0 defaults. Like, 0.7:0 for psy-rd with aq set to 0.7. This combined with gradfun2db works wonders.

[Zelos]

thanks merlin , i tried it and the result works .
I have some shadows around some persons sometimes after gradfun2db, is there anyh settings for it ?

[Dark Shikari]

thanks merlin , i tried it and the result works .
I have some shadows around some persons sometimes after gradfun2db, is there anyh settings for it ?Don't use such a high strength value with gradfun2db. Maybe 1.5-2.5, not 6.

[Zelos]

yeah !

thanks Dark , that's perfect !

i thank everyone for the help.

Since i'm a starting with avisynth and encoding , i'm lost with scripts and filters.

[Comatose]

Tip: Start at the default settings and tweak from there, don't just make up a number >_>