How to deal with this severe color banding?

[asarian]

I wonder if some of the more experienced folks here could give me some settings advice on how to get rid of some severe color banding on THIS (http://www.mediafire.com/file/tyvjzfm2fzz/blade2test.mkv) small 19MB Blade Runner clip? I use 'MCTemporalDenoise(settings="medium")' on it as only filter. And encode as follows:

cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x133 / me=tesa / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / constrained_intra=0 / bframes=8 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=14.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / vbv_maxrate=60000 / vbv_bufsize=70000 / crf_max=0.0 / ip_ratio=1.40 / aq=2:1.00 / nal_hrd=non

The quality of the resultant file, overall, though very small, is surprisingly good; but in certain scenes things have really GONE BAD (http://www.mediafire.com/file/znynoymnjym/gonebad.mkv).

I've been trying to get rid of the severe color banding; but, alas, to no avail. :( I tried adding things like '--deadzone-inter 0 --deadzone-intra' and/or '--no-dct-decimate', etc. But the output is still pretty bad. I can achieve a somewhat bearable look by adding '--qpmin 0' and lowering CRF all the way to 5. But that makes the output file nearly twice the size of the original. Needless to say, that's not ideal.

So, I was hoping someone can give me some better advice than the stuff I could come up with myself (which was, pretty much, not much more than rasising bitrate insanely).

Thanks.

[Keiyakusha]

Don't know what you expecting to see but it looks great. If you want it to be visually identical to the source set lower crf so resulting bitrate will be 2-3 times higher.
However current crf behavior also seems strange to me. maybe It was rebalanced and I didn't noticed... Оо
Some time ago with higher crf values bitrates was higher and so result was closer to the source. There was no need to touch qpmin. Now we need to set qpmin and crf lower...

[asarian]

Don't know what you expecting to see but it looks great.
Well, it looks okay when seen on my PC with, say, GOM Player. :) On my Sony 40" Bravia TV, however, the banding (to the left of Rachel's face) becomes painfully obvious. As in being unwatchable.

If you want it to be visually identical to the source set lower crf so resulting bitrate will be 2-3 times higher.
In this case, 'closer the original' has no real meaning, as I MCTemporalDenoise the source first before feeding it to x264.

To be fair, the original, denoised input actually already has some color banding (viewed in GOM Player with just the output of the AVS script, sans x264); x264 just aggrevates the issue. So the object is to tell x264 to not do that.

[Keiyakusha]

asarian
Well, I mean closer to the input.
As for your TV, this is expected. Sometimes you even need to add dither with strong noise/grain to make it look good on TV :P
EDIT: and yes, in this case any kind or denoising is not your friend.

[asarian]

However current crf behavior also seems strange to me. maybe It was rebalanced and I didn't noticed... ??
Some time ago with higher crf values bitrates was higher and so result was closer to the source. There was no need to touch qpmin. Now we need to set qpmin and crf lower...
I've pretty much given up on understanding the intricacies on the new CRF meanings. :) For this test, for example, I lowered CRF to as low as 1 (from 14,9,7,5,3); and in the various stages the output would sometimes be smaller even on a lower CRF than its predecessor; go figure. :) Indeed, only also lowering qcomp seems to really make a significant difference.

For the current encode, I'm looking at an average bitrate of ~43 Mbp/s now, with a peak at ~80 Mbp/s; all of which I can still stream easily, but is, objectively, too high a bitrate for this source. I'm thinking it would be uebercool if you could set like 'zones' in x264: areas (of known, problematic frame-ranges or some such) for which you want special override bitrate settings. Who knows? Maybe in a distant future. :)

[mp3dom]

Well, it looks okay when seen on my PC with, say, GOM Player. :) On my Sony 40" Bravia TV, however, the banding (to the left of Rachel's face) becomes painfully obvious. As in being unwatchable.

Most probably, the Bravia TV is more well calibrated than your PC monitor. To be precise, the banding is on the left (as you said) but even on the right of Rachel's face. It's less visible because it has some subtle grain but it's present. If you don't want to raise bitrate (or lower crf) you can:
- try to increase the AQ values (1.2~1.5)
- Apply gradfun2db prior to encoding as this add a bit of subtle dithering
- Add a bit of noise with Blockbuster. You can add noise only on uniform areas avoiding to add noise to the whole picture.

[Lyris]

Most probably, the Bravia TV is more well calibrated than your PC monitor.
I highly doubt that. PC monitors don't do half of the goofy bit-shifting nonsense that TVs are set up to do usually. It's almost certainly the TV's fault - what model of BRAVIA is it? I have experience with some of those.

What are the TV settings? Make sure any "Dynamic Contrast" modes are shut off, for starters.

[kemuri-_9]

I'm thinking it would be uebercool if you could set like 'zones' in x264: areas (of known, problematic frame-ranges or some such) for which you want special override bitrate settings. Who knows? Maybe in a distant future. :)

welcome to 5 years ago:
see the documentation on --zones (http://mewiki.project357.com/wiki/X264_Settings#zones)

[asarian]

Most probably, the Bravia TV is more well calibrated than your PC monitor. To be precise, the banding is on the left (as you said) but even on the right of Rachel's face. It's less visible because it has some subtle grain but it's present. If you don't want to raise bitrate (or lower crf) you can:
- try to increase the AQ values (1.2~1.5)
- Apply gradfun2db prior to encoding as this add a bit of subtle dithering
- Add a bit of noise with Blockbuster. You can add noise only on uniform areas avoiding to add noise to the whole picture.
As usual, you came up with some very useful tips! :) Thank you very much!

Man, there's a lot to learn about this stuff. It's only recently that I actually started paying attention to per-job settings. Once you do that, you'll learn soon enough that there really is no 'one-for-all' setings profile.

[mp3dom]

I highly doubt that. PC monitors don't do half of the goofy bit-shifting nonsense that TVs are set up to do usually. It's almost certainly the TV's fault - what model of BRAVIA is it? I have experience with some of those.
I don't know which monitor asarian have, nor the quality of the Bravia TV in general, but if it's a TN LCD monitor there's no surprise. TN are 6 bpc monitors and to achieve 16.7M colors they use dithering. Also they suffer from color shifts, reduced viewing angle etc etc.
Anyway I've watched the result posted by asarian on an Eizo ColorEdge hardware-calibrated monitor and the encoded file indeed suffers from color banding on the left and on the right of the Rachel's face as I stated above.

zones are good feature as it allows to change settings in particular areas of the video, but this would require twice the time as you need to do 1st CRF pass, watch the video and then redo a full 2nd CRF pass

[asarian]

I don't know which monitor asarian have, nor the quality of the Bravia TV in general, but if it's a TN LCD monitor there's no surprise. TN are 6 bpc monitors and to achieve 16.7M colors they use dithering. Also they suffer from color shifts, reduced viewing angle etc etc.
Anyway I've watched the result posted by asarian on an Eizo ColorEdge hardware-calibrated monitor and the encoded file indeed suffers from color banding on the left and on the right of the Rachel's face as I stated above.
I quickly had to look up the model nr. again, but it's actually a Sony Bravia KDL-40W4000 LCD TV.

[Didée]

That's not the TV's fault. It's simply the very old story of 8-bit colorspace limitations in general, and YV12 in particular.

Listen to those that advised to not denoise, or to add dithering after the denoising. Dithering works very good during playback ("Deband" in ffdshow). When using dithering before encoding, you need rather slow settings and/or high bitrate in order to keep the dithering in the compressed result.

[Atak_Snajpera]

I can only advise to not denoise at all. Let x264's PsyRDO do it's magic.