x264 CRF16 Animation preset not good enough for animation?

[pwnsweet]

So I'm encoding the 1080p Evangelion 1.11 Blu-ray. I'm using x86 x264 (8bit) v2273 in MeGUI. Here are my x264 settings (basically the Animation preset with merange set to 24 instead of 16, bframes set to 16 instead of 10 and ref set to 4 instead of 16):

cabac=1 / ref=4 / deblock=1:1:1 / analyse=0x3:0x113 / me=umh / subme=11 / psy=1 / psy_rd=0.40:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=36 / lookahead_threads=8 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:0.60

My Avisynth script is simply:

Dss2("G:\Movies\Eva\Evangelion 1.11.mkv")
converttoyv12()

Comparing the encoded file to the lossless source, the following can be seen in certain scenes (mainly slow/no movement scenes):

http://imgbox.com/gallery/edit/vQ6WNDOwwe/kRXxrCdqtcL0Jn2o

On the left is the source, and on the right is the encode. You can clearly see what looks to be banding/blocking in the encode. By switching from the Animation preset to the Film preset I was able to reduce the issue somewhat, but lowering the CRF value to 0 eliminated it entirely.

This seems to suggest that it is the result of low bitrate. But how can that be? According to MeGUI, CRF16 is supposed to result in "near perfect quality" and in most scenes it does, but then there are scenes like this one that look terrible. In the past I used CRF16 on western animation such as the Lion King and I've never seen this before. Furthermore, I actually got worse results using the Animation preset on - not something I would expect given the source is...animation!

So my question is, can anyone explain why this problem is occurring and does anyone know how to 'fix' it without having to lower the CRF below 16?

[sneaker_ger]

If you search the forum you will see that this is a known problem of 8bit x264. The easiest way to fix it is to use 10bit x264, but the resulting file will not be compatible anymore with any kind of consumer hardware decoder (found in Blu-Ray players, Phones, TVs, etc.) and basically only run on the PC.

I suggest you look here for further suggestions for 8 bit encoding and do a forum search, before asking further questions:
http://forum.doom9.org/showthread.php?p=1564253

Btw, why did you deviate from the presets and for example increased bframes and subme but reduced refframes?

[pwnsweet]

Thanks for the link.

10bit x264 is not an option for me, because I require broad compatibility. That's also the reason why I decreased reference frames to 4, to maintain Level 4.2 compatibility. I encode on a dual xeon hexacore rig, so I can afford to increase me-range and subme without a significant hit to fps.

On that note, I've also tried limiting threads to 16 to see if it reduces the banding/blocking - it doesn't.

[paradoxical]

On that note, I've also tried limiting threads to 16 to see if it reduces the banding/blocking - it doesn't.

Why would it have? The banding issues are due to rounding errors introduced by the use of 8-bit values for calculation.

[Dark Shikari]

Avoiding banding can be very difficult in 8-bit encoding on some footage. Your best bet is probably to try to add some dither and compress with grain/dither-retention in mind, instead of animation. No matter what, it's going to waste a lot of bits.

[mp3dom]

So my question is, can anyone explain why this problem is occurring and does anyone know how to 'fix' it without having to lower the CRF below 16?

There's no way to fix it while keeping CRF at 16, unless you put a huge amount of dither noise to prevent such problem (you can try with the Blockbuster plugin, that puts noise or dither noise on flat pattern)... but at the end it would be a 'different' movie. The original bluray release of Rebuild of Evangelion have an average bitrate of 33 Mbps. There's a reason why it is so high... and the encoder was also fine-tuned to keep such fine dithering at maximum.

[bikar]

+ for evangelion psy_rd=0.40:0.00 , aq=1:0.60 is crap
do some test encodes and start with psy_rd=1.00:00 ,aq=1:1.0