Encoding 3d Movies. How to avoid blockyness

[cogman]

I have just finished encoding Ratatouille and am having some problems with blockyness, especially noticeable in dark scenes and on the rats hair. I am using CRF of 20 with no filters (I figure a cartoon like this needs no filters) and very high settings.

I wish I could include some pictures, but I guess part of the problem is that zoom player is lightening up the picture a bit (which makes the defects all the more obvious) AVS on the other hand renders the picture a bit darker. Here are my encoding settings anyways, maybe you can point out something that could be changed.

--crf 20.0
--ref 16
--mixed-refs
--no-fast-pskip
--bframes 16
--b-pyramid
--b-rdo
--bime
--weightb
--direct auto
--filter -1,-2
--subme 7
--trellis 1
--analyse all
--8x8dct
--vbv-maxrate 25000
--me esa
--threads auto
--thread-input
--cqm "jvt"
--progress
--no-psnr
--no-ssim
--output "output" "input"

[foxyshadis]

AQ. Old AQ or Dark Shikari's new AQ, both will prevent the blocks.

Any special reason for JVT, or just a preference? m4g-smooth looked better on rattatouille, for me, but I didn't test jvt with that.

[Dark Shikari]

AQ is your friend.

I'd suggest you avoid CQMs unless, when not using a CQM, you run into a specific problem that the CQM you're using sets out to solve.

Ref 16/ESA is probably ridiculous overkill unless you don't care about encoding time.

[Atak_Snajpera]

I agree with guys above. Remove custom matrix and use AQ. Start with --aq-strength 0.5 --aq-sensitivity 5

[cogman]

I'm using MeGui (and the CQM was actually there by mistake, I just forgot to change it :)), is the option to configure AQ strength there or am I going to have to go to the command line?

I didn't really care all that much about encoding time hence the almost pointlessly high settings, I just didn't expect any blockieness with my current settings.

Anyways, Ill try and hunt down how to set AQ strength in me-gui (and if not, I'm not too scared of the console)

*edit*

Ok, so MeGui doesn't have any ability to do AQ strength, so I went straight to the console. Ill tell you in a little while if it worked or not. (BTW this is not HD so on my rig it is going at about 5.5 fps, not bad in my opinion :))

*edit again*

Hmm, so I have bumped the AQ sensitivity to 5 and strength to 1.0, it has helped, but the blocking is still noticeable. So I have changed the CRF to 19, dropped the sensitivity to 2, and added no-dct-decimation to see if that helps at all. Hope it works :)

[Dark Shikari]

I'm using MeGui (and the CQM was actually there by mistake, I just forgot to change it :)), is the option to configure AQ strength there or am I going to have to go to the command line?

I didn't really care all that much about encoding time hence the almost pointlessly high settings, I just didn't expect any blockieness with my current settings.

Anyways, Ill try and hunt down how to set AQ strength in me-gui (and if not, I'm not too scared of the console)

*edit*

Ok, so MeGui doesn't have any ability to do AQ strength, so I went straight to the console. Ill tell you in a little while if it worked or not. (BTW this is not HD so on my rig it is going at about 5.5 fps, not bad in my opinion :))

*edit again*

Hmm, so I have bumped the AQ sensitivity to 5 and strength to 1.0, it has helped, but the blocking is still noticeable. So I have changed the CRF to 19, dropped the sensitivity to 2, and added no-dct-decimation to see if that helps at all. Hope it works :)Turning off DCT decimation is probably not going to help blocking at all. Raising AQ sensitivity is more likely to help blocking issues, since low AQ sensitivity simply applies AQ to all blocks instead of the specific blocks where problems occur.

You could also try using my experimental AQ.

[kumi]

I have had much more success combating macroblocking using Dark's AQ with --aq-sensitivity 0 than with any other non-zero sensitivity setting. But since you are in CRF mode, it's probably going to make final output size unpredictable.

[*.mp4 guy]

I'd suggest you avoid CQMs unless, when not using a CQM, you run into a specific problem that the CQM you're using sets out to solve.

The Smooth matrix is supposed to reduce blocking without introducing extra (visible) ringing, by sacrificing details, hence "smooth". It generally looks good on cg. All the same, I don't want to push matrices on people, I just thought your comment might be a little misleading, given that the OP is trying to reduce blocking, and this matrix is supposed to help in that regard.

[cogman]

So here is what I am finally doing for the final encode. I am using Dark's AQ, but not the automatic detection mode (I'm using a 30 if you can believe it) and my AQ strength is fairly high as well (a 1.5...) As well I have changed my CRF to 18 and threw in mp4 guys smooth matrix. Still, there is some blockyness, for those that have the movie, the worst point is where the cook (garbage) dude is going to drown the main rat character in. They have a combination of fog, and dark blues, x264 does not seem to like that at all :). doing the changes I listed, it is not horrible at that scene but it is still fairly bad (not as bad as it was to start out with) However I fear if I go any hire with the settings I wont be saving any bits over the original dvd rip.

Thanks for all the help. I recommend this scene be used for any upcoming codec shoot out, it is a beast to encode.