pretty clean video, flat areas to little bitrate

[moviefan]

Hey guys,

I would like to compress a rather clean video (pre-processed with avisynth) with x264. Its resolution is 720x576 and I would like to keep the DVD specs, since it looks to me a lot sharper than a resized to square pixels version and additionally because I want to have the option, to author it to Bluray or HD-DVD at some time when it gets cheaper.
My problem is that the video after pre-processing is very clean and sharp, gradient areas don't seem to have banding, if so, only very little, hardly visible, but when I encode it with x264 with CRF 20, these flat areas have so little detail that they look horrible. I tried applying AQ, but then the picture looks blurred. Is there anything I can do to the x264 settings to assign more bitrate to the flat areas or any other way to make the picture look better?

My command line:
--crf 20.0 --level 4.1 --keyint 14 --min-keyint 2 --ref 5 --mixed-refs --bframes 16 --b-pyramid --bime --weightb --direct auto --filter -2,-1 --subme 6 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 14475 --vbv-maxrate 25000 --qcomp 0.8 --me umh --threads auto --thread-input --cqmfile "eqm_avc_hr.cfg" --progress --no-psnr --no-ssim --output "output" "input" --mvrange 511 --aud --nal-hrd

[Dark Shikari]

I tried applying AQ, but then the picture looks blurred. Is there anything I can do to the x264 settings to assign more bitrate to the flat areas or any other way to make the picture look better?Which AQ are you using--Haali's old AQ, or the current VAQ? VAQ builds can be found at x264.tk.

AQ is really the only way to deal with flat areas, in my experience.

[moviefan]

I use x264 736-2 Jarod's patched build from meGUI, which has the HD-DVD compliancy patch, but I think it still uses the Haali AQ. What settings would you suggest for AQ anyhow? I have experienced blur with something like --aq-sensitivity 15 --aq-strength between 0.5 and 1.0. The pre-processed picture looks to me really clean, so there's little noise, no dither but of course flat areas don't consist of only 1 colour, but maybe 2 or three, for my eyes. But that should be normal, since it's not an anime... The result after encoding then is nothing close to a flat area with a tiny gradient, more like two areas with two colours separated very clearly. I hope you understand, what I mean... How would you deal with that problem?

[Dark Shikari]

The result after encoding then is nothing close to a flat area with a tiny gradient, more like two areas with two colours separated very clearly. I hope you understand, what I mean... How would you deal with that problem?Unfortunately, its basically impossible to keep such a single-color-step gradient.

You can try gradfun2db or adding grain for preprocessing, but its really no different from lowering --qpmin and adding even more bitrate.

[moviefan]

Hm, I have found out that lowering --qmax to something around 25 at CRF 19 gives a similar file size and better quality than before. Is there any reason not do that? I am not very experienced with x264 and don't know about possible things, I should not do for any reason... And how come that CRF 20 with --qmin 10, --qmax 51 is similar in size with CRF 19 and --qmin 1, --qmax 28?

[Dark Shikari]

Hm, I have found out that lowering --qmax to something around 25 at CRF 19 gives a similar file size and better quality than before. Is there any reason not do that? I am not very experienced with x264 and don't know about possible things, I should not do for any reason... And how come that CRF 20 with --qmin 10, --qmax 51 is similar in size with CRF 19 and --qmin 1, --qmax 28?It depends if you're using AQ or not, and which AQ algorithm; I think Jarod's build uses the new AQ.

Lowering --qpmax is probably a bad idea.

[moviefan]

I don't use any AQ right now... Why is lowering --qpmax a bad idea?

[Dark Shikari]

I don't use any AQ right now... Why is lowering --qpmax a bad idea?Because it needlessly wastes bits where its not needed.

The problem you have can be solved with gradfun2db before processing, and AQ. Note that AQ is on by default with Jarod's build.

[moviefan]

OK, but then can you give advise what settings I should use? I really don't know what makes sense in my case...

[Sagekilla]

Grad2fundb() should work fine. default is grad2fundb(thr=1.2) Try increasing thr in small increments (fractional, not whole integers) if it's not enough.

[moviefan]

And what do you say about raising inter/intra luma quantization deadzone? I've read this value determines the amount of details of the stream passed on to x264. Would that make sense and if so, how should I increase both values - what does intra and inter mean?

[Dark Shikari]

And what do you say about raising inter/intra luma quantization deadzone? I've read this value determines the amount of details of the stream passed on to x264. Would that make sense and if so, how should I increase both values - what does intra and inter mean?Lowering them increases detail (but costs more bits). Its probably not going to help in your case.

Additionally, note that deadzone is incompatible with trellis (they're two different algorithms that do the same thing in different ways; you can't run both).

[moviefan]

Hm, I thought it is the other way around... raising = more details (http://mewiki.project357.com/wiki/Video_configuration_dialog/X264_Configuration)

Hm, so if this isn't going to help, what magic does AQ actually do to help? Would trellis be HD-DVD compliant and if so, should I use trellis 1 or 2?

[Dark Shikari]

Hm, I thought it is the other way around... raising = more details (http://mewiki.project357.com/wiki/Video_configuration_dialog/X264_Configuration)

Hm, so if this isn't going to help, what magic does AQ actually do to help? Would trellis be HD-DVD compliant and if so, should I use trellis 1 or 2?
Inter / Intra luma quantization deadzone: Also known as inter/intra deadzone. Specifies the level of detail to be discarded and not taken into account by the encoder. Lower values discard less information, and therefore hand the encoder a more complex stream. The main use of these settings is to set a detail threshold. Lower settings can help grain retention, while higher settings may reduce noise. Use of deadzone settings requires trellis to be turned off.No, it says lower = more details.

Trellis is just a quantization algorithm, it doesn't change compliance.

AQ's magic is that it takes bits away from complex areas (edges/high detail areas) and gives them to flatter areas.

[moviefan]

Ah, okay. And how would you choose AQ values for a slight shift from complex to flat areas?

[Dark Shikari]

Ah, okay. And how would you choose AQ values for a slight shift from complex to flat areas?The default (AQ strength 0.5) is likely fine. For a stronger effect, raise it more, of course. Above 1.0 is strongly discouraged.

[Sharktooth]

I use x264 736-2 Jarod's patched build from meGUI, which has the HD-DVD compliancy patch, but I think it still uses the Haali AQ.
No. It was patched with the latest VAQ.

[Chainmax]

That's awesome news, thanks for confirming it Sharktooth.