Does Trellis affect VAQ?

[mahsah]

Sorry, I'm a newbie with x264, and I just want to reduce banding and blocking in dark areas in an anime I am encoding. The new VAQ seems to be a good way to do this, but since I normally use the AE-Maxquality profile in megui (which uses Trellis 2), will this effect the VAQ at all (they are both quantization methods, right?)

Also what bitrate would be good for a 720x480 anamorphic encode with VAQ enabled? I understand it is better to have a bit more bitrate with VAQ?

[Dark Shikari]

VAQ chooses the quantizer--trellis performs the quantization. They're two separate things, and neither will conflict with the other.

[mahsah]

Alright, thanks. Also you wrote the VAQ right? Great job, I get awesome results with it.

[Dark Shikari]

Alright, thanks. Also you wrote the VAQ right? Great job, I get awesome results with it.Correct, and welcome :)

[bluebebe]

how can i enable VAQ in the encodings? uses this the params --aq-strength and threshold?

[Dark Shikari]

how can i enable VAQ in the encodings? uses this the params --aq-strength and threshold?If you have a patched build, AQ is on by default. Strength can be raised with --aq-strength.

[bluebebe]

thanks for the hint, and "--aq-strength 0.0" will completly disable the adaptive quantizer of x264 = means best quality results at cost of higher bitrate?

[Dark Shikari]

thanks for the hint, and "--aq-strength 0.0" will completly disable the adaptive quantizer of x264 = means best quality results at cost of higher bitrate?Huh? Generally AQ is beneficial for quality.

[Sharktooth]

the old AQ was sacrificing quality to remove blocking in some areas, Dark Shikari's new V(ariance)AQ is completely different and helps quality in almost every situation.

[bluebebe]

i learned in xvid encoding lession that AQ will compress complete dark or white areas with a lower bitrate and i thought it was so in x264 too. it is not so okay, what is the min / max range for the --aq-strength param and esp. the best setting which will produce the best results? it is better to use a low strength under 1.0 or is it better to set a higher number of it?

[gav1577]

@bluebebe I would not recommend going over --aq-strength 1.0 personally i leave it at 1.0 always
i find best at this level for combating the effects of banding :)

[Sharktooth]

i learned in xvid encoding lession that AQ will compress complete dark or white areas with a lower bitrate and i thought it was so in x264 too. it is not so okay, what is the min / max range for the --aq-strength param and esp. the best setting which will produce the best results? it is better to use a low strength under 1.0 or is it better to set a higher number of it?
there are several kinds of possible "adaptive" quantizations, ppl keep inventing new algos every day... adaptive is just a generic term. xvid's AQ is lumimasking... that's completely different from Haali's AQ and Dark Shikari's Variance AQ.

[TheRyuu]

there are different kinds of possible "adaptive" quantizations. adaptive is just a generic term. xvid's AQ is lumimasking... that's completely different from Haali's AQ and Dark Shikari's Variance AQ.

Unless you're using an xvid built with VAQ.... :)

[Sharktooth]

of course

[Dark Shikari]

Unless you're using an xvid built with VAQ.... :)Though even then its quite a bit different both because of the quantizer field smoothing Xvid uses, along with the fact that MPEG-4 ASP's AQ ability is somewhat restricted.