Adaptive Quantization woes in x264

[Sagekilla]

After reading a bit and finding out about the AQ feature.. and subsequently downloading it, I seem to have run into a bit of trouble using it. To save you guys a somewhat unnecessary story.. I've redone all my settings and came out with the following settings:

--crf 18 --ref 4 --bframes 3 --b-pyramid --weightb --b-rdo --bime --direct auto --filter -2:-1 --analyse all --8x8dct --no-fast-pskip --subme 6 --me umh --trellis 2 --mixed-refs --progress --threads auto --aq-strength 0.5 --aq-sensitivity 15 --thread-input --merange 24

It gives me pretty good quality, yet I still get the blocking issue in the dark areas, even with AQ enabled. I've placed all my source files (Except for the .d2v and the .vob file) in a folder here. (http://skdotnet.sytes.net:45312/X264/) You can take a look at the png I took of the video playing back in VLC, and the little blocking I get. It's a bit annoying to me, I see it throughout the video and you can view the exact video that x264 creates in there too. I'd also like to mention I've done away with GUIs altogether since I like using the CLI. I don't use Staxrip, Mencoder, or MeGUI. Any help with this issue would be much appreciated! If anyone can suggest some tweaks to my settings that may be beneficial I'd really appreciate it too. I'm off to go tweak my settings a bit!

[ToS_Maverick]

try increasing aq-strength to 0.9 and 1.2, if that doesn't help, lower the sensitivity to 10.
also, try using subme 5 and no trellis. merange 24 is also a bit overkill, for me, 16 is enough.

[Sagekilla]

Yeah, I've been just testing settings a bit. I want to see how the settings correlate to the overall quality (and filesize more specifically) I just tested --me 16 and --me 24, and you're right.. bit overkill on my part. I'm going to try increasing the aq strength and sensitivity to what you said, then try out subme 5 + no trellis. Thanks!


Edit: I just tried out aq-strength 0.9 and it increased the filesize a bit too much; a good portion of the problem areas in the darkness went away, but most are still there. Testing out aq-strength 0.7, aq-sensitivity 12 now.

Edit 2: Decreasing sensitivity to 11 seems to have done it, along with leaving the strength at 0.5.. if I had it even at 0.6, the bitrate jumped by 17% I still have some blocks, but at least they're not as bad as it was on aq-sensitivity 15.

[Sagekilla]

Wow.. thanks for the trellis suggestion. It helped, A LOT..

--trellis 0: 924 kb/s @ 16.27 fps (Global PSNR: 45.138, SSIM: 0.9793955)
--trellis 2: 928 kb/s @ 11.29 fps (Global PSNR: 45.257, SSIM: 0.9798104)

I don't know how much of a difference in SSIM is supposed to be "meaningful," but from that data above alone makes me wanna use --trellis 0. 44% faster and 0.5% less bitrate... not too shabby.

[ToS_Maverick]

because of this, i use me 16, hex sub5 and no trellis ;)

good quality and good speed, especially for HD stuff

[Sagekilla]

Yeah, only setting I don't really agree on is the subme 5; the bitrate increase compared to the encoding speed increase isn't worth it for me, but otherwise all your other suggestions REALLY helped a lot! Heh.. Now I'm off to go into the world of matrices.

[ToS_Maverick]

for this i can recommend you the prestige matrix:
http://forum.doom9.org/showthread.php?t=124755&highlight=prestige+matrix

or the ones from mp4-guy
http://forum.doom9.org/showthread.php?t=117041

[Sagekilla]

Thanks again.. after combining all the stuff so far in this thread, the final quality of my videos has improved quite a bit.. I've been doing comparisons between the CRF'd version and the 2-pass one I did a few days ago, and the CRF'd seems to be much better quality. It's encoding a hell of a lot faster then the old one did too..

[Dark Shikari]

Yeah, --trellis 2 can cause blockiness. If you're using AQ and you're trying to get a video without blocks, don't use --trellis 2.

[Terranigma]

Yeah, --trellis 2 can cause blockiness. If you're using AQ and you're trying to get a video without blocks, don't use --trellis 2.

Oh my, Dark Shikari beat me to it :P
Trellis-2 can cause unexpected results with AQ.

[elguaxo]

Trellis-2 can cause unexpected results with AQ.

Is there an explanation why you get unexpected results/blockiness using --trellis 2 with AQ?

Thanks.

[Dark Shikari]

Is there an explanation why you get unexpected results/blockiness using --trellis 2 with AQ?

Thanks.
Trellis (--trellis 2 especially) causes some blocking in order to obtain overall higher bitrate efficiency. In videos where you want to avoid blocks, AQ allows you to reduce blocking. However, AQ costs more bitrate than Trellis saves, so you're creating blocks with trellis and then trying to remove them with AQ.

[Terranigma]

Is there an explanation why you get unexpected results/blockiness using --trellis 2 with AQ?

Thanks.
first, check out this (http://www.neuron2.net/library/avc/overview_x264_v8_5.pdf) .pdf document hosted at neuron2's site. Review 2.4.2 (Trellis in x264) to understand how it works.

Also, check my reply here (http://forum.doom9.org/showpost.php?p=1028772&postcount=28) about trellis-2.


What you could probably do, is do 2 or more passes, but only use adaptive quantization for the first pass (with either trellis-1 or trellis disabled), then encode using trellis-2 for the second/nth pass (With AQ disabled).

[elguaxo]

Thanks Dark Shikari and Terranigma for your quick answers!

[Sagekilla]

first, check out this (http://www.neuron2.net/library/avc/overview_x264_v8_5.pdf) .pdf document hosted at neuron2's site. Review 2.4.2 (Trellis in x264) to understand how it works.

Also, check my reply here (http://forum.doom9.org/showpost.php?p=1028772&postcount=28) about trellis-2.


What you could probably do, is do 2 or more passes, but only use adaptive quantization for the first pass (with either trellis-1 or trellis disabled), then encode using trellis-2 for the second/nth pass (With AQ disabled).


I could only do this is if I was doing a 2-pass bitrate based encode though, correct? I remember playing around with the CLI a bit and crf was a 1-pass deal. I could add --pass 1 and make it output a stats file.. but it wouldn't do a second pass based off a stats file.

[Terranigma]

I could only do this is if I was doing a 2-pass bitrate based encode though, correct? I remember playing around with the CLI a bit and crf was a 1-pass deal. I could add --pass 1 and make it output a stats file.. but it wouldn't do a second pass based off a stats file.
crf can be used in 2 passes as well, instead of using abr for the first pass, you'd use crf:
e.g.:
ABR For 1st pass.
--pass 1 --bitrate 1000

CRF For 1st Pass
--pass 1 --crf xx.x

But if you're trying to do this with megui, then for custom commandline options through the "Zones" tab, add --crf xx.x for "Custom Commandline Options". It doesn't matter what you specify for the bitrate in 2-pass ABR, because CRF will just override it anyways.

[Sagekilla]

crf can be used in 2 passes as well, instead of using abr for the first pass, you'd use crf:
e.g.:
ABR For 1st pass.
--pass 1 --bitrate 1000

CRF For 1st Pass
--pass 1 --crf xx.x

But if you're trying to do this with megui, then for custom commandline options through the "Zones" tab, add --crf xx.x for "Custom Commandline Options". It doesn't matter what you specify for the bitrate in 2-pass ABR, because CRF will just override it anyways.

Alright thanks.. Yeah I just use straight CLI actually, never touched megui in my life!