Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Sagekilla]

Ehhh you mean AQ ON/AQ OFF? and don't you think those settings are a little insane ;)

D'oh, yeah it should be AQ on/off. Fixed that. Those settings are perfectly sane for me :) 8 fps on my dual core Opty 170. Besides, that's partially the reason why I can get good quality @ 1400 kbps on CoR.

[jmnk]

Ehhh you mean AQ ON/AQ OFF? and don't you think those settings are a little insane ;)
@CruNcher: could you elaborate on why these settings are 'little insane'?

[burfadel]

Just a slightly irrelevent question, but its something that may be asked or have confusion about when your patch build is updated! Afterall, depending on the actual change, using the same settings from 720 to 721 may result in inaccurate comparisons?...

On the revision log (from www.x264.nl) for rev 721, its says:

- change the meaning of --ref: it now selects DPB size (including B-frames), rather than L0 size (which B-frames are added to)

Does that mean if we normally select 5 reference frames and set b frames to 16, that --ref should now be set to 21?...

How would this affect the use of Megui/Staxrip etc?

[Dark Shikari]

Just a slightly irrelevent question, but its something that may be asked or have confusion about when your patch build is updated! Afterall, depending on the actual change, using the same settings from 720 to 721 may result in inaccurate comparisons?...

On the revision log (from www.x264.nl) for rev 721, its says:

- change the meaning of --ref: it now selects DPB size (including B-frames), rather than L0 size (which B-frames are added to)

Does that mean if we normally select 5 reference frames and set b frames to 16, that --ref should now be set to 21?...

How would this affect the use of Megui/Staxrip etc?No, that's not what the patch means.

Previously, DPB size required was equal to (--ref + [0,1,2]), with 0 in the case of no b-frames, 1 in the case of b-frames, and 2 in the case of b-pyramid. Now, I think its equal to (--ref + 1), no matter what. Akupenguin, feel free to correct me if I'm not exactly correct.

[Sagekilla]

@CruNcher: could you elaborate on why these settings are 'little insane'?

It's insane because I enabled every setting short of esa using 16 reference threads, and unlimited merange. Some of these settings can easily be turned off without increasing the bitrate massively (think 1500 kbps instead of 1400 kbps) while providing a sizeable speed boost. But, I prefer the extra quality. ESA I don't use because the speed vs quality tradeoff isn't worth it yet.

[Dark Shikari]

It's insane because I enabled every setting short of esa using 16 reference threads, and unlimited merange. Some of these settings can easily be turned off without increasing the bitrate massively (think 1500 kbps instead of 1400 kbps) while providing a sizeable speed boost. But, I prefer the extra quality. ESA I don't use because the speed vs quality tradeoff isn't worth it yet.I'd say you could probably improve that commandline by adding a few more refs--I've found that 4 is generally not where the the curve begins to top out, even for live-action.

[burfadel]

No, that's not what the patch means.

Previously, DPB size required was equal to (--ref + [0,1,2]), with 0 in the case of no b-frames, 1 in the case of b-frames, and 2 in the case of b-pyramid. Now, I think its equal to (--ref + 1), no matter what. Akupenguin, feel free to correct me if I'm not exactly correct.

How does that affect b-pyramid then, if its always +1 instead of +2?

[Sagekilla]

I'd say you could probably improve that commandline by adding a few more refs--I've found that 4 is generally not where the the curve begins to top out, even for live-action.

I probably could yes, especially if you merged the fast ref search diff to this build. I use very slow preprocessing though (to remove grain, noise, etc), which manages to cut down on the bitrate needed by a huge margin compared to increasing refs. I actually was using 6 refs back when your fast ref build was being actively developed, but since the AQ on this build is vastly better than the older AQ I'd rather use this one. All a matter of tradeoffs, no?

On a side note, CoR is coming out very nicely.. 154k of 192k frames encoded and the file is a mere 1.09 GB big. Looking at the end file being around 1.36 GB, possibly more. Throw on an extra 440 MB for the audio track though.

[CruNcher]

I i finished my 3 longcut encodes (CRF 25) :)

No AQ

x264 [info]: SSIM Mean Y:0.9849869
x264 [info]: PSNR Mean Y:46.721 U:47.332 V:51.035 Avg:47.291 Global:46.857 kb/s:
4125.27

encoded 6842 frames, 10.16 fps, 4132.79 kb/s

AQ ON --aq-strength 1.0

x264 [info]: SSIM Mean Y:0.9843622
x264 [info]: PSNR Mean Y:45.762 U:46.754 V:50.071 Avg:46.394 Global:46.023 kb/s:
4228.15

encoded 6842 frames, 9.72 fps, 4235.46 kb/s

Old Aq --aq-strength 0.9

x264 [info]: SSIM Mean Y:0.9858071
x264 [info]: PSNR Mean Y:46.987 U:47.763 V:51.368 Avg:47.590 Global:47.106 kb/s:
6594.71

encoded 6842 frames, 9.64 fps, 6602.20 kb/s

[Sagekilla]

Can we some screen caps of that clip, CruNcher?

[CruNcher]

First i have todo a subjective test of all 3 to determine the problematic areas for this the 3rd encode has to finish first

[foxyshadis]

With AQ or None AQ?
If you get it with revision 721 SVN, please report a way to reproduce.

With AQ, I didn't bother to test multiple identical SVN encodes.

How does that affect b-pyramid then, if its always +1 instead of +2?

It now uses the same number of references for all frames, eliminating the need to trade off b or b-pyramid for extra references.

Cruncher, pleeeease link instead of inlining the images this time. ;)

[akupenguin]

Now, I think its equal to (--ref + 1), no matter what. Akupenguin, feel free to correct me if I'm not exactly correct.
DPB is equal to --ref.
Unless --ref is less than the minimum (1 for P only, 2 for B-frames, 3 for pyramid), in which case --ref selects the number of L0 refs used for P-frames (like it did before) and DPB is equal to said minimum.

[CruNcher]

Ok im finished and the result is the same as what i told you before from the visual side the difference in Bright Scenes between NoAQ and NewAQ is almost 0 (if their is a enhancement i couldn't see it ,most probably the bitrate for those scenes is allready perfectly chosen by the encoder), but the Black Scenes with very defined gradients (and especialy when noise was left) did improved with the NewAQ compared to noAQ a little, see bellow. And as you allready see from the Size difference the OldAQ gave the best visual results also in the Dark Scenes (but this is still unbalanced and not really useable that way most probably this behaveiour can be tweaked alot).

Im gonna post pictures from every Scene this testcut contains and the problematic scenes (that doesn't get enhanced enough by either the NewAQ or NoAQ).

The Testcut is buildup from 6 scenes 5 of them are dark 1 is very bright and detailed (the Dark scenes are also detailed and when wrong encoded band extremely and most of the visible problems are in the ROI then, so a very bad situation)

Scene 1 = http://s6.directupload.net/images/080119/xjbzbe2m.png
Scene 2 = http://s6.directupload.net/images/080119/hc5re8nm.png
Scene 3 = http://s6.directupload.net/images/080119/hc5re8nm.png
Scene 4 = http://s5.directupload.net/images/080119/bfs48jze.png
Scene 5 = http://s2.directupload.net/images/080119/8bmbnscd.png
Scene 6 = http://s2.directupload.net/images/080119/lxw5ntsx.png

Problematic Scenes (areas)

Scene 2
No AQ = http://s4.directupload.net/images/080119/b3r6jc5i.png
New AQ = http://s3.directupload.net/images/080119/j4933af4.png
Old AQ = http://s6.directupload.net/images/080119/ojc2ha8s.png

Scene 3
No AQ = http://s5.directupload.net/images/080119/ackxyl3o.png
New AQ = http://s3.directupload.net/images/080119/gfl2dcw9.png
Old AQ = http://s1.directupload.net/images/080119/dggqn965.png

Scene 5-1
No AQ = http://s3.directupload.net/images/080119/oxuq5zo3.png
New AQ = http://s1.directupload.net/images/080119/zm3m6gj3.png
Old AQ = http://s2.directupload.net/images/080119/afa45iww.png

Scene 5-2
No AQ = http://s6.directupload.net/images/080119/mvz5ahzn.png
New AQ = http://s1.directupload.net/images/080119/ebgix8it.png
Old AQ = http://s3.directupload.net/images/080119/qdocncsk.png

Scene 6
No AQ = http://s1.directupload.net/images/080119/e29e7rgw.png
New AQ = http://s2.directupload.net/images/080119/tnenotri.png
Old AQ = http://s5.directupload.net/images/080119/yno3ves7.png

All in all the NewAQ is very good compared to noAQ at all (especialy in those Dark Scenes they look better in Motion alot then with NoAQ this way) and this just for 100 kbps more bitrate usage (think about the OldAQ result and the bitrate difference yada yada think about 100 kbps more *g*), everyone should invest that for this big HVS Improvement :)
and it definately should go into SVN :D

Final Size
No AQ = 140 mb
New AQ = 144 mb
Old AQ = 224 mb

Tough in a ABR Encode @ 3 Mbit Visualy the New AQ fails in the Problematic ROI Scenes and the old AQ is supirior in those and shows no visible sign of any Visual degredation to any of the other scenes.

[Sagekilla]

Nice results there CruNcher. Very bothersome that we keep getting closer to that "great" flat area performance yet are still quite far from it.

Dark Shikari, you said increasing aq-strength affects how high and low a qp can raise for a frame. Is this all that it affects in the new build or do higher values give a greater tendency to allocate more bits? Because in my testing, it seemed like going from 0.5 to 1, and then to 2 had little effect on overall quality of flat areas.

[Dark Shikari]

Nice results there CruNcher. Very bothersome that we keep getting closer to that "great" flat area performance yet are still quite far from it.

Dark Shikari, you said increasing aq-strength affects how high and low a qp can raise for a frame. Is this all that it affects in the new build or do higher values give a greater tendency to allocate more bits? Because in my testing, it seemed like going from 0.5 to 1, and then to 2 had little effect on overall quality of flat areas.One of the problems with the existing AQ is that it makes a compromise--that it won't move bits considerably between frames. As a result, it avoids screwing up ratecontrol--but that also means that in extremely flat frames, it isn't willing to lower the framewide quantizer to compensate.

[DeathTheSheep]

Which is a darn shame if you use CQ. :)

[Dark Shikari]

Which is a darn shame if you use CQ. :)Actually, one interesting thought: AQ currently limits the amount it raises and lowers QPs. If I make an option to customize or remove that limitation on the commandline, then what you can do is use a static (non-automatic) sensitivity--and use AQ for ratecontrol with CQ as a base :devil:

[Sagekilla]

Any chance of a patch coming out to try this out? *wink wink* :)

[Dark Shikari]

Line 293, ratecontrol.c clips the QPs to (-5 * AQ strength, 5 * AQ strength). Comment that out, or change it to whatever you want.

That's all you need to do.

I *strongly* suggest you don't put the strength too high when doing this, and use a sensitivity between 15 and 30 or something like that--you'll have to fool around to find the best values.

[CruNcher]

this is so funny i did now a Encode @ 3 Mbit with the Old Aq and geez it looks stunning in those scenes even better then the CRF 25 New Aq encode with 4 mbit so the Old one is really thought for ABR or 2 Pass but not CRF :) those scenes look now as if they used the 6 mbit :D and the degredation on the other scenes is visualy not visible :) This is what i wanted (balanced distribution) :D

SSIM everything lowered as expected but the Visual Quality is Perfect :) no ROI problems anymore in any of those Problematic Scenes i posted above and the Final size is 104 Mb :)

x264 [info]: SSIM Mean Y:0.9807726
x264 [info]: PSNR Mean Y:44.941 U:46.045 V:49.483 Avg:45.600 Global:45.188 kb/s:
3043.12

encoded 6842 frames, 9.49 fps, 3050.65 kb/s

Line 293, ratecontrol.c clips the QPs to (-5 * AQ strength, 5 * AQ strength). Comment that out, or change it to whatever you want.

That's all you need to do.

I *strongly* suggest you don't put the strength too high when doing this, and use a sensitivity between 15 and 30 or something like that--you'll have to fool around to find the best values.

gonna try that and do the 3 mbit again, i think this should improve it like the Old AQ also for ABR :)

[TheRyuu]

I've built it with the newest rev. (r721) with this patch, mp4 output, pthreads, and avis input.

http://www.fileducky.com/NWvOdpST/

I've tested it to the point were it built and it didn't error.
Just thought I'd share this build.

[Dark Shikari]

Here are some things that I want people to feel free to experiment with my AQ (since the function itself is relatively easy to modify):

1. I find that the cap I've placed on the AQ adjustment seems like a bad solution to the issue of raising/lowering quantizers too much. Can anyone here come up with a better solution?

2. As mentioned earlier, I tried to avoid the problem of AQ redistributing bits with the automatic sensitivity. However, this doesn't bode well for very flat scenes, it seems. Is there a way to avoid screwing up ratecontrol while biasing the AQ a bit in the favor of flat scenes getting lower quantizers? This would most likely take the form of some sort of bias in the summing of the x264_aq_autosense() function.

3. Would the AQ formula benefit from any change in scaling method?

Etc.

[CruNcher]

Dark I tried now to max it out @ CRF 25 like the oldAQ does i did it with --aq-sensitivity 40 without comenting out that line now the dark small test scene gets the same enhancement as the OldAQ did @ --ag-strength 0.9 --aq-sensitivity 15 (even more bitwise but not visualy have to find the perfect visual max out point now) :) so the key for the new AQ seems to be as i thought at first really to be in the --aq-sensitivity :D

This is the max i get now with the new AQ
x264 [info]: SSIM Mean Y:0.9918510
x264 [info]: PSNR Mean Y:49.527 U:50.649 V:52.874 Avg:50.108 Global:49.794 kb/s:
8130.14

encoded 701 frames, 9.09 fps, 8136.86 kb/s

This is the max i got with the OldAQ @ --aq-strength 0.9 --aq-sensitivity 30
x264 [info]: SSIM Mean Y:0.9916371
x264 [info]: PSNR Mean Y:49.032 U:50.341 V:52.570 Avg:49.657 Global:49.356 kb/s:
7589.89

encoded 701 frames, 9.13 fps, 7596.56 kb/s

Btw isn't this now the same behaveiour like a VBR Ratecontroll ? (you get the best possible output, without any limitations) i think such a mode is really missing and this seems a cheap way to force this by useing AQ (Atemes Encoder has this mode and seems X264 now too) :D (and it seems to work perfectly) and by maxing it out it seems to work perfectly then in ABR for lower bitrates too (best possible visual result for the bitrate target) :) i think this is indeed a big big step. I think this is even better then CRF because now you can decide with the tools of H.264 how much compression and complexity in the end you wan't (like a Stream Complexity Ratecontrol) and how much Speed loss you wan't to invest for that (both sides Encoding/Decoding or based on Compression Efficiency) if it really works out as i think (this is what i allways dreamed of for a mode for EDP) :) as base for this CRF 19 or 21 should be used i think.

If my ideas work out i think there are 2 new Modes born for X264 in this moment :D
Variable Bitrate Mode (useing CRF or QP X as start)
Stream Complexity Mode (Encoding/Decoding Speed Mode)


It looks like this now

OldAQ
CRF/ABR Visually max out (CRF doesn't max out as good as with New AQ)

New AQ
CRF Visually maxes out (better then OldAQ)
ABR didn't found yet how to max it out Visually

gonna try that aproach now on ParkRun with the Old AQ maxed out :)

[DeathTheSheep]

Actually, one interesting thought: AQ currently limits the amount it raises and lowers QPs. If I make an option to customize or remove that limitation on the commandline, then what you can do is use a static (non-automatic) sensitivity--and use AQ for ratecontrol with CQ as a base :devil:

Wow, I was actually just thinking about doing something like that... sheesh. Well, I'm encoding with it now. Strength is at 0.7, and I'm trying both 15 and 30 sensitivity.

[CruNcher]

This is really a hard balance act definately the OldAQ can't improve Parkrun in any way, now i tried to balance the New AQ between Parkrun and my testcut and got better results then before in the ROI scenes (a little SSIM decrease in Parkrun, but still visualy better then No AQ) but still it's nowhere near the OldAQ results on my testcut (ABR).

[DeathTheSheep]

Okay, some results.
0.9741502 SSIM for original code, q29 AQ07. (4248KB)
0.9737121 SSIM for RC mod, q28 AQ07. (4258KB)

Visually, undecided.
Of course, I'll try other settings... but at least as far as sensitivity goes, 15 might just be too low.

[Dark Shikari]

Okay, some results.
0.9741502 SSIM for original code, q29 AQ07. (4248KB)
0.9737121 SSIM for RC mod, q28 AQ07. (4258KB)

Visually, undecided.
Of course, I'll try other settings... but at least as far as sensitivity goes, 15 might just be too low.It is too low. That sets the "center variance" at about 25000. 25 will set it at around 200,000.

[DeathTheSheep]

I'm using sensitivity 30 now (the "other extreme"), and things might be looking up.

[edit] Filesize is ridiculously high, I'm going to have to up my qp by at least 2 notches...

[chipzoller]

I'm finding this AQ function very handy. Do you know if it is currently used in any other encoders? And this may be a stupid question as I'm not an expert on the inner-workings of x264 and, indeed, H.264 in general, but does using AQ in any way tweak or perhaps "break" the H.264 spec to your knowledge? Is this a profile-limited tool, or can it be used in the creation of any stream?

[CruNcher]

I think i found a good balance :) have to test it on the longcut first :D

[Sharktooth]

@chipzoller: no. it doesnt break anything. other encoders use some kind of AQ too.

[DeathTheSheep]

Q31: 0.9750365 (RC, strength 1.0, size 4293KB)
Q29: 0.9732632 (std strength 1.0, size 4129KB)
Q29: 0.9741502 (std strength 0.6, size 4248KB)

Since it's quants we're dealing with here, it's hard to settle on a single, uniform bitrate. The three points here seem almost linear in bitrate/SSIM scaling, so I really can't say. (Needless to say, it takes forever to take derivatives of each one's SSIM vs bitrate curve for comparison).

Should I post the resultant clips?

[Dark Shikari]

Q31: 0.9750365 (RC, strength 1.0, size 4293KB)
Q29: 0.9732632 (std strength 1.0, size 4129KB)
Q29: 0.9741502 (std strength 0.6, size 4248KB)

Since it's quants we're dealing with here, it's hard to settle on a single, uniform bitrate. The three points here seem almost linear in bitrate/SSIM scaling, so I really can't say. (Needless to say, it takes forever to take derivatives of each one's SSIM vs bitrate curve for comparison).

Should I post the resultant clips?
Compare visual quality, not SSIM.

Eliminating blocking in very flat areas, for example, doesn't generally help SSIM.

[DeathTheSheep]

That's true, but at exactly the same bitrate (<1KB difference out of >4000), the large SSIM increase and PSNR slight increase isn't easy to count out.

As for blocking, it's a non-issue for me anyway at Q31...or is it?
And now for a complete 180 degree turnaround, I'm going to check the visual quality of the clips in dark scenes, if you'll excuse me. :p

[CruNcher]

I give up it seems impossible that way (changeing settings) to achive good results on the problematic ROI stuff in this testcut @ 3 Mbit with ABR and the New AQ (with CRF it's no problem or ABR and OldAQ)

[DeathTheSheep]

Okay, here's a test pack. The settings (and SSIM) can be inferred from the filenames. One of the files is without AQ, and one of them uses standard .431 at strength 0.7 with auto threshold, I believe. All AQ strengths are 1.0 unless specified otherwise in the filename with st_. As you can tell, I tried to achieve roughly 4201KB for each file; however, 9749861_RC_q29_s21 was blatantly oversized (lowering the sensitivity even one point makes it severely undersized).

http://gabe.ej.am/samples/

[CruNcher]

Okay, here's a test pack. The settings (and SSIM) can be inferred from the filenames. One of the files is without AQ, and one of them uses standard .431 at strength 0.7 with auto threshold, I believe. All AQ strengths are 1.0 unless specified otherwise in the filename with st_. As you can tell, I tried to achieve roughly 4201KB for each file; however, 9749861_RC_q29_s21 was blatantly oversized (lowering the sensitivity even one point makes it severely undersized).

http://gabe.ej.am/samples/

Wussa what for fast pictures (not used to this speed) very hard to realize any difference @ all, but they definitely all behind my visual understanding of quality, especialy the ringing and extreme blurienes would annoy me to death (it allready does in my stuff) and especialy with H.264 (blurrienes of X264 can for sure be enhanced tough). Im only used @ this problem from past codecs and ASP (except blurrienes) since the better partitioning, Qpel and Inloop deblocker should prevent such stuff from happening in H.264 (was the source that bad?). I would say no real visual improvement there visible that might come from the New AQ.

[DeathTheSheep]

Yes, the source wasn't exactly par excellence, if you catch my drift. I do see markedly better quality (detail preservation) in the q29 (and to some extent the q30) samples as opposed to both original and AR-std.

Also keep in mind this is baseline QVGA resolution (320x240), not even VGA, much less HD, not to mention q30+ equivalent bitrate, which might account for some "blurriness." :D

[edit] CruNcher: for your viewing pleasure, a max setting DivX6.8Pro and CruNcherEDP1.4.5 have been added to the site. Even at max settings, ASP (at least without tons of filtering) looks like sheer buttocks compared to baseline AVC at this bitrate/resolution.

[CruNcher]

(detail preservation) - ehh what details if i might allowed to ask, i see only a bunch of colors flipping around in ultra speed ;)

*hides far far away deep inside of looney tunes and hanna babera land from the wrath of the anime community that might come over him like a Dragonball @ the sunset dawn, for this statement*

[DeathTheSheep]

NAPA: Vegeta, what does the SSIM say about its quality level?

VEGETA: It's over nine thousaaaand!!

NAPA: What 9000?!! There's no WAY that can be right!

You don't watch much anime, do you, CruNcher? Take a look at how your EDP with max settings (except GMC) does on the clip, for instance... tsk, tsk. If you don't know the rubric, maybe you shouldn't judge. ;)

[Dark Shikari]

My hunch was right.

A constant AQ sensitivity with unlimited (no clipping) AQ, in a sense using AQ to hack at ratecontrol, is IMPRESSIVE, to say the least. And I mean astounding.

Before (this thread's automatic-sensing AQ):

http://i1.tinypic.com/80neaz9.png

After:

http://i16.tinypic.com/6l8ke14.png

Obviously the framesize is drastically different, but that's the point; this scene needed more bits. The effect of this is simply astounding; it may make this AQ even better, though it cannot work with CRF if you want any hope of a bitrate close to that you would get without AQ.

[DeathTheSheep]

You just took out that one line and used sensitivity X, right? That's what I did (see previous posts).

If so, what sensitivity precisely did you use?

[CruNcher]

Indeed i don't, but don't forget your sample uses inloop deblocking and ASP isn't haveing that feature it can make only use of outside Postprocessing, but you know all that. Think about the same AVC Baseline sample but with the sharpness of ASP, im sure it's doable someone just has to find the right way to achive that and nope you won't with only turning off deblocking unfortunately it's not that easy ;). Btw i watched anime back when i was a kid but todays stuff is ehh different i loved http://www.imdb.com/title/tt0185110/ , now you know my depest secret ;)

Yes Dark as i said it's like a VBR then it gives the bitrates to those scenes that need it the most, but i have problems to reproduce the same effect in ABR with the OldAQ it works in both RC modes, tough the OldAQ doesn't improve ParkRun the way the new does. :)

[Dark Shikari]

You just took out that one line and used sensitivity X, right? That's what I did (see previous posts).

If so, what sensitivity precisely did you use?Sensitivity was 30, which seemed like a good medium. And yes, I just removed that line.

[DeathTheSheep]

Hmm, I use 25. 30 seems rather excessive to me. Besides, as you can see from my previous post (or gabe.ej.am/samples/), there's many ways to get at the same filesize via QP, with SSIM and visual quality to match. I find qX without ratecontrol is approximately equal to -qX --aq-sensitivity ~25 OR -q[X+1] --aq-sensitivity ~30 for a good range of balanced sources.

[Dark Shikari]

Also, it was requested that I post this updated, working version of me-prepass (supposedly, I haven't tested it) on doom9, so I will post it here.

ME-prepass patch, working with 721, supposedly. (http://pastebin.com/f387ef1d)

[Reuf Toc]

The result of your hack is really awesome :eek:

Here is a gif of your pictures with contrast enhanced :

http://img255.imageshack.us/my.php?image=sanstitre1rl1.gif

[DeathTheSheep]

May I hope you have the updated, working satd floating around there as well? If so, here's your request... :devil:

[Dark Shikari]

Working SATD doesn't exist at the moment, peng will have to updated the patch.

Also, apparently that Prepass patch is broken and won't compile. Blegh. At least I did make it.

Also, major major major major update. Read the new instructions, and be shocked at the quality improvement.