Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Don_Genaro]

Those are really bizarre settings--UMH/8x8dct/many refs, but subme 1?!

I think they´re taken from the megui CRF profile wich has --subme 1

[nurbs]

Aq version 0.47

avs:
SelectRangeEvery(1000, 50)

Commandline:
--crf 20.0 --level 3.1 --ref 3 --bframes 3 --b-pyramid --weightb --filter -1,-1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-maxrate 12000 --me umh --threads auto --thread-input --progress --no-psnr --no-ssim

--aq-strength 1.0 when AQ is used

Coffee and Cigarettes:
No AQ : 1387
aq-sensitivity 9 : 1079
aq-sensitivity 10 : 1516

Clerks 2:
No AQ : 3123
aq-sensitivity 12 : 2677
aq-sensitivity 13 : 3161

A Farscape Episode(4x22):
No AQ : 1755
aq-sensitivity 8 : 1676
aq-sensitivity 9 : 2170

Severance:
No AQ : 1362
aq-sensitivity 8 : 1071
aq-sensitivity 9 : 1413

The Great Dictator:
No AQ : 1627
aq-sensitivity 12 : 1552
aq-sensitivity 13 : 1857

Babylon 5 - Lost Tales:
No AQ : 1108
aq-sensitivity 7 : 946
aq-sensitivity 8 : 1709

[ToS_Maverick]

Dark Shikari, what have you done, it's even more insane now :D

Black.Pearl.Sample test crf 18 aq00.mkv 35,3 MB
Black.Pearl.Sample test crf 18 aq10 sens10.mkv 33,3 MB
Black.Pearl.Sample test crf 18 aq10 sens11.mkv 40,2 MB !!!
Black.Pearl.Sample test crf 20 aq10 sens12.mkv 34,2 MB

SAW comp test crf 18 aq00.mkv 15,4 MB
SAW comp test crf 18 aq10 sens09.mkv 17,3 MB
SAW comp test crf 18 aq10 sens10.mkv 21,7 MB

Casablanca test crf 18 aq00.mkv 13,5 MB
Casablanca test crf 18 aq10 sens10.mkv 11,2 MB
Casablanca test crf 18 aq10 sens11.mkv 13,3 MB

quality for BP crf 18@10 and crf 20@12 seems to be quite identical, bot got SSIM 0.973xx and look good. i just had a quick look over them, but they seem fine.

so my vote at the moment is sens for default at 10. nice number and seems to be quite ok :cool:

[salehin]

Thanks a lot, guys. :)

...It vastly cleaned up the code, and more importantly, it fixed a rounding error with low QPs which should somewhat improve visual quality in flat areas. However, this may change the results of your sensitivity tests.

I may be completely wrong, but does it mean that now the flat areas gets more balanced bits and thus making the QP in those sections (and consequently the overall avg QP) better. It would be great if you can kindly explain in laymen's terms.

Regarding: I noticed that in very clean source (like bbchd power of the planet h264 1080p) that contains very small amount of dark sequences, the encode quality improved a signficantly with weak quantization (0.30 - 0.50) and sensitivity of 20-25 (based on bob0r's v .45). Did anyone had similar experience?

Also, what if the source is quite dark, or full of dirts due to old age, 60's handheld cameras etc? Thanks :)

[Morte66]

Here we go again at the Blue Balls Lagoon (Entourage season 2 episode 11, about 0.94GB of MPEG2), with v0.47.

Levelled/deblocked/denoised/debanded:

crf 18 no AQ: 372MB
crf 18 VAQ strength 1.0 sensitivity 17: 538MB
crf 18 VAQ strength 1.0 sensitivity 14: 414MB
crf 18 VAQ strength 1.0 sensitivity 13: 370MB
crf 18 VAQ strength 1.0 sensitivity 12: 325MB

I would say sensitivity 13 (which matched the bitrate of non-AQ) produced a well-balanced encode. No one imperfection stood out ahead of the others. Blocking/banding was there if I looked for it, but it did not leap out as the dominant issue. It also seemed a little soft compared to no AQ, it could use another crf level or two to get back the crispness.


Same material just levelled, no other cleanup:

crf 18 no AQ: 623MB
crf 18 VAQ strength 1.0 sensitivity 17: 1.17GB (25% bigger than source)
crf 18 VAQ strength 1.0 sensitivity 14: 900MB
crf 18 VAQ strength 1.0 sensitivity 13: 792MB
crf 18 VAQ strength 1.0 sensitivity 12: 685MB
crf 18 VAQ strength 1.0 sensitivity 11: 580MB


Well, call it 13 for denoised and 11.5 for noisy. It's getting tighter, but I don't think you can remove the sensitivity parameter yet.

[Dark Shikari]

Well, call it 13 for denoised and 11.5 for noisy. It's getting tighter, but I don't think you can remove the sensitivity parameter yet.The idea is that some videos do indeed need more bits than others--and therefore noisy videos will get more bits at the same "quality level." Therefore, sensitivity isn't needed, because all that matters is that two videos at the same AQ strength and CRF are "the same quality."

[CruNcher]

Nice with the latest patch i don't need the Prestige Matrix anymore Aku's 0.47 :) and save some details --aq-strength 1.0 alone is now able to handle all of the scenes accordingly (except the very dark scene when he gets out of the watter (killer scene) that's still a mess but that was expected, just to less bitrate would need a 2pass for that or very strong oldaq) and doesn't lose anything of it's visual detail enhancement the SSIM lowered a little again (steady with every patch) but the visual balance is much better now :)

[Yoshiyuki Blade]

Hmmm so how do you guys determine the best AQ sensitivity (at a given clip)? The one that produces a file size approximately equal to the same clip without AQ?

EDIT: I didnt mean to use the word "best" but rather "recommended" or something.

[akupenguin]

It doesn't matter. Any sensitivity is equally good. What we're tuning is the mapping of CRF value to quality level, so that hopefully CRF will mean the same thing with and without AQ.

[Dark Shikari]

Hmmm so how do you guys determine the best AQ sensitivity (at a given clip)? The one that produces a file size approximately equal to the same clip without AQ?

EDIT: I didnt mean to use the word "best" but rather "recommended" or something.It seems to be roughly around 10-12 for most clips.

[EuropeanMan]

Perhaps a stupid question...

My custom parametre in Command line: --b-pyramid doesn't work...it errors out in MeGUI...any suggestion on how to fix it? For now, I eliminated this...and the encode went through...

thanks in advance.

I'm using the latest version of MeGUI on vista ultimate. all plugins updated...and i used ONLY the build version of AQ (don't know what this means) & installed...

And yes I'm doing my very very first x264 encode. and PS...since noone has answered this yet in my other thread...is it possible to use DTS audio in my mp4? :)

[DeathTheSheep]

In the interest of being honest, I don't think this new AQ with "fixed" rounding does as well on low-bitrate (anime). Here's what's happening qualitatively as a result of the new AQ:

1. Sensitivity must be 3 or 4 lower at the same QP for comparable bitrate.
2. SSIM at similar bitrate is decreased (tried with many QP/sensitivity combinations).
3. Dark, crappy scenes (pardon my slang) are allocated *less* bitrate in the video stream, and bright, high-contrast ones are allotted more than with previous AQ. Increasing AQ strength seems to take bits away from both.

[edit]To actually see them, cut from the original which is the same (total) size under both AQs, http://gabext.com/samples/

Maybe there should be an anime-lowpass. Blocks that are too flat shouldn't be given quantizers so low, in order to improve what we actually see.

[akupenguin]

An effect of the rounding (before I fixed it) was to add a small constant to all the variances, thus reducing the difference in QPs (but not exactly the same as reducing aq-strength). If that is desirable, we can explicitly add such a constant, and still benefit from the higher arithmetic precision.

[CruNcher]

@Death
Aku shifted it very nicely :) at least for mixed Quality stuff it makes alot of sense (Flat and Non Flat) look @ this

http://mirror05.x264.nl/CruNcher/force.php?file=./noaq.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./newaq-dark.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./roi-fixed-newaq-dark-cruncher.mkv (my aproach shifting Darks AQ with a custom matrix)
http://mirror05.x264.nl/CruNcher/force.php?file=./newaq-aku.mkv akus work :)

i know people rated the newaq-dark scene before allready as ok, but there were clearly problems visible to me (tough they are hard to realize with a correctly calibrated viewing device, especialy as there aren't really in the ROI in this example (hidden in the luminance @ the left side )), but they are in some situations in middle of the ROI and sometimes for a long time and so the whole thing became painfull to watch before.
Now the AQ does good with every of those situation (very balanced, and it still enhances the Details even at such insane low bitrates) and don't forget PSY is not about SSIM or PSNR it is about the most efficient Visual Perception @ best even at such low bitrates you could put it on a floppy ;) (actually im not far away from putting this HD 400 frame on a floppy ;))

[DeathTheSheep]

I agree with you CruNcher in that it produces considerably better visual quality on live footage. But this seems to be at the direct expense of quality in other sources.
If that is desirable, we can explicitly add such a constant, and still benefit from the higher arithmetic precision.
Yes, I think that this is a good idea with much possible benefit. Perhaps this constant can be made user-adjustable as well, for even further potential gain.

It's somewhat counter-intuitive to me that the recently instated increase in QP differences would yield such significantly lower quality (and bitrate) on such a flat scene with very low contrast.

[Razorholt]

So, is there a significant vidual improvement between 0.46 and 0.47 @ low rates ?

[CruNcher]

@Razorholt
should be yes :)

http://mirror05.x264.nl/CruNcher/force.php?file=./noaq-crf40.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./newaq-aku-crf40.mkv (--aq-strength 0.1 --aq-sensitivity 11)

[ditche]

I think they´re taken from the megui CRF profile wich has --subme 1
Yeah, I tried to edit my post (for subme 5) but the forum was down...

[Morte66]

Therefore, sensitivity isn't needed, because all that matters is that two videos at the same AQ strength and CRF are "the same quality."

So would AQ-strength remain as a parameter users can override?

Putting it another way: if AQ shifts the definition of "quality" from regular encoding to something more concerned with smooth/dark areas, can users choose how far they want to shift it?

[Dark Shikari]

So would AQ-strength remain as a parameter users can override?

Putting it another way: if AQ shifts the definition of "quality" from regular encoding to something more concerned with smooth/dark areas, can users choose how far they want to shift it?Yes. AQ strength will stay.

[Morte66]

Yes. AQ strength will stay.

Phew. Thanks, that's what I was missing. I got so caught up in testing sensitivity that I forgot that was there. Sorry for the distraction.

[CruNcher]

No AQ CRF 29

x264 [info]: SSIM Mean Y:0.9797418
x264 [info]: PSNR Mean Y:45.548 U:45.836 V:49.471 Avg:46.021 Global:45.525 kb/s:
2972.16

encoded 9336 frames, 7.90 fps, 2975.00 kb/s


New AQ CRF 29 (--aq-strength 1.0 --aq-sensitivity 11)

x264 [info]: SSIM Mean Y:0.9791818
x264 [info]: PSNR Mean Y:43.834 U:45.212 V:47.979 Avg:44.493 Global:43.912 kb/s:
5078.05

encoded 9336 frames, 7.57 fps, 5080.97 kb/s

That's way off :( might have to be lower then --aq-sensitivity 10 to reach the same size

[ToS_Maverick]

this seems to be a very strange video... CRF29 with SSIM 0.98
my guess would be sens 6 or 7 for the same bitrate

i don't get 0.98 even at CRF18 with BlackPearl...

[Atak_Snajpera]

I found something weird in 0.47.

x264 settings
--crf 20 --level 4.1 --sar 1:1 --filter 0,0 --aq-strength 1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output

1280x544
0.45 version gave size 5.12 MB , NOAQ gave me 4.63 but 0.47 gives me 19.4 MB !!!!

source
http://rapidshare.com/files/59949596/Logo.zip.html

[nurbs]

@Atak_Snajpera:
Filesize explosion and implosion depending on aq-sensitivity happened to me with both 0.45 and 0.47. Maybe you could try to lower the sensitivity to somewhere around 10 and see if that helps.

Also while I don't know if this is important I made a graph with the bitrate for a sample clip at a given crf and sensitivity with varying strenghts.

http://img246.imageshack.us/img246/3500/strvariation1qy5.th.png (http://img246.imageshack.us/my.php?image=strvariation1qy5.png)

[Atak_Snajpera]

Maybe you could try to lower the sensitivity to somewhere around 10 and see if that helps.

but I want to use automatic sensitivity.

[nurbs]

If I understand the first post correctly the default sensitivity is now 20. What you want is 0.

[Atak_Snajpera]

ok thanks. Sensitivity 12 now gives 4.11 MB compared to 4.63 MB. (0 = 5.12 MB)

default sensitivity is now 20
Default value is way to high.

[Jawed]

If qcomp isn't at default 0.6 (I presume that's still the default) will VAQ still deliver the same behaviour?

Does anyone care about qcomp?

It strikes me that qcomp could become more useful (or more meaningful) with VAQ than with old AQ and I'm wondering if qcomp might even be a good way of tweaking VAQ, particularly as VAQ apparently isn't the wild horse that AQ seems to be.

Jawed

[Morte66]

DVD...

Deadwood season 3 episode 7, cleaned up, SelectEvery(1000,50):
crf 16 no AQ: 17.4MB
crf 16 VAQ strength 1.0 sensitivity 8: 15.3MB
crf 16 VAQ strength 1.0 sensitivity 9: 17.8MB

Battlestar Galactica Razor, cleaned up, SelectEvery(1000,50):
crf 16 no AQ: 96.3MB
crf 16 VAQ strength 1.0 sensitivity 8: 75.6MB
crf 16 VAQ strength 1.0 sensitivity 9: 93.5MB
crf 16 VAQ strength 1.0 sensitivity 10: 110MB

Trois Couleus Blanc, cleaned up, SelectEvery(1000,50):
crf 16 no AQ: 49.2MB
crf 16 VAQ strength 1.0 sensitivity 8: 42.3
crf 16 VAQ strength 1.0 sensitivity 9: 51.8

Irreversible, cleaned up, all of it:
crf 16 no AQ: 1.11GB
crf 16 VAQ strength 1.0 sensitivity 13: 1.86GB
crf 16 VAQ strength 1.0 sensitivity 10: 1.37GB
crf 16 VAQ strength 1.0 sensitivity 9: 1.18GB


Whole lotta nines...

[Jawed]

DVD...

Deadwood season 3 episode 7, cleaned up, SelectEvery(1000,50):
Battlestar Galactica Razor, cleaned up, SelectEvery(1000,50):
Trois Couleus Blanc, cleaned up, SelectEvery(1000,50):

What am I missing?: those clips are going to have a way way higher proportion of I frames than a "normal" encode, so how is the crf algorithm going to be acting anything like it would when encoding the real film?

Jawed

[Dark Shikari]

What am I missing?: those clips are going to have a way way higher proportion of I frames than a "normal" encode, so how is the crf algorithm going to be acting anything like it would when encoding the real film?

JawedIn this case, that doesn't actually really matter for comparison.

[akupenguin]

A typical film has one scenecut per 50-100 frames. SelectEvery adds an additional one I-frame per 50 frames, i.e. 2-3x the total number of I-frames. That's not enough to significantly affect this test.

[burfadel]

I tried a small clip (24 sec) of 'That 70's show', season 3, that just had Eric sitting on the couch in the basement eating pizza, then Hyde comes in and there's some panning. The background was completely still until Hyde comes through the door. I resized the image to 512x384, quite low I know but it was just for this test. Anyways, I found a problem with the AQ (0.47), especially with a sensitivity of 9 such as what Morte66 used.

The settings I used (apart from AQ), where all the same:
--crf 24 --keyint 450 --ref 5 --bframes 16 --b-pyramid --weightb --b-bias 40 --b-rdo --bime --direct auto --analyse all --8x8dct --subme 7 --me umh --trellis 1 --mixed-refs --progress

With the default AQ sensitivity (that is, not specified), and strength 1.0 the image looked quite good and very close to the original, but slightly larger (685kb compared to 571kb).

At AQ 9, the filesize was a tiny 201kb, and at 8 it was 170kb! This is with using the same settings. The quality at these sensitivities was shocking, both the background and the foreground were extremely blocky, and the slight moving parts of Eric in the first part caused severe artifacting. Even at a CRF of 17 it did not resolve all of this artifacting.

Therefore, in the case of a standard quality encode, the results with low sensitivities are absolutely disgusting!

I wouldn't go any lower than say, 17... even then the filesize was larger than without AQ, and the picture quality in the bright areas was still slightly reduced!

With these tests and a few others I have done, I am not fussed at all with the slightly larger files, its worth it for the quality increase. That said, I would like to suggest one thing in regards to sensitivity.

Having automatic sensitivity seems to be the best as no one sensitivity seems to be good for all situations. What you could have is sensitivity ratings of say, low, medium, and normal, where normal is the default (naturally).

For low and medium settings, you could still have it as auto, but subtract say, 2 for low, and 1 for medium (or something along those lines). It just seems too much of a disadvantage having a fixed sensitivity for every encoding scenario from bright to dark, low res to full HD, almost no motion to high motion.

[Dark Shikari]

I tried a small clip (24 sec) of 'That 70's show', season 3, that just had Eric sitting on the couch in the basement eating pizza, then Hyde comes in and there's some panning. The background was completely still until Hyde comes through the door. I resized the image to 512x384, quite low I know but it was just for this test. Anyways, I found a problem with the AQ (0.47), especially with a sensitivity of 9 such as what Morte66 used.

The settings I used (apart from AQ), where all the same:
--crf 24 --keyint 450 --ref 5 --bframes 16 --b-pyramid --weightb --b-bias 40 --b-rdo --bime --direct auto --analyse all --8x8dct --subme 7 --me umh --trellis 1 --mixed-refs --progress

With the default AQ sensitivity (that is, not specified), and strength 1.0 the image looked quite good and very close to the original, but slightly larger (685kb compared to 571kb).

At AQ 9, the filesize was a tiny 201kb, and at 8 it was 170kb! This is with using the same settings. The quality at these sensitivities was shocking, both the background and the foreground were extremely blocky, and the slight moving parts of Eric in the first part caused severe artifacting. Even at a CRF of 17 it did not resolve all of this artifacting.

Therefore, in the case of a standard quality encode, the results with low sensitivities are absolutely disgusting!

I wouldn't go any lower than say, 17... even then the filesize was larger than without AQ, and the picture quality in the bright areas was still slightly reduced!

With these tests and a few others I have done, I am not fussed at all with the slightly larger files, its worth it for the quality increase. That said, I would like to suggest one thing in regards to sensitivity.

Having automatic sensitivity seems to be the best as no one sensitivity seems to be good for all situations. What you could have is sensitivity ratings of say, low, medium, and normal, where normal is the default (naturally).

For low and medium settings, you could still have it as auto, but subtract say, 2 for low, and 1 for medium (or something along those lines). It just seems too much of a disadvantage having a fixed sensitivity for every encoding scenario from bright to dark, low res to full HD, almost no motion to high motion.I'm thinking of --aq-strength 0.7 --aq-sensitivity 14 as a medium... I would think this should generate a relatively decent encode with ordinary CRF settings and be comparable between different clips.

[burfadel]

Having only a 24 second clip at 512x384 makes for some quick testing!

At --aq-strength 0.7 and --aq-sensitivity 14, the image was still blocky where without it its no problem...! The file size is still smaller than that of with no AQ.

With a senstivity of 16 (strength 0.7), the file size is only 20kb smaller but the area of blocking is still not as good as without AQ... This may not be a problem with 1920x1080 encodes, but for dvd encodes (720x576 aka PAL) the problem is still there!

[CruNcher]

(--aq-strength 1.0 --aq-sensitivity 7)

x264 [info]: SSIM Mean Y:0.9687704
x264 [info]: PSNR Mean Y:41.195 U:43.895 V:46.262 Avg:42.077 Global:41.401 kb/s:
1607.20

encoded 9336 frames, 8.24 fps, 1610.42 kb/s


eh hehe

[Jawed]

A typical film has one scenecut per 50-100 frames. SelectEvery adds an additional one I-frame per 50 frames, i.e. 2-3x the total number of I-frames. That's not enough to significantly affect this test.
Well, it seems to me that an I frame will have a significantly lower quantiser, so the test is being biased towards "low quantisers" where VAQ is going to have relatively little effect.

Anyway, who am I to argue...

Also, why are people posting SSIM and PSNR data when it's the fact that these measures don't match human perception that calls for the VAQ algorithm?

Jawed

[Yoshiyuki Blade]

I've noticed many clips jump significantly in file size when AQ sensitivity gets bumped up a notch. Does AQ strength have anything to do with this?

[CruNcher]

The Size is important here and it's flipping like crazy :P

[lexor]

Having only a 24 second clip at 512x384 makes for some quick testing!

At --aq-strength 0.7 and --aq-sensitivity 14, the image was still blocky where without it its no problem...! The file size is still smaller than that of with no AQ.


Well, since the testing is so quick, how about trying strength 1 and going with sensitivity from say 7 to 20 (maybe step by 2, to limit the number of encodes) and telling us which one works on that particular clip? We really can't tell you what to do to improve the scene since we aren't working with the same material.

[akupenguin]

I've noticed many clips jump significantly in file size when AQ sensitivity gets bumped up a notch. Does AQ strength have anything to do with this?
That's every clip, and it's perfectly predictable and intended. In static sensitivity mode, --aq-sensitivity simply determines a constant that gets added to the QP. Because if we didn't add anything, then VAQ would introduce a huge bias in quality-per-CRF-value, much more than HaaliAQ. Come to think of it, it should be in units of QP; dunno why Dark Shikari didn't do that.

[Yoshiyuki Blade]

That's every clip, and it's perfectly predictable and intended. In static sensitivity mode, --aq-sensitivity simply determines a constant that gets added to the QP. Because if we didn't add anything, then VAQ would introduce a huge bias in quality-per-CRF-value, much more than HaaliAQ. Come to think of it, it should be in units of QP; dunno why Dark Shikari didn't do that.

Ah ok. I used the word "many" for being uncertain of its predictability.

[CruNcher]

(--aq-strength 1.0 --aq-sensitivity 8)

x264 [info]: SSIM Mean Y:0.9721481
x264 [info]: PSNR Mean Y:42.009 U:44.243 V:46.741 Avg:42.817 Global:42.172 kb/s:
2204.62

encoded 9336 frames, 7.83 fps, 2207.73 kb/s

Seems like i endup with a --aq-sensitivity of 9 like Morte66 before me :)

[nurbs]

With my clips I also ended up at --aq-sensitivity 9 with --aq-strength 1.0 to get similar bitrates to pure crf. But the settings don't work well on all clips. If I use them on Clerks 2 for instance I end up with a small really bad looking file.

[Inventive Software]

@Dark_Shikari, akupenguin: Can you leave the sensitivity option in for those "unusual" sources if the patch gets committed, but conceal it as an "advanced" option, i.e only list it in --longhelp?

[akupenguin]

No. Sensitivity is a property of the algorithm, it's useless as an option. If your source needs more bitrate, lower the value of CRF.

[CruNcher]

yep i would also say so, because it's really like sitting infront of a Roulette Table makeing your bets this way better to use CRF ;)
Reja ne va plue

9 Black wins

x264 [info]: SSIM Mean Y:0.9748461
x264 [info]: PSNR Mean Y:42.691 U:44.574 V:47.179 Avg:43.439 Global:42.820 kb/s:
2989.76

encoded 9336 frames, 8.71 fps, 2992.77 kb/s

So for me this would mean most probably --aq-sensitivity 20 and a CRF somewhere @ 32 i guess instead of currently without AQ 29 :), the result above also looks far from Optimal compared to the No AQ (hard edges, looks oversharped in motion, tough fixes the problematic scenes but the Detailed Scenes now look horrible)

[Dark Shikari]

Especially since moving sensitivity is equivalent to changing CRF :)

[Inventive Software]

Damn... who knew AQ could be so complicated eh? :D