Variance AQ Megathread (AQ v0.48 update--defaults changed)

[CruNcher]

i don't need to swap between you can allready see from the first sight less ringing :) that's amazing :D

New AQ
- less ringing
- less banding
- more details

:D that's 3 for one and with very little speed loss (and only smal complexity disadvantage) :)

yeah dark --aq-strength 1.5 enhanced it i go higher now :)

final ratefactor: 24.74 <- before that where somewhere @ 30 :D

but tough i still don't reach that optimum visual quality i reached with the old one @ 0.9
hmm and the higher i go with the new one the more the bitrate decreases and the file gets smaller in the end but still those spots are not enhanced jesus i try a strength of 5.0 now ;)
strength of 5.0 is extreme it shows all the faces basicly smashed up in blocks but the background seems to get ignored completly at least it doesn't enhance the way it did with the old AQ
i think a little more balance to the new AQ and it could be perfect for all situations you just have to find that :), but it seemes easy now comparing the Old AQ with the new AQ and the spots they actually enhance/disenhance mixing those both and voila = Super AQ :D

@Dark
here is the visual result with strength 5.0
http://mirror05.x264.nl/CruNcher/pearl-newdark-5.0.mkv

i didn't found the setting that results in the same visual quality for the whole scene yet as the oldaq did (still searching)
http://forum.doom9.org/showthread.php?p=1088356#post1088356

-aq-strength 2.0 seems to come near the visual result of the old but it's not quiet the same but i think i get closer (at least it gets harder to spot the difference)
maybe i should take the final size as the indicator for it :)

Old AQ --aq-strength 0.9 --aq-sensitivity 15 = 11.273.040 Bytes
New AQ --aq-strength 2.0 = 10.643.143 Bytes

jep it seems to be somewhere bellow the old AQ i come nearer :)
i reached now the same filesize @ --aq-strength 0.3 but the visual result is suboptimal for this scene compared with the oldAQ @ --aq-strenth 0.9 and --aq-sensitivity 15

[desta]

I'm a bit lost now - the higher the strength of the new AQ, the less overall bitrate it needs?!

[CruNcher]

jep that's how it looks to me, im almost sure the problem lies somewhere in the adaptive --aq-sensitivity in the new AQ, because that's also what is crashing the encoding process if you try to set it manualy.

here is the nearest i could get with the new AQ to the Old AQ visualy wise for this testcut (with just adjusting --aq-strength for both)

New AQ (--aq-strength 0.3) SSIM Mean Y:0.9870448 PSNR Mean Y:47.322 U:48.604 V:51.085 Avg:47.950 Global:47.431 final ratefactor: 25.26
http://mirror05.x264.nl/CruNcher/pearl-aq-newdark-0.3.mkv

Old AQ (--aq-strength 0.9) SSIM Mean Y:0.9838171 PSNR Mean Y:45.361 U:47.704 V:49.915 Avg:46.192 Global:45.801 final ratefactor: 30.14
http://mirror05.x264.nl/CruNcher/pearl-aq-olddark-0.9.mkv

Now it's up to the viewer to decide wich gives better results for this scene


I don't giva a shit about SSIM and PSNR for years now doing my visual optimization stuff keep that in mind (because SSIM isn't realizing this visual problems @ all) (im more an Artist then a Mathematician) (im absolutely thriled by HVS optimization and tricking the eye)

[Dark Shikari]

I'm a bit lost now - the higher the strength of the new AQ, the less overall bitrate it needs?!Generally, yes, because it raises quantizers on high-detail areas that don't need as many bits.

[bob0r]

@CruNcher
You deleted all your test files?
As in the last two links above?

[CruNcher]

No their are up they just werent up before i wrote it yes the --aq-strength 5.0 i deleted i wasn't sure if Dark really needs it as he can reproduce it but maybe it's interesting for the rest to watch it that's why i uploaded it again ;)

[Dark Shikari]

No their are up they just werent up before i wrote it yes the --aq-strength 5.0 i deleted i wasn't sure if Dark really needs it as he can reproduce it but maybe it's interesting for the rest to watch it that's why i uploaded it again ;)Strength 5.0 is retarded since it allows QP adjustments of up to 25 in each direction :p

[CruNcher]

Dark if i see those Metric results did you really blindly optimized for SSIM ?
I think above is the best example to showof where such Optimizations end and where you should better go the HVS (Psy Optimization way) the Ateme guys really understood that fast after i did my first beta tests of their encoder back then and really did some great HVS research since then (in the whole area Picture Sharpness, Perfect Masking and some more clever things) :)

[Dark Shikari]

Dark if i see those Metric results did you really blindly optimized for SSIM ?The reason SSIM rises is simple: SSIM is directly related to the variance of a block. One of its main advantages over PSNR is that it measures distortion relative to the detail already present in a block--which makes perfect sense visually.

My AQ has the exact same approach, and for the same reasons; X amount of lossiness in a low detail block looks far worse than X amount of lossiness in a high detail block. Therefore, low-detail blocks should have much less lossiness than high-detail blocks.

I actually learned that SSIM worked this way slightly after I wrote my first version of this AQ, but the results are not at all surprising given what the AQ does. It doesn't specifically optimize for SSIM, but by its very nature it should increase SSIM.

[desta]

Generally, yes, because it raises quantizers on high-detail areas that don't need as many bits.
So to preserve detail evenly, it's best to use moderate strength and/or lower sensitivity?

[Dark Shikari]

So to preserve detail evenly, it's best to use moderate strength and/or lower sensitivity?Sensitivity should be kept at automatic whenever possible.

Obviously too high strength will ruin quality.

The trick is that with no AQ, detail is kept far better in high-detail areas than in low detail. This is because of how quantization works. Let's say we have two blocks, each of which consist of a single frequency (for simplicity):

Block 1 frequency: 102.9

Block 2 frequency: 7.4

Let's say our quantizer allows values 0, 5, 10, etc.

Block 1 will be rounded to 105, resulting in an error of about 2%. Block 2 will be rounded to 5, resulting in an error of about 30%. Yet they both used the same quantizer!

This AQ tries to resolve this by forcing Block 1 to use a higher quantizer and Block 2 to use a lower quantizer.

[desta]

Right, yeah I understand. Thanks for the explanation.

It threw me a bit when CruNcher showed the stronger AQ using less bits, but I see now it's pretty much the same principle as your original AQ - just that pushing the AQ strength too much will tip the results in the opposite direction.

[CruNcher]

Might be Dark but that doesn't change the fact that your new AQ Visualy isn't @ the optimum as the Old one wich renders especialy the very problematic background edges that they eye imidiatly realizes (in motion even more) better.
This is even clearly viewable with a correct calibrated color and brightness representation as how CoreAVC shows it and with a bad calibration (like for sure many Windows users have them) it even looks more worse with your new aproach compared to your old aproach, you really should think about it.
Im not sure if more details are worth it to have such scenes apearing that way to the eye and the viewer get distracted and the illusion of a very clean source is gone in that moment (talking especialy for very low bitrates here).

[Dark Shikari]

Might be Dark but that doesn't change the fact that your new AQ Visualy isn't @ the optimum as the Old oneI've found it to be much better, personally; but perhaps you're looking for an AQ with a stronger QP-lowering effect at very low variances? This would act more like Haali's AQ, and have a much stronger effect on dark backgrounds.

[CruNcher]

Dark im now conducting some more improvements stuff on this source project im gonna todo 2 encodes of the complete 3 hour with your AQ and the old one @ the given settings and then do a subjective compare of most of the scenes and tell you the final results, first im doing a extreme cut of what you just saw with many different scenes put together (also used that to evaluate Atemes quality) and show you those results with both AQs and my Visual understanding of this (the result will be the Encode that has the lowest amount of scenes where you could realize this is a lossy encode wins) :) later im also doing that with a almost complete dark Movie and see how good each of those AQs does in each situation, definately will take some time.

[DeathTheSheep]

This RCRD is revolutionary... (and slow, of course, but the slower the revolution, the more exquisite the results).

Looks like it's ignoring/washing out complex detail that's only shown for a split second, keeping all the goodies that persist in the following frames. This reminds me of some beta vp6 builds of yore.

[Dark Shikari]

Dark im now conducting some more improvements stuff on this source project im gonna todo 2 encodes of the complete 3 hour with your AQ and the old one @ the given settings and then do a subjective compare of most of the scenes and tell you the final results, first im doing a extreme cut of what you just saw with many different scenes put together (also used that to evaluate Atemes quality) and show you those results with both AQs and my Visual understanding of this :)How about you encode using a source that isn't atrocious?

And try using a sane bitrate? :p

Encoding with unrealistic settings is not a way to test an AQ.

[DeathTheSheep]

Haha, maybe unrealistic settings on an atrocious source brings out the best in an unrealistic, atrocious AQ? :)

[CruNcher]

Dark you know the EBU (Broadcast) test sequence ? my stuff isn't far away from that testcut just that i completly use hollywood (film) stuff.
And we all know that X264 still has problems just run ParkRun throug it with low bitrate and you see there could be alot still done in Psy Optimizations for X264

[Dark Shikari]

Dark you know the EBU (Broadcast) test sequence ? my stuff isn't far away from that testcut just that i completly use hollywood (film) stuff.
And we all know that X264 still has problems just run ParkRun throug it with low bitrate and you see there could be alot still done in Psy Optimizations for X264Given what my AQ does, I suspect it would give extremely good results for ParkRun...

[DeathTheSheep]

Are there any plans for improving the quality algorithm further? You mentioned lambda support and such being removed "for the time being" since they "screw up" B frames on high quantizers or whatnot, but are there plans to pump the algo to the next level soon enough?

[Dark Shikari]

Are there any plans for improving the quality algorithm further? You mentioned lambda support and such being removed "for the time being" since they "screw up" B frames on high quantizers or whatnot, but are there plans to pump the algo to the next level soon enough?At this point the only real possible benefits are:

1) Adding the ability to drop QP even further in extremely flat areas, in addition to the current method.

2) Trellising the QP_deltas to save bits.

3) Lambda modification to save bits (will be a hell of an annoyance to do, and I'm not sure how good an idea it is). Lambda modification is better in Xvid or similar, where you can't move quantizers around as effectively.

At this point, for the most part, I think this AQ is ready for primetime; I have yet to see it have a negative effect on anything but a cartoon source at low bitrates. The positive effects are of course huge.

[DeathTheSheep]

Mm, good stuff. I'm curious as to the origin of this statement: "Its not particularly good at cartoons; I wouldn't use it on anime/cartoons," especially in light of its groundbreaking success on my anime test clips at high QP, and whether the improvements you mention have the potential to remedy what problems may remain in this regard.

[Danisan]

Mm, good stuff. I'm curious as to the origin of this statement: "Its not particularly good at cartoons; I wouldn't use it on anime/cartoons," especially in light of its groundbreaking success on my anime test clips at high QP, and whether the improvements you mention have the potential to remedy what problems may remain in this regard.

Do you have any comparison pictures for the anime tests you've made? :thanks:

[CruNcher]

Dark whats about the --aq-sensitivity crash in your latest build ?

[Dark Shikari]

Dark whats about the --aq-sensitivity crash in your latest build ?What crash? If someone reported a bug, I missed it...

[ditche]

Yep, I have a crash with v0.42, there's no encoding...

-[Information] Log for job1 (video, video test.avs -> video test new aq.mp4)
--[Information] [17/01/2008 20:26:02] Started handling job
--[Information] [17/01/2008 20:26:02] Preprocessing
--[NoImage] Job commandline: "C:\Program Files\megui\x264.exe" --qp 23 --ref 10 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --bime --weightb --direct auto --filter -2,-1 --trellis 2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads 3 --thread-input --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --output "D:\VIDEO\rip\video test new aq.mp4" "D:\VIDEO\rip\video test.avs" --aq-strength 0.6
--[Information] [17/01/2008 20:26:02] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
--[Information] [17/01/2008 20:26:02] Job completed

:helpful::)

[Dark Shikari]

Found one crash bug. Note to self: you cannot say if(floating point value == 0) before calling x264_cpu_restore(). Updated builds/patch.

[CruNcher]

Wow this really almost fliped me out of the chair :D

No AQ
http://mirror05.x264.nl/CruNcher/parkrun-noaq.mkv

New AQ
http://mirror05.x264.nl/CruNcher/parkrun-newaq-1.0.mkv

Old AQ
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-0.9.mkv
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-1.0.mkv

[Dark Shikari]

Wow this really almost fliped me out of the chair :D

No AQ
http://mirror05.x264.nl/CruNcher/parkrun-noaq.mkv

New AQ
http://mirror05.x264.nl/CruNcher/parkrun-newaq-1.0.mkv

Old AQ
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-0.9.mkv
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-1.0.mkvHoly crap! I thought it was just a ratecontrol issue until I realized the I-frames were the same size between the three encodes... holy shit!

The difference in some of the later I-frames between those is unbelievable.

[Dark Shikari]

>>>WOW<<<

http://i8.tinypic.com/6pocmcy.png

http://i2.tinypic.com/87k6ipf.png

:eek::eek::eek::eek::eek::eek::eek::eek:

I didn't realize any AQ could be this effective. Wow.

(both I-frames, same size)

[Romario]

Dark Shikari,what about Vista users, especially Vista 64 users?

Do you have a plan to compile 64-bit build?

[Dark Shikari]

Dark Shikari,what about Vista users, especially Vista 64 users?

Do you have a plan to compile 64-bit build?I don't have a 64-bit OS, so I can't make a 64-bit build--but the patch is available for someone else to.

Also, here's an animated GIF of the difference, since its so shocking:

http://i18.tinypic.com/82u8c9j.gif

[CruNcher]

just the pulseing from the closed gop that needs to be workedaround visualy, then it would be perfect :)

[Snowknight26]

The I-frames really detract from the visual asthetics when watching those samples.
Pretty impressive nonetheless.

[Dark Shikari]

The I-frames really detract from the visual asthetics when watching those samples.The main problem is that Cruncher used 1-pass ABR. The second problem being his crappy settings :p

[Razorholt]

@Darky: what build did you use?

[Dark Shikari]

Apparently (and not surprisingly) people want to see comparisons of P-frames in properly encoded clips, rather than Cruncher's subme1 atrocities, so I will post a real comparison in a bit.

[CruNcher]

i know Dark but it's realtime that way on my 2.8 GHZ Dualcore machine :D sure with subme 5 this would look much better and useing b-frames and more ref and all the stuff but complexity would go up and speed down and that's not the way i balance stuff my goal is also another one then best compression existing ;)

[DeathTheSheep]

Is 1.0 now the recommended strength? Everyone seems to scream 1.0 is the best, but I see no evidence. Did you do your tests with 1.0, DS?

[Dark Shikari]

Is 1.0 now the recommended strength? Everyone seems to scream 1.0 is the best, but I see no evidence. Did you do your tests with 1.0, DS?Yeah, I've been using 1.0.

1.0 = max QP adjustment of +/- 5

[Atak_Snajpera]

PS3 profile 2-pass 6144 kbps

frame 370
no AQ
http://img174.imageshack.us/img174/5076/noaquq8.th.jpg (http://img174.imageshack.us/my.php?image=noaquq8.jpg)

AQ1.0
http://img156.imageshack.us/img156/5189/aq1yu6.th.jpg (http://img156.imageshack.us/my.php?image=aq1yu6.jpg)

[Dark Shikari]

:eek:

I just finished my own test encodes, 5 megabits at 25 frames per second, maxed out settings.

AQ got a 37% SSIM boost.

The quality difference is mindblowing--the AQ, even in motion, looks nearly transparent, while the non-AQ looks atrocious.

Linkage to download both clips (AQ, no AQ) (http://www.mediafire.com/?62dzmttn0n4).

[Inventive Software]

And the award for next SVN entry goes to:


Dark Shikari! :D

[DeathTheSheep]

You know, I'd like a link to the clips (preferably the anime) you claim didn't benefit visually from the AQ.
I want to test if any settings combo can give a better effect, then see if I can generalize these to other problem anime samples.

[Dark Shikari]

You know, I'd like a link to the clips (preferably the anime) you claim didn't benefit visually from the AQ.
I want to test if any settings combo can give a better effect, then see if I can generalize these to other problem anime samples.I haven't tested a lot on cartoons--once I fix interlacing to work correctly with AQ, I'll go back to them.

[CruNcher]

I haven't tested a lot on cartoons--once I fix interlacing to work correctly with AQ, I'll go back to them.

Arghh don't deoptimize X264 for Anime stuff at least don't unbalance the Real Footage behaveiour that would be a disaster.

This i-Frame pulseing should be looked into even @ 6 Mbits it's visible and even with heavy encoding settings (but i think that problem is very deep inside X264 since the first days and has todo with the partitioning itself won't be easy to fix it for sure (make it more consistent), hmm playing arround with the deadzone reduces it a little but it's still their)

[Dark Shikari]

Interlacing's stream corruption appears to not be a bug in AQ bug actually a long-standing bug in x264--the CABAC context for interlaced QP_deltas seems to be wrong in interlaced mode, since it only appears with CABAC, and appears even if I entirely remove AQ and do nothing but randomely apply quantizers to different blocks.

[DeathTheSheep]

Crunch: Nobody's de-optimizing anything.

DS: If interlacing is causing you trouble, don't use it. It's more visually difficult to spot differences in screencaps anyway with the presence of two fields. Instead (just for testing purposes, if nothing else), deinterlace the source well first; it might even be advisable to discard one field entirely and interpolate the width (via spline36) to maintain the AR--this way, you'll have a great low-res, low-br test clip on your hands.

[Dark Shikari]

Crunch: Nobody's de-optimizing anything.

DS: If interlacing is causing you trouble, don't use it. It's more visually difficult to spot differences in screencaps anyway with the presence of two fields. Instead (just for testing purposes, if nothing else), deinterlace the source well first; it might even be advisable to discard one field entirely and interpolate the width (via spline36) to maintain the AR--this way, you'll have a great low-res, low-br test clip on your hands.I don't have this choice. Dakaz insists that interlacing and AQ work together :p