Variance AQ Megathread (AQ v0.48 update--defaults changed) [Archive] - Page 4

burfadel

16th January 2008, 11:18

This patch definately makes a better quality image, without sacrificing bitrate, at least in crf mode :) Its to the point where I can say it should be enabled by default, and with a strength of 1.0 (100 percent? :) ) by default only because it works so well, and unlike the old patch such a high AQ seems to be better. Except of course, for the bug that cruncher has mentioned which I haven't seen! Hopefully this is a real step towards including it in the svn!

acrespo

16th January 2008, 12:22

That's because... you need pthreadGC2.dll?

You can get it here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

Why the new version needs this file? I notice that the file size decrease too. Is that because you remove this library inside the .EXE?

Atak_Snajpera

16th January 2008, 13:28

(look @ the top left when the calendar pages are turned)

Could you make a screenshot and mark spot because I can't see it.
BTW 3MBps for 1440x1080... You are crazy :)

Good advice use mediafire.com instead of Rapidshare.
(I've just reached the limit for user)

Dark Shikari

16th January 2008, 15:11

Why the new version needs this file? I notice that the file size decrease too. Is that because you remove this library inside the .EXE?I'm too lazy to fix it for now ;_;

Also, I fixed the link.

Sharktooth

16th January 2008, 17:53

nice one DS! really... :)

ToS_Maverick

16th January 2008, 18:32

Dark Shikari, could you implement the additional switches you suggested, in the next release? i'd really like to play around with a fully enabled AQ (also on B frames).

Dark Shikari

16th January 2008, 18:35

Dark Shikari, could you implement the additional switches you suggested, in the next release? i'd really like to play around with a fully enabled AQ (also on B frames).This one is fully enabled on B-frames :)

I completely took out all frame-specific code and lambda-based code for the meantime.

DeathTheSheep

16th January 2008, 19:12

Even so, with r681, it yields an average of 33.4% bitrate-adjusted SSIM increase on my anime test clips (as compared to r680) in conjunction with hadamard, me-prepass, and 681's subme7 tweak. (Baseline profile, relatively high quantizers too).

Hmm, I wonder what all of these quality patches have in common? Like common place of origin, etc? Hmmm, nope, nothing. :p

Dark Shikari

16th January 2008, 19:19

Even so, with r681, it yields an average of 33.4% bitrate-adjusted SSIM increase on my anime test clips:eek:

Dark Shikari

16th January 2008, 19:27

Here is the Bug with --aq-strength 1.0 (also happens for --aq-strength 0.9 if --no-dct-decimate is used), excuse me if this problem allready has been encountered before i didn't read the whole thread yet (and maybe it's not even happening with the new patch anymore)
http://rapidshare.com/files/84200572/testseq-pearl-darkaq-bug.mkv (look @ the top left when the calendar pages are turned)
Something is HORRIBLY screwed up with the quantizers in that frame.

What commandline are you using, and where can I get your source? There seems to be something like a complete AQ reversal (!?!?!) It almost looks like the AQ sensitivity for that frame is negative (?!!!) I'm guessing it has something to do with the dark/light areas and an overflow of some sort in the calculations.

DeathTheSheep

16th January 2008, 19:34

...and the more modest 17.1% without hadamard, me-prepass, and subme7 patch (r680).

(This "expectedly unexpected" development only serves to add fuel to the fire in favor of updating the old patches; for some reason, quality increase is almost doubled when all used together! :D).

[edit]By the way, I'm testing in constant quantizer mode only. Are these results at all expected?

Sagekilla

16th January 2008, 19:40

DeathTheSheep

16th January 2008, 19:46

What do you mean? As in the qcomped "crf" instead of "qp"?

It is bitrate adjusted, after all (bitrate delta compared with SSIM delta over 3 qp range works too).

Also, it's fun to note that q29 aq strength 0.5 produces nearly identical filesize (on 2 of my anime test clips) as q30 without aq, but SSIM goes up from [av] 0.9714289 to [av] 0.9747350.

bob0r

16th January 2008, 20:02

@Dark Shikari
Great work, you really solved parts of the Banding Problem with this, im amazed by the results :)

Without Dark Shikaris Magic AQ
http://rapidshare.com/files/84197179/testseq-pearl-nodarkaq.mkv

With Dark Shikaris Magic AQ (HVS Quality is greatly improved)
http://rapidshare.com/files/84197485/testseq-pearl-darkaq.mkv

Get it now, it makes your "darkest" dreams come true ;D

...

Wow what a difference!!!
Truely stunning to see this must improvement... i have mirrored the files: http://files.x264.nl/cruncher/

CruNcher

16th January 2008, 21:01

Thanks bob0r for mirroring, but i might have bad news the new Patch does nothing on this scene, seems the adaptive --aq-sensitivity is failing and even if i set it manualy to 15 and strength to 0.9 or 1.0 nothing changes anymore as with the old patch :(

CMD is

Old patch (x264_aq-brdo.diff) (improves visual percepted quality of this scene massively)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 0.9 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

Old patch Reveresed AQ Bug (x264_aq-brdo.diff) (still improves but shows a problem in one sequence)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

New Patch (No reaction, also non with strength 0.9/1.0 and sensitvitiy 15)
x264-newaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

Dark Shikari

16th January 2008, 21:03

Thanks bob0r for mirroring, but i might have bad news the new Patch does nothing on this scene, seems the adaptive --aq-sensitivity is failing and even if i set it manualy to 15 and strength to 0.9 or 1.0 nothing changes anymore as with the old patch :(

CMD is

Old patch (x264_aq-brdo.diff)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 0.9 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

Old patch Reveresed AQ Bug (x264_aq-brdo.diff)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

New Patch (No reaction, also non with strength 0.9/1.0 and sensitvitiy 15)
x264-newaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv
Can you get me the source for this so I can find and fix the bug?

Sharktooth

16th January 2008, 21:04

bobor, i hope you've messed up the file names on your mirror, coz the nodarkaq looks better and less blocky than the darkaq sequence...

Dark Shikari

16th January 2008, 21:06

bobor, i hope you've messed up the file names on your mirror, coz the nodarkaq looks better and less blocky than the darkaq sequence...That's probably due to the bug in my AQ that seems to show its ugly face on certain parts of this clip.

Sharktooth

16th January 2008, 21:18

no, i saw blockiness in bright areas.

DeathTheSheep

16th January 2008, 21:20

Funny I noticed similar... but at my bitrates, the increase in quality of everything else more than makes up for it.

Dark Shikari

16th January 2008, 21:32

no, i saw blockiness in bright areas.Uh, that's part of the bug. My AQ has absolutely zero, zip, zilch to do with "brightness" or "darkness."

Sharktooth

16th January 2008, 21:39

but the blockiness was accentuated from the nodarkaq encode... so IMHO you should watch out from stealing bits from those areas unless you want them to look worse than without AQ.

Dark Shikari

16th January 2008, 21:43

but the blockiness was accentuated from the nodarkaq encode... so IMHO you should watch out from stealing bits from those areas unless you want them to look worse than without AQ.The entire bug is that AQ is reversed in certain frames. This means its INCREASING blockiness in blocky areas--doing the opposite of what it should. How much do I have to explain this? Look at the quantizer distribution yourself--its obviously completely reversed.

So far, I have been unable to replicate the bug using his encoded source as my source, with a CRF pass.

CruNcher

16th January 2008, 21:45

The percived quality improvement here is in the really dark scenes but even with standard VMR9 and calibrated you can see those problems without Dark Shikaris AQ (especialy LCD). They even become visible in the gradient in the non Dark (Green/Blue) Background with the text (The Rise and fall of our Empire is at stake) you can clearly see how Darks AQ improves this Background Gradient drasticly :). Sure you wont see the stuff in the Dark area before that if you turn down the Brightness a little, but in this Area you would still see this Gradient Problems even if you lower your Brightness. And for me Personaly that (Banding) is more anoying then the little more blocks it introduces mostly in the Fast Action :D, never forget we working lossy here we have to set visual priorities and my is clear in that case ;)

DeathTheSheep

16th January 2008, 21:50

In my case, it seems there were flaws in the original mpeg2, which I misconstrued as problems with the AQ. No worries for me...

What was the exact binary, avs script, decoder, and commandline used to produce the bug? I want to replicate this thing--I can't believe it would favor one system over another.

CruNcher

16th January 2008, 21:52

Indeed the source is bad one of the first Blu-Ray Mpeg-2 encodes (Full of Filmgrain mixed with Encoder Quantization Noise, and Banding couseing problems when cleaning it), so perfect to test extreme cases ;). Ok found a problem useing --sar 16:9 makes this hardware incompatible :( --sar 4:3 works but it's huge now im sure it wasn't intended to be displayed that way, jesus i never gonna understand AR as it's AR is 2.40 it's wrong i think that it's displayed now with --sar 4:3 @ AR 16/9 how crazy is that ;).
Ok so no go for staying hardware compatible and also in the right AR (2.40) onscreen itself, you must include the letterbox i see no other way or im blind ;)

Ok here another one with the old AQ, this time it should be almost 100% Hardware Decoding Compatible (and it's low complexity) :)
http://mirror05.x264.nl/CruNcher/new-olddarkaq-hdc.mkv

Another thing i encountered now is that with VM9 Windowed (at least on my Nvidia 8800 GT) i see every mini block specialy now the problems in the Letterbox area (grey/white flashing because brightness is much higher now seems the old TV Level thing most probably).
With VMR9 Renderless it looks much better (because it's darker and so alot of problems aren't visible (Luma HVS tricks) on the first sight and even hard to spot on the second). Darks AQ improves the rest mostly the visible banding in the bright fire flash gradient, then the background gradient of the Text scene and also in the dark blue gradient background outside :) I also left out --no-fast-pskip, this seems to have overall improved it a little. Viewing this from aprox 75cm-1m away looks really nice :). Only thing im not happy with is the color representation compared to the source it seems wrong much duller it's strange it seems to be no Colormatrix problem as the uncompressed sample FFV1 shows all the color fidelity. As expected it's the way Cyberlinks Decoder does it under VMR9 and even Overlay Mixer the only Decoder doing it right from the start is CoreAVC it shows imidiatly the Full Color Representation as the Source (and by doing that it even gets rid of more of the visual flaws that you would see with other decoders even Mplayer and Videolan show the same dull colors :(

Dark Shikari

17th January 2008, 03:08

Same settings as you cruncher--and absolutely no problem like I encountered in your stream. QPs came out exactly as I expected:

http://i18.tinypic.com/8dvvvqq.png

Also, I vastly optimized the AQ, removing the need for duplicate cache_loads and SADs and SSDs. The main result is that the CPU cost of AQ is cut drastically.

CruNcher

17th January 2008, 03:16

Dark Shikari

17th January 2008, 03:21

Dark did you used the old AQ patch or the new one ? the new one didn't worked @ all for me and the old one showed that bug @ --aq-strength 1.0, strange this is.
This here is the one i mean http://files.x264.nl/force.php?file=./dark/x264_aq-brdo.diff that one i patched on 720 and got those --aq-strength 1.0 results.That's an ancient and bugged AQ patch.

Here, I'll give you a little something that will make you all warm and fuzzy inside: a combined patch of RDRC and AQ... AQ 0.43, not 0.42! Nice and fast... until you turn on RCRD ;)

Patch (http://pastebin.com/f227baccb)

Build (http://www.mediafire.com/?bcd2dd5ygtj)

CruNcher

17th January 2008, 03:28

Nice and where is Lookahead ? a complete RDRC but no Lookahead for ABR geez ;)

Sharktooth

17th January 2008, 03:31

could you encode the clip again with this one? just AQ, no RDRC...

Dark Shikari

17th January 2008, 03:35

could you encode the clip again with this one? just AQ, no RDRC...That's what I just did--just AQ, not RCRD.

RCRD is unrelated and I'm not using it for most of the AQ work; its just something some people were requesting.

CruNcher

17th January 2008, 03:55

Dark Shikari

17th January 2008, 03:57

Dark ehh i tried your new build (not patched it your .exe) but --aq-strength 1.0 does nothing compared to the old (that you call buggy AQ) in the key spots im talked about it just ignores them and when doing --aq-strength 1.0 --aq-sensitivity 15 your .exe is crashing the encode right after the 1 frame :(Well it works just fine here... :rolleyes: same source, same commandlines...

Can you come on Freenode and we'll try to resolve this?

mahsah

17th January 2008, 03:58

Wow, this is great! I tried this on a test clip of Futurama, and while the results are not very noticable (and I was doing two pass encoding), there are fewer blocks with AQ on. Also the png filesize is lower, which I guess means it is more compressable.

NoAQ:
http://img47.imageshack.us/my.php?image=noaqtq1.png

AQ:
http://img337.imageshack.us/img337/5960/aqgx4.png

command line:

C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 1350 --stats "hfyu_go.stats" --ref 8 --mixed-refs --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 12 --threads auto --thread-input --sar 243:200 --progress --no-psnr --no-ssim --output "hfyu_go.mkv" "hfyu_go.avs"

CruNcher

17th January 2008, 05:03

Dark wich commandline i pasted 3 :)

so ok i did a test again with the old aq and your new aq both same settings --aq-streangth 0.9 this time as 1.0 is extreme buggy as you said no --aq-sensitivity was used for both (not correct the old one has --aq-sensitivity 15 as default) and here is now a visual comparsion analysis of they key spots i talked about and their subjectivly percepted visual differences.

Not many guys for sure have such trained eyes as me and maybe that's the problem why i realize such scenes so fast and they really hurt me (please note it's not about the individual frames here it's about the scenes they mark and the time they stay in focus and are perceptable to the eye) :(

New AQ
http://s3.directupload.net/images/080117/llnhtezh.png
http://s1.directupload.net/images/080117/tu7hg58j.png
http://s2.directupload.net/images/080117/gueurani.png
http://s1.directupload.net/images/080117/ft79axio.png

Old AQ
http://s1.directupload.net/images/080117/gh3325fq.png
http://s2.directupload.net/images/080117/2dp7ut4p.png
http://s5.directupload.net/images/080117/dvit3x7j.png
http://s5.directupload.net/images/080117/hgc6ar9j.png

These are the Key Areas for me that decide what is good and what is bad and how the whole sequence is rated by the viewer more rated by me as Sharktooth for example doesn't like the blocks in the luminant areas but those are so fast anyway this really hurts my eyes about your new AQ compared to the old :)
I have overexergerated the Decoding brightness Problems (useing VMR9 Windowed and ffdshow is the best combination to reach that hehe) to show those Key areas this is ofcourse not the same what you would see with a good calibration on screen and the best to rate this on Windows seems CoreAVC (Perfect Rendering Quality right color levels right brigthness levels everything perfect) :)

These Key Spots have to be seen in Motion (especialy the color gradients) and then it's clear that the new AQ is not for everyone :)

Dark Shikari

17th January 2008, 05:11

It'd be far more useful if you showed me what you thought was a "good" quantizer distribution and what you thought was a "bad" one.

I can't tell crap from those images without seeing the quantizers.

Sagekilla

17th January 2008, 05:22

Dark Shikari, does your latest linked build feature your RCRD as well? I downloaded what I thought had this feature and I found it produced bit-for-bit identical encodes as the current r720 build, with the exception of r720 being marginally faster (6.88 vs 6.47 fps).

Nevermind, I forgot to enable the switches to begin with. Are there any suggestions for --rcrd-lambda, like a starting point for example? :)

Dark Shikari

17th January 2008, 05:24

For lambda, note that lambda does not correspond to a specific CRF--the bitrate result of lambda will vary based on your window size, source, and AQ settings.

As I mentioned in the OP, lambda 500 will give you something roughly similar to CRF 24-26.

CruNcher

17th January 2008, 05:26

@Dark
Sorry gonna upload the results from both showing the quantizers and some results showing both rendered by CoreAVC in the key spots (also interesting) :)

This is the nices Representation of the Encode (wich comes closest to the source in color fidelity and brightness)

CoreAVC New AQ
http://s3.directupload.net/images/080117/g7vcy5ab.png
http://s3.directupload.net/images/080117/jtx66peq.png
http://s3.directupload.net/images/080117/o3jlae5c.png

CoreAVC Old AQ
http://s5.directupload.net/images/080117/wz44kip6.png
http://s3.directupload.net/images/080117/y73468f4.png
http://s4.directupload.net/images/080117/pgqe79ch.png

Even with Hardware Decoding it's not possible to reach this Quality at least not with MPCHC + Cyberlink and Nvidia on VMR9 or Overlay the colors will allways be duller then the actuall source was :(

Sagekilla

17th January 2008, 05:38

@CruNcher: Sounds like you're not properly adjusted for PC levels.. If you're not, then video will always look dull and washed out (black will be gray, whites won't be as white, etc) I forgot which setting, but it involves YUV conversion in hardware. Don't do that, that's what causes it to be set to TV scale instead of PC scale.

On a side note, is there any possibility of speed optimizations for RCRD or is it inherently that slow?

Dark Shikari

17th January 2008, 05:49

On a side note, is there any possibility of speed optimizations for RCRD or is it inherently that slow?I have already made a number of optimizations, but more are possible.

1) Take more shortcuts when doing the lookahead encodes.

2) Take more shortcuts when choosing the QP.

3) The full effect of RDRC can be simulated by analyzing the video on the first pass to figure out, for every case in which data is added to the video stream, how much that data is used in all future frames. This is an extremely difficult problem, but if we could somehow get an answer to that question, we would have an RC method that would get results close to RDRC and absolutely blow away all other video codecs out there. I already have some theories on how to do this, but I'm not sure how good an approximation they would be.

Sagekilla

17th January 2008, 06:16

Interesting work.. Any chance of any portion of this being added to the RC for crf mode? Or is this something that'll only work for 2-pass?

Dark Shikari

17th January 2008, 06:18

Interesting work.. Any chance of any portion of this being added to the RC for crf mode? Or is this something that'll only work for 2-pass?The speed cost is extremely heavy; the only thing I could think of for general use would be to use it to make I-frame/scenecut quantizer decisions or something.

It doesn't need to be twopass--that's just to make the coding easier (to avoid having to decide frametypes).

CruNcher

17th January 2008, 06:24

Here dark as you can see in the gradient areas the Quantization is in the new AQ to high or uneven distributed

New AQ Quantization Distribution
http://s4.directupload.net/images/080117/qiil5rwn.png
http://s6.directupload.net/images/080117/92elpdmw.png
http://s4.directupload.net/images/080117/dj435r2x.png
http://s3.directupload.net/images/080117/5xdn8u82.png

Old AQ Quantization Distribution
http://s1.directupload.net/images/080117/kgtd8olt.png
http://s3.directupload.net/images/080117/p86p3bdv.png
http://s3.directupload.net/images/080117/st6beg4c.png
http://s3.directupload.net/images/080117/9ar89obb.png

Maybe a Quantization of 15-20 (oldaq is most of the time really low like 15-17 over several frames) will help in such areas :) 24-28 seems to be to much and then it seems the result is very visible banding.

Here are also both clips for you to analyze
http://mirror05.x264.nl/CruNcher/pearl-aq-newdark.mkv
http://mirror05.x264.nl/CruNcher/new-olddarkaq-hdc.mkv

Dark Shikari

17th January 2008, 06:27

The explanation there is that the new AQ (when on automatic sensitivity) tries to avoid redistributing bits among frames, while older AQs didn't.

I would suggest you try a higher AQ strength, also, as the new AQ uses a different formula.

Atak_Snajpera

17th January 2008, 13:31

@CruNcher

...and don't forget to use RGB32 instead of YV12 !

CruNcher

17th January 2008, 15:58

@Sagekilla and Atak
I used the high quality YV12->RGB32 conversion but enableing/disableing doesn't anything to the picture quality with fffdshow and vmr9 windowed/renderless.
Don't know tough what Sagekilla means with the YUV conversion in hardware, never saw such an option in MPCHC Cyberlink or the VMR9 renderer.
I think the problem here arises once again from the Nvidia driver and somehow the DVI output is set on TV Level their is a registry fix for that, gonna try if this improves anything :).
It's strange that CoreAVC does compensate this somehow i think it's more then just that(also somehting with color telemetry and the decoder).
Saw a Bt709 option now in ffdshow that's something new and also the Range setting but both didn't improved anything on the visible picture quality.

@Dark Shikari
i try a higer Strength now thx for that info :)