Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Dark Shikari]

A test of the AQ, at 2 megabits per second:

Original (no AQ) (http://mirror05.x264.nl/Dark/force.php?file=./Original.mkv)

New AQ (http://mirror05.x264.nl/Dark/force.php?file=./AQTest.mkv)

Notice the massively better grain retention with the AQ.

[Sharktooth]

what settings did you use for those 2 encodings?
also how does it behave at very low quantizers (below 18)?
it seems in the original one, background grain is there on the i frame, but on Bs it gets washed. it does the same with AQ, but it "drops" much less.

[Dark Shikari]

what settings did you use for those 2 encodings?
also how does it behave at very low quantizers (below 18)?
it seems in the original one, background grain is there on the i frame, but on Bs it gets washed. it does the same with AQ, but it "drops" much less.I used 3pass with pretty much maxed settings, with --no-dct-decimate. Trellis 1.

I-frames will obviously have better grain because they have lower quantizers.

I'm testing a 5 megabit encode right now to see behavior at lower quantizers.

Also note that the original video clip isn't the best quality--some blurring and even blocking is actually from the original.

[Gilgamesh83]

This would be some sort of motion-based AQ. I've never seen one, personally.
That's called brightness-based AQ. Elecard supports this, and regular x264 AQ, though not brightness-based, is thresholded by brightness.

Regular x264 AQ finds the flattest parts of the frame (least complex) and gives the more bits. Mine is somewhat similar, but does it using different math and is much more willing to move bits around.

thx for the answer. would be cool if a motion-based aq existed.

[foxyshadis]

Motion-based isn't that interesting. You'd really want some sort of area of visual interest AQ, but that is firmly in the Hard Problem territory. You'd practically need the director or someone intimately familiar with the film highlight the area the eye is focusing on in every frame. Researching is progressing in this every year, though, there's a lot of papers out there if anyone ever wants to take a (quixotic imho) stab at it.

The quantizer, deadzone, and custom matrix are all there to tweak how the codec responds to motion and detail, and generally do a fine job; it's grain that has generated nearly all of the complaints over the years. The lousy performance of h.264 with grain is the main reason VC-1 even exists.

btw, xvid and ffmpeg use method b and call it lumimasking. x264 doesn't need that since it doesn't have the same overquantization problems in dark areas they do, although there's some overlap when dark areas are flat but grainy.

[ToS_Maverick]

@Dark Shikari:
did you find something out during your testing? just curious ;)

[foxyshadis]

Also: It definitely helps reduce blocking of gradients in anime. It can increase bitrate by a pretty inordinate amount in some scenes, though, even with zero visual difference - anime's probably so flat that the algorithm goes a little crazy.

[Valeron]

hi, Dark Shikari, not good news from my anime encode experience.
crf18 same setting, one with ur new AQ strength 1.0 and threshold 0(automatic) enable, the other disable AQ, the AQ enable one looks bad compare to the no AQ encode. And is 27MB larger in size.
If u would like some screen shot, I can post here tomorrow.

[Dark Shikari]

hi, Dark Shikari, not good news from my anime encode experience.
crf18 same setting, one with ur new AQ strength 1.0 and threshold 0(automatic) enable, the other disable AQ, the AQ enable one looks bad compare to the no AQ encode. And is 27MB larger in size.
If u would like some screen shot, I can post here tomorrow."Bad" is sort of a bad term--screenshots are definitely useful to illustrate. Also, comparing to files with different sizes is generally bad, too.

[burfadel]

"Bad" is sort of a bad term--screenshots are definitely useful to illustrate. Also, comparing to files with different sizes is generally bad, too.

Especially since CRF is constant quality, not constant bitrate (which would end up with the same file size). The use of P and B frames, as well as macroblocks would also be different with constant quality and AQ enabled (?), so although the image may have the same CRF, the filesize will end up being different! The filesize could go either way?...

[Dark Shikari]

Especially since CRF is constant quality, not constant bitrate (which would end up with the same file size). The use of P and B frames, as well as macroblocks would also be different with constant quality and AQ enabled (?), so although the image may have the same CRF, the filesize will end up being different! The filesize could go either way?...And of course comparing individual frames is also bad unless the GOPs have the same structure--comparing a B-frame to an I-frame, for example, is just retarded.

Best way to compare is just to run two two-pass encodes, one with AQ and one without, and comparing the result.

[Sharktooth]

i'd suggest to encode at a target bitrate and see if the AQ encode looks better...

[zbutsam]

I read in a post on this topic that this AQ patch could improve the encoding of scenes with grass textures and I had a great clip to test it on. It is a trailer of the film "Kicking and Screaming" which contains a lot of action on football fields.
The clip (originally in 720p) was resized to 560x304 for speed's sake and encoded in MeGUI using the HQ-Fast profile and with a two-pass target bitrate of 700kbits (rather low I know but I wanted to see how the AQ would react with lower bitrates).
As a reference I used the latest x264 build that MeGUI would download (709).
For the AQ x264.exe I added the switches
--aq-strength 1.0 --aq-sensitivity 20.

The results were great:) Whereas the original would struggle with the low bitrate having to blur details on the grass and producing a flat effect, the new AQ managed to preserve much more detail without noticable loss of quality anywhere else.

On a 1000 kbit 2-pass I tried just with the AQ build it retained much more detail and the picture was sharp (however I will have to go back and repeat this last test with the unmodified build to see how that does).

Speed-wise the build with the AQ patch is about 10% slower for me.

I will try to post more tests later but, so far, congratulations :) it seems to be working great

[Sagekilla]

I read in a post on this topic that this AQ patch could improve the encoding of scenes with grass textures and I had a great clip to test it on. It is a trailer of the film "Kicking and Screaming" which contains a lot of action on football fields.
The clip (originally in 720p) was resized to 560x304 for speed's sake and encoded in MeGUI using the HQ-Fast profile and with a two-pass target bitrate of 700kbits (rather low I know but I wanted to see how the AQ would react with lower bitrates).
As a reference I used the latest x264 build that MeGUI would download (709).
For the AQ x264.exe I added the switches
--aq-strength 1.0 --aq-sensitivity 20.

The results were great:) Whereas the original would struggle with the low bitrate having to blur details on the grass and producing a flat effect, the new AQ managed to preserve much more detail without noticable loss of quality anywhere else.

On a 1000 kbit 2-pass I tried just with the AQ build it retained much more detail and the picture was sharp (however I will have to go back and repeat this last test with the unmodified build to see how that does).

Speed-wise the build with the AQ patch is about 10% slower for me.

I will try to post more tests later but, so far, congratulations :) it seems to be working great

700 kbps actually isn't that too low for 560x304. I manage to get around 1.1 mbps (300, surprisingly) to about 2 mbps on most of my encodes @ 864x480. Then again, I do typically enable most settings except for esa. On some of my encodes I've decided to just go with the full 16 refs since they're so slow to begin with because of preprocessing.


In any case, that's very interesting to hear. I'm waiting for the preprocessing to finish on one of my videos before I decide how to tackle it with AQ. Last encode I ran on it, I ended up using an older build of the new AQ and the newer builds seem to be doing an even better job, so I'm redoing it for the probably 8th time now.

[Dark Shikari]

700 kbps actually isn't that too low for 560x304. I manage to get around 1.1 mbps (300, surprisingly) to about 2 mbps on most of my encodes @ 864x480.The main issue that I find, however, is that without my AQ, you need very high bitrates to retain fine background detail, like grass; it simply doesn't put the bits where they need to be. As a result, you end up needing vastly higher bitrates to achieve transparency, even though most of those bits end up wasted.

[Sagekilla]

The main issue that I find, however, is that without my AQ, you need very high bitrates to retain fine background detail, like grass; it simply doesn't put the bits where they need to be. As a result, you end up needing vastly higher bitrates to achieve transparency, even though most of those bits end up wasted.


Neat, all the more for me to be excited about re-encoding 300 for the zillionth time. At the bitrates I said above, I usually find the videos to be mostly transparent (except backgrounds which tend to be slightly blocked but manageable) I just hope the rate control won't get screwed up badly using crf 18. By the way, why did you say that grass tends to be unfairly smeared by x264 again? I always found that a bit of an odd quirk.

[Sharktooth]

metrics do not represent the eyes perception. they're just numbers that represent an average deviation from the source picture. since x264 internal stuff was made to get the best compression keeping high level of metrics, sometimes the codec produce unwanted (visually speaking) results.
however every codec developer is more or less using the same method coz it's easier to compare eventual (metric) improvements and that leads to a faster development. when algos are optimized and the compression gets close to the theoretical maximum, then visual optimizations (psy and other stuff) are introduced to obtain a visually pleasing picture.
in other words, x264 has a very good compression but the picture quality can be improved drastically.

[rhester72]

Since there are no current diffs, any chance we could get the test EXEs compiled with MP4 output support?

Rodney

[Dark Shikari]

Here's a diff... (http://pastebin.com/f54db05b9)

I really need to work on the automatic thresholding and the formula though--there are some cases in which the AQ really doesn't work well. I've been busy lately though--final exams and watching Haruhi.

[Sagittaire]

Well here a well know psy optimisation for noise/grain: for HVS noise in dark area is useless.

1) make pre-process for reduce noise in dark area. Make strong denoising/degraining in dark area (with lumi < 40 for example).

2) Use lower quant in dark area for better HVS quality in dark area. After "dark denoising" dark area will be more compressible. Use high quality (low quantizer) for dark area is really important for TFT screen.

3) Use higher quant in for complex texture. With this HVS AQ the quality for flat area will be really better. Use "dark denoising pre-process" and "spacial complexity AQ" will produce directly better quality for dark area.

[ToS_Maverick]

Dear Dark Shikari, I got a pre-christmas present for you :D

i recorded PotC1 some time ago from HDTV and can now present you, the same sample in 1080p broadcasted at about 6 MBit:
http://www.megaupload.com/de/?d=O262L4JJ

to get the same screen size and picture area, i use this script:
directshowsource("Black.Pearl.Sample HD.mkv",audio=false,fps=25)
trim(82,3703)
ColorMatrix(mode="Rec.709->Rec.601")
crop(0,140,0,-140)
lanczosresize(768,320)

the broadcast isn't perfect, but the very fine detail in the background is preserved very well, which should be good input for your AQ!

have fun :cool:

[kumi]

Good luck with your exams, and Haruhi :) I hope you find time to adjust the automatic thresholding for use with crf mode. I know that you said this isn't like constant bitrate mode and we shouldn't expect filesize parity, but a little more predictability would be real nice. I just finished an encode that came out massively oversized, (2.3GB vs 1.4GB without AQ). Other movies haven't been nearly as bad, though.

Well I guess I should have been doing a prediction pass from the start... stupid me :p

Happy holidays, everyone

[Dark Shikari]

Good luck with your exams, and Haruhi :) I hope you find time to adjust the automatic thresholding for use with crf mode. I know that you said this isn't like constant bitrate mode and we shouldn't expect filesize parity, but a little more predictability would be real nice. I just finished an encode that came out massively oversized, (2.3GB vs 1.4GB without AQ). Other movies haven't been nearly as bad, though.

Well I guess I should have been doing a prediction pass from the start... stupid me :p

Happy holidays, everyoneThe other issue is that edges are really getting screwed up in some cases with my AQ; my algorithm seems to work at its absolute best when there are no sharp edges in the video, and worst when there are plenty--so perhaps I have to deal with edge masking or similar.

[Sagekilla]

Hmm, the latest build (0.3) is rather quirky in your transmagical adaptive sensitivity mode with crf . Strength 1 I got 1.6 mbps on a 720p resized portion of that PotC source. Increasing strength to 2 made the bitrate go down further, to 1.3 mbps! I'm decreasing crf to 16.5 to see if that'll change anything, but the rate control really gets screwed up with your transmorphmagical mode :(

[kumi]

Rate control gets out of whack, but it does have potential... I've been trying to match the (overall) quality achieved with aq-sensitivity 0 on a certain source... and no amount of fiddling with >0 aq-sensitivity values was able to approach it.

[Dark Shikari]

Rate control gets out of whack, but it does have potential... I've been trying to match the (overall) quality achieved with aq-sensitivity 0 on a certain source... and no amount of fiddling with >0 aq-sensitivity values was able to approach it.It seems to be absurdly source-dependent--which is what AQ should try to avoid.

This winter, I'll try to find a good way to fix it if I can. Its a tough problem.

[Gromozeka]

When you can add AQ in official build? :)
It already now gives very big positive effect.
It would allow more better testing yours AQ many people From the different countries
Thank you

[Sharktooth]

AQ is not ready yet. stop asking those silly questions.
all Dark Shikari needs is testing from qualified persons, not all idiots on the planet.

[ToS_Maverick]

@Dark Shikari and Sharktooth:
What type of samples, genres, ... still need to be tested? It would be nice to have a list or sth, to know which samples are still open.

maybe i got some other useful stuff that could be tested.

[Gromozeka]

To Sharktooth
Слышь ты, умник, производное децибела, аля мозг планеты, будь поскромнее! Мои вопросы может и не блещут ни познанием английского языка, ни алгоритмами программирования, но я подозреваю, что ты мог бы покумекать своими мозгами и быть несколько вежливее.
А если ты на это не способен то закрой свой ротик, зачехли рога на башке и сопи в тряпочку!
С нескрытым раздражением, но уважением, Игорь

[Sagekilla]

When you can add AQ in official build? :)
It already now gives very big positive effect.
It would allow more better testing yours AQ many people From the different countries
Thank you


Personally I don't think it's ready yet. Currently the rate control for constant rate factor is completely off when using adaptive sensitivity for AQ, so it'll take a lot of testing before AQ will be introduced to svn since it's completely failing with crf.

[sp@rrow]

Gromozeka
Бугага, эта 5 :-)))

[fields_g]

To Sharktooth
Слышь ты, умник, производное децибела, аля мозг планеты, будь поскромнее! Мои вопросы может и не блещут ни познанием английского языка, ни алгоритмами программирования, но я подозреваю, что ты мог бы покумекать своими мозгами и быть несколько вежливее.
А если ты на это не способен то закрой свой ротик, зачехли рога на башке и сопи в тряпочку!
С нескрытым раздражением, но уважением, Игорь

Igor (Gromozeka),
You've missed years of people asking for AQ to be added to SVN. Not only that, but you have also missed years of different methods, tweaks, and revisions. AQ (or other psy enhancements) is greatly needed, but is not currently stable.

I think the x264 community can be proud of the integrity of our SVN. Patches are accepted into SVN when the developers are comfortable with what it does and feel they can maintain the addition properly. Feel free to compile x264 yourself with any available patches you choose. In fact, I would guess that the majority of people here use builds that are NOT pure SVN.

BTW.... You might want to brush up on Rule 13.

[Sharktooth]

just as a reminder...
13) The official language is English. Outside the translator forum English is the only allowed language.

[Gromozeka]

I think the x264 community can be proud of the integrity of our SVN. Patches are accepted into SVN when the developers are comfortable with what it does and feel they can maintain the addition properly. Feel free to compile x264 yourself with any available patches you choose. In fact, I would guess that the majority of people here use builds that are NOT pure SVN.

BTW.... You might want to brush up on Rule 13.

I have understood you on 75 %. Russian language is combined even for Russian people at times. And to communicate here on it it is wrong. Thanks for all

[Chainmax]

fields_g: Gromozeka's post was polite and, more importantly, was volunteering for testing. What you described is not an excuse for Shartooth's awfully rude, insulting and snobbish retort. He should know better.


just as a reminder...
13) The official language is English. Outside the translator forum English is the only allowed language.

Just another reminder:
4) Be nice to each other and respect the moderator. Profanity and insults will not be tolerated. If you have a problem with another member turn to the respective moderator and if the moderator can't help you send a private message to Doom9

[bond]

yeah everyone keep cool please...

[desta]

Sorry to ask probably an obvious question, but does "--aq-sensitivity 0" have to be input to use the 'automatic' thresholding? Going by --help and the info in the first post, I would've assumed that automatic is.. automatic, but going from certain other posts in this thread, it seems it does need to be input.

[Dark Shikari]

Sorry to ask probably an obvious question, but does "--aq-sensitivity 0" have to be input to use the 'automatic' thresholding? Going by --help and the info in the first post, I would've assumed that automatic is.. automatic, but going from certain other posts in this thread, it seems it does need to be input.Automatic wasn't the default until version 0.3. Now it is.

[desta]

Ah I see. Thanks for clarifying. :)

[Ranguvar]

I have a video that would be very good for testing this, IMO. I'd like to do so and provide screenshots.

It's an HD trailer for a video game. Not capped by myself, provided by GameTrailers. Would it be a Rule 6?

[Dark Shikari]

I have a video that would be very good for testing this, IMO. I'd like to do so and provide screenshots.

It's an HD trailer for a video game. Not capped by myself, provided by GameTrailers. Would it be a Rule 6?Nope, those are fine to use.

[Sharktooth]

fields_g: Gromozeka's post was polite and, more importantly, was volunteering for testing. What you described is not an excuse for Shartooth's awfully rude, insulting and snobbish retort. He should know better.
ppl need to learn to speak only when necessary and just not to blow air out of their mouth.
the fact there were like 1 million of ppl requesting AQ in the SVN it doesnt mean it DESERVES to be there.
There are several test builds and i cant see the reason why incomplete and experimental code should be put in the x264 SVN.
Also, since it has been asked SO MUCH TIMES (and the answer was always the same), he could SEARCH before posting (you should too).
No offense, it's just my way...
and excuse me for the OT.

[bond]

guys, again, keep cool and nice please. next offense, no matter which one, will get striked

[ToS_Maverick]

@Dark Shikari
i played around with VC-1 a bit, while an idea struck me:
would it make sense, to apply AQ only to I or I and P frames?

i think it would be interesting to see the effect of this. quality and metric-wise and if you could save some bitrate by leaving the B frames out.

[Dark Shikari]

@Dark Shikari
i played around with VC-1 a bit, while an idea struck me:
would it make sense, to apply AQ only to I or I and P frames?

i think it would be interesting to see the effect of this. quality and metric-wise and if you could save some bitrate by leaving the B frames out.That's not a bad idea, since the main disadvantage of my AQ is actually the cost of encoding the qp_delta bits.

[ToS_Maverick]

great that i could be helpful!

well since VC1 is using things from AVC, why not have a look at what they are doing.

at the MS VC1 codec you can set this. i got the idea because the background grain started to "update" with every I frame. i could not test the I/P setting since AVS2ASF seems bit buggy.

[mahsah]

Any idea what settings (if any) for AQ I could use to retain the dithering added by Gradfun2db?

[Sagekilla]

Any idea what settings (if any) for AQ I could use to retain the dithering added by Gradfun2db?

Dithering is actually quite "complex" since it's not a uniform, flat color like the sky would be. Since it makes use of bunching together a bunch of different colors in a set pattern to make it LOOK like another color, it'll end up getting mushed because it looks like grain or noise.

And since AQ looks to increase the bits allocated towards flat areas so that it isn't blocky (dithering looks flat, but to the human eye, not to an algorithm) but since dithering isn't like this, like I just explained above, it won't work well.

[Dark Shikari]

And since AQ looks to increase the bits allocated towards flat areas so that it isn't blocky (dithering looks flat, but to the human eye, not to an algorithm) but since dithering isn't like this, like I just explained above, it won't work well.My algorithm will most definitely consider dithered blocks to be very very close to flat, and as a result will decrease their quantizer (though not necessarily by enough to keep the dither accurately).