Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Dark Shikari]

No, it's a great idea, IgorC.


That's not the whole story. If it was simply a matter of adjusting strengths, one would just lower 0.47's strength to get the better result. 0.47 does not distribute QPs in a visually pleasing way in the scenes that need it most, whereas 0.46 does, at the same bitrate. Using no AQ doesn't benefit the scenes of interest at all. Keep in mind, for dark/flat scenes (which is all that matters at this bitrate--who cares about a little ringing here and there when you can't even see what's happening in darker frames?), 0.46 is better than 0.47, which is the purpose of the AQ. SSIM is higher, which you certainly tote as a key point, too.

All that's left is to do the double blind. I tried to convey the expression in my test post containing screenshots of the worst-looking scenes. I thought, people who see that the horrible quality in these areas (which is why we're using AQ even at its potential ringing cost), can get a good idea of which is better. The people I've shown agreed with me here, so I made the post.

We just need IgorC's test to prove it.Testing specific frames on a single source is not a way to prove that an algorithm is useful. It only proves that the algorithm is useful on those frames, on that source. The fact that your source is incredibly low resolution means the test is particularly invalid, because it does not correlate with what most sources are like.

Just because one algorithm is better than another for one source does not mean it is useful in general.

Moreso, the fact that you're not using qcomp is basically rigging the test in favor of AQ versus non-AQ, even though in any normal encode one would use qcomp.

[IgorC]

Nobody told that it would be evaluated by frames. There is MSU ap that admites blind test by playback. Real conditions.

[Dark Shikari]

Nobody told that it would be evaluated by frames. There is MSU ap that admites blind test by playback. Real conditions.Yup, I've used it.

Anyways, this thread is now pretty much pointless.

1. I am not going to work any more on the AQ algorithm for the time being.

2. 0.46 cannot possibly be committed, for the reason I stated earlier.

3. Therefore, if you want any aspect of 0.46's AQ to be committed, someone else must code it.

[CruNcher]

I have a better idea, just commit it and then lets wait what happens, if we see a big demand from low bitrate encoders that have ringing problems that they didn't had before, we know something should be changed don't we? In some way that's a big subejctive test and it doesn't need to be setup ;)
And yeah i used the MSU Subjective test to rate 0.46 vs 0.47 too :) the left/right decission setup

[Guest]

If people cannot accept simple limitations of the algorithm like this, I'm going to simply delete this thread and continue any further development in private. For the record, you're allowed to delete only your own posts. Yes, I know the forum software allows you to delete the entire thread, but mods will restore it if you do that.

I can see that some friction is developing here. Maybe CruNcher can start a new thread for his variant so that Dark Shikari can continue this thread in the direction of his vision?

[Dark Shikari]

For the record, you're allowed to delete only your own posts. Yes, I know the forum software allows you to delete the entire thread, but mods will restore it if you do that.

I can see that some friction is developing here. Maybe CruNcher can start a new thread for his variant so that Dark Shikari can continue this thread in the direction of his vision?I don't need this thread anymore; development is done, and while I don't deny there could certainly be improvements to the algorithm, I am done discussing it for now. Nothing constructive has been discussed in a dozen pages.

I'm happy to support an attempt to make a better AQ--but I'm not going to be the one coding, at least not now.

[CruNcher]

For me it's also not important to continue for now as time will tell how people gona react to the New AQ in a bigger user scale my usage behaviour isn't really important here and as i said i've got my workaround so nope no need to continue. People know what for problems could arise from the New AQ Visualy under some circumstances and how to avoid them by now and so everything has discussed and is publicly available :)
And Dark Shikari please don't feel in anyway atacked you really did a great job with the AQ it's just criticism about the way it gets implemented now and what for possible outcomes this might have in terms of Visual Perceptive Problems.

[DeathTheSheep]

Before you make any drastic decisions, I'm going to try to choose my words carefully here in an attempt to offset the risks of coming off "silly," upsetting anyone, or disseminating an inaccurate impression of my intentions.

Firstly, it may be noted that the frames I chose were representative of the worst-looking scenes in the original. They are not intended to represent all scenes in the source video, nor are they to be taken as the net visual result of the video in its entirety. They are, however, intended to represent what I have alluded to before as "scenes of interest," a denotation which I had understood to be self-explanatory, but which I will now undertake to elaborate upon in greater degree and verbosity.

Consider the following scenario. Suppose we have a test clip in which there are some mid-motion dark scenes and some low-motion/no-motion high-contrast scenes. Qcomp would assign lower QPs to the high-contrast still scenes at the expense of the darker ones. Also recall that the same specific QP X assigned to a low-contrast/dark scene will tend to look worse than the same QP X assigned to a high-contrast scene, since a high-contrast scene is allotted more bitrate, the uniform rounding mechanism would favor the higher bitrate of the high-contrast scene over the the low bitrate of the low-contrast one. In this case, the fact that qcomp is motion compensated turns into a deleterious aspect of the algorithm in that the darker scenes (or darker areas in brighter scenes) will remain characteristically lower in subjective quality than their high-contrast counterparts. I do not profess an intimate knowledge of the precise means by which x264 gives rise to said degradation of visual quality in dark/flat/low-contrast areas with uniform QP, but I will make so bold as to profess intimate familiarity with the results of such a distribution (in both QP and CRF modes).

I thereby take "scenes of interest" to denote these areas wherein the degradation of visual quality is apparent in proportion to that of the higher-bitrate, heightened-contrast scenes of prior allusion.

It might also be worth mention that at lower bitrates, quality degradation is perhaps more apparent and more distracting in the scenes of interest than the presence of artifacts in high-contrast regions, for ringing and noise artifacts will inevitably abound at this bitrate in any scene. Obvious reduction of the overall "fogginess" and "blockiness" of the scenes of interest, however, which alleviates the difficulty to discern even rudimentary shapes and other elements therein (even more so in the adverse qcomp situation I described above), is often well worth the introduction of such artifacts elsewhere in otherwise "clearer" scenes.

Please bear in mind also that qcomp is an adaptive quality rate-control mechanism, as is VAQ; however, whereas qcomp relies on a complexity metric (which may and does backfire, as in such cases as aforementioned), VAQ relies on a different, variance-based metric. This is the reason you advise against using qcomp with VAQ enabled at any significant strength. It stands to reason that one such algorithm can potentially be chosen over the other.

I believe you are asserting a logical fallacy as well. You assume that my source and settings are not representative of the bulk of consumer AVC encoding; in fact, you claim the entire setup to be abnormal. However, in so doing, you are neglecting to account for the considerable hand-held, mobile, compatibility, and low-complexity demands that dominate many aspects of the AVC encoding arena. I'll grant you that the bitrate in the above tests is low; however, it was chosen for illustrative purposes such that the visual differences might be made more readily apparent.

The topic of discussion at hand isn't even whether or not the usage of AQ is beneficial, since any large-scale optimization for one subjective quality metric/preference is bound to contradict with another to some extent, and different scenes tend to benefit differently for each. The point at hand is that VAQ 0.46 produces significantly higher visual quality on the scenes of interest, objectively and metrically, than does 0.47. It can also be argued, unlike as in the case of no-AQ vs AQ, that 0.46 produces better results on the entire clip, and perhaps even on a range of sources and bitrates, if CruNcher's remarks are to be taken into account, than 0.47.

I also believe a large-scale blind test on a wide variety of sources, bitrates, and AQ settings is in order to verify these and other assertions, as it is foolhardy to proceed with what may actually be a substandard algorithm.

[Atak_Snajpera]

@Dark Shikari
You've made alot to improve quality in x264 and I'm fully satisfied with default settings! There is no need to adjust anything! I don't really understand why some people are still looking for holy gral? They use low bitrate with crazy settings (no cabac...) and They are still crying that they cannot achive perfect results. This patch should be commited to SVN as soon as possible! No further pointless discussion. Anybody who is not satisfied with current AQ should write his own algorithm. Let's see what you can do 'smart' boys :)

[bob0r]

I love you all, let's group-hug.

[DeathTheSheep]

Lets!

[Inventive Software]

AQ is not perfect, by design, and cannot make a clip perfect on it's own without assistance from other factors. End-of.

[DeathTheSheep]

Correct, there's always some degree of sacrifice involved. It must draw the bits from somewhere--they don't spontaneously spring into existence. It's simply a new method controlling the distribution of QPs by redistribution.

[ChronoCross]

can you duplicate this on another clip?

[Adub]

Okay, big group hug continued!!!

I just want to say that I really love your hard work, DS, and I hope that you continue your efforts with a brilliant eye towards the future. And I hope that, with time, I join you in your efforts and all will be good. Just got to get through college right now.

Thanks again and code on!

[ToS_Maverick]

@Dark Shikari
is there any way we can donate, so that you (and maybe your girlfriend) could go and have a nice dinner?

[*.mp4 guy]

Dark Shikari, Thank you for all of your work on aq, I must admit that I haven't been keeping up to date with X264, but If vaq gets added to the svn I'll finally have a good reason to hunt down a new revision.

Feel free to ignore this next part, I'm sure your quite tired of the debating by now, but my curiosity is getting the better of me.

Avoiding any debate as to the merits of the various vaq incarnations, is it possible that the differences people are posting about are caused by a bias in the rate of 4x4/8x8/intra/inter blocks chosen by x264, introduced by the rounding erros in 0.46? from the screenshots I looked at, I can't see quantization alone acounting for some of the differences in the examples, it looks to me like 0.46 may be using the 4x4 transform less often then noaq/0.48/0.47. Obviously this is just wild speculation, and honestly I don't see what all the noise is about, 0.46 looks better sometimes, 0.47/0.48 looks better other times; but there does apear to be a trend in the way they look compared to each other, 0.46 looks to me like it has more 8x8 transform induced ringing, while the others apear to have more 4x4 transform induced aliasing/microblocking. I suppose it could also be a peculiarity of the AVC deblocking filter, reacting to the different realtive quantizations between blocks or something...Meh.

I bet I'll end up dieing of unfounded curiosity someday. "no really I'm not a terrorist, that screensaver just looked so cool...".

[Atak_Snajpera]

AQ is not perfect, by design, and cannot make a clip perfect on it's own without assistance from other factors. End-of.

Since when any compression is perfect. I think your expectations from h.264 standard are a little bit to high. Use higher bitrate and get used to it.

[CruNcher]

Atak ehh sure it's not perfect but to find the correct Visual (or near that) Balance for every bitrate should be the goal, it shouldn't drift in 1 direction. But hell it will improve over time im sure, and what *.mp4 guy mentions is interesting indeed :)
Most people will be fine @ the moment with the AQ some will hit walls with it but those that do will find ways to visually circumvent these walls ("that's what a good compresionist makes") :) So this is only a problem for beginners, but those are not trying such ultra crazy low bitrate stuff so it will be no real problem @ all for the mass of users as i said. But all of this doesn't change the fact that this AQ is better then no AQ @ all and perfect work by Dark Shikari and a big visual quality step for X264 in many situations.

[Inventive Software]

Since when any compression is perfect. I think your expectations from h.264 standard are a little bit to high. Use higher bitrate and get used to it.

I think you mis-understood the point of my post. It was to clear up the arguments over how (non)important AQ is to x264 at the moment. I don't have any expectations of H.264, and I personally don't use AQ because my H.264 encodes at CRF20 are very pleasing on the eye. ;)

[Atak_Snajpera]

I don't have any expectations of H.264, and I personally don't use AQ because my H.264 encodes at CRF20 are very pleasing on the eye.

Blocks are less visible on gloss LCDs but more visible on mat LCDs

Friend of mine had HDTV 40'' Samsung 1360x768 (mat lcd) and blocks were quite visible then he bought 43'' samsung 1920x1080 (gloss) and all ugly blocks on flat areas suddenly disappear despite bigger screen! Before you ask. In both cases colors were set correctly. Black was black,white didn't kill eyes and so on. FFDshow was set to HQ-RGB32. I've also notice the same results on LCD monitors (gloss vs mat)

[akupenguin]

Blocks are less visible on gloss LCDs but more visible on mat LCDs
I noticed the same thing, but I attributed it to the fact that my glossy LCD has a much brighter black than my matte LCD. Not a difference in adjustment, but rather that black pixels don't completely shut out the backlight.

[Maccara]

Please all who have complains / praises, also specify your display devices!

There's a HUGE difference on properly calibrated (for graphics work, not video) CRT (~10y old, still haven't found decently priced LCD that comes even close) & LCD (<1y old), especially in dark scenes.

I always compare those side by side, as there's zero possibility to make an encode look "perfect" on both (except when I tested Eizo ColorEdge LCD, which came close to CRT qual) so I'll have to compromise (or target one platform).

[*.mp4 guy]

In your experience in what areas are LCD's more demanding then CRT's and vice versa.

[Maccara]

In your experience in what areas are LCD's more demanding then CRT's and vice versa.

Some (many?) LCDs still have 6bit panels to me it seems smooth color gradients can be quite challenging (and YV12 already poses a bit of challenge here, so it is only pronounced on LCDs). Also due to black levels, blocking in dark areas can become quite prominent.

I've noticed that I've had to sometimes throw quite a bit much more bitrate to get rid of blocking in dark areas whereas CRT copes already with much less. (also, AQ seems to help quite a lot and other parameters, of course)

Of course, this is all subjective. Without going into personal preferences, I just wanted to remind that different display technologies have quite a bit different characteristics and it might be a good idea to keep that in mind when tuning PSY algorithms and possibly test on various hardware before making any conclusions.

[TheRyuu]

I ran some tests in anime.
New AQ vs old (Haali) AQ.

The old AQ was done using aq-strength 1.0, and the new AQ was done with the defaults (1.0 on the old aq could be the default, not sure).

What I found was that not old did the new VAQ result in a lower average quant (18.5 vs 19, not too big :p), the new VAQ also, in MOST scenes that I looked at, did a better job at keeping details and avoiding blocks.

There were places were the old AQ was ever so slightly better (really, really, flat places), but on a whole, I would definity go with the new VAQ because as a WHOLE, it performs better then the old AQ, even on anime.

I did this with the new AQ v0.48.
I'm testing v0.46 but from what I've seen so far, v0.48 is the way to go.

The tests I did are subjective but I'm just reporting what I've found.
They are also done at a bitrate of ~900, which is fine for the anime dvd backups I'm doing. (avg quant goes about 17-19).

However, I'm not sure which is better if bitrate isn't a factor.

[TheRyuu]

I ran some additional tests with VAQ v0.46 as well with anime.
From what I can tell, 0.46 does perform better then 0.48 in anime from a "visual" standpoint.

I'm not sure of the technical details of it, but not matter how you look at it, using AQ is a trade off.

Once again, looking at it from an "overall" standpoint, 0.46 is slightly superior to that of 0.48. I was looking for blocking and background details and it looks like 0.46 comes out on top.

I compared 0.48 with 0.5 strength and 13 sensitivity, with 0.46 with 0.6 strength and 20 sensitivity.

I'm not sure if 0.48 can be tuned to perform similar to that of 0.46, but I'm still testing stuff out. This is just what I've found SO FAR. I think next I'll try 0.48 with strength of 0.5 and a higher sensitivity like 15-20 which may be the cause of 0.46's better visual quality.

[Terranigma]

Can't anyone see that in reality, this thread's closed? There's no new posts by Dark_Shikari: He asked for this thread to be closed/deleted, and although he didn't get exactly what he wished, he took a different approach called the silent treatmemnt. :D
You can blame it on rio (that's right, that's a movie, but there are blamers to be blamed: just do some reading =P).

[Dark Shikari]

Can't anyone see that in reality, this thread's closed? There's no new posts by Dark_Shikari: He asked for this thread to be closed/deleted, and although he didn't get exactly what he wished, he took a different approach called the silent treatmemnt. :DThis is moreso do to the fact that I'm spending all my coding time on fixing 1pass VBV ;)

[microchip8]

This is moreso do to the fact that I'm spending all my coding time on fixing 1pass VBV ;)


oh, what about QNS? you forgot about that? :P :P

[TheRyuu]

Can't anyone see that in reality, this thread's closed? There's no new posts by Dark_Shikari: He asked for this thread to be closed/deleted, and although he didn't get exactly what he wished, he took a different approach called the silent treatmemnt. :D
You can blame it on rio (that's right, that's a movie, but there are blamers to be blamed: just do some reading =P).

Just reporting what I see.
Either case, the new VAQ is superior to the old Haali AQ even in anime.

Thanks Dark Shirkari! :)
:thanks:

[Atak_Snajpera]

I noticed the same thing, but I attributed it to the fact that my glossy LCD has a much brighter black than my matte LCD. Not a difference in adjustment, but rather that black pixels don't completely shut out the backlight.

I also see difference on blue sky. Gloss = no dancing blocks , mat = blocks. Difference is not huge but visible :(

[DeathTheSheep]

0.46 works even better if you use strength near 1.1, and possibly better if you use --ipratio 1.45 or 1.5 (to account for keyframe effects like we discussed before).

Lower sensitivities work okay too on anime. Just make sure you lower the crf to compensate. Cheers!

[Sharktooth]

VAQ 0.46 is dead. if you still didnt understood, it's useless you compare it to the newer one coz 0.46 is going nowhere.

[Morte66]

So if I buy that LCD TV I've been thinking about, it should be glossy...

[DeathTheSheep]

VAQ 0.46 is dead. if you still didnt understood, it's useless you compare it to the newer one coz 0.46 is going nowhere.

Not necessarily true... As long as it exists and is used, I'll be happy to maintain it myself. Same with me-prepass. (I also keep satd-me modified to my standards). Nothing is dead in OS if there's someone around to maintain/use it. :)

[Atak_Snajpera]

So if I buy that LCD TV I've been thinking about, it should be glossy...

It depends. Mat LCD gives more accurate colors (in my opinion) therefore you see blocks.
Glossy LCD shows less colors so blocks are less visible.

Some my old post regarding LCDs
http://forum.doom9.org/showthread.php?p=1044240#post1044240
http://forum.doom9.org/showthread.php?p=1044306#post1044306

[CruNcher]

It depends. Mat LCD gives more accurate colors (in my opinion) therefore you see blocks.
Glossy LCD shows less colors so blocks are less visible.

Some my old post regarding LCDs
http://forum.doom9.org/showthread.php?p=1044240#post1044240
http://forum.doom9.org/showthread.php?p=1044306#post1044306

But glossy they look more hmm how could you say "vivid"? (you can very well compare it with a Print on glossy photo paper compared to normal paper) tough the bigest problem with glossy panels is the reflection problem with ambient light sources, and so even a small light in direction to it can couse viewing problems, the same with sunlight coming from a window. So you should know the enviroment you gonna use such a LCD in very well or be ready to change the light sources if it seems to be problematic. You really should go in a shop and compare what fits best for you it's also a very subjective thing as Atak allready noted.

And the problems about all this lossy viewing stuff goes very deep (especialy in Windows) their are many factors that play a role in how the end result looks on your monitor (thats also a reason why for DTP and Video MACs are used so often, they have a very good defined color, brightness,contrast and gama system)

in Windows you have (The Driver, The YV12 renderer, The Decoder, The Player) and if 1 does it wrong or in bad combination with those others the end result will look worse (and haveing all from different Developers is gonna make problems for sure maybe not today but tommorow, especialy with DirectShow) on your system compared to a correctly calibrated one. Especialy as alot of stuff is hidden in Dark areas and correctly calibrated can't be seen most of the times in motion (by untrained eyes) but wrong it becomes anoying to everyone (of course you can use that knowledge to optimize the stuff visualy in those extreme uncalibrated viewing conditions and that is very powerfull in the end result for a calibrated system specialy for trained eyes) and if you work in a calibrated system and would like to know what might can go wrong in a lossy source visualy Avisynths histogram is perfect to show you that :) histogram(mode="luma").

[ChronoCross]

Not necessarily true... As long as it exists and is used, I'll be happy to maintain it myself. Same with me-prepass. (I also keep satd-me modified to my standards). Nothing is dead in OS if there's someone around to maintain/use it. :)

until of course pengvado changes something that breaks it and only adjusts 0.48. More than likely it would completely break or alter the ability of 0.46 beyond a point that simply changing values would fix (happened with haali's patch quite a few times).

[DeathTheSheep]

Ah, too true. We'll cross that road when we get to it though, won't we?

...Or so they say. :eek:

[Morte66]

My computer monitor is well calibrated and ICM profiled.

As for the LCD TV, I was prepared to darken the room for a projector so glossy LCD shouldn't be too bad. OF course I will look at the TV before buying, I'm mostly waiting for the whole 24fps --> 120fps thing to become ubiquitous (in Britain).

[Stingrey]

Is it possible, that with your build die CRF parameter can't be chosen in 0.1 step's, only in 0.5 step's?
CRF 21.0 to 21.4 all gives me the same bitrate, 21.5 gives me a much lower one!
I would need 21.1!
With CRF 21.0 1.109,16 kb/s
CRF 21.5 885,24 kb/s

--crf 21.5 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --threads auto --thread-input --progress --output "output" "input"

[Dark Shikari]

Is it possible, that with your build die CRF parameter can't be chosen in 0.1 step's, only in 0.5 step's?
CRF 21.0 to 21.4 all gives me the same bitrate, 21.5 gives me a much lower one!
I would need 21.1!
With CRF 21.0 1.109,16 kb/s
CRF 21.5 885,24 kb/s

--crf 21.5 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --threads auto --thread-input --progress --output "output" "input" Yes, this is because AQ + CRF = AQ + CQP, because qcomp gets disabled.

What I could do is map fractional changes in CRF to changes in sensitivity.

[Stingrey]

Mhm, I think that could be a good thing.
The jump in the bitrate is to high, some points between would be great.

[Dark Shikari]

Mhm, I think that could be a good thing.
The jump in the bitrate is to high, some points between would be great.In the meantime, feel free to adjust sensitivity up and down very slightly (fractionally) in order to change the bitrate between CRFs.

[Stingrey]

Ok, I will do some tests an then post it here.

Test:
CRF 21, sens 13: 1109.16 kb/s
CRF 21, sens 12: 995,98 kb/s
CRF 21, sens 11: 885,24 kb/s equal to CRF 21.5 with sens 13

so just changing the sensitivity won't do much.

[DeathTheSheep]

In the meantime, feel free to adjust sensitivity up and down very slightly (fractionally) in order to change the bitrate between CRFs.

Hey, that's precisely the kind of adjusting I did to fine-tune my filesizes, and you yelled at me for it! :devil:

:)

[Dark Shikari]

Hey, that's precisely the kind of adjusting I did to fine-tune my filesizes, and you yelled at me for it! :devil:

:)You were fine-tuning by more than 1 sensitivity ;)

You went all the way up to 19 or 20, which appeared to cause quantization problems.

[DeathTheSheep]

Ah, too true. Oddly enough the sensitivity 19 encodes looked, well, not too bad! My resolution is also smaller (people mentioned that differing resolutions have an impact on how well the threshold stays true to the original crf/cqp), and a higher sensitivity was needed to keep my original QP anyway.

If you don't mind me asking, what were the quantization problems?

[Dark Shikari]

Ah, too true. Oddly enough the sensitivity 19 encodes looked, well, not too bad! My resolution is also smaller (people mentioned that differing resolutions have an impact on how well the threshold stays true to the original crf/cqp), and a higher sensitivity was needed to keep my original QP anyway.

If you don't mind me asking, what were the quantization problems?For some reason which I do not understand, using high sensitivity on your small clip caused AQ to do basically nothing at all. As you raised sensitivity from 13, it progressed from acting normal to doing nothing, in a quite smooth fashion.