aq-mode 3, 4 / fade-compensate / treillis 3 & subme 11

[sirt]

Hello,

Here I am to ask you for answering another couple of questions. Those are rather specifical but somebody may answer :

1) I have tried different versions of x264 ; among them, I got the tMod variant (https://astrataro.wordpress.com/category/encode/x264/) and it features additional AQ modes : 3 & 4 ; what are they intended for ? How will they affect the video and in what way are they different and/or better than standard aq-mode 2 ?

2) I downloaded a patch called fade.compensate.2208.diff from komisar's home page and compiled it myself. Let's image I have a video containing a large amount of fades. Will it be really effective to use --fade-compensate 1 for example or using the official x264 build will give me almost same output ?

3) I've read on git that developpers are working on a new treillis mode, treillis 3. Accordingly, subme 11 has been implemented and it seems the combination --treillis 3 --subme 11 should be used in the future. Am I wrong ? Moreover what will be the difference between treillis 2 & 3 and what is the point of using subme 11 with treillis 2 right now ?

[LoRd_MuldeR]

(1) Those are additional experimental "Adaptive Quantization" modes. If any of those AQ modes was generally better than the current AQ modes "1" and "2", it would have been added to official x264 and made the default. Still it may depend on the kind of source you are encoding and (not less important!) on your personal preferences whether you prefer AQ mode "2" or "3" over the current modes. You can only give them a try and see what happens ;)

(2) Since support for "Weighted P-Prediction" (Weight-P) has been added to x264, it should handle "fades" much better than before. Anyway, the "fade compensate" patch, as the name implies, moves more bits into "fades". Those bits obviously will be missing at other places. So while the "fades" are supposed to look better with that patch (there is no guarantee though!), it will not necessarily improve the overall quality! Moving too many bits into fades will certainly be counterproductive. But if you think that "fades" look noticeably worse than the rest of your encode, you may still give this patch a try. Note that it has a parameter to control the "strength" of the patch's effect.

(3) According to the commit message, Sub-ME mode 11 "disables all early terminations in analysis". This will probably be much slower, with only tiny improvements (if at all). Use it, iff time doesn't matter (or use "--preset placebo" right away ^^).

[sirt]

Thanks for those explanations LoRd_MuldeR.

Regarding the first point, what do you mean by "the kind of source" chosen ? Unfortunately, I have no precise idea what AQ mode 1,2,3 or even 4 will change. As from I've read in the past, AQ 2 is already "experimental" even though it seems it is currently exhorted to use it. But I ignore if one or another mode is more adapted to a precise type of video (slow or fast motion, black and white, fades...).

Second point : in fact another idea came to my mind to process fades advantageously : I wanted to use --zones and set a very high bitrate for each fades, one consisting in some frames. We already discussed about the use of zones and I remember it would be conceivable to use such parameter in typical situations (video opening, closing...) only. So It is probably not a good idea to do this ; by the way, let's say I encode in 2 pass at 2500 kbits/s and I use --zones to raise the bitrate in 10 places (fades, at 10000 kbits/s for example). Do you think that will have the same impact than using fade compensate namely allocating too many bits into fades ?

Third point : what if I use subme 11 and treillis 0, AQ 0 ? Does that have a sence ?

[LoRd_MuldeR]

Thanks for those explanations LoRd_MuldeR.
Regarding the first point, what do you mean by "the kind of source" chosen ? Unfortunately, I have no precise idea what AQ mode 1,2,3 or even 4 will change. As from I've read in the past, AQ 2 is already "experimental" even though it seems it is currently exhorted to use it. But I ignore if one or another mode is more adapted to a precise type of video (slow or fast motion, black and white, fades...).

What I mean is that there may be sources where AQ mode "3" or "4" may give an improvement and others where they don't give an improvement or even hurt quality.

You simply will have to give them a try with the kind of footage your are encoding...

Again: If AQ modes "3" and "4" generally performed better than the current modes "1" and "2", the x264 developers would already have adopted those modes into the official codebase.

The fact that they don't have, indicates that if modes "3" and "4" perform better in certain cases, there also are enough cases where they perform worse...

(It could also indicate that there simply is no good theoretical justification for those modes)

Second point : in fact another idea came to my mind to process fades advantageously : I wanted to use --zones and set a very high bitrate for each fades, one consisting in some frames. We already discussed about the use of zones and I remember it would be conceivable to use such parameter in typical situations (video opening, closing...) only. So It is probably not a good idea to do this ; by the way, let's say I encode in 2 pass at 2500 kbits/s and I use --zones to raise the bitrate in 10 places (fades, at 10000 kbits/s for example). Do you think that will have the same impact than using fade compensate namely allocating too many bits into fades ?

Using zones to manually increase the bitrate in "fade" sections is pretty much the same thing that the "fade compensate" patch is supposed to do automatically.

Third point : what if I use subme 11 and treillis 0, AQ 0 ? Does that have a sence ?

Completely turning off Adaptive Quantization doesn't seem like a good idea, unless you are a fan of blocking and banding :p

Also using a super-slow "placebo" setting like Sub-ME 11, but at the same time turning off Trellis quantization completely, doesn't seem reasonable either... :confused:

You better follow the x264 Preset system!

[sirt]

You better follow the x264 Preset system!

As you have perhaps guessed, I am relatively reluctant to use the all-in-one presets. Obviously, what I do is purely educative and I'm after "what will happen" if I try this or that.

For example, I tried to see which bitrate I should not go above with slow settings in order to preserve the video...

[mandarinka]

(1) Those are additional experimental "Adaptive Quantization" modes. If any of those AQ modes was generally better than the current AQ modes "1" and "2", it would have been added to official x264 and made the default.

In theory, yes. In practice though, nobody really seemed to have cared to test those modes. So we don't really know.

Fade Compensate simply pumps more bits into parts that are detected as fades, when you use mbtree. It was hacked so because people thought that some fades look worse with mbtree. Dunno if you experience that. In crf mode, it shouldn't remove bits from other parts IIRC - it will just increase size of fade frames.

By "trellis 3" you apparently mean --me trellis. That method is not in development anymore AFAIK, due to difficulties with plugging it into x264 and making it actually give significant benefits (it requires a lot of cpu cycles and competes with the already present analysis techinques for the "fruit"). And yes, subme 11 was meant for that, because the method needed all the candidate vectors to work on, including those that early terminations would otherwise rule out before.

[sirt]

Fade Compensate simply pumps more bits into parts that are detected as fades, when you use mbtree. It was hacked so because people thought that some fades look worse with mbtree. Dunno if you experience that. In crf mode, it shouldn't remove bits from other parts IIRC - it will just increase size of fade frames.

By "trellis 3" you apparently mean --me trellis. That method is not in development anymore AFAIK, due to difficulties with plugging it into x264 and making it actually give significant benefits (it requires a lot of cpu cycles and competes with the already present analysis techinques for the "fruit"). And yes, subme 11 was meant for that, because the method needed all the candidate vectors to work on, including those that early terminations would otherwise rule out before.


Thanks for your detailed answers. Then how can you be sure the fade compensate algorithm will indeed recognize each fade ? To my mind, using --zones to manually increase the bitrate when a fade occurs seems to be a better alternative in so far as it will - as LoRd_MulDer explained - allocate the bits in the same way when encoding in 2 Pass mode. What strikes me is that, according to you, the allocation of bits in CRF mode will be slightly different by just increasing size of fade frames. But why, in 2 Pass mode, would the encoder remove bits from other places when it is forced to move more into fades ? Is it only to respect the bitrate / final size intended ? In this case is it possible to force the encoder to allocate the necessary bits to fades without sacrifying the rest ?

About treillis 3 : do you mean its developpement has been totaly suspended ? In this case, does using subme 11 worth it with treillis 2 ?

[mandarinka]

What strikes me is that, according to you, the allocation of bits in CRF mode will be slightly different by just increasing size of fade frames. But why, in 2 Pass mode, would the encoder remove bits from other places when it is forced to move more into fades ? Is it only to respect the bitrate / final size intended ?

About treillis 3 : do you mean its developpement has been totaly suspended ? In this case, does using subme 11 worth it with treillis 2 ?

If you are doing a two-pass encode, then the final filesize is fixed. When the encoder allocates more bits to fades, then of course, those bits will come at the expense of the rest of the video. If you want to use fadecompensate in 2pass mode, use higher overall bitrate to make up for it.

Development of --me trellis ended circa in september-october 2011.
--subme 11 has virtually no effect despite the significant slowdown - AFAIK.

[sirt]

Thanks for those answer.

Anyway is there a place where I could find valid diff patches for AQ Mode 4 & FGO ? I tried to get latest x264 code via git (git clone...) then I went there https://astrataro.wordpress.com/category/encode/x264/ where I got a folder full of patches but it looks like anyone is compliant with latest x264 code because both diff fails. I also tried to get the source for x264 Tmod but I can't compile it : I mostly get errors about "undeclared" parameters. Howhever, Any patch from komisar's site (fade.compensate.2208, aq-mode3.2208...) work but I didn't find AQ mode 4 & FGO there.

[LoRd_MuldeR]

Patches only work with the exact revision they have been made for/from.

Trying to apply the patch on a different revision may or may not work. And even if the "patch" command runs through without error/warning, there is absolutely no guarantee that the patch still works as intended.

The older the patch, the higher the chance that something goes wrong. So if the original author of the patch doesn't update it anymore, you can only try to ping the author or update the patch yourself...

(And even more problems can pop up, if you try to mix different patches that have not been intended to be used together)

[Blue_MiSfit]

@sirt: You're free to experiment, of course, but you're really honestly best off using the latest official build and a preset. I promise :)

[sirt]

@sirt: You're free to experiment, of course, but you're really honestly best off using the latest official build and a preset. I promise :)

Yes, I know but I am only experimenting !

So if the original author of the patch doesn't update it anymore, you can only try to ping the author or update the patch yourself...



Or I Can ping LoRd_MuldeR ! Well, the fact is I tried to manually modify the code where needed (according to the modifications done) but too many source files are different, notably x264.c : Tmod's version is really not similar to standard one gotten via git. It's true I'm not fond of editing that myself but well...anyway source code tMod from https://github.com/astrataro/x264_tMod (I've got the zip file) can't be compiled ("undeclared" parameters and many annoying errors) ; you can maybe have a look at that code ? I've tried many things such as editing the makefile not to compile some things, delelting functions that are not found...I've probably f**** up everything !

[jpsdr]

About AQ Mode, from MasterNobody less than 2 months ago :

The idea was to improve --aq-mode 2 in dark areas by allowing lower (for negatives numbers it mean higher absolute values) QP offsets there.
P.S. As users didn't really take active part in my testing I decided to give up on mode 4 [(0.25f - 3.3125f / (qp_adj * qp_adj - 0.75f) one] as too weak (it changes curve only at extremely low variance).


About subme 11 and treillis 3 : According DS (it was a "long" time ago), treillis 3 will probably never be released, it was something very experimental, and had probably not given the expected results => subme 11 has actualy no real purpose.

[LoRd_MuldeR]

Or I Can ping LoRd_MuldeR ! Well, the fact is I tried to manually modify the code where needed (according to the modifications done) but too many source files are different, notably x264.c : Tmod's version is really not similar to standard one gotten via git. It's true I'm not fond of editing that myself but well...anyway source code tMod from https://github.com/astrataro/x264_tMod (I've got the zip file) can't be compiled ("undeclared" parameters and many annoying errors) ; you can maybe have a look at that code ?

Nope, because I'm not involved with the "tMod" modification (which appears to be a fork of "L-SMASH", which in turn is a fork of the original x264) at all.

If it doesn't compile for you (and the cause of the error cannot be sorted out), ask the authors. Which would be "astrataro" in that case.

I've tried many things such as editing the makefile not to compile some things, delelting functions that are not found...I've probably f**** up everything !

If the Makefile file tries to compile a file, that's most likely for a reason.

Removing that file from the Makefile "because it doesn't compile properly" is certainly going to fix the problem :rolleyes:

And randomly deleting functions will make things even better... :p

[sirt]

Well I finally found out what is wrong : I can't compile the code with extra libraries (lavf,...) on but it works without. So I guess - again - it's my fault...hwohever those additional librairies perfectly compiles with latest regular x264 (git), so is this again a problem of compatibility ?

[LoRd_MuldeR]

In order to compile with "external libraries" you have to make sure that the required libraires are in place, obviously.

For C/C++ this means that you must make sure that the required Header files (.h) are present in your "inlcude" path and that the corresponding library files (.a or .lib) are present in your "lib" path.

Unless you have "installed" those libraries into your compiler's standard "include" and "lib" directories, you will have to use the "-I" and "-L" compiler flags to add the required directories.

Furthermore, unless you have compiled all the "external libraries" yourself, i.e. when you have downloaded pre-compiled .a or .lib files, you must make sure these are compatible with your compiler (version)!

Finally, the software you are compiling may not work with arbitrary versions of the "external libraries". Often a certain minimum version is needed. Though the very latest version may not be supported yet.

After all, running the "./configure" script should check which of the (optional) libraries are in place and also whether they have the "right" version.

If "./configure" doesn't recognize a library that you have installed (and thus expected to be recognized), then you may look at the "config.log" file to see what exactly went wrong...

[sirt]

Yes, well I don't really know the reason but the libraries seem to be recognized after ./configure because they are enabled ("yes") but during the compilation I get errors which doesn't happen with regurlar x264 source. I will have to look by myself I think but, as we already discussed in another thread, I think it's a waste of time for me to try to compile all those libraries, am I wrong ? I just encode samples and don't decode anthing with x264.exe in fact.

Another question : I've seen tMod features --weightp 3 , do you know what it is ?

[LoRd_MuldeR]

Another question : I've seen tMod features --weightp 3 , do you know what it is ?

I didn't know, but I just had a look at the source code (help screen) to figure it out. You could have done the same :devil:
https://github.com/astrataro/x264_tMod/blob/tMod/x264.c#L853

AFAIK, the "k-means" mode was dropped at some point because it had some issues. And it has not been finished and re-added since then...

This entry from the x264 TODO list probably is related:
Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.

[sirt]

You could have done the same :devil:
https://github.com/astrataro/x264_tMod/blob/tMod/x264.c#L853



Yes, it's what I did but it didn't mean anything to me. Then, I deduce it is again unfinished. I guess those are the drawbacks of free lance programs : many people can interfer, modify the code and even disagree one with another when it is about to commit or not.

[LoRd_MuldeR]

It's like with every "real world" software:

Not every idea that seems promising in theory will turn out to be feasible and beneficial when you actually implement it and try to integrate it with the existing code/design.

Also, unless somebody steps up to implement+integrate+test a new feature (or is willing to pay for the new feature), it won't be done. It's as simple as that ;)

(Hacking in a load of unpolished features or features of questionable gain will result in an unmaintainable mess. New stuff needs to be tested carefully. Often "less is more".)

[h3nry]

Third point : what if I use subme 11 and treillis 0, AQ 0 ? Does that have a sence ?

but subme>9,requires trellis=2,isn't it?or else no matter what you set for subme,it will fall back to 9 unless trellis=2

[LoRd_MuldeR]

but subme>9,requires trellis=2,isn't it?

You are right:
x264 | branch: master | Jason Garrett-Glaser <darkshikari at gmail.com> | Thu Jul 23 12:20:39 2009 -0700

Add QPRD support as subme=10
Refactor trellis lambda selection to be done in analyse_init instead of in trellis.
This will allow for more easy adaption of lambda later on; for now it allows constant lambda across variable QPs.
QPRD is only available with adaptive quantization enabled and generally improves SSIM and visual quality.

[benwaggoner]

Am I correct in thinking that aq-mode 3 is simply aq-mode 2 except with relatively lower quants in dark areas? Since that's a common quality issue in our post perceptually uniform gamma era (darn LCDs), it sound quite promising.

Does anyone have any theoretical or experimental reason why aq-mode 3 would be worse than aq-mode 2 for general use?

[sirt]

Well I gave a try to aq-mode 4 / subme 11 / fgo 6 + preset very-slow by using tMod build and it took me...8 hours to encode a small video from Vimeo without any type of enhancing. It only took half of the time with aq-mode 2 /subme 10 + preset very-slow by using regular build. And I don't really see anything different with my eyes.

It's not the first time I've used the tMod build but it has always been incredibly slow.

[LoRd_MuldeR]

What do you want to learn from changing multiple options at the same time ???

If you want to learn whether a specific option is beneficial or harmful (or has no noticeable effect), you should compare encodes which differ in exactly that option - and nothing else.

It's even problematic to compare encodes that have been created with different revisions of x264, because other things might have changed "behind the scenes" and thus the results might not be comparable anymore.

Also both files need to have the exactly same average bitrate (file size) in order to make the comparison "fair". Comparing the quality of files with different size is kind of pointless, so use 2-Pass mode for the test!

Finally, you cannot expect any quality "enhancement" from re-encoding! It's in the nature of lossy compression that it always degrades quality. At best, the quality loss can be minimized...

(And if you can't see a difference between the two options you are comparing, it's quite possible that the chosen bitrate simply was too high)

[sirt]

By enhancement I mean using an avisynth filter to reduce noises, grains and things like this.

[LoRd_MuldeR]

By enhancement I mean using an avisynth filter to reduce noises, grains and things like this.

That doesn't make much sense to me :confused:

Above you said that you compared the "tMod" build against a "regular" build of x264, but didn't see any enhancement, compared to the source.

But of course you won't see an enhancement - compared to the source. Instead both re-encodes will look worse than the source! That's the nature of a lossy compression ;)

The real question is: Which one of the two re-encodes has the smaller quality loss - at the same target bitrate ???

And if both re-encodes look pretty much indistinguishable from the source, it means that the chosen bitrate was too high. In that case, repeat the test with lower bitrate...

[sirt]

In fact I was just doing random tests, I didn't want to compare much more one build with another.Moreover, I was simply pointing out I often noticed tMod is really slow.

Then, about using an avisynth filter : I've read many times that applying a filter such as a denoiser can help compression ; I don't understand what that means. I gave a try to mctemporal denoise applied on an entire video + very slow preset but - when doing this on my home computer - it was totally insane with HD content (1920 x 1080). First pass would took 23 hours in one case and an eternity in another. Of course it depends on the video, settings ect but in what what way applying a filter would help compression ?

[LoRd_MuldeR]

In fact I was just doing random tests, I didn't want to compare much more one build with another.Moreover, I was simply pointing out I often noticed tMod is really slow.

Comparing different builds and, at the same time, using different settings for each build, means you don't learn much :p

If one of the encodes looks better/worse, what do you know? Is it because of the different build? Or is it because of the different settings used ???

Maybe you could have gotten the same effect by just switching the settings instead of using a different builds. Maybe not. Who knows?

Then, about using an avisynth filter : I've read many times that applying a filter such as a denoiser can help compression ; I don't understand what that means. I gave a try to mctemporal denoise applied on an entire video + very slow preset but - when doing this on my home computer - it was totally insane with HD content (1920 x 1080). First pass would took 23 hours in one case and an eternity in another. Of course it depends on the video, settings ect but in what what way applying a filter would help compression ?

Now that's a completely different topic! :rolleyes:

Anyway: Video compression works by eliminating redundant information. At the same time, noise, by definition, is a random signal!

Consequently noise is very hard to compress. It's the nemesis of the video compressor. It means you will need a very high bitrate, if you want to preserve the noise.

Removing the noise beforehand, e.g. by using one of the many Avisynth denoisers, can make the video more "compressible", i.e. you might be able to get away with a lower bitrate.

However denoisng can easily cause other defects! Bad denoisers cause "ghosting". And strong denoising kills the details. Denoisng can also destroy the "authentic" film look.

(And indeed: Good motion-adaptive denoise filters are slow. As always in life, nothing is for free. But the denoise filter might have options to tweak it more for speed)

[jpsdr]

Well I gave a try to aq-mode 4 / subme 11 / fgo 6 + preset very-slow by using tMod build and it took me...8 hours to encode a small video from Vimeo without any type of enhancing. It only took half of the time with aq-mode 2 /subme 10 + preset very-slow by using regular build. And I don't really see anything different with my eyes.

It's not the first time I've used the tMod build but it has always been incredibly slow.

In my post, i've report that MasterNobody said he gave up on mode 4, because curve was indeed too close to mode 2. If you want to test, test mode 3, speed for mode 2 or 3 is the same.

[sirt]

jpsdr, I tested mode 3 & 2 but I'm not able to see the difference ; how could I notice the difference between mode 3 and 2 or at least mode 1 & 2 ? I don't know what or where to look.


LorD_MulDer : in fact I used --tune grains or --fgo or even both, what do you think of these ? I have some grainy streams to test.

[LoRd_MuldeR]

jpsdr, I tested mode 3 & 2 but I'm not able to see the difference ; how could I notice the difference between mode 3 and 2 or at least mode 1 & 2 ? I don't know what or where to look.

Pick a build that supports both modes you want to compare and create two encodes from the identical source:
One with mode 2 and one with mode 3. Either that or one with mode 1 and one with mode 2.

Also use 2-Pass mode with the identical bitrate for both encodes. And, apart from AQ mode, do not change any other options.

Finally you have to pick a bitrate that is low enough to make the results (re-encodes) look different from the source!
Though not so low that both results look equally messed up ;)

Then it's up to your eyes to decide whether encode #1 or encode #2 looks better. Or whether the difference is negligible...

(And I wouldn't start with a "low quality" source already full of artifacts. Use something "ordinary" like Black Perl (http://www.mediafire.com/?mymhmje0iki))

in fact I used --tune grains or --fgo or even both, what do you think of these ? I have some grainy streams to test.

The "--tune grain" is intended for grain preservation at very high bitrates. Besides other things, significantly reduces Deblocking and AQ strength.

[sirt]

Okay, so here you go with your sample !

Settings :

x264.2230kMod.x86.exe --preset veryslow --profile high --level 3.0 --aq-mode 2 --bitrate 2000 --pass 1 -o black_aq2_pass1.mkv black.avs
x264.2230kMod.x86.exe --preset veryslow --profile high --level 3.0 --aq-mode 2 --bitrate 2000 --pass 2 -o black_aq2_pass2.mkv black.avs


x264.2230kMod.x86.exe --preset veryslow --profile high --level 3.0 --aq-mode 3 --bitrate 2000 --pass 1 -o black_aq3_pass1.mkv black.avs
x264.2230kMod.x86.exe --preset veryslow --profile high --level 3.0 --aq-mode 3 --bitrate 2000 --pass 2 -o black_aq3_pass2.mkv black.avs

pause

AVS Script :

DirectShowSource("C:\Users\AQ_TEST\Black.Pearl.Sample.m2v")
trim(319,500)


Results :

http://www.sendspace.com/file/lt2l1q

Then, it looks like aq_mode3 is somewhat sharpening, but I may be wrong.