Few questions about x264

[Pier]

Hi everyone, i have a few questions that i'd like to make about x264.

-qcomp (without mbtree) increases the qp for the more complex parts of the image in the spatial domain since they need more bits for the coding while retaining the simplest parts since they need less bits for the coding. And this is applyed on every frame type, right?

-qcomp (with mbtree) does the same as above but instead of the spatial domain, the temporal domain is used instead. This allow to get a better visual quality since compression artefacts are harder to spot into fast motion scene where there is no time to focus on the details, right? Is this applyed only on the bframes or on every frame type? If it is applyed only on the bframes this means that for the other the spatial domain is used instead???

-aq-strength increases/decreases the qp on the flat zones, a value of 1.0 means that nothing changes, right? But if a qcomp with a value of 0.6 decides that the qp in this flat block must be increased while using an aq value of 0.5, which is calculated first? The qcomp or the aq value? Will that qp be increased 2 times? And even if i use an aq value of 1.0, will it still be affected by the qcomp?

-deadzone drops fine details since they are hard to code, right? My question is if for example i use --crf 10, will it be scaled to a lower value due the high bitrate in order to retain details or it will still drops the same ammount of details making a low crf value useless?

Assuming that i got how they works, i'd like to give some opinion about how they could be optimized, and if you have any, i'd like to hear yours as well.

-I noticed a problem about how x264 handle the dithering and the static grain, especially in anime, basically, into a fast motion scene where you give less bitrate on the edges since there is no time to focus on them, is not the same for the flat zones. The compression artefacts in the flat zones are easy to spot even if you don't have the time to focus on them, this gives a "wrong" concept of "visual quality" since you will see your dithering or your layer of static grain destroied and thus the banding that they are hiding will come out, and in order to fix that aq-strength must be increased, but doing so the bitrate will be also incrased in the static zones where is not needed. I don't know if it possible to code, but i think that if we could use a qcomp value for the flat zones and a qcomp value for the no-flat zones, this could be avoided, saving more bitrate with the same visual quality, this should also avoid varius problemens related to the fades.

-Same as above, but related to the chroma. I noticed that while trying to encode some 4:4:4 video. This source was full of chroma banding while the luma was not, to keep the dithering made by the debanding filter i had to increase the aq-strength value, doing so the bitrate given to the luma was increased as well, and the filesize of the ouput was not great. Using the chroma-qp-offset instead of the aq-strength was even worse since the bitrate given to the edges was increased as well. Another time instead, this source was full of red obects with sharp edges, and the AVS preview looked awesome. The problem was that those edges where blurred by the low bitrate given due the qcomp throwing away all the good that a 4:4:4 video could get compared to a 4:2:0 video. Thus if we could use a different aq/qcomp value for the chroma, this should allow the user to get a better visual quality.

-I have also noticed that the --deadzone is the first responsabile for the wrong allocation of bitrate in the blocks with a low luma value, while they do a pretty good job on the blocks with an high luma value, thus, if it was possible to set a different deadzone value based on the luma value, something like 256:128:64 so that you could set the deadzone to 21:11:6. Also having the deadzones working on edges and flat zones with differents values would be nice as well, especially for anime, same for the chroma.

I guess this is pretty much everything, thanks in advance and sorry for the bad english :)

[Dark Shikari]

Hi everyone, i have a few questions that i'd like to make about x264.

-qcomp (without mbtree) increases the qp for the more complex parts of the image in the spatial domain since they need more bits for the coding while retaining the simplest parts since they need less bits for the coding. And this is applyed on every frame type, right?Not quite; --qcomp only applies to the temporal domain; frames that are more complex get fewer bits. Also, higher qcomp (more towards 1) means qcomp has less effect.

-qcomp (with mbtree) does the same as above but instead of the spatial domain, the temporal domain is used instead. This allow to get a better visual quality since compression artefacts are harder to spot into fast motion scene where there is no time to focus on the details, right? Is this applyed only on the bframes or on every frame type? If it is applyed only on the bframes this means that for the other the spatial domain is used instead???MB-tree applies to all frame types.

-aq-strength increases/decreases the qp on the flat zones, a value of 1.0 means that nothing changes, right? But if a qcomp with a value of 0.6 decides that the qp in this flat block must be increased while using an aq value of 0.5, which is calculated first? The qcomp or the aq value? Will that qp be increased 2 times? And even if i use an aq value of 1.0, will it still be affected by the qcomp?AQ and qcomp are completely orthogonal. 1.0 doesn't mean "nothing happens", it means "use the default strength, which is hardcoded to 1.0".

Actual MB QP = RC QP + MBtree QP offset * (1 - qcomp) + AQ QP offset * (AQ strength)

-deadzone drops fine details since they are hard to code, right?This description is probably too much of an oversimplification to meaningfully make decisions about how to use the feature. I'd recommend not playing with it unless you understand how quantization works, and even then there's really not much of a reason to poke at it. Additionally, when trellis is on, deadzone isn't used (and if you want to bias heavily towards detail retention at the expense of quality, psy-trellis is probably a good start).

Keep in mind that a lower deadzone value can very much mean more loss of detail; this is probably hard to explain why without the rest of the quantization process.

...
Maintaining dither is an extremely difficult problem; there's no magic bullet, and generally the only solution is to throw an unreasonable number of bits at the encoder. You're probably a lot better off using 10-bit instead.

-I have also noticed that the --deadzone is the first responsabile for the wrong allocation of bitrate in the blocks with a low luma value, while they do a pretty good job on the blocks with an high luma value, thus, if it was possible to set a different deadzone value based on the luma value, something like 256:128:64 so that you could set the deadzone to 21:11:6. Also having the deadzones working on edges and flat zones with differents values would be nice as well, especially for anime, same for the chroma.This doesn't really have anything to do with deadzone; I think what you're looking for is luma adaptive quantization. The problem with this -- and the reason I've never committed anything to that effect -- is that its primary purpose ends up being to compensate for badly calibrated monitors that display dark colors too brightly (thus amplifying artifacts). If the monitors were at least consistently bad, it might be solvable, but they tend to all be different, so any particular choice of algorithm parameters tended to look great in some cases and terrible in others.

It's not inherently a bad idea, but I was never able to come up with something consistently effective. Patches are welcome, though I'd preferably like at least some academic justification for the luma quantization curve used (the ones I've seen in literature actually say to make quality worse in dark areas, which is exactly the opposite of what you want -- another reason why I was lax to commit anything!).

[Pier]

Thanks for the answers, now is all clear :)

[Pier]

I will ask here instead of making another post since it is still related.
I'd like to know the default CMQ file that is used by x264-10bit (with trellis 0) for each of those http://mewiki.project357.com/wiki/X264_Settings#cqmfile

[Dark Shikari]

The default CQM is the flat CQM (all 16s).

[Pier]

Thanks again :cool:

[smok3]

It's not inherently a bad idea, but I was never able to come up with something consistently effective. Patches are welcome, though I'd preferably like at least some academic justification for the luma quantization curve used (the ones I've seen in literature actually say to make quality worse in dark areas, which is exactly the opposite of what you want -- another reason why I was lax to commit anything!).

Is
Contrast frame = less bits for dark areas
implemented?

[Pier]

If for example i have (of course it dosn't make sense but it is* for practicality)

INTRA8X8_LUMA =
0xy,01y,02y,03y,04y,05y,06y,07y,
01x,2xy,3xy,4xy,5xy,6xy,7xy,7xy,
02x,3xy,4xy,5xy,6xy,7xy,7xy,8xy,
03x,4xy,5xy,6xy,7xy,7xy,8xy,8xy,
04x,5xy,6xy,7xy,7xy,8xy,8xy,9xy,
05x,6xy,7xy,7xy,8xy,8xy,9xy,9xy,
06x,7xy,7xy,8xy,8xy,9xy,9xy,9xy,
07x,7xy,8xy,8xy,9xy,9xy,9xy,9xy

And i want to use --cqm8i instead of --cqmfile, do i have to start from left to right
--cqm8i 0xy,01y,02y...,01x,2xy...
or with the zig zag order
--cqm8i 0xy,01y,01x,02x,2xy,02y,...04x,4xy,4xy,4xy,04y...
I don't know if is clear what i mean, but thanks in advance.

[Dark Shikari]

It should be raster order (not zigzag).

[Pier]

It should be raster order (not zigzag).
This means --cqm8i 0xy,01x,01y,02x,2xy,02y... right? Sorry if i'm asking but i i want to be sure since the raster order can be also from left to right...:confused:

[Dark Shikari]

I'm not sure what you mean; there's only one raster order (http://en.wikipedia.org/wiki/Raster_scan#Scanning_pattern), and that's left to right -- it's the one you posted above under the label "INTRA8X8_LUMA =".

[Pier]

I'm not sure what you mean; there's only one raster order (http://en.wikipedia.org/wiki/Raster_scan#Scanning_pattern), and that's left to right -- it's the one you posted above under the label "INTRA8X8_LUMA =".
Sorry i was confused :D Thanks :)

[Pier]

Are deadzones used to lower dct values in the source so that they can be discarted by the values from the qm during the check without the need to use high value so to retain other type of information that are not much affected by deadzones or i was not even closer?

[Dark Shikari]

"Deadzone" is a simple scalar rounding algorithm in which rounding towards zero is more likely than rounding away. For example:

[0.0,0.7) -> 0
(-0.7,0.0] -> 0
[0.7,1.7) -> 1
(-1.7,-0.7] -> -1
[1.7,2.7) -> 2
(-2.7,-1.7] -> -2
(and so on)

would be a deadzone algorithm. The exact threshold depends on the deadzone setting.

The primary reason deadzone is useful is because 0s are a lot cheaper than 1s bit-wise, so getting more 0s improves compression. While not at all smart or particularly RD-optimized, it's also very fast (unlike trellis).

[Pier]

Thanks for the explanation :)
Now that made me curious about which algorithm is used by dct-decimate to choose which blocks are unnecessary, since now i'm looking at the QMs that i was making with different eyes :D

[Dark Shikari]

DCT decimate is probably best understood by reading the code; it basically scores each block based on the run lengths and if the total score is lower than a certain amount in an 8x8 or 16x16 block, it decimates it. If any coefficient's absolute value is larger than 1, it terminates early.

[Pier]

That would mean that the higher the deadzone value the easier the chance that the block is decimate, right? Also, in order to understand if i got it or not, a deadzone value of 32 would mean always the closest value to 0, while a deadzone value of 0 would mean always the outmost value from 0, thus a value of 16/17 would mean to round to the closest value, that means "deadzone is disabled", right?

[Dark Shikari]

No, a deadzone of 0 means "round to nearest", while a deadzone of 32 (I think) means "round down".

[Pier]

I see, now i have something to work on, thanks :D

[Emulgator]

I found x264 deadzones left on default (21,11) hurting dark details as well as Pier,
so from the beginning my suggestion was to use --deadzone-inter 0, --deadzone-intra 0.
(My main purpose here: Blu-ray)

For the chroma issue: On launching of TMPG Authoring Works 5 (using x264)
I spotted TAW5's x264 commandline using a --chroma-qp-offset of -2,
which I liked and used since then.

Furthermore: From version 0.25 on Hank315 has successfully introduced an algorithm for his MPEG-2 encoder HC
(called "luminance gain") which allocates more bits to dark scenes by adapting matrix.
Matrix values are decreased for dark scenes. This helped a lot in keeping dark detail blockfree.

Maybe this would be a feature worth implementing into x264 ?

On a Sonic show here in Berlin I remember a Sonic representative suggesting to clip close to black values to black.
Back then I thought: Ouch. The other ones in the audience seemed to feel the same.
This because their encoder (Cinevision back then) might not handle it correctly ?

Please not. Lets rather work on these fine details, these are worth it.

[Dark Shikari]

Maybe this would be a feature worth implementing into x264 ?
The problem with this -- and the reason I've never committed anything to that effect -- is that its primary purpose ends up being to compensate for badly calibrated monitors that display dark colors too brightly (thus amplifying artifacts). If the monitors were at least consistently bad, it might be solvable, but they tend to all be different, so any particular choice of algorithm parameters tended to look great in some cases and terrible in others.

It's not inherently a bad idea, but I was never able to come up with something consistently effective. Patches are welcome, though I'd preferably like at least some academic justification for the luma quantization curve used (the ones I've seen in literature actually say to make quality worse in dark areas, which is exactly the opposite of what you want -- another reason why I was lax to commit anything!).Like I said before, patches are welcome. Simply saying "let's make dark areas better" isn't enough; I need an actual, detailed algorithm with mathematical justification.

[Pier]

I found x264 deadzones left on default (21,11) hurting dark details as well as Pier,
so from the beginning my suggestion was to use --deadzone-inter 0, --deadzone-intra 0.
(My main purpose here: Blu-ray)
For the chroma issue: On launching of TMPG Authoring Works 5 (using x264)
I spotted TAW5's x264 commandline using a --chroma-qp-offset of -2,
which I liked and used since then.
I agree with the solutions, but my point was "we need the capacity to give more bits where we can see the difference to save filesize, thus quality". In my opinion an high deadzone value on the flat zones with an high luma value dosn't change much the result most times while it changes a lot the filesize, while a very high deadzone value on the edges looks bad becouse it looks blurred while a very low value makes it look bad, due DCT's fake sharpening effect and the edges artefacts enhancement. The same for the chroma, what looked bad was on the edges, thus i would get a better visual quality in the same filesize if i could increase the bitrate/precision only where the video looked bad.
Furthermore: From version 0.25 on Hank315 has successfully introduced an algorithm for his MPEG-2 encoder HC
(called "luminance gain") which allocates more bits to dark scenes by adapting matrix.
Matrix values are decreased for dark scenes. This helped a lot in keeping dark detail blockfree.

Maybe this would be a feature worth implementing into x264 ?
Personally i think that x264 with --deadzone-intra 0 --deadzone-inter 0 and --trellis 0 with the default matrix does a perfect job on the blocks with a low luma value, but it is also pretty bad when we talk about filesize. And since the default matrix can keep those detail, i think that to use lower values on them would just create artefacts thus loose more details, due the way how (i think) the dct works. My opinion is that a some kind of --cqmfile for the deadzones where instead of putting the dct value for that blocksize/frequency you could put the deadzone value that you need on that blocksize/frequency, this way you could use a low value to retaind the noise/dither while using an halfway value for the edges, plus you could set a value in that file where you could say the luma/chroma value where it needs to be applyed, something like: if anything has a lower value, it is not processed with those settings, so you could specific any deadzone value at any luma/chroma value. This would fix both my problems, becouse it is not about "we need to incrase the precision used to retain the details" but is more like "we need to incrase the precision used to discard * the details". I don't even know if what i said is possible, but to guess dosn't hurt anyone :D

[Emulgator]

I just PM'ed hank to invite him to this thread.
He can code, I can not ;-)

[Marin85]

Re dark areas: I am surprised no one mentions Haali AQ. It has been around for ages now and it takes care of dark&&blue. It is already effective at moderate strength without excessive quality penalty to other elements.

[Pier]

Re dark areas: I am surprised no one mentions Haali AQ. It has been around for ages now and it takes care of dark&&blue. It is already effective at moderate strength without excessive quality penalty to other elements.
It is not about bitrate but about details, i will try to explain what i mean with this my theory/opinion. If you look at the sun, you won't last much, since the light is too strong, making you impossible to focus on it, the same happens when you read something with a pure white background and a text with sharp edges and bright colors. The pure white background and the contrast with the edges makes your eyes hurt if you focus too much, due to that your eyes are unable to get used to bright lights, while is not for dark lights. If you turn off the lights of your room, you won't be able too see anything at first, but after some time you will be used to the darkness and you will start to spot things, becouse your eye can focus with that ammount of light and it will get used to that condition. The same happens with monitors, the higher is the luma value the harder becomes for your eyes to focus on it, while is easier with a low luma value. This makes you to prefer a block without details instead of a detailed block with a notable ammount of artefacts when it has an high luma value, since it is harder to spot fine details, while is the inverse with low luma values, since a block without details and witout artefacts is more ugly than a block full of details and full of artefacts (--tune grain is a conceptual example). Putting aside the things about the edges, the point/problem is the way how trellis and deadzone are meant to work, if you use them you will get the wanted result on the blocks with high luma values, while if you disable them you will get the wanted result on the blocks with a low luma value and you will have to make a choice, to save filesize and destroy the dark blocks, or to bloat the filesize to save the dark blocks. If it was possible to set differents value on differents luma values, that could be fixed, while looking for something "generic" it would end up on forcing a subjective concept of visual quality on the user, while is not possible, like --aq-mode 3 that should gives more bitrate to the blocks with a low luma value, but that dosn't not allow to change "how much" is given to the blocks with a low luma and the blocks with an high luma. Thinking about it, something like --zones where you set the luma range instead of the frame range could be a better solution? As above, to dream dosn't hurt anyone :D

[Dark Shikari]

the point/problem is the way how trellis and deadzone are meant to work, if you use them you will get the wanted result on the blocks with high luma values, while if you disable them you will get the wanted result on the blocks with a low luma valueThis is mathematically equivalent to lowering lambda on darker blocks, which is again equivalent to lowering the quantizer.

If you think you want "lower deadzone in darker blocks", the real thing you're asking for is "lower lambda/quantizer in darker blocks".

[Pier]

Not sure if i got what you mean with "lowering the quantizer" but if i encode a video with
--qp 24
vs
--qp 24 --trellis 0 --deadzone-intra 0 --deadzone-inter 0
The difference is that the first has a lower filesize/bitrate with the same quantizer of the second, that has a larger filesize/bitrate but with more details. Thus, i believe i want an higher quantizer in darker blocks (more compression artefacts) but with an higher detail retention.

[Dark Shikari]

The lambda is the value that trades off between detail retention and detail removal -- the relative value of quality vs bits. Lowering deadzone is effectively equivalent to lowering lambda, because you're telling x264 to use a different tradeoff. You can get exactly the same effect with trellis by changing its lambda (see the tables in encoder/analyse.c).

The quantizer choice is largely dependent on the lambda; if you lower the lambda, it will often be better for the encoder to pick a lower quantizer, in that it will give strictly better results at that lambda. Lowering lambda but forcing the quantizer to be the same is a particularly bad idea, because you're basically telling the encoder to be inefficient for no reason. In short, in an ideal encoder, lambda is the independent variable, quantizer is the dependent variable. x264 (except with QP RD) takes a bit of a shortcut here, but the concept is mostly the same.

On a practical level I find that in dark and flat areas, leaving the quantizer high actually makes it physically impossible for the encoder to retain details, because those details don't fit in the quantizer step.

If you believe that encoders really do need to use a lower lambda relative to QP (and I can see a reason or two to consider this; it's not insane), please write a psychovisual metric for RDO that causes this to happen (and post a patch); simply forcing the lambda lower on its own won't make things better if the encoder's algorithms still believe otherwise.

[Pier]

Question:
Trellis picks the most suitable trade off between detail retention and detail removal for the block, right? Could you please tell me a pratical example where it would decide to retain more details and another example where it would decide to remove more details?

[Dark Shikari]

Trellis is RD-optimal quantization; it picks the quantization choice that optimizes RD = BITS * LAMBDA^2 + SSD. Deadzone is an attempt to (very very simply) approximate the results of trellis; trellis is much smarter and will make better choices for the same value of lambda because it's aware of the interdependencies between coefficients.

Here's a very hypothetical example, I guess:

8.7, -2.2, -0.8, 0.7, -0.7, 0.3, -0.5, 0.4, -0.6, 0.3, 0.9

Deadzone might give:

8, -2, -1, 0, 0, 0, 0, 0, 0, 1

Trelliis might give:

9, -2, -1, 1, -1, 0, 0, 0, 0, 0

because it understands that coding that final 1 is very costly because of the long zero-run, and omitting it will save many bits, which it can then use to improve the earlier coefficients. Deadzone is extremely fast and simple and does not have this intelligence.

There's better approximations than deadzone; libvpx has an algorithm (if I recall correctly) that's somewhere in between, attempting to consider run length costs while still not doing a full trellis. It's a lot slower than deadzone, but naturally still an order of magnitude or two faster than trellis.

[Pier]

I see, thanks :)