why is it not possible (To use Trellis with deadzone options)?

[Terranigma]

As the question states, why is it supposedly not possible to use trellis with the deadzone options? I added deadzone options for inter and intra in my commandline with trellis-1 and 2, and avinaptic shows my lowered deadzone setting and trellis being used. :devil:

Inter / Intra luma quantization deadzone: Also known as inter/intra deadzone. Specifies the level of detail to be discarded and not taken into account by the encoder. Lower values discard less information, and therefore hand the encoder a more complex stream. The main use of these settings is to set a detail threshold. Lower settings can help grain retention, while higher settings may reduce noise. Use of deadzone settings requires trellis to be turned off.

[HeadBangeR77]

I've been wondering as well, why it's not possible (not that I'm a big fan of trellis :P). I can understand the approach of using dead zones with e.g. M4G's MRM and HRM matrices, and not to use trellis, to preserve some fine small details and grain. However, why it's technically not possible, if at all, I've got no clue. :confused:

According to my beloved avinaptic the flat matrix uses 21,11 for dead zones, and trellis can be safely use. So is it not: "dont' mess with dead zones when using trellis!" instead of "both exclude each other"???

[Terranigma]

Here's a screenshot I took of me using trellis-1 and a lowered deadzone setting.
http://img368.imageshack.us/img368/3368/trellisanddeadzonetf9.png

:D

[akupenguin]

There are two different quantization algorithms: deadzone and trellis. Any given dct of a given block can only use one of them. Because, well, if you had two different values for each coefficient, what would you do with them?
--deadzone-inter and --deadzone-intra tweak a parameter of the deadzone algorithm. (The parameter in question is "deadzone", that's where the algorithm got its name.) --trellis determines when to use the trellis algorithm (0=never, 1=sometimes, 2=almost always); any dcts that don't use trellis must use deadzone instead.
So, the fact that --deadzone* still has some minor effect when you enable trellis does not contradict the fact that deadzone and trellis are incompatible.
Don't do that. If you enable trellis, then deadzone is only used as a fast estimate of trellis, so a reduced value of --deadzone does not provide any grain retention, it just reduces the precision of that estimate and thus of the mode decision process. Sure I could make x264 ignore --deadzone* if --trellis is enabled, but that would contradict my policy of "give the luser enough rope to hang himself".

That said, it might be possible to tweak trellis in a way that has similar results to --deadzone*. In which case --deadzone* could apply to trellis too, despite the name.

BTW, just because an option is printed in the userdata sei doesn't mean it had any effect on the encode. I remove some of the extraneous options (e.g. most of ratecontrol when using cqp), but still some options may be printed even if they're overridden by some other option.

[Terranigma]

it might be possible to tweak trellis in a way that has similar results to --deadzone*. In which case --deadzone* could apply to trellis too, despite the name.
I think that'd be awesome. :)
btw, thanks for clearing this up, it's been nagging at me for quite some time. :cool:

[*.mp4 guy]

(not that I'm a big fan of trellis :P)
I rather like trellis, myself, but since it isn't possible to set a deadzone(like) parameter for it, it would take more bitrate using trellis to get the desired grain/detail retention in some situations, mostly at high bitrates.

This also seems like as good a place as any to ask about "Quantizer noise shaping" (from ffmpeg) which sounds a bit like trellis, whats the difference? I read somewhere that "qns" uses a special dct transform, which would indicate to me that it is different then trellis, which only uses a speciall method of dropping dct coeficients.

[akupenguin]

qns computes weighted ssd in the pixel domain, whereas trellis computes ssd in the dct domain. Without weights the two are the same (modulo idct rounding), but with weights qns can do more precise spatial complexity masking.

[TheRyuu]

If your using AQ with Trellis, which quantization method would take precedence?

Or would the quantization use Trellis first, then tweak the quantization values with AQ? Or am I just totally wrong here. :p (am I trying to compare apples to oranges?)

I'm attempting to learn how each of these works and I'm doing several test to see what I like best. Just wondering how it works.

Thanks.

[akupenguin]

AQ is not a quantization algorithm. AQ is more related to ratecontrol: it chooses QP, which is an input to quantization.

[Sagekilla]

AQ is not a quantization algorithm. AQ is more related to ratecontrol: it chooses QP, which is an input to quantization.

Hmm.. this piqued my interest a bit. If the OP doesn't mind me asking, would that mean the output of AQ would be used for Trellis input?

[akupenguin]

Quantization (no matter whether it's deadzone or trellis) takes some DCT coefficients and a qscale (qscale = 2^(QP/6)), and divides the coefficients by the qscale. Deadzone and trellis differ in how they round the result of the division.
Ratecontrol chooses QP for a whole frame. If AQ is enabled, then AQ modifies the QP for some macroblocks.