Some explanation about new x264's features

[max-holz]

1) Trellis quantization value=2 is good for High Profile using an high bitrate or degrade video quality?

2) What's RDO?

[Sharktooth]

1) 2 is an "insane" value... however trellis (1 or 2) should be always good (at least for PSNR). the difference between 1 and 2 is not so big... but 2 kills the encoding speed.

2) :search:

[bond]

1) 2 is an "insane" value... however trellis (1 or 2) should be always good (at least for PSNR). the difference between 1 and 2 is not so big... but 2 kills the encoding speed.my small trellis analysis:
http://forum.doom9.org/showthread.php?p=733643#post733643

[CiNcH]

What about not using Trellis R-D or Adaptive Quantization in a fast first pass? Do those features have big impact on rate-control?

[charleski]

2) What's RDO?
A question which seems to have been asked several times (or at least I can remember thinking the same question several times :) ). Given the context, it seems that it's a reference to Rate-Distortion Optimization. As far as I could find out, this term originated with this (http://www.lnt.de/LMS/publications/web/lnt1998_041.pdf) 1998 paper by Sullivan and Wiegand (I found the link through google, but the site was down when I tried to get the paper). An abstract of this paper can be found here (http://ieeexplore.ieee.org/xpls/absprintf.jsp?arnumber=733497).
If you rummage around a bit there are plenty of other papers available that discuss Rate-Distortion Optimization, one example is Xin et al 2004 (http://www.merl.com/reports/docs/TR2004-079.pdf).

As far as I can make out from looking at the code, RDO in the context of x264 appears to apply to the use of trellis quantization. Here are a couple of quotes concerning trellis quantization:
Trellis quantization optimizes the motion estimation following a rate distortion function. It slightly decreases encoding speed and leads to a slight quality loss along (but the decrease in used bitrate is considerable in comparison).
Trellis quantization is actually very dependent on bitstream format: It's purpose is to reduce some inefficiencies in Huffman coding of DCT coefficients. But H.264 uses Arithmetic coding, which doesn't suffer the same problems.
Trellis uses rate/distortion optimization, which means that it may decrease the quality of the video if it can save enough bits to be "worth it" (where the definition of worth depends on quantizer). Or it might have increased the quality and you could be underestimating the improvement by looking at only filesize. The tradeoff threshold is supposed to be calibrated such that it doesn't modify quality much, but it's not perfect, and can only be accurately measured in 2pass mode. (The same advice applies to all options, but especially vhq, bvhq, and anything related to B-frames.)
my own results on trellis for the usual matrix clip i test
...
speed:
no trellis: 3.63 fps
trellis 1: 3.34 fps (- 8%)
trellis 2: 3.05 fps (- 16%)

average psnr:
no trellis: 44.090
trellis 1: 44.203
trellis 2: 44.216

visual quality:

i am a little bit disappointed of trellis, because it wasnt able to really show me a visual quality difference compared to when using no trellis

BTW, if anyone could point me to a good technical paper on trellis quantization I'd be interested in taking a look.

[akupenguin]

The original paper about trellis quantization: http://www.ee.ucla.edu/~ipl/Publications/Journal/trellis-based_r-d_optimal_quantization_in_h263+.pdf
Trellis quantization in H.264 is not exactly the same algorithm, but I haven't published an updated paper yet ;)

[charleski]

Excellent! Thanks.

[lexor]

So wait, from what I'm reading Trellis is supposed to make coding more efficient without deviating quality too much. So why was bond complaining about not seeing quality improvement? I mean if he didn't increase his settings he shouldn't see any quality improvement just a slight fileseize reduction, right?

[charleski]

As far as I understand it, trellis improves quality in xvid, but the effect on quality in h.264 is much less. Even on xvid, the differences aren't dramatic, but may be worthwhile. To quote the Wen et al. paper from akupenguin:We have presented an algorithm which performs the R-D optimal
quantization of DCT coefficients in the H.263+ video coding system
including run length and Huffman coding. With a fixed PSNR, the R-D
approach reduces the bitrate by approximately 3.5%, or equivalently
with a fixed bitrate, PSNR performance is improved by an average of
0.17 dBbond measured a 0.126dB improvement with trellis 2 in x264.

[foxyshadis]

So wait, from what I'm reading Trellis is supposed to make coding more efficient without deviating quality too much. So why was bond complaining about not seeing quality improvement? I mean if he didn't increase his settings he shouldn't see any quality improvement just a slight fileseize reduction, right?
If you keep the quality constant, it will reduce bitrate, so if you think about it, a constant bitrate would be like raising the quality by that amount. In this case it would save very little, so it also improves very little.

Bond is simply surprised that it doesn't help nearly as much as it does with xvid.

Is it possible to improve the trellis implementation at this point, or does it already give the optimal results here?

[DryFire]

Might as well add:

Do subme 7 (RDO 2), mixed refs and trellis work optimally together or will we still see improvement? By how much? I recall that the notes for subme 7 and mixed refs said performance together was sub optimal.