CAVLC Trellis

[DeathTheSheep]

Since x264's trellis isn't implemented in CAVLC mode, and folks here have mentioned that CAVLC and CABAC can be converted to one another losslessly anyway (but no tool exists), is there a particular reason why Trellis cannot be enabled for CAVLC encoding? This would greatly improve playback quality on mobile devices or baseline-type players that need the speed of CAVLC playback. :)

[Manao]

Shortly said, no, it wouldn't.

Trellis is efficient only when computed with the entropy coding used. If you compute trellis for CABAC and then convert it to CAVLC, you're likely not to take the proper decisions.

Anyway, even for CABAC, trellis doesn't greatly improve things, so it's not a big loss.

Edit: The same is true for RDO, to a lesser extent. Since CAVLC & CABAC sizes are loosely correlated, RDO will still work, but will be less efficient if you compute them for CABAC but encode in CAVLC. Trellis is also a kind of RDO, but a much finer one, for which I think the correlation isn't good enough to allow it to work.

[akupenguin]

To put it another way:
If CABAC trellis had helped on a CAVLC encode, then I wouldn't have disabled trellis in CAVLC mode.

[DeathTheSheep]

Gotcha. Anyways, keep up the good work :)