Wierd treillis behavior ?

[DarkZell666]

After having searched for the word "treillis" in the MPEG4/AVC forum and not finding anything much relevant about this problem, I'm reposting something I posted in the "AVC (X264) stupid questions" thread about some wierd behavior with treillis :

just a remark about treillis :

I tried several --crf 22 encodes with a couple of differing settings,
and re-did them all with treillis 1 enabled ... the filesize of the treillis version of each encode was always bigger, but I couldn't see any quality improvement (well ok it was bicubicresized 512*384 anime so It was a bit blurry, but still...).

Since someone said earlier that treillis allowed more compression of data I wanted to understand : why I could witness the opposite effect ?

A non exhaustive list of the settings I used :

CRF22, 6 bframes with bframe ratio = 1.6, RDO for Bframes and --subme 7, -minqp 20, --qcomp 0.7, CABAC enabled (obviously ), weighed biprediction, --bime, chromaQPoffset 4 ... the rest is pretty much the default settings.

Just one question :
- Should treillis be any more efficient in high/low motion scenes or does it simply not matter ?

Just trying to understand the "wierd" figures I've observed recently


And a reply from OvejaNegra, stating the same fact :

THE SAME HAPPENS TO ME!!!
I always encode with fixed quantizer and trellis makes my files biggers.
I have the same question: IS TRELLIS REALLY USEFUL?

BTW:: using Xvid with a fixed quantizer and trellis, makes my files smallers..
a bug in x264??

Any suggestions ? Any other conflicting options with which treillis shouldn't be used ?

[unmei]

Try searching for TRELLIS (without a "i" after the "e" :P)

[OvejaNegra]

At the moment i´m encoding without trellis, it really makes my files largers....

The same happens to you...

[Manao]

the filesize of the treillis version of each encode was always bigger, but I couldn't see any quality improvementEncode a movie twice, once with a bitrate XXX, then once with a bitrate XXX + 1%, then try to spot a difference. You can't, it's normal. It's the same with trellis. It marginaly increases / reduces the quality / the bitrate. In the overall ( ie if either the quality is fixed, or the bitrate ), the result will be better ( ie the size smaller / the quality higher ). But it's hard to judge.

And there's also a post by bond, where he states his visual impression ( size was constant iirc ), and where trellis didn't come out as a big win.

[DarkZell666]

well making the size bigger @ same quality is pretty useless I think, which means that if the filesize had to stay the same, the quality wouldn't be as good ...


But it's hard to judge.


Sure, the size increase wasn't that dramatic but still, it's a loss ...
Though I only tried with --crf 22 on anime with specific settings, I'll probably do a couple of tests using other material (3D movie, real-life footage, etc) and other modes (bitrate, --cqp, --crf with no bframes, etc ...) to figure all this out later =)

[Manao]

well making the size bigger @ same quality is pretty useless I think, which means that if the filesize had to stay the same, the quality wouldn't be as good ...Ok, I obviously didn't make myself understood.

Trellis is an option that, with a fixed quantizer, or fixed pseudo quality ( crf ), can either raise or lower the quality, and lower or raise the bitrate. But, with a fixed size, trellis raises the quality, and with a fixed quality ( here, quality = PSNR ), trellis reduces the size.

Now, in your case, the filesize is bigger. It means the (mathematical) quality is higher. You can't notice it, because, as I said, your eyes can't see that an encode 1% bigger than another is better - all other settings being identical.

[DarkZell666]

I do get the point, it's just some bitrate/quality "optimisation" (well... sort of ^^'), but I really can't understand the use of it (in my case for sure :p) ...

The other question that I believe your explanation raises is :
- In what cases does the file effectively come out smaller with treillis enabled ? oO

I'll test all this extensively this w-e since I'll have LOADS of spare time =)
(anime, real-life footage, 3D animation, @ different resolutions and bitrates, bframes/no bframes, etc ...)

Stay tuned :)

[akupenguin]

In what cases does the file effectively come out smaller with treillis enabled ?
When you tell it to reduce the bitrate.
It's that simple. Don't expect a given --qp or --crf to always produce exactly the same quality, and especially don't measure the performance of any option by how much it affects bitrate at cqp.

[DarkZell666]

well well well ...

After a bunch of many tests on anime (for now, real life footage coming soon ;)) some observations are the following :

Trellis 2 doesn't always give a smaller filesize than Trellis 1
Trellis 1 doesn't always give a smaller filesize than Trellis 0 (no trellis)
Trellis 0 doesn't always give a bigger filesize than Trellis 1
Trellis 1 doesn't always give a bigger filesize than Trellis 2 ...
pretty useful huh ? ;)

Consider fs<x> the final filesize using --trellis x :
This has happened twice : fs0 < fs1 < fs2
But the opposite : fs2 < fs1 < fs0 also happened !

... which is quite amusing :) I have the x264 logs somewhere on my laptop (with PSNR calculated), I'll be able to post them a bit later for comparison.

I've done about 50 encodes (1min sample) with different bframe settings, different ratecontrol methods (crf, cqp, and 1pass bitrate) and switching between --me umh and --me hex for each test, trying to tie some setting to trellis' efficiency ... I still need to calculate the % difference between the encoded samples to have objective interpretation.

Conclusion : I find trellis is something rather "unstable" ...

@akupenguin :


When you tell it to reduce the bitrate.
It's that simple.

How, as a user, can I ask it to reduce the bitrate ? ;)
Unless I missed something out, I only have a --trellis x option ^^'

[Manao]

How, as a user, can I ask it to reduce the bitrate ?You don't want to do that. Trellis optimizes the tradeoff bitrate / quality. In order to do so, it'll try, anywhere it can, to minimize "rate + L * distorsion", where L is a constant which depends on the quantizer. L is the parameter you'd want to have access to. In your case, you want to minimize bitrate, so you'd want to have L=0. But in that case, anywhere it can, the trellis would try not to encode the macroblock, since it's the best way to minimize the rate. You'd end up with a video where all macroblocks are roughly skips ( needless to say, it'd be ugly ).

So, you really have to set a tradeoff ( hence, L > 0 ). But if you set a tradeoff, sometimes, the rate will need to be bigger in order the distorsion not to be too big.

Another thing to know is that the constant L is really important in the efficiency of the trellis. It'll work only if the constant is properly set, else, it'll in fact degrade the rate/distorsion tradeoff. Needless to say, the values are currently correctly set in x264.

[akupenguin]

How, as a user, can I ask it to reduce the bitrate ? ;)
Using the --bitrate option.

[DarkZell666]

Great explanation Manao, that L constant was indeed what I was after, from what you explained ;)

@akupenguin:

Can the --bitrate option be used at the same time as --crf ? oO
I was talking about --crf/--qp at first, which is why I didn't understand this :


When you tell it to reduce the bitrate.
It's that simple.


I was actually asking how to impose upon trellis to reduce the bitrate, maybe I wasn't clear about that either, sorry ^^

[Teegedeck]

What the good people tried to tell you is that quantizer 20 with Trellis should show better quality than, say, quantizer 19 without Trellis. (Not that I tested that, I still sadly neglect x264, though I'm improving. ;))

So if you absolutely must use constant quantizer and have a lower filesize with Trellis than without Trellis, just use a higher quantizer.

[Manao]

if you absolutely must use constant quantizer and have a lower filesizeIt's the issue in a nutshell : using a constant quantizer / constant crf means that you don't care about filesize. So ask yourself : why do you care ? If you want a filesize, use bitrate rate control mode instead.

[DarkZell666]

Well, since someone once said that trellis allowed more compression of data, I thought that having trellis enabled would give me the same psnr than without it, but with a smaller filesize. I thought in the first place that trellis was kind of a lossless optimisation, hence the misunderstanding I had of the hole thing.

Sorry for that, I see clearly now that I went completely the wrong way :(

Thx for your advice all of you ;)
Anyway, here are the filesizes of the encoded clips I made, just for the fun of it ;)
NB : I used bframe bias = 100 to enforce the 6 consecutive Bframes

Edit : Here is the link to the logs concerning the umh encodes (it's the only log I've actually kept) : http://tpechoc2.free.fr/x264log_kngb_512x384_allumh.txt


http://tpechoc2.free.fr/trellistest_kngb_512x384.png

I'll upload the source clip tomorrow (konjiki no gash bell opening, trimmed @ 1min)

[LoRd_MuldeR]

What the good people tried to tell you is that quantizer 20 with Trellis should show better quality than, say, quantizer 19 without Trellis. (Not that I tested that, I still sadly neglect x264, though I'm improving. ;))

So if you absolutely must use constant quantizer and have a lower filesize with Trellis than without Trellis, just use a higher quantizer.

So this means, it is usually better to use Trellis with higher a Quantizer than a lower Quantizer without Trellis. Correct?