X264 Trellis and CRF Mode [Archive]

ron spencer

28th February 2010, 14:34

Just a quick question...I am putting some DVDs on a media server in mkv via Handbrake...if I am using CRF mode, should Trellis be off? I have seen some people leave it at 1 and others say must be off for CQ.

Also, just a general question, does Trellis set at 2 for a two pass really add to quality vs. the time it takes?

thanks in advance

LoRd_MuldeR

28th February 2010, 14:40

I can't answer your question regarding Trellis and CQP mode, but why the heck do you want to use CQP mode at all ??? :eek:

If you want something that is as close as possible to a "constant quality" mode, then use CRF mode. Also be aware that with CQP mode you would loose AQ and MB-Tree!

EDIT: The only difference between Trellis=1 and Trellis=2 is that Trellis=2 uses Trellis quantization more often than Trellis=1 would do.

So if Trellis=1 (aka "Trellis is used for some decisions") gives an improvement over Trellis=0 (aka "no Trellis quantization at all"), then we can conclude that Trellis is helpful.

And then Trellis=2 (aka "Trellis is used for all decisions") will be even more helpful than Trellis=1 ;)

Warpman

28th February 2010, 14:43

I can't answer your question regarding Trellis and CQ mode, but why the heck do you use CQ mode at all ???

If you want something that is as close as possible to a "constant quality" mode, then use CRF mode. Also be aware that with CQ mode you would loose AQ and MB-Tree!

++++

don't use CQ mode! use crf!

28th February 2010, 14:52

Ron probably refers to HandBrake's "Constant Quality" control, which is CRF. Trellis works fine with it.

Also, just a general question, does Trellis set at 2 for a two pass really add to quality vs. the time it takes?
Check the presets listed by x264 --fullhelp to get an idea of how useful each setting is. Trellis=2 is enabled at preset "slower", so it may be practical in some cases.

nurbs

28th February 2010, 14:53

CRF mode is called CQ (Constant Quality) in Handbrake. You can use trellis in any encoding mode and it's always beneficial.

edit:
I'm always a minute too slow :(

LoRd_MuldeR

28th February 2010, 14:56

Ron probably refers to HandBrake's "Constant Quality" control, which is CRF. Trellis works fine with it.

That naming is suboptimal :rolleyes:

ron spencer

28th February 2010, 14:56

sorry....was in a rush.

yes I did mean crf...will fix title.

I had seen a guide which said not to use trellis if using crf. I could not find the guide(s) but remember reading about it. Just want to be sure.

thanks for the fast responses...very much appreciated. Will use 1.

:thanks:

LoRd_MuldeR

28th February 2010, 14:58

I had seen a guide which said not to use trellis if using crf. I could not find the guide(s) but remember reading about it. Just want to be sure.

That guide is either outdated or simply wrong. Trellis works perfectly fine with CRF mode. Also see the "edit" in my first (http://forum.doom9.org/showpost.php?p=1378617&postcount=2) post.

(Back in the old days, some people said that Trellis had a tendency to smooth out fine detail. But that definitely is different today, now that Psy-RD and Psy-Trellis were added ^^)

yes I did mean crf...will fix title.

Title can be edit after going "Advanced" from the ad-hoc editing dialog.

28th February 2010, 15:14

Ron probably refers to HandBrake's "Constant Quality" control, which is CRF. Trellis works fine with it.
That naming is suboptimal :rolleyes:
Well, I think "Constant Quality" is more descriptive than "Constant Rate Factor" to the average user. Also note that the acronym CQ is not used in any HandBrake user interface.

HandBrakeCLI lists these parameters:
-q, --quality <float> Set video quality (0.0..1.0)
-Q, --cqp Use with -q for CQP instead of CRF

ron spencer

28th February 2010, 15:18

Well, I think "Constant Quality" is more descriptive than "Constant Rate Factor" to the average user. Also note that the acronym CQ is not used in any HandBrake user interface.

HandBrakeCLI lists these parameters:
-q, --quality <float> Set video quality (0.0..1.0)
-Q, --cqp Use with -q for CQP instead of CRF

Actually CQ itself is not used, but Constant Quality is. Go to video tab and you will see Constant Quality...then to far right you see RF. Title probably should be changed to Constant Quality (RF). I saw the Constant Quality, which stood out as CQ in my mind when I made the original title of this thread.

ron spencer

28th February 2010, 15:46

Here are my settings (wait for attachment to be approved).

Look ok?

thx

me7

28th February 2010, 16:00

Actually CQ itself is not used, but Constant Quality is. Go to video tab and you will see Constant Quality...then to far right you see RF. Title probably should be changed to Constant Quality (RF). I saw the Constant Quality, which stood out as CQ in my mind when I made the original title of this thread.

For 99% of doom9 readers "constant quantizer" is the first thing they think of when they read "CQ".

Constant Quality is a bad name because their is no reliable metric for quality. Constant Rate Factor is a metric that tries to achieve constant quality but it is not eqivalent with the meaning of "constant quality".

Warpman

28th February 2010, 16:02

Here are my settings (wait for attachment to be approved).

Look ok?

thx

it's easier to upload to some imagehoster and linking the picture here instead of waiting for the approvel which can take a bit of time ;)

Stephen R. Savage

28th February 2010, 16:24

Just for the record, I always use deadzones=0, trellis=0, decimate=0 because I find that trellis greatly increases banding, even at absurdly low RF.

LoRd_MuldeR

28th February 2010, 16:56

Just for the record, I always use deadzones=0, trellis=0, decimate=0 because I find that trellis greatly increases banding, even at absurdly low RF.

The only thing that reliably helps against "banding" is dither. Try to add dither before encoding and use x264 settings that keep the dither.

Psy-Trellis should help here, which of course requires Trellis quantization to be enabled. Note that Psy-Trellis is not enabled by default, but enabled with the "film" tuning.

Stephen R. Savage

28th February 2010, 17:02

The only thing that reliably helps against "banding" is dither. Try to add dither before encoding and use x264 settings that keep the dither.

Exactly. And these x264 settings are: qmin=0, deadzone=0, trellis=0, mbtree=0.

ron spencer

28th February 2010, 17:02

it's easier to upload to some imagehoster and linking the picture here instead of waiting for the approvel which can take a bit of time ;)

see here

http://img186.imageshack.us/img186/2666/sshot1vc.jpg

http://img186.imageshack.us/img186/2666/sshot1vc.jpg

mp3dom

28th February 2010, 17:16

Exactly. And these x264 settings are: qmin=0, deadzone=0, trellis=0, mbtree=0.

Is the new AQ=2 (the 'old' AQ4 committed) suitable to avoid banding or it is better to stay with AQ=1?

LoRd_MuldeR

28th February 2010, 17:19

@ron spencer

(1) You should enabled Adaptive 8x8 DCT, if possible!
(2) If you use Trellis quantization and Trellis=1 gives an improvement, then Trellis=2 will be even better (and slower) than Trellis=1
(3) B-Pyramid should give some improvement and it now works fine with MB-Tree, so there's no reason to not use it. However that GUI is outdated, as B-Pyramid is not a boolean option anymore
(4) Analysis "all" is overkill, because the 4x4 partitions usually won't help at all. So it's recommended to stick with x264's default value
(5) Some Psy-Trellis should help for "filmic" content. Try something like Psy-Trellis = 0.15
(6) Usually you should keep "DCT-Decimation" and "Fast P-Skip" enabled, which means you should keep the corresponding "No" options unchecked
(7) Raise Sub-ME as much as possible. You have currently selected 8, but you can try 9 or even 10 to squish out some more quality (at the cost of encoding speed)
(8) Where are the MB-Tree and Lookahead options ???

sneaker_ger

28th February 2010, 17:43

LoRd_MuldeR

28th February 2010, 17:46

(3) B-Pyramid should give some improvement and it now works fine with MB-Tree, so there's no reason to not use it. However that GUI is outdated, as B-Pyramid is not a boolean option anymore
Or it's super up-to-date. (http://git.videolan.org/gitweb.cgi?p=x264.git;a=commit;h=1d94f305999864aaa1da79050da2f900f69b4365) ;)

What is that supposed to tell me?

The fact that "--b-pyramid normal" is the new default now and that "--b-pyramid 0/1/2" can now be used as aliases for "--b-pyramid none/strict/normal" does not make "--b-pyramid" a boolean option ;)

However in ron spencer's screenshot the "Pyramidal B-Frames" option is implemented as a simple checkbox. Consequently that GUI is outdated/incomplete. MB-Tree is missing too...

MuLTiTaSK

28th February 2010, 18:42

HandBrake Snapshots (bit.ly/a1yRi6)
http://img28.imageshack.us/img28/4894/ss20100228124025.png
http://img28.imageshack.us/img28/8761/ss20100228124048.png

sneaker_ger

28th February 2010, 23:15

ron spencer

1st March 2010, 01:38

What is that supposed to tell me?

The fact that "--b-pyramid normal" is the new default now and that "--b-pyramid 0/1/2" can now be used as aliases for "--b-pyramid none/strict/normal" does not make "--b-pyramid" a boolean option ;)

However in ron spencer's screenshot the "Pyramidal B-Frames" option is implemented as a simple checkbox. Consequently that GUI is outdated/incomplete. MB-Tree is missing too...

nothing I can do about the GUI....it is a simple and good approach though. perhaps developers will update it sometime

LoRd_MuldeR

1st March 2010, 02:27

nothing I can do about the GUI....it is a simple and good approach though. perhaps developers will update it sometime

I have to admit that I didn't check Handbrake recently, but if the latest version doesn't allow you to enable "--b-pyramid 2" (aka "--b-pyramid normal") or if MB-Tree Ratecontrol is missing --- as it seems to me from the screenshots --- then I'd use another GUI until they add the missing options.

ron spencer

1st March 2010, 02:35

what would you recommend?

I just liked Handbrake because of its layout and ease of use...

Dark Shikari

1st March 2010, 03:08

I have to admit that I didn't check Handbrake recently, but if the latest version doesn't allow you to enable "--b-pyramid 2" (aka "--b-pyramid normal") or if MB-Tree Ratecontrol is missing --- as it seems to me from the screenshots --- then I'd use another GUI until they add the missing options.The latest handbrake supports MB-tree (on by default) and pyramid just fine.

lych_necross

1st March 2010, 07:30

I think only the latest dev build supports MB-tree while the regular stable build does not. I could be mistaken. Handbrake dev builds are available here (http://handbrake.fr/snapshot.php).

Dark Shikari

1st March 2010, 07:35

I think only the latest dev build supports MB-tree while the regular stable build does not. I could be mistaken. Handbrake dev builds are available here (http://handbrake.fr/snapshot.php).No, 0.9.4 has MB-tree on by default. In fact, MB-tree is one of the reasons 0.9.4 was released.

lych_necross

1st March 2010, 09:01

No, 0.9.4 has MB-tree on by default. In fact, MB-tree is one of the reasons 0.9.4 was released.
I stand corrected. I wasn't sure as I haven't used Handbrake in a while (I'm an Avidemux guy personally).