x264 papers

[GeoffreyA]

This morning, searching for information on x264's deadzone parameters, I came across a paper by Loren Merritt and Rahul Vanam, detailing rate control, quantisation, and other matters. It might be a good place to bring together other papers, so I link Dark Shikari's MB-tree one as well.

http://akuvian.org/src/x264/overview_x264_v8_5.pdf
https://huyunf.github.io/blogs/2017/12/06/x264_slice_type_decision/MBtree%20paper.pdf

[Emulgator]

Nice, thanks Geoffrey !

[Z2697]

Did you find what's the deal with deadzone?

[GeoffreyA]

Did you find what's the deal with deadzone?

Seems to be related to x264 getting rid of fine detail, higher values having more of an effect. In practice, it's hard to see the difference but the bitrate drops.

Set the size of the inter luma quantization deadzone for non-trellis quantization (default: 21). Lower values help to preserve fine details and film grain (typically useful for high bitrate/quality encode), while higher values help filter out these details to save bits that can be spent again on other macroblocks and frames (typically useful for bitrate-starved encodes). It is recommended that you start by tweaking deadzone_intra before changing this parameter.

http://www.mplayerhq.hu/DOCS/man/en/mplayer.1.html

[Z2697]

non-trellis huh... so one should only expect meaningful difference when using preset <= 2 (veryfast) or --trellis 0.
I didn't know that, after I test it this way it does make a significant "blur". (same QP, not bitrate normalized)

For trellis quantization, it still made some change but non-conclusive, even the extreme ends, so I guess just don't bother with it until you have no trellis? :)

I'm testing on a noise source by the way, that's one way to get a "even distribution" of DCT coefficients right?

[GeoffreyA]

Indeed, when trellis is disabled, it has most effect, but even with trellis, there's a minor change in bitrate. So for most encodes, it probably doesn't matter, trellis being a staple in x264.

I was expecting it to be the original of x265's nr-inter/intra, but for me the effect was weaker.

Come to think of it, where is trellis quantisation in x265?

[Z2697]

Come to think of it, where is trellis quantisation in x265?

Probably --rdoq(-level). --psy-rdoq is then the psy-trellis, the "second part" of x264's --psy-rd.

Although, I couldn't find any traces of --trellis being once in the x265's code (searching through git log).
That said, maybe they just used the different name from beginning.
(x265 code base actually starts from HM, with x264 features gradually added in)

[GeoffreyA]

Probably --rdoq(-level). --psy-rdoq is then the psy-trellis, the "second part" of x264's --psy-rd.

Although, I couldn't find any traces of --trellis being once in the x265's code (searching through git log).
That said, maybe they just used the different name from beginning.
(x265 code base actually starts from HM, with x264 features gradually added in)

Hmm. Trellis is supposed to have a sharpening effect in x264? Not sure. If x265 has no x264-like trellis, can that be the cause of its softness, even when the smoothing features are all disabled?

I see. I was under the impression the codebase was a fork of x264, the HEVC features being implemented little by little. That explains much and also why the code seems messier than x264.

Edit: Searching the x264 codebase for trellis shows that it's pretty involved with other things, so it wouldn't be easy, without deep understanding of x265's code, to port it over.

[Z2697]

Most of the features (trellis included) will not actively sharpen or blur the image.
But as you can see, psy-rd and trellis prefer higher energy results, and combined with whole encoding process being lossy (suprise), result in sometimes the less accurate but sharper mode being chosen.
The real image altering filters are rare so to speak, and often designed to tackle some hard problem, such as deblocking.

[benwaggoner]

Indeed, when trellis is disabled, it has most effect, but even with trellis, there's a minor change in bitrate. So for most encodes, it probably doesn't matter, trellis being a staple in x264.

I was expecting it to be the original of x265's nr-inter/intra, but for me the effect was weaker.
--nr in x264 is essentially identical to x265's --nr-inter.