Fairly recent, pretty comprehensive, home made x265 3.2 benchmark

[Losko]

I was curious of comparing performances of the latest x265 releases, since I read the article about change into slower/veryslow preset parameters after 3.0 release. I just wanted to see how crf+preset combination could affect compression time and final file size, but never stumbled into a comprehensive benchmark, so I did one my way, using a recent x265 build and a command line as simple as crf+preset (and always "-D 10"). And here you have it too.

Encoding content:
y4m file, 1920x1080 2000 frames (~ 6 GB)
Source is the "Snowpiercer" bluray movie untouched. Selected footage include action (1000 frames) and more relaxed (not so static, indeed) scenes (1000 frames) for a 2000 frames total.
A (resized) sample can be viewed here.

Encoder & cpu capabilities:
x265 [info]: HEVC encoder version 3.2.1+5-0af31fef64a7
x265 [info]: build info [Linux][GCC 6.3.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

Current kernel:
Linux cindy 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux

Encoding settings:
always 10 bit, preset + crf

Encoding duration taken from the x265 final statistics.

              |  crf 28   |  crf 26   |  crf 24   |  crf 22   |  crf 20   |  crf 18   |
 PRESET       | (default) |           |           |           |           |           |
---------------------------------------------------------------------------------------
              | T:  77.8% | T:  78,5% | T:  85.8% | T:  86.8% | T:  99.1% | T: 103.6% |
 4 (fast)     | S:  96.5% | S: 126.3% | S: 170.0% | S: 240.0% | S: 357.7% | S: 550.6% |
---------------------------------------------------------------------------------------
 5 (medium,   | T:  100%  | T:  104%  | T:  117%  | T:  132%  | T:  158%  | T:  167%  |
  default)    | S:  100%  | S:  132%  | S:  182%  | S:  263%  | S:  403%  | S:  637%  |
---------------------------------------------------------------------------------------
              | T:  341%  | T:  361%  | T:  403%  | T:  444%  | T:  528%  | T:  625%  |
 6 (slow)     | S:  118%  | S:  161%  | S:  230%  | S:  348%  | S:  550%  | S:  867%  |
---------------------------------------------------------------------------------------
              | T: 1176%  | T: 1251%  | T: 1440%  | T: 1618%  | T: 1927%  | T: 2300%  |
 7 (slower)   | S:  126%  | S:  174%  | S:  251%  | S:  380%  | S:  590%  | S:  907%  |
---------------------------------------------------------------------------------------
              | T: 2211%  | T: 2313%  | T: 2724%  | T: 3051%  | T: 3669%  | T: 4364%  |
 8 (veryslow) | S:  125%  | S:  172%  | S:  248%  | S:  376%  | S:  586%  | S:  903%  |
---------------------------------------------------------------------------------------

T = encoding time
S = raw file size

On a first sight, increase in file size is expected looking at any single row from left to right. The less crf (at a given preset), the more quality requested and the more data need to be encoded.

On the other hand, I didn't expect such an increase in file size in vertical direction (same crf value, increasing preset). It seems at lower presets the encoder deliberately discards some data, which is encoded at higher preset values, up to a point where file size remains the same (slower and veryslow). I can figure out more data is encoded more efficiently, so the file size doesn't increase; rather, it is always smaller with veryslow preset, BUT this comes at a cost of doubling the encoding time with respect to the slower preset.

[Losko]

In this second run I encoded the same uncompressed source material (y4m file, 1920x1080 2000 frames, ~ 6 GB) BUT filtered using MCDegrain filter.

Same encoder, same OS, same environment.

              |  crf 28   |  crf 26   |  crf 24   |  crf 22   |  crf 20   |  crf 18   |
 PRESET       | (default) |           |           |           |           |           |
---------------------------------------------------------------------------------------
              | T:  84.4% | T:  70.3% | T:  73.0% | T:  82.7% | T:  80.6% | T:  87.1% |
 4 (fast)     | S:  97.7% | S: 122.5% | S: 154.5% | S: 197.8% | S: 258.7% | S: 343.1% |
---------------------------------------------------------------------------------------
 5 (medium,   | T:  100%  | T:   89%  | T:   97%  | T:  119%  | T:  120%  | T:  134%  |
  default)    | S:  100%  | S:  126%  | S:  161%  | S:  209%  | S:  278%  | S:  378%  |
---------------------------------------------------------------------------------------
              | T:  353%  | T:  313%  | T:  332%  | T:  457%  | T:  403%  | T:  425%  |
 6 (slow)     | S:  114%  | S:  145%  | S:  188%  | S:  247%  | S:  334%  | S:  462%  |
---------------------------------------------------------------------------------------
              | T: 1183%  | T: 1067%  | T: 1150%  | T: 1627%  | T: 1414%  | T: 1571%  |
 7 (slower)   | S:  116%  | S:  149%  | S:  192%  | S:  254%  | S:  344%  | S:  476%  |
---------------------------------------------------------------------------------------
              | T: 2077%  | T: 1927%  | T: 2137%  | T: 2943%  | T: 2679%  | T: 2947%  |
 8 (veryslow) | S:  115%  | S:  147%  | S:  191%  | S:  252%  | S:  342%  | S:  473%  |
---------------------------------------------------------------------------------------

T = encoding time
S = raw file size

Again, the size of the files produced by slower and veryslow presets are the same for the same crf value.
The first thing I notice is the different file size growth, unlike the previous run I didn't have an "explosion" up to 9 times the reference size, and this is due to the filter.
Further, there is a weird "jump down" in encoding times between both crf28/crf26 and crf22/crf20, even though visual quality should go up the whole scene compressed easily (but may this only be due to this particular source material?).
Also, times are generally shorter: for crf18 encoding times are 20-30% shorter here.

In case anyone is wondering, the reference encoding (medium preset, crf28) created a file large 85% than the previous reference, and the encoding took 5% longer.

[Boulder]

It is an interesting finding compared to x264 is that you get a bigger file when you go to a slower preset. If I remember correctly, in x264 it was said that slower presets generally compress more efficiently. x265 uses more B- and ref frames in slower presets but they don't seem to offset the other things which cause bitrate to rise (probably smoothes the image less so more bits required).

[excellentswordfight]

Originally Posted by Boulder

It is an interesting finding compared to x264 is that you get a bigger file when you go to a slower preset. If I remember correctly, in x264 it was said that slower presets generally compress more efficiently. x265 uses more B- and ref frames in slower presets but they don't seem to offset the other things which cause bitrate to rise (probably smoothes the image less so more bits required).

Ofc the slower presets can achieve higher compression at the same quality, and that is very much the case here as well.

This behavior has been noted and talked about around here for a long time; slower presets create larger files at the same CRF. It might seem a bit of counterintuitive, but remember that an CRF value does not specify a specific level of quality or fidelity, and its never "constant" between different settings for the encoder. For exempel, try to encode something with fine detail with the faster presets, it will be soft for most if the CRF range, while the slower ones will keep quality at much higher values (and ofc at the same bitrate).

It would however be nice if someone could give a more technical explanation of the behavior (more specify maybe what settings that effects it) cause generally with x264 slower presets gave an smaller size then faster ones at the same crf value, the image quality still woudlnt wont be identical though, but the encoder behavior did change in x265.

[Sharc]

Interesting table. Indeed the strong increase in file size from fast to slow(er) preset for x265 is surprising.
Slower settings producing a larger file size for the same CRF is also possible with x264 however. For example, a stronger (slower) setting for --me and --subme may increase the file size for same CRF, if I remember this correctly.

[benwaggoner]

I think the potential higher size at the same CRF is due to additional rate distortion and psychovisual algorithms being used, which will send more bits to some parts of the image (and less to others, which is why the size gain isn't consistent between sources). However, the file size at the same visual quality goes down as presets go up. For apples-to-apples comparisons, 2-pass VBR to a specified average bitrate is more accurate. Also probably better for benchmarking, as more bits also means more CABAC, so comparing presets at CRF is going to yield slower relative performance at higher presets than would be seen in fixed bitrate or fixed subjective quality. In the real world, no one would encode using the same CRF at different presets, because the whole point of the higher presets is to get more quality per bit.

[Tadanobu]

For proper comparison you should add VMAF to your table.

[Losko]

Originally Posted by Tadanobu

For proper comparison you should add VMAF to your table.

Mmmmhhhh, truth is this benchmark wasn't born with quality-wise evaluation in mind but it's a good idea.
I'm currently running an encoding series for another benchmark but I'll see what I can do.

[Losko]

Added second run details in the second post.

[benwaggoner]

Originally Posted by Tadanobu

For proper comparison you should add VMAF to your table.

VMAF isn't a super good metric to look at subtle differences between CRF18 encodes. And there's no standard way to get an overall VMAF for a clip more than 10-20 seconds long. Harmonic of individual frame scores can work okay if it's just a GOP or two. But for a 10 min clip, the same mean VMAF can indicate a lot of quality variability, or not.

[Atlantis]

Question. Imagine you are at Medium CRF 22. Now you want to go one step higher quality. Better to go Slow CRF 22 or Medium CRF 21? Are they the same quality wise? What's the difference?

I think we need a benchmark with a third value. quality if that's possible.

[benwaggoner]

Originally Posted by Atlantis

Question. Imagine you are at Medium CRF 22. Now you want to go one step higher quality. Better to go Slow CRF 22 or Medium CRF 21? Are they the same quality wise? What's the difference?

I think we need a benchmark with a third value. quality if that's possible.

I imagine Slow 22 and Medium 21 would look better or worse depending on content. A single CRF step really isn't that big. The slower presets actually look at things the faster presets don't, and so can wind up preserving more detail and/or finding newer ways to preserve bits. The difference between CRF 21 and 22 in quality can easily be smaller than the content-dependent difference between presets.

Sent from my SM-T837V using Tapatalk

[Atlantis]

So I did a test myself.

x265 3.3+10-g08d895bb6 GCC 10.0.1

medium 22
encoded 976 frames in 347.54s (2.81 fps), 5511.89 kb/s, Avg QP:24.58

medium 21
encoded 976 frames in 352.39s (2.77 fps), 6488.14 kb/s, Avg QP:23.57

slow 22
encoded 976 frames in 1174.07s (0.83 fps), 6862.54 kb/s, Avg QP:24.61

As you see preset slow is extremely slower. Uses a lot more bitrate and with my eyes medium 21 looks a lot better than slow 22. Not to mention it's faster and uses less bitrate!

What's the point of Slow preset? Why they made it, if you can get better results faster with Medium and lower CRF.

[imhh11]

Originally Posted by Atlantis

So I did a test myself.

x265 3.3+10-g08d895bb6 GCC 10.0.1

medium 22
encoded 976 frames in 347.54s (2.81 fps), 5511.89 kb/s, Avg QP:24.58

medium 21
encoded 976 frames in 352.39s (2.77 fps), 6488.14 kb/s, Avg QP:23.57

slow 22
encoded 976 frames in 1174.07s (0.83 fps), 6862.54 kb/s, Avg QP:24.61

As you see preset slow is extremely slower. Uses a lot more bitrate and with my eyes medium 21 looks a lot better than slow 22. Not to mention it's faster and uses less bitrate!

What's the point of Slow preset? Why they made it, if you can get better results faster with Medium and lower CRF.

Do the same comparison with grainy content. You will notice that slow preset (CRF22) will look a lot better than medium preset (CRF21). I even compared slow CRF22 vs medium CRF17 and slow preset was still better.

Same story when comparing Slower (CRF22) vs Slow (CRF21).

Personally, after a lot of HDR10 encode comparisons, I decided to use CRF 17 slower for grainy contents and CRF15 slower for everything else. It takes about 35-40h per movie but the quality is great.

[Atlantis]

Wow, that takes a lot of time and disk space. I try to apply filters and remove grain to help the compression. I use RemoveGrain which is not excessive on mode 2.

[Boulder]

Downsizing also helps quite a lot.. maybe down to 1440p for sources that are true 4K and 1080p for sources that are actually 2K or so and just upsized to 4K.

[imhh11]

Originally Posted by Atlantis

Wow, that takes a lot of time and disk space.

Well in my experience with UHD HDR encoding, I often(clean source) get ridiculously low bitrate(10mb/s or less) when I use CRF17+ slower preset.

CRF 15 usually gets me in the 15/20mb/s bitrate which is the size I try to target.

[excellentswordfight]

Originally Posted by Atlantis

Wow, that takes a lot of time and disk space. I try to apply filters and remove grain to help the compression. I use RemoveGrain which is not excessive on mode 2.

X265 usually spend less bits with the same crf value for HDR content, I usually lower the crf value by 2 for HDR compared to SDR.

So did you use filters on your tests above? If so, please always mention this in the future. Its very relevant information when discussing encoder behavior.

The point of slower presets is ofc better compression, and that means for a lot of cases, more detail at the same bitrate. Encode something at fixed bitrate (2pass) and compare medium and slower and you will see that slower will keep alot more detail. If you pre-process the material to remove detail, making the source easier to compress the difference could very much be somewhat harder to notice. And if you dont mind the reduction of grain and detail, then good for you, you dont need to wait days for encodes

[benwaggoner]

Originally Posted by excellentswordfight

X265 usually spend less bits with the same crf value for HDR content, I usually lower the crf value by 2 for HDR compared to SDR.

To try and keep bitrates consistent?

Generally speaking HDR looks better at the same bitrate as the same content in an SDR grade, due to being more perceptually uniform.

[ChaosKing]

Hmm so the higher the crf the faster the encoding is. It would be also very interesting to see how ssim / vmaf metrics would look like with your results.