Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 4th March 2020, 23:24   #1  |  Link
Losko
Registered User
 
Join Date: Dec 2010
Posts: 20
Fairly recent, pretty comprehensive, home made x265 3.2 benchmark

I was curious of comparing performances of the latest x265 releases, since I read the article about change into slower/veryslow preset parameters after 3.0 release. I just wanted to see how crf+preset combination could affect compression time and final file size, but never stumbled into a comprehensive benchmark, so I did one my way, using a recent x265 build and a command line as simple as crf+preset (and always "-D 10"). And here you have it too.

Encoding content:
y4m file, 1920x1080 2000 frames (~ 6 GB)
Source is the "Snowpiercer" bluray movie untouched. Selected footage include action (1000 frames) and more relaxed (not so static, indeed) scenes (1000 frames) for a 2000 frames total.
A (resized) sample can be viewed here.

Encoder & cpu capabilities:
x265 [info]: HEVC encoder version 3.2.1+5-0af31fef64a7
x265 [info]: build info [Linux][GCC 6.3.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

Current kernel:
Linux cindy 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux

Encoding settings:
always 10 bit, preset + crf

Encoding duration taken from the x265 final statistics.

Code:
              |  crf 28   |  crf 26   |  crf 24   |  crf 22   |  crf 20   |  crf 18   |
 PRESET       | (default) |           |           |           |           |           |
---------------------------------------------------------------------------------------
              | T:  77.8% | T:  78,5% | T:  85.8% | T:  86.8% | T:  99.1% | T: 103.6% |
 4 (fast)     | S:  96.5% | S: 126.3% | S: 170.0% | S: 240.0% | S: 357.7% | S: 550.6% |
---------------------------------------------------------------------------------------
 5 (medium,   | T:  100%  | T:  104%  | T:  117%  | T:  132%  | T:  158%  | T:  167%  |
  default)    | S:  100%  | S:  132%  | S:  182%  | S:  263%  | S:  403%  | S:  637%  |
---------------------------------------------------------------------------------------
              | T:  341%  | T:  361%  | T:  403%  | T:  444%  | T:  528%  | T:  625%  |
 6 (slow)     | S:  118%  | S:  161%  | S:  230%  | S:  348%  | S:  550%  | S:  867%  |
---------------------------------------------------------------------------------------
              | T: 1176%  | T: 1251%  | T: 1440%  | T: 1618%  | T: 1927%  | T: 2300%  |
 7 (slower)   | S:  126%  | S:  174%  | S:  251%  | S:  380%  | S:  590%  | S:  907%  |
---------------------------------------------------------------------------------------
              | T: 2211%  | T: 2313%  | T: 2724%  | T: 3051%  | T: 3669%  | T: 4364%  |
 8 (veryslow) | S:  125%  | S:  172%  | S:  248%  | S:  376%  | S:  586%  | S:  903%  |
---------------------------------------------------------------------------------------

T = encoding time
S = raw file size
On a first sight, increase in file size is expected looking at any single row from left to right. The less crf (at a given preset), the more quality requested and the more data need to be encoded.

On the other hand, I didn't expect such an increase in file size in vertical direction (same crf value, increasing preset). It seems at lower presets the encoder deliberately discards some data, which is encoded at higher preset values, up to a point where file size remains the same (slower and veryslow). I can figure out more data is encoded more efficiently, so the file size doesn't increase; rather, it is always smaller with veryslow preset, BUT this comes at a cost of doubling the encoding time with respect to the slower preset.
Losko is offline   Reply With Quote
Old 4th March 2020, 23:24   #2  |  Link
Losko
Registered User
 
Join Date: Dec 2010
Posts: 20
In this second run I encoded the same uncompressed source material (y4m file, 1920x1080 2000 frames, ~ 6 GB) BUT filtered using MCDegrain filter.

Same encoder, same OS, same environment.

Code:
              |  crf 28   |  crf 26   |  crf 24   |  crf 22   |  crf 20   |  crf 18   |
 PRESET       | (default) |           |           |           |           |           |
---------------------------------------------------------------------------------------
              | T:  84.4% | T:  70.3% | T:  73.0% | T:  82.7% | T:  80.6% | T:  87.1% |
 4 (fast)     | S:  97.7% | S: 122.5% | S: 154.5% | S: 197.8% | S: 258.7% | S: 343.1% |
---------------------------------------------------------------------------------------
 5 (medium,   | T:  100%  | T:   89%  | T:   97%  | T:  119%  | T:  120%  | T:  134%  |
  default)    | S:  100%  | S:  126%  | S:  161%  | S:  209%  | S:  278%  | S:  378%  |
---------------------------------------------------------------------------------------
              | T:  353%  | T:  313%  | T:  332%  | T:  457%  | T:  403%  | T:  425%  |
 6 (slow)     | S:  114%  | S:  145%  | S:  188%  | S:  247%  | S:  334%  | S:  462%  |
---------------------------------------------------------------------------------------
              | T: 1183%  | T: 1067%  | T: 1150%  | T: 1627%  | T: 1414%  | T: 1571%  |
 7 (slower)   | S:  116%  | S:  149%  | S:  192%  | S:  254%  | S:  344%  | S:  476%  |
---------------------------------------------------------------------------------------
              | T: 2077%  | T: 1927%  | T: 2137%  | T: 2943%  | T: 2679%  | T: 2947%  |
 8 (veryslow) | S:  115%  | S:  147%  | S:  191%  | S:  252%  | S:  342%  | S:  473%  |
---------------------------------------------------------------------------------------

T = encoding time
S = raw file size
Again, the size of the files produced by slower and veryslow presets are the same for the same crf value.
The first thing I notice is the different file size growth, unlike the previous run I didn't have an "explosion" up to 9 times the reference size, and this is due to the filter.
Further, there is a weird "jump down" in encoding times between both crf28/crf26 and crf22/crf20, even though visual quality should go up the whole scene compressed easily (but may this only be due to this particular source material?).
Also, times are generally shorter: for crf18 duration times are 20-30% shorter here.

In case anyone is wondering, the reference encoding (medium preset, crf28) created a file large 85% of the previous reference, and the encoding was 5% longer.

Last edited by Losko; 7th March 2020 at 13:32.
Losko is offline   Reply With Quote
Old 5th March 2020, 06:16   #3  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,697
It is an interesting finding compared to x264 is that you get a bigger file when you go to a slower preset. If I remember correctly, in x264 it was said that slower presets generally compress more efficiently. x265 uses more B- and ref frames in slower presets but they don't seem to offset the other things which cause bitrate to rise (probably smoothes the image less so more bits required).
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 5th March 2020, 11:29   #4  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 130
Quote:
Originally Posted by Boulder View Post
It is an interesting finding compared to x264 is that you get a bigger file when you go to a slower preset. If I remember correctly, in x264 it was said that slower presets generally compress more efficiently. x265 uses more B- and ref frames in slower presets but they don't seem to offset the other things which cause bitrate to rise (probably smoothes the image less so more bits required).
Ofc the slower presets can achieve higher compression at the same quality, and that is very much the case here as well.

This behavior has been noted and talked about around here for a long time; slower presets create larger files at the same CRF. It might seem a bit of counterintuitive, but remember that an CRF value does not specify a specific level of quality or fidelity, and its never "constant" between different settings for the encoder. For exempel, try to encode something with fine detail with the faster presets, it will be soft for most if the CRF range, while the slower ones will keep quality at much higher values (and ofc at the same bitrate).

It would however be nice if someone could give a more technical explanation of the behavior (more specify maybe what settings that effects it) cause generally with x264 slower presets gave an smaller size then faster ones at the same crf value, the image quality still woudlnt wont be identical though, but the encoder behavior did change in x265.

Last edited by excellentswordfight; 5th March 2020 at 12:11.
excellentswordfight is offline   Reply With Quote
Old 5th March 2020, 13:19   #5  |  Link
Sharc
Registered User
 
Join Date: May 2006
Posts: 3,597
Interesting table. Indeed the strong increase in file size from fast to slow(er) preset for x265 is surprising.
Slower settings producing a larger file size for the same CRF is also possible with x264 however. For example, a stronger (slower) setting for --me and --subme may increase the file size for same CRF, if I remember this correctly.

Last edited by Sharc; 5th March 2020 at 13:29.
Sharc is offline   Reply With Quote
Old 5th March 2020, 17:24   #6  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,104
I think the potential higher size at the same CRF is due to additional rate distortion and psychovisual algorithms being used, which will send more bits to some parts of the image (and less to others, which is why the size gain isn't consistent between sources). However, the file size at the same visual quality goes down as presets go up. For apples-to-apples comparisons, 2-pass VBR to a specified average bitrate is more accurate. Also probably better for benchmarking, as more bits also means more CABAC, so comparing presets at CRF is going to yield slower relative performance at higher presets than would be seen in fixed bitrate or fixed subjective quality. In the real world, no one would encode using the same CRF at different presets, because the whole point of the higher presets is to get more quality per bit.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 5th March 2020, 17:31   #7  |  Link
Tadanobu
Registered User
 
Join Date: Sep 2019
Posts: 28
For proper comparison you should add VMAF to your table.
Tadanobu is offline   Reply With Quote
Old 6th March 2020, 10:33   #8  |  Link
Losko
Registered User
 
Join Date: Dec 2010
Posts: 20
Quote:
Originally Posted by Tadanobu View Post
For proper comparison you should add VMAF to your table.
Mmmmhhhh, truth is this benchmark wasn't born with quality-wise evaluation in mind but it's a good idea.
I'm currently running an encoding series for another benchmark but I'll see what I can do.

Last edited by Losko; 6th March 2020 at 10:42. Reason: typo
Losko is offline   Reply With Quote
Old 7th March 2020, 13:33   #9  |  Link
Losko
Registered User
 
Join Date: Dec 2010
Posts: 20
Added second run details in the second post.
Losko is offline   Reply With Quote
Old 10th March 2020, 21:51   #10  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,104
Quote:
Originally Posted by Tadanobu View Post
For proper comparison you should add VMAF to your table.
VMAF isn't a super good metric to look at subtle differences between CRF18 encodes. And there's no standard way to get an overall VMAF for a clip more than 10-20 seconds long. Harmonic of individual frame scores can work okay if it's just a GOP or two. But for a 10 min clip, the same mean VMAF can indicate a lot of quality variability, or not.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 8th April 2020, 23:29   #11  |  Link
Atlantis
Registered User
 
Join Date: Feb 2002
Posts: 361
Question. Imagine you are at Medium CRF 22. Now you want to go one step higher quality. Better to go Slow CRF 22 or Medium CRF 21? Are they the same quality wise? What's the difference?

I think we need a benchmark with a third value. quality if that's possible.

Last edited by Atlantis; 8th April 2020 at 23:45.
Atlantis is offline   Reply With Quote
Old Yesterday, 17:02   #12  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,104
Quote:
Originally Posted by Atlantis View Post
Question. Imagine you are at Medium CRF 22. Now you want to go one step higher quality. Better to go Slow CRF 22 or Medium CRF 21? Are they the same quality wise? What's the difference?

I think we need a benchmark with a third value. quality if that's possible.
I imagine Slow 22 and Medium 21 would look better or worse depending on content. A single CRF step really isn't that big. The slower presets actually look at things the faster presets don't, and so can wind up preserving more detail and/or finding newer ways to preserve bits. The difference between CRF 21 and 22 in quality can easily be smaller than the content-dependent difference between presets.

Sent from my SM-T837V using Tapatalk
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 00:43.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.