HEVC Video Codecs Comparison 2017 by MSU

[IgorC]

http://www.compression.ru/video/codec_comparison/hevc_2017/

Overall, the leaders in this comparison are Kingsoft HEVC encoder and x265!

[littlepox]

7. x265 MulticoreWare, Inc. 1.9+169-e5b5bdc3c154


1.9+169
23 Aug 2017

Serious?

[Asmodian]

Wow, 2.2 was such a big upgrade too, and there has been a lot more done since. :(

[x265_Project]

It looks like they used v2.3 (see page 80 of their free report... http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_free.pdf).

FYI - We didn't actively support this latest comparison test (providing a build, settings, feedback), as MSU didn't respond positively to our request to improve their test design to meet basic standards for video encoder comparison tests, which I ended up publishing here - http://x265.org/compare-video-encoders/

[littlepox]

It looks like they used v2.3 (see page 80 of their free report... http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_free.pdf).

FYI - We didn't actively support this latest comparison test (providing a build, settings, feedback), as MSU didn't respond positively to our request to improve their test design to meet basic standards for video encoder comparison tests, which I ended up publishing here - http://x265.org/compare-video-encoders/

frankly speaking, asking them to compare the whole video using eyes is not reasonable at all. That's simply too much work for them, also they will be questioned for objectivity coz it's impossible to digitize the comparison of quality.

[nevcairiel]

frankly speaking, asking them to compare the whole video using eyes is not reasonable at all. That's simply too much work for them, also they will be questioned for objectivity coz it's impossible to digitize the comparison of quality.

They could've included Netflix' VMAF metric though, instead of relying on the mediocre SSIM metric only (at least they don't seem to use PSNR anymore, like most other "scientific" tests seem to use)

[kolak]

If you use few metrics and all show that file A is better quality than B how likely it's that out of 10 people more will say opposite after watching files?

[littlepox]

If you use few metrics and all show that file A is better quality than B how likely it's that out of 10 people more will say opposite after watching files?

If neither encoders are specially optimized for the metric, then PSNR is 70% relevant and ssim is almost 90% if the benchmark values are materially different.

However, If encoders are inherently optimized for a specific metric instead of visual quality, then that's a completely different story. 5 years ago you could find tons of encoders with better PSNR values than x264. They were not lying to you, but your eyes wouldn't agree with them at all. In that case, metric itself speaks very little about visual quality.

[x265_Project]

frankly speaking, asking them to compare the whole video using eyes is not reasonable at all. That's simply too much work for them, also they will be questioned for objectivity coz it's impossible to digitize the comparison of quality.
I disagree. You can limit the number of clips you use, to minimize the overall effort. But if you compare video encoders using objective metrics like SSIM, you will only be able to conclude which encoders can produce the highest SSIM score, and not which encoders produce the best visual quality.

Producing an encoder comparison study without comparing the visual quality is like comparing wines without tasting them.

[IgorC]

They could've included Netflix' VMAF metric though, instead of relying on the mediocre SSIM metric only
Yes, Netflix claims better correlation between visual perception and VMAF metric.

However is there any independent study which indicates that VMAF is really superior to SSIM?


P.S. It's strange that Intel HEVC encoder wasn't tested this time. It had the highest score in previous test.
As far as I know Intel has updated lately their encoder.
http://www.compression.ru/video/codec_comparison/hevc_2016/
http://www.compression.ru/video/codec_comparison/hevc_2016/figures/graph3.png

Also there is no any of Ateme, Mainconcept, Elecard or DiVX HEVC encoders. Nor competitors/anchors like VP9 or even current AV1.

[sonnati]

SSIM alone is absolutely not representative of perceptual picture quality. Similarly to PSNR it has very poor correlation with subjective MOS (Mean Opinion Score). VMAF is much more correlated to subjective MOS even if I see problems in correctly evaluate quality in dark areas. Then there's always the problem of pooling values: execute simply the mean of SSIM (or other metrics) across frames regardless of local and global contrast, motion level, luminosity masking and attention is too simplistic and it's easy to obtain an higher SSIM even with lower MOS compared to other encoders...so why use SSIM with different encoders and give value to difference of 0.001 ? it's like "estimate an actor from a photo"

[Blue_MiSfit]

I wish they'd included Vanguard HEVC - that's what Netflix uses AFAIK.

There's also some fun encoders from Apple and Dolby that are being use for HDR / Dolby Vision encoding that are apparently quite good.

[birdie]

Some words about Kingsoft HEVC codec.

Its website is http://www.ksyun.com/solution/video_cloud_solution
They have demo encoder/decoder here: https://github.com/ksvc/ks265codec - they claim they are far superior to x265.

[Sagittaire]

Some words about Kingsoft HEVC codec.

Its website is http://www.ksyun.com/solution/video_cloud_solution
They have demo encoder/decoder here: https://github.com/ksvc/ks265codec - they claim they are far superior to x265.

I make test with ksyun codec. It's really fast codec. Anyway x265 produce by far better metric and better visual quality.

[IgorC]

http://www.compression.ru/video/codec_comparison/hevc_2018/

Anyway there isn't much in free version

[Forteen88]

http://www.compression.ru/video/codec_comparison/hevc_2018/Anyway there isn't much in free versionI think they'll release more for free since their website says: "Express report on 2018 HEVC codecs comparison released!"
Other year's reports doesn't say "express report".

EDIT: Where can I get the HW265 and Tencent Shannon Encoder?
I want to make small tests to see if they really got better SSIM-score than x265.

[zub35]

The SSIM metric is easily fooled by blur. For purity test comparisons needs to do Blurring test.
If we compare codecs by only SSIM metric, then x264 and x 265 can be disabled PSY and get very high values

Unfortunately MSU does not do this. Although MSU VQMT allows you to do this.

For example

( SSIM(dB) + MSU_Blurring(dB*) ) / 2

* log(1-(encode_blur / source_blur))×(-10)

if encode_blur < source_blur then (encode_blur / source_blur) - [blurry image]
if encode_blur > source_blur then (source_blur / encode_blur) - [sharpness image]
if encode_blur = source_blur then 0.9999..9 - [depending on the number of decimal places in the calculation of the SSIM metric]

But here, all on the honesty of the tester. And it is unacceptable for low bitrates with very lossy

[Forteen88]

If we compare codecs by only SSIM metric, then x264 and x 265 can be disabled PSY and get very high values

Unfortunately MSU does not do this. Although MSU VQMT allows you to do this.Oh, I thought that MSU did set --tune ssim in x265 & x264. Oh, I'll better do my own tests then.

[Forteen88]

"HEVC Video Codecs Comparison 2018",
"FullHD report on 2018 HEVC codecs comparison released!",
http://www.compression.ru/video/codec_comparison/hevc_2018/

EDIT: Oh, you need to give email etc to get the more in-depth free info.

[ChaosKing]

http://download.compression.ru/codec_comparison_hevc_2018/MSU_HEVC_comparison_2018_FullHD_free-vajaz_1shivom.com-4827.zip
Don't know how long the link is valid.

[Forteen88]

ChaosKing :thanks:

[iwod]

http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/HEVC-VP9-AV1-and-VVC-Presenting-a-Codec-Update-in-11-Charts-127956.aspx

That shows HW265 is 20% better in VMAF than x265! ( Not shown in the free Report ) Has anyone used / tested that encoder?

[hajj_3]

http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/HEVC-VP9-AV1-and-VVC-Presenting-a-Codec-Update-in-11-Charts-127956.aspx

That shows HW265 is 20% better in VMAF than x265! ( Not shown in the free Report ) Has anyone used / tested that encoder?

HW265 is made by Huawei, as far as i know the general public can't get their hands on it.

[Atak_Snajpera]

http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/HEVC-VP9-AV1-and-VVC-Presenting-a-Codec-Update-in-11-Charts-127956.aspx

That shows HW265 is 20% better in VMAF than x265! ( Not shown in the free Report ) Has anyone used / tested that encoder?

You can immediately tell that those tests are fishy when scale starts from 60% ;)
https://dzceab466r34n.cloudfront.net/Images/ArticleImages/InlineImages/118265-CodecPic1.jpg-ORG.jpg

Let's be honest those newer codecs barely retain more details than old good x264 using veryslow preset.

[iwod]

You can immediately tell that those tests are fishy when scale starts from 60% ;)

Let's be honest those newer codecs barely retain more details than old good x264 using veryslow preset.

While the graph were made exaggerate the difference, the numbers don't lie. 20% difference is quite a lot, especially with VMAF.

x264 is still king at 8Mbps+, but x265 is much better at 1-2Mbps 2K and 4Mbps 4K. I would love to test HW265 myself.... ><

[benwaggoner]

Recent results comparing HEVC v. VVC v. xx265 v. LibAOM at SMPTE last week had some interesting attributes. Note that the x265 settings turned off all psy, had no tune, no b-adapt, fixed QP, no AQ, fixed GOP cadence, and a lot of other stuff one would never use in reality. Comparing HM to x265 WITHOUT using —tune psnr and disabling nearly all the x265 features that optimize quality did show HM producing a better PSNR, which is not surprising. Another interesting result is that HM offered worse objective metrics than libAOM, but significantly better subjective test results. VMAF wasn’t trained on HEVC or AV1 artifacts, so it’s unsurprising that it isn’t good at distinguishing between them.

VVC is also showing better subjective than objective metrics, with the proposed mechanism being more “natural” motion prediction that doesn’t yield a visually distracting block pattern.

[CSMR]

Note that the x265 settings turned off all psy, had no tune, no b-adapt, fixed QP, no AQ, fixed GOP cadence, and a lot of other stuff one would never use in reality.
Are you saying they chose settings which are worse than default for x265? They could hardly be expected to run millions of tests with all the possible combinations of settings.

Do you think x265 is in practice near competitive (in quality/speed terms) in a standard configuration with the four encoders ahead of it in the chart above?

[benwaggoner]

Are you saying they chose settings which are worse than default for x265? They could hardly be expected to run millions of tests with all the possible combinations of settings.
"Worse" is subjective. I asked the presenter about this in Q&A. They were really trying to figure out fundamental bitstream efficiency and algorithmic complexity. So they tried to turn off all "non-normative" features like psychovisual optimization, SIMD acceleration, etcetera. So it's nothing like a real-world encode, where exactly we want to use those things that do better than just a fixed QP encode.

As x265 wasn't ever designed to produce optimal efficiency in PSNR or SSIM without using --tune for one of those, nor was it designed to deliver optimal psychovisual quality without being able to use any per-frame or intraframe AQ, that it wasn't competitive doing those isn't particularly relevant. Letting x265 do its thing tuning for the test at hand with its full set of features almost certainly would yield better subjective ratings than any reference encoder, and at least competitive PSNR and SSIM scores.

As much as wish there was a "neutral" setting for an encoder, there are tons of implicit psychovisual tunings in encoders, bitstreams, and the basic math of video. Gamma is a perceptual optimization! The idea that mean PSNR or SSIM is useful is also a huge hand-wave. A mean PSNR of 30 could mean every frame is 30, or that half the frames are 10 and half are 50; those would be extremely different perceptual experiences. I would expect x265 would be biased towards the worse frames looking better, even if that reduces the mean, because that's a better subjective experience.

The researchers tested what they wanted to test, and documented that pretty well. The big issue for me was that the results are challenging to extrapolate to real-world scenarios. It'd be easy for someone not reading closely to say things like "AV1 is 50x slower than HEVC" without "comparing the reference encoders with all multithreading and SIMD optimization turned off. With SIMD on, AV1 is 4x slower." Or that "x265 is 500x faster than AV1" without "comparing single-threaded without SIMD. Encoding 10-bit UHD resolutions on a c5.8xlarge instance, x265 is <my ballpark estimate> 5000x faster than AV1 encoding with both encoders optimally configured to deliver VMAF 80 at 15 Mbps."

Do you think x265 is in practice near competitive (in quality/speed terms) in a standard configuration with the four encoders ahead of it in the chart above?
All the data so far is that HEVC produces the best subjective quality at a given bitrate of any of H.264, HEVC, and AV1 today. I am also very confident that x265 will produce superior results, MUCH faster, than the HM for real-world scenarios. My most optimistic estimate is that an AV1 encoder wouldn't be capable of matching x265 bit-for-bit for real-world encoding scenarios before end of 2019.

[cyberbrain]

Need link for downlaod tencent shannon software encoder (HEVC)