MSU HEVC/AV1 Video Codecs Comparison 2019

[Dyomich]

MSU Video-Codec Comparison 2019 FullHD and Subjective reports are released http://compression.ru/video/codec_comparison/hevc_2019/

Free versions of the reports are available from direct links (FullHD (http://compression.ru/video/codec_comparison/hevc_2019/download/main_report_free_version/Video_Codec_Comparison_FullHD_2019_report_free_version.zip) and Subjective (http://compression.ru/video/codec_comparison/hevc_2019/download/subjective_report_free_version/Free_Subjective_Video_Codec_Comparison_Report_2019_by_MSU.zip)).
If someone is interested in the enterprise version, you can get it with 20% discount before 30/11/19 (use this link (https://store.payproglobal.com/checkout?products[1]=56089&coupon-code-to-add=HEVC-NOVEMBER-2019-DN) or a promo HEVC-NOVEMBER-2019-DN)

This year we used 100 videos for our tests (list of videos and descriptions: http://compression.ru/video/codec_comparison/hevc_2019/videos.html)

The winners in objective comparison are:
[I]HW265 (Huawei) by SSIM
Tencent V265 Encoder by VMAF
Best bitrate handling: x264 in fast encoding use-case, HW265 in universal and ripping encoding use-cases
Most often was Pareto-optimal: HW265 in fast encoding use-case, Tencent V265 Encoder in universal, SIF and x264 in ripping use-case.

Results of subjective comparison:
http://compression.ru/video/codec_comparison/hevc_2019/figures/subjective_overall.jpg

Soon the comparison of cloud transcoding services will be released, and also we will perform 4K and high-quality (including AV1) comparisons this year.
Any feedback is very welcome! :)

[Tadanobu]

These reports are always very interesting. Thank you.

I can't wait to see another study with the whole SVT family, libaom, rav1e and all the recent newcomers.

[Atak_Snajpera]

I just love when y Axis does not start from 0...

Fixed
https://i.postimg.cc/DZQVGpzk/graph.png

[benwaggoner]

Is there a detailed description anywhere of the subjective testing methodology?

[Emulgator]

I just love when y Axis does not start from 0...
Hits me too. I always want to see where zero is.

[Dyomich]

You can find the description in the PDF of Subjective report 2019 (http://compression.ru/video/codec_comparison/hevc_2019/download/subjective_report_free_version/Free_Subjective_Video_Codec_Comparison_Report_2019_by_MSU.zip)

Also, last year we described study method in a separate PDF (2018 subjective study method (http://compression.ru/video/codec_comparison/hevc_2018/pdf/subjective_comparison_method.pdf)). This year the methodology was the same, with one difference that the participants of FullHD comparison could use another codec binary and presets for subjective comparison.

Some questions are considered on subjectify.us (http://www.subjectify.us) website. If there are other questions, we can discuss them here:)

[TEB]

Im curious about the HW and Tencent encoders, are they available for purchase, testing etc?

[birdie]

WZAurora AV1 codec in case anyone wonders is from https://www.visionular.com

[birdie]

Their AV1 codec marketing presentation (https://yx-web-nosdn.netease.im/quickhtml%2Fassets%2Fyunxin%2Fdefault%2F20191030181322.pdf).

[Capella Systems]

I couldn't find out from the PDF since it doesn't have the command lines used. But, x264/x265 need different 'tune' for optimizing for PSNR, SSIM or for subjective tests. Were different versions created for the metrics-based tests and for the subjective tests? And if not, what were the settings tuned for?

[RanmaCanada]

I do love how it appears that almost none of these "encoders" are available to the general public.

[Dyomich]

I couldn't find out from the PDF since it doesn't have the command lines used. But, x264/x265 need different 'tune' for optimizing for PSNR, SSIM or for subjective tests. Were different versions created for the metrics-based tests and for the subjective tests? And if not, what were the settings tuned for?

Participants had the ability to send different binaries and preset for objective and subjective comparisons. Regarding x264 and x265, this year they used the same preset for subjective and objective tests. But, our 2017 subjective comparison (http://compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_P3_subjective.pdf) showed, that '--tune ssim' increases visual quality, so using this option in subjective tests makes sense.