MSU Cloud Benchmark 2020

[Dyomich]

Last year our video-codecs testing group released a report on video transcoding clouds comparison-2019 (http://compression.ru/video/codec_comparison/cloud_2019/).
We compared a price and a quality of encoding in 6 services (Alibaba, Amazon Elastic Transcoder, AWS Elemental MediaConvert, Coconut, Qencode, Zencoder). We are going to expand this comparison this year: compare more services, add more analytical tools, use cases and provide maximally accurate results.

To provide more reliable comparison results in the report, we invite you to validate the 2020 comparison results in cloud-encoding services. If you use any cloud encoding service, you can use your own account for testing (or register a new one). We will provide enterprise reports for all contributors and ready to pay for cloud encoding expenses. To participate, you can fill up this form (http://compression.ru/video/codec_comparison/cloud_2020/#contribute) or contact us: cloud@compression.ru
More info about contribution to Cloud Benchmark 2020 (http://compression.ru/video/codec_comparison/cloud_2020/figures/page_for_cloud_report_contribute.png)

Due to events related to COVID-19, the dates might be extended. Relevant information will be posted on MSU Cloud Benchmark 2020 page: http://compression.ru/cloud2020/ and via newsletter (http://compression.ru/video/codec_comparison/cloud_2020/#subscribe).
Results obtained in autumn 2019:
http://compression.ru/video/codec_comparison/cloud_2019/figures/cloud2019_cost_quality.png

[benwaggoner]

What does "average relative bitrate" indicate? Some sort of compression efficiency (VMAF @ bitrate)?

And are the axes linear? So is cost of 8 8x more than a cost of 1? And a 0.5 average relative bitrate means same quality at 1/2 the bitrate?

Also, the symbols on the chart vary a lot in size; it would help for the smaller ones to match the larger in area for better legibility.

[Dyomich]

What does "average relative bitrate" indicate? Some sort of compression efficiency (VMAF @ bitrate)?
And are the axes linear? So is cost of 8 8x more than a cost of 1? And a 0.5 average relative bitrate means same quality at 1/2 the bitrate?
Yes, here "average relative bitrate" indicates the difference between RD-curves (YUV-SSIM quality) of competitors and reference (an H.264 AWS Elemental Media Convert was chosen as a reference) integrated by quality. So, and 0.5 average relative bitrate means the same quality at 1/2 the bitrate, and the cost of "8" is 8x more than a cost of 1. The axes are linear.
(more explanation is given in the report (D.4. Bitrate Ratio for the Same Quality) (http://compression.ru/video/codec_comparison/cloud_2019/download/MSU_Cloud_Services_Comparison_Report_2019.pdf), in this report encoding speed was replaced by encoding cost. In the next report we are planning to take encoding speed into account too)

Also, the symbols on the chart vary a lot in size; it would help for the smaller ones to match the larger in area for better legibility.
This chart was exported from the interactive version, but it can be zoomed in HTML report or here: http://compression.ru/video/codec_comparison/cloud_2019/#report_info

[benwaggoner]

Thanks for the update.

For MS-SSIM is that just the arithmetic mean of the scores of each frame? I worry that makes rate control matter a lot less, since quality variability can be a huge problem, but not captured by mean-of-frames metrics.

Also, what a medium preset means can vary a lot between vendors. A useful alternative view would be looking at quality at fixed cost (time or $).

[foxyshadis]

Thanks for the update.

For MS-SSIM is that just the arithmetic mean of the scores of each frame? I worry that makes rate control matter a lot less, since quality variability can be a huge problem, but not captured by mean-of-frames metrics.

Also, what a medium preset means can vary a lot between vendors. A useful alternative view would be looking at quality at fixed cost (time or $).

Given how coarse some of the platforms are, even just a chart of fast-medium-slow for each would be substantially more expressive. Based on the appendix, they seem to have run quite a few runs on each.