MSU released benchmark of online video encoding services

[Dyomich]

Hi! MSU Graphics & Media Lab Video Group has released a new Codecs Comparison Report and we are happy to present it to you.

This test included encoding performance comparison using services' default settings. This comparison is based on these main points:

• 7 cloud-based video-encoding services were compared: 3 with their own storages (Alibaba ApsaraVideo for Media Processing, Kingsoft Cloud and Tencent Media Processing Service) and 4 with storages at Amazon S3 (AWS Elemental MediaConvert, Coconut, Qencode, and Zencoder)
• Encoding time measurement. We analyzed how much time a job waits in the services’ queue.
• Default encoding settings efficiency. We launched encoding out-of-the-box without changing services options and measured objective video quality.
• Automatic sequence-based encoding options (per-title encoding) for services that support this feature.
• Test setup: 15 FullHD videos of different types and content (including UGC). Two types of measurements: the main case involved encoding into FullHD, and the additional case included transcoding into several resolutions. And finally, we tried to use anonymous accounts for testing for fair comparison

In our previous comparison (2019 Cloud Comparison) the results showed that for the same quality, the file-size difference can reach 100% and the cost per minute difference is 700%. This year’s comparison showed about 80% file-size difference and 150% cost per minute difference. Below we show the winners by different categories.

Best services in speed/quality trade-off
We measured encoded video quality and encoding time using services’ callbacks. This nomination shows how frequently a service was Pareto-optimal for encoding speed and quality scores for HEVC and H.264 averaged for all test videos (SSIM metric). Results of AWS Elemental MediaConvert were used as a reference to get quality scores.

The most frequent Pareto-optimal services in terms of encoding speed and quality were:

• Tencent Media Processing Service
• Zencoder
• Kingsoft Cloud

Also Alibaba ApsaraVideo for Media Processing showed good quality for higher encoding speed.

The plot below shows an average encoding speed and file size among all videos. This statistics is a bit different from a number of videos, where a service showed optimal scores.




Best services in cost/quality trade-off
This nomination shows how frequently a service was Pareto-optimal for video encoding cost and quality among all test videos. The following chart shows relative bitrate and encoding cost for the same quality (SSIM) averaged for all videos. The best quality for the least prices has:

• Tencent Media Processing Service showed best encoding efficiency
• “Kingsoft Cloud, Qencode (for H.264 and HEVC encoding) and Coconut (for HEVC encoding) are the cheapest and keep a good bitrate at the same time

Per-title encoding test
In addition to default encoding settings, per-sequence encoding optimization was tested for services, which supported this feature during testing. Unfortunately, we didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started, and Qencode sequence-based encoding, because it produces only one output without any bitrate or resolution variation.

Per-title encoding option optimizes encoding settings for each input video to get better encoding performance. The first and second places in nomination for the best quality per-title encoding:

• Tencent Media Processing Service
• Alibaba ApsaraVideo for Media Processing

Here we show the winners in different categories in one table.


We should say that using only default settings may not provide a fully comprehensive comparison. Some services can set base profiles by default for better streams compatibility. On the other hand, when we tried to set the same settings for all services, we found out that sometimes we can not vary encoding options in service interfaces.

There may be different ways on how to compare services, and we would really love to hear which scenarios are interesting for you. Please, tell us what information in our comparison was the most useful for you? And, more importantly, what would you like to see in future comparisons? Or what would you like to change in our comparisons?
You can help to make codecs comparisons better!

Stay tuned!

[benwaggoner]

The mean YUV-SSIM value of all frames in a piece of content really isn't a very useful metric. It's better than Y-only PSNR (which we see a lot). But the mean of a bunch of frames doesn't account for variability in the quality throughout an encode. A file that oscillates between -5 dB and -15 dB has the same mean as a consistent -10 dB, but will look much worse.

A single value rating of the quality of a file of >>10 second is really an unsolved problem for the industry, and an essential one if we want to make broad comparisons like this.

Even just publishing the variance of the metrics would help a lot.