Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd April 2020, 16:19   #1  |  Link
Dyomich
Codec Analysis Expert
 
Join Date: Feb 2006
Location: Moscow
Posts: 37
MSU Cloud Benchmark 2020

Last year our video-codecs testing group released a report on video transcoding clouds comparison-2019.
We compared a price and a quality of encoding in 6 services (Alibaba, Amazon Elastic Transcoder, AWS Elemental MediaConvert, Coconut, Qencode, Zencoder). We are going to expand this comparison this year: compare more services, add more analytical tools, use cases and provide maximally accurate results.

To provide more reliable comparison results in the report, we invite you to validate the 2020 comparison results in cloud-encoding services. If you use any cloud encoding service, you can use your own account for testing (or register a new one). We will provide enterprise reports for all contributors and ready to pay for cloud encoding expenses. To participate, you can fill up this form or contact us: cloud@compression.ru
More info about contribution to Cloud Benchmark 2020

Due to events related to COVID-19, the dates might be extended. Relevant information will be posted on MSU Cloud Benchmark 2020 page: http://compression.ru/cloud2020/ and via newsletter.
Results obtained in autumn 2019:

Last edited by Dyomich; 2nd April 2020 at 16:20. Reason: fix typos
Dyomich is offline   Reply With Quote
Old 2nd April 2020, 21:47   #2  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
What does "average relative bitrate" indicate? Some sort of compression efficiency (VMAF @ bitrate)?

And are the axes linear? So is cost of 8 8x more than a cost of 1? And a 0.5 average relative bitrate means same quality at 1/2 the bitrate?

Also, the symbols on the chart vary a lot in size; it would help for the smaller ones to match the larger in area for better legibility.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 3rd April 2020, 14:04   #3  |  Link
Dyomich
Codec Analysis Expert
 
Join Date: Feb 2006
Location: Moscow
Posts: 37
Quote:
Originally Posted by benwaggoner View Post
What does "average relative bitrate" indicate? Some sort of compression efficiency (VMAF @ bitrate)?
And are the axes linear? So is cost of 8 8x more than a cost of 1? And a 0.5 average relative bitrate means same quality at 1/2 the bitrate?
Yes, here "average relative bitrate" indicates the difference between RD-curves (YUV-SSIM quality) of competitors and reference (an H.264 AWS Elemental Media Convert was chosen as a reference) integrated by quality. So, and 0.5 average relative bitrate means the same quality at 1/2 the bitrate, and the cost of "8" is 8x more than a cost of 1. The axes are linear.
(more explanation is given in the report (D.4. Bitrate Ratio for the Same Quality), in this report encoding speed was replaced by encoding cost. In the next report we are planning to take encoding speed into account too)

Quote:
Originally Posted by benwaggoner View Post
Also, the symbols on the chart vary a lot in size; it would help for the smaller ones to match the larger in area for better legibility.
This chart was exported from the interactive version, but it can be zoomed in HTML report or here: http://compression.ru/video/codec_co...9/#report_info
Dyomich is offline   Reply With Quote
Old 3rd April 2020, 17:00   #4  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Thanks for the update.

For MS-SSIM is that just the arithmetic mean of the scores of each frame? I worry that makes rate control matter a lot less, since quality variability can be a huge problem, but not captured by mean-of-frames metrics.

Also, what a medium preset means can vary a lot between vendors. A useful alternative view would be looking at quality at fixed cost (time or $).
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 6th April 2020, 10:42   #5  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Quote:
Originally Posted by benwaggoner View Post
Thanks for the update.

For MS-SSIM is that just the arithmetic mean of the scores of each frame? I worry that makes rate control matter a lot less, since quality variability can be a huge problem, but not captured by mean-of-frames metrics.

Also, what a medium preset means can vary a lot between vendors. A useful alternative view would be looking at quality at fixed cost (time or $).
Given how coarse some of the platforms are, even just a chart of fast-medium-slow for each would be substantially more expressive. Based on the appendix, they seem to have run quite a few runs on each.
foxyshadis is offline   Reply With Quote
Reply

Tags
cloud, codec comparison, control, testing, transcoding

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:53.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.