MSU Video Codecs Comparison 2019 - CALL FOR CODECS

[Dyomich]

Dear video codec developers,

Moscow State University Graphics&Media Lab starts the fourteenth annual video codecs comparison.
We invite all codec developers to take part in the competition.
Applications are open until March 1.

Participation is free with the publishing of all obtained results.
Also, you can compare your codec staying incognito (contact us for private participation if you are interested).

Detailed information can be found on call-for-codecs page.

Nominations (encoding use-cases)
FullHD: Fast (60fps)
FullHD: Universal (25fps)
FullHD: Ripping (1fps)
FullHD: Ultra-Ripping
4K (20fps)
Subjective comparison
Cloud solutions

New features of MSU Video Codecs Comparisons

• Increasing the number of test videos up to 100+ in fast use case, adding UGC (user-generated) videos (vlogs, noise videos, etc.)
• Open to compare cloud-based solutions
• Enlarging of video collection (up to 20 000 videos)
• Subjective comparisons (with Subjectify.us platform)
• Upgraded testing hardware to Coffee Lake
• Supporting VMAF metric

You can apply for the contest via the form or e-mail.

Useful Links
Call-for-codecs 2019 page
http://compression.ru/video/codec_co...codecs_19.html
MSU Video Codecs Comparison 2018 (3 parts: FullHD, Subjective, 4K)
http://compression.ru/video/codec_comparison/hevc_2018/
Link for subscription to report news
http://compression.ru/video/codec_co...018/#subscribe

Also, we will be happy if you leave any feedback or suggestions for our previous and future reports: http://compression.ru/video/codec_co.../feedback.html

-----
Best regards,
Dr. Dmitriy Kulikov,
Moscow State University (MSU)
Graphics&Media Lab
Videocodec Testing Team
videocodec-testing@graphics.cs.msu.ru

[benwaggoner]

It's nice to see this year's updates! I look forward to the results. I do have a few questions and comments, based on this post and the linked rules.

1. It is very odd to read "ripping" in 2019. Might be better to call it "offline" or "high quality."
2. Ultra-ripping is only half the speed of ripping; not a very big jump. I'd suggest making encoding time unlimited (but reported), or <=0.25fps. AV1 can get SLOW
3. Having UHD only at 20fps seems really limited; that's fewer MIPS/pixel than FullHD Fast! Having a 1fps test and perhaps a 0.2fps would allow use of more advanced HEVC features in UHD.
4. It seems there should be SOME limit on GOP duration, probably somewhere 4-10 seconds. That will ensure IDR placement and quality of inter-GOP transitions is included.
5. I also suggest having some VBV limits and level limits. It would be reasonable to limit to at least the maximum allowed in the lowest profile @ level that supports the frame size and fps targeted. Accurate VBV compliance and maintaining quality at VBV peaks is an important encoder feature.
6. VMAF now supports mobile, HD, and UHD scores. It would be good to have all of those available. And the UHD tests should focus on the UHD VMAF. I also recommend using the harmonic mean (HVMAF) instead of just mean, as that is more sensitive to individual low-quality frames.
7. It isn't specified if this will be 8-bit or 10-bit encoding. Probably 8-bit?
8. It isn't specified if this is all SDR or if there is some HDR. Optimal HDR encoding would require different parameters.
9. 1 Mbps might be too high a bottom floor for 1080p24 HEVC. I've gotten interesting results <<1 Mbps in my codec shootout http://forum.doom9.org/showthread.php?t=175776
10. Personally, I'd love to see some dual-Xeon examples as performance optimization for multi-socket systems is challenging, but that is probably what most UHD HEVC encoding is done on.
11. I see that it is possible to nominate a "cloud solution" but there is no data on what that covers/entails.

More broadly, I encourage some rumination and clarity around objective versus subjective metrics. The interaction of tuning and metrics is complex. Many modern encoders have modes that will optimize for PSNR, SSIM, subjective quality, and in some cases VMAF. Which metric should proponents tune for? I would argue that subjective quality is the only meaningful metric, and the others are informational to ballpark subjective quality without having to do subjective evaluation. VMAF itself is a ML system to predict subjective quality ratings, which is what VMAF score attempts to predict. So, what's the goal here? If it is to see how well each codec can do for each metric, it seems reasonable to let proponents provide settings tuned for each metric, and then to score the tuned version for each metric to rate that metric. If the goal is to make a "general" encode intended to score well in PSNR, SSIM, VMAF, and subjective ratings, that seems muddled, and will result in parameters that somewhat degrade subjective quality in order to bump scores on the other metrics. It is silly to tune an encoder to produce better PSNR at the expense of worse looking video! I would suggest focusing on subjective ratings, and specifying that is what proponents should tune for. PSNR, SSIM, and VMAF can still be measured at reported; it would be interesting to see how they do and don't correlate with subjective ratings. But someone picking an encoder for real-world use should ONLY care about subjective quality.

I argue that the focus on PSNR in codec development and encoder tuning has become more of a distraction than a help. PSNR made sense decades ago as something with some subjective correlation that was cheap to calculate. But it's always been inaccurate, and its subjective quality correlation goes down the more encoders tune for PSNR specifically. This happens with every metric; once it because a key differentiator, developers start to develop for the metric in ways that improve the metric more than they improve subjective quality, meaning the metric has less subjective correlation. Encoders with --tune vmaf will have less meaningful VMAF ratings (although tuning for VMAF will certainly be subjectively superior to tuning for PSNR!). We've already seen studies where libaom produces better PSNR but lower subjective ratings than the HM.

Given the huge number of clips that'll be produced, I don't know if it is feasible to get MOS scores for all of them. But doing as many as you can (for perhaps a limited set of sources and bitrates that the objective metrics show to be "interesting") will make your report much more meaningful.

After all, these reports trigger lots of discussion about "my codec is best!" so, fronting the best "best" is great for the industry.

[Blue_MiSfit]

I'd LOVE to see some focus on CBR / very capped VBR (e.g. maxrate = 1.1x bitrate) as this is quite commonly used for ABR streaming.

[benwaggoner]

Originally Posted by Blue_MiSfit

I'd LOVE to see some focus on CBR / very capped VBR (e.g. maxrate = 1.1x bitrate) as this is quite commonly used for ABR streaming.

2-sec CBR fixed Closed GOP is used a lot out there. Preventing keyframe strobing with that is an interesting challenge.

When I do encoder comparisons I normally do a 1-pass 2-sec fixed closed GOP CBR and a 2-pass max 5 sec GOP with VBR (with VBV limitations). Those two captures two good real world scenarios but exercising quite different encoder features.

I care about "without VBV" tests only slightly more than I care about PSNR plots. My customers will never see either .

[excellentswordfight]

As benwaggoner touched as well, 20fps for the UHD-test seems to be a far to high target, especially if the test platform is a consumer grade CPU. Somewere in the 1-5fps range seem far more reasonable, or at least have two UHD tests with different speed targets.

I also dont really see the ripping and ultra ripping use cases. If these are "ripps" with basically unrestricted vbv limits and GOP settings, this is a consumer scenario, cause I cant see any professional use case for this, and if its a consumer scenario, I dont see any point in going for settings that has such a big speed penilty that they become irrelevent for everything except for academic purpose, cause I dont see a huge market for "ripping" FHD at <1fps.

[IgorC]

It also will be great to see some results for HDR content. MSU have included 4K in comparison but they left HDR out. Also some formats greatly benefit from 10 bits like H.264 and specially VP9.

[Dyomich]

The team of MSU codecs comparisons thank you for your valuable comments and advice.

Originally Posted by benwaggoner

1. It is very odd to read "ripping" in 2019. Might be better to call it "offline" or "high quality."

Thank you for suggesting, it's true and we will change the name next year (in this year all rules are already fixed and have been sent to participants).

Originally Posted by benwaggoner

2. Ultra-ripping is only half the speed of ripping; not a very big jump. I'd suggest making encoding time unlimited (but reported), or <=0.25fps. AV1 can get SLOW

Yes, of course. We had a typo on site, in our comparisons Ultra-ripping use-case doesn't have speed limits (actually it's limited by AV1 speed at about 0.005fps)

Originally Posted by benwaggoner

3. Having UHD only at 20fps seems really limited; that's fewer MIPS/pixel than FullHD Fast! Having a 1fps test and perhaps a 0.2fps would allow use of more advanced HEVC features in UHD.

It's a reasonable notice and I think we will correct it or make 2 use cases for 4K (but also next year), because some encoders users have interest in high-speed UHD encoding too.

Originally Posted by benwaggoner

4. It seems there should be SOME limit on GOP duration, probably somewhere 4-10 seconds. That will ensure IDR placement and quality of inter-GOP transitions is included.

Maybe there should be some limitations, but it is hard to choose a particular value. We decided to discard limiting GOP durations in according to experts (and participants) recommendations when we started the developments of our methodology in early comparisons. Also, comparisons participants often send us codecs of their own standards (not only HEVC and H.264). Thus, limiting GOP is unfair for such multi-standard comparison.

Originally Posted by benwaggoner

5. I also suggest having some VBV limits and level limits. It would be reasonable to limit to at least the maximum allowed in the lowest profile @ level that supports the frame size and fps targeted. Accurate VBV compliance and maintaining quality at VBV peaks is an important encoder feature.

This idea again doesn't work very well for other standards. We can set it, but in many cases, we simply couldn't check it. So it'd be unfair for participants who follow the rules if someone doesn't do it. Also, many codecs are tested with profiles that include VBV limitations (all encoding presets are chosen by the developers). And finally, this is quite an uncommon case for which we haven't received many requests.

Originally Posted by benwaggoner

6. VMAF now supports mobile, HD, and UHD scores. It would be good to have all of those available. And the UHD tests should focus on the UHD VMAF. I also recommend using the harmonic mean (HVMAF) instead of just mean, as that is more sensitive to individual low-quality frames.

Using HVMAF and different scores sounds well and we will consider this question. But it should be noticed that VMAF is not perfect still. This was already confirmed by independent subjective tests (https://euclidiq.com/2018/01/16/well...video-quality/) and also by our research that will be published soon. However, VMAF is constantly changing so we will definitely follow the updates and use newer versions in comparisons.

Originally Posted by benwaggoner

7. It isn't specified if this will be 8-bit or 10-bit encoding. Probably 8-bit?

Now we use only 8-bit. Soon we will publish a full description of our methodology for codecs testing.

Originally Posted by benwaggoner

8. It isn't specified if this is all SDR or if there is some HDR. Optimal HDR encoding would require different parameters.

For now we use only SDR, but we are considering adding HDR in future comparisons.

Originally Posted by benwaggoner

9. 1 Mbps might be too high a bottom floor for 1080p24 HEVC. I've gotten interesting results <<1 Mbps in my codec shootout http://forum.doom9.org/showthread.php?t=175776

That's an interesting suggestion, but on the other hand we have already done it (in previous comparisons) and many users and participants were not satisfied when we used low bitrates (<1Mbps) for FullHD content. So it may depend on opinion. Now we have a start bitrate 1Mbps in according to a big number of participants requests. Of course, many codecs are not optimized for low bitrates and don't keep them. But it's also not clear what use case is suitable for low bitrates on FullHD encoding (instead of decreasing resolution).

Originally Posted by benwaggoner

10. Personally, I'd love to see some dual-Xeon examples as performance optimization for multi-socket systems is challenging, but that is probably what most UHD HEVC encoding is done on.

It is an interesting direction and we have already done a comparison on server platform (Xeon E5, in 2015 http://compression.ru/video/codec_comparison/hevc_2015/). Unfortunately, now we don't have such new server hardware and as an academic organization, we currently don't have an ability to purchase it . By the way, the results in our 2015 comparison weren't very different from the desktop platform. Also, this is a separate case for which not all participants could be easily optimized.

Originally Posted by benwaggoner

11. I see that it is possible to nominate a "cloud solution" but there is no data on what that covers/entails.

This year it will be small experiment - we are going to register on a number of cloud transcoding platforms and encode a set of videos with default settings in the cloud. Then, we will perform subjective (and some objective) comparisons for the report.
Actually, we did such kind of comparison several years ago, but the results weren't worthy of publication (low number of videos, and it was hard to perform big subjective comparisons). Now we want to try this format again and if it becomes popular we hope to make it a part of our annual reports.


We agree with the importance of subjective comparisons, and we are trying to improve ours. At the moment, we decided to increase the number of tested videos by user requests, but so far a hundred of videos will be used only in the fast use case and objective comparison. For subjective comparison, there will not be such a significant increase this year. However, --tune options in objective studies don’t always make it less representative. In 2017 subjective report (Part 3) http://compression.ru/video/codec_co...ubjective.pdf/ we performed an additional study on --tune ssim option which showed that the subjective quality score earned by the codec with the --tune ssim option enabled is always higher than the score of the same codec with this option disabled. The attached plot reveals that codecs with the --tune ssim option enabled outperform their siblings with this option disabled by a huge margin. The results of this study justify keeping this option enabled in the main study.

[Dyomich]

Originally Posted by Blue_MiSfit

I'd LOVE to see some focus on CBR / very capped VBR (e.g. maxrate = 1.1x bitrate) as this is quite commonly used for ABR streaming.

This is an interesting case for investigation which we haven't done yet. Of course, we don't cover all use cases in our annual reports and firstly concentrate on the participants requests. But in some years we do special comparisons and reports on specific use cases (like cloud comparison now). So thank you for your suggestion, we will consider this case in future reports.

[Dyomich]

Originally Posted by excellentswordfight

As benwaggoner touched as well, 20fps for the UHD-test seems to be a far to high target, especially if the test platform is a consumer grade CPU. Somewere in the 1-5fps range seem far more reasonable, or at least have two UHD tests with different speed targets.

I also dont really see the ripping and ultra ripping use cases. If these are "ripps" with basically unrestricted vbv limits and GOP settings, this is a consumer scenario, cause I cant see any professional use case for this, and if its a consumer scenario, I dont see any point in going for settings that has such a big speed penilty that they become irrelevent for everything except for academic purpose, cause I dont see a huge market for "ripping" FHD at <1fps.

Thank you for your comment on UHD fps, we are going to change it or add additional use case next year.
Well, there is a lot of participants every year who are interesting in participating only in "Ripping" nominations (you can check it in the reports). AV1 and VP9 are not an exception and I don't think we can say that they don't have a market

[Dyomich]

Originally Posted by IgorC

It also will be great to see some results for HDR content. MSU have included 4K in comparison but they left HDR out. Also some formats greatly benefit from 10 bits like H.264 and specially VP9.

Thank you for your comment, these are interesting cases. We have added 4K as it became quite popular, but I think it's a bit early to say so about HDR and 10bit. Of course, we consider adding it in our future comparisons, but yet not this/next year.

[benwaggoner]

Originally Posted by Dyomich

Thank you for your comment, these are interesting cases. We have added 4K as it became quite popular, but I think it's a bit early to say so about HDR and 10bit. Of course, we consider adding it in our future comparisons, but yet not this/next year.

Amazon launched HDR 10-bit back in June 2015, and most Amazon and Netflix original programming is in HDR now. It's an important current scenario.

Also, lots of UHD SDR content is also delivered in 10-bit.

There is a lack of good metrics for HDR, so this would be a great area to see some research.

[IgorC]

Originally Posted by Dyomich

We have added 4K as it became quite popular, but I think it's a bit early to say so about HDR and 10bit.

As Ben has already said Amazon and Netflix already stream in HDR.

Also UltraHD/4k Blu Ray has mandatory support for HDR. It means that pretty every Ultra HD BR title is at least HDR.

Netflix 4k titles as well are always in 10 bits . https://www.streamingmedia.com/Artic...ticleID=118853