View Single Post
Old 4th March 2019, 13:11   #7  |  Link
Dyomich
Codec Analysis Expert
 
Join Date: Feb 2006
Location: Moscow
Posts: 37
The team of MSU codecs comparisons thank you for your valuable comments and advice.
Quote:
Originally Posted by benwaggoner View Post
1. It is very odd to read "ripping" in 2019. Might be better to call it "offline" or "high quality."
Thank you for suggesting, it's true and we will change the name next year (in this year all rules are already fixed and have been sent to participants).
Quote:
Originally Posted by benwaggoner View Post
2. Ultra-ripping is only half the speed of ripping; not a very big jump. I'd suggest making encoding time unlimited (but reported), or <=0.25fps. AV1 can get SLOW
Yes, of course. We had a typo on site, in our comparisons Ultra-ripping use-case doesn't have speed limits (actually it's limited by AV1 speed at about 0.005fps)
Quote:
Originally Posted by benwaggoner View Post
3. Having UHD only at 20fps seems really limited; that's fewer MIPS/pixel than FullHD Fast! Having a 1fps test and perhaps a 0.2fps would allow use of more advanced HEVC features in UHD.
It's a reasonable notice and I think we will correct it or make 2 use cases for 4K (but also next year), because some encoders users have interest in high-speed UHD encoding too.
Quote:
Originally Posted by benwaggoner View Post
4. It seems there should be SOME limit on GOP duration, probably somewhere 4-10 seconds. That will ensure IDR placement and quality of inter-GOP transitions is included.
Maybe there should be some limitations, but it is hard to choose a particular value. We decided to discard limiting GOP durations in according to experts (and participants) recommendations when we started the developments of our methodology in early comparisons. Also, comparisons participants often send us codecs of their own standards (not only HEVC and H.264). Thus, limiting GOP is unfair for such multi-standard comparison.
Quote:
Originally Posted by benwaggoner View Post
5. I also suggest having some VBV limits and level limits. It would be reasonable to limit to at least the maximum allowed in the lowest profile @ level that supports the frame size and fps targeted. Accurate VBV compliance and maintaining quality at VBV peaks is an important encoder feature.
This idea again doesn't work very well for other standards. We can set it, but in many cases, we simply couldn't check it. So it'd be unfair for participants who follow the rules if someone doesn't do it. Also, many codecs are tested with profiles that include VBV limitations (all encoding presets are chosen by the developers). And finally, this is quite an uncommon case for which we haven't received many requests.
Quote:
Originally Posted by benwaggoner View Post
6. VMAF now supports mobile, HD, and UHD scores. It would be good to have all of those available. And the UHD tests should focus on the UHD VMAF. I also recommend using the harmonic mean (HVMAF) instead of just mean, as that is more sensitive to individual low-quality frames.
Using HVMAF and different scores sounds well and we will consider this question. But it should be noticed that VMAF is not perfect still. This was already confirmed by independent subjective tests (https://euclidiq.com/2018/01/16/well...video-quality/) and also by our research that will be published soon. However, VMAF is constantly changing so we will definitely follow the updates and use newer versions in comparisons.
Quote:
Originally Posted by benwaggoner View Post
7. It isn't specified if this will be 8-bit or 10-bit encoding. Probably 8-bit?
Now we use only 8-bit. Soon we will publish a full description of our methodology for codecs testing.
Quote:
Originally Posted by benwaggoner View Post
8. It isn't specified if this is all SDR or if there is some HDR. Optimal HDR encoding would require different parameters.
For now we use only SDR, but we are considering adding HDR in future comparisons.
Quote:
Originally Posted by benwaggoner View Post
9. 1 Mbps might be too high a bottom floor for 1080p24 HEVC. I've gotten interesting results <<1 Mbps in my codec shootout http://forum.doom9.org/showthread.php?t=175776
That's an interesting suggestion, but on the other hand we have already done it (in previous comparisons) and many users and participants were not satisfied when we used low bitrates (<1Mbps) for FullHD content. So it may depend on opinion. Now we have a start bitrate 1Mbps in according to a big number of participants requests. Of course, many codecs are not optimized for low bitrates and don't keep them. But it's also not clear what use case is suitable for low bitrates on FullHD encoding (instead of decreasing resolution).
Quote:
Originally Posted by benwaggoner View Post
10. Personally, I'd love to see some dual-Xeon examples as performance optimization for multi-socket systems is challenging, but that is probably what most UHD HEVC encoding is done on.
It is an interesting direction and we have already done a comparison on server platform (Xeon E5, in 2015 http://compression.ru/video/codec_comparison/hevc_2015/). Unfortunately, now we don't have such new server hardware and as an academic organization, we currently don't have an ability to purchase it . By the way, the results in our 2015 comparison weren't very different from the desktop platform. Also, this is a separate case for which not all participants could be easily optimized.
Quote:
Originally Posted by benwaggoner View Post
11. I see that it is possible to nominate a "cloud solution" but there is no data on what that covers/entails.
This year it will be small experiment - we are going to register on a number of cloud transcoding platforms and encode a set of videos with default settings in the cloud. Then, we will perform subjective (and some objective) comparisons for the report.
Actually, we did such kind of comparison several years ago, but the results weren't worthy of publication (low number of videos, and it was hard to perform big subjective comparisons). Now we want to try this format again and if it becomes popular we hope to make it a part of our annual reports.


We agree with the importance of subjective comparisons, and we are trying to improve ours. At the moment, we decided to increase the number of tested videos by user requests, but so far a hundred of videos will be used only in the fast use case and objective comparison. For subjective comparison, there will not be such a significant increase this year. However, --tune options in objective studies don’t always make it less representative. In 2017 subjective report (Part 3) http://compression.ru/video/codec_co...ubjective.pdf/ we performed an additional study on --tune ssim option which showed that the subjective quality score earned by the codec with the --tune ssim option enabled is always higher than the score of the same codec with this option disabled. The attached plot reveals that codecs with the --tune ssim option enabled outperform their siblings with this option disabled by a huge margin. The results of this study justify keeping this option enabled in the main study.

Dyomich is offline   Reply With Quote