Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > VP9 and AV1

Reply
 
Thread Tools Search this Thread Display Modes
Old 30th January 2019, 01:00   #1401  |  Link
TD-Linux
Registered User
 
Join Date: Aug 2015
Posts: 34
Quote:
Originally Posted by TomV View Post
The patent chart Tim is using is out of date and inaccurate. Fraunhofer sold their patents to GE, which is why GE has HEVC patents. Canon licenses their patents through MPEG LA.
The chart is the one used in Leonardo's blog, though I think it's originally from streamingmedia.com. I've attached an updated version for future presentations.


Last edited by TD-Linux; 30th January 2019 at 01:45.
TD-Linux is offline   Reply With Quote
Old 30th January 2019, 02:14   #1402  |  Link
jonatans
Registered User
 
Join Date: Oct 2017
Posts: 56
I originally created the figure and presented it for the first time during Streaming Tech Sweden 2017. There have been some changes since then.

Here is an updated figure:



Please note that the figure is only based on public information available from ISO/IEC/ITU and from the patent pools. Please also note that not all of the MPEG LA patent holders are shown in the figure.
__________________
Jonatan Samuelsson
Co-founder and CEO at Divideon

www.divideon.com | xvc.io
jonatans is offline   Reply With Quote
Old 30th January 2019, 02:32   #1403  |  Link
hajj_3
Registered User
 
Join Date: Mar 2004
Posts: 1,014
Quote:
Originally Posted by jonatans View Post
I originally created the figure and presented it for the first time during Streaming Tech Sweden 2017. There have been some changes since then.

Here is an updated figure:



Please note that the figure is only based on public information available from ISO/IEC/ITU and from the patent pools. Please also note that not all of the MPEG LA patent holders are shown in the figure.
I think i read that Franhaufer sold their HEVC patents to General Electric, if true you should remove Franhaufer from your diagram.
hajj_3 is offline   Reply With Quote
Old 30th January 2019, 02:51   #1404  |  Link
TomV
VP Strategy, Beamr
 
Join Date: Jan 2018
Location: Palo Alto, CA
Posts: 50
Quote:
Originally Posted by TD-Linux View Post
The chart is the one used in Leonardo's blog, though I think it's originally from streamingmedia.com. I've attached an updated version for future presentations.

It's originally from Jonatan Samuelsson, a.k.a. jonatans

Even with the updates, the main problem with this chart is that it's a bit misleading.. for 2 reasons. First, I think most people in the video industry assume that AVC patent licensing is and was perfectly clean and simple... that all necessary standard-essential patents were licenseable in the MPEG LA patent pool. That's not true. Nokia, Qualcomm, Broadcomm, Blackberry, Texas Instruments, MIT all hold standard-essential AVC patents outside the MPEG LA pool (although Qualcomm messed up and a judge ruled they can't assert them for AVC). Multiple legal battles have been fought over AVC patents, including some pretty big cases... Microsoft v Motorola, and Apple v Nokia. Today, everyone can agree that the patent licensing situation for AVC is much better than it is for HEVC. But it didn't start out that way, and it took some time for the situation to settle. Also, there are quite a few more patent holders in some of these HEVC pools than shown in this chart.

In his talk, Tim mentioned that patents are issued that may not be valid, and then said "and you could go around and try to invalidate them all, but they're really expensive to do that, and there's a lot of them". Well, if you're a multi-billion dollar company (Apple, Samsung, Google, Amazon, etc.), you have a lot of lawyers, and that's what they're paid to do. If you're being asked to pay tens or hundreds of millions of dollars a year in patent license fees, you have all the motivation in the world to spend whatever it takes on legal fees to right-size the problem. When multiple multi-billion dollar companies have this issue, collectively there is a lot of motivation. It turns out that when challenged in court, most patents don't hold up. They can be invalidated for many reasons... prior art, unpatentable claims, obviousness, the invention was anticipated, etc. This type of effort is being undertaken by Unified Patents (as a service to many large tech companies), and there is a relatively new law called the America Invents Act that provides a faster, less expensive way to get rid of bad patents, called an Inter Partes Review (IPR). Unified already filed an IPR against Velos Media, and you can expect more such filings under their Video Codec domain. But you don't even have to invalidate patents in order not to pay a fortune.

Again, keep in mind that no one is asking for patent license fees for content distribution (streaming, etc.), except for UHD-Blu-ray disc (a small per-disc fee to HEVC advance). Only hardware device manufacturers need to license HEVC patents, and they are dealing with that issue and they continue to support HEVC in every device they make that supports video. For video services, HEVC is free. Now that the majority of active end-user devices support HEVC, it makes a lot of financial sense for video services to make their VOD catalog, or the majority of their live channels available in both AVC and HEVC (not just 4K and HDR content... all content). The bandwidth savings and customer experience improvement far outweigh the additional cost of encoding and CDN storage.
TomV is offline   Reply With Quote
Old 30th January 2019, 02:58   #1405  |  Link
TomV
VP Strategy, Beamr
 
Join Date: Jan 2018
Location: Palo Alto, CA
Posts: 50
Quote:
Originally Posted by hajj_3 View Post
I think i read that Franhaufer sold their HEVC patents to General Electric, if true you should remove Franhaufer from your diagram.
That's true. I don't think it was ever announced, but I can assure you that I got confirmation from a very reliable source.
TomV is offline   Reply With Quote
Old 30th January 2019, 03:02   #1406  |  Link
TomV
VP Strategy, Beamr
 
Join Date: Jan 2018
Location: Palo Alto, CA
Posts: 50
Quote:
Originally Posted by jonatans View Post
I originally created the figure and presented it for the first time during Streaming Tech Sweden 2017. There have been some changes since then.

Here is an updated figure:



Please note that the figure is only based on public information available from ISO/IEC/ITU and from the patent pools. Please also note that not all of the MPEG LA patent holders are shown in the figure.
Hey... we were both responding at the same time (cross posting). Thanks for posting an update Jonatan. Your efforts on this, and in the MC-IF are really appreciated.
TomV is offline   Reply With Quote
Old 30th January 2019, 04:39   #1407  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 739
Interesting that some of the champions supporting or helping AOM are in the problematic(?) group of unpooled HEVC licensors... you would say these companies support AV1 because they hated that.
mandarinka is offline   Reply With Quote
Old 30th January 2019, 05:55   #1408  |  Link
kuchikirukia
Registered User
 
Join Date: Oct 2014
Posts: 466
Quote:
Originally Posted by TD-Linux View Post
Here's a link to AWCY as shown in Tim's presentation:

https://beta.arewecompressedyet.com/...y-525f981376bd

tl;dr it is better than x264 at every bitrate, but still worse that libvpx VP9. It is also currently about 10x slower than x264, which is blazing fast compared to libaom but still has a lot of room for improvement.
Taking x264 PSNR and SSIM values without setting --tune PSNR and SSIM is fail.
kuchikirukia is offline   Reply With Quote
Old 30th January 2019, 10:35   #1409  |  Link
nevcairiel
Registered Developer
 
Join Date: Mar 2010
Location: Hamburg/Germany
Posts: 10,072
Quote:
Originally Posted by kuchikirukia View Post
Taking x264 PSNR and SSIM values without setting --tune PSNR and SSIM is fail.
You get one encode to compare, do you really want that to be one tuned for PSNR? Because that would be the real fail.
One encode, several metrics. Not re-encoding targeted for metrics.
__________________
LAV Filters - open source ffmpeg based media splitter and decoders
nevcairiel is offline   Reply With Quote
Old 30th January 2019, 10:42   #1410  |  Link
TD-Linux
Registered User
 
Join Date: Aug 2015
Posts: 34
Quote:
Originally Posted by kuchikirukia View Post
Taking x264 PSNR and SSIM values without setting --tune PSNR and SSIM is fail.
There are more metrics than those at the link, but fair. I compared against x264 --tune PSNR as well and it still beats x264 at PSNR:

https://beta.arewecompressedyet.com/...y-525f981376bd
TD-Linux is offline   Reply With Quote
Old 30th January 2019, 12:29   #1411  |  Link
MoSal
Registered User
 
Join Date: Jun 2013
Posts: 89
Quote:
Originally Posted by benwaggoner View Post
In particular I worry that VMAF is insufficiently sensitive to temporal shifts in video quality. A VMAF of 70, 65, 60, 60, 65, 60, 60 might come out as a nice "VMAF=65.3" but be a annoying to watch. Frame strobing was a weakness of libvpx.
That's not really a problem. Per-frame data is available (wrote this mostly in a couple of hours). It's even available for multiple metrics, which is nice.

The real problem is that VMAF is not that good. It, for example, spectacularly fails with samples that greatly benefit from AQ (yes, I know you already hinted at this).
__________________
saldl: a command-line downloader optimized for speed and early preview.
MoSal is offline   Reply With Quote
Old 30th January 2019, 13:57   #1412  |  Link
jonatans
Registered User
 
Join Date: Oct 2017
Posts: 56
Quote:
Originally Posted by TomV View Post
Thanks for posting an update Jonatan. Your efforts on this, and in the MC-IF are really appreciated.
Thank you Tom. And thanks for providing additional context to this interesting and complicated matter.

Quote:
Originally Posted by hajj_3 View Post
I think i read that Franhaufer sold their HEVC patents to General Electric, if true you should remove Franhaufer from your diagram.
This is correct. But my understanding is that Fraunhofer did not sell all their HEVC patents. They are listed as licensor in HEVC Advance. In the latest patent list from HEVC Advance there are two Fraunhofer patents listed: https://www.hevcadvance.com/pdfnew/H...nuary-2019.pdf
__________________
Jonatan Samuelsson
Co-founder and CEO at Divideon

www.divideon.com | xvc.io
jonatans is offline   Reply With Quote
Old 30th January 2019, 17:56   #1413  |  Link
Beelzebubu
Registered User
 
Join Date: Feb 2003
Location: New York, NY (USA)
Posts: 82
Quote:
Originally Posted by benwaggoner View Post
In particular I worry that VMAF is insufficiently sensitive to temporal shifts in video quality. A VMAF of 70, 65, 60, 60, 65, 60, 60 might come out as a nice "VMAF=65.3" but be a annoying to watch. Frame strobing was a weakness of libvpx.
First and foremost: yes! It's great to see some technical & independent thinking of how good VMAF really is.

Netflix uses "hVMAF" as official notation in their charts. "h" means "harmonic", which means it uses harmonic means, which bias towards the least favourable. So in your example, the harmonic mean would be 62.66, whereas the average would be 62.86. Neither of these is 65.3. So I'm personally not as concerned about the averaging mechanism aspect of your concern. (In the CLI, use --pool harmonic_mean or something similar, depending on which exact tool you use.) On the other hand, I don't believe that VMAF uses temporal consistency in the reconstruction (the "motion" component is calculated from the source), so that particular concern ("frame throbbing" - i.e. keyframe pulsing or grain/textured-background tearing) I agree with.

[edit] Actually, I have to hedge a little here, since I'm not 100% sure VIF (another VMAF component) has a temporal component to it. I don't think it does but I'm not 100% sure. [/edit]

Since we're on the subject, here's some more of my personal concerns about VMAF:
* it's luma-only;
* AQ (x264/5) or SAO (x265) appear to have a negative impact on vmaf score, which is inconsistent with the reported visual results. I do have more detailed thoughts on this but let's leave that for some other time;
* the actual MOS/VMAF correlation depends very strongly on the viewing environment and therefore on the used model file, but most poeple simply use the default model without knowing what viewing environment it represents.

Just to be clear, I'm not trying to talk badly about VMAF, I think it's a great tool, it's better than the alternatives and it's fantastic that they opensourced the library as well as the models so that we can learn and understand how it works and constructively critique it. Hopefully, over time, that will make it even better, which should be the ultimate goal.

Separately, I also do agree with you that in the end, we should probably make a distinction between codecs optimized using VMAF vs. those that did not. This isn't an excuse to suck at writing encoders or to not use VMAF when writing encoders, but at the end of the day, we have to acknowledge that as in any metric, we're assuming a perfect correlation between our metric-of-the-day and the visual experience (or MOS score). That correlation will in practice always be imperfect, and therefore tuning towards/using that metric needs to be done with care and with visual confirmation (otherwise queue up the incoming VMAF artifacts - I wonder what they will look like?).

Last edited by Beelzebubu; 30th January 2019 at 22:11.
Beelzebubu is offline   Reply With Quote
Old 30th January 2019, 21:57   #1414  |  Link
MoSal
Registered User
 
Join Date: Jun 2013
Posts: 89
@Beelzebubu

What's really funny, Netflix will not be using VMAF on their published AOM content as is. Why? Because of film grain synthesis
__________________
saldl: a command-line downloader optimized for speed and early preview.
MoSal is offline   Reply With Quote
Old 31st January 2019, 06:00   #1415  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,596
Quote:
Originally Posted by MoSal View Post
@Beelzebubu

What's really funny, Netflix will not be using VMAF on their published AOM content as is. Why? Because of film grain synthesis
Well, if VMAF is used on the reconstructed video, it should be as good as VMAF is at dealing with film grain.

If VMAF is bad at dealing with film grain, they need to address that.

The nice thing about VMAF is that it's really a machine learning framework. They can keep on adding new clips and kinds of encodings and training it to rate those. The big expenses is getting the subjective ratings to use as ground-truth data. But VMAF itself can always be as good as the ground truth data from subjective testing.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 31st January 2019, 06:38   #1416  |  Link
TD-Linux
Registered User
 
Join Date: Aug 2015
Posts: 34
Quote:
Originally Posted by benwaggoner View Post
The nice thing about VMAF is that it's really a machine learning framework. They can keep on adding new clips and kinds of encodings and training it to rate those. The big expenses is getting the subjective ratings to use as ground-truth data. But VMAF itself can always be as good as the ground truth data from subjective testing.
The inputs to VMAF itself are the outputs of a bunch of simpler metrics. In that way, the machine-learned part is sort of a "meta-metric". That also means that if the input metrics all respond poorly to film grain, no amount of machine learning is going to be able to make sense of it. I think more work on the input metrics will be needed before VMAF can be used to make film grain decisions.

I don't know what Netflix currently does, but if I were them I would filter the grain from the video, run the VMAF-targeting dynamic optimizer to produce the rate controlled stream, and then add the noise parameters back as a final step.
TD-Linux is offline   Reply With Quote
Old 31st January 2019, 08:55   #1417  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,383
Quote:
Originally Posted by hajj_3 View Post
Franhaufer
Fraunhofer

Frau = woman
Hof = yard
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 31st January 2019, 19:27   #1418  |  Link
TomV
VP Strategy, Beamr
 
Join Date: Jan 2018
Location: Palo Alto, CA
Posts: 50
HEVC Advance standard essential patent owned by GE challenged as likely invalid
TomV is offline   Reply With Quote
Old 31st January 2019, 20:33   #1419  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,596
Quote:
Originally Posted by TD-Linux View Post
The inputs to VMAF itself are the outputs of a bunch of simpler metrics. In that way, the machine-learned part is sort of a "meta-metric". That also means that if the input metrics all respond poorly to film grain, no amount of machine learning is going to be able to make sense of it. I think more work on the input metrics will be needed before VMAF can be used to make film grain decisions.

I don't know what Netflix currently does, but if I were them I would filter the grain from the video, run the VMAF-targeting dynamic optimizer to produce the rate controlled stream, and then add the noise parameters back as a final step.
Good point on the underlying metrics. In particular I think the temporal metric was quite weak. They changed it for the most recent VMAF, but I'm not confident it'll catch all common kinds of visible temporal distortions. Two frames can look equally "good" but switching between them can be terribly jarring. Open GOP and RADL exist in large part to smooth inter-GOP transitions. And that still requires some cleverness around GOP boundaries to do well.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 1st February 2019, 20:13   #1420  |  Link
Mr_Khyron
Member
 
Mr_Khyron's Avatar
 
Join Date: Nov 2002
Posts: 156
SVT-AV1 encoder!

https://github.com/OpenVisualCloud/SVT-AV1
Quote:
Welcome to the GitHub repo for the SVT-AV1 encoder! To see a list of feature request and view what is planned for the SVT-AV1 encoder, visit our Trello page: http://bit.ly/SVT-AV1 Help us grow the community by subscribing to our SVT-AV1 mailing list

Quote:
Hardware

The SVT-AV1 Encoder library supports the x86 architecture

CPU Requirements

In order to achieve the performance targeted by the SVT-AV1 Encoder, the specific CPU model listed above would need to be used when running the encoder. Otherwise, the encoder runs on any 5th Generation Intel® Core™ processor, (Intel® Xeon® CPUs, E5-v4 or newer).

RAM Requirements

In order to run the highest resolution supported by the SVT-AV1 Encoder, at least 48GB of RAM is required to run a 4k 10bit stream multi-threading on a 112 logical core system. The SVT-AV1 Encoder application will display an error if the system does not have enough RAM to support this. The following table shows the minimum amount of RAM required for some standard resolutions of 10bit video per stream:
Resolution Minimum Footprint (GB)
4k 48gb
1080p 16gb
720p 8gb
480p 4gb
Mr_Khyron is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:35.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.