More than half of the H.264 patents have expired

[Blue_MiSfit]

A service I worked on called Movies Anywhere uses HEVC wherever possible. This is a free service dealing with premium Hollywood content. It was free because it was tied to the digital rights you get when you buy a title on disc or at a streaming service.

We had no HEVC licensing costs, and were comfortable taking the risk. We delivered, 4K, with HDR, Dolby Vision, Atmos, etc.

[benwaggoner]

Originally Posted by kurkosdr

Copying and pasting from above:
- Facebook
- Twitch
- Various X-rated sites
- Microsoft Stream
- IBM Video
- various YouTube alternatives (Rumble etc)

All of them H.264-only, except Twitch which uses H.264 and VP9. None of them uses HEVC.

Facebook also has big Feels about Patents. They've really backed away from their AV1 goals, however. I can't speak about Twitch in detail as they are a partner company. I don't have much experience with the other providers/markets. It doesn't sound like content sees would apply in those cases, and companies like Microsoft and IBM have extensive patent cross-licensing portfolios which should reduce or eliminate encoder/decdoer fees.

Real world, it gets down to the potential benefits of reduced bandwidth costs versus the extra storage costs of having multiple codecs and the complexity of encoding and managing a multi codec service. Plus if a service is live and not being hugely profitable, the instinct is to leave it as it is rather than make a big capital investment in long-term operational cost savings. And HEVC in browsers is relatively new and kinda came out of the blue. Services could be working on multi-codec but not have launched yet.

Also, a lot of services you've mentioned aren't really monetizing quality, and so reducing bitrate, even with a quality hit, is a much easier knob to control storage and bandwidth costs.

And if a service didn't archive the source, but just kept distribution encodes, reencoding a mediocre H.264 into HEVC doesn't offer nearly the same benefits as encoding fresh from the source. This has been pretty common to save on storage costs.

Real-time HEVC encoding takes a few times more compute/pixel, and would require hardware upgrades if a service is using low-power fixed function H.264 encoders. We're not seeing much live AV1 for similar reasons (although it is several times slower yet than HEVC). The extra capital and/or operational costs per channel may not pencil out for low-viewership streams. And, at least, would require a new budget cycle.

Again, if HEVC made financial sense for free-to-view web video, one of those services would be using it.

There are lots of ways it may not have made sense for a given free-to-view service, as above. Sometimes it doesn't make sense as the leadership of a service may not be even thinking in these directions. It's a complex thing to model! And capital investments have a much higher bar these days with VC funding trying up and interest rates so high.

[kurkosdr]

Originally Posted by benwaggoner

Facebook also has big Feels about Patents.

Please try to get this into your skull: it's not about "feels", it's about financial realities. I personally lol'ed hard at the graph on the previous page: 2 patent pools, 8 unaffiliated entities. Nice. Unless you have a reliable source from all 10 entities pledging there won't be "content fees", jumping into HEVC carries the huge risk that one or those 10 entities will come to collect content fees years after you've spent time and electricity encoding your content to HEVC. Now, if you are a pay-to-view service, those fees can be rolled into the cost the customer pays to view the video, but free-to-view services serving tons of video to anyone who asks and making tight margins per video from ads don't have that option.

Originally Posted by benwaggoner

and companies like Microsoft and IBM have extensive patent cross-licensing portfolios which should reduce or eliminate encoder/decdoer fees.

Irrelevant, unless they have patent cross-licensing portfolios with every patent holder in HEVC. The patent holders they do not have patent cross-licensing portfolios with can always come to collect content fees.

Originally Posted by benwaggoner

Real world, it gets down to the potential benefits of reduced bandwidth costs versus the extra storage costs of having multiple codecs and the complexity of encoding and managing a multi codec service. Plus if a service is live and not being hugely profitable, the instinct is to leave it as it is rather than make a big capital investment in long-term operational cost savings. And HEVC in browsers is relatively new and kinda came out of the blue. Services could be working on multi-codec but not have launched yet.

Also, a lot of services you've mentioned aren't really monetizing quality, and so reducing bitrate, even with a quality hit, is a much easier knob to control storage and bandwidth costs.

Reality is, any free-to-view service choosing to use some more modern format alongside H.264 has gone to VP9. So there is a desire to use formats more modern than H.264 by some of those services, just not HEVC. If there was some kind of desire to use HEVC by some free-to-view service, we would have heard it by now. Pay-to-view services have already done it.

[kurkosdr]

Originally Posted by Blue_MiSfit

A service I worked on called Movies Anywhere uses HEVC wherever possible. This is a free service dealing with premium Hollywood content. It was free because it was tied to the digital rights you get when you buy a title on disc or at a streaming service.

We had no HEVC licensing costs, and were comfortable taking the risk. We delivered, 4K, with HDR, Dolby Vision, Atmos, etc.

Then it's not a free service, it's part of the package the customer buys when buying the disc of streaming service in question, which means it's actually a bundled service. This means any potential content fee (I guess that's what you mean when you say "risk") can be bundled into the purchase price (the margins the Hollywood studio makes per disc or subscriber likely allow it).

[benwaggoner]

Originally Posted by kurkosdr

Please try to get this into your skull: it's not about "feels", it's about financial realities. I personally lol'ed hard at the graph on the previous page: 2 patent pools, 8 unaffiliated entities. Nice. Unless you have a reliable source from all 10 entities pledging there won't be "content fees", jumping into HEVC carries the huge risk that one or those 10 entities will come to collect content fees years after you've spent time and electricity encoding your content to HEVC. Now, if you are a pay-to-view service, those fees can be rolled into the cost the customer pays to view the video, but free-to-view services serving tons of video to anyone who asks and making tight margins per video from ads don't have that option.

You may feel there is a problem in theory, but lots of companies have adopted HEVC without any problems in practice. I concur that getting a clear straight answer from all participants is way more complicated than it should ever have been. But unless you have some examples of someone actually having to pay content licenses for free-to-play content, this is largely conjecture.

Irrelevant, unless they have patent cross-licensing portfolios with every patent holder in HEVC. The patent holders they do not have patent cross-licensing portfolios with can always come to collect content fees.

In practice, the patent cross-licensing can have a big effect on these concerns, as it allows a whole lot of patents that could apply be ruled out. IANAL, but this seems to be the general case. There's no patent issue without a patent holder wiling to sue.

Reality is, any free-to-view service choosing to use some more modern format alongside H.264 has gone to VP9. So there is a desire to use formats more modern than H.264 by some of those services, just not HEVC. If there was some kind of desire to use HEVC by some free-to-view service, we would have heard it by now. Pay-to-view services have already done it.

That's like saying premium content must have a big reason not to adopt VVC as no one has yet. Market timing is a big factor here. Bear in mind that VP9 has been broadly playable in web browsers for seven years now, while HEVC has been broadly supported for just shy of a year.

VP9 also got a lot of very optimistic promises made about bitrate savings over H.264 that never really paid off, largely due to x264, but also because of some poor design decisions that forced a lot of serialization for encoding and decoding. The VP9 toolchain was fine for YouTube splitting up lots of single-threaded chunks across lots of instances, but not for a lot of core x264 scenarios.

[Spyros]

Small update, a newer patent list is available (August 1, 2023).

I didn't notice earlier, because the website changed (from mpegla.com to via-la.com).

19 pages with unexpired patents, 37 pages with expired ones.

[benwaggoner]

There's a few pages worth that are expiring this or next month as well. Dolby really dominates the remaining ones.

[kurkosdr]

Originally Posted by Spyros

Small update, a newer patent list is available (August 1, 2023).

I didn't notice earlier, because the website changed (from mpegla.com to via-la.com).

19 pages with unexpired patents, 37 pages with expired ones.

In the US, the last patent for the first version of the H.264 standard will expire on 29 nov 2027:
https://scratchpad.fandom.com/wiki/M...#H.264_patents
So it's not worth holding your breath (when it comes to US patents).

I am interested how many EU patents remain active after 17 August 2004 (the date the first version of the H.264 standard was published), since the EU doesn't have a habit of granting multi-year patent extensions willy-nilly like the US does. For example, MPEG4 ASP is still patented in the US (and in Brazil, due to an one-time extension given to some patents in that country) but is royalty-free everywhere else.

MPEG LA updates their patents lists every 3 months btw.

[Avclover]

In the US, the last patent for the first version of the H.264 standard will expire on 29 nov 2027

It seems the current VIA LA patent list features at least one early US patent that expires only at the end of 2030:
US9356620B2.

It is also worth noting that the ubiquitous high profile was introduced only after version 1 of AVC.

[kurkosdr]

Originally Posted by Avclover

It seems the current VIA LA patent list features at least one early US patent that expires only at the end of 2030:
US9356620B2.

It is also worth noting that the ubiquitous high profile was introduced only after version 1 of AVC.

VIA LA's patent list includes extensions such as SVC and MVC which are technically part of the AVC standard but are not essential to implement (both SVC and MVC are backwards-compatible extensions).
https://www.via-la.com/licensing-2/avc-h-264/

You are correct about the High Profile though: The High profile was added on 1st March 2005, so that's the patent priority date to look for, I guess.

[benwaggoner]

Originally Posted by kurkosdr

VIA LA's patent list includes extensions such as SVC and MVC which are technically part of the AVC standard but are not essential to implement (both SVC and MVC are backwards-compatible extensions).
https://www.via-la.com/licensing-2/avc-h-264/

You are correct about the High Profile though: The High profile was added on 1st March 2005, so that's the patent priority date to look for, I guess.

Since High mainly added 8x8 blocks, which were the pre-H.264 standard, I wonder if there are all that many essential patents for it. How to choose between and single 4x4 versus 8x8, perhaps?

[rwill]

Originally Posted by kurkosdr

VIA LA's patent list includes extensions such as SVC and MVC which are technically part of the AVC standard but are not essential to implement (both SVC and MVC are backwards-compatible extensions).

You write that MVC is backwards compatible. I have a question. If I write 128 as Profile IDC into a SPS to signal MVC Stereo High Profile, how is this backwards compatible to Profile IDC = 100 which is High Profile? If a Decoder knows nothing about Profile IDC = 128 or 132 and the corresponding change in syntax, would it not flat our fail to decode?

Originally Posted by benwaggoner

Since High mainly added 8x8 blocks, which were the pre-H.264 standard, I wonder if there are all that many essential patents for it. How to choose between and single 4x4 versus 8x8, perhaps?

That would be news to me that the very specific 8x8 Transform of AVC was a standard pre-H.264.

[Avclover]

VIA LA's patent list includes extensions such as SVC and MVC which are technically part of the AVC standard but are not essential to implement (both SVC and MVC are backwards-compatible extensions).

While that's true, the specific patent originated in 2002, got a ridiculous term extension and is not related to SVC or MVC.

Since High mainly added 8x8 blocks, which were the pre-H.264 standard, I wonder if there are all that many essential patents for it. How to choose between and single 4x4 versus 8x8, perhaps?

High profile also introduced quantization scaling matrices and separate C_B and C_R QP control to the specification.

[benwaggoner]

Originally Posted by Avclover

High profile also introduced quantization scaling matrices and separate C_B and C_R QP control to the specification.

Oh, right!

Not all encoders and bitstreams use those tools, but some certainly do, or at least can.

[Avclover]

I guess the amount of decoders with support is what counts from a licensing perspective.

[kurkosdr]

Originally Posted by rwill

You write that MVC is backwards compatible. I have a question. If I write 128 as Profile IDC into a SPS to signal MVC Stereo High Profile, how is this backwards compatible to Profile IDC = 100 which is High Profile? If a Decoder knows nothing about Profile IDC = 128 or 132 and the corresponding change in syntax, would it not flat our fail to decode?

There is a simple answer to that: You won't do that, you will instead do what Blu-ray 3D wants, because it's the only use of MVC that matters (and fortunately it is backwards compatible with High profile).

And even if you don't care about Blu-ray 3D playback (say you are making MVC in MP4), then you will do whatever it takes to to not break High Profile anyway. I am sure there are multiple ways to produce a stream that's technically compliant with the latest version of the H.264/AVC standard but breaks the first High profile decoders, but don't do that if you want to get the benefits of the huge H.264/AVC installed base out there. That's true even if we are not talking about hardware players but software players. No real-world H.264/AVC stream out there breaks High Profile, not ones anyone would care about anyway.

So, as a person writing a software player, implementing the first version of the H.264/AVC standard that includes the High profile achieves compatibility with practically all real-world H.264/AVC streams out there.

[benwaggoner]

Originally Posted by kurkosdr

There is a simple answer to that: You won't do that, you will do what Blu-ray 3D wants, because that's the only use of MVC that matters (and is backwards compatible with High profile).

Except 3D Blu-ray is H.264, and Apple specifies using MV-HEVC. There will need to be some work to get it working for HEVC, >1080p, and for HDR. It's all pretty obvious how to do it, and the HC reference encoder includes MV-HEVC support.

So, implementing the first version of High Profile achieves compatibility with practically all real-world H.264/AVC streams.

Yeah. Thank goodness the days of "subset of Main and Baseline + B-frames" defacto subsets are behind us.

[kurkosdr]

Originally Posted by benwaggoner

Except 3D Blu-ray is H.264, and Apple specifies using MV-HEVC. There will need to be some work to get it working for HEVC, >1080p, and for HDR. It's all pretty obvious how to do it, and the HC reference encoder includes MV-HEVC support.

MV-HEVC has nothing to do with H.264/AVC, so it doesn't belong in this thread.

[oibaf]

Originally Posted by Spyros

Small update, a newer patent list is available (August 1, 2023).

I didn't notice earlier, because the website changed (from mpegla.com to via-la.com).

19 pages with unexpired patents, 37 pages with expired ones.

BTW, in that list unexpired patents are:

• US: 540
• CN: 132
• JP: 129
• KR: 97
• GB: 95
• DE: 92
• FR: 91
• IT: 68
• NL: 67
• ES: 65
• BR: 57
• CA: 31

There is also an interesting analysis here: https://scratchpad.fandom.com/wiki/M...#H.264_patents : sometimes they differ in their expiration dates.

[kurkosdr]

It's been 3 months since August 1, 2023, so MPEG LA pushed a new list:
https://www.via-la.com/wp-content/up...tachment-1.pdf

----

Originally Posted by oibaf

BTW, in that list unexpired patents are:

• US: 540
• CN: 132
• JP: 129
• KR: 97
• GB: 95
• DE: 92
• FR: 91
• IT: 68
• NL: 67
• ES: 65
• BR: 57
• CA: 31

There is also an interesting analysis here: https://scratchpad.fandom.com/wiki/M...#H.264_patents : sometimes they differ in their expiration dates.

I may try and make a csv list or spreadsheet of the MPEG LA document sometime so I can feed it to Google Patents, but if you already have one, can you post it? Basically trying to come up with a "isn't royalty-free before" date for every country (only taking into account patents with a priority date from before March 1, 2005)