More than half of the H.264 patents have expired

[Spyros]

Just something I noticed on the official MPEG LA website.

The previous patent list (February 21, 2023 (https://web.archive.org/web/20230426143723/https://www.mpegla.com/wp-content/uploads/avc-att1.pdf)) listed 32 pages with unexpired patents and 23 pages with expired ones.

The latest patent list (May 1, 2023 (https://web.archive.org/web/20230503205906/https://www.mpegla.com/wp-content/uploads/avc-att1.pdf)) lists 24 pages with unexpired patents and 32 pages with expired ones.

Yes, I know, quick-and-dirty calculation. And that percentage doesn't change anything, all the patents (or almost all of them) must expire in order to use the codec freely. But that day is getting closer. :cool:

UPDATES:

August 1, 2023 (https://web.archive.org/web/20230907021043/https://www.via-la.com/wp-content/uploads/Final-August-1-2023-AVC-Attachment-1.pdf): 19 pages with unexpired patents, 37 pages with expired ones (post #46 (https://forum.doom9.org/showthread.php?p=1991662#post1991662))
November 1, 2023 (https://web.archive.org/web/20231124114858/https://www.via-la.com/wp-content/uploads/Final-November-1-2023-AVC-Attachment-1.pdf): 17 pages with unexpired patents, 40 pages with expired ones (post #60 (https://forum.doom9.org/showthread.php?p=1993227#post1993227))
February 1, 2024 (https://web.archive.org/web/20240503101957/https://www.via-la.com/wp-content/uploads/Final-February-1-2024-AVC-Attachment-1.pdf): 17 pages with unexpired patents, 41 pages with expired ones (post #77 (https://forum.doom9.org/showthread.php?p=1996897#post1996897))
May 1, 2024 (https://web.archive.org/web/20240503094239/https://www.via-la.com/wp-content/uploads/Final-May-1-2024-AVC-Attachment-1.pdf): 14 pages with unexpired patents, 45 pages with expired ones (post #99 (https://forum.doom9.org/showthread.php?p=2001343#post2001343))

[kurkosdr]

Keep in mind that MPEG LA adds the patents essential to implementing the backwards-compatible extensions SVC (Scalable Video Coding) and MVC (Multiview Video Coding) to the list of patents, so the essential patents required to decode AVC video ignoring those extensions will expire sooner.

The best I have found is this list, but it's only for US patents:
https://www.osnews.com/story/24954/us-patent-expiration-for-mp3-mpeg-2-h264/

[FranceBB]

Nice. Given how widespread H.264 already is, it might make things even easier.
Perhaps H.264 is gonna stay with us for a little longer ehehehehehe

[benwaggoner]

Nice. Given how widespread H.264 already is, it might make things even easier.
Perhaps H.264 is gonna stay with us for a little longer ehehehehehe
H.264 is already a particularly long-lived codec, and will be around for some years to come.
I can imagine a streaming service viably dropping H.264 entirely in maybe 2-3 years. >90% of player devices support HEVC today, so it's really older long-tail things that don't. And H.264 will remain reliably compatible far into the future.

H.264 was really a revolution and revelation, the first to truly break out of the H.261 model with lots of fundamentally new approaches to encoding. Multiple reference frames! 4x4 intra blocks! In-loop deblocking! Arithmetic entropy encoding!

Its timing with the birth of the ripping/warez/torrent community was also fortuitous, as that lead to x264. Past professional encoders largely refined reference encoders, focused on PSNR, and started with a similar basic set of test clips. x265 had a whole different target audience and developer goals as an open source project. There were thousands of people across the world throwing all kinds of wildly variable content at it, tuning and testing and sharing results like crazy. Subjective quality was all that matters, with PSNR and SSIM afterthoughts at best. Animation was a first-class content category (a BIG blind spot in classic codec development). Crazy ideas like MBTree were tried. Bad ones got discarded quickly, and the good ones refined.

And real-world performance mattered; groups were competing to get the best quality at the smallest file size published before everyone else. Optimization and multithreading were pushed farther than anyone had before.

x264 kept raising the bar for H.264 encoders, and encoders in general. Anyone who wanted to sell a H.264 encoder had to compete with the free x264, so lots of bitstreams and code were analyzed, and techniques ported over or at least caused inspiration.

All of that meant that H.264 kept getting better, longer, than almost any other codec (MPEG-2 as the arguable exception). H.264 wound up being more efficient for important use cases than the people who made the standard ever predicted.

HEVC got a big boost from x264 as well, as MultiCoreWare licensed the source code and started x265 as a mash-up of the H.265 reference encoder and x264, and then went from there.

[FranceBB]

I can imagine a streaming service viably dropping H.264 entirely in maybe 2-3 years. >90% of player devices support HEVC today

I don't really think they will, unless they're gonna use something like AV1 as a backwards-compatibility option.
The issue being that H.265 is still struggling in browsers compared to H.264, so much so that unless the user has a GPU with its own hardware decoder, it won't be able to play it (and this is definitely not guaranteed).
Although lots of people watch those contents on either a smartphone (if they're on the go) or with their TV (if they're at home) where none of this is a problem, there are still some folks watching contents on a PC via the browser (especially if it's a laptop and they're, dunno, on a business trip or whatever). As much as it would be nice to have H.265 replace H.264 once and for all, I don't really see that happening. The only feasible replacement for H.264 would (probably) be AV1 as it has a totally free software decoding.

The "big boys" in terms of streaming services can innovate and afford to have multiple different encodes of the same content in different codecs to achieve the best quality and that's fine.
If we look at smaller, non global, country-specific services, however, they all use the same fatal combination: H.264 + AAC as the only thing that they're sure works everywhere.

- BBC iPlayer -> H.264 + AAC
- ITVX -> H.264 + AAC
- All 4 -> H.264 + AAC

The list goes on, but essentially they're all using the H.264 + AAC combo displaying SDR contents only when you watch them on a browser. For instance, even the BBC, which offers HDR HLG BT2020 contents via the iPlayer, doesn't really offer them via browser but rather only via the installed app on the devices they know they can support.

[Blue_MiSfit]

Google ships a software HEVC decoder in Chrome now (though without Widevine DRM support), so the browser argument against HEVC is winding down :)

Safari of course has full HEVC support. That leaves the small footprint of Firefox and others.

[StainlessS]

Safari of course has full HEVC support.
So far as I know, Safari version is same as IOS version (at least on iPad), and so HEVC support relies on IOS version,
but which version has the full support ?

EDIT:
Dont think my iPad 4 supports (IOS v10.3.3),
iPad mini 4 not sure, (recent updated to IOS v15.7.7 due to security bug:- https://forum.doom9.org/showthread.php?p=1989479#post1989479 )

[Ritsuka]

The A8 chip should support HEVC, A9 surely does.

[benwaggoner]

Google ships a software HEVC decoder in Chrome now (though without Widevine DRM support), so the browser argument against HEVC is winding down :)
Yeah, 18 months ago Chrome was the single biggest platform where HEVC couldn't be used. And it had AV1 support, so there was a better-than-H.264 alternative. Clearly that wasn't enough to justify broad adoption of AV1 for premium content that requires DRM. YouTube still remains the primary distributor of AV1 bitstreams.

Safari of course has full HEVC support. That leaves the small footprint of Firefox and others.
Edge is Chromium derived, and I believe supports HEVC correctly (again, finally) as well. So that's the top three browsers by usage. FireFox has dropped below 6% on Windows/Mac now.

[Blue_MiSfit]

Indeed, I just tested hardware accelerated HEVC in Edge with multi-key PlayReady. Yay!

Now... if only Google would get this hooked up on Chrome.

[FranceBB]

Google ships a software HEVC decoder in Chrome now (though without Widevine DRM support), so the browser argument against HEVC is winding down :)

Ah! I didn't know that!
Nice, very nice. About time.
So now it just needs DRM and the game is on. :)


By the way, I was trying to play with: https://reference.dashif.org/dash.js/nightly/samples/dash-if-reference-player/index.html
I used the following stream: https://dash.akamaized.net/dash264/TestCasesHEVC/1a/5/MultiRate.mpd
But all I could see was dark:

https://i.imgur.com/L8E1NT5.png


Google Chrome 116.0.5845.42 (Official Build) beta (64-bit)
Revision d6ccfdce8c8e575e0a33c92686a29707d0dd4e4f-refs/branch-heads/5845@{#571}

Are you guys sure it supports software decoding and it's not just hardware decoding using the GPU decoder?
Have you actually got it working with software decoding?

[kurkosdr]

H.264 was the only case of the patent pools pulling their heads out of their posteriors and deciding to not charge "content fees" for free-to-view web content. All the other ISO/ITU video formats have been failures on the web mainly because of the "content fee", with websites opting for literally anything else than the ISO/ITU formats. The only exception is some subscription services using H.265, and even then some subscription services have opted for AV1 instead.

So, even if the problem of the decoder fees was solved for H.265 (which is a much bigger problem than it was for H.264 due to the stacking of multiple patent pool royalties), so that browsers can support H.265 reliably without dependence on the particular hardware configuration they run on, H.265 also has the problem of the "content fee".

The only viable formats for the web are VP8, H.264, VP9, and AV1. VP9 and AV1 have a non-zero possibility of a successful patent assertion, so the only 100% safe choice is VP8 and H.264, and since H.264 is better than VP8 and also universally supported by hardware acceleration, it's gonna be with us for a long time. Also, any attempt to replace it is not worth the effort since it's going to be royalty-free before any attempt to replace it can succeed.

[Ritsuka]

Do you have a link to where those "content fees" are defined?

[excellentswordfight]

Do you have a link to where those "content fees" are defined?
AFIK it was removed in 2018 (except for physical media e.g. Bluray).

[Snap]

AFIK it was removed in 2018 (except for physical media e.g. Bluray).

Yeah. I saved the page/link of an article covering it at the time:

http://www.streamingmedia.com/Articles/News/Online-Video-News/HEVC-Advance-Cuts-Content-Fees-on-Streaming-123828.aspx

[kurkosdr]

You are aware that HEVC Advance is only one of the several patent pools that you need to license patens from to implement HEVC, right? If one of the patent pools charges a "content fee" for streaming, then HEVC has a content fee for streaming

[excellentswordfight]

You are aware that HEVC Advance is only one of the several patent pools that you need to license patens from to implement HEVC, right? If one of the patent pools charges a "content fee" for streaming, then HEVC has a content fee for streaming
Im fully aware, but non of the other holders do that, I work at a broadcast/streaming company and I havnt heard anyone talk about this fee for years, and I can find multiple industry sources stating that there are no fees;

from harmonic:

"we estimate that HEVC still has large growth potential, especially as streaming fees are waived from patent pools."

https://www.harmonicinc.com/insights/blog/hevc-market-perspectives-broadcast-and-streaming

From Tom Vaughan (ex VP for MulticoreWare)

"Again, keep in mind that no one is asking for patent license fees for content distribution (streaming, etc.), except for UHD-Blu-ray disc (a small per-disc fee to HEVC advance). Only hardware device manufacturers need to license HEVC patents, and they are dealing with that issue and they continue to support HEVC in every device they make that supports video. For video services, HEVC is free."

https://forum.doom9.org/showthread.php?p=1864184#post1864184

So what are your sources for this? Cause it seems like you know something we dont.

[Blue_MiSfit]

It's FUD, IMO.

Yes it's possible that someone may declare and start demanding per-content royalties.

Do I think that will happen? Personally, no.

[benwaggoner]

H.264 was the only case of the patent pools pulling their heads out of their posteriors and deciding to not charge "content fees" for free-to-view web content. All the other ISO/ITU video formats have been failures on the web mainly because of the "content fee", with websites opting for literally anything else than the ISO/ITU formats. The only exception is some subscription services using H.265, and even then some subscription services have opted for AV1 instead.
I'm not aware of any premium services are using AV1 for even 10% of their streams, or for UHD and HDR at all. I could be wrong.

Content fees haven't wound up being a problem in practice, but the ambiguity until things have clarified sure have (MPEG-4 pt 2 in particular).

So, even if the problem of the decoder fees was solved for H.265 (which is a much bigger problem than it was for H.264 due to the stacking of multiple patent pool royalties), so that browsers can support H.265 reliably without dependence on the particular hardware configuration they run on, H.265 also has the problem of the "content fee".
I think the decoder fee has been solved for H.265. It is pretty well universal in mobile, TV, and attached devices. Web browsers were really the last big gap. Certainly HDR content can be done in just HEVC.

The only viable formats for the web are VP8, H.264, VP9, and AV1. VP9 and AV1 have a non-zero possibility of a successful patent assertion, so the only 100% safe choice is VP8 and H.264, and since H.264 is better than VP8 and also universally supported by hardware acceleration, it's gonna be with us for a long time. Also, any attempt to replace it is not worth the effort since it's going to be royalty-free before any attempt to replace it can succeed.
"Viable" in what sense? UGC HEVC is now viable on the web with Safari + Chrome + Edge, with HW decoder support broadly available.

I concur that as of August 2021, if you can only encode to one codec, H.264 is the obvious choice. But for more complex efforts where multi-codec is viable, <20% of clients would get H.264 instead of something better.

[kurkosdr]

I'm not aware of any premium services are using AV1 for even 10% of their streams, or for UHD and HDR at all.

Netflix streams AV1 if the device supports it

So what are your sources for this? Cause it seems like you know something we dont.
I read somewhere that MPEG LA used to charge content fees, but that may be outdated. Anyway, unless there is a reliable source for every patent pool (that they have waived content fees), I don't believe it, there are just too many patent pools and entities getting their beak wet from H.265, and every one has to agree for H.265 to be free of content fees.

I think the decoder fee has been solved for H.265. It is pretty well universal in mobile, TV, and attached devices. Web browsers were really the last big gap. Certainly HDR content can be done in just HEVC.
It hasn't been solved for browsers, since the business model of the entire industry revolves around giving away the browser for $0. Just because a company like Google begrudingly pays the H.264 royalties for Chrome (and Cisco for Firefox) doesn't mean they will do it again for HEVC. Chrome "solves" the issue by using HEVC hardware acceleration if supported by the system (and not decoding HEVC if not), which means that as a service provider you will have to also encode in H.264 or VP8. Which is why if you can't encode and store multiple formats (on top of encoding and storing multiple resolutions that you have to do anyway), H.264 and VP8 are your only viable formats. But if you can encode in multiple-formats, then every format can be said to be viable, even something like VVC+Dolby Vision can be said to be viable if there is an H.264 or VP8 stream to fall back to.

[benwaggoner]

Netflix streams AV1 if the device supports it
For HDR as well? If so, I wasn't aware! Do you have any links?

I read somewhere that MPEG LA used to charge content fees, but that may be outdated. Anyway, unless there is a reliable source for every patent pool (that they have waived content fees), I don't believe it, there are just too many patent pools and entities getting their beak wet from H.265, and every one has to agree for H.265 to be free of content fees.
MPEG-LA has had some commercial content fees before, but they've been reasonable. The bigger friction has been from ambiguity before the patent pool(s) have finalized and published details. This was more complex for HEVC which had three pools, not all of which published any license details and required an attorney and NDA to get the specifics.

It hasn't been solved for browsers, since the business model of the entire industry revolves around giving away the browser for $0. Just because a company like Google begrudingly pays the H.264 royalties for Chrome (and Cisco for Firefox) doesn't mean they will do it again for HEVC. Chrome "solves" the issue by using HEVC hardware acceleration if supported by the system (and not decoding HEVC if not), which means that as a service provider you will have to also encode in H.264 or VP8. Which is why if you can't encode and store multiple formats (on top of encoding and storing multiple resolutions that you have to do anyway), H.264 and VP8 are your only viable formats. But if you can encode in multiple-formats, then every format can be said to be viable, even something like VVC+Dolby Vision can be said to be viable if there is an H.264 or VP8 stream to fall back to.
It has been "solved" in the sense that HEVC decode hardware is now ubiquitous in most markets, and a free browser can pass a HEVC bitstream to a system-level SW or HW decoder (just like with AV1). There will be some edge cases without a HEVC decoder, of course, so full ubiquity still requires H.264 as a fallback.

For premium content requiring hardware DRM, just HEVC is more viable because the systems without HEVC HW decoders rarely have sufficiently protected HW DRM. Same with HDR, as the 10-bit requirement rules out most H.264 decoders in systems without HEVC decoders.

I am using HEVC as an illustrative example here, as the same market dynamics apply to AV1, AV2, VVC, and on into the future. I doubt we'll see any major content platform adopt a single codec solution other than H.264 in the next decade.

[kurkosdr]

For HDR as well? If so, I wasn't aware! Do you have any links?

They are considering AV1 HDR in the future, but not doing it at the moment:
https://www.audioholics.com/hdtv-formats/netflix-av1
Don't ask what's taking them so long, YouTube is doing it with AV1 and VP9.


MPEG-LA has had some commercial content fees before, but they've been reasonable. The bigger friction has been from ambiguity before the patent pool(s) have finalized and published details. This was more complex for HEVC which had three pools, not all of which published any license details and required an attorney and NDA to get the specifics.

Quite honestly, this is why I hate HEVC with a passion when it comes to free-to-view content: Nobody out there has a straight "yes" or "no" answer to the thorny issue of content fees. Which is why I think it's best to avoid it for free-to-view content.

At this point it makes you wonder whether investing in HDR H.264 encoders is indeed the best option. Resolutions above 1080p are unafforable for free-to-view content anyway, and the bitrate impact of HDR will be marginal.

[rwill]

Free-to-view sounds nice. YouTube made $29.2 billion revenue last year though... maybe they could afford content fees if they had to without having to close shop?

[kurkosdr]

Free-to-view sounds nice. YouTube made $29.2 billion revenue last year though... maybe they could afford content fees if they had to without having to close shop?

1) Free-to-view doesn't mean YouTube.

2) How much did YouTube made in profit? Because that's the excess they could theoretically spend in content fees. And how much do the HEVC patent pools want in content fees per video?

There is a reason websites serving free-to-view content (which make a razor-thin margin per each video) avoid formats with content fees like the plague, going all the way back to IGN serving Quicktime videos and later WMV (but not MPEG-2 or MPEG 4 Part 2), which was before YouTube was even a thing btw. Even MPEG-1 didn't became a thing for free-to-view videos until the last patent expired.

[rwill]

1) Free-to-view doesn't mean YouTube.

2) How much did YouTube made in profit? Because that's the excess they could theoretically spend in content fees. And how much do the HEVC patent pools want in content fees per video?

There is a reason websites serving free-to-view content (which make a razor-thin margin per each video) avoid formats with content fees like the plague, going all the way back to IGN serving Quicktime videos and later WMV (but not MPEG-2 or MPEG 4 Part 2), which was before YouTube was even a thing btw. Even MPEG-1 didn't became a thing for free-to-view videos until the last patent expired.

Can you give examples of free-to-view websites other than YouTube? Maybe some with a different business model?

It appears Alphabet does not tell how much YouTube made them in profit. But if Youtube can give some cents per video view to the content creator I think they can also afford paying a little for a highly efficient format. They have no need to do that of course because of AV1 and VP9 - which are well suited for the quality the user has come to expect from YouTube.

I think you are mistaken about MPEG-1, MPEG-2 and MPEG-4 not being a thing on the web because of content fees. Its the formats videos were shared in when the videos were put up for download for offline viewing. I think websites looking to embed videos in the page, because of ad revenue, were using RealVideo, Quicktime, Flash or some Windows IE extension only because there was no other widely supported browser support. These used patented and/or proprietary codecs internally anyway. And there were no content fees back then, just a per unit license for hard- or software. The licensing business was also less fragmented.

[benwaggoner]

They are considering AV1 HDR in the future, but not doing it at the moment:
https://www.audioholics.com/hdtv-formats/netflix-av1
Don't ask what's taking them so long, YouTube is doing it with AV1 and VP9.
Netflix, as a premium content company, has a lot higher quality bar than YouTube. And while AV1 doesn't seem to have any fundamental deficit for HDR, encoding in PQ and Rec. 2020 requires a fair amount of retuning away from gamma and Rec. 709. For a simple example, there's the --hdr10-opt from x265.
AV1 also encodes more slowly than HEVC in general, and as HDR comes with UHD, those extra pixels can really slow down encoding. There's a big difference between taking 12 hours and 36 hours to encode a movie.

Quite honestly, this is why I hate HEVC with a passion when it comes to free-to-view content: Nobody out there has a straight "yes" or "no" answer to the thorny issue of content fees. Which is why I think it's best to avoid it for free-to-view content.
I believe all the stakeholders did clarify that free-to-view content doesn't have license fees as of a few years ago. Which was always true, but one of the licensing groups wouldn't tell anyone they wouldn't charge a content license expect under NDA. Which was infuriating.

At this point it makes you wonder whether investing in HDR H.264 encoders is indeed the best option. Resolutions above 1080p are unafforable for free-to-view content anyway, and the bitrate impact of HDR will be marginal.
HDR requires 10-bit, and there are a lot more 10-bit HEVC HW decoders out there than 10-bit H.264. And HDR is also most often UHD, where the compression efficiency improvements of HEVC are even bigger than at lower resolutions. Having a 32x32 max block size is really useful at higher resolutions versus a max of 8x8. H.264 UHD HDR would need to at least double bitrates, and with much less compatibility.

And, obviously, premium content HDR playback launched HEVC only over eight years now, and is still exclusively HEVC other than YouTube's user generated content.

[kurkosdr]

Can you give examples of free-to-view websites other than YouTube? Maybe some with a different business model?

X-rated sites are a big example, then there is Twitch, Microsoft Stream, IBM Video, plus the various YouTube alternatives (Rumble etc).

None of them uses H.265, it's H.264 and maybe VP9 and AV1.

It appears Alphabet does not tell how much YouTube made them in profit. But if Youtube can give some cents per video view to the content creator I think they can also afford paying a little for a highly efficient format. They have no need to do that of course because of AV1 and VP9 - which are well suited for the quality the user has come to expect from YouTube.

Correction: YouTube gives creators a cut of the ad revenue. You don't know whether what's left after the cut given to creators and after hosting expenses are taken into account is enough to pay for the content fee. I am always amazed by the insistence of H.265 fanboys to spend other people's money without knowing any of the numbers involved. I follow a more pragmatic approach which says that H.265 has been chosen by premium video providers but not free-to-view video providers and I trust that the economics work in either case.

My main gripe is the lack of an official baseline (mandatory format) for the html5 video tag, because I hate "open-ended" specs, but H.264 and VP8 have become the closest thing we have to a defacto baseline. I still hate the fact there is still no official baseline.

I think you are mistaken about MPEG-1, MPEG-2 and MPEG-4 not being a thing on the web because of content fees. Its the formats videos were shared in when the videos were put up for download for offline viewing. I think websites looking to embed videos in the page, because of ad revenue, were using RealVideo, Quicktime, Flash or some Windows IE extension only because there was no other widely supported browser support. These used patented and/or proprietary codecs internally anyway. And there were no content fees back then, just a per unit license for hard- or software. The licensing business was also less fragmented.
I remember downloading a mov file from IGN, not streaming it. But I've never come across MPEG-2 or MPEG4 Part 2 in streaming or downloadable form unless talking about pirate releases.

[benwaggoner]

1) Free-to-view doesn't mean YouTube.
It dominates, but yeah, Facebook has a ton, and it is a very long tail down to self-hosted .mp4 files.

2) How much did YouTube made in profit? Because that's the excess they could theoretically spend in content fees. And how much do the HEVC patent pools want in content fees per video?
The bandwidth savings of using HEVC are also substantial, which needs to go into ROI calculations like that. A model where the most popular 1% of titles get HEVC would spread out extra storage and encoding costs and still offer savings for >50% of streams delivered.

There is a reason websites serving free-to-view content (which make a razor-thin margin per each video) avoid formats with content fees like the plague, going all the way back to IGN serving Quicktime videos and later WMV (but not MPEG-2 or MPEG 4 Part 2), which was before YouTube was even a thing btw. Even MPEG-1 didn't became a thing for free-to-view videos until the last patent expired.
MPEG-2 licensing was so streaming unfriendly, yeah. But it didn't matter, as the decoder and bitrate requirements of MPEG-1 and MPEG-2 weren't viable for streaming until much better alternatives became available (originally better decoder performance like Cinepak and Indeo, later better compression efficiency like WMV9, Sorenson Video 3, and RealVideo 8). MPEG-1 was $0 licensing. MPEG-2's would have been prohibitively expensive if it were technical viable for IP delivery. I evaluated using MPEG-1 for progressive download after QuickTime added a free MPEG-1 decoder, but it just didn't offer practical benefits. The MP2 audio alone really needed at least 128 Kbps to be minimally acceptable. MPEG-4 pt 2 + AAC was a lot better by then.

MPEG-4 pt 2's licensing wasn't so bad in the end, but the ambiguity of it more than offset the relatively meager compression efficiency advantages it offered. The friction from "I don't know what it would cost" is often much bigger than the actual costs themselves. I certainly did use it for free to play projects back in late 90's.

I think the biggest reason why free sites use H.264 + AAC is that it is a good-enough codec that is universally compatible, fast to encode, and has an excellent free encoder that works well with a very wide variety of content in x264. Since a H.264 fallback is required, any additional codec is an additional thing to encode and store, and requires the complexity of making sure the right codec goes to the right player.

At a big enough scale, multi-codec will easily pay for itself in bandwidth savings, but paying for bandwidth can be a lot easier and less risk than hiring staff to deal with all the extra complexity required.

Not that we couldn't do it. I worked on a lot of web sites in the 90's that did auto selection of RealVideo, QuickTime, or Windows Media based on what browser plugins were available. It was a huge pain to have more than a few dozen videos available, of course.

[kurkosdr]

I believe all the stakeholders did clarify that free-to-view content doesn't have license fees as of a few years ago. Which was always true, but one of the licensing groups wouldn't tell anyone they wouldn't charge a content license expect under NDA. Which was infuriating.
Reliable source or else it didn't happen. The whole "there are no content fees but it's under NDA" sounds like an internet rumour. So, there is still not a straight "yes" or "no" answer to the question.

[kurkosdr]

It dominates, but yeah, Facebook has a ton, and it is a very long tail down to self-hosted .mp4 files.

I have posted some examples above. I missed Facebook though.

[kurkosdr]

It
The bandwidth savings of using HEVC are also substantial, which needs to go into ROI calculations like that. A model where the most popular 1% of titles get HEVC would spread out extra storage and encoding costs and still offer savings for >50% of streams delivered.

If a vacuum where only H.265 exists, maybe, but YouTube and Twitch use VP9 in addition to H.264. And YouTube also uses AV1.

If H.265 made financial sense, they'd use that instead.

[benwaggoner]

Free-to-view sounds nice. YouTube made $29.2 billion revenue last year though... maybe they could afford content fees if they had to without having to close shop?
From a licensing cost and business perspective, it's important to distinguish between pure free-to-view version ad-funded content with in-stream commercials. Actual free content has always had no content licensing cost. That's not always been true for FAST content, which mimics the classic over the air broadcast experience and business model.

[benwaggoner]

If a vacuum where only H.265 exists, maybe, but YouTube and Twitch use VP9 in addition to H.264. And YouTube also uses AV1.

If H.265 made financial sense, they'd use that instead.
Google has long had a strong philosophical objection to codecs with software patents, and has done many things that have disadvantaged them in order to encourage use of "free" codecs like VPx and AV1. They absolutely could have saved a bunch of money (from bandwidth savings) and improved customer experience (with better quality) by using HEVC on popular videos. And could have delivered some significant CO2 emissions savings; the aggregate power consumption of all those VP8/9 software decodes when HW H.264 was available, and SW AV1 when HEVC was avilable has to be pretty substantial.

YouTube has been the primary user of V8, VP9, and AV1 to date, and building out the whole ecosystem for what's largely internal use wasn't done out of clear-eyed operating cost ROI optimization.

AV1 is certainly the most technical viable and competitive codec to come out of their On2 aquisiton, but even still we don't see much use of it by other companies in terms of % of streams delivered. Lots of companies have announced they're supporting VPx and AV1 while still largely delivering in other codecs.

[kurkosdr]

Google has long had a strong philosophical objection to codecs with software patents, and has done many things that have disadvantaged them in order to encourage use of "free" codecs like VPx and AV1. They absolutely could have saved a bunch of money (from bandwidth savings) and improved customer experience (with better quality) by using HEVC on popular videos. And could have delivered some significant CO2 emissions savings; the aggregate power consumption of all those VP8/9 software decodes when HW H.264 was available, and SW AV1 when HEVC was avilable has to be pretty substantial.

YouTube has been the primary user of V8, VP9, and AV1 to date, and building out the whole ecosystem for what's largely internal use wasn't done out of clear-eyed operating cost ROI optimization.

AV1 is certainly the most technical viable and competitive codec to come out of their On2 aquisiton, but even still we don't see much use of it by other companies in terms of % of streams delivered. Lots of companies have announced they're supporting VPx and AV1 while still largely delivering in other codecs.
Thing is, all the other free-to-view sites mentioned above have also steered clear of H.265 for some reason.

Now that Chrome passes H.265 to hardware decoders (to the detriment of application consistency across different hardware configurations), you'd expect all those other free-to-view sites to switch to H.265 if H.265 makes so much financial sense over H.264 or VP9, but they don't.

Again, this goes back to H.265-everywhere fanboys willing to spend other people's money to gloss over their beloved format's licensing issues (instead of admitting the format makes financial sense in some markets but not in others).

[rwill]

I just noticed no one mentioned EVC Baseline yet, sad.

[kurkosdr]

I just noticed no one mentioned EVC Baseline yet, sad.
It's a dead format, too little too late compared to VP9.

[excellentswordfight]

I believe all the stakeholders did clarify that free-to-view content doesn't have license fees as of a few years ago. Which was always true, but one of the licensing groups wouldn't tell anyone they wouldn't charge a content license expect under NDA. Which was infuriating..
I assume you are refering to Velos, and it should be noted that they have ended their license pool for hevc, some of the members have now joined Access Advance (former hevc advance), it now looks like this:https://images.fonearena.com/blog/wp-content/uploads/2023/02/Access_Advance_HEVC_Patent_Sharp-1024x735.jpg

And regarding costs for av1, i agree with you there, the encoding requerments shoudlnt be overlooked. For example, just look at the aws MediaConvert pricing for av1, its A LOT more expensive to encode cause of the processing requerments.

[benwaggoner]

Thing is, all the other free-to-view sites mentioned above have also steered clear of H.265 for some reason.
Presumably because the technical complexity of supporting multiple codecs didn't make sense for them. The audience size has to be pretty decent before the economies of scale that make multi-codec pay for itself in reduced bandwidth outweigh the complexity and storage costs of having multiple codecs.

[Now that Chrome passes H.265 to hardware decoders (to the detriment of application consistency across different hardware configurations), you'd expect all those other free-to-view sites to switch to H.265 if H.265 makes so much financial sense over H.264 or VP9, but they don't.
Can you give some examples of these companies? I'm not sure of who you are thinking of and so can't really speculate.

Again, this goes back to H.265-everywhere fanboys willing to spend other people's money to gloss over their beloved format's licensing issues (instead of admitting the format makes financial sense in some markets but not in others).
I think we're exactly talking about how it makes sense in some markets but not others.

The original question was when a service could viably have no H.264 encodes at all. And I think it could be viable in a few years, but not today for anyone aiming for broad reach. A mobile-only service could viably have been all HEVC and xHE-AAC for several years now, due to the rapid replacement rate of those devices. There are still old pre-HEVC Smart TVs and streaming devices still in use due to their longer replacement cycles, although that's a small and shrinking share of the market.

A related question is when would it be viable to have only HDR content. which will probably take a few years beyond when H.264 could be deprecated.

H.264 has had an amazingly long run for a codec, a testament to the very innovative original design and the huge amount of optimization its encoders have seen over nearly 20 years. The broad early support for High Profile made a big difference. Baseline Profile and VC-1 were pretty competitive for a few years, but once there was High with both 4x4 and 8x8 blocks, B-frames, and CABAC, it was the king of the heap for years. It took its own next generation, HEVC, to have a broadly viable and superior replacement. HEVC is of course built out from H.264, which allowed existing H.264 encoders to make compliant bitstreams with relatively minor modifications, and refine from there.

If H.264 High10 decoders had become standard like High decoders were, H.264 could have had longer legs with decent 1080p HDR support and the efficiency improvements of 10-bit. But that was not to be. In any case, UHD really required HEVC, as it could deliver 4x the pixels in similar bitrates to H.264 High 1080p for content without too much grain.

[kurkosdr]

Can you give some examples of these companies? I'm not sure of who you are thinking of and so can't really speculate.

Copying and pasting from above:
- Facebook
- Twitch
- Various X-rated sites
- Microsoft Stream
- IBM Video
- various YouTube alternatives (Rumble etc)

All of them H.264-only, except Twitch which uses H.264 and VP9. None of them uses HEVC.

Again, if HEVC made financial sense for free-to-view web video, one of those services would be using it.

[rwill]

HEVC in Browser support is still fresh, give it some time. HEVC is already well established in OTT Streaming and Broadcast.

[Blue_MiSfit]

A service I worked on called Movies Anywhere uses HEVC wherever possible. This is a free service dealing with premium Hollywood content. It was free because it was tied to the digital rights you get when you buy a title on disc or at a streaming service.

We had no HEVC licensing costs, and were comfortable taking the risk. We delivered, 4K, with HDR, Dolby Vision, Atmos, etc.

[benwaggoner]

Copying and pasting from above:
- Facebook
- Twitch
- Various X-rated sites
- Microsoft Stream
- IBM Video
- various YouTube alternatives (Rumble etc)

All of them H.264-only, except Twitch which uses H.264 and VP9. None of them uses HEVC.
Facebook also has big Feels about Patents. They've really backed away from their AV1 goals, however. I can't speak about Twitch in detail as they are a partner company. I don't have much experience with the other providers/markets. It doesn't sound like content sees would apply in those cases, and companies like Microsoft and IBM have extensive patent cross-licensing portfolios which should reduce or eliminate encoder/decdoer fees.

Real world, it gets down to the potential benefits of reduced bandwidth costs versus the extra storage costs of having multiple codecs and the complexity of encoding and managing a multi codec service. Plus if a service is live and not being hugely profitable, the instinct is to leave it as it is rather than make a big capital investment in long-term operational cost savings. And HEVC in browsers is relatively new and kinda came out of the blue. Services could be working on multi-codec but not have launched yet.

Also, a lot of services you've mentioned aren't really monetizing quality, and so reducing bitrate, even with a quality hit, is a much easier knob to control storage and bandwidth costs.

And if a service didn't archive the source, but just kept distribution encodes, reencoding a mediocre H.264 into HEVC doesn't offer nearly the same benefits as encoding fresh from the source. This has been pretty common to save on storage costs.

Real-time HEVC encoding takes a few times more compute/pixel, and would require hardware upgrades if a service is using low-power fixed function H.264 encoders. We're not seeing much live AV1 for similar reasons (although it is several times slower yet than HEVC). The extra capital and/or operational costs per channel may not pencil out for low-viewership streams. And, at least, would require a new budget cycle.

Again, if HEVC made financial sense for free-to-view web video, one of those services would be using it.
There are lots of ways it may not have made sense for a given free-to-view service, as above. Sometimes it doesn't make sense as the leadership of a service may not be even thinking in these directions. It's a complex thing to model! And capital investments have a much higher bar these days with VC funding trying up and interest rates so high.

[kurkosdr]

Facebook also has big Feels about Patents.
Please try to get this into your skull: it's not about "feels", it's about financial realities. I personally lol'ed hard at the graph on the previous page: 2 patent pools, 8 unaffiliated entities. Nice. Unless you have a reliable source from all 10 entities pledging there won't be "content fees", jumping into HEVC carries the huge risk that one or those 10 entities will come to collect content fees years after you've spent time and electricity encoding your content to HEVC. Now, if you are a pay-to-view service, those fees can be rolled into the cost the customer pays to view the video, but free-to-view services serving tons of video to anyone who asks and making tight margins per video from ads don't have that option.

and companies like Microsoft and IBM have extensive patent cross-licensing portfolios which should reduce or eliminate encoder/decdoer fees.
Irrelevant, unless they have patent cross-licensing portfolios with every patent holder in HEVC. The patent holders they do not have patent cross-licensing portfolios with can always come to collect content fees.

Real world, it gets down to the potential benefits of reduced bandwidth costs versus the extra storage costs of having multiple codecs and the complexity of encoding and managing a multi codec service. Plus if a service is live and not being hugely profitable, the instinct is to leave it as it is rather than make a big capital investment in long-term operational cost savings. And HEVC in browsers is relatively new and kinda came out of the blue. Services could be working on multi-codec but not have launched yet.

Also, a lot of services you've mentioned aren't really monetizing quality, and so reducing bitrate, even with a quality hit, is a much easier knob to control storage and bandwidth costs.
Reality is, any free-to-view service choosing to use some more modern format alongside H.264 has gone to VP9. So there is a desire to use formats more modern than H.264 by some of those services, just not HEVC. If there was some kind of desire to use HEVC by some free-to-view service, we would have heard it by now. Pay-to-view services have already done it.

[kurkosdr]

A service I worked on called Movies Anywhere uses HEVC wherever possible. This is a free service dealing with premium Hollywood content. It was free because it was tied to the digital rights you get when you buy a title on disc or at a streaming service.

We had no HEVC licensing costs, and were comfortable taking the risk. We delivered, 4K, with HDR, Dolby Vision, Atmos, etc.

Then it's not a free service, it's part of the package the customer buys when buying the disc of streaming service in question, which means it's actually a bundled service. This means any potential content fee (I guess that's what you mean when you say "risk") can be bundled into the purchase price (the margins the Hollywood studio makes per disc or subscriber likely allow it).

[benwaggoner]

Please try to get this into your skull: it's not about "feels", it's about financial realities. I personally lol'ed hard at the graph on the previous page: 2 patent pools, 8 unaffiliated entities. Nice. Unless you have a reliable source from all 10 entities pledging there won't be "content fees", jumping into HEVC carries the huge risk that one or those 10 entities will come to collect content fees years after you've spent time and electricity encoding your content to HEVC. Now, if you are a pay-to-view service, those fees can be rolled into the cost the customer pays to view the video, but free-to-view services serving tons of video to anyone who asks and making tight margins per video from ads don't have that option.
You may feel there is a problem in theory, but lots of companies have adopted HEVC without any problems in practice. I concur that getting a clear straight answer from all participants is way more complicated than it should ever have been. But unless you have some examples of someone actually having to pay content licenses for free-to-play content, this is largely conjecture.

Irrelevant, unless they have patent cross-licensing portfolios with every patent holder in HEVC. The patent holders they do not have patent cross-licensing portfolios with can always come to collect content fees.
In practice, the patent cross-licensing can have a big effect on these concerns, as it allows a whole lot of patents that could apply be ruled out. IANAL, but this seems to be the general case. There's no patent issue without a patent holder wiling to sue.

Reality is, any free-to-view service choosing to use some more modern format alongside H.264 has gone to VP9. So there is a desire to use formats more modern than H.264 by some of those services, just not HEVC. If there was some kind of desire to use HEVC by some free-to-view service, we would have heard it by now. Pay-to-view services have already done it.
That's like saying premium content must have a big reason not to adopt VVC as no one has yet. Market timing is a big factor here. Bear in mind that VP9 has been broadly playable in web browsers for seven years now, while HEVC has been broadly supported for just shy of a year.

VP9 also got a lot of very optimistic promises made about bitrate savings over H.264 that never really paid off, largely due to x264, but also because of some poor design decisions that forced a lot of serialization for encoding and decoding. The VP9 toolchain was fine for YouTube splitting up lots of single-threaded chunks across lots of instances, but not for a lot of core x264 scenarios.

[Spyros]

Small update, a newer patent list is available (August 1, 2023 (https://web.archive.org/web/20230907021043/https://www.via-la.com/wp-content/uploads/Final-August-1-2023-AVC-Attachment-1.pdf)).

I didn't notice earlier, because the website changed (from mpegla.com to via-la.com).

19 pages with unexpired patents, 37 pages with expired ones.

[benwaggoner]

There's a few pages worth that are expiring this or next month as well. Dolby really dominates the remaining ones.

[kurkosdr]

Small update, a newer patent list is available (August 1, 2023 (https://web.archive.org/web/20230907021043/https://www.via-la.com/wp-content/uploads/Final-August-1-2023-AVC-Attachment-1.pdf)).

I didn't notice earlier, because the website changed (from mpegla.com to via-la.com).

19 pages with unexpired patents, 37 pages with expired ones.
In the US, the last patent for the first version of the H.264 standard will expire on 29 nov 2027:
https://scratchpad.fandom.com/wiki/MPEG_patent_lists#H.264_patents
So it's not worth holding your breath (when it comes to US patents).

I am interested how many EU patents remain active after 17 August 2004 (the date the first version of the H.264 standard was published), since the EU doesn't have a habit of granting multi-year patent extensions willy-nilly like the US does. For example, MPEG4 ASP is still patented in the US (and in Brazil, due to an one-time extension given to some patents in that country) but is royalty-free everywhere else.

MPEG LA updates their patents lists every 3 months btw.

[Avclover]

In the US, the last patent for the first version of the H.264 standard will expire on 29 nov 2027

It seems the current VIA LA patent list features at least one early US patent that expires only at the end of 2030:
US9356620B2 (https://patents.google.com/patent/US9356620B2/).

It is also worth noting that the ubiquitous high profile was introduced only after version 1 of AVC.

[kurkosdr]

It seems the current VIA LA patent list features at least one early US patent that expires only at the end of 2030:
US9356620B2 (https://patents.google.com/patent/US9356620B2/).

It is also worth noting that the ubiquitous high profile was introduced only after version 1 of AVC.

VIA LA's patent list includes extensions such as SVC and MVC which are technically part of the AVC standard but are not essential to implement (both SVC and MVC are backwards-compatible extensions).
https://www.via-la.com/licensing-2/avc-h-264/

You are correct about the High Profile though: The High profile was added on 1st March 2005, so that's the patent priority date to look for, I guess.