More than half of the H.264 patents have expired

[benwaggoner]

Originally Posted by kurkosdr

Netflix streams AV1 if the device supports it

For HDR as well? If so, I wasn't aware! Do you have any links?

I read somewhere that MPEG LA used to charge content fees, but that may be outdated. Anyway, unless there is a reliable source for every patent pool (that they have waived content fees), I don't believe it, there are just too many patent pools and entities getting their beak wet from H.265, and every one has to agree for H.265 to be free of content fees.

MPEG-LA has had some commercial content fees before, but they've been reasonable. The bigger friction has been from ambiguity before the patent pool(s) have finalized and published details. This was more complex for HEVC which had three pools, not all of which published any license details and required an attorney and NDA to get the specifics.

It hasn't been solved for browsers, since the business model of the entire industry revolves around giving away the browser for $0. Just because a company like Google begrudingly pays the H.264 royalties for Chrome (and Cisco for Firefox) doesn't mean they will do it again for HEVC. Chrome "solves" the issue by using HEVC hardware acceleration if supported by the system (and not decoding HEVC if not), which means that as a service provider you will have to also encode in H.264 or VP8. Which is why if you can't encode and store multiple formats (on top of encoding and storing multiple resolutions that you have to do anyway), H.264 and VP8 are your only viable formats. But if you can encode in multiple-formats, then every format can be said to be viable, even something like VVC+Dolby Vision can be said to be viable if there is an H.264 or VP8 stream to fall back to.

It has been "solved" in the sense that HEVC decode hardware is now ubiquitous in most markets, and a free browser can pass a HEVC bitstream to a system-level SW or HW decoder (just like with AV1). There will be some edge cases without a HEVC decoder, of course, so full ubiquity still requires H.264 as a fallback.

For premium content requiring hardware DRM, just HEVC is more viable because the systems without HEVC HW decoders rarely have sufficiently protected HW DRM. Same with HDR, as the 10-bit requirement rules out most H.264 decoders in systems without HEVC decoders.

I am using HEVC as an illustrative example here, as the same market dynamics apply to AV1, AV2, VVC, and on into the future. I doubt we'll see any major content platform adopt a single codec solution other than H.264 in the next decade.

[kurkosdr]

Originally Posted by benwaggoner

For HDR as well? If so, I wasn't aware! Do you have any links?

They are considering AV1 HDR in the future, but not doing it at the moment:
https://www.audioholics.com/hdtv-formats/netflix-av1
Don't ask what's taking them so long, YouTube is doing it with AV1 and VP9.

Originally Posted by benwaggoner

MPEG-LA has had some commercial content fees before, but they've been reasonable. The bigger friction has been from ambiguity before the patent pool(s) have finalized and published details. This was more complex for HEVC which had three pools, not all of which published any license details and required an attorney and NDA to get the specifics.

Quite honestly, this is why I hate HEVC with a passion when it comes to free-to-view content: Nobody out there has a straight "yes" or "no" answer to the thorny issue of content fees. Which is why I think it's best to avoid it for free-to-view content.

At this point it makes you wonder whether investing in HDR H.264 encoders is indeed the best option. Resolutions above 1080p are unafforable for free-to-view content anyway, and the bitrate impact of HDR will be marginal.

[rwill]

Free-to-view sounds nice. YouTube made $29.2 billion revenue last year though... maybe they could afford content fees if they had to without having to close shop?

[kurkosdr]

Originally Posted by rwill

Free-to-view sounds nice. YouTube made $29.2 billion revenue last year though... maybe they could afford content fees if they had to without having to close shop?

1) Free-to-view doesn't mean YouTube.

2) How much did YouTube made in profit? Because that's the excess they could theoretically spend in content fees. And how much do the HEVC patent pools want in content fees per video?

There is a reason websites serving free-to-view content (which make a razor-thin margin per each video) avoid formats with content fees like the plague, going all the way back to IGN serving Quicktime videos and later WMV (but not MPEG-2 or MPEG 4 Part 2), which was before YouTube was even a thing btw. Even MPEG-1 didn't became a thing for free-to-view videos until the last patent expired.

[rwill]

Originally Posted by kurkosdr

1) Free-to-view doesn't mean YouTube.

2) How much did YouTube made in profit? Because that's the excess they could theoretically spend in content fees. And how much do the HEVC patent pools want in content fees per video?

There is a reason websites serving free-to-view content (which make a razor-thin margin per each video) avoid formats with content fees like the plague, going all the way back to IGN serving Quicktime videos and later WMV (but not MPEG-2 or MPEG 4 Part 2), which was before YouTube was even a thing btw. Even MPEG-1 didn't became a thing for free-to-view videos until the last patent expired.

Can you give examples of free-to-view websites other than YouTube? Maybe some with a different business model?

It appears Alphabet does not tell how much YouTube made them in profit. But if Youtube can give some cents per video view to the content creator I think they can also afford paying a little for a highly efficient format. They have no need to do that of course because of AV1 and VP9 - which are well suited for the quality the user has come to expect from YouTube.

I think you are mistaken about MPEG-1, MPEG-2 and MPEG-4 not being a thing on the web because of content fees. Its the formats videos were shared in when the videos were put up for download for offline viewing. I think websites looking to embed videos in the page, because of ad revenue, were using RealVideo, Quicktime, Flash or some Windows IE extension only because there was no other widely supported browser support. These used patented and/or proprietary codecs internally anyway. And there were no content fees back then, just a per unit license for hard- or software. The licensing business was also less fragmented.

[benwaggoner]

Originally Posted by kurkosdr

They are considering AV1 HDR in the future, but not doing it at the moment:
https://www.audioholics.com/hdtv-formats/netflix-av1
Don't ask what's taking them so long, YouTube is doing it with AV1 and VP9.

Netflix, as a premium content company, has a lot higher quality bar than YouTube. And while AV1 doesn't seem to have any fundamental deficit for HDR, encoding in PQ and Rec. 2020 requires a fair amount of retuning away from gamma and Rec. 709. For a simple example, there's the --hdr10-opt from x265.
AV1 also encodes more slowly than HEVC in general, and as HDR comes with UHD, those extra pixels can really slow down encoding. There's a big difference between taking 12 hours and 36 hours to encode a movie.

Quite honestly, this is why I hate HEVC with a passion when it comes to free-to-view content: Nobody out there has a straight "yes" or "no" answer to the thorny issue of content fees. Which is why I think it's best to avoid it for free-to-view content.

I believe all the stakeholders did clarify that free-to-view content doesn't have license fees as of a few years ago. Which was always true, but one of the licensing groups wouldn't tell anyone they wouldn't charge a content license expect under NDA. Which was infuriating.

At this point it makes you wonder whether investing in HDR H.264 encoders is indeed the best option. Resolutions above 1080p are unafforable for free-to-view content anyway, and the bitrate impact of HDR will be marginal.

HDR requires 10-bit, and there are a lot more 10-bit HEVC HW decoders out there than 10-bit H.264. And HDR is also most often UHD, where the compression efficiency improvements of HEVC are even bigger than at lower resolutions. Having a 32x32 max block size is really useful at higher resolutions versus a max of 8x8. H.264 UHD HDR would need to at least double bitrates, and with much less compatibility.

And, obviously, premium content HDR playback launched HEVC only over eight years now, and is still exclusively HEVC other than YouTube's user generated content.

[kurkosdr]

Originally Posted by rwill

Can you give examples of free-to-view websites other than YouTube? Maybe some with a different business model?

X-rated sites are a big example, then there is Twitch, Microsoft Stream, IBM Video, plus the various YouTube alternatives (Rumble etc).

None of them uses H.265, it's H.264 and maybe VP9 and AV1.

Originally Posted by rwill

It appears Alphabet does not tell how much YouTube made them in profit. But if Youtube can give some cents per video view to the content creator I think they can also afford paying a little for a highly efficient format. They have no need to do that of course because of AV1 and VP9 - which are well suited for the quality the user has come to expect from YouTube.

Correction: YouTube gives creators a cut of the ad revenue. You don't know whether what's left after the cut given to creators and after hosting expenses are taken into account is enough to pay for the content fee. I am always amazed by the insistence of H.265 fanboys to spend other people's money without knowing any of the numbers involved. I follow a more pragmatic approach which says that H.265 has been chosen by premium video providers but not free-to-view video providers and I trust that the economics work in either case.

My main gripe is the lack of an official baseline (mandatory format) for the html5 video tag, because I hate "open-ended" specs, but H.264 and VP8 have become the closest thing we have to a defacto baseline. I still hate the fact there is still no official baseline.

Originally Posted by rwill

I think you are mistaken about MPEG-1, MPEG-2 and MPEG-4 not being a thing on the web because of content fees. Its the formats videos were shared in when the videos were put up for download for offline viewing. I think websites looking to embed videos in the page, because of ad revenue, were using RealVideo, Quicktime, Flash or some Windows IE extension only because there was no other widely supported browser support. These used patented and/or proprietary codecs internally anyway. And there were no content fees back then, just a per unit license for hard- or software. The licensing business was also less fragmented.

I remember downloading a mov file from IGN, not streaming it. But I've never come across MPEG-2 or MPEG4 Part 2 in streaming or downloadable form unless talking about pirate releases.

[benwaggoner]

Originally Posted by kurkosdr

1) Free-to-view doesn't mean YouTube.

It dominates, but yeah, Facebook has a ton, and it is a very long tail down to self-hosted .mp4 files.

2) How much did YouTube made in profit? Because that's the excess they could theoretically spend in content fees. And how much do the HEVC patent pools want in content fees per video?

The bandwidth savings of using HEVC are also substantial, which needs to go into ROI calculations like that. A model where the most popular 1% of titles get HEVC would spread out extra storage and encoding costs and still offer savings for >50% of streams delivered.

There is a reason websites serving free-to-view content (which make a razor-thin margin per each video) avoid formats with content fees like the plague, going all the way back to IGN serving Quicktime videos and later WMV (but not MPEG-2 or MPEG 4 Part 2), which was before YouTube was even a thing btw. Even MPEG-1 didn't became a thing for free-to-view videos until the last patent expired.

MPEG-2 licensing was so streaming unfriendly, yeah. But it didn't matter, as the decoder and bitrate requirements of MPEG-1 and MPEG-2 weren't viable for streaming until much better alternatives became available (originally better decoder performance like Cinepak and Indeo, later better compression efficiency like WMV9, Sorenson Video 3, and RealVideo 8). MPEG-1 was $0 licensing. MPEG-2's would have been prohibitively expensive if it were technical viable for IP delivery. I evaluated using MPEG-1 for progressive download after QuickTime added a free MPEG-1 decoder, but it just didn't offer practical benefits. The MP2 audio alone really needed at least 128 Kbps to be minimally acceptable. MPEG-4 pt 2 + AAC was a lot better by then.

MPEG-4 pt 2's licensing wasn't so bad in the end, but the ambiguity of it more than offset the relatively meager compression efficiency advantages it offered. The friction from "I don't know what it would cost" is often much bigger than the actual costs themselves. I certainly did use it for free to play projects back in late 90's.

I think the biggest reason why free sites use H.264 + AAC is that it is a good-enough codec that is universally compatible, fast to encode, and has an excellent free encoder that works well with a very wide variety of content in x264. Since a H.264 fallback is required, any additional codec is an additional thing to encode and store, and requires the complexity of making sure the right codec goes to the right player.

At a big enough scale, multi-codec will easily pay for itself in bandwidth savings, but paying for bandwidth can be a lot easier and less risk than hiring staff to deal with all the extra complexity required.

Not that we couldn't do it. I worked on a lot of web sites in the 90's that did auto selection of RealVideo, QuickTime, or Windows Media based on what browser plugins were available. It was a huge pain to have more than a few dozen videos available, of course.

[kurkosdr]

Originally Posted by benwaggoner

I believe all the stakeholders did clarify that free-to-view content doesn't have license fees as of a few years ago. Which was always true, but one of the licensing groups wouldn't tell anyone they wouldn't charge a content license expect under NDA. Which was infuriating.

Reliable source or else it didn't happen. The whole "there are no content fees but it's under NDA" sounds like an internet rumour. So, there is still not a straight "yes" or "no" answer to the question.

[kurkosdr]

Originally Posted by benwaggoner

It dominates, but yeah, Facebook has a ton, and it is a very long tail down to self-hosted .mp4 files.

I have posted some examples above. I missed Facebook though.

[kurkosdr]

Originally Posted by benwaggoner

It
The bandwidth savings of using HEVC are also substantial, which needs to go into ROI calculations like that. A model where the most popular 1% of titles get HEVC would spread out extra storage and encoding costs and still offer savings for >50% of streams delivered.

If a vacuum where only H.265 exists, maybe, but YouTube and Twitch use VP9 in addition to H.264. And YouTube also uses AV1.

If H.265 made financial sense, they'd use that instead.

[benwaggoner]

Originally Posted by rwill

Free-to-view sounds nice. YouTube made $29.2 billion revenue last year though... maybe they could afford content fees if they had to without having to close shop?

From a licensing cost and business perspective, it's important to distinguish between pure free-to-view version ad-funded content with in-stream commercials. Actual free content has always had no content licensing cost. That's not always been true for FAST content, which mimics the classic over the air broadcast experience and business model.

[benwaggoner]

Originally Posted by kurkosdr

If a vacuum where only H.265 exists, maybe, but YouTube and Twitch use VP9 in addition to H.264. And YouTube also uses AV1.

If H.265 made financial sense, they'd use that instead.

Google has long had a strong philosophical objection to codecs with software patents, and has done many things that have disadvantaged them in order to encourage use of "free" codecs like VPx and AV1. They absolutely could have saved a bunch of money (from bandwidth savings) and improved customer experience (with better quality) by using HEVC on popular videos. And could have delivered some significant CO2 emissions savings; the aggregate power consumption of all those VP8/9 software decodes when HW H.264 was available, and SW AV1 when HEVC was avilable has to be pretty substantial.

YouTube has been the primary user of V8, VP9, and AV1 to date, and building out the whole ecosystem for what's largely internal use wasn't done out of clear-eyed operating cost ROI optimization.

AV1 is certainly the most technical viable and competitive codec to come out of their On2 aquisiton, but even still we don't see much use of it by other companies in terms of % of streams delivered. Lots of companies have announced they're supporting VPx and AV1 while still largely delivering in other codecs.

[kurkosdr]

Originally Posted by benwaggoner

Google has long had a strong philosophical objection to codecs with software patents, and has done many things that have disadvantaged them in order to encourage use of "free" codecs like VPx and AV1. They absolutely could have saved a bunch of money (from bandwidth savings) and improved customer experience (with better quality) by using HEVC on popular videos. And could have delivered some significant CO2 emissions savings; the aggregate power consumption of all those VP8/9 software decodes when HW H.264 was available, and SW AV1 when HEVC was avilable has to be pretty substantial.

YouTube has been the primary user of V8, VP9, and AV1 to date, and building out the whole ecosystem for what's largely internal use wasn't done out of clear-eyed operating cost ROI optimization.

AV1 is certainly the most technical viable and competitive codec to come out of their On2 aquisiton, but even still we don't see much use of it by other companies in terms of % of streams delivered. Lots of companies have announced they're supporting VPx and AV1 while still largely delivering in other codecs.

Thing is, all the other free-to-view sites mentioned above have also steered clear of H.265 for some reason.

Now that Chrome passes H.265 to hardware decoders (to the detriment of application consistency across different hardware configurations), you'd expect all those other free-to-view sites to switch to H.265 if H.265 makes so much financial sense over H.264 or VP9, but they don't.

Again, this goes back to H.265-everywhere fanboys willing to spend other people's money to gloss over their beloved format's licensing issues (instead of admitting the format makes financial sense in some markets but not in others).

[rwill]

I just noticed no one mentioned EVC Baseline yet, sad.

[kurkosdr]

Originally Posted by rwill

I just noticed no one mentioned EVC Baseline yet, sad.

It's a dead format, too little too late compared to VP9.

[excellentswordfight]

Originally Posted by benwaggoner

I believe all the stakeholders did clarify that free-to-view content doesn't have license fees as of a few years ago. Which was always true, but one of the licensing groups wouldn't tell anyone they wouldn't charge a content license expect under NDA. Which was infuriating..

I assume you are refering to Velos, and it should be noted that they have ended their license pool for hevc, some of the members have now joined Access Advance (former hevc advance), it now looks like this:

And regarding costs for av1, i agree with you there, the encoding requerments shoudlnt be overlooked. For example, just look at the aws MediaConvert pricing for av1, its A LOT more expensive to encode cause of the processing requerments.

[benwaggoner]

Originally Posted by kurkosdr

Thing is, all the other free-to-view sites mentioned above have also steered clear of H.265 for some reason.

Presumably because the technical complexity of supporting multiple codecs didn't make sense for them. The audience size has to be pretty decent before the economies of scale that make multi-codec pay for itself in reduced bandwidth outweigh the complexity and storage costs of having multiple codecs.

[

Now that Chrome passes H.265 to hardware decoders (to the detriment of application consistency across different hardware configurations), you'd expect all those other free-to-view sites to switch to H.265 if H.265 makes so much financial sense over H.264 or VP9, but they don't.

Can you give some examples of these companies? I'm not sure of who you are thinking of and so can't really speculate.

Again, this goes back to H.265-everywhere fanboys willing to spend other people's money to gloss over their beloved format's licensing issues (instead of admitting the format makes financial sense in some markets but not in others).

I think we're exactly talking about how it makes sense in some markets but not others.

The original question was when a service could viably have no H.264 encodes at all. And I think it could be viable in a few years, but not today for anyone aiming for broad reach. A mobile-only service could viably have been all HEVC and xHE-AAC for several years now, due to the rapid replacement rate of those devices. There are still old pre-HEVC Smart TVs and streaming devices still in use due to their longer replacement cycles, although that's a small and shrinking share of the market.

A related question is when would it be viable to have only HDR content. which will probably take a few years beyond when H.264 could be deprecated.

H.264 has had an amazingly long run for a codec, a testament to the very innovative original design and the huge amount of optimization its encoders have seen over nearly 20 years. The broad early support for High Profile made a big difference. Baseline Profile and VC-1 were pretty competitive for a few years, but once there was High with both 4x4 and 8x8 blocks, B-frames, and CABAC, it was the king of the heap for years. It took its own next generation, HEVC, to have a broadly viable and superior replacement. HEVC is of course built out from H.264, which allowed existing H.264 encoders to make compliant bitstreams with relatively minor modifications, and refine from there.

If H.264 High10 decoders had become standard like High decoders were, H.264 could have had longer legs with decent 1080p HDR support and the efficiency improvements of 10-bit. But that was not to be. In any case, UHD really required HEVC, as it could deliver 4x the pixels in similar bitrates to H.264 High 1080p for content without too much grain.

[kurkosdr]

Originally Posted by benwaggoner

Can you give some examples of these companies? I'm not sure of who you are thinking of and so can't really speculate.

Copying and pasting from above:
- Facebook
- Twitch
- Various X-rated sites
- Microsoft Stream
- IBM Video
- various YouTube alternatives (Rumble etc)

All of them H.264-only, except Twitch which uses H.264 and VP9. None of them uses HEVC.

Again, if HEVC made financial sense for free-to-view web video, one of those services would be using it.

[rwill]

HEVC in Browser support is still fresh, give it some time. HEVC is already well established in OTT Streaming and Broadcast.