Versatile Video Coding (VVC) / H.266: HEVC successor

[Z2697]

Originally Posted by edison

https://git.ffmpeg.org/gitweb/ffmpeg...7.1:/Changelog

version 7.1:
- Raw Captions with Time (RCWT) closed caption demuxer
- LC3/LC3plus decoding/encoding using external library liblc3
- ffmpeg CLI filtergraph chaining
- LC3/LC3plus demuxer and muxer
- pad_vaapi, drawbox_vaapi filters
- vf_scale supports secondary ref input and framesync options
- vf_scale2ref deprecated
- qsv_params option added for QSV encoders
- VVC decoder compatible with DVB test content
- xHE-AAC decoder
- removed DEC Alpha DSP and support code
- VVC encoding support via libvvenc
- perlin video source
- D3D12VA HEVC encoder
- Cropping metadata parsing and writing in Matroska and MP4/MOV de/muxers
- Intel QSV-accelerated VVC decoding
- MediaCodec AAC/AMR-NB/AMR-WB/MP3 decoding
- YUV colorspace negotiation for codecs and filters, obsoleting the
  YUVJ pixel format
- Vulkan H.264 encoder
- Vulkan H.265 encoder
- stream specifiers in fftools can now match by stream disposition
- LCEVC enhancement data exporting in H.26x and MP4/ISOBMFF
- LCEVC filter
- MV-HEVC decoding
- minor stream specifier syntax changes:
    - when matching by metadata (:m:<key>:<val>), the colon character
      in keys or values now has to be backslash-escaped
    - in optional maps (-map ....?) with a metadata-matching stream specifier,
      the value has to be separated from the question mark by a colon, i.e.
      -map ....:m:<key>:<val>:? (otherwise it would be ambiguous whether the
      question mark is a part of <val> or not)
    - multiple stream types in a single specifier (e.g. :s:s:0) now cause an
      error, as such a specifier makes no sense
- Mastering Display and Content Light Level metadata support in hevc_nvenc
  and av1_nvenc encoders
- libswresample now accepts custom order channel layouts as input, with some
  constrains
- FFV1 parser

Which Intel GPU support QSV VVC decoding?

I think it's currently only on the Lunar Lake (mobile chips), not even on the same generation desktop chips, not to say the Arc video cards... so that euqals to... nothing? Anyone buying Lunar Lake?

(I guess "not to say" isn't quite fit in here, because Arc Battlemage was release after Lunar Lake... but you get the point, right? (am I getting the timeline correct?))

[GeoffreyA]

Originally Posted by modus-ms325c

actually VVC would still be avoided like the plague, leading to AVC still being used regardless of anything.
plus, tech companies would've scrambled to rush out a video codec that's easier to adopt while putting on a show pretending that their shiny new codec is actually much better than MPEG's offerings.

all-in-all, doesn't matter anymore. if VVC can be declared dead owing to patent situations making it difficult/impossible to even use it, let's just follow suit by calling it such, dance on its "corpse", have a ball at doing so, and call it a day.

I think AVC hit critical mass in the field of encoding, and HEVC tightened it up a bit. Both haven't been matched in transparent encoding. AV1 found its place in low-bitrate encoding and anime (though HEVC still leads in the latter). As for VVC, well, I lament that the baby was stillborn, especially as one who awaited its birth and rapid growth to adulthood.

[excellentswordfight]

Originally Posted by GeoffreyA

I think AVC hit critical mass in the field of encoding, and HEVC tightened it up a bit. Both haven't been matched in transparent encoding. AV1 found its place in low-bitrate encoding and anime (though HEVC still leads in the latter). As for VVC, well, I lament that the baby was stillborn, especially as one who awaited its birth and rapid growth to adulthood.

Yeah, at least HEVC could ride on the UHD and HDR wave, pretty much forcing any content provider interested in those two to use HEVC. Whats the USP for VVC? Yes, yes, 640K ought to be enough for anyone and so on, but tbh with 4k/UHD/HDR/HEVC I think we are starting to get to the top of the s curve were diminishing returns is very much in effect. I think the only killer feature today would be a clear savings cost, either from licensing and or encoding/decoding/distrubution cost.

[GeoffreyA]

Originally Posted by Z2697

I think it's currently only on the Lunar Lake (mobile chips), not even on the same generation desktop chips, not to say the Arc video cards... so that euqals to... nothing? Anyone buying Lunar Lake?

(I guess "not to say" isn't quite fit in here, because Arc Battlemage was release after Lunar Lake... but you get the point, right? (am I getting the timeline correct?))

Seems to be only Lunar Lake at present. As far I can tell, the media engine of discrete Battlemage doesn't have VVC decoding. I imagine the cards were finalised with the earlier media engine, or at least the B580 was, whereas the CPU SoC division had the newer one included earlier in development. By the way, the B580 turns out to be an excellent GPU, the best in its category from a price point of view. If it had VVC decoding, that would have been a cherry on top.

[GeoffreyA]

Originally Posted by excellentswordfight

Yeah, at least HEVC could ride on the UHD and HDR wave, pretty much forcing any content provider interested in those two to use HEVC. Whats the USP for VVC? Yes, yes, 640K ought to be enough for anyone and so on, but tbh with 4k/UHD/HDR/HEVC I think we are starting to get to the top of the s curve were diminishing returns is very much in effect. I think the only killer feature today would be a clear savings cost, either from licensing and or encoding/decoding/distrubution cost.

I agree that we're hitting a limit; I don't know what the exact numbers are, but surely, the human eye is nearing saturation. Will 8K be placebo in the home? Streaming and broadcasting have different concerns, but for us encoders, what we need from modern codecs is H.264/5 calibre at post-AV1 sizes, something contemporary codecs are not providing.

[kurkosdr]

Originally Posted by excellentswordfight

I think the only killer feature today would be a clear savings cost, either from licensing and or encoding/decoding/distrubution cost.

There would be a case for reduction of distribution costs with VVC if the patent holders charged reasonable "content fees", but this isn't the case, in fact the patent holders have split themselves into 20 separate entities or so to maximize the royalties they could milk without technically running afoul of FRAND, so as distributor and you have to negotiate a "content fee" with each one. Good luck with that. Also keep in mind that internet bandwidth and storage get cheaper over time, while the patent holders responsible for the VVC licensing mess are getting greedier over time (VVC has double the patent-licensing entities of HEVC, and it might get even worse).

The streaming industry shifted to AV1 as its post-HEVC standard and that's it. The broadcast industry is stuck in HEVC for the foreseeable future (and some of it is still trying to get rid of MPEG2) so they aren't relevant to the VVC discussion, save for a few countries.

[benwaggoner]

Originally Posted by GeoffreyA

I agree that we're hitting a limit; I don't know what the exact numbers are, but surely, the human eye is nearing saturation. Will 8K be placebo in the home?[-/QUOTE]
8K is already placebo for moving images, where motion blur eliminates any use of the actual precision. We tested carefully selected 8K 10-bit uncompressed 24p moving image content, viewed by expert golden eye viewers in an A/B/A/B comparison. There were a couple of clips where people sitting 55" form a 65" TV who had 20/10 vision could tell the difference.

Literally no consumer will be able to tell you if something is 4K or 8K while watching something without a reference.

For very static, sharp stuff like looking at text on a computer, it's possible for some people to tell, but people also lean in very close with computers. There being value at >>24 fps hasn't been disproven, but by definition that will only matter where there is motion and thus motion blur, so it seems unlikely. And has 100/120 fps is a sports thing, Even having a real-time 8Kp120 encoder that can produce flawless sharp quality at distribution bandwidth is many years away.

[Streaming and broadcasting have different concerns, but for us encoders, what we need from modern codecs is H.264/5 calibre at post-AV1 sizes, something contemporary codecs are not providing.

Really, the only place where modern encoders can deliver flawless quality at practical bitrates is with film grain or other random noise. The killer feature of AV1 could have been Film Grain Synthesis, but a reliable infrastructure of encoders and complain decoders hasn't emerged (a lot of early AV1 products had broken FGS, because there weren't any conformance tests available yet).

Now that is becoming a general standard that could be applied to any codec, so it's not AV1 specific (it's not in-loop, just metadata driven post processing). Hopefully AV2 will get universally compatible decoders so it could be an on-by-default option there at least.

AV1's FGS + VVC, or MPEG FGS + VVC are both technically viable too, with some good demos already done.

[benwaggoner]

Originally Posted by kurkosdr

There would be a case for reduction of distribution costs with VVC if the patent holders charged reasonable "content fees", but this isn't the case, in fact the patent holders have split themselves into 20 separate entities or so to maximize the royalties they could milk without technically running afoul of FRAND, so as distributor and you have to negotiate a "content fee" with each one. Good luck with that. Also keep in mind that internet bandwidth and storage get cheaper over time, while the patent holders responsible for the VVC licensing mess are getting greedier over time (VVC has double the patent-licensing entities of HEVC, and it might get even worse).

The streaming industry shifted to AV1 as its post-HEVC standard and that's it. The broadcast industry is stuck in HEVC for the foreseeable future (and some of it is still trying to get rid of MPEG2) so they aren't relevant to the VVC discussion, save for a few countries.

That's overstated. The user generated content industry (Facebook, YouTube) has tried to shift to AV1 where possible, as software decode without HW DRM is viable for them.

In premium content, it exists from both Netflix and Prime Video, given the installed base of decoders, there's no way that most premium HDR content isn't still delivered in HDR. We're still having mid-tier mobile devices and some TVs being launched without AV1 support. Once they're pretty much universal, we'll have AV1 mostly available in mobile within four years and in living room within ten years. But it's still a while before it could be even half of premium content.

Still, at a big enough scale, saving 30% of bandwidth costs on 20% of sessions can be more than enough savings to justify the effort. Of course, you'll get increased encoding and storage costs for the time being, as you can't stop using HEVC or even H.264 within the next few years, so AV1 would be an additional thing, not a replacement.

[GeoffreyA]

Originally Posted by benwaggoner

Really, the only place where modern encoders can deliver flawless quality at practical bitrates is with film grain or other random noise. The killer feature of AV1 could have been Film Grain Synthesis, but a reliable infrastructure of encoders and complain decoders hasn't emerged (a lot of early AV1 products had broken FGS, because there weren't any conformance tests available yet).

Now that is becoming a general standard that could be applied to any codec, so it's not AV1 specific (it's not in-loop, just metadata driven post processing). Hopefully AV2 will get universally compatible decoders so it could be an on-by-default option there at least.

AV1's FGS + VVC, or MPEG FGS + VVC are both technically viable too, with some good demos already done.

That's interesting, Ben, about the testing at 8K. It confirms my suspicions that the biological limit is being reached. A good thing, so we can put effort elsewhere. As for high frame rates, it's much in demand these days. I find it disagreeable for non-gaming content; there's a different feeling, losing that film effect, along with a mental strain, as if I would get a headache. Drop back to 24 and it's normal. No doubt, there is some early neurological training at play here.

Indeed, grain or random noise is the main challenge to video codecs, and proper film-grain synthesis could address this. Being an implementation problem, it will improve as time goes by.

[benwaggoner]

Originally Posted by GeoffreyA

That's interesting, Ben, about the testing at 8K. It confirms my suspicions that the biological limit is being reached. A good thing, so we can put effort elsewhere. As for high frame rates, it's much in demand these days. I find it disagreeable for non-gaming content; there's a different feeling, losing that film effect, along with a mental strain, as if I would get a headache. Drop back to 24 and it's normal. No doubt, there is some early neurological training at play here.

24p seems pretty indelible for scripted content, although people like James Cameron try higher from time to time.

Where very high frame rates shine is sports. 120 is better for lots of people than 60, even.

Indeed, grain or random noise is the main challenge to video codecs, and proper film-grain synthesis could address this. Being an implementation problem, it will improve as time goes by.

No doubt. I'm not sure we've nailed it yet, but we should within a decade.

[birdie]

The State of the Video Codec Market 2025

VVC is still DOA.

There are a few realities worth noting about the percentages shown in Figure 1. First, you’ll realize those savings only when streaming to compatible platforms, which will be well short of 100% of the time. Second, you’ll realize those savings only at the top rung of your encoding ladder—which, in fairness, is typically the most widely viewed rung. However, if you distribute lots of middle-rung content to mobile devices, you’ll be swapping a 2Mbps H.264 rung for a 2Mbps AV1 rung. The video should look better, improving the QoE for your viewers, but you’ll see no bandwidth savings.

In addition, as bandwidth costs drop, the savings drop as well. Twenty years ago, when it cost $0.50 to deliver a GB of video, a 50% savings was substantial. Today, BlazingCDN offers low-volume pricing starting at around $0.005/GB, making the bandwidth savings worth 100 times less today.

Beyond reduced savings, most codec proponents ignore the cost side of the equation. Before deploying a new codec, you have significant testing costs; if you have your own encoding pipeline, you have integration and testing costs. After deployment, you have the additional transcoding costs since you typically can’t drop one codec because you’re adopting another. The costs are all additive. You also have additional storage costs for the transcoded files and decreased caching efficiency at the edge because you may have to cache multiple versions of the encoded file for full coverage.

For VVC, almost 100% of TVs don't support HW VVC decoding and their SoCs are too weak to support it in software. For mobile users, again no HW support, so VVC videos will kill your battery.

Maybe 50 thousand or so Lunar Lake owners? Nope, that won't work for PC/laptop users either.

Looks like VVC will not be adopted in anything consumer any time soon or maybe ever.

Oh, what about the [warez] scene? Nothing. Completely ignored because the encoder is crazy slow.

[Blue_MiSfit]

I was fairly impressed with Spin Digital's live 9 (?) Mbps 4kp60 HDR VVC encode at NAB. It resoundingly outperformed NVENC AV1 and x265 at equal speed, despite using very fast settings and no tiles.

I didn't get too deep into the weeds about exactly how much CPU it was using, maybe 16 cores or something?

[GeoffreyA]

Originally Posted by birdie

Oh, what about the [warez] scene? Nothing. Completely ignored because the encoder is crazy slow.

There was a good example of "Last Night in Soho," one of the first VVC encodes. A high-quality QP20 specimen, it took the group about a week using vvenc medium; its mild loss of grain only evident if one compares against the BluRay. As a bonus, it sported USAC audio.

[Z2697]

Originally Posted by Blue_MiSfit

I was fairly impressed with Spin Digital's live 9 (?) Mbps 4kp60 HDR VVC encode at NAB. It resoundingly outperformed NVENC AV1 and x265 at equal speed, despite using very fast settings and no tiles.

I didn't get too deep into the weeds about exactly how much CPU it was using, maybe 16 cores or something?

Realtime 4K 60FPS on CPU? That's almost too good to be true.

[LigH]

@ birdie + GeoffreyA:

As much as I remember the last 2-3 decades, moviez pirates did not avoid encoding efforts to gain shareability while the average internet bandwidth was limited... having Gbps connections and TByte media today moves the goalpost. Burning a CD/DVD is out since many "Smart" TV sets have USB3 ports and rather compatible decoders for many established media formats.

[GeoffreyA]

Originally Posted by LigH

@ birdie + GeoffreyA:

As much as I remember the last 2-3 decades, moviez pirates did not avoid encoding efforts to gain shareability while the average internet bandwidth was limited... having Gbps connections and TByte media today moves the goalpost. Burning a CD/DVD is out since many "Smart" TV sets have USB3 ports and rather compatible decoders for many established media formats.

Yes. Not so much the encoding complexity, but its offering little gain over AV1 at the cost of less compatibility.

[benwaggoner]

Originally Posted by Blue_MiSfit

I was fairly impressed with Spin Digital's live 9 (?) Mbps 4kp60 HDR VVC encode at NAB. It resoundingly outperformed NVENC AV1 and x265 at equal speed, despite using very fast settings and no tiles.

I imagine it was using 64+ cores. Demos like this are normally using the fastest Xeon or EPYC processor they can find that'll run the demo. IIRC, MultiCoreWare were using at least 36 cores when they demoed live 4K HEVC software encoding ~7 years ago. Basically they use as many cores as they can scale to before the lower per-core performance winds up being a net negative. Xeon goes up to a max of 86 performance cores now.

[birdie]

Some interesting thoughts about the codec from Jan Ozer.

If VVC advocates want to make the codec relevant before hardware catches up, they need to follow the AV1 playbook. That means:

• Share Real-World Software Decode Performance: Meta’s 2023 post on Reels explained exactly how AV1 performed in practice: a 15% bitrate reduction at the same quality, 10% lower battery consumption, and 6% faster playback start time. They also shared performance results and the benchmarking effort needed to maintain high viewer QoE. That kind of transparency builds trust and sets expectations. VVC groups haven’t done this, and it shows. If companies are currently using VVC with software decode successfully, tell us about it. If not, say they're not, and quit alluding to a use case that doesn't exist.
  
• Launch a VVC Equivalent of VCAT. Meta’s Video Codec Analysis Tool (VCAT) will give the industry a shared, testable way to evaluate decoder performance across device classes. If VVC is being tested internally, no one’s publishing results. Without shared metrics, no one can benchmark or optimize around it.
  
• Fund an efficient, open-source decoder like dav1d. AOMedia funded dav1d to ensure AV1 playback worked on low-end phones and across platforms. It’s now the most widely used AV1 decoder, integrated into browsers, apps, and OS platforms. In contrast, VVC’s open-source decoder options are unoptimized and untested in the wild. When I shared in a LinkedIn post that I was testing VVC decoders and found them twice as CPU-intensive as AV1, the author claimed this was unfair because VVC software players are “not as optimized as the AV1 one.” I snickered, stifled my sarcastic response ("that's the freaking point"), and commented, “I can only test what’s available.” That’s the gap. That’s the problem.

[modus-ms325c]

what are they hiding!? why obfuscate codec adoption flaws with insane tech demos that we'll never get to see in a million years?
i mean, what year do they think we're in, 1994?

[ksec]

Originally Posted by birdie

Some interesting thoughts about the codec from Jan Ozer.

Until MC-IF and its partners focus on decode-first deployment

That is not the job of MC-IF. In fact there is no such equivalent to AOM on the VVC Camp. Although one could argue it *should* be doing that job.

Consider Qualcomm still dont have Hardware encode and Decode for VVC suggest something is holding up. May be hardware complexity. I remember the New Adreno VPU was suppose to go into the X Elite but that didn't happen. Some last minutes changes but everything since has been using the old VPU block.

Nothing about VPU from Qualcomm on Twitter. I guess we will have to wait and see. ( Its not like x266 is even here anyway )