Versatile Video Coding (VVC) / H.266: HEVC successor

[birdie]

At IBC 2024:

4K 120fps VVC Playback with Ali266 on Snapdragon X Elite PC by Qualcomm and Alibaba
Qualcomm's Aytac Biber and Yan Ye from Alibaba will demonstrate the real-world benefits and capabilities of a decoder optimized for VVC running on Qualcomm's Snapdragon X Elite platform. This demonstration highlights the Snapdragon platform's capabilities in delivering 4K at 120 frames per second without dropping frames of skipping video. Qualcomm and Alibaba will illustrate the significant compression benefits of VVC, which allow for up to 50 percent reduction in video size and improvement of video start-up latency by 5 percent.

VVC and Web RTC Low Latency for Cloud Gaming by Ericsson
Ericsson's Per-Erik Brodin and Lukasz Litwic will showcase one the world's first integration of VVC with WebRTC for low latency video streaming in real time. Addressing the challenges of cloud gaming, Ericsson experts will demonstrate streaming a remotely rendered game to a web application running in a browser, and on mobile devices (both old and new). The application of this integration will generate an enhanced user experience with less buffering, increased game responsiveness and faster times accessing content.

IMAX, MediaKind, Fraunhofer HHI and Ericsson present their work on encoder optimisations and film grain processing
The encoder optimization papers include a means to optimize the trade-off between bit storage and transcoding in multi-profile video delivery systems, whilst the other uses a Large Modal Model (LMM) to successfully improve content-aware encoding decisions.

Next, film grain. A well-known problem which presents increasing challenges as codec efficiency improves - and it is not going away, as movie directors continue to choose film for artistic effect. One of our papers presents and illustrates the Film Grain Synthesis system employed in VVC, while the other aims to produce an objective perceptual assessment metric by training a model with examples from expert viewers’ subjective assessments. This is important as typical video coding assessment metrics do not work well when comparing synthesized film grain.

[Z2697]

The current form of FG synthesis is TRASH, using it in any case other than "contents designed for FG synthesis" (which apparently is not a real thing) is abuse.

[benwaggoner]

Originally Posted by Z2697

The current form of FG synthesis is TRASH, using it in any case other than "contents designed for FG synthesis" (which apparently is not a real thing) is abuse.

You mean the old one from AVC that has been recycled? AFAIK the only customer-facing use of it was a singular HD-DVD release circa 2007.

I was asking some questions about it at IBC, and wasn't able to get a straight answer for how and how well it works with HDR. If it is based on 709 code values, it could give some weird results with HDR.

It's hard to judge the quality of the synthesis without having better parameterization and removal algorithms up front. IMAX demoed a Film Grain Similarity metric at IBC that looks promising; having to do everything with subjective evaluation really slow things down.

[benwaggoner]

Originally Posted by birdie

At IBC 2024:

All the papers are now available for download here:

https://www.ibc.org/technical-papers...s/9995.article

[Z2697]

Originally Posted by benwaggoner

You mean the old one from AVC that has been recycled? AFAIK the only customer-facing use of it was a singular HD-DVD release circa 2007.

I was asking some questions about it at IBC, and wasn't able to get a straight answer for how and how well it works with HDR. If it is based on 709 code values, it could give some weird results with HDR.

It's hard to judge the quality of the synthesis without having better parameterization and removal algorithms up front. IMAX demoed a Film Grain Similarity metric at IBC that looks promising; having to do everything with subjective evaluation really slow things down.

I mean what is widely and easily available now, which just mean those in AV1 (there's no one-click solution on MPEG side as far as I know of).
The parameterization and removal algorithms up front is exactly what I mean. I know perceptual noise substitution is working in audio codecs pretty well but it's just far more complicated when it comes to video. As of now they are just putting some funny dancing dots on the screen and demolishing high frequency.
(Is there a way to make a static or less dynamic pattern out of AV1's FGS?)

[benwaggoner]

Originally Posted by Z2697

I mean what is widely and easily available now, which just mean those in AV1 (there's no one-click solution on MPEG side as far as I know of).
The parameterization and removal algorithms up front is exactly what I mean. I know perceptual noise substitution is working in audio codecs pretty well but it's just far more complicated when it comes to video. As of now they are just putting some funny dancing dots on the screen and demolishing high frequency.
(Is there a way to make a static or less dynamic pattern out of AV1's FGS?)

There's a lot that can be tweaked in AVFG1.

At IBC IMAX demonstrated their new Film Grain Similarity objective metric. I've not tested hands-on with it, but even if it is only mediocre, that's going to be a huge step forward in making FGS works. Right now we have to eyeball how similar grain is subjectively to know if it did a decent job.

[GeoffreyA]

This is reminscent of the advanced features in MPEG-4 ASP that were not very usable. Then, H.264 did some of those things right and in simpler fashion. Or, at least, that was the case with quarter pixel. GMC's ideas saw later use in AV1.

[ksec]

4K 120fps VVC Playback with Ali266 Software decoder on Snapdragon X Elite PC with no drop frames? The future is bright!

Any news about H.266 encoders?

[benwaggoner]

Originally Posted by ksec

4K 120fps VVC Playback with Ali266 Software decoder on Snapdragon X Elite PC with no drop frames? The future is bright!

Any news about H.266 encoders?

A number of live encoders were demonstrated at IBC, and some academic papers. Most of the academic work seems to be using VVEnc for testing now, and are getting competitive results out of it.

There was at least one 8Kp60 live encoder demonstrated, which was pretty amazing.

[benwaggoner]

Originally Posted by ksec

4K 120fps VVC Playback with Ali266 Software decoder on Snapdragon X Elite PC with no drop frames? The future is bright!

VVC is really impressive in its balance of compression efficiency improvement for increase in decoder complexity. It offers significantly better compression than AV1 with significantly less silicon.

Of course, given Moore's Law, the actual relative cost of decoders keep dropping generation by generation. It was only a few decades ago that $5/decoder was considered an acceptable license fee for a MPEG-2 decoder, given it was only a fraction of the cost.

Having a decoder that pushed a phone SoC cost up by even $2 would be a dealbreaker today for high volume low margin devices.

[LigH]

Modern video codecs are highly asymmetrical in general. The main effort in the encoder is searching for reduncancies that can be used to avoid intra coding. Reproducing content from previously known content in a decoder, in relation, hardly costs any CPU power.

[birdie]

BTW ffmpeg 7.1 with VVC decoding support elevated to official status has been released:

https://git.ffmpeg.org/gitweb/ffmpeg...7.1:/Changelog

There's also an xHE-AAC decoder though its implementation is incomplete.

[kurkosdr]

Originally Posted by benwaggoner

VVC is really impressive in its balance of compression efficiency improvement for increase in decoder complexity. It offers significantly better compression than AV1 with significantly less silicon.

Of course, given Moore's Law, the actual relative cost of decoders keep dropping generation by generation. It was only a few decades ago that $5/decoder was considered an acceptable license fee for a MPEG-2 decoder, given it was only a fraction of the cost.

The corollary is also true: Thanks to Moore's Law, silicon is relatively cheap nowadays, so it's preferable to throw some extra silicon to an AV1 encoder (compared to an VVC encoder) than deal with the licensing costs of VVC (with every one of the 10 patent-licensing entities). This explains the success of AV1 in services like Netflix.

[ksec]

Originally Posted by birdie

BTW ffmpeg 7.1 with VVC decoding support elevated to official status has been released:

https://git.ffmpeg.org/gitweb/ffmpeg...7.1:/Changelog

There's also an xHE-AAC decoder though its implementation is incomplete.

Not only VVC but also LC-EVC. Exciting times! ( But again lack of encoder for me to play with )

[LigH]

You can already include xeve and xevd in ffmpeg.

[excellentswordfight]

Originally Posted by benwaggoner

Of course, given Moore's Law, the actual relative cost of decoders keep dropping generation by generation.

Moore's Law at an economical level is deader then dead, the cost of a 5/4nm TSMC waffer has pretty much stayed the same for four years according to all estimates i've seen, and that goes for a lot other process nodes as well, and new advance nodes has worse transistor/usd than previous nodes. It used to be the case that you wanted to migrate to a smaller node cause it was cheaper (maybe not the most cutting edge one initially), but nowdays the cheapest nodes are really old at this point, and even there I dont think cost is dropping cause of inflation and lack of technical advancements.

There is a reason why we barely seen any drop in consumer electronic the last couple of years. Part is ofc the pandemic and inflation, but there is also just a matter of physics and reality of increase in complexity of new nodes.

It has been discussed for at least 10 years; that this was going to happen, and yes we are now there, and has been for a couple of years now.

"In the dynamic field of semiconductor technology, the ongoing discourse surrounding Moore’s Law has experienced a notable evolution, prominently featuring Zvi Or-Bach’s (MonolithIC 3D’ CEO) 2014 assertion. His statement that transistor cost scaling reached a pivotal juncture at 28 nm has attracted significant attention. The statement was recently validated by Milind Shah from Google in the Short Course (SC1.6 ) at IEDM 2023. The unequivocal statement, “Transistor cost scaling (0.7X) stalled at 28 nm and remains flat gen over gen,” confirms what was initially foreseen in earlier public viewpoints and blogs in 2014 predicting the conclusion of Moore’s Law."

https://www.semiconductor-digest.com...opped-at-28nm/

Even Nvidia flagged for this back in 2012 that this trend would lead to a huge spike in costs if we wanted to continue to scale performance with transistors count...

[benwaggoner]

Originally Posted by excellentswordfight

Moore's Law at an economical level is deader then dead, the cost of a 5/4nm TSMC waffer has pretty much stayed the same for four years according to all estimates i've seen, and that goes for a lot other process nodes as well, and new advance nodes has worse transistor/usd than previous nodes. It used to be the case that you wanted to migrate to a smaller node cause it was cheaper (maybe not the most cutting edge one initially), but nowdays the cheapest nodes are really old at this point, and even there I dont think cost is dropping cause of inflation and lack of technical advancements.

There is a reason why we barely seen any drop in consumer electronic the last couple of years. Part is ofc the pandemic and inflation, but there is also just a matter of physics and reality of increase in complexity of new nodes.

It has been discussed for at least 10 years; that this was going to happen, and yes we are now there, and has been for a couple of years now.

"In the dynamic field of semiconductor technology, the ongoing discourse surrounding Moore’s Law has experienced a notable evolution, prominently featuring Zvi Or-Bach’s (MonolithIC 3D’ CEO) 2014 assertion. His statement that transistor cost scaling reached a pivotal juncture at 28 nm has attracted significant attention. The statement was recently validated by Milind Shah from Google in the Short Course (SC1.6 ) at IEDM 2023. The unequivocal statement, “Transistor cost scaling (0.7X) stalled at 28 nm and remains flat gen over gen,” confirms what was initially foreseen in earlier public viewpoints and blogs in 2014 predicting the conclusion of Moore’s Law."

Yeah, things have really slowed down. It's amazing we had a run as long as we had; I remember working on an industrial marketing video in the mid 90's touting "deep sub micron" support - down to 400 nm!

For. compression it's not so bad, as we benefit a lot from multiple cores and SIMD instructions, so we about as big a generation-on-generation throughput improvement as anyone.

And smaller processes still give us better density and thermals, which should allow for more performance (less time for a signal to cross from one edge to the other of a SoC) and probably better FLOPS/watt and thus FLOPS/$ due to reduced power consumption for the processing itself and the cooling required.

Some process stability would still allow for better refinement and optimization for existing processes. When each architecture revision is coupled with a new process, that doesn't really leave that long to make sure the bang-per-transistor is really optimized, and means that backwards compatibility is a higher priority, as a new revision will be be out before existing software is deprecated.

If we knew we were going to be stuck at 1 nm for a decade plus, we could really think hard about the optimal long-term ISA and architectures, and be able to build hardware and software towards much less of a moving target.

I was at the Samsung Developer Conference yesterday, and they were really touting RISC-V for Tizen (their OS for pretty much everything but phones). Perhaps related?

[benwaggoner]

tl;dr we should be thinking about throughput per watt as a big factor in the ROI of new processes at least as much as $/transistor. For many applications the lifetime cost of power for computing and cooling will be a lot more than the cost of the processor. So we still have some runway of economic benefit from smaller processes even if the $/transistor starts rising significantly.