MPAI-EVC - enhancing video compression with portable ML models

[benwaggoner]

Another very interesting longer-term project in the encoding space is MPAI-EVC (https://mpai.community/?s=mpai-evc), an effort to combine updatable machine learning/AI models with EVC for improved compression efficiency and playback quality. While EVC is the base for this effort, the concepts could be easily applied to other codecs.

We've all seen exciting uses of ML in signal processing. That's how TVs have gotten so good at upscaling, among other things. And ML use is rapidly growing in encoding. However, use in decoding was hard even conceptually, as there wasn't any standard about how to make ML models portable, updatable, and well integrated into a codec.

MPAI-EVC is working to do exactly that, to couple bitstreams and ML models that can be used in a resource-constrained Profile @ Level framework required for broad implementation in low-cost decoders. It's early stages, with current work focusing on enhancing intra prediction.

Thoughts or insights?

[videoh]

Thoughts or insights? Sounds like spam. Are you involved?

[FranceBB]

I haven't read the document just yet, but neural networks are involved in all sort of operations nowadays so it's probably not gonna be a surprise to see their involvement in video compression as well in the future...

[ksec]

I could see this being used along side with LC-EVC? So you have LC and AI all working on top of EVC.

I skim read the document, ( I tend to look immediately for cost benefits analysis ) one of the thing I recognise was the possibility of hardware decoder cost saving. You will have an AI processor ( commonly known as NPU ) in your Smartphone regardless. So you are immediately saving some die space and cost. Although that is (likely) at the expense of power usage during decoding.

[benwaggoner]

Sounds like spam. Are you involved?
Not spam :sly:. https://forum.doom9.org/member.php?u=90717

I only heard about this effort right before I started this thread, but I'm very involved in codec and encoder evaluation, selection, and tuning.

[benwaggoner]

I could see this being used along side with LC-EVC? So you have LC and AI all working on top of EVC.
It certainly could work with LC-EVC. I do wish LC-EVC was called something else as the name implies it is connected with the EVC codec in some way.

I skim read the document, ( I tend to look immediately for cost benefits analysis ) one of the thing I recognise was the possibility of hardware decoder cost saving. You will have an AI processor ( commonly known as NPU ) in your Smartphone regardless. So you are immediately saving some die space and cost. Although that is (likely) at the expense of power usage during decoding.
Yeah, that's the challenge. We could imagine a LC-EVC type thing with ML enabled. But integrating the ML in-loop so it can be used for prediction is what would offer real improvements to encoding efficiency.

I'm not sure about integrating with general NPU features on a SoC. Those are in different blocks and using them together would add power usage and particularly latency. It's likely cleaner to have constrained ML processing contained in the decoder itself, if it is in-loop. Plus defining Profile @ Level would require precise definition of ML model requirements.

For "ML enhanced LC-EVC post-processing" latency wouldn't matter as it would operate on fully decoded frames and tight HW coupling wouldn't matter as much. LC-EVC overlays can even be generated outside of secure memory and then blended back with the decoded frame in secure memory. Postprocessing input could be just metadata, possibly enhanced with a heatmap of CUs and their QPs.