Any MV-HEVC encoders?

[benwaggoner]

Apple has announced that their forthcoming Vision Pro AR headset will support MV-HEVC encoding for stereoscopic 3D.

What MV-HEVC encoding options are people aware of? The HM reference encoder can, but it's really slow and not really set up for practical use.

The last round of Stereo 3D tools used for Blu-ray etcetera were H.264 and 1080 SDR only.

Anything practical out there, or in development? Extending x265 to support MV-HEVC seems like it would be pretty straightforward, but I don't recall hearing anything about anyone working on it.

[Blue_MiSfit]

I've heard that Ateme has a solution. I haven't touched it personally.

[benwaggoner]

Originally Posted by Blue_MiSfit

I've heard that Ateme has a solution. I haven't touched it personally.

Ah!

https://www.ateme.com/press/ateme-in...o-compression/

enable Spatial Computing, creating immersive and engaging 3D user experiences by efficiently transmitting content from different camera angles using Multiview High Efficiency Video Coding (MV-HEVC).

[kurkosdr]

Wait, does x264 even support MVC? MVC is not side-by-side H.264, in MVC the left-eye view is encoded as standard H.264 but the right-eye view is encoded as the difference from the left-eye view, which means the right-eye view is not H.264 and cannot be decoded as such (you can also do the opposite, aka right-eye view encoded as standard H.264 and left-eye view as difference, but that's far less common for some reason).

I don't remember x264 ever encoding MVC.

Since x265 is based on the x264 codebase, it's not a case of extending x265 to support MV-HEVC but writing multi-view coding from scratch for both MVC and MV-HEVC.

Considering stereoscopic 3D is still a niche and may stay that way, I wouldn't hold my breath, get a proprietary tool that already does the job (see comment above).

[Emulgator]

There was a non-public x264 patch regarding MVC, more than 6 years ago, IIRC.

[kurkosdr]

Originally Posted by Emulgator

There was a non-public x264 patch regarding MVC, more than 6 years ago, IIRC.

Link to source code or binary or else it didn't happen.

[Selur]

he was probably refering to: https://forum.doom9.org/showthread.php?t=168010 -> https://pastebin.com/qZ1xSmuc

[Emulgator]

Pegasys it was, yes.

[sturmen]

I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs: https://github.com/sturmen/SpatialMediaKit

[ShortKatz]

Originally Posted by sturmen

I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs: https://github.com/sturmen/SpatialMediaKit

Very cool!
By the way, you explicitly say "for Apple Silicon Macs", but I was able with no issue to build it also for my Intel Mac.

[sturmen]

Originally Posted by ShortKatz

Very cool!
By the way, you explicitly say "for Apple Silicon Macs", but I was able with no issue to build it also for my Intel Mac.

Glad to hear it! But… does it actually work on the Intel Mac?

[benwaggoner]

Originally Posted by sturmen

Glad to hear it! But… does it actually work on the Intel Mac?

The documentation says otherwise.

It's also ~3x faster than realtime on my M1 Pro, which suggests it's using their SoC HEVC encoder. While there were GPU HW encoders on Intel Macs, they would have been a lot more varied in functionality. Apple likely knew they were going to want MV-HEVC down the road as they were designing their computer SoCs.

[ShortKatz]

Originally Posted by sturmen

Glad to hear it! But… does it actually work on the Intel Mac?

It seems not. I've tested just now to split your test file and it only gives an error:

MV-HEVC decoding not supported on this device! Please try again on Apple Silicon and macOS 14+

But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?

[benwaggoner]

Originally Posted by ShortKatz

But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?

Not if it is leveraging the Silicon-specific hardware encoder.

[sturmen]

Originally Posted by ShortKatz

It seems not. I've tested just now to split your test file and it only gives an error:


But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?

That message comes from my code simply asking the computer if it supports decoding MV-HEVC: https://github.com/sturmen/SpatialMe....swift#L50-L55

Apple decides which Macs return "true" or "false" for this. It is conceivable Apple could have written pure software to encode/decode MV-HEVC, but I think they only wrote a hardware-accelerated pipeline and so that becomes the requirement.

[benwaggoner]

Originally Posted by sturmen

That message comes from my code simply asking the computer if it supports decoding MV-HEVC: https://github.com/sturmen/SpatialMe....swift#L50-L55

Apple decides which Macs return "true" or "false" for this. It is conceivable Apple could have written pure software to encode/decode MV-HEVC, but I think they only wrote a hardware-accelerated pipeline and so that becomes the requirement.

That makes sense. Apple doesn't have much motivation to spend a lot of effort optimizing for older x86-64 Macs.

[sturmen]

Originally Posted by benwaggoner

That makes sense. Apple doesn't have much motivation to spend a lot of effort optimizing for older x86-64 Macs.

By contrast to Apple: say what you will about Microsoft, but they have a sterling reputation when it comes to backcompat/backporting.

[benwaggoner]

Originally Posted by sturmen

By contrast to Apple: say what you will about Microsoft, but they have a sterling reputation when it comes to backcompat/backporting.

Very true. I worked there for six years, and "how will this new thing work on old corporate machines" was a constant consideration.

I strongly suspect that Apple's MV-HEVC implementation is leveraging specific features of the Silicon SoC hardware encoder. It can do faster than realtime MV-HEVC 1080p on my M1 Pro. Perhaps it even has a specific MV-HEVC mode? I can imagine getting it working on an Intel Mac could be 10x more effort, with a lot lower performance possible.

[davidt]

Originally Posted by sturmen

I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs

That's great, thank you very much! I've seen few similar tools, all closed source, even paid ones and I guess all of them work in the same way by wrapping AVFoundation APIs:
- https://blog.mikeswanson.com/spatial
- https://haloclinetech.com/
- https://apps.apple.com/us/app/spatialify/id6471922894

Originally Posted by benwaggoner

What MV-HEVC encoding options are people aware of? The HM reference encoder can, but it's really slow and not really set up for practical use.

Now the real question is what encoder does Disney use for the 4K 3D movies:
https://press.disneyplus.com/news/di...-entertainment

Using Dolby Vision and Multiview High Efficiency Video Coding (MV-HEVC), 3D movies on Disney+ will deliver exceptional UHD resolution in HDR, unfiltered and independent for each eye, and at a high frame rate for several titles – ensuring that the filmmakers’ creative intent is fully preserved and reflected.

I don't think it was Apple's one because hardware encoders aren't very good at quality/bitrate ratio...

Originally Posted by benwaggoner

Perhaps it even has a specific MV-HEVC mode?

Of course. Layers in MV-HEVC are part of the bitstream (of a single track), I can't think a way you can encode first layer in hardware and then add another in software.

[benwaggoner]

Originally Posted by davidt

Now the real question is what encoder does Disney use for the 4K 3D movies:
https://press.disneyplus.com/news/di...-entertainment
I don't think it was Apple's one because hardware encoders aren't very good at quality/bitrate ratio...

Ateme has the only released commercial MV-HEVC encoder I am aware of. The Apple one seems fine for hobbyist use, but I don't know that's it's functional to create a full, high quality adaptive bitrate ladder.

Of course. Layers in MV-HEVC are part of the bitstream (of a single track), I can't think a way you can encode first layer in hardware and then add another in software.

Hmm. This is probably feasible. The second layer is a lot like b-frame encoding. And you could reuse motion search and other features of the HW encoder as hints to the enhancement layer encoding.

It'd be a lot of code and a lot slower than a full HW implementation, and I don't know if it would be worth it. But it is certainly possible.