Any MV-HEVC encoders?

[benwaggoner]

Apple has announced that their forthcoming Vision Pro AR headset will support MV-HEVC encoding for stereoscopic 3D.

What MV-HEVC encoding options are people aware of? The HM reference encoder can, but it's really slow and not really set up for practical use.

The last round of Stereo 3D tools used for Blu-ray etcetera were H.264 and 1080 SDR only.

Anything practical out there, or in development? Extending x265 to support MV-HEVC seems like it would be pretty straightforward, but I don't recall hearing anything about anyone working on it.

[Blue_MiSfit]

I've heard that Ateme has a solution. I haven't touched it personally.

[benwaggoner]

I've heard that Ateme has a solution. I haven't touched it personally.
Ah!

https://www.ateme.com/press/ateme-introduces-its-next-generation-video-compression/

enable Spatial Computing, creating immersive and engaging 3D user experiences by efficiently transmitting content from different camera angles using Multiview High Efficiency Video Coding (MV-HEVC).

[kurkosdr]

Wait, does x264 even support MVC? MVC is not side-by-side H.264, in MVC the left-eye view is encoded as standard H.264 but the right-eye view is encoded as the difference from the left-eye view, which means the right-eye view is not H.264 and cannot be decoded as such (you can also do the opposite, aka right-eye view encoded as standard H.264 and left-eye view as difference, but that's far less common for some reason).

I don't remember x264 ever encoding MVC.

Since x265 is based on the x264 codebase, it's not a case of extending x265 to support MV-HEVC but writing multi-view coding from scratch for both MVC and MV-HEVC.

Considering stereoscopic 3D is still a niche and may stay that way, I wouldn't hold my breath, get a proprietary tool that already does the job (see comment above).

[Emulgator]

There was a non-public x264 patch regarding MVC, more than 6 years ago, IIRC.

[kurkosdr]

There was a non-public x264 patch regarding MVC, more than 6 years ago, IIRC.
Link to source code or binary or else it didn't happen.

[Selur]

he was probably refering to: https://forum.doom9.org/showthread.php?t=168010 -> https://pastebin.com/qZ1xSmuc

[Emulgator]

Pegasys it was, yes.

[sturmen]

I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs: https://github.com/sturmen/SpatialMediaKit

[ShortKatz]

I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs: https://github.com/sturmen/SpatialMediaKit

Very cool! :cool:
By the way, you explicitly say "for Apple Silicon Macs", but I was able with no issue to build it also for my Intel Mac.

[sturmen]

Very cool! :cool:
By the way, you explicitly say "for Apple Silicon Macs", but I was able with no issue to build it also for my Intel Mac.

Glad to hear it! But… does it actually work on the Intel Mac?

[benwaggoner]

Glad to hear it! But… does it actually work on the Intel Mac?
The documentation says otherwise.

It's also ~3x faster than realtime on my M1 Pro, which suggests it's using their SoC HEVC encoder. While there were GPU HW encoders on Intel Macs, they would have been a lot more varied in functionality. Apple likely knew they were going to want MV-HEVC down the road as they were designing their computer SoCs.

[ShortKatz]

Glad to hear it! But… does it actually work on the Intel Mac?

It seems not. I've tested just now to split your test file and it only gives an error:
MV-HEVC decoding not supported on this device! Please try again on Apple Silicon and macOS 14+

But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?

[benwaggoner]

But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?
Not if it is leveraging the Silicon-specific hardware encoder.

[sturmen]

It seems not. I've tested just now to split your test file and it only gives an error:


But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?

That message comes from my code simply asking the computer if it supports decoding MV-HEVC: https://github.com/sturmen/SpatialMediaKit/blob/7d4e7253cf30388ca0a34cad2fb107ce317adc9b/Sources/SpatialMediaKitTool/SpatialMediaKitTool.swift#L50-L55

Apple decides which Macs return "true" or "false" for this. It is conceivable Apple could have written pure software to encode/decode MV-HEVC, but I think they only wrote a hardware-accelerated pipeline and so that becomes the requirement.

[benwaggoner]

That message comes from my code simply asking the computer if it supports decoding MV-HEVC: https://github.com/sturmen/SpatialMediaKit/blob/7d4e7253cf30388ca0a34cad2fb107ce317adc9b/Sources/SpatialMediaKitTool/SpatialMediaKitTool.swift#L50-L55

Apple decides which Macs return "true" or "false" for this. It is conceivable Apple could have written pure software to encode/decode MV-HEVC, but I think they only wrote a hardware-accelerated pipeline and so that becomes the requirement.
That makes sense. Apple doesn't have much motivation to spend a lot of effort optimizing for older x86-64 Macs.

[sturmen]

That makes sense. Apple doesn't have much motivation to spend a lot of effort optimizing for older x86-64 Macs.

By contrast to Apple: say what you will about Microsoft, but they have a sterling reputation when it comes to backcompat/backporting.

[benwaggoner]

By contrast to Apple: say what you will about Microsoft, but they have a sterling reputation when it comes to backcompat/backporting.
Very true. I worked there for six years, and "how will this new thing work on old corporate machines" was a constant consideration.

I strongly suspect that Apple's MV-HEVC implementation is leveraging specific features of the Silicon SoC hardware encoder. It can do faster than realtime MV-HEVC 1080p on my M1 Pro. Perhaps it even has a specific MV-HEVC mode? I can imagine getting it working on an Intel Mac could be 10x more effort, with a lot lower performance possible.

[davidt]

I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs

That's great, thank you very much! I've seen few similar tools, all closed source, even paid ones and I guess all of them work in the same way by wrapping AVFoundation APIs:
- https://blog.mikeswanson.com/spatial
- https://haloclinetech.com/
- https://apps.apple.com/us/app/spatialify/id6471922894

What MV-HEVC encoding options are people aware of? The HM reference encoder can, but it's really slow and not really set up for practical use.

Now the real question is what encoder does Disney use for the 4K 3D movies:
https://press.disneyplus.com/news/disney-plus-on-apple-vision-pro-ushers-in-a-new-era-of-storytelling-innovation-and-immersive-entertainment
Using Dolby Vision and Multiview High Efficiency Video Coding (MV-HEVC), 3D movies on Disney+ will deliver exceptional UHD resolution in HDR, unfiltered and independent for each eye, and at a high frame rate for several titles – ensuring that the filmmakers’ creative intent is fully preserved and reflected.I don't think it was Apple's one because hardware encoders aren't very good at quality/bitrate ratio...

Perhaps it even has a specific MV-HEVC mode?
Of course. Layers in MV-HEVC are part of the bitstream (of a single track), I can't think a way you can encode first layer in hardware and then add another in software.

[benwaggoner]

Now the real question is what encoder does Disney use for the 4K 3D movies:
https://press.disneyplus.com/news/disney-plus-on-apple-vision-pro-ushers-in-a-new-era-of-storytelling-innovation-and-immersive-entertainment
I don't think it was Apple's one because hardware encoders aren't very good at quality/bitrate ratio...
Ateme has the only released commercial MV-HEVC encoder I am aware of. The Apple one seems fine for hobbyist use, but I don't know that's it's functional to create a full, high quality adaptive bitrate ladder.

Of course. Layers in MV-HEVC are part of the bitstream (of a single track), I can't think a way you can encode first layer in hardware and then add another in software.
Hmm. This is probably feasible. The second layer is a lot like b-frame encoding. And you could reuse motion search and other features of the HW encoder as hints to the enhancement layer encoding.

It'd be a lot of code and a lot slower than a full HW implementation, and I don't know if it would be worth it. But it is certainly possible.

[rwill]

Well if one does not need inter layer prediction it is pretty easy to merge two HEVC streams into one MV-HEVC stream...

Doing inter layer is pretty easy too if one can modify the reference picture list setup of layer > 0.

I wonder why there is no patch for x265 yet.

[benwaggoner]

Well if one does not need inter layer prediction it is pretty easy to merge two HEVC streams into one MV-HEVC stream...

Doing inter layer is pretty easy too if one can modify the reference picture list setup of layer > 0
And interlayer is pretty important! That gives ~20% instead of 100% added bitrate for stereo 3D.

I wonder why there is no patch for x265 yet.
Why? It's work that engineers would have to do instead of doing something else.

Among other things, x265 would need a way to input multiple synchronized sources to encode from. Getting interlayer working, including rate control. Getting all the signaling working and tested. The R&D is much more D than R, but it's still work for multiple developers for multiple months.

[kagami.h]

Ben, could you please look at https://forum.doom9.org/showthread.php?p=1997769#post1997769
I made a post here, but then it disappeared for some reason.

[rwill]

And interlayer is pretty important! That gives ~20% instead of 100% added bitrate for stereo 3D.


Well from my tests the second layer is more like 40%-70% of the first layer, depending on how clean the content is or if there is grain etc. For real 3D Cinema content anyway. Sure one can make the second view a little worse quality but this was not accepted here.


Why? It's work that engineers would have to do instead of doing something else.

Among other things, x265 would need a way to input multiple synchronized sources to encode from. Getting interlayer working, including rate control. Getting all the signaling working and tested. The R&D is much more D than R, but it's still work for multiple developers for multiple months.

Strange, I know of one HEVC production encoder that got extended to MV-HEVC where it took a single engineer around 2 weeks, including changing certain parts of the reference decoder for testing.

[kagami.h]

Well from my tests the second layer is more like 40%-70% of the first layer, depending on how clean the content is or if there is grain etc. For real 3D Cinema content anyway. Sure one can make the second view a little worse quality but this was not accepted here.

Right, they report 32% savings for MV-HEVC versus simulcast in the paper (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7258339).

[MSwanson]

Thought I'd weigh-in on some of this. For context, I've written a bit about what Apple is doing with MV-HEVC here: https://blog.mikeswanson.com/post/744337392062873601/spatial-video

Also, the Apple streams I've looked at show ~50% savings for the second layer. Interestingly, this doesn't appear to be the same encoder as what is currently provided in AVFoundation. Or at least it's a much more customized/future version. As far as I can tell, AVFoundation (Mac, iPhone, and Vision Pro) is encoding two separate HEVC streams and stitching them together with no inherent savings or inter-layer prediction. The AVFoundation encoder also doesn't seem to support multi-pass encoding or rate control beyond simple bitrate averages. I'm hoping that all of this improves with an upcoming release or perhaps at WWDC24.

[benwaggoner]

Thought I'd weigh-in on some of this. For context, I've written a bit about what Apple is doing with MV-HEVC here: https://blog.mikeswanson.com/post/744337392062873601/spatial-video

Also, the Apple streams I've looked at show ~50% savings for the second layer. Interestingly, this doesn't appear to be the same encoder as what is currently provided in AVFoundation. Or at least it's a much more customized/future version. As far as I can tell, AVFoundation (Mac, iPhone, and Vision Pro) is encoding two separate HEVC streams and stitching them together with no inherent savings or inter-layer prediction. The AVFoundation encoder also doesn't seem to support multi-pass encoding or rate control beyond simple bitrate averages. I'm hoping that all of this improves with an upcoming release or perhaps at WWDC24.
Yeah, the 32% efficiency improvement always seemed much lower than what should be achievable with a perceptually optimized encode; most times the eyes look pretty similar, with big regions often identical.

I'm pleased but unsurprised Apple has done better.