Apple HEVC Stereo Video

[rwill]

Apple published this document:
https://developer.apple.com/av-foundation/HEVC-Stereo-Video-Profile.pdf

Interesting times ahead?

[benwaggoner]

It's been a while since I've encoded any 3D video, but the Apple spec looks like a straightforward implementation of the "good" way to do stereo video in HEVC. One eye is a normal stream, the other eye is encoded as an enhancement layer based on that. This will offer better quality than the old vertical split technique (Each frame has L and R embedded as the L and R halves on the screen with a 2:1 horizontal squeeze), with a lot less than 2x bitrate increase (one eye is an excellent predictor for what the other sees almost all the time).

Thinking back fondly on running the QuickTime VR tools inside the command line Macintosh Programmer's Workshop. On a ~110 MHz Mac of the era, It took minutes just to stitch 24 pictures into a non-moving panorama.

I did my first 3D video encode targeting VR glasses over 25 years ago now. I'm still waiting for a compelling use case for that consumption model.

[rwill]

It's been a while since I've encoded any 3D video, but the Apple spec looks like a straightforward implementation of the "good" way to do stereo video in HEVC.

Multiview HEVC seems to work more or less like AVC-MVC minus the DPB control hassle.


Thinking back fondly on running the QuickTime VR tools inside the command line Macintosh Programmer's Workshop. On a ~110 MHz Mac of the era, It took minutes just to stitch 24 pictures into a non-moving panorama.

When I was a kid I found "Star Trek: The Next Generation Interactive Technical Manual" kinda impressive on my 386. Well, 'I was a kid' is a good excuse anyway I think.

[benwaggoner]

Multiview HEVC seems to work more or less like AVC-MVC minus the DPB control hassle.
Exactly. Reasonably well understood from a compression perspective.

When I was a kid I found "Star Trek: The Next Generation Interactive Technical Manual" kinda impressive on my 386. Well, 'I was a kid' is a good excuse anyway I think.
Did that have a stereo 3D version?

[rwill]

Did that have a stereo 3D version?

Sadly not, but it used Quicktime VR. PCs with a CD-ROM drive enabled a lot of things back then.

[microchip8]

3D video is dead! Long live 3D video!

[ShortKatz]

I was already wondering how Apple is going to encode the 3D video for the Apple Vision Pro. Now I know. Interesting.

[benwaggoner]

Sadly not, but it used Quicktime VR. PCs with a CD-ROM drive enabled a lot of things back then.
Oh, YEAH, I remember it now! I did video encoding for several 90's Star Trek games, but I didn't work on that one.

TrueMotion and TrueMotion-S, IIRC, essentially On2 VP1 and VP2.

[Blue_MiSfit]

Is anyone aware of ANY shipping solution for encoding MV-HEVC? I presume Apple will add support for this to their APIs and encoding implementations at some point but...

[benwaggoner]

Is anyone aware of ANY shipping solution for encoding MV-HEVC? I presume Apple will add support for this to their APIs and encoding implementations at some point but...
I want to think there was at least one commercial implementation back in the 3D era. But maybe that was a preview release or something. I only ever worked with side-by-side, and haven't done any stereo 3D since maybe 2014.

[rwill]

Is anyone aware of ANY shipping solution for encoding MV-HEVC? I presume Apple will add support for this to their APIs and encoding implementations at some point but...

There are some solutions but none are 'shipping' as far as I know. So either wait or pony up that baksheesh for the x265 people...

[FranceBB]

I only ever worked with side-by-side

Same here, I've only ever worked with 3D side by side for our linear channels back then. I remember the struggle of manually checking that the two sides were actually aligned and that the eyes were not inverted. I used to spend time testing those contents on a 3D reference monitor with active glasses, purposely inverting the sides to see whether everything was correct or not etc etc etc.
Honestly I hated those days, so much stress, so much struggle with lots of manual checks involved.
Luckily 3D died an horrible death around the world and we, like many other broadcasters, closed our linear channels.
I would rather not go back to this madness.

[rwill]

New Dolby Vision Profile 20 uses MV-HEVC:

https://professionalsupport.dolby.com/s/article/What-is-Dolby-Vision-Profile?language=en_US

[Emulgator]

iPhone 15 will be able to use 2 of its 3 cams for 3D, presentable later using AppleVision Pro.
(I am not affiliated, a Huawei user myself)

Während des Wonderlust-Events erklärte Apples Greg Joswiak, dass Apple die Ultraweit- und Hauptkamera des iPhone Pro 15 nutze, um ein "dreidimensionales Video" zu erstellen.
Die aufgezeichneten Erinnerungen könne man später mit Apple Vision Pro nacherleben.
"Räumliche Videos geben Ihnen das Gefühl, dass Sie sich wieder in diesem Moment befinden", sagte Joswiak.

During the Wonderlust event, Apple's Greg Joswiak explained that Apple uses the ultra-wide and main camera of the iPhone Pro 15 to create a "three-dimensional video".
The recorded memories could later be relieved with Apple Vision Pro.
"Spatial videos give you the feeling that you are back at that moment," said Joswiak.

[benwaggoner]

New Dolby Vision Profile 20 uses MV-HEVC:

https://professionalsupport.dolby.com/s/article/What-is-Dolby-Vision-Profile?language=en_US
Good catch! I see a lot of the details are in other docs which I'm now trying to track down.
One important question is whether both views share the same non-backwards compatible image processing and dynamic metadata, which would be a useful and reasonable constraint, so there's no need to have two different LUTs running simultaneously.

[benwaggoner]

iPhone 15 will be able to use 2 of its 3 cams for 3D, presentable later using AppleVision Pro.
My understanding is that the iPhone 15 Pro stereo mode will be a different file format than the MV-HEVC stereo 3D videos. My best guess it is fundamentally a 2D recording with depth map where stereoscopic effects are rendered device side.