A Step By Step Guide To Encoding 3D Blu-rays To The Apple Vision Pro

[sturmen]

Introduction

This is the workflow I've been using to convert my 3D Blu-ray collection to Apple's new MV-HEVC codec. As of Feb 12, on the Apple Vision Pro these can only be played back in the native Files and Photos applications, but I am optimistic that other, more powerful player apps will be released in the coming months.

Hardware Used

Windows PC with a processor capable of decoding MVC (I believe this means an Intel processor that contains an iGPU that supports QuickSync Video)
Apple Silicon Mac
Blu-ray drive (MakeMKV-compatible)
Portable SSD with like 500+ GB of space (optional but recommended)


Software Used

MakeMKV (https://makemkv.com/)
FFmpeg (https://ffmpeg.org/)
FRIM Decoder (https://forum.doom9.org/showthread.php?t=169651)
SpatialMediaKit (https://github.com/sturmen/SpatialMediaKit)
MP4Box (https://github.com/gpac/gpac)


Steps
I recommend saving all files to an external SSD, and simply moving the drive between the computers at the section break.

Section 1: Windows

Rip your Blu-ray with MakeMKV. Make sure to include the stereo video track! It is unchecked by default and needs to be manually selected.
Use FFmpeg to extract the MVC bitstream and audio to separate files.
ffmpeg.exe -i 'Gravity_t00.mkv' -map 0:v -c:v copy -bsf:v h264_mp4toannexb "Gravity.h264" -map 0:a:0 -c:a pcm_s24le "Gravity_5.1_LPCM.mov"
Use FRIM Decode to split the MVC video track into left and right streams. This will take three separate PowerShell windows. Execute the commands in the following order in their own window:
FRIMDecode64.exe -i:mvc Gravity.h264 -o \\.\pipe\lefteye \\.\pipe\righteye
ffmpeg -f rawvideo -s 1920x1080 -r 23.976 -pix_fmt yuv420p -i \\.\pipe\lefteye -c:v libx265 -x265-params keyint=1:min-keyint=1:lossless=1 -vtag hvc1 -movflags +faststart Gravity_left.mov
ffmpeg -f rawvideo -s 1920x1080 -r 23.976 -pix_fmt yuv420p -i \\.\pipe\righteye -c:v libx265 -x265-params keyint=1:min-keyint=1:lossless=1 -vtag hvc1 -movflags +faststart Gravity_right.mov

Section 2: Mac

Combine left and right movies into an MV-HEVC stream.
spatial-media-kit-tool merge -l Gravity_left.mov -r Gravity_right.mov -q 50 --left-is-primary --horizontal-field-of-view 40 -o Gravity_MV-HEVC.mov
(Optional) Transcode the audio to AAC to save space.
ffmpeg -i Gravity_5.1_LPCM.mov -c:a libfdk_aac -b:a 384k Gravity_5.1_AAC.mov
Use MP4Box to remux the audio back into the MV-HEVC file.
mp4box -new -add Gravity_MV-HEVC.mov -add Gravity_5.1_AAC.mov Gravity_AVP.mov


Notes and Explanations, Presented in Q&A Format

Which eye is primary? This differs based on the source material. In FFmpeg, look at the input printout of the MKV file. You should see "stereo_mode" under Metadata. "block_lr" means left is primary, "block_rl" means right is primary.
What's a good quality value for -q? I don't actually have a good answer for this yet. q=50 is what Apple calls "medium quality" and it seems fine on my 5K Studio Display, but maybe compression artifacts will be glaringly obvious inside the Vision Pro. In my testing, q=50 yields about 2.5 GB per hour of 1080p stereoscopic video. Apple calls q=75 "high quality" and that yields approximately 4x bigger files, which are still about smaller than the size of the source H.264 MVC file.
Can this be done with only a Mac? I believe so: MakeMKV does support Macs, and I read about a tool called h264-tools (https://github.com/Vargol/h264-tools) that can decode the MVC stream into the separate files. I haven't used it though.
What if my film isn't 1080p at 23.976 fps? Change your FFmpeg commands accordingly. FRIMDecoder should print out the right resolution, dimension, and colorspace. (I420 is the same as yuv420p)
Why x265 lossless? I actually use hevc_nvenc lossless, but I didn't want this guide to rely on that. We don't want to use any lossy codecs until the very end, so that eliminates a bunch of codecs, and H.264/HEVC have the advantage of being directly readable by spatial-media-kit-tool, unlike other popular lossless codecs like FFV1 or HuffyYUV. And the final tiebreaker is: lossless H.264 doesn't play in QuickTime whereas HEVC does (useful for spot-checking), so HEVC is the winner.
Can spatial-media-kit-tool work on anything other than an Apple Silicon Mac? No. It's actually a pretty thin wrapper around the proper macOS APIs that actually do all the heavy lifting for encoding. We'll have to wait x265 to add MV-HEVC support before we'll see this on any other OS.
Why MOV containers? This is again driven by compatibility with the QuickTime framework, which spatial-media-kit-tool heavily relies on. The audio file will also carry through the original chapters from the original MKV into the final MP4box mux, which is nice.
Why libfdk_aac with a bitrate of 384 kbps? According to Wikipedia, 384 is "transparent" for 5.1. I've also encountered discs that have 7.1, where I figured 512 kbps ought to be enough. I use libfdk because it supports up to 7.1 channels. If you have 5.1, I recommend you try encoding with the aac_at encoder instead, as it's supposedly even better for the same bitrate.
Why MP4box for the final mux? FFmpeg does not preserve the MV-HEVC atoms. See the GitHub readme of SpatialMediaKit.

[benwaggoner]

Good stuff!

I note that one might as well set --keyint 1 for the lossless x265. Long GOP doesn't save that many bits in lossless, and IDR-only can encode a lot faster and decode somewhat faster.

[sturmen]

Good stuff!

I note that one might as well set --keyint 1 for the lossless x265. Long GOP doesn't save that many bits in lossless, and IDR-only can encode a lot faster and decode somewhat faster.

As I mentioned in the appendix, I don't actually use libx265 so my memory on the syntax is a little hazy. It would be this, right?
-c:v libx265 -x265-params keyint=1:min-keyint=1:lossless=1

[benwaggoner]

I don't use ffmpeg for HEVC encoding myself, but it looks right.

[CleverTrousers]

So after all that - what app will you use to play the files?

[rwill]

So after all that - what app will you use to play the files?

I think the Apple Vision Pro comes with a Player. Or maybe Quicktime on an Apple Silicon device.

[sturmen]

So after all that - what app will you use to play the files?

Right now, you can play them through the Files app, or the Photos app. Neither are ideal, as they don't give you a beautiful theater environment like Apple TV does. But it's early days. Personally, I'm waiting excitedly for Infuse (https://community.firecore.com/c/platforms/vision/34) and SKYBOX (https://forum.skybox.xyz/d/2239-apple-vision-pro-support/2).

[benwaggoner]

They got my files working, thanks for the tips.

And I can make stereo 3D movies now, at least for local playback, using the Apple API.

[kagami.h]

Can this be done with only a Mac? I believe so: MakeMKV does support Macs, and I read about a tool called h264-tools (https://github.com/Vargol/h264-tools) that can decode the MVC stream into the separate files. I haven't used it though.

What about reference decoder JMVC (https://vcgit.hhi.fraunhofer.de/jvet/jmvc)? In theory it should be able to decode H.264 MVC.

[sturmen]

What about reference decoder JMVC (https://vcgit.hhi.fraunhofer.de/jvet/jmvc)? In theory it should be able to decode H.264 MVC.

I've settled on this workflow that works for me, but if you get a Mac-only workflow going I'm sure there are people who would appreciate a writeup!

[vadlerg]

On Windows PC BD3D2MK3D to convert 3D BDs or MKV to 3D SBS, TAB or FS MKV the best choice for me. (https://forum.doom9.org/showthread.php?t=177317)

[cbusillo]

Hey, I took this method and wrapped it in an easy to use script. I've been using it for a week or so. If you notice any bugs or have any suggestions, please let me know! The goal was to make it as easy as possible for people and also create a script that could be used in automated environments.

https://github.com/cbusillo/BD_to_AVP

[cbusillo]

What about reference decoder JMVC (https://vcgit.hhi.fraunhofer.de/jvet/jmvc)? In theory it should be able to decode H.264 MVC.

I found a Mac binary of this. While it did work, it was really slow. I found FRIMDecode64 through Wine to be three times faster.

[benwaggoner]

I found a Mac binary of this. While it did work, it was really slow. I found FRIMDecode64 through Wine to be three times faster.
Yeah, reference decoders, like reference encoders, aren't designed for practical real-world performance.

[cbusillo]

Yeah, reference decoders, like reference encoders, aren't designed for practical real-world performance.

I figured it was worth a try just in case it was as fast as Wine's translation layer. The process does work which is cool, I just figured it wasn't worth being Mac Native for 30% of the speed. :)

[benwaggoner]

I figured it was worth a try just in case it was as fast as Wine's translation layer. The process does work which is cool, I just figured it wasn't worth being Mac Native for 30% of the speed. :)
As speed would be the main benefit of being Mac native, nope!

Interesting that Macs can decode MV-HEVC but not MV-AVC. Or...have you tried on M-series Mac? They added a lot of video functionality that wasn't in the x86-64 models.

[cbusillo]

As speed would be the main benefit of being Mac native, nope!

Interesting that Macs can decode MV-HEVC but not MV-AVC. Or...have you tried on M-series Mac? They added a lot of video functionality that wasn't in the x86-64 models.

This was on my M2 Max. I guess there was no reason for Apple to support an old 3D format that seems to be only used on Blu-ray. MVC AVC up until now, would have been kind of pointless on the Apple hardware. It is neat that the 2D stream just works on non 3D stuff. This was the first time I really played with video encoding and decoding, so I learned a lot of basic stuff.

Edit: Ha, you literally wrote the book on compression and I just recently learned AVC is h264.

[benwaggoner]

This was on my M2 Max. I guess there was no reason for Apple to support an old 3D format that seems to be only used on Blu-ray. MVC AVC up until now, would have been kind of pointless on the Apple hardware. It is neat that the 2D stream just works on non 3D stuff. This was the first time I really played with video encoding and decoding, so I learned a lot of basic stuff.

Edit: Ha, you literally wrote the book on compression and I just recently learned AVC is h264.
So I did. Hard to believe the 2nd edition was fifteen years ago now. I keep hoping my publisher will ask me to do a third. Although I'm not sure what the market/audience would be like these days. In 2009 there were still a variety of commercial workstation compression tools. Now is much more automated in the cloud with a lot less scope for individual tuning.

It seems like it'd be easier to target an audience of 200 people who would all pay $1000 than for 4000 people who would pay $50. Of course, my employer may not want me to publish the sorts of things I know that people would pay $1000 for ;).

"Encoding and ladder design for premium content at global scale" wouldn't exactly leap off the shelves at Barnes & Noble.

[cbusillo]

Or one guy that pays $200,000 for that content. That guy is Netflix. Amazon might not like that.

[sturmen]

A new app (well, new to the Vision Pro, at least) claims to be able to play back MVC files directly on the AVP. The price is $30. https://x.com/4XVR_Player/status/1881156694539145658

[Z2697]

FRIM can use software decoding on not-intel CPUs.
x265 can do MV-HEVC encoding natively now. (you need standalone x265 CLI, with the compile-time feature enabled)