Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th October 2023, 17:36   #1  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,893
Any MV-HEVC encoders?

Apple has announced that their forthcoming Vision Pro AR headset will support MV-HEVC encoding for stereoscopic 3D.

What MV-HEVC encoding options are people aware of? The HM reference encoder can, but it's really slow and not really set up for practical use.

The last round of Stereo 3D tools used for Blu-ray etcetera were H.264 and 1080 SDR only.

Anything practical out there, or in development? Extending x265 to support MV-HEVC seems like it would be pretty straightforward, but I don't recall hearing anything about anyone working on it.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 20th October 2023, 18:42   #2  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,997
I've heard that Ateme has a solution. I haven't touched it personally.
__________________
These are all my personal statements, not those of my employer :)
Blue_MiSfit is offline   Reply With Quote
Old 20th October 2023, 19:51   #3  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,893
Quote:
Originally Posted by Blue_MiSfit View Post
I've heard that Ateme has a solution. I haven't touched it personally.
Ah!

https://www.ateme.com/press/ateme-in...o-compression/

enable Spatial Computing, creating immersive and engaging 3D user experiences by efficiently transmitting content from different camera angles using Multiview High Efficiency Video Coding (MV-HEVC).
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 22nd November 2023, 21:46   #4  |  Link
kurkosdr
Registered User
 
Join Date: Aug 2009
Posts: 385
Wait, does x264 even support MVC? MVC is not side-by-side H.264, in MVC the left-eye view is encoded as standard H.264 but the right-eye view is encoded as the difference from the left-eye view, which means the right-eye view is not H.264 and cannot be decoded as such (you can also do the opposite, aka right-eye view encoded as standard H.264 and left-eye view as difference, but that's far less common for some reason).

I don't remember x264 ever encoding MVC.

Since x265 is based on the x264 codebase, it's not a case of extending x265 to support MV-HEVC but writing multi-view coding from scratch for both MVC and MV-HEVC.

Considering stereoscopic 3D is still a niche and may stay that way, I wouldn't hold my breath, get a proprietary tool that already does the job (see comment above).

Last edited by kurkosdr; 22nd November 2023 at 22:21.
kurkosdr is offline   Reply With Quote
Old 23rd November 2023, 00:39   #5  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,738
There was a non-public x264 patch regarding MVC, more than 6 years ago, IIRC.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Old 24th November 2023, 16:16   #6  |  Link
kurkosdr
Registered User
 
Join Date: Aug 2009
Posts: 385
Quote:
Originally Posted by Emulgator View Post
There was a non-public x264 patch regarding MVC, more than 6 years ago, IIRC.
Link to source code or binary or else it didn't happen.
kurkosdr is offline   Reply With Quote
Old 24th November 2023, 16:42   #7  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,495
he was probably refering to: https://forum.doom9.org/showthread.php?t=168010 -> https://pastebin.com/qZ1xSmuc
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 24th November 2023, 19:31   #8  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,738
Pegasys it was, yes.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Old 30th January 2024, 02:59   #9  |  Link
sturmen
Registered User
 
Join Date: May 2016
Posts: 8
I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs: https://github.com/sturmen/SpatialMediaKit
sturmen is offline   Reply With Quote
Old 1st February 2024, 23:31   #10  |  Link
ShortKatz
Registered User
 
Join Date: Aug 2018
Location: Germany
Posts: 134
Quote:
Originally Posted by sturmen View Post
I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs: https://github.com/sturmen/SpatialMediaKit
Very cool!
By the way, you explicitly say "for Apple Silicon Macs", but I was able with no issue to build it also for my Intel Mac.

Last edited by ShortKatz; 1st February 2024 at 23:43.
ShortKatz is offline   Reply With Quote
Old 2nd February 2024, 13:30   #11  |  Link
sturmen
Registered User
 
Join Date: May 2016
Posts: 8
Quote:
Originally Posted by ShortKatz View Post
Very cool!
By the way, you explicitly say "for Apple Silicon Macs", but I was able with no issue to build it also for my Intel Mac.
Glad to hear it! But… does it actually work on the Intel Mac?
sturmen is offline   Reply With Quote
Old 3rd February 2024, 08:19   #12  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,893
Quote:
Originally Posted by sturmen View Post
Glad to hear it! But… does it actually work on the Intel Mac?
The documentation says otherwise.

It's also ~3x faster than realtime on my M1 Pro, which suggests it's using their SoC HEVC encoder. While there were GPU HW encoders on Intel Macs, they would have been a lot more varied in functionality. Apple likely knew they were going to want MV-HEVC down the road as they were designing their computer SoCs.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 3rd February 2024, 20:59   #13  |  Link
ShortKatz
Registered User
 
Join Date: Aug 2018
Location: Germany
Posts: 134
Quote:
Originally Posted by sturmen View Post
Glad to hear it! But… does it actually work on the Intel Mac?
It seems not. I've tested just now to split your test file and it only gives an error:
Quote:
MV-HEVC decoding not supported on this device! Please try again on Apple Silicon and macOS 14+
But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?

Last edited by ShortKatz; 3rd February 2024 at 21:04.
ShortKatz is offline   Reply With Quote
Old 4th February 2024, 04:34   #14  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,893
Quote:
Originally Posted by ShortKatz View Post
But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?
Not if it is leveraging the Silicon-specific hardware encoder.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 4th February 2024, 15:03   #15  |  Link
sturmen
Registered User
 
Join Date: May 2016
Posts: 8
Quote:
Originally Posted by ShortKatz View Post
It seems not. I've tested just now to split your test file and it only gives an error:


But the necessary AVFoundation APIs should also work on an Intel Mac, shouldn't it?
That message comes from my code simply asking the computer if it supports decoding MV-HEVC: https://github.com/sturmen/SpatialMe....swift#L50-L55

Apple decides which Macs return "true" or "false" for this. It is conceivable Apple could have written pure software to encode/decode MV-HEVC, but I think they only wrote a hardware-accelerated pipeline and so that becomes the requirement.
sturmen is offline   Reply With Quote
Old 10th February 2024, 20:20   #16  |  Link
davidt
Registered User
 
Join Date: Jan 2019
Posts: 1
Quote:
Originally Posted by sturmen View Post
I glued together Apple’s AVFoundation APIs to make an encoder/decoder for Apple Silicon Macs
That's great, thank you very much! I've seen few similar tools, all closed source, even paid ones and I guess all of them work in the same way by wrapping AVFoundation APIs:
- https://blog.mikeswanson.com/spatial
- https://haloclinetech.com/
- https://apps.apple.com/us/app/spatialify/id6471922894

Quote:
Originally Posted by benwaggoner View Post
What MV-HEVC encoding options are people aware of? The HM reference encoder can, but it's really slow and not really set up for practical use.
Now the real question is what encoder does Disney use for the 4K 3D movies:
https://press.disneyplus.com/news/di...-entertainment
Quote:
Using Dolby Vision and Multiview High Efficiency Video Coding (MV-HEVC), 3D movies on Disney+ will deliver exceptional UHD resolution in HDR, unfiltered and independent for each eye, and at a high frame rate for several titles – ensuring that the filmmakers’ creative intent is fully preserved and reflected.
I don't think it was Apple's one because hardware encoders aren't very good at quality/bitrate ratio...

Quote:
Originally Posted by benwaggoner View Post
Perhaps it even has a specific MV-HEVC mode?
Of course. Layers in MV-HEVC are part of the bitstream (of a single track), I can't think a way you can encode first layer in hardware and then add another in software.
davidt is offline   Reply With Quote
Old 12th February 2024, 20:41   #17  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,893
Quote:
Originally Posted by davidt View Post
Now the real question is what encoder does Disney use for the 4K 3D movies:
https://press.disneyplus.com/news/di...-entertainment
I don't think it was Apple's one because hardware encoders aren't very good at quality/bitrate ratio...
Ateme has the only released commercial MV-HEVC encoder I am aware of. The Apple one seems fine for hobbyist use, but I don't know that's it's functional to create a full, high quality adaptive bitrate ladder.

Quote:
Of course. Layers in MV-HEVC are part of the bitstream (of a single track), I can't think a way you can encode first layer in hardware and then add another in software.
Hmm. This is probably feasible. The second layer is a lot like b-frame encoding. And you could reuse motion search and other features of the HW encoder as hints to the enhancement layer encoding.

It'd be a lot of code and a lot slower than a full HW implementation, and I don't know if it would be worth it. But it is certainly possible.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 16th February 2024, 22:36   #18  |  Link
kagami.h
Registered User
 
Join Date: Feb 2024
Posts: 4
Ben, could you please look at https://forum.doom9.org/showthread.p...69#post1997769
I made a post here, but then it disappeared for some reason.
kagami.h is offline   Reply With Quote
Old 26th March 2024, 05:50   #19  |  Link
MSwanson
Registered User
 
Join Date: Mar 2024
Location: Sammamish, WA
Posts: 1
Thought I'd weigh-in on some of this. For context, I've written a bit about what Apple is doing with MV-HEVC here: https://blog.mikeswanson.com/post/74.../spatial-video

Also, the Apple streams I've looked at show ~50% savings for the second layer. Interestingly, this doesn't appear to be the same encoder as what is currently provided in AVFoundation. Or at least it's a much more customized/future version. As far as I can tell, AVFoundation (Mac, iPhone, and Vision Pro) is encoding two separate HEVC streams and stitching them together with no inherent savings or inter-layer prediction. The AVFoundation encoder also doesn't seem to support multi-pass encoding or rate control beyond simple bitrate averages. I'm hoping that all of this improves with an upcoming release or perhaps at WWDC24.
MSwanson is offline   Reply With Quote
Old 27th March 2024, 23:58   #20  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,893
Quote:
Originally Posted by MSwanson View Post
Thought I'd weigh-in on some of this. For context, I've written a bit about what Apple is doing with MV-HEVC here: https://blog.mikeswanson.com/post/74.../spatial-video

Also, the Apple streams I've looked at show ~50% savings for the second layer. Interestingly, this doesn't appear to be the same encoder as what is currently provided in AVFoundation. Or at least it's a much more customized/future version. As far as I can tell, AVFoundation (Mac, iPhone, and Vision Pro) is encoding two separate HEVC streams and stitching them together with no inherent savings or inter-layer prediction. The AVFoundation encoder also doesn't seem to support multi-pass encoding or rate control beyond simple bitrate averages. I'm hoping that all of this improves with an upcoming release or perhaps at WWDC24.
Yeah, the 32% efficiency improvement always seemed much lower than what should be achievable with a perceptually optimized encode; most times the eyes look pretty similar, with big regions often identical.

I'm pleased but unsurprised Apple has done better.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Reply

Tags
multiview, mv-hevc, stereo 3d, stereoscopic 3d

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:49.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.