Stereoscopic 3D using MVC with x264 ?

[vsv]

"TDVision's patented technology has also been implemented as Intra-view
coding in the MVC MPEG-4 H.264 extension recommendation initiative by
comparing the difference between left and right channels. This eliminates
redundant information and optimizes the bandwidth required for a given
infrastructure such as cable, satellite, ATSC 1.0/2.0 and Blu-ray disc."
http://3dcinecast.blogspot.com/2009/01/tdvision-to-show-world-first-3d2d.html

A version of the reference software for MVC including both encoder and
decoder source code is publicly available too:
http://wftp3.itu.int/av-arch/jvt-site/2008_07_Hannover/JVT-AB207.zip

Panasonic, TDVision and third must be X264 ;)

[Dark Shikari]

It's insane to try to implement such a spec when they're still finding bugs in it every other week.

[vsv]

What better for x264 encoding with two views
side-by-side or over-under ?

[dukey]

i would say if you are going to encode 2 images in the stream
i would go with one image on top of the other as opposed to side by side. If you want to render the video with a 3d api direct x etc, side by side you'll run out of texture resolution for older cards, where as on top you won't :o

[Manao]

As far as I can tell, MVC is just AVC with a bunch of SEI messages. No new coding tools have been added (so your post is outdated). So you don't need to modify x264 internals, you just need to create those SEI messages properly.

[BeyondTheEyes]

Well, it's a little more than that: while it's true that MVC does not add new encoding tools (e.g. new macroblock modes) on top of AVC, MVC does specify new high-level syntax (from NAL units to slice headers) for inter view prediction (an MVC picture is composed of N >= 2 view components). Therefore I believe MVC could be supported by x264 core engine but (especially) DPB (e.g. reference picture lists) management would require specific adaptation changes.

[LoRd_MuldeR]

What better for x264 encoding with two views
side-by-side or over-under ?

Given that both views show a pretty similar picture, I'd say:
Interleave the views before encode and separate them after decode (at playback time).
That should give more efficient encoding and avoid ultra-long motion vectors...

[vsv]

As far as I can tell, MVC is just AVC with a bunch of SEI messages.


Interleave the views before encode and separate them after decode (at playback time).

Yes! Now we need to compare efficience of multiview encoding (MVC) vs over-under and side-by-side with
same stereo3D source.

[Zerofool]

Sorry for bringing that old topic up, but now that 3D Blu-Ray standard is finalized (using MVC), will you guys consider adding MVC support in x264 at some later time? First we'll have to wait because "The completed specification will be available shortly and provides individual manufacturers and content providers with the technical information and guidelines necessary to develop, announce and bring products to market pursuant to their own internal planning cycles and timetables". So it's finalized, but not out yet :). |sources: 1 (http://www.techpowerup.com/110796/_quot_Blu-ray_3D_quot_Expected_to_Reach_Consumers_in_2010_Los_Angeles.html) 2 (http://www.mtbs3d.com/cgi-bin/news.cgi) 3 (http://www.betanews.com/article/With-Avatar-close-at-hand-the-Bluray-3D-spec-is-launched/1261060196) 4 (http://www.engadget.com/2009/12/17/blu-ray-3d-specifications-finalized-your-ps3-is-ready/) 5 (http://www.forbes.com/feeds/businesswire/2009/12/17/businesswire132950591.html) |
There's an Annex H in H.264 standard (http://www.itu.int/rec/T-REC-H.264-200903-I/en) about MVC and the "Multiview High profile", but I guess you already know that ;).
Considering 2010 will be the year of Stereoscopic 3D, this will be a great addition to x264.

[shon3i]

will you guys consider adding MVC support in x264 at some later time?They should first finish that current x264 became full blu-ray compilant :P

[Biggiesized]

Can you imagine 3-D blocking artifacts?! Trippy!

[Zerofool]

Can you imagine 3-D blocking artifacts?! Trippy!

Yes, it's strange, but your brains deal with it better than in 2D, I've seen some old side-by-side XviDs ;). For low quality, blurry 3D H.264 encodes just head over to YouTube :p.

They should first finish that current x264 became full blu-ray compilant :P

Well, of course, but I'm not talking about BD compatibility, it's about efficient storage and playback of 3D videos on the PC. In all these news about MVC, everybody's mentioning about 50% more space needed for the second image (in most cases). And from my CRF test with current x264 builds in side-by-side and over/under modes (both almost identical in size) the results are 96-100% additional bitrate consummation compared to only the left or right video (which are again very close in size to each other). So for the second image we have additional size ~98% (current x264, no MVC) vs. ~50% (H.264 with MVC, on paper ;)).
If someone's interested I could post some results data, used settings and sources.
And yes, updated decoders will be needed, but obviously right now Nvidia is working with Arcsoft, Corel and CyberLink (devs of TotalMdedia Theater, WinDVD and PowerDVD), and AMD - with CyberLink. Both companies will host demonstrations of Blu-Ray 3D playback at the upcoming CES (HW Accelerated in the case of nv, no word about amd's). Most likely other decoders will soon follow.

[juGGaKNot]

VideoLeft = AVIsource("C:\x264\movie.avi", audio=false)
VideoRight = AVIsource("C:\x264\movie3D.avi", audio=false)
VideoStacked = StackHorizontal(VideoRight,VideoLeft)
ConvertToYV12(VideoStacked)

works for me

[jakor]

Given that both views show a pretty similar picture, I'd say:

Sorry to say that but your statement above is fundamentally wrong.
Given that two pictures represent the same scene - but even if the cameras are parallel - it is not possible to perform efficent MC between them. For example - with 2 views given - the ratio between two P frames - one from the same view - the next in time axis and the other - the same scene, but other view is 1:20.

[Blue_MiSfit]

I wouldn't say I've observed 2x more bitrate being necessary in the split-screen 3D test clips I've encoded for various HDTV manufacturers using x264. More, sure, but definitely not 2x more. For example, I did an encode of a segment from some really awful movie about dinosaurs and giant snakes attacking a city in 3D, and got quite good results at ~8mbps CBR for 1080p, with a 1 second buffer :p... streaming is a real bummer lols

~MiSfit

[BlackSharkfr]

I use x264 for encoding stereoscopic 3D content.
My #1 issue at the moment isn't really the bitrate or the increase of bitrate required for stereo video.

The main issue I have at the moment is media player not being able to detect the stereo video format automatically.
At the moment, watching stereo video systematically requires user input to find out what format is being used.

Many different stereo video formats are being used :
-Separate files for L and R views
-Dual streams L+R in single file
-line interleave (encoded as progressive)
-line interleave (encoded as interlaced)
-Stacked L+R views in single video frame
>-stacked horizontally
>-stacked vertically
>->- left eye view then right eye view
>->- right ee view then left eye view

or any ther possible way of storing two eye views in a video file.

This requires tagging the files/streams in a standard way, and then the stereoscopy enabled media player to reckognize these tags.
I have seen the stereoscopy tag for matroska files in mkvmerge-gui, but not only it is unsufficient (it only supports dual streams orhas a vague "both views in one stream") but no media player is capable of actually using them.


I am not a developer and i can't code, but i wish that the arrival of MVC will bring more attention to the issues encountered by stereo video users.

[moviefan]

One short question concerning this 3D stuff for Blu-rays as I find it very interesting: There's for example the movie "My Bloody Valentine 3D", is this movie based on MVC? I guess not, since the standard isn't in use for product sale right now correct? What is the difference between this MVC and 3D Blu-rays that are available by now?

[Reimar]

I am not a developer and i can't code, but i wish that the arrival of MVC will bring more attention to the issues encountered by stereo video users.

I don't think this kind of "nobody can be bothered to fix the small and easy issues, so let's make a really difficult solution"-thinking is really going to work.
If stereo doesn't work well with the current stuff it IMHO is because nobody cares, and MVC sure isn't going to help there.
And concerning bandwidth: I find it hard to imagine that MVC will do significantly better than encoding the views as successive frames and increasing the number of reference frames...

[Manao]

I find it hard to imagine that MVC will do significantly better than encoding the views as successive frames [...]That is exactly what MVC is intended for. As BlackSharkfr noticed, what is missing in his case is mostly signalization stuff (like AR, frame rate, overscan, but for 3D material), and MVC does just that, and nothing else. Originally, they did want to improve coding efficiency, but that would have broken compatibility and cost too much.

[BeyondTheEyes]

That is exactly what MVC is intended for. As BlackSharkfr noticed, what is missing in his case is mostly signalization stuff (like AR, frame rate, overscan, but for 3D material), and MVC does just that, and nothing else. Originally, they did want to improve coding efficiency, but that would have broken compatibility and cost too much.

I would like to clarify something, as already stated in a previous post (here (http://forum.doom9.org/showthread.php?p=1246638#post1246638)): MVC is not about SEI messages, but also specifies a new decoding process (cf. Annex H of the H.264/AVC recommendation) that enables the interview prediction of one view (for instance the right-eye view) from the so-called base view (in that example the left-eye view). A legacy H.264/AVC decoder (i.e. that is capable of decoding High profile Elementary Streams), even if some sort of signalization is present, will not be able to completely handle an MVC Elementary Stream (i.e. decode both views), since new NAL unit types will be present in that stream, and (among other things) a modified (in comparison with the one specify in legacy H.264/AVC profiles) process for reference picture lists will be signaled in that stream. While it's true that the currently specified MVC profiles do not specify any new encoding tools at the macroblock level beyond legacy H.264/AVC ones, MVC does specify a bunch of new stuffs compared with H.264/AVC. BTW, coding efficiency is also improved compared with simulcast or legacy interleaving methods, but obviously not by a very significant percentage. What is key is the backward compatibility of MVC with H.264/AVC (both for the video specification and the MPEG-2 TS specification), which will enable 3D Blu-ray titles to be decoded on legacy 2D Blu-ray players (their integrated H.264/AVC High profile decoder will only output the H.264/AVC compatible base view).

[BlackSharkfr]

One short question concerning this 3D stuff for Blu-rays as I find it very interesting: There's for example the movie "My Bloody Valentine 3D", is this movie based on MVC? I guess not, since the standard isn't in use for product sale right now correct? What is the difference between this MVC and 3D Blu-rays that are available by now?

Currently available Blu-Ray discs labelled "3D" are all anaglyph.
These discs come with coloured glasses (red/cyan, magenta/green or yellow/blue). The images on the discs have their colours permanently altered to match these glasses, and are then compressed as traditional BluRay discs. (they do not use MVC of course).
If you've got a proper 3D display which can display full colour images to both eyes like you can see at the cinema (you've seen AVATAR haven't you ?), it won't work.
Also you can't watch this disc in 2D either since the colours are permanently altered.

As beyond the eyes mentionned. the key point of MVC is not storing 3D images. We already know how to do it and it just works great through 99% of the current workflow, i do it myself with unmodified Megui/x264, unmodified mkvmerge(gui), unmodified yamb, and unmodified haali's matroska splitter or divx splitter. All you need is a 3D enabled player able to decode the final information and the user to say "hey this video is 3D, please activate the 3D mode" and there you go (although it would be great if that final step could be overcome).

The key issue of MVC is that MVC contains both 2D and 3D versions and is backwards compatible with 2D BluRay players. This means :
-people with 3D players/displays will get the full 3D quality for the full effect.
-people with 2D players/displays will get the full 2D quality just like if it was a 2D movie.
->with the same file/disc/stream

MVC will also certainly help 3D broadcast as it requires only 150% the bandwidth to transfer both 2D and 3D videos simultaneously in a single channel.
(whereas transferring both 2D and 3D separately would use two separate channels and requires 250~300% the bandwidth)


Now I can't code, and i don't expect the very skilled x264/VLC/MPC-HC/etc... coders to suddently support stereo video users just because 3 unknown guys kindly ask. I think they won't do it until they feel they need these features. My guess is that we'll have to wait for consumer use of MVC and future 3D bluray backups to make it happen.
I just hope it could come sooner than later.

[sethk]

I think the 3D onslaught will be in full force come CES 2010 in a few weeks, and I bet launch consumer level 3D hardware / software is a lot closer than people may think - i.e. 3D Blu-ray discs, players and compliant TVs will be announced at CES 2010 and available within months, rather than years. This is especially true if it turns out that the current PS3 slim can play back 3D. I could well be wrong, but I think they'd want to capitalize on the current 3D craze in theaters and transfer some of that excitement into the sales of new hardware (espcially Sony, since they have interests in the entire chain - movies, players, TVs).

[BlackSharkfr]

Well the Sony PS3 will be capable of playing 3D BluRays, sony made sure the specs would fit within the technical capabilities of the console. Now it just needs the software which sony will publish via firmware update when they are ready to sell their own 3D Bravia TVs.

The big question is : will the PS3 be able to hand over the full resolution images to the display ? Sony will have to give us some explanations because at the moment, the PS3 is limited to 1/2 resolution 3D due to the single hdmi v1.3 plug.

[sethk]

Well the Sony PS3 will be capable of playing 3D BluRays, sony made sure the specs would fit within the technical capabilities of the console. Now it just needs the software which sony will publish via firmware update when they are ready to sell their own 3D Bravia TVs.

The big question is : will the PS3 be able to hand over the full resolution images to the display ? Sony will have to give us some explanations because at the moment, the PS3 is limited to 1/2 resolution 3D due to the single hdmi v1.3 plug.

I thought the current model (known commonly as the PS3 slim) is HDMI 1.3, which supports full bandwidth transmissions over a single plug (340Mhz, 10.2Gbit TDMS transmission, same as 1.4 versus 165/4.95 on 1.2 and older). It does not technically support the official 3D over HDMI spec that was added in 1.4 but I thought that was more of a command / spec thing than a bandwidth issue?

[BlackSharkfr]

I thought the current model (known commonly as the PS3 slim) is HDMI 1.3, which supports full bandwidth transmissions over a single plug (340Mhz, 10.2Gbit TDMS transmission, same as 1.4 versus 165/4.95 on 1.2 and older). It does not technically support the official 3D over HDMI spec that was added in 1.4 but I thought that was more of a command / spec thing than a bandwidth issue?

Yes I already heard from someone else that the bandwidth was identical between hdmi 1.3 and 1.4. The question I do not know is whether the console could be firmware upgraded to add the 3D features of hdmi 1.4 into it's hdmi 1.3 output.

and additionally on the same theme :
If such a thing could ever be possible on a PS3, could it also be possible on a computer graphics card ?

[jakor]

there is AVC stereo profile. Intended to be the standard for two view video.

[lnatan25]

Yes I already heard from someone else that the bandwidth was identical between hdmi 1.3 and 1.4. The question I do not know is whether the console could be firmware upgraded to add the 3D features of hdmi 1.4 into it's hdmi 1.3 output.

and additionally on the same theme :
If such a thing could ever be possible on a PS3, could it also be possible on a computer graphics card ?
http://www.highdefdigest.com/news/show/Industry_Trends/HDMI/3D/HDMI_1.3_Will_Support_3D_After_All/4002,

Not optimal, but still...

[moviefan]

And what requirements do TVs have to fulfill?

[Willyfan]

PS3 will be compatible with 3D. Panasonic have a bluray player annunced for the first month of 2010. There is some BD 3D in authoring phase from Sony and Disney (not anaglyph, TRUE 3D). A lot of TV producer (like samsung) have 3D screens in production. So, I think that adding MVC in x264 can be a good idea...

[Stephen R. Savage]

Slightly off-topic, but what does it look like when watching a stereoscopic video with the left/right fields swapped? I'd check it out myself, but I lack any stereoscopic equipment.

[Dark Shikari]

Slightly off-topic, but what does it look like when watching a stereoscopic video with the left/right fields swapped? I'd check it out myself, but I lack any stereoscopic equipment.Z axis is inverted, I think. I did this by flipping my glasses backwards at an Avatar showing.

[Willyfan]

They should first finish that current x264 became full blu-ray compilant :P

Why x264 is not blu-ray compliant? Some times ago, I asked for slice support (a bluray compliant issue). Now, I read that there is a version with slices:

Date: Mon Dec 7 00:49:41 2009 -0800

Bring back slice-based threading support
Enabled with --sliced-threads
Unlike normal threading, adds no encoding latency.
Less efficient than normal threading, both performance and compression-wise.
Useful for low-latency encoding environments where performance is still important, such as HD videoconferencing.
Add --tune zerolatency, which eliminates all x264 encoder-side latency (no delayed frames at all).
Some tweaks to VBV ratecontrol and lookahead (in addition to those required by sliced threading).
Commit sponsored by a media streaming company that wishes to remain anonymous.


So, I want to try, but I think that x264 is compliant with bluray. Or not?
MVC can be the next step...

[nurbs]

I guess the NAL-HRD patch would have to be committed for full blu-ray compliance.

[nm]

Why x264 is not blu-ray compliant? Some times ago, I asked for slice support (a bluray compliant issue). Now, I read that there is a version with slices:

Date: Mon Dec 7 00:49:41 2009 -0800

Support for multiple slices with frame-based threading was actually added before that, back in August (http://git.videolan.org/?p=x264.git;a=commit;h=4d553edf178bf0ae01547731a48e1fb08c5cc1f4):

Mon, 31 Aug 2009 13:36:41 +0000 (06:36 -0700)

Multi-slice encoding support
Slicing support is available through three methods (which can be mixed):
--slices sets a number of slices per frame and ensures rectangular slices (required for Blu-ray). Overridden by either of the following options:
--slice-max-mbs sets a maximum number of macroblocks per slice.
--slice-max-size sets a maximum slice size, in bytes (includes NAL overhead).
Implement macroblock re-encoding support to allow highly accurate slice size limitation. Might be useful for other things in the future, too.

This is what you'll want to use when encoding Blu-rays, not sliced threads.

So, I want to try, but I think that x264 is compliant with bluray. Or not?
x264 still needs to be patched for NAL-HRD support, as nurbs said.

[Willyfan]

x264 still needs to be patched for NAL-HRD support, as nurbs said.

Many thanks for your post, very clear!
Do you know if the patch for NAL-HRD support exist?

[nm]

Many thanks for your post, very clear!
Do you know if the patch for NAL-HRD support exist?
It has existed since 2007. Current, generally used version is this one: http://komisar.gin.by/x.patch/last.used/x264_hrd_pd_interlace.16_r1369.diff
There's also a newer version (18) and a different implementation, but there may have been some problems with them.

Some Windows builds that include this patch are announced within the Current Patches thread (http://forum.doom9.org/showthread.php?t=130364). For example: http://forum.doom9.org/showthread.php?t=130364&page=142

Few notes on usage: http://mewiki.project357.com/wiki/X264_Settings#x264_hrd_pulldown..23_interlace

[Trahald]

This thread is 'Stereoscopic 3D using MVC with x264 ?' - Anything else is off topic.

[JohnPeterson]

What's the current status of 3D mkvs?

Now that amd is joining the 3d fray with its amd hd3d it seems like 3d popularity is heating up. And the powerdvd 10 mark ii update with mvc-dxva decoding has been available since this summer. Could we see 3d mkvs before the end of the year?

The mvc manual should be all that's needed to add mvc support to x264/mkv right? And now both the nvidia drivers and catalyst drivers have api's for video playback software (to sync the frames) that mpc-hc could utilize?

[kieranrk]

Patch welcome.

[popper]

Patch welcome.

talking about patches , how's your TS muxer patch coming ?, its been a long time now since i read anything about it's progress or lack of.

[kieranrk]

talking about patches , how's your TS muxer patch coming ?, its been a long time now since i read anything about it's progress or lack of.

Waiting to have access to an analyzer that actually gives me the right buffer data for audio and video. (Mainconcept have promised me they'll get theirs fixed) I got fed up of believing there were bugs when in fact the analyzer was wrong.

H.264 video on it's own works apart from the odd corner case.

[sneaker_ger]

Which odd corner case?

[kieranrk]

Which odd corner case?

When the transport bitrate is much higher than Rxn (transport buffer flow rate). It needs some code to write null packets so that the transport buffer doesn't overflow. However, if audio is present you might have a chance to write some audio packets to the buffer, which is why I need a working verifier to see the actual buffer flows.