Any free MVC encoder yet?

[rica]

Guys, appreciate to you all but i've missed all those things while i 've been busy to help HW/ SW vendors on 3D :)
I hope they will help us someday. :rolleyes:

[Dark Shikari]

Sweet! So you mean they should be in revision 1835?More like 1861.

[BigPines]

More like 1861.
I am really looking forward to it. I will wait to do any other 3D work until this is ready because I think it is that important. I think a lot of people don't yet understand the importance of Frame Sequential. The quality benefits are huge and I think this is eventually going to be THE way of storing 3D content outside of Blu-ray disc. As a side note, I tired a sample Frame Sequential video on my PS3 and it plays back 2D on a non-3D display. That's a nice touch!

Mike

[BlackSharkfr]

This sounds more and more interesting.
I have some 3D content at hand so I'll be very happy to test it out.

However I won't convert my whole 3D video library to frame sequential just yet. I see x264 frame sequential as a milestone on the long road to having a new proper 3D storage and broadcast format.
I have yet be see proper reliable playback, frame sequential is widely known to be unreliable and requires the additional metadata to identify the frames. x264 framepacking data is too recent, no playback software supports it yet. My 3D displays aren't frame sequential and I need the playback software to convert the frame format on the fly.

To my user's eyes, MVC still remains the gold standard I wish I could use. It's the only format to my knowledge that befefits from :
-the 50% compression improvement
-the 2D player and display backwards compatibility without decoding the other eye view
and
-has wide industry support thanks to BluRay 3D

[BigPines]

To my user's eyes, MVC still remains the gold standard I wish I could use. It's the only format to my knowledge that befefits from :
-the 50% compression improvement
-the 2D player and display backwards compatibility without decoding the other eye view
and
-has wide industry support thanks to BluRay 3D
I would be very interested in hearing about the reliability issues of Frame Sequential. If done correctly, it seems like it should work well.

From my understanding, Frame Sequential has the same compression improvement.

At least with the PS3, I have backwards compatibility. This is really easy if you think about it, the player just suppresses the alternate frames and changes the framerate accordingly. Then poof, you have normal 2D!

You have a point on industry support but MKV didn't have any industry support when it took off either.

Let's talk about the benefits of frame sequential:

One of the major benefits of frame sequential is it is contained in one file. I personally don't want multiple streams to keep track of for each film.

Frame Sequential should also be able to be streamed via traditional media servers. I doubt we will ever see that for MVC.

x264 framepacking data is too recent, no playback software supports it yet. My 3D displays aren't frame sequential and I need the playback software to convert the frame format on the fly.
I think most people are going to need the player to detect and appropriately convert the stream. Isn't the x264 flag a standard? Won't my PS3 detect it? I just assumed it would. If it won't, then that will be a problem. The sample I used was a video game trailer and it worked great. However, I don't know how it was flagged.

EDIT: I just saw your post (I assume it is yours) here http://www.avsforum.com/avs-vb/showthread.php?p=19768645&posted=1#post19768645 "The PS3 frame sequential videos are not raw frame sequential, they use a proprietary marking system implemented by Sony for this purpose. I do not know any software that can produce it."

That is very disappointing. If I could get around this, I would be in business. :(

Mike

[BlackSharkfr]

That was indeed me at AVS forums.

Raw frame sequential has a terrible history of unreliability, sync problems and eye inversion issues when transmitting picture from a computer to a display, do a small search about the issues with cheap 3D DLP-link projectors, you'll quickly find people trying to find fixes to get their sync working properly.
You could kind of get it to work by defining some kind of convention like all the odd frames are left and even frames are right but in case of any missed frame, corrupt data or whatever and the eye views would swap. It's really a horrible mess and I do not want that nightmare to happen again with video storage.
If you want something reliable, you really need the frames to be paired and marked for each eye, which is (I believe) what the x264 frame packing #5 option tries to do.

I did not read the MVC spec, but I remember reading a Fraunhofer PDF stating that MVC could be stored in a single file too. We're used to having mp4 or mkv files with multiple audio and/or video tracks inside them. It would be very easy to store MVC into these files, the 2D video data would be the primary video stream and the 3D delta would be stored a secondary video stream inside the same file. I have not seen the PS3 video files, I couldn't find anyone uploading these file on the internet. I even wonder if these are really frame sequential of if they wouldn't be disguised MVC.
If you have them I'd love to get a copy to test them.

The separate 2D+delta odd/even numbered m2ts files thing is the way BluRay works. It's not very surprising, we're used to having that kind of crap on retail discs, DVDs have multiple VOBs, and 2D BluRay discs sometimes split the movie into multiple m2ts files. I don't know the exact reason for this but I guess it's been done that way to ensure backwards compatibility with 2D hardware BluRay players that could potentially fail to load and parse a very high bitrate MVC file.

[BigPines]

Well crap. I guess I was getting excited for nothing. So I guess I'll go SBS like most everybody else. Seems a shame since I will lose 1/2 my resolution. If this is the case, I will probably not embrace 3D at all. I'd rather watch full res 2D than crappy 1/2 res 3D. I have to be able to retain 1080p24 or I am not really interested. It sounds like SBS Full is the only way to do that and then my files will be 50% larger and I am very limited on what player will accept it. I'm pretty sure the PS3 won't accept it. What a shame.

Mike

[rica]

I wanted to point out the common confusion here again:

http://www.avsforum.com/avs-vb/showpost.php?p=19770090&postcount=47

_ _ _ _ _

[BigPines]

I wanted to point out the common confusion here again:

http://www.avsforum.com/avs-vb/showpost.php?p=19770090&postcount=47

_ _ _ _ _
Nope. I don't think you are understanding. If all the odd frames are left-eye and all the even frames are right-eye and the framerate is doubled and the appropriate frame packing flag is set in x264, you have encoded Frame Sequential H264. This is what I wanted but I wanted something that the PS3 could handle.

Check out the video here for an example of this: http://poseidon.dl.playstation.net/cdn/UP9000/NPUL00108_00/FREE_CONTENT6h5UXVMKmPkL0JiN7yIY/E320103DSizzle_Trailer-3D720.MP4?product=0084&country=us

Mike

[rica]

Nope. I don't think you are understanding. If all the odd frames are left-eye and all the even frames are right-eye and the framerate is doubled and the appropriate frame packing flag is set in x264, you have encoded Frame Sequential H264. This is what I wanted but I wanted something that the PS3 could handle.

Check out the video here for an example of this: http://poseidon.dl.playstation.net/cdn/UP9000/NPUL00108_00/FREE_CONTENT6h5UXVMKmPkL0JiN7yIY/E320103DSizzle_Trailer-3D720.MP4?product=0084&country=us

Mike

I haven't watched the video you uploaded but as far as i understand, you are trying to re-code to interlaced format because your display didn't understand frame packed format which your PS3 streaming and instead of using a foolish converter box you want to re-encode to interlaced format.
You will loose the half of the video again:

http://www.avsforum.com/avs-vb/showpost.php?p=19770823&postcount=52

And sorry but i still don't understand why you will need a PS3 if you have an HTPC?

[nm]

Check out the video here for an example of this: http://poseidon.dl.playstation.net/cdn/UP9000/NPUL00108_00/FREE_CONTENT6h5UXVMKmPkL0JiN7yIY/E320103DSizzle_Trailer-3D720.MP4?product=0084&country=us

That file uses the same standard frame packing SEI (45) as x264 does. The only difference is that the stream has the SEI before every frame (with current_frame_is_frame0 flag set for each other frame) while x264 only inserts the message once for each GOP.

So go ahead and try --frame-packing 5 with x264. It might work with PS3 if your linked file plays right. If it doesn't work, complain to Sony and in the meantime maybe patch x264 for yourself to produce frame-packing SEIs that the decoder expects.

I haven't watched the video you uploaded but as far as i understand, you are trying to re-code to interlaced format

No. This is about encoding full frames of two views one after another, doubling the framerate of the stream. SEI messages are added to signal the decoder which frame belongs to which view. Maybe you should read this whole thread again...

[rica]

No. This is about encoding full frames of two views one after another, doubling the framerate of the stream. SEI messages are added to signal the decoder which frame belongs to which view. Maybe you should read this whole thread again...

Would you give me the exact link directly? This doesn't seem to me logical.

[LoRd_MuldeR]

Would you give me the exact link directly? This doesn't seem to me logical.

http://forum.doom9.org/showthread.php?p=1464365#post1464365

[rica]

http://forum.doom9.org/showthread.php?p=1464365#post1464365

OK, thanks.
I have to do my homework.

[BigPines]

Would you give me the exact link directly? This doesn't seem to me logical.
Just download the video I linked to and play it or step through it one frame at a time. It will become obvious what is happening.

Mike

[BigPines]

That file uses the same standard frame packing SEI (45) as x264 does. The only difference is that the stream has the SEI before every frame (with current_frame_is_frame0 flag set for each other frame) while x264 only inserts the message once for each GOP.

So go ahead and try --frame-packing 5 with x264. It might work with PS3 if your linked file plays right. If it doesn't work, complain to Sony and in the meantime maybe patch x264 for yourself to produce frame-packing SEIs that the decoder expects.
Thanks for looking into this! Sounds like I may be back in business.

Yeah, I will give it a shot. I have been too busy to play lately but I'll try to do some experiments tonight.

Mike

[BigPines]

And sorry but i still don't understand why you will need a PS3 if you have an HTPC?
I don't have an HTPC.

I think the more appropriate question is why would I need an HTPC if my PS3 can do everything I want?

Mike

[mariner]

Thanks for looking into this! Sounds like I may be back in business.

Yeah, I will give it a shot. I have been too busy to play lately but I'll try to do some experiments tonight.

Mike

Greetings BigPines.

Would like to know if the PS3 could handle this as well:

Name: 700_2D3D_interleave.mkv
Size: 105.14MB
Description: 1920x1080/60p 2D to 3D Interleave
http://www.sendspace.com/file/pxhas6

Many thanks for testing.

[sneaker_ger]

The PS3 cannot read Matroska files in the first place.

[mariner]

Touche, sneaker_ger.

[BigPines]

The PS3 cannot read Matroska files in the first place.
Ahhh, but through PS3 Media Server it should be able to (or I can just remux it to M2TS). I'll definitely test tonight.

Mike

[BigPines]

Greetings BigPines.

Would like to know if the PS3 could handle this as well:

Name: 700_2D3D_interleave.mkv
Size: 105.14MB
Description: 1920x1080/60p 2D to 3D Interleave
http://www.sendspace.com/file/pxhas6

Many thanks for testing.
Well, the file streamed to the PS3 using PS3 Media Server. However, it didn't recognize it as Frame Sequential.

Did you make this yourself? What camera did you use? Is this really a 2D conversion? What did you do that with? Did you flag it with x264 framepacking 5? It looks like it is 119.88 FPS. So the original material was 59.94 fps per eye/view?

Beautiful bird by the way.

Mike

[mariner]

Well, the file streamed to the PS3 using PS3 Media Server. However, it didn't recognize it as Frame Sequential.

Did you make this yourself? What camera did you use? Is this really a 2D conversion? What did you do that with? Did you flag it with x264 framepacking 5? It looks like it is 119.88 FPS. So the original material was 59.94 fps per eye/view?

Beautiful bird by the way.

Mike

Yep. Frame Packing 5. Credit for the beautiful birds goes to Ken Ross.


The goal is to create a 1920x1080/119.88p frame-sequential pseudo-3D video using Martin Haverland's simple 2D to 3D conversion script found here (http://forum.doom9.org/showthread.php?p=1421042). The source is 1920x1080/59.94p .mts (http://hdcam.web-pda.info/21639208%20-%20ken%20ross%2003.mts) stream.

There is also the more complicated script by Branko Jermanis (http://free-ri.htnet.hr/Branko/10.html). Appreciate if anyone familiar with it could lend a helping hand.

BTW, AssumeFPS("ntsc_double" ) outputs 59.94 fps with the bit-rate halved.

2Dto3D_interleave.avs:
## Open the video file for conversion, change the video file name
video=FFVideoSource ("D:\3D\left.mts")
audio=FFAudioSource("D:\3D\left.mts")
video2d = AudioDub(video,audio)

## Increase video brightnes on dark videos, good for 3D Vision owners
# video2d = video2d.Tweak(Bright=10)

## Convert to RGB32 to avoid the width restrictions
video2d = ConvertToRGB32(video2d)

## Optional aspect ratio maintaining quality resize for 3d monitor target resolution.
## Very cpu intensive, may be for offline use only, e.g. in virtualdubmod.
## 2x 3.0ghz cpu may give you a framerate of 16fps while running the whole script in virtualdubmod including xvid compression in HDTV quality saving setting.
## Offers great quality in the result video for fullscreen playback in every .avs capable player later.
## Also reduces ghosting if the original file resolution is smaller than the target resolution.
# videoW = width(video2d)
# videoH = height(video2d)
## For 19" Zalman use 1280, for 22" Zalman Trimon it is 1680 etc.
# hzTargetSize = 1280
# video2d = Lanczos4Resize(video2d, hzTargetsize, hzTargetsize * videoH / videoW)
## Commenting out the above resizing maintains realtime capability!

## Get video width/height and set the frame stretch factor
## Lower the value 100 to increase frame stretch, may introduce ghosting
videoW = width(video2d)
videoH = height(video2d)
ResW = videoW + (videoW / 100)
CropW = (ResW - videoW) / 2

## Create variables for left and right frame with one frame difference
## This is the Plufrich-like simulation that creates illusion of depth from movement
f1 = video2d
f2 = DeleteFrame(video2d, 0)

## Stretch the right frame to further the depth effect
f1 = LanczosResize(f1, ResW, videoH)
f1 = Crop(f1, 0, 0, videoW, videoH)

## Stretch the left frame to further the depth effect
f2 = LanczosResize(f2, ResW, videoH)
f2 = Crop(f2, CropW, 0, videoW, videoH)

## Output the two video frames in a side-by-side / parallel format
## Use this as a default for playing back on 3D Vision (Side by Side L/R)
#StackHorizontal(f2, f1)

## Output the two video frames in a Above/Below format (like Sony?)
# StackVertical(f2,f1)

VideoInterleave = Interleave(f1,f2)
ConvertToYV12(VideoInterleave)
AssumeFPS(120000, 1001)
#AssumeFPS("ntsc_double" )

x264.exe:
x264 --frame-packing 5 --crf 17 --tune film -o 700_2D3D_interleave.mkv 2Dto3D_interleave.avs

General
Complete name : D:\3D\700_2D3D_interleave.mkv
Format : Matroska
File size : 105 MiB
Duration : 13s 972ms
Overall bit rate : 63.1 Mbps
Writing application : x264 r1834 a51816a
Writing library : Haali Matroska Writer b0

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5.1
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 13s 972ms
Bit rate : 61.9 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 119.880 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.249
Stream size : 103 MiB (98%)
Writing library : x264 core 112 r1834 a51816a
Encoding settings : cabac=1 / ref=3 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=17.0 / qcomp=0.60 / qpmin=0 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Language : English


Name: 700_2D3D_interleave.mkv
Size: 105.14MB
Description: 1920x1080/60p 2D to 3D Interleave
http://www.sendspace.com/file/pxhas6

Here are some raw Panasonic HS700 .mts samples for you to play with:
http://hdcam.web-pda.info/San%20Fran%20trolley.MTS
http://hdcam.web-pda.info/21639208%20-%20ken%20ross%2003.mts
http://www.megaupload.com/?d=3EBAWEWN

@rica: it would be interesting to see if your MVCV encoder would output a "non-compliant" 1920x1080/60p BD3D.



Here is one made by Dark Shikari:

My initial testing on 3D content is done.

Free lossless 3D test sequence (http://x264.nl/developers/Dark_Shikari/lossless3d.mkv) (L/R interleaved).

I have some tweaking I plan to do, but the crux of the matter is that MVC is totally pointless, as are all coding schemes besides interleaved mode. DVB's insane attempt to use T/B half-resolution using regular AVC is going to cost billions of dollars in wasted bandwidth.

Using that source, at crf 25, without any changes to x264:

Two separate streams: 3709 and 3700kbps, for 7.4mbps total
One stream, interleaved: 5503kbps (improvement: ~34.5%)
PSNR improvement: ~0.381db (~7.5%)
Overall improvement: 42%

Also, even on superfast x264 is able to capture "motion" perfectly between two views, even if they're widely spaced from each other. No special 3D-aware motion estimation methods are required. I do recommend using at least --ref 2 though, even with superfast, so it can predict from both the previous frame and the previous view.

Many thanks for testing.

[BigPines]

My initial testing on 3D content is done.

Free lossless 3D test sequence (http://x264.nl/developers/Dark_Shikari/lossless3d.mkv) (L/R interleaved).

I have some tweaking I plan to do, but the crux of the matter is that MVC is totally pointless, as are all coding schemes besides interleaved mode. DVB's insane attempt to use T/B half-resolution using regular AVC is going to cost billions of dollars in wasted bandwidth.

Using that source, at crf 25, without any changes to x264:

Two separate streams: 3709 and 3700kbps, for 7.4mbps total
One stream, interleaved: 5503kbps (improvement: ~34.5%)
PSNR improvement: ~0.381db (~7.5%)
Overall improvement: 42%
That file uses the same standard frame packing SEI (45) as x264 does. The only difference is that the stream has the SEI before every frame (with current_frame_is_frame0 flag set for each other frame) while x264 only inserts the message once for each GOP.
Dark Shikari,

I have tested your sample file and unfortunately, the PS3 does not recognize it as Frame Sequential. nm outlines the reason why above. I believe Sony may have chosen this route to mitigate the inherent problems with Frame Sequential that were mentioned by BlackSharkfr earlier in this thread. Is there any chance you could offer a frame packing option that would be PS3 compliant and apply the flag prior to every frame? This would give us a player that has a large install base to play back Frame Sequential encodes.

I appreciate your consideration and am interested in your thoughts.

Mike

[nm]

I have tested your sample file and unfortunately, the PS3 does not recognize it as Frame Sequential.

How did you try it? By remuxing to MP4 and playing from an USB stick I hope?

I believe Sony may have chosen this route to mitigate the inherent problems with Frame Sequential that were mentioned by BlackSharkfr earlier in this thread.

One message per GOP is enough to avoid sync problems. It's Sony's fault if that doesn't work.

[Dark Shikari]

You can of course trivially hack x264 to add it per-frame and see if it works. I'm not going to consider it unless someone can show me that it actually helps.

[BigPines]

How did you try it? By remuxing to MP4 and playing from an USB stick I hope?
I used PS3 Media Server. The same way I streamed the official one that worked great. The server is not remuxing the stream so it should work but it doesn't.

Mike

[nm]

I used PS3 Media Server. The same way I streamed the official one that worked great.

But the other file was also originally in MP4 container. I'm not familiar with PS3 Media Server, so I don't trust that it doesn't do more than what you asked for. It must be remuxing the video and audio to something else than MKV, at least.

[BigPines]

You can of course trivially hack x264 to add it per-frame and see if it works. I'm not going to consider it unless someone can show me that it actually helps.
I understand your position.

I am a .net software developer but I don't have much experience with C. Although I believe that the hack is probably trivial, I don't think I have the skill to create the hack at this time. Would anyone be willing to patch it for me so I could try it? Sorry to be such a leach.

Mike

[BigPines]

But the other file was also originally in MP4 container. I'm not familiar with PS3 Media Server, so I don't trust that it doesn't do more than what you asked for. It must be remuxing the video and audio to something else than MKV, at least.
It is remuxing but not transcoding (it uses tsmuxer behind the scenes). I could try it on USB but I know it won't work at this point.

The file basically stutters like it is trying to work but constantly goes back and forth.

Mike

[nm]

The file basically stutters like it is trying to work but constantly goes back and forth.

So it works part of the time? How long periods?

[BigPines]

So it works part of the time? How long periods?
Fraction of seconds. It is hard to explain but it is like watching regular Frame Sequential on a PC monitor with both frames visible but every once in a while (fraction of a second) it switches so the interleaving is not happening anymore but then just as fast as it changes, it changes back. It happens VERY quickly so it is hard to see exactly what is going on.

It is like starts and fits (really fast hiccups).

Mike

[nm]

Ok, that could be a display refresh effect. I'll try to get my hands on a PS3 to try some things. Which firmware version does it need to work?

[BigPines]

Ok, that could be a display refresh effect. I'll try to get my hands on a PS3 to try some things. Which firmware version does it need to work?
I'm not sure when the 3D update was pushed out. I have a PS3 slim with firmware 3.55.

I'll also put these examples on a USB stick just to rule out PS3 Media Server but I'm pretty sure that is not the problem.

Mike

[BlackSharkfr]

It sounds like the PS3 does not support frame sequential as in frame sequential mode, but it does support the left/right tagging via SEI messages and forwards it to the display as 3D video.
When it reads the left right SEI message it triggers the 3D mode but switches back to 2D when it finds out the next messages are missing. It's a clever way of allowing instant switching between 2D and 3D broadcast within a single video stream but requires every frame to be tagged with the SEI message.

No need to pre-declare the file as frame sequential, no need to tell when 3D starts or stops, if a frame is skipped, the decoder won't even bother (L message : send the picture to left buffer, R message : send the picture to the right buffer, no message : default to normal video).
All you need is the 3D encoder and 3D decoder to follow some convention like "the L frame shall always be first" (or the opposite) and the frame alternating stream will resist frame loss. I don't know if Sony actually does this but it sounds good to me.


While waiting for an update, maybe you can try to make a test x264 file that has super-short gop of a single frame. Compression wise it won't be efficient at all but since x264 only marks the 1st frame of a gop, you'll have all frames marked.

[nm]

It sounds like the PS3 does not support frame sequential as in frame sequential mode, but it does support the left/right tagging via SEI messages and forwards it to the display as 3D video.
When it reads the left right SEI message it triggers the 3D mode but switches back to 2D when it finds out the next messages are missing. It's a clever way of allowing instant switching between 2D and 3D broadcast within a single video stream but requires every frame to be tagged with the SEI message.

Such switching could be implemented as well with a single frame-packing message at the point of the switch, and at the start of each GOP as usual. The encoder would probably place a keyframe at the point of the switch anyway.

All you need is the 3D encoder and 3D decoder to follow some convention like "the L frame shall always be first" (or the opposite) and the frame alternating stream will resist frame loss.

Frame loss between encoder and decoder would cause significant corruption. Stereo frame sync is not an issue at that point.

If the decoder decides to drop frames in frame-packed stereo video, it should probably do it in pairs and follow incoming messages to switch between mono and stereo mode.

[BlackSharkfr]

Such switching could be implemented as well with a single frame-packing message at the point of the switch, and at the start of each GOP as usual. The encoder would probably place a keyframe at the point of the switch anyway.



Frame loss between encoder and decoder would cause significant corruption. Stereo frame sync is not an issue at that point.

If the decoder decides to drop frames in frame-packed stereo video, it should probably do it in pairs and follow incoming messages to switch between mono and stereo mode.

The picture would indeed be partially corrupted, but due the the great similarity between the left an right eye views, you'll still get something that is watchable. A left/right inversion screws the entire picture up, a brief flash is annoying already but something as long a an entire GOP is really bad, and GOPs can get very very long with unrestricted x264 profiles.
If I were to have a problem, I'd rather try to get something mild with partial recovery rather than the total picture failure.
When a player detects a corruption, it has the choice between showing the corrupted picture to the user (what most decoders do) or blacking/freezing out the screen (what TVs do). I prefer watching the picture, so I'd rather at least get the eye order corrected instantly than on the next GOP.

I don't like the idea of the enable/disable stereo switches, what would happen if a switch is missing, or missed ?
I find Sony's approach of marking every frame to use the 3D mode much simpler, cleverer and safer.
It does everything I'd ever want for a pure frame alternate 3D stream with a single measure.

[nm]

The picture would indeed be partially corrupted, but due the the great similarity between the left an right eye views, you'll still get something that is watchable.

A significant loss where whole frames are missed will lead to the whole GOP being corrupted and totally unwatchable in pretty much any case. And for error mitigation in general, keyframe interval needs to be short: 1 to 2 seconds.

I don't like the idea of the enable/disable stereo switches, what would happen if a switch is missing, or missed ?

It's repeated in every GOP, so the error only lasts for a second or two in a broadcast.

I find Sony's approach of marking every frame to use the 3D mode much simpler, cleverer and safer.

I'm not sure if it's worth the few extra bits.

[BlackSharkfr]

A significant loss where whole frames are missed will lead to the whole GOP being corrupted and totally unwatchable in pretty much any case. And for error mitigation in general, keyframe interval needs to be short: 1 to 2 seconds.



It's repeated in every GOP, so the error only lasts for a second or two in a broadcast.



I'm not sure if it's worth the few extra bits.

I'm not that convinced the picture will be such a mess, it will certainly depend on "where" the missing frame strikes. A missing B-frame in the middle of a static scene won't cause the same amount of damage than in the middle of an action scene. I remain convinced I'd rather see the corrupt picture with the correct eye order than with the eyes reversed.

I agree GOPs should not be extended too much, but H264 does offer this flexibility and we should think about what will happen for the people using x264 with long GOPs.
There is also the issue of streaming mode. Although I've never actually used it (it's the open gop mode right ?), I remember vaguely reading about this mode a few months ago in Dark Shikari's blog, with the scanning I-macroblocks thing that replaces I-frames.

I consider 1 or 2 seconds of eye inversion a significant enough issue to make these few bits of data on every frame worth it when doing 3D (especially given the usual higher bitrates used for 3D video due to the image quality requirements). And I am very serious about this.
Sony's approach is very simple, and prevents bugs by design so we won't have to come back and plea x264 developers for bug fixes in ultra-niche usages that interests them even less than generic 3D video.

[BigPines]

How did you try it? By remuxing to MP4 and playing from an USB stick I hope?
Just confirming that I remuxed the test clips using mpeg streamclip to mp4 and put them on a USB drive. It didn't help at all. Mariner's file worked exactly the same. That is to say I could view both eyes (one eye every other frame) because the PS3 didn't recognize the stream as Frame Sequential. Dark Shikari's lossless clip wouldn't play at all. It said unsupported data. Not sure what the problem with it was. Although it was 720p, tsmuxer recognized it as "-642:88i" which is definitely strange. I noticed tsmuxer saw it as level 3.2. It also seemed to have a problem in it (I downloaded it twice) where tsmuxer complained with the following message: "B-pyramid level 2 detected. Shift DTS to 3 frames" I'm not sure if that had anything to do with the error on the PS3 but I thought I'd mention it.

While waiting for an update, maybe you can try to make a test x264 file that has super-short gop of a single frame. Compression wise it won't be efficient at all but since x264 only marks the 1st frame of a gop, you'll have all frames marked.
Excellent idea! This will be my next experiment. :)

I consider 1 or 2 seconds of eye inversion a significant enough issue to make these few bits of data on every frame worth it when doing 3D (especially given the usual higher bitrates used for 3D video due to the image quality requirements). And I am very serious about this.
For what it is worth, I agree with this 100%. How much overhead are we really talking about? It looks like it would be maybe 1KB per second. Most of us interested in high quality would eagerly invest such a small amount to get an improvement in reliability. What is the real downside?

Mike

[Dark Shikari]

I understand your position.

I am a .net software developer but I don't have much experience with C. Although I believe that the hack is probably trivial, I don't think I have the skill to create the hack at this time. Would anyone be willing to patch it for me so I could try it? Sorry to be such a leach.

MikeC# is just C with much messier syntax, it's not hard.

The if ( h->param.i_frame_packing >= 0 ) block in encoder.c is inside an if( h->fenc->b_keyframe ) block. Move it outside. Done.

[BigPines]

I have done some more testing. I made the hack suggested by Dark Shikari above (thank you very much - it was indeed easy) and compiled the modified x264. As a bonus I was able to compile it for OS X so I don't have to do all my testing under a virtual machine. :) I used the following to re-encode the two Frame Sequential test videos I have been using:

./x264 --crf 24 --frame-packing 5 -o lossless3d_flagged.h264 lossless3d.mp4

result:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101010a00] multiple edit list entries, a/v desync might occur, patch welcome
lavf [info]: 1280x720p 0:1 @ 60/1 fps (vfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 3.2
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] illegal short term buffer state detected
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] illegal short term buffer state detected
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] illegal short term buffer state detected
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] illegal short term buffer state detected
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] illegal short term buffer state detected
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
[h264 @ 0x101008000] illegal short term buffer state detected
[h264 @ 0x101008000] reference picture missing during reorder
[h264 @ 0x101008000] Missing reference picture
x264 [info]: frame I:14 Avg QP:21.66 size: 84250
x264 [info]: frame P:636 Avg QP:24.92 size: 22147
x264 [info]: frame B:825 Avg QP:27.81 size: 4858
x264 [info]: consecutive B-frames: 4.0% 51.3% 18.8% 25.9%
x264 [info]: mb I I16..4: 4.0% 64.9% 31.1%
x264 [info]: mb P I16..4: 1.4% 7.0% 2.4% P16..4: 44.2% 18.0% 8.8% 0.0% 0.0% skip:18.3%
x264 [info]: mb B I16..4: 0.2% 0.5% 0.1% B16..8: 34.7% 4.6% 1.0% direct: 2.1% skip:56.8% L0:43.7% L1:55.2% BI: 1.1%
x264 [info]: 8x8 transform intra:64.5% inter:67.8%
x264 [info]: coded y,uvDC,uvAC intra: 71.8% 51.0% 13.0% inter: 17.5% 10.0% 0.1%
x264 [info]: i16 v,h,dc,p: 42% 21% 8% 28%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 16% 21% 6% 8% 8% 10% 7% 9%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 20% 19% 6% 8% 7% 8% 5% 7%
x264 [info]: i8c dc,h,v,p: 58% 19% 19% 4%
x264 [info]: Weighted P-Frames: Y:5.5% UV:2.5%
x264 [info]: ref P L0: 55.5% 29.1% 10.8% 4.4% 0.2%
x264 [info]: ref B L0: 82.5% 15.8% 1.8%
x264 [info]: ref B L1: 93.4% 6.6%
x264 [info]: kb/s:6271.94

encoded 1475 frames, 22.71 fps, 6271.94 kb/s


./x264 --crf 24 --frame-packing 5 -o 700_2D3D_interleave_flagged.h264 700_2D3D_interleave.mp4

result:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101010a00] multiple edit list entries, a/v desync might occur, patch welcome
lavf [info]: 1920x1080p 0:1 @ 120000/1001 fps (vfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 5.1
x264 [info]: frame I:8 Avg QP:20.73 size:155668
x264 [info]: frame P:720 Avg QP:23.33 size: 27677
x264 [info]: frame B:934 Avg QP:25.66 size: 4874
x264 [info]: consecutive B-frames: 15.5% 19.6% 2.4% 62.5%
x264 [info]: mb I I16..4: 7.4% 81.4% 11.2%
x264 [info]: mb P I16..4: 1.7% 5.9% 0.1% P16..4: 53.1% 9.1% 6.6% 0.0% 0.0% skip:23.4%
x264 [info]: mb B I16..4: 0.2% 0.6% 0.0% B16..8: 25.8% 0.8% 0.1% direct: 0.5% skip:71.9% L0:45.7% L1:53.3% BI: 1.1%
x264 [info]: 8x8 transform intra:77.0% inter:83.7%
x264 [info]: coded y,uvDC,uvAC intra: 48.7% 79.9% 13.6% inter: 11.7% 19.6% 0.8%
x264 [info]: i16 v,h,dc,p: 12% 24% 12% 52%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 19% 34% 5% 6% 4% 7% 4% 7%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 19% 20% 7% 11% 7% 10% 5% 6%
x264 [info]: i8c dc,h,v,p: 61% 23% 14% 3%
x264 [info]: Weighted P-Frames: Y:0.1% UV:0.0%
x264 [info]: ref P L0: 56.2% 12.2% 25.2% 6.3% 0.0%
x264 [info]: ref B L0: 87.8% 8.6% 3.6%
x264 [info]: ref B L1: 96.7% 3.3%
x264 [info]: kb/s:14844.38

encoded 1662 frames, 19.56 fps, 14844.38 kb/s

As you can see, I got several errors from Dark Shikari's lossless3d file. mariner's 700_2D3D_interleave video encoded with no errors. Interestingly, I was able to play back both files on my computer with no obvious problems.

I tried both videos on the PS3 and unfortunately neither of them were recognized as Frame Sequential. :( So my next question is, did my hack not work for some reason or is there something else we need to do to make this work?

nm, how did you determine the PS3 file had flags on every frame? How can I look at my videos to determine the same? I opened them up in a hex editor but couldn't see anything obvious.

For anyone interested, you can download my converted test video files:

700_2D3D_interleaved_flagged.m2ts: http://www.fileserve.com/file/eDqPU8x
lossless3d_flagged.m2ts: http://www.fileserve.com/file/gR7KaBf

I didn't know the legality of providing the x264 compiled for OS X so I didn't do that. I don't want to break any rules.

I would appreciate any feedback. I sure wish I could make this work!

Mike

[popper]

"I didn't know the legality of providing the x264 compiled for OS X so I didn't do that."
there's no problem, you can make x264 work on ANY Hardware/OS (with SIMD)as per LGPL rules.

at this point *, Bonus points for porting and writing More ARM A8/A9 NEON SIMD for it and actually submitting that on the #x264dev IRC channel (rather than say you will and never Do) so if you know any ARM cortex assembly guys, tell them to get over there and contribute if they want a faster ARM x264/ffmpeg.

*Quad-Core Freescale i.MX 6 ARM/NEON
http://www.linuxfordevices.com/c/a/News/Freescale-iMX-6/
Multicore Cortex-A9 SoC touted for 3D video
By Eric Brown 2011-01-03
The single-core i.MX 6Solo, dual-core i.MX 6Dual, and quad-core i.MX 6Quad are all clocked at 1.2GHz and touted as delivering 1080p60 video decode, 720p60 encode, and 3D video playback at 50Mbps.

[BigPines]

"I didn't know the legality of providing the x264 compiled for OS X so I didn't do that."
there's no problem, you can make x264 work on ANY Hardware/OS (with SIMD)as per LGPL rules.

at this point *, Bonus points for porting and writing More ARM A8/A9 NEON SIMD for it and actually submitting that on the #x264dev IRC channel (rather than say you will and never Do) so if you know any ARM cortex assembly guys, tell them to get over there and contribute if they want a faster ARM x264/ffmpeg.

*Quad-Core Freescale i.MX 6 ARM/NEON
http://www.linuxfordevices.com/c/a/News/Freescale-iMX-6/
Multicore Cortex-A9 SoC touted for 3D video
By Eric Brown 2011-01-03
The single-core i.MX 6Solo, dual-core i.MX 6Dual, and quad-core i.MX 6Quad are all clocked at 1.2GHz and touted as delivering 1080p60 video decode, 720p60 encode, and 3D video playback at 50Mbps.
All I did was compile it for OS X. I don't know assembly and didn't make any performance enhancements to the code.

I just didn't know if I could redistribute a modified version compiled for OS X.

Mike

[popper]

sure, i get that BigPines, your fine telling/linking people where your OS X binary is, your not breaking any rules.

the other part was more general info for the general 3D readers/dev's that are and will come across this thread in the future to keep them aware of what options are available.

business dev's especially seem to have problems understanding that IRC is where development takes place and where they should be spending their time contributing :D

[Dark Shikari]

Your best bet is to look at the working file and compare the SEIs to the x264 version. They could be in different places, have different content, or both.

[BigPines]

Your best bet is to look at the working file and compare the SEIs to the x264 version. They could be in different places, have different content, or both.
Yep, that is what I would like to do but I don't know how. If anyone will point me in the right direction, I will dig into it right away.

Sorry I'm such a noob at this stuff.

Mike


P.S. Since it appears it is OK to share the compiled app, the OS X 64 bit version is here: http://www.fileserve.com/file/R9fJcZc It has been tested on 10.6.6 and appears to work well. Please keep in mind, this is a beta that includes some Frame Sequential optimizations by Dark Shikari as well as an additional hack that should flag each frame of the video as Frame Sequential (intended for PS3 compatibility) when --frame-packing 5 is used.

[nm]

nm, how did you determine the PS3 file had flags on every frame? How can I look at my videos to determine the same?

I used h264_parse from the MPEG4IP project on a demuxed elementary stream, and since my version didn't support the frame packing SEI, I had to parse its contents manually from the hex dump.

PS3 probably expects you to set current_frame_is_frame0_flag=1 for the left view frame and current_frame_is_frame0_flag=0 for the right.

[BigPines]

I used h264_parse from the MPEG4IP project on a demuxed elementary stream, and since my version didn't support the frame packing SEI, I had to parse its contents manually from the hex dump.

PS3 probably expects you to set current_frame_is_frame0_flag=1 for the left view frame and current_frame_is_frame0_flag=0 for the right.
Bummer, it looks like that project is no longer under development and the last download is corrupt: http://mpeg4ip.sourceforge.net/downloads/index.php

Anybody know what other options I have to parse these streams to try to figure out what is going on here?

EDIT: I found these Windows binaries. http://pbx.mine.nu/mpeg4ip_binaries_for_windows_free_download/ They should probably work. If anyone has any other suggestions, I'm all ears.

Mike

[nm]

There are H264Visa and Elecard StreamEye. I haven't used either and don't know how the trial versions are limited.