Mpeg2 chroma artifacting in 3D Videos

[clydd2004]

Hi I dont quite know how to begin because i've posted in so many forums about the same topic but cant find an answer.
Im hoping the very involved group here will be my breakthrough.

In short: Im into 3d video the spk-kids kind (red-cyan) or anaglyph video as its called. Well the problem here is that no matter how good your source quality video and how high your mpeg2 encoder is set at, everytime you encode video that is in anaglyph format there is an "embossing" artifact that crops up. Now this embossed look or fringe or "halo" cant be seen with the naked eye, but the moment you put on the red-cyan glasses the horrid thing is visible.
Its even worse on areas of the scene that has high contrast, such as a building silloutte against a clear sky bakground.

The Long Explaination as I have posted it on the virtual Dub forum and some on-going debate with the experts there is tabulated here:
http://neuron2.net/ipw-web/bulletin/bb/viewtopic.php?t=398

Sorry if this post gets long and out of hand.. im just desperate for an answer and am convinced it can be cracked within the existing Mpeg2 framework. Heres the longer explaination....
--------------------------**from neuron's VDub board***--------------
Hi All,
Im a newbie to video processing and my expierence on this is to put it politely.... 2%.

I work with 3d video and have taken up the challenge of doing full color 3dvideo for DVD using the red/cyan glasses made popular with various movies.

A brief backgrounder (forgive me if you all know this)..
Anaglyph 3d video (the red-cyan glasses type) is very good at reproducing stereo left right views as long as its in some uncompressed codec or codecs like HuffyUV or Mjpeg with 4:4:4 colorsampling.

With Mpeg2 encoding, what happens is there is some artefacting on high contrast areas in a scene. As you know the entire scene of an anaglyph video is a superimposition of a Left and right view as recorded by 3d cameras...so the final composition is a superimpositing of the two images as viewed via a red filter and cyan filter.

Thus you have an image with Red and Cyan color fringes around it.
Now what happens is... as long as the avi is uncompressed all is fine, but MPEG2 encoding leaves an "embossed" in areas of contrasting detail in the scene where the red-cyan fringes were.
Meaning you cant see the embossed effect with the naked eye, but with the red-cyan glasses on it looks terrible.

No matter what bitrate i encode at ..even at 15mbps and greater this embossed look is still present.

I want to know a few things..

1) Does anyone know wether this effect is beacuse of mpeg sampling algorithm? meaning cause its 4:2:2 or 4:2:0 or is there some other reason?

2) Is there a filter that will smoothen or eliminate this effect in VirtualDub?

Answers by Others:

VeryWetPaint: clydd2004,

Anaglyphic 3d is a tall order for MPEG, but surely there must be some room for slop.

With red-cyan 3d, information will be concentrated in the Y and Cr channels. (Depending on sign and intensity, the Cr channel turns white into either red or cyan.) There is only one Cr sample for every four Y samples, so there must be decode errors unless all four pixels in each subsampled group have identical Y values to go with a single Cr value. That is, all four pixels in each 2x2 grouping must be identical after decoding.

To accomplish this in a 720x480 image, there must be 360 congruent pairs of pixels on each line and 240 congruent pairs of pixels in each column. That is, the image must be a 360x240 matrix of 2x2 pixel-groups. A simple averaging filter won't accomplish this.

Hmm.

Try pixelizing it with VirtualDub, using TWO resize filters in series. First resize the image to 360x240 using your choice of resampling (bilinear, bicubic, etc). Then resize it to 720x480 using "nearest neighbor". It'll be ugly, but it just might encode correctly at really high bit-rates.

I'll keep my fingers crossed.
---------------------------------------------------------------------

Hoping for a solution
Regards All,
Clyde

[Mug Funky]

i would say the effect is from the 4:2:0 sampling without a doubt.

you could mitigate this effect by blurring the luma channel, but that's a very stupid thing to do :)

the problem is that 4:2:0 assumes less colour sensitivity for the naked eye. as soon as a filter comes between the screen and the eye, this assumption becomes invalid and you get the embossing effect (embossing comes from subtracting a blur from the original, which is what happens when luma and chroma are suddenly on equal terms through the glasses).

i currently have no idea how to fix this though :(

perhaps something could be added to libavcodec, as that's still being actively developed, unlike most other mpeg-2 encoders, and also the developers might just be open to suggestions like these (as long as they're possible)

[Nic]

@Mug Funky: although it could be added to libavcodec, libavcodec doesn't even decode 4:2:2 MPEG-2 video yet...nevermind encode it :( So it's a long way off.

@clydd2004: If Mug Funky is right, then you'll need a MPEG-2 Encoder that can encode using a 4:2:2 profile...which is quite rare :( (I don't know of a software one...)

[clydd2004]

bbmpeg does 4:2:2 but the embossod prob still exists even there :(
the thing is the solution needs to be done with 4:2:0 as thats the standard Mp@ML (damn im just learning all this techno speak as best I can so bear with me) that DVB and DVD uses.
Digital Video Broadcast and DVD is what im looking at for ThreeD anaglyph video.

Thanks for shedding some light on this Mug Funky. Im willing to try anything even at blurring luma (how is that done).
Resolution wont be the *most important* factor, as the filters will anyway disguise to some extent a lss of resolution detail, and hopefully people are so caught up in the :wow effect" of ThreeD stereo video they wont complain if resolution is a bit downgraded.

Regards
Clyde

[Mug Funky]

hmm. a couple of questions, because i just realised i might be barking up the wrong tree:

a) what is your source of analglyph video? DVD? if so, then it's obviously possible to do this properly (unless the embossing is present in the DVD too!)

b) does this effect work on a TV (running VHS maybe?)

if the above conditions are met, there wouldn't be a chroma problem from MPEG-2 to speak of - especially in NTSC where the chroma resolution for broadcast TV is seldom better than 120 lines horizontal (and usually much, much worse, like 50 or less)

Chroma gets treated disgracefully on TV in general. this may be why analglyph hasn't taken off (though it is a cool effect)

[clydd2004]

a) what is your source of analglyph video? DVD? if so, then it's obviously possible to do this properly (unless the embossing is present in the DVD too!)

The source is Interlaced Stereo Video (captured via DV camera) then taken into VDub and the interlaced RGB filter is used. Taking out the red channel from odd(or even) and the green+blue channel from even (or odd as the case may be).
This makes it into an anaglyph video. then I use the Deinterlace filter and blend fields to get a progressive video in VDub. then i save as uncompressed RGB for the best quality.
at this stage thereis NO embossing effect.

Then I take RGB into BBMPEG (or every other Mpeg2 editor ive tried to no use) and encode at dvd high rate 9.8mbps.
Bang-- Embossed Ghosting!

The anaglyph effect works on todays good TVs and perfectly on Lcd Tvs.
The embossed ghosting is present and seen even on a hi-res monitor and LCD tv screen, In fact on a normal household Tv, tho the embossed effect is there you can actually "blur" it a bit with the TV's Sharpness contrl turned all the way down.

I think the problem is in what you said .. the differences with Luma and chroma. the embossed halo effect cant be seen with the naked eye. only when the 3d glasses are put on it appears.

Regards
Clyde

BTW: todays state of the art Anaglyph encoding techniques are Just as good as shutter glasses in reproducing full color visuals 9except the color red which gives a shiny effect) Im sure you wont believe me but check out the pictures with a good pair of spykids type anaglyph glasses on www.brightland.com (their akumira section).

You can download some pics from there and pass that thru any Mpeg2 encoder to see how Mpeg encoders destroy it :mad:

[Mug Funky]

hmm... the sample pics in that site were jpeg, possibly 4:2:0 sampled.

have you tried these with the glasses? (i don't have any cellophane handy to make a pair, and didn't see spykids3d)

if they are indeed 4:2:0, and there's no embossing effect, then it's possible that mpeg-2 could still work for you.

i'm clutching at straws, but maybe the colours are out of range when you compress to mpeg-2, and so edges become dead.

also if you are encoding interlaced instead of progressive (you deinterlaced first...), there might be some additional chroma smear going on. try it both ways and see what happens.

give nic's quenc a try - it's much less in the way of variables then bbmpeg, and gives very good quality (no 4:2:2 sampling though, like bbMpeg gives the option for)

[Kika]

I have one of those glasses and can see the problem.
I guess this "embossing" effect ist the combination of two things:

1. YUV 4:2:0
2. Compression Artefacts (Mosquito Noise)

I have tried to smooth the embossing out by using TMPGEncs Soften Block Noise Function - it works. The effect is still there, but it looks imho better now.
But i had only one of those Pics from the site for testing, on a Video it may be a different thing.
Take care to set the correct Colorspace in TMPGEnc, try them both, CCIR and YCbCr. Also use 10Bit-Precission, not 8 or 9 Bit.

[clydd2004]

Hi Kika,
I will try what you said in TmpgEnc. I mean any smoothing is good smoothinG! Mpeg2 beggars cant choose I guess :)
If this works out for DVD at least half the battle will be won.
For DVB (Digital video Broadcast) aka satellite TV, They use Mpeg2 as well, but I will have no control over that i guess as they will have their own Mpeg2 hardware encoders, and That will muck up the anaglyph anyways.

I think logically thinking the only way is to preprocess the Video to make it Mpeg2 friendly, (suggestions thus far have been to pixellize it into 2x2 blocks, smear luma channel, and use Tv safe chroma values.)
I'll try and experiment with all these variables and see what i can come up with, meanwhile any, I mean any, brainstorm is welcome.

@MugFunky
.if they are indeed 4:2:0, and there's no embossing effect, then it's possible that mpeg-2 could still work for you.

The Images are possibly saved at Jpeg Quality 8 and above. These dont exhibit emossing, but any value below photoshop Jpeg 7 do show the embossing.

When you say "then its possible that mpeg2 could sill work for you' your flirting with me arent yoU! :p How how is it possble thats the answer i need! :)

also if you are encoding interlaced instead of progressive (you deinterlaced first...), there might be some additional chroma smear going on. try it both ways and see what happens

I tried both ways ..interlace and progressive.
Theres less embossing happening in interlace, but playing the resulting anaglyph on TV will kill any one with a headache or epilepsy as it flickers between red field - cyan field so much that its unbearable.

regards
Clyde

PS a good way to make homemade glasses is to just print red and cyan squares on good quality transparency.