4:2:0 -> 4:2:2 upsampling methods

[Movie_Maker]

Hi,
I am able to understand how 4:1:1 or 4:2:2 could be upsampled to 4:4:4 using various methods or interpolation, but I am quite intrigued about how would you upsample 4:2:0 to 4:2:2 first (even before upsampling it to 4:4:4) ?

4:1:1 and 4:2:2 streams have chroma values tied to one pixel, but in 4:2:0 the chroma sample is actually an average(?) of two chroma samples - one of each line. See image below - (lets assume progressive for simplicity).


http://www.hometheaterhifi.com/volume_8_2/images/chroma-figure-3.jpg


So I guess the first step would be to 'split' the averaged chroma sample back into two full individual chroma samples. I understand that good interpolation can get your very close to actual values of missing samples, but here there are really no 'missing' samples. Its not a case for interpolation. Long story short - how does good 4:2:0 to 4:2:2 upsampling work ?

[Movie_Maker]

hmmm.. 71 views and 0 replies ? did I post it in the wrong forum ?

[Guest]

It's well-known stuff, easily findable with Google. Nobody wants to rehash it.

[Movie_Maker]

I tried searching on google a lot, the only things that kept coming up were all somehow related to 'Chroma upsampling error/bug'. Nothing in particular and specific about the way upsampling is done for 4:2:0.
I am not looking for upsampling in general, but specifically for 4:2:0.
Would you mind posting a link or two if you have any on hand ?

[Guest]

http://avisynth.org/mediawiki/Sampling

[Movie_Maker]

Thanks! thats a great start.
I realize thats how AviSynth's function does it, but really that's how its done elsewhere too ? Like dvd players and TVs, and dedicated video processors too ? Just an addition of a certain fixed ratio of the chroma samples on pre-determined lines ?
I had thought it would be different but didn't expect this. Very interesting.

[Guest]

I had thought it would be different but didn't expect this. What did you think it would be?

[Movie_Maker]

just keeping general oversampling and interpolation as my reference I was expecting it to be some kind of a complex mathematical form. Something that would recreate the two original(or close to original) values from which the 'averaged' chroma sample was derived.

[2Bdecided]

You never use "accurate" filters (i.e. the kind you might use for audio) for video - they ring far too much, and ringing is easily visible. Whereas the frequency response errors due to shorter filters (which would be easily audible) are quite difficult to see.

Cheers,
David.

[drmpeg]

From mpeg2decode:


/* vertical 1:2 interpolation filter */
static void conv420to422(src,dst)
unsigned char *src,*dst;
{
int w, h, i, j, j2;
int jm6, jm5, jm4, jm3, jm2, jm1, jp1, jp2, jp3, jp4, jp5, jp6, jp7;

w = Coded_Picture_Width>>1;
h = Coded_Picture_Height>>1;

if (progressive_frame)
{
/* intra frame */
for (i=0; i<w; i++)
{
for (j=0; j<h; j++)
{
j2 = j<<1;
jm3 = (j<3) ? 0 : j-3;
jm2 = (j<2) ? 0 : j-2;
jm1 = (j<1) ? 0 : j-1;
jp1 = (j<h-1) ? j+1 : h-1;
jp2 = (j<h-2) ? j+2 : h-1;
jp3 = (j<h-3) ? j+3 : h-1;

/* FIR filter coefficients (*256): 5 -21 70 228 -37 11 */
/* New FIR filter coefficients (*256): 3 -16 67 227 -32 7 */
dst[w*j2] = Clip[(int)( 3*src[w*jm3]
-16*src[w*jm2]
+67*src[w*jm1]
+227*src[w*j]
-32*src[w*jp1]
+7*src[w*jp2]+128)>>8];

dst[w*(j2+1)] = Clip[(int)( 3*src[w*jp3]
-16*src[w*jp2]
+67*src[w*jp1]
+227*src[w*j]
-32*src[w*jm1]
+7*src[w*jm2]+128)>>8];
}
src++;
dst++;
}
}
else
{
/* intra field */
for (i=0; i<w; i++)
{
for (j=0; j<h; j+=2)
{
j2 = j<<1;

/* top field */
jm6 = (j<6) ? 0 : j-6;
jm4 = (j<4) ? 0 : j-4;
jm2 = (j<2) ? 0 : j-2;
jp2 = (j<h-2) ? j+2 : h-2;
jp4 = (j<h-4) ? j+4 : h-2;
jp6 = (j<h-6) ? j+6 : h-2;

/* Polyphase FIR filter coefficients (*256): 2 -10 35 242 -18 5 */
/* New polyphase FIR filter coefficients (*256): 1 -7 30 248 -21 5 */
dst[w*j2] = Clip[(int)( 1*src[w*jm6]
-7*src[w*jm4]
+30*src[w*jm2]
+248*src[w*j]
-21*src[w*jp2]
+5*src[w*jp4]+128)>>8];

/* Polyphase FIR filter coefficients (*256): 11 -38 192 113 -30 8 */
/* New polyphase FIR filter coefficients (*256):7 -35 194 110 -24 4 */
dst[w*(j2+2)] = Clip[(int)( 7*src[w*jm4]
-35*src[w*jm2]
+194*src[w*j]
+110*src[w*jp2]
-24*src[w*jp4]
+4*src[w*jp6]+128)>>8];

/* bottom field */
jm5 = (j<5) ? 1 : j-5;
jm3 = (j<3) ? 1 : j-3;
jm1 = (j<1) ? 1 : j-1;
jp1 = (j<h-1) ? j+1 : h-1;
jp3 = (j<h-3) ? j+3 : h-1;
jp5 = (j<h-5) ? j+5 : h-1;
jp7 = (j<h-7) ? j+7 : h-1;

/* Polyphase FIR filter coefficients (*256): 11 -38 192 113 -30 8 */
/* New polyphase FIR filter coefficients (*256):7 -35 194 110 -24 4 */
dst[w*(j2+1)] = Clip[(int)( 7*src[w*jp5]
-35*src[w*jp3]
+194*src[w*jp1]
+110*src[w*jm1]
-24*src[w*jm3]
+4*src[w*jm5]+128)>>8];

dst[w*(j2+3)] = Clip[(int)( 1*src[w*jp7]
-7*src[w*jp5]
+30*src[w*jp3]
+248*src[w*jp1]
-21*src[w*jm1]
+5*src[w*jm3]+128)>>8];
}
src++;
dst++;
}
}
}


Ron

[Movie_Maker]

Ron, at first glance looks like a digital low pass filter that recalculates a pixel based on 6 vertical pixels at a time ? How did you come up with those filter coefficients ? And how/where do you use this filter ?
Would be interesting to try it out. Does it exhibit the ringing that David mentioned ?

[drmpeg]

It's from the original MPEG-2 reference decoder.

http://www.mpeg.org/pub_ftp/mpeg/mssg/mpeg2v12.zip

I believe the filter coefficients (and number of taps) are selected to provide better performance during motion. For still images, you will see ringing from this kind of upsampling. Here's an example of the ringing from a very similar FIR filter that does 4:2:2 to 4:4:4 (from the same MPEG-2 source code, store.c).

http://www.w6rz.net/bars709f.png

Ron

[Movie_Maker]

Hi, sorry didn't get back to this one sooner.

Thanks for posting the link. Looks like MSSG (http://www.mpeg.org/MPEG/video/mssg-free-mpeg-software.html) does interesting things. Definitely worth trying.

I also wanted to find out more about mpeg2 4:2:2(or 4:4:4) -> 4:2:0 subsampling methods, for which hopefully I should be able to find in a code snippet like the one you posted above.

[Manao]

I believe the filter coefficients (and number of taps) are selected to provide better performance during motionNo, during motion, you aren't sensitive to either sharpness / bluriness / ringing. The only thing that must be avoided at all cost during motion is aliasing (since it isn't temporally stable)

As for the screenshot drmpeg provide, indeed, ringing does show sometimes. But the example you provide show the limit of subsampling in itself, not of the FIR you've chosen (said otherwise, you can't create a nice colorbar in 4:2:0, whatever the chosen subsampling/upsampling)

[drmpeg]

The only thing that must be avoided at all cost during motion is aliasing (since it isn't temporally stable).
Thanks, that's what I meant to say but I could only vaguely remember reading about it in the Poynton book. The actual reference is here (last paragraph on page 93):

http://books.google.com/books?id=ra1lcAwgvq4C&pg=RA1-PA93&lpg=RA1-PA93&dq=simple+averaging+filters+poynton&source=web&ots=bOh2BDTU17&sig=Pbh5R9bo7Vwi51IRDWT0g0Dmsb8&hl=en&sa=X&oi=book_result&resnum=1&ct=result

Ron