YUV->RGB Chroma upsample testing help ...

[trevlac]

Hi,

I've been playing with ways to 'better' upsample the chroma when converting from YUY2 to RGB.

I was looking to see if anyone could suggest some test source or a method of testing the various methods.


I made a avs filter that converts YUY2->RGB32. Effectively this goes from 4:2:2 to 4:4:4 1st. (I'd be happy to test 4:2:0, but put this off due to interlace issues. Besides, ConvertToYUY2 seems to do a great job with YV12 source.) The filter has ~ 8 ways of upsampling the chroma:

Duplicate Pixels
1/2 1/2 avg (this is what avs does)
3/4 1/4 | 1/2 1/2 | 1/4 3/4 avg based upon luma
7/8 1/8 | 1/2 1/2 | 1/8 7/8 avg based upon luma
Dup pixels or 1/2 1/2 based upon luma
Dont recreate missing chroma - set to grey (for testing)
Use an external 4:4:4 sized chroma (like lanczos)
Remove the luma that has no chroma (1/2 size result for testing)

http://trevlac.us/docs/UpSampleTest.gif

The above is a test I ran. Subtitle RGB source was down sampled using ConvertBackToYUY2() and then UpSampled with various methods.
The results are:

Original RGB
3/4 1/4 using luma for selection
Lanczos
7/8 1/8 using luma for selection
Dup Pixels
1/2 1/2 Avg
Don't make chroma (so you can see what the downsample did)


Frankly, I'm not sure there is enough of a difference to really go thru the effort. The above test goes out of it's way to make Dup look bad, due to the lack of interpolation on the down sample. But there seems to be a fair amount of people who 'see' a difference if they convert to RGB ... so maybe they can help me.

Thanks

PS: Here is the dll/source and the script i used for the test.
http://trevlac.us/docs/UpSample.rar

[Soulhunter]

Hmm, I prefer method No. 2 and method No. 4 !!!


Bye

[trevlac]

Originally posted by Soulhunter
Hmm, I prefer method No. 2 and method No. 4 !!!


That's good news since those are 'my new' methods. But do you have any suggestons for further testing?

Thanks

[MfA]

Test for all primary colours, and at a couple different intensities.

[Soulhunter]

I did a small test some time ago...

Original
http://img31.photobucket.com/albums/v94/Soulhunter-No.1/Original2.jpg

Encoded as XviD (YV12)
http://img31.photobucket.com/albums/v94/Soulhunter-No.1/Wrong_Upsample.jpg

YV12 converted to RGB with AviSynth
http://img31.photobucket.com/albums/v94/Soulhunter-No.1/AviSynth_Upsample.jpg


IIRC, there is a "good test" scene (a red logo) in the intro of Battle Royal !!!

I still have this movie somewhere... :rolleyes:

When I find it, I will upload a pic the next days !!!


Bye

[scharfis_brain]

Hey, that looks pretty cool!

could this method be modified for aligning low-bandwidth chroma of VHS recordings to the luma?

[trevlac]

@Soulhunter, MfA;

Thanks for the test ideas. I've run a bunch on RGB (actually, 'A' is a nice letter for this. The results are fairly much the same regardless of the color or intensity. Of course, for low intensity blue, it is near imposible to see anything. Also, I've always used black as a background. I will put together a picture/matrix. There is 1 catch. How you get to 4:2:2 matters ... see below.


@Scharif,

I saw the method a bit ago and got a recent tip for some minor changes. Basically it is a classification method that tries to determine the color transitions based upon the luma changes.

Given [Y1 C1 Y2] & [Y3 C3 Y4] {2 YUV 4:2:2 Pixels}
Trying to determine C2
Calculate the absolute difference from Y2 of Y1,Y3,(Y1+Y3)/2

If the Y1 difference is smallest, C2 = C1*7/8 + C3*1/8
If the Y3 difference is smallest, C2 = C1*1/8 + C3*7/8
If the Avg Y1+Y2 diff is smallest,C2 = C1*1/2 + C3*1/2

For the 1st 2 cases, I found some averaging was needed or you get dup pixels artifacts. 7/8 vs 3/4 is a tradeoff between blur and blocky diagonals.

As far as VHS ... you could try, but I'm not sure of the results. Read the catch.

The Catch
It seems to matter how the Chroma was originally downsampled. I tested 3 cases

1)ConvertToYUY2 - which (from what I can tell) does an average over 4 chroma values [2 full RGB pixels]. Not sure of this. I thought it did 1/2 1/2 avg ... but the code looks like {1/4 1/4 1/4 1/4}

2)ConvertBackToYUY2 - which uses only the left 4:4:4 chroma - no blend

3)Make a 4:4:4 pic and downsize the chroma with Lanczos.

For #1, 3/4 weighting is about the same as a regular convertoRGB32. Maybe slightly better. 7/8 is worse due to blocky.

For #2, 7/8 is 'much' better than standard methods. 3/4 is better.

For #3, The classification methods may be slightly better.

Here is the big question: It is my understanding that for 4:2:2 and MPEG-2 4:2:0 the chroma should be co-sited with the left luma. So, is #2 the closest to the way it should be done? I'm not sure on a Lanczos downsize if you keep the left or the right when you center the filter.

So ... YMMV The more the original detail is blurred away, the less effective the upsample is. I will try to put to gether a chart.

All comments very welcome. :)

[MfA]

Originally posted by trevlac
Here is the big question: It is my understanding that for 4:2:2 and MPEG-2 4:2:0 the chroma should be co-sited with the left luma.

Only for 4:2:2, unless you meant to say it was aligned with the left luma in 4:2:0 (which it is, but halfway vertically between two luma samples ... check out the mpeg-2 standard (http://le-hacker.org/hacks/mpeg-drafts/is138182.pdf), which also has nice pictures to show the interlaced sampling).

#2 is a point sampling approach, chrominance isnt very important so it is unlikely to introduce aliasing (moire for instance). In principle the correct way is to do bandwith limited downsampling. You can use lanczos, or you can simply use horizontal blur before using approach #2 (Id favour this, because if you are not carefull lanczos could cause a 1/2 sample shift for chrominance which degrades the image).

[trevlac]

@MfA,

Yes ... I ment that chroma was alinged with luma for MPEG-2 4:2:0, seeing I am only thinking 4:2:2->4:4:4. AVS 4:2:0->4:2:2 seems to do a fine job anyway. (I did save the pdf for future reading ... :)

I do like your suggestion of a blur and then a convertbacktoYUY2 (point sample). This seems to be what proper source should be. The 1/2 sample shift is exactly what I am trying to avoid. Hopefully this will give a reasonably valid test.

PS: RGB->YUY2 seems to shift the chroma 1/2 sample to the right. On a line C0, C1, C2 ... C1 definately is only weighted with C2. Probably 1/2C1 1/2C2 (not that 1/4 ... rubish I thought before). I think it would be better for it to be 1/4C0, 1/2C1, 1/4C2.

[Soulhunter]

Maybe a good picture for testing...

http://img20.imageshack.us/img20/544/BattleRoyale.png


Bye

[morsa]

Don't know if I'm really stupid or the like, but seeing we begin with 4:2:0 color sampling, can't it be treated as a Bayer mosaic?
If the answer is yes here is this link with demosaicking algos:

http://www-ise.stanford.edu/~tingchen/main.htm

I guess the Variable Number Gradients method would give a really sharp image, far sharper than what we are used to see...

They are being used by the well known opensource digital camera soft "I don't remember its name now" which probably begins with a "g":)

what about EDI instead of Lanczos?

[trevlac]

Originally posted by morsa
Don't know if I'm really stupid or the like, but seeing we begin with 4:2:0 color sampling, can't it be treated as a Bayer mosaic?
If the answer is yes here is this link with demosaicking algos:

http://www-ise.stanford.edu/~tingchen/main.htm

I guess the Variable Number Gradients method would give a really sharp image, far sharper than what we are used to see...

what about EDI instead of Lanczos?

Hey Morsa ... yes, I took a look at that demosaicking stuff you pointed out at everwicked. Seems to me that this 'using the luma' thing is a Gradient method. Of course a Bayer array is different from the info we start with ... seeing they have to interpolate the luma too. I think that is why they get a bigger bang than just messing with the chroma.

I found the following write up the most clear:
http://www4.ncsu.edu/~rramana/Research/demosaicking-JEI-02.pdf


Also ... this stuff assumes a progressive source. A Bayer array is after all, the CCD array as it takes one big sample. No?

So going from 4:2:2 -> 4:4:4 requires only 3 classifications. Where as 4:2:0 -> would need 9 for a pixel radius of 1. Besides, avs going from 4:2:0->4:2:2 does a fine job and takes interlaced into account.

Anyway ... I still need to play with getting a 'valid' subsampled test source. ConvertToYUY2 shifts the chroma over by 1/2 pixel. I don't know if this makes a difference, but I want to test where the chroma on a 4:2:2 source is aligned with the left luma sample.

Sorry ... but I have not had a chance.


@Soulhunter

Thanks for posting that pic. I did some quick tests and frankly, I can't really see much of a difference between the normal bilinear and point sample methods. Maybe it's the PNG. Maybe if it moves it is different. Maybe I need new glasses. :)

[morsa]

well I've posted this because a Bayer array is in fact a 4:2:0 color sampling scheme.
So if we have a source with 4:2:0 color sampling we could apply the demosaicking algos, but without the need of interpolating Luma:)
(this doesn't apply for 4:2:2 though).
Always talking about progressive.If we talk about interlaced I guess there is no reason to go RGB because it will always end up being seen on an interlaced display (aka TV) which usually doesn't accept more than 4:2:2.But, even if this were the case, what would happen if we seperate the fields both on luma and chroma to process the same way?
I don't quite understand what you are talking about when you say that thing about Bayer , the CCD and one big sample ,etc, sorry.

[trevlac]

Originally posted by morsa

Always talking about progressive.If we talk about interlaced I guess there is no reason to go RGB because it will always end up being seen on an interlaced display (aka TV) which usually doesn't accept more than 4:2:2.But, even if this were the case, what would happen if we seperate the fields both on luma and chroma to process the same way?


I don't quite understand what you are talking about when you say that thing about Bayer , the CCD and one big sample ,etc, sorry.

Hi,

Your 2nd question 1st. I may well be wrong ... but I thought a Bayer array was as follows:
http://www.siliconimaging.com/Images/a_Bayer_Pattern_Diagram.gif

This is used in a 1 CCD camera. Each sensor in the ccd has a color filter and samples only 1 color. The other colors are then interpolated from the suronding samples. In a Bayer array, there are 2 greens due to the higher luma in green. I suppose there are other types of arrays/combos.
http://www.siliconimaging.com/RGB%20Bayer.htm

On the 1st. I think there is value in higher quality RGB because RGB is a better color space for processing. Even if the final output is 4:2:2. As far as interlaced goes, I have not really thought about it. But if I did, I'd say that there is a major problem with interpolating interlaced source. Namely, the vertical dimension is seriously undersampled. To interpolate vertical well, you need to use other frames for still images, and montion compensation for non still images. So basically, I don't think about it. :)

PS: I think the demosaicing methods work well because 1/2 of the luma is being interpolated. Therefore, you get more bang for your effort. 4:2:2 -> 4:4:4 gets you less visual bang. But still a fun project.

[morsa]

Well, You are right about Bayer:)
Second, I don't understand very well what you are saying and what are the differences with my posts, sorry.
Anyway, I guess in a near future I would make my own version going from YUV 4:2:0 to RGB, I've been testing it by hand and looks promising (I mean treating color planes with demosaicking algos based on Luma)

[trevlac]

Originally posted by morsa
I guess the Variable Number Gradients method would give a really sharp image, far sharper than what we are used to see...



This is the part I don't think is true. The reason I don't think it is true is that most 'sharpness' is seen in the luma. And since going from 4:2:0 -> 4:4:4 does not have anything to do with luma, we will not get as much improvement as you hope.

I hope I am wrong. I am still 'slowly' working on this myself. But, I am not sure there is too much improvement to be seen.

[morsa]

May be, but my tests show something different.Make one yourself, try this:
Take a real RGB source image, then convert it to 4:2:0 and compare the two.Can you see the difference in sharpness?
Not trying to be a j*rk, it's just that I work always upsizing images, and believe it or not, even a HDCAM source has a lot of haze because of this kind of thinking.I'm just trying to get the most out of 4:2:0 sampling, just that:)
Anyway I'm not trying to start a nonsense and I promise I will stay on topic from now on.
Sorry Trevlac.


P.S: Ok,OK...If someone is going to tell me that HDCAM has a 7.5:1 lossy compression, its resolution isn't 1920x1080 but in fact 1440x1080, it isn't actually progressive but some kind of intermediate solution and that its color sampling is something like 3:1:1, I already know it :D

[trevlac]

Hey,

You are not being a jerk to me. I'm very interested in this process and happy to talk about all aspects of it. I have only been focusing on 4:2:2->4:4:4. Your demosaicing comments helped me come up with more ways to test.

Originally posted by morsa
Take a real RGB source image, then convert it to 4:2:0 and compare the two.Can you see the difference in sharpness?


Yes I can see a difference. Of course to do this compare, I need to convert from 4:4:4 -> 4:2:0 -> 4:4:4. Some questions about this are:
1) How much damage is done by the down convert?
2) How good can the up convert be?

For Avisynth, I've found RGB->ConvertToYUY2() does more damage than it probably should. It actually shifts the color plane by 1/2 sample. I'm not sure if this messes up my tests of different ways to up convert. I can of course use ConvertBackToYUY2() which is a point sample back down. However, a 'real' 4:2:2 should have been created using a better filter. So I guess I need to do a 1,2,1 convolution on a 4:4:4 source and then point sample it.

My point on all of this is that my answer is yes, but I am very unsure of HOW much better, and what is a valid test.

Cheers

[morsa]

Happy to hear all that.I always test the images upsampling a ROI after the different processings to see exactly how bad or good it gets...

[shae]

Why isn't something like this integrated into AviSynth?

[IanB]

Lack of developer cycles.

No one added a well worded synopsis to the RfE pages on our Sourceforge site.

And yes Avisynth uses a [1, 0] algorithm in ConvertBackToYUY2() and a [0.5, 0.5] algorithm in ConvertToYUY2(), it probably should be a [0.25, 0.5, 0.25] style algorithm, to avoid chroma creep.

ConvertToRGB() uses a [[1, 0], [0.5, 0.5]] algorithm.

So yes repeated usage will shift the chroma 0.5 pixels right per cycle.