YV12 chroma placement

[Guest]

Be very careful about averaging! You can make things worse. let's discuss it here before you change it. The 75%-25% weighting is arguably an error. I believe simple copying is fine. Do it as shown here:

http://forum.doom9.org/showthread.php?s=&postid=297236#post297236

[EDITED to remove inflammatory language.]

[sh0dan]

The 75%-25% weighting is an ERROR made by people who don't know what they are doing.

Be careful what you proclaim! Averaging is needed for proper choma placement and it helps stepping - The dilemma is the same as with RGB<->YUY2.

Averaging chroma gives visible better results than copying - especially when dealing with interlaced sources. Yes - chroma will get slightly blurred, but there is no easy way around that. 25/75 is not 100% accurate, but the difference to a 20/80 is so extremely small that it's not worth the relatively big speed penalty it'll give.

We could implement a "copy-mode" conversion routine. Just as the first versions of the converters were, and have them as ConvertBackToYUY2. An "interpolate" flag would probably be the best solution for this.

Note - the luma lines are lying between the black lines, in the following YV12 -> YUY2 illustration (The black lines illustrate the separation of two luma lines):

[Si]

@sh0dan
I'm sure your diagrams mean something to you but to us ...:confused: (esp the bottom pair)

regards
Simon

[TheRealMoh]

@neuron2:

My code works exactly as your illustration in the other thread.

@sh0dan

I understand your diagrams, but why do you say it's 80/20? According to my calculations, it's 75/25.

Here is the calc for non-interlaced (for interlaced, I always think of it as separatefields().convert().weave()):
Suppose you start with a YUV picture. Consider the first four rows, which have V values V1, V2, V3, V4. When these values are converted to YV12, there are only two rows left: newV1 = (V1+V2)/2 and newV2 = (V3+V4)/2.

Let's make a simplifying assumption: the progression of pixels values is linear. Linear progression is basis of all averaging, so it's not a *terrible* assumption. In other words, the orginal 4 pixels are really:
V1 = V
V2 = V + d
V3 = V + 2 d
V4 = V + 3 d

where 'd' is the constant delta between the values. The YV12 values are therefore: newV1 = (2 V + d)/2 and newV2 = (2 V + 5 d)/2.

The point of this exercise is reconstruct V2. Notice that
newV2-newV1 = 2 d
V2 is then (newV2 - newV1)/4 + newV1 = (newV2 + 3*newV1)/4
In other words, a 75/25 ratio.

Confused yet :confused: :)

At any rate, the 25% doesn't seem so important. If anyone really wants it that way, they can just load up the avi using avisynth and use its conversion routines.

[trbarry]

V2 is then (newV2 - newV1)/4 + newV1 = (newV2 + 3*newV1)/4

I believe that is correct. And even nicer, since there is a pavgb assembler instruction that averages:

V2 = pavgb(newV1,pavgb(newV2,newV1))

or

movq mm0, newV2 // eight 1-byte newV2's
pavgb mm0, newV1
psubusb mm0, AllOnes // adjust for repeated rounding bias
pavgb mm0, newV1
movq V2, mm0


- Tom

[sh0dan]

@neuron: Glad we agree - I just had the 80/20 from some page in the "Please explain this...."-thread, but they also recommended the 25/75 for speed. Don't know where they got their result from though.

@trbarry: I use pavgb now - but I can't see why you would subtract 1? Just using pavgb twice should produce correct rounding.

[Guest]

Originally posted by sh0dan
@neuron: Glad we agree - I just had the 80/20 from some page in the "Please explain this...."-thread, but they also recommended the 25/75 for speed. Don't know where they got their result from though. Who said I agree? This whole averaging business is silly.

I'll throw the onus on you. Justify it. Then I'll refute you. :)

[trbarry]

@trbarry: I use pavgb now - but I can't see why you would subtract 1? Just using pavgb twice should produce correct rounding.

There is a white paper somewhere on the Intel site that discusses this issue but today I couldn't find the darn thing. I'll try to reconstruct as I see it.

Say you wanted the average of 4 values a,b,c,d. First you take the avg(a,b). 50% of the time the sum a+b will be even and after adding 1 and dividing by 2 the answer will be the same as if you did it with real numbers. Error = 0

But the other 50% of the time a+b will be an odd number. After you add 1 and divide by 2 your answer will be too high by 1/2 over the real number answer. Error = +1/2.

So over the long run with random input there is a .25 upward bias by rounding up this way. And if both avg(a,b) and avg(c,d) are too high by .25 then pavg( pavgb(a,b), pavg(c,d)) is too high by a expected average of .5 after the second round of averaging.

But when you use pavgb if you first subtract 1 from one of the numbers then you are subracting .5 from the answer. Combined with the .25 upward bias of the instruction this means a .25 downward bias, or the same as rounding downward instead of up. So every second time when repeating pavgb combinations on the same values it's more accurate to round down this way.

Note in my example from the post above the second pavgb was applied not to the result of 2 previous pavgb instructions but only to one and one unaveraged number. Thus my final result was actually low by an expected 1/8, but still better than high by 3/8.

- Tom

[sh0dan]

@neuron:

Please - I don't want to get into that discussion again. Look at:

http://www.hometheaterhifi.com/volume_8_2/dvd-benchmark-special-report-chroma-bug-4-2001.html See "Encoding 4:4:4 to 4:2:0" and "Converting 4:2:0 Back to 4:2:2 or 4:4:4". Also look at the Last pages of the Sticky "Video Frame Properties" threads.

In short, there are pros and cons of both. I get VISIBLY better results on chroma intensive material using interpolation.

@trbarry:

Did a small test program (accumulating all errors between the two routines and float values), and it seems like you (and Intel) are right - there is slightly less bias, when subtracting one before the second average. Except for the slight memory read pentalty, there is no problem implementing it into the current conversion routines.

No manual rounding: Average bias: +0.3750
Manual rounding: Average Bias: -0.1250

Well spotted!

[sh0dan]

I implemented the new testing, and made a minor adjustment to progressive YV12 -> YUY2 upsampling.

PSNR is about 55-60dB even after 5 conversions back and forth.

[Guest]

@sh0dan

I know all about those links. If you're not interested in having your dogmas challenged so be it.

Still for the record... Sampling 80-20 or 75-25 when doing interlaced YV12 chroma subsampling is just wrong. That is not to say that people don't do it. But more and more people are realizing how wrong it is and they're stopping it. Whether our upsampling should allow for clips done the wrong way as well as the right way (possibly via an option) is another matter.

[sh0dan]

That is not to say that people don't do it. But more and more people are realizing how wrong it is and they're stopping it.
Then please provide us with references. Right now we are getting nowhere with this discussion.

[Guest]

Originally posted by sh0dan
Right now we are getting nowhere with this discussion. I'm simply raising the point that there are different subsampling schemes used in the real world.

Unfortunately my references are STMicroelectronics and customer proprietary information. I will try to find public-domain citations for you. I remember seeing one and I'll try to find the link again.

You can do what you want but you have to acknowledge that there are 50-50 samplers out there. If the upsampler always assumes 80-20, that is wrong. Should it be configurable?

[sh0dan]

Originally posted by neuron2
You can do what you want but you have to acknowledge that there are 50-50 samplers out there. If the upsampler always assumes 80-20, that is wrong. Should it be configurable?

Yes - there are 50/50 upsamplers out there. There are also 100/0 upsamplers outthere. We can also agree that there probably is a lot of YV12 (or I420) material out there, that has been created with slightly displaced chroma.

IMO however AviSynth should assume that material is delviered as according to the specs. In the specs, there is no doubt that YV12 and I420 have chroma placed between luma lines. Therefore a 25/75 is the correct upsampling method for this.
If chroma is NOT placed where it should, it is not the job of the converter to correct this.

Can we agree on some of the above?

PS. I feel this is becoming one of the "arguments for the sake of arguments" discussion. Kinda like Hifi-nerds discussing gold-cables and green markers on their CD's.

[MfA]

Where does this 80:20 mix come from BTW? H.263 specifies a shift of 1/4 pel, so I dont see why the 75:25 mix is termed an approximation (strangely enough the MPEG4 standard while showing a vertical shift does not specify the exact sampling position of chroma samples, but given the backward compatibility to h.263 we can take that as saying MPEG4 uses the 1/4 pel shift too).

[sh0dan]

Where does this 80:20 mix come from BTW?

Probably some strange place in my mind ;)
Not sure where I got it from - just ignore it, and think 25/75 instead :)

[MfA]

I think it is all MPEG's fault, at least ITU saw the ludicrousness of leaving something this fundamental undefined ... so they just put the quarter pel shift in an annex.

AFAICS MPEG not only forgot to define it in their MPEG-2 standard (beyond saying and showing it is neither halfway between luma samples nor co-sited) but has been trying to pretend for all these years that simply no problem exists by not acknowledging it ... leaving the industry and us to sort out the mess.

Marco

PS. does anyone actually understand MPEG's justification for the positioning BTW?

In each field of an interlaced frame, the chrominance samples do not lie (vertically) mid way between the luminance samples of the field, this is so that the spatial location of the chrominance samples in the frame is the same whether the frame is represented as a single frame-picture or two field-pictures.

I just dont see it.

PPS. on second thought, MPEG probably didnt forget to define it ... it probably was just as controversial a topic at the time as it is now, and they didnt have the guts to make a decision :)

PPPS. looking around I see that Philips uses vertical chrominance filtering in their encoder ICs for 4:2:2 to 4:2:0 conversion, which assumes neither a co-sited nor a half-way sampling position (although the exact position they assume cannot be deduced from the datasheet). Also the MSSG MPEG-2 encoder based on TM5 (an unofficial document where MPEG did specify the sampling position and interpolation) uses interpolation for 4:2:2 to 4:2:0 conversion which assumes a 1/4 pel shift. I think a statement that both them, Sh0dan and a lot of other people are wrong based on some internal info from STM is silly ... there is no absolute right or wrong here. It is non normative for MPEG-2 (for MPEG-4 our friends at ITU have thankfully forced through a usefull standard ... and for it 75:25 interpolation is more right than 100:0 or 50:50).

[MfA]

It would be nice to have a util which used correlation of edges between luminance and chrominance planes to find what kind of interpolation has been used on existing 4:2:0 sources.

BTW, the arguement that everyone should do it wrong because the majority does it wrong is a reasonable one ... if MPEG leaves too many things in the standard open-ended it is up to the industry to reach a consensus. In this case the consensus does not seem to have been reached entirely, so to me it seems best to leave it up to the user. Philips seems to disagree with STM, as does the closest thing to MPEG-2 reference software which is available in the open (the decoder also assumes 1/4 pel offset).

[Guest]

H.263 does not define interlaced encoding. So taking it as a guideline for that and concluding that 75-25 is correct is the error I am talking about.

Think about it. If you encode a field picture, the chroma is sited between two adjacent lumas of that field. That is a 50-50 sampling. There is no justification for doing anything else, other than that so many have done it wrong.

I am curious to know whether it is worse to upsample a 50-50 as a 75-25, or to upsample a 75-25 as a 50-50. Perhaps some experiments are in order. But it seems to me that the best implementation would support both.

[MfA]

As I said ... in an annex. It is the closest thing to a specification we have, since MPEG seems so determined to be vague (well apart from TM5).

W.6.3.11 Interlaced field indications
<snip>
interlaced field coding of a top field picture are specified as shifted up by 1/4 luminance sample height relative to the field sampling grid in order for these samples to align vertically to the usual position relative to the full-picture sampling grid. The vertical sampling positions of the chrominance samples in interlaced field coding of a bottom field picture are specified as shifted down by 1/4 luminance sample height relative to the field sampling grid in order for these samples to align vertically to the usual position relative to the full-picture sampling grid. The horizontal sampling positions of the chrominance samples are specified as unaffected by the application of interlaced field coding

Also the MPEG standards clearly are indicating that the sampling position is not at the halfway position ... the piece of text I quoted earlier was from the MPEG-4 standard. Once more :
In each field of an interlaced frame, the chrominance samples do not lie (vertically) mid way between the luminance samples of the field, this is so that the spatial location of the chrominance samples in the frame is the same whether the frame is represented as a single frame-picture or two field-pictures.

When you say that the vast majority of decoders out there assume 50:50 Ill believe you ... but it aint consistent with the standard.

[Guest]

When you say that the vast majority of decoders out there assume 50:50 Ill believe you ... but it aint consistent with the standard. I didn't say that. :)

[TheRealMoh]

@neuron2

H.263 does not define interlaced encoding. So taking it as a guideline for that and concluding that 75-25 is correct is the error I am talking about.


75/25 applies equally well to interlaced as well as non-interlaced as long as we keep our heads clear and do interlaced with:
separatefields().convert().weave().

My math above is purely progressive and it does not invoke any arguments by authority (standards). BTW, you haven't said one word about what may be wrong with the math.

@everybody

Looking at the equations again, it's easy to figure out what may give rise to a 50/50: bad choice of YUV->YV12 conversion. If YUV->YV12 conversion is done via throwing data away, in other words, newV1 = V1 and newV2 = V3, then reconstructing V2 is best done with V2 = (newV1 + newV2)/2. In other words, a 50/50 average.

So the question becomes, how do hardware and software codec convert YUV->YV12? At this point, the question of standards does come into play. If many codecs throw data away, then avisynth should provide a choice to support the (broken) codecs. I also think those codecs should be fixed.

[trbarry]

I agree that if we can isolate one or more very common pathological YUY2->YV12 conversions it might be nice to implement special options to deal with them.

But I also agree with Sh0dan that without any special information we should assume it follows the spec and assume the 1/4+3/4 averaging.

- Tom

[MfA]

Hmm just noticed that an old version of the MPEG-2 reference software is actually a publicly available standard ... and it is mostly identical to the MPEG Software Simulation Group TM5 software. It has the same "FIR filter with 0.25 sample interval phase shift" in the the interlaced 4:2:2 to 4:2:0 conversion.

[Guest]

I think TheRealMoh has cut through the confusion (at least my confusion). He asserts that sampling 50-50 and upconverting 75-25 is always the right thing to do, progressive or interlaced. I've now looked at his math and I do find it convincing. I've been arguing that sampling 50-50 is the right thing to do for interlaced, which it appears is correct, even though a lot of implementations sample 75-25. That is what I have been saying is wrong.

Where I went wrong was thinking that 50-50 sampling required 50-50 upsampling.

So we have been arguing past each other. sh0dan was staunchly defending 75-25 upsampling (correctly), and I was staunchly defending 50-50 interlaced sampling. TheRealMoh showed that those are not incompatible!

I hope we can all get along now. :)

[TheRealMoh]

I've been arguing that sampling 50-50 is the right thing to do for interlaced, which it appears is correct.

As long as you mean 50/50 on alternate, not consecutive lines (which is really the same as separatefields().convert().weave()), then we're in agreement.

We could throw a monkey wrench into this whole thing and do motion mask upsampling where areas that are motion-free are done differently, using the information from the other field :) But that's too complex and confusing.
:devil:

[MfA]

I am pretty sure he agrees with you ... I wouldnt call the lines alternate myself though, they are consecutive inside the field. How fields are stored is best ignored, it only interferes with an intuitive understanding of the sampling (IMO storing 2 fields from an interlaced sequence in a single frame in an interleaved manner is a daft thing to do in the first place, but hey ... MPEG does it, so who am I to disagree).

BTW, this method of downsampling might be correct .... but only in the "in total disagreement with the reference software and the standard" sense of correct :)

[Guest]

Originally posted by TheRealMoh
As long as you mean 50/50 on alternate, not consecutive lines (which is really the same as separatefields().convert().weave()), then we're in agreement. Yes, of course, 50-50 between adjacent lines of a field.

@MfA

There is ample precedent for "reinterpreting" or ignoring standards when to do so brings substantial gains.

[Guest]

@sh0dan

Here is the promised link. You don't think I make these things up do you? :)

http://members.aol.com/ajaynejr/vidbug2.htm

Scroll down to the section on the "Interlaced Chroma Problem". It speaks of the 75-25 sampling as a fallacy and points out that many production houses nevertheless use it. Furthermore, he describes the inherent problem of interlaced YV12 and shows that doing 50-50 sampling minimizes its effects.

He also discusses TheRealMoh's point about the possibility of using motion-adaptive upsampling.

It is for these reasons, I believe, that 50-50 sampling is becoming more common, notwithstanding the specification.

Finally, check out the whole of the linked site. The video section is very good and the rest is hilarious. I love the article on "post anal drip". :)

[MfA]

That might be so, but why would you assume it would look better?

If you use the down/upscaling from the reference software you will get a higher color accuracy on every second line in the field ... and in the next field the color accuracy will have an opposite phase. This in itself is the same sort of trade off as interlacing.

Marco

PS. This (http://standards.pictel.com/ftp/video-site/0201_Gen/JVT-B116.doc) is also interesting, it could be another reason for chroma bugginess ... the problem will occur for both 1/4 and 1/2 sample phase shift though, it is caused by the skipping of every second chroma value not its precise location, so it isnt entirely relevant.

[trbarry]

All the details aren't fresh in my head anymore but I think when upsampling interlaced 4:2:0 to 4:2:2 there are two different ratios used. Depending upon whether your luma line has a croma line next to it you use either a 7/8+1/8 or a 3/8+5/8 ratio.

My reasoning was always based upon linear interpolation where the weights you give any sample point are inversely proportional to the distance from that sample point.

You can see the way I view it and my long winded explanation with pictures in this old MPEG2DEC3 for YV12 (http://forum.doom9.org/showthread.php?s=&threadid=36942&perpage=20&highlight=yv12%20chroma&pagenumber=10) post.

- Tom

[Si]

@trbarry
I think I've finally clicked with YV12 chroma placement thanks to your diagram and explanation.

The only trouble is that it is so clear, concise and reasonable that I wonder if you've got it right :)

regards
Simon

[TheRealMoh]

I have to say that I've never been comfortable about this pixel positioning thing that people talk about. It seems topsy-turvy to me, not to mention confusing. I tend to look at things from a very basic point of view. Basic, as in basic science.

Light is always measured over an area, not a point. A pixel is typically a rectangular area. In the RGB format, each component represents the same area (details of CCD's not withstanding). AFAIK, the only measuring devices are RGB (not YUV).

When RGB is converted to YUY2, the V and U values represent an area that covers two Y values. The V value is not somehow "in between" the two Y values. From a purely mathematical point of view, the question really is, is the averaged V the same as if it were measured in between the two Y values? The answer is pretty straight forward: in most cases, NO! (It is only true if the V component varies linearly over the area).

Converting YUY2 to YV12 is in a sense no different that RGB->YUY2: each V value represents the area of 4 Y pixels. I tried to draw it (see below).

Of course, for interlaced encoding things are more complicated, but I maintain that the correct way to deal with interlaced is to
separatefields().convert().weave().

[MfA]

If you dont want to assume linearity then how do you justify 75:25 interpolation during UPsampling? :) You should be advocating straight copies.

The assumption of linearity is a simplification of an assumption that the underlying signal was band limited before sampling BTW (using this stronger assumption it is easy to see why the longer FIR filter should be better than simple weighted averaging between 2 samples).

[TheRealMoh]

@MfA

I was hoping someone would ask that question :)

When we're writing code to upsample YV12->YUY2, we have to choose an algorithm (100/0, 75/25, etc). No matter what we choose, it will not restore the original data: it's an approximation.

However, when we're talking and thinking about the problem, there is no particular reason to approximate. In fact, the point of my post was to show that I find it easier to think in non-approxiate terms instead.

But you do raise a good point, one that's been lingering in the back of my head: what happens when the linear approximation breaks down? The best way I know to tackle this is with an example. Let's take an important case that is non-linear: an sharp edge. After all, edges make a picture what it is. Edges also present the hardest problems for codecs (even analog ones): that's why video test patterns have lots of sharp edges between various colors.

Here is the example. Let's start with 6 rows in YUY2 with these V values: 100, 100, 100, 200, 200, 200. There is an edge on an odd pixel boundary. Let's downsample that to YV12 (50/50, of course): 100, 150, 200. Now let's upsample using 3 algorithms:

orig: 100, 100, 100, 200, 200, 200
100/0: 100, 100, 150, 150, 200, 200
75/25: 100, 113, 138, 163, 188, 200
50/50: 100, 125, 150, 175, 200, 200

So which one is best? 100/0 creates a step. 75/25 is a soft edge, 50/50 bleeds too much on the low end. None of them are great, but I think 75/25 is no worse than any other.

[trbarry]

That also serves as a great example of why I like to stay in YV12 all the way through unless I really want the smoothing.

- Tom

[MfA]

MSE wise the straight copy clearly wins.

[avih]

ladies (?) and gentlemen, this thread is one of the finest on the board ;) i find great pleasure in following it.

[MfA]

So ... how many reasonable conversions does that make to support? :)

For downsampling :

Standard compliant :
FIR filter resampling
3/4:1/4 average
Maybe copy closest

Non standard compliant :
1/2:1/2 average

For upsampling :

Maybe copy closest (doesnt really need to assume any sampling position for upsampling)

Standard compliant :
FIR filter resampling
5/8:3/8 average

Non standard compliant :
3/4:1/4 average

Default should probably be the FIR filters from the MPEG2 reference code (with smart code you are probably near memory bandwith limited whatever method you choose, so speed is not an issue). Did I miss any?

Oh and a filter which detects the most probable downsampling method for a frame would be nice ;)

[TheRealMoh]

Take a look at this:
http://ce.sharif.edu/~ce712/materials/mpeg2.pdf
It claims that for YUY2 (4:2:2), the chroma values are co-sited with every other luma value. Does that mean in RGB->YUY2 conversion, values are thrown out?

It also puts the chroma values 50% between luma value for 4:2:0. That can only mean 50/50 downsampling. Can the reference code be wrong?

looks like XVID is using 50/50 downsampling:
http://edu.bnhof.de/pipermail/xvid-devel/2003-January/001867.html

[trbarry]

I didn't wade through all the code but I think a picture in the Xvid document is wrong, or at least misleading in where the YV12 chroma bytes are:


basically for a four block RGB pixel this gives:

RGB RGB Y Y
will become UV
RGB RGB Y Y


This seems to imply the UV pixels are horizontally centered between the RGB pairs after downconverting to YV12, which is not true at least for MPEG2 or MPEG4. It should look more like the horizontally co-sited:


RGB RGB Y Y
will become UV
RGB RGB Y Y


- Tom

[MfA]

TheRealMoh, yes with 4:2:2 every second sample in horizontal direction is simply dropped.

As for the MPEG2 reference code ... I found a copy of the MPEG2 standard and it says the exact same thing as the MPEG4 standard :

In each field of an interlaced frame, the chrominance samples do not lie (vertically) mid way between the luminance samples of the field, this is so that the spatial location of the chrominance samples in the frame is the same whether the frame is represented as a single frame-picture or two field-pictures.

Really, between this and the reference code the standard way of doing things is pretty clear.

Xvid does seem to use a 50:50 average for 4:2:2 to 4:2:0 conversion ... dunno if the RGB to 4:2:0 conversion is as buggy as the documentation trbarry quoted suggests, the macro'd C version of the conversions gives me a headache and the assembly code is a bit too undercommented and dense for me to decipher now.