color space conversions problems

[dividee]

I did a test and chained about 20 conversions from RGB To YUY2 and back (using the ColorBars filter as source).

Avisynth has an MMX routine for converting to RGB, but also a C routine which is called if the clip width isn't a multiple of 4.

col=ColorBars(320,240)

col.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB
ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB
ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB
mmx=ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB

col2=col.AddBorders(0,0,2,0)

col2.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB
ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB
ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB
nommx=ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB.ConvertToYUY2.ConvertToRGB

s1=StackHorizontal(col,col2)
s2=StackHorizontal(mmx,nommx)
StackVertical(s1,s2)


The two clips on top are the originals, the MMX routine is bottom left and the C routine bottom right.

Colors are OK, but what you'll notice if you run this script is that with the MMX routine, chroma is shifted to the left quite heavily. So I thought the MMX routine was faulty and started investigating. But that wasn't so simple.
In YUY2, you have 2 luma samples for one chroma sample. The chroma sample isn't placed between the two luma samples, but on the left one:
(Y,U,V) (Y) (Y,U,V) (Y)
and not
(Y) (U,V) (Y) (Y) (U,V) (Y)
When converting to RGB, the missing chroma samples should be interpolated... And this is exactly what the MMX version of ConvertToRGB does! The C code doesn't go to such lengths and just use the given chroma for both pixel, and is therefore less precise. The problem then comes from the RGB-->YUY2 conversion. It uses averaging, as if the chroma sample was in the middle. So basically, each time you use ConvertToYUY2, you shift the chroma information by half a pixel to the left (in addition to lowpassing it).

Probably the "right thing" to do would be to change the C version of ConvertToRGB and modify ConvertToYUY2 to only use the left pixel of each pair for chroma. I tested it and it solves the problem above and seems OK. But i'm a bit reluctant to change something that noone ever complained about.

[Acaila]

But i'm a bit reluctant to change something that noone ever complained about.
You feel better about leaving a piece of code in that you know is faulty? Can't say I've ever noticed this, but bug squashing is always good.

[trbarry]

Probably the "right thing" to do would be to change the C version of ConvertToRGB and modify ConvertToYUY2 to only use the left pixel of each pair for chroma. I tested it and it solves the problem above and seems OK. But i'm a bit reluctant to change something that noone ever complained about.

To me your fix sound like the proper way to go, especially if you can already see it gives better results.

Good work. :)

- Tom

[dividee]

Thanks. I guess I was afraid that discarding the chroma info of half of the pixels would somehow cause more information to get lost than averaging.

[Richard Berg]

> Ok I tracked down some documents. The UV is aligned with the first Y for
> YUY2. (Oddly it seems that for YUV4:2:0, UV is centered between the Y's
> for jpeg and mpeg1, and it is aligned with the first Y for mpeg2.)

I actually don't know of any definitive spec anywhere that says what
the chroma positioning is for various formats, but the BT878A specs
state that that particular capture chip outputs it coaligned and with
headroom, so I've considered it a fairly safe assumption. Chroma
interpolation for 4:2:0 is actually pretty expensive in software,
particularly for MPEG-1 positioning.

> This is not the 134 you started with. What is happening is that you are
> applying a horizontal low-pass filter to the chroma. You could probably
> correct this by using more sophisticated interpolation filters but I
> don't really want something that slow. And I am not convinced it would
> be of much benefit, since you are going to convert back to YUY2 in the
> end.

All resampling filters are low pass filters, but yes, the triangle
filter definitely isn't the best.

The non-interpolated way is essentially a box filter, so it is
theoretically worse when you have image processing in the middle. I
suppose the shift is of neglegible consequence for a noise filter,
although it would be much more important for titling or composition.

If you are doing block matching for your noise filter, though, why
don't you keep the UV planes at half size? A very common heuristic for
MPEG encoders is to avoid doing half-pel checks until a reasonably
good minima has been found with full-pel scanning. You can also keep
the UVs interleaved, which would speed up the conversions to and from
YUY2.

--
Avery Lee <phaeron@virtualdub.org> http://www.virtualdub.org
The latest version of VirtualDub is V1.4.7 (build 13130).

[tenebrenz]

I guess I was afraid that discarding the chroma info of half of the pixels would somehow cause more information to get lost than averaging.
Especially at lower resolutions, or in finely detailed area's, discarding half the chroma could be a real (visible) problem. I remember a website which showed conclusively that using a codec which took the top left pixel for chroma when dealing with rgb input definately produced visible problems compared to an averaging method. Hmm.

[dividee]

Thanks Richard for digging out that post from the avisynth-dev mailing list.
So discarding isn't a good idea.
But at least we should use a triangle filter [1 2 1] instead of the current [0 1 1] don't you think ? That's what Avery recommends in the post preceding the one you quoted for converting 4:4:4 --> 4:2:2

[trbarry]

The reason I was in favor of using the even bytes is because you sort of have to figure where the data first came from. If the data started on DVD or HDTV then the odd chroma bytes were just created by interpolaton anyway.

So while it might be mathematically correct to always average I'm a little wary of repeatedly averaging in data that was already created by averaging. The will accumulate softness.

So what percentaqe of our data actually started out life as a 4:4:4 source? ;)

- Tom

[poptones]

Ummm, pardon if this seems overtly simplistic, but why not just do it the same way the mmx routine does? You're not "averaging" data that's not already "averaged." I've played with consecutive yuy>rgb conversions and even after twenty steps found in=out pretty much exactly.

I never noticed the odd size problem, I guess because they're so rarely used. The mmx routine could pretty easily be adapted to process two pixels at a time, which is mandatory for YUY images anyway. At that point this is not an issue at all, since we know it works properly within all legal parameters.

[dividee]

@poptones:
The problem isn't really about odd size & ConvertToRGB (which is easily fixed), but in ConvertToYUY2. If you played with 20 conversions and didn't see the chroma shift, it was probably because you didn't know what to look for. For film content it's not as visible as with color bars, but it's there (look below).

@Tom:
I get your point. I tested both approach (triangle filter and discard)on film content and couldn't tell the difference even after 20 conversions. Anyway we can have both, if we add an argument to ConvertToYUY2. I'd default it to the discard method, since it's faster and probably good enough.

[ARDA]

@dividee

From the point of view of a newbie.I should say and I think you will
agree with me that before changing something in avisynth; why shouldn't you
release a plugin ? ; or two with a new name something like GotoYUY2 with two
or three parameters in a way many of us could make some tests and help
in such a difficult decision.Just a quick idea.

Thanks once more. Arda

[poptones]

Actually, I tested it (this was long ago, note) by doing a bunch of ops then subtracting - as in the example in the avisynth primer. And differences at all would be obvious, even shifts in hue.

What I mean is the problem isn't that it's done this way and that, the problem is it's not done the SAME way in both cases. It's not reciprocal. Within the app itself what's most important is the operations are as reciprocal as possible, not that they are "perfect."

What is wrong with interpolating the pixels so long as one does it both directions the same way? The errors would be cumulative, but not in the way shown here. They would be "normal" rounding errors.

Why not "lock" the left pixel (or the center, or the right), then interpolate? Make it reciprocal. Then there would be no shifting, and no smoothing.

Likewise, what I meant about the mmx routine was just to make it "correct" so it's not an issue anymore. It always works one way no matter how many pixels the screen is.

It seems like consistency is the problem here, not that one way is always technically superior to the other. If it were consistent there would be no need to add more potentially confusing "switches" to otherwise trivial operations.

[bill_baroud]

hmm, when i see the screenshot ot dividee, that's remain me a artefact i've got on very poor source (like HK DVD )

... it's there a way to shift the Chroma at this orignal position, or a better position than the actual one ?


thanks anyway for all your works.


bill

[dividee]

There is more than one way to make it reciprocal:

Option 1) Use left pixel (RGB-->YUY2) and interpolation (YUY2-->RGB)
+ Best for 4:2:2, 4:1:1 and MPEG2 4:2:0 sources
- Can create small (one pixel) visible artefacts for other material (with subtitles, for example)

Option 2) Mean (RGB-->YUY2) and copy (YUY2-->RGB)
+ Best for RGB, 4:4:4 and MPEG1/H.263 4:2:0 sources, and material that has been resampled or heavily filtered in one of these modes.

Option 3) Triangle filter (RGB-->YUY2) and interpolation (YUY2-->RGB)
= Compromise between the 2 above, but less reciprocal (altough much better than current situation).

And who's gonna make more than one or two conversions back and forth anyway ?

[EDIT:]
It would be intersting to see what other softwares uses for these conversions. I already checked that Virtualdub uses "copy" for YUY2-->RGB and DVD2AVI uses "interpolation" for the same process. Didn't see an example of RGB-->YUY2 conversion, though.

@bill_baroud:
I can't think of a way to undo that effect. The term "shifting" is maybe not the best, it's more like "bleeding".