Chroma shift in interlaced YV12 to YUY2 conversion

[Mini-Me]

BTW, speaking of ye good old ConvertBackToYUY2. What was its reason for existance? :p

To allow people to losslessly convert back and forth between YUY2 and YV12 200 times as part of a perfectly ordinary and logical workflow? ;) Point taken of course, but the real reason for repeated conversions in this thread is just to make flaws in the conversion routines more apparent.

I just realized something though: Although my "corrected" offsets smear chroma less (apparent with 200 applications ;)), it turns out that they are not "corrected" at all but wrong. They have another problem: If you perform a one-way conversion from YUY2 to YV12 or vice-versa without converting back, the chroma will bob from field to field, relative to the luma. (Converting back cancels out the bobbing and masks the problem, which is why I didn't recognize it earlier.) In contrast, Gavino's original offsets keep chroma perfectly aligned with luma, at least in the sources I tested. Since his offsets seem perfectly correct, my best guess is that the smearing comes from Avisynth's resizer core not playing as well with the smaller offsets...or something to that effect.

In other words, my doubled offsets are wrong, even though they don't cause smearing. I noticed the chroma bobbing in some of my own captures when I viewed them with separated fields, where bright reds especially would appear to slightly "swim" around the details of the image as they bobbed. I'm not sure whether deinterlacing with Q/TGMC would exacerbate or alleviate the chroma bobbing, but either way, it's not great news.

So, using Gavino's offsets results in gradual chroma smearing, and using my doubled offsets results in significant chroma bobbing from field-to-field. Instead, you can use the methods I included in my "ConvertAlmostEquivalent" function further up the page. They don't have perpetual chroma smearing, and they don't displace chroma relative to luma either.* If you want to do it this way, convert to YV12 by:
yuy2clip.SeparateFields.ConvertToYV12.Weave()
and you have to convert from YV12 to YUY2 with either:
yv12clip.ConvertToYUY2(interlaced=True) #, or
yv12clip.SeparateFields.ConvertToYUY2.Weave()
This is assuming the current ConvertToYV12 and ConvertToYUY2 implementations in Avisynth 2.6. I've edited this post and changed my mind about a half-dozen times, but now that I've sorted things out a bit better, I think the above methods are probably the most accurate, both for one-way and round-trip conversion. At the very least, I cannot find any issues with them.

Before editing this post again, I previously wrote that the "separated fields" methods bob chroma by a very subtle amount. Upon further testing, it turns out this is untrue, at least if you're using Virtualdub to view your test videos like I'm doing. The slight chroma bobbing I saw with these methods was already present in my YUY2 source, and it was not introduced by the "separated fields" YV12 conversion. (It's a long story, but my VHS captures are a bit of a pain sometimes. ;)) The bottom line is, the above methods will not perpetually smear, and they will also introduce no chroma bobbing (from what I can tell). Using Virtualdub as a player, the chroma is in the same place for YUY2 and YV12. In contrast, I just realized that Gavino's method actually does very slightly displace the chroma relative to the luma if you do a one-way conversion, but in my case it was actually fixing the error present in my source, at least for a one-way conversion. :o Using resizer offsets of 0.0 will eliminate the bobbing entirely, but 0.0 offsets greatly exacerbate the smearing, so Gavino's offsets appear to be a better compromise.

I still don't have a whole lot of certainty about all this though, because it ultimately depends on the assumptions your player makes about YV12 chroma placement. I mean, I'm using Virtualdub as my player, so if it upscales YV12 incorrectly, then I just came to the wrong conclusion about which conversion method bobs chroma and which does not. ;)

[Gavino]

Hi guys, I have been occupied elsewhere today so I've missed the latest round of postings from the last 24 hours.

I was pretty sure the offsets I used (worked out from first principles rather than by experiment) were the right ones to get the chroma placement correct, and I'm relieved to see that in the end Mini-Me agrees with that.

The various test results are interesting, and I will look into them in more detail tomorrow in an attempt to understand what lies behind them. What I can say is that the reason the 'double shifts' (though wrong) do not show any chroma shift on repeated conversion of YV12 to YUY2 and back is because the error in one side of the conversion is exactly reversed on the other, so there is no error on the 'round trip'.

BTW ConvertBackToYUY2 is only for converting from RGB, and cannot be used for YV12->YUY2.

[IanB]

Strip away all the gory details and you are left with :-BilinearResize(width, height/2, 0, -0.5).BilinearResize(width, height, 0, 0.25)Of course this is going to smear after lots of iteration. You might as well have strung 200 iterations of Blur() together. Doh!

[Mini-Me]

Strip away all the gory details and you are left with :-BilinearResize(width, height/2, 0, -0.5).BilinearResize(width, height, 0, 0.25)Of course this is going to smear after lots of iteration. You might as well have strung 200 iterations of Blur() together. Doh!

I guess it's really not surprising that it blurs with 0.125 and 0.25 offsets. What surprises me is that using 0.0 offsets makes it blur faster (rather than less or not at all), and using 0.25 and 0.5 offsets doesn't blur (although it bobs chroma terribly). Gavino mentioned the doubled offsets don't blur because the errors cancel each other out, but I haven't yet worked out how/why they cancel out when his own offsets and especially 0.0 offsets don't. (A lot of it is because I'm a noob; I'm still not sure whether the resizer offsets are given in terms of input or output resolution. :o)

Anyway, the more surprising thing is that clip.SeparateFields.ConvertToYV12.Weave() and clip.SeparateFields.ConvertToYUY2.Weave() work so well, because the progressive conversion used internally isn't even meant to be used on individual half-height fields (and Gavino said as much on page 1). I should be jumping for joy that they work without any apparent issues, but I keep suspecting there's some hidden snag waiting to bite me. It seems too good to be true, since it's giving me no perpetual blurring and no chroma bobbing. I feel like something has to be wrong with it, but it keeps giving me perfect results.

Technically speaking, Gavino's 0.125 and 0.25 offsets are supposed to be correct (and are definitely the best compromise for the functions on page 1), but I noticed that a one-way conversion actually introduces a very slight chroma bob from field to field if I compare the image between YUY2 and YV12 in Virtualdub (using two windows side-by-side and flipping between fields, since you can't actually stack or interleave different formats). Of course, the offsets shift the chroma on purpose to accomodate the chroma placement of the new format, but I assume they're not supposed to introduce a perceptible chroma bob from field to field, relative to the luma. The picture is supposed to look the same in both formats (or as close as possible, given half the vertical chroma resolution in YV12 and hairy interaction with interlacing). In contrast, simply separating fields and converting results in identical chroma placement in YUY2 and YV12, when I compare using Virtualdub.

So I guess my question is: What's going on? Is Virtualdub making unusual assumptions about chroma placement and upsampling from YV12 incorrectly for display (relative to MPEG standards and the behavior we expect from players)? If so, I can see why that would erroneously introduce chroma bob to Gavino's correct version and inadvertantly fix the hacky separated fields version. Or is Virtualdub upsampling from YV12 correctly, meaning the best results actually come from separating fields and converting the colorspace as progressive?

[Gavino]

Having now found time to look at your points in detail, I think the trees are preventing you from seeing the forest (so to speak) and you're looking at the wrong thing. Let's take a step back and revisit the problem we're trying to solve.

The original problem

At the start of this thread, I pointed out that Avisynth's method of converting interlaced YV12 to YUY2 introduced a chroma shift (which was in a different direction for each field). To make the effect more visible, I demonstrated the problem with a script that repeatedly converts interlaced YV12 to YUY2 and back, showing that a two-way (combed) chroma shift accumulates with each round of conversions.

As I also stated in that very first post, the expected result of a round trip YV12->YUY2->YV12 conversion is a slight blurring (aka smearing) of chroma (because of upscaling followed by downscaling). The error demonstrated by the original test script is the shift which occurs in addition to the (expected) blurring.

The workaround

Later (over 18 months later), when it became clear that the problem still existed in v2.60, I posted a workaround consisting of a set of script functions which could be used in place of Avisynth's built-in YV12<->YUY2 conversions.
[The original problem was just about interlaced YV12->YUY2, but the replacement functions also include YUY2->YV12 (both progressive and interlaced) with chroma anti-aliasing (emulating the 'manao functions'), as well as a progressive YV12->YUY2 for completeness.]

The 'offsets'

The functions use BilinearResize to do the chroma resampling, with appropriate sub-pixel offsets where necessary to get the correct chroma placement.
Note that a correct set of offsets must necessarily 'cancel out' on a round trip YV12->YUY2->YV12, as this has to return chroma to its starting placement. So if some offset y is used for upsizing (YV12->YUY2), it must be matched by an offset of -2*y for downsizing (YUY2->YV12).
(The factor of two is because the offset is expressed in terms of source pixels, and the distance between the upscaled pixels is half that between the downscaled ones.)

Conversely however, you cannot confirm the validity of the individual offsets by using them in pairs on a round trip - any pair of offsets y and -2*y will combine on a round trip without a net shift, even if these offsets actually give an incorrect chroma placement for the individual ('one-way') conversions. All you will see from this test is the expected chroma blurring (whose strength varies with the offsets - see later).

To check the individual offsets, you must modify the round trip test to combine my interlaced YV12->YUY2 function with the built-in ConvertToYV12 (which is known to give the correct chroma placement).
If you do that, you will find that my offsets work, and no shift is seen - with any other offsets, the combed shift is still present to some degree.

You can do the same test using your SeparateFields.ConvertToYUY2.Weave() for the YV12->YUY2 part and see that it also creates a combed shift. This is because it treats top and bottom fields the same way, which is wrong as they have different chroma alignment in YV12.
The error (here with chroma) is similar in nature to the one you get (with both luma and chroma) resizing an interlaced clip with SeparateFields.xxxResize.Weave (which produces an incorrect field alignment).

Other points

We are left to explain why some offsets blur more than others on the round trip.
This is simply a side-effect of where the resampling positions fall. I believe the reason an upscaling offset of (+ or -) 0.25 gives the smallest blur is that this preserves the original pixels when upscaling and so each pixel retains a higher proportion of its original value after downscaling again.
Since the Avisynth resizers maintain the image centre position, an offset of zero does not preserve the original pixels, so blurs more on the round trip. (Note that we don't want to preserve the original pixels when converting because YV12 and YUY2 have different chroma sampling points.)

Regarding VDub, I believe it always treats YV12 as progressive when converting to RGB (eg to apply certain filters or even simply for display), so interlaced YV12 is displayed incorrectly. See this post from IanB:
Over in VirtualDub land Avery Lee rigidly follows the fourCC definition for YV12 to the letter, ie. Mpeg2 chroma sitting, Black=16, White=235, Progressive picture structure, Rec601 colorimetry, etc ... It saves him a whole lot of grief from his users but it limits the available functionality. Here in Avisynth land we are a bit more laid back, we provide an internal format misnamed YV12. It provides 8 bit data planes with chroma information subsampled 50% both horizontally and vertically. That's it, everything else is freely a user choice. Various filters assume certain attributes about the data and it is the script authors responsibility to use or abuse those assumptions.

Also see here:
http://forum.videohelp.com/threads/309449-Adobe-Premiere-MainConcept-s-H-264-vs-x264-open-source-encoder?p=1952599#post1952599
http://forum.videohelp.com/threads/316415-Check-Over-My-Options-From-DVD-to-XviD-Conversion-Thread?p=1958565#post1958565

[Mini-Me]

Regarding VDub, I believe it always treats YV12 as progressive when converting to RGB (eg to apply certain filters or even simply for display), so interlaced YV12 is displayed incorrectly. See this post from IanB:


Also see here:
http://forum.videohelp.com/threads/309449-Adobe-Premiere-MainConcept-s-H-264-vs-x264-open-source-encoder?p=1952599#post1952599
http://forum.videohelp.com/threads/316415-Check-Over-My-Options-From-DVD-to-XviD-Conversion-Thread?p=1958565#post1958565

Aha! Thank you for your detailed summary, Gavino, and thank you especially for looking into the Virtualdub issue. Now that I know that Virtualdub's resampling is incorrect, everything makes way more sense.

I never fed Virtualdub actual interlaced YV12, so I never saw its chroma upsampling bug. Instead, I tested YV12 by separating fields before viewing in Virtualdub, so I could view each field clearly and individually and see how much the chroma would bob from field to field. Since Virtualdub always treats YV12 as progressive, it just so happened to precisely reverse my SeparateFields.ConvertToYV12 conversion...but that same process failed to correctly upsample your own conversion.

I know I've been a pain, but I appreciate your patience guiding me through this. I'll make sure to use your conversion routines from here on out! :)

[Wilbert]

Aha! Thank you for your detailed summary, Gavino, and thank you especially for looking into the Virtualdub issue. Now that I know that Virtualdub's resampling is incorrect, everything makes way more sense.

I never fed Virtualdub actual interlaced YV12, so I never saw its chroma upsampling bug.
To confuse things a bit more :) Avery Lee will tell you that Virtualdub is not incorrect, since YV12 is progressive by definition. What we call interlaced YV12 is actually interlaced 4:2:0. Not that the problem doesn't go away though :)

edit: typo

[Mini-Me]

To confuse things a bit more :) Avery Lee will tell you that Virtualdub is not incorrect, since YV12 is progress by definition. What we call interlaced YV12 is actually interlaced 4:2:0. Not that the problem doesn't go away though :)

Considering one of the most common video standards in the world (DVD) calls for interlaced 4:2:0 for interlaced sources, how did anyone think it was a good idea to omit interlacing from the YV12 spec in the first place? Granted, interlaced 4:2:0 is a mess, but the "official" YV12 spec is pretty incomplete if it doesn't allow straight encoding to common end-user formats (without another colorspace conversion). All standard definition TV content is interlaced, 1080i content is interlaced, and for a while hard-telecining was popular for DVDs. (DVD players and conforming MPEG2/MPEG4-ASP/MPEG4-AVC software and hardware players do all use the same chroma placement for interlaced 4:2:0, right?)

[IanB]

@Mini-Me,

Avery is 100% correct in his interpretation of YV12. The "official" fourcc YV12 spec is a complete definition of the data it represents.

The definition used in Avisynth is somewhat of a misnomer, in hind site we probably should have called the initial planar format just "4:2:0" as that is the only attribute that is guaranteed. Everything else is the responsibility of the script author.

None of the interlaced example you quote are YV12. Avisynth lets you import any 4:2:0 format as what we call YV12 and do useful things with that data.

[Mini-Me]

@Mini-Me,

Avery is 100% correct in his interpretation of YV12. The "official" fourcc YV12 spec is a complete definition of the data it represents.

The definition used in Avisynth is somewhat of a misnomer, in hind site we probably should have called the initial planar format just "4:2:0" as that is the only attribute that is guaranteed. Everything else is the responsibility of the script author.

None of the interlaced example you quote are YV12. Avisynth lets you import any 4:2:0 format as what we call YV12 and do useful things with that data.

I understand that Avery is technically correct, because fourCC YV12 is always progressive, and industry 4:2:0 formats are not actually called YV12. It just seems shortsighted for people to have defined fourCC YV12 without a specified interlaced mode in the first place. AFAIK YV12 is meant to match up with progressive 4:2:0 end formats, but interlaced 4:2:0 is also so common that the computer world needed a standards-compliant interlaced 4:2:0 format just as badly as a progressive one. (Obviously I'm not alone in thinking this, since you guys created an ad hoc/de facto extension to YV12 to address it. ;))

I also understand that if Avisynth is fed 4:2:0 input, there's no way of telling the chroma placement, and it just has to accept the chroma planes it's given. What ultimately matters to me is knowing whether Avisynth's conversion from 4:2:2 to 4:2:0 uses the same chroma sampling as my end formats'. Confusion aside, I'm not really concerned with what Avisynth's (or MPEG-2's, or MPEG-4 AVC's) 4:2:0 formats are called; they could be identified by YV12 or HACK or KRZY, but the important thing is that Avisynth's 4:2:2->4:2:0 conversion places chroma samples at the same locations that MPEG formats and software/hardware players are expecting. It's probably only a small minority of Avisynth scripts that are actually affected by chroma placement, but the final display is always affected by the manner of upsampling, which makes it a larger issue. They do match, correct? (I'm asking for confirmation, but it seems logical that Avisynth would have defined its 4:2:0 placement - upon conversion - to match up with the placement defined by common end formats.)

[Gavino]

the important thing is that Avisynth's 4:2:2->4:2:0 conversion places chroma samples at the same locations that MPEG formats and software/hardware players are expecting. It's probably only a small minority of Avisynth scripts that are actually affected by chroma placement, but the final display is always affected by the manner of upsampling, which makes it a larger issue. They do match, correct? (I'm asking for confirmation, but it seems logical that Avisynth would have defined its 4:2:0 placement - upon conversion - to match up with the placement defined by common end formats.)
The default(*) conversions are defined to use MPEG-2 chroma placement for 4:2:0.
The implementation matches the definition, except for interlaced 4:2:0 to 4:2:2 (the problem identified by this thread).

(*)In 2.60, you can specify the chroma placement explicitly (but the implementation is wrong in the current release).

[Mini-Me]

The default(*) conversions are defined to use MPEG-2 chroma placement for 4:2:0.
The implementation matches the definition, except for interlaced 4:2:0 to 4:2:2 (the problem identified by this thread).

(*)In 2.60, you can specify the chroma placement explicitly (but the implementation is wrong in the current release).

That's what I expected, so thank you for confirming. :)

[Mounir]

After reading all these posts i can certify that i'm totally lost.
My goal is to convert yuy2 to yv12

Should i use YUY2ToYV12_26(interlaced=true) (IanB script)
... or a simple ConverttoYV12(interlaced=true)

[Gavino]

After reading all these posts i can certify that i'm totally lost.
My goal is to convert yuy2 to yv12

Should i use YUY2ToYV12_26(interlaced=true) (IanB script)
... or a simple ConverttoYV12(interlaced=true)
The chroma shift problem only occurs going the other way (interlaced YV12->YUY2).
For YUY2->YV12, there is just a potential problem of chroma aliasing, but for many (perhaps most?) sources a simple ConverttoYV12(interlaced=true) will produce acceptable results. If not, then use my function YUY2ToYV12_26(interlaced=true). (post #16 (http://forum.doom9.org/showthread.php?p=1471623#post1471623))
(IanB's changes in post #20 were only for YV12->YUY2 and in any case require a fixed 2.6 release.)

I'm assuming from your question that you are using v2.6 - otherwise you would remove "_26", ie use function YUY2ToYV12.

BTW I'm not surprised you're confused. Every time this problem comes up, I find I've forgotten the details and it takes me a while to get my head around it again. :(

[ariga]

Maybe a bit off-topic,
Will AutoYUY2() be a suitable replacement to YV12ToYUY2() ?

[IanB]

It is off topic.

AutoYUY2() would be an excellent replacement for YV12ToYUY2(interlaced=True) but not for straight YV12ToYUY2() [progressive mode].

It is an advanced filter that tries to determine "static" versus "in motion" areas of a frame. For static areas it uses "progressive style" chroma interpolation which has double the vertical resolution i.e. YV12ToYUY2(interlaced=False). For motion areas it uses "interlaced style" chroma interpolation which has the normal half vertical resolution i.e YV12ToYUY2(interlaced=True) but with no bug. So for interlaced conversion it is probably a win on 2 fronts, 1. no bug and 2. advanced chroma interpolation for static areas.

[Mounir]

Today i have spotted a problem of chroma, please note i am using the v2.6.0.2 i'd like to know if this has been fixed or not in the latest avisynth release:
sample (yuy2, tff, interlaced, vhs):
http://www.mediafire.com/?716kzh9lacez3e5

I have spotted it because i did a ivtc (which i rarely do), there is chroma misplacement, it is ahead of 1 frame when i do the ivtc

the solution i use to fix that (note that i do yuy2>yv12>rgb32>yv12 conversions in most of my scripts):

avisource("...")
assumetff()

separatefields()
pointresize(last.width, last.height*2).converttoyv12(matrix="Rec601",interlaced=false).pointresize(last.width, last.height)
weave()

separatefields()
pointresize(last.width, last.height*2).ConverttoRGB32(matrix="rec601",interlaced=false).pointresize(last.width, last.height)
weave()


separatefields()
pointresize(last.width, last.height*2).converttoyv12(matrix="Rec601",interlaced=false).pointresize(last.width, last.height)
weave()


Note the placement of the separatefields() , must happen before the pointresize or it doesn't work otherwise(chroma still is ahead of 1 frame)

This way the result is lossless

Edit:
It seems i was wrong in a way, my capture has a strange fps (29.968 ) i don't know how that happened (!), hence the problem during the ivtc , the pointresizing don't hurt though

Any idea how i can have a 29.970 without re capturing, changefps do more harm than good it appears.
After the ivtc i end up with 23.974 fps, answer me in private 'cause it's off topic

[zambelli]

I nominate this thread to be a sticky. :) Thanks for the education, guys.

[real.finder]

now in avs 2.6 ConvertToYV16(interlaced=true) is ok as I see, but ConvertToYV12(interlaced=true) with yuy2 not yv16 not fixed

[Wilbert]

now in avs 2.6 ConvertToYV16(interlaced=true) is ok as I see, but ConvertToYV12(interlaced=true) with yuy2 not yv16 not fixed
I have no idea what you mean here. Please elaborate. What is your source and what goes wrong?

Also if your source is anything else then YV12, the interlaced option won't do anything in ConvertToYV16(interlaced=true).

[real.finder]

I have no idea what you mean here. Please elaborate. What is your source and what goes wrong?

Also if your source is anything else then YV12, the interlaced option won't do anything in ConvertToYV16(interlaced=true).

I just do a test with Gavino script in 1st post

function Convert(clip c, int times) {
times == 0 ? c : c.ConvertToYUY2(interlaced=true).ConvertToYV12(interlaced=true).Convert(times-1)
}

ColorBars(pixel_type="YV12").TurnRight()
Animate(0, 10, "Convert", 0, 10)

and edit it like this

function Convert(clip c, int times) {
times == 0 ? c : c.ConvertToYV16(interlaced=true).ConvertToYV12(interlaced=true).Convert(times-1)
}

ColorBars(pixel_type="YV12").TurnRight()
Animate(0, 10, "Convert", 0, 10)

and of course I also do an other edits as an experiment

and then compared the result with his YV12ToYUY2i and YUY2ToYV12i functions

and since the conversion between yuy2 and yv16 is lossless, we can do ConvertToYV16 then ConvertToYV12(interlaced=true) if we have an interlaced yuy2 source and we want converted to yv12

and in opposite case ConvertToYV16(interlaced=true) then ConvertToYUY2

[Wilbert]

I see. It's a bit late to fix this now for 2.60 ;( The reason that yuy2 and yv16 produce difference results is that they take different code paths. I wonder what kernel is used by YV16<->YV12. Must be tUsubSampling[4] and bUsubSampling[4] in ConvertToPlanarGeneric. Anyone?