Matching RGB->YUY2/YV12 results between Avisynth 2.5 and 2.6

[Robert Martens]

Is there any way in Avisynth 2.5 to get the same output from ConvertToYUY2 that you get from Avisynth 2.6?

I'm back to a seemingly endless quest to finish my current plugin development project, and part of that involves regression tests that track the MD5 hash of test scripts' output frames. I recently discovered that while the majority of my tests produce the same output whether run under 2.5 or 2.6, some involving the creation of YUY2 and YV12 clips are giving different output depending on the host.

According to this post (http://forum.doom9.org/showthread.php?p=1038027#post1038027), Avisynth <= 2.58 uses a 0-1-1 kernel for RGB->YUY2 conversions, which was implemented for performance reasons way back when. It works quickly, but causes a slight horizontal chroma shift in the process. In 2.6, however, it was decided that the performance penalty of an improved approach was acceptable, and the 1-2-1 kernel referred to in the aforementioned post was implemented for Avisynth 2.6 (http://forum.doom9.org/showthread.php?p=1644457#post1644457).

Having heard of this issue before, I began by assuming it could be the cause of the different results, and after eliminating my plugin and test executable from the equation, I thought I'd managed to piece together a solution:

base = BlankClip(width=1, height=1, pixel_type="RGB24")
a = BlankClip(base, color=$FF0000)
b = BlankClip(base, color=$00FF00)
c = BlankClip(base, color=$0000FF)
d = BlankClip(base, color=$FFFF00)
two = StackVertical(StackHorizontal(a, b), StackHorizontal(c, d))
four = StackVertical( \
StackHorizontal(two, two.FlipVertical()), \
StackHorizontal(two.Turn180(), two.FlipHorizontal()) \
)



avs26 = four.ConvertToYUY2(matrix="PC.601")

w = four.Width()
h = four.Height()
avs25_chroma = four.BilinearResize(w, h, -0.5, 0).ConvertToYUY2(matrix="PC.601")

avs25 = MergeChroma(avs26, avs25_chroma)

return VersionNumber() < 2.6 ? avs25: avs26

I expect this script to produce identical output whether run under 2.5 or 2.6, and it seems to. Saving the results as EBMP files using ImageWriter, then loading both results and using Subtract, shows they're the same. The MD5 hash is exact, as well. Opening the script in AvsPmod, zooming in with "Fit inside window", then hovering over each square shows the same YUV readouts, too.

Unfortunately it only seems to work for that one contrived, small example. When I replace 'four' with a more involved clip, for example this HSL plot (https://github.com/ItEndsWithTens/TurnsTile/raw/master/test/scripts/clips/hsl.png), I still see different results. Subtracting the 2.5 results from the 2.6 results shows that the Y components are identical at 126, but scanning across the frame in AvsPmod shows that U and V jitter between roughly 127 and 130, when they should both be 128.

Am I missing something in this process, or is there just no way to achieve the output of 2.6 in 2.5? For my purposes I have a few other options for getting around this (save a reference frame from 2.6 as an EBMP so I can bypass conversions, maintain separate test scripts and reference data for 2.5, force developers running the test suite to use 2.6, etc.), but for my own edification I'd like to know if I'm just stumbling over the implementation of the idea, and so far despite scouring old threads I haven't figured out what I'm doing wrong.

[Gavino]

I don't think there is a way to replicate the 2.6 results exactly with 2.5.
I think the differences you see are due to rounding errors.
By adding a BilinearResize() before ConvertToYUY2() on 2.5, you are successfully compensating for the chroma shift introduced by the 0-1-1 kernel, so the results will be close to those of 2.6. However, the intermediate results (from BilinearResize) will be rounded to 8 bits, so some accuracy will normally be lost in the combined process.

By the way, I am very impressed that you do such detailed regression testing - I wonder how many other developers go to the same lengths.

[Robert Martens]

That makes sense, thanks. I did get excited for a moment thinking I could just introduce those rounding errors on purpose for the 2.6 clip (since I don't really need accurate results in my case, just a consistent approach that gets me a YUY2 image with some sort of interesting content in it at low filesize and decent speed) with some combination of Bilinear and Point resizing, but it hasn't worked out so far. Getting down to the basics with a minimal script like

clip = ImageSource("hsl.png")
clip.BilinearResize(clip.Width(), clip.Height(), -0.5, 0)

and comparing hashes shows a difference between the two host versions. A crop of 1.0 is identical, but the subpixel stuff must round differently somewhere, maybe something to do with AsFloat returning an actual float in 2.6 instead of a double in 2.5? I don't know, I'm out of my league here, there have been a couple of decent sized changes in the resizing code since 2.58, and trying to trace the code is making my eyes bug out.

Well, it was just curiosity anyway, so thank you again for the insight, and for the compliment on my tests! To be fair, I don't really have to go to much of a length to get new ones up and running. I've got over two hundred scripts so far, but between Bulk Rename Utility (http://www.bulkrenameutility.co.uk/) for naming them, TextCrawler (http://www.digitalvolcano.co.uk/textcrawler.html) for multi-file regex search and replace, and Catch (https://github.com/philsquared/Catch) for simple C++ test cases, I can get them written up in no time. I only wish I'd been doing that since the start. This process has really helped me focus on the ideas behind the features, forcing me to think them through more carefully than I had before, and has exposed a few bugs: check the commit history (https://github.com/ItEndsWithTens/TurnsTile/commits/master) and you'll see a few such updates sprinkled through my recent test work.