ConvertToYV12 - bug or feature?

[au]

Hello, guys!

I'm trying to encode CG-sequence using x264 & AVISynth. So, there is a strange bug with ConvertToYV12() function.
AFAIK, in theory it just reduces chroma resolution with 4:2:0 sampling. But then I try to convert 24bit sources to YV12 colorspace, strange posterization effects appear. It looks like color depth has been reduced due to inproper function of converter.

I did such transformation in eyeon fusion (RGB24->YUV420->RGB24) and there was no posterization.

So, what's wrong with it?

Here are the crops of source(!RGB24.JPG) and converted output(!YV12.JPG):

[Dark Shikari]

Of course, you're using JPG for your pictures... and JPG doesn't support the RGB colorspace if I recall correctly :p

[au]

:) Right.
I've got the same result with uncompressed TGA and x264 output. So, the RGB->YUV->RGB is done with color degradation.

[MfA]

Maybe, or maybe you are doing something wrong. If you aren't paying a developer to listen to bad bug reports they very often simply won't. For open source software especially good bug reporting (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html) is the best way of getting heard. In this case good bug reporting would be to give the avisynth version, the script, and an example input for which the problem occurs.

Lets sidestep all that for the moment though, did you try adding converttorgb at the end of your script and verifying the problem is still there?

[foxyshadis]

It's because 8-bit RGB value can't be represented in an 8-bit YUV, and vice versa, without roundoff errors. This is why 10-, 12-, and 16-bit are so popular and so often requested for avisynth. You can make a table of all 16 million possible combinations, run them through the RGB->YUV equations and back, and see that most will have at least one value off by 1. In rare cases it can even be off by more.

[Konrad Klar]

Here are the crops of source(!RGB24.JPG) and converted output(!YV12.JPG):

I see only two completely black images (in FF 3b3, K-Meleon, IE7).

[Leak]

I see only two completely black images (in FF 3b3, K-Meleon, IE7).
Turn up your screen's brightness... :)

np: The Orb vs. Meat Beat Manifesto - 1855 BC (Battersea Shield)

[sh0dan]

You will never get an exact match, but you can use ConvertToYV12("PC.709") and ConvertToRGB("PC.709") to maintain a larger precision.

Also - you might check that Virtual Dub is not is in it's (default at one time) 16 bit mode previews.

[au]

It's because 8-bit RGB value can't be represented in an 8-bit YUV, and vice versa, without roundoff errors.

As I said before, I've done such convertion in Eyeon Fusion (pipeline compositing software) using custom tool. I wrote formulas for each channel by myself. So, there was no posterization after RGB-YUV420-RGB transcoding in 8bit color mode. I've expected some kind of 2x2 or 4x4 blocking, but it did not appear.

You will never get an exact match, but you can use ConvertToYV12("PC.709") and ConvertToRGB("PC.709") to maintain a larger precision.

Invalid arguments to function "ConvertToYV12()" in 2.5.7 version.
Which type of precision do you use in ConvertToYV12 function? Integer or Float?

Lets sidestep all that for the moment though, did you try adding converttorgb at the end of your script and verifying the problem is still there?

Of course, I did. Here is the sample of RGB24->YV12->RGB24 transformation (see attached picture).
Moreover, x264 output gives me the same posterized picture - that is the main problem.

[sh0dan]

Sorry - meant ConvertToYV12(matrix="PC.709") ConvertToRGB(matrix="PC.709")

Which type of precision do you use in ConvertToYV12 function? Integer or Float?
We use "enough" precision. The problem isn't the internal precision, but the precision of the YV12 data, which is 8bpp.

By default that is reduced to about 220 values in the Rec601 TV range, which is why you should use Rec709 in PC Range, which is full 255.


Edit: Use options "--fullrange on --colormatrix bt709 --colorprim bt709 --transfer bt709" in x264 (some may be redundant).

Edit 2: You could also add a "graining feature", that adds grain to your video on uniform areas. I drafted something up:
v = last
grained=blankclip(v,pixel_type="yv12",color_yuv=$7f7f7f).addgrain(1.0,0,0)
m = mt_edge(v,mode="prewitt",thY1=0,thY2=10).invert().mt_inpand().mt_deflate()
mergeluma(v,overlay(v,grained,mode="softlight",mask=m))

#m.greyscale() #Uncomment to see mask

Adjust "1.0" in addgrain to adjust the amount of noise. Adjust "thY2" to adjust sensitivity - uncomment m.greyscale to see the mask. Requires masktools v2 and AddGrain.

[au]

Edit: Use options "--fullrange on --colormatrix bt709 --colorprim bt709 --transfer bt709" in x264 (some may be redundant).


thanks for advice!