Enconding in 444 YV24 from RGB source Avisynth.

[sephirotic]

I color graded a m2ts episode from a BD I own on Premiere CS6. Only truly lossless export Option I had was mov with animation codec (RLE).
Now, I wanted to convert to a lossless x264 file (qp0 preset slow) with 4:4:4 subsampling.

I'm truly lost as how to effectively generate a qp0 x264 file with the less possible loss in color acurancy. I understand that a 100% lossless RGB->YV24 conversion isn't possible, but all the attempts I made so far seem to give me a file that was converted to 4:2:0 at some point.

Initially I tried inputing the file directly on x264.exe CLI, but I didn't figure out how to make the mov file acceptable by x264 (and even if it is). --input-fmt uses lav filter, right? Should it be configured as default spliter on windows for this parameter to work? My default filter is ffdshow.
Then I heard Avisynt version 2.60 now accepts YV24 output format, but I'm not sure, the regular version do or just some beta build?
Anyway, I just added ConvertToYV24 on avs script and encoded with megui with the following settings:

program --preset slower --qp 0 --b-pyramid none --no-weightb --qpmax 51 --merange 24 --psy-rd 0.00:0 --no-dct-decimate --no-fast-pskip --output-csp i444 --output "output" "input"

I manually added the output-csp i444 line. This generated me a true 444 output file but the colors seems just like 420. I suspect avisynth is indeed outputting 420.

I further tought about exporting a TIF image sequence from Premiere and trying to feed it into x264.exe directly but I have no idea how to do that.

After a lot of reading around Doom's9 and other websites, I also saw people saying that I should specify the input color space for encoding on x264. Original m2ts was Rec.709 limited range, the RGB conversion on Premiere seemed to be fully lossless as the output mov files looks identical to the original 4:2:0 m2ts file. So should I just specify Rec 709 anyway?

I'm not even sure this should be in the avisynth section or x264 section.
Thanks a lot.

[feisty2]

"I understand that a 100% lossless RGB->YV24 conversion isn't possible"
nah nah nah, see "YCgCo"

--output-csp i444 --input-csp i444 --profile high444

[sephirotic]

Originally Posted by feisty2

"I understand that a 100% lossless RGB->YV24 conversion isn't possible"
nah nah nah, see "YCgCo"

--output-csp i444 --input-csp i444 --profile high444

I have researched about YCgCo and it seems that it is only compatible with Madvr, with is not good enough for me.

I discovered why Avisynth wasn't converting properly, even tough the input was RGB, I still should have specified the matrix: ConvertToYV24 (matrix="rec709")
By just adding the parameter I managed to get 100% lossless result.
Thanks for the quick reply, tough.

[feisty2]

By just adding the parameter I managed to get 100% lossless result.

YCbCr (601, 709, 2020..) <---> RGB IS NOT mathematically lossless under finite precision

[vivan]

sephirotic,
Color being different is a different issue.
For RGB->YCbCr conversion x264 uses BT.601 matrix, which is wrong for HD video.
By default avisynth's ConvertTo uses it too, but it can and should be specified ("Rec709" is the correct one if you want tv-range video).

Do you really need 4:4:4 YCbCr? x264 supports RGB encoding which is a bit less effective but way more convenient (and is actually lossless).

Originally Posted by feisty2

nah nah nah, see "YCgCo"

Don't forget to add that it also requires 9 bit precision for chroma to be lossless. Which means 10 bit encoder, Dither tools and 16 bit output from avisynth.

Originally Posted by feisty2

YCbCr (601, 709, 2020..) <---> RGB IS NOT mathematically lossless under finite precision

Of course it's lossless, under higher but still finite precision.

[Jamaika]

Originally Posted by sephirotic

I color graded a m2ts episode from a BD I own on Premiere CS6. Only truly lossless export Option I had was mov with animation codec (RLE).
Now, I wanted to convert to a lossless x264 file (qp0 preset slow) with 4:4:4 subsampling.

At the very beginning you need to be aware that m2ts input material was compressed YUV BT709, that you have lost some information.
Then you added effects and wanted to make the movie lossless at x264.

Originally Posted by sephirotic

I further tought about exporting a TIF image sequence from Premiere and trying to feed it into x264.exe directly but I have no idea how to do that.

I don't know what you created TIFF images MOV. You can download a free plugin VP9 (sRGB, i444 undef).
Chroma sampling 4:4:4 (YUV i444) will be converted to 8bit sRGB monitor. ):
BT2020 is for depth 10bit files and currently isn't recognized by most programs only TV.
sRGB isn't recognized by most programs.

[feisty2]

Originally Posted by vivan

Of course it's lossless, under higher but still finite precision.

there's something called "Infinite decimal", which is a mathematic concept even known to kids from elementary school.
and iirc, floats can do "finite decimals", not fractions
edit: typo

[Motenai Yoda]

Yet but it's lossless like, as float precision is enough to get same values for 8bit RGB -> (16bit RGB) -> >9bit YUV -> (16bit RGB) -> 8bit RGB.
as IIRC you need about 9.6bit YUV to have all 8bit RGB values, ergo 10 or more bit.

[feisty2]

Originally Posted by Motenai Yoda

Yet but it's lossless like, as float precision is enough to get same values for 8bit RGB -> (16bit RGB) -> >9bit YUV -> (16bit RGB) -> 8bit RGB.
as IIRC you need about 9.6bit YUV to have all 8bit RGB values, ergo 10 or more bit.

the YCbCr -- RGB stuff will make infinite decimals, and the infinite decimal gets truncated to some finite decimal and then converted to an even lower precision and BY CHANCE it's "lossless" at such low precision, that's not "mathematically" lossless, that's coincidence
and YCgCo is free from all this at the mathematic level, no infinite decimal will be produced long as the input is not infinite decimal, and that's "mathematically" lossless

[vivan]

Originally Posted by feisty2

there's something called "Infinite decimal", which is a mathematic concept even known to kids from elementary school.
and iirc, floats can do "finite decimals", not fractions
edit: typo

there's also something called "precision". I know, they don't teach it to kids so it's totally ok not to know about it.

Originally Posted by feisty2

the YCbCr -- RGB stuff will make infinite decimals, and the infinite decimal gets truncated to some finite decimal and then converted to an even lower precision and BY CHANCE it's "lossless" at such low precision, that's not "mathematically" lossless, that's coincidence

Yeah, sure, 100% chance is really just a coincidence.

void RGBtoYCbCr (unsigned char* RGB, unsigned short* YCbCr) {
	YCbCr[0] = (unsigned short)(( 16.0 + 0.257 * RGB[0] + 0.504 * RGB[1] + 0.098 * RGB[2]) * 257.0 + 0.5);
	YCbCr[1] = (unsigned short)((128.0 - 0.148 * RGB[0] - 0.291 * RGB[1] + 0.439 * RGB[2]) * 257.0 + 0.5);
	YCbCr[2] = (unsigned short)((128.0 + 0.439 * RGB[0] - 0.368 * RGB[1] - 0.071 * RGB[2]) * 257.0 + 0.5);
}

void YCbCrToRGB (unsigned short* _YCbCr, unsigned char* RGB) {
	double YCbCr[3] = {_YCbCr[0] / 257.0 - 16.0, _YCbCr[1] / 257.0 - 128.0, _YCbCr[2] / 257.0 - 128.0};
	RGB[0] = (unsigned char)(1.164 * YCbCr[0] + 0.000 * YCbCr[1] + 1.596 * YCbCr[2] + 0.5);
	RGB[1] = (unsigned char)(1.164 * YCbCr[0] - 0.392 * YCbCr[1] - 0.813 * YCbCr[2] + 0.5);
	RGB[2] = (unsigned char)(1.164 * YCbCr[0] + 2.017 * YCbCr[1] + 0.000 * YCbCr[2] + 0.5);
}

int _tmain(int argc, _TCHAR* argv[]) {
	for (int R = 0; R < 256; R++)
	for (int G = 0; G < 256; G++)
	for (int B = 0; B < 256; B++) {
		unsigned char RGB[3] = {R, G, B};
		unsigned char newRGB[3];
		unsigned short YCbCr[3];
		RGBtoYCbCr (RGB, YCbCr);
		YCbCrToRGB (YCbCr, newRGB);
		if (newRGB[0] != RGB[0] || newRGB[1] != RGB[1] || newRGB[2] != RGB[2]) {
			printf ("%i %i %i %i %i %i\n", newRGB[0], RGB[0], newRGB[1], RGB[1], newRGB[2], RGB[2]);
		}
	}

	return 0;
}

Run it and tell me what you see (or, more precisely, don't).

[feisty2]

yeah, yeah
something is only lossless when you manually clip the extra imprecise "lsb" part (aka. error part) off, right, 0.123456 = 0.123000 when you zero the "456" lsb out and that is just mathematically lossless??
back to the point, it's lossless cuz you were rounding a short to a char, which almost never ever happens in the wild, cuz you know what? people will dither a short to a char, and boom, the lossless magic is gone with dithering, so you gonna tell everyone, just rounding 16bits YUV to 8bits RGB, don't dither? and I highly doubt if anyone will ever follow that advice
and YCgCo is ALWAYS lossless, dithering or truncating or rounding, cuz you know what, it's mathematically lossless (it has no lsb (no error), so nothing to dither)

[feisty2]

and don't even bother to write c programs to test all these
dither package + mt_makediff + mt_lut (diff, "x 128 - 20 * 128 +") are everything you need

[sephirotic]

Originally Posted by vivan

sephirotic,
By default avisynth's ConvertTo uses it too, but it can and should be specified ("Rec709" is the correct one if you want tv-range video).

yest, that is what I've done after further digging.

Originally Posted by vivan

Do you really need 4:4:4 YCbCr? x264 supports RGB encoding which is a bit less effective but way more convenient (and is actually lossless).

I've done a second encode with x264 rgb for keeping the master, I only needed YCBCR for the final 10 bit encode.

Originally Posted by Jamaika

At the very beginning you need to be aware that m2ts input material was compressed YUV BT709, that you have lost some information.
Then you added effects and wanted to make the movie lossless at x264.

Well, Premiere interpreted the color matrix correctly so lost information was minimal.

Originally Posted by Jamaika

I don't know what you created TIFF images MOV. You can download a free plugin VP9 (sRGB, i444 undef).
Chroma sampling 4:4:4 (YUV i444) will be converted to 8bit sRGB monitor. ):
BT2020 is for depth 10bit files and currently isn't recognized by most programs only TV.
sRGB isn't recognized by most programs.

sRGB would be interesting for the master, but then again, premiere doesn't export it Afaik so taht would only add another conversion and more rounding losses in the process.

Thanks for all the help, I was able to attain virtually imperceivable loss in color information just by using ConverTToYV24 (matrix="rec709")

Here a sample to anyone curious.
http://diff.pics/ssFVuuBc4Agi/1
Lossless in this case is qp0 veryslow no bframes x264 yv24 encode, but I'm in the middle of doing a RGB x264 encode right now as a V2.

[vivan]

Originally Posted by feisty2

yeah, yeah
something is only lossless when you manually clip the extra imprecise "lsb" part (aka. error part) off, right, 0.123456 = 0.123000 when you zero the "456" lsb out and that is just mathematically lossless??

First, you should learn the definition of lossless. Lossless transform is a transform that can be reversed so that you will get original data. And no, "can" is not about probability - it's about conditions, which is higher precision.

Second, you should really learn something about precision. 0.123 and 0.123000 have different meaning - second implies much greater precision. What original "0.123" actually means is that value lies in the +- 0.0005 range. And this is how you're supposed to compare numbers.

Originally Posted by sephirotic

I've done a second encode with x264 rgb for keeping the master, I only needed YCBCR for the final 10 bit encode.

But can't you encode from your x264 rgb master?

[feisty2]

I don't wanna waste my time on this pointless argument no more

colorbars (pixel_type="rgb32")

ycgco = dither_convert_rgb_to_yuv (output="yv24",lsb=true,matrix="ycgco",tv_range=false)
ycbcr = dither_convert_rgb_to_yuv (output="yv24",lsb=true,matrix="601",tv_range=false)

rgb48ycgco = dither_convert_yuv_to_rgb (ycgco, output="rgb48y",matrix="ycgco",tv_range=false,lsb_in=true)
rgb48ycbcr = dither_convert_yuv_to_rgb (ycbcr, output="rgb48y",matrix="601",tv_range=false,lsb_in=true)

rgb48cgco = mergergb (rgb48ycgco.selectevery (3,0), rgb48ycgco.selectevery (3,1), rgb48ycgco.selectevery (3,2)).subtitle ("ycgco")
rgb48cbcr = mergergb (rgb48ycbcr.selectevery (3,0), rgb48ycbcr.selectevery (3,1), rgb48ycbcr.selectevery (3,2)).subtitle ("ycbcr")

StackHorizontal(rgb48cgco,rgb48cbcr)

just stare at the LSB part of "ycgco" and "ycbcr" real carefully and tell me what's the difference, which one is "lossless"

[feisty2]

0.123000 (ycgco) = 0.123 is lossless, cuz 0.123000=0.123 mathematically
0.123456 (ycbcr) ≈ 0.123 is NOT lossless, ≈ is still some kind of != obviously

[colours]

It's vaguely ironic to see feisty2 post about "mathematical precision" and using Dither tools to illustrate that, when Dither tools has off-by-one errors scattered throughout the full-range colour conversion functions.

Also, good job missing vivan's point. How surprising it is that you find the argument "pointless" when you just ignore what everyone else says.

[feisty2]

Originally Posted by colours

It's vaguely ironic to see feisty2 post about "mathematical precision" and using Dither tools to illustrate that, when Dither tools has off-by-one errors scattered throughout the full-range colour conversion functions.

there's no such thing as infinite precision for computers, I don't see any other better option to do it, anyways, I'll explain what I was saying with the following

Originally Posted by colours

Also, good job missing vivan's point. How surprising it is that you find the argument "pointless" when you just ignore what everyone else says.

I get that stuff vivan or you stated

suppose:
1. there was infinite precision for computers
2. all rgb-yuv444-rgb stuff was done correctly under infinite precision
3. the input value was 0.123

the output value of both "ycgco" and "ycbcr" should be 0.123000000... or 0.1229999999... both are mathematically equal to 0.123, the input value
perfect, end of the story

now the reality is, there's no infinite precision, only higher but still finite precision

say, the higher precision here is "6 significant figures"
the output of ycgco would be like, 0.123000, still mathematically equals to 0.123, cuz there're only things like /2 and /4 inside ycgco stuff, and they won't produce infinite decimals long as the input is not infinite precision

the ycbcr output might be like 0.123456, which != 0.123, cuz obviously ycbcr requires infinite precision for intermediate values to stay the zero error status, some intermediate value gets truncated to 6 significant figures and that leads to the error presented in the output value

the disagreement starts right from here

you and vivan think 0.123456 will become 0.123 if rounded down to 3-significant-figure precision and will =0.123 again so it's lossless

and I think it's cheating, the mathematically lossless method should have an output like ycgco, lossless without rounding

[Rumbah]

Well, I'd define lossless like this in this case:

I have an image in space A.
I have a transformation f from space A to space B.
I have a transformation g from space B to space A.

If for every pixel x in A: x = g(f(x)) is true then the transformation is lossless.

It does not matter if there is rounding or definite presicion involved as long as every starting pixel has the same value as a transformed and back tranformed pixel.

With this you see that the Dither color space tranformations are not lossless (because of dithering they can be off by +-1).

But indeed you can do a lossless RGB8 to YCbCr10 to RGB8 transformation as vivan has shown.

[feisty2]

Originally Posted by Rumbah

Well, I'd define lossless like this in this case:

I have an image in space A.
I have a transformation f from space A to space B.
I have a transformation g from space B to space A.

If for every pixel x in A: x = g(f(x)) is true then the transformation is lossless.

It does not matter if there is rounding or definite presicion involved as long as every starting pixel has the same value as a transformed and back tranformed pixel.

With this you see that the Dither color space tranformations are not lossless (because of dithering they can be off by +-1).

But indeed you can do a lossless RGB8 to YCbCr10 to RGB8 transformation as vivan has shown.

and that "lossless" for ycbcr is just useless like flopped celebrities going to reality shows (the voice, ...) trying to regain attention

will you ever "round" or "truncate" a 16bits YUV vid to <= 14bits RGB? guess not

and the losslessness of ycgco will survive thru rounding or truncating or error diffusing, long as no random noise is applied, ycgco takes its crown by being mathematically lossless at finite precision