Bug of x264 10bit or of ffdshow?

[aufkrawall]

Hello,
I hope the x264 devs are reading. :)
I can't say if the following is a bug of ffdshow or x264:
A x264 I444 10bit video doesn't look right, red is rather orange.
Here I made a bug report for it for ffdshow, there you can also download a (totally legal) sample:
https://sourceforge.net/projects/ffdshow-tryout/forums/forum/597593/topic/4934748

The problem is that ffdshow either chooses the wrong YCbCr specification (709 instead of 601) or x264 10bit does something wrong. If encoded with --colormatrix fcc ffdshow chooses the right YCbCr specification.

Now, is it a problem of ffdshow or x264 10bit?

[sneaker_ger]

Doesn't really sound like a bug to me. If there is no info about the matrix in the file, players have to guess. Ffdshow chooses BT.709 for files with your characteristics, and that is neither wrong nor right - it's really just a guess. If you need the player to know the matrix used, you have to include that info in the file (with --colormatrix, like you already found out).

On a side note, I'm wondering if the high bitdepth BT.601 conversions are implemented correctly in the common programs, because there were problems with the corresponding BT.709 ones, pointed out by cyberbeing (IIRC) for x264 and ffdshow. Maybe it would be safer to convert the 8 bit sources to BT.709 with Colormatrix() before doing any high bitdepth work for the time being?

[aufkrawall]

Doesn't really sound like a bug to me. If there is no info about the matrix in the file, players have to guess.

But why does it work correctly (without specifying colormatrix) with 8bit I444 output?
It's the same source, of course.


On a side note, I'm wondering if the high bitdepth BT.601 conversions are implemented correctly in the common programs, because there were problems with the corresponding BT.709 ones, pointed out by cyberbeing (IIRC) for x264 and ffdshow.

They are just playing fine, with bare eye I can't see a quality difference to 8bit RGB.


Maybe it would be safer to convert the 8 bit sources to BT.709 with Colormatrix() before doing any high bitdepth work for the time being?
Wouldn't it pretty useless to use 10bit encoder if the source had only 8?
Source is Fraps RGB video and with 10bit x264 compression is muuch better.

[sneaker_ger]

But why does it work correctly (without specifying colormatrix) with 8bit I444 output?
It's the same source, of course.

A guess is a guess is a guess. It can happen to be correct or incorrect. The ffdshow devs probably think that 10 bit content is mostly BT.709 and therefore use that matrix if there is no info.


Wouldn't it pretty useless to use 10bit encoder if the source had only 8?
Source is Fraps RGB video and with 10bit x264 compression is muuch better.

You are contradicting yourself here. RGB is 8 bit. And no, it is actually useful to encode 8 bit sources with 10 bit, like you have already found out. This has been discussed several times on this forum already.

[nevcairiel]

I don't know about ffdshow specifically, but usually the rules are pretty simple if the header doesn't specify any information.
Its simply determined by resolution, i doubt 10-bit has any special rules (if it does, its stupid).

Anything > 576 in height or > 1024 in width is usually considered "HD", and gets the 709 matrix. PC resolutions don't really match those specs, as even 800x600 is already "HD" in that definition.
I would suggest to simply include the color matrix flag in the h264 encoding process, this is the only sure way you know it'll be right.

PS:
I didn't download the file, so i don't know its resolution.
If the only difference between your files truely is 8bit vs 10bit, it is indeed weird. It may however also be the case that ffdshows "good" RGB converter does not support 10-bit 4:4:4 input, and therefor falls back to a "bad" one.
For comparison sake, you could try my decoder (see link in signature), which fully supports 10-bit 4:4:4 -> RGB conversion while taking the matrix into account.
If the problem persists then, its quite possible that the encoding process did something odd

[mandarinka]

By default, ffdshow outputs yuv IIRC, that would mean that the colormatrix guessing is left to the video renderer, which also decides according to resolution. The renderer probably won't receive the header's information, no?

[naoan]

Nope, madvr get the flag right for my x264 768p game encode.

[nevcairiel]

The renderer probably won't receive the header's information, no?

There is a way to communicate it, however ffdshow doesnt support sending it, nor do most renderers support receiving it.

[aufkrawall]

I don't know about ffdshow specifically, but usually the rules are pretty simple if the header doesn't specify any information.
Its simply determined by resolution, i doubt 10-bit has any special rules (if it does, its stupid).

Anything > 576 in height or > 1024 in width is usually considered "HD", and gets the 709 matrix. PC resolutions don't really match those specs, as even 800x600 is already "HD" in that definition.
I would suggest to simply include the color matrix flag in the h264 encoding process, this is the only sure way you know it'll be right.

PS:
I didn't download the file, so i don't know its resolution.

It's PC resolution, 1280*1024.


For comparison sake, you could try my decoder (see link in signature), which fully supports 10-bit 4:4:4 -> RGB conversion while taking the matrix into account.
If the problem persists then, its quite possible that the encoding process did something odd
I already use your LAV Decoder for x264 RGB, there it works great. :)
But for I444 it doesn't look right either, neither with 8bit video. :scared:

Example screens:
RGB x264 (LAV, bit exact to Fraps RGB output):
http://www.ld-host.de/uploads/thumbnails/8bfb211381666aae6131469bd5a031f0.png (http://www.ld-host.de/show/8bfb211381666aae6131469bd5a031f0.png)

I444 8bit:
http://www.ld-host.de/uploads/thumbnails/f9b7eaed107cb0b6260584a1547e84d6.png (http://www.ld-host.de/show/f9b7eaed107cb0b6260584a1547e84d6.png)

I444 10bit with BT 709 in ffdshow:
http://www.ld-host.de/uploads/thumbnails/2ff7bad451b64759310bf1ae69ab538e.png (http://www.ld-host.de/show/2ff7bad451b64759310bf1ae69ab538e.png)

I444 10bit with LAV RGB output (doesn't look better with AYUV output):
http://www.ld-host.de/uploads/thumbnails/d7f84b12eec3ef689e68ac1b1d7fa2df.png (http://www.ld-host.de/show/d7f84b12eec3ef689e68ac1b1d7fa2df.png)

Unfortunately, LAV also doesn't look right for the I444 10bit video with FCC colormatrix, maybe a bug of LAV (tested with both AYUV and RGB output)?
Here's a sample (totally legal):
http://www.multiupload.com/N2H6N6YE9W

[STaRGaZeR]

I suffered this a while ago. Feeding Fraps RGB to x264 and encoding to 4:4:4 resulted in wrong colors, it happened with 8 bit too. Tried everything about color matrixes, levels, etc. with no success. Pure green wasn't pure green, pure red wasn't pure red, and so on. Just like in those screenshots.

[aufkrawall]

I suffered this a while ago. Feeding Fraps RGB to x264 and encoding to 4:4:4 resulted in wrong colors, it happened with 8 bit too. Tried everything about color matrixes, levels, etc. with no success. Pure green wasn't pure green, pure red wasn't pure red, and so on. Just like in those screenshots.
Did you find any solution/workaround?
With the following config everything works for me:
-ffdshow for I444
-also ffdshow for 10bit I444 (use --colormatrix fcc with x264 or change YCbCr spec.)
-set up ffdshow to output RGB32 and to use full RGB range
-LAV Decoder for x264 RGB
Player is MPC HC x64 with bundled demuxers.

I use Simple x264 Launcher as encoder with parameters, it works by 100% with RGB output, I444 and 10bit I444.
Edit: --range pc is important.

[STaRGaZeR]

Nope, didn't found a solution. At that point it was just an experiment. My intention was to open a thread like this when I finally needed to do an actual encode. In my case flags were properly set, but played with everything and still got wrong colors unfortunately.

[nevcairiel]

What Matrix is fcc exactly? I only support 601, 709 and smpte 240m. without flags your file will always decode as 709, there really isn't a better way at guessing.

I can have a look at adding support for that matrix so that at least properly flagged files decode ok.

Edit:
nvm, found the matrix in the h264 spec, and it seems to be very close to BT.601.

So fixed it, should at least with matrix look good now.
Regarding the original question, there really isn't much that can be done. I guess if we were to scan for PC resolutions (800x600, 1024x768, 1600x1200, and maybe some more), and would default them to BT.601 when no flag is found, other people would still complain that their files now changed color.
Add the flags, and everything is good! :)

I can also add a matrix option to LAV, although i would still prefer if people flag their stuff. :p

[aufkrawall]

What Matrix is fcc exactly? I only support 601, 709 and smpte 240m. without flags your file will always decode as 709, there really isn't a better way at guessing.

I can have a look at adding support for that matrix so that at least properly flagged files decode ok.

Edit:
nvm, found the matrix in the h264 spec, and it seems to be very close to BT.601.

Do you know the command to make x264 set the right flag for BT.601?
There was only a little difference when comparing pixel difference pictures of manual BT.601 and FCC (I couldn't see them without making them visible) but it'd be nice if there was a more ideal solution. :)
Otherwise I'd have to try out every color matrix in hope to find something that matches.


So fixed it, should at least with matrix look good now.

Cool, I'll try out the next release.
:thanks:

[nevcairiel]

The difference between FCC and 601 is very marginal.

The matrix coefficients for FCC are 0.300 and 0.110, BT.601 is 0.299 and 0.114.
So you might get very minor differences, but i doubt anyone can see them without looking at individual pixels.

"--colormatrix bt470bg" should do BT.601 (known under many different names)

[aufkrawall]

"--colormatrix bt470bg" should do BT.601 (known under many different names)
Thanks again, I'll try it out. :)
Might be a good idea to always add this command when encoding Fraps RGB video.
Edit: Except when encoding to x264 RGB, of course.

[aufkrawall]

--colormatrix bt470bg works great, with LAV Decoder and ffdshow too.
But just like with ffdshow I have to set output CS to RGB in the LAV Decoder settings or colors look wrong with I444.
Also RGB output level must be set to PC, otherwise I420 DXVA videos are too bright with LAV.
But techninally everything works.

[STaRGaZeR]

Just did some tests. Stuff has changed since I tried it. I see it as follows. If you do this:

x264 --qp 0 frapsRGBInput.avi -o outI444.mkv --output-csp i444


x264 will convert the RGB frames (PC levels) into YUV 4:4:4 using the BT.601 matrix (or FCC, too little difference to tell, but I highly doubt it) AND it'll convert PC to TV levels. So, it you use that command line, decoders will guess the stream is TV levels and the matrix depending on resolution, which can lead to issues. To avoid them, flag the stream properly:

x264 --qp 0 frapsRGBInput.avi -o outI444.mkv --output-csp i444 --input-range tv --range tv --colorprim bt470bg --transfer bt470bg --colormatrix bt470bg


This works for me with both 8 bit and 10 bit x264. However it'd be nice to be able to tell x264 how to convert stuff, both levels and matrices. This isn't possible, but Avisynth can do it just fine:

AVISource("Source.avi") ### Your RGB Fraps source
ConvertToYV24(matrix="rec709") ### "rec709" or "rec601" if you want TV levels, "PC.709" or "PC.601" if you want PC levels

Use this script and enjoy full control of what you send to x264. Just don't forget to change the flags accordingly. I recommend "rec709" or "rec601" depending on resolution (use "rec709" unless you record on a prehistoric monitor), this will save you trouble with decoders that ignore flags.


@nev, it seems LAVVideo outputs NV12 for the original Fraps file (LAV Splitter reports it as bgr24) instead of RGB32 or RGB24 with default settings. This causes severe damage to IQ of course.

[Snowknight26]

@nev, it seems LAVVideo outputs NV12 for the original Fraps file (LAV Splitter reports it as bgr24) instead of RGB32 or RGB24 with default settings. This causes severe damage to IQ of course.

It could be because the Fraps video you're testing was YV12 instead of BGR24.

[nevcairiel]

@nev, it seems LAVVideo outputs NV12 for the original Fraps file (LAV Splitter reports it as bgr24) instead of RGB32 or RGB24 with default settings. This causes severe damage to IQ of course.

Only if you use EVR, because EVR is incapable of changing its pixel format after it connects, and i don't think Fraps lets me figure out which pixel format was used before decoding starts - and sadly a decoder doesn't get any data before connecting the renderer. (And 8-bit 4:2:0 is the default mode, because it covers like 90% of all content)

I was meaning to add an interface so that LAV Splitter can tell LAV Video which pixel format it detected (LAV Splitter decodes a few frames to figure it out), so this issue can be hacked around.

[aufkrawall]

So, it you use that command line, decoders will guess the stream is TV levels and the matrix depending on resolution, which can lead to issues. To avoid them, flag the stream properly:
"x264 --qp 0 frapsRGBInput.avi -o outI444.mkv --output-csp i444 --input-range tv --range tv --colorprim bt470bg --transfer bt470bg --colormatrix bt470bg"

Why --range TV? Fraps RGB/BGR input has PC range, that's why I used --range pc.
I simply use this:
--range pc --output-csp i444 --colormatrix bt470bg
The only thing to consider is setting RGB output for decoder when playing, but I can live with that.

[STaRGaZeR]

It could be because the Fraps video you're testing was YV12 instead of BGR24.

Nope, it is bgr24 actually. LAV Splitter got it right.

@nev, yikes!

Why --range TV? Fraps RGB/BGR input has PC range, that's why I used --range pc.

Huh, copypaste fail. Copied a commandline of the Avisynth tests :p

I simply use this:

The only thing to consider is setting RGB output for decoder when playing, but I can live with that.

If you do that and you don't output RGB from the decoder, the conversion YUV->RGB has to be done on the renderer. Everything except LAV Video (AFAIK) doesn't send any flag information downstream, and all renderers except madVR (AFAIK) just ignores them even if you send them, they have to guess the matrix and levels. As said previously, they always assume TV levels and matrix depending on resolution, in your case 1280x1024 is assumed to be BT.709. That's why you have wrong colors if you don't output RGB, as the original conversion to YUV was done using the 601 matrix. To avoid this it's best to convert it to TV levels and BT.709. That way even a retarded decoder/renderer combo will display it properly.

That said, AYUV support in EVR is pretty lame, so unless you use madVR it's best to untick AYUV in LAV Video so it outputs RGB anyway, saving you of all hassle.

[aufkrawall]

Everything except LAV Video (AFAIK) doesn't send any flag information downstream, and all renderers except madVR (AFAIK) just ignores them even if you send them, they have to guess the matrix and levels.

That should be done by ffdshow, that's why manually changing YCbCr spec in ffdshow "fixed" the output colors. The same does --colormatrix bt470bg when leaving the option default in ffdshow.
Or is there still anything wrong when setting the flag with --colormatrix bt470bg?


To avoid this it's best to convert it to TV levels and BT.709. That way even a retarded decoder/renderer combo will display it properly.

Is the conversion from PC to TV range lossless?

[STaRGaZeR]

That should be done by ffdshow, that's why manually changing YCbCr spec in ffdshow "fixed" the output colors. The same does --colormatrix bt470bg when leaving the option default in ffdshow.
Or is there still anything wrong when setting the flag with --colormatrix bt470bg?

ffdshow doesn't do it. All options in the RGB conversion tab control ffdshow's internal YUV->RGB converter, and don't have any effect if the output isn't RGB as the name implies. The colormatrix flag tells the ffdshow RGB converter what matrix to use, but if ffdshow doesn't convert to RGB that information is not passed downstream. That's why you get wrong colors when outputting YUV. The renderer guesses, and guesses wrong. That's why I suggest to encode it in a way that the guesses will be right: TV levels, and BT.709 for HD resolutions or BT.601 for SD resolutions.

Is the conversion from PC to TV range lossless?

Nope, you'll lose dinamic range, but you won't see it (I don't see it anyway). Try for yourself. The RGB->YUV->RGB process you're doing now isn't lossless either.

[aufkrawall]

Now I understand and I see your point.
I rather avoid any "unnecessary" conversions to maintain maximum image quality and to me it's not really a disadvantage to force the decoder to output RGB (very most CPUs should deal with it), other videos are still decoded correct with it (at least the ones I tried).

But just converting to RGB is always lossless?

[STaRGaZeR]

But just converting to RGB is always lossless?

What do you mean? If you want to avoid all conversions just encode with --output-csp rgb, and if you want lossless, with --qp 0. You won't need to change the decoder's output to RGB either, it'll output RGB natively.

[aufkrawall]

What do you mean? If you want to avoid all conversions just encode with --output-csp rgb, and if you want lossless, with --qp 0. You won't need to change the decoder's output to RGB either, it'll output RGB natively.
x264 RGB is to ressources consuming, unfortunately.
I mean I444 output conversion to RGB in ffdshow, is it lossless?
Since RGB24 can portray every color I'd suppose yes.

[J_Darnley]

I mean I444 output conversion to RGB in ffdshow, is it lossless?
Since RGB24 can portray every color I'd suppose yes.

That's an odd question, why would you let ffdshow output RGB only to convert back to yuv444?

In general, RGB<->YUV conversions are not lossless. Not every value in one maps to a unique value in the other meaning you will not be able to get the original data again. But, just to be clear, without repeated conversions and getting the range and matrix correct, you will never see the loss in these conversions.

[poisondeathray]

whoops <ignore>

[STaRGaZeR]

x264 RGB is to ressources consuming, unfortunately.
I mean I444 output conversion to RGB in ffdshow, is it lossless?
Since RGB24 can portray every color I'd suppose yes.

Nope. RGB to 4:4:4 YUV inside x264 is lossy, 4:4:4 YUV to RGB in ffdshow is lossy again. Double lossy conversion. But I bet you won't discern the result from the original, just like with my TV levels BT.709 proposal :p Try it and decide!

[Asmodian]

Nope. RGB to 4:4:4 YUV inside x264 is lossy, 4:4:4 YUV to RGB in ffdshow is lossy again. Double lossy conversion. But I bet you won't discern the result from the original, just like with my TV levels BT.709 proposal :p Try it and decide!

I think this is not true. Though I do agree that converting to TV.709 is a good idea as it will look correct anywhere.

"YUV is not inherently any more compact than RGB. Unless the chroma is downsampled, a YUV pixel is the same size as an RGB pixel. Also, the conversion from RGB to YUV is not lossy. If there is no downsampling, a YUV pixel can be converted back to RGB with no loss of information."

[Dark Shikari]

I think this is not true.MSDN is wrong. YUV<->RGB is not a one-to-one mapping. If you want a YUV-like colorspace that can losslessly represent RGB, YCgCo is what you're looking for, not YCbCr.

[aufkrawall]

That's an odd question, why would you let ffdshow output RGB only to convert back to yuv444?

Because with other output colorspaces the colors of the encoded Fraps RGB to x264 I444 video don't look right (since I want to keep PC range).


In general, RGB<->YUV conversions are not lossless.

I understood that RGB to I444 isn't lossless.
But what about the RGB output (by ffdshow) of the I444 video?
Will there be, just for this single step, a quality loss?
In my simple theorie RGB24/32 are able to render every color of I444 since they can render any important color anyway.
Now, is that right?

[Stephen R. Savage]

The RGB colorspace and the YUV colorspace are not equivalent for two reasons:

1. Due to rounding error, not every RGB value maps to a unique YUV value.
2. Not all YUV values represent valid RGB colors (the mapping is not one-to-one). This is because YUV/RGB are defined on the (normalized) interval [0, 1] and not on R^3. See: http://forum.doom9.org/showthread.php?t=154731

[aufkrawall]

In my simple theorie RGB24/32 are able to render every color of I444 since they can render any important color anyway.
Now, is that right?
Ah, sorry:

4:4:4 YUV to RGB in ffdshow is lossy again.

But there's still almost no difference between the pictures, difference pic with little tolerance:
http://www.ld-host.de/uploads/thumbnails/b9d840acc8771e6e3832b0c6f64735ab.png (http://www.ld-host.de/show/b9d840acc8771e6e3832b0c6f64735ab.png)
The question is what is more lossy? Dropping RGB PC range or doing a second conversion to RGB while decoding?

It's unsatisfying that there seems to be no possibility to play the video with AYUV output and full range. :mad:

[STaRGaZeR]

Ah, sorry:

But there's still almost no difference between the pictures, difference pic with little tolerance:
http://www.ld-host.de/uploads/thumbnails/b9d840acc8771e6e3832b0c6f64735ab.png (http://www.ld-host.de/show/b9d840acc8771e6e3832b0c6f64735ab.png)
The question is what is more lossy? Dropping RGB PC range or doing a second conversion to RGB while decoding?

It's unsatisfying that there seems to be no possibility to play the video with AYUV output and full range. :mad:

Told you: not lossless, but visually indistinguishable ;)

That AYUV has to be converted somewhere to RGB again before displaying it in your monitor, you can't avoid that. It's either x264 RGB or double lossy conversion. To the question as to what mis more lossy, I don't know. You'd need a comparison program.

If I can't do lossless, I don't care about the degree of lossyness, as long as I can't see it. But that's just me.

[Asmodian]

The RGB colorspace and the YUV colorspace are not equivalent for two reasons:

1. Due to rounding error, not every RGB value maps to a unique YUV value.
2. Not all YUV values represent valid RGB colors (the mapping is not one-to-one). This is because YUV/RGB are defined on the (normalized) interval [0, 1] and not on R^3. See: http://forum.doom9.org/showthread.php?t=154731

As a clarification and so I can stop spreading misinformation is there a loss in precision when doing I444->RGB->I444?

If every color present in the source is valid RGB and I444 are there some YUV colors that map to the same RGB, either due to rounding errors or just because that is the way the space works?

[jp2k]

Hi everyone,
I have some gray level images with 16 bits depth and I would like to save it as a video (h.264 ).Would it be possible to help me how I can do it.
Thank you in advance.

[Asmodian]

Thanks. :)

[aufkrawall]

Cool, with madVR as renderer and LAV as decoder MPC HC shows the original AYUV CSP of the I444 video, at least if it has bt470bg flag. Colors look like they should.
So, lossy conversion to RGB while decoding when playing is avoided. :)

Riddle fully solved.

[sneaker_ger]

So, lossy conversion to RGB while decoding when playing is avoided. :)

It is not. MadVR always converts to RGB. But as STaRGaZeR said: "not lossless, but visually indistinguishable".

[aufkrawall]

It is not. MadVR always converts to RGB. But as STaRGaZeR said: "not lossless, but visually indistinguishable".
So, it does the same like when forcing LAV to output only RGB?

[Asmodian]

It is not. MadVR always converts to RGB. But as STaRGaZeR said: "not lossless, but visually indistinguishable".

Well any renderer will convert to RGB in the end; the monitor needs to know how much red, green, and blue to display after all.

[aufkrawall]

Well any renderer will convert to RGB in the end; the monitor needs to know how much red, green, and blue to display after all.
So, it doesn't matter after all?

[Asmodian]

So, it doesn't matter after all?

I think you just want to make sure it is done right, MadVR is quite good at it and it knows more about your display device if setup so people suggest using it. LAV Video works quite well too.

[nevcairiel]

So, it doesn't matter after all?

madVR is one of the few renderers that can actually deal with flagged streams and doesn't only decide based on resolution, so it is indeed better.
As the others said however, for display you need RGB, so at some point you will need to convert it. :)

[Zerofool]

Here's my experience with FRAPS RGB video and 4:4:4 10-bit encoding, and what I do to get the same colors/levels as the input:

First, I make sure that the "Force lossless RGB capture (may be slow)" checkbox in FRAPS' window is on before I start capturing, because it disappeared one time after updating FRAPS :).

Then, I use Avisynth script for the input, such as this:
FFVideoSource("My_FRAPS_RGB_vid.avi")
ConvertToYV24(matrix="PC.709")
It's important to use the "PC.709" matrix, to keep the full range, otherwise (e.g. "Rec709") it will be converted to 16-235.
You'll need Avisynth 2.6 for YV24 support, I use the MT version by SEt.

Then, during encoding (naturally, with the 10bit version of x264) I specify these parameters:
--range pc --input-range pc --colorprim bt709 --transfer bt709 --colormatrix bt709 --output-csp i444
Without specifying "--input-range pc" and only "--range pc", it appears that some kind of internal conversion takes place, as if the input was tv range. In that aspect, it was easier with just "--fullrange" back in the day.

For the playback, I personally use MPC-HC with madVR as both renderer AND decoder, but it's also OK when I use different decoder (LAVF, ffdshow).

aufkrawall, give it a try and compare it with your avisynth-less approach.



If you want a YUV-like colorspace that can losslessly represent RGB, YCgCo is what you're looking for, not YCbCr.
Now this is interesting, can you give us more details on how this can be done with x264? Because the available avisynth plug-in (http://forum.doom9.org/showthread.php?t=161736) doesn't support the "lossless" part, it would need 9bit chroma for this, according to the posts in the thread.
Are there any plans for future integration of "full-precision" RGB-to-YCgCo internal conversion inside (10bit) x264?

[Dark Shikari]

Since x264 doesn't support chroma-having-more-bits-than-luma, it can't really support YCgCo without major changes.

[aufkrawall]

aufkrawall, give it a try and compare it with your avisynth-less approach.

I couldn't get it work right. :(
I used the following script with with AviSynth 2.6 and Simple x264 Launcher with your parameters:
LoadPlugin("C:\Users\Julian\Desktop\megui\tools\ffms\ffms2.dll")
FFVideoSource("C:\Users\Julian\Desktop\rgb.avi", threads=1).AssumeFPS(30.000)
ConvertToYV24(matrix="PC.709")
The result looks like this:
http://www.ld-host.de/uploads/thumbnails/0ebf212311c84c46af0a5851afac990d.png (http://www.ld-host.de/show/0ebf212311c84c46af0a5851afac990d.png)
It doesn't look like I444, there must have been a conversion to a worse CSP at some point.

[Zerofool]

I couldn't get it work right. :(
I used the following script with with AviSynth 2.6 and Simple x264 Launcher with your parameters:
LoadPlugin("C:\Users\Julian\Desktop\megui\tools\ffms\ffms2.dll")
FFVideoSource("C:\Users\Julian\Desktop\rgb.avi", threads=1).AssumeFPS(30.000)
ConvertToYV24(matrix="PC.709")
It doesn't look like I444, there must have been a conversion to a worse CSP at some point.

That's strange, I haven't encountered such problems, it's all OK, I've used test patterns to be sure. I just did a short test now, to see if the problem lies in the latest revision of x264, but it's still OK here.
Are you sure you copied the entire line of parameters, specifically the last one? Check your stream with mediainfo, to make sure it's 4:4:4. From there... you can troubleshoot your script - first, leave only the source (and the loadplugin) and add a line "info()". Then open the script in let's say VirtualDub. The colorspace of the FRAPS video should be RGB32. Then add the "info()" line to your original script. It should be YV24 now. If all that is OK... then the problem should lie in the Simple x264 Launcher, especially if you use the x64 build of x264. Most probably the piping tool this app uses doesn't correctly handle YV24 video yet. I don't use it, so I don't know which one exactly it incorporates, but you can try to manually replace it with this one (http://forum.doom9.org/showthread.php?t=162656). Still, your best bet is to use CLI with the 32-bit x264 binary for this test, that way you'll know for sure what exactly happens.
Then again, it also may be the way you play the file... :)


Since x264 doesn't support chroma-having-more-bits-than-luma, it can't really support YCgCo without major changes.

I see... Well, I guess my understanding about it is very wrong, so please excuse me, but here's how I imagine it. Let's say we have a 8bpp RGB clip, so for its conversion we'll need at least 8bit luma and at least 9bit chroma. So what if we do the conversion and then we upscale the result to 10bit? That way both luma and chroma will have the same depth. Wouldn't the color values remain the same and therefore reproduce the initial 8bpp RGB in the end? Wouldn't it work that way?
Talking about upconverting, now I remember there was a discussion (http://forum.doom9.org/showthread.php?t=161915&highlight=10bit&page=4) about 8bit to 10bit, and how it was done wrong in the current state, and I couldn't find any final conclusion, just a 3rd party patch for limited range only, so it might not be exactly "lossless" this way.

And another thing - I looked more closely at this Amendment 1 PDF (http://sipl.technion.ac.il/new/Archive/Special_Events/H264-2004/H264-FR-Extensions.pdf) and on page 16 there's this thing they say about YCgCo:
– In 4:4:4 color format there’s also a trick to eliminate
conversion rounding errors without the two additional
bits (residual color transform)
Do you know what this guy is talking about, does it make sense?

P.S. Now I looked it up, obviously this Residual Color Transform is another feature from FRExt, does x264 support it? And if not, is it on the to-do list? :)
Cheers