Problem with YUV->RGB hardware & MKV/H264 delivery

[Lazarus9]

I'm releasing a video in MKV/H264 format and have a problem - unless VLC Player has 'Use Hardware YUV->RGB conversions' unselected, the colour balance of the video is very off.


The problem affects specifically nvidia cards and VLC player - unfortunately most of my viewers are precisely in that demographic; most have nvidia and most use VLC.

I have been told the problem is due to a use of YUV, which is a television standard. I'd be happy to try and remove YUV from my work but I'm not sure where to go about doing this. My development process is as follows:

1) Video is made in Adobe Premiere and AE
2) A large, uncompressed raw file is extracted
3) I use Staxrip to make this file into an MKV.
4) Unfortunately the resulting MKV is vulnerable to the YUV issue described above.

I have tried using Handbrake to do the encoding instead but still run into the same issue with the VLC/Nvidia combination.

Since the option is ticked by default and I don't have a delivery method to ensure that people untick it before viewing, I need to make, somehow, an MKV/H264 file that isn't vulnerable to this issue.

From some years of lurking on Doom9, I figure you fine people are the best placed folk on the internet to tell me how :)

[Guest]

Post a link to a sample that shows the problem.

And what is the content you are releasing?

[Lazarus9]

Post a link to a sample that shows the problem.

The problem wouldn't manifest in a screenshot, since it's a problem with the playback on VLC with Hardware YUV>RGB selected on Nvidia cards (which it is by default). The file played back on an ATi machine or on Media Player, for example, would look normal.

The problem itself seems to be reasonably well known, e.g. threads like http://forum.doom9.org/showthread.php?t=132716

Essentially by default with that option ticked, VLC plays back YUV MKV content with "RGB 16 to 235" instead of 0, 255.

I understand that unticking this option would be a clientside fix but what I need, from a delivery perspective, is a to change my MKV such that it is not read as YUV content in the first place - if such a thing is possible.

And what is the content you are releasing?

It's made from video game footage, encompassing voice acting and suchlike. Adobe Premiere & After Effects generate the source file which is then MKV'd.

It's possible that this source file, i.e. the uncompressed AVI being produced at the end of the Premiere/AE cycle is inherently in the YUV colour space - in which I case I need to know how to change that (there appears to be no option in Adobe concerning this matter when extracting the file).

Alternatively, it may be that the YUV space is being picked up later when running through avisynth, staxrip, handbrake, etc. for the MKV process, I don't know. But in the end, I suppose what I am asking is:

1) Can an MKV/H264 video file NOT be in YUV? So that we bypass the VLC/nvidia hardware problem?
2) Assuming a native MKV/H264 RGB is possible, how do I go about getting one from a large, uncompressed avi, given that Staxrip, Handbrake and the like appear to have no options for it?

[sneaker_ger]

I'm not familiar with this problem, but you can't really get around YUV. If PC levels don't work correctly, what about TV levels?
i.e. in AviSynth:
ColorYUV(levels="PC->TV")

?

/edit:
nvm, I'm still half asleep and didn't read correctly.

[Nevilne]

You can try to convert video to fullrange pc.601 and add colormatrix flags to x264, that way software and hardware decoding would be same. However, VLC is _bad_, so your problem might be unsolvable (for example some software decoders dont respect colormatrix and go by resolution, or ignore fullrange).

[Lazarus9]

You can try to convert video to fullrange pc.601 and add colormatrix flags to x264, that way software and hardware decoding would be same.


Is this something I would do to the source, uncompressed avi? And if so, will those flags remain after running through Staxrip or Handbrake? (I'm not an encoding expert :( )

There are lots of people complaining about washed out blacks on nvidia cards on VLC's forum, e.g.

http://forum.videolan.org/viewtopic.php?f=2&t=73578

and the solution given is always to turn off YUV->RGB hardware conversion on those cards and it works. Unfortunately I can't personally ask each person who downloads the MKV to do this (target is about 100k), which is why I'm looking for a more permanent solution given the prevalence of nvidia and VLC in the gaming community.

Presumably an MKV in pc.601/fullrange would be immune to the problem, since no hardware conversion would be taking place?

[ForceX]

If I am not mistaken, it is a renderer side issue, not a container issue; you can not fix it with a video flag. All modern videos are compressed in YUV colorspace. The problem is that nvidia drivers, by default, output RGB to Limited (16-235) range, instead of Full (0-255) range used in most monitors. There's an option in nvidia's control panel which lets you enable Full range for videos, but that's not feasible in your specific case. Maybe you can recommend your users to use a different player like MPC-HC.

[Nevilne]

Is this something I would do to the source, uncompressed avi? And if so, will those flags remain after running through Staxrip or Handbrake? (I'm not an encoding expert :( )


Yes, you will have to make avisynth file for your lossless encode

Avisource(filename.avi)
ColorMatrix(mode="Rec.709->Rec.601", clamp=0, outputFR=true)

open it in virtualdub or something and resave as same lossless encode.

Then open your GUI encoder and manually add those lines to x264 configuration:

--fullrange on --colorprim smpte170m --transfer smpte170m --colormatrix smpte170m

(I think pc.601 is smpte170m?)

I'm not familiar with GUI tools, you might have to open their text preset and add those lines there.

In any case, you will have to see if it works out, gl.

[Lazarus9]

Yes, you will have to make avisynth file for your lossless encode

Avisource(filename.avi)
ColorMatrix(mode="Rec.709->Rec.601", clamp=0, outputFR=true)

open it in virtualdub or something and resave as same lossless encode.

Then open your GUI encoder and manually add those lines to x264 configuration:

--fullrange on --colorprim smpte170m --transfer smpte170m --colormatrix smpte170m

(I think pc.601 is smpte170m?)

I'm not familiar with GUI tools, you might have to open their text preset and add those lines there.

In any case, you will have to see if it works out, gl.

Thanks for the instructions :)

Most of it makes sense, but Virtualdub returns an error on the 'outputFR' command, saying ColorMatrix does not know of it.

If I remove the outputFR part, I get an error saying 'Avisynth open failure: ColorMatrix: input to filter must be YV12 or YUY2'.

Not where to go from that.

I have the colormatrix plugin in the avisynth plugin directory.