YUV color conversion during MeGUI encoding

[vib]

Hello.

This is my first post here, but despite that I tried to find an answer before actually bothering you.

So, I'm capturing a scene in-game, having tga pictures as the output, which I am then saving as an avi file, using VirtualDub with 'Lagarith lossless codec' being the video compression.

Then I'm going into Vegas (no color adjusting there) and I'm rendering the whole as an uncompressed avi file.

Note: the uncompressed Vegas output looks great color and quality wise. Apart from not being suited for playback ; )

Then I'm finally running MeGUI and encoding the uncompressed Vegas source there. Unfortunately, the MeGUI output looks bad: it has dull and bad looking colors. I am no expert in those things (that's why I am asking you!), but I suspect that the YUV color conversion might be the problem.

Here some samples:

After MeGUI encoding: http://i39.tinypic.com/10eeoow.png
Before MeGUI encoding: http://i44.tinypic.com/o6xq4j.png
Encoded with Sony Vegas: http://i44.tinypic.com/feqy6b.png

The clip which Sony Vegas produced isn't looking as good as a properly encoded MeGUI output, but at least gives you a picture of the whole thing having nothing to do with the bit rate.

On a side note, I've tried different MeGUI versions (0.2.6.1046 & 0.3.4) with different, decent x264 configurations and different x264 revisions.

Here's my AviSynth script:

AVISource("c:\fm\sony\project\0 blur 16 sharpen.avi")
ConvertToYV12()

I realized that I can't just cut out the 'ConvertToYV12()' part : (.

[Blue_MiSfit]

Try this:

ConvertToYV12(matrix="pc.709")

and add "--fullrange on" to your x264 settings.

[vib]

Well thanks, I'll try!

Expect an answer in about 30 minutes.

Edit:

This has fixed the problem, thank you very much!

[Blue_MiSfit]

No probs :)

What happened:

convertToYV12 assumes the TV range rec601 matrix, which ignores data in the 0-15 and 236-255 range. Since your input is HD, it's best to use rec709, but in full range (hence pc.709).

adding --fullrange on to x264 makes x264 signal a full range signal in the bitstream so the decoder knows to assume this when doing RGB conversion / passing this to the renderer. ... I think

[vib]

Okay. This is nearly not noticeable at all to me and I can perfectly live with that, however if I could fix even that I would be very happy.

Before MeGUI encoding: http://i44.tinypic.com/o6xq4j.png
After MeGUI encoding: http://i38.tinypic.com/2mz02f.jpg

As you see, the colors appear to be more blueish after the MeGUI encoding.

[PhrostByte]

This seems to be a common question -- the answer would be good on the wiki page for Convert*. I also wonder if flash/youtube even support fullrange video.

[Blue_MiSfit]

use PC.601 instead of PC.709. Your decoder appears to assume 601 instead of 709 colorimetry :)

Derel

[vib]

could you please explain the background stuff behind your answer? and of course I'm very thankful for your answer again.

Edit: I tried what you said and it fixed the colors. I am very, very grateful to you!

[Blue_MiSfit]

Happy to be of assistance.

Basically, there are 2 different things at play here, each with two possibilities :)

1) Colorimetry - either:
BT. 601 / Rec. 601: http://en.wikipedia.org/wiki/BT.601, OR
BT. 709 / Rec. 709: http://en.wikipedia.org/wiki/Rec._709

2) Level range - either:
"TV" Range (16-235 luma out of 0-255 possible values), OR
"PC" Range (0-255 luma out of 0-255 possible values)

You have RGB video, and must convert it to YV12 for x264 or most any other MPEG* encoder. When you convert RGB to YV12, you must specify a matrix that defines whether the conversion should follow the 601 or 709 standards, and whether it should output YV12 using the full PC range, or scaled to comply with the TV range.

It doesn't REALLY matter what matrix you use, so long as the reverse process in the playback application (YV12 -> RGB for your display) uses the same matrix! So you see, it's kind of a backwards choice you have to make depending on your target.

If you're playing back on Media Player Classic using the EVR-CP renderer, it will (if I recall correctly) use TV ranges by default, and 601 for SD and 709 for HD.

If you're playing back on Flash, I think it assumes PC.601 in all cases, but I'm not totally sure. This is why one must experiment as you have done.

The differences between 601 and 709 are usually pretty minor, but they were quite visible in your sample (the color shift). The differences between TV and PC are, however quite huge!! This produces the washed out look you originally asked about.

Ultimately if you get lost, just try all 4 possibilities until you find the one that ends up looking correct in your player of choice :D

Final note - x264 assumes TV range unless you tell it otherwise - hence the suggestion to set --fullrange on to override this behavior.

Derek

[vib]

Wow, thank you for the great and easy to understand explanation.

[Blue_MiSfit]

No problem :)

BTW, a great way to play with different YV12 -> RGB conversion matricies is to use Haali Renderer in MPC-HC. Its config page while running has the option of switching these 4 different parameters on the fly, so you can see the differences back and forth.

Derek

[vib]

How can I enable the Halli renderer? As you can see on the picture, it is grayed out.

http://i37.tinypic.com/689bpv.png

[Blue_MiSfit]

Install Haali media splitter. The renderer comes with it.

I guess there are some things wrong with Haali renderer (some slightly incorrect math for RGB conversion I think), but it's useful for testing.

Derek

[Dr@ke]

Hi sorry for bringing up a old thread, my first noob post and I thought the question I'm having is similar to the above.

I'm currently making backups of my blu-ray using megui but I subbed in the x264 encoder with 10 bit patched versions (it works! but need to add some other commands into the cmdline thingy).

I understand that I have to use for my blu ray rips usually at 720p:

ConvertToYV12(matrix="pc.709")

and add "--fullrange on" to your x264 settings.

But I also make another copy; a SD 480p/576p downscale from blu-ray to play on my not so high performance portable notebook. So is it correct to assume I will need to use ConvertToYV12(matrix="PC.601") instead of the above 709 since from reading around 709 is for HD and 601 is for SD?

Thanks in advance for anyones input, much appreciated.

[J_Darnley]

Blurays are already YV12, so those convert lines do nothing. For SD you can either use the ColorMatrix filter or just use --colormatrix bt709 when encoding.

[EDIT] Also, blurays are not full range.

[Dr@ke]

Thanks for the advice!!!

Sorry for being a pain in the butt, the 601 and 709 and full range or PC levels (0-255) / limited range or TV range(16-235) are the things thats confusing the hell out of me.

Oh I see so considering my source is Blu-ray then it doesn't matter if I downscale to SD resoultion or not then I just use the --colormatrix bt709 / ColorMatrix(hints=true, dest=0, interlace=false). I have been reading this in regards to whether I need to use 609 or 709 - http://avisynth.org/mediawiki/Colorimetry

Sorry I just want to completely nail this - would it be correct if I'm encoding from DVD source then i need to use ColorMatrix(hints=true, dest=2, interlace=false) or --colormatrix smpte170m ?

I also like to use b-pyramid on the x264 settings and I understand if I encode the blu-ray I need to use strict. So is it correct to assume even though I'm also doing another downscale encode from the blu-ray to SD resolution, I still need to use strict not normal as the source itself is blu-ray?

Also, blurays are not full range.

Ah I was reading this in regards to luminance levels - http://avisynth.org/mediawiki/Luminance_levels

And since I only watch the backups I made on notebook monitors / PC especially when I'm on the move I thought it be best to change it to full/pc range (0-255) instead of leaving it as TV range from the source Blu-ray; I wasn't sure also on how to apply these ranges on the encodes via avisynth until came across the --fullrange on x264 settings. Have I mis-understood this?:confused:

Thanks again for your help J_Darnley and also in advance for anybody else's advice ----- sorry again for being a pain as well T_T

[J_Darnley]

Okay, it can be confusing. The 601 and 709 refers to standards which define two numbers which get used in an algorithm to convert RGB into Y'CbCr (commonly written as YUV despite them being different) and the reverse. The PC or TV ranges refer to which algorithm to use which produce 0-255 or 16-235 (luma) and 156-240 (chroma) respectively.

At present the --colormatrix and --fullrange flags just set these options in the output file to let players know what to do with the video.

If you are encoding from a bluray source you should set --colormatrix bt709. If you alter the colour to bt601 then set --colormatrix bt470bg or --colormatrix smpte170m (they are functionally equivalent).

If you're encoding from DVD then you can probably set --colormatrix bt470bg or --colormatrix smpte170m.

I guess some people have found DVDs with different colour matrices because ways of dealing with them exist. DGIndex will display the matrix detected (or assumed) in the information window and print it in the log file and the d2v file. MPEG2Source can output hints which will let ColorMatrix know the input.

My suggestion is to let ffdshow decode the video and convert it into RGB, that way I know how it is doing it and will do so automatically based on the flags you set when encoding.

[nm]

Oh I see so considering my source is Blu-ray then it doesn't matter if I downscale to SD resoultion or not then I just use the --colormatrix bt709 / ColorMatrix(hints=true, dest=0, interlace=false). I have been reading this in regards to whether I need to use 609 or 709 - http://avisynth.org/mediawiki/Colorimetry

As that page explains, it is indeed a good idea to convert to Rec.601 when downscaling HD to SD. Some renderers and players assume that SD is always Rec.601 and HD is Rec.709.

Convert with:

ColorMatrix(mode="Rec.709->Rec.601", interlaced=false)

(Those are default values, so plain ColorMatrix() should do the same.)


I also like to use b-pyramid on the x264 settings and I understand if I encode the blu-ray I need to use strict. So is it correct to assume even though I'm also doing another downscale encode from the blu-ray to SD resolution, I still need to use strict not normal as the source itself is blu-ray?

No. Use whatever the target playback device supports. The original source format is irrelevant.

And since I only watch the backups I made on notebook monitors / PC especially when I'm on the move I thought it be best to change it to full/pc range (0-255) instead of leaving it as TV range from the source Blu-ray

Nearly all video has 16-235 ("TV") luminance levels and that's what software players and renderers expect too. You are asking for trouble by converting to full range.

[Dr@ke]

Thanks again for your advice guys, helped alot =D.

Really appreciate it!

[hello_hello]

And since I only watch the backups I made on notebook monitors / PC especially when I'm on the move I thought it be best to change it to full/pc range (0-255) instead of leaving it as TV range from the source Blu-ray; I wasn't sure also on how to apply these ranges on the encodes via avisynth until came across the --fullrange on x264 settings. Have I mis-understood this?:confused:

Have a look at the options for your video card. Mine for example (nvidia), has an option under "adjust video color settings" in the nvidia control panel which can be set to "Full (0-255)".
That way any video you play using any media player should be expanded from TV levels to PC levels and will display correctly on your monitor.
Why video cards don't expand video to PC levels by default I don't really know....

I never use ColorMatrix(hints=true, dest=0, interlace=false), especially when converting from HD to SD as I'm pretty sure if there's no colorimetry info in the HD video stream (or SD for that matter) then ColorMatrix assumes R.601 and no color conversion will take place. I always manually convert as has been mentioned previously.

As a side issue.... I did some experimenting a while back, and at least when it comes to XP and the WM9 renderer, as best as I could tell, for video to display using R.709 the width must be equal to or greater than 1200 AND the height must be equal to or greater than 578.
So disappointingly, a cropped 720p encode with dimensions such as 1280x544 or 1280x528 etc will display using the wrong colorimetry (R.601 instead of R.709) when played on a PC.

[roozhou]

Have a look at the options for your video card. Mine for example (nvidia), has an option under "adjust video color settings" in the nvidia control panel which can be set to "Full (0-255)".
That way any video you play using any media player should be expanded from TV levels to PC levels and will display correctly on your monitor.
Why video cards don't expand video to PC levels by default I don't really know....

I never use ColorMatrix(hints=true, dest=0, interlace=false), especially when converting from HD to SD as I'm pretty sure if there's no colorimetry info in the HD video stream (or SD for that matter) then ColorMatrix assumes R.601 and no color conversion will take place. I always manually convert as has been mentioned previously.

As a side issue.... I did some experimenting a while back, and at least when it comes to XP and the WM9 renderer, as best as I could tell, for video to display using R.709 the width must be equal to or greater than 1200 AND the height must be equal to or greater than 578.
So disappointingly, a cropped 720p encode with dimensions such as 1280x544 or 1280x528 etc will display using the wrong colorimetry (R.601 instead of R.709) when played on a PC.
What video card and driver are you using? I never use VMR9 in XP. Instead I use either Overlay Mixer for DXVA or MadVR for SD->full screen playback. Neither gives me 601/709 issue.