Inaccurate colors on final encode

[Soichiro]

Hello, I was working with a bluray source, and after making the final encode I noticed that the colors on the x264 encode were very different from the lossless. The lossless was done in YV12 LAGS and encoded using the BT.601 colormatrix.

Proper colors: http://ayamachi-subs.org/pics/ef01_lossless.png
x264 colors: http://ayamachi-subs.org/pics/ef01_final.png

The only filter used on the x264 encode was the latest version of TextSub, which shouldn't have caused any problem like this. I even tried experimenting in Avisynth with colormatrix() to see if the matrix being detected improperly could cause the problem, but the testing didn't offset the colors nearly as much as in the actual encode. I know it's not a quality problem, as the final quants were in the 13-14 range.

Here are the x264 settings I used on it (with x264 r1088):

x264.exe --pass 1 --bitrate 1691 --stats "%~dp1.stats" --keyint 400 --min-keyint 12 --bframes 6 --b-adapt 2 --b-pyramid --weightb --direct auto --subme 5 --partitions none --me dia --merange 32 --threads auto --thread-input --progress --no-psnr --no-ssim --colormatrix "smpte170m" --output null %1

x264.exe --pass 2 --bitrate 1691 --stats "%~dp1.stats" --keyint 400 --min-keyint 12 --ref 8 --mixed-refs --no-fast-pskip --bframes 6 --b-pyramid --weightb --direct auto --subme 9 --trellis 2 --partitions all --8x8dct --me umh --merange 32 --threads auto --thread-input --progress --no-psnr --no-ssim --colormatrix "smpte170m" --output "%~dpn1.mkv" %1

Can anyone help figure out what could be causing this?

[scharfis_brain]

looks like luma range got stretched from 16...235 to 0...255.

but I don't know why.

Post your avs-script

[Atak_Snajpera]

looks like luma range got stretched from 16...235 to 0...255.
Source is already 0..255 so it looks like unnecessary TV -> PC conversion was made

[Soichiro]

For the lossless the script is
AVCSource("00001.dga")
LanczosResize(1280,720,0,2,0,-2)
ColorMatrix()
#ep 1
Trim(0,34188)
undot()
And for the x264:
AVISource("ef BD 01.avi")
TextSub("ts.ass")


I was playing around with my decoder settings and wonder if it might be related to CoreAVC. The colors don't seem as off when using ffdshow's H.264 decoder (using r1355) as in CoreAVC (1.8.5).

http://ayamachi-subs.org/pics/ef01_ffdshow.png

Edit again: It seems like it was related to the TV->PC conversion. If I set CoreAVC to always detect input as PC levels, the output looks exactly like ffdshow's.

[lexor]

what happens if you remove: --colormatrix "smpte170m", I'm pretty sure that will make it go away, for the reasons Atak_Snajpera stated.

[Soichiro]

I had two encodes, one with that and one without, and it didn't seem to make any difference.

[lexor]

OK, now I'm confused. What is the source of what? Your lossless script has AVC source, your x264 script has AVI source. So does x264 output goes to lossless? And you have 2 x264 encodes.

You've got to clear this up for us to help better.

[Soichiro]

BD source (H.264 m2ts) -> DgAVCDecode -> First avs script -> LAGS lossless (AVI) -> Second script -> x264 output

And yes, I made two x264 encodes to test whether the colormatrix string would have an effect (which it didn't seem to)

[Esurnir]

dumb suggestion of the day : does '--fullrange on' help ?

So you don't have to do a conversion

[lexor]

does your edit in #4 mean it's working to your satisfaction now?

since the sources for two images are different, it could be that the lossless decoder and h264 decoder are not working in exactly the same colour space. Decoders take liberty in deciding if TV is 16...235 or 16...240 (though I don't know if it's that big of a difference)

[Comatose]

edit: This is what I get for not clicking the images at first D: Like others have said, it looks like TV->PC conversion... can you cut a section from the file the screenshot is from and upload it?

[Conquerist]

colors don't seem as off when using ffdshow's H.264 decoder (using r1355) as in CoreAVC (1.8.5).

http://ayamachi-subs.org/pics/ef01_ffdshow.png
The difference between ef01_lossless.png and ef01_ffdshow.png is definately BT.709 vs. BT.601 colorspace.

It looks like ef01_ffdshow.png was converted to RGB using BT.709 while ef01_lossless.png used BT.601.

AddingConvertToRGB(matrix="Rec709")to the source script (or the x264 script) should yield colors identical to ef01_ffdshow.png.

PS: Why convert to BT.601 in the first place? Since BT.709 is the standard for HD content, the renderer should use BT.709 on video with a 1280x720 resolution. In my experience, VMR7, VMR9, and Haali Renderer (set to "auto" YUV colorspace) always use the correct colorspace. Adding an unnecessary ColorMatrix() will cause "inaccurate colors" on a correctly configured player.

[Soichiro]

Okay, I got that problem fixed, thank you everyone. (As for your PS, Conquerist, it's because I keep hearing all these conflicting things about what color matrixes to use >.<). However, I managed to come up with a new problem. In some of my SD encodes, the original source will have PC range colors, but the x264 encode gets decoded as TV range (or something?), making the colors look all faded. If I manually set CoreAVC to convert from TV->PC (note: ffdshow does the same thing) then the colors match the lossless, but I don't really want to have to change the CoreAVC settings every time I switch between an SD and HD video. This seems like it happens on most of my SD H.264 encodes but none of my HD ones, and I have encodes both with and without the --fullrange on setting that have this problem.

Edit: Never mind, I think I figured it out, lol.

[Esurnir]

Okay, I got that problem fixed, thank you everyone. (As for your PS, Conquerist, it's because I keep hearing all these conflicting things about what color matrixes to use >.<). However, I managed to come up with a new problem. In some of my SD encodes, the original source will have PC range colors, but the x264 encode gets decoded as TV range (or something?), making the colors look all faded. If I manually set CoreAVC to convert from TV->PC (note: ffdshow does the same thing) then the colors match the lossless, but I don't really want to have to change the CoreAVC settings every time I switch between an SD and HD video. This seems like it happens on most of my SD H.264 encodes but none of my HD ones, and I have encodes both with and without the --fullrange on setting that have this problem.

Edit: Never mind, I think I figured it out, lol.

Did the fullrange had -any- effect ?

Very basicaly, what was the solution, some other people (me) could be interested in it.

[Conquerist]

All video content (save some screen captures and video game footage) has a 16-235 ("TV") luma range, regardless of SD or HD. So, a 16-235 to 0-255 (TV -> PC) conversion is necessary for nearly all content. In a DirectShow player it's performed by either the decoding filter (CoreAVC or ffdshow for H.264) or the renderer (VMR7, VMR9, Haali, etc.). It should be done by the renderer, and doing it in the decoding filter is only a quick-fix solution since most decoding filters don't have an option to perform a levels conversion.

Bottom line, it has nothing to do with the encoder, but rather the playback environment. Any decoding filter with no luma levels correction applied together with Haali Renderer set to TV levels and auto colorspace should produce the right results.

Concerning --fullrange, as the x264 --longhelp states, the VUI settings are not used by the encoder but are merely suggestions to the playback equipment. As far as I know, no DirectShow H.264 decoder takes these flags into account.

Back to Bt.709/BT.601: The YUV to RGB conversion is normally done in the renderer and most decoders don't even do any type of colorspace conversion (except notably CoreAVC and ffdshow). With a correctly configured renderer, colorspace conversion in the decoding filter isn't necessary and as demonstrated can only become a nuisance. Since the renderer chooses which matrix to use when converting YUV to RGB based on resolution, BT.709 to BT.601 conversion is only needed when making a SD resolution encode from a HD source.