x264 color range input

[Niddhogg]

:o

Okay, I did this :
- read the info included with the binaries
- read an Octillian threads ;)
- searched source for keywords that might help

Buttttt... no success in finding out if x264 expects reduced range YUV or can also use full range scales.
Then, is it actually at all possible to have 0-255 output with x264 ?
The samples I've seen so far were all capped.

(Since I don't use a tv to watch back dvd's there's no need for me to cap the 0-255 dvd's to tv range.
And ditching what's outside the range doesn't seem that appealing.)

Thanksy.

[foxyshadis]

x264 doesn't care either way (it only would if it could import rgb streams), only the decoder's colorspace conversion method is important.

[Niddhogg]

:thanks:

Okay, so :
- upsample YV12 --> RGB 0-255
- ConvertToYV12(matrix="PC.709") (or 601 ?)

Would keep the full range, providing I don't induce clipping anywhere ?
I suppose that rules out MPEG2Source() then.

http://img124.imageshack.us/img124/1493/operavsie1sd.gif

[cacepi]

I suppose that rules out MPEG2Source() then.
DGDecode has LumaYV12(), which outputs 0-255 color.

[Wilbert]

MPEG2Source doesn't clip the luma range to [16,235]. At least if you request YV12 from it.

[Niddhogg]

DGDecode has LumaYV12(), which outputs 0-255 color.
How would I use LumaYV12() for opening the d2v ?
It keeps replying "Invalid arguments to...".

MPEG2Source doesn't clip the luma range to [16,235]. At least if you request YV12 from it.
What switch of these would enable that :
MPEG2Source(str "d2v", int "idct", int "cpu", bool "iPP", int "moderate_h", int "moderate_v", str "cpu2", bool "upConv", bool "iCC", bool "i420", int "info", bool "showQ", bool "fastMC")
The default MPEG2Source() is clipping to TV ranges, this is from the DGVfapi readme :
DGVfapi 1.4.7
...
DVD2AVI's clipping feature is not supported.

I tried a few samples to determine what sampling methods / sequences would keep the color range closest to the original :
0 - 193517 - VOB (http://img368.imageshack.us/img368/4923/0193517vob4zm.png)
1 - 188804 - MPEG2Source (http://img368.imageshack.us/img368/3626/1188804mpeg2source6ec.png)
2 - 182041 - MPEG2Source.ColorYUV(TV to PC) (http://img368.imageshack.us/img368/3061/2182041mpeg2sourcecoloryuvtvto.png)
3 - 198723 - VFAPI & AviSource (http://img368.imageshack.us/img368/3299/3198723vfapiavisource4fb.png)
4 - 193941 - VFAPI & AviSource.ConvertToYV12(PC.601) (http://img368.imageshack.us/img368/5310/4193941vfapiavisourceconvertto.png)
5 - 196765 - VFAPI & AviSource.ConvertToYV12(PC.709) (http://img368.imageshack.us/img368/3170/5196765vfapiavisourceconvertto.png)
6 - 183115 - VFAPI & AviSource.ConvertToYV12(PC.601) PF (http://img138.imageshack.us/img138/8154/6183115vfapiavisourceconvertto.png)
7 - 188993 - VFAPI & AviSource.ConvertToYV12(PC.709) PF (http://img138.imageshack.us/img138/4194/7188993vfapiavisourceconvertto.png)

The second numbers indicate the total number of colors in the decoded frame.
PF = Post Filters.

[edit]

ImageShack was offline, had to use slower host.

[cacepi]

How would I use LumaYV12() for opening the d2v ?
MPEG2Decode().LumaYV12(int "lumoff", float "lumgain")

Or you can just use ColorYUV(levels="TV->PC")


What switch of these would enable that :

MPEG2Source() doesn't change the colorspace unless you've told it to do so.


I tried a few samples to determine what sampling methods / sequences would keep the color range closest to the original:
Ah, Nadesico movie. In DGIndex it reports Colorimetry "ITU-R BT.709"; it's already clipped. This is the Region 2 disc.

So how to preserve original colors? Don't perform any conversions (i.e. no 0-255).

[Wilbert]

What switch of these would enable that :
By default.

The default MPEG2Source() is clipping to TV ranges
No, it doesn't. You can check by using

MPEG2Source()
ColorYUV(analyze=true)

[Niddhogg]

MPEG2Decode().LumaYV12(int "lumoff", float "lumgain")
Or you can just use ColorYUV(levels="TV->PC")
MPEG2Source() doesn't change the colorspace unless you've told it to do so.

First method didn't work, second gives me different colors (see samples). I'd think if MPEG2Source would pass on the YV12 space as it is in the VOB it would display the exact same image after YV12 upsampling. But it doesn't, while using VFAPI & AviSource (which uses ConvertToRGB) does display an identical image, try switching between the VOB and this image.

Ah, Nadesico movie. In DGIndex it reports Colorimetry "ITU-R BT.709"; it's already clipped. This is the Region 2 disc.
Yes, but afaik 602 / 709 only specify sampling methods, not clipping per se ?

By default.
Yes, I can see it displays YV12, but this is different from the image when viewed as the VOB.
I would expect the VOB using the same YV12 upsampling as MPEG2Source, but the images differ.
Especially green is noticable and so is the general brightness.


No, it doesn't. You can check by using

MPEG2Source()
ColorYUV(analyze=true)

Not in output color range, but the color count does go down and display is different.
So either MPEG2Source clips before parsing to the YV12 upsampler, OR the VOB uses a different upsampler.
Is there a way to check which is the case ?

Also, the only 2 images that keep the detail in the dark grey shadow area underneath the ship are :
1 - 188804 - MPEG2Source (YV12)
3 - 198723 - VFAPI & AviSource (RGB)

[Wilbert]

Yes, I can see it displays YV12, but this is different from the image when viewed as the VOB.
I would expect the VOB using the same YV12 upsampling as MPEG2Source, but the images differ.
Especially green is noticable and so is the general brightness.
Look up the AviSynth plugin ColorMatrix. It corrects this. The vobs should be upsampled using Rec.709 coefficients (not always true, but in this case it is), but the standard tools use Rec.601.

I think this answers also your second question. Nb, it's not that the vob uses a different upsampler, it's the decoder (VDubMod, or a software dvd player or whatever you use) decoding it correctly.

[Niddhogg]

Look up the AviSynth plugin ColorMatrix. It corrects this. The vobs should be upsampled using Rec.709 coefficients (not always true, but in this case it is), but the standard tools use Rec.601.

I think this answers also your second question. Nb, it's not that the vob uses a different upsampler, it's the decoder (VDubMod, or a software dvd player or whatever you use) decoding it correctly.
I've read the manual of your :rolleyes: ColorMatrix before, but was put off by this tiny line :

Version 1.9. February 23th, 2005
Copyright (C) 2004-2005 Wilbert Dijkhof.
contact: wilbertdijkhof at yahoo.com

As a final note. This filter will clamp (= round) your input video to CCIR-601 compliant values (these ranges are 16-235 for the luma component and 16-240 for the chroma component).
Ok, tried anyway and it gave me these values :

1b - 185993 - MPEG2Source.ColorMatrix() (http://img76.imageshack.us/img76/4065/1b185993mpeg2sourcecolormatrix.png) (for YV12)
1c - 229550 - MPEG2Source.ColorMatrix().ConvertToRGB(Rec601) (http://img217.imageshack.us/img217/5091/1c230016mpeg2sourcecolormatrix.png) (for RGB)

Even though the color count goes down 3.9 % compared to the vob the image is very close to it, just some red that's a bit off (rounding I suppose).
More important is that after ConvertToRGB the image is identical to using VFAPI & AviSource, but with 15.8 % more colors.
Important since I prefer (or am forced to) using RGB space for filters.

Don't mind admitting I'm now more terribly confused than is good for me (sleepless nights ;) ) :

1.
Using VDM, is the VOB display accurate and does this rely on the installed YV12 upsampler (think it's a more accurate name than 'decoder') ?
Currently using Helix's for this since it gives a slightly larger range compared to XviD's.
Before I also had ATI's installed which would always output clipped values to the display.
2.
Can it be that both Helix and XviD detect Rec.709 and decide to output to 0-255 since they are feeding a monitor, not a tv ?
3.
Something which then ATI's doesn't do and instead displays the vob as 'intended' ?

Thanksy (again)... ;)

[Wilbert]

1.
Using VDM, is the VOB display accurate and does this rely on the installed YV12 upsampler (think it's a more accurate name than 'decoder') ?
VDubMod uses its own upsampler. Yes that one is good.

2.
Can it be that both Helix and XviD detect Rec.709 and decide to output to 0-255 since they are feeding a monitor, not a tv ?
No. It is true that XviD (and perhaps Helix) is not clipping the luma range. However, if you request RGB from XviD (upon display or whatever), it will clip to YUV [16,235] before converting to RGB [0,255].

3.
Something which then ATI's doesn't do and instead displays the vob as 'intended' ?
? How exactly are you displaying the vob here?

[Niddhogg]

No. It is true that XviD (and perhaps Helix) is not clipping the luma range. However, if you request RGB from XviD (upon display or whatever), it will clip to YUV [16,235] before converting to RGB [0,255].
Which would mean viewing ANY YV12 source on a monitor gets clipped to start with...

4.
Does that also make MPC's internal MPEG-2's filter setting 'TV -> PC' less effective as it is applied after upsampling ?
5.
If so, are there actually YV12 upsamplers that do not clip prior to display ?

? How exactly are you displaying the vob here?
Ehrm... via VDM, but here it gives absolute 16-235 ranges whereas both XviD and Helix upsampling gives 0-255.

[Wilbert]

Ehrm... via VDM, but here it gives absolute 16-235 ranges
I'm pretty sure it uses the conversion YUV [16,235] -> RGB [0,255].

Btw, some dv codecs keep the luma range intact (ie YUV [0,255] -> RGB [0,255]).

4.
Does that also make MPC's internal MPEG-2's filter setting 'TV -> PC' less effective as it is applied after upsampling ?
Why do you think it is applied after and not during the upsampling?

[Niddhogg]

I'm pretty sure it uses the conversion YUV [16,235] -> RGB [0,255].
I've uninstalled codec so can't test now, but i'll try again.

Btw, some dv codecs keep the luma range intact (ie YUV [0,255] -> RGB [0,255]).
Good, i'd like to compare those to the two currently in use.
Wiki time :

The quantization process represents each sampled signal by an array of numbers; one for Y, one for U, and one for V. The Y channel has numbers appearing 13.5 million times per second, while the other two have 3.375 million numbers per second each. Each number in these arrays is between 0 and 255, ...

The chroma subsampling is 4:1:1 for NTSC or 4:2:0 for PAL, ...

Why do you think it is applied after and not during the upsampling?
Hunch, since doing it before would require MPC to include its own support for YV12 sampling.
Inventing the wheel twice while it could also rely on present sampler, it does the same with swf support.

[Wilbert]

Hunch, since doing it before would require MPC to include its own support for YV12 sampling.
Qualitywise, it wouldn't be very good to convert to RGB [16,235] first before stretching to [0,255]. So, i doubt this is happening here. I think all MPEG-2 decoders use [16,235] YUV -> RGB [0,255] by default.

[Niddhogg]

I 'm pretty sure it uses the conversion YUV [16,235] -> RGB [0,255].
Managed to dig up ATI's YV12 (v4.12) again, here's what it outputs in VDM :

1d - 190021 - MPEG2Source (ATI) (http://img89.imageshack.us/img89/8523/1d190021mpeg2sourceati9cs.png)

It gives the highest number of colors when compared to XviD's and Helix's, but 'black' is at RGB 16,16,16...
For this frame Helix gave 188804 and XviD 188805, nearly identical, while ATI's is 1217 colors 'off'.
Annoying to say the least, more so since I didn't expect this to be possible.

Btw, some dv codecs keep the luma range intact (ie YUV [0,255] -> RGB [0,255]).

Found some, but think PAL DV's 4:2:0 differs from MPEG-2's ?
Too bad, I was interested in finding out if (some) dvd's used the full range,
took this from the Rec. 601 (didn't have 709 a.t.t.) :


3.5.3 Quantization

In the case of the 4:2:2 systems described in this Recommendation, levels 0 and 255 are reserved for synchronization
data, while levels 1 to 254 are available for video.
Given that the luminance signal is to occupy only 220 levels, to provide working margins, and that black is to be at level
16, the decimal value of the luminance signal, Y -, prior to quantization, is:
Y - = 219 (EY ¢ ) + 16
and the corresponding level number after quantization is the nearest integer value.

...

3.5.5 Limiting of Y, CR , CB signals

Digital coding in the form of Y, CR, CB signals can represent a substantially greater gamut of signal values than can be
supported by the corresponding ranges of R, G, B signals. Because of this it is possible, as a result of electronic picture
generation or signal processing, to produce Y, CR, CB signals which, although valid individually, would result in out-ofrange
values when converted to R, G, B. It is both more convenient and more effective to prevent this by applying
limiting to the Y, CR, CB signals than to wait until the signals are in R, G, B form. Also, limiting can be applied in a way
that maintains the luminance and hue values, minimizing the subjective impairment by sacrificing only saturation.

Maybe it's silly, but is there actually any need for dvd's (stored format) to include 'synchronization
data' or 'working margins' in its pixel representation as it would in the broadcast flavour ?

YUV Wiki :

Digital television and DVDs preserve their compressed video streams in the MPEG-2, which uses a full YUV color space.


I think all MPEG-2 decoders use [16,235] YUV -> RGB [0,255] by default.

ATI's seems not to for one, but I'm keen on knowing for sure that DVD's are stored constricted or not.
If they do and the MPEG-2 decoder in use is discarding 1/8 th. of the possible values even prior to upsampling to RGB 0-255...

[Wilbert]

Managed to dig up ATI's YV12 (v4.12) again, here's what it outputs in VDM :
Remove it :) Let XviD or Helix do the YV12->RGB conversion (see YV12 FAQ).

Found some, but think PAL DV's 4:2:0 differs from MPEG-2's ?
Indeed. See http://www.avisynth.org/Sampling

but I'm keen on knowing for sure that DVD's are stored constricted or not.
It's stored at the full range [0,255], but the black is centered around Y=16, and white around Y=235. I say centered because you can find values which are a bit smaller than 16 and a bit larger than 235. Use Limiter(analyze=true) (see avs documentation) to make the out of bound values visible, and see for yourself.

[Niddhogg]

Remove it :) Let XviD or Helix do the YV12->RGB conversion (see YV12 FAQ).
Done. :)
Odd thing is that I can't remember ever installing it in first place, don't even have an ATI card...
Makes you wonder how many people still have it installed unwittingly.

It's stored at the full range [0,255], but the black is centered around Y=16, and white around Y=235. I say centered because you can find values which are a bit smaller than 16 and a bit larger than 235. Use Limiter(analyze=true) (see avs documentation) to make the out of bound values visible, and see for yourself.
Yes, it shows many off limit values when using only MPEG2Source() and almost none when applying ColorMatrix() after that.
But that's too late, isn't it ?
If I understood correctly ColorMatrix is correcting values that were obtained by a Rec.709 interpreted using Rec.601 conversion.
Values that as result of this conversion resulted in < 0 or > 255 are set off as being 0 or 255 respectively.
ColorMatrix can't know / prevent this since it's always applied after conversion.

Tried a comparison using :
frame0 = ImageSource("C:\0 - 193517 - VOB.png", 0, 0)
frame1 = ImageSource("C:\1b - 185993 - MPEG2Source.ColorMatrix().png", 0, 0)
frame2 = ImageSource("C:\3 - 198723 - VFAPI & AviSource.png", 0, 0)
frame3 = ImageSource("C:\1c - 230016 - MPEG2Source.ColorMatrix().ConvertToRGB(Rec601).png", 0, 0)
clip10 = ConvertToYUY2(frame0, matrix = "PC.709")
clip11 = Overlay(frame0, frame1, mode = "Difference", output = "YUY2", pc_range = true).ColorYUV(off_y=-110).ColorYUV(analyze = true)
clip12 = Overlay(frame0, frame2, mode = "Difference", output = "YUY2", pc_range = true).ColorYUV(off_y=-110).ColorYUV(analyze = true)
clip13 = Overlay(frame0, frame3, mode = "Difference", output = "YUY2", pc_range = true).ColorYUV(off_y=-110).ColorYUV(analyze = true)
StackVertical(StackHorizontal(clip10, clip11), StackHorizontal(clip12, clip13))


Result gives :

Top left VOB, tr MPEG2Source.ColorMatrix(), bl VFAPI & AviSource, br MPEG2Source.ColorMatrix().ConvertToRGB(Rec601) (http://instant.myonlineimages.com/Uploads/Images/462690703750010 - tl VOB, tr MPEG2Source.ColorMatrix(), bl VFAPI AviSource, br MPEG2Source.ColorMatrix().ConvertToRGB(Rec601).png)

That's a lot of difference, provided I followed a correct method to compare.
Perhaps you can explain (again ;) ) ?

Thanksy.

[Wilbert]

1) ColorMatrix clamps to CCIRR range ([16,235]) as explained in the docs.

2) Could you upload a ~10 frames VOB? Then i can check it myself :)

[Niddhogg]

1) ColorMatrix clamps to CCIRR range ([16,235]) as explained in the docs.
Myes, that's also why I was hesistant to use it.
But it shouldn't matter for the RGB output on screen, right ?

sample (http://0.dr.ag/s/Testy4Wilbert.vob.HtMl)

(oopsy, erroneously read 10 secs... hope you're not on copper thread ? ;) )

[Wilbert]

I'm unable to connect to that site. Could you put it on rapidshare.de?

[Niddhogg]

RapidShare (http://rapidshare.de/files/16832338/Testy4Wilbert.vob.html)

[Wilbert]

You should use the following comparison:

frame0 = ImageSource("C:\0 - 193517 - VOB.png", 0, 0).ConvertToYUY2()
frame1 = ImageSource("C:\1b - 185993 - MPEG2Source.ColorMatrix().png", 0, 0).ConvertToYUY2()
frame2 = ImageSource("C:\3 - 198723 - VFAPI & AviSource.png", 0, 0).ConvertToYUY2()
frame3 = ImageSource("C:\1c - 230016 - MPEG2Source.ColorMatrix().ConvertToRGB(Rec601).png", 0, 0).ConvertToYUY2()
clip10 = frame0
clip11 = Subtract(frame0, frame1).ColorYUV(analyze = true)
clip12 = Subtract(frame0, frame2).ColorYUV(analyze = true)
clip13 = Subtract(frame0, frame3).ColorYUV(analyze = true)
StackVertical(StackHorizontal(clip10, clip11), StackHorizontal(clip12, clip13))

to minimize errors ...

I'm not sure how you made clip10 (screenshot in DGIndex or VDubMod)? If you did it in DGIndex, the wrong coefficients will be used (always Rec.601). If it is from VDubMod should be good. For the vfapi screenshot, also the wrong coefficients are used (Rec.601). frame1 and frame3 should be the same.

I can't watch your picture anymore, so i can't check the above.

[tritical]

I'm not sure how you made clip10 (screenshot in DGIndex or VDubMod)? If you did it in DGIndex, the wrong coefficients will be used (always Rec.601). If it is from VDubMod should be good. For the vfapi screenshot, also the wrong coefficients are used (Rec.601). frame1 and frame3 should be the same.
It depends on the version of dgindex. If it is v1.4.1 or later, than the correct coefficients (FCC, rec601, rec709, or smpte 240) are used in the rgb conversion. That applies to the frame display in dgindex, vfapi (d2v not avs), and upconv=2 in dgdecode. When that conversion is done the input yuv values are not clamped (nowhere in dgindex or dgdecode are the pixel values ever clamped to 16-235/240 range). Output values from the rgb conversion that are below 0 are set to 0 and above 255 are set to 255. If scale is set to pc_scale then all coefficients are scaled by 255/219 and 255/240, if scale is set to tv_scale then the coefficients are not scaled. So tv_scale is equal to matrix="pc.xxx" in avisynth's conversions and pc_scale is equal to matrix="recxxx" in avisynth's conversions.

[Wilbert]

It depends on the version of dgindex. If it is v1.4.1 or later, than the correct coefficients (FCC, rec601, rec709, or smpte 240) are used in the rgb conversion. That applies to the frame display in dgindex, vfapi (d2v not avs), and upconv=2 in dgdecode.
Thanks for your correction!

When that conversion is done the input yuv values are not clamped (nowhere in dgindex or dgdecode are the pixel values ever clamped to 16-235/240 range). Output values from the rgb conversion that are below 0 are set to 0 and above 255 are set to 255.
I think it shouldn't matter whether you clamp first and convert to RGB after that, or you convert first and clamp after that?

[NoX1911]

One word to the initial question always doing a TV->PC conversion:

Overlay seems to correct YUV color ranges automatically (for correct onscreen output). If you change output renderer to VMR9 instead you get the original, uncorrected/low luma range (16..235). You need a software converter like ffdshow then to do the additional TV-PC conversion to fix the luma range.

So theres no need to do a TV->PC conversion on YUV material if you use a standard system with overlay output. It shouldn't depend on the codec, just on colorspace and output renderer.

[tritical]

I think it shouldn't matter whether you clamp first and convert to RGB after that, or you convert first and clamp after that? I guess it depends on if all sets of input containing values outside of the 16-235/240 range lead to values outside of 0-255 range after the conversion. It might be interesting to see if it leads to a difference and if so how much.