New Color Conversion Function

[josey_wells]

I recently got into the foray of converting my DVD collection to H.264. I was frustrated for a period of time with the colors of my converted movies being washed out or incorrect.

To this end I decided to put my programming/math skills to use and create my own colorconversion function from the MPEG2 DVD ISO standard. It is all based on integer arithmetic but is not mutlithreaded at the moment.

It currently only supports YUY2 colorspace. I use the upConv=1 function of MPEG2Source to get to this colorspace since I believe it gives more accurate color conversion.

The usage is as follows:

BZColorspace(clip c, string SourceColorimetry="BT709", string DestinationColorimetry="BT709", bool InputFullrange=false, bool OutputFullRange=true)

SourceColorimetry & Destination Colorimetry
1. BT709
2. BT470_2
3. SMTP170M
4. SMTP240M
5. FCC

Input & Output Ranges

true - Luma and chromas [0, 255]
false - Luma [16, 235] and chroma[16, 240] CCIR(DVD sources)

I use FFDSHOW and set it to use full range RGB32 output with BT709 and everything looks fantastic.

For example:
BZColorspace(SourceColorimetry="SMTP170M", DestinationColorimetry="BT709", InputFullRange=false, OutputFullRange=true)

The main reason I wrote this instead of using the ColorMatrix function was that conversion from YUY2 colorspace 1 to YUY2 colorspace 2 without an intermediate step of converting to the linear luminance and blue, red difference channels is technically incorrect.

The flow is as follows:
YUY2 Colorspace 1(Y1, Cb, Y2, Cr) -> EY1, Eb, EY2, Er Colorspace 1-> EY1, Eb, Ey2, Er Colorspace 2 - > YUY2 Colorspace 2(Y1, Cb, Y2, Cr)

where the conversion to EY1, Eb, EY2, Er Colorspace 1 uses the input full range parameters and the conversion to YUY2 colorspace 2 uses the output full range parameter.

If there is interest I can provide the matrices and source code.

Good luck.

[IanB]

Yes source code is always of interest.

... linear luminance and blue, red difference channels ...

I take this means you "unGamma" the values, convert, then reapply Gamma. How do you do this fast? (I guess when I see the source, I'll know or not)

[josey_wells]

Source code is attached in VS2008.

The YUY2 values are in linear colorspace meaning that the gamma corrections have been applied to the original sources to convert from non linear to linear.

I convert from linear YUY2 to linear luminance and red, green difference channels.

Then I convert the colorspaces and convert back to linear YUY2.

The output device, i.e. monitor/graphics card, performs the gamma correction of the linear data for display.

To be technically correct one would have to convert from linear to gamma source back to linear for colorspaces where the white points differ but I don't believe this will change the results discernibly.

[IanB]

A very nicely documented piece of source code.

Okay you do no Gamma processing

You do purely linear 3x3 transforms, but all of your matrices have no Luma contribution in the Rdiff or Bdiff translations, this is unexpected. Perhaps we need to discuss your matrices a lot more.

Given you do no gamma processing, it seems you have another implementation of ColorMatrix but with different matrices Given the work done to make ColorMatrix super fast, might it not be more fruitfull to just add your matrices to it.


P.S. Also no 601 (default for Avisynth) colour space alias.

[Wilbert]

The main reason I wrote this instead of using the ColorMatrix function was that conversion from YUY2 colorspace 1 to YUY2 colorspace 2 without an intermediate step of converting to the linear luminance and blue, red difference channels is technically incorrect.

I haven't looked at your matrices, but you should get the same results. Why do you think that doing it without intermediate step is incorrect?

[josey_wells]

These are all good questions. I want to explain basic color flow first and then come back to the questions. I apologize if anyone is already familiar with this. Also I am using the ISO/IEC 13818-2:1995 H.262 standard.

When a source captures video it is done in [X, Y, Z]. The [X, Y, Z] are converted to non-linear [R, G, B] based on the white point and x, y primaries.

So [X, Y, Z] -> White Point/Primaries -> [R, G, B] non-linear i.e Table 6-7.

The [R, G, B] are then converted to linear [R', G', B'] in the range [0, 1] through the transfer charateristics in Table 6-8, i.e. Gamma.

[R, G, B]-> Gamma Correction -> [R', G', B'] documented as E'G, E'R, & E'B.

My BIG assumption is that once all the different source transformations are taken into account they will produce identical linear [R',G',B'] values for the same real world color. The white points for all colorspaces are all D65 except for ITU-R BT470-2 System M which is C. I have not done a full blown transformation to verify this.

Now the source linear [R', G', B'] values are converted to [Ey', Eb', Er'] using table 6-9 which produces the linear luminance in range [0, 1] and red blue difference channels in range [-0.5, 0.5].

These values are then converted to Y in the range [16, 235], and Cb and Cr in the range [16, 240] using the formulas.

Y=(219*Ey')+16
Cb=(224*Eb')+128
Cr=(224*Er')+128

These final Y, Cb, & Cr values are what is stored on the DVD.

To reproduce the values on a display the simplified process is
[Y, Cb, Cr] -> Limited/Full Range -> [Y', Eb', Er']-> Color Matrix Coefficients -> [R', G', B']-> Gamma of output device -> [R, G, B]

So to transfer the Y, Cb, Cr values from one colorspace to another I perform the following:

[Y1, Cb1, Cr1] -> Limited/Full Range -> [Y1', Eb1', Er1'] -> [Y2', Eb2', Er2'] -> Limited/Full Range -> [Y2, Cb2, Cr2]

Now I will come back to the questions:

Question 1 - Intermediate step

The conversion from [Y1, Cb1, Cr1] to [Y1', Eb1', Er1'] can be represented in matrix form as [Y1', Eb1', Er1']=C*([Y1, Cb1, Cr1]-B) where B is an offset vector and C is a scaling matrix.

To convert from [Y1', Eb1', Er1'] to [Y2', Eb2', Er2'] can be represented as a pure matrix multiplication since one can solve for the [R', G', B'] from [Y1', Eb1', Er1'] and then convert back to a [Y2', Eb2', Er2'] so [Y2', Eb2', Er2']=D*[Y1', Eb1', Er2'].

Converting back to [Y2, Eb2, Er2] is [Y2, Eb2, Er2]=E*[Y2', Eb2', Er2']+F where E is a scaling matrix and F is an offset vector.

Combining all the equations results in the following:

[Y2, Eb2, Er2]=E*D*C*([Y1, Cb1, Cr1]-B)+F=E*D*C*[Y1, Cb1, Cr1]-E*D*C*B+F

This shows that the conversion cannot be done as a pure matrix multiplication, i.e. a single matrix multiply.

Question 2 - BT601

I want to clarify that color matrix cofficients for SMTP170M are the same as BT470-2 System B, G which are also BT601.

The BT470_2 option is really BT470-2 system M.

Also I am in the camp that if the colorspace of the DVD is not given then the default is BT709 and not BT601.

Question 3 - Speed

I could add threads to easily speed this up but since I use the multithreaded version of AviSynth, I didn't think it was necessary.

Question 4 - Gamma Correction

Since I am assuming that the linear [R', G', B'] values all correspond to the same real world color gamma correction is necessary. But I have not done the the complete transformations to verify this.

I am not trying to offend anyone and enjoy the debates.

[IanB]

... 601 ... alias

Alias as in alternate name. Rec.601 is the term generally used in Avisynth, hence the nudge for an alias.

ColorMatrix implements in your terms as:-

[Y2, Cb2, Cr2] = F + ((E*D*C) * ([Y1, Cb1, Cr1]-B))
where (E*D*C) is precalculated for speed,
and B = F = [16, 128, 128]

Assuming your coefficients end up being the same, the 2 filters should be identical (okay, ColorMatrix doesn't do full range).


And from my opening post, I had the presumption you were investigating the effect that the signals we regularly process within Avisynth are gamma compressed. This does cause measurable, and sometime even visible effects :- to wit my recent combat with the antialiaser code in the text drawing routines. That will teach me to presume evidence not in fact.

[tritical]

I looked at your code, and

BZColorspace(InputFullrange=false,OutputFullRange=false)

is exactly equivalent to:

Colormatrix(clamp=false)

aside from rounding differences. Actually, the only difference between your filter and colormatrix is when you set either InputFullrange=true or OutputFullRange=true or both in BZColorspace. The difference is that Colormatrix always subtracts 16 from the luma before doing the transformation, and adds 16 to the luma at the end. With InputFullrange=true BZColorspace doesn't subtract 16 at the beginning, and with OutputFullRange=true it doesn't add 16 back at the end. To test my theory, I modified Colormatrix to do this and as expected I got exactly the same output as BZColorspace aside from rounding errors. The question then is: is not subtracting 16 at the start and not adding it back at the end ever correct?.. and: is doing one and not the other (such as subtracting 16 at the beginning and not adding 16 at the end) ever correct? Such functionality could easily be added to colormatrix.

EDIT:
Actually, there is another difference that occurs when InputFullrange/OutputFullRange are given different values. You end up scaling the coefficients on one end of the transformation differently than on the other. Colormatrix never scales the coefficients because it always assumes the same scaling at both ends, which ends up canceling out (and is therefore unneeded).

[josey_wells]

You guys are tough.

I originally started playing with AviSynth back in November 07 and was using ColorMatrix and levels="TV->PC" and got unsatisfactory results with the people being a little orange. Instead of digging into ColorMatrix I decided I would write a function which can perform both functions together and be easily expandable. I had selfish reasons for doing this being that I am also in the process of writing routines to process solar eclipse pictures and needed to fully understand the color flow process. Also I have been programming numerical/data processing routines in C/C++ for 18 years.

I wanted all my converted movies to be BT709 full range.

I will first go into the matrix derivations and then into talk about the scaling.

using MatLab the code to derive the SMTP170 matrix is as follows:

C170MR = 0.299;
C170MB = 0.114;
C170MG = 0.587;
C170MBScale = .5/(1-C170MB);
C170MRScale = .5/(1-C170MR);
C170M = [C170MR C170MG C170MB
-C170MR*C170MBScale -C170MG*C170MBScale (1-C170MB)*C170MBScale
(1-C170MR)*C170MRScale -C170MG*C170MRScale -C170MB*C170MRScale];

What I want to show here is that the luminace coefficients are the only important ones. These three constants define the entire matrix. In fact only 2 are necessary since R+G+B=1 means the third can be derived from the other two. With that being said all the color conversion matrixices become as below. There is some rounding in the coefficients for the second and third row of the matrices below but I carry the exact matrices in follow on calculations. Also this shows that there is some rounding in the Table 6-9 also for the PB and PR rows.

C170M=
0.2990 0.5870 0.1140
-0.1687 -0.3313 0.5000
0.5000 -0.4187 -0.0813

BT709=
0.2125 0.7154 0.0721
-0.1145 -0.3855 0.5000
0.5000 -0.4542 -0.0458

C240M=
0.2120 0.7010 0.0870
-0.1161 -0.3839 0.5000
0.5000 -0.4448 -0.0552

FCC=
0.3000 0.5900 0.1100
-0.1685 -0.3315 0.5000
0.5000 -0.4214 -0.0786

To convert from one colorspace to another I use the following MATLAB code:

C170MToBT709=BT709*(C170M^-1);
int32(ScaleFactor*C170MToBT709)

where I am using a ScaleFactor=16384. All the code uses 32 bit integer multiples and almost the full range of the 32 bit range to try to avoid any round off errors. So the conversion matrices are as follows once the data has been converted to [Ey', Eb', Er'] the linear luminance and red and blue difference channels.

C170MToBT709=
16384 -1940 -3489
0 16690 1880
0 1232 16800

C240MToBT709=
16384 -499 -100
0 16390 54
0 317 16458

FCCToBT709=
16384 -1787 -3470
0 16678 1870
0 1135 16766

BT709ToC170M=
16384 1667 3216
0 16218 -1815
0 -1189 16112

C240MToC170M=
16384 1231 3137
0 16189 -1770
0 -878 16180

FCCToC170M=
16384 133 12
0 16383 -7
0 -95 16352

BT709ToC240M=
16384 497 97
0 16379 -53
0 -315 16312

C170MToC240M=
16384 -1427 -3332
0 16681 1825
0 905 16689

FCCToC240M=
16384 -1274 -3313
0 16669 1814
0 809 16657

BT709ToFCC=
16384 1537 3219
0 16218 -1808
0 -1098 16133

C170MToFCC=
16384 -133 -12
0 16385 7
0 95 16416

C240MToFCC=
16384 1100 3139
0 16189 -1764
0 -786 16202

These can be compared to colormatrix to see if they are the same.

Range scaling:

When input or output full range is selected the following conversion is used which come from the JPEG standard and utilizies the entire [0, 255] range for luminance and chromanince where EY' range is [0, 1] and Eb' & Er' range is [-0.5, 0.5].

Y=256*E'Y
Cb=256*Eb'+128
Cr=256*Er'+128

When input or output not full range is selected the following conversion is used

Y=(219*Ey')+16
Cb=(224*Eb')+128
Cr=(224*Er')+128

This shows the offsets and scale factors are different for the two processes. Which is why I have an intermediate step to convert from input Y, Cb, Cr to linear convert color transformation and convert back to output Y, Cb, Cr.

This transformation does extend the color range for the file for further processing but it is not a gamma transformation.

Gamma Compression-

If by gamma compression you mean the banding that results in some of the processing after H.264 conversion then I think that this is unavoidable. I had to use the deband function in FFDSHOW no matter how good the settings were for the H.264 compression.

I know that this color conversion function will not introduce any banding issues that were not originally in the source since no round off/overflow/underflow errors are introduced.

I use the following script for doing DVD processing:

SetMTMode(2,0)

FileName="VTS_01_1.d2v"

#decode and deblock
MPEG2Source(FileName, idct=5, cpu=4, moderate_h=20, moderate_v=20, upconv=1)

#deinterlace, decomb, and decimate
BryanDDD(bInterlaced=false)

#crop and anamorphic resize
crop(0, 0, -0, -0) #left, top, -right, -bottom
Spline36Resize(((Width+15)/16)*16, ((Height+15)/16)*16)

#change colorspace(BT470_2, BT709, FCC, SMTP170M, SMTP240M)
BZColorspace(SourceColorimetry="SMTP170M", DestinationColorimetry="BT709", InputFullRange=false, OutputFullRange=true)

#motion denoise
BryanDenoise(iSearch=2)

#trim for test
#trim(1, 1000)

#deinterlace, decomb, and decimate
function BryanDDD(clip c, bool "bInterlaced")
{
bInterlaced=default(bInterlaced, false) #default not interlaced

processed=(binterlaced==true) ? Eval("""
#interlaced
interp=c.nnedi()
interlaced=c.tdeint(edeint=interp, emask=c.TMM(), slow=2, denoise=true)
return c.tfm(mode=5, pp=7, slow=2, clip2=interlaced).tdecimate(hybrid=1, denoise=true)
""") : Eval("""
#non interlaced
return c.tfm(mode=5, pp=7, slow=2).tdecimate(hybrid=1, denoise=true)
""")
return processed
}

#motion denoise
function BryanDenoise(clip c, int "iTHSAD", int "iBlksize", int "iOverlap", int "iPel", bool "bChroma", bool "bTrueMotion", int "iSearch", int "iSharp", int "iDct", int "iIdx")
{
iBlksize =default(iBlksize, 8) #default block size
iOverlap =default(iOverlap, iBlksize/2) #default 1/2 block size
iTHSAD =default(iTHSAD, 400) #default
iPel =default(iPel, 4) #quarter pixel
bChroma =default(bChroma, true) #use chroma
bTrueMotion=default(bTrueMotion, true) #use true motion
iSearch =default(iSearch, 3) #exhaustive search
iSharp =default(iSharp, 2) #6 tap sharp filter
iDct =default(iDct, 1) #use DCT for SAD
iIdx =default(iIdx, 1) #index
backward_vec2=c.MVAnalyse(isb=true, delta=2, pel=iPel, blksize=iBlksize, overlap=iOverlap, chroma=bChroma, truemotion=bTrueMotion, search=iSearch, sharp=iSharp, dct=iDct, idx=iIdx)
backward_vec1=c.MVAnalyse(isb=true, delta=1, pel=iPel, blksize=iBlksize, overlap=iOverlap, chroma=bChroma, truemotion=bTrueMotion, search=iSearch, sharp=iSharp, dct=iDct, idx=iIdx)
forward_vec1 =c.MVAnalyse(isb=false, delta=1, pel=iPel, blksize=iBlksize, overlap=iOverlap, chroma=bChroma, truemotion=bTrueMotion, search=iSearch, sharp=iSharp, dct=iDct, idx=iIdx)
forward_vec2 =c.MVAnalyse(isb=false, delta=2, pel=iPel, blksize=iBlksize, overlap=iOverlap, chroma=bChroma, truemotion=bTrueMotion, search=iSearch, sharp=iSharp, dct=iDct, idx=iIdx)
return c.MVDegrain2(backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=iTHSAD, idx=iIdx)
}

When viewed in VirtualDub it doesn't look quite correct since VirtualDub uses BT601 and non full range values. But when viewed after processing in FFSHOW everything looks correct.

I have also taken all the source code for all the filters I use and corrected scope/conformance issues and fixed uncessary temporaries in loops, i.e. i++ where ++i can be used. The only source code I don't have is NNEDI.

Thanks tritical for writing a lot of thse functions.

If colormatrix can be modified for the general user environment then I agree that that is the way to go.

Bryan Z.

Well, I have a closet to go build now and will check back later.

[Fizick]

josey_wells,
thanks for your contribution!
IMO, full range is important, in particular for non-DVD sources too, such as DV and analog capture.
I do not like "forced" range clipping in Avisynth functions
(i like have an option!)
I vote for ColorMatrix have full range too.

Are you really corrected scope/conformance issues and fixed uncessary temporaries in loops of MVTools filters?
Whats the gain?

[josey_wells]

Fizick,

Thanks for the response. I love MVTools and think that this is one of the greatest filters for improving compressibility and cleaning up all DVD sources.

I have attached the updated source code for you to look at and try. Unfortunately it is based on 1.8.5.1, but I don't think that many files have changed. I think I achieved about a 10% increase. I didn't modify all the files since I was only interested in the MVAnalyse/MVDegrain functions at the moment. If I have some more time I would like to come back and try to achieve some more speed improvements.

In the mean time give my color filter a shot until ColorMatrix is modified.

[tritical]

Turns out that I was slightly incorrect in my original statements.

Colormatrix(clamp=false)

is actually equivalent to both of these cases (aside from rounding):

BZColorspace(InputFullrange=false,OutputFullRange=false)
BZColorspace(InputFullrange=true,OutputFullRange=true)

With InputFullrange=true/OutputFullRange=true the subtraction/addition of 16 cancel out the same way the scaling does. Therefore, the interesting part is the ability to set them to different values (i.e. start with limited range YUV and end up with full range YUV, and vice versa). This is actually combining two separate processes: colorimetry conversion and range expansion (or contraction). josey_wells, you seem to always be using the combination of InputFullrange=false/OutputFullrange=true. Is that because using that in combination with ffdshow's full range RGB conversion looks better than using OutputFullrange=false with ffdshow's standard RGB conversion?

Your points about the coefficient calculations make sense. I'll change colormatrix to calculate the PB/PR rows from the luminance coefficients in the future. Although there are some differences between your luminance coefficients for BT.709 and the ones in colormatrix:

+0.7152, +0.0722, +0.2126 (Colormatrix)

vs

+0.7154, +0.0721, +0.2125 (BZColorspace)

I think the difference between these two sets was discussed a while ago when the new conversion functions were added to ffdshow. IIRC, the ones in Colormatrix are the more recent (Wilbert can you comment?)?

[IanB]

From Outdated Rec.709 coefficients

In the first draft of the spec, they use: kr = 0.2125, kg = 0.7154, kb = 0.0721.

In the final spec, they use: kr = 0.2126, kg = 0.7152, kb = 0.0722.


@Tritical,

Yes building the fuller expressions directly from the Kr and Kb constants tends to avoid errors, I have started using this model to good effect when I fix such problems inside Avisynth.

[josey_wells]

Tritical,

First I started out with colormatrix encoded video looking correct on some computers and not correct on others due to different video cards interpreting YV12 as full range or not. So I tried TV->PC in combination with Colormatrix and got unsatisfactory results. That is when I decided I wanted to output full range BT709 and use FFDShow. I figure that all video will be full range one day. Second, It gives a larger colorspace for further filters to work in.

I have also found updated ISO/IEC 14496-2:2004 (MPEG-4 standard) which confirms several things:

BT709 - The new coefficients are indeed 0.7152, 0.0722, and 0.02126

A video_range parameter was added where for 8 bit video the source is encoded with the following:

Video_Range=0
Luminance and Chromaniance [16, 235] & [16, 240] respectively

Video_Range=1
Luminace and Chrominance [0, 255]

With the scaling and offset parameters the same as my previous emails.

This would be a good reason to combine the color conversion and input/output range functions into a single color function.

This begs the following questions:
1. Do these updated coefficients also apply to the MPEG-2 standard?
2. Does the updated MPEG-2 standard also have a video range paramater as part of the sequence extension?
3. Do the DGIndex programs for DVD and AVC take into account this video_range and report it for MPEG-4 and MPEG-2 if it applies in an updated document?

I have also attached modified source code of some of your filters if you would like to take a look.

[tritical]

Thanks, I will take a look at those sources.

Unfortunately, I don't know the answers for your first two questions. I do know that DGIndex/DGDecode don't parse a video_range parameter or anything that signals that information. Almost no one requests rgb from DGDecode, in which case it just passes the yuv values through untouched, so it wouldn't really matter regardless.

[josey_wells]

I agree that hardly no one uses the RGB32 function of DGIndex. However, the video_range parameter would be important to know in the log file or GUI because it would impact whether one would use Input Full Range in BZColorspace or ColorMatrix. I will post a question in the DGIndex thread.

Maybe someone else(Wilbert?) can post in on the MPEG 4 BT709 coefficients and whether they apply to the MPEG 2 standard also. If so then I need to update my fucntion. If not then technically a conversion between MPEG2 to MPEG4 BT709 would need to be done for correct display in FFDSHOW and I would need to add another conversion.

Is AviSynth V3.0 going to be based on 16/14/12/10 bit internal video bit depth vs the 8 it currently uses? I only ask since I noticed the new standard allows for higher bit depths than 8. Although increasing bit depth internally will not help banding issues for sources that are already encoded as 8 bit.