yCMS - Color Management System

[yesgrey]

Introduction:
yCMS is a Color Management System (CMS) that uses 3-dimensional lookup tables ("3D LUTs") to perform the conversion between the input and output color representations. You can read a little more about CMS here (http://en.wikipedia.org/wiki/Color_management).

The idea of using 3D LUTs was due to the intensive computing that needs to be performed when converting video data between different color spaces. When using 3D LUTs all computing is performed offline during its creation. The computed 3D LUTs (three, one for each component) are then written to a file which can later be used, when playing videos or viewing photos, simply by mapping the input data through the 3D LUTs. This way, the conversion can be performed at runtime using lower computational power and leaving more CPU available for other tasks. Since all the computing is performed offline its precision can be increased by using 64bit floating point (per component) through out all the conversion chain.

Features:
- Color space conversion
- Gamut correction
- Chromatic adaptation
- Gamma correction
- HD, SD, xvYCC, sYCC & Computer video standards
- Handles R'G'B', RGB, Y'CbCr and YCbCr video data
- 6 bit (64x64x64), 7 bit (128x128x128) or 8 bit (256x256x256) input bit depth 3D LUTs
- 8 or 16 bit output bit depth 3D LUTs
- Full 64 bit FP processing chain
- Both x86 and x64 versions

Requisites:
- Windows XP or newer
- x86 or x64 CPU with SSE or above

Changelog:

v1.12 - 2012/11/07
- New command Denoise_Black_Level to denoise black level range
- Changed Input/Output_Primaries range to allow negative numbers in R,G,B


Download:
http://yesgrey.com/ycms.html

[yesgrey]

How to use yCMS:
(1)Read the detailed instructions inside the Manual and create 3DLUT file(s) that fit your needs.
(2)Use your 3DLUT file(s) with madVR, t3dlut or rgb3dlut, the software options which support 3DLUT files. Below are instructions on how to use them.

How to use madVR v0.66+:
See instructions on this (http://forum.doom9.org/showpost.php?p=1510363&postcount=637) post.

How to use madVR up to v0.61:
madVR is a DirectShow video renderer created by madshi. If you don't know what madVR is or how to set it up and use it go here (http://forum.doom9.org/showthread.php?t=146228).

madVR supports four different 3DLUT files. Two for watching HD media (PC and video levels) and two for watching SD media (PC and video levels). The selection between HD and SD files is automatic, according to media resolution. For "height > 576" or "width > 1024" it uses the HD files, otherwise it uses the SD files. You can manually choose between PC and video levels in madVR's settings dialog.

You should create a 3DLUT file for each of the available options. To make madVR use them, just copy them into madVR's installation directory (where madVR.ax is contained) and name them properly. The file names expected by madVR are:
HD media with PC levels: "hd - pc.3dlut"
HD media with video levels: "hd - video.3dlut"
SD media with PC levels: "sd - pc.3dlut"
SD media with video levels: "sd - video.3dlut"

After creating, copying and renaming the 3DLUT files simply start your media player and check the "Use 3DLUT" checkbox in madVR's settings dialog.

note: madVR only supports Y'CbCr 8 bit input / R'G'B' 16 bit output 3DLUT files. The output is dithered to screen bit depth by madVR.


How to use t3dlut/rgb3dlut:
t3dlut/rgb3dlut are Avisynth plugins created by tritical. If you don't know what they are or how to set them up and use them go here (http://bengal.missouri.edu/~kes25c). rgb3dlut is included in ddcc.

You can use t3dlut/rgb3dlut for watching your media or for processing your video files using Avisynth. t3dlut supersedes rgb3dlut by allowing any combination of RGB and YCbCr input/output, while rgb3dlut only supports RGB output. rgb3dlut is faster because it has some assembly optimizations, so you will have to decide which would fit better your needs. Using t3dlut with a 3DLUT file should give you slightly more accurate colorspace conversions than using ColorMatrix.

Due to the high number of possible configurations no example is offered at this time. I will add examples when requested.

note: t3dlut supports 8 bit input / 8 or 16 bit output 3DLUT files, and all of the video formats combinations available in yCMS. When using 16 bit output 3DLUT files, the output is automatically dithered to 8 bit by t3dlut.

Example using rgb3dlut with ffdshow: Instructions here (http://forum.doom9.org/showthread.php?p=1426031#post1426031).

[yesgrey]

Calibration example:
Here are the results after calibrating my projector (JVC M15) using yCMS and madVR.

I started by measuring my projector primaries and white point at 75 IRE. For that I ran a video with 75 IRE R,G,B color fields, unchecking the 3DLUT option in madVR and selecting PC levels (my projector only accepts this):
Red: x = 0.6662; y = 0.3280
Green: x = 0.2654; y = 0.6818
Blue: x = 0.1425; y = 0.0551
White: x = 0.3109; y = 0.3324

Then, I created a yCMS configuration file for it:
# Set input format
Input_Format HD YCbCr 8

# Set output format
Output_Format HD RGB_PC 16

# Gamut correction
Gamut_Measurements 0.6662 0.3280 0.2654 0.6818 0.1425 0.0551 0.3109 0.3324

Here are the results only with Gamut correction:
11197
In the picture, the HD color gamut (BT.709) is the black triangle, and my projector's color gamut is the white triangle.
After using gamut correction the results are slightly better. The green primary is nearer to the HD green primary, but the red and blue primaries were not corrected at all.

I then measured my display's grayscale response (gamma curve):
11196
As you can see, it's not very nice...:(

I tried again, but this time including gamma correction by using my display's grayscale response:
# Set input format
Input_Format HD YCbCr 8

# Set output format
Output_Format HD RGB_PC 16

# Gamut correction
Gamut_Measurements 0.6662 0.3280 0.2654 0.6818 0.1425 0.0551 0.3109 0.3324

# Gamma correction
Grayscale_Measurements
30.0 6.29
40.0 11.82
50.0 20.12
60.0 30.89
70.0 44.31
80.0 62.52
90.0 83.82
100.0 100.00

Here are the results with both Gamut and Gamma corrections:
11198
As you can see in the picture, all primaries are in the correct positions now, over the HD primaries.:cool:

[Thunderbolt8]

will your version of the 3d lut be automatically included in madvr? if not, is there any reason why I should or shouldnt get it when I just want to watch blu-rays via my PC & hdmi connection to a tv? are there any advantages or disadvantages compared to when I don't use your version?

[madshi]

Congrats to yesgrey for (finally :)) releasing yCMS!

will your version of the 3d lut be automatically included in madvr?
Our plan is that a future madVR version will no longer include cr3dlut/yCMS by default, but instead madVR will automatically download and "install" yCMS, when you ask madVR to use 3dlut files.

if not, is there any reason why I should or shouldnt get it when I just want to watch blu-rays via my PC & hdmi connection to a tv? are there any advantages or disadvantages compared to when I don't use your version?
If you use 3dlut files with madVR, then you should get rid of cr3dlut and use yCMS instead. That's very simple.

Now if you're asking whether you should or should not use 3dlut files, that's a more complicated question. Let me give you a list of advantages and disadvantages of using 3dlut files vs. not using them in madVR:

advantages:
(1) yCMS can make your display show a perfect gamma response
(2) yCMS can correct the gamut/primaries of your display (color management system)
(3) when watching SD content, yCMS can convert the SD gamut/primaries to HD
(4) madVR needs to do less math (faster performance?)

disadvantages:
(1) ever so slightly higher loading times
(2) higher memory usage on your GPU
(3) madVR needs to do more GPU memory accesses (slower performance?)

In order to use advantages (1) and (2), you will need to have a meter (e.g. Spyder3 or Chroma5 etc) and a calibration software (e.g. HCFR or Calman).

[cyberbeing]

What exactly is yCMS doing when you input values for Grayscale_Measurements?

When I use it, it greatly brightens up the picture and reduces overall contrast considerably. Blacks turn gray. Whites get slightly crushed. It's like a light gray film gets placed over the video.

http://img218.imageshack.us/img218/6271/nolutpc.th.png (http://img218.imageshack.us/img218/6271/nolutpc.png)http://img97.imageshack.us/img97/2900/hdpc.th.png (http://img97.imageshack.us/img97/2900/hdpc.png)http://img80.imageshack.us/img80/7955/hdpcirew.th.png (http://img80.imageshack.us/img80/7955/hdpcirew.png)
From left to right.
| madVR TV->PC | yCMS LUT TV->PC | yCMS LUT with IRE TV->PC |
Funnily enough, yCMS LUT /w IRE 16-235 -> 0-255 makes what should be a 0 black, brighter than yCMS LUT 0-255 -> 0-255 (seen below)...

http://img297.imageshack.us/img297/2169/noluttv.th.png (http://img297.imageshack.us/img297/2169/noluttv.png)http://img245.imageshack.us/img245/7381/hdtv.th.png (http://img245.imageshack.us/img245/7381/hdtv.png)http://img375.imageshack.us/img375/705/hdtvire.th.png (http://img375.imageshack.us/img375/705/hdtvire.png)
From left to right.
| madVR PC->PC | yCMS LUT PC->PC | yCMS LUT with IRE PC->PC |

# Set input format
Input_Format HD YCbCr 8

# Set output format
Output_Format HD RGB_PC 16

# Grayscale IRE Measurements
Grayscale_Measurements
10 0.486085 0.472363 0.497613
20 2.855507 2.751755 2.748106
30 7.040504 6.968139 7.044012
40 13.197106 13.020103 13.015621
50 21.422743 21.428576 21.564960
60 31.921806 31.852612 31.998531
70 45.422878 45.317829 45.406380
80 60.435123 60.174709 60.314659
90 78.802399 78.381996 78.620575
100 97.811592 98.124657 98.652870

# Display Gamut Measurements
Gamut_Measurements 0.626 0.343 0.294 0.607 0.149 0.078 0.3127 0.3290

The above is what I'm using with yCMS. How can I use the new IRE function and keep my Blacks, Black?

[FoLLgoTT]

@yesgrey
Good work! :)

Would it be possible to input multiple gamut measurements? Right now we measure primaries/secondaries at 75% or 100%. Like calibrating greyscale there should be the possibility to input primary/secondary measurements for different IRE levels (e.g. in 10 IRE steps). I think this would higher the accuracy of calibration.

Another question: in which color space do you interpolate the colors?

[FoLLgoTT]

Ah, another question: can luma (Y) of primaries/secondaries be corrected by yCMS? I can only see x and y in the manual. But a full gamut correction also includes Y.

[pankov]

yesgrey,
I've just looked over the madVR thread and I can confirm that the selection of SD or HD luts is done not only based on the height but also on the width of the video - HD = width > 1024 or height > 576.
It will be nice if you correct the definition in your second post

[janos666]

I have the same problem what cyberbeing described.

Here is my input file built with copy-pasteing HCFR 2.1 measurements:

# Source video format
Input_Format HD YCbCr 8

#3DLUT output format
Output_Format HD RGB_PC 16

# Gamut correction - measured native x;y coordinates
Gamut_Measurements 0.657073 0.329413 0.231911 0.690324 0.140940 0.090552 0.311880 0.329555

#Gamma correction - measured grayscale luminance Y;Z values
Grayscale_Measurements
10 1.469370 1.549970 1.633230
20 4.510128 4.779058 5.217576
30 9.946090 10.541579 11.273838
40 18.012747 18.977659 20.425848
50 28.926823 30.672146 32.918686
60 42.545696 44.938530 48.570366
70 60.392231 63.847328 69.020111
80 80.578819 85.209976 92.178696
90 105.731606 111.789406 121.205627
100 134.328751 141.941650 154.436478

(I know, it looks strange.)

The result is a brightened image with low contrast feeling.

I also tried to use the videoStandard preset (I calibrated my monitor with sRGB curve):
Output_Transfer_Function 0 0
Output_Transfer_Function 1 0
It is much better than it was with cr3dlut but it is still noisy.
I could not see it from normal viewing distance but I had the "something isn't right" feeling, so I lean closer to the display and it is clearly visible. (There is a lot of noise on dark areas.)

I made some screenshots with Output_Transfer_Function 0 0 enabled/disabled:
Sometimes I can see more geometry details with slightly less color accuracy: 01 (http://img12.tar.hu/janos666/img/76125895.png) - 02 (http://img12.tar.hu/janos666/img/76125896.png) ; 11 (http://img12.tar.hu/janos666/img/76125728.png) - 12 (http://img12.tar.hu/janos666/img/76125893.png)
But sometimes it causes some strange "sour cream" look: 21 (http://img12.tar.hu/janos666/img/76126042.png) - 22 (http://img12.tar.hu/janos666/img/76126043.png) ; 31 (http://img12.tar.hu/janos666/img/76128065.png) - 32 (http://img12.tar.hu/janos666/img/76128066.png)

I think it works well with SD sources but still useless with HD materials. The image is too bright and there is still too much quality loss. (I didn't noticed this with PAL DVDs but that was only a quick test, I do not watch them anymore...) After all, we may need that 10 bit output (and I need a new display then...).

[cyberlolo]

I only have one question: why do you offer two different options for setting the chromaticity coordinates of the primaries and the white point? I mean, we can set it with the option:

Gamut_Measurements redX redY greenX greenY blueX blueY whiteX whiteY

and also, we can set it with:

Output_Primaries redX redY greenX greenY blueX blueY whiteX whiteY

What's the difference(s) between them?

Thanks for your great work. Now I can get rid of ffdshow's levels filter. Finally!!!

:thanks:

[madshi]

I'll let yesgrey answer most of the questions, because he can do that much better than me, but I know the answer to some questions, at least.

It is much better than it was with cr3dlut but it is still noisy.
I could not see it from normal viewing distance but I had the "something isn't right" feeling, so I lean closer to the display and it is clearly visible. (There is a lot of noise on dark areas.)
Can you post a screenshot of a frame where you see so much noise in dark areas? Please also post the same frame without yCMS processing, so that we can compare. Thanks.

I only have one question: why do you offer two different options for setting the chromaticity coordinates of the primaries and the white point?
The commands serve different purposes. If you read the manual from top to bottom, the "mandatory" commands are explained first, then the "optional" commands, then the "calibration" commands. That's really all you should care about.

The remaining commands under the section "standards definition" are mainly meant to be used by people who want to convert a video from one standard/format to another. E.g. if you want to reencode video by upscaling SD to HD, you should also modify the primaries. The "Input_Primaries" should then be PAL or NTSC and the "Output_Primaries" should be HD. When reencoding video, you do NOT want to measure your display and use calibration commands. You want to define standard conversion parameters instead.

"Gamut_Measurements" is for display calibration during media playback. This command only supports measured coordinates.

"Output_Primaries" is for special purposes like e.g. reencoding video. This command mainly supports shortcuts to the well knowns standards (e.g. "0" for "HD"). But it also supports custom coordinates, in case a reencoding shall be done to a custom format, not directly supported by yCMS through a shortcut.

[cyberlolo]

The commands serve different purposes. If you read the manual from top to bottom, the "mandatory" commands are explained first, then the "optional" commands, then the "calibration" commands. That's really all you should care about.

The remaining commands under the section "standards definition" are mainly meant to be used by people who want to convert a video from one standard/format to another. E.g. if you want to reencode video by upscaling SD to HD, you should also modify the primaries. The "Input_Primaries" should then be PAL or NTSC and the "Output_Primaries" should be HD. When reencoding video, you do NOT want to measure your display and use calibration commands. You want to define standard conversion parameters instead.

"Gamut_Measurements" is for display calibration during media playback. This command only supports measured coordinates.

"Output_Primaries" is for special purposes like e.g. reencoding video. This command mainly supports shortcuts to the well knowns standards (e.g. "0" for "HD"). But it also supports custom coordinates, in case a reencoding shall be done to a custom format, not directly supported by yCMS through a shortcut.

So, if I understood it correctly: for HD sources (input and output formats will be the same), I can use the "Gamut_Measurements" command to set the measured coordinates of my display. But for SD sources, if I want them to be reencoded to HD (which I do), it's better not using that command, and set input_format to NTSC/PAL and output_format to HD. Is this right?

[cyberbeing]

#Gamma correction - measured grayscale luminance Y;Z values


According to the manual it should be | Red Luminance | Blue Luminance | Green Luminance | in the three IRE value columns. You shouldn't be putting X or Z values there unless I'm misunderstanding something.

Grayscale_Measurements
10 1.469370 1.549970 1.633230
From your example, 1.469370 should be Red Luminance, 1.549970 should be Green Luminance, and 1.633230 should be Blue Luminance. If you have a good display, all three should have near identical values with very little deviance.

Optionally you can also just list a single value (only using the first column) for overall Gray Luminance at a given IRE.

@yesgrey
Could the overly bright problem be because you aren't applying any sort of Black Point Compensation with yCMS?

I've also noticed yCMS only allows for 10 IRE values, but it would be nice if you could make it 11 (increments of 10 + zero) or even 21 (increments of 5 + zero), so you could have a value for 0 IRE to represent your Black Point.

[yesgrey]

Congrats to yesgrey for (finally :)) releasing yCMS!
Thanks.:)

are there any advantages or disadvantages compared to when I don't use your version?

advantages:
(1) yCMS can make your display show a perfect gamma response
(2) yCMS can correct the gamut/primaries of your display (color management system)
(3) when watching SD content, yCMS can convert the SD gamut/primaries to HD
(4) madVR needs to do less math (faster performance?)

I would like to correct (2):
yCMS can overcome the incorrect gamut/primaries of your display and let you watch the colors as accurately as possible. If your display's gamut is wider than the source video gamut (if it contains entirely the source video gamut) all colors are shown accuratelly, otherwise the colors that lay outside the display's gamut would be clipped.

note: madshi answer was not wrong in a user's point of view, because it appears that the display's gamut/primaries are corrected by yCMS, but in reality they aren't. That could be done only within the display, and yCMS is external to it.

To (3) I would like to add:
(3) when watching SD content, yCMS can convert the SD gamut/primaries to HD, which should be done if you have a HD display.

[janos666]

According to the manual it should be | Red Luminance | Blue Luminance | Green Luminance | in the three IRE value columns. You shouldn't be putting X or Z values there unless I'm misunderstanding something.

Yes, it was my mistake. :rolleyes: I wrote that note because I wasn't sure when I have tried to use this new software at the first time.
I already figured it out that I should use the RGB values from HCFR. And yes, they are nearly identical values and the 100% IRE is close to my calibration target luminance. :cool:

But the problem is still exist with the correct matrix. The image is too bright and it has low contrast...

And I would ask the same thing: I want to add 11 IREs...


Can you post a screenshot of a frame where you see so much noise in dark areas? Please also post the same frame without yCMS processing, so that we can compare. Thanks.

Yes. I will play more with the settings first to make sure it is not my fault (like poor movie quality, ect.) and I will make some screenshots.
But you can take a look at my actual screenshots above. They meant to represent the loss in color quality but you may see some strange on dark areas.
The main problem there that the overall image looks too bright and this brightening results in washed-out colors. (Ok, it is not a very serious mess -as it was with cr3dlut- but I can easily notice it on a S-PVA display...)

PS: I almost forgot to say thanks for your work. Aside from my continuous "problem reports" I appraise it. So, may be it isn't perfect but better than anything else which I can use now...

[yesgrey]

What exactly is yCMS doing when you input values for Grayscale_Measurements?
It removes the studio gamma encoding and then encodes using the inverse function of the display.

The above is what I'm using with yCMS. How can I use the new IRE function and keep my Blacks, Black?
I have the same problem what cyberbeing described.
Investigating...

Good work! :)
Would it be possible to input multiple gamut measurements?
Can luma (Y) of primaries/secondaries be corrected by yCMS?

Thanks.:)
Both are on the ToDo list.
Though, don't forget that in the second a perfect Y might run against a perfect xy, so it might ends up as an optional setting.

in which color space do you interpolate the colors?
Sorry, but i didn't understand your question. What did you mean with "interpolate colors"?

It will be nice if you correct the definition in your second post
Done. Thanks.

It is much better than it was with cr3dlut but it is still noisy.
After all, we may need that 10 bit output (and I need a new display then...).
Yes, I know it's not perfect yet, but I preferred to release yCMS now and keep working on it. It should improve in future versions. And no, we don't need the 10 bit output...

I only have one question: why do you offer two different options for setting the chromaticity coordinates of the primaries and the white point?

What's the difference(s) between them?

madshi answered it really well. I'll just add that the Gamut_Measurements command was created also having in mind the gamut correction for multiple saturations (not yet supported), and that would not fit in the Output_Primaries command.
With only one saturation allowed there is an overlapping between the two commands, but we decided to still introduce this command now because it's more user friendly than the Output_Primaries. A big part of the users would not need (or dare :D) to go past the Warning, so I must say that you are one of the braves...;)

So, if I understood it correctly: for HD sources (input and output formats will be the same), I can use the "Gamut_Measurements" command to set the measured coordinates of my display. But for SD sources, if I want them to be reencoded to HD (which I do), it's better not using that command, and set input_format to NTSC/PAL and output_format to HD. Is this right?
No.
The "Gamut_Measurements" command should always be used when the display's coordinates are measured, no matter what the source is. The source should be reflected only in the Input commands, and in none of the output commands. The commands relative to measures are all output commands, because you are measuring the output (display).

[janos666]

Can you post a screenshot of a frame where you see so much noise in dark areas.

I made some screenshots from different movies and different type of scenes to show all my concerns.
I made a 3dlut with these settings:
Input_Format HD YCbCr 8
Output_Format HD RGB_PC 16
Output_Transfer_Function 0 0
And I used madVR with these settings:
[scaling settings]
luma upsampler=Spline64 luma downsampler=Spline64 chroma resampler=Spline64
[output settings]
video levels=0 use 3dlut=1
[trade quality for performance]
don't use dithering=0 use 10bit luma buffer=0 use 10bit chroma buffer=0 disable anti-tearing fix=0

I changed only this value through the CP:
use 3dlut=0

I take these from a re-encoded 1080p source (x264_L4.1 @ 7971 kbps) to represent how this brightening boost the perception of the noise: 01 (http://img12.tar.hu/janos666/img/76143352.png) - 02 (http://img12.tar.hu/janos666/img/76143353.png) ; 11 (http://img12.tar.hu/janos666/img/76143354.png) - 12 (http://img12.tar.hu/janos666/img/76143355.png)

I take these from a real Blu-Ray source (AVC_L4.1 @ 39.2 Mbps) to represent how this option kills the color detail: 21 (http://img12.tar.hu/janos666/img/76143634.png) - 22 (http://img12.tar.hu/janos666/img/76143635.png)

I take these from a re-encoded 1080p source (x264_L4.1 @ 8706 kbps) to show that image is sometimes too bright: 31 (http://img12.tar.hu/janos666/img/76143356.png) - 32 (http://img12.tar.hu/janos666/img/76143357.png)

And these comes from the Blu-Ray again to show them all (too bright scene with noisy dark spots and washed-out colors...): 41 (http://img12.tar.hu/janos666/img/76143497.png) - 42 (http://img12.tar.hu/janos666/img/76143498.png)

And here is the result of these new settings: 51 (http://img12.tar.hu/janos666/img/76144446.png) - 52 (http://img12.tar.hu/janos666/img/76144447.png)
Grayscale_Measurements
10 1.564805 1.551411 1.491997
20 4.668003 4.810841 4.791349
30 10.406371 10.604137 10.320132
40 19.016638 18.991844 18.722298
50 30.179890 30.871260 30.149662
60 44.580005 45.085168 44.541965
70 63.151944 64.109925 63.294323
80 84.184549 85.582548 84.539725
90 110.365791 112.272175 111.200465
100 140.120883 142.500616 141.767822
(The calibration targets were sRGB curve, ~140 cd/m^2 and D65 WP.)

[madshi]

So, if I understood it correctly: for HD sources (input and output formats will be the same), I can use the "Gamut_Measurements" command to set the measured coordinates of my display. But for SD sources, if I want them to be reencoded to HD (which I do), it's better not using that command, and set input_format to NTSC/PAL and output_format to HD. Is this right?
No, that's wrong. As I said in my previous comment, you should simply forget all commands in the "defining standards" section. They are for people who want to reencode (not play/watch) videos, e.g. Blu-Ray -> x264 MKV.

All commands you'll ever need for madVR and video playback are in the first 3 sections of the manual (mandatory, optional, calibration). Please simple *forget* the commands in the "defining standards" section.

Yes, I know it's not perfect yet, but I preferred to release yCMS now and keep working on it. It should improve in future versions. And no, we don't need the 10 bit output...
So you're aware of a noise problem in dark areas and there's hope for an improvement?

I made some screenshots from different movies and different type of scenes to show all my concerns.
I'm sorry, but I don't really understand which of your images shows what exactly. What I hoped to get was just 2 images of a movie frame where you see that "noise in dark areas", with the following settings:

(1) Full 3dlut processing (without Gamut_Measurements, though, since it doesn't seem to work properly yet).
(2) 3dlut turned off.

I expect to see the "noise in dark areas" in image (1), but not in image (2). Thanks!

[FoLLgoTT]

Though, don't forget that in the second a perfect Y might run against a perfect xy, so it might ends up as an optional setting.

I don't quite understand that. Luma (Y) should be independant from xy. Maybe I missed something?

Sorry, but i didn't understand your question. What did you mean with "interpolate colors"?

Maybe I'm wrong, but from my understanding of color management you have to correct all mixed colors if the primaries/secondaries get corrected. Here is an interesting paragraph from Lumagen's Radiance calibration guide (http://www.lumagen.com/docs/Tip0002_GamutCalibration.pdf):

Many displays and video processors claim to have primary color controls but, instead,
they really only have Color/Hue/Offset controls. Color/Hue/Offset controls should not be
used to try to correct the primary colors, because these controls operate in the wrong
color space. Other manufactures attempt to calibrate the primaries using HLS color
space. While this is an improvement over Color/Hue/Offsets, it is not as good as
correcting in the RGB color space.

Lumagen uses RGB color space for primary color correction. Lumagen believes that RGB
color space – the native color space for the primary and secondary colors – is the proper
color space to use when correcting primary and secondary colors. This is because it
allows the primary and secondary colors to be corrected, while maintaining the correct
interpolation of colors. The Radiance performs this calibration in linear RGB color space
to assure the best possible image quality. Figure 2 shows an example of corrected
primary and secondary colors.

So I'm wondering how yCMS interpolate the mixed colors when correcting gamut. :)

[madshi]

yCMS does its internal calcuations in linear light RGB, AFAIK, just like the Lumagen Radiance. However, yCMS calculates in 64bit floating point. I don't think the Radiance is that exact.

[cyberlolo]

madshi answered it really well. I'll just add that the Gamut_Measurements command was created also having in mind the gamut correction for multiple saturations (not yet supported), and that would not fit in the Output_Primaries command.
With only one saturation allowed there is an overlapping between the two commands, but we decided to still introduce this command now because it's more user friendly than the Output_Primaries. A big part of the users would not need (or dare :D) to go past the Warning, so I must say that you are one of the braves...;)
The "Gamut_Measurements" command should always be used when the display's coordinates are measured, no matter what the source is. The source should be reflected only in the Input commands, and in none of the output commands. The commands relative to measures are all output commands, because you are measuring the output (display).

The Output_Primaries command should be used only when a user cannot measure the display's primaries but he has an idea of what videoStandard his display might be. Then, he should use this command to set it accordingly.

Ok, I think I got it now. I won't use Output_Primaries at all, and will use Gamut_Measurements with both SD and HD sources. Thanks for explaining!

And hey, I'm one of the braves because I need to use that awesome Input/Output_Range to set my black and white points at 16 and 240, as I asked you some time ago. By the way, it works like a charm (no banding), so thank you again!!

No, that's wrong. As I said in my previous comment, you should simply forget all commands in the "defining standards" section. They are for people who want to reencode (not play/watch) videos, e.g. Blu-Ray -> x264 MKV.

All commands you'll ever need for madVR and video playback are in the first 3 sections of the manual (mandatory, optional, calibration). Please simple *forget* the commands in the "defining standards" section.

Thank you too for your help, madshi. But I think there's at least one command in the "Standards definition" section that it's really useful for watching/playing videos: that is the "Input/Output_Range", which allows to set black and white levels of the video data, defining the valid range (all video data outside of it is clipped). This command solves the problem that I posted in your madVR thread some time ago, as gives the users the chance to set top white level at any point between 235 and 254, without introducing banding (which ffdshow's levels filter does). So with this, anyone can decide if to discard all WTW info (white level @ 235), only a part of it (let's say setting white at 245), or if preserve all WTW up to 254. And to me, this is a really important feature. Don't you think?

[cyberbeing]

There appears to be a bug where if you don't use Input_Format & Output_Format and instead use the "Advanced settings" for everything, yCMS complains that there is no Compression_Method value. Since it is supposedly optional, it should be defaulting to 0 when no value is specified.

Also, could you please explain what Compression_Method does? Is it currently just a placeholder at this point?

Other than "0" what would be an example value? None are listed in the manual.

[yesgrey]

How can I use the new IRE function and keep my Blacks, Black?
I've found the bug. It will be corrected in next version.

So you're aware of a noise problem in dark areas and there's hope for an improvement?
Yes. I simply forgot to include it in the first post as "Known issues". I improved it compared to cr3dlut, but the definitive solution would take me a little more time...

I don't quite understand that. Luma (Y) should be independant from xy.
In a perfect display yes, but I think there isn't any in the market yet... I only will know for sure when I have it working. It might be possible to have it all correct.

Maybe I'm wrong, but from my understanding of color management you have to correct all mixed colors if the primaries/secondaries get corrected.
That's the idea: correct all colors. The primaries/secondaries are only the reference values.

So I'm wondering how yCMS interpolate the mixed colors when correcting gamut. :)
There is no interpolation involved in yCMS. All colors are corrected in linear RGB color space using 64 bit FP for each component. Then, it's applied gamma encoding and the result is converted into the output format bit depth selection.
The only interpolation involved is in madVR, which interpolates 3DLUT input/output using tri-linear interpolation for finding more accurate values for its 32 bit FP calculated values.

There appears to be a bug...
Thanks. It will be corrected in next version.

Also, could you please explain what Compression_Method does? Is it currently just a placeholder at this point?
Yes.
When we defined the 3DLUT file format specification we considered the possibility of compressing the files, due to its big size and high loading time, so this option is to be used when the 3DLUT files could be compressed.

madshi and I are currently working on adding compression to the 3DLUT files.

[Blue_MiSfit]

It seems to really take advantage of this, one must have good calibration equipment, correct?

~MiSfit

[Mark_A_W]

Get a HCFR Blue_misfit.


But I don't think it's ready yet. I don't think there is a clear path to go from measurements to the 3DLUTs yet.

[yesgrey]

yCMS v1.1 released

http://yesgrey.com/ycms.html

- Increased Grayscale_Measurements maximum to 11.
- Compression_Method was not being considered as optional.
- Fixed incorrect black point when Grayscale_Measurements ire 0 was not set.

[cyberbeing]

- Fixed incorrect black point when Grayscale_Measurements ire 0 was not set.


Well blacks are now black, but near black shadows are crushed into bright solid tones, which makes the problem orders of magnitude worse.

http://img710.imageshack.us/img710/9760/normall.pnghttp://img37.imageshack.us/img37/6268/irec.png
| Regular 3DLUT on LEFT | 3DLUT with IRE on RIGHT |

All of the below give horrible results. Try it with a dark scene and see for yourself.
# Set input format
Input_Format HD YCbCr 8

# Set output format
Output_Format HD RGB_PC 16

# Grayscale IRE Measurements
Grayscale_Measurements
10 0.486085 0.472363 0.497613
20 2.855507 2.751755 2.748106
30 7.040504 6.968139 7.044012
40 13.197106 13.020103 13.015621
50 21.422743 21.428576 21.564960
60 31.921806 31.852612 31.998531
70 45.422878 45.317829 45.406380
80 60.435123 60.174709 60.314659
90 78.802399 78.381996 78.620575
100 97.811592 98.124657 98.652870

# Display Gamut Measurements
Gamut_Measurements 0.626 0.343 0.294 0.607 0.149 0.078 0.3127 0.3290

# Set input format
Input_Format HD YCbCr 8

# Set output format
Output_Format HD RGB_PC 16

# Grayscale IRE Measurements
Grayscale_Measurements
0 0.000000
10 0.486085 0.472363 0.497613
20 2.855507 2.751755 2.748106
30 7.040504 6.968139 7.044012
40 13.197106 13.020103 13.015621
50 21.422743 21.428576 21.564960
60 31.921806 31.852612 31.998531
70 45.422878 45.317829 45.406380
80 60.435123 60.174709 60.314659
90 78.802399 78.381996 78.620575
100 97.811592 98.124657 98.652870

# Display Gamut Measurements
Gamut_Measurements 0.626 0.343 0.294 0.607 0.149 0.078 0.3127 0.3290

# Set input format
Input_Format HD YCbCr 8

# Set output format
Output_Format HD RGB_PC 16

# Grayscale IRE Measurements
Grayscale_Measurements
0 0.000000 0.000000 0.000000
10 0.486085 0.472363 0.497613
20 2.855507 2.751755 2.748106
30 7.040504 6.968139 7.044012
40 13.197106 13.020103 13.015621
50 21.422743 21.428576 21.564960
60 31.921806 31.852612 31.998531
70 45.422878 45.317829 45.406380
80 60.435123 60.174709 60.314659
90 78.802399 78.381996 78.620575
100 97.811592 98.124657 98.652870

# Display Gamut Measurements
Gamut_Measurements 0.626 0.343 0.294 0.607 0.149 0.078 0.3127 0.3290

http://img56.imageshack.us/img56/1585/gammaplot2.gif

Do you think you are running into a problem because the BT.709 curve (red line in the above image) crushes low luminance blacks. It appears yCMS may be adjusting those crushed (non-existant) BT.709 transfer curve blacks into actual visible colors when you shouldn't be.

[Blue_MiSfit]

I have a Pantone Huey Pro. It doesn't look like HCFR supports it, but we'll see...

[kostik]

I have the same issue that cyberbeing reported. The shadows turn into red with or withoust the 0 IRE:( evrything but this issue works well, hope it will be fixed soon.
Thanks for your work :)

[yesgrey]

I have the same issue that cyberbeing reported. The shadows turn into red with or withoust the 0 IRE:( evrything but this issue works well, hope it will be fixed soon.
Thanks for your work :)
I'm working on it...

[BeNooL]

What are the value measurement expected by yCMS for the Grayscale_Measurements option?

In the manual, the values given as example are quite large (more that 450 at 100IRE) while the one shown by cyberbeing are more in the 0 to 100 range (same for me).
So just wondering if the proper data is given for the computation of this correction?

[yesgrey]

What are the value measurement expected by yCMS for the Grayscale_Measurements option?
Like said in the manual: >= 0.0

In the manual, the values given as example are quite large (more that 450 at 100IRE) while the one shown by cyberbeing are more in the 0 to 100 range (same for me).
That was intentional to let people know they could use any values.
The units are not important, because internally all values are divided by the maximum value.

[BeNooL]

Okay so let me rephrase the question: in HCFR Colorimeter which of the grayscale measurement has to be copied over to yCMS config file?
X, Y, Z, R, G, B, x, y etc?

PS: thank you very much for your development effort on this project.

[yesgrey]

in HCFR Colorimeter which of the grayscale measurement has to be copied over to yCMS config file?
X, Y, Z, R, G, B, x, y etc?
I don't have HCFR installed, but I think it should be Y (Luminance)...

[cyberbeing]

I don't have HCFR installed, but I think it should be Y (Luminance)...

Y (Overall Luminance) if you are only inputting a single value.

R (Red Luminance), G (Green Luminance), B (Blue Luminance) if you are inputting multiple values.

[Thunderbolt8]

a question regarding 1.1 (and other upcoming versions): I only need those 4 3dlut files for madVR, 2xHD and 2xSD as described in your example. I created these files with 1.0, should I now recreate those files with v1.1 and all other upcoming versions?

[BeNooL]

Thanks for the clarifications Yesgrey and cyberbeing.

Thunderbolt8, yes you have to regenerate the 3Dlut files with each new version of yCMS.

[yesgrey]

a question regarding 1.1 (and other upcoming versions): I only need those 4 3dlut files for madVR, 2xHD and 2xSD as described in your example. I created these files with 1.0, should I now recreate those files with v1.1 and all other upcoming versions?
If the new version does not correct any bugs shown in your current 3DLUT files there is no need to.
If you haven't used the Grayscale_Measurements you're should be ok, but I strongly advise to recreate the files whenever any bugs are fixed. It would be safer.
You can create a simple batch file that calls each of the input files and creates all the 4 files, which would take you only a few seconds. I don't think that's too much work... ;)

[janos666]

Nice progress. The Grayscale_Measurements command gives me better result than I had with the Output_Primaries command and it is nearly usable (for me) now.

But there is a huge bug with the out of gamut colors (I think) in this version. Take a look at the blue area on the upper left corner: off (http://img12.tar.hu/janos666/img/76364851.png) - on (http://img12.tar.hu/janos666/img/76364853.png)
I made these screenshots with enabled/disabled 3dlut which was built with these settings:
# Source video format
Input_Format HD YCbCr 8

#3DLUT output format
Output_Format HD RGB_PC 16

# Gamut correction - measured native x;y coordinates
Gamut_Measurements 0.657073 0.329413 0.231911 0.690324 0.140940 0.090552 0.311880 0.329555

I do not want to guess about the real problem (output bitrate, my perception, poor quality video source, poor quality display) but I think that gamma correction sometimes kills the shadow details. I can hear as shadows are screaming and dieing here: off (http://img12.tar.hu/janos666/img/76364850.png) - on (http://img12.tar.hu/janos666/img/76364852.png) (Sorry but I was a Lexx fan. :rolleyes:)
I used the lines above with these:
#Gamma correction - measured grayscale RGB luminances
Grayscale_Measurements
0 0.151752 0.146392 0.138733
10 1.564805 1.551411 1.491997
20 4.668003 4.810841 4.791349
30 10.406371 10.604137 10.320132
40 19.016638 18.991844 18.722298
50 30.179890 30.871260 30.149662
60 44.580005 45.085168 44.541965
70 63.151944 64.109925 63.294323
80 84.184549 85.582548 84.539725
90 110.365791 112.272175 111.200465
100 140.120883 142.500616 141.767822

[cyberlolo]

@yesgrey: I think I've found a bug too with the Input_Range parameter. Please take a look at this two images: With Input_Range (http://www.gandario.com/cal_img/Weeds_on.png) - Without Input_Range (http://www.gandario.com/cal_img/Weeds_off.png).

In the first one, you can clearly see the problem (I've added a white ellipse around it). In the second one, this problem is not there, but of course the levels are then wrong (they're 0-255 instead of 16-240).

I made both screenshots with enabled 3dlut, which was built with these settings. For the first image:


Input_Format NTSC YCbCr 8

Output_Format HD RGB_Video 16

Gamut_Measurements 0.6830 0.3170 0.3130 0.5810 0.1390 0.0500 0.2890 0.2800

Input_Range 16 240
Output_Range 0 255

For the second image, I used the same file, but removing the Input_Range parameter:


Input_Format NTSC YCbCr 8

Output_Format HD RGB_Video 16

Gamut_Measurements 0.6830 0.3170 0.3130 0.5810 0.1390 0.0500 0.2890 0.2800

Output_Range 0 255

Thanks in advance.

[yesgrey]

I think I've found a bug too with the Input_Range parameter.
Thanks for the report, it's the same bug. I'm currently working on it...

[cyberlolo]

Thanks for the report, it's the same bug. I'm currently working on it...

Ok then, I thought it could be a different one, as this seemed to be caused by a different parameter.

Thanks again for your work.

[cyberlolo]

yesgrey, here is more info in case it can help you: I had a surprise today when I tested again the same video, and the problem had disappeared. The only things I made are:
1) Suspend+wakeup PC.
2) Enable ffdshow's levels filter+disable 3dlut in madVR. After that, I switched back by disabling ffdshow's levels filter and re-enabling 3dlut (using the exact same file than before) in madVR, and then everything was Ok. The problem has gone by now.

Weird. Isn't it?

[yesgrey]

(using the exact same file than before)
Which 3DLUT file (with or without the Input_Range command)?

[cyberlolo]

Which 3DLUT file (with or without the Input_Range command)?

The one WITH the Input_Range command.

[yesgrey]

Yes, that's weird, though it doesn't help me...
I already know what the problem is, but I need to find time for correcting it...

[janos666]

I tried to add more low IREs to grayscale measurements to increase the shadow detail with gamma correction but I got an error that I should give the numbers in increaseing order. My IRE list was: 0, 1, 2, 3, 4, 10, 20, 30, 50, 70, 100. So, the real problem is that I should keep equal steps between values or I can not use too little steps.

Can you upload the v1.0 version for me? I can not enjoy any improvements/fixes but I have a bug which I didn't noticed with v1.0 (some - may be out-of-gamut - colors bands to black -> I don't know if this bug exists in v1.0 too but I didn't remember if I saw that)

[Mark_A_W]

Guys

How do I go from an HCFR measurement to a custom 3DLUT?

Can we do that yet?


Thanks

Mark

[janos666]

Do a measurement set with HCFR (primary colors and white will be enough for now but grayscale could be useful later, so choose the icon with the most rings...) and use copy-paste (CTRL+C and CTRL+V keys on your keyboard) the xyY values to a text file, delete the Y values and sort the xy values (in R G B W order) to a row and copy-paste it after the Gamut_Measurements command in your yCMS settings file.
You can do the same with grayscale RGB values but I do not think it is useful with current softwares.