yCMS - Color Management System

[tschi]

Hi, just come back from work, I test madVRprecision3 but I still have pink for red primary :(
I try to generate a 3dlut and load it in madVR using this code
# Set input format
Input_Format yRGB RGB_Video 8

# Set output format
Output_Format HD RGB_PC 16

Gamut_Measurements 1
19.283 0.652 0.320
83.538 0.280 0.593
9.781 0.142 0.077
114.411 0.313 0.329
but madVR said the 3dlut file does not match the output format required.
I also test
Output_Format yRGB RGB_PC 16 without any sucess, what should be a correct code ? thanks

[madshi]

You need to use "Output_Format yRGB RGB_Video 16". Switching between video <-> PC levels is done via shader code. The 3dlut must always be video levels input/output for madVR v0.61+.

[tschi]

Thanks !
I generate 3Dlut using this code
# Set input format
Input_Format yRGB RGB_Video 8

# Set output format
Output_Format yRGB RGB_Video 16

Gamut_Measurements 1
19.283 0.652 0.320
83.538 0.280 0.593
9.781 0.142 0.077
114.411 0.313 0.329
I am able to load it in madVR :)
But red is still pink :(
I can provide video pattern and 3dlut files if needed...

[yesgrey]

I did get confused by the formats. You want Yxy. HCFR has xyY. Are they the same?

Yes, only the position of Y is different. Both expressions are used.

But the results are awful. The colours are wrong. White below 235 is completely purple and black below 16 is green.
But red is still pink :(
There is a bug in madVR. madshi and I are working on it... until then, you should use v0.61 if you want to use the 3DLUT files.

[tschi]

ok, thank you both for all your hard work :cool:

[fairchild]

Here's comes the next try. I think this one should work pretty well:

http://madshi.net/madVRprecision3.rar

Alright, just got home and gave this new one a try. It fixed the black/white levels but the original issue is back. (blue glitching and being clipped, red being clipped early, red tinge in white levels in APL clipping pattern)

@Mark, can you post or link me your chc file generated by HCFR after the calibration. Because I think your values are not correct. It shouldn't be causing results like you posted in that grayscale image.

[Mark_A_W]

Yes, only the position of Y is different. Both expressions are used.



There is a bug in madVR. madshi and I are working on it... until then, you should use v0.61 if you want to use the 3DLUT files.

I'm sorry, I don't quite understand what you mean by "both expressions are used".

Both Yxy and xyY are the same aren't they, just the values in different order? Hope so, as that's how I transcribed the numbers.

I tried the precision3 madVR version, that's not the problem. I think either my measurements or transcribing are funky.


Alright, just got home and gave this new one a try. It fixed the black/white levels but the original issue is back. (blue glitching and being clipped, red being clipped early, red tinge in white levels in APL clipping pattern)

@Mark, can you post or link me your chc file generated by HCFR after the calibration. Because I think your values are not correct. It shouldn't be causing results like you posted in that grayscale image.

Fairchild, chc file is here: http://dl.dropbox.com/u/1372034/XG-1101LC%20MP%20Mods.chc

And here is my settings.bin, you can grab the values out of it, if you like: http://dl.dropbox.com/u/1372034/settings.bin

Thanks

[pankov]

guys,
how do you export the values from colorHCFR in madVR/yCMS compatible format?
Is there a .chc to yCMS "converter"?
Please, don't tell me I'll have to convert (copy&paste) them manually one by one.

[fairchild]

Fairchild, chc file is here: http://dl.dropbox.com/u/1372034/XG-1101LC%20MP%20Mods.chc

And here is my settings.bin, you can grab the values out of it, if you like: http://dl.dropbox.com/u/1372034/settings.bin

Thanks

Ok first thing, this was the absoulte best results you could achieve on your grayscale? dE's optimally you want to have them under 5.0 from 30-100 IRE and you had a bunch over 10 and only one under 5.0 (30 IRE you have at 3.7).

Secondly, once you have your grayscale how you want, you work on the color. If you are going to use this calibration, your luminance on red is way too high, you want it to be 0.21 of your white (I'm assuming you took measurements of your primaries with the same level of the white). So if that's correct your white Luminance is 10.833 so you want your red luminance to around 2.275 and yours is over twice that much. If your display has a working CMS, you could bring the dE of the primaries much better.

If you are confused on this, here is a guide which is really good and breaks stuff down clearly.

http://www.curtpalme.com/forum/viewtopic.php?t=10457

Moving along, this is what you would enter into yCMS given the data from your calibration file. You can compare this to what you entered. First is the primaries/gamut, it seems you entered some incorrect grayscale values at the early IRE's which is probably what is screwing you up:

Gamut_Measurements 1
5.112210 0.628470 0.336796
9.017853 0.293739 0.595465
1.672253 0.155996 0.077642
10.833004 0.321126 0.341560

Then these are for the grayscale/gamma:

Grayscale_Measurements
30 1 1.179433 0.315635 0.333479
40 1 2.705139 0.311259 0.334691
50 1 5.036874 0.310992 0.340356
60 1 7.476284 0.315073 0.340151
70 1 10.363506 0.320395 0.341030
80 1 10.603373 0.320395 0.341423
90 1 10.748604 0.320739 0.342232
100 1 10.849747 0.320797 0.342380

@pankov
The fastest way I know of is just to copy and paste from the colorHCFR interface. It only takes me around 3-5 minutes to copy and paste everything formatted properly like I did in this post for another forum member. You can copy the 3 values at once (xyY or XYZ then just format it like I did above)

[yesgrey]

Both Yxy and xyY are the same aren't they, just the values in different order?
Yes.

I tried the precision3 madVR version, that's not the problem. I think either my measurements or transcribing are funky.
Trust me. madVR is not working right. Until it gets fixed it would be a waste of time looking for problems on your measurements transcription.

Please, don't tell me I'll have to convert (copy&paste) them manually one by one.
Go to Advanced->Export and export the values to a xls or csv file. Then you can copy all the values to the format yCMS needs and then create your 3DLUT file manually, or using madVR.

[Mark_A_W]

Ok first thing, this was the absoulte best results you could achieve on your grayscale? dE's optimally you want to have them under 5.0 from 30-100 IRE and you had a bunch over 10 and only one under 5.0 (30 IRE you have at 3.7).

Secondly, once you have your grayscale how you want, you work on the color. If you are going to use this calibration, your luminance on red is way too high, you want it to be 0.21 of your white (I'm assuming you took measurements of your primaries with the same level of the white). So if that's correct your white Luminance is 10.833 so you want your red luminance to around 2.275 and yours is over twice that much. If your display has a working CMS, you could bring the dE of the primaries much better.

If you are confused on this, here is a guide which is really good and breaks stuff down clearly.

http://www.curtpalme.com/forum/viewtopic.php?t=10457

Moving along, this is what you would enter into yCMS given the data from your calibration file. You can compare this to what you entered. First is the primaries/gamut, it seems you entered some incorrect grayscale values at the early IRE's which is probably what is screwing you up:

Gamut_Measurements 1
5.112210 0.628470 0.336796
9.017853 0.293739 0.595465
1.672253 0.155996 0.077642
10.833004 0.321126 0.341560

Then these are for the grayscale/gamma:

Grayscale_Measurements
30 1 1.179433 0.315635 0.333479
40 1 2.705139 0.311259 0.334691
50 1 5.036874 0.310992 0.340356
60 1 7.476284 0.315073 0.340151
70 1 10.363506 0.320395 0.341030
80 1 10.603373 0.320395 0.341423
90 1 10.748604 0.320739 0.342232
100 1 10.849747 0.320797 0.342380

@pankov
The fastest way I know of is just to copy and paste from the colorHCFR interface. It only takes me around 3-5 minutes to copy and paste everything formatted properly like I did in this post for another forum member. You can copy the 3 values at once (xyY or XYZ then just format it like I did above)


Thanks Fairchild.

The first thing is I have a CRT projector. It's quirky. And no, there is no CMS, you can't move the primaries, they are set by the phosphur and the filtering on the lenses.

(The forum you link to is the CRT projector forum BTW.)

The second thing is no, that is not the best I can do, I didn't even try to tweak the CRT colours, this was more about the process. I just measured and then dove into yCMS. Like said I recently swapped some relevant boards (testing mods) and haven't recalibrated.

Also, I put the PC gamma control back to flat before I started - I normally use a custom curve with different curves per colour. To compensate for a stubborn CRT. This caused the greyscale to play up a bit.


As for the red luminance...I really can't explain why it's high. If you look at the greyscale on the previous page, the red and green are identical at 100% white.
You can see the typical blue hump of a CRT (blue saturates) and NEC XGs always seem to have way too much red down low.
My gamut is not that far off REC709, two colours are a touch shy, and one is over, but not by much.


And I'll check the numbers again, I thought I did it without error, but there are a lot of numbers, and the order from HCFR to yCMS (in madVR) differ.

However I also recognise that Yesgrey said there's a bug.

Cheers

Mark

[fairchild]

(The forum you link to is the CRT projector forum BTW.)

That guide works with Plasma, LCD, DLP, SXRD, LCOS, CRT, etc.

[Mark_A_W]

Yeah, I know, but Curt Palme is the CRT projector forum. The other stuff has just sort of grown around the sides, but when we all got shoved off AVS forum we all went to Curt's forum.

[jmone]

madshi suggested I post here for advice over the madVR thread on what affordable gear I need to buy if I want to do my own calibration. Any suggestions? I have a Pio LX608a that has been calibrated by an ISF Tech a few years ago. It is fed via a Yami V2700, from a HTPC running madVR over HDMI from a HD5670, I did book him in to recalibrate the TV and grab some measurements but he postponed.
Thanks
Nathan

[jimwhite]

check out the ColorMunki.... i1pro type performance for much less $$$$ :p


_EDIT_: Wow!! it took 6 years to post 22 times :eek: Master Level lurker :D

[tschi]

jmone : I use x-rite eye-one display 2, yeah, I know that's not the best one :p but only 200$, it's ok for watching tv and if no professional use. I have a friend with eye one pro model , I should compare the result with a 1000$ stuff ... but I guess it 's not so far on my 5 years LCD :rolleyes:

[madshi]

Here's a new test build, anybody willing to test?

http://madshi.net/madVRfinally.rar

[fairchild]

Here's a new test build, anybody willing to test?

http://madshi.net/madVRfinally.rar

I am still experiencing the same issues.

[tschi]

Here's a new test build, anybody willing to test?
http://madshi.net/madVRfinally.rar
I have still the same pink-red color :(

[madshi]

Argh, thanks for testing, though.

[jmone]

Thanks Jim/tschi, lools like the ColorMunki is in the right sort of price/performance bracket for me, but there seems to be different bundles all at similar prices:
ColorMunki Photo Vs ColorMunki Design
(Image Specific Vs Spot Color Optimization)
(Nil Vs Pantone Matching and Geo)
(DigitalPouch Image Transport Vs Nil)
(Interactive CD Vs Nil)

Then there are bundles like "ColorMunki Photo + Colorchecker Passport":

I "presume" that Photo is all I need or are there other addons/accessories?

Thanks
Nathan

[*Touche*]

I have an ICC profile for my monitor, but no access to a meter to re-do the measurements. It would be great if it could be converted to a yCMS compatible format and used in madVR. Any plans and possible ETA for this feature?

[yesgrey]

I have an ICC profile for my monitor, but no access to a meter to re-do the measurements. It would be great if it could be converted to a yCMS compatible format and used in madVR. Any plans and possible ETA for this feature?
It's in my plans, but no ETA. However, do a search, because I guess some people here were able to create a 3DLUT file by using some data from their ICC files...

[Dogway]

Is there something wrong in my code?

# Set input format
Input_Format HD YCbCr 8

# Set output format
Output_Format HD RGB_Video 16


# Gamut correction
Gamut_Measurements 0.608 0.324 0.292 0.575 0.146 0.087 0.323 0.354

# Grayscale IRE Measurements
Grayscale_Measurements
0.320 0.290 0.329
1.057 0.314 0.349
2.712 0.319 0.352
6.367 0.322 0.358
11.875 0.320 0.355
19.461 0.320 0.353
28.516 0.322 0.355
39.409 0.323 0.355
52.761 0.325 0.357
69.888 0.325 0.355
88.791 0.323 0.355

yCMS says "Error! Line 17 - Invalid Format: Gamut_Measurements '0.608'"

I will make another one for SD, which one do I supply to madVR?

[yesgrey]

Is there something wrong in my code?
Which yCMS version are you using? The Gamut_Measurements and Grayscale_Measurements commands syntax is not that any more since v1.9.

which one do I supply to madVR?
Maybe you would be better by feeding your measurements into madVR and let it create the 3DLUT for you...;)

[Dogway]

Maybe you would be better by feeding your measurements into madVR and let it create the 3DLUT for you...;)

True, but learning always comes handy. I ended up using the next txt:
# Set input format
Input_Format yRGB RGB_Video 8

# Set output format
Output_Format yRGB RGB_Video 16

# Gamut correction
Gamut_Measurements 1
20.471 0.608 0.324
61.856 0.292 0.575
7.109 0.146 0.087
88.739 0.323 0.354

# Grayscale IRE Measurements
Grayscale_Measurements
10 1 1.057 0.314 0.349
20 1 2.712 0.319 0.352
30 1 6.367 0.322 0.358
40 1 11.875 0.320 0.355
50 1 19.461 0.320 0.353
60 1 28.516 0.322 0.355
70 1 39.409 0.323 0.355
80 1 52.761 0.325 0.357
90 1 69.888 0.325 0.355
100 1 88.791 0.323 0.355
Thank you
Warn me if something is wrong.

[yesgrey]

True, but learning always comes handy.
100% agreed. :)

Warn me if something is wrong.
I would remove the comments above the first two commands. I'm not sure that madVR is ignoring them...
Input_Format yRGB RGB_Video 8
Output_Format yRGB RGB_Video 16

[madshi]

madVR is stupid, but not that stupid... :D The comments don't harm.

[yesgrey]

OK. I did not test it thoroughly.;)

[Dogway]

What software can I use to tweak the results of my calibrated display? (laptop monitor) I used the spyder2 software, but it defaults to D55, where it seems digital content should use D65. How do I fix that and other off parameters after calibration? Sorry if its a bit offtopic, I tried looking for a calibration thread in doom9. I guess the values I feed to yCMS are measurements done with my profile already loaded, is it right? So madVR can compensate the slightly off values...

I also measured my TV, I used the internal TV menu sets to approximate to ideal Gamut and Greyscale, by measure->tweaking a dozen times. I finally got fairly decent parameters and gamma curve by setting contrast to 25 and brightness to 55 in TV, but it looks my whites are clipped and a bit low, if I set contrast to 70 ("before" line) it looks ok but my gamma curve is back to wrong place. I thought on using the "before" parameters (straight line), and just use in madVR a 1.80 value for Power Curve, Is this correct? If so which of them, the one in "calibration" or "color & gamma"?
http://i212.photobucket.com/albums/cc35/Dogway/Misc/gamma.png

[tschi]

Hi, If I make 3dlut with like this (madVR 0.61) Input_Format HD YCbCr 8
Input_Range 13 235
Output_Format HD RGB_PC 16
....
Would I have BTB informations like ffdshow level ?
http://red.skully.free.fr/Divers/levels_2.jpg
With madVR>0.61, (and yRGB 3dlut format) is it working ?
thx

[TheElix]

Does anybody know how EDID Color characteristics data of the monitor influences anything? Here's what I have:
Color characteristics
Default color space...... Non-sRGB
Display gamma............ 2.20
Red chromaticity......... Rx 0.640 - Ry 0.345
Green chromaticity....... Gx 0.291 - Gy 0.635
Blue chromaticity........ Bx 0.163 - By 0.093
White point (default).... Wx 0.288 - Wy 0.296
Additional descriptors... None
This isn't sRGB, and this isn't REC.709 either. I have my HDTV calibrated as REC.709 with 2.35 gamma. And I used that data with yCMS. Now I'm wondering how this EDID data influences anything...

[TheElix]

Something I've looked up about EDID:
Color Characteristics – defines the RGB color space conversion technique to be used by the display
So that means that those numbers really do influence the picture? If so then I'm not content with this, beacause those numbers are wrong.

[janos666]

Something I've looked up about EDID:

So that means that those numbers really do influence the picture? If so then I'm not content with this, beacause those numbers are wrong.

No, those numbers do nothing. They are just there if a software would use them but it's very rare.

How do you know they are wrong? These numbers can be either very accurate readings from a reference grade sensor in the factory or completely false decorations.
What kind of sensor did you use for measuring?

[TheElix]

What kind of sensor did you use for measuring?i1 Display 2. I know it's not so accurate yet I calibrated my plasma with it to a satisfactory result and can tell those numbers in EDID are way off. Nor they are close to any color stadard.
Well, if you say they don't influence anything then I'm relieved...

[janos666]

i1 Display 2. I know it's not so accurate yet I calibrated my plasma with it to a satisfactory result and can tell those numbers in EDID are way off. Nor they are close to any color stadard.
Well, if you say they don't influence anything then I'm relieved...

I don't think they are unbelievable if I just look at them.
They don't need to be close to any standards, they are supposed to show the actual characteristics. (And by actual, I mean the real physical primaries, not the emulated ones. The hardware CMS settings don't affect these numbers.)

Which plasma? All of the primaries are over-saturated on my PDP.
I don't have it anymore to test it on my PDP but my old i1LT was useless on my U2410. I could tell it without questions that the calibration result is worse than the factory state. I picked up a ColorMunki Photo and I found that it improved the calibration of my older l220x as well. It had a sRGB<X<AdobeRGB gamut (smaller but more similar to my current PDP).

However the low light accuracy of the ColorMunki doesn't enough for my current PDP, so I need to buy a colorimeter too. I want one of those new Display III sensors but not until ArgyllCMS will support them.

[madshi]

The latest madVR version (v0.66) should now finally fix the problems introduced by the last couple releases:

http://madshi.net/madVR.zip

I've once again changed the 3dlut requirements. If you let madVR create the 3dlut for you then you can skip the rest of this post.

If you want to manually create a 3dlut, it should be created like this:

(1) input/output must be RGB video levels
(2) input bitdepth must be 6, 7 or 8 bits
(3) output bitdepth must be 16 bit
(4) input transfer function must be a 2.20 pure power curve
(5) input/output primaries must match the measured gamut
(6) input white point must be D65

madVR v0.66 will refuse to use external 3dluts with different bitdepths. It will also refuse to use external 3dluts which contain the text "RGB_PC" or "YCbCr", or which do *not* contain the "Input_Primaries" command.

Here's an example of how madVR v0.66 itself creates 3dluts now:

Input_Primaries 0.640000000000000 0.330000000000000 0.300000000000000 0.600000000000000 0.150000000000000 0.060000000000000 0.31272661468101209 0.32902313032606195
Output_Primaries 0.640000000000000 0.330000000000000 0.300000000000000 0.600000000000000 0.150000000000000 0.060000000000000 0.31272661468101209 0.32902313032606195

Input_Transfer_Function 1.0 0.0 0.45454545454545454545454545454545 0.0
Output_Transfer_Function 1.0 0.0 0.45454545454545454545454545454545 0.0

Input_Matrix_Coefficients 0
Output_Matrix_Coefficients 0

Input_Range 16 235
Output_Range 16 235

Input_Bit_Depth 8
Output_Bit_Depth 16

Gamut_Measurements 1
1.0 0.640000000000000 0.330000000000000
1.0 0.300000000000000 0.600000000000000
1.0 0.150000000000000 0.060000000000000
1.0 0.312726614681012 0.329023130326062
Make sure you always replace the last two floating point numbers in "Input_Primaries" with "0.31272661468101209 0.32902313032606195", which is the D65 white point. For the "Output_Primaries" use the measured white point instead.

Please note that this is just the way it *currently* works. I might go back to yRGB sooner or later.

[TheElix]

Are you using Grayscale_Measurements command? Without it is not likely for you to have perfect results...
Yes, here's my data:
20, Yxy, 3.143497, 0.313949, 0.320639
30, Yxy, 7.019581, 0.314259, 0.326342
40, Yxy, 12.641442, 0.314473, 0.326627
50, Yxy, 20.403746, 0.312119, 0.326721
60, Yxy, 31.174128, 0.315227, 0.332607
70, Yxy, 43.673073, 0.315350, 0.331502
80, Yxy, 58.332012, 0.314300, 0.329078
90, Yxy, 76.145157, 0.314849, 0.330899
100, Yxy, 96.699242, 0.313689, 0.328451

[yesgrey]

Yes, here's my data
OK, so it's your display that is limiting you. The blue and red are limited, hence why you cannot achieve the standard values. In the green the difference is minimal, and you would not notice any difference.

[TheElix]

OK, so it's your display that is limiting you. The blue and red are limited, hence why you cannot achieve the standard values. In the green the difference is minimal, and you would not notice any difference.The point of yCMS then...?

[tschi]

I just made a very small test w/o meter and the red primary is red now, with my eyes at least. Thanks :cool:

[yesgrey]

The point of yCMS then...?
With your current display only to correct any flaws in your display's gamma curve, if there are any. Try with the 3DLUT and without and judge for yourself what looks better...

[cyberbeing]

* instead of yRGB, madVR/yCMS are now using the measured display primaries
Can you explain why you are doing this?

All the below would do is correct the white point, since the RGB primaries are identical in input and output. Also keep in mind that Gamut_Measurements overrides Output_Primaries so Gamut_Mesurments isn't needed how you currently are creating the 3dlut because they both have identical data.

Where in the display chain is the gamut being corrected, considering the 3dlut is no longer doing so? Are the shaders now taking into account the display primaries and doing the gamut adjustment before feeding the 3dlut?

Input_Primaries 0.626473564822893 0.34138544756587 0.294038817005545 0.608576709796673 0.148466540292538 0.0774905646709701 0.31272661468101209 0.32902313032606195
Output_Primaries 0.626473564822893 0.34138544756587 0.294038817005545 0.608576709796673 0.148466540292538 0.0774905646709701 0.313032323542798 0.329082368907846

Input_Transfer_Function 1.0 0.0 0.45454545454545454545454545454545 0.0
Output_Transfer_Function 1.0 0.0 0.45454545454545454545454545454545 0.0

Input_Matrix_Coefficients 0
Output_Matrix_Coefficients 0

Input_Range 16 235
Output_Range 16 235

Input_Bit_Depth 8
Output_Bit_Depth 16

Gamut_Measurements 2
44.8790 24.4560 2.3025
31.8150 65.8480 10.5370
18.2410 9.5207 95.1010
95.1206 99.9977 108.7500

[madshi]

Can you explain why you are doing this?
Because we've run into some problems with yRGB and we're not sure yet where they're coming from. So the v0.66 solution is a workaround for now. Maybe I'll stay with this solution, or maybe I'll go back to yRGB once the problems are sorted out. Not decided yet.

All the below would do is correct the white point, since the RGB primaries are identical in input and output.
The white point is corrected by the 3dlut, with chromatic adaption (which madVR can't do). Also the 3dlut has specific algorithms to handle out of gamut colors. Which madVR also doesn't have. Furthermore, the gamma response of the display is fixed, if you provide grayscale measurements.

Also keep in mind that Gamut_Measurements overrides Output_Primaries so Gamut_Mesurments isn't needed how you currently are creating the 3dlut because they both have identical data.
Having both doesn't harm.

Are the shaders now taking into account the display primaries and doing the gamut adjustment before feeding the 3dlut?
Yes. But the shaders are not doing white point correction (because the 3dlut can do that better) and are not handling out of gamut colors in any way.

[cyberbeing]

The shaders are doing gamut correction but not handling out of gamut colors? How does the 3dlut handle out of gamut colors in this case? Setting Input_Primaries to the display primaries makes the 3dlut assume their are no out-of-gamut colors. Unless you are talking about out-of-gamut colors created during white point correction? In any case this shouldn't be a problem, just trying to understand how it works now.

When I have some time later I'll do some measurements and see if this resolved my problem with 0.65.

[madshi]

The shaders are doing gamut correction but not handling out of gamut colors? How does the 3dlut handle out of gamut colors in this case? Setting Input_Primaries to the display primaries makes the 3dlut assume their are no out-of-gamut colors.
Out of gamut colors are in the BTB and WTW areas. The 3dlut is RGB *video* levels. Some out of gamut colors might be clipped by the shader math into the BTB/WTW range, though. Of course using yRGB would be better for out of gamut handling, but it was about time we fixed the broken calibration. So I preferred to offer a 99% solution over a 0% solution.

[cyberbeing]

I'm a bit confused since BTB and WTW has to do with levels not gamut, but you are referring to them as out-of-gamut colors.

BT.709 is a gamut. Your display primaries represent a gamut. If your display primaries have a larger gamut than the BT.709 gamut, the shaders would need to handle out-of-gamut colors during gamut conversion, before feeding the 3dlut in madVR 0.66. That is what's happening, correct?

[madshi]

I'm a bit confused since BTB and WTW has to do with levels not gamut, but you are referring to them as out-of-gamut colors.

BT.709 is a gamut. Your display primaries represent a gamut. If your display primaries have a larger gamut than the BT.709 gamut, the shaders would need to handle out-of-gamut colors during gamut conversion, before feeding the 3dlut in madVR 0.66. That is what's happening, correct?
I think we're talking about different out-of-gamut colors here. You're thinking about a bigger display gamut than source gamut. I was thinking of a smaller display gamut than source gamut. You're thinking about colors which are out of gamut *before* the gamut conversion. I was thinking about colors which are out of gamut *after* the gamut conversion.

Anyway, I'm not really an expert in calibration matters.

What I can say is that madVR does not have any out-of-gamma specific processing at all while I know that yCMS has some sort of out-of-gamma special handling. It'd probably be better if I left this discussion to yesgrey.

[cyberbeing]

yCMS 3dlut's only handle BTB/WTW and White Point how they are being created with madVR 0.66.
The Input_Primaries needs to represent the primaries of the video madVR is using as input for the 3dlut.

Which means madVR 0.66 is as follows with a BT.709 video? Gamut correction performed by shaders? WTW/BTB and White Point correction performed by the 3dlut?
madVR:
(1) input = 8bit Y'CbCr, 709 primaries/gamut, 709 gamma, WTW/BTB, D65 White Point
(2) convert -> 32bit float R'G'B', 709 primaries/gamut, 709 gamma, WTW/BTB, D65 White Point
(3) convert -> 32bit float RGB, 709 primaries/gamut, linear light, WTW/BTB, D65 White Point
(4) convert -> 32bit float RGB, Display primaries/gamut, linear light, WTW/BTB, D65 White Point
(5) convert -> 32bit float R'G'B', Display primaries/gamut, pure power 2.2 gamma, WTW/BTB, D65 White Point
3dlut:
(6) input = 8bit float R'G'B', Display primaries/gamut, pure power 2.2 gamma (with trilinear interpolation), WTW/BTB, D65 White Point
(7) convert -> 64bit float RGB, Display primaries/gamut, linear light, 16-235, D65 White Point
(8) convert -> 64bit float RGB, Display primaries/gamut, linear light, 16-235, Display White Point
(9) convert -> 16bit int R'G'B', Display primaries/gamut, Display corrected gamma, 16-235, Display White Point
madVR:
(10) input = 16bit int R'G'B', Display primaries/gamut, Display corrected gamma, 16-235, Display White Point
(11) dither down to 8bit R'G'B', Display primaries/gamut, Display corrected gamma, 16-235, Display White Point

[yesgrey]

(7) convert -> 64bit float RGB, Display primaries/gamut, linear light, 16-235, Display White Point

(7) is not correct. It should be:

(7) convert -> 64bit float RGB, Display primaries/gamut, linear light, 16-235, D65 White Point