yCMS - Color Management System

[6233638]

I made some others tests and always the same result:(
but now I understand what is happening :)
yCMS try to adjust gamut to the reference using xy position but the reference are out of my tv gamut :
http://a.imageshack.us/img822/9329/gamut.png
So in my case, may be the best things would be to compute a target which minimize the distance from the reference instead of trying to adjust to xy reference postion.Use DeltaE 94 broken into LCH components to find the best compromise for where green should be. Distance (deltaE uv) is not always most important for color. (it is for grayscale) Remember color is 3D not 2D that chart only shows half the picture.

In order of importance unless theres major deficiencies its Luminance > Hue > Saturation. YCMS is probably doing the best correction so your greens have enough yellow in them.

Also make sure your using a uv chart for gamut and both axes are the same scale to give an accurate estimate of color, xy is not perceptually correct.

[tschi]

I will try to find a solution for it...
Thank you, it would be great !:)

Use DeltaE 94 broken into LCH components to find the best compromise for where green should be. Distance (deltaE uv) is not always most important for color. (it is for grayscale) Remember color is 3D not 2D that chart only shows half the picture.

In order of importance unless theres major deficiencies its Luminance > Hue > Saturation. YCMS is probably doing the best correction so your greens have enough yellow in them.

Also make sure your using a uv chart for gamut and both axes are the same scale to give an accurate estimate of color, xy is not perceptually correct.

I am not sure to understand your technical lingo :confused:
Anyway, my grayscale delta E is quite good between 40 to 100 IRE (see imageshack link) with and without ycms, but I would like to adjust the gamut near the reference as best as can. I am not sure it would make a big visual difference but geometrically speaking it would be the optimum :rolleyes:

[6233638]

Sorry. If your measuring distance from the point in two dimensions thats the deltaE uv number. Theres different ways of calculating color error with deltaE 76, 94, 2000, uv....
dE76 is default in HCFR I think. Its suitable for grayscale errors but not color error. Ideal measure for grayscale is dE uv because it only measures distance from D65 and ignores gamma error but its very strict.

For color errors you need to use dE94. dE94 is a combination of three separate dEs. Luminance which is color brightness. Hue which is color tint. Saturation which is color.... vividness. Some programs, not sure about hcfr, let you see the LCH on their own which is great for adjusting a CMS.

Color has three dimensions. Hue, Saturation and Luminance. The gamut chart you posted only shows hue (shift left/right) and saturation (in/out) ignoring luminance. If you only look at the distance on the gamut chart this is the same as using dEuv which is not a good measure for color performance as it ignores luminance.

In order of importance for being correct its 1. Luminance 2. Hue 3. Saturation. What you said sounds like your putting saturation first and ignoring luminance completely.

The chart you posted also looks like its the default one in HCFR. You need to right click it and set it to a uv chart and export it as a square image so the x y axes are the same scale. Only then can you judge distance between the measured point and target if you dont have deltaE uv numbers.
The default xy gamut chart is not a good representation of the data its misleading.

[janos666]

I made a custom test chart file for DispCalGUI and an excel sheet which converts the results into an yCMS v1.7 compatible input file format.
It can be very useful for ColorMunki owners (or anybody), so I decided to "release it": yCMS_Argyll_comp_v0.1 (http://www.tar.hu/janos666/yCMS_Argyll_comp_v0.1.rar)

But the real reason why I uploaded it that I want to ask you if my calculations are correct or not. :)
I am not sure about the color space of the RGB values. I used the Rec709 color space now. Should I use the device color space?
Should I use gamma weighting for the RGB values? (I think I should not because yCMS will use them for gamma correction... but I am not sure about your assumptions about the input values. I think you tested it with the values from HCFR, and I don't know how HCFR works.)

I decided to use 128 gray steps because yCMS gave me an error when it found slightly decreasing values (instead of increasing values) and I couldn't really insert 256 IREs. [###]
But I believe that it is my display's fault because it varies with various hardware settings (I couldn't find a proper solution with this display mode yet.) It is safer for a "public release" anyway, and should be enough.
And the last ArgyCMS version still has a ColorMunki related bug (but there is no any alternative and the next bug fix version will be released soon, so...)
* I recommend the Adaptive High Resolution mode for the i1 Pro and ColorMunki spectrophotometers! (And the latest ArgyllCMS version. You should check if the new bug fix version released!)
** I think I want to go ever further and start from the spectral data (you can ask ArgyllCMS to record it) because my latest paranoia is the EIZO white paper where they stated that we should use an XYZ color space with 10˙filed of view instead of the default 2˙field of view for wide gamut displays. I think this story will never end. :p

[###] I think I figured out the source of my old problem. I tired to insert my 256 values like this:
Grayscale_Measurements 0 0.072061393 0.109495877 0.165268895
0.7874 0.075778028 0.112976172 0.168187243
Because I thought that a first empty line counts like an IRE (when I got an error that I inserted more than 256 values) but it actually ignored the first line, so there were no black offset and the near-black scale was erroneous.
I think this little format problem caused the black swatches.

I can't notice any significant detail losses on dark areas now. And I think I like it better than the hardware emulation. (The contrast ratio is slightly higher and the device itself works with much better color accuracy in the native gamut mode, so the final quality depends on the software quality which looks great now... for the first look...)


But I need an advice: Is it theoretically better to set the RGB Gains first in the device's OSD?
Pro:
- It is a dynamic iteration. I can measure, correct and measure again until I find the best solution.
- It will be handled with 12 bit internal accuracy.
- It will be close but not perfect. The RGB Level graphs will fluctuate (and sometimes) cross each others (a little bit). (So, the software will always try to make further fine tunings.) And may be it worse if we consider the further circumstances in the "contra" section ->)
Contra:
- This optimization will belong to the native characteristics. It will permanently reduce the contrast ratio and the number of the available shades before we know the final corrections.
yCMS will make heavy gamma and gamut corrections. So, may be it is better to keep as much of the available shades and contrast ratio as possible. MadVR will assume that every shade is available...
(And may be it is still available because the device works with 12 bit internal accuracy. But I can't be sure until I measure it. And I can't be absolutely sure until I measure it with a very expensive instrument which would never happen...)

I tried to calibrate and profile my display (with BasicColor) with and without the RGB Gain optimization. I achieved significantly lower dE94 values and slightly lower contrast ratio with optimized Gains (as I expected).
But this was an iterative calibration. (It could measure, correct, and measure again. In theory -> I think BasicColor use much more simple process. But, for example, ArgyllCMS works with up to 4 iteration steps...)
And this is the first ColorMunki compatible BasicColor release, so it can also have some bugs. (And it is a great software but not the best. I couldn't find anything which is better and theoretically bug free now. :p)

I think I will try to make an excel sheet for dE94 and/or dE2000 calculations. But I am not sure how can I measure video test patterns (with a ColorMunki to get an usable output).

[yesgrey]

I am not sure about the color space of the RGB values. I used the Rec709 color space now. Should I use the device color space?
Yes. By the way, good work with the Excel ws, but the next yCMS version will support XYZ and xyY.

Should I use gamma weighting for the RGB values?
What do you mean with that?

I decided to use 128 gray steps because yCMS gave me an error when it found slightly decreasing values (instead of increasing values) and I couldn't really insert 256 IREs. [###]
But I believe that it is my display's fault because it varies with various hardware settings (I couldn't find a proper solution with this display mode yet.)
I don't believe it's your display's fault, because that would mean that different non-consecutive input values would have the same output value, and that's very unlikely... though, with digital displays anything is possible... but don't worry, even 128 points is overkill.

I think I figured out the source of my old problem. I tired to insert my 256 values like this:
Grayscale_Measurements 0 0.072061393 0.109495877 0.165268895
0.7874 0.075778028 0.112976172 0.168187243
Because I thought that a first empty line counts like an IRE (when I got an error that I inserted more than 256 values) but it actually ignored the first line, so there were no black offset and the near-black scale was erroneous.
The example on the manual shows an empty line, so, why have you put values on it? I thought you were reporting a bug, but now I understand what you were saying with you example... I will update yCMS to give an error when data is put on the command's line.

But I need an advice: Is it theoretically better to set the RGB Gains first in the device's OSD?
As I've said previously, I don't know. You need to try both ways and decide for yourself. It would depend on the display's calibration software/hardare precision. Sometimes it might be preferable to adjust the display and then fine-tune with yCMS, othertimes it might be preferable to adjust only brightness and contrast on display and perform the full calibration with yCMS.

[madshi]

The example on the manual shows an empty line, so, why have you put values on it? I thought you were reporting a bug, but now I understand what you were saying with you example... I will update yCMS to give an error when data is put on the command's line.
I'm wondering: Why did janos666 get black swatches without the IRE 0 measurement? Is the IRE 0 measurement so much different from what yCMS guessed for IRE 0? If there's no IRE 0, yCMS guesses it, right?

[yesgrey]

If there's no IRE 0, yCMS guesses it, right?
No. When there isn't no IRE 0 yCMS considers it to be 0.0. When the lowest given IRE is not too small, it's not a problem, but when, like in janos666 case, it's very small, it's a problem, because it ends up as a big discontinuity in the gamma curve.
I guess I will have to make yCMS detect and correct it and/or outputing a warning about it...

[janos666]

So the problem overview:
I couldn't insert 256 IREs because I got an error ("you can't insert more than 256 IREs"), so I tried to use the first empty line and yCMS accepted the input file.
I thought that the first line can be used and it always counts as an IRE (even when it is an empty line). But no, the first line was ignored and the other 255 values were used, so there were no black offset (here come the black swatches).

But it is still a bug: The real IRE limit is 255 instead of 256.


I believe this is a display fault because the luminance graph is very smooth in the factory calibrated sRGB mode but it always behaves strangely in the Standard mode. And it also depends on other hardware settings. I just figured out this behavior yesterday when I tried to optimize the WP with the Gains (I think I finally suppled this bastard :cool:). I will post some results later (with an updated excel sheet for validation purposes ; and input_transfer_function instead of gamma_curve, because I "rediscovered" the old bug with the gamma_curve and gamut emulation accuracy :D I am trying to calculate some CIE-Lab dE values right now).

[janos666]

Here are my results:

Uncalibrated state (hardware Gains at 255-255-255, linear VGA LUT, etc):

http://img12.tar.hu/janos666/img/84234651.png ; http://img12.tar.hu/janos666/img/84234652.png ; http://img12.tar.hu/janos666/img/84234653.png


yCMS Calibrated state (128 IREs + Primaries ; verified with 10 IREs + Primaries at 100% luminance):

http://img12.tar.hu/janos666/img/84234654.png ; http://img12.tar.hu/janos666/img/84234656.png ; http://img12.tar.hu/janos666/img/84234657.png

It's not bad. The gamut and gamma are very close. But the white balance correction is not so good (if even exists, I think it doesn't).

A little fine-tuning with the hardware RGB Gains (a little contrast ratio reduction as the "side effect" - but this is a 12 bit display, so it can hurt a 8 bit display...):

http://img12.tar.hu/janos666/img/84234658.png ; http://img12.tar.hu/janos666/img/84234659.png ; http://img12.tar.hu/janos666/img/84234660.png


yCMS Calibrated state (128 IREs + Primaries ; verified with 10 IREs + Primaries at 100% luminance):

http://img12.tar.hu/janos666/img/84234661.png ; http://img12.tar.hu/janos666/img/84234663.png ; http://img12.tar.hu/janos666/img/84234664.png

Well, this is fine. Nearly perfect gamut, gamma and white balance. (I think the WB is untouched here as well as earlier. Don't forget that it looks much smoother because I used much less sample points. Or yCMS would do small changes only.).

The only remaining question is the ratio of the lost and remained shades (if any -- I don't want to start it over now/yet/never but it is not measured/calculated yet*) and the accuracy of the mixed colors (I found it too hard to calculate CIE-Lab dE94 values in an excel sheet to do it only for curiosity...).

* I think you can do some kind of check on the 3DLUTs to calculate the number of the lost shades (clipped values, etc).

** Another note: SpotRead doesn't support Adaptive measurement mode, so these measures were made in normal High Resolution (spectrum sampling) mode. It can be more accurate in Adaptive mode. (And I can use the Adaptive mode for capturing the uncalibrated characteristics...)


And my personal problem that the two darkest grays are equal in this display mode. But this is nothing when compared to other possible calibration "side effects".
But this display mode offers significantly higher contrast ratio than the built-in sRGB emulation mode and contrast is good for movies.
(And the gamut is closer but it can be the difference between my instrument and the factory meters. It is below the "just noticeable difference" dE94 value anyway...)

[madshi]

I'm not sure why but I think primaries are usually measured at 75% Luminance?

[BeNooL]

Thanks janos666, I'll give your solution a try this week end.
I have a HCFR colorimeter, not the best but that's all I have for now.

[janos666]

Thanks janos666, I'll give your solution a try this week end.
I have a HCFR colorimeter, not the best but that's all I have for now.

yCMS_Argyll_comp_v0.2 (http://tar.hu/janos666/yCMS_Argyll_comp_v0.2.rar) (new: graphical validation ; changed: input_transfer_function instead of gamma_curve).

I'm not sure why but I think primaries are usually measured at 75% Luminance?

The near-black and near-white scale may have some defects on some (digital) displays (with various hardware settings). But no, most of the softwares measure them at 100% luminance. (Instruments like the higher light intensity.)

But I also thought about it and I already measured them at 75% luminance. I think it is good for error checking (to see if the gamut emulation is valid for every luminance levels or not).

Results with inaccurate device WP and hardware corrected device WP at 75% luminance (and 70% for the WP):

http://img12.tar.hu/janos666/img/84234655.png ; http://img12.tar.hu/janos666/img/84234662.png

[yesgrey]

changed: input_transfer_function instead of gamma_curve.

Nice work and nice results. :)
However, you should not forget that by changing the Input_Transfer_Function instead of Gamma_Curve you get "perfect" gamut graphs but that's not a guaranty that the same will happen with your movies. I still think that the gamma should be changed by using the Gamma_Curve command.

[janos666]

However, you should not forget that by changing the Input_Transfer_Function instead of Gamma_Curve you get "perfect" gamut graphs but that's not a guaranty that the same will happen with your movies. I still think that the gamma should be changed by using the Gamma_Curve command.

In my understanding, the Input_Transfer_Function defines the transfer function of the assumed end-user display. And according to my current "faith" (we can only guess but I also made some subjective tests, like a Rec709 VGA LUT calibration and uncorrected or linearized image through this calibration, etc) this should be a pure exponential curve with an exponent about 2.35 (I am not absolutely sure about the exponent [2.2 ... 2.4] but I believe that it should be a pure exponential curve).
In this case, the gamma_curve command is redundant and it's result is less accurate (theoretically and subjectively).

I watched two Blu-Ray movies (Infestation - a comedy with many daylighted or bright indoor scenes ;and; The Cave - a horror with lot of extremely dark scenes) and my overall subjective experience was good.
Of course, it depends on many other things and placebo exists, so I don't want to make a judge yet. (But it felt good. I didn't experienced the "something is not right" subconscious feeling.)


I have an idea to make an assumption about the detail loss. You should check for identical sub-matrices* in the resulted 3DLUT and divide this by the number of the available shades in the 3x16 bit palette. (Different inputs shouldn't result in equal outputs.) I also want to see this number with 8 bit 3DLUTs. What do you think? Would it be easy or hard to implement? [*how do english people call them? I think I will make a visual illustration if you don't get it. :rolleyes:]

[BeNooL]

So I gave a try at AgyllCMS along with janos666's excel sheet to generate yCMS input file. I only have a HCFR colorimeter and first tested with my PC's monitor: a Iiyama ProLite E2003WS

In a nutshell, it's a disaster.

First of all, as the colorimeter is not very accurate I end up having yCMS complain that the grayscale measurement are not ordered.
Exact error message is
Error! 'Grayscale_Measurements' line n.xx is invalid.
All measurements must be specified in increasing order.

I proceeded to delete those faulty lines, but apparently yCMS has a bug and the reported lines don't match. I know it counts starting from the Grayscale_Measurements line onward still at some point it would always report line n-1 at faulty. (ie: error line 111, I delete 111, now I have error line 110, repeat)

So I updated the excel sheet to include a validation that all measurements from line n are inferior to line n+1. It prints an "err" on lines that do not satisfy this check.

After deleting all lines having an "err" flag yCMS accepts the input file!

It took me some time to figure out how to make the validation measurements but already by simply looking a video with madvr and 3DLut activated I could tell something was seriously messed up.

I still did the validation measurements with AVS test patterns.

Below are the results: left before with no correction and right with ycms active.

Gamut:
http://benool.free.fr/iiyama/gamut_uncal.pnghttp://benool.free.fr/iiyama/gamut_ycms.png

White balance:
http://benool.free.fr/iiyama/white_uncal.pnghttp://benool.free.fr/iiyama/white_ycms.png

Gamma:
http://benool.free.fr/iiyama/gamma_uncal.pnghttp://benool.free.fr/iiyama/gamma_ycms.png

We clearly see a lack of blue component but what's happening with the correction made by yCMS?!
It manages not so bad at first but what happens halfway?

I will try this afternoon with my RPTV and report how it goes.

[yesgrey]

I proceeded to delete those faulty lines, but apparently yCMS has a bug and the reported lines don't match.
We clearly see a lack of blue component but what's happening with the correction made by yCMS?!
It manages not so bad at first but what happens halfway?
Can you send me both your input files so I can test them? The first with faulty lines and the last one used for the measurements.

[BeNooL]

and this time I tried to calibrate my normal movie viewing device, a Sony KDS55A2000.

Gamut:White balance:Gamma
http://benool.free.fr/kds/gamut_uncal.png:http://benool.free.fr/kds/white_uncal.png:http://benool.free.fr/kds/gamma_uncal.png

As you can see I ended up having these strange seesaw for red component. This rendered the Input file unusable as I had to delete most of the lines due to values jumping up and down from one measure to the next.

I then ran ColorHCFR to check if still had more or less the same results as I did last time.

Gamut:White balance:Gamma (measured via ColorHCFR)
http://benool.free.fr/kds/gamut_uncal_HCFR.png:http://benool.free.fr/kds/white_uncal_HCFR.png:http://benool.free.fr/kds/gamma_uncal_hcfr.png

Except for the red shift it matches what I expected.

In the end it seem argyll doesn't like my HCFR sensor so much but the process to generate yCMS input file with it and the excel sheet is working great.

[janos666]

Sorry about that. I wanted to include this kind of error checking but I decided to exclude it (it was ready for the 256 IREs but I deleted it from the 128 IREs version). I forgot about the yCMS bug but I ran into the same trouble when I tried to use 256 IREs.

About your results...
You should read the ArgyCMS documentation. It talks about the HCFR sensor (unresolvable problems with 64bit Windows, poor accuracy, etc.).
And don't forget: It is a simple colorimeter. It won't work on wide gamut displays until you calibrate it to the specific display. You can do it with a reference instrument (spectrophotometers in this case). But I am not sure if you can save this calibration to the instrument itself or you are limited to the HCFR software.

The native wide gamut usually looks more saturated with uncorrected colorimeters but you could measure the corrected gamut (more or less). This is why the corrected gamut is too small.

This should be a bug but you can really have jagged graphs. I have strange jumps (not as much as you but something like this) when I set my display to Game mode (low input lag mode).

And I already mentioned that yCMS doesn't make a good job with the white balance. You should try to correct it with hardware controls first (it is necessary to get nice gamut.) -> After you calibrated or replaced your instrument.

[BeNooL]

You don't have to be sorry at all.
I know the sensor isn't very accurate still with the HCFR software simply adjusting the white balance yields excellent subjective results (HCFR requires a reference file for the display you are calibrating so this must help).

[Thunderbolt8]

there seem to be a few japanese blu-rays which have the wrong black levels, they might have used the wrong colourspace here. so they either mixed up 0-255 with 16-235, or the other way round.

any way to fix this using a 3dlut file? (for PC and TV)? if so, how?

[yesgrey]

any way to fix this using a 3dlut file? (for PC and TV)? if so, how?
A short sample would help to see what kind of correction should be done...

[Thunderbolt8]

A short sample would help to see what kind of correction should be done...
here are 2 short samples from the movie vengeance is mine (eureka master of cinema BD). picked a dark scene in which there are as well shadows and if you compare the darkest colours there with for example the black of the (mpc) black bars (in windowed mode), then you'll notice the difference.

http://remixshare.com/download/w5ybd
http://remixshare.com/download/dgd8e

http://forum.blu-ray.com/united-kingdom/106707-masters-cinema-moc-discussion-thread-50.html#post3597679
http://forum.blu-ray.com/united-kingdom/106707-masters-cinema-moc-discussion-thread-50.html#post3629961
http://forum.blu-ray.com/united-kingdom/106707-masters-cinema-moc-discussion-thread-52.html#post3657537

and here http://www.glennchan.info/articles/technical/setup/75IREsetup.html
"In Japan, the standard black level was changed from 7.5 IRE to 0 IRE in 1985"

[yesgrey]

here are 2 short samples from the movie vengeance is mine
Short answer
Add this line at the end of your input file:
Input_Range 30 218

Detailed answer
Looking at the samples it seems that the conversion might have been something like from 0-255 to 16-235, but the black|white on the source were on 16|235 and not 0|255, as the conversion assumed. So, you should compensate for it by changing the Input_Range on your yCMS input file.

The theoretical values are 29.741 (16+(16*219/255)) for black and 217.824 (16+(235*219/255)) for white. yCMS only accepts integer values, so you should round them. This is not a big issue, because the error caused by the bad conversion is so big that the slightly error created from this rounding is not significant.

Fine tune (+/-1 on both values) to your personal taste.

[Thunderbolt8]

thanks for that answer. since Im a complete noob on that, I use your HD & SD PC & Video template files, so I add that line you suggested right here

# Example input file for yCMS v1.0 and up
#
# Settings for creating a 3DLUT file for watching the following Video formats:
# Blu-ray, HD DVD, ATSC HD Broadcast, DVB HD Broadcast
# without any Display correction
# Includes YCbCr->RGB conversion using PC Levels (Black: 0 and White: 255)

# Set input format
Input_Format HD YCbCr 8
Input_Range 30 218

# Set output format
Output_Format HD RGB_PC 16

and here?

# Example input file for yCMS v1.0 and up
#
# Settings for creating a 3DLUT file for watching the following Video formats:
# Blu-ray, HD DVD, ATSC HD Broadcast, DVB HD Broadcast
# without any Display correction
# Includes YCbCr->RGB conversion using Video Levels (Black: 16 and White: 235)

# Set input format
Input_Format HD YCbCr 8
Input_Range 30 218

# Set output format
Output_Format HD RGB_Video 16

apart from that:

- how did you get those result values (or those with whom you did the calculations)? did you put that sample in some kind of parser? would be good to know in case if the other BDs with the same issue should have slightly different values.

- how can I switch between the different .3dlut files in madvr? can you have more than those 4 different input names as listed in the first post which are accept by madvr, so you'd not have to rename the files manually each time before starting?

- when trying to have the colours exactly the way they are meant to be, how come you'd suggest fine tuning with +/-1, while the mathematical rounding clearly suggests using that one value and not the other? or are there maybe other slight errors which could occur and the result could mathematically be off by +/-1? or is it really just for personal taste which then might not be 100% correct in the end?

- are softsubtitles played together with that movie also affected by that colour change? if so, what to do that they keep their standard colouration (.ass subs) ?

[yesgrey]

so I add that line you suggested right here ... and here?
Yes.

how did you get those result values (or those with whom you did the calculations)? did you put that sample in some kind of parser?
No, I've just paused the image and used "ColorPic" (it's a free tool) to see if there were any pixels with values below 16. Since there weren't, then I suspected that had happened what you said, they probably converted a video that was already on video levels to 16-235, hence rising the black levels and, probably, lowering the white level. I couldn't confirm the white level lowering because the video samples were only of dark parts, but you can confirm it for yourself using ColorPic.

Next, assuming the standard values of 16 and 235 for black/white levels, here is the math of a conversion from 0-255 -> 16-235. It's a compression from a dynamic range of 255 (255-0) to 219 (235-16), adding an offset of 16:
out = 16 + (in * 219/255)
Assuming the original video had black at 16 and white at 235, the locations of the original black and white levels on the new video can be calculated using the formula above. For the black level, it's pretty safe to use 16 as the value, because any data below this value is not valid video data. However, for the white level this is not completely true, because some videos might have valid data above 235. If you prefer playing safe, you could use a higher value for the white level, for example 240, or 245, but I think that you could use 235 without any problems.

how can I switch between the different .3dlut files in madvr? can you have more than those 4 different input names as listed in the first post which are accept by madvr, so you'd not have to rename the files manually each time before starting?
Currently you can use 2 3DLUTs for HD and 2 for SD. The HD/SD selection is automatically performed by madVR according to the rules explained in my second post. If you want to use more, you will have to rename them. However, you could create a simple batch file just for the renaming, so you would need only a single click before watching your movies, not a big issue. I think madshi plans to support more 3DLUTs in future madVR versions, but I don't know when.

when trying to have the colours exactly the way they are meant to be, how come you'd suggest fine tuning with +/-1, while the mathematical rounding clearly suggests using that one value and not the other?
It's not exactly for personal taste, but more for "mental" comfort.

Let's look at the black level. Considering the calculated theoretical value (29.741) is the correct one, if you use 30, the correct math rounding, then the blacks would be slightly clipped - hardly noticeable. If you use 29, you will increase slightly the black level - hardly noticeable too. So, just use the value that gives you more "mental" comfort, because in the end you probably can't tell the difference...;)

are softsubtitles played together with that movie also affected by that colour change? if so, what to do that they keep their standard colouration (.ass subs)?
Yes, because they are rendered on the image before the 3DLUT usage. If you want to keep their standard colouration you should change their colours on the subtitle renderer filter using the inverse formula. However, try it first, because they should only become darker, so it might not be a big issue...

[Thunderbolt8]

I've just paused the image and used "ColorPic" (it's a free tool) to see if there were any pixels with values below 16.
how did you do that? I dont have any field for black or white from 0-255, only hue, sat, val, red, green and blue. black only seems to be expressed as a percentage there.

in case you mean the val field, ive found values as low as 11 so far (from the samples I made) and also 255.

heres a sample for val 255 (parts of the umbrella): http://remixshare.com/download/bncy5

[yesgrey]

how did you do that? I dont have any field for black or white from 0-255, only hue, sat, val, red, green and blue. black only seems to be expressed as a percentage there.

in case you mean the val field, ive found values as low as 11 so far (from the samples I made) and also 255.
First, setup madVR disabling the 3DLUT and selecting PC levels. Then, look for the R,G,B values to see the highest of them (black and white are when R,G,B have exactly the same values). I've looked and val is always the same value as the highest of R, G or B, so you can use it. I confirmed that the umbrella have both R and G at 255, so it seems only the blacks are wrong...

Considering the above it might be preferable using:
Input_Range 25 235
(25 because: 16 + 11*219/255 = 25.45)
For the output, select video levels or PC levels according to what your display is expecting...

To be more precise, all the process also might have messed up the gamma, so probably you will also need to perform some gamma correction, but let's first try setting the range and see how do you feel about the image...

[Thunderbolt8]

now I've found val = 6, but its R2 G4 B6, that doesn't count, right? in that case, I cannot confirm if that val 11 I found had 2 times the same value for R,B,G

Im a bit confused now. so e.g. val 14 would be ok if you had like R 14, G something else, and B 14 again? what if you had R & B 14, but G and therefore val would be 16?

edit: ok, Ive definately found now val 11 with RGB all 11.

edit²: according to a changed 3dlut like that, if we did it all right then we should be able to find val 0 RGB 0 and val 255 RGB 255 when checking on those samples again, right?

can only find val 255 RG 255, but blue seems to be a little off like around 248 at best. so I guess the assumed black end for the input range still wasnt entirely correct?

[jaydee81]

Does it make any sense to use this with a PC Monitor?
Thanks,
JD

[Thunderbolt8]

next update, I've found a value with 0, RGB = 0 now somewhere in the movie. in case that should somehow not be valid, then val & RGB 3.


so in the end, the line should be "input_range 16 235" ? (in case of 0)

edit: but then theres no difference at all when using the lut file or not -.- but the picture definately looks somehow wrong that way

[janos666]

Does it make any sense to use this with a PC Monitor?
Thanks,
JD

Yes, very much. PC monitors may require more corrections because TVs and Home Cinema projectors were made for movies (this is another topic if they are really optimized for this purpose or not -> like WCG HDTVs without internal color management...) while PC displays were made for general PC usage.

[yesgrey]

next update, I've found a value with 0, RGB = 0 now somewhere in the movie. in case that should somehow not be valid, then val & RGB 3.
I forgot one setting. For analyzing the image you must set all scaling algorithms to "nearest neighbour". That's the only way of knowing which are the lowest/highest values on the source image, because when other scaling algorithms are used some lower/higher values could appear but only as a result of the chroma upsampling and/or rescaling of the image, which is not real data.

[Thunderbolt8]

hm it didnt change anything, I still keep finding val 0 RGB 0 or val 2 RGB 2. so 0 really seems to be the lowest value in the movie (quite at the beginning of the movie).

heres a sample for some 0 val 0 rgb values: http://remixshare.com/download/bmqe6

but why does the movie then still look so wrong? looks much better with that lut file which keeps input black at 16 or 11. even in places where its very dark and there are some 0 val blacks around, the colour is still greyer than the black of the mpc-hc black bars.

[yesgrey]

but why does the movie then still look so wrong?
Because it's wrong.

The fact that you could find some RGB 0 values only shows that the video suffered some kind of processing, and those values are the symptom of that. The black level is clearly wrong, so you have to move it down. OTOH the white level seems to be fine, so I think you should only correct for the black.

[Thunderbolt8]

yes, but to which values for the input_range? if I take those lowest and highest values of 0 and 255 into that formula you gave me, then Ill end up with 16 - 235 which then makes no difference compared to watching with PC levels & without the 3dlut file (unless I made some math mistake).

and why does finding values below 16 make the black levels wrong, while finding values over 235 makes white correct? im confused again :S shouldnt it either be both correct or both wrong? why taking the 255 we found into consideration for the formula, but not the 0 we found?

[yesgrey]

yes, but to which values for the input_range? if I take those lowest and highest values of 0 and 255 into that formula you gave me, then Ill end up with 16 - 235
The white level is fine, so you should use 255 into the formula. The black level is not, so you should not use 0, but one of the other values instead. If you want to play a little safer you could go with 11, but from looking at the samples you upload you don't lose anything if you use 16 into the formula.

and why does finding values below 16 make the black levels wrong, while finding values over 235 makes white correct? im confused again :S shouldnt it either be both correct or both wrong?
If you look at the sample n.3, where the 255 values are, it's perfectly noticeable the crushing on the whites if you do not use 255. Look at the umbrella, it almost look as a 1 full bright umbrella. Furthermore, the machinery behind the umbrella also disappears.

If you look at the other samples, with dark imagery, you cannot watch any detail loss when using 11 or 16. The only thing you would notice is the lowering of the black level, nothing more. The lower values you are detecting are just noise, and not valid data.

Now, let's look into this in another way... Let's quote a sentence from one of your previous posts:
"In Japan, the standard black level was changed from 7.5 IRE to 0 IRE in 1985"
This also justify what's happening... This movie is from 1979, so probably they simply got the old video with the black level at 7.5 IRE, hence only the black level being flawed.
Now, let's find which value in 8 bit corresponds to 7.5 IRE:
7.5/100*256 = 19.2
So, probably you could go a little further and use 19 into the formula (gives 32) without losing any shadow detail, because it seems that below that value there is not any valid data, only noise/processing artifacts. I've tried it, and did not found any losses on shadow detail, but maybe the image is too much dark for your taste... try it for yourself and decide.

[Thunderbolt8]

ok so what you say in order to find out I basically need to try and compare values, using them and not using them in a 3dlut files and in case theres a difference, I would have to use the full spectrum (as with white) and if no difference in detail (as with black), then Im fine with 16 (or maybe 19.2 to be more accurate)? I only find it very hard to spot differences in detail in such darkness :S

is there any way I can compare those 2 different settings with 30 and 32 by switching (e.g. switching 3dlut files on the fly)? cant take screenshots with madvr yet.

To be more precise, all the process also might have messed up the gamma, so probably you will also need to perform some gamma correctionalright, since the black level thing above seems to be solved now, what about this, how to do this?

[yesgrey]

ok so what you say in order to find out I basically need to try and compare values, using them and not using them in a 3dlut files and in case theres a difference, I would have to use the full spectrum (as with white) and if no difference in detail (as with black), then Im fine with 16 (or maybe 19.2 to be more accurate)?
Right.
There is a shortcut in madVR for switching from PC->video levels and back without opening the settings dialog (Ctrl+Alt+Shift+Y).
I always use it when I want to compare 2 different 3DLUT files. It's easy to spot differences this way, in case they exist.

I just find it very hard to spot differences in detail in such darkness :S
This sentence almost answer your own question... If it's hard for you to see any detail in that area, even if exist any, why bother? Simply lower the black level and enjoy a better viewing experience. Watching those films with such a raised black level would be a lot worse, IMHO.;)

I will reply later about the gamma correction. I have to think about it, and now I'm very busy working on the next version of yCMS...

[yesgrey]

is there any way I can compare those 2 different settings with 30 and 32 by switching (e.g. switching 3dlut files on the fly)?
Yes. In addition to what I've said on my previous post about the shortcut, you just need to rename one of the files as the PC levels and the other as Video levels. When you switch a OSD label appears showing which is in use. Use exclusive fullscreen for the testing, because on windowed mode the label sometimes doesn't appear...

[yesgrey]

yCMS v1.8 released

http://yesgrey.com/ycms.html

- Added verification of measurements below 30 IRE.
- Added new parameter 'format' to Grayscale_Measurements.
- Added Yxy and XYZ support on Grayscale_Measurements.
- Removed RGB support on Grayscale_Measurements.
- Changed error messages to show the input file line number where it ocurrs.
- Added test to verify if a command has surplus parameters.
- Fixed bug on maximum number of grayscale measurements. It was max-1.


yCMS does not support RGB measurements on Grayscale_Measurements anymore. From now on should be used Yxy or XYZ formats instead.

These new formats should allow yCMS to perform color balance correction, but I did not test it. I haven't my new PC configured for using my calibration equipment, so I would like if anyone could test these two modes and report back their results.

These new formats should also allow a more accurate final gamma curve when using the Gamma_Curve command. Please confirm that too.

Enjoy.:)

[janos666]

Thanks for the new version. It looks promising. I will test it. ;)
By the way, last time I decided to calibrate my display with the VGA LUT (targets: D65 and gamma 2.35) and do only gamut correction with yCMS. The reason was the better error handling.
- If yCMS receives equal or slightly decreasing Y values for different (increasing) IREs, it simply refuses to make a 3DLUT file. It always assumes that it is a measurement error.
- If ArgyCMS/Dispcal finds that a display is broken, it tries to fix it. For example, when R=0, R=1, R=2 produces the same luminance (or it shows decrease instead of increase - oh ya, the random measurement deviation exists as well, but this is another story), the LUT will start from R>2

I think you should consider this error handling. You should let the user to decide if he believes in the measurement data and thinks that the display is broken (and he wants to compensate it) or he wants to ignore the seemingly erroneous measurements.

I am sure that in my specific case, the measurements are correct and the display is faulty. I can see it with my own eyes as well and I can measure much lower luminance levels with my instrument. This is not some extraordinary thing with digital LCD displays.

[tschi]

Hi yesgrey,
Thanks for the new version :)
here some results (colorhcfr files)
http://www.mediafire.com/?9gjodlhsvhgoxpb
no ycms /ycms with XYZ formats + gamma_curve 0 2.2 / ycms with XYZ formats default gamma curve
Is the choice of the format (XYZ / Yxy) can change the result ?

[janos666]

Is the choice of the format (XYZ / Yxy) can change the result ?

You can easily convert the values between the XYZ and xyY formats with a simple linear formula. So, no, it can't change anything or something is wrong.
You should use the XYZ format because the xyY is calculated from it. (You can avoid the conversion errors and rounding, etc.)

[tschi]

ok, thanks

[yesgrey]

here some results (colorhcfr files)?
Wow, you were fast! :)

The results look promising... Above 40/50 IRE the results are very good, but below don't. That made me think: did you use the XYZ or xyz values from HCFR? HCFR users must be very careful with this, because they should select the "XYZ", all in capitals, and not the "xyz" in lower case.

You should use the XYZ format because the xyY is calculated from it. (You can avoid the conversion errors and rounding, etc.)
I agree with you on this, but considering the above I think I would suggest to use Yxy instead. The conversion errors by the software should be insignificant, but if people use xyz instead of XYZ it would be way worse...

[tschi]

I use XYZ because colorHCFR does not export xyY to .xls :(
And it 's so easy & quick to copy and paste results in ycms text files using .xls :rolleyes:

[yesgrey]

I think you should consider this error handling. You should let the user to decide if he believes in the measurement data and thinks that the display is broken (and he wants to compensate it) or he wants to ignore the seemingly erroneous measurements.
The current method was made to call the attention to the user that he might have switched, by mistake, some values. I did not consider that it could be errors from the meter. I will add some error handling on the next version. Thanks for the suggestion.

I use XYZ because colorHCFR does not export xyY to .xls :(
No, problem. As janos666 said it's preferable to use XYZ, only be careful to not use xyz by mistake... ;)

[janos666]

Well. I didn't find any other bugs but I think the white balance is still not corrected.
I inserted 255 XYZ points (I had to skip the second erroneously decreasing value) but the white balance looks untouched. And the offset white point still degrades the precision of the gamut emulation too.

You could see my results with hardware gain corrected WP and offset WP. Now, here are my new results with offset WP and XYZ inputs:

http://img12.tar.hu/janos666/img/88329968.png http://img12.tar.hu/janos666/img/88329969.png http://img12.tar.hu/janos666/img/88329970.png
http://img12.tar.hu/janos666/img/88329965.png http://img12.tar.hu/janos666/img/88329966.png http://img12.tar.hu/janos666/img/88329967.png

Just a small notice, but can you accept XYZ format for the gamut_measurements command too?
For example, the xy format would go into the first line and an empty first line would mean XYZ format where you can copy your XYZ values in separate lines.

By the way, it is redundant to ask for white xy coordinates when you specify the 100% IRE with three dimensional format. It will ask for a slight correction which is the measurement deviation and/or format conversion related difference.

[Thunderbolt8]

any idea of what I could try about the gamma correction of that movie of mine yet? ;)

[janos666]

http://forum.doom9.org/showpost.php?p=1445481&postcount=4605
"I do not plan to do any CMS kind of stuff myself. That is yCMS' job. What you could do is go to the yCMS thread and ask yesgrey there to add an ICC -> 3dlut conversion utility."

@yesgrey: What do you think?