.ti3 Parser / .3dlut Generator for madVR + .3dlut toolset

[nand chan]

Originally Posted by alph@

nand chan,when I want to create a 3dlut with LinkICC, i have a error message 'input string was not in a correct format'' I forgot something?

Might be the locale issue again, afaik all that could be causing it is Double.Parse() which is used to generate a number from a source string (in the .ti3 files).

Originally Posted by cyberbeing

I'd be interested to test that once you release 0.13, since something was causing a large gamma shift using LinkICCGUI.exe in 0.11, even though auto-calibrate was disabled. You probably should add a checkbox to disable yCMS grayscale measurements, if I assume that was the cause and not littlecms2. At first I was thinking some significant modification would need to be done to ArgyllCMS because of all the PCS and LAB code integrated in colprof and ICC profiles, but I completely forgot that profile linking could be used to create a cLUT with src -> dst gamut mapping.

yCMS values were only being used for very dark values, the problem with LittleCMS is that it seemingly did not compensate for the screen's minimum brightness, and as such, every value that in theory would decode to a value below the screen's minimum would get cut off - so you lost a lot of shadow detail on screens with poor black levels (like mine).

My (naive) solution was to let LittleCMS handle the bright things (the vast majority of stuff), and yCMS to handle the very dark values.

Can you test if linkicc.exe -i sRGB.icc -d monitor.icc -o - | tag3dlut -i - -o output.3dlut --legacy -7 works?

Also, have you tried modes other than absolute chromatic?

I assume 'standard.icm' is your ArgyllCMS display profile and 'sRGB.icm' could be replaced with a custom 'REC709.icm' source profile?

Standard.icm is the profile for my monitor's “standard” mode (which I use to test as it is a lot wider gamut than my sRGB emulation mode, which is already at delta E 2 average)

sRGB.icm could be replaced by a BT.709 source profile, I don't know for 100% sure whether madVR encodes for the screen using an sRGB gamma curve or a pure power 2.2 curve.

I also don't recommend using the BT.709 source profile I provided earlier since I still don't quite know how to work with LittleCMS in terms of its white point shifting (it automatically normalizes all generated profiles to D50 even though I'm working with D65 everywhere, so if I want to combine them I'll have to use relative chromatic mode to fix the white point back to D65).

[cyberbeing]

Originally Posted by nand chan

Can you test if linkicc.exe -i sRGB.icc -d monitor.icc -o - | tag3dlut -i - -o output.3dlut --legacy -7 works?

Also, have you tried modes other than absolute chromatic?

I'll test that command line later. The other day I tested both absolute colorimetric and relative colorimetric and the results had identically bad gamma shift, as far as I could tell.

For future reference, how do I go about disabling yCMS gamma corrections (aka grayscale measurements) with your tools? You really shouldn't have that enabled by default, unless someone is calibrating to a pure power-curve (i.e. the only type of curve which yCMS agrees with ArgyllCMS) . Both yCMS (grayscale measurements) and littlecms2 (in mpc-hc) are seemingly unable to properly linearize dispcal's CIECAM02 scaled REC709/sRGB gamma curves, or return the curve to its pre-linearized state unmolested.

I'm crossing my fingers that the ArgyllCMS collink method you came up with, will give the results I desire.

Originally Posted by nand chan

sRGB.icm could be replaced by a BT.709 source profile, I don't know for 100% sure whether madVR encodes for the screen using an sRGB gamma curve or a pure power 2.2 curve.

madVR uses a 2.2 pure power-curve by default, not sRGB. I guess that means I need to create a profile with REC709/sRGB primaries, but has 2.2 gamma power-curve as my src link?

Originally Posted by nand chan

I also don't recommend using the BT.709 source profile I provided earlier since I still don't quite know how to work with LittleCMS in terms of its white point shifting (it automatically normalizes all generated profiles to D50 even though I'm working with D65 everywhere, so if I want to combine them I'll have to use relative chromatic mode to fix the white point back to D65).

That's exactly why I always have used Scarse for creating REC709 profiles (up to 16bit 256x256x256). It's too bad that project was abandoned in alpha status 6 years ago, and was left in state where creating non-calibrated custom ICC profiles is all it's good for. Unfortunately, there just aren't many tools that I'm aware for creating uncalibrated ICC profiles with user-defined settings.

[Audionut]

Originally Posted by cyberbeing

That's exactly why I always have used Scarse

Sorry for taking this off topic, but since Scarse has no documentation, does this CL look like what I want to be doing. Display is calibrated to a Gamma of 1.9

ipb -c i -i RGB -o RGB:1.9 -p HDTV test.icm

[nand chan]

Originally Posted by cyberbeing

I'll test that command line later. The other day I tested both absolute colorimetric and relative colorimetric and the results had identically bad gamma shift, as far as I could tell.

Okay. Would you mind also giving me a further detailing of the problem? Test case? Screenshots? It might help me either debug a problem with my tools or a potential misconfiguration.

For future reference, how do I go about disabling yCMS gamma corrections (aka grayscale measurements) with your tools? You really shouldn't have that enabled by default, unless someone is calibrating to a pure power-curve (i.e. the only type of curve which yCMS agrees with ArgyllCMS) . Both yCMS (grayscale measurements) and littlecms2 (in mpc-hc) are seemingly unable to properly linearize dispcal's CIECAM02 scaled REC709/sRGB gamma curves, or return the curve to its pre-linearized state unmolested.

You can't currently do that I'm afraid, it includes the grayscale measurements one way or another - however, what's the point in using yCMS if you leave out the grayscale measurements? At that point it's just a simple gamut mapping, which can be done with EncodePrimaries() in LutScript or the tools in ArgyllCMS.

I'm crossing my fingers that the ArgyllCMS collink method you came up with, will give the results I desire.

If it does not then it is either an issue with your profiles, ArgyllCMS or madVR - all my program does with the above described method is interpolate the Argyll values (using LittleCMS).

madVR uses a 2.2 pure power-curve by default, not sRGB. I guess that means I need to create a profile with REC709/sRGB primaries, but has 2.2 gamma power-curve as my src link?

Yes, basically. Actually, if you want to go a step further, what you can do is input a profile that has the same primaries as your display - madVR does gamut mapping to match the .3dlut's input, after all. I'm going to guess it's not as advanced as ArgyllCMS's gamut mapping though. So yeah, basically, just use a BT.709 primaries 2.2 power curve input profile, because that's exactly what madVR spits into the .3dlut.

That's exactly why I always have used Scarse for creating REC709 profiles (up to 16bit 256x256x256). It's too bad that project was abandoned in alpha status 6 years ago, and was left in state where creating non-calibrated custom ICC profiles is all it's good for. Unfortunately, there just aren't many tools that I'm aware for creating uncalibrated ICC profiles with user-defined settings.

Okay, good to know. I'll go check it out.

[alph@]

Originally Posted by nand chan

Might be the locale issue again, afaik all that could be causing it is Double.Parse() which is used to generate a number from a source string (in the .ti3 files)..

ok,it's work now,i have solved the probleme -panels of configuration -region and language-formats-additional parameters-in 'decimal symbol' replace the comma ',' by a point '.', Now I can test linkICC.

[Graeme Gill]

Originally Posted by cyberbeing

It has now been over 2 years since people began using 3dluts for color correction on their GPUs with madVR. All nand-chan is doing is making tools to ease the process of 3dlut creation and modification. Previously people were pretty much restricted to manually copying and pasting measurements from TI3 files or tools like ColorHCFR and feeding them to yesgrey's tool yCMS to create a 3dlut for use with madshi's video renderer, madVR.

From what I can see of it, it's all a bit weird - a very round about, cumbersome and inflexible approach to dealing with video color management, and an awful lot of re-inventing the wheel.

Originally Posted by cyberbeing

Recently madshi and yesgrey have added support for 6bit (64x64x64) and 7bit (128x128x128) 3DLUTs where madVR uses trilinear interpolation to generate the missing values, in addition to the full 8bit (256x256x256) lut. I personally have been using a 6bit 3DLUT for awhile, the results are very good, but on average I've measured a yCMS created 6bit 3DLUT having 0.5 worse dE values than a full 8bit 3DLUT with my i1pro rev. D. The difference is indeed nearly invisible (gamma correction aside), but with modern GPUs $80+ having zero performance issues with the full 16.7million color 3DLUT it's more of case of why not? The more significant issue with the current implementation is not gamut correction with 6bit 3DLUT, but rather gamma correction where the difference between 64 points and 256 points (non-smoothed) can be extreme when attempting to represent non-power-curves like REC709 or sRGB in the lower tones.

You simply shouldn't be using the 3D tables to do the per channel curves. That's why any ICC device link worth it's salt has per channel curves as well as the 3D table. For the 3D table to be a reasonable prospect, it needs to be in a colorspace that best leveraged its advantages, while minimises its limitations. So typically the color mapping would be designed to be in a perceptually
linear space, with the input curves (or even more sophisticated transforms if you use ICC V4) serving to transform between the device space and the 3D mapping space.

Originally Posted by cyberbeing

That said, there is a lot of room for improvement in 3dlut creation. All yCMS created 3dluts take into account are white point, primary colors (100% Red/Green/Blue), and IRE grayscale, making the resulting correction extremely basic and likely worse than an Argyll created Matrix/Shaper ICC profile. This no doubt hurts the potential accuracy of a 6bit 3DLUT.

One of the things that I see as being rather suspect, is the lack of clear distinction between a device profile (which relates the device color values like RGB to device independent color values such as XYZ or L*a*b*), and a device link (which transforms from one device space to another, such as REC709 to display RGB).

Originally Posted by cyberbeing

As I've been saying for the past year or so on Doom9, what would be amazing is if ArgyllCMS colprof could be modified to create a madVR-compliant 3DLUT instead of a ICC profile during a normal ArgyllCMS calibration workflow. Graeme would modifying your existing cLUT code to output a 6bit(?) 3DLUT interest you personally at all (since nobody else has been interested...)? What we are really lacking is a high quality CMS like ArgyllCMS that takes a full set of measurement patches into account so we can actually get high quality results with less data points in the 3DLUT. The work nand chan has been doing is a step in the right direction, but an official ArgyllCMS solution which meets your quality standards and approval would be so much better. madshi as said a few times already that he'd love to be able to make 6bit 3DLUTs the default in madVR if the quality improved.

This doesn't seem the right approach at all to me. I'd strongly suggest instead implementing direct support for ICC device links in the video display path, taking as a model how ArgyllCMS's cctiff works. You don't even have to do any work as a first cut to see if the quality of transform is of the right order - take a single frame and use the whole of the Argyll toolchain to make a video standard to device RGB transform, and apply it to the frame using cctiff. I'm quite prepared to investigate any quality issues with this (as I've done in the past). This approach leverages established color management tools, and minimises a lot of duplication and rediscovery of how not to do things :-)

[nand chan]

Originally Posted by Graeme Gill

From what I can see of it, it's all a bit weird - a very round about, cumbersome and inflexible approach to dealing with video color management, and an awful lot of re-inventing the wheel.

Re-inventing the wheel is not always a bad thing - if the existing wheels become too large or complex, it can make sense to start a “fresh”, application-specific, limited approach.

You simply shouldn't be using the 3D tables to do the per channel curves. That's why any ICC device link worth it's salt has per channel curves as well as the 3D table. For the 3D table to be a reasonable prospect, it needs to be in a colorspace that best leveraged its advantages, while minimises its limitations. So typically the color mapping would be designed to be in a perceptually
linear space, with the input curves (or even more sophisticated transforms if you use ICC V4) serving to transform between the device space and the 3D mapping space.

Or you increase the bit depth to compensate.

One of the things that I see as being rather suspect, is the lack of clear distinction between a device profile (which relates the device color values like RGB to device independent color values such as XYZ or L*a*b*), and a device link (which transforms from one device space to another, such as REC709 to display RGB).

I personally think one should draw a line between data information and usage information. So, for example, while it makes sense to store, say, the input primaries, or the color encoding, or bit depth - I don't think it makes sense to store things such as which display it's used for, or what the applied transformations are.

Such things, in my opinion, belong in the hands of the usage implementation, and not the data format. If different implementations, for example a monitor calibration system, want to store additional meta-data (which file means what, which display they're associated with), they should introduce their own structures.

Encoding such trivial “information” into the format itself is, in my opinion, more of a waste of space than additional bit depth.

This doesn't seem the right approach at all to me. I'd strongly suggest instead implementing direct support for ICC device links in the video display path, taking as a model how ArgyllCMS's cctiff works. You don't even have to do any work as a first cut to see if the quality of transform is of the right order - take a single frame and use the whole of the Argyll toolchain to make a video standard to device RGB transform, and apply it to the frame using cctiff. I'm quite prepared to investigate any quality issues with this (as I've done in the past). This approach leverages established color management tools, and minimises a lot of duplication and rediscovery of how not to do things :-)

Don't quote me on it but I believe the problem is that yesgrey, madshi, etc. simply don't have enough time to create a full implementation of the ICC profile, and they refuse to use existing libraries such as LittleCMS or ArgyllCMS to do the job, as that would not only take the fun out of it but also deny a learning experience, credit/license and ultimately defeat the point.

So they decided to create their own, stripped down format instead, and it's up to programmers such as me who have both time and no obligations to reject third party code, to create the middleware / bridgeware.

[nand chan]

Version 0.13
This is a bug fix release, mainly to fix a blatant issue in the CalFile parser that was introduced last version. It's a shame I didn't notice it sooner.

Download link (v0.13)

Changelog:

Version 0.13:
* ICC profiles without added primaries information no longer get rejected (the color space gets set as “unknown”)
* Calibration files no longer have swapped primaries, bug introduced in version 0.12
  This fix affects: CalFile(), applycal, imagecal and --calibrate in make3dlut

[madshi]

Originally Posted by Graeme Gill

From what I can see of it, it's all a bit weird - a very round about, cumbersome and inflexible approach to dealing with video color management, and an awful lot of re-inventing the wheel.

The basic idea is that having a very big 3dlut allows us to use as complex a calibration math as we want, without affecting rendering performance. Furthermore it allows us to strictly separate calibration programming from video rendering programming. Personally, I've more than enough on my hands just with the video rendering, I don't really want to get into calibration programming too much. So I very much the approach to just use a 3dlut in my renderer and rely on external tools to do the math stuff in offline 3dlut creation tools.

Originally Posted by Graeme Gill

I'd strongly suggest instead implementing direct support for ICC device links in the video display path, taking as a model how ArgyllCMS's cctiff works.

I'd have to write all that as GPU pixel shaders, and I doubt it would run fast enough on any current Intel GPU for 1080p60. Especially considering that I want to use the GPU shaders for other things like chroma upsampling, scaling, sharpening etc etc, too.

And what happens if we aim for a "perfect" calibration? E.g. if we do a couple hundred (or even thousand) measurements and try to achieve perfect correction based on those measurements? Considering all the funny things like too small/big display gamuts, out of gamut source data, perception based stuff etc, the math to achieve "perfect" calibration could quickly become very complicated. Too complicated to run it in realtime on current Intel GPUs at least, I think.

[nand chan]

Originally Posted by madshi

The basic idea is that having a very big 3dlut allows us to use as complex a calibration math as we want, without affecting rendering performance. Furthermore it allows us to strictly separate calibration programming from video rendering programming. Personally, I've more than enough on my hands just with the video rendering, I don't really want to get into calibration programming too much. So I very much the approach to just use a 3dlut in my renderer and rely on external tools to do the math stuff in offline 3dlut creation tools.

But that doesn't conflict with what he suggests - doing the calculations offline as you mentioned and storing the result in a small RGB -> RGB link profile (ICC), the same as a .3dlut would. His only gripe is with you re-inventing the format when the ICC profiles already exist.

I'm not quite sure I agree with his philosophy though, since implementing such a huge and complex standard as ICC profiles takes a lot more work than re-inventing some simple format.

I'd have to write all that as GPU pixel shaders, and I doubt it would run fast enough on any current Intel GPU for 1080p60. Especially considering that I want to use the GPU shaders for other things like chroma upsampling, scaling, sharpening etc etc, too.

And what happens if we aim for a "perfect" calibration? E.g. if we do a couple hundred (or even thousand) measurements and try to achieve perfect correction based on those measurements? Considering all the funny things like too small/big display gamuts, out of gamut source data, perception based stuff etc, the math to achieve "perfect" calibration could quickly become very complicated. Too complicated to run it in realtime on current Intel GPUs at least, I think.

It would be the same as, say, a 6-bit .3dlut. If those run on Intel GPUs in realtime, then an ICC link profile would run too.

[madshi]

So an ICC link profile can basically contain a 6bit input 16bit output 3dlut? I have no knowledge about ICC at all, so I don't really have any clue.

[nand chan]

Originally Posted by madshi

So an ICC link profile can basically contain a 6bit input 16bit output 3dlut? I have no knowledge about ICC at all, so I don't really have any clue.

If I've interpreted Graeme Gill's posts correctly, yes.

Ps. Another disadvantage of using .icc profiles is that it might confuse people who are attempting to use the more common /device/ profiles instead of a link profile - since they all have the same extension they might not know the difference, or struggle in figuring out how to create a link profile before settling for some inferior solution.

By forcing .3dlut, we have people actually being forced to search for .3dlut programs in particular, which would be our implementations - and there's no confusion between ICC profiles.

[IceB]

Hi.
You are doing a great job - keep on !

I am trying to generate the 3DLUT file using the LinkICCGUI and getting a following error on Win 32 when i press generate.
"Error: Could not find a part of the path
'C:\Users\#MyUsername#\AppData\Roaming\TI3Parser\dfdd367b-09e8-4086-96b8-373cd55f9f4d'."
There is no TI3Parser folder at AppData\Roaming at all.
My extracted 0.13 version is a C:\ .
Latest .NET Frameworks installed.
I choose both ICM and TI3 files created with dispcalGUI as linked on the guide.
Is there something I miss ?

Thanks for advance.

[nand chan]

Originally Posted by IceB

Hi.
You are doing a great job - keep on !

I am trying to generate the 3DLUT file using the LinkICCGUI and getting a following error on Win 32 when i press generate.
"Error: Could not find a part of the path
'C:\Users\#MyUsername#\AppData\Roaming\TI3Parser\dfdd367b-09e8-4086-96b8-373cd55f9f4d'."
There is no TI3Parser folder at AppData\Roaming at all.
My extracted 0.13 version is a C:\ .
Latest .NET Frameworks installed.
I choose both ICM and TI3 files created with dispcalGUI as linked on the guide.
Is there something I miss ?

Thanks for advance.

Hmm, it might be a permissions issue? Are you on a restricted account maybe?

(It would surprise me though since even restricted accounts are supposed to have access to their appdata folders)

[IceB]

Originally Posted by nand chan

Hmm, it might be a permissions issue? Are you on a restricted account maybe?

(It would surprise me though since even restricted accounts are supposed to have access to their appdata folders)

It is strange.
I have tried it on 2 computers an the laptop - all with admin account returned the same exact error.
I also tried to copy the TI3Parser folder to the Appdata\Roaming folder - then pressing the "generate" button returns different error:
"Error: The system can no find the file specified "

What file does the app looking for and why at the first place it needs to look at AppData\Roaming\TI3Parser while this folder is not there.

Is there any installation process I should do or should I use the new TI3Parser as is after the extraction.

I tried it with different profiles and different machines - all of them return the same error - i must be missing something.

[nand chan]

Originally Posted by IceB

It is strange.
I have tried it on 2 computers an the laptop - all with admin account returned the same exact error.
I also tried to copy the TI3Parser folder to the Appdata\Roaming folder - then pressing the "generate" button returns different error:
"Error: The system can no find the file specified "

What file does the app looking for and why at the first place it needs to look at AppData\Roaming\TI3Parser while this folder is not there.

Is there any installation process I should do or should I use the new TI3Parser as is after the extraction.

I tried it with different profiles and different machines - all of them return the same error - i must be missing something.

Come to think of it, try running “updateycms.exe” (you don't have to run this from the console, just double clicking it should be fine).

I just realized that I didn't add the dialog to automatically download yCMS to LinkICCGUI as well, I only added it to TI3ParserGUI.

[IceB]

Well done !

The “updateycms.exe” trick worked for my laptop.
I'll try to manage it on the PC later.

Thank you.

[Audionut]

Originally Posted by nand chan

Come to think of it, try running “updateycms.exe”

Can you add an option to select ycms from a user folder.

Seems a bit redundant needing to have ycms in a specific folder.

Cheers.

[nand chan]

Originally Posted by Audionut

Can you add an option to select ycms from a user folder.

Seems a bit redundant needing to have ycms in a specific folder.

Cheers.

I'll add something like it for ver 0.14. I'm also moving the temporary files into, well, %temp%.

Edit: Where do you think I should save the setting for this? 1. A registry key, 2. A config file in appdata or 3. A local configuration file in the \bin folder?

Advantage of 3. is portability, but 1./2. is invisibility and the ability to retain settings / have settings effect every copy of the program.

[Audionut]

Originally Posted by nand chan

Where do you think I should save the setting for this?

Personally, I would prefer 2 as it makes it easy to use the same config when upgrading.

I prefer 2 over 1 for this, as I think there shouldn't been things added to the registry unless really needed, although I not really that bothered as I don't install 9 million programs.

Also some people get really anal about their registries.

3 would work fine if you switch to an installer which can upgrade over a previous release.

edit: If you switch to an installer, you should provide an option to select the location of YCMS during install, or the option to download.
I'd personally be happy to have the installer copy my download of YCMS to it's own bin folder (in case I delete the YCMS folder for instance without thinking).