.ti3 Parser / .3dlut Generator for madVR + .3dlut toolset

[nand chan]

v0.12 release

Not all of the things I described above made it into the release, and I spent quite some time reversing some of the changes. Either way, the release should be largely working now, but I haven't tested every program - only the basics.

I also decided to get rid of ScaleAbs() and renamed ScaleRelative to simply Scale. Most importantly, however, this release introduces gamma and gamut functions.

Download link: http://www.mediafire.com/?l3toc3v20l03g6v

Changelog:
Version 0.12:
* Fixed a bug in the merge3dlut description
~ Internal ColorTriple<T> structure re-ordered to have the same channel order for BGR and YCbCr, most tools are now channel
agnostic unless they need to be otherwise. This might cause some issues where the channels end up swapped - please report
them asap.
+ New LutScript functions: GammaDec(Number), GammaEnc(Number), ColorDec(Primaries), ColorEnc(Primaries) - these add primitive
gamma and gamut adjustment. The proper procedure is to first decode gamma, then colors, then encode colors and encode gamma.
“Primaries” can be either of the form BT709 or it can be 8 separate numbers, referring to the x/y values of the primaries'
Red, Green, Blue and White points - in that order. For example:
!Pixel({
GammaDec(2.2) # This converts from weighted R/G/B to linear r/g/b
ColorDec(BT709) # This converts from linear r/g/b to absolute X/Y/Z
ColorEnc(0.682, 0.304, 0.1975, 0.7036, 0.1457, 0.0486, 0.313, 0.3308) # This converts from X/Y/Z to some other linear r/g/b
GammeEnc(2.35) # This converts from linear r/g/b back to weighted R/G/B, here scaled slightly differently
})
+ New LutScript function: Scale(Number, Number) - this scales to a given value range, for example Scale(0.2, 0.8)
It is important to note here that this is relative to the input range - the above function in a Limited range .3dl2 will scale
to something like 59.8 - 198.4.
+ New LutScript function: Clamp(Number, Number) - this clamps output to a certain value range. Useful when you get artifacts
in madVR when dealing with BTB/WTW levels in Limited range .3dluts (my own code does not exhibit this behavior, so it is not
needed for usage with progams like imagecal.exe)
* LinkIccGUI no longer locks up the interface while it is processing

[Graeme Gill]

I think the point of Graeme Gill (apparently it's the first post on Doom9 from the author of ArgyllCMS... welcome !) is that you don't need such precision when storing the filter on e.g. your hard disk, and that the actual, 256x256x256 3dlut should be generated on the time it is used (i.e. beginning of playback in the case of madVR).

Hi,
the point is that there is no reason at all to create a 256x256x256 table. A device->device transform with resolution somewhere between 17 and 50 as a maximum (when combined with higher res. per channel curves) is all that can be justified given the fairly coarse characterisation of the actual output device. It has additional advantages in scaling to greater than 8 bit inputs smoothly. Modern processors have lots of compute speed, but relatively limited memory bandwidth, so trading computation for smaller table size is going to give better performance. A GPU has the advantage of having built in hardware that does the core interpolation - the trilinear 3D texture lookup instruction. See for instance: GPU Gems 2, page 381: "Using Lookup Table to Accelerate Color Transformations", The OpenEXR exrdisplay application, and CompICC:
<http://sourceforge.net/apps/mediawiki/compicc/index.php?title=Main_Page>

[Graeme Gill]

Is ArgyllCMS capable of creating such large input profiles? eg. 256x256x256



Yes, but you would be crazy to do so, and nothing would be gained by it. Why generate 16 million points from a few thousand measurements ?


Because the lower bit-depth you clamp your result to, the less precise it gets when applying multiple transformations. There's a reason every single decent video filter outputs a higher bit depth then it was fed, usually at least twice the bit-depth of the input. Remember, once you destroy information, it's gone. Why not preserve it if you can?


Sounds like a severe case of premature optimisation to me, to go for 2x10^16 times the precision of the final result. A factor of 2 - 10 is typically sufficient, but the final proof is in the viewing. If you can see it in a double blind test amongst all the natural image noise, JPEG quantisation artefacts and motion estimation artefacts, then by all means use a higher precision - say 16 bits.


Looking up a table index is quite fast, far faster than 64-bit square root calculations etc.


Why would you want to do a square root ? That's not likely to be fast....


So you're suggesting re-interpolating the values 49766400 times per second, giving you the same result? Why not interpolate them once and then do nothing, greatly reducing playback stress?


Yes, because that optimises the memory bandwidth, which is one of the main bottlenecks on modern processors. You also get the advantage of scaling to greater than 8 bit input precision.



Chances are the calculations you are performing are less in magnitude. Proper gamut mapping should be done with polar coordinates and converting to/from those is quite expensive.



Yes, but none of that has anything to do with the final transformation. Argyll gamut mapping is actually done in CIECAM02 space, quite a complex transformation in both directions. But the end result of a device -> device link is a cLUT stored in an ICC device link, something that can be executed very fast using just integer CPU code, such as Argyll's IMDI, lcms or a GPU texture lookup.

[cyberbeing]

Yes, but you would be crazy to do so, and nothing would be gained by it. Why generate 16 million points from a few thousand measurements ?
Graeme, you may want to direct your issues with the 3dlut format at yesgrey (yCMS) (http://yesgrey.com/ycms.html) and tritical (ddcc/t3dlut) (http://web.missouri.edu/~kes25c/), since they are the ones who originally created the 3dlut format. madshi in that same time period back in 2009 created the video renderer madVR (http://forum.doom9.org/showthread.php?t=146228) to use a 3dlut for gamut correction.

It has now been over 2 years since people began using 3dluts for color correction on their GPUs with madVR. All nand-chan is doing is making tools to ease the process of 3dlut creation and modification. Previously people were pretty much restricted to manually copying and pasting measurements from TI3 files or tools like ColorHCFR and feeding them to yesgrey's tool yCMS to create a 3dlut for use with madshi's video renderer, madVR.

Recently madshi and yesgrey have added support for 6bit (64x64x64) and 7bit (128x128x128) 3DLUTs where madVR uses trilinear interpolation to generate the missing values, in addition to the full 8bit (256x256x256) lut. I personally have been using a 6bit 3DLUT for awhile, the results are very good, but on average I've measured a yCMS created 6bit 3DLUT having 0.5 worse dE values than a full 8bit 3DLUT with my i1pro rev. D. The difference is indeed nearly invisible (gamma correction aside), but with modern GPUs $80+ having zero performance issues with the full 16.7million color 3DLUT it's more of case of why not? The more significant issue with the current implementation is not gamut correction with 6bit 3DLUT, but rather gamma correction where the difference between 64 points and 256 points (non-smoothed) can be extreme when attempting to represent non-power-curves like REC709 or sRGB in the lower tones.

That said, there is a lot of room for improvement in 3dlut creation. All yCMS created 3dluts take into account are white point, primary colors (100% Red/Green/Blue), and IRE grayscale, making the resulting correction extremely basic and likely worse than an Argyll created Matrix/Shaper ICC profile. This no doubt hurts the potential accuracy of a 6bit 3DLUT.

As I've been saying for the past year or so on Doom9, what would be amazing is if ArgyllCMS colprof could be modified to create a madVR-compliant 3DLUT instead of a ICC profile during a normal ArgyllCMS calibration workflow. Graeme would modifying your existing cLUT code to output a 6bit(?) 3DLUT interest you personally at all (since nobody else has been interested...)? What we are really lacking is a high quality CMS like ArgyllCMS that takes a full set of measurement patches into account so we can actually get high quality results with less data points in the 3DLUT. The work nand chan has been doing is a step in the right direction, but an official ArgyllCMS solution which meets your quality standards and approval would be so much better. madshi as said a few times already that he'd love to be able to make 6bit 3DLUTs the default in madVR if the quality improved.

[Audionut]

Graeme, you may want to direct your issues with the 3dlut format at yesgrey (yCMS) (http://yesgrey.com/ycms.html) and tritical (ddcc/t3dlut) (http://web.missouri.edu/~kes25c/), since they are the ones who originally created the 3dlut format.

It's probably best to keep it here actually, as yesgrey won't be available for a few months and nand chan looks set to improve the 3dlut format.

All nand-chan is doing is making tools to ease the process of 3dlut creation and modification.

As above, it has since been expanded to include a new 3dlut format. (Mainly due to lack of time on yesgreys part atm afaik).

As I've been saying for the past year or so on Doom9, what would be amazing is if ArgyllCMS colprof could be modified to create a madVR-compliant 3DLUT instead of a ICC profile during a normal ArgyllCMS calibration workflow.

That would be nice. I would be keen to donate again if this were to happen.

[cyberbeing]

It's probably best to keep it here actually, as yesgrey won't be available for a few months and nand chan looks set to improve the 3dlut format.
I probably should have stated that differently. If Graeme wants more information on why the 3DLUT format was originally created instead of using existing ICC profile implementation, he would need to talk to tritical or yesgrey, as both I'm sure have a stronger opinion of the merits of a 3DLUT vs an ICC profile compared to nand chan who has only recently taken on the task of expanding the format.

It's very true that yesgrey is unavailable for the time being since he is in the process of moving to another country, but contacting tritical may not be a bad idea if Graeme desires some information on the logic behind the format's creation. Considering tritical's background as a Ph.D in Math as well as a college professor, I'm sure he and Graeme could have a lively discussion on the technical merits of 3DLUT vs ICC if they so desired, which may be much more Graeme's cup of tea than debating things with nand chan or even yesgrey. ;)

Now if Graeme wants to discuss changes and future of the 3dlut format, rather than why it exists as it is now, nand chan (and yesgrey when he returns) are the proper people to direct suggestions, complaints, and constructive criticism.

Either way, I'm happy to see Graeme has finally made his way over to Doom9 to provide his valuable input on all things CMS related. :D

[nand chan]

Yes, but you would be crazy to do so, and nothing would be gained by it. Why generate 16 million points from a few thousand measurements ?

Sounds like a severe case of premature optimisation to me, to go for 2x10^16 times the precision of the final result. A factor of 2 - 10 is typically sufficient, but the final proof is in the viewing. If you can see it in a double blind test amongst all the natural image noise, JPEG quantisation artefacts and motion estimation artefacts, then by all means use a higher precision - say 16 bits.

Why use less when you can use more? That's my line of thinking, anyhow.

Why would you want to do a square root ? That's not likely to be fast....

LittleCMS uses polar coordinates to desaturate colors, the chromaticity of which is calculated as the square root of the sum of squares of the a and b values (from L*ab).

Yes, because that optimises the memory bandwidth, which is one of the main bottlenecks on modern processors. You also get the advantage of scaling to greater than 8 bit input precision.

Modern graphics cards are more than optimized for fast random memory access - this is just as much not a limiting factor as calculation speed, so arguing over the two is sort of pointless.

Yes, but none of that has anything to do with the final transformation. Argyll gamut mapping is actually done in CIECAM02 space, quite a complex transformation in both directions. But the end result of a device -> device link is a cLUT stored in an ICC device link, something that can be executed very fast using just integer CPU code, such as Argyll's IMDI, lcms or a GPU texture lookup.

I see what you mean by this. Since the result is (arguably) the same (as long as LUT precision isn't reduced below measurement precision), it boils down to “computation power” vs “storage size”.

Either way, let me clear up some potential misunderstandings here: I'm not claiming that the benefits of 256x256x256 LUTs are tangible, I'm just providing reasons to prove that a smaller LUT would not be better.

The matter of fact is, madVR uses full .3dluts and it's not within my power to change that - if you can provide a good reason for madshi to change that then by all means do so and I will adapt my toolset accordingly. But, for the time being, all I'm interested in doing is providing the tools to better work with madVR.

Graeme, you may want to direct your issues with the 3dlut format at yesgrey (yCMS) (http://yesgrey.com/ycms.html) and tritical (ddcc/t3dlut) (http://web.missouri.edu/~kes25c/), since they are the ones who originally created the 3dlut format. madshi in that same time period back in 2009 created the video renderer madVR (http://forum.doom9.org/showthread.php?t=146228) to use a 3dlut for gamut correction.

I would like to mention that there is a separate thread to discuss the .3dlut topic. If possible, I would like to keep this thread related only to the tool which I am implementing (eg. bug reports, issues, questions)

Link to .3dlut discussion topic: http://forum.doom9.org/showthread.php?t=139389&page=36

Recently madshi and yesgrey have added support for 6bit (64x64x64) and 7bit (128x128x128) 3DLUTs where madVR uses trilinear interpolation to generate the missing values, in addition to the full 8bit (256x256x256) lut.

I'd just like to mention that madVR also interpolates the 8bit lut when using 10bit video.

That said, there is a lot of room for improvement in 3dlut creation. All yCMS created 3dluts take into account are white point, primary colors (100% Red/Green/Blue), and IRE grayscale, making the resulting correction extremely basic and likely worse than an Argyll created Matrix/Shaper ICC profile. This no doubt hurts the potential accuracy of a 6bit 3DLUT.

As I've been saying for the past year or so on Doom9, what would be amazing is if ArgyllCMS colprof could be modified to create a madVR-compliant 3DLUT instead of a ICC profile during a normal ArgyllCMS calibration workflow.

My program is capable of doing this. The IccSingle() function loads a single .icc profile and transforms its input accordingly (eg. for a source profile, it would input RGB and output XYZ*, or for a link profile, it would transform RGB to RGB).

So you can generate your link profile using colprof + collink and then generate an 8-bit .3dlut using lutscript, which uses LittleCMS internally.

* Currently, only the .3dl2 file format is capable of XYZ output - .3dlut (v1) must have RGB input and RGB output and my toolset will complain to you otherwise.

(I'll go test this myself now so I can back up my claims)

[nand chan]

And, the results are in.

Step 1: Profile generated using collink.exe sRGB.icm standard.icm link.icm -i a

(I fixed a bug here in v0.13 that would cause my tools to reject any ICC profiles that don't have the tags for r/g/bXYZ + wtpt)

Step 2: I ran the following lutscript:

!Filetype(3DLUT)
!Input_Primaries(BT709)
!Input_Depth(6, 6, 6)

!Pixel(IccSingle("link.icc"))
For this .3dlut, I had the standard.icm's CLUT tables loaded to my graphics card

And then the same with !Input_Depth(8, 8, 8)

Here is a grayscale comparison: http://screenshotcomparison.com/comparison/79911
Here is a color image comparison: http://screenshotcomparison.com/comparison/79912 (one of my favorite scenes for testing gamut mapping because of the green tones)

Step 3: In order to compare the quality of the actual gamut mapping to that of LittleCMS, I have modified the script slightly:

!Filetype(3DLUT)
!Input_Primaries(BT709)
!Input_Depth(6, 6, 6)

!Pixel({
IccSingle("link.icc")
CalFile("standard.ti3")
})
For this .3dlut, I have my video card's gamma table reset

Here is a comparison image between the .3dlut generated using the above script, and a previous, 8-bit .3dlut that I had generated using LittleCMS and yCMS:

http://screenshotcomparison.com/comparison/79913

I believe the results can be said to be satisfactory.

[alph@]

nand chan,when I want to create a 3dlut with LinkICC, i have a error message 'input string was not in a correct format'' I forgot something?

[cyberbeing]

And, the results are in.

Step 1: Profile generated using collink.exe sRGB.icm standard.icm link.icm -i a

(I fixed a bug here in v0.13 that would cause my tools to reject any ICC profiles that don't have the tags for r/g/bXYZ + wtpt)

I'd be interested to test that once you release 0.13, since something was causing a large gamma shift using LinkICCGUI.exe in 0.11, even though auto-calibrate was disabled. You probably should add a checkbox to disable yCMS grayscale measurements, if I assume that was the cause and not littlecms2. At first I was thinking some significant modification would need to be done to ArgyllCMS because of all the PCS and LAB code integrated in colprof and ICC profiles, but I completely forgot that profile linking could be used to create a cLUT with src -> dst gamut mapping.

I assume 'standard.icm' is your ArgyllCMS display profile and 'sRGB.icm' could be replaced with a custom 'REC709.icm' source profile?

It's amazing how quickly you put together all these tools in less than a month. I look away for a couple weeks and the number of tools and functions double. :scared:

Keep up the good work, at some point I hope you take a break to write up some documentation w/ example uses and workflows for all these tools.

[nand chan]

nand chan,when I want to create a 3dlut with LinkICC, i have a error message 'input string was not in a correct format'' I forgot something?

Might be the locale issue again, afaik all that could be causing it is Double.Parse() which is used to generate a number from a source string (in the .ti3 files).

I'd be interested to test that once you release 0.13, since something was causing a large gamma shift using LinkICCGUI.exe in 0.11, even though auto-calibrate was disabled. You probably should add a checkbox to disable yCMS grayscale measurements, if I assume that was the cause and not littlecms2. At first I was thinking some significant modification would need to be done to ArgyllCMS because of all the PCS and LAB code integrated in colprof and ICC profiles, but I completely forgot that profile linking could be used to create a cLUT with src -> dst gamut mapping.

yCMS values were only being used for very dark values, the problem with LittleCMS is that it seemingly did not compensate for the screen's minimum brightness, and as such, every value that in theory would decode to a value below the screen's minimum would get cut off - so you lost a lot of shadow detail on screens with poor black levels (like mine).

My (naive) solution was to let LittleCMS handle the bright things (the vast majority of stuff), and yCMS to handle the very dark values.

Can you test if linkicc.exe -i sRGB.icc -d monitor.icc -o - | tag3dlut -i - -o output.3dlut --legacy -7 works?

Also, have you tried modes other than absolute chromatic?

I assume 'standard.icm' is your ArgyllCMS display profile and 'sRGB.icm' could be replaced with a custom 'REC709.icm' source profile?

Standard.icm is the profile for my monitor's “standard” mode (which I use to test as it is a lot wider gamut than my sRGB emulation mode, which is already at delta E 2 average)

sRGB.icm could be replaced by a BT.709 source profile, I don't know for 100% sure whether madVR encodes for the screen using an sRGB gamma curve or a pure power 2.2 curve.

I also don't recommend using the BT.709 source profile I provided earlier since I still don't quite know how to work with LittleCMS in terms of its white point shifting (it automatically normalizes all generated profiles to D50 even though I'm working with D65 everywhere, so if I want to combine them I'll have to use relative chromatic mode to fix the white point back to D65).

[cyberbeing]

Can you test if linkicc.exe -i sRGB.icc -d monitor.icc -o - | tag3dlut -i - -o output.3dlut --legacy -7 works?

Also, have you tried modes other than absolute chromatic?

I'll test that command line later. The other day I tested both absolute colorimetric and relative colorimetric and the results had identically bad gamma shift, as far as I could tell.

For future reference, how do I go about disabling yCMS gamma corrections (aka grayscale measurements) with your tools? You really shouldn't have that enabled by default, unless someone is calibrating to a pure power-curve (i.e. the only type of curve which yCMS agrees with ArgyllCMS) . Both yCMS (grayscale measurements) and littlecms2 (in mpc-hc) are seemingly unable to properly linearize dispcal's CIECAM02 scaled REC709/sRGB gamma curves, or return the curve to its pre-linearized state unmolested.

I'm crossing my fingers that the ArgyllCMS collink method you came up with, will give the results I desire.

sRGB.icm could be replaced by a BT.709 source profile, I don't know for 100% sure whether madVR encodes for the screen using an sRGB gamma curve or a pure power 2.2 curve.
madVR uses a 2.2 pure power-curve by default, not sRGB. I guess that means I need to create a profile with REC709/sRGB primaries, but has 2.2 gamma power-curve as my src link?

I also don't recommend using the BT.709 source profile I provided earlier since I still don't quite know how to work with LittleCMS in terms of its white point shifting (it automatically normalizes all generated profiles to D50 even though I'm working with D65 everywhere, so if I want to combine them I'll have to use relative chromatic mode to fix the white point back to D65).

That's exactly why I always have used Scarse for creating REC709 profiles (up to 16bit 256x256x256). It's too bad that project was abandoned in alpha status 6 years ago, and was left in state where creating non-calibrated custom ICC profiles is all it's good for. Unfortunately, there just aren't many tools that I'm aware for creating uncalibrated ICC profiles with user-defined settings.

[Audionut]

That's exactly why I always have used Scarse

Sorry for taking this off topic, but since Scarse has no documentation, does this CL look like what I want to be doing. Display is calibrated to a Gamma of 1.9

ipb -c i -i RGB -o RGB:1.9 -p HDTV test.icm

[nand chan]

I'll test that command line later. The other day I tested both absolute colorimetric and relative colorimetric and the results had identically bad gamma shift, as far as I could tell.

Okay. Would you mind also giving me a further detailing of the problem? Test case? Screenshots? It might help me either debug a problem with my tools or a potential misconfiguration.

For future reference, how do I go about disabling yCMS gamma corrections (aka grayscale measurements) with your tools? You really shouldn't have that enabled by default, unless someone is calibrating to a pure power-curve (i.e. the only type of curve which yCMS agrees with ArgyllCMS) . Both yCMS (grayscale measurements) and littlecms2 (in mpc-hc) are seemingly unable to properly linearize dispcal's CIECAM02 scaled REC709/sRGB gamma curves, or return the curve to its pre-linearized state unmolested.

You can't currently do that I'm afraid, it includes the grayscale measurements one way or another - however, what's the point in using yCMS if you leave out the grayscale measurements? At that point it's just a simple gamut mapping, which can be done with EncodePrimaries() in LutScript or the tools in ArgyllCMS.

I'm crossing my fingers that the ArgyllCMS collink method you came up with, will give the results I desire.

If it does not then it is either an issue with your profiles, ArgyllCMS or madVR - all my program does with the above described method is interpolate the Argyll values (using LittleCMS).

madVR uses a 2.2 pure power-curve by default, not sRGB. I guess that means I need to create a profile with REC709/sRGB primaries, but has 2.2 gamma power-curve as my src link?

Yes, basically. Actually, if you want to go a step further, what you can do is input a profile that has the same primaries as your display - madVR does gamut mapping to match the .3dlut's input, after all. I'm going to guess it's not as advanced as ArgyllCMS's gamut mapping though. So yeah, basically, just use a BT.709 primaries 2.2 power curve input profile, because that's exactly what madVR spits into the .3dlut.

That's exactly why I always have used Scarse for creating REC709 profiles (up to 16bit 256x256x256). It's too bad that project was abandoned in alpha status 6 years ago, and was left in state where creating non-calibrated custom ICC profiles is all it's good for. Unfortunately, there just aren't many tools that I'm aware for creating uncalibrated ICC profiles with user-defined settings.

Okay, good to know. I'll go check it out.

[alph@]

Might be the locale issue again, afaik all that could be causing it is Double.Parse() which is used to generate a number from a source string (in the .ti3 files)..
ok,it's work now,i have solved the probleme -panels of configuration -region and language-formats-additional parameters-in 'decimal symbol' replace the comma ',' by a point '.', Now I can test linkICC.

[Graeme Gill]

It has now been over 2 years since people began using 3dluts for color correction on their GPUs with madVR. All nand-chan is doing is making tools to ease the process of 3dlut creation and modification. Previously people were pretty much restricted to manually copying and pasting measurements from TI3 files or tools like ColorHCFR and feeding them to yesgrey's tool yCMS to create a 3dlut for use with madshi's video renderer, madVR.


From what I can see of it, it's all a bit weird - a very round about, cumbersome and inflexible approach to dealing with video color management, and an awful lot of re-inventing the wheel.


Recently madshi and yesgrey have added support for 6bit (64x64x64) and 7bit (128x128x128) 3DLUTs where madVR uses trilinear interpolation to generate the missing values, in addition to the full 8bit (256x256x256) lut. I personally have been using a 6bit 3DLUT for awhile, the results are very good, but on average I've measured a yCMS created 6bit 3DLUT having 0.5 worse dE values than a full 8bit 3DLUT with my i1pro rev. D. The difference is indeed nearly invisible (gamma correction aside), but with modern GPUs $80+ having zero performance issues with the full 16.7million color 3DLUT it's more of case of why not? The more significant issue with the current implementation is not gamut correction with 6bit 3DLUT, but rather gamma correction where the difference between 64 points and 256 points (non-smoothed) can be extreme when attempting to represent non-power-curves like REC709 or sRGB in the lower tones.


You simply shouldn't be using the 3D tables to do the per channel curves. That's why any ICC device link worth it's salt has per channel curves as well as the 3D table. For the 3D table to be a reasonable prospect, it needs to be in a colorspace that best leveraged its advantages, while minimises its limitations. So typically the color mapping would be designed to be in a perceptually
linear space, with the input curves (or even more sophisticated transforms if you use ICC V4) serving to transform between the device space and the 3D mapping space.


That said, there is a lot of room for improvement in 3dlut creation. All yCMS created 3dluts take into account are white point, primary colors (100% Red/Green/Blue), and IRE grayscale, making the resulting correction extremely basic and likely worse than an Argyll created Matrix/Shaper ICC profile. This no doubt hurts the potential accuracy of a 6bit 3DLUT.


One of the things that I see as being rather suspect, is the lack of clear distinction between a device profile (which relates the device color values like RGB to device independent color values such as XYZ or L*a*b*), and a device link (which transforms from one device space to another, such as REC709 to display RGB).


As I've been saying for the past year or so on Doom9, what would be amazing is if ArgyllCMS colprof could be modified to create a madVR-compliant 3DLUT instead of a ICC profile during a normal ArgyllCMS calibration workflow. Graeme would modifying your existing cLUT code to output a 6bit(?) 3DLUT interest you personally at all (since nobody else has been interested...)? What we are really lacking is a high quality CMS like ArgyllCMS that takes a full set of measurement patches into account so we can actually get high quality results with less data points in the 3DLUT. The work nand chan has been doing is a step in the right direction, but an official ArgyllCMS solution which meets your quality standards and approval would be so much better. madshi as said a few times already that he'd love to be able to make 6bit 3DLUTs the default in madVR if the quality improved.

This doesn't seem the right approach at all to me. I'd strongly suggest instead implementing direct support for ICC device links in the video display path, taking as a model how ArgyllCMS's cctiff works. You don't even have to do any work as a first cut to see if the quality of transform is of the right order - take a single frame and use the whole of the Argyll toolchain to make a video standard to device RGB transform, and apply it to the frame using cctiff. I'm quite prepared to investigate any quality issues with this (as I've done in the past). This approach leverages established color management tools, and minimises a lot of duplication and rediscovery of how not to do things :-)

[nand chan]

From what I can see of it, it's all a bit weird - a very round about, cumbersome and inflexible approach to dealing with video color management, and an awful lot of re-inventing the wheel.

Re-inventing the wheel is not always a bad thing - if the existing wheels become too large or complex, it can make sense to start a “fresh”, application-specific, limited approach.

You simply shouldn't be using the 3D tables to do the per channel curves. That's why any ICC device link worth it's salt has per channel curves as well as the 3D table. For the 3D table to be a reasonable prospect, it needs to be in a colorspace that best leveraged its advantages, while minimises its limitations. So typically the color mapping would be designed to be in a perceptually
linear space, with the input curves (or even more sophisticated transforms if you use ICC V4) serving to transform between the device space and the 3D mapping space.

Or you increase the bit depth to compensate.

One of the things that I see as being rather suspect, is the lack of clear distinction between a device profile (which relates the device color values like RGB to device independent color values such as XYZ or L*a*b*), and a device link (which transforms from one device space to another, such as REC709 to display RGB).

I personally think one should draw a line between data information and usage information. So, for example, while it makes sense to store, say, the input primaries, or the color encoding, or bit depth - I don't think it makes sense to store things such as which display it's used for, or what the applied transformations are.

Such things, in my opinion, belong in the hands of the usage implementation, and not the data format. If different implementations, for example a monitor calibration system, want to store additional meta-data (which file means what, which display they're associated with), they should introduce their own structures.

Encoding such trivial “information” into the format itself is, in my opinion, more of a waste of space than additional bit depth.

This doesn't seem the right approach at all to me. I'd strongly suggest instead implementing direct support for ICC device links in the video display path, taking as a model how ArgyllCMS's cctiff works. You don't even have to do any work as a first cut to see if the quality of transform is of the right order - take a single frame and use the whole of the Argyll toolchain to make a video standard to device RGB transform, and apply it to the frame using cctiff. I'm quite prepared to investigate any quality issues with this (as I've done in the past). This approach leverages established color management tools, and minimises a lot of duplication and rediscovery of how not to do things :-)

Don't quote me on it but I believe the problem is that yesgrey, madshi, etc. simply don't have enough time to create a full implementation of the ICC profile, and they refuse to use existing libraries such as LittleCMS or ArgyllCMS to do the job, as that would not only take the fun out of it but also deny a learning experience, credit/license and ultimately defeat the point.

So they decided to create their own, stripped down format instead, and it's up to programmers such as me who have both time and no obligations to reject third party code, to create the middleware / bridgeware.

[nand chan]

Version 0.13
This is a bug fix release, mainly to fix a blatant issue in the CalFile parser that was introduced last version. It's a shame I didn't notice it sooner.

Download link (v0.13) (http://www.mediafire.com/?ebf348k5ksjt9w3)

Changelog:

Version 0.13:
* ICC profiles without added primaries information no longer get rejected (the color space gets set as “unknown”)
* Calibration files no longer have swapped primaries, bug introduced in version 0.12
This fix affects: CalFile(), applycal, imagecal and --calibrate in make3dlut

[madshi]

From what I can see of it, it's all a bit weird - a very round about, cumbersome and inflexible approach to dealing with video color management, and an awful lot of re-inventing the wheel.
The basic idea is that having a very big 3dlut allows us to use as complex a calibration math as we want, without affecting rendering performance. Furthermore it allows us to strictly separate calibration programming from video rendering programming. Personally, I've more than enough on my hands just with the video rendering, I don't really want to get into calibration programming too much. So I very much the approach to just use a 3dlut in my renderer and rely on external tools to do the math stuff in offline 3dlut creation tools.

I'd strongly suggest instead implementing direct support for ICC device links in the video display path, taking as a model how ArgyllCMS's cctiff works.
I'd have to write all that as GPU pixel shaders, and I doubt it would run fast enough on any current Intel GPU for 1080p60. Especially considering that I want to use the GPU shaders for other things like chroma upsampling, scaling, sharpening etc etc, too.

And what happens if we aim for a "perfect" calibration? E.g. if we do a couple hundred (or even thousand) measurements and try to achieve perfect correction based on those measurements? Considering all the funny things like too small/big display gamuts, out of gamut source data, perception based stuff etc, the math to achieve "perfect" calibration could quickly become very complicated. Too complicated to run it in realtime on current Intel GPUs at least, I think.

[nand chan]

The basic idea is that having a very big 3dlut allows us to use as complex a calibration math as we want, without affecting rendering performance. Furthermore it allows us to strictly separate calibration programming from video rendering programming. Personally, I've more than enough on my hands just with the video rendering, I don't really want to get into calibration programming too much. So I very much the approach to just use a 3dlut in my renderer and rely on external tools to do the math stuff in offline 3dlut creation tools.

But that doesn't conflict with what he suggests - doing the calculations offline as you mentioned and storing the result in a small RGB -> RGB link profile (ICC), the same as a .3dlut would. His only gripe is with you re-inventing the format when the ICC profiles already exist.

I'm not quite sure I agree with his philosophy though, since implementing such a huge and complex standard as ICC profiles takes a lot more work than re-inventing some simple format.

I'd have to write all that as GPU pixel shaders, and I doubt it would run fast enough on any current Intel GPU for 1080p60. Especially considering that I want to use the GPU shaders for other things like chroma upsampling, scaling, sharpening etc etc, too.

And what happens if we aim for a "perfect" calibration? E.g. if we do a couple hundred (or even thousand) measurements and try to achieve perfect correction based on those measurements? Considering all the funny things like too small/big display gamuts, out of gamut source data, perception based stuff etc, the math to achieve "perfect" calibration could quickly become very complicated. Too complicated to run it in realtime on current Intel GPUs at least, I think.

It would be the same as, say, a 6-bit .3dlut. If those run on Intel GPUs in realtime, then an ICC link profile would run too.

[madshi]

So an ICC link profile can basically contain a 6bit input 16bit output 3dlut? I have no knowledge about ICC at all, so I don't really have any clue.

[nand chan]

So an ICC link profile can basically contain a 6bit input 16bit output 3dlut? I have no knowledge about ICC at all, so I don't really have any clue.

If I've interpreted Graeme Gill's posts correctly, yes.

Ps. Another disadvantage of using .icc profiles is that it might confuse people who are attempting to use the more common /device/ profiles instead of a link profile - since they all have the same extension they might not know the difference, or struggle in figuring out how to create a link profile before settling for some inferior solution.

By forcing .3dlut, we have people actually being forced to search for .3dlut programs in particular, which would be our implementations - and there's no confusion between ICC profiles.

[IceB]

Hi.
You are doing a great job - keep on !

I am trying to generate the 3DLUT file using the LinkICCGUI and getting a following error on Win 32 when i press generate.
"Error: Could not find a part of the path
'C:\Users\#MyUsername#\AppData\Roaming\TI3Parser\dfdd367b-09e8-4086-96b8-373cd55f9f4d'."
There is no TI3Parser folder at AppData\Roaming at all.
My extracted 0.13 version is a C:\ .
Latest .NET Frameworks installed.
I choose both ICM and TI3 files created with dispcalGUI as linked on the guide.
Is there something I miss ?

Thanks for advance.

[nand chan]

Hi.
You are doing a great job - keep on !

I am trying to generate the 3DLUT file using the LinkICCGUI and getting a following error on Win 32 when i press generate.
"Error: Could not find a part of the path
'C:\Users\#MyUsername#\AppData\Roaming\TI3Parser\dfdd367b-09e8-4086-96b8-373cd55f9f4d'."
There is no TI3Parser folder at AppData\Roaming at all.
My extracted 0.13 version is a C:\ .
Latest .NET Frameworks installed.
I choose both ICM and TI3 files created with dispcalGUI as linked on the guide.
Is there something I miss ?

Thanks for advance.

Hmm, it might be a permissions issue? Are you on a restricted account maybe?

(It would surprise me though since even restricted accounts are supposed to have access to their appdata folders)

[IceB]

Hmm, it might be a permissions issue? Are you on a restricted account maybe?

(It would surprise me though since even restricted accounts are supposed to have access to their appdata folders)

It is strange.
I have tried it on 2 computers an the laptop - all with admin account returned the same exact error.
I also tried to copy the TI3Parser folder to the Appdata\Roaming folder - then pressing the "generate" button returns different error:
"Error: The system can no find the file specified "

What file does the app looking for and why at the first place it needs to look at AppData\Roaming\TI3Parser while this folder is not there.

Is there any installation process I should do or should I use the new TI3Parser as is after the extraction.

I tried it with different profiles and different machines - all of them return the same error - i must be missing something.

[nand chan]

It is strange.
I have tried it on 2 computers an the laptop - all with admin account returned the same exact error.
I also tried to copy the TI3Parser folder to the Appdata\Roaming folder - then pressing the "generate" button returns different error:
"Error: The system can no find the file specified "

What file does the app looking for and why at the first place it needs to look at AppData\Roaming\TI3Parser while this folder is not there.

Is there any installation process I should do or should I use the new TI3Parser as is after the extraction.

I tried it with different profiles and different machines - all of them return the same error - i must be missing something.

Come to think of it, try running “updateycms.exe” (you don't have to run this from the console, just double clicking it should be fine).

I just realized that I didn't add the dialog to automatically download yCMS to LinkICCGUI as well, I only added it to TI3ParserGUI.

[IceB]

Well done !

The “updateycms.exe” trick worked for my laptop.
I'll try to manage it on the PC later.

Thank you.

[Audionut]

Come to think of it, try running “updateycms.exe”

Can you add an option to select ycms from a user folder.

Seems a bit redundant needing to have ycms in a specific folder.

Cheers.

[nand chan]

Can you add an option to select ycms from a user folder.

Seems a bit redundant needing to have ycms in a specific folder.

Cheers.

I'll add something like it for ver 0.14. I'm also moving the temporary files into, well, %temp%.

Edit: Where do you think I should save the setting for this? 1. A registry key, 2. A config file in appdata or 3. A local configuration file in the \bin folder?

Advantage of 3. is portability, but 1./2. is invisibility and the ability to retain settings / have settings effect every copy of the program.

[Audionut]

Where do you think I should save the setting for this?

Personally, I would prefer 2 as it makes it easy to use the same config when upgrading.

I prefer 2 over 1 for this, as I think there shouldn't been things added to the registry unless really needed, although I not really that bothered as I don't install 9 million programs.

Also some people get really anal about their registries.

3 would work fine if you switch to an installer which can upgrade over a previous release.

edit: If you switch to an installer, you should provide an option to select the location of YCMS during install, or the option to download.
I'd personally be happy to have the installer copy my download of YCMS to it's own bin folder (in case I delete the YCMS folder for instance without thinking).

[nand chan]

I'm not a fan of installers, I think programs should be extract + run wherever possible - especially if they're command line programs (just add the bin folder to PATH and done).

There won't be such a thing as “upgrading over a previous release” since there's nothing to upgrade - you just download the new version and paste it over the old, replacing all of the files. If any new tool makes further use of this proposed configuration file, it can simply add the keys when it first tries to access them. A formal installer is not needed.

I personally don't see the difference between a file and a registry key since they both end up saved on your hard drive either way. I imagine a registry key has /less/ overhead than a file, though.

If I *do* create an installer (or rather a small setup script), I can provide the option to choose where you want to save the config file. (With the program checking all three on load, local files take the highest precedence).

[Thunderbolt8]

I need a 3dlut file (compatible to madvr) with input_range 16 235 and output_range 0 255. everything else should stay the same. how can I do this in highest quality possible?

[nand chan]

I need a 3dlut file (compatible to madvr) with input_range 16 235 and output_range 0 255. everything else should stay the same. how can I do this in highest quality possible?

Just a question, why don't you just set your monitor's output range to 0-255 in madVR?

[Thunderbolt8]

not sure whether I have done this already (output is set to PC levels 0-255). I need to correct a japanese BD which has wrong levels.

[nand chan]

not sure whether I have done this already (output is set to PC levels 0-255). I need to correct a japanese BD which has wrong levels.

I see. Well, do you mean that the BD compressed the levels twice? Eg. it stores 32-188 instead of 16-235? (Result: Everything in the BD looks more flat and gray than it should)

If so, you can fix that by expanding the levels using the following script:

# Set the properties properly
!Input_Range(Limited)
!Output_Range(Limited)
!Filetype(3DLUT)
!Input_Primaries(BT709)

!Pixel( {

Scale(-0.07306, 1.09)
Clamp(0, 1)

} )

For an explanation:

If we consider 0-1 to be 16-235, then 0-255 will be (-0.07306)-1.09.

As such, scaling with these two coefficients *stretches* the levels out to full range.

(Alternatively, you can just write !Output_Range(Full) since madVR does not check to see if the output is limited range as well, but this is not the ideal way to solve it as it will use slightly different scaling coefficients, those of 16-bit output instead of 8-bit).


If you want to do the opposite of this, that is, correct a BD which has mistakenly encoded 0-255 levels (blacks are crushed and whites are bleached, too high contrast), you would use the same script but change the line to Scale(0.06275, 0.92157).

Edit: Oh and if you're OCD about accuracy, use Scale(-0.07305936073059360730593607305936, 1.0913242009132420091324200913242).

[Thunderbolt8]

alright, but how to create the script, what tool to use in this case? used to do it with ycms with simply input_range & output_range stuff, but you said there seems to be a tiny more accurate way to do it.

[nand chan]

alright, but how to create the script, what tool to use in this case? used to do it with ycms with simply input_range & output_range stuff, but you said there seems to be a tiny more accurate way to do it.

Save it in a file anywhere (the official extension is .3ls), then run:

lutscript -i <path to file> -o Limited2Full.3dlut

[Thunderbolt8]

thanks, but madvr complains that the 3dlut file is missing primaries information (picture stays black)

I could copy & paste

Input_Primaries 0.640000000000000 0.330000000000000 0.300000000000000 0.600000000000000 0.150000000000000 0.060000000000000 0.31272661468101209 0.32902313032606195
Output_Primaries 0.640000000000000 0.330000000000000 0.300000000000000 0.600000000000000 0.150000000000000 0.060000000000000 0.31272661468101209 0.32902313032606195

from the ycms template file into the script you gave me, but would this be valid input for lutscript as well or accurate enough?


and how did you get the values for 0.07306..., 1.09... how to calculate this?

[nand chan]

thanks, but madvr complains that the 3dlut file is missing primaries information (picture stays black)

Did you forget the line !Input_Primaries(BT709) maybe? That line is supposed to add the correct primaries already. Either that, or your region locale uses commas instead of decimals (which wouldn't be the first time the issue has cropped up).

I could copy & paste

Input_Primaries 0.640000000000000 0.330000000000000 0.300000000000000 0.600000000000000 0.150000000000000 0.060000000000000 0.31272661468101209 0.32902313032606195
Output_Primaries 0.640000000000000 0.330000000000000 0.300000000000000 0.600000000000000 0.150000000000000 0.060000000000000 0.31272661468101209 0.32902313032606195

from the ycms template file into the script you gave me, but would this be valid input for lutscript as well or accurate enough?

Not quite, the LutScript format is !Input_Primaries(0.640000000000000, 0.330000000000000, 0.300000000000000, 0.600000000000000, 0.150000000000000, 0.060000000000000, 0.31272661468101209, 0.32902313032606195) (And the same for !Output_Primaries).

and how did you get the values for 0.07306..., 1.09... how to calculate this?

-0.07306... is (0 - 16) ÷ (235 - 16)
1.09... is (255 - 16) ÷ (235 - 16)

Explanation: The way to convert from 0-1 to limited range is: λi ↦ i × (235 - 16) + 16

However, we want this constant to be equal to 255 for the input 1 (while keeping the .3dlut in limited space since that's what madVR works with). In other words, the input 235 (which gets mapped to 1) should output 255 - thus fulfilling our desired goal.

So we want to introduce a scaling function ε so that λ(ε(0)) = 0:

ε(0) × (235 - 16) + 16 = 255 | -16
ε(0) × (235 - 16) = (-16) | ÷ (235 - 16)
ε(0) = (-16) ÷ (235 - 16)
ε(0) = (-16) ÷ (219)
ε(0) ≈ -0.07305936073059360730593607305936

This is now our lower limit.

For the upper limit, we proceed similarly, under the assumption that λ(ε(1)) = 255:

ε(1) × (235 - 16) + 16 = 255 | - 16
ε(1) × (235 - 16) = (255 - 16) | ÷ (235 - 16)
ε(1) = (255 - 16) ÷ (235 - 16)
ε(1) = (255 - 16) ÷ (219)
ε(1) ≈ 1.0913242009132420091324200913242

[Thunderbolt8]

Did you forget the line !Input_Primaries(BT709) maybe? That line is supposed to add the correct primaries already. Either that, or your region locale uses commas instead of decimals (which wouldn't be the first time the issue has cropped up).no, I copy & pasted the whole script box like you wrote it down. also tried it with comma, but that didnt change anything

edit: at the command line, I didnt use .3dl2 as output file extension, but .3dlut (this is what you wrote though). maybe thats the mistake? do I have to use .3dl2 and convert that file afterwards to a madvr compatible .3dlut file (if so, how?)

[nand chan]

no, I copy & pasted the whole script box like you wrote it down. also tried it with comma, but that didnt change anything

No, I mean going to Control Panel -> Region and Language -> Formats -> Additional Settings and setting the “Decimal symbol” to a period.

I'll see if I can definitively fix this issue by the next version.

edit: at the command line, I didnt use .3dl2 as output extension, but .3dlut. maybe thats the mistake? do I have to use .3dl2 and convert that file afterwards to a madvr compatible .3dlut file (if so, how?)

No, .3dlut is correct.

The line !Filetype(3DLUT) means it's a .3dlut file, had you written !Filetype(3DL2) or left it out completely, it would have been a .3dl2 file.

The !Filetype command already ensures compatibility with madVR.

[Thunderbolt8]

No, I mean going to Control Panel -> Region and Language -> Formats -> Additional Settings and setting the “Decimal symbol” to a period.

I'll see if I can definitively fix this issue by the next version.that might have helped a bit. Im also from germany, but use an english version of win7 (region set to ger, lg set to eng)

after that change, I dont get that error any more, but the picture stays black. video plays at least and theres also sound, but screen remains black.

[nand chan]

that might have helped a bit. Im also from germany, but use an english version of win7 (region set to ger, lg set to eng)

after that change, I dont get that error any more, but the picture stays black. video plays at least and theres also sound, but screen remains black.

Copying the script I posted earlier 1:1 and doing nothing else but running lutscript as described, it works fine here (and the levels get expanded appropriately).

Maybe there's still some locale issue going on. (I'm from Germany as well but I have my locale completely set to English formats)

I'll try uploading version 0.14 which has the en-US culture forced, see if it works.

[nand chan]

I'll try uploading version 0.14 which has the en-US culture forced, see if it works.

Link: http://www.mediafire.com/?o8wds588w66ot66

Change log for reference:

Version 0.14:
* LinkICCGUI now automatically updates yCMS on first run as well
* imagecal.exe clamps the levels to 0-255 properly now, before it would overflow
* Another attempt at fixing the locale bug, this time by forcing en-US explicitly instead of the invariant culture

[Thunderbolt8]

with this version, Im back to the no primaries information error message (after reverting the comma to decimal point change again in windows)

[nand chan]

with this version, Im back to the no primaries information error message (after reverting the comma to decimal point change again in windows)

Strange.

Do you think you could upload the output of inspect3dlut -i Limited2Full.3dlut --params?

[Thunderbolt8]

corretion, it now does work, but only if I keep this comma decimal point change applied. when I revert back to how it usually is (what I actually like to do, since no other programs have problems with it) then it doenst work.

[nand chan]

corretion, it now does work, but only if I keep this comma decimal point change applied. when I revert back to how it usually is (what I actually like to do, since no other programs have problems with it) then it doenst work.

Try this version?

http://www.mediafire.com/?ezjufd54ft6o9y6

I set the locale to “German” on my machine and it seems to work using this version. What I did is went through the code and forced every single .Parse() and .ToString() to use the en-US culture... silly culture issues.

If only .NET would just get rid of the whole globalization crap and standardize the sane English number formats we wouldn't have this problem. Oh well.

[Thunderbolt8]

it does work now, thanks :-)

strangely though, it only works properly with LAVsplitter, but not with haali.

with haali, the file doesnt seem to load up properly, playback wont start. in order to get it work with haali, what I needed to do is to start with an invalid 3dlut file and then switch over to that correct 3dlut file. then the file would start playing.

but when trying to get it to work with the correct 3dlut file directly from the start, theres just a black screen with no playback.

[nand chan]

it does work now, thanks :-)

strangely though, it only works properly with LAVsplitter, but not with haali.

with haali, the file doesnt seem to load up properly, playback wont start. in order to get it work with haali, what I needed to do is to start with an invalid 3dlut file and then switch over to that correct 3dlut file. then the file would start playing.

but when trying to get it to work with the correct 3dlut file directly from the start, theres just a black screen with no playback.

What I *THINK* might be causing the issue: Since madVR loads the .3dlut at the start, it causes a short delay before playback starts (same when switching to/from fullscreen). Maybe this short playback delay causes Haali to fail, but not LAV Splitter? I can't test it myself, Haali works fine here.

Does playback with Haali work immediately using any other .3dlut?

Also, what you could try is adding the line !Input_Depth(7) or !Input_Depth(6) which should reduce the file size quite dramatically (at the cost of a bit of quality).

But, if that works right away, it offers further proof for my theory - these are not as large, ergo faster to load.