.ti3 Parser / .3dlut Generator for madVR + .3dlut toolset

[Audionut]

How about uploading to a host that in turn uploads to various hosts.

http://www.multiupload.com/

[nand chan]

Originally Posted by Audionut

How about uploading to a host that in turn uploads to various hosts.

http://www.multiupload.com/

http://www.multiupload.com/VANWR6WMUF

I won't do that for anything serious though since none of those downloading services are remotely as good as mediafire. (Except maybe the multiupload's own DD)

I can't stand wait times and such, if I'm forced to download from such shitty hosts I'd rather not download the file at all.

Edit: MEDIAFIRE IS BACK UP http://www.mediafire.com/?0sbwzrd8f0nowbw

[Audionut]

Originally Posted by nand chan

I won't do that for anything serious though

No worries. Looks like I successfully converted the base64 anyway.

Never heard of that before.

Thanks again.

[Audionut]

Ok, something is wrong.

I used,

linkicc -i Rec.709_2.35_gamma.icc -d device.icm -o output.3dl2

tag3dlut -i output.3dl2 -o output.3dlut --legacy --compression none

Results in this.

[nand chan]

Originally Posted by Audionut

Ok, something is wrong.

I used,

linkicc -i Rec.709_2.35_gamma.icc -d device.icm -o output.3dl2

tag3dlut -i output.3dl2 -o output.3dlut --legacy --compression none

Results in this.

<snip>

Yeah, I just noticed that as well. The issue seems to be deeper than simply a bugged tag. It seems like all of the conversions done by LittleCMS are completely different as well when the tags change to the correct (?) values.

I've posted to the lcms-user mailing group in order to find out what the cause of this bug is, but in the meantime, what you can do is use the incorrect profile (re-uploaded here: http://www.mediafire.com/?9g66hzyg92jmlye) and fix the tagging manually with -7.

So, basically,

linkicc -i Rec.709_2.35_gamma.icc -d device.icm -o output.3dl2

tag3dlut -i output.3dl2 -o output.3dlut --legacy --compression none -7

For the next version (0.11) of linkicc.exe, I've made it ignore the ICC profile tags by default and assume everything to be BT.709, since this is simply unacceptably flawed and I don't want to trouble my end users with having to deal with LittleCMS's bullshit.

[Audionut]

That fixed it.

[nand chan]

Another comparison between the two:

http://screenshotcomparison.com/comparison/78975

(BPC) indicates black point correction, a flag which I've enabled starting version 0.11 (without BPC it's basically the same but the blacks are a tiny bit cut off, since it maps the input to a true black level which of course the LCD monitor can't actually reproduce)

I haven't quite tracked down the reason why the yCMS image seems a bit brighter in general, but I'm guessing it's because of different gamma settings.

Edit: yeah, it was a different gamma setting. yCMS uses a gamma of 2.2 by default, BT.709 uses 2.35. That's why the image appears slightly darker. I'll upload an sRGB comparison image in a moment.

[nand chan]

Here is an sRGB comparison image:

http://screenshotcomparison.com/comparison/78977

And here is a 2.2 gamma BT.709 comparison image: (this should, in theory, be closest to the yCMS result)

http://screenshotcomparison.com/comparison/78979

I've uploaded the 2.2 gamma profile if you want to make use of it: http://www.mediafire.com/?8svo69bqavubn4b (the same warning applies as for the last)

What it boils down to: LittleCMS handles the high-sat colors better but yCMS handles the low spectrum better.

I think it's time for me to create a program that can combine the two, by using LittleCMS values above a certain brightness, and yCMS values below them. Then we'll have a perfectly linear tone curve mapping that accounts for LCD inaccuracies, with the full 3D color correction of LittleCMS!

Sounds like a plan? You bet it does!

[Audionut]

Getting an error from mediafire trying to download the 2.2 gamma profile.

[Graeme Gill]

This may be a dumb question, but why all this messing around with .ti3 files ? They are raw device measurement information, and that leaves a lot of work in creating a device model and linking it to a input profile to create a 3D transform.

There are a bunch of tools already out there that do most of this work. ArgyllCMS has the profile creation capability, and both lcms and Argyll have device link capability. (Argyll's is probably a bit more sophisticated though).

ICC device links are a standard "3dlut", so why invent some new format ? All you should have to work on for use with video is 1) Creating suitable input profiles to suite video encoding standards 2) Providing a mechanism to apply a 3Dlut and 1D curves to video in realtime using the GPU, and 3) provide a loader to load ICC device to device links + display calibration curves so the GPU code can use it. Note that the equivalent machinery (implemented as non-real time, general CPU code) exists in Argyll imdi. lcms has similar code.

[nand chan]

Originally Posted by Graeme Gill

This may be a dumb question, but why all this messing around with .ti3 files ? They are raw device measurement information, and that leaves a lot of work in creating a device model and linking it to a input profile to create a 3D transform.

I am aware of the amount of work involved here, this is why I do not do any transformations myself - I use yCMS and LittleCMS (which itself relies on ArgyllCMS for the profile creation), all of which are highly complete and well-done tools.

There are a bunch of tools already out there that do most of this work. ArgyllCMS has the profile creation capability, and both lcms and Argyll have device link capability. (Argyll's is probably a bit more sophisticated though).

linkicc uses lCMS to create the device link. You can also use LutScript (starting version 0.11) to load a single device link, eg. from ArgyllCMS.

ICC device links are a standard "3dlut", so why invent some new format?

ICC profiles are not complete 3dluts. If you look at the filesizes, a typical “3dlut-based” ICC profile is about 20-40 KB large, and a typical .3dlut is about 96 MB large. There are further issues. For example, ICC profile 3dluts are natively 8-bit.

All you should have to work on for use with video is 1) Creating suitable input profiles to suite video encoding standards

This you can/need to do either way, for example the BT.709 profile above.

2) Providing a mechanism to apply a 3Dlut and 1D curves to video in realtime using the GPU

This is not doable in realtime. 64-bit interpolation would unacceptably strain the GPU, especially on GPUs that simply cannot handle it.

and 3) provide a loader to load ICC device to device links + display calibration curves so the GPU code can use it. Note that the equivalent machinery (implemented as non-real time, general CPU code) exists in Argyll imdi. lcms has similar code.

The reason why they are non-realtime / general is because they are so slow.

You're missing the point of the .3dlut format. The .3dlut format is not a general purpose / comprehensive “transformation” format, the point of the .3dlut format is to pre-calculate /every single value/ so you can use the result in real-time during video playback.

Have you ever loaded a 1920x1080 image on a color managed application like Firefox? It takes about ~5 seconds depending on the image to apply the color correction fully, and that's without 64-bit floating point logic.

Using a properly multi-threaded code (linkicc), it still takes about 10 seconds to generate a full .3dlut from ICC profiles with all of the bells and whistles (FLAG_HIGHRESPRECALC, BPC, application of 3x1D TRCs and so forth).

[nand chan]

Originally Posted by nand chan

What it boils down to: LittleCMS handles the high-sat colors better but yCMS handles the low spectrum better.

I think it's time for me to create a program that can combine the two, by using LittleCMS values above a certain brightness, and yCMS values below them. Then we'll have a perfectly linear tone curve mapping that accounts for LCD inaccuracies, with the full 3D color correction of LittleCMS!

Sounds like a plan? You bet it does!

I've implemented this for now,

Here's what I dubbed a composite image vs yCMS:

http://screenshotcomparison.com/comparison/79074

They should basically be the same in terms of appearance and black level handling, but the composite image will be, in general, a /bit/ more accurate (since it accounts for 3D errors as well).

For an explanation of how the system works:

http://img40.imageshack.us/img40/582...positeluma.png

The areas where the screen is red means LittleCMS is being used, the areas where it's blue means yCMS is being used. Purple means they're being interpolated. The determining factor is luma, and you can adjust the cross-over point and amount of spread (interpolation) being performed.

Here is a grayscale comparison: http://img823.imageshack.us/img823/7021/compvsycms.png

On my monitor, I notice that yCMS's grays are slightly too blue, since it doesn't perform the 3D correction. It also shows a tiny bit more banding than the composite.

I've also disabled black point compensation again since composite mode eliminates the problem.

[nand chan]

Version 0.11 release



This release spotlights the addition of a new GUI program called LinkIccGUI. This GUI program serves as a front-end for the entire process from .ti3 parsing to ICC linking up to and including auto-calibration, legacy tagging, composite joining and so forth.

Users who simply wish to get good color management out of the box now have a one-click solution that doesn't require any further in-depth knowledge of the system.

Special note: If “auto-calibrate” is checked, you have to make sure that /no/ calibration curve is interfering with madVR. Either disable Windows color management or set your monitor to the sRGB device profile. Alternatively, you could use one of the many available tools to disable the curves (dispcalGUI, dispclut.exe, ArgyllCMS, CalibrationTester.exe and so forth).

Download link: http://www.mediafire.com/?c6hlon3r1g0ohfa

Changelog:

Version 0.11:
- IccTransform() was removed
+ IccSingle(String, [Intent]) added. Intent is optional and defaults to Relative.
  This IccSingle filter decodes between a profiles PCS and ColorSpace and can go both ways (eg. XYZ->RGB, RGB->XYZ)
+ IccLink(String, String, [Intent]) added - this links two profiles together and can only go one way (no PCS support)
* Missing ICC profiles no longer result in a crash
* Comments after LutScript lines with strings in them get filtered out properly now
* Fixed a bug where tag3dlut would still mark a lut as compressed even in legacy mode (the actual output wasn't
  compressed, but the tag was incorrectly set)
* linkicc.exe now automatically assumes the input to be BT.709 instead of detecting the primaries from the ICC profile,
  due to a strange issue in LittleCMS where the input tags will be completely incorrect.
+ New tag --auto-detect (-a) reverses this behavior [linkicc]
+ New program: join3dlut.exe, this program joins together two .3dl2s, switching between the two depending on
  the luminance of the input.
+ New program: LinkICCGUI.exe. This is the new preferred front-end for everything. It accepts both an .icc and a .ti3,
  and applies the following steps automatically:
   make3dlut
   linkicc (auto-generates a BT.709 / 2.2 profile in memory)
   join3dlut
   applycal
   tag3dlut
  The result is a fully functional v1 .3dlut ready for madVR compatibility. No further steps are needed.

[Thunderbolt8]

the whole thing still seems rather complicated to me, because I dont know what is required to start, what information or input files and where to take them from

basically, all I want to to is to create a .3dlut file for black level fixing, so changing input levels from default 0-255 to something like input range 16+X - 235+X (x= depending on how much black and/or white is off)
and sometimes changing gamma a little as well which worked in ymcs like this:

Output_Transfer_Function 4.5 0.099 0.45 0.018

by modifying the value 0.45 which then resulted in more or less gamma.

with ycms, I only needed to put some lines into a .txt file, but how does this work with your programm?

[alph@]

can you explain why you mix a icm file with a ti3 ,for the precision of the 3dlut?

[nand chan]

Originally Posted by Thunderbolt8

the whole thing still seems rather complicated to me, because I dont know what is required to start, what information or input files and where to take them from

basically, all I want to to is to create a .3dlut file for black level fixing, so changing input levels from default 0-255 to something like input range 16+X - 235+X (x= depending on how much black and/or white is off)

That's not currently possible (with the X offset thing). What you can do is change the input level from 0-255 to 16-235, but you can't apply an offset to either manually. I can add it as a feature for 0.12.

If you still want to create a .3dlut that will map from 0-255 to 16-235, you can do that by using:

gen3dlut.exe -o levels.3dl2 --range Full
changedepth.exe -i levels.3dl2 --output-range Limited

and sometimes changing gamma a little as well which worked in ymcs like this:

Output_Transfer_Function 4.5 0.099 0.45 0.018

by modifying the value 0.45 which then resulted in more or less gamma.

with ycms, I only needed to put some lines into a .txt file, but how does this work with your programm?

If you want to change the gamma afterwards you can create a Limited-range .3dlut that just changes the gamma in yCMS, and then merge it onto the .3dlut from earlier:

merge3dlut -o result.3dl2 levels.3dl2 gamma.3dlut

Now that I have LittleCMS integrated, I can add a program for changing the gamma level of a .3dlut by version 0.12 as well.

Ps. Alternatively, just use LutScript:

!Input_Range(Full)
!Output_Range(Limited)

!Pixel({
  LutFile("gammafixing.3dlut")
})

Originally Posted by alph@

can you explain why you mix a icm file with a ti3 ,for the precision of the 3dlut?

Yes, basically. The .icm file doesn't handle near-black levels the way I want it, so I handle the near-black levels manually using the .ti3 (and yCMS).

[Thunderbolt8]

Originally Posted by nand chan

That's not currently possible (with the X offset thing). What you can do is change the input level from 0-255 to 16-235, but you can't apply an offset to either manually. I can add it as a feature for 0.12.

that would be very helpful, because sometimes only black leves are affected, while white leves are fine, or they are affected between 0-16 input range, so simply 16-235 would be too far off.

Originally Posted by nand chan

gen3dlut.exe -o levels.3dl2 --range Full
changedepth.exe -i levels.3dl2 --output-range Limited

If you want to change the gamma afterwards you can create a Limited-range .3dlut that just changes the gamma in yCMS, and then merge it onto the .3dlut from earlier:

merge3dlut -o result.3dl2 levels.3dl2 gamma.3dlut

Now that I have LittleCMS integrated, I can add a program for changing the gamma level of a .3dlut by version 0.12 as well.

Ps. Alternatively, just use LutScript:

!Input_Range(Full)
!Output_Range(Limited)

!Pixel({
  LutFile("gammafixing.3dlut")
})

I would be able to create these 2 things with whatever aspect or tool is most precise, so whatever gives the best results (and is accepted by madvr as input file of course). so using littlecms for both is recommended atm?

[nand chan]

Originally Posted by Thunderbolt8

that would be very helpful, because sometimes only black leves are affected, while white leves are fine, or they are affected between 0-16 input range, so simply 16-235 would be too far off.

It seems like I will have to support dynamic ranges after all :P.

Oh well, this is why I split off the 3DL2 from 3DLUT in the first place, so I can make changes like this and modify the spec as feedback comes in, before pushing it back upstream.

I would be able to create these 2 things with whatever aspect or tool is most precise, so whatever gives the best results (and is accepted by madvr as input file of course). so using littlecms for both is recommended atm?

I personally trust yCMS more than LittleCMS. LittleCMS has a lot of little funny things, and I don't like the way it handles black levels at all.

It's important to note here that LutScript is just a front-end, it's basically just a script-form that wraps all of the functions that the 3dLutLib.dll exposes.

So if I add some feature like YCbCr conversion, gamut mapping or gamma conversion to LutScript, it will just use the respective functions from yCMS and/or LittleCMS.

Ps. If you're doing gamma correction, why not use madVR's built in function for this? Or is the application going to be something other than madVR?

[Thunderbolt8]

I have no clue, no clue what all these progs can go, what gamut correction even means or that madvr has a built in function for this. Ive been given that line once to change gamma and that was the only way I knew how to change this.

[nand chan]

Originally Posted by Thunderbolt8

I have no clue, no clue what all these progs can go, what gamut correction even means or that madvr has a built in function for this. Ive been given that line once to change gamma and that was the only way I knew how to change this.

Sorry, I mistyped - I meant /gamma/ correction. madVR has a built-in function for gamma correction, you can scale the input (2.2 gamma) to any pure power curve / BT.709 curve you want, eg. 2.35 or 2.0, without needing a .3dlut for the task.