gamut conversions through Avisynth ?

[Kamus]

I've got good results, but should they be better?
I'm using MadVR+cr3dlut, I'm wondering if I've done something wrong here:

http://i139.photobucket.com/albums/q296/Saintkamus14/gamut.jpg

Pic on the left is from the native gamut of my JVC RS1x projector using an eye one 2 LT, the one on the right is the corrected gamut, and while it's better i still get a delta E that's quite a bit off.
Green has a delta E of 11, 6.7 on red and 17 on blue with the 3dlut file enabled.
While these aren't horrible results, i'm obsessed. (and supposedly delta E needs to be about < 3 to be undetectable by people right?)

My HD_PC file looks like this:

# Example input file for cr3dlut v2.1 and up
#
# Settings for creating a 3D LUT for watching the following Video formats:
# Blu-ray, HD DVD, ATSC HD Broadcast, PAL HD Broadcast
# without any Display correction
# Includes YCbCr->RGB conversion using PC Levels (Black: 0 and White: 255)

# Set input bitdepth
Input_Bit_Depth 8

# Set source video format
Input_Video_Format HD YCbCr

# Set output bitdepth
Output_Bit_Depth 16

# Set display video format
Output_Video_Format HD RGB_PC

# set Output Primaries
Output_Primaries 9 0.651 0.348 0.294 0.697 0.140 0.043 0.312 0.330

BTW, i'm calibrated to D65 if it means anything, and gamma is at 2.2 (i simply used the eye one match 3 software to set the grayscale & gamma with very good results, or so says HCFR. Maybe the fact that i'm not using any gamma settings on the file has something to do with it? I figured that it was pointless since the projector is already at 2.2 gamma and i figured it shouldn't be related in the first place)

Also, I'm using "PC levels" since my ATi videocard is set to RGB limited and that takes care of the video levels. The color patterns i used are from the AVS HD disc. (http://www.avsforum.com/avs-vb/showthread.php?t=948496) by extracting the files and playing them trough MPCHC. (also used color facts test patterns to see if there was any significant difference in the native gamut, compared to using the files, and there wasn't)

Anyway, any tips on how i could make this more accurate? I'm overall pleased with the results, but if i can improve them even more, then all the better.

Thanks in advance for anyone that takes the time to reply, and Kudos to everyone involved in making HTPC a very viable and cheap solution.

[yesgrey]

I've got good results, but should they be better?
Try adding this line at the end of your cr3dlut setting file:
Output_Gamma 9 1.0 0.0 0.454 0.0
If it's not good enough or if you notice any banding at low levels, try adding this too:
Input_Gamma 9 1.0 0.0 0.454 0.0

[Kamus]

Thanks, that seems to have done the trick for blue and green almost to perfection (had to use both input and output, just the output messed with the levels it seems) red is somewhat off, but it's due to the fact that my bulb doesn't seem to be able to do a full red anyway.

[yesgrey]

Great.:)

[janos666]

I calibrated my PC monitor with this target values: WP - exact D65 ; Tonal response curve - x^2.22 ; White luminance - 140 cd/m^2 (Balck luminance - Min. Neutral)
My hardwares and softwares: Q6600@3600Mhz (FPS@450) ; Radeon HD 5850 1Gb with Cat 9.12hotfix ; Windows 7 x64
I installed the latest AviSynth 2.6 alpha and the latest MPC-HC (stable x86), and I have a recent SVN build of FFDShow (x86). I used the latest cr3dlut and t3dlut from this thread.
This is all the 3 lines I have in the AviSynt box (I move the # for SD movies...):
yv12toyuy2(itype=2, threads=4)
t3dlut("C:\Program Files (x86)\AviSynth 2.5\hd - pc.3dlut", threads=4, destcs=2)
#t3dlut("C:\Program Files (x86)\AviSynth 2.5\sd - pc.3dlut", threads=4, destcs=2)
And I created my HD - SD.t3dlut file with this settings:
# Set source video format
Input_Video_Format HD YCbCr
Input_Bit_Depth 8

# Set display video format
Output_Video_Format HD RGB_PC
Output_Bit_Depth 8

# Gamut and gamma correction - measured native coordinates
Output_Primaries 9 0.656725 0.329082 0.228974 0.690610 0.140753 0.086776 0.313918 0.328022

# Gamut and Gamma Correction settings
Chromatic_Adaptation 2
Out_Of_Gamut_Clipping 1
Output_Gamma 1

And I have some questions:
Is it normal that I had to reduce the Output_Bit_Depth to 8 bit? I had constant 75% CPU utilization and only 18-20 FPS under Blu-Ray playback with 16-bit (720p was ok). But I do not have a lot of free CPU time with 8-bit anyway. 1080p playback is smooth until I do not try to apply another filters like 720p->1080p resize with FFDShow (there is a reason why would I do that but that is offtopic...).

When I set the Output_Gamma to 9 1.0 0.0 0.45 0.0 then the image is too bright. The result looks exactly the same when I miss the Output Levels (16-235 signal for a 0-255 PC monitor). I think that the output level parameter is ignored when this parameter is used in a later line.
I do not know if the HDTV standard is x^2.22 or something else (like sRGB has an own tonal response curve -> I can calibrate my monitor with an sRGB curve but there is no Rec 709 preset...). Should I worry about it?

I try to use this file with madVR but it would not load my 3dlut files. The malfunction is another question but:
- madVR is internally limited to YV12 input - (I tested it with unchecked AviSynt and FFDShow output settings)
- I can use this renderer with the AviSynt script and I can see corrected colors.
- This would imply that t3dlut is sending a YV12 output for madRV. But this could not be right with destcs=2 settings (and RGB_PC generation settings). It should send an RGB32 output and nothing else. And madVR should accept only YV12. (I did not set up any other conversion and I unchecked every box in FFDShow's output tab, expect YV12.) Where is the trick? :confused:

Anyway, this is a very nice stuff. I tried the old PS script but it caused some wired violet instead of blue. (I have a wide gamut display with over-saturated red and neon-green, but it misses some of the Rec709 blue area. :( )

[yesgrey]

And I created my HD - SD.t3dlut file with this settings:
# Set source video format
Input_Video_Format HD YCbCr

You should change the line above for SD sources.
Use NTSC instead of HD for ntsc dvds.
Use PAL_DVD instead of HD for PAL dvds.
(Read the ReadMe for further details)

This is all the 3 lines I have in the AviSynt box (I move the # for SD movies...):
yv12toyuy2(itype=2, threads=4)
t3dlut("C:\Program Files (x86)\AviSynth 2.5\hd - pc.3dlut", threads=4, destcs=2)
#t3dlut("C:\Program Files (x86)\AviSynth 2.5\sd - pc.3dlut", threads=4, destcs=2)
Is it normal that I had to reduce the Output_Bit_Depth to 8 bit? I had constant 75% CPU utilization and only 18-20 FPS under Blu-Ray playback with 16-bit (720p was ok). But I do not have a lot of free CPU time with 8-bit anyway. 1080p playback is smooth until I do not try to apply another filters like 720p->1080p resize with FFDShow (there is a reason why would I do that but that is offtopic...).

When you use 16 bit output t3dlut dithers the output to 8 bit, hence the higher cpu load. If you have low cpu power available you have to stick with 8 bit output.
You can also try using ConvertYV12ToYUY2() instead of yv12toyuy2, because the later is slower.
Another option would be to use rgb3dlut, which is also faster than t3dlut, but only supports 8 bit output.

When I set the Output_Gamma to 9 1.0 0.0 0.45 0.0 then the image is too bright.
When you change the gamma curve, you have to correct the brightness level of your display, that's why it looks brighter. You have to decrease the display's brightness until you get the correct black level.

I do not know if the HDTV standard is x^2.22
HDTV is BT.709.

I try to use this file with madVR but it would not load my 3dlut files.
madVR only works with 16 bit output 3DLUT files.


- madVR is internally limited to YV12 input - (I tested it with unchecked AviSynt and FFDShow output settings)
- I can use this renderer with the AviSynt script and I can see corrected colors.
- This would imply that t3dlut is sending a YV12 output for madRV. But this could not be right with destcs=2 settings (and RGB_PC generation settings). It should send an RGB32 output and nothing else. And madVR should accept only YV12.

Correct

(I did not set up any other conversion and I unchecked every box in FFDShow's output tab, expect YV12.) Where is the trick?
That's the trick.;)
When you only check YV12 at ffdshow's output, ffdshow converts the RGB32 output from t3dlut to YV12, and feed it to madVR.

Summing up, you should try disabling the avisynth script, and simply create a 16 bit output 3DLUT file to use with madVR. This should give you better performance, and better image quality.

[janos666]

Thanks for the detailed reply but I have to honor it with more questions ans corrections for misunderstandings. :)

You should change the line above for SD sources.
(Read the ReadMe for further details)

I already read that. And I am sorry, it was my mistake. I want to write: "HD - PC.3dlut" and not "HD - SD.t3dlut". You can see that I talked about 1080p and 720p movies later. But I made both SD and HD files with PC levels.

If you have low cpu power available you have to stick with 8 bit output.

I can live with that. (I am not sure about this dithering anyway. It usually means "noise" for me, not quality, and I am irritable about noise. But movies does not contain figures with color gradients. It is not so easy to imagine it with random colors on a random picture. I did not compare the results visually, yet.)

My concrete question supposed to be: Is it normal if my Q6600 (which runs at 3600Mhz with FSB450) can not handle a Blu-Ray movie with 16-bit 3dlut? I did not see another posts about this kind of performance problems in this topic and I think it is not me who has the slowest CPU here. So, I assumed it could be some kind of compatibility problem which causes high CPU load on my system. (Like I should use a specific AviSynth/FFDShow/MCP-HC build and not the last one, ect.) There is no any other active background task.

When you change the gamma curve, you have to correct the brightness level of your display, that's why it looks brighter.

Oh. This is not a good news because I would not do that. And it does not sounds like a correct solution anyway!
I calibrated my display to 140cd/m^2 because this was the default value in the calibration software. I would keep this white luminance value because it feels good for everyday usage (document reading/editing, web browsing, ect.)
Is there an exact value in the ITU Rec BT.709 standard? I could not see that. It should be a varying number which considered by the user (this is a function of ambient light conditions, personal feeling, ect.)

I can not go below 120 cd/m^2 anyway, it won't help now. The white luminance value with factory default OSD settings was ~280 cd/m^2, nearly two times more than it is after the calibration. And black was black (and not gray) in the movies with 0-255 output level settings in FFDShow. (Of course without t3dlut. I started to deal with this after I bought my calibrator...)

I think that brightness has nothing to do with this.
The image looks exactly when I miss the PC-TV levels. (Like the program ignores the PC level settings when it uses custom gamma settings.)
Or do you mention about some software brightness control in the player? That would not be a nice thing anyway.

HDTV is BT.709
Yes. And I would ask question about that.
Does this standard have specific tonal response curve (like the sRGB standard)? Isn't it match with some x^y (like gamma 2.22), or sRGB curve? Is there any EyeOne compatible software which can calibrate to this target?

madVR only works with 16 bit output 3DLUT files.
I thought about that. But I tried to use madvr first. When it refused to load the 3dlut files I came here to learn about them. I found the t3dlut plugin and it was later when I figured it out that I have to reduce the bit depth for smooth Blu-Ray playback...
It is not well documented how to use 3dluts with madvr. I could see 3 kind of template file labeling: "HD_PC" ; "HD - PC" ; "template HD-PC". I do not know what would be the correct file name for 3dlut files and where should I place them. (It was tricky to figure out that here is no real madvr installer, it is only a shortcut to register the filter, no matter where it is located momentarily, blahhh -> I know the install directory now, I only mentioned that...)

ffdshow converts the RGB32 output from t3dlut to YV12

I thought about that but I could not imagine that it could be true. Softwares can be smart only when they confuse the user with that. :p
Here is another CPU eater...
Is it a lossless conversion or does it have deficit? (Not if I would like to keep with this.)

*** May be it will solve some of my other problems if I eliminate this conversion. (Like the brightness/gamma/TV-PC level shit...) :)

create a 16 bit output 3DLUT file to use with madVR
This would be my goal. But I have another problems with madvr. (But I can solve them when I will have more free CPU power, so...)


UPDATE: I corrected the FFDShow output settings to RGB32 only. I have less CPU load with 8-bit 3dluts, I will try to generate 16-bit one again.
But black areas are still "washed out" with Output_Gamma 9 1.0 0.0 0.45 0.0. (The monitor is still calibrated to x^2.22, 140cd/m^2)
And the first (full black) frame of a movie tell the truth: It looks exactly the same when I miss PC-TV levels. I can see a gray rectangular between two black rectangulars. (16:9 source on 16:10 display...)
UPDATE2: And no. I can not play 1080p Blu-Ray movies with 16-bit 3dluts.

UPDATE3: I could solve my problems with madrv. It works well now. (I had to disable something in FFDShow...)
MadVR gives me better result with 9 1.0 0.0 0.45 0.0 and PC output settings than t3dlut. But this is not perfect yet. (At least, it won't miss the output levels, so black is nearly black and not gray now.)
There is some strange (and intensive) noise on dark areas with custom gamma settings. (I could see something like this when I watched old SD rips with very low bitrates.) It is not correct. I can not see this noise without custom gamma settings, and black is perfectly black when I clear this line with output_gamma.

Should I try to recalibrate my display with L* or sRGB tonal response curve and use one of these presets in cr3dlut for output_gamma?

[yesgrey]

My concrete question supposed to be: Is it normal if my Q6600 (which runs at 3600Mhz with FSB450) can not handle a Blu-Ray movie with 16-bit 3dlut?
No. You have to be doing something wrong...

I calibrated my display to 140cd/m^2 because this was the default value in the calibration software. I would keep this white luminance value because it feels good for everyday usage (document reading/editing, web browsing, ect.)
When you change gamma, you change the black level. Unless you recalibrate your brightness level you will get bad black levels. Though, considering this is only for watching movies and for your day-to-day usage you will use a different gamma, it's better keeping the gamma untouched.;)

UPDATE3: I could solve my problems with madrv. It works well now. (I had to disable something in FFDShow...)
MadVR gives me better result with 9 1.0 0.0 0.45 0.0 and PC output settings than t3dlut. But this is not perfect yet. (At least, it won't miss the output levels, so black is nearly black and not gray now.)
There is some strange (and intensive) noise on dark areas with custom gamma settings. (I could see something like this when I watched old SD rips with very low bitrates.) It is not correct. I can not see this noise without custom gamma settings, and black is perfectly black when I clear this line with output_gamma.

Should I try to recalibrate my display with L* or sRGB tonal response curve and use one of these presets in cr3dlut for output_gamma?
No. That's a problem that I know for a while that I'm trying to find a solution, but I think it would not be possible.
The problem is not from using custom settings, the problem is when someone use an output gamma curve with a low level segment different from the input gamma curve. For now, just use as output gamma the same as input gamma or, if you want it, try this:
9 4.5 0.099 0.45 0.018
For different gamma values, just change the 0.45 value for any value between [0.4, 0.5]

Note 1: performance wise, you should disable the avisynth filter and resizing in ffdshow, and do it all with madVR.

Note 2: Due to your updates, I've not answered some of the other questions.

[janos666]

I calibrated my display with sRGB tonal response curve and I used this input file for "HD - PC.3dlut":

# Source video format
Input_Video_Format HD YCbCr
Input_Bit_Depth 8

# Display video format
Output_Video_Format HD RGB_PC
Output_Bit_Depth 16

# Gamut and Gamma Conversion - settings
Chromatic_Adaptation 2
Out_Of_Gamut_Clipping 0
Output_Gamma 0
Input_YCbCr_Full_Range 0
Output_RGB_Black_White 0 255

# Gamut conversion - measured native coordinates
Output_Primaries 9 0.658 0.328 0.230 0.691 0.141 0.087 0.313 0.328

It should be perfect but it is not. But I am close to see the source of the problem. I think that cr3dlut handles this PC-TV range thing incorrectly when it also does gamma correction.
I watched Pandorum (Blu-Ray disk) some days ago with t3dlut and VRM9 (8-bit RGB table without gamma correction) and I used it to test my settings with madVR today, because this movie has a lot of dark scenes.
I can see very odd things. Sometimes I can see a very dark scene where black is true black but every black area changes to noisy gray as soon as a little light source shows up on the screen. It can render true black but any little light will push it to gray. It is a noisy gray because the little differences (between dark and dark areas) are magnified. There is a very big jump between black and the darkest gray.
I guess YCrCb=RBG looks like this: 0=0 and 1=4, 2=8, 3=16, or 0=0, 1=16, 2=17, and so on. (And dithering may help to make it more noisy.)
I tried to change it but Out_Of_Gamut_Clipping has nothing to do with this problem. The Reducing of RGB White to 235 helped a bit, but the result was far from good. The Increasing of RGB Black wont help, and negative values won't make sense here...
So I removed this gamma setting for now. Dark scenes are more natural without it.
This gamut conversion helped a lot, a more accurate gamma won't make big difference (the Rec709 tonal response curve is not far from sRGB one). But I have to live with this knowledge that it is not perfect yet. :rolleyes:

SUPPLEMENT: Sorry, I already wrote this post when I noticed your answer.
No. That's a problem that I know for a while that I'm trying to find a solution, but I think it would not be possible.
Did You try to change the order between TV->PC and source_gamma>output_gamma conversion precess? May be gamma correction should be done first.

[yesgrey]

I think that cr3dlut handles this PC-TV range thing incorrectly when it also does gamma correction.

Did You tried to change the order between TV->PC and source_gamma>output_gamma conversion precess? May be gamma correction should be done first.
Thanks for trying to help, but that's not the problem.
Currently I think the problem is related to the bit depth.
At lower levels, the 8 bit bit depth is too low for the small changes.
If we don't change the gamma curve for the low level part, there is no problem, because all the fine gradation levels are kept, but when we change the low level part of the gamma function, there is a problem, because some of the low level detail is completely lost, because some values will be coded as the same output level instead of different ones, hence the blocky images in the dark areas.
I still don't know if this can be solved with the new clipping method I am working with, but I'm afraid that it's not.
So, for now, we should always use the same gamma function for input and output. If you want, you could try changing the gamma value, as I showed you in my previous post, but not the gamma function.

[janos666]

Currently I think the problem is related to the bit depth.
At lower levels, the 8 bit bit depth is too low for the small changes.


It is not hard to imagine as soon as we are in the same interval. But I thought that the extra 16 steps (between PC and TV levels) can span this limitation. So the steps between 16 and K (during this non linear gamma function is effective) can be stretched over this 0-K interval (which is approximately 1,5-2 times wider than the original interval. - It is my guess only, I did not calculated it...)
But ok, I stop giving advices because I can not reflect my thoughts in this language and I do not have experience with this kind of transformations...

[yesgrey]

But I thought that the extra 16 steps (between PC and TV levels) can span this limitation.
No, because I think the problem only happens when you are outputting PC levels. The bit depth seems to be too low to allow the conversion... but it also could be any problem with the clipping, so without further testing is not easy to know.

But ok, I stop giving advices because I can not reflect my thoughts in this language[/COLOR]
Ideas are always welcome, help me think of what might be the reason. It's also hard for me to reflect my thoughts in this language, so just try, like I do.;)

[janos666]

No, because I think the problem only happens when you are outputting PC levels. The bit depth seems to be too low to allow the conversion...

It sounds like a proof that problem is elsewhere. I also noticed that there are no noisy gray blocks with TV output levels. Of course, it is always too bright on my PC monitor and it never hits real black but the transition between gray levels is smooth. How can it loose so much detail when it is extended to a wider interval?
May be RGB values shouldn't be extended but simply reduced by 16 in this tricky 0-0.081 interval and the remained K;235 interval should be extended to fill the (K-16);255 interval. (In this case, gamma correction should done first in the original range.)


But I am not sure if this conversion is needed or not. I found this info when I searched on the web for HDTV transfer functions:

My understanding is that Rec709 is a camera transfer function, not a display transfer function. So while the gamma curves detailed above do define the input reference of a Rec709 image, those images are then presumed to be displayed at gamma 2.2

In my understanding this document from ICC confirms this statement: http://www.color.org/sRGB.xalter The most interesting paragraph is: sRGB and ITU-R BT.709 Compatibility
It supposes to clarify the confusions but this document isn't consistent anyway. :rolleyes:
Sometimes it says:
we can solve for the ideal target monitor gamma of 2.2
It is not always clear that it speaks about the whole sRGB standard or the sRGB color-space only.
sRGB color space provides a monitor definition that can be used independently from the ITU-R BT.709 standard while maintain compatibility
But sRGB and Rec 709 have the same primaries so their compatibility shouldn't be further clarified.
And gamma 2.2, again:
In summary, there has been some concern with the choice of a 2.2 CRT gamma with a 1.0 LUT gamma as opposed to a 1.571 (2.2/1.4) or a 1.294 (2.2/1.7) display gamma. We feel that there are many reasons to support a 2.2 CRT, including;
- PC's with 256+ colors
- HDTV
- [....many more...]

The biggest conflict in this document is that the same image cannot be perfect with both sRGB and Gamma 2.2 because they are close but different tonal response curves. And it doesn't mention any color management (with gamma correction...) for HD video playback on PC. This is your "innovation".

On the other hand there are another conflicts with calibration softwares:
HCFR knows the Rec 709 (and some other) standard color spaces but it uses a reference gamma (like x^2.2) and not complex tonal response curves.
I found some information about an X-Rite software which accepts user defined gamma functions with mathematical formulas, and someone made an XML with Rec709 formulas.
Cheaper and bundle calibration softwares do not care about the sRGB curve. So a lot of people calibrate their displays with x^2.2 and working happily (Yes, working, not just watching movies. There is very few people like us who buy a colorimeter to enjoy movies...)


In my experience Blu-Ray movies looked better with gamma 2.22 than they looks now with sRGB calibration. (I am speaking about monitor calibration here...)
I can see more details in dark areas now but the whole image was sharper and more coherent for me with x^2.22 tonal response. (It applies to movies and PDF documents as well.)

[IanB]

The classic cheat for this is like sRGB does it.

Have a small linear region at the black end then change to the gamma curve. i.e.Y=[16..235] => y=[0..219] being the zero based luma values

(y<10) ? (y*2) : ((y/219)**(1/2.2)-(10/219)**(1/2.2))/(1-(10/219)**(1/2.2))*((219-10*2)/219)+10*2

2.2 being the Gamma value
10 being the size of the linear region
2 being the linear region coefficient

Example lookup values :-
In Gam Lin/Gam
0 0 0
1 19 2
2 26 4
3 31 6
4 36 8
5 39 10
6 43 12
7 46 14
8 49 16
9 51 18
10 54 21
11 56 24
12 58 27
13 61 29
14 63 32
15 65 34
16 67 36
17 69 39
18 70 41
19 72 43
20 74 45
...
58 120 100
59 121 101
60 122 102
61 122 103
62 123 104
63 124 105
...
77 136 120
78 137 121
79 138 122
80 139 123
81 139 124
...
214 217 216
215 217 217
216 218 217
217 218 218
218 219 218
219 219 219

[htpc66]

I notice that the free program MonInfo from Entech provides chromaticity co-ordinates for xyz and white points.

Wonder if someone please guide on whether those co-ordinates could be used as input primaries to generate .3dlut files? If not, what would the relevance of those chromaticity co-ordinates given by MonInfo as far as color correction is concerned?

[yesgrey]

Wonder if someone please guide on whether those co-ordinates could be used as input primaries to generate .3dlut files?
I don't know if the chromaticity coordinates are accurate, but for using them is self-explanatory. The names are the same. Rx in moninfo is rx in cr3dlut, etc., so it's just copying the values to the cr3dlut settings file.

[janos666]

I decided to recalibrate my monitor. This time I wasn't lazy and checked back the results. I used the latest MPC-HC, FFDShow and MadVR builds without any post-process filters to measure the AVCDH video test palette with Color HCFR 2.1.
Here is the animated gif with 3 CIE diagrams: monitor's native, Out_Of_Gamut_Clipping 1 and 0: click (http://img12.tar.hu/janos666/img/76100716.gif)
It was worth to check this because I thought that the default clipping 1 setting is better. And I thought that it is a little more accurate. So, I can appreciate my friend's results with his U2410 in sRGB mode. (That display uses the results of the one time factory measures to do gamut conversion and that is more accurate than this software with fresh measures.).

[yesgrey]

So, I can appreciate my friend's results with his U2410 in sRGB mode. (That display uses the results of the one time factory measures to do gamut conversion and that is more accurate than this software with fresh measures.).
That's one of the reasons I've started the new project: this wasn't good enough, it has some limitations, and I want something a lot better.;)

[tritical]

I updated the 'rgb3dlut' function of ddcc to support 3DLUT format LUT files. Even though t3dlut has all of the functionality of rgb3dlut, it doesn't have assembly versions of some of the code paths making rgb3dlut preferable in some cases. I also updated ddcc to output 3DLUT format files, and fixed a small bug in t3dlut for yuy2 input with cplaceU=0/itypeU=1. All are available at the usual place.

I'm considering writing a standalone program that would create a full 256*256*256 entry LUT for use with rgb3dlut/t3dlut/madvr from a partial subset of mappings.

[leeperry]

I'm considering writing a standalone program that would create a full 256*256*256 entry LUT for use with rgb3dlut/t3dlut/madvr from a partial subset of mappings.
A simple GUI where you could input/load/save RGBW xy coordinates and choose the output gamut for direct use within mVR would be very nice indeed :cool:

[yesgrey]

We have started a project on sourceforge to host everything related to the 3DLUT (http://thr3dlut.sourceforge.net/) file format.

When madshi find the time he will start a new thread for discussing the compression of the 3DLUT files.

@tritical
If you also want to be part of the 3DLUT project admins let me know.

[yesgrey]

I've updated 3DLUT (http://thr3dlut.sourceforge.net/)'s sourceforge project by uploading the files to the "Download 3DLUT files" area.

Sorry for any trouble, but I'm new to this and previously only have put the files in the bazaar repository.

[yesgrey]

Updated 3DLUT (http://thr3dlut.sourceforge.net/)'s project: created license file.

[Yellow_]

Would ddcc & 3DLUT improve colourspace conversion from YV12 to RGB for DSLR video, h264 over AVS's usual ConvertTORGB()? Reading through the ReadMe for ddcc it appears there are more options for control? The Rec709 v EBU looks like something that may come into play or is DSLR h264 almost certainly Rec709?

[IanB]

AVS's usual ConvertTORGB() is a linear algebraic conversion. It offers 2 sets of colour coefficients Rec.601 and Rec.709 with either PC, [0..255], or TV, [16..235], levels scaling.

Ddcc and 3DLut offer arbitrary lookup table translation from each YUV value to a RGB value. A LUT can be generated to apply any linear or non-linear function you desire. The standard LUT generator programs offer a useful selection of standard conversions profiles. You could modify the source code to generate any conceivable LUT you desire. I seem to remember some discussion about csv files and Excel as a means to generate arbitrary LUT's.

[Yellow_]

I think ddcc also offers Bicubic conversion and others, as well a linear. I believe 2.6 (development) will have Bicubic as default?

Do you think there is any benefit in that respect, in pursuing different conversions in ddcc, just for the YV12 to RGB?

Thank you for the explaination, I'll read more on 3D LUTS.

[yesgrey]

Would ddcc & 3DLUT improve colourspace conversion from YV12 to RGB for DSLR video, h264 over AVS's usual ConvertTORGB()?
If you use 16 bit 3DLUTs with t3dlut() then the quality should be better, because t3dlut dithers them to 8 bit, and I think the YV12->RGB conversion on ConvertToRGB() is performed using only 8 bit (please correct me if I'm wrong). This would avoid any banding that might be created during the conversion.

[yesgrey]

Do you think there is any benefit in that respect, in pursuing different conversions in ddcc, just for the YV12 to RGB?
For that you should take a look into t3dlut, and should use it in combination with yv12toyuy2. The chroma upsampling benefits from using the bicubic, we've tested it. However, it's just on images with very saturated reds in very dark backgrounds, so it might not be worthy the extra processing time...

[Yellow_]

yesgrey, thanks for the reply, I will look into t3dlut further, it sounds encouraging.

Is there a point in the process where 16bit can be exported out rather than dither to 8bit and do that later after grading?

With regard to saturated reds, I was looking at an example of that last night in a shot of mine from a 550D, http://blendervse.wordpress.com/2010/09/03/video-import-update/, difference between Fast_Bilinear & bicubic conversion, red / magenta on black gives a rather rough edge, where as the white on black was much smoother.

I understand that the Canon DSLR's are a bit red and can be calibrated by the user adjusting manual white balance and picture style.

http://www.hurlbutvisuals.com/blog/2010/03/30/color-correction-put-your-best-foot-forward/

[yesgrey]

yesgrey, thanks for the reply, I will look into t3dlut further, it sounds encouraging.
Be aware that t3dlut is slower than rgb3dlut because is a more generic function, but if you want more quality, and the time is not a big issue, use it instead.

Is there a point in the process where 16bit can be exported out rather than dither to 8bit and do that later after grading?
No, because we are limited by Avisynth 8 bit processing. If/When Avisynth support 16 bit processing then it might be possible.

As it is now I only see two solutions for your problem:
(1) Try to do the grading in YV12 and then convert to RGB
or
(2) Try to create a 3DLUT that not only performs the YCbCr->RGB conversion but also performs the grading.

The (1) I don't know if are there any tools available for it, but I might be able to help you with (2) if you could specify a set of changes to each of RGB channels that applies to the entire video...

[Yellow_]

yesgrey, thanks for your time. I think the best to achieve presently for me is the assumed improved YV12 to RGB conversion, I'll try it out and see.

I don't think (1) & (2) are very useable for me but thanks for suggestions.

Wilbert has a plugin called Immaav which uses ImageMagick to read but also write out 8 & 16bit to image files.

http://forum.doom9.org/showthread.php?t=135928

There's hope maybe some way of writing 16bit out via a Q16 build of IM? Although he thinks it's maybe not possible again due to AVS 8bit processing.

Anyway, very interesting processing and just improving YV12 to RGB over normal ConvertToRGB is a definite plus. :-)

[Yellow_]

I've started to give this ago, but I'm unsure as to chroma placement, left or centre? Source is progressive mpeg2 HDV and h264 in the conversion from YV12 to YUY2.

When I read through the t3dlut manual I can't find under mandatory settings a full range YCbCr 0 - 255 only 16 - 235. I'm aware most stuff is authored in that range, like BluRay, DVD etc, however I'm hoping to try the process on HD & DV vid cam source. I deliberately expose right up to hard clipping point using the waveform and histogram to gauge expsore, to get the most into the 8bit.

I'm almost certain both the camera's I use capture full range, depending on conditions. Certainly using AVS ConvertToRGB(PC.709...) gives me usable data in the headroom, not so much down below. I assume my camera sources are YCbCr, I've always used YV12 in AVS before. :-)

[poisondeathray]

I'm not sure if this is correct, but when I fidded with it , I generated a 3DLUT file from yCMS first. It's in that 3DLUT file that you can specify whatever range, and the characteristics etc...

http://forum.doom9.org/showthread.php?t=154719

Then I used t3dlut's YV12toYUY2(), and then t2dlut() with the 3DLUT file to do the RGB conversion

A while back , maybe 20 pages or so tritcal posted some comparison screenshots between the standard ConvertToRGB() vs. the other sampling methods. There was also discussion of whether left, or center is preferred.

Perhaps I wasn't doing it correctly , but my limited testing - assuming I'm doing it correctly - (along with tritical's comments) have shown it not that beneficial over the standard avisynth method ... but of course if you come to some new understanding or findings please share :)

[Yellow_]

Yeah, that's where I'm at really. I've got the AVS script sorted for the conversion except to confirm chroma placement.

Used YV12ToYUY2 from t3dlut and dropped out with AVS's ConvertToRGB and saw no difference tested against AVS's ConvertToYUY2, but hoping the whole chain via the LUT may yield better results, is that how you tested it?

The whole process I'd imagine has got to be useful coming back from RGB too after grading to delivery codec.

I've used a HD-PC template for the 3DLUT file but can't find a mandatory full range HD or SD option, so not got very far. :-(

Don't really want to get into the 'only use these settings if you know what you're doing' section of the LUT generation process.

A lot of reading to do. :-)

[yesgrey]

I've used a HD-PC template for the 3DLUT file but can't find a mandatory full range HD or SD option, so not got very far.
All HD and SD (Blu-ray, HD DVD, DVD and broadcasts) are not full range, hence the absence of such a setting from the mandatory commands. In the rare cases where it is needed, you need to use an advanced command.

Don't really want to get into the 'only use these settings if you know what you're doing' section of the LUT generation process.
You can always ask for help to achieve what you need... ;)

In your case, just add this line at the bottom of your file:
Input_Range 0 255
If you also want to guaranty full-range on the output side (the mandatory only allows the selection for RGB. YCbCr is always standard) add:
Output_Range 0 255

[Yellow_]

yesgrey,

re Input_Range 0 255 & Output_Range 0 255, that's nice and simple. :-)

I did read through the custom settings but saw mention that mandatory would overide incomplete custom choices so sort of backed off using them for now until I can understand them more, if necessary.

My other query re post #582 was that the sources are h264 and mpeg2 hdv progressive, so not sure whether it should be left or centre chroma placement.

Looking forward to trying this all out later and thank you for your time.

[yesgrey]

I did read through the custom settings but saw mention that mandatory would overide incomplete custom choices
No, you misunderstood it. What it says is that if you only use some of the advanced commands you still need to include the mandatory command in the file. As long as you put any custom commands below the mandatory commands, they will override their part of the settings.
The mandatory commands are some kind of internal scripts with default values for all the commands available.;)

My other query re post #582 was that the sources are h264 and mpeg2 hdv progressive, so not sure whether it should be left or centre chroma placement.
For H.264 is left, but for mpeg2 I don't remember. Read on avisynth's doc, there is a description of the different positions and when they apply. The problem is that some cameras use a different kind of chroma placement, so try to see which applies to your case.

Read this (http://avisynth.org/mediawiki/Sampling), it's a good place to start...

[Yellow_]

Ok, I have success and I think improvement based on one test. :-) So far.

Using t3dlut I managed to raise the number of unique colours by 23680, above the AVS ConvertToYUY2 -> ConvertToRGB. :-)

That was with bicubic coefs and set at 0.0 & 0.0

Using the defaults 0.0 & 0.5 it raised the unique colours by 25946.

Is it a fair assumption that the more unique colours generated the more resistant to grading and banding?

I've added the output frames on the blog if interested.

http://blendervse.wordpress.com/2010/09/10/ycbcr-to-rgb-by-3d-lut-via-avisynth/

Based on the full 1440x1080 .png frames, although they are just an example, is there anything else that could be done to improve general image quality of mpeg2 / h264, subjective I know, for example dealing with the blockiness, as seen in the sky, adding discreet noise, remove grain / add grain etc.

[poisondeathray]

How valid is the gimp colorcube analysis? And what does it really indicate ?

Is having more unique colors necessarily better ? e.g. if , because of the changed interploation the pixels are shifted slightly, or maybe use left instead of center, that may lead to more "unique colors", but perhaps less accurate


Is it a fair assumption that the more unique colours generated the more resistant to grading and banding?


You should test it .

For example, in other programs, you can do some levels or curves manipulations and watch the histogram along with the graded picture. It will band up more (stair steps in the histogram) with 8-bit footage compared to 8-bit footage interpolated correctly to 10-bit footage . I'm sure you can try the same thing in blender . Some programs (e.g. nuke) can take 8-bit images and work in 32-bit float then dither down to 8-bit when exporting (or you can export 32bit or 16bit images as well) , I'm not that familiar with blender or how it works internally

I doubt it will make that much of a difference (at least in terms of banding) by using using different YV12=> 8bit RGB methods . I suspect there will be zero visible difference by using avisynth converttorgb() vs. t3dlut in terms of banding because you are limited by 8-bit RGB (2^8 =256 "shades" for each channel component)

[Yellow_]

Yes, I ask the same question, but for the time being I use it as a 'measurement', it's just a statistic and my feeling is that 'proof' is in the learning process which comes with manipulating the image source and seeing what it will and won't stand. :-) It does come across as if I'm trying to proof by colour count I know. :-)

[poisondeathray]

I couldn't find any more information on it (colorcube analysis), but I did start playing with it because of another thread you mentioned it in

Just "eyeballing" the png images at various zooms , it's not a big difference (slight shift) , but not even as big as the red on black letter images earlier in the thread

In terms of grading, and actually using the images in blender - I'm still interested in seeing what comes out of your tests , please keep sharing your findings :)

[Yellow_]

How valid is the gimp colorcube analysis? And what does it really indicate ?

Is having more unique colors necessarily better ? e.g. if , because of the changed interploation the pixels are shifted slightly, or maybe use left instead of center, that may lead to more "unique colors", but perhaps less accurate


Cross posts. :-)


You should test it .

For example, in other programs, you can do some levels or curves manipulations and watch the histogram along with the graded picture. It will band up more (stair steps in the histogram) with 8-bit footage compared to 8-bit footage interpolated correctly to 10-bit footage . I'm sure you can try the same thing in blender . Some programs (e.g. nuke) can take 8-bit images and work in 32-bit float then dither down to 8-bit when exporting (or you can export 32bit or 16bit images as well) , I'm not that familiar with blender or how it works internally

Blender works 32bit float internally and has basic colour management that is just sRGB and Linear at the moment, but next iteration will probably be 3D LUT based rather than hardcoded as it is now, also possible to export DPX (gamma encoded and log) and OpenEXR (multilayer , linear too). All image imports are converted to linear colour space and all compositing / lighting / rendering calculations are done in linear colour space. OpenCL is beginning to appear for the compositing pipeline as a GSOC project, so 3 way colour correct, curves etc hopefully soon to be GPU accelerated.

I doubt it will make that much of a difference (at least in terms of banding) by using using different YV12=> 8bit RGB methods . I suspect there will be zero visible difference by using avisynth converttorgb() vs. t3dlut in terms of banding because you are limited by 8-bit RGB (2^8 =256 "shades" for each channel component)

Yes, more tests needed and thanks for the encouragement. :-)

[Dogway]

There's something strange on some png's I exported from Final Cut, they look clearer on avisynth and the default Windows Viewer, than some other applications like Photoshop, or Nuke, being the latter the correct look.
I dont understand why this happens, its 1080pixels and I have a calibrated monitor with its profile.

1Mb Sendspace rar:
(I pack the image so no color conversion is made upon browser.)
http://www.sendspace.com/file/lwh4rl

Actually it looks just if I disabled proof->monitorRGB on photoshop thus ignoring my monitor profile.

[Dogway]

Someone please?

[cretindesalpes]

Your PNG has a gamma correction profile attached, which interpretation may vary, depending on the displaying application. Use TweakPNG (http://entropymine.com/jason/tweakpng/) to remove it manually. FFmpeg will also work fine for batch conversion.

[Dogway]

Didnt notice the answer, yes I found out there was a gamma issue, and by some heavy thinking decided to tweak gamma by 1.25 to accomplish the desired 2.2, (1.8*1.25=2.2). That makes my image clearer, and then it shows a more natural look with more tonal range and hue shifts. But as you suggested I just used TweakPNG and corrected the internal 0.555 gamma to 0.454 and what I get is the darker version. So now I dont know where my logic goes.

Or maybe the internal gamma correction of 0.555 is correct and the software showing the clearer version are doing the proper thing because they are profile aware(?), in which case I dont need to do anything?

[Yellow_]

@Dogway, what are you trying to achieve? or are you sorted now?

[Dogway]

Kind of. Im color grading that footage, just want to be sure Im working in the original/tonal range of the source.

What I have are a png sequence and a prores encoded video footage.

-The pngs at first glance are darker, only avisynth and the default windows image viewer read the built in 0.555 gamma correction thus clearing up the image.
-The video footage is already clear, I think because no yuv->rgb conversion was done.

So Im assuming the clearer version is the correct, and will work better with the yuv source because it has a wider tonal range to begin with. I already converted the video source to png (yuv->rgb) through avisynth and grading in Nuke with that. Just correct me if Im wrong.

[Yellow_]

Ok, I don't quite get it? Why gamma correction from FCP to png's?

Are you trying to linearize your png's, with a reverse gamma correction?

It's straightforward to export png's from ProRes with AVISynth using the latest QTSource plugin.

Your frame image looks like it's had the levels messed with, the histogram looks rough. It doesn't look like it's been cleanly converted from the original.

[Dogway]

Yes, that's what I did, exported from Prores to png in avisynth.
You're true, good idea the levels thing, yes, it looks tweaked. What further confirms the clearer version is the correct. Don't ask me why gamma correction from FCP to png, I don't use that program, so I just wanted to hint what the embedded 0.555 gamma thing on the png meant and choosing the right one to grade on Nuke.