gamut conversions through Avisynth ?

[pitch.fr]

hello world,

there's a new SAMSUNG projector, that's ISF(Imaging Science Foundation) certified and made under the supervision of Joe Kane(he's quite a video guru)

that's the SP-A800B(DLP 1080p), and as they say "Paramount, DreamWorks, ABC, and Universal Studios are among the professional users of this model."

http://www.projectorcentral.com/Samsung-SP-A800B.htm

it's one of the very few projectors(if not the only one) where you can change the gamut on the fly.

mostly, the SD movies are encoded through the REC601 matrix, and the HD movies through REC709.

but this is the theory...

in practice(and I know that sounds crazy), the mastering engineers still use this kind of monitoring displays :

http://www.soundandvisionmag.com/assets/image/2007/W49/1242007175956.jpg

the engineer is using a BVM CRT monitor(with SMPTE-C phosphores gamut)

this thing :
http://assets.sonybiz.net/products/BVM-A14F5M(img1).jpg
http://www.sony.co.uk/biz/view/ShowProduct.action?BIZ_SESSIONID=yndHL19ZmGTb7HJwGSbJyQRmYPJYqtv1GjHhgvYr6TzXZNjLbf26!-1724456250&product=BVM-A14F5M&pageType=Overview&category=BVM&site=biz_en_GB

so basically, SD and HD movies are not mastered on sRGB/HDTV gamuts as we've all been led to believe.

video monitors used in Europe :
SONY BVM-A14F5M (EBU phosphores)
SONY BVM-A20F1M (EBU phosphores)
SONY BVM-A32E1WM (EBU phosphores)
SONY BVM-L230 (LCD)
SONY BVM-L420 (LCD)

and in USA:
SONY BVM-A20F1U(SMPTE-C phosphores)


so basically, to see movies the way they were mastered, you'd need :

1)YCbPr decoding :

- Rec. ITU-R BT.601-5 for SD primaries (in theory only)
Y’ = 0,299R’ + 0,587G’ + 0,114B’ (601)

- Rec. ITU-R BT.709-4 for HD primaries (in theory only)
Y’ = 0,2126R’ + 0,7152G’ + 0,0722B’ (709)

2)display gamut :

- REC-601 SMPTE-C for SMPTE-C primaries (NTSC SD & HD, 90% of bluray discs as well, and some PAL DVD's)

- REC-601 EBU Tech. 3213 for EBU primaries (PAL/SECAM SD and european bluray discs)

- REC-709 HDTV (for US and EUR HDTV)


that's what my DLP projector looks like, when calibrated in sRGB(the gamut is identical to the HDTV gamut) :

http://pix.nofrag.com/a/3/c/deff50ab2a4b5c063c43e15a16155.png

and here's a gamut comparison :

http://www.homecinema-fr.com/forum/download/file.php?id=54227

so is there any AVS guru that'd be interested to find a way to get 1:1 colors with the way movies are being mastered in the first place ?

of course it'd have to 10bit if possible in order to avoid banding and stuff.

I know that sounds crazy, but that's the way it is.
this Samsung projector is the ultimate thing to watch movies "the way they were meant to be watched"(it supports sRGB/HDTV/EBU/SMPTE/etc.. gamuts).

surely enough, you could revert this colorspace hiccup in AVS ?

....and run it in realtime in ffdshow ? that'd be the shiznit :D

here's a list made by the french ISF CEO "Julien Berry", that lists which gamut has to be used for each Bluray movie :

http://www.wysios.com/jkp/gamut2.asp

from what he told me, here's what happens when using a display with sRGB/HDTV gamut :

for telecines that were made on an EBU monitor, only the green primary will be off.

for telecines that were made on an SMPTE-C monitor, the colors will be over-saturated.

when you read that most of the bluray's are made on SMPTE-C monitors, and that usually you end up with over-saturated colors on sRGB/HDTV displays.....that suddenly would make a lot of sense to fix the colors.....wouldn't it ?

here's a technical PDF about gamuts :
http://www.teksite.co.kr/osilo/down/Understandin%20color%20and%20gamut.pdf


and basically you can do this gamut conversion with a scaler :
http://www.google.com/search?hl=en&q=gamut+conversion&btnG=Search&lr=

its job is to convert gamuts, but that costs a hell lot of money(and usually only SDI input and DVI output).....so doing it in AVS in 10bit would be beyond words :)

some ppl have worked on it with PS scripts :
http://www.avsforum.com/avs-vb/showthread.php?t=912720

the idea would be to :

-do REC-601/REC-709 matrix conversions for SD/HD to get sRGB content
-do REC-601 SMPTE-C / REC-601 EBU Tech. 3213 > HDTV gamut conversions to get proper primaries on sRGB/HDTV displays :)

[pitch.fr]

an even easier solution would be to input the primaries locations(after calibration) and input them in AVISYNTH :
http://www.avsforum.com/avs-vb/showthread.php?p=11726737#post11726737

is that doable ?

here are the primaries locations on my sRGB CRT :
R:[0.658,0.328]; G:[0.334,0.619]; B:[0.150,0.070]

EDIT : another option might be to use ColorMatrix.
but it doesn't support either "REC-601 EBU" or "REC-601 SMPTE-C" :(

anyhow here's a BT.709>RGB32 conversion(through ffdshow)
+ the same with ColorMatrix(mode="Rec.601->Rec.709") added upfront :

http://thumbnails8.imagebam.com/915/73e4979147374.gif (http://www.imagebam.com/image/73e4979147374)http://thumbnails8.imagebam.com/915/0b83a59147373.gif (http://www.imagebam.com/image/0b83a59147373)
http://thumbnails8.imagebam.com/915/7778be9147377.gif (http://www.imagebam.com/image/7778be9147377)http://thumbnails8.imagebam.com/915/36e09b9147376.gif (http://www.imagebam.com/image/36e09b9147376)
http://thumbnails8.imagebam.com/915/a0f7ef9147379.gif (http://www.imagebam.com/image/a0f7ef9147379)http://thumbnails8.imagebam.com/915/30d22c9147378.gif (http://www.imagebam.com/image/30d22c9147378)

I would say that it looks A LOT better on my sRGB/HDTV calibrated displays :)

this is a US bluray source btw, so most likely telecined in a "REC-601 SMPTE-C" gamut ;)

hopefully I've convinced a few people now :D

[yesgrey]

some ppl have worked on it with PS scripts....but they are confusing "decoding matrix" and "display gamut" :
http://www.avsforum.com/avs-vb/showthread.php?t=912720

It's me who have started the AVSForum thread you refer, and no, I am not confusing "decoding matrix" and "display gamut".

It seems to be you who are confusing "encoding matrix" and "monitoring color gamut". The authoring process being monitored with a different color gamut, does not means that the colors are wrong. They could be correcting the colors on their monitoring displays the same way I suggested in that thread.

All the thread is considering the mastering process is monitored with displays the same color gamut of the encoding matrix; what you are saying is that it's not done that way. In fact, if the above correction in the monitoring displays is not performed, the correction should be a little different, but the math is all there, we just need to adapt it to this situation. I will take a look and update the thread accordingly.

When I started the thread I started using avisynth, but when I realized the correction must be done in the color linear space JohnAd suggested to use pixel shaders and mplayerc.
It seems to be the better option for now, you have it already done and working, why requesting it in avisynth?

[pitch.fr]

well many people speak about different ways to do it in this thread, sorry I didn't understand it this way :(

anyhow, great to have you here :)

tritical is improving ColorMatrix to allow this kind of CIExyz conversions on the fly in AVS/ffdshow :)

well, for one I only use Haali's Renderer, which is the de facto video renderer on PC to get smooth video....so these PS scripts are not usable.

and ColorMatrix will allow to input the primaries coordinates we get from colorimetry applications, so it will offer tailor-made gamut conversions between SMPTE-C > your own calibrated display for 1:1 colors(no need for an expensive scaler anymore).

well the ISF CEO and many other professionals are showing that the only way to watch bluray with 1:1 colors is to do REC.709 conversion, and watch it on an SMPTE-C gamut if it was mastered in the US, and an EBU gamut if it was done in europe.

these guys are calibrating broadcasting studios with Minolta CS colorimeters all over the world, and they are pushing this new Samsung as the ultimate projector because it supports any kind of gamut you could possibly want :eek:

even PLANAR does that now with their newest projectors as well.

you can find above a list with all the bluray that are supposed to be watched on an SMPTE-C gamut.

OTOH, the only thing you need to use this kind of PS script is 2D convolution, that some AVS plugins can do.....only problem is that converting to RGB in AVS is single threaded and very slow compared to ffdshow, and you also get those pesky "chroma upsampling bugs" :
http://forum.doom9.org/showpost.php?p=1137196&postcount=1868

[pitch.fr]

alright, so this PS script is really nice :eek:

http://www.avsforum.com/avs-vb/showthread.php?t=912720

...too bad that doesn't work with HR :(

here's my 19" iiyama CRT calibrated CIE :

http://thumbnails9.imagebam.com/931/c3ad2b9303768.gif (http://www.imagebam.com/image/c3ad2b9303768)

here's my XLS script with the primaries coordinates measured with an Eye One Display 2 colorimeter in Color.HCFR :

http://thumbnails9.imagebam.com/931/fdb6989303770.gif (http://www.imagebam.com/image/fdb6989303770)

and here's the correction from SMPTE-C to my CRT gamut, with REC.709 RGB32HQ HD content from ffdshow in VMR9.

left is untouched, right is corrected to match my CRT gamut.

http://thumbnails9.imagebam.com/931/4ad07f9303793.gif (http://www.imagebam.com/image/4ad07f9303793)http://thumbnails9.imagebam.com/931/8bfb8d9303796.gif (http://www.imagebam.com/image/8bfb8d9303796)
http://thumbnails9.imagebam.com/931/76e0769303773.gif (http://www.imagebam.com/image/76e0769303773)http://thumbnails9.imagebam.com/931/01b8319303776.gif (http://www.imagebam.com/image/01b8319303776)
http://thumbnails9.imagebam.com/931/b562359303778.gif (http://www.imagebam.com/image/b562359303778)http://thumbnails9.imagebam.com/931/7e556c9303781.gif (http://www.imagebam.com/image/7e556c9303781)
http://thumbnails9.imagebam.com/931/25028c9303784.gif (http://www.imagebam.com/image/25028c9303784)http://thumbnails9.imagebam.com/931/10f5e19303785.gif (http://www.imagebam.com/image/10f5e19303785)
http://thumbnails9.imagebam.com/931/2aedfd9303788.gif (http://www.imagebam.com/image/2aedfd9303788)http://thumbnails9.imagebam.com/931/3d12c09303791.gif (http://www.imagebam.com/image/3d12c09303791)

I'd say it removes the green cast on faces(my biggest problem at this point!), and it gives much more natural colors :cool:

too bad my red level is now lower, and because it's 8bit/pixel this could possibly introduce banding.

that's also too bad that I've never managed to get good results with Reclock in VMR9......HR is way smoother, so this is not a viable solution for me at this point.

there is some AVS plugins that can do 2D convolution :
http://www.google.com/search?hl=en&q=avisynth+convolution&btnG=Search&lr=

but as I said earlier RGB32 conversion in AVS is slow as hell(not multithreaded and/or optimized), and full of "chroma upsampling bugs".......except if tritical can make miracles happen :)

otherwise maybe you could create an .icm file ?
or a .cal file to be opened with dispwin.exe(part of ARGYLLCMS www.argyllcms.com) ?

ATi's CLUT is 30bit AFAIK :)

ARGYLL's coder is very knowledgeable and always willing to help :)

it's still 8bit/pixel, but better get proper 8bit corrected colors than greenish movies :D

[Wilbert]

but as I said earlier RGB32 conversion in AVS is slow as hell, and full of "chroma upsampling bugs"
If you feed interlaced material, you need to use

ConvertToRGB32(interlaced=true)

[pitch.fr]

ok, thanks for the tip but I've never managed to get RGB32 content out of AVS w/o chroma bugs.

here's a good test sample :
http://forum.doom9.org/showpost.php?p=1137196&postcount=1868

only HR and ffdshow(with "high quality YV12 to RGB conversion" checked) manage to do bug-free conversions AFAIK...

and HR/ffdshow BT.709 matrixes are also different.....ffdshow's is 1 notch greener, and 1 notch less blue and less red...

[Wilbert]

ok, thanks for the tip but I've never managed to get RGB32 content out of AVS w/o chroma bugs.

here's a good test sample :
http://forum.doom9.org/showpost.php?...postcount=1868
Ok. Post a script and a screenshot which shows these chroma upsampling bugs.

[pitch.fr]

Ok. Post a script and a screenshot which shows these chroma upsampling bugs.

http://img207.imageshack.us/img207/2374/blaud9.png

there is also some AVS commands to increase the chroma upsampling, but that didn't work either.

I'm not an AVS guru, so maybe I'm missing something :D

what is required is "progressive chroma upsampling", at this point this is not done :eek:

[Didée]

That's fine and all, but where's the bug you're talking about? There's nothing wrong in your pic. :)

When converting YUV to RGB, it's a matter of choice if you replicate the subsampled UV planes 1:1, or if you interpolate during resampling. The former is what Avisynth and ffdshow(default) are doing, the latter is what ffdshow's HQ=true does.

Both methods are valid, there is no "right" or "wrong".

Definetly, it is not a bug. A missing option, yes. But not a bug.

[pitch.fr]

oh ok thanks for the clarification Didée...well everyone calls it "chroma upsampling bug" :
http://www.google.com/search?hl=en&q=%22chroma+upsampling+bug%22&btnG=Search&lr=

it basically makes the red blocky to hell...and I'm not watching HD to get blocky reds :D

so is there a way to overcome this issue in AVS ?

I think I read that when you checked "HQ" in ffdshow, it was using some YV12>YUY2 conversion script(from Avisynth actually).....I read it in the ffdshow thread :
http://forum.doom9.org/showpost.php?p=1106325&postcount=3250

anyhow ConvertToRGB32 can't be multithreaded, so this can't be used in realtime in ffdshow :(

[tritical]

pitch.fr how did you generate that image? Avisynth uses interpolation in its yv12->yuy2->rgb conversions, but the image for converttorgb32() looks more like replication was used. Btw, 'chroma upsampling bug' usually refers to using progressive upsampling for yv12->yuy2 when interlaced is needed or vice versa.

Maybe someone who knows for sure can comment on this, but I was under the assumption that the high quality rgb conversion in ffdshow actually used the yv12->yuy2->rgb conversion code from avisynth, and that the other colorspace conversions in ffdshow used xvid's conversion routines? I took a quick glance at ffdshow-tryout's subversion repository, and saw avisynth's/xvid's conversion code in there, but couldn't quickly see when each was called.

[pitch.fr]

hi tritical, did you get my PM ? any chance getting a reply please ? :D

well I was using Haali's Renderer in MPC, then I pressed the "print screen" key, zoomed to 300% in photoshop and did some cut/paste.

yes it seems that ffdshow is using some YV12>YUY2>RGB32 conversion when you tick the "HQ conversion" option, I dunno why it ends up like that with ConvertToRGB32 :(

anyhow, I'll try to watch a movie with EVR in MPC HC, Reclock and that PS script......but I don't think this will be as smooth as HR in 1920*1080 24Hz :(

[tritical]

Yeah, I'll get back to you, gonna go on a bike ride right now though. I quickly downloaded that file, and I get pretty much exactly the same image using ffdshow's hq rgb conversion versus using converttorgb24() in avisynth. image (http://bengal.missouri.edu/~kes25c/tt.bmp) (top is ffdshow default, second is ffdshow hq, third is converttorgb24()) (you might have to zoom in to see the differences). Are you sure that when you tested with converttorgb in avisynth that you didn't accidently still have ffdshow outputting rgb when it decoded the video or something along those lines?

[pitch.fr]

I didn't use ConvertToRGB24, only ConvertToRGB32.....as even if I do RGB24 in AVS, then I will need to do convert to 32 again....which will mean more CPU time required :(

I've just tried again with ConvertToRGB24(matrix="rec709") + RGB24>RGB32 conversion in ffdshow(as neither VMR9/EVR or HR will accept RGB24 input) and I still get pixelated red.....are you sure you got this output from ConvertToRGB24 ?!

maybe it's ffdshow messing the whole stuff up, then ? even tho I kinda doubt it :(

[yesgrey]

The problem is the color correction must be done in the linear space. So, after you convert from YUV to R'G'B', you must remove gamma and convert it to RGB, apply the color correction, reapply gamma and then output to your screen. This process has a big advantage, you can also perform gamma correction all in the same operation, to apply a gamma to the signal more indicated for your display.

This is not a Convolution 2D, is simply multiplying vectors and matrices. The problem is it being slow, and the PS have the great advantage of being highly multithreaded and all the clip math is already implemented, no cpu work.

I use reclock and it is great. Tell me more about your problems, maybe I can help you.

When I created that thread, I asked Haali to add that PS funcionallity to HR, since he is already using PS for the YUV->RGB conversion. He told me he would look into it, but never replyed me again about the subject, so I think he did not approve the idea... Let's keep using mplayerc and vmr9...

[tritical]

@pitch.fr
Something is wrong in your chain. Here is how I tested:

1.) ffdshow default. Use directshowsource("Bronz_s.mkv") in avisynth to open the mkv file using ffdshow, and set ffdshow to output RGB w/o hq conversion. Open in vdub.
2.) ffdshow hq. Same as method 1 setup, but check the hq rgb conversion box in ffdshow.
3.) avisynth conversion. Again use directshowsource, but make sure ffdshow is set to output YV12 (check that it is in vdub), and use converttorgb24 or converttorgb32 after directshowsource() in the avisynth script.

Using this, I get that 2 and 3 give almost identical output (image (http://bengal.missouri.edu/~kes25c/tt2.bmp)).

[pitch.fr]

Let's keep using mplayerc and vmr9...
yeah, except if tritical can make some highly multithreaded and optimized code, I don't see it working in realtime in ffdshow :(

anyhow, I've just run through all my test movies on my DLP projector.

I'd say the colors look way more realistic :eek:

the best example being an HD bollywood movie, where red clothes look too dark, dull and artificial and skin tones slightly too green.....and when I enable the tailor-made PS script, bam! it looks natural (again) :devil:

well my main issue with not using HR w/ Reclock is that it's simply not as smooth.....HR is just so smooth, that's quite amazing...

I can watch a 2H movie in 24fps@24Hz with very low jitter and no dropped frame.

EVR or VMR9 can't offer me this kind of super low jitter.

but I'll run more tests, considering I'm starting to enjoy getting 1:1 colors on a 2 meters wide projection screen :p

@pitch.fr
Something is wrong in your chain. Here is how I tested:

1.) ffdshow default. Use directshowsource("Bronz_s.mkv") in avisynth to open the mkv file using ffdshow, and set ffdshow to output RGB w/o hq conversion. Open in vdub.
2.) ffdshow hq. Same as method 1 setup, but check the hq rgb conversion box in ffdshow.
3.) avisynth conversion. Again use directshowsource, but make sure ffdshow is set to output YV12 (check that it is in vdub), and use converttorgb24 or converttorgb32 after directshowsource() in the avisynth script.

Using this, I get that 2 and 3 give almost identical output (image (http://bengal.missouri.edu/~kes25c/tt2.bmp)).
that's really weird ?! I'll run more tests tomorrow ?!

and could you do that gamut conversion in 10 bit in AVS(even if it's not officially supported yet) ?

[yesgrey]

yeah, except if tritical can make some highly multithreaded and optimized code, I don't see it working in realtime in ffdshow :(

It would be great if the gpu drivers could allow this. It would be so simple to them, but I don't believe they would do this kind of stuff to just a few guys...

I'm starting to enjoy getting 1:1 colors on a 2 meters wide projection screen :p
I understand you. My projector is a bit old (JVC M15), and the green is slightly off. It never bothered me. In fact, I never noticed it. After starting using mplayerc with the PS for correcting the colors, I started noticing and prefering the standard colors...:)

[yesgrey]

and could you do that gamut conversion in 10 bit in AVS(even if it's not officially supported yet) ?

For this the PS is better. I think it operates with more than 14 bit precision. But the biggest problem is we not having bit depths higher than 8 bit per color in the pc world. Due to this, in theory, you could get some banding with the color correction, in practice, I never noticed it.

[pitch.fr]

well even BD's are 8bit, there's nothing further on the market at this point AFAIK..

oh really 14bit ? I had no idea :)

well I've run some tests this evening, and EVR/VMR9 still can't compete with HR's smoothness...

maybe I'm willing to give up some smoothness to get perfect colors :rolleyes:

lucky you, your projector has a Xenon lamp so you get massive reds, due to my UHP lamp mine are orangey now :(

[tritical]

Whether avisynth is the right place or not, I created a filter to do the conversions. ddcc v1.2 (http://bengal.missouri.edu/~kes25c/ddcc.zip). It takes in rgb24 or rgb32, and does the following conversion:

gamma corrected RGB -> linear RGB -> CIE XYZ -> CIE XYZ -> linear RGB -> gamma corrected RGB

Chromaticity coordinates and transfer functions of the source and output are fully adjustable... some presets are built in. If white points of chromaticity coordinates differ, then a chromatic adaptation is done using Bradford matrix. More details are in the readme. I tested some, but not enough to really be sure it is correct (I simply implemented the conversions as I understand them). For this to run real time on HD video you're gonna need a processor with SSE3 support (I didn't write SSE or SSE2 routines, so its either SSE3 or the C routine), and probably more than 1 core (using only 1 core on my 2.8Ghz Q6600 doesn't quite make it). Hopefully someone can comment on whether the results are anywhere near correct :).

[yesgrey]

tritical,
I will take a look and update my AVSForum thread if it's working. It's nice to have other options.:)

Have you think about doing it using the GPU instead of the CPU? Since this is a highly parallelizable task the GPU could speed it up a lot!... The mplayerc Pixel Shader version works great without any cpu overhead...

Thanks for your work!

[pitch.fr]

cool thanks tritical, gonna give it a shot!

wow 32float, hope this will work in realtime :D

I've spoken to a friend of mine who's a PS script coder, and he said PS scripts also work in 32bit float :
http://msdn.microsoft.com/en-us/library/bb509646(VS.85).aspx

EDIT : ooh, your plugin is highly technical stuff :eek:

all I wanna do is map SMPTE-C to my DLP projector gamut, which is

Your Display xy

Red Green Blue White
x 0.656 0.339 0.151 0.311
y 0.329 0.611 0.068 0.328
z 0.015 0.05 0.781 0.361

maybe you could give some real life examples in the readme for the Avisynth challenged amongst us :D

well I can check very easily if the conversion is done properly, by doing a Color.HCFR calibration using the official test patterns DVD in MPC

I could even compare the MPC PS script and your plugin.

I'm gonna try to see if it works in realtime, but some little help on the syntax would be much appreciated :)

[pitch.fr]

well it's actually very usable in realtime with HR :eek:

man, 32bit gamut correction with HR and Reclock, this is too awesome :)

at this point I'm decoding h264 w/ ffdshow, then it enters the AVS filter in YV12, for 1) LSF 2) multithreaded ConvertToRGB32 conversion and 3) your colorimetry plugin.

[pitch.fr]

oh btw yesgrey3, tritical is using 6 digits after the coma for the xyz coeffs, may wanna update your script ?

for BT470-2 you have more digits after the coma than he does.......but you simply call it "NTSC" so I dunno if it's the same exact matrix.

and for "Offset" which is set to 24 in the XLS script, what's a good value ? what does it stand for ?

[tritical]

@yesgrey3
I'll look into it. Unfortunately, I don't have any experience using pixel shaders... so it may be unlikely.

@pitch.fr
Not sure if you figured this out already, but to map to a custom set of chromaticity coordinates you'll have to use the ofile parameter:

ddcc(ofile="file.txt")

file.txt would contain your values as follows:

0.151
0.068
0.781
0.339
0.611
0.050
0.656
0.329
0.015
0.311
0.328
0.361

You also need to specify the 5 values for the transfer function. You can just take one of the sets of values out of the readme, or to match the transfer function of the pixel shader code use the following five values:

1.0
0.0
0.45
0.0
0.0

I set the defaults of the filter to assume SMPTE-C input primaries and BT.709 input transfer function... so you'll probably also need to change the input transfer function depending on what values you use for the output transfer function.

Also, I put up v1.1... it fixes one small bug, and adds one new built in transfer function (gam_i/gam_o = 5) which matches the transfer function of the pixel shader code.

[pitch.fr]

great, thanks!

actually I've spent quite some time trying to find a way to run this thing in realtime in ffdshow :)

I've had to cut quite a lot of my audio enhancements in ffdshow audio(96KHz resample and stuff), because it sucks a lot more CPU to use your script + RGB32 conversion in the AVS filter, than to let ffdshow do the RGB32HQ conversion + using the PS script to fix the gamut :rolleyes:

what do you think is a good/best transfer function to choose ?
any theoritical improvement over the way the PS script works ?

and what do you guys think of that SAMSUNG pj that can auto-detect whether the YCbPr input is receiving BT.601 or BT.709 ?!

they've got to use some sort of auto-detection algorithm I guess ? or maybe they actually found a way to do that ?

AFAIK this is not possible :confused:

there is already a PS script parser for AVS :
http://www.google.com/search?hl=en&q=avisynth+pixel+shader&btnG=Search&lr=

maybe there is room for improvement ?

only the GPU works on these scripts, that's a good thing because I've always heard ppl saying that CPU's were pretty lousy DSP's

[yesgrey]

I'll look into it. Unfortunately, I don't have any experience using pixel shaders... so it may be unlikely.


I was not thinking exclusivelly in PS. CUDA would be too complicated? Haven't you tried something with the gpu for your NNEDI?;)

[tritical]

@yesgrey3
A CUDA implementation would be easy to do, but I was thinking that it would be pretty limiting (in terms of supported video cards) compared to a pixel shader.

@pitch.fr
The best choice of transfer functions would be the one corresponding to what was originally used to produce the gamma corrected RGB, and the one corresponding to the transfer function of your display. Of course, finding those out may be impossible. The one in the pixel shader code is just a generic one with ~2.222 display gamma and no linear segment... it's a reasonable choice if the true one is not known. I have no idea how autodetection of BT.601 vs BT.709 would work.

[pitch.fr]

yeah, I prefer the PQ of ATi cards........please no nvidia proprietary stuff :D

so what do u think of the PS AVS plugin, maybe a good starting point ?

I'm not sure I understand what you call the "Transfer Function", but it's 4AM here :D

you mean that it's a wild guess at what the gamma curve was before they encoded their RGB master to YV12 ?

I know what the gamma curve looks like on my DLP pj, it start at 2.3 and ends at 2.1

BUT the professional broadcasting equipment runs 2.5 gamma, that's the rule.......only consumer equipment is 2.22

do you know what I should change in the PS script generator to reflect this please ?

there's also many ways to compute gamma, and there's 2 formulas being used the most.

I've spoken about that with the HCFR color engineers, the most widely used formula seems to be this one :

y' = (y - black) / (white - black)
gamma = log (y') / log (x)
or y' = x ^gamma

which can also be written as follow
we agree that
Y = Ymin when V = 0
Y = Ymax when V = 1
gamma = log ((Y - Ymin) / (Ymax - Ymin)) / log (V)
for 0<V<1

if we wanna take the offset in account, we need to use the following formula :

Y = ((V+offset)/(1+offset))^gamma

if V = 1 then Y = 1
if V = 0 then Ymin = ( offset / (1 + offset)) ^ gamma

it's the one used by Colorfacts, X-Rite, Color.HCFR, Calman etc...

[tritical]

When I say 'transfer function' I just mean the function which maps linear RGB to gamma corrected RGB. The inverse of the transfer function maps gamma corrected RGB to linear RGB. In ddcc they are implemented as:

x = lv*C if C < thresh1
(1.0+av)*pow(C,pv)-av otherwise

x = C/lv if C < thresh2
pow((C+av)/(1.0+av),1.0/pv) otherwise


Here, x is the output value, C is a linear or gamma corrected RGB value in the range [0,1], and lv, av, pv, thresh1, and thresh2 are the variables. This is the same as the function you describe, if thresh1 and thresh2 are set to 0 (i.e. no linear segment around 0), offset = av, and pv = 1.0/gamma. All of the presets that I put into ddcc (with the exception of the one to match the pixel shader code, and the srgb one which I got from http://en.wikipedia.org/wiki/SRGB) I took from the mpeg2 specs ('transfer_characteristics', Table 6-8).

[pitch.fr]

humm OK, I will read that slowly again I'm not quite a coder :D

BTW, what happens to off-gamut colors ?

white is my DLP pj, black is SMPTE-C :

http://pix.nofrag.com/a/3/c/deff50ab2a4b5c063c43e15a16155.png

what happens to the green colors that my pj can't output ? they're clipped ? :eek:

so they don't appear(banding?), or it still shows yellow instead like it does w/o correction ?

and are you sure this can't be done through the CLUT ?!

ARGYLLCMS has pretty advanced CLUT capabilities, and it's GPL software.

considering you know the formula part, why would that be impossible to generate CLUT entries ?

then we can "capture" it with powerstrip, and we're free from any ressource hogging process.

of course I will test it in every possible way with my colorimeter :D

DESCRIPTOR "Argyll Device Calibration State"
ORIGINATOR "Argyll dispcal"
CREATED "Tue Jun 17 13:01:45 2008"
KEYWORD "DEVICE_CLASS"
DEVICE_CLASS "DISPLAY"
KEYWORD "DEVICE_TYPE"
KEYWORD "TARGET_WHITE_XYZ"
TARGET_WHITE_XYZ "47.889141 50.384253 54.837810"
KEYWORD "TARGET_GAMMA"
TARGET_GAMMA "2.200000"
KEYWORD "BLACK_POINT_CORRECTION"
BLACK_POINT_CORRECTION "1.000000"
KEYWORD "QUALITY"
QUALITY "high"

KEYWORD "RGB_I"
NUMBER_OF_FIELDS 4
BEGIN_DATA_FORMAT
RGB_I RGB_R RGB_G RGB_B
END_DATA_FORMAT

NUMBER_OF_SETS 256
BEGIN_DATA
0.0000 0.084613 0.0000 2.0434e-003
3.9216e-003 0.087834 0.0000 6.7763e-003
7.8431e-003 0.091075 1.0884e-003 0.011527
0.011765 0.094334 8.1334e-003 0.016295
0.015686 0.097613 0.015385 0.021081
0.019608 0.10091 0.022857 0.025885
0.023529 0.10423 0.030658 0.030707
0.027451 0.10757 0.038978 0.035546
0.031373 0.11092 0.047935 0.040408
0.035294 0.11430 0.057523 0.045302
0.039216 0.11771 0.067448 0.050240
0.043137 0.12117 0.077293 0.055230
0.047059 0.12466 0.086703 0.060272
0.050980 0.12820 0.095542 0.065369
0.054902 0.13178 0.10384 0.070522
0.058824 0.13540 0.11162 0.075722
0.062745 0.13906 0.11897 0.080956
0.066667 0.14275 0.12592 0.086205
0.070588 0.14647 0.13252 0.091446
0.074510 0.15019 0.13878 0.096661
0.078431 0.15392 0.14473 0.10184
0.082353 0.15765 0.15039 0.10697
0.086275 0.16136 0.15583 0.11205
0.090196 0.16507 0.16107 0.11710
0.094118 0.16877 0.16614 0.12210
0.098039 0.17246 0.17107 0.12706
0.10196 0.17616 0.17589 0.13197
0.10588 0.17986 0.18059 0.13685
0.10980 0.18357 0.18519 0.14168
0.11373 0.18728 0.18972 0.14646
0.11765 0.19100 0.19416 0.15121
0.12157 0.19471 0.19853 0.15589
0.12549 0.19843 0.20279 0.16054
0.12941 0.20214 0.20699 0.16514

and many more lines.......

[yesgrey]

BTW, what happens to off-gamut colors ?
what happens to the green colors that my pj can't output ? they're clipped ? :eek:
so they don't appear(banding?), or it still shows yellow instead like it does w/o correction ?

The final color gamut will be the intersection of the two triangles; a display cannot output colors out of it's own color gamut. What this correction does is remapping the "original" colors into the display gammut so they are the same. It does this changing the RGB values sent to your display in a way that the resulting colors would be the same as the "original".

As I told you before, in theory could exist banding, in practice, I've never noticed it. As you see, the colors clipped are the more saturated ones, which don't appear too often in a movie...

There are two kinds of possible banding, one avoidable and the other not:
1-colors out of display gammut: there is nothing to do, due to the clipping, some colors will be the same. Not many and not very frequent in a movie.
2-colors inside of display gammut: If you perform the correction and have a display with >= 10 bit per color, you could avoid this kind of banding, but in the pc we are currently "limited" by 8 bit per color. Maybe there is any hardware solution...

[yesgrey]

A CUDA implementation would be easy to do, but I was thinking that it would be pretty limiting (in terms of supported video cards) compared to a pixel shader.

Well, it would be a start... and much of the people who will use this probably have a Nvidia card.
The ATI "CUDA version" is too complicated?
I understand that currently it's not very appealing working with these new GPU/CPU APIs, because probably only one will survive or even none of them will survive (a Microsoft one maybe?...)

Also, we cannot forget that the currently existing alternative to this, using mplayerc with PS, works in all of them and gives pretty good results. I believe you're solution in theory is better, because you are using the correct gamma functions, but at the end, the result should be very similar...

I left the decision in your ends (as it should be).:)

[pitch.fr]

As I told you before, in theory could exist banding, in the practice, I've never noticed it. As you see, the colors clipped are the more saturated ones, which don't appear too often in a movie...

There are two kinds of possible banding, one avoidable and the other not
ok, gracias for the explanation ;)

well CLUT is 30 bit, so no banding :)

I've spoken to Graeme Gill(the author of www.argyllcms.com) and he said this could be achieved through the CLUT.

then I could use it on top of Haali's Renderer.............this would be awesome for me coz this is the smoothest renderer ever :)

here's what he told me :

do you think that would be doable through the CLUT ?

Yes. It's been done many times. See the example code in the openEXR package and their exrdisplay utility. Rather than doing simple matrix arithmetic, you set up a texture to hold the CLUT, and have the fragment shader do a texturelookup for each pixel.

the issue is that the smoothest video renderer on PC(Haali's Renderer) doesn't support Pixel Shaders......and I was hoping to get SMPTE-C corrected colors with it, but I guess it's just a wild dream at this point...

The code here http://www.avsforum.com/avs-vb/showthread.php?t=912720 is already using a shader, it's just a matter of converting it to use a text3D and loading the ICC device link into the texture, instead of the pow/mul etc. functions.

so are you saying that once the CLUT is built, it could work w/o further realtime pixel manipulations ?

Yes, it only needs initializing if you change the device link it represents.
The OpenEXR utility was written to preview motion picture effects in real time,that's why they use a shader program to do it.

if any of you two code gurus could do that, that'd save a helluvalot of CPU time :)

as I understand it, this should be possible to output an .ICC/.ICM file instead of a realtime correction through PS or AVS ? :p

maybe this PDF would help ?
http://www.inventoland.net/imaging/JEI/159.PDF

OTOH, I'm trying your AVS plugin in ffdshow....as nothing's as smooth as HR on my box with Reclock in 24/48Hz :(

I've copied this to C:\PJ_SMPTE.txt :

0.151
0.068
0.781
0.339
0.611
0.050
0.656
0.329
0.015
0.311
0.328
0.361
1.0
0.0
0.45
0.0
0.0

and I'm using :

MT("""ConvertToRGB32(matrix="rec709")""",4)
ddcc(chr_i=3,ofile="C:\PJ_SMPTE.txt",threads=4,opt=1)

all I get is a black screen, ddcc(chr_i=3,chr_o=0,threads=4,opt=1) works fine.

[IanB]

@Tritical,

One very minor issue, you min(max(X, 0.0f), 1.0f) in 3 places. The input gamma table, the result of the matrix multiply and again for the output gamma table. I would have expected only the output gamma table which generates the 8bit output pixels needed to be clamped, all the other cases could be free to have temporary head room excursions outside the nominal range.

i.e.

1.001*0.55 + 0.7*0.20 + 0.9*0.25 = 0.91555
1.000*0.55 + 0.7*0.20 + 0.9*0.25 = 0.91500

I guess it's a style thing, allow headroom on intermediate results or clamp rigorously at every stage.

[tritical]

@IanB
Well, only one of those three clamps is actually doing anything. The first clamp (on building the igamma table) shouldn't ever trigger since the inverse transfer function maps [0,1]->[0,1], and at that point it will only get [0,1] input. The second clamp, after the linear RGB -> XYZ -> linear RGB conversion, will trigger for out-of-gamut colors (i.e. a color in the input gamut is outside of the output gamut). I believe this is the right place to clamp because the final RGB values should be in the range [0,1], and the transfer function is defined as taking [0,1] input. The final clamp (on building the fgamma table) will again never trigger since it maps [0,1]->[0,1] and because of the previous clamp it will only be getting [0,1] input. I put those clamps in the gamma table creation mainly to remind myself that those values should be in the range [0,1], and since it is run only in the constructor the cost is negligible.

@pitch.fr
Having the filter output a 3D LUT is possible, just need to know the format to use. Also, the dark screen when using the ofile parameter was due to a bug I introduced in v1.1 (a missing multiply if the white points differed causing chromatic adaptation to be used). I put up a fixed version.

[pitch.fr]

well, Graeme Gill's told me that an .ICC file would do the trick! possibly v4 ?

the LUT is 30 bit on ATi cards, dunno with nvidia ?

If you could generate an .ICC file, this would be really awesome........as the Avisynth support in ffdshow is too slow(or maybe it's ConvertToRGB32's fault), and this PS script isn't working with HR..

at this point, nothing could be better than a way to generate .ICC profiles

I'm actually very happy to hear that it's possible, because getting 1:1 colors with Haali's Renderer and Reclock would be just TOO AWESOME :eek:

OK I'm gonna try again with the new version

there is another slight detail, though.
a friend of mine believes that gamut color conversion can only be done on the original 16-235 signal........applying the gamut conversion on 0-255 expanded content would require different coeffs ?!

when we do REC.xxx conversions to RGB32 in ffdshow, luma goes from 16-235 to 0-255 and chroma from 16-240 to 1-255.....so are you still able to map the 0-15 and 241-255 colors :confused:

or should we output in full range and let your ICC do the PC>TV conversion ? or a custom range ?

also, here's a very interesting thread about these gamut stories :
http://www.avsforum.com/avs-vb/showpost.php?p=14071122&postcount=1

and another one on doom9 :
http://forum.doom9.org/showthread.php?t=132745

and a PDF about ICC corrections in the movie industry :
http://www.color.org/ICC_Chiba_07-06-19_PM_DMP_Float.pdf

thanks for your help tritical !

[tritical]

I don't know anything about ICC profiles... How exactly would that solve the problem (assuming you had one, how would you use it)? Also, is there a document which describes the format of icc profiles, i.e. how to create one?

a friend of mine believes that gamut color conversion can only be done on the original 16-235 signal........applying the gamut conversion on 0-255 expanded content would require different coeffs ?!

when we do REC.xxx conversions to RGB32 in ffdshow, luma goes from 16-235 to 0-255 and chroma from 16-240 to 1-255.....so are you still able to map the 0-15 and 241-255 colors

All of the equations in this filter work on values in the [0,1] range. The conversion to and from [0,1] currently assumes the min/max values of the RGB channels are 0 and 255. I could allow the user to specify different min/max values if desired (for instance if the RGB channels had a range of [16,235]), but usually RGB is [0,255]. The range question is usually related to whether YUV is [16,235],[16-240] or [0-255]. Depending on which it is, you need different scalings of the YUV->RGB conversion coefficients.

[pitch.fr]

well an ICC profile is the most universal solution by far.

people could compute the numbers on windows, then use the ICC profile on any OS and even hardware device :)

and it would enable gamut correction in any software/player(except PowerDVD because it only works in OVERLAY and this doesn't support ICC's).......but all the other players could beneficiate from this......even games, and there's no CPU time required, just like Pixel Shaders :)

I believe you can find all kind of white papers through google considering it's an open file format I think :
http://www.google.com/search?hl=en&q=ICC+specs&btnG=Search&lr=

this one maybe ?
http://www.color.org/ICC1v42_2006-05.pdf

I can ask Graeme Gill as well.

you can apply it within windows(for automatic correction) or with various apps, but I would personally use dispwin.exe(from the ARGYLLCMS package) with a batch for each of my displays(CRT/pj)

[pitch.fr]

I'm still a bit confused between LUT and profiles.
LUT is the final result of the profile, so you can't quite output LUT raw data.

here's what he told me

maybe he could simply output "raw lut data", that can be read with this software : http://www.exactscan.com/lutmanager/

You are confusing calibration and profiles.
The "lut data" is the calibration curves.
A device link is the result of linking two device ICC profiles,and is a clut, a 3D dimensional interpolation table.
Gerhard's code http://www.mail-archive.com/lcms-user@lists.sourceforge.net/msg02326.html , which uses lcms to link two device profiles and load it into a textureis basically a complete example.
I can't help you much more than pointing youat that code. Graeme Gill.

I've asked him for simple specs to an .ICC file

ok, so using the settings from the previous page with DDCC 1.2, compared to these settings with the .XLS script(updated the script with your coeffs that have 5 digits after the coma) :
http://rapidshare.com/files/130373214/pj.xls.html

the correction is quite different actually.

this is the original test video :

http://rapidshare.com/files/130375581/rec709.mkv.html

http://thumbnails8.imagebam.com/950/b5e6859499794.gif (http://www.imagebam.com/image/b5e6859499794)

this is with the PS script, and SMPTE-C conversion :
http://thumbnails8.imagebam.com/950/3498ff9499795.gif (http://www.imagebam.com/image/3498ff9499795)

this is with ddcc 1.2, and SMPTE-C conversion :
http://thumbnails8.imagebam.com/950/3c2ca29499796.gif (http://www.imagebam.com/image/3c2ca29499796)

I'd say the gamma has been lowered way too much....clearly not 2.2 ?

or maybe we need to finetune the settings ? :D

EDIT : there's an ICC profile SDK btw :
http://www.littlecms.com/downloads.htm

interestingly enough, they have used that SDK to make a simple example, that is said to :

This command line program does compute colorspace conversion based on icc profiles. Additionally, it can show XYZ and Lab values of PCS, and up to 16 bits of precision (48, 64 bits per pixel). If you ever have been searching for a universal colorspace conversion utility, check this one!

http://www.littlecms.com/newutils.htm

but these things are going way over my head :scared:

[tritical]

The differences to the pixel shader are due to two things:

1.) ddcc and the ps code are using different input gamma functions. You need to set gam_i=5 in ddcc to have it use the same one as the ps code. So the script line would become:

ddcc(chr_i=3,gam_i=5,ofile="ofile.txt")

2.) When the white point coordinates differ between the source and destination, ddcc performs chromatic adaptation using the Bradford method. The pixel shader excel file does not. This will slightly change the resulting coefficients. (Other than this, the ps excel file and ddcc produce the same coefficients).

I don't think I'll mess with ICC profiles. However, the 3D lut method via a 3d texture on the gpu looks easy enough. I will try to implement it when I have the time.

[pitch.fr]

OK that sounds great, indeed they're identical now...tritical yo'ure my hero :D

original :

http://thumbnails8.imagebam.com/956/b5e6859555408.gif (http://www.imagebam.com/image/b5e6859555408)

PS :

http://thumbnails8.imagebam.com/956/3498ff9555409.gif (http://www.imagebam.com/image/3498ff9555409)

AVS :

http://thumbnails8.imagebam.com/956/9a4b919555410.gif (http://www.imagebam.com/image/9a4b919555410)

I will make colorimeter measures anytime soon :)

too bad the ConvertToRGB32 sucks so much CPU......it's hardly usable in real time :(

[pitch.fr]

OK I've played around with Profile Maker 5.0

http://pix.nofrag.com/a/e/5/7812bec244bd2cba7acec57c666bdtt.jpg (http://pix.nofrag.com/a/e/5/7812bec244bd2cba7acec57c666bd.html)

here's what their help file says :

Mechanisms of the ICC Profile
There are a variety of mathematical mechanisms for color conversion that are used by the Color Management Module. The data required for this operation is contained in the profile.

There are two methods of defining a conversion:

1)Mathematical Functions (TRC: Tone Reproduction Curves and Matrix Models)

2)Conversion Tables (LUT: Look Up Table)

The ICC profiles of the different device categories work with a variety of mathematical models.
Using Matrix Functions and TRCs
For the quite simple conversions between purely additive color systems, ICC profiles use Matrix Functions. These models are sufficiently accurate and produce relatively small color profiles.

The advantage of functions is the compactness of the parameter sets, but unfortunately they are poorly suited to accurately describe complex mapping characteristics such as exist between color models that have different dimensions (RGB > LAB versus CMYK > LAB).

Monitor profiles work mostly with TRCs and 3x3 Matrix Operations.

Using LUTs
For more complex color systems, for example in the case of color transformations involving a change in dimension, and for the description of non-linear color spaces, Tables (LUTs) must be used.

Tables offer unlimited accuracy and an unlimited number of dimensions, but are also unlimited in terms of size.

As a rule, the direction of conversion from Scanner RGB to LAB is described in a scanner profile using three-dimensional LUTs. Normally, a separate table would have to be created for each Rendering Intent, for a total of four tables. However, since the Rendering Intents for Relative and Absolute colorimetric interpretation can be calculated from the same table, only three LUTs are required. In output profiles, both conversion directions are described using LUTs. In this case, three four-dimensional LUTs are created for the direction CMYK to LAB and three thee-dimensional LUTs for the direction LAB to CMYK.

so clearly a LUT is more accurate than an ICC :)

and Powerstrip's coder had given me infos on how to input custom LUT's within his app :

LUTs are stored by PowerStrip in the registry, and read/written to SRAM in the GPU. Here's the text format for a Broadcast D65 LUT - it uses 8 bit values because it predates deeper depths, but PowerStrip has been able to read LUTs stored with 16 bit data since the Matrox Parhelia.

So you have here 768 bytes (or words) in the form R[0], R[1]...R[254],R[255],G[0]...G[255],B[0]...B[255].


"Broadcast 2.2 Gamma D65"=hex:00,00,00,01,01,02,02,02,03,03,04,04,04,05,05,06,\
07,08,09,0a,0b,0c,0d,0f,10,11,12,13,14,15,16,17,19,1a,1b,1c,1d,1f,20,21,22,\
23,24,26,27,28,29,2a,2b,2c,2d,2f,30,31,32,33,34,35,36,38,39,3a,3b,3c,3d,3e,\
3f,40,42,43,44,45,46,47,48,49,4a,4b,4c,4d,4f,50,51,52,53,54,55,56,57,58,59,\
5a,5b,5c,5e,5f,60,61,62,63,64,65,66,67,68,69,6a,6b,6c,6d,6e,6f,70,71,72,73,\
74,76,77,78,79,7a,7b,7c,7d,7e,7f,80,81,82,83,84,85,86,87,88,89,8a,8b,8c,8d,\
8e,8f,90,91,92,93,94,95,96,97,98,99,9a,9b,9c,9d,9e,9f,a0,a1,a2,a3,a4,a5,a6,\
a7,a8,a9,aa,ab,ad,ae,af,b0,b1,b2,b3,b4,b5,b6,b7,b8,b9,ba,bb,bc,bd,bd,be,bf,\
c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,ca,cb,cc,cd,ce,cf,d0,d1,d2,d3,d4,d5,d6,d7,d8,\
d9,da,db,dc,dd,de,df,e0,e1,e2,e3,e4,e5,e6,e7,e8,e9,ea,eb,ec,ed,ee,ef,f0,f1,\
f2,f3,f4,f5,f6,f7,f8,f9,fa,fb,fb,fc,fd,fe,ff,00,00,00,01,01,02,02,02,03,03,\
04,04,04,05,05,06,07,08,09,0a,0b,0c,0d,0f,10,11,12,13,14,15,16,18,19,1a,1b,\
1c,1d,1f,20,21,22,23,24,26,27,28,29,2a,2b,2c,2e,2f,30,31,32,33,34,35,37,38,\
39,3a,3b,3c,3d,3e,3f,40,42,43,44,45,46,47,48,49,4a,4b,4c,4e,4f,50,51,52,53,\
54,55,56,57,58,59,5a,5b,5c,5e,5f,60,61,62,63,64,65,66,67,68,69,6a,6b,6c,6d,\
6e,6f,70,71,72,74,75,76,77,78,79,7a,7b,7c,7d,7e,7f,80,81,82,83,84,85,86,87,\
88,89,8a,8b,8c,8d,8e,8f,90,91,92,93,94,95,96,97,98,99,9a,9b,9c,9d,9e,9f,a0,\
a1,a3,a4,a5,a6,a7,a8,a9,aa,ab,ac,ad,ae,af,b0,b1,b2,b3,b4,b5,b6,b7,b8,b9,ba,\
bb,bc,bd,be,bf,c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,ca,ca,cb,cc,cd,ce,cf,d0,d1,d2,\
d3,d4,d5,d6,d7,d8,d9,da,db,dc,dd,de,df,e0,e1,e2,e3,e4,e5,e6,e7,e8,e9,ea,eb,\
ec,ed,ee,ef,f0,f1,f2,f3,f4,f5,f6,f7,f8,f9,fa,fb,fc,fd,fe,ff,ff,00,00,00,01,\
01,02,02,02,03,03,04,04,04,05,05,06,07,08,09,0a,0b,0c,0d,0f,10,11,12,13,14,\
15,16,17,19,1a,1b,1c,1d,1e,20,21,22,23,24,25,27,28,29,2a,2b,2c,2d,2f,30,31,\
32,33,34,35,36,38,39,3a,3b,3c,3d,3e,3f,40,41,43,44,45,46,47,48,49,4a,4b,4c,\
4d,4e,50,51,52,53,54,55,56,57,58,59,5a,5b,5c,5d,5e,60,61,62,63,64,65,66,67,\
68,69,6a,6b,6c,6d,6e,6f,70,71,72,73,74,75,77,78,79,7a,7b,7c,7d,7e,7f,80,81,\
82,83,84,85,86,87,88,89,8a,8b,8c,8d,8e,8f,90,91,92,93,94,95,96,97,98,99,9a,\
9b,9c,9d,9e,9f,a0,a1,a2,a3,a4,a5,a6,a7,a8,a9,aa,ab,ac,ad,ae,af,b0,b1,b2,b3,\
b4,b5,b6,b7,b8,b9,ba,bb,bc,bd,be,bf,c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,ca,cb,cc,\
cd,ce,cf,d0,d1,d2,d3,d4,d5,d6,d7,d8,d9,da,db,dc,dd,de,df,e0,e1,e2,e3,e4,e5,\
e6,e7,e8,e9,ea,eb,ec,ed,ee,ef,f0,f1,f2,f3,f4,f5,f6,f6,f7,f8,f9,fa,fb,fc,fd,\
fe,ff

maybe using your voodoo magic, you could offer a way to output powerstrip compatible LUT's :eek:

he has also told me that you can output 16bit LUT, which will improve the accuracy to 10 bit on ATi cards, dunno about nvidia

If you use 16-bit values, PowerStrip automatically adjusts down to the maximum bit depth of the DAC.

If you use 8-bit values, PowerStrip will make 8 bit correction.

the advantage of doing the conversion through the LUT is that it doesn't use any CPU time and it's compatible with any picture viewer/video renderer(except OVERLAY) w/o any requirement(like PS script compatibility, that only MPC and KMPlayer support at this point)

and the LUT can be ported to other OS such as Linux :)

thanks!

[pitch.fr]

OK I've read the AVS thread again.

it seems that people were confusing ICC and LUT(just like me, but the other way around)

an ICC only works in compatible applications, Adobe Gamma can force it though.....but only in window mode.

so the best solution is to output 16 bit LUT to powerstrip, which will render it in 10 bit on the Radeon.

the LUT is compatible with any application, except the OVERLAY video renderer.

here's the powerstrip coder's notes about importing 16bit pstrip-compatible LUT's to the registry(for instant use in pstrip in 10 bit) :

Here is the same Broadcast D65 LUT, but with 16 bit values instead of 8 bit.


"Broadcast 2.2 Gamma D65 (16 bit)"=hex:00,00,00,00,00,00,00,01,00,01,00,02,00,\
02,00,02,00,03,00,03,00,04,00,04,00,04,00,05,00,05,00,06,00,07,00,08,00,09,\
00,0a,00,0b,00,0c,00,0d,00,0f,00,10,00,11,00,12,00,13,00,14,00,15,00,16,00,\
17,00,19,00,1a,00,1b,00,1c,00,1d,00,1f,00,20,00,21,00,22,00,23,00,24,00,26,\
00,27,00,28,00,29,00,2a,00,2b,00,2c,00,2d,00,2f,00,30,00,31,00,32,00,33,00,\
34,00,35,00,36,00,38,00,39,00,3a,00,3b,00,3c,00,3d,00,3e,00,3f,00,40,00,42,\
00,43,00,44,00,45,00,46,00,47,00,48,00,49,00,4a,00,4b,00,4c,00,4d,00,4f,00,\
50,00,51,00,52,00,53,00,54,00,55,00,56,00,57,00,58,00,59,00,5a,00,5b,00,5c,\
00,5e,00,5f,00,60,00,61,00,62,00,63,00,64,00,65,00,66,00,67,00,68,00,69,00,\
6a,00,6b,00,6c,00,6d,00,6e,00,6f,00,70,00,71,00,72,00,73,00,74,00,76,00,77,\
00,78,00,79,00,7a,00,7b,00,7c,00,7d,00,7e,00,7f,00,80,00,81,00,82,00,83,00,\
84,00,85,00,86,00,87,00,88,00,89,00,8a,00,8b,00,8c,00,8d,00,8e,00,8f,00,90,\
00,91,00,92,00,93,00,94,00,95,00,96,00,97,00,98,00,99,00,9a,00,9b,00,9c,00,\
9d,00,9e,00,9f,00,a0,00,a1,00,a2,00,a3,00,a4,00,a5,00,a6,00,a7,00,a8,00,a9,\
00,aa,00,ab,00,ad,00,ae,00,af,00,b0,00,b1,00,b2,00,b3,00,b4,00,b5,00,b6,00,\
b7,00,b8,00,b9,00,ba,00,bb,00,bc,00,bd,00,bd,00,be,00,bf,00,c0,00,c1,00,c2,\
00,c3,00,c4,00,c5,00,c6,00,c7,00,c8,00,c9,00,ca,00,cb,00,cc,00,cd,00,ce,00,\
cf,00,d0,00,d1,00,d2,00,d3,00,d4,00,d5,00,d6,00,d7,00,d8,00,d9,00,da,00,db,\
00,dc,00,dd,00,de,00,df,00,e0,00,e1,00,e2,00,e3,00,e4,00,e5,00,e6,00,e7,00,\
e8,00,e9,00,ea,00,eb,00,ec,00,ed,00,ee,00,ef,00,f0,00,f1,00,f2,00,f3,00,f4,\
00,f5,00,f6,00,f7,00,f8,00,f9,00,fa,00,fb,00,fb,00,fc,00,fd,00,fe,00,ff,00,\
00,00,00,00,00,00,01,00,01,00,02,00,02,00,02,00,03,00,03,00,04,00,04,00,04,\
00,05,00,05,00,06,00,07,00,08,00,09,00,0a,00,0b,00,0c,00,0d,00,0f,00,10,00,\
11,00,12,00,13,00,14,00,15,00,16,00,18,00,19,00,1a,00,1b,00,1c,00,1d,00,1f,\
00,20,00,21,00,22,00,23,00,24,00,26,00,27,00,28,00,29,00,2a,00,2b,00,2c,00,\
2e,00,2f,00,30,00,31,00,32,00,33,00,34,00,35,00,37,00,38,00,39,00,3a,00,3b,\
00,3c,00,3d,00,3e,00,3f,00,40,00,42,00,43,00,44,00,45,00,46,00,47,00,48,00,\
49,00,4a,00,4b,00,4c,00,4e,00,4f,00,50,00,51,00,52,00,53,00,54,00,55,00,56,\
00,57,00,58,00,59,00,5a,00,5b,00,5c,00,5e,00,5f,00,60,00,61,00,62,00,63,00,\
64,00,65,00,66,00,67,00,68,00,69,00,6a,00,6b,00,6c,00,6d,00,6e,00,6f,00,70,\
00,71,00,72,00,74,00,75,00,76,00,77,00,78,00,79,00,7a,00,7b,00,7c,00,7d,00,\
7e,00,7f,00,80,00,81,00,82,00,83,00,84,00,85,00,86,00,87,00,88,00,89,00,8a,\
00,8b,00,8c,00,8d,00,8e,00,8f,00,90,00,91,00,92,00,93,00,94,00,95,00,96,00,\
97,00,98,00,99,00,9a,00,9b,00,9c,00,9d,00,9e,00,9f,00,a0,00,a1,00,a3,00,a4,\
00,a5,00,a6,00,a7,00,a8,00,a9,00,aa,00,ab,00,ac,00,ad,00,ae,00,af,00,b0,00,\
b1,00,b2,00,b3,00,b4,00,b5,00,b6,00,b7,00,b8,00,b9,00,ba,00,bb,00,bc,00,bd,\
00,be,00,bf,00,c0,00,c1,00,c2,00,c3,00,c4,00,c5,00,c6,00,c7,00,c8,00,c9,00,\
ca,00,ca,00,cb,00,cc,00,cd,00,ce,00,cf,00,d0,00,d1,00,d2,00,d3,00,d4,00,d5,\
00,d6,00,d7,00,d8,00,d9,00,da,00,db,00,dc,00,dd,00,de,00,df,00,e0,00,e1,00,\
e2,00,e3,00,e4,00,e5,00,e6,00,e7,00,e8,00,e9,00,ea,00,eb,00,ec,00,ed,00,ee,\
00,ef,00,f0,00,f1,00,f2,00,f3,00,f4,00,f5,00,f6,00,f7,00,f8,00,f9,00,fa,00,\
fb,00,fc,00,fd,00,fe,00,ff,00,ff,00,00,00,00,00,00,00,01,00,01,00,02,00,02,\
00,02,00,03,00,03,00,04,00,04,00,04,00,05,00,05,00,06,00,07,00,08,00,09,00,\
0a,00,0b,00,0c,00,0d,00,0f,00,10,00,11,00,12,00,13,00,14,00,15,00,16,00,17,\
00,19,00,1a,00,1b,00,1c,00,1d,00,1e,00,20,00,21,00,22,00,23,00,24,00,25,00,\
27,00,28,00,29,00,2a,00,2b,00,2c,00,2d,00,2f,00,30,00,31,00,32,00,33,00,34,\
00,35,00,36,00,38,00,39,00,3a,00,3b,00,3c,00,3d,00,3e,00,3f,00,40,00,41,00,\
43,00,44,00,45,00,46,00,47,00,48,00,49,00,4a,00,4b,00,4c,00,4d,00,4e,00,50,\
00,51,00,52,00,53,00,54,00,55,00,56,00,57,00,58,00,59,00,5a,00,5b,00,5c,00,\
5d,00,5e,00,60,00,61,00,62,00,63,00,64,00,65,00,66,00,67,00,68,00,69,00,6a,\
00,6b,00,6c,00,6d,00,6e,00,6f,00,70,00,71,00,72,00,73,00,74,00,75,00,77,00,\
78,00,79,00,7a,00,7b,00,7c,00,7d,00,7e,00,7f,00,80,00,81,00,82,00,83,00,84,\
00,85,00,86,00,87,00,88,00,89,00,8a,00,8b,00,8c,00,8d,00,8e,00,8f,00,90,00,\
91,00,92,00,93,00,94,00,95,00,96,00,97,00,98,00,99,00,9a,00,9b,00,9c,00,9d,\
00,9e,00,9f,00,a0,00,a1,00,a2,00,a3,00,a4,00,a5,00,a6,00,a7,00,a8,00,a9,00,\
aa,00,ab,00,ac,00,ad,00,ae,00,af,00,b0,00,b1,00,b2,00,b3,00,b4,00,b5,00,b6,\
00,b7,00,b8,00,b9,00,ba,00,bb,00,bc,00,bd,00,be,00,bf,00,c0,00,c1,00,c2,00,\
c3,00,c4,00,c5,00,c6,00,c7,00,c8,00,c9,00,ca,00,cb,00,cc,00,cd,00,ce,00,cf,\
00,d0,00,d1,00,d2,00,d3,00,d4,00,d5,00,d6,00,d7,00,d8,00,d9,00,da,00,db,00,\
dc,00,dd,00,de,00,df,00,e0,00,e1,00,e2,00,e3,00,e4,00,e5,00,e6,00,e7,00,e8,\
00,e9,00,ea,00,eb,00,ec,00,ed,00,ee,00,ef,00,f0,00,f1,00,f2,00,f3,00,f4,00,\
f5,00,f6,00,f6,00,f7,00,f8,00,f9,00,fa,00,fb,00,fc,00,fd,00,fe,00,ff


As you can see from the very last LUT entry above (0xFF00 for B[256]) when represented as text the msb are in the 2nd byte of the couplet.

[pitch.fr]

OK I've made some colorimetry measurements with my Eye One Display 2 on my HC3100 as promised.

the black triangle is SMPTE RP 145 in all these graphs, and the ghost gamut is the original one.

I also give the Delta-E and the Distance from the reference in the CIExy space

the projector has been D65 calibrated with a Delta-E <3 before doing the tests(with the official test patterns DVD and Color.HCFR in MPC) :

http://pix.nofrag.com/b/2/6/c48e09b6e79d3305ce11b26fa27edtt.jpg (http://pix.nofrag.com/b/2/6/c48e09b6e79d3305ce11b26fa27ed.html)

that's the original gamut :

http://pix.nofrag.com/2/0/8/2b4fde6c873d93324cb13e538db76tt.jpg (http://pix.nofrag.com/2/0/8/2b4fde6c873d93324cb13e538db76.html)

http://pix.nofrag.com/d/a/7/993d1813bee28cdd09acd5f647378.png


now with the Pixel Shaders script :

http://pix.nofrag.com/4/1/0/6146664a9e7e23f8f006fc5e3875dtt.jpg (http://pix.nofrag.com/4/1/0/6146664a9e7e23f8f006fc5e3875d.html)

http://pix.nofrag.com/6/8/f/de5e32a36e2732028bbe1ab14e1f3.png


now with ddcc(chr_i=3,gam_i=5,ofile="C:\PJ.txt",threads=4,opt=1) :

http://pix.nofrag.com/2/c/c/aea54933ce35849d7bf9774522bf3tt.jpg (http://pix.nofrag.com/2/c/c/aea54933ce35849d7bf9774522bf3.html)

http://pix.nofrag.com/1/2/f/1b9c92223b2fef175a7d7fe61cb10.png


now with ddcc(chr_i=3,gam_i=2,ofile="C:\PJ.txt",threads=4,opt=1) :

http://pix.nofrag.com/a/f/a/7b96e6cba9e016e1c02893d743e26tt.jpg (http://pix.nofrag.com/a/f/a/7b96e6cba9e016e1c02893d743e26.html)

http://pix.nofrag.com/f/3/4/941c1cae7095aa14a0fed85b9d41d.png

PJ.txt contained :

0.151
0.068
0.782
0.338
0.611
0.052
0.653
0.330
0.002
0.312
0.328
0.360
1.0
0.0
0.45
0.0
0.0


it's really impressive how close the PS script managed to get to the refs in the CIExy space....I'm speechless :eek:

maybe some brush-up of the DDCC maths and/or the text file could improve the results ?

together with a way to output 16 bit Powerstrip-compatible LUT's please ? :D

TIA,

[pitch.fr]

OK forget it, this can't be done through the LUT :

> I'm afraid there is a misunderstanding. The VCGT LUTs which can be
> loaded into the graphics card with ARGYLL are just one-dimensional
> LUTs. Such LUTs do not allow arbitrary color transformation, but only
> VERY LIMITED ones (like e.g. adjusting the gamma of the RGB channnels).
> For the kind of color transformation you desire, you would need a
> three-dimensional LUT. I'm not aware of graphics cards which support
> that (except in conjunction with a pixel shader program), but I also
> don't want to rule out that such graphics cards might exist.

tritical, any idea what I could do to get the same accurate results that the PS script is giving ?

thanks,

[yesgrey]

Now I understand what is a LUT. You could change it through the color calibration you could perform in the graphic card drivers, this was debated in the AVS Forum. The problem of it is that each component only depends on itself, and for this kind of correction, each component should depends on all three components.
Example:
LUT
Rd = f(Rs)
Gd = f(Gs)
Bd = f(Bs)

What we need:
Rd = f(Rs,Gs,Bs)
Gd = f(Rs,Gs,Bs)
Bd = f(Rs,Gs,Bs)

It's the 3D LUT table you referred...

There is also a very simple and fast solution for your problem... Haali uses Pixel Shaders in his renderer for the YUV->RGB conversion, you could ask him to add this PS code to his renderer, in which we just need to specify our conversion matrix.
In fact, I have already tried that some time ago without any luck. Give it a try, maybe if more people start to request this to him he feel the need to do it.;)

[tritical]

Your measurements aren't telling the whole story here for a number of reasons. First, you are ignoring white point differences between source/destination, and chromatic adaptation. As I said before, the only difference between the PS script and ddcc (assuming ddcc is setup to use the same gamma function assumed by the ps script, etc...) is that ddcc performs chromatic adaptation when the white points between the source and destination differ. Chromatic adaptation, with regards to the HVS, refers to the ability of the HVS to make an object appear the same (color wise) under a wide range of illuminants. What this means is that to a person different points in the CIE XYZ colorspace will appear to be the same color when viewed under different illuminants. When it is used in regards to colorspace conversions (specifically those based on CIE XYZ where the white point differs) it refers to trying to alter the XYZ tristimulus values such that to a person the new color under the new illuminant would look the same as the original color under the original illuminant. The are three main methods for doing chromatic adaptation: XYZ scaling, Von Kries, and Bradford. ddcc uses Bradford. To see how this effects the coefficients in the BGR->CIE->BGR transform, I had ddcc print them out. The first set shows the coefficients calculated when using the ddcc() line you specified. Each line after that modifies your white point to be progressively closer to the D65 illuminant, such that in the last case no chromatic adaptation is needed (in this case the PS script's coefficients match ddcc's to 7 or more decimal places).
0.312713,0.329016,0.358271 -> 0.312,0.328,0.360

1.109563,-0.129243,0.016116
0.028285,0.972018,-0.001302
0.008913,0.054374,0.938853


0.312713,0.329016,0.358271 -> 0.313,0.329,0.358

1.105802,-0.123381,0.019919
0.028030,0.971643,-0.000281
0.008032,0.052882,0.938751


0.312713,0.329016,0.358271 -> 0.3127,0.3290,0.3583

1.108651,-0.127532,0.018811
0.028307,0.971852,-0.000172
0.008176,0.053085,0.938775


0.312713,0.329016,0.358271 -> 0.312713,0.329016,0.358271

1.108620,-0.127480,0.018860
0.028306,0.971848,-0.000154
0.008164,0.053063,0.938774

While the coefficients aren't altered all that much (since the white point you specified was already pretty close to D65), it is definitely enough to account for the minor differences you measured between the PS script and ddcc.

Another thing to remember is that the assumed gamma function (in this case x^2.2222) is not an exact match to any real world device (let alone an exact match to both the source/destination). This will introduce error into the calculations.

Basically, while measurements are nice, the only way to truely know if the colors are accurate would be to see the original source on the original display device, and see if to you it looks the same as what you are seeing on your display device. Unfortunately this is unlikely to ever happen.

Anyways, to get back to the point, there isn't anything wrong with the math in ddcc.

[pitch.fr]

@yesgrey3 : this is not going to happen. Haali won't support PS scripts in his renderer, he clearly said it several times in his thread.

you can only achieve 3D LUT's in ICC v4, but this only works in windowed mode with HR, so it's useless to us.

@tritical : wow, the mad scientist is at work again :D

a friend of mine was explaining me that the CIE chart doesn't mean much because CIExy is a 3D space.....from the view we see the CIE from there's an unlimited number of heights.

the PS scripts prefers to get closer to the source, even if the Delta-E gets worse.

your explanation is going a bit over my head, I will need to read it several times again :D

so there's no way to measure what DDCC does ?

it doesn't clip colors like the PS script does, it's smarter than that then ? :eek:

for SMPTE-C/EBU conversion on an HDTV'ish display, the DDCC settings and the text file settings seem "optimal" to you ? is it better to go gam_i=2 or 5 ?

Thanks,

[yesgrey]

this is not going to happen. Haali won't support PS scripts in his renderer, he clearly said it several times in his thread

But he doesn't need to support PS scripts, he could only add this correction to his renderer, which should be a very simple task to do...

The other solution seems to be if tritical write ddcc with some kind of gpu support. PS would be better because will work with all gpu. CUDA would be easier to him, but will only work with NVidia. I vote for the later. (I have a nvidia card :D)

[yesgrey]

tritical,
Where did you get your D65 white point coordinates? (0.312713,0.329016,0.358271)
In Rec. ITU-R BT.709-5 the coordinates are: (0.3127,0.3290,0.3583)

In the begining I also used yours, but aparently they are wrong...

[tritical]

@yesgrey3
If you go through all the calculations based on how D65 is defined, 0.312713,0.329016,0.358271 are closer to the true values. BT.709 just rounds to 4 decimal places. I guess the BT.709 chromaticity coordinates definition in ddcc should be changed to the truncated values, but I don't think the difference is enough to matter (i.e. release a new version just for that). Especially if you are simply going to use pow(x,0.45) as the transfer function, when you consider that BT.709 actually defines the transfer function as:
x = 4.5*C if C < 0.018
(1.0+0.099)*pow(C,0.45)-0.099 otherwise

Again, I think most of these differences are too small to worry about.

@pitch.fr
Chromatic adaptation tries to adjust for changes in illuminant (between the source and destination) such that the image after conversion will appear to a person to look as much like the original as possible. Basically, CATs (chromatic adaptation transforms) are adjusting for a phenomena of the HVS, and like psychovisual methods in codecs, CATs are primarily based on human viewer tests. Using a CAT means that the XYZ values of each pixel will be different than if you just did a straight conversion while ignoring the CA phenomena. Most major photo/image editors like photoshop use CATs during conversions, so I assume it is desirable (photoshop uses the Bradford method).

On the gam_i=2 or 5 question, I'm not sure. SMPTE-C only defines chromaticity coordinates, not a transfer function.

[pitch.fr]

But he doesn't need to support PS scripts, he could only add this correction to his renderer, which should be a very simple task to do...

The other solution seems to be if tritical write ddcc with some kind of gpu support. PS would be better because will work with all gpu. CUDA would be easier to him, but will only work with NVidia. I vote for the later. (I have a nvidia card :D)

oh yeah GPU support would be awesome, but no proprietary stuff please.
nvidia cards don't work with Powerstrip, so no spot-on 48Hz for you.....and their PQ contrast and sharpness-wise it's nothing like the ATi's IMHO.

and what happens to the off-gamut green with the PS script in my case ? it clips to the most saturated green I can get ? strangely enough, I haven't seen any banding

On the gam_i=2 or 5 question, I'm not sure. SMPTE-C only defines chromaticity coordinates, not a transfer function.

well SMPTE-C is supposed to be in watched in the SMPTE RP 145 gamut I think, so how does that look ? and what about EBU Tech. 3213 for EBU ?
gam_i=5 seems to offer more contrast, because the low IRE gamma of 2 seem too bright.

what about the last 5 figures of the text file, do they look "optimal" to you ?

and does the DDCC algorithm offer improvements over the PS script ? or do they basically go different ways but achieve the same end result ? like off-gamut colors for instance ?

I'm getting a new CPU on thursday, hopefully a 4GHz C2D will help with real time use in ffdshow and HD content :D

[yesgrey]

I guess the BT.709 chromaticity coordinates definition in ddcc should be changed to the truncated values, but I don't think the difference is enough to matter (i.e. release a new version just for that).
I agree. I did not referred the chromaticity coordinates difference to justify what is happening, I know the difference is too small, I was just noting it.

nvidia cards don't work with Powerstrip, so no spot-on 48Hz for you...

and what happens to the off-gamut green with the PS script in my case ? it clips to the most saturated green I can get ? strangely enough, I haven't seen any banding

I have spot-on 47.952Hz with my geforce 8600gt. See my thread in powerstrip forum on how to achieve it. In fact, I also have spot on 50.000Hz and 59.940Hz. I wrote a little program to calculate the resolutions to achieve this. Maybe when I get the time I will create a thread about it. I have to test the new powerstrip functionallity to see if it overrides my little program...

Yes, your green and all the other off-gamut colors are clipped to the nearest most saturated colors you could get.
Remember that a movie is not full of highly saturated colors, and our eyes color perception is also non-linear, so don't worry about that and simply enjoy your movies!:)
For me, would be more of a concern the possibillity of banding inside the color gammut, when converting from one to another,
but I also never noticed it.
One of this days I will create a few test images to see if we could really notice the banding...;)

[pitch.fr]

I can't find any posts from yesgrey3 on the pstrip forum, do you have the link please ?

well quite frankly I've tried 2 8600GT(MSI & PNY), they looked really bad compared to the Sapphire 2400Pro/2600XT on both my CRT and my HC3100 in DVI.

like some ugly EE even on the windows desktop, and very bad contrast.

many friends have also switched after I told them, IMO the ATI are far ahead as far as PQ is concerned.

a friend of mine tried the new "auto search" feature of pstrip on a 9600GT, it didn't work at all!

I personally prefer 48.000Hz to get the real 24fps cinema speed, I find 47.952 too slow :D

[yesgrey]

Search for yesgrey33. I lost the password of the yesgrey3...

I prefer 47.952. If you set an exact refresh in your graphic card you don't need to resample the audio with reclock. I currently only use it to output bit perfect with my RME Fireface. For PAL movies, the pitch correction could be done lowering the audio card clock, without any resampling...

[pitch.fr]

well I don't see any tool ?!

Use powerstrip timings with GeForce 8 custom resolutions
yesgrey33 6 5125 Fri Mar 21, 2008 3:58 am

Custom resolution feature request
yesgrey33 10 2570 Fri Sep 28, 2007 9:33 pm

well I don't mind resampling, I'm outputting analog anyway.

and pstrip has that new feature that does automatic searches until the graphic card says "48.0000000000000000000 Hz"

so together with Reclock set to "24fps", it's a winner....very low jitter in HR 8)

[yesgrey]

I haven't posted the tool yet, and I believe the new pstrip function works very similar to mine, so maybe I will never post it.

When you resample the sound it's quality will be worse than without resampling... even when you use the Excelent quanlity, and this is very slow... and if you don't resample, you will have more cpu power for processing the color correction.

[pitch.fr]

the auto-search feature doesn't work at all on the 9600GT.
a friend of mine was searching for 48.000000 but all he got was 59Hz :D

maybe he could try your app ?

yes you have a point, if Reclock doesn't suck 20% of my CPU time to do "Excellent" resampling, this can be put to good use with DDCC :)

I'll see how it goes with my new E7200 tomorrow :D

[pitch.fr]

to get back on topic, would that be possible to fix the saturations of the display upfront in DDCC ?

that's a chart of my HC3100 saturations :

http://pix.nofrag.com/0/2/8/a990a92dce9da10f80a7f76c10e18.png

ideally, they should all be on the 0% line and at worse ±5%

you can see that at 75% the green primary is +12% and yellow +10%, which has a very bad effect on people faces.

and cyan is falling down, which increases the red saturation.

very few displays allow to set each color saturation(the Sanyo pj are very good at that).

could you set an offset or something in DDCC for the primaries/secondaries saturations ? because it's a major point not to be missed to get accurate colors on the whole IRE scale.

and a friend of mine is asking again if the luminance is taken in account ? considering we've done a TV>PC levels conversion.



PS : here's another thread that says that HD is mastered with REC.709 matrix and SMPTE-C primaries :
http://www.avsforum.com/avs-vb/showthread.php?p=14072175#post14072175

actually there's a very good reason why all the US telecine studios use this exact SONY 20" 4:3 CRT :
http://catalogs.infocommiq.com/avcat/CTL901/index.cfm?mlc_id=203&mrc_id=901&prodid=398485

that because it's built like a tank(2.5 gamma like all the broadcast equipment), and it already has less dynamic than a telecine, but LCD and plasma can't match its native contrast and also have other major drawbacks...

I just don't understand why they don't convert gamut from SMPTE to HDTV when they do BT.601>BT.709 transcoding :o

[pitch.fr]

I think I've killed tritical with all my questions :eek: :D

anyhow, I just got my new CPU, and it lets me run DDCC in real time with all my audio enhancements and Haali's Renderer.....this is awesome :D

http://pix.nofrag.com/0/9/9/f76a843109dbd01e7bec409131558tt.jpg (http://pix.nofrag.com/0/9/9/f76a843109dbd01e7bec409131558.html)

what do you guys think of this TV>PC levels AVS script ?
http://forum.doom9.org/showthread.php?t=137479

it basically does a TV>PC levels conversion doing chroma smoothing, to avoid banding, which is very pleasing to the eye....

but OTOH it's messing with the gamma curve, which increases the contrast :)

here's a comparison(both get through my pj SMPTE-C PS script), top is with Ulevels(), bottom is with ffdshow at the RGB32 conversion :

http://thumbnails8.imagebam.com/991/b61b119905830.gif (http://www.imagebam.com/image/b61b119905830)http://thumbnails8.imagebam.com/991/9b2ef49905832.gif (http://www.imagebam.com/image/9b2ef49905832)http://thumbnails8.imagebam.com/991/3b5c889905834.gif (http://www.imagebam.com/image/3b5c889905834)

http://thumbnails8.imagebam.com/991/f1ffe19905831.gif (http://www.imagebam.com/image/f1ffe19905831)http://thumbnails8.imagebam.com/991/0b3b449905833.gif (http://www.imagebam.com/image/0b3b449905833)http://thumbnails8.imagebam.com/991/fa99c29905835.gif (http://www.imagebam.com/image/fa99c29905835)

there's an ongoing war on AVS around SMPTE-C reds being way too orangey, and I have to agree.

basically the CARS hero will never look as red in the movie as he is on the cover....so using this thing kinda fixes this problem :cool:

here's what I'm running in ffdshow at this point :
MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",4)
MT("""ULevels(preset="tv2pc")""",4)
MT("""ConvertToRGB32(matrix="PC.709")""",4)
ddcc(chr_i=3,gam_i=5,ofile="C:\PJ.txt",threads=4,opt=1)

[pitch.fr]

just wanted to say that DDCC works perfectly fine in real time in ffdshow with any 720p/2.35 1080p content on a G0 Q6600 o/c to 3.4Ghz :)

it's too slow for 1080p 1.78/1.85 movies, though(even when downscaled to 720p).......ffdshow is wasting too many CPU cycles on 15/20mbit h264

but hopefully the h264 Remoulade decoder will solve this issue, because it's got much less latency than ffdshow and is far more MT optimized :eek:

[tritical]

While working on a project requiring rgb->Lab/Luv/Lch conversions, for which I stole code from ddcc, I discovered a bug in ddcc's chromatic adaptation code (it's used when the white points of the source and destination differ). Specifically, it should have been multiplying by the transpose a matrix when it was just using the matrix straight away. Fortunately, the errors caused by this were small. Anyways, I put up a fixed v1.3 at the usual location (http://bengal.missouri.edu/~kes25c/ddcc.zip). Also, I still don't have any plans to port this to the GPU.

[yesgrey]

tritical,
The correction matrix is always the same for each display. So, why not make a very simple thing like this:
Pick this filter and divide it in two parts:
1-One small software tool that will create files with 3D LUTs for all our display modes we need.
2-The Avisynth filter only has to load that file and output the correct RGB values just by looking at the 3D LUTs. This will not use lots of memory and it will speed up all this process. It's not very efficient calculating each RGB value one by one, since for each RGB value the corrected value will always be the same. This way, if we have the 3D LUTs in a file format, we could simply ask the ffdshow developers to add this to ffdshow, which would be very simple, just load the 3D LUTs and applying it after the YUV->RGB conversion, without any increase in cpu load, just some memory usage.
From my calculations, it would be: 3 3D LUTs: 3x256^3 = 48MB
For the current memory available in a PC, it's an insignificant amount...
What do you think?
The porting to the GPU is completelly useless, I agree with you. This is a one time calculation, so it could also be all written in C, then it's simply using the 3D LUTs...

[tritical]

yesgrey3, when I read your post my first thought was that accessing a 48MB LUT would be really slow on a typical desktop comp. Then I got curious about how slow it would be... Turns out it wasn't that slow at all (faster than the sse3 code path of ddcc on the dual core comp I'm on right now).

So I added a parameter to ddcc to output a 3D LUT ('lutfile'). The table is stored in binary format. The offset into the table in bytes is calculated as ((g<<16)+(b<<8)+r)*3, and at that location the new values are stored in b,g,r order (one byte each). I chose that arrangement because it made the assembly implementation fastest/easiest. I also added a new function to ddcc.dll called 'rgb3d'. It takes a LUT of the format I just described as input, and performs the 3D LUT operation. 'rgb3d' requires rgb24 input. The new dll is at the same place as before.

[yesgrey]

tritical,
Thank you very much! It was a good Christmas gift!:)
I will take a look and test it and will let you know...

:thanks:

[leeperry]

goddamn, you know you're just a bunch of mad scientists right :D

ConvertToRGB32(matrix="rec709")
= 302 fps

ConvertToRGB32(matrix="rec709").AviShader("C:\effect.fx", "ColorCorrection")
= 86 fps
(using this script : http://www.avsforum.com/avs-vb/showthread.php?t=912720 )

ConvertToRGB32(matrix="rec709")
ddcc(chr_i=3,gam_i=2,ofile="C:\coeffs.txt",threads=4,opt=1)
= 96 fps

ConvertToRGB24(matrix="rec709")
= 283 fps

ConvertToRGB24(matrix="rec709")
rgb3d(lutfile="C:\lut.txt")
= 220 fps

ConvertToRGB24(matrix="rec709")
ConvertToRGB32()
= 196 fps

ConvertToRGB24(matrix="rec709")
rgb3d(lutfile="C:\lut.txt")
ConvertToRGB32()
= 170 fps

anyway me likes it A LOT, thanks a lot fellas & merry xmas http://forum-images.hardware.fr/images/perso/astrid72.gif

if it could work natively/output in RGB32 & support RAR format for the lut file(so I can copy them on a ramdisk), that would be great too...but it's already awesome as it is!

[yesgrey]

tritical,
It's great as it is, but I would like to make a suggestion... I think the name rgb3dlut would be a little more suggestive...

leeperry,
I was worried to not have hear anything yet from you...;-)

[leeperry]

I was worried to not have hear anything yet from you...;-)
well when it comes to Reclock/ffdshow/KMP/HR or 3D LUT's, I'm all ears :p

besides, I think it's good to have real world benchmarks to see how efficient the new code is :o

PS: the lut stuff seems to be more accurate than the realtime SSE3 mode...or at least the PNG's are slightly bigger than both the SSE3 mode & the PS script.

[tritical]

I changed the name to rgb3dlut, and added rgb32 support.

[leeperry]

awesome, thanks a bunch tritical!

ConvertToRGB32(matrix="rec709")
rgb3dlut(lutfile="C:\lut.txt")
= 216 fps

I've RAR'ed up the LUT files(1 mb a pop) on a ramdisk and I unrar them through a rar.exe batch, so everything's cool...damn I'm drunk, gonna get some more Champagne :D

[yesgrey]

PS: the lut stuff seems to be more accurate than the realtime SSE3 mode...
Read the readme.txt file.
With all calculations not needed to be done in Real-time, no simplifications are used, the values in the 3D LUT are as accurate as they could be!

[yesgrey]

tritical,
Thanks a lot for your work, and also for your curiosity in knowing how slow my suggestion would be...;)
:thanks:

[leeperry]

Read the readme.txt file.
With all calculations not needed to be done in Real-time, no simplifications are used, the values in the 3D LUT are as accurate as they could be!
right, I remembered I read something about that, but I couldn't find it in this thread :rolleyes:

[yesgrey]

tritical,
Why do you request all chromaticity coordinates in the file. Is not the rule x+y+z=1 always valid? Usually, the colorimeters only give us the x,y coordinates, so we have to perform the calculation above to get the z coordinate...

Another question: in the readme, you don't say what is the gamma for SMPTC-E and EBU, only the chromaticity coordinates...

If you release a new version to remove the z coordinates, also take a look at this, in your post #54, maybe you still want to change it, even not being needed...

If you go through all the calculations based on how D65 is defined, 0.312713,0.329016,0.358271 are closer to the true values. BT.709 just rounds to 4 decimal places. I guess the BT.709 chromaticity coordinates definition in ddcc should be changed to the truncated values, but I don't think the difference is enough to matter (i.e. release a new version just for that).

[leeperry]

colors look definitely even more true to life with that LUT stuff, quite a blast to watch BARAKA in BD in its native gamut this accurately :eek:

I tried gam_i=2,gam_o=5 as yesgrey suggested, but then the gamma curves were not dark enough...gam_i=5,gam_o=5 looks identical to the PS script gamma-wise and really good on my ±2.2 calibrated display :cool:

[tritical]

I require z in the files just because that's how I wrote it. Calculating z isn't that much extra work :p. I don't list gamma functions for SMPTE-C because it doesn't specify a specific gamma function. AFAIK, with SMPTE-C you should use NTSC gamma (BT.470-2 System M, 2.2 gamma) for NTSC stuff and PAL gamma (BT.470-2 System B,G, 2.8 gamma) for PAL stuff. EBU should usually use BT.470-2 System B,G gamma AFAIK. On the truncating of white point coefficients for BT.709, I'm not sure if the standard actually states truncated values or simply says the white point is D65 illuminant (in which case more accurate values would be fine). Either way, the difference wouldn't be noticeable. I might change it in the next version.

One thing I definitely want to change is the way out-of-gamut colors are handled when using lut creation. Right now, each channel (R,G,B) is simply capped to [0,1] (before gamma is applied). That method is used for speed. A better way would be to adjust all channels together. In the case of values < 0, subtract min(r,g,b,0) from all channels (i.e. add white). In the case of values > 1, scale all channels using the same value such that the maximum channel is at 1.

[leeperry]

..that still wouldn't show off-gamut colors I guess ? :D

actually that'd be great to have an option to set off-gamut colors to blink or sumthing(for troubleshooting purposes), because from what yesgrey3 said the most saturated tints are hardly ever used...and I've never witnessed banding myself.

it would also be great if you could import a .chc file from Color.HCFR and use the primaries/secondaries saturations data to counterbalance it in the LUT :eek:

I was discussing it with JohnAd here, he said it would be possible :
http://www.avsforum.com/avs-vb/showthread.php?p=14594838#post14594838

:thanks:

[yesgrey]

Either way, the difference wouldn't be noticeable.
I agree with that, that's why I think it's preferable using the truncated values. The standard says D65 and the x,y coordinates as 0.3127, 0.3290. I agree that the other values are more accurate, but using the truncated values avoid the repetition of this question over and over... do as you prefer, for me it's not an issue.:)

If SMPTE-C and EBU do not specify any gamma curve, should we use option 5 but using the gamma values you suggested?
I think it would be a good idea to include in the readme that explanation about the gamma curves for SMPTE-C and EBU.

[yesgrey]

actually that'd be great to have an option to set off-gamut colors to blink or sumthing(for debug purposes), because from what yesgrey3 said the most saturated tints are hardly ever used...and I've never witnessed banding myself.

With the current displays, the off-gamut colors is almost not an issue, because their gammuts exceed the standards gammuts; maybe only ntsc could have some problems with the reds...

The more problematic would be the inside-gamut colors. Due to the PC only working with 8 bit per component, the same as the video sources, some source colors will be showed as the same color after the gammut conversion (shrinking).
If we had 10 bit per component color depth in our PCs... some graphics cards already support it, but Windows don't.:(

[tritical]

@yesgrey3
For SMPTE-C I would use gam_i/o = 3 for NTSC and 4 for PAL, which is the same as using 5 but with 2.2 or 2.8 gamma respectively. I will add that to the readme.

@leeperry
it would also be great if you could import a .chc file from Color.HCFR and use the primaries/secondaries saturations data to counterbalance it in the LUT
In the saturation error graph you posted in the avsforum thread, how is saturation being computed?... From Luv colorspace or XYZ or HSV? How are those % errors computed? Is it computing the saturation of the desired color, the saturation of the color measured by the probe, and then computing the relative error? What values are in the .chc file (I can't download the file you posted since I don't have an account there). Given those values, how do you propose to modify the calculation?

[leeperry]

@leeperry

In the saturation error graph you posted in the avsforum thread, how is saturation being computed?... From Luv colorspace or XYZ or HSV? How are those % errors computed? Is it computing the saturation of the desired color, the saturation of the color measured by the probe, and then computing the relative error? What values are in the .chc file (I can't download the file you posted since I don't have an account there). Given those values, how do you propose to modify the calculation?
apparently JohnAd could answer you on that, as he managed to import the .chc data and try to fix it through the PS script.

but because it was a very simple 3D LUT in the PS script, it wasn't quite possible to fix it entirely.

with your LUT stuff, you should be able to take it in account in the conversion computing I would guess ?

all I can say right now is that there is an option in Color.HCFR to choose whether you want the saturations to be computed from the reference gamut or from the display native gamut.

I will ask your questions to one of the Color.HCFR coders, and I'll get back to you.

I've uploaded the CHC file here BTW :
http://www.yousendit.com/download/TTZuUWVuTWN6RStGa1E9PQ

you need Color.HCFR 2.01 :
http://www.homecinema-fr.com/colorimetre/release/Setup_v2_0_1.exe

For SMPTE-C I would use gam_i/o = 3 for NTSC and 4 for PAL, which is the same as using 5 but with 2.2 or 2.8 gamma respectively. I will add that to the readme.
well SMPTE-C is used on US/ASIAN DVD/BD, and EBU on EUR BD/DVD...you're saying we should use 3/3 for SMPTE-C stuff ? what about EBU stuff ?

what if you have values at the end of the ofile and still specify gam_o in the AVS call, is gam_o prefered ?

I've tried to put this at the end of my ofile(same values as gam_o=3) but it gave much darker gamma than gam_o=5 :
1
0.0
1.0/2.2
0.0
0.0

this is 0.45454545~ where gam_o=5 is 0.45...the difference I see seems much bigger than that, maybe the "/2.2" part has been ignored :confused:

I was under the impression than 5/5 worked in any given situation :rolleyes:

[tetsuo55]

With the current displays, the off-gamut colors is almost not an issue, because their gammuts exceed the standards gammuts; maybe only ntsc could have some problems with the reds...

The more problematic would be the inside-gamut colors. Due to the PC only working with 8 bit per component, the same as the video sources, some source colors will be showed as the same color after the gammut conversion (shrinking).
If we had 10 bit per component color depth in our PCs... some graphics cards already support it, but Windows don't.:(

Windows 7 has full support for 10bit, it goes as far as 16bit actually.
I personally cannot wait because i have a 10bit videocard AND a 10bit display

I really hope Windows 7 is smart enough to detect all the different colorspaces and converts them accurately to 10bit BT.709

[leeperry]

i have a 10bit videocard
the TMDS encoder of your graphic card works in 3*8 bits, no soup for you :D

[tetsuo55]

the TMDS encoder of your graphic card works in 3*8 bits, no soup for you :D


Why do you think so?

According to all ATI documentation i read on the subject it has full support, also read some posts on random forums where people are using it.(incidentally with almost the same panel as i have)

Quoting Ati's PR:
-Full 30-bit display processing
-Spatial/temporal dithering provides 30-bit color quality on 24-bit and 18-bit displays
-Primary supports 18-, 24-, and 30-bit digital displays at all resolutions up to 1920x1200 (single-link DVI) or 2560x1600 (dual-link DVI)1
Secondary supports 18-, 24-, and 30-bit digital displays at all resolutions up to 1920x1200 (single-link DVI only)1

[leeperry]

According to all ATI documentation i read on the subject it has full support, also read some posts on random forums where people are using it.(incidentally with almost the same panel as i have)
well we've discussed this a bit with Seb.26(an occasional ffdshow coder from HCFR), basically you can output 30 bits on VGA....considering it's analog and the LUT is 3*10 bits, you can even measure the LUT accuracy with ARGYLLCMS and a colorimeter.

DVI/HDMI 1.0 is 3*8 bits TMDS, and I don't see any >1.0 HDMI graphic card at this point...so these nice HDMI 1.3 inputs on top of the line displays are m00t at this point(even so considering there's no 30 bits source at this point, and there won't be before a long while).

so 30 bits/xvYCC would only be useful to the HTPC color freaks...like us :D

apparently DVI can only do 30 bits in dual link configuration :
http://techreport.com/forums/viewtopic.php?t=47715

but dual link DVI & HDMI 1.3 are not compatible AFAIK...what's your display that does 30 bits on DVI ?

[tetsuo55]

Interesting...

I have a Sony HDTV, its a 1080P/10bit(30bit) panel.
My current videocard is a HD2400pro.

The card seems to have a HDMI 1.2 port, which supports everything except TrueHD/DTS-MA bitstream and 10bit(30bit)
My card does support up to 10bit(30bit) but only with DVI

I guess you're right.
Currently you need a panel that supports 10bit over DVI, not HDMI(Many of these panels exist, but they are small and expensive as they are sold to the graphic industry)

I know there is no content available, but at least it would fix the fact that all are mediacenters are stuck in sRGB(which is the only colorspace windows understands)

[yesgrey]

Windows 7 has full support for 10bit, it goes as far as 16bit actually.

I was hoping for this...
In fact, when they start W7 development, I have posted to their blog requesting that... I don't think they have done by my request, but it's great to know that it's coming...

Maybe you can not use it with the hdmi port, but with the vga port yes, I believe the dacs are 10 or 12bit...

[leeperry]

there's still no HDMI 1.3 soundcard that can do bitstream....so wait for a (long) while and HDMI 1.3 graphic cards will pop up, eventually :D

many pj work internally in 10/12 bits, meaning the gamma/colorimetry settings in the OSD won't create terrible banding

even if we get HDMI 1.3 graphic cards & 10 bits native windows 7, don't count on anyone but tritical, yesgrey3 & JohnAd to watch our back.

just like Reclock, ppl who care about butter smooth movies and proper colorspaces are not legions.

actually, there's this card from Asus that looked most promising "12-bit gamma correction" :
http://vr-zone.com/articles/ASUS_Combines_Splendid_HD_w_HD3850_MXM_Card/5753.html?doc=5753

too bad they didn't put HDMI 1.3 outputs :rolleyes:

[tritical]

Well, after reading the documentation of color.hcfr I know how they calculate saturation (euclidean distance from the white point in xyY space) and the saturation error percentages. However, I'm still not sure how you'd go about accuractly correcting anything. All you have are measures for a few points that fall along the 6 lines running from the white point to the 3 primary and 3 secondary colors in xyY colorspace.

[leeperry]

actually Colorfacts doesn't measure these saturations at all.

all you get is gray levels & gamut measurements, so maybe it's not that important after all.

in my case cyan is going down the drain >70%, and other colors are oversaturated...if you tell me what data YOU'd need, I can try to get the Color.HCFR coders to provide you with them.

http://pix.nofrag.com/0/2/8/a990a92dce9da10f80a7f76c10e18.png

[yesgrey]

For SMPTE-C I would use gam_i/o = 3 for NTSC and 4 for PAL, which is the same as using 5 but with 2.2 or 2.8 gamma respectively. I will add that to the readme.


tritical,
This is my understanding of all this formats thing...

For the output, whichever format we use, I think we should always use our display coordinates, and gam_o=5 with a gamma value equal to our display gamma... if our display has a gamma of 2.0, we should use 2.0.

For PAL we should use BT.470-2 System B,G
primaries and gamma curve, chr_i=2 and gam_i=4.

and from this (http://en.wikipedia.org/wiki/NTSC), it appears that for NTSC we should use SMPTE C primaries and SMPTE 170M gamma curve, chr_i=3 and gam_i=1.

What do you think of this?

[yesgrey]

leeperry,

could you post the same graph but without performing the color correction? Just with the original primaries and secondaries?

Thanks.

[leeperry]

leeperry,

could you post the same graph but without performing the color correction? Just with the original primaries and secondaries?

Thanks.
well that's the second part of the problem actually.
it's easy to measure them with the automatic test patterns built into Color.HCFR...but then it's done through simple GDI(no DS filter).

to do it through ddcc, you'd need to use the manual DVD test patterns.....and to get this chart you need to go through at least 24 of them, enough to turn anyone nuts.

so I've asked the Color.HCFR guys if they could somehow drive MPC by sending "next chapter" keystrokes, but well if these data can't be "processed" into ddcc's LUT then it's pointless.

[tetsuo55]

there's still no HDMI 1.3 soundcard that can do bitstream....so wait for a (long) while and HDMI 1.3 graphic cards will pop up, eventually :D

many pj work internally in 10/12 bits, meaning the gamma/colorimetry settings in the OSD won't create terrible banding

even if we get HDMI 1.3 graphic cards & 10 bits native windows 7, don't count on anyone but tritical, yesgrey3 & JohnAd to watch our back.

just like Reclock, ppl who care about butter smooth movies and proper colorspaces are not legions.

actually, there's this card from Asus that looked most promising "12-bit gamma correction" :
http://vr-zone.com/articles/ASUS_Combines_Splendid_HD_w_HD3850_MXM_Card/5753.html?doc=5753

too bad they didn't put HDMI 1.3 outputs :rolleyes:Auzentech has a HDMI1.3 soundcard, you connect the device or videocard with 1.2 or lower to the hdmi in on the soundcard, then it converts it to hdmi1.3(this does not add any new information except for the audio)

[leeperry]

so they finally rlsed it, and it does HD bitstream audio ?
I guess the fees to offer HDMI 1.3 must be very high, but soon or later we'll get HDMI 1.3 graphic cards I guess.

maybe with the new nvidia cards early next year ? :)

[yesgrey]

tritical,
I had another idea to increase speed and lower the size of the 3D LUT file... Why not include in the 3D LUT also the YUV->RGB conversion? (I think YUY2 would be enough)
The LUT size would be: 3x220x254x254 ~ 40.6MB < 48MB.
And is just adding the specification of the matrix we would like to use (BT.601 or BT.709) and the levels we want... (you already have part of that done in Colormatrix)

So the 3D LUT would become equivalent to:
Y'U'V'->R'G'B'->RGB->Color Correction->RGB->R'G'B'.

I think this would be a more powerfull and interesting solution, because some people only need the YUV->RGB conversion with a 3D LUT... perhaps it would be faster than the ConvertToRGB32()...;)

If you don't have time let me know, I could try changing just the C part of the code for calculating the 3D LUT file...

[tritical]

I am working on the next version, current list of changes to make:

1.) don't require z in the input files
2.) change the way out-of-gamut colors are handled
3.) Add an option to ddcc to use lut (basically it will just programmically invoke rgb3dlut after creating the lut file so you don't have to open the script twice).

@yesgrey3
Coding a yuy2 to rgb 3d lut filter wouldn't take much work (small modifications to rgb3dlut). I might include it in the next release of ddcc. If you could write a program to actually calculate the LUT files for use with it that would great :).

@leeperry
the difference I see seems much bigger than that, maybe the "/2.2" part has been ignored
It will be... it just reads fp values (sscanf(buf,"%lf",&val)).

On the topic of saturation correction. Maybe you could ask JohnAd what he did, because I still don't know how you could make any corrections based on the data you have.

[leeperry]

Coding a yuy2 to rgb 3d lut filter wouldn't take much work (small modifications to rgb3dlut). I might include it in the next release of ddcc.
well mark0077 has been recently pointing out that the nvidia drivers in YV12(using software renderers) were offering better chroma upsample than even ConvertToRGB32() :
http://forum.doom9.org/showpost.php?p=1230065&postcount=27

I'm not sure how they do it, but could you please set up a very HQ chroma upsample scheme ? like spline36 ? apparently Convert() is using bicubic, and so does ffdshow....but the ugly ATi drivers would be using pointresize(from what Leak said) :rolleyes:

the test sample VOB is available here :
http://www.mediafire.com/?9g9ddlfzxhv
@leeperry

It will be... it just reads fp values (sscanf(buf,"%lf",&val)).

On the topic of saturation correction. Maybe you could ask JohnAd what he did, because I still don't know how you could make any corrections based on the data you have.
well the real issue will be to verify how it went, the HCFR coders are too busy with the next release to set up some sort of automatization with MPC and the DVD manual patterns....and going through 24 manual patterns is too cumbersome. nevertheless, I will ask JohnAd :)

if at some point, you could sorta give some hints for the gam_i/o formulas to use, that'd be really great.
atm I'm using 5/5 and it looks good to me on my ±2.2 calibrated display....maybe I should put 0.454545454545 instead of 0.45 ?

[tritical]

What playback chain do I need to use to ensure that nvidia's drivers are doing the upsampling (I have a 9800gtx with 178.28 drivers)? Avisynth's color conversions operate as described here: http://avisynth.org/Sampling. The only problems with the descriptions on that page are that the rgb->yuy2 conversion doesn't use a [1 2 1] kernel, and the C code version of yuy2->rgb doesn't average to get every other chroma sample (see http://forum.doom9.org/archive/index.php/t-129316.html). ffdshow's hq rgb conversion uses avisynth's code (as in taken out of avisynth cvs) so it uses the same sampling. Any differences between avisynth's conversion and ffdshow's hq rgb conversion are probably due to coefficient scaling.

My plan for yuy2 to rgb lut is to use linear interpolation to create the chroma for the odd pixels (same method described on the avisynth sampling page).

[leeperry]

the test was conducted in YV12 with EVR on Vista SP1 :
http://forum.doom9.org/showthread.php?t=143818

apparently what also helped was Leak's progressively upsampling PS script(embedded in MPC HC) :
http://forum.doom9.org/showpost.php?p=1184975&postcount=32

that'd be more convenient if you could allow YV12 input instead of YUY2 if any possible, so ddcc gets the untouched video stream(no ConvertToYUY2() implied).

[yesgrey]

The LUT size would be: 3x220x254x254 ~ 40.6MB < 48MB.
This is wrong. Since we also want to convert values of Y<16 and Y>235, the correct size would be:
3x254x254x254 ~ 48MB.
In YUV values 0 and 255 should be clipped. So, I think the better is just keeping the same size of the RGB, 3x256x256x256...

If you could write a program to actually calculate the LUT files for use with it that would great :).
Yes, I am thinking in this. I will use parts of your code, and am also thinking in finding a way of performing the correction for the saturations problem.
Probably I will use Fortran - is my prefered language -, are you ok with it?

[yesgrey]

that'd be more convenient if you could allow YV12 input instead of YUY2 if any possible

This is not possible.
A LUT maps an input value to an output value.
In YV12 you don't have all the input values, only half, so you cannot map it through the 3D LUT. You have to compute the values that are missing, so, you always have to convert to YUY2. The idea of including the YUV->RGB conversion is only to avoid the extra cpu load of YUV->RGB conversion, you cannot avoid the YV12->YUY2 conversion...

Of course rgb3dlut function could support YV12 input, but then it will have to perform the YV12->YUY2 internally beforing mapping it out via the 3D LUT, so there is no gain in avoiding ConvertToYUY2...;)

[leeperry]

you cannot avoid the YV12->YUY2 conversion
right :D
well still, seeing how inaccurate ConvertToRGB32() is, maybe the YV12>YUY2 conversion could also benefit from some higher quality upsample :o

[tritical]

In YUV values 0 and 255 should be clipped. So, I think the better is just keeping the same size of the RGB, 3x256x256x256...
I agree... it also makes the code simpler. My code makes the lookup in to the table as: ((V<<16)+(U<<8)+Y)*3, then the three bytes at that location are stored in b,g,r order.

Yes, I am thinking in this. I will use parts of your code, and am also thinking in finding a way of performing the correction for the saturations problem.
Probably I will use Fortran - is my prefered language -, are you ok with it?
Yep.

[IanB]

YV12 to RGB is possible in exactly the same way as the proposed YUY2 to RGB, you just have to pre-interpolate vertically as well.

As for out of gamut values, maintaining Hue is the all important factor. The eye is very sensitive to hue, especially skin tones (pink), but is quite ambivalent about saturation particularly at high saturations.

In YUV space, Hue is analogous to the ratio of U/V, saturation is analogous to the length of the UV vector, i.e. U**2+V**2, so just clamp the offending U or V value at the gamut limit and scale the other value to maintain the original ratio of the preclamped values.

e.g. Using normalised values, conversion results in U=1.06, V=0.42. Clamp U at 1.0, set V=1.0*(0.42/1.06)=0.396

In RGB space hue is the ratio of the 2 dominant primaries after the value of the minority primary has been subtracted from each. These values are best corrected in linear (non-gamma) space.

e.g. Using normalised values, conversion results in R=0.72, G=1.07, B=0.42.
B is the current minority primary, R'=0.72-0.42=0.3 and G'=1.07-0.42=0.65
Maintain Hue by keeping the ratio G'/R'=0.65/0.3=2.167 constant.
Clamp G to 1.0, scale R''=R'*G''/G'=0.3*(1.0-0.42)/0.65=0.267
Thus adjusted values become R=R''+B=0.267+0.42=0.688, G=1.0, B=0.42
Apply Gamma correction and scale to output range.

[tritical]

I agree that maintaining hue while decreasing brightness and saturation is the best way to handle the case of values greater than 1.0, but I don't follow your math. You say that the ratio of G'/R' must be kept constant, and that ratio is equal to (G-B)/(R-B). Yet you only modify G and R. It's not possible for that ratio to remain the same without scaling all three values by the same factor (and it will stay constant if you do). I think the way to do it is simply to divide all three values by the maximum value... which in your example would give:

R=0.72/1.07, G=1.07/1.07, B=0.42/1.07
R=0.673, G=1.000, B=0.393

[yesgrey]

YV12 to RGB is possible in exactly the same way as the proposed YUY2 to RGB, you just have to pre-interpolate vertically as well.
Yes, I referred that, but it's not possible to include the interpolation in the 3D LUT values. The interpolation should be done before mapping with the 3D LUT, it's the only way to know what to map from...;)

[yesgrey]

These values are best corrected in linear (non-gamma) space.
Well, these values only appear in the linear space, when we are performing the gammut conversion, so we are safe.:)

[yesgrey]

that ratio is equal to (G-B)/(R-B). Yet you only modify G and R. It's not possible for that ratio to remain the same without scaling all three values by the same factor...
It's possible. Look:
a) C = (G-B)/(R-B); you know R,G,B and calculate C
b) C = (G''-B)/(R''-B)
c) Set G''=1.0
d) R'' = (G''-B)/C + B; you know G'',B,C and calculate R''

I think this method is better because affects less the values not to be clipped.

[tritical]

You're are right. I must have been up to late last night.

However, I still don't think it's clear cut that that method is better. In terms of HSL coordinates IanB's method is better... both methods produce the same H/S values, but IanB's method reduces L less. However, in terms of HSV coordinates my method is better... both methods produce the same H/V values, but IanB's method reduces S while mine does not.

[yesgrey]

tritical,
Your method has the advantage of being much easier to code.
Since the hue is kept with both methods, and the HSL vs HSV is not conclusive, maybe it's better just doing the easiest...;)
IanB, what do you think?

[IanB]

Yes, both methods are compromises, the value is out of gamut, you must compensate it.

Tritical's all scaling method maintains hue and saturation at the expense of luminance.

My dominant pair scaling method maintains hue at the expense of both saturation and luminance, but to a lesser degree.

Another alternative is to prune saturation even harder and maintain luminance. i.e.

R=0.72, G=0.42, B=1.07.
Y=0.2126*0.72+0.7152*0.42+0.0722*1.07=0.531

All scaling
R=0.673, G=0.393, B=1.0
Y=0.496

Dominant pair scaling
R=0.688, G=0.42, B=1.0
Y=0.519

Plus minority primary scaling
R=0.695, G=0.437, B=1.0
Y=0.531

Each method has weaknesses. Probably a selection of methods might be needed, depending on how the colour is out of gamut. I chose to make the Blue channel over valued in this example because it was not possible to achieve the Y correction needed with the original Green over value example. Minority primary scaling can work best for excess Blue excursions and worst for excess Green excursions.

Likewise dominant pair scaling should not be used for low saturation examples, the eye is quite sensitive to changes in low saturation values. Think of white point difference between 9300K, 6500K and 5000K to the eye these are all very low saturation blues and reds

And of course small changes in mid scale luminance for high saturation colours are more noticable, making all scaling a less desirable choice.

We are probably being excessively picky here, but gamut correction is for the perfectionist's anyway. Most of the population watch coloured TV and just don't care/don't know any better.

[yesgrey]

We are probably being excessively picky here, but gamut correction is for the perfectionist's anyway.
Well, with the 3D LUT method, implementing all of this will not increase the cpu load, and it's not too hard to code, so let's go be picky.:)

[tritical]

I'll admit that I'm one of those people who don't really care. When I watch stuff on my computer I never worry about colorimetry or gamut correction. I have a cheap 19 inch flat panel monitor :).

Anyways, initial speed tests show the yuy2->rgb lut method is much faster than calling converttorgb32().rgb3dlut(), but it isn't going to beat out just converttorgb32(). Some tests on my quad core Q6600 using 720x480 mpeg2 video decoded with dgdecode to yv12:

220fps converttorgb32()

76fps converttorgb32().ddcc(threads=1)
111fps converttorgb32().ddcc(threads=2)
143fps converttorgb32().ddcc(threads=4)

118fps converttorgb32().rgb3dlut(threads=1)
138fps converttorgb32().rgb3dlut(threads=2)
145fps converttorgb32().rgb3dlut(threads=4)

175fps converttoyuy2().rgb3dlut(threads=1)
200fps converttoyuy2().rgb3dlut(threads=2)
210fps converttoyuy2().rgb3dlut(threads=4)

These results seem a little strange... have to investigate.

[leeperry]

I'll admit that I'm one of those people who don't really care. When I watch stuff on my computer I never worry about colorimetry or gamut correction. I have a cheap 19 inch flat panel monitor :).
I believe you start to care when you use a projector, as these things have wide gamuts.....and a flashy picture on a big projection screen looks really ugly :o

nice speed improvement! with proper chroma upsampling at that...can't wait to try a new beta :thanks:

[yesgrey]

I'll admit that I'm one of those people who don't really care. When I watch stuff on my computer I never worry about colorimetry or gamut correction. I have a cheap 19 inch flat panel monitor :).
But have you tryed it? probably not, because I think that maybe you don't know the coordinates of your monitor primaries.
I also don't care much about it, because since I'm not english native, sometimes I have to use subtitles, and with it I spend more time looking at the subtitles than at the movie colors...:D but when I disable the subtitles and listen to the audio directly, the difference is very noticeable and pleasant.:)

Anyways, initial speed tests show the yuy2->rgb lut method is much faster than calling converttorgb32().rgb3dlut(), but it isn't going to beat out just converttorgb32().
Well, if we could load the 3D LUT in the graphics card memory and performing the mapping through it, maybe it would beat the convertorgb32() alone. Even as it is now, is very fast, but it's strange that the speed changes with the number of threads... it should be limited by the memory access speed, right?

[tritical]

I put up the new version. I thought the fps numbers were weird compared to tests I ran two weeks ago, but those numbers were on a dual core using xvid encoded input. For rgb3dlut speed increases going from 1 to 2 threads, but hardly at all from 2 to 4. I think it is probably because this quad core is actually two dual cores each with its own 4MB L2 cache.

[leeperry]

nice! trying it as we speak...it looks fast as hell :eek:

but because ddcc takes care of the RGB conversion, shouldn't it let us choose the YCbCr>RGB decoding matrix ?

Rec. ITU-R BT.601-5 => PAL / SECAM / NTSC (SD)
Rec. ITU-R BT.709-4 => HD

you can have a SD video w/ EBU gamut, or a HD video w/ SMPTE-C gamut as you know.

also letting us choose the input levels would be nice, as I'm sending 0-255 content to ddcc(or just assume 0-255 input and let ppl use colorYUV(levels="tv->pc"))

and I've played around again w/ gam_i/o=3 or 5 but I always go back to 5/5.

[yesgrey]

well mark0077 has been recently pointing out that the nvidia drivers in YV12(using software renderers) were offering better chroma upsample than even ConvertToRGB32() :
http://forum.doom9.org/showpost.php?p=1230065&postcount=27

I'm not sure how they do it, but could you please set up a very HQ chroma upsample scheme ? like spline36 ?
Try using avisynth 2.6. With it you can select the resizer for the chroma upsampling, even spline64.

[yesgrey]

but because ddcc takes care of the RGB conversion, shouldn't it let us choose the YCbCr>RGB decoding matrix ?
Yes. I will write a little program, based on some of ddcc code, for creating the 3D LUT files. I will consider this, and even the possibility of defining custom levels for the conversion, like it's possible now with ffdshow. This way, I could use just rgb3dlut and control myself the yuv->rgb conversion.
I also want to add custom gamma correction, including the possibility of using the display gamma curve.

[leeperry]

Try using avisynth 2.6. With it you can select the resizer for the chroma upsampling, even spline64.
ORLY ? well MT 0.7 doesn't support Avisynth 2.58.
I know chromaresample="spline36" exists, but it's not used for chroma upsampling in RGB32 conversion from what IanB told me.
Yes. I will write a little program, based on some of ddcc code, for creating the 3D LUT files. I will consider this, and even the possibility of defining custom levels for the conversion, like it's possible now with ffdshow. This way, I could use just rgb3dlut and control myself the yuv->rgb conversion.
I also want to add custom gamma correction, including the possibility of using the display gamma curve.
oh OK, I thought tritical would simply add an option to choose REC601/709(and assume 0-255 input), and that everything would be fine :o

anyhow, I failed to understand how I could build the LUT in RGB32 in ddcc() and then use it in YUY2 in rgb3dlut()....so that would explain :D

so rgb3dlut() will assume 0-255 input for YUY2>RGB32 conversion then ?

I'm back to ddcc 1.5 for the time being..or maybe I'll rebuild the LUT's as off-gamut colors seem to be even better handled now :cool:

[leeperry]

OK so I've rebuilt the LUT's, my 2 displays(CRT/DLP pj) both carry off-gamut colors, and the new ddcc looks absolutely stunning :eek:

playing HD SMPTE-C movies yields very true to life colors....actually they've never looked so good :cool:

as soon as I started playing around w/ the PS script, I quickly realized that when you watch movies in their native gamut the contrast ratio increases accordingly....mainly coz colors are perfectly spot-on and not "polluted" anymore.

a friend of mine, who got me into that gamut craziness also told me the same thing....he's got the holy Samsung SP-A800B pj and runs a website about gamuts & stuff : www.hdsoir.com (sorry, french only :o )

@yesgrey : if you can get ddcc working equally accurately in YUY2, that sure will be a blast! :D

[cyberbeing]

I've read the documentation, but I'm a bit confused about how to properly use this filter. I currently have two GDM-F520 CRT monitors calibrated and profiled with an Eye-One Pro spectrophotometer to D65 2.2 for photoediting/graphic design use.

Could someone give me a step-by-step of how to create the required txt file(s)?

Do I only need to create an ofile?
Why would I need an ifile?
Is lutfile automatically created by ddcc or do I have to create that as well?

It sounds like rgb3dlut is faster and therefore preferred to ddcc?
Once I have the required files, what would be an example script for bt.709 video? For bt.601 video?
How should I setup ffdshow's output panel?

[yesgrey]

First you have to measure the chromaticity coordinates of your displays, in this case of both your crt's, you will need it for filling the data in the ofile.
Generally you don't need the ifile, only if you will use source material from a standard different than the already included in ddcc.

Currently the 3D lut file is created by ddcc when you specify a name for it when calling the function. Then, rgb3dlut uses that file.

I'm writting a little program, using some parts of ddcc, to create the 3D LUT files in an alternative way. These files would then be used by rgb3dlut.
Wait a few more days (I'm waiting for the weekend to finish it), I will post my program with instructions and scripts for the more usual setups.;)

[cyberbeing]

First you have to measure the chromaticity coordinates of your displays
That is my main confusion. I'm unsure what chromaticity coordinates you are talking about.

What program do I use? If not easy to figure out, how would I measure what is needed?
What do I need to measure?
After measuring, where in the results would I find the chromaticity coordinates needed for the ofile?

[yesgrey]

What program do I use? If not easy to figure out, how would I measure what is needed?

Well, you can look here (http://www.avsforum.com/avs-vb/showthread.php?t=912720) to understand what are the coordinates we need, and you can look here (http://www.homecinema-fr.com/colorimetre/index_en.php) for a free software to help you measuring the coordinates.

[cyberbeing]

OK, I think I figured it out, but ddcc is throwing an error:
ddcc: error reading from file (C:\Program Files\Avisynth 2.5\bt709.txt)

I was using the following script to try to generate the lut file:

dss2("F:\test.mkv",fps=23.976)
ConvertToRGB24()
ddcc(chr_i=0, gam_i=1, ofile="C:\Program Files\AviSynth 2.5\bt709.txt", lutfile="C:\Program Files\AviSynth 2.5\3D_LUT_BT709.txt", threads=0, opt=-1)


the bt709.txt contains:
http://img144.imageshack.us/img144/296/ofilegi8.png

What am I doing wrong?

[yesgrey]

What am I doing wrong?
If you are using ddcc v1.6, you should supply only the x and y coordinates. In your file you have specified x,y and z.

[cyberbeing]

Oh, I guess that explains the error. I was just basing my file off one of the old examples earlier in this thread, but I didn't catch it had changed. I removed the z value and it worked.

Now I've run into another problem. Using YUY2 input with rgb3dlut gives me messed up chroma (http://img144.imageshack.us/img144/5751/0000ur6.png):
ConvertToYUY2()
rgb3dlut(lutfile="C:\Program Files\AviSynth 2.5\3D_LUT_BT709.txt", threads=2)

If I use RGB input the colors seem correct:
ConvertToRGB32(matrix="Rec709")
rgb3dlut(lutfile="C:\Program Files\AviSynth 2.5\3D_LUT_BT709.txt", threads=2)

Am I missing something else, or in ddcc 1.6 are you unable to use YUY2 input with rgb3dlut?

[yesgrey]

Now I've run into another problem. Using YUY2 input with rgb3dlut gives me messed up chroma...
Am I missing something else, or in ddcc 1.6 are you unable to use YUY2 input with rgb3dlut?
Read this (http://forum.doom9.org/showthread.php?p=1236588#post1236588).

[leeperry]

Read this (http://forum.doom9.org/showthread.php?p=1236588#post1236588).
I was reading again that you would let us input the display gamma curve.

does that mean that we could input all the data from ColorHCFR ? 9 points for each primary color ? :cool:

http://pix.nofrag.com/2/f/a/731617f4652b055fc4d4498489ac4.png

and to get back on the orangey red problem in SMPTE-C, I'm often seeing this kind of UGLY red's :
http://pix.nofrag.com/d/e/c/18ccaea39e7817bbb9a670494f9a8tt.jpg (http://pix.nofrag.com/d/e/c/18ccaea39e7817bbb9a670494f9a8.html)

this is the Dumb & Dumber BD, mostly the damn cameras didn't sample the Ferrari red properly...the major flaw in SMPTE-C.

in oversaturated REC709 demos, red cars are actually red :
http://img141.imageshack.us/img141/7003/31bv.jpg

it's discussed here :
http://www.avsforum.com/avs-vb/showthread.php?t=1038602

and here :
http://www.google.com/search?hl=en&q=SMPTE-C+orangey+red&btnG=Search&lr=

EDIT: using this Ferrari colors panel, maybe it was a "Rosso Corsa" red after all....but still undersaturated :rolleyes:

http://www.jb330gt.com/color/1995FerrariBig.jpg

[cyberbeing]

Read this (http://forum.doom9.org/showthread.php?p=1236588#post1236588).

I don't see the answer to my question in what you linked.

It would be much easier if you could just give me simple answers to my questions instead of linking me all over the place... I've already previously read through everything you have linked me to so far before I even thought about asking for help... I need a clear answer beyond what has already been posted... I don't want to play 20 questions...

Can rgb3dlut be used with YUY2 input? The documentation suggests the answer is yes.

What is causing the problem I'm seeing with YUY2 input? The lutfile? Something else?

If the lutfile isn't the problem, how did tritical get it to work with YUY2?

If the lutfile is the problem, then I assume that you are working on an app for creating the needed lutfile for YUY2 input?

[leeperry]

If the lutfile is the problem, then I assume that you are working on an app for creating the needed lutfile for YUY2 input?
it'll be possible when the app will be ready, now just wait like all of us :o

[yesgrey]

It would be much easier if you could just give me simple answers to my questions instead of linking me all over the place...
Some answers are not simple, and is boring having to repeat long answers several times. Each long answer that I repeat, is less time I can put in coding...;)

If the lutfile is the problem, then I assume that you are working on an app for creating the needed lutfile for YUY2 input?
Yes.

[yesgrey]

I was reading again that you would let us input the display gamma curve.

does that mean that we could input all the data from ColorHCFR ? 9 points for each primary color ?
Yes, I am thinking in it, but not in the first release. First it will be just the same as ddcc v1.6, but including YCbCr->RGB conversion.

[leeperry]

Yes, I am thinking in it
http://forum-images.hardware.fr/images/perso/nico54.gif

[cyberbeing]

Yes.
Thank you, that is all I needed to know.

The lutfile is the problem and without a properly created lutfile, it won't work with YUY2 input.

You are working on a solution to make that lutfile, and until it is ready I'll have to wait.
:thanks:

[yesgrey]

Here is a link (http://www.megaupload.com/?d=JAVPPV3Q) with 4 3D LUT files just for testing purposes, to keep you busy while I finish the first version of my little program...;)

The 3D LUTs are only for performing the YCbCr->RGB conversion. This way, we can test the speed and the quality of using rgb3dlut versus other options available for the conversion (like ConverToRGB32, ffdshow, graphic cards drivers).
There are 4 files:
-BT.601 with 16-235 levels
-BT.601 with 0-255 levels
-BT.709 with 16-235 levels
-BT.709 with 0-255 levels

Don't forget that the speed will be exactly the same with the full options available for creating the 3D LUT files.

[leeperry]

awesome, thanks!

so from the readme :

itype -

For yuy2 input this sets how to compute the u/v values for the second y value in each yuyv set.
Possible settings:

0 - duplicate (use u/v of first y value)
1 - linear interpolation (average u/v of first y with u/v of first y in next yuyv set)

ConvertToRGB32(matrix="rec601")
http://www.image-load.eu/out.php/t141613_convert32601.png (http://www.image-load.eu/out.php/i141613_convert32601.png)

ConvertToYUY2()
rgb3dlut(lutfile="C:\3dluts\3dlut_ycbcr_16-235.txt",itype=0)

http://www.image-load.eu/out.php/t141614_ddccitype0.png (http://www.image-load.eu/out.php/i141614_ddccitype0.png)

ConvertToYUY2()
rgb3dlut(lutfile="C:\3dluts\3dlut_ycbcr_16-235.txt",itype=1)

http://www.image-load.eu/out.php/t141615_ddccitype1.png (http://www.image-load.eu/out.php/i141615_ddccitype1.png)

major changes :eek:

the chroma looks a lot more accurate, not some smearing pixelating hollow anymore :eek:

PS: I didn't have the BT.709 LUT when I ran the tests, but this doesn't really matter.

[yesgrey]

YV12 to RGB is possible in exactly the same way as the proposed YUY2 to RGB, you just have to pre-interpolate vertically as well.

Yes, I referred that, but it's not possible to include the interpolation in the 3D LUT values. The interpolation should be done before mapping with the 3D LUT, it's the only way to know what to map from...;)

IanB,
Now that I am messing with YCbCr->RGB conversion, I realized that I haven't understood exactly what you have said in that quote. I thought that YUY2 had all the CbCr data for the correponding Y values, but now I know it doesn't, it only has 1 CbCr data for 2 Y data.
Now I see that the YV12 input could also be used in rgb3dlut. I hope it's not too much work, and that tritical could find the time for doing it...;)

[yesgrey]

the chroma looks a lot more accurate, not some smearing pixelating hollow anymore :eek:
Yes, I agree that rgb3dlut YCbCr->RGB conversion looks better.
I have zoomed to compare, and when the linear interpolation is used (itype=1), the color grading is exactly the same from ConvertToRGB32, but it doesn't have the smearing around it...
This makes me think that if rgb3dlut would also accept YV12 input the result could be even better... maybe ConvertToYUY2 is also adding some smearing of it's own...

[leeperry]

This makes me think that if rgb3dlut would also accept YV12 input the result could be even better... maybe ConvertToYUY2 is also adding some smearing of it's own...
exactly my words when you guys started that YCbCr thingie....tritical's algorithms are a lot more accurate than Convert(), so we should bypass it altogether if any possible :cool:

until your code is ready, I will now use rgb3dlut() twice in a row, once for RGB32 conversion and once for gamut conversion :D

EDIT: I might also need to lower my sharpening in ffdshow, it's too damn sharp now :eek:

[mark0077]

so there is no direct yv12 to rgb32 lut? Is one in production :)

[yesgrey]

so there is no direct yv12 to rgb32 lut? Is one in production :)
The yv12 to rgb32, as the yuy2 to rgb32 conversions could not be performed by a 3D lut. The 3D LUT converts only betwwen YCbCr and RGB, and for that you must have the Y and CbCr values of each pixel.
In YUY2 you have 2 Y and 1 CbCr for 2 pixels, and in YV12 you have 4 Y and 1 CbCr for 4 pixels. The CbCr values for the other 1 or 3 pixels, have to be calculated by rgb3dlut. Currently, it's already doing it for YUY2, let's hope the same could happen for YV12...

So, what we need for YV12->RGB32 is not a new lut, but a new rgb3dlut version...;)

[yesgrey]

leeperry,
Could you also post a screenshot using ffdshow's conversion?
Thanks!

[leeperry]

here's ffdshow in RGB32/601 :

http://thumbnails3.imagebam.com/2490/fe9a1124894461.gif (http://www.imagebam.com/image/fe9a1124894461)

and in RGB32HQ/601 :

http://thumbnails9.imagebam.com/2490/6a0f6e24894469.gif (http://www.imagebam.com/image/6a0f6e24894469)

http://img132.imageshack.us/img132/1697/31370080kq2.png

the file sizes are pretty self-explanatory....the smearing red costs, and RGB32HQ=ConvertToRGB32()

also the black background is actually R0-G0-B0 in ddcc, but in Convert()(full range 601)/ffdshow(full range 601) it's R2-G0-B1 :confused:

the sample is still available here :
http://forum.doom9.org/showpost.php?p=1137196&postcount=1868

and here's some real world comparisons, top is Convert(709), bottom is ddcc(709) :

http://thumbnails16.imagebam.com/2490/f405bd24894487.gif (http://www.imagebam.com/image/f405bd24894487)http://thumbnails15.imagebam.com/2490/f137f324894509.gif (http://www.imagebam.com/image/f137f324894509)http://thumbnails10.imagebam.com/2490/054af024894527.gif (http://www.imagebam.com/image/054af024894527)http://thumbnails11.imagebam.com/2490/208cb824894559.gif (http://www.imagebam.com/image/208cb824894559)
http://thumbnails11.imagebam.com/2490/257a6b24894500.gif (http://www.imagebam.com/image/257a6b24894500)http://thumbnails2.imagebam.com/2490/922c4d24894517.gif (http://www.imagebam.com/image/922c4d24894517)http://thumbnails10.imagebam.com/2490/aa353524894543.gif (http://www.imagebam.com/image/aa353524894543)http://thumbnails14.imagebam.com/2490/635f1324894578.gif (http://www.imagebam.com/image/635f1324894578)

http://img262.imageshack.us/img262/4982/42859825xo0.png

the red shade is slightly different in ddcc, more saturated :confused:

[yesgrey]

as I told you in PM the 2 BT.709 LUT's are mixed up, the 0-255 is actually 16-235 ;)
The Luts designation refers to the black and white in RGB.
The BT.709_0-255 is
Y:16-235 -> RGB: 0-255
and the BT.709_16-235 is
Y:16-235 -> RGB: 16-235

and RGB32HQ=ConvertToRGB32()
As it's supposed to be, the HQ mode uses Avisynth's code.

also the black background is actually R0-G0-B0 in ddcc, but in Convert()(full range 601)/ffdshow(full range 601) it's R2-G0-B1 :confused:

Probably some small error in the matrix coefficients...

[leeperry]

The Luts designation refers to the black and white in RGB.
The BT.709_0-255 is
Y:16-235 -> RGB: 0-255
and the BT.709_16-235 is
Y:16-235 -> RGB: 16-235
well, I think you told me that PC/TV levels didn't matter for colorimetry matters...what I need is full range YUY2 to full range RGB32 conversion(in 601 & 709)

BTW I've definitely had to lower my sharpening in ffdshow one notch, that smearing red was tempering w/ the PQ apparently..

so here we go again :

Convert() 609 / ddcc() 609 / ffdshow 609 RGB32(not HQ, same as the ATi drivers in YV12/YUY2)

http://thumbnails12.imagebam.com/2491/2d15aa24903098.gif (http://www.imagebam.com/image/2d15aa24903098)http://thumbnails14.imagebam.com/2491/4a0ede24903105.gif (http://www.imagebam.com/image/4a0ede24903105)http://thumbnails11.imagebam.com/2491/dd153924903110.gif (http://www.imagebam.com/image/dd153924903110)
http://thumbnails15.imagebam.com/2491/c7b08524903114.gif (http://www.imagebam.com/image/c7b08524903114)http://thumbnails11.imagebam.com/2491/51812224903124.gif (http://www.imagebam.com/image/51812224903124)http://thumbnails14.imagebam.com/2491/ec623824903131.gif (http://www.imagebam.com/image/ec623824903131)

can't really notice less red blocking between Convert() and ddcc() on these screenshots, only that red is more saturated w/ ddcc.

[mark0077]

So how will the new ffdshow rgb32hq that is being developed compare to the final version of this? I assume by definition they should be exactly the same but will they?

[leeperry]

So how will the new ffdshow rgb32hq that is being developed compare to the final version of this?
you'll have to pray for some good soul to add tritical's chroma upsampling algorithm in ffdshow :o

BTW, if you care to run compares w/ the nvidia drivers...you're most welcome :cool:

[mark0077]

I will do the compare in about 2 hours when I'm home from work ;) will be interested to see which is better. Pretty sure this conversion to rgb32 in software is a great idea because of the difference between nvidia and ati, not to mention the diff between conversion with nvidia and different renderers (sometimes bad conversion with evr-cp).

I'll post the new pics up soon with my test pattern.

EDIT: Got the code working, no avisynth errors, but ffdshow seems to still do a conversion which means I can't test this :( Any workarounds...? ffdshow musn't know that avisynth is outputting 32bit rgb?

Using ffdshow set to rgb32 output only, and full range and the following two sets of codes, but ffdshow is still doing some conversion

colorYUV(levels="tv->pc")
ConvertToYUY2()
rgb3dlut(lutfile="c:\3dlut_ycbcr_BT601_16-235.txt",itype=0)

and

colorYUV(levels="tv->pc")
ConvertToYUY2()
rgb3dlut(lutfile="c:\3dlut_ycbcr_BT601_0-255.txt",itype=0)

[leeperry]

colorYUV(levels="tv->pc")
ConvertToYUY2()
rgb3dlut(lutfile="c:\3dlut_ycbcr_BT601_16-235.txt",itype=0)

and

colorYUV(levels="tv->pc")
ConvertToYUY2()
rgb3dlut(lutfile="c:\3dlut_ycbcr_BT601_0-255.txt",itype=0)
you don't have to use colorYUV(), I do coz I got a longer script and I want GrainFactory3() to be processed on the full range video...

this SHOULD work :

ConvertToYUY2()
rgb3dlut(lutfile="c:\3dlut_ycbcr_BT601_0-255.txt",itype=1)

610 for SD, 709 for HD of course ;)

we spoke about it in PM, but if you only have RGB32 checked in ffdshow, ffdshow should bypass the Avisynth stream....I dunno what's up :confused:

instead of using Avisynth 2.5.8, you could try 2.5.7 + MT 0.7 on XP ? that's what I got....or as a last resort the alpha 2.60, which is already patched for MT.

try to set the RGB conversion of ffdshow to bogus levels, and see if it has any effect....this doesn't on my box :

http://www.image-load.eu/out.php/i142077_plop0.png

[mark0077]

Disabling ffdshows internal decoders fixed the problem, now avisynth seems to be doing the conversion...

Well I am kind of disappointed. Scaling upto 1920 x 1080 and tried

ffdshow rgb32hq and rgb32lut with the following settings

ConvertToYUY2()
rgb3dlut(lutfile="c:\3dlut_ycbcr_BT601_0-255.txt",itype=1)

Both show the ugly scaling problem again at the edge of the letters in my test. rgb32lut with itype=1 is 2kb larger file size than ffdshow's rgb32hq. The Nvidia yv12 -> rgb32 is absolutely perfectly smooth again as usual.

DISCOVERY ! :P
Now I have a feeling why I am seeing these ugly artifacts on this test, and I may be wrong but I think my hunch might be right in this case. When I view my test as part of the entire DVD, it is the first title and the image is scaled to the correct aspect ratio, BUT when its played alone, as a seperate vob file, this same image is displayed much narrower.

I am wondering if this image I am using to test is originally in the aspect ratio

x pixels : y pixels

and when played as part of the DVD it is signalled to be scaled to a non x:y aspect ratio (as I think it should by just looking at the width of the fonts) to

(x*z) : y

I think this is what is happening, and this horizontal stretching in certain configurations is what is causing these weird little jaggy lines (all but nvidias own conversions). What can be done software wise to counteract this. I am trying to find a combination of ffdshow resize, before and after avisynth to see what nvidia might be doing, or what order they do things in.

Any input on this would be great, because I think this may be important. Could it be that the resizers that we / I use are better in yv12/yuy2, than rgb32? This is my hunch so far anyways.

[yesgrey]

but ffdshow seems to still do a conversion which means I can't test this :( Any workarounds...? ffdshow musn't know that avisynth is outputting 32bit rgb?

Are you using any filter after the avisynth filter in ffdshow?
Some of the filters require YV12 input, so, even if you are outputing RGB32 from the avisynth filter, ffdshow will convert back to YV12 to feed the filter, then, it will convert to RGB32.

[leeperry]

Now I have a feeling why I am seeing these ugly artifacts on this test, and I may be wrong but I think my hunch might be right in this case. When I view my test as part of the entire DVD, it is the first title and the image is scaled to the correct aspect ratio, BUT when its played alone, as a seperate vob file, this same image is displayed much narrower.
it's splitter/player/renderer dependent I guess.

I gave up on your test pattern coz it's SD, and needs to be upscaled....kinda ruins the point of checking how the chroma looks.

anyway here it is, in spline36 upscale to 1280*768, and zoomed at 400% :

http://www.image-load.eu/out.php/t142192_01.png (http://www.image-load.eu/out.php/i142192_01.png)

[leeperry]

@tritical : Haruhiko has implemented a new chroma upsampling algorithm in ffdshow :
http://forum.doom9.org/showpost.php?p=1243154&postcount=6366

the results look really good :
http://forum.doom9.org/showpost.php?p=1243161&postcount=6368

your input would be much appreciated :thanks:

[tritical]

Nice that it is 20% faster, but I don't see how the extra bit depth during the yv12->yuy2->yv24 conversions, using linear interpolation, could make any noticeable difference. During yv12->yuy2 you do 75/25 and 25/75 averaging. The most error you could incur while rounding to 8-bit result is 0.5 (only possible results are: 1.0,0.75,0.5,0.25,0.0, rounding .5 to 1.0 gives the largest error). During yuy2->yv24 you keep all of the u/v values you calculated during yv12->yuy2, and do 50/50 averaging to get the other half. Now if you happened to introduce .5 error into both pixel values during yv12->yuy2 you could end up with a difference of at most 1.0 in the new u/v values calculated during yuy2->yv24. A difference of +-1 in u/v is not noticeable, and is certainly not the cause behind the differences in mark0077's image.

The explanation above is also why I'm not keen to add yv12 support to rgb3dlut... I don't think it would be at all noticeably different than if you called converttoyuy2(). Certainly it wont be faster. The only way it would be visibly different is if it used a different interpolation method during yv12->yuy2 conversion, and, if a more complex upsampling method is used, I think it would be better to have it as a separate filter (so that yuy2 frames are fed to rgb3dlut). However, I did add a cubic interpolation mode to rgb3dlut for yuy2->yv24 conversion, but before I release it I'd like to do a blind test. If anyone has samples that would really show the difference between chroma upsampling algorithms please post them :thanks:.

[leeperry]

A difference of +-1 in u/v is not noticeable, and is certainly not the cause behind the differences in mark0077's image.
it would appear that what you see in mark0077's compares is mostly due to bogus upscaling...so the results are to be taken with a grain of salt, as he has said himself here :
http://forum.doom9.org/showpost.php?p=1243013&postcount=156
If anyone has samples that would really show the difference between chroma upsampling algorithms please post them
it only showns on extreme test patterns, such as these red rolling end credits :
http://rapidshare.com/files/122925763/Bronz_s.mkv.html

if you zoom *a lot*, you can see that Haruhiko's new algorithm looks slightly better than ddcc :
http://forum.doom9.org/showpost.php?p=1243161&postcount=6368

but ddcc is already a major improvement over ConvertToRGB32() when zoomed....still it doesn't really show in real world practice :
http://forum.doom9.org/showpost.php?p=1242783&postcount=149

[tritical]

Well, on second thought I don't want to take the time for the test. Everyone can do their own tests and make up their own mind. ddcc v1.7 is on my site, changes:

+ added cubic interpolation option to rgb3dlut
+ added adobe 1998 chromaticity and linear gamma presets to ddcc
+ added yv12toyuy2 filter

[yesgrey]

tritical,
Once again...
:thanks:

[leeperry]

ddcc v1.7 is on my site, changes:

+ added cubic interpolation option to rgb3dlut
+ added adobe 1998 chromaticity and linear gamma presets to ddcc
+ added yv12toyuy2 filter

looking good, thanks!

so I ran a quick test on my o/c Q6600 :

yv12toyuy2(threads=4)
=750 fps

yv12toyuy2(itype=1,threads=4)
=1400 fps

ConvertToYUY2()
=1400 fps

is "itype 1" the same exact thing as what ConvertToYUY2() does?

and w/ rg3dlut() :

yv12toyuy2(itype=1,threads=4)
rgb3dlut(lutfile="Y:\BT709_0-255.txt",itype=1,threads=4)
=550 fps

yv12toyuy2(itype=1,threads=4)
rgb3dlut(lutfile="Y:\BT709_0-255.txt",itype=2,threads=4)
=480 fps

the speed drop is hardly noticeable, my full script(w/ LSF+Grainfactory3) falls from 62 to 61.5 fps...I'll be doing visual comparisons tomorrow :p

[madshi]

Dear gamut experts. Unfortunately I'm not an expert in this area at all. But I've one question:

The Lumagen Radiance video processor offers gamut correction which seems to be comparable to what tritical's solution does. But there seems to be a limitation of the Radiance, which is described here:

http://www.avsforum.com/avs-vb/showpost.php?p=15033893&postcount=3101
http://www.avsforum.com/avs-vb/showpost.php?p=15045760&postcount=3111
http://www.avsforum.com/avs-vb/showpost.php?p=15126353&postcount=3130

Does the tritical gamut solution have the same limitation? If so, could it be improved to beat the Radiance? :D

[yesgrey]

Does the tritical gamut solution have the same limitation? If so, could it be improved to beat the Radiance? :D
Yes, currently it does, and probably it will remain like that.
Since I am creating a little program to create the 3D LUT files all the future developments should be done in this program, that will create the 3D LUT files to use with rgb3dlut. These are our (tritical and I) plans.
The final goal would be to create a color correction application with complete customization of the output to fit all display irregularities that we can measure. Just to name a few: gamma correction without any predefined function (all data points you measure from your display), and for each color channel; chromatic adaptation to several white points (color temperatures), because a display cannot maintain the same white point from 0 to 100 IRE, and usually the higher on/off contrast is not at 6500k. Gammut correction at several levels, not only at one point as other solutions available, with better off-gammut colors handling. Etc...
With the 3D LUT, the main problem of adding all these corrections is solved: speed. It will always be the same speed, because it will be just mapping values. Now it will be just coding the algorythms for performing all the corrections we want.;)
Just a final note: this will be free. The program will be released under GNU GPL.

[madshi]

Yes, currently it does, and probably it will remain like that.
Since I am creating a little program to create the 3D LUT files all the future developments should be done in this program, that will create the 3D LUT files to use with rgb3dlut. These are our (tritical and I) plans.
The final goal would be to create the "ultimate" color correction application, a complete customization of the output to fit all display irregularities that we can measure. Just to name a few: gamma correction without any predefined function (all data points you measure from your display), and for each color channel; chromatic adaptation to several white points (color temperatures), because a display cannot maintain the same white point from 0 to 100 IRE, and usually the higher on/off contrast is not at 6500k. Gammut correction at several levels, not only at one point as other solutions available, with better off-gammut colors handling. Etc...
With the 3D LUT, the main problem of adding all these corrections is solved: speed. It will always be the same speed, because it will be just mapping values. Now it will be just coding the algorythms for performing all the corrections we want.;)
Just a final note: this will be free. The program will be released under GNU GPL.
Well, that sounds *really* good. I think the biggest problem may be on how to make all this potential functionality available in a way which is easy and intuitive to use? Ideally, I guess, your little program would contain a calibration "assistant" which would guide the consumer through a set of test screens (contained in and displayed by your program) and then ask for measurements of each of those test screens and then automatically do all necessary calculations based on those measurement results? If implemented in such a way, I guess even a calibration dummy (like me) would be able to get near to perfection, as long as good measurement hardware is used? Would that be "goodbye" to hiring ISF calibrators?

:)

[yesgrey]

Ideally, I guess, your little program would contain a calibration "assistant" which would guide the consumer through a set of test screens...
Well, I really haven't thought about it. Currently, my idea is just a simple console application (like eac3to;)) in which the user inputs a file with all the data, and the program outputs a 3D LUT file accordingly. Your idea is very good, and maybe it could end to something like that, but I don't know if I will have the time and the skills for all of that... Let's just see how the things progress... Currently there are already some free software for measuring and testing the displays (HCFR, for example), maybe we can find a way to use it for collecting the data needed for creating the 3D LUT...

Would that be "goodbye" to hiring ISF calibrators?
To someone with a HTPC, yes. Maybe it's better keeping my identity secret...:D

[madshi]

Well, I really haven't thought about it. Currently, my idea is just a simple console application (like eac3to;)) in which the user inputs a file with all the data, and the program outputs a 3D LUT file accordingly. Your idea is very good, and maybe it could end to something like that, but I don't know if I will have the time and the skills for all of that... Let's just see how the things progress... Currently there are already some free software for measuring and testing the displays (HCFR, for example), maybe we can find a way to use it for collecting the data needed for creating the 3D LUT...
I demand a perfect solution. NOW!!

Just joking. I was a little day dreaming, of course... :)

If you provided a command line tool which does all the dirty work which is necessary to realize all the fancy features you mentioned, then that would a very awesome first step.

But it would be great, if your tool just needed a specific set of measurements and would do all the necessary calculations itself. Finding a way to provide those measurements comfortably should be easy enough to add for other people then (e.g. the HCFR guys). I'm just hoping that your tool won't require users to be calibration experts. Well, because I'm not... :o

To someone with a HTPC, yes. Maybe it's better keeping my identity secret...:D
:D

[yesgrey]

I'm just hoping that your tool won't require users to be calibration experts. Well, because I'm not...
It won't.:)

[madshi]

It won't.:)
Great! You might be responsible for holding me in the HTPC camp then. Was already thinking about going external media player. Now I have to reconsider... :rolleyes:

[leeperry]

To someone with a HTPC, yes. Maybe it's better keeping my identity secret
not quite, we're still a far cry from real ISF calibration.
not because of their software tools(Color.HCFR has actually more features than ColorFacts, like the saturation measurements which ColorFacts doesn't do at all)....but the real difference is the sensor.

home users will have an Eye One Display 2 colorimeter from X-Rite at best, which is as accurate as the Eye One Pro in *most* cases :
http://www.avsforum.com/avs-vb/showthread.php?p=9495885#post9495885 (oops pix have gone AWOL...)

it's almost as accurate as the i1pro on DLP projectors, but on plasma or SXRD...it's a whole different story.

ISF ppl use Minolta spectrophotometers, that are recalibrated every 6 months(mastering houses get their CRT's recalibrated on a weekly basis w/ these professional tools).

http://www.konicaminolta.com/instruments/products/display/index.html

when an i1d2 gives a ΔE of 3, god knows if it wouldn't be 1 or 15 w/ a true Minolta spectrophotometer...it'd be like having a SP-A800B pj, and calibrate it w/ a spyder 2 :D

[yesgrey]

You might be responsible for holding me in the HTPC camp then.
It's funny, because you, with eac3to, are also responsible for holding a lot of people in the HTPC camp...;)

[leeperry]

so here I am w/ comparisons :)

1)
colorYUV(levels="tv->pc")
yv12toyuy2(itype=1,threads=4)
rgb3dlut(lutfile="Y:\BT709_16-235.txt",itype=2,threads=4)

2)
colorYUV(levels="tv->pc")
yv12toyuy2(itype=2,threads=4)
rgb3dlut(lutfile="Y:\BT709_16-235.txt",itype=2,threads=4)

3)
colorYUV(levels="tv->pc")
convertToYUY2()
rgb3dlut(lutfile="Y:\BT709_16-235.txt",itype=2,threads=4)

4)
colorYUV(levels="tv->pc")
convertToYUY2()
rgb3dlut(lutfile="Y:\BT709_16-235.txt",itype=1,threads=4)

5)
colorYUV(levels="tv->pc")
+ ffdshow RGB32 709 full range

6)
colorYUV(levels="tv->pc")
+ ffdshow RGB32HQ 709 full range

7)
colorYUV(levels="tv->pc")
+ ffdshow RGB32HQ 709 full range / new test build from Haruhiko

http://www.image-load.eu/out.php/t142371_1.png (http://www.image-load.eu/out.php/i142371_1.png)http://www.image-load.eu/out.php/t142372_2.png (http://www.image-load.eu/out.php/i142372_2.png)http://www.image-load.eu/out.php/t142373_3.png (http://www.image-load.eu/out.php/i142373_3.png)
http://www.image-load.eu/out.php/t142374_4.png (http://www.image-load.eu/out.php/i142374_4.png)http://www.image-load.eu/out.php/t142375_5.png (http://www.image-load.eu/out.php/i142375_5.png)http://www.image-load.eu/out.php/t142376_6.png (http://www.image-load.eu/out.php/i142376_6.png)
http://www.image-load.eu/out.php/t142377_7.png (http://www.image-load.eu/out.php/i142377_7.png)

first comparison :

http://www.image-load.eu/out.php/t142378_orig.png (http://www.image-load.eu/out.php/i142378_orig.png)

zoomed at 400% :

http://www.image-load.eu/out.php/t142379_origx4.png (http://www.image-load.eu/out.php/i142379_origx4.png)

second comparison :

http://www.image-load.eu/out.php/t142381_2orig.png (http://www.image-load.eu/out.php/i142381_2orig.png)

zoomed at 400% :

http://www.image-load.eu/out.php/t142382_2origx4.png (http://www.image-load.eu/out.php/i142382_2origx4.png)

after careful examination, it seems to me that the sharpest and best looking results are w/ Haruhiko's new algorithm...any chance adding it in rgb3dlut please ? plus it works in 10 bits or so, so going yv12>rgb32 in more than 8 bits would avoid sloppy roundings and increase the LUT accuracy. Haruhiko talked about it here :
http://forum.doom9.org/showpost.php?p=1243635&postcount=6402
this is really über-nitpicking, though...feel free to throw rocks at me if you like http://forum-images.hardware.fr/images/perso/antp.gif

[yesgrey]

leeperry, how have you peerformed your tests? Are you using any sharpenning? My tests show differences, but not so big as yours... I think the comparison must be done at the most raw level...

Here are my test results...
ffdshow new algorythm:
9365
yv12toyuy2 and rgb3dlut with itype=2, b=0, c=0.5
9365

Comparing both images, the ffdshow new algorythm has a slightly less sharp image, but the colors are more even (Look at the capital J's).
There is also a strange thing... if you compare both images switching from one to another (I open both with Paint, and then switch using Alt+Tab), you can notice a little horizontal shift between them.
Maybe the slightly loss of precision due to the yv12->yv24 being performed in two steps with yv12toyuy2+rgb3dlut would be the responsible for that?

[leeperry]

nope, no sharpening.

remoulade(divx7) decoding > colorYUV() in HR/full range 32bits and that's it.

you can't trust VMR/EVR as they do postprocessing, HR does not.

I can't see your screenshots, maybe you could put them on imagebam?

anyway, we both agree that using Haruhiko's new code and doing the LUT thingie in 10 bits w/o any in-between 8 bits conversion could potentially yield more pleasing results...let's wait to know what tritical thinks about all this, and whether this would be technically achievable http://forum.slysoft.com/images/smilies/agreed.gif

[leeperry]

the funny thing is that if you save the 7 original screenshots to your PC, then quickly pass through them w/ the windows picture viewer....they all smear a bit, except the seventh.

in RGB32HQ stock, the background is R3/G1/B1 like all the other screenshots...but w/ the new experimental yv12 algorithm it's R2/G0/B0

this time I haven't messed up w/ the LUT's, and I used the exact same settings between stock ffdshow and the new Haruhiko's version...I only updated ffdshow :o

[FoLLgoTT]

in RGB32HQ stock, the background is R3/G1/B1 like all the other screenshots...but w/ the new experimental yv12 algorithm it's R2/G0/B0

The overall saturation is too high. Is this a basic problem of this algorithm or just a matter of implementation?

[tritical]

There is also a strange thing... if you compare both images switching from one to another (I open both with Paint, and then switch using Alt+Tab), you can notice a little horizontal shift between them.
Maybe the slightly loss of precision due to the yv12->yv24 being performed in two steps with yv12toyuy2+rgb3dlut would be the responsible for that?
It's because the new ffdshow algorithm assumes centered chroma placement in yuy2, and not left aligned (which is the mpeg2 standard, and what rgb3dlut uses). As I explained before, the extra bits in the conversion during the 4:2:0 -> 4:4:4 interpolation steps will be absolutely unnoticable (there are bigger differences between yv12toyuy2(itype=1) and converttoyuy2(), try to spot those :)). Now a change in assumed chroma position is a different matter. I have added chroma placement options to rgb3dlut/yv12toyuy2, which allow the user to select between all possible scenarios. In yuy2 there are only 2, centered or left aligned. In yv12 there are a few more, depending on interlaced vs non-interlaced and non-standard ways the conversion could be performed... which has been discussed in the past.

[Jeremy Duncan]

It's because the new ffdshow algorithm assumes centered chroma placement in yuy2, and not left aligned (which is the mpeg2 standard, and what rgb3dlut uses). As I explained before, the extra bits in the conversion during the 4:2:0 -> 4:4:4 interpolation steps will be absolutely unnoticable (there are bigger differences between yv12toyuy2(itype=1) and converttoyuy2(), try to spot those :)). Now a change in assumed chroma position is a different matter. I have added chroma placement options to rgb3dlut/yv12toyuy2, which allow the user to select between all possible scenarios. In yuy2 there are only 2, centered or left aligned. In yv12 there are a few more, depending on interlaced vs non-interlaced and non-standard ways the conversion could be performed... which has been discussed in the past.

- "left aligned (chroma) is the mpeg2 standard"
- "ffdshow uses centered aligned (chroma)"
- "you can spot a change in the assumed chroma"

My question to you, Tritical.
- Mpeg2 assumes left, and I use yv12.
- ffdshow output changes yv12 to rgb32 with (16-235 levels).
- Will the picture be degraded? because the yv12 being changed into rgb 32 is treated like it's centered chroma when in fact it's left chroma?

[tritical]

My question to you, Tritical.
- Mpeg2 assumes left, and I use yv12.
- ffdshow output changes yv12 to rgb32 with (16-235 levels).
- Will the picture be degraded? because the yv12 being changed into rgb 32 is treated like it's centered chroma when in fact it's left chroma?

The thing is, just because the standard specifies left aligned 4:2:2 doesn't mean all encoders do that. Heck, avisynth's rgb->yuy2 conversion averages every two pixels... resulting in centered chroma (yet its yuy2->rgb conversion assumes left aligned, and averages to create the right pixel value, which will result in visible shifting if you chain enough conversions together). Xvid's old color conversion routines (circa 2003-2004, I have no idea if they are the same now) operated based on centered chroma. I assume there are commercial encoders that operate that way... so in these cases assuming centered chroma in the yuy2->rgb conversion will look better. That said... if you are watching video, and not zooming in on still frames, you probably wouldn't be able to notice the difference.

Also, when I mentioned yuy2 vs yv12 and number of positioning scenarios... for yuy2 I was talking only about horizontal positioning (what matters in the yuy2<->rgb conversion) and in yv12 I was talking only about vertical positioning (what matters in the yuy2<->yv12 conversions). Generally, progressive yuy2->yv12 vertical positioning is consistent. It's the interlaced yuy2->yv12 conversion that is more likely to result in non-standard chroma placement (for example: separating fields and then doing a progressive yv12->yuy2 conversion on each field results in non-standard placement). The nice thing about mpeg2 standard 4:2:0 interlaced vertical chroma positioning, is that if the content is actually progressive, but is downsampled using interlaced conversion, then you can use normal progressive upsampling on it because the two positionings are actually the same (the difference is how the values are created). If the positioning is non-standard though, then the progressive conversion's assumptions wont match up correctly.

[madshi]

It's because the new ffdshow algorithm assumes centered chroma placement in yuy2, and not left aligned (which is the mpeg2 standard, and what rgb3dlut uses).
Do you happen to know what the h264 and VC-1 specifications say about chroma placement?

[tritical]

h.261,h.263,mpeg1 -> centered
mpeg2,mpeg4,h.264 -> left

Don't know about vc-1 for sure, but I would guess left.

[madshi]

Pretty interesting - thanks!

[leeperry]

I have added chroma placement options to rgb3dlut/yv12toyuy2
in ddcc 1.7? you mean itype? which one is left and which one is centered please?
there are bigger differences between yv12toyuy2(itype=1) and converttoyuy2(), try to spot those :)
haha, couldn't tell :D
I'm using yv12toyuy2(itype=1)/rgb3dlut(itype=2) and it looks good to me.
h.261,h.263,mpeg1 -> centered
mpeg2,mpeg4,h.264 -> left

Don't know about vc-1 for sure, but I would guess left.
ok, I'll try to make automatic profiles in ffdshow then :p
if you are watching video, and not zooming in on still frames, you probably wouldn't be able to notice the difference.
indeed, this is major nitpicking :devil:
anyway what I *can* see is that yesgrey3's REC601/709 matrixes are more accurate than ffdshow.
so we finally have very accurate decoding and conversion, this is too awesome :thanks:

[leeperry]

I've got some h264/DTS MKV samples that make yv12toyuy2() crash ffdshow instantly(using HMS/divx7 decoder/sonic audio 4.2), whatever in type 1 or 2...itype 0 works fine, and so does ConvertToYUY2() :confused:

here's one of them :
http://www.megaupload.com/?d=4A4TRTNB

btw, look at the native gamut of the Epson TW5000 :
http://www.homecinema-fr.com/BE/TW5000/usine-cie.jpg

it's got a full CMS, BenQ/Epson are now offering it in their projectors...got to be a good sign ;)

[tritical]

in ddcc 1.7? you mean itype? which one is left and which one is centered please?

It's in 1.8, haven't released it yet. Still need to test it a little more, and need to check out your crashing report.

[yesgrey]

Finally it's ready the first version of cr3dlut, a program for creating a 3D LUT file to use with tritical's rgb3dlut. This first version is partially based in tritical's ddcc code, so the results should be very similar.
You can get both the program and source code at my web page. Here is the link:
http://yesgrey3.totalh.com/

It's not (yet) very user friendly, but I will also improve the user-friendliness at the same time I will improve the program. For that, it will be very important your comments and suggestions.;)

A special thanks to tritical for his rgb3dlut. Now I can implement and test several ideas I have about YCbCr->RGB conversion and color correction.
I would also like to thank leeperry for promptly test our implementations, and for helping me correcting some typos in the readme file.

[leeperry]

thanks yesgrey! I was busy doing stuff today, but now that I've RTFM several times I'll look into it tomorrow http://forum-images.hardware.fr/images/perso/d4buff.gif

[Mug Funky]

heywow!

3d luts in avisynth, 1 step closer to being quite practical :)

i'm just R'ingTFM now.

just thinking, probably the best balance of performance versus complexity might be in the way programs like lustre, scratch, resolve etc do it - a 16x16x16 LUT (in a text file) with processing and interpolation between these "cube points" happening on the GPU. i'm pretty confident most modern GPU's can handle it, and the advantage is you can go ape with the precision and it all happens at the output stage.

right now i'm just trying to figure out how to port the Arri DCI LUT for log-to-print emulation, so i can preview film scans in avisynth as they'd appear in a cinema.

[edit]

would it be advantageous to make the 3d lut file format such that it looks meaningful when loaded raw into an image editor? that way you could create a series of adjustments in gimp or photoshop or whatever, use them on a "flat" lut file, then save the result as a raw image that can be used as a lut in avisynth. does that make sense? this way i could drop the flat file into scratch, apply the arri lut, then render to a new pic that i can use.

[canuckerfan]

i feel a little lost in all this but would using yv12toyuy2() be advantageous in any way over avisynth's native converttoyuy2()?

[tritical]

@All
Put up ddcc v1.8 on my website... only change was adding 'cplace' parameter to yv12toyuy2 and rgb3dlut for specifying chroma placement.

@leeperry
The crash happened because I coded most of yv12toyuy2 for mod 4 height, but didn't put in the necessary error checking. Since itype=0 with progressive upsampling was the only combination that worked with mod 2 height, I decided to require all input be mod 4.

just thinking, probably the best balance of performance versus complexity might be in the way programs like lustre, scratch, resolve etc do it - a 16x16x16 LUT (in a text file) with processing and interpolation between these "cube points" happening on the GPU. i'm pretty confident most modern GPU's can handle it, and the advantage is you can go ape with the precision and it all happens at the output stage.
Well, I have no plans to port it to the gpu, but someone else might. In terms of programming for the cpu, 16x16x16 lut with interpolation would be more complicated. If the interpolation was something simple it might be faster, but the current method (with full 48MB table) is pretty quick on most recent computers. Working with 8-bit in -> 8-bit out you certainly can't get better precision than with the full table (precision of the mapping is limited by the program computing the table). Personally, I think it would be easier to write a separate program that converts other lut formats into the format rgb3dlut currently uses, and since that would be offline you could make it as complex as you want (interpolation methods, etc...). Plus, that way rgb3dlut is never the limiting the factor, in terms of interpolation methods offered for smaller luts, etc...

would it be advantageous to make the 3d lut file format such that it looks meaningful when loaded raw into an image editor? that way you could create a series of adjustments in gimp or photoshop or whatever, use them on a "flat" lut file, then save the result as a raw image that can be used as a lut in avisynth. does that make sense? this way i could drop the flat file into scratch, apply the arri lut, then render to a new pic that i can use.
What format are you thinking? Again, I think it would be easier just to have a separate program that converts from one format to another and leave rgb3dlut as it is.

i feel a little lost in all this but would using yv12toyuy2() be advantageous in any way over avisynth's native converttoyuy2()?
That question is like asking if bicubicresize() would be advantageous over bilinearresize(). Compared to converttoyuy2(), yv12toyuy2() simply offers more choices for interpolation function, and more choices for chroma placement. Are those differences large? Not really, people aren't that sensitive to chroma. As a test, I created an image in paint, and converted it to yv12 using converttoyv12(matrix="Rec709",interlaced=false). Then I converted it back to rgb using a number of methods: cpic-center.png (http://bengal.missouri.edu/~kes25c/cpic-center.png). For the images that are named "xxx,yyy", 'xxx' specifies the interpolation method for yv12->yuy2, and 'yyy' specifies the interpolation method for yuy2->yv24 (dup=duplicate,lin=linear,cub=cubic). Those images were converted using yv12toyuy2/rgb3dlut. Since avisynth's rgb->yuy2 conversion averages every two pixels, I used centered chroma placement in rgb3dlut. Here is the result using left aligned chroma placement for the yv12toyuy2/rgb3dlut images: cpic-left.png (http://bengal.missouri.edu/~kes25c/cpic-left.png). Are there differences among the conversions? Yep. Are they that big between linear and cubic? Not really.

[cyberbeing]

I'm finding ddcc 1.8 nearly 30-50% slower then ddcc 1.7 which makes it unusable for me. This is a major problem considering that the 30-50% slower causes me to be unable to view some 720p video in real-time on my AMD X2 computer with yv12toyuy2, rgb3dlut, and reclock. If just adding the chroma placement option is the only thing changed/fixed and is the entire cause of this slowdown, I'll just continue using 1.7.

Below are the options I use:

yv12toyuy2(itype=2, interlaced=false, threads=2, b=0.33, c=0.33)
rgb3dlut(itype=2, lutfile="C:\Program Files\AviSynth 2.5\bt709lut.txt", threads=2, b=0.33, c=0.33)

Edit: Oh and thank you yesgrey3 for the cr3dlut app, initial impressions is it seems to work as designed.

[canuckerfan]

thanks for the explanation, tritical. the visuals helped:)

[leeperry]

@leeperry
The crash happened because I coded most of yv12toyuy2 for mod 4 height, but didn't put in the necessary error checking. Since itype=0 with progressive upsampling was the only combination that worked with mod 2 height, I decided to require all input be mod 4.
any chance you could make it fall back to itype=0 if it's not mod4? but well I'm not sure I could even see a diff between the itypes :o

anyway, I'll look into the new version! :thanks:

[tritical]

@cyberbeing
Is this faster: [removed]?

[madshi]

Finally it's ready the first version of cr3dlut, a program for creating a 3D LUT file to use with tritical's rgb3dlut. This first version is partially based in tritical's ddcc code, so the results should be very similar.
You can get both the program and source code at my web page. Here is the link:
http://yesgrey3.totalh.com/

It's not (yet) very user friendly, but I will also improve the user-friendliness at the same time I will improve the program. For that, it will be very important your comments and suggestions.;)
Thanks!

Some comments:

(1) There are some "*/" in the readme which probably aren't supposed to be there?
(2) The readme is very technical. I didn't understand half of it... :) Would it make sense to split the readme into two separate files: One for technical gurus, where all the funny details are explained. And one for every user where there are more explanations and less technical terms? I can't help much with double checking the technical stuff. But I could easily tell you which parts are difficult to understand for a noob like me. Some of the current text of the readme is only useful for programmers, but not for users, e.g. "The table is created such that the offset into the table" is not useful for users, but might be useful for programmers, I think. Such information could also be moved to a separate file.
(3) For some parameters you're listing a number of ITU/SMTP/whatever specs. That's ok, but it doesn't help a noob like me at all. I'd need an explanation what these specs are actually usually used for. E.g. for "Source_primaries" both options 1 and 3 contain the word "NTSC", so if I wanted to use the default source primaries for DVD NTSC discs, I wouldn't know whether to use 1 or 3. Also the word "ATSC" doesn't appear, anywhere, so I wouldn't know which source primaries to use for ATSC, either. I think for each of these options the first line should explain what they are usually used for, and then maybe the ITU/SMPTE spec listings under that. Personally, I couldn't care less about ITU/SMPTE. If you ask me, I'd move the ITU/SMTPE numbers to the "technical readme" and remove them from the normal readme. I think for each option which requires different values depending on the source type, each of the following source types should be contained somewhere in one of the options: Blu-Ray/HD DVD, NTSC DVD, PAL DVD, NTSC SD broadcasts, ATSC HD broadcasts, PAL SD broadcasts, PAL HD broadcasts. So that we noob users know exactly which option is the right one to use.
(4) What does "GBR" stand for? I'd simply remove that and only write "RGB Input, no YCbCr->RGB conversion" there instead.
(5) I'd rename "YCbCr_Full_range" to "YCbCr_Input_Full_range" to make it clear that this is for *input*. It's clear enough if you think about it (after all output of 3dlut is always RGB, so YCbCr can only be the input and not the output), but adding "input" to the parameter name means you don't even need to use intelligent logic to find out that input is meant and not output.
(6) "RGB_BW" should be renamed to "RGB_Output_BW".
(7) Would it make sense to add an "RGB_Input_BW" option?
(8) "Chromatic_adaptation" definitely needs an explanation. I've no idea what this is good for and which value it should be set to under which circumstances.
(9) What should "Display_primaries" be set to for people who don't have this information about their display?
(10) Is there a direct correlation between different source types (e.g. PAL DVD, NTSC DVD, Blu-Ray, ...) and "Source_gamma"?
(11) What should "Display_gamma" be set to for people who don't have this information about their display?
(12) An explanation would be helpful about whether YCbCr and RGB data are usually linear or gamma corrected. E.g. are compressed sources usually gamma corrected (probably yes)? And if you transport YCbCr via HDMI, is that also gamma corrected or linear? And if you transport RGB via HDMI, is that also gamma corrected or linear? How about if you use HDMI 1.3 DeepColor. Is that still gamma corrected or linear?
(13) I'd like to have an option to output 16bit RGB instead of 8bit... ;)

(Of course my questions are not meant to be answered by you here in the forum. The intent of my questions is that they should be answered by the readme.)

[cyberbeing]

It seems like I underestimated before how slow the initial version of 1.8 was. Instead of 10-20% it was actually nearer to 30-50% slower then 1.7 when I retested and actually calculated the percentage instead of guessed. The new version you posted is much better but still slower then 1.7 by about 5-15%.

CPU graphs from left to right, |ddcc 1.7|original ddcc 1.8|new ddcc 1.8|
http://img17.imageshack.us/img17/6731/17st7.png http://img10.imageshack.us/img10/6216/old18iu9.png http://img15.imageshack.us/img15/9736/new18ew8.png

[yesgrey]

But I could easily tell you which parts are difficult to understand for a noob like me.
madshi,
Thank you very much for your excellent post! I need feedback like this, because it's the only way of improving things.
You should consider the actual readme as the technical readme.;)
I did not want to delay much more the release of the program, so I only put the more technical stuff. For helping the newbies, I've created 3 typical input files, for Blu-ray, dvd-pal and dvd-ntsc, but I know the instructions must be a lot more detailed and simpler.
As you know very well, only when people start using our software we can see what's good, what's bad, what's useful, what's useless...

(5) I'd rename "YCbCr_Full_range" to "YCbCr_Input_Full_range"
(6) "RGB_BW" should be renamed to "RGB_Output_BW".

I'm not so sure about this. I was thinking that maybe could be a good idea of allowing the 3D LUT to work both ways:
YCbCr->RGB or RGB->YCbCr.
tritical, what do you think about it? Some people are reporting slightly better results when performing YCbCr->RGB with the 3D LUT, maybe the RGB->YCbCr could also be more accurate with the 3D LUT?
I think that would be a good idea starting a new thread about rgb3dlut. Now, there is a new option available for the Avisynth usage (using 3D LUTs), and the current thread name is not very meaningfull about it; some potential users could be missing it...
If you agree with using the 3dlut in both ways, we have two optios:
-rename rgb3dlut to a more generic name
-keep rgb3dlut as it is and create ycbcr3dlut or yuv3dlut (i prefer the former because is the correct designation), etc. As you wish.

[leeperry]

It seems like I underestimated before how slow the initial version of 1.8 was.

on an o/c Q6600 :

ddcc 1.7

yv12toyuy2(itype=1,threads=4)
=1400 fps

rgb3dlut(lutfile="C:\BT709_16-235.txt",itype=2,threads=4)
=622 fps

ddcc 1.8

yv12toyuy2(itype=1,threads=4)
=1450 fps

rgb3dlut(lutfile="C:\BT709_16-235.txt",itype=2,threads=4)
=320 fps

[madshi]

You should consider the actual readme as the technical readme.;)
Ok, that readme is probably very good then... :)

I'm not so sure about this. I was thinking that maybe could be a good idea of allowing the 3D LUT to work both ways:
YCbCr->RGB or RGB->YCbCr.
tritical, what do you think about it? Some people are reporting slightly better results when performing YCbCr->RGB with the 3D LUT, maybe the RGB->YCbCr could also be more accurate with the 3D LUT?
I think that would be a good idea starting a new thread about rgb3dlut. Now, there is a new option available for the Avisynth usage (using 3D LUTs), and the current thread name is not very meaningfull about it; some potential users could be missing it...
If you agree with using the 3dlut in both ways, we have two optios:
-rename rgb3dlut to a more generic name
-keep rgb3dlut as it is and create ycbcr3dlut or yuv3dlut (i prefer the former because is the correct designation), etc. As you wish.
Well, if you go that way then you may also want to support YCbCr -> YCbCr. Also you may want to support RGB computer levels -> RGB video levels. In any case, if the options are not clear about whether they're supposed to affect input or output, there can be all kinds of misunderstandings. Just think about the ffdshow RGB controls which were backwards (are finally fixed) and about Haali's RGB levels control, which is also backwards. It's all because the options don't clearly say whether they're meant to control input or output levels. If you want to do RGB -> RGB then you probably have to offer RGB_BW options for both input and output. And if you want to support YCbCr -> YCbCr you probably have to offer full_range options for both input and output. So IMHO the options should be clearly separated for input and output.

[yesgrey]

So IMHO the options should be clearly separated for input and output.
Yes, you're right. I will change it for the next version.
Thanks

[tritical]

v1.9 should fix the speed issues... it was the compiler sucking at inlining some functions. Also, when I was explaining yv12toyuy2 vs converttoyuy2 I forgot to mention that converttoyuy2 is significantly faster (it has mmx/isse versions, whereas yv12toyuy2 is just written in c). So unless you really need the extra functionality of yv12toyuy2 I would recommend using converttoyuy2. I wrote yv12toyuy2 mainly for comparison purposes.

@yesgrey3
Adding rgb->yuy2, yuy2->yuy2, and yv12->yv12 support is a good idea. I would probably just rename rgb3dlut to 3dlut, and move it to its own dll at that point.

[leeperry]

v1.9 should fix the speed issues... it was the compiler sucking at inlining some functions.
indeed! on an o/c Q6600 :

ddcc 1.7

yv12toyuy2(itype=1,threads=4)
=1400 fps

rgb3dlut(lutfile="C:\BT709_16-235.txt",itype=2,threads=4)
=622 fps

ddcc 1.9

yv12toyuy2(itype=1,threads=4)
=1410 fps

rgb3dlut(lutfile="C:\BT709_16-235.txt",itype=2,threads=4)
=700 fps

OK now yv12toyuy2() doesn't crash if it's not mod4, it simply gives an error msg.
is there any potentially visible improvement using yv12toyuy2(itype=0) over ConvertToYUY2() ? or if you don't mind forcing it to use 0 if it's not mod4 :thanks:

[yesgrey]

Adding yuy2->yuy2, and yv12->yv12
What's the idea of this, changing the luma matrix coefficients and/or levels?
Please tell me when you are thinking in adding that, so I can set my priorities for cr3dlut development...;)
I would probably just rename rgb3dlut to 3dlut, and move it to its own dll at that point.
Yes, and perhaps it would also be the time for starting a new thread about it...
Can a function name start with a number? I think that would be the best name, but i thought that it was not possible...

[yesgrey]

(13) I'd like to have an option to output 16bit RGB instead of 8bit... ;)
This would be very simple to add, but completelly useless for now... You will need a video renderer that supports 16bit per component, and Avisynth also only supports 8bit per component. I've read something about Avisynth 3.0 supporting RGB45 (15bit per component), but I don't know if it's still being developed.
I also need to know the format of the 3D LUT. I could simply use the current format just changing the offset considering 16bit instead of 8bit, but I don't know if that would be the desired format...
I think you will have to wait a little more. For cr3dlut, it will be less than a day of work, but all the other stuff that we need could take a little bit longer...:(

[leeperry]

ah...yv12toyuy2(itype=0) used to work w/ non-mod4, but now it also gives an error msg....back to ConvertToYUY2() :o

[madshi]

This would be very simple to add, but completelly useless for now... You will need a video renderer that supports 16bit per component, and Avisynth also only supports 8bit per component. I've read something about Avisynth 3.0 supporting RGB45 (15bit per component), but I don't know if it's still being developed.
I also need to know the format of the 3D LUT. I could simply use the current format just changing the offset considering 16bit instead of 8bit, but I don't know if that would be the desired format...
I think you will have to wait a little more. For cr3dlut, it will be less than a day of work, but all the other stuff that we need could take a little bit longer...:(
Here comes my suggestion:

The 3dlut files *have* to get a header. The header should contain at least the following information:

(1) signature, e.g. "3dlut"
(2) header size
(3) file format version number
(4) program which created this file (e.g. "cr3dlut")
(5) version of the program which created this file (Windows version information "a.b.c.d" = 4 words)
(6) input color space (RGB or YUY2 something else)
(7) output color space (YUY2 or YCbCr or something else)
(8) input bitdepth (8bit or 16bit) - even if 16bit input doesn't seem to make sense right now
(9) output bitdepth (8bit or 16bit)
(10) detailed list of *every single* parameter used to create the file
(11) maybe some reserved fields for future use

The file format should never break. In the worst case a new "file format version number" could be used, e.g. if you added support for display primaries for different luminance levels or things like that. If you can create a file format which is already fit for all future extensions you plan to do that would be awesome, of course...

Let's just imagine someone decided to create a DirectShow filter doing gamut correction based on your 3dlut solution. That DirectShow filter should be able to find out whether a specific 3dlut file has the expected format. If it does not, the filter could just delete it and recreate it on the fly by using cr3dlut. The filter could offer the consumer a list of controls (e.g. display gamma and primaries) and further options like RGB output levels (video or computer) etc. The necessary 3dlut files could always be created on the fly before video playback is started. But such a logic would definitely require a clearly defined header for the 3dlut files. Finally, such a DirectShow filter could easily make use of 16bit RGB output, e.g. to feed a potential new Windows 7 16bit RGB renderer or to dither down the 16bit RGB 3dlut output to any desired RGB bitdepth. Saying that AviSynth doesn't support more than 8bit RGB yet and that there's no renderer for 16bit RGB yet feels a bit short sighted to me. Maybe Haali would be motivated to update his renderer to support Windows 7 16bit if there was a ready to use gamut correction solution outputting 16bit RGB? He won't add 16bit output if there's no argument for it, obviously. So *please* let's not play the chicken and egg game.

I think if you guys provided the necessary framework with exact specifications, that might increase motivation for other programmers to jump in and provide the missing pieces of the puzzle. Saying: "Maybe we will add this later" and "I don't know how the file format of 16bit output would look like" etc makes all your work feel like "it's not ready to be used by other programmers yet". That is likely to slow down adoption by other programmers...

IMHO the first thing you should do is create a file format for 3dlut. The 2nd thing to do would be to make cr3lut create files in that format. That would be the minimum needed for other programmers to jump in and make use of it, I think.

[yesgrey]

Here comes my suggestion:
The 3dlut files *have* to get a header.
That's a very good suggestion, because with all the possible options that could be added to the 3dlut it would be very hard to know which will do what... I will start working on it.
The file format should never break...
If you can create a file format which is already fit for all future extensions you plan to do that would be awesome, of course...

For a 3dlut this is easy. The output will always be the same for the same output bit depths. For example, for the current 8bit version, the output will always be an array of 3x[256,256,256] 1 byte entries. What could change is just the way the output values are computed, nothing more.
So *please* let's not play the chicken and egg game.
Ok, I will put the egg.:D
I will add 16bit and possibly other output bit depths.
Let's hope he grows in a beautiful chicken...;)

@tritical,
I will start the specification of a header for the 3dlut files, based on madshi suggestions. For the 3d lut files with 16bit output are you ok with this format:
offset: ((v<<16)+(u<<8)+y)*3 2 bytes entries
offset: ((g<<16)+(b<<8)+r)*3 2 bytes entries
At that location, the associated rgb value should be stored in b, g, r order.
Would it be better considering only 8bit and 16bit output? It would be easier to code... what about performance wise?
With all the options we are considering for rgb3dlut, maybe it would also be better start using some kind of compression with the lut files. If we need to have several in our computer it will consume some hard disk space... any suggestion?

[FoLLgoTT]

There is a 10 bit per component mode in DirectShow called MEDIASUBTYPE_A2R10G10B10 (http://msdn.microsoft.com/en-us/library/dd407253(VS.85).aspx). Is it somehow possible to use that mode with VMR9 or other Renderers?

While searching a time ago I found nearly nothing about displaying graphics with more than 8 bit per component on Windows platform.

[yesgrey]

There is a 10 bit per component mode in DirectShow called MEDIASUBTYPE_A2R10G10B10 (http://msdn.microsoft.com/en-us/library/dd407253(VS.85).aspx). Is it somehow possible to use that mode with VMR9 or other Renderers?
With VMR9 I don't think so, in the Video Mixing Renderer Subtypes (http://msdn.microsoft.com/en-us/library/dd407345(VS.85).aspx) it doesn't appear... With another renderers, I don't know.
While searching a time ago I found nearly nothing about displaying graphics with more than 8 bit per component on Windows platform.
Apparently only Windows7 will allow it.
I have read a user report in this forum that when using a 10bit lcd monitor with dvi connection the current windows version would show a 10bit per component graphic mode, but I don't know if it's true.

[madshi]

For a 3dlut this is easy. The output will always be the same for the same output bit depths. For example, for the current 8bit version, the output will always be an array of 3x[256,256,256] 1 byte entries. What could change is just the way the output values are computed, nothing more.
True for the data array. But how about the header? I would really want to have all the parameters in the header which were used to create the 3dlut data array. So if the parameter logic changes, the header might have to change, too. E.g. currently users can define only one set of primaries for the display, IIRC. Maybe some day you will allow primaries for different luminance levels. If you do that, the header may have to change, if you really want to put all the parameters in there which were used to create the 3dlut file.

Or maybe you could simply store the cr3dlut config text file in the header which was used to create the 3dlut file?

Ok, I will put the egg.:D
Great - thanks!!

I will add 16bit and possibly other output bit depths.
What other output bit depths would make sense? Maybe 8bit and 16bit is all we need? More than 16bit should be overkill. And any intermediate value between 8bit and 16bit could easily be calculated on the fly by the software (e.g. DirectShow filter) which does all the work. If I were to design a software (e.g. DirectShow filter) which "executes" gamut correction based on 3dlut I'd probably not even use 8bit 3dlut files at all. I'd only use 16bit 3dlut and then dither down to 8bit, if needed. I wouldn't really care much if that costs a few percent of performance...

[yesgrey]

Or maybe you could simply store the cr3dlut config text file in the header which was used to create the 3dlut file?
Yes, I think this would be the best. If we'll edit the 3dlut file with a text editor we could see the run parameters settings, and it will be a small overhead...
What other output bit depths would make sense?
Well, for people who doesn't want to use dithering, it would be better having it in the display's native bit depth.
More than 16bit should be overkill.
16bit is already overkill...;)

[yesgrey]

Here is the first iteration for the definition of a file format for the 3D LUT:
struct
{
char sig[6];
int size, ver, biti, bito, cci, cco;
char pname[20];
int pver, sizerp, reserv1, reserv2;
} h3dlut;

/* 3D LUT file specification:

// Header
sig - File signature, must be: '3DLUT'
size - File header size in bytes
ver - File format version number
biti - Input bit depth per component (same for all three)
bito - Output bit depth per component (same for all three)
cci - Input color coding
0 - R'G'B'
> 0 - Y'Cb'Cr' - index to luma_matrix_coeffs
cco - Output color coding
0 - R'G'B'
> 0 - Y'Cb'Cr' - index to luma_matrix_coeffs
pname - Name of the program that created the file
pver - Version of the program that created the file
sizerp - Size in bytes of the array of char with a copy of the
run parameters settings used for creating the file
reserv1 - Reserved for future usage
reserv2 - Reserved for future usage

// Parameters Settings
sizerp bytes

// Data
3*((2^biti)^3)*bito/8 bytes
This is the general formula, in reality it's only usable the biti=8 version
*/
Any suggestions/corrections are welcome.
@tritical,
If you agree, let me know so I can implement it and release the first version of the 3DLUT file format.

[cyberbeing]

Would there be any quality advantage of completely bypassing the video card LUT (i.e. leaving your 8bit display uncalibrated) and applying your calibrated gamma ramp instead through 3dlut before it does its other adjustments?

My limited understanding is, considering that most graphics cards only have a 8bit or 10bit LUT, doing the adjustment with 3dlut which is 16bit would result in less banding. Is this correct?

If so, could support for applying a calibrated gamma ramp (using something like the values exported from CalibrationTester) before the CMS compensations are done be added?

[leeperry]

Would there be any quality advantage of completely bypassing the video card LUT (i.e. leaving your 8bit display uncalibrated) and applying your calibrated gamma ramp instead through 3dlut before it does its other adjustments?

My limited understanding is, considering that most graphics cards only have a 8bit or 10bit LUT, doing the adjustment with 3dlut which is 16bit would result in less banding. Is this correct?

If so, could support for applying a calibrated gamma ramp (using something like the values exported from CalibrationTester) before the CMS compensations are done be added?
good point, even though I'm dubious about any banding improvement?!
ARGYLLCMS outputs this sort of 1D LUT for calibration :
NUMBER_OF_SETS 256
BEGIN_DATA
0.0000 0.083189 0.021229 0.023934
3.9216e-003 0.087305 0.047484 0.028051
7.8431e-003 0.091516 0.061404 0.032320
0.011765 0.095825 0.071292 0.036751
0.015686 0.10023 0.079941 0.041354
0.019608 0.10475 0.088256 0.046137
0.023529 0.10937 0.096318 0.051114
0.027451 0.11409 0.10400 0.056294
0.031373 0.11882 0.11121 0.061668
0.035294 0.12350 0.11782 0.067159
0.039216 0.12807 0.12387 0.072689
0.043137 0.13247 0.12948 0.078181
0.047059 0.13671 0.13473 0.083563
0.050980 0.14080 0.13966 0.088789
0.054902 0.14475 0.14437 0.093852
0.058824 0.14860 0.14889 0.098760
0.062745 0.15235 0.15323 0.10354
0.066667 0.15601 0.15747 0.10823
0.070588 0.15960 0.16162 0.11283
0.074510 0.16315 0.16566 0.11736
0.078431 0.16664 0.16961 0.12183
0.082353 0.17010 0.17345 0.12626
0.086275 0.17353 0.17722 0.13063
0.090196 0.17695 0.18092 0.13497
0.094118 0.18034 0.18457 0.13927
0.098039 0.18369 0.18818 0.14352
0.10196 0.18701 0.19175 0.14771
0.10588 0.19030 0.19534 0.15185
0.10980 0.19355 0.19889 0.15593
0.11373 0.19679 0.20245 0.15996
0.11765 0.20002 0.20604 0.16396
0.12157 0.20324 0.20968 0.16794
0.12549 0.20647 0.21337 0.17190
0.12941 0.20970 0.21707 0.17586
0.13333 0.21293 0.22077 0.17980
0.13725 0.21616 0.22444 0.18373
0.14118 0.21938 0.22807 0.18765
0.14510 0.22257 0.23167 0.19155
0.14902 0.22575 0.23524 0.19542
0.15294 0.22891 0.23872 0.19925
0.15686 0.23207 0.24214 0.20306
0.16078 0.23521 0.24550 0.20685
0.16471 0.23833 0.24883 0.21061
0.16863 0.24143 0.25216 0.21436
0.17255 0.24450 0.25548 0.21808
0.17647 0.24758 0.25880 0.22179
0.18039 0.25068 0.26216 0.22546
0.18431 0.25379 0.26554 0.22912
0.18824 0.25693 0.26891 0.23277
0.19216 0.26008 0.27230 0.23641
0.19608 0.26324 0.27569 0.24003
0.20000 0.26642 0.27910 0.24366
0.20392 0.26961 0.28251 0.24731
0.20784 0.27282 0.28593 0.25096
0.21176 0.27602 0.28935 0.25463
0.21569 0.27921 0.29280 0.25832
0.21961 0.28238 0.29625 0.26200
0.22353 0.28554 0.29964 0.26570
0.22745 0.28867 0.30297 0.26940
0.23137 0.29177 0.30625 0.27309
0.23529 0.29482 0.30950 0.27678
0.23922 0.29783 0.31273 0.28045
0.24314 0.30081 0.31594 0.28412
0.24706 0.30375 0.31910 0.28778
0.25098 0.30666 0.32227 0.29146
0.25490 0.30954 0.32544 0.29513
0.25882 0.31239 0.32864 0.29879
0.26275 0.31521 0.33185 0.30242
0.26667 0.31801 0.33510 0.30606
0.27059 0.32080 0.33837 0.30971
0.27451 0.32357 0.34171 0.31334
0.27843 0.32633 0.34507 0.31698
0.28235 0.32909 0.34844 0.32062
0.28627 0.33184 0.35180 0.32425
0.29020 0.33460 0.35515 0.32787
0.29412 0.33736 0.35850 0.33149
0.29804 0.34013 0.36182 0.33510
0.30196 0.34292 0.36512 0.33870
0.30588 0.34573 0.36840 0.34229
0.30980 0.34856 0.37163 0.34587
0.31373 0.35141 0.37483 0.34944
0.31765 0.35430 0.37801 0.35302
0.32157 0.35723 0.38120 0.35660
0.32549 0.36019 0.38439 0.36018
0.32941 0.36319 0.38759 0.36376
0.33333 0.36622 0.39079 0.36735
0.33725 0.36926 0.39400 0.37095
0.34118 0.37232 0.39727 0.37455
0.34510 0.37539 0.40062 0.37815
0.34902 0.37847 0.40404 0.38178
0.35294 0.38152 0.40742 0.38542
0.35686 0.38457 0.41073 0.38906
0.36078 0.38759 0.41400 0.39271
0.36471 0.39061 0.41727 0.39636
0.36863 0.39361 0.42054 0.40003
0.37255 0.39661 0.42379 0.40371
0.37647 0.39957 0.42703 0.40738
0.38039 0.40251 0.43029 0.41101
0.38431 0.40542 0.43356 0.41463
0.38824 0.40834 0.43686 0.41826
0.39216 0.41125 0.44017 0.42191
0.39608 0.41415 0.44352 0.42554
0.40000 0.41704 0.44689 0.42916
0.40392 0.41992 0.45025 0.43277
0.40784 0.42281 0.45363 0.43636
0.41176 0.42569 0.45704 0.43995
0.41569 0.42858 0.46047 0.44353
0.41961 0.43148 0.46390 0.44712
0.42353 0.43438 0.46733 0.45071
0.42745 0.43729 0.47076 0.45430
0.43137 0.44021 0.47416 0.45790
0.43529 0.44316 0.47755 0.46154
0.43922 0.44611 0.48092 0.46519
0.44314 0.44907 0.48427 0.46886
0.44706 0.45203 0.48760 0.47255
0.45098 0.45500 0.49092 0.47624
0.45490 0.45800 0.49422 0.47994
0.45882 0.46104 0.49752 0.48364
0.46275 0.46410 0.50088 0.48734
0.46667 0.46717 0.50426 0.49102
0.47059 0.47023 0.50763 0.49470
0.47451 0.47329 0.51097 0.49837
0.47843 0.47633 0.51429 0.50204
0.48235 0.47937 0.51760 0.50570
0.48627 0.48239 0.52091 0.50936
0.49020 0.48540 0.52423 0.51301
0.49412 0.48838 0.52756 0.51662
0.49804 0.49131 0.53090 0.52021
0.50196 0.49420 0.53424 0.52377
0.50588 0.49704 0.53761 0.52730
0.50980 0.49984 0.54101 0.53079
0.51373 0.50260 0.54441 0.53426
0.51765 0.50535 0.54780 0.53770
0.52157 0.50809 0.55119 0.54112
0.52549 0.51083 0.55459 0.54452
0.52941 0.51358 0.55797 0.54792
0.53333 0.51637 0.56136 0.55132
0.53725 0.51917 0.56478 0.55473
0.54118 0.52201 0.56821 0.55817
0.54510 0.52486 0.57156 0.56162
0.54902 0.52773 0.57486 0.56510
0.55294 0.53063 0.57815 0.56863
0.55686 0.53355 0.58143 0.57219
0.56078 0.53649 0.58470 0.57579
0.56471 0.53943 0.58796 0.57941
0.56863 0.54237 0.59122 0.58308
0.57255 0.54532 0.59449 0.58680
0.57647 0.54827 0.59776 0.59054
0.58039 0.55119 0.60104 0.59430
0.58431 0.55408 0.60432 0.59807
0.58824 0.55695 0.60763 0.60185
0.59216 0.55978 0.61094 0.60565
0.59608 0.56259 0.61424 0.60945
0.60000 0.56539 0.61753 0.61327
0.60392 0.56820 0.62085 0.61709
0.60784 0.57102 0.62419 0.62091
0.61176 0.57386 0.62754 0.62471
0.61569 0.57671 0.63089 0.62851
0.61961 0.57960 0.63422 0.63229
0.62353 0.58252 0.63753 0.63606
0.62745 0.58548 0.64084 0.63981
0.63137 0.58849 0.64420 0.64354
0.63529 0.59153 0.64767 0.64726
0.63922 0.59461 0.65129 0.65096
0.64314 0.59772 0.65505 0.65466
0.64706 0.60087 0.65889 0.65832
0.65098 0.60405 0.66261 0.66193
0.65490 0.60723 0.66619 0.66547
0.65882 0.61039 0.66964 0.66895
0.66275 0.61351 0.67301 0.67239
0.66667 0.61660 0.67637 0.67586
0.67059 0.61968 0.67973 0.67937
0.67451 0.62277 0.68309 0.68291
0.67843 0.62587 0.68643 0.68648
0.68235 0.62898 0.68974 0.69004
0.68627 0.63209 0.69302 0.69359
0.69020 0.63519 0.69628 0.69712
0.69412 0.63827 0.69951 0.70066
0.69804 0.64136 0.70272 0.70421
0.70196 0.64446 0.70594 0.70777
0.70588 0.64758 0.70915 0.71135
0.70980 0.65070 0.71237 0.71493
0.71373 0.65383 0.71561 0.71852
0.71765 0.65695 0.71883 0.72214
0.72157 0.66009 0.72207 0.72578
0.72549 0.66323 0.72529 0.72943
0.72941 0.66636 0.72850 0.73310
0.73333 0.66948 0.73172 0.73678
0.73725 0.67259 0.73496 0.74048
0.74118 0.67569 0.73824 0.74418
0.74510 0.67880 0.74155 0.74790
0.74902 0.68191 0.74488 0.75164
0.75294 0.68502 0.74823 0.75539
0.75686 0.68812 0.75159 0.75914
0.76078 0.69120 0.75499 0.76290
0.76471 0.69427 0.75840 0.76668
0.76863 0.69734 0.76185 0.77046
0.77255 0.70040 0.76535 0.77424
0.77647 0.70346 0.76888 0.77803
0.78039 0.70653 0.77243 0.78182
0.78431 0.70960 0.77603 0.78563
0.78824 0.71269 0.77971 0.78946
0.79216 0.71580 0.78351 0.79329
0.79608 0.71890 0.78727 0.79712
0.80000 0.72198 0.79095 0.80093
0.80392 0.72505 0.79458 0.80476
0.80784 0.72814 0.79821 0.80863
0.81176 0.73124 0.80183 0.81250
0.81569 0.73433 0.80544 0.81638
0.81961 0.73743 0.80903 0.82027
0.82353 0.74054 0.81259 0.82416
0.82745 0.74367 0.81613 0.82806
0.83137 0.74680 0.81968 0.83195
0.83529 0.74994 0.82325 0.83583
0.83922 0.75307 0.82684 0.83970
0.84314 0.75619 0.83043 0.84356
0.84706 0.75930 0.83406 0.84741
0.85098 0.76242 0.83776 0.85126
0.85490 0.76555 0.84142 0.85511
0.85882 0.76867 0.84502 0.85896
0.86275 0.77178 0.84862 0.86283
0.86667 0.77489 0.85219 0.86669
0.87059 0.77799 0.85573 0.87054
0.87451 0.78109 0.85927 0.87438
0.87843 0.78420 0.86280 0.87821
0.88235 0.78733 0.86635 0.88202
0.88627 0.79048 0.86992 0.88581
0.89020 0.79364 0.87344 0.88959
0.89412 0.79680 0.87696 0.89338
0.89804 0.79998 0.88047 0.89716
0.90196 0.80316 0.88399 0.90093
0.90588 0.80634 0.88752 0.90468
0.90980 0.80953 0.89105 0.90842
0.91373 0.81271 0.89456 0.91217
0.91765 0.81589 0.89809 0.91591
0.92157 0.81906 0.90164 0.91964
0.92549 0.82222 0.90518 0.92335
0.92941 0.82538 0.90873 0.92704
0.93333 0.82854 0.91228 0.93072
0.93725 0.83169 0.91582 0.93440
0.94118 0.83484 0.91938 0.93809
0.94510 0.83801 0.92293 0.94178
0.94902 0.84119 0.92647 0.94547
0.95294 0.84438 0.93002 0.94916
0.95686 0.84757 0.93356 0.95285
0.96078 0.85077 0.93710 0.95654
0.96471 0.85398 0.94065 0.96023
0.96863 0.85720 0.94419 0.96392
0.97255 0.86041 0.94773 0.96760
0.97647 0.86363 0.95128 0.97127
0.98039 0.86683 0.95483 0.97493
0.98431 0.87004 0.95838 0.97858
0.98824 0.87324 0.96194 0.98222
0.99216 0.87643 0.96550 0.98586
0.99608 0.87962 0.96907 0.98948
1.0000 0.88281 0.97263 0.99310
END_DATA
CAL
its author is all for GPL, and very helpful if needed.
but this graphic card 1D LUT is also used on the desktop, so you get D65 calibration in picture viewers and games....plus this is done in 10 bits, ARGYLLCMS has a tool to check the LUT accuracy.
considering the TMDS of the HDMI/DVI outputs will be encoded in RGB24 anyhow, and that the 3D LUT computing is done in 64bit floating point per component, I don't see it getting any more accurate and show *VISIBLE* improvement :o

[yesgrey]

Would there be any quality advantage of completely bypassing the video card LUT (i.e. leaving your 8bit display uncalibrated) and applying your calibrated gamma ramp instead through 3dlut before it does its other adjustments?
IMHO for video it would be like this:
-3D LUT 8bit and GC LUT 8bit: do all processing in 3D LUT
-3D LUT 8bit and GC LUT 10bit: do all processing in 3D LUT except gamma correction to fit displays gamma, this should be done in the GC LUT, which has higher accuracy
-3D LUT 16bit dithered to 8bit: do all processing in 3D LUT.
The last option is not available, and I don't know if it will ever be.
If so, could support for applying a calibrated gamma ramp (using something like the values exported from CalibrationTester) before the CMS compensations are done be added?
This is part of my plans, but in fact the gamma correction would be part of the CMS, the last step. I'm already working on it, but I'm waiting feedback about the 3DLUT file format specification.

[yesgrey]

but this graphic card 1D LUT is also used on the desktop, so you get D65 calibration in picture viewers and games....
Yes. The 3D LUT we are talking about only could be used for videos and photos, if you use a program that uses it (currently only an avisynth scrip directly or via ffdshow support). For the desktop, you are better with the GC 1D LUT.

[leeperry]

Yes. The 3D LUT we are talking about only could be used for videos and photos, if you use a program that uses it (currently only an avisynth scrip directly or via ffdshow support). For the desktop, you are better with the GC 1D LUT.
well we could use the GC 1D LUT for the desktop, and OVERLAY for videos(as it doesn't care for the GC LUT) but :
-I don't wanna use Overlay, HR is far smoother and free of video drivers sharpening blabla
-the GC LUT is 10 bits, it won't create more banding than needed by the calibration IMHO

BTW I'm being lazy as my gamut config works perfectly fine as it is, just that I have to use one YUY2 LUT for RGB conversion + one RGB LUT for gamut conversion....I will try to look into creating an all-in-one LUT w/ your app later today :p

[cyberbeing]

3D LUT 16bit dithered to 8bit: do all processing in 3D LUT.
The last option is not available, and I don't know if it will ever be.

That last option is what I was thinking of. What is the reason against implementing this? Would it be too slow? I was under the assumption that rgb3dlut already had this capability and all it would need was a specially created lut file to do this.

[leeperry]

That last option is what I was thinking of. What is the reason against implementing this? Would it be too slow?
the LUT would be HUGE, and the visible improvement inexistent? it's already done in 64 bits float, and you're gonna have to wait forever before display drivers actually support 10 bits.

OTOH maybe 10 bits could be implement for futureproof sake's?

[yesgrey]

That last option is what I was thinking of. What is the reason against implementing this? Would it be too slow? I was under the assumption that rgb3dlut already had this capability and all it would need was a specially created lut file to do this.
I will add the 16 bit output 3D LUT very soon, so it will only depends if tritical would want to add dithering to 8bits in rgb3dlut. It will be slower than without dithering, but essentially it will depend on the dither algorythm used... Floyd-Steinberg could be the best balance between quality and speed. From the dithering wiki-page (http://en.wikipedia.org/wiki/Dithering) the stucki method looks the best for me, but probably it would be too slow and/or too complicated to add...
the LUT would be HUGE
The LUT would be HUGE for 16 bit input. For 8bit input/16 bit output it will be 96MB, still pretty usable...
it's already done in 64 bits float
That's the internal computing resolution, the output resolution will always be a limiting factor...;)
So it should be: "16 bit" better than "16bit dithered to 8bit" better than "8bit".

[leeperry]

the output resolution will always be a limiting factor
true, but all we got now is RGB24 TMDS anyhow.
maybe in a few years if we've not been naughty, we'll get 10 bits...so adding 10 bits/10 bits dither would be more useful IMHO.

of course we'll have to buy spanking new HDMI 1.3 displays.....personally I've never seen any banding using your algorithms in 8 bits, so I'm not too worried :cool:

and before we get >8 bits native video sources, it will take a LONG while.

[cyberbeing]

so it will only depends if tritical would want to add dithering to 8bits in rgb3dlut.
You heard the man, so tritical, any interest in adding 16bit-->8bit dithering in rgb3dlut?


the LUT would be HUGE
The LUT would be HUGE for 16 bit input. For 8bit input/16 bit output it will be 96MB, still pretty usable...
Photoshop does all processing with 16bit per component and 128bit floating point precision via lut tables. Out of curiosity how do you assume Photoshop does this without taking up massive diskspace or memory with 16bit and 32bit input images?

Is there some better alternative method which is being overlooked? Do you think Photoshop generates a partial lut table wherever needed, on the fly? Something else? Too slow for your uses?

[leeperry]

Photoshop does all processing with 16bit per component and 128bit floating point precision via lut tables.
I demand 1024bit accuracy :p j/k ;)

pshop uses ICC v4 files for softproofing AFAIK

[cyberbeing]

I demand 1024bit accuracy :p j/k ;)
Duotriguple Precision??? :scared:


pshop uses ICC v4 files for softproofing AFAIK

If that is the case, and it somehow makes the situation better, why aren't we using 16bit ICC profiles in ddcc?

[tetsuo55]

Awesome news.

Just want to add view.

-The default Lut's should be 16bit per component, lower bits are optional(for slower systems)
-All internal processing should be 64bit or higher
-The should be a choice between dithering:none, fast, balanced, slow (increasing in quality over speed)
-The output bitdepth should be selectable: 6,8,10,12,16(I'm not sure if 6bit is possible and if it helps those 6bit lcd panels at all)


Once this all works we will need a proof of concept.

-A player that supports 16bit
-A renderer that supports 16bit
-A 16bit sample

Once we have all this in place we can start demanding support (the egg is in place)

-Write all the videocard developers that we want:
*10/12/16bit per component output support
*HDMI1.3C on their videocards
*We want updated drivers supporting this for current hardware

-Write all videocard reseller brands(like asus, msi) the same email.
-Write all display manufacturers the same letter:
*16 bit and lower per componenent input
*Native 16 panels
*Driver updates for current 10/12 bit panels

-File support tickets with the oem companies that you cannot use this 16bit setup because the system they sold you does not support it :(
-Ask the oem companies for 16bit capable hardware and get dissapointed for them not selling it

We might have to push Micrsoft too..

PS.

i know there is no commercial >8 bit content at this time. Maybe a big wave like the one above can spark more interest

[yesgrey]

before we get >8 bits native video sources, it will take a LONG while.
For >8bit sources the 3DLUT will not be usable...
Input/output: 3D LUT size
8bit/8bit: 48MB
8bit/16bit: 96MB
9bit/9bit: 432MB
10bit/10bit: 3840MB
But since the sources should being kept at 8bit...

You heard the man, so tritical, any interest in adding 16bit-->8bit dithering in rgb3dlut?

It doesn't have to be tritical doing it... the source code is available, so anyone who wishes to can do it. But I think it would be better tritical doing it... let's hope he agrees with it and have the time for it.;)

Photoshop does all processing with 16bit per component and 128bit floating point precision via lut tables. Out of curiosity how do you assume Photoshop does this without taking up massive diskspace or memory with 16bit and 32bit input images?
Photoshop should be using the lut only for the gamma correction, and for that we only need 1D LUTs - each component only depends on itself.
The size formula for 3 1D LUT is: 3*(2^biti)*bito/8 bytes
So, for a 16bit in/16bit out the size is only: 384kB

why aren't we using 16bit ICC profiles in ddcc?
Because the ICC profiles should be used at the software player renderer level.

-All internal processing should be 64bit or higher
No need for higher than 64bit FP. I have tryed 80bit FP and the results were exactly the same, only 15% slower in the computation.
I have also tryed 32bit FP, but with that was a slightly difference against 64bit. Some values differ of +/- 1. Nothing visually noticeable, but since we are performing the computation offline there is no reason to get less accurate results.
-The should be a choice between dithering:none, fast, balanced, slow (increasing in quality over speed)
I think it would be good enough just a basic algorythm like Floyd-steinberg or Sierra-lite (I think this is preferable, since it's the same visual quality as floyd and a little less cpu intensive). Even without dithering we aren't noticing any banding, so let's keep this simple.
I'm not sure if 6bit is possible and if it helps those 6bit lcd panels at all
It's possible, but it would be useless. The 6bit lcd panels perform their own dithering from an 8bit input for their native 6bit.
Once we have all this in place we can start demanding support (the egg is in place)
A massive attack!!!:D

[leeperry]

Photoshop should be using the lut only for the gamma correction, and for that we only need 1D LUTs - each component only depends on itself.
The size formula for 3 1D LUT is: 3*(2^biti)*bito/8 bytes
So, for a 16bit in/16bit out the size is only: 384kB
apparently you can have 3D LUT's within ICC v4 profiles.
I played around w/ them in X-Rite Profile Maker 5, too bad they only work in color managed apps(pshop, firefox, etc..)
pshop can do full gamut conversions w/ its softproofing options apparently.
but going ICC v4 in rgb3dlut would be pointless, as this is not even an open standard I think? not sure

[madshi]

sig - File signature, must be: '3DLUT'
'3DLUT' are only 5 chars. So maybe the definition should be "char sig[5]"?

int size, ver, biti, bito, cci, cco;
char pname[20];
int pver, sizerp, reserv1, reserv2;
I'd prefer much much MUCH longer names. Look at all the win32 structures. Your average field name is about 20-30 chars there...

Does "size" include the signature and the "size" field itself?

cci - Input color coding
0 - R'G'B'
> 0 - Y'Cb'Cr' - index to luma_matrix_coeffs
cco - Output color coding
0 - R'G'B'
> 0 - Y'Cb'Cr' - index to luma_matrix_coeffs
What does "luma_matrix_coeffs" mean? How about this?

COLOR_CODING_RGB = 0;
COLOR_CODING_YCbCr = 1;
COLOR_CODING_LUMA_MATRIX_COEFFS = 0x10000;

The matrix coeffs (whatever they mean) would then be "cci & 0xffff". And we'd have a lot of room for additional values between YCbCr and luma_matrix_coeffs for potential future use. But since I don't really know what is meant with matrix coeffs, my suggestion may very well be stupid. So take it with a pinch of salt, please...

pver - Version of the program that created the file
That should be a long int (64bit) to be win32 file version compatible.

Rest looks fine to me.

IMHO for video it would be like this:
-3D LUT 8bit and GC LUT 8bit: do all processing in 3D LUT
-3D LUT 8bit and GC LUT 10bit: do all processing in 3D LUT except gamma correction to fit displays gamma, this should be done in the GC LUT, which has higher accuracy
-3D LUT 16bit dithered to 8bit: do all processing in 3D LUT.
The last option is not available, and I don't know if it will ever be.
I think you're being too pessimistic... :)

This is part of my plans, but in fact the gamma correction would be part of the CMS, the last step.
That would be very nice!!

Floyd-Steinberg could be the best balance between quality and speed. From the dithering wiki-page (http://en.wikipedia.org/wiki/Dithering) the stucki method looks the best for me, but probably it would be too slow and/or too complicated to add...
I think the differences between the colors are small enough with RGB24 so that Stucki won't have much of an advantage over Floyd-Steinberg. Actually I think maybe using random dithering (like in audio processing) would even be superior to using either Stucki or Floyd-Steinberg. Random dithering looks bad if the color differences are big. But I think with small color differences random dithering could play out to be surprisingly good. Well, I'm only speculating, of course...

So it should be: "16 bit" better than "16bit dithered to 8bit" better than "8bit".
Fully agreed. However, "16bit dithered to 8bit" should be a lot nearer to 16bit than to 8bit. That is, if there's any difference visible between 16bit and 8bit at all. Jury is still out on that, I have to admit...

[tritical]

I can add 16 bit to 8 bit support, but it will take a little while. rgb3dlut really needs to be rewritten so that the yuy2 -> packed 4:4:4 upsampling step is separate from the packed 4:4:4 -> RGB conversion via lut step. Right now 14 separate code paths exist in rgb3dlut (12 for yuy2 input w/ rgb output), and I'd have to add 16->8 support to all of them. Separating the upsampling out would leave only 4 code paths. I'm skeptical if 16-bit output with dithering to 8-bit, given 8-bit input, will make any real difference though (we're talking about changing r/g/b values by +-1).

My current plans are:

separate rgb3dlut and yv12toyuy2 from ddcc
rename rgb3dlut to 3dlut
add rgb->yuy2, yuy2->yuy2, and yv12->yv12 support
separate upsampling/downsampling steps inside 3dlut (4:2:2->4:4:4,4:4:4->4:2:2) from lut step

After that, I will add 16-bit lut output w/ dithering to 8-bit. I can't say how fast all of this will happen. I do have other stuff that I actually get paid to work on :D.

The proposed header looks ok to me, but it doesn't make much difference to rgb3dlut anyways... aside from some error checking and automatic parameter setting. I think the specific values of cci/cco should be listed though.

[madshi]

@tritical, that sounds awesome - thanks! It will be very interesting to see whether dithering does or does not make a visible difference. I guess we'll need some fancy test screens for that then with very smooth color and brightness gradations...

@yesgrey3 & tcritical, here are some additional thoughts about bitdepth and array / file format:

input bitdepth

For unscaled progressive sources 8bit input should be good enough. But eventually if deinterlacing and scaling is made use of, having slightly higher input bitdepth may be worth a thought. But of course the needed memory size grows astronomical, unfortunately. But:

(1) Deinterlacing and scaling are usually done in YCbCr and not in RGB, AFAIK.
(2) Brightness information is more important than color information.
(3) For the array access we need to convert to YCbCr 4:4:4, so we put too much weight on color information compared to brightness information.

So my thought is this: Would it make sense to (optionally) up the Y bitdepth to 10bit while leaving Cb and Cr at 8bit? That would increase needed memory by factor 4x, which is a lot, but still manageable, I think. Of course this input format would make sense only if there was a deinterlacing/scaling algorithm which outputs more than 8bit. Don't know if any such thing exists yet. But even if it doesn't exist yet, who knows what the future will bring. FWIW, all the good hardware video processing chips calculate at least in 10bit YCbCr internally when doing deinterlacing & scaling etc...

One minor problem is that the current file format draft wouldn't support a funny input format like this. So I'd suggest splitting the "input/output bitdepth" field into 3 fields, one for each input/output component. E.g. "int inputBitdepth [3]".

array pack format

Thinking about memory sizes, and how to pack the output data:

We could split the array into 3 parts, one for each output component. This would give the program loading the 3dlut the chance to load the array into memory in 3 separate chunks. That is a big advantage, especially if the array sizes get rather big. E.g. Y10Cb8Cr8 input and R16G16B16 output would require an array of 384MB. Now the memory consumption itself is still manageable, but the bigger problem is the limited address range in a 32bit process. There's only 4GB of memory address range available per 32bit process and the whole range gets fragmented already during creation and initialization of the process. Furthermore some of those 4GB are reserved to the OS. So it will be hard to find a memory address where you can put a continuous block of 384MB into. In my experience you can have good or bad luck allocating such a big continuous memory block. There's a chance that it might fail - especially inside of a typical DirectShow media player, which has loads of DirectShow filter dlls loaded. Chances would be much much higher for allocating 3 separate memory blocks à 128MB. So that's a really good argument for splitting the array into 3 chunks, one for each output component.

[yesgrey]

'3DLUT' are only 5 chars. So maybe the definition should be "char sig[5]"?
I was counting with the '\0' character, but I will change to 5.
Does "size" include the signature and the "size" field itself?
Yes. size is the size of the header. signature and size are part of the header.
That is, if there's any difference visible between 16bit and 8bit at all. Jury is still out on that, I have to admit...
That's the big question... We are very happy playing with all this bits stuff, but in the end probably we will not notice any difference (I will for sure, because I will not admit that all this work was for nothing...:D)
My current plans are:

Thanks for the planning. I will adjust my planning to yours.
I guess we'll need some fancy test screens for that then with very smooth color and brightness gradations...
You could start creating some...;)
If you don't know how to put them in video just post the pics here and I will do it.
here are some additional thoughts about bitdepth...
Interesting thoughts, but I think that without seeing it we will never know...
IMHO all this >8bit processing is important, because it's the only way of preserving the full 8bit color depth of the source... If we perform all these computations using 8bit, in the end we will keep only 6 or 7 bits of the source color depth...:(
Would it make sense to (optionally) up the Y bitdepth to 10bit while leaving Cb and Cr at 8bit?
Why not? Since I've decided to play the chicken I will put one more egg...:D
We could split the array into 3 parts, one for each output component.
I think that would be a bad idea. I agree with the splitting, but not like that. When we access the 3D LUT, we always retrieve 3 values (r,g,b or y,cb,cr), so, these three values should be stored together. I think it would be better splitt the 3D LUT file for Cb values
For example, for YCbCr(10bit;8bit;8bit):
(Y:0-1023;Cb:0-255;Cr:0-63)+(Y:0-1023;Cb:0-255;Cr:64-127)+(Y:0-1023;Cb:0-255;Cr:128-191)+(Y:0-1023;Cb:0-255;Cr:192-255)

Let's see what tritical thinks about it...

[yesgrey]

Here is the second iteration for the definition of a file format for the 3D LUT (with names more like at Microsoft way...;)):
struct
{
char signature[5];
int headerSize, fileVersion;
int inputBitDepth[3], outputBitDepth[3];
int inputColorEncoding, outputColorEncoding;
char programName[20];
long int programVersion;
int parametersSize;
int reserved1, reserved2;
} h3dlut;

/* 3D LUT file specification:

// Header
signature - File signature; must be: '3DLUT'
headerSize - File header size in bytes
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or G,B,R)
outputBitDepth - Output bit depth per component (Y,Cb,Cr or G,B,R)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array of char with a copy of the
run parameters settings used for creating the file
reserved1 - Reserved for future usage
reserved2 - Reserved for future usage

// Parameters Settings
parametersSize bytes
// Data
outputBitDepth=8:
offset = (cr<<(2*inputBitDepth[1])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(2*inputBitDepth[1])+ b<<(inputBitDepth[0])+g)*3 (RGB input)
size = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2]) bytes
outputBitDepth<=16:
offset = (cr<<(2*inputBitDepth[1])+cb<<(inputBitDepth[0])+y)*6 (YCbCr input)
offset = ( r<<(2*inputBitDepth[1])+ b<<(inputBitDepth[0])+g)*6 (RGB input)
size = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])*2 bytes
(This is the general formula for the 3 3D LUT tables offset and size)
*/
I have not decided yet how to indicate the ColorEncoding values, I will update the specification later. I need a little more time to think about it.

I think that the offset navigation should be in bytes, so, when outputBitDepth > 8 it will assume always a 16bit output 3D LUT size. Maybe it's better only considering 8bit or 16bit output, or we will have to decide how to deal with the bits not used...

Comments/sugestions are welcome.

[Mug Funky]

you're gonna have to wait forever before display drivers actually support 10 bits.

http://h20331.www2.hp.com/hpsub/cache/596803-0-0-225-121.html

nvidia's higher end cards will do 10 (or 12) bit output via an SDI option (HDMI can carry it too, i believe).

also sony BVM monitors have been doing 10-bit for years (since 1992?). whether it achieves a noise floor below -48dB i'm not sure, but they certainly don't visibly band on gentle gradients.

[madshi]

You could start creating some...;)
If you don't know how to put them in video just post the pics here and I will do it.
The problem is that ideally a test screen should be created in YCbCr and not in RGB. E.g. a smooth gradiant from lowest to highest brightness in YCbCr in one bit steps would be helpful. And I don't know how to create YCbCr pics.

Why not? Since I've decided to play the chicken I will put one more egg...:D
Great - thanks!!

I think that would be a bad idea. I agree with the splitting, but not like that. When we access the 3D LUT, we always retrieve 3 values (r,g,b or y,cb,cr), so, these three values should be stored together.
I changed my mind. Please ignore my splitting suggestion. Because I think it is not the responsibility of a file format to dictate how the data has to be stored in RAM. I think it's the task of the software which loads the 3dlut data to store the data in RAM in the best possible way. If the array needs to be split in order to be mappable it into RAM, then let the software which loads the data do this work. I think you should just write one big continuous data array (just like you originally planned) and be done with it.

Here is the second iteration for the definition of a file format for the 3D LUT (with names more like at Microsoft way...;)):
struct
{
char signature[5];
int headerSize, fileVersion;
int inputBitDepth[3], outputBitDepth[3];
int inputColorEncoding, outputColorEncoding;
char programName[20];
long int programVersion;
int parametersSize;
int reserved1, reserved2;
} h3dlut;

/* 3D LUT file specification:

// Header
signature - File signature; must be: '3DLUT'
headerSize - File header size in bytes
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or G,B,R)
outputBitDepth - Output bit depth per component (Y,Cb,Cr or G,B,R)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array of char with a copy of the
run parameters settings used for creating the file
reserved1 - Reserved for future usage
reserved2 - Reserved for future usage
Looks fine to me! Could you please add "char parameter data [?]" to the end of the "h3dlut" structure definition, just to make clear where the parameter data is stored?

// Parameters Settings
parametersSize bytes
// Data
outputBitDepth=8:
offset = (cr<<(2*inputBitDepth[1])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(2*inputBitDepth[1])+ b<<(inputBitDepth[0])+g)*3 (RGB input)
size = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2]) bytes
outputBitDepth<=16:
offset = (cr<<(2*inputBitDepth[1])+cb<<(inputBitDepth[0])+y)*6 (YCbCr input)
offset = ( r<<(2*inputBitDepth[1])+ b<<(inputBitDepth[0])+g)*6 (RGB input)
size = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])*2 bytes
(This is the general formula for the 3 3D LUT tables offset and size)
*/
I think that the offset navigation should be in bytes, so, when outputBitDepth > 8 it will assume always a 16bit output 3D LUT size. Maybe it's better only considering 8bit or 16bit output, or we will have to decide how to deal with the bits not used...
Agreed, using only 8bit or 16bit byte packed output sounds just fine to me. If the software loading the file thinks its better to use bit packing, it can rearrange the array to its liking.

But, I think the offset calculation is wrong. Shouldn't it be this way?

offset = (cr << (inputBitDepth[1] + inputBitDepth[0]) + cb << (inputBitDepth[0]) + y) * 3 (YCbCr input)

[leeperry]

http://h20331.www2.hp.com/hpsub/cache/596803-0-0-225-121.html

nvidia's higher end cards will do 10 (or 12) bit output via an SDI option (HDMI can carry it too, i believe).

also sony BVM monitors have been doing 10-bit for years (since 1992?). whether it achieves a noise floor below -48dB i'm not sure, but they certainly don't visibly band on gentle gradients.
well, any video card can do 30bit on VGA, it's mostly 8 bit + 10bit LUT...I can use ARGYLLCMS to measure that the LUT is in 10bit.
we're stuck to RGB24 over HDMI/DVI coz the TMDS encoder works in RGB24 anyhow.
oh sure, there must be some high end cards that can do true 10bit on SDI/dual link DVI, but these are not consumer items AFAIK.
it would appear that the HDMI 1.3 licences would be so high that there's hardly any soundcard compatible, let alone graphic card.

[honai]

This might be a useful 1920x1080 pattern to detect banding/dithering issues:

http://web.comhem.se/zacabeb/repository/spectrum_rgb.png

[madshi]

I've just had an idea about how to make *any* input bitdepth work with a relatively small (e.g. 8bit input, 16bit output) 3dlut array:

Instead of rounding the input down to the native 3dlut bitdepth, we read the 8 nearest RGB output values from the 3dlut array. Then we average these 8 values down (weighted average) to interpolate the final RGB output value.

I'm thinking of the 3dlut array like a net of 3D cubes. Each value in the 3dlut array is one cube corner. Now if we have an input bitdepth higher than the native 3dlut array, the input usually doesn't fall directly on a 3dlut array cube corner. Instead it's somewhere in the middle of one of those cubes. So we read the surrounding cube corner values and interpolate the final output value, based on how far the input value is away from each of the 8 corners.

Does that make any sense to you? Of course my suggestion would result a noticeable performance loss, cause we'd have to read 8 array values instead of just one and we'd have to use some crazy formula to combine these 8 values into the final output value. But I think the result should be very near to perfect. It should *almost* be as good as having a 3dlut with a much higher native input bitdepth.

Thoughts?

Anyone willing to write a formula to calculate the final output value? Unfortunately my math sucks...

[tritical]

I would like to change the lookup into the rgb lut to be:

(r<<16)+(g<<8)+b

Also, I am dropping the idea of adding yv12->yv12 support as it just doesn't make sense (internally it would have to convert to 4:4:4 somehow, so might as well require yuy2 input and let the user decide how to do the yv12->yuy2 conversion).

@madshi
Your idea is basically the same idea that is used with smaller luts, such as 16x16x16, applied to higher bitdepths... for input values that don't fall on the samples defined by the lut interpolate inside the cube. I think that for input sources with greater than 8-bits this is the way to go due to memory requirements. So stick with an 8-bit input, 16-bit output and interpolate.

[yesgrey]

But, I think the offset calculation is wrong. Shouldn't it be this way?
offset = (cr << (inputBitDepth[1] + inputBitDepth[0]) + cb << (inputBitDepth[0]) + y) * 3 (YCbCr input)
Yes, you're right. I will correct it in iteration 3.:o
I would like to change the lookup into the rgb lut to be:
(r<<16)+(g<<8)+(b<<8)

Is this correct? or would it be: (r<<16)+(g<<8)+b
I have suggested: (r<<16)+(b<<8)+g
to keep the correlation Y->G; Cb->B; Cr->R.
as indicated in the ITU specifications, but for me it's ok to use it as you suggested... and what about the output format, do you also agree to always use 1 byte or 2byte?

[tritical]

Yep, I meant

(r<<16)+(g<<8)+b

and

(v<<16)+(u<<8)+y

So that lookups using rgb packed as: b,g,r in memory, and yuv packed as: y,u,v in memory form the lookup the same way in terms of reading/shifting from memory locations. I agree only 1 byte and 2 byte outputs.

[madshi]

Also, I am dropping the idea of adding yv12->yv12 support as it just doesn't make sense (internally it would have to convert to 4:4:4 somehow, so might as well require yuy2 input and let the user decide how to do the yv12->yuy2 conversion).
I'm wondering: As far as I understand yv12 is 4:2:0 and yuy2 is 4:2:2, right? Is there also an "official" fourcc for 4:4:4 YCbCr? And wouldn't it be faster and better to convert 4:2:0 directly to 4:4:4 instead of doing 4:2:0 -> 4:2:2 -> 4:4:4?

Your idea is basically the same idea that is used with smaller luts, such as 16x16x16, applied to higher bitdepths... for input values that don't fall on the samples defined by the lut interpolate inside the cube. I think that for input sources with greater than 8-bits this is the way to go due to memory requirements.
And there I thought I had a brand new idea... :o

Anyway, is there code available anywhere (or at least a formula) on how to "interpolate inside the cube"?

Thanks!

[yesgrey]

Is there also an "official" fourcc for 4:4:4 YCbCr?
YV24.
And wouldn't it be faster and better to convert 4:2:0 directly to 4:4:4 instead of doing 4:2:0 -> 4:2:2 -> 4:4:4?

This was discussed a few posts back. That was the reason for tritical create the yv12toyuy2 filter, so we could test if we could notice any quality difference, since ffdshow is performing yv12->yv24 with no intermediate yuy2 step.;)
My first tests showed that yv12->yv24 was slightly better, but I haven't tested yet the more recent versions of rgb3dlut and yv12toyuy2 with the new options for the chroma sampling position...
In the end, maybe the difference is only noticeable in still zoomed pictures, and nobody see a movie that way...:D

[yesgrey]

Here is the third iteration for the definition of a file format for the 3D LUT:
// 3D LUT file specification (fileVersion=1):
struct
{
char signature[5];
int headerSize, fileVersion;
int inputBitDepth[3], outputBitDepth[3];
int inputColorEncoding, outputColorEncoding;
char programName[20];
long int programVersion;
int parametersSize;
int reserved1, reserved2;
} h3dlut;
char parametersData[];
void lut3d[];
/*
// Header
signature - File signature; must be: '3DLUT'
headerSize - File header size in bytes
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or G,B,R)
outputBitDepth - Output bit depth per component (Y,Cb,Cr or G,B,R)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array of char with a copy of the
run parameters settings used for creating the file
reserved1 - Reserved for future usage
reserved2 - Reserved for future usage

// Parameters Settings
parametersData - pointer to an array of char with size parametersSize
// 3D LUT Data
lut3d - pointer to an array that contains the 3dlut output values.
The type of the lut3d array entries is defined by the outputBitDepth values:
-If all outputBitDepth values are 8, it's an array of unsigned char.
-If any of outputBitDepth values is >8 and <=16, it's an array of unsigned short.
The offset and dimension of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
dimension = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(This is the general formula for the 3 3D LUT tables offset and size)
*/

I have not decided yet how to indicate the ColorEncoding values, I will update the specification later. I need a little more time to think about it.

We could simplify it a little by using outputBitDepth as the same for all output channels, but the way it is now is more future proof...

Comments/sugestions are welcome.

If you all agree with this I will start working in cr3dlut v2.0 to release the first "official" version of the 3DLUT file format.

[madshi]

YV24.
Ah, never heard of that one yet! I wonder why ffdshow doesn't allow YV24 output?

Here is the third iteration for the definition of a file format for the 3D LUT
Looks good to me. Two minor cosmetic things:

(1) "char parametersData[];" should be the last element of the h3dlut struct definition, so it should be inside the structure, not outside. At least that's how win32 structures like that are usually done.

(2) offset and dimension can be either "3*" or "6*", depending on output bitdepth.

Thanks!

[honai]

I wonder why ffdshow doesn't allow YV24 output?


Probably because no DirectShow video renderer exists for YV24 on the Windows platform.

[yesgrey]

(1) "char parametersData[];" should be the last element of the h3dlut struct definition
I have think on that, but did not want to add the pointer to the header, because it would be completelly useless to store it. But you're right, more 4 bytes will make no harm.
(2) offset and dimension can be either "3*" or "6*", depending on output bitdepth.
No. The offset and dimension are always the same, what changes is the entries type. 'char' for 8bit output, and 'short' for 16 bit output.

[madshi]

I have think on that, but did not want to add the pointer to the header, because it would be completelly useless to store it. But you're right, more 4 bytes will make no harm.
I thought the first character was stored right after "reserved2" without any pointers?

[yesgrey]

Here is the fourth iteration for the definition of a file format for the 3D LUT:
// 3D LUT file specification (fileVersion=1):
struct
{
char signature[5];
int headerSize, fileVersion;
int inputBitDepth[3], outputBitDepth[3];
int inputColorEncoding, outputColorEncoding;
char programName[20];
long int programVersion;
int parametersSize;
int reserved1, reserved2;
char parametersData[];
} h3dlut;
void *o3dlut;
/*
// Header
signature - File signature; must be: '3DLUT'
headerSize - File header size in bytes
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or R,G,B)
outputBitDepth - Output bit depth per component (Y,Cb,Cr or R,G,B)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array of char with a copy of the
run parameters settings used for creating the file
reserved1 - Reserved for future usage
reserved2 - Reserved for future usage
parametersData - pointer to an array of char with size parametersSize
// Output data
o3dlut - pointer to an array that contains the 3dlut output values.
The type of the o3dlut array entries is defined by the outputBitDepth values:
-If all outputBitDepth values are 8, it's an array of unsigned char.
-If any of outputBitDepth values is >8 and <=16, it's an array of unsigned short.
The offset and dimension of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
dimension = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(This are the general formulas for the 3D LUT offset and dimension
assuming: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/

I have not decided yet how to indicate the ColorEncoding values, I will update the specification later. I need a little more time to think about it.

@tritical,
We could define it later, but how should we store <16bit values in the 'short'? MSB or LSB?

Comments/sugestions are welcome.

[tritical]

I'm wondering: As far as I understand yv12 is 4:2:0 and yuy2 is 4:2:2, right? Is there also an "official" fourcc for 4:4:4 YCbCr? And wouldn't it be faster and better to convert 4:2:0 directly to 4:4:4 instead of doing 4:2:0 -> 4:2:2 -> 4:4:4?
4:2:0 -> 4:4:4 is essentially 1 vertical resize and 1 horizontal resize of the chroma planes. Since we are talking about separable operations, it is usually faster to do them separately (except for nearest neighbor, linear is close, cubic and above is definitely faster). The question is whether you should store the result of each step with greater than 8-bit accuracy. ffdshow's new algorithm uses two point linear interpolation in each step (75/25 weighted averaging). Thus, to get the correct value you need 4 extra bits (2 extra for the first step result, and another 2 for the second). If you truncate to 8-bit after each step the largest difference you can incur (on 8-bit scale) is 0.5 after the first step and 1.0 after the second step. Therefore, once you get to 4:4:4 the u/v values could have a difference of up to 1.0. I have argued this is unnoticeable, and so far everyones comments support this... if you can't tell the difference between different chroma placements or linear vs cubic interpolation, both of which create much larger differences, then there is no way you can tell the difference here. For reference, avisynth's resizers separate horizontal/vertical steps, and store the intermediate result (if both width and height are altered) to 8-bit. They operate on luma as well as chroma.

Using higher intermediate bitdepths would yield more of a difference as the quantization error of a larger number pixels comes into play (like convolution with 7x7 kernel after resizing or chaining a bunch of single pixel operations together).

Also, I don't believe there is an "official" fourCC for 4:4:4. Avisynth 2.6 uses 'YV24' to identify its planar 4:4:4 yuv colorspace.

We could define it later, but how should we store <16bit values in the 'short'? MSB or LSB?
I say just cast it to unsigned 16-bit (so satured conversion to [0-65535] range), and store as little endian. My program can then just operate on it as though it was a 16 bit value... all that needs to change is how much to shift by at the end.

Anyway, is there code available anywhere (or at least a formula) on how to "interpolate inside the cube"?
If we make the assumption of linearity between points defined by the lut, then it is simply trilinear interpolation involving 8 points. I think anything else would be a pain.

[yesgrey]

For reference, avisynth's resizers separate horizontal/vertical steps, and store the intermediate result (if both width and height are altered) to 8-bit. They operate on luma as well as chroma.
Recently there was a user suggesting in ffdshow's thread to perform the resize and 4:2:0 -> 4:4:4 all in one step... that could be the best of all.;)

Also, I don't believe there is an "official" fourCC for 4:4:4. Avisynth 2.6 uses 'YV24' to identify its planar 4:4:4 yuv colorspace.
I think you are right. If we google yv24 there is little info about it... only Avisynth2.6 talks about it...

I say just cast it to unsigned 16-bit (so satured conversion to [0-65535] range), and store as little endian. My program can then just operate on it as though it was a 16 bit value... all that needs to change is how much to shift by at the end.
Let's look at an example to see if I understood it...
For 10bit output, the values will be stored in bits 6-15, and bits 0-5 will be set at 0, right? Then you just need to >>6 to get the 10bit value, right?
If it's right, there are two ways of doing it:
-When creating the 3D LUT I convert the float to 10bit and <<6
-When creating the 3D LUT I convert the float to 16bit, and then your application will take care of any rounding to 10bit
The second is the same of only creating 8bit or 16bit output 3D LUTs, but maybe the first would be a little more faster, because you will only need to shift and ignore the rounding.

[tritical]

Let's look at an example to see if I understood it...
For 10bit output, the values will be stored in bits 6-15, and bits 0-5 will be set at 0, right? Then you just need to >>6 to get the 10bit value, right?
I was thinking that 10 bit values in a lut would be stored in bits 0-9 of each 2 byte unsigned short.. so instead of the full [0-65535] range being used only [0-1023] would be used. That way, all calculations in rgb3dlut can be performed as if the values were full range, but when it comes time to calculate the final output values the right shift amount would be, assuming 8-bit output, 2 instead of 8, and the addition for rounding would be 4 instead of 128.

Using that method, it makes no sense to create a lut with 2 byte entries with < 16-bit values stored in them... unless all of the mapped values can be precisely represented in < 16 bit. Also, I think that there should be only one value allowed, i.e. the same for all planes, for outputBitDepth (talking about bit depth of values stored in the lut).

It's confusing talking about outputBitDepth with respect to values stored in the lut because the bit depth of the values output by a program using the lut (like rgb3dlut), and the bit depth of the values stored in the lut, are two different things. Maybe in the format description they could be called 'indexBitDepth' and 'storedBitDepth' instead of 'inputBitDepth' and 'outputBitDepth'. Again, I can see the usefullness in allowing different indexBitDepths for each plane, but not for different storedBitDepths.

[madshi]

4:2:0 -> 4:4:4 is essentially 1 vertical resize and 1 horizontal resize of the chroma planes. Since we are talking about separable operations, it is usually faster to do them separately (except for nearest neighbor, linear is close, cubic and above is definitely faster).
I didn't know that doing it separately would be faster... :eek:

Also, I don't believe there is an "official" fourCC for 4:4:4.
Strange.

If we make the assumption of linearity between points defined by the lut, then it is simply trilinear interpolation involving 8 points. I think anything else would be a pain.
Ok, thanks.

I was thinking that 10 bit values in a lut would be stored in bits 0-9 of each 2 byte unsigned short.. so instead of the full [0-65535] range being used only [0-1023] would be used. That way, all calculations in rgb3dlut can be performed as if the values were full range, but when it comes time to calculate the final output values the right shift amount would be, assuming 8-bit output, 2 instead of 8, and the addition for rounding would be 4 instead of 128.

Using that method, it makes no sense to create a lut with 2 byte entries with < 16-bit values stored in them...
So in other words only 8bit and 16bit should be supported and no intermediate bitdepths? Would be fine with me, since rounding from 16bit to e.g. 10bit would be *really* quick and easy, anyway...

Also, I think that there should be only one value allowed, i.e. the same for all planes, for outputBitDepth (talking about bit depth of values stored in the lut). [...] Again, I can see the usefullness in allowing different indexBitDepths for each plane, but not for different storedBitDepths.
Hmmmm... I have to agree.

It's confusing talking about outputBitDepth with respect to values stored in the lut because the bit depth of the values output by a program using the lut (like rgb3dlut), and the bit depth of the values stored in the lut, are two different things. Maybe in the format description they could be called 'indexBitDepth' and 'storedBitDepth' instead of 'inputBitDepth' and 'outputBitDepth'.
Well, using the 3dlut array is a simple process. You feed data in and you get data out. Of course the program using the lut can do additional processing on the array "output". But for me using the words "input" and "output" is intuitive, while using "indexBitDepth" and "storedBitDepth" (while making sense) is more difficult to understand. I'm thinking of the 3dlut array as one conversion step in a chain of conversions. After all before the 3dlut array you have to convert e.g. yv12 to yv24. And afterwards you may have to dither or round 16bit RGB down to 8bit. But still the 3dlut process itself for me is one separate conversion step which has an input and output. DirectShow filters also all have an input and output. And my own eac3to tool internally also has input and output for every conversion step.

[yesgrey]

Using that method, it makes no sense to create a lut with 2 byte entries with < 16-bit values stored in them... unless all of the mapped values can be precisely represented in < 16 bit.
Well, but that's the idea... I don't believe that if Windows7 brings 16 bit, the only options would be 8 bit or 16 bit, unless the drivers developers only want that...
Currently some displays only support 10 bit or 12 bit connection at most, and for those would be good a format like that, and with a LUT with the same bit depth there is no need for roundings inside rgb3dlut... But since this is just the future, and we don't know how will it be, we could simply stick with 8 bit and 16 bit modes. The proposed specification will support other bit depths between those, we could leave the option of "how" to a later stage, if it will ever be needed...

Also, I think that there should be only one value allowed, i.e. the same for all planes, for outputBitDepth ... Again, I can see the usefullness in allowing different indexBitDepths for each plane, but not for different storedBitDepths.
If we think only in using the 3D lut one time in a chain, I also can't see the usefullness in allowing different outputBitDepths, but if we think in using the 3D LUT two times in a chain, maybe it could be useful.
For example:
a YCbCr->YCbCr->RGB chain, with indexing YCbCr(10|8|8), it would be useful to have a stored YCbCr(10|8|8) to use when indexing into the 2nd 3D LUT.
Would this kind of usage make any sense? Maybe not, and we could always create a stored YCbCr(16|16|16) 3D LUT and reduce the bith depth for YCbCr(10|8|8) when indexing into the 2nd stage in the chain...
To be honest, I would be happy only with YCbCr->RGB with 8bit, 16bit and 16bit dithered to 8bit output. All the other options are useless to me. I'm only proposing them in the case someone else might find them useful...:)

It's confusing talking about outputBitDepth with respect to values stored in the lut because the bit depth of the values output by a program using the lut (like rgb3dlut), and the bit depth of the values stored in the lut, are two different things. Maybe in the format description they could be called 'indexBitDepth' and 'storedBitDepth' instead of 'inputBitDepth' and 'outputBitDepth'.
But for me using the words "input" and "output" is intuitive, while using "indexBitDepth" and "storedBitDepth" (while making sense) is more difficult to understand. I'm thinking of the 3dlut array as one conversion step in a chain of conversions.
I am ok with any of the namings. The index/stored is more accurate, but the input/output is a little more intuitive.
If I was the only person to use the specification, I would prefer the former, but maybe the later is more intuitive for other possible users...

[madshi]

To be honest, I would be happy only with YCbCr->RGB with 8bit, 16bit and 16bit dithered to 8bit output.
So it seems that none of us (tcritical, you and me) really wants different output bitdepths per plane, nor any other output bitdepth other than 8bit and 16bit. So if we 3 agree on that, then let's keep things simple and do it that way.

Still I'd keep using an "int" for output bitdepth instead of a "bool" byte/word switch, just to keep the door open for intermediate bitdepths at a later day...

[yesgrey]

So if we 3 agree on that, then let's keep things simple and do it that way.

Still I'd keep using an "int" for output bitdepth instead of a "bool" byte/word switch, just to keep the door open for intermediate bitdepths at a later day...
Simple, I agree with it.
No "bool", I also agree with it. I don't like "bools", I only use them if not avoidable (when using functions not mine that use them).

So it seems we just need to agree with the index/stored vs input/output naming...:)

[yesgrey]

After some discussion between me and madshi via PM, here is the fifth iteration for the definition of a file format for the 3D LUT:
// 3D LUT file specification (fileVersion=1):
typedef struct
{
char signature[5];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[20];
long int programVersion;
int parametersSize;
} h3dlut;
struct
{
h3dlut header;
char parametersData[1];
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
} f3dlut;
/*
// Header
signature - File signature; must be: '3DLUT'
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or R,G,B)
outputBitDepth - Output bit depth per component (The same for all components)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array parametersData
// Parameters data
parametersData - array of char with size parametersSize that contains a copy of the
run parameters settings used for creating the file
// 3D LUT output data
lut.dataxx - array with size lutSize that contains the 3dlut output values.
The type used (lut.dat8 or lut.data16) is defined by the
outputBitDepth field:
- lut.data8 when outputBitDepth = 8
- lut.data16 when outputBitDepth = 16
The offset and size of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
lutSize = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(Size in bytes:
- lutSize*1, when outputBitDepth = 8
- lutSize*2, when outputBitDepth = 16)
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/

I have not decided yet how to indicate the ColorEncoding values, I will update the specification later. I need a little more time to think about it.

tritical, do you agrre with it? It seems we only need to agree with the BitDepths naming...;)

Comments/sugestions are welcome.

[tetsuo55]

I'm not sure what you guys mean with only output 8 or 16 bit.

I would do everything with 16bit, and then allow dithering down to 8/10/12

[madshi]

I would do everything with 16bit, and then allow dithering down to 8/10/12
We're only talking about the 3dlut file format here. A software could do just what you suggested.

------------------

@yesgrey3 and @tritical: We still need a default file extension. How about *.3dl"? Or should be use "*.3dlut"? Or maybe "*.lut"? Hmmmmm... I think my favorite would be "*.3dlut". Your opinion?

[tetsuo55]

Looks like the first hdmi 1.3 10bit videocard is out:

http://www.s3graphics.com/en/products/desktop/chrome_540gtx/index.jsp

I have no seen any hard confirmation, purely based on its specs

[tritical]

It seems we only need to agree with the BitDepths naming...
Leaving the names as they are is fine with me.

So in other words only 8bit and 16bit should be supported and no intermediate bitdepths? Would be fine with me, since rounding from 16bit to e.g. 10bit would be *really* quick and easy, anyway...
To be honest, I would be happy only with YCbCr->RGB with 8bit, 16bit and 16bit dithered to 8bit output. All the other options are useless to me. I'm only proposing them in the case someone else might find them useful...
Well, but that's the idea... I don't believe that if Windows7 brings 16 bit, the only options would be 8 bit or 16 bit, unless the drivers developers only want that...
Currently some displays only support 10 bit or 12 bit connection at most, and for those would be good a format like that, and with a LUT with the same bit depth there is no need for roundings inside rgb3dlut... But since this is just the future, and we don't know how will it be, we could simply stick with 8 bit and 16 bit modes. The proposed specification will support other bit depths between those, we could leave the option of "how" to a later stage, if it will ever be needed...
I agree that only 8-bit and 16-bit should be supported. From my point of view, the lut should always use the full range it can (8-bit for byte output, 16-bit for 2 byte output), as that gives the dithering algorithm in the program using the lut the most information to work with if it is outputting at a lower bitdepth. If that program doesn't use dithering, then its conversion down to a different bitdepth should end up giving the same value that the program creating the lut would have given (technically it could be off by 1 at the lower bitdepth, but only for values very close to x.5), and the speed cost of its having to do that conversion is negligible.

@yesgrey3 and @tritical: We still need a default file extension. How about *.3dl"? Or should be use "*.3dlut"? Or maybe "*.lut"? Hmmmmm... I think my favorite would be "*.3dlut". Your opinion?
I like .3dlut as well.

[yesgrey]

@yesgrey3 and @tritical: We still need a default file extension. How about *.3dl"? Or should be use "*.3dlut"? Or maybe "*.lut"? Hmmmmm... I think my favorite would be "*.3dlut". Your opinion?
I prefer .3dlut too.
I agree that only 8-bit and 16-bit should be supported.
So, it seems we have our file format ready.:)
I will change cr3dlut to output this file format and do the ColorEncoding description, to complete the specification.

[tritical]

So, it seems we have our file format ready.
I will change cr3dlut to output this file format and do the ColorEncoding description, to complete the specification.
Great. I've completed all of the things I mentioned earlier, and have dithering from 16 to 8 added. I went with Floyd-Steinberg due to speed. Other fast diffusion methods are: burkes, sierra2, and filter lite... but they are all very similar. Just need to change it to read the new lut format, and do some testing. Should have time tommorrow.

[cyberbeing]

tritical, for videos which are not mod4, could you add the ability to automatically fallback to ConvetToYUY2() if desired, instead of throwing an error? (e.g. Fallback=True or Fallback=False with the default being False).

[jmartinr]

I agree that only 8-bit and 16-bit should be supported.

In some avisynth 3.0 documents there is talk about YV45: YV24, a 4:4:4 YUV planar color format, and RGB45 and YV45 which are 15 bits depth versions of RGB24 and YV24 (NB: even though RGB45 is 8 byte aligned, the extra is not alpha, just garbage).I wouldn't know myself, but maybe there's a good reason for using a 15 bits depth version?

[yesgrey]

Great. I've completed all of the things I mentioned earlier, and have dithering from 16 to 8 added.
I have to speed up to catch you and release cr3dlut at the same time...;)
In some avisynth 3.0 documents there is talk about YV45: I wouldn't know myself, but maybe there's a good reason for using a 15 bits depth version?
I think we should not start adding things that we don't know yet how they would end...
Currently, we are stuck with RGB32 and 8bit per component. Yes, we could be using RGB32 with 10bit per component and 2bit alpha, but we don't have it. DirectShow had considered it, but it seems it never appeared. For example, RGB24 was dropped by all graphic cards drivers developers in favor of RGB32.
It seems with Windows7 we will get 16bit per component. Let's see what else we will get... because in the end, the graphics card drivers define what we could use...

[leeperry]

tritical, for videos which are not mod4, could you add the ability to automatically fallback to ConvetToYUY2() if desired, instead of throwing an error? (e.g. Fallback=True or Fallback=False with the default being False).
or imode=0 which used to work fine w/ non-mod4

the real question is : will it show any visible improvement over ConvertToYUY2() as this is much more optimized and faster...

[madshi]

Great. I've completed all of the things I mentioned earlier, and have dithering from 16 to 8 added. I went with Floyd-Steinberg due to speed. Other fast diffusion methods are: burkes, sierra2, and filter lite... but they are all very similar. Just need to change it to read the new lut format, and do some testing. Should have time tommorrow.
Awesome! Have you had a chance to check whether dithering makes a visible difference or not? Thanks!

[IanB]

Seeing this is a file header, I would suggest you explicitly align all the elements independent of any compiler.

And seeing the lut is some 48+ megabytes, catering for accessing it via MMAPing technology in the future might be prudent, so align the lut data now to the largest architecture page size likely, i.e 8192 or maybe 16384. Also pad the entire file length to this page size.
// 3D LUT file specification (fileVersion=1):
typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int parametersSize;
} h3dlut;
enum { PAGE_SIZE = 8192; };
struct
{
h3dlut header;
char parametersData[PAGE_SIZE-sizeof(h3dlut)];
// The next element should start on a page aligned offset to allow MMAPing the raw data
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
// char padding[1]; // entire file length should be an exact multiple of Page Size.
} f3dlut;
/*
// Header
signature - File signature; must be: '3DLUT'
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or R,G,B)
outputBitDepth - Output bit depth per component (The same for all components)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array parametersData
// Parameters data
parametersData - array of char with size parametersSize that contains a copy of the
run parameters settings used for creating the file
// 3D LUT output data
lut.dataxx - array with size lutSize that contains the 3dlut output values.
The type used (lut.dat8 or lut.data16) is defined by the
outputBitDepth field:
- lut.data8 when outputBitDepth = 8
- lut.data16 when outputBitDepth = 16
The offset and size of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
lutSize = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(Size in bytes:
- lutSize*1, when outputBitDepth = 8
- lutSize*2, when outputBitDepth = 16)
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/


MMAPing involves directing the OS to page directly from a specified file, there are usually many constraints, like the disk image must be physical page aligned, etc. The technology avoids the startup overhead of reading a hugh file into memory. As the elements of data are accessed the OS page faults the required chunk into the address space. If certain elements are never used then the elements are never read or allocated physical memory.

[yesgrey]

Seeing this is a file header, I would suggest you explicitly align all the elements independent of any compiler.

And seeing the lut is some 48+ megabytes, catering for accessing it via MMAPing technology in the future might be prudent, so align the lut data now to the largest architecture page size likely, i.e 8192 or maybe 16384. Also pad the entire file length to this page size.
Thank you for your suggestions.:)
For me it's ok doing it like this, even considering than MMAPing could be useless for the 3D LUT, since we want it all in memory for performance reasons...
tritical, are you also ok with this?

[madshi]

What happens if parametersData is longer than "PAGE_SIZE-sizeof(h3dlut)"? What happens if parametersData is only 100 bytes? How does an application know that it's only 100 bytes and not "PAGE_SIZE-sizeof(h3dlut)"? Ok, the bytes 101+ could be zeroed out. But that's not really a pretty solution...

MMAPing involves directing the OS to page directly from a specified file, there are usually many constraints, like the disk image must be physical page aligned, etc. The technology avoids the startup overhead of reading a hugh file into memory. As the elements of data are accessed the OS page faults the required chunk into the address space. If certain elements are never used then the elements are never read or allocated physical memory.
That's true. But for real time playback it's better to have a startup overhead than to have the video stutter for a few seconds until the whole 48MB (or 96MB with 16bit output) are read in. So I don't really like the idea of memory mapping the array, at least not for media playback purposes. Reencoding may be a different situation...

[yesgrey]

How does an application know that it's only 100 bytes and not "PAGE_SIZE-sizeof(h3dlut)"? Ok, the bytes 101+ could be zeroed out. But that's not really a pretty solution...
By looking to parametersSize. I also agree it's not a pretty solution, but the idea is to get a general solution. We could dimension pametersData to a higher multiple of PAGE_SIZE just to be safe... For example: [4*PAGE_SIZE-sizeof(h3dlut)]; or putting parametersData at the end of the file.

So I don't really like the idea of memory mapping the array, at least not for media playback purposes. Reencoding may be a different situation...
I agree, but remember that MMAPing only will be used if the application wants it.

[IanB]

...
parametersSize - Size in bytes of the array parametersData
...And where the actual LUT data starts can either be implicitly at the first page aligned offset after the parameter data or you could explicitly add an offset field/padding length to the header.
I agree, but remember that MMAPing only will be used if the application wants it.I am just saying make provision for this.

[madshi]

We could dimension pametersData to a higher multiple of PAGE_SIZE just to be safe... For example: [4*PAGE_SIZE-sizeof(h3dlut)]
Ok, then let's do that.

It has also the advantage that we don't need to calculate the offset of the 3dlut array, anymore. We can directly access it through "f3dlut->lut" then.

[yesgrey]

What happens if parametersData is longer than "PAGE_SIZE-sizeof(h3dlut)"? What happens if parametersData is only 100 bytes? How does an application know that it's only 100 bytes and not "PAGE_SIZE-sizeof(h3dlut)"?
I was thinking in copying to parametersData exactly the same strings I read from the input file, with a '\n' character at the end of each parameter setting valid line, and an EOF at the end of it.
And where the actual LUT data starts can either be implicitly at the first page aligned offset after the parameter data or you could explicitly add an offset field/padding length to the header.
I see two ways of doing it:
1) dimension parametersData to a higher multiple of PAGE_SIZE just to be safe (I think with [4*PAGE_SIZE-sizeof(h3dlut)] we would be safe enough), and use parametersSize to indicate the portion of parametersData that has valid data.
2) Don't dimension parametersData to a fixed size. Set parametersSize to: n*PAGE_SIZE-sizeof(h3dlut); where n is the number of pages needed to contain the header and parametersData. Navigate through parametersData using the C characters like if it was a text file.

I believe 4*PAGE_SIZE-sizeof(h3dlut) will be big enough to hold any parameters file that could be created in the future, but we'll never know... and it would be strange to create a new file version just to increase the size of the parametersData field... but we can always do it.;)

There is still one thing left: compression. We haven't talked nothing about it, and with the size of the 3DLUT files it's important, because some users will need to have 3 or 4 in their disks... a 48MB uncompressed 3DLUT file can be reduced to a <1MB file...

[madshi]

There is still one thing left: compression. We haven't talked nothing about it, and with the size of the 3DLUT files it's important, because some users will need to have 3 or 4 in their disks... a 48MB uncompressed 3DLUT file can be reduced to a <1MB file...
Good idea. Reading in a compressed 1MB file and uncompressing it in RAM might also be faster than reading in a full 48MB file (if the decompression doesn't take too much time).

Should we use a custom method (e.g. difference to previous Lut index)? Or should we use e.g. some ZIP-like algorithm? In the latter case we should use not too strong compression, I'd say, so that decompression doesn't take too much time. It doesn't matter much if the 3dlut file is 1MB or 2MB, I think.

Should we *always* compress the lut file? I'd almost vote for "yes". But that would also mean that we could forget about aligning the file format for memory mapping purposes. Because all advantages of memory mapping would be gone if the lut data must be decompressed, anyway.

[leeperry]

There is still one thing left: compression. We haven't talked nothing about it, and with the size of the 3DLUT files it's important, because some users will need to have 3 or 4 in their disks... a 48MB uncompressed 3DLUT file can be reduced to a <1MB file...
you can compress a 48mb LUT to 1 mb w/ WinRAR....but the process is rather slow.
I've personally copied the LUT on a ramdisk(plus the RAR compressed ones), and I'm using shortcuts in the start menu to decompress the right one...I think you should let the end user take care of this, as decompressing in memory each time you open a video file would really slow things down..

and provided that you mostly watch US movies, all you need is SMPTE-C.....I seldom use EBU or HDTV, so that avoids wasting time decompressing LUT's each time.

start /b /high Z:\WinRAR\Rar.exe e -y Z:\pj_SMPTE-C.rar Z:\

and then :

rgb3dlut(lutfile="z:\lut.txt",threads=4)

a ramdisk is also good to avoid fragmentation and accelerate the media opening time..I think you can easily find some freeware ramdisk apps.

[madshi]

I think you should let the end user take care of this, as decompressing in memory each time you open a video file would really slow things down..
That depends a lot on the algorithm and compression ratio. I think a light ZIP compression should uncompress very fast. Of course it won't be as effective as RAR.

[yesgrey]

The problem with all this is that we will keep delaying the release of new versions of rgb3dlut and cr3dlut.
I think it might be a good idea releasing now a version and then update the file format specification after some more debate. Since currently there are only two applications using the 3DLUT format, we could update it in a week or two and continue calling it the version 1.0.
I know it will be a little annoying for the users, because they will have to run the cr3dlut file again to recreate their 3DLUT files, but they will always have to do it when cr3dlut add new features...

tritical,
my next release will use the following file specification. Are you ok with it?
typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int parametersSize;
} h3dlut;
/*struct
{
h3dlut header;
char parametersData[1];
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
} f3dlut; */


Please let continue the discussion about the file format.

[madshi]

Just did a small test: Loading an 48MB 3dlut file from harddisk, while my PC was a bit busy took 844ms. Unzipping a 3dlut file costs 984ms on my (not very fast) PC. However, the zipping method I chose was *very* light. It only brought the 48MB file down to roughly half its size. Still, I think if we choose the right compression method, uncompression at realtime doesn't have to be much of a penalty, especially since CPUs are getting faster at a quicker pace compared to harddisks.

But my test was only quick and dirty and may be flawed. Some more tests may be in order...

The problem with all this is that we will keep delaying the release of new versions of rgb3dlut and cr3dlut.
I think it might be a good idea releasing now a version and then update the file format specification after some more debate. Since currently there are only two applications using the 3DLUT format, we could update it in a week or two and continue calling it the version 1.0.
Sure, why not. I'd suggest to set fileversion to 0 for now, and then move it to 1 when the final file format is finally completed.

[yesgrey]

Sure, why not. I'd suggest to set fileversion to 0 for now, and then move it to 1 when the final file format is finally completed.
Ok, will set it to 0.
I think we are all very curious to see if the dithering is worth it or not...:D

[tritical]

my next release will use the following file specification. Are you ok with it?
What is the final verdict on calculating/indicating the amount of padding between parametersData and the actual lut values?

My suggestion is to keep everything the way it is, but state that the lut values start at the next multiple of PAGE_SIZE after the end of the parametersData array. Then all we have to do is add a definition for PAGE_SIZE to the specification.

[73ChargerFan]

How about using the 7-zip lzma sdk (http://www.7-zip.org/sdk.html) for compression? It is public domain, compressed a 48mb 3dlut file to 4.6mb & decompressed in less than 1 second (q6600 2.4 ghz).


* Compression speed: 2 MB/s on 2 GHz dual-core CPU.
* Decompression speed: 20-30 MB/s on 2 GHz Intel Core2 or AMD Athlon 64.
* Small memory requirements for decompression: 8-32 KB + DictionarySize
* Small code size for decompression: 2-8 KB (depending on speed optimizations)

Add a 10 character field to the header for the compression format (or none) of the data. And don't compress the header.

Loading 5mb is much better than 48mb.

[leeperry]

well, if you can still allow uncompressed LUT's I'll be a happy camper :o

[madshi]

What is the final verdict on calculating/indicating the amount of padding between parametersData and the actual lut values?

My suggestion is to keep everything the way it is, but state that the lut values start at the next multiple of PAGE_SIZE after the end of the parametersData array. Then all we have to do is add a definition for PAGE_SIZE to the specification.
I would be fine with that. But what is your opinion about compressing the lut data in the file? If we compress it then aligning the lut array to a page boundary doesn't make much sense, anymore.

[IanB]

If you are keen on compression, have a look at the source code for TCPSource(), sh0dan has done quite a bit research on various high performance compression modules.

Also the golden rule with compression is "know they enemy".

Any organised pre-processing that can be applied to data to lower the entropy of the stream going into the compression library will be a big win.

I suspect a predict-left style pre-processing would yield a very small population of delta values, i.e. -2, -1, 0, 1 & 2. This would compress extremely well with even trivial engines.

[yesgrey]

My suggestion is to keep everything the way it is, but state that the lut values start at the next multiple of PAGE_SIZE after the end of the parametersData array. Then all we have to do is add a definition for PAGE_SIZE to the specification.
If we don't go with the compression, this is the way I think it should be.
Do you want to consider the compression?

[tritical]

I don't particularly care one way or the other on compression, but if it is added I like 73ChargerFan's suggestion of adding a field to the header indicating what type or none. Then have the compressed or uncompressed lut values start at the next PAGE_SIZE multiple after the header and parameterData array. For now we could just support 'none' for compression to get everything going, and then discuss about what type(s), pre-processing, and other details in the future.

[madshi]

If you are keen on compression, have a look at the source code for TCPSource(), sh0dan has done quite a bit research on various high performance compression modules.

Also the golden rule with compression is "know they enemy".

Any organised pre-processing that can be applied to data to lower the entropy of the stream going into the compression library will be a big win.

I suspect a predict-left style pre-processing would yield a very small population of delta values, i.e. -2, -1, 0, 1 & 2. This would compress extremely well with even trivial engines.
Good thinking. A little pre-processing and then a fast compression algorithm should be a good solution. I'm using a similar approach in one of my applications (not audio/video data related, though).

I don't particularly care one way or the other on compression, but if it is added I like 73ChargerFan's suggestion of adding a field to the header indicating what type or none. Then have the compressed or uncompressed lut values start at the next PAGE_SIZE multiple after the header and parameterData array. For now we could just support 'none' for compression to get everything going, and then discuss about what type(s), pre-processing, and other details in the future.
Sounds good to me.

So one more field in the header. E.g. "compressionMethod". Set to 0 means "no compression". I don't care in which position in the header the new field is added. Just add it where you see fit.

Nice, I think we're finally done... :)

[yesgrey]

Here is the updated 3DLUT file format specification:
// 3D LUT file specification (fileVersion=1):
typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int parametersSize;
int lutCompression;
int reserved;
} h3dlut;
enum {PAGE_SIZE = 8192};
struct
{
h3dlut header;
char parametersData[1];
// The next element should start on the next PAGE_SIZE multiple to allow MMAPing the raw data
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
// char padding[1]; // entire file length should be an exact multiple of PAGE_SIZE.
} f3dlut;
/*
// Header
signature - File signature; must be: '3DLUT'
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or R,G,B)
outputBitDepth - Output bit depth per component (The same for all components)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array parametersData
lutCompression - Indicates if lut.dataxx array is compressed and which method was used
0 - None (not compressed)
reserved - Reserved for future usage
// Parameters data
parametersData - array of char with size parametersSize that contains a copy of the
run parameters settings used for creating the file
// 3D LUT output data
lut.dataxx - array with size lutSize that contains the 3dlut output values.
The type used (lut.dat8 or lut.data16) is defined by the
outputBitDepth field:
- lut.data8 when outputBitDepth = 8
- lut.data16 when outputBitDepth = 16
The offset and size of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
lutSize = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(Size in bytes:
- lutSize*1, when outputBitDepth = 8
- lutSize*2, when outputBitDepth = 16)
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/

I have named the new variable lutCompression to show that only the LUT data is compressed. The parametersData will always be very small, so no need to compress it, and this way we could always edit the 3DLUT file in a text editor and see the parameter settings used for its creation.
I have added again one reserved entry, in case we would need it.
Should we always pad the file length to a PAGE_SIZE multiple, or only when it's not compressed?
The parametersSize, should indicate the size of parametersData that contains valid data, or should also include the padding to reach the PAGE_SIZE multiple before the LUT data?

[madshi]

Here we go again... ;)

Things got more complicated because of the page aligning. So maybe we should reconsider the separation of header and file structures? Because using "sizeof(h3dlut)" doesn't make any sense, anymore, when we use page aligning.

So how about this?

typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int lutFileOffset;
int lutCompressionMethod;
int lutCompressedSize;
int lutUncompressedSize;
int reserved;
int parametersSize;
char parametersData[1];
} header3dlut;

typedef union
{
unsigned char data8[1];
unsigned short data16[1];
} data3dlut;
Suggested changes:

(1) A new field "lutFileOffset" would make it easy to find the lut array in the file.
(2) The new fields "lutCompressedSize" and "lutUncompressedSize" are simply helpers for memory allocation.
(3) Moved "parametersData" to the header cause "parametersData" and "lut.dataXX" are separated now due to page aligning.
(4) The 2nd structure shouldn't contain the header, anymore, cause due to page alignment the lut data is now separated from the header.
(5) Doing (3) and (4) leaves the 2nd structure with only the lut array, so we can simplify that structure.
(6) Moved "reserved" up, so that "parametersSize" and "parametersData" are next to each other.
(6) Renamed "lutCompression" to "lutCompressionMethod" for better understanding.

Comments?

[yesgrey]

So maybe we should reconsider the separation of header and file structures? Because using "sizeof(h3dlut)" doesn't make any sense, anymore, when we use page aligning.
If parametersSize is set to: n*PAGE_SIZE-sizeof(h3dlut); where n is the number of pages needed to contain the header and parametersData, then "sizeof(h3dlut)" will still make sense.
I am ok with your suggestions, but the previous was good because it had very clear all the file structure.
Let's hear what other people think about it.
Both me and tritical need to know how to write/read the 3DLUT files...

[madshi]

If parametersSize is set to: n*PAGE_SIZE-sizeof(h3dlut); where n is the number of pages needed to contain the header and parametersData, then "sizeof(h3dlut)" will still make sense.
Can you define a structure with a dynamic "n" element in C(++) and then use "sizeof"? In Delphi that is not possible.

the previous was good because it had very clear all the file structure.
We could also integrate everything into one big structure with a big comment in front of the "lut data" element (page alignment). That would also make the file structure clear. The only reason to split the structure into two parts was to make "sizeof" possible.

[yesgrey]

Can you define a structure with a dynamic "n" element in C(++) and then use "sizeof"? In Delphi that is not possible.

No, but I wasn't thinking in putting the dynamic "n" on it.
The paging of the file structure does not bring anything new. The parametersSize was already not fixed, the only difference is that before it was only the size of the array that contained a copy of the input file, and now we want to put in there some more null bytes just to reach a PAGE_SIZE multiple.
The only reason to split the structure into two parts was to make "sizeof" possible.
But the sizeof will not be very friendly, it will include the first character of parametersData...

To be honest, this is not very important to me, because whichever model we decide, I cannot make it work completelly in C.:o
I always have to replace the:
char parametersData[1];
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;

with:
char parametersData[16384];
union
{
unsigned char *data8;
unsigned short *data16;
} lut;

Perhaps my C skills are short in handling this kind of stuff, so feel free to specify it the way you feel more confortable with...;)

[madshi]

The paging of the file structure does not bring anything new.
Yes, it does. We were planning to locate the lut data by using the following formula:

sizeof(header) + parametersSize
And that "sizeof" in that formula was the one and only reason why we used a separate structure for the header. We cannot use that formula anymore now, because "parametersSize" does not include the padding to page size. At least I don't think it does/should.

Now doing some weird math to find the beginning of the lut data is not to my liking, that's why I suggested adding a "lutFileOffset" field. But then having such an offset field in the header means that we don't need "sizeof(header)", anymore. And that means we don't need a separate structure for the header, anymore, either. So I'd say: Let's put all fields in one structure, where the lut data union is the last element.

typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int lutFileOffset;
int lutCompressionMethod;
int lutCompressedSize;
int lutUncompressedSize;
int reserved;
int parametersSize;
char parametersData[1];

// the next element should start on the next PAGE_SIZE multiple
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
} file3dlut;

Just my opinion, of course. What do tritical and IanB think?

[73ChargerFan]

and this way we could always edit the 3DLUT file in a text editor and see the parameter settings used for its creation.Could the parameter settings field be initialized to spaces (as opposed to random data) to make it easy to see view in a text editor? That would also visually differentiate it from the data.

[yesgrey]

We cannot use that formula anymore now, because "parametersSize" does not include the padding to page size. At least I don't think it does/should.
Now doing some weird math to find the beginning of the lut data is not to my liking, that's why I suggested adding a "lutFileOffset" field.
I also agree with this. What I said was considering that "parametersSize" included the padding to page size, and adding a C 'EOF' character at the end of the valid "parametersData" data. If you prefer that "parametersSize" only refers to the size without the padding, for me it's ok. In fact, it would be better if someone wants to show it (no need to go through the array looking for the 'EOF' character).;)
So I'd say: Let's put all fields in one structure, where the lut data union is the last element.

It looks good to me.:)

Could the parameter settings field be initialized to spaces (as opposed to random data) to make it easy to see view in a text editor? That would also visually differentiate it from the data.
From the reply above, the "parametersData" will only contain the parameters settings. We could set all the padding bytes until the PAGE_SIZE to the character space to make it look cleaner, but I think that setting it to zero would be preferable.
By the way, what do you prefer in "parametersData":
1) a clean copy of the parameters settings with only the valid lines.
2) a clean copy of the parameters settings with the valid lines and the comment lines (lines started by '#')
3) an exact copy of the input file.

[madshi]

By the way, what do you prefer in "parametersData":
1) a clean copy of the parameters settings with only the valid lines.
2) a clean copy of the parameters settings with the valid lines and the comment lines (lines started by '#')
3) an exact copy of the input file.
I've vote for (3) because I'm thinking that if an application has the ability to recreate missing 3dlut files on the fly by using cr3dlut, such an application might also want to double check whether an existing file matches exactly the needed parameters. In order to do that check, the application would create the parameter settings string in RAM and then compare that to what is stored in the 3dlut file's "parametersData". If it matches, the file is ok. If it doesn't match, the file needs to be recreated.

[tritical]

What is the point of the union and

char parametersData[1];

line? Why not just make it:

struct H3DLUT
{
char signature[8]; // File signature; must be: '3DLUT'
int fileVersion; // File format version number
int inputBitDepth[3]; // Input bit depth per component (Y,Cb,Cr or R,G,B)
int outputBitDepth; // Output bit depth for all components (valid values are 8 and 16)
int inputColorEncoding; // Input color encoding standard
int outputColorEncoding; // Output color encoding standard
char programName[32]; // Name of the program that created the file
long int programVersion; // Version of the program that created the file
int parametersSize; // Size in bytes of the array parametersData
int lutFileOffset; // number of bytes between end of parametersData and lut data
int lutCompressionMethod;// type of compression used if any (0 = none, ...)
int lutCompressedSize; // if compressed, size of compressed lut values in bytes
int lutUncompressedSize; // uncompressed size of lut values in bytes
int reserved; // reserved for future use
// This header if followed by the char array 'parametersData', of length 'parametersSize'.
// The lut data starts 'lutFileOffset' bytes after the end of the 'parametersData' array,
// and that pos must be on a 16384 byte boundary if the data is uncompressed.
};

Now in the uncompressed case we could read it using code like (no error checking):

FILE *f = fopen("lut","rb");
H3DLUT h3dlut;
fread(&h3dlut,sizeof(H3DLUT),1,f);
fseek(f,h3d.parametersSize+h3d.lutFileOffset,SEEK_CUR);
lut3d = (unsigned char*)_aligned_malloc(h3d.lutUncompressedSize,16);
fread(lut3d,1,h3d.lutUncompressedSize,f);
fclose(f);

[madshi]

What is the point of the union and

char parametersData[1];

line?
If you read in (or memory map in) the whole file at once, you can access parametersData directly through the structure this way. It's not necessary, but I think it's useful.

Having the union in the structure, however, only serves the purpose to let everyone know that the lut data is supposed to come "somewhere" after the parametersData, with nothing else (of importance) in between. It's just meant as a source code level explanation of how the file is structured. I wouldn't mind having this removed from the structure, but personally I like it in there just as an explanation.

Why not just make it
I could live with that.

fseek(f,h3d.parametersSize+h3d.lutFileOffset,SEEK_CUR);
I intended "lutFileOffset" to work from the beginning of the file ("fseek(f,h3d.lutFileOffset,SEEK_BEGIN)"). That's more comfortable when working with a memory map.

[yesgrey]

Why not just make it...
For me it's ok too.
Where the lutOffset starts is indiferent to me. Just let me know what you prefer so I could write the correct value.

The only change I suggest is always filling the lutCompressed Size. If it's compressed, the size is different from the lutUncompressedSize. If it's uncompressed, both have the same value.

[yesgrey]

I've vote for (3) ...
In order to do that check, the application would create the parameter settings string in RAM and then compare that to what is stored in the 3dlut file's "parametersData". If it matches, the file is ok. If it doesn't match, the file needs to be recreated.

For that, the 1) is the better option, not the 2) nor the 3).
I already coded the 1). Then I realized that could be interesting allowing some comments to help a user identifying a 3DLUT if he opens it in a text editor, but for that, maybe it's better adding a new field to the header to contain a description of the 3DLUT file (for example a "char fileDescription[64]"), and keep in the parametersData only the parameters and respective settings. The 3) could contain a lot of trash, that's why I prefer the option 1), which retains only the essential.

[madshi]

For that, the 1) is the better option, not the 2) nor the 3).
I disagree.

Look, if a program creates its 3dlut files automatically by calling cr3dlut then this program will have a function like "CreateParameterSettingsString" which it will then feed to cr3dlut. Now if this program wants to check whether a 3dlut file on harddisk has the expected format, it will call "CreateParametersSettingsString" again and compare the result to the "parametersData" string in the 3dlut file. If the two strings are bit perfectly identical, the 3dlut file is "good", otherwise it's "bad". No need to parse the actual "parametersData" string this way. If you do (1) then the program is forced to write complicated code to parse "parametersData" and compare every single parameter to make sure that the 3dlut data was created with the correct parameters...

Why not simply storing the full input file as it is into parametersData? That should be the most simple thing to do. It would also allow the user to add comments to the file. E.g. the user could write "this 3dlut is meant to be used only for the movie XYZ, because it has the colors encoded in the wrong way". If you stored the settings file unchanged and if you then open the 3dlut file in a (hex) editor, you could still see that comment.

Then I realized that could be interesting allowing some comments to help a user identifying a 3DLUT if he opens it in a text editor, but for that, maybe it's better adding a new field to the header to contain a description of the 3DLUT file (for example a "char fileDescription[64]"), and keep in the parametersData only the parameters and respective settings.
I don't think we need a description in the file. IMHO the readme shipping with cr3dlut should be good enough.

[yesgrey]

If you do (1) then the program is forced to write complicated code to parse "parametersData" and compare every single parameter to make sure that the 3dlut data was created with the correct parameters...

The code is already created in cr3dlut code, but ok, I will do 3). If that application appears probably no one will be messing with the input files anymore, so no problem of breaking them.;)
It would also allow the user to add comments to the file. E.g. the user could write "this 3dlut is meant to be used only for the movie XYZ, because it has the colors encoded in the wrong way". If you stored the settings file unchanged and if you then open the 3dlut file in a (hex) editor, you could still see that comment.

I don't think we need a description in the file. IMHO the readme shipping with cr3dlut should be good enough.
The description I was talking about was like the example you gave above "this 3dlut is meant to be used only for the movie XYZ, because it has the colors encoded in the wrong way". So, I will let the description being put in the input file as a comment that will be kept in the parametersData array.

[madshi]

Thanks!

So what is missing to finally complete the spec? "lutFileOffset"? We have two different interpretations of what i means.

tritical suggests to interpret it as "number of bytes between end of parametersData and lut data". The code then looks like this:
// sequential file reading:
fseek(f,h3d.parametersSize+h3d.lutFileOffset,SEEK_CUR);

// memory mapping:
lutPtr = (PVOID) ((ULONG_PTR) lutMemoryMap + sizeof(H3DLUT) + lutMemoryMap->parametersSize + lutMemoryMap->lutFileOffset);

I suggest to interpret it as "number of bytes between beginning of file and lut data". Code:
// sequential file reading:
fseek(f,h3d.lutFileOffset,SEEK_BEGIN);

// memory mapping:
lutPtr = (PVOID) ((ULONG_PTR) lutMemoryMap + lutMemoryMap->lutFileOffset);
:)

[yesgrey]

So what is missing to finally complete the spec? "lutFileOffset"?
Yes.
tritical's interpretation is more logical. Since lutOffset appears right after parametersSize, it's more logical thinking that it's just the space between parametersData end and lutData begin.
But your interpretation is more programmer friendly...;)
For me is indifferent, I'm only creating and writing the file.:)

[yesgrey]

if a program creates its 3dlut files automatically by calling cr3dlut then this program will...
Are you considering writing that program, or are you only conjecturing...;)

[madshi]

Are you considering writing that program, or are you only conjecturing...;)
What? Huh? Have you asked me something? I've some strange noise in my ears, so I couldn't hear you. <plugging fingers into my ears> "La la la la la"...

[cyberbeing]

I have somehow confused myself when I realized cr3dlut has sRGB as the display gamma for the _PC example input files. I found this odd considering almost no one would have their display at a gamma of 2.4.

For a CRT monitor calibrated to a gamma of 2.2 with something like ProfileMaker, EyeOne Match, or ColorEyes, which output/display gamma should I be using?

sRGB (2.4 gamma?):
12.92,0.055,1.0/2.4,0.0031308,0.04045

sRGB modified with a gamma of 2.2 instead of 2.4:
12.92,0.055,1.0/2.2,0.0031308,0.04045

BT.709/BT.601:
4.5,0.099,1.0/(1.0/0.45),0.018,0.081

General 2.2:
1.0,0.0,1.0/2.2,0.0,0.0
or
1.0,0.0,0.45,0.0,0.0

Is there any test I could do to determine which of the above is the correct one to use?

@yesgrey3
Does cr3dlut only accept decimal values or do fractions also work? What is the maximum number of decimal places it will accept?
Also is Display_gamma 9 lv av pv thresh OR Display_gamma 9 lv av pv thresh1 thresh2 correct?
I ask since you have the example as Display_gamma 9 1.0 0.0 0.45 0.0 without a second thresh value in the readme.

[yesgrey]

"La la la la la"...
A new project always makes us sing...:D

I have somehow confused myself when I realized cr3dlut has sRGB as the display gamma for the _PC example input files. I found this odd considering almost no one would have their display at a gamma of 2.4.
When you look to a transfer function you cannot consider only the gamma value (2.4 in sRGB). In reality, the sRGB transfer function, due to the small linear part at the black levels, has a curvature of almost a pure 1/2.2 power curve. So yes, if your monitor was calibrated to a gamma of 2.2, you could use the sRGB transfer function, that was the reason I have selected it.
You can also use the pure power curve with a 2.2 gamma. Try it, and see what you prefer. There is not an exact value for the gamma; it will depend always of the viewing lighting conditions...
I never liked very much watching stuff in my CRT monitor, always too dark. Since I started using it with rgb3dlut and output set to sRGB, I am liking what I see...:)

Does cr3dlut only accept decimal values or do fractions also work? What is the maximum number of decimal places it will accept?
Also is Display_gamma 9 lv av pv thresh
Only decimals, the number you want, but since it read it as a double, you gain nothing in putting more than 14 decimal places.;)
The gamma values are the specifyed in the ReadMe. I have not followed tritical's usage of thresh1 and thresh2, because thresh2 can be calculated from thresh1. So, I only need one. My thresh is the same as tritical's thresh1.

[yesgrey]

About the 'lutCompressedSize' and 'lutUncompressedSize', I think we should use them like this:
lutCompressedSize - size of the lutData array inside the 3DLUT file, whether compressed or not.
lutUncompressedSize - size of the lutData array when loaded in memory for usage, outside of the 3DLUT file. It will always be uncompressed.
Do you agree with this, or should we specify the 'lutCompressedSize' only when compressed, and set it to zero when uncompressed?

[madshi]

A new project always makes us sing...:D
What new project? :p

About the 'lutCompressedSize' and 'lutUncompressedSize', I think we should use them like this:
lutCompressedSize - size of the lutData array inside the 3DLUT file, whether compressed or not.
lutUncompressedSize - size of the lutData array when loaded in memory for usage, outside of the 3DLUT file. It will always be uncompressed.
Do you agree with this
Yes, absolutely.

[yesgrey]

What new project?
My cr3dlut, of course.:)
"La la la la la!..."

[tritical]

The offset being from the beginning or after parametersData doesn't matter to me. Using the compressed/uncompressed size for in the lut and in memory is fine also.

[yesgrey]

Here is another iteration of the file format. It's tritical's suggestion with a few comment lines for describing with a little more detail the lutData array:
// 3D LUT file specification (fileVersion=1):
enum {PAGE_SIZE = 16384};
struct H3DLUT
{
char signature[8]; // file signature; must be: '3DLUT'
int fileVersion; // file format version number
int inputBitDepth[3]; // input bit depth per component (Y,Cb,Cr or R,G,B)
int outputBitDepth; // output bit depth for all components (valid values are 8 and 16)
int inputColorEncoding; // input color encoding standard
int outputColorEncoding; // output color encoding standard
char programName[24]; // name of the program that created the file
long int programVersion; // version of the program that created the file
int parametersSize; // size in bytes of the array parametersData
int lutFileOffset; // number of bytes between the beginning of the file and array lutData
int lutCompressionMethod;// type of compression used if any (0 = none, ...)
int lutCompressedSize; // size in bytes of the array lutData inside the 3DLUT file, whether compressed or not
int lutUncompressedSize; // size in bytes of the array lutData when in memory for usage, outside of the 3DLUT file
int reserved; // reserved for future use
// This header is followed by the char array 'parametersData', of length 'parametersSize',
// and by the array 'lutData', of length 'lutCompressedSize'.
// The array 'lutData' starts 'lutFileOffset' bytes after the beginning of the file,
// and that pos must be on a 16384 byte boundary if 'lutData' is uncompressed.
};
/*
parametersData - char array with size parametersSize that contains an exact copy of the
input file with the parameters settings used for creating the 3DLUT file
lutData - array with size lutSizeUncompressed that contains the 3dlut output values.
The type used depends on the 'outputBitDepth' field:
- unsigned char, if outputBitDepth = 8
- unsigned short, if outputBitDepth = 16
The offset inside the array is calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
The 'lutUncompressedSize' of the array is calculated as:
lutDim = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
lutUncompressedSize = lutDim*1, if outputBitDepth = 8
lutUncompressedSize = lutDim*2, if outputBitDepth = 16
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/

I have changed 'programName' size to [24]. With [32] the struct alignment created an empty space of 4 bytes, due to the long int programVersion. Another way of solving this problem is keeping the [32] size and adding a "#pragma pack(4)" before the struct definition. I think the former is the best; [24] would be enough for a program's name...
Another thing to consider is the long int type. In some compilers, the long int is treated as an int, only 32bit (with mine, for example). I have to change it for long long to get a 64bit integer. Should I change it also in the specification above, or do your compilers work ok with it?
With the new meaning of 'lutCompressedSize' and 'lutUncompressedSize', maybe the names 'lutPackingMethod' 'lutPackedSize' and 'lutUnpackedSize' are more meaningfull...

[cyberbeing]

When you look to a transfer function you cannot consider only the gamma value (2.4 in sRGB). In reality, the sRGB transfer function, due to the small linear part at the black levels, has a curvature of almost a pure 1/2.2 power curve. So yes, if your monitor was calibrated to a gamma of 2.2, you could use the sRGB transfer function, that was the reason I have selected it.
You can also use the pure power curve with a 2.2 gamma. Try it, and see what you prefer. There is not an exact value for the gamma; it will depend always of the viewing lighting conditions...
I never liked very much watching stuff in my CRT monitor, always too dark. Since I started using it with rgb3dlut and output set to sRGB, I am liking what I see...:)

Aren't you just making things more inaccurate by skewing things from the source gamma? The difference between setting the display gamma to source gamma and setting the display gamma to 2.2 or sRGB is very extreme. Everything has much less contrast.

I never liked very much watching stuff in my CRT monitor, always too dark. Since I started using it with rgb3dlut and output set to sRGB, I am liking what I see...:)

This sounds like more of an issue with your CRT monitor then anything else. Lets forget about CRTs for a second, since you seem to think yours should be treated differently.

If you had a LCD/projector/Plasma calibrated to a gamma of 2.2 which setting for display gamma should you be using to match your calibrated gamma?

I would also still like to see this question answered:

When you calibrate your display to a gamma of 2.2, which gamma curve do apps like Profile Maker, Eye One Match, and Color Eyes calibrate it to? I've never seen an option which can be set in any of these apps, so it seems the question needs to be asked.

2.2 Power Curve?
sRGB Curve adapted to 2.2?
BT.601/BT.709 Curve?
Random 2.2 Curve that doesn't match anything standard?

Are their calibration apps which do let you set a specific type of gamma curve?

Is there any test which can be done to determine which type of gamma curve your calibrated display matches closest? Is there a way to generate your own lv av pv thresh values to match how your TV/Monitor displays gamma exactly?

The reason this came up is because using the sRGB setting my shadows were gray (looked identical to viewing PC levels without conversion) and the whole video was brighter with less contrast overall. The same goes for the 2.2 power curve.

My main concern is if I am actually doing harm rather then good by setting the display gamma any different from the source gamma.

Since the difference is so extreme, doing one or the other must be very wrong. Which one is it? Can it be proven?

[leeperry]

When you calibrate your display to a gamma of 2.2, which gamma curve do apps like Profile Maker, Eye One Match, and Color Eyes calibrate it to?
you can use Color.HCFR afterwards to spy on your ICC app.
I personally use ARGYLLCMS w/ 2.1 gamma on my CRT, then I always doublecheck with Color.HCFR

I've done numerous tests, it looks better on both my 2.1 CRT/2.2 DLP pj to use the 1.0/0.45 ddcc gamma curve(same as the PS script)

[madshi]

Here is another iteration of the file format.
Looks fine to me.

programName[24] is fine.

"long long" for programVersion would be fine with me. Another option would be "word programVersion[4]". Use whatever you prefer.

maybe the names 'lutPackingMethod' 'lutPackedSize' and 'lutUnpackedSize' are more meaningfull...
For programmers "packed" is usually used as in "#pragma pack". So I'd prefer "compressed" to avoid confusion...

One final thought: Now we have "lutFileOffset". Maybe we should also use "parametersFileOffset"? I know it's the same as "sizeof(H3DLUT)". But having an extra field "parametersFileOffset" would mean that we would be more flexible. E.g. we could add more fields to the header without breaking anything. Because of that we could also drop "reserved" from the header, because we could later readd as many new fields as we like, simply by adjusting parametersFileOffset and lutFileOffset accordingly...

[IanB]

I have changed 'programName' size to [24]. With [32] the struct alignment created an empty space of 4 bytes, due to the long int programVersion. Another way of solving this problem is keeping the [32] size and adding a "#pragma pack(4)" before the struct definition. I think the former is the best; [24] would be enough for a program's name...
Another thing to consider is the long int type. In some compilers, the long int is treated as an int, only 32bit (with mine, for example). I have to change it for long long to get a 64bit integer. Should I change it also in the specification above, or do your compilers work ok with it?
With the new meaning of 'lutCompressedSize' and 'lutUncompressedSize', maybe the names 'lutPackingMethod' 'lutPackedSize' and 'lutUnpackedSize' are more meaningfull...Seeing this is a file header, I would suggest you explicitly align all the elements independent of any compiler.You could just shuffle the order a little avoid the problem...

Hint :- Put big things early, put small things late ...

[yesgrey]

Here is another iteration of the 3DLUT specification:
/* 3D LUT file specification (fileVersion=1):
enum {PAGE_SIZE = 16384};
struct H3DLUT
{
char signature[8]; // file signature; must be: '3DLUT'
int fileVersion; // file format version number
char programName[32]; // name of the program that created the file
unsigned short programVersion[4]; // version of the program that created the file (a.b.c.d -> [3].[2].[1].[0])
int inputBitDepth[3]; // input bit depth per component (Y,Cb,Cr or R,G,B -> [0],[1],[2])
int outputBitDepth; // output bit depth for all components (valid values are 8 and 16)
int inputColorEncoding; // input color encoding standard
int outputColorEncoding; // output color encoding standard
int parametersFileOffset; // number of bytes between the beginning of the file and array parametersData
int parametersSize; // size in bytes of the array parametersData
int lutFileOffset; // number of bytes between the beginning of the file and array lutData
int lutCompressionMethod; // type of compression used if any (0 = none, ...)
int lutCompressedSize; // size in bytes of the array lutData inside the 3DLUT file, whether compressed or not
int lutUncompressedSize; // size in bytes of the array lutData when in memory for usage (outside of the 3DLUT file)
// This header is followed by the char array 'parametersData', of length 'parametersSize',
// and by the array 'lutData', of length 'lutCompressedSize'.
// The array 'parametersData' starts 'parametersFileOffset' bytes after the beginning of the file.
// The array 'lutData' starts 'lutFileOffset' bytes after the beginning of the file,
// and that pos must be on a 16384 byte boundary if 'lutData' is uncompressed.
};

parametersData - char array with size parametersSize that contains an exact copy of the
input file with the parameters settings used for creating the 3DLUT file
lutData - array with size lutSizeUncompressed that contains the 3dlut output values.
The type used depends on the 'outputBitDepth' field:
- unsigned char, if outputBitDepth = 8
- unsigned short, if outputBitDepth = 16
The offset inside the array is calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
The 'lutUncompressedSize' of the array is calculated as:
lutDim = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
lutUncompressedSize = lutDim*1, if outputBitDepth = 8
lutUncompressedSize = lutDim*2, if outputBitDepth = 16
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte)
*/
I have moved up 'programName' and 'programVersion'. I think it's preferable having them before the file format variables instead of in the middle, as they were. I also have followed madshi suggestion and changed 'programVersion' from long int to unsined short [4]; this also solves the alignement problem, so I have returned to 'programName' size as [32].

Should we also shift positions of these?
From:
int inputBitDepth[3];
int outputBitDepth;
int inputColorEncoding;
int outputColorEncoding;
To:
int inputBitDepth[3];
int inputColorEncoding;
int outputBitDepth;
int outputColorEncoding;
It means output following the input...
You could just shuffle the order a little avoid the problem...
Hint :- Put big things early, put small things late ...
The problem is the 'fileVersion' variable, and I think the right place for it is right after the signature. We can consider that 'programName' and 'programVersion' will never change, and then put 'fileVersion' after them...
Any thoughts?...

[madshi]

Should we also shift positions of these?
From:
int inputBitDepth[3];
int outputBitDepth;
int inputColorEncoding;
int outputColorEncoding;
To:
int inputBitDepth[3];
int inputColorEncoding;
int outputBitDepth;
int outputColorEncoding;
It means output following the input...
Doesn't matter to me.

the problem is the 'fileVersion' variable, and I think the right place for it is right after the signature. We can consider that 'programName' and 'programVersion' will never change, and then put 'fileVersion' after them...
"programVersion" should change with every new cr3dlut build! Personally, I'd prefer keeping fileVersion directly after the signature because (just like you) I think that is logically the right place.

Can't we simply define the structure with "#pragma pack (1)" and be done with it? Or are there some C++ compilers which don't support byte packing?

Alternatively: Would shorting the signature to 4 byte ("3DLT") help?

[yesgrey]

"programVersion" should change with every new cr3dlut build!
I was talking about the format, not the value...;)
Can't we simply define the structure with "#pragma pack (1)" and be done with it? Or are there some C++ compilers which don't support byte packing?
"#pragma pack (4)" would be good enough...
Alternatively: Would shorting the signature to 4 byte ("3DLT") help?
Yes, that would help, and we could return to set programVersion to long long...
IanB, what do you think? What's your suggestion to make the alignment compiler independent?

[yesgrey]

@cyberbeing,
I will look at your questions a little later, when I start working in the custom gamma curves completelly defined by the users, to mimic their display's gamma response. That will be the next step, after completing this 3DLUT file specification thing.;)

[IanB]

IanB, what do you think? What's your suggestion to make the alignment compiler independent?Assume the size of each element is also it's alignment base for the element and order the elements such that the whole structure is intrinsically aligned.

Don't stuff with any "#pragma pack ..." like options, you just don't know how any given compiler will behave. It also make it easier for others to port the structure to other languages.

The short programVersion[4] trick works for the current windows compilers and probably will work everywhere, but thinking it might not someday costs nothing and hence coding defensively to avoid the issue means never having to say sorry.

[madshi]

Assume the size of each element is also it's alignment base for the element and order the elements such that the whole structure is intrinsically aligned.
That's fine as long as the reordering makes sense, logically. But I'm not a fan of using an illogical order just to make alignment work.

Don't stuff with any "#pragma pack ..." like options, you just don't know how any given compiler will behave. It also make it easier for others to port the structure to other languages.
I think any language and any compiler (which is any good) has some way to produce byte aligned structures. Because otherwise even calling simple win32 APIs wouldn't work. Sure, "#pragma pack" might not work with every compiler, but then some other option will do the trick. So I don't really see the problem. If the structure is clearly specified as being byte aligned, then I think that's all anybody can ask for.

Of course if it's possible to find a good and logical order of the fields which also happens to be good for alignment, then I'm all for it. But please let us not shuffle the fields to a "random" order just to avoid having to use a compiler option like "#pragma pack".

[IanB]

Of course if it's possible to find a good and logical order of the fields which also happens to be good for alignment, then I'm all for it. But please let us not shuffle the fields to a "random" order just to avoid having to use a compiler option like "#pragma pack".Well that is the challenge. And you are allowed to manually pad the structure. As I keep saying "Intrinsically aligned".

Anyway for a binary header logical order is somewhat meaningless, but rational ordering like good comments is good programming. Anything that prints the data should of course do so in a sensible order.

[yesgrey]

Probably this is a very dumb idea, but since the 'signature' field is a char array of 8, and we only use 5, why not use the last to indicate the file version? With a char we can have 127 different versions, I think it would be good enough... this way we can remove the fileVersion field...
We can simply set the fileVersion to be a long long, which would solve the alignment problem...

[madshi]

Anyway for a binary header logical order is somewhat meaningless, but rational ordering like good comments is good programming.
We have a different way of thinking here, it seems. But maybe we can find a perfect alignment for the 3dlut files without taking a hit on logical ordering. Then we could all be satisfied.

Probably this is a very dumb idea, but since the 'signature' field is a char array of 8, and we only use 5, why not use the last to indicate the file version? With a char we can have 127 different versions, I think it would be good enough... this way we can remove the fileVersion field...
We can simply set the fileVersion to be a long long, which would solve the alignment problem...
I don't like misusing the last byte of a field for another purpose. Using long long for fileVersion would be fine with me, though. But if it solves the alignment problem, I'd like using a 4 byte signature and a simple "int" for fileVersion even better.

Anyway, I'd love to finally get the spec completed. So if either using "long long fileVersion" or a 4 byte signature + 4 byte fileVersion solves all remaining problems, then please just use whatever you like more and let's finally call it finished... :)

[yesgrey]

So here it goes another iteration... hopefully the last one.;)
// 3DLUT file format specification (fileVersion=1):
enum {PAGE_SIZE = 16384};
struct H3DLUT
{
char signature[4]; // file signature; must be: '3DLT'
int fileVersion; // file format version number
char programName[32]; // name of the program that created the file
long long programVersion; // version number of the program that created the file
int inputBitDepth[3]; // input bit depth per component (Y,Cb,Cr or R,G,B)
int inputColorEncoding; // input color encoding standard
int outputBitDepth; // output bit depth for all components (valid values are 8 and 16)
int outputColorEncoding; // output color encoding standard
int parametersFileOffset; // number of bytes between the beginning of the file and array parametersData
int parametersSize; // size in bytes of the array parametersData
int lutFileOffset; // number of bytes between the beginning of the file and array lutData
int lutCompressionMethod; // type of compression used if any (0 = none, ...)
int lutCompressedSize; // size in bytes of the array lutData inside the file, whether compressed or not
int lutUncompressedSize; // true size in bytes of the array lutData when in memory for usage (outside the file)
// This header is followed by the char array 'parametersData', of length 'parametersSize',
// and by the array 'lutData', of length 'lutCompressedSize'.
// The array 'parametersData' starts 'parametersFileOffset' bytes after the beginning of the file.
// The array 'lutData' starts 'lutFileOffset' bytes after the beginning of the file.
// When creating a 3DLUT file, 'lutData' should be positioned on a 16384 byte boundary.
};
/*
parametersData - char array with size parametersSize that contains an exact copy of the
input file with the parameters settings used for creating the 3DLUT file
lutData - array with size lutSizeUncompressed that contains the 3dlut output values.
The type used depends on the 'outputBitDepth' field:
- unsigned char, if outputBitDepth = 8
- unsigned short, if outputBitDepth = 16
The offset inside the array is calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
The 'lutUncompressedSize' of the array is calculated as:
lutDim = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
lutUncompressedSize = lutDim*1, if outputBitDepth = 8
lutUncompressedSize = lutDim*2, if outputBitDepth = 16
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long long = 8 byte)
*/
I've preferred changing the signature to "3DLT" due to the previously discussed alignment problems. "3DLUT" is the name always used to refer to this type of LUTs, so, since we are specifying a file format, it's better changing the name slightly to avoid confusion when talking about the file format vs the "3D LUT" term. Also, since the name is 3D Lookup Table, "3DLT" are the real initials, so I agreed with madshi and also voted for "3DLT". With this change, I think we should also change the file extension to "*.3dlt".

Comments are welcome...

[madshi]

So here it goes another iteration... hopefully the last one.;)
For me it's perfect. No complaints, nor further improvement ideas about the structure itself. Just one minor suggestion about one of the comments:

// and that pos must be on a 16384 byte boundary if 'lutData' is uncompressed.
I'd change the comment slightly to make clear that this page alignment requirement is a requirement which must be met by programs *creating* a 3dlut file and not by programs reading such a file. So maybe:

"when creating a 3dlut file, you should position "lutData" on a 16384 byte boundary"

Also, since the name is 3D Lookup Table, "3DLT" are the real initials
Tricky... :)

so I agreed with madshi and also voted for "3DLT". With this change, I think we should also change the file extension to "*.3dlt".
In other words we rename the whole concept from "3D lut" to "3D Lookup Table"? I like that! And I agree that the file extension then must be "*.3dlt".

So when is cr3dlut - - ... - scratch that.

So when is cr3dlt version 2.0 coming out? :D

[yesgrey]

In other words we rename the whole concept from "3D lut" to "3D Lookup Table"?
In reality is almost the same, is "3D LookUp Table"... just sounds better a lut than a lt.:D
I think we should keep the whole concept as 3D LUT, and use 3DLT just as the file format designation...
So when is cr3dlut - - ... - scratch that.
So when is cr3dlt version 2.0 coming out? :D
I have not yet decided if I will change the program's name. My first thought was to do it, but since I think we should keep the concept as 3DLUT, cr3dlut still makes sense, as tritical's rgb3dlut. The only difference is that now our programs are creating/using a 3DLUT in the file format 3DLT.;)
Let's hear how tritical feels about this...
My program is already creating 8 and 16 bit 3DLT files, and I already send some to tritical, so he can test it with his program version. Now I will start working on the specification of the ColorEncoding variables. I want to create a more user friendly specification of it.;)

[yesgrey]

I'd change the comment slightly to make clear that this page alignment requirement is a requirement which must be met by programs *creating* a 3dlut file
I've edited the specification. See if it's ok to you.

[madshi]

In reality is almost the same, is "3D LookUp Table"...
Ooops, didn't know that... :)

I think we should keep the whole concept as 3D LUT, and use 3DLT just as the file format designation...
In that case I'd prefer using "*.3dlut" as file extension. Because if "cr3dlut" and "rgb3dlut" both keep using the full name "3dlut" then so should the file extension, I think. It would be strange if the tool "cr(eate)3dlut" would create a "3dlt" file!

(It doesn't matter that the "U" is missing in the file header "signature" field. No normal end user would ever care - or even notice.)

I've edited the specification. See if it's ok to you.
I'm wondering whether the first 2 comment lines are needed at all. I think without those it would still be clear. Maybe only these 3 lines?

// The array 'parametersData' starts 'parametersFileOffset' bytes after the beginning of the file.
// The array 'lutData' starts 'lutFileOffset' bytes after the beginning of the file.
// When creating a 3DLT file, you should position "lutData" on a 16384 byte boundary.
I'd like "When creating" to be in a new line. And I've removed the "if 'lutData' is uncompressed" because I think it wouldn't harm if "lutData" was always aligned, regardless of whether it's compressed or not. It would keep things simple to always align instead of only aligning under specific circumstances. What do you think?

[yesgrey]

In that case I'd prefer using "*.3dlut" as file extension...
(It doesn't matter that the "U" is missing in the file header "signature" field. No normal end user would ever care - or even notice.)
Ok, so I will revert the file format name to 3DLUT.
It will be similar to the BMP file format, which has a signature of "BM", and we all keep calling it BMP...;)

About your other suggestions, I agree with all, except removing of the two first comment lines - I think it's better keep them.
I will edit the specification accordingly.

[madshi]

Great!

Any objections from tritical or IanB about the latest revision of the file format?

[tritical]

yesgrey3, my program is working with the lut files you sent. Could you send me one with the new header format for testing? If you have your program to the point of generating rgb->yuv tables one of those would be good.

Btw, I've changed the name of my filter to 3dlut, but avisynth doesn't seem to like function names starting with numbers so it is now t3dlut().

[IanB]

Looks okay, I would put back the outer structure you had in earlier versions as a pseudo structure in the comments to help understand how the parametersData[], the page padding and lutData[] are organised. Use example pseudo numbers if you think it will help.

[yesgrey]

IanB,
I agree. I will update the spec accordingly.

[madshi]

Hey guys,

stumbled over this interesting article:

http://www.glennchan.info/articles/technical/chroma/chroma1.htm

As far as I understand, it's written for people who *down*sample chroma. But I still find it interesting. E.g. it seems that chroma upsampling (and probably also any upscaling) should be done in linear color space and not in gamma corrected color space!!

That brings me to the following conclusion: To get perfect quality, we should first remove gamma correction to move the image to linear color space. Then we should upsample chroma and scale the image (if needed). Finally we should do gamut & gamma correction and eventually convert to RGB or YCbCr.

So I guess that means cr3dlut should also support linear YCbCr input in addition to gamma corrected YCbCr and RGB input?

BTW, the article also explains why chroma placement can be "center" instead of "left" even though MPEG2 specifies is should be left.

Edit: It seems that only Y is gamma corrected, while Cb and Cr are linear? But then why does the article say that chroma subsampling should be done linear and not gamma corrected? I'm confused...

[yesgrey]

So I guess that means cr3dlut should also support linear YCbCr input in addition to gamma corrected YCbCr and RGB input?
It already supports it.;)
I will read the article tomorrow.

[tritical]

I think the author is saying he thinks you will get better results if you first convert gamma corrected RGB to linear RGB before converting to YCbCr (in that case you technically get "luminance" - Y instead of "luma" - Y') and downsampling. I highly doubt this is the case if you have to store the resulting Y with only 8-bits of precision. In almost all video/image processing, the conversion involves gamma-corrected RGB so you technically get Y'CbCr. Really it makes no difference in terms of chroma upsampling, because unless the person somehow converted to Y'CbCr, downsampled, and then undid gamma correction (which I'm not even sure how you would go about doing as the gamma correction we're talking about, performing the inverse of the transfer function of an rgb display, is really an RGB only concept) you'd want to do the upsampling in the same system (linear or non-linear) that the downsampling was performed in.

Now whether it is better to perform image processing operations in linear RGB instead of gamma corrected RGB is another question. The main thing you have to remember is that a person's perceptual response to luminance, or lightness, is non-linear (close to a 0.4 power function). So if you have limited precision (like 256 steps in 8-bit processing) in which to store intermediate values, you'll get much better results if you operate in a non-linear space close to a 0.4 power function. Otherwise, at the low end you have only a few samples covering a large perceptual difference range, and at the high end you have a lot of samples covering a small perceptual difference range. It just happens that most monitors have a transfer function roughly equal to a 2.2 power curve. So when you perform gamma correction to adjust for the monitor's transfer function by taking x^(1/2.2) you actually get pretty close to the 0.4 power function of human perception. Therefore, working with Y'CbCr (Y' derived from gamma corrected RGB) works pretty well. Of course, what you could do is have each operation undo the gamma correction (storing the result at higher precision), do its work in higher precision, and then reapply gamma before storing the result in limited precision.

I think 12 bits or more of precision, per color component, is required to avoid banding issues with linear rgb.

[madshi]

I think the author is saying he thinks you will get better results if you first convert gamma corrected RGB to linear RGB before converting to YCbCr (in that case you technically get "luminance" - Y instead of "luma" - Y') and downsampling. I highly doubt this is the case if you have to store the resulting Y with only 8-bits of precision.
Is it easily possible to convert YCbCr to Y'CbCr? Maybe what he was suggesting was to do "R'G'B' -> RGB -> YCbCr -> chroma subsampling -> Y'CbCr"? Don't know if that makes any sense. Of course if he really wanted to end up with YCbCr instead of Y'CbCr that would make no sense at all.

[tritical]

Is it easily possible to convert YCbCr to Y'CbCr?
You'd have to go YCbCr -> RGB -> R'G'B' -> Y'CbCr.

After reading it again, I'm fairly certain he means to do R'G'B -> RGB -> YCbCr because when you compute Y' using R'G'B' you aren't computing the true "luminance". So when you store the difference components as Cb = B'-Y' and Cr = R'-Y', introduce errors into them through downsampling/upsampling,and then rederive R'G'B', errors in Cb/Cr will cause some change in the luminance. Whereas if you store the true luminance, Y, then errors you introduce into Cb/Cr (Cb = B-Y and Cr = R-Y) cannot affect the luminance. The only thing is that he doesn't mention anything about precision. Although, at that point you are inventing your own system which isn't going to be compatible with current software/devices (which he mentions)... so nothing is preventing you from performing some kind of non-linear scaling on the YCbCr values prior to storing them in limited precision.

[yesgrey]

Edit: It seems that only Y is gamma corrected, while Cb and Cr are linear?
Y, Cb, Cr are not gamma corrected, only R, G, B are gamma corrected.
Y, Cb, Cr are always a linear combination of R, G, B, so, their content will always linearly reflect the way R, G, B are represented. If R, G, B are not gamma corrected, Y, Cb, Cr will also represent the colors in a linear form, if R, G, B are gamma corrected, they will represent the colors in a non-linear form, but not in a gamma corrected form.
Remember the formula:
Y = kr*R + Kg*G + kb*B; where kg = 1 - kr - kb (kr, kb are defined by the standard's specification)
Is it easily possible to convert YCbCr to Y'CbCr?
No. From the description above you can clearly see that:
Y' = kr*pow(R,0.45) + Kg*pow(G,0.45) + kb*pow(B,0.45) is not the same as Y'=pow(Y,0.45) , so you always have to do:
YCbCr->RGB->R'G'B'->Y'CbCr
I think the author is saying he thinks you will get better results if you first convert gamma corrected RGB to linear RGB before converting to YCbCr (in that case you technically get "luminance" - Y instead of "luma" - Y') and downsampling.
Yes, I also agree with this.
I have looked through the source code he supplyed, and in there he always works in the non-linear domain - he never removes the gamma from the images. So, the images he has there were not obtained with the code supplyed, or he hasn't tested it performing the calculations in the linear domain...:confused:

[yesgrey]

I was a bit late...:)

It's a very interesting paper, but with real world images the difference should be almost none. I am not writing cr3dlut to see red characters in grey backgrounds in all its glory.:D

But we can always try to create a version which works like this:
-Use one 3D LUT to convert from Y'CbCr->YCbCr
-perform chroma upsample in YCbCr
-Use one 3D LUT to convert YCbCr->Y'CbCr (only to apply on upsampled chroma)
-Use one 3D LUT to convert from Y'CbCr->R'G'B' (using original Y' values, and the upsampled CbCr values)
For the upsampling we will need a higher input bit depth 3D LUT for the YCbCr->Y'CbCr to avoid banding...

I can offer myself to create the LUTs.:D

[madshi]

But we can always try to create a version which works like this:
-Use one 3D LUT to convert from Y'CbCr->YCbCr
-perform chroma upsample in YCbCr
In order to do this we'd have to go Y'CbCr -> R'G'B' -> RGB -> YCbCr, right? But can we go from Y'CbCr 4:2:0 to R'G'B' without upsampling chroma first? I don't think so. Probably we'd have to downsample Y' to make it match CbCr resolution. Then we could convert the "quarter resolution" Y'CbCr intermediate to R'G'B' -> RGB -> YCbCr, and then finally upsample the chroma.

Ouch. I wonder if there'd even be a visible difference. The problem is: We will probably only be able to tell once we tried it... :(

[yesgrey]

In order to do this we'd have to go Y'CbCr -> R'G'B' -> RGB -> YCbCr, right?
Yes, that's how cr3dlut should build the 3D LUT.

But can we go from Y'CbCr 4:2:0 to R'G'B' without upsampling chroma first?
For a 3D LUT only interest Y'CbCr -> R'G'B'. Remember that a 3D LUT only relates the input and output values; it is applyed at the pixel level, and is always the same for each pixel; it has no relationship with chroma upsampling, that's another different thing...
For converting an 1920x1080 image (2073600 pixels) you should map each pixel through it, so it would be accessed 2073600 times. In Y'CbCr 4:2:0 you have a Y' value for each pixel (2073600 Y' values for the entire image), and a CbCr value for each 2x2 block of pixels (518400 CbCr values for the entire image). The chroma resampling is just to calculate the missing CbCr values.

Ouch. I wonder if there'd even be a visible difference.
If we'll use it with red characters in a grey background, it should be visible...:D

[madshi]

For converting an 1920x1080 image (2073600 pixels) you should map each pixel through it, so it would be accessed 2073600 times. In Y'CbCr 4:2:0 you have a Y' value for each pixel (2073600 Y' values for the entire image), and a CbCr value for each 2x2 block of pixels (518400 CbCr values for the entire image). The chroma resampling is just to calculate the missing CbCr values.
Trust me, I know what chroma upsampling is.

For a 3D LUT only interest Y'CbCr -> R'G'B'. Remember that a 3D LUT only relates the input and output values; it is applyed at the pixel level, and is always the same for each pixel; it has no relationship with chroma upsampling, that's another different thing...
My point is this: You suggested to do chroma upsampling in YCbCr (instead of Y'CbCr). But our source is Y'CbCr 4:2:0. How can we go from Y'CbCr 4:2:0 to YCbCr 4:2:0? This is not as easy as it seems. Because in order to go from Y'CbCr to YCbCr we have to go through RGB. And doing Y'CbCr 4:2:0 -> RGB 4:4:4 is not possible, I believe. You first have to upsample chroma to 4:4:4 to do Y'CbCr -> R'G'B'. But we want to upsample chroma in YCbCr and not in Y'CbCr. Do you see the problem?

[yesgrey]

Trust me, I know what chroma upsampling is.
I believe so, but what I wanted to make clear is that to create the 3D LUT does not matter if the color is 4:2:0, 4:1:1, 4:2:2, 4:4:4, etc... it's an independent thing.

How can we go from Y'CbCr 4:2:0 to YCbCr 4:2:0? This is not as easy as it seems.
I think it will be more clear with an example. Feel free to correct me if anything is wrong.;)
Let's see how it would be done using the method "same chroma for all pixels":
Let's call CbCr in Y'CbCr as C' and CbCr in YCbCr as C.
In Y'CbCr 4:2:0 we have Y'11, Y'12, Y'21, Y'22, and we have C'11.
The current method is doing C'12=C'21=C'22=C'11;
What I was proposing was:
-Use a Y'CbCr->YCbCr 3D LUT to find:
Y12=3dlut(Y'12,C'11)
Y21=3dlut(Y'21,C'11)
Y22=3dlut(Y'22,C'11)
C11=3dlut(Y'11,C'11)
-do C12=C21=C22=C11
-Use a YCbCr->Y'CbCr 3D LUT to find:
C'12=3dlut(Y12,C12)
C'21=3dlut(Y21,C21)
C'11=3dlut(Y22,C22)
-Use original Y'11,Y'12,Y'21,Y'22,C'11 with calculated C'12, C'21, C'22.
So, we have to perform two chroma upsamplings. One upsampling when converting Y'C' -> YC because to find the Y values through the 3D LUT we need a C' value. Another when doing it in YCbCr.

[madshi]

I think it will be more clear with an example. Feel free to correct me if anything is wrong.;)
Let's see how it would be done using the method "same chroma for all pixels":
Let's call CbCr in Y'CbCr as C' and CbCr in YCbCr as C.
In Y'CbCr 4:2:0 we have Y'11, Y'12, Y'21, Y'22, and we have C'11.
The current method is doing C'12=C'21=C'22=C'11;
What I was proposing was:
-Use a Y'CbCr->YCbCr 3D LUT to find:
Y12=3dlut(Y'12,C'11)
Y21=3dlut(Y'21,C'11)
Y22=3dlut(Y'22,C'11)
C11=3dlut(Y'11,C'11)
-do C12=C21=C22=C11
-Use a YCbCr->Y'CbCr 3D LUT to find:
C'12=3dlut(Y12,C12)
C'21=3dlut(Y21,C21)
C'11=3dlut(Y22,C22)
-Use original Y'11,Y'12,Y'21,Y'22,C'11 with calculated C'12, C'21, C'22.
So, we have to perform two chroma upsamplings. One upsampling when converting Y'C' -> YC because to find the Y values through the 3D LUT we need a C' value. Another when doing it in YCbCr.
The problem with this approach is that you're using the worst chroma upsampling filter possible. Using "C12=C21=C22=C11" is really bad. This will definitely look worse than the best we can do right now.

How about this instead:

(1) Create image A (1080p 4:4:4): Do the best quality chroma upsampling we can do on the original unprocessed source to get a Y'CbCr 4:4:4 image.
(2) Create image B (540p 4:4:4): Downsample Y' to quarter resolution, but keep chroma unchanged. This way we get a Y'CbCr 4:4:4 image, too, but with a quarter of the pixels.
(3) Convert both images (1) and (2) to YCbCr.
(4) Combine the Y from image (1) and CbCr from image (2). This should give us 1080p YCbCr 4:2:0.
(5) Do the best quality chroma upsampling we can do to convert (4) to YCbCr 4:4:4.
(6) Finally run this through 3dlut to correct gamut etc...

If possible we should avoid 3dlut until the final step to keep bitdepth as high as possible. Steps (1) to (5) should be done in floating point. Step (6) should use trilinear interpolation, so that we can make full use of the full bitdepth data we ended up with.

Of course this would be quite CPU consuming, but I think that should give the best results?

[yesgrey]

The problem with this approach is that you're using the worst chroma upsampling filter possible.
I used the simplest only to describe the idea more easilly, that was my only intent. We could use whatever filter we want.:)
Now, for the first upsampler, since we will use it just for calculating the Y values, maybe we could use that, and use a better one only for the chroma upsampling in YCbCr.
I'm just thinking lowd.

[madshi]

I used the simplest only to describe the idea more easilly, that was my only intent. We could use whatever filter we want.:)
Now, for the first upsampler, since we will use it just for calculating the Y values, maybe we could use that, and use a better one only for the chroma upsampling in YCbCr.
I'm just thinking lowd.
I don't think your approach works correctly. What do you think about my alternative suggestion (see my edited previous post)?

[yesgrey]

I don't think your approach works correctly. What do you think about my alternative suggestion (see my edited previous post)?
Your suggestion is almost the same as mine. The only difference is that instead of using Y'11 to convert CbCr to the linear domain you prefer using the subsampling of Y'11, Y'12, Y'21, Y'22.
I think there would be almost no difference between the results of our suggestions... mine will be slightly faster because avoids the Y' subsampling, and the quality would depend of the kind of chroma subsampling that was done in the source.
Of course I would prefer using float or double to perform all the calculations, but that would be very heavy to the cpu, unless we start thinking in using the GPU, which is very indicated for this kind of work... CUDA or OpenCL.;) it would also be great for the 3D LUT usage, because the Video card memory is a lot faster than the main memory.:)

[tritical]

This idea doesn't make any sense to me. If you start with a 4:2:0 Y'CbCr source (I'm going to designate this Y'Cb'Cr' to avoid confusion), then Cb'/Cr' were downsampled after having been calculated from R'G'B'. The more accurately you recompute the Cb'/Cr' of each Y', the closer you will get to the original. Why do you think it is better to recalculate those Cb'/Cr' values by transforming into YCbCr (using some method to estimate Y' at each chroma point if chroma placement does not line up with Y'), then upsampling Cb/Cr, and then transforming back to Y'Cb'Cr' using the upsampled Cb/Cr and Y values computed from combining the upsampled Cb'Cr' with stored Y'... as opposed to simply upsampling (interpolating in) Cb'/Cr'?

[madshi]

@tritical, you're probably right. I was thinking that chroma has been downsampled in linear light, while actually it most probably has been downsampled gamma corrected. So if it was downsamples that way, we should probably also upsample it that way.

FWIW, I've emailed the author of the article and asked for his opinion about this and also about his opinion on how to do best quality chroma upsampling. Here's his reply:

The very highest quality would come from either:
a- A pretty blurry reconstruction filter, something like a Gaussian blur/filter. We are not very good at spotting the loss of chroma resolution.
Don't use anything with negative lobes for chroma reconstruction if you are at the very end of the signal chain... this will cause colored ringing and looks bad.

You'd always want to do in-range chroma reconstruction, and for that to work perfectly you have to match your filters when you encode + decode the material (in practice this will never happen... people just do whatever when downsampling the chroma).
b- From a practical POV, I would just use triangle filters. This might get the chroma siting for 4:2:0 MPEG2 wrong... I'm not sure about that.

2- I don't believe you can do the upsampling in linear light if you start off with Y'CbCr data. That information is already thrown away I think.

3- In practice, you will likely have illegal values coming into your decoder. e.g. Y' at white level but Cb and Cr aren't neutral (so some of the channels will exceed white level), or Y' above white level with non-neutral Cb and Cr.

So when doing in-range chroma reconstruction, you have to relax the triangle weighing function as a compromise. That's how I would deal with that.

4- There may be other kinds of signal processing that yields more visual improvement than in-range chroma reconstruction. Very few people notice this stuff. Getting rid of macroblocking and deinterlacing artifacts would be my first priority.

Also see Stacey Spears and Don Munsil's article on the chroma bug... I believe I've linked to it from my article.
I see 2 things worth mentioning:

(1) Bicubic upsampling is probably not better than linear upsampling, because bicubic rings while linear upsampling doesn't ring.

(2) Not sure about the "illegal values". Do you think there's anything we could/should do there?

[yesgrey]

Why do you think it is better to recalculate those Cb'/Cr' values by transforming into YCbCr...
(2) Not sure about the "illegal values". Do you think there's anything we could/should do there?
According to the paper, I think this could be better but only for the out of range values. For all the other values, I think the current upsampling method is better.
Since the out of range values are not many (except in some particular images), I think it's not worthy having all that trouble. If in the future, we have some free time, maybe we could take a look for curiosity, but probably the differences could only be noted in special images created only for showing the problem, like the one in the paper.;)

[yesgrey]

(1) Bicubic upsampling is probably not better than linear upsampling, because bicubic rings while linear upsampling doesn't ring.
Testing it with yv12toyuy2 tritical's function, the bicubic looks better than the linear. To avoid ringing, is just keeping c<0.6 (according to Avisynth's manual description of Bicubic Resize filter). I use b=0 and c=0.5 (tritical, IMHO these should be the default values)

[madshi]

Any news about cr3dlut v2? :)

[yesgrey]

I will try to release it during the weekend...

[yesgrey]

I've just released here (http://yesgrey3.totalh.com/) cr3dlut v2.0.
v2.0 - 2009/03/23
- Added: creation of 3D LUT for R'G'B' -> Y'CbCr conversion chain
- Added: creation of 3D LUT for Y'CbCr -> Y'CbCr conversion chain
- Added: chromatic adaptation method using Bradford Transform (Chromatic_adaptation 2)
- Added: new run parameters "Input_Video_Format" and "Output_Video_Format"
- Added: new run parameters "Input_Bit_Depth" and "Output_Bit_Depth"
- Changed: renaming of the run parameters
- Changed: perform color gamut conversion only when the color gamuts are different (speed gain)


I've tryed to make it a little more user friendly.
Please note that some of the new functionalities are not usable, because are not supported by rgb3dlut. We have to wait for tritical's t3dlut to use it, but I don't know when (or if) it would be released...

Comments/suggestions are welcome.

[leeperry]

looks good I'll look into it, thanks!

BTW, 10 bit might appear sooner than you'd think :
http://www.amdzone.com/index.php/news/37/8998-nvidia-geforce-8200-hdmi-13-compliant

[madshi]

I've just released cr3dlut v2.0.
Thanks!!

[yesgrey]

The first time I've read this (http://www.poynton.com/papers/Discreet_Logic/index.html) Charles Poynton's article was a big surprise to me. I never thought that all our sources that claim to be 8bit per color component were in fact 8bit per color component but with lots of holes inside, kind of a swiss cheese.
When I started developing cr3dlut, this idea came through my mind and now I've decided to use cr3dlut's code to perform some "measures" and give me a better idea of it.
I simply have put some counters inside the code to count the number of colors that we could get when converting between the two representations commonly used: YCbCr and RGB. I also tryed different combinations, with different levels and different bit depths at both sides. I'm sure the results I will post here are nothing new to some of you, but I'm sure they will be for others.;)
I have decided to use the word "color" instead of the correct term "codeword", and the terms RGB and YCbCr instead of the correct R'G'B' and Y'CbCr, to simplify the understanding of the results to the less technical of us.
The "n. of available colors", is all the colors that could be represented using the ranges indicated.
The "n. of valid colors", is all the colors in the source representation that are within the ranges of the target representation.
The "n. of different colors obtained", is all the different colors that are obtainable in the target representaion. Some different colors in the source representation could correspond to the same color in the target representation.
For these tests, I have used BT.709 coefficients for the YCbCr<->RGB conversions.


YCbCr->RGB Standard Conversion

YCbCr(8bit) n. of available colors: 11 137 500 (Y: 16-235; CbCr: 16-240)
RGB (8bit) n. of available colors: 10 648 000 (RGB: 16-235)
YCbCr n. valid colors: 2 689 427
RGB n. of different colors obtained: 2 689 427 (25.3%)
---
YCbCr(9bit) n. of available colors: 88 502 839 (Y: 32-470; CbCr: 32-480)
RGB (8bit) n. of available colors: 10 648 000 (RGB: 16-235)
YCbCr n. valid colors: 21 515 482
RGB n. of different colors obtained: 10 595 662 (99.5%)


YCbCr->RGB Full Conversion

YCbCr(8bit) n. of available colors: 16 777 216 (Y: 0-255; CbCr: 0-255)
RGB (8bit) n. of available colors: 16 777 216 (RGB: 0-255)
YCbCr n. valid colors: 4 058 320
RGB n. of different colors obtained: 4 058 320 (24.2%)
---
YCbCr(9bit) n. of available colors: 134 217 728 (Y: 0-511; CbCr: 0-511)
RGB (8bit) n. of available colors: 16 777 216 (RGB: 0-255)
YCbCr n. valid colors: 32 657 781
RGB n. of different colors obtained: 16 702 766 (99.6%)

From these numbers, we can see that our current sources that use YCbCr(8bit) are our limiting factor. We would need YCbCr(9bit) to be able to use our current displays RGB(8bit) at its full potential...

This could raise one question: If the RGB(8bit) is not used at its full potential, would we really need or benefit of higher bit depths in our displays, like the new RGB(16bit) that we should get with Windows7?
Let's look at some more numbers...

RGB->YCbCr Standard Conversion

RGB (8bit) n. of available colors: 10 648 000 (RGB: 16-235)
YCbCr(8bit) n. of available colors: 11 137 500 (Y: 16-235; CbCr: 16-240)
RGB n. valid colors: 10 648 000
YCbCr n. of different colors obtained: 2 809 180 (25.2%)
---
RGB (10bit) n. of available colors: 674 526 133 (RGB: 64-940)
YCbCr(8bit) n. of available colors: 11 137 500 (Y: 16-235; CbCr: 16-240)
RGB n. valid colors: 674 526 133
YCbCr n. of different colors obtained: 2 828 080 (25.4%)
---
RGB (11bit) n. of available colors: 1 092 017 481 (RGB: 16-235)
YCbCr(8bit) n. of available colors: 11 137 500 (Y: 16-235; CbCr: 16-240)
RGB n. valid colors: 1 092 017 481
YCbCr n. of different colors obtained: 2 831 230 (25.4%)

From these numbers, we can see that no matter the bitdepth of the RGB, if the YCbCr bitdepth is 8bit per component, we will get almost no benefit by increasing the RGB bitdepth.

The question above has more than one answer.
If we watch our sources at its original resolutions and without any kind of processing, the answer is no, we do not need more than 8bit per component in our displays.
If we scale our sources to match our display's resolution or if we perform any kind of processing in YCbCr(<10bit), the answer is no, we do not need more than 8bit per component in our displays.
If we scale our sources to match our display's resolution or if we perform any kind of processing in YCbCr(>9 bit) or in RGB(>8 bit), then the answer may be yes, we could benefit of more than 8bit per component in our displays...

Now a question that is asked so many times: Should we convert from YCbCr->RGB using standard (16-235->16-235) levels or expanding (16-235->0-255) levels?
Let's look at a few more numbers...


YCbCr->RGB Expanding Conversion

YCbCr(8bit) n. of available colors: 11 137 500 (Y: 16-235; CbCr: 16-240)
RGB (8bit) n. of available colors: 16 777 216 (RGB: 0-255)
YCbCr n. valid colors: 2 689 427
RGB n. of different colors obtained: 2 689 427 (16.0%)
---
YCbCr(9bit) n. of available colors: 88 502 839 (Y: 32-470; CbCr: 32-480)
RGB (8bit) n. of available colors: 16 777 216 (RGB: 0-255)
YCbCr n. valid colors: 21 515 482
RGB n. of different colors obtained: 15 997 163 (95.4%)
---
YCbCr(10bit)n. of available colors: 705 642 093 (Y: 64-940; CbCr: 64-960)
RGB (8bit) n. of available colors: 16 777 216 (RGB: 0-255)
YCbCr n. valid colors: 172 122 708
RGB n. of different colors obtained: 16 777 200 (99.9999%)


Comparing the numbers of the Standard and the Expanding conversions, we can see that the n. of different colors is the same, but there is a significative difference between the two conversions, the "color density".
We can see that the standard conversion has the colors confined to a narrower range, so the colors have a smoother transition between them, which will show as less banding in color transitions.

[yesgrey]

I know that it could be a bit shocking that we are getting much less colors (less than 25%) that we usually think. So, to be able to see it this is true, I've created a 3DLUT file that will output as red pixels all the invalid colors specifyed in the numbers above. I've used the Full Conversion, so we could also be able to see if there are any colors outside the standard (16-235) range.

The first picture, was considering only the valid values contained in the standard (16-235) RGB values.
The image has a few red points. These points could be some of the measured invalid colors above that are valid, or could be valid values that are above the 235 range.
9689

The second picture, was considering the valid values contained in the full (0-255) RGB values. The image does not has any red points, so the points that appeared in the first picture were points above the 235 range.
9688

The pictures above prove that it's true.

[madshi]

That's pretty interesting - thanks!

Some thoughts:

(1) Our eyes are more sensitive to brightness than they are to colors. So I'd say: Screw the colors, they're not *that* important. Probably more important is the brightness gradiant representation ability. Is there a way to measure that? Based on your calculations it seems that using 7bit RGB might already be good enough. But just think of a test pattern which shows a simple grayscale ramp. I think you will agree that using only 7bit RGB would result in noticable banding. This already hints that some of the logic you've used might not be correct. More about this in point (2):

(2) You write "If [...], the answer is no, we do not need more than 8bit per component in our displays." I don't think that is correct. Why? Because I think your logic somehow ignors the fact that doing YCbCr -> RGB conversion results in floating point numbers and not in integer numbers. The "n. of different colors obtained" number is very interesting, but it only tells half the truth. Let me give you a simple example:

Let's say we had a 1bit YCbCr full range source and a 1bit full range RGB display. That would result in:

YCbCr(1bit) n. of available colors: 8 (Y: 0-1; CbCr: 0-1)
RGB (1bit) n. of available colors: 8 (RGB: 0-1)
YCbCr n. valid colors: 8
RGB n. of different colors obtained: 8
Probably the number of valid/different colors would be even less than 8, but let's just pretend they'd be 8 to keep the discussion simple. Now based on the logic you're using 1bit RGB would be all we need, right? *Wrong*. Let's say the valid 8 YCbCr colors map to RGB "(0, 0, 0), (0, 0, 1), (0, 1, 0), (0, 1, 1), (1, 0, 0), (1, 0, 1), (1, 1, 0), (1, 1, 1)". In this case you would be right. But the mapping will not be like that. Instead it will be something like: "(0.0, 0.0, 0.0), (0.1, 0.1, 0.9), (0.2, 0.8, 0.0), (0.1, 0.9, 0.85), (0.95, 0.0, 0.08), (0.9, 0.05, 0.8), (0.9, 0.95, 0.02), (1.0, 1.0, 1.0)".

Do you see the problem? Using only 1bit RGB would result in grossly incorrect display. Using e.g. 12bit RGB output would produce a much more exact display - even though there are only 8 valid YCbCr colors to start with! Ok, we could work around the problem by using dithering. But dithering is always only a workaround. Having higher RGB bitdepth means that we can represent the YCbCr data better without having to do any workarounds.

(3) Already hinted by your post: Any (good quality) scaling will result in intermediate chroma and brightness values, which will "increase" source bitdepth.

IMHO a 16bit output 3dlut should be used and the higher the RGB bitdepth is the better. Only this way we can get near the correct display of the YCbCr source. The number of different valid colors doesn't matter. The exactness of the representation does! Even if you only have a low number of valid colors to start with, having to round RGB data might result in banding to appear/increase which would be less visible if more correct RGB colors would be used. Just think of a RGB gradiant of (80.0, 82.5, 85.0). With 8bit RGB you'd have to round that to (80, 83, 85). With 9bit RGB you could properly display the gradiant as (80.0, 82.5, 85.0). The 9bit RGB display would show a more evenly distributed gradiant compared to the 8bit RGB display.

[FoLLgoTT]

Do you see the problem? Using only 1bit RGB would result in grossly incorrect display. Using e.g. 12bit RGB output would produce a much more exact display - even though there are only 8 valid YCbCr colors to start with! Ok, we could work around the problem by using dithering. But dithering is always only a workaround. Having higher RGB bitdepth means that we can represent the YCbCr data better without having to do any workarounds.

Very interesting!

Let me think about this. Let's assume the master was already converted to 8bit RGB (from e.g. 12 bit). So when converting to YCbCr we get floating point numbers which are rounded to integer values. Now the data is getting reduced by the MPEG encoder.
While playing back the video the conversion YCbCr->RGB again results in floating point numbers which have to be rounded to 8 bit integer. Who guarantees that both rounding methods are the same? Maybe the first is a floor and the second a ceiling function as a worst case example. Is it even possible to correctly reconstruct the 8 bit RGB values?

[leeperry]

for what it's worth, the Beliyal MPC HC builds apparently support 10bit on Vista :
http://forum.doom9.org/showthread.php?t=145203

w/ nvidia supporting HDMI 1.3 on their latest IGP, and prolly next PCI-E cards...this might be good news ^^

[73ChargerFan]

Let's look at some more numbers...
Code: RGB->YCbCr Standard Conversion

So YCbCr(8)->RGB(8) is a many-to-one map, and the full resolution of RGB(8) isn't utilized in this conversion.

Could you similarly analyze YCbCr(8)->RGB(10) ? Or do you predict the same 4:1 mapping?

[yesgrey]

So YCbCr(8)->RGB(8) is a many-to-one map
No. That is a one-to-one map. If you look closely, we get in RGB the same number of colors that the YCbCr n. of valid colors.
The many-to-one map will only be true for YCbCr(>8bit )->RGB(8bit)

Could you similarly analyze YCbCr(8)->RGB(10) ?
No, I will need lots of memory for doing it, or I would have to use some kind of linked list, which would be too cpu intensive.
Or do you predict the same 4:1 mapping?
Yes. Remember, the limiting factor is the bit depth used in YCbCr, so, no matter what bit depth you use in RGB, you will always get almost the same number of colors. That's the reason I have used RGB(10)->YCbCr(8), which is almost the same thing, and computable.;)
Look at it this way: YCbCr is a much wider representation of colors than RGB, so, it contains RGB entirely in it and still has a lot of free values without any correspondence within the RGB space. If they use the same bit depth, the YCbCr representation of the RGB values will have lower resolution. That's revealed when we see that YCbCr (at the same bit depth of RGB) only can contain <25% of the colors available in RGB.

[madshi]

Let me think about this. Let's assume the master was already converted to 8bit RGB (from e.g. 12 bit). So when converting to YCbCr we get floating point numbers which are rounded to integer values.
Studios sometimes use rounding and sometimes they use dithering. See this very interesting thread:

http://www.avsforum.com/avs-vb/showthread.php?t=1011359

Now the data is getting reduced by the MPEG encoder.
While playing back the video the conversion YCbCr->RGB again results in floating point numbers which have to be rounded to 8 bit integer. Who guarantees that both rounding methods are the same?
First of all "have to be rounded" is not true. We don't have to use simple rounding. The 3dlut logic which was discussed and defined in this thread can optionally do 8bit YCbCr -> 16bit RGB. In this case the floating point RGB data is rounded down to 16bit integer which should be plenty enough precision for our eyes. Still, if the display is only 8bit, those 16bit have to be brought down to 8bit somehow. You can use rounding for that again. Or you can use dithering.

IMHO dithering is a must if you want really faithful image reproduction, because simple rounding introduces too big deviation from the floating point RGB data. Actually the lower the bitdepth of the display, the higher the rounding errors are! So when using 8bit RGB rendering, using dithering is much more important/helpful than when using e.g. 16bit RGB rendering.

yesgrey3's calculations about how many colors YCbCr can code are (most probably) correct. But how many colors YCbCr can code is one thing. How to accurately convert those limited number of colors to RGB is IMHO a totally separate topic. Even if YCbCr only allowed 5 colors to be coded, we still wanted to display them as accurately as possible. And for that purpose using 8bit integer RGB with simple rounding just doesn't cut it IMHO.

[leeperry]

@yesgrey : that's quite striking to read you saying that outputting 8 or 10bit doesn't make much difference...if any, and that we hardly use 25% of the gamut :o

no wonder I couldn't detect any banding on my green-deficient DLP pj :rolleyes:

anyway, Epson finally offers no-brainer gamut conversion:

http://www.hdsoir.com/wp-content/uploads/2009/03/epson-tw5000-gamut-menu.jpg

I'd guess that it works in 10bit too ;)

it comes from this tw5000 test(french link, sorry) :
http://www.hdsoir.com/?p=5700&page=2

ideally over the next years, some 10bit sources will show up(Bluray Reloaded?), HDMI 1.3 graphic cards(already the case w/ nvidia IGP's) and projectors that do conversions internally :devil:

[tetsuo55]

So is this an accurate conclusion?:

-8bit source should look the same on all 8+ panels, no need for higher rgb bitdepth(and judging from the results a good 6bit panel should be able to display all the required colors also)

-A very VERY good dithering, while using a very high bitdepth YCbCr->RGB conversion (minimising the errors into the 0.0000x%) could result in a slightly more realistic looking/colorfull image, while deviating from bitperfectness

We basically have almost 75% of the space left to do all kinds of fun post-processing stuff.

I doubt the results would be the same for luminance though

[tritical]

t3dlut v1.0 (http://bengal.missouri.edu/~kes25c/t3dlut.zip). Readme is a little sparse, and I haven't had time to test it much.

[yesgrey]

Thanks tritical!:)
I will give it a try.

[yesgrey]

If we scale our sources to match our display's resolution or if we perform any kind of processing in YCbCr(<10bit), the answer is no, we do not need more than 8bit per component in our displays.
After some tests using t3dlut, it seems my quote above is true.
I have used t3dlut with 3dlut files with 8bit output and 16bit output dithered to 8bit. You can get some of the files here (http://www.megaupload.com/?d=283EM50Y).
As you can see, the biggest difference between the pure 8bit output files and the 16bit dithered to 8bit output files is the file size.
Looking into the images, we cannot see any difference. We have to zoom a lot to see some slightly differences, and even then we have to look very carefully. I also created a pic with the difference between both images, applyed to a 50% gray screen (I have used avisynth subtract function). Looking into it, it seems to be a plain 50% gray image, but if we use a tool to see the values of each pixel, we can see that several values are different by +/-1.

Conclusion: unless we have higher bit depth sources, or higher bit depth processing, our best bet is keep using the 8bit output 3dlut's. The 16bit output would be just wasting hard disk space...

[tetsuo55]

After some tests using t3dlut, it seems my quote above is true.
I have used t3dlut with 3dlut files with 8bit output and 16bit output dithered to 8bit. You can get some of the files here (http://www.megaupload.com/?d=283EM50Y).
As you can see, the biggest difference between the pure 8bit output files and the 16bit dithered to 8bit output files is the file size.
Looking into the images, we cannot see any difference. We have to zoom a lot to see some slightly differences, and even then we have to look very carefully. I also created a pic with the difference between both images, applyed to a 50% gray screen (I have used avisynth subtract function). Looking into it, it seems to be a plain 50% gray image, but if we use a tool to see the values of each pixel, we can see that several values are different by +/-1.

Conclusion: unless we have higher bit depth sources, or higher bit depth processing, our best bet is keep using the 8bit output 3dlut's. The 16bit output would be just wasting hard disk space...

I see a very subtle difference between the 2 screenshots.

The 16bit version almost seems to have a darker hue in some area's, especially the skin.
In this sample case i like the 8bit version better.

Could be a complete placebo though.

I compared the images on a CRT

[leeperry]

well I've never been convinced by that "16bit is higher precision, then dither back to 8 will look better than plain 8 bit" myself :o

you can't quite create data that's not there, only create additional distorsion/noise. But it's good that it supports 10bit output...I think? maybe not :D

OK I will follow the infos yesgrey3 gave me in PM, and try to make a simple guide coz it's getting kinda confusing w/ the zillion options in the new cr3dlut/t3dlut(thank you tritical BTW!) versions

[yesgrey]

Based on your calculations it seems that using 7bit RGB might already be good enough. ...
This already hints that some of the logic you've used might not be correct.
No, it's not good enough. RGB(7bit) has 78% of the n. of colors available in YCbCr(8bit), and YCbCr will not map into RGB(7bit) like it maps into RGB(8bit).

Let me give you a simple example:

Let's say we had a 1bit YCbCr full range source and a 1bit full range RGB display. That would result in:

YCbCr(1bit) n. of available colors: 8 (Y: 0-1; CbCr: 0-1)
RGB (1bit) n. of available colors: 8 (RGB: 0-1)
YCbCr n. valid colors: 8
RGB n. of different colors obtained: 8
Probably the number of valid/different colors would be even less than 8, but let's just pretend they'd be 8 to keep the discussion simple. Now based on the logic you're using 1bit RGB would be all we need, right? *Wrong*
No, your example is flawed. YCbCr(1bit) only has 2 valid colors in RGB(1bit), so, RGB(1bit) would continue to exceed all our needs (it has 8 colors available), still the <25% rule.;)
I see what you are trying to say, and I agree with you... for less than 8bit depths.:)
With bit depths of 8 and above, it's not too difficult for us to differentiate between two adjacent colors when looked alone. Imagine a full image at RGB[128,128,128] vs a full image at RGB[128,129,128]. If you compare both images switching between them or side by side, you can notice the difference (not very easy though), but if you watch each of them separatelly, you would not be able to tell the difference.
With RGB(1bit), it's easy to spot that difference.
The number of different valid colors doesn't matter. The exactness of the representation does! Even if you only have a low number of valid colors to start with, having to round RGB data might result in banding to appear/increase which would be less visible if more correct RGB colors would be used.
I don't agree with this. Banding is the result of using the same color to represent different colors. The exactness of the representation does not affect this, because it will always exist some "space" between the adjacent colors due to the low number of different valid colors. Also, the exactness of the representation is already damaged by the original conversion from RGB->YCbCr.:(

[yesgrey]

well I've never been convinced by that "16bit is higher precision, then dither back to 8 will look better than plain 8 bit" myself :o
We cannot rule out yet the dither because currently there is no way to see if it's useful or not. Currently, all software that we are using reduces the bit depth to 8bit, so the dithering of 16bit to 8 bit is not used at its full potential, it's only being used in t3dlut for the YCbCr->RGB conversion, nothing more, and as I showed with my previous posts, it could only be useful with other types of processing.

[leeperry]

We cannot rule out yet the dither because currently there is no way to see if it's useful or not.
well I took the analogy of upsampling 44.1 to 96 Khz > process w/ EQ then downsampling to 44.1 again....no added value, only distortion.

but if I understand, what we crave for badly is of course 10bit input, but more importantly 10bit output so the gamut conversion from 8bit wouldn't be so destructive anymore :)

anyway Win7 supports it and the latest nvidia IGP do too, so it should be even usable as we speak :eek: ....too bad I'll be sticking to XP until the end of days, and my HC3100 is not 1.3 compliant anyway :devil:

[madshi]

I don't agree with this. Banding is the result of using the same color to represent different colors. The exactness of the representation does not affect this, because it will always exist some "space" between the adjacent colors due to the low number of different valid colors.
My understanding of the term "banding" is: >> Visible steps in a gradiant. <<

Such banding can be in the original studio master. It can be caused by the encoding. It can be caused by the decoding. It can be caused by the processing, by the color conversion or by the display. I think our eyes are quite sensitive to even small errors in gradiants.

If the YCbCr source has a gradiant which maps to the RGB values e.g. "80.0", "81.5" and "83.0", then each banding step is "1.5" steps away from the next. If you round that, the gradiant steps will be "1" and "2" instead of "1.5" and "1.5". I think such a situation might show as increased banding to our eyes.

But maybe we should stop discussing this topic for now because we don't really have a good way to actually test/proof this. Yet (!).

[madshi]

well I took the analogy of upsampling 44.1 to 96 Khz > process w/ EQ then downsampling to 44.1 again....no added value, only distortion.
That is a very bad analogy.

We are *not* trying to increase the bitdepth of YCbCr data. The part you seem to be missing is that the inevitable YCbCr -> RGB conversion automatically ends up in floating point RGB data. So all Blu-Rays, DVDs and also all broadcasts don't have a bitdepth of 8bit RGB. They have a bitdepth of 8bit YCbCr. The native RGB bitdepth of all these sources is floating point. Say it together with me to make the message sink in: "Decoding and converting Blu-Rays, DVDs and broadcasts to RGB colorspace means we end up with floating point RGB data". And it's a simple mathematical fact that you can't convert floating point data to 8bit integer data without doing violence to the data. Using a higher RGB output bitdepth doesn't mean that we're trying to "upsample" anything. Instead we're trying to minimize the mathematical error caused by converting the floating point RGB data to a format we can transport to the display.

yesgrey3 and I do agree that it's mathematically more correct to use e.g. 10bit RGB output than 8bit RGB output. The only thing we do not agree on is the question whether the mathematical difference will be visible to our eyes.

[leeperry]

Say it together with me to make the message sink in
hehe, OK I tried to say it w/ you, but I dunno if we were in sync :confused:

well you can go "10bit>8 bit" or ">8bit" directly, it won't make the slightest visible difference....or maybe even worsen the roundings(like going 16>14>12>10>8 sure would make things worse :o )

but I'm glad you took the time to correct me, thank you for that!

[yesgrey]

Imagine a full image at RGB[128,128,128] vs a full image at RGB[128,129,128]. If you compare both images switching between them or side by side, you can notice the difference (not very easy though), but if you watch each of them separatelly, you would not be able to tell the difference.
If the YCbCr source has a gradiant which maps to the RGB values e.g. "80.0", "81.5" and "83.0", then each banding step is "1.5" steps away from the next. If you round that, the gradiant steps will be "1" and "2" instead of "1.5" and "1.5". I think such a situation might show as increased banding to our eyes.
yesgrey3 and I do agree that it's mathematically more correct to use e.g. 10bit RGB output than 8bit RGB output. The only thing we do not agree on is the question whether the mathematical difference will be visible to our eyes.
madshi, I agree with stopping the discussion for now until we have a method of testing our claims.;)

I only want to note that we do not disagree. You can see at my quote that it's the same as in yours. When the colors are side-by-side we could notice the difference.
The only thing that I say is that (considering your example) if you look to an image with only the RGB value "81.5" or only the value "82.0", we could not notice any difference in both colors.
Rephrasing it: I agree that the mathematical difference will be visible to our eyes if in a gradient of color. If it's an isolated color, we would not notice it. If we have RGB values "100.0", "81.5" and "63.0", we could not distinguish it from "100.0", "82.0" and "63.0".

[leeperry]

Rephrasing it: I agree that the mathematical difference will be visible to our eyes if in a gradient of color. If it's an isolated color, we would not notice it. If we have RGB values "100.0", "81.5" and "63.0", we could not distinguish it from "100.0", "82.0" and "63.0".
so as long as the source will be 8 bit, there'll be too many pieces of the puzzle missing? but why would 10bit output help then? you don't plan on "debanding" stuff AFAIK

plus some dithered(from the uncompressed source) 8bit would have more pieces than plain 8 bit I think :confused:

oh well, you already answered me on that point, I think it has to do w/ the available "headroom" after the gamut conversion or sumthing :D

well, anyway 8 bit is cool w/ me considering my pj is not 1.3 compliant....I'll recalibrate all my displays and try to get the whole thing to sing :devil:

[madshi]

hehe, OK I tried to say it w/ you, but I dunno if we were in sync :confused:
:)

well you can go "10bit>8 bit" or ">8bit" directly, it won't make the slightest visible difference....
How do you know that? Have you tested it? With a good test pattern (ideally a "slow" gray ramp)?

Rephrasing it: I agree that the mathematical difference will be visible to our eyes if in a gradient of color. If it's an isolated color, we would not notice it. If we have RGB values "100.0", "81.5" and "63.0", we could not distinguish it from "100.0", "82.0" and "63.0".
Agreed.

[leeperry]

@tritical: hi, I was using a test pattern video in ddcc(realtime) and in rgb3dlut() and they definitely don't match :o

it's available here : http://rapidshare.com/files/221027650/rec709.mkv.html

and the results are here : http://forum.doom9.org/showpost.php?p=1273996&postcount=288

am I doing something wrong? I get the same burned up green w/ yesgrey's app in madVideoRenderer and 16bit LUT's..

this is the PS script :

sampler s0 : register(s0);
float4 p0 : register(c0);

static float4x4 r2r =
{
1.15934746091817,-0.152088743917486,0.0260432549966361,0,
0.0250652234853865,0.974389775516909,-0.00709869095234219,0,
0.0100702399922673,0.0548005860451516,0.93638237486273,0,
0, 0, 0, 0
};

float4 main(float2 tex : TEXCOORD0) : COLOR
{
float4 c0 = tex2D(s0, tex);
c0 = pow(c0, 1/0.45);
c0 = mul(r2r, c0);
c0 = saturate(c0);
c0 = pow(c0, 0.45);

return c0;
}


and HC3100.txt :

0.151
0.071
0.342
0.610
0.656
0.332
0.311
0.330
1
0.0
0.45
0.0
0.0

:thanks:

How do you know that? Have you tested it? With a good test pattern (ideally a "slow" gray ramp)?
I'm colorblind :D

[yesgrey]

tritical,
I've found what's the problem. I will make some tests and then will post the results here...

[yesgrey]

I've just released here (http://yesgrey3.totalh.com/) cr3dlut v2.1.
v2.1 - 2009/04/17
- Added: new run parameter "Out_Of_Gamut_Clipping"
- Added: chromatic adaptation method using von Kries model in XYZ (Chromatic_Adaptation 3)
- Changed: basic parameter Video_Format settings were renamed
- Changed: basic parameter Video_Format do not clip anymore BTB and WTW when using RGB_Video
- Fixed: RGB->YCbCr was always considering Input_RGB_Black_White 0 255.


This time I've just released the executable. The source code is still v2.0.
This was just an interim version due to a problem described by leeperry. I'm currently working in adding custom gamma curves to cr3dlut, so the source code is still a bit messy. In the next release, I should have time to clean up the source code...

Comments/suggestions are welcome.

[yesgrey]

am I doing something wrong? I get the same burned up green w/ yesgrey's app in madVideoRenderer and 16bit LUT's..
I will make some tests and then will post the results here...
There is no bug.
The problem is that leeperry projector's color gamut is not large enough to contain entirelly the SMPTE-C color gamut, so, when converting between color gamuts, some colors are not representable, because there is no valid mapping inside leeperry's projector color gamut.
When ddcc is working without a 3D LUT, it simply clips the out of gammut colors by rounding the rgb values to the nearest possible. When ddcc works with a 3D LUT, it uses a clipping method that calculates a new color mantaining the hue and saturation of the original color, because the eye is more sensitive to hue variatons than to saturation or luminance variations.
Here are 4 images that show this effect:

The first two are using leeperry projector's color gamut, which is not capable of outputting all colors contained in the SMPTE-C color gamut, more specifically, in the green region. As you could see, the two clipping methods give very different greens corresponding to the most saturated green possible.
9778
9779

The other two, are using my projector's (JVC M15) color gamut, which has a color gamut larger than the SMPTE-C color gamut, so it can maps the entire SMPTE-C color gamut during the conversion. Both clipping methods give the same results.
9780
9781

[leeperry]

ah, so I'm stuck to ddcc in realtime/the PS script? they output perfect colors on my green deficient HC3100.

in the past, you've proved that 8bit movies were hardly using 25% of the gamut anyway, so these most saturated green tints are hardly ever used IRL..I haven't noticed any banding anyway, and clipping's fine(most DLP's exhibit a lack of green) :o

even the interim cr3dlut version you gave me gives too saturated red against ddcc/the PS script, but that's OK! because I'm color blind on the red, I'm very happy w/ the colors actually :cool:

to answer your PM, no I wasn't kidding...I'm truly color blind. there's many types of color blindness, mine is mostly red-based, basically my gamut is a lot less refined than "normal" people.
I see colors differently, just for fun I should add that the Army uses color blind ppl to see camouflage normal ppl can't see...we have a complete different perception of colors. growing older, my brain has learned to "compensate" somehow for the lack of red cones in my eyes :devil:

that's why I'm very sensitive to colorimetry, "proper" colors look natural, slightly wrong colors look artificial...it's a case of all black/all white, no "in between" if you get my drift

anyway I'll try the new build http://forum-images.hardware.fr/images/perso/athome.gif

PS: I only see 25 and 56 on this url: http://www.toledo-bend.com/colorblind/Ishihara.asp

I can hardly guess 29, but can't really see it..

[cyberbeing]

The gamma clipping setting is broken.

Error! Invalid Keyword setting: Out_Of_Gamut_Clipping 1

[yesgrey]

The gamma clipping setting is broken.
Error! Invalid Keyword setting: Out_Of_Gamut_Clipping 1
No, it's working; it was just the testing that was broken - the dangers of the "Copy & Paste"... ;) You were using Chromatic_Adaptation 2.

I've made a silent update. Please re-download the pack again.

Thanks.

[yesgrey]

even the interim cr3dlut version you gave me gives too saturated red against ddcc/the PS script, but that's OK!
Let me know if with the new version this still happens, so I could look into it to see what's happening...

to answer your PM, no I wasn't kidding...I'm truly color blind.
...
that's why I'm very sensitive to colorimetry, "proper" colors look natural, slightly wrong colors look artificial...it's a case of all black/all white, no "in between" if you get my drift

I think this is the reason why you dislike the "keep hue" kind of clipping. Being colorblind, the hue of a color does not have to you the same meaning that for people with regular color sensitivity. So, I've just added to cr3dlut a new option that let the user specify the clipping method to use when handling out of gamut colors. This way, you (and probably other color blind people) could achieve the results that look more appealing to their vision.:)

[leeperry]

Let me know if with the new version this still happens, so I could look into it to see what's happening...

I think this is the reason why you dislike the "keep hue" kind of clipping. Being colorblind, the hue of a color does not have to you the same meaning that for people with regular color sensitivity. So, I've just added to cr3dlut a new option that let the user specify the clipping method to use when handling out of gamut colors. This way, you (and probably other color blind people) could achieve the results that look more appealing to their vision.:)
well ddcc works fine in realtime, and burns up the green in a LUT....I don't really see how this is related to my color blindness..
I just would like to get the ddcc "realtime" colors in a LUT :o

OTOH if I can get the red slightly more saturated, the better :)

and not that ugly black bar in the middle of the green patterns like in CA0 in the "interim" build would be cool too...I'll try again and report back, thanks!

[yesgrey]

well ddcc works fine in realtime, and burns up the green in a LUT....I don't really see how this is related to my color blindness.
Let me try to explain... ddcc uses two clipping methods for out of gamut colors. Let's consider an example tritical and IanB talked some posts back...
R=0.72; G=1.07; B=0.42
The valid values should be between 0.0 and 1.0.
The first clipping method, used without the 3DLUT, just set the rgb values to be inside the valid range:
R=0.72; G=1.00; B=0.42
The second clipping method, used within the 3DLUT, set the rgb values in a way that maintains the hue of the original rgb value. The values would then be:
R=0.72/1.07, G=1.07/1.07, B=0.42/1.07
R=0.673, G=1.000, B=0.393
Since the hue perception for color blind people is different than the hue perception for people with regular color vision, probably the method above is not very friendly to them...
It would be a good test if you could bring anyone with regular color vision to watch your projector, and then show him/her a movie using both clipping methods to see what would he/she prefer... A woman would be more safe, because they are less prone to be color blind...
OTOH if I can get the red slightly more saturated, the better :)
You can try undersaturating the source red primary coordinates to see if it would help...;)

[leeperry]

ok I understand color blind ppl don't see the "hue" the same way normal ppl do, but I'm happy w/ ddcc in realtime, which does NOT give the same results in a LUT(burning the green)...if I got it right it doesn't use the same algorithm for out of gamut colors in a LUT, which you have fixed in your latest cr3dlut build :confused:

even "normal" ppl can see that :
http://forum.doom9.org/showpost.php?p=1273996&postcount=288

oh these questions are banned in my house, I'm not allowed to ask anyone "so what do you say, does the red look better like this <click> or like that?" :D

[yesgrey]

if I got it right it doesn't use the same algorithm for out of gamut colors in a LUT, which you have fixed in your latest cr3dlut build :confused:
Yes, but I have not fixed it, because there is nothing to fix. I've only added the option of selecting the algorithm used for handling out of gamut colors, to let people try and choose which they prefer.

even "normal" ppl can see that :

I haven't said that "normal" people could not see the difference, what I said was that, probably, "normal" people would find the colors by the the "maintain hue" algorythm more appealing than the simple clipping algorythm...;)

[leeperry]

so provided that you had a DLP w/ a lack of green(like mine), SOME ppl could/would prefer to have the green burned up over the regular "realtime" ddcc() algorithm? I don't see it happening as it really makes everything green to death, but OK :)

so to get the "realtime" algorithm, I have to select CA3 right?

[yesgrey]

so provided that you had a DLP w/ a lack of green(like mine), SOME ppl could/would prefer to have the green burned up over the regular "realtime" ddcc() algorithm? I don't see it happening as it really makes everything green to death, but OK :)
Maybe you are right, I don't know. Would it be possible that you see all as green because you can't see the red part that is in the image?:confused:

so to get the "realtime" algorithm, I have to select CA3 right?
No.
-CA3 and "Out_Of_Gamut_Clipping 0" is the same as the PS script.
-CA1 and "Out_Of_Gamut_Clipping 0" is the same as ddcc realtime

[leeperry]

I mean, the PS script and ddcc realtime look quite similar...where ddcc in LUT burns the green gamma in my previous tests :
http://forum.doom9.org/showpost.php?p=1273996&postcount=288

don't tell me that the green is identical between all the charts...please :D

anyway, I will compare them again...if I can get the same colors as ddcc realtime I'll be a happy camper :thanks:

[yesgrey]

I mean, the PS script and ddcc realtime look quite similar...
Yes, but they are not equal, because ddcc performs chromatic adaptation using Linearized Bradford Transform, while PS script uses von Kries in XYZ.

don't tell me that the green is identical between all the charts...please :D
But I never said that the green is the same, what I say is that it could be more appealing... But there is a problem in the HUE method that you never noticed it, and it's pretty obvious... banding. If you look carefully, you see that with the simple clipping there are 4 tones of green, and with the "maintain hue dithering" there are only two tones of green and red. If you used that in your projector you should get lot's of banding, and that would be ugly. Even if the colors would be more appealing, it would be pretty useless.

[leeperry]

well, as I understand it there's some banding because my gamut exceed the red and doesn't match the green SMPTE-C references(clipping green and lowering the red "scale") :

http://www.image-load.eu/out.php/t158097_gamut.png (http://www.image-load.eu/out.php/i158097_gamut.png)

I sent you screenshots, the "green burning" makes everything green in movies.

anyway, I'll try the new cr3dlut build and hopefully I'll find something that I like http://forum-images.hardware.fr/images/perso/caloub.gif

the PS script already looks great you know, and I only have an Eye One Display 2 colorimeter, so w/o an uber-accurate freshly recalibrated spectrophotometer...this is major nit-picking IMVHO.

even though my i1d2 gives very close results to the Eye One Pro(the Calman coders said on AVS that the measurements difference wouldn't be visible on DLP front projectors), if the i1d2 gives a ΔE of 1, god knows if it wouldn't be 5 or 6 w/ a Minolta CS-200 :o

and the most saturated colors in the gamut are hardly ever used anyway?

[leeperry]

alright, I was kinda surprised to see how slow my o/c Q6600 was doing on 16bit LUT's...so I've done a few benchmarks :

ConvertToYUY2()
t3dlut("C:\16bit.3dlut",threads=4)
=117

ConvertToYUY2()
t3dlut("C:\8bit.3dlut",threads=4)
=360

ConvertToRGB32(matrix="rec709")
ddcc(chr_i=3,gam_i=5,ofile="C:\HC3100.txt",threads=4,opt=1)
=123

ConvertToYUY2()
rgb3dlut(lutfile="C:\HC3100_SMPTE-C.txt",itype=2,threads=4)
=480

ConvertToYUY2()
rgb3dlut(lutfile="C:\HC3100_SMPTE-C.txt",threads=4)
=555

yv12toyuy2(itype=2,threads=4)
rgb3dlut(lutfile="C:\HC3100_SMPTE-C.txt",threads=4)
=410

yv12toyuy2(itype=2,threads=4)
rgb3dlut(lutfile="C:\HC3100_SMPTE-C.txt",itype=2,threads=4)
=355

I don't really get why t3dlut is much slower than rgb3dlut on the same 8bit LUT(created in cr3dlut)...I guess it's not because of the new file format, so maybe some of the default options make it slower?

I understand 16bit LUT's give much better results in gray ramps, but it'd need further testing to see whether it's worth the HUGE performance drop. But mVR doesn't use any CPU cycles to run the 16bit LUT anyway? so it'd be a non-issue there.

Also, I think the picture is somewhat sharper in mVR...might need to play around w/ rgb3dlut's settings :)

[tritical]

t3dlut is slower than rgb3dlut because with all of the extra conversions I decided not to special case everything. Also there is no assembly. In rgb3dlut everything was special cased, and everything but itype=2 was in assembly.

in itype=2 I get an error msg w/ either b/c=0.0,1.0 or b/c="0.0,1.0"....anyone's got an idea how to use that?
What is the error message?

[leeperry]

It said that it was missing a " or a ( )...probably the "/" is confusing Avisynth, or I'm really missing some character?

And is it my imagination or yv12toyuy2(itype=2) + rgb3dlut(itype=2) look sharper than ConvertToYUY2() + rgb3dlut(itype=1) ? the PQ is really great!

also, you set the bicubic coeff to 0.75 in rgb3dlut and its yv12toyuy2 but it's 0.5 in t3dlut and its yv12toyuy2. I thought 0.6 was the most mathematically accurate, and 0.75 slightly sharper?

[leeperry]

so I've compared 8bit against 16bit dithered.

native BD content w/o rescaling in ffdshow or HR :

http://thumbnails14.imagebam.com/3312/7e4d6333119757.gif (http://www.imagebam.com/image/7e4d6333119757) http://thumbnails15.imagebam.com/3312/09600333119759.gif (http://www.imagebam.com/image/09600333119759)http://thumbnails19.imagebam.com/3312/8e730533119762.gif (http://www.imagebam.com/image/8e730533119762)

that's the cr3dlut .ini :
# Example input file for cr3dlut v2.0 and up
#
# Settings for creating a 3D LUT for a Custom display
# (including YCbCr->RGB conversion)
#

# Do not mess up with these settings
Chromatic_Adaptation 1
Input_Bit_Depth 8
Input_YCbCr_Full_Range 0
Input_Gamma 1
Output_Bit_Depth 16
Output_YCbCr_Matrix 0
Output_RGB_Black_White 0 255

# You can change the following settings

# for BT.709 use '1'; for BT.601 use '3'
Input_YCbCr_Matrix 1

# for HDTV use '0'; for EBU use '2'; for SMPTE-C use '3'
Input_Primaries 0

# put here your primaries and white point xy coordinates, RGBW order.
Output_Primaries 1

# if you want to try other gamma values change the 0.45 value accordingly
Output_Gamma 1

left side is
yv12toyuy2(itype=2,threads=4)
t3dlut("C:\test_8bit.3dlut",threads=4)

right side is
yv12toyuy2(itype=2,threads=4)
t3dlut("C:\test_16bit.3dlut",threads=4)

http://thumbnails18.imagebam.com/3312/4bf71433119768.gif (http://www.imagebam.com/image/4bf71433119768)http://thumbnails14.imagebam.com/3312/9c07d933119771.gif (http://www.imagebam.com/image/9c07d933119771)
http://thumbnails3.imagebam.com/3312/6c78d433119774.gif (http://www.imagebam.com/image/6c78d433119774)http://thumbnails5.imagebam.com/3312/ecc28233119777.gif (http://www.imagebam.com/image/ecc28233119777)
http://thumbnails18.imagebam.com/3312/44233233119780.gif (http://www.imagebam.com/image/44233233119780)http://thumbnails8.imagebam.com/3312/ce24aa33119782.gif (http://www.imagebam.com/image/ce24aa33119782)
http://thumbnails9.imagebam.com/3312/13ab1f33119790.gif (http://www.imagebam.com/image/13ab1f33119790)http://thumbnails18.imagebam.com/3312/3e81e533119791.gif (http://www.imagebam.com/image/3e81e533119791)
http://thumbnails.imagebam.com/3312/79db7633119794.gif (http://www.imagebam.com/image/79db7633119794)http://thumbnails10.imagebam.com/3312/7900e533119799.gif (http://www.imagebam.com/image/7900e533119799)

some of my comparisons didn't make it through to my imagehost coz they were >3MB, but I can't really say that it ever made any visible difference on that BD(MPEG2).

Kazuya has been kind enough to encode some test patterns in 16-235 h264, so they could go through my Avisynth ffdshow pipeline like any other video.

left side is 8bit/right side is 16 :

http://thumbnails14.imagebam.com/3313/caf19c33121658.gif (http://www.imagebam.com/image/caf19c33121658)http://thumbnails19.imagebam.com/3313/38621933121661.gif (http://www.imagebam.com/image/38621933121661)
http://thumbnails17.imagebam.com/3313/47073033121664.gif (http://www.imagebam.com/image/47073033121664)http://thumbnails11.imagebam.com/3313/178c1d33121667.gif (http://www.imagebam.com/image/178c1d33121667)
http://thumbnails19.imagebam.com/3313/47817033121669.gif (http://www.imagebam.com/image/47817033121669)http://thumbnails19.imagebam.com/3313/ebd8a133121672.gif (http://www.imagebam.com/image/ebd8a133121672)

[leeperry]

last but not least, I wanted to try 0-255 native content so we could see if xvYCC would make an even bigger difference....but ImageSource() seems to convert to TV levels anyway :confused:

so left side is 8bit, right side is 16. Both using this script :
ImageSource("C:\Faces_1920x1080.jpg",pixel_type="RGB32")
ConvertToYUY2(matrix="pc.709")
t3dlut("C:\test_16bit.3dlut",threads=4)

http://thumbnails15.imagebam.com/3313/487c7733121647.gif (http://www.imagebam.com/image/487c7733121647)http://thumbnails11.imagebam.com/3313/90783233121651.gif (http://www.imagebam.com/image/90783233121651)
http://thumbnails.imagebam.com/3313/fc70fa33121654.gif (http://www.imagebam.com/image/fc70fa33121654)http://thumbnails8.imagebam.com/3313/6ed56833121656.gif (http://www.imagebam.com/image/6ed56833121656)

if anyone knows how to encode image files in 0-255 h264, I'll update my posts. The test patterns are available here : http://www.burosch.de/shop/shop_content.php?coID=136&

[tetsuo55]

so I've compared 8bit against 16bit dithered.

native BD content w/o rescaling in ffdshow or HR :

http://thumbnails14.imagebam.com/3312/7e4d6333119757.gif (http://www.imagebam.com/image/7e4d6333119757) http://thumbnails15.imagebam.com/3312/09600333119759.gif (http://www.imagebam.com/image/09600333119759)http://thumbnails19.imagebam.com/3312/8e730533119762.gif (http://www.imagebam.com/image/8e730533119762)

that's the cr3dlut .ini :
# Example input file for cr3dlut v2.0 and up
#
# Settings for creating a 3D LUT for a Custom display
# (including YCbCr->RGB conversion)
#

# Do not mess up with these settings
Chromatic_Adaptation 1
Input_Bit_Depth 8
Input_YCbCr_Full_Range 0
Input_Gamma 1
Output_Bit_Depth 16
Output_YCbCr_Matrix 0
Output_RGB_Black_White 0 255

# You can change the following settings

# for BT.709 use '1'; for BT.601 use '3'
Input_YCbCr_Matrix 1

# for HDTV use '0'; for EBU use '2'; for SMPTE-C use '3'
Input_Primaries 0

# put here your primaries and white point xy coordinates, RGBW order.
Output_Primaries 1

# if you want to try other gamma values change the 0.45 value accordingly
Output_Gamma 1

left side is
yv12toyuy2(itype=2,threads=4)
t3dlut("C:\test_8bit.3dlut",threads=4)

right side is
yv12toyuy2(itype=2,threads=4)
t3dlut("C:\test_16bit.3dlut",threads=4)

http://thumbnails18.imagebam.com/3312/4bf71433119768.gif (http://www.imagebam.com/image/4bf71433119768)http://thumbnails14.imagebam.com/3312/9c07d933119771.gif (http://www.imagebam.com/image/9c07d933119771)
http://thumbnails3.imagebam.com/3312/6c78d433119774.gif (http://www.imagebam.com/image/6c78d433119774)http://thumbnails5.imagebam.com/3312/ecc28233119777.gif (http://www.imagebam.com/image/ecc28233119777)
http://thumbnails18.imagebam.com/3312/44233233119780.gif (http://www.imagebam.com/image/44233233119780)http://thumbnails8.imagebam.com/3312/ce24aa33119782.gif (http://www.imagebam.com/image/ce24aa33119782)
http://thumbnails9.imagebam.com/3312/13ab1f33119790.gif (http://www.imagebam.com/image/13ab1f33119790)http://thumbnails18.imagebam.com/3312/3e81e533119791.gif (http://www.imagebam.com/image/3e81e533119791)
http://thumbnails.imagebam.com/3312/79db7633119794.gif (http://www.imagebam.com/image/79db7633119794)http://thumbnails10.imagebam.com/3312/7900e533119799.gif (http://www.imagebam.com/image/7900e533119799)

some of my comparisons didn't make it through to my imagehost coz they were >3MB, but I can't really say that it ever made any visible difference.

Kazuya has been kind enough to encode some test patterns in 16-235 h264, so they could go through my Avisynth ffdshow pipeline like any other video.

left side is 8bit/right side is 16 :

http://thumbnails14.imagebam.com/3313/caf19c33121658.gif (http://www.imagebam.com/image/caf19c33121658)http://thumbnails19.imagebam.com/3313/38621933121661.gif (http://www.imagebam.com/image/38621933121661)
http://thumbnails17.imagebam.com/3313/47073033121664.gif (http://www.imagebam.com/image/47073033121664)http://thumbnails11.imagebam.com/3313/178c1d33121667.gif (http://www.imagebam.com/image/178c1d33121667)
http://thumbnails19.imagebam.com/3313/47817033121669.gif (http://www.imagebam.com/image/47817033121669)http://thumbnails19.imagebam.com/3313/ebd8a133121672.gif (http://www.imagebam.com/image/ebd8a133121672)

I don't think you used the best samples, both of those movies are mostly computer generated using 8bit rgb(or lower) textures

[yesgrey]

in itype=2 I get an error msg w/ either b/c=0.0,1.0 or b/c="0.0,1.0"....anyone's got an idea how to use that?

b=0.0, c=1.0

[leeperry]

I don't think you used the best samples, both of those movies are mostly computer generated using 8bit rgb(or lower) textures
I can't seem to find any free imagehost that accepts files bigger than 3MB, so I've put the remaining comparisons there : http://rapidshare.com/files/223274625/t3dlut.rar.html

except for the inflated filesize in 16bit, they look identical to me.

but in the test patterns, gray ramps on top look better in the photo w/ the 2 women in 16bit.....very slight improvement, though.

b=0.0, c=1.0
ah! thanks. didn't think about that one :rolleyes:

so what would be some good value ? 0.0/0.75? or 0.0/0.60?


BTW, I was trying to get this test pattern through HR and mVR....but colors don't come off the same way ?! :o

http://www.image-load.eu/out.php/t158401_Zone70pp19201080.jpg (http://www.image-load.eu/out.php/i158401_Zone70pp19201080.jpg)

the mVR .PNG is 4MB, so I had to make a RAR again w/ the 2 files :
http://rapidshare.com/files/223278460/compare.rar.html

the idea was to check for sharpening, and apart from the different colors this should be OK :)

the test pattern is available here(it's the last one in the MP4 container) : http://www.megaupload.com/?d=B6R60TE0

in HR, I put this in ffdshow :
yv12toyuy2(itype=2,threads=4)
t3dlut("C:\madVR\HD.3dlut")

and in mVR 0.5, I disabled ffdshow and used the same "HD.3dlut"....so I can't really explain this color difference :o

I tried it both in the MP4 file, and using ImageSource()...colors always come off different..

[yesgrey]

so what would be some good value ? 0.0/0.75? or 0.0/0.60?
You could try 0.0/0.5 or 1.0/0.0.

[yesgrey]

BTW, I was trying to get this test pattern through HR and mVR....but colors don't come off the same way ?! :o
HR image is using Video levels, and madVR image is using PC levels. Check your settings.;)

[leeperry]

@tritical: any chance you could make yv12toyuy2() fallback to itype=0 if it's not mod4 please? coz I can't make rules in ffdshow on that criteria, so yv12toyuy2() can't really be used on all videos at this point :(

[leeperry]

You could try 0.0/0.5 or 1.0/0.0.
b/c -

Adjusts properties of cubic interpolation (itype=2). Same as Avisynth's BicubicResize filter.

default: 0.0,0.75
http://avisynth.org/mediawiki/Resize
BicubicResize is similar to BilinearResize, except that instead of a linear filtering function it uses the Mitchell-Netravali two-part cubic. The parameters b and c can be used to adjust the properties of the cubic, they are sometimes referred to as "blurring" and "ringing" respectively.

With b = 0 and c = 0.75 the filter is exactly the same as VirtualDub's "precise bicubic" and the results are identical except for the VirtualDub scaling problem mentioned above. The default is b = 1./3. and c = 1./3., which were the values recommended by Mitchell and Netravali as yielding the most visually pleasing results in subjective tests of human beings. Larger values of b and c can produce interesting op-art effects--for example, try b = 0 and c = -5.

If you are magnifying your video, you will get much better-looking results with BicubicResize than with BilinearResize. However, if you are shrinking it, you are probably just as well off, or even better off, with BilinearResize. Although VirtualDub's bicubic filter does produce better-looking images than its bilinear filter, this is mainly because the bicubic filter sharpens the image, not because it samples it better. Sharp images are nice to look at--until you try to compress them, at which point they turn nasty on you very quickly. The BicubicResize default doesn't sharpen nearly as much as VirtualDub's bicubic, but it still sharpens more than the bilinear. If you plan to encode your video at a low bitrate, I wouldn't be at all surprised if BilinearResize yields better quality.

You have to set "b + 2 * c = 1" for the numerically most accurate filter. This gives for b = 0 the maximum value for c = 0.5, which is the Catmull-Rom spline and a good suggestion for sharpness.

From c>0.6 the filter starts to "ring". You won't get real sharpness, what you'll get is crispening like on a TV set. No negative values are allowed for b. Then stay on b=0.

I'll try to force 0.0/0.5, because it's the default values in t3dlut() but not in rgb3dlut(0.75)

[leeperry]

HR image is using Video levels, and madVR image is using PC levels. Check your settings.;)
ah indeed, I didn't pay attention that there was some new settings in mVR...

and whatever I set "c" to 0.5/0.6/0.75/1.0 in rgb3dlut(), this test pattern gives CRC identical outputs : http://www.image-load.eu/out.php/i158401_Zone70pp19201080.jpg

[tritical]

That test pattern is grayscale. b and c only effect chroma upsampling.

Also, I didn't say it in the readme, but t3dlut expects raw lut files (not 3dlut format) to be stored such that the lookup is calculated r<<16+g<<8+b (which is the same as the 3dlut format). While rgb3dlut expects g<<16+b<<8+r. So lut files created with ddcc can't be used with t3dlut. However, yesgrey3's app can create LUT's the same as ddcc that will work with t3dlut so I don't plan to change ddcc.

@tritical: any chance you could make yv12toyuy2() fallback to itype=0 if it's not mod4 please? coz I can't make rules in ffdshow on that criteria, so yv12toyuy2() can't really be used on all videos at this point

I will think it about it, but why not just use converttoyuy2()? Especially if you are willing to accept itype=0 (nearest neighbor) in yv12toyuy2().

[leeperry]

That test pattern is grayscale. b and c only effect chroma upsampling.

Also, I didn't say it in the readme, but t3dlut expects raw lut files (not 3dlut format) to be stored such that the lookup is calculated r<<16+g<<8+b (which is the same as the 3dlut format). While rgb3dlut expects g<<16+b<<8+r. So lut files created with ddcc can't be used with t3dlut. However, yesgrey3's app can create LUT's the same as ddcc that will work with t3dlut so I don't plan to change ddcc.

I will think it about it, but why not just use converttoyuy2()? Especially if you are willing to accept itype=0 (nearest neighbor) in yv12toyuy2().
oh OK, so t3dlut only works with .txt LUT's, not .3dlut...now that's confusing...how come it even works then?

well, the idea is that most files are mod4 anyway, so you can use yv12toyuy2(itype=2) but sometimes you have files that are mod2...and IIRC only itype=0 works on mod2.

so if yv12toyuy2() could either fall back to a mod2-compatible mode, or to ConvertToYUY2(it's GPL right? only a matter of cut/paste?)...that would make life easier when you run them in ffdshow instead of getting an error msg.

:thanks:

[yesgrey]

Also, I didn't say it in the readme, but t3dlut expects raw lut files (not 3dlut format) to be stored such that the lookup is calculated r<<16+g<<8+b (which is the same as the 3dlut format). While rgb3dlut expects g<<16+b<<8+r. So lut files created with ddcc can't be used with t3dlut. However, yesgrey3's app can create LUT's the same as ddcc that will work with t3dlut so I don't plan to change ddcc.
tritical,
cr3dlut currently works like this:
-3dlut format: r<<16+g<<8+b
-raw format: g<<16+b<<8+r
I made it like this to let cr3dlut create files for both t3dlut and rgb3dlut. I thought that t3dlut's raw format was the same as rgb3dlut... Do you think I should change my raw format to r<<16+g<<8+b?

but why is the default 0.75 in rgb3dlut/yv12toyuy2 and 0.5 in t3dlut/yv12toyuy2? apparently 0.75 is sharper but 0.5 is "neutral" ?
Avisynth's help says that 0.0/0.5 is the most sharp without ringing. I have suggested tritical to change the defaults to these values, and it seems he agreed with the idea.;)

and is it my imagination or using itype=2 in yv12toyuy2()/rgb3dlut() make the picture sharper than the default options?
Yes it makes. The default in rgb3dlut is itype=1.
You should start reading the help more thoroughly...;)

[leeperry]

Avisynth's help says that 0.0/0.5 is the most sharp without ringing. I have suggested tritical to change the defaults to these values, and it seems he agreed with the idea.;)


Yes it makes. The default in rgb3dlut is itype=1.
You should start reading the help more thoroughly...;)
1) OK, I'll try several values w/ a test pattern tomorrow but indeed 0.5 looks good in yv12toyuy2/rg3dlut and ffdshow(when downscaling BD to 720p)
2) I did. but it's not too self-explanatory that "linear interpolation" doesn't look as good as "Mitchell-Netravali two-part cubic interpolation"...especially when ppl advise to have very blurry chroma :o

And what are the real world differences between the different itypes of yv12toyuy2() and ConvertToYUY2()...the way they align chroma pixels and how sharp they upsample chroma I think? so if itype2 0.5 is the best, how does Convert() fare against it?

[yesgrey]

And what are the real world differences between the different itypes of yv12toyuy2() and ConvertToYUY2()
I think ConvertToYUY2() is the same as yv12toyuy2(itype=1).

[tritical]

oh OK, so t3dlut only works with .txt LUT's, not .3dlut...now that's confusing...how come it even works then?
t3dlut works with 3dlut format files (it doesn't rely on the extension to detect them), and with raw lut files (i.e. the lut file contains just the entries without a header). rgb3dlut worked only with raw lut files since it was created before the 3dlut format. The difference is that rgb3dlut expects the entries in the raw lut file to be stored in different positions than t3dlut does.

I made it like this to let cr3dlut create files for both t3dlut and rgb3dlut. I thought that t3dlut's raw format was the same as rgb3dlut... Do you think I should change my raw format to r<<16+g<<8+b?
No, I think the way it is is good. If the user wants to use t3dlut they can always output as 3dlut format, or if they want to use rgb3dlut they can output as raw. For uses that don't need t3dlut's extra features, rgb3dlut is better since it is faster.

And what are the real world differences between the different itypes of yv12toyuy2() and ConvertToYUY2()...the way they align chroma pixels and how sharp they upsample chroma I think? so if itype2 0.5 is the best, how does Convert() fare against it?
As yesgrey3 said, converttoyuy2() is the same as yv12toyuy2 with itype=1 and yv12toyuy2's default chroma placement settings. And yes, the differences between converttoyv12() and yv12toyuy2() are simply the upsampling method and chroma placement assumption.

There isn't really a best setting for b/c. Your question is the same as how does bilinearresize() compare to bicubicresize()? and how do different values of b/c effect bicubicresize()?... just applied to chroma resampling. I personally can't tell the difference between converttoyuy2() and converttoyv12(itype=2), assuming the same chroma placement, except when zooming in under extreme cases.

Also, why can't your script use a line like:

mod_4_check() ? yv12toyuy2() : converttoyuy2()

Obviously you'd have to script your own mod_4_check, but it should be pretty easy.

[leeperry]

alright, I'll just stick to ConvertToYUY2() + rgb3dlut(itype=2,b=0.00,c=0.5)...no worries, PQ is great and it's very fast :)

[leeperry]

ah well, maybe it's psychological but I still find yv12toyuy2(itype=2,threads=4,b=0.00,c=0.5) more appealing than ConvertToYUY2()

there's no chance you could get it to work w/ mod2? like padding additional lines and removing them afterwards?

madshi advises to go for blurry chroma(softcubic), so I'm not sure...maybe it's all in my head :o

[madshi]

Avisynth's help says that 0.0/0.5 is the most sharp without ringing. I have suggested tritical to change the defaults to these values, and it seems he agreed with the idea.;)
Unfortunately the AviSynth help is wrong. 0.0/0.5 *does* ring. Not as much as e.g. 0.0/0.75, and not as much as e.g. Lanczos, but 0.0/0.5 does ring. If you don't want ringing, use e.g. 0.33/0.33 (Mitchell-Netravali). Or maybe 0.5/0.5 (madVR calls this "SoftCubic50"). Personally, I even prefer 1.0/0.0 which is *very* soft, but totally aliasing free and ringing free.

[leeperry]

I even prefer 1.0/0.0 which is *very* soft, but totally aliasing free and ringing free.
indeed ConvertToYUY2() + rgb3dlut(itype=2,b=1.0,c=0.0) look really great http://forum-images.hardware.fr/images/perso/otakonleboss.gif

maybe it appears sharper because the chroma is very soft, and LSF only sharpens the luma I think...so at least the chroma doesn't "interfere".

ffdshow only offers one negative parameter for bicubic downscaling(from 1080p to 720p) so I've set it to -0.5....looks good too I think, not too sharp, not too blurry :)

[leeperry]

@tritical: so is it normal that ddcc in realtime and rgb3dlut w/ a ddcc RGB LUT don't output the same colors ? :(

http://forum.doom9.org/showpost.php?p=1273996&postcount=288

cr3dlut still suffers from the same kind of problem, I guess I'll be using ddcc in realtime in HR for a while...at least colors are great(identical to the PS script) and HR is very smooth in 48Hz(unlike EVR/VMR/etc) :)

that'd be great if you could let ddcc output identical RGB LUT's to ddcc in realtime, if any possible :o

:thanks:

[tritical]

The only difference between ddcc directly and ddcc's lut in rgb3dlut is the handling of out-of-gamut colors (capping of channels individually vs adjusting all 3 to maintain hue). Didn't yesgrey3 add an option in his program to do it both ways? Does that not work as expected?

[yesgrey]

Didn't yesgrey3 add an option in his program to do it both ways? Does that not work as expected?
Yes, I did. He's right, though, there is a strange thing with the results. I'm investigating to see what's happenning...

Another question.
Do you think it should be preferable (more accurate) to use t3dlut to convert from YCbCr->YCbCr instead of using ColorMatrix?

[tritical]

Yes, I did. He's right, though, there is a strange thing with the results. I'm investigating to see what's happenning...

Alright. I think I did a test where I changed the lut creation code in ddcc to cap the channels individually and then compared rgb3dlut with that lut vs ddcc directly and the outputs were identical. I'll go back and check though.

Do you think it should be preferable (more accurate) to use t3dlut to convert from YCbCr->YCbCr instead of using ColorMatrix?
Maybe. Colormatrix basically upsamples chroma using nearest neighbor, and does the conversion using integer approximation for speed. So mathematically t3dlut would be more accurate, but I don't know if the difference would be noticeable.

[yesgrey]

I'll go back and check though.
I don't know yet if it's a problem with both ddcc and cr3dlut 3DLUT's or if it's only a problem with cr3dlut... Maybe it's better if you wait until I see if the problem is the samew with both...;)

[leeperry]

The only difference between ddcc directly and ddcc's lut in rgb3dlut is the handling of out-of-gamut colors (capping of channels individually vs adjusting all 3 to maintain hue). Didn't yesgrey3 add an option in his program to do it both ways? Does that not work as expected?
ok, but if you try my settings, you can see that the green levels are completely burned using a LUT...

yesgrey has changed this behavior in his latest build of cr3dlut, but colors are still quite off..mostly red is too saturated this time.

if you could please make the out-of-gamut colors behave the same exact way in a LUT as they do w/ realtime ddcc..that'd be really great :)

I very much enjoyed ConvertToYUY2() + rgb3dlut(itype=2,b=1.0,c=0.0)...it doesn't use much CPU and the chroma is very soft, which makes the luma cleaner and clearer to look at.

:thanks:

[leeperry]

@yesgrey: if you ever plan to import Color.HCFR saturations into cr3dlut, I've just found out that you can measure them on 10 points :

http://img19.imageshack.us/img19/5627/satz.png

http://rapidshare.com/files/233048376/05-14_HC3100.chc.html

and hopefully if tritical has time to find the diff between LUT/realtime out-of-gamut colors management, will you be able to export the change to cr3dlut? seems like mVR will be ripe anytime soon, and I'd like to play too :)

:thanks:

[leeperry]

hi guys, any news by any chance? :)

ideally if ddcc could treat out-of-gamut colors in a LUT the same way it does in realtime, that would enable far more ppl(including DLP users like me) to be able to use a LUT to convert gamuts...atm I'm playing around w/ 48fps content, and I simply don't have enough CPU cycles to run ddcc in realtime :(

:thanks:

[tritical]

I changed ddcc to clip in the same fashion during lut file creation that it does during realtime usage, and when using the resulting lut file with rgb3dlut the output is identical to ddcc realtime. I put up a new ddcc version with the ability to set the clipping method during lut creation on my website. However, the lut format that ddcc uses for output isn't compatible with anything but rgb3dlut. If I get time I might add the ability to output in .3dlut format. It'll take more investigating to figure out what the difference is between ddcc and cr3dlut.

[leeperry]

awesome, thank you tritical!

I was about to give up on gamut conversion for now on 48p content, this definitely cheers me up! http://forum-images.hardware.fr/images/perso/sniperr.gif

ideally, if it could also output YUY2 LUT's, that'd be even better...or maybe yesgrey could commit your code change to cr3dlut? :)

[tritical]

That's just it, cr3dlut already has that change (Out_Of_Gamut_Clipping=0). Anyways, I did some quick tests, and using rgb3dlut with cr3dlut's lut file is giving me exactly the same results as with ddcc's lut file (for both Out_Of_Gamut_Clipping=0/oog=0 and Out_Of_Gamut_Clipping=1/oog=1) . Could you give an example that is producing different output. I'll see if there is any difference with cr3dlut's .3dlut output loaded with t3dlut.

Using t3dlut with rgb input and cr3dlut's .3dlut output is fine.

There seem to be some major differences in flat color areas between:

convertToRGB32().rgb3dlut()
converttoyuy2().t3dlut()

the second one using cr3dlut's .3dlut output. I'll do some more testing.

[leeperry]

well, did you try this? http://forum.doom9.org/showpost.php?p=1273996&postcount=288

the latest build of cr3dlut definitely doesn't output the same colors as ddcc in realtime(even w/ Out_Of_Gamut_Clipping=0), and yesgrey confirmed it. red is too saturated and the green gamma is still way too bright.

I will give a go at the new version, w/ the settings from this url :)

PS: oops, I gave all the detailed files to yesgrey, but didn't post them on the forum...I'll look into it tomorrow morning and will post every file so you can reproduce(if a problem still occurs).

[tritical]

I believe I tested that exact example, or close to it, using your monitor values that you posted a while ago. cr3dlut file used to create "cr3dlut":

Chromatic_Adaptation 1
Out_Of_Gamut_Clipping 0
Input_Bit_Depth 8
Input_YCbCr_Matrix 0
Input_YCbCr_Full_Range 0
Input_RGB_Black_White 0 255
Input_Primaries 3
Input_Gamma 9 1.0 0.0 0.45 0.0
Output_Bit_Depth 8
Output_YCbCr_Matrix 0
Output_RGB_Black_White 0 255
Output_YCbCr_Full_Range 0
Output_Primaries 9 0.656 0.332 0.342 0.610 0.151 0.071 0.311 0.330
Output_Gamma 9 1.0 0.0 0.45 0.0

ddcc(chr_i=3,gam_i=5,ofile="t.txt",lutfile="ddcclut",oog=0)

t.txt is:

0.151
0.071
0.342
0.610
0.656
0.332
0.311
0.330
1
0.0
0.45
0.0
0.0

then tested the difference between:

converttorgb32().ddcc(chr_i=3,gam_i=5,ofile="t.txt")
converttorgb32().rgb3dlut(lutfile="cr3dlut")
converttorgb32().rgb3dlut(lutfile="ddcclut")

and all were the same. I also had cr3dlut output a matching .3dlut format file, and used it with t3dlut. That gave the same output. So far the only differences I have found are when cr3dlut includes the yuv->rgb conversion.

[leeperry]

that was something along the lines

# Example input file for cr3dlut v2.0 and up
#
# Settings for creating a 3D LUT for a Custom display
# (including YCbCr->RGB conversion)
#

# Do not mess up with these settings
Chromatic_Adaptation 1
Out_Of_Gamut_Clipping 0
Input_Bit_Depth 8
Input_YCbCr_Full_Range 1
Input_Gamma 9 1.0 0.0 0.45 0.0
Output_Bit_Depth 8
Output_YCbCr_Matrix 0
Output_RGB_Black_White 0 255

# You can change the following settings

# for BT.709 use '1'; for BT.601 use '3'
Input_YCbCr_Matrix 1

# for HDTV use '0'; for EBU use '2'; for SMPTE-C use '3'
Input_Primaries 3

# put here your primaries and white point xy coordinates, RGBW order.
Output_Primaries 9 0.6530 0.3300 0.3390 0.6120 0.1500 0.0700 0.3120 0.3280

# if you want to try other gamma values change the 0.45 value accordingly
Output_Gamma 9 1.0 0.0 0.45 0.0

0.150
0.070
0.339
0.612
0.653
0.330
0.312
0.328
1
0.0
0.45
0.0
0.0

even this last bit didn't yield the same colors between ddcc realtime/ddcc LUT w/ this test pattern : http://www.megaupload.com/?d=F8DES6ZN

[tritical]

Do you get the same output when you don't have cr3dlut perform the yuv->rgb conversion? From my testing it appears that differences only arise when cr3dlut's lut includes the yuv->rgb conversion as opposed to converting to rgb in avisynth first.

EDIT 1:
After some more tests, cr3dlut's yuv->rgb lut works when no gamma/gamut changes are required... i.e. only doing yuv->rgb conversion. Once the gamma/gamut conversions are thrown in the values are off.

EDIT 2:
I think there is a bug in v2.1 of cr3dlut. I built v2.0 from the source, and building a lut with it, "lut.3dlut", using the following config file:

Chromatic_Adaptation 1
Input_Bit_Depth 8
Input_YCbCr_Matrix 1
Input_YCbCr_Full_Range 0
Input_RGB_Black_White 0 255
Input_Primaries 3
Input_Gamma 9 1.0 0.0 0.45 0.0
Output_Bit_Depth 8
Output_YCbCr_Matrix 0
Output_YCbCr_Full_Range 0
Output_RGB_Black_White 0 255
Output_Primaries 9 0.656 0.332 0.342 0.610 0.151 0.071 0.311 0.330
Output_Gamma 9 1.0 0.0 0.45 0.0

the output of these two match:

converttorgb32(interlaced=false,matrix="Rec709").ddcc(chr_i=3,gam_i=5,ofile="t.txt",ulut=true,lutfile="t",oog=1)
converttoyuy2(interlaced=false).t3dlut(itypeU=1,cplaceU=0,lutfile="lut.3dlut",destcs=2)

but building a lut file using v2.1 with this config file (the only difference is the Out_Of_Gamut_Clipping=1 which shouldn't change anything):

Chromatic_Adaptation 1
Out_Of_Gamut_Clipping 1
Input_Bit_Depth 8
Input_YCbCr_Matrix 1
Input_YCbCr_Full_Range 0
Input_RGB_Black_White 0 255
Input_Primaries 3
Input_Gamma 9 1.0 0.0 0.45 0.0
Output_Bit_Depth 8
Output_YCbCr_Matrix 0
Output_YCbCr_Full_Range 0
Output_RGB_Black_White 0 255
Output_Primaries 9 0.656 0.332 0.342 0.610 0.151 0.071 0.311 0.330
Output_Gamma 9 1.0 0.0 0.45 0.0

gives completely different output. Hopefully yesgrey3 can look into it.

[yesgrey]

So far the only differences I have found are when cr3dlut includes the yuv->rgb conversion.
Yes, I have already reached this point too. I'm still trying to arrange some free time to find what's the problem...

Edit: tritical, thanks for your help, I will test with your examples.;)

[leeperry]

http://thumbnails14.imagebam.com/3767/81271937662519.gif (http://www.imagebam.com/image/81271937662519) http://thumbnails10.imagebam.com/3767/3dc7c437662523.gif (http://www.imagebam.com/image/3dc7c437662523) http://thumbnails14.imagebam.com/3767/99738b37662525.gif (http://www.imagebam.com/image/99738b37662525)
http://thumbnails6.imagebam.com/3767/26e99037662527.gif (http://www.imagebam.com/image/26e99037662527) http://thumbnails17.imagebam.com/3767/d21d9f37662529.gif (http://www.imagebam.com/image/d21d9f37662529) http://thumbnails15.imagebam.com/3767/4a5c4837662531.gif (http://www.imagebam.com/image/4a5c4837662531)

1)
colorYUV(levels="tv->pc") in ffdshow
dithered 0-255 RGB32HQ in ffdshow > HR

2)
colorYUV(levels="tv->pc") in ffdshow
dithered 0-255 RGB32HQ in ffdshow > VMR9 Renderless (mixer mode disabled)

1 and 2 match!

3)
colorYUV(levels="tv->pc") in ffdshow
dithered 0-255 RGB32HQ in ffdshow > VMR9 Renderless (mixer mode disabled)
+ PS script to convert gamut to SMPTE-C

4)
colorYUV(levels="tv->pc")
ConvertToRGB32(matrix="pc.709")
ddcc(chr_i=3,gam_i=5,ofile="X:\HC3100.txt",threads=4,opt=1)
RGB32 in ffdshow > HR

===

ddcc(chr_i=3,gam_i=5,ofile="X:\HC3100.txt",lutfile="X:\3100_SMPTE-C_oog0.txt",oog=0,threads=4,opt=1)

ddcc(chr_i=3,gam_i=5,ofile="X:\HC3100.txt",lutfile="X:\3100_SMPTE-C_oog1.txt",oog=1,threads=4,opt=1)

===

5)
colorYUV(levels="tv->pc")
ConvertToRGB32(matrix="pc.709")
rgb3dlut(lutfile="X:\3100_SMPTE-C_oog0.txt",threads=4,itype=2,b=1.0,c=0.0)
RGB32 in ffdshow > HR

6)
colorYUV(levels="tv->pc")
ConvertToRGB32(matrix="pc.709")
rgb3dlut(lutfile="X:\3100_SMPTE-C_oog1.txt",threads=4,itype=2,b=1.0,c=0.0)
RGB32 in ffdshow > HR

3/4/5 match! well done :)

I don't see how 6 would be useful to anyone ?! maybe if your original gamut can fully include the new one?

well, yesgrey knows there's some problems in cr3dlut, I'll be most happy to try any new version :thanks:

[leeperry]

but, rgb3dlut() used to be faster than t3dlut() in the past...not the case anymore :confused:

ConvertToRGB32(matrix="pc.709")
=275

ConvertToRGB32(matrix="pc.709")
t3dlut(lutfile="X:\3100_SMPTE-C_oog0.txt",threads=4)
=200

ConvertToRGB32(matrix="pc.709")
rgb3dlut(lutfile="X:\3100_SMPTE-C_oog0.txt",threads=4,itype=2,b=1.0,c=0.0)
=198

or it's only much faster in YUY2, but not in RGB32?

EDIT: ConvertToRGB32() is really wasting CPU cycles here :(
I was able to get 480fps from
ConvertToYUY2()
rgb3dlut(lutfile="C:\HC3100_SMPTE-C.txt",itype=2,threads=4)

http://forum.doom9.org/showpost.php?p=1275498&postcount=405

[yesgrey]

I don't see how 6 would be useful to anyone ?! maybe if your original gamut can fully include the new one?
That's the most common situation. In general, the color gamuts of the new projectors are much larger than the standards...

[leeperry]

ok well, I wonder how it'd differ to oog=0 in that case...but anyway I'm happy w/ oog=0

all the 720p DLP have a lack of green AFAIK..

anyway, if I got it right cr3dlut 2.0 is fine? all it needs is oog=0 basically?

[tritical]

Speed difference between t3dlut and rgb3dlut should be pretty small for rgb->rgb and using multiple threads. rgb3dlut should be faster for yuy2->rgb, as you've observed.

I also found a bug in t3dlut with yuy2 input and itypeU=1/cplaceU=0... the right edge pixels are not handled correctly.

[yesgrey]

I've just released here (http://yesgrey3.totalh.com/) cr3dlut v2.2.
v2.2 - 2009/06/03
- Fixed: when performing both Y'CbCr->R'G'B' and color gamut conversions some errors could occur


Once again, I've just released the executable. The source code is still v2.0.
This is just an interim version due to a problem described by leeperry. I'm currently working in adding custom gamma curves to cr3dlut, so the source code is still a bit messy. In the next release, I should have time to clean up the source code...

Comments/suggestions are welcome.

[73ChargerFan]

Article on ars technica: Expanding the gamut: Sharp to increase color range of LCDs (http://arstechnica.com/gadgets/news/2009/06/expanding-the-gamut-sharp-to-increase-color-range-of-lcds.ars)


The company's new design adds cyan and yellow to the standard red, green, and blue subpixels found in other LCDs. This allows the display to reproduce nearly the color defined by the Pointer color space, a standard based on measurements of real world inks, paints, and pigments. Sharp claims this makes images displayed on the device "identical in appearance to real-world objects."

[leeperry]

@yesgrey3: w00t, same colors as ddcc/ogg0 would be a god bless http://forum-images.hardware.fr/images/perso/otakonleboss.gif

@73ChargerFan: they keep increasing the gamuts, but the movies are still SMPTE-C....and usually they don't offer any option to shrink it(or poorly, like on the latest JVC pj) :rolleyes:
well, Epson starts to offer options to switch gamuts on their consumer projectors, but you have to dig into menus and submenus to reach it...only the Samsung shown in the OP has direct access buttons on its remote control.

[leeperry]

oh my, oh my....it's working :D

http://thumbnails19.imagebam.com/3777/3a81a937766600.gif (http://www.imagebam.com/image/3a81a937766600) http://thumbnails6.imagebam.com/3777/dca22037766601.gif (http://www.imagebam.com/image/dca22037766601) http://thumbnails16.imagebam.com/3777/ae452a37766602.gif (http://www.imagebam.com/image/ae452a37766602)

1)
colorYUV(levels="tv->pc")
ConvertToRGB32(matrix="pc.709")

2)
colorYUV(levels="tv->pc")
ConvertToRGB32(matrix="pc.709")
ddcc(chr_i=3,gam_i=5,ofile="X:\HC3100.txt",threads=4,opt=1)

3)
colorYUV(levels="tv->pc")
ConvertToYUY2()
rgb3dlut(lutfile="X:\HC3100_SMPTE-C.txt",threads=4,itype=2,b=1.0,c=0.0)

HC3100.txt
0.150
0.070
0.339
0.612
0.653
0.330
0.312
0.328
1
0.0
0.45
0.0
0.0

# Example input file for cr3dlut v2.0 and up
#
# Settings for creating a 3D LUT for a Custom display
# (including YCbCr->RGB conversion)
#

# Do not mess up with these settings
Chromatic_Adaptation 3
Out_Of_Gamut_Clipping 0
Input_Bit_Depth 8
Input_YCbCr_Full_Range 1
Input_Gamma 9 1.0 0.0 0.45 0.0
Output_Bit_Depth 8
Output_YCbCr_Matrix 0
Output_RGB_Black_White 0 255

# You can change the following settings

# for BT.709 use '1'; for BT.601 use '3'
Input_YCbCr_Matrix 1

# for HDTV use '0'; for EBU use '2'; for SMPTE-C use '3'
Input_Primaries 3

# put here your primaries and white point xy coordinates, RGBW order.
Output_Primaries 9 0.6530 0.3300 0.3390 0.6120 0.1500 0.0700 0.3120 0.3280

# if you want to try other gamma values change the 0.45 value accordingly
Output_Gamma 9 1.0 0.0 0.45 0.0

I will try BT.601 later on, and also the different other options...but I've never been able to see any difference between ddcc and the PS script, to me they go different ways to achieve the same result :o

way to go yesgrey, thanks a bunch! http://forum-images.hardware.fr/images/perso/ginie.gif

[Neeto]

@yesgrey: if you ever plan to import Color.HCFR saturations into cr3dlut, I've just found out that you can measure them on 10 points :

http://img19.imageshack.us/img19/5627/satz.png

http://rapidshare.com/files/233048376/05-14_HC3100.chc.html

and hopefully if tritical has time to find the diff between LUT/realtime out-of-gamut colors management, will you be able to export the change to cr3dlut? seems like mVR will be ripe anytime soon, and I'd like to play too :)

:thanks:

Do you have to do the import from Color.HCFR?
Perhaps a new "10" Custom type with the first row being the same as format as "9" and muliple following rows with each row being the parameters for a particular pecentage saturation?
I guess this means messing with the calcuations to produce the 3dlut quite a lot ;-)

:thanks: for all the work!

[Neeto]

I've been playing around with rgb3dlut and thougth I had it working just fine (silly me!)
Take a look at the following two images which are a crowd seen from an Elton John One Night Only NTSC DVD played using MPC-HC (v1.2.1160) with EVR, fddshow (revsion 3014), using libmpeg2 codec to a ATI 4650 card
The first is with AviSynth off & thus no rgb3dlut & the second is with AViSynth on & thus rgb3dlut doing its magic.

http://thumbnails10.imagebam.com/4251/d0d52c42503305.gif (http://www.imagebam.com/image/d0d52c42503305)

http://thumbnails10.imagebam.com/4251/5a07c542503306.gif (http://www.imagebam.com/image/5a07c542503306)

It is clear that something is badly wrong with the second image - all sorts of "squares" have appeared.
It is most noticable on sceens where it is low light with mainly single colors.

All I've done to gernerate these images is start MPC-HT with AviSynth on in ffdshow, then pause "Print Screen", save image, turn AviSynth off, and step a few frames foward "Print Screen", save image.

AviSynth Script:

Last=ffdshow_source()
isYV12() ? yv12toyuy2(2,true) : last
rgb3dlut("C:\Program Files\AviSynth 2.5\plugins\rgb3dlut_NTSC.raw")

I've tried:
1. outputing the image as NV12 (i.e. NV21,VU)
2. changing codec form libmpeg2 to libavcodec
3. changing to EVR Custom
4 change yv12toyuy2 interplotation method to 0, 1 & 2

Any other ideas?

Neeto

[Neeto]

This look like it the chroma upsampling bug people talk about with the ATI cards - can someone confirm?
By changing the to RGB on the ffdshow Output tab, rather than NV12 I was able to get to get back to a good picture, just with crappy de-interlacing.
But this does not make sense with the about "AviSynth/rgb3dlut" on & off giving better with AviSynth/rgb3dlut ON giving worse images - going to recheck everything.
I really want this to work with hardware de-interlacing as the advanced modes of de-interlacing are not supported in RGB on the ATI cards.

[yesgrey]

rgb3dlut_NTSC.raw
What settings have you used to create this file?

[leeperry]

This look like it the chroma upsampling bug people talk about with the ATI cards - can someone confirm?
[..]
I really want this to work with hardware de-interlacing as the advanced modes of de-interlacing are not supported in RGB on the ATI cards.
well, simple...the ATi will do ugly chroma upsampling if you don't send them RGB...or you'll have to use Leak's chroma blurring script in MPC, but it'll be compatible w/ VMR9/EVR only(the script also works in KMP).

[Neeto]

What settings have you used to create this file?

Input file to cr3dlut

#
# Settings for creating a 3D LUT for watching the following Video formats:
# NTSC DVD, NTSC SD Broadcast
# without any Display correction
# Includes YCbCr->RGB conversion preserving Video Levels

# Set input bitdepth
Input_Bit_Depth 8

# Set source video format
Input_Video_Format NTSC YCbCr

# Set output bitdepth
Output_Bit_Depth 8

# Set display video format
Output_Video_Format NTSC RGB_PC

[Neeto]

well, simple...the ATi will do ugly chroma upsampling if you don't send them RGB...or you'll have to use Leak's chroma blurring script in MPC, but it'll be compatible w/ VMR9/EVR only(the script also works in KMP).

This is the script from this post?
http://forum.doom9.org/showthread.php?p=1184975#post1184975

I was hoping to avoid using that script, but it seems wiser folk have trodden this path before me and concluded this is the best we can do.
:thanks:

[Neeto]

Point taken about the "lossy" - everything is a compromise at the moment as I've not been able to find a software delinterlacer that is as good as ATI hardware de-interlacer.
The half-resolution chorma of the YV12 is definitely lossy

DVD is in YCrCb which MPEG2 decoder output as RGB or YV12 (correct?) & rgb3dlut want it in YUY2, so need the yv12toyuy2, so the video is going to go:
DVD source> (Splitter):MPEG2-YCrCb > (MPEG2 decoder):YV12 > (yv12toyuy2):YUY2 > (rgb3dlut):RGB32 > (ffdshow output convert):NV12 > (ATI drivers, deinterlacing, scaling)

I'm really not sure how "lossy" this will be compared with something like (is this correct?):

DVD source> (Splitter):MPEG2-YCrCb > (MPEG2 decoder):YV12/RGB32/YUY2 > (ffdshow deinterlace):RGB32/YUY2 > (ffdshow scaling):RGB32/YUY2 > (rgb3dlut):RGB32 > ATI drivers

All I know so far is I don't like the output of the deinterlace (software).

I did some more testing to confirm my original findings.

Additional info:

MPEG2 codec must be libmpeg2 as the libmpeg2 codec seems to be the only one that supports it output being rendered set to the renderer in NV12 & RGB output, the libavcodec only seems to support it being sent in RGB.

Definitenly re-confirmed When playing MPC-HT :
1. With Avisynth/rgb3dlut "OFF" and NV12 output, image is "good" (as per first image posted above) and de-interlacing good
2. With Avisynth/rgb3dlut "ON" and NV12, image is "ugly" (as per second image posted above), and de-interlacing good
3. With Avisynth/rgb3dlut "OFF" and RGB output, image is "good" (as per first image posted above) and de-interlacing bad
4. With Avisynth/rgb3dlut "ON" and RGB output, image is "good" (as per first image posted above) and de-interlacing bad

From this I conclude that Avisynth/rgb3dlut is doing something to confuse the renderer/ATI hardware acceleration implementation, because the first test "proves(?)" that NV12 input can be handled correctly by the ATI drivers.

Perhaps it is somehow "triggering" the upscaling problem - perhaps levels? out of range colors?
Another thought is that if rgb3dlut is actually producing frames in RGB (someone confirm please or is it coming out of rgb3dlut in YUY2 - I'm starting to doubt the most basic of assumptions) then ffdshow will need to convert RGB to NV12 & perhaps this is where the problem is.
Are there any avisynth filters to convert back to NV12? - I've not been able to find any.

Thoughts?

[yesgrey]

Thoughts?
Yes, don't use rgb3dlut, you don't need it.:)

You should use rgb3dlut in one of two scenarios:
a)you want to perform some kind of video adaptation to your display.
b)you want to perform YV12->RGB32 with more accuracy.

Looking at the file you used to create the 3dlut file, you are not changing anything in your video to adapt it to your display. So you're not in a).

You're also not in b), because after using rgb3dlut you use ffdshow to convert back from RGB32->NV12 to use the ATI hardware deinterlacing.

So, keep it simple, do not use rgb3dlut and feed directly the deinterlacer with the dvd output, you will get the best quality.;)

[Neeto]

Ahhhh - forgot to say that I haven't yet got my caliabration for my new Samsung LED LCD - so can't put int he custom levels/colour space yet.
I'm doing all this testing in anticipation.

What about the idea of having rgb3dlut output YUY2?

[yesgrey]

What about the idea of having rgb3dlut output YUY2?
Have you heard about t3dlut?;)
Look a few posts back. tritical created t3dlut that supersedes rgb3dlut and can use all combinations output by cr3dlut, even YUY2->YUY2.

[Neeto]

Thanks for the pointer yesgrey3.

I've played around with tritical's t3dlut function.
I can get the RGB output to work, with the same image problems described in earlier post - no surprises there.

When I change the output color space on t3dlut to YUY2, the colors are all messed up.
I get a "Pink" FBI Warning on a "Green" background as per this image on the opening of an NTSC DVD.

http://thumbnails15.imagebam.com/4268/7ddf4242671846.gif (http://www.imagebam.com/image/7ddf4242671846)

It's like all the Blue in RGB went missing.

This is the avisynth script:
isYV12() ? yv12toyuy2(interlaced=true) : last
t3dlut("C:\Program Files\AviSynth 2.5\plugins\t3dlut_NTSC_Video.3dlut", destcs=0)

I've tried:
1. changing output color space in ffdshow Output tab to "RGB32", "NV12" & "YV12" - no difference
2. adding a yuy2toyv12 to the script & changing output color space in ffdshow Output tab to "RGB32", "NV12" & "YV12" - no difference

Do I need to generate the 3dlut file different?
I don't think so cause the cr3dlut documentation says we have all the conversions needed:

" In reality, our 3D LUT file contains three 3D LUTs, one 3D LUT for each component:
- R'G'B' -> R'G'B':
R'out = 3D LUT(R'in,G'in,B'in)
G'out = 3D LUT(R'in,G'in,B'in)
B'out = 3D LUT(R'in,G'in,B'in)
- R'G'B' -> Y'CbCr:
Y'out = 3D LUT(R'in,G'in,B'in)
Cbout = 3D LUT(R'in,G'in,B'in)
Crout = 3D LUT(R'in,G'in,B'in)
- Y'CbCr -> R'G'B':
R'out = 3D LUT(Y'in,Cbin,Crin)
G'out = 3D LUT(Y'in,Cbin,Crin)
B'out = 3D LUT(Y'in,Cbin,Crin)
- Y'CbCr -> Y'CbCr:
Y'out = 3D LUT(Y'in,Cbin,Crin)
Cbout = 3D LUT(Y'in,Cbin,Crin)
Crout = 3D LUT(Y'in,Cbin,Crin)"

Thanks, Neeto

[yesgrey]

Do I need to generate the 3dlut file different?
Yes.
I don't think so cause the cr3dlut documentation says we have all the conversions needed
No. The documentation shows all the possible conversions.
For YUY2->YUY2 you should use:
Output_Video_Format NTSC YCbCr

[Neeto]

:thanks::thanks::thanks:

Yep it's all working now.

I can confirm that by using the following script:

isYV12() ? yv12toyuy2(interlaced=true) : last
t3dlut("t3dlut_NTSC__YCbCr_Video.3dlut",destcs=0)

and having ONLY NV12 checked on the Output tab of ffdshow you can indeed get hardware acceleration for the deinterlacer AND color space adjustment/conversion.

Note: Don't be tempted to add a yuy2toyv12(interlaced=true) for some reason this reintroducates the "squares" - see previous posts.

Again thanks for all the help!

Now if I can just workout why EVR Custom Pres rendere will not obey aspect information - and why jitter has gone to hell in a handbasket - all would be good - SIGH.....

[yesgrey]

Great.:)

you can indeed get hardware acceleration for the deinterlacer AND color space adjustment/conversion.
Good finding, I will give it a try, maybe it's a good solution for the interlaced stuff...;)

[Neeto]

Things where going along swimmingly until decided to upgrade to latest ATI 9.8 drivers.
This is because 9.8 driver fixes the "double expansion" problem when using H/W acceleration.

Now having AviSyth in ffdshow with this script

isYV12() ? yv12toyuy2(interlaced=true) : last
t3dlut("t3dlut_PAL_DVD_YCbCr_Video.3dlut",destcs=0)

and having ONLY NV12 checked on the Output tab of ffdshow will "clip" i.e. convert anything < 16 to 16 & anything >235 to 235. All other levels appear to be fine.
If I disable AviSyth in ffdshow no more clipping - i.e. 0 to 254 displayed just fine.

It would seem the problem might be one of the following:

a. A bug in ffdshow?
I'm using ffdshow version 3014 & Avisynth version 2.5.8
Perhaps AviSynth doesn't talk to the NV12 interface correctly.
How might I check this?

b. A bug in t3dlut.dll - I hope not.

c. The way it's meant to be & I've mis-understood all this time.

d. Badly configured 3dlut input file. Unlikely as I've not changed it - but can someone check I'm not doing something silliy in the file I use to generate the 3dlut file.

e. Something else??? Ideas??

Thanks, Neeto

# Settings for creating a 3D LUT for watching the following Video formats:
# PAL DVD
# without any Display correction
# Excludes YCbCr->RGB conversion preserving Video Levels (Black: 16 and White: 235)

Chromatic_Adaptation 2

Out_Of_Gamut_Clipping 1

Input_Bit_Depth 8

Input_Video_Format PAL_DVD YCbCr

Input_YCbCr_Matrix 3

Input_YCbCr_Full_Range 0

Input_RGB_Black_White 16 235

Input_Primaries 2

Input_Gamma 1

Output_Bit_Depth 8

Output_Video_Format PAL_DVD YCbCr

Output_YCbCr_Matrix 3

Output_YCbCr_Full_Range 0

Output_RGB_Black_White 16 235

Output_Primaries 0

Output_Gamma 1

[yesgrey]

Try changing the following lines:
Input_YCbCr_Full_Range 1

Input_RGB_Black_White 0 255

Output_YCbCr_Full_Range 1

Output_RGB_Black_White 0 255


And be sure to use the latest version, v2.2.
Post your results.;)

[Neeto]

Yet again the experts are in victory! :)
All working, but now I'm curious why. Can you explain?
It makes me wonder what else I need to change in the config files for "non-hardware" accelerated HD/NTSC/PAL sources.

Thanks, Neeto

[yesgrey]

All working, but now I'm curious why. Can you explain?

I've made some changes in the latest versions of cr3dlut and now when Input_YCbCr_Full_Range is set to 0 it only converts the YCbCr values from 16-235, clipping the <16 and >235, as it should.
I still have to made some changes in cr3dlut about these conversions, because it's not yet exactly how I want, but I have to think in it more carefully...

[leeperry]

I still have to make some changes in cr3dlut about these conversions, because it's not yet exactly how I want, but I have to think about it more carefully...
yeah! I'm bored, everything works fine :scared:

I demand new features to beta-test http://img513.imageshack.us/img513/6779/icecream.gif

[yesgrey]

yeah! I'm bored, everything works fine :scared:
That's the bad thing of a perfect world...;)

Unfortunatelly, not everything is working fine.:(
Yesterday, I was trying to work a little in cr3dlut and I don't know what happenned, because now it's not compiling... The last time I opened the project it compiled fine, now, don't. I think I will have to uninstall and re-install VS2008 and ICL11.:mad:

[Neeto]

If you're messing with parameters then may I suggest keeping some consistency with those in ffdshow video RGB Conversion tab.
Up until the last set of changes, I thought they where consistent, but now I'm confused.

I started thinking how I would configure the 3dlut config file if I needed to display on a monitor i.e blacks really at 0 & whites really at 255 with HA acceleration.
I can't see a way at the moment.

Perhaps this is a missing parameter that indicates whether to clip or not clip below 16 and above 235 on the output, because sometimes you really do want black 16 & white at 235 and sometimes you want the black at 0 and white at 255 and when you have black at 16 and white at 235 sometimes you want <16 & > 235 info included and sometimes you don't.
In the 3dlut config parameters I can't see a way of indicating whether you want <16 & > 235 info to be included or not.

[yesgrey]

may I suggest keeping some consistency with those in ffdshow video RGB Conversion tab.
Don't worry. In case you don't know, I've participated actively in the discussion of the new ffdshow tab, and some of its ideas were suggested by me.;)

In the 3dlut config parameters I can't see a way of indicating whether you want <16 & > 235 info to be included or not.
I think cr3dlut has everything that you need.
In first place, you need to distinguish between input/output levels and YCbCr/RGB levels.
Generally, the input is YCbCr and the output is RGB, hence the possibility of confusion between the terms, but as you are using it, YCbCr can be both input and output.
The parameter that sets the YCbCr black and white levels is the XXX_YCbCr_Full_Range. If set to 1, it considers the video black at 0 and video white at 255; if set to 0, it considers the video black at 16 and video white at 235, hence clipping all <16 and >235 values. If you want to use YCbCr and all <16 and >235 values, you should set the parameter always to 1.

What I am missing, is the option to customize the levels for values different than the standard 0/16 and 235/255, like ffdshow does, because despite the XXX_RGB_Black_White parameter allowing that kind of customization, since the XXX_YCbCr_Full_Range performs clipping when set to 0, this is useless, because even if I set the RGB white to 245, I will not see the >235 values of the source, because they are clipped. My current idea is doing it similar to ffdshow, and substituting the XXX_YCbCr_Full_Range with a XXX_YCbCr_Black_White with the possibility of custom values, but I still need to think a little more about it.
I want the parameters setting to be clear, logic and simple, and that's not the case yet...

[Neeto]

That seems to make sense.
if I undestand correctly, what we need is Input AND Output config parameters for BOTH RGB and YCbCr with custom black/white levels for both RGB & YCbCr on both input and output.
If I understand correctly, we're missing the setting of custom levels on output of YCbCr.


The parameter that sets the YCbCr black and white levels is the XXX_YCbCr_Full_Range. If set to 1, it considers the video black at 0 and video white at 255; if set to 0, it considers the video black at 16 and video white at 235, hence clipping all <16 and >235 values. If you want to use YCbCr and all <16 and >235 values, you should set the parameter always to 1.


I'm not sure what's happening on the input YCbCr with regard to Input_RGB_Black_White parameters. Is it like the "Custom" in the "Input levels" on the ffdshow RGB conversion tab where you can choose the black level & white level? or is it simply ignored by cr3dlut when using Input_YCbCr_Full_Range set to 1?
I guess it's the use of "RGB" in the parameter name when we're dealing with YCbCr input info that is confusing me (or perhaps it is the screwy ATI drivers and how they deal with YCbCr input levels when taking NV12 and doing hardware acceleration - it seems to want YCbCr in Full Range Y:0-255 CbCr: 1-255 where as I would have expected it want it in Standard range Y:16-235, CbCr:16:240)

I've confirmed that by using setting:

Input_YCbCr_Full_Range 1
Input_RGB_Black_White 16 235
Output_YCbCr_Full_Range 1
Output_RGB_Black_White 16 235

On initial veiwing I get the same results when using HA as using:

Input_YCbCr_Full_Range 1
Input_RGB_Black_White 0 255
Output_YCbCr_Full_Range 1
Output_RGB_Black_White 0 255

i.e. no clipping. The first set of parameters seem more logical to me.

[yesgrey]

You are confusing the YCbCr and RGB Input/Output parameters, which shows that you have not read the readme carefully... but this confusion of yours is good, because it showed me one thing that is not very clear.
What defines the Input and output formats is the xxx_YCbCr_Matrix parameter. If set to 0 the format is RGB, if set to other values is YCbCr, and uses the specified matrix.
If you read the readme carefully, you will see that the xxx_RGB_Black_White parameter setting is ignored when the corresponding xxx_YCbCr_Matrix parameter is not set to 0, and that the xxx__YCbCr_Full_Range parameter setting is ignored when the corresponding xxx_YCbCr_Matrix parameter is set to 0. So, when you specify both parameters for Input and Output, only one is considered, depending of your YCbCr_Matrix setting.;)
Yes, I know, this is a bit confusing, but when this started the idea was only YCbCr->RGB, so it was natural to put the YCbCr and RGB words in the parameters names. With the current state of cr3dlut, it seems a better idea to substitute xxx_YCbCr_Full_Range and xxx_RGB_Black_White by a single parameter, like xxx_Black_White_Levels. The only problem I see is that this could be less intuitive for some people, because the YCbCr and RGB words are missing... but I feel that I am nearer to what I feel is right.:)

[Neeto]

Thanks for the clarification - as usual I was confused - situation normal ;-)

[canuckerfan]

any chance the yv12toyuy2() function can support matrix strings like converttoyuy2() does so that we can keep the full range of 0-255?

and in its current state does yv12toyuy2(), by default, clamp the range to 16-235?

[yesgrey]

any chance the yv12toyuy2() function can support matrix strings like converttoyuy2()
That does not make any sense.
converttoyuy2 needs the matrix strings because it can be used to perform the conversion from RGB -> YUY2. When it's used to perform the conversion from YV12->YUY2 the matrix string is not used.
yv12toyuy2 only performs the conversion from YV12->YUY2, hence no need for the matrix string.
Since there is no RGB->YUY2 conversions invloved, the levels are untouched. If it enters 16-235 it outputs 16-235, if it enters 0-255 it outputs 0-255.

[darkbasic]

Hi!
I have a wide gamut display (similar to AdobeRGB) and I'dd like to have color correct video playback.
Can someone explain me how to achieve it?
Thank you!

[leeperry]

do you have a colorimeter?

[darkbasic]

do you have a colorimeter?
Yes, I have a spectrophotometer (Colormunki Photo).

[leeperry]

ok, cool..but it's not supported by Color.HCFR, is it? but maybe you're using Calman or another similar app?

anyway, do you know the RGBW coordinates of your display gamut? you also need to decide whether you wanna use madVR or t3dlut/rgb3dlut in the Avisynth filter of ffdshow...you can also use MPC HC's PS script in MPC(only for VMR9/EVR)

[yesgrey]

Hi!
I have a wide gamut display (similar to AdobeRGB)
Yes, I have a spectrophotometer (Colormunki Photo).
The better option would be to measure your primaries coordinates and use them when creating a 3DLUT for your display. The other option, easier, would be select option 4 for the output primaries (set them as Adobe RGB) when creating your 3DLUT using cr3dlut.

[darkbasic]

ok, cool..but it's not supported by Color.HCFR, is it?
I don't know, but I have also an Eye-One Display 2. I already have a monitor profile (both matrix+shaper and LUT) and I know my RGB coordinates (I usually use Spectraview II for hardware calibration and argyll 1.1 beta for profiling). I use gamma 2.2 and D65 white point.

you also need to decide whether you wanna use madVR or t3dlut/rgb3dlut in the Avisynth filter of ffdshow...you can also use MPC HC's PS script in MPC(only for VMR9/EVR)
Well, what's the difference? Which one should I use? How to use them?
I don't know much about video encoding standards and video color management, I know only about photography...

Thank you,
Darkbasic

[leeperry]

yes, the i1d2 in Color.HCFR might be an easier solution to measure coordinates.

well, it depends on many things actually...all the video renderers have their advantages:
-madVR supports YV12 input, is very smooth and makes highly accurate color conversions, but it's still in beta stage, needs a beefy GPU, is not very stable yet and doesn't support subs or DXVA(you can use CoreAVC CUDA, though)
-EVR and VMR9 support exclusive mode(for smooth playback if your system is not optimized), and they also support DXVA...the gamut conversion PS script hardly needs any GPU power, and no CPU power at all. you can use PS scripts in MPC/MPC HC and KMPlayer
-if you want to use Haali's Renderer(which is quite buggy), you have to use t3dlut/rgb3dlut in RGB32..which is a CPU hog. I don't recommend it at this point, madVR is a better choice now.

I'd say if you watch your movies in 60Hz, EVR w/ the PS script in MPC or KMPlayer might be the easiest solution(a few clicks and you're good to go!): Color Correction with a HTPC - Simpler solution and now it really works! - AVS Forum (http://www.avsforum.com/avs-vb/showthread.php?t=912720)

if you watch movies with Reclock in a 24Hz multiple to get butter smooth playback, don't need DVD support and don't mind fiddling around, madVR would be better.

[pbmtp]

Hi yesgrey3,

I emailed you (@gmail) some HCFR files about differences between cr3dlut & t3dlut and pixel shader script, did you had any time to have a look at it ?

[yesgrey]

Hi yesgrey3,

I emailed you (@gmail) some HCFR files about differences between cr3dlut & t3dlut and pixel shader script, did you had any time to have a look at it ?

I will try to take a look at them as soon as possible. I have to instal hcfr to be able to see the files...;)

[pbmtp]

Thanks a lot

[darkbasic]

I already tried the Shader Editor solution, but I'm not fully satisfied with it. I want to achieve the best quality possible and if I can have a smooth playback I don't care about CPU/GPU usage (you were talking about Reclock in a 24Hz multiple, can you tell me more about it?). madVR seems to be very nice...
I have an AMD Athlon64 3800+X2@2700 MHz, 2GB ram (fsb synchronous) and an Ati Radeon 3870HD and I use Windows 7 x64 and Media Player Classic HC x64 (with K-Lite Codec Pack x64).
I think I can also use CHROMiX ColorThink Pro 3 or Argyll's iccdump utility and a chromatic adaptation calculator (bradford) to find the RGBW coordinates. Do you need them referred to D50 or D65?

[leeperry]

I already tried the Shader Editor solution, but I'm not fully satisfied with it. I want to achieve the best quality possible and if I can have a smooth playback I don't care about CPU/GPU usage (you were talking about Reclock in a 24Hz multiple, can you tell me more about it?). madVR seems to be very nice...
I have an AMD Athlon64 3800+X2@2700 MHz, 2GB ram (fsb synchronous) and an Ati Radeon 3870HD and I use Windows 7 x64 and Media Player Classic HC x64 (with K-Lite Codec Pack x64).
I think I can also use CHROMiX ColorThink Pro 3 or Argyll's iccdump utility and a chromatic adaptation calculator (bradford) to find the RGBW coordinates. Do you need them referred to D50 or D65?
the PS script is as accurate as can get...if that doesn't satisfy you, you either need to make the right conversion(SMPTE-C for US/ASIAN movies, EBU for european/russian) and/or make sure your RGBW coordinates are OK.

theoritically, madVR/cr3dlut in 16bit/ddcc would yield better results on extreme test patterns...but the PS script does the job, it's actually my personal reference as it looks so damn nice on my DLP pj and it's been double-proofed on several occasions by either ppl on AVS or myself on my own setup :cool:

you can extrapolate zillion values, but the source movies are only 16-235 SMPTE-C/EBU to begin with..when xvYCC and so will show up it'll be a different story, though.

look at this doctored JVC RS1 w/ the PS script : http://www.avsforum.com/avs-vb/showpost.php?p=11937064&postcount=90

and that's Kazuya's PLV-Z4 in SMPTE-C w/ cr3dlut in madVR: http://img301.imageshack.us/img301/1904/clip54.th.png (http://img301.imageshack.us/i/clip54.png/)

D65 is required AFAIK, and you'd be better off using ColorHCFR w/ your i1d2 to measure coordinates.

oh well, Reclock carries a pretty steep learning curve...you can start here: http://forum.slysoft.com/forumdisplay.php?f=85

[yesgrey]

the PS script is as accurate as can get...
No. Using madVR with a 3DLUT file is more accurate than the PS script.

oh well, Reclock carries a pretty steep learning curve...you can start here: http://forum.slysoft.com/forumdisplay.php?f=85
Also, you're using mpc-hc x64, and reclock only is available in 32bit version. You need to use mpc-hc 32bit.

[leeperry]

No. Using madVR with a 3DLUT file is more accurate than the PS script.
well, show me some real world movies where they make a clear difference ;)

Seb.26 told me that the HLSL PS scripts work in double-precision 64bit float, the source movies are still 16-235 SMPTE-C...you can't squeeze a lemon forever :o

when fiddling w/ cr3dlut, my goal has always been to reach the same colors as the PS script...which has been double-proofed on many occasions, and looks amazing anyway.

when we'll have xvYCC, then yes...cr3dlut+madVR will most likely kill the PS script.

anyway, the PS script w/ EVR is an easy way to get gamut conversion(you even get a nice menu for different conversion presets in MPC or KMP), if you wanna fiddle a lot more you can go cr3dlut+madVR..that's only what I was saying, no offense.

[yesgrey]

Thanks a lot
I think it's better if we discuss the problem here.
Start by telling me how have you measured the primaries in all three cases. Only the results is not enough.

well, show me some real world movies where they make a clear difference ;)
The PS use an aproximate transfer function to remove the gamma, while cr3dlut uses the exact transfer function inverse of each format, and it uses 64bitFP per component, while PS uses at max 32bitFP.
The fact that you cannot see the difference doesn't mean that it's as accurate as can get.;)

Seb.26 told me that the HLSL PS scripts work in double-precision 64bit float
No they don't. Pixel Shaders currently only support 32bit float per component (what madshi is using in madVR), and I don't know if mpc-hc PS is also working with that level of precision.

when fiddling w/ cr3dlut, my goal has always been to reach the same colors as the PS script...which has been double-proofed on many occasions, and looks amazing anyway.
That would make the colors more nice looking, which is not the same as more accurate.;)

anyway, the PS script w/ EVR is an easy way to get gamut conversion(you even get a nice menu for different conversion presets in MPC or KMP).
Yes, I agree, currently the PS solution is more user friendly. I believe madshi will take care of it.

that's only what I was saying, no offense.
I am not offended.:)

[leeperry]

The fact that you cannot see the difference doesn't mean that it's as accurate as can get.
[..]
That would make the colors more nice looking, which is not the same as more accurate.
as you know, I'm colorblind anyway :D

if the gamut conversion is spot-on in ColorHCFR and if my brain is stunned by the natural colors(as it's always been the case w/ the PS script), I'm cool!

getting spot-on SMPTE-C seems to increase the CR, maybe because colors are not mixed or oversaturated anymore...anyway it's like Reclock, try a butter smooth movie w/ perfect colorimetry and you'll never look back :devil:

professional grade equipment works w/ a 2.5 gamma(including the Samsung projectors from Joe Kane), but I think my HC3100 was built to work at 2.22..like every consumer display.

[pbmtp]

Hi yesgrey,

I used the following patterns in ts files which are adaptation of AVSHD for being use with a video player. Here is the link http://kvcd.net/downloads/MIRES_1080P_POUR_MPCHC_V3.rar The ones I used were 100 % Color (folder ColorHCFR Fields\100% Color).

For first measure, video were played using MPC-HC EVR configured in 0-255 without D3D, decoded with CoreAVC in YV12 16-235, conversion to RGB32 0-255 was done using ffdshow (High quality YV12 to RGB32 conversion), ColorHCFR probe was configured in 0-255 plugged on a second computer. This measurement gave me the exact same results as when using ColorHCFR pattern generator.

The next measure was done using the exact same configuration but with the pixel shader corresponding to the primaries of my Z3000 applied in MPC-HC, in that case CIE diagram was close to spot on with BT.709 reference.

Final measure was done using same config for CoreAVC, MPC-HC without any shader, and the provided avisynth script with t3dlut using the LUT generated by cr3dlut with the config file i gave you by email. ffdshow was configured in RGB32 0-255 output. I also tried with a more simple cr3dlut input file recommended in the manual for HD BT.709 with just the added output primaries but got the same results.

Also all 3 measures were done in a row without turning off the video projector.

Hope you have all you need, i can also mail you my ffdshow profile if needed.

Thanks

[yesgrey]

Hope you have all you need, i can also mail you my ffdshow profile if needed.
Yes, please send it.

[pbmtp]

done by email

[Kazuya]

and that's Kazuya's PLV-Z4 in SMPTE-C w/ cr3dlut in madVR: http://img301.imageshack.us/img301/1904/clip54.th.png (http://img301.imageshack.us/i/clip54.png/)

D65 is required AFAIK, and you'd be better off using ColorHCFR w/ your i1d2 to measure coordinates.

oh well, Reclock carries a pretty steep learning curve...you can start here: http://forum.slysoft.com/forumdisplay.php?f=85

Of course, it works perfect, and D65 is not required for that !
Color temperature only affects secondaries colors and their position on the gamut.

Thanks Yesgrey !!! :thanks:

[leeperry]

D65 is not required for that
movies are mastered in D65/2.2/SMPTE-C or EBU...but well, whatever floats your boat. D50 is yellowish and D75 blueish...it kinda defeats the whole point of gamut conversion, innit?

[yesgrey]

Thanks Yesgrey !!! :thanks:
You're welcome.:)
But you also should thank other people, like Charles Poynton, JohnAd, tritical and madshi (listed by chronological order), who helped me achieving the current state.
:thanks:

[yesgrey]

movies are mastered in D65/2.2/SMPTE-C or EBU...but well, whatever floats your boat. D50 is yellowish and D75 blueish...it kinda defeats the whole point of gamut conversion, innit?
It would be preferable D65, but the Chromatic_Adaptation is exactly for compensating that...

[pbmtp]

Hi yesgrey,

Did you find anything about my issue ? Is it something miss configured ?

[leeperry]

It would be preferable D65, but the Chromatic_Adaptation is exactly for compensating that...
ahhhhh your LUT can also compensate for D75>D65? cool stuff :devil:
miss configured
did she win any pageant :confused:

[yesgrey]

Did you find anything about my issue ?
Please send me by e-mail the .reg file. The copy paste thing is not working...

[yesgrey]

ahhhhh your LUT can also compensate for D75>D65? cool stuff :devil:
ddcc already did that. With cr3dlut I improved it slightly because not running in real time allowed me to use the full Bradford transform to perform the chromatic adaptation, instead of the linear version of it; but it would always be better to use a display calibrated for the same illuminant (white point) as the source.;)

[leeperry]

it would always be better to use a display calibrated for the same illuminant (white point) as the source.;)
less banding?

[yesgrey]

less banding?
More accurate colors.

[pbmtp]

yesgrey you've got mail :)

[leeperry]

More accurate colors.
compared to what? how to measure the colors "accuracy"? ColorHCFR will give the RGBW coordinates and saturations, that's it.

BTW, if you could somehow allow us to import the ColorHCFR saturations within cr3dlut, that'd be really awesome...that's the Achilles' heel of gamut conversion at this point IMHO.

http://img141.imageshack.us/img141/7501/satu.png

[darkbasic]

ColorHCFR will give the RGBW coordinates
Please explain me because I can't understand and so I don't know which coordinates I need. Let's make an example with AdobeRGB!

According to the documenation,the primaries are at:

Red x=0.6400, y=0.3300
Green x=0.2100, y=0.7100
Blue x=0.1500, y=0.0600
White x=0.3127, y=0.3290 (the reference white is D65)


If I dump the wtpt, rXYZ, gXYZ and bXYZ tags form the profile I obtain:

C:\Argyll_V1.0.4\bin>iccdump.exe -v3 -t wtpt -t rXYZ -t gXYZ -t bXYZ AdobeRGB1998.icc
XYZArray:
No. elements = 1
0: 0.950455, 1.000000, 1.089050 [Lab 100.000000, -2.387320, -19.404505]
XYZArray:
No. elements = 1
0: 0.609741, 0.311111, 0.019470 [Lab 62.601347, 90.371212, 78.149349]
XYZArray:
No. elements = 1
0: 0.205276, 0.625671, 0.060867 [Lab 83.214105, -129.089932, 87.172524]
XYZArray:
No. elements = 1
0: 0.149185, 0.063217, 0.744568 [Lab 30.210038, 69.243738, -113.612302]

The tristimulus values (XYZ) of the "white point" (0.950455, 1.000000, 1.089050) are in the "AdobeRGB white" source (so D65, If I had chosen my monitor profile instead of AdobeRGB would be ~ D65).

In fact if I convert it from XYZ to xyY I obtain 0,312701 0,329001 0,999999 which is the value from the documentation, of course.


Now let's see rXYZ, gXYZ, bXYZ.
They are the XYZ values of the R, G, B primaries and they form a 3x3 matrix which converts the normalized RBG values to XYZ.

If the white of the device is D50 they are exactly the colorimetric coordinates of the primaries. If not, the colorimetric coordinates had to be chromatic adaptated, in fact ICC specifications want relative colorimetry, so the matrix have to port the device white to the PCS white (D50).

For rXYZ we have 0.609741, 0.311111, 0.019470. If we do chromatic adaptation (from D50 to D65) and then we convert from XYZ to xyY we obtain 0,640014 0,330001 0,297363 which is the value from the documentation.

So, what are the RGBW coordinates I need? XYZ or xyY? Chromatic adaptated or not?
Can you make and example for AdobeRGB so I can use use the spectrophotometer (which is not comatible with ColorHCFR) instead of the Eye-One Display v2 (which is not so good with extreme gamuts)?

[yesgrey]

Please explain me because I can't understand and so I don't know which coordinates I need. Let's make an example with AdobeRGB!

According to the documenation,the primaries are at:

Red x=0.6400, y=0.3300
Green x=0.2100, y=0.7100
Blue x=0.1500, y=0.0600
White x=0.3127, y=0.3290 (the reference white is D65)

So, what are the RGBW coordinates I need? XYZ or xyY?
Use only xy from the xyY. The chromatic adaptation is not needed, you only need to put the coordinates of the white point, and cr3dlut will perform the chromatic adaptation.

The values you'll need are like the above values. If your display is AdobeRGB, you don't need to specify the RGBW colors, simply select option 4, cr3dlut already knows the standard colors. You only need to specify them when they are not accurate relative to any standard.

[yesgrey]

I used the following patterns in ts files which are adaptation of AVSHD for being use with a video player. Here is the link http://kvcd.net/downloads/MIRES_1080P_POUR_MPCHC_V3.rar The ones I used were 100 % Color (folder ColorHCFR Fields\100% Color).

That's the problem.
These files were not created with the BT.709 transfer function, but with an aproximation of it, hence the good results you get with the PS (which uses the same approximated formula) vs the 3DLUT file, which was created with the exact BT.709 transfer function.
Here is the file for cr3dlut that will give you the same results as the PS script:
# Settings for creating a 3D LUT for watching the following Video formats:
# Blu-ray, HD DVD, ATSC HD Broadcast, PAL HD Broadcast
# without any Display correction
# Includes YCbCr->RGB conversion using PC Levels (Black: 0 and White: 255)
# using advanced parameters instead of 'x_Video_Format'

# Do not mess up with these settings
Chromatic_Adaptation 3 # same as PS
Out_Of_Gamut_Clipping 0 # same as PS
Input_Bit_Depth 8
Input_YCbCr_Full_Range 0 # Y: 16-235; CbCr: 16-240
Input_Gamma 9 1.0 0.0 0.45 0.0 # same as PS
Output_Bit_Depth 8
Output_YCbCr_Matrix 0 # RGB
#Output_YCbCr_Full_Range 1 # Ignored
Output_RGB_Black_White 0 255 # RGB: 0-255

# You can change the following settings
Input_YCbCr_Matrix 1 # BT.709
Input_Primaries 0 # BT.709

Output_Primaries 9 0.675 0.324 0.289 0.711 0.142 0.052 0.313 0.332 # custom primaries for video proj
Output_Gamma 9 1.0 0.0 0.45 0.0 # same as PS

Remember that official video material should be mastered using the exact standard curves.

You can compare the results using a tool like colorpic, so you can avoid turning on your projector and all the calibration stuff.;)

[pbmtp]

Hi yesgrey,

thanks for taking time investigating the problem, i am glad everything is working as expected in cr3dlut, i will try your input file and compared it with pixel shader to validate when i have some time to launch "all the calibration stuff" :). And i will switch back to cr3dlut + t3dlut.

Do you know if mp4 version of AVSHD files (http://www.avsforum.com/avs-vb/showthread.php?t=948496 ) suffer from the same problem ? If so maybe you could report the issue to the AVS HD thread so that it can be improved.

[darkbasic]

According to the documenation,the primaries are at:

Red x=0.6400, y=0.3300
Green x=0.2100, y=0.7100
Blue x=0.1500, y=0.0600
White x=0.3127, y=0.3290 (the reference white is D65)

The values you'll need are like the above values.

This is a problem, because those values are referred to an observer adaptated to the device white, while the values stored in the ICC tags are referred to an observer adaptated to D50.

From the ICC tags I can see my monitor's Red is x=0,675968 y=0,313882 which is _ABOUT_ x=0,671004 y=0,312852 referred to an observer adaptated to the device white. I told _ABOUT_ because I found those values with this (http://www.brucelindbloom.com/index.html?ColorCalculator.html) calculator (click on "Calc" and then on "Chromatic Adaptation Calculator") which doesn't let me choose the temperature in Kelvin but only D50/65/... and my monitor temperature is not exactly D65, but is 6439,1K (as you can see from the white point coordinates x=0,313635 y=0,330961).

[yesgrey]

And i will switch back to cr3dlut + t3dlut.
Remember that some of the settings are only for you to be able to get the same results as the PS cript. After that, you should use the recomended settings.

If so maybe you could report the issue to the AVS HD thread so that it can be improved.
Yes, I will take a look into it. Apparently, in the issues list on the first post, they already noted that something is not correct...;)

[yesgrey]

which is _ABOUT_ x=0,671004 y=0,312852 referred to an observer adaptated to the device white.
... and my monitor temperature is not exactly D65, but is 6439,1K (as you can see from the white point coordinates x=0,313635 y=0,330961).

Maybe I can find some time to include a calculator into cr3dlut... Until then, try using those coordinates and D65 coordinates and see the results.

[darkbasic]

How can I check which is the color space of a movie (BT 709, EBU, SMPTE-C...)? Is there an utility to check it?

[leeperry]

oh that's the easy part :)

SD: BT.601
HD: BT.709

european/russian/brazilian movies = EBU
USA/ASIAN: SMPTE-C

it's all explained in the OP I think ;)

[yesgrey]

european/russian/brazilian movies = EBU
USA/ASIAN: SMPTE-C
That would be for SD. For HD it's supposed to be BT.709 primaries.

[leeperry]

That would be for SD. For HD it's supposed to be BT.709 primaries.
I couldn't more disagree...movies are not mastered w/ BT.709 primaries.

We've discussed it many times, it's been discussed many times on AVS too..if you check the OP, the french CEO of the ISF has a list on his website and no movie on bluray whatsoever is ever mastered w/ BT.709 primaries :o

the HDTV gamut is used for demos, but not for movies...well at least for mastering studios that still master their stuff on CRT(99% of them?)

the CARS hero will always remain dark orangey in the movie, way to go! :D

[yesgrey]

I couldn't less disagree...movies are not mastered w/ BT.709 primaries.
According to some people that work in the area they use their displays gamut corrected to BT.709. Nobody knows for sure, there is not a 100% agreement on that. I only stated what the standards say, but I accept perfectly that in some cases they do not gamut correct their displays.;)

[leeperry]

that's the aforementioned thread: Question : Only for those that are 6500K/D65/REC709 calibrated ... - AVS Forum (http://www.avsforum.com/avs-vb/showthread.php?t=1038602)

it matches exactly what the ISF and Joe Kane say, movies on bluray are mastered in SMPTE-C on CRT :/

they get their CRT recalibrated on a weekly basis, and it goes off as SMPTE-C encoded in BT.709..

[yesgrey]

These files were not created with the BT.709 transfer function, but with an aproximation of it, hence the good results you get with the PS (which uses the same approximated formula) vs the 3DLUT file, which was created with the exact BT.709 transfer function.
Do you know if mp4 version of AVSHD files (http://www.avsforum.com/avs-vb/showthread.php?t=948496 ) suffer from the same problem ? If so maybe you could report the issue to the AVS HD thread so that it can be improved.
pbmtp,
I have looked into the files and the way they were created and there is not anything wrong with them.
I've thought a little more about it, and I think that the problem is only at the output gamma setting.
The displays usually have a gamma function that mimics a pure power curve, and the BT.709 gamma curve is not exactly like that.
It seems to me that the bad results you are getting are due to using an output gamma curve that does not fit correctly your display's gamma curve.
So, I would suggest that you also run your tests with:
Input_Gamma 1 # BT.709 gamma encoding
Output_Gamma 9 1.0 0.0 0.45 0.0 # same as PS

Let me know if it worked...

[pbmtp]

Hi yesgrey,

I will do some test next week and tell you what happened.

[Kazuya]

movies are mastered in D65/2.2/SMPTE-C or EBU...but well, whatever floats your boat. D50 is yellowish and D75 blueish...it kinda defeats the whole point of gamut conversion, innit?

Sorry I was'nt notified !

I didn't say D65 is useless, I say it is not necessary to have a perfect gamut on RGB points.
D65 only puts YCM at the right place.

Pbmtp : in my opinion, you have a problem on your display.

[pbmtp]

kaz, what could it be as when using the pixel shader script the CIE of my Z3000 is very close to be spot on with BT.709 ?

anyway i will try what yesgrey suggested next week and see what happens.

[Kazuya]

I don't understand your question.

It would be preferable D65, but the Chromatic_Adaptation is exactly for compensating that...

Yes, I noticed it when I measured greyscale with KMP ! :D

Is there a way to avoid it if for some reason we don't want a temperature correction ?


Well, Yesgrey, I have an other question for your colorimetric skills ! :p

I made some color patterns at only 20 IRE.
And, I was wondering how should be the gamut with this colors.
I used a 20IRE white pattern too.

Should the gamut be exactly the same than the CIE ?
Or should it be equaly narrow ?
Or whatelse ?

This is the measure on my Z4 :

http://img694.imageshack.us/img694/4351/clip.png

But maybe the Eye one is not enough sensitive to measure correctly those dark patterns ?
(probably the better explanation for the yellow position)

[yesgrey]

Is there a way to avoid it if for some reason we don't want a temperature correction ?
If you want to avoid the chromatic adaptation, setting it to 0 will disable it.

I made some color patterns at only 20 IRE.
And, I was wondering how should be the gamut with this colors.
I used a 20IRE white pattern too.

Should the gamut be exactly the same than the CIE ?
Or should it be equaly narrow ?
Or whatelse ?
Yes, The gamut should be exactly the same.

But maybe the Eye one is not enough sensitive to measure correctly those dark patterns ?
That's a possibility, because the sensors are less accurate at the low IRE levels, but I already noticed that the gamut correction is not working good at very low levels. I'm currently working on it to solve the problem, but probably I will simple disable the gamut correction at low levels, because our sensitivity to colors is not very high at those levels, and the low bit depth of the source (8bit) is a problem when working at those levels...

[Kazuya]

Ok, thanks ! :)
I will do more tests at different levels.

[yesgrey]

Test also without any gamut correction, that way you can have an idea of the accuracy of the meter at lower levels...

[Kazuya]

Yes, I forgot to do it on my Z4 but I do it on my LCD display :

http://img204.imageshack.us/img204/3699/clip12.png

I think the gamut conversion is working properly, even at 20IRE.
I watched Knowing yesterday, it was awesome colorimetricaly speaking, and I watched tonight an episode of Gossip Girl (DVD) and it was totally incredible ! :o Even in dark scenes.

[yesgrey]

I think the gamut conversion is working properly, even at 20IRE.
The problem is not the gamut conversion not working properly at low levels, it's the handling of out of gamut values at low levels, that could cause severe banding. I notice it some times, at very low levels. I'm currently investigating it to find the cause and how to solve it.

I watched Knowing yesterday, it was awesome colorimetricaly speaking, and I watched tonight an episode of Gossip Girl (DVD) and it was totally incredible ! :o Even in dark scenes.
Yes, I know what you're talking. After we start watching at accurate colorimetry, it's very hard to go back again...;)

[Kazuya]

I didn't notice severe banding at all, maybe depends of display ?
I will take a better look.

Well, yesterday I make gamut correction on a VW60 Sony blackpearl, with 334hours lamps.
This is what I obtained with shaders on MPC-HC, all patterns in MPC, even greyscale :

http://img17.imageshack.us/img17/272/clip19.png

The most interesting part is the gamut with 20% colors : it is narrow, but it looks great ! :o
Can I conclude the eye one is not so bad in low level colors ?
And that there's big differences between displays, like I suspected ?

[yesgrey]

I didn't notice severe banding at all, maybe depends of display ? I will take a better look.
With the shaders method there is no problem, only with the 3DLUT method, and if you change the gamma.
A good place to notice the banding is the start of the movie "Batman: The Dark Knight". At the beginning, when the logo of DC Comics appears, it will show severe banding around it.

Can I conclude the eye one is not so bad in low level colors ?
And that there's big differences between displays, like I suspected ?
From the images you posted that seems to be the most likely...

[Kazuya]

The 3Dlut method but where ? In madVR ?

[yesgrey]

The 3Dlut method but where ? In madVR ?
In madVR or t3dlut, the results are the same, so it should be a problem in the 3DLUT. I'm working on it...

[leeperry]

SD: BT.601
HD: BT.709

european/russian/brazilian movies = EBU
USA/ASIAN: SMPTE-C
apparently you can add Australia to the EBU list: http://www.indietalk.com/showthread.php?t=4240

and China too I think?

[pbmtp]

Hi yesgrey,

I finally had some time to make the measurement you asked me, all configuration file and HCFR chc can be found at the following location:
http://pbmtp.free.fr/3dlut-z3000/
On my Z3000, the clear winner is HD_PC_Z3000_PS.txt (same configuration for cr3dlut/rgb3dlut as the pixel shader). The corresponding HCFR chc file has all the possible measurement possible done, you can see that CIE is perfect and saturation are really as good as they can be.

Damien

[yesgrey]

Hi,
Great!
Thanks for posting the results. It's good to see that our work give such good results.:)

I suggest you to use Chromatic_Adaptation 2 with the PS, it should give you more accurate colors. The difference you are getting between HD_PC_Z3000_PS and HD_PC_Z3000_2, should be due to the different input gamma curves.

[Kazuya]

What is the difference between chromatic adaptation 2 and 3 ?

[yesgrey]

What is the difference between chromatic adaptation 2 and 3 ?

chromatic adaptation 2 is the preferred because it causes less errors. I only added 3 to be able to get with cr3dlut the same correction like the pixel shader method. We should always use 2. 1 and 3 are only for testing purpose.

If you want a more detailed description, read this (http://infoscience.epfl.ch/record/34077/files/FinlaysonS00.pdf). It explains the three methods implemented in cr3dlut, and also describes a fourth that, maybe one day, I will also add...

[Kazuya]

Ah ok, I was in 3 for chromatic adaptation, and didn't see anything wrong.
But I will put it at 2 and will see !
Thanks.

[leeperry]

chromatic adaptation 2 is the preferred because it causes less errors. I only added 3 to be able to get with cr3dlut the same correction like the pixel shader method. We should always use 2. 1 and 3 are only for testing purpose.
but only "3" works if the destination gamut is smaller, otherwise the green gamma is burned to death..and most DLP's have a lack of green in the first place, so you get a very greenish picture basically.

I know you're working on it as there seems to be a glitch in some part of the code, but "3" is mandatory on my DLP...please don't drop it in the next version :)

[yesgrey]

Ah ok, I was in 3 for chromatic adaptation, and didn't see anything wrong.
When the difference between the source and the output white points coordinates is small you should not notice any difference, but since it will take exactly the same time when watching your movies, I suggest setting it to 2.
Remember that this is not a 100% sure thing, because if you have read the article I linked above, you could see that this is just the results of several tests, so it's possible that some people prefer other settings.

but only "3" works if the destination gamut is smaller, otherwise the green gamma is burned to death..and most DLP's have a lack of green in the first place, so you get a very greenish picture basically.
From what I have tested previously, I thought that your problem was more due to the out of gamut clipping, than to the chromatic adaptation... Also, since you're color blind, I don't think that you would fit in the profile of the people used for when the chromatic adaptation algorythms were tested... but it's good to have you onboard, so we can make this work to all people.:)
Also, don't worry, usually I don't drop any features.;)

[leeperry]

ahhh don't give me the colorblind stuff again :D

I'm sure you can easily see that we got a problem w/ the green gamma in the sixth screenshot, and some of that code is in cr3dlut: http://forum.doom9.org/showpost.php?p=1292804&postcount=444

but indeed, it's got to do w/ the OOG...but the only way we could get the exact same colors as the PS script was through "Chromatic_Adaptation=3".

anyway, we already spoke about all this...the average guy will recalibrate his display every few weeks, and we don't use highly accurate spectrophotometers...and projectors MH lamps have fluctuating R/G/B levels, so all that stuff is slighty overkill IMVHO.

when you recalibrate on a weekly basis w/ a Minolta CS200 sensor(that's factory recalibrated every few months)...then yes, we can talk about this level of accuracy...but quite frankly ddcc and the PS script were going different ways(ddcc being more accurate), but even on test patterns screenshots you'd have a hard time identifying them in a DBT and tell whichever one is better(especially w/ a 16-235 SMPTE-C source).

[FoLLgoTT]

@yesgrey3
The primaries are defined by three coordinates in color space (usually xyY). Sometimes it is not sufficient to correct only x and y. It would be very cool if cr3dlut would support the luminance Y in the future.

[yesgrey]

The primaries are defined by three coordinates in color space (usually xyY).
Yes, I've already read that in several other places. I am considering looking into that so I could try to add it in a future version...
Thanks for the tip anyway.:)

[FoLLgoTT]

Yes, I've already read that in several other places. I am considering looking into that so I could try to add it in a future version...
Thanks for the tip anyway.:)

Great news. :)

Btw. a complete CMS with independent controls for all primaries and secondaries in RGB color space would be a killer application for the HTPC. Only the $5000 Lumagen Radiance (http://www.lumagen.com/docs/Tip0002_GamutCalibration.pdf) has implemented such a CMS. But now I'm only dreaming a bit. It is great what you have achieved with cr3dlut! :)

[Kamus]

I've got good results, but should they be better?
I'm using MadVR+cr3dlut, I'm wondering if I've done something wrong here:

http://i139.photobucket.com/albums/q296/Saintkamus14/gamut.jpg

Pic on the left is from the native gamut of my JVC RS1x projector using an eye one 2 LT, the one on the right is the corrected gamut, and while it's better i still get a delta E that's quite a bit off.
Green has a delta E of 11, 6.7 on red and 17 on blue with the 3dlut file enabled.
While these aren't horrible results, i'm obsessed. (and supposedly delta E needs to be about < 3 to be undetectable by people right?)

My HD_PC file looks like this:

# Example input file for cr3dlut v2.1 and up
#
# Settings for creating a 3D LUT for watching the following Video formats:
# Blu-ray, HD DVD, ATSC HD Broadcast, PAL HD Broadcast
# without any Display correction
# Includes YCbCr->RGB conversion using PC Levels (Black: 0 and White: 255)

# Set input bitdepth
Input_Bit_Depth 8

# Set source video format
Input_Video_Format HD YCbCr

# Set output bitdepth
Output_Bit_Depth 16

# Set display video format
Output_Video_Format HD RGB_PC

# set Output Primaries
Output_Primaries 9 0.651 0.348 0.294 0.697 0.140 0.043 0.312 0.330

BTW, i'm calibrated to D65 if it means anything, and gamma is at 2.2 (i simply used the eye one match 3 software to set the grayscale & gamma with very good results, or so says HCFR. Maybe the fact that i'm not using any gamma settings on the file has something to do with it? I figured that it was pointless since the projector is already at 2.2 gamma and i figured it shouldn't be related in the first place)

Also, I'm using "PC levels" since my ATi videocard is set to RGB limited and that takes care of the video levels. The color patterns i used are from the AVS HD disc. (http://www.avsforum.com/avs-vb/showthread.php?t=948496) by extracting the files and playing them trough MPCHC. (also used color facts test patterns to see if there was any significant difference in the native gamut, compared to using the files, and there wasn't)

Anyway, any tips on how i could make this more accurate? I'm overall pleased with the results, but if i can improve them even more, then all the better.

Thanks in advance for anyone that takes the time to reply, and Kudos to everyone involved in making HTPC a very viable and cheap solution.

[yesgrey]

I've got good results, but should they be better?
Try adding this line at the end of your cr3dlut setting file:
Output_Gamma 9 1.0 0.0 0.454 0.0
If it's not good enough or if you notice any banding at low levels, try adding this too:
Input_Gamma 9 1.0 0.0 0.454 0.0

[Kamus]

Thanks, that seems to have done the trick for blue and green almost to perfection (had to use both input and output, just the output messed with the levels it seems) red is somewhat off, but it's due to the fact that my bulb doesn't seem to be able to do a full red anyway.

[yesgrey]

Great.:)

[janos666]

I calibrated my PC monitor with this target values: WP - exact D65 ; Tonal response curve - x^2.22 ; White luminance - 140 cd/m^2 (Balck luminance - Min. Neutral)
My hardwares and softwares: Q6600@3600Mhz (FPS@450) ; Radeon HD 5850 1Gb with Cat 9.12hotfix ; Windows 7 x64
I installed the latest AviSynth 2.6 alpha and the latest MPC-HC (stable x86), and I have a recent SVN build of FFDShow (x86). I used the latest cr3dlut and t3dlut from this thread.
This is all the 3 lines I have in the AviSynt box (I move the # for SD movies...):
yv12toyuy2(itype=2, threads=4)
t3dlut("C:\Program Files (x86)\AviSynth 2.5\hd - pc.3dlut", threads=4, destcs=2)
#t3dlut("C:\Program Files (x86)\AviSynth 2.5\sd - pc.3dlut", threads=4, destcs=2)
And I created my HD - SD.t3dlut file with this settings:
# Set source video format
Input_Video_Format HD YCbCr
Input_Bit_Depth 8

# Set display video format
Output_Video_Format HD RGB_PC
Output_Bit_Depth 8

# Gamut and gamma correction - measured native coordinates
Output_Primaries 9 0.656725 0.329082 0.228974 0.690610 0.140753 0.086776 0.313918 0.328022

# Gamut and Gamma Correction settings
Chromatic_Adaptation 2
Out_Of_Gamut_Clipping 1
Output_Gamma 1

And I have some questions:
Is it normal that I had to reduce the Output_Bit_Depth to 8 bit? I had constant 75% CPU utilization and only 18-20 FPS under Blu-Ray playback with 16-bit (720p was ok). But I do not have a lot of free CPU time with 8-bit anyway. 1080p playback is smooth until I do not try to apply another filters like 720p->1080p resize with FFDShow (there is a reason why would I do that but that is offtopic...).

When I set the Output_Gamma to 9 1.0 0.0 0.45 0.0 then the image is too bright. The result looks exactly the same when I miss the Output Levels (16-235 signal for a 0-255 PC monitor). I think that the output level parameter is ignored when this parameter is used in a later line.
I do not know if the HDTV standard is x^2.22 or something else (like sRGB has an own tonal response curve -> I can calibrate my monitor with an sRGB curve but there is no Rec 709 preset...). Should I worry about it?

I try to use this file with madVR but it would not load my 3dlut files. The malfunction is another question but:
- madVR is internally limited to YV12 input - (I tested it with unchecked AviSynt and FFDShow output settings)
- I can use this renderer with the AviSynt script and I can see corrected colors.
- This would imply that t3dlut is sending a YV12 output for madRV. But this could not be right with destcs=2 settings (and RGB_PC generation settings). It should send an RGB32 output and nothing else. And madVR should accept only YV12. (I did not set up any other conversion and I unchecked every box in FFDShow's output tab, expect YV12.) Where is the trick? :confused:

Anyway, this is a very nice stuff. I tried the old PS script but it caused some wired violet instead of blue. (I have a wide gamut display with over-saturated red and neon-green, but it misses some of the Rec709 blue area. :( )

[yesgrey]

And I created my HD - SD.t3dlut file with this settings:
# Set source video format
Input_Video_Format HD YCbCr

You should change the line above for SD sources.
Use NTSC instead of HD for ntsc dvds.
Use PAL_DVD instead of HD for PAL dvds.
(Read the ReadMe for further details)

This is all the 3 lines I have in the AviSynt box (I move the # for SD movies...):
yv12toyuy2(itype=2, threads=4)
t3dlut("C:\Program Files (x86)\AviSynth 2.5\hd - pc.3dlut", threads=4, destcs=2)
#t3dlut("C:\Program Files (x86)\AviSynth 2.5\sd - pc.3dlut", threads=4, destcs=2)
Is it normal that I had to reduce the Output_Bit_Depth to 8 bit? I had constant 75% CPU utilization and only 18-20 FPS under Blu-Ray playback with 16-bit (720p was ok). But I do not have a lot of free CPU time with 8-bit anyway. 1080p playback is smooth until I do not try to apply another filters like 720p->1080p resize with FFDShow (there is a reason why would I do that but that is offtopic...).

When you use 16 bit output t3dlut dithers the output to 8 bit, hence the higher cpu load. If you have low cpu power available you have to stick with 8 bit output.
You can also try using ConvertYV12ToYUY2() instead of yv12toyuy2, because the later is slower.
Another option would be to use rgb3dlut, which is also faster than t3dlut, but only supports 8 bit output.

When I set the Output_Gamma to 9 1.0 0.0 0.45 0.0 then the image is too bright.
When you change the gamma curve, you have to correct the brightness level of your display, that's why it looks brighter. You have to decrease the display's brightness until you get the correct black level.

I do not know if the HDTV standard is x^2.22
HDTV is BT.709.

I try to use this file with madVR but it would not load my 3dlut files.
madVR only works with 16 bit output 3DLUT files.


- madVR is internally limited to YV12 input - (I tested it with unchecked AviSynt and FFDShow output settings)
- I can use this renderer with the AviSynt script and I can see corrected colors.
- This would imply that t3dlut is sending a YV12 output for madRV. But this could not be right with destcs=2 settings (and RGB_PC generation settings). It should send an RGB32 output and nothing else. And madVR should accept only YV12.

Correct

(I did not set up any other conversion and I unchecked every box in FFDShow's output tab, expect YV12.) Where is the trick?
That's the trick.;)
When you only check YV12 at ffdshow's output, ffdshow converts the RGB32 output from t3dlut to YV12, and feed it to madVR.

Summing up, you should try disabling the avisynth script, and simply create a 16 bit output 3DLUT file to use with madVR. This should give you better performance, and better image quality.

[janos666]

Thanks for the detailed reply but I have to honor it with more questions ans corrections for misunderstandings. :)

You should change the line above for SD sources.
(Read the ReadMe for further details)

I already read that. And I am sorry, it was my mistake. I want to write: "HD - PC.3dlut" and not "HD - SD.t3dlut". You can see that I talked about 1080p and 720p movies later. But I made both SD and HD files with PC levels.

If you have low cpu power available you have to stick with 8 bit output.

I can live with that. (I am not sure about this dithering anyway. It usually means "noise" for me, not quality, and I am irritable about noise. But movies does not contain figures with color gradients. It is not so easy to imagine it with random colors on a random picture. I did not compare the results visually, yet.)

My concrete question supposed to be: Is it normal if my Q6600 (which runs at 3600Mhz with FSB450) can not handle a Blu-Ray movie with 16-bit 3dlut? I did not see another posts about this kind of performance problems in this topic and I think it is not me who has the slowest CPU here. So, I assumed it could be some kind of compatibility problem which causes high CPU load on my system. (Like I should use a specific AviSynth/FFDShow/MCP-HC build and not the last one, ect.) There is no any other active background task.

When you change the gamma curve, you have to correct the brightness level of your display, that's why it looks brighter.

Oh. This is not a good news because I would not do that. And it does not sounds like a correct solution anyway!
I calibrated my display to 140cd/m^2 because this was the default value in the calibration software. I would keep this white luminance value because it feels good for everyday usage (document reading/editing, web browsing, ect.)
Is there an exact value in the ITU Rec BT.709 standard? I could not see that. It should be a varying number which considered by the user (this is a function of ambient light conditions, personal feeling, ect.)

I can not go below 120 cd/m^2 anyway, it won't help now. The white luminance value with factory default OSD settings was ~280 cd/m^2, nearly two times more than it is after the calibration. And black was black (and not gray) in the movies with 0-255 output level settings in FFDShow. (Of course without t3dlut. I started to deal with this after I bought my calibrator...)

I think that brightness has nothing to do with this.
The image looks exactly when I miss the PC-TV levels. (Like the program ignores the PC level settings when it uses custom gamma settings.)
Or do you mention about some software brightness control in the player? That would not be a nice thing anyway.

HDTV is BT.709
Yes. And I would ask question about that.
Does this standard have specific tonal response curve (like the sRGB standard)? Isn't it match with some x^y (like gamma 2.22), or sRGB curve? Is there any EyeOne compatible software which can calibrate to this target?

madVR only works with 16 bit output 3DLUT files.
I thought about that. But I tried to use madvr first. When it refused to load the 3dlut files I came here to learn about them. I found the t3dlut plugin and it was later when I figured it out that I have to reduce the bit depth for smooth Blu-Ray playback...
It is not well documented how to use 3dluts with madvr. I could see 3 kind of template file labeling: "HD_PC" ; "HD - PC" ; "template HD-PC". I do not know what would be the correct file name for 3dlut files and where should I place them. (It was tricky to figure out that here is no real madvr installer, it is only a shortcut to register the filter, no matter where it is located momentarily, blahhh -> I know the install directory now, I only mentioned that...)

ffdshow converts the RGB32 output from t3dlut to YV12

I thought about that but I could not imagine that it could be true. Softwares can be smart only when they confuse the user with that. :p
Here is another CPU eater...
Is it a lossless conversion or does it have deficit? (Not if I would like to keep with this.)

*** May be it will solve some of my other problems if I eliminate this conversion. (Like the brightness/gamma/TV-PC level shit...) :)

create a 16 bit output 3DLUT file to use with madVR
This would be my goal. But I have another problems with madvr. (But I can solve them when I will have more free CPU power, so...)


UPDATE: I corrected the FFDShow output settings to RGB32 only. I have less CPU load with 8-bit 3dluts, I will try to generate 16-bit one again.
But black areas are still "washed out" with Output_Gamma 9 1.0 0.0 0.45 0.0. (The monitor is still calibrated to x^2.22, 140cd/m^2)
And the first (full black) frame of a movie tell the truth: It looks exactly the same when I miss PC-TV levels. I can see a gray rectangular between two black rectangulars. (16:9 source on 16:10 display...)
UPDATE2: And no. I can not play 1080p Blu-Ray movies with 16-bit 3dluts.

UPDATE3: I could solve my problems with madrv. It works well now. (I had to disable something in FFDShow...)
MadVR gives me better result with 9 1.0 0.0 0.45 0.0 and PC output settings than t3dlut. But this is not perfect yet. (At least, it won't miss the output levels, so black is nearly black and not gray now.)
There is some strange (and intensive) noise on dark areas with custom gamma settings. (I could see something like this when I watched old SD rips with very low bitrates.) It is not correct. I can not see this noise without custom gamma settings, and black is perfectly black when I clear this line with output_gamma.

Should I try to recalibrate my display with L* or sRGB tonal response curve and use one of these presets in cr3dlut for output_gamma?

[yesgrey]

My concrete question supposed to be: Is it normal if my Q6600 (which runs at 3600Mhz with FSB450) can not handle a Blu-Ray movie with 16-bit 3dlut?
No. You have to be doing something wrong...

I calibrated my display to 140cd/m^2 because this was the default value in the calibration software. I would keep this white luminance value because it feels good for everyday usage (document reading/editing, web browsing, ect.)
When you change gamma, you change the black level. Unless you recalibrate your brightness level you will get bad black levels. Though, considering this is only for watching movies and for your day-to-day usage you will use a different gamma, it's better keeping the gamma untouched.;)

UPDATE3: I could solve my problems with madrv. It works well now. (I had to disable something in FFDShow...)
MadVR gives me better result with 9 1.0 0.0 0.45 0.0 and PC output settings than t3dlut. But this is not perfect yet. (At least, it won't miss the output levels, so black is nearly black and not gray now.)
There is some strange (and intensive) noise on dark areas with custom gamma settings. (I could see something like this when I watched old SD rips with very low bitrates.) It is not correct. I can not see this noise without custom gamma settings, and black is perfectly black when I clear this line with output_gamma.

Should I try to recalibrate my display with L* or sRGB tonal response curve and use one of these presets in cr3dlut for output_gamma?
No. That's a problem that I know for a while that I'm trying to find a solution, but I think it would not be possible.
The problem is not from using custom settings, the problem is when someone use an output gamma curve with a low level segment different from the input gamma curve. For now, just use as output gamma the same as input gamma or, if you want it, try this:
9 4.5 0.099 0.45 0.018
For different gamma values, just change the 0.45 value for any value between [0.4, 0.5]

Note 1: performance wise, you should disable the avisynth filter and resizing in ffdshow, and do it all with madVR.

Note 2: Due to your updates, I've not answered some of the other questions.

[janos666]

I calibrated my display with sRGB tonal response curve and I used this input file for "HD - PC.3dlut":

# Source video format
Input_Video_Format HD YCbCr
Input_Bit_Depth 8

# Display video format
Output_Video_Format HD RGB_PC
Output_Bit_Depth 16

# Gamut and Gamma Conversion - settings
Chromatic_Adaptation 2
Out_Of_Gamut_Clipping 0
Output_Gamma 0
Input_YCbCr_Full_Range 0
Output_RGB_Black_White 0 255

# Gamut conversion - measured native coordinates
Output_Primaries 9 0.658 0.328 0.230 0.691 0.141 0.087 0.313 0.328

It should be perfect but it is not. But I am close to see the source of the problem. I think that cr3dlut handles this PC-TV range thing incorrectly when it also does gamma correction.
I watched Pandorum (Blu-Ray disk) some days ago with t3dlut and VRM9 (8-bit RGB table without gamma correction) and I used it to test my settings with madVR today, because this movie has a lot of dark scenes.
I can see very odd things. Sometimes I can see a very dark scene where black is true black but every black area changes to noisy gray as soon as a little light source shows up on the screen. It can render true black but any little light will push it to gray. It is a noisy gray because the little differences (between dark and dark areas) are magnified. There is a very big jump between black and the darkest gray.
I guess YCrCb=RBG looks like this: 0=0 and 1=4, 2=8, 3=16, or 0=0, 1=16, 2=17, and so on. (And dithering may help to make it more noisy.)
I tried to change it but Out_Of_Gamut_Clipping has nothing to do with this problem. The Reducing of RGB White to 235 helped a bit, but the result was far from good. The Increasing of RGB Black wont help, and negative values won't make sense here...
So I removed this gamma setting for now. Dark scenes are more natural without it.
This gamut conversion helped a lot, a more accurate gamma won't make big difference (the Rec709 tonal response curve is not far from sRGB one). But I have to live with this knowledge that it is not perfect yet. :rolleyes:

SUPPLEMENT: Sorry, I already wrote this post when I noticed your answer.
No. That's a problem that I know for a while that I'm trying to find a solution, but I think it would not be possible.
Did You try to change the order between TV->PC and source_gamma>output_gamma conversion precess? May be gamma correction should be done first.

[yesgrey]

I think that cr3dlut handles this PC-TV range thing incorrectly when it also does gamma correction.

Did You tried to change the order between TV->PC and source_gamma>output_gamma conversion precess? May be gamma correction should be done first.
Thanks for trying to help, but that's not the problem.
Currently I think the problem is related to the bit depth.
At lower levels, the 8 bit bit depth is too low for the small changes.
If we don't change the gamma curve for the low level part, there is no problem, because all the fine gradation levels are kept, but when we change the low level part of the gamma function, there is a problem, because some of the low level detail is completely lost, because some values will be coded as the same output level instead of different ones, hence the blocky images in the dark areas.
I still don't know if this can be solved with the new clipping method I am working with, but I'm afraid that it's not.
So, for now, we should always use the same gamma function for input and output. If you want, you could try changing the gamma value, as I showed you in my previous post, but not the gamma function.

[janos666]

Currently I think the problem is related to the bit depth.
At lower levels, the 8 bit bit depth is too low for the small changes.


It is not hard to imagine as soon as we are in the same interval. But I thought that the extra 16 steps (between PC and TV levels) can span this limitation. So the steps between 16 and K (during this non linear gamma function is effective) can be stretched over this 0-K interval (which is approximately 1,5-2 times wider than the original interval. - It is my guess only, I did not calculated it...)
But ok, I stop giving advices because I can not reflect my thoughts in this language and I do not have experience with this kind of transformations...

[yesgrey]

But I thought that the extra 16 steps (between PC and TV levels) can span this limitation.
No, because I think the problem only happens when you are outputting PC levels. The bit depth seems to be too low to allow the conversion... but it also could be any problem with the clipping, so without further testing is not easy to know.

But ok, I stop giving advices because I can not reflect my thoughts in this language[/COLOR]
Ideas are always welcome, help me think of what might be the reason. It's also hard for me to reflect my thoughts in this language, so just try, like I do.;)

[janos666]

No, because I think the problem only happens when you are outputting PC levels. The bit depth seems to be too low to allow the conversion...

It sounds like a proof that problem is elsewhere. I also noticed that there are no noisy gray blocks with TV output levels. Of course, it is always too bright on my PC monitor and it never hits real black but the transition between gray levels is smooth. How can it loose so much detail when it is extended to a wider interval?
May be RGB values shouldn't be extended but simply reduced by 16 in this tricky 0-0.081 interval and the remained K;235 interval should be extended to fill the (K-16);255 interval. (In this case, gamma correction should done first in the original range.)


But I am not sure if this conversion is needed or not. I found this info when I searched on the web for HDTV transfer functions:

My understanding is that Rec709 is a camera transfer function, not a display transfer function. So while the gamma curves detailed above do define the input reference of a Rec709 image, those images are then presumed to be displayed at gamma 2.2

In my understanding this document from ICC confirms this statement: http://www.color.org/sRGB.xalter The most interesting paragraph is: sRGB and ITU-R BT.709 Compatibility
It supposes to clarify the confusions but this document isn't consistent anyway. :rolleyes:
Sometimes it says:
we can solve for the ideal target monitor gamma of 2.2
It is not always clear that it speaks about the whole sRGB standard or the sRGB color-space only.
sRGB color space provides a monitor definition that can be used independently from the ITU-R BT.709 standard while maintain compatibility
But sRGB and Rec 709 have the same primaries so their compatibility shouldn't be further clarified.
And gamma 2.2, again:
In summary, there has been some concern with the choice of a 2.2 CRT gamma with a 1.0 LUT gamma as opposed to a 1.571 (2.2/1.4) or a 1.294 (2.2/1.7) display gamma. We feel that there are many reasons to support a 2.2 CRT, including;
- PC's with 256+ colors
- HDTV
- [....many more...]

The biggest conflict in this document is that the same image cannot be perfect with both sRGB and Gamma 2.2 because they are close but different tonal response curves. And it doesn't mention any color management (with gamma correction...) for HD video playback on PC. This is your "innovation".

On the other hand there are another conflicts with calibration softwares:
HCFR knows the Rec 709 (and some other) standard color spaces but it uses a reference gamma (like x^2.2) and not complex tonal response curves.
I found some information about an X-Rite software which accepts user defined gamma functions with mathematical formulas, and someone made an XML with Rec709 formulas.
Cheaper and bundle calibration softwares do not care about the sRGB curve. So a lot of people calibrate their displays with x^2.2 and working happily (Yes, working, not just watching movies. There is very few people like us who buy a colorimeter to enjoy movies...)


In my experience Blu-Ray movies looked better with gamma 2.22 than they looks now with sRGB calibration. (I am speaking about monitor calibration here...)
I can see more details in dark areas now but the whole image was sharper and more coherent for me with x^2.22 tonal response. (It applies to movies and PDF documents as well.)

[IanB]

The classic cheat for this is like sRGB does it.

Have a small linear region at the black end then change to the gamma curve. i.e.Y=[16..235] => y=[0..219] being the zero based luma values

(y<10) ? (y*2) : ((y/219)**(1/2.2)-(10/219)**(1/2.2))/(1-(10/219)**(1/2.2))*((219-10*2)/219)+10*2

2.2 being the Gamma value
10 being the size of the linear region
2 being the linear region coefficient

Example lookup values :-
In Gam Lin/Gam
0 0 0
1 19 2
2 26 4
3 31 6
4 36 8
5 39 10
6 43 12
7 46 14
8 49 16
9 51 18
10 54 21
11 56 24
12 58 27
13 61 29
14 63 32
15 65 34
16 67 36
17 69 39
18 70 41
19 72 43
20 74 45
...
58 120 100
59 121 101
60 122 102
61 122 103
62 123 104
63 124 105
...
77 136 120
78 137 121
79 138 122
80 139 123
81 139 124
...
214 217 216
215 217 217
216 218 217
217 218 218
218 219 218
219 219 219

[htpc66]

I notice that the free program MonInfo from Entech provides chromaticity co-ordinates for xyz and white points.

Wonder if someone please guide on whether those co-ordinates could be used as input primaries to generate .3dlut files? If not, what would the relevance of those chromaticity co-ordinates given by MonInfo as far as color correction is concerned?

[yesgrey]

Wonder if someone please guide on whether those co-ordinates could be used as input primaries to generate .3dlut files?
I don't know if the chromaticity coordinates are accurate, but for using them is self-explanatory. The names are the same. Rx in moninfo is rx in cr3dlut, etc., so it's just copying the values to the cr3dlut settings file.

[janos666]

I decided to recalibrate my monitor. This time I wasn't lazy and checked back the results. I used the latest MPC-HC, FFDShow and MadVR builds without any post-process filters to measure the AVCDH video test palette with Color HCFR 2.1.
Here is the animated gif with 3 CIE diagrams: monitor's native, Out_Of_Gamut_Clipping 1 and 0: click (http://img12.tar.hu/janos666/img/76100716.gif)
It was worth to check this because I thought that the default clipping 1 setting is better. And I thought that it is a little more accurate. So, I can appreciate my friend's results with his U2410 in sRGB mode. (That display uses the results of the one time factory measures to do gamut conversion and that is more accurate than this software with fresh measures.).

[yesgrey]

So, I can appreciate my friend's results with his U2410 in sRGB mode. (That display uses the results of the one time factory measures to do gamut conversion and that is more accurate than this software with fresh measures.).
That's one of the reasons I've started the new project: this wasn't good enough, it has some limitations, and I want something a lot better.;)

[tritical]

I updated the 'rgb3dlut' function of ddcc to support 3DLUT format LUT files. Even though t3dlut has all of the functionality of rgb3dlut, it doesn't have assembly versions of some of the code paths making rgb3dlut preferable in some cases. I also updated ddcc to output 3DLUT format files, and fixed a small bug in t3dlut for yuy2 input with cplaceU=0/itypeU=1. All are available at the usual place.

I'm considering writing a standalone program that would create a full 256*256*256 entry LUT for use with rgb3dlut/t3dlut/madvr from a partial subset of mappings.

[leeperry]

I'm considering writing a standalone program that would create a full 256*256*256 entry LUT for use with rgb3dlut/t3dlut/madvr from a partial subset of mappings.
A simple GUI where you could input/load/save RGBW xy coordinates and choose the output gamut for direct use within mVR would be very nice indeed :cool:

[yesgrey]

We have started a project on sourceforge to host everything related to the 3DLUT (http://thr3dlut.sourceforge.net/) file format.

When madshi find the time he will start a new thread for discussing the compression of the 3DLUT files.

@tritical
If you also want to be part of the 3DLUT project admins let me know.

[yesgrey]

I've updated 3DLUT (http://thr3dlut.sourceforge.net/)'s sourceforge project by uploading the files to the "Download 3DLUT files" area.

Sorry for any trouble, but I'm new to this and previously only have put the files in the bazaar repository.

[yesgrey]

Updated 3DLUT (http://thr3dlut.sourceforge.net/)'s project: created license file.

[Yellow_]

Would ddcc & 3DLUT improve colourspace conversion from YV12 to RGB for DSLR video, h264 over AVS's usual ConvertTORGB()? Reading through the ReadMe for ddcc it appears there are more options for control? The Rec709 v EBU looks like something that may come into play or is DSLR h264 almost certainly Rec709?

[IanB]

AVS's usual ConvertTORGB() is a linear algebraic conversion. It offers 2 sets of colour coefficients Rec.601 and Rec.709 with either PC, [0..255], or TV, [16..235], levels scaling.

Ddcc and 3DLut offer arbitrary lookup table translation from each YUV value to a RGB value. A LUT can be generated to apply any linear or non-linear function you desire. The standard LUT generator programs offer a useful selection of standard conversions profiles. You could modify the source code to generate any conceivable LUT you desire. I seem to remember some discussion about csv files and Excel as a means to generate arbitrary LUT's.

[Yellow_]

I think ddcc also offers Bicubic conversion and others, as well a linear. I believe 2.6 (development) will have Bicubic as default?

Do you think there is any benefit in that respect, in pursuing different conversions in ddcc, just for the YV12 to RGB?

Thank you for the explaination, I'll read more on 3D LUTS.

[yesgrey]

Would ddcc & 3DLUT improve colourspace conversion from YV12 to RGB for DSLR video, h264 over AVS's usual ConvertTORGB()?
If you use 16 bit 3DLUTs with t3dlut() then the quality should be better, because t3dlut dithers them to 8 bit, and I think the YV12->RGB conversion on ConvertToRGB() is performed using only 8 bit (please correct me if I'm wrong). This would avoid any banding that might be created during the conversion.

[yesgrey]

Do you think there is any benefit in that respect, in pursuing different conversions in ddcc, just for the YV12 to RGB?
For that you should take a look into t3dlut, and should use it in combination with yv12toyuy2. The chroma upsampling benefits from using the bicubic, we've tested it. However, it's just on images with very saturated reds in very dark backgrounds, so it might not be worthy the extra processing time...

[Yellow_]

yesgrey, thanks for the reply, I will look into t3dlut further, it sounds encouraging.

Is there a point in the process where 16bit can be exported out rather than dither to 8bit and do that later after grading?

With regard to saturated reds, I was looking at an example of that last night in a shot of mine from a 550D, http://blendervse.wordpress.com/2010/09/03/video-import-update/, difference between Fast_Bilinear & bicubic conversion, red / magenta on black gives a rather rough edge, where as the white on black was much smoother.

I understand that the Canon DSLR's are a bit red and can be calibrated by the user adjusting manual white balance and picture style.

http://www.hurlbutvisuals.com/blog/2010/03/30/color-correction-put-your-best-foot-forward/

[yesgrey]

yesgrey, thanks for the reply, I will look into t3dlut further, it sounds encouraging.
Be aware that t3dlut is slower than rgb3dlut because is a more generic function, but if you want more quality, and the time is not a big issue, use it instead.

Is there a point in the process where 16bit can be exported out rather than dither to 8bit and do that later after grading?
No, because we are limited by Avisynth 8 bit processing. If/When Avisynth support 16 bit processing then it might be possible.

As it is now I only see two solutions for your problem:
(1) Try to do the grading in YV12 and then convert to RGB
or
(2) Try to create a 3DLUT that not only performs the YCbCr->RGB conversion but also performs the grading.

The (1) I don't know if are there any tools available for it, but I might be able to help you with (2) if you could specify a set of changes to each of RGB channels that applies to the entire video...

[Yellow_]

yesgrey, thanks for your time. I think the best to achieve presently for me is the assumed improved YV12 to RGB conversion, I'll try it out and see.

I don't think (1) & (2) are very useable for me but thanks for suggestions.

Wilbert has a plugin called Immaav which uses ImageMagick to read but also write out 8 & 16bit to image files.

http://forum.doom9.org/showthread.php?t=135928

There's hope maybe some way of writing 16bit out via a Q16 build of IM? Although he thinks it's maybe not possible again due to AVS 8bit processing.

Anyway, very interesting processing and just improving YV12 to RGB over normal ConvertToRGB is a definite plus. :-)

[Yellow_]

I've started to give this ago, but I'm unsure as to chroma placement, left or centre? Source is progressive mpeg2 HDV and h264 in the conversion from YV12 to YUY2.

When I read through the t3dlut manual I can't find under mandatory settings a full range YCbCr 0 - 255 only 16 - 235. I'm aware most stuff is authored in that range, like BluRay, DVD etc, however I'm hoping to try the process on HD & DV vid cam source. I deliberately expose right up to hard clipping point using the waveform and histogram to gauge expsore, to get the most into the 8bit.

I'm almost certain both the camera's I use capture full range, depending on conditions. Certainly using AVS ConvertToRGB(PC.709...) gives me usable data in the headroom, not so much down below. I assume my camera sources are YCbCr, I've always used YV12 in AVS before. :-)

[poisondeathray]

I'm not sure if this is correct, but when I fidded with it , I generated a 3DLUT file from yCMS first. It's in that 3DLUT file that you can specify whatever range, and the characteristics etc...

http://forum.doom9.org/showthread.php?t=154719

Then I used t3dlut's YV12toYUY2(), and then t2dlut() with the 3DLUT file to do the RGB conversion

A while back , maybe 20 pages or so tritcal posted some comparison screenshots between the standard ConvertToRGB() vs. the other sampling methods. There was also discussion of whether left, or center is preferred.

Perhaps I wasn't doing it correctly , but my limited testing - assuming I'm doing it correctly - (along with tritical's comments) have shown it not that beneficial over the standard avisynth method ... but of course if you come to some new understanding or findings please share :)

[Yellow_]

Yeah, that's where I'm at really. I've got the AVS script sorted for the conversion except to confirm chroma placement.

Used YV12ToYUY2 from t3dlut and dropped out with AVS's ConvertToRGB and saw no difference tested against AVS's ConvertToYUY2, but hoping the whole chain via the LUT may yield better results, is that how you tested it?

The whole process I'd imagine has got to be useful coming back from RGB too after grading to delivery codec.

I've used a HD-PC template for the 3DLUT file but can't find a mandatory full range HD or SD option, so not got very far. :-(

Don't really want to get into the 'only use these settings if you know what you're doing' section of the LUT generation process.

A lot of reading to do. :-)

[yesgrey]

I've used a HD-PC template for the 3DLUT file but can't find a mandatory full range HD or SD option, so not got very far.
All HD and SD (Blu-ray, HD DVD, DVD and broadcasts) are not full range, hence the absence of such a setting from the mandatory commands. In the rare cases where it is needed, you need to use an advanced command.

Don't really want to get into the 'only use these settings if you know what you're doing' section of the LUT generation process.
You can always ask for help to achieve what you need... ;)

In your case, just add this line at the bottom of your file:
Input_Range 0 255
If you also want to guaranty full-range on the output side (the mandatory only allows the selection for RGB. YCbCr is always standard) add:
Output_Range 0 255

[Yellow_]

yesgrey,

re Input_Range 0 255 & Output_Range 0 255, that's nice and simple. :-)

I did read through the custom settings but saw mention that mandatory would overide incomplete custom choices so sort of backed off using them for now until I can understand them more, if necessary.

My other query re post #582 was that the sources are h264 and mpeg2 hdv progressive, so not sure whether it should be left or centre chroma placement.

Looking forward to trying this all out later and thank you for your time.

[yesgrey]

I did read through the custom settings but saw mention that mandatory would overide incomplete custom choices
No, you misunderstood it. What it says is that if you only use some of the advanced commands you still need to include the mandatory command in the file. As long as you put any custom commands below the mandatory commands, they will override their part of the settings.
The mandatory commands are some kind of internal scripts with default values for all the commands available.;)

My other query re post #582 was that the sources are h264 and mpeg2 hdv progressive, so not sure whether it should be left or centre chroma placement.
For H.264 is left, but for mpeg2 I don't remember. Read on avisynth's doc, there is a description of the different positions and when they apply. The problem is that some cameras use a different kind of chroma placement, so try to see which applies to your case.

Read this (http://avisynth.org/mediawiki/Sampling), it's a good place to start...

[Yellow_]

Ok, I have success and I think improvement based on one test. :-) So far.

Using t3dlut I managed to raise the number of unique colours by 23680, above the AVS ConvertToYUY2 -> ConvertToRGB. :-)

That was with bicubic coefs and set at 0.0 & 0.0

Using the defaults 0.0 & 0.5 it raised the unique colours by 25946.

Is it a fair assumption that the more unique colours generated the more resistant to grading and banding?

I've added the output frames on the blog if interested.

http://blendervse.wordpress.com/2010/09/10/ycbcr-to-rgb-by-3d-lut-via-avisynth/

Based on the full 1440x1080 .png frames, although they are just an example, is there anything else that could be done to improve general image quality of mpeg2 / h264, subjective I know, for example dealing with the blockiness, as seen in the sky, adding discreet noise, remove grain / add grain etc.

[poisondeathray]

How valid is the gimp colorcube analysis? And what does it really indicate ?

Is having more unique colors necessarily better ? e.g. if , because of the changed interploation the pixels are shifted slightly, or maybe use left instead of center, that may lead to more "unique colors", but perhaps less accurate


Is it a fair assumption that the more unique colours generated the more resistant to grading and banding?


You should test it .

For example, in other programs, you can do some levels or curves manipulations and watch the histogram along with the graded picture. It will band up more (stair steps in the histogram) with 8-bit footage compared to 8-bit footage interpolated correctly to 10-bit footage . I'm sure you can try the same thing in blender . Some programs (e.g. nuke) can take 8-bit images and work in 32-bit float then dither down to 8-bit when exporting (or you can export 32bit or 16bit images as well) , I'm not that familiar with blender or how it works internally

I doubt it will make that much of a difference (at least in terms of banding) by using using different YV12=> 8bit RGB methods . I suspect there will be zero visible difference by using avisynth converttorgb() vs. t3dlut in terms of banding because you are limited by 8-bit RGB (2^8 =256 "shades" for each channel component)

[Yellow_]

Yes, I ask the same question, but for the time being I use it as a 'measurement', it's just a statistic and my feeling is that 'proof' is in the learning process which comes with manipulating the image source and seeing what it will and won't stand. :-) It does come across as if I'm trying to proof by colour count I know. :-)

[poisondeathray]

I couldn't find any more information on it (colorcube analysis), but I did start playing with it because of another thread you mentioned it in

Just "eyeballing" the png images at various zooms , it's not a big difference (slight shift) , but not even as big as the red on black letter images earlier in the thread

In terms of grading, and actually using the images in blender - I'm still interested in seeing what comes out of your tests , please keep sharing your findings :)

[Yellow_]

How valid is the gimp colorcube analysis? And what does it really indicate ?

Is having more unique colors necessarily better ? e.g. if , because of the changed interploation the pixels are shifted slightly, or maybe use left instead of center, that may lead to more "unique colors", but perhaps less accurate


Cross posts. :-)


You should test it .

For example, in other programs, you can do some levels or curves manipulations and watch the histogram along with the graded picture. It will band up more (stair steps in the histogram) with 8-bit footage compared to 8-bit footage interpolated correctly to 10-bit footage . I'm sure you can try the same thing in blender . Some programs (e.g. nuke) can take 8-bit images and work in 32-bit float then dither down to 8-bit when exporting (or you can export 32bit or 16bit images as well) , I'm not that familiar with blender or how it works internally

Blender works 32bit float internally and has basic colour management that is just sRGB and Linear at the moment, but next iteration will probably be 3D LUT based rather than hardcoded as it is now, also possible to export DPX (gamma encoded and log) and OpenEXR (multilayer , linear too). All image imports are converted to linear colour space and all compositing / lighting / rendering calculations are done in linear colour space. OpenCL is beginning to appear for the compositing pipeline as a GSOC project, so 3 way colour correct, curves etc hopefully soon to be GPU accelerated.

I doubt it will make that much of a difference (at least in terms of banding) by using using different YV12=> 8bit RGB methods . I suspect there will be zero visible difference by using avisynth converttorgb() vs. t3dlut in terms of banding because you are limited by 8-bit RGB (2^8 =256 "shades" for each channel component)

Yes, more tests needed and thanks for the encouragement. :-)

[Dogway]

There's something strange on some png's I exported from Final Cut, they look clearer on avisynth and the default Windows Viewer, than some other applications like Photoshop, or Nuke, being the latter the correct look.
I dont understand why this happens, its 1080pixels and I have a calibrated monitor with its profile.

1Mb Sendspace rar:
(I pack the image so no color conversion is made upon browser.)
http://www.sendspace.com/file/lwh4rl

Actually it looks just if I disabled proof->monitorRGB on photoshop thus ignoring my monitor profile.

[Dogway]

Someone please?

[cretindesalpes]

Your PNG has a gamma correction profile attached, which interpretation may vary, depending on the displaying application. Use TweakPNG (http://entropymine.com/jason/tweakpng/) to remove it manually. FFmpeg will also work fine for batch conversion.

[Dogway]

Didnt notice the answer, yes I found out there was a gamma issue, and by some heavy thinking decided to tweak gamma by 1.25 to accomplish the desired 2.2, (1.8*1.25=2.2). That makes my image clearer, and then it shows a more natural look with more tonal range and hue shifts. But as you suggested I just used TweakPNG and corrected the internal 0.555 gamma to 0.454 and what I get is the darker version. So now I dont know where my logic goes.

Or maybe the internal gamma correction of 0.555 is correct and the software showing the clearer version are doing the proper thing because they are profile aware(?), in which case I dont need to do anything?

[Yellow_]

@Dogway, what are you trying to achieve? or are you sorted now?

[Dogway]

Kind of. Im color grading that footage, just want to be sure Im working in the original/tonal range of the source.

What I have are a png sequence and a prores encoded video footage.

-The pngs at first glance are darker, only avisynth and the default windows image viewer read the built in 0.555 gamma correction thus clearing up the image.
-The video footage is already clear, I think because no yuv->rgb conversion was done.

So Im assuming the clearer version is the correct, and will work better with the yuv source because it has a wider tonal range to begin with. I already converted the video source to png (yuv->rgb) through avisynth and grading in Nuke with that. Just correct me if Im wrong.

[Yellow_]

Ok, I don't quite get it? Why gamma correction from FCP to png's?

Are you trying to linearize your png's, with a reverse gamma correction?

It's straightforward to export png's from ProRes with AVISynth using the latest QTSource plugin.

Your frame image looks like it's had the levels messed with, the histogram looks rough. It doesn't look like it's been cleanly converted from the original.

[Dogway]

Yes, that's what I did, exported from Prores to png in avisynth.
You're true, good idea the levels thing, yes, it looks tweaked. What further confirms the clearer version is the correct. Don't ask me why gamma correction from FCP to png, I don't use that program, so I just wanted to hint what the embedded 0.555 gamma thing on the png meant and choosing the right one to grade on Nuke.

[Yellow_]

@Dogway, I understand now. :-)

yesgrey, if you're still about, is it possible to use yCMS to create a LUT that does this, the way Cineform handles levels, quoted by Cineform:

"Downstream tools generally don't handle full range YUV well, so we take the 0-255 input, bump that to 10-bit 0-1023, then range correct to 64-940, compressing the results. So we made a standard range YUV without clipping, and without loss of codewords as we bumped to 10-bit first. This greatly simplifies YUV playback or HDSDI/HDMI devices."

I'm aware yCMS can do the work in 16bit, then dither down to 8bit and input and output levels can be specified. So is it possible to do the above at 16bit?

Also I find out that Canon DSLR video is BT601 not 709, so should I use YCbCr or HD or PAL or specify 601 for primaries and colour coefficients when going to RGB?

Cheers

[yesgrey]

"Downstream tools generally don't handle full range YUV well, so we take the 0-255 input, bump that to 10-bit 0-1023, then range correct to 64-940, compressing the results. So we made a standard range YUV without clipping, and without loss of codewords as we bumped to 10-bit first. This greatly simplifies YUV playback or HDSDI/HDMI devices."
yCMS already does better than that. It takes the 0-255 input, converts that to 64 bit fp 0.0-1.0, then range correct according to specified output bit depth, which could be 8bit or 16bit. If you use 16bit it would be better than the above.

I'm aware yCMS can do the work in 16bit, then dither down to 8bit and input and output levels can be specified. So is it possible to do the above at 16bit?
yCMS never uses dithering, because it only creates the 3DLUT file. Dithering should be used only when reducing the bit depth for the processed streams/images, not for the 3DLUT. What you're referring is the t3dlut working mode, and that I'm afraid could not be changed, because Avisynth is limited to 8 bit processing. So, it's not a yCMS limitation, but Avisynth's. However, by dithering from 16 bit to 8 bit the compression would not be a problem, only a slightly higher noise level on the images,

Also I find out that Canon DSLR video is BT601 not 709, so should I use YCbCr or HD or PAL or specify 601 for primaries and colour coefficients when going to RGB?
You should select PAL, if you're from PAL land, or NTSC, if you're from NTSC land.

[Yellow_]

yesgrey, thanks for the clarifications, my bad with regard to 8bit 16bit in yCMS, not dither but range correct, dithering in t3dlut.

Thanks for the tool, it's excellent.

[yesgrey]

not dither but range correct
Yes, you can do range correct in yCMS. I decided to use only values between 0-255 to simplify the interface, because it's easier to think that we will convert 0-255 -> 16-235. However, the accurate range values are calculated internally. So, if you select 16-235 as output, but you're using a 16 bit bit-depth, the real output range values would be 4096-60160.

[Yellow_]

Just to clarify.

For doing 601 full range PAL to 8bit RGB:

Input_Format PAL YCbCr 8
Output_Format PAL RGB_PC 16

Gives me a LUT suitable for taking 0 - 255 and 'squashing' into 16 - 235 out

Substituting 'PAL' for 'HD' would give me a LUT suitable for the same but with for example HDV video camera sources?

Adding the line 'Output_Range 0 255' to either would give me full range output in either PAL or HD?

Is it as simple as that, no other parameters?

[yesgrey]

For doing 601 full range PAL to 8bit RGB:

Input_Format PAL YCbCr 8
Output_Format PAL RGB_PC 16

Gives me a LUT suitable for taking 0 - 255 and 'squashing' into 16 - 235 out
No.

When using the mandatory commands (Input/Output_Format) YCbCr is always considered to be within the standard range:
Y: 16-235
CbCr: 16-240

For full range you should add the line:
Input_Range 0 255

And for 'squashing' to 16-235 you should add this other line:
Output_Range 16 235

Substituting 'PAL' for 'HD' would give me a LUT suitable for the same but with for example HDV video camera sources?
Only if the 3DLUT's are meant to 'squash' the sources and keep the video format. If you will use them for watching the files then you should set as output format your display's video standard, and not the same as the source.

[Yellow_]

Thanks again yesgrey.

Considering that the Canon DSLR is BT601 but HD. Then really I should use the transfer function to 709 as well when going to an intermediate file for editing and reencoding?

Just thinking about final playback, aiming for widest compatibility with playback devices, should I really be encoding HD size material with 709, I assume that's what players will expect, or is 601/709 flagged in the encoding added by the encoder, say with x264.

I've read players are renowned for doing there own thing, but as best practice, 709?

[yesgrey]

Considering that the Canon DSLR is BT601 but HD. Then really I should use the transfer function to 709 as well when going to an intermediate file for editing and reencoding?
What do you mean with "Canon DSLR is BT601 but HD". Are you referring to the primaries, the transfer function, or the encoding matrix?
The transfer function is not a problem, because it's the same in both BT.601 and BT.709. The differences between them are the encoding matrices and the primaries.

Just thinking about final playback, aiming for widest compatibility with playback devices, should I really be encoding HD size material with 709, I assume that's what players will expect
Yes. According to what I've read everywhere, the players and TV sets use BT.709 when they detect a HD signal. It might happen that some equipments don't, but that's the exception (manufacturer mistake) and not the rule.

[Yellow_]

What do you mean with "Canon DSLR is BT601 but HD". Are you referring to the primaries, the transfer function, or the encoding matrix?
The transfer function is not a problem, because it's the same in both BT.601 and BT.709. The differences between them are the encoding matrices and the primaries.

Whether it's colour primaries or matrix, I'm unsure, all i know is that 601 is said to be "flagged in the bitstream"? FFMPEG for example reads Canon 7D and 550D DSLR video as 601 where as HDV video from my other vidcam is read as 709.

[yesgrey]

That's strange... I've googled a bit about those two models and I saw no reference whether it's PAL or NTSC, so it would be very strange if the cameras use BT.709 primaries and BT.601 encoding matrix... Did you see any reference to PAL or NTSC in the user manuals?

[poisondeathray]

Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M


This is the mediainfo metadata for the 7D, it's the same for 5D MK2

[yesgrey]

Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M
For characterizing this source in yCMS it should be used:

Input_Format HD YCbCr 8
Input_Matrix_Coefficients 3

if you prefer a simpler form, or

Input_Primaries 0
Input_Transfer_Function 1
Input_Matrix_Coefficients 3
Input_Bit_Depth 8
Input_Range 16 235

if you prefer the advanced/detailed form.

[Yellow_]

Thanks guys. I've now installed Mediainfo, a fine tool. Canon 550D/T2i is same as 7D & 5D mkII. All sources are PAL.

Prefer the advanced / detailed form for yCMS. :-)

So, for example, using the detailed form above gives me suitably 'decoded', colourmetrics 'neutral' source which I can then use to go to full range RGB using RGB_PC 16 and / or Output_Range 0 255 and then from that RGB intermediate to whatever, 601, 709 just using matrix co efs without the need for transfer function?

Looking at some HD encoded files done previously from edits and also source mpeg2 from a HDV cam, mediainfo just lists colormetry as 4:2:0, assume then these are not flagged sufficiently, so does that become more of a lottery at playback, as to what colourmetrics the player uses, perhaps based on frame size/pixel count where colourmetrics are not flagged?

Thanks for your patience with this, it must quite frustrating. :-)

[yesgrey]

Thanks guys. I've now installed Mediainfo, a fine tool. Canon 550D/T2i is same as 7D & 5D mkII. All sources are PAL.
If it's the same as above it can't be PAL, it has different primaries.

Prefer the advanced / detailed form for yCMS.
Me too, but I'm suspect.;)

So, for example, using the detailed form above gives me suitably 'decoded', colourmetrics 'neutral' source which I can then use to go to full range RGB using RGB_PC 16 and / or Output_Range 0 255 and then from that RGB intermediate to whatever, 601, 709 just using matrix co efs without the need for transfer function?
RGB_PC assumes always full range (0 255), so you won't need to use the Output_Range command. Of course you could also use the advanced/detailed form for the output, you just need to know what you want to get.

The RGB intermediate only makes sense if you plan to process the image in RGB, but then, if you want to output as YCbCr, you would need to use another 3DLUT for performing the RGB->YCbCr conversion. Is that your intention?

Looking at some HD encoded files done previously from edits and also source mpeg2 from a HDV cam, mediainfo just lists colormetry as 4:2:0, assume then these are not flagged sufficiently, so does that become more of a lottery at playback, as to what colourmetrics the player uses, perhaps based on frame size/pixel count where colourmetrics are not flagged?
If you know which camera was used for the capture you could try to know, otherwise it's safer to stick to the standards. Another option would be using the standard and the other and decide which look best.

[Yellow_]

If it's the same as above it can't be PAL, it has different primaries.

I'll check again. The 550D/T2i does both PAL & NTSC, but I use it mainly on PAL, just NTSC to get the 60fps for slow mo. (720P)

The RGB intermediate only makes sense if you plan to process the image in RGB,

Unfortuneatley the NLE / Compositor / 3D app I use stuffs up video sources with regard to levels and colourmetrics, (FFMPEG based) as it works solely in RGB. So I prefer to do a controlled conversion with AVISynth, then finally encode back to video for final delivery. Also want to maintain full range levels in editing / grading then squash 16 - 235 for final delivery.

but then, if you want to output as YCbCr, you would need to use another 3DLUT for performing the RGB->YCbCr conversion. Is that your intention?

Yes, I've used ConvertToYUY2 or YV12. However since finding yCMS + t3dlut I've created some 'Output' LUTS via yCMS instead which include 16 - 235 levels. I assume encoders don't provide method to set levels, generally just pass what they get through?

If you know which camera was used for the capture you could try to know, otherwise it's safer to stick to the standards. Another option would be using the standard and the other and decide which look best.

The 4:2:0 files are from a Canon HV30 HDV cam and the other 4:2:0 files are generally XVid and h264 final encodes that don't flag colourmetrics I guess.

Thanks again.

[Yellow_]

Check mediainfo and get this for 550D videos:

Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M

The camera is set to it's PAL mode as opposed to NTSC. Whether that's strictly to spec I don't know.

[yesgrey]

The BT.709 primaries are not much different from the PAL primaries, only the green changes, and it's a very small change, so the effect would not be significant. What's more important is using the right matrix, and that it does. However, due to the slight difference of the green primary it would not follow exactly the standards.
Just FYI the PAL's green coordinates are: 0.2900, 0.6000, while BT.709's are 0.3000, 0.6000 .

Just for curiosity, what do you get when the camera is set to NTSC mode?

[Yellow_]

What's more important is using the right matrix, and that it does.

You mentioned in an earlier post that it would be strange for the camera to use 709 primaries and 601 matrix. But above that it uses the correct matrix, ie: 601?

Just for curiosity, what do you get when the camera is set to NTSC mode?

Mediainfo gives exact same for NTSC mode from camera.

Assume th mix of primaries and matrix, increases chances of apps misjudging how to handle it? For example FFMPEG suggests 601?

I see variations in different apps with regard to brightness (assume levels stretching there) and colour hue mainly a difference in red. One app will show something of reddish colour more orange, generally when using 709 and the same source with a definite pink hue instead of orange, generally when using 601.

There's much 'folk law' about Canon being on the 'red' side, wonder how much that is to do with camera settings (I use Neutral profile and knocked magenta down a bit in custom white balance) or assuming incorrect colour primaries 601 instead of 709, would that make any real difference I wonder?

Lightworks NLE goes Free as Open Source on the 29th and offers handling of DSLR sources + GPU effects and playback/editing without transcoding to DNxHD or similar first. Time will tell how successful that is. Which may or may not result in a second workflow, other than the RGB one, staying YCbCr. I think there are still benefits to using your LUT + td3lut for precision and eeking most out. Many advocate Cineform products for similar reasons, high precision, upsampling and more lightweight codec.

Considering whether it's beneficial to decode the Canon sources as we've discussed 709 primaries & 601 matrix and put them into 709 and use a lossless codec like UT / lossless h264 / Matrox I Frame ? Sort of a 'poor mans' Neoscene. :-) Or maybe better?

[yesgrey]

You mentioned in an earlier post that it would be strange for the camera to use 709 primaries and 601 matrix. But above that it uses the correct matrix, ie: 601?
Yes, but I did not remember that BT.709 and PAL primaries are almost the same. Yes, it uses the correct matrix.

Mediainfo gives exact same for NTSC mode from camera.
With NTSC it would not be that good... Look at all the primaries (R;G;B):
BT.709: 0.6400, 0.3300; 0.3000, 0.6000; 0.1500, 0.0600
PAL: 0.6400, 0.3300; 0.2900, 0.6000; 0.1500, 0.0600
NTSC: 0.6300, 0.3400; 0.3100, 0.5950; 0.1550, 0.0700

There's much 'folk law' about Canon being on the 'red' side... or assuming incorrect colour primaries 601 instead of 709, would that make any real difference I wonder?
It might be from the primaries... If you compare the Red coordinates from above you could see that the NTSC Red is more saturated than the BT.709 one. If you capture using BT.709 and then process assuming NTSC the end result might be just that... To be sure you would need to process the same file considering both options and compare.

Assume th mix of primaries and matrix, increases chances of apps misjudging how to handle it? For example FFMPEG suggests 601?
Ideally you should use all correct parameters, but considering not all applications allow the selection of the primaries at least choose the right coefficients matrix.

Considering whether it's beneficial to decode the Canon sources as we've discussed 709 primaries & 601 matrix and put them into 709 and use a lossless codec like UT / lossless h264 / Matrox I Frame ? Sort of a 'poor mans' Neoscene. :-) Or maybe better?
I can't help you on that. I'm not in the editing area, so I really don't know the results...

[Yellow_]

yesgrey, I really appreciate your continued replies.

In trying to read around the subject and try to get a better understanding I found info on xvYCC, http://en.wikipedia.org/wiki/XvYCC and stated there:

xvYCC-encoded video retains the same color primaries and white point as BT.709, and uses either a BT.601 or BT.709 RGB-to-YCC conversion matrix and encoding. This allows it to travel through existing digital YCC data paths, and any colors within the normal gamut will be compatible.

Which sounds like what the Canon DSLR is doing? It's h264 AVC.

Although it suggests for final output:

xvYCC is not supported by DVD-Video or Blu-ray, but is supported by the high-definition recording format AVCHD and PlayStation 3.

However when reading about AVCHD it is suggested it is compatible and designed for Blu-ray, I assume there are various profiles and the higher end of AVCHD accommodate the xvYCC gamut?

Is it possible to use a LUT and retain the xvYCC data in an intermediate for grading / image manipulation and then encoding out to a lesser gamut like 709 or 601?.

Would it need to be as RGB? maybe described in a wider colourspace than sRGB, AdobeRGB perhaps? I see yCMS includes an AdobeRGB output option.

[Yellow_]

yesgrey, could you clarify whether I have this right:

Source is 709 primaries, 601 matrix. Want to convert to 709 primaries and matrix and export from Vdub as uncompressed YCbCr HD. The bit I'm unsure about is the need for the Output transfer matrix or not required?

# Set input format


Input_Primaries 0

Input_Transfer_Function 1

Input_Matrix_Coefficients 3

Input_Bit_Depth 8

Input_Range 0 255



# Set output format



Output_Primaries 0

Output_Transfer_Function 0

Output_Matrix_Coefficients 1

Output_Bit_Depth 16

Output_Range 0 255

re Adobe RGB & xvYCC query from previous post, when converting using 709 pri and 601 matrix to AdobeRGB, I get less colour info per frame using AdobeRGB than sRGB, I'm ensuring that Gimp uses the correct ICC under colour management for importing the image frames, I'd assumed I see better colour info per frame, ie: wider gamut.

For AdobeRGB I was using the same input details as above with these outputs:

Output_Primaries 4

Output_Transfer_Function 5

Output_Matrix_Coefficients 0

Output_Bit_Depth 16

Output_Range 0 255

Does all look correct for generating the LUT's?

[yesgrey]

Source is 709 primaries, 601 matrix. Want to convert to 709 primaries and matrix and export from Vdub as uncompressed YCbCr HD.
# Set input format
Input_Primaries 0
Input_Transfer_Function 1
Input_Matrix_Coefficients 3
Input_Bit_Depth 8
Input_Range 0 255

# Set output format
Output_Primaries 0
Output_Transfer_Function 0
Output_Matrix_Coefficients 1
Output_Bit_Depth 16
Output_Range 0 255

Almost correct. You shouldn't change the TF. You should use:
Output_Transfer_Function 1

If you're not using any of the mandatory commands you must use all STANDARDS DEFINITION commands for input and output.

re Adobe RGB & xvYCC query from previous post, when converting using 709 pri and 601 matrix to AdobeRGB, I get less colour info per frame using AdobeRGB than sRGB, I'm ensuring that Gimp uses the correct ICC under colour management for importing the image frames, I'd assumed I see better colour info per frame, ie: wider gamut.

For AdobeRGB I was using the same input details as above with these outputs:
Is your source full range YCbCr? If not, try using:
Input_Range 16 235

[diffid]

Post removed, wrong thread.

[Yellow_]

Almost correct. You shouldn't change the TF. You should use:
Output_Transfer_Function 1

If you're not using any of the mandatory commands you must use all STANDARDS DEFINITION commands for input and output.

Just to clarify, I have missed the mandatory? So should add:

Input_Format HD YCbCr 8 & Output_Format HD YCbCr 8

When you say almost correct, which is almost correct the 709pri /601 matrix to 709 or the AdobeRGB route? Obviously I want to get the LUTs right before using them on a full project. :-)

If I'm for example going from 709 pri / 601 matrix to 709 for both, then will either YCbCr or HD work the same in Output_Format? Which would be correct?

Is your source full range YCbCr? If not, try using:
Input_Range 16 235

Yes always full range from the cameras I'm using. But if I haven't been doing the LUT generation correctly for these latest tests then that might explain the AdobeRGB problem. :-)

[yesgrey]

Just to clarify, I have missed the mandatory? So should add:
Input_Format HD YCbCr 8 & Output_Format HD YCbCr 8
If you're using all STANDARDS DEFINITION commands the mandatory lose their mandatory state, so, no, you don't need to add those two lines. If you miss any command yCMS would let you know, because it never assumes any default values.

When you say almost correct, which is almost correct the 709pri /601 matrix to 709 or the AdobeRGB route?
The first. You were using the sRGB transfer function (0) instead of the BT.601/BT.709 one (1). The AdobeRGB route is correct.

If I'm for example going from 709 pri / 601 matrix to 709 for both, then will either YCbCr or HD work the same in Output_Format? Which would be correct?
If you are comfortable in using the standards definition commands stick to them. The mandatory always consider YCbCr to be in the standard range, so it's safer for you to have full control on the Input and Output settings.

Yes always full range from the cameras I'm using. But if I haven't been doing the LUT generation correctly for these latest tests then that might explain the AdobeRGB problem. :-)
Right. Let me know the results with the correct LUTs then.

[Yellow_]

Right. Let me know the results with the correct LUTs then.

Ok, I've been trying to establish whether Canon DSLR h264 with 709 priimaries / 609 matrix is xvYCC and whether it is more beneficial to try putting that into a wider RGB gamut than sRGB.

My assumptions to establish either way are:

That sRGB is not sufficient for conversion from xvYCC to RGB and hold all data. ie enough for 709 primaries and matrix, but not xvYCC.

That a wider gamut like AdobeRGB would be needed.

However using the following spec for the 3DLUT:

# Set input format

Input_Primaries 0

Input_Transfer_Function 1

Input_Matrix_Coefficients 3

Input_Bit_Depth 8

Input_Range 0 255


# Set output format

Output_Primaries 4

Output_Transfer_Function 5

Output_Matrix_Coefficients 0

Output_Bit_Depth 16

Output_Range 0 255


Gives me far less unique colours in the exported image frames, (using .tif and reading in Gimp with assumed ICC profile AdobeRGB) than exactly the same process and t3dlut settings.


The sRGB LUT used the same LUT config but substituting sRGB instead and reading the images with an sRGB ICC profile.

# Set input format

Input_Primaries 0

Input_Transfer_Function 1

Input_Matrix_Coefficients 3

Input_Bit_Depth 8

Input_Range 0 255


# Set output format

Output_Primaries 0

Output_Transfer_Function 0

Output_Matrix_Coefficients 0

Output_Bit_Depth 16

Output_Range 0 255

I've tried different methods of wrting the frames out, copy to clip in Vdub, image seq out in Vdub, imagewriter to tif in AVISynth and even Wilberts Imagemagick write to tif, all AdobeRGB ICC images are identical output and far less colours than the sRGB ones. :-(

Are my assumptions about xvYCC totally wrong or is the LUT config still not right?

[yesgrey]

Are my assumptions about xvYCC totally wrong or is the LUT config still not right?
yCMS does not support xvYCC colorspace yet, so that might be the reason for your current results. I plan to support it, but I don't know when.

[Yellow_]

lol yep that could be it. :-)

Oh well, thanks for your time and I'll look out occasionally in case you feel the urge to add support. :-)

[yesgrey]

I'll look out occasionally in case you feel the urge to add support. :-)
I guess I haven't expressed myself clearly enough...

It's not a question of feeling the urge for to add support, is because I don't have the xvYCC standard's documentation. So, until I could be able to access reliable information about it I could not do it. For the current supported formats the specs are widely spread, but for xvYCC I don't.

If you have it let me know and I will consider it soon.;)

[Yellow_]

Sorry for misunderstanding, urge was really wrong word anyway, time and resources.

I'll see what I can find, maybe others on Doom9 may shout out.

Don't want to provide links to what might not be 'reliable', it is Sony after all, but here's a link.

http://www.sony.net/SonyInfo/technology/technology/theme/xvycc_02.html

KMO offers two links in the bottom of first post here and a test pattern in second post. I think you are a part time resident at avsforum anyway. :-)

http://www.avsforum.com/avs-vb/showthread.php?t=1170632

And Neuron2s site for h264AVC additional info:

http://www.avsforum.com/avs-vb/showthread.php?p=17051065#post17051065

I'll try to find written, ratified, for purchase doc if necessary.

Thanks

[yesgrey]

Thanks for the links.

I've also searched in the documents I have and I think I have all the info needed. If it's not too much work I will try to add xvYCC support to yCMS v1.9.

[Yellow_]

Hey, that's good news.

I notice the one link I gave above to avsforum, Neuron2's site needs a password, Neuron2's suggested you PM him at his site if you'd like access.

[yesgrey]

OK, thanks.

[Yellow_]

Hi, I'm unable to eek out anymore from the Canon DSLR video source using xvYCC, maybe the camera doesn't record it. :-( Although from what I've read it appears to be an extension of ITU601 and ITU709 where the range 1 to 254 is used so assumed the camera was xvYCC as I can capture above and below 16 - 235, especially with a flatter picture style camera curve.

Did you see anything / were successful with the gold source?

Sorry to ask again but with the following header info from the camera .mov files:

Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M

I've tried various combinations of settings to generate the LUT. If I want full range RGB from the above source which is 0 - 255 what settings would I need again if I want to assume the source is xvYCC.

Also will using 16bit output dithered down skew the results, been using 8bit out only anyway.

And last query :-), Do you think when going to RGB via the LUT, AdobeRGB or sRGB?

[yesgrey]

Hi, I'm unable to eek out anymore from the Canon DSLR video source using xvYCC, maybe the camera doesn't record it. :-(
That's what I thought too. The Gold sample seems to have the same number of available colors when using BT.709 or xvYCC.

I've tried various combinations of settings to generate the LUT. If I want full range RGB from the above source which is 0 - 255 what settings would I need again if I want to assume the source is xvYCC.
With that you can go as simple as:
Input_Format xvYCC_SD YCbCr 8
Output_Format videoStandard RGB_PC 16

on the Output format use as videoStandard the RGB space you want.

Also will using 16bit output dithered down skew the results, been using 8bit out only anyway.
Using the 16 bit output dithered to 8 bit is always preferable, unless you plan to compress the source, but since you're talking about RGB space I don't think you are considering any compression...

[yesgrey]

Do you think when going to RGB via the LUT, AdobeRGB or sRGB?
If your display supports AdobeRGB it would be preferable. It's always a good idea to use the widest color gamut supported by the display.

[Yellow_]

In your Config above you've used SD rather than HD is that deliberate ie because the transfer is BT601?

I wonder whether the cameras 'Picture Style' camera curves play a part? Would the curve be applied in the conversion from linear to gamma encoded I wonder, could that reduce the chances of seeing xvYCC? More testing I think. :-)

I don't think the curve styles affect actual captured range ie 0 - 255 but only colour content.

However I think I used 'Faithful' for the Gold source as I had the thought in mind then. I also use the Marvel Styles that emulate a LOG space.

Whats peoples general opinion of camera curves here I wonder?

Also about colour gamut, the app I'm using is linear RGB is that a wide gamut compared to AdobeRGB or does gamut not apply to liear space?

Would it be worth transforming to linear RGB or detrimental many apps appear to do all processing in linear space such as Nuke and AE.

[yesgrey]

In your Config above you've used SD rather than HD is that deliberate ie because the transfer is BT601?
Yes. According to the spec the BT.601 coefficients matrix is to be used with SD and the BT.709 with HD, hence why I named them like I did. I think it's more user friendly than xvYCC601 and xvYCC709.;)

I wonder whether the cameras 'Picture Style' camera curves play a part?
Sorry, but I don't know what you're referring to...

Also about colour gamut, the app I'm using is linear RGB is that a wide gamut compared to AdobeRGB or does gamut not apply to liear space?
The colour gamut depends on the RGB primaries only. Being linear or non linear doesn't matter, the colour gamut would be exactly the same. You need to know which are the primaries coordinates of the linear RGB space the application works with.

Would it be worth transforming to linear RGB or detrimental many apps appear to do all processing in linear space such as Nuke and AE.
If the application could accept the 16 bit output of the 3DLUT, then it might be beneficial, but if it only accepts 8 bit I think you should go with a non linear output, and let the application perform the linearisation itself.

[zcream]

A Q about xvYCC. For this extended color gamut, do we need YUV 10-bit 4:2:2 ?
The HDC-SD9 advertises xvYCC output via HDMI.
However, every other camcorder I have seen gives 8-bit YUV 4:2:2 via HDMI. For greater color depth would we need to go from 8-bit to 10-bit HDMI ?

[yesgrey]

A Q about xvYCC. For this extended color gamut, do we need YUV 10-bit 4:2:2 ?
No. xvYCC is an extension of BT.709, so it's still 8-bit. The gamut extension is achieved by using the CbCr values of 1-15 and 241-254.

[zcream]

So, if I use a HDMI capture card to get lossless 8-bit 4:2:2 data, do I need to do something to map the extra data ?
I would assume that it gets recorded normally.

[yesgrey]

So, if I use a HDMI capture card to get lossless 8-bit 4:2:2 data, do I need to do something to map the extra data ?
I would assume that it gets recorded normally.
Sorry, but I don't quite understand what you are intending...

[zcream]

Hi! Chroma is usually 16-240 for YUV video. For xvYCC it is 0-255. My question related to capturing this signal via HDMI. If I use a video capture program like VirtualDub, does it capture the extra chroma values for chroma ?

Problem is that a HDMI capture card outputs HDYC, which is then converted to RGB24 (as the video preview is seen on the monitor), then it is converted to YUV 4:2:2 for capture.
The extra chroma values would give a negative RGB value, so I would think they get thrown away.

[Yellow_]

Hi! Chroma is usually 16-240 for YUV video. For xvYCC it is 0-255. My question related to capturing this signal via HDMI. If I use a video capture program like VirtualDub, does it capture the extra chroma values for chroma ?

If your camera captures outside the 16-240 range as most do, certainly my Canon DSLR, HV30 & old JVC DV do then you have access to both. But just how much is xvYCC data is the thing. :-)

f your NLE or whatever is strictly 16 - 240 then you'll loose the rest and I think it's the case that if you're NLE or whatever works above 8bit processing(ie 10, 16 or 32bit) then they may well allow a full range workflow holding onto whatever extra that may have been captured.

Problem is that a HDMI capture card outputs HDYC, which is then converted to RGB24 (as the video preview is seen on the monitor), then it is converted to YUV 4:2:2 for capture.
The extra chroma values would give a negative RGB value, so I would think they get thrown away.

From what I've found so far doing xvYCC / full range to R'G'B is that out of the potential far wider gamut that xvYCC is said to have by the time the 4:2:0 subsampling has been done there is I think only the potential for 2.75million colors to be utilised out of the 16million and therefore where I thought I needed to use AdobeRGB or some other wide gamut profile to work with xvYCC converted to R'G'B I don't think it's necessary. :-) However if it were possible to capture 4:2:2 or 4:4:4 then something bigger than sRGB gamut might be needed I guess?

To eek out most there maybe some more to find using certain 'picture profiles' to pack more values in, possibly. Shoot Neutral or with Marvels profile but wonder whether a vivid maybe better for xvYCC?


Which leads me onto a couple of questions. I think I know the reason why I'm seeing no more colour values than before the xvYCC addition to yCMS. Here my thinking to be put right. :-)

Normal NLE handling would be 16 - 235/240 in 8bit processing, The conversion to RGB by taking reference black as 16 and reference white as 235 and putting them on that 0 1 abstract scale. To my understanding that is Rec709 to R'G'B. sRGB.

What I've been doing is using the full range because the camera captures it and in AVISynth using PC.709 as the matrix and yesgrey's yCMS with Full Range Input/Output, getting all the range and putting it in R'G'B so already doing a xvYCC conversion to R'G'B.

Which raises the question about the abstract scale, the conversion to R'G'B and handling there on in an NLE.

If an NLE only extracts 16 - 235/240 and puts it on that 1 0 scale and applies 8bit processing, I make the distinction because I think that 10,16 or 32bit and that it's float, is required for holding onto the negative values and values over 1 ie xvYCC. Then applies colour processing on a 0 1 scale set to 0 to 255 that would be incorrect handling unless they had used the full range in a bit depth greater than 8bit in the initial YCbCr to R'G'B conversion or I guess this is why they scale 16 - 235 to 0 - 255 for R'G'B processing in 8bit thus staying inside the supposed 'legal' boundaries for video?

[yesgrey]

Hi! Chroma is usually 16-240 for YUV video. For xvYCC it is 0-255.
Not quite. For xvYCC is 1-254. The 0 and 255 values should not be used with video data.

Problem is that a HDMI capture card outputs HDYC, which is then converted to RGB24 (as the video preview is seen on the monitor), then it is converted to YUV 4:2:2 for capture.
The extra chroma values would give a negative RGB value, so I would think they get thrown away.
I think it would be easier if you tell me which HDMI capture card are you thinking of (post a link, please)...

[yesgrey]

From what I've found so far doing xvYCC / full range to R'G'B is that out of the potential far wider gamut that xvYCC is said to have by the time the 4:2:0 subsampling has been done there is I think only the potential for 2.75million colors to be utilised out of the 16million and therefore where I thought I needed to use AdobeRGB or some other wide gamut profile to work with xvYCC converted to R'G'B I don't think it's necessary. :-) However if it were possible to capture 4:2:2 or 4:4:4 then something bigger than sRGB gamut might be needed I guess?
4:2:0, 4:2:2 and 4:4:4 have nothing to do with the number of available colors. In all the modes you will have the same number of available colors. The difference between them is the color resolution of an image. With 4:4:4 the chroma and luminance resolutions are the same. With 4:2:2 the chroma is half the resolution of the Luminance, and with 4:2:0 it's 1/4.

I will not comment any of the other parts of your post because it's very hard to understand what you are trying to say. Sorry.

[zcream]

Hi
I am using the Blackmagic intensity capture card.
http://www.blackmagic-design.com/products/intensity/

The colorspace output by the card was discussed here, and a special build of Huffyuv created.
http://forums.virtualdub.org/index.php?act=ST&f=6&t=16116&#entry66552

We did a test posted at vimeo
http://vimeo.com/10174263 - Comparison of Blackmagic MJPEG and Huffyv colorspace on Vimeo
We played out a SMPTE color bar from a 5d Mark II. It was ingested with a BM Intensity card using Virtualdub. There were 2 files, a special Huffyuv build for HDYC and the default BM MJPEG. As you scroll through the video it can be seen that the colorspace changes when Huffyuv records. This is a confirmation of the problem first noticed at
etfinder.net/capturepics/

To my knowledge, only AMV2-MT and Cineform handle the color conversion properly to capture a direct feed.



Not quite. For xvYCC is 1-254. The 0 and 255 values should not be used with video data.


I think it would be easier if you tell me which HDMI capture card are you thinking of (post a link, please)...

[Yellow_]

yesgrey, here again. :-)

I've started to read up on DCP (Digital Cinema Package).

http://reduser.net/forum/showthread.php?t=33118

Open Source route:

https://github.com/wolfgangw/digital_cinema_tools/wiki/Open-source-tools-for-a-digital-cinema-pipeline

Part of the process is getting from sRGB to X’Y’Z’ (Gamma 2.6) however my interest is from Canon DSLR YCbCr 1 - 254 xvYCC to X’Y’Z’ (Gamma 2.6) and then output Jpeg2000 image sequences via Wilberts Imagemagick AVISynth plugin.

What would be involved in getting a 3D LUT created to X’Y’Z’ (Gamma 2.6) is it possible within what you've created so far in yCMS?

[yesgrey]

What would be involved in getting a 3D LUT created to X’Y’Z’ (Gamma 2.6) is it possible within what you've created so far in yCMS?
Yes, it should be possible, but I haven't tested it yet. I'm planning to include that option in next yCMS version, but you can already use it with the current one by defining it on your own by using:

Output_Primaries 1.000 0.000 0.000 1.000 0.000 0.000 0.3127 0.3290
Output_Transfer_Function 1.0 0.0 0.384615384615385 0.0

Note: I've used D65 white point, but I think you can use any other you would prefer. I've read the DCI spec and did not find any reference for a specific white point.

[Yellow_]

Excellent, once again thanks, going to give it a go. :-)

Looks like last bit of the jigsaw is a avisynth plugin using Openjpeg to export the jpeg2000 images unless imagemagick does it now, via Wilberts plugin.

[yesgrey]

Hi! Chroma is usually 16-240 for YUV video. For xvYCC it is 0-255. My question related to capturing this signal via HDMI. If I use a video capture program like VirtualDub, does it capture the extra chroma values for chroma ?
I don't know, but if it works according to spec it should, because the entire range 1-254 is supposed to contain video data, but to be sure ask VirtualDub's author about it.

Problem is that a HDMI capture card outputs HDYC, which is then converted to RGB24 (as the video preview is seen on the monitor), then it is converted to YUV 4:2:2 for capture.
The extra chroma values would give a negative RGB value, so I would think they get thrown away.
If you perform any conversion from HDYC to an RGB that could not hold the entire range of colors you will lose them. However, why would you perform that conversion if the capture would always be in YUV? Just capture the original data without any processing.

Am I missing something?

[Yellow_]

Queried xvYCC handling with Virtualdub author and Vdub can't handle an RGB gamut wide enough to contain xvYCC in RGB such as AdobeRGB, think Vdub is sRGB only.

That could well explain why I wasn't seeing anymore colors than just doing full range conversion prior to you implementing xvYCC, both must have been clipping at the boundary of sRGB.

Need to find an alternative tool to process avisynth scripts, maybe FFmpeg on the command line? Hack Blender to support AdobeRGB, as it only supports sRGB and linear currently but can process avs scripts via ffmpeg.

Or perhaps when I was using Wilberts Imagemagick plugin it was writing out images of sRGB gamut, maybe need to revisit that and see if there is a way to set gamut to AdobeRGB in imagemagick via his plugin.

Still haven't got my head round linear space yet, I assume the width of gamut is the same as the source?

If an app has a 32bit float linear compositing pipeline and writes images out either as linear exr or gamma encoded formats like png, jpg etc restricted to sRGB, is the linear exr restricted sRGB gamut or does linear have a capability for a wide gamut?

[yesgrey]

Still haven't got my head round linear space yet, I assume the width of gamut is the same as the source?
Yes, it's exactly the same width. The linear vs gamma only affects the distance between the different points inside the gamut, and not the gamut itself.

[zcream]

Just wondering how we could even get xvYCC data in a video file anyway ?

I checked with Blackmagic Design, and it turns out that the chipset they use clips the colors from 16-240. Most HDMI capture cards use the same HDMI chipset - hence its not possible to capture this color range.

xvYCC is only present in the HDMI specs, I dont see it on any other video transport spec.

So how can one get xvYCC video data anyway ?

Also, with VD, if the input stream is YUY2 or HDYC, you can use Fast Recompress and thus turn off any RGB conversion. This way, any xvYCC data should be retained during conversion.

[Yellow_]

Just wondering how we could even get xvYCC data in a video file anyway ?

xvYCC is only present in the HDMI specs, I dont see it on any other video transport spec.

So how can one get xvYCC video data anyway ?



http://forum.doom9.org/showthread.php?p=1464035#post1464035

[yesgrey]

I checked with Blackmagic Design, and it turns out that the chipset they use clips the colors from 16-240.
This kind of thing always amazes me... Why do they do it? Even the ITU.BT-709 standard clearly says that video data is all the range from 1-254, so why do they clip it?
Then, this kind of thing happens. A new standard is developed considering the previous one was fully respected, but it wasn't...

[Yellow_]

Majority I guess, are there many sources that are full range by specification?

Broadcast, cable, satalite? Video cameras? a few video cameras support xv.color but it's not widely used.

But then what codecs support full range by specification? h264AVC is the only one I know.

Uncompressed capture will need a RAID.

Then having captured full range, other than playback on a TV with hdmi 1.3 and able to display full range correctly, what else to do with it, burn it to Bluray, not supported.

Edit it, what NLE's actually handle xvYCC without squashing it into 16 - 235/240 in some intermediate codec to do all the RGB processing on it? Not many I think. :-(

[poisondeathray]

Edit it, what NLE's actually handle xvYCC without squashing it into 16 - 235/240 in some intermediate codec to do all the RGB processing on it? Not many I think. :-(

This is true, but some can access Y'CbCr in filters even though they function in RGB . These filters are applied before the NLE converts to RGB. eg. if you have "illegal levels" you can use YUV filters to bring back into legal ranges. Premiere Pro CS5 does this for some filters. But I don't know how xvYCC is handled specifically

[Yellow_]

Yes, I think Premiere CS5 is one of the can handle it based on a comment from a Adobe representative:

Adobe CS5 reads the H.264 files natively into Premiere Pro and After Effects at the highest possible quality. Our color gamut and dynamic range for tonal detail from shadow to highlight is unsurpassed. There is even support for over-brights beyond 100% in After Effects.

The magic comes from the use of proprietary interpretation algorithms and I might also mention that we bypass QuickTime for this process, which avoids the whole gamma conundrum. Once the file is living inside our apps on the timeline or project, we deal with the image information at the 32 bit float level. Now that is not saying we can make an 8 bit H.264 DSLR video capture look like perfectly shot IMAX footage scanned at 16 bits, but what we do offer up is the ability to edit, apply effects and color corrections within our apps. at an unprecedented level of quality.

Obviously not proof but interesting. :-)

[xbox360]

What is the gamut for NTSC-J ? & how do I use it in an Avisynth script ?

[yesgrey]

What is the gamut for NTSC-J ?
Is it an analog source? If it is you might need to use the "US NTSC 1953" options. In addition to that you should use an input range that would compensate for the black point difference.

how do I use it in an Avisynth script ?
With t3dlut.

[xbox360]

In addition to that you should use an input range that would compensate for the black point difference.


With t3dlut.

And what would that input range be ?

[yesgrey]

And what would that input range be ?
It would depend on how the capture is performed. How are you doing it?

[xbox360]

sRGB captures.

[yesgrey]

sRGB captures.
Are you sure? Which card are you using? Also, tell me which output do you want? Tell me as much information as you can, only then I could help you creating a configuration file to use with yCMS.

[xbox360]

sRGB captures from Camera Fujifilm A100 .avi JPEG convert to NTSC-J. It would be great if you can make me a PAL to NTSC-J & NTSC to NTSC-J.

[yesgrey]

sRGB captures from Camera Fujifilm A100 .avi JPEG convert to NTSC-J. It would be great if you can make me a PAL to NTSC-J & NTSC to NTSC-J.
Start by trying this:

# PAL to NTSC-J
Input_Format PAL RGB_PC 8
Output_Format NTSC RGB_PC 8

and

# NTSC to NTSC-J
Input_Format NTSC RGB_PC 8
Output_Format NTSC RGB_PC 8
Input_Range 16 255

If you prefer you can use 16 bit depth on the Output_Format commands.

I'm not sure if these would be the correct configuration files, so let me know how they work out.

[leeperry]

A CUDA implementation would be easy to do, but I was thinking that it would be pretty limiting (in terms of supported video cards) compared to a pixel shader.
hi tritical, 3 years later times have changed...how about OpenCL then?

I kinda like the idea of keeping the gamut mapping stuff outside the VR...so we can set automatic rules in ffdshow, and get ddcc to use SMPTE-C/EBU/HDTV gamut mapping depending on the frame rate and the video stream native resolution. Also, there's no need to wait for the 96MB 3DLUT to be loaded each and every time you wanna watch a movie...so you get all the pros w/o any of the cons, too good!

Doing all this in the VR would require auto-detection of upscaled SD(madVR currently forces the HDTV gamut when x>1024) and a lof new automatic rules code that's already in ffdshow...no need to reinvent the wheel really.

I realize that doing all this in 8bit might not be the best idea, but SmoothLevels() does very impressive dithering...so this point is more or less covered. And SmoothLevels() is also using a kludge to allow 16bit in Avisynth 2.6(stacking MSB/LSB in YV16/YV24 (http://forum.doom9.org/showpost.php?p=1504207&postcount=210)). I guess that would be fairly easy to implement in ddcc? So provided that some small changes in ffdshow would be made(I think I know someone who could commit them), we could have a full 16bit avisynth pipeline in ffdshow, and feed it as P216/RGB48 to madVR :devil:

I was also told that it would be a far better idea to post-process video when already using the right colors, instead of changing them afterwards...that'd make a lot more sense to use ddcc in 16bit at the very top of my ffdshow/avisynth scripts, than using a 3DLUT in madVR at the very last stage.

Hope you'll consider it, :thanks: in advance! I really miss ddcc in Avisynth, but it's a CPU hog...getting it to run off the GPU would be a God bless, and a perfect solution for on-the-fly and automatic rules based gamut mapping. A killer combo together w/ the supreme smoothness of madVR :cool:

[leeperry]

@tritical: any chance you could make yv12toyuy2() fallback to itype=0 if it's not mod4 please?
which can easily be overcome by using this script (http://forum.doom9.org/showpost.php?p=1453641&postcount=22), that will pad black borders in order to become mod4.

it's nice to see that solutions do arise over time...now all that'd be required is OpenCL GPU acceleration for ddcc, and then this would become an entirely workable solution :cool:

[Kazuya]

Hope you'll consider it, :thanks: in advance! I really miss ddcc in Avisynth, but it's a CPU hog...getting it to run off the GPU would be a God bless, and a perfect solution for on-the-fly and automatic rules based gamut mapping. A killer combo together w/ the supreme smoothness of madVR :cool:

Automatic switching gamut would be great of course !!! :)

[tritical]

Do you want ddcc() or rgb3dlut() on the gpu?

[leeperry]

hi tritical, thanks for the reply!

well, ddcc() works in realtime, doesn't need to read LUT's and can provide gamut mapping on its own...IIRC, it was chosen to go LUT in order to drastically lower the CPU usage. Going GPU accelerated would alleviate this problem, so I don't really see the point to bother w/ LUT's anymore?

OTOH, ddcc() requires RGB24/RGB32 input, which takes a lot of CPU cycles....rgb3dlut() uses small 8bit LUT files and only needs YUY2 input.

tough call! I guess rgb3dlut() would make a lot more sense indeed...and doing the RGB32 conversion via the GPU is always a nice touch :)

[tritical]

Yeah, I guess what I was asking was what colorspace would you be using for input/output? ddcc is much easier to put on the gpu - in terms of complete feature set - (in fact I did it last night in cuda since I have a lot of experience with cuda and none with opencl), but has the restriction of rgb24/rgb32 input/output. Of course, that input/output restriction is what makes it an easier task than putting rgb3dlut on the gpu. I could add yuy2 input to ddcc, but what output format makes sense in that case (yuy2, rgb24, rgb32, all three)? Plus with yuy2 input I'd have to handle different chroma placements, upsampling methods, etc... which is kind of a pain.

[leeperry]

Oh, YUY2 input on ddcc() would rock! I'd be using YV12 input in the best case scenario, otherwise YUY2. And I'd need RGB32 output, I don't see much point to RGB24? Ideally YV12 input/output would be best, but that'd require a lot of lossy conversions I guess...or maybe you could add dithering to make it less painful? :o

That's for the regular 8bit Avisynth pipeline, because P216/Y416 input and RGB48 output could be useful sometime in the future.

I'm from team nvidia, so a CUDA version would be truly fantastic...you could always go OpenCL in the future if there's any request for it(and that you want to bother w/ OpenCL in the first place).

Indeed, a CUDA version of ddcc() that would accept YUY2 input and converts to RGB32 using the GPU would really be beyond words! If it could also convert from a YV12 input to YUY2 internally using the GPU would be even better :cool:

And feeding RGB32 to madVR wouldn't require using ColorMatrix() to convert the 601 decoding matrix coeffs to 709 for upscaled SD(because madVR uses 709 if x>1024) anymore, so that'd mean one less 8bit Avisynth plugin to process :)

:thanks:

[tritical]

In the interest of actually getting something released, I put up a new version of ddcc which has a CUDA implementation of ddcc(). Syntax wise the only thing that changed is 'opt' can be set to 2, which forces cuda to be used. By default, it will use the cuda implementation if a suitable device is found. I do not perform a search over all devices, just query the default device for the thread that ddcc is created in. I only tested it on an old 9800 gtx+, speed was the same as the cpu version on my q6600 quadcore.

I might add yuy2 support or other things in the future as time/motivation allows.

[leeperry]

ouh that was a fast, thanks a lot!

I'll be testing it and report back then :)

[leeperry]

having second thoughts, do you think it would be possible to get ConvertToRGB32() to use CUDA as well? that would be a perfect solution as a combo w/ the CUDA build of ddcc(), and far less work for you than reinventing the wheel? I guess the source code should be fairly easy to locate :)

[leeperry]

having even more thoughts about it, if you could embed a CUDA version of ConvertToRGB32() and ddcc() into one single plugin, this would avoid one very lossy 8bit pass? going YV12 > 32fp RGB(or RGB48 for that matter) > ddcc > RGB32(and RGB48 as well, as madVR supports it natively...but that'd require some modifications in ffdshow).

:thanks:

[tritical]

Creating a full converttorgb (with different interpolation functions, chroma placements, interlaced vs progressive for yv12, etc...) would take more time than I want to spend. I would be willing to start with a single case for ddcc: yuy2 input with mpeg2 chroma placement and linear interpolation for upsampling to 4:4:4 and output rgb32. Would that be usable?

[leeperry]

Oh, you've got me a bit lost here...is it what RGB32HQ in ffdshow and ConvertToRGB32() do? then that would be great :)

from reading the rgb3dlut() manual, "mpeg2/mpeg4/h264" chroma placement sounds good, I think the "Mitchell-Netravali two-part cubic interpolation" used to look better than "linear interpolation"...but if that's too much trouble, let's forget about it. I could always blur the chroma in madVR if I like(I upscale before going through the Avisynth scripts then I let madVR downscale using the GPU, otherwise LSF gives jaggies at 1:1 resolution...hence the requirement to use "SuperSampling" as they call it).

And I guess MPEG1 wouldn't look too ugly when using MPEG2 chroma placement? I dunno what placement madVR and ffdshow assume, prolly MPEG2 as well :o

The main goal would be to have the RGB32 conversion done through CUDA, in order to save CPU cycles indeed. And possibly output RGB48 to madVR when Avisynth/ffdshow will allow it.

:thanks: for all once more, as you're one of the shapers of my dreams :cool:

[leeperry]

OK, I finally found the ddcc() calls I was using a few years back, but now ddcc() complains that I don't have "cudart32_40_17.dll" :o

I run a 9600GSO(rebadged G92 8800GS w/ 96SP) and the 257.21 drivers on XPSP3, and I've got "nvcuda.dll 6.14.12.5721". CoreAVC 2.55 CUDA works like a charm, and "CUDA" support is checked in GPU-Z.

Is that a DLL that can be found in the dev.kit? http://developer.nvidia.com/cuda-toolkit-40

Or shall I update to the newest WHQL drivers? They don't mention CUDA 4.0, though: http://www.nvidia.com/object/winxp-275.33-whql-driver.html

[jmac698]

I'm new to t3dlut etc., is there a way to convert yuv (rec709) to Lab?

[leeperry]

Because of the CUDA error message, I cannot seem to be able to use ddcc() in SSE3 mode either...I did set "opt=1" so it shouldn't even try to use CUDA? :o

PS: ok, using 1.11 and spot-on gamut mapping in plain RGB32 fed to madVR is really major bliss! I can't use spline SuperSampling anymore because the software RGB32 conversion and gamut mapping are quite a CPU hog...can't wait for the full CUDA version. :thanks:

[Yellow_]

Having now established a way to export 10bit h264 and 16bit RGB using avs2yuv is it possible to use t3dlut to export 10 / 16bit rather than it dither back to 8? Is setting output bitdepth to 16 in the LUT generation all that is reqired?

[tritical]

I'm new to t3dlut etc., is there a way to convert yuv (rec709) to Lab?

Not at the moment. Why do you need to convert to LAB? Conversion to LAB from XYZ is simple, but outputting lab as 8-bit would not work very well.

Having now established a way to export 10bit h264 and 16bit RGB using avs2yuv is it possible to use t3dlut to export 10 / 16bit rather than it dither back to 8? Is setting output bitdepth to 16 in the LUT generation all that is reqired?

Current version of t3dlut only supports 8-bit output. I am not update to date on avisynth 2.6 or whether it supports higher bit depths. The 8-bit output requirement is a restriction of avisynth (at least with 2.5.8), if avisynth supports higher bit depths then t3dlut could be made to dither to those.

but now ddcc() complains that I don't have "cudart32_40_17.dll"
Is that a DLL that can be found in the dev.kit?
Yes, it is part of the 4.0 toolkit. The ddcc.dll is linked statically against it so it must be found in the system path in order for the dll to load at all. I believe I included it in the ddcc.zip archive.

[jmac698]

I want to do processing in L*a*b, but output to 8bit yuv again. I'm using this to colormatch videos, so lab is the ideal colourspace for this.

If we wait for high bit support in avisynth, it will be a long time. I suggest supporting one of the workaround formats. Smooth levels for example supports high bit depths and dithering down to 8 bit. It uses MSB stacked vertically on top of LSB. My deepcolor tools uses a separate clip for MSB and LSB. I also support levels adjustments, and I can input true highbit videos. I helped with the update to sashimi to support reading high bit quicktime formats.

[Yellow_]

Current version of t3dlut only supports 8-bit output. I am not update to date on avisynth 2.6 or whether it supports higher bit depths. The 8-bit output requirement is a restriction of avisynth (at least with 2.5.8), if avisynth supports higher bit depths then t3dlut could be made to dither to those..

Thanks to the Dither 1.9 functions, stacked lsb/msb and avs2yuv, 10bit h264 encoding and 16bit image writing via Imagemagick using standard out successfully works with Avisynth 2.5.8.

A git build of ffmpeg is required to successfully play 10bit h264 however and that also appears to work.

[leeperry]

Yes, it is part of the 4.0 toolkit. The ddcc.dll is linked statically against it so it must be found in the system path in order for the dll to load at all. I believe I included it in the ddcc.zip archive.
Oops, my bad! So I've copied it to the system32 folder but now it says "opt = 2 but no cuda device detected".

Can I send you a log somehow? Should I install the dev.kit in your opinion?

And once that'll work, how to figure out what to set for "oog" if you don't mind me asking?

:thanks:

[tritical]

leeperry, it is probably a driver issue. I would try upgrading to the latest driver from nvidia (275.33?). As for 'oog', I think 1 would generally be preferable since it maintains hue/saturation. It's really just a personal preference.

jmac698/Yellow_, I will look in the stacked MSB/LSB output. Or someone else can modify it if they feel so inclined. I can merge the changes.

[leeperry]

As for 'oog', I think 1 would generally be preferable since it maintains hue/saturation.
Oh indeed, my CRT barely reaches SMPTE-C and when I map gamuts from EBU...if I set oog=0 then the colors end up utterly undersatured :o

leeperry, it is probably a driver issue. I would try upgrading to the latest driver
you nailed it! I was afraid to update coz that'd mean setting up all my custom res all over again, but they seem to have improved the custom res panel..and it all went like a breeze :)

BTW, some tests results in VDUB(Q9450@3.5Ghz on XPSP3):

ddcc(chr_i=3,gam_i=5,gam_o=5,ofile="P:\CRT.txt",threads=4,opt=1)
http://www.pixelz.fr/e/3/4/ca018bb46b31e6ba87f2858bae662.png

=252 fps

ddcc(chr_i=3,gam_i=5,gam_o=5,ofile="P:\CRT.txt",opt=2)
http://www.pixelz.fr/e/6/8/850977083923dfb2fa24777dd5582.png

=230 fps

How could I change the cores affinity? I'd rather have the load evenly balanced between the four cores if any possible :o

I've tried to force "threaded optimization" for my media player in there, but that gives randomly dropped frames(that don't occur when it's left on "auto"): http://www.pixelz.fr/9/f/4/dd152000bd4ff9f4d18e332d11adct.jpg (http://www.pixelz.fr/9/f/4/dd152000bd4ff9f4d18e332d11adc.png)

PS: ddcc() works beautifully in combination w/ CoreAVC CUDA of course :cool:

That's an untouched BD w/ CoreAVC CUDA 2.55 + dddc() CUDA + madVR Spline downscaling: http://thumbnails45.imagebam.com/13719/e97bf1137184129.jpg (http://www.imagebam.com/image/e97bf1137184129) http://thumbnails46.imagebam.com/13719/d707a4137184130.jpg (http://www.imagebam.com/image/d707a4137184130)

PPS: it's pretty annoying that "ConvertToRGB32" crashes most of the time when used in MT("",4) mode...lotsa wasted CPU cycles running it non-MT :o

[Yellow_]

jmac698/Yellow_, I will look in the stacked MSB/LSB output. Or someone else can modify it if they feel so inclined. I can merge the changes.

Excellent, enquring if you've had time to give this any more consideration.

[leeperry]

Hi tritical,

I've been using the CUDA build of ddcc() ever since you released it(XPSP3/96SP 8800GS/275.33), but I've encountered 2 problems when switching back and forth between input gamuts(chr_i=0/2/3) in ffdshow(without restarting the video). In both cases, I wasn't using CoreAVC CUDA...only PotPlayer/madVR and ffdshow audio+video.

That's the call I was using:
ddcc(chr_i=3,gam_i=5,gam_o=5,ofile="P:\CRT.txt",opt=2)

1) A few days ago, after switching back and forth like 4 times, I got the "opt = 2 but no cuda device detected" error message...so I closed the video, reopened it and it worked fine again.

2) a few mins ago, I was switching like +8 times(trying to figure out if this 30fps european SD video was EBU or SMPTE-C :rolleyes:) then I got a brief green flash at the second third of the screen, the computer locked up on the last video frame before the green flash...and bye bye Kansas, not even a soft reset would allow the computer to work again. My mobo BIOS decided that the graphic card was crashed for good and ran a hard reset instead.

That's too bad I've got no log to provide you w/, hopefully you could bulletproof the realtime changes in order to avoid any crash/deadlock please?

:thanks: again for your hard work, much appreciated!

[leeperry]

Ahhh, there's a new option in madVR 0.67 to force prebuffering when seeking, so I was trying it and enabling/disabling LSF on the fly in ffdshow to see how the new option would react(I use several scripts in ffdshow: SmoothLevels(for TV>PC conversion)>LSF>GrainFactory3>ddcc). I was running madVR in FullScreen Exclusive mode, and after I had toggled LSF like 6 times in a row, I got the very same green flash at the second third of the screen(like this: http://thumbnails50.imagebam.com/14100/129389140995987.jpg (http://www.imagebam.com/image/129389140995987) ) for a half second, then the picture froze, the mouse froze a few secs later...and I was forced to reboot. My mobo BIOS managed to get away w/ a soft reset this time.

Any chance you could reproduce tritical? Any chance for a fix please please please?

I will now refrain from making realtime changes in my ffdshow Avisynth scripting, and I have to say that I'm really amazed by the colors and PQ when using proper gamut mapping via ddcc() :eek:

:thanks:

[tritical]

@leeperry I would fix it if it had anything to do with ddcc, but there's no way ddcc can cause that or anything in the code I can change. It must be either a driver/card issue, or the software you are running avisynth in is not closing the associated thread(s) (and associated cuda context(s)) properly.

Also, for anyone using it, I would not advise using setmtmode or other avisynth multithreading methods with ddcc+cuda. However, it should work correctly as long as the getframe method is called in the same thread as the constructor/destructor.

[leeperry]

oh ok, thanks for the reply! but ddcc() works perfectly fine in software mode when messing in realtime w/ the avisynth filter of ffdshow...it seems rather clear that sometimes the CUDA connection is not closed properly and hell breaks loose at the GPU drivers level =/

so indeed ffdshow doesn't close the threads properly, and "Leak"(the coder who took care of the avisynth support in ffdshow) has recently made it clear to me that he doesn't plan on developing it any further.

[leeperry]

OK I've found my old Avisynth scripts and going:
ConvertToYUY2()
rgb3dlut(lutfile="P:\smpte-c",threads=4,itype=2,b=1.0,c=0.0)

carries many advantages:
-no CUDA drivers required, meaning that ffdshow won't crash my computer anymore when rolling gamuts on the fly
-no need to make a CPU hogging ConvertToRGB32() conversion
-the ability to use oog=1, which isn't possible in CUDA mode
-ddcc() CUDA still uses quite a lot of CPU cycles from what I've seen, mostly due to the nvidia drivers I presume...OTOH, rgb3dlut() is very light CPU-wise.

I gave up on this solution because up to very recently, mVR only supported YV12 input.

Also, using rgb3dlut() in ffdshow allows automatic profiles based on resolution & frame rate, and mVR now uses a new colorimetry system called "yRGB" that requires a 16bit 96MB LUT(that takes forever to load), won't remember the last used gamut and won't allow automatic profiles (yet)...so all in all, going back to the old school sounds very reasonable. Shelling out $500 on a new i7 box in order to get CPU based gamut mapping simply does not :rolleyes:

I fully realize that doing all this in 8bit is a terrible idea, but 16-235 SMPTE-C(the smallest gamut in the industry) isn't quite as data intensive as xvYCC I guess. And tbh SmoothLevels() does such an amazing job at dithering that I highly doubt anyone could DBT it from native 10/12bit.

PS: ends up 8bit LUT's are 48MB, I've got 3 gamuts and 2 YUY2 LUT's for each decoding matrix...so that's 240MB of LUT's(I don't see how HDTV could use BT601).

PPS: I might as well input and output YUY2 using t3dlut(), but then I'd have to rely on mVR to either go 601 or 709 RGB...and t3dlut() requires far more CPU cycles than rgb3dlut().

Actually, rgb3dlut() in YUY2 is so light on the CPU that I can use Gavino's script (http://forum.doom9.org/showpost.php?p=1453641&postcount=22) to automatically become mod4 and yv12toyuy2(itype=2,threads=4) http://forum.slysoft.com/images/smilies/agreed.gif

yv12toyuy2(itype=2,threads=4,b=0,c=0.75) is way too edgy...0.33/0.33 would appear far more reasonable, as madshi advised (http://forum.doom9.org/showpost.php?p=1276180&postcount=426).

[leeperry]

Alright, I will need to do screenshots comparisons against vanilla ddcc() in order to ensure that gamuts mappings & YCbCr decoding matrixes are processed properly, I'll put my 240MB of LUT's on a PC8500 ramdisk for instant access then it'll all be looking very much kosher to me :D

It's great how it all fits perfectly together in order to provide a very impressive HTPC experience :)

The ability to change the bicubic coeff for both the YV12>YUY2 interpolation and RGB32 conversion really allows you to finetune it all perfectly to your 1) video processing pipeline 2) equipment 3) taste...I'm really stunned by the PQ tbh :cool:

[leeperry]

Bumpity, Bump :)

AVS 2.60 supports ConvertToYUY2(chromaresample="Spline64") in MT mode, so that more or less renders yv12toyuy2(itype=2,threads=4) useless? And it's not mod4-only so that makes my life a lot easier.

Also, I might consider creating a bigger ramdisk and use 16bit YUY2 LUT's instead of 8bit, but why is t3dlut() so much more power hungry than rgb3dlut() again? :o

[nand chan]

Does anybody have plans for the future of the .3dlut spec, or possibly a v2/v1.1?
These are some of the issues I've noticed while working with the current, and I'd be more than happy to help discuss changes / alterations to improve these:


Many programss still have to scan through the parameters file to find out some information that is not present in the tags
No tag for the value range (full range, limited range, min/max, etc)
No concept of color space - a color-aware 3dlut could easily be achieved with optional gamut tags, this would allow tools such as madVR to transform color into the input color space
It seems like nothing actually uses the “color encoding” fields, they're set to 0 even on YCbCr LUTs. Am I mistaken of its purpose? This could be improved to reflect the color encoding nature (YUV, RGB, BGR, etc)
No currently defined compression methods

[madshi]

yesgrey was planning to specify the meaning of the color encoding fields for a while. He hasn't had the time to do that yet.

I'd welcome an addition to the 3dlut spec with the changes you're suggesting. Hopefully yesgrey can participate? The problem is that yesgrey is taking a break for personal reasons, so this is a bad time for him to extend the spec. I don't know how much time (if any at all) he will have to participate in the next couple of days/weeks.

[nand chan]

yesgrey was planning to specify the meaning of the color encoding fields for a while. He hasn't had the time to do that yet.

I'd welcome an addition to the 3dlut spec with the changes you're suggesting. Hopefully yesgrey can participate? The problem is that yesgrey is in the process of moving to another country, so this is a bad time for him to extend the spec. I don't know how much time (if any at all) he will have to participate in the next couple of days/weeks.

Yeah, I've contacted him via PM and it seems like he won't have time for a few months at least.

It seems we will have to make do with ourselves for now, it's just a spec after all - not the implementation in yCMS.

In effort to get this moving forwards, I shall propose some suggested changes:

In addition to modifying the current color encoding fields, we will have to add new tags for some things. Should the space in the current header not suffice, the solution will be to increase the offset of the parameters file, which is helpfully encoded in the format itself. As such, all new tags will simply be appended to the end of the last tag, lutUncompressedSize, to maintain backwards compatibility with v1.

Encoding the input and output value ranges

Three ways I have thought of dealing with this:

Encoding the value range as enumeration of possibilities, eg. 0 = (0-255), 1 = (16-235), possibly more in the future (xvYCC?)
Encoding the lower and upper bounds as separate numbers, eg. 0 and 255 - this has the possibility of added flexibility, but it would create a situation where we have to either complicate the rendering chain further or risk creating unsupported LUTs
Encoding the lower and upper bounds as floating point numbers, relative to the maximum for that bit-depth, eg. 0.0627 and 0.922 for 16-235 (as relative to 255). While this may seem like an ideal solution in practice, and required if we are to use 32-bit floating point LUTs, it will also complicate the rendering chain and possibly add inaccuracies.

I'm personally leaning towards either options 1 or 3, the latter because it offers the greatest flexibility, and the former because it's the easiest to work with, especially if you don't want to add the conversion logic and simply want to detect whether a LUT is 16-235 (and reject it otherwise, like madVR does).

Color Encoding field

Given that the current color encoding value of 0 represents BGR, in both value order and enumeration order, we can introduce the value of 1 for YCbCr and the value of 2 for RGB. If the value is 2, the values would simply be flipped around, meaning that the offset would now be (b<<(inputBitDepth[1]+inputBitDepth[0])+g<<(inputBitDepth[0])+r)*3, and so forth.

If this is too contrived or seems unnecessary, it could also be simply limited to 0 and 1.

Color space encoding

This tag will be a strictly optional, with 0 for every value meaning “disabled” or “unknown”. Some things we would have to take care of:

Encoding the red point, green point, blue point and white point of RGB data
Potentially encode the conversion constants for YCbCr output, but this alone will not remove color space ambiguity

A simple method would be to introduce the following fields:

float primaryRedX
float primaryRedY
float primaryGreenX
float primaryGreenY
float primaryBlueX
float primaryBlueY
float primaryWhiteX
float primaryWhiteY

Another possibility would be to encode the values using doubles.

File compression:

We will probably have to make a trade-off here between compression quality and decompression speed, as waiting a few seconds to decompress the LUT when starting playback might be more of a trade-off than it's worth, especially with increasing hard drive sizes.

In effort to measure the gain, I have performed some test compressions on the .3dlut which I use for daily viewing, sorted by size:

Original size: 100 MB
.bz2: 83.59 MB
.gz: 77.66 MB
.zip (normal): 75.84 MB
.rar (fastest): 66.42 MB
.rar (fast): 64.34 MB
.7z (LZMA): 34.61 MB
.rar (normal): 7.35 MB

Gamma

Another thing that might be useful to store is the input/output gamma transfer functions, but I don't know where this would truly be necessary.

XYZ output and PCS

Finally, in effort to get the power of 3dluts closer to something like ICC profiles, we could perhaps allow a PCS such as XYZ as an acceptable output color encoding, that way you could generate two RGB -> PCS LUTs and link them using implementation specific logic to form a single RGB -> RGB LUT.

On the matter of backwards compatibility (for tools and implementations):

To indicate the presence and/or alterations of these fields, the “fileVersion” shall be updated to 2. Should a program supporting version 2 encounter a version 1 LUT, the appropriate backup response shall be to ignore the ColorEncoding fields and draw information from the parameters file as currently. Similarly, if creating a v1 YCbCr LUT, the field shall be kept as “0” to avoid confusing older programs unnecessarily.

Thoughts?

If nobody has any suggestions or further input, then I'll create the first draft of the spec v2 myself and implement it for my toolset.

[nand chan]

If nobody has any suggestions or further input, then I'll create the first draft of the spec v2 myself and implement it for my toolset.

Said and done:

The updated .3dlut spec proposal: OBSOLETE - See the 3DL2 spec below

________________________________________________________________________________

3DLUT file format specification (version 2)
________________________________________________________________________________

enum {PAGE_SIZE = 16384};
struct H3DLUT
{
char signature[4]; // file signature; must be: '3DLT'
long fileVersion; // file format version number (currently 2)
char programName[32]; // name of the program that created the file
long long programVersion; // version number of the program that created the file
long inputBitDepth[3]; // input bit depth per component (Y,Cb,Cr or B,G,R)
long inputColorEncoding; // input color encoding standard (0 = BGR, 1 = YCbCr)
long outputBitDepth; // output bit depth for all components (valid values are 8, 16, 32 and 64)
long outputColorEncoding; // output color encoding standard (0 = BGR, 1 = YCbCr, 2 = XYZ)
long parametersFileOffset; // number of bytes between the beginning of the file and array parametersData
long parametersSize; // size in bytes of the array parametersData
long lutFileOffset; // number of bytes between the beginning of the file and array lutData
long lutCompressionMethod; // type of compression used if any (0 = none, ...)
long lutCompressedSize; // size in bytes of the array lutData inside the file, whether compressed or not
long lutUncompressedSize; // true size in bytes of the array lutData when in memory for usage (outside the file)
long inputValueRange; // value range for the input (0 = Full range (0-255), 1 = Limited (16-235))
long outputValueRange; // value range for the output
// Input color space - this section is strictly optional, a value of 0 for all means “disabled/unknown”
double inputPrimaryRedX; // Though the X and Y are capitalized here for the naming convention,
double inputPrimaryRedY; // they refer to the x and y coordinates on the CIE xy chromaticity diagram,
double inputPrimaryGreenX; // and *not* the XY values of the XYZ color space.
double inputPrimaryGreenY;
double inputPrimaryBlueX;
double inputPrimaryBlueY;
double inputPrimaryWhiteX;
double inputPrimaryWhiteY;
// This header is followed by the char array 'parametersData', of length 'parametersSize',
// and by the array 'lutDataxx', of length 'lutCompressedSize'.
};
char parametersData[1];
union LUTDATA
{
unsigned char lutData8[1];
unsigned short lutData16[1];
float lutData32[1];
double lutData64[1];
};
// The array 'parametersData' starts 'parametersFileOffset' bytes after the beginning of the file.
// The array 'lutDataxx' starts 'lutFileOffset' bytes after the beginning of the file.
// When creating a 3DLUT file, 'lutDataxx' should be positioned on a 16384 byte boundary.
//
// parametersData - char array with size parametersSize that contains an exact copy of the
// input file with the commands and settings used for creating the 3DLUT file
// lutDataxx - array with size lutSizeUncompressed that contains the 3D LUTs output values.
// The type used depends on the 'outputBitDepth' field:
// - unsigned char, if outputBitDepth = 8
// - unsigned short, if outputBitDepth = 16
// - float, if outputBitDepth = 32
// - double, if outputBitDepth = 64
// The value ranges are assumed to be as follows:
// Full range (integers): 0 to (2^depth) - 1, for example 0-255 and 0-65535
// Full range (floats): 0 to 1
// Limited range (integers): 16 to 235, left/right shifted to the correct bit depth, eg. 4096-60160)
// Limited range (floats): (16/255) to (235/255), approx. 0.06275 - 0.92157.
// For XYZ output:
// XYZ must be appropriately normalized so that the luminosity of white = the range limit.
// eg. for limited range 8-bit output, the luminosity of white would be 235. If the value range is full range,
// this would mean that the values are essentially capped to 255. As such, using limited range for XYZ is
// preferable for integer output. It is strongly recommended that one uses full range floats for XYZ however,
// where the luminosity of white would be normalized to 1.0
// The offset inside the array is calculated as:
// offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 // YCbCr input
// offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 // BGR input
// The output order inside the array is:
// Y = lutDataxx(offset); Cb = lutDataxx(offset+1); Cr = lutDataxx(offset+2) // YCbCr output
// B = lutDataxx(offset); G = lutDataxx(offset+1); R = lutDataxx(offset+2) // BGR output
// X = lutDataxx(offset); Y = lutDataxx(offset+1); Z = lutDataxx(offset+2) // XYZ output
// The 'lutUncompressedSize' of the array is calculated as:
// lutDim = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
// lutUncompressedSize = lutDim*outputBitDepth/8
// This specification assumes:
// char = 1 byte; short = 2 byte; float = 4 byte; long = 4 byte; long long = 8 byte; double = 8 byte

Notable changes:

The inputBitDepth field has been redefined to be ordered as B,G,R instead of R,G,B
64 is now an acceptable outputBitDepth, and an appropriate array has been added to LUTDATA
The addition of fields inputValueRange and outputValueRange, and the definition of their ranges in the specification itself
Fields inputPrimaryRedX through inputPrimaryWhiteY define the input color space of a .3dlut
The inputColorEncoding and outputColorEncoding fields have been elaborated on
XYZ is now an acceptable output color encoding, and the implementation is defined
Due to the increased header size, the parametersFileOffset must not fall short of 168, a value of 176 is recommended

Reference implementation:

Preview release of the v0.9 of the TI3Parser toolset which includes full support for .3dlut v2 can be found here: http://www.mediafire.com/?aix71ea5y5kmqw8 (source code included)

This includes a utility to convert v2 .3dluts into v1 .3dluts (by running tag3dlut -1 -i <somefile>.3dlut) so we can use the new spec until madVR supports them.

Full changelog of the version 0.9 up to now:

Version 0.9:
! Created and added support for the .3dlut specification v2, changes include:
+ Added support for 64-bit floating point LUTs
+ Added support for color-space aware LUTs
+ Added support for RGB/YCbCr -> XYZ LUTs for profiling purposes, similar to how ICC device profiles work
+ Added support for file-encoded value ranges
* Utilities such as gen3dlut and changedepth which create new LUTs now use the v2, so you no longer need to fix tagging
using tag3dlut
? Note: Since programs such as madVR still only support v1 .3dluts, you need to “convert” it to a v1 for those programs
+ --one (-1) tag added to tag3dlut in order to degrade the version number to v1, this also overwrites the params in order
to correctly tag everything for madVR etc.
+ --color-space (-c) flag added to inspect3dlut
+ !Input_Primaries(rx, ry, gx, gy, bx, by, wx, wy) tag added to LutScript
+ !Input_Primaries(String) overload added to LutScript, acceptable values are currently: "BT709" and "None"
+ --ycbcr (-y) flag added to gen3dlut to generate a blank YCbCr LUT
* ColorTriple<T> is now aware of its encoding, in order to ensure transformations are properly chained
eg. the grayscale filter now simply sets Cr and Cb to 0 if the input is already YCbCr
~ Significantly increased performance here and there by operating on a single value in memory instead of copying it
over and over, however the old style of doing it is still available for cases where it's needed

[yesgrey]

The updated .3dlut spec:

You are forcing a spec without any of the original authors agreeing with it, so you are at your own risk.

Sorry, but I don't have time for this now. I don't see any reason for all this rush...

[nand chan]

You are forcing a spec without any of the original authors agreeing with it, so you are at your own risk.

Sorry, but I don't have time for this now.

As I've said, it's a first draft.

I don't see any reason for all this rush...

Because I'm having issues working with the existing .3dlut format. If you would prefer I not touch the original spec, then I shall simply create a new file format from scratch for my projects.

[nand chan]

If you would prefer I not touch the original spec, then I shall simply create a new file format from scratch for my projects.

Actually, this seems like a good idea regardless, so I can re-order the existing header without worrying about backwards compatibility.

.3dl2 specification:

________________________________________________________________________________

3DLUT2 file format specification
________________________________________________________________________________

enum {PAGE_SIZE = 16384};
struct H3DLUT2
{
byte signature[4]; // file signature; must be: “3DL2” as encoded using the ASCII standard (0x33444C32)
int fileVersion; // file format version number (currently 1)
byte programName[32]; // name of the program that created the file
long programVersion; // version number of the program that created the file
int inputBitDepth[3]; // input bit depth per component (Y,Cb,Cr or B,G,R)
int inputColorEncoding; // input color encoding standard (0 = BGR, 1 = YCbCr)
int inputValueRange; // value range for the input (0 = Full range (0-255), 1 = Limited (16-235))
int outputBitDepth; // output bit depth for all components (valid values are 8, 16, 32 and 64)
int outputColorEncoding; // output color encoding standard (0 = BGR, 1 = YCbCr, 2 = XYZ)
int outputValueRange; // value range for the output
int parametersFileOffset; // number of bytes between the beginning of the file and array 'parametersData'
int parametersSize; // size in bytes of the array 'parametersData'
int lutFileOffset; // number of bytes between the beginning of the file and array lutData
int lutCompressionMethod; // type of compression used if any (0 = none, 1 = LZO, ...)
int lutCompressedSize; // size in bytes of the array 'lutData' inside the file, whether compressed or not
PRIMARIES inputColorSpace; // Input color space - this section is strictly optional,
// a value of 0 for all means “disabled/unknown”
PRIMARIES outputColorSpace; // Output color space - the same rules apply as for input color space.
// In addition, if the output is XYZ, this section is to be set to 0 for all values

// This header is followed by the byte array 'parametersData', of length 'parametersSize',
// and by the array 'lutDataxx', of length 'lutCompressedSize'.
};
struct PRIMARIES
{
double primaryRedX; // Though the X and Y are capitalized here for the naming convention,
double primaryRedY; // they refer to the x and y coordinates on the CIE xy chromaticity diagram,
double primaryGreenX; // and *not* the XY values of the XYZ color space.
double primaryGreenY;
double primaryBlueX;
double primaryBlueY;
double primaryWhiteX;
double primaryWhiteY;
};
byte parametersData[1];
union LUTDATA
{
byte lutData8[1];
ushort lutData16[1];
float lutData32[1];
double lutData64[1];
};
// The array 'parametersData' starts 'parametersFileOffset' bytes after the beginning of the file.
// The array 'lutDataxx' starts 'lutFileOffset' bytes after the beginning of the file.
// When creating a 3DLUT2 file, 'lutDataxx' should be positioned on a 16384 byte boundary.
//
// parametersData - byte array with size 'parametersSize' that contains an exact copy of the
// input file with the commands and settings used for creating the 3DLUT2 file
//
// lutDataxx - array with size lutSizeUncompressed that contains the 3D LUTs output values.
// The type used depends on the outputBitDepth field:
// - unsigned byte, if outputBitDepth = 8
// - unsigned short, if outputBitDepth = 16
// - float, if outputBitDepth = 32
// - double, if outputBitDepth = 64
//
// The value ranges are assumed to be as follows:
// Full range (integers): 0 to (2∧depth)−1, for example 0–255 and 0–65535
// Full range (floats): 0 to 1
// Limited range (integers): 16–235, left/right shifted to the correct bit depth, eg. 4096–60160)
// Limited range (floats): (16÷255)≈0.06275 to (235÷255)≈0.92157.
//
// For XYZ output:
// XYZ must be appropriately normalized so that the luminosity of white = the range limit.
// eg. for limited range 8-bit output, the luminosity of white would be 235. If the value range is full range,
// this would mean that the values are essentially capped to 255. As such, using limited range for XYZ is
// preferable for integer output. It is strongly recommended that one uses full range floats for XYZ however,
// where the luminosity of white would be normalized to 1.0
//
// The offset inside the array is calculated as:
// offset = (Cr<<(inputBitDepth[1]+inputBitDepth[0])+Cb<<(inputBitDepth[0])+Y)×3 // YCbCr input
// offset = ( R<<(inputBitDepth[1]+inputBitDepth[0])+ G<<(inputBitDepth[0])+B)×3 // BGR input
//
// The output order inside the array is:
// Y = lutDataxx[offset]; Cb = lutDataxx[offset+1]; Cr = lutDataxx[offset+2] // YCbCr output
// B = lutDataxx[offset]; G = lutDataxx[offset+1]; R = lutDataxx[offset+2] // BGR output
// X = lutDataxx[offset]; Y = lutDataxx[offset+1]; Z = lutDataxx[offset+2] // XYZ output
//
// The lutUncompressedSize of the array is calculated as:
// lutDim = 3 × 2∧inputBitDepth[0] × 2∧inputBitDepth[1] × 2∧inputBitDepth[2] × outputBitDepth÷8
//
// This specification assumes:
// byte = 1 byte; short = 2 byte; int = 4 byte; long = 8 byte; // integers
// float = 4 byte; double = 8 byte; // floating point numbers
//
// << is used to denote an arithmetic left shift operator. Where strict/large inequality is wanted, ≪ is used.

I've updated the signature to 3DL2 and decreased the fileVersion back to 1. This also makes code supporting both very easy, you simply check to see if the signature is 3DLT or 3DL2 and is what I do in my code.

Other changes include the addition of an output color space field as well (I'll eventually create a fully color aware pipeline that can link together multiple 3dluts by mapping the gamuts between each step, so I've added this as a prerequisite) and the fixing of some stylistic issues (eg. incorrect ASCII symbols for mathematical operators being used where appropriate Unicode replacements exist)

[madshi]

Original size: 100 MB
.bz2: 83.59 MB
.gz: 77.66 MB
.zip (normal): 75.84 MB
.rar (fastest): 66.42 MB
.rar (fast): 64.34 MB
.7z (LZMA): 34.61 MB
.rar (normal): 7.35 MB
That's all too slow for real time use. I've already experimented with LZO and although it doesn't compress too well, at least it decompresses faster than reading from harddisk. I've already fully working LZO compression projects for 3dluts on my harddisk, but haven't had time yet to polish and publish it all.

[nand chan]

That's all too slow for real time use. I've already experimented with LZO and although it doesn't compress too well, at least it decompresses faster than reading from harddisk. I've already fully working LZO compression projects for 3dluts on my harddisk, but haven't had time yet to polish and publish it all.

I've added LZO compression to v0.10 of mine, using CompressionMethod = 1 to denote LZO.

Some metrics:
16-bit “blank” .3dl2 (input=output): Reduced from ~100 MB to ~50 MB

8-bit “blank” .3dl2: /increased/ from ~49 MB to ~50 MB. LutCompressedSize is larger than LutUncompressedSize.

16-bit gamut mapping .3dl2 created using LittleCMS + ICC profiles: Decreased from ~100 MB to ~70 MB.

Doesn't seem too bad, and decompressing is quite fast as mentioned. I'll leave it in for now.

[Yellow_]

A bit of a weird query but if I wanted a 3D LUT to convert from rec709/sRGB to 'another' with primaries as follows: Red CIE x = 0.73470 CIE y = 0.26530, Green CIE x = 0.00000, CIE y = 1.00000 and Blue CIE x 0.00010 & CIE y -0.07700.

Neutral Axis CIE x = 0.32168, y = 0.33767 Approx CIE D60 and Reference Midpoint Grey at CIE XYZ {0.1715, 0.1800, 0.1816}

What would I need to put in the 3dlut config file?

[madshi]

I've added LZO compression to v0.10 of mine, using CompressionMethod = 1 to denote LZO.

Some metrics:
16-bit “blank” .3dl2 (input=output): Reduced from ~100 MB to ~50 MB

8-bit “blank” .3dl2: /increased/ from ~49 MB to ~50 MB. LutCompressedSize is larger than LutUncompressedSize.

16-bit gamut mapping .3dl2 created using LittleCMS + ICC profiles: Decreased from ~100 MB to ~70 MB.

Doesn't seem too bad, and decompressing is quite fast as mentioned. I'll leave it in for now.
I've found that pre-processing the data before compression helps a lot with LZO compression. E.g. compression the difference instead of the absolute values.

[nand chan]

I've found that pre-processing the data before compression helps a lot with LZO compression. E.g. compression the difference instead of the absolute values.

So we make a LZOM (Lempel-Ziv-Oberhumer-Madshi) variation of it? :P

I'll experiment with what you suggested later. The only immediately jarring problem I see is that you need a signed integer to encode a negative offset, and if you use a signed integer then you can't encode a difference of something like 200. For example, if I want to make a 3dlut that inverts its output, I'll either need two signed integers or an unsigned integer to encode the difference offset.

A bit of a weird query but if I wanted a 3D LUT to convert from rec709/sRGB to 'another' with primaries as follows: Red CIE x = 0.73470 CIE y = 0.26530, Green CIE x = 0.00000, CIE y = 1.00000 and Blue CIE x 0.00010 & CIE y -0.07700.

Neutral Axis CIE x = 0.32168, y = 0.33767 Approx CIE D60 and Reference Midpoint Grey at CIE XYZ {0.1715, 0.1800, 0.1816}

What would I need to put in the 3dlut config file?

Just use yCMS? Something like

Input_Primaries 0.640312221900565 0.331429429181046 0.309178195500916 0.598966865076147 0.146091462964633 0.0570065026117047 0.3127266146811209 0.32902313032606195
Output_Primaries 0.73470 0.26530 0.00000 1.00000 0.00010 -0.07700 0.32168 0.33767

By the way, that is one *HUGE* color space you are talking about. 180% NTSC laser projector or what are you mapping to?

[Yellow_]

By the way, that is one *HUGE* color space you are talking about. 180% NTSC laser projector or what are you mapping to?

hehe, yep and pretty pointless from rec709/sRGB for 90% of uses. :-)

It's ACES IIF a overview here: Need to scroll down a bit to the ACES section.

http://www.fxguide.com/featured/the-art-of-digital-color/

And

http://www.oscars.org/science-technology/council/projects/iif.html

Thanks for the yCMS config.

[nand chan]

hehe, yep and pretty pointless from rec709/sRGB for 90% of uses. :-)

It's ACES IIF a overview here: Need to scroll down a bit to the ACES section.

http://www.fxguide.com/featured/the-art-of-digital-color/

And

http://www.oscars.org/science-technology/council/projects/iif.html

Thanks for the yCMS config.

Oh, thanks a lot for pointing me to this.

So it's basically like the PRMG except less shitty? It would be a good reference gamut for perceptual conversions in ICCv2 profiles (too bad the ICCv4 is already standardized to the PRMG).

(Then again, for perceptual conversions, it doesn't make a difference).

I like the idea of using the ACES as an *essentially* device independent gamut, we could use RGB as a PCS without needing to modify existing RGB gamut correction technologies (eg. yCMS).

That way you could make a 3dlut from BT.709 to ACES, and a 3dlut from ACES to any device, and link the two together using technologies such as merge3dlut.exe.

[Yellow_]

I'm pleased it's useful to you, most of what you've said is over my head, not really got into ICC stuff. :-)

On the subject of gamut's and conversions though, could I point you to this too in case it is of use, was meaning to ask here if anyone is looking at a plugin for Avisynth using it.

http://opencolorio.org/

http://code.google.com/p/opencolorio/

Although it's by Sony, it's open standard / opensource and being adopted pretty much everywhere, including VFX apps like the Foundary's Nuke Compositor and OSS tools like Blender.

OpenColorIO is also in the process of supporting ACES.

[nand chan]

I'm pleased it's useful to you, most of what you've said is over my head, not really got into ICC stuff. :-)

On the subject of gamut's and conversions though, could I point you to this too in case it is of use, was meaning to ask here if anyone is looking at a plugin for Avisynth using it.

http://opencolorio.org/

http://code.google.com/p/opencolorio/

Although it's by Sony, it's open standard / opensource and being adopted pretty much everywhere, including VFX apps like the Foundary's Nuke Compositor and OSS tools like Blender.

OpenColorIO is also in the process of supporting ACES.

Looks interesting. I wonder how much it differs from LittleCMS as far as gamut mapping goes. I may have to provide a LutScript function for this :P

[nand chan]

Updating my 3DL2 spec, it seems like we will have to support dynamic ranges after all. It's good that I split it off since now I can experiment and change the spec as the different needs and wants come in, and adapt my own implementation - until the changes are ready to be pushed back upstream.

For the sake of future flexibility, I am going to encode the black and white levels as 64 bit floating point values on the scale 0-1.

So, for example, a limited range 8-bit LUT (value ranges 16-235) would be (16/255) and (235/255) respectively, or approx. 0.0627451 - 0.921561.

Edit: Having done some testing, it is revealed to me that this method is not quite perfect, since that way you can't simply shift the bit depth without accounting for the value range, eg. (16/255) * 65535 is not, as correct, 4096, but in fact 4112. I'm still trying to come up with an elegant solution for this problem.

Edit 2: The solution dawned on me just as I type this. The value will not be represented as a fraction of the maximum value, but that of /one above it/ (eg. the range limit).

So, a 16-235 value range would be encoded as 16/256 and 235/256, or about 0.0625 and 0.91796875 respectively. This way, the scaling is preserved - 0.0625 * 65536 = 4096 and 0.91796875 * 65536 = 60160.

Some minor rewriting of my pullup/down engine will be required but very doable.

Floating point values will still be kept as their respective exact values, eg. a 32-bit .3dl2 with the output white level set to 0.2 will have a white point of exactly 0.8.

[IanB]

... For the sake of future flexibility, I am going to encode the black and white levels as 64 bit floating point values on the scale 0-1.
...
So, a 16-235 value range would be encoded as 16/256 and 235/256, or about 0.0625 and 0.91796875 respectively. This way, the scaling is preserved - 0.0625 * 65536 = 4096 and 0.91796875 * 65536 = 60160.
You may be better off using rational pairs (like FPS is done with numerator and denominator) to store this concept. This lets you store the exact values and do any required rounding at the time you implement them some time later. Once rounding is done it cannot be undone.


Edit 2: The solution dawned on me just as I type this. The value will not be represented as a fraction of the maximum value, but that of /one above it/ (eg. the range limit).This might be better expressed as the number of values in the range, i.e. [0..255] has 256 values, [16..235] has 220 values.

[nand chan]

You may be better off using rational pairs (like FPS is done with numerator and denominator) to store this concept. This lets you store the exact values and do any required rounding at the time you implement them some time later. Once rounding is done it cannot be undone.

This might be a good idea, the only issue I see here is an implementation issue: If you have some arbitrary value limit, how would you compute the best matching fraction (unless you allow floating point fractions as well)?

For example, say I have some 32-bit LUT with value limits of, I dunno, 0.19439583945 and 0.938593485 - you'd have to generate some sort of integer fraction pair for those and the process for that might as well introduce more rounding inaccuracies than the process of storing a floating point value.

Also, you have to realize that technically, floating points are themselves just a collection of fractions, eg. 1/4 + 1/8 + 1/16 = 0.4375.

And with 64-bit precision the difference between storing the value and computing the value at runtime will be negligible.

This might be better expressed as the number of values in the range, i.e. [0..255] has 256 values, [16..235] has 220 values.

I don't see how that would get us anywhere, elaborate?

[IanB]

This might be a good idea, the only issue I see here is an implementation issue: If you have some arbitrary value limit, how would you compute the best matching fraction (unless you allow floating point fractions as well)?

For example, say I have some 32-bit LUT with value limits of, I dunno, 0.19439583945 and 0.938593485 - you'd have to generate some sort of integer fraction pair for those and the process for that might as well introduce more rounding inaccuracies than the process of storing a floating point value.

Also, you have to realise that technically, floating points are themselves just a collection of fractions, eg. 1/4 + 1/8 + 1/16 = 0.4375.

And with 64-bit precision the difference between storing the value and computing the value at runtime will be negligible.
Yes, floating point numbers are built from fractions of the form 1/2^n so only fractional values that have a last decimal digit of 5 after the point can be exactly represented, and then only within the available precision of the mantissa. But yes it is still a very good representation of the value.

Using rational pair to store a value is a way of never actually doing the divisions in the calculation of the value, thus never having a rounding issue.

If you are always just plonking pre-calculated decimal values into the LUT header then you gain nothing and you might as well use a double.

However if the values are calculations resulting in a normalised (a+b/c) result then rational pairs have a lot to offer.


For your 2 example number with simple normalising and also approximated with continued fractions limited to your last digit :-

0.19439583945 = (3887916789/20000000000) ~ (231749/1192150) = 0.19439583944973

0.938593485 = (187718697/200000000) ~ (106811/113799) = 0.938593485004

Of course I don't know your original calculation, I am working with what appears as an already rounded double result. The true result maybe a much simpler rational pair.

Some rational pairs for PI are 22/7, 333/106, 355/113


This might be better expressed as the number of values in the range, i.e. [0..255] has 256 values, [16..235] has 220 values.
I don't see how that would get us anywhere, elaborate?For the original idea you were expressing, the justification to use 256 and 220 as the divisors in your specification were that they are the count of unique values in the range not the range limit, e.g. which would be 236 for the [16.235] subset.

[nand chan]

Yes, floating point numbers are built from fractions of the form 1/2^n so only fractional values that have a last decimal digit of 5 after the point can be exactly represented, and then only within the available precision of the mantissa. But yes it is still a very good representation of the value.

Using rational pair to store a value is a way of never actually doing the divisions in the calculation of the value, thus never having a rounding issue.

Negligible since the calculations of the value will be performed millions of times during both generation and usage either way - even if you /could/ store some theoretically perfect fraction it would offer no single usage case.

If you are always just plonking pre-calculated decimal values into the LUT header then you gain nothing and you might as well use a double.

The only situations in which you /wouldn't/ want to put a pre-calculated value into the header is for the Full range situation (0-1), but both 0 and 1 can be accurately represented as a double. Everything else will be implicitly tied to the generation, whether it's during gamut mapping using ICC profiles, or yCMS' 64 bit creation, or even simple I=O mapping (pulldown and pullup get performed regardless to transfer the values from an 8-bit range to a 16-bit range, for example - so precision is lost)

You have to remember also we are dealing with color here and the difference between, say, 0.123456789 and 0.123456788 approximates to a difference in XYZ of <0.0000001, even after gamma pullup and mapping.

However if the values are calculations resulting in a normalised (a+b/c) result then rational pairs have a lot to offer

Such exact calculations are not performed here and are in the realm of mathematical accuracy implementations, not color management.

Even if you could achieve such results after /one/ step of the operation (eg. pulldown/pullup which will indeed get calculated as an a + b*c or (a-b)/c triple, storing them that way would be detrimental to both storage space and calculation speed - the point of .3dluts is to pre-calculate everything already so the resulting program just has to run the values through.

For your 2 example number with simple normalising and also approximated with continued fractions limited to your last digit :-

0.19439583945 = (3887916789/20000000000) ~ (231749/1192150) = 0.19439583944973

0.938593485 = (187718697/200000000) ~ (106811/113799) = 0.938593485004

Of course I don't know your original calculation, I am working with what appears as an already rounded double result. The true result maybe a much simpler rational pair.

Some rational pairs for PI are 22/7, 333/106, 355/113

For the original idea you were expressing, the justification to use 256 and 220 as the divisors in your specification were that they are the count of unique values in the range not the range limit, e.g. which would be 236 for the [16.235] subset.

Yes, I understood that much, it's just that I'm wondering what dividing by the count of values would achieve - the point of dividing by 256 in the first place is so you can express the range limits - if you divide by some factor of that range in the first place then the point is destroyed - we would end up with a situation where every range limit turns out to be 1.0 (or every value from 0-1 plus some number (base/lim)).

Unless you are suggesting that we define the divisor as the amount of unique values within the integer range we are representing (and not the value range), in which case that's just expressing the same thing using different words (and would pose a problem to floating point values).

Also, I'm considering removing the whole variable-range proposal in the first place, since it complicates implementations - a true implementation will now have to perform an addition, a subtraction, a multiplication and a division for each components of each pixel of each frame - four things that would previously all not have been necessary.

The point of .3dluts was to be to pre-calculate values, so I'm wondering why re-calculation should be necessarily - .3dluts should be simple look up tables.

The original idea for the variable range limits was to fix a problem that never existed (and indeed, introducing variable ranges didn't fix the problem either - I had forgotten that the range gets normalized within the pipeline either way).

And lastly, the only situation in which flexible range limits would be of benefit is for adjustable white and black levels - and for those, a normalized, full range floating point .3dlut would be ideal, since you can just express, say, 1.15 easily.

Edit: I've also slightly rewritten my copy of the 3DL2 proposal to remove the distinction between R/G/B or Y/Cb/Cr - calculations are now simply performed as “A, B and C” and are unaware of their own encoding, the encoding type determines which channel means what. Encoding = 0 means “BGR”, so A = Blue, B = Green and C = Red.

[nand chan]

I've removed the silly variable ranges again, here is my current copy:

________________________________________________________________________________

3DL2 file format specification
________________________________________________________________________________

enum {PAGE_SIZE = 16384};
struct H3DLUT2
{
byte signature[4]; // file signature; must be: “3DL2” as encoded using the ASCII standard (0x33444C32)
int fileVersion; // file format version number (currently 2)
byte programName[32]; // name of the program that created the file
long programVersion; // version number of the program that created the file
int inputBitDepth[3]; // input bit depth per component (Y,Cb,Cr or B,G,R)
int inputColorEncoding; // input color encoding standard (0 = BGR, 1 = YCbCr)
int inputValueRange; // value range for the input (0 = Full range (0-255), 1 = Limited (16-235)) [up to v1]
int outputBitDepth; // output bit depth for all components (valid values are 8, 16, 32 and 64)
int outputColorEncoding; // output color encoding standard (0 = BGR, 1 = YCbCr, 2 = XYZ)
int outputValueRange; // value range for the output
int parametersFileOffset; // number of bytes between the beginning of the file and array 'parametersData'
int parametersSize; // size in bytes of the array 'parametersData'
int lutFileOffset; // number of bytes between the beginning of the file and array lutData
int lutCompressionMethod; // type of compression used if any (0 = none, 1 = LZO, ...)
int lutCompressedSize; // size in bytes of the array 'lutData' inside the file, whether compressed or not
PRIMARIES inputColorSpace; // Input color space - this section is strictly optional,
// a value of 0 for all means “disabled/unknown”
PRIMARIES outputColorSpace; // Output color space - the same rules apply as for input color space.
// In addition, if the output is XYZ, this section is to be set to 0 for all values

// This header is followed by the byte array 'parametersData', of length 'parametersSize',
// and by the array 'lutDataxx', of length 'lutCompressedSize'.
};
struct PRIMARIES
{
double primaryRedx; // The x and y refer here to the coordinates on the CIE xy chromaticity diagram,
double primaryRedy; // Y is assumed to be 1.0 for these primaries
double primaryGreenx;
double primaryGreeny;
double primaryBluex;
double primaryBluey;
double primaryWhitex;
double primaryWhitey;
};
byte parametersData[1];
union LUTDATA
{
byte lutData8[1];
ushort lutData16[1];
float lutData32[1];
double lutData64[1];
};
// The array 'parametersData' starts 'parametersFileOffset' bytes after the beginning of the file.
// The array 'lutDataxx' starts 'lutFileOffset' bytes after the beginning of the file.
// When creating a 3DLUT2 file, 'lutDataxx' should be positioned on a 16384 byte boundary.
//
// parametersData - byte array with size 'parametersSize' that contains an exact copy of the
// input file with the commands and settings used for creating the 3DLUT2 file
//
// lutDataxx - array with size lutSizeUncompressed that contains the 3D LUTs output values.
// The type used depends on the outputBitDepth field:
// - unsigned byte, if outputBitDepth = 8
// - unsigned short, if outputBitDepth = 16
// - float, if outputBitDepth = 32
// - double, if outputBitDepth = 64
//
// The value ranges are assumed to be as follows:
// Full range (integers): 0 to (2∧depth)−1, for example 0–255 and 0–65535
// Full range (floats): 0 to 1
// Limited range (integers): 16–235, left/right shifted to the correct bit depth, eg. 4096–60160)
// Limited range (floats): (16÷255)≈0.06275 to (235÷255)≈0.92157.
//
// For XYZ output:
// XYZ must be appropriately normalized so that the luminosity of white = the range limit.
// eg. for limited range 8-bit output, the luminosity of white would be 235. If the value range is full range,
// this would mean that the values are essentially capped to 255. As such, using limited range for XYZ is
// preferable for integer output. It is strongly recommended that one uses full range floats for XYZ however,
// where the luminosity of white would be normalized to 1.0
//
// The offset inside the array is calculated as:
// offset = (A + B << (inputBitDepth[0]) + C << (inputBitDepth[1]+inputBitDepth[0])) × 3
//
// The output order inside the array is:
// A = lutDataxx[offset]; B = lutDataxx[offset+1]; C = lutDataxx[offset+2]
//
// A, B and C refer to the respective channels of the used encoding (eg. B, G, R or Y', Cb, Cr)
//
// The lutUncompressedSize of the array is calculated as:
// lutDim = 3 × 2∧inputBitDepth[0] × 2∧inputBitDepth[1] × 2∧inputBitDepth[2] × outputBitDepth÷8
//
// This specification assumes:
// byte = 1 byte; short = 2 byte; int = 4 byte; long = 8 byte; // integers
// float = 4 byte; double = 8 byte; // floating point numbers
//
// << is used to denote an arithmetic left shift operator. Where strict/large inequality is wanted, ≪ is used.

[leeperry]

I might consider creating a bigger ramdisk and use 16bit YUY2 LUT's instead of 8bit, but why is t3dlut() so much more power hungry than rgb3dlut() again? :o
hi tritical, if you're still around: is there any way you could provide a way to apply 16bit LUT's w/ the same CPU load as rgb3dlut() please? t3dlut() is a CPU hog :/

[leeperry]

BTW, it's funny to see that in 2013 they still use a Sony CRT for TV shows production(and more than likely color correction): http://s.tf1.fr/mmdia/i/33/5/plateau-speciale-10-ans-ci-26-10592335hxsqq.jpg

It's one of those: http://www.highwayav.com/Portals/0/Products/Production_Monitors/BVM_PVM_Monitors/BVM-A14F5M/1169568028-191.jpg

[fijam]

Does anyone still have a copy of ddcc.zip? The original links are down and it wasn't archived by the Wayback machine.

[StainlessS]

ddcc_v17.zip, @ SendSpace below in my sig [~0.1MB].

Source (still there), https://forum.doom9.org/showthread.php?t=168012, posted by Wilbert (bottom of first post, 350MB+).

EDIT: I think there were also at least a couple of earlier versions in the archive zip posted by Wilbert.

EDIT: File linked by Wilbert:- Plugins4_20130531.zip, again here (~350MB):- http://www.avisynth.nl/users/warpenterprises/Plugins4_20130531.zip

Contents of Zip

avsfilmcutter_Source_0020b
ffmpegsource_119
flash3kyuu_deband
QTGMC
tc2mp4_20070124
_FSubstitute.zip
_FSubstitute_121122.zip
_FSubstitute_121202.zip
_FSubstitute_121223.zip
_GPU25_002_test3.rar
3dubois3.zip
AddGrainC-1.5.1.7z
AddGrainC-1.5.3.7z
AddGrainC-1.6.1.7z
AddGrainC-1.7.0.7z
addgrainc_1.4.zip
anisotools-v1.0a1.zip
anisotools-v1.0a2.zip
anisotools-v1.0a3.zip
anisotools-v1.0a4.zip
anisotools-v1.0a5.zip
assrender-0.18.7z
assrender-0.18-src.7z
assrender-0.24.1.7z
assrender-0.24.7z
assrender_0.1.7z
assrender_0.1-src.7z
autolevels0.3.zip
autolevels0.3b.zip
autolevels_0.6_20101226.zip
autolevels_0.6_20110109.zip
autolevels_20101104.zip
autoyuy2_101.zip
Average_v10.zip
Average_v11.zip
avfs_1.0.0.3.zip
avfs_20080602.zip
AVIInfo_v03.zip
AviSynth26.7z
avisynth26_20110913.7z
AviSynth26_old.7z
AVISynth UI 0.01.zip
avisynth_pascal_v5.rar
AviSynth_R3D_Source_V1.0.zip
AvisynthStudio 1.2 Beta (Source).7z
AvisynthStudio 1.2 Beta.7z
AvisynthTrackin.1.0.binary.zip
AvisynthTrackin.1.0.source.zip
AVS2ASF_NIC.zip
avs2avi-140.7z
avs2avi-140a.zip
avs2bdnxml-1.3.tar.bz2
avs2bdnxml-2.04.tar.bz2
avs2pipe260-0.0.2.zip
avs2pipe-0.0.2.zip
avs2pipemod-0.1.2.7z
avs2pipemod-0.3.0.7z
avs2pipemod-20110509.zip
avs2pipemod-20110703-2.zip
avs2pipemod-20110919.zip
avs2wav.2011-05-21.zip
avs2yuv-0.24bm2.zip
avs.coder
AVSCurveFlow-2007.03.06.zip
avsfilmcutter_Source_0020b.rar
AVSInpaint-2008.01.06.zip
AVSInpaint-2008.02.23.zip
AVSMeter100.zip
AVSMeter112.zip
AVSMeter115.zip
AVSMeter144.zip
AVSMeter145.zip
AVSMeter149.zip
avsproxy_gui.2008-02-29.zip
avsproxy_gui.2008-03-01.zip
avss.7z
avss-src.7z
avstp-1.0.0.zip
avstp-1.0.1.zip
avsViewer.zip
aWarpSharp.rar
aWarpSharp_20090619.rar
aWarpSharp_20090619_fix1.rar
BassAudio2423.7z
BeHappy_r13913.7z
BlendBob_1.2.zip
BlindDeRing.dll
BZColorspace_Source.rar
ccc_v0.4a_avs.zip
chikuzen-HighBitDepth_VFWSource-f6729f2.zip
chikuzen-VS_AvsReader-8c6c94a.zip
chroma_optimize v0.2.0 (alpha).zip
CMVSource.zip
ColorBalance_0.26.zip
Colorit.zip
colorkeyframe02.zip
ColorMatrix_ICL11.rar
ColorMatrixv23.zip
ColorMatrixv24.zip
ColorMatrixv25.zip
ColorMatrixv25ICL11.zip
ColorScreenMask.zip
ColorSource_original.rar
ColorYUV Graffer.zip
colourlike_25_dll_20050825.zip
ColourLike_25Aug05.dll
Comparison-187.zip
ddcc.zip
ddcc_v16.zip
ddcc_v17.zip
DDigit_25_dll_20101017.zip
DDigitTest_25&26_v1-04_dll_20130420.zip
Deathray_1_00_DLL.zip
Deathray_1_00_src.zip
decomb524src_pitch_fix.zip
defish.zip
demosaic.zip
demosaic_ignus2.zip
depan1100.zip
depan1101.zip
depanestimate192.zip
despot351.zip
dfttest_281107.zip
dfttestv13.zip
dfttestv14.zip
dfttestv15.zip
dfttestv18.zip
dgavcdec100a8.zip
dgavcdec100a13.zip
dgavcdec101.zip
DGDecode158_Mod.zip
DGIndex158b.zip
dgmpgdec154.zip
dgmpgdec154src.zip
dgmpgdec156.zip
dgmpgdec157.zip
dgmpgdec158.zip
dgmpgdec158src.zip
DirectShowSource_2587.zip
Displace v0.1.rar
Dither16to8Bit.zip
dither-1.4.zip
dither-1.5.zip
dither-1.8.zip
dither-1.9.5.zip
dither-1.12.1.zip
dlAviShader042.rar
Dsynth based on 2.55 release.v9.rar
dvinfo_20100602.zip
EEDI2MT___(v0.9.2_-_2007-02-23).7z
eedi3.zip
EffectsMany.zip
exinpaint01.zip
exinpaint02.zip
FanFilter_19102008.zip
FanFilter_22102008.zip
fauxD_v010.zip
fdecimate110.zip
ffmpeg_eac3_mlp_rev10905.7z
ffmpeg_eac3_mlp_rev10905.patch
FFmpegSource-1.13.rar
FFmpegSource-1.14.rar
FFmpegSource-1.19.rar
FFmpegSource-2.00b1.rar
FFmpegSource-2.00b3.rar
ffmpegsource-2.15.7z
FFmpegSource_+++.7z
FFMpegSource_audio_test_281007.rar
FFmpegSource_r10891.7z
FFmpegSource_rev10905.7z
ffms2-2[1].13.7z
ffms2-2[1].13_src.7z
ffms2-r339.7z
ffms-2.16-src.tar.bz2
ffms-2.17.7z
FilmCutterVer0020beta.zip
flash3kyuu_deband_1.0.1.7z
FluaG_v01.zip
FluaG_v02.zip
FluaG_v04.zip
FluaG_v05.zip
FluxSmooth-1.1b.zip
framenumber_version_0.2c.zip
framenumber_version_0.4b.zip
fssetup-2[1].7.exe
FSubstitute.txt
fusion2.zip
fusion.zip
GetProcessName_dll_20121229(Take-2).zip
GetSystemEnv_v020.zip
GetSystemEnv_v030.zip
GPUBilateralFilter.Binary.Win32.1.0.zip
GPUBilateralFilter.Binary.Win32.1.2.zip
GPUBilateralFilter.Source1.0.zip
GPUBilateralFilter.Source.1.2.zip
GrainOptimizer_v12.zip
GrainOptimizer_v101.zip
GraMaMa_v02.zip
greyc01.zip
GRunT101.zip
GRunT.zip
GScript_10.zip
GScript_11.zip
HighBitDepth_VFWSource-0.1.3.7z
HSVAdjust_v01.zip
imagesequence_20080227.zip
imck-2.0.0.zip
imck-2.0.0-src.zip
InterFrame-1.0.zip
iuf_v1.5.zip
javisynth_0_2.tar.gz
libass-static-mingw32.tar.bz2
LoadDll.zip
loadfont.cpp
makeavis1603.zip
Manamanator_Installer.zip
Manamanator_Source.zip
MaskHS_src.zip
masktools-v2.0a33.zip
masktools-v2.0a34.zip
masktools-v2.0a44.zip
masktools-v2.0a46.zip
masktools-v2.0a47.zip
masktools-v2.0a48.zip
mcbobbers.zip
MCBobUv5.7z
McM_2D_to_3D_v0.2.rar
MDec2_source.zip
MinMaxAudio_v02.zip
mpasource_20080220.zip
MpegAutoIndex.zip
MSU Frame Rate Conversion Filter.zip
MT_07_with_Avisynth258.rar
mt_masktools-26-for-2.6alpha4.7z
mvtools-v1.8.6.zip
mvtools-v1.9.2.zip
mvtools-v1.9.3.zip
mvtools-v1.9.5.zip
mvtools-v2.5.11.3.zip
mxfsource_v01.zip
netAvs2yuv-0.3.1.tar.bz2
NicAudio_204.7z
NicAudio_205.7z
NicAudio_206.7z
NicAudio_r202.7z
NLMeansCL_0.1.1.zip
NLMeansCL_0.3.1.zip
NLMeansCL_0.3.1_source.zip
NLMeansCL_0.3.2.zip
NLMeansCL_0.3.2_source.zip
nnedi2.zip
nnedi2_v10.zip
nnedi2_v13.zip
nnedi2_v15.zip
nnedi3_v09.zip
nnedi_20100820.tar.bz2
overlua-0.7-alpha.rar
Pantarheon3DAviSynthToolbox-1-1.zip
pixelinfo0_3.zip
PointSize_0.1.7z
pop.zip
PremiereAVSPlugin-v1.91.exe
PremiereAVSPlugin-v1.91-src.zip
PremiereAVSPlugin-v1.95.exe
PremiereAVSPlugin-v1.95-src.zip
QTGMC2.47r.zip
QTGMC-3.32.zip
QTGMC 32-bit Plugins.zip
QTSource_0_0_8_bin.zip
QTSource_20100915_Test01.zip
quad_20121101.zip
quad-no-sse.zip
R_pack_23.zip
R_pack_020708.zip
R_pack_191107.zip
RaWav10b.7z
RawSource_25_dll_20110521.zip
RawSource_25_dll_20110523.zip
RawSource_25_dll_20110529.zip
RawSource_26_dll_20110614.zip
rawsource_26_dll_20110925.zip
rawsource_26_dll_20120831.zip.zip
rawsourceSeq_25_dll_20060412.7z
RedAverage___(1.4.3_-_2011-12-02).7z
RemoveGrain_v09.zip
ResampleHQ-v1.zip
ResampleHQ-v2.zip
rotate11.zip
rotate131.zip
RT_Stats_25_dll_0.0b_20120724.zip
RT_Stats_25_dll_1.05_20121003.zip
RT_Stats_25_dll_1.06_20121205.zip
RT_Stats_25_dll_1.07Beta_20121211.zip
Sample.avs
Sample.d2v
Sample.m2v
Sample_Wilbert.zip
sashimi_073.zip
sashimi_073_src.zip
sashimi_074.zip
sashimi_074_src.zip
sashimi_075.zip
sashimi_075_src.zip
sashimi_085.zip
Sashimi_085_src.zip
showpixelvalues1.4c.zip
SimpleResize_MOD.zip
SincResize4.7z
SmoothAdjust-v1.00.zip
SmoothAdjust-v1.16.zip
SmoothAdjust-v1.40.zip
SmoothAdjust-v2.00alpha.zip
SmoothAdjust-v2.00beta3.zip
SmoothAdjust-v2.00beta4.zip
SmoothD008.zip
SmoothD008_src.zip
sora_mtmp_package_20120211.7z
SoundOut-1.0.3.zip
SoundOut-1.1.1.zip
specials-src_davidhorman.zip
SSIM0.25.1.0.rar
SUPtext.zip
tc2mp4_doom9dotorg_20070124.rar
tcanny.zip
tcv2tov1_01.zip
TDeintv11.zip
ThreadRequest102a.zip
TIVTCv103.zip
TurnsTile_v010.zip
TurnsTile_v020.zip
TurnsTile_v030.zip
TurnsTile_v031.zip
TWriteAvi.zip
twriteavi_squid_80_mod.zip
unblock11.zip
Unfurl.zip
variableblur_20110314.zip
VariableBlur_30082008.zip
variableblur_v06.zip
VerticalCleaner.rar
ViDBuG.Beta3.msi
ViDBuG.Beta3.Source.7z
vsavsreader-4dafb1b.7z
vsfilter_250508.rar
waveform0.1.zip
waveform0.2.zip
WMCmd_20071011.zip
x264Resize.7z
yadif13.zip
yadif17.zip
yadifmod_v1.zip
yatta.txt
yatta_7-130-beta6.rar
yv12torgb24hq.zip


File list, extracted from Windows Explorer using NirSoft SysExporter.
Perhaps posting full contents might assist in future forum search.

EDIT: The zip posted by Wilbert seems to be a copy of WarpEnterprises plugs, but WarpEnterprises does not seem to
include the ddcc zip, perhaps there are other differences, anyways here is WarpEnterprises on the Avisynth.nl:- http://www.avisynth.nl/users/warpenterprises/