gamut conversions through Avisynth ?

[tritical]

I'm wondering: As far as I understand yv12 is 4:2:0 and yuy2 is 4:2:2, right? Is there also an "official" fourcc for 4:4:4 YCbCr? And wouldn't it be faster and better to convert 4:2:0 directly to 4:4:4 instead of doing 4:2:0 -> 4:2:2 -> 4:4:4?
4:2:0 -> 4:4:4 is essentially 1 vertical resize and 1 horizontal resize of the chroma planes. Since we are talking about separable operations, it is usually faster to do them separately (except for nearest neighbor, linear is close, cubic and above is definitely faster). The question is whether you should store the result of each step with greater than 8-bit accuracy. ffdshow's new algorithm uses two point linear interpolation in each step (75/25 weighted averaging). Thus, to get the correct value you need 4 extra bits (2 extra for the first step result, and another 2 for the second). If you truncate to 8-bit after each step the largest difference you can incur (on 8-bit scale) is 0.5 after the first step and 1.0 after the second step. Therefore, once you get to 4:4:4 the u/v values could have a difference of up to 1.0. I have argued this is unnoticeable, and so far everyones comments support this... if you can't tell the difference between different chroma placements or linear vs cubic interpolation, both of which create much larger differences, then there is no way you can tell the difference here. For reference, avisynth's resizers separate horizontal/vertical steps, and store the intermediate result (if both width and height are altered) to 8-bit. They operate on luma as well as chroma.

Using higher intermediate bitdepths would yield more of a difference as the quantization error of a larger number pixels comes into play (like convolution with 7x7 kernel after resizing or chaining a bunch of single pixel operations together).

Also, I don't believe there is an "official" fourCC for 4:4:4. Avisynth 2.6 uses 'YV24' to identify its planar 4:4:4 yuv colorspace.

We could define it later, but how should we store <16bit values in the 'short'? MSB or LSB?
I say just cast it to unsigned 16-bit (so satured conversion to [0-65535] range), and store as little endian. My program can then just operate on it as though it was a 16 bit value... all that needs to change is how much to shift by at the end.

Anyway, is there code available anywhere (or at least a formula) on how to "interpolate inside the cube"?
If we make the assumption of linearity between points defined by the lut, then it is simply trilinear interpolation involving 8 points. I think anything else would be a pain.

[yesgrey]

For reference, avisynth's resizers separate horizontal/vertical steps, and store the intermediate result (if both width and height are altered) to 8-bit. They operate on luma as well as chroma.
Recently there was a user suggesting in ffdshow's thread to perform the resize and 4:2:0 -> 4:4:4 all in one step... that could be the best of all.;)

Also, I don't believe there is an "official" fourCC for 4:4:4. Avisynth 2.6 uses 'YV24' to identify its planar 4:4:4 yuv colorspace.
I think you are right. If we google yv24 there is little info about it... only Avisynth2.6 talks about it...

I say just cast it to unsigned 16-bit (so satured conversion to [0-65535] range), and store as little endian. My program can then just operate on it as though it was a 16 bit value... all that needs to change is how much to shift by at the end.
Let's look at an example to see if I understood it...
For 10bit output, the values will be stored in bits 6-15, and bits 0-5 will be set at 0, right? Then you just need to >>6 to get the 10bit value, right?
If it's right, there are two ways of doing it:
-When creating the 3D LUT I convert the float to 10bit and <<6
-When creating the 3D LUT I convert the float to 16bit, and then your application will take care of any rounding to 10bit
The second is the same of only creating 8bit or 16bit output 3D LUTs, but maybe the first would be a little more faster, because you will only need to shift and ignore the rounding.

[tritical]

Let's look at an example to see if I understood it...
For 10bit output, the values will be stored in bits 6-15, and bits 0-5 will be set at 0, right? Then you just need to >>6 to get the 10bit value, right?
I was thinking that 10 bit values in a lut would be stored in bits 0-9 of each 2 byte unsigned short.. so instead of the full [0-65535] range being used only [0-1023] would be used. That way, all calculations in rgb3dlut can be performed as if the values were full range, but when it comes time to calculate the final output values the right shift amount would be, assuming 8-bit output, 2 instead of 8, and the addition for rounding would be 4 instead of 128.

Using that method, it makes no sense to create a lut with 2 byte entries with < 16-bit values stored in them... unless all of the mapped values can be precisely represented in < 16 bit. Also, I think that there should be only one value allowed, i.e. the same for all planes, for outputBitDepth (talking about bit depth of values stored in the lut).

It's confusing talking about outputBitDepth with respect to values stored in the lut because the bit depth of the values output by a program using the lut (like rgb3dlut), and the bit depth of the values stored in the lut, are two different things. Maybe in the format description they could be called 'indexBitDepth' and 'storedBitDepth' instead of 'inputBitDepth' and 'outputBitDepth'. Again, I can see the usefullness in allowing different indexBitDepths for each plane, but not for different storedBitDepths.

[madshi]

4:2:0 -> 4:4:4 is essentially 1 vertical resize and 1 horizontal resize of the chroma planes. Since we are talking about separable operations, it is usually faster to do them separately (except for nearest neighbor, linear is close, cubic and above is definitely faster).
I didn't know that doing it separately would be faster... :eek:

Also, I don't believe there is an "official" fourCC for 4:4:4.
Strange.

If we make the assumption of linearity between points defined by the lut, then it is simply trilinear interpolation involving 8 points. I think anything else would be a pain.
Ok, thanks.

I was thinking that 10 bit values in a lut would be stored in bits 0-9 of each 2 byte unsigned short.. so instead of the full [0-65535] range being used only [0-1023] would be used. That way, all calculations in rgb3dlut can be performed as if the values were full range, but when it comes time to calculate the final output values the right shift amount would be, assuming 8-bit output, 2 instead of 8, and the addition for rounding would be 4 instead of 128.

Using that method, it makes no sense to create a lut with 2 byte entries with < 16-bit values stored in them...
So in other words only 8bit and 16bit should be supported and no intermediate bitdepths? Would be fine with me, since rounding from 16bit to e.g. 10bit would be *really* quick and easy, anyway...

Also, I think that there should be only one value allowed, i.e. the same for all planes, for outputBitDepth (talking about bit depth of values stored in the lut). [...] Again, I can see the usefullness in allowing different indexBitDepths for each plane, but not for different storedBitDepths.
Hmmmm... I have to agree.

It's confusing talking about outputBitDepth with respect to values stored in the lut because the bit depth of the values output by a program using the lut (like rgb3dlut), and the bit depth of the values stored in the lut, are two different things. Maybe in the format description they could be called 'indexBitDepth' and 'storedBitDepth' instead of 'inputBitDepth' and 'outputBitDepth'.
Well, using the 3dlut array is a simple process. You feed data in and you get data out. Of course the program using the lut can do additional processing on the array "output". But for me using the words "input" and "output" is intuitive, while using "indexBitDepth" and "storedBitDepth" (while making sense) is more difficult to understand. I'm thinking of the 3dlut array as one conversion step in a chain of conversions. After all before the 3dlut array you have to convert e.g. yv12 to yv24. And afterwards you may have to dither or round 16bit RGB down to 8bit. But still the 3dlut process itself for me is one separate conversion step which has an input and output. DirectShow filters also all have an input and output. And my own eac3to tool internally also has input and output for every conversion step.

[yesgrey]

Using that method, it makes no sense to create a lut with 2 byte entries with < 16-bit values stored in them... unless all of the mapped values can be precisely represented in < 16 bit.
Well, but that's the idea... I don't believe that if Windows7 brings 16 bit, the only options would be 8 bit or 16 bit, unless the drivers developers only want that...
Currently some displays only support 10 bit or 12 bit connection at most, and for those would be good a format like that, and with a LUT with the same bit depth there is no need for roundings inside rgb3dlut... But since this is just the future, and we don't know how will it be, we could simply stick with 8 bit and 16 bit modes. The proposed specification will support other bit depths between those, we could leave the option of "how" to a later stage, if it will ever be needed...

Also, I think that there should be only one value allowed, i.e. the same for all planes, for outputBitDepth ... Again, I can see the usefullness in allowing different indexBitDepths for each plane, but not for different storedBitDepths.
If we think only in using the 3D lut one time in a chain, I also can't see the usefullness in allowing different outputBitDepths, but if we think in using the 3D LUT two times in a chain, maybe it could be useful.
For example:
a YCbCr->YCbCr->RGB chain, with indexing YCbCr(10|8|8), it would be useful to have a stored YCbCr(10|8|8) to use when indexing into the 2nd 3D LUT.
Would this kind of usage make any sense? Maybe not, and we could always create a stored YCbCr(16|16|16) 3D LUT and reduce the bith depth for YCbCr(10|8|8) when indexing into the 2nd stage in the chain...
To be honest, I would be happy only with YCbCr->RGB with 8bit, 16bit and 16bit dithered to 8bit output. All the other options are useless to me. I'm only proposing them in the case someone else might find them useful...:)

It's confusing talking about outputBitDepth with respect to values stored in the lut because the bit depth of the values output by a program using the lut (like rgb3dlut), and the bit depth of the values stored in the lut, are two different things. Maybe in the format description they could be called 'indexBitDepth' and 'storedBitDepth' instead of 'inputBitDepth' and 'outputBitDepth'.
But for me using the words "input" and "output" is intuitive, while using "indexBitDepth" and "storedBitDepth" (while making sense) is more difficult to understand. I'm thinking of the 3dlut array as one conversion step in a chain of conversions.
I am ok with any of the namings. The index/stored is more accurate, but the input/output is a little more intuitive.
If I was the only person to use the specification, I would prefer the former, but maybe the later is more intuitive for other possible users...

[madshi]

To be honest, I would be happy only with YCbCr->RGB with 8bit, 16bit and 16bit dithered to 8bit output.
So it seems that none of us (tcritical, you and me) really wants different output bitdepths per plane, nor any other output bitdepth other than 8bit and 16bit. So if we 3 agree on that, then let's keep things simple and do it that way.

Still I'd keep using an "int" for output bitdepth instead of a "bool" byte/word switch, just to keep the door open for intermediate bitdepths at a later day...

[yesgrey]

So if we 3 agree on that, then let's keep things simple and do it that way.

Still I'd keep using an "int" for output bitdepth instead of a "bool" byte/word switch, just to keep the door open for intermediate bitdepths at a later day...
Simple, I agree with it.
No "bool", I also agree with it. I don't like "bools", I only use them if not avoidable (when using functions not mine that use them).

So it seems we just need to agree with the index/stored vs input/output naming...:)

[yesgrey]

After some discussion between me and madshi via PM, here is the fifth iteration for the definition of a file format for the 3D LUT:
// 3D LUT file specification (fileVersion=1):
typedef struct
{
char signature[5];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[20];
long int programVersion;
int parametersSize;
} h3dlut;
struct
{
h3dlut header;
char parametersData[1];
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
} f3dlut;
/*
// Header
signature - File signature; must be: '3DLUT'
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or R,G,B)
outputBitDepth - Output bit depth per component (The same for all components)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array parametersData
// Parameters data
parametersData - array of char with size parametersSize that contains a copy of the
run parameters settings used for creating the file
// 3D LUT output data
lut.dataxx - array with size lutSize that contains the 3dlut output values.
The type used (lut.dat8 or lut.data16) is defined by the
outputBitDepth field:
- lut.data8 when outputBitDepth = 8
- lut.data16 when outputBitDepth = 16
The offset and size of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
lutSize = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(Size in bytes:
- lutSize*1, when outputBitDepth = 8
- lutSize*2, when outputBitDepth = 16)
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/

I have not decided yet how to indicate the ColorEncoding values, I will update the specification later. I need a little more time to think about it.

tritical, do you agrre with it? It seems we only need to agree with the BitDepths naming...;)

Comments/sugestions are welcome.

[tetsuo55]

I'm not sure what you guys mean with only output 8 or 16 bit.

I would do everything with 16bit, and then allow dithering down to 8/10/12

[madshi]

I would do everything with 16bit, and then allow dithering down to 8/10/12
We're only talking about the 3dlut file format here. A software could do just what you suggested.

------------------

@yesgrey3 and @tritical: We still need a default file extension. How about *.3dl"? Or should be use "*.3dlut"? Or maybe "*.lut"? Hmmmmm... I think my favorite would be "*.3dlut". Your opinion?

[tetsuo55]

Looks like the first hdmi 1.3 10bit videocard is out:

http://www.s3graphics.com/en/products/desktop/chrome_540gtx/index.jsp

I have no seen any hard confirmation, purely based on its specs

[tritical]

It seems we only need to agree with the BitDepths naming...
Leaving the names as they are is fine with me.

So in other words only 8bit and 16bit should be supported and no intermediate bitdepths? Would be fine with me, since rounding from 16bit to e.g. 10bit would be *really* quick and easy, anyway...
To be honest, I would be happy only with YCbCr->RGB with 8bit, 16bit and 16bit dithered to 8bit output. All the other options are useless to me. I'm only proposing them in the case someone else might find them useful...
Well, but that's the idea... I don't believe that if Windows7 brings 16 bit, the only options would be 8 bit or 16 bit, unless the drivers developers only want that...
Currently some displays only support 10 bit or 12 bit connection at most, and for those would be good a format like that, and with a LUT with the same bit depth there is no need for roundings inside rgb3dlut... But since this is just the future, and we don't know how will it be, we could simply stick with 8 bit and 16 bit modes. The proposed specification will support other bit depths between those, we could leave the option of "how" to a later stage, if it will ever be needed...
I agree that only 8-bit and 16-bit should be supported. From my point of view, the lut should always use the full range it can (8-bit for byte output, 16-bit for 2 byte output), as that gives the dithering algorithm in the program using the lut the most information to work with if it is outputting at a lower bitdepth. If that program doesn't use dithering, then its conversion down to a different bitdepth should end up giving the same value that the program creating the lut would have given (technically it could be off by 1 at the lower bitdepth, but only for values very close to x.5), and the speed cost of its having to do that conversion is negligible.

@yesgrey3 and @tritical: We still need a default file extension. How about *.3dl"? Or should be use "*.3dlut"? Or maybe "*.lut"? Hmmmmm... I think my favorite would be "*.3dlut". Your opinion?
I like .3dlut as well.

[yesgrey]

@yesgrey3 and @tritical: We still need a default file extension. How about *.3dl"? Or should be use "*.3dlut"? Or maybe "*.lut"? Hmmmmm... I think my favorite would be "*.3dlut". Your opinion?
I prefer .3dlut too.
I agree that only 8-bit and 16-bit should be supported.
So, it seems we have our file format ready.:)
I will change cr3dlut to output this file format and do the ColorEncoding description, to complete the specification.

[tritical]

So, it seems we have our file format ready.
I will change cr3dlut to output this file format and do the ColorEncoding description, to complete the specification.
Great. I've completed all of the things I mentioned earlier, and have dithering from 16 to 8 added. I went with Floyd-Steinberg due to speed. Other fast diffusion methods are: burkes, sierra2, and filter lite... but they are all very similar. Just need to change it to read the new lut format, and do some testing. Should have time tommorrow.

[cyberbeing]

tritical, for videos which are not mod4, could you add the ability to automatically fallback to ConvetToYUY2() if desired, instead of throwing an error? (e.g. Fallback=True or Fallback=False with the default being False).

[jmartinr]

I agree that only 8-bit and 16-bit should be supported.

In some avisynth 3.0 documents there is talk about YV45: YV24, a 4:4:4 YUV planar color format, and RGB45 and YV45 which are 15 bits depth versions of RGB24 and YV24 (NB: even though RGB45 is 8 byte aligned, the extra is not alpha, just garbage).I wouldn't know myself, but maybe there's a good reason for using a 15 bits depth version?

[yesgrey]

Great. I've completed all of the things I mentioned earlier, and have dithering from 16 to 8 added.
I have to speed up to catch you and release cr3dlut at the same time...;)
In some avisynth 3.0 documents there is talk about YV45: I wouldn't know myself, but maybe there's a good reason for using a 15 bits depth version?
I think we should not start adding things that we don't know yet how they would end...
Currently, we are stuck with RGB32 and 8bit per component. Yes, we could be using RGB32 with 10bit per component and 2bit alpha, but we don't have it. DirectShow had considered it, but it seems it never appeared. For example, RGB24 was dropped by all graphic cards drivers developers in favor of RGB32.
It seems with Windows7 we will get 16bit per component. Let's see what else we will get... because in the end, the graphics card drivers define what we could use...

[leeperry]

tritical, for videos which are not mod4, could you add the ability to automatically fallback to ConvetToYUY2() if desired, instead of throwing an error? (e.g. Fallback=True or Fallback=False with the default being False).
or imode=0 which used to work fine w/ non-mod4

the real question is : will it show any visible improvement over ConvertToYUY2() as this is much more optimized and faster...

[madshi]

Great. I've completed all of the things I mentioned earlier, and have dithering from 16 to 8 added. I went with Floyd-Steinberg due to speed. Other fast diffusion methods are: burkes, sierra2, and filter lite... but they are all very similar. Just need to change it to read the new lut format, and do some testing. Should have time tommorrow.
Awesome! Have you had a chance to check whether dithering makes a visible difference or not? Thanks!

[IanB]

Seeing this is a file header, I would suggest you explicitly align all the elements independent of any compiler.

And seeing the lut is some 48+ megabytes, catering for accessing it via MMAPing technology in the future might be prudent, so align the lut data now to the largest architecture page size likely, i.e 8192 or maybe 16384. Also pad the entire file length to this page size.
// 3D LUT file specification (fileVersion=1):
typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int parametersSize;
} h3dlut;
enum { PAGE_SIZE = 8192; };
struct
{
h3dlut header;
char parametersData[PAGE_SIZE-sizeof(h3dlut)];
// The next element should start on a page aligned offset to allow MMAPing the raw data
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
// char padding[1]; // entire file length should be an exact multiple of Page Size.
} f3dlut;
/*
// Header
signature - File signature; must be: '3DLUT'
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or R,G,B)
outputBitDepth - Output bit depth per component (The same for all components)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array parametersData
// Parameters data
parametersData - array of char with size parametersSize that contains a copy of the
run parameters settings used for creating the file
// 3D LUT output data
lut.dataxx - array with size lutSize that contains the 3dlut output values.
The type used (lut.dat8 or lut.data16) is defined by the
outputBitDepth field:
- lut.data8 when outputBitDepth = 8
- lut.data16 when outputBitDepth = 16
The offset and size of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
lutSize = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(Size in bytes:
- lutSize*1, when outputBitDepth = 8
- lutSize*2, when outputBitDepth = 16)
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/


MMAPing involves directing the OS to page directly from a specified file, there are usually many constraints, like the disk image must be physical page aligned, etc. The technology avoids the startup overhead of reading a hugh file into memory. As the elements of data are accessed the OS page faults the required chunk into the address space. If certain elements are never used then the elements are never read or allocated physical memory.

[yesgrey]

Seeing this is a file header, I would suggest you explicitly align all the elements independent of any compiler.

And seeing the lut is some 48+ megabytes, catering for accessing it via MMAPing technology in the future might be prudent, so align the lut data now to the largest architecture page size likely, i.e 8192 or maybe 16384. Also pad the entire file length to this page size.
Thank you for your suggestions.:)
For me it's ok doing it like this, even considering than MMAPing could be useless for the 3D LUT, since we want it all in memory for performance reasons...
tritical, are you also ok with this?

[madshi]

What happens if parametersData is longer than "PAGE_SIZE-sizeof(h3dlut)"? What happens if parametersData is only 100 bytes? How does an application know that it's only 100 bytes and not "PAGE_SIZE-sizeof(h3dlut)"? Ok, the bytes 101+ could be zeroed out. But that's not really a pretty solution...

MMAPing involves directing the OS to page directly from a specified file, there are usually many constraints, like the disk image must be physical page aligned, etc. The technology avoids the startup overhead of reading a hugh file into memory. As the elements of data are accessed the OS page faults the required chunk into the address space. If certain elements are never used then the elements are never read or allocated physical memory.
That's true. But for real time playback it's better to have a startup overhead than to have the video stutter for a few seconds until the whole 48MB (or 96MB with 16bit output) are read in. So I don't really like the idea of memory mapping the array, at least not for media playback purposes. Reencoding may be a different situation...

[yesgrey]

How does an application know that it's only 100 bytes and not "PAGE_SIZE-sizeof(h3dlut)"? Ok, the bytes 101+ could be zeroed out. But that's not really a pretty solution...
By looking to parametersSize. I also agree it's not a pretty solution, but the idea is to get a general solution. We could dimension pametersData to a higher multiple of PAGE_SIZE just to be safe... For example: [4*PAGE_SIZE-sizeof(h3dlut)]; or putting parametersData at the end of the file.

So I don't really like the idea of memory mapping the array, at least not for media playback purposes. Reencoding may be a different situation...
I agree, but remember that MMAPing only will be used if the application wants it.

[IanB]

...
parametersSize - Size in bytes of the array parametersData
...And where the actual LUT data starts can either be implicitly at the first page aligned offset after the parameter data or you could explicitly add an offset field/padding length to the header.
I agree, but remember that MMAPing only will be used if the application wants it.I am just saying make provision for this.

[madshi]

We could dimension pametersData to a higher multiple of PAGE_SIZE just to be safe... For example: [4*PAGE_SIZE-sizeof(h3dlut)]
Ok, then let's do that.

It has also the advantage that we don't need to calculate the offset of the 3dlut array, anymore. We can directly access it through "f3dlut->lut" then.

[yesgrey]

What happens if parametersData is longer than "PAGE_SIZE-sizeof(h3dlut)"? What happens if parametersData is only 100 bytes? How does an application know that it's only 100 bytes and not "PAGE_SIZE-sizeof(h3dlut)"?
I was thinking in copying to parametersData exactly the same strings I read from the input file, with a '\n' character at the end of each parameter setting valid line, and an EOF at the end of it.
And where the actual LUT data starts can either be implicitly at the first page aligned offset after the parameter data or you could explicitly add an offset field/padding length to the header.
I see two ways of doing it:
1) dimension parametersData to a higher multiple of PAGE_SIZE just to be safe (I think with [4*PAGE_SIZE-sizeof(h3dlut)] we would be safe enough), and use parametersSize to indicate the portion of parametersData that has valid data.
2) Don't dimension parametersData to a fixed size. Set parametersSize to: n*PAGE_SIZE-sizeof(h3dlut); where n is the number of pages needed to contain the header and parametersData. Navigate through parametersData using the C characters like if it was a text file.

I believe 4*PAGE_SIZE-sizeof(h3dlut) will be big enough to hold any parameters file that could be created in the future, but we'll never know... and it would be strange to create a new file version just to increase the size of the parametersData field... but we can always do it.;)

There is still one thing left: compression. We haven't talked nothing about it, and with the size of the 3DLUT files it's important, because some users will need to have 3 or 4 in their disks... a 48MB uncompressed 3DLUT file can be reduced to a <1MB file...

[madshi]

There is still one thing left: compression. We haven't talked nothing about it, and with the size of the 3DLUT files it's important, because some users will need to have 3 or 4 in their disks... a 48MB uncompressed 3DLUT file can be reduced to a <1MB file...
Good idea. Reading in a compressed 1MB file and uncompressing it in RAM might also be faster than reading in a full 48MB file (if the decompression doesn't take too much time).

Should we use a custom method (e.g. difference to previous Lut index)? Or should we use e.g. some ZIP-like algorithm? In the latter case we should use not too strong compression, I'd say, so that decompression doesn't take too much time. It doesn't matter much if the 3dlut file is 1MB or 2MB, I think.

Should we *always* compress the lut file? I'd almost vote for "yes". But that would also mean that we could forget about aligning the file format for memory mapping purposes. Because all advantages of memory mapping would be gone if the lut data must be decompressed, anyway.

[leeperry]

There is still one thing left: compression. We haven't talked nothing about it, and with the size of the 3DLUT files it's important, because some users will need to have 3 or 4 in their disks... a 48MB uncompressed 3DLUT file can be reduced to a <1MB file...
you can compress a 48mb LUT to 1 mb w/ WinRAR....but the process is rather slow.
I've personally copied the LUT on a ramdisk(plus the RAR compressed ones), and I'm using shortcuts in the start menu to decompress the right one...I think you should let the end user take care of this, as decompressing in memory each time you open a video file would really slow things down..

and provided that you mostly watch US movies, all you need is SMPTE-C.....I seldom use EBU or HDTV, so that avoids wasting time decompressing LUT's each time.

start /b /high Z:\WinRAR\Rar.exe e -y Z:\pj_SMPTE-C.rar Z:\

and then :

rgb3dlut(lutfile="z:\lut.txt",threads=4)

a ramdisk is also good to avoid fragmentation and accelerate the media opening time..I think you can easily find some freeware ramdisk apps.

[madshi]

I think you should let the end user take care of this, as decompressing in memory each time you open a video file would really slow things down..
That depends a lot on the algorithm and compression ratio. I think a light ZIP compression should uncompress very fast. Of course it won't be as effective as RAR.

[yesgrey]

The problem with all this is that we will keep delaying the release of new versions of rgb3dlut and cr3dlut.
I think it might be a good idea releasing now a version and then update the file format specification after some more debate. Since currently there are only two applications using the 3DLUT format, we could update it in a week or two and continue calling it the version 1.0.
I know it will be a little annoying for the users, because they will have to run the cr3dlut file again to recreate their 3DLUT files, but they will always have to do it when cr3dlut add new features...

tritical,
my next release will use the following file specification. Are you ok with it?
typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int parametersSize;
} h3dlut;
/*struct
{
h3dlut header;
char parametersData[1];
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
} f3dlut; */


Please let continue the discussion about the file format.

[madshi]

Just did a small test: Loading an 48MB 3dlut file from harddisk, while my PC was a bit busy took 844ms. Unzipping a 3dlut file costs 984ms on my (not very fast) PC. However, the zipping method I chose was *very* light. It only brought the 48MB file down to roughly half its size. Still, I think if we choose the right compression method, uncompression at realtime doesn't have to be much of a penalty, especially since CPUs are getting faster at a quicker pace compared to harddisks.

But my test was only quick and dirty and may be flawed. Some more tests may be in order...

The problem with all this is that we will keep delaying the release of new versions of rgb3dlut and cr3dlut.
I think it might be a good idea releasing now a version and then update the file format specification after some more debate. Since currently there are only two applications using the 3DLUT format, we could update it in a week or two and continue calling it the version 1.0.
Sure, why not. I'd suggest to set fileversion to 0 for now, and then move it to 1 when the final file format is finally completed.

[yesgrey]

Sure, why not. I'd suggest to set fileversion to 0 for now, and then move it to 1 when the final file format is finally completed.
Ok, will set it to 0.
I think we are all very curious to see if the dithering is worth it or not...:D

[tritical]

my next release will use the following file specification. Are you ok with it?
What is the final verdict on calculating/indicating the amount of padding between parametersData and the actual lut values?

My suggestion is to keep everything the way it is, but state that the lut values start at the next multiple of PAGE_SIZE after the end of the parametersData array. Then all we have to do is add a definition for PAGE_SIZE to the specification.

[73ChargerFan]

How about using the 7-zip lzma sdk (http://www.7-zip.org/sdk.html) for compression? It is public domain, compressed a 48mb 3dlut file to 4.6mb & decompressed in less than 1 second (q6600 2.4 ghz).


* Compression speed: 2 MB/s on 2 GHz dual-core CPU.
* Decompression speed: 20-30 MB/s on 2 GHz Intel Core2 or AMD Athlon 64.
* Small memory requirements for decompression: 8-32 KB + DictionarySize
* Small code size for decompression: 2-8 KB (depending on speed optimizations)

Add a 10 character field to the header for the compression format (or none) of the data. And don't compress the header.

Loading 5mb is much better than 48mb.

[leeperry]

well, if you can still allow uncompressed LUT's I'll be a happy camper :o

[madshi]

What is the final verdict on calculating/indicating the amount of padding between parametersData and the actual lut values?

My suggestion is to keep everything the way it is, but state that the lut values start at the next multiple of PAGE_SIZE after the end of the parametersData array. Then all we have to do is add a definition for PAGE_SIZE to the specification.
I would be fine with that. But what is your opinion about compressing the lut data in the file? If we compress it then aligning the lut array to a page boundary doesn't make much sense, anymore.

[IanB]

If you are keen on compression, have a look at the source code for TCPSource(), sh0dan has done quite a bit research on various high performance compression modules.

Also the golden rule with compression is "know they enemy".

Any organised pre-processing that can be applied to data to lower the entropy of the stream going into the compression library will be a big win.

I suspect a predict-left style pre-processing would yield a very small population of delta values, i.e. -2, -1, 0, 1 & 2. This would compress extremely well with even trivial engines.

[yesgrey]

My suggestion is to keep everything the way it is, but state that the lut values start at the next multiple of PAGE_SIZE after the end of the parametersData array. Then all we have to do is add a definition for PAGE_SIZE to the specification.
If we don't go with the compression, this is the way I think it should be.
Do you want to consider the compression?

[tritical]

I don't particularly care one way or the other on compression, but if it is added I like 73ChargerFan's suggestion of adding a field to the header indicating what type or none. Then have the compressed or uncompressed lut values start at the next PAGE_SIZE multiple after the header and parameterData array. For now we could just support 'none' for compression to get everything going, and then discuss about what type(s), pre-processing, and other details in the future.

[madshi]

If you are keen on compression, have a look at the source code for TCPSource(), sh0dan has done quite a bit research on various high performance compression modules.

Also the golden rule with compression is "know they enemy".

Any organised pre-processing that can be applied to data to lower the entropy of the stream going into the compression library will be a big win.

I suspect a predict-left style pre-processing would yield a very small population of delta values, i.e. -2, -1, 0, 1 & 2. This would compress extremely well with even trivial engines.
Good thinking. A little pre-processing and then a fast compression algorithm should be a good solution. I'm using a similar approach in one of my applications (not audio/video data related, though).

I don't particularly care one way or the other on compression, but if it is added I like 73ChargerFan's suggestion of adding a field to the header indicating what type or none. Then have the compressed or uncompressed lut values start at the next PAGE_SIZE multiple after the header and parameterData array. For now we could just support 'none' for compression to get everything going, and then discuss about what type(s), pre-processing, and other details in the future.
Sounds good to me.

So one more field in the header. E.g. "compressionMethod". Set to 0 means "no compression". I don't care in which position in the header the new field is added. Just add it where you see fit.

Nice, I think we're finally done... :)

[yesgrey]

Here is the updated 3DLUT file format specification:
// 3D LUT file specification (fileVersion=1):
typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int parametersSize;
int lutCompression;
int reserved;
} h3dlut;
enum {PAGE_SIZE = 8192};
struct
{
h3dlut header;
char parametersData[1];
// The next element should start on the next PAGE_SIZE multiple to allow MMAPing the raw data
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
// char padding[1]; // entire file length should be an exact multiple of PAGE_SIZE.
} f3dlut;
/*
// Header
signature - File signature; must be: '3DLUT'
fileVersion - File format version number
inputBitDepth - Input bit depth per component (Y,Cb,Cr or R,G,B)
outputBitDepth - Output bit depth per component (The same for all components)
inputColorEncoding - Input color encoding standard
- ...
outputColorEncoding - Output color encoding standard
- ...
programName - Name of the program that created the file
programVersion - Version of the program that created the file
parametersSize - Size in bytes of the array parametersData
lutCompression - Indicates if lut.dataxx array is compressed and which method was used
0 - None (not compressed)
reserved - Reserved for future usage
// Parameters data
parametersData - array of char with size parametersSize that contains a copy of the
run parameters settings used for creating the file
// 3D LUT output data
lut.dataxx - array with size lutSize that contains the 3dlut output values.
The type used (lut.dat8 or lut.data16) is defined by the
outputBitDepth field:
- lut.data8 when outputBitDepth = 8
- lut.data16 when outputBitDepth = 16
The offset and size of the array are calculated as:
offset = (cr<<(inputBitDepth[1]+inputBitDepth[0])+cb<<(inputBitDepth[0])+y)*3 (YCbCr input)
offset = ( r<<(inputBitDepth[1]+inputBitDepth[0])+ g<<(inputBitDepth[0])+b)*3 (RGB input)
lutSize = 3*(2^inputBitDepth[0]*2^inputBitDepth[1]*2^inputBitDepth[2])
(Size in bytes:
- lutSize*1, when outputBitDepth = 8
- lutSize*2, when outputBitDepth = 16)
(This specification assumes: char = 1 byte; short = 2 byte; int = 4 byte; long int = 8 byte)
*/

I have named the new variable lutCompression to show that only the LUT data is compressed. The parametersData will always be very small, so no need to compress it, and this way we could always edit the 3DLUT file in a text editor and see the parameter settings used for its creation.
I have added again one reserved entry, in case we would need it.
Should we always pad the file length to a PAGE_SIZE multiple, or only when it's not compressed?
The parametersSize, should indicate the size of parametersData that contains valid data, or should also include the padding to reach the PAGE_SIZE multiple before the LUT data?

[madshi]

Here we go again... ;)

Things got more complicated because of the page aligning. So maybe we should reconsider the separation of header and file structures? Because using "sizeof(h3dlut)" doesn't make any sense, anymore, when we use page aligning.

So how about this?

typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int lutFileOffset;
int lutCompressionMethod;
int lutCompressedSize;
int lutUncompressedSize;
int reserved;
int parametersSize;
char parametersData[1];
} header3dlut;

typedef union
{
unsigned char data8[1];
unsigned short data16[1];
} data3dlut;
Suggested changes:

(1) A new field "lutFileOffset" would make it easy to find the lut array in the file.
(2) The new fields "lutCompressedSize" and "lutUncompressedSize" are simply helpers for memory allocation.
(3) Moved "parametersData" to the header cause "parametersData" and "lut.dataXX" are separated now due to page aligning.
(4) The 2nd structure shouldn't contain the header, anymore, cause due to page alignment the lut data is now separated from the header.
(5) Doing (3) and (4) leaves the 2nd structure with only the lut array, so we can simplify that structure.
(6) Moved "reserved" up, so that "parametersSize" and "parametersData" are next to each other.
(6) Renamed "lutCompression" to "lutCompressionMethod" for better understanding.

Comments?

[yesgrey]

So maybe we should reconsider the separation of header and file structures? Because using "sizeof(h3dlut)" doesn't make any sense, anymore, when we use page aligning.
If parametersSize is set to: n*PAGE_SIZE-sizeof(h3dlut); where n is the number of pages needed to contain the header and parametersData, then "sizeof(h3dlut)" will still make sense.
I am ok with your suggestions, but the previous was good because it had very clear all the file structure.
Let's hear what other people think about it.
Both me and tritical need to know how to write/read the 3DLUT files...

[madshi]

If parametersSize is set to: n*PAGE_SIZE-sizeof(h3dlut); where n is the number of pages needed to contain the header and parametersData, then "sizeof(h3dlut)" will still make sense.
Can you define a structure with a dynamic "n" element in C(++) and then use "sizeof"? In Delphi that is not possible.

the previous was good because it had very clear all the file structure.
We could also integrate everything into one big structure with a big comment in front of the "lut data" element (page alignment). That would also make the file structure clear. The only reason to split the structure into two parts was to make "sizeof" possible.

[yesgrey]

Can you define a structure with a dynamic "n" element in C(++) and then use "sizeof"? In Delphi that is not possible.

No, but I wasn't thinking in putting the dynamic "n" on it.
The paging of the file structure does not bring anything new. The parametersSize was already not fixed, the only difference is that before it was only the size of the array that contained a copy of the input file, and now we want to put in there some more null bytes just to reach a PAGE_SIZE multiple.
The only reason to split the structure into two parts was to make "sizeof" possible.
But the sizeof will not be very friendly, it will include the first character of parametersData...

To be honest, this is not very important to me, because whichever model we decide, I cannot make it work completelly in C.:o
I always have to replace the:
char parametersData[1];
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;

with:
char parametersData[16384];
union
{
unsigned char *data8;
unsigned short *data16;
} lut;

Perhaps my C skills are short in handling this kind of stuff, so feel free to specify it the way you feel more confortable with...;)

[madshi]

The paging of the file structure does not bring anything new.
Yes, it does. We were planning to locate the lut data by using the following formula:

sizeof(header) + parametersSize
And that "sizeof" in that formula was the one and only reason why we used a separate structure for the header. We cannot use that formula anymore now, because "parametersSize" does not include the padding to page size. At least I don't think it does/should.

Now doing some weird math to find the beginning of the lut data is not to my liking, that's why I suggested adding a "lutFileOffset" field. But then having such an offset field in the header means that we don't need "sizeof(header)", anymore. And that means we don't need a separate structure for the header, anymore, either. So I'd say: Let's put all fields in one structure, where the lut data union is the last element.

typedef struct
{
char signature[8];
int fileVersion;
int inputBitDepth[3], outputBitDepth;
int inputColorEncoding, outputColorEncoding;
char programName[32];
long int programVersion;
int lutFileOffset;
int lutCompressionMethod;
int lutCompressedSize;
int lutUncompressedSize;
int reserved;
int parametersSize;
char parametersData[1];

// the next element should start on the next PAGE_SIZE multiple
union
{
unsigned char data8[1];
unsigned short data16[1];
} lut;
} file3dlut;

Just my opinion, of course. What do tritical and IanB think?

[73ChargerFan]

and this way we could always edit the 3DLUT file in a text editor and see the parameter settings used for its creation.Could the parameter settings field be initialized to spaces (as opposed to random data) to make it easy to see view in a text editor? That would also visually differentiate it from the data.

[yesgrey]

We cannot use that formula anymore now, because "parametersSize" does not include the padding to page size. At least I don't think it does/should.
Now doing some weird math to find the beginning of the lut data is not to my liking, that's why I suggested adding a "lutFileOffset" field.
I also agree with this. What I said was considering that "parametersSize" included the padding to page size, and adding a C 'EOF' character at the end of the valid "parametersData" data. If you prefer that "parametersSize" only refers to the size without the padding, for me it's ok. In fact, it would be better if someone wants to show it (no need to go through the array looking for the 'EOF' character).;)
So I'd say: Let's put all fields in one structure, where the lut data union is the last element.

It looks good to me.:)

Could the parameter settings field be initialized to spaces (as opposed to random data) to make it easy to see view in a text editor? That would also visually differentiate it from the data.
From the reply above, the "parametersData" will only contain the parameters settings. We could set all the padding bytes until the PAGE_SIZE to the character space to make it look cleaner, but I think that setting it to zero would be preferable.
By the way, what do you prefer in "parametersData":
1) a clean copy of the parameters settings with only the valid lines.
2) a clean copy of the parameters settings with the valid lines and the comment lines (lines started by '#')
3) an exact copy of the input file.

[madshi]

By the way, what do you prefer in "parametersData":
1) a clean copy of the parameters settings with only the valid lines.
2) a clean copy of the parameters settings with the valid lines and the comment lines (lines started by '#')
3) an exact copy of the input file.
I've vote for (3) because I'm thinking that if an application has the ability to recreate missing 3dlut files on the fly by using cr3dlut, such an application might also want to double check whether an existing file matches exactly the needed parameters. In order to do that check, the application would create the parameter settings string in RAM and then compare that to what is stored in the 3dlut file's "parametersData". If it matches, the file is ok. If it doesn't match, the file needs to be recreated.

[tritical]

What is the point of the union and

char parametersData[1];

line? Why not just make it:

struct H3DLUT
{
char signature[8]; // File signature; must be: '3DLUT'
int fileVersion; // File format version number
int inputBitDepth[3]; // Input bit depth per component (Y,Cb,Cr or R,G,B)
int outputBitDepth; // Output bit depth for all components (valid values are 8 and 16)
int inputColorEncoding; // Input color encoding standard
int outputColorEncoding; // Output color encoding standard
char programName[32]; // Name of the program that created the file
long int programVersion; // Version of the program that created the file
int parametersSize; // Size in bytes of the array parametersData
int lutFileOffset; // number of bytes between end of parametersData and lut data
int lutCompressionMethod;// type of compression used if any (0 = none, ...)
int lutCompressedSize; // if compressed, size of compressed lut values in bytes
int lutUncompressedSize; // uncompressed size of lut values in bytes
int reserved; // reserved for future use
// This header if followed by the char array 'parametersData', of length 'parametersSize'.
// The lut data starts 'lutFileOffset' bytes after the end of the 'parametersData' array,
// and that pos must be on a 16384 byte boundary if the data is uncompressed.
};

Now in the uncompressed case we could read it using code like (no error checking):

FILE *f = fopen("lut","rb");
H3DLUT h3dlut;
fread(&h3dlut,sizeof(H3DLUT),1,f);
fseek(f,h3d.parametersSize+h3d.lutFileOffset,SEEK_CUR);
lut3d = (unsigned char*)_aligned_malloc(h3d.lutUncompressedSize,16);
fread(lut3d,1,h3d.lutUncompressedSize,f);
fclose(f);