FranceBB LUT Collection

[FranceBB]

Hi,
LUTs are generally used to grade contents, but they can be useful to do quick and dirt color space conversions, especially in this day and age.
Although a good color conversion has to be achieved manually and/or via algorithms, LUTs can still be helpful; unfortunately, I've seen people selling their LUTs for simple color conversion and it incredibly pissed me off 'cause they are supposed to be free, so today I'm gonna share my LUT collection, hoping that it can be useful to someone.

Before you apply any of my LUTs, remember that although you can apply them at any bit depth, I strongly suggest you to do that with 16bit precision, otherwise you'll get banding and other terrible issues.
LUTs are 65x65x65, which is supposed to be high precision, but some of them are 33x33x33 which is just about right. None of them are 17x17x17 'cause I don't consider it to be accurate enough.

LUTs: Link (https://github.com/FranceBB/LinearTransformation)

How to use my LUTs in Avisynth:

FFVideoSource("example.mxf")

ConvertBits(16)

ConvertToPlanarRGB()

Cube("C:\Programmi\AviSynth+\LUTs\example.cube")

How to use my LUTs in VapourSynth:


import vapoursynth as vs
core = vs.core

core.ffms2.Source(r'example.mxf')
vid = core.resize.Spline64(vid, format=vs.RGBS)
vid = core.timecube.Cube(vid, cube=r"C:\Programmi\AviSynth+\LUTs\example.cube")

vid.set_output ()

How to use my LUTs in FFMpeg:


ffmpeg.exe -i "source.m2ts" -vf lut3d='example.cube' -c:v whatever -c:a whatever -f mkv "output.mkv"



Alternatively, you can use my plugin "Linear Transformation": Link (https://github.com/FranceBB/LinearTransformation/blob/master/LinearTransformation.avsi)

Introducing LinearTransformation (https://github.com/FranceBB/LinearTransformation).

With the increasing number of matrices to do any kind of linear transformations, I kinda felt the need to make something more accessible for users, which is why I'm introducing now "LinearTransformation".
This plugin performs Linear Transformations based on a matrix conversion. Matrices are made by me and can be used independently from Avisynth as well.

Parameters
LinearTransformation(clip clp, string "Input", string "Output", string "Interpolation", string "LUT_Path")

clip = input clip
string Input = source input you're starting from
string Output = destination output you want to go to
string Interpolation = interpolation method to use (Trilinear or Tetrahedral)
string LUT_Path = path to the folder where the LUTs are ( e.g D:\MyLUTs\ while default is C:\Program Files (x86)\AviSynth+\LUTs\ )


The function has two mandatory parameters: Input and Output.
The function also has two optional parameters: Interpolation and LUT_Path.

The interpolation parameter can either be set to "Trilinear" or "Tetrahedral" and by default it's set to Tetrahedral as it achieves better results.

The LUT_Path allows you to specify a different path if you prefer to have the .cube files in a different place that isn't the default C:\Program Files (x86)\AviSynth+\LUTs.

Source (left) - Trilinear (center) - Tetrahedral (right):
https://images2.imgbox.com/ce/d3/lsm1LWs8_o.png


As you can see, in the right hand side waveform (tetrahedral) there are no ripples, unlike the ones you can see in the trilinear interpolation at the center.
Currently, only those conversions are supported:

LinearTransformation(Input="Linear_BT601_NTSC", Output="Linear_BT601_NTSC-J")
LinearTransformation(Input="Linear_BT601_NTSC-J", Output="Linear_BT601_NTSC")
LinearTransformation(Input="Linear_BT601_NTSC", Output="Linear_BT709")
LinearTransformation(Input="Linear_BT601_PAL", Output="Linear_BT709")
LinearTransformation(Input="Linear_BT709", Output="Linear_BT601_NTSC")
LinearTransformation(Input="Linear_BT709", Output="Linear_BT601_PAL")
LinearTransformation(Input="Linear_BT709", Output="Linear_BT2020")
LinearTransformation(Input="Linear_BT709", Output="BT2020_HLG")
LinearTransformation(Input="Linear_BT709", Output="BT2100_PQ")
LinearTransformation(Input="Linear_BT709", Output="DCI_XYZ")
LinearTransformation(Input="Linear_BT2020", Output="Linear_BT709")
LinearTransformation(Input="BT2100_PQ", Output="Linear_BT2020")
LinearTransformation(Input="BT2100_PQ", Output="Linear_BT709")
LinearTransformation(Input="BT2100_PQ", Output="BT2020_HLG")
LinearTransformation(Input="BT2100_PQ", Output="DCI_XYZ")
LinearTransformation(Input="ALog", Output="Linear_BT709")
LinearTransformation(Input="ALog", Output="BT2020_HLG")
LinearTransformation(Input="ALog", Output="BT2100_PQ")
LinearTransformation(Input="CLog3", Output="Linear_BT709")
LinearTransformation(Input="CLog3", Output="Linear_BT601_PAL")
LinearTransformation(Input="CLog3", Output="Linear_BT601_NTSC")
LinearTransformation(Input="CLog3", Output="BT2020_HLG")
LinearTransformation(Input="CLog3", Output="BT2100_PQ")
LinearTransformation(Input="SLog2", Output="Linear_BT709")
LinearTransformation(Input="SLog3", Output="Linear_BT709")
LinearTransformation(Input="SLog3", Output="Linear_BT601_PAL")
LinearTransformation(Input="SLog3", Output="Linear_BT601_NTSC")
LinearTransformation(Input="BT2020_HLG", Output="Linear_BT709")
LinearTransformation(Input="BT2020_HLG", Output="BT2100_PQ")
LinearTransformation(Input="BT2020_HLG", Output="DCI_XYZ")
LinearTransformation(Input="DCI_XYZ", Output="Linear_BT709")
LinearTransformation(Input="DCI_XYZ", Output="BT2020_HLG")
LinearTransformation(Input="DCI_XYZ", Output="BT2100_PQ")
LinearTransformation(Input="LogC", Output="Linear_BT709")
LinearTransformation(Input="LogC", Output="Linear_BT601_PAL")
LinearTransformation(Input="LogC", Output="Linear_BT601_NTSC")
LinearTransformation(Input="VLog", Output="Linear_BT709")
LinearTransformation(Input="VLog", Output="Linear_BT601_PAL")
LinearTransformation(Input="VLog", Output="Linear_BT601_NTSC")
LinearTransformation(Input="ZLog", Output="Linear_BT709")
LinearTransformation(Input="ZLog2", Output="Linear_BT709")
LinearTransformation(Input="dvhe0509", Output="Linear_BT709")

A Linear Transformation is essentially a matrix that maps all points of a certain space to another, which includes of course points belonging to a certain curve to other in order to get a different curve.
Of course, a linear transformation can be used in encoding to map some values to some other values and therefore have conversions from curves like PQ to HLG and so on.
The transformation is performed with 16bit precision, which means that if your input source is lower, let's say, 8bit planar yv12, it will be brought to 16bit planar RGB internally, the linear transformation will be applied with 16bit planar precision and then the result will be brought down to 8bit planar yv12.
Planar RGB 16bit is strongly suggested as your source as it's gonna be faster, in fact 4:2:0, 4:2:2, 4:4:4 planar up to 16bit will be converted back and forth internally.
Inside the plugin, the path specified for the matrices by default is: "C:\Program Files (x86)\AviSynth+\LUTs" which means that it's gonna look for my LUTs in a folder in such a location. It's not mandatory to have my LUTs there, you can have them in any location you want, provided that you do update that string via the LUT_Path parameter while calling the function.

A huge thanks to StainlessS and to the whole community for greatly improving the script and helping me writing it.

Plugin: https://github.com/FranceBB/LinearTransformation/blob/master/LinearTransformation.avsi
Matrices: https://github.com/FranceBB/LinearTransformation

Usage Example:

Example 1: From Slog3 to Linear BT709


FFVideoSource("mySlog3source.mxf")
LinearTransformation(Input="SLog3", Output="Linear_BT709")


Example 2: From PQ to HLG


FFVideoSource("myPQsource.mxf")
LinearTransformation(Input="BT2100_PQ", Output="BT2020_HLG")



Video:

https://i.ytimg.com/vi/XL8kkeEfQuk/hqdefault.jpg (https://www.youtube.com/watch?v=XL8kkeEfQuk&t=1s)

Images:

Slog3 pic1:
https://i.imgur.com/9E6yr7S.png

BT709 pic1:
https://i.imgur.com/7DHYloZ.png

-

Slog3 pic2:
https://i.imgur.com/glV6rHA.png

BT709 pic2:
https://i.imgur.com/icGXfbo.png

-

Slog3 pic3:
https://i.imgur.com/kYlmAet.png

BT709 pic3:
https://i.imgur.com/VaDIB0M.png

Slog3 to BT709
https://i.imgur.com/TIRdtG5.png

Who made those LUTs?
Those matrices are made by Francesco Bucciantini (FranceBB) (https://www.linkedin.com/in/francesco-bucciantini-3392b4ab/), computer science engineer and Linear Algebra lover and Livio Aloja (aligia) (https://www.linkedin.com/in/livio-aloja-9a287424/), former Senior Sky Editor and Encoder, using both free open source tools and closed source ones and cross-checking them with broadcast grade equipment (TVs, waveform monitors, scopes) provided by Sony, Canon and Tektronix.

Who uses those LUTs?
Those LUTs are used by everyone for free, from companies that deal with broadcasting video content in any way (Post-production, OTT, etc) like public and private broadcasters all around the world to regular people that want to convert their video archives.

Mentions in other programs:
These LUTs are officially shipped with
- FFASTrans - FFMpeg Avisynth Transcoder by Steinar Apalnes, Benjamin Dissoubret and Francesco Bucciantini (https://www.ffastrans.com/)
- Hybrid by Selur (https://www.selur.de/)

Compatibility with third party programs:
- Compatible with Avisynth+
- Compatible with VapourSynth
- Compatible with AVID Media Composer
- Compatible with Davinci Resolve
- Compatible with Adobe Premiere
- Compatible with Colorfront Transkoder

[gonca]

Thank you for this.
One question
Should it be fulldepth=true or false
since false would return 8 bit I believe

[FranceBB]

Should it be fulldepth=true or false
since false would return 8 bit I believe

Nope, they both return 16bit, but fullrange=true is for full range (PC Range) sources, while fullrange=false is for limited range (TV Range) sources.
Limited range for 8bit is 16-235, for 10bit is 64-940 and so on.
Full range for 8bit is 0-255, for 10bit is 0-1020 and so on.
It depends on your source, but keep in mind that the majority of sources are TV Range and if you encode something for the TV, it has to be Limited TV Range.

[gonca]

My mistake
I originally read fulldepth in your sample script, but after re reading it I realize that it actually says fullrange

[StainlessS]

Full range for 8bit is 0-255, for 10bit is 0-1024 and so on.
Nit Pick, 1023.

EDIT: Hmmm, I'm wrong too.

[FranceBB]

Sorry about that.
Actually, this is the right one, I think (from the Avisynth Wiki):

https://i.imgur.com/aZtIN8V.png

[huhn]

these "issues" come from the understanding that 10 bit is 4 times as big as 8 bit so we just have to multiply by 4.

8 bit RGB 255 255 255 as 10 bit is 1023 1023 1023 not 1020 1020 1020 has a very slightly lower brightness and is just 4x the original number.

the correct way to increase 8 bit to 10 bit is taking the first 2 bits from the 8 bit source and add them to the end.

or as an small example:

11111111=255 100 % brightness
1111111100=1020 ~99.7% brightness
1111111111= 1023 100% brightness

[wonkey_monkey]

the correct way

But according to whom? The wiki seems to indicate the other way - simple shifting - is correct. What's the broadcast standard?

And what's the correct way to go back down? Is there bit-ty way to do it, or do you have to divide by 4.0117647059?

And I thought 16-235 was a headache...

[huhn]

i just ask you a different question who stops you from creating a native 10 bit source with 1023 in it?

a bad way to go back to 8 bit is by removing the last too bits truncation. another one would be dithering.

[wonkey_monkey]

i just ask you a different question who stops you from creating a native 10 bit source with 1023 in it?

Umm, okay, but my questions weren't rhetorical. Nothing stops you from creating a native 8 bit source with 255 in it, but that is - according to broadcast standards - an "illegal" value.

a bad way to go back to 8 bit is by removing the last too bits truncation. another one would be dithering.

Dithering still needs a convention as to what values, integer of fractional, correspond between the bit depths.

[huhn]

Umm, okay, but my questions weren't rhetorical. Nothing stops you from creating a native 8 bit source with 255 in it, but that is - according to broadcast standards - an "illegal" value.

full range is a thing. computer games work in it it's part of display port and HDMI.

while BD source is all ways limited range this doesn't change that even full range encoding is a thing and totally supported by programs like x264.

and be aware that this is about RGB not YCbCr.
so clearly not illegal
Dithering still needs a convention as to what values, integer of fractional, correspond between the bit depths.
does it? how should i even know what an dithering algorithm is working internal. i know it using the "errors" truncated parts to spread noise if it is max 1023 or 1020 shouldn't matter. it will just use what it has available.

[wonkey_monkey]

Okay, we're getting off-track. The original point is: the Avisynth wiki says 8-bit 255 is equivalent to 10-bit 1020. You say it's 1023. Who's right?

[silverwing]

FranceBB, thank you!
So.
These are examples with your lut "PQ_to_HLG.cube".
I get a similar result only with the help of a long selection of curves in the plugin "SmoothCurve". Good work. :thanks:

[FranceBB]

FranceBB, thank you!
So.
These are examples with your lut "PQ_to_HLG.cube".
I get a similar result only with the help of a long selection of curves in the plugin "SmoothCurve". Good work.

Thanks, I tried to tweak things correctly.


Update: Added Slog3 to BT709 LUT

Whenever I work for the News channel or for our Sport channel, I receive footages shot normally, but whenever I work on TV Series, I receive footages shot in Slog3 and I always have to bring them to BT709 to encode everything in a standard old-fashioned FULL HD SDR.
I'm gonna share with you my Slog3 to BT709 LUT as well.

https://i.imgur.com/TIRdtG5.png

Slog3 first, BT709 second:

https://i.imgur.com/9E6yr7S.png
https://i.imgur.com/7DHYloZ.png

https://i.imgur.com/glV6rHA.png
https://i.imgur.com/icGXfbo.png

Some people may think that it's ok-ish, but slightly dark on some low-light scenes, but actually this is because I added a knee to the curve and there's a reason for that: trying to stay in the Limited Tv Range avoiding clipping.
Let's take a look at this example here with a strong light coming from behind.
Raw footage in Slog3:

https://i.imgur.com/kYlmAet.png

If I try to apply a transformation without using a knee, the white level gets too high and I get out of legal range, therefore everything over 235 gets clipped out:

https://i.imgur.com/c8ItGyB.png

By adding a knee, however, I manage to get thinks right in 16-235 without the need to clip so many details out:

https://i.imgur.com/VaDIB0M.png

If you take a look at the curve of the transformation you can see how the unclipped one gets so high that it almost looks like a function that reaches a vertical asymptote:

https://i.imgur.com/oT6Tw7B.png

While the knee manages to smooth things out and generates a proper curve:

https://i.imgur.com/8Q3DQBi.png

[FranceBB]

Update: Added C-Log3 to BT709 SDR / C-Log3 to PQ HDR / C-Log3 to HLG HDR

I added three new LUTs that are meant to be used by Canon C-300 Camera users.
It's a 4K professional camera that can record up to 12bit with the C-Log3 curve and it can definitely be used for HDR productions and SDR ones alike.
What's interesting about this curve with the C-300 is that it's possible to shoot and then grade it to HDR PQ and get approximately 3500 nits worth of data (I mean, real data).
When I created the LUTs, I tried to keep the BT709 SDR one in range and I applied a knee to get the right values.
As to the HDR PQ one, I left it "unclipped" to 10'000 nits, however you're not gonna get anything over 3500 nits. The reason why I left it unclipped up to 10'000 nits, though, is that Canon might release new cameras in the future that are gonna use the same curve and perhaps they'll be able to get more than 3500 nits worth of data.
As to the HDR HLG one, I had to make a very big compromise, 'cause as you know, HLG has been invented in order to display both HDR and SDR using the same file, which basically means that whenever you make a LUT you have two choices: 1) make it more HDR-like and sacrifice the view of the ones with an SDR TV or 2) make it more SDR-like and sacrifice the view of the ones with an HDR TV.
Since many people still have SDR TVs, my choice (as well as the one of other encoders working in broadcast) is to make it more SDR-like, 'cause this way no one is ever gonna complain.
In order to do so, I had to sacrifice the representation of the black values and of course since HLG is limited to 1'000 nits, I had to lower down the highlights as well.
The result looks good and it's definitely more natural than the SDR, but comparing it to the PQ is... like... you know... not quite good enough, but as broadcaster it's kinda acceptable.

Alright, let me show you the graphs.

Let's start with the C-Log3 to BT709 SDR:

https://i.imgur.com/IwzbKey.png
https://i.imgur.com/IwzbKey.png
https://i.imgur.com/NC0Qvik.png

As you can see, the C-Log3 curve is very different compared to the BT709 and it's a curve that it's meant for HDR productions as well, so although it can be used for SDR, the grading should really be done manually on a scene by scene basis, 'cause otherwise you might end up with oversaturated or wrong values.

C-Log 3 to BT2100 HDR PQ:

https://i.imgur.com/t7kdtcs.png
https://i.imgur.com/h3xyoPh.png
https://i.imgur.com/iF3h0BL.png

As you can see, the C-Log 3 curve pretty much resembles the PQ curve, this is also why it's actually pretty easy to map it to HDR PQ BT2100.

C-Log 3 to BT2100 HDR HLG:

https://i.imgur.com/FnUukXL.png
https://i.imgur.com/hEC7UbZ.png
https://i.imgur.com/2RS5aCl.png

As you can see, I tried to keep the HLG curve as tweaked as possible to SDR, which actually limited the quality of the blacks for the HDR viewers. As to the peak, HLG is limited to 1000 nits by definition.

[FranceBB]

Alright, let's crack on and view some examples.
You'll find the original C-Log3 footage, the BT709 SDR one, the HDR HLG one and the HDR PQ one.
I tried to keep the BT709 LUT as usable as possible, but limiting C-Log3 to BT709 SDR means that many details are really gonna be clipped out and you can definitely see it from the examples below.
As to the HDR HLG BT2100, it pretty much resembles the BT709 SDR version, but it does show more details, despite being still capped out.
Finally, the HDR PQ BT2100 is the best looking version, with a very natural look and it almost feel like as if you were there for real with 3500 nits worth of data.

C-Log3
https://i.imgur.com/OGsGwCC.jpg
BT709 SDR
https://i.imgur.com/PEf8dPS.png
BT2100 HDR HLG
https://i.imgur.com/Ty2R7go.png
BT2100 HDR PQ
https://i.imgur.com/3D4nNem.png

C-Log3
https://i.imgur.com/fwxqH16.jpg
BT709 SDR
https://i.imgur.com/hisJcBM.png
BT2100 HDR HLG
https://i.imgur.com/iycLYSo.png
BT2100 HDR PQ
https://i.imgur.com/bgXCnHn.png

C-Log3
https://i.imgur.com/u32LPm4.png
BT709 SDR
https://i.imgur.com/RV23zGM.png
BT2100 HDR HLG
https://i.imgur.com/GPV7Oqf.png
BT2100 HDR PQ
https://i.imgur.com/yyqGQ5S.png

C-Log3
https://i.imgur.com/Y5nHirT.png
BT709 SDR
https://i.imgur.com/8LGSkWj.png
BT2100 HDR HLG
https://i.imgur.com/64zMdVa.png
BT2100 HDR PQ
https://i.imgur.com/aAbjw8L.png

[gonca]

Once again, thank you for your contribution :thanks:

[Sharc]

Perhaps a dumb question: How do people select the "correct" LUT? Is it a matter of personal taste, or do we need reference pictures for comparison?

[FranceBB]

Perhaps a dumb question: How do people select the "correct" LUT? Is it a matter of personal taste, or do we need reference pictures for comparison?

What do you mean?
Each LUT is a matrix of linear transformation that goes from a curve to the other, which basically means that, depending on your source color curve, you use the one to go from it to your target color curve.
For instance, if you have an HDR PQ video and you wanna go to HDR HLG, you are gonna use my PQ_to_HLG LUT.

If you are asking how to create a LUT yourself, then that's a whole different question and... no, it's not a matter of personal taste only, it's a matter of linear algebra, defined standards and personal taste.
You generally start with a linear conversion from a curve to the other and then tweak it trying to get into the right values according to the standard. After that, you can fine-tune it 'till you'll get something that looks best for your taste, as long as you are still respecting the standards.
Anyway, it really depends on your source and your target, 'cause if you have two curves that are really different one to the other, then you are gonna have to choose which parts of the input you wanna keep while you go to your desired output.
Let's suppose that you have a C-LOG3 in input and you wanna go to BT709 SDR in output; you can basically get the calculations right, but you still have to choose what you wanna sacrifice as the BT709 SDR is able to retain very little informations compared to the input and THAT is a personal choice according to what you think it's best.
When I make my LUTs, I always try to keep as many things as possible and make them usable for pretty much every scenario. Let's suppose that we have a scene with snow, a white dog and the sky: you are gonna have three different tonalities of white, one for the snow, one for the dog and one for the highlights (sky). If you originally shoot in BT709 SDR, you have one of those three tonalities of white clipped out, while if you shoot in C-LOG3 you are gonna have as many details as your camera can get, then, in encoding, you can decide to retain the desired part of the image, with the desired tonalities of white. In this specific case, you wouldn't be able to do it with my LUT, 'cause my LUT is gonna take a decision for you and clip what I thought it was supposed to be an acceptable compromise for every scenario, but it's better to do it manually, 'cause by doing it manually you can choose which things you are gonna preserve for that specific shot.
I hope this answer your question; if you want, I can get into more technical details.

Cheers,
Frank.

[Sharc]

Thank you Frank for enlightening me. Your short tutorial answered my nebulous question about the LUTs nicely :)

[FranceBB]

Hello Frank,

Thanks a lot for this! It will be useful for sure.
I wonder if you work on Sony Slog 2? I must admit it will be useful for me :)
Slog 2 to rec 709.
Thanks again.
Cheers.

Benjamin

I don't use Slog2 anymore, but I do have some old contents shot using that curve, so it's gonna be useful to me as well.
Slog2 is the second revision of the old version of the Sony log, which is significantly different if you compare it to Slog3 as we can see from here:

https://i.imgur.com/Rl5Y0io.png


(The scale used in the graph is the same, so it's easier to spot the differences).

As you can see from the graph, Slog2 curve is pretty much the same (although slightly lower) on the low terms with a more gradual raise, while as it goes up it intersects with Slog3 and it ends up raising faster. If you take a look at this Slog2 shot, you can see the differences between the two curves:

This is the orignal Slog2 footage:

https://i.imgur.com/F09jiif.png

This is how it would have been if it was recorded in Slog3:

https://i.imgur.com/0h3sM4a.png

Keep in mind that this specific example is kinda "bogus" 'cause the content has been originally shot in Slog2, so a linear transformation matrix had to be applied in order to convert it to Slog3; luckily the transformation was 1 to 1, but it's not onto. The reason is that every element in Slog2 can be mapped to Slog3, however the contrary is not true: for instance, Slog2 is a 14 stop maximum curve, so the peak level is recorded right at the top of the recording range, while the S-Log3 curve is designed to go beyond 14 stops. In other words, an element that resides within the >14 area can't be mapped back, so the linear transformation is not onto which basically means that it's not an isomorphism, which is basically why I said that the example above is just to give you something else other than the graph.

Enough digression for Slog2 vs Slog3 and linear algebra lovers, so let's crack on and take a look at the graph that goes from Slog2 and then we'll take a look at some examples.

This is what I've got so far for Slog2 to BT709 SDR:

https://i.imgur.com/EOM31i6.png

Please note that it's not just the curve, but also the Gamut as we have S-Gamut with Slog2 (it would have been S-Gamut3 with Slog3), which is not exactly so straightforward to convert to BT709 SDR as the color science inside it's slightly different from the one we use to deal with.

Let's take a look at our first example here:

Slog2:
https://i.imgur.com/QDdGSDG.png
BT709 SDR:
https://i.imgur.com/XI9mCKc.png

This is exactly what I mean with "it's not exactly so straightforward". It's very easy to get the different tonalities of the leaves/ground wrong, especially on a general purpose LUT, which is why I tried to keep it as low as possible. For instance, when I tweaked it to a scene, I've got a red-ish colors on brown tonalities, so this is the compromise I managed to get.

Let's take a look at a normal shot you are usually going to get:

Slog2:
https://i.imgur.com/pNElEE7.png
BT709 SDR:
https://i.imgur.com/Dm9njIH.png

As you can see, skin tones are natural and chroma grading is in general kinda conservative, as well as levels.

Let's take a look at the last example here:

Slog2:
https://i.imgur.com/SHg1JY4.png
BT709 SDR:
https://i.imgur.com/CZur4ZT.png

In this case, although the overall picture appears fine, in the BT709 SDR output we've got combing in the hat. Don't get me wrong, it wasn't introduced as an artifact by the conversion, but it was on the Slog2 S-Gamut already, however while it looked like a fine gradient, it became a noticeable "artifact" once it has been converted to BT709, so in this case it might be worth doing it manually.


I'm gonna finish a few things and perhaps tweak it a little bit more and then I'm gonna upload it next week.

Cheers,
Frank.

[FranceBB]

When logarithmic curves are life, linear algebra is your passion and you can't start the day without using a matrix of linear transformation :P

https://i.imgur.com/0oUrdz3.jpg

[ChaosKing]

Nice and tasty results :D

[FranceBB]

Over the last few days I've been taking pictures of pretty much everything in LogC.
I'm gonna upload the LUT in a matter of hours, but right now I uploaded a few images here 'cause it's "educative" to show how linear transformations performed by the matrix created (my LUT) are ok, however there's always margin for manual adjustment: the so called "artistic-choice" which cannot be included in a LUT by default as it would produce a good result on some occasions and bad results on some others.

(Examples are in BT709 SDR as the sensor I used is very bad and with a ridiculously low amount of stops).

Example1:

Log-C:
https://i.imgur.com/T2AAJQM.png

Linear Transformation BT709 SDR:
https://i.imgur.com/qVRLFoF.png

Artistic Choice BT709 SDR:
https://i.imgur.com/nE8lgGK.png


Log-C:
https://i.imgur.com/9Hltiqk.png

Linear Transformation BT709 SDR:
https://i.imgur.com/PmAD6FR.png

Artistic Choice BT709 SDR:
https://i.imgur.com/SyauXJi.png


Log-C:
https://i.imgur.com/664H6mJ.png

Linear Transformation BT709 SDR:
https://i.imgur.com/9Fkg42y.png

Artistic Choice BT709 SDR:
https://i.imgur.com/ys19Czx.png


As you can see, the matrix cannot tell whether it's ok to raise the highlights or not 'cause in some occasions it is, in some others it's not, so it just sticks with a rather conservative linear transformation; after all, it is my personal choice to raise certain highlights.
For instance, the cat should be black 'cause it was completely black in the reality, but I wanted to raise the grass a little bit and give a bit more life to the fur at the expense of making it "dirty" a little bit.
Anyway, those are just examples, they are shot with a ridiculously crappy sensor and I literally spent 2 minutes to grade them, however I do believe that they were good examples of what I mean by "artistic adjustment".

This is the LUT I'm gonna upload this evening:

https://i.imgur.com/5iCRvdR.png
https://i.imgur.com/jkSeYRe.png
https://i.imgur.com/Mb1bFhA.png

[FranceBB]

Have you ever noticed that many of the latest music videos have been originally shot in Log? I just noticed that Rita Ora - Lonely Together, Ed Sheeran - Beautiful People and Ellie Goulding - Sixteen were all Log; you can tell by the way they have been graded and brought to BT709 linear with a matrix of linear transformation (LUT).

This is a frame from "Ed Sheeran - Beautiful People":

https://i.imgur.com/TVbfP9c.png

[FranceBB]

Here I am again, ready to play with matrices of linear transformation as I have a little bit more spare time.
A few days ago, a user, Gser (https://forum.doom9.org/member.php?u=148121) received a DCP package in M-JPEG2000 which contained an XYZ 4:4:4 12bit video.
He was trying to convert it to RGB and then YUV to encode it and I got interested in the subject as the very first thing I thought about was "we can do it with a matrix of linear transformation!" :D
Unfortunately, though, I didn't really have time to spend on that as I was both working and studying for my Calculus III exam (which I just passed; hurray! :)).
Anyway, in the mean time another encoder popped up to help Gser (https://forum.doom9.org/member.php?u=148121) and made his own matrix using Davinci Resolve; the encoder who made the matrix is Bradley Greer, founder of Kyotocolor and a very famous colorist who worked for several movie productions and who has been invited to speak on behalf of Kodak, Sony, and Avid multiple times...
I know what you are thinking: "C'mon Frank, I know you like matrices, but can't you just give us a link to his matrix and stop?"
Well, if you just wanna stop reading and you wanna use his LUT, you can download it from this website: Link (http://24p.com/wordpress/wp-content/uploads/2016/11/xyz-709-xyz.zip)
However, there are some interesting findings I made for what follows...
As an engineer who truly loves Linear Algebra and who likes math, I can't stop thinking about matrices of linear transformation and although there was already a matrix made by a really talented encoder, I really wanted to make my own matrix in order to see how far I was going to get, so first things first.
First of all, a DCI it's actually relatively easier to handle as we're gonna make our conversion in linear gamma, specifically from 2.60 to 1.90, as we're gonna have to bring it down to make use of it in our final BT709 target. The second think we need to do of course is to convert the XYZ colorspace. X, Y and Z are extrapolations of RGB created mathematically to avoid negative numbers and are called Tristimulus values: Y means luma (as we all know), Z is somewhat equal to blue, and X is a mix of cone response curves chosen to be orthogonal to the luma and non-negative.
In my attempt to make a decent-looking matrix, I used the monitor I have here at work to test it and I'm not very satisfied with the result as I think that the black level is too low and the white level has been compressed so much to avoid clipping that the final look actually greatly differs from what the original intent was, however it doesn't look bad and I think it can be used in production.

Anyway, before trying to spot the conversion differences, let's take a look at the graph of the curves first:

https://i.imgur.com/3HPo5aU.png
https://i.imgur.com/28Mav1M.png

If you remember how logaritmic curves behave, you can see that they're completely different.
Logaritmic curves generally use to start higher than the linear BT709 one and then they generally curve 'till they get almost straight as if they were y=x+3 in the first quadrant, crossing the linear BT709 curve two times, at the beginning after the non-so-black values are displayed and at the end right when the linear BT709 one gets a correction in order to avoid to get out of range values, therefore applying a sort of soft-clipping, which someone likes to call "knee". In the DCI case, however, there are no points in common whatsoever between the two curves and the DCI one is always above the BT709 one. We kinda expected this as we knew that it was going to be higher, but the interesting thing is that the difference between the two curves isn't constant as we expected in a linear scenario, on the contrary, it varies as there's less difference in the mid-tones than there's in the black-ish tones. As to the white tones, the DCI curve quickly peaks to high values while the BT709 one is smoothed to make sure to avoid out of range values (i.e clipped whites and so on), so it diverges but we expected this and it makes sense.
Alright, let's take a look at the comparison, shall we?


Original XYZ footage 1:

https://i.imgur.com/BgxWxda.png

BT709 Bradley Greer matrix of linear transformation:

https://i.imgur.com/l4RnUzO.png

BT709 FranceBB matrix of linear transformation:

https://i.imgur.com/KJRG26z.png

Original XYZ footage 2:

https://i.imgur.com/xPBe02t.png

BT709 Bradley Greer matrix of linear transformation:

https://i.imgur.com/Nzg3HYr.png

BT709 FranceBB matrix of linear transformation:

https://i.imgur.com/ERSyBwH.png


We can see how the Bradley Greer matrix has both luma and chroma right to the right level and it offers a more natural representation of how the scene should look like. As to the matrix I used, blacks are slightly too low I think and in order to have whites in range I ended up scaling everything down and getting perhaps too smoothed values. I'm gonna post the matrix by tomorrow, so feel free to use it anyway if you prefer it over the Bradley Greer one and you like a darker, more contrast-looking output with kinda saturated colors, or just use the Bradley one as many other people are.
When it comes to these things, it's really up to you and your eyes as both matrices are mathematically correct.

Temporary Link (I'll add it to the main link inside the folder soon): Link (https://wetransfer.com/downloads/3df66d4386679baca613270e8c2945d420190728155059/0c4977)

Cheers,
Frank.

[FranceBB]

A few days ago I was explaining how HLG is useful but it looks different for people who do have a BT2020 SDR capable TV but not an HLG HDR capable one.

I strongly invite you to read it here: Link (https://forum.doom9.org/showthread.php?t=176797)

As a follow up, I decided to do a few more comparisons to show you how it looks different across different TVs.

On the left hand side, you'll find the BT709 SDR footage.
In the middle, you'll find the HLG version as it should look like for people who have an HLG Capable TV.
On the right hand side, you'll find the HLG version as it should look like for people who have a BT2020 SDR capable TV.

https://i.imgur.com/0XEspy6.png
https://i.imgur.com/NMettmD.png
https://i.imgur.com/lMqVJEr.png
https://i.imgur.com/t2JQnRZ.png
https://i.imgur.com/X7wdfb3.png
https://i.imgur.com/wOpP2Ab.png
https://i.imgur.com/SImWPia.png
https://i.imgur.com/KxKJHuZ.png
https://i.imgur.com/y6oLnmT.png
https://i.imgur.com/t9UnEgl.png
https://i.imgur.com/Uu4SVJe.png
https://i.imgur.com/OzkEew4.png
https://i.imgur.com/Ih182V8.png

Link to sample files:

BT709 SDR & BT2020 HLG HDR: https://we.tl/t-cXvmayyONR
Comparison Video (BT709 SDR - HLG BT2020 - BT2020 SDR): https://we.tl/t-NkikWq8AwO

[FranceBB]

Hi there,
it's been a while.
Today I had to encode a few contents shot with a Nikon camera in Z-Log.
I gotta say that I was surprised 'cause I totally ignored the fact that Z-Log even existed, however it seems that it's really a thing and it has been developed around mid-2018 or so.
Anyway, since I didn't really want to get into HDR as I have no flipping clue about how many stops (nits) Nikon cameras have, I decided to go to Linear BT709 SDR 100 nits as target for my matrix of linear transformation.
To be fair, even though I don't know the exact amount of stops that those cameras have, from the tests I made, I don't really think that there are enough stops to get a decent amount of nits that would justify going to HDR (either HLG or PQ), so it's better to stick with BT709 SDR.

I'm about to show you some Z-Log 10bit footages before and after applying my matrix of linear transformation (LUT):

Z-Log Original 1

https://i.imgur.com/zW5r1yu.png

Linear Transformation to BT709 1

https://i.imgur.com/hDa8VXp.png

Z-Log Original 2

https://i.imgur.com/Ml03NR2.png

Linear Transformation to BT709 2

https://i.imgur.com/quH5HF4.png

Z-Log Original 3

https://i.imgur.com/NGPjVOv.png

Linear Transformation to BT709 3

https://i.imgur.com/PbYrWF3.png


I know what you're thinking "you generally plot the input and output curves in a graph and show the differences between the input and the output, why didn't you do it this time?".
Well, mathematical laziness at its finest I say, so I'm asking you to fire up your brain and picture a 2D plane in your mind.

https://i.imgur.com/aGX2DOk.png

You should be familiar by now with the classic BT709 curve, so let's focus now on the Z-Log one.
Well, Z-Log basically has the black level that starts higher than normal, however it quickly goes up with its highlights to the point it doesn't even look like a logarithmic curve anymore at the very top. This is because its main focus is to prioritize blacks and offer a better way to deal with situations that would otherwise be difficult to deal with and would get blacks crashed. So, if you're ever going to shoot in Z-Log, keep in mind that although it does a fine job in retaining the details of the blacks, it won't help you with highlights. In other words, if you are using Z-Log, don't expect to have skies not clipped out and a very wide dynamic range, it's not what it's meant for! The reason for that, I suppose, is that sensors used by Nikon cameras don't have many stops and although they do a relatively fine job with dark areas, they are not able to get bright areas and highlights.

I'm gonna add the .LUT file called "Z-Log_to_BT709.cube" tomorrow.
As a plus, this time it's gonna be compatible not just with Avisynth, but also with ffmpeg.

Cheers,
Frank.

[anatol2002]

Hi Frank
I tried to access the LUT's link, but it did not work for. Is it broken?

Actually I am looking for HLG to PQ conversion. Do u have that LUT?

Thanks,
Anatol

[FranceBB]

Hi Frank
I tried to access the LUT's link, but it did not work for. Is it broken?

Actually I am looking for HLG to PQ conversion. Do u have that LUT?

Thanks,
Anatol

It works fine on my end. https://i.imgur.com/ZqYl5I9.png
Please note that Mega is blocked by some ISP via DNS, so you might wanna try to change your DNS to the google ones or open dns ones or cloudflare ones.

FYI
Google DNS are 8.8.8.8, 8.8.4.4
Cloudflare one is 1.1.1.1

As to the LUTs, I did make a PQ to HLG but I didn't make an HLG to PQ. If you need it, I might take a look at that and make it. Ideally I would make a 1000 nits HLG to 1000 nits PQ, if that's alright. Would you be willing to test it if I send you a LUT?

[anatol2002]

Hi Frank
Thanks for your advice - I succeeded to download the LUT's files from another connection.

And thanks again - I will surely test the HLG to PQ LUT! Much appreciated!

Anatol

[FranceBB]

Ok, perfect. The reason why I use Mega as a host is because it's fast and I have 50 GB of space free of charge, but it's blocked in some countries.

Anyway I'm gonna be out of office 'till Christmas, so I won't be able to work on the LUT (heck I don't even have a 1000 nits capable display at home T_T), but I will as soon as I get back to work.

For those curious, I'm actually eating a delicious half roast chicken with a pig-in-blanket, roast potatoes, gingerbread stuff, red wine jus and of course two Yorkshire puddings.
As you probably figured out, I'm currently in the UK as there's hardly anything more British than a Sunday Roast. https://i.imgur.com/k0N36Cg.jpg

[anatol2002]

Merry Christmas Frank!
Have a great time in UK!

P.S. - is it a roast chicken or broadcast chicken :)

[FranceBB]

Alright, before getting to HLG to PQ, I wanted to review my HDR PQ to HLG matrix and add a new one.
I gotta say though that making a PQ to HLG matrix is a bit tricky 'cause it's NOT an homomorphism 'cause the PQ input is so much wider than the HLG output, so I can't match each and every point of the input to a single point of the output (not 1-1) but every member of the input is mapped to all the output (so it is onto) but definitely not an homomorphism. In other words, you can't go from PQ to HLG without losing something.
A matrix of linear transformation is just... you know... a matrix, which basically means that can map some points of the input to some other points of the output, which is exactly what I'm doing here for PQ to HLG, however PQ can span from 0 to 10'000 nits while HLG can only span 0 to 1000 nits which is far less than PQ. The thing is that I can't make the LUT change on a scene-by-scene basis or anything, it's static, it's just a "mapping", therefore there's no magical matrix that works for any content in the world.
I started with the BBC White Paper as a good reference point for doing just this as I did last time for the other LUT, however I was trying to adapt this in order to be good with any kind of contents with many more nits than 1000 nits without brutally clipping them in my attempt to bring them down to 1000 in HLG, but I wasn't satisfied. The results were poor and my matrix only behaved correctly on not-so-bright contents. Fortunately, I noticed that NHK published a similar reference paper for HDR PQ to HLG which has been translated in English (thanks God as I don't really understand Japanese other than a few words I learned from watching anime in my younger age) and it was far different from the BBC one. After trying a few options, the curve I got has a much greater slope compared to what the BBC states and I gave a lot of importance to the black level. I sacrificed whites, but that's an acceptable compromise; after all, I gotta map everything to just 1000 nits, so I don't really have much choice, but I tried not to brutally clip whites, but to smooth them out a bit instead, although I do also sort of "clip" the very very high ones.

https://i.imgur.com/UKuw5lT.png

As you can see, blacks are preserved, but whites are smoothed out before eventually clipping them.

https://i.imgur.com/em86uxA.png

If you take a look at the nits of this HDR PQ test image, for instance, you can see that there's a lot in the low area with very low nits: the background is at about 200 nits, the sky is pretty low as it's evening so it's slightly over 500 nits at like 520 nits and the skin is between 240 to 500 nits, depending on the part of the face we're actually analyzing as some parts gets more photons from the light emitting source. Lastly, the power source peaks at 1000 nits and it's therefore clipped out as the PQ test file I used has just 1000 nits. Please note that the picture you're seeing is a PQ on an HDR Waveform Scope, which basically means that you're already seeing how it's supposed to look like, but for the sake of references, here's how you would see it if you open the raw pixel file without taking into account the curve (i.e NOT how is supposed to look):

https://i.imgur.com/DP7nBaG.png

From here we can see that the windows at the center pops up as it peaks at 1000 nits as well as each and every "candle" (heck, there are no candles anymore, just AC light bulbs getting a mono-phase current and emitting photons :( ).
Walls, chairs made out of wood and other things are at about 400 nits and so on.

https://i.imgur.com/IF78vur.png


Alright, enough waveforms, now I'm gonna post a few screenshot comparisons about HDR PQ (on the left) and HDR HLG (on the right).
As you can see, I couldn't quite match the same tones of the input, but I preserved as many things as possible while mapping.
I gotta say that all the test files I used have about 1300 nits but nothing more, so it was relatively easier to re-map to 1000 nits, but it should be fine for contents slightly higher than 1300 nits (although I didn't test it).

HDR PQ (left) vs HDR HLG (right):

https://i.imgur.com/8pADg58.jpg
https://i.imgur.com/u8wegI2.jpg
https://i.imgur.com/NKP1KIc.jpg
https://i.imgur.com/OIkrqHw.jpg
https://i.imgur.com/Eqq2gZm.jpg
https://i.imgur.com/SyJuLnV.jpg
https://i.imgur.com/FP9WMMk.jpg
https://i.imgur.com/20cLkOH.png
https://i.imgur.com/r6b2Wiw.png
https://i.imgur.com/MlYafZP.jpg
https://i.imgur.com/QBfbkpx.jpg
https://i.imgur.com/FwYPh9p.jpg
https://i.imgur.com/EqiSrN3.jpg
https://i.imgur.com/OwxpF4p.jpg
https://i.imgur.com/WPl3Ky7.jpg

[FranceBB]

https://i.imgur.com/NVFFakh.jpg
https://i.imgur.com/ISwHWj6.jpg
https://i.imgur.com/ByNJKiX.jpg
https://i.imgur.com/CrwDhOm.jpg
https://i.imgur.com/eKwPjqe.jpg
https://i.imgur.com/V42Q9l7.jpg
https://i.imgur.com/Ytrcldb.jpg


I added it to the list with the name "PQ_to_HLG_Nspec.cube" which stands for "NHK specifications based" + of course my modifications.
Now I'm finally ready to work on your LUT, which is the inverse: from HLG to PQ, which should be fairly easier as this time I can really map each and every point of the input space (HLG) to each and every point of the output space (PQ) without losing quality as HLG is less wide than PQ.
I decided to set them to scale in a linear fashion: in other words, if you have an HLG at 250 nits you'll get a PQ at 250 nits, if you have an HLG at 400 nits you'll get a PQ at 400 nits and if you have an HLG of 1000 nits you'll get a PQ at 1000 nits.
One of the common mistakes people make is to map the maximum of a certain input curve to the maximum of the output curve, 'cause generally when you're dealing with matrices of linear transformation and you have an input space and an output space, you tend to do it that way.
Think about BT709 to HLG: many people think that the correct way to do it is to map the whole linear BT709 to HLG thus getting the peak brightness of BT709 SDR to 1000 nits in HLG HDR, but that's a very doggy approach. As a matter of fact, in HLG to PQ I've done the same: they're both mapped in a linear fashioned so that instead of having the peak HLG be mapped to the peak PQ, they're mapped to get equal results on the output; in other words, instead of mapping the 1000 nits HLG to 10'000 nits PQ, I mapped it to 1000 nits PQ.
That's the way it should be done anyway in my opinion.
I'm doing it in this very moment, it should be ready soon!



Merry Christmas Frank!
Have a great time in UK!

P.S. - is it a roast chicken or broadcast chicken :)

I'm afraid that the canteen we have at work is way worse. :(

[FranceBB]

Alright, your LUT is ready now.
I mapped it out to 1000 nits in PQ in order to have the same amount of nits in both the resulting HDR, rather than remapping the maximum of HLG to the maximum of PQ.
This is the resulting curve:

https://i.imgur.com/p5FStgB.png
https://i.imgur.com/rJ0x8dP.png
https://i.imgur.com/NDsixAP.png

I know what you're thinking "hold on a second! You said that you were going to map it to 1000 nits but the resulting PQ curve is left unclipped 'till 10000 nits, why is that?"
Well, that's not gonna affect the results at all.
Yes, it's true, I left the curve unclipped 'till 10'000 nits, but since I mapped every point of the HLG to its reciprocal of PQ 'till 1000 nits, those other 9000 nits that are there left are NOT gonna be used at all.
I.e there isn't going to be any data in there.
As an example ('cause mine are just words), I'm gonna show you a frame.



This is how the waveform would look like if I clip the resulting curve of the matrix to 1000 nits by forcing it to be 1000 nits directly from the matrix:

https://i.imgur.com/msHqcM8.png


Now, look at what happens when I use the very same curve but I leave it unclipped to 10'000 nits:

https://i.imgur.com/AwRNkS8.png


Nothing, exactly. This is because although I left the curve unclipped to 10'000 nits, I mapped the resulting points in a way that NOTHING is gonna get mapped over 1000 nits.

Now, for the sake of knowledge, this is how it would be if I had to map the maximum of the HLG to the maximum of the PQ (i.e 10'000 nits):

https://i.imgur.com/Oa3uBrD.png

See the curve? See the waveform? Now I have a completely different slope and a completely different peak as points that were lower will be mapped to an higher point, but I don't really like it, I think it's wrong, 'cause essentially you're making something out of nowhere for apparently no reason at all, so...
It would be like when I see broadcasters mapping the 100 nits of the linear BT709 to the maximum of the HLG so that I get a "nice" 1000 nits of white and completely clipped out sky -.-
So, that's why I decided to take this other route.
I didn't actually manage to get it 100% the same, but I think I'm fairly close and it's gonna be fine for your needs.

A few screenshots (on the left the original HDR HLG 1000 nits and on the right the HDR PQ 1000 nits output from the matrix of linear transformation):

https://i.imgur.com/V2253FY.jpg
https://i.imgur.com/RbXFxGk.png
https://i.imgur.com/XC22HHi.jpg
https://i.imgur.com/uO7GEk2.png
https://i.imgur.com/GzPQLLW.png
https://i.imgur.com/VuuSHkl.png
https://i.imgur.com/ucWEDio.jpg
https://i.imgur.com/1Hc2frs.jpg
https://i.imgur.com/jeJQhAG.jpg
https://i.imgur.com/ckc4ZmH.jpg


As you can see, the very low dark levels are slightly different, but that it's really gonna depend on how low your monitor can go, so it's gonna be tricky to check, especially if you have an LCD, while you'll be perfectly able to see it if you have an OLED. This is because in HLG the low end of the curve resembles a sort of linear SDR one and the mid-high end part is more log-prone (so more inclined to the HDR), hence the reason why it's called Hybrid Log Gamma. When it's mapped to PQ, though, which is purely meant for HDR, the low end part is gonna be mapped to several completely different points that are way different from the HLG one and if your monitor isn't very good at displaying low lights it may seem as if they're not there, but really they are, 'cause I doubled checked with my waveform monitor. This is particularly "funny" 'cause I can see details in the shadows in the HLG one that get mapped to points in PQ that make them invisible to my monitor at home but they're still there and I can see them in the waveform (and I probably will be able to see them at work in one of the monitor we have there).

You should find the new LUT in the folder (Mega) in the afternoon.
I look forward for your reply.

Cheers,
Frank.

[anatol2002]

Hi Frank
Thanks for prompt response!
Unfortunately I did not time to get to test your new LUT. Just overloaded with lots of other stuff.
Hope that will get some time in couple of days.

Happy New Year!

Anatol

[FranceBB]

Unfortunately I did not time to get to test your new LUT. Just overloaded with lots of other stuff.
Hope that will get some time in couple of days.

Happy New Year!

Sure, no problem, happy new year! :)

[FranceBB]

Alright, so I've come across some Panasonic Cameras that shoot in V-Log.
I gotta say that I do not like Panasonic Cameras nor V-Log, however I had to quickly do something to get something out of this.
Please note that this is a very rough linear transformation from V-Log to Linear BT709 SDR.
Unfortunately although I've got the Luma right, I don't quite seem to understand the internal color science of Panasonic that much, that's why I didn't quite get the tone I wanted, anyway if you wanna use it feel free to use it (it will be added to the link in the first page soon). Temporary Link: Link (https://we.tl/t-YFng3JQUUY)

https://i.imgur.com/d3KOzw4.png
https://i.imgur.com/a92vFLj.png
https://i.imgur.com/KQisM8j.png
https://i.imgur.com/43E9jdJ.png
https://i.imgur.com/1vH8xd8.png
https://i.imgur.com/BijYyVt.png
https://i.imgur.com/z0s9wtl.png
https://i.imgur.com/r8p7dLp.png
https://i.imgur.com/q09TkaK.png

[hydra3333]

thank you

[ClubM]

Just wanted to say thank you FranceBB for sharing these!

[FranceBB]

New day, new matrix of linear transformation.
This matrix has been inspired by something that hello_hello was talking about here (https://forum.doom9.org/showthread.php?p=1897823&posted=1#post1897823) which is basically going from BT2100 HDR PQ not to BT709 SDR but rather to BT2020 SDR. Although, generally, whenever we have PQ contents we go to HLG to keep HDR compatibility as much as possible up to 1000 nits, however the more it gets closer to the original HDR source, the more it gets dull on BT2020 SDR monitors/TVs, so it does actually make sense to have such a matrix as it can come in handy.
So, the idea behind this is to make use of the wide BT2020 but limiting the nits to get an SDR material. In other words, this content will look far better on UHD SDR TV that can handle 4K but that don't interpret any curve and would just ignore information about HLG in an HLG source and would interpret the matrix correctly.

Since we're dealing with BT2100 PQ and our output is going to be a BT2020 it's very strongly suggested to work with 16bit and output either a 12bit or a 10bit file also 'cause 8bit BT2020 is not officially supported (although Sony likes to make weird files and I've seen cameras being able to encode an 8bit file with BT2020 which is a mediocre compromise).

Alright, enough talking, let's see the curve!
If you take a look at it, it basically bring an HDR BT2100 PQ and maps it in order to get a gamma 1.90 (which is something we expect from an SDR source) BUT making use of the wide BT2020 matrix.
The knee at the very top is to avoid hard clipping and its slope is kinda high 'cause the amount of nits we have in our output is gonna be extremely low compared to the input, so our linear transformation is "onto" which means that it maps different points of the input to a single point of the output. If we call our input PQ "X" and our output Linear BT2020 SDR gamma 1.90 "Y", then the function is "onto" because every element in the codomain is the value of f(x) for at least one element x in the domain.

https://i.imgur.com/XFlmosJ.png
https://i.imgur.com/hr9EFjF.png
https://i.imgur.com/mMnTvMJ.png

Screenshots are gonna be a little bit tricky though 'cause I have nothing to make you visualize them correctly if you have an old monitor which is not compatible with BT2020 SDR as I'm going to compare BT2100 HDR PQ with BT2020 SDR, none of which is supported by old monitors.



BT2100 HDR PQ (Top)
Linear BT2020 SDR (Bottom)

https://i.imgur.com/S8pJItb.png
https://i.imgur.com/jQHYj7A.png
https://i.imgur.com/FVHOeYz.png
https://i.imgur.com/pUAUNkb.png
https://i.imgur.com/XjCFmjT.png
https://i.imgur.com/DaruT1D.png
https://i.imgur.com/UhO0JTR.png
https://i.imgur.com/FjIceXn.png


Temporary Link to the LUT for testing (it will be added to the main repository soon): Link (https://we.tl/t-lxprs0qH6X)

[DavidK_]

Frank!

I am helping Anatol on this conversion issue and tested your HLG to PQ lut. Very close - nice work - just had a couple of notes/adjustments if possible.

We're trying to get FFMPEG to match a source file and an output from an non-FFMPEG encoder. I measured the source, the version from an adobe media encoder and your lut via ffmpeg. Here are some of the results/observations comparing the three outputs:

1) Histogram:
Source MXF: RGB values: 7.774 6.701 7.267
Adobe Media Encoder H265: RGB values: 5.829 5.803 6.125
FFMPEG HLG TO PQ LUT: RGB values: 1.554 1.459 1.512
It does feel too red still and you'll notice that yours has just slightly elevated Rs instead of G and B with Blue being too low compared to Source and Media Encoder.

2) RGB peaks stop well below what the source and Media Encoder. Yours are at around 80/.8 whereas the source and Adobe version hit 100/1.

3) It feels too contrasty and darker with the LUT compared to what is expected. But just a little.

[FranceBB]

Frank!

I am helping Anatol on this conversion issue and tested your HLG to PQ lut. Very close - nice work - just had a couple of notes/adjustments if possible.

We're trying to get FFMPEG to match a source file and an output from an non-FFMPEG encoder. I measured the source, the version from an adobe media encoder and your lut via ffmpeg. Here are some of the results/observations comparing the three outputs:

1) Histogram:
Source MXF: RGB values: 7.774 6.701 7.267
Adobe Media Encoder H265: RGB values: 5.829 5.803 6.125
FFMPEG HLG TO PQ LUT: RGB values: 1.554 1.459 1.512
It does feel too red still and you'll notice that yours has just slightly elevated Rs instead of G and B with Blue being too low compared to Source and Media Encoder.

2) RGB peaks stop well below what the source and Media Encoder. Yours are at around 80/.8 whereas the source and Adobe version hit 100/1.

3) It feels too contrasty and darker with the LUT compared to what is expected. But just a little.

A feedback is always welcome 'cause other people will always test more scenarios than I do on my own. I gotta say that I did notice a light dominance in red but that probably was because of my test source, so although the intent was to match it as much as possible, I think I "fixed" it when I was creating it so that the output was going to be ok, however that probably ended up with me matching the wrong points; the blue makes sense as it's the one that got compensated in favor of red. Anyway, I'll try to make another one but it would help to have a very short sample if you can provide one so that I can send you multiple outputs.

[PatchWorKs]

Hi there, just a question: do you think I need a "color conversion" lut for Canon HF100' shootings ?

I guess it *should* record in 4:2:0 8-bit AVCHD (.mts) Canon RGB...

[FranceBB]

Hi there, just a question: do you think I need a "color conversion" lut for Canon HF100' shootings ?

I guess it *should* record in 4:2:0 8-bit AVCHD (.mts) Canon RGB...

Uhm... so it's basically an H.264 yv12 (4:2:0 planar) 8bit muxed in an .mts container, but that tells me nothing, honestly.
I mean, I tried to google it up and it doesn't seem to be able to record in any logarithmic curve, so my guess is that it actually records in Linear BT709. If that's the case, then no, you don't need a matrix of linear transformation, however I don't really know your model. Just scroll through the settings and check whether it says anything about "C-Log" (since it's Canon) or HLG. If it doesn't, then you're probably fine and it should say somewhere "BT.709". If you're uncertain, just record a short sample and upload it here via WeTransfer.

[PatchWorKs]

Ok, i did some searches too and found this interesting article:
Video editing for scientific analysis (http://nerdfever.com/video-editing-for-scientific-analysis/)

That claims:
Color spaces – Studio RGB vs. Computer RGB vs. Canon RGB

One thing you may notice when using Vegas or viewing the output files after processing in Vegas is that the video looks “washed out” or too dark or contrasty. These problems come from color space conversion problems. Different video players use different color spaces, so the same video file might look fine in one player, and bad in another.

The brightness and color of pixels displayed on your computer screen is determined by three numbers, for the brightness of Red, Green, and Blue in each pixel. Each number can range from 0 (darkest) to 255 (brightest). If Red, Green, and Blue (RGB) are all 255, they combine to make bright white. This is the “Computer RGB” color space – each color has the range from 0 to 255.

The “Studio RGB” color space uses only the values from 16 to 235 to represent the RGB brightness. Values in the range of 0 to 15 are called “superblacks” and values 236-255 are “superwhites”. Standard televisions and video cameras are supposed to use the Studio RGB system. On a TV display, superblacks are all displayed as the same color of absolute black, and superwhites are all displayed as the same shade of whitest white, so 16 is as black as you can get, and 235 is as white as you can get.

But this is not true in the Computer RGB system. 16 is not pure black – it’s dark, but 0 is a lot darker. And 235 is not as bright as the computer display can get – full brightness comes at 255. So if you show a Studio RGB video (16-235) on a computer (0-255), it will look washed out, with low contrast. The blacks won’t be very black, and the whites will be grayish. And vice-versa; if you show a Computer RGB video on a TV, there will be too much contrast, with all the dark details (0 to 15) lost as black, and all the white highlights (236-255) “blown out” to pure white.

So a color space correction is needed when converting from one system to the other. Sony Vegas has the “Sony Color Corrector” video effect that you can use to do this conversion – it has presets for converting Computer to Studio RGB and another for the other direction.

But. If you look at the video files the Canon HF100 generates, you’ll find it uses the values 16-255, instead of either of the two standard systems. If you treat this as Studio RGB, the video will lose the bright highlights (236-255). If you treat it as Computer RGB, the black levels are wrong (16 instead of 0).

You can fix this by converting Canon RGB to Computer RGB (assuming you’ll view on a computer).

In the Sony Color Corrector, set Gain to 1.067 (this is (17+ 255)/255), and set an Offset of -17. The Color Corrector first multiplies all the color values by the Gain setting, so 16-255 becomes 17-272. Then it adds the Offset to all the values, so the range 17-272 becomes 0-255. I saved this setting as “Canon HFxxx to Comptuer RGB”.

...i still don't know if a lut is needed, but according to the article's author a "color shift" would be useful.

[FranceBB]

Ok so the first part is just talking about Limited TV Range and Full Range.
For 8bit:

Limited TV Range is 16-235 (0.0-0.7V)
Full PC Range is 0-255.

Your camera is behaving like some other Canon camera do while shooting in Linear BT709 8bit: they're set in Limited TV Range 16-235 however they leave the highlights unclipped so that they can go as high as 255.
In their mind, it's the cameraman that has to look at the waveform monitor and make sure that it doesn't overshoot and it's at the sweet spot instead (235).
This happens on professional productions while shooting live sports events as well, which is why I generally put a hard clipping to 16-235 at the very end of the chain just to be sure in case the cameraman goes mad and a player passes through a very strong light almost straight into the camera.
In your case, I would advise you to check your waveform monitor (if the camera has one built-in) and then make sure that your highlights never exceed 235 as you're gonna lose them.
Once in post-production, you can do a soft-clipping to bring what exceeded down to 235 and just be safe.
I don't really use Sony Vegas as NLE, I use AVID Media Composer instead (and sometimes Davinci Resolve for color-correction and masks), so I can't tell you how to do soft-clipping there, but if there's something called "Broadcast Safe" or "Broadcast colors" then go for it.
On Avisynth there are two ways of doing it as well:



#Indexing
FFMpegSource2("whatever.mts", atrack=-1)

#Clipping
Limiter(min_luma=16, max_luma=235, min_chroma=16, max_chroma=240)


OR


#Indexing
FFMpegSource2("whatever.mts", atrack=-1)

#Canon Levels to Limited TV Range
Levels(0, 16, 255, 16, 235, coring=false)


The latter will assume that you shot on Canon Levels 16-255 and that your white is left unclipped to 255, so it keeps the black level at 16 and lowers down all the white values from 255 to 235. It should work.
Please keep in mind that the reason why I'm suggesting you that you work in Limited TV Range instead of using the Full PC Range 0-255 by just lowering the black of your Canon from 16 to 0 is that although you can encode and flag a stream with Full Range, it would be correctly reproduced by a bunch of computers only and a very little number of TVs will be able to correctly reproduce it; besides, not even ALL the players for computer will read the flag correctly and respect it, so my suggestion is to play safe and go for Limited TV Range 16-235.

This is an example of what I mean:

https://i1.creativecow.net/u/19614/0_levels_vs_broadcast_colors.png

As you can see, clipping brutally "cuts" everything above 235 (which in that scale is 100% luma) while Levels just scales it down.

Oh, as a final remark, you don't really need a LUT for that; a LUT is a matrix of linear transformation, it maps a space into another and - in this case - points of a curve into points of another curve. In your case it would be from Linear BT709 with highs unclipped to Linear BT709 in limited TV Range, so it shouldn't really be needed considering that there are tons of other tools available for video levels that do a way better job. Anyway, just out of curiosity, a matrix of linear transformation from out of range Linear BT709 values to legal Linear BT709 values would look something like this (where 100% in this case means 255):

https://i.imgur.com/SPIRQ0B.png
https://i.imgur.com/CFbnkZD.png




Please note that there are tons of topics about this subject here on Doom9 if you're curious. ;)

I hope this clarifies your doubts.


Cheers,
Frank.

[DavidK_]

A feedback is always welcome 'cause other people will always test more scenarios than I do on my own. I gotta say that I did notice a light dominance in red but that probably was because of my test source, so although the intent was to match it as much as possible, I think I "fixed" it when I was creating it so that the output was going to be ok, however that probably ended up with me matching the wrong points; the blue makes sense as it's the one that got compensated in favor of red. Anyway, I'll try to make another one but it would help to have a very short sample if you can provide one so that I can send you multiple outputs.



Okay I think you're right - we should reference something together. Here is a link to the file I used:

https://4kmedia.org/lg-new-york-hdr-uhd-4k-demo/

I'll reply back tomorrow with settings that I'm trying to match. And thank you!!!! You're insight has been very enlightening.

[DavidK_]

A feedback is always welcome 'cause other people will always test more scenarios than I do on my own. I gotta say that I did notice a light dominance in red but that probably was because of my test source, so although the intent was to match it as much as possible, I think I "fixed" it when I was creating it so that the output was going to be ok, however that probably ended up with me matching the wrong points; the blue makes sense as it's the one that got compensated in favor of red. Anyway, I'll try to make another one but it would help to have a very short sample if you can provide one so that I can send you multiple outputs.

Okay I think you're right - we should reference something together. Here is a link to the file I used:

https://4kmedia.org/lg-new-york-hdr-uhd-4k-demo/

I'll reply back tomorrow with settings that I'm trying to match. And thank you!!!! You're insight has been very enlightening.



And here are the settings to match:
LG Sample levels to match:
Source Timecode: 00:00:01:00
Source: R .802 G .672 B .795
Adobe Match: R 1.472 G 1.202 B 1.334

At least keep those proportions. And keep black levels a bit lower - the LUT feels a bit too contrasty/dark/inky I thought. Your lut currently is still too red - your lut levels were:R 2.877 G 2.654 B 2.56 - notice your's is too green and too red. Red and Blue should be more even etc.