Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th February 2019, 19:15   #1  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
FranceBB LUT Collection

Hi,
LUTs are generally used to grade contents, but they can be useful to do quick and dirt color space conversions, especially in this day and age.
Although a good color conversion has to be achieved manually and/or via algorithms, LUTs can still be helpful; unfortunately, I've seen people selling their LUTs for simple color conversion and it incredibly pissed me off 'cause they are supposed to be free, so today I'm gonna share my LUT collection, hoping that it can be useful to someone.

Before you apply any of my LUTs, remember that although you can apply them at any bit depth, I strongly suggest you to do that with 16bit precision, otherwise you'll get banding and other terrible issues.
LUTs are 65x65x65, which is supposed to be high precision, but some of them are 33x33x33 which is just about right. None of them are 17x17x17 'cause I don't consider it to be accurate enough.

LUTs: Link

How to use my LUTs in Avisynth:

Code:
FFVideoSource("example.mxf")

ConvertBits(16)

ConvertToPlanarRGB()

Cube("C:\Programmi\AviSynth+\LUTs\example.cube")
How to use my LUTs in FFMpeg:

Code:
ffmpeg.exe -i "source.m2ts" -vf lut3d='example.cube' -c:v whatever -c:a whatever -f mkv "output.mkv"

Alternatively, you can use my plugin "Linear Transformation": Link

Introducing LinearTransformation.

With the increasing number of matrices to do any kind of linear transformations, I kinda felt the need to make something more accessible for users, which is why I'm introducing now "LinearTransformation".
This plugin performs Linear Transformations based on a matrix conversion. Matrices are made by me and can be used independently from Avisynth as well.
The function has two parameters: Input and Output.
Currently, only those linear transformations are supported:

LinearTransformation(Input="Linear_BT601_NTSC", Output="Linear_BT709")
LinearTransformation(Input="Linear_BT601_PAL", Output="Linear_BT709")
LinearTransformation(Input="Linear_BT709", Output="Linear_BT601_NTSC")
LinearTransformation(Input="Linear_BT709", Output="Linear_BT601_PAL")
LinearTransformation(Input="Linear_BT709", Output="BT2020_HLG")
LinearTransformation(Input="Linear_BT709", Output="BT2100_PQ")
LinearTransformation(Input="Linear_BT709", Output="DCI_XYZ")
LinearTransformation(Input="Linear_BT709", Output="ZLog")
LinearTransformation(Input="BT2100_PQ", Output="Linear_BT2020")
LinearTransformation(Input="BT2100_PQ", Output="Linear_BT709")
LinearTransformation(Input="BT2100_PQ", Output="BT2020_HLG")
LinearTransformation(Input="CLog3", Output="Linear_BT709")
LinearTransformation(Input="CLog3", Output="BT2020_HLG")
LinearTransformation(Input="CLog3", Output="BT2100_PQ")
LinearTransformation(Input="SLog2", Output="Linear_BT709")
LinearTransformation(Input="SLog3", Output="Linear_BT709")
LinearTransformation(Input="BT2020_HLG", Output="Linear_BT709")
LinearTransformation(Input="BT2020_HLG", Output="BT2100_PQ")
LinearTransformation(Input="DCI_XYZ", Output="Linear_BT709")
LinearTransformation(Input="LogC", Output="Linear_BT709")
LinearTransformation(Input="VLog", Output="Linear_BT709")
LinearTransformation(Input="ZLog", Output="Linear_BT709")
LinearTransformation(Input="ZLog2", Output="Linear_BT709")
LinearTransformation(Input="dvhe0509", Output="Linear_BT709")

A Linear Transformation is essentially a matrix that maps all points of a certain space to another, which includes of course points belonging to a certain curve to other in order to get a different curve.
Of course, a linear transformation can be used in encoding to map some values to some other values and therefore have conversions from curves like PQ to HLG and so on.

The transformation is performed with 16bit precision, which means that if your input source is lower, let's say, 8bit planar yv12, it will be brought to 16bit planar RGB internally, the linear transformation will be applied with 16bit planar precision and then the result will be brought down to 8bit planar yv12.

Planar RGB 16bit is strongly suggested as your source as it's gonna be faster, in fact 4:2:0, 4:2:2, 4:4:4 planar up to 16bit will be converted back and forth internally.

Inside the plugin, the path specified for the matrices is "mypath" and by default is: mypath = "C:\Program Files (x86)\AviSynth+\LUTs" which means that it's gonna look for my LUTs in a folder in such a location. It's not mandatory to have my LUTs there, you can have them in any location you want, provided that you do update that string.

A huge thanks to StainlessS and to the whole community for greatly improving the script and helping me writing it.

Plugin: https://github.com/FranceBB/LinearTr...formation.avsi
Matrices: https://github.com/FranceBB/LinearTransformation

Usage Example:

Example 1: From Slog3 to Linear BT709

Code:
FFVideoSource("mySlog3source.mxf")
LinearTransformation(Input="SLog3", Output="Linear_BT709")
Example 2: From PQ to HLG

Code:
FFVideoSource("myPQsource.mxf")
LinearTransformation(Input="BT2100_PQ", Output="BT2020_HLG")

Video:



Images:

Slog3 pic1:


BT709 pic1:


-

Slog3 pic2:


BT709 pic2:


-

Slog3 pic3:


BT709 pic3:


Slog3 to BT709


Who made those LUTs?
Those matrices are made by Francesco Bucciantini (FranceBB), computer science engineer and Linear Algebra lover and Livio Aloja (aligia), former Senior Sky Editor and Encoder, using both free open source tools and closed source ones and cross-checking them with broadcast grade equipment (TVs, waveform monitors, scopes) provided by Sony, Canon and Tektronix.

Who uses those LUTs?
Those LUTs are used by everyone for free, from companies that deal with broadcasting video content in any way (Post-production, OTT, etc) like public and private broadcasters all around the world to regular people that want to convert their video archives.

Mentions in other programs:
These LUTs are officially shipped with
- FFASTrans - FFMpeg Avisynth Transcoder by Steinar Apalnes, Benjamin Dissoubret and Francesco Bucciantini
- Hybrid by Selur

Compatibility with third party programs:
- Compatible with Avisynth+
- Compatible with VapourSynth
- Compatible with AVID Media Composer
- Compatible with Davinci Resolve
- Compatible with Adobe Premiere
- Compatible with Colorfront Transkoder

Last edited by FranceBB; 25th February 2022 at 21:24.
FranceBB is offline   Reply With Quote
Old 8th February 2019, 23:24   #2  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Posts: 1,213
Thank you for this.
One question
Should it be fulldepth=true or false
since false would return 8 bit I believe
gonca is offline   Reply With Quote
Old 9th February 2019, 04:12   #3  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
Originally Posted by gonca View Post
Should it be fulldepth=true or false
since false would return 8 bit I believe
Nope, they both return 16bit, but fullrange=true is for full range (PC Range) sources, while fullrange=false is for limited range (TV Range) sources.
Limited range for 8bit is 16-235, for 10bit is 64-940 and so on.
Full range for 8bit is 0-255, for 10bit is 0-1020 and so on.
It depends on your source, but keep in mind that the majority of sources are TV Range and if you encode something for the TV, it has to be Limited TV Range.

Last edited by FranceBB; 9th February 2019 at 16:40.
FranceBB is offline   Reply With Quote
Old 9th February 2019, 12:37   #4  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Posts: 1,213
My mistake
I originally read fulldepth in your sample script, but after re reading it I realize that it actually says fullrange
gonca is offline   Reply With Quote
Old 9th February 2019, 14:56   #5  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
Full range for 8bit is 0-255, for 10bit is 0-1024 and so on.
Nit Pick, 1023.

EDIT: Hmmm, I'm wrong too.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 9th February 2019 at 18:22.
StainlessS is offline   Reply With Quote
Old 9th February 2019, 16:44   #6  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Sorry about that.
Actually, this is the right one, I think (from the Avisynth Wiki):

FranceBB is offline   Reply With Quote
Old 9th February 2019, 20:48   #7  |  Link
huhn
Registered User
 
Join Date: Oct 2012
Posts: 7,890
these "issues" come from the understanding that 10 bit is 4 times as big as 8 bit so we just have to multiply by 4.

8 bit RGB 255 255 255 as 10 bit is 1023 1023 1023 not 1020 1020 1020 has a very slightly lower brightness and is just 4x the original number.

the correct way to increase 8 bit to 10 bit is taking the first 2 bits from the 8 bit source and add them to the end.

or as an small example:

11111111=255 100 % brightness
1111111100=1020 ~99.7% brightness
1111111111= 1023 100% brightness
huhn is online now   Reply With Quote
Old 9th February 2019, 21:41   #8  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,492
Quote:
Originally Posted by huhn View Post
the correct way
But according to whom? The wiki seems to indicate the other way - simple shifting - is correct. What's the broadcast standard?

And what's the correct way to go back down? Is there bit-ty way to do it, or do you have to divide by 4.0117647059?

And I thought 16-235 was a headache...
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 9th February 2019, 22:02   #9  |  Link
huhn
Registered User
 
Join Date: Oct 2012
Posts: 7,890
i just ask you a different question who stops you from creating a native 10 bit source with 1023 in it?

a bad way to go back to 8 bit is by removing the last too bits truncation. another one would be dithering.
huhn is online now   Reply With Quote
Old 9th February 2019, 22:19   #10  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,492
Quote:
i just ask you a different question who stops you from creating a native 10 bit source with 1023 in it?
Umm, okay, but my questions weren't rhetorical. Nothing stops you from creating a native 8 bit source with 255 in it, but that is - according to broadcast standards - an "illegal" value.

Quote:
a bad way to go back to 8 bit is by removing the last too bits truncation. another one would be dithering.
Dithering still needs a convention as to what values, integer of fractional, correspond between the bit depths.
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 9th February 2019, 22:39   #11  |  Link
huhn
Registered User
 
Join Date: Oct 2012
Posts: 7,890
Quote:
Originally Posted by wonkey_monkey View Post
Umm, okay, but my questions weren't rhetorical. Nothing stops you from creating a native 8 bit source with 255 in it, but that is - according to broadcast standards - an "illegal" value.
full range is a thing. computer games work in it it's part of display port and HDMI.

while BD source is all ways limited range this doesn't change that even full range encoding is a thing and totally supported by programs like x264.

and be aware that this is about RGB not YCbCr.
so clearly not illegal
Quote:
Dithering still needs a convention as to what values, integer of fractional, correspond between the bit depths.
does it? how should i even know what an dithering algorithm is working internal. i know it using the "errors" truncated parts to spread noise if it is max 1023 or 1020 shouldn't matter. it will just use what it has available.
huhn is online now   Reply With Quote
Old 9th February 2019, 22:49   #12  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,492
Okay, we're getting off-track. The original point is: the Avisynth wiki says 8-bit 255 is equivalent to 10-bit 1020. You say it's 1023. Who's right?
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 25th February 2019, 22:30   #13  |  Link
silverwing
Registered User
 
Join Date: Nov 2017
Location: Russia, Nizhny Novgorod
Posts: 25
FranceBB, thank you!
So.
These are examples with your lut "PQ_to_HLG.cube".
I get a similar result only with the help of a long selection of curves in the plugin "SmoothCurve". Good work.
Attached Images
  
silverwing is offline   Reply With Quote
Old 19th March 2019, 05:59   #14  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
FranceBB, thank you!
So.
These are examples with your lut "PQ_to_HLG.cube".
I get a similar result only with the help of a long selection of curves in the plugin "SmoothCurve". Good work.
Thanks, I tried to tweak things correctly.


Update: Added Slog3 to BT709 LUT

Whenever I work for the News channel or for our Sport channel, I receive footages shot normally, but whenever I work on TV Series, I receive footages shot in Slog3 and I always have to bring them to BT709 to encode everything in a standard old-fashioned FULL HD SDR.
I'm gonna share with you my Slog3 to BT709 LUT as well.



Slog3 first, BT709 second:







Some people may think that it's ok-ish, but slightly dark on some low-light scenes, but actually this is because I added a knee to the curve and there's a reason for that: trying to stay in the Limited Tv Range avoiding clipping.
Let's take a look at this example here with a strong light coming from behind.
Raw footage in Slog3:



If I try to apply a transformation without using a knee, the white level gets too high and I get out of legal range, therefore everything over 235 gets clipped out:



By adding a knee, however, I manage to get thinks right in 16-235 without the need to clip so many details out:



If you take a look at the curve of the transformation you can see how the unclipped one gets so high that it almost looks like a function that reaches a vertical asymptote:



While the knee manages to smooth things out and generates a proper curve:


Last edited by FranceBB; 24th March 2019 at 05:31.
FranceBB is offline   Reply With Quote
Old 24th April 2019, 03:41   #15  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Update: Added C-Log3 to BT709 SDR / C-Log3 to PQ HDR / C-Log3 to HLG HDR

I added three new LUTs that are meant to be used by Canon C-300 Camera users.
It's a 4K professional camera that can record up to 12bit with the C-Log3 curve and it can definitely be used for HDR productions and SDR ones alike.
What's interesting about this curve with the C-300 is that it's possible to shoot and then grade it to HDR PQ and get approximately 3500 nits worth of data (I mean, real data).
When I created the LUTs, I tried to keep the BT709 SDR one in range and I applied a knee to get the right values.
As to the HDR PQ one, I left it "unclipped" to 10'000 nits, however you're not gonna get anything over 3500 nits. The reason why I left it unclipped up to 10'000 nits, though, is that Canon might release new cameras in the future that are gonna use the same curve and perhaps they'll be able to get more than 3500 nits worth of data.
As to the HDR HLG one, I had to make a very big compromise, 'cause as you know, HLG has been invented in order to display both HDR and SDR using the same file, which basically means that whenever you make a LUT you have two choices: 1) make it more HDR-like and sacrifice the view of the ones with an SDR TV or 2) make it more SDR-like and sacrifice the view of the ones with an HDR TV.
Since many people still have SDR TVs, my choice (as well as the one of other encoders working in broadcast) is to make it more SDR-like, 'cause this way no one is ever gonna complain.
In order to do so, I had to sacrifice the representation of the black values and of course since HLG is limited to 1'000 nits, I had to lower down the highlights as well.
The result looks good and it's definitely more natural than the SDR, but comparing it to the PQ is... like... you know... not quite good enough, but as broadcaster it's kinda acceptable.

Alright, let me show you the graphs.

Let's start with the C-Log3 to BT709 SDR:





As you can see, the C-Log3 curve is very different compared to the BT709 and it's a curve that it's meant for HDR productions as well, so although it can be used for SDR, the grading should really be done manually on a scene by scene basis, 'cause otherwise you might end up with oversaturated or wrong values.

C-Log 3 to BT2100 HDR PQ:





As you can see, the C-Log 3 curve pretty much resembles the PQ curve, this is also why it's actually pretty easy to map it to HDR PQ BT2100.

C-Log 3 to BT2100 HDR HLG:





As you can see, I tried to keep the HLG curve as tweaked as possible to SDR, which actually limited the quality of the blacks for the HDR viewers. As to the peak, HLG is limited to 1000 nits by definition.

Last edited by FranceBB; 24th April 2019 at 03:47.
FranceBB is offline   Reply With Quote
Old 24th April 2019, 03:41   #16  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Alright, let's crack on and view some examples.
You'll find the original C-Log3 footage, the BT709 SDR one, the HDR HLG one and the HDR PQ one.
I tried to keep the BT709 LUT as usable as possible, but limiting C-Log3 to BT709 SDR means that many details are really gonna be clipped out and you can definitely see it from the examples below.
As to the HDR HLG BT2100, it pretty much resembles the BT709 SDR version, but it does show more details, despite being still capped out.
Finally, the HDR PQ BT2100 is the best looking version, with a very natural look and it almost feel like as if you were there for real with 3500 nits worth of data.

C-Log3

BT709 SDR

BT2100 HDR HLG

BT2100 HDR PQ


C-Log3

BT709 SDR

BT2100 HDR HLG

BT2100 HDR PQ


C-Log3

BT709 SDR

BT2100 HDR HLG

BT2100 HDR PQ


C-Log3

BT709 SDR

BT2100 HDR HLG

BT2100 HDR PQ
FranceBB is offline   Reply With Quote
Old 25th April 2019, 22:54   #17  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Posts: 1,213
Once again, thank you for your contribution
gonca is offline   Reply With Quote
Old 1st May 2019, 08:13   #18  |  Link
Sharc
Registered User
 
Join Date: May 2006
Posts: 3,997
Perhaps a dumb question: How do people select the "correct" LUT? Is it a matter of personal taste, or do we need reference pictures for comparison?
Sharc is offline   Reply With Quote
Old 1st May 2019, 14:08   #19  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,869
Quote:
Originally Posted by Sharc View Post
Perhaps a dumb question: How do people select the "correct" LUT? Is it a matter of personal taste, or do we need reference pictures for comparison?
What do you mean?
Each LUT is a matrix of linear transformation that goes from a curve to the other, which basically means that, depending on your source color curve, you use the one to go from it to your target color curve.
For instance, if you have an HDR PQ video and you wanna go to HDR HLG, you are gonna use my PQ_to_HLG LUT.

If you are asking how to create a LUT yourself, then that's a whole different question and... no, it's not a matter of personal taste only, it's a matter of linear algebra, defined standards and personal taste.
You generally start with a linear conversion from a curve to the other and then tweak it trying to get into the right values according to the standard. After that, you can fine-tune it 'till you'll get something that looks best for your taste, as long as you are still respecting the standards.
Anyway, it really depends on your source and your target, 'cause if you have two curves that are really different one to the other, then you are gonna have to choose which parts of the input you wanna keep while you go to your desired output.
Let's suppose that you have a C-LOG3 in input and you wanna go to BT709 SDR in output; you can basically get the calculations right, but you still have to choose what you wanna sacrifice as the BT709 SDR is able to retain very little informations compared to the input and THAT is a personal choice according to what you think it's best.
When I make my LUTs, I always try to keep as many things as possible and make them usable for pretty much every scenario. Let's suppose that we have a scene with snow, a white dog and the sky: you are gonna have three different tonalities of white, one for the snow, one for the dog and one for the highlights (sky). If you originally shoot in BT709 SDR, you have one of those three tonalities of white clipped out, while if you shoot in C-LOG3 you are gonna have as many details as your camera can get, then, in encoding, you can decide to retain the desired part of the image, with the desired tonalities of white. In this specific case, you wouldn't be able to do it with my LUT, 'cause my LUT is gonna take a decision for you and clip what I thought it was supposed to be an acceptable compromise for every scenario, but it's better to do it manually, 'cause by doing it manually you can choose which things you are gonna preserve for that specific shot.
I hope this answer your question; if you want, I can get into more technical details.

Cheers,
Frank.

Last edited by FranceBB; 1st May 2019 at 16:44.
FranceBB is offline   Reply With Quote
Old 1st May 2019, 14:49   #20  |  Link
Sharc
Registered User
 
Join Date: May 2006
Posts: 3,997
Thank you Frank for enlightening me. Your short tutorial answered my nebulous question about the LUTs nicely

Last edited by Sharc; 1st May 2019 at 14:52.
Sharc is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:20.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.