Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 6th January 2022, 01:56   #81  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 413
Thank you!

I'd like to wait until 0.4.0 for listing it. That's going to have arguably better tone mapping. (It will map the RGB channels independently, which doesn't allow out-of-gamut results.)

I need to get some scripts ported to Windows (PowerShell?) and then that can happen.
wswartzendruber is offline   Reply With Quote
Old 6th January 2022, 02:14   #82  |  Link
ErazorTT
Registered User
 
Join Date: Mar 2003
Location: Germany
Posts: 215
No need to port scripts to windows. Just install the bash shell on windows, its in the package of git for windows:
https://gitforwindows.org/

Just install this, and you have a fully functional bash shell in windows.
ErazorTT is offline   Reply With Quote
Old 6th January 2022, 07:09   #83  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 413
Here's 0.4.0:

https://github.com/wswartzendruber/h...elease%2F0.4.0

I ported the BASH scripts over to PowerShell (for the Windows zip) so people could use them more easily.
wswartzendruber is offline   Reply With Quote
Old 6th January 2022, 08:58   #84  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
Quote:
Originally Posted by ErazorTT View Post
That first comparison I saw with the official LUTs showed me that what wswartzendruber implemented is actually spot on! He did a great job!
Yep. @wswartzendruber you should have got the file I sent which has been encoded using the official BBC LUT through Avisynth, namely:

Code:
video=FFVideoSource("\\mibctvan000\Ingest\MEDIA\temp\PQ2HLG-20220105T125539Z-001\PQ2HLG\PQ.mkv")
audio=BlankClip(length=607, fps=24000, fps_denominator=1001, channels=1)
AudioDub(video, audio)

#From 4:2:2 16bit planar Narrow Range to RGB Planar 16bit Full Range
z_ConvertFormat(pixel_type="RGBP16", colorspace_op="2020:st2084:2020:limited=>rgb:st2084:2020:full", resample_filter_uv="spline64", dither_type="error_diffusion")

#From PQ to HLG with 16bit precision
Cube("C:\Program Files (x86)\AviSynth+\LUTs\1a_PQ1000_HLG_mode-nar_in-nar_out-nar_nocomp.cube", fullrange=true)

#From RGB 16bit planar Full Range to YUV422 10bit planar Narrow Range with dithering
z_ConvertFormat(pixel_type="YUV422P10", colorspace_op="rgb:std-b67:2020:full=>2020:std-b67:2020:limited", resample_filter_uv="spline64", dither_type="error_diffusion")
and the result is close.

@ErazorTT I'm now encoding the test patters you sent me with the official BBC LUT and I'll post the 7 days link to the result here if you don't mind, so that everyone else can take a look at it too.

The idea, after that, would be to try to recreate a LUT as close as possible to the official one, so I'll send you back a new LUT file as well later in the afternoon.

Last edited by FranceBB; 6th January 2022 at 09:36.
FranceBB is offline   Reply With Quote
Old 6th January 2022, 10:10   #85  |  Link
ErazorTT
Registered User
 
Join Date: Mar 2003
Location: Germany
Posts: 215
Quote:
Originally Posted by wswartzendruber View Post
Here's 0.4.0
Awesome! I will test!

Quote:
Originally Posted by FranceBB View Post
@ErazorTT I'm now encoding the test patters you sent me with the official BBC LUT and I'll post the 7 days link to the result here if you don't mind, so that everyone else can take a look at it too.
Sure, I have created those patterns with my own avisynth scripts, which are part of my calibration pattern suite. So I could actually also post those, if anybody needs PQ or HLG test patterns.
ErazorTT is offline   Reply With Quote
Old 6th January 2022, 11:56   #86  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
There we go:

Color Saturation Steps: https://we.tl/t-gwqgHChcyZ
Gray Steps: https://we.tl/t-q8f4O5AGpt

(link valid for 7 days, so hurry up eheheheh)


This should produce really really similar results to the official BBC LUT: https://github.com/FranceBB/LinearTr...PQ_to_HLG.cube



BBC LUT (left) - My LUT, new Implementation (right)









FranceBB is offline   Reply With Quote
Old 6th January 2022, 15:38   #87  |  Link
ErazorTT
Registered User
 
Join Date: Mar 2003
Location: Germany
Posts: 215
Quote:
Originally Posted by FranceBB View Post
There we go:
Awesome! After looking closely into the results I must again complement you guys!

So FranceBB's v1.9 PQ_to_HLG.cube LUT is alomst identical as the results with wswartzendruber's hlg-tools v0.3.0 and 0.4.0 with the settings of max-cll 1000 and ref-white 203. The differences are only single 8bit values apart, and that only in very view of all the test squares of the patterns.
And the best is that both are extremely simular to the output using the official LUT! So a big thumbs up there!

The big advantage of the hlg-tools generator is that the LUT can be adjusted to whatever max limunosity is needed, while in contrast the fixed LUT's all will cut abruptly at 1000nits, which might be undesirable. And if more tuning is needed, there is also the possibility of changing the ref-white (or the brightness factor).

I have to ask though, was that expected that 0.3.0 and 0.4.0 return exaclty identical results? Perhaps however the differences are below the 8bit threshold for which I tested.
ErazorTT is offline   Reply With Quote
Old 6th January 2022, 16:43   #88  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 413
Quote:
Originally Posted by ErazorTT View Post
I have to ask though, was that expected that 0.3.0 and 0.4.0 return exaclty identical results? Perhaps however the differences are below the 8bit threshold for which I tested.
For a reference white of 203 nits and a MaxCLL of 1,000 nits, they should produce identical results.

Functionally, 0.4.0 does tone mapping a bit differently, but that only matters for MaxCLL values above 1,000 nits (or if you boost the brightness so that MaxCLL gets pushed up above that internally).

Internally, there was a lot of code refactoring that went into 0.4.0 to better organize things.

EDIT: I kind of want to get a wiki going of different 4K UltraHD discs and the pq2hlg settings that should be put into it for each movie.

Last edited by wswartzendruber; 6th January 2022 at 17:05.
wswartzendruber is offline   Reply With Quote
Old 6th January 2022, 17:28   #89  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
Quote:
Originally Posted by ErazorTT View Post
Awesome! After looking closely into the results I must again complement you guys!

So FranceBB's v1.9 PQ_to_HLG.cube LUT is alomst identical as the results with wswartzendruber's hlg-tools v0.3.0 and 0.4.0 with the settings of max-cll 1000 and ref-white 203. The differences are only single 8bit values apart, and that only in very view of all the test squares of the patterns.
And the best is that both are extremely simular to the output using the official LUT! So a big thumbs up there!
Nice!
Thank you for testing and making me work on this once more, I'm always happy to improve things.

Quote:
Originally Posted by ErazorTT View Post
The big advantage of the hlg-tools generator is that the LUT can be adjusted to whatever max limunosity is needed, while in contrast the fixed LUT's all will cut abruptly at 1000nits, which might be undesirable.
Yes, in particular, professionally wise, at Sky we have 3000 LUTs (I'm not joking) one for each nit, from 1000 nits to 4000 nits and we receive a masterfile like an Apple ProRes 4:4:4 10bit or 12bit or a Motion JPEG2000 4:4:4 12bit or a DNXHQX 4:2:2 12bit and an xml file in which the MaxCLL value is stored and from that we have a workflow that picks up that value and generates an Avisynth Script with the right LUT. Anything lower than 1000 nits uses the 1000 nits LUT, anything higher than 4000 nits uses the 4000 nits LUT 'cause we haven't got anything higher than that so far. Unfortunately I can't share those 'cause they're for internal use only (just like the BBC LUTs, they are licensed by the BBC to the individual broadcasters).
Still I gotta say, @wswartzendruber, impressive work, looks like you finally got it right this time!
Amazingly enough, something that you started as a hobby-like project, turned out to be so reliable that is able to produce professional-like results. Well done!
I don't know why many people over the past months didn't bother to reply or help you, but I knew you would have got this on your own eventually.

Last edited by FranceBB; 6th January 2022 at 17:30.
FranceBB is offline   Reply With Quote
Old 6th January 2022, 17:44   #90  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 413
I'm curious as to how pq2hlg's generated LUTs compete with the licensed ones you have.

Do you have the means+time+patience to compare pq2hlg at 4,000 nits MaxCLL to one of your licensed LUTs?
wswartzendruber is offline   Reply With Quote
Old 6th January 2022, 20:50   #91  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
Quote:
Originally Posted by wswartzendruber View Post
Do you have the means+time+patience to compare pq2hlg at 4,000 nits MaxCLL to one of your licensed LUTs?
Absolutely.
I think it's gonna be a useful exercise for us all. I stand on the free and open source software side, so the more open source softwares improve, the better.
Tomorrow I'm gonna be back at work and I'm gonna be busy hardsubbing the UHD HDR HLG Italian dubbed version of The Town with the Italian typesetting and it will take me a while 'cause it's always a bit of a pain in the... *Beep* to work with typesetting in HLG, especially given that Aegisub is stuck in the stone age (2014) and has no idea whatsoever of any colour curve and I have to use the colour picker to get exactly the same levels of the HLG video and then use a separate waveform monitor to cross check everything...

(You may think that official studios and professional use fancy paid HDR aware subtitling/typesetting software, but no, we're in the same tedious painful mess as the rest of the world eheheheh).

Anyway, I'm digressing, long story short, if I'm gonna have time tomorrow, I'll repeat the tests with the official 4000 nits LUT and see what happens.
FranceBB is offline   Reply With Quote
Old 7th January 2022, 11:46   #92  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
p.s "What's the most common thing you do at work?"

Me: Wait for indexing

LMAO

https://i.imgur.com/5QTbeCy.png

(456 GB worth of mezzanine file, it's gonna take forever ehehehehe)

Last edited by FranceBB; 7th January 2022 at 11:53.
FranceBB is offline   Reply With Quote
Old 10th January 2022, 20:15   #93  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 413
Of course, if this turns out too well, I'll probably have the BBC pissed off at me for taking away a source of revenue.
wswartzendruber is offline   Reply With Quote
Old 14th January 2022, 02:33   #94  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 413
I've just now realized that GitHub has wikis!

https://github.com/wswartzendruber/h...ovies-Database

I'll be filling this out as I work through my Marvel collection (really mad at DC right now).
wswartzendruber is offline   Reply With Quote
Old 15th January 2022, 17:38   #95  |  Link
ErazorTT
Registered User
 
Join Date: Mar 2003
Location: Germany
Posts: 215
Quote:
Originally Posted by FranceBB View Post
Anyway, I'm digressing, long story short, if I'm gonna have time tomorrow, I'll repeat the tests with the official 4000 nits LUT and see what happens.
Any news on that front? I'm still holding my breath :P
ErazorTT is offline   Reply With Quote
Old 15th January 2022, 18:16   #96  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
Quote:
Originally Posted by ErazorTT View Post
Any news on that front? I'm still holding my breath :P
Unfortunately I've had a bit of a rough series of days, as I've been busy with several issues (XDCAM frame properties and indexers reporting the wrong chroma location value, which screwed up some workflows I had. You can see the whole story on Avisynth Development, Avisynth+. As soon as that thing is solved/fixed, I'll spend time on the LUTs again, I promise).
FranceBB is offline   Reply With Quote
Old 17th January 2022, 13:25   #97  |  Link
ErazorTT
Registered User
 
Join Date: Mar 2003
Location: Germany
Posts: 215
I ran tests with different lum-scales and I think there is something I do not understand.

Let's say I have a PQ clip with a maxcll of 1000, then I'm first going to try out a LUT setting maxcll of 1000 and a scale of 1.0.

Now lets say I find that a scale 2.0 would be more fitting. What happens is that everything is brightened, including the whites.
My tests showed that the whites are all pushed to the max brightness bin (1023 or 940 whatever you prefer)! So now there is loss of information since all whites are pushed together into the max brighteness.
Shouldn't maxcll be thus increased according to the scale? Thus when the scale is increased to 2 the maxcll should also be increased to 2000, to leave place for all the whites?
2000 is perhaps too much, but I'm pretty sure it should be increased to something above 1000.

Or is all that not the case and I somehow mess up the processing?

Last edited by ErazorTT; 17th January 2022 at 15:01.
ErazorTT is offline   Reply With Quote
Old 17th January 2022, 17:13   #98  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 413
Quote:
Originally Posted by ErazorTT View Post
I ran tests with different lum-scales and I think there is something I do not understand.

Let's say I have a PQ clip with a maxcll of 1000, then I'm first going to try out a LUT setting maxcll of 1000 and a scale of 1.0.

Now lets say I find that a scale 2.0 would be more fitting. What happens is that everything is brightened, including the whites.
My tests showed that the whites are all pushed to the max brightness bin (1023 or 940 whatever you prefer)! So now there is loss of information since all whites are pushed together into the max brighteness.
Shouldn't maxcll be thus increased according to the scale? Thus when the scale is increased to 2 the maxcll should also be increased to 2000, to leave place for all the whites?
2000 is perhaps too much, but I'm pretty sure it should be increased to something above 1000.

Or is all that not the case and I somehow mess up the processing?
MaxCLL will be internally adjusted by the --lum-scale amount. So after specifying 1,000, the mapper will consider it to be 2,000 in light of --lum-scale 2.0. Tone mapping will then push everything down to 1,000 nits to prep for HLG conversion.
wswartzendruber is offline   Reply With Quote
Old 17th January 2022, 17:18   #99  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
Ok, so, the issue doesn't seem to be closed and it's making us all talk a lot here: https://forum.doom9.org/showthread.php?t=181351 and yet the FFMpeg developers don't seem to care: https://trac.ffmpeg.org/ticket/9598 which is actually annoying me as this is a problem which affects our production environment.
That being said, I've actually temporarily worked around the thing by pretending to be using Avisynth 3.7.0 instead of 3.7.1 or 3.7.2 by using propClearAll() after every file I index, which is far from being perfect, but still...

Anyway, here are the samples encoded with the official BBC LUT 4000 nits

ColorSaturationSteps BT.2100 PQ - ColorSaturationSteps BT.2100 PQ_BBC_LUT_4000_nits: https://we.tl/t-KLyDHIyaeq

GraySteps All - GraySteps All_BBC_LUT_4000_nits: https://we.tl/t-dqPcNT4B3y

(Link valid for 7 days)
FranceBB is offline   Reply With Quote
Old 17th January 2022, 19:06   #100  |  Link
ErazorTT
Registered User
 
Join Date: Mar 2003
Location: Germany
Posts: 215
Hm that is weird. The BBC lut produces two steps at almost exactly the same non-white brightness (red arrows in the grayBBC.png picture). These steps are apart 5% of the signal, which for PQ is really a lot in absolute terms. So that must be wrong!

Addionally, I took out my colorspace calculation sheet and found that a 4000nits white is at a 8bit value of 198 for a limited range signal. So actually in the picture the step with an orange arrow should already have been completly white, since that is at a value of 202. (Always assuming that I implemented my calcualtion sheet correctly which is quite a big if.)

The lut generator however produces anything as I expected. See picture gray_4000_1.0.png.

I also added the original grayPQ.png but with the signal values written as text so that anybody knows which step is which.

https://drive.google.com/drive/folde...QT?usp=sharing

Last edited by ErazorTT; 17th January 2022 at 19:26.
ErazorTT is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 04:14.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.