Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 12th July 2021, 15:43   #121  |  Link
chmars
Registered User
 
Join Date: Apr 2021
Posts: 127
Quote:
Originally Posted by FranceBB View Post
At this point I'm almost tempted to ask you one last attempt if the last LUT doesn't work: namely trying HLG, let the camera adjust itself dynamically and hope for the best.
Yes, I could do what you need but I'm afraid camera auto-adjust will not work. The system will gonna turn crazy with this huge black borders and this flashing blue LED.
Too bad there is not yet an embedded version of Avisynth to allow treating only one part of the frame!

Film is always different.
Film was changing a lot. every batch of reels had its curve, colors (little changes, but still)... If you add exposure errors and age, you get a nightmare.

When we use to shoot in film for features, they always asked to get the same batch to avoid testing again and again to know exactly how to expose.

If some characteristics common to all films exist, the LUT must be limited to those (probably your original Slog3-LUT?). The rest is to be made by hand or automatic filters.
Maybe would VideoFred or JohnMeyer have some clever suggestions...

One point I don't get: why applying the LUT seems to turn to a so unstable result, exaggerating sometimes the yellow-green, sometimes the blue?
chmars is offline   Reply With Quote
Old 12th July 2021, 18:09   #122  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Quote:
Originally Posted by chmars View Post
Yes, I could do what you need but I'm afraid camera auto-adjust will not work. The system will gonna turn crazy with this huge black borders and this flashing blue LED.
Right, there's also that LED that will screw things up...


Quote:
Originally Posted by chmars View Post
Film is always different.
Film was changing a lot. every batch of reels had its curve, colors (little changes, but still)... If you add exposure errors and age, you get a nightmare.
I see...



Quote:
Originally Posted by chmars View Post
If some characteristics common to all films exist, the LUT must be limited to those (probably your original Slog3-LUT?). The rest is to be made by hand or automatic filters.
Yeah, it's constantly changing, there isn't much I can do.
I was trying to make a comparison with this:

Code:
FFVideoSource("Super8_Slog3_AppleProRes.mov")
original=Subtitle("Original Analog Super8 film", size=60)

p0=FreezeFrame(0, 1, 0).Trim(0, 1).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(64, 0.40, 940, 64, 940, coring=false)
p1=trim(1, 344).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1)
p2=trim(345, 717).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(64, 0.20, 940, 64, 940, coring=false)
p3=trim(718, 1400).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1)
p4=trim(1401, 1451).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().tweak(hue=-4, dither=true).ConvertBits(bits=10, dither=1)
p5=trim(1452, 1599).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(200, 1, 810, 64, 940, coring=false)
p6=trim(1600, 1772).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(200, 1, 810, 64, 940, coring=false)
p7=trim(1773, 2344).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(200, 1, 810, 64, 940, coring=false)
p8=trim(2345, 2504).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(64, 0.40, 900, 64, 940, coring=false).tweak(sat=1.81, dither=true)
p9=trim(2505, 2634).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(64, 1.0, 800, 64, 940, coring=false).tweak(hue=-24, dither=true)
p10=trim(2635, 3204).ConvertBits(16).ConverttoPlanarRGB().Cube("Slog3_to_BT709.cube").ConverttoYUV422().ConvertBits(bits=10, dither=1).Levels(64, 0.40, 940, 64, 940, coring=false).ColorYUV(autogain=true, autowhite=true).tweak(sat=1.81, dither=true)

p0++p1++p2++p3++p4++p5++p6++p7++p8++p9++p10

Crop(90, 20, -20, -0)

SpotLess(RadT=16, ThSAD=1000000, ThSAD2=1000000, pel=2, chroma=false, BlkSz=64, Olap=4, tm=false, glob=false, bBlur=0.0)
SpotLess()

super = MSuper(pel=2, sharp=1)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super,bv1,fv1,bv2,fv2,thSADC=1400, thSAD=1400)

SpotLess()

dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=0, dither=0)


FrostyBorders(1650, 1150, frosty=false)
Subtitle("Digital Avisynth Remaster", size=60)
filtered=last

StackHorizontal(original, filtered)
first=last

waveform=TurnRight.Histogram.TurnLeft.Crop(0, 0, -0, -1150).Spline64Resize(3300, 706)

StackVertical(waveform, first)

FrostyBorders(3840, 2160, frosty=false)
and it's encoding at the whopping speed of 0.04fps on a 56c/112th at 7% CPU usage.
1 Day and 2 hours remaining... DX


Quote:
Originally Posted by chmars View Post
Maybe would VideoFred or JohnMeyer have some clever suggestions...

They're definitely more expert than me in terms of Remastering old reels. They're the masters of Remastering. They were doing this long before I was, so maybe they're gonna have a trick or two for us, I don't know. :P


Quote:
Originally Posted by chmars View Post
One point I don't get: why applying the LUT seems to turn to a so unstable result, exaggerating sometimes the yellow-green, sometimes the blue?
'cause the input is unstable, it constantly changes, it's incredibly hard to work with!
It's really really hard when you have each "cut" different from the other. In your sample I had to divide it in 10 parts and threat each one of them individually in a different way 'cause otherwise I would have got some pretty bad results.
On top of that, the results I've got in the end are far from being great as I should have opened Davinci Resolve and played with them a bit, but I didn't want to and since I was doing everything in Avisynth I ended up with a compromise.
Anyway, I'll send you the results just for curiosity.
At least I'm happy about one thing, though: SpotLess() got rid of (almost) all the spots.
FranceBB is offline   Reply With Quote
Old 12th July 2021, 22:19   #123  |  Link
chmars
Registered User
 
Join Date: Apr 2021
Posts: 127
Yumy, rich answer!

If I dare: I am afraid we might spread our minds all over and loose our point if we talk about spot removal, stabilisation, denoising, etc.
However, I am very curious to see your results color-wise, sure, but also about your cleaning approach, frosty borders...

Only some words...
[Off topic parenthesis begin]
Just to explain: Alone, Spotless eats parts of moving subjects (balls, feet, hands...). It needs a colleague to bring them back.
A whole study was made in this thread about spot removal (and, somehow, grain keeping).
Brilliant results thanks to Stainlesss with Spotless and Zorr with Delta Restore. Really efficient and impressing. The flavour I use and now tested a lot is here.
Here, you can see an example (Zorr) of what Spotless removed and what Delta Restore bought back:

You know that place, don't you?

[Off topic parenthesis end]

Quote:
Originally Posted by FranceBB View Post
the results I've got in the end are far from being great as I should have opened Davinci Resolve and played with them a bit, but I didn't want to and since I was doing everything in Avisynth I ended up with a compromise.
Yes, I had great results with DR but it is totally anti-productive for such a task. I know John also uses an NLE to grade sometimes.
It would really be great having a tool that automates some part of the work like Fred/John scripts do but a bit further ifever possible.

I tried with Gammac and Autolevels.
Major problem is that Gammac will also correct the sky, sea or grass color when they fill the whole image.
Maybe, good approach would be to have one correction mixing all the scenes, plus one scene-by-scene correction. And percentages to balance these two corrections.
Well, off again....

Still: By looking at the scenes differences with the projector on a screen, differences are not so extreme as when LUT is applied on the capture. I am sure there is something to understand here.
Just intuition. No tools, no knowledge. Maybe you could see some moment in the process where colors go crazy?
chmars is offline   Reply With Quote
Old 15th September 2021, 11:05   #124  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Released Linear Transformation v1.7

Quote:
Changelog:

- Added support for ZLog2 HDR to BT709 SDR 100 nits
Full Screen Pictures: ZLog2 HDR - BT709 SDR

Comparison:




Waveform:



Plotted matrix of transformation with In and Out:

FranceBB is offline   Reply With Quote
Old 16th September 2021, 17:29   #125  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
I'm gonna take the chance to post a conversation occurred in PM with a user.

So basically they've asked my opinion for a BT2020 UHD stream and I noticed that the color transfer was set to BT2020, which means BT2020 SDR 100 nits, however the user was concerned about the possibility that it could have been an HDR HLG stream.

Original Picture: https://i.imgur.com/NbxFvXN.jpg


So, I indexed the stream to take a look at it and... Oh my God, what a mess they've done...

I gotta say that I don't know SK 4K as a channel, but they've done a mess.

See here:



In the middle you see those "arcs" and then straight line. Those are in the middle, namely in the 3.18 Y HLG relative value. In other words, that's the peak for reference white in HLG, in fact if you look at the bottom, this should be white:




So far so good, 'cause we would think that it's just a stream tagged wrongly and we can just easily tag it as arib-std-b67, so HLG in color transfer and that's it, right?

WRONG!

THIS is where they screwed up:



They overlayed a BT2020 SDR 100 nits logo in a BT2020 HLG stream!!
You can't do that, of course, in fact the waveform skyrockets to 940 as it shouldn't...

Now, this is the original picture: https://i.imgur.com/NbxFvXN.jpg

This is what TVs get with the "wrong" BT2020 SDR 100 nits flag (as they're currently airing): https://i.imgur.com/3YfmBY9.jpg


And this is what you would get with the BT2020 HLG HDR flag (as they should be airing): https://i.imgur.com/WN9S2b0.jpg

Of course, by doing that the TV will have to compensate the logo as it's not HLG and has therefore the wrong values...
FranceBB is offline   Reply With Quote
Old 22nd October 2021, 09:14   #126  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
2021, what a year we're living in: a year in which users are now responsible to set the right frame properties themselves ehehehehehe

FranceBB is offline   Reply With Quote
Old 23rd October 2021, 17:00   #127  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Linear Transformation v1.8 Released!!

Quote:
Changelog:

- Introduced frame properties support for Avisynth 3.7.1
https://github.com/FranceBB/LinearTr...ation/releases
FranceBB is offline   Reply With Quote
Old 5th November 2021, 20:40   #128  |  Link
MaxLt
Registered User
 
Join Date: Feb 2016
Posts: 7
HLG -> PQ via FFMpeg?

Is this something I can use to "convert" HLG video to PQ?

Via some unusual events, I ended up with dozens of short HLG 10-bit videos that look exactly as needed on equipment that supports HLG correctly. So I do not need to colour grade them or edit them - the goal is to keep them looking as close to the way they do now but be compatible. I'd like to try to convert them to PQ because PQ videos display correctly on all equipment I have, and if that fails, convert them to SDR.

I tried doing it in Davinci Resolve with just one video (without these LUTs), and it seems to work very well. But ideally, I'd be able to do it in a batch with FFMpeg. I batch transcode them to the needed bitrate via FFMpeg anyway, so if I can also convert to PQ in that step, it would be ideal. Am I on the right track?
MaxLt is offline   Reply With Quote
Old 5th November 2021, 21:27   #129  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Quote:
Originally Posted by MaxLt View Post
ideally, I'd be able to do it in a batch with FFMpeg. I batch transcode them to the needed bitrate via FFMpeg anyway, so if I can also convert to PQ in that step, it would be ideal. Am I on the right track?
Definitely. You can totally use my LUTs to go from HLG to PQ, either by using LinearTransformation() in Avisynth or by applying the LUT directly from FFMpeg. Additionally, if you need something that takes care of automatic encoding and works with both Avisynth and FFMpeg, you might wanna check our product: FFAStrans. It's free and available for everyone to use and it's the one that is handling the encoding of the overwhelming majority of contents here at Sky.
I think that FFAStrans is what you're looking for

https://ffastrans.com/wp/
FranceBB is offline   Reply With Quote
Old 5th November 2021, 23:51   #130  |  Link
MaxLt
Registered User
 
Join Date: Feb 2016
Posts: 7
Quote:
Originally Posted by FranceBB View Post
I think that FFAStrans is what you're looking for

https://ffastrans.com/wp/
Thank you! Just tried it, and my mind is blown. It is exactly what I need if I can make it work. So far the PQ result is too dark for some reason (and I cannot figure out how to use CRF rather than fixed bit rate with HEVC output module there, which would be a good option to have but not critical).

I seem to have been able to localize the cause of dark image to the 2020 HLG -> 2100 PQ linear transform filter. Using all the same settings for PQ but BT709 source and BT709->PQ linear transform works fine. Similarly, using 2020 HLG source with HLG to BT709 transform works fine. Only 2020 HLG to 2100 PQ produces a very dark image but with correct colors (white is about the level of SDR white and shadows are all near black). Either there is a bug or I'm doing something wrong. I will try directly with FFMpeg, but I have difficulty figuring out the command line parameters there.

Last edited by MaxLt; 6th November 2021 at 03:49.
MaxLt is offline   Reply With Quote
Old 6th November 2021, 15:24   #131  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Quote:
Originally Posted by MaxLt View Post
Thank you! Just tried it, and my mind is blown. It is exactly what I need if I can make it work.
Perfect.

Quote:
Originally Posted by MaxLt View Post
So far the PQ result is too dark for some reason
This is weird.

Quote:
Originally Posted by MaxLt View Post
I seem to have been able to localize the cause of dark image to the 2020 HLG -> 2100 PQ linear transform filter. Using all the same settings for PQ but BT709 source and BT709->PQ linear transform works fine. Similarly, using 2020 HLG source with HLG to BT709 transform works fine. Only 2020 HLG to 2100 PQ produces a very dark image but with correct colors (white is about the level of SDR white and shadows are all near black). Either there is a bug or I'm doing something wrong. I will try directly with FFMpeg, but I have difficulty figuring out the command line parameters there.

Perhaps you found a bug.
Can you please check this build: Link

and do a simple test with: Watch -> Encoder

and choose Auto LUT in the encoder?

If it still doesn't work correctly, then I'm gonna have to ask you to provide me privately a very small sample to work with.
FranceBB is offline   Reply With Quote
Old 6th November 2021, 21:08   #132  |  Link
MaxLt
Registered User
 
Join Date: Feb 2016
Posts: 7
I tried the new build and also tried with FFMpeg directly with just commend (to remove possible issues) and got the same result in every case. I'll upload the source and result samples and PM a link to it later today. Thanks for all your help!
MaxLt is offline   Reply With Quote
Old 7th November 2021, 22:23   #133  |  Link
MaxLt
Registered User
 
Join Date: Feb 2016
Posts: 7
I figured it out, I think. The LUT is for BBC HLG and what I had was "base HLG". I was able to generate a LUT that worked well with this tool: https://cameramanben.github.io/LUTCa...alc/index.html
MaxLt is offline   Reply With Quote
Old 8th November 2021, 11:10   #134  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
That's totally fine, no problem.
You can apply a custom Avisynth Script within FFAStrans and apply your custom LUT just as easily
FranceBB is offline   Reply With Quote
Old 8th November 2021, 22:15   #135  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 412
What is the difference between "BBC HLG" and "Base HLG?"

I know that in the early days, HLG had the reference white put at 0.5. Now the ITU has come along and put it at 0.75. Is that the difference between them?
wswartzendruber is offline   Reply With Quote
Old 9th November 2021, 04:19   #136  |  Link
MaxLt
Registered User
 
Join Date: Feb 2016
Posts: 7
Quote:
Originally Posted by wswartzendruber View Post
What is the difference between "BBC HLG" and "Base HLG?"

I know that in the early days, HLG had the reference white put at 0.5. Now the ITU has come along and put it at 0.75. Is that the difference between them?
Yes, the "Spec Base/NHK HLG" white is 6 stops above neutral and "BBC HLG" is 4.5 stops above neutral. The red line here is Spec Base and the blue line is BBC. I'm surprised how few sources even mention this difference. Sony cameras seem to output the "Spec base/NHK HLG" with the brighter white point.
Attached Images
 

Last edited by MaxLt; 9th November 2021 at 04:24.
MaxLt is offline   Reply With Quote
Old 9th November 2021, 05:07   #137  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 412
Quote:
Originally Posted by MaxLt View Post
Yes, the "Spec Base/NHK HLG" white is 6 stops above neutral and "BBC HLG" is 4.5 stops above neutral. The red line here is Spec Base and the blue line is BBC. I'm surprised how few sources even mention this difference. Sony cameras seem to output the "Spec base/NHK HLG" with the brighter white point.
I think of these two as being labelled "ARIB STD-B67" (reference white = 0.5) and "ITU-R BT.2408" (reference white = 0.75).

Am I right in assuming that BT.2408 is the norm?
wswartzendruber is offline   Reply With Quote
Old 9th November 2021, 09:01   #138  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Quote:
Originally Posted by MaxLt View Post
Sony cameras seem to output the "Spec base/NHK HLG" with the brighter white point.
Of course, Japanese people are playing at home with Sony Cameras ehehehehehe

Quote:
Originally Posted by wswartzendruber View Post
I think of these two as being labelled "ARIB STD-B67" (reference white = 0.5) and "ITU-R BT.2408" (reference white = 0.75).

Am I right in assuming that BT.2408 is the norm?
Not here. Over here, we definitely use arib-std-b67 and we follow BBC Specs.
FranceBB is offline   Reply With Quote
Old 9th November 2021, 17:24   #139  |  Link
wswartzendruber
hlg-tools Maintainer
 
wswartzendruber's Avatar
 
Join Date: Feb 2008
Posts: 412
Quote:
Originally Posted by FranceBB View Post
Not here. Over here, we definitely use arib-std-b67 and we follow BBC Specs.
You guys are putting reference white at 0.5? Because that's what ARIB STD-B67 calls for. At least on the copy I read.
wswartzendruber is offline   Reply With Quote
Old 9th November 2021, 17:54   #140  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Quote:
Originally Posted by wswartzendruber View Post
You guys are putting reference white at 0.5? Because that's what ARIB STD-B67 calls for. At least on the copy I read.
Yes, definitely.
FranceBB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:54.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.