Process in between converting HDR content to reduced HDR content

[jriker1]

I have been chatting on here about a ffmpeg script to take H265 content and convert it to smaller H265 content but maintain the HDR10 metadata. So I have that, but there are some cases even though the video may be a super high bitrate, that there is still noticeable noise in the video that I have the opportunity to remove before doing the final encode. Problem is I don't want to mess with the color of the video and perhaps although I may not have a good understanding of HDR and color primaries, basically want to do the following:

1. I have an HEVC, HDR10 file.
2. Would like to reduce noise. Normally do this in Premiere with NeatVideo or Davinci Resolve Studio but usually the first tool.
3. Want to export the video in step 2 reducing noise uncompressed and maintain the video coloring as is. I know I'm going to loose the "HDR metadata".
4. I then want to run it thru ffmpeg and put in the original videos HDR10 Metadata while re-encoding.

Obviously the key to this is whatever comes out of Premiere maintains the coloring and doesn't loose any data or brightness or darkness. Any thoughts how I would do steps 2 and 3? Always afraid if I open a file in Premiere or Davinici Resolve and export it the underlying color will be lost. Again probably have a poor understanding at this point of HDR and what it means.

Info from MediaInfo:
Video
Count : 379
Count of stream of this kind : 1
Kind of stream : Video
Kind of stream : Video
Stream identifier : 0
StreamOrder : 0
Inform : 53.6 Mb/s, 3840*2160 (16:9), at 23.976 (24000/1001) FPS, HEVC (Main 10@L5.1@High), HDR10
ID : 1
ID : 1
Format : HEVC
Format : HEVC
Format/Info : High Efficiency Video Coding
Format/Url : http://www.itu.int
Commercial name : HEVC
Format profile : Main 10@L5.1@High
HDR format : SMPTE ST 2086
HDR format : SMPTE ST 2086, HDR10 compatible
HDR_Format_Commercial : HDR10
HDR_Format_Compatibility : HDR10
Internet media type : video/H265
Codec ID : hev1
Codec ID/Info : High Efficiency Video Coding
Codec ID/Url : http://www.itu.int/
Duration : 5262174
Duration : 1 h 27 min
Duration : 1 h 27 min 42 s 174 ms
Duration : 1 h 27 min
Duration : 01:27:42.174
Duration : 01:27:43;13
Duration : 01:27:42.174 (01:27:43;13)
Source duration : 5262215
Source duration : 1 h 27 min
Source duration : 1 h 27 min 42 s 215 ms
Source duration : 1 h 27 min
Source duration : 01:27:42.215
Bit rate : 53587537
Bit rate : 53.6 Mb/s
Width : 3840
Width : 3 840 pixels
Height : 2160
Height : 2 160 pixels
Sampled_Width : 3840
Sampled_Height : 2160
Pixel aspect ratio : 1.000
Display aspect ratio : 1.778
Display aspect ratio : 16:9
Rotation : 0.000
Frame rate mode : CFR
Frame rate mode : Constant
Frame rate : 23.976
Frame rate : 23.976 (24000/1001) FPS
FrameRate_Num : 24000
FrameRate_Den : 1001
Frame count : 126167
Source frame count : 126168
Color space : YUV
Chroma subsampling : 4:2:0
Chroma subsampling : 4:2:0 (Type 2)
ChromaSubsampling_Position : Type 2
Bit depth : 10
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.269
Stream size : 35248639594
Stream size : 32.8 GiB (100%)
Stream size : 33 GiB
Stream size : 33 GiB
Stream size : 32.8 GiB
Stream size : 32.83 GiB
Stream size : 32.8 GiB (100%)
Proportion of this stream : 0.99995
Source stream size : 35248642671
Source stream size : 32.8 GiB (100%)
Source stream size : 33 GiB
Source stream size : 33 GiB
Source stream size : 32.8 GiB
Source stream size : 32.83 GiB
Source stream size : 32.8 GiB (100%)
Source_StreamSize_Proportion : 0.99995
colour_description_present : Yes
colour_description_present_Source : Container / Stream
Color range : Limited
colour_range_Source : Container / Stream
Color primaries : BT.2020
colour_primaries_Source : Container / Stream
Transfer characteristics : PQ
transfer_characteristics_Source : Container / Stream
Matrix coefficients : BT.2020 non-constant
matrix_coefficients_Source : Container / Stream
Mastering display color primaries : Display P3
MasteringDisplay_ColorPrimaries_Source : Stream
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2
MasteringDisplay_Luminance_Source : Stream
Maximum Content Light Level : 10000 cd/m2
MaxCLL_Source : Stream
Maximum Frame-Average Light Level : 1166 cd/m2
MaxFALL_Source : Stream
mdhd_Duration : 5262174
Codec configuration box : hvcC



info from ffprobe:
"color_range": "tv",
"color_space": "bt2020nc",
"color_primaries": "bt2020",
"color_transfer": "smpte2084",
"chroma_location": "topleft",
"side_data_list": [
{
"side_data_type": "Mastering display metadata",
"red_x": "34000/50000",
"red_y": "16000/50000",
"green_x": "13250/50000",
"green_y": "34500/50000",
"blue_x": "7500/50000",
"blue_y": "3000/50000",
"white_point_x": "15635/50000",
"white_point_y": "16450/50000",
"min_luminance": "50/10000",
"max_luminance": "40000000/10000"
},
{
"side_data_type": "Content light level metadata",
"max_content": 10000,
"max_average": 1166
}
]

[benwaggoner]

One challenge with HDR is that it doesn't have a simple EOTF like Rec 709 gamma. Lots of SDR filters just do math based on code values, which kinda works for the limited range of values, although linear light processing will generally look better with a lot of filtering.

But HDR10 uses the PQ EOTF, which is a lot more complex, and works even less well with non-linear-light processing than SDR. And lots of traditional video filters are based around SDR code values and will do weird things with HDR. Noise is challenging, because it can often be a lot brighter than the rest of the content, and so denoising can dim the overall luma of the frame significantly in a way that's just not a concern with SDR. Maybe it looks better than way, but the HDR metadata of the original frame is just not accurate to the output frame. This even comes up with simple scaling. If a star is a 4000 nit single pixel on a black background at 4K, it should become a 1000 nit single pixel at 1080p, per =AVERAGE(4000, 0, 0, 0). But a SDR or code based filter would give you something much dimmer.

Even with platforms like Premiere/After Effects that have native 32-bit float linear light processing, tons of built in and 3rd party filters aren't updated to support linear float. At least the ones that don't get a warning symbol next to them when in a linear float project. You can get accurate scaling and color correction, sure, but any older plugin will need to be significantly upgraded.

And forget something like ffmpeg, which doesn't have anything close to end-to-end linear float.

[Blue_MiSfit]

Don't be afraid of the metadata in HDR10. it's quite simple. Dynamic metadata with HDR10+ or Dolby Vision gets more complicated.

Here's what you need to worry about with HDR10:

1) The color primaries, transfer, and matrix/differencing/"color space" in ffprobe. HDR10 will always use the same values here:

- Primaries = BT. 2020
- Transfer = SMPTE ST. 2084
- Differencing / Matrix / "Color Space" = BT. 2020 (specifically the non-constant luminance version hence bt2020nc)

Again, these are _ALWAYS_ set this way for all HDR10 content. This metadata just tells the display "this is HDR10 content" and is all that's technically needed to engage HDR mode and display the content.

2) The mastering display metadata describes how the colorist's mastering display was set up in terms of color volume. This blob of data describes the X / Y coordinates of each color primary and the white point, plus the minimum and maximum luminance of the display. In theory this is useful during HDR10 playback because your display could use it when tone mapping the HDR10 video to match the panel capabilities of the display. In practice it has little to no effect.

3) The content light level metadata indicates the luminance of the single brightest pixel in the entire content, and also the average luminance of the single brightest frame in the entire content. Again in theory this can help tone mapping but in practice it has little to no effect.

All of this information can be taken from the source and specified in the x265 command line to inject it during your final encode. It's just a matter of translating between different syntax styles :)

Now... technically the content light level metadata would change since you're filtering the video... so I'd just omit it entirely. It's optional per HDR10 (whereas technically the mastering display metadata is _not_ optional). You could carry it over from the source but it would be incorrect so why bother?

In any case, I agree with Ben above. Filtering PQ video should ideally be done with filters understanding PQ. You might get noise reduction that you like, but you might not. Good luck :)

[benwaggoner]

2) The mastering display metadata describes how the colorist's mastering display was set up in terms of color volume. This blob of data describes the X / Y coordinates of each color primary and the white point, plus the minimum and maximum luminance of the display. In theory this is useful during HDR10 playback because your display could use it when tone mapping the HDR10 video to match the panel capabilities of the display. In practice it has little to no effect.
Yeah, I'm not aware of any display that doesn't ignore master display metadata.

3) The content light level metadata indicates the luminance of the single brightest pixel in the entire content, and also the average luminance of the single brightest frame in the entire content. Again in theory this can help tone mapping but in practice it has little to no effect.
MaxCLL IS used by some TVs, notably LG. The higher the MaxCLL, the dimmer the brightest part of the image gets displayed.

Now... technically the content light level metadata would change since you're filtering the video... so I'd just omit it entirely. It's optional per HDR10 (whereas technically the mastering display metadata is _not_ optional). You could carry it over from the source but it would be incorrect so why bother?
I'm not 100% sure what an LG does when MaxCLL is not set; it might assume an arbitrary value which might not be optimum. Setting MaxCLL accurately will maximize accuracy and brightness at the high end.

In any case, I agree with Ben above. Filtering PQ video should ideally be done with filters understanding PQ. You might get noise reduction that you like, but you might not. Good luck :)
And you might get NR you like with one piece of content, but not others. It'll behave quite differently at different luma ranges.

[jriker1]

Thanks for the responses. I'm starting to wonder if I should skip noise reduction on the content going forward. I used NeatVideo all the time before with my 1080p content and feel I was able to compress the content more and still retain good quality, but sounding like this may not be ideal now. Especially if the HDR info will not be accurate when doing this any longer anyway. Supposedly NeatVideo supports 4k HDR content but since Premiere isn't maintaining the metadata, not entirely sure what it's maintaining of the video itself.

Out of curiosity, when they say the material will look washed out or whatever when viewing/editing on a non HDR monitor, is that because of the original video material or the HDR data? Would think if HDR can provide brighter brights and darker darks, that on a standard monitor it would still look decent, but everything I read speaks to the contrary so curious where this extra detail resides, the physical video or the HDR data. From what I read here, sounds like in the case of my LG OLED CX display, it pretty much doesn't use the HDR10 data if I read things right. Is it just that the colorspace is different so looks different on an HDR display?

One other item, I noticed that on my video MediaInfo says the Color Space is YUV and ffprobe says it's bt2020nc. What's up with that?

[rwill]

Out of curiosity, when they say the material will look washed out or whatever when viewing/editing on a non HDR monitor, is that because of the original video material or the HDR data? Would think if HDR can provide brighter brights and darker darks, that on a standard monitor it would still look decent, but everything I read speaks to the contrary so curious where this extra detail resides, the physical video or the HDR data. From what I read here, sounds like in the case of my LG OLED CX display, it pretty much doesn't use the HDR10 data if I read things right. Is it just that the colorspace is different so looks different on an HDR display?

HDR10 is using the PQ transfer function instead of the SDR BT.709 gamma transfer function. HDR10 is also using BT.2020 color matrix/primaries which fits a larger color volume than BT.709. PQ fits the BT.709 brightness range in roughly the first half of the PQ codeword range and if you take a BT.709 color value and transform it to BT.2020 the values become smaller. So if you display HDR10 as if it were SDR BT.709 most of the luma codewords will be at or below half brightness and chroma will have smaller and different values, leading to a desaturated and false color image unless the colorist went wild with the HDR10 version. So thats what people mean with washed out.

To display HDR10 on a panel there has to be a process that maps PQs 0-10000 nits and the full BT.2020 color volume ( or at least the color volume of the mastering display ) as most panels only have 400-800 nits and smaller color volume. This process should not change the artistic intent. As this mapping is display specific it happens in the display most of the time. It would be really hard doing it in an application window of an operating system, unless you map HDR10 to SDR BT.709 and display that. So when most people view HDR10 content on a PC monitor they view it as SDR BT.709.

[benwaggoner]

Yeah, HDR code values are just wrong on a SDR monitor. Beyond those issues with contrast given above, color primaries are also different, so hue is off. In the early days of HDR (2014-2016) I still had to do a fair amount of QA on SDR monitors as there weren't good HDR from PC solutions then, and it was like looking at a TV through an inch of black tea.

Today Windows 10 does a pretty decent job of mapping HDR10 to whatever your panel can do, so at least hue is correct and you can generally see all the info. It's more reliable on a laptop as the exact panel can be known. External monitors require that the monitor accurately report its display properties via EDID. But those are static values, so Windows can't account for different picture modes. Plus they have errors all too often.

[jriker1]

OK here I am again. So I have a nice ffmpeg setup I think and did some conversions. Keeps the luminance, and whitelevel info and I apply the color metadata. I'm going to hope per this thread that keeping that info is accurate if I'm just lowering the bitrate using CRF 13. So great now I have a reduced bitrate version of the prior footage. Won't get into the 7 days to encode it. Please new model Threadripper you need to come out!!!! Anyway I digress. Maybe it's just my perception, but the footage is garbage. And I'm not talking about my conversion, talking the source. I have never liked film grain or whatever kind of grain this is so I took out one of my BD Movies. Let's look at Harry Potter Sorcerer's Stone, it's 4k 2016p HDR content. It looks super grainy. I'm sitting probably 17 feet away looking at a 77" TV and I can see it. I have NeatVideo for Premiere and VirtualDub but have mentioned previously those tools seem to strip out all HDR data other than it's flagged as HDR. Not specifically referring to NeatVideo but Premiere.

Should I just assume the grain is what it is and leave it in there or do I have any options? I was starting to look at nlmeans but sounds like grain removal or reduction wouldn't be prudent in ffmpeg.

[Boulder]

You could always watch The Lord of the Rings 4K transfers, Mr. Jackson decided to DNR them all the way so they look like The Hobbitses :devil: If it wasn't for the HDR and Dolby Atmos tracks, they would be worse than the old Blu-rays.

Seriously speaking, if you need any filtering, Avisynth+ is the way to go. There are many ready-made functions meant for denoising/degraining, for example Dogway's SMDegrain. I also downsize 4K's to 1440p (native 4K) or 1080p (very noisy native 4Ks or ones that are upscaled to 4K from 2K). It also removes some excessive noise, makes encodes faster and also smaller files.

Do not worry about people telling you that you cannot process HDR sources with regular Avisynth methods. They work just fine even if it's not scientifically 100% correct.

[benwaggoner]

You could always watch The Lord of the Rings 4K transfers, Mr. Jackson decided to DNR them all the way so they look like The Hobbitses :devil: If it wasn't for the HDR and Dolby Atmos tracks, they would be worse than the old Blu-rays.

Seriously speaking, if you need any filtering, Avisynth+ is the way to go. There are many ready-made functions meant for denoising/degraining, for example Dogway's SMDegrain. I also downsize 4K's to 1440p (native 4K) or 1080p (very noisy native 4Ks or ones that are upscaled to 4K from 2K). It also removes some excessive noise, makes encodes faster and also smaller files.

Do not worry about people telling you that you cannot process HDR sources with regular Avisynth methods. They work just fine even if it's not scientifically 100% correct.
Does AVISynth+ have PQ-native filters? Filters that assume code values are gamma per Rec.709 often have weirdness and defects working with PQ.

PQ filters should really operate in floating point linear light.

[jriker1]

What's interesting and doesn't help me, is when I look at the video on my 4k HDR TV I can't help but stare at the grain noise. Really quite bad. But when I open the same video on my non HDR 24" computer monitor, I don't really see it.

[Boulder]

Does AVISynth+ have PQ-native filters? Filters that assume code values are gamma per Rec.709 often have weirdness and defects working with PQ.

PQ filters should really operate in floating point linear light.

Not that I know of - you can however convert to linear RGB for example with the HDRTools plugin and do things, I for example do all the downscaling that way but denoise and deband while the clip is regular 16-bit YV12 (Rec.709). I do not see a difference but then again, what I do is quite subtle and just try to press down the noise unnoticably to improve compression efficiency.

What's interesting and doesn't help me, is when I look at the video on my 4k HDR TV I can't help but stare at the grain noise. Really quite bad. But when I open the same video on my non HDR 24" computer monitor, I don't really see it.
Do you mean graded to SDR on your monitor? By which method?

[Stereodude]

What's interesting and doesn't help me, is when I look at the video on my 4k HDR TV I can't help but stare at the grain noise. Really quite bad. But when I open the same video on my non HDR 24" computer monitor, I don't really see it.
Do you have the sharpness cranked up on your TV?

[benwaggoner]

Not that I know of - you can however convert to linear RGB for example with the HDRTools plugin and do things, I for example do all the downscaling that way but denoise and deband while the clip is regular 16-bit YV12 (Rec.709). I do not see a difference but then again, what I do is quite subtle and just try to press down the noise unnoticably to improve compression efficiency.
PQ video is never Rec. 709. PQ has many more code values near black and a lot fewer around SDR's white. And PQ can go 100x brighter than nominal SDR (10,000 nits max in PQ, 100 in SDR). And with Rec 2020 primaries, HDR can have tons of colors that cannot mathematically exist in SDR, like bright violet and dark red.

The differences are fundamental, and filters that change code values presuming Rec 709 and gamma will result in quite different and less accurate results on the screen when applied to PQ.

[jriker1]

Not sure what this image repository will do to my images and wish my window in the kitchen wasn't covering parts of Dumbledore to show how crappy he looks but this is the original content unmodified at 78 kbps.

https://i.ibb.co/PhkTjYP/E93-DE812-FBA3-4-E8-D-A22-E-CFF02-C637-D35.jpg

https://i.ibb.co/7nnhXf6/D0-B3-F88-A-FC0-E-43-EB-A0-EA-2-C2-D0-E20-D730.jpg

I'll check the sharpness when the family isn't around so they don't get pissed off but think it's set to some LG OLED default setting. Talking like Movie, Game, etc but some LG one. Other things look good but this stuff is brutal.

[jriker1]

Submit separate details on this:

LG CX 77"
Picture Mode APS
Contrast: 100
Brightness: 50
Sharpness: 25
Color: 60
Tint: 0
Color Temperature: C20
Noise Reduction: Low
MPEG Noise Reduction: Low
Smooth Gradation: Low
Black Level: Auto
Real Cinema: On
Motion Eye Care: Off
TruMotion: Natural
Dynamic Contrast: Medium
Dynamic Color: Medium
Color Gamut: Wide
Super Resolution: Medium
Gamma: Medium
Peak Brightness: Off

[Boulder]

PQ video is never Rec. 709. PQ has many more code values near black and a lot fewer around SDR's white. And PQ can go 100x brighter than nominal SDR (10,000 nits max in PQ, 100 in SDR). And with Rec 2020 primaries, HDR can have tons of colors that cannot mathematically exist in SDR, like bright violet and dark red.

The differences are fundamental, and filters that change code values presuming Rec 709 and gamma will result in quite different and less accurate results on the screen when applied to PQ.

The resizing part is okay as far as I know, I've confirmed it with the author of HDRTools earlier. If you're interested, the docs are here:
http://avisynth.nl/index.php/HDRTools

What could cause any issues is the denoising part (I do debanding very rarely with UHD sources). I've never noticed any myself. Somehow I think that the re-encoding process itself is changing the image quite a lot more than a very restricted motion compensated denoiser.

[jriker1]

By the way, here is a small clip where I attached the pictures above. Is it very grainy for anyone else on an HDR display?

https://www.dropbox.com/s/w5hhqbs2trltu65/HarryPotter.mkv?dl=0

[Stereodude]

Submit separate details on this:

LG CX 77"
Picture Mode APS
Contrast: 100
Brightness: 50
Sharpness: 25
Color: 60
Tint: 0
Color Temperature: C20
Noise Reduction: Low
MPEG Noise Reduction: Low
Smooth Gradation: Low
Black Level: Auto
Real Cinema: On
Motion Eye Care: Off
TruMotion: Natural
Dynamic Contrast: Medium
Dynamic Color: Medium
Color Gamut: Wide
Super Resolution: Medium
Gamma: Medium
Peak Brightness: Off
Try turning the sharpness down to 0. Additionally, you have a number of image altering options turned on.

[Stereodude]

By the way, here is a small clip where I attached the pictures above. Is it very grainy for anyone else on an HDR display?

https://www.dropbox.com/s/w5hhqbs2trltu65/HarryPotter.mkv?dl=0
It looks pretty clean and not very grainy for me on my 75" Sony Z9D.

[Boulder]

Try turning the sharpness down to 0. Additionally, you have a number of image altering options turned on.Rtings.com usually has a good general setup based on their review, so it would be a good idea to also check them (they recommend Sharpness '0').

https://www.rtings.com/tv/reviews/lg/cx-oled/settings

[jriker1]

Rtings.com usually has a good general setup based on their review, so it would be a good idea to also check them (they recommend Sharpness '0').

https://www.rtings.com/tv/reviews/lg/cx-oled/settings

Thanks I saw that but it was in reference to SDR so wasn't sure it applied here. I know my cables are very high end and say they support up to 8K and HDR so guessing that's not a thing. I also opened Jurassic World Fallen Kingdom which is a known digital recorded movie and not with film stock. That picture looked fantastic on my screen.

[Boulder]

Thanks I saw that but it was in reference to SDR so wasn't sure it applied here. I know my cables are very high end and say they support up to 8K and HDR so guessing that's not a thing. I also opened Jurassic World Fallen Kingdom which is a known digital recorded movie and not with film stock. That picture looked fantastic on my screen.

To me it looked like they suggest the same base settings for both and then letting the TV automatically adjust the settings it does when the HDR mode is switched on.

[Balling]

"aware of any display that doesn't ignore master display metadata."

Would not it clip if it is set to DCI-P3 or BT.709?

[benwaggoner]

"aware of any display that doesn't ignore master display metadata."

Would not it clip if it is set to DCI-P3 or BT.709?
It'd have to be tested on different models. I would hope most TVs would do the right thing, but when given combinations of parameters that don't exist in real-world content, results could be unexpected or suboptimal.

[Balling]

It'd have to be tested on different models. I would hope most TVs would do the right thing, but when given combinations of parameters that don't exist in real-world content, results could be unexpected or suboptimal.

Well, if you have Planet Earth 2 Blu-ray with PQ that is what it is set to: all zeroes, so it does not clip :_) https://forum.doom9.org/showpost.php?p=1828960&postcount=48010

madvr also at least shows that it understands that 2020 ->> DCI does not happen in that case, but normal 3D LUT is applied...

BTW, do you have direct link to any documentation that talks about how to apply mastering metadata? SMPTE docs do not have this, alas.

[benwaggoner]

Well, if you have Planet Earth 2 Blu-ray with PQ that is what it is set to: all zeroes, so it does not clip :_) https://forum.doom9.org/showpost.php?p=1828960&postcount=48010

madvr also at least shows that it understands that 2020 ->> DCI does not happen in that case, but normal 3D LUT is applied...

BTW, do you have direct link to any documentation that talks about how to apply mastering metadata? SMPTE docs do not have this, alas.
For static metadata, zero is a special case. It means "undefined" not "none."

[Balling]

For static metadata, zero is a special case. It means "undefined" not "none."

I know. I read CTA 861-H.

[benwaggoner]

I know. I read CTA 861-H.
Sorry. My point was that we shouldn't assume that "0" will not cause clipping. I would hope that displays would do some sort of adaptive mapping. But if a TV assumes a MaxCLL of 1000 or something when unspecified, clipping is possible. Undefined behavior makes me nervous, especially with more and more companies making HDR TVs.