AVS Softlight

[Argaricolm]

That's your personal choice. It doesn't mean your TV is doing something wrong just because you can override it to a setting that you think looks better.

In my case I can't override it.
Also i'v not mentioned it here:
If you watch video on PC using MPC-HC - its default renderer "MPC Video Renderer" will output video in limited range even if your video is full range and even if video has full range tag.
To watch full range video you need to select EVR renderer in options.

Also if you want to process limited range input and get limited range output - I think its better to convert it to full before and then back to limited.
Because neutralization functions are designed for full range. They get negative average sum this way: 255 - average.
I have not tested it with (235 - average) for limited input. And it will be more complicated, because limited input technically can have noise levels > 235. So for proper average calculations I need to cut those levels before sum.
Will try it later but not soon.

[hello_hello]

And also I'm not some guru, so as a normal user when I see some settings about full range or limited range I think that "full" is better than "limited". And I think most other normal TV users think like so.

TVs generally expect limited range for YUV and expand it to full range to display it. Likewise they should/could expect RGB to be full range and therefore shouldn't expand it.

My old Samsung TV doesn't have a setting labelled full/limited range either. Samsung decided to label it "HDMI Black Level", where "Normal" means the TV expects full range and "Low" means it expects limited range. It only applies when the input is RGB over HDMI though. Any other time it's greyed out as it expects YUV to be limited range, and it expects RGB at it's VGA input to be full range.

The LCD monitor connected to this PC does have a setting labelled RGB full/limited range for it's HDMI inputs.

And in result we see limited color range without convertion. So we watch data in its intermediate state designed to be converted to full range before output. And I think it's a strange design for our days when we have fast transfer speeds and space - no need for limited color range anymore.

As long as the output and input match in respect to levels the picture should look correct, and any input/output settings probably only apply to RGB over HDMI anyway. Here's a screenshot showing how to change the levels in the Nvidia control panel. https://i.redd.it/x44a5pwwned51.png

And here is example (https://imgsli.com/MjYyNTYz) of what we see and what we should see.

I'd place a bet on the limited range version being the correct one, although if your TV has a really horrible black level I can see why the full range picture might look better.

It's fairly easy to check when you're using a PC with MPC-HC, as it has pixel shaders for changing the levels. If you play a video with black borders encoded and you tell MPC-HC to expand the levels with a pixel shader.... if the picture gets darker but the borders don't then they were already black and the levels were already being expanded. Like this:

https://imgur.com/PrVqOYK.png

https://i.ibb.co/W0W98WB/1.jpg

https://i.ibb.co/VMNM2CR/2.jpg

However if the black borders do get darker then the levels are wrong.

Here's a small YV12 video you can try. There's a histogram on top.
The first 500 frames have the correct levels. Black=16.
The next 500 frames have those levels expanded. If it's being displayed correctly the black borders won't/can't get any darker even though the picture will.
For the last 500 frames a PC to TV conversion was applied so the black borders should be dark grey rather than black and the picture will look a bit washed out. If the borders still look black rather than dark grey it means the levels are being expanded unnecessarily, maybe due to a mismatched input/output.

Black Test.mkv (https://files.videohelp.com/u/210984/Black%20Test.mkv)

[Argaricolm]

I'm making an update with support for lossless yuv<-> rgb convertion. And also I'v added rec 601 & rec 709 so far (not published).
What will be better:
1) Use based on height: Rec601 when height <= 576; Rec709 for 577-1080; Rec2020 for > 1080
2) Or use rec 709 always by default.
It will be possible to select input rec and output rec separately.

[Argaricolm]

Black Test.mkv (https://files.videohelp.com/u/210984/Black%20Test.mkv)

Tried this video from TV and from TVbox.
My TV shows it same way as on PC.
I see limited range in first part (with bad blacks), PC range in middle (ideal) and bright last.
No settings in TV about color range, no settings in any android video player about it. Only "adaptive lightness control" in TV - but it does a little different work and it change full range too.

Looks like I'm stuck with converting video or playing it from PC.

[DTL]

I'm making an update with support for lossless yuv<-> rgb convertion. And also I'v added rec 601 & rec 709 so far (not published).
What will be better:
1) Use based on height: Rec601 when height <= 576; Rec709 for 577-1080; Rec2020 for > 1080
2) Or use rec 709 always by default.
It will be possible to select input rec and output rec separately.

For fail-safe operation it may be better to disable any auto-estimation and force user to provide required param.

You can set init value to -1 for example and if it is left as default at class constructor - throw an error about required param not set.

Also if you want to try new AVS+ features you can read frame/clip property of the colour matrix (if set). So possible ways are:
1. Force user input only.
2. Auto-estimate based on the frame height.
3. Read input frame/clip properties.

[Argaricolm]

Example (https://imgsli.com/MjY4NzIz)of OETF function.
It makes all movie bright like on the Sun...

[Argaricolm]

Version 1.15 (https://github.com/ArturAlekseev/AVS_SoftLight/releases/tag/v1.15) is out.

[Selur]

yuvin & yuvout options are used for modes where yuv <-> rgb conversion is used and they define formula used for decode and encode 0 = Default is Rec.709. Or you can select 601, 709, 2020. Like Softlight(yuvin=601,yuvout=601)
okay, 0 = Rec.709 What are 601 and 2020?

[DTL]

It looks current version uses direct numbering - https://github.com/ArturAlekseev/AVS_SoftLight/blob/a9f046fd8a60d187dc9364197601bced443ca4dd/kernel.cu#L2892

So 0 and 709 are the same and others are 601 and 2020 integers for rec.601 and rec.2020 matrices. Any other integer numbers except 601 and 2020 will cause default to rec.709 matrix ?

[Selur]

@DTL: Thanks.
Any other integer numbers except 601 and 2020 will cause default to rec.709 matrix ?
yes.

[Argaricolm]

Anyone knows something about OETF and EOTF functions?
The interesting thing here for me is:
From their description in docs I'v found that OETF is used to encode original light and EOTF is reverse.
So first I thought that if I use EOTF on data I will get "original lightness of the scene". But EOFT function makes everything MUCH darker. And OETF makes brighter. It can't be that original scene lightness was so dark.
Or it is just a misunderstanding because of "light" differs from "electric signal" - so this functions are not designed to be used such way (maybe they are designed to be used only inside camera and inside display hardware).
But then it is strange that OETF function gives great results on some content.
For example Divergent 2 and 3 blurays processed by it (Rec.709 limited -> full range -> OETF) results it bright picture but with good contrast. While I'v tested on other BDs and result are just bright picture with lost contrast (its possible to get better contrast if limited -> full conversion is used twice before OETF, but this burns a lot of darks). So it looks like for some content there should be another OETF function. But same OETF is specified for Rec 601 and Rec 709.

[DTL]

"But EOFT function makes everything MUCH darker. And OETF makes brighter. It can't be that original scene lightness was so dark."

Your display device doing EOTF for you (and it is at current PCs typically rec.709/sRGB EOTF). So if you apply EOTF to data in the distribution domain and feed to standard display - you got EOTF applied twice and darker displaying.

To check linear light values you need to understand how digital values maps to physical light. And inspect digital values directly (or switch your E->O display device to linear EOTF if possible).

" looks like for some content there should be another OETF function."

Most of cine-titles (not direct live broadcasts) are mastered not with real scene light but after complex colour-grading process for artistic reasons and by choice of product director. So usase of standard EOTF and standard display only guarantees you will see same optical displaying of the content as was expected at production.

For live broadcasts depending on camera settings you can get real OETF only tracking and can restore linear light using EOTF transform. But real OETF tracking for physical production camera rarely usable (only in some 100% controlled studio lighting) so for real ENG and other open air shooting cases some (unknown) non-linearity added like KNEE (and AUTO KNEE) control to soft-compress highlights and this data not sent as metadata with encoded content so you can not restore full linear light with EOTF only.

Only for special closed digital imaging like medical highly linear path may be used from camera to display so doctor can see as much real image as possible. Most of other digital imaging are somehow distorted at production to have 'more nice look'. Same and much more changes applied to colours view.

You can take some digital camera with RAW output and look how awful is it looks in both colour and tone in most of real life scenes.

[Argaricolm]

I'v tested my beelink android TV box and MPC player again. This time with my own specially created files.
Results:
TV box does not care about HEVC flag - full or limited color space. If video is in full space but flagged limited - it will show it full anyway.
But looks like it cares for range itself. So I'v created image with half RGB(16,16,16) and half RGB(235,235,235). And then it shows it as black and white (really white not some gray nearly white).
So looks like TV box can fail in limited color space determinations if first frames will contain values < 16.

MPC player:
* MPC Video Renderer cares about full or limited color space flag and displays correctly.
* MadVR is same. (But with decoding AV1 its faster and does not freeze). So I better use it.
* EVR Renderer does not care about it. It always rerange YUV from limited to full. If video is already full - it will rerange it again.
Same in Staxrip (avisynth) - if source is YUV - preview will always be reranged to full.

[Argaricolm]

What bothers me now is that why twice reranged video to full looks more like it should be so:
Limited -> Full (https://imgsli.com/MjcyMzg0)
Full -> Twice Full (https://imgsli.com/MjcyMzg1)

[DTL]

Content mastered at master control monitor. See EBU Tech 3320 and 3325 for requirements and testing as example.

https://tech.ebu.ch/docs/tech/tech3320.pdf
https://tech.ebu.ch/docs/tech/tech3325.pdf

These documents also list EOTF and test methods and requirements for different tiers of quality. To see how it was mastered you need to simulate master control monitor and viewing environment (different for SDR and HDR too). Also you can tweak as you want with local display controls - it will be your local version of image.

[Argaricolm]

New release v1.16 (https://github.com/ArturAlekseev/AVS_SoftLight/releases/tag/v1.16).

Now you can easily get limited range back like so:
softlight(8) = full range
softlight(3) = average
softlight(9) = back to limited

You can change default range parameters for limited range like so: Softlight(8, rangemin = 16, rangemax = 235)
Its needed on some sources to play with OETF function (it needs black to be absolute - or contrast will be lost).

Also added softlight(7) mode. Some video sources with limited range contain values < 16 and > 235. This mode will change them to 16 and 235. This will ensure correct range identification by your decoding hardware (seems it looks at first frame for values outside limited range to identify video as full range). Plus it will allow to compress your video better.

[Argaricolm]

OETF makes magic in Alien 1979 without any levels correction.
Just > full range > OETF and much more details are visible without contrast lost. Example (https://imgsli.com/MjcyOTE0)

[DTL]

It looks like you do not understand why users need some levels and colour correction plugins. The studio mastered titles are always perfectly mastered for colour and tone so do not need any corrections. You can read also the article from Ken Rockwell how studio shootings always get perfect colour and tone - https://www.kenrockwell.com/tech/highlight-shadow.htm
But the job of studio colour grading costs a lot.

On some days (like 19xx..201x) there were home end users video unexperienced shooting with bad natural lighting on cheap hardware and bad auto-exposure and bad auto-colour etc. This result in bad colour and tone in most cases. So users typically want to make old home and amateur recordings better using some free tools. Thus to show how the tool can help end users with badly recorded home video it is better to use the real samples - not perfectly mastered studio titles. Some real content may be downloaded from forum posts at videohelp forum where users show some bad footages for repair.

[Selur]

I'm a bit confused what mode 8 (tv2pc) does.
8 mode: TV to PC color range conversion (use it on videos where you see no total black and only grays).
I expected it to do the same as:
clip = core.std.Levels(clip=clip, min_in=16, max_in=235, min_out=0, max_out=255) scaling tv to pc scale.
but:
clip = core.std.Levels(clip=clip, min_in=16, max_in=235, min_out=0, max_out=255)
clip = core.Argaricolm.Softlight(clip, mode=11, yuvin=601, yuvout=601)

gives me:
https://i.ibb.co/KLsy80t/grafik.png (https://ibb.co/YydjVR5)
while:
clip = core.Argaricolm.Softlight(clip, mode=8, yuvin=601, yuvout=601)
clip = core.Argaricolm.Softlight(clip, mode=11, yuvin=601, yuvout=601)
gives me:
https://i.ibb.co/Jym72CH/grafik.png (https://ibb.co/xfXjG36)

I also looked at:
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
clip = core.Argaricolm.Softlight(clip, mode=11, yuvin=601, yuvout=601)
https://imgsli.com/MjcyOTUw

=> any inside on this? What does tv2pc do?

Cu Selur

[Argaricolm]

TV2PC changes range this way for 8 bit (in RGB for each pixel):
(R/G/B - 16) / 219 * 255
if input is yuv - it is converted to rgb for processing (without changing range)

As for core.std.Levels maybe you need to rerange channels in yuv differently (from vp docs):
clip = std.Levels(clip, min_in=16, max_in=235, min_out=0, max_out=255, planes=0)
clip = std.Levels(clip, min_in=16, max_in=240, min_out=0, max_out=255, planes=[1,2])

As for examples. I see difference here too.
AviSynth and VapourSynth give different results (https://imgsli.com/MjczNTQx).
Btw you should rerange your result back to limited range using softlight(9)
Otherwise you will have full range in yuv. If you preview it in staxrip (and maybe other software) you will see it twice reranged to full (for me preview is always reranged from limited to full - if I already have full - in preview I see it even more dark). To not have such issue you should try to work in RGB space.

As for different result in VapourSynth looks like a bug in VapourSynth (thou its strange).
Here what I'v found.

Using this script:
import vapoursynth as vs
from vapoursynth import core
core.std.LoadPlugin(path='c:\\..\\softlight.dll')
clip = core.std.BlankClip(width=640,height=480, format=vs.YUV420P8, length=500, fpsnum=2997, fpsden=125, color=[16, 128, 128])
clip = core.Argaricolm.Softlight(clip, mode=8)
clip = core.Argaricolm.Softlight(clip, mode=11)
clip.set_output()

When I open in VirtualDub - mode 8 is never executed.
Y = 16 is passed to mode 11 resulting in 55
Then 55 is passed again to mode 11 resulting 115
So it does mode 11 twice without doing mode 8.
I don't see how it can be from my code.

If only mode 8 is in script - it is executed correctly resulting 0 in Y.

If I do mode 8 then 11 then 9 - only mode 9 is executed 3 times.
I'll try to update VapourSynth (have R65).

[Selur]

Here what I'v found.
Okay, if multiple calls to Softlight (in Vapoursynth) atm. result in just the last mode called multiple times, I will stick to calling it only once.

I'll try to update VapourSynth (have R65).
I'm using R68.

[Argaricolm]

Fixed it.
v1.17 (https://github.com/ArturAlekseev/AVS_SoftLight/releases/tag/v1.17)

The problem was in different parameters passing in VapourSynth than in AviSynth.
AviSynth uses different objects for each call - so there I use object variables to store parameters.
But VapourSynth uses same object for different calls.
It first calls filterCreate for each call (in same object). So last call replaced previous parameters. And after that it starts to call filterGetFrame.
So I made it to pass parameters for each vsapi->createVideoFilter call.
Looks like it is not a bug. Strange, that filter skeleton does not have parameters passing example.

[Argaricolm]

The studio mastered titles are always perfectly mastered for colour and tone so do not need any corrections.
I can understand that they master color & tone perfectly. Or it may be director vision of the picture.
What I don't understand is that why nearly all latest titles are so dark.
Same goes for old titles too.
I don't think the problem was with the lack of light.

[DTL]

To display dark scenes - the lower code values of standard 8bit encoding used. The sRGB and rec.709 dynamic range with 8bit encoding is about 1000:1 (as ratio from first non-zero code value of 17 to nominal white of code value 235)so for daylight scenes higher part used and for night scenes - lower part. The total range is fixed in relative lighting (not as possilble with Dolby HDR with dymanic metadata ?) so it is easy enough to show dark scenes as dark and bright as bright. But this only work best in required viewing conditions - see viewing conditions for HDTV ITU-R BT.2022 https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.2022-0-201208-W!!PDF-E.pdf and ITU-R BT.2035 https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.2035-0-201307-I!!PDF-E.pdf and also check your display gamma-tracking for rec.709 EOTF first and also display nominal white level (or peak luminance) if you do not see dark scenes correctly.

Typically for end-users HDTV and rec.709 -
(BT.2022)
1.1 General viewing conditions for subjective assessments in a laboratory environment
The assessors’ viewing conditions should be arranged as follows:
a) Room illumination: low
b) Chromaticity of background: D65
c) Peak luminance1: 70-250 cd/m2
(See § 1.7.2)
d) Monitor contrast ratio: < 0.02 (See § 1.7.1)
e) Ratio of luminance of background behind picture monitor to
peak luminance of picture:
~ 0.15
1.2 General viewing conditions for subjective assessments in a home environment
a) Environmental illuminance on the screen (incident light from
the environment falling on the screen, should be measured
perpendicularly to the screen): 200 lux
b) Peak luminance1
: 70-500 cd/m2
(See § 1.7.2)
c) Ratio of luminance of inactive screen to peak luminance
monitor contrast ratio:
< 0.02 (See § 1.7.1)

(BT.2035)
1.1 Viewing environment for subjective assessment
a) Room illumination: 10 Lux
b) Chromaticity of background: D65 (optionally D93 in some
regions)
c) Ratio of luminance of background behind picture
monitor to peak luminance of picture:
≈ Between 10% ±2% of reference white value

10 Lux of room illumination is really very dim lighting (indoor and after sunset typically in room with windows).

"I don't think the problem was with the lack of light."

It may be with too many light in your viewing environment and too low display contrast (or too low display peak brightness too). Too much external lighting shifts viewer's brightness of adaptation to high levels and viewer lost ability to see low brightness levels (if even display tracks EOTF completely perfect).

[Selur]

Fixed it.
v1.17

Thanks, I can confirm it works fine now.

Cu Selur

[Argaricolm]

Another question.
Does anyone know why some sources have incorrect range?
For example this:
Solo A Star Wars Story.
This is a frame from bluray.
As you can see on the frame with a lot of blacks the lowerst value is 22. But should be 16 (at least some pixels should be absolute black).

For comparison same frame reranged:
softlight(8,rangemin=22,rangemax=235)
softlight(9)

Screenshots are from YUV (so you see both of them as they will appear on TV).

And this is a "new" movie. So this is not a result of something wrong. By unknown reason this range was intended.
I'v already seen some of such sources where lowerst range limit is ~22 instead of 16.

Example (https://imgsli.com/Mjc3ODYx)

By the way. If you want normal blacks after softlight(1 or 3 or 11) you should rerange such sources this way.

[Emulgator]

But should be 16 (at least some pixels should be absolute black)
I would not assume that for every frame, given that the lighting intent of your sample frame might well have been "close to dark, but just not sunk into blackness"
Watched on a OLED it makes sense, watched on a TFT you may want to lower that black because TFT blacks are poor to begin with.
But tying that to 16 for all audiences means loss of shadow detail, and crushed blacks on any slighty misadjusted monitor.
Just had such a source where this was attempted, the real blacks were sitting around 8. Was hard to reverse without banding.

[Argaricolm]

I'v found that some sources are twice converted to limited range (full -> limited -> limited).
To determine such issue you should check first black frames of video for levels (can do so using ShowChannels plugin). Normally first black frame should be (Y = 16, U = 128, V = 128). It may differ a little due to compression.
And normally each video should contain such totally black frame or frames at the begining. This signals hardware that video is in limited range.
When your hardware decodes video and displays - it always converts limited range to full before displaying.
But in case of twise converted range - hardware converts twise limited range to limited range and displays limited range to you.
For example (https://imgsli.com/MjkyNzc3)is the first season of Rick & Morty that has Y = 30 in first key frames. I have not checked all seasons, but 6 & 7 do not have such issue.

[Argaricolm]

New version 1.19 (https://github.com/ArturAlekseev/AVS_SoftLight/releases/tag/v1.19-release2).
Critical fix in TV2PC 10 bit color range conversion function. It was working wrong.
Added fullrange option.
By default functions will treat source as limited color range. It will be converted inside. Only OETF & EOTF functions are same.

So now you can do just:

softlight(3)

instead of:

softlight(8)
softlight(3)
softlight(9)

If you want to do same as before:
softlight(8)
softlight(3,fullrange=1)
softlight(9)

This is so only for YUV color space, because it is treated as limited color range by default.
I'm thinking to make same changes for RGB color space (add ability to change color space inside call). But by default it is treated as full range. Maybe I should do it reverse way (do not change it for RGB by default, but change it if fullrange = 1).

[Selur]

My 2 cents about this:
when:

fullrange = 0, SoftLight should assume input is 16-235 and its output should be 16-235 (unless TV->PC conversion is used)
fullrange = 1, SoftLight should assume input is 0-255 and its output should be 0-255 (unless PC->TV conversion is used)

[Argaricolm]

Made a new release. Fullrange is changed to "changerange". By default (0) it will treat YUV as limited range (and will rerange for processing) and RGB as full range (and will not rerange it). OETF and EOTF functions don't use this param and will treat any source as full range (you need to rerange it to full and back yourself using 8 & 9).
This param will correspond with my plugin ImageSourceNV where I make CUDA conversion functions. Will publish it's updated version soon.

[Selur]

So, if I make sure that my source is full range for all modes != 8 and limited range when using mode = 8, can simply skip the 'changerange' parameter?

[Argaricolm]

If your source is YUV and it is in full range (that is not normal), then you should use changerange=1. Otherwise it will rerange it from limited to full and back to limited.
If your source is YUV and it is in limited range (that is normal), then you should not use changerange.
If your source is RGB and it is in limited range (that is not normal), then you should use changerange=1.
If your source is RGB and it is in full range (that is normal), then you should not use changerange.

Simply said YUV is treated as limited range and RGB is treated as full range by default. And if it is not so, then you need to use changerange.
8,9,10,11,12 modes don't use this parameter.

[Selur]

okay,... how about adjusting the filter, so that the user, just reports whether the source should be treated as limited or full and the filter does rest and output the same range it was fed?
Current handling seems to be unnecessarily complicated.

8,9,10,11,12 modes don't use this parameter.
Does this mean they work without any requirement, that they always require limited or full range, or what?

Cu Selur

[Argaricolm]

Well, it's hard to determine is input full range or limited range.
Ideally limited range should not have values from 0-15 and from 236 to 255 (in 8 bit). I'v also read that values near 0 were used for some sort of TV signal syncronization.
But in real life limited range source can have such values by unknown reasons. Maybe related to compression. Maybe related to buggy encoding software. Some sort of noise.
And the only way I'v found to determine range is to look at it with eyes and see.
Mostly you will not find source in full range, because nearly all content I see is encoded in YUV and it is by standard that YUV is limited range. Thou technically it can contain full.

Plugin does same as you said, just for default it does not need range to be specified. YUV -> limited, RGB -> full.
And if by some reason you have somethig weird like YUV full range, then you use changerange.
So normally you don't need to use it :)

Example when I use it myself:
I have avisynth plugin that works only for RGB source. And I just like to use it on source without range conversion.
So I use my ImageSourceNV plugin to convert YUV to RGB without range conversion. So I get RGB in limited range (that is weird). Then I use this plugin. And then I use ImageSourceNV to convert limited RGB back to limited YUV.

For 8-12 modes. They just work same way no matter what input range you give.
If you use 8 on full range input - it will be treated as limited. So 0-15 and 236-255 levels will be just lost (cut) and middle levels will be reranged.
Same for mode 9. If you give limited range input to it you will get twice limited output. The weird thing is that I'v already found a lot of video sources with twice limited range. And the even more weird thing is that such sources are not from some home video. They are from TV, they are from streaming services and they are even from blurays!
I'v even found some cartoons, that some studios do in twice limited range. And they stream them in such way in streaming services. I don't get it why.
Twice limited range video has its lowerst blacks near 30 (instead of 16).
Example (https://imgsli.com/MzUxMzUz). <- and it is from bluray!

But also there are some sources that looks like twice limited, but they are not.
Such example is "Spy x Family" anime. Its levels look like twice limited. If you will use mode 8 on it you will get colors closer to primary. But on dark scenes you will see that too much visual information is not seen. So cartoon was designed to be in that range. And it's not some fault.

For 11 & 12 source should be full range. Otherwise result will be more weird. Image will be very bright after mode 11 if source is limited range. But you should not use 11 & 12 in some production. It just to play with. Mostly it gives weird results. But I'v found some sources where mode 11 do cool results:
5to7 (https://imgsli.com/MzUxMzQ4)
Insurgent (https://imgsli.com/MzUxMzQ5)
Insurgent (https://imgsli.com/MzUxMzUx)
But if you use mode 11 on whole movie you will get terrible result.

And mode 10 just removes colors. Range is not changed.

And for automatic detection:
I'v found that video sources mostly use first black frames to show in which range video is encoded. That's why most sources have black frames at start. But I'v also found sources in twice limited range that had these key black frames in 1 time limited range. So it's not very reliable. Much easier to do it by eyes and hands.

[Argaricolm]

I'm thinking to change mode 1 (already changed for myself and now testing). I want your thoughts about it.
So far in mode 1 after all RGB channels are processed by softlight - Saturation and Volume (from HSV) are restored to original. So hue (colors) and saturation are not changed, but only Volume. This causes image to be mostly brighter (in most cases source is darker). But it also causes result image to become oversaturated. Because original saturation is for darker image and it looks like for brighter image saturation should be lower. I have changed it to not restore saturation. So now it is also changed. It shows results like mode 3, but because Hue is original results have original colors (no greeny faces).
The question is - do you need original mode 1 or I can replace it with a change?

[DTL]

.
But in real life limited range source can have such values by unknown reasons. Maybe related to compression. Maybe related to buggy encoding software. Some sort of noise.


There are lots of reasons to have code values +-nominal black value in narrow range encoded files. One is to save from false DC offset from nominal black by cutting out 'negative' code values added by real noise or dithering process or something else.

Example:

For samples sequence

14, 16, 18

Average value is 16 and it is good black.

If you hard limit 14 under-black to 16 the output will be

16, 16, 18

Average is (16+16+18)/3=16.6 rounded to 17. So we got lost (damage) of contrast by increasing average black to about +1 code value.

[Selur]

Well, it's hard to determine is input full range or limited range
okay, then simply let the user set whether the input is full or limited and do the rest internal instead of having the user jump through hoops.

[Argaricolm]

Technically value 14 in limited color range should not exist at all. Because it is LIMITED to 16-235. And I'm talking about RGB. As it is limited in RGB. Because 0-255 range is reranged to 16-235. So 0 will become 16.

Limited range is not something that exist by itself. It is a middle in transformation.
Record Full -> Limited -> Stream -> Limited -> Full for Viewer.
And Full RGB can't be correctly converted to limited to have values < 16 or > 235. Because RGB can't be lower than 0. And 0 will be 16.
Because rerange from full to TV is R / 255 * 219 + 16.

And Y will not be < 16 in this case. Because minimum RGB in limited range is 16 16 16.
It can become lower only because of rounding problems (or maybe also because of YUV compression/decompression). Rounding problems like here (https://www.mikekohn.net/file_formats/yuv_rgb_converter.php). Where he uses Math.floor for rounding and gets Y = 15 for RGB (16,16,16).

On another part TV to Full conversion must cut all values <16 or >235. Or result will be incorrect.
This way:
(X - 16) / 219 * 255
And in the end we do clamping to 0,255 (just to be sure no negative values). Because in RGB negative can't exist.
So if you have 14 in limited range. You get 0 in full. You get 0 for 0-16 (limited) and 255 for 235-255 (limited).
You can test it using photoshop + imagesource in avisynth.
I'v created image.mkv (https://disk.yandex.ru/d/PAbbpzvGJGp3NQ) for this test.
First 5 frames are full RGB 16 16 16. To be key frames for hardware to see it is in limited range. Other frames has lines paint in photoshop that is of color (10,10,10).
If I open it in mpc-hc (MadVR decoder) it treats video as limited range and all frames are complete black. You can pause at any frame and save image as bmp. All bytes will be 0 (except only few some that are 01) - must be because of compression/decompression.
But I'v also opened it in my android tv device in Kodi (android tv version). And there it is different.
I can see lines on TV and they are black. So Kodi somehow use 10 as the lowerst limit of range (10 becomes absolute black). Maybe Kodi is not the cause because it uses hardware here for AV1 decoding.
This is example why <16 should not exist in limited color range video. Because this noise may be treated as color info and it will ruin contrast and colors because of wrong rerange. What should be black is treated as gray.

[DTL]

"Because it is LIMITED to 16-235."

It is not hard limited - it is narrow range. Where 16 is nominal black (physical zero) and 235 is nominal white. The digital domain is not completely equal to physical light domain so it can have 'negative' non-exist in positive light physics code values. But they are required for many math operations in digital domain to keep quality. So valid code values for narrow 8 bit digital are at least from 1 to 254 if compatible with SDI or from 0 to 255 if compatibility with SDI interface is not required.

"Because in RGB negative can't exist."

You still mix digital RGB and physical light RGB. The digital domain RGB can be any signed value (negative too). Only if you go at the end of digital chain to physical light RGB you make cutting out (clamping to zero) negative non-possible to display RGB power values.

"You can pause at any frame and save image as bmp. All bytes will be 0"

Typically personal computers are end-users devices to playback to the RGB physical display and they like to use 'monitor-RGB' (usually sRGB) with black at code value of 0 and no negative RGB encoded. But it was from old days when PCs where too slow for moving images processing and process only static images encoded in different from moving pictures digital. sRGB really encodes black at zero.
Nowdays if you run some software player of digital moving images (encoded in narrow YUV) it make decode and scale to PC monitor resolution and send data in sRGB domain with zero at 0 (ready to display at physical monitor without scaling). If you grab bitmap from operating system in RGB (PC sRGB) you got black at zero and no under-blacks.

If your display support YUV narrow or RGB narrow you can try to grab this data from frame buffer and see the negative under-blacks in it. It is when you connect some device like TV set/panel via HDMI and feed YUV 4:2:2 (or 4:4:4) or RGB narrow to the device.

If you work with digital moving images you typically use industry standards like ITU bt.601/709/2020 and they uses different physical levels to code values mapping (with negative Y and RGB values too). If you create software for moving pictures processing at computers it is good to start from learning industry standards for digital levels encoding and colour space conversions. They are free and open for download from ITU website.

"it's hard to determine is input full range or limited range."

You can try simple statistical analysis:
1. Count number of samples equal to 0 to Cnt_0 and equal to 16 to Cnt_16.
2. If Cnt_0 > Cnt_16 - clip mostly probably uses 'full' levels maping scheme. Else - 'narrow'.

You can do counting in the single pass using SIMD of comparison with constant 0 and constant 16 to mask and masked addition of 1 to counters.

It is not best way for very low noise content. For low noise levels it need some changes (like addition of nosie with average delta value about 16/2 ?).

From the math point of view the task is simple enough - you need to create computationally cheap and SIMD-friendly algoriphm to discriminate between 2 possible distributions of random value in the range 0..16+7 :
1. If the max number of samples are located in the code values range 0..8 - result if 'full'
2. If the max number of samples are located in the code values range 16+-7 (9 to 23) - result is 'narrow'

May be it is enough to calculate some math average in the range 0..16+7 and compare with middle value of 8. If it is below 8 - the result is 'full'. Where average may be some of mean or median or mode (see in the wiki how they are calculated - the 'mode' implementation with SIMD available in vsTTempSmooth plugin in Asd-g github - https://github.com/Asd-g/AviSynth-vsTTempSmooth/blob/master/src/vsTTempSmooth_pmode1_AVX2.cpp ).

Some description for 'math center' calculations - https://www.khanacademy.org/math/statistics-probability/summarizing-quantitative-data/mean-median-basics/a/mean-median-and-mode-review

Mean, median, and mode are different measures of center in a numerical data set. They each try to summarize a dataset with a single number to represent a "typical" data point from the dataset.
Mean: The "average" number; found by adding all data points and dividing by the number of data points.
Example: The mean of 4, 1, and 7 is (4+1+7)/3 = 12/3 = 4.
Median: The middle number; found by ordering all data points and picking out the one in the middle (or if there are two middle numbers, taking the mean of those two numbers).
Example: The median of 4, 1, and 7 is 4 because when the numbers are put in order 1, 4, 7, the number 4 is in the middle.
Mode: The most frequent number — that is, the number that occurs the highest number of times.
Example: The mode of {4, 2, 4, 3, 2, 2} is 2 because it occurs three times, which is more than any other number.

The 'mode' analysis allows to get some data about reliability of result - if there is 'no mode' in samples dataset it mean the detection is not reliable and other dataset (frame or set of frames area) need to be checked. Example sequence 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 have 'no mode'.