Fix distorted aliasing from bad upscales

[simple_simon]

Is there any way to reverse this?: https://ibb.co/G5kXwBz
It looks like the video was badly upsized from 480p to 1080p without correcting the existing line aliasing so now the aliasing is stretched and impossible to fix with normal dealiasing filters. I came up with this filter chain:

propSet("_FieldBased",0).deep_resize(720,240,grain=0,qual=2,gpuid=-1,show=false)
maa2()
deep_resize(720,480,grain=0,qual=2,gpuid=-1,show=false)
maa2()
lsfplus(strength=40,preset="slow",preblur="neo_fft3d(sigma=4,u=2,v=2)")
deep_resize(960,720,grain=0,qual=2,gpuid=-1,show=false)

which works decently but figured I'd crowdsource and see if there a better solution I'm unaware of.

[poisondeathray]

Downscale, perhaps using an inverse kernel / desampler, then use 1x_AnimeUndeint_Compact_130k_net_g, then upscale. Adjust to taste.

I fiddled a bit with a height below 480 which seemed helped better with some aliasing artifacts, you can play with that and other kernels as well. FMTC has inverse kernels too

https://imgsli.com/MjU2Mjcx


ImageSource("snapshot-02-39-411.png")
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ncnn(network_path="PATH\1x_AnimeUndeint_Compact_130k_net_g.onnx", builtin=false)
mlrt_ncnn(network_path="PATH\2X_DigitalFilmV5_Lite.onnx", builtin=false)
z_convertformat(width=1440, height=1080, use_props=0)


The pavement is slightly less red from one of the models. When you convert it back to YUV, smoothtweak(hue2=1) will make it more like the original

[simple_simon]

Those are a bunch of tools I'm not familiar with so it'll take me some time to play around with that (I don't even have a need to convert colorspace enough to know offhand how to convert back to YUV). But the results are damn impressive. Thanks for the suggestions.

[Emulgator]

Wow PDR, that is impressive.

[poisondeathray]

Those are a bunch of tools I'm not familiar with so it'll take me some time to play around with that (I don't even have a need to convert colorspace enough to know offhand how to convert back to YUV). But the results are damn impressive. Thanks for the suggestions.



What I mean is the models work in 32bit RGB float (the mlrt_ncnn steps). But usually people encode to 8 or 10bit 4:2:0 YUV (but some people use 4:4:4 when models are used because some of them sharpen the chroma significantly)

The output of the script as-is before the last line is 1440x800 because the downscale was 720x400, then a 2x model was applied

The last line resamples to 1440x1080 (to compare to same as input dimensions) . If you wanted 8bit 4:2:0 , you can add pixel_type="YV12" or whatever format you're encoding to .

Models are slow - you you could use deep_resize instead like you did before instead of 2X_DigitalFilmV5_Lite - whatever you feel like . Most of the problem fixing for the aliasing is from 1x_AnimeUndeint_Compact_130k_net_g on a downscaled input . It's really good model for anime or cartoon type aliasing issues

Before commiting to the Smoothtweak adjustment, I would examine other frames with other colors as well. Look at histogram("levels") and compare the U,V channels


z_convertformat(width=1440, height=1080, pixel_type="YV12", use_props=0)
Smoothtweak(hue2=1)



If you're stuck and can't find the answers just ask and someone will help out

[lollo2]

Downscale, perhaps using an inverse kernel / desampler, then use 1x_AnimeUndeint_Compact_130k_net_g, then upscale. Adjust to taste.

I fiddled a bit with a height below 480 which seemed helped better with some aliasing artifacts, you can play with that and other kernels as well. FMTC has inverse kernels too

https://imgsli.com/MjU2Mjcx

Excellent result, master!

[tormento]

ImageSource("snapshot-02-39-411.png")
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ncnn(network_path="PATH\1x_AnimeUndeint_Compact_130k_net_g.onnx", builtin=false)
mlrt_ncnn(network_path="PATH\2X_DigitalFilmV5_Lite.onnx", builtin=false)
z_convertformat(width=1440, height=1080, use_props=0)
Can AVSLibPlacebo or FMTC replace the 2 z_convertformat instructions?

[DTL]

You can even try internal and eveready ConvertBits(32) to go into 32bits float RGB.

[tormento]

you can even try internal and eveready convertbits(32) to go into 32bits float rgb.
rgb=rgbps?

[DTL]

RGB with 32bit (float) per sample is RGBPS. AVS do not support integer 32bit samples so ConvertBits(32) can only convert to float.

So you first convert to Planar RGB and next convert any sample format to float32. Typically RGB float32 is the only common format accepted by 'neural-networks' libraries as most fail safe and most dynamic range capable. It is not optimized for performance but easy to implement by current very poor residuals of opensource developers.

The last z_convertformat(width=1440, height=1080, use_props=0) do not change sample format and only resize so you again can use any internal resize (or any other installed).

To go back to some usable by encoders you anyway need to add something like ConvertToYV12() or if it can not make direct convert from float32 - ConvertBits(8).ConvertToYV12()

See example how to feed that NN-plugins with RGBPS format with internal conversions and back - https://forum.doom9.org/showthread.php?p=1983255#post1983255

[simple_simon]

The last line resamples to 1440x1080 (to compare to same as input dimensions) . If you wanted 8bit 4:2:0 , you can add pixel_type="YV12" or whatever format you're encoding to .

Models are slow - you you could use deep_resize instead like you did before instead of 2X_DigitalFilmV5_Lite - whatever you feel like . Most of the problem fixing for the aliasing is from 1x_AnimeUndeint_Compact_130k_net_g on a downscaled input . It's really good model for anime or cartoon type aliasing issues

Before commiting to the Smoothtweak adjustment, I would examine other frames with other colors as well. Look at histogram("levels") and compare the U,V channels

I've got this setup and have been playing around with it. The results seem like some sort of wizardry. It's really incredible. I can see these and other AI models being incredibly useful in restoring a whole stack of badly encoded dvd's and old avi files that no longer have original sources that I've never been able to fix in any sufficiently satisfying way.

As to this particular clip I didn't end up using the smoothtweak because looking at it in AvsPmod with Prefetch RGB Display Conversion turned on I didn't see any color difference before and after filtering. I made a few minor modifications to your script but can you verify whether I'm making any mistakes converting back to YUV?

propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()

[StvG]

The line z_convertformat(pixel_type="RGBPS", use_props=0) means colorspace_op="601=>rgb". If you want 709->rgb do z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left").

[simple_simon]

The line z_convertformat(pixel_type="RGBPS", use_props=0) means colorspace_op="601=>rgb". If you want 709->rgb do z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left").

Since the source is 1080p isn't it already in full colorspace? I wouldn't want to force limited-->full. Shouldn't it be full-->full?

[StvG]

The resolution doesn't define if the video is full or limited. If it's YUV (especially DVD or BD, or UHD) then it's usually (read 99% of the cases) limited.

[DTL]

Since the source is 1080p isn't it already in full colorspace? I wouldn't want to force limited-->full. Shouldn't it be full-->full?

It was a note like "Never use complex plugins with lots of hidden defaults left unchanged. The internal defaults may not match your source and no metadata magic may helps."

The main issue is not with range mapping but with system matrix. Your upscaled source may or may not use 709 hd-typical matrix (depending on previous history of conversion from sd source with sd-typical 601 matrix). And z_convertformat if options left unchanged will use 601 matrix.

"z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full","

For moving pictures typically used limited/narrow range mapping in YUV too. The RGBPS allows to keep all required data. So better to use z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:limited", . I also not know where do you going to use possible YUV420P8 format with full range mapping. It may be unknown to many YUV decoders.

There is unfortunately no any warnings output in AVS yet in some non-stopping of processing way so Avsresize do not emit stop-exception on the YUV+full format request and expect user understand how it will be looks like. In a more happy way it may simply ignore full-flag for RGB->YUV conversions.

The line z_convertformat(pixel_type="RGBPS", use_props=0) means colorspace_op="601=>rgb". If you want 709->rgb do z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left").

In the script of

ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)

the z_convertformat is used as RGBPx -> RGBP 32bit float conversion engine only. No de-matrix YUV to RGB. So it should not need matrix operation param ?

So user must provide correct matrix param to ConvertToPlanarRGB() instead of default left to have correct conversion.

[StvG]

... In the script of

ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)

the z_convertformat is used as RGBPx -> RGBP 32bit float conversion engine only. No de-matrix YUV to RGB. So it should not need matrix operation param ?

So user must provide correct matrix param to ConvertToPlanarRGB() instead of default left to have correct conversion.

Correct.

@poisondeathray's script is using ConvertToPlanarRGB() because his source is image (png - packed rgb).

@simple_simon's script doesn't need ConvertToPlanarRGB() and I have been thinking of:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()

[DTL]

The resolution doesn't define if the video is full or limited. If it's YUV (especially DVD or BD, or UHD) then it's usually (read 99% of the cases) limited.

Because current AVS interface do not support non-stop warnings - there is an idea to add bool param like 'treat_warnings_as_errors=(default)true' and throw stop-error if user provide any very highly looking like error combinations of params like YUV+full. And if user insist on the conversion - it is required manually force it by setting treat_warnings_as_errors=false.

[simple_simon]

@poisondeathray's script is using ConvertToPlanarRGB() because his source is image (png - packed rgb).

@simple_simon's script doesn't need ConvertToPlanarRGB() and I have been thinking of:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()

So are you now saying that the "z_convertformat(pixel_type="RGBPS", use_props=0)" line is correct but just need to remove "ConvertToPlanarRGB()"?

I already noticed I got an error when using:
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left")
so had to remove "ConvertToPlanarRGB()" already anyway.

[StvG]

So are you now saying that the "z_convertformat(pixel_type="RGBPS", use_props=0)" line is correct but just need to remove "ConvertToPlanarRGB()"?

No.
If the source is 709:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left")
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()

If the source is 601:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()

You can check with mediainfo what's the source matrix or you can with avs:
sourcefilter
propshow()

[DTL]

"If the source is 709:
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)

If the source is 601:
z_convertformat(pixel_type="YUV420P8", dither_type="error_diffusion").propSet("_ColorRange",1)
"

Why we have such big difference in z_convertformat arguments only with input matrix difference (and also this difference eliminated with conversion to RGB before NN-processing) ?

Also documentation http://avisynth.nl/index.php/Internal_functions#ColorRange says
_ColorRange
int _ColorRange
Full or limited range (PC/TV range). Primarily used with YUV formats.
0=full range, 1=limited range.

So with arguments
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)

the plugin simply ignores "=>709:709:709:full" and always outputs limited/narrow (for requested YUV420P8) so forcing of .propSet("_ColorRange",1) is valid ?

Documentation for ConvertBits() http://avisynth.nl/index.php/ConvertBits makes hints about (possible ?) valid YUV full-range for AVS+:
bool fulls = (auto)
Use the default value unless you know what you are doing.
Default value can come from _ChromaRange frame property
If true (RGB default), scale by multiplication: 0-255 → 0-65535;
Note: full scale U and V chroma is specially handled if false (YUV default), scale by bit-shifting.

Documentation on Avsresize http://avisynth.nl/index.php/Avsresize do not lists 'auto-ignoring of full range for YUV' and lists example of RGB to YUV420P8 as:
RGB to 8-bit YV12 (YUV 4:2:0 Rec. 709):
z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:limited")

Also if user later make resize to 720p hd-like size it may be better to make RGB to rec.709 conversion to be more compatible with decode/display devices ignores metadata ? So backward conversion to YUV420P8 may be better
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:601:601:full=>709:709:709:limited", dither_type="error_diffusion").propSet("_ColorRange",1) for rec 601 input matrix. If followed by resize to 960,720.

[simple_simon]

So with arguments
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)

the plugin simply ignores "=>709:709:709:full" and always outputs limited/narrow (for requested YUV420P8) so forcing of .propSet("_ColorRange",1) is valid ?

The reason I was using full-->full in the conversion to YUV was because of this note on the avsresize filter page:

Note that the examples above use full=>full since it does not truncate the brightest and darkest pixels, as opposed to limited=>limited.
https://forum.doom9.org/showthread.php?p=1906754&highlight=full#post1906754

And then the propSet("_ColorRange",1) makes sure it's correctly labeled as limited even though it was converted full-->full.

Testing in AvsPmod, both
z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:f=>709:709:709:l", dither_type="error_diffusion")

and
z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:f=>rgb:709:709:f", chromaloc_op="left=>left")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:f=>709:709:709:f", dither_type="error_diffusion").propSet("_ColorRange",1)

seem (to my eyes) to give the same results


I think the only thing I'm still unsure about is what "use_props=0" is doing in the rgb conversion. Is it just passing on any existing properties in the file or is it writing the new properties from the rgb conversion to the file? Should I be adding "use_props=0" to the YUV conversion line also?

[DTL]

"seem (to my eyes) to give the same results"

As expected it may mean plugin do not try to do 'full' mapping at converting to YUV420P8 and ignores this setting. In limited mapping you typically got 1.0f nominal white mapped to code value 235 of 8bit Y and with full mapping it must be mapped to code value 255 and you will see increasing of contrast (brighter whites) and clipping of possible super-whites presented in input RGBPS.

To see difference with YUV420P8 and 'full' mapping version you can try internal (to the end of script)
ConvertBits(8, fulls=false, fulld=true).

Or check difference at testscript:

ColorBarsHD()

fullmap=ConvertBits(8, fulls=false, fulld=true)

Interleave(last.propShow(), fullmap.propShow())


where fullmap clip is YV24 with full-YUV levels mapping (you can see how contrast increases).

""use_props=0" is doing in the rgb conversion."

It may force skipping of any input properties (and also may cause some performance increase ?).
http://avisynth.nl/index.php/Avsresize
int use_props =
Whether to read and set frame properties:
0: If frame properties are supported - only set frame properties.
use_props=0 is faster than use_props=1

"to the file?"

Intermediate script filters do not write to file - they can also read some input properties from input (previous filter) and set some properties for next downstream filter. If all next filters pass through the properties may reach AVS output and be read by file writing application (or ignored).

[simple_simon]

As expected it may mean plugin do not try to do 'full' mapping at converting to YUV420P8 and ignores this setting. In limited mapping you typically got 1.0f nominal white mapped to code value 235 of 8bit Y and with full mapping it must be mapped to code value 255 and you will see increasing of contrast (brighter whites) and clipping of possible super-whites presented in input RGBPS.

It shows the difference when doing limted-->full or full-->limted. I was just saying that when doing full-->full in both the rgb and back yuv conversions it seems to just maintain the source colorspace even if the source colorspace is limited and without any truncation of brightest/darkest pixels as referenced earlier.

[DTL]

"when doing full-->full in both the rgb and back yuv conversions it seems to just maintain the source colorspace even if the source colorspace is limited and without any truncation of brightest/darkest pixels as referenced earlier."

Yes - it also possible. But is cause passing of 'non-typical' float32 to intermediate plugins (with raised black level) and can (or can not) cause some issues. If you force treat limited YUV8bit as full at conversion to RGBPS it should map black code value of 16 to 16/219 float value instead of 0.0f. So some algorithms with black level search may not found it at all for example.

[simple_simon]

Note that the examples above use full=>full since it does not truncate the brightest and darkest pixels, as opposed to limited=>limited.

So I should just ignore this?

[DTL]

Note that the examples above use full=>full since it does not truncate the brightest and darkest pixels, as opposed to limited=>limited.

So I should just ignore this?

I think it may really depends on the NN-engine used in your avs-mlrt calls (up to exact versions of kernels/models used). Some may need 0.0f black. Some may really clips overwhites >1.0f and underblacks < 0.0f. Some may process all as expected. It may be only checked in the testing with your workflow (with exact models/kernels files and versions of plugins used that RGBPS format).

Everything with NN-engines may change every day and with every version. Because (some) developers of NN are far from real broadcasting and 'video industry' and typically develop engines just for profit (academic level raised) and report at computer-vision conferences and may test/train with PC-bitmaps of 'full-mapping'. Not with industry datasets with underblacks and superwhites.

That post from 2020 year may really had benefits with the plugins at that year and models/kernels used if being feed with some 'distorted' range with raised blacks and compressed whites (the float32 format allows to do it without adding significant quantization noise).

[StvG]

"If the source is 709:
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)

If the source is 601:
z_convertformat(pixel_type="YUV420P8", dither_type="error_diffusion").propSet("_ColorRange",1)
"

Why we have such big difference in z_convertformat arguments only with input matrix difference (and also this difference eliminated with conversion to RGB before NN-processing) ?

In order to use use_props=0 but...

The first conversion (709) has full->full that is wrong if you do YUV->RGB with limited->full. It should be full->limited. Also this line can be written - z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb=>709", dither_type="error_diffusion", use_props=0). propSet("_ColorRange",1) can be safely omitted.

The second conversion (601) is missing use_props=0. It should be - z_convertformat(pixel_type="YUV420P8", dither_type="error_diffusion", use_props=0). propSet("_ColorRange",1) can be safely omitted.

Also documentation http://avisynth.nl/index.php/Internal_functions#ColorRange says
_ColorRange
int _ColorRange
Full or limited range (PC/TV range). Primarily used with YUV formats.
0=full range, 1=limited range.

So with arguments
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)

the plugin simply ignores "=>709:709:709:full" and always outputs limited/narrow (for requested YUV420P8) so forcing of .propSet("_ColorRange",1) is valid ?

If you have YUV->RGB with limited->full and you do z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1) you will have full YUV420P8 (0-255) with wrong _ColorRange frame property that can lead to unexpected (wrong) results if further YUV->RGB conversion will be done.

If you have YUV->RGB with full->full and you do z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1) you will have the initial YUV420P8 range (usually 16-235) with correct _ColorRange frame property (if the initial input is limited).

I think it may really depends on the NN-engine used in your avs-mlrt calls (up to exact versions of kernels/models used). Some may need 0.0f black. Some may really clips overwhites >1.0f and underblacks < 0.0f. Some may process all as expected. It may be only checked in the testing with your workflow (with exact models/kernels files and versions of plugins used that RGBPS format).

Everything with NN-engines may change every day and with every version. Because (some) developers of NN are far from real broadcasting and 'video industry' and typically develop engines just for profit (academic level raised) and report at computer-vision conferences and may test/train with PC-bitmaps of 'full-mapping'. Not with industry datasets with underblacks and superwhites.

That post from 2020 year may really had benefits with the plugins at that year and models/kernels used if being feed with some 'distorted' range with raised blacks and compressed whites (the float32 format allows to do it without adding significant quantization noise).

For such use cases (AI models) it's recommended always to clamp [0,1] before processing the models:
YUV->RGB (limited->full or full->full)
Expr("x 0 1 clip")
mlrt
...

[DTL]

"For such use cases (AI models) it's recommended always to clamp [0,1] before processing the models:"

It looks like additional sign of severe degradation of residuals of former great civilization - the very memory costly RGBPS format (both RAM size and RAM performance costly) was expected to be most clear and perspective (up to linear-RGB forever with any HDR) and fail-safe. And in 2024 we see how with 'very perspective and cutting-edge NN/AI-technologies' it become again non-standard and distortive or error-prone. It is way back to the beginning of computer era for image processing with PC-RGB bitmaps of 'full' levels encoding (usable for static images processing usually as film scans).

Also sadly the AI/NN algorithms looks like can not make benefit of 'linear RGB' processing and/or input of the system transfer data to process nominal and super-whites range more correctly with the models where available.

Clamping of 'standard RGBPS limited/narrow' of 0.0f nominal black and 1.0f nominal white to 0.0f..1.0f will cause irreversible distortions of valid super whites and under blacks in the industry standards of moving pictures data encoding.
The hack of assuming 'full' mapping to real limited/narrow range may cause processing engine algorithms confusing with no-black present and low contrast (also degradation of 'nominal white' to lower code level). Some algorithms may treat this as additional 'distortion' and may try to 'fix' adding more distortions.

So the only current end-users way is 'try and check' either 1 or 2 method. Also as experiment the linear-RGB may be tested for quality if user uses z_convertformat to convert to and from RGB.

[StvG]

These super whites and super blacks are undefined (especially in float) and they are acceptable only for the final result. When you have filter accepting RGBPS input you expect the input to be in a standard range ([0,1]).

[DTL]

In the moving pictures industry the superwhites are used both for peaking 'video-look/makeup' code values storage (to make more 'sharper' look) and to store soft clipped super whites (sort of non-standard HDR compression in SDR footage). And underblacks are used for peaking code values storage at least. So with limiting code values to only nominal black and nominal white there is a danger to distort both shaprness and lost soft clipped highlights to more poorly hard clipped white patches.

Such clipping is mostly safe only for film-look/makeup (softer) without over/undershoots and specially mastered content with all levels limited to nominal white only. For hand painted animation use case in may be some valid solution without significant distortions.

Float completely safely keeps any superwhites and underblacks so it was expected to be the one and the only used samples format in the future moving pictures industry (when data systems can finally handle 4 bytes 32bits code words instead of old poor 1 byte 8(10) bit code words). After the 8(10)bit standrards in 19xx years the digital industry speed and storage volumes increased much higher than 4x times - but we still have some complex mix of 8 to 32bits formats.

As a one more intermediate solution we may use half-precision 16bit floats. They are also much better partially-shifted and compressed integer encodings. But it is new format for computing and not yet supported in AVS+.

For AI/NN plugins I hope we still have chances to get new models/kernels with proper support of 'unlimited float32 RGB' with simply defined nominal black at 0.0f and nominal white at 1.0f but with correct handling of all other valid for industry code values.

I also understand currently provided to 'video-users' AI/NN models/kernels may be as badly designed for such use case so may cause visible distortions if input data contains float code values below 0.0f and above 1.0f. So it only can be recommended to test both methods and select best performance with currently provided kernels/models.