fmtconv/Avisynth+: resize, bitdepth and colorspace conversions

[cretindesalpes]

tormento:

In [2], replace the last line with:

fmtc_bitdepth (bits=8, dmode=8, fulld=true)
propSet ("_ColorRange", 1) # Limited

The reason:

AviSynth+ uses full and limited ranges with float, using 16/256 and 235/256 [Edit: /255, not /256] for black and white in limited range, same with chroma data. Avs+ fmtconv is derived from its original Vapoursynth version, where nominal range for float is always [0 ; 1] and [-0.5 ; +0.5] for chroma information, and full/limited concept doesn’t apply (however range information can be carried as frame property).

So, just before the bitdepth conversion, we have a YUV clip in “limited” float. fmtc_bitdepth cannot handle such a range: converting gives a double-limited range. However it is possible to restore the single-limited range at the function output by using the inverse factor, therefore converting to full range with fulld=true. This is a dirty hack, it requires to overwrite the frame property afterwards, but it should work.

[tormento]

replace the last line with
Thus doing, can I use standard convertbits(32) or have I still to use the fulls fulld switches?

[cretindesalpes]

You can do whatever you want, both configurations should be handled gracefully by the subsequent ConvertToPlanarRGB(), giving the same output.

[DTL]

"#We go to 32bit float limited tv range
ConvertBits(32)
"

It looks the 'float32' format in AVS may be very different between authors ? So not compatible between different filters/plugins/AVS-internal.


"BlankClip(pixel_type="YV12", color_yuv=$100000)

ScriptClip(Last, """
luma = AverageLuma ## gives the average luma of the current frame
Subtitle("luma=" + String(luma), align=2)
""")


return luma=16.0000 as expected for integer 10Hex = 16 decimal black


BlankClip(pixel_type="YV12", color_yuv=$100000)

ConvertBits(32)


ScriptClip(Last, """
luma = AverageLuma ## gives the average luma of the current frame
Subtitle("luma=" + String(luma), align=2)
""")

ConvertBits(8)


returns luma=0.062745 . It is magic number really not zero black expected in full range scaled. It is 255*0.062745=15.9999 (16). So it looks AVS internal 32bit float is mapping in 'float-narrow' of 0..255 integer data range to 0.0f..1.0f float range. It is really not logical engineering mapping of nominal video range to 0..1.0 data range as I think. The other plugins may not use this mapping so produce errors in processing ?

So the comment in the AVS docs about 'float is always full range' may be correct for 0..255 (and 0..65535) to 0.0f..1.0f mapping - but may be not what is expected by other plugins developers (using also external libraries and so on) ?

So AVS sequence
ConvertBits(32)
ConvertBits(8)
is transparent without range convert because it uses the single AVS-rule of range mapping to float32. Other plugins/filters may be use real zero black (and some other for nominal white) and things going wrong.

"#We go from Limited TV Range YUV to Full Range RGB 32bit float
ConvertToPlanarRGB()"

It looks not change range:

BlankClip(pixel_type="RGB24", color=$101010)
ConvertToYV12(matrix="PC.709") # to finally have correct zero black UVs

ConvertBits(32)

ConvertToPlanarRGB()

ScriptClip(Last, """
# avg = AverageLuma ## gives the average luma of the current frame
avg = AverageR ## gives the average luma of the current frame
Subtitle("avg=" + String(avg), align=2)
""")

ConvertBits(8)
ConvertToRGB24()


return also avg=0.062745 . So
ConvertBits(32)
ConvertToPlanarRGB()
still keeps range in 'AVS-float-narrow'. Not convert black to zero 0.0f.

"AviSynth+ uses full and limited ranges with float, using 16/256 and 235/256 for black and white in limited range, same with chroma data. Avs+ fmtconv is derived from its original Vapoursynth version, where nominal range for float is always [0 ; 1] and [-0.5 ; +0.5] for chroma information, and full/limited concept doesn’t apply (however range information can be carried as frame property)."

This confirms the idea of different range mapping to float in different environments/plugins. One more point of failure in AVS scripting.

"So, just before the bitdepth conversion, we have a YUV clip in “limited” float. fmtc_bitdepth cannot handle such a range: converting gives a double-limited range. However it is possible to restore the single-limited range at the function output by using the inverse factor, therefore converting to full range with fulld=true. This is a dirty hack, it requires to overwrite the frame property afterwards, but it should work."

It turns that going into float do not makes things easy and clear and border-less and virtually infinite precision as expected - it again adds lots of range-mapping nightmare.

[StvG]

"#We go to 32bit float limited tv range
ConvertBits(32)
"

It looks the 'float32' format in AVS may very different between authors ? So not compatible between different filters/plugins/AVS-internal.

This was discussed several times in the past. Here (https://forum.doom9.org/showthread.php?p=1926448#post1926448) (read from this post) an example (https://github.com/sekrit-twc/zimg/issues/134 https://github.com/AviSynth/AviSynthPlus/issues/189)

[DTL]

May be it is good to supplement ConvertBits() description at wiki http://avisynth.nl/index.php/ConvertBits with description of internal AVS-float format (range mapping) and some warning because it may be not directly compatible with plugins. With HDR times people like to go into float and mix plugins so faces this issues more frequently ?

As a 'natural and logical progress' in digital precision enhancement (8->10...16 bit) I expect also different range mapping of integer to float as 16..235 to 16.0f..235.0f for example. But things going differently in different environments/plugins.

Also may be AVS developers may add one more frame/clip property to auto-handle this issue like _FloatRangeMapping of different possible ways.

The
ConvertBits(32, fulls=false, fulld=true)
really makes range expansion so narrow range black of 16 integer finally maps to 0.0f of float (ConvertToPlanarRGB keeps it too). So it may be way to convert into some 'Vapoursynth' plugins float format.

[cretindesalpes]

fmtconv r30 (https://forum.doom9.org/showthread.php?t=183139):
matrix: The _ColorRange frame property is now set when a matrix preset is used.
transfer: Added ACEScct transfer function.
primaries: Added DCI P3+ and Cinema Gamut presets.
primaries: Added wconv parameter for full conversion.
Changed the configure options to compile with Clang.
Updated datatypes in the examples.

[Dogway]

Thanks!
I was having an issue tweaking fmtc_transfer() lws and lwd, they didn't seem to do anything, am I doing it wrong?
ConvertBits(32)
fmtc_resample (css="444")
fmtc_matrix (mats="2020", matd="rgb")
fmtc_transfer (transs="PQ",transd="linear",lws=4000.0,lwd=100.0,sceneref=false) # Also tested with '1886' transd

[cretindesalpes]

I should document better all the parameter effects in the documentation.

1. If you want to match the peak luminance of the transfer functions (and not the reference white), use match=2

However this won’t work here:

2. You can specify lws or lwd only when the component scale is not specified by the EOTF.

PQ uses an explicit value of 10000 cd/m² for component F’ = 1.0, and this cannot be overridden. Same with the 1886 transfer curve, which is in the generic SDR category (100 cd/m² for F’ = 1.0). lws and lwd are mainly meant for HLG operations, or transfer functions with unspecified luminance scale. Anyway you can use the cont parameter to force a scale change, and possibly use match=0 to get a full control on the transfer scales.

Check fmtc_transfer (/* parameters, */ debug=1) propShow () to know a bit more what’s happening under the hood.

Maybe should I relax some constraints to allow more flexibility?

[Dogway]

Check fmtc_transfer (/* parameters, */ debug=1) propShow () to know a bit more what’s happening under the hood.

Maybe should I relax some constraints to allow more flexibility?

Thanks! No I don't think so, maybe only adding some clarifications, like using a tonemapper or EETF for proper HDR scaling. I positively see fmtconv as strongly abiding to the standard so that's fine if the feature fits in 2446 or an EETF better.
I ported 2446 (method C only) here (https://forum.doom9.org/showthread.php?p=1973845#post1973845) and combining it with fmtconv gave me pleasant results, specially on highlights. It has some milky shadows and didn't implement achromatic highlights but I will be adding more TM and shapers.

[tormento]

I should document better all the parameter effects in the documentation.
Is possible to use fmtc_resample to invert a bicubic interpolation?

Getnative gives me (from a 1080p source):

Resize Kernel: Bicubic b 0.33 c 0.33
Native resolution: 720p

I just would like to see if I can get better results than DeBicubicResizeMT(target_width=1280,target_height=720,threads=1), as it gives me some aliasing.

What parameters would you suggest me?

[cretindesalpes]

tormento:

fmtc_resample (1280, 720, kernel="bicubic", a1=0.33, a2=0.33, invks=True)

is probably what you’re looking for. Please note that invks works differently of DeBicubic. It performs the inversion in the spectral domain instead of solving an equation system to find the kernel coefficients, so the result may be slightly softer. You can increase the invkstaps parameter (default: 4) but this can introduce more ringing.

[tormento]

is probably what you’re looking for
Thanks!

About a1(=b ?) and a2(=c ?) parameters, should they be the same of the getnative utility or is there a better way to find their optimal values?

As they form a ∞² space, it's not trivial to have a proper domain of search. Any help is welcome.

[cretindesalpes]

Yes a1 = b and a2 = c, and they are the same as the one found by getnative. I don’t know if there are better ways to find these values.

[tormento]

Yes a1 = b and a2 = c, and they are the same as the one found by getnative. I don’t know if there are better ways to find these values.
I have tried

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 3)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Ishuzoku reviewers\01.dgi")
fmtc_resample (1490, 838, kernel="bicubic", a1=0.3, a2=0.3, invks=True)
Prefetch(x)

on a YV12 video and it gives me YUV420P16 back.

Is it wanted?

Plus, there is almost no advantage in using a Prefetch >=2 (fps 60ish), while something else such as DeBicubicResizeMT accelerates almost linearly (fps 200ish).

Why?

[cretindesalpes]

on a YV12 video and it gives me YUV420P16 back.
Is it wanted?
Yes.

Plus, there is almost no advantage in using a Prefetch >=2 (fps 60ish), while something else such as DeBicubicResizeMT accelerates almost linearly (fps 200ish).
Why?
fmtc_resample is already multithreaded via AVSTP. Use avstp_set_threads(1) to disable internal multithreading.

[tormento]

is already multithreaded via AVSTP
I don't use avstp dll.

[hello_hello]

Just a question to clear up a little ambiguity in the FMTConv (and DitherTools) help file.

The fmtc.resample section shows the default for taps as 4.

arrayi taps (4),
arrayi tapsh (taps),
arrayi tapsv (taps),

Under the kernel section, it says Lanczos is the same as LanczosResize.
The default for LanczosResize is 3 taps though, so I'm just wondering which it is.

Cheers.

[tormento]

Yes.
Just saw that you moved the repo to GitLab (https://gitlab.com/EleonoreMizo/fmtconv/)and there are some changes. When do you plan to release a new build?

[tormento]

I'd like to replace the two lines:

z_ConvertFormat(pixel_type="RGBP16", colorspace_op="2020:st2084:2020:limited=>rgb:st2084:2020:full", resample_filter_uv="Spline64", dither_type="error_diffusion", use_props=0)

z_ConvertFormat(pixel_type="YUV420P10", colorspace_op="rgb:std-b67:2020:full=>2020:std-b67:2020:limited", resample_filter_uv="Spline64", dither_type="error_diffusion", use_props=0)

with fmtconv equivalents, as it's usually faster on my pc.

Tried to read the manual and I have lost my mind.

Can someone help me?

[StvG]

I guess there will be some additional filtering between the two conversion.

I'd like to replace the two lines:

z_ConvertFormat(pixel_type="RGBP16", colorspace_op="2020:st2084:2020:limited=>rgb:st2084:2020:full", resample_filter_uv="Spline64", dither_type="error_diffusion", use_props=0)

fmtc_bitdepth(32)
fmtc_resample(css="444", kernel="spline64")
fmtc_matrix (mat="2020", col_fam="RGB")
fmtc_bitdepth(16, dmode=6)

z_ConvertFormat(pixel_type="YUV420P10", colorspace_op="rgb:std-b67:2020:full=>2020:std-b67:2020:limited", resample_filter_uv="Spline64", dither_type="error_diffusion", use_props=0)

fmtc_bitdepth(32)
fmtc_matrix (mat="2020", col_fam="YUV")
fmtc_resample (css="420", kernel="spline64")
fmtc_bitdepth(10, dmode=6)

[tormento]

I guess there will be some additional filtering between the two conversion.
Indeed, there is:

DGCube("2a_PQ4000_HLG_mode-nar_in-nar_out-nar_nocomp.cube", in="full", lut="full", out="full")

Are those fmtconv lines still valid?

:thanks:

[StvG]

Are those fmtconv lines still valid?

Yes, they are valid.

[tormento]

Yes, they are valid.


Have I to convert32 even the 2nd time?

[StvG]

Have I to convert32 even the 2nd time?

Yes, if you want to mimic as close as possible the zimg calls.

[DTL]

Looks like found an issue with 'forcing' :

Documentation says the fh param is both for kernel additional scaling and also to force convolution if size not changed if fh<0. But too low negative fh values like -0.1 causes significant kernel distortion (though working) and too close to zero negatives like -0.000001 to keep kernel as unchanged as possible causes freeze (forever ?).

Test script:

LoadPlugin("fmtconv.dll")

Function AddBordersHF(clip c, int left, int right, int flt_rad)
{
unflt=AddBorders(c, left, 0, right, 0)
flt=GaussResize(unflt, unflt.width, unflt.height, p=10, b=2.71828, s=0, force=1)
uf_internal=Crop(c, flt_rad, 0, c.width-flt_rad*2, c.height)
return Overlay(flt, uf_internal, x=left+flt_rad, y=0)
}

Function AddBordersHF_FMTC(clip c, int left, int right, int flt_rad)
{
unflt=AddBorders(c, left, 0, right, 0)
# flt=fmtc_resample(unflt, w=unflt.width, h=unflt.height, kernel="gauss", taps=10, a1=10, fh=-0.000001).ConvertBits(8)
flt=fmtc_resample(unflt, w=unflt.width, h=unflt.height, kernel="gauss", taps=10, a1=10, fh=-0.1).ConvertBits(8)
uf_internal=Crop(c, flt_rad, 0, c.width-flt_rad*2, c.height)
return Overlay(flt, uf_internal, x=left+flt_rad, y=0)
}

ColorBarsHD(2000,2000)
UserDefined2Resize(width/10, height/10)

r1=2

left=20

std=AddBorders(left, 0, 0, 0, r=r1).SubTitle("AVS 3.7.4", align=5)

#return std

s1=AddBordersHF(last, left, 0, r1).SubTitle("AVS s1", align=5)

s2=AddBordersHF_FMTC(last, left, 0, r1).SubTitle("FMTC p=10", align=5)

StackVertical(std, s1, s2)
#StackVertical(std, s1)


The line
flt=fmtc_resample(unflt, w=unflt.width, h=unflt.height, kernel="gauss", taps=10, a1=10, fh=-0.1).ConvertBits(8)
is working, but too significant kernel distortion (too wide and soft)

The line
flt=fmtc_resample(unflt, w=unflt.width, h=unflt.height, kernel="gauss", taps=10, a1=10, fh=-0.000001).ConvertBits(8)
cause CPU load high enough but no result returned in many seconds on 200x200 frame size at E7500 CPU.

Tried with latest version r30 (and AVS+ core r4246).

Is it possible to force processing in other way ? May be setting of sx != 0 will work too as with AVS resampler ?

Addition: The sx !=0 is also working for forcing.
flt=fmtc_resample(unflt, w=unflt.width, h=unflt.height, kernel="gauss", taps=10, a1=10, sx=0.000001).ConvertBits(8)

[tormento]

I just ported my Vapoursynth plug-in to Avisynth+.
Please, two requests:
update the project link to the new one (https://gitlab.com/EleonoreMizo/fmtconv/)
I saw that the last update to the project was 7 months ago; would you please provide us a new build for avs+?

[tormento]

There is a probable "bug" about frame properties.

The problem is fmtc_bitdepth(bits=32) has frame properties _ColorRange=1 while in reality is _ColorRange=0.

Have a look here (https://github.com/Asd-g/libplacebo_Render/issues/4#issuecomment-4104255351).

[cretindesalpes]

DTL:
You should use fh = -1 and fv = -1. 1 means the kernel is used unscaled (default), and negative sign forces the processing. Very small values will create a gigantic kernel, probably not what you want.

tormento:
1. This is not really a bug. Maybe should I explicitly write the property as “full range” (0) in case of 32-bit float output whatever the fulld parameter value?
2. Thank you for the report, link update done.
3. See below.

[cretindesalpes]

fmtconv r31 (https://forum.doom9.org/showthread.php?t=183139):
bitdepth/Vapoursynth: Can work on clips of variable frame size. The function was already designed to handle this feature, but the parameter check was more strict than necessary. Thanks to Vardë for the report.
matrix: Added the SMPTE ST 2128 IPT-PQ-C2 transform to and from L’M’S’.
resample: Fixed 14 to 16 bit AVX2 conversion path, thanks to NSQY for the report.
resample: Fixed chroma placement: U and V vertical positions for interlaced PAL-DV were swapped and vertical subsampling > 2 was not handled at all. Thanks to Chortos-2 for the report.
transfer: Added L* transfer function.
transfer: Added Arri LogC4 transfer function.
transfer: Rescaled the S-Log3 curve to be used with limited-range code values, like the other S-Log curves.
primaries: Added Arri Wide Gamut 4 colorspace.
primaries: Fixed white XYZ coordinates for the NTSC-J colorspace (exact D93 location), thanks to Chortos-2 for the report.
Program path without x86 SIMD: fixed wrong conversions affecting a lot of functions (noticed on ARM/Apple). Thanks to SaltyChiang for the fix.
On Windows/x64, the Vapoursynth plug-in can be installed with the command line pip install vapoursynth-fmtconv. Requires Vapoursynth r74 or above.

[tormento]

1. This is not really a bug. Maybe should I explicitly write the property as “full range” (0) in case of 32-bit float output whatever the fulld parameter value?
I will leave the last word to the more experienced users here.

My issue is that I have to invoke libplacebo_render with src_levels="full" or it outputs the wrong range. Once I knew, I could easily fix it.

Thanks for your new build.

[tormento]

This is not really a bug. Maybe should I explicitly write the property as “full range” (0) in case of 32-bit float output whatever the fulld parameter value?
Why:

DGSource("M:\In\La collina dei papavery ~Lucky Red\collina.dgi", ct=24, cb=24, cl=0, cr=0)
propSet("_Matrix", 1)
propSet("_Transfer", 1)
propSet("_Primaries", 1)
propSet("_ColorRange", 1)
fmtc_bitdepth(bits=32, fulls=false, fulld=false)

expands the video to full range and doesn't leave it as it is?

A

fmtc_bitdepth(bits=16, fulls=false, fulld=false)

correctly leaves limited as limited.

Original image:

https://i.ibb.co/PZDRZgbW/image.png (https://ibb.co/0jskjrW9)

Script image:

https://i.ibb.co/nqnxRH6G/image.png (https://ibb.co/p6RYntb7)

[Z2697]

Because it's floating point. Values should always be in the range of 0.0-1.0, or -0.5-+0.5
Who knows. :rolleyes:

You see, by definition the full or limited range is the use of 0-255 or 16-235 (scale for other bitdepths and channels in your mind) to represent 0.0-1.0/-0.5-+0.5

[StvG]

0. YUV limited => resulted to YUV float [0, 1]
1. YUV limited => resulted to YUV float [16/255, 235/255]

_ColorRange does show which of the cases we have.

@tormento:
fmtc_bitdepth(bits=32, fulls=false, fulld=false)

expands the video to full range and doesn't leave it as it is?


You probably want fmtc_bitdepth(bits=32, fulls=true)

[Z2697]

The purpose of the limited range is to preserve the over/undershoot.
The IEEE 754 floating point number inherently "just have that attribute".
It is reasonable to have a legal range of 0.0–1.0 (-0.5 to +0.5) for everything.

However the color range frame property may be used to indicate the source's range.
So you would actually expect some over/undershoot value beyond 0.0-1.0 if the source is limited range.
Full range source will have those values clipped out.

Not that it really matters... if you work in float32 format, there will be over/undershoots introduced...

[StvG]

The purpose of the limited range is to preserve the over/undershoot.
The IEEE 754 floating point number inherently "just have that attribute".
It is reasonable to have a legal range of 0.0–1.0 (-0.5 to +0.5) for everything.

However the color range frame property may be used to indicate the source's range.
So you would actually expect some over/undershoot value beyond 0.0-1.0 if the source is limited range.
Full range source will have those values clipped out.

Not that it really matters... if you work in float32 format, there will be over/undershoots introduced...

I think we are not in the same page.

The practical matter is that in the Avisynth YUV 32-bit (float) world you can have [16/255, 235/255] when [0, 1] is expected - range mismatch (double compression, double expansion and what not). Examples:
- https://github.com/AviSynth/AviSynthPlus/issues/189
- https://forum.doom9.org/showthread.php?p=1926770#post1926770
- ColorBars(pixel_type="yuv420ps") / BlankClip(pixel_type="yuv420ps")

So in practice scripts and plugins read _ColorRange even for float and can treat it as "limited".
You have example of such script few posts above (Tormento, PixelScope) - that reads _ColorRange and based on the value it decide how to do YUV=>RGB - YUV [16/255, 235/255] => RGB [0, 1] or YUV [0, 1] => RGB [0,1]. fmtconv has YUV [0, 1] values but in some cases it has _ColorRange 0 (if fulld explicitly set) and in other cases for the same values it has _ColorRange 1.

[Z2697]

What I found more interesting is that libplacebo is doing 16/256-255/256 :)
It seems to convert both range to limited first. (so 255 in full range become 235/256, in limited range become 255/256...)

And swscale is doing something more quirky.
From limited range 235 is converted to 0.917983 which may suggest 235/256 with some error, but 255 is 0.999985... (update: I used 255 for all 3 channels by accident)
I had to use grayf32le as dest because there's no float YUV format in FFmpeg.
When input is YUV, it clamps the value to 16-235 then divides it by 256 (with low accuracy). When input is GRAY, it divides the value by 255 without clamping.
From full range there's no clamping or range conversion, just divide by 256 or 255 depending on the input...

Both libplacebo and swscale don't have their output value range depend on the output range setting.

Looks like everyone except zimg and fmtc have their own mind.
I think zimg and fmtc are clearly better.

[StvG]

/ off-topic
libplacebo with Vulkan processing? If so, it is in own area due to how GPUs handle image data.

I guess you used accurate_rnd for swscale?
/ end off-topic

[Z2697]

/ off-topic
libplacebo with Vulkan processing? If so, it is in own area due to how GPUs handle image data.

I guess you used accurate_rnd for swscale?
/ end off-topic

It's still floating point (fp16, probably) so shouldn't be very different.
But maybe because divide by 256 is faster than 255, just decrese the exponent.

I've tried both with and without accurate_rnd, it doesn't change the result (at least for a solid color).

[tormento]

If the default for fp is to normalize values, i.e. expand the range from limited to full, why AVS+ internal functions keep them in limited range?

LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("M:\In\[CR] Chainsaw Man - Il Film La storia di Reze.dgi",ct=140,cb=140,cl=0,cr=0)
propSet("_Matrix", 1)
propSet("_Transfer", 1)
propSet("_Primaries", 1)
propSet("_ColorRange", 1)
ConvertToYUV444()
ConvertBits(32)
VideoTek(Mode="SDR", Type="nits", Detailed=true)
Prefetch(4)

https://i.ibb.co/RT3VtyyX/image.png (https://ibb.co/60HfGBBK)

You probably want fmtc_bitdepth(bits=32, fulls=true)

That one expands range too:

fmtc_bitdepth(bits=32)
fmtc_resample(css="444", kernel="spline36", fulls=true)

https://i.ibb.co/wr4hq0R0/image.png (https://ibb.co/cXQh9g1g)

There is no way I can get limited range YUV444PS with fmtc_conv. I tried any combination of fulls & fulld.

libplacebo_Render(src_csp="709", out_fmt="YUV444PS", src_levels="limited", dst_levels="limited")

works as intended:

https://i.ibb.co/QvnqWRVN/image.png (https://ibb.co/s9QDTG4w)

With z_ConvertFormat I had to fake the origin color range:

z_ConvertFormat(resample_filter="Spline64", pixel_type="YUV444PS", colorspace_op="709:709:709:f=>709:709:709:f")

https://i.ibb.co/gZRqFtS9/image.png (https://ibb.co/23kVYcvh)

In general, as I am converting to YUV444PS to feed BM3D_CUDA, is it better to apply noise reduction to the original, uninterpolated, limited range or to the expanded, interpolate, one?

What are those "black lines" that I can see in the expanded luma range?

[Z2697]

That's wrong.

fmtc_bitdepth(bits=32, fulls=true)

fmtc_bitdepth(bits=32)
fmtc_resample(css="444", kernel="spline36", fulls=true)

Can't you see the difference between the two?
The idea is the same as you faking input flag for zimg.

What you see as "black lines" is the result from not converting it back to it's original range.
The range in floating point doesn't matter. Binary32 have enough precision to hold either set of the values.
There's no interpolation being done.
uint8/235*255 creates some gap, not surprising.
x/235*235 or x/255*255, gives you exactly same thing (almost).

BM3D is designed for "full range".
The paper discuss firstly RGB and then "opponent" colorspace which is pretty much like some quirky YCgCo.
Both are most likely in the "full range".

(I put quotes on the "full range" because fp32 have much much wider actual range.)

[real.finder]

There is no way I can get limited range YUV444PS with fmtc_conv. I tried any combination of fulls & fulld.


I think this should be a bug, maybe report it here https://gitlab.com/EleonoreMizo/fmtconv/-/boards

[tormento]

I think this should be a bug, maybe report it here https://gitlab.com/EleonoreMizo/fmtconv/-/boards
Could you try too and let me know your results?

I'd prefer not to submit a false issue.

[Z2697]

You need to confirm the range not by converting it back to integer format (https://github.com/FranceBB/VideoTek/blob/6012d45f255e0ea12385d0c3f6e9ae54dd0b7e04/VideoTek.avsi#L97).

[tormento]

You need to confirm the range not by converting it back to integer format.
Such as?

What's wrong with my script?

[Z2697]

What you want really is just fmtc_bitdepth(bits=32, fulls=true, fulld=false)

[tormento]

What you want really is just fmtc_bitdepth(bits=32, fulls=true, fulld=false)


Does it work for you?

I get range expansion anyway.

[StvG]

Does it work for you?

I get range expansion anyway.

Yes, it does work. The following results to YUV "limited".

source 8-16 bit
propSet("_Matrix", 1)
propSet("_Transfer", 1)
propSet("_Primaries", 1)
propSet("_ColorRange", 1)
fmtc_bitdepth(bits=32, fulls=true, fulld=false)

Share your exact script.

[tormento]

Yes, it does work. The following results to YUV "limited".

Now I see where the problem is.

It works when you use fulls=true, fulld=false in fmtc_bitdepth but not in fmtc_resample. WTF.

Thanks.

P.S: Why doesn't the internal AVS+ conversion plugins "expand" to full range?

[tormento]

Can't you see the difference between the two? The idea is the same as you faking input flag for zimg. What you see as "black lines" is the result from not converting it back to it's original range.
This point is a bit obscure to me.
BM3D is designed for "full range".
So, my "correct" question becomes: should I feed video to BM3D by using

fmtc_bitdepth(bits=32, fulls=true, fulld=false)
fmtc_resample(css="444", kernel="spline36")

or

fmtc_bitdepth(bits=32)
fmtc_resample(css="444", kernel="spline36")

and think about getting back to limited after the noise reduction?