fmtconv/Avisynth+: resize, bitdepth and colorspace conversions

[cretindesalpes]

Fmtconv always uses ”full range” floating point values whatever the specified full* parameter value, because the narrow range is meaningless for float. Avisynth+ converters don’t work like this and use a narrow range for float by default. You can add fulls=true, fulld=false to ConvertBits(8) and get the expected result.

[StvG]

Thanks for adding top_left chroma location.

fmtc_matrix - with csp data type cannot be changed (int->float). It seems the same is valid for bits parameter too (but not mentioned in the readme)?
csp doesn't accept rgbpx values: ColorBars(pixel_type="yv24")
fmtc_matrix(mat="rgb", mats="2020", csp="rgbp10")

[DTL]

Fmtconv always uses ”full range” floating point values whatever the specified full* parameter value"

If software currently can not provide any warning about non-compatible user input with other enviroment (and internal software features)- may be better to fail processing with stop-error (and some useful hint messages why this stop occur) instead of silently ignoring user control param and creating unexpected output.

Also float32 processing can process shifted and scaled "limited" levels encoding without significant losses but may reqiure more operations and will be just slower.

[cretindesalpes]

StvG:

Yes at the moment the data type cannot be changed. This may change later, but for now I’ll update the doc.

DTL:

In my previous message I enclosed full range between double quotes, because range hasn’t any meaning for float. It applies to integer and indicates the values coding 0.0 and 1.0 or -0.5 and +0.5. But the float values are the real values we want to process, the real number scale for the algorithms. There is no need for this kind of full/TV coding, it doesn’t apply. And reflecting the narrow range coding in float with an offset and scale doesn’t make any sense, I would even say it’s an insanity. With floating point data, we have both the headroom and the dynamic range, and even more than with integers. Why make things more complicated than they should be? And issuing an error when using the fulls or fulld with float would most likely break wrapping functions that can process different data formats.

[DTL]

"And reflecting the narrow range coding in float with an offset and scale doesn’t make any sense, I would even say it’s an insanity."

It is good if programmer live in float world only. But in current reality we still use generally integer-coded low bits number values but in the new awful HDR-world need to deep into float to not lost precision in linear HDR (I not sure if using integer 'limited' 32bit per sample available/possible - not tested, though 32bit integer in theory enough to cover 1..10 milion linear range of typical todays HDR PQ). So most of workflows in nowdays HDR reality (in september 2021 EBU Recs r153 and r154 make deprecated SDR transfer at all) are 'integer(transfer_domain compressed)->float(linear,expanded)->integer(transfer_domain compressed)" and if float domain silently shift and scale the range it creates unexpected errors. User assumes dataset already in 'limited' encoding and if create again conversion to 'limited' looks like error situation - twice conversion to 'limited' that in typical life will cause levels error.
It is good to add to documentation about 'fmtc always will shift and scale 'limited' integer encoded input data to 0..1 range so if user need 'transparent' workflow - the output data again need to be converted to 'limited' mode (and before touching integer domain to prevent rounding and limiting errors).

"Why make things more complicated than they should be?"

It is just 'common practice' that most of moving pictures processing world uses not very programmer-friendly 'limited' mode of black and white data encoding (in the integers). Also to use signed integers with black at zero is mean to lost almost half of available integer range. It is again sad shadows of poor past where every codelevel of 8 and 10 bit integers was long debated. So the best result for 8 and 10 bit poor world was 'shifted unsigned integer' encoding.

[StvG]

StvG:

Yes at the moment the data type cannot be changed. This may change later, but for now I’ll update the doc.

Just to be sure you didn't overlooked it:

csp doesn't accept rgbpx values: ColorBars(pixel_type="yv24")
fmtc_matrix(mat="rgb", mats="2020", csp="rgbp10")

This is ok:BlankClip(pixel_type="rgbp")
fmtc_matrix(matd="709", mats="rgb", csp="yuv444p10")

This doesn't work:BlankClip(pixel_type="yv24")
fmtc_matrix(matd="rgb", mats="709", csp="rgbp10")

[cretindesalpes]

Ah sorry I see now, RGBP* values were not interpreted as planar. The bug fix will be available in the next release.

[cretindesalpes]

fmtconv r26 (https://forum.doom9.org/showthread.php?t=183139):
matrix: fixed the output colorspace when using singleout.
matrix: fixed the output colorspace when using a custom matrix. Thanks to vxzms for the report.
matrix/Vapoursynth: _Matrix and _ColorSpace frame properties are actually deleted when a custom matrix is used or the final colorspace is unknown. Thanks to mysteryx93 for the report.
resample: fixed empty custom impulse that could cause a crash.
resample: added top-left chroma location.
resample/Avisynth+: interlacing detection now uses the global stream information when the frame properties are not available.
resample/Avisynth+: fixed the planes parameter.
transfer: Linear light can now be display- or scene-referred. Added sceneref, lw, lws, lwd, lb, ambiant and deprecated blacklvl.
transfer: By default, automatically matches the reference white levels for source and destination transfer curves. This may cause some backward incompatibilities. match parameter added.
transfer: Removed the planes parameter introduced in r24 because planes are no longer independent of each other.
transfer: debug provides information about the transfer operation and levels as a frame property.
transfer: BT.470M characteristic is now a pure power curve, instead of a copy of the sRGB curve.
transfer/Avisynth+: fixed flt + bits combination that was ignored. Thanks to DTL for the report.
transfer/Avisynth+: fixed case sensitivity to transs and transd.
transfer/Vapoursynth: fixed logceis and logceid that were missing from the function registration since their introduction in r23… Thanks to groucho86 for the report.
Fixed compatibility with Avisynth+ 3.7.1.
Avisynth+: fixed "RGBPxx" colorspace values that weren’t correctly interpreted as planar.

[StainlessS]

Whats new pussycat :)

Alert from Firefox about insecure download [due to non https], nothing to worrry about.

Thanks.

[hello_hello]

Would someone be so kind as to clarify my interpretation of the fmtconv help file in relation to the "center" argument?

The help file says the default for center is true.

center
Like the Avisynth standard resizers, this resizer preserves the position of the picture center.

my interpretation:
"This resizer assumes mpeg1 chroma placement by default, as Avisynth's resizers do."

So far so good, I assume to change that behaviour, "center" should be false.

cplace, cplaces, cplaced
Placement of the chroma samples.
The chroma placement is ignored when center is set to False or kernel to "point".

But then if centre=false the chroma placement is ignored?
Should the last line say "when center is set to True...."?

[cretindesalpes]

The help file says the default for center is true.
center
Like the Avisynth standard resizers, this resizer preserves the position of the picture center.
my interpretation:
"This resizer assumes mpeg1 chroma placement by default, as Avisynth's resizers do."
No, this is not really related to chroma placement, the statement works for any plane of any colorspace. It just means that the sampling grid coincides with the center of each (rectangular) displayed pixel, and that resize operations keep the center of the picture at the same relative location. Borders are not cropped nor extended. It’s sounds trivial because this is what users expect in general, but mathematically speaking it isn’t. There are other possible implementations. The most naive one, preserving the sampling position of the top-left corner for example, results in a slight picture shift along the top-left–bottom-right axis. This is what center = False does. PointResize is an exception in Avisynth built-in resizers and does it too; notice the shift between both pictures:
Version ().ConvertToPlanarRGB ().BicubicResize (1920, 720)
Interleave (PointResize (640, 240), GaussResize (640, 240, p=100))
Remarks:
1. GaussResize (p=100) is more or less equivalent to PointResize with a correct center preservation, so it’s easier to compare both.
2. I added ConvertToPlanarRGB because interleaved RGB frames are stored upside down like BMP pictures, and the shift would be vertically inverted.

BTW in fmtconv, default chroma placement is MPEG-2.

cplace, cplaces, cplaced
Placement of the chroma samples.
The chroma placement is ignored when center is set to False or kernel to "point".
But then if centre=false the chroma placement is ignored?
Should the last line say "when center is set to True...."?
The documentation stands correct. center = True is the default behaviour. When center = False, all planes are processed “naively”, without compensation to preserve the center of the picture and the plane’s relative positions to each other.

[hello_hello]

cretindesalpes,
thank you very much for the clarification.

[cretindesalpes]

fmtconv r27 (https://forum.doom9.org/showthread.php?t=183139):
matrix: fixed coefficient errors when using singleout.
primaries: added Sony P22 primaries.
transfer: better selection between linear and logarithmic input for LUTs, LUT size reduced.
transfer: fixed crash with grey colorspace. Thanks to vxzms for the report.

[gispos]

Why is the download blocked by Firefox? Security risk.

[cretindesalpes]

Probably because the download link is from a http site, not https. You can tell Firefox to bypass this security check. The releases are also available on Github (https://github.com/EleonoreMizo/fmtconv/releases) using https if http gives you shivers.

[gispos]

Probably because this is a http site, not https. You can tell Firefox to bypass this security check. The releases are also available on Github (https://github.com/EleonoreMizo/fmtconv/releases) using https if http gives you shivers.
OK, thanks

[tormento]

Probably because this is a http site
Actually, I open doom9 by https. Did you mean the file link?

[cretindesalpes]

fmtconv r28 (https://forum.doom9.org/showthread.php?t=183139):
bitdepth: Faster void and cluster pattern generation. Larger patterns are allowed.
matrix: Added an alias for "fcc"
primaries: New presets: Free Scale-gamut, DaVinci Wide Gamut, DRAGONcolor, DRAGONcolor2, REDcolor, REDcolor2, REDcolor3, REDcolor4 and REDWideGamutRGB.
primaries: Fixed Sony P22 primaries.
primaries: Removed the deprecated "dcip3" alias for "p3d65".
transfer: Added DaVinci Intermediate, RED Log3G10, REDlog, Cineon and Panalog.
Vapoursynth and Avisynth+ plug-ins are now contained in the same binary file.

[StainlessS]

Why is the download blocked by Firefox? Security risk.
Actually, I open doom9 by https. Did you mean the file link?
Firefox still blocks download, but you can open "Downloads", right click the RED marked download and "Allow Download",
its just as cretindesalpes said
You can tell Firefox to bypass this security check.

Thanks for the update C.

[Dogway]

Thank you!

[Dogway]

@cretindesalpes: Can you have a look at the graph (https://www.desmos.com/calculator/fddyni7beb) and check my weights are correct? I compared for example BlackmanMinLobe (Nuttall in the graph) and my version is softer:

fmtc_resample(round(w*1.5),round(h*1.5),kernel="impulse",kovrspl=2,cnorm=true,center=true,impulse=[-0.00024,0,0.01005,0,-0.09096,0,0.5812,1.0,0.5812,0,-0.09096,0,0.01005,0,-0.00024])

This is 4 taps or supposed to be. Also is it possible to use Jinc (as in EWA) weights?

[cretindesalpes]

I haven’t checked the coefficient details, but the kernel oversampling looks way too low. Currently custom kernels are linearly interpolated (this could change in the future with spline or sinc interpolation) so with 2× you’ll get some triangular shapes between zero crossings—far from the function you could expect. You need something like 16× or more to get decent results.

Jinc is a different beast because the kernel is not separable in horizontal and vertical passes. It requires a single 2D pass; the current code is not designed to handle this. It may be a future addition (https://github.com/EleonoreMizo/fmtconv/issues/4) too.

[Dogway]

I see, I thought it was spline interpolated. With kovrspl=16 I get grey output or scanline type artifacts (depends if normalization is true).
The way I compute coefficients in the graph is by taking the y-intercept every 0.5 units in x. This looked similar to DTL's impulse values (https://forum.doom9.org/showthread.php?p=1952985#post1952985)

imp=[-0.00024*16,0,0.01005*16,0,-0.09096*16,0,0.5812*16,1.0*16,0.5812*16,0,-0.09096*16,0,0.01005*16,0,-0.00024*16]
imp=[-0.00024,0,0.01005,0,-0.09096,0,0.5812,1.0,0.5812,0,-0.09096,0,0.01005,0,-0.00024]
fmtc_resample(1280*2,634*2, kovrspl=16, cnorm=true, center=true, kernel="impulse",impulse=imp)

[cretindesalpes]

Ah no actually the 16× factor applies to the kernel sampling interval. So you have to fill the array with the kernel values for each 1/16 unit in [-4 ; 4], instead of each 1/2 unit with kovrspl=2. Don’t multiply the values by 16.

[Dogway]

Wow thanks, that's gonna be a long array, thank you!

EDIT: Maybe I can do something with Spline (http://avisynth.nl/index.php/Internal_functions#Spline).

[Dogway]

Finally went with 64 (32 each side). But I had to set kovrspl to 8 and was about to ask why. Well I guess I found out, oversampling by 8 in unit interval, so "32 / kovrspl = taps".

Here's my call, 'nuttall' matches 'blackmanminlobe'. I guess kovrspl doesn't affect performance. Thanks again!

nuttall = [1,0.968978694959837,0.8800877014862263,0.7450789501548785,0.5811536099884803,0.4079380255054445,0.24421357419883338,0.10506827778662947,0,-0.0677506795383086,-0.1006542288117901,-0.10540908368482169,-0.09096328276239886,-0.06652252976938802,-0.0399481722582298,-0.016788722421354926,0,0.0097560148734007, \
0.013447841939631484,0.012890350335053615,0.010046723030434222,0.0065452031473278455,0.0034500048853390547,0.0012517909430134226,0,-0.0005100722653928198,-0.000565616944011163,-0.00041987662546901855,-0.00023981418211867437,-0.00010446859341761584,-0.00003063039438713025,-0.000003718533785179404]

a=fmtc_resample(1280*2,634*2, kovrspl=8, cnorm=true, center=true, kernel="impulse",impulse=ArraySym(nuttall))
b=fmtc_resample(1280*2,634*2, cnorm=true, center=true, kernel="blackmanminlobe")
ex_makediff(a,b, metric="none", aug=true, dif=true, show=2)

[DTL]

Jinc can be calculated in 1D as usual function. So it can be provided as impulse kernel to exist resampler of 1D. It is simply replaced sin(x)/x to bessel_J1(x)/x. Also typically as Lanczos is sinc weighted by sinc so real working low taps jinc is jinc weighted by jinc (or any other weighting). Though as jinc(x) fades significantly faster in compare with sinc(x) with increasing of x it require less weighting.
https://github.com/Asd-g/AviSynth-JincResize/blob/b7fbf5d680a2950dff65b907134e6719efd11916/src/JincResize.cpp#L915

Also _j1(x) is standard lib C function and not need complex calculating each time in each handcrafted software. https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/bessel-functions-j0-j1-jn-y0-y1-yn?view=msvc-170

The JincResize is 2D processing 1-pass. But any other kernel may be calculated as 2D kernel f_2D(x,y) = f_1D(raduis_vector_length(x,y)) and used in 2D 1-pass processing. The difference is only in the processing engine - 2D single pass or 1D+1D 2 passes.

Main difference between sinc and jinc - the bessel_J1 is not = 0 at the regular offset like sinc(x) (neither 1D offsets nor 2D offsets). So when doing jinc-interpolation (upsizing) - the input samples do not keeps its values in the output. It is somehow viloates the idea of upsize interpolation as calculating only intermediate samples and keep input unchanged. Same for sub-sample shifts. So it looks all nice math of sinc in 1D do not work in 2D and with jinc too.

"I thought it was spline interpolated."

May be it is better to provide more 'real' kernel samples instead of rely on interpolation inside. Current input is not limited with number of provided kernel samples ?

"You need something like 16× or more to get decent results."

May be it is possible to somehow allow user to provide analytical math equation for kernel (with weighting) ? So it will save user from providing lots of kernel samples for non-integer resampling ratios or some filtering.
Like via some math interpreter. It may slow down processing very because of lots of kernel samples required for each output sample. But for some science projects it may make development faster. Or provide some software tool (plugin to some math CAD ?) to convert analytical kernel formula into sampled vector of any required precision (sampling interval) and generate text string to copy-paste into AVS script.

[Dogway]

Playing with some fmtc_resample() I noticed that "fmtc_resample(css="41")" doesn't work, since fmtc_resample() always output 16-bit and YUV411 HBD is not supported in AVS+.
Also checked "fmtc_resample(cplaced="tl")" which doesn't write frame properties.
And:
fmtc_resample(1920,1034,invks=true,kernel="bicubic",fh=2.6,fv=2.6,a1=-0.6,a2=0.4)
produces a crash

[cretindesalpes]

fmtconv r29 (https://forum.doom9.org/showthread.php?t=183139):
Vapoursynth: switched to API v4. Fmtconv now requires Vapoursynth r55 or later, API v3 is not supported anymore.
resample: Fixed degnerated cases with invks causing a crash. Thanks to Dogway for the report.
resample/Avisynth+: fixed frame property writing when only cplaced is defined, thanks to Dogway for the report.
transfer: Added sigmoid curve as a transfer function.
transfer: Fixed the float to integer path.

[hello_hello]

cretindesalpes,
Using something similar to one of the examples in your help file (Avisynth+ 3.7.2 and fmtconv r29), propShow() says for the output video _ColorRange=0=full, when for a YV12 source I would've expected it to be either limited or not set at all.

fmtc_bitdepth(bits=16)
fmtc_resample(css="444")
fmtc_matrix(mat="709", bits=16)
fmtc_transfer(transs="709", transd="linear")
fmtc_primaries(prims="709", primd="601-625")
fmtc_transfer(transs="linear", transd="601")
fmtc_matrix(mat="601", bits=16)
fmtc_resample(css="420")
fmtc_bitdepth(bits=8)
propShow()

Best as I can tell the transfer function is setting _ColorRange=0 (why is that when the input is RGB?) but the second matrix function isn't updating _ColorRange when converting back to YUV, and fmtc_matrix(mat="601", bits=16, fulld=false) is required to correct it.

The output looks the same with or without fulld=false, so I assume it's just the _ColorRange property that's incorrect and the video is still being converted to limited range.

Is that expected behaviour?

Cheers.

[cretindesalpes]

hello_hello:

Yes it is the expected behaviour. From the matrix doc:
The _ColorRange frame property is set if the fulld parameter has been explicitely defined.
And if fulld is not defined, the property is just copied from the source frame. But this doesn’t make sense when using a matrix preset because the target range is always known. So I might change this behaviour in the next release.

[tormento]

Can I use fmtconv to replace the lines:

z_ConvertFormat(pixel_type="RGBP16", colorspace_op="2020:st2084:2020:limited=>rgb:st2084:2020:full", resample_filter_uv="spline64", dither_type="error_diffusion")

and

z_ConvertFormat(pixel_type="YUV420P10", colorspace_op="rgb:std-b67:2020:full=>2020:std-b67:2020:limited", resample_filter_uv="spline64", dither_type="error_diffusion")

?

I'd like to try if there is any speed improvement over z.

[Dogway]

I think there might some error in the docs, for bobbing an interlaced source it shows interlaced(d) args as boolean, but in the function description as int type.

Also I'm trying to understand how do you do a chromatic adaptation with fmtconv. I tried the next but the result looks noop.

fmtc_resample(css="444")
fmtc_matrix(mat="709", bits=16)
fmtc_transfer(transs="709", transd="linear")
fmtc_primaries(prims="709", primd="709", ws=[0.3127,0.3290], wd=[0.33242, 0.34743]) # From D65 to D55
fmtc_transfer(transs="linear", transd="709")
fmtc_matrix(mat="709", bits=16)

[tormento]

I just ported fmtconv Vapoursynth plug-in to Avisynth+.
We need you. :)

[cretindesalpes]

tormento:

Yes you can. I’m pretty sure z is faster, but you can try.

Dogway:

Thanks, indeed the examples haven’t been checked for long, and function prototypes slightly evolved since. I just fixed the doc.

Your chromatic adaptation is right. How do you check it? Results may be subtle. Or maybe what you’re looking for is complete conversion, for example to make a picture displayed on a D55 monitor look exactly like the same picture displayed on a D65 monitor placed side by side?

[Dogway]

Yes I was expecting a warmer output, maybe I should try with D50. Eyes adapt to the white point specially at matching environment temperatures as there's not a single "white" in psychovisually terms. I was double checking as I had difficulties on color space conversions on non-standard illuminants.

[tormento]

tormento:Yes you can. I’m pretty sure z is faster, but you can try.
Can you please translate them for me? The AVS porting doc is a bit obscure and there are missing parts too.

[Dogway]

Can I use fmtconv to replace the lines:

z_ConvertFormat(pixel_type="RGBP16", colorspace_op="2020:st2084:2020:limited=>rgb:st2084:2020:full", resample_filter_uv="spline64", dither_type="error_diffusion")


Maybe something like this?

fmtc_bitdepth (bits=16)
fmtc_resample (css="444",kernel="spline64")
fmtc_matrix (mats="2020", matd="rgb", fulls=false, fulld=true)
fmtc_bitdepth (bits=16, dmode=6)

EDIT: Correction, fmtconv does dither from float to 16-bit int, but the effect is minimal.

[StvG]

I would say this is the equivalent of that z_ line:
fmtc_bitdepth(bits=32)
fmtc_resample(css="444",kernel="spline64")
fmtc_matrix(mats="2020", matd="rgb")
fmtc_bitdepth(bits=16, dmode=6)

[tormento]

I would say this is the equivalent of that z_ line:
fmtc_bitdepth(bits=32)
fmtc_resample(css="444",kernel="spline64")
fmtc_matrix(mats="2020", matd="rgb")
fmtc_bitdepth(bits=16, dmode=6)
I guess 4 calls are slower than a single z one.

[DTL]

It may also cause massive memory overload with lots of cached frames with massive multithreading at todays AVS+ at todays massive multicore end-user CPUs. So single filter processing may be much more RAM friendly.

[tormento]

I was getting crazy, trying to find why BM3D_CUDA had some slight tint change and I found out that it is the combination of Convertbits and fmtconv that makes strange things.

Look at this comparison (https://slow.pics/c/LkQ6sKsA).

Reality is that BM3D_CUDA needs RGB to work and... am I doing something silly or there is something terribly wrong?

Image 0:

original

Image 1: OK

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 3)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
ConvertBits(32)
ConvertBits(8)
Prefetch(6)

Image 2: WRONG!!!

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 3)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
ConvertBits(32)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)[

Image 3: WRONG!!!

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 3)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
fmtc_bitdepth (bits=32,dmode=8)
ConvertBits(8)
Prefetch(6)

Image 4: OK

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 3)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
fmtc_bitdepth (bits=32,dmode=8)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

[StvG]

I was getting crazy, trying to find why BM3D_CUDA had some slight tint change and I found out that it is the combination of Convertbits and fmtconv that makes strange things.

Look at this comparison (https://slow.pics/c/LkQ6sKsA).

Reality is that BM3D_CUDA needs RGB to work and... am I doing something silly or there is something terribly wrong?

For Image 2 change ConvertBits(32) to ConvertBits(32, fulls=false, fulld=true).
For Image 3 change ConvertBits(8) to ConvertBits(8, fulls=true, fulld=false)

[tormento]

For Image 2 change ConvertBits(32) to ConvertBits(32, fulls=false, fulld=true).
For Image 3 change ConvertBits(8) to ConvertBits(8, fulls=true, fulld=false)
Please explain me why and why it's not "transparent" to fmtconv but they work only with the same function properly.

[tormento]

Ok, here things get tricky.

I need to convert to 32, then planar RGB and back to 8.

I am getting some tint changes (https://slow.pics/c/403yxn91) again.

00

original

https://i2.lensdump.com/i/1ELwFZ.md.png (https://lensdump.com/i/1ELwFZ)

01

LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
ConvertBits(32)
ConvertToPlanarRGB()
ConvertToYUV420()
ConvertBits(8)

https://i3.lensdump.com/i/1ELy0P.md.png (https://lensdump.com/i/1ELy0P)

02

SetMemoryMax()
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
ConvertBits(32, fulls=false, fulld=true)
ConvertToPlanarRGB()
ConvertToYUV420()
fmtc_bitdepth (bits=8,dmode=8)

https://i.lensdump.com/i/1EL5Bm.md.png (https://lensdump.com/i/1EL5Bm)

Even a

ConvertToPlanarRGB()
ConvertToYUV420()
fmtc_bitdepth (bits=8,dmode=8)

Gives slight wrong tint again.

https://i1.lensdump.com/i/1ELGoi.md.png (https://lensdump.com/i/1ELGoi)

It happens even with:

ConvertToPlanarRGB()
ConvertToYUV420()

I am getting crazy with conversions.

EDIT: it seems that ConvertToYUV420(matrix="Rec709") fixed things. WTF.

[tormento]

I talked too early (https://slow.pics/c/9GiDGmGp)...

0

Original

1

SetMemoryMax()
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
DeBicubicResizeMT(target_width=1280,target_height=720,threads=1)
ConvertBits(32)
ConvertToPlanarRGB()
w2xncnnvk(noise=0,model=0,gpu_thread=2)
ConvertToYUV420(matrix="Rec709")
DeBicubicResizeMT(target_width=1920,target_height=1080,threads=1)
ConvertBits(8)

2

SetMemoryMax()
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\Knights of Sidonia\Knights 01A.dgi")
DeBicubicResizeMT(target_width=1280,target_height=720,threads=1)
ConvertBits(32, fulls=false, fulld=true)
ConvertToPlanarRGB()
w2xncnnvk(noise=0,model=0,gpu_thread=2)
ConvertToYUV420(matrix="Rec709")
DeBicubicResizeMT(target_width=1920,target_height=1080,threads=1)
fmtc_bitdepth (bits=8,dmode=8)

I smashed my head against the wall enough, need your help.

[DTL]

For Image 2 change ConvertBits(32) to ConvertBits(32, fulls=false, fulld=true).


Do ConvertBits(32) really process fulld argument anyway ? As I start to learn in the world of AVS and plugins programmers the 32bit float always treated as 0..1.0 nominal range with black at 0.0f and always internally converted into this range. It is very important to know that any going into float format around AVS and its plugins always require to convert back to narrow range in integer.

"ConvertBits(32)
ConvertToPlanarRGB()
w2xncnnvk(noise=0,model=0,gpu_thread=2)
ConvertToYUV420(matrix="Rec709")"

It looks in 32bit float matrix=Rec709 is no-op for range change ? It may be hidden feature of 32bit float AVS format.

"ConvertBits(8) "

It looks after any return back from float to integer you need to convert to narrow range (as usual integer moving picture data representation), so correct may be
ConvertBits(8, fulls=true, fulld=false)
or shorter
ConvertBits(8, fulld=false) because fulls for 32bit float input will be auto-detected as true (?).

AVS defaults looks like source of error if going into float: http://avisynth.nl/index.php/ConvertBits
bool fulls = (auto)

Conversion from and to float is always full-scale.

bool fulld = fulls

So as default 32bit float is full range and default fulld = fulls the simple
ConvertBits(32)
ConvertBits(8)
will cause range convert from narrow to full. The only safe and transparent ConvertBits is 8<->16 integer.

[tormento]

Conversion from and to float is always full-scale.
Ok but why the "fix" on Convertbits(32) worked and now, that I have a planar to YUV, not anymore?

[FranceBB]

Conversion from and to float is always full-scale.

bool fulld = fulls

So as default 32bit float is full range and default fulld = fulls the simple
ConvertBits(32)
ConvertBits(8)
will cause range convert from narrow to full.

Well... that's what I thought too, but actually it's not.
When you go to 32bit float, but stay in YUV, looks like it's still limited tv range, in fact even a simple:


ColorBars(848, 480, pixel_type="YV12")
ConvertBits(32)
TurnRight.Histogram.TurnLeft


shows that levels are still limited tv range:

https://i.imgur.com/0fG17aB.png


The conversion to Full Range happens when Tormento calls "ConverttoPlanarRGB()" 'cause RGB by standard is full range.
My "theory" was like yours, namely that when he called back ConverttoYUV420(matrix="Rec709") it was going to clip values rather than converting them back to limited, but no, I was wrong, in fact by doing all the operations he has in the script (except for Waifu) we get a perfectly identical image in terms of levels:

#Limited tv range source
ColorBars(848, 480, pixel_type="YV12")
original=last

#We go to 32bit float limited tv range
ConvertBits(32)

#We go from Limited TV Range YUV to Full Range RGB 32bit float
ConvertToPlanarRGB()

#We go back to Limited TV Range YUV 32bit float
ConverttoYUV420(matrix="Rec709")

#We go back to Limited TV Range YUV 8bit planar
ConvertBits(8)
converted=last

#We compare the two images
StackHorizontal(converted, original)

https://i.imgur.com/41Fyapw.png


So... it must be Waifu screwing it up, there's no other way...

[tormento]

So... it must be Waifu screwing it up, there's no other way...
So both convertbits and fmtc (as last command) should give the same output, thing that doesn't happen.