ResampleHQ replacement for VS. ¿is this?

[sl1pkn07]

Hi

according of this:

http://forum.doom9.org/showthread.php?p=1545006#post1545006

this is a ResampleHQ replacement for VS? (uses a fmtconv (http://forum.doom9.org/showthread.php?t=166504) and dither (http://forum.doom9.org/showthread.php?t=171525))


video = core.fmtc.bitdepth (video, bits=16)
video = dither.gamma_to_linear(video)
video = dither.Resize16nr(video, w=1280, h=720)
video = dither.linear_to_gamma(video)

optional:*

video = core.fmtc.bitdepth (video, bits=8)

*: what is better? add 16->8 conversion to VS (the optional)? or active the 16bits hack in the x264 (or i make blind shots)

greetings

[feisty2]

gamma linear functions are designed for rgb colorspaces, don't use them on yuv clips, convert your clip to rgb first and do it under rgb then

[mawen1250]

You should apply the gamma conversion in RGB color space for proper and correct gamma-aware processing, which is also what ResampleHQ does internally.
As for the final output bit depth, you can do whatever you want as long as the encoder support it and the input depth is correctly specified. With y4m output you don't need to specify color format explicitly.

[sl1pkn07]

thanks @feisty2 and @mawen1250 (and @Are_ in the IRC)

then is like this? (remove the 16bit conversion due a make automatic by resize16nr)


video = core.fmtc.resample(video, css="444")
video = core.fmtc.matrix(video, mat="709",col_fam=vs.RGB)
video = dither.gamma_to_linear(video)
video = dither.Resize16nr(video, w=1280, h=720)
video = dither.linear_to_gamma(video)
video = core.fmtc.matrix(video, mat="709", col_fam=vs.YUV)
video = core.fmtc.resample(video, css="420")


greetings

edit: forgotten steps

[feisty2]

http://forum.doom9.org/showthread.php?t=171741
see #13
edit: your script should be correct also, didn't test tho.

[Are_]

Something like this maybe resamplehq (https://gist.github.com/4re/64642122e359c37543fe).

[sl1pkn07]

thanks @Are_!

[MonoS]

@Are_: What if i need to upscale the chroma using nnedi??

Do i need to use your script only for the luma channel and then merge the upscaled chroma back in??

[Are_]

Not sure, but if you are gonna use nnedi for linear scaling just go with feisty2's script, why bother overcomplicating stuff?

[MonoS]

Because i'd like to understand of things should works before blindly using some script [and maybe rewrite it as i did with my GCResizer].

Feisty2's script don't do any processing in RGB [only in YUV by default] and don't linearize the chroma channel [if i didn't misunderstood the script], but as far as i can understand reading some threads [linked here and there is resizing threads] not doing so introduce some incorrectness in the process.

[feisty2]

Feisty2's script don't do any processing in RGB [only in YUV by default] and don't linearize the chroma channel

yet again, gamma stuff are meant to be applied on RGB clips, use them on luma is wrong enough and use them on chroma..., that's even more so.
take the following steps
1 convert your clip to yuv444 with EDIResample
2 convert your clip to RGB with fmtc.matrix
3 use the script at #13 from EDIResample thread or #4 of this thread

[mawen1250]

https://github.com/mawen1250/VapourSynth-script

Some days ago I wrote a VS script named nnedi3_resample, mainly designed to do the same things as my AVS script nnedi3_resize16 does, with many changes:
1. Remove some processes in the original nnedi3_resize16 which makes the script too complicated and confusing, such as pre-padding to mod4 (for AVS 2.5 without Y8 support), pre-cropping (improve nnedi3 processing speed if lots of pixels will be cropped).
2. Make the center/chroma shift calculation clearer and more robust.
3. Remove the mixing mode of resizer and nnedi3, since it doesn't make much sence as long as VS nnedi3 supports 16bit processing.
4. Gamma-aware resizing is always done in RGB/Gray color space instead of YUV.
5. Many kinds of conversions are designed to support, including color family, matrix, transfer characteristics(gamma), full/limited range, subsampling, bit depth, etc. However, this results in somewhat complicated conversion/processing in the script, and currently it's far from being completed. Apart from some simple conversions such as RGB<->YUV and subsampling, there may be many bugs in the conversions right now.

Note:
1. I've only tested some commonly used cases, so use it at your own risk.
2. I'm too lazy to improve it recently.

[MonoS]

I'm making some modification to mawen new nnedi3_resample [i already did one some weeks ago].
First, I'm only concerned about gamma corrected resizing, i didn't study the code for the other modes.

I'd like to make the code use a different kernel [and relative settings] for luma and chroma, but being in RGB colorspace there is no distinction between Y and UV.

So i was thinking about doing something like this
1) Convert the original clip in RGB Linear colorspace
2) Apply resizing twice with different kernels
3) Reconvert both to YUV Gamma colorspace
4) Pick the right channels from the respective clips

Does it have at least some minimum sense??

Thanks for the attention

[foxyshadis]

You might want to page mp4 guy or colors to the thread, or PM one of them, for an answer to that. Aside from that, testing it and comparing visually wouldn't be too hard, with a quick:

final=std.ShufflePlanes(clips=[spline_clip,bicubic_clip],planes=[0,1,2],colorfamily=vs.YUV)

[mawen1250]

I'm making some modification to mawen new nnedi3_resample [i already did one some weeks ago].
First, I'm only concerned about gamma corrected resizing, i didn't study the code for the other modes.

I'd like to make the code use a different kernel [and relative settings] for luma and chroma, but being in RGB colorspace there is no distinction between Y and UV.

So i was thinking about doing something like this
1) Convert the original clip in RGB Linear colorspace
2) Apply resizing twice with different kernels
3) Reconvert both to YUV Gamma colorspace
4) Pick the right channels from the respective clips

Does it have at least some minimum sense??

Thanks for the attention
I guess this will not make much sense. My idea is to convert the linear RGB color space to a linear YUV-like color space. Perhaps opponent color space can be a good choice:
Right now I'm implementing a BM3D denoising filter for VapourSynth, which filters a color image in opponent color space (suggested by BM3D related articles), and the result is better than filtering in RGB, YCgCo, BT.709 YCbCr color space in my tests.

The matrix coefficients for RGB->OPP is:
1/3 1/3 1/3
1/2 0 -1/2
1/4 -1/2 1/4
And OPP->RGB:
1 1 2/3
1 0 -4/3
1 - 1 2/3
Custom matrix can be assigned to fmtc.matrix with "coef".

[MonoS]

Ooook, i did some tests:

doppio = False
sigmoid = True
linear = False
test = edi.nnedi3_resample(src, 1920, 1036, csp=vs.RGB48).fmtc.bitdepth(flt=True)

test = haf.GammaToLinear(test, True, True, "709", sigmoid=sigmoid) if linear else test
test1 = core.fmtc.bitdepth(test, flt=True).fmtc.matrix(fulls=True, fulld=True, coef=[1/3, 1/3, 1/3, 1, 1/2, 0, -1/2, 1, 1/4, -1/2, 1/4, 1])

test1Y = core.fmtc.resample(test1, 960, 518, kernel="bilinear", invks=True, planes=[3, 1, 1])
test1UV = core.fmtc.resample(test1, 960, 518, kernel="bicubic", invks=True, planes=[1, 3, 3])

test1 = core.std.ShufflePlanes([test1Y, test1UV], [0, 1, 2], vs.RGB)
if doppio:
testY = core.fmtc.resample(test, 960, 518, kernel="bilinear", invks=True)
testUV = core.fmtc.resample(test, 960, 518, kernel="bicubic", invks=True)
else:
test = core.fmtc.resample(test, 960, 518, kernel="bilinear", invks=True)

test1 = core.fmtc.matrix(test1, fulls=True, fulld=True, coef=[1, 1, 2/3, 1, 1, 0, -4/3, 1, 1, -1, 2/3, 1])#.fmtc.bitdepth(bits=16, dmode=1)
if doppio:
testY=haf.LinearToGamma(testY, True, True, "709", sigmoid=sigmoid) if linear else testY
testUV=haf.LinearToGamma(testUV, True, True, "709", sigmoid=sigmoid) if linear else testUV
else:
test=haf.LinearToGamma(test, True, True, "709", sigmoid=sigmoid) if linear else test

test1 = haf.LinearToGamma(test1, True, True, "709", sigmoid=sigmoid) if linear else test1

if doppio:
testY = core.fmtc.matrix(testY, mat="709", fulls=True, fulld=False, col_fam=vs.YUV, singleout=-1)
testUV = core.fmtc.matrix(testUV, mat="709", fulls=True, fulld=False, col_fam=vs.YUV, singleout=-1)
test = core.std.ShufflePlanes([testY, testUV], [0, 1, 2], vs.YUV)
else:
test = core.fmtc.matrix(test, mat="709", fulls=True, fulld=False, col_fam=vs.YUV, singleout=-1)

test = core.fmtc.matrix(test, mat="709", fulls=True, fulld=False, col_fam=vs.YUV, singleout=-1)
test1 = core.fmtc.matrix(test1, mat="709", fulls=True, fulld=False, col_fam=vs.YUV, singleout=-1)

#test1 = core.fmtc.resample(fix, 960,518, kernel="bilinear", invks=True, css="444")

core.std.MakeDiff(test, test1).fmtc.bitdepth(dmode=1, bits=16).fmtc.nativetostack16().std.ShufflePlanes([0], vs.GRAY).set_output()


there are some issues:
-If i try to make the conversion in linear light i have to do the calculation in integer format, if i do so i get very different results, instead if i do in gamma RGB i only get minimum differences between the RGB clip and the OPP clip [that if i didn't screwed the script]
-I always get some minimum differences in the luma channel when using different kernel for Y and UV in the OPP version, even if i use the same kernel for the RGB version, but i think this is to be expected [in fact the differences are only present in the MSB]

What do you think??

[mawen1250]

I've compared the output of fmtc.matrix with my implemented bm3d.RGB2OPP.
The Y is identical but the U and V of fmtc.matrix is almost zero, maybe it's a bug of the custom coef?

More tests:
With integer input, the UV are not properly offset (I've set col_fam=vs.YUV).
With float input, the result is almost identical.

I always get some minimum differences in the luma channel when using different kernel for Y and UV in the OPP version
Sorry I don't get what you mean...

[MonoS]

I've tested the output of fmtc.matrix with my implemented bm3d.RGB2OPP.
The Y is nearly identical but the U and V of fmtc.matrix is almost zero, maybe it's a bug of the custom coef?


Please, before claiming bug in cretindesalpes code, are you sure that my code is correct??

[cretindesalpes]

Indeed I think there is a problem in the fmtc.matrix code… I’m going to check it.

[MonoS]

Sorry I don't get what you mean...

in you set doppio at True and set the same kernel to the opp and the rgb version, but different between the Y and UV, i get some differences in the Y channel when i downconvert to YUV, these difference don't exist if i set the same kernel in the OPP version [that match the kernel on the Y call from the RGB version]

With this code i get some stuff in the MSB of the resulting diff

test1Y = core.fmtc.resample(test1, 960, 518, kernel="bilinear", invks=True, planes=[3, 1, 1])
test1UV = core.fmtc.resample(test1, 960, 518, kernel="bicubic", invks=True, planes=[1, 3, 3])

test1 = core.std.ShufflePlanes([test1Y, test1UV], [0, 1, 2], vs.RGB)
if doppio:
testY = core.fmtc.resample(test, 960, 518, kernel="bilinear", invks=True)
testUV = core.fmtc.resample(test, 960, 518, kernel="bicubic", invks=True)


Instead with this one i get no difference

test1Y = core.fmtc.resample(test1, 960, 518, kernel="bilinear", invks=True, planes=[3, 1, 1])
test1UV = core.fmtc.resample(test1, 960, 518, kernel="bilinear", invks=True, planes=[1, 3, 3])

test1 = core.std.ShufflePlanes([test1Y, test1UV], [0, 1, 2], vs.RGB)
if doppio:
testY = core.fmtc.resample(test, 960, 518, kernel="bilinear", invks=True)
testUV = core.fmtc.resample(test, 960, 518, kernel="bicubic", invks=True)


This happens only with the Y channel, not in the UV channels

[mawen1250]

I think it is not that strange.
Employing different kernels for Y and UV is mainly because they are of very different characteristics: different SNR, different kinds of information, etc. Y, U, V channels is decorrelation of the R, G, B channels, and Y, R, G, B are of the similar characteristics.
Thus applying different kernels on RGB/RGB and converting it back to Y/UV won't involve the characteristics of UV, and this kind of combination is meaningless IMO.

[MonoS]

I think it is not that strange.
Employing different kernels for Y and UV is mainly because they are of very different characteristics: different SNR, different kinds of information, etc. Y, U, V channels is decorrelation of the R, G, B channels, and Y, R, G, B are of the similar characteristics.
Thus applying different kernels on RGB/RGB and converting it back to Y/UV won't involve the characteristics of UV, and this kind of combination is meaningless IMO.

As expected you are right.

Tried with this code

fin1=core.std.ShufflePlanes([core.fmtc.resample(src, 1280,720, kernel="bilinear", invks=False, planes=[3,1,1]),core.fmtc.resample(core.nnedi3.nnedi3_rpow2(src, rfactor=2, correct_shift=True), 1280,720, kernel="spline64", invks=False, planes=[1,3,3], css="444")], [0,1,2], vs.YUV)
"""
res = edi.nnedi3_resample(src, 1920, 1080, csp=vs.RGB48, curves="709", curved="linear", mats="709", matd="709", fulls=True, fulld=True, sigmoid)
lin = has.GammaToLinear(res, True, True, "709", sigmoid=True) if linear else res

linY = core.fmtc.resample(lin, 1280,720, kernel="bilinear", invks=True)
linUV= core.fmtc.resample(lin, 1280,720, kernel="spline64", invks=False)

gammaY = has.LinearToGamma(linY, True, True, "709", sigmoid=True) if linear else linY
gammaUV= has.LinearToGamma(linUV, True, True, "709", sigmoid=True) if linear else linUV

Y = core.fmtc.matrix(gammaY, mat="709", fulls=True, fulld=False, col_fam=vs.YUV, singleout=-1)
UV = core.fmtc.matrix(gammaUV, mat="709", fulls=True, fulld=False, col_fam=vs.YUV, singleout=-1)
fin2 = core.std.ShufflePlanes([Y, UV], [0, 1, 2], vs.YUV)
"""

resY = edi.nnedi3_resample(src, 1280, 720, kernel="bilinear", mats="709", curves="709", sigmoid=True, fulls=True, csp=vs.YUV444P16)
resUV = edi.nnedi3_resample(src, 1280, 720, kernel="spline64", mats="709", curves="709", sigmoid=True, fulls=True, scale_thr=1.0, csp=vs.YUV444P16)

fin2 = core.std.ShufflePlanes([resY, resUV], [0,1,2], vs.YUV)

fin2.set_output()
core.std.MakeDiff(fin1, fin2).fmtc.nativetostack16().std.ShufflePlanes([0], vs.GRAY).set_output()

and all three planes are pretty different between the two methods.

I got interested of your OPP version and wanted to try ASAP how it worked and never tried the standard method.

In your knowledge, is possible to use both different kernel for Y and UV with scaling in linear light??

[mawen1250]

In your knowledge, is possible to use both different kernel for Y and UV with scaling in linear light??
Then the question is how to define the Y,U,V here:
1. If you just want to employ the characteristics of YUV color space in linear resize, then converting linear RGB to linear YUV is the proper solution, as what I proposed at first.
2. If you want to do the resizing in original Y'Cb'Cr' color space. No way, since the original Y'Cb'Cr' color space is not a linear one, it is defined to be a gamma-corrected one.
3. Alternatively, you can convert the Y' to linear Y, then resize in YCb'Cr'. It applies linear resize to luma but not chroma, and the result is not correct from the viewpoint of color science.

Besides, I see you were using inverse kernel for downscaling, then I assume you were trying to reverse upscaled source?
If it is, rather than considering resizing in linear light, it's more appropriate to resize in a color space where the source is originally upscaled. May be Y'Cb'Cr, R'G'B, RGB, etc. depending on the source.