EDIResample

[feisty2]

well, problem fixed and no one else replied to this thread yet, I'll just change it to a release thread for the script

basically it's a vaporsynth port of mawen1250's nnedi3_resize16 with a few mods

what's this badass script?
- it's an upgraded version of fmtc.resample, it can resize your clip or resample the chroma (420-422-444) or do them both by one single call

what does it actually do, why wouldn't I just use fmtc.resample instead?
- it adopts both edi algorithm (only nnedi3 by now) and regular algorithm (spline or lanczos or whatever, your choice) for image upscaling and chroma upsampling, merges the results of 2 algos by Dither_limit_dif16, uses the edi result if dif >= 1.0 (8bit scale), blends 2 results if dif < 1, so, quality wise, and it supports gamma corrected resize and sigmoid curve resize also, the gamma corrected resizing is different from the original nnedi3_resize16, nnedi3_resize16 only resizes linearly at regular algo part due to the precision limit of nnedi3, this script resizes everything linearly cuz vaporsynth nnedi3 got support for 16bit

readme?
- only yuv420/422/444p16 and gray16 input supported, no support for 8 bit clips
- useless parameters like fh, fv... are removed
- no support for rgb output, do it urself with fmtc.matrix/matrix2020cl

link and dependencies?
link https://github.com/IFeelBloated/VaporSynth-Functions/blob/master/EDIResample.py
nnedi3 http://forum.doom9.org/showthread.php?t=166434
Dither http://forum.doom9.org/showthread.php?t=171525 (updated, don't use the one at #2, it got a range error in linearandgamma function)
fmtconv http://forum.doom9.org/showthread.php?t=166504

alternative version (from the original author of avs version) https://github.com/mawen1250/VapourSynth-script/blob/master/nnedi3_resample.py

[feisty2]

Dither.py, if it's really needed

import vapoursynth as vs

def get_msb (src):
core = vs.get_core ()
clip = core.fmtc.nativetostack16 (src)
return core.std.CropRel (clip, 0, 0, 0, (clip.height // 2))

def get_lsb (src):
core = vs.get_core ()
clip = core.fmtc.nativetostack16 (src)
return core.std.CropRel (clip, 0, 0, (clip.height // 2), 0)

def add16 (src1, src2, dif=True):
core = vs.get_core ()
if dif == True:
clip = core.std.MergeDiff (src1, src2)
else:
clip = core.std.Expr ([src1, src2], ["x y +"])
return clip

def sub16 (src1, src2, dif=True):
core = vs.get_core ()
if dif == True:
clip = core.std.MakeDiff (src1, src2)
else:
clip = core.std.Expr ([src1, src2], ["x y -"])
return clip

def max_dif16 (src1, src2, ref):
core = vs.get_core ()
clip = core.std.Expr ([src1, src2, ref], ["x z - abs y z - abs > x y ?"])
return clip

def min_dif16 (src1, src2, ref):
core = vs.get_core ()
clip = core.std.Expr ([src1, src2, ref], ["x z - abs y z - abs > y x ?"])
return clip

def limit_dif16 (flt, src, ref=None, thr=0.25, elast=3.0):
core = vs.get_core ()
thr = thr * 256
alpha = 1 / (thr * (elast - 1))
beta = elast * thr
ref = src if ref is None else ref
clip = core.std.Expr ([flt, src, ref], ["x z - abs {thr} <= x x z - abs {beta} >= ? y y {alpha} x y - * {beta} x y - abs - * + ?".format (thr=thr, alpha=alpha, beta=beta)])
return clip

def merge16_8 (src1, src2, mask):
core = vs.get_core ()
mask16 = core.fmtc.bitdepth (mask, bits=16, fulls=True, fulld=True)
clip = core.std.MaskedMerge (src1, src2, mask16)
return clip

def build_sigmoid_expr (string, inv=False, thr=0.5, cont=6.5):
core = vs.get_core ()
x1m0 = "1 {thr} 1 - {cont} * exp 1 + / 1 {cont} {thr} * exp 1 + / -".format (thr=thr, cont=cont)
x0 = "1 {cont} {thr} * exp 1 + /".format (thr=thr, cont=cont)

if inv == True:
expr = "{thr} 1 " + string + " {x1m0} * {x0} + 0.000001 max / 1 - 0.000001 max log {cont} / -".format (x1m0=x1m0, x0=x0, thr=thr, cont=cont)
else:
expr = "1 1 {cont} {thr} " + string + " - * exp + / {x0} - {x1m0} /".format (x1m0=x1m0, x0=x0, thr=thr, cont=cont)
return expr.format (thr=thr, cont=cont)

def sigmoid_direct (src, thr=0.5, cont=6.5):
core = vs.get_core ()
expr = build_sigmoid_expr ("x 65536 /", False, thr, cont)
clip = core.std.Expr ([src], [expr + " 65536 *"])
return clip

def sigmoid_inverse (src, thr=0.5, cont=6.5):
core = vs.get_core ()
expr = build_sigmoid_expr ("x 65536 /", True, thr, cont)
clip = core.std.Expr ([src], [expr + " 65536 *"])
return clip

def linear_and_gamma (src, l2g_flag=True, fulls=True, fulld=None, curve="srgb", gcor=1.0, sigmoid=False, thr=0.5, cont=6.5):
core = vs.get_core ()
if curve == "srgb":
k0 = "0.04045"
phi = "12.92"
alpha = "0.055"
gamma = "2.4"
elif curve == "709":
k0 = "0.081"
phi = "4.5"
alpha = "0.099"
gamma = "2.22222"
elif curve == "240":
k0 = "0.0912"
phi = "4.0"
alpha = "0.1115"
gamma = "2.22222"
elif curve == "2020":
k0 = "0.08145"
phi = "4.5"
alpha = "0.0993"
gamma = "2.22222"
else:
k0 = "0.04045"
phi = "12.92"
alpha = "0.055"
gamma = "2.4"

fulld = fulls if fulld is None else fulld

if fulls == True:
expr = "x 4096 - 56064 /"
else:
expr = "x 65536 /"

g2l = "{expr} {k0} <= {expr} {phi} / {expr} {alpha} + 1 {alpha} + / log {gamma} * exp ?".format (expr=expr, k0=k0, phi=phi, alpha=alpha, gamma=gamma)

if gcor != 1.0:
g2l = "{g2l} 0 >= {g2l} log {gcor} * exp {g2l} ?".format (g2l=g2l, gcor=gcor)

if sigmoid == True:
g2l = build_sigmoid_expr (g2l , True , thr, cont)
l2g = build_sigmoid_expr (expr , False , thr, cont)
else:
l2g = expr

if gcor != 1.0:
l2g = "{l2g} 0 >= {l2g} log {gcor} * exp {l2g} ?".format (l2g=l2g, gcor=gcor)

l2g = "{l2g} {k0} {phi} / <= {l2g} {phi} * {l2g} log 1 {gamma} / * exp {alpha} 1 + * {alpha} - ?".format (l2g=l2g, k0=k0, phi=phi, alpha=alpha, gamma=gamma)

if l2g_flag == True:
expr = l2g
else:
expr = g2l

if fulld == True:
expr = expr + " 56064 * 4096 +"
else:
expr = expr + " 65536 *"

clip = core.std.Expr ([src], [expr])
return clip

def gamma_to_linear (src, fulls=True, fulld=None, curve="srgb", gcor=1.0, sigmoid=False, thr=0.5, cont=6.5):
core = vs.get_core ()
clip = linear_and_gamma (src, False, fulls, fulld, curve, gcor, sigmoid, thr, cont)
return clip

def linear_to_gamma (src, fulls=True, fulld=None, curve="srgb", gcor=1.0, sigmoid=False, thr=0.5, cont=6.5):
core = vs.get_core ()
clip = linear_and_gamma (src, True, fulls, fulld, curve, gcor, sigmoid, thr, cont)
return clip

def sbr16 (src):
core = vs.get_core ()
rg11 = core.rgvs.RemoveGrain(src, 11)
rg11D = core.std.MakeDiff (src, rg11)
rg11DR = core.rgvs.RemoveGrain(rg11D, 11)
rg11DD = core.std.Expr ([rg11D, rg11DR], ["x y - x 32768 - * 0 < 32768 x y - abs x 32768 - abs < x y - 32768 + x ? ?"])
clip = core.std.MakeDiff (src, rg11DD)
return clip

def clamp16 (src, bright_limit, dark_limit, overshoot=0, undershoot=0):
core = vs.get_core ()
os = overshoot * 256
us = undershoot * 256

clip = core.std.Expr ([src, bright_limit, dark_limit], ["x y {os} + > y {os} + x ? z {us} - < z {us} - x ?".format (os=os, us=us)])
return clip

def Resize16nr (src, w=None, h=None, sx=0, sy=0, sw=0, sh=0, kernel="spline36", kernelh=None, kernelv=None, fh=1, fv=1, taps=4, a1=None, a2=None, a3=None, kovrspl=1, cnorm=True, center=True, fulls=None, fulld=None, cplace="mpeg2", invks=False, invkstaps=4, noring=True):
core = vs.get_core ()
w = src.width if w is None else w
h = src.height if h is None else h
kernelh = kernel if kernelh is None else kernelh
kernelv = kernel if kernelv is None else kernelv
sr_h = float (w / src.width)
sr_v = float (h / src.height)
sr_up = max (sr_h, sr_v)
sr_dw = 1.0 / min (sr_h, sr_v)
sr = max (sr_up, sr_dw)

thr = 2.5
nrb = (sr > thr)
nrf = (sr < thr + 1.0 and noring)
nrr = min (sr - thr, 1.0) if nrb else 1.0
nrv = [round ((1.0 - nrr) * 65535), round ((1.0 - nrr) * 65535), round ((1.0 - nrr) * 65535)] if nrb else [0, 0, 0]
nrm = core.std.BlankClip (clip=src, width=w, height=h, color=nrv) if nrb and nrf else 0

main = core.fmtc.resample (src, w=w, h=h, sx=sx, sy=sy, sw=sw, sh=sh, kernel=kernel, kernelh=kernelh, kernelv=kernelv, fh=fh, fv=fv, taps=taps, a1=a1, a2=a2, a3=a3, kovrspl=kovrspl, cnorm=cnorm, center=center, fulls=fulls, fulld=fulld, cplace=cplace, invks=invks, invkstaps=invkstaps)
nrng = core.fmtc.resample (src, w=w, h=h, sx=sx, sy=sy, sw=sw, sh=sh, kernel="gauss", a1=100, center=center, fulls=fulls, fulld=fulld, cplace=cplace) if nrf else main

clip = core.rgvs.Repair (main, nrng, 1) if nrf else main
clip = core.std.MaskedMerge (main, clip, nrm) if nrf and nrb else clip
return clip

[feisty2]

found a typo in EDInter,
now it's working :)

[feisty2]

updated

[foxyshadis]

Nice, this is going right into my toolbox. Does the blending give any difference compared to just using the nnedi result, once dithered down to 8 or 10 bit?

[feisty2]

yes, I guess, 1.0 is at 8bit scale (0-255), not 16bit scale (0-65535), so it's gonna be 256.0 at 16bit
when pscrn is on, only few parts of the image get interpolated by edi algo, bicubic would be used to interpolate other parts
the script would try to "replace" the bicubic interpolation by nnedi3
edit: mindless error

[DarkSpace]

yes, I guess, 1.0 is at 8bit scale (0-255), not 16bit scale (0-65535), so it's gonna be 16.0 at 16bit
No, it isn't. 2**8 is 256, so it's 256 at 16-bit scale...

[feisty2]

No, it isn't. 2**8 is 256, so it's 256 at 16-bit scale...

right.. my brain was wandering outside my body :p
it's 256

[DarkSpace]

It should be all right.
Also, I wasn't totally awake yet when I wrote that, so I'd like to apologize if my comment seemed rude. It was not intended to be (but I have a tendency to be even more concise when I'm tired than when I'm awake).

[MonoS]

Probably i'm doing it wrong, but i'm unable to get the correct result

This is my script
core.std.LoadPlugin("C:/Program Files (x86)/VapourSynth/plugins64/imwri/imwri.dll")

mat = "709"
core.std.LoadPlugin("C:/Program Files (x86)/VapourSynth/plugins64/imwri/imwri.dll")

src = core.imwri.Read("C:/Users/MonoS/Desktop/gamma_dalai_lama_gray.jpg")

yuv = core.fmtc.matrix(src,fulls=True, fulld=False,mats="RGB",matd=mat, csp=vs.YUV444P16)

downg = core.fmtc.resample(yuv, 129,111, invks=False, kernel="spline36")

y = core.fmtc.matrix(src,fulls=True, fulld=False,mats="RGB",matd=mat, csp=vs.GRAY16)
downEdi = edi.EDIResample(y, 129,111, curve=mat)
Ediyuv = core.std.ShufflePlanes([downEdi, downg], [0,1,2], vs.YUV)

lin = dit.gamma_to_linear(yuv, curve=mat, sigmoid=True, fulls=False)
down = core.fmtc.resample(lin, 129,111, invks=False, kernel="spline36")
gam = dit.linear_to_gamma(down, curve=mat, sigmoid=True, fulls=False)

core.std.StackVertical([downg,Ediyuv, gam]).set_output()

I get this result http://abload.de/img/provegammaxbuo9.png
The first one is to be expected but the other two are plain wrong [if i wrote it right].

The sample image is this one http://abload.de/img/gamma_dalai_lama_gray4wujj.jpg took from this site http://www.4p8.com/eric.brasseur/gamma.html? , the expected result should be this for the last two http://abload.de/img/gamma_dalai_lama_gray9aug3.png

[feisty2]

should be curve="709", there's no "curve 601" thing, I set anything other than valid curve values to srgb curve, so you are see the srgb curve result

EDIT: thx for the report, checking where the script went wrong

[MonoS]

Changed all the curve and mat parameter to 709, but still the same problem [but result is a bit different] http://abload.de/img/nuovovapoursynthpytho12u4n.png

[feisty2]

Is this correct?

src = core.imwri.Read("gamma_dalai_lama_gray4wujj.jpg")
r = core.std.ShufflePlanes(src, 0, vs.GRAY).fmtc.bitdepth (bits=16, fulls=True, fulld=True)
g = core.std.ShufflePlanes(src, 1, vs.GRAY).fmtc.bitdepth (bits=16, fulls=True, fulld=True)
b = core.std.ShufflePlanes(src, 2, vs.GRAY).fmtc.bitdepth (bits=16, fulls=True, fulld=True)
edir = EDIResample.EDIResample (r, 129,111, curve="srgb", sigmoid=False, fulls=True, fulld=True)
edig = EDIResample.EDIResample (g, 129,111, curve="srgb", sigmoid=False, fulls=True, fulld=True)
edib = EDIResample.EDIResample (b, 129,111, curve="srgb", sigmoid=False, fulls=True, fulld=True)
core.std.ShufflePlanes([edir, edig, edib], [0,0,0], vs.RGB).set_output ()

[feisty2]

the "expected" result is don't use gamma curve at all, just scale directly anyways

src = core.imwri.Read("gamma_dalai_lama_gray4wujj.jpg")
yuv = core.fmtc.matrix(src,fulls=True, fulld=False, mats="RGB",matd="709", csp=vs.YUV444P16)
edi = EDIResample.EDIResample(yuv, 129,111, curve="linear", fulls=False, sigmoid=False)
edi.set_output ()


EDIT: don't enable gamma corrected resizing on YUV clips, do it under RGB colorspace like #13

[feisty2]

Update: Fixed an error might lead to a logic crash when downscaling

[MonoS]

The fact is that: i was using that test image as a test for making a simple helper script for resampling in linear light [without using your EDIResampler] and was testing the result.

So this let me think that i misused fmtc.matrix()

[asarian]

Thanks for these! :) Just found em!