FrameRateConverter (Official)

[MysteryX]

I'm pretty sure this output is stranger then it should be :p
I'll answer like Microsoft. It's by design.

Just finished porting the first function. Only problem: GaussResize doesn't exist in VapourSynth. Also is BoxBlur equivalent to Avisynth Blur?

I could use some help here :stupid:


import vapoursynth as vs
core = vs.get_core()

def frc_GaussianBlur42(input, var = None, rad = None, vvar = None, vrad = None, p = None):
{
if not isinstance(input, vs.VideoNode):
raise vs.Error('frc_GaussianBlur42: input is not a clip')

var = max(0.0, float(var or 1.0)))
rad = max(1.0, float(rad or pow(var, 0.5)))
var = pow(min(max(0.0, rad), 60.0), 1.9) ## arbitrary max radius = 60

vvar = max(0.0, float(vvar or var)))
vrad = max(1.0, float(vrad or pow(vvar, 0.5)))
vvar = pow(min(max(0.0, vrad), 60.0), 1.9)
p = p or 19

w0 = input.width
h0 = input.height
w1 = round(w0/rad)
h1 = round(h0/vrad)

B = input.resize.Bilinear( \
min(max(4, w1 + (w1 % 2)), w0), \
min(max(4, h1 + (h1 % 2)), h0))

B = B.std.BoxBlur(hpasses=2, vpasses=2)

if var < 0.01 and vvar < 0.01:
return input
elif B.Width > 8 and B.Height > 8):
return B.std.GaussResize(w0, h0, p=p) ### NOT IN VAPOURSYNTH
else:
return B.std.Bilinear(w0, h0)
}


Note: just realized that ConvertFpsLimit can be quite useful in VapourSynth since ConvertFps doesn't exist! Good thing it's already ported.

[kedautinh12]

fmtconv have GaussResize
https://github.com/AkarinVS/fmtconv/blob/4f187bf5b7fe116d98a55910d597146b6b3a79a8/doc/fmtconv.html#L1080

[MysteryX]

Thanks. Another issue. I had already ported ConvertFps, but ChangeFps (no blending) doesn't exist in VapourSynth either. What's the best solution? It's actually used several times for masks.

Also to consider: what to do in case of variable framerate input?

Finally... MaskTools2 hasn't been ported to VapourSynth??

[kedautinh12]

Mt_lutspa: https://github.com/kewenyu/VapourSynth-Draw

mt_motion: https://github.com/dubhater/vapoursynth-motionmask

[kedautinh12]

havsfunc have ChangeFPS: https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/019ad633afa2982dfee8959435b2a0deb862c9f7/havsfunc.py#L5353

[kedautinh12]

I think you need convert masktool2 from avisynth to internal functions of Vapoursynth. Here for example convert:
https://forum.doom9.org/showthread.php?p=1802951

[MysteryX]

I need:
mt_inpand
mt_lut
mt_lutxy
mt_expand
mt_binarize
mt_circle
mt_merge

Pretty much the whole suite

[Reel.Deel]

I need:
mt_inpand
mt_lut
mt_lutxy
mt_expand
mt_binarize
mt_circle
mt_merge

Pretty much the whole suite


mt_lut and mt_lutxy: http://www.vapoursynth.com/doc/functions/expr.html

mt_binarize: http://www.vapoursynth.com/doc/functions/binarize.html

mt_merge: http://www.vapoursynth.com/doc/functions/maskedmerge.html#std.MaskedMerge

mt_inpand and mt_expand: http://www.vapoursynth.com/doc/functions/minimum_maximum.html


Not sure about mt_circle.

[kedautinh12]

mt_lut and mt_lutxy: http://www.vapoursynth.com/doc/functions/expr.html

mt_binarize: http://www.vapoursynth.com/doc/functions/binarize.html

mt_merge: http://www.vapoursynth.com/doc/functions/maskedmerge.html#std.MaskedMerge

mt_inpand and mt_expand: http://www.vapoursynth.com/doc/functions/minimum_maximum.html


Not sure about mt_circle.

That's was i said

[Reel.Deel]

That's was i said


Umm I guess ... You posted a link to another thread which has a script with all kinds of things that are mostly not relevant to the question that MysteryX asked ... but hey, it's nice to see you actually writing something instead of 'thanks' on every post :p

[kedautinh12]

Umm I guess ... You posted a link to another thread which has a script with all kinds of things that are mostly are not relevant to the question that MysteryX asked ...

Are you sure???

mt_merge = maskmerge
https://forum.doom9.org/showthread.php?p=1803090#post1803090

And some functions of Mastool2 in this script:
http://avisynth.nl/images/MfToon-v0.54.avsi

Was converted to internal functions from vapoursynth here:
https://gist.github.com/Frechdachs/b3a05afe2f7d25316a10cf7239d53d0c

And you say all kinds of things that are mostly are not relevant to the question that MysteryX asked???

[MysteryX]

I also need to convert this


Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true)
Overlay(MergeRGB(Blank, EM, EM), mode="Add", opacity=0.40, pc_range=true)

[ChaosKing]

I also need to convert this


Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true)
Overlay(MergeRGB(Blank, EM, EM), mode="Add", opacity=0.40, pc_range=true)


https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/019ad633afa2982dfee8959435b2a0deb862c9f7/havsfunc.py#L5427

[MysteryX]

mt_inpand and mt_expand: http://www.vapoursynth.com/doc/functions/minimum_maximum.html


this though...

mt_expand(mode= mt_circle(zero=true, radius=8))


Good news is that everything else is "converted" and non-tested. Just missing that last function.

[kedautinh12]

I seen here, but how to use avs functions in Vapoursynth scripts???
https://github.com/darealshinji/vapoursynth-plugins/blob/614d367e826b5f6bb0af33e1f91453f3ff7999ec/scripts/edgecleaner.py#L40

[MysteryX]

Making good progress.

This gives me "failed to convert '^' to float"

core.std.Expr(EM, "x 1.09 * 1.28 ^")
to replace
mt_lut(EM, "x " + string(dct_mult) + " * " + string(dct_pow) + " ^")


Performance looks much higher in Vapoursynth so far, but there are a couple of restrictions. Analysis can only be done with blksize 4x4, 8x8, 16x16 or 32x32, whereas in Avisynth I was using 8, 12, 16, 24,32

[ChaosKing]

I seen here, but how to use avs functions in Vapoursynth scripts???
https://github.com/darealshinji/vapoursynth-plugins/blob/614d367e826b5f6bb0af33e1f91453f3ff7999ec/scripts/edgecleaner.py#L40

VS can use (some) avs plugins. But avs script functions can't be used. For that there's avsw.Eval https://forum.doom9.org/showthread.php?t=175141



Performance looks much higher in Vapoursynth so far, but there are a couple of restrictions. Analysis can only be done with blksize 4x4, 8x8, 16x16 or 32x32, whereas in Avisynth I was using 8, 12, 16, 24,32
mvtools-sf by feisty seems to support even lower blksizes. BUT it will be slower and only supports 32bit float.
https://github.com/IFeelBloated/vapoursynth-mvtools-sf/blob/master/src/MVAnalyze.hxx#L304

https://forum.doom9.org/showthread.php?t=172525

[kedautinh12]

Thanks

[MysteryX]

mvtools-sf by feisty seems to support even lower blksizes. BUT it will be slower and only supports 32bit float.
It's blksize 12 and 24 that would be useful, which it doesn't support either.

Remains the issues of core.std.Expr(EM, "x 1.09 * 1.28 ^") and mt_expand with mt_circle

[wonkey_monkey]

Remains the issues of core.std.Expr(EM, "x 1.09 * 1.28 ^")

core.std.Expr(EM, "x 1.09 * 1.28 pow")

[MysteryX]

Comparing MVTools2 in VapourSynth vs Avisynth, performance is MUCH higher in VapourSynth, and the interpolation output is slightly different (slightly better in VapourSynth). The Mask output, however, is VERY different! A lot more detailed in VapourSynth. This means I'll have to retest and rebalance everything for VapourSynth.

VapourSynth version doesn't support DCT=0. The whole CalculateDiff was between DCT=4 and DCT=0, or between DCT=1 and DCT=0. Will have to see whether it makes sense to compare DCT=4 and DCT=1.

[real.finder]

I think that because MT is on in vs by default
LWLibavVideoSource("test.avi")
RequestLinear(clim=100)
super_search = MSuper(rfilter=4)

bv2 = super_search.MAnalyse(isb = true, delta = 2, overlap= 4)
bv1 = super_search.MAnalyse(isb = true, delta = 1, overlap= 4)
fv1 = super_search.MAnalyse(isb = false, delta = 1, overlap= 4)
fv2 = super_search.MAnalyse(isb = false, delta = 2, overlap= 4)

MDegrain2(MSuper(levels=1), bv1, fv1, bv2, fv2, thSAD=300, thSADC=150)
Prefetch()

as fast as

import vapoursynth as vs
core = vs.get_core()
clip=core.lsmas.LWLibavSource(source=r'test.avi')
super_search = core.mv.Super(clip,rfilter=4)

bv2 = core.mv.Analyse(super_search, isb = True, delta = 2, overlap= 4)
bv1 = core.mv.Analyse(super_search, isb = True, delta = 1, overlap= 4)
fv1 = core.mv.Analyse(super_search, isb = False, delta = 1, overlap= 4)
fv2 = core.mv.Analyse(super_search, isb = False, delta = 2, overlap= 4)

clip=core.mv.Degrain2(clip,core.mv.Super(clip,levels=1), bv1, fv1, bv2, fv2, thsad=300, thsadc=150)
clip.set_output()

don't know about output, didn't check if there are some parameter default difference between them

[MysteryX]

Good catch about MT. DCT=0: "usual spatial blocks, do not use DCT". So I wonder why I was using that for the slowest preset.

As for masks, there are MAJOR differences. I wonder why.

[MysteryX]

Raw mask is working fine, but I'm having issues converting it into a decent mask.

Here I do Binarize, followed by Bicubic, and it gives a weird vertical stretching. Here's what it does with Point resize at the end. Binarize+Point should give pure black&white. What's happening here? Mask is in Gray8 format.


EM = EM.resize.Bicubic(round(C.width/blkSize/4.0)*4, round(C.height/blkSizeV/4.0)*4)
EM = EM.std.Maximum()
EM = EM.std.Binarize(maskThr)
EM = EM.resize.Point(C.width, C.height)

https://i.postimg.cc/5HbfvqDw/Mask-Point.png (https://postimg.cc/5HbfvqDw)

Then I also have the issue that BoxBlur makes the blur too wide. I tried havsfunc.MinBlur but it makes smaller mask areas dimmer.

Here's a good mask produced in Avisynth. How can I reproduce something similar?

https://i.postimg.cc/jWcthnrB/MaskAvs.png (https://postimg.cc/jWcthnrB)

Here's the Avisynth code


EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(MaskThr)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

[kedautinh12]

B = input.resize.Bilinear( \
min(max(4, w1 + (w1 % 2)), w0), \
min(max(4, h1 + (h1 % 2)), h0))

B = B.std.BoxBlur(hpasses=2, vpasses=2)


Maybe replace boxblur with RemoveGrain


B = input.resize.Bilinear( \
min(max(4, w1 + (w1 % 2)), w0), \
min(max(4, h1 + (h1 % 2)), h0))

B = B.rgvs.RemoveGrain(mode=12).rgvs.RemoveGrain(mode=12) #Deprecated. Use Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1]) instead.

[MysteryX]

ah I can use this instead of Blur(1)
Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1])

but what about Blur(0.6) ?

[kedautinh12]

but what about Blur(0.6) ?

Dogway use this replaced blur(.6) with Expr, you can convert it to Vapoursynth code
https://github.com/Dogway/Avisynth-Scripts/blob/6eb7e81ea48717c648fc5140be0f3ce41d553123/EX%20mods/FrameRateConverter.avsi#L246

[kedautinh12]

Like ChaosKing say VS can use (some) avs plugins. You can use this script in VS replace EM.std.Maximum()
EM.avs.mt_expand(mode=avs.mt_circle(zero=true, radius=1))

Same here: https://github.com/darealshinji/vapoursynth-plugins/blob/614d367e826b5f6bb0af33e1f91453f3ff7999ec/scripts/edgecleaner.py#L40

[amayra]

Ported to VapourSynth


finally

[MysteryX]

I think that using Avisynth's MaskTools2 is the best option until someone decides to port it. It's really not difficult to write shared code for both platforms, I wish more people did that.

[kedautinh12]

Hope anyone port VapourSynth-RIFE-ncnn-Vulkan to avisynth

[MysteryX]

Like ChaosKing say VS can use (some) avs plugins. You can use this script in VS replace EM.std.Maximum()

Same here: https://github.com/darealshinji/vapoursynth-plugins/blob/614d367e826b5f6bb0af33e1f91453f3ff7999ec/scripts/edgecleaner.py#L40

How is he importing AVS into VapourSynth? All I can find is AvsProxy (https://forum.doom9.org/showthread.php?t=175141), which only has an Eval function.

That's not what he's using here, his syntax is much nicer. He lists his dependencies but doesn't say how he's loading them.

Then I still have the weird vertical blur issue on resize.

EM.std.Expr("x[0,1] 0.12 * x[0,0] 0.76 * x[0,-1] 0.12 * + +")
"Failed to convert x[0,1] to float." And is this exactly .6 strength?

[kedautinh12]

Here for load avs plugins in Vapoursynth
http://www.vapoursynth.com/doc/functions/loadpluginavs.html

[kedautinh12]

AvsProxy used for load scripts .avsi file in Vapoursynth not .dll. please read slowly what ChaosKing says

[kedautinh12]

Bad news
Avisynth plugins are never autoloaded. Support for this may be added in the future.
From here: http://www.vapoursynth.com/doc/plugins.html

[real.finder]

EM.std.Expr("x[0,1] 0.12 * x[0,0] 0.76 * x[0,-1] 0.12 * + +")
"Failed to convert x[0,1] to float." And is this exactly .6 strength?

cuz it use http://avisynth.nl/index.php/Expr#Pixel_addressing and seems it's only in avs+

[kedautinh12]

"Failed to convert x[0,1] to float." And is this exactly .6 strength?

Dogway say that: https://github.com/Dogway/Avisynth-Scripts/issues/13#issuecomment-867535798

[MysteryX]

Converting to VapourSynth isn't so simple..


core.std.LoadPlugin(path=r'C:\GitHub\FrameRateConverter\masktools2.dll')

"No entry point found". Note that if it doesn't find the file or it's a x86 DLL, it instead says "Failed to load".

List of pending issues:
- Blur(.6)
- Loading Avisynth DLL
- Resize weird vertical blurring

Blur(.6) may not be as fast as Blur(1), but there's a reason I put it to .6: otherwise the mask edges are too soft and makes artefacts leak through.

Oh wait, Dogway created ex_expand (https://github.com/Dogway/Avisynth-Scripts/blob/ccabc3d261a787e38f762b4976cc3e698f226880/ExTools.avsi#L531)(1,mode="circle"), does that work in VapourSynth? It's using the same Pixel Addressing feature; is that supported or not?

EDIT:
For AVS plugin, simply replace core.std.LoadPlugin with core.avs.LoadPlugin !

Now, this crashes without any message. EM format is Gray8.

EM = EM.avs.mt_expand(mode= core.avs.mt_circle(zero=True, radius=1))

[MysteryX]

OK now I'm running only mt_expand() to debug it. It "works" in YUV420P8 (with weird output?) and CRASHES in GRAY8.

Now if I run this (in YUV420P8), I get that.


video = video.avs.mt_expand(core.avs.mt_circle(zero=True, radius=1))
> Python exception: could not convert string to float: b'-1 0 0 -1 0 0 0 1 1 0 '


I don't know how others are using it, but converting Dogway's function may be the best option?

[kedautinh12]

Don't know cause Dogway still use same Pixel Addressing in his script

Edit: can you try AvsProxy??

[MysteryX]

Edit: can you try AvsProxy??
Could run the whole thing under AvsProxy; but then it isn't really a port.

Porting the Expr code supporting pixel addressing would allow to solve both problems (mt_expand and blur). Or can I run those expressions with AVS LoadPlugin? It's kind of weird if Avisynth's Expr would be more powerful than Vapoursynth's Expr.

[kedautinh12]

Porting the Expr code supporting pixel addressing would allow to solve both problems (mt_expand and blur).

You can ask Vapoursynth's developer about this porting
https://github.com/vapoursynth/vapoursynth/issues

[MysteryX]

This works to create the mask.

EM = EM.resize.Bicubic(round(C.width/blkSize/4.0)*4, round(C.height/blkSizeV/4.0)*4) \
.std.Maximum(coordinates=[0, 1, 0, 1, 1, 0, 1, 0]) \
.std.Binarize(maskThr) \
.std.Convolution(matrix=[0, 1, 0, 1, 3, 1, 0, 1, 0]) \
.resize.Bicubic(C.width, C.height)

Maximum with those coordinates is like mt_circle(radius=1). Convolution balanced in this way gives a decent output.

Blur(1) can be rewritten like this and then you can fine-tune as desired.

.std.Convolution(matrix=[10, 20, 10, 20, 40, 20, 10, 20, 10])

Apparently it also supports floats, which I never saw any mention before.

[MysteryX]

Having another issue with Overlay


# Prepare output=Over: Mask(cyan), Stripes(yellow)
FlowOver = Flow.Overlay(MergeRGB(Blank, EM, EM), mode="Add", opacity=0.40, pc_range=true)

becomes

Overlays = core.std.ShufflePlanes(clips=[Blank, EM, EM], planes=[0, 0, 0], colorfamily=vs.RGB).resize.Point(format=B.format, matrix_s="709")
FlowOver = havs.Overlay(Flow, Overlays, mode="addition", opacity=0.40)

This however causes the whole image to turn purple! Instead of adding the mask on top of it.

EDIT: if I convert Flow to RGB then it works

[MysteryX]

Here's an example of how the mask is different in VapourSynth than in Avisynth

Here's the mask with Analyze (without Recalculate)

Avisynth
https://i.postimg.cc/SXWjzsP2/MaskAvs.png (https://postimg.cc/SXWjzsP2)

VapourSynth
https://i.postimg.cc/cKqrHYFG/MaskVpy.png (https://postimg.cc/cKqrHYFG)

[kedautinh12]

Blur(1) can be rewritten like this and then you can fine-tune as desired.

.std.Convolution(matrix=[10, 20, 10, 20, 40, 20, 10, 20, 10])


But document here Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1]) same RemoveGrain (12) and blur(1)
http://www.vapoursynth.com/doc/functions/convolution.html

[kedautinh12]

Having another issue with Overlay


# Prepare output=Over: Mask(cyan), Stripes(yellow)
FlowOver = Flow.Overlay(MergeRGB(Blank, EM, EM), mode="Add", opacity=0.40, pc_range=true)

becomes

Overlays = core.std.ShufflePlanes(clips=[Blank, EM, EM], planes=[0, 0, 0], colorfamily=vs.RGB).resize.Point(format=B.format, matrix_s="709")
FlowOver = havs.Overlay(Flow, Overlays, mode="addition", opacity=0.40)

This however causes the whole image to turn purple! Instead of adding the mask on top of it.

EDIT: if I convert Flow to RGB then it works

try increase opacity in Vapoursynth

[MysteryX]

But document here Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1]) same RemoveGrain (12) and blur(1)
http://www.vapoursynth.com/doc/functions/convolution.html
It averages the value of a 3x3 square to determine its middle pixel. If you multiply all the values by 10, the weighting is exactly the same.

[MysteryX]

Great news! The VapourSynth script is now functional!

> Download Beta 3 here (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta)

Requires havsfunc and FMTC.

I changed all DLL parameter names to lowercase. FrameRateConverter took parameter stp to enable/disable stripe mask. Stp now specifies the mask threshold. Default was previously 23, now it is set to 35 by default. 0 to disable.

In Avisynth, DCT=0 is faster and was used for various presets. In VapourSynth, DCT=0 gives the same as DCT=4. Which brings 2 questions: what DCT to use for better performance, and what DCTs to use with the DIFF mode?

Need to play around with mask levels and thresholds. Stripe mask seemed to be working better before, might just be threshold settings. Needs more testing.

Also for now you need to set blkSize manually, as blkSize 12 and 24 aren't supported; but perhaps could still be used for StripeMask.

Right now most presets behave like "slow", with Calculate and Recalculate with DCT=4. If you want to do some testing, need to see what to offer at the various preset levels.

btw I'm getting MUCH better performance in VapourSynth. Same script preset "slow"

Avisynth: 10327ms at 29% CPU usage (100% would give 2994ms)
VapourSynth: 6442ms at 52% CPU usage (100% would give 3349ms ?)

Cheers!

[kedautinh12]

Thanks for your hardwork, MysteryX