How to extend filldrops with gpu support?

[Selur]

Anyone got an idea how to extend filldrops (https://github.com/JJKylee/Filter-Scripts/blob/main/VapourSynth/filldrops.py) to use SVP (or RIFE) for interpolation?

Something along the lines of:
def FillDrops(clip, thresh=0.1, method="mv",debug=False):
core = vs.core
if not isinstance(clip, vs.VideoNode):
raise vs.Error('FillDrops: This is not a clip')
if method == "mv":
super = core.mv.Super(clip, pel=2)
vfe = core.mv.Analyse(super, truemotion=True, isb=False, delta=1)
vbe = core.mv.Analyse(super, truemotion=True, isb=True, delta=1)
filldrops = core.mv.`(clip, super, mvbw=vbe, mvfw=vfe, time=50)
elif method == "svp_gpu":
# SVP Interpolation
else:
raise vs.Error('FillDrops: Unknown method '+method)

def selectFunc(n, f):
# return clip if more than thresh changed
if f.props['PlaneStatsDiff'] > thresh:
if debug:
return core.text.Text(clip=clip,text="Org, diff: "+str(f.props['PlaneStatsDiff']),alignment=8)
return clip
else:
if debug:
return core.text.Text(clip=filldrops,text="Int, diff: "+str(f.props['PlaneStatsDiff']),alignment=8)
return filldrops

diffclip = core.std.PlaneStats(clip, clip[0] + clip)
fixed = core.std.FrameEval(clip, selectFunc, prop_src=diffclip)
return fixed
Using

super = core.svp1.Super(clip,"{gpu:1}")
vectors= core.svp1.Analyse(super["clip"],super["data"],clip,"{}")
filldrops = core.svp2.SmoothFps(clip,super["clip"],super["data"],vectors["clip"],vectors["data"],"{}")

for '# SVP Interpolation' doesn't work since it would not return the right frame.
So the questions is: Can this be done using SVP (or RIFE) ?

Cu Selur

[poisondeathray]

If you use differencefromprevious (ie. clip, clip[0] + clip) , frame zero will always be interpolated as a false positive (the original avs filldrops did that too) - because frame zero is duplicated to shift the stream for the metrics. (If you use differencetonext , the last frame will be interpolated as a false positive)

So for the selectFunc last line I think it should be:

return clip[0] + filldrops.std.Trim(first=1)



For RIFE I use an interpolation helper function , similar to RX for avisynth, and call that within the selectFunc

I hardcoded this to use the opencl version (because everyone can use it, instead of Nvidia only), but it would be better to parameterize model version as an option and cuda version. Right now I use separate call functions (RX2,3 etc.. and 2.x, 3.x, 4.0 model versions). I also hardcoded RGBS 709 conversion to/from src . I know, I know bad coding practices, but I'm just a simple user, anyone should feel free to improve it

Also for debugging I have the levels gamma=2 to "flash", but it's commented out below



def RX1(clip, firstframe=None, Model=1):
clip1 = core.std.AssumeFPS(clip, fpsnum=1, fpsden=1)
start = core.std.Trim(clip1, first=firstframe-1, length=1)
end = core.std.Trim(clip1, first=firstframe+1, length=1)
startend = start + end
r = core.resize.Point(startend, format=vs.RGBS, matrix_in_s="709")
r = core.rife.RIFE(r, model=Model)
r = core.resize.Point(r, format=clip.format, matrix_s="709")
r = core.std.Trim(r, first=1, last=1)
r = core.std.AssumeFPS(r, fpsnum=1, fpsden=1)
a = core.std.Trim(clip1, first=0, last=firstframe-1)
b = core.std.Trim(clip1, first=firstframe+1)
join = a + r + b
join = core.std.AssumeFPS(join, src=clip)
return join

def FillDropsRifeOCL(clip, thresh=0.3, debug=False, model=1):
core = vs.core
if not isinstance(clip, vs.VideoNode):
raise ValueError('This is not a clip')

def selectFunc(n, f):
if f.props['PlaneStatsDiff'] > thresh:
if debug:
return core.text.Text(clip=clip,text="Org, diff: "+str(f.props['PlaneStatsDiff']),alignment=8)
return clip

elif n < 1:
if debug:
return core.text.Text(clip=clip,text="Org, diff: "+str(f.props['PlaneStatsDiff']),alignment=8)
return clip[0]
else:
if debug:
return core.text.Text(clip=RX1(clip, firstframe=n),text="Int, diff: "+str(f.props['PlaneStatsDiff']),alignment=8)
return RX1(clip, firstframe=n)#.std.Levels(gamma=2, planes=[0])

diffclip = core.std.PlaneStats(clip, clip[0] + clip) #differencefromprevious
#diffclip = core.std.PlaneStats(clip, clip.std.Trim(first=1)) #differencetonext
fixed = core.std.FrameEval(clip, selectFunc, prop_src=diffclip)
return fixed

[poisondeathray]

The current filldrops (and avs filldrops) interpolates the wrong frame for the drop,dup scenario

There should be a way to use multiple conditionals to differentiate vs. dupe,drop vs. drop,dupe , or 2,3,x dupe placeholders, etc... but it gets complicated for my basic skills . No doubt it's possible with python . eg. if frame duplicates, then do the 2 frame interpolation ,if triplicates , then do the 3 frame interp...

[Selur]

Nice had a few busy days, but will look at it tomorrow.

[Selur]

Rearrange the script a bit:
def fillWithMVTools(clip):
super = core.mv.Super(clip, pel=2)
vfe = core.mv.Analyse(super, truemotion=True, isb=False, delta=1)
vbe = core.mv.Analyse(super, truemotion=True, isb=True, delta=1)
return core.mv.FlowInter(clip, super, mvbw=vbe, mvfw=vfe, time=50)

def fillWithRIFE(clip, firstframe=None, rifeModel=1, rifeTTA=False, rifeUHD=False):
clip1 = core.std.AssumeFPS(clip, fpsnum=1, fpsden=1)
start = core.std.Trim(clip1, first=firstframe-1, length=1)
end = core.std.Trim(clip1, first=firstframe+1, length=1)
startend = start + end

r = core.resize.Point(startend, format=vs.RGBS, matrix_in_s="709")
r = core.rife.RIFE(r, model=rifeModel, tta=rifeTTA,uhd=rifeUHD)
r = core.resize.Point(r, format=clip.format, matrix_s="709")

r = core.std.Trim(r, first=1, last=1)
r = core.std.AssumeFPS(r, fpsnum=1, fpsden=1)
a = core.std.Trim(clip1, first=0, last=firstframe-1)
b = core.std.Trim(clip1, first=firstframe+1)
join = a + r + b
return core.std.AssumeFPS(join, src=clip)

def fillWithSVP(clip, firstframe=None, gpu=False): # Here I go wrong since I select the wrong frames
clip1 = core.std.AssumeFPS(clip, fpsnum=1, fpsden=1)
start = core.std.Trim(clip1, first=firstframe-1, length=1)
end = core.std.Trim(clip1, first=firstframe+1, length=1)
startend = start + end

if gpu:
super = core.svp1.Super(startend,"{gpu:1}")
else:
super = core.svp1.Super(startend,"{gpu:0}")
vectors= core.svp1.Analyse(super["clip"],super["data"],startend,"{}")
r = core.svp2.SmoothFps(startend,super["clip"],super["data"],vectors["clip"],vectors["data"],"{}")

r = core.std.Trim(r, first=1, last=1)
r = core.std.AssumeFPS(r, fpsnum=1, fpsden=1)
a = core.std.Trim(clip1, first=0, last=firstframe-1)
b = core.std.Trim(clip1, first=firstframe+1)
join = a + r + b
return core.std.AssumeFPS(join, src=clip)

def FillDrops(clip, thresh=0.3, method="mv", rifeModel=1, rifeTTA=False, rifeUHD=False, debug=False):
core = vs.core
if not isinstance(clip, vs.VideoNode):
raise ValueError('This is not a clip')

def selectFunc(n, f):
if f.props['PlaneStatsDiff'] > thresh or n == 0:
if debug:
return core.text.Text(clip=clip,text="Org, diff: "+str(f.props['PlaneStatsDiff']),alignment=8)
return clip
else:
if method == "mv":
filldrops=fillWithMVTools(clip)
elif method == "svp":
filldrops=fillWithSVP(clip,n)
elif method == "svp_gpu":
filldrops=fillWithSVP(clip,n,gpu=True)
elif method == "rife":
filldrops = fillWithRIFE(clip,n,rifeModel,rifeTTA,rifeUHD)
else:
raise vs.Error('FillDrops: Unknown method '+method)

if debug:
return core.text.Text(clip=filldrops,text=method+", diff: "+str(f.props['PlaneStatsDiff']),alignment=8)
return filldrops

diffclip = core.std.PlaneStats(clip, clip[0] + clip) #differencefromprevious
#diffclip = core.std.PlaneStats(clip, clip.std.Trim(first=1)) #differencetonext
fixed = core.std.FrameEval(clip, selectFunc, prop_src=diffclip)
return fixed

clip = FillDrops(clip,thresh=0.01, method="mv", rifeModel=0, rifeTTA=False, rifeUHD=False, debug=True)
Thanks that works fine now! :)


There should be a way to use multiple conditionals to differentiate vs. dupe,drop vs. drop,dupe , or 2,3,x dupe placeholders, etc... but it gets complicated for my basic skills . No doubt it's possible with python . eg. if frame duplicates, then do the 2 frame interpolation ,if triplicates , then do the 3 frame interp...
yeah,...

Cu Selur

[Emulgator]

That is what I came up with back in 2012, reinterpolates single and double drops:

# filldrops3 based on MugFunky's filldrops from 19.December 2005, extended by Emulgator on 26.Feb.2012 to accommodate interpolation of Doupledrops
# http://forum.doom9.org/showthread.php?p=753779#post753779
# "...so i wrote a little function to replace all frames that are identical to the previous one with a motion-interpolation of the adjacent frames.
# it could be useful for captures that you can't get again, but are full of drops..."
# Needs MVTools2, deviation now tunable.
function filldrops3 (clip c, float "dev")
{
blksz=16 # Blocksize for Motion search. Default = 8. My Default 16: Delivers a smoother picture. 4: Gives too many artefacts with poorer source.
searchp=32 # Radius for Motion search. Default = 2, My Default = 32
bld=true # Decides what to do if a Scenechange is detected by the following parameters: true will blend, false will repeat previous frame.
SCDpb=800 # Default = 400. thSCD1. SAD (pixels*luma values) Threshold per Block. A Block exceeding this SAD is considered changed.
## SAD = Sum of Absolute Differences = Number of pixels per 8x8 block (=64) multiplied with their luma difference.
## SCDpb is always given related to 8x8 blocks and scaled internally to match other block sizes. SCDbp is thSCD1 and is evaluated before SCDpf=thSCD2
SCDpf=200 # 0-255, Default=130. thSCD2. Changed Blocks percentage threshold per frame. 0=0%, 127=50%, 255 = 100%.
## This Threshold sets which minimal percentage of changed blocks a frame has to contain to be considered as a scene change.

dev=default(dev, 2.0)
super=MSuper(c,pel=2)

back_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=true, delta=1, temporal=false, dct=0, divide=0, trymany=false)#delta=framedistance;isb=IS Backward
# Looks 1 frame ahead and calculates vectors backward to reference frame.
forw_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=false, delta=1, temporal=false, dct=0, divide=0, trymany=false)#search 3:=Exhaustive; 4:=Hex(default); 5:=UMH
# Looks 1 frame behind and calculates vectors forward to reference frame.
dblback_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=true, delta=2, temporal=false, dct=0, divide=0, trymany=false)#delta=framedistance;isb=IS Backward
# Looks 2 frames ahead and calculates vectors backward to reference frame. To be used when reference frame + 1 has to be excluded.
dblforw_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=false, delta=2, temporal=false, dct=0, divide=0, trymany=false)#search 3:=Exhaustive; 4:=Hex(default); 5:=UMH
# Looks 2 frames behind and calculates vectors forward to reference frame. To be used when reference frame -1 has to be excluded.

global fillsingdrops = MFlowInter(c,super,back_vec,forw_vec,time=50, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global filldoubdrops33 = MFlowInter(c,super,dblback_vec,dblforw_vec,time=33, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global filldoubdrops67 = MFlowInter(c,super,dblback_vec,dblforw_vec,time=67, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global oric=c
global devthresh=dev


#ConditionalFilter(c, c, fillsingdrops, "YDifferenceFromPrevious()", ">", "devthresh", show=true)##works for mending of single framerepeats
#ConditionalFilter(c, c, filldoubdrops33, "YDifferenceFromPrevious()>devthresh", "&&", "YDifferenceToNext()>devthresh", show=true)##wrong syntax

c.scriptclip("""(YDifferenceFromPrevious<devthresh) && (YDifferenceToNext<devthresh) ? (filldoubdrops33++filldoubdrops67.Trim(1,0)) : (YDifferenceFromPrevious<devthresh) ? fillsingdrops : oric""")

}

Might be a hint to get this into vapoursynth...

[DVB-Freak]

Hello Selur,

I have a question how i implement your script.
I made a "FillDrops4.avsi" with the text of your script in it and put in the plugins-folder of avisynth.
In my script i add than "Filldrops4(fillWithRIFE)". VirtualDub show me a syntax-error. What is the correct way?

[Selur]

@DVB-Freak: This is a Vapoursynth script.

# This is a VapourSynth port of FillDrops by johnmeyer
# https://forum.doom9.org/showthread.php?p=1775184#post1775184

the Avisynth script from johnmeyer is probably what you want. (but that does not have RIFE support)
My latest version for Vapoursynth can be found here: https://github.com/Selur/VapoursynthScriptsInHybrid/blob/master/filldrops.py

Cu Selur

[DVB-Freak]

Okay, thank you