Frame rate interpolation artifact problem

[Selur]

I got a sample forInterpolation.mp4 (https://drive.google.com/drive/folders/0B_WxUS1XGCPASUZibG5XZkRfeTg?usp=sharing) and when trying to get it from 29,97 fps to 60fps, I get a lot of artifacts in the fence that is visible.

I tried Interframe/SVP for Vapoursynth and I also stumbled over:
clip = core.std.AssumeFPS(clip, fpsnum=30000, fpsden=1001)
sup = core.mv.Super(clip, pel=2, hpad=0, vpad=0)
bvec = core.mv.Analyse(sup, blksize=16, isb=True, chroma=True, search=3, searchparam=1)
fvec = core.mv.Analyse(sup, blksize=16, isb=False, chroma=True, search=3, searchparam=1)
clip = core.mv.FlowFPS(clip, sup, bvec, fvec, num=60, den=1, mask=2)
#clip = core.mv.BlockFPS(clip, sup, bvec, fvec, num=60, den=1, mask=2)
problem is all those methods have lots of problems with fences.

-> Is there a solution for this?

Cu Selur

[burfadel]

There is this active thread about a script with basically the same purpose:
https://forum.doom9.org/showthread.php?t=174410

The script MysteryX is working on still needs tweaking I think. I played around with it myself an came up with the script in this post (same thread as above):
https://forum.doom9.org/showthread.php?p=1804437#post1804437

It's a lot more full on than your script, and a little more full on that MysteyX's combined script, but I found it handled some things better.

[Selur]

Thanks will try those scripts, I somehow overlooked that thread. :)
Ah, okay I mainly looked for a way using Vapoursynth. :)

[burfadel]

I forgot to mention in the post that modplus plugin is required for the modded script:
http://www.avisynth.nl/users/vcmohan/modPlus/modPlus.html

[Selur]

Thanks for the info

[Selur]

Just tried it with the default values and it produced some even worse artifacts than any of the others. :)
http://media.mediahump.com/image/259123/1/0/Frame 63.png (http://www.mediahump.com/image/259123/)

Cu Selur

[aegisofrime]

Hi Selur. That script is a Avisynth script right? Were you able to use it in Vapoursynth? I vaguely remember that Vapoursynth is able to load avs scripts, but my memory is rusty regarding this and Google is failing me.

[Selur]

Only tried the script from MysterX only using Avisynth MT and not Vapoursynth. :)

[Sharc]

Only tried the script from MysterX only using Avisynth MT and not Vapoursynth. :)It's much better now with MysteryX latest version (https://forum.doom9.org/showpost.php?p=1805171&postcount=186) and SkipOver=20, but the fence is still a challenge

[Selur]

Thanks for the info, really busy in real life atm. but will check it out at the weekend. :D

[Selur]

argh, did a fast check with 'FrameRateConverter(SkipOver=20)' and the output is jumping between the next frame and a frame multiple frames in the past,...

[Sharc]

Hmmm...I observed the jerkiness as well, before I resized to 1280x720 or set output="flow", I think.
Here my script which I used for comparison:
clip=DGSource("C:\Temp\Selur\forInterpolation.dgi")
clip=clip.bilinearresize(1280,720)
conv1=clip.changefps(60000,1001).subtitle("ChangeFPS",x=20,y=20,size=32)
conv2=clip.framerateconverter(NewNum=60000,NewDen=1001,preset="normal",BlkSize=32,output="auto",SkipOver=25).subtitle("FrameRateConverter",x=20,y=20,size=32)
conv3=clip.interframe(cores=4, gpu=false, newnum=60000, newden=1001, tuning="film").subtitle("Interframe",x=20,y=20,size=32)

return stackhorizontal(conv2,conv3)

[Selur]

framerateconverter(NewNum=60000,NewDen=1001,preset="normal",BlkSize=32,output="auto",SkipOver=25)
is crap to, because it simply repeats each frame which could be done a lot faster ;)

[Sharc]

No way.
conv1 duplicates, conv2 and conv3 interpolate.

Note that SkipOver=25 replaces the worst pictures only by duplicates. Set it to 0 in order to interpolate all frames.

[Selur]

Note that I use it on the full resolution version and:
# Frame Rate Converter
# Version: 26-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only;
## mask=mask only; skip=mask used by SkipOver; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc", int "SkipOver", clip "Prefilter")
{
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
Preset = Default(Preset, "normal")
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskGam = Default(MaskGam, .5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = Default(MaskOcc, 150)
MaskOcc = MaskTrh > 0 ? MaskOcc : 0
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Preset != "faster"
Prefilter = Default(Prefilter, Preset != "faster" ? C.RemoveGrain(22) : C)

Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster",
\ "FrameRateConverter: Preset must be slow, normal, fast or faster")
Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
Assert(MaskOcc >= 0 && MaskOcc <= 255, "FrameRateConverter: MaskOcc must be between 0 and 255")
Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast, faster
Recalculate = preset == "slow" || preset == "normal"
DCT = preset == "slow" ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255) : BlankClip(C)
EM = EM.ConvertToY8()
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(100)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
EMskipOut = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip, BlankClip(EM, color=$FFFFFF), BlankClip(EM),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM)
Sc = Sc.mt_binarize(128)

# Display SkipOver value on Output="over
Global GEMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver = Flow.ScriptClip("Subtitle(string(GEMskip.AverageLuma()))")

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "flow")==0) [** flow: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? B
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? EM
\ : (StrCmpi(Output, "skip")==0) [** skip: skip mask *]
\ ? EMskipOut
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ FlowOver.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|over)")
return R
}
LoadPlugin("G:\Hybrid\AVISYN~1\LSMASHSource.dll")
LoadPlugin("G:\Hybrid\AVISYN~1\masktools2.dll")
LoadPlugin("G:\Hybrid\AVISYN~1\mvtools2.dll")
LoadPlugin("G:\Hybrid\AVISYN~1\rgtools.dll")
# loading source: C:\Users\Selur\Desktop\forInterpolation.mp4
# input luminance scale tv
LWLibavVideoSource("C:\Users\Selur\Desktop\forInterpolation.mp4",cache=false,stacked=true,format="YUV420P8",repeat=true)
# current resolution: 3840x2160
framerateconverter(NewNum=60000,NewDen=1001,preset="normal",BlkSize=32,output="auto")
return last
still leaves me with every (aside from second frame a duplicate aside from the first)
Will do some testing over the weekend,..

[Sharc]

I couldn't reproduce the frame jumping anymore, but apparently Stainless experienced it as well (and fixed it in his version).
For comparison I encoded your clip with x264 and streamed it to my 50" TV. My subjective ranking:
1. (best): ConvertFPS (blending)
2. FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
3. ChangeFPS (frame duplication). Probably the TV applied its own interpolation here, introducing its own artefacts. I didn't tweak the TV options, maybe later ...
4. (worst): Interframe/SVPflow (tune="film")

Edit:
I have to revise my rating. Unless one focuses on the fence but rather on the girl on the bike, the viewing impression is:
1. (winner): FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
2. ConvertFPS (blending)
3. Change FPS (with interpolation by the TV)
4. Interframe/SVPflow (the fence artefacts are dominant)

[MysteryX]

1. (winner): FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)

:D

Now try with the new fallback option -- which I designed specifically for a fence-like problem I was having. It didn't actually help much there, but it's helping in other places.

[aegisofrime]

I couldn't reproduce the frame jumping anymore, but apparently Stainless experienced it as well (and fixed it in his version).
For comparison I encoded your clip with x264 and streamed it to my 50" TV. My subjective ranking:
1. (best): ConvertFPS (blending)
2. FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
3. ChangeFPS (frame duplication). Probably the TV applied its own interpolation here, introducing its own artefacts. I didn't tweak the TV options, maybe later ...
4. (worst): Interframe/SVPflow (tune="film")

Edit:
I have to revise my rating. Unless one focuses on the fence but rather on the girl on the bike, the viewing impression is:
1. (winner): FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
2. ConvertFPS (blending)
3. Change FPS (with interpolation by the TV)
4. Interframe/SVPflow (the fence artefacts are dominant)

Did you manage to run it in Vapoursynth? How's the speed like?