Frame rate interpolation artifact problem

[Selur]

I got a sample forInterpolation.mp4 and when trying to get it from 29,97 fps to 60fps, I get a lot of artifacts in the fence that is visible.

I tried Interframe/SVP for Vapoursynth and I also stumbled over:

clip = core.std.AssumeFPS(clip, fpsnum=30000, fpsden=1001)
sup = core.mv.Super(clip, pel=2, hpad=0, vpad=0)
bvec = core.mv.Analyse(sup, blksize=16, isb=True, chroma=True, search=3, searchparam=1)
fvec = core.mv.Analyse(sup, blksize=16, isb=False, chroma=True, search=3, searchparam=1)
clip = core.mv.FlowFPS(clip, sup, bvec, fvec, num=60, den=1, mask=2)
#clip = core.mv.BlockFPS(clip, sup, bvec, fvec, num=60, den=1, mask=2)

problem is all those methods have lots of problems with fences.

-> Is there a solution for this?

Cu Selur

[burfadel]

There is this active thread about a script with basically the same purpose:
https://forum.doom9.org/showthread.php?t=174410

The script MysteryX is working on still needs tweaking I think. I played around with it myself an came up with the script in this post (same thread as above):
https://forum.doom9.org/showthread.p...37#post1804437

It's a lot more full on than your script, and a little more full on that MysteyX's combined script, but I found it handled some things better.

[Selur]

Thanks will try those scripts, I somehow overlooked that thread.
Ah, okay I mainly looked for a way using Vapoursynth.

[burfadel]

I forgot to mention in the post that modplus plugin is required for the modded script:
http://www.avisynth.nl/users/vcmohan...s/modPlus.html

[Selur]

Thanks for the info

[Selur]

Just tried it with the default values and it produced some even worse artifacts than any of the others.


Cu Selur

[aegisofrime]

Hi Selur. That script is a Avisynth script right? Were you able to use it in Vapoursynth? I vaguely remember that Vapoursynth is able to load avs scripts, but my memory is rusty regarding this and Google is failing me.

[Selur]

Only tried the script from MysterX only using Avisynth MT and not Vapoursynth.

[Sharc]

Originally Posted by Selur

Only tried the script from MysterX only using Avisynth MT and not Vapoursynth.

It's much better now with MysteryX latest version and SkipOver=20, but the fence is still a challenge

[Selur]

Thanks for the info, really busy in real life atm. but will check it out at the weekend.

[Selur]

argh, did a fast check with 'FrameRateConverter(SkipOver=20)' and the output is jumping between the next frame and a frame multiple frames in the past,...

[Sharc]

Hmmm...I observed the jerkiness as well, before I resized to 1280x720 or set output="flow", I think.
Here my script which I used for comparison:

clip=DGSource("C:\Temp\Selur\forInterpolation.dgi")
clip=clip.bilinearresize(1280,720)
conv1=clip.changefps(60000,1001).subtitle("ChangeFPS",x=20,y=20,size=32)
conv2=clip.framerateconverter(NewNum=60000,NewDen=1001,preset="normal",BlkSize=32,output="auto",SkipOver=25).subtitle("FrameRateConverter",x=20,y=20,size=32)
conv3=clip.interframe(cores=4, gpu=false, newnum=60000, newden=1001, tuning="film").subtitle("Interframe",x=20,y=20,size=32)

return stackhorizontal(conv2,conv3)

[Selur]

framerateconverter(NewNum=60000,NewDen=1001,preset="normal",BlkSize=32,output="auto",SkipOver=25)

is crap to, because it simply repeats each frame which could be done a lot faster

[Sharc]

No way.
conv1 duplicates, conv2 and conv3 interpolate.

Note that SkipOver=25 replaces the worst pictures only by duplicates. Set it to 0 in order to interpolate all frames.

[Selur]

Note that I use it on the full resolution version and:

# Frame Rate Converter
# Version: 26-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools
##
## @ NewNum      - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen      - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset      - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize     - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
## 
## @ BlkSizeV    - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output      - Output mode [auto|inter|none|mask|skip|over] (default=auto)
##                 auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; 
##                 mask=mask only; skip=mask used by SkipOver; over=mask as cyan overlay for debugging
##
## @ MaskGam     - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh     - The treshold where a block is considered bad, between 0 and 255.
##                 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc     - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver    - Skip interpolation of frames when artifacts cover more than specified treshold, 
##                 0 to disable. (Default=16)
##          
## @ Prefilter   - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##          
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV", 
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc", int "SkipOver", clip "Prefilter")
{
    Output     = Default(Output, "auto")
    FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
    NewNum     = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
    NewDen     = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
    Preset     = Default(Preset, "normal")
    BlkSize    = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
    BlkSizeV   = Default(BlkSizeV, BlkSize)
    blkmin     = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
    MaskGam    = Default(MaskGam, .5)
    MaskTrh    = Default(MaskTrh, 120)
    MaskOcc    = Default(MaskOcc, 150)
    MaskOcc    = MaskTrh > 0 ? MaskOcc : 0
    SkipOver   = Default(SkipOver, 16)
    CalcPrefilter = Defined(Prefilter) || Preset != "faster"
    Prefilter  = Default(Prefilter, Preset != "faster" ? C.RemoveGrain(22) : C)

    Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster", 
    \      "FrameRateConverter: Preset must be slow, normal, fast or faster")
    Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
    Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
    Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
    Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
    Assert(MaskOcc >= 0 && MaskOcc <= 255, "FrameRateConverter: MaskOcc must be between 0 and 255")
    Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

    # Performance settings: slow, normal, fast, faster
    Recalculate = preset == "slow" || preset == "normal"
    DCT = preset == "slow" ? 1 : 0

    ## "B" - Blending, "BHard" - No blending
    BHard = C.ChangeFPS(NewNum, NewDen)
    B  = C.ConvertFPS(NewNum, NewDen)
    B  = FrameDouble ? SelectOdd(B) : B

    ## jm_fps interpolation
    superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
    super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
    bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
    fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
    fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
    bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
    Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

    ## "EM" - error or artifact mask
    # Mask: SAD
    EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255) : BlankClip(C)
    EM = EM.ConvertToY8()
    # Mask: Temporal blending
    EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
    EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
    EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
    # Mask: Occlusion
    EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
    \                     .ConvertToY8().mt_inpand() : BlankClip(C)
    EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
    
    ## Mask processing
    EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    EMskip = EM.mt_binarize(100)
    EM = EM.mt_binarize(255-MaskTrh)
    \       .Blur(.6)
    \       .BicubicResize(C.Width, C.Height)
    EMskipOut = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")
    
    ## "Sc" - scene detection / SkipOver
    Sc = SkipOver > 0 ? ConditionalFilter(EMskip, BlankClip(EM, color=$FFFFFF), BlankClip(EM), 
    \    "AverageLuma", ">", string(SkipOver)) : BlankClip(EM)
    Sc = Sc.mt_binarize(128)

    # Display SkipOver value on Output="over
    Global GEMskip = EMskip.ChangeFPS(NewNum, NewDen)
    FlowOver = Flow.ScriptClip("Subtitle(string(GEMskip.AverageLuma()))")

    ## Convert masks to desired frame rate
    EM = EM.ChangeFPS(NewNum, NewDen)
    Sc = Sc.ChangeFPS(NewNum, NewDen)

    ## the FrameRateConverter magic happens
    M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
    M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

    R = (StrCmpi(Output, "auto")==0)        [** auto:  artifact masking     *] 
    \     ? (FrameDouble ? Interleave(C, M) : M)
    \ : (StrCmpi(Output, "flow")==0)        [** flow: interpolation only   *] 
    \     ? Flow
    \ : (StrCmpi(Output, "none")==0)        [** none: ConvertFPS only       *] 
    \     ? B
    \ : (StrCmpi(Output, "mask")==0)        [** mask:  mask only            *] 
    \     ? EM
    \ : (StrCmpi(Output, "skip")==0)        [** skip:  skip mask          *] 
    \     ? EMskipOut
    \ : (StrCmpi(Output, "over")==0)        [** over:  mask as cyan overlay *] 
    \     ? mt_merge(
    \         FlowOver.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
    \         BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
    \ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|over)")
    return R
}
LoadPlugin("G:\Hybrid\AVISYN~1\LSMASHSource.dll")
LoadPlugin("G:\Hybrid\AVISYN~1\masktools2.dll")
LoadPlugin("G:\Hybrid\AVISYN~1\mvtools2.dll")
LoadPlugin("G:\Hybrid\AVISYN~1\rgtools.dll")
# loading source: C:\Users\Selur\Desktop\forInterpolation.mp4
#  input luminance scale tv
LWLibavVideoSource("C:\Users\Selur\Desktop\forInterpolation.mp4",cache=false,stacked=true,format="YUV420P8",repeat=true)
# current resolution: 3840x2160
framerateconverter(NewNum=60000,NewDen=1001,preset="normal",BlkSize=32,output="auto")
return last

still leaves me with every (aside from second frame a duplicate aside from the first)
Will do some testing over the weekend,..

[Sharc]

I couldn't reproduce the frame jumping anymore, but apparently Stainless experienced it as well (and fixed it in his version).
For comparison I encoded your clip with x264 and streamed it to my 50" TV. My subjective ranking:
1. (best): ConvertFPS (blending)
2. FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
3. ChangeFPS (frame duplication). Probably the TV applied its own interpolation here, introducing its own artefacts. I didn't tweak the TV options, maybe later ...
4. (worst): Interframe/SVPflow (tune="film")

Edit:
I have to revise my rating. Unless one focuses on the fence but rather on the girl on the bike, the viewing impression is:
1. (winner): FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
2. ConvertFPS (blending)
3. Change FPS (with interpolation by the TV)
4. Interframe/SVPflow (the fence artefacts are dominant)

[MysteryX]

Originally Posted by Sharc

1. (winner): FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)

Now try with the new fallback option -- which I designed specifically for a fence-like problem I was having. It didn't actually help much there, but it's helping in other places.

[aegisofrime]

Originally Posted by Sharc

I couldn't reproduce the frame jumping anymore, but apparently Stainless experienced it as well (and fixed it in his version).
For comparison I encoded your clip with x264 and streamed it to my 50" TV. My subjective ranking:
1. (best): ConvertFPS (blending)
2. FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
3. ChangeFPS (frame duplication). Probably the TV applied its own interpolation here, introducing its own artefacts. I didn't tweak the TV options, maybe later ...
4. (worst): Interframe/SVPflow (tune="film")

Edit:
I have to revise my rating. Unless one focuses on the fence but rather on the girl on the bike, the viewing impression is:
1. (winner): FrameRateConverter (MysteryX/Stainless version, default settings except SkipOver=0)
2. ConvertFPS (blending)
3. Change FPS (with interpolation by the TV)
4. Interframe/SVPflow (the fence artefacts are dominant)

Did you manage to run it in Vapoursynth? How's the speed like?