SVP-like frame interpolation?

[manolito]

Originally Posted by MysteryX

Which part of the script requires Avisynth+? Unless modPlus.dll requires Avisynth+ but it should work with v2.6 too

You are right, contrary to what vcmohan himself says about his plugin it does work with v2.6...

for 2.5 not available for 2.6 Not Available for 32 & 64 bit avisynth+ modPlus

But compared to the old RemoveGrain command there is quite a speed sacrifice. I think I will stick with RemoveGrain...

Originally Posted by MysteryX

I decided last-minute to compile ConditionalMT with XP support -- glad it serves someone.

I have no use for AviSynth MT so I took out ConditionalReaderMT and replaced it with the standard ConditionalReader command.

Originally Posted by MysteryX

Which of these lines fail? BicubicResize and Blur are very unlikely. mt_expand or mt_binarize are most likely the ones crashing .. but I wonder why. Neither take blksize as parameter.

It is this line:

EM = EM.BicubicResize(C.Width, C.Height)

For SD sources and BlkSize 32 the resized mask is very small, maybe there is a problem within AviSynth to resize such a small mask back to the original size?? No idea, but my workaround sure fixes it.


Cheers
manolito

[StainlessS]

W=4
H=4
Colorbars(Width=W,Height=H,Pixel_Type="YV12").KillAudio
BicubicResize(1920,1280)

Error message (perhaps referring to chroma size) [have seen this error before, was confused by it].

Resize: source image too small for this resize method. Width=2, Support=2.

EDIT: W=6 H=6 works ok.

EDIT: YV12, I think that error can only happen if source dimensions (as per Manolito) less than 192.
(192 / 32 / 4.0) = 1.5, * 4 = 6, so less than 192 is error [Round(191.9999 / 32 / 4.0) rounds down to 1.0, so 1 * 4 = 4 = error] .

EDIT: If YV24, then same resize error if W=2 H=2 [W=3 H=3 OK], so for YV24 error if less than 192 / 2 = 96.

EDIT: YV12 BilinearResize with source W=4 H=4 OK, Bilinear needs 2 (chroma) samples minimum, BicubicResize needs 3 (chroma) samples.

EDIT: Fixed some cockups.

EDIT: For YV411, (not suggesting that anyone should use that), then minimum width with that script would be 384 else error (assuming that it would work at all with YV411).

[MysteryX]

Median indeed is slower than RemoveGrain -- a LOT.

RemoveGrain: 636fps @ 24% CPU
Median: 30fps @ 82% CPU

I edited the script to use Median with presets "slower|slow", RemoveGrain with presets "normal|fast", and none with preset "faster". It is done with Eval so that you don't need a reference to a library you're not using.

Output="over" image brightening issue is fixed, as well as another similar mask issue.

I fixed Avisynth+'s conditional filters code so the standard ConditionalFilter will work with MT mode in the next release of AVS+; thus I removed ConditionalFilterMT.

This code is tweaked for YV24 sources; use that for testing.

TODO:
- Mask luma is different between YV12 and YV24; it needs to be normalized (Pinterf)
- SkipOver only works in 8-bit, as the 16-bit value of AverageLuma is 220x higher and needs to be normalized (Pinterf)
- ChangeFps on the mask somehow causes the CPU to stall at 20% (Pinterf)
- Release the next version of AVS+ with conditional MT (Pinterf)

After some testing, I'm not seeing any difference at all between prefilters. I might just go back to RemoveGrain(22); even RemoveGrain(21) is still a *LOT* faster than Median. I'm open to suggestions.

# Frame Rate Converter
# Version: 24-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2
## Prefilter: modPlus for presets "slow|slower", otherwise rgtools
##
## @ NewNum      - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen      - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Preset      - The speed/quality preset [slower|slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize     - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
## 
## @ BlkSizeV    - The vertical block size (default = BlkSize)
##
## @ Output      - Output mode [auto|inter|none|mask|over] (default=auto)
##                 auto=normal artifact masking; flow=interpolation only
##                 none=ConvertFPS only; mask=mask only; over=mask as cyan overlay for debugging
##
## @ MaskStr     - The artifact masking strength (1 to 100). This will apply a gamma curve on the mask 
##                 before processing, where 100 applies no gamma curve and 5 applies 0.05 gamma (Default=5)
##
## @ MaskSAD     - Artifact masking strength for bad motion, 0 to disable (Default=190)
##
## @ thSCD1      - MSCDetection scene detection treshold 1 (Default=500)
##
## @ thSCD2      - MSCDetection scene detection treshold 2 (Default=150)
##
## @ SkipOver    - Skip interpolation of frames when artifacts cover more than specified treshold, 
##                 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##          
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble", string "Preset",
\ int "BlkSize", int "BlkSizeV", string "Output", float "MaskStr", int "MaskSAD", int "thSCD1", int "thSCD2", int "SkipOver")
{
    Output     = Default(Output, "auto")
    FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
    NewNum     = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
    NewDen     = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
    Preset     = Default(Preset, "normal")
    BlkSize    = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
    BlkSizeV   = Default(BlkSizeV, BlkSize)
    blkmin     = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
    MaskStr    = Default(MaskStr, 5)
    MaskSAD    = Default(MaskSAD, 190)
    #MaskOcc    = Default(MaskOcc, 64)
    thSCD1     = Default(thSCD1, 500)
    thSCD2     = Default(thSCD2, 150)
    SkipOver   = Default(SkipOver, 48)

    Assert(Preset == "slower" || Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster", 
    \      "FrameRateConverter: Preset must be slower, slow, normal, fast or faster")
    Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
    Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
    Assert(MaskStr > 0 && MaskStr <= 100, "FrameRateConverter: MaskStr must be between 1 and 100")
    Assert(SkipOver >= 0 && MaskStr <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

    # Performance settings: slower, slow, normal, fast, faster
    Recalculate = preset == "slow" || preset == "normal"
    prefilter = (preset == "slower" || preset == "slow") ? "Median(C, uu = true, vv = true)" : 
    \           (preset == "normal" || preset == "fast") ? "C.RemoveGrain(21)" : "C"
    DCT = preset == "slower" ? 1 : 0

    ## "B" - Blending, "BHard" - No blending
    BHard = C.ChangeFPS(NewNum, NewDen)
    B  = C.ConvertFPS(NewNum, NewDen)
    B  = FrameDouble ? SelectOdd(B) : B

    ## jm_fps interpolation
    prefiltered = C.RemoveGrain(21) #Eval(prefilter)
    superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
    super = Prefilter != "C" ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
    bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
    fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
    fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
    bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
    Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

    ## "EM" - error or artifact mask
    EM = MaskSAD > 0 ? C.MMask(bak, ml=MaskSAD, kind=1) : BlankClip(C)   # kind=SAD
    EMfwd = MaskSAD > 0 ? C.MMask(fwd, ml=MaskSAD, kind=1) : EM           # kind=temporal blending
    EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
    #EMocc = MaskOcc > 0 ? C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand() : BlankClip(C)  # kind=occlusion
    EM = MaskSAD > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
    #EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=0.5, mode="lighten", pc_range=true) : EM
    SkipEM = EM

    ## mask strength
    EM = EM.Levels(0, MaskStr / 100.0, 255, 0, 255, coring=false)
    
    EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    \       .mt_binarize(92)
    \       .Blur(.6)
    \       .BicubicResize(C.Width, C.Height)
    
    ## "Sc" - scene detection
    Sc = thSCD1 > 0 && thSCD2 > 0 ? C.MSCDetection(bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(EM)
    Sc = SkipOver > 0 ? ConditionalFilter(SkipEM, BlankClip(EM, color=color_white), Sc, "AverageLuma()", ">", string(SkipOver)) : Sc

    ## Convert masks to desired frame rate
    EM = EM.ChangeFPS(NewNum, NewDen)
    Sc = Sc.ChangeFPS(NewNum, NewDen)

    ## the FrameRateConverter magic happens
    M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
    M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

    R = (StrCmpi(Output, "auto")==0)        [** auto:  artifact masking     *] 
    \     ? (FrameDouble ? Interleave(C, M) : M)
    \ : (StrCmpi(Output, "flow")==0)        [** flow: interpolation only   *] 
    \     ? Flow
    \ : (StrCmpi(Output, "none")==0)        [** none: ConvertFPS only       *] 
    \     ? B
    \ : (StrCmpi(Output, "mask")==0)        [** mask:  mask only            *] 
    \     ? EM
    \ : (StrCmpi(Output, "over")==0)        [** over:  mask as cyan overlay *] 
    \     ? mt_merge(
    \         Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
    \         BlankClip(EM, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true)
    \ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|over)")
    return R
}

[burfadel]

I didn't realise median was so slow, but it makes sense. It's like mode 4 of Removegrain but more advanced. It's probably overkill. Removegrain mode 22/21 is very fast by nature, but probably a little weak and inprecise for best results, which is why median had nicer output.

Maybe you could try something like:

    prefiltered = C.minvar(lx=120, ty=100) #Eval(prefilter)

Minvar is part of the same package as Median (modplus).

Small values for lx and ty as larger values would be slower, it's for the computing of global variance. I just chose 45 and 35 as an example, these can be higher or lower (wouldn't go too low though).

EDIT: I originally had the lx and ty a little smaller, but chose higher numbers because it is very fast.

[manolito]

OK, since the ToDo list only contains stuff for AVS+ (pinterf), I decided that this script is static for my needs.

I did the same to this MysteryX script which I did a while ago to the original JohnMeyer script. I modified the interface so now it is a regular fps converter which can be used like ConvertFPS or ChangeFPS. The only exposed params are FPS, Preset and BlkSize. (Update: Two additional exposed parametes "Dct" and "Mask")

I call it "mx_fps" just like the previous "jm_fps" (which seems to stick...). In my tests it works beautifully with all kinds of sources. Only slightly slower than jm_fps, but better results with many source clips, and so far never worse than jm_fps. Looks like a winner to me...

mx_fps.avsi

# Frame Rate Converter
# Version: 2-June-2019
# By Etienne Charland aka MysteryX
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Pinterf is the one who spent the most time working on the core libraries, adding features and fixing bugs
# Slightly simplified user interface and code cleanup by manolito
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### mx_fps
### Changes the frame rate with interpolation and fine artifact removal.
##
## YV12/YUY2
## Requires: FrameRateConverter.dll, MaskTools2, MvTools2, GRunT, RemoveGrain, FFTW3.dll (in the system32 or syswow64 folder) for Dct values other than 0
##
## @ fps         - The new framerate.
##
## @ Preset      - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize     - The block size. Supported values are 8, 16 and 32.
##                 Defaults for 4/3 video of height:
##                 0-359:  8
##                 400-1199: 16
##                 1200-2160: 32
## 
## @ BlkSizeV    - The vertical block size. (default = BlkSize)
## 
## @ Output      - Output mode [auto|flow] (default = auto)
##                 auto=normal artifact masking; flow=interpolation only
##
## @ Prefilter   - Specified a custom prefiltered clip. (default = C.RemoveGrain(22))
##
## @ Mask        - Enable artifact masking (default = true)
##
## @ MaskTrh     - The treshold where a block is considered bad, between 0 and 255. Smaller = stronger.
##                 0 to disable artifact masking. (default = 100)
##
## @ MaskOcc     - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (default = 105)
##
## @ SkipTrh     - The treshold where a block is counted for the skip mask, between 0 and 255. Smaller = stronger.
##                 Must be smaller (stronger) than MaskTrh. (default = 55)
##
## @ BlendOver   - Try fallback block size when artifacts cover more than specified treshold, or 0 to disable.
##                 If it fails again, it will revert to frame blending. (default = 60)
##
## @ SkipOver    - Skip interpolation of frames when artifacts cover more than specified treshold, 
##                 or 0 to disable. (default = 120)
##
## @ Stripes     - How to deal with stripes [none|skip|blend] (default=blend)
##
## @ Dct         - Overrides DCT parameter (default: Fast=0, Normal=0, Slow=1)
##                 Useful values are 0, 4 and 1.
##
## @ BlendRatio  - Changes the blend ratio used to fill artifact zones. 0 = frame copy and 100 = full blend.
##                 Other values provide a result in-between to eliminate ghost effects. Default = 40.
##
## Presets
## Fast:    Basic interpolation
## Normal:  Fast + prefilter + MSuper on prefilter + MRecalculate
## Slow:    Normal + DCT=1
##

function mx_fps(clip C, float "fps", string "Preset", int "BlkSize", int "Dct", bool "Mask")
{
    Preset = Default(Preset, "normal")
    P_SLOW = 1  P_NORMAL = 2  P_FAST = 3
    Pset = Preset == "slow" ? P_SLOW : Preset == "normal" ? P_NORMAL : Preset == "fast" ? P_FAST : -1
    Assert(Pset != -1, "mx_fps: 'Preset' must be slow, normal or fast {'" + Preset + "'}")
    Mask   = Default(Mask, true)
    Output = Mask ? "auto" : "flow"
    O_AUTO = 0  O_FLOW = 1
    OPut = Output == "auto" ? O_AUTO : Output == "flow" ? O_FLOW : -1
    Stripes = "blend"
    S_NONE = 0  S_SKIP = 1  S_BLEND = 2
    Stp = Stripes == "none" ? S_NONE : Stripes == "skip" ? S_SKIP : Stripes == "blend" ? S_BLEND : -1
    
    fps         = default(fps, 25.000)
    NewNum      = int(fps * 1000)
    NewDen      = 1000
    DefH        = Max(C.Height, C.Width/4*3)
    BlkSize     = Default(BlkSize, DefH<360 ? 8 : DefH<1200 ? 16 : 32)
    Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "mx_fps: BlkSize must be 8, 16 or 32")
    BlkSizeV    = BlkSize
    MaskTrh     = 100
    SkipTrh     = 55
    MaskOcc     = MaskTrh > 0 ? 105 : 0
    BlendOver   = 60
    SkipOver    = 120
    CalcPrefilter = Pset != P_FAST
    Prefilter   = CalcPrefilter ? C.RemoveGrain(22) : C
    Recalculate = PSET <= P_NORMAL
    Dct         = Default(Dct, PSET == P_SLOW ? 1 : 0)
    BlendRatio  = 40
    
    ## "B" - Blending, "BHard" - No blending
    Try {
    B = C.ConvertFpsLimit(NewNum, NewDen, ratio=BlendRatio)
    }
    Catch(Err_Msg) {
             B = C.ChangeFps(floor(fps * 3/2)).ConvertFpsLimit(NewNum, NewDen, ratio=BlendRatio)
        }
    BHard = C.ChangeFps(NewNum, NewDen)
    Blank = BlankClip(C.ConvertToY8(), color_yuv=$000000)

    ## Adjust parameters for different block sizes, causing stronger or weaker masks
    blk = Max(BlkSize, BlkSizeV)
    MaskTrh = MaskTrh + (blk<=8 ? -20 : blk<=16 ? 0 : blk<=32 ? 20 : 35)
    SkipTrh = SkipTrh + (blk<=8 ? -18 : blk<=16 ? 0 : blk<=32 ? 16 : 30)
    MaskTrh = Max(Min(MaskTrh, 255), 0)
    SkipTrh = Max(Min(SkipTrh, 255), 0)
    gam = blk<=8 ? .56 : blk<=16 ? .50 : blk<=32 ? .36 : .14

    ## jm_fps interpolation
    superfilt = MSuper(prefilter, hpad=16, vpad=16, sharp=1, rfilter=4) # all levels for MAnalyse
    super = CalcPrefilter ? MSuper(C, hpad=16, vpad=16, levels=1, sharp=1, rfilter=4) : superfilt # one level is enough for MRecalculate
    bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizev=BlkSizeV, overlap = (BlkSize/4+1)/2*2, overlapv = (BlkSizeV/4+1)/2*2, search=3, dct=Dct)
    fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizev=BlkSizeV, overlap = (BlkSize/4+1)/2*2, search=3, dct=Dct)
    fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizev=BlkSizeV/2, overlap = BlkSize/2>4?(BlkSize/8+1)/2*2:0, overlapv = BlkSizeV/2>4?(BlkSizeV/8+1)/2*2:0, thSAD=100) : fwd
    bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizev=BlkSizeV/2, overlap = BlkSize/2>4?(BlkSize/8+1)/2*2:0, overlapv = BlkSizeV/2>4?(BlkSizeV/8+1)/2*2:0, thSAD=100) : bak
    Flow = MFlowFps(C, super, bak, fwd, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255)

    ## "EM" - error or artifact mask
    # Mask: SAD
    EM = MaskTrh > 0 ? C.ConvertToY8().MMask(bak, ml=255, kind=1, gamma=1/gam, ysc=255, thSCD2=255) : Blank
    # Mask: Temporal blending
    EMfwd = MaskTrh > 0 ? C.ConvertToY8().MMask(fwd, ml=255, kind=1, gamma=1/gam, thSCD2=255) : EM
    EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true) : EM
    # Mask: Occlusion
    EMocc = MaskOcc > 0 ? C.ConvertToY8().MMask(bak, ml=MaskOcc, kind=2, gamma=1/gam, ysc=255, thSCD2=255).mt_inpand() : Blank
    EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM

    # Last mask frame is white. Replace with previous frame.
    EM = EM.DeleteFrame(EM.Framecount-1).Loop(2, EM.Framecount-1)

    # Create skip mask
    EMskip = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    \       .mt_binarize(SkipTrh)
    
    ## Create artifact correction mask
    Try {
    EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    \       .mt_binarize(MaskTrh)
    \       .Blur(.6)
    \       .BicubicResize(C.Width, C.Height)
    }
    Catch(Err_Msg) {
             Try {
             BlkSize = BlkSize/2
             EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
             \       .mt_expand(mode= mt_circle(zero=true, radius=1))
             \       .mt_binarize(MaskTrh)
             \       .Blur(.6)
             \       .BicubicResize(C.Width, C.Height)
         }
         Catch(Err_Msg) {
                  BlkSize = BlkSize/2
                  EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
                  \       .mt_expand(mode= mt_circle(zero=true, radius=1))
                  \       .mt_binarize(MaskTrh)
                  \       .Blur(.6)
                  \       .BicubicResize(C.Width, C.Height)
             }
    }

    # Mask: Stripes
    EMstp = C.StripeMask(blksize=BlkSize, blksizev=BlkSizeV, str=min(SkipTrh*2+20, 255), strf=min(SkipTrh+10, 255), thr=23)
    \      .BicubicResize(Round(C.Width/BlkSize)*4, Round(C.Height/BlkSizeV)*4)
    \      .ContinuousMask(22)
    EMstp = EMstp.BicubicResize(EMstp.Width/2, EMstp.Height/2)
    \      .mt_binarize(82)
    \      .mt_inpand()
    \      .mt_expand(mode= mt_circle(zero=true, radius= Stp==S_SKIP ? 12 : 8))
    \      .FRC_GaussianBlur42(Stp==S_SKIP ? 8.0 : 2.8)
    \      .BicubicResize(C.Width, C.Height)
        
    ## "M" - Apply artifact removal
    EM = EM.ChangeFPS(NewNum, NewDen)
    EMskip = EMskip.ChangeFPS(NewNum, NewDen)
    EMstp = EMstp.ChangeFPS(NewNum, NewDen)
    M = mt_merge(Flow, B, EM, luma=true, chroma="process")
    M = Stp != S_NONE ? mt_merge(M, Stp == S_SKIP ? BHard : B, EMstp, luma=true, chroma="process") : M

    ## Apply BlendOver and SkipOver
    M2 = SkipOver > 0 ? ConditionalFilterMT(EMskip, B, BHard, "AverageLuma", "<", string(SkipOver)) : B
    M = BlendOver > 0 ? ConditionalFilterMT(EMskip, M, M2, "AverageLuma", "<", string(BlendOver)) : M

    # Output modes
    R= (Oput==O_AUTO)                               [** auto: artifact masking     *]
    \     ? M
    \ : (Oput==O_FLOW)                              [** flow: interpolation only   *]
    \     ? Flow
    \ : nop

    return R
}


##################################
### Emulate [[VariableBlur/GaussianBlur]] 
##  For YUV, effective chroma blur varies depending on source 
##  color subsampling - YUV444 has *more* chroma blur, others less.
##
## @ var - works like GaussianBlur's varY
## @ rad - blur radius (<var> squared); overrides <var>
## @ vvar, vrad - vertical var & rad; default same as horizontal
## @ p  - final [[GaussResize]] sharpness. Default 19
##        (if > 25, blockiness; if < 15, loss of contrast)
##
## version 2013-10-23 raffriff42 
## version 2014-05-31 discrete hor. and vert. args
## version 2017-05-21 bugfix: blockiness
##
function FRC_GaussianBlur42(clip C, 
\               float "var", float "rad",
\               float "vvar", float "vrad", float "p")
{
    var = Max(0.0, Float(Default(var, 1.0)))
    rad = Max(1.0, Float(Default(rad, Pow(var, 0.5))))
    var = Pow(Min(Max(0.0, rad), 60.0), 1.9) ## arbitrary max radius = 60

    vvar = Max(0.0, Float(Default(vvar, var)))
    vrad = Max(1.0, Float(Default(vrad, Pow(vvar, 0.5))))
    vvar = Pow(Min(Max(0.0, vrad), 60.0), 1.9)
    p    = Default(p, 19)

    w0 = C.Width
    h0 = C.Height
    w1 = Round(w0/rad)
    h1 = Round(h0/vrad) 

    B = C.BilinearResize(
    \         Min(Max(4, w1 + (w1 % 2)), w0), 
    \         Min(Max(4, h1 + (h1 % 2)), h0))

    B = B.Blur(1.0).Blur(1.0)
    
    return (var<0.01 && vvar<0.01)   ? C 
    \    : (B.Width>8 && B.Height>8) ? B.GaussResize(w0, h0, p=p)
    \                                : B.BilinearResize(w0, h0)
}

Cheers
manolito


//EDIT//
Here is a complete AIO package with all the required plugins:
https://files.videohelp.com/u/172211/mx_fps%20AIO.zip
Please note that I used older (but stable) plugin versions which also run on old hardware.
This AIO package also contains a version of the script which does not require a SSE2 capable CPU (had to remove the stripe mask feature for this).

//EDIT 2//
Changed the user interface to expose the DCT value which will override the default. Useful if you want to use DCT=4. Also added a "Mask" parameter which makes it easy to enable or disable artifact masking.
NOTE: This script version needs the latest version 1.3 of "FrameRateConverter.dll". It comes with the AIO package.

//EDIT3//
Added the "ConvertFpsLimit" function from the latest FrameRateConverter 1.3

[StainlessS]

Manolito,

Requires: MaskTools2, MvTools2, RemoveGrain and fftw3.dll (in the System32 or SysWOW64 folder)

What is it that needs fftw3.dll, I see nothing that sticks out, and does not seem to be required in earlier mods ?

[manolito]

It is required for DCT=1 in mvtools2 (at least in Fizick's latest version 2.5.11.22). In this script Preset = "Slow" will use "DCT=1", and without fftw3.dll you will get a crash...

Cheers
manolito

[StainlessS]

Cheers Mani, had no idea that it was required, not noted in docs.
There is mention that it uses fftw3 header, and several mentions of fft regards to DCT but no specific requirement mentioned anywhere.
Deleted fftw3.dll from system32 and still worked ok, then remembered something about something also being able to use lib named as
libfftw3f-3.dll, so deleted that too, and crashes, just as you said.

God, its a bit slow with that setting, dont think I'm ever likely to use it.

[manolito]

Originally Posted by StainlessS

God, its a bit slow with that setting, dont think I'm ever likely to use it.

Yeah right, but for some sources it does make a real difference...

Cheers
manolito

[MysteryX]

A caveat to be careful about: YV12 and YV24 will have different artifact mask strengths. YV12 will have much weaker mask.

Also, for mask strength, besides adjusting Gamma, the other line that will make a difference is "binarize(92)".

You guys can play around with Gamma and Binarize, in YV12 or YV24, and see what works best for you.

[StainlessS]

Originally Posted by MysteryX

A caveat to be careful about: YV12 and YV24 will have different artifact mask strengths. YV12 will have much weaker mask.

If chroma masks cannot be normalized somehow, then perhaps chroma introduces too big a random variation to be useful, as cannot be separately processed/controlled/combined. (or everything converted to YV24, or everything to YV12).

[johnmeyer]

We had a discussion about DCT=1 making a difference, and I did some tests and couldn't see a difference. As I remember, the evidence posted in that thread was not convincing (that DCT=1 helped reduce ME artifacts). The only reason I bring this up is that I'd hate to see lots of effort chasing issues related to DCT=1, especially since, in addition to maybe not making much difference, it is so doggedly slow. Instead, I'm hoping most of the focus will be on enhancing the effort to produce a version that can detect and mask artifacts. I've been waiting for the dust to settle before I jump in and start testing this.

[burfadel]

Would there be any advantage running the script natively as YV24 (that is, using converttoyv24() if not already in YV24)?

Have you looked into minvar function? It should provide the benefit of the median filter without the speed penalty.

[MysteryX]

Any benefit to processing in YV24 will be lost just as soon as you re-convert to YV12, but if you do further processing in YV24, you can see better chroma precision before discarding the data.

But this script so far was tweaked for YV24. We may want to compare YV12, YV24 and Luma-only masks. Who knows -- maybe a Luma-only mask would be more accurate by discarding chroma noise -- only one way to know.

Edit: Mask cannot be applied to only Luma, because it is based on the vectors analysis which takes all planes for motion interpolation.

Right now, YV12 produces stronger mask than YV24.

YV24 / YV12


Did another test using MMask's Gamma. This time YV12 produces weaker mask. YV24 masks just look better in general.

[MysteryX]

I've done plenty of changes to the artifact removal by tweaking MMask's arguments.

MMask's ml is unecessary because it simply sets to 255 values above that treshold -- which is exactly what mt_binarize is for; thus I set that to 255 and removed MaskSAD parameter.

MMask has a gamma property, which removes the need for another call to Levels. The mask level adjustment function has been removed.

MMask also has ysc which says what to do on scene changes: set everything to 255. Thus, I no longer need MSCDetection.

Mask adjustment settings are now MaskGam (gamma, applied during MMask) and MaskTrh (treshold, applied during mt_binarize).

MaskGam affects SkipOver -- and I believe I've found the right gamma value. If we find no need to change it, I might remove that property.

Because SkipOver now operate on a gamma-adjusted mask, it alters SkipOver behaviors.

After plenty of testing, I've set default to MaskGam=.6, MaskTrh=90, SkipOver=15

I removed Median and "slower" preset, went back to RemoveGrain(22) until someone can prove benefits of doing more than that.

To please manolito, I changed the arguments order for simplicity. First parameters are now: NewNum, NewDen, Preset, BlkSize

Again, settings are tweaked for YV24, and YV12 produces a different mask strength so settings will need to be adjusted.

Because now I know what I'm doing (instead of randomly using old code), the mask now both covers artifacts more precisely and leaves out more areas. I found it to be better in almost every cases.

# Frame Rate Converter
# Version: 25-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools
##
## @ NewNum      - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen      - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset      - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize     - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
## 
## @ BlkSizeV    - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output      - Output mode [auto|inter|none|mask|over] (default=auto)
##                 auto=normal artifact masking; flow=interpolation only
##                 none=ConvertFPS only; mask=mask only; over=mask as cyan overlay for debugging
##
## @ MaskGam     - A gamma to be applied to the raw mask, between 0 and 1. 
##                 Higher value means stronger mask. (Default=.6)
##
## @ MaskTrh     - The treshold where a block is considered bad, between 0 and 255. 0 to disable artifact masking.
##                 Higher value means stronger mask. (Default=90)
##
## @ SkipOver    - Skip interpolation of frames when artifacts cover more than specified treshold, 
##                 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##          
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV", 
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "SkipOver")
{
    Output     = Default(Output, "auto")
    FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
    NewNum     = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
    NewDen     = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
    Preset     = Default(Preset, "normal")
    BlkSize    = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
    BlkSizeV   = Default(BlkSizeV, BlkSize)
    blkmin     = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
    MaskGam    = Default(MaskGam, .6)
    MaskTrh    = Default(MaskTrh, 90)
    SkipOver   = Default(SkipOver, 15)

    Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster", 
    \      "FrameRateConverter: Preset must be slow, normal, fast or faster")
    Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
    Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
    Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
    Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
    Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

    # Performance settings: slow, normal, fast, faster
    Recalculate = preset == "normal"
    prefilter = preset != "faster"
    DCT = preset == "slower" ? 1 : 0

    ## "B" - Blending, "BHard" - No blending
    BHard = C.ChangeFPS(NewNum, NewDen)
    B  = C.ConvertFPS(NewNum, NewDen)
    B  = FrameDouble ? SelectOdd(B) : B

    ## jm_fps interpolation
    prefiltered = Prefilter ? C.RemoveGrain(22) : C
    superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
    super = Prefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
    bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
    fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
    fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
    bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
    Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

    ## "EM" - error or artifact mask
    EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, ysc=255, gamma=1.0/MaskGam) : BlankClip(C)   # kind=SAD
    EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam) : EM           # kind=temporal blending
    EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
    EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
    SkipEM = EM
    
    EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    \       .mt_binarize(255-MaskTrh)
    \       .Blur(.6)
    \       .BicubicResize(C.Width, C.Height)
    
    ## "Sc" - scene detection / SkipOver
    Sc = SkipOver > 0 ? ConditionalFilter(SkipEM, BlankClip(EM, color=color_white), BlankClip(EM), 
    \                    "AverageLuma()", ">", string(SkipOver)) : BlankClip(EM)

    ## Convert masks to desired frame rate
    EM = EM.ChangeFPS(NewNum, NewDen)
    Sc = Sc.ChangeFPS(NewNum, NewDen)

    ## the FrameRateConverter magic happens
    M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
    M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

    R = (StrCmpi(Output, "auto")==0)        [** auto:  artifact masking     *] 
    \     ? (FrameDouble ? Interleave(C, M) : M)
    \ : (StrCmpi(Output, "flow")==0)        [** flow: interpolation only   *] 
    \     ? Flow
    \ : (StrCmpi(Output, "none")==0)        [** none: ConvertFPS only       *] 
    \     ? B
    \ : (StrCmpi(Output, "mask")==0)        [** mask:  mask only            *] 
    \     ? EM
    \ : (StrCmpi(Output, "over")==0)        [** over:  mask as cyan overlay *] 
    \     ? mt_merge(
    \         Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
    \         BlankClip(EM, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true)
    \ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|over)")
    return R
}

[MysteryX]

I was wondering what it would change if I scaled the mask down to 4 pixels per block instead of 1 pixel per block.

Regular

4 pixels per block


I'll definitely leave it as it is. This change is not bringing good results: too much false positives and chip-chopping between clips. I'm surprised at the difference though.

[manolito]

Thanks MX for the improvements...

I think there are a few small oversights in the script, could you please check?

Line 54: Wrong default value for SkipOver
Line 80: Recalculate really only for preset "normal", not for "slow"?
Line 82: DCT = 1 for the no longer existing preset "slower" instead of "slow"?


Cheers
manolito

[burfadel]

Did you try:

    prefiltered = Prefilter ? C.minvar(lx=120, ty=100) : C

Probably the only adjustment would be xgrid (which automatically sets ygrid if not specified). Default is 5, your can go lower number for weaker, or higher for stronger effect. The more noise that there is, the greater the noise reduction, so I think it's ideal for the prefilter (especially considering its very fast). You can choose higher or lower values for lx and ty (these must be specified). I think they're reasonable values.

It should possibly help to reduce some artifacting as well, if the artifacting is caused by noise as was the case with the improvement with median. This filter is very much faster though, probably faster than Removegrain (or so it seems).

[raffriff42]

Noise is not usually the major issue (for me) -- all the things you see in JM's parade clip are. Motion-detection-busting occlusions, or "antlers" for short

Most of the time a simple Blur gives plenty of prefiltering. Why not a prefilter clip argument, as used by SMDegrain("prefilter") , Interp2("denoised") and probably others, to allow users with noise problems to supply their own denoising? There are many filters available to handle the many types of noise -- high frequency? low frequency? gaussian? mosquito? RF? jitter? flicker? streaking? herringbone?

[burfadel]

Blur blur's edges though, you want to maintain the picture detail whilst removing noise, obviously with minimal speed impact.