SVP-like frame interpolation?

[MysteryX]

Originally Posted by manolito

The original jm_fps script works fine using blksize=32. The libraries I use may be outdated, but so is my computer, and obviously so am I.

I get you. Still, you must find which component fails, and only you can do that as everything works here. You have to identify the line of code that fails. Then, we can explore the options from there.

Originally Posted by burfadel

I have noticed the converter runs fairly slow as well compared to you earlier versions. Much less CPU use as well, I think it's hitting a major bottleneck.

Again, we have to identify which component creates a bottleneck...

Originally Posted by burfadel

Why the divide by 4 then straight away multiplying by 4?

The first is to reduce the mask so that we have 1 pixel per block. Meaning, each block is either left or replaced. The second resize is to scale it back to the clip size. I'm finally starting to understand what it's doing. That line looks good.

Originally Posted by burfadel

What does the purpose of MergeRGB(Blankclip(EM), EM, EM) do for a normal avisynth script?

This is only for output=over, to add a cyan mask overlay. The mask gets added to the G and B channels. However, there is a bug that causes the image to get brighter (with output=over) which I still haven't identified.

Originally Posted by burfadel

Also, wouldn't converting the FPS in the middle of an overlay command be a bad thing, in that it would be overlaying the wrong frame data?

Masks have the source frame rate until they are converted to match the destination frame rate. If you don't place ChangeFps at the right places, the mask won't match the clip it's being applied to.

[burfadel]

Ah ok! You could probably get rid of the blur function though if you play around with the b and c parameters of bicubicresize, which would have practically the same effect. Saves one function. This is for the enlargment.

For the downsize, couldn't bilinear be used instead seeing as it's faster and output extremely similarly? This is particularly true when going to such a small resolution.

[MysteryX]

Warning: the output is currently very different between YV12 and YV24!! See here. This completely changes the artifact detection behaviors.

I've mostly been doing my tests in YV24, so the settings are tweaked for YV24.

Burfadel, I don't think changing these functions will make any difference at all, they are very lightweight compared to the rest of the code.

[MysteryX]

Good news: 16-bit now works with Pinterf's latest MvTools and MaskTools.

Performance-wise:
- YV24 8-bit: 13.22fps @ 27% CPU
- YV24 16-bit: 11fps @ 27% CPU

Not a big performance cost. Quality-wise, it removes some interpolation distortion in some areas. I also saw a few instances where it instead added slight distortion.

[MysteryX]

Here's a weird one.

ColorBarsHD()

C=ColorBarsHD()
C.Overlay(BlankClip(C), mode="Add", opacity=0.40)

There is unwanted brightening. If I were to take a guess, it might be because black is Y=36, and it applies a brightness of 36/255. What would be the solution?

Edit: Adding pc_range=true to Overlay fixes it.

[MysteryX]

The performance bottleneck is... roll-drum...

EM = EM.ChangeFPS(NewNum, NewDen)

!??

CPU drops from 80% to 20% if I place a "return EM" after this line than before

As for 16-bit support, SkipOver doesn't yet work as expected. It needs to be disabled for testing, or set to a value 220x higher in 16-bit.

YV12 doesn't work as expected, 16-bit doesn't work as expected, and CPU drops to 20%, but other than that it's perfect

[burfadel]

It must have driven you crazy finding that, it's basically the least likely place you would see a performance bottleneck.

Have you tried renaming EM to something else, like EM2?

EM2 = EM.ChangeFPS(NewNum, NewDen)
Return EM2

Maybe using a frame rate change and EM=EM.ChangeFPS is causing it to loop.

[manolito]

Originally Posted by MysteryX

I get you. Still, you must find which component fails, and only you can do that as everything works here. You have to identify the line of code that fails. Then, we can explore the options from there.

The code fails here:

EM  = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    \       .mt_binarize(92)
    \       .Blur(.6)
    \       .BicubicResize(C.Width, C.Height)

But only for an SD source (720 x 480) when I force a Block Size of 32. HD sources do not cause problems.

I made a workaround reducing the BlkSize variable to 16 for SD sources. I inserted this code just before the above section so the interpolator is not affected:

BlkSize = C.Width>1200||C.Height>900 ? BlkSize : BlkSize == 32 ? 16 : BlkSize
BlkSizeV = C.Width>1200||C.Height>900 ? BlkSizeV : BlkSizeV == 32 ? 16 : BlkSizeV

This produces very nice results so far.


Otherwise I have to tell you guys that I am outta here. The last incarnation of the script requires AviSynth+, and this pushes it over the edge for me. You guys do live on a different planet. The only guy who is even further out is stax76 who seriously wanted to raise the minimum requirement for StaxRip to Win 10 with the recent Creator's update (I think he revoked it for the time being).


Whatever, have fun...

Cheers
manolito

[burfadel]

Originally Posted by MysteryX

The performance bottleneck is... roll-drum...

EM = EM.ChangeFPS(NewNum, NewDen)

!??

Shouldn't it be:

EM = Changefps(EM, NewNum, NewDen)

Does that make any difference? I notice that in several places in the script different functions are written as (example) EM.ChangeFPS instead of ChangeFPS(EM,), or ChangeFPS(clip = EM,)

I wonder if adjusting all of these (using the clip function of the command rather than having it piped? from outside via EM.) would make any different to the performance?

This last updated code with the difference syntax applied:

# Frame Rate Converter
# Version: 23-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, modPlus, ConditionalMT
##
## @ NewNum      - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen      - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Preset      - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize     - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
## 
## @ BlkSizeV    - The vertical block size (default = BlkSize)
##
## @ Output      - Output mode [auto|inter|none|mask|over] (default=auto)
##                 auto=normal artifact masking; flow=interpolation only
##                 none=ConvertFPS only; mask=mask only; over=mask as cyan overlay for debugging
##
## @ MaskStr     - The artifact masking strength (1 to 100). This will apply a gamma curve on the mask 
##                 before processing, where 100 applies no gamma curve and 5 applies 0.05 gamma (Default=5)
##
## @ MaskSAD     - Artifact masking strength for bad motion, 0 to disable (Default=190)
##
## @ thSCD1      - MSCDetection scene detection treshold 1 (Default=500)
##
## @ thSCD2      - MSCDetection scene detection treshold 2 (Default=150)
##
## @ SkipOver    - Skip interpolation of frames when artifacts cover more than specified treshold, 
##                 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##          
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble", string "Preset",
\ int "BlkSize", int "BlkSizeV", string "Output", float "MaskStr", int "MaskSAD", int "thSCD1", int "thSCD2", int "SkipOver")
{
    Output     = Default(Output, "auto")
    FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
    NewNum     = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
    NewDen     = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
    Preset     = Default(Preset, "normal")
    BlkSize    = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
    BlkSizeV   = Default(BlkSizeV, BlkSize)
    blkmin     = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
    MaskStr    = Default(MaskStr, 5)
    MaskSAD    = Default(MaskSAD, 190)
    #MaskOcc    = Default(MaskOcc, 64)
    thSCD1     = Default(thSCD1, 500)
    thSCD2     = Default(thSCD2, 150)
    SkipOver   = Default(SkipOver, 48)

    Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster", "FrameRateConverter: Preset must be slow, normal, fast or faster")
    Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
    Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
    Assert(MaskStr > 0 && MaskStr <= 100, "FrameRateConverter: MaskStr must be between 1 and 100")
    Assert(SkipOver >= 0 && MaskStr <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

    # Performance settings: slow, normal, fast, faster
    Recalculate= preset == "slow" || preset == "normal"
    Prefilter  = preset == "slow" || preset == "normal" || preset == "fast"
    DCT        = preset == "slow" ? 1 : 0

    ## "B" - Blending, "BHard" - No blending
    BHard = ChangeFPS(C, NewNum, NewDen)
    B  = ConvertFPS(C, NewNum, NewDen)
    B  = FrameDouble ? SelectOdd(B) : B

    ## jm_fps interpolation
    prefiltered = Prefilter ? (C.IsYUV ? Median(C, uu = true, vv = true) : Median(C)) : C
    superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
    super = Prefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
    bak = MAnalyse(superfilt, isb = true, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = DCT)
    fwd = MAnalyse(superfilt, isb = false, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = DCT)
    fwd = Recalculate ? MRecalculate(super, fwd, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : fwd
    bak = Recalculate ? MRecalculate(super, bak, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : bak
    Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

    ## "EM" - error or artifact mask
    EM  = MaskSAD > 0 ? MMask(C, bak, ml=MaskSAD, kind=1) : BlankClip(C)   # kind=SAD
    EMfwd = MaskSAD > 0 ? MMask(C, fwd, ml=MaskSAD, kind=1) : EM           # kind=temporal blending
    EMfwd = FrameDouble ? DeleteFrame(EMfwd,0) : EMfwd
    #EMocc = MaskOcc > 0 ? MMask(C, bak, ml=MaskOcc, kind=2).mt_inpand() : BlankClip(C)  # kind=occlusion
    EM  = MaskSAD > 0 ? Overlay(EM, EMfwd, opacity=0.5, mode="lighten") : EM
    #EM  = MaskOcc > 0 ? Overlay(EM, EMocc, opacity=0.5, mode="lighten") : EM
    SkipEM = EM

    ## mask strength
    EM  = Levels(EM, 0, MaskStr / 100.0, 255, 0, 255, coring=false)
    
    EM  = BicubicResize(EM, Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    \       .mt_binarize(92)
    \       .Blur(.6)
    \       .BicubicResize(C.Width, C.Height)
    
    ## "Sc" - scene detection
    Sc  = thSCD1 > 0 && thSCD2 > 0 ? MSCDetection(C, bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(EM)
    Sc  = SkipOver > 0 ? ConditionalFilterMT(SkipEM, BlankClip(EM, color=color_white), Sc, "AverageLuma", ">", string(SkipOver)) : Sc

    ## Convert masks to desired frame rate
    EM = ChangeFPS(EM, NewNum, NewDen)
    Sc = ChangeFPS(SC, NewNum, NewDen)

    ## the FrameRateConverter magic happens
    M   = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
    M   = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

    R = (StrCmpi(Output, "auto")==0)        [** auto:  artifact masking     *] 
    \     ? (FrameDouble ? Interleave(C, M) : M)
    \ : (StrCmpi(Output, "flow")==0)        [** flow: interpolation only   *] 
    \     ? Flow
    \ : (StrCmpi(Output, "none")==0)        [** none: ConvertFPS only       *] 
    \     ? B
    \ : (StrCmpi(Output, "mask")==0)        [** mask:  mask only            *] 
    \     ? EM
    \ : (StrCmpi(Output, "over")==0)        [** over:  mask as cyan overlay *] 
    \     ? mt_merge(
    \         Overlay(Flow, MergeRGB(BlankClip(EM), EM, EM).ConvertFPS(NewNum, NewDen), mode="Add", opacity=0.40),
    \         BlankClip(EM, color=color_darkgoldenrod), mt_lut(Sc, "x 2 / "), luma=true)
    \ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|over)")
    return R
}

[stax76]

Otherwise I have to tell you guys that I am outta here. The last incarnation of the script requires AviSynth+, and this pushes it over the edge for me. You guys do live on a different planet. The only guy who is even further out is stax76 who seriously wanted to raise the minimum requirement for StaxRip to Win 10 with the recent Creator's update (I think he revoked it for the time being).

I admit it was stupid and I realized, revoked and apologized after only one day. I test staxrip on Win 7 using wmware almost daily! staxrip will probably be the first app to give up Win 7 support, I'm not proud of it, but since it was always like this it's also something that can be expected.

[raffriff42]

Originally Posted by burfadel

I notice that in several places in the script different functions are written as (example)
EM.ChangeFPS instead of
ChangeFPS(EM,), or ChangeFPS(clip = EM,)

There is no difference in performance, as far as I know.

http://avisynth.nl/index.php/Grammar

The fourth form is an alternate syntax called "OOP notation" in AviSynth:

• expression . function_name ( argument_list ) is equivalent to
• function_name ( expression , argument_list )

However, occasionally there is a need for the normal, non-"OOP" form; for example, if a filter accepts multiple clips in the left-most set of arguments, implicit Last may insert itself as the first clip when not intended. When that happens, the output is obviously wrong; it's not a subtle performance hit. (except possibly in the case of ScriptClip)

It's only an issue where implicit Last is involved. With explicit variables, both forms are equivalent.

...and ChangeFPS(clip = EM,...) will not work. There is no argument named 'clip' - it's a unnamed argument.

[StainlessS]

FrameRateConverter() script is for Avs+ only, but there is also a Median() function in Median.dll v2.5 plugin.

Median         "c+[CHROMA]b[SYNC]i[SAMPLES]i[DEBUG]b"
MedianBlend    "c+[LOW]i[HIGH]i[CHROMA]b[SYNC]i[SAMPLES]i[DEBUG]b"
TemporalMedian "c[RADIUS]i[CHROMA]b[DEBUG]b"

So maybe should use "ModPlus_Median()" instead (to explicitly name the required dll).
http://avisynth.nl/index.php/Plugins...Function_Names

EDIT: Leastwise I take it that Median exists in the modPlus dll.
EDIT: Yes it does:- http://www.avisynth.nl/users/vcmohan...s/modPlus.html
EDIT: Would not be surprised if filter 'Median()' exists in more than those two dll's.

[Groucho2004]

Originally Posted by manolito

The last incarnation of the script requires AviSynth+, and this pushes it over the edge for me.

What's wrong with AVS+? It should work on your SSE CPU:

if(CMAKE_SIZEOF_VOID_P EQUAL 4)
    # VC++ enables the SSE2 instruction set by default even on 32-bits. Step back a bit.
    add_definitions("/arch:SSE")
    #add_definitions("/arch:SSE2") # Better use this one, it's 2016 now
endif()

[pinterf]

Originally Posted by Groucho2004

What's wrong with AVS+? It should work on your SSE CPU:

if(CMAKE_SIZEOF_VOID_P EQUAL 4)
    # VC++ enables the SSE2 instruction set by default even on 32-bits. Step back a bit.
    add_definitions("/arch:SSE")
    #add_definitions("/arch:SSE2") # Better use this one, it's 2016 now
endif()

My builds override this flag. I always published SSE2-only builds, to have optimized functions (mainly because of high bit depths that had less SSE2 optimized filters) for C-only parts. No one complained so far.

[Groucho2004]

Originally Posted by pinterf

My builds override this flag. I always published SSE2-only builds, to have optimized functions (mainly because of high bit depths that had less SSE2 optimized filters) for C-only parts.

Seems reasonable. Manolito's CPU (Athlon XP, I think) does not support SSE2 and above.

[manolito]

I followed the AVS+ threads by Utim and pinterf loosely, I tried AVS+ a few times (on the ancient desktop where it didn't work and on the newer laptop where it did). My conclusion so far is very clear:

All the new stuff which AVS+ offers does not do anything for me. I do not need 64-bit, I do not need MT, I do not need the high bit depths.
But I do need an AviSynth build which is stable (looking at the AVS+ thread all I can say is that this is work in progress, looks far from stable to me).
I absolutely do need to use older 32-bit filters like the VDub filters.
And I do need to use it in a variety of older software (without switching versions - even if Groucho made this quite convenient).

So it looks like I will stick with the plain vanilla AviSynth versions for some time...


Cheers
manolito

[manolito]

Originally Posted by Groucho2004

Manolito's CPU (Athlon XP, I think) does not support SSE2 and above.

No, it is an Intel Celeron (Coppermine) 1.1 GHz. Hitting the 1 GHz limit was a milestone at that time. And I do not have an upgrade path for the old desktop because the big tower only accomodates BAT mainboards, not ATX. I did upgrade to the latest available BAT MoBo, but that was in the year 2000...

And I have to say that this is by far the most reliable computer I ever had. The original machine is from 1993, it started as an AMD 486DX266. I upgraded it constantly, but the power supply is still the original one, and my drive A: is still a 5.25" floppy drive. And with my highly optimized WinXP installation it only feels slow when I try to play HD videos...


Cheers
manolito

[MysteryX]

Which part of the script requires Avisynth+? Unless modPlus.dll requires Avisynth+ but it should work with v2.6 too

I decided last-minute to compile ConditionalMT with XP support -- glad it serves someone.

EM  = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
    \       .mt_expand(mode= mt_circle(zero=true, radius=1))
    \       .mt_binarize(92)
    \       .Blur(.6)
    \       .BicubicResize(C.Width, C.Height)

Let's split this up.

EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
EM = EM.mt_expand(mode= mt_circle(zero=true, radius=1))
EM = EM.mt_binarize(92)
EM = EM.Blur(.6)
EM = EM.BicubicResize(C.Width, C.Height)

Which of these lines fail? BicubicResize and Blur are very unlikely. mt_expand or mt_binarize are most likely the ones crashing .. but I wonder why. Neither take blksize as parameter.

[burfadel]

I found there was no speed difference with the different syntax including the different reference to EM.

Could this have something to do with it?

## "B" - Blending, "BHard" - No blending
BHard = ChangeFPS(C, NewNum, NewDen)
B = ConvertFPS(C, NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## Convert masks to desired frame rate
EM = ChangeFPS(EM, NewNum, NewDen)
Sc = ChangeFPS(SC, NewNum, NewDen)

M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

Aren't you mixing changefps and convertfps which work slightly differently?

[MysteryX]

Originally Posted by burfadel

Aren't you mixing changefps and convertfps which work slightly differently?

Blending for artifact masking, ChangeFps for scene changes and skipped bad scenes.