FrameRateConverter (Official)

[Selur]

Nice! Thanks!

[Selur]

Hmm, when using FrameRateConverter in Vapoursynth I get:
vapoursynth.Error: BlankClip: nodes foreign to this core passed as input, improper api usage detected
script I used was:
# Imports
import os
import sys
import vapoursynth as vs
core = vs.get_core()
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.append(os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/FrameFilter/FramerateConverter/FrameRateConverter-x64.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import havsfunc
import FrameRateConverter
# source: 'G:\TestClips&Co\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/test.avi",cachefile="E:/Temp/avi_9dec25d3f707eb4813d42334c7f1a8d6_853323747.ffindex",fpsnum=25,format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting frame count&rate with FramerateConverter
clip = FrameRateConverter.FrameRateConverter(clip, frameDouble=True, blkSize=8, blkSizeV=8, output="auto", skipOver=210) # new fps: 50
# adjusting output color from: YUV420P8 to YUV420P10 for x265Model (i420@8)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

-> Any idea what could cause this?

Okay, replacing:
core = vs.get_core()
with:
from vapoursynth import core
in FrameRateConverter.py (and deleting the Python cache files) fixed the issue.

Cu Selur

Ps.: Should such thinks better be reported here, over at github or at both places?

[MysteryX]

I cannot reproduce your problem. This works for me

import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
import FrameRateConverter as frc
import vapoursynth as vs
core = vs.get_core()

core.std.LoadPlugin(path=r'C:\GitHub\FrameRateConverter\Src\x64\release\FrameRateConverter.dll')
core.avs.LoadPlugin(path=r'C:\GitHub\FrameRateConverter\masktools2.dll')
#file='E:\AVSMeter\Motion Estimation Torture Clip.avi'
file="E:\AVSMeter\MotionTest2.mp4"
#file="E:\NaturalGrounding\Girl's Day\Original\Female President.mp4"
video = core.ffms2.Source(source=file, format=vs.YUV420P8, alpha=False)
clip = video.std.SetFrameProp("_FieldBased", intval=0)

clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
clip = frc.FrameRateConverter(clip, frameDouble=True, blkSize=8, blkSizeV=8, output="auto", skipOver=210) # new fps: 50
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip.set_output()

To keep note:

- Need to validate that _FieldBased == 0 or throw an error. The weird resizing issue I had was due to this flag being improperly set.

- Need to call AssumeFPS or validate that the input is already fixed frame rate

[MysteryX]

Better report issues here. vs.get_core or import core, why is it making a difference?

[Selur]

My guess is:
get_core(..) -> seems to create a new core-instance
import core -> takes the existing core-instance
compared how havsfunc got the core and there 'import ..' was used.

[MysteryX]

I'm doing vs.get_core() and it doesn't crash.

AH! -- libraries may need to specifically import core. Is this documented anywhere? Even there... I'm not getting your error.

[Selur]

Might be related to that I use a portable Vapoursynth version,...

[Selur]

A few questions regarding FrameRateConverter in Vapoursynth:

Is it correct/intended that FrameRateConverter always outputs RGB24?
Using:
# Imports
import os
import sys
import vapoursynth as vs
core = vs.get_core()
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.append(os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/FrameFilter/FramerateConverter/FrameRateConverter-x64.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import havsfunc
import FrameRateConverter

clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",fpsnum=25,format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)

# adjusting frame count&rate with FramerateConverter
clip = FrameRateConverter.FrameRateConverter(clip, newNum=25, newDen=1, blkSize=8, blkSizeV=8, output="over", maskThr=100, blendOver=65, skipOver=255) # new fps: 25

# Output
clip.set_output()
the output is RGB24 (I was expecting YUV420P8, same as the input).
If this is correct, does it respect the color matrix and luma range durint the yuv->rgb conversion?
any plans for high bit depth support?


Cu Selur

[MysteryX]

That's because the Overlay function in VapourSynth only behaves as expected when done in the RGB space; gives weird result in YUV. I guess I could convert it back to the original format afterwards so that there's no matrix issue; and I should do that YUV->RGB->Overlay->YUV conversion in 16-bit.

There shouldn't be much issue with HBD, just need to try and see where it blocks.

[poisondeathray]

That's because the Overlay function in VapourSynth only behaves as expected when done in the RGB space; gives weird result in YUV.

In what way is the Overlay result "weird" in YUV ?

[Selur]

I guess I could convert it back to the original format afterwards so that there's no matrix issue; and I should do that YUV->RGB->Overlay->YUV conversion in 16-bit.
Looking forward to it. :)

There shouldn't be much issue with HBD, just need to try and see where it blocks.
clip = FrameRateConverter.FrameRateConverter(clip, frameDouble=True, blkSize=8, blkSizeV=8)
with a yuv420p10 clip produces:
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2242, in vapoursynth.vpy_evaluateScript
File "src\cython\vapoursynth.pyx", line 2243, in vapoursynth.vpy_evaluateScript
File "C:\Users\Selur\Desktop\test.vpy", line 30, in <module>
clip = FrameRateConverter.FrameRateConverter(clip, frameDouble=True, blkSize=8, blkSizeV=8) # new fps: 50
File "I:\Hybrid\64bit\vsscripts\FrameRateConverter.py", line 178, in FrameRateConverter
EM = ToGray(C.mv.Mask(bak, ml=255, kind=1, gamma=1/gam, ysc=255, thscd2=skipOver))
File "src\cython\vapoursynth.pyx", line 2067, in vapoursynth.Function.__call__
vapoursynth.Error: Mask: input clip must be GRAY8, YUV420P8, YUV422P8, YUV440P8, or YUV444P8, with constant dimensions.
comes from https://github.com/dubhater/vapoursynth-mvtools/blob/6d805e232397cb08074dbb58cc02849f63b96714/src/MVMask.c#L328
Seems like the documentation (https://github.com/dubhater/vapoursynth-mvtools/blob/master/readme.rst) and functionality of mvtools don't match. :/
known 'missing feature' -> https://github.com/dubhater/vapoursynth-mvtools/issues/16

[MysteryX]

Actually a 8-bit mask is perfectly fine. Just need to convert it to Gray16 to make it compatible.

[MysteryX]

As an update on the discussion about Pixel Addressing, it is now supported in this VapourSynth plugin (https://github.com/AkarinVS/vapoursynth-plugin/releases/tag/v0.60).

[kedautinh12]

Time to replace masktool with ex_ family in Vapoursynth
https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/FrameRateConverter.avsi

[MysteryX]

ex_ scripts haven't been ported to VapourSynth AFAIK. Plus the VPY replacement methods are highly optimized for HBD.

[kedautinh12]

About BicubicResize
https://forum.doom9.org/showthread.php?p=1946623#post1946623
https://forum.doom9.org/showthread.php?p=1946622#post1946622

[MysteryX]

DogWay, to replace Blur(.6) (https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/FrameRateConverter.avsi#L246) with a Convolution like you're doing, this convolution matrix gives a better result. If the blur is too wide, it lets artefacts leak through.

Convolution(matrix=[0, 1, 0, 1, 3, 1, 0, 1, 0])

for StripeMask giving "worse" results, I get similar results as before if doing it on PC range instead of Full range. Playing with the color range... seems like you can get better results by reducing the range. I might add a parameter to adjust the value range on top of the threshold.

[wonkey_monkey]

Wouldn't something like

Convolution(matrix=[1, 3, 1, 3, 9, 3, 1, 3, 1])

be more accurate? Ignoring the corner pixels seems wrong to me.

[Dogway]

I run some tests and yes Mysterys's kernel is more similar to either blur(0.6) or blur(0.5), actually a practical match to blur(0.5) besides the kernel is shorter so faster to compute.

Dog: Expr("x[-1,1] x[0,1] x[1,1] x[-1,0] x[0,0] 6 * x[1,0] x[-1,-1] x[0,-1] x[1,-1] + + + + + + + + 0.071428571 *","")
monkey: Expr("x[-1,1] x[0,1] 3 * x[1,1] x[-1,0] 3 * x[0,0] 9 * x[1,0] 3 * x[-1,-1] x[0,-1] 3 * x[1,-1] + + + + + + + + 0.04 *","")
Mystery: Expr("x[0,1] x[-1,0] x[0,0] 3 * x[1,0] x[0,-1] + + + + 0.1428571428 *","")
mergeluma(blur(0.6))
mergeluma(blur(0.5))

[MysteryX]

It's not about accuracy (of a pure white/black mask), but about having the right shape: not too wide, and not too square.

[MysteryX]

btw the existing StripeMask has a memory leak. I wonder why that never came up before. I cannot reproduce the memory leak in v2 beta in Avisynth. In VapourSynth, memory usage goes up from 1.3GB to 1.8GB over time but that could just be the way it handles the cache.

I've been playing with parameters to try to match previous results; the bar is pretty high, my best efforts barely come to 90% of the previous results!

Working with Luma-only (ShufflePlanes) works better than converting to Grey8. Btw, why do ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY) and resize.Point(format=vs.GRAY8) give different results? The latter gives a slightly darker output.

This is the pre-filtering that worked the best so far... yet it's not as good as the previous results! The linear gamma conversion is a bit different. I guess I got very lucky on the last released version (except that the old version assumes PC range and would behave different with Full range input)


video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1/2.2, min_in=16, max_in=235, min_out=65, max_out=190)


The question is: what makes it that I can't fully match the accuracy of the last version? It was using thr=23. Now I nearly match it with thr=15, range=125 (considerably compressing the data range)

Before I was using ColorYUV to change gamma. Perhaps the difference is the way Gamma is applied, over Full or PC range? I think it was applying full-range gamma on a PC-range clip... somehow improving results. What impact did it really have? Anything to learn here in terms of better tweaking the input before processing?

Edit: I suppose converting to GRAY8 takes chroma into account during conversion. If I use ShufflePlanes, then coloured stripes with similar Luma would fail to be detected.

Note: StripeMask could very well run on an image with Width and Height reduced by half to increase performance.

[wonkey_monkey]

blur(0.6) produces the following result on a single white pixel (in RGB32, anyway; it's slightly different in RGB24 for some reason):

8 29 8
29 110 29
8 29 8

So it does spread to the corners, and these would be the best numbers if you want the closest match.

EDIT: My mistake, that's in Avisynth. Vapoursynth blur() might be different.

[MysteryX]

I could use some help here. Here's a new DLL vesion. (https://mega.nz/file/mBhCFTKK#0ZaI0jc48Aioeq3jXaOJFAsoHD7Tcy8I4S2Lj2DMu_I)

The download link contains a sample video clip. Can you try to get a more accurate stripe detection, which means, more of the fence being detected, and less false positives. You can compare to Avisynth StripeMask from before. Best I got so far is about 90% the quality of before for some reason. You can play with levels, contrasts, gamma, perhaps sharpening. Havsfunc has SmoothLevels which perhaps could be useful. Resize to Gray8 or ShufflePlane? This is time-consuming trial-and-error task that requires a lot of testing. This version has an additional parameter "skip". When set, it bypasses the pre-filter calls so you can do it manually.

Let me know what you find out!

Also for the crash about invalid core -- I made a small change. Can you test whether you still get a crash?

As for ContinuousMask, I fixed it to process all planes (instead of leaving garbage on chroma planes). It should behave more appropriately now. There really isn't much to debug in there... (https://github.com/mysteryx93/FrameRateConverter/blob/master/Src/Common/ContinuousMaskBase.cpp) Perhaps I could use a Threshold parameter so that it processes pixels above a certain value (other than 0), if it gets useful for anyone else.

[zorr]

You can compare to Avisynth StripeMask from before. Best I got so far is about 90% the quality of before for some reason.

Just to be clear - so this difference is not just between the Avisynth and Vapoursynth versions but with the current and previous Avisynth versions as well?

This is time-consuming trial-and-error task that requires a lot of testing.

Sounds like a perfect job for Zopti. Since we have a good result (the old Avisynth version) we can compare the result with it, I think a simple diff will do instead of some fancy SSIM or GMSD metric. If you want even better results you could create a mask manually (even for just one frame).

I'll try to come up with a Zopti script and do some testing. First I will need to recreate the good result and save it as lossless video.

[MysteryX]

Previous version was using ColorYUV to change gamma. It was never converting to Gray8 and was only using the Luma plane, so that's the equivalent of ShufflePlanes. I don't know whether there are other "unknown" factors or errors coming into play.

[zorr]

Is this the correct code for the good result (using FrameRateConverter v1.3)?

src = FFVideoSource("MotionTest2.mp4")
src = src.BicubicResize(src.width / 2, src.height / 2)
mask = src.StripeMask(thr=23)
return mask

[zorr]

The VapourSynth version is returning a mask with two gray values, is that expected?

https://i.postimg.cc/hvY5RgyN/vs-stripemask.png

The Avisynth version returns a mask with luma 0 or 255, VS version returns a mask with luma values 0, 100 or 200.

EDIT: Ah, that's controlled by the parameters str and strf. So I guess we should use the default values to match the Avisynth version?

[zorr]

StripeMask now has parameter range. What's it for and what is the usable value range? Should it always be max_out - max_in?

[MysteryX]

The lesser-gray is adding the next frame into it. Temporal stabilizer. Str sets the color of matching areas, strf(orward) sets the color of next frame data.

BTW......... could I improve performance 2x by caching the data of next frame instead of calculating it twice? I guess so. Or perhaps initialize the plugin twice with 1-frame offset and merge them.

Btw reducing size by half is only for performance reason because I don't need the clip to be that big; can reduce it to various sizes to test with various block sizes. Also see the impact of processing on half-resolution or quarter-resolution vs full-resolution (with half or quarter block sizes).

Range. If you want Full range, set 255. If you want PC range, set 220. To further compress the data range, I've set it to 125. Note that if skip=1, range isn't doing anything because it's only used for pre-filtering.

The ZIP contains the VapourSynth script that is working good,


import os
import sys
dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(dir)

import vapoursynth as vs
core = vs.get_core()

core.std.LoadPlugin(path=dir + r"\FrameRateConverter-x64.dll")
file="MotionTest2.mp4"
video = core.ffms2.Source(source=file, format=vs.YUV420P8)
video = video.resize.Bicubic(video.width / 2, video.height / 2)

video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1/2.2, min_in=16, max_in=235, min_out=65, max_out=190) # range=125
video = core.frc.StripeMask(video, blksize=16, str=200, strf=100, thr=15, range=125, skip=1)

video.set_output()


btw I'm curious what would be the effect of sharpening.

[zorr]

I've got some interesting results.

Short version: You can use thr=23, min_out=0, max_out=208 and gamma=1/2.078 to get a pretty good result. I measured similarity against the older Avisynth version and average SSIM is 0.987 per frame (1.0 is perfect score).

Long version: Naturally I used Zopti to find the optimal values. First I needed a "reference video" so I ran StripeMask with FrameRateConverter v1.3 and saved the resulting mask as lossless video. I also created a pre-scaled lossless version of the original video to avoid unnecessary work during optimization.

I created this VapourSynth / Zopti script:

import os
import sys
from zoptilib import Zopti

dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(dir)

import vapoursynth as vs
core = vs.get_core()
core.std.LoadPlugin(path=dir + r"\FrameRateConverter-x64.dll")

zopti = Zopti(r'results.txt', metrics=['ssim', 'time'])

file="MotionTest2_scaled.avi"
video = core.ffms2.Source(source=file, format=vs.YUV420P8)

goodmask = core.ffms2.Source(source="StripeMask_good.avi")
goodmask = core.std.ShufflePlanes(goodmask, planes=0, colorfamily=vs.GRAY)

gamma = 220/100.0 # optimize gamma = _n_/100.0 | 1..300 | gamma
min_out = 65 # optimize min_out = _n_ | 0..100 ; max:max_out 1 - | min_out
max_out = 190 # optimize max_out = _n_ | 100..255 ; min:min_out 1 + | max_out
thr = 15 # optimize thr = _n_ | 1..100 | thr

video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1/gamma, min_in=16, max_in=235, min_out=min_out, max_out=max_out) # range=125
video = core.frc.StripeMask(video, blksize=16, thr=thr, range=max_out-min_out, skip=1)

frame = 20
video = video[frame]
goodmask = goodmask[frame]

zopti.run(goodmask, video)

So there are four optimized parameters: gamma, min_out, max_out and thr. I didn't include min_in or max_in because those should be correct for this video.

Gamma range is 1 - 300 divided by 100 so effectively 0.01 - 3.0 in steps of 0.01. max_out and max_in have dependencies, max should always be larger than min so I set a min limit for max_out and max limit for min_out.

I only measured SSIM on one frame to save time and just to see if even one frame can be matched well (if we can't do that then doing it for the whole clip is hopeless). The frame number 20 contained interesting mask features so that's the one I chose. I later found out that if the result matches well with this one frame it also matches well with the rest of the frames.

I ran a couple of Zopti tests with
zopti StripeMaskTest.vpy -alg mutation -iters 1000 -pop 48 -threads 6 and then set the iterations to 10000 and population to 30. With VapourSynth you don't need that many threads to saturate all cores, with Avisynth I usually run -threads 16 or 24 (and 16 usually only when there isn't enough memory to run more).

The best result was 0.9778275 so already quite promising and I verified visually that it's pretty close. You can output the best result script with zopti -mode evaluate -scripts best

Next I wanted to see some heatmaps to better understand where the good values are. The first one was for max_out and thr.

https://i.postimg.cc/rpL5Qbf9/Stripe-Mask-Test-ex-thr-max.png

There's a nice linear area of good values. In theory you could find a good match with any thr or max_out value as long as you adjust the other one. The red dot is the best found result.

The next one is min_out and max_out.

https://i.postimg.cc/d0v4LLcv/Stripe-Mask-Test-ex-min-max.png


This revealed to me what you probably already knew - that the absolute values of min_out and max_out don't matter, only their distance (which is the range). Ok, this means I can run the next tests with min_out=0 and leaving only gamma, max_out and thr to optimize.

What about gamma + min_out?

https://i.postimg.cc/PqtQd915/Stripe-Mask-Test-ex-gamma-min.png

That's odd... there's kind of checkerboard pattern going on. Changing gamma slightly makes the result go up and down. Let's take a closer look at the gamma alone:

https://i.postimg.cc/m2YdRQKV/Stripe-Mask-Test-ex-gamma-2021-07-05-01-56-46-optimize-exhaustive-run-01.png

Oh wow, no wonder you had trouble finding a good match, the gamma behaves quite erraticly! I wonder what's going on, is the up-down swing also there in the brightness of the image? I don't know yet.

Meanwhile I ran some tests with 100 000 iterations and using more focused search ranges. I decreased the gamma step to 0.001. I had trouble finishing those runs as the execution would sometimes stop with error. Once I got

Script exceeded memory limit. Consider raising cache size.
terminate called after throwing an instance of 'std::out_of_range'
what(): _Map_base::at
even though I had plenty of memory left (about 40 gigs). Usually the script just reports Output 1 frames in 0.62 seconds (1.61 fps) or similar. Zopti has a tendency to find rare bugs so this could be a sign of that.

Anyway, two complete and one almost-complete runs later they all found the best result 0.98015726 so I think that's as good as it gets. I think giving gamma even more decimals could improve things a bit so that's something I'll try.

I looked at some heatmaps of these 100000 iteration runs and they look like they're straight out of a horror film:

https://i.postimg.cc/zBfmHdYJ/Stripe-Mask-Test2-run-01-1.png

I have no idea why there are those vertical stripes but I know they use different thr than the best found value. First I thought they are just some randomness but the stripes were found in every one of the three runs at the same locations. Investigation continues...

[MysteryX]

If you took the v1.3 results as reference, those aren't perfect at all. Perhaps you could manually draw the areas in Photoshop as a perfect reference mask, and see how close the tests get. Comparing to v1.3, there are random fluctuations of areas that are included or excluded that would randomize the test results. You need a fixed and clean reference.

As for performance, it could be optimized by doing this in a script function:
- Create 2 clips with 1-frame offset
- Output with 255 (pure black & white)
- Apply each mask over a blank clip of the desired colour
- Merge the 2 masks together, taking the highest value of each clip

Could someone write that function? It should double the performance.

Did you try sharpening just out of curiosity? Anything else that could be tried?

P.S. your last test, it's a galactic war between two fleets. The Dark Fleet tried to take over the galaxy.

The gamma could be altering the way stripe patterns are being detected or not past certain thresholds. Your white bars kind of look like the stripes we're trying to detect. If playing with the gamma considerably alters the results, there's also the option of running the mask different times with various gamma values and merging them. Or a parameter: run with 1, 2 or 3 passes.

[wonkey_monkey]

Oh wow, no wonder you had trouble finding a good match, the gamma behaves quite erraticly!

Rounding errors/artefacts?

[MysteryX]

This function gives just over 25% performance gain with strf

def StripeMask2(clip, blksize = None, blksizev = None, overlap = None, overlapv = None, thr = None, range = None, comp = None, compv = None, str = None, strf = None, lines = None, skip = None):
if not isinstance(clip, vs.VideoNode):
raise vs.Error('StripeMask: This is not a clip')

mask = clip.frc.StripeMask(blksize=blksize, blksizev=blksizev, overlap=overlap, overlapv=overlapv, thr=thr, range=range, comp=comp, compv=compv, str=str, strf=0, lines=lines, skip=skip)
if strf > 0:
maskf = mask.std.DeleteFrames(frames=[0]).std.DuplicateFrames(frames=[clip.num_frames-2])
return core.std.Expr(clips=[mask, maskf], expr=f"x {str} y {strf} 0 ? ?")
else:
return mask

[zorr]

I ran one more test to see how much even more decimals for gamma helps. Not much, best result was 0.98015726 with gamma 1/2.0778.

Rounding errors/artefacts?

I decided to take a closer look at how the gamma behaves. This is the average brightness of the source frame 20 with gammas between 1/2.05 and 1/2.1.

https://i.postimg.cc/Bnv8hB7h/Brightness-Test-2021-07-07-20-45-16-optimize-exhaustive-run-01.png

So while it's not buttery smooth it doesn't look like it's doing anything radical. If we draw a straight line from gamma 1/2.05 to gamma 1/2.1 and calculate the offset it looks like this:

https://i.postimg.cc/jdM2JJqy/Brightness-Test-nonlinear-2021-07-07-20-14-42-optimize-exhaustive-run-01.png

Not pretty but the errors are small.

If you took the v1.3 results as reference, those aren't perfect at all. Perhaps you could manually draw the areas in Photoshop as a perfect reference mask, and see how close the tests get.


Yes this is just a starting point, at least we now know that there is nothing fundamentally wrong with the VS version as it's capable of emulating the Avisynth version.

Doing one or more masks manually and letting Zopti play with the parametes is probably a good next step. With Zopti you can develop algorithms on the idea level not having to bother with the gruesome testing of whether this or that idea actually delivers. Of course setting up the test scripts and running them still takes time but it's still tremendously helpful.

Comparing to v1.3, there are random fluctuations of areas that are included or excluded that would randomize the test results. You need a fixed and clean reference.

Yes there are some differences, curiously the VS version usually makes the mask one pixel wider than Avisynth. I'm not sure what you mean by fixed and clean reference. Using the same clip and frame counts as fixed, right?

Did you try sharpening just out of curiosity? Anything else that could be tried?

Didn't try sharpening, that (and blurring) should be tested as well. My first idea on how to improve the mask was to remove the smaller areas using the RemoveSmallSpots-function I used with Delta Restore (https://forum.doom9.org/showthread.php?p=1944708#post1944708). And likewise fill the small holes of the mask using a similar FillSmallHoles -function. I also tried offsetting the image half blksize and taking another StripeMask and combining it with the original. This seems to stablilize the mask somewhat just like using strf. Here's the script (including aforementionded functions) I played with:


src = FFVideoSource("MotionTest2.mp4")
src = src.BicubicResize(src.width / 2, src.height / 2)

blksize = 16
thr=20
offset = 8
spotSize = 14
holeSize = 20
finalHoleSize = 22

src2 = src.AddBorders(offset,0,0,0)

mask = src.StripeMask(thr=thr, blksize=blksize)
mask2 = src2.StripeMask(thr=thr, blksize=blksize)
mask2 = mask2.Crop(offset,0,0,0)
mask2 = mask2.RemoveSmallSpots(spotSize).FillSmallHoles(holeSize)

mask = mask.RemoveSmallSpots(spotSize).FillSmallHoles(holeSize)
mask_final = mt_logic(mask, mask2, "and")
mask_final = mask_final.RemoveSmallSpots(spotSize).FillSmallHoles(finalHoleSize)

return src.Overlay(mask_final, opacity=0.5)


function FillSmallHoles(clip mask, int holeSize) {
return mask.mt_invert().RemoveSmallSpots(holeSize).mt_invert()
}

function RemoveSmallSpots(clip mask, int spotSize) {
mask_orig = mask

# remove too small spots
for (i = 1, spotSize) {
mask = mask.mt_inpand()
}
mask = mt_hysteresis(mask, mask_orig, chroma="-128")
return mask
}

Perhaps the area around the found mask should be allowed to match the stripes with a lower threshold, that way some of those cracks in the mask could be filled. This can be implemented by running another StripeMask with lower threshold and anding it with the original which has been expanded about half blocksize.


P.S. your last test, it's a galactic war between two fleets. The Dark Fleet tried to take over the galaxy.

:D

there's also the option of running the mask different times with various gamma values and merging them. Or a parameter: run with 1, 2 or 3 passes.

Yes, good ideas that could be tested.

[MysteryX]

Yes there are some differences, curiously the VS version usually makes the mask one pixel wider than Avisynth.

Mask areas are wider, or the output clip is wider?


I'm not sure what you mean by fixed and clean reference. Using the same clip and frame counts as fixed, right?

I mean, creating a mask manually that covers 100% of the right areas with no false positive.


I also tried offsetting the image half blksize and taking another StripeMask and combining it with the original. This seems to stablilize the mask somewhat just like using strf.

strf stabilizes in time (after all, the mask hides all generated frames between this frame and the next), whereas this will help fill spacial holes. It's actually a really good idea! Cutting the image in half, and working on a single grey plane, the performance hit will be minimal.

Small spots will be removed by ContinuousMask. mt_expand fills small holes.

[zorr]

Mask areas are wider, or the output clip is wider?

The mask, and just the right edge. It's actually two pixels, and sometimes the bottom edge is one pixel wider. Like this:

https://i.postimg.cc/26HyzjYm/vs-avisynth-mask-comparison.gif

I mean, creating a mask manually that covers 100% of the right areas with no false positive.

Oh yes. And that's what I did. Well, perhaps not 100% right areas as that depends on how MaskTools sees the frame (where the stripes are causing trouble). But I did roughly select the fence, if StripeMask can do that it's going to be a significantly improved result. I chose 4 frames, numbers 5, 20, 32 and 72. This is the mask in frame 5:

https://i.postimg.cc/8C4M6SGX/ref-mask-05.png

I've started running Zopti on this new challenge, first using the same three params gamma, max_out and thr. Initial results indicate that this is a much harder problem.

It's actually a really good idea! Cutting the image in half, and working on a single grey plane, the performance hit will be minimal.

Not sure I follow but I'm glad I was able to help! :)

Small spots will be removed by ContinuousMask. mt_expand fills small holes.

Ok, we just have to test the ContinuousMask as well. Actually we should follow the whole procedure in the .avsi and include new parameters from there.

mt_expand will also make the mask larger, if you only want the holes filled then it must be combined with mt_hysteresis. But that comes with a performance penalty (I haven't measured but since mt_hysteresis is a fill algorithm it's probably not super fast) and might not be worth it.

[MysteryX]

The mask, and just the right edge. It's actually two pixels, and sometimes the bottom edge is one pixel wider. Like this:
Is that between VapourSynth and Avisynth or between v1.3 and v2.0? I did fix a bug where sometimes the area to mark wasn't closed properly and a huge bar would appear; so I did a change to the closing of the masked area. That could be the pixel difference.

I'm not sure it's worth testing the whole procedure. The rest of the AVSI code is to give the shape I want for a mask. I do want it larger so that artefacts are properly hidden.

What I really need is just the right thr, range and gamma parameters. If doing 2 or 3 passes with different values gives a better mask, then need to find which set of params to use. Also do these parameters need to be adjusted for various videos, or are the right values right for everything?

We're better to include more and have a bit more false positive here; the small spots get removed with ContinuousMask, and the most important is for stripe areas to definitely be masked because they're UGLY otherwise.

In terms of fine-tuning the analysis, I think we've got enough data to implement something.

Unless someone wants to write machine-learning AI for stripe detection? If it's not over-kill.

[zorr]

Is that between VapourSynth and Avisynth or between v1.3 and v2.0? I did fix a bug where sometimes the area to mark wasn't closed properly and a huge bar would appear; so I did a change to the closing of the masked area. That could be the pixel difference.

Yes between v1.3 and 2.0. And that's a very logical explanation.

What I really need is just the right thr, range and gamma parameters.

Here's some results, using the default StripeMask parameters except for thr. The best SSIM found was 0.3176762 so much lower than before, especially since the maximum possible is now 4.0. I did two 100k iteration runs.

This is the best mask for each of the four test frames (target is overlaid in red).

https://i.postimg.cc/137dwDk2/Stripe-Mask-Test4-PARETO-01-0-3176762-470-vpy-0.png

The best found combo was gamma 1/1.811, max_out 247 and thr 16.

I'm leaving for a little trip for a couple of days, I'll adress the rest of your comments after that. :)

[zorr]

I'm not sure it's worth testing the whole procedure. The rest of the AVSI code is to give the shape I want for a mask. I do want it larger so that artefacts are properly hidden.
...
We're better to include more and have a bit more false positive here; the small spots get removed with ContinuousMask, and the most important is for stripe areas to definitely be masked because they're UGLY otherwise.

Perhaps the whole procedure is not needed but anything that ends up doing more than just cosmetic changes (like blurring) to the final mask should be included, otherwise we're measuring the wrong thing. For example if the ContinuousMask is left out then we won't know how much larger the mask needs to be for it to be optimal for ContinuousMask.

I actually tried ContinuousMask but seems like it's not designed to remove large areas completely. Then I did some tests with RemoveSmallSpots to see how including it will change the optimal gamma, max_out and thr. The best result was 0.32462016 with gamma=1/1.763, max_out=225, thr=13 and spotsize=20. So removing spots afterwards seems to change the optimal values.

Here's what the optimal mask looks like with RemoveSmallSpots:

https://i.postimg.cc/TwvF7FMZ/Stripe-Mask-Test5-PARETO-01-0-32462016-640-vpy-0.png

It looks much cleaner than the previous attempt.

After this I also wanted to see if changing the defaults of parameters blksize, overlap and comp will give better results. This resulted in SSIM of 0.43589097 using gamma=1/1.345, max_out=230, thr=14, blksize=27, overlap=26, comp=5 and spotsize=38.

Something I observed: optimal overlap is always the largest possible (blksize-1) and optimal comp is the largest supported 5.

After that I also included filling the small holes in the mask using FillSmallHoles and the best result was 0.45771345 using gamma=1/1.034, max_out=204, thr=9, blksize=24, overlap=23, comp=5, spotsize=61 and holesize=43.

But while the SSIM is now much better the mask looks way too large, it seems SSIM is not measuring this the way we want:

https://i.postimg.cc/15LxTJFc/Stripe-Mask-Test8-PARETO-01-0-45771345-820-vpy-0.png

So I think the next step is to replace SSIM with something better. As you mentioned it's very important to cover all of the striped areas we are better off measuring that directly - how large percentage of the target area is covered. But in addition we need another measurement to determine how much larger the mask is than what is needed to cover the target (otherwise it would be easy to get perfect score by making the mask fill 100% of the screen), this one should be minimized. Zopti does multiobjective optimization by default so this is not a problem - the results will be a pareto front with increasing target mask cover and increasing overflow area and we just need to pick the one that is the best compromise. I have already added custom props to ZoptiLib so that the similarity metric(s) can be calculated in the script.

If doing 2 or 3 passes with different values gives a better mask, then need to find which set of params to use. Also do these parameters need to be adjusted for various videos, or are the right values right for everything?

These are also interesting to test and we'll get to those once the similarity metric has been fixed. We'll need some more test clips in order to determine if one set of values works well. I think one of those could be the one I'm doing MVTools tests with (https://forum.doom9.org/showthread.php?p=1941970#post1941970).

Unless someone wants to write machine-learning AI for stripe detection? If it's not over-kill.

I'd like to try frequency domain filters such as DeFreq to create a stripe mask. The filter is not meant for creating masks but the output can be turned into a mask quite easily.

[kedautinh12]

@zorr, this is FrameRateConverter.avsi support HBD
https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/FrameRateConverter.avsi

[zorr]

Sorry about the delay, I now have some new results to share.

The latest incarnation calculates the percentage of covered mask in each of the four test frames and also the percentage of StripeMask covering the area outside of the target mask. The first property we'd like to maximize and the latter we'd like to minimize. I gave gamma two decimal accuracy in these tests (I also had to rerun some tests with extended gamma range as I noticed it maxed out the initial limits, now gamma range extends all the way to 1/20.00).

I first tested the basic arguments gamma, max_out and thr. The result after 312600 iterations (I used dynamic iteration count) was a pareto front with 516 different results (the red dots below).

https://i.postimg.cc/63qqzkVz/Stripe-Mask-Test13-2021-07-18-22-27-34-optimize-spea2-pop-100-mutcount-60-1-mutamount-0-3-0-01-cros.png

The maxCover means the percentage of target mask covered, and the maximum is 400 because we have 4 frames. The minOutside is the percentage of area outside of the mask, the max is 400 also.

The largest target percentage is about 354% but then also about 324% outside of the mask is selected which is way too much.

It's no fun checking each result one by one so I turned the results into a video, one result per frame. You can download it here (https://drive.google.com/file/d/1VsRYrt9E1mlU7lI3hiR1orArYoAg5JiB/view?usp=sharing). NOTE: the preview sucks, download the video for good quality.

Second test was running StripeMask twice and combining the masks. I tried combining them both as union and intersection and union turned out much better. The results are below:

https://i.postimg.cc/sxZXC5Nh/Stripe-Mask-Test12-2021-07-17-12-56-15-optimize-spea2-pop-100-mutcount-60-1-mutamount-0-3-0-01-cros.png

Results were better which was to be expected. Now the maximum coverage is about 387%. I have a comparison chart later in this post. Download a video of the results here (https://drive.google.com/file/d/1ckyzV6Tsdmgo1Agh7WennZ01E0JbE9FD/view?usp=sharing).

Next I tried adding the rest of StripeMask parameters blksize, overlap and comp but with a single StripeMask. Results were:

https://i.postimg.cc/NGW89DYZ/Stripe-Mask-Test15-2021-07-21-13-24-15-optimize-spea2-pop-100-mutcount-60-1-mutamount-0-3-0-01-cros.png

Video here (https://drive.google.com/file/d/1rlImnzezyQBM0EO1SY0PS9rHODUUwtrI/view?usp=sharing). This is already pretty good result, for example this is a result with about 356% mask covered and only 11% outside.

https://i.postimg.cc/zXvmvJzx/Stripe-Mask-Test15-x264-00.png

Finally I tried the previous StripeMask parameters and two masks combined.

https://i.postimg.cc/7ZNSRy7y/Stripe-Mask-Test16-2021-07-22-19-18-46-optimize-spea2-pop-100-mutcount-60-1-mutamount-0-3-0-01-cros.png

Video here (https://drive.google.com/file/d/1BL4BYMQCGSZzPj5JSodGv8P3R4YODIrc/view?usp=sharing). Results were still better but not by a whole lot.

Here are the first three runs together in the same chart.

https://i.postimg.cc/3NJb6ZfK/Stripe-Mask-Test-run-03-0.png

Pareto 0 is the basic run, pareto 1 added a second mask and pareto 2 has one mask but more StripeMask parameters.

And this is a comparison of the last two runs:

https://i.postimg.cc/s2dHyzHH/Stripe-Mask-Test-run-04-0.png

So adding another mask was not nearly as beneficial than when it was done with the basic parameters.

The results can be improved further for example by adding RemoveSmallSpots or ContinuousMask.

[sofakng]

Does anybody have an example of how I can use this with mpv + vapoursynth + SVP?

[Selur]

btw. any news regarding incoporating rife into FrameRateConverter?

[ChaosKing]

It looks like this could be the new standard in video interpolation:

https://github.com/uzh-rpg/rpg_timelens
https://www.youtube.com/watch?v=dVLyia-ezvo&feature=youtu.be
https://www.youtube.com/watch?v=G00A1Fyr5ZQ&feature=emb_title

[takla]

Cuda only. Yikes.

[real.finder]

Cuda only. Yikes.

it can be ported to c++ (for cpu and others) I think

[zorr]

That TimeLens-method relies on event data which is generated by custom hardware.

[Dogway]

As I see it it's only a dataset that you can use for AI based slomo splash sequences.

[takla]

That TimeLens-method relies on event data which is generated by custom hardware.

Imagine if in the future, streaming bandwidth for videos is cut in half or more, since frames will be reconstructed via hardware. (the stream provider sends an extra file for reconstruction to the client) simply amazing.

Eventually, I'd expect a similar thing to happen for resolution, too.

[Milardo]

Hi, does anybody have the current FrameRateConverter.avsi for the latest beta release? It only comes with vapoursynth script.