FrameRateConverter (Official)

[kedautinh12]

Hi, does anybody have the current FrameRateConverter.avsi for the latest beta release? It only comes with vapoursynth script.

Still use old .avsi, or if you want use in HBD can use here
https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/FrameRateConverter.avsi

[MysteryX]

Sorry had been travelling 6 weeks across Mexico.

It looks like this could be the new standard in video interpolation:

https://github.com/uzh-rpg/rpg_timelens
https://www.youtube.com/watch?v=dVLyia-ezvo&feature=youtu.be
https://www.youtube.com/watch?v=G00A1Fyr5ZQ&feature=emb_title

So this method isn't applicable here, correct? Requires extra hardware while recording.

[MysteryX]

Zorr, very interesting research! But after all this, it's still not clear at all, what are the optimal parameters when using 1 mask, and when combining 2 masks?

RemoveSmallSpots is a great idea, but searching about it returns nothing. Where do you find that plugin?

I wouldn't use the max overlap for performance reasons. I don't expect much quality difference between half-block size and blksize-1. Setting Comp to 5? If it's consistently better, then perhaps. As for BlkSize, it entirely depends on the video size, and is set to the same as the motion estimation blksize. It's 8, 12, 16, 24, 32, 48 or 64. No point in testing that with other values. Actually no point in testing it here, because it should be set based on motion estimation results.

[StainlessS]

but searching about it returns nothing. Where do you find that plugin?
https://forum.doom9.org/showthread.php?p=1944487#post1944487

[zorr]

Zorr, very interesting research! But after all this, it's still not clear at all, what are the optimal parameters when using 1 mask, and when combining 2 masks?

There is no one optimal result. The pareto front is the set of optimal results, each better than the others in some way. You have to use your own judgement as to which one of them suits you best. For example if you think that it's acceptable to have no more than 5% error rate (at most 5% of area outside of the stripes selected) then you can look for the result with largest maxCover that has minOutside less or equal to 20 (400*0.05 = 20), that would be 367.02933 19.190443 gamma=364 max_out=143 thr=19 blksize=16 overlap=15 comp=5 gamma2=105 max_out2=219 thr2=44 blksize2=25 overlap2=24 comp2=5) in the previous results. But I think it's easier to look at the video with all the results and choose one that looks like a good compromise. Even better would be to test at least some of those results with FrameRateConverter and see the resulting video.

RemoveSmallSpots is a great idea, but searching about it returns nothing. Where do you find that plugin?

S*ssS already provided a link to it, it's from a larger set of mask filters called Delta Restore. I got the idea from some old message of Didee.

I wouldn't use the max overlap for performance reasons. I don't expect much quality difference between half-block size and blksize-1.

Performance was not the focus of these tests but it should be considered, of course. I could add the runtime as another variable to optimize but then the results would be a set of points in three dimensions so that would be even more difficult to decipher. Perhaps there could be a runtime limit, anything slower than x milliseconds would be rejected.

Some people don't care about the runtime that much, perhaps you could add a preset for those that want maximum quality?

Setting Comp to 5? If it's consistently better, then perhaps.

It seems to be a very popular choice among the pareto front. And its impact on performance can also be tested just like with the overlap.

As for BlkSize, it entirely depends on the video size, and is set to the same as the motion estimation blksize. It's 8, 12, 16, 24, 32, 48 or 64. No point in testing that with other values. Actually no point in testing it here, because it should be set based on motion estimation results.

I don't see where the motion estimation (or even the video size) enters the equation. StripeMask doesn't take any motion vectors as input so it doesn't care about those. I believe the optimal blksize depends on the general size of the stripe patterns in the video, in this video it's about 18. It remains to be seen what the optimal size is in other videos.

By the way I did another test where I compared ContinuousMask and RemoveSmallSpots, I'll post those results next. :)

[MysteryX]

This does give pretty good resuts!

video = frc.StripeMask2(video, blksize=16, str=200, strf=100, thr=17, range=205, gamma=2.87)


Setting Comp=5 makes the image a lot noisier; if set, it seems to move the pareto front and everything needs to be configured differently.

Default Overlap is 1/4 (4) of blksize. Setting it to half (8) does give better results. Setting it to 3/4 (12) gives even better results in hard-to-detect edge cases. Setting to 15 gives even better results indeed. Rarely makes much difference but edge cases are more clearly detected. Adjusting between 1/4, 2/4 or 3/4 of blksize might be a reasonable tuning.

[MysteryX]

First of all, here's a new build with Range and Gamma parameters. (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta) I confirm: Range min doesn't matter, all that matters is the range between min and max, so we only need to specify RangeMax.

Second, there remains a difference between Avisynth and VapourSynth versions. VapourSynth converts to grey with ShufflePlanes, keeping the first plane intact. Avisynth uses ConvertToY. I believe it weights the colors to give a different output, considering red more visually important or something? Whatever it does, these 2 methods produce an output with a different brightness. Which will give inconsistent output between platforms. Which approach is best here, ShufflePlanes or ConvertToY? TODO: in VapourSynth, I'll need to convert to YUV in case it's in something else.

Third, I'm currently mostly testing VapourSynth. If you test in Avisynth with the current discrepancy, the results you'll get won't be fully valid for VapourSynth.

I've been having a hard time getting consistent results with the settings you came up with. Particularly the settings with weird extreme values, it works well in some places and less in others, thus I'd rather avoid extreme settings.

As for Gamma, higher (2.8) gives better output in some places... whereas lower (1.4) gives better output in other places, depending on the stripe shades. The very first frame has one part better defined with 2.8 and another part better defined with 1.4 ... so overall 2.2 is more consistent.

BlkSize depends on the particular video. The tests are for the general algorithm, not for a specific video, thus we can't include that in the variations. What I *could* do is merging 2 masks with different blksize so that patterns of various sizes get better detected.

Comp=5, from my tests, causes certain areas to be more clearly defined, whereas it misses other finer details. If merging 2 masks, perhaps it could be used with the larger block size.

I'm still unsure of the best settings to use, but I'm inclined to use gamma 1.8, 2.0 or 2.2 for single-pass and 1.4/2.8 for double-pass.

First need to deal with the inconsistency between ShufflePlanes and ConvertToY; how to do ShufflePlanes in Avisynth. Experts?

[Reel.Deel]

First need to deal with the inconsistency between ShufflePlanes and ConvertToY; how to do ShufflePlanes in Avisynth. Experts?

For regular 8-bit YUV you can use ConvertToY8/UToY8/UToY8/ to extract and YToUV (http://avisynth.nl/index.php/Swap) to merge. For 8-bit RGB there's ShowRed/Green/Blue (http://avisynth.nl/index.php/ShowAlpha) to extract and MergeRGB (http://avisynth.nl/index.php/MergeRGB) to merge. In AviSynth+ there's ExtractX/PlaneToY/ShowY/U/V (http://avisynth.nl/index.php/Extract) to extract and CombinePlanes (http://avisynth.nl/index.php/CombinePlanes) to merge. Anything in particular that you're trying to do?

[MysteryX]

For regular 8-bit YUV you can use ConvertToY8/UToY8/UToY8/ to extract and YToUV (http://avisynth.nl/index.php/Swap) to merge. For 8-bit RGB there's ShowRed/Green/Blue (http://avisynth.nl/index.php/ShowAlpha) to extract and MergeRGB (http://avisynth.nl/index.php/MergeRGB) to merge. In AviSynth+ there's ExtractX/PlaneToY/ShowY/U/V (http://avisynth.nl/index.php/Extract) to extract and CombinePlanes (http://avisynth.nl/index.php/CombinePlanes) to merge. Anything in particular that you're trying to do?

hum. There's no difference between ConvertToY and ExtractY, so that clarifies that. Now I'm comparing with ShufflePlanes, getting same output. Had issues in the past... not sure what it was. Must have been the TV-PC conversion methods that caused discrepancy.

In that case, VapourSynth and Avisynth should be consistent in theory. Haven't re-tested it yet.

[Reel.Deel]

hum. There's no difference between ConvertToY and ExtractY, so that clarifies that. Now I'm comparing with ShufflePlanes, getting same output. Had issues in the past... not sure what it was.

In that case, VapourSynth and Avisynth should be consistent in theory. Haven't re-tested it yet.

No difference in ConvertToY and ExtractY provided that the input clip is YUV, otherwise ConvertToY does an RGB to YUV conversion. ExtractY would just throw and error in that case.

Combined, Extract and CombinePlanes do everything that ShufflePlanes does. Although ShufflePlanes having arrays does make some task a bit easier, scripting wise.

[zorr]

I've been having a hard time getting consistent results with the settings you came up with. Particularly the settings with weird extreme values, it works well in some places and less in others, thus I'd rather avoid extreme settings.

That could very well be because only a few frames of one video was used for optimization - the settings will work very well for this video but probably not for others. Selur recently posted (https://forum.doom9.org/showthread.php?p=1951704#post1951704) a couple of interesting clips with stripe patterns - perhaps we should do another round of tests but take frames from multiple videos and see what settings work for all of them.

[MysteryX]

For Gamma, putting it higher or lower gets better results in specific scenes and worse in others -- so IMO linear light is best (2.2)

For Levels adjustment in Avisynth, I needed to add Coring=false. Without it, it was giving funky results rendering all the tests useless... I released a new version that sets coring=false. (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta)

This is giving the best results so far... marginally better than in v1
video = frc.StripeMask2(video, blksize=16, str=200, strf=100, thr=26, range=241, gamma=2.2)

Now I'm still struggling with slight difference between Avisynth and VapourSynth.

This pre-processing should give the same output

#VapourSynth
video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1.0 / 2.2, min_in=16, max_in=235, min_out=0, max_out=241)

#Avisynth
ConvertToY()
Levels(16, 1.0 / 2.2, 235, 0, 241, coring=false)


It's *almost* the same, brightness is the same, but white contrasts are different for some reason. If I copy/paste both clips into a photo editor, then the image looks the same, making me think it could be just an interpretation issue, but I manually set both to full-range Rec.601 in VirtualDub and still see a difference. Then when I run StripeMask, the output is slightly different. What could it be?

Finally, if doing 2-pass detection, to set a different block size for the 2nd pass... are the missing details generally smaller or larger stripes? Could multiply blksize by 1.25 or 1.5 for a 2nd pass. If setting comp=5, then larger block size probably makes sense. Unless it's more struggling with small details.

[poisondeathray]

Now I'm still struggling with slight difference between Avisynth and VapourSynth.

This pre-processing should give the same output

#VapourSynth
video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1.0 / 2.2, min_in=16, max_in=235, min_out=0, max_out=241)

#Avisynth
ConvertToY()
Levels(16, 1.0 / 2.2, 235, 0, 241, coring=false)


It's *almost* the same, brightness is the same, but white contrasts are different for some reason. If I copy/paste both clips into a photo editor, then the image looks the same, making me think it could be just an interpretation issue, but I manually set both to full-range Rec.601 in VirtualDub and still see a difference. Then when I run StripeMask, the output is slightly different. What could it be?


I get the same Y levels for both, checked with makediff . At first with colorbars, but also on a 0-255 8it420 perfect gradient

You can import avs script into vpy script using core.avisource.AVISource , or import vpy script into avs script using VSImport


clip = core.avisource.AVISource(r'8bit420_greyscale_gradient_256x256_utvideo.avi', pixel_type="YV12")
vpylevels = core.std.ShufflePlanes(clip, planes=0, colorfamily=vs.GRAY)
vpylevels = core.std.Levels(vpylevels, gamma=1.0 / 2.2, min_in=16, max_in=235, min_out=0, max_out=241)

avslevels = core.avisource.AVISource(r'levels_grad.avs')

d = core.std.MakeDiff(avslevels, vpylevels, planes=[0])
da = core.std.Levels(d, min_in=127, max_in=129, gamma=1, min_out=0, max_out=255, planes=[0])
da.set_output()

da.set_output()


where levels_grad.avs is


AVISource("8bit420_greyscale_gradient_256x256_utvideo.avi", pixel_type="YV12")
ConvertToY()
Levels(16, 1.0 / 2.2, 235, 0, 241, coring=false)


I can upload test video if you want
https://www.mediafire.com/file/0nk29m6sqnvak55/8bit420_greyscale_gradient_256x256_utvideo.avi/file

[MysteryX]

Considering copy/pasting it into a photo editor removes the visual difference, I can assume it's just a display issue?

If that's the case, what else could be the difference? The platform-specific code is very simple. It only does the range/gamma conversion and then calls shared code. VapourSynth (https://github.com/mysteryx93/FrameRateConverter/blob/master/Src/VapourSynth/StripeMaskVpy.cpp) and Avisynth (https://github.com/mysteryx93/FrameRateConverter/blob/master/Src/Avisynth/StripeMaskAvs.cpp) code. Nothing in there to justify any difference... Any idea?

[poisondeathray]

How are you viewing the "output" ? What application ?

Exactly what you do mean by "white contrasts are different" ? How are they different ?

Copy/pasting into photo editor means you've converted to RGB somewhere, so that introduces other variables, including how you converted it. Limited range equations conversion of YUV to RGB will clip data and all values Y <16, >235 will look the same in a photo editor.

MakeDiff compares Y channel directly, it should be more reliable , or at least is a direct measure of Y

[MysteryX]

I'm using VirtualDub for both Avisynth and VapourSynth. Ctrl+1 copies the output to clipboard.

Here's the full code for both.

LoadPlugin("FrameRateConverter.dll")
file="MotionTest2.mp4"
LWLibavVideoSource(file, cache=False)
ConvertToYV12(matrix="Rec709")
StripeMask(blksize=16, str=200, strf=100, thr=26, range=241, gamma=2.2)


import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
import FrameRateConverter as frc
import vapoursynth as vs
core = vs.get_core()

core.std.LoadPlugin(path=r'FrameRateConverter.dll')
file=r"MotionTest2.mp4"
video = core.ffms2.Source(source=file, format=vs.YUV420P8)
video = video.std.SetFrameProp("_FieldBased", intval=0)
video = video.std.SetFrameProp("_ColorRange", intval=0)
video = core.frc.StripeMask(video, blksize=16, str=200, strf=100, thr=26, range=241, gamma=2.2)
video.set_output()

I tried with a different video file that is more standard, the difference remains.

[poisondeathray]

I'm using 2.0 beta-6

The stripemask shape is slightly different between avs and vpy, but for the most part the corresponding values are the same. Is the shape difference what you are referring to ?

I used one of Selur's test clips, "Fensterladen.mkv" and used Lsmash for both avs and vpy to be consistent
https://drive.google.com/drive/folders/1hG4-vEDn4l_B0QTlsrUnyl8JrOAuRgBy


You can view pixel values with vapoursynth editor color picker; eg. so if G=200 in the vpy version, you can see that the corresponding "block" in the avs version is G=200

a.avs

LWLibavVideoSource("Fensterladen.mkv")
StripeMask(blksize=16, str=200, strf=100, thr=26, range=241, gamma=2.2)


vpy

import FrameRateConverter as frc
import vapoursynth as vs
core = vs.get_core()

video = core.lsmas.LWLibavSource(r'Fensterladen.mkv')
video = video.std.SetFrameProp("_FieldBased", intval=0)
video = video.std.SetFrameProp("_ColorRange", intval=0)
video = core.frc.StripeMask(video, blksize=16, str=200, strf=100, thr=26, range=241, gamma=2.2)

avs = core.avisource.AVISource(r'a.avs')

stack = core.std.StackHorizontal([avs, video])

stack.set_output()



The mask values seem binarized to 100 or 200 (or 0)

This is what the output of that stack script looks on frame 66
https://postimg.cc/jD6NvnBD

[MysteryX]

Ya. Output is similar but different. What am I missing?

[poisondeathray]

Ya. Output is similar but different. What am I missing?

Can you clarify what you're referring to specifically ?

Did you mean the difference in the general mask (areas included/excluded) ?

Or did you mean "It's *almost* the same, brightness is the same, but white contrasts are different for some reason" ?

Because I don't see the difference in brightness, and the gradient ramp test should detect that if there was a fundamental math difference between "levels" filter in avs vs. vpy

[Selur]

btw. I added another test clip called 'bicycle wheel' which shown a spinning animated wheel that RIFE with it's anime-Model can handle fine, but from the looks of it FrameRateConverter can't handle (just inserts dublicates).
->setting dctRe=2 helps, a lot, but result is still behind RIFE.

---

Also when using Vapoursynth version with 'stp=False':
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/FrameFilter/FramerateConverter/FrameRateConverter-x64.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
import havsfunc
import FrameRateConverter
# source: 'C:\Users\Selur\Desktop\bicycle wheel.mp4'
# current color space: YUV420P10, bit depth: 10, resolution: 840x760, fps: 29.97, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\bicycle wheel.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Selur/Desktop/bicycle wheel.mp4", format="YUV420P10", cache=0, fpsnum=30000, fpsden=1001, prefer_hw=0)
# making sure input color matrix is set as 709
clip = core.resize.Bicubic(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P10 to YUV444P8 for vsFrameRateConverter
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P8, range_s="limited")
# adjusting frame count&rate with FramerateConverter
clip = FrameRateConverter.FrameRateConverter(clip, frameDouble=True, preset="anime", blkSize=8, blkSizeV=8, maskThr=40, maskOcc=45, skipThr=25, stp=False, dctRe=1) # new fps: 59.94
# adjusting output color from: RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="709", range_s="limited")
# set output frame rate to 59.940fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=60000, fpsden=1001)
# Output
clip.set_output()
it fails with:
Failed to evaluate the script:
Python exception: local variable 'EMstp' referenced before assignment

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2242, in vapoursynth.vpy_evaluateScript
File "src\cython\vapoursynth.pyx", line 2243, in vapoursynth.vpy_evaluateScript
File "E:\Temp\tempPreviewVapoursynthFile15_54_12_915.vpy", line 31, in <module>
clip = FrameRateConverter.FrameRateConverter(clip, frameDouble=True, preset="anime", blkSize=8, blkSizeV=8, maskThr=40, maskOcc=45, skipThr=25, stp=False, dctRe=1) # new fps: 59.94
File "I:\Hybrid\64bit\vsscripts\FrameRateConverter.py", line 264, in FrameRateConverter
EMstp = havs.ChangeFPS(EMstp, newNum, newDen)
UnboundLocalError: local variable 'EMstp' referenced before assignment

Cu Selur

Ps.: Also added "jump_1440x1080i" QTGMC bob brings it to 50fps using RIFE looks fine, haven't found good values for FrameRateConverter so far.
PPs.: Is there any downside to use dctRec=2 and dec=2 aside from the speed drop? (haven't seen any example where it had a negative effect on the results, only none or positive effects)

[MysteryX]

Can you clarify what you're referring to specifically ?

Did you mean the difference in the general mask (areas included/excluded) ?

Or did you mean "It's *almost* the same, brightness is the same, but white contrasts are different for some reason" ?

Because I don't see the difference in brightness, and the gradient ramp test should detect that if there was a fundamental math difference between "levels" filter in avs vs. vpy
You're seeing that the input (after pre-processing) of StripeMask is the same for both yet the mask output is different. That's the problem. Most of the code is shared except pre-processing and parsing parameter values. Unless there's a bug in the way parameters are passed in either version?

Selur, the script isn't finished yet.

[MysteryX]

An idea that just came to my mind. Instead of just saying "values above thr will have value 200", what if I say "values between thr1 and thr2 will have values between 0 and 200"? Let's say thr1=17 thr2=20, value 17 gives mask 50, 18 gives 100, 19 gives 150 and 20 gives 200. Would that better represent details or not? Will have to think of how the lighter tones would affect the overall mask.

EDIT: Never mind this makes no sense. Threshold is to determine contrast lines; and from those lines, I detect patterns.

[MysteryX]

Found the problems and released beta 7. (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta)

In Avisynth, the new Gamma parameter was assigned to an int instead of double, and Coring wasn't passed to Levels because although the function had 7 parameters defined, I was then passing only 6 parameters to the call.

Now both versions produce the same output :D

[MysteryX]

Released beta 8. (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta)

Renamed StripeMask to StripeMaskPass and removed strf parameter.

VapourSynth script now contains function StripeMask(blksize, blksizev, str, strf). I think you'll be pleased with the results! It kicks v1 out of the water. Thanks Zorr for your research.


blksizev = blksizev or blksize
mask1 = clip.frc.StripeMaskPass(blksize=blksize, blksizev=blksizev, overlap=blksize//2+1, overlapv=blksizev//2+1, thr=29, range=241, gamma=2.2, str=str)
blksize *= 1.25
blksizev *= 1.25
mask2 = clip.frc.StripeMaskPass(blksize=blksize, blksizev=blksizev, overlap=blksize//2+1, overlapv=blksizev//2+1, thr=42, range=214, gamma=2.2, comp=5, str=str)


Perhaps it can be further tweaked, but it's working pretty well. Now need to look at the mask post-processing...

The changes will break the Avisynth script. That will be released later.

EDIT: updated the beta 8 package with proper post-processing. using output="stripe" now gives pretty good results!

[MysteryX]

I'm having an issue with VapourSynth distorting the colors; not related to my plugin, but simply loading the video clip. Compared to playing the source video, it's fine in Avisynth, but the color matrix (?) is wrong in VapourSynth but even setting _Matrix property doesn't fix it. Both of these don't work correctly. What a I missing?


video = core.ffms2.Source(source=file, format=vs.YUV420P8)
video = core.lsmas.LWLibavSource(file, cache=False)


Btw where is LSMashWorks syntax for VapourSynth documented? It took me a long while to find the correct function and had to dig through source code.

[ChaosKing]

Just for the function names (parameters will follow soon™) vsdb can be helpful https://vsdb.top/plugins/lsmas
Readme: https://github.com/AkarinVS/L-SMASH-Works/tree/master/VapourSynth

[kedautinh12]

For new ver here:
https://github.com/AkarinVS/L-SMASH-Works/blob/master/VapourSynth/README

[Selur]

I always use something like:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)

# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
to make sure color matrix, luma range and fps are set to the values I expect.

[MysteryX]

I see my _ColorRange was set to 0 instead of 1, now it opens fine. Great, now it shifts the colours again when I run FrameRateConverter(output="flow") ...

[Selur]

Using 2.0beta8 and:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/FrameFilter/FramerateConverter/FrameRateConverter-x64.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import havsfunc
import FrameRateConverter
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",fpsnum=25,format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting frame count&rate with FramerateConverter
clip = FrameRateConverter.FrameRateConverter(clip, frameDouble=True, blkSize=8, blkSizeV=8, output="flow") # new fps: 50
# adjusting output color from: RGB24 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 50.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=50, fpsden=1)
# Output
clip.set_output()
colors dont' seem shifted here.

[MysteryX]

I'm looking at Dogway's version. Output="diff" shows nothing. Masks are very different than they should be and the output is completely different. Output hasn't been compared with the base script for equivalence.

v2.0 is nearly working.

Quick question. How do I access the clip's _Matrix property (and format) to convert it back and forth correctly?

video = video.resize.Bicubic(format=vs.RGBS)
video = video.rife.RIFE(uhd=False)
video = video.resize.Bicubic(format=vs.YUV420P8, matrix_s="709")


and...... RIFE doesn't have any parameter to specify the frame rate??

[MysteryX]

Beta 9. Looks functional! (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta)

Test it and let me know if you find problems. It contains both VapourSynth and Avisynth scripts.

TODO:
- implement RIFE interpolation with format conversion to RGBS back-and-forth, then you can call FrameRateConverter(rife=True)
- add fullRange parameter in the Avisynth version.

[Dogway]

I'm waiting on 2.0 so I can finish my mix mod.
I normally compare the outputs to the T so they match, and on any bitdepth, this has been so with ExTools and other mods.
As for FrameRateConverter there were a few odd things:

Was this really necessary?
.mt_expand(mode= mt_circle(zero=true, radius=8))

This is utterly slow, even in ExTools is slow so I used an approximation, so yes is not mathematically exact, if it was me personally I would resize down (further) and use rad=4.

Another thing, FRC_GaussianBlur42(2.8).
Again the concept is good but the values represent nothing, you are blurring on the downscaled clip and hence breaking the blur linearity.
I made ex_GaussBlur() based on this, removed the explicit blur() and parametrized rad to respond to an approximated gaussian standard deviation, so again not mathematically exact to your output, but things make more sense. If you want to be mathematically accurate to a gaussian use vsTCanny() in mode=-1, but it will never match FRC_GaussianBlur42().

Other than that I don't recall changing anything else.

[poisondeathray]

How do I access the clip's _Matrix property (and format) to convert it back and forth correctly?



format=clip.format

Not sure about matrix, but doesn't really matter, because you're using the same one both ways . So if you randomly choose "709" it "reverses" when you go back and forth. Also if input clip is "unspec" or unassigned, it might cause problem so just picking one should work



RIFE doesn't have any parameter to specify the frame rate??

RIFE only does powers of 2, you need to also use mvtools2 or other for intermediate rates

There was talk on the official implementation page of eventually including other rates, but it hasn't materialized yet

[MysteryX]

Dogway, I'm getting entirely different mask output -- doesn't even compare. Wrong dependency perhaps?


FrameRateConverter(60, preset="slower", blkSize=16, output="auto", debug=True)


Using
FrameRateConverter-Dogway 1.4.2 (28-06-2021)
ExTools v2.7 (01-07-2021)
Utils-r41 v0.41 (2017-05-13)

RIFE only does powers of 2, you need to also use mvtools2 or other for intermediate rates

There was talk on the official implementation page of eventually including other rates, but it hasn't materialized yet
...

Well that's a "slightly" limiting factor.

Still, could very well double the frame rate before using the exact same frame blending.

[kedautinh12]

Why you don't test new ver??
Frame Rate Converter mix mod Version: 1.5.0 (06-09-2021)
ExTools v5.8 (18-09-2021)

[poisondeathray]

RIFE only does powers of 2, you need to also use mvtools2 or other for intermediate rates

There was talk on the official implementation page of eventually including other rates, but it hasn't materialized yet




Well that's a "slightly" limiting factor.

Still, could very well double the frame rate before using the exact same frame blending.

It's still beneficial in scenes where RIFE works well. I posted some examples, such as 23.976=> 47.952 RIFE => 59.94 MVTools2. The "distance" that mvtools2 has to interpolate in the 2nd step is smaller, and the end result has minimal/negligible artifacts. But if you use MVtools2 directly from 23.976 => 59.94, there are the usual artifacts

Having the option to use RIFE I think is still a plus overall. Also consider implementing a switch for model version (anime model, vs. 2.4, vs. 3.8 etc...)

[MysteryX]

Beta 10. (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta)

The moment you've all been waiting for: RIFE support. Set rife=1 to run it once, or rife=2 to run it twice on the frame blending clip used for artefact masking. This will reduce frame blending in artefact areas.

It however doesn't have as huge of an impact as one might think, but definitely helps in some frames. You can also lower the mask threshold to see more of the RIFE clip.

First running RIFE and then running FrameRateConverter on top of it would also be an option.

btw I'm unable to set gpu_id, how do I set it to my 2nd NVIDIA graphic card? Haven't found how to use list_gpu param either.

TODO: when preset="anime", set model=2

btw I had to comment this validation in the source code to avoid an error when doubling framerate twice. Any idea what this validation was for? (https://github.com/mysteryx93/FrameRateConverter/blob/master/Src/Common/ConvertFpsLimitBase.cpp#L16) That was copy/pasted from "somewhere" LOL (isn't that how doom9 works?)

[MysteryX]

Playing with it, couple of issues.

When I use RIFE in 2 separate VirtualDub windows, sometimes junk comes through (in artefact areas).

As for difficult frames marked as full-frame blending, RIFE isn't doing any magic so it's not giving a good output.

More tweaking will be required.

[MysteryX]

Good news. After keeping full frame blending on difficult frames, and simply adding Maximum() to the mask when using RIFE, it gives a pretty good result.

Here's a sample frame with Blend / Rife / 2x Rife

https://i.postimg.cc/hJLrXVB0/Blend.png (https://postimg.cc/hJLrXVB0) https://i.postimg.cc/R6vTKqCx/Rife1.png (https://postimg.cc/R6vTKqCx) https://i.postimg.cc/G8zxQr98/Rife2.png (https://postimg.cc/G8zxQr98)

Can do further tweaking on mask thresholds when using RIFE.

There are however some corner cases where RIFE gives worse results, such as quick spotlights. You might get 1 worse frame for 20 better frames.

[StainlessS]

MX,
Not totally sure, but think it was you who posted a K-Pop "Girl's Day - Female President" clip,
I loved that and have went out of my way to find more of GD. Also I found I liked "Itzy - Wannabee",
plus more of them too. but my favourite is probably "BlackPink - Kill this Love" [and their entire collection],
I also liked the "Making Of, Kill This Love" where they get showered with boulders [not the Boulder on this site]
during the explosion. Great stuff, and thanks for putting me on the this K-Pop thing, love it.

BlackPink - Kill This Love:- https://www.youtube.com/watch?v=2S24-y0Ij3Y&list=RD2S24-y0Ij3Y

EDIT: Yep I thought so, your triple image above this post is from "Female President":- https://www.youtube.com/watch?v=xF3MC8PWgJE

[MysteryX]

StainlessS, that's what the Natural Grounding Player is for, attuning to the energy of those videos. Although that software now has broken download feature and it will require a lot more work before the next version is ready. I recently published the whole database of videos here. (https://docs.google.com/spreadsheets/d/1jpjKgEDgcKxxzSdtDG2gXO59xMsDX_Yx/edit?usp=sharing&ouid=101904282551013923658&rtpof=true&sd=true) Long list, and great stuff in there. Try this one (https://www.youtube.com/watch?v=wCPZTjqvN0Q) and that one (https://www.youtube.com/watch?v=V2hlQkVJZhE).

[poisondeathray]

Yes, RIFE can sometimes produce worse results on seemingly "simple" content that mvtools2 breezes through without issues

I've been doing quite a bit of testing and there are frames where 2.4 does better job than 3.1 or 3.8. I'm finding 2.3 seems the best model overall on varied content (but it's not included in vapoursynth version - But YMMV as usual . Scene changes are not handled well either - probably the same underlying issue with those flashing lights

Also , often you can get a "clean solve" on some frames by pre manipulations such as turnleft , or flipping (then reverting the change post rife) ; so I asked zorr about zopti maybe figuring combinations out on a per frame basis

It would be pure awesomeness if an automatic tiered choice algorithm could choose what works based on least artifacts, then fallback on some of the masking, or full blending if none work . MVTools2 is much faster still, and if it works - great . That's my preference. But if that doesn't work, replace that frame with RIFE 2.4 , and if that doesn't work replace with xyz etc...

[MysteryX]

It would be pure awesomeness if an automatic tiered choice algorithm could choose what works based on least artifacts, then fallback on some of the masking, or full blending if none work . MVTools2 is much faster still, and if it works - great . That's my preference. But if that doesn't work, replace that frame with RIFE 2.4 , and if that doesn't work replace with xyz etc...
That's pretty much what I'm doing; except that RIFE doesn't provide any info as to what artefacts it produces. Full-frame blending or skipping is done based on the raw masks from MvTools. Which handles scene changes correctly.

If there remains specific situations causing trouble, the only way would be to have a specific plugin to scan for that problem, just like StripeMask is doing.

BTW can someone explain this. I run twice with output="raw" debug=true in VapourSynth, and although the raw mask visually looks the same, the raw value varies randomly. Why does Vpy MvTools have this "random" factor?

I confirm that model=1 gives better results. The random junk that sometimes appear when running multiple instances does make it harder to test. Also, VirtualDub process doesn't terminate correctly. It does have a few bugs that need to be rounded up.

[StainlessS]

Thanks for that MX, I'm aware of some of them.
Love this [Japanese] one from Kill Bill II [closing credits], "Urami Bushi" by Meiko Kaji:- https://www.youtube.com/watch?v=yT9zJ2V9lfw

EDIT: Didn't realize it but Meiko Kaji has serveral more tracks from Kill Bill I.

[MysteryX]

Beta 11 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v2.0-beta)

Added "rife" and "rifeanime" presets
Preset "rife" sets rife=2 and maskThr=80. If doubling framerate, rife is set to 1-pass.
Rife uses model=1 (v2.4)
Changed maskOcc from 105 to 125, and blending with .7 opacity instead of .4
Avisynth version now has FullRange parameter

Using diff mode with RIFE is counter-productive, so preset "rife" uses preset "slow".

Remains a weird randomness to the masks. Run the same clip twice (with preset="normal") and you'll get slightly different masks.

I created a thread for RIFE problems here. (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan/issues/7) Reported the MvTools randomness here. (https://github.com/dubhater/vapoursynth-mvtools/issues/53)

This now gives seriously good results!! Please test it and post your reports.

Pay particular attention to the occlusion mask. VapourSynth occlusion mask is a lot narrower than in Avisynth where it marks large areas. If you find that occlusions aren't being detected enough, yet show on the raw mask, let me know and we can tweak either maskOcc or the merge opacity. Also do a bit of testing about StripeMask as to whether it includes too much.

So please test these 3 areas and let me know whether it feels weak, just right or too strong.
- Occlusion mask
- Stripe mask
- Overall mask

[poisondeathray]

Thanks for the updates


Rife uses model=1 (v2.4)



That was for v1, which is loaded with a .dll
model: Model to use.

0 = rife-v3.1
1 = rife-v2.4
2 = rife-anime



v1.1 uses model_ver
3.1
3.5
3.8

You can check __init__.py

v1.1 is accessible when you use pip install method


To access all the models for the vapoursynth version, I've been both loading the old R1 .dll and importing the R1.1 py. I've been using stackvertical/horizontal to check versions, also other non vapoursynth versions to check other model versions


core.std.LoadPlugin(r'PATH\RIFE.dll') #RIFE-r1
from vsrife import RIFE as RF3 #RIFE r1.1

r24 = core.rife.RIFE(clip , model=1)
ranime = core.rife.RIFE(clip , model=2)
r31 = RF3(clip , model_ver=3.1) #3.1,3.5,3.8
r35 = RF3(clip , model_ver=3.5)
r38 = RF3(clip , model_ver=3.8)



Remains a weird randomness to the masks. Run the same clip twice (with preset="normal") and you'll get slightly different masks.

Did you check source filter, maybe try threads=1

[poisondeathray]

I can't get vapoursynth version to run, never seen this error before


File "FrameRateConverter.py", line 170, in FrameRateConverter
Blank = core.std.BlankClip(C.resize.Point(format=vs.GRAY8))
File "src\cython\vapoursynth.pyx", line 2067, in vapoursynth.Function.__call__
vapoursynth.Error: BlankClip: nodes foreign to this core passed as input, improper api usage detected


The py says Version: 2.0 (2021-09-18) beta 9 , not beta 11 - could it be wrong bundled version ?

EDIT - NM , some "weirdness" going on , but you need to specify output mode and blkSize, but sometimes that error message comes back... weird . You have to close vsedit and restart and message goes away. It's really flaky right now, maybe it's related to that weird mask randomness. Similar thing happens in vdub2 - 1st time works, but if you push f2 (reopen video file) that error message pops up

Selur had the same issue a few pages back
https://forum.doom9.org/showpost.php?p=1946115&postcount=402

Might be related
https://forum.doom9.net/showthread.php?p=1801028#post1801028
https://forum.doom9.net/showthread.php?p=1801033#post1801033

If I edit the FrameRateConverter.py to

from vapoursynth import core


instead of

core = vs.get_core()

the issue goes away

[MysteryX]

Ah yes there was that. Probably related to the way I'm playing with Core instance in StripeMask code. I remember doing a hack I wasn't sure about.

vsrepo gives me v1.0 I believe? Where do I find that v1.1?

[ChaosKing]

I belive he means https://github.com/HolyWu/vs-rife
These are 2 different plugins

vsrepo currently only has https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan
Not sure if vs-rife is worth adding if you still have have many post install steps.