SVP-like frame interpolation?

[StainlessS]

EDIT: Note, John's script keeps every even output frame same as input, at double rate.

Oops, no it does not. John's original script did but manolito left that bit out.

Here John's script that I was talking about:- https://forum.doom9.org/showthread.php?p=1788725#post1788725

And here an alternative which does same for any integer multiple rate, eg if multiply by 3, then every third frame is an original frame.
MulRate():- https://forum.doom9.org/showthread.php?p=1789461#post1789461

[MysteryX]

One issue I'm seeing compared to Interframe is strange warping on text objects -- and that's something that's recurring throughout the whole videos.

Quite a lot of artifacts too, in some scenes.

By truncating my script and removing whatever runs after jm_fps, I'm able to play it at a decent speed and visualize the artifacts at about half speed. Anything else won't show me where it fails. I'm concerned, however, about recurring artifacts such as on texts and subtitles.

Perhaps the jm_fps function would also benefit from exposing a few parameters to tweak settings.

[StainlessS]

I did further tests in YV24, and strangely, it generates a LOT of strong artifacts!! This seems more like a bug.
http://screenshotcomparison.com/comparison/204834

Yip, seems like a bug to me too. Post a few frames that produce same as your posted image, Pinterf is likely to want to see that.

[MysteryX]

Here's a clip. Encoding speed is about the same using this script in YV24 vs Interframe in YV12, in my case.

Here's a sample clip. (https://mega.nz/#!7VpRARjI!exq4QdhnCC0zGEuuuEe6udwruUev_lS9La6RniscSqI)

mvtools2 *does* support 16-bit processing with Pinterf's version; but I don't believe it supports stack16 format (which I'm still currently using)

[MysteryX]

Another question. How does SVP's artifact masking feature work?

At a minimum, it would make sense to keep that feature and make it revert to frame blending on the partial areas it detects, before going through the video manually and marking whole frames.

[johnmeyer]

How do you play it at 2x speed? In order to see the result, all the frames must be calculated. You won't reach 2x speed unless the processing is faster than real-time, which it won't if it's slower than SVP.You are assuming a playback method which achieves faster than real time playback by dropping frames. However, a good NLE can play frames at a faster speed, spitting out 30, 40, 50 or more fps. In Vegas, this can be stretched even further if you select one of the spatially degraded preview resolutions. By moving fewer pixels around, it permits playback at even higher speeds before frames must be dropped to keep up.

I'm not sure what you meant by your reference to SVP because when I perform the operation of cutting between the motion estimated version and the frame blended version, both those videos have already been rendered, i.e., I am not frame serving through an AVISynth script into Vegas via SVP, or anything else.

[kolak]

I have made a quick test to compare Interframe(preset="smooth") with jm_fps. This is part of a script that does more processing, and does an extra frame double which makes the changes more visible. This is a test on very bad quality content to see how it behaves.

http://screenshotcomparison.com/comparison/204832

http://screenshotcomparison.com/comparison/204833

The difference is huge. It's much better than I expected. Worth investigating some more. Plus, it allows processing in YV24.

Can the jm_fps script work in 16-bit, using either DitherTools hack or native AVS+ format?

I did further tests in YV24, and strangely, it generates a LOT of strong artifacts!! This seems more like a bug.
http://screenshotcomparison.com/comparison/204834

Your source quality is terrible :)
Mvtools are definitely better, although waaaaay slower. Looks like SVP does many quality v speed shortcuts.

[kolak]

I believe Interframe's anti-artifact results in disabling frame interpolation when it detects artifacts, which isn't ideal either.




There are different modes for this. There is one which forces only interpolated frames, not original or blended. Mode 3 if I remember well.
There is no trial for Tachyon. License is something like 20K£.

[MysteryX]

You are assuming a playback method which achieves faster than real time playback by dropping frames. However, a good NLE can play frames at a faster speed, spitting out 30, 40, 50 or more fps. In Vegas, this can be stretched even further if you select one of the spatially degraded preview resolutions. By moving fewer pixels around, it permits playback at even higher speeds before frames must be dropped to keep up.

I'm not sure what you meant by your reference to SVP because when I perform the operation of cutting between the motion estimated version and the frame blended version, both those videos have already been rendered, i.e., I am not frame serving through an AVISynth script into Vegas via SVP, or anything else.
Ah, you first generate the 2 videos, and THEN edit the 2 timelines of regular AVI files. That makes sense, and yes would allow playing in real-time -- except that the files must be generated first which takes more time.

Your source quality is terrible :)
That's why I'm trying to improve them. I have a bunch of such VCDs, all produced between 2002 and 2006. Unfortunately, in terms of content, I haven't seen anything produced after 2006 from Thailand that compares to these older "classic" releases. And even today, in Thailand, many people are still playing these songs from 2002 to 2006.

[MysteryX]

I'd see a few improvements to the jm_fps script.

1. First, exposes a few arguments to tweak settings (such as block size).

2. When doubling frame rate, keep source frames

3. Implement SVP/Interframe's artifact masking -- because often the artifacts are small edges of objects, and it would be a waste to not only take the time to mark those frames one by one, but also to waste the whole frame for a partial object that often could be fixed automatically. Of course not all artifacts will be detected in that way, but if half are remove, that's half less work left to do -- and done better, with partial masks.

I'm not familiar enough with mvtools (I know nothing about how it works) to implement those.

[StainlessS]

You could play with settings via the posted code I linked as MulRate(), or ChangeFrameRate():- https://forum.doom9.org/showthread.php?p=1789461#post1789461
After doing so, then decide which external settings you wish to keep and remove the rest.
The big list of args to ChangeFrameRate() were intended for experimentation purposes, and will be only a few milliseconds slower before frame serving starts.

Might also be an idea to detect almost identical FrameRates, eg 29.97 and 30.0 FPS and do an AssumeFPS(...,sync_audio=true).ResampleAudio(OriginalSampleRate)
instead. Also cope with similar 29.97 -> 60.0 via AssumeFPS and integer DoubleRate() to 60.0 FPS, keeping alternate original frames if required.
EDIT: Code to keep original frames where integer multi-rate embedded as nested function within MulRate() function.

Function MulRate(clip c,Bool "Blend",Bool "Flow",int "mul",Bool "ILeave",
\ Bool "Prefilter",Bool "Chroma",Bool "Chroma1",Bool "Chroma2",Int "BlkSize",Int "Recalc", Int "OverLap",Int "OverLap1",Int "OverLap2",
\ Int "SearchParam",Int "SearchParam1",Int "SearchParam2",Int "thSAD1",Int "thSAD2",Float "Flow_ml",Int "Block_Mode") {
Function MulRate_EvS(String s,int n,int mul) {
s="SelectEvery("+String(mul)+","+String(n)+")"+((s!="")?","+s:s)
# RT_DebugF("s='%s'",s,name="MulRate: ")
return (n>1)?MulRate_EvS(s,n-1,mul):s
}
c
mul=default(mul,2)
Ileave=Default(Ileave,True)
ChangeFrameRate(Blend,Flow,c.FramerateNumerator*mul,c.FramerateDenominator,Prefilter,Chroma,Chroma1,Chroma2,BlkSize,Recalc,
\ OverLap,OverLap1,OverLap2,SearchParam,SearchParam1,SearchParam2,thSAD1,thSAD2,Flow_ml,Block_Mode)
(ILeave&&mul>=2) ? Eval("Interleave(c,"+MulRate_EvS("",mul-1,mul)+")") : NOP
Return Last
}


EDIT: Fragment of code to detect almost identical framerate, that I use in some scripts.
May not be best available for the purpose, but what I use (the subject tends to result in unresolved arguments on forums).


Function IsClose(Float a, Float b, Float threshold) {
threshold=Abs(threshold)
return ((abs(a-b)+threshold) <= threshold*2.0)
}

[MysteryX]

SVP/Interframe has different anti-artifact strengths. The downside of setting it higher is that it disables frame interpolation wherever it detects potential artifacts.

If we fallback to frame interpolation, then the downside of such artifact removal is lesser and it can safely be set higher.

As for cutting/merging frames using different methods, that's exactly what SVP/Interframe does with artifact masking, and it does give a good result.

2 areas of improvement with artifact masking
- Fallback to frame blending
- blending of both methods on frames adjacent to where masking occurs

[MysteryX]

How is this artifact removal script? It already supports frame blending to fill the masks.
http://avisynth.nl/index.php/YFRC

[poisondeathray]

2. When doubling frame rate, keep source frames



Having the option to do either is a good idea.

Source frames are kept by default for mvtools2 when you have integer framerate multiples. Pros/cons eitherway. Conventional thinking or "knee jerk" reaction would say keep source frames... but...

Interpolation generally creates blurrier , lower quality frames . On some types of material (eg. high quality source with lots of details, or grainy sources, or noisy sources, ) , the difference is very noticable and results in a "strobing" or flickering quality . This difference is farther amplified in scenarios where the goal is slow motion (longer display time per frame, you "see" the differences more easily) . In those types of scenarios resampling all frames is sometimes preferrable. On lower quality source, or ones with low detail , the difference between frames is not as noticable.

[MysteryX]

Totally agree. I hate this flickering effect.

I also want to point out that although I understand the skepticism with automatic artifact removal, I don't think it's a good idea to not use it. There are lots of small artifacts that can be detected and removed that we just can't do by hand. We'd replace whole frames while it can replace only the affected areas. We still probably have to review manually afterwards, but we want to remove as much as possible first. Manual review should be a quick final step; not the bulk of the work.

[poisondeathray]

I also want to point out that although I understand the skepticism with automatic artifact removal, I don't think it's a good idea to not use it. There are lots of small artifacts that can be detected and removed that we just can't do by hand. We'd replace whole frames while it can replace only the affected areas. We still probably have to review manually afterwards, but we want to remove as much as possible first. Manual review should be a quick final step; not the bulk of the work.


I agree.

Certainly having the option to use "automatic" artifact handling with various algorithms or approaches is a good idea. Sometimes it works great. And if it can reduce the amount of manual work, that's awesome and a "win". But there are cases where it produces worse results (you have a halo effect of blur) , or it misses areas, or messes up good areas - and you actually end up having to do more work than not using it at all . This is where compositing comes into play and you combine layers done with different settings/iterations with masks. You NEED human guidance for this. No way around it.

As much as it would be nice for everything to work automatically with as little work as possible, there are situations that require some user guidance and manual work - if you want good results. (Maybe my definition of "good result" is different than what some people are using). Commercial plugins are a good an example of this - you can plug in motion tracking data, animated splines to guide motion estimation. You use accurate rotoscoped mattes to separate foreground objects and object boundaries so you reduce the edge morphing artifacts (not just "blur" like svp) . But this requires work and human intervention (ie. time). But in "automatic" mode, they are about the same as mvtools2 or svp.

[chainik_svp]

kolak
SVP made to many speed shortcuts so quality is not as good.

The only one I could think of is "adaptive search radius".
However I found large "adaptive" radius to be better than fixed one especally on regular structures like window blinds. And you can easily turn it off and use good old fixed search raduis.


They find problematic areas and then replace them with frame blended or nearest frame using making and feathering.

https://forum.doom9.org/showthread.php?t=172688#post1741208

and this's what SVP already does (much simplier of cause ;))


The result is a mixed clip: frame doubling on "difficult" scenes (fast moving objects) and frame interpolation on "easy" scenes (slow panning for example).

and this's called "adaptive interpolation mode" in SVP

MysteryX

I have made a quick test to compare Interframe(preset="smooth") with jm_fps. ... The difference is huge.

Yeah, probably because they use completely different options. Don't you want to compare them using the same options? :)

[videoFred]

and this's what SVP already does (much simplier of cause ;))


The result is a mixed clip: frame doubling on "difficult" scenes (fast moving objects) and frame interpolation on "easy" scenes (slow panning for example).

and this's called "adaptive interpolation mode" in SVP

How do I set this "adaptive interpolation mode" in Interframe() ?

Fred.

[kolak]

kolak
SVP made to many speed shortcuts so quality is not as good.

The only one I could think of is "adaptive search radius".
However I found large "adaptive" radius to be better than fixed one especally on regular structures like window blinds. And you can easily turn it off and use good old fixed search raduis.


They find problematic areas and then replace them with frame blended or nearest frame using making and feathering.

https://forum.doom9.org/showthread.php?t=172688#post1741208

and this's what SVP already does (much simplier of cause ;))


The result is a mixed clip: frame doubling on "difficult" scenes (fast moving objects) and frame interpolation on "easy" scenes (slow panning for example).

and this's called "adaptive interpolation mode" in SVP

MysteryX

I have made a quick test to compare Interframe(preset="smooth") with jm_fps. ... The difference is huge.

Yeah, probably because they use completely different options. Don't you want to compare them using the same options? :)

I've used svp a lot, but quality of current mvtools is better (definitely less artefacts).
SVP is way faster of course which is great. Adaptive mode is not the same as I was describing. It switches mode for whole frames which is not the same as Tachyon's fallback. Alchemist uses same way as svp if I'm correct.
I'm not looking for frame doubling, but fps conversion (doubling is very specific case).

Are you saying I should get about the same quality. You can't match setting as they are different compared to mvtools.

[chainik_svp]

"Current mvtools" is exactly the same it used to be 6 (7?) year ago.

What mvtools option you can't match to SVP? (except for "dct", SVP works with "dct=5", which is SATD, by default)

[MysteryX]

Yeah, probably because they use completely different options. Don't you want to compare them using the same options? :)
I compared the highest settings preset with SVP/Interframe, compared to jm_fps's script (which honestly I don't know what it does). I don't know where the difference comes from, but there's a very considerable difference.

How can I get the same result with SVP/Interframe?

Then mvtools has the benefit of supposedly supporting YV24 which you didn't care to implement; but then YV24 processing is buggy :)

Johnmeyer talked about some settings and features that were stripped out of SVP, it would be good to hear more specifics on that.

[chainik_svp]

How can I get the same result with SVP/Interframe?

By using the same options (https://www.svp-team.com/wiki/Plugins:_SVPflow), obviously ;)
Ensure that block sizes, search type(s), penalties and others are the same. And don't forget about defaults that can be different between mvtools and svp for some options... :D

[johnmeyer]

If some of you want to try out SVP, Interframe, MVtools2, the Yushko Frame Rate Converter script (with masking that attempts to suppress artifacts), then I suggest you try a real torture test.

All of these tools, as well as the expensive commercial equivalents, fail for two reasons: they cannot predict objects that are "revealed" when a foreground object moves across the frame in front of them; and they have trouble with motion of vertical objects, especially when they don't conform to the general motion in the frame (panning across a picket fence is a nightmare, and the legs of people walking in front of the camera often "break" grotesquely). All of these problems are accentuated at lower frame rates because the temporal gap between frames is larger, and the motion estimation algorithm has to make bigger assumptions.

My test clip is at 14.985 fps, exactly half the 29.97 NTSC rate. Even though your software may report it as interlaced, I assure you that it is 100% progressive: it is from my own film transfer of a 1940 Flint Michigan parade. If needed, add a line to your AVISynth script to force the software to treat it as progressive, although if you don't, nothing bad will happen.

Here are the things likely to show problems: the waving flags, the crowd getting "revealed" as floats cross in front of them; the rotating spokes on the old cars; the legs of the marching soldiers; the bayonets on the shouldered rifles as the marchers make a left turn; the drummer's drumsticks, and most of all, the moose antlers at the end. Good luck with those. They represent the ultimate challenge.

Here's the clip:

Flint Michigan Parade Clip (14.985 fps; progressive; DV format) (https://www.mediafire.com/?cb739wsyzws3z53)

P.S. If you want to see what this clip looks like when I used MVTools2 to increase fps to 29.97, here is the YouTube version of the all the film I transferred from this event. I increased the frame rate because it makes the film easier to watch, even though it introduces a LOT of artifacts. I did not spend the time to do the manual fixup that I described earlier, because this was a non-paying job for an old friend.

1940 Flint Michigan parade (https://www.youtube.com/watch?v=t8HjRN0rw5M)

I just watched it again, and it reminds me of why I always recommend to those who are doing their own film transfers and restoration that they NOT use motion estimation and instead simply use pulldown, even if the resulting video looks jerky.

[poisondeathray]

John's parade clip really is "torture", because it has many of the issues where you expect problems with motion vectors and interplation

John mentioned some of these, but common "prototypical" issues that predispose to "failure" include:

1) occlusions. Ie. object layers, foreground , background, z-depth crossing over one another (e.g. walking, legs crossing)
2) repeating patterns ("picket fence")
3) object deformation (e.g. bouncing ball, waving flags in john's video) , the algorithm can no longer accurately track because it "looks" like a different object
4) rotational movements, global and object axial. Object axial is especially bad, because textures are rotated away (different faces, effectively a different object)
5) motion blur - you cannnot distinguish between object boundaries as accurately. Where does one object edge begin or end?
6) compression artifacts
7) illumination changes within a scene
8) complex motions, e.g speed (long "distance" betwen frames or low frame rate), or non standard motion paths, or "random" motions. Camera motion vs. object motion. perspective changes



Understanding why something fails is the first step in actually doing something about it.

"Covering up" bad results is one approach; But I'm more interested in higher quality, address the underlying problem in the first place approach. This is for non realtime scenarios , like the OP wanted in the first post.

The underlying problem for issues like occlusions is the inability to delineate boundaries of objects. As one object crosses over another, or background objects, the vectors are no longer accurate. So one way around this is to guide the vectors or edit the optical flow data. Commercial applications use motion tracking (track points) , and mattes to indicate object layers and edges, to help guide the motion estimation . This isn't "covering up" a bad result, this is helping to get the interpolation right in the first place. There is user input , but it's not entirely "manual" the way some people think it is. You don't manually create masks per frame, every frame - you might only have to do every nth frame, and minor adjustments because of mask interpolation) , and there is motion tracking to assist with rotoscoping , mask tracking, as well as semi automated tools like rotobrush which is basically rototracking

For some types of complex motions, such as complex camera motions, there are "camera solves" which basically track the camera motion. ie. You can separate the camera motion from object motions to help simplfy the interpolation. You need rock stable VFX tracking for this, not something like deshaker or mercalli. You add back in the camera motion once everything is fixed and composited back together

So we start with a more simple, prototypical "fail" scenario. Low level tests are useful in understanding what the problem is, what parameters to tweak, developing strategies to improve. This is a synthetic low level test. Lagarith RGB . A repetitive checker background, simple object rotation, but no object axial rotation, or deformation. The goal is to double the frames. What settings mvtools2 or svpflow or other would you use ? It's a simple solve with 1 matte for commercial plugins, and this is basically automatic when you use a motion tracker - you draw a white circle and attach it to the motion track data. That luma matte (or "mask" in avisynth terms) input to the plugin is what helps in assisting the motion estimation. For matte generation there are free tools too , with motion trackers such as blender, natron, nuke (non commercial version).
https://www.mediafire.com/?ocs0abw49wlj4fo


I asked this before in this forum , you can visualize mvtools2 mv's with mshow(), but there is no way that I know of to re-input the edited data back in. But that would be one way to improve results. Some commercial applications can manipulate input vector fields directly and almost all of them have matte inputs (which indirectly assist with increasing mv accuracy) . Or is there a way to use mattes (or masks) in avisynth to assist mvtools2 motion estimation ? (ie. not just as an alpha channel or layer overlay) . But how can avisynth/mvools2 interpolation be improved using commercial approaches like mattes ? Things like multiple inputs, splines, mattes, motion track points are what separate commercial "pro" interpolation tools but I'd like to see some of them adapted for avisynth but I don't see a way.

A free / open source example editing motion data to improve results is slomovideo .
https://www.youtube.com/watch?v=aLtIvEiDD2k

[MysteryX]

Improving the failed areas is one topic. Improving the successful areas is another topic. Why do I get a much clearer and better defined image using jm_fps?

[poisondeathray]

Why do I get a much clearer and better defined image using jm_fps?


First thing is make sure you seek linearly with svpflow . Go back about 30 frames and advance forward. It's like dirt removal or srestore - non linear seeks cause inconsistent results , typically lots of blurring

But what settings are you using for svpflow ?

It looks like jm_fps is using mrecalculate and removegrain prefilter. svpflow doesn't have mrecalculate , it uses refine. Did you try matching the other settings ?

[johnmeyer]

That's a great post poison!

Motion tracking/rotoscoping is definitely the only way to deal with some of these issues. That technology lets you define real objects, like legs, and then track those, rather than simply -- and stupidly -- just arbitrarily breaking up each frame into blocks of pixels (4x4, or 8x8, or 16x16, etc.) and then tracking blocks which really have nothing to do with the actual content of the video.

MysteryX's states that he gets better results with some script than others. He makes it sound like he always gets better results with jm_fps, but I'll bet a few dollars that, on other clips, he would get better results with some other approach. It all depends on the nature of the specific scene you use.

As I have said already in this thread, block size is the single most important thing to vary if you want to optimize the results for a given clip. Given that this is true (at least in my experience), I have what might be an interesting idea:

Would it make sense to devise a script which provided the ability to alter the block size, scene by scene, and then choose which one to use? It would certainly be trivial to write a script which simultaneously created 4x4, 8x8, 16x16, and 32x32 results. The "trick" would be devising good enough artifact detection to choose between them. My sense is that, if such detection could be written, the result would be more pleasing than using similar detection to try to fix a bad result using masks. I haven't tried the script posted here which does that, but I have very low expectations that it would consistently produce a result that was much better than a script without it, especially since others have already reported that sometimes it actually makes things worse (halos and blurring is what I think one person reported).

[kolak]

If some of you want to try out SVP, Interframe, MVtools2, the Yushko Frame Rate Converter script (with masking that attempts to suppress artifacts), then I suggest you try a real torture test.

All of these tools, as well as the expensive commercial equivalents, fail for two reasons: they cannot predict objects that are "revealed" when a foreground object moves across the frame in front of them; and they have trouble with motion of vertical objects, especially when they don't conform to the general motion in the frame (panning across a picket fence is a nightmare, and the legs of people walking in front of the camera often "break" grotesquely). All of these problems are accentuated at lower frame rates because the temporal gap between frames is larger, and the motion estimation algorithm has to make bigger assumptions.


I can confirm those findings. Exactly same issues which I've seen for any tool I've tried. I've done fps conversion on 500 hours of footage.

[kolak]

By using the same options (https://www.svp-team.com/wiki/Plugins:_SVPflow), obviously ;)
Ensure that block sizes, search type(s), penalties and others are the same. And don't forget about defaults that can be different between mvtools and svp for some options... :D

Tried changing many settings.
mvtools is simply producing less artefacts. svp quite often has double edges on objects (even on not difficult scenes).
Also- older (much older) versions of svp dlls actually produced better results. The newer version the more artefacts :)

You are saying that svp should be as good as mvtools? I would like to get match, as svp is way faster.

[kolak]

Would it make sense to devise a script which provided the ability to alter the block size, scene by scene, and then choose which one to use? It would certainly be trivial to write a script which simultaneously created 4x4, 8x8, 16x16, and 32x32 results. The "trick" would be devising good enough artifact detection to choose between them. My sense is that, if such detection could be written, the result would be more pleasing than using similar detection to try to fix a bad result using masks. I haven't tried the script posted here which does that, but I have very low expectations that it would consistently produce a result that was much better than a script without it, especially since others have already reported that sometimes it actually makes things worse (halos and blurring is what I think one person reported).

ffmpeg motion estimation new filter has adaptive block size switching, if I'm correct, but results are not very good.

[chainik_svp]

Why do I get a much clearer and better defined image using jm_fps?

the thing is - no one will tell
in mvtools's world even a little change of some "secondary" option lake "lambda" or "pnew" can change the vectors field significantly


lets compare the magic "jm_fps" w/o using pre-filter to see if it really helps ;)


> svpflow doesn't have mrecalculate , it uses refine.

it's the same thing
MRecalculate can switch from any block size to any other block size while SVP's "refine" can only split a block into four smaller ones, but this's exactly what we want here

[MysteryX]

I'm just coming back from holidays. No further updates on this since?

I'd be curious to know where the difference of quality comes from.

[MysteryX]

I've done a bit of progress on this. I've taken both scripts and stripped it down to the actual scripts being generated.

Here I use Interframe with Tuning="Smooth". RemoveGrain has a very minimal effect and isn't the cause of the difference.

All I can do at this point is contemplate the huge difference between both scripts. The 2 functions' parameters are completely different so it's difficult to translate one with the other. Perhaps someone else can chime in from there.

Btw, does MVTools2 allow implementing artifact removal in the same way as SVP?


function InterFrameProcess(clip Input) {
SuperString = "{scale:{up:0,down:4},gpu:1,rc:false}"
VectorsString = "{block:{w:8,overlap:2},main:{search:{distance:0,coarse:{distance:-10,bad:{sad:2000}}}},refine:[{thsad:250}]}"
SmoothString = "{rate:{num:60,den:1,abs:true},algo:23,mask:{area:150,area_sharp:1.2},scene:{blend:true, mode:0}}"

# Make interpolation vector clip
Super = SVSuper(Input, SuperString)
Vectors = SVAnalyse(Super, VectorsString)

# Put it together
smooth_video = SVSmoothFps(Input, Super, Vectors, SmoothString, url="www.svp-team.com", mt=2)
smooth_video
}

function jm_fps(clip source) {
fps_num = 60
fps_den = 1

prefiltered = RemoveGrain(source, 22)
super = MSuper(source, hpad = 16, vpad = 16, levels = 1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = 16, overlap = 4, search = 3, dct = 0)
forward = MAnalyse(superfilt, isb = false, blksize = 16, overlap = 4, search = 3, dct = 0)
forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

return out
}

[MysteryX]

Here's YFMC that implements artifact removal, but only supports frame doubling.

function YFRC(clip clp_Input, int "BlockH", int "BlockV", int "OverlayType", int "MaskExpand")
{
#Yushko Frame Rate Converter 2x
#ColorSpace: YV12
#OverlayType: 0 - soft (blend), 1 - hard (SelectOdd)

clp_input = ConvertToYV12(clp_input) # script produce YV12 colorspace (3 times faster than YUY2!!!)
clp_Super = MSuper(clp_Input.blur(1), chroma=true, pel=2) # blur(1) - smooth edges for better analysing
ox = clp_Input.width()
oy = clp_Input.height()
fps_num = FrameRateNumerator(clp_input)*2 # Numerator , always 2X
fps_den = FrameRateDenominator(clp_input) # Denominator, always same as input clip

fps_num = ((fps_num==0||fps_den==0)) ? FramerateNumerator(clp_input)*2 : fps_num
fps_den = ((fps_num==0||fps_den==0)) ? FramerateDenominator(clp_input) : fps_den
MaskExpand = default(MaskExpand, 1) # 1 or 2
OverlayType = default(OverlayType, 0) # 0 - like ConvertFPS (blend); 1 - like ChangeFPS (strong);
blendSOFT = (OverlayType==0) ? clp_Input.ConvertFPS(fps_num, fps_den).SelectOdd() : DeleteFrame(clp_Input, 0)
BlockH = default(BlockH, 16) # use 8 for 320x240 (WEB); 16 for 720x576 (SD); 32 for 1280x720 (720p HD); 32 for 1920x1080 (1080p HD)
BlockV = default(BlockV, 16) # use 8 for 320x240 (WEB); 16 for 720x576 (SD); 32 for 1280x720 (720p HD); 32 for 1920x1080 (1080p HD)
blendHARD = DeleteFrame(clp_Input, 0) # SceneChange detection

bw1_vec116 = clp_Super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=true , chroma=false, search=1, searchparam=1, truemotion=true, lambda=2000, global=true, dct=0, divide=2)
fw1_vec116 = clp_Super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=false, chroma=false, search=1, searchparam=1, truemotion=true, lambda=2000, global=true, dct=0, divide=2)
ErrorMask16L = MMask(clp_input, bw1_vec116, kind=1)
ErrorMask16R = DeleteFrame(MMask(clp_input, fw1_vec116, kind=1), 0)
ErrorMask16 = Overlay(ErrorMask16L, ErrorMask16R, opacity=0.5, mode="Lighten")#.ColorYUV(gain_y=256)
SceneChange = MSCDetection(clp_input, bw1_vec116, thSCD2=130)
FPSconverted16 = clp_input.MFlowFps(clp_input.MSuper(levels=1), bw1_vec116, fw1_vec116, num=fps_num, den=fps_den, blend=false, mask=0) #mask=0 - doesn't matter what mode is

CircleExpand = mt_circle(radius=MaskExpand, zero=true)
CircleInpand = mt_circle(radius=1 , zero=true)
ErrorMask16 = ErrorMask16.BicubicResize(Round((Ox/BlockH)/4)*4, Round((Oy/BlockV)/4)*4)
\ .mt_expand(mode=CircleExpand).mt_inpand(mode=CircleInpand).mt_binarize(64).Blur(1).BicubicResize(ox, oy)

ClipToReturn = mt_merge(SelectOdd(FPSconverted16), blendSOFT, ErrorMask16, luma=true)
ClipToReturn = mt_merge(ClipToReturn, blendHARD, SceneChange, luma=true)
ClipToReturn = Interleave(clp_Input, ClipToReturn)

return ClipToReturn
}

Here are sample screenshots to compare methods, with frame doubling in all cases.

Interframe
https://s9.postimg.org/o7mfaszff/Interframe.png (https://postimg.org/image/o7mfaszff/)

fm_fps
https://s9.postimg.org/41ixbx3rv/jm_fps.png (https://postimg.org/image/41ixbx3rv/)

YFRC
https://s9.postimg.org/5hufu26or/yfrc.png (https://postimg.org/image/5hufu26or/)

The best would be to take jm_fps and add YFRC's artifact removal.

[johnmeyer]

The only result that is even close to acceptable is the jm_fps. I think that is based on some code I posted. If that is the case, I can tell you that there is absolutely nothing special about it, and it is almost identical to some of the sample code in the MVTools2 documentation.

The reason I am writing this is that I wonder if there is something missing, or some setting that isn't quite right for the other two scripts (Interframe and yfrc)? I say this because of all the fuzziness on sections that weren't moving that fast. To me, they looked blended rather than interpolated. I know that some of these scripts have the ability to do blending rather than ME, based on metrics, and I wonder if that is what is happening.

I think the results you get for this simple case should be much closer to each other.

[MysteryX]

Agree.

Solution 1 is to add YFMC's artifact removal to jm_fps script.

Solution 2 is to figure out what setting is missing to fix SVP/Interframe.

Solution 2 would be best.

Having a "good" and "bad" script using the same DLL is going to make it easier as SVP and MVTools2 have a completely different syntax.

[MysteryX]

Alright I think I'm getting at something.

jm_fps
https://s16.postimg.org/4v898rhbl/jm_fps.png (https://postimg.org/image/4v898rhbl/)

YFMC
https://s16.postimg.org/5m0zejjox/yfmc.png (https://postimg.org/image/5m0zejjox/)

YFMC with artifact removal disabled
https://s16.postimg.org/y06eyf78x/yfmc_noremoval.png (https://postimg.org/image/y06eyf78x/)

Artifact auto-removal seems to be the culprit. Whenever we see the image doubling effect, that's frame blending due to artifact removal.

[MysteryX]

There's more than artifact removal

SVP
https://s13.postimg.org/42yxgfyv7/SVP1.png (https://postimg.org/image/42yxgfyv7/)

SVP with artifact removal disabled
https://s13.postimg.org/d9h83q43n/SVP_noremoval.png (https://postimg.org/image/d9h83q43n/)

jm_fps
https://s13.postimg.org/xsc48si0z/jm_fps.png (https://postimg.org/image/xsc48si0z/)

What I had started doing was to take YFMC script and rewrite it to look like jm_fps, and looking at the difference of script and result every step of the way.

[raffriff42]

YFRC is extremely conservative regarding artifacts - it falls back to frame blending a lot. For some sources, that is exactly what is needed.

For high motion stuff, I like YFRC a lot. For me, false interpolation artifacts are way more annoying than blended frames.

Here is a version I have made with comments added and variables renamed for clarity, and more importantly, a way to see the artifact mask in action so it can be tuned to your liking: YFRC2(..., output="over"). You can also output the mask only (output="mask") to use with a different interpolator like jm_fps.# Version: "01dd-10mm-2015yyyy" modded raffriff42 31-Mar-2017
# Author: RunForLife(Oleg Yushko) http://videomontazh.com.ua
# http://avisynth.nl/index.php/YFRC
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Yushko Frame Rate Converter 2x, modded 2017 raffriff42
### doubles the frame rate with strong artifact detection and scene change detection.
##
## YV12/YV24/Y8/YUY2
##
## @ hardblend - how to handle scene change and artifact fallback:
## if false (default), blend like ConvertFPS; else repeat like ChangeFPS
##
## @ weakmask - if true, allow a few more artifacts to get through; default false
##
## @ output - (auto|inter|none|mask|over) default "auto"=normal artifact masking;
## "inter"=interpolation only; "none"=ConvertFPS only;
## "mask"=mask only; "over"=mask as cyan overlay for debugging
##
function YFRC2(clip C,
\ bool "hardblend", bool "weakmask",
\ string "output")
{
ox = C.Width
oy = C.Height

fps_num = C.FrameRateNumerator * 2
fps_den = C.FrameRateDenominator

bigblock = (ox>1270 || oy>710)
BlockH = (bigblock) ? 32 : 16
BlockV = (oy>710) ? 32 : 16

hardblend = Default(hardblend, false)
weakmask = Default(weakmask, false)
output = Default(output, "auto")

blendHARD = C.DeleteFrame(0)
blendSOFT = (hardblend)
\ ? blendHARD
\ : C.ConvertFPS(fps_num, fps_den).SelectOdd

## blur(1) - smooth edges for better analysing
sup = MSuper(C.blur(1), chroma=true, pel=2)

## TODO: try merge w/ jm_fps
bak = sup.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=true,
\ chroma=false, search=1, searchparam=1, truemotion=true,
\ lambda=2000, global=true, dct=0, divide=2)
fwd = sup.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=false,
\ chroma=false, search=1, searchparam=1, truemotion=true,
\ lambda=2000, global=true, dct=0, divide=2)

## "Flow" - MFlowFps double framerate
Flow = C.MFlowFps(C.MSuper(levels=1), bak, fwd,
\ num=fps_num, den=fps_den, blend=false, mask=0)

## "EM" - error or artifact mask
EM = C.MMask(bak, ml=190, kind=1) [** kind=SAD *]
EM = EM.Overlay(
\ C.MMask(fwd, ml=190, kind=1).DeleteFrame(0),
\ opacity=0.5, mode="lighten")

EM = EM.Overlay(
\ C.MMask(bak, ml=64, kind=2).mt_inpand, [** kind=occlusion *]
\ opacity=0.5, mode="lighten")

## mask strength
## TODO: tuning ('ml=' values etc)
EM = (!weakmask) ? EM : EM.Levels(0, 0.7, 255, 0, 191, coring=false)

EM = EM.BicubicResize(Round((ox/BlockH)/4.0)*4, Round((oy/BlockV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ [*.mt_inpand(mode=mt_circle(zero=true, radius=1)) *]
\ .mt_binarize(92)
\ .Blur(1.0)
\ [*.ColorYUV(cont_y=_f2c(2.0)) *]
\ .BicubicResize(ox, oy)

## "Sc" - scene detection
Sc = C.MSCDetection(bak, thSCD2=130)

## the YFRC magic happens
## note hard coded 2x interpolation
B = mt_merge(SelectOdd(Flow), blendSOFT, EM, luma=true)
B = mt_merge(B, blendHARD, Sc, luma=true)

R = (StrCmpi(output, "auto")==0) [** auto: artifact masking *]
\ ? Interleave(C, B)
\ : (StrCmpi(output, "inter")==0) [** inter: interpolation only *]
\ ? Flow
\ : (StrCmpi(output, "none")==0) [** none: ConvertFPS only *]
\ ? (hardblend)
\ ? C.ChangeFPS(fps_num, fps_den)
\ : C.ConvertFPS(fps_num, fps_den)
\ : (StrCmpi(output, "mask")==0) [** mask: mask only *]
\ ? EM.ConvertFPS(fps_num, fps_den).Grayscale.Invert
\ : (StrCmpi(output, "over")==0) [** over: mask as cyan overlay *]
\ ? Flow.Overlay(
\ MergeRGB(BlankClip(EM), EM, EM)
\ [*.RGBAdjust(2, 2, 2)*]
\ .ConvertFPS(fps_num, fps_den),
\ mode="Add", opacity=0.25)
\ : Assert(false,
\ "YFRC2: 'output' not one of (auto|inter|none|mask|over)")
return R
}

[StainlessS]

Looks good Raff.
I note that you use Blur(1.0) in "sup = MSuper(C.blur(1), chroma=true, pel=2) ",

Dont see anything wrong in that, however Didée's goto value for blur is 0.6,
https://forum.doom9.org/showthread.php?p=1508638#post1508638

and he seemed to have some idea what he was doing :)

Just a suggestion, give it a try.

[raffriff42]

Blur(1) is from the original, I actually have not tried other values.
http://avisynth.nl/index.php/YFRC

[MysteryX]

YFRC with weak mask is almost as good as jm_fps, with reasonable artifact handling. It looks better than SVP/Interframe, so I'm still not sure what it would take to get those results with SVP.

YFRC definitely is too conservative, but the danger of weak mask is that it leaves artifacts. If we're to do a manual review of videos, instead of marking zones to replace manually, I'd rather have the choice between
- interpolated clip
- weak mask
- strong mask
- blend
- copy

Then we can set a default for the video, then override frames or segments with a different clip.

[MysteryX]

So far, "weak mask" gives the best results.

Here's the YFMC script with jm_fps + artifact masking


# Version: "01dd-10mm-2015yyyy" modded raffriff42 31-Mar-2017
# Author: RunForLife(Oleg Yushko) http://videomontazh.com.ua
# http://avisynth.nl/index.php/YFRC
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Yushko Frame Rate Converter 2x, modded 2017 raffriff42
### doubles the frame rate with strong artifact detection and scene change detection.
##
## YV12/YV24/Y8/YUY2
##
## @ hardblend - how to handle scene change and artifact fallback:
## if false (default), blend like ConvertFPS; else repeat like ChangeFPS
##
## @ weakmask - if true, allow a few more artifacts to get through; default false
##
## @ output - (auto|inter|none|mask|over) default "auto"=normal artifact masking;
## "inter"=interpolation only; "none"=ConvertFPS only;
## "mask"=mask only; "over"=mask as cyan overlay for debugging
##
function YFRC2(clip C,
\ bool "hardblend", bool "weakmask",
\ string "output")
{
ox = C.Width
oy = C.Height

fps_num = 50 #C.FrameRateNumerator * 2
fps_den = 1 #C.FrameRateDenominator

bigblock = (ox>1270 || oy>710)
BlockH = (bigblock) ? 32 : 16
BlockV = (oy>710) ? 32 : 16

hardblend = Default(hardblend, false)
weakmask = Default(weakmask, false)
output = Default(output, "auto")

blendHARD = C.DeleteFrame(0)
blendSOFT = (hardblend)
\ ? blendHARD
\ : C.ConvertFPS(fps_num, fps_den).SelectOdd

prefiltered = RemoveGrain(C, 22)
super = MSuper(C, hpad = BlockH, vpad = BlockV, levels = 1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = BlockH, vpad = BlockV) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
forward = MAnalyse(superfilt, isb = false, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
forward_re = MRecalculate(super, forward, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
Flow = MFlowFps(C, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

## "EM" - error or artifact mask
EM = C.MMask(backward_re, ml=190, kind=1) [** kind=SAD *]
EM = EM.Overlay(
\ C.MMask(forward_re, ml=190, kind=1).DeleteFrame(0),
\ opacity=0.5, mode="lighten")

EM = EM.Overlay(
\ C.MMask(backward_re, ml=64, kind=2).mt_inpand, [** kind=occlusion *]
\ opacity=0.5, mode="lighten")

## mask strength
## TODO: tuning ('ml=' values etc)
EM = (!weakmask) ? EM : EM.Levels(0, 0.7, 255, 0, 191, coring=false)

EM = EM.BicubicResize(Round((ox/BlockH)/4.0)*4, Round((oy/BlockV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ [*.mt_inpand(mode=mt_circle(zero=true, radius=1)) *]
\ .mt_binarize(92)
\ .Blur(1.0)
\ [*.ColorYUV(cont_y=_f2c(2.0)) *]
\ .BicubicResize(ox, oy)

## "Sc" - scene detection
Sc = C.MSCDetection(backward_re, thSCD2=130)

## the YFRC magic happens
## note hard coded 2x interpolation
B = mt_merge(SelectOdd(Flow), blendSOFT, EM, luma=true)
B = mt_merge(B, blendHARD, Sc, luma=true)

R = (StrCmpi(output, "auto")==0) [** auto: artifact masking *]
\ ? Interleave(C, B)
\ : (StrCmpi(output, "inter")==0) [** inter: interpolation only *]
\ ? Flow
\ : (StrCmpi(output, "none")==0) [** none: ConvertFPS only *]
\ ? (hardblend)
\ ? C.ChangeFPS(fps_num, fps_den)
\ : C.ConvertFPS(fps_num, fps_den)
\ : (StrCmpi(output, "mask")==0) [** mask: mask only *]
\ ? EM.ConvertFPS(fps_num, fps_den).Grayscale.Invert
\ : (StrCmpi(output, "over")==0) [** over: mask as cyan overlay *]
\ ? Flow.Overlay(
\ MergeRGB(BlankClip(EM), EM, EM)
\ [*.RGBAdjust(2, 2, 2)*]
\ .ConvertFPS(fps_num, fps_den),
\ mode="Add", opacity=0.25)
\ : Assert(false,
\ "YFRC2: 'output' not one of (auto|inter|none|mask|over)")
return R
}


jm_fps
https://s7.postimg.org/4uon7sz5z/jm_fps.png (https://postimg.org/image/4uon7sz5z/)

YFMC (weakmask)
https://s7.postimg.org/noag4sxdz/YFMC.png (https://postimg.org/image/noag4sxdz/)

jm_fps + YFMC's artifact removal (weakmask)
https://s7.postimg.org/l88mqyfbb/YFMC-jm_fps.png (https://postimg.org/image/l88mqyfbb/)

With artifact removal, I'm honestly not sure whether YFMC or jm_fps gives the best result...

TODO: make the script work either for frame doubles, or for all frames.

I could create a plugin/script that allows easy switching between various clip versions for manual tweaking.

If we flip through the videos while seeing mask overlays, it would then be easy to say whether to tune settings up or down for each frame. This will require more testing but I'd expect to use weakmask in most cases and fallback to strong mask on a few frames when problems occur.


hum... weakmask does only this. It seems to work so far, but is that enough?

EM = (!weakmask) ? EM : EM.Levels(0, 0.7, 255, 0, 191, coring=false)


We still haven't resolved any fundamental problem, but at least we're getting to "something"

[MysteryX]

I've done good progress. Modified YFMC with additional arguments, allowed specifying frame rate other than FrameDouble (while still allowing FrameDouble), cleaned up the code, used jm_fps, and changed a bunch of things.

Considering it's containing pieces of code from me, johnmeyer, raffriff42 as well as Yushko, it doesn't make sense anymore to call it "Yushko", and even less YFRC2. I changed the name to FrameRateConverter.

I changed weakmask with MaskStr. It would make sense to allow more settings here, and the code for weak masking was left as a TODO:tweaking. For now it's an int that takes 0(weak) or 1(normal), but that should be changed.

Finally, between YFRC and jm_fps interpolation, we have to decide which gives the best results -- which settings to use for interpolation. From initial testing, both seem to be slightly better than the other in 50% of cases. It might be good to test how each perform in worst cases like the video johnmeyer posted.


# Frame Rate Converter
# Version: 19-Apr-2017
# Authors: Yushko, johnmeyer, raffriff42, MysteryX
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Doubles the frame rate with strong artifact detection and scene change detection.
##
## YV12/YV24/Y8/YUY2
## Requires: masktools2, mvtools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ BlockH - The horizontal block size (default = Width > 1270 || Height > 710 ? 32 : 16)
##
## @ BlockV - The vertical block size (default = BlockH or Height > 710 ? 32 : 16)
##
## @ Blend - Whether to use frame blending for artifact masking (default = true)
##
## @ Output - (auto|inter|none|mask|over) default "auto"=normal artifact masking;
## "inter"=interpolation only; "none"=ConvertFPS only;
## "mask"=mask only; "over"=mask as cyan overlay for debugging
##
## @ MaskStr - The artifact masking strength (1 to 10). A lower value will not change
## artifact detection but will make the mask softer. (Default=7)
##
## @ MaskSAD - Artifact masking strength for bad motion (Default=190)
##
## @ MaskOcc - Artifact masking strength for occlusion (Default=64)
##
## @ thSCD1 - Scene change detection treshold 1 of MSCDetection. 0 to disable. (Default=400)
##
## @ thSCD2 - Scene change detection treshold 2 of MSCDetection. (Default=130)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble",
\ int "BlockH", int "BlockV", bool "Blend", string "Output", int "MaskStr", int "MaskSAD", int "MaskOcc", int "thSCD1", int "thSCD2")
{
Blend = Default(Blend, true)
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlockV = Default(BlockV, Defined(BlockH) ? BlockH : (C.Height > 710 ? 32 : 16))
BlockH = Default(BlockH, C.Width > 1270 || C.Height > 710 ? 32 : 16)
MaskStr = Default(MaskStr, 7)
MaskSAD = Default(MaskSAD, 190)
MaskOcc = Default(MaskOcc, 64)
thSCD1 = Default(thSCD1, 400)
thSCD2 = Default(thSCD2, 130)

B = Blend ? C.ConvertFPS(NewNum, NewDen) : FrameDouble ? C.DeleteFrame(0) : C.ChangeFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
prefiltered = RemoveGrain(C, 22)
super = MSuper(C, hpad = BlockH, vpad = BlockV, levels = 1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = BlockH, vpad = BlockV) # all levels for MAnalyse
bak = MAnalyse(superfilt, isb = true, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
fwd = MAnalyse(superfilt, isb = false, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
fwd = MRecalculate(super, fwd, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
bak = MRecalculate(super, bak, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

## YFMK interpolation
# super = MSuper(C.blur(.6), chroma=true, pel=2)
# bak = super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=true,
# \ chroma=false, search=1, searchparam=1, truemotion=true,
# \ lambda=2000, global=true, dct=0, divide=2)
# fwd = super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=false,
# \ chroma=false, search=1, searchparam=1, truemotion=true,
# \ lambda=2000, global=true, dct=0, divide=2)
# Flow = C.MFlowFps(C.MSuper(levels=1), bak, fwd, num=NewNum, den=NewDen, blend=false, mask=0)

## "EM" - error or artifact mask
EMfwd = C.MMask(fwd, ml=MaskSAD, kind=1)
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = C.MMask(bak, ml=MaskSAD, kind=1) [** kind=SAD *]
EM = EM.Overlay(EMfwd, opacity=0.5, mode="lighten")

EM = EM.Overlay(
\ C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand, [** kind=occlusion *]
\ opacity=0.5, mode="lighten")

## mask strength
EM = MaskStr < 10 ? EM.Levels(0, float(MaskStr) / 10, 255, 0, 255 - 9*(10-MaskStr), coring=false) : EM

EM = EM.BicubicResize(Round((C.Width/BlockH)/4.0)*4, Round((C.Height/BlockV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
EM = EM.ChangeFPS(NewNum, NewDen)

## "Sc" - scene detection
ScDetect = thSCD1 > 0 && thSCD2 > 0
Sc = ScDetect ? C.MSCDetection(bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(C)
Sc = Sc.ChangeFPS(NewNum, NewDen)
#Sc = ConditionalFilter(EM, BlankClip(EM, color=color_white), EM, "AverageLuma()", ">", "100")

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = ScDetect ? mt_merge(M, B, Sc, luma=true) : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "inter")==0) [** inter: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? Blend ? C.ConvertFPS(NewNum, NewDen) : C.ChangeFPS(NewNum, NewDen)
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? mt_merge(
\ EM.ConvertFPS(NewNum, NewDen),
\ BlankClip(EM, color=color_white),
\ ScDetect ? sc : BlankClip(EM), luma=true).Grayscale.Invert
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM).ConvertFPS(NewNum, NewDen), mode="Add", opacity=0.30),
\ BlankClip(EM, color=color_darkgoldenrod), ScDetect ? sc.Levels(0, 1, 255, 0, 128, coring=false) : BlankClip(EM), luma=true)
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|inter|none|mask|over)")
return R
}


Edit: found the bug and updated the script. There was a ChangeFps missing. This script is working, all that's left is testing and tweaking some details.

[MysteryX]

Weak masking is exactly the same artifact detection. The difference is that it doesn't replace a huge area around it. I think weak mask should be the default; I see no reason not to use it. Perhaps the artifact detection itself could be configured?

Here are some tests on the Motion Estimation Torture Clip
Flint Michigan Parade Clip (14.985 fps; progressive; DV format) (https://www.mediafire.com/?cb739wsyzws3z53)

All are interpolated to 60fps and using weak artifact masking.

Frame 112. InterframeSmooth / YFMC / JM
https://s29.postimg.org/t1qpoub37/Interframe_Smooth112.png (https://postimg.org/image/t1qpoub37/) https://s2.postimg.org/wqaitayph/112_YFMC.png (https://postimg.org/image/wqaitayph/) https://s2.postimg.org/6gpgaics5/112_JM.png (https://postimg.org/image/6gpgaics5/)
All 3 are decent but there are slight artifacts with JM

Frame 141. InterframeFilm / InterframeSmooth / YFMC / JM
https://s29.postimg.org/tby886ppf/Interframe_Film141.png (https://postimg.org/image/tby886ppf/) https://s29.postimg.org/vx8e9g4gz/Interframe_Smooth141.png (https://postimg.org/image/vx8e9g4gz/) https://s2.postimg.org/q2dx6pf79/141_YFMK.png (https://postimg.org/image/q2dx6pf79/) https://s2.postimg.org/oytsuquk5/141_JM.png (https://postimg.org/image/oytsuquk5/)
YFMK is better and JM shows some artifacts. Interframe is less detailed.

Frame 582. Interframe does an ugly job while YFMC's artifact masking skips the frames of that section altogether.
https://s29.postimg.org/8gbxwxtib/Interframe_Film582.png (https://postimg.org/image/8gbxwxtib/) https://s2.postimg.org/aw7vfrp6d/582_YFMK.png (https://postimg.org/image/aw7vfrp6d/) https://s2.postimg.org/8dm6f33g5/582_JM.png (https://postimg.org/image/8dm6f33g5/)
Though call...

So far, Interframe doesn't perform so badly, but I see a much stronger difference on my own clips

InterframeSmooth / YFMC / JM
https://s2.postimg.org/95ouea7n9/6558_Interframe_Smooth.png (https://postimg.org/image/95ouea7n9/) https://s2.postimg.org/8vlx8oiet/6558_YFMK.png (https://postimg.org/image/8vlx8oiet/) https://s2.postimg.org/cqkpxic6t/6558_JM.png (https://postimg.org/image/cqkpxic6t/)

[MysteryX]

I have updated the script above.
- Scene change detection must use frame blending, otherwise, when a scene is detected as repeated scene change, the frames would keep bouncing back and forth (between deleted and interpolated frames)
- All artifact detection parameters are now exposed as parameters (MaskSAD, MaskOcc, thSCD1, thSCD2)
- MaskStr is now a number that fades the mask. "weak mask" is MaskStr=7, 5 fades it further, and 9 only fades it a little. Use 10 for no fading.
- Scene changes are now included in "over" and "mask" outputs. In "over" mode, it will appear as a dark-golden color with 50% opacity.
- Scene change detection can be disabled with thSCD1=0

For mode "over", cyan overlays can be hard to see on cyan background. Is there a way to show the mask in a way that will be equally visible for any color?

RemoveGrain has a bug in YV24. Is there a good alternative to it?

You can test it out to see if everything is working. Also test various settings to see what works best for you, and perhaps we can decide which artifact detection settings to use as defaults. Also it would be good to test with other types of content such as anime that requires different settings.

For the most part, it seems to be working good, but there are still ugly effects with artifact removal, such as spotlights moving by and showing up as "spots" of different colors. There's also nothing uglier than a scene change with only half the frame showing up through artifact removal mask.

[Motenai Yoda]

I have updated the script above.
- Scene change detection must use frame blending, otherwise, when a scene is detected as repeated scene change, the frames would keep bouncing back and forth (between deleted and interpolated frames)

Shouldn't be more reliable to use ChangeFPS instead of ConvertFPS for scene-change detections?

[MysteryX]

Shouldn't be more reliable to use ChangeFPS instead of ConvertFPS for scene-change detections?

The problem I was having is that on a slowly-rotating scene at 45° camera angle, it is triggering scene change detection repeatedly. If half the frames are interpolated and half the frames are dropped, then instead of a smooth panning, you get a zigzag camera rotation bouncing back and forth.

If I want to do a full frame blending whenever more than 30% is part of the artifact masking, is this the right syntax to do it? (right after MSCDetection)

Sc = ConditionalFilter(EM, BlankClip(EM, color=color_white), EM, "AverageLuma()", ">", "80")


I've done a good bulk of the work. I'd appreciate others who are more expert to chime in with feedback and improvements. Perhaps instead of applying Sc and EM separately, it would be good to merge them into 1 mask and then apply only 1 mt_merge. This should be better on performance.

I could also write a simple plugin that is similar to ConditionalFilter but takes a string to specify which frames to take from which clip. Then you could specify something like "1-10A,11-100B,50C,200-210A" and frames 1-10,200-210 would take clip A (frame blending), frames 11-100 would take clip B (interpolation) and frame 50 would take clip C (delete frame). But for now, we want to get as good as possible with the basic script.

[johnmeyer]

BTW, if you are going to use my Flint Michigan parade clip, once you get your script exactly the way you want, try it out on the moose antlers. If it can handle that, it can handle anything. That is the ultimate torture test, IMHO.

[MysteryX]

Arggg there was a nasy bug! Artifact masking mask remained in original frame rate which was out of phase with output video, which caused artifacts masks to be out of sync, and over half of the video being treated as a scene change!! So the tests I did earlier were not good.

I fixed the bug and re-uploaded new images. Now, YFMK and JM are much closer to each other. Still hard to say which one is best.

BTW, if you are going to use my Flint Michigan parade clip, once you get your script exactly the way you want, try it out on the moose antlers. If it can handle that, it can handle anything. That is the ultimate torture test, IMHO.
A considerable portion of the image is detected as artifacts.

I'm considering dropping MSCDetection and simply considering a scene change if more than X% of the image is in the mask. I could do a ChangeFps in those cases, and ConvertFps when applying masks. In the case of the moose, we just have to set it to a value so that this whole scene is seen as "scene changes" and replaces the whole frame. The best way to handle it is to disable frame interpolation -- and avoid blending to not have double shadows.

Forget what I said earlier about scene changes during panning -- scene changes were out of wack in the code.

With this fixed, it's actually working pretty well as-is! With an adjustable "artifact mask treshold" to consider as scene change, this just might be enough. This treshold would be much easier to configure than thSCD1/thSCD2, and be much faster.