SVP-like frame interpolation? [Archive]

CruNcher

18th March 2017, 11:20

Spent some more time tuning the script parameters for this particular test clip. Please keep in mind that this clip is extremely demanding, usually I would never recommend to use this script on anime.

What I found out is that a dct value of 3 does give better results on the control panel to the left, but it does not improve the vertical grille to the right. What does improve this grille is using a larger block size of 32 for MAnalyze. For MRecalculate I kept the original block size of 8, increasing it to 16 gave worse results.

So for this particular clip my preferred settings are dct =3 and blksize = 32 for MAnalyse. The resulting file is here:
http://www59.zippyshare.com/v/l6Ouh8fw/file.html

In my experience any settings which work for this clip will also give good results for other sources. I need to test if the increased block size will indeed work for HD sources... ;)

Cheers
manolito

This version introduced many new perceptable problems though you fought 1 problem and created a whole bunch of new ones which would most probably result now in a even lower mos score then before (predicted).

Though this indicates their must be a better result hiding in between :)

if you split all the best parts and put them back together you would have a overall better result, from the timing this could even work.

Temporal i can split this in 3 different parts that show different problems

Wherby the amount of problems are different from each case.

In this test case you could say you have 3 scene parts Beginning,Middle and the End part

and depending on your setup now they come out different the beginning part never completely fixed the middle and end part with some very good overall results except in your latest result the end part comes out perceptually much worse then in any result before, therfore though the middle part is perceptually better (no fast texture failure).

Logic now says take all the good parts and put them together and you have a pretty good overall result.

Which also could indicate you need something more adaptively clever to get this result in 1 try,without putting it manually together ;)

My reasoning still stays dct 1 gives the overall best result except 1 difference now to the latest output therefore 1 part in the latest output is now totally ending up bad perceptually.

i would also weight the temporal failure of the latest result in the end part higher even critical now then the small problem in the middle part on the dct 1 result before (fast texture failure).

manolito

18th March 2017, 11:53

Thanks for looking into it...

I need to test if the increased block size will indeed work for HD sources...

Well, I found that it does not... :devil:

The best results for this clip came from using dct=1, but this is so slow that it is not really usable for me.

Of course you can optimize the results by finetuning the params for each and every source, maybe even split the source into several parts. But this is not what I'm after. I want a script which works well for the vast majority of sources without tweaking the params.

And I believe that johnmeyer's original params do an excellent job for most sources (maybe use dct=3 instead of 0). Way better than MotionProtectedFPS. And keep in mind that I am not interested in watching still frames. I want to just watch the movie, and it should look better than using ChangeFPS or ConvertFPS. I do not mind some motion artifacts as long as they do not get too annoying. And I use this only on film, not on anime. Plus I mostly use this type of fps conversion for PAL <-> NTSC conversion, not for slow motion.

Cheers
manolito

CruNcher

18th March 2017, 12:23

yes but overall this result could endup in a catastrophe now if it creates the same temporal issues for every source that heavy perceptual jitter is really annoying for moving objects like those angels it destroys everything and i really wonder how you can percept it now as better then anything before.

you fixed the texture problem yes but this is hyper annoying now in motion compared to that small glitch that was only visible mere milliseconds before.

Good thing in the end you at last saw the problems on the panel with the light scan ;)

Also there are much much more problems to think about you using AVC currently as output you don't know yet how this will be perceived actually with HEVC/VP9 for example it could come out even worse (better perceptible).

It partly though could be also get fixed in the other direction as internaly a hevc encoder is smarter then mvtools and you might throw out extra resources for nothing ;)

manolito

19th March 2017, 15:06

I don't know but I think you are being a little too critical. I showed the converted clips to some of my friends who are journalists working for Deutsche Welle TV. They do have trained eyes, but just like me they focus on the overall visual impression, and all of them agreed that the MVTools based conversions were quite pleasing to watch.

For this particular clip my ultimate conversion is here:
http://www82.zippyshare.com/v/7bRTiwej/file.html

dct=1 and blocksize for MAnalyse = 32. Way too slow, but I cannot find any annoying flaws.

My main problem with these fps conversion methods are hard coded subs or movie credits. The letters mostly get warped to a point where they are painful to watch. But this is the only occasion where I would consider splitting up the source and convert the different parts separately...

Cheers
manolito

kolak

19th March 2017, 20:08

MvTools are almost as good as Alchemist or Tachyon which are used in braodcast and cost small fortune.
It just would be nice to have it ported to GPU, like svp does. SVP made to many speed shortcuts so quality is not as good.

CruNcher

22nd March 2017, 04:07

manolito

22nd March 2017, 12:46

end = horrible (full of motion perceptible fails, heat haze effect)

Are you talking about the three small angels in the background?

The human brain works differently. Everybody just sees the two large angels in the foreground, and they do not display motion artifacts. The information in the background is pretty much discarded because the brain determines that this information is not important. (This is gone by the "Formatio Reticularis" which works like a spotlight).

Cheers
manolito

CruNcher

22nd March 2017, 16:28

Are you talking about the three small angels in the background?

The human brain works differently. Everybody just sees the two large angels in the foreground, and they do not display motion artifacts. The information in the background is pretty much discarded because the brain determines that this information is not important. (This is gone by the "Formatio Reticularis" which works like a spotlight).

Cheers
manolito

Everything is connected the edge warping of the moving object 2 Angels is causing the problems and stays visible up until the end the edge of the moving object is "warping" everything it moves in front of gets distorted, that is pretty visible and was less visible some versions before these problems where better hidden.

Im pretty sure these warping distortions at the end got amplified i didn't percept them so crazy heavy before on the first sight.

just make a experiment glue both results together :)

the beginning and middle of your newest results and the older result of the end and voila a pretty stable overall result :)

And of course you concentrating on those 2 Angels if you would conduct a Eye tracker experiment you would see the ROI being the 2 Angels and everything in their close proximity is being percepted with highest priority.

MysteryX

24th March 2017, 17:46

Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi

Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.php?p=1789031#post1789031

Cheers
manolito

How does this script compare with InterFrame?

johnmeyer

24th March 2017, 18:50

How does this script compare with InterFrame?For the answer, read my post here (https://forum.doom9.org/showpost.php?p=1725099).

That post also explains why some in this thread thought my script provided some good results: I was able to tweak some things which are not "exposed" with SVP and its Interframe front end.

As for things said in this thread, I still am not convinced that any of the DCT settings are going to provide any substantial, real improvements, i.e., it won't produce differences that actually matter. Note I didn't say there wouldn't be differences, only that those differences won't really get at the reasons why motion estimation fails when doing frame rate changes or other operations which involve creating new frames from adjacent frames.

The real issue is how to define "objects" and how to track them. The motion estimation done by any of these MVTools-derived filters relies on nothing more than tracking pre-determined blocks of pixels rather than pre-identifying actual objects in the frame. This is why block size is the most important variable to change when trying to get good results. Depending on the video and the size of the things being tracked (like people's legs, vertical fence posts, and other difficult-to-track items), different block sizes will work on some videos better than others. I find a block size of 16 to be a good starting point, but sometimes find 8 or 32 works better. The block overlap can then provide some fine tuning.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.

So, motion estimation is not a "set it, and forget it" tool, and if you use it that way for creating new frames, you will get burned, and it will be sooner rather than later.

MysteryX

25th March 2017, 04:56

In the Natural Grounding Player / Yin Media Encoder (https://github.com/mysteryx93/NaturalGroundingPlayer/), I upscale videos from 288p 25fps into 768p 60fps. I get the best results by running Interframe between the 2 frame doubles, and it is one of the most important steps. For low quality videos with lots of artifacts, SVP is actually removing a lot of those artifacts by creating the interframe animations! Kind of too good to be true, but it works well.

I'm wondering whether the approach you suggest here would give better results, by combining 2 approaches to reduce the severity of artifacts when it fails. I'm looking for a generic script that will work most of the time, with a few simple tweakeable settings.

Do you have a specific script I could try to see the difference in my case?

CruNcher

25th March 2017, 07:24

For the answer, read my post here (https://forum.doom9.org/showpost.php?p=1725099).

That post also explains why some in this thread thought my script provided some good results: I was able to tweak some things which are not "exposed" with SVP and its Interframe front end.

As for things said in this thread, I still am not convinced that any of the DCT settings are going to provide any substantial, real improvements, i.e., it won't produce differences that actually matter. Note I didn't say there wouldn't be differences, only that those differences won't really get at the reasons why motion estimation fails when doing frame rate changes or other operations which involve creating new frames from adjacent frames.

The real issue is how to define "objects" and how to track them. The motion estimation done by any of these MVTools-derived filters relies on nothing more than tracking pre-determined blocks of pixels rather than pre-identifying actual objects in the frame. This is why block size is the most important variable to change when trying to get good results. Depending on the video and the size of the things being tracked (like people's legs, vertical fence posts, and other difficult-to-track items), different block sizes will work on some videos better than others. I find a block size of 16 to be a good starting point, but sometimes find 8 or 32 works better. The block overlap can then provide some fine tuning.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.

So, motion estimation is not a "set it, and forget it" tool, and if you use it that way for creating new frames, you will get burned, and it will be sooner rather than later.

Exactly and in this case the Quantization noise from the lossy MPEG Input in manolitos test conversion case causes some of the tracking problems.

johnmeyer

25th March 2017, 22:05

Do you have a specific script I could try to see the difference in my case?Definitely not. Like so many video issues, you have to do things manually. In this case, you have to watch the video, see where it fails, and then switch over to the other approach at those points.

I put both versions on two timelines in my NLE, with the motion estimated version on the dominant (i.e., default) track. I play the video, usually at 1.5x - 2x normal speed (to get through it in a hurry). When I see bad frame (or several bad frames), I simply cut to the other track until the problem goes away.

For somewhat critical work, I will sometimes crossfade from the ME version to the other version in order to make the switch less apparent. For really critical work, I will create a motion mask, feathered at the edges, to replace only the parts of the frame that are broken. This produces virtually perfect results, but it obviously takes quite a bit of time. When I do paid work (once in awhile some of my stuff ends up in movies or on TV), it is worth the time to do this.

MysteryX

26th March 2017, 00:22

I put both versions on two timelines in my NLE, with the motion estimated version on the dominant (i.e., default) track. I play the video, usually at 1.5x - 2x normal speed (to get through it in a hurry). When I see bad frame (or several bad frames), I simply cut to the other track until the problem goes away.
That's a perfect case where automation provided by a software would be most useful -- but this process would be somewhat complicated to implement.

johnmeyer

26th March 2017, 00:44

That's a perfect case where automation provided by a software would be most useful -- but this process would be somewhat complicated to implement.Complicated? No. Impossible? Yes.

You simply cannot anticipate the problems which get created. Also, things which "look good" to an algorithm look like heck to the human eye.

I've written over 100 AVISynth scripts, and have spent many thousands of hours editing video over the past twenty years. I understand both and, as an EE, programmer, and former software manager, I think I know what can and cannot be done (although I am often amazed and surprised what software people manage to create).

The best way to express my skepticism about finding an algorithmic solution to this problem is that if you could actually detect the anomaly created by motion estimation, then the motion estimation itself would be able to avoid creating it in the first place. Since even the commercial software cannot do this (i.e., even people with advanced programming skills and economic incentives), and since they've been working on this for a long time, I don't think it will happen anytime soon.

MysteryX

26th March 2017, 01:07

MysteryX

26th March 2017, 04:25

Essentially, what you're doing is pretty simple if I understand correctly. You use 2 algorithms: a more aggressive frame interpolation (that causes more artifacts), and a safer method for when it fails. Then the idea is to identify which frames or area to use the alternative method.

If this approach was to be semi-automated, it would be best done as an Avisynth script than as a software. A script or plugin could be designed that takes a string with "100-115,140-144,160-180", and perhaps even allow specifying rectangles or zones for masks, and automatically do everything you're manually scripting.

The only challenge I'm seeing is that this would require conditional filters, but ScriptClip doesn't currently work with MT.

If you're doing a lot of it and are spending a lot of time on this process, perhaps it would be worth it to develop such an utility that systematizes your process.

johnmeyer

26th March 2017, 04:57

What I'm saying is that a software could allow you to manually go over the video at 50% speed to mark which frames are corrupt, and handle the rest. Exactly the same as you're doing, but without having to hack around with manual scripts.

The first part of your process could probably easily be done.

Cross-fade maybe could maybe be semi-automated too. But that last part with motion masks, that can only be done manually by someone who really knows what he's doing.

Right now I'm implementing Deshaker (VirtualDub filter), which is difficult to use manually with its 2 passes, especially if you want to preview various settings -- and especially if you want to adjust settings for various segments.1. If you Google my name and "Deshaker" you will find that I wrote a very complex set of scripts to completely automate the process. One set of scripts was written within Sony Vegas Pro, my NLE. It provides its own scripting language. I also wrote a script in VirtualDub that interacts with the Vegas script. The end result is that, when you want to stabilize a clip on the timeline, you press one button, and the entire process happens with no further interaction on your part, including both Deshaker passes.

This entire process is well-documented by posts that I did in the Vegas forum (https://www.vegascreativesoftware.info/us/forum/deshaker-vegas-script-back-again--57432/) many years ago.

2. Because Vegas provides scripting, I already do the automation that you suggest. If, for instance, I want to cut to the other clip, but do it via a two-frame cross-fade, I can simply press one key and it will do the entire cut and cross fade. What's more, if I wanted to, I could instead simply insert markers at each place that I want to fix, and then do all the cuts using a batch version of the same script.

Again, if you look at the scripting portion of the Vegas forum, you will find many of my posts describing some of my dozens of Vegas scripts.

Even though Sony pretty much abandoned Vegas, and the new owner, Magix, doesn't appear to be doing anything to make it better, it is still the most productive editing tool on the planet because of its scripting capability.

MysteryX

26th March 2017, 06:47

It still could be interesting to see how your process could be translated into pure Avisynth programming.

TheFluff

26th March 2017, 17:55

It still could be interesting to see how your process could be translated into pure Avisynth programming.

please don't

kolak

26th March 2017, 23:30

Tachyon (used in broadcast for fps conversion) uses fallback method with masking and fathering:
https://www.telestream.net/pdfs/app-notes/app_Vantage_Tachyon_VPL.pdf

page 5,6.

They find problematic areas and then replace them with frame blended or nearest frame using making and feathering. Looks like they use quality of vectors as deterministic process.
I'm just surprised that this doesn't break "local" motion coherency between frames.

Love to see this in mvtools :)

poisondeathray

27th March 2017, 00:53

I'm just surprised that this doesn't break "local" motion coherency between frames.

How do you know it doesn't ?

Have you tested it or seen samples ?

Manually doing it usually looks poor when you try to take care of occlusions in that manner (blending or nearest through accurate user defined rotoscoped masks), so I doubt an "automatic" method using a less accurate method would look any better

johnmeyer

27th March 2017, 00:56

It still could be interesting to see how your process could be translated into pure Avisynth programming.

please don't

Since there is no point, I am not even tempted.

MysteryX

27th March 2017, 06:47

Johnmeyer, the first part of your process is actually very simple to do.

Write a filter that takes 2 clips and a string as parameters. The filter returns either clip based on the frame number as configured in the string. Frame blending and making transitions transparent, that's a whole other story. Perhaps blending both in the transition frames. Is it like an artist's work where you have to draw it differently in each situation?

I'm just curious, what part of your process can't be systematized?

videoFred

27th March 2017, 10:07

A very simple solution is creating two clips from the same source: one with changeFPS() and one with interpolation (this can be done with MVTools2 or Interframe() ).

Then we can select scenes with clipclop():

source = Avisource("L:\VdP\VdP_Sp4_gekuist.avi").converttoYV12()
changed = source.ChangeFPS(25)

V0 = changed
V1 = InterFrame(source,Newnum=25, Newden=1, Cores=8, GPU = true)

NickNames =""" # Psuedonyms for clips

I = 1
"""

SCMD="""
I 0,20
I 836,1285
I 1723,2312
I 3032,3202
I 3986,4456
I 4682,4938
I 6060,6600

"""

SHOW= True

ClipClop(V0,V1,scmd=SCMD,nickname=NickNames,show=SHOW)

The result is a mixed clip: frame doubling on "difficult" scenes (fast moving objects) and frame interpolation on "easy" scenes (slow panning for example). Of cource this requires manual searching for the scenes who can be interpolated.

Fred.

kolak

27th March 2017, 11:15

How do you know it doesn't ?

Have you tested it or seen samples ?

Manually doing it usually looks poor when you try to take care of occlusions in that manner (blending or nearest through accurate user defined rotoscoped masks), so I doubt an "automatic" method using a less accurate method would look any better

Yes, I seen results. They are quite good.

johnmeyer

27th March 2017, 16:55

StainlessS

27th March 2017, 17:41

You can use Sawbones/FrameSurgeon Replace FXn range:- https://forum.doom9.org/showthread.php?t=173158&highlight=sawbones
to replace ranges using clipClop, with additional functionality included.

SawBones v1.02

SawBones/FrameSurgeon, a VirtualDub/Avisynth script, utility combo to edit bad frames.
Create Command file in VirtualDub with Sawbones, and use Command file in Avisynth script function FrameSurgeon().

SawBones, is a compiled AutoIt script utility, intended to assist in creating Avisynth command file for FrameSurgeon.avs script.

SawBones is used together with VirtualDub and NotePad. You must run the app, with BOTH VirtualDub and the
NotePad Editor VISIBLE, you can scroll through a video clip, and press eg CTRL/DELETE to insert a FrameSurgeon DELETE (DEL n) command
for the current frame into the NotePad Editor. You press the keys with VDub as the active window (not NotePad).
SawBones requires VirtualDub, VirtualDubMod will not be recognised (It does not provide marked ranges in Status Bar).
After creating Command file in NotePad, Save as eg Command.txt and provide it as eg FrameSurgeon(Cmd="Command.txt").

*** Current frame is shown in VirtualDub in the middle of the status bar. ***
*** Ranges are marked in VirtualDub via HOME and END keys, shown at Left of status bar (when marked). ***
*** FXd clips (where d is Digit 1-9) are user provided to FrameSurgeon.avs function and default to source clip if not user supplied. ***
*** Interpolation commands in FrameSurgeon are for YV12 and YUY2 only, others any colorspace. ***

It is important to NOT do any VirtualDub clip editing, if you eg delete a frame in VirtualDub, then all frames after that frame will
be off by 1, and so all SawBones edited ranges/frames inserted later into NotePad file will be also off by 1.
The VDub loaded clip can be either an AVI or AVS clip, it makes no difference, you could eg have a stacked multi-window frame
open in VDub so as to choose from your FXd clips.

All SawBones Keyboard commands that are inserted into NotePad Text file:-

CTRL/F1 CopyFromPrevious frame to current frame (ie replace current frame n with frame n - 1. (CP n)
CTRL/F2 CopyFromNext frame to current frame (ie replace current frame n with frame n + 1. (CN n)

CTRL/1 Replace current frame with same frame from FX1 clip. (FX1 n)
CTRL/2 Replace current frame with same frame from FX2 clip. (FX2 n)
CTRL/3 Replace current frame with same frame from FX3 clip. (FX3 n)
CTRL/4 Replace current frame with same frame from FX4 clip. (FX4 n)
CTRL/5 Replace current frame with same frame from FX5 clip. (FX5 n)
CTRL/6 Replace current frame with same frame from FX6 clip. (FX6 n)
CTRL/7 Replace current frame with same frame from FX7 clip. (FX7 n)
CTRL/8 Replace current frame with same frame from FX8 clip. (FX8 n)
CTRL/9 Replace current frame with same frame from FX9 clip. (FX9 n)

CTRL/SHIFT/1 Replace range with same range from FX1 clip. (FX1 s,e)
CTRL/SHIFT/2 Replace range with same range from FX2 clip. (FX2 s,e)
CTRL/SHIFT/3 Replace range with same range from FX3 clip. (FX3 s,e)
CTRL/SHIFT/4 Replace range with same range from FX4 clip. (FX4 s,e)
CTRL/SHIFT/5 Replace range with same range from FX5 clip. (FX5 s,e)
CTRL/SHIFT/6 Replace range with same range from FX6 clip. (FX6 s,e)
CTRL/SHIFT/7 Replace range with same range from FX7 clip. (FX7 s,e)
CTRL/SHIFT/8 Replace range with same range from FX8 clip. (FX8 s,e)
CTRL/SHIFT/9 Replace range with same range from FX9 clip. (FX9 s,e)

CTRL/SHIFT/ALT/1 Interpolate current frame n using n-1 and n+1 as source frames. (I1 n)
CTRL/SHIFT/ALT/2 Interpolate 2 frames starting at current frame n, using n-1 and n+2 as source frames. (I2 n)
CTRL/SHIFT/ALT/3 Interpolate 3 frames starting at current frame n, using n-1 and n+3 as source frames. (I3 n)
CTRL/SHIFT/ALT/4 Interpolate 4 frames starting at current frame n, using n-1 and n+4 as source frames. (I4 n)
CTRL/SHIFT/ALT/5 Interpolate 5 frames starting at current frame n, using n-1 and n+5 as source frames. (I5 n)
CTRL/SHIFT/ALT/6 Interpolate 6 frames starting at current frame n, using n-1 and n+6 as source frames. (I6 n)
CTRL/SHIFT/ALT/7 Interpolate 7 frames starting at current frame n, using n-1 and n+7 as source frames. (I7 n)
CTRL/SHIFT/ALT/8 Interpolate 8 frames starting at current frame n, using n-1 and n+8 as source frames. (I8 n)
CTRL/SHIFT/ALT/9 Interpolate 9 frames starting at current frame n, using n-1 and n+9 as source frames. (I9 n)

CTRL/SHIFT/ALT/? Interpolate Range. Ie in Avisynth Inclusive mode, where range is 100,101 [2 frames] then result = (I2 100)
The specified range start and end are the outer bad frames.

CTRL/DELETE Delete current frame. (DEL n)
CTRL/SHIFT/DELETE Delete range (DEL s,e)

CTRL+SHIFT+ALT+PAUSE is TERMINATE Program (or close via System Tray icon). PAUSE is also known as BREAK, usually next to Scroll Lock.

NOTE: A range shown in VDub status bar as eg "Selecting Frames 100-102(2 frames)" represents frames 100 and 101, frame 102 is exclusive and
does not count. By default, SawBones uses Inclusive END frame ranges (same as Avisynth where end frame DOES count).
You can change the default behaviour to behave the same as VDub by Running SawBones.Exe at least once and changing the auto created
SawBones.ini file contents from "RangeEndIsExclusive=0" to "RangeEndIsExclusive=1". When using VDub "RangeEndIsExclusive=1" mode,
we subtract 1 from the End Frame to convert to Avisynth End Frame Inclusive specification when writing range to NotePad command.
When sending a command using VDub exclusive mode with a status bar range of "100-100(frames=0)", it will beep and show a "No Frames"
type error message for a few seconds, in Avisynth Inclusive mode it will send a 1 frame range command to the NotePad window.

It is easy to see what is happening as the NotePad window will be visible and after each insertion into NotePad, an ENTER key
will also be sent to move the cursor down one line, each command is on its own line. If you make a mistake, it is easy to
just switch to NotePad window and delete the erroneous line.

Where same frames are flagged for replacment mulitple times, later one will take precidence.
All frame/range deletes will be done AFTER replacements, multiple deletes on same frame will only result on single frame deletion.
Already Interpolated frame/range CANNOT be replaced and will produce an error (In avs script), but they can be deleted.

FrameSurgeon.avs requires MvTools, GScript, RT_Stats, FrameSel, ClipClop and Prune Plugins.

AutoiIt compiled executable with source provided, just click Menu Tools/build to create executable (In Scite4AutoIt3 editor).
Requires AutoIt3 and Scite4AutoIt3 editor to re-build executable.

EDIT: You could use an input Avisynth Stacked clip for viewing multiple clips side by side to choose best FXn option.

poisondeathray

27th March 2017, 17:43

Yes, I seen results. They are quite good.

Can you post some examples ?

I have never understood why people insist on making AVISynth into an editing tool. It really is not well-suited to that job. But, if you want to do it, knock yourself out and have fun!

I completely agree . Once people actually use the other tools , they will never go back to using avisynth for those types of operations. Different tools better suited for different things

Along similar lines, a way to use NLE is a multicam edit where you have more than 2 layers. You have multiple iterations/settings of videos on different layers (e.g. large block size , smaller block size, dct=1, etc....) and just switch between them easily . It would take easily 20-30x longer to do this type of edit in avisynth

MysteryX

27th March 2017, 18:15

Of course you can use ClipClop, but to do that you already have to know frame numbers for your in/out points. To get those, you've already done all the work, presumably in your NLE. In Vegas (my NLE), once I arrive at the frame where the switch should be made, I just make it then and there. It takes less time to finish the job "on site" than actually type or write out the frame number, transfer those numbers to a script, and then execute the script. I see zero benefit to that workflow: it adds extra steps, takes more time, and doesn't let me nudge or make slight changes easily.

I have never understood why people insist on making AVISynth into an editing tool. It really is not well-suited to that job. But, if you want to do it, knock yourself out and have fun!
Not everybody uses Sony Vegas.

I also don't like writing Avisynth manually.

What I personally do is write a software that generates Avisynth code for my needs in a simple way, and then handles all the files and encoding steps.

Avisynth plugins must provide all the features, and then I can write software interface to make the job as simple as it can be. In this case, it's definitely possible to create a window where you can navigate the video frame by frame and mark the positions.

It shouldn't be that hard to do. The hardest part would be to open the Avisynth scripts directly in .NET without using Windows Media Player to do the preview; but others have already written that code.

It seems you know what you're doing. I'd just like to better understand the procedure you are using, and perhaps see some samples. If there is considerable improvement over Interframe, I might decide to program it. I see no reason why this couldn't be done. Plus, you say that the fade-in and fade-out between clip segments requires extra time. I see no reason why this wouldn't be automated.

kolak

27th March 2017, 19:29

Can you post some examples ?

Don't have them anymore, but could not do it anyway.
I've seen some sample where fallback occurred and it was better than massive morphing artefacts which typically appear in such a places.

MysteryX

28th March 2017, 00:21

Don't have them anymore, but could not do it anyway.
I've seen some sample where fallback occurred and it was better than massive morphing artefacts which typically appear in such a places.
I'd like to see it. Doing a blend of both algorithms on the transition frames should make it smooth; perhaps even gradual blending over 2 or 3 adjacent frames. All automated of course.

I'd just like to have more specific details and algorithms to give it a try.

I believe Interframe's anti-artifact results in disabling frame interpolation when it detects artifacts, which isn't ideal either.

Also, SVP doesn't support YV24, but does other libraries allow for YV24 processing? That's one area that would make a considerable difference in my case because I do a frame double after Interframe.

MysteryX

28th March 2017, 03:03

I re-read the whole thing to better understand. I have 2 questions.

I play the video, usually at 1.5x - 2x normal speed (to get through it in a hurry).
How do you play it at 2x speed? In order to see the result, all the frames must be calculated. You won't reach 2x speed unless the processing is faster than real-time, which it won't if it's slower than SVP.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.
How do I do frame blending interpolation in Avisynth? Are there other methods you use?

StainlessS

28th March 2017, 03:15

How do I do frame blending interpolation in Avisynth?

Well for blending, ConvertFPS :- http://avisynth.nl/index.php/FPS.

Play at 2x speed, presume he just does AssumeFPS(FrameRate*2.0). [or equivalent in Vegas]

MysteryX

28th March 2017, 03:32

I have made a quick test to compare Interframe(preset="smooth") with jm_fps. This is part of a script that does more processing, and does an extra frame double which makes the changes more visible. This is a test on very bad quality content to see how it behaves.

http://screenshotcomparison.com/comparison/204832

http://screenshotcomparison.com/comparison/204833

The difference is huge. It's much better than I expected. Worth investigating some more. Plus, it allows processing in YV24.

Can the jm_fps script work in 16-bit, using either DitherTools hack or native AVS+ format?

I did further tests in YV24, and strangely, it generates a LOT of strong artifacts!! This seems more like a bug.
http://screenshotcomparison.com/comparison/204834

MysteryX

28th March 2017, 03:32

Play at 2x speed, presume he just does AssumeFPS(FrameRate*2.0). [or equivalent in Vegas]
This won't show you the interpolation artifacts.

StainlessS

28th March 2017, 03:37

This won't show you the interpolation artifacts.

I never said that it did. John said "to get through it in a hurry". (got good high speed eyes has John).

EDIT: Note, John's script keeps every even output frame same as input, at double rate.

StainlessS

28th March 2017, 03:51

EDIT: Note, John's script keeps every even output frame same as input, at double rate.

Oops, no it does not. John's original script did but manolito left that bit out.

Here John's script that I was talking about:- https://forum.doom9.org/showthread.php?p=1788725#post1788725

And here an alternative which does same for any integer multiple rate, eg if multiply by 3, then every third frame is an original frame.
MulRate():- https://forum.doom9.org/showthread.php?p=1789461#post1789461

MysteryX

28th March 2017, 03:54

One issue I'm seeing compared to Interframe is strange warping on text objects -- and that's something that's recurring throughout the whole videos.

Quite a lot of artifacts too, in some scenes.

By truncating my script and removing whatever runs after jm_fps, I'm able to play it at a decent speed and visualize the artifacts at about half speed. Anything else won't show me where it fails. I'm concerned, however, about recurring artifacts such as on texts and subtitles.

Perhaps the jm_fps function would also benefit from exposing a few parameters to tweak settings.

StainlessS

28th March 2017, 04:29

I did further tests in YV24, and strangely, it generates a LOT of strong artifacts!! This seems more like a bug.
http://screenshotcomparison.com/comparison/204834

Yip, seems like a bug to me too. Post a few frames that produce same as your posted image, Pinterf is likely to want to see that.

MysteryX

28th March 2017, 05:45

Here's a clip. Encoding speed is about the same using this script in YV24 vs Interframe in YV12, in my case.

Here's a sample clip. (https://mega.nz/#!7VpRARjI!exq4QdhnCC0zGEuuuEe6udwruUev_lS9La6RniscSqI)

mvtools2 *does* support 16-bit processing with Pinterf's version; but I don't believe it supports stack16 format (which I'm still currently using)

MysteryX

28th March 2017, 06:45

Another question. How does SVP's artifact masking feature work?

At a minimum, it would make sense to keep that feature and make it revert to frame blending on the partial areas it detects, before going through the video manually and marking whole frames.

johnmeyer

28th March 2017, 07:51

How do you play it at 2x speed? In order to see the result, all the frames must be calculated. You won't reach 2x speed unless the processing is faster than real-time, which it won't if it's slower than SVP.You are assuming a playback method which achieves faster than real time playback by dropping frames. However, a good NLE can play frames at a faster speed, spitting out 30, 40, 50 or more fps. In Vegas, this can be stretched even further if you select one of the spatially degraded preview resolutions. By moving fewer pixels around, it permits playback at even higher speeds before frames must be dropped to keep up.

I'm not sure what you meant by your reference to SVP because when I perform the operation of cutting between the motion estimated version and the frame blended version, both those videos have already been rendered, i.e., I am not frame serving through an AVISynth script into Vegas via SVP, or anything else.

kolak

28th March 2017, 11:20

kolak

28th March 2017, 11:23

I believe Interframe's anti-artifact results in disabling frame interpolation when it detects artifacts, which isn't ideal either.

There are different modes for this. There is one which forces only interpolated frames, not original or blended. Mode 3 if I remember well.
There is no trial for Tachyon. License is something like 20K£.

MysteryX

28th March 2017, 16:33

You are assuming a playback method which achieves faster than real time playback by dropping frames. However, a good NLE can play frames at a faster speed, spitting out 30, 40, 50 or more fps. In Vegas, this can be stretched even further if you select one of the spatially degraded preview resolutions. By moving fewer pixels around, it permits playback at even higher speeds before frames must be dropped to keep up.

I'm not sure what you meant by your reference to SVP because when I perform the operation of cutting between the motion estimated version and the frame blended version, both those videos have already been rendered, i.e., I am not frame serving through an AVISynth script into Vegas via SVP, or anything else.
Ah, you first generate the 2 videos, and THEN edit the 2 timelines of regular AVI files. That makes sense, and yes would allow playing in real-time -- except that the files must be generated first which takes more time.

Your source quality is terrible :)
That's why I'm trying to improve them. I have a bunch of such VCDs, all produced between 2002 and 2006. Unfortunately, in terms of content, I haven't seen anything produced after 2006 from Thailand that compares to these older "classic" releases. And even today, in Thailand, many people are still playing these songs from 2002 to 2006.

MysteryX

28th March 2017, 16:55

I'd see a few improvements to the jm_fps script.

1. First, exposes a few arguments to tweak settings (such as block size).

2. When doubling frame rate, keep source frames

3. Implement SVP/Interframe's artifact masking -- because often the artifacts are small edges of objects, and it would be a waste to not only take the time to mark those frames one by one, but also to waste the whole frame for a partial object that often could be fixed automatically. Of course not all artifacts will be detected in that way, but if half are remove, that's half less work left to do -- and done better, with partial masks.

I'm not familiar enough with mvtools (I know nothing about how it works) to implement those.

StainlessS

28th March 2017, 18:39

You could play with settings via the posted code I linked as MulRate(), or ChangeFrameRate():- https://forum.doom9.org/showthread.php?p=1789461#post1789461
After doing so, then decide which external settings you wish to keep and remove the rest.
The big list of args to ChangeFrameRate() were intended for experimentation purposes, and will be only a few milliseconds slower before frame serving starts.

Might also be an idea to detect almost identical FrameRates, eg 29.97 and 30.0 FPS and do an AssumeFPS(...,sync_audio=true).ResampleAudio(OriginalSampleRate)
instead. Also cope with similar 29.97 -> 60.0 via AssumeFPS and integer DoubleRate() to 60.0 FPS, keeping alternate original frames if required.
EDIT: Code to keep original frames where integer multi-rate embedded as nested function within MulRate() function.

Function MulRate(clip c,Bool "Blend",Bool "Flow",int "mul",Bool "ILeave",
\ Bool "Prefilter",Bool "Chroma",Bool "Chroma1",Bool "Chroma2",Int "BlkSize",Int "Recalc", Int "OverLap",Int "OverLap1",Int "OverLap2",
\ Int "SearchParam",Int "SearchParam1",Int "SearchParam2",Int "thSAD1",Int "thSAD2",Float "Flow_ml",Int "Block_Mode") {
Function MulRate_EvS(String s,int n,int mul) {
s="SelectEvery("+String(mul)+","+String(n)+")"+((s!="")?","+s:s)
# RT_DebugF("s='%s'",s,name="MulRate: ")
return (n>1)?MulRate_EvS(s,n-1,mul):s
}
c
mul=default(mul,2)
Ileave=Default(Ileave,True)
ChangeFrameRate(Blend,Flow,c.FramerateNumerator*mul,c.FramerateDenominator,Prefilter,Chroma,Chroma1,Chroma2,BlkSize,Recalc,
\ OverLap,OverLap1,OverLap2,SearchParam,SearchParam1,SearchParam2,thSAD1,thSAD2,Flow_ml,Block_Mode)
(ILeave&&mul>=2) ? Eval("Interleave(c,"+MulRate_EvS("",mul-1,mul)+")") : NOP
Return Last
}

EDIT: Fragment of code to detect almost identical framerate, that I use in some scripts.
May not be best available for the purpose, but what I use (the subject tends to result in unresolved arguments on forums).

Function IsClose(Float a, Float b, Float threshold) {
threshold=Abs(threshold)
return ((abs(a-b)+threshold) <= threshold*2.0)
}

MysteryX

28th March 2017, 18:49

SVP/Interframe has different anti-artifact strengths. The downside of setting it higher is that it disables frame interpolation wherever it detects potential artifacts.

If we fallback to frame interpolation, then the downside of such artifact removal is lesser and it can safely be set higher.

As for cutting/merging frames using different methods, that's exactly what SVP/Interframe does with artifact masking, and it does give a good result.

2 areas of improvement with artifact masking
- Fallback to frame blending
- blending of both methods on frames adjacent to where masking occurs

MysteryX

28th March 2017, 22:11

How is this artifact removal script? It already supports frame blending to fill the masks.
http://avisynth.nl/index.php/YFRC

poisondeathray

28th March 2017, 22:16

2. When doubling frame rate, keep source frames

Having the option to do either is a good idea.

Source frames are kept by default for mvtools2 when you have integer framerate multiples. Pros/cons eitherway. Conventional thinking or "knee jerk" reaction would say keep source frames... but...

Interpolation generally creates blurrier , lower quality frames . On some types of material (eg. high quality source with lots of details, or grainy sources, or noisy sources, ) , the difference is very noticable and results in a "strobing" or flickering quality . This difference is farther amplified in scenarios where the goal is slow motion (longer display time per frame, you "see" the differences more easily) . In those types of scenarios resampling all frames is sometimes preferrable. On lower quality source, or ones with low detail , the difference between frames is not as noticable.

MysteryX

28th March 2017, 23:11

Totally agree. I hate this flickering effect.

I also want to point out that although I understand the skepticism with automatic artifact removal, I don't think it's a good idea to not use it. There are lots of small artifacts that can be detected and removed that we just can't do by hand. We'd replace whole frames while it can replace only the affected areas. We still probably have to review manually afterwards, but we want to remove as much as possible first. Manual review should be a quick final step; not the bulk of the work.

poisondeathray

28th March 2017, 23:53

I also want to point out that although I understand the skepticism with automatic artifact removal, I don't think it's a good idea to not use it. There are lots of small artifacts that can be detected and removed that we just can't do by hand. We'd replace whole frames while it can replace only the affected areas. We still probably have to review manually afterwards, but we want to remove as much as possible first. Manual review should be a quick final step; not the bulk of the work.

I agree.

Certainly having the option to use "automatic" artifact handling with various algorithms or approaches is a good idea. Sometimes it works great. And if it can reduce the amount of manual work, that's awesome and a "win". But there are cases where it produces worse results (you have a halo effect of blur) , or it misses areas, or messes up good areas - and you actually end up having to do more work than not using it at all . This is where compositing comes into play and you combine layers done with different settings/iterations with masks. You NEED human guidance for this. No way around it.

As much as it would be nice for everything to work automatically with as little work as possible, there are situations that require some user guidance and manual work - if you want good results. (Maybe my definition of "good result" is different than what some people are using). Commercial plugins are a good an example of this - you can plug in motion tracking data, animated splines to guide motion estimation. You use accurate rotoscoped mattes to separate foreground objects and object boundaries so you reduce the edge morphing artifacts (not just "blur" like svp) . But this requires work and human intervention (ie. time). But in "automatic" mode, they are about the same as mvtools2 or svp.

chainik_svp

29th March 2017, 09:01

videoFred

29th March 2017, 14:55

and this's what SVP already does (much simplier of cause ;))

The result is a mixed clip: frame doubling on "difficult" scenes (fast moving objects) and frame interpolation on "easy" scenes (slow panning for example).

and this's called "adaptive interpolation mode" in SVP

How do I set this "adaptive interpolation mode" in Interframe() ?

Fred.

kolak

29th March 2017, 15:06

kolak
SVP made to many speed shortcuts so quality is not as good.

The only one I could think of is "adaptive search radius".
However I found large "adaptive" radius to be better than fixed one especally on regular structures like window blinds. And you can easily turn it off and use good old fixed search raduis.

They find problematic areas and then replace them with frame blended or nearest frame using making and feathering.

https://forum.doom9.org/showthread.php?t=172688#post1741208

and this's what SVP already does (much simplier of cause ;))

The result is a mixed clip: frame doubling on "difficult" scenes (fast moving objects) and frame interpolation on "easy" scenes (slow panning for example).

and this's called "adaptive interpolation mode" in SVP

MysteryX

I have made a quick test to compare Interframe(preset="smooth") with jm_fps. ... The difference is huge.

Yeah, probably because they use completely different options. Don't you want to compare them using the same options? :)

I've used svp a lot, but quality of current mvtools is better (definitely less artefacts).
SVP is way faster of course which is great. Adaptive mode is not the same as I was describing. It switches mode for whole frames which is not the same as Tachyon's fallback. Alchemist uses same way as svp if I'm correct.
I'm not looking for frame doubling, but fps conversion (doubling is very specific case).

Are you saying I should get about the same quality. You can't match setting as they are different compared to mvtools.

chainik_svp

29th March 2017, 15:54

"Current mvtools" is exactly the same it used to be 6 (7?) year ago.

What mvtools option you can't match to SVP? (except for "dct", SVP works with "dct=5", which is SATD, by default)

MysteryX

29th March 2017, 17:59

Yeah, probably because they use completely different options. Don't you want to compare them using the same options? :)
I compared the highest settings preset with SVP/Interframe, compared to jm_fps's script (which honestly I don't know what it does). I don't know where the difference comes from, but there's a very considerable difference.

How can I get the same result with SVP/Interframe?

Then mvtools has the benefit of supposedly supporting YV24 which you didn't care to implement; but then YV24 processing is buggy :)

Johnmeyer talked about some settings and features that were stripped out of SVP, it would be good to hear more specifics on that.

chainik_svp

29th March 2017, 18:15

How can I get the same result with SVP/Interframe?

By using the same options (https://www.svp-team.com/wiki/Plugins:_SVPflow), obviously ;)
Ensure that block sizes, search type(s), penalties and others are the same. And don't forget about defaults that can be different between mvtools and svp for some options... :D

johnmeyer

29th March 2017, 19:09

If some of you want to try out SVP, Interframe, MVtools2, the Yushko Frame Rate Converter script (with masking that attempts to suppress artifacts), then I suggest you try a real torture test.

All of these tools, as well as the expensive commercial equivalents, fail for two reasons: they cannot predict objects that are "revealed" when a foreground object moves across the frame in front of them; and they have trouble with motion of vertical objects, especially when they don't conform to the general motion in the frame (panning across a picket fence is a nightmare, and the legs of people walking in front of the camera often "break" grotesquely). All of these problems are accentuated at lower frame rates because the temporal gap between frames is larger, and the motion estimation algorithm has to make bigger assumptions.

My test clip is at 14.985 fps, exactly half the 29.97 NTSC rate. Even though your software may report it as interlaced, I assure you that it is 100% progressive: it is from my own film transfer of a 1940 Flint Michigan parade. If needed, add a line to your AVISynth script to force the software to treat it as progressive, although if you don't, nothing bad will happen.

Here are the things likely to show problems: the waving flags, the crowd getting "revealed" as floats cross in front of them; the rotating spokes on the old cars; the legs of the marching soldiers; the bayonets on the shouldered rifles as the marchers make a left turn; the drummer's drumsticks, and most of all, the moose antlers at the end. Good luck with those. They represent the ultimate challenge.

Here's the clip:

Flint Michigan Parade Clip (14.985 fps; progressive; DV format) (https://www.mediafire.com/?cb739wsyzws3z53)

P.S. If you want to see what this clip looks like when I used MVTools2 to increase fps to 29.97, here is the YouTube version of the all the film I transferred from this event. I increased the frame rate because it makes the film easier to watch, even though it introduces a LOT of artifacts. I did not spend the time to do the manual fixup that I described earlier, because this was a non-paying job for an old friend.

1940 Flint Michigan parade (https://www.youtube.com/watch?v=t8HjRN0rw5M)

I just watched it again, and it reminds me of why I always recommend to those who are doing their own film transfers and restoration that they NOT use motion estimation and instead simply use pulldown, even if the resulting video looks jerky.

poisondeathray

29th March 2017, 21:52

John's parade clip really is "torture", because it has many of the issues where you expect problems with motion vectors and interplation

John mentioned some of these, but common "prototypical" issues that predispose to "failure" include:

1) occlusions. Ie. object layers, foreground , background, z-depth crossing over one another (e.g. walking, legs crossing)
2) repeating patterns ("picket fence")
3) object deformation (e.g. bouncing ball, waving flags in john's video) , the algorithm can no longer accurately track because it "looks" like a different object
4) rotational movements, global and object axial. Object axial is especially bad, because textures are rotated away (different faces, effectively a different object)
5) motion blur - you cannnot distinguish between object boundaries as accurately. Where does one object edge begin or end?
6) compression artifacts
7) illumination changes within a scene
8) complex motions, e.g speed (long "distance" betwen frames or low frame rate), or non standard motion paths, or "random" motions. Camera motion vs. object motion. perspective changes

Understanding why something fails is the first step in actually doing something about it.

"Covering up" bad results is one approach; But I'm more interested in higher quality, address the underlying problem in the first place approach. This is for non realtime scenarios , like the OP wanted in the first post.

The underlying problem for issues like occlusions is the inability to delineate boundaries of objects. As one object crosses over another, or background objects, the vectors are no longer accurate. So one way around this is to guide the vectors or edit the optical flow data. Commercial applications use motion tracking (track points) , and mattes to indicate object layers and edges, to help guide the motion estimation . This isn't "covering up" a bad result, this is helping to get the interpolation right in the first place. There is user input , but it's not entirely "manual" the way some people think it is. You don't manually create masks per frame, every frame - you might only have to do every nth frame, and minor adjustments because of mask interpolation) , and there is motion tracking to assist with rotoscoping , mask tracking, as well as semi automated tools like rotobrush which is basically rototracking

For some types of complex motions, such as complex camera motions, there are "camera solves" which basically track the camera motion. ie. You can separate the camera motion from object motions to help simplfy the interpolation. You need rock stable VFX tracking for this, not something like deshaker or mercalli. You add back in the camera motion once everything is fixed and composited back together

So we start with a more simple, prototypical "fail" scenario. Low level tests are useful in understanding what the problem is, what parameters to tweak, developing strategies to improve. This is a synthetic low level test. Lagarith RGB . A repetitive checker background, simple object rotation, but no object axial rotation, or deformation. The goal is to double the frames. What settings mvtools2 or svpflow or other would you use ? It's a simple solve with 1 matte for commercial plugins, and this is basically automatic when you use a motion tracker - you draw a white circle and attach it to the motion track data. That luma matte (or "mask" in avisynth terms) input to the plugin is what helps in assisting the motion estimation. For matte generation there are free tools too , with motion trackers such as blender, natron, nuke (non commercial version).
https://www.mediafire.com/?ocs0abw49wlj4fo

I asked this before in this forum , you can visualize mvtools2 mv's with mshow(), but there is no way that I know of to re-input the edited data back in. But that would be one way to improve results. Some commercial applications can manipulate input vector fields directly and almost all of them have matte inputs (which indirectly assist with increasing mv accuracy) . Or is there a way to use mattes (or masks) in avisynth to assist mvtools2 motion estimation ? (ie. not just as an alpha channel or layer overlay) . But how can avisynth/mvools2 interpolation be improved using commercial approaches like mattes ? Things like multiple inputs, splines, mattes, motion track points are what separate commercial "pro" interpolation tools but I'd like to see some of them adapted for avisynth but I don't see a way.

A free / open source example editing motion data to improve results is slomovideo .
https://www.youtube.com/watch?v=aLtIvEiDD2k

MysteryX

29th March 2017, 23:10

Improving the failed areas is one topic. Improving the successful areas is another topic. Why do I get a much clearer and better defined image using jm_fps?

poisondeathray

29th March 2017, 23:23

Why do I get a much clearer and better defined image using jm_fps?

First thing is make sure you seek linearly with svpflow . Go back about 30 frames and advance forward. It's like dirt removal or srestore - non linear seeks cause inconsistent results , typically lots of blurring

But what settings are you using for svpflow ?

It looks like jm_fps is using mrecalculate and removegrain prefilter. svpflow doesn't have mrecalculate , it uses refine. Did you try matching the other settings ?

johnmeyer

29th March 2017, 23:30

That's a great post poison!

Motion tracking/rotoscoping is definitely the only way to deal with some of these issues. That technology lets you define real objects, like legs, and then track those, rather than simply -- and stupidly -- just arbitrarily breaking up each frame into blocks of pixels (4x4, or 8x8, or 16x16, etc.) and then tracking blocks which really have nothing to do with the actual content of the video.

MysteryX's states that he gets better results with some script than others. He makes it sound like he always gets better results with jm_fps, but I'll bet a few dollars that, on other clips, he would get better results with some other approach. It all depends on the nature of the specific scene you use.

As I have said already in this thread, block size is the single most important thing to vary if you want to optimize the results for a given clip. Given that this is true (at least in my experience), I have what might be an interesting idea:

Would it make sense to devise a script which provided the ability to alter the block size, scene by scene, and then choose which one to use? It would certainly be trivial to write a script which simultaneously created 4x4, 8x8, 16x16, and 32x32 results. The "trick" would be devising good enough artifact detection to choose between them. My sense is that, if such detection could be written, the result would be more pleasing than using similar detection to try to fix a bad result using masks. I haven't tried the script posted here which does that, but I have very low expectations that it would consistently produce a result that was much better than a script without it, especially since others have already reported that sometimes it actually makes things worse (halos and blurring is what I think one person reported).

kolak

29th March 2017, 23:32

kolak

29th March 2017, 23:39

By using the same options (https://www.svp-team.com/wiki/Plugins:_SVPflow), obviously ;)
Ensure that block sizes, search type(s), penalties and others are the same. And don't forget about defaults that can be different between mvtools and svp for some options... :D

Tried changing many settings.
mvtools is simply producing less artefacts. svp quite often has double edges on objects (even on not difficult scenes).
Also- older (much older) versions of svp dlls actually produced better results. The newer version the more artefacts :)

You are saying that svp should be as good as mvtools? I would like to get match, as svp is way faster.

kolak

29th March 2017, 23:46

Would it make sense to devise a script which provided the ability to alter the block size, scene by scene, and then choose which one to use? It would certainly be trivial to write a script which simultaneously created 4x4, 8x8, 16x16, and 32x32 results. The "trick" would be devising good enough artifact detection to choose between them. My sense is that, if such detection could be written, the result would be more pleasing than using similar detection to try to fix a bad result using masks. I haven't tried the script posted here which does that, but I have very low expectations that it would consistently produce a result that was much better than a script without it, especially since others have already reported that sometimes it actually makes things worse (halos and blurring is what I think one person reported).

ffmpeg motion estimation new filter has adaptive block size switching, if I'm correct, but results are not very good.

chainik_svp

30th March 2017, 07:25

Why do I get a much clearer and better defined image using jm_fps?

the thing is - no one will tell
in mvtools's world even a little change of some "secondary" option lake "lambda" or "pnew" can change the vectors field significantly

lets compare the magic "jm_fps" w/o using pre-filter to see if it really helps ;)

> svpflow doesn't have mrecalculate , it uses refine.

it's the same thing
MRecalculate can switch from any block size to any other block size while SVP's "refine" can only split a block into four smaller ones, but this's exactly what we want here

MysteryX

11th April 2017, 03:49

I'm just coming back from holidays. No further updates on this since?

I'd be curious to know where the difference of quality comes from.

MysteryX

15th April 2017, 03:14

I've done a bit of progress on this. I've taken both scripts and stripped it down to the actual scripts being generated.

Here I use Interframe with Tuning="Smooth". RemoveGrain has a very minimal effect and isn't the cause of the difference.

All I can do at this point is contemplate the huge difference between both scripts. The 2 functions' parameters are completely different so it's difficult to translate one with the other. Perhaps someone else can chime in from there.

Btw, does MVTools2 allow implementing artifact removal in the same way as SVP?

function InterFrameProcess(clip Input) {
SuperString = "{scale:{up:0,down:4},gpu:1,rc:false}"
VectorsString = "{block:{w:8,overlap:2},main:{search:{distance:0,coarse:{distance:-10,bad:{sad:2000}}}},refine:[{thsad:250}]}"
SmoothString = "{rate:{num:60,den:1,abs:true},algo:23,mask:{area:150,area_sharp:1.2},scene:{blend:true, mode:0}}"

# Make interpolation vector clip
Super = SVSuper(Input, SuperString)
Vectors = SVAnalyse(Super, VectorsString)

# Put it together
smooth_video = SVSmoothFps(Input, Super, Vectors, SmoothString, url="www.svp-team.com", mt=2)
smooth_video
}

function jm_fps(clip source) {
fps_num = 60
fps_den = 1

prefiltered = RemoveGrain(source, 22)
super = MSuper(source, hpad = 16, vpad = 16, levels = 1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = 16, overlap = 4, search = 3, dct = 0)
forward = MAnalyse(superfilt, isb = false, blksize = 16, overlap = 4, search = 3, dct = 0)
forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

return out
}

MysteryX

17th April 2017, 23:51

Here's YFMC that implements artifact removal, but only supports frame doubling.

function YFRC(clip clp_Input, int "BlockH", int "BlockV", int "OverlayType", int "MaskExpand")
{
#Yushko Frame Rate Converter 2x
#ColorSpace: YV12
#OverlayType: 0 - soft (blend), 1 - hard (SelectOdd)

clp_input = ConvertToYV12(clp_input) # script produce YV12 colorspace (3 times faster than YUY2!!!)
clp_Super = MSuper(clp_Input.blur(1), chroma=true, pel=2) # blur(1) - smooth edges for better analysing
ox = clp_Input.width()
oy = clp_Input.height()
fps_num = FrameRateNumerator(clp_input)*2 # Numerator , always 2X
fps_den = FrameRateDenominator(clp_input) # Denominator, always same as input clip

fps_num = ((fps_num==0||fps_den==0)) ? FramerateNumerator(clp_input)*2 : fps_num
fps_den = ((fps_num==0||fps_den==0)) ? FramerateDenominator(clp_input) : fps_den
MaskExpand = default(MaskExpand, 1) # 1 or 2
OverlayType = default(OverlayType, 0) # 0 - like ConvertFPS (blend); 1 - like ChangeFPS (strong);
blendSOFT = (OverlayType==0) ? clp_Input.ConvertFPS(fps_num, fps_den).SelectOdd() : DeleteFrame(clp_Input, 0)
BlockH = default(BlockH, 16) # use 8 for 320x240 (WEB); 16 for 720x576 (SD); 32 for 1280x720 (720p HD); 32 for 1920x1080 (1080p HD)
BlockV = default(BlockV, 16) # use 8 for 320x240 (WEB); 16 for 720x576 (SD); 32 for 1280x720 (720p HD); 32 for 1920x1080 (1080p HD)
blendHARD = DeleteFrame(clp_Input, 0) # SceneChange detection

bw1_vec116 = clp_Super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=true , chroma=false, search=1, searchparam=1, truemotion=true, lambda=2000, global=true, dct=0, divide=2)
fw1_vec116 = clp_Super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=false, chroma=false, search=1, searchparam=1, truemotion=true, lambda=2000, global=true, dct=0, divide=2)
ErrorMask16L = MMask(clp_input, bw1_vec116, kind=1)
ErrorMask16R = DeleteFrame(MMask(clp_input, fw1_vec116, kind=1), 0)
ErrorMask16 = Overlay(ErrorMask16L, ErrorMask16R, opacity=0.5, mode="Lighten")#.ColorYUV(gain_y=256)
SceneChange = MSCDetection(clp_input, bw1_vec116, thSCD2=130)
FPSconverted16 = clp_input.MFlowFps(clp_input.MSuper(levels=1), bw1_vec116, fw1_vec116, num=fps_num, den=fps_den, blend=false, mask=0) #mask=0 - doesn't matter what mode is

CircleExpand = mt_circle(radius=MaskExpand, zero=true)
CircleInpand = mt_circle(radius=1 , zero=true)
ErrorMask16 = ErrorMask16.BicubicResize(Round((Ox/BlockH)/4)*4, Round((Oy/BlockV)/4)*4)
\ .mt_expand(mode=CircleExpand).mt_inpand(mode=CircleInpand).mt_binarize(64).Blur(1).BicubicResize(ox, oy)

ClipToReturn = mt_merge(SelectOdd(FPSconverted16), blendSOFT, ErrorMask16, luma=true)
ClipToReturn = mt_merge(ClipToReturn, blendHARD, SceneChange, luma=true)
ClipToReturn = Interleave(clp_Input, ClipToReturn)

return ClipToReturn
}

Here are sample screenshots to compare methods, with frame doubling in all cases.

Interframe
https://s9.postimg.org/o7mfaszff/Interframe.png (https://postimg.org/image/o7mfaszff/)

fm_fps
https://s9.postimg.org/41ixbx3rv/jm_fps.png (https://postimg.org/image/41ixbx3rv/)

YFRC
https://s9.postimg.org/5hufu26or/yfrc.png (https://postimg.org/image/5hufu26or/)

The best would be to take jm_fps and add YFRC's artifact removal.

johnmeyer

18th April 2017, 00:45

The only result that is even close to acceptable is the jm_fps. I think that is based on some code I posted. If that is the case, I can tell you that there is absolutely nothing special about it, and it is almost identical to some of the sample code in the MVTools2 documentation.

The reason I am writing this is that I wonder if there is something missing, or some setting that isn't quite right for the other two scripts (Interframe and yfrc)? I say this because of all the fuzziness on sections that weren't moving that fast. To me, they looked blended rather than interpolated. I know that some of these scripts have the ability to do blending rather than ME, based on metrics, and I wonder if that is what is happening.

I think the results you get for this simple case should be much closer to each other.

MysteryX

18th April 2017, 02:06

Agree.

Solution 1 is to add YFMC's artifact removal to jm_fps script.

Solution 2 is to figure out what setting is missing to fix SVP/Interframe.

Solution 2 would be best.

Having a "good" and "bad" script using the same DLL is going to make it easier as SVP and MVTools2 have a completely different syntax.

MysteryX

18th April 2017, 02:55

Alright I think I'm getting at something.

jm_fps
https://s16.postimg.org/4v898rhbl/jm_fps.png (https://postimg.org/image/4v898rhbl/)

YFMC
https://s16.postimg.org/5m0zejjox/yfmc.png (https://postimg.org/image/5m0zejjox/)

YFMC with artifact removal disabled
https://s16.postimg.org/y06eyf78x/yfmc_noremoval.png (https://postimg.org/image/y06eyf78x/)

Artifact auto-removal seems to be the culprit. Whenever we see the image doubling effect, that's frame blending due to artifact removal.

MysteryX

18th April 2017, 03:13

There's more than artifact removal

SVP
https://s13.postimg.org/42yxgfyv7/SVP1.png (https://postimg.org/image/42yxgfyv7/)

SVP with artifact removal disabled
https://s13.postimg.org/d9h83q43n/SVP_noremoval.png (https://postimg.org/image/d9h83q43n/)

jm_fps
https://s13.postimg.org/xsc48si0z/jm_fps.png (https://postimg.org/image/xsc48si0z/)

What I had started doing was to take YFMC script and rewrite it to look like jm_fps, and looking at the difference of script and result every step of the way.

raffriff42

18th April 2017, 13:32

YFRC is extremely conservative regarding artifacts - it falls back to frame blending a lot. For some sources, that is exactly what is needed.

For high motion stuff, I like YFRC a lot. For me, false interpolation artifacts are way more annoying than blended frames.

Here is a version I have made with comments added and variables renamed for clarity, and more importantly, a way to see the artifact mask in action so it can be tuned to your liking: YFRC2(..., output="over"). You can also output the mask only (output="mask") to use with a different interpolator like jm_fps.# Version: "01dd-10mm-2015yyyy" modded raffriff42 31-Mar-2017
# Author: RunForLife(Oleg Yushko) http://videomontazh.com.ua
# http://avisynth.nl/index.php/YFRC
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Yushko Frame Rate Converter 2x, modded 2017 raffriff42
### doubles the frame rate with strong artifact detection and scene change detection.
##
## YV12/YV24/Y8/YUY2
##
## @ hardblend - how to handle scene change and artifact fallback:
## if false (default), blend like ConvertFPS; else repeat like ChangeFPS
##
## @ weakmask - if true, allow a few more artifacts to get through; default false
##
## @ output - (auto|inter|none|mask|over) default "auto"=normal artifact masking;
## "inter"=interpolation only; "none"=ConvertFPS only;
## "mask"=mask only; "over"=mask as cyan overlay for debugging
##
function YFRC2(clip C,
\ bool "hardblend", bool "weakmask",
\ string "output")
{
ox = C.Width
oy = C.Height

fps_num = C.FrameRateNumerator * 2
fps_den = C.FrameRateDenominator

bigblock = (ox>1270 || oy>710)
BlockH = (bigblock) ? 32 : 16
BlockV = (oy>710) ? 32 : 16

hardblend = Default(hardblend, false)
weakmask = Default(weakmask, false)
output = Default(output, "auto")

blendHARD = C.DeleteFrame(0)
blendSOFT = (hardblend)
\ ? blendHARD
\ : C.ConvertFPS(fps_num, fps_den).SelectOdd

## blur(1) - smooth edges for better analysing
sup = MSuper(C.blur(1), chroma=true, pel=2)

## TODO: try merge w/ jm_fps
bak = sup.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=true,
\ chroma=false, search=1, searchparam=1, truemotion=true,
\ lambda=2000, global=true, dct=0, divide=2)
fwd = sup.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=false,
\ chroma=false, search=1, searchparam=1, truemotion=true,
\ lambda=2000, global=true, dct=0, divide=2)

## "Flow" - MFlowFps double framerate
Flow = C.MFlowFps(C.MSuper(levels=1), bak, fwd,
\ num=fps_num, den=fps_den, blend=false, mask=0)

## "EM" - error or artifact mask
EM = C.MMask(bak, ml=190, kind=1) [** kind=SAD *]
EM = EM.Overlay(
\ C.MMask(fwd, ml=190, kind=1).DeleteFrame(0),
\ opacity=0.5, mode="lighten")

EM = EM.Overlay(
\ C.MMask(bak, ml=64, kind=2).mt_inpand, [** kind=occlusion *]
\ opacity=0.5, mode="lighten")

## mask strength
## TODO: tuning ('ml=' values etc)
EM = (!weakmask) ? EM : EM.Levels(0, 0.7, 255, 0, 191, coring=false)

EM = EM.BicubicResize(Round((ox/BlockH)/4.0)*4, Round((oy/BlockV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ [*.mt_inpand(mode=mt_circle(zero=true, radius=1)) *]
\ .mt_binarize(92)
\ .Blur(1.0)
\ [*.ColorYUV(cont_y=_f2c(2.0)) *]
\ .BicubicResize(ox, oy)

## "Sc" - scene detection
Sc = C.MSCDetection(bak, thSCD2=130)

## the YFRC magic happens
## note hard coded 2x interpolation
B = mt_merge(SelectOdd(Flow), blendSOFT, EM, luma=true)
B = mt_merge(B, blendHARD, Sc, luma=true)

R = (StrCmpi(output, "auto")==0) [** auto: artifact masking *]
\ ? Interleave(C, B)
\ : (StrCmpi(output, "inter")==0) [** inter: interpolation only *]
\ ? Flow
\ : (StrCmpi(output, "none")==0) [** none: ConvertFPS only *]
\ ? (hardblend)
\ ? C.ChangeFPS(fps_num, fps_den)
\ : C.ConvertFPS(fps_num, fps_den)
\ : (StrCmpi(output, "mask")==0) [** mask: mask only *]
\ ? EM.ConvertFPS(fps_num, fps_den).Grayscale.Invert
\ : (StrCmpi(output, "over")==0) [** over: mask as cyan overlay *]
\ ? Flow.Overlay(
\ MergeRGB(BlankClip(EM), EM, EM)
\ [*.RGBAdjust(2, 2, 2)*]
\ .ConvertFPS(fps_num, fps_den),
\ mode="Add", opacity=0.25)
\ : Assert(false,
\ "YFRC2: 'output' not one of (auto|inter|none|mask|over)")
return R
}

StainlessS

18th April 2017, 13:52

Looks good Raff.
I note that you use Blur(1.0) in "sup = MSuper(C.blur(1), chroma=true, pel=2) ",

Dont see anything wrong in that, however Didée's goto value for blur is 0.6,
https://forum.doom9.org/showthread.php?p=1508638#post1508638

and he seemed to have some idea what he was doing :)

Just a suggestion, give it a try.

raffriff42

18th April 2017, 14:33

Blur(1) is from the original, I actually have not tried other values.
http://avisynth.nl/index.php/YFRC

MysteryX

18th April 2017, 21:04

YFRC with weak mask is almost as good as jm_fps, with reasonable artifact handling. It looks better than SVP/Interframe, so I'm still not sure what it would take to get those results with SVP.

YFRC definitely is too conservative, but the danger of weak mask is that it leaves artifacts. If we're to do a manual review of videos, instead of marking zones to replace manually, I'd rather have the choice between
- interpolated clip
- weak mask
- strong mask
- blend
- copy

Then we can set a default for the video, then override frames or segments with a different clip.

MysteryX

18th April 2017, 21:51

So far, "weak mask" gives the best results.

Here's the YFMC script with jm_fps + artifact masking

# Version: "01dd-10mm-2015yyyy" modded raffriff42 31-Mar-2017
# Author: RunForLife(Oleg Yushko) http://videomontazh.com.ua
# http://avisynth.nl/index.php/YFRC
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Yushko Frame Rate Converter 2x, modded 2017 raffriff42
### doubles the frame rate with strong artifact detection and scene change detection.
##
## YV12/YV24/Y8/YUY2
##
## @ hardblend - how to handle scene change and artifact fallback:
## if false (default), blend like ConvertFPS; else repeat like ChangeFPS
##
## @ weakmask - if true, allow a few more artifacts to get through; default false
##
## @ output - (auto|inter|none|mask|over) default "auto"=normal artifact masking;
## "inter"=interpolation only; "none"=ConvertFPS only;
## "mask"=mask only; "over"=mask as cyan overlay for debugging
##
function YFRC2(clip C,
\ bool "hardblend", bool "weakmask",
\ string "output")
{
ox = C.Width
oy = C.Height

fps_num = 50 #C.FrameRateNumerator * 2
fps_den = 1 #C.FrameRateDenominator

bigblock = (ox>1270 || oy>710)
BlockH = (bigblock) ? 32 : 16
BlockV = (oy>710) ? 32 : 16

hardblend = Default(hardblend, false)
weakmask = Default(weakmask, false)
output = Default(output, "auto")

blendHARD = C.DeleteFrame(0)
blendSOFT = (hardblend)
\ ? blendHARD
\ : C.ConvertFPS(fps_num, fps_den).SelectOdd

prefiltered = RemoveGrain(C, 22)
super = MSuper(C, hpad = BlockH, vpad = BlockV, levels = 1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = BlockH, vpad = BlockV) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
forward = MAnalyse(superfilt, isb = false, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
forward_re = MRecalculate(super, forward, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
Flow = MFlowFps(C, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

## "EM" - error or artifact mask
EM = C.MMask(backward_re, ml=190, kind=1) [** kind=SAD *]
EM = EM.Overlay(
\ C.MMask(forward_re, ml=190, kind=1).DeleteFrame(0),
\ opacity=0.5, mode="lighten")

EM = EM.Overlay(
\ C.MMask(backward_re, ml=64, kind=2).mt_inpand, [** kind=occlusion *]
\ opacity=0.5, mode="lighten")

## mask strength
## TODO: tuning ('ml=' values etc)
EM = (!weakmask) ? EM : EM.Levels(0, 0.7, 255, 0, 191, coring=false)

EM = EM.BicubicResize(Round((ox/BlockH)/4.0)*4, Round((oy/BlockV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ [*.mt_inpand(mode=mt_circle(zero=true, radius=1)) *]
\ .mt_binarize(92)
\ .Blur(1.0)
\ [*.ColorYUV(cont_y=_f2c(2.0)) *]
\ .BicubicResize(ox, oy)

## "Sc" - scene detection
Sc = C.MSCDetection(backward_re, thSCD2=130)

## the YFRC magic happens
## note hard coded 2x interpolation
B = mt_merge(SelectOdd(Flow), blendSOFT, EM, luma=true)
B = mt_merge(B, blendHARD, Sc, luma=true)

R = (StrCmpi(output, "auto")==0) [** auto: artifact masking *]
\ ? Interleave(C, B)
\ : (StrCmpi(output, "inter")==0) [** inter: interpolation only *]
\ ? Flow
\ : (StrCmpi(output, "none")==0) [** none: ConvertFPS only *]
\ ? (hardblend)
\ ? C.ChangeFPS(fps_num, fps_den)
\ : C.ConvertFPS(fps_num, fps_den)
\ : (StrCmpi(output, "mask")==0) [** mask: mask only *]
\ ? EM.ConvertFPS(fps_num, fps_den).Grayscale.Invert
\ : (StrCmpi(output, "over")==0) [** over: mask as cyan overlay *]
\ ? Flow.Overlay(
\ MergeRGB(BlankClip(EM), EM, EM)
\ [*.RGBAdjust(2, 2, 2)*]
\ .ConvertFPS(fps_num, fps_den),
\ mode="Add", opacity=0.25)
\ : Assert(false,
\ "YFRC2: 'output' not one of (auto|inter|none|mask|over)")
return R
}

jm_fps
https://s7.postimg.org/4uon7sz5z/jm_fps.png (https://postimg.org/image/4uon7sz5z/)

YFMC (weakmask)
https://s7.postimg.org/noag4sxdz/YFMC.png (https://postimg.org/image/noag4sxdz/)

jm_fps + YFMC's artifact removal (weakmask)
https://s7.postimg.org/l88mqyfbb/YFMC-jm_fps.png (https://postimg.org/image/l88mqyfbb/)

With artifact removal, I'm honestly not sure whether YFMC or jm_fps gives the best result...

TODO: make the script work either for frame doubles, or for all frames.

I could create a plugin/script that allows easy switching between various clip versions for manual tweaking.

If we flip through the videos while seeing mask overlays, it would then be easy to say whether to tune settings up or down for each frame. This will require more testing but I'd expect to use weakmask in most cases and fallback to strong mask on a few frames when problems occur.

hum... weakmask does only this. It seems to work so far, but is that enough?

EM = (!weakmask) ? EM : EM.Levels(0, 0.7, 255, 0, 191, coring=false)

We still haven't resolved any fundamental problem, but at least we're getting to "something"

MysteryX

19th April 2017, 03:13

I've done good progress. Modified YFMC with additional arguments, allowed specifying frame rate other than FrameDouble (while still allowing FrameDouble), cleaned up the code, used jm_fps, and changed a bunch of things.

Considering it's containing pieces of code from me, johnmeyer, raffriff42 as well as Yushko, it doesn't make sense anymore to call it "Yushko", and even less YFRC2. I changed the name to FrameRateConverter.

I changed weakmask with MaskStr. It would make sense to allow more settings here, and the code for weak masking was left as a TODO:tweaking. For now it's an int that takes 0(weak) or 1(normal), but that should be changed.

Finally, between YFRC and jm_fps interpolation, we have to decide which gives the best results -- which settings to use for interpolation. From initial testing, both seem to be slightly better than the other in 50% of cases. It might be good to test how each perform in worst cases like the video johnmeyer posted.

# Frame Rate Converter
# Version: 19-Apr-2017
# Authors: Yushko, johnmeyer, raffriff42, MysteryX
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Doubles the frame rate with strong artifact detection and scene change detection.
##
## YV12/YV24/Y8/YUY2
## Requires: masktools2, mvtools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ BlockH - The horizontal block size (default = Width > 1270 || Height > 710 ? 32 : 16)
##
## @ BlockV - The vertical block size (default = BlockH or Height > 710 ? 32 : 16)
##
## @ Blend - Whether to use frame blending for artifact masking (default = true)
##
## @ Output - (auto|inter|none|mask|over) default "auto"=normal artifact masking;
## "inter"=interpolation only; "none"=ConvertFPS only;
## "mask"=mask only; "over"=mask as cyan overlay for debugging
##
## @ MaskStr - The artifact masking strength (1 to 10). A lower value will not change
## artifact detection but will make the mask softer. (Default=7)
##
## @ MaskSAD - Artifact masking strength for bad motion (Default=190)
##
## @ MaskOcc - Artifact masking strength for occlusion (Default=64)
##
## @ thSCD1 - Scene change detection treshold 1 of MSCDetection. 0 to disable. (Default=400)
##
## @ thSCD2 - Scene change detection treshold 2 of MSCDetection. (Default=130)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble",
\ int "BlockH", int "BlockV", bool "Blend", string "Output", int "MaskStr", int "MaskSAD", int "MaskOcc", int "thSCD1", int "thSCD2")
{
Blend = Default(Blend, true)
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlockV = Default(BlockV, Defined(BlockH) ? BlockH : (C.Height > 710 ? 32 : 16))
BlockH = Default(BlockH, C.Width > 1270 || C.Height > 710 ? 32 : 16)
MaskStr = Default(MaskStr, 7)
MaskSAD = Default(MaskSAD, 190)
MaskOcc = Default(MaskOcc, 64)
thSCD1 = Default(thSCD1, 400)
thSCD2 = Default(thSCD2, 130)

B = Blend ? C.ConvertFPS(NewNum, NewDen) : FrameDouble ? C.DeleteFrame(0) : C.ChangeFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
prefiltered = RemoveGrain(C, 22)
super = MSuper(C, hpad = BlockH, vpad = BlockV, levels = 1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = BlockH, vpad = BlockV) # all levels for MAnalyse
bak = MAnalyse(superfilt, isb = true, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
fwd = MAnalyse(superfilt, isb = false, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
fwd = MRecalculate(super, fwd, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
bak = MRecalculate(super, bak, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

## YFMK interpolation
# super = MSuper(C.blur(.6), chroma=true, pel=2)
# bak = super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=true,
# \ chroma=false, search=1, searchparam=1, truemotion=true,
# \ lambda=2000, global=true, dct=0, divide=2)
# fwd = super.MAnalyse(blksize=BlockH, blksizeV=BlockV, isb=false,
# \ chroma=false, search=1, searchparam=1, truemotion=true,
# \ lambda=2000, global=true, dct=0, divide=2)
# Flow = C.MFlowFps(C.MSuper(levels=1), bak, fwd, num=NewNum, den=NewDen, blend=false, mask=0)

## "EM" - error or artifact mask
EMfwd = C.MMask(fwd, ml=MaskSAD, kind=1)
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = C.MMask(bak, ml=MaskSAD, kind=1) [** kind=SAD *]
EM = EM.Overlay(EMfwd, opacity=0.5, mode="lighten")

EM = EM.Overlay(
\ C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand, [** kind=occlusion *]
\ opacity=0.5, mode="lighten")

## mask strength
EM = MaskStr < 10 ? EM.Levels(0, float(MaskStr) / 10, 255, 0, 255 - 9*(10-MaskStr), coring=false) : EM

EM = EM.BicubicResize(Round((C.Width/BlockH)/4.0)*4, Round((C.Height/BlockV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
EM = EM.ChangeFPS(NewNum, NewDen)

## "Sc" - scene detection
ScDetect = thSCD1 > 0 && thSCD2 > 0
Sc = ScDetect ? C.MSCDetection(bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(C)
Sc = Sc.ChangeFPS(NewNum, NewDen)
#Sc = ConditionalFilter(EM, BlankClip(EM, color=color_white), EM, "AverageLuma()", ">", "100")

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = ScDetect ? mt_merge(M, B, Sc, luma=true) : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "inter")==0) [** inter: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? Blend ? C.ConvertFPS(NewNum, NewDen) : C.ChangeFPS(NewNum, NewDen)
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? mt_merge(
\ EM.ConvertFPS(NewNum, NewDen),
\ BlankClip(EM, color=color_white),
\ ScDetect ? sc : BlankClip(EM), luma=true).Grayscale.Invert
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM).ConvertFPS(NewNum, NewDen), mode="Add", opacity=0.30),
\ BlankClip(EM, color=color_darkgoldenrod), ScDetect ? sc.Levels(0, 1, 255, 0, 128, coring=false) : BlankClip(EM), luma=true)
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|inter|none|mask|over)")
return R
}

Edit: found the bug and updated the script. There was a ChangeFps missing. This script is working, all that's left is testing and tweaking some details.

MysteryX

19th April 2017, 19:44

Weak masking is exactly the same artifact detection. The difference is that it doesn't replace a huge area around it. I think weak mask should be the default; I see no reason not to use it. Perhaps the artifact detection itself could be configured?

Here are some tests on the Motion Estimation Torture Clip
Flint Michigan Parade Clip (14.985 fps; progressive; DV format) (https://www.mediafire.com/?cb739wsyzws3z53)

All are interpolated to 60fps and using weak artifact masking.

Frame 112. InterframeSmooth / YFMC / JM
https://s29.postimg.org/t1qpoub37/Interframe_Smooth112.png (https://postimg.org/image/t1qpoub37/) https://s2.postimg.org/wqaitayph/112_YFMC.png (https://postimg.org/image/wqaitayph/) https://s2.postimg.org/6gpgaics5/112_JM.png (https://postimg.org/image/6gpgaics5/)
All 3 are decent but there are slight artifacts with JM

Frame 141. InterframeFilm / InterframeSmooth / YFMC / JM
https://s29.postimg.org/tby886ppf/Interframe_Film141.png (https://postimg.org/image/tby886ppf/) https://s29.postimg.org/vx8e9g4gz/Interframe_Smooth141.png (https://postimg.org/image/vx8e9g4gz/) https://s2.postimg.org/q2dx6pf79/141_YFMK.png (https://postimg.org/image/q2dx6pf79/) https://s2.postimg.org/oytsuquk5/141_JM.png (https://postimg.org/image/oytsuquk5/)
YFMK is better and JM shows some artifacts. Interframe is less detailed.

Frame 582. Interframe does an ugly job while YFMC's artifact masking skips the frames of that section altogether.
https://s29.postimg.org/8gbxwxtib/Interframe_Film582.png (https://postimg.org/image/8gbxwxtib/) https://s2.postimg.org/aw7vfrp6d/582_YFMK.png (https://postimg.org/image/aw7vfrp6d/) https://s2.postimg.org/8dm6f33g5/582_JM.png (https://postimg.org/image/8dm6f33g5/)
Though call...

So far, Interframe doesn't perform so badly, but I see a much stronger difference on my own clips

InterframeSmooth / YFMC / JM
https://s2.postimg.org/95ouea7n9/6558_Interframe_Smooth.png (https://postimg.org/image/95ouea7n9/) https://s2.postimg.org/8vlx8oiet/6558_YFMK.png (https://postimg.org/image/8vlx8oiet/) https://s2.postimg.org/cqkpxic6t/6558_JM.png (https://postimg.org/image/cqkpxic6t/)

MysteryX

19th April 2017, 22:29

I have updated the script above.
- Scene change detection must use frame blending, otherwise, when a scene is detected as repeated scene change, the frames would keep bouncing back and forth (between deleted and interpolated frames)
- All artifact detection parameters are now exposed as parameters (MaskSAD, MaskOcc, thSCD1, thSCD2)
- MaskStr is now a number that fades the mask. "weak mask" is MaskStr=7, 5 fades it further, and 9 only fades it a little. Use 10 for no fading.
- Scene changes are now included in "over" and "mask" outputs. In "over" mode, it will appear as a dark-golden color with 50% opacity.
- Scene change detection can be disabled with thSCD1=0

For mode "over", cyan overlays can be hard to see on cyan background. Is there a way to show the mask in a way that will be equally visible for any color?

RemoveGrain has a bug in YV24. Is there a good alternative to it?

You can test it out to see if everything is working. Also test various settings to see what works best for you, and perhaps we can decide which artifact detection settings to use as defaults. Also it would be good to test with other types of content such as anime that requires different settings.

For the most part, it seems to be working good, but there are still ugly effects with artifact removal, such as spotlights moving by and showing up as "spots" of different colors. There's also nothing uglier than a scene change with only half the frame showing up through artifact removal mask.

Motenai Yoda

19th April 2017, 23:34

I have updated the script above.
- Scene change detection must use frame blending, otherwise, when a scene is detected as repeated scene change, the frames would keep bouncing back and forth (between deleted and interpolated frames)

Shouldn't be more reliable to use ChangeFPS instead of ConvertFPS for scene-change detections?

MysteryX

20th April 2017, 00:08

Shouldn't be more reliable to use ChangeFPS instead of ConvertFPS for scene-change detections?

The problem I was having is that on a slowly-rotating scene at 45° camera angle, it is triggering scene change detection repeatedly. If half the frames are interpolated and half the frames are dropped, then instead of a smooth panning, you get a zigzag camera rotation bouncing back and forth.

If I want to do a full frame blending whenever more than 30% is part of the artifact masking, is this the right syntax to do it? (right after MSCDetection)

Sc = ConditionalFilter(EM, BlankClip(EM, color=color_white), EM, "AverageLuma()", ">", "80")

I've done a good bulk of the work. I'd appreciate others who are more expert to chime in with feedback and improvements. Perhaps instead of applying Sc and EM separately, it would be good to merge them into 1 mask and then apply only 1 mt_merge. This should be better on performance.

I could also write a simple plugin that is similar to ConditionalFilter but takes a string to specify which frames to take from which clip. Then you could specify something like "1-10A,11-100B,50C,200-210A" and frames 1-10,200-210 would take clip A (frame blending), frames 11-100 would take clip B (interpolation) and frame 50 would take clip C (delete frame). But for now, we want to get as good as possible with the basic script.

johnmeyer

20th April 2017, 00:39

BTW, if you are going to use my Flint Michigan parade clip, once you get your script exactly the way you want, try it out on the moose antlers. If it can handle that, it can handle anything. That is the ultimate torture test, IMHO.

MysteryX

20th April 2017, 03:10

Arggg there was a nasy bug! Artifact masking mask remained in original frame rate which was out of phase with output video, which caused artifacts masks to be out of sync, and over half of the video being treated as a scene change!! So the tests I did earlier were not good.

I fixed the bug and re-uploaded new images. Now, YFMK and JM are much closer to each other. Still hard to say which one is best.

BTW, if you are going to use my Flint Michigan parade clip, once you get your script exactly the way you want, try it out on the moose antlers. If it can handle that, it can handle anything. That is the ultimate torture test, IMHO.
A considerable portion of the image is detected as artifacts.

I'm considering dropping MSCDetection and simply considering a scene change if more than X% of the image is in the mask. I could do a ChangeFps in those cases, and ConvertFps when applying masks. In the case of the moose, we just have to set it to a value so that this whole scene is seen as "scene changes" and replaces the whole frame. The best way to handle it is to disable frame interpolation -- and avoid blending to not have double shadows.

Forget what I said earlier about scene changes during panning -- scene changes were out of wack in the code.

With this fixed, it's actually working pretty well as-is! With an adjustable "artifact mask treshold" to consider as scene change, this just might be enough. This treshold would be much easier to configure than thSCD1/thSCD2, and be much faster.

MysteryX

20th April 2017, 04:26

hum... am I supposed to see artifact masks during scene changes due to blocks being completely different? It doesn't seem to show up -- unless there's a setting I'm missing?

burfadel

20th April 2017, 08:55

I thought I'd try playing around with script settings, with 'good' reason :). I believe jm_fps is designed for the best results using the least amount of processing power, not specifically producing the best results.

super = MSuper(C, hpad = BlockH, vpad = BlockV, levels = 1) # one level is enough for MRecalculate

From the MVTools doc:
rfilter: hierarchical levels smoothing and reducing (halving) filter.
0 is simple 4 pixels averaging like unfiltered SimpleResize (old method);
1 is triangle (shifted) filter like ReduceBy2 for more smoothing (decrease aliasing);
2 is triangle filter like BilinearResize for even more smoothing;
3 is quadratic filter for even more smoothing;
4 is cubic filter like BicubicResize(b=1,c=0) for even more smoothing.
Default is 2 (since v2.3.1). You may also try to apply some external filter to superclip or its coarse bottom part (by appropriate crop and overlay).
This goes in the MSuper line. Maybe do some comparisons with 3 or 4 and see if it is any better or worse :).

As for the levels, try removing it from the line (such that it is set to auto). I think it might be slightly improved in some situations. I believe 1 is set to reduce computational load slightly. Could also try 2 etc, not sure how many 'levels' a video would have?

Under MAnalyse:
bak = MAnalyse(superfilt, isb = true, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
fwd = MAnalyse(superfilt, isb = false, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0)
fwd = MRecalculate(super, fwd, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)
bak = MRecalculate(super, bak, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100)

Try:
bak = MAnalyse(superfilt, isb = true, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0, searchparam = 32, truemotion = true, trymany = true, divide = 2)
fwd = MAnalyse(superfilt, isb = false, blksize = BlockH, blksizeV = BlockV, overlap = 4, search = 3, dct = 0, searchparam = 32, truemotion = true, trymany = true, divide = 2)
fwd = MRecalculate(super, fwd, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100, searchparam = 32, truemotion = true)
bak = MRecalculate(super, bak, blksize = BlockH/2, blksizeV = BlockV/2, overlap = 2, thSAD = 100, searchparam = 32, truemotion = true)

The addition of truemotion could help with motion scenes, at the expense of compution time. This is for both MAnalyse and the MRecalculate. I believe this does help. The trymany is only valid for MAnalyse. I did have a clip where it seemed beneficial when using another motion interpolation script a while ago, it's at least a thought. The searchparam at 32 is from the script filldrops3, supposedly 'better'. I guess these could be added as option. Maybe if those options are beneficial at the expense of computational time they could simply be enabled through a HQ setting in the script.

The divide option may possibly be beneficial in some circumstances.
divide: post-processing motion vectors by dividing every block into 4 subblocks.
0 - do not divide;
1 - divide blocks and assign the original vector to all 4 subblocks;
2 - divide blocks and assign median (with 2 neighbors) vectors to subblocks;
Default = 0. Block size and overlap values must be selected to be acceptable after internal dividing.

All this might seem overkill, but I do believe the combination of the levels, truemotion, searchparam, and maybe trymany does improve the output. Not sure on the benefit of rfilter, but these are all things that can be played with and tested.

StainlessS

20th April 2017, 09:28

As for the levels, try removing it from the line (such that it is set to auto). I think it might be slightly improved in some situations. I believe 1 is set to reduce computational load slightly. Could also try 2 etc, not sure how many 'levels' a video would have?

If used for super clip only used in MRecalculate, anything other than Levels=1, just eats CPU (if not enough levels, then would complain about it as MAnalyse does).
(Levels=0=all levels needed for MAnalyse)

MAnalyse(dct=0) is the default (so that arg of 0, dont do anything other than default).

MAnalyse(truemotion = true) is default.

burfadel

20th April 2017, 10:07

I also played around with this:
EM = EM.Spline36Resize(Round((C.Width/BlockH)/4.0)*4, Round((C.Height/BlockV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .GBlur()
\ .SincResize(C.Width, C.Height)
EM = EM.ChangeFPS(NewNum, NewDen)

Probably makes no difference using those resize methods considering the function, but worth playng around with in case.

Notice gblur(), I also used it here:
## jm_fps interpolation
prefiltered = gblur(C)

It's specifically for that kind of use and is part of the modplus pack:
http://www.avisynth.nl/users/vcmohan/modPlus/modPlus.html

Specific link:
http://www.avisynth.nl/users/vcmohan/modPlus/GBlur.html

real.finder

20th April 2017, 10:39

there are another gblur in TCannyMod, Completely different one (I think one of them should be changed to avoid conflict)

burfadel

20th April 2017, 12:59

There is ghosting in some frames with motion caused by:
EM = EM.Overlay(
\ C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand, [** kind=occlusion *]
\ opacity=0.5, mode="lighten")

In many instances, raising the maskocc value to 120 help. I thought stuff it, and using mode 5 which doesn't exhibit that behaviour.

Also the () is missing after mt_inpand. It may be slightly better as mt_inpand(mode="both")

I've modded the script a bit. It works better for the video I was using, however that may not be the case with others. Seems to be improved with motion for the test clip.

Edit: Removed as MysterX's latest version resolves all the issues!

MysteryX

20th April 2017, 17:45

divide: post-processing motion vectors by dividing every block into 4 subblocks.
0 - do not divide;
1 - divide blocks and assign the original vector to all 4 subblocks;
2 - divide blocks and assign median (with 2 neighbors) vectors to subblocks;
Default = 0. Block size and overlap values must be selected to be acceptable after internal dividing.

Isn't this what MRecalculate does? What's the difference?

burfadel, I see you're now doing MRecalculate twice -- is this a good idea or is it overkill?

MysteryX

20th April 2017, 18:29

I believe the updated images I posted were using hard blending instead of soft blending -- fixing the code. I'll post more comparison images after doing further tweaking.

MysteryX

20th April 2017, 19:49

The video I'm using is quite harder than Motion Estimation Torture Clip so I'm using it for stress-testing. With Motion Estimation Torture Clip, I wasn't quite sure whether YFMK or JM was best for interpolation, but with my low quality clips, JM clearly performs better. Burfadel's interpolation performs worse.

YFMK / JM / Burfadel
https://s2.postimg.org/4e1u7d8sl/2495yfmc.png (https://postimg.org/image/4e1u7d8sl/) https://s2.postimg.org/yhgczbc1x/2495jm.png (https://postimg.org/image/yhgczbc1x/) https://s2.postimg.org/8xe2svqo5/2495burfadel.png (https://postimg.org/image/8xe2svqo5/)

GBlur softens and widens the mask a lot -- which is basically what MaskStr is for (so we're doing it twice). We just want to ask ourselves what shape we want our mask to have.

Raising MaskOcc basically disables it. A good approach here is to observe the specific scenes causing trouble, and decide what would work best: blending, no blending, or what shape and strength for the artifact masking. Begin with the end in mind. Once you know what mask shape you want, we can tune the settings to give that.

mt_inpand(mode="both") causes them to disappear from the mask entirely.

Using block size smaller than 16 gives Access Violation Error.

Another benefit of this over Interframe is that with MvTools, MaskTools2 and Avisynth+, it will be possible to perform interpolation in high-bit-depth.

MysteryX

20th April 2017, 20:30

Here's an updated script with scene detection and artifact masking working correctly.

For the moose scene, set SkipOver=50 which will skip the entire scene. SkipOver uses ConditionalFilter which doesn't work in Avisynth+ MT mode so the default is 0. Is there an alternative filter that could accomplish the job and work in MT?

I found that in my video, scene detection was triggering many times in a row during high-action scenes. I tuned the settings up (500/160 instead of 400/130).

One point to test: is it better to do ChangeFps or ConvertFps on the artifact mask? Right now it uses ChangeFps. If we use ConvertFps, it will cause the mask to be weaker (blended) so that should also be compensated for. Please test it out.

Also have to test where Access Violation comes from when using blocks of 8x8 or 4x4.

Haven't tested FrameDouble for a while, have to test to ensure that's still working.

Is there a command more efficient than this to divide the mask by 2?
Sc.Levels(0, 1, 255, 0, 128, coring=false)

# Frame Rate Converter
# Version: 21-Apr-2017
# Authors: Yushko, johnmeyer, raffriff42, MysteryX
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: masktools2, mvtools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ Output - (auto|inter|none|mask|over) default "auto"=normal artifact masking;
## "inter"=interpolation only; "none"=ConvertFPS only;
## "mask"=mask only; "over"=mask as cyan overlay for debugging
##
## @ MaskStr - The artifact masking strength (1 to 10). A lower value will not change
## artifact detection but will make the mask softer. (Default=7)
##
## @ MaskSAD - Artifact masking strength for bad motion, 0 to disable (Default=190)
##
## @ MaskOcc - Artifact masking strength for occlusion, 0 to disable (Default=64)
##
## @ thSCD1 - MSCDetection scene detection treshold 1 (Default=500)
##
## @ thSCD2 - MSCDetection scene detection treshold 2 (Default=160)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble",
\ int "BlkSize", int "BlkSizeV", string "Output", float "MaskStr", int "MaskSAD", int "MaskOcc", int "thSCD1", int "thSCD2", int "SkipOver")
{
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskStr = Default(MaskStr, 6)
MaskSAD = Default(MaskSAD, 190)
MaskOcc = Default(MaskOcc, 64)
thSCD1 = Default(thSCD1, 500)
thSCD2 = Default(thSCD2, 160)
SkipOver = Default(SkipOver, 48)

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
prefiltered = RemoveGrain(C, 22)
super = MSuper(C, hpad = 16, vpad = 16, levels = 1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
bak = MAnalyse(superfilt, isb = true, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = 0)
fwd = MAnalyse(superfilt, isb = false, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = 0)
fwd = blkmin > 4 ? MRecalculate(super, fwd, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : fwd
bak = blkmin > 4 ? MRecalculate(super, bak, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

## "EM" - error or artifact mask
EM = MaskSAD > 0 ? C.MMask(bak, ml=MaskSAD, kind=1) : BlankClip(C) # kind=SAD
EMfwd = MaskSAD > 0 ? C.MMask(fwd, ml=MaskSAD, kind=1) : EM # kind=temporal blending
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand() : BlankClip(C) # kind=occlusion
EM = MaskSAD > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=0.5, mode="lighten") : EM
SkipEM = EM

## mask strength
EM = MaskStr < 10 ? EM.Levels(0, float(MaskStr) / 10, 255, 0, int(255 - 10*MaskStr), coring=false) : EM

EM = EM.BicubicResize(Round((C.Width/BlkSize)/4.0)*4, Round((C.Height/BlkSizeV)/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

## "Sc" - scene detection
Sc = thSCD1 > 0 && thSCD2 > 0 ? C.MSCDetection(bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(EM)
Sc = SkipOver > 0 ? ConditionalFilter(SkipEM, BlankClip(EM, color=color_white), Sc, "AverageLuma()", ">", string(SkipOver)) : Sc

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "inter")==0) [** inter: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? B
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? EM
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM).ConvertFPS(NewNum, NewDen), mode="Add", opacity=0.30),
\ BlankClip(EM, color=color_darkgoldenrod), Sc.Levels(0, 1, 255, 0, 128, coring=false), luma=true)
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|inter|none|mask|over)")
return R
}

burfadel

20th April 2017, 22:06

MysteryX, that script works much better, particularly the new occlusion changes. I modded my previous script, just added back in the final recalculate, changed the blur to a much weaker gblur (rad=1), and this:
EMocc = C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand # kind=occlusion
to this:
EMocc = C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand() # kind=occlusion

You forgot the () :)

I updated the script above. It's now basically the same as yours, except using glbur(rad=1,sd=1) (much weaker), and adding back in that final mrecalculate. I also added blur to the first recalculate. The idea of doing removegrain for the Manalyse and not further is that on whichever 'Super' it is applied to it affects the output. However, the use of it is for better analysis which noise affects, so it is relatively important in principle. So it's a catch 22 in many ways as you can't apply it to the final recalculate, but smaller block sizes are affected by noise more! I therefore applied it to the '8' blocksize (the first recalculate when using it for under 1280x720). The added last stage of 'blocksize' 4 doesn't have the blur applied, so doesn't blur the output. Remember the additional recalculate is only for bad motion vectors, so if the previous stages do their things its use will be limited apart from not applying the removegrain/blur to the final output.

Using a weak gblur instead of removegrain is because the scripts people use probably already has a denoiser, the removegrain is a bit redundant and wouldn't be strong enough for the purpose. In my last script I had it way too strong :).

You mentioned that using gblur for the masking is redundant, but if so then wouldn't using blur(0.6) be redundant as well? (again, now using a very much weaker setting).

I'm not saying I'm right in any way, just encouraging the thought process :)

MysteryX

20th April 2017, 22:32

FrameDouble is working.

Code updated.

For JM's video, it works better with block size 8. Block size 4 generates more artifacts. I updated the way block sizes and paddings are handled. Default values are now Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8

hpad and vpad are what were causing the Access Violation if changed. They must remain at 16 no matter the block size?

As for artifact mask interpolation, in my opinion it is better to do without interpolation so that artifact masking is more specifically applied to the areas having troubles, otherwise we'll get parts of blending clip showing up where it doesn't need to be.

MysteryX

20th April 2017, 22:44

Changed the blur to a much weaker gblur (rad=1)
I'm not seeing much difference at all between gblur and blur, except that it produces a very slightly weaker mask. I see no reason to require an additional library for it.

However, looking at the artifact mask, I prefer Blur(1) over Blur(.6)... it makes the mask more discrete. It could be tested out further but I think 1 is a good value here.

If we want to refine the shape of the gradient, we should do so in the MaskStr's levels. GBlur doesn't bring anything more to the table.

MysteryX

20th April 2017, 22:58

Here are some comparison images between Interframe(smooth) and FrameRateConverter.

Frame 112
http://screenshotcomparison.com/comparison/207437

Frame 141
http://screenshotcomparison.com/comparison/207438

Frame 582 (Interframe normal preset)
http://screenshotcomparison.com/comparison/207439

These are HUGE improvements!

As for the moose, the correct value is SkipOver=45 to skip that clip section. In my videos, SkipOver=45 results in way too many frames being flagged so it must be higher -- or disabled.

Here's a comparison of my full script with Interframe(smooth) in YV12 vs FrameRateConverter in YV24
http://screenshotcomparison.com/comparison/207440

It would work in 16-bit but MvTools' MMask doesn't yet work in 16-bit (latest from Pinterf).

burfadel

20th April 2017, 23:33

I'm not seeing much difference at all between gblur and blur, except that it produces a very slightly weaker mask. I see no reason to require an additional library for it.

However, looking at the artifact mask, I prefer Blur(1) over Blur(.6)... it makes the mask more discrete. It could be tested out further but I think 1 is a good value here.

If we want to refine the shape of the gradient, we should do so in the MaskStr's levels. GBlur doesn't bring anything more to the table.

Fair enough :), and makes sense! What do you think about the removegrain prefiltered comment though? I think there could be slight improvements there. For many scripts the removegrain is probably redundant as well considering they would already have a denoiser in place, and likewise, you could use blur(1) there as well. This would also eliminate the need for RGTools. I'm not sure of the last recalculate I added, but in principle you would want to apply a filter to the noise on at least the first recalculate. If you left it though, the output would contain the blur which is why the second recalculate I think could be 'beneficial'. Even if very few bad motion vectors are recalculated, the output wouldn't have the blur applied.

burfadel

21st April 2017, 05:43

Hmm, the occlusion is still causing some ghosting artifacts in some particular scenes.

raffriff42

21st April 2017, 12:48

Wow, great work you're doing! I'm just gonna sit back and see what you come up with!It would work in 16-bit but MvTools' MMask doesn't yet work in 16-bit (latest from Pinterf).you can get away with using an 8-bit motion mask in an otherwise 16-bit filter, I would think? MMask(source.ConvertBits(8), vectors.ConvertBits(8), ...).ConvertBits(16) ## ?

pinterf

21st April 2017, 13:39

Seems that masks at 16 bits need some work.

MysteryX

21st April 2017, 15:17

Wow, great work you're doing! I'm just gonna sit back and see what you come up with!you can get away with using an 8-bit motion mask in an otherwise 16-bit filter, I would think? MMask(source.ConvertBits(8), vectors.ConvertBits(8), ...).ConvertBits(16) ## ?
That's the first thing I tried. Unfortunately, the vectors data can't be converted that way.

MysteryX

21st April 2017, 19:40

I really don't like the way artifact masks are being generated. This is NOT a good artifact removal mask. It should be smooth round zones over a perfectly blank background. I'm open to suggestions for a better algorithm to transform the raw mask into something usable.
https://s14.postimg.org/d2hr3t55p/Mask_AR.png (https://postimg.org/image/d2hr3t55p/)

Can someone tell me what [* *] does? I've never seen that syntax before. I thought it was some type of comments, but apparently not.

[*.ColorYUV(cont_y=_f2c(2.0)) *]

I have restored these lines from RaffRiff's code and it's better, but it still has problems. Here's the mask for frame 128. RiffRaff, could you look at this if you can do better?
https://s1.postimg.org/io1ggtmqj/Mask128.png (https://postimg.org/image/io1ggtmqj/)

I have updated the code above, re-added the lines with [* *] for mask transformation, allowed disabling artifact /occlusion removal with MaskSAD=0 and MaskOcc=0, and SkipOver moved to be applied after mask transformation -- which unfortunately means that SkipOver will be directly affected by MaskStr.

Mask transformation code needs more work.

Edit: Never mind, RaffRiff's code appears to be working now. Will post post new pictures with the mask working correctly.

MysteryX

21st April 2017, 21:19

It seems MMask sets the Luma and leaves other planes at random values, and that's where the dirt comes from and what confused me. If that random data is then ignored, it's fine.

I have updated the script with the following changes
- It looks better with Blur(.6) than Blur(1), it makes the output image look less blurry.
- SkipOver now being applied again before mask transformation, as we want to evaluate the raw data for interpolation performance, and use a consistent default value that isn't affected by MaskStr. As a rule of thumb, "in doubt, don't interpolate" seems like a good advice. I've set SkipOver to 48 by default which skips the entire moose scene. In my clip, it skips quite a lot, but those skipped animated frames contain serious artifacts anyway.
- I think we don't need the "blend" parameter, because if we merge an interpolated frame with a previous frame, we're going to get a double-shadow. I can't see how that would ever be good. I removed it.
- Since SkipOver is highly recommended, and serves as an additional scene change detection, I don't think thSCD1 and thSCD2 are necessary to configure. MaskStr is affecting how many artifacts are included in the mask, so MaskSAD and MaskOcc may not be necessary either; but for now I'll leave them for testing.

It will be important to make SkipOver work in some way or another in MT -- it's an important feature.

MysteryX

21st April 2017, 21:53

Changing MaskStr default from 7 to 6.

I'm playing around with a clip that causes horrible double shadows. It looks better with MaskStr=6. JM's clip also looks better with MaskStr=6.

Interframe / FRC(MaskStr=7) / FRC(MaskStr=6)
https://s22.postimg.org/ac4btgq4t/3106svp.png (https://postimg.org/image/ac4btgq4t/) https://s22.postimg.org/f9hweks3x/3106ar7.png (https://postimg.org/image/f9hweks3x/) https://s22.postimg.org/n0u0ze6vh/3106ar6.png (https://postimg.org/image/n0u0ze6vh/)

https://s22.postimg.org/qqy96183x/5503svp.png (https://postimg.org/image/qqy96183x/) https://s22.postimg.org/5g0p1rpzh/5503ar7.png (https://postimg.org/image/5g0p1rpzh/) https://s22.postimg.org/rqok1qn9p/5503ar6.png (https://postimg.org/image/rqok1qn9p/)

SVP/Interframe generates very ugly frames now and then. FrameRateConverter doesn't have that problem, or at least, it does a much better job on hard scenes.

One could spend a LOT of time tweaking settings for MaskSAD, MaskOcc, raw mask levels adjustments based on MaskStr, and mask transformation parameters, to get optimal results. If you want to play with this, take your clips where you get the worst artifacts, and play with those until you get the "least worst" results in most cases. Use output="over" to see how the artifact masks are being generated, then compare the output.

Also test if SkipOver=48 is an acceptable value for your clips. With output="over", skipped frames will appear with orange-gold overlay. I'm not doing frame blending on bad frames, I'm just skipping them altogether.

Overall, there are 2 aspects to this script: frame interpolation, and frame removal. For interpolation, no matter what I try, I keep coming back to johnmeyer's code, and I trust he did plenty of testing and tweaking on it. Artifact removal is what can be tweaked. In particular, when things look ugly, we can adjust settings to make it more decent.

MysteryX

22nd April 2017, 01:49

Mask levels range reduction changed from 255-36 to 255-60. This is causing less "false positives" and is handling double-shadows slightly better. In JM's clip, I only found 1 frame where 2 minor artifacts were left out. Other than that, I have only seen improvements.

Changing gamma by more than .6 is causing a loss of smoothness in artifact masking.

Honestly, I'm quite satisfied by these results. The clip that had lots of double shadows barely has any artifacts now. I have to look hard to find a defective frame, and none are horrible.

As for MaskSAD, MaskOcc, thSCD1 and thSCD2 I might leave them simply because it can be used to disable each component.

raffriff42

22nd April 2017, 17:06

Took a quick look. Seems very good. The artifact mask seems to come into play much less often (viewed using output="over"), but it is there when it's needed.

FrameRateConverter is a better name, but I would like to see "formerly known as YFRC" in the comments, both as credit to the original and simply because it's useful information. Also, people searching for "YFRC" will stumble on FrameRateConverter, so it's a win-win. (I guess they would anyway, since "YFRC" is all over this thread)

MysteryX

23rd April 2017, 00:24

Updated code for today: added some validations, and added comments about YFRC and what code comes from who.

Also released ConditionalMT (https://forum.doom9.org/showthread.php?t=174550) which is now required by the script. It will allow for ConditionalFilter to work with MT mode.

Pinterf just released a new version of masktools2, but this still doesn't work in 16-bit. Probably soon.

# Frame Rate Converter
# Version: 23-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, modPlus, ConditionalMT
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Preset - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ Output - Output mode [auto|inter|none|mask|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only
## none=ConvertFPS only; mask=mask only; over=mask as cyan overlay for debugging
##
## @ MaskStr - The artifact masking strength (1 to 100). This will apply a gamma curve on the mask
## before processing, where 100 applies no gamma curve and 5 applies 0.05 gamma (Default=5)
##
## @ MaskSAD - Artifact masking strength for bad motion, 0 to disable (Default=190)
##
## @ thSCD1 - MSCDetection scene detection treshold 1 (Default=500)
##
## @ thSCD2 - MSCDetection scene detection treshold 2 (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble", string "Preset",
\ int "BlkSize", int "BlkSizeV", string "Output", float "MaskStr", int "MaskSAD", int "thSCD1", int "thSCD2", int "SkipOver")
{
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
Preset = Default(Preset, "normal")
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskStr = Default(MaskStr, 5)
MaskSAD = Default(MaskSAD, 190)
#MaskOcc = Default(MaskOcc, 64)
thSCD1 = Default(thSCD1, 500)
thSCD2 = Default(thSCD2, 150)
SkipOver = Default(SkipOver, 48)

Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster", "FrameRateConverter: Preset must be slow, normal, fast or faster")
Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskStr > 0 && MaskStr <= 100, "FrameRateConverter: MaskStr must be between 1 and 100")
Assert(SkipOver >= 0 && MaskStr <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast, faster
Recalculate= preset == "slow" || preset == "normal"
Prefilter = preset == "slow" || preset == "normal" || preset == "fast"
DCT = preset == "slow" ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
prefiltered = Prefilter ? (C.IsYUV ? Median(C, uu = true, vv = true) : Median(C)) : C
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
super = Prefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb = true, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = DCT)
fwd = MAnalyse(superfilt, isb = false, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

## "EM" - error or artifact mask
EM = MaskSAD > 0 ? C.MMask(bak, ml=MaskSAD, kind=1) : BlankClip(C) # kind=SAD
EMfwd = MaskSAD > 0 ? C.MMask(fwd, ml=MaskSAD, kind=1) : EM # kind=temporal blending
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
#EMocc = MaskOcc > 0 ? C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand() : BlankClip(C) # kind=occlusion
EM = MaskSAD > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
#EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=0.5, mode="lighten") : EM
SkipEM = EM

## mask strength
EM = EM.Levels(0, MaskStr / 100.0, 255, 0, 255, coring=false)

EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

## "Sc" - scene detection
Sc = thSCD1 > 0 && thSCD2 > 0 ? C.MSCDetection(bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(EM)
Sc = SkipOver > 0 ? ConditionalFilterMT(SkipEM, BlankClip(EM, color=color_white), Sc, "AverageLuma", ">", string(SkipOver)) : Sc

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "flow")==0) [** flow: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? B
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? EM
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM).ConvertFPS(NewNum, NewDen), mode="Add", opacity=0.40),
\ BlankClip(EM, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true)
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|over)")
return R
}

TODO: How can we have the mask overlay not lighten the colors of the clip?

I just checked the "mask" output on JM's clip -- it does indeed mask *much* less, even less than I expected. But so far it seems to be working just right. If we find cases where artifacts aren't being masked, then we can do further tweaks.

MysteryX

23rd April 2017, 03:55

Oups, I left some really bad debugging code ... the mask was being applied a leveling of .9/10 instead of 6/10 ... surprising it looked good. Fixed.

MysteryX

23rd April 2017, 05:12

Putting such a low gamma was totally an accident, but it's actually working better.

Look at this sample image with horrible artifact -- with mask gamma leveling of
.6 / .2 / .1 / .06
https://s16.postimg.org/pz6pl2575/Gamma60.png (https://postimg.org/image/pz6pl2575/) https://s16.postimg.org/8kmhcs829/Gamma20.png (https://postimg.org/image/8kmhcs829/) https://s16.postimg.org/i3w668vkh/Gamma10.png (https://postimg.org/image/i3w668vkh/) https://s16.postimg.org/osy48irip/Gamma06.png (https://postimg.org/image/osy48irip/)

I might leave it at a constant of .1 -- making that adjustable may be a hassle. Adjusting the other leveling parameter might be enough for MaskStr.

Edit: Updated code. MaskStr now adjusts only gamma (1-100), and by default applies a gamma curve of 0.05. The second adjustable parameter of Levels is now left at 255, it looks better that way.

EM = EM.Levels(0, MaskStr / 100.0, 255, 0, 255, coring=false)

If my worst interpolation clip and JM's worst interpolation clip can make it through this, I believe any clip will work with this.

There is one bug: SkipOver's mask doesn't fully cover the image. When an image is repeated, the 2nd frame has a slight shadow of the interpolated frame. Something is wrong with the mask. Got to investigate.

manolito

23rd April 2017, 09:41

Thanks a lot MysteryX for your work on this plugin... :thanks:

I used your latest script for some comparisons with the original jm_fps script, as usual I threw my anime torture clip at it. The results are here:
http://www5.zippyshare.com/v/0tGQFlMd/file.html

There is a bug in your script which crashes AviSynth when I specify a block size of 32. I get this message:
[avisynth @ 0335ea20] ResizeH: ISSE code could not be compiled.

I have to say that I use older plugin versions because of my Non-SSE2 CPU and working under WinXP. I use plain vanilla AviSynth 2.60, MaskTools 2.0a48, MVTools2 latest version by Fizick and the old original RemoveGrain package by Kassandro.

For this specific anime source a block size of 32 seems to work better, but I could not test it with your script. Using DCT=1 makes a significant difference in the jm_fps script, with your script the difference is much smaller (but the speed sacrifice is also much smaller).

Overall I do like your script very much, the results are very good for real world sources. And the speed is surprisingly good, even on my ancient computer.

Cheers
manolito

pinterf

23rd April 2017, 09:59

burfadel

23rd April 2017, 10:59

Have you looked into:
prefiltered = RemoveGrain(C, 22)

Excerpt from the MVTools doc:
To use prefiltered clip for more reliable motion estimation, but compensate motion of not-prefiltered clip (denoising example)

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
# Use some denoiser (blur) or deflicker for prefiltering
prefiltered = blur(1.0)
super = MSuper(levels=1) # one level is enough for MCompensate
superfilt = MSuper(prefiltered) # all levels for MAnalyse
backward_vectors = MAnalyse(superfilt, isb = true)
forward_vectors = MAnalyse(superfilt, isb = false)

On a very clean source or a clip that has already been denoised, the addition of removegrain with mode 22 would likely do very little, and just add the unnecessary removegrain filtering and extra MSuper step.

From the Removegrain doc:
21 Clips pixels using the averages of opposite neighbour
22 Same as mode 21 but simpler and faster.

I believe it was chosen purely for speed, however I believe that a stronger filter, whether it be another one of the Removegrain modes or a different filter, could produce better results in certain circumstances. It does not affect the output apart from more potentially more accurate motion compensation (or worse if you get it wrong), so the best filter to remove noise, reasonably fast, but retain picture structure for motion compensation would be ideal. I am thinking along the lines of a median filter like this one:
http://www.avisynth.nl/users/vcmohan/modPlus/Median.html

Maybe with something like:
prefiltered = Median(C, uu = true, vv = true)

Technically it wouldn't be an additional library because it would be used in place of RGtools, that said RGtools would likely be on the users system anyway. The median filter above is part of modplus that does have other useful functions in it.

From the doc:
"Ordinary median filters perform well as long as the spatial density of impulsive noise is small. This Median filter can suppress impulsive noise with larger probablity. An additional benefit is this seeks to preserve detail while smoothing nonimpulse noise something that the traditional median filter does not do.

The algorithm described has three purposes.
1.To remove salt and pepper (impulse) noise.
2.To smooth other noise which may not be impulsive
3.To reduce distortion such as excessive thinning or thickening of object boundaries.
"

That sounds ideal for the intended purpose, since the comparison to other median filters would be similar to the comparison to removegrain. I added uu=true and vv=true to process the colour planes, since the MAnalyse works on chroma as well.

Anyways, just a thought, but I do believe such a weak filter like removegrain mode 22, particularly on an already denoised or clean video, would not be effective, and on a noisy clip simply not effective enough.

burfadel

23rd April 2017, 11:05

Oh, forgot to mention!
Sc = SkipOver > 0 ? ConditionalFilterMT(SkipEM, BlankClip(EM, color=color_white), Sc, "AverageLuma", ">", string(SkipOver)) : Sc

Why MT? Are you using MT Avisynth? In Avisynth+ (using r2455) there is no such function, it is simply ConditionalFilter. Therefore, on Avisynth+ it simply fails when that line executes.

pinterf

23rd April 2017, 11:07

It's a new workaround filter from MysteryX until avs+ is fixed.

burfadel

23rd April 2017, 11:19

It's a new workaround filter from MysteryX until avs+ is fixed.

Oops completely missed that! I guess the code of the post took the focus! :)

MysteryX

23rd April 2017, 14:23

MysteryX: new MvTools2 with 10+ bits MMask has been released.
You mentioned an Access Violation crash, what should I change in the script to reproduce it?
Remark: When avisynth (manolito case) or a plugin is crashing, it's never the script's fault but the core itself needs bugfix.
It was when using hpad and vpad at 8 or 4 instead of 16 I believe.

I used your latest script for some comparisons with the original jm_fps script, as usual I threw my anime torture clip at it. The results are here:
http://www5.zippyshare.com/v/0tGQFlMd/file.html
Nice to know it works for anime too! The advice used to be to tune down settings a LOT to make it decent for anime, but here it seems to work out-of-the-box.

In your sample, comparing both outputs of this script, DCT=1 does make a serious difference on the artifacts on the machine dials and conveyor belt's vertical lines. I'd say FrameRateConverter with DCT=1 is the only one that is good here. It will have to be exposed as an option.

[avisynth @ 0335ea20] ResizeH: ISSE code could not be compiled.
blisize=32 works here. The bug is in one of your outdated libraries, not in this script. You'll have to test it out by reducing the script until you find the exact line that crashes -- and then look for a replacement library for that.

Have you looked into:
prefiltered = RemoveGrain(C, 22)
No not yet. Honestly I don't even know how it's working exactly. I'll have to test out various options.

It would be good to offer a faster preset, and that would include disabling pre-processing. Options for faster performance:
- disabling RemoveGrain and the 2nd MSuper
- disabling MRecalculate
- disabling occlusion mask -- have to do some testing on this as this is exactly what tends to produce double shadows with frame blending, maybe those could be left out
- replacing MSCDetection, it seems to be heavy on performance. It seems MSuper already does scene detection in its interpolation because it isn't producing a strong mask on most scene changes. Is there a way to avoid running scene detection twice, or to disable MSuper's scene detection so that strong masks are produced on scene changes and then later discarded?
- what other options could be tuned down for minimal quality loss and good performance gain?

MysteryX

23rd April 2017, 15:47

Performance with JM's test clip

For some reason, I only get 48% CPU usage even with Prefetch(8)

FPS (min | max | average): 7.265 | 155894 | 59.82
Memory usage (phys | virt): 597 | 595 MiB
Thread count: 29
CPU usage (average): 48%

With current settings, occlusion mask doesn't appear at all -- ever -- on this clip. And if it did, frame blending of moving objects would cause a double shadow that won't be better than the artifact itself. Might as well remove occlusion mask altogether: performance goes from 59.82 to 63.10fps

Disabling prefiltering and MRecalculate, I get 90fps !!

DCT=1 cuts performance in half -- 40fps

Question: Is MRecalculate after MAnalyse still useful if we're not doing prefiltering and both are being done on the same clip? (Edit: after testing, it does still make a difference)

After testing, MSCDetection has low impact on performance so that's fine.

MysteryX

23rd April 2017, 16:15

Have you looked into:
prefiltered = RemoveGrain(C, 22)
I tested JM's test clip with KNLMeans running first. Even after a denoiser, I still see benefits to pre-filtering. However, I have to look very carefully to find any difference at all, so it is very subtle.

Frame 248 shows difference on the yellow lines on the ground.

Prefiltering: None, RemoveGrain, Median
https://s23.postimg.org/ekoudkyav/Filt_No.png (https://postimg.org/image/ekoudkyav/) https://s23.postimg.org/pljzilqjr/Filt_Remove_Grain.png (https://postimg.org/image/pljzilqjr/) https://s23.postimg.org/wyzdhkalj/Filt_Median.png (https://postimg.org/image/wyzdhkalj/)

Median is also faster. Sold.

manolito

23rd April 2017, 17:51

blisize=32 works here. The bug is in one of your outdated libraries, not in this script. You'll have to test it out by reducing the script until you find the exact line that crashes -- and then look for a replacement library for that.

Well, this is not really acceptable for me... :devil:

The original jm_fps script works fine using blksize=32. The libraries I use may be outdated, but so is my computer, and obviously so am I.

These libraries are the latest which run on my machine, and I am not going to replace it just to get this script to work. I have a lot of other scripts which work well using these libraries. I do have a problem with all these "plugin modernizations" which have no regard whatsoever for backwards compatibility.

Cheers
manolito

burfadel

23rd April 2017, 18:31

I tested JM's test clip with KNLMeans running first. Even after a denoiser, I still see benefits to pre-filtering. However, I have to look very carefully to find any difference at all, so it is very subtle.

Frame 248 shows difference on the yellow lines on the ground.

I think the difference depends on the amount of noise. A noisy clip would likely have greater gains from prefiltering. The same probably goes for blocking artifacts so a deblocking filter could help in those instances. Maybe upping the maxgrid in the Median filter I linked could accomplish this. The default is 5, maximum is 9 (keeping uu=true, and vv=true).

I have noticed the converter runs fairly slow as well compared to you earlier versions. Much less CPU use as well, I think it's hitting a major bottleneck.

EM.BicubicResize(Round((C.Width/BlkSize)/4.0)*4, Round((C.Height/BlkSizeV)/4.0)*4)

Why the divide by 4 then straight away multiplying by 4? For standard resolutions (horizontal and vertical) of 480, 576, 720, 1080, 1280 etc, a block size of 16 fits nicely into these as whole numbers, such that the rounding isn't necessary. However, if the resolution (due to cropping) was say, 1280x716, 716 divided by the block size of 16 would be 44.75.

I believe the /4 and *4 is there to identify these resolution and make this number nice (716/4=179). The rounding would therefore only be used the number is something like 714.

So the question is, why is it divide by 4 and not 8 or 2? Is it performance related versus quality? There are also TWO additional resize steps in this part of the function alone, surely there can be a faster way of doing it assuming that there is some performance penalty.

For this line:
Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM).ConvertFPS(NewNum, NewDen), mode="Add", opacity=0.30)

What does the purpose of MergeRGB(Blankclip(EM), EM, EM) do for a normal avisynth script?

Also, wouldn't converting the FPS in the middle of an overlay command be a bad thing, in that it would be overlaying the wrong frame data?

MysteryX

23rd April 2017, 18:33

Previous tests had artifact masking disabled so they ran faster. I added performance presets. Here are benchmarks on JM's clip in YV12.

Presets:
- Faster: 85fps @ 55% CPU (no recalculate, no prefilter)
- Fast: 30fps @ 29% CPU (no prefilter)
- Normal: 25fps @ 31% CPU
- Slow: 19fps @ 30% CPU (dct=1)

Faster / Fast / Normal / Slow
https://s8.postimg.org/e4lbh249t/Preset_Faster.png (https://postimg.org/image/e4lbh249t/) https://s8.postimg.org/esu6002zl/Preset_Fast.png (https://postimg.org/image/esu6002zl/) https://s8.postimg.org/i28l6gr35/Preset_Normal.png (https://postimg.org/image/i28l6gr35/) https://s8.postimg.org/49yrokppd/Preset_Slow.png (https://postimg.org/image/49yrokppd/)

I'm open to suggestions to improve each preset.

burfadel

23rd April 2017, 18:37

Well, this is not really acceptable for me... :devil:

The original jm_fps script works fine using blksize=32. The libraries I use may be outdated, but so is my computer, and obviously so am I.

These libraries are the latest which run on my machine, and I am not going to replace it just to get this script to work. I have a lot of other scripts which work well using these libraries. I do have a problem with all these "plugin modernizations" which have no regard whatsoever for backwards compatibility.

Cheers
manolito

Unfortunately backwards compatibility creates greater code complexity and use of programmers time for the benefit of a small percentage of people that may one day use the plugin. Their time is precious, so they're not going to spend time in updating the old source code alongside the new source code when that time could be better spent elsewhere. You would have to find someone willing to do that for you.

You're probably lucky it runs at all considering the non-SSE2 CPU, Windows XP, plain AviSynth 2.60, MaskTools 2.0a48, and the old original RemoveGrain package by Kassandro.

I can't imagine that if it did work that you would get very encode speed considering. The encoder is also important. No point on having really good quality output from framerateconverter if you have to use very basic encoder settings to get it to anything that isn't painfully slow on such a system.

MysteryX

23rd April 2017, 18:59

The original jm_fps script works fine using blksize=32. The libraries I use may be outdated, but so is my computer, and obviously so am I.
I get you. Still, you must find which component fails, and only you can do that as everything works here. You have to identify the line of code that fails. Then, we can explore the options from there.

I have noticed the converter runs fairly slow as well compared to you earlier versions. Much less CPU use as well, I think it's hitting a major bottleneck.
Again, we have to identify which component creates a bottleneck...

Why the divide by 4 then straight away multiplying by 4?
The first is to reduce the mask so that we have 1 pixel per block. Meaning, each block is either left or replaced. The second resize is to scale it back to the clip size. I'm finally starting to understand what it's doing. That line looks good.

What does the purpose of MergeRGB(Blankclip(EM), EM, EM) do for a normal avisynth script?
This is only for output=over, to add a cyan mask overlay. The mask gets added to the G and B channels. However, there is a bug that causes the image to get brighter (with output=over) which I still haven't identified.

Also, wouldn't converting the FPS in the middle of an overlay command be a bad thing, in that it would be overlaying the wrong frame data?
Masks have the source frame rate until they are converted to match the destination frame rate. If you don't place ChangeFps at the right places, the mask won't match the clip it's being applied to.

burfadel

23rd April 2017, 19:09

Ah ok! You could probably get rid of the blur function though if you play around with the b and c parameters of bicubicresize, which would have practically the same effect. Saves one function. This is for the enlargment.

For the downsize, couldn't bilinear be used instead seeing as it's faster and output extremely similarly? This is particularly true when going to such a small resolution.

MysteryX

23rd April 2017, 19:21

Warning: the output is currently very different between YV12 and YV24!! See here. (https://forum.doom9.org/showthread.php?p=1804832#post1804832) This completely changes the artifact detection behaviors.

I've mostly been doing my tests in YV24, so the settings are tweaked for YV24.

Burfadel, I don't think changing these functions will make any difference at all, they are very lightweight compared to the rest of the code.

MysteryX

23rd April 2017, 19:43

Good news: 16-bit now works with Pinterf's latest MvTools and MaskTools.

Performance-wise:
- YV24 8-bit: 13.22fps @ 27% CPU
- YV24 16-bit: 11fps @ 27% CPU

Not a big performance cost. Quality-wise, it removes some interpolation distortion in some areas. I also saw a few instances where it instead added slight distortion.

MysteryX

24th April 2017, 00:18

Here's a weird one.

ColorBarsHD()
https://s27.postimg.org/6bxbm2fvz/Overlay1.png (https://postimg.org/image/6bxbm2fvz/)

C=ColorBarsHD()
C.Overlay(BlankClip(C), mode="Add", opacity=0.40)

https://s27.postimg.org/dt6j1a5f3/Overlay2.png (https://postimg.org/image/dt6j1a5f3/)

There is unwanted brightening. If I were to take a guess, it might be because black is Y=36, and it applies a brightness of 36/255. What would be the solution?

Edit: Adding pc_range=true to Overlay fixes it.

MysteryX

24th April 2017, 04:00

The performance bottleneck is... roll-drum...

EM = EM.ChangeFPS(NewNum, NewDen)
!??

CPU drops from 80% to 20% if I place a "return EM" after this line than before

As for 16-bit support, SkipOver doesn't yet work as expected. It needs to be disabled for testing, or set to a value 220x higher in 16-bit.

YV12 doesn't work as expected, 16-bit doesn't work as expected, and CPU drops to 20%, but other than that it's perfect :)

burfadel

24th April 2017, 04:25

It must have driven you crazy finding that, it's basically the least likely place you would see a performance bottleneck.

Have you tried renaming EM to something else, like EM2?
EM2 = EM.ChangeFPS(NewNum, NewDen)
Return EM2

Maybe using a frame rate change and EM=EM.ChangeFPS is causing it to loop.

manolito

24th April 2017, 09:22

I get you. Still, you must find which component fails, and only you can do that as everything works here. You have to identify the line of code that fails. Then, we can explore the options from there.

The code fails here:
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

But only for an SD source (720 x 480) when I force a Block Size of 32. HD sources do not cause problems.

I made a workaround reducing the BlkSize variable to 16 for SD sources. I inserted this code just before the above section so the interpolator is not affected:
BlkSize = C.Width>1200||C.Height>900 ? BlkSize : BlkSize == 32 ? 16 : BlkSize
BlkSizeV = C.Width>1200||C.Height>900 ? BlkSizeV : BlkSizeV == 32 ? 16 : BlkSizeV

This produces very nice results so far.

Otherwise I have to tell you guys that I am outta here. The last incarnation of the script requires AviSynth+, and this pushes it over the edge for me. You guys do live on a different planet. The only guy who is even further out is stax76 who seriously wanted to raise the minimum requirement for StaxRip to Win 10 with the recent Creator's update (I think he revoked it for the time being).

Whatever, have fun...

Cheers
manolito

burfadel

24th April 2017, 10:04

The performance bottleneck is... roll-drum...

EM = EM.ChangeFPS(NewNum, NewDen)
!??

Shouldn't it be:
EM = Changefps(EM, NewNum, NewDen)

Does that make any difference? I notice that in several places in the script different functions are written as (example) EM.ChangeFPS instead of ChangeFPS(EM,), or ChangeFPS(clip = EM,)

I wonder if adjusting all of these (using the clip function of the command rather than having it piped? from outside via EM.) would make any different to the performance?

This last updated code with the difference syntax applied:
# Frame Rate Converter
# Version: 23-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, modPlus, ConditionalMT
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Preset - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ Output - Output mode [auto|inter|none|mask|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only
## none=ConvertFPS only; mask=mask only; over=mask as cyan overlay for debugging
##
## @ MaskStr - The artifact masking strength (1 to 100). This will apply a gamma curve on the mask
## before processing, where 100 applies no gamma curve and 5 applies 0.05 gamma (Default=5)
##
## @ MaskSAD - Artifact masking strength for bad motion, 0 to disable (Default=190)
##
## @ thSCD1 - MSCDetection scene detection treshold 1 (Default=500)
##
## @ thSCD2 - MSCDetection scene detection treshold 2 (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble", string "Preset",
\ int "BlkSize", int "BlkSizeV", string "Output", float "MaskStr", int "MaskSAD", int "thSCD1", int "thSCD2", int "SkipOver")
{
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
Preset = Default(Preset, "normal")
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskStr = Default(MaskStr, 5)
MaskSAD = Default(MaskSAD, 190)
#MaskOcc = Default(MaskOcc, 64)
thSCD1 = Default(thSCD1, 500)
thSCD2 = Default(thSCD2, 150)
SkipOver = Default(SkipOver, 48)

Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster", "FrameRateConverter: Preset must be slow, normal, fast or faster")
Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskStr > 0 && MaskStr <= 100, "FrameRateConverter: MaskStr must be between 1 and 100")
Assert(SkipOver >= 0 && MaskStr <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast, faster
Recalculate= preset == "slow" || preset == "normal"
Prefilter = preset == "slow" || preset == "normal" || preset == "fast"
DCT = preset == "slow" ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = ChangeFPS(C, NewNum, NewDen)
B = ConvertFPS(C, NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
prefiltered = Prefilter ? (C.IsYUV ? Median(C, uu = true, vv = true) : Median(C)) : C
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
super = Prefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb = true, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = DCT)
fwd = MAnalyse(superfilt, isb = false, blksize = BlkSize, blksizeV = BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search = 3, dct = DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize = BlkSize/2, blksizeV = BlkSizeV/2, overlap = blkmin>8?2:0, thSAD = 100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

## "EM" - error or artifact mask
EM = MaskSAD > 0 ? MMask(C, bak, ml=MaskSAD, kind=1) : BlankClip(C) # kind=SAD
EMfwd = MaskSAD > 0 ? MMask(C, fwd, ml=MaskSAD, kind=1) : EM # kind=temporal blending
EMfwd = FrameDouble ? DeleteFrame(EMfwd,0) : EMfwd
#EMocc = MaskOcc > 0 ? MMask(C, bak, ml=MaskOcc, kind=2).mt_inpand() : BlankClip(C) # kind=occlusion
EM = MaskSAD > 0 ? Overlay(EM, EMfwd, opacity=0.5, mode="lighten") : EM
#EM = MaskOcc > 0 ? Overlay(EM, EMocc, opacity=0.5, mode="lighten") : EM
SkipEM = EM

## mask strength
EM = Levels(EM, 0, MaskStr / 100.0, 255, 0, 255, coring=false)

EM = BicubicResize(EM, Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

## "Sc" - scene detection
Sc = thSCD1 > 0 && thSCD2 > 0 ? MSCDetection(C, bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(EM)
Sc = SkipOver > 0 ? ConditionalFilterMT(SkipEM, BlankClip(EM, color=color_white), Sc, "AverageLuma", ">", string(SkipOver)) : Sc

## Convert masks to desired frame rate
EM = ChangeFPS(EM, NewNum, NewDen)
Sc = ChangeFPS(SC, NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "flow")==0) [** flow: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? B
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? EM
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Overlay(Flow, MergeRGB(BlankClip(EM), EM, EM).ConvertFPS(NewNum, NewDen), mode="Add", opacity=0.40),
\ BlankClip(EM, color=color_darkgoldenrod), mt_lut(Sc, "x 2 / "), luma=true)
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|over)")
return R
}

stax76

24th April 2017, 12:58

Otherwise I have to tell you guys that I am outta here. The last incarnation of the script requires AviSynth+, and this pushes it over the edge for me. You guys do live on a different planet. The only guy who is even further out is stax76 who seriously wanted to raise the minimum requirement for StaxRip to Win 10 with the recent Creator's update (I think he revoked it for the time being).

I admit it was stupid and I realized, revoked and apologized after only one day. I test staxrip on Win 7 using wmware almost daily! staxrip will probably be the first app to give up Win 7 support, I'm not proud of it, but since it was always like this it's also something that can be expected.

raffriff42

24th April 2017, 13:07

I notice that in several places in the script different functions are written as (example)
EM.ChangeFPS instead of
ChangeFPS(EM,), or ChangeFPS(clip = EM,)

There is no difference in performance, as far as I know.
http://avisynth.nl/index.php/Grammar

The fourth form is an alternate syntax called "OOP notation" in AviSynth:
expression . function_name ( argument_list ) is equivalent to
function_name ( expression , argument_list )

However, occasionally there is a need for the normal, non-"OOP" form; for example, if a filter accepts multiple clips in the left-most set of arguments, implicit Last may insert itself as the first clip when not intended. When that happens, the output is obviously wrong; it's not a subtle performance hit. (except possibly in the case of ScriptClip) (http://forum.doom9.org/showthread.php?t=168698)

It's only an issue where implicit Last is involved. With explicit variables, both forms are equivalent.

...and ChangeFPS(clip = EM,...) will not work. There is no argument named 'clip' - it's a unnamed argument.

StainlessS

24th April 2017, 14:24

FrameRateConverter() script is for Avs+ only, but there is also a Median() function in Median.dll v2.5 plugin.

Median "c+[CHROMA]b[SYNC]i[SAMPLES]i[DEBUG]b"
MedianBlend "c+[LOW]i[HIGH]i[CHROMA]b[SYNC]i[SAMPLES]i[DEBUG]b"
TemporalMedian "c[RADIUS]i[CHROMA]b[DEBUG]b"

So maybe should use "ModPlus_Median()" instead (to explicitly name the required dll).
http://avisynth.nl/index.php/Plugins#Plugin_Autoload_and_Conflicting_Function_Names

EDIT: Leastwise I take it that Median exists in the modPlus dll.
EDIT: Yes it does:- http://www.avisynth.nl/users/vcmohan/modPlus/modPlus.html
EDIT: Would not be surprised if filter 'Median()' exists in more than those two dll's.

Groucho2004

24th April 2017, 14:54

The last incarnation of the script requires AviSynth+, and this pushes it over the edge for me.What's wrong with AVS+? It should work on your SSE CPU:
if(CMAKE_SIZEOF_VOID_P EQUAL 4)
# VC++ enables the SSE2 instruction set by default even on 32-bits. Step back a bit.
add_definitions("/arch:SSE")
#add_definitions("/arch:SSE2") # Better use this one, it's 2016 now
endif()

pinterf

24th April 2017, 15:38

What's wrong with AVS+? It should work on your SSE CPU:
if(CMAKE_SIZEOF_VOID_P EQUAL 4)
# VC++ enables the SSE2 instruction set by default even on 32-bits. Step back a bit.
add_definitions("/arch:SSE")
#add_definitions("/arch:SSE2") # Better use this one, it's 2016 now
endif()

My builds override this flag. I always published SSE2-only builds, to have optimized functions (mainly because of high bit depths that had less SSE2 optimized filters) for C-only parts. No one complained so far.

Groucho2004

24th April 2017, 15:57

My builds override this flag. I always published SSE2-only builds, to have optimized functions (mainly because of high bit depths that had less SSE2 optimized filters) for C-only parts.Seems reasonable. Manolito's CPU (Athlon XP, I think) does not support SSE2 and above.

manolito

24th April 2017, 16:08

I followed the AVS+ threads by Utim and pinterf loosely, I tried AVS+ a few times (on the ancient desktop where it didn't work and on the newer laptop where it did). My conclusion so far is very clear:

All the new stuff which AVS+ offers does not do anything for me. I do not need 64-bit, I do not need MT, I do not need the high bit depths.
But I do need an AviSynth build which is stable (looking at the AVS+ thread all I can say is that this is work in progress, looks far from stable to me).
I absolutely do need to use older 32-bit filters like the VDub filters.
And I do need to use it in a variety of older software (without switching versions - even if Groucho made this quite convenient).

So it looks like I will stick with the plain vanilla AviSynth versions for some time...

Cheers
manolito

manolito

24th April 2017, 16:30

Manolito's CPU (Athlon XP, I think) does not support SSE2 and above.

No, it is an Intel Celeron (Coppermine) 1.1 GHz. Hitting the 1 GHz limit was a milestone at that time. And I do not have an upgrade path for the old desktop because the big tower only accomodates BAT mainboards, not ATX. I did upgrade to the latest available BAT MoBo, but that was in the year 2000...

And I have to say that this is by far the most reliable computer I ever had. The original machine is from 1993, it started as an AMD 486DX266. I upgraded it constantly, but the power supply is still the original one, and my drive A: is still a 5.25" floppy drive. And with my highly optimized WinXP installation it only feels slow when I try to play HD videos... :D

Cheers
manolito

MysteryX

24th April 2017, 16:36

Which part of the script requires Avisynth+? Unless modPlus.dll requires Avisynth+ but it should work with v2.6 too

I decided last-minute to compile ConditionalMT with XP support -- glad it serves someone.

EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

Let's split this up.

EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
EM = EM.mt_expand(mode= mt_circle(zero=true, radius=1))
EM = EM.mt_binarize(92)
EM = EM.Blur(.6)
EM = EM.BicubicResize(C.Width, C.Height)

Which of these lines fail? BicubicResize and Blur are very unlikely. mt_expand or mt_binarize are most likely the ones crashing .. but I wonder why. Neither take blksize as parameter.

burfadel

24th April 2017, 17:05

I found there was no speed difference with the different syntax including the different reference to EM.

Could this have something to do with it?
## "B" - Blending, "BHard" - No blending
BHard = ChangeFPS(C, NewNum, NewDen)
B = ConvertFPS(C, NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## Convert masks to desired frame rate
EM = ChangeFPS(EM, NewNum, NewDen)
Sc = ChangeFPS(SC, NewNum, NewDen)

M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

Aren't you mixing changefps and convertfps which work slightly differently?

MysteryX

24th April 2017, 17:47

Aren't you mixing changefps and convertfps which work slightly differently?
Blending for artifact masking, ChangeFps for scene changes and skipped bad scenes.

manolito

24th April 2017, 18:36

Which part of the script requires Avisynth+? Unless modPlus.dll requires Avisynth+ but it should work with v2.6 too

You are right, contrary to what vcmohan himself says about his plugin it does work with v2.6...
for 2.5 not available for 2.6 Not Available for 32 & 64 bit avisynth+ modPlus
But compared to the old RemoveGrain command there is quite a speed sacrifice. I think I will stick with RemoveGrain...

I decided last-minute to compile ConditionalMT with XP support -- glad it serves someone.

I have no use for AviSynth MT so I took out ConditionalReaderMT and replaced it with the standard ConditionalReader command.

Which of these lines fail? BicubicResize and Blur are very unlikely. mt_expand or mt_binarize are most likely the ones crashing .. but I wonder why. Neither take blksize as parameter.

It is this line:
EM = EM.BicubicResize(C.Width, C.Height)

For SD sources and BlkSize 32 the resized mask is very small, maybe there is a problem within AviSynth to resize such a small mask back to the original size?? No idea, but my workaround sure fixes it.

Cheers
manolito

StainlessS

25th April 2017, 01:33

W=4
H=4
Colorbars(Width=W,Height=H,Pixel_Type="YV12").KillAudio
BicubicResize(1920,1280)

Error message (perhaps referring to chroma size) [have seen this error before, was confused by it].
Resize: source image too small for this resize method. Width=2, Support=2.

EDIT: W=6 H=6 works ok.

EDIT: YV12, I think that error can only happen if source dimensions (as per Manolito) less than 192.
(192 / 32 / 4.0) = 1.5, * 4 = 6, so less than 192 is error [Round(191.9999 / 32 / 4.0) rounds down to 1.0, so 1 * 4 = 4 = error] .

EDIT: If YV24, then same resize error if W=2 H=2 [W=3 H=3 OK], so for YV24 error if less than 192 / 2 = 96.

EDIT: YV12 BilinearResize with source W=4 H=4 OK, Bilinear needs 2 (chroma) samples minimum, BicubicResize needs 3 (chroma) samples.

EDIT: Fixed some cockups.

EDIT: For YV411, (not suggesting that anyone should use that), then minimum width with that script would be 384 else error (assuming that it would work at all with YV411).

MysteryX

25th April 2017, 03:43

Median indeed is slower than RemoveGrain -- a LOT.

RemoveGrain: 636fps @ 24% CPU
Median: 30fps @ 82% CPU

I edited the script to use Median with presets "slower|slow", RemoveGrain with presets "normal|fast", and none with preset "faster". It is done with Eval so that you don't need a reference to a library you're not using.

Output="over" image brightening issue is fixed, as well as another similar mask issue.

I fixed Avisynth+'s conditional filters code so the standard ConditionalFilter will work with MT mode in the next release of AVS+; thus I removed ConditionalFilterMT.

This code is tweaked for YV24 sources; use that for testing.

TODO:
- Mask luma is different between YV12 and YV24; it needs to be normalized (Pinterf)
- SkipOver only works in 8-bit, as the 16-bit value of AverageLuma is 220x higher and needs to be normalized (Pinterf)
- ChangeFps on the mask somehow causes the CPU to stall at 20% (Pinterf)
- Release the next version of AVS+ with conditional MT (Pinterf)

After some testing, I'm not seeing any difference at all between prefilters. I might just go back to RemoveGrain(22); even RemoveGrain(21) is still a *LOT* faster than Median. I'm open to suggestions.

# Frame Rate Converter
# Version: 24-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2
## Prefilter: modPlus for presets "slow|slower", otherwise rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Preset - The speed/quality preset [slower|slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ Output - Output mode [auto|inter|none|mask|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only
## none=ConvertFPS only; mask=mask only; over=mask as cyan overlay for debugging
##
## @ MaskStr - The artifact masking strength (1 to 100). This will apply a gamma curve on the mask
## before processing, where 100 applies no gamma curve and 5 applies 0.05 gamma (Default=5)
##
## @ MaskSAD - Artifact masking strength for bad motion, 0 to disable (Default=190)
##
## @ thSCD1 - MSCDetection scene detection treshold 1 (Default=500)
##
## @ thSCD2 - MSCDetection scene detection treshold 2 (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", bool "FrameDouble", string "Preset",
\ int "BlkSize", int "BlkSizeV", string "Output", float "MaskStr", int "MaskSAD", int "thSCD1", int "thSCD2", int "SkipOver")
{
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
Preset = Default(Preset, "normal")
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskStr = Default(MaskStr, 5)
MaskSAD = Default(MaskSAD, 190)
#MaskOcc = Default(MaskOcc, 64)
thSCD1 = Default(thSCD1, 500)
thSCD2 = Default(thSCD2, 150)
SkipOver = Default(SkipOver, 48)

Assert(Preset == "slower" || Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster",
\ "FrameRateConverter: Preset must be slower, slow, normal, fast or faster")
Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskStr > 0 && MaskStr <= 100, "FrameRateConverter: MaskStr must be between 1 and 100")
Assert(SkipOver >= 0 && MaskStr <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slower, slow, normal, fast, faster
Recalculate = preset == "slow" || preset == "normal"
prefilter = (preset == "slower" || preset == "slow") ? "Median(C, uu = true, vv = true)" :
\ (preset == "normal" || preset == "fast") ? "C.RemoveGrain(21)" : "C"
DCT = preset == "slower" ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
prefiltered = C.RemoveGrain(21) #Eval(prefilter)
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
super = Prefilter != "C" ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

## "EM" - error or artifact mask
EM = MaskSAD > 0 ? C.MMask(bak, ml=MaskSAD, kind=1) : BlankClip(C) # kind=SAD
EMfwd = MaskSAD > 0 ? C.MMask(fwd, ml=MaskSAD, kind=1) : EM # kind=temporal blending
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
#EMocc = MaskOcc > 0 ? C.MMask(bak, ml=MaskOcc, kind=2).mt_inpand() : BlankClip(C) # kind=occlusion
EM = MaskSAD > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
#EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=0.5, mode="lighten", pc_range=true) : EM
SkipEM = EM

## mask strength
EM = EM.Levels(0, MaskStr / 100.0, 255, 0, 255, coring=false)

EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(92)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

## "Sc" - scene detection
Sc = thSCD1 > 0 && thSCD2 > 0 ? C.MSCDetection(bak, thSCD1=thSCD1, thSCD2=thSCD2) : BlankClip(EM)
Sc = SkipOver > 0 ? ConditionalFilter(SkipEM, BlankClip(EM, color=color_white), Sc, "AverageLuma()", ">", string(SkipOver)) : Sc

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "flow")==0) [** flow: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? B
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? EM
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(EM, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true)
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|over)")
return R
}

burfadel

25th April 2017, 09:00

I didn't realise median was so slow, but it makes sense. It's like mode 4 of Removegrain but more advanced. It's probably overkill. Removegrain mode 22/21 is very fast by nature, but probably a little weak and inprecise for best results, which is why median had nicer output.

Maybe you could try something like:
prefiltered = C.minvar(lx=120, ty=100) #Eval(prefilter)

Minvar is part of the same package as Median (modplus).

Small values for lx and ty as larger values would be slower, it's for the computing of global variance. I just chose 45 and 35 as an example, these can be higher or lower (wouldn't go too low though).

EDIT: I originally had the lx and ty a little smaller, but chose higher numbers because it is very fast.

manolito

25th April 2017, 16:06

OK, since the ToDo list only contains stuff for AVS+ (pinterf), I decided that this script is static for my needs.

I did the same to this MysteryX script which I did a while ago to the original JohnMeyer script. I modified the interface so now it is a regular fps converter which can be used like ConvertFPS or ChangeFPS. The only exposed params are FPS, Preset and BlkSize. (Update: Two additional exposed parametes "Dct" and "Mask")

I call it "mx_fps" just like the previous "jm_fps" (which seems to stick...). In my tests it works beautifully with all kinds of sources. Only slightly slower than jm_fps, but better results with many source clips, and so far never worse than jm_fps. Looks like a winner to me...

mx_fps.avsi
# Frame Rate Converter
# Version: 2-June-2019
# By Etienne Charland aka MysteryX
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Pinterf is the one who spent the most time working on the core libraries, adding features and fixing bugs
# Slightly simplified user interface and code cleanup by manolito
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### mx_fps
### Changes the frame rate with interpolation and fine artifact removal.
##
## YV12/YUY2
## Requires: FrameRateConverter.dll, MaskTools2, MvTools2, GRunT, RemoveGrain, FFTW3.dll (in the system32 or syswow64 folder) for Dct values other than 0
##
## @ fps - The new framerate.
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The block size. Supported values are 8, 16 and 32.
## Defaults for 4/3 video of height:
## 0-359: 8
## 400-1199: 16
## 1200-2160: 32
##
## @ BlkSizeV - The vertical block size. (default = BlkSize)
##
## @ Output - Output mode [auto|flow] (default = auto)
## auto=normal artifact masking; flow=interpolation only
##
## @ Prefilter - Specified a custom prefiltered clip. (default = C.RemoveGrain(22))
##
## @ Mask - Enable artifact masking (default = true)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255. Smaller = stronger.
## 0 to disable artifact masking. (default = 100)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (default = 105)
##
## @ SkipTrh - The treshold where a block is counted for the skip mask, between 0 and 255. Smaller = stronger.
## Must be smaller (stronger) than MaskTrh. (default = 55)
##
## @ BlendOver - Try fallback block size when artifacts cover more than specified treshold, or 0 to disable.
## If it fails again, it will revert to frame blending. (default = 60)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## or 0 to disable. (default = 120)
##
## @ Stripes - How to deal with stripes [none|skip|blend] (default=blend)
##
## @ Dct - Overrides DCT parameter (default: Fast=0, Normal=0, Slow=1)
## Useful values are 0, 4 and 1.
##
## @ BlendRatio - Changes the blend ratio used to fill artifact zones. 0 = frame copy and 100 = full blend.
## Other values provide a result in-between to eliminate ghost effects. Default = 40.
##
## Presets
## Fast: Basic interpolation
## Normal: Fast + prefilter + MSuper on prefilter + MRecalculate
## Slow: Normal + DCT=1
##

function mx_fps(clip C, float "fps", string "Preset", int "BlkSize", int "Dct", bool "Mask")
{
Preset = Default(Preset, "normal")
P_SLOW = 1 P_NORMAL = 2 P_FAST = 3
Pset = Preset == "slow" ? P_SLOW : Preset == "normal" ? P_NORMAL : Preset == "fast" ? P_FAST : -1
Assert(Pset != -1, "mx_fps: 'Preset' must be slow, normal or fast {'" + Preset + "'}")
Mask = Default(Mask, true)
Output = Mask ? "auto" : "flow"
O_AUTO = 0 O_FLOW = 1
OPut = Output == "auto" ? O_AUTO : Output == "flow" ? O_FLOW : -1
Stripes = "blend"
S_NONE = 0 S_SKIP = 1 S_BLEND = 2
Stp = Stripes == "none" ? S_NONE : Stripes == "skip" ? S_SKIP : Stripes == "blend" ? S_BLEND : -1

fps = default(fps, 25.000)
NewNum = int(fps * 1000)
NewDen = 1000
DefH = Max(C.Height, C.Width/4*3)
BlkSize = Default(BlkSize, DefH<360 ? 8 : DefH<1200 ? 16 : 32)
Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "mx_fps: BlkSize must be 8, 16 or 32")
BlkSizeV = BlkSize
MaskTrh = 100
SkipTrh = 55
MaskOcc = MaskTrh > 0 ? 105 : 0
BlendOver = 60
SkipOver = 120
CalcPrefilter = Pset != P_FAST
Prefilter = CalcPrefilter ? C.RemoveGrain(22) : C
Recalculate = PSET <= P_NORMAL
Dct = Default(Dct, PSET == P_SLOW ? 1 : 0)
BlendRatio = 40

## "B" - Blending, "BHard" - No blending
Try {
B = C.ConvertFpsLimit(NewNum, NewDen, ratio=BlendRatio)
}
Catch(Err_Msg) {
B = C.ChangeFps(floor(fps * 3/2)).ConvertFpsLimit(NewNum, NewDen, ratio=BlendRatio)
}
BHard = C.ChangeFps(NewNum, NewDen)
Blank = BlankClip(C.ConvertToY8(), color_yuv=$000000)

## Adjust parameters for different block sizes, causing stronger or weaker masks
blk = Max(BlkSize, BlkSizeV)
MaskTrh = MaskTrh + (blk<=8 ? -20 : blk<=16 ? 0 : blk<=32 ? 20 : 35)
SkipTrh = SkipTrh + (blk<=8 ? -18 : blk<=16 ? 0 : blk<=32 ? 16 : 30)
MaskTrh = Max(Min(MaskTrh, 255), 0)
SkipTrh = Max(Min(SkipTrh, 255), 0)
gam = blk<=8 ? .56 : blk<=16 ? .50 : blk<=32 ? .36 : .14

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad=16, vpad=16, sharp=1, rfilter=4) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad=16, vpad=16, levels=1, sharp=1, rfilter=4) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizev=BlkSizeV, overlap = (BlkSize/4+1)/2*2, overlapv = (BlkSizeV/4+1)/2*2, search=3, dct=Dct)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizev=BlkSizeV, overlap = (BlkSize/4+1)/2*2, search=3, dct=Dct)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizev=BlkSizeV/2, overlap = BlkSize/2>4?(BlkSize/8+1)/2*2:0, overlapv = BlkSizeV/2>4?(BlkSizeV/8+1)/2*2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizev=BlkSizeV/2, overlap = BlkSize/2>4?(BlkSize/8+1)/2*2:0, overlapv = BlkSizeV/2>4?(BlkSizeV/8+1)/2*2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.ConvertToY8().MMask(bak, ml=255, kind=1, gamma=1/gam, ysc=255, thSCD2=255) : Blank
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.ConvertToY8().MMask(fwd, ml=255, kind=1, gamma=1/gam, thSCD2=255) : EM
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true) : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.ConvertToY8().MMask(bak, ml=MaskOcc, kind=2, gamma=1/gam, ysc=255, thSCD2=255).mt_inpand() : Blank
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM

# Last mask frame is white. Replace with previous frame.
EM = EM.DeleteFrame(EM.Framecount-1).Loop(2, EM.Framecount-1)

# Create skip mask
EMskip = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(SkipTrh)

## Create artifact correction mask
Try {
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
}
Catch(Err_Msg) {
Try {
BlkSize = BlkSize/2
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
}
Catch(Err_Msg) {
BlkSize = BlkSize/2
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
}
}

# Mask: Stripes
EMstp = C.StripeMask(blksize=BlkSize, blksizev=BlkSizeV, str=min(SkipTrh*2+20, 255), strf=min(SkipTrh+10, 255), thr=23)
\ .BicubicResize(Round(C.Width/BlkSize)*4, Round(C.Height/BlkSizeV)*4)
\ .ContinuousMask(22)
EMstp = EMstp.BicubicResize(EMstp.Width/2, EMstp.Height/2)
\ .mt_binarize(82)
\ .mt_inpand()
\ .mt_expand(mode= mt_circle(zero=true, radius= Stp==S_SKIP ? 12 : 8))
\ .FRC_GaussianBlur42(Stp==S_SKIP ? 8.0 : 2.8)
\ .BicubicResize(C.Width, C.Height)

## "M" - Apply artifact removal
EM = EM.ChangeFPS(NewNum, NewDen)
EMskip = EMskip.ChangeFPS(NewNum, NewDen)
EMstp = EMstp.ChangeFPS(NewNum, NewDen)
M = mt_merge(Flow, B, EM, luma=true, chroma="process")
M = Stp != S_NONE ? mt_merge(M, Stp == S_SKIP ? BHard : B, EMstp, luma=true, chroma="process") : M

## Apply BlendOver and SkipOver
M2 = SkipOver > 0 ? ConditionalFilterMT(EMskip, B, BHard, "AverageLuma", "<", string(SkipOver)) : B
M = BlendOver > 0 ? ConditionalFilterMT(EMskip, M, M2, "AverageLuma", "<", string(BlendOver)) : M

# Output modes
R= (Oput==O_AUTO) [** auto: artifact masking *]
\ ? M
\ : (Oput==O_FLOW) [** flow: interpolation only *]
\ ? Flow
\ : nop

return R
}

##################################
### Emulate [[VariableBlur/GaussianBlur]]
## For YUV, effective chroma blur varies depending on source
## color subsampling - YUV444 has *more* chroma blur, others less.
##
## @ var - works like GaussianBlur's varY
## @ rad - blur radius (<var> squared); overrides <var>
## @ vvar, vrad - vertical var & rad; default same as horizontal
## @ p - final [[GaussResize]] sharpness. Default 19
## (if > 25, blockiness; if < 15, loss of contrast)
##
## version 2013-10-23 raffriff42
## version 2014-05-31 discrete hor. and vert. args
## version 2017-05-21 bugfix: blockiness
##
function FRC_GaussianBlur42(clip C,
\ float "var", float "rad",
\ float "vvar", float "vrad", float "p")
{
var = Max(0.0, Float(Default(var, 1.0)))
rad = Max(1.0, Float(Default(rad, Pow(var, 0.5))))
var = Pow(Min(Max(0.0, rad), 60.0), 1.9) ## arbitrary max radius = 60

vvar = Max(0.0, Float(Default(vvar, var)))
vrad = Max(1.0, Float(Default(vrad, Pow(vvar, 0.5))))
vvar = Pow(Min(Max(0.0, vrad), 60.0), 1.9)
p = Default(p, 19)

w0 = C.Width
h0 = C.Height
w1 = Round(w0/rad)
h1 = Round(h0/vrad)

B = C.BilinearResize(
\ Min(Max(4, w1 + (w1 % 2)), w0),
\ Min(Max(4, h1 + (h1 % 2)), h0))

B = B.Blur(1.0).Blur(1.0)

return (var<0.01 && vvar<0.01) ? C
\ : (B.Width>8 && B.Height>8) ? B.GaussResize(w0, h0, p=p)
\ : B.BilinearResize(w0, h0)
}

Cheers
manolito

//EDIT//
Here is a complete AIO package with all the required plugins:
https://files.videohelp.com/u/172211/mx_fps%20AIO.zip
Please note that I used older (but stable) plugin versions which also run on old hardware.
This AIO package also contains a version of the script which does not require a SSE2 capable CPU (had to remove the stripe mask feature for this).

//EDIT 2//
Changed the user interface to expose the DCT value which will override the default. Useful if you want to use DCT=4. Also added a "Mask" parameter which makes it easy to enable or disable artifact masking.
NOTE: This script version needs the latest version 1.3 of "FrameRateConverter.dll". It comes with the AIO package.

//EDIT3//
Added the "ConvertFpsLimit" function from the latest FrameRateConverter 1.3

StainlessS

25th April 2017, 16:17

Manolito,

Requires: MaskTools2, MvTools2, RemoveGrain and fftw3.dll (in the System32 or SysWOW64 folder)

What is it that needs fftw3.dll, I see nothing that sticks out, and does not seem to be required in earlier mods ?

manolito

25th April 2017, 16:49

It is required for DCT=1 in mvtools2 (at least in Fizick's latest version 2.5.11.22). In this script Preset = "Slow" will use "DCT=1", and without fftw3.dll you will get a crash...

Cheers
manolito

StainlessS

25th April 2017, 17:14

Cheers Mani, had no idea that it was required, not noted in docs.
There is mention that it uses fftw3 header, and several mentions of fft regards to DCT but no specific requirement mentioned anywhere.
Deleted fftw3.dll from system32 and still worked ok, then remembered something about something also being able to use lib named as
libfftw3f-3.dll, so deleted that too, and crashes, just as you said.

God, its a bit slow with that setting, dont think I'm ever likely to use it.

manolito

25th April 2017, 17:35

God, its a bit slow with that setting, dont think I'm ever likely to use it.

Yeah right, but for some sources it does make a real difference...

Cheers
manolito

MysteryX

25th April 2017, 17:42

A caveat to be careful about: YV12 and YV24 will have different artifact mask strengths. YV12 will have much weaker mask.

Also, for mask strength, besides adjusting Gamma, the other line that will make a difference is "binarize(92)".

You guys can play around with Gamma and Binarize, in YV12 or YV24, and see what works best for you.

StainlessS

25th April 2017, 18:07

A caveat to be careful about: YV12 and YV24 will have different artifact mask strengths. YV12 will have much weaker mask.

If chroma masks cannot be normalized somehow, then perhaps chroma introduces too big a random variation to be useful, as cannot be separately processed/controlled/combined. (or everything converted to YV24, or everything to YV12).

johnmeyer

25th April 2017, 18:23

We had a discussion about DCT=1 making a difference, and I did some tests and couldn't see a difference. As I remember, the evidence posted in that thread was not convincing (that DCT=1 helped reduce ME artifacts). The only reason I bring this up is that I'd hate to see lots of effort chasing issues related to DCT=1, especially since, in addition to maybe not making much difference, it is so doggedly slow. Instead, I'm hoping most of the focus will be on enhancing the effort to produce a version that can detect and mask artifacts. I've been waiting for the dust to settle before I jump in and start testing this.

burfadel

25th April 2017, 18:33

Would there be any advantage running the script natively as YV24 (that is, using converttoyv24() if not already in YV24)?

Have you looked into minvar function? It should provide the benefit of the median filter without the speed penalty.

MysteryX

25th April 2017, 19:06

Any benefit to processing in YV24 will be lost just as soon as you re-convert to YV12, but if you do further processing in YV24, you can see better chroma precision before discarding the data.

But this script so far was tweaked for YV24. We may want to compare YV12, YV24 and Luma-only masks. Who knows -- maybe a Luma-only mask would be more accurate by discarding chroma noise -- only one way to know.

Edit: Mask cannot be applied to only Luma, because it is based on the vectors analysis which takes all planes for motion interpolation.

Right now, YV12 produces stronger mask than YV24.

YV24 / YV12
https://s1.postimg.org/gge0ottu3/Mask_YV24.png (https://postimg.org/image/gge0ottu3/) https://s1.postimg.org/qp6hunhvv/Mask_YV12.png (https://postimg.org/image/qp6hunhvv/)

Did another test using MMask's Gamma. This time YV12 produces weaker mask. YV24 masks just look better in general.
https://s13.postimg.org/v806swshv/Mask_YV24.png (https://postimg.org/image/v806swshv/) https://s13.postimg.org/moc7heurn/Mask_YV12.png (https://postimg.org/image/moc7heurn/)

MysteryX

26th April 2017, 00:11

I've done plenty of changes to the artifact removal by tweaking MMask's arguments.

MMask's ml is unecessary because it simply sets to 255 values above that treshold -- which is exactly what mt_binarize is for; thus I set that to 255 and removed MaskSAD parameter.

MMask has a gamma property, which removes the need for another call to Levels. The mask level adjustment function has been removed.

MMask also has ysc which says what to do on scene changes: set everything to 255. Thus, I no longer need MSCDetection.

Mask adjustment settings are now MaskGam (gamma, applied during MMask) and MaskTrh (treshold, applied during mt_binarize).

MaskGam affects SkipOver -- and I believe I've found the right gamma value. If we find no need to change it, I might remove that property.

Because SkipOver now operate on a gamma-adjusted mask, it alters SkipOver behaviors.

After plenty of testing, I've set default to MaskGam=.6, MaskTrh=90, SkipOver=15

I removed Median and "slower" preset, went back to RemoveGrain(22) until someone can prove benefits of doing more than that.

To please manolito, I changed the arguments order for simplicity. First parameters are now: NewNum, NewDen, Preset, BlkSize

Again, settings are tweaked for YV24, and YV12 produces a different mask strength so settings will need to be adjusted.

Because now I know what I'm doing (instead of randomly using old code), the mask now both covers artifacts more precisely and leaves out more areas. I found it to be better in almost every cases.

# Frame Rate Converter
# Version: 25-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only
## none=ConvertFPS only; mask=mask only; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1.
## Higher value means stronger mask. (Default=.6)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255. 0 to disable artifact masking.
## Higher value means stronger mask. (Default=90)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. Does not work in Avisynth+ MT mode. (Default=48)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "SkipOver")
{
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
Preset = Default(Preset, "normal")
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskGam = Default(MaskGam, .6)
MaskTrh = Default(MaskTrh, 90)
SkipOver = Default(SkipOver, 15)

Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster",
\ "FrameRateConverter: Preset must be slow, normal, fast or faster")
Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast, faster
Recalculate = preset == "normal"
prefilter = preset != "faster"
DCT = preset == "slower" ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
prefiltered = Prefilter ? C.RemoveGrain(22) : C
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
super = Prefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2)

## "EM" - error or artifact mask
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, ysc=255, gamma=1.0/MaskGam) : BlankClip(C) # kind=SAD
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam) : EM # kind=temporal blending
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
SkipEM = EM

EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
\ .mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(SkipEM, BlankClip(EM, color=color_white), BlankClip(EM),
\ "AverageLuma()", ">", string(SkipOver)) : BlankClip(EM)

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true) : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "flow")==0) [** flow: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? B
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? EM
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Flow.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(EM, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true)
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|over)")
return R
}

MysteryX

26th April 2017, 02:37

I was wondering what it would change if I scaled the mask down to 4 pixels per block instead of 1 pixel per block.

Regular
https://s8.postimg.org/ggqos82xd/Mask1.png (https://postimg.org/image/ggqos82xd/)
4 pixels per block
https://s8.postimg.org/4g58rhvip/Mask2.png (https://postimg.org/image/4g58rhvip/)

I'll definitely leave it as it is. This change is not bringing good results: too much false positives and chip-chopping between clips. I'm surprised at the difference though.

manolito

26th April 2017, 08:01

Thanks MX for the improvements...

I think there are a few small oversights in the script, could you please check?

Line 54: Wrong default value for SkipOver
Line 80: Recalculate really only for preset "normal", not for "slow"?
Line 82: DCT = 1 for the no longer existing preset "slower" instead of "slow"?

Cheers
manolito

burfadel

26th April 2017, 11:53

Did you try:
prefiltered = Prefilter ? C.minvar(lx=120, ty=100) : C

Probably the only adjustment would be xgrid (which automatically sets ygrid if not specified). Default is 5, your can go lower number for weaker, or higher for stronger effect. The more noise that there is, the greater the noise reduction, so I think it's ideal for the prefilter (especially considering its very fast). You can choose higher or lower values for lx and ty (these must be specified). I think they're reasonable values.

It should possibly help to reduce some artifacting as well, if the artifacting is caused by noise as was the case with the improvement with median. This filter is very much faster though, probably faster than Removegrain (or so it seems).

raffriff42

26th April 2017, 13:37

Noise is not usually the major issue (for me) -- all the things you see in JM's parade clip are. Motion-detection-busting occlusions, or "antlers" for short :)

Most of the time a simple Blur gives plenty of prefiltering. Why not a prefilter clip argument, as used by SMDegrain (http://avisynth.nl/index.php/SMDegrain)("prefilter") , Interp2 (https://forum.doom9.org/showthread.php?p=1391871#post1391871)("denoised") and probably others, to allow users with noise problems to supply their own denoising? There are many filters available to handle the many types of noise -- high frequency? low frequency? gaussian? mosquito? RF? jitter? flicker? streaking? herringbone?

burfadel

26th April 2017, 14:01

Blur blur's edges though, you want to maintain the picture detail whilst removing noise, obviously with minimal speed impact.

raffriff42

26th April 2017, 14:27

Most of the time a simple Blur gives plenty of prefiltering...
Blur blur's edges though, you want to maintain the picture detail whilst removing noise, obviously with minimal speed impact.The output of Blur (or any pre-filter) goes to motion analysis, not the main video path.

burfadel

26th April 2017, 15:44

I know, but the motion analysis wouldn't be accurate if the edges are not representative of the main clip. You want to motion picture detail, not noise, so a good noise filter that maintains lines would product the most accurate analysis results.

MysteryX

26th April 2017, 16:00

Did you try:
prefiltered = Prefilter ? C.minvar(lx=120, ty=100) : C

How about posting a benchmark comparison between RemoveGrain and minvar?

Then you can post a screenshot comparison of improvements you see.

We can make up our mind from there.

manolito

26th April 2017, 19:17

Did a couple of tests comparing the old mask functions to the new ones, The only real difference I found is that the new SkipOver threshold is way lower than it used to be in the old version. IMO skipping starts much too early, so far I raised the threshold from 15 to 20 which looks better to me. I'll do more testing...

Cheers
manolito

MysteryX

26th April 2017, 20:13

Line 54: Wrong default value for SkipOver
Line 80: Recalculate really only for preset "normal", not for "slow"?
Line 82: DCT = 1 for the no longer existing preset "slower" instead of "slow"?

Thanks, fixing those

I'm also re-adding occlusion masking. On its own, it doesn't do anything, but when added to the raw mask, it does expand the mask to cover larger. I'm tweaking it to alter the mask just the right way.

This time, I'm tweaking settings for YV12 instead of YV24, as MVTools2 will normalize to the YV12 mask levels (although Pinterf's latest version still doesn't do it right).

I'll post the new script soon.

MysteryX

26th April 2017, 21:27

Here's the updated code. I'm re-added occlusion masks. Now it is based on YV12 mask levels.

Overall, it seems to be giving great results.

On some clips, however, SkipOver is too high; but if I set it any higher the moose scene doesn't get detected. I found that gamma=.5 works best in most cases but it "may" help reduce the SkipOver discrepancy with a gamma of .4 or .3

There's a bit more work to do on the "SkipOver" part.

MMask's scene detection also seems to trigger too often -- I disabled it and it's working better, but that might require more investigation.

# Frame Rate Converter
# Version: 26-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only;
## mask=mask only; skip=mask used by SkipOver; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc", int "SkipOver", clip "Prefilter")
{
Output = Default(Output, "auto")
FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
Preset = Default(Preset, "normal")
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskGam = Default(MaskGam, .5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = Default(MaskOcc, 150)
MaskOcc = MaskTrh > 0 ? MaskOcc : 0
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Preset != "faster"
Prefilter = Default(Prefilter, Preset != "faster" ? C.RemoveGrain(22) : C)

Assert(Preset == "slow" || Preset == "normal" || Preset == "fast" || Preset == "faster",
\ "FrameRateConverter: Preset must be slow, normal, fast or faster")
Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
Assert(MaskOcc >= 0 && MaskOcc <= 255, "FrameRateConverter: MaskOcc must be between 0 and 255")
Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast, faster
Recalculate = preset == "slow" || preset == "normal"
DCT = preset == "slow" ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255) : BlankClip(C)
EM = EM.ConvertToY8()
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(100)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
EMskipOut = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip, BlankClip(EM, color=$FFFFFF), BlankClip(EM),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM)
Sc = Sc.mt_binarize(128)

# Display SkipOver value on Output="over
Global GEMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver = Flow.ScriptClip("Subtitle(string(GEMskip.AverageLuma()))")

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

R = (StrCmpi(Output, "auto")==0) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (StrCmpi(Output, "flow")==0) [** flow: interpolation only *]
\ ? Flow
\ : (StrCmpi(Output, "none")==0) [** none: ConvertFPS only *]
\ ? B
\ : (StrCmpi(Output, "mask")==0) [** mask: mask only *]
\ ? EM
\ : (StrCmpi(Output, "skip")==0) [** skip: skip mask *]
\ ? EMskipOut
\ : (StrCmpi(Output, "over")==0) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ FlowOver.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : Assert(false, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|over)")
return R
}

MysteryX

27th April 2017, 00:33

Code updated. I've done a lot of work to improve SkipOver, to separate important vs unimportant noise. The logic has been changed. I had the problem that the moose creates small but serious artifacts while lighting changes create weak but large artifacts -- so I had to give more importance to small/strong artifacts and discard big/weaker ones.

Scene detection has also been disabled on MFlowFps and MMask which were creating additional skipped frames.

Added output="skip" to view the mask being used by SkipOver.

I'd like to display the "skip" value on each frame with output="over" but haven't figured out how to do so.

MysteryX

27th April 2017, 03:36

Code updated. Output="over" now displays the "SkipOver" value of each frame, and masks are now processed in Y8 which results in higher performance.

MysteryX

27th April 2017, 05:12

Added "Prefilter" argument to use custom prefilter.

SC mask was in the 16-235 range which resulted in unwanted blending of frames. This has been fixed and results in much better performance.

It plays weird in AVS+, even without MT... it plays a bunch of frames, pauses, then continues playing a bunch of frames. With MT, it stalls at low CPU usage, but first it would be good to debug the issue without MT.

StainlessS

27th April 2017, 13:41

MysteryX, no idea if of help but RT_Stats has eg

Last.RT_Subtitle("%d ] %f",current_frame,Last.AverageLuma()) # MUCH faster than subtitle, MONOSPACED fixed size font (10x20)
RT_WriteFile("Bug.log","%d ] %f",current_frame,Last.AverageLuma,Append=True)
RT_DebugF("%d ] %f",current_frame,Last.AverageLuma,name="MysteryX: ") # to DebugView (google)

EDIT: In blue Added

StainlessS

27th April 2017, 15:24

MX,

# Display SkipOver value on Output="over
Disp_EMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver = Flow.GScriptClip("Subtitle(string(Disp_EMskip.AverageLuma))",args="Disp_EMskip",Local=True)

Above removes Global entirely and so function is not single instance limited,
new req Grunt, but that should be built-in to Avisynth, along with GScript (GScript is builtin in AVS+, already).

Completely untested.

EDIT: Having multi-instace ability to see side-by-side differences with alternate settings is a good thing to have.
EDIT: I think SRestore() and QTGMC() are screwed in that respect, ie single instance only (Although I think Martin53 may have fixed recent version of SRestore).

MysteryX

27th April 2017, 16:57

Agree, GRunT would work better -- but that would add extra dependency only for debug code. I also agree GRunT should be built into AVS+, but even so, it then wouldn't work under AVS 2.6

The code I wrote at least works for a single instance, and is only for debug mode. There's no point in activating debug mode for 2 instances simultaneously.

StainlessS

27th April 2017, 18:44

GRunt only used in Debug mode, no globals.

NOTE, Preset Fast is not used anywhere in script (other than Preset string check).

# Frame Rate Converter
# Version: 26-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only;
## mask=mask only; skip=mask used by SkipOver; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc", int "SkipOver", clip "Prefilter")
{
Output = Default(Output, "auto")
Preset = Default(Preset, "normal")
P_SLOW=0 P_NORMAL=1 P_FAST=2 P_FASTER=3 # NOTE, Preset="Fast" is NOT used anywhere in Script
O_AUTO=0 O_FLOW=1 O_NONE=2 O_MASK=3 O_SKIP=4 O_OVER=5
Pset=Preset=="slow"?P_SLOW:Preset=="normal"?P_NORMAL:Preset=="fast"?P_FAST:Preset=="faster"?P_FASTER:-1
OPut=Output=="auto"?O_AUTO:Output=="flow"?O_FLOW:Output=="none"?O_NONE:Output=="mask"?O_MASK:Output=="skip"?O_SKIP:Output=="over"?O_OVER:-1
Assert(Pset!=-1, "FrameRateConverter: Preset must be slow, normal, fast or faster")
Assert(OPut!=-1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|over)")

FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskGam = Default(MaskGam, .5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = Default(MaskOcc, 150)
MaskOcc = MaskTrh > 0 ? MaskOcc : 0
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Pset != P_FASTER
Prefilter = Default(Prefilter, Pset != P_FASTER ? C.RemoveGrain(22) : C)

Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
Assert(MaskOcc >= 0 && MaskOcc <= 255, "FrameRateConverter: MaskOcc must be between 0 and 255")
Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast, faster
Recalculate = PSet==P_SLOW || PSET==P_NORMAL
DCT = PSet==P_SLOW ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255) : BlankClip(C)
EM = EM.ConvertToY8()
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(100)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
EMskipOut = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip, BlankClip(EM, color=$FFFFFF), BlankClip(EM),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM)
Sc = Sc.mt_binarize(128)

# Display SkipOver value on Output="over
Disp_EMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver = (Oput==O_OVER) ? Flow.GScriptClip("Subtitle(string(Disp_EMskip.AverageLuma))",args="Disp_EMskip",Local=True) : 0

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

R= (Oput==O_AUTO) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (Oput==O_FLOW) [** flow: interpolation only *]
\ ? Flow
\ : (Oput==O_NONE) [** none: ConvertFPS only *]
\ ? B
\ : (Oput==O_MASK) [** mask: mask only *]
\ ? EM
\ : (Oput==O_SKIP) [** skip: skip mask *]
\ ? EMskipOut
\ : (Oput==O_OVER) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ FlowOver.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")
return R
}

Oops, changed GScript to GRunt

EDIT: Untested, I dont have requirements setup.

StainlessS

27th April 2017, 22:10

MX, I presume that you are more familiar with the code than I, so take a peek here,
Puts up a load of stacked clips, looks to me like M is wrong, but then I'm not sure what it should look like
(I mean for the weird backwards forwards jumping thing).

# Frame Rate Converter
# Version: 26-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast|faster]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only;
## mask=mask only; skip=mask used by SkipOver; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc", int "SkipOver", clip "Prefilter",Bool "Debug")
{
Function FRSub(clip c,string tit){ # Auto Convert clips to YV12 if not already (NOP if YV12)
c.ConvertToYV12.Subtitle(tit,align=5)
}
Output = Default(Output, "auto")
Preset = Default(Preset, "normal")
P_SLOW=0 P_NORMAL=1 P_FAST=2 P_FASTER=3 # NOTE, Preset="Fast" is NOT used anywhere in Script
O_AUTO=0 O_FLOW=1 O_NONE=2 O_MASK=3 O_SKIP=4 O_OVER=5
Pset=Preset=="slow"?P_SLOW:Preset=="normal"?P_NORMAL:Preset=="fast"?P_FAST:Preset=="faster"?P_FASTER:-1
OPut=Output=="auto"?O_AUTO:Output=="flow"?O_FLOW:Output=="none"?O_NONE:Output=="mask"?O_MASK:Output=="skip"?O_SKIP:Output=="over"?O_OVER:-1
Assert(Pset!=-1, "FrameRateConverter: Preset must be slow, normal, fast or faster")
Assert(OPut!=-1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|over)")

FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskGam = Default(MaskGam, .5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = Default(MaskOcc, 150)
MaskOcc = MaskTrh > 0 ? MaskOcc : 0
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Pset != P_FASTER
Prefilter = Default(Prefilter, Pset != P_FASTER ? C.RemoveGrain(22) : C)
Debug = Default(Debug,False)

Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
Assert(MaskOcc >= 0 && MaskOcc <= 255, "FrameRateConverter: MaskOcc must be between 0 and 255")
Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast, faster
Recalculate = PSet==P_SLOW || PSET==P_NORMAL
DCT = PSet==P_SLOW ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255) : BlankClip(C)
EM = EM.ConvertToY8()
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(100)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
EMskipOut = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip, BlankClip(EM, color=$FFFFFF), BlankClip(EM),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM)
Sc = Sc.mt_binarize(128)

# Display SkipOver value on Output="over
Disp_EMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver=(Oput==O_OVER||DEBUG) ? Flow.GScriptClip("Subtitle(string(Disp_EMskip.AverageLuma))",args="Disp_EMskip",Local=True) : 0

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

Z_AUTO = (DEBUG||OPut==O_AUTO) ? (FrameDouble ? Interleave(C, M) : M) : 0
Z_FLOW = (DEBUG||OPut==O_FLOW) ? Flow : 0
Z_NONE = (DEBUG||OPut==O_NONE) ? B : 0
Z_MASK = (DEBUG||OPut==O_MASK) ? EM : 0
Z_SKIP = (DEBUG||OPut==O_SKIP) ? EMskipOut : 0
Z_OVER = (DEBUG||Oput==O_OVER)
\ ? mt_merge(FlowOver.Overlay(MergeRGB(BlankClip(EM), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : 0

R=(Debug)
\ ? StackHorizontal(
\ StackVertical(FRSub(Z_AUTO,"Auto"),FRSub(Z_FLOW,"Flow"),FRSub(Z_NONE.ChangeFps(NewNum, NewDen),"None")),
\ StackVertical(FRSub(Z_MASK,"Mask"),FRSub(Z_SKIP,"Skip"),FRSub(Z_OVER,"Over")),
\ StackVertical(FRSub(EM,"EM"),FRSub(SC,"SC"),FRSub(M,"M"))
\ )
\ :(Oput==O_AUTO) ? Z_AUTO [** auto: artifact masking *]
\ : (Oput==O_FLOW) ? Z_FLOW [** flow: interpolation only *]
\ : (Oput==O_NONE) ? Z_NONE [** none: ConvertFPS only *]
\ : (Oput==O_MASK) ? Z_MASK [** mask: mask only *]
\ : (Oput==O_SKIP) ? Z_SKIP [** skip: skip mask *]
\ : (Oput==O_OVER) ? Z_OVER [** over: mask as cyan overlay *]
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")
return R
}

Avisource("F:\v\XMen2.avi")

FrameRateConverter(Debug=True)
#FrameRateConverter(Output="Auto")
#FrameRateConverter(Output="Flow")
#FrameRateConverter(Output="None")
#FrameRateConverter(Output="Mask")
#FrameRateConverter(Output="Skip")
#FrameRateConverter(Output="Over")

Could be expanded / modded to find out exactly at what point if goes wrong.

EDIT: M aint right again :(

MysteryX

27th April 2017, 22:22

Thanks StainlessS. Preset "faster" was for no prefilter, but considering RemoveGrain(22) is so fast and that it can be customized, I removed that preset.

I added output="raw" to see the raw artifact mask.

Made a few other minor tweaks.

# Frame Rate Converter
# Version: 27-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools (default prefilter), GRunT (output="over")
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|raw|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; mask=mask only;
## skip=mask used by SkipOver; raw=raw mask; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc", int "SkipOver", clip "Prefilter")
{
Preset = Default(Preset, "normal")
P_SLOW=0 P_NORMAL=1 P_FAST=2
Pset=Preset=="slow"?P_SLOW:Preset=="normal"?P_NORMAL:Preset=="fast"?P_FAST:-1
Assert(Pset!=-1, "FrameRateConverter: Preset must be slow, normal or fast")
Output = Default(Output, "auto")
O_AUTO=0 O_FLOW=1 O_NONE=2 O_MASK=3 O_SKIP=4 O_RAW=5 O_OVER=6
OPut=Output=="auto"?O_AUTO:Output=="flow"?O_FLOW:Output=="none"?O_NONE:Output=="mask"?O_MASK:Output=="skip"?O_SKIP:Output=="raw"?O_RAW:Output=="over"?O_OVER:-1
Assert(OPut!=-1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|raw|over)")

FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskGam = Default(MaskGam, .5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = Default(MaskOcc, 150)
MaskOcc = MaskTrh > 0 ? MaskOcc : 0
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Pset != P_FAST
Prefilter = Default(Prefilter, C.RemoveGrain(22))

Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
Assert(MaskOcc >= 0 && MaskOcc <= 255, "FrameRateConverter: MaskOcc must be between 0 and 255")
Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast
Recalculate = PSet==P_SLOW || PSET==P_NORMAL
DCT = PSet==P_SLOW ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255).ConvertToY8() :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = EM.ConvertToY8()
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
OutRaw = EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(100)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
OutSkip = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip, BlankClip(EM, color_yuv=$ff0000), BlankClip(EM, color_yuv=$000000),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM, color_yuv=$000000)

# Display SkipOver value on Output="over
Disp_EMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver = (Oput==O_OVER) ? Flow.GScriptClip("Subtitle(string(Disp_EMskip.AverageLuma))",args="Disp_EMskip",Local=True) : 0

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

R= (Oput==O_AUTO) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, M) : M)
\ : (Oput==O_FLOW) [** flow: interpolation only *]
\ ? Flow
\ : (Oput==O_NONE) [** none: ConvertFPS only *]
\ ? B
\ : (Oput==O_MASK) [** mask: mask only *]
\ ? EM
\ : (Oput==O_SKIP) [** skip: skip mask *]
\ ? OutSkip
\ : (Oput==O_RAW) [** raw: raw mask *]
\ ? OutRaw
\ : (Oput==O_OVER) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ FlowOver.Overlay(MergeRGB(BlankClip(EM, color_yuv=$000000), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")
return R
}

Performance is much better in this version than in previous versions. It no longer freezes every few frames, and with Prefetch(8), it stalls at 50% CPU instead of 20%.

Preset="fast"

FPS (min | max | average): 52.66 | 224.8 | 92.41
Memory usage (phys | virt): 535 | 541 MiB
Thread count: 29
CPU usage (average): 56%

Preset="normal"

Frames processed: 749 (0 - 748)
FPS (min | max | average): 8.133 | 146152 | 57.98
Memory usage (phys | virt): 586 | 593 MiB
Thread count: 29
CPU usage (average): 48%

Preset="slow"

FPS (min | max | average): 3.729 | 167031 | 32.70
Memory usage (phys | virt): 461 | 470 MiB
Thread count: 29
CPU usage (average): 38%

StainlessS

27th April 2017, 23:28

Ill repost previous one with DEBUG option, really good idea to fully implement this, not fully working, only working for FrameDouble mode.
Need to ensure same framerate for each debug stack window, FRSub() nested function auto converts non YV12 to YV12 (so in one place only).

Take a look, I recommend you get it fully working, will make future development/debugging a whole lot easier.

Posting update now.

EDIT: The flashes are coming from same time at original framerate, ie at frame 10000, flash frame is source frame 5000.

EDIT: Perhaps easier to fail if debug and not FrameDouble.

MysteryX

27th April 2017, 23:35

Here are screenshot comparisons.

- FrameRateConverter SkipOver=0
- FrameRateConverter SkipOver=0, dct=1
- Interframe
- Interframe Tuning="smooth"

JM's clip, frame 106
https://s23.postimg.org/oa6zk53if/106-frc0.png (https://postimg.org/image/oa6zk53if/) https://s23.postimg.org/up60gta87/106-frc1.png (https://postimg.org/image/up60gta87/) https://s23.postimg.org/4i4tkuryf/106-inter0.png (https://postimg.org/image/4i4tkuryf/) https://s23.postimg.org/ax3uhiyo7/106-inter1.png (https://postimg.org/image/ax3uhiyo7/)

Frame 377
https://s23.postimg.org/dfpji7kef/377-frc0.png (https://postimg.org/image/dfpji7kef/) https://s23.postimg.org/azsbi3rpj/377-frc1.png (https://postimg.org/image/azsbi3rpj/) https://s23.postimg.org/q9s6pan7r/377-inter0.png (https://postimg.org/image/q9s6pan7r/) https://s23.postimg.org/6gg2wl9tz/377-inter1.png (https://postimg.org/image/6gg2wl9tz/)

Frame 582
https://s23.postimg.org/3nmvck9hj/582-frc0.png (https://postimg.org/image/3nmvck9hj/) https://s23.postimg.org/hvck17m6f/582-frc1.png (https://postimg.org/image/hvck17m6f/) https://s23.postimg.org/9ed1qahhj/582-inter0.png (https://postimg.org/image/9ed1qahhj/) https://s23.postimg.org/8q477cirr/582-inter1.png (https://postimg.org/image/8q477cirr/)

My dreaded "frame 4477"
https://s23.postimg.org/vg3c0c1zb/4477-frc0.png (https://postimg.org/image/vg3c0c1zb/) https://s23.postimg.org/dek72j7yf/4477-frc1.png (https://postimg.org/image/dek72j7yf/) https://s23.postimg.org/7rnu525fr/4477-inter0.png (https://postimg.org/image/7rnu525fr/) https://s23.postimg.org/rax0s5tl3/4477-inter1.png (https://postimg.org/image/rax0s5tl3/)
Frame 4655
https://s23.postimg.org/4aqdftvrb/4655-frc0.png (https://postimg.org/image/4aqdftvrb/) https://s23.postimg.org/4c0b98xl3/4655-frc1.png (https://postimg.org/image/4c0b98xl3/) https://s23.postimg.org/a1gjtk3rb/4655-inter0.png (https://postimg.org/image/a1gjtk3rb/) https://s23.postimg.org/fqwsdv9xj/4655-inter1.png (https://postimg.org/image/fqwsdv9xj/)

MysteryX

27th April 2017, 23:36

Frame 6836
https://s23.postimg.org/cy3ktu9l3/6836-frc0.png (https://postimg.org/image/cy3ktu9l3/) https://s23.postimg.org/m75r3yih3/6836-frc1.png (https://postimg.org/image/m75r3yih3/) https://s23.postimg.org/fhz7nxx53/6836-inter0.png (https://postimg.org/image/fhz7nxx53/) https://s23.postimg.org/ptbkglouf/6836-inter1.png (https://postimg.org/image/ptbkglouf/)

Observations:

- The difference between Interframe and FrameRateConverter is HUGE!

- dct=1 brings significant improvements, but causes the CPU to stall. It's not just twice slower, it makes VirtualDub freeze for 10 seconds.

- dct=1 causes a slightly stronger artifact mask, which causes more blending to occur, and thus more double shadows. Settings need to be adjusted consequently.

- SVP/Interframe always appears softer, as if it was applying some level of frame blending on every frame

- On dreaded frame 4477, Interframe shows a "hard blending", followed by that same image mixed with ugly artifacts, which brings an original frame followed by gradually increasing artifacts on a still image.

- To compare scenes with heavy artifacts, I had to set SkipOver=0 otherwise these ugly frames aren't being generated

johnmeyer

27th April 2017, 23:58

It looks like you are in the home stretch, so I really need to dig into this so I can provide some feedback before you're all through.

To my surprise, the DCT=1 does indeed seem to provide better results. However, I want to compare the results to a "native" MVTools2 implementation, like that script I posted that people have been using, which is nothing more than the MVTools2 sample code, but with some very "finely tuned" parameters. I think that would provide a better benchmark than Interframe which, based on reading lots of posts, is really not the thing to use as the basis for comparison when you are looking at quality, because SVPFlow/Interframe is mostly about speed, not quality.

StainlessS

28th April 2017, 00:27

MX, are you not getting the weird frame flashes that I'm getting (and Selur metions in VS forum) ?
@ FrameDouble, Flash frames @ output frame 10000, are coming from input frame 2500.

Avisource("F:\v\XMen2.avi")
ShowFrameNumber

FrameRateConverter(Debug=True)

I'll make same changes to my version script as yours and repost.

EDIT: Also, debug frame M, @ output frame ~10000 is alternating between input frames ~5000 and ~10000.
Something weird in there somewhere.

StainlessS

28th April 2017, 00:56

Modified similar to your last post, Still getting weird flashes, and M jumps from eg frame 10000, to frame 5000 when flashing.

# Frame Rate Converter
# Version: 27-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools (default prefilter), GRunT (output="over")
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|raw|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; mask=mask only;
## skip=mask used by SkipOver; raw=raw mask; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc", int "SkipOver", clip "Prefilter",Bool "Debug")
{
Function FRSub(clip c,string tit){ # Auto Convert clips to YV12 if not already (NOP if YV12)
c.ConvertToYV12.Subtitle(tit,align=5)
}
Preset = Default(Preset, "normal")
P_SLOW=0 P_NORMAL=1 P_FAST=2
Pset=Preset=="slow"?P_SLOW:Preset=="normal"?P_NORMAL:Preset=="fast"?P_FAST:-1
Assert(Pset!=-1, "FrameRateConverter: Preset must be slow, normal or fast")
Output = Default(Output, "auto")
O_AUTO=0 O_FLOW=1 O_NONE=2 O_MASK=3 O_SKIP=4 O_RAW=5 O_OVER=6
OPut=Output=="auto"?O_AUTO:Output=="flow"?O_FLOW:Output=="none"?O_NONE:Output=="mask"?O_MASK:Output=="skip"?O_SKIP:Output=="raw"?O_RAW:Output=="over"?O_OVER:-1
Assert(OPut!=-1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|raw|over)")

FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = BlkSize > BlkSizeV ? BlkSizeV : BlkSize
MaskGam = Default(MaskGam, .5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = Default(MaskOcc, 150)
MaskOcc = MaskTrh > 0 ? MaskOcc : 0
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Pset != P_FAST
Prefilter = Default(Prefilter, C.RemoveGrain(22))
Debug = Default(Debug,False)

Assert(BlkSize == 8 || BlkSize == 16 || BlkSize == 32, "FrameRateConverter: BlkSize must be 8, 16 or 32")
Assert(BlkSizeV == 8 || BlkSizeV == 16 || BlkSizeV == 32, "FrameRateConverter: BlkSizeV must be 8, 16 or 32")
Assert(MaskGam > 0 && MaskGam <= 1, "FrameRateConverter: MaskGam must be between 0 and 1")
Assert(MaskTrh >= 0 && MaskTrh <= 255, "FrameRateConverter: MaskTrh must be between 0 and 255")
Assert(MaskOcc >= 0 && MaskOcc <= 255, "FrameRateConverter: MaskOcc must be between 0 and 255")
Assert(SkipOver >= 0 && SkipOver <= 255, "FrameRateConverter: SkipOver must be between 0 and 255")

# Performance settings: slow, normal, fast
Recalculate = PSet==P_SLOW || PSET==P_NORMAL
DCT = PSet==P_SLOW ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)
B = FrameDouble ? SelectOdd(B) : B

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, blksizeV=BlkSizeV, overlap = blkmin>8?4:blkmin>4?2:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, blksizeV=BlkSizeV/2, overlap = blkmin>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255).ConvertToY8() :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = EM.ConvertToY8()
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
OutRaw = EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(100)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
OutSkip = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip, BlankClip(EM, color_yuv=$ff0000), BlankClip(EM, color_yuv=$000000),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM, color_yuv=$000000)

# Display SkipOver value on Output="over
Disp_EMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver = (Oput==O_OVER||DEBUG) ? Flow.GScriptClip("Subtitle(string(Disp_EMskip.AverageLuma))",args="Disp_EMskip",Local=True) : 0

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

Z_AUTO = (DEBUG||OPut==O_AUTO) ? (FrameDouble ? Interleave(C, M) : M) : 0
Z_FLOW = (DEBUG||OPut==O_FLOW) ? Flow : 0
Z_NONE = (DEBUG||OPut==O_NONE) ? B : 0
Z_MASK = (DEBUG||OPut==O_MASK) ? EM : 0
Z_SKIP = (DEBUG||OPut==O_SKIP) ? OutSkip : 0
Z_RAW = (DEBUG||OPut==O_RAW) ? OutRaw : 0
Z_OVER = (DEBUG||Oput==O_OVER)
\ ? mt_merge(
\ FlowOver.Overlay(MergeRGB(BlankClip(EM, color_yuv=$000000), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : 0

R=(Debug)
\ ? StackHorizontal(
\ StackVertical(FRSub(Z_AUTO,"Auto"),FRSub(Z_FLOW,"Flow"),FRSub(Z_NONE.ChangeFps(NewNum, NewDen),"None")),
\ StackVertical(FRSub(Z_MASK,"Mask"),FRSub(Z_SKIP,"Skip"),FRSub(Z_OVER,"Over")),
\ StackVertical(FRSub(EM,"EM"),FRSub(SC,"SC"),FRSub(M,"M"))
\ )
\ :(Oput==O_AUTO) ? Z_AUTO [** auto: artifact masking *]
\ : (Oput==O_FLOW) ? Z_FLOW [** flow: interpolation only *]
\ : (Oput==O_NONE) ? Z_NONE [** none: ConvertFPS only *]
\ : (Oput==O_MASK) ? Z_MASK [** mask: mask only *]
\ : (Oput==O_SKIP) ? Z_SKIP [** skip: skip mask *]
\ : (Oput==O_RAW) ? Z_RAW [** raw: raw mask *]
\ : (Oput==O_OVER) ? Z_OVER [** over: mask as cyan overlay *]
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")
return R
}

Avisource("F:\v\XMen2.avi")
ShowFrameNumber
#return last

FrameRateConverter(Debug=True)
#FrameRateConverter(Output="Auto")
#FrameRateConverter(Output="Flow")
#FrameRateConverter(Output="None")
#FrameRateConverter(Output="Mask")
#FrameRateConverter(Output="Skip")
#FrameRateConverter(Output="Raw")
#FrameRateConverter(Output="Over")

EDIT: is this an error

B = FrameDouble ? SelectOdd(B) : B # B is original rate when FrameDouble
...
## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen) # EM is double original rate when FrameDouble
...
## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
<-original rate when framedouble -->, B Orig, EM is Doublerate

Above in blue, different rates ???

MysteryX

28th April 2017, 01:04

Just tried FrameDouble again, and everything is fine. No, I'm not getting any weird flash. What ColorSpace are you using? (YV12 here) What Avisynth version are you using?

Btw I'm trying to split long lines so that it displays nicely in the browser.

Your script is also working fine here, no frame jumps. However, "SC" isn't displaying and "Mask" and "EM" are displaying the same. I believe Mask should be the raw mask. I think it's showing too much to be useful. 4 views would be more than enough.

Or perhaps, on top of "over", to have "debug" which shows 4 views and "debug2" which shows 9 views ... but isn't it starting to be overkill when the script is already ready and working?

One advantage of your "debug" mode is to help others understand what this script is doing under the hood. It took me a while to understand the basic YFRC script.

If you're going to display 9 clips, I'd recommend this order, from top-left to bottom-right: Output, MFlow, Blending, Raw, Mask, Skip --- these 6 are enough. The 3 others are redundant and unnecessary.

If we display 4 clips, Output, Over, Raw, Skip -- or something like that.

But first, what would be the intent? With the script working, it's only about tweaking settings by comparing details. If too many clips are showing up the details of each clip are too small to see and compare.

Edit: OK I see some frames not matching the others. That happens because some clips are 15fps while others are 60fps. If you display a 15fps clip next to a 60fps clip, you're going to see such mismatch.

StainlessS

28th April 2017, 01:17

See edit

EDIT: I'm using AVS standard, YV12. With test script given including ShowFrameNumber.

M clip bottom RHS debug is jumping 5000 -> 10000 -> 5000 etc. (original frame numbers)

Only just updated to latest mvtools and masktools, was getting exact same type thing with older versions
(several versions ago)

EDIT: OverKill, not when your trying to find a bug (and its called debug mode).

EDIT: 20 frames flashing, ~ 200KB:- http://www.mediafire.com/file/cj3kwpaq748bwzm/MX.mp4
With Output=Auto.
Selur has been posting about the flashes in vapoursynth forum.

MysteryX

28th April 2017, 01:52

I see no problem with AVS 2.6 here... nor with FrameDouble.

I did see a bug though: with FrameDouble, scene changes aren't being applied.

StainlessS

28th April 2017, 01:57

FlowX=FrameDouble ? SelectOdd(Flow) : Flow
RT_DebugF("Flow_Rate=%f B_Rate=%f EM_Rate=%f",FlowX.FrameRate,B.FrameRate,EM.FrameRate)
RT_DebugF("BHard_Rate=%f SC_Rate=%f",BHard.FrameRate,SC.FrameRate)

## the FrameRateConverter magic happens
M = mt_merge(FrameDouble ? SelectOdd(Flow) : Flow, B, EM, luma=true, chroma="process")
RT_DebugF("M_1 Rate=%f",M.FrameRate)
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M
RT_DebugF("M_2 Rate=%f",M.FrameRate)

Debug output

[3888] RT_DebugF: Flow_Rate=25.000000 B_Rate=25.000000 EM_Rate=50.000000
[3888] RT_DebugF: BHard_Rate=50.000000 SC_Rate=50.000000
[3888] RT_DebugF: M_1 Rate=25.000000
[3888] RT_DebugF: M_2 Rate=25.000000

M_2 Rate=25.0 but that is probably just because the first clip framerate FlowX was 25.0,
The Mt_merge stuff obviously aint working proper there.

I think that mt_merge pair is where the flashes originate, result of first one mangles things up, but result is 25.0 fps
and probably no longer than first clip (FlowX).

EDIT: Re-did above debug output, more than one problem there

EDIT: As when DoubleRate you use SelectOdd(Flow), so should not all of the other clips be SelectOdd,
however, B has already had SelectOdd applied.
Also, would that mean that when NOT double rate, then no SelectOdd should be applied to any, including B.
I'm not sure I understand what its supposed to do so I'm not sure how to correct it.

MysteryX

28th April 2017, 02:53

I think the whole "SelectOdd" and "FrameDelete" in the processing parts of the clip are unnecessary. It's only at the end that you interlace source and processed clips. Processed clip will only be generated for such requested frames.

I'll make those changes tomorrow.

Is 60fps working for you, or are both modes failing?

StainlessS

28th April 2017, 03:17

I have only been trying it with default doublerate, I'de already reached same conclusion as you about selectodd stuff and already implemented my fix,
seems to be working just fine, except I did not know what to do with this line (so I've left it for you :) )

EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd

here fix, seems to be working just fine, no flashes at all.

# Frame Rate Converter
# Version: 27-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools (default prefilter), GRunT (output="over")
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The horizontal block size (default = Width>1200||Height>900 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ BlkSizeV - The vertical block size (default = BlkSize)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|raw|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; mask=mask only;
## skip=mask used by SkipOver; raw=raw mask; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", int "BlkSizeV",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc",
\ int "SkipOver", clip "Prefilter",Bool "Debug")
{
Function FRSub(clip c,string tit){ # Auto Convert clips to YV12 if not already (NOP if YV12)
c.ConvertToYV12.Subtitle(tit,align=5)
}
Preset = Default(Preset, "normal")
P_SLOW=0 P_NORMAL=1 P_FAST=2
Pset=Preset=="slow"?P_SLOW:Preset=="normal"?P_NORMAL:Preset=="fast"?P_FAST:-1
Assert(Pset!=-1,"FrameRateConverter: 'Preset' must be slow, normal or fast {'"+Preset+"'}")
Output = Default(Output, "auto")
O_AUTO=0 O_FLOW=1 O_NONE=2 O_MASK=3 O_SKIP=4 O_RAW=5 O_OVER=6
OPut=Output=="auto"?O_AUTO:Output=="flow"?O_FLOW:Output=="none"?O_NONE:Output=="mask"?O_MASK:
\ Output=="skip"?O_SKIP:Output=="raw"?O_RAW:Output=="over"?O_OVER:-1
Assert(OPut!=-1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|raw|over) {'"+Output+"'}")

FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1200||C.Height>900 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
BlkSizeV = Default(BlkSizeV, BlkSize)
blkmin = Min(BlkSize,BlkSizeV)
MaskGam = Default(MaskGam, 0.5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = MaskTrh > 0 ? Default(MaskOcc, 150) : 0
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Pset != P_FAST # If !Defined(Prefilter) && P_FAST : Prefilter MUST be Input C
Prefilter = Default(Prefilter, CalcPrefilter ? C.RemoveGrain(22) : C)
Debug = Default(Debug,False)

Assert(BlkSize==8||BlkSize==16||BlkSize==32,String(BlkSize,"FrameRateConverter: BlkSize must be 8, 16 or 32 {%.f}"))
Assert(BlkSizeV==8||BlkSizeV==16||BlkSizeV==32,String(BlkSizeV,"FrameRateConverter: BlkSizeV must be 8, 16 or 32 {%.f}"))
Assert(MaskGam>0.0&&MaskGam<=1.0,String(MaskGam,"FrameRateConverter: MaskGam must be between 0.0 and 1.0 {%.1f}"))
Assert(MaskTrh>=0&& MaskTrh<= 255,String(MaskTrh,"FrameRateConverter: MaskTrh must be between 0 and 255 {%.f}"))
Assert(MaskOcc>=0&&MaskOcc<=255,String(MaskOcc,"FrameRateConverter: MaskOcc must be between 0 and 255 {%.f}"))
Assert(SkipOver>=0&&SkipOver<=255,String(SkipOver,"FrameRateConverter: SkipOver must be between 0 and 255 {%.f}"))

# Performance settings: slow, normal, fast
Recalculate = PSet==P_SLOW || PSET==P_NORMAL
DCT = PSet==P_SLOW ? 1 : 0

## "B" - Blending, "BHard" - No blending
BHard = C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad = 16, vpad = 16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad = 16, vpad = 16, levels = 1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt,isb=true,blksize=BlkSize,blksizeV=BlkSizeV,overlap=blkmin>8?4:blkmin>4?2:0,search=3,dct=DCT)
fwd = MAnalyse(superfilt,isb=false,blksize=BlkSize,blksizeV=BlkSizeV,overlap=blkmin>8?4:blkmin>4?2:0,search=3,dct=DCT)
fwd = Recalculate ? MRecalculate(super,fwd,blksize=BlkSize/2,blksizeV=BlkSizeV/2,overlap=blkmin>8?2:0,thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super,bak,blksize=BlkSize/2,blksizeV=BlkSizeV/2,overlap=blkmin>8?2:0,thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num = NewNum, den = NewDen, blend = false, ml = 200, mask = 2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255).ConvertToY8() :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = EM.ConvertToY8()
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EMfwd = FrameDouble ? EMfwd.DeleteFrame(0) : EMfwd
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=0.5, mode="lighten") : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
OutRaw = EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSizeV/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(100)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
OutSkip = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip,BlankClip(EM, color_yuv=$ff0000),BlankClip(EM,color_yuv=$000000),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM, color_yuv=$000000)

# Display SkipOver value on Output="over
Disp_EMskip = EMskip.ChangeFPS(NewNum, NewDen)
FlowOver = (Oput==O_OVER||DEBUG) ? Flow.GScriptClip("Subtitle(string(Disp_EMskip.AverageLuma))",
\ args="Disp_EMskip",Local=True) : 0

## Convert masks to desired frame rate
EM = EM.ChangeFPS(NewNum, NewDen)
Sc = Sc.ChangeFPS(NewNum, NewDen)

## the FrameRateConverter magic happens
M = mt_merge(Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

Z_AUTO = (DEBUG||OPut==O_AUTO) ? (FrameDouble ? Interleave(C, SelectOdd(M)) : M) : 0
Z_FLOW = (DEBUG||OPut==O_FLOW) ? Flow : 0
Z_NONE = (DEBUG||OPut==O_NONE) ? B : 0
Z_MASK = (DEBUG||OPut==O_MASK) ? EM : 0
Z_SKIP = (DEBUG||OPut==O_SKIP) ? OutSkip : 0
Z_RAW = (DEBUG||OPut==O_RAW) ? OutRaw : 0
Z_OVER = (DEBUG||Oput==O_OVER)
\ ? mt_merge(
\ FlowOver.Overlay(MergeRGB(BlankClip(EM,color_yuv=$000000),EM,EM),mode="Add",opacity=0.40,pc_range=true),
\ BlankClip(Flow, color=color_darkgoldenrod), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : 0

R=(Debug)
\ ? StackHorizontal(
\ StackVertical(FRSub(Z_AUTO,"Auto"),FRSub(Z_FLOW,"Flow"),FRSub(Z_NONE.ChangeFps(NewNum, NewDen),"None")),
\ StackVertical(FRSub(Z_MASK,"Mask"),FRSub(Z_SKIP,"Skip"),FRSub(Z_OVER,"Over")),
\ StackVertical(FRSub(EM,"EM"),FRSub(SC,"SC"),FRSub(M,"M"))
\ )
\ :(Oput==O_AUTO) ? Z_AUTO [** auto: artifact masking *]
\ : (Oput==O_FLOW) ? Z_FLOW [** flow: interpolation only *]
\ : (Oput==O_NONE) ? Z_NONE [** none: ConvertFPS only *]
\ : (Oput==O_MASK) ? Z_MASK [** mask: mask only *]
\ : (Oput==O_SKIP) ? Z_SKIP [** skip: skip mask *]
\ : (Oput==O_RAW) ? Z_RAW [** raw: raw mask *]
\ : (Oput==O_OVER) ? Z_OVER [** over: mask as cyan overlay *]
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")
return R
}

My test

Avisource("F:\v\XMen2.avi")
#ShowFrameNumber # EDIT: Only to find flash frames, no longer needed
#return last

FrameRateConverter(Debug=True)
#FrameRateConverter(Output="Auto")
#FrameRateConverter(Output="Flow")
#FrameRateConverter(Output="None")
#FrameRateConverter(Output="Mask")
#FrameRateConverter(Output="Skip")
#FrameRateConverter(Output="Raw")
#FrameRateConverter(Output="Over")

chainik_svp

28th April 2017, 09:08

ok, I've testes JM's clip and I must say:
1. don't use "algo=13" (default in Interframe w/o "smooth" preset)!
2. higher mask.area values lead to more blurry image (Interframe uses 150 in "smooth" preset, which is too high)

so, Interframe:
- default mode: algo=13, mask.area=0
- smooth preset: algo=23, mask.area=150

BOTH modes are BAD, at least for this extremely low-quality source
this doesn't mean they're ALWAYS bad, I believe SubJunk made a lot of comparisons when he selected these values

set algo=21/23, mask.area <= 50 and you'll be fine ;)

kolak

28th April 2017, 13:45

To get some values which work with specific source is 1 thing, to find good general settings is very different.

StainlessS

28th April 2017, 14:57

MX, your times for fast are probably quite a bit off, bug in fast processing for prefilter.

Mod

CalcPrefilter = Defined(Prefilter) || Pset != P_FAST # If !Defined(Prefilter) && P_FAST : Prefilter MUST be Input C
# Prefilter = Default(Prefilter, C.RemoveGrain(22) )
Prefilter = Default(Prefilter, CalcPrefilter ? C.RemoveGrain(22) : C)
# EDIT: OR alternative: Prefilter = Default(Prefilter, !P_FAST ? C.RemoveGrain(22) : C)

Am doing a few changes (mainly cosmetic).

EDIT: Post #207 updated as per above.
+ Asserts show error value.

EDIT: In addition to above bug, both SuperFilt and Super would both be from C.RemoveGrain(22), ie quality loss,
output always Prefiltered. Above fix fixes that too.
Extreme care should be taken if making alterations anywhere in vicinity of those settings.

MysteryX

28th April 2017, 14:57

Comparison with prefilter and artifact masking disabled.

FrameRateConverter(60, output="auto", prefilter=last, masktrh=0, skipover=0)
https://s1.postimg.org/gk7oin67f/106-frc.png (https://postimg.org/image/gk7oin67f/)

InterFrame(Tuning="Smooth", NewNum=60, NewDen=1, GPU=true, OverrideArea=0, Cores=1)
https://s13.postimg.org/as2thdd77/106-inter.png (https://postimg.org/image/as2thdd77/)

chainik_svp

28th April 2017, 15:53

"OverrideArea" IS zero by default... which means "override" is off

===
BTW, here's another sample video (http://www.svp-team.com/files/temp/telnyashka.avi)

It's almost impossible with MVTools to get rid of this:
http://www.svp-team.com/files/temp/telnyashka-321.png

this thing is not static, it jumps all over the striped vest randomly ;)
(well, there's one solution - set search radius to 1, but it's not great)

MysteryX

28th April 2017, 16:39

Whole frame blurring is one thing. Artifact masking is another.

I'll do some test later with 1080p source that has severe artifacts problems.

This is the script generated by Interframe, which I isolated earlier; but then didn't know how to convert settings to/from MFlow syntax

function InterFrameProcess(clip Input) {
SuperString = "{scale:{up:0,down:4},gpu:1,rc:false}"
VectorsString = "{block:{w:8,overlap:2},main:{search:{distance:0,coarse:{distance:-10,bad:{sad:2000}}}},refine:[{thsad:250}]}"
SmoothString = "{rate:{num:60,den:1,abs:true},algo:23,mask:{area:150,area_sharp:1.2},scene:{blend:true, mode:0}}"

# Make interpolation vector clip
Super = SVSuper(Input, SuperString)
Vectors = SVAnalyse(Super, VectorsString)

# Put it together
smooth_video = SVSmoothFps(Input, Super, Vectors, SmoothString, url="www.svp-team.com", mt=1)
smooth_video
}

chainik_svp

28th April 2017, 17:11

in your message above:
Comparison with prefilter and artifact masking disabled.
artifact masking for Interframe IS NOT disabled
it's still equal to "150"

MysteryX

28th April 2017, 17:59

HD sample: Girl's Day - Female President (https://www.youtube.com/watch?v=v0f9ifrDSp8)

FrameRateConverter(60, output="over", blksize=32)
FrameRateConverter(60, output="over", blksize=16)
FrameRateConverter(60, output="over", blksize=16, preset="slow")
InterFrame(Tuning="Smooth", NewNum=60, NewDen=1, GPU=true, Cores=1)
InterFrame(Tuning="Smooth", NewNum=60, NewDen=1, GPU=true, Cores=1, overridearea=1)

Frame 3497
https://s10.postimg.org/51co8xzet/girl3497-frc32.png (https://postimg.org/image/51co8xzet/) https://s10.postimg.org/bp99vjix1/girl3497-frc16.png (https://postimg.org/image/bp99vjix1/) https://s10.postimg.org/z4r70w2o5/girl3497-frc16-dct.png (https://postimg.org/image/z4r70w2o5/) https://s10.postimg.org/iwayrettx/girl3497-inter.png (https://postimg.org/image/iwayrettx/) https://s28.postimg.org/ju9ng0m89/girl3497-inter-1.png (https://postimg.org/image/ju9ng0m89/)

Frame 8900
https://s10.postimg.org/a53xx5ait/girl8900-frc32.png (https://postimg.org/image/a53xx5ait/) https://s10.postimg.org/bhlmz17yd/girl8900-frc16.png (https://postimg.org/image/bhlmz17yd/) https://s10.postimg.org/x5al9h8cl/girl8900-frc16-dct.png (https://postimg.org/image/x5al9h8cl/) https://s10.postimg.org/je6479jet/girl8900-inter.png (https://postimg.org/image/je6479jet/) https://s28.postimg.org/xpcjc8615/girl8900-inter-1.png (https://postimg.org/image/xpcjc8615/)

Note: DCT=1 is prohibitively slow in 1080p!!!

Note: DCT's darker images is because it produces stronger artifact masks and darker means the frames will be skipped -- but those frames actually look better

Note: For frame 8900, most such frames are skipped by my script, but this bad frame does go through -- as an exception

Performance:

blksize=32

FPS (min | max | average): 3.728 | 137555 | 18.62
Memory usage (phys | virt): 1389 | 1395 MiB
Thread count: 29
CPU usage (average): 70%

blksize=16

FPS (min | max | average): 3.314 | 111354 | 15.65
Memory usage (phys | virt): 1483 | 1496 MiB
Thread count: 29
CPU usage (average): 66%

blksize=16, preset="slow"

FPS (min | max | average): 0.040 | 53146 | 0.123
Memory usage (phys | virt): 1465 | 1466 MiB
Thread count: 26
CPU usage (average): 44%

EDIT: I just had an idea!

If frames are "skipped" at blksize=16, I could fallback to blksize=32, and if it fails again, then skip. It would be a bit more processing but only on bad frames, and frame 8900 would then render correctly.

MysteryX

28th April 2017, 22:45

Here's the problem with DCT mask strength

DCT=0
https://s13.postimg.org/8zg5r1hf7/mask-dct0.png (https://postimg.org/image/8zg5r1hf7/)

DCT=1
https://s13.postimg.org/6jecd6zcj/mask-dct1.png (https://postimg.org/image/6jecd6zcj/)

Now I have to find a way to adjust to DCT=1 mask strength. Average luminosity is almost twice higher!!

Update: I'm finding it hard to do something decent with DCT=1. The raw image looks nicer, but then the mask is also much stronger, and while it removes some artifacts, it creates others. Sometimes it puts lines in subtitles, and to remove it, the mask treshold must be high, which causes a lot more artifact removal to occur; and thus more frame blending and more double shadows.

Combining the facts that...
- DCT=1 is gruelly slow (see benchmark above)
- DCT=1 generates much stronger artifact masks
- DCT=1 creates some ugly artifacts preventing lowering the mask treshold much

...

I'm considering dropping DCT altogether.

Here's what I managed to achieve.

DCT=0 / DCT=1 with MaskTrh-40
https://s7.postimg.org/i0x9qrptj/dct0.png (https://postimg.org/image/i0x9qrptj/) https://s7.postimg.org/yrd8tudmf/dct1.png (https://postimg.org/image/yrd8tudmf/)

If I lower the treshold by 50, then I get artifacts like this in the text
https://s7.postimg.org/n0upypvg7/dct0-50.png (https://postimg.org/image/n0upypvg7/) https://s7.postimg.org/qzwiva9h3/dct1-50.png (https://postimg.org/image/qzwiva9h3/)

kolak

29th April 2017, 00:43

HD sample: Girl's Day - Female President (https://www.youtube.com/watch?v=v0f9ifrDSp8)

FrameRateConverter(60, output="over", blksize=32)
FrameRateConverter(60, output="over", blksize=16)
FrameRateConverter(60, output="over", blksize=16, preset="slow")
InterFrame(Tuning="Smooth", NewNum=60, NewDen=1, GPU=true, Cores=1)
InterFrame(Tuning="Smooth", NewNum=60, NewDen=1, GPU=true, Cores=1, overridearea=1)

Frame 3497
https://s10.postimg.org/51co8xzet/girl3497-frc32.png (https://postimg.org/image/51co8xzet/) https://s10.postimg.org/bp99vjix1/girl3497-frc16.png (https://postimg.org/image/bp99vjix1/) https://s10.postimg.org/z4r70w2o5/girl3497-frc16-dct.png (https://postimg.org/image/z4r70w2o5/) https://s10.postimg.org/iwayrettx/girl3497-inter.png (https://postimg.org/image/iwayrettx/) https://s28.postimg.org/ju9ng0m89/girl3497-inter-1.png (https://postimg.org/image/ju9ng0m89/)

Frame 8900
https://s10.postimg.org/a53xx5ait/girl8900-frc32.png (https://postimg.org/image/a53xx5ait/) https://s10.postimg.org/bhlmz17yd/girl8900-frc16.png (https://postimg.org/image/bhlmz17yd/) https://s10.postimg.org/x5al9h8cl/girl8900-frc16-dct.png (https://postimg.org/image/x5al9h8cl/) https://s10.postimg.org/je6479jet/girl8900-inter.png (https://postimg.org/image/je6479jet/) https://s28.postimg.org/xpcjc8615/girl8900-inter-1.png (https://postimg.org/image/xpcjc8615/)

Note: DCT=1 is prohibitively slow in 1080p!!!

Note: DCT's darker images is because it produces stronger artifact masks and darker means the frames will be skipped -- but those frames actually look better

Note: For frame 8900, most such frames are skipped by my script, but this bad frame does go through -- as an exception

Well, this is exactly my problem with interframe- double edges. Don't have this issue with mvtools.

MysteryX

29th April 2017, 01:04

Mask strength discrepancy isn't only an issue with YV12/YV24, and DCT=0 or 1, but also blksize 8, 16 and 32 produce a different mask strength...

Right now it is tweaked for YV12 with blksize=16. A different format or block size won't give as good artifact masking.

MysteryX

29th April 2017, 04:43

Does anyone *ever* use non-square blocks? Handling 16x8 and 32x16 would make mask strength adjustments considerably more complex.

Any objection to dropping BlkSizeV?

Instead, what I'm working on is fallback mode, so you can process in 16x16, and fallback to either 8x8 or 32x32 before reverting back to "hard blending" if it fails too. It's just that doing such fallback with 16x8 would make it considerably more complex.

For SD videos, 16x16 with fallback to 8x8 works best, while for HD videos, 16x16 with fallback to 32x32 works best, so both options must be available.

MysteryX

29th April 2017, 05:48

OK I made a bunch of changes again.

I removed StainlessS's debug code and added a new debug option that displays AverageLuma details in the corner, in any output mode.

I removed DCT. Preset Slow is still there but it's not currently in use.

I made some adjustments based on the block size. Now it should work fine with any block sizes and any video sizes.

MaskTrh = MaskTrh + (BlkSize == 8 ? 30 : BlkSize == 32 ? -60 : 0)
SkipTrh = SkipTrh + (BlkSize == 8 ? 25 : BlkSize == 32 ? -25 : 0)

I removed BlkSizeV.

I added Fallback so that we try a different block size before skipping. 1 to try larger, -1 to try smaller, 0 to disable. For some videos, I found it to make nearly no difference. For other videos, however, a considerable amount of Fallback frames are coming through. Please test and give your feedback. You can view Fallback frames with Debug=true.

In theory, the fallback mode should only fetch the frames when Skip is triggered, but in practice, it makes the whole script run twice slower, which indicates it is calculating the fallback frame every time. Any way to improve this? Also, Fallback+Debug causes the info to be written twice on top of each other; anyone is welcomed to fix this.

# Frame Rate Converter
# Version: 28-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, rgtools (default prefilter), GRunT (output="over")
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The block size (default = Width>1600||Height>1200 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|raw|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; mask=mask only;
## skip=mask used by SkipOver; raw=raw mask; over=mask as cyan overlay for debugging
##
## @ MaskGam - A gamma to be applied to the raw mask, between 0 and 1. (Default=.5)
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255.
## 0 to disable artifact masking. (Default=120)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=150)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## 0 to disable. (Default=16)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
## @ Debug - Whether to display AverageLuma values of Skip, Mask and Raw. (Default=false)
##
## @ Fallback - When SkipOver is triggered, -1 to try with smaller block size,
## 1 to try with smaller block size, or 0 to disable. (Default=0)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize",
\ bool "FrameDouble", string "Output", float "MaskGam", int "MaskTrh", int "MaskOcc",
\ int "SkipOver", clip "Prefilter", bool "Debug", int "Fallback")
{
Preset = Default(Preset, "normal")
P_SLOW = 0 P_NORMAL = 1 P_FAST = 2
Pset = Preset == "slow" ? P_SLOW : Preset == "normal" ? P_NORMAL : Preset == "fast" ? P_FAST : -1
Assert(Pset != -1, "FrameRateConverter: 'Preset' must be slow, normal or fast {'" + Preset + "'}")
Output = Default(Output, "auto")
O_AUTO = 0 O_FLOW = 1 O_NONE = 2 O_MASK = 3 O_SKIP = 4 O_RAW = 5 O_OVER = 6
OPut = Output == "auto" ? O_AUTO : Output == "flow" ? O_FLOW : Output == "none" ? O_NONE : Output == "mask" ? O_MASK :
\ Output == "skip" ? O_SKIP : Output == "raw" ? O_RAW : Output == "over" ? O_OVER : -1
Assert(OPut != -1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|raw|over) {'" + Output + "'}")

FrameDouble= Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1600||C.Height>1200 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
MaskGam = Default(MaskGam, .5)
MaskTrh = Default(MaskTrh, 120)
MaskOcc = MaskTrh > 0 ? Default(MaskOcc, 150) : 0
SkipTrh = 155
SkipOver = Default(SkipOver, 16)
CalcPrefilter = Defined(Prefilter) || Pset != P_FAST
Prefilter = Default(Prefilter, CalcPrefilter ? C.RemoveGrain(22) : C)
Debug = Default(Debug, false)
#Fallback = Default(Fallback, C.Width>934||C.Height>700 ? (BlkSize<32?1:-1) : (BlkSize>8?-1:1))
Fallback = Default(Fallback, 0)
OutFps = OPut!=O_MASK && OPut!=O_SKIP && OPut!=O_RAW # Whether output will have altered frame rate

Assert(BlkSize==8 || BlkSize==16 || BlkSize==32, String(BlkSize, "FrameRateConverter: BlkSize must be 8, 16 or 32 {%.f}"))
Assert(MaskGam > 0.0 && MaskGam <= 1.0, String(MaskGam, "FrameRateConverter: MaskGam must be between 0.0 and 1.0 {%.1f}"))
Assert(MaskTrh >= 0 && MaskTrh <= 255, String(MaskTrh, "FrameRateConverter: MaskTrh must be between 0 and 255 {%.f}"))
Assert(MaskOcc >= 0 && MaskOcc <= 255, String(MaskOcc, "FrameRateConverter: MaskOcc must be between 0 and 255 {%.f}"))
Assert(SkipOver >= 0 && SkipOver <= 255, String(SkipOver, "FrameRateConverter: SkipOver must be between 0 and 255 {%.f}"))
Assert((Fallback >= -1 && Fallback <= 1) || Fallback == 69, String(SkipOver, "FrameRateConverter: Fallback must be -1, 0 or 1 {%.f}"))
Assert(Fallback != -1 || BlkSize != 8, "FrameRateConverter: Fallback cannot be -1 with BlkSize = 8")
Assert(Fallback != 1 || BlkSize != 32, "FrameRateConverter: Fallback cannot be 1 with BlkSize = 32")

# Performance settings: slow, normal, fast
Recalculate = PSET <= P_NORMAL

## "B" - Blending, "BHard" - No blending
BHard = Fallback == 1 || Fallback == -1 ? C.FrameRateConverter(NewNum, NewDen, Preset,
\ Fallback == -1 ? BlkSize/2 : BlkSize*2, FrameDouble, Output, MaskGam, MaskTrh, MaskOcc, SkipOver, Prefilter, Debug, 69)
\ : C.ChangeFPS(NewNum, NewDen)
B = C.ConvertFPS(NewNum, NewDen)

## Adjust parameters for different block sizes, causing stronger or weaker masks
MaskTrh = MaskTrh + (BlkSize == 8 ? 30 : BlkSize == 32 ? -60 : 0)
SkipTrh = SkipTrh + (BlkSize == 8 ? 25 : BlkSize == 32 ? -25 : 0)

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad=16, vpad=16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad=16, vpad=16, levels=1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, overlap = BlkSize>8?4:BlkSize>4?2:0, search=3, dct=0)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, overlap = BlkSize>8?4:BlkSize>4?2:0, search=3, dct=0)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, overlap = BlkSize>8?2:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, overlap = BlkSize>8?2:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=1.0/MaskGam, ysc=255, thSCD2=255).ConvertToY8() :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=1.0/MaskGam, thSCD2=255).ConvertToY8() : EM
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true) : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=255-MaskOcc, kind=2, gamma=1.0/MaskGam, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
OutRaw = EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSize/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(255-SkipTrh)
EM = EM.mt_binarize(255-MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
OutSkip = EMskip.BicubicResize(C.width, C.Height)
ShowEM = EM

## "Sc" - scene detection / SkipOver
Sc = SkipOver > 0 ? ConditionalFilter(EMskip,BlankClip(EM, color_yuv=$ff0000),BlankClip(EM,color_yuv=$000000),
\ "AverageLuma", ">", string(SkipOver)) : BlankClip(EM, color_yuv=$000000)

## Convert masks to desired frame rate
EM = OutFps ? EM.ChangeFPS(NewNum, NewDen) : EM
Sc = OutFps ? Sc.ChangeFPS(NewNum, NewDen) : Sc

## the FrameRateConverter magic happens
M = mt_merge(Flow, B, EM, luma=true, chroma="process")
M = SkipOver > 0 ? mt_merge(M, BHard, Sc, luma=true, chroma="process") : M

R= (Oput==O_AUTO) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, SelectOdd(M)) : M)
\ : (Oput==O_FLOW) [** flow: interpolation only *]
\ ? Flow
\ : (Oput==O_NONE) [** none: ConvertFPS only *]
\ ? B
\ : (Oput==O_MASK) [** mask: mask only *]
\ ? EM
\ : (Oput==O_SKIP) [** skip: skip mask *]
\ ? OutSkip
\ : (Oput==O_RAW) [** raw: raw mask *]
\ ? OutRaw
\ : (Oput==O_OVER) [** over: mask as cyan overlay *]
\ ? mt_merge(
\ Flow.Overlay(MergeRGB(BlankClip(EM, color_yuv=$000000), EM, EM), mode="Add", opacity=0.40, pc_range=true),
\ BlankClip(Flow, color=$B8860B), Sc.mt_lut("x 2 / "), luma=true, chroma="process")
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")

# Display AverageLuma values of Skip, Mask and Raw
ShowSkip = OutFps ? EMskip.ChangeFPS(NewNum, NewDen) : EMskip
ShowRaw = OutFps ? OutRaw.ChangeFPS(NewNum, NewDen) : OutRaw
R = Debug ? R.GScriptClip("""(Fallback == 69 || Fallback == 0 || Sc.AverageLuma < 255) ?
\ Subtitle("Skip: " + string(ShowSkip.AverageLuma) + "\nMask: " + string(EM.AverageLuma) +
\ "\nRaw: " + string(ShowRaw.AverageLuma) +
\ (Fallback == 69 && Sc.AverageLuma == 0 ? "\nFallback" : "") +
\ ((Fallback == 69 || Fallback == 0) && Sc.AverageLuma == 255 ? "\nSkipped" : ""), lsp=0) : last""",
\ args = "ShowSkip,EM,ShowRaw,Sc,Fallback", Local=true) : R
return R
}

EDIT: The debug display bug is fixed.

Selur

29th April 2017, 07:40

Hmm,.. tried this version on my sample (see: https://forum.doom9.org/showthread.php?t=174541) with 'framerateconverter()' after frame 32 or so, each frame is simply doubled,...

MysteryX

29th April 2017, 09:30

Set SkipOver=0 to interpolate all frames and Debug=true to see what's going on under the hood and find the right SkipOver value. You can also use the new Fallback option to try a different block size before reverting back to doubling frames.

MysteryX

29th April 2017, 10:19

Something I could add: FallbackOver, BlendOver and SkipOver. We can do different actions based on the severity of the frame.

Sharc

29th April 2017, 12:27

This sounds like a good idea to me :)

chainik_svp

29th April 2017, 22:04

BTW, this's almost what SVP does in "adaptive" interpolation mode - calculates scene "quality" and modifies intermediate frames times (switching source frames only in the worst case) accordingly.

Let me quote the docs:

scene - Extended "scene change" controls.

scene.mode - Frames interpolation mode:
0 - uniform interpolation for maximum smoothness. For example for 24->60 conversion output will be: "1mmmm1mmmm...", where "1" stands for original frame and "m" for interpolated one.
1 - "1m" mode that gives "1mm1m1mm1m..." output in the above example => less artifacts at the cost of less smoothness.
2 - "2m" mode: "1m11m11m11..." => much less artifacts and much less smoothness.
3 - adaptive mode that switches between modes 0,1,2 based on overall vector field quality.

scene.blend - Blend frames at scene change like ConvertFps if true, or repeat last frame like ChangeFps if false.

scene.limits - Limits for vector field quality / scene change detection.
For example scene change will be detected if number of blocks with "adjusted SAD" > "limits.scene" will be more than "limits.blocks" percents of all blocks, that has "adjusted SAD" value > "limits.zero", where "adjusted SAD" is "block SAD"/"block average luma".

scene.limits.m1 - Limit for changing uniform mode to "1m".
scene.limits.m2 - Limit for changing "1m" mode to "2m".
scene.limits.scene - Limit for scene change detection.
scene.limits.zero - Vectors with "adjusted SAD" less than this value are excluded from consideration.
scene.limits.blocks - Threshold which sets how many blocks in percents have to change.

MysteryX

29th April 2017, 22:08

BTW, this's almost what SVP does in "adaptive" interpolation mode - calculates scene "quality" and modifies intermediate frames times (using source frames only in the worst case) accordingly.
The question remains: why am I achieving much better results than SVP? The logic is the same, but somewhere along the line, quality gets lost.

chainik_svp

29th April 2017, 22:21

Because you're not using SVP.

Interframe != SVP

> why am I achieving much better results than SVP?

where?
keep in mind that both 13th "SVP shader" (*) and "area masking" are missing from MVTools
w/o these new features you got almost the same still images as with MVTools

(*) "13th shader" is the one from MBlockFps, but applied per-pixel rather than per-block

MysteryX

30th April 2017, 00:25

Because you're not using SVP.

Interframe != SVP
Interframe is a preset to generate the same scripts as SVP. Is there something that Interframe is doing wrong that we could fix?

> why am I achieving much better results than SVP?

where?

https://forum.doom9.org/showthread.php?p=1805304#post1805304
https://forum.doom9.org/showthread.php?p=1805396#post1805396

keep in mind that both 13th "SVP shader" (*) and "area masking" are missing from MVTools
w/o these new features you got almost the same still images as with MVTools

(*) "13th shader" is the one from MBlockFps, but applied per-pixel rather than per-block
We can talk about theory all day long -- and at the end of the day, it's the result that matters. We bring screenshots and compare between various methods to see which is better.

Your comments here haven't been exactly useful. You've only said that "SVP is fine" but we still aren't any closer to getting similar results with SVP (and with GPU acceleration) than we do with FrameRateConverter (slower, unfortunately).

MysteryX

30th April 2017, 02:44

The latest script introduced some problems. I'm working on it.

MysteryX

30th April 2017, 06:23

OK this version will work much better than the latest script.

Fallback wasn't working. The reason is that although some frames may generate better with blksize 8 or 16, the artifact mask doesn't change enough to know for sure which is better. More often than not, if the artifact mask is too strong, interpolation should be discarded and if the fallback frame passes, it risks being a "false good". If a frame is flagged as bad, better revert to Blending or Skip right away.

There is now BlendOver and SkipOver. Frames that aren't good will revert to blending, and frames that are really bad and scene changes will revert to skip. That's working very well.

The adjustments I made for varying block sizes were wrong. I corrected and re-adjusted it so that 8, 16 and 32 work.

So far it's all for YV12. YV24 has slightly stronger mask so I still need to make up for that.

This version should skip much less than before. Use Debug=true to see what it's doing.

# Frame Rate Converter
# Version: 29-Apr-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, GRunT, rgtools (default prefilter)
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The block size: 8, 16 or 32.
## (default = Width>1600||Height>1200 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|inter|none|mask|skip|raw|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; mask=mask only;
## skip=mask used to Skip; raw=raw mask; over=mask as cyan overlay for debugging
##
## @ Debug - Whether to display AverageLuma values of Skip, Mask and Raw. (Default=false)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255. Smaller = stronger.
## 0 to disable artifact masking. (Default=135)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=105)
##
## @ BlendOver - Try fallback block size when artifacts cover more than specified treshold, or 0 to disable.
## If it fails again, it will revert to frame blending. (default=20)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## or 0 to disable. (Default=45)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", bool "FrameDouble",
\ string "Output", bool "Debug", clip "Prefilter", int "MaskTrh", int "MaskOcc", int "BlendOver", int "SkipOver")
{
Preset = Default(Preset, "normal")
P_SLOW = 0 P_NORMAL = 1 P_FAST = 2
Pset = Preset == "slow" ? P_SLOW : Preset == "normal" ? P_NORMAL : Preset == "fast" ? P_FAST : -1
Assert(Pset != -1, "FrameRateConverter: 'Preset' must be slow, normal or fast {'" + Preset + "'}")
Output = Default(Output, "auto")
O_AUTO = 0 O_FLOW = 1 O_NONE = 2 O_MASK = 3 O_SKIP = 4 O_RAW = 5 O_OVER = 6
OPut = Output == "auto" ? O_AUTO : Output == "flow" ? O_FLOW : Output == "none" ? O_NONE : Output == "mask" ? O_MASK :
\ Output == "skip" ? O_SKIP : Output == "raw" ? O_RAW : Output == "over" ? O_OVER : -1
Assert(OPut != -1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|raw|over) {'" + Output + "'}")

FrameDouble = Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>1600||C.Height>1200 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
MaskTrh = Default(MaskTrh, 100)
SkipTrh = 90
MaskOcc = MaskTrh > 0 ? Default(MaskOcc, 105) : 0
BlendOver = Default(BlendOver, 20)
SkipOver = Default(SkipOver, 45)
CalcPrefilter = Defined(Prefilter) || Pset != P_FAST
Prefilter = Default(Prefilter, CalcPrefilter ? C.RemoveGrain(22) : C)
Debug = Default(Debug, false)
OutFps = OPut!=O_MASK && OPut!=O_SKIP && OPut!=O_RAW # Whether output will have altered frame rate
Recalculate = PSET <= P_NORMAL

Assert(BlkSize==8 || BlkSize==16 || BlkSize==32, String(BlkSize, "FrameRateConverter: BlkSize must be 8, 16 or 32 {%.f}"))
Assert(MaskTrh >= 0 && MaskTrh <= 255, String(MaskTrh, "FrameRateConverter: MaskTrh must be between 0 and 255 {%.f}"))
Assert(MaskOcc >= 0 && MaskOcc <= 255, String(MaskOcc, "FrameRateConverter: MaskOcc must be between 0 and 255 {%.f}"))
Assert(BlendOver >= 0 && BlendOver <= 255, String(BlendOver, "FrameRateConverter: BlendOver must be between 0 and 255 {%.f}"))
Assert(SkipOver >= 0 && SkipOver <= 255, String(SkipOver, "FrameRateConverter: SkipOver must be between 0 and 255 {%.f}"))
Assert(BlendOver==0 || SkipOver==0 || SkipOver > BlendOver, "FrameRateConverter: SkipOver must be greater than BlendOver")

## "B" - Blending, "BHard" - No blending
B = C.ConvertFps(NewNum, NewDen)
BHard = C.ChangeFps(NewNum, NewDen)

## Adjust parameters for different block sizes, causing stronger or weaker masks
MaskTrh = MaskTrh + (BlkSize == 16 ? 25 : BlkSize == 32 ? 50 : 0)
SkipTrh = SkipTrh + (BlkSize == 16 ? 30 : BlkSize == 32 ? 70 : 0)

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad=16, vpad=16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad=16, vpad=16, levels=1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=0)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=0)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=2, ysc=255, thSCD2=255).ConvertToY8() :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.MMask(fwd, ml=255, kind=1, gamma=2, thSCD2=255).ConvertToY8() : EM
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true) : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.MMask(bak, ml=MaskOcc, kind=2, gamma=2, ysc=255, thSCD2=255)
\ .ConvertToY8().mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
OutRaw = EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSize/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(SkipTrh)
EM = EM.mt_binarize(MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
OutSkip = EMskip.BicubicResize(C.width, C.Height).ScriptClip("Subtitle(string(AverageLuma()))")

## "M" - Apply artifact removal
EM = OutFps ? EM.ChangeFPS(NewNum, NewDen) : EM
EMskip = OutFps ? EMskip.ChangeFPS(NewNum, NewDen) : EMskip
M = OutFps ? mt_merge(Flow, B, EM, luma=true, chroma="process") : Flow

## Apply BlendOver and SkipOver
M = M.GScriptClip("Skip = EMskip.AverageLuma()
\ (" + string(SkipOver) + " > 0 && Skip >= " + string(SkipOver) + ") ? BHard :
\ (" + string(BlendOver) + " > 0 && Skip >= " + string(BlendOver) + ") ? B : M",
\ args = "EMskip,M,B,BHard", Local=true)

# Output modes
R= (Oput==O_AUTO) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, SelectOdd(M)) : M)
\ : (Oput==O_FLOW) [** flow: interpolation only *]
\ ? Flow
\ : (Oput==O_NONE) [** none: ConvertFPS only *]
\ ? B
\ : (Oput==O_MASK) [** mask: mask only *]
\ ? EM
\ : (Oput==O_SKIP) [** skip: skip mask *]
\ ? OutSkip
\ : (Oput==O_RAW) [** raw: raw mask *]
\ ? OutRaw
\ : (Oput==O_OVER) [** over: mask as cyan overlay *]
\ ? Flow.Overlay(MergeRGB(BlankClip(EM, color_yuv=$000000), EM, EM), mode="Add", opacity=0.40, pc_range=true)
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")

# Debug: display AverageLuma values of Skip, Mask and Raw
ShowRaw = OutFps ? OutRaw.ChangeFPS(NewNum, NewDen) : OutRaw
R = Debug ? R.GScriptClip("""Skip = EMskip.AverageLuma()
\ SkipSoft = BlendOver > 0 && Skip >= BlendOver && (Skip < SkipOver || SkipOver == 0)
\ Subtitle("Skip: " + string(Skip) + "\nMask: " + string(EM.AverageLuma) +
\ "\nRaw: " + string(ShowRaw.AverageLuma) +
\ "\nBlkSize: " + string(BlkSize) +
\ (SkipSoft ? " - Blend" : "") +
\ (SkipOver > 0 && Skip >= SkipOver ? " - Skip" : ""), lsp=0)""",
\ args = "EMskip,EM,ShowRaw,BlkSize,SkipOver,BlendOver", Local=true) : R
return R
}

chainik_svp

30th April 2017, 10:52

MysteryX
Your comments here haven't been exactly useful. You've only said that "SVP is fine" but we still aren't any closer to getting similar results with SVP (and with GPU acceleration) than we do with FrameRateConverter (slower, unfortunately).

I'm sorry my English is so bad you can't understand it :(
I'll try to write as short sentences as I can.

Rule #1. Don't use algo=13. Use algo=21 or 23 instead.
Rule #2. Don't set mask.area to 150. Use 50 at maximum.

> https://forum.doom9.org/showthread.php?p=1805304#post1805304

this's a comparison between Interframe and MVTools, NOT between SVPflow and MVTools
you can get results similar to MVTools by following rules #1 and #2 above

which is clear enough in your 2nd comparison, first scene:
> https://forum.doom9.org/showthread.p...96#post1805396

2nd scene with stripes is a mess, this's the only one where I can agree "yes, SVPflow with default settings has a problem here"
however you prefer to ignore the opposite example - https://forum.doom9.org/showthread.php?p=1805379#post1805379

And if you think that my comment about "adaptive interpolation mode" was not useful - I'm sorry to interrupt you, please continue your re-invention of the wheel...

MysteryX

30th April 2017, 14:35

Chainik, this code is what Interframe(tuning="smooth") does, and I changed area from 150 to 1. It uses algo 23.

Do you have a better script to propose for testing?

function InterFrameProcess(clip Input) {
SuperString = "{scale:{up:0,down:4},gpu:1,rc:false}"
VectorsString = "{block:{w:8,overlap:2},main:{search:{distance:0,coarse:{distance:-10,bad:{sad:2000}}}},refine:[{thsad:250}]}"
SmoothString = "{rate:{num:60,den:1,abs:true},algo:23,mask:{area:1,area_sharp:1.2},scene:{blend:true, mode:0}}"
Super = SVSuper(Input, SuperString)
Vectors = SVAnalyse(Super, VectorsString)
smooth_video = SVSmoothFps(Input, Super, Vectors, SmoothString, url="www.svp-team.com", mt=1)
smooth_video
}

chainik_svp

30th April 2017, 15:39

VectorsString = "{refine:[{thsad:250}]}"
SmoothString = "{rate:{num:60,den:1,abs:true},algo:23}"

MysteryX

30th April 2017, 17:23

VectorsString = "{refine:[{thsad:250}]}"
SmoothString = "{rate:{num:60,den:1,abs:true},algo:23}"

FrameRateConverter / SVP with above settings

https://s9.postimg.org/llf9zsarf/377-frc.png (https://postimg.org/image/llf9zsarf/) https://s9.postimg.org/nre6813ln/377-svp.png (https://postimg.org/image/nre6813ln/)

kolak

30th April 2017, 17:49

I agree with you. Whatever setting you try by average mvtools offers better quality.
For me main issue in svp are double edges (even on fairly easy scenes).
Another thing- with much older svp libraries quality was better.

Groucho2004

30th April 2017, 18:17

Here (https://www.dropbox.com/s/s63qy9spnfohnrw/str.mkv?dl=0) is a test clip that produces artefacts (on the striped patterns) with almost all methods I tried.
Mystery's latest script works very well on that clip, SVP/Interframe (even with chainik_svp's suggested parameters) is the worst by far.

Sharc

30th April 2017, 18:32

@MysteryX
Very nice work.
Just a cosmetic note: The default values in the ##comments and the actual default values should be aligned for SkipOver and BlendOver

chainik_svp

30th April 2017, 22:10

> FrameRateConverter / SVP with above settings

ok, assuming your effective MVTools options are

bak = MAnalyse(superfilt, isb=true, blksize=8, overlap = 2, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=8, overlap = 2, search=3, dct=DCT)
fwd = MRecalculate(super, fwd, blksize=4, overlap = 0, thSAD=100)
bak = MRecalculate(super, bak, blksize=4, overlap = 0, thSAD=100)
MFlowFps(C, super, bak, fwd, num = 60, den = 1, blend = false, ml = 200, mask = 2, thSCD2=255)

here's the closest SVPflow settings:

VectorsString = "{block:{w:8,overlap:2},main:{search:{distance:2,coarse:{satd:false,distance:2}},penalty:{lambda:20.0}},refine:[{thsad:100}]}"
SmoothString= "{gpuid:11,rate:{num:60,den:1,abs:true},algo:23}"

MysteryX

30th April 2017, 22:21

here's the closest SVPflow settings:

VectorsString = "{block:{w:8,overlap:2},main:{search:{distance:2,coarse:{satd:false,distance:2}},penalty:{lambda:20.0}},refine:[{thsad:100}]}"
SmoothString= "{gpuid:11,rate:{num:60,den:1,abs:true},algo:23}"

FrameRateConverter / SVP
https://s8.postimg.org/98pcphimp/377-frc.png (https://postimg.org/image/98pcphimp/) https://s8.postimg.org/7i6bo013l/377-svp.png (https://postimg.org/image/7i6bo013l/)

MysteryX

30th April 2017, 22:29

Made minor changes to the script above. Increased overlap with BlkSize==32, and fixed comments default values.

manolito

1st May 2017, 15:14

Thanks MysteryX, this script has come a long way...

I agree that using DCT=1 is forbiddingly slow, but for some sources it just makes the difference. Right now the Preset="slow" does not do anything at all, so you might just as well remove it completely.

I modified the script so the slow preset does invoke DCT=1 and also forces Output="flow". For some of my sources this gives the best results since artifact removal using error masks does not really work well with DCT=1.

BTW in the comments for the params you need to replace the output value "inter " by "flow".

Cheers
manolito

StainlessS

1st May 2017, 16:58

Small speed increase possible (In several places).

EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=2, ysc=255, thSCD2=255).ConvertToY8() :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)

You ConvertToY8 after MMask, so throwing away processed chroma, how bout convertToY8 beforehand instead, avoid chroma processing.
(unless due to MaskTools not supporting Y8). [EDIT: I presume MMAsk processes chroma. Does it ?. EDIT: Looks like it does]

Every little helps.

bin.n2f

1st May 2017, 19:33

any chance we get FrameRateConverter script translated into vapoursynth lang.

thanx

SpoCk0nd0pe

1st May 2017, 21:34

Thank you very much for your efforts! I read this with a lot of interest.

MysteryX

1st May 2017, 23:58

You ConvertToY8 after MMask, so throwing away processed chroma, how bout convertToY8 beforehand instead, avoid chroma processing.

It definitely processes the chroma because the mask strength is different between Y12, Y16 and Y24 because of the chroma plane.

However, the mask only contains junk in the chroma planes.

If I create the mask only on the Luma plane, there could be slight performance gain, and also resolve the problem of mask strength discrepancy between formats, but I'd have to see what kind of impact it has on quality and mask accuracy.

any chance we get FrameRateConverter script translated into vapoursynth lang.

thanx
Perhaps, if someone with knowledge of Vapoursynth takes an interest. I've never touched it.

StainlessS

2nd May 2017, 01:02

any chance we get FrameRateConverter script translated into vapoursynth lang.

You best ask in VapourSynth forum, where the VS people live, and also a good idea to wait until finalized, aint nobody gonna wanna keep doing a conversion several times a day.

MysteryX

2nd May 2017, 04:58

Why is it that DCT=1 is so crushingly slow? But only on HD sources. On low-resolution sources, you're barely seeing the difference. Any way to improve this?

hydra3333

2nd May 2017, 05:37

Hello, a related link just for interest (not my code, saw it whilst googling for OpenCL ICD loader stuff)
https://github.com/dthpham/butterflow

MysteryX

2nd May 2017, 06:22

Hello, a related link just for interest (not my code, saw it whilst googling for OpenCL ICD loader stuff)
https://github.com/dthpham/butterflow
Can someone try it and post some comparison screenshots? Thanks

ahah! DCT=1 is crushingly slow with BlkSize=32 !!

With BlkSize=8, it is almost as fast. With BlkSize=16, it's useable. With BlkSize=32, forget it. Plus, it's giving me BAD results with BlkSize=32 on HD content. I'll just make it work with BlkSize 8 and 16.

Small speed increase possible (In several places).

EM = MaskTrh > 0 ? C.MMask(bak, ml=255, kind=1, gamma=2, ysc=255, thSCD2=255).ConvertToY8() :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)

You ConvertToY8 after MMask, so throwing away processed chroma, how bout convertToY8 beforehand instead, avoid chroma processing.
(unless due to MaskTools not supporting Y8). [EDIT: I presume MMAsk processes chroma. Does it ?. EDIT: Looks like it does]

Every little helps.
This change doesn't affect the mask. It's working with the vectors data and I don't think it's using C at all except to determine the output format. So I can call ConvertToY8 first but I don't think it will change performance much at all.

johnmeyer

2nd May 2017, 06:29

Why is it that DCT=1 is so crushingly slow? But only on HD sources. On low-resolution sources, you're barely seeing the difference. Any way to improve this?This would be a great improvement to MVTools2, if it is technically possible. I still haven't had time to play with this new script, but I have used DCT=1 on many projects to reduce flicker. For some videos, it is actually better than Deflicker and other dedicated plugins, but it is unbelievably slow.

The fact that it can reduce flicker perhaps provides a hint of why it might be slow: the only way I know to reduce flicker is to take averages across many, many frames. I have never looked at the MVTools2 source code, but if I did, I'd be looking for some interaction between the DCT setting and the number of frames being evaluated for each block. I suspect that the number may be much larger when DCT is set to something other than zero. If so, improving performance may require the same sort of math genius like what was needed to create the FFT from standard Fourier analysis. If you've ever looked into how that is done, it is completely non-intuitive and requires a math aptitude far beyond anything I posses.

MysteryX

2nd May 2017, 06:47

Updated script.
- Re-added DCT=1 for preset="slow", for BlkSize 8 and 16.
- Tweaked tresholds for DCT=1, and made a few other adjustments
- Replaced ScriptClip with ConditionalFilterMT so that it works with Avisynth+ MT. If you don't want to use it, comment those 2 lines and uncomment the lines below it.
- Other minor changes

# Frame Rate Converter
# Version: 01-May-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, GRunT, ConditionalMT, rgtools (default prefilter)
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The block size: 8, 16 or 32.
## (default = Width>2000||Height>1200 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|flow|none|mask|skip|raw|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; mask=mask only;
## skip=mask used to Skip; raw=raw mask; over=mask as cyan overlay for debugging
##
## @ Debug - Whether to display AverageLuma values of Skip, Mask and Raw. (Default=false)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255. Smaller = stronger.
## 0 to disable artifact masking. (Default=100)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=105)
##
## @ BlendOver - Try fallback block size when artifacts cover more than specified treshold, or 0 to disable.
## If it fails again, it will revert to frame blending. (default=30)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## or 0 to disable. (Default=60)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", bool "FrameDouble",
\ string "Output", bool "Debug", clip "Prefilter", int "MaskTrh", int "MaskOcc", int "BlendOver", int "SkipOver")
{
Preset = Default(Preset, "normal")
P_SLOW = 0 P_NORMAL = 1 P_FAST = 2
Pset = Preset == "slow" ? P_SLOW : Preset == "normal" ? P_NORMAL : Preset == "fast" ? P_FAST : -1
Assert(Pset != -1, "FrameRateConverter: 'Preset' must be slow, normal or fast {'" + Preset + "'}")
Output = Default(Output, "auto")
O_AUTO = 0 O_FLOW = 1 O_NONE = 2 O_MASK = 3 O_SKIP = 4 O_RAW = 5 O_OVER = 6
OPut = Output == "auto" ? O_AUTO : Output == "flow" ? O_FLOW : Output == "none" ? O_NONE : Output == "mask" ? O_MASK :
\ Output == "skip" ? O_SKIP : Output == "raw" ? O_RAW : Output == "over" ? O_OVER : -1
Assert(OPut != -1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|raw|over) {'" + Output + "'}")

FrameDouble = Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>2000||C.Height>1200 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
MaskTrh = Default(MaskTrh, 100)
SkipTrh = 80
MaskOcc = MaskTrh > 0 ? Default(MaskOcc, 105) : 0
BlendOver = Default(BlendOver, 30)
SkipOver = Default(SkipOver, 60)
CalcPrefilter = Defined(Prefilter) || Pset != P_FAST
Prefilter = Default(Prefilter, CalcPrefilter ? C.RemoveGrain(22) : C)
Debug = Default(Debug, false)
OutFps = OPut!=O_MASK && OPut!=O_SKIP && OPut!=O_RAW # Whether output will have altered frame rate
Recalculate = PSET <= P_NORMAL
# DCT=1 is crushingly slow with BlkSize=32 and gives bad results
DCT = PSET == P_SLOW && BlkSize < 32 ? 1 : 0

Assert(BlkSize==8 || BlkSize==16 || BlkSize==32, String(BlkSize, "FrameRateConverter: BlkSize must be 8, 16 or 32 {%.f}"))
Assert(MaskTrh >= 0 && MaskTrh <= 255, String(MaskTrh, "FrameRateConverter: MaskTrh must be between 0 and 255 {%.f}"))
Assert(MaskOcc >= 0 && MaskOcc <= 255, String(MaskOcc, "FrameRateConverter: MaskOcc must be between 0 and 255 {%.f}"))
Assert(BlendOver >= 0 && BlendOver <= 255, String(BlendOver, "FrameRateConverter: BlendOver must be between 0 and 255 {%.f}"))
Assert(SkipOver >= 0 && SkipOver <= 255, String(SkipOver, "FrameRateConverter: SkipOver must be between 0 and 255 {%.f}"))
Assert(BlendOver==0 || SkipOver==0 || SkipOver > BlendOver, "FrameRateConverter: SkipOver must be greater than BlendOver")

## "B" - Blending, "BHard" - No blending
B = C.ConvertFps(NewNum, NewDen)
BHard = C.ChangeFps(NewNum, NewDen)

## Adjust parameters for different block sizes, causing stronger or weaker masks
MaskTrh = Min(MaskTrh + (BlkSize == 16 ? 25 + (DCT==1?90:0) : BlkSize == 32 ? 50 : DCT==1?15:0), 255)
SkipTrh = Min(SkipTrh + (BlkSize == 16 ? 30 + (DCT==1?80:0) : BlkSize == 32 ? 70 : DCT==1?10:0), 255)

## jm_fps interpolation
superfilt = MSuper(prefilter, hpad=16, vpad=16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad=16, vpad=16, levels=1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.ConvertToY8().MMask(bak, ml=255, kind=1, gamma=2, ysc=255, thSCD2=255) :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.ConvertToY8().MMask(fwd, ml=255, kind=1, gamma=2, thSCD2=255) : EM
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true) : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.ConvertToY8().MMask(bak, ml=MaskOcc, kind=2, gamma=2, ysc=255, thSCD2=255)
\ .mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
OutRaw = EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSize/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(SkipTrh)
EM = EM.mt_binarize(MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
OutSkip = EMskip.BicubicResize(C.width, C.Height)

## "M" - Apply artifact removal
EM = OutFps ? EM.ChangeFPS(NewNum, NewDen) : EM
EMskip = OutFps ? EMskip.ChangeFPS(NewNum, NewDen) : EMskip
M = OutFps ? mt_merge(Flow, B, EM, luma=true, chroma="process") : Flow

## Apply BlendOver and SkipOver
M2 = ConditionalFilterMT(EMskip, B, BHard, "AverageLuma", "<", string(SkipOver))
M = ConditionalFilterMT(EMskip, M, M2, "AverageLuma", "<", string(BlendOver))
# M = M.GScriptClip("Skip = EMskip.AverageLuma()
# \ (" + string(SkipOver) + " > 0 && Skip >= " + string(SkipOver) + ") ? BHard :
# \ (" + string(BlendOver) + " > 0 && Skip >= " + string(BlendOver) + ") ? B : M",
# \ args = "EMskip,M,B,BHard", Local=true)

# Output modes
R= (Oput==O_AUTO) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, SelectOdd(M)) : M)
\ : (Oput==O_FLOW) [** flow: interpolation only *]
\ ? Flow
\ : (Oput==O_NONE) [** none: ConvertFPS only *]
\ ? B
\ : (Oput==O_MASK) [** mask: mask only *]
\ ? EM
\ : (Oput==O_SKIP) [** skip: skip mask *]
\ ? OutSkip
\ : (Oput==O_RAW) [** raw: raw mask *]
\ ? OutRaw
\ : (Oput==O_OVER) [** over: mask as cyan overlay *]
\ ? Flow.Overlay(MergeRGB(BlankClip(EM, color_yuv=$000000), EM, EM), mode="Add", opacity=0.40, pc_range=true)
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")

# Debug: display AverageLuma values of Skip, Mask and Raw
ShowRaw = OutFps ? OutRaw.ChangeFPS(NewNum, NewDen) : OutRaw
R = Debug ? R.GScriptClip("""Skip = EMskip.AverageLuma()
\ SkipSoft = BlendOver > 0 && Skip >= BlendOver && (Skip < SkipOver || SkipOver == 0)
\ Subtitle("Skip: " + string(Skip) + "\nMask: " + string(EM.AverageLuma) +
\ "\nRaw: " + string(ShowRaw.AverageLuma) +
\ "\nBlkSize: " + string(BlkSize) +
\ (SkipSoft ? " - Blend" : "") +
\ (SkipOver > 0 && Skip >= SkipOver ? " - Skip" : ""), lsp=0)""",
\ args = "EMskip,EM,ShowRaw,BlkSize,SkipOver,BlendOver", Local=true) : R
return R
}

Performance on 1080p content:
FrameRateConverter(60, BlkSize=16)
Prefetch(8)

FPS (min | max | average): 3.234 | 32478 | 11.60
Memory usage (phys | virt): 1518 | 1531 MiB
Thread count: 29
CPU usage (average): 60%

pinterf

2nd May 2017, 07:46

Afaik 8x8 dct is using dctint instead of fft3 library

MysteryX

2nd May 2017, 16:12

Afaik 8x8 dct is using dctint instead of fft3 library
and it's MUCH faster. What's the reason for 16x16 being a bit slower, and 32x32 being 100x slower? Normally, larger block size means faster.

manolito

3rd May 2017, 15:57

Again I threw my anime torture clip at the latest version of this script...

For this specific source DCT=1 with BlkSize=32 does make sense. The test results are here:
https://www.sendspace.com/file/kyj2vg

First of all I believe that for a standard NTSC source frame size of 720 x 480 the default block size should be 16 instead of 8, so I edited this line:
BlkSize = Default(BlkSize, C.Width>2000||C.Height>1200 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
to look like this:
BlkSize = Default(BlkSize, C.Width>2000||C.Height>1200 ? 32 : C.Width>=720||C.Height>=480 ? 16 : 8)

Then I edited the script and replaced this line:
DCT = PSET == P_SLOW && BlkSize < 32 ? 1 : 0
with these two lines:
DCT = PSET == P_SLOW ? 1 : 0
OPut = DCT == 1 && BlkSize == 32 && OPut == O_AUTO ? O_FLOW : OPut

The result looks real good to me, much better than the default settings.

Of course this source is special, for most natural film sources these settings will not work well, but this is what the default settings are for...

Cheers
manolito

MysteryX

3rd May 2017, 16:43

The parade video is 480 in height and looks better with BlkSize==8 because the objects to track are small. This value should be set based on the size of the objects to track.

As for DCT==1, artifact masking isn't a problem, I could tweak the settings to make it work anyway. But did you try DCT==1 with BlkSize==32? It's *extremely* slow!! It makes no sense to use it with BlkSize==32. With BlkSize==8, yes.

manolito

3rd May 2017, 17:07

The parade video is 480 in height and looks better with BlkSize==8 because the objects to track are small. This value should be set based on the size of the objects to track.

Yes, I always had my reservations about setting the block size depending on the frame size of the source. But how could setting the block size depending on the size of the objects to track be automated?

As for DCT==1, artifact masking isn't a problem, I could tweak the settings to make it work anyway. But did you try DCT==1 with BlkSize==32? It's *extremely* slow!! It makes no sense to use it with BlkSize==32. With BlkSize==8, yes.

But did you try DCT==1 with BlkSize==32?

Of course I did. Have you looked at my conversions? This anime source absolutely needs a block size of 32, otherwise the vertical grille in the second half of the clip will be totally warped. And of course using DCT=1 with BlkSize=32 (without artifact masking) will result in a very slow encode, but the result is by far the best I could get...

Cheers
manolito

//EDIT//
In one of the earlier threads about the johnmeyer params for the Mflowfps based fps conversion methods kolak suggested to try other DCT options like DCT=3. Is this a feasible way?

MysteryX

3rd May 2017, 17:39

Normally, the size of the objects to track depends on the video resolution. If default settings don't work, then it can be customized but I think it should be OK in most cases.

Oh, I see what you're talking about... anime.

Anime require different settings because motion interpolation usually fails with it. We can add either preset="anime" or tuning="anime" to specify custom settings for this particular type of video. Here it would make sense to use larger block size because smaller means better interpolation but more artifacts, while larger means more gross interpolation and fewer artifacts. For anime, generating less artifacts is what we want so it makes sense. You can play around with other DCT options and let us know what you find.

manolito

3rd May 2017, 18:46

You can play around with other DCT options and let us know what you find.

Well, I did play with DCT=3 on this particular anime clip. It was almost as fast as DCT=0, but the output quality was nowhere near when using DCT=1.

Cheers
manolito

MysteryX

3rd May 2017, 19:07

ok... so in your case you *do* see benefits to using DCT=1 with BlkSize=32, even without artifact masking -- BUT you are also working with low-resolution clips which makes it bearable.

I can see what I can do to allow this option then, with artifact masking still working.

pinterf

3rd May 2017, 20:09

and it's MUCH faster. What's the reason for 16x16 being a bit slower, and 32x32 being 100x slower? Normally, larger block size means faster.
Yes, larger block size means less blocks, on the other hand fftw complexity is n*log n (by wiki). Perhaps I could profile it where the bottleneck is.

MysteryX

3rd May 2017, 21:02

Yes, larger block size means less blocks, on the other hand fftw complexity is n*log n (by wiki). Perhaps I could profile it where the bottleneck is.
When you're done also run a profiler of 16-bit SMDegrain, it's WAAAYYYY slower than 8-bit.

pinterf

3rd May 2017, 21:07

Smdegrain is a complex script, try eliminating the filters from it.

burfadel

3rd May 2017, 22:47

If you quarter the resolution (half horizontal and vertical), such that 1920x1080 becomes 960x540, if you use the right resize method wouldn't it effectively make a blocksize of 16 a blocksize of 32? Then simply resize the analysis back to the original size. For a blocksize of 8 on a full image you would have to add another recalculate line with blocksize 4 due to the sizing. It's supposedly only used when results from previous lines are 'bad').

MysteryX

3rd May 2017, 23:32

The analysis provides vectors information in a "fake" clip. You can't resize that data as it's not a clip.

StainlessS

4th May 2017, 02:54

MX, have you ever tried this (I have not)

MScaleVect

MScaleVect (
clip vectors,
float scale (2),
float scaleV (scale),
int mode (0),
bool flip (scale < 0 && scale == scaleV)
bool adjustSubPel (false)
)

Rescales motion vectors / blocksize. Main purpose is to allows vectors to be used on a differently sized clip than they were analyzed from.

Example steps:

Use MAnalyze on a half-sized clip at block size 16
Use this plugin to scale the vectors by 2 to block size 32
Use resulting vectors for MFlowFPS, MDegrain,… on the full sized frame.

Saves doing the MAnalyze on the full size frame, which may be faster and saves memory (good for multi-threading). Note that you need a super clip for each frame size. The padding (hpad, vpad) on each super clip must be manually scaled to match the vector scaling (can be easier to set hpad = 0 and vpad = 0 everywhere). See the example below. Similar functionality was available in MVTools through the function MVIncrease, but it was removed.

EDIT: Only just before the Examples in MvTool docs.
EDIT: If D9 search, there are only about 10 posts mentioning "MScaleVect".

burfadel

4th May 2017, 04:53

That's exactly what I meant, it makes sense. As I said, you'd probably have top add another recalculate to cover blksize 8 as it would only be blksize 16 if the motion vectors were scaled.

MysteryX

4th May 2017, 07:18

Good idea.

It could be done with something like this, "half" to process a reduced image. However, "super" needs to be full-size, and is needed by MRecalculate. Or perhaps only MAnalyze would be done in half and MRecalculate would be done in full?

prefilter = half ? prefilter.BicubicResize(C.Width/2, C.Height/2) : prefilter
superfilt = MSuper(prefilter, hpad=16, vpad=16) # all levels for MAnalyse
ch = half ? C.BicubicResize(C.Width/2, C.Height/2) : C
super = CalcPrefilter ? MSuper(ch, hpad=16, vpad=16, levels=1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=DCT)
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : bak
fwd = half ? fwd.MScaleVect() : fwd
bak = half ? bak.MScaleVect() : bak
Flow = MFlowFps(C, super, bak, fwd, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255)

MysteryX

4th May 2017, 07:31

Initial test works, with MAnalyze in half and MRecalculate in full. "half" shows no performance benefit with DCT=0 on HD source. However, if it makes DCT=1 beareable by turning BlkSize=32 into BlkSize=16, then it might have its use. Will need to test more.

MysteryX

4th May 2017, 07:44

Yes, larger block size means less blocks, on the other hand fftw complexity is n*log n (by wiki). Perhaps I could profile it where the bottleneck is.
Here's an interesting one.

BlkSize=4 with DCT=1 also runs EXTREMELY slow!!

I think you'll find something good with the profiler.

BlkSize=8 with DCT=1

FPS (min | max | average): 5.295 | 61538 | 20.63
Memory usage (phys | virt): 75 | 82 MiB
Thread count: 21
CPU usage (average): 11%

BlkSize=8 with "half" and DCT=1, so BlkSize=4

FPS (min | max | average): 0.049 | 178.2 | 0.087
Memory usage (phys | virt): 72 | 77 MiB
Thread count: 18
CPU usage (average): 10%

StainlessS

4th May 2017, 07:53

prefilter = half ? prefilter.BicubicResize(C.Width/2, C.Height/2) : prefilter

Maybe

prefilter = half ? prefilter.BicubicResize(C.Width/4*2, C.Height/4*2) : prefilter

Or

prefilter = half ? prefilter.BicubicResize((C.Width+3)/4*2, (C.Height+3)/4*2) : prefilter

MysteryX

4th May 2017, 07:56

Here's an experimental script with "half" argument.

Play around with it and see whether you find some use for it.

# Frame Rate Converter
# Version: 04-May-2017
# By Etienne Charland
# Based on Oleg Yushko's YFRC artifact masking,
# johnmeyer's frame interpolation code, and
# raffriff42's "weak mask" and output options.
# Special thanks to Pinterf for adding 16-bit support to MvTools2 and MaskTools2
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA, or visit
# http:#www.gnu.org/copyleft/gpl.html.

#######################################################################################
### Frame Rate Converter
### Increases the frame rate with interpolation and strong artifact removal.
##
## YV12/YV24/Y8/YUY2
## Requires: MaskTools2, MvTools2, GRunT, ConditionalMT, rgtools (default prefilter)
##
## @ NewNum - The new framerate numerator (if FrameDouble = false, default = 60)
##
## @ NewDen - The new framerate denominator (if FrameDouble = false, default = 1)
##
## @ Preset - The speed/quality preset [slow|normal|fast]. (default=normal)
##
## @ BlkSize - The block size: 8, 16 or 32.
## (default = Width>2000||Height>1200 ? 32 : Width>720||C.Height>480 ? 16 : 8)
##
## @ FrameDouble - Whether to double the frame rate and preserve original frames (default = true)
##
## @ Output - Output mode [auto|flow|none|mask|skip|raw|over] (default=auto)
## auto=normal artifact masking; flow=interpolation only; none=ConvertFPS only; mask=mask only;
## skip=mask used to Skip; raw=raw mask; over=mask as cyan overlay for debugging
##
## @ Debug - Whether to display AverageLuma values of Skip, Mask and Raw. (Default=false)
##
## @ Prefilter - Specified a custom prefiltered clip. (Default=RemoveGrain(22))
##
## @ MaskTrh - The treshold where a block is considered bad, between 0 and 255. Smaller = stronger.
## 0 to disable artifact masking. (Default=100)
##
## @ MaskOcc - Occlusion mask treshold, between 0 and 255. 0 to disable occlusion masking. (Default=105)
##
## @ BlendOver - Try fallback block size when artifacts cover more than specified treshold, or 0 to disable.
## If it fails again, it will revert to frame blending. (default=30)
##
## @ SkipOver - Skip interpolation of frames when artifacts cover more than specified treshold,
## or 0 to disable. (Default=60)
##
function FrameRateConverter(clip C, int "NewNum", int "NewDen", string "Preset", int "BlkSize", bool "FrameDouble",
\ string "Output", bool "Debug", clip "Prefilter", int "MaskTrh", int "MaskOcc", int "BlendOver", int "SkipOver", bool "half")
{
Preset = Default(Preset, "normal")
P_SLOW = 0 P_NORMAL = 1 P_FAST = 2
Pset = Preset == "slow" ? P_SLOW : Preset == "normal" ? P_NORMAL : Preset == "fast" ? P_FAST : -1
Assert(Pset != -1, "FrameRateConverter: 'Preset' must be slow, normal or fast {'" + Preset + "'}")
Output = Default(Output, "auto")
O_AUTO = 0 O_FLOW = 1 O_NONE = 2 O_MASK = 3 O_SKIP = 4 O_RAW = 5 O_OVER = 6
OPut = Output == "auto" ? O_AUTO : Output == "flow" ? O_FLOW : Output == "none" ? O_NONE : Output == "mask" ? O_MASK :
\ Output == "skip" ? O_SKIP : Output == "raw" ? O_RAW : Output == "over" ? O_OVER : -1
Assert(OPut != -1, "FrameRateConverter: 'Output' not one of (auto|flow|none|mask|skip|raw|over) {'" + Output + "'}")

FrameDouble = Default(FrameDouble, Defined(NewNum) ? false : true)
NewNum = FrameDouble ? C.FrameRateNumerator * 2 : Default(NewNum, 60)
NewDen = FrameDouble ? C.FrameRateDenominator : Default(NewDen, 1)
BlkSize = Default(BlkSize, C.Width>2000||C.Height>1200 ? 32 : C.Width>720||C.Height>480 ? 16 : 8)
MaskTrh = Default(MaskTrh, 100)
SkipTrh = 80
MaskOcc = MaskTrh > 0 ? Default(MaskOcc, 105) : 0
BlendOver = Default(BlendOver, 30)
SkipOver = Default(SkipOver, 60)
CalcPrefilter = Defined(Prefilter) || Pset != P_FAST
Prefilter = Default(Prefilter, CalcPrefilter ? C.RemoveGrain(22) : C)
Debug = Default(Debug, false)
Half = Default(Half, false)
OutFps = OPut!=O_MASK && OPut!=O_SKIP && OPut!=O_RAW # Whether output will have altered frame rate
Recalculate = PSET <= P_NORMAL
# DCT=1 is crushingly slow with BlkSize=32 and gives bad results
DCT = PSET == P_SLOW && BlkSize < 32 ? 1 : 0

Assert(BlkSize==8 || BlkSize==16 || BlkSize==32, String(BlkSize, "FrameRateConverter: BlkSize must be 8, 16 or 32 {%.f}"))
Assert(MaskTrh >= 0 && MaskTrh <= 255, String(MaskTrh, "FrameRateConverter: MaskTrh must be between 0 and 255 {%.f}"))
Assert(MaskOcc >= 0 && MaskOcc <= 255, String(MaskOcc, "FrameRateConverter: MaskOcc must be between 0 and 255 {%.f}"))
Assert(BlendOver >= 0 && BlendOver <= 255, String(BlendOver, "FrameRateConverter: BlendOver must be between 0 and 255 {%.f}"))
Assert(SkipOver >= 0 && SkipOver <= 255, String(SkipOver, "FrameRateConverter: SkipOver must be between 0 and 255 {%.f}"))
Assert(BlendOver==0 || SkipOver==0 || SkipOver > BlendOver, "FrameRateConverter: SkipOver must be greater than BlendOver")

## "B" - Blending, "BHard" - No blending
B = C.ConvertFps(NewNum, NewDen)
BHard = C.ChangeFps(NewNum, NewDen)

## Adjust parameters for different block sizes, causing stronger or weaker masks
MaskTrh = Min(MaskTrh + (BlkSize == 16 ? 25 + (DCT==1?90:0) : BlkSize == 32 ? 50 : DCT==1?15:0), 255)
SkipTrh = Min(SkipTrh + (BlkSize == 16 ? 30 + (DCT==1?80:0) : BlkSize == 32 ? 70 : DCT==1?10:0), 255)

## jm_fps interpolation
prefilter = half ? prefilter.BicubicResize(C.Width/2, C.Height/2) : prefilter
superfilt = MSuper(prefilter, hpad=16, vpad=16) # all levels for MAnalyse
super = CalcPrefilter ? MSuper(C, hpad=16, vpad=16, levels=1) : superfilt # one level is enough for MRecalculate
bak = MAnalyse(superfilt, isb=true, blksize=half?BlkSize/2:BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=DCT)
fwd = MAnalyse(superfilt, isb=false, blksize=half?BlkSize/2:BlkSize, overlap = BlkSize>4?BlkSize/4:0, search=3, dct=DCT)
fwd = half ? fwd.MScaleVect() : fwd
bak = half ? bak.MScaleVect() : bak
fwd = Recalculate ? MRecalculate(super, fwd, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : fwd
bak = Recalculate ? MRecalculate(super, bak, blksize=BlkSize/2, overlap = BlkSize/2>4?BlkSize/4:0, thSAD=100) : bak
Flow = MFlowFps(C, super, bak, fwd, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255)

## "EM" - error or artifact mask
# Mask: SAD
EM = MaskTrh > 0 ? C.ConvertToY8().MMask(bak, ml=255, kind=1, gamma=2, ysc=255, thSCD2=255) :
\ BlankClip(C, pixel_type="Y8", color_yuv=$000000)
# Mask: Temporal blending
EMfwd = MaskTrh > 0 ? C.ConvertToY8().MMask(fwd, ml=255, kind=1, gamma=2, thSCD2=255) : EM
EM = MaskTrh > 0 ? EM.Overlay(EMfwd, opacity=.6, mode="lighten", pc_range=true) : EM
# Mask: Occlusion
EMocc = MaskOcc > 0 ? C.ConvertToY8().MMask(bak, ml=MaskOcc, kind=2, gamma=2, ysc=255, thSCD2=255)
\ .mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM
OutRaw = EM

## Mask processing
EM = EM.BicubicResize(Round(C.Width/BlkSize/4.0)*4, Round(C.Height/BlkSize/4.0)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=1))
EMskip = EM.mt_binarize(SkipTrh)
EM = EM.mt_binarize(MaskTrh)
\ .Blur(.6)
\ .BicubicResize(C.Width, C.Height)
OutSkip = EMskip.BicubicResize(C.width, C.Height)

## "M" - Apply artifact removal
EM = OutFps ? EM.ChangeFPS(NewNum, NewDen) : EM
EMskip = OutFps ? EMskip.ChangeFPS(NewNum, NewDen) : EMskip
M = OutFps ? mt_merge(Flow, B, EM, luma=true, chroma="process") : Flow

## Apply BlendOver and SkipOver
M2 = ConditionalFilterMT(EMskip, B, BHard, "AverageLuma", "<", string(SkipOver))
M = ConditionalFilterMT(EMskip, M, M2, "AverageLuma", "<", string(BlendOver))
# M = M.GScriptClip("Skip = EMskip.AverageLuma()
# \ (" + string(SkipOver) + " > 0 && Skip >= " + string(SkipOver) + ") ? BHard :
# \ (" + string(BlendOver) + " > 0 && Skip >= " + string(BlendOver) + ") ? B : M",
# \ args = "EMskip,M,B,BHard", Local=true)

# Output modes
R= (Oput==O_AUTO) [** auto: artifact masking *]
\ ? (FrameDouble ? Interleave(C, SelectOdd(M)) : M)
\ : (Oput==O_FLOW) [** flow: interpolation only *]
\ ? Flow
\ : (Oput==O_NONE) [** none: ConvertFPS only *]
\ ? B
\ : (Oput==O_MASK) [** mask: mask only *]
\ ? EM
\ : (Oput==O_SKIP) [** skip: skip mask *]
\ ? OutSkip
\ : (Oput==O_RAW) [** raw: raw mask *]
\ ? OutRaw
\ : (Oput==O_OVER) [** over: mask as cyan overlay *]
\ ? Flow.Overlay(MergeRGB(BlankClip(EM, color_yuv=$000000), EM, EM), mode="Add", opacity=0.40, pc_range=true)
\ : Assert(false, "FrameRateConverter: 'Output' INTERNAL ERROR")

# Debug: display AverageLuma values of Skip, Mask and Raw
ShowRaw = OutFps ? OutRaw.ChangeFPS(NewNum, NewDen) : OutRaw
R = Debug ? R.GScriptClip("""Skip = EMskip.AverageLuma()
\ SkipSoft = BlendOver > 0 && Skip >= BlendOver && (Skip < SkipOver || SkipOver == 0)
\ Subtitle("Skip: " + string(Skip) + "\nMask: " + string(EM.AverageLuma) +
\ "\nRaw: " + string(ShowRaw.AverageLuma) +
\ "\nBlkSize: " + string(BlkSize) +
\ (SkipSoft ? " - Blend" : "") +
\ (SkipOver > 0 && Skip >= SkipOver ? " - Skip" : ""), lsp=0)""",
\ args = "EMskip,EM,ShowRaw,BlkSize,SkipOver,BlendOver", Local=true) : R
return R
}

manolito

4th May 2017, 13:23

As you suggested I did play around using this "Half" parameter on my anime torture clip... :D

Here's what I found:
First of all the script does not work with Fizick's last version of MVTools2. The "MScaleVect" command is missing (and I cannot use a newer pinterf build because my CPU does not support SSE2). I needed to install MVExtras by ViT and rename "MScaleVect" to "MScaleVectors".

When I used the "Half" parameter (DCT=1, BlkSize=32) the conversion speed increased from 0.3fps to 0.5fps which is not bad. But unfortunately the conversion quality decreased quite a bit, too. It was almost the same as if I had used a block size of 16 to begin with. Not really useful for me.

I then disabled the "Half" parameter and rechecked if for this anime clip artifact masking damaged the quality. Here are the results:
https://www.sendspace.com/file/asqto0

The version without artifact masking really looks better to me. When the light beam goes over the control panel for the first time some blending does occur. This segment looks better without artifact masking.

So like you already said, it is probably due to the mask being stronger for higher block sizes and for DCT=1.

Cheers
manolito

MysteryX

4th May 2017, 15:08

You get 0.3 fps with BlkSize=32 and DCT=1? It seems your version of the DLL isn't as crushingly slow as Pinterf's version with DCT=1

pinterf

4th May 2017, 16:48

@MysteryX
Meanwhile found something regarding dct slowness.
Plus made the luma:chroma SAD ratio to be always 4:2, colorspace independent (both for YV12 and YV24), but one can choose chroma SAD weight /4, /2 or *2, *4 though a new parameter in MAnalyze and MRecalculate.
PM sent

manolito

4th May 2017, 17:50

You get 0.3 fps with BlkSize=32 and DCT=1?
Yes, but for an SD source (720 x 480) on an ancient Intel Coppermine CPU running at 1.1 GHz.

MysteryX

7th May 2017, 15:10

Hey, I just thought of something. The one thing that always fail are horizontal and vertical stripes.

Is there a way to detect such stripes without too much performance cost? Then we could safely discard such frames.

The best would be to manually scan for stripes to add them reliably to the mask. No need to discard the whole frame unless these stripes cover too much.

Btw, here's what the mask looks like on stripes. Useless here; to detect stripes, we have to start off the source image.
https://s24.postimg.org/h9roif1up/Stripes_Mask.png (https://postimg.org/image/h9roif1up/)

Only way I can think of is a custom specialized filter that takes the source clip, raw mask, blksize and blkoverlay as parameters, scans each block for stripes (Luma plane only), and sets the mask area to 255 when a block contains stripes.

Oh and by the way, Pinterf fixed the mask strength discrepancies of MvTools, I tested it and it works. DCT=1 also works better now. Looking forward to his official release. I'll be able to remove all the code that adjusts mask thresholds based on settings.

manolito

7th May 2017, 17:14

I'll be able to remove all the code that adjusts mask thresholds based on settings.

Please don't forget the folks who need to continue using Fizick's latest version (the pnterf versions need a CPU with SSE2 support, maybe they also do not work under WinXP...)

Cheers
manolito

MysteryX

7th May 2017, 18:07

Please don't forget the folks who need to continue using Fizick's latest version (the pnterf versions need a CPU with SSE2 support, maybe they also do not work under WinXP...)

Cheers
manolito
Then you'll need to adjust thresholds manually; I'll make sure the arguments are exposed as parameters.

MysteryX

7th May 2017, 20:48

If I wanted to detect stripes, for each block, we could calculate the sum of each line and of each column. From the sequence of those sums, it would then be relatively easy to detect stripes.

But most of the work would be assembly programming, which I know nothing about.

MysteryX

9th May 2017, 18:45

Here's a good math problem. You have a sequence of 32 SAD numbers. You want to detect whether there is a repeating sequence.

There are "pattern finder" algorithms for strings, but in this case, it must allow for a certain threshold of tolerance. We could use block size twice larger than for motion estimation, and if more than 50% is "repeating pattern", flag it; or flag the part that's repeating.

Any idea on how to get this done? There must be some specialized algorithm that is designed for such a task.

MysteryX

9th May 2017, 19:58

Scanning for stripes won't be easy but it can be done.

Let's say we use BlkSize=16. Start with 32x16 block at the top left. Get the Luma sum for each column (S[32]). Scan S for 3 consecutive similar values +/- 2%. If found, start scanning for another block of 3 consecutive similar values that are at least 20% apart. If found, scan for the first value again for 3 consecutive lines, and continue scanning until the pattern is broken. We then know where the pattern starts and ends, and can mark those in the mask. Advance by 28 pixels and repeat until we reach the end of first line. Then move 2 or 4 pixels down and repeat. Once we've scanned the whole image, do the same with the sum of each line.

This could detect stripes quite precisely; I just don't know what the performance cost would be. Perhaps it would work better with larger block sizes.

MysteryX

10th May 2017, 01:28

OK this one is simple.

1. Start with a full band of 32 pixels height.
2. Get the sum of each vertical line of that row. (S[width])
3. Run code along those lines, for each band.
4. Move down 24 pixels and repeat with new band.
5. Turn 90° and repeat.

Something like this should work -- code totally untested

edit: it's missing the code to set StripeValid=true. A stripe is valid when StripeValue1 is found for 3 lines, then StripeValue2 is found for 3 lines, and StripeValue1 is found again for 3 lines; then that whole area is marked until the pattern is broken.

width = 100;
int S[width];
int Same = (int)(2.0 / 100 * 255); // how much variation is allowed to consider same value
int Different = (int)(20.0 / 100 * 255); // how much variation is required to consider a different stripe
int Tolerance = 3; // how many transition lines are allowed between stripes
int StripeValue1 = -1, StripeValue2 = -1, StripeStart = -1, StripeIter = 0, FaultCount = 0;
bool StripeValid = false;

for (int i=0; i<width; i++) {
if (StripeStart < 0) {
// Start detecting stripe
if (i < width - 2 && abs(S[i] - S[i+1]) <= Same && abs(S[i] - S[i+2]) <= Same) {
FaultCount = 0;
StripeStart = i;
StripeValue1 = (S[i] + S[i+1] + S[i+2]) / 3;
StripeValue2 = -1;
StripeEnd = -1;
StripeIter = 0;
StripeValid = false;
}
} else {
// Continue stripe processing
if (abs(S[i] - StripeIter == 0 ? StripeValue1 : StripeValue2) > Same) {
if (i < width - 2 && abs(S[i] - S[i+1]) <= Same && abs(S[i] - S[i+2]) <= Same) {
// New stripe detected
int StripeTemp = (S[i] + S[i+1] + S[i+2]) / 3;
if (StripeValue2 < 0) {
// Check if new stripe is different enough.
if (abs(StripeTemp - StripeValue1) >= Different) {
StripeValue2 = StripeTemp;
StripeIter = 1;
} else
FaultCount++;
}
else {
// Check if stripe matches existing pattern
if (abs(StripeTemp - StripeIter == 0 ? StripeValue2 : StripeValue1)
StripeIter = StripeIter == 0 ? 1 : 0;
else
FaultCount++;
}
}
else
FaultCount++;
}

// end of stripe
if (i == width - 1 || FaultCount >= Tolerance) {
int StripeEnd = i - (FaultCount >= Tolerance ? Tolerance : 0);
if (StripeValid == true)
SetMaskValue(StripeStart, StripeEnd);
StripeStart = -1;
}
}
}

johnmeyer

10th May 2017, 04:35

Before you spend too much time on picket fences, you might want to check whether the ME algorithms are only failing on perfectly vertical or horizontal lines. I think you may find that the problem may have more to do with the regularity of the pattern rather than its rotational orientation. As a thought experiment, imagine a picket fence. Then imagine that the camera pans horizontally across that fence. Then -- and this is the key -- suppose the video frame rate and camera pan rate are such that at each subsequent frame, the next picket in the fence now occupies the same pixels as the previous picket in the previous frame. How should the algorithm interpolate between those frames???

Obviously that is a pathological case, but I think it gets at the problem, namely that with a regular pattern one thing looks exactly like the next thing.

I think you might get more mileage by simultaneously doing multiple estimations using different block sizes. In my almost-extensive experience with this technology, the one "tweak" that makes the biggest difference is block size (and the related overlap setting). Larger block sizes usually work better, but not always. If you have your artifact detection working pretty well, then perhaps switching to a different block size for the mask might work better.

MysteryX

10th May 2017, 04:40

That wouldn't really work... a simple pole in the sky would be detected as stripes. Plus, it's only black/white stripes between 2 colors, but you can have a fence, the fence shadow, and the grass behind, alternating. So it's "repeating patterns" that matters.

Whatever we do, the Lighthouse image would be a perfect test case.
https://s13.postimg.org/5jlu9mldv/Lighthouse.png (https://postimg.org/image/5jlu9mldv/)

Vertical lines of repeating patterns, but this one poses several challenges. The patterns can be small, the distance varies, and although they are vertical lines, they're not on the same horizontal axis. A good code would detect the whole fence, and probably the house on the left depending on required threshold.

This really isn't "necessary" as most videos won't even benefit from this -- but when such patterns occur, it fails spectacularly. In that sense, it would really benefit to detect it.

If we work with full bands of pixel-sums, then we get those sums, either horizontally or vertically, and pass it to a function that works with that array of sums. Horizontal or vertical won't make much difference and no need to rotate the image.

Here's another idea.

From the array of sums, we go by blocks of 16. For each block, scan the previous block for at least 2 loosely matching sequences of the start of the block. Going like that, block by block, scanning the previous block for potential matches. Or something like that.

This still leaves the issue that some patterns repeat every 3 or 4 pixels and other HD patterns repeat over 30 pixels.

If anyone have ideas, let me know.

MysteryX

10th May 2017, 04:43

I think you might get more mileage by simultaneously doing multiple estimations using different block sizes. In my almost-extensive experience with this technology, the one "tweak" that makes the biggest difference is block size (and the related overlap setting). Larger block sizes usually work better, but not always. If you have your artifact detection working pretty well, then perhaps switching to a different block size for the mask might work better.
I tried before and it didn't work because the masks couldn't be compared and weren't accurate enough. Now that Pinterf normalized the masks between block sizes, I might be able to do more with this.

For fallback, blending is working fine. However, the mask generation of repeating patterns is very inconsistent and thus fallback methods don't get triggered consistently.

johnmeyer

10th May 2017, 04:58

Yes, you are right: blending IS a better fallback. That's true for picket fences and stripes as well. Ignore my previous comments.

I think you may find that the toughest masking challenge is (are?) legs. When a person walks across the frame, fairly near to the camera, the back-and-forth motion of the person's legs causes the legs to "break" in rather unsettling ways.

While I have been following this thread, and while I still want to experiment with the script, I've been too busy to do that yet. As a result of not reading every last post, I may have missed an answer to the following question: does MVTools2 provide any internals, either via debug or some other mechanism? If so, would any of these additional metrics be useful in determining if any fallback is needed? I just looked again at the MVTools2 user documentation, and there is a debug mode but I'm not at my my main computer where I can look to see what information is provided.

raffriff42

10th May 2017, 10:24

In theory, stripes are easily detected with H- and V- low-pass filters:
https://www.dropbox.com/s/1kfk3cuhpwx3p44/stripe-detect3.png?raw=1
When blurred in one direction only, stripes going at right angles will lose contrast (YPlaneMinMaxDifference (http://avisynth.nl/index.php/Internal_functions/YPlaneMinMaxDifference))
in that direction more than in the other direction.

In the real world, the results are a little ambiguous:
https://www.dropbox.com/s/r9xq63pphu0qd3l/stripe-detect2.png?raw=1

EDIT it's not stripes IMHO, but occlusion -- things passing in front of other things; see the "antlers" shot -- that is the major stumbling block here. Occlusion issues can't be solved by mvtool's block motion detection; they require advanced machine vision (https://en.wikipedia.org/wiki/Machine_vision) techniques -- making sense of an image at a higher level, tracking objects as they move about. Cutting edge technology, very difficult, mostly proprietary or even top secret.

MysteryX

10th May 2017, 14:57

From the current script, occlusions generally get detected enough to either cover them, or discard the whole frame. I haven't seen any major issue there.

The only places where it failed where with some stripe patterns. Some frames get rendered OK, some frames get discarded, some frames get partially masked, and some ugly frames pass through.

As for using MvTools to detect them, as we see from the mask, the output of MvTools is junk and we shouldn't rely on that to detect stripes.

MysteryX

10th May 2017, 15:36

Pinterf, you've gone through the MvTools2 code. Is there anything internal that would be of use to us here?

StainlessS

10th May 2017, 17:25

Dont want to send you on a wild goose chase, but maybe put some stripey stuff through this, and see if anything pops out as obvious.
(probably will not, take a look at all of the bars, use the demo script or variation of):- https://forum.doom9.org/showthread.php?t=167663&highlight=Zebra

pinterf

10th May 2017, 17:39

Pinterf, you've gone through the MvTools2 code. Is there anything internal that would be of use to us here?
I can't really see inside the core of the task and how the mvtools code could be reused. Mvtools is a difficult code, anyway. I'm happy that it is working automagically as-is. :)

MysteryX

10th May 2017, 18:06

So basically it's a code written a long time ago and nobody ever touched since because nobody understands it. You rewrote parts but even you don't understand how the thing works. Good to know.

Yes, it's good that it works automagically.

MysteryX

10th May 2017, 21:24

I played around with it and am getting somewhere.

Here I put the Lighthouse in Y8, scanned bands to calculate the average value, and then set the band value to that average.

BlkSize = 8, 16, 32
https://s4.postimg.org/vnfhu3znt/Mask_Patterns8.png (https://postimg.org/image/vnfhu3znt/) https://s4.postimg.org/ja2nn79zd/Mask_Patterns16.png (https://postimg.org/image/ja2nn79zd/) https://s4.postimg.org/r4398lhs9/Mask_Patterns32.png (https://postimg.org/image/r4398lhs9/)

Stripe patterns are very visible in the average bands. Everything else is muffled into gray tones.

I don't think a pattern between 100 and 120 should be considered an issue. A pattern is visible when it goes up and down at least 5 times -- at least that's what *we* consider a pattern when we look at it. For blksize=32, the pattern gradually fades as the fence takes a lesser portion of the band, so there's a specific threshold where we'll consider it's not going up and down enough.

MysteryX

10th May 2017, 22:43

OH. Here's a VERY simple one.

Sequence on high contrast changes.

With the right parameters and thresholds, just the right areas would pop out automatically.

MysteryX

10th May 2017, 23:05

Here I produced bands with the contrast changes.

Diff with previous
https://s28.postimg.org/kr4cyf095/Mask_Dif1.png (https://postimg.org/image/kr4cyf095/)

Diff with previous and next, max - min of 3 values (I think this one is better)
https://s28.postimg.org/yyu1n2cy1/Mask_Dif2.png (https://postimg.org/image/yyu1n2cy1/)

At this point, I could almost do a binarize and get a decent result for the mask; or apply a gamma curve. The areas highlighted are exactly the ones sensitive to problems. Then the rest of the script triggers artifact removal when the white is too dense in this picture.

The next question is then: what to do with overlap values. Bands would overlap, then we process horizontally and then vertically. If we have the right way of handling overlapping values, it might just work like that.

Edit: for overlaps, we must take the highest value. If vertically has high contrast changes but horizontally none, the horizontal value must be discarded. For the fence, when the block is partially on the fence, the value is lower, and when fully on the fence, it will be higher. It should appear on the mask accordingly with the max value.

Which means I can start with an existing mask, and in each pass, merge the values in "lighten" mode, meaning I'll only brighten the mask as I keep processing band by band.

This is just perfect :)

Horizontal processing (image then doubled for easy view)
blksize=8 overlap=2 // blksize=16 overlap=4 // blksize=32 overlap=8
https://s16.postimg.org/jmb2lnbtd/Mask_H8.png (https://postimg.org/image/jmb2lnbtd/) https://s11.postimg.org/gox49q8kv/Mask_H.png (https://postimg.org/image/gox49q8kv/) https://s16.postimg.org/k0cel8vwx/Mask_H32.png (https://postimg.org/image/k0cel8vwx/)

Repeat the same vertically and we're good

This gets added to the raw mask, with gamma / transparency adjustment, and is then resized and binarized with the rest of the mask as desired.

StainlessS

10th May 2017, 23:24

I was thinking maybe the band at 90 degrees to the stripes, and maybe something like
Median constant and MinMax also constant but hi-value. As I said, look at all of the bands.
I have not tested anything myself at all, just a daft sort of thing. I like to explore all avenues,
even the daft ones. [EDIT: also, just come in from pub, so less than sensible :) ]

EDIT: Sometimes, with an insoluble problem, the only possible solution is a daft one.

kolak

10th May 2017, 23:26

Is DCT=0 default one?
Are you guys not getting issues on fades?

StainlessS

10th May 2017, 23:39

Dct=0, is default for MvTools, DCT=1, is horrendously slow (block size dependent).

MysteryX

11th May 2017, 00:32

Full mask 8, 16, 32
https://s17.postimg.org/x8k5dxmgb/Mask_Full8.png (https://postimg.org/image/x8k5dxmgb/) https://s17.postimg.org/4k77apka3/Mask_Full16.png (https://postimg.org/image/4k77apka3/) https://s17.postimg.org/pidd8sk4r/Mask_Full32.png (https://postimg.org/image/pidd8sk4r/)

Lovely isn't it :)

DCT=1 will work better in Pinterf's upcoming major release. It will remain slow, but usable.

What's the formula to apply a gamma of .5 ? If we apply a gamma, all the weaker contrasts will fade out. Is it simply adding ^2 to the result? .5 becomes .25, .95 becomes .90

johnmeyer

11th May 2017, 01:14

I just spent a couple of hours playing around with the 01-May-2017 version of the script which I think is the latest. I used the animation clip as well as my old 1940s Detroit parade clip. I did a simple frame double to make things simple. I then set up my test script so I could see the Frame Rate Converter result first, followed immediately by the results from a standard MVTools2 MFlowFPS script using the same block size and overlap (8/2) as the Frame Rate Converter (FRC) script and, unfortunately, almost an equal number where the simple script produced a frame that seemed to be better-looking.

So, there were differences, but they all seemed to be a matter of degree. In most cases, when either script broke down, the resulting bad spots looked similar, although each had its own "look."

The only non-default option I tried was the slow preset, in order to test out DCT=1. I didn't find that it made any significant difference in my two clips.

More testing is required. I would love for this to be a significant improvement over the stock MVTools2/MFlowFPS result. I next need to look at the masking code in the script to see if I can figure it out and perhaps make or suggest some modifications.

MysteryX

11th May 2017, 01:45

Masking with blending indeed often doesn't look great. False positives look worse than MVTools2 alone; but when MVTools2 makes really ugly artifacts, it's important to take those out and it does a better job at that.

While you test, also look with output="mask" or "overlay" to see where the differences are, and whether the mask is being applied correctly.

If there are areas where you would manually fix differently, you can tell me exactly how you would do it.

johnmeyer

11th May 2017, 02:49

While you test, also look with output="mask" or "overlay" to see where the differences are, and whether the mask is being applied correctly.Thanks for the tip. I'll try that next.

MysteryX

11th May 2017, 02:53

Now I can do something with this.

StripeMask(blksize=8, gam=2, str=2)

https://s10.postimg.org/wcw4y6zid/Stripe_Mask_Lighthouse.png (https://postimg.org/image/wcw4y6zid/)

I tried on a video that fails drastically
https://s10.postimg.org/60mletkjp/Stripe_Mask_Fem.png (https://postimg.org/image/60mletkjp/)

Blksize 16, 32
https://s10.postimg.org/c2ytj1ydh/Stripe_Mask_Fem16.png (https://postimg.org/image/c2ytj1ydh/) https://s10.postimg.org/4bi3khu85/Stripe_Mask_Fem32.png (https://postimg.org/image/4bi3khu85/)

From there, I call mt_expand and got good data to work with.

While passing through a regular video, stuff does get detected, but it's generally areas that cause problems anyway, so it might just help remove areas that are borderline. It's kind of funny to look at a live video with those sticks though.

MysteryX

11th May 2017, 04:10

This is looking pretty good.

This frame...
https://s22.postimg.org/g6maqk01p/4088orig.png (https://postimg.org/image/g6maqk01p/)

Gives this...
https://s22.postimg.org/ny30p4471/4088flow.png (https://postimg.org/image/ny30p4471/)

With this raw mask...
https://s22.postimg.org/hmxt8p2yl/4088raw.png (https://postimg.org/image/hmxt8p2yl/)

StripeMask enhances the mask like this
https://s22.postimg.org/pgyeu3arh/4088raw2.png (https://postimg.org/image/pgyeu3arh/)

and turns the skip mask from this...
https://s22.postimg.org/scbi0yerh/4088skip.png (https://postimg.org/image/scbi0yerh/)

into this
https://s22.postimg.org/j5xsreywt/4088skip2.png (https://postimg.org/image/j5xsreywt/)

Now it's just a matter of testing and tweaking the settings.

MysteryX

11th May 2017, 05:08

Johnmeyer, for now, focus on testing with blksize=8, as with the next release of MVTools2 (https://github.com/pinterf/mvtools/blob/mvtools-pfmod/README.md), the mask is going to be normalized to blksize=8 (I believe). The version you have "hacks" for other block sizes by increasing the thresholds, but that hack won't be necessary anymore, and it won't produce the same output.

I don't know whether the original MVTools2 also had this issue of mask strength changing with blksize and with dct. Pinterf appears to be saying it was a regression, and perhaps earlier versions are fine.

Now I have a question.

I have StripeMask of the current frame. Is it possible to add StrikeMask of the next frame with 50% opacity without needing to calculate it twice? It would need to go into the cache, but I'm not sure how to go about that.

What happens if I write something like this? Does it calculate each frame once or twice?

EM = C.DeleteFrame(0).StripeMask(C.StripeMask(EM, str=2), str=1)

johnmeyer

11th May 2017, 16:34

Johnmeyer, for now, focus on testing with blksize=8, <snip>

Now I have a question.

I have StripeMask of the current frame. Is it possible to add StrikeMask of the next frame with 50% opacity without needing to calculate it twice? It would need to go into the cache, but I'm not sure how to go about that.

What happens if I write something like this? Does it calculate each frame once or twice?

EM = C.DeleteFrame(0).StripeMask(C.StripeMask(EM, str=2), str=1)
That looks like something Gavino needs to address. I know nothing about how AVISynth handles reentrant code.

MysteryX

11th May 2017, 17:20

btw you might want to wait until I get the new version out before testing as the behaviors will change with the new MVTools2 and with StripeMask. Everything needs to be re-adjusted.

Also for testing, set debug=true to view more metrics

And by the way, this doesn't work

EM = C.DeleteFrame(0).StripeMask(C.StripeMask(EM, str=2), str=1)

I'll use only the previous frame for the mask (which can serve for several interpolated frames) unless someone provides a solution here.

MysteryX

12th May 2017, 06:40

I spent a lot of time on StripeMask and can't get perfect results.

This simple algorithm works 80% of the times, but it tends to take too much geometric forms (especially of high contrasts) and skips low-contrast patterns.

MysteryX

12th May 2017, 20:38

Victory, sweet victory!!

I knew I could do this.

From the mask of contrast lines, I looked for repeating patterns OR 2 full blocks of continual contrasts lines.

I also process the next frame with 50% transparency.

Sweet lighthouse
https://s10.postimg.org/5knv9yx2t/Lighthouse.png (https://postimg.org/image/5knv9yx2t/)

Contrasts are high and small so lines form continuous zones
https://s10.postimg.org/6bglfqzg5/Lighthouse_Lines.png (https://postimg.org/image/6bglfqzg5/)

After detecting patterns and continuous zones
https://s10.postimg.org/y038n9mgl/Lighthouse_Patterns.png (https://postimg.org/image/y038n9mgl/)

This image was giving me trouble because contrasts are too low, requiring to set a low threshold (causing the lighthouse to go so white)
https://s10.postimg.org/7leeust85/4015.png (https://postimg.org/image/7leeust85/)

Lines get partially detected on that frame
https://s10.postimg.org/n85o86705/4015lines.png (https://postimg.org/image/n85o86705/)

That's enough to detect a lot of patterns
https://s10.postimg.org/n9fm1l8tx/4015patterns.png (https://postimg.org/image/n9fm1l8tx/)

This frame was giving me trouble because it shows just as much lines as previous frame
https://s10.postimg.org/9ywclt7n9/1405.png (https://postimg.org/image/9ywclt7n9/)

Lots of geometric forms being detected
https://s10.postimg.org/9ani2v8xh/1405lines.png (https://postimg.org/image/9ani2v8xh/)

But very few patterns
https://s10.postimg.org/koa1e2jg5/1405patterns.png (https://postimg.org/image/koa1e2jg5/)

That will do the job :D

cork_OS

13th May 2017, 00:41

It looks very promising, congratulations!

johnmeyer

13th May 2017, 01:29

So is your plan to replace your current masking approach with this mask, and then, within the mask, switch to something besides motion estimation?

MysteryX

13th May 2017, 03:37

So is your plan to replace your current masking approach with this mask, and then, within the mask, switch to something besides motion estimation?
No, this is mostly useful for the Skip threshold to know which frames to blend and which ones to skip. Otherwise, bad frames are passing through.

MysteryX

13th May 2017, 21:03

Now that mask levels are normalized, I played around with masks of different settings.

DCT=1 generally gives better quality but often adds artifacts, while removing others. Let's just say artifacts are different. However, if I take DCT=1 mask and substract DCT=0 mask, MT_Binarize, MT_Expand, Blur and then add DCT=0 image to the areas where DCT=1 has more artifacts, then it consistently gives better results. The areas that are only in the DCT=1 mask always look better on the DCT=0 image. That's consistent from my tests.

As for using masks of different block size, the problem is that the mask still tends to be slightly stronger with higher block sizes. Thus, it makes it very hard to compare. If the fallback is a larger block size, the logic explained above will almost never trigger, and if the fallback is a smaller block size, the fallback triggers way too often.

raffriff42

14th May 2017, 08:32

By the way, the name of the lighthouse (I have learned) is Pemaquid Point Light (https://en.wikipedia.org/wiki/Pemaquid_Point_Light), located in Maine. In case you want a nicer image, Wikipedia has got some:
https://upload.wikimedia.org/wikipedia/commons/thumb/d/da/Preston-pemaquid-lighthouse.jpg/180px-Preston-pemaquid-lighthouse.jpg (https://en.wikipedia.org/wiki/File:Preston-pemaquid-lighthouse.jpg) https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Pemaquid-lighthouse.jpg/320px-Pemaquid-lighthouse.jpg (https://en.wikipedia.org/wiki/File:Pemaquid-lighthouse.jpg)

There are lots more online, but they are not open-source like Wikipedia's.

cork_OS

14th May 2017, 10:25

MysteryX

14th May 2017, 14:45

and now you'll want me to test my script on THAT video!??

burfadel

15th May 2017, 10:34

Now that there are a whole lot more permissable block sizes, are any of these useful for this script? The 64x64 would be beneficial for >HD material, I can imagine 48x48, 32x32 etc being good for lower resolutions (such as HD as 32!).

The other point of interest are block sizes like 32x16. I was thinking that maybe these odd block sizes could be beneficial. Most video movement is horizontal in nature, with much less vertical movement. If the normal system is 16x16 then 8x8, woudn't 24x12, 16x8, and maybe another recalculate step (just recalculates detected bad vectors from the first two) with 12x6? Obviously if autosetting if you are encoding 3840x2160 you might want it to autoselect 64x32 for example?

MysteryX

15th May 2017, 16:30

Please play around with block sizes and see if there are cases where you get better results with non-square blocks.

Also test which block size works best for various video sizes.

This will help me tweak the script. Meanwhile, I got other things to test.

kolak

16th May 2017, 10:25

MysteryX

16th May 2017, 15:12

If you want to test various block sizes, disable artifact masking and look at the raw interpolation only. Set Output="flow".

There is still a mask strength discrepancy between block sizes, and if we do non-square sizes, it will make it more complex to sort this one out.

Perhaps it would help to understand exactly the cause of this difference, and what kind of mathematical relationship there is.

MysteryX

17th May 2017, 00:34

Alpha Release: 2017-05-16 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.1-alpha)

I think I did a good job at normalizing the masks per block size.

StripeMask also is working. I've done a lot of changes, so here's a version you guys can play with. You'll need both the DLL and the AVSI. Pinterf's latest MvTools2 is highly recommended. If there are mask discrepancy problems, manually adjust MaskTrh and SkipTrh.

https://github.com/mysteryx93/FrameRateConverter/releases

I think everything should be good as it is. The one setting that will require testing is MaskTrh. A lower value will result in larger artifact masks, and a higher value will keep more interpolated images. I've put a default of 140 but haven't tested it much. Try with other values and report what value works best for your content.

To test, what I generally do is open various VirtualDub windows. I set the script to MaskTrh=120, open in VirtualDub, then MaskTrh=130, open in VirtualDub, then MaskTrh=140, open in VirtualDub, etc. to have 5 or 6 versions to compare. I place all instances at the same frame, then switch between them to see which one looks better on difficult areas.

If you set BlkSizeV to a different value than BlkSize, for now I'm only using BlkSize for normalization so you may have to adjust MaskTrh and SkipTrh manually for the difference. Larger block sizes result in a slightly stronger mask and require higher thresholds. By default, BlkSizeV>BlkSize will result in stronger artifact detection and BlkSize<BlkSizeV will result in weaker artifact detection, but the difference shouldn't be much.

This code can work in MT mode when debug=false.

StripeMask currently only supports 8-bit and isn't optimized, but it works.

manolito

17th May 2017, 03:12

Wanted to test this new Alpha, but I can't... :devil:

The framerateconverter.dll crashes, probably because it requires a CPU with SSE2 support, which I do not have.

Commenting out the ConditionalReaderMT calls and the call for stripemask fixes it, but where's the fun without the stripe mask? I tried to compile a stripemask.dll myself (using the DigitalMars compiler), but this was unsuccessful.

Whatever, unless you can compile the DLL without the need for SSE2 I will be outta here...

Cheers
manolito

MysteryX

17th May 2017, 04:27

StripeMask has no ASM code. Did ConditionalMT work for you before? It's the same, and can be replaced easily.

Sharc

17th May 2017, 08:08

I think I did a good job .....
You really did! Thanks for all your efforts and sharing your results :)

I will upload some test results for various MaskTrh later today ....

Sharc

17th May 2017, 10:44

Here some first results for various MaskTrh settings, for comparison. All other parameters = default.
(The original source has been provided by Selur)

http://www.mediafire.com/file/3d7x9y562jfbfru/FRC.mkv

manolito

17th May 2017, 20:42

StripeMask has no ASM code. Did ConditionalMT work for you before? It's the same, and can be replaced easily.

I redid the tests several times, but it is clearly the "FrameRateConverter.dll" which is not working on my system...

The ConditionalFilterMT never worked on my computer, and I never expected it to work since my CPU is single threaded. Thankfully the AVSI contains a workaround (comment out 2 lines, uncomment the following lines) which I have always used successfully.

Alright, here is the detailed report:

I used Groucho's STR test clip (the Japanese girl with the striped stockings). This is the AVS script:
video = DSS2("F:\Download\str.mpg", fps=25.000, preroll=15)
audio = DirectShowSource("F:\Download\str.mpg", video=false)
AudioDub(video, audio)
Crop(0,0,-Width % 8,-Height % 8)
ConvertToYV12()
framerateconverter(NewNum = 50, NewDen = 1)

With the untouched "FrameRateConverter.avsi" I get this error:
Avisynth open failure:
Evaluate: System exception - Illegal instruction
The error occurs at line #160 of the AVSI, this is the first call to "ConditionalFilterMT".

After commenting out "ConditionalFilterMT" I get this error message:
CAVIStreamSynth:
System exception - Illegal instruction at 0x4b860c4

Then I commented out lines 139 and 140 which disables the stripe mask, and after this the conversion went without problems.

(The funny thing is that this conversion gave good results, and it was quite a bit faster than the previous version which did not have the stripe mask yet).

This is my system setup:
Intel Coppermine CPU, single threaded, MMX and SSE, no SSE2 and above
Win XP SP3
AviSynth 2.60
MVTools 2.5.11.22 (latest Fizick release)
MaskTools 2.0a48
RemoveGrain 1.0b by Kassandro (non-SSE2 version)

All these plugin versions are the latest which work on my machine, and I use them for a couple of other plugins where they have always worked flawlessly.

So I am quite sure that your FrameRateConverter.dll is to blame. Probably the newer compilers always optimize for SSE2 unless you tell them not to do this...

Cheers
manolito

MysteryX

17th May 2017, 21:44

do you have Visual C++ Runtime 2017 installed?

It has no ASM code and is compiled with WinXP support.

MysteryX

17th May 2017, 21:54

I have added Preset="slower" and I think you'll be pleased with the results

https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi

Frame 644: normal / slow / slower, output="flow" (no artifact masking)
https://s16.postimg.org/q3iy81tmp/644-normal.png (https://postimg.org/image/q3iy81tmp/) https://s16.postimg.org/syw1ewxmp/644-slow.png (https://postimg.org/image/syw1ewxmp/) https://s16.postimg.org/5lxzwehj5/644-slower.png (https://postimg.org/image/5lxzwehj5/)

Normal: DCT=0
Slow: DCT=1
Slower: Both DCT=1 and DCT=0, and take from DCT=0 the areas where the mask is better

MysteryX

18th May 2017, 00:50

I updated the code again. Replaced preset="slower" with DiffBlkSize and DiffBlkSizeV
https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi

To achieve the same result as
blksize=8, preset="slower"
use this
blksize=8, preset="slow", diffblksize=8

Now you can try with all kinds of other block sizes, such as "blksize=8, diffblksize=12" or "blksize=16, diffblksize=24"

Now that mask strengths are normalized I'm able to compare various masks, but I'm not getting as consistent results with different block sizes as with comparing DCT=1 and DCT=0.

Yet, it gives a lot more options and possibilities to play with. You can try "blksize=8, dct=1" and "blksize=12" as fallback, or you can try "blksize=12, dct=1" and "blksize=8" as fallback.

Or maybe I'll go back to preset="slower" to simplify. Play with it and see what works for you.

Also added output="diff" to see the areas affected by the diff.

Now john you can spend a few hours testing the whole thing

manolito

18th May 2017, 01:10

do you have Visual C++ Runtime 2017 installed?

It has no ASM code and is compiled with WinXP support.

No, the latest VC++ Runtime I had installed was 2015. But after your post I did upgrade to the latest 2017 version, without success though. The error messages stayed exactly the same.

MysteryX

18th May 2017, 18:29

Manolito, I have no idea then. Perhaps someone else has an idea.

You can get the latest script here (https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi)

- Slightly increased MaskTrh from 140 to 150

- Re-added Preset="slower", results are truly impressive

- You can still perform a diff mask with other block sizes with DiffBlkSize and DiffBlkSizeV. With preset="slower", it default to same block size.

noisyfart

18th May 2017, 19:10

Manolito, I have no idea then. Perhaps someone else has an idea.A quick look at the documentation (https://msdn.microsoft.com/en-us/library/7t5yh4fd.aspx) might help. If "-arch" is not specified, it defaults to SSE2.

MysteryX

18th May 2017, 19:58

A quick look at the documentation (https://msdn.microsoft.com/en-us/library/7t5yh4fd.aspx) might help. If "-arch" is not specified, it defaults to SSE2.

ok, try this version (https://mega.nz/#!jdoGBB5D!n9_tWPd5YhfoyHrueszaPq89pcXyawSlnwhezaPBBjc)

manolito

18th May 2017, 20:25

Thanks so much, also to noisyfart... :thanks:

This version works perfectly, even with the VC++ Runtime 2015. Both ConditionalFilterMT and the stripe mask work now.

But I got another problem with my older MaskTools 2.0a48. The last line in the CalcDiff section:
EM = CalcDiff ? mt_merge(EM, EM2, EMdiff, luma=true, chroma="process") : EM

throws this error message:
[avisynth @ 0335ea20] mt_merge : "luma" is unsupported in 422 and 444

My source file is 420, but it looks like one of the mt_merge input masks have become 422 or 444. I got it working by removing the "luma=true" param, but the result might not be what you intended...

Cheers
manolito

MysteryX

18th May 2017, 20:27

It may not recognize the mask in Y8 format and expects a YV12 mask. Try to convert the mask to YV12. Do that for other MT_Merge(s) as well.

Sharc

18th May 2017, 22:15

- You can still perform a diff mask with other block sizes with DiffBlkSize and DiffBlkSizeV. With preset="slower", it default to same block size.
Does preset="slower" overrule any DiffBlkSize settings? Means when I want to specify DiffBlkSize and DiffBlkSizeV I must not use any of the presets?

MysteryX

18th May 2017, 22:20

Look at the script how parameters are handled. But basically, it calculates 2nd version if Preset=slower or DiffBlkSize is specified.

Then if we calculate 2nd version, if DiffBlkSize isn't specified, it defaults to BlkSize.

Sharc

18th May 2017, 22:57

Ah it's clear now. Thanks.

nhope

19th May 2017, 14:46

I did some testing of MysteryX's 18th May FrameRateConverter vs jm_fps.avsi in manolito's post (https://forum.doom9.org/showthread.php?p=1800439#post1800439), converting the 50p 1080p file linked to here (https://www.vegascreativesoftware.info/us/forum/50-to-60p-for-3d--106617/#ca660294) to 59.94p. So this sort of line:

FrameRateConverter(NewNum=60000, NewDen=1001, Preset="medium", MaskTrh=140)

With Preset="medium" and MaskTrh=120, 140 or 160, the render was very fast and the picture was identical to jm_fps.avsi except for some interpolation artefacts along the left hand edge.

With Preset="slower" and MaskTrh=150 it was extremely slow and hung after 98 frames. That may have been down to my SetMemoryMax and MT settings which I left at the same as I had optimised for previous MFlowFps scripts (SetMemoryMax(3072), Prefetch(12)).

With Preset="slow" and MaskTrh=150 it was still very slow and still getting artefacts down the left hand edge. It also made a few changes to other parts of the image (compared to Preset="normal" or jm_fps.avsi) but unfortunately mostly for the worse.

MysteryX

19th May 2017, 15:29

Make sure to preview with output="mask", or with output="over" debug=true to see where it's detecting artifacts and whether the masks are accurate.

Also, this version of StripeMask doesn't currently work in MT; there was a bug I couldn't figure out and the temporary work-around isn't MT-compatible.

12 threads with DCT=1? That can definitely bring a memory issue.

To be clear, under normal preset, it's designed to give the exact same output as jm_fps except in areas of bad artifacts where it falls back to frame blending, and on bad frames that it either skips or blends.

manolito

19th May 2017, 20:14

It may not recognize the mask in Y8 format and expects a YV12 mask. Try to convert the mask to YV12. Do that for other MT_Merge(s) as well.

Yes, that's the problem. I tried all the newer MaskTools versions by tp and pinterf, but none of them runs on my machine...

I modified the script like this:
EM = CalcDiff ? EM.ConvertToYV12() : EM
EM2 = CalcDiff ? EM2.ConvertToYV12() : EM2
Flow = CalcDiff ? mt_merge(Flow, Flow2, EMdiff, luma=true, chroma="process") : Flow
EM = CalcDiff ? mt_merge(EM, EM2, EMdiff, luma=true, chroma="process").ConvertToY8() : EM

Does this look correct to you?
Flow and Flow2 are already YV12 so I think I do not need to convert them. And for EMdiff I was not sure if it was YV12 or Y8. I left it alone and it seems to work.

Cheers
manolito

MysteryX

19th May 2017, 21:13

I believe you only need to call ConvertToY12() on the 3rd argument of MT_Merge -- unless your version doesn't recognize Y8 *at all*, in which case anything passed to it must be in YV12.

Or simply remove any mention to Y8 conversion, as the masks returned by default are in YV12 format -- except StripeMask which returns in Y8.

manolito

19th May 2017, 23:19

No, the third argument for mt_merge is EMdiff, and it does not matter if I convert EMdiff to YV12 or not. I do not really understand how EMdiff is created, but I suppose that it already is YV12. (Is there a way in AviSynth to inspect variable properties at a certain point in the script?)

//EDIT//
but I suppose that it already is YV12
No, it isn't...
Flow and Flow2 are YV12, EM, EM2 and EMdiff are all Y8.
The old mt_merge function seems to have no problem with EMdiff being Y8, but it sure wants the first 2 arguments to be YV12.
//END EDIT//

The culprit for the luma error are EM and EM2, they need to be YV12.

During my tests I found some other interesting things (I used johnmeyer's Parade clip):

1. Commenting out the ConditionalFilterMT lines and using the GScript based lines instead gave very different results. The moose scene had the interpolated frames with ConditionalFilterMT, using the GrunT alternative the moose scene was blended (which looks better).

2. For this clip using the stripe mask introduced some very juddery movement. Commenting out the two lines where the stripe mask is invoked gave a much much better result.

For my tests I used Preset = "slower", all other settings were at their defaults.

Cheers
manolito

MysteryX

20th May 2017, 00:38

The culprit for the luma error are EM and EM2, they need to be YV12.
Duh! "luma" makes no sense for a Y8 clip; the error is correct. Luma is to apply the Luma part of the mask to both luma and chroma planes. In this case it makes no sense. You can safely remove "luma".

1. Commenting out the ConditionalFilterMT lines and using the GScript based lines instead gave very different results. The moose scene had the interpolated frames with ConditionalFilterMT, using the GrunT alternative the moose scene was blended (which looks better).
Moose scene should be blended or skipped. When you put Debug=true, what is the Skip value of those frames?

2. For this clip using the stripe mask introduced some very juddery movement. Commenting out the two lines where the stripe mask is invoked gave a much much better result.
Where?

raffriff42

20th May 2017, 01:34

manolito, testing on my end with older plugins (but running AVS+), these changes allowed the script to work (Version 16-May-2017)
* source clip must be YV12;
* in script, replace all ConvertToY8 with ConvertToYV12;
* in script, replace both ConditionalFilterMT's with ConditionalFilter;
* in script, delete all "overlap=..." (accepting default overlap values)

manolito

20th May 2017, 04:36

Alright, some tests are here:
https://www.sendspace.com/file/9nmqry

I do not really know what to make of them...

Test setup:
Source is johnmeyer's Parade clip. Latest FrameRateConverter script. Used Preset = "slower" and block size 16 (IMO looks better for 720x480 sources than a block size of 8). Otherwise all default params.

I created three test conversions:

1. Using the stripe mask and the ConditionalFilterMT function. Sorry but for this conversion I forgot to turn on debugging. But for the moose scene it is quite obvious that there is no blending or skipping, the interpolated frames are used. Looks bad...

2. Using the stripe mask, but disabled ConditionalFilterMT and used the GRunT based alternative instead. The moose scene is now mainly blended (as it should), but there are ugly motion artifacts. This one looks really bad...

3. Disabled the stripe mask and the ConditionalFilterMT function (removed the FrameRateConverter.dll file from my AviSynth\plugins folder). IMO this conversion beats the other ones by a huge margin.

Cheers
manolito

manolito

20th May 2017, 04:46

@ raffriff42

Thanks for the tips, will test...

But I believe that most of my old plugins do not need this treatment (using plain vanilla AviSynth 2.60).

My version of MaskTools (mt_masktools-26.dll) supports the new AVS 2.60 color spaces quite well, Y8 is no problem.

The MysteryX script already contains an alternative method for ConditionalFilterMT (based on GRunT) which seems to work better.

And Fizick's latest version of MVTools2 has no problem using custom values for Overlap.

Cheers
manolito

MysteryX

20th May 2017, 06:49

The parade clip is the one I used for testing and tweaking. If it's not using blending on the moose scene, then your script isn't running right.

I also set it to 8 for 480 specifically because I'm getting better results on that specific clip with 8 than with 16. Generally, lower gives better results but also more artifacts. So for 1080p, we have the choice between 16, 24 and 32. 16 is generally the best choice -- except for anime.

manolito

20th May 2017, 19:54

The parade clip is the one I used for testing and tweaking. If it's not using blending on the moose scene, then your script isn't running right.

That's what I thought, but I could not find anything on my side... :scared:

Here is a new set of tests, this time using Preset = "normal". I can confirm that the CalcDiff routine has nothing to do with these issues.
https://www.sendspace.com/file/41ntzd

What I did:
Downloaded the latest version of the script from 18-May-2017.
Modified the script in 2 places. I removed "luma=true" from the last mt_merge line, and I changed the default block size for 720x480 from 8 to 16 (still like it better this way).

I had to use older versions of MVTools2 and MaskTools2, plus I needed to use the special Non-SSE2 version of FrameRateConverter.dll.

My observations:
1. When using ConditionalFilterMT (lines 207 and 208) as in conversions #1 and #4 then the moose scene gets weird. The debug output says "blend", but the frames sure do not look blended. The horns are flapping badly making the scene look horrible.
2. Conversion #2 puzzles me the most. That's the one using the stripe mask, but not ConditionalFilterMT. The moose scene looks alright, but the marching guys in the red uniforms have blending all over the place, and this blending occurs at all the wrong positions. Very weird...
3. Conversion #3 still looks best to my eyes. Only the moose scene is blended, the other scenes look good without blending. How could disabling the stripe mask (commenting out lines 186 and 187) change the result so massively?

What could be the reason for these issues? Is the Non-SSE2 version of the DLL not working correctly?

Cheers
manolito

MysteryX

20th May 2017, 20:36

The clip actually looks best with BlkSize=12, but you can't use that with the old MvTools2. If you use BlkSize=16, however, the artifact mask is much stronger and a lot of parade frames are being blended, so you'd have to set BlendOver to 60 instead of 50.

If the masks are showing up all at the wrong places, it's generally that the mask hasn't been converted to the destination frame rate -- and the 2nd half of the clip then has no mask at all.

MysteryX

20th May 2017, 20:52

Here is the parade clip
Source (https://mega.nz/#!nJZHEKab!H7TnhbXDzXV8lBxJs6th_hkrZQObgXFuvGIgb0kpxo4)

file="Motion Estimation Torture Clip.avi"
LWLibavVideoSource(file, cache=False)
ConvertToYV12()
FrameRateConverter(60, blksize=8, preset="slower")

BlkSize=8 (https://mega.nz/#!rB5EnLSA!gV_FthFn0EycbrO0471ajYzXavT-IVT1vnkHd2ez2Uc)

BlkSize=12 (https://mega.nz/#!SFoBUaBa!J8WIG81ZKvZ_M2DmH9yn0cd40x0Q5i64wjSHxsSwpKE)

Even though this doesn't yet work right with MT, running with a single thread already has internal multi-threading as it runs at 40%-50% CPU on my 8 cores. BlkSize=8 encoded at 6.45fps while BlkSize=12 encoded at 2.18fps... huge difference!

manolito

21st May 2017, 00:29

Are you serious? Did you even watch your BlkSize=8 conversion?

This one looks horrible. Look at the scene with the car. Didn't you notice that the people in the background are totally butchered?
The lower speed for the BlkSize=12 conversion certainly paid off. And still comparing it to my test conversion #3 I think mine looks better, even using Preset = "normal"...

Cheers
manolito

manolito

21st May 2017, 03:17

Alright, I decided that for me this plugin will be static from now on...

The issues I have are most likely caused by my older plugin versions, but that's fine because I think that the latest improvements which cause my issues are not too important. Generally I believe that development for this tool has reached its peak, it won't get much better than it is now.

My latest and probably final modded version can be found in this post:
https://forum.doom9.org/showthread.php?p=1805050#post1805050

I modified the user interface so this is a normal fps converter now. The only exposed parameters are fps, Preset and BlkSize.

I removed the StripeMask and the ConditionalFilterMT functions so the FrameRateConverter.dll plugin is no longer needed. Also removed all debug related code from the script.

Also Preset = "slower" was removed, and the CalcDiff routine is no longer there. BlkSizes are limited to 8, 16 and 32 again.

In my tests this version is mostly just as fast as the core jm_fps script. In many cases the output will be almost indistinguishable, but for some sources the artifact removal routines do make a real difference.

Thanks to MysteryX and all the other contributors... :thanks:

Cheers
manolito

MysteryX

21st May 2017, 03:50

Keep in mind that if you don't use StripeMask (which isn't an issue unless you have a source with stripes), it weakens the mask and you have to slightly lower the thresholds to make up for it.

And why would you remove the debug option? Before encoding a video, you may want to at least quickly review to make sure it's skipping and blending at the right tresholds.

manolito

21st May 2017, 23:19

Keep in mind that if you don't use StripeMask (which isn't an issue unless you have a source with stripes), it weakens the mask and you have to slightly lower the thresholds to make up for it.

You mean all the way down to the values of the latest script version without the stripe mask from 1-May-2017, like this?
MaskTrh = Default(MaskTrh, 100)
SkipTrh = 80
MaskOcc = MaskTrh > 0 ? Default(MaskOcc, 105) : 0
BlendOver = Default(BlendOver, 30)
SkipOver = Default(SkipOver, 60)

And why would you remove the debug option? Before encoding a video, you may want to at least quickly review to make sure it's skipping and blending at the right tresholds.

That's exactly what I do not want. I want the script to work just like ChangeFPS(), with default thresholds which work in the vast majority of cases right out of the box. The least thing I need is the requirement to adjust params for every different source. For a while I even thought about removing "Preset" and "BlkSize" from the interface... :p

I remember having exactly the same discussion with johnmeyer after I started promoting the jm_fps script. He insisted that his parameters would probably give bad results for a lot of sources, so the parameters needed to be tuned for each new source. I strongly disagreed because in my tests his params for the first time did work well for almost all sources I tried it with, so for me this script was a godsend.

And this is what I want to achieve for the FrameRateConverter script, too. I want defaults which work each and every time.

Cheers
manolito

MysteryX

22nd May 2017, 01:20

Then contribute into improving the defaults. Removing options does nothing to help anyone.

Defaults work well so far, except with the parade with blksize=16 where whole-frame blending starts to happen too often.

manolito

22nd May 2017, 01:56

Removing options does nothing to help anyone.

It sure does help users who don't have a clue about how this whole artifact removal thingy works. And where else do you have AviSynth plugins which feature a plethora of debug options? Debug code is for developers, release versions usually have the debug options removed.

For users who have all the required expertise, there always is your FrameRateConverter.avsi. My modification is for a different group of users...

Regarding my question from my previous post how much I should reduce the thresholds when not using the stripe mask, I did some test conversions using the parade clip and the str clip (Groucho's Japanese girl with the striped stockings), and using these different thresholds did not make any visual difference.

I also converted the parade clip again using jm_fps, and IMO only the moose scene shows a visual improvement when using FrameRateConverter (uses blending instead of interpolating). The other scenes look just fine with jm_fps, I see no need to use artifact removal for these other scenes.

My goal for FrameRateConverter is to have defaults which make any conversion look at least as good as a jm_fps conversion (looking worse than jm_fps is not acceptable whatsoever), but additionally improve the results of jm_fps for demanding scenes like the moose scene. I can probably achieve this by using rather high thresholds for the masks.

Cheers
manolito

raffriff42

22nd May 2017, 02:26

So make a wrapper function (https://en.wikipedia.org/wiki/Wrapper_function) that exposes only the arguments you care about. Your preferred interface can coexist with MysteryX's.

MysteryX

22nd May 2017, 03:51

So make a wrapper function (https://en.wikipedia.org/wiki/Wrapper_function) that exposes only the arguments you care about. Your preferred interface can coexist with MysteryX's.

Even there, a wrapper function wouldn't provide any value. You'd call the wrapper with 3 arguments, while those 3 first arguments are exactly the same as FrameRateConverter, so might as well call FrameRateConverter directly. Unless you want to use different defaults (eg: preset) for your needs, then you could write a wrapper to alter the defaults so you don't need to specify them every time.

As far as I'm concerned, development isn't done here.

Pinterf saw some issues with MvTools2 and MT

Inconsistent output of the same script under mt. These are binary algorithms and not some fuzzy logic with AI intuition and randomness.

Although SVP has been working that way for a while

StripeMask needs to be written in assembly and some bug fixed in it; Pinterf will work on that when he's done with MvTools.

I haven't gotten much feedback on StripeMask yet. Does it work correctly on bad clips? StripeMask normally should only alter the Skip value and not the mask, but in some cases it may slightly alter the mask. Are there cases where it's giving worse output?

Then there's the Diff mode where I also need feedback. Finding any good uses for it besides preset="slower"?

And then with the new block sizes, the defaults will need to be changed. It requires testing a variety of video formats and listing what block sizes work best, and building the defaults from there. And anyone sees anything to gain from non-square block sizes?

For the output, the changes are subtle. It always looks better to leave jm_fps as-is than masking with frame blending, except when there are bad artifacts. It looks best with small and precise masks. As for slower preset, it fixes several details and small artifacts, but it's still only details here and there. It's when you add up all those details properly tweaked that you get a considerably better result.

Slightly improved the Diff mask here (removed blockiness)
https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi

manolito

22nd May 2017, 22:39

It always looks better to leave jm_fps as-is than masking with frame blending, except when there are bad artifacts.

That's exactly what I think... :devil:
So my motto for this kind of tasks is (taken from the old "Alchemist" thread): Try to compensate first, fall back to blending if compensate does not work.

I had been playing a lot with the old salFPS3 (Mug Funky and Didée), but it only worked in theory, the real results were not so good. See here: https://forum.doom9.org/showthread.php?p=899222#post899222

After doing many more tests I concluded that all the recent additions to FrameRateConverter don't do it for me. So I stripped down my modded version even further. Basically all additions since the 1-May-2017 version had to go.

The code is much simpler and clearer now, and all my tests so far did not show any degraded quality compared to the latest edition. The modded script can be found here:
https://forum.doom9.org/showthread.php?p=1805050#post1805050

Cheers
manolito

burfadel

23rd May 2017, 08:54

I am only using normal as slow and slower seem overkill. I feel slower does produce the best results though, but the difference is probably much more noticeable in the encode stats than visually. If it weren't so much slower I'd probably be using it. I'm guessing using DCT currently means doing a whole second lot of calculations? If it could be used only on the part of the frame where there is artifacts and only when necessary (a threshold for example), I think normal + this would be the ultimate solution. I realise this isn't supported in the support tools currently. As it stands, slow seems a bit wasteful, and slower very wasteful since a large amount of the calculated stuff is thrown away, and the useful stuff is always around the areas where you intend the information to be used.

hello_hello

23rd May 2017, 10:23

The code is much simpler and clearer now, and all my tests so far did not show any degraded quality compared to the latest edition. The modded script can be found here:
https://forum.doom9.org/showthread.php?p=1805050#post1805050

I don't use frame interpolation much but I stumbled into this thread and tried the script & it seems quite good, relative to Interframe at least.
I'll confess I changed the function name to xfps though. Life's too short to be typing underscores, and xfps seems kinda cool. ;)

A question though....

What's the intention with the speed presets? The script seems a bit undecided.

## @ Preset - The speed/quality preset [slower|slow|normal|fast]. (default=normal)

P_SLOWER = 0 P_SLOW = 1 P_NORMAL = 2 P_FAST = 3
Pset = Preset == "slow" ? P_SLOW : Preset == "normal" ? P_NORMAL : Preset == "fast" ? P_FAST : -1
Assert(Pset != -1, "mx_fps: 'Preset' must be slow, normal or fast {'" + Preset + "'}")

Should the Assert line be permitting Preset="slower"?

Cheers.

Sharc

23rd May 2017, 13:22

After doing many more tests I concluded that all the recent additions to FrameRateConverter don't do it for me. So I stripped down my modded version even further. Basically all additions since the 1-May-2017 version had to go.

The code is much simpler and clearer now, and all my tests so far did not show any degraded quality compared to the latest edition. The modded script can be found here:
https://forum.doom9.org/showthread.php?p=1805050#post1805050

Cheers
manolito

Comparison here (default settings, except presets)
http://www.mediafire.com/file/oranq24vuccna10/FRC_comparison.mkv

Edit:
Somewhat better results I got with MaskThr=255 in MysteryX's script:
http://www.mediafire.com/file/4vwdtbc2gutvlw9/FRC_comparison2.mkv

Comments, suggestions for better tweaking?

MysteryX

23rd May 2017, 15:59

Should the Assert line be permitting Preset="slower"?

Official script allows preset slower. (https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi)

Manolito took the diff feature out. DCT=1 is overkill, but preset="slower" isn't much slower, so I'd rather go with either Normal for HD or Slower for SD.

Somewhat better results I got with MaskThr=255
MaskThr=255 completely disables artifact masking.

It won't look good where there are actually artifacts, so I suggest comparing on high-artifacts scenes.

Since many high-artifacts scenes are being skipped altogether, it's easier to compare MaskTrh by setting BlendOver=0 and SkipOver=0, then you can look at what it's doing with ugly scenes.

manolito

23rd May 2017, 17:04

What's the intention with the speed presets? The script seems a bit undecided.
Should the Assert line be permitting Preset="slower"?

I removed Preset = "slower", but obviously I wasn't thorough enough... :eek:

Script is fixed now.

There is a reason for removing "slower". Switching between DCT=1 and DCT=0 is a good idea in therory, but in my tests it did not work as intended. I have this anime clip where using DCT=1 removes tons of bad artifacts, but with the "slower" preset the script mostly picks the frames which use DCT=0. Which means that a conversion with Preset "slow" looks much better than with Preset = "slower" (and it is a little faster also).

Please note that for DCT=1 AND BlkSize=32 I disabled artifact masking completely (by forcing Output = "flow"). Looked better in my tests...

Cheers
manolito

manolito

23rd May 2017, 19:15

Comparison here (default settings, except presets)
http://www.mediafire.com/file/oranq24vuccna10/FRC_comparison.mkv

Edit:
Somewhat better results I got with MaskThr=255 in MysteryX's script:
http://www.mediafire.com/file/4vwdtbc2gutvlw9/FRC_comparison2.mkv

Comments, suggestions for better tweaking?

The second test with MaskThr=255 shows that the core jm_fps script without any artifact treatment often looks better than the elaborate FrameRateConverter script. This is my experience, too.

The first test is interesting. The two Preset = "normal" conversions should look identical, but for some frames my script seems to be a little better. I can't explain this because the parameters are identical. The only differences are that my script has no stripe mask and that I do not use ConditionalFilterMT.

Comparing the two "slower" and "slow" conversions IMO it is a tossup. At the beginning the "slower" clip looks better, later in the clip I think my "slow" script gives better results.

For me the bottom line is that none of the scripts can get rid of the artifacts, but when watching the clip in real time I find the quality quite good. If I had encoded this scene with the "normal" preset I would never see the need to try a slower preset because of real bad artifacts.

Cheers
manolito

//EDIT//
It might be worth a try to encode the clip using my script and specifying Preset = "slow" and BlkSize = 32. In my experience this works well with objects like the fence.

MysteryX

23rd May 2017, 20:05

Sharc

23rd May 2017, 21:20

Interesting. First, the fence doesn't get handled properly.

Then, a *lot* of frames are done with Blending; yet look different between my script and Manolito's script? It's as if whole-frame blending wasn't taking effect. Full-frame blending should look exactly the same.

Then, there are frames where the fence really doesn't look good with blending either; it would look better to skip it.

Can you upload your source so I can try with it?
Here the source (originally provided by Selur):
http://www.mediafire.com/file/4xfo0poblri72su/forInterpolation.mp4

For my tests I resized it to 960x540.

MysteryX

23rd May 2017, 23:12

OK. Good news is that StripeMask is accurately detecting the barrier.

The philosophy right now is that StripeMask is weak enough so that it gets counted for Skip value but doesn't alter the Mask (or very little), so most of the clip gets blended as expected.

Question is: for clips where only small areas have stripes, should StripeMask mask those areas? If so, then there's the risk that it's going to enlarge the mask in other areas where it isn't beneficial.

Darkening StripeMask so that it masks those areas would introduce a bunch of other problems. For example, those marks are the size of a block, so if we mask all full blocks in the mask, then the whole stripemask would trigger artifact removal which is bad.

This clip is like the moose scene: the best is to blend it all. In this particular case, setting BlendOver=41 would give better results, but I don't think I'd want to lower the default to 40. Since we only do frame blending, this isn't a good clip to test any of the other features.

There is also a bug where the last frames have a completely white mask. I saw this before and fixed it by deleting the last frame and repeating the other one before; but in this case the 3 last frames are white! Anyone knows what is the cause of this? Manolito, does the old version of MvTools2 have the same behavior?

I fixed a minor issue in my script to make StripeMask's strength relative to SkipTrh to get consistent output with various block sizes.

The difference between using StripeMask or not is that without it, you should get inconsistent frame blending even if you set BlendOver=41. Of course in this case, if you just set it low enough, it will all get blended, but then other scenes that don't need frame blending will get blended as well. With it, you get more consistent detection to blend the frames.

The other option would be to process the stripe mask entirely separately with a different logic to detect large chunks, let's say, 4x larger than blksize, and then merge that mask with the first mask. This would allow to mask only those areas and interpolate the rest of the image, and wouldn't require lowering BlendOver. I think that idea could work nice actually. Mask processing isn't heavy anyway compared to interpolation.

MysteryX

24th May 2017, 00:21

Alright I implemented masking of large patches of StripeMask :) It's working fine.

Try the latest script, and you'll get best results with BlendOver=0. The girl will be smooth the whole clip, there will be no serious artifacts (a little bit around her head only), and since we're focused on the pretty girl, we won't even notice the blended background.

There is still the issue with the last 3 frames having a white mask.

hello_hello

24th May 2017, 04:03

I removed Preset = "slower", but obviously I wasn't thorough enough... :eek:

Script is fixed now.

Cheers.

I thought as a tiny contribution I'd share my new found fondness for wrapper functions, brought on by typing laziness.
It just calls your script with preset frame rates by adding the frame rate to the function name. ie

xfps() = Double frame rate by default, or fps can be specified
x23fps() = 23.976fps
x24fps() = 24fps
x25fps() = 25fps etc.

It's easily edited to use a different preset or BlkSize by default.
Admittedly in this case it doesn't save a lot of typing, but every little bit helps. :)

# xfps - mx_fps wrappers
#
# Wrapper functions for mx_fps
# https://forum.doom9.org/showthread.php?p=1805050#post1805050
#
# xfps() = Double frame rate by default, or fps can be specified
# x23fps() = 23.976fps
# x24fps() = 24fps
# x25fps() = 25fps
# x29fps() = 29.97fps
# x50fps() = 50fps
# x59fps() = 59.94fps
# x60fps() = 60fps
#

# ========== xfps Wrapper Function - Double frame rate, or fps can be specified =====================

function xfps(clip c, float "fps", string "Preset", int "BlkSize")
{
fps = default(fps, c.framerate * 2.0)
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

# ========== x23fps Wrapper Function ====================================================

function x23fps(clip c, string "Preset", int "BlkSize")
{
fps = 24.0/1.001
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

# ========== x24fps Wrapper Function ====================================================

function x24fps(clip c, string "Preset", int "BlkSize")
{
fps = 24.0/1.0
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

# ========== x25fps Wrapper Function ====================================================

function x25fps(clip c, string "Preset", int "BlkSize")
{
fps = 25.0/1.0
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

# ========== x29fps Wrapper Function ====================================================

function x29fps(clip c, string "Preset", int "BlkSize")
{
fps = 30.0/1.001
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

# ========== x50fps Wrapper Function ====================================================

function x50fps(clip c, string "Preset", int "BlkSize")
{
fps = 50.0/1.0
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

# ========== x59fps Wrapper Function ====================================================

function x59fps(clip c, string "Preset", int "BlkSize")
{
fps = 60.0/1.001
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

# ========== x60fps Wrapper Function ====================================================

function x60fps(clip c, string "Preset", int "BlkSize")
{
fps = 60.0/1.0
Preset = default(Preset, "normal")
BlkSize = default(BlkSize, c.width>2000||c.height>1200 ? 32 : c.width>=720||c.height>=480 ? 16 : 8)
return c.mx_fps(fps, Preset, BlkSize)
}

hello_hello

24th May 2017, 04:18

Official script allows preset slower. (https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi)

Manolito took the diff feature out. DCT=1 is overkill, but preset="slower" isn't much slower, so I'd rather go with either Normal for HD or Slower for SD.

Cheers.

For funzies, I put together similar wrapper functions for FrameRateConverter as I did for manolito's version of the script. I didn't know which options others might want to change from their defaults, so I kept them all. This time it saves a bit more typing. FrameRateConverter Wrapper - xFRC.avsi (https://files.videohelp.com/u/210984/FrameRateConverter%20Wrapper%20-%20xFRC.avsi)

Unfortunately I couldn't test the script when encoding as it'd encode for a short period, then x264 would crash with an error apparently relating to FrameRateConverter.dll. I didn't make note of any details but I can try encoding again and provide what I can if you like.

Mind you it could be an XP thing, or my PC. Aside from the occasional reboot it's been running for about four years with zero maintenance, so it's due for a reformat. Or should I be using AVIsynth+ instead of the standard version?

I was using the script dated 18-May-2017.

PS MysteryX. The new version of the script is still dated 2017-05-18. I just thought I'd let you know in case you want to change it to avoid confusion.
Cheers.

raffriff42

24th May 2017, 07:15

MysteryX, pardon my ignorance, but where is the latest version of your script?

EDIT: thanks!

Sharc

24th May 2017, 07:34

MysteryX, pardon my ignorance, but where is the latest version of your script?
Good question. My guess is from here:
https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi

hello_hello

24th May 2017, 07:34

It says the last update was 7 hours ago, so I'm guessing:

https://github.com/mysteryx93/FrameRateConverter

The only place I could find FramerateConverter.dll was in the zip file here:

https://github.com/mysteryx93/FrameRateConverter/releases

Sharc

24th May 2017, 12:57

MysteryX

24th May 2017, 15:04

Here the update of the comparison with your latest script and BlendOver=0. It looks fine real-time on TV.
http://www.mediafire.com/file/mx4w2vuel255vfl/FRC_comparison4a.mkv

:)

Now I'm only updating the AVSI script on GitHub. I'll only release another package when there are changes to the DLL.

For the crashes, are you using ST or MT? It doesn't yet work with MT.

Admittedly in this case it doesn't save a lot of typing, but every little bit helps. :)

Sometimes I forget that not everybody types at 200 words per minutes, especially in an age of cellphones.

pinterf

24th May 2017, 16:17

Does it really work?

EMocc = MaskOcc > 0 ? C.ConvertToY8().MMask(bak, ml=MaskOcc, kind=2, gamma=1/gam, ysc=255, thSCD2=255).mt_inpand() : BlankClip(C, pixel_type="Y8", color_yuv=$000000)
EM = MaskOcc > 0 ? EM.Overlay(EMocc, opacity=.4, mode="lighten", pc_range=true) : EM

MMask is getting its format from the input clip, which is Y8 here.
But MMask does not support it (but it should, now I see).
The input clip format is not checked at all in this filter, I'm getting
"Filter error: GetPlaneWidthSubsampling not available on greyscale pixel type." when it tries to get subsampling for the non-existant chroma planes.

But when you happen to use a YV12 clip instead of Y8, either convert it to greyscale _before_ mt_inpand, or specify "process" mode (U=3, V=3) for mt_inpand, or else chroma planes will be garbage (This is important only if you ever want to use this mask for chroma later other than mt_merge(.. luma=true))

And the last one:
"pc_range" in Overlay is only used for RGB input or output, because Overlay "lighten" works in YUV colorspace

Then I have mentioned the mvtools MFlowFPS MT mode problem: I was getting different file sizes for each encoding.

There were two or three old bugs, but this is the nastiest one:
part of the bottom mask area was not cleared up and contained some random memory garbage in each run. The bug affected the bottom or bottom right part of the occlusion time mask. This mask is used by all MxxxxFPS when they are run in specific modes. It occured under specific circumstances, depending on the relationship between clip dimensions and blocksize and padding. Finding the culprit one single missing letter took me more than 30 hours, I was getting mad. Release later, I have to clean up all the debugging garbage I have put in the source.

MysteryX

24th May 2017, 16:51

Well... the script *is" working here so I'm not sure if something needs to be changed.

If I don't specify pc_range, then there were issues where the masks wouldn't go full black and full white, and instead the whole images would have partial blending (using 16 and 235 values instead of 0 and 255). It's the only way I found to fix it.

Those nasty spelling errors are the worst, I hear you! Undetectable to the naked eye! When bugs don't make any sense, I had taken the habit of looking for the stupidest causes.

Sharc

24th May 2017, 18:02

For the crashes, are you using ST or MT? It doesn't yet work with MT
ST only. Avisynth 2.60

MysteryX

24th May 2017, 18:31

ST only. Avisynth 2.60
There's a weird bug in StripeMask that causes crashes even in ST. I added code that checks memory boundaries on every assignment and I haven't been getting crashes since; but apparently you've got some anyway. Pinterf will look into it when he gets time; or I'll go deeper into debugging it.

Sharc

24th May 2017, 19:05

I didn't get crashes... must be someone else, unless you mean the last 3 frames issue ....

MysteryX

24th May 2017, 20:23

I didn't get crashes... must be someone else, unless you mean the last 3 frames issue ....
hello_hello got a crash in FrameRateConverter.dll

MysteryX

25th May 2017, 00:22

I also just got an access violation crash -- so definitely something still needs to be fixed in the DLL.

Script updated to improve the masking of stripe areas. The bicycle scene now looks very smooth.

Sharc

25th May 2017, 08:38

Updated comparison here:
http://www.mediafire.com/file/p8qccveqgvxtjgq/FRC_comparison4b.mkv

I did some blind tests on TV with "uneducated" viewers. Interestingly, nobody made comments about the fence. Only after I told them to focus on it, they noticed "some differences" but were still undecided about the ranking.

MysteryX

25th May 2017, 14:49

The latest version mostly has differences with the trees above the fence, resulting from more consistent and precise coverage of the fence. You have to know what to look for as these are details.

Also the fact that you're showing each version small makes it nearly impossible to see such details. Adding to the fact that this video only has 1 of 2 frames interpolated and the other half identical.

hello_hello

25th May 2017, 19:32

It occured under specific circumstances, depending on the relationship between clip dimensions and blocksize and padding. Finding the culprit one single missing letter took me more than 30 hours, I was getting mad. Release later, I have to clean up all the debugging garbage I have put in the source.

The 30 hours is appreciated. Thank you.

MysteryX

26th May 2017, 04:43

Another update. (https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi)

- Further improved the stripe mask to reduce false positives and give smoother results.
- Raw stripe mask will no longer affect artifact masking and will be processed separately. To reflect the slight weakening of the mask, MaskTrh default changed from 150 to 145.
- Adjusted default block sizes as follow:
Defaults for 4/3 video of height:
0-359: 8
360-749: 12
750-1199: 16
1200-1699: 24
1600-2160: 32

Note that stripe masking will cover some flag scenes in the parade. It may not look better (some frames are better, some are worse), but that's technically correct because that's what it's designed to do. It's not "worse" and it will give more consistent results across a range of videos.

Sharc

26th May 2017, 09:15

MysteryX

26th May 2017, 13:09

Error in Line 98
BlkSize = Default(BlkSize, DefH<360 ? 8 : DefH<750 ? 12 : DefH<1200 ? 24 : DefH<1600 ? 32)

should be (I think):
BlkSize = Default(BlkSize, DefH<360 ? 8 : DefH<750 ? 12 : DefH<1200 ? 16 : DefH<1700 ? 24 : 32)
But then I get an error in line 149 with the blocksizes x*y .....
Uh... how did this one slip through?

After fixing that, I'm not getting more errors. What syntax are you using when getting error at line 149?

Sharc

26th May 2017, 13:26

I am getting:
MAnalyze: Block's size must be 4x4, 8x4,8x8,16x2,16x8,16x16,32x16,32x32
(FrameRateConverter.avsi, line 149

I don't get the error when I set BlkSize=8 or 16 or 32 explicitly. I get the error when I set BlkSize=24 (or leave it default)

MysteryX

26th May 2017, 13:29

and what block size are you using?

12 and 24 and only supported in Pinterf's latest version

Sharc

26th May 2017, 13:44

I left it at default. So I am guessing that it used 12 or 24 now (?).
In the previous version it picked 16 for the same clip.

Maybe I have to doublecheck with pinterf's (very)latest mvtools .... (will try later as I have to leave for now....)

burfadel

26th May 2017, 15:53

I get the error as well:
MAnalyze: Block's size must be 4x4, 8x4,8x8,16x2,16x8,16x16,32x16,32x32
(FrameRateConverter.avsi, line 149

The issue is this:
BlkSize = Default(BlkSize, DefH<360 ? 8 : DefH<750 ? 12 : DefH<1200 ? 16 : DefH<1600 ? 24 : 32)

If DefH is such that 12 or 24 is used, it seems Manalyse won't accept it as it states it only allows 4x4, 8x4,8x8,16x2,16x8,16x16,32x16,32x32, meaning 12x12 and 24x24 aren't supported.

To get the interim sizes this works, but probably stuffs up the masks and other things:
BlkSize = Default(BlkSize, DefH<360 ? 8 : DefH<750 ? 16 : DefH<1200 ? 16 : DefH<1600 ? 32 : 32)
BlkSizeV = Default(BlkSizeV, DefH<360 ? 8 : DefH<750 ? 8 : DefH<1200 ? 16 : DefH<1600 ? 16 : 32)

Instead of using invalid block sizes 12x12 and 24x24, it will use 16x8 and 32x16.

Sharc

26th May 2017, 16:14

With pinterf's mvtools2 2.7.19.22 from here (https://github.com/pinterf/mvtools/releases/tag/2.7.19.22) with depans it works now here even with BlkSize 12 and 24.
I am on avisynth 2.60 32bit (standard)

burfadel

26th May 2017, 16:27

Version 2.7.20.22 of mvtools2 just released. The bug has been fixed that caused the crash preventing me from using 2.7.19.22. The official framerateconverter script works fine now.

Sharc

26th May 2017, 17:18

I just tried it. Seems to work fine here and is even a bit faster.

MysteryX

27th May 2017, 03:11

New release: 2017-05-26 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.2-alpha)

The DLL access violation has been fixed, and performance improved. It now fully supports MT. It still only supports 8-bit with no SIMD.

MysteryX

27th May 2017, 07:06

I thought it would be interesting to show how the stripe mask works.

Frame 91, Interpolated image
https://s12.postimg.org/xxfs7e821/91flow.png (https://postimg.org/image/xxfs7e821/)

Scan blocks by calculating line averages and detecting contrast changes. Dynamic content blends into greys and show no such contrast.
https://s12.postimg.org/wwfji9r2h/91lines.png (https://postimg.org/image/wwfji9r2h/)

From these lines, detect recurring patterns and mark those areas.
https://s12.postimg.org/3wpqg0xu1/91patterns.png (https://postimg.org/image/3wpqg0xu1/)

Blur

\ .BicubicResize(Round(C.Width/BlkSize)*4, Round(C.Height/BlkSizeV)*4)
\ .FRC_GaussianBlur2(6)

https://s12.postimg.org/n9c18jy2x/91blur.png (https://postimg.org/image/n9c18jy2x/)

Binarize

\ .mt_binarize(90)

https://s12.postimg.org/wfubvu3bd/91binarize.png (https://postimg.org/image/wfubvu3bd/)

Blur again

\ .mt_expand(mode= mt_circle(zero=true, radius=5))
\ .FRC_GaussianBlur2(1.4)
\ .BicubicResize(C.Width, C.Height)

Final mask
https://s12.postimg.org/u3mby8qq1/91mask.png (https://postimg.org/image/u3mby8qq1/)

Final result
https://s12.postimg.org/oyl4gmds9/91auto.png (https://postimg.org/image/oyl4gmds9/)

Sharc

27th May 2017, 09:11

Very interesting demo. Thanks.

Edit:
Comparison "normal,flow" (interpolation) ..... "slower,auto" (masking). All other settings = default
http://www.mediafire.com/file/7z90z0l5b49t5e7/FRC_comparison4e.mkv
The masking works very well now IMO, also for higher resolutions of the picture.

burfadel

28th May 2017, 06:32

Final result
https://s12.postimg.org/oyl4gmds9/91auto.png

Would there be a way to do a mask to avoid the artifact of certain types of motion, as seen to the top left of the girls shoulder? If you have a edge mask of the non-motion interpolated frames, and one of the motion-interpolated frame, you could do a motion analysis on them. The artifacts would shown as additional information in the edgemask, in which case the original frames information can be used instead of the interpolated frame. Or at least, a principle based on this concept?

I guess it's similar to stripemask, where you resolved the artifact in the fence.

However, compared with:

Frame 91, Interpolated image
https://s12.postimg.org/xxfs7e821/91flow.png

The remainder of the fence looks like it has motion distortion. Wouldn't that be unfavourable? Would it be possible to only apply the stripemask to the area where the artifact was created?

MysteryX

28th May 2017, 07:48

hum... I see what you're saying... if the StripeMask is still in an area, then it means it has no artifact. When artifacts appear, it means the stripes are broken.

I think you're onto something.

I was also thinking of a way of generating a mask of continuous areas in a way that would preserve more precise borders. A filter that combines unidirectional blur and binarize. Take the average of x pixels in a certain direction, and if the average is above a treshold, include that pixel in the mask. Repeat in all 4 directions.

MysteryX

29th May 2017, 04:22

New version 2017-05-28! (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.3-alpha)

hum... I see what you're saying... if the StripeMask is still in an area, then it means it has no artifact. When artifacts appear, it means the stripes are broken.
I've tried it and had no luck with it. The results are too erratic and inconsistent. It looks better to blend the whole fence, and while animated, you barely see it.

I have created a new filter ContinousMask. It turns [left] into [right]
https://s29.postimg.org/ns3te0pr7/Continuous_Mask1.png (https://postimg.org/image/ns3te0pr7/) https://s29.postimg.org/3m0bf4u3n/Continuous_Mask2.png (https://postimg.org/image/3m0bf4u3n/)

You specify a radius, and it takes [radius] pixels to the right, to the left, to the top and to the bottom and makes the average of them in the output. It only processes pixels that have a value > 0 in the source. Perfect for detecting continuous areas! Then you just have to call mt_binarize.

Using that, I've edited the stripe masking to cover the fence more precisely. Barely any artifact is making it through. So far it looks great, but it might require a bit more testing to see if it sometimes gives false positives, it maybe can be tweaked some more. You can also test this vs the last version.

MysteryX

29th May 2017, 21:13

StirpeMask will now work in Linear Light (2.2 gamma correction) which gives more consistent results. Adding high-bit-support, will release after a few other things are fixed.

Script updated. Stripes are now covered with no blending as it looks better. Stripes don't look good even with blending. It however causes artifacts at the junctions of no-blend zones but they're precise enough so that it doesn't do too much such artifacts. Also added Stripes argument to enable/disable stripe masking (default=true).

MysteryX

30th May 2017, 16:21

New version: 2017-05-30 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.4-alpha)

In this release, stripes detection is now done in linear light which results in more accurate detection across the spectrum. Added parameter Stripes to specify how to mask stripes: 0=disabled, 1=skip, 2=blend (default=1)

I'm leaving to California for the next 5 days so won't be giving updates until I'm back. This is a good time to test it. Send me positive vibes and wish me luck.

Edit: release updated with a fix for ContinuousMask's 16-bit support. Now it should work in 8 or 16-bit. What's not yet working in 16-bit is ConditionalFilterMT with AverageLuma that needs to be normalized. The normalization code is there but it isn't being called somehow.

Edit: release updated again. 16-bit mode now working 16-bit mode now working (except that debug won't display normalized values, but result will be good)

Sharc

30th May 2017, 23:11

No luck here. Crashes for "slower", stripes=1 or stripes=2, after few frames:
- for Stripes=1: Softwire: caught an access violation at 0x1390e25b(code+83)', attempting to read from 0xfffffff [ScriptClip], line 7
- for Stripes=2: MVTools: invalid vector stream, [Script Clip], line2
Single threaded, avisynth 2.6

Enjoy your trip to California :)

MysteryX

31st May 2017, 00:42

No luck here. Crashes for "slower", stripes=1 or stripes=2, after few frames
!?

It works perfectly fine here. Perhaps try with AVS+?

What's even more strange is that you're getting a crash on MVTools2 "MVTools: invalid vector stream"

Sharc

31st May 2017, 07:56

Perhaps try with AVS+?
Yep! All ok with AVS+. Thanks.

For comparison:
http://www.mediafire.com/file/3k7x9tpk71n6ve8/FRC_comparison6.mkv
This difficult clip is getting better and better. The bottom left picture (slower, other settings default) looks very good (stepping through frames and watching real-time)

burfadel

31st May 2017, 15:56

I get an error that there is no function named 'StripeMask' with the latest build. I believe there is an error with the dll's, they're practically identical! Only a few lines difference and 1 byte between the x64 and 32-bit DLL's when doing a file compare. With the 28 May version the DLL payload and file sizes are quite different as you would expect between 32-bit and 64-bit. It seems the 64-bit version of the 30 May buld is the 32-bit code, maybe at a slightly different commit because they aren't identical.

Here are the differences:
Comparing files FrameRateConverter.dll and FRAMERATECONVERTER-X64.DLL
***** FrameRateConverter.dll
L
¦Í-Y

***** FRAMERATECONVERTER-X64.DLL
L
BÍ-Y

*****

***** FrameRateConverter.dll

¦Í-Y

***** FRAMERATECONVERTER-X64.DLL

BÍ-Y

*****

***** FrameRateConverter.dll

¦Í-Y

***** FRAMERATECONVERTER-X64.DLL

BÍ-Y

*****

***** FrameRateConverter.dll

¦Í-Y

***** FRAMERATECONVERTER-X64.DLL

BÍ-Y

*****

***** FrameRateConverter.dll

¦Í-Y

***** FRAMERATECONVERTER-X64.DLL

BÍ-Y

*****

***** FrameRateConverter.dll

RSDSxOvûâ5H*;S`*hG

***** FRAMERATECONVERTER-X64.DLL

RSDSxOvûâ5H*;S`*hF

*****

***** FrameRateConverter.dll

¦Í-Y

***** FRAMERATECONVERTER-X64.DLL

BÍ-Y

*****

So yes, something drastically wrong there! The 28 May x64 DLL works fine in its place for normal use, but obviously would be preferable to use the correct version.

Groucho2004

31st May 2017, 16:44

I get an error that there is no function named 'StripeMask' with the latest build. I believe there is an error with the dll's, they're practically identical! Only a few lines difference and 1 byte between the x64 and 32-bit DLL's when doing a file compare. With the 28 May version the DLL payload and file sizes are quite different as you would expect between 32-bit and 64-bit. It seems the 64-bit version of the 30 May buld is the 32-bit code, maybe at a slightly different commit because they aren't identical.

The DLLs are both x86 from the same source. Minor differences will always be there because the linker puts a time stamp in the image. Therefore, using fc (or whatever binary compare tool you used) is pointless in this case.

Also, you can easily check this with AVSMeter:
[Avisynth info]
VersionString: AviSynth+ 0.1 (r2489, MT, x86_64)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 6
Multi-threading support: Yes
Avisynth.dll location: D:\WINNT\system32\avisynth.dll
Avisynth.dll time stamp: 2017-05-29, 09:04:37 (UTC)
PluginDir2_5 (HKLM, x64): E:\Apps\VideoTools\AVSPlugins\AutoLoad64
PluginDir+ (HKLM, x64): E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins

[CPP 2.5 / 64 Bit plugins]
E:\Apps\VideoTools\AVSPlugins\AutoLoad64\flash3kyuu_deband.dll

[CPP 2.6 / 64 Bit plugins]
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\ConvertStacked.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\DirectShowSource.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\ImageSeq.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\Shibatch.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\TimeStretch.dll [2.2.6.0]

[Plugin errors/warnings]
______________________________________________________

Error loading "E:\Apps\VideoTools\AVSPlugins\AutoLoad64\FrameRateConverter-x64.dll"
Cannot load 32 bit DLL with 64 bit Avisynth
______________________________________________________

burfadel

31st May 2017, 16:59

The DLLs are both x86 from the same source. Minor differences will always be there because the linker puts a time stamp in the image. Therefore, using fc (or whatever binary compare tool you used) is pointless in this case.

Also, you can easily check this with AVSMeter:
[Avisynth info]
VersionString: AviSynth+ 0.1 (r2489, MT, x86_64)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 6
Multi-threading support: Yes
Avisynth.dll location: D:\WINNT\system32\avisynth.dll
Avisynth.dll time stamp: 2017-05-29, 09:04:37 (UTC)
PluginDir2_5 (HKLM, x64): E:\Apps\VideoTools\AVSPlugins\AutoLoad64
PluginDir+ (HKLM, x64): E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins

[CPP 2.5 / 64 Bit plugins]
E:\Apps\VideoTools\AVSPlugins\AutoLoad64\flash3kyuu_deband.dll

[CPP 2.6 / 64 Bit plugins]
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\ConvertStacked.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\DirectShowSource.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\ImageSeq.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\Shibatch.dll
E:\Apps\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\TimeStretch.dll [2.2.6.0]

[Plugin errors/warnings]
______________________________________________________

Error loading "E:\Apps\VideoTools\AVSPlugins\AutoLoad64\FrameRateConverter-x64.dll"
Cannot load 32 bit DLL with 64 bit Avisynth
______________________________________________________

Ah ok! True. Either way, it shows the DLL is wrong. AVSmeter of course saying it directly.

Just in time for MysteryX's five day's away!

MysteryX

5th June 2017, 05:41

Just in time for MysteryX's five day's away!

DLL updated (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.4-alpha)

From LAX airport

MysteryX

6th June 2017, 00:18

New version 2017-06-05 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.5-alpha)

What's new:
- ContinuousMask no longer counts center pixel twice
- StripeMask threshold raised from 22 to 25 (after adding gamma correction)
- FrameRateConterter Stripes parameters is now [none|skip|blend]
- Stripes masking has higher blur radius when Stripes=skip, looks better
- Added Output=stripe to view the stripes mask

The parade clip triggers minor stripes masking in a few scenes. With Stripes=skip, it just looks great; can barely notice the mask at all

MysteryX

6th June 2017, 00:32

I just tried with Avisynth 2.6 and am not getting any crash. However, the results is different for some reason. The parade clip has issues with the crowd by the car, and it appears worse in Avisynth 2.6.

MysteryX

6th June 2017, 00:46

Raised StripeMask Trh from 25 to 26 and the parade crowd now looks fine. Updated release.

Sharc

6th June 2017, 17:58

Hmmm... I found the version of 30-May with skip=1 better (less artefacts for the fence) for the clip with the girl on the bike.....

MysteryX

6th June 2017, 18:41

There is a bug in this build, will release another one

MysteryX

6th June 2017, 23:18

New version (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.5.1-alpha)

What's new:
- ContinuousMask now counts central pixel twice again
- Increased ContinuousMask search radius from 18 to 20
- Increased Stripe Mask mt_expand from 8 to 12 when Stripes="skip"

Sharc

7th June 2017, 08:20

Comparing with version of 30-May I'd say that some frames are better, some are worse.....
But in any case it looks good. The improvements compared to the early versions are significant :)

MysteryX

7th June 2017, 18:10

I increased the blurring radius to remove "hard cuts"; perhaps it could be tweaked some more.

MysteryX

8th June 2017, 01:13

I see what you're saying about May 30th version giving better output.

New version: 2017-06-07 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.6-alpha)

Try this version. I had to tone down the settings a little bit so that the parade crowd doesn't get distorted -- it's tweaked on a fine line so that the fence looks good and the parade is excluded.

Sharc

8th June 2017, 09:59

It looks good now.
Question is perhaps how much is it tuned for the selected clips (fence, parade), or whether the current version/optimization is generally good for other clips as well....

MysteryX

8th June 2017, 16:45

It looks good now.
Question is perhaps how much is it tuned for the selected clips (fence, parade), or whether the current version/optimization is generally good for other clips as well....
Test it and let me know. It should be mostly good, and by testing with a variety of other videos, we can fine-tune settings further.

For example, after adding stripes masking and detecting after gamma correction, it might be possible to increase the blend over threshold.

MysteryX

8th June 2017, 19:18

I'm making tests with other videos. Stripes masking tends to trigger on parts of subtitles; which isn't bad since artifacts tend to show up around the text. However, "skip" doesn't look good when applied randomly in such places. "skip" is better in the clip you posted with large fences, but in most cases, "blend" looks better and is safer. Thus, I'll change Stripes default to "blend". I'm also changing another settings in StripeMask, Comp default for blksize=16 will be 2 instead of 3 (comparing 2 pixels to detect contrast changes instead of 3 for blksize>16)

For the most part, things are looking great.

MysteryX

8th June 2017, 21:44

MysteryX

9th June 2017, 02:25

Here's an encode of a full sample video (difficult case with strong artifacts). It's part of a script that denoises and upscales from 288p into 768p.

Source (https://www.spiritualselftransformation.com/files/media-encoder-old.mpg)

Previous results with InterFrame (https://www.spiritualselftransformation.com/files/media-encoder-new.mkv)

Current results with FrameRateConverter in YV24 (https://mega.nz/#!iEAwFAaL!WzWjT5CllT9YlCxT9iJxcwEB4pbOptEEC5nXCKo9Zi0)

There are some artifacts with the lights near the beginning, but everything else is perfect. The quality is drastically higher than my previous results!!

Here's the script I'm using

file="Meu-Ayw-Tua-Lae-Tur.mpg"
LWLibavVideoSource(file, cache=False)
AudioDub(LWLibavAudioSource(file, cache=False))
Crop(0, 0, -8, -0)
ConvertBits(16)
ConvertToYUV444(chromaresample="Spline36", ChromaInPlacement="MPEG1")
ConvertToStacked()
KNLMeansCL(D=2, A=2, h=1.5, channels="YUV", device_type="GPU", device_id=0, lsb_inout=true)
ConvertFromStacked()
SuperResXBR(5, 1, 0, XbrStr=2.6, XbrSharp=1.3, MatrixIn="Rec601")
ConvertBits(8, dither=1)
FrameRateConverter(NewNum=60, NewDen=1)
SuperResXBR(3, 1, 0, XbrStr=2.6, XbrSharp=1.3, fWidth=1012, fHeight=778, fKernel="Bicubic", fB=0, fC=.75, FormatOut="YV12")
ResizeX(1004, 768, 0, 5, -8, -5)
Prefetch(8)

I'm having some performance issues though.

Sharc

10th June 2017, 10:07

New version: 2017-06-08 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.7-alpha)

Several tweaks:
- StripeMask's comparison width is now 2 instead of 3 for blksize=16
- BlendOver default raised from 50 to 60
- Stripes default changed from "skip" to "blend" -- safer
- Output=over now displays stripes mask as yellow (and artifact mask as cyan)
- Debug now displays Diff and Stripe values, and 4 digits instead of 6 digits
- Works correctly in 16-bit mode while maintaining compatibility with AVS 2.6 (debug mode still doesn't work correctly in 16-bit)
This version is very strong, for both stripes=skip and stripes=blend mode. Very nice!
The only minor issue is that it crashes under avisynth 2.60 here. No problem with avs+ though.

Edit:
Probably just the debug mode which created the avisynth 2.60 crash...

Sharc

11th June 2017, 19:26

Test it and let me know. It should be mostly good, and by testing with a variety of other videos, we can fine-tune settings further.

For example, after adding stripes masking and detecting after gamma correction, it might be possible to increase the blend over threshold.
Here another example for framerate doubling, with the version of 08-June-2017. Looks good, doesn't it?
http://www.mediafire.com/file/f06xv53etvf1se3/FRC_ma.mkv

MysteryX

15th June 2017, 00:52

I'm not satisfied with the video I posted earlier.

Here's an updated script. (https://github.com/mysteryx93/FrameRateConverter/blob/master/FrameRateConverter.avsi) StripeMask no longer affecting Skip, as Skip is designed for dealing with inconsistent masks with a threshold, while StripeMask is consistent in its results and can be accurately masked. Because of the lights causing distortions at the beginning of my clip, I lowered MaskTrh from 145 all the way down to 100. It looks a lot better, but still has a lot of artifacts with the flashes of light. Overall, the mask isn't strong enough to blend those frames, so it must be dealt with in other ways. Lowering the mask didn't have as much effect as I though it would on other clips; I tested the parade clip and it's not worse. It probably has limited impact because of the gamma curve that was applied on the mask first. I'll have to test on other clips if that still works.

The flashing lights still cause massive distortions that I'm not able to get rid of. Any idea on how to deal with this?

Sharc

15th June 2017, 08:29

The flashing lights still cause massive distortions that I'm not able to get rid of. Any idea on how to deal with this?
I am afraid that this presents a basic problem to motion estimation. Not sure that there is a solution to this (flashing lights, strobing)

kolak

15th June 2017, 09:58

1 thing for sure DCT=1

MysteryX

15th June 2017, 17:20

1 thing for sure DCT=1
That video is perfectly fine with DCT=1 (preset="slower")! and it requires it

MysteryX

17th June 2017, 18:51

Even with DCT=1 (preset="slower"), I get better results on my video with MaskTrh=100 than with 110 or 145. Can you test with your videos?

Sharc

18th June 2017, 19:33

Even with DCT=1 (preset="slower"), I get better results on my video with MaskTrh=100 than with 110 or 145. Can you test with your videos?
MaskTrh=145 has the edge in my case. Little to no difference between 100 or 110.

MysteryX

19th June 2017, 00:54

Is 145 just slightly better and 100 still good? (and safer for general use)

Sharc

19th June 2017, 08:13

Is 145 just slightly better and 100 still good? (and safer for general use)
Differences are subtle only and frame dependent. 100 is definitely still good. No problem.

MysteryX

19th June 2017, 18:13

One option is to add Artifacts=Weak|Medium|Strong. But if the difference is minor and 100 still looks good, then I'll just keep that as default.

Sharc

19th June 2017, 20:14

Agree. Better not overdo with too many options.

MysteryX

20th June 2017, 22:47

Agree. Better not overdo with too many options.
I did some more tests and this version seems good. Nothing more to add at the moment.

hello_hello

21st June 2017, 19:06

FrameRateConverter seems to be causing havoc on my PC (XP and Avisynth 2.6). I'm using a script such as the following one. AssumeFPS(25) was added for testing to see if going from a non-integer frame rate to an integer frame rate was the problem. FrameRateConverter.dll version 2017-06-08 with the updated version of the script. Most of the time programs simply crash without an error message as soon as I try to preview the script, but I managed to catch a few.

LoadPlugin("C:\Program Files\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\VTS_02_1.d2v")
AssumeFPS(25)
FrameRateConverter(50)

VirtualDub said:

An out-of-bounds memory access (access violation) occurred in module 'avisynth'...
...reading address 00000004...
...while running thread "Dub-I/O" (thread.cpp:197).

For AvsPMod it'd start out:

http://image.ibb.co/j5tm55/error_1.gif

Then a couple of times it got funky claiming errors in scripts in the Avisynth plugins folder that weren't actually being used in the script being opened.

http://image.ibb.co/bsu4sk/error_2.gif

http://image.ibb.co/m7Ztk5/error_3.gif

As soon as I removed FrameRateConverter() from the script things returned to normal.

Cheers.

Sharc

21st June 2017, 19:19

I had to switch to AVS+ (x86, 32bit) in order to make it work.
It crashed under Avisynth 2.6.0.6 here as well.

Groucho2004

21st June 2017, 21:31

I had to switch to AVS+ (x86, 32bit) in order to make it work.
It crashed under Avisynth 2.6.0.6 here as well.
False assumption below, please ignore.

The explanation is quite simple:
From Init.cpp

// Convert input to 8-bit; nothing to gain in processing at higher bit-depth.
int SrcBit = input->GetVideoInfo().BitsPerComponent();
if (SrcBit > 8) {
...
}
Calling a function that is exclusive to AVS+ without any error handling in place will of course cause a hard crash with the classic Avisynth versions.

MysteryX

21st June 2017, 21:55

Didn't Pinterf say that BitsPerComponent was implemented in the header file in a way that was backwards compatible?

manolito

21st June 2017, 22:14

This is my main complaint about FrameRateConverter... :scared:

MysteryX develops his app using only the latest and greatest versions of AviSynth, MaskTools and MVTools without any regard to users who prefer older (and more stable) versions of these filters.

Please note that I do not complain about his app not working on my ancient WinXP machine with a Non-SSE2 CPU. I am talking about a ThinkPad running Win7-64 on an Intel Core i5 CPU.

It probably comes down to Pinterf's efforts to modernize older AviSynth filters. It is fine with me to add new functionality like new color spaces and higher bit depths to these filters. But I firmly believe that feeding those filters with a script which does not make use of these new features should produce identical output compared to the older versions. (Maybe with the exception when a real bug has been discovered with the old versions).

But with Pinterf's latest versions of AVS+, MaskTools and MVTools this is not the case. When I run FrameRateConverter with identical parameters and switch between the different plugin versions, I either get crashes (AVS 2.6 vs AVS+), or the results are very different, probably due to different mask strengths.

This is the main reason why I gave up on FrameRateConverter in its current incarnation.

Cheers
manolito

MysteryX

21st June 2017, 22:48

I just haven't done tests yet with v2.6. Reporting bugs is sufficient, no need to complain.

StainlessS

21st June 2017, 23:11

Didn't Pinterf say that BitsPerComponent was implemented in the header file in a way that was backwards compatible?

AVS+ is "backwards compatible" with standard, but AVS standard is not necessarily "forwards compatible" with AVS+.

If coder uses only AVS standard methods, then no probs. :helpful:

Groucho2004

21st June 2017, 23:25

Didn't Pinterf say that BitsPerComponent was implemented in the header file in a way that was backwards compatible?
I had a look at the AVS+ header and there is indeed a fallback mechanism in place where for example BitsPerComponent() will return a constant 8 in classic Avisynth.

So, I don't know what's causing the crash but it does seem to be something in FramerateConverter.

MysteryX

22nd June 2017, 00:04

I had a look at the AVS+ header and there is indeed a fallback mechanism in place where for example BitsPerComponent() will return a constant 8 in classic Avisynth.
Told ya :)

Will have to debug later.

manolito

22nd June 2017, 16:35

The crash under AVS 2.60 is definitely caused by FrameRateConverter.dll. The last version which does not cause a crash is this version from May 28th:
https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.3-alpha

The following version from May 30th and all newer versions do crash AVS 2.60 reliably... :mad:

The problem with the old version from May 28th is that the conversion result is much worse.

Cheers
manolito

hello_hello

22nd June 2017, 18:06

It probably comes down to Pinterf's efforts to modernize older AviSynth filters. It is fine with me to add new functionality like new color spaces and higher bit depths to these filters. But I firmly believe that feeding those filters with a script which does not make use of these new features should produce identical output compared to the older versions. (Maybe with the exception when a real bug has been discovered with the old versions).

You made me curious so I had a look, as I regularly use those plugins with QTGMC (makstools2, mvtools2, rgtools). I tested with QTGMC 3.33.

I'll confess when comparing the Avisynth output I don't think I could pick differences between the new plugins and the old ones visually, but there are some differences because when I saved identical screenshots (png) the image file size when using the old plugins was generally marginally higher (ie 849.6MB vs 849.4MB).

When I saved screenshots after encoding, the screenshots taken from the encode using the new plugins were slightly larger, which I thought was odd, given it was the other way around before encoding (ie 819.7MB vs 820.4MB).

Whatever the differences in the plugins, you can visually see it causes the video to be encoded slightly differently when using QTGMC. Not necessarily better or worse (at CRF18), which is what your post made me paranoid about and why I looked, but a little differently.

Even using my old Q9450 CPU though, the same NTSC video de-interlaced with QTGMC 3.33 encodes a couple of fps faster using the new plugins than it does with the old ones.

MysteryX

22nd June 2017, 18:53

Small differences? Pseudo-random AI code in action :)

Please isolate the function causing a difference so that Pinterf can look into it.

MysteryX

23rd June 2017, 02:29

I had a look at the AVS+ header and there is indeed a fallback mechanism in place where for example BitsPerComponent() will return a constant 8 in classic Avisynth
Found the problem. It instead returns 0.

MysteryX

23rd June 2017, 02:41

New version available: 2017-06-22 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.8-beta)

- Fix compatibility bug with Avisynth 2.6
- Reduced MaskTrh from 145 to 100, and SkipTrh from 60 to 55
- StripeMask is no longer affecting BlendOver/SkipOver values nor the artifact masking. It is processed entirely separate.

hello_hello

23rd June 2017, 03:57

Please isolate the function causing a difference so that Pinterf can look into it.

Well I don't think my previous observations were wrong, but if there are differences they're probably not enough to see easily with the human eye when viewing the raw Avisynth output, but I can only assume the difference is enough to effect the way the video is encoded, and that can be seen visually.

Old_Plugins.avs
LoadPlugin("E:\RGTools.dll")
LoadPlugin("E:\mvtools2 2.5.11.22.dll")
LoadPlugin("E:\masktools2 2.1.0.0.dll")
Import("E:\QTGMC 3.33.avsi")
LWLibavVideoSource("E:\video.mkv")
QTGMC()

New_Plugins.avs:
LoadPlugin("E:\RgTools 0.96.dll")
LoadPlugin("E:\mvtools2 2.7.20.22.dll")
LoadPlugin("E:\masktools2 2.2.10.dll")
Import("E:\QTGMC 3.33.avsi")
LWLibavVideoSource("E:\video.mkv")
QTGMC()

Comparing Old_Plugins.avs with itself (as a test):
AA=Import("E:\Old_Plugins.avs")
BB=AA
Return Compare(AA,BB)

http://image.ibb.co/ichjA5/compare1.jpg

Comparing Old_Plugins.avs with New_Plugins.avs:
AA=Import("E:\Old_Plugins.avs")
BB=Import("E:\New_Plugins.avs")
Return Compare(AA,BB)

http://image.ibb.co/eFJPA5/compare2.jpg

Is the difference enough to care about, or is it even a bad thing? If it is, I'm not sure I'd know where to start looking.

hello_hello

23rd June 2017, 04:48

New version available: 2017-06-22 (https://github.com/mysteryx93/FrameRateConverter/releases/tag/v0.8-beta)

- Fix compatibility bug with Avisynth 2.6

So far, so good.

Thanks!

manolito

23rd June 2017, 12:42

@hello_hello

My "Old_Plugins" line looks a little different from yours:

LoadPlugin("E:\RGTools.dll")
LoadPlugin("E:\mvtools2 2.5.11.22.dll")
LoadPlugin("E:\masktools2 2.1.0.0.dll")

Mine:

LoadPlugin("E:\RemoveGrain.dll") # Version 1.0 from August 2005
LoadPlugin("E:\mvtools2.dll") # Version 2.5.11.22 by Fizick
LoadPlugin("E:\mt_masktools-26.dll") # Version 2.0a48

These are the latest versions which work without SSE2. I need to use these versions on my old desktop machine. On the newer laptop with the Core i5 CPU I could of course install the latest plugin versions, but I hate to maintain 2 different AviSynth\Plugin folders. The newer laptop has to run the same plugins which the old machine does. The sacrifice in speed is insignificant (so far I refuse to use the highly unstable multithreading versions of AviSynth).

Cheers
manolito

hello_hello

23rd June 2017, 15:58

manolito,
I just asked google and found some older versions of the plugins for that test. I wasn't trying to compare any specific older version, but I'll run some more tests later (maybe tomorrow) to see if it's just the Pinterf plugins that produce a different result, and maybe try to work out which functions are different.

I use the standard Avisynth myself as a couple of times I started some quite lengthy encodes, walked away and returned many hours later to find an error message on the monitor and very little encoding had been done, so I went back to single threaded Avisynth. If filtering is a serious bottleneck, I run two encodes at a time.

MysteryX

23rd June 2017, 18:23

MysteryX

23rd June 2017, 18:27

I've done a new encoding test

Source (https://www.spiritualselftransformation.com/files/media-encoder-old.mpg)

Previous results with InterFrame (https://www.spiritualselftransformation.com/files/media-encoder-new.mkv)

Current results with FrameRateConverter in YV24 (https://mega.nz/#!vVA2yDxD!4BfZZ1bL5MfCRrmLOEmz9MR_E72nlk8y4wWXTfood0w)

Quality is great and a HUGE improvement. Performance, however, is unacceptable. It ran at ~15% CPU usage.

Script

file="Meu-Ayw-Tua-Lae-Tur.mpg"
LWLibavVideoSource(file, cache=False)
AudioDub(LWLibavAudioSource(file, cache=False))
Crop(0, 0, -8, -0)
ConvertBits(16)
ConvertToYUV444(chromaresample="Spline36", ChromaInPlacement="MPEG1")
ConvertToStacked()
KNLMeansCL(D=2, A=2, h=2.1, channels="YUV", device_type="GPU", device_id=0, lsb_inout=true)
ConvertFromStacked()
SuperResXBR(5, 1, 0, XbrStr=2.6, XbrSharp=1.3, MatrixIn="Rec601")
ConvertBits(8, dither=1)
FrameRateConverter(NewNum=60, NewDen=1, Preset="slower")
SuperResXBR(3, 1, 0, XbrStr=2.6, XbrSharp=1.3, fWidth=1012, fHeight=778, fKernel="Bicubic", fB=0, fC=.75, FormatOut="YV12")
ResizeX(1004, 768, 0, 5, -8, -5)
Prefetch(8)

manolito

23rd June 2017, 23:31

This is what I found out using the latest version of FrameRateConverter (2017-06-22):

The new "FrameRateConverter.dll" really made all the difference. It turned out that all my bad looking conversions I had done so far were due to the old DLL from 2017-05-28.

With the current DLL it did not really matter if I used the latest MaskTools and MVTools, or if I used the old versions (which work without SSE2). There were slight differences, but nothing anybody could detect while viewing the conversion result. My apologies to pinterf... :stupid:

Of course I had to modify the script slightly to work with the old Fizick MVTools version. I had to restrict the default block sizes to 8, 16 and 32, and I needed to remove "luma = true" from line 196 of the script.

Quality was very good using the defaults. The "Slower" preset was not really worth the speed sacrifice in most cases.

On my ancient Non-SSE2 computer the "FrameRateConverter.dll" did of course not work at all. I had to remove all references to ConditionalFilterMT and to the Stripe Mask from the script to make it work. Fortunately this turned out to be no big loss, the conversion results are pretty good without the stripe mask.

Cheers
manolito

//EDIT//
I also updated my simplified and stripped down version which I call "mx_fps.avsi" here:
https://forum.doom9.org/showthread.php?p=1805050#post1805050

hello_hello

24th June 2017, 00:29

To compare versions, you can take a frame and save it to PNG, and the 2 versions should have exactly the same size. If there are a few bytes of difference between the 2 versions, then something is wrong. You need to isolate the script line that causes such difference.

I haven't done any MaskTool2 testing yet, but so far RGTools 0.96 seems consistent with the original version. MVTools2 though, not so much. The summary:

MVTools2 version 2.6.0.5 (cretindesalpes) has a different output to version 2.5.11.22 (Fizick)
Version 2.7.0.1 (1st Pinterf) has identical output to version 2.6.0.5 (cretindesalpes)
Version 2.7.20.22 (latest Pinterf) has a different output to version 2.7.0.1 (1st Pinterf)

And I think I found a bug.
I posted about it all in the MVTools2 thread (https://forum.doom9.org/showthread.php?p=1810302#post1810302).

MysteryX

25th June 2017, 04:34

Upon looking at my test video carefully, there's a misplaced frame at 9048. It's a wrong frame that somehow got inserted there for 3 frames. No idea how or why that happened.

manolito

1st July 2017, 21:08

Could you determine what happened so you got those misplaced extra frames?

I tried to reproduce it using your latest FrameRateConverter version, just adapted the script for the older MaskTools and MVTools versions. With FPS=60 and Preset="slower" for your test clip I got a CPU usage of 33%, and there were no misplaced frames at the location you posted.

Could it be my older plugin versions, or could it be related to some of the other filters in your script?

Whatever, I believe this plugin could go to Stable status pretty soon...

Cheers
manolito

MysteryX

2nd July 2017, 15:21

I have a program that can run the script in various segments, pause/resume, and merge all the segments at the end. It is possible that the glitch happened there. Nobody else encountered such misplaced frame issue?

Quality seems to be good enough, and it looks stable enough. Performance is the main issue at this point. Either I can run MvTools2 through a profiler see if something pops up, or I could modify my software to run scripts in 8 single-threaded segments and merge in the end, kind of like how downloader software are doing.

edcrfv94

2nd July 2017, 15:37

Some animation will introduces a lot of artifacts.
FrameRateConverter(NewNum=60000, NewDen=1001, Preset="slower")
https://mega.nz/#F!pQ4R3YQY!468FTficImO1crYM9T0qDA

Tempter57

2nd July 2017, 18:36

Some animation will introduces a lot of artifacts.
FrameRateConverter(NewNum=60000, NewDen=1001, Preset="slower")
https://mega.nz/#F!pQ4R3YQY!468FTficImO1crYM9T0qDA
Used for anime DoubleFPS( mode=2, mopro=true, bias=128, show=false) https://mega.nz/fm/YaYG0LTD

MysteryX

2nd July 2017, 19:58

For anime, you might get better results with Preset="slow" as the DCT=0 clip is bad. Also, limiting to FrameDouble will greatly reduce artifacts. Also try various block sizes and that makes the greatest difference.

MysteryX

3rd July 2017, 12:57

With MT, MvTools2 isn't only slow. It freezes after a while, too.

burfadel

4th July 2017, 00:00

With MT, MvTools2 isn't only slow. It freezes after a while, too.

Do you mean using prefetch in the script, like prefetch(8)? I found encoding really only stable to prefetch(3). Internal multi-threading is much better, to state the obvious!

Mvtools should have a pixel based (loosely like mflow function) prescan. From pixel movement grouping you could automatically determine the ideal blocksize for each picture element instead of just blindly applying 8x8 etc to the whole frame. It would eliminate the need for mrecalculate and produce the best results since the best blocksize for each picture element is used. The prescan would be in its own thread, and because of the prescan you can safely divide work of the block movements without potentially affecting quality.

Faster, and much better results! Considering all the extra bocksize options they could all potentially automatically be utilised.

MysteryX

4th July 2017, 05:47

Nobody even understands how MvTools2 work, so it's unlikely somebody is going to do such development. (at least, Pinterf updated the code without even understanding the logic)

Fixing bugs, however, is another story. There are still stability issues in MvTools2 with MT, especially with DCT=1.

I posted a benchmark of jm_fps function with MT here here (https://forum.doom9.org/showthread.php?p=1811108#post1811108)

burfadel

10th July 2017, 03:17

I think I've come across those stability issues with MT, I have a R7-1700X, if I disable SMT so only 8c/16t it is much more reliable. Basically it just seems to stall. It could also be knlmeanscl, nnedi3, or something else as well. Ideally the filter shouldn't see 16 individual threads, it should see the 8 cores, apply work as necessary on these, and then divide the work with the core and SMT thread. This way it helps the data stay in the suitable cpu logic as cache. I think at the moment the data isn't divided efficiently.

MysteryX

12th July 2017, 05:30

The filter doesn't see threads, Avisynth manages that.

I've been using KNLMeansCL and NNEDI3 extensively and they are stable. The stalls are due to MvTools2, and it seems to happen mostly with DCT=1.

manolito

16th July 2017, 03:13

No updated version since beta 0.8 from 2017-06-22 ? Is it becoming stable now?

I have been using this version quite a bit during the last weeks and I am quite happy with it. I mostly used the defaults, mainly because I found that Preset="slower" and also the DiffBlkSize settings are not very useful (at least their usefulness is not worth the speed sacrifice). The automatic detection routine which DCT value or which block size looks better is not working IMO. In most cases the choice which FrameRateConverter made automatically is different from the choice I would have made visually.

Since I use old versions of MVTools2 and MaskTools2 plus plain vanilla AVS 2.60 I did not see any stability problems. I believe that especially the latest MVTools2 incarnations are still work in progress and far from stable.

So I think it's time for me (again) to strip down the AVSI. My plans are like this:

1. Make the script compatible with old MVTools2 by Fizick (restrict default block sizes to 8, 16 or 32 and remove "luma=true" from line 196 of the script).
2. Remove all debug code
3. Only export the FPS, Preset and BlkSize parameters.
4. Remove Preset="slower" and DiffBlkSize routines.
5. Maybe disable the mask if DCT==1 AND BlkSize==32 (i.e. force output to "flow" instead of "auto" in this case).
I am not absolutely sure about this last change, but so far I did get better results and a better speed with this setting.

Whatever, please let me know if there are any improved versions to expect in the near future. If not, I will go ahead with my changes to the script.

Cheers
manolito

MysteryX

16th July 2017, 13:44

No updated version since beta 0.8 from 2017-06-22 ? Is it becoming stable now?
Stable in single-threaded mode. MT has stability and performance issues, due to MvTools2 (mostly with DCT=1 it seems). Pinterf is the only one who could solve that, and it most likely would require a lot of work.

I'm thinking of editing my encoder program to encode videos in segments (single-threaded) similar to how downloader software split in many parts to do parallel downloads, then I can merge all the segments together. This would be a work-around; more of the programming required for this is already done, I'd just have to add code for automatic segments management. However this wouldn't help those using Avisynth directly.

I mostly used the defaults, mainly because I found that Preset="slower" and also the DiffBlkSize settings are not very useful (at least their usefulness is not worth the speed sacrifice).
It depends on the content. Many of my VCDs require preset="slower". Anime require preset="slow" as they look crap with DCT=0. Most other content works with preset="normal".

The automatic detection routine which DCT value or which block size looks better is not working IMO. In most cases the choice which FrameRateConverter made automatically is different from the choice I would have made visually.
How about sharing useful data about which settings work best for specific types of content? Thanks!

By the way, are you using MvTools2 in single-threaded mode or with MT?

and remove "luma=true" from line 196 of the script
You sure about that? This may change the behavior of the script.

luma is a special mode, where only the luma plane of the mask is used to process all three channels.

The mask may contain garbage in the chroma planes, which would then be applied against the clip's chroma.

manolito

17th July 2017, 04:59

Stable in single-threaded mode.

This is all I need. My modified and simplified version absolutely has to work with the old single-threaded versions of all helper plugins.

It depends on the content. Many of my VCDs require preset="slower". Anime require preset="slow" as they look crap with DCT=0. Most other content works with preset="normal".

I agree that anime mostly requires DCT=1. But the automatic switching between DCT=1 and DCT=0 in Preset="slower" has never worked for me.

By the way, are you using MvTools2 in single-threaded mode or with MT?

No MT on my machine. Even on my Core i5 computer I found that almost all MT AviSynth stuff is not reliable, so I ditched it.

You sure about that? This may change the behavior of the script.
The mask may contain garbage in the chroma planes, which would then be applied against the clip's chroma.

This was your own suggestion from this post:
http://forum.doom9.org/showthread.php?p=1807366#post1807366
For the modded version it does not make a difference anyways because the whole CalcDiff routine has been removed.

Meanwhile I went ahead and updated my modded script version. The difference to the previous version is that I kept the stripe mask. This means that FrameRateConverter.dll is required this time, and therefore a CPU with SSE2 support is also required.
You can find the new version here:
https://forum.doom9.org/showthread.php?p=1805050#post1805050

Cheers
manolito

MysteryX

19th July 2017, 20:26

and therefore a CPU with SSE2 support is also required.
Wasn't non-SSE2 CPU the sole reason for not using the latest MvTools2?

manolito

20th July 2017, 06:35

Wasn't non-SSE2 CPU the sole reason for not using the latest MvTools2?

Yes, it was for my ancient desktop computer. On my newer Core i5 notebook I could of course use the latest MvTools2 by pinterf. Two reasons why I don't:

1. I like to keep the plugin folders for both computers consistent. It gets too confusing for me to maintain two different plugin configurations. So I use the least common denominator...

2. Following pinterf's MvTools2 thread I can only conclude that his versions are work in progresss and far from stable. Since I have no use for all the new features I prefer to stick with the old and stable versions. (Maybe this attitude comes with getting older...)

Cheers
manolito

MysteryX

20th July 2017, 15:47

AFAIK Pinterf's version only has some issues with MT -- which the old version didn't support well at all. It's mostly bug fixed. On a quad-core, however, single-threaded processing is useless.

Do whatever you want to do -- but regressing from Pinterf's work and reverting my work won't help anyone else here. You're spending your energy on undoing work instead of actually improving things.

Instead, you could put your energy into further improving development. If you see any issue with MvTools2, report them. As for FrameRateConverter, DCT is a manual setting, there is no automatic detection there. If you say different block sizes work better for you, report which settings work best for you -- but considering you're using a subset of available block sizes, it's unlikely you'll get better results that way but still tweaks can be done.

manolito

20th July 2017, 19:34

Do whatever you want to do -- but regressing from Pinterf's work and reverting my work won't help anyone else here. You're spending your energy on undoing work instead of actually improving things.

We obviously do have very different philosophies here. I am not undoing your work, I am just removing things which for me proved to be useless. I want a simple frame rate converter which works well most of the time without tweaking tons of parameters. This frame interpolation method has its inherent drawbacks which noboby will be able to overcome. All the various directions you have tried (stripe mask, the whole calcdiff routine) only produced tiny improvements at the price of vastly reduced speed and the need for the user to tune parameters for every different source. Nothing I want to deal with...

As for FrameRateConverter, DCT is a manual setting, there is no automatic detection there.

What? So what is the whole calcdiff thingy doing?
Slower: Slow + calculate diff between DCT=1 and DCT=0 to take the best from both
So you say there is no automatic DCT decision between DCT=1 and DCT = 0 for Preset = "slower" ?

The same goes for DiffBlkSize. If the user specifies a different block size, of course FrameRateConverter makes an automatic decision which of the two block sizes should be used. The problem is that this automatic decision is mostly wrong (of course depending on the source) for DCT as well as for BlkSize.

As far as I am concerned this project has reached its peak. Thanks a lot for your work, but I am outta here... :devil:

Cheers
manolito

MysteryX

20th July 2017, 20:00

You don't seem to understand how this works.

All the various directions you have tried (stripe mask, the whole calcdiff routine) only produced tiny improvements at the price of vastly reduced speed
Not true. The script is only slower when user slower features. An "off" feature or a "non-existent" feature have exactly the same performance.

and the need for the user to tune parameters for every different source. Nothing I want to deal with...
So far automatic settings, with either of the 3 speed presets, is working very well.

What? So what is the whole calcdiff thingy doing?
Nothing unless you set preset="slower" or configure it manually. So far, quality gain is consistent with preset="slower" except for anime where DCT=0 gives crap, but applying it in other ways hasn't brought any further benefits.

So you say there is no automatic DCT decision between DCT=1 and DCT = 0 for Preset = "slower" ?
Preset Normal = DCT=0 (fastest method for good results)
Preset Slow = DCT=1 (recommended for anime)
Preset Slower = DCT=1 with DCT=0 diff (generally better results but slower)

That's a fully manual setting.

As far as I am concerned this project has reached its peak. Thanks a lot for your work, but I am outta here... :devil:
You're trying to undo the work because you don't understand how it works. Seek to understand it first. Since default settings work great (according to feedback), it's not complicated.

Here's what would make it easier. Forget about the code. Simply try to understand the use. Using default settings and specifying a preset is easy.

Telling anyone to do otherwise for no reason is doing them a disservice and creating confusion.

Of course your reaction will be to attack this post because your mind is set on destroying not creating -- but before you do, pause.

raffriff42

21st July 2017, 01:09

As far as I'm concerned FrameRateConverter works very well, even with the "fast" preset. I tried some things but could not improve upon it. Thanks for your hard work. (suggest you start a new thread and let this one fade away)

MysteryX

21st July 2017, 17:04

TODO list:

- Adding assembly code for FrameRateConverter.dll functions (low priority due to bottlenecks elsewhere)

- Testing performance and stability, both in ST and MT modes. This would allow identifying if issues are purely related to MT, and whether it is only DCT=1 that causes instabilities or whether there are also issues with DCT=0.

- Somehow improving MT support (higher CPU usage). What I'm thinking of doing is editing my encoder software (Yin Media Encoder shipping with the Natural Grounding Player) to encode in segments similar to how download accelerator softwares do, and then merge those segments at the end. But that's a work-around not a solution. SMDegrain also performs poorly in MT most likely because of the same issues in MvTools, so both could be improved at the same time.

StainlessS

21st July 2017, 17:13

MysteryX,

You are one of Gods creatures with limited lifespan, we are so glad that you give a little (well a lot) to us :)

MysteryX

22nd July 2017, 03:52

you truly have no idea what my main focuses have been the past few years -- glad you appreciate

MysteryX

22nd July 2017, 23:58

I'm doing a few encodings. With normal preset, it works great. Adding x264 encoding, CPU usage goes to 80% and it runs smoothly.

But it's a pity. Preset="slower" gives much better results, but I can't get even a short clip to complete without it freezing and having to do it in segments. It's just there to tease us -- until Pinterf or someone else gets his hands dirty and finds a solution.

My recommendation. Preset="normal" works with MT. Presets "slow" and "slower" shouldn't be mixed with MT for now.

burfadel

23rd July 2017, 01:39

I found even on normal preset it is problematic with MT for extended periods of encoding. It seems maybe it chews through the desktop heap memory and doesn't release it.

MysteryX

23rd July 2017, 03:03

I found even on normal preset it is problematic with MT for extended periods of encoding. It seems maybe it chews through the desktop heap memory and doesn't release it.
Memory leak is highly possible. Does the memory usage gradually go up?

I just did my short clip encodings in ST and it went fine. Dreadfully slow but at least it goes through the end without freezing.

burfadel

23rd July 2017, 08:29

There's plenty of memory free, however the free memory isn't the only memory that an application uses. I'm not sure whether all the different memory pools are shows in task manager, I didn't think to look at the time. If you open task manager, go to details view, right click at the top where the titles are (such as Title, PID, Status etc), and select 'Select Columns', you can view the use of memory by applications. You have options for working set, peak working set, working set delta, memory (private working set), memory (shared working set), commit size, paged pool, NP pool (that's non paged pool), PF Delta, handles, threads, user objects, GDI objects. I guess if you add those columns in MT mode, and check whether any become excessive for the encoder (avs2pipmod64, ffmpeg, x264, whatever is processing the avisynth side of things). This still doesn't show the heap usage from what I can tell, the Microsoft application to do this is from 2006.

Not sure whether Process Explorer will show everything, it's procexp64.exe from https://live.sysinternals.com/

For me if I set the MT for MVtools as Multi-Instance and Prefetch(4), it appears to work, but doing length encoding of a batch of files over say, 30 hours, the whole system becomes slow and sluggish to the point where you have to reset. Another program I had open said it was an out of memory error, and if you google it the message will show not only if your physical memory is exhausted but the system allocated memory pools as well. It's the latter that I believe is happening as I had plenty of memory free at the time. This happened even when I just set the MT mode for MVtools and nothing else (although setting prefetch(4) I believe enabled internal mutlithreading in Avisynth+?). Since then I have not set MT mode for MVtools functions, but have for everything else using mode compatibility info I could find. This is in an imported .avsi file. Most filters are MT_NICE_FILTER, including all the Masktools functions, and literally every other filter I have set it apart from fft3dfilter, and no issues.

Sorry I couldn't be more helpful in actually seeing what is being used, but I believe it's a leak or issue related to those memory pools etc and not just in physical memory usage.

MysteryX

1st August 2017, 22:25

I'm testing code that runs several ST instances in parallel and then merges at the end. On 8 cores, 8 instances eats way too much memory and doesn't give any better performance than 4 instances. With 4 instances, CPU still stagnates at 45-50%, spread equally across 8 cores, both with DCT=0 or DCT=1. At least, I'm able to run DCT=1 with stability and acceptable performance. I still don't find it satisfying.

Edit: after running a while CPU went up to 60-70% so it's not bad. 6.2GB total memory used on my system though ... and then later goes down to 25-45%. I wonder what makes it vary like that.

After running more tests, with DCT=0, running 4 instances gives 4.2fps @ 50% CPU usage. With MT=8, I get 3.5fps @ 25-32% CPU usage. Segmenting the video into ST processes gives a 20% performance increase for a 50% higher electric bill. This method shows more benefits for DCT=1, and the real solution would be to fix that.

MysteryX

2nd August 2017, 03:18

I see one improvement. With DCT=0, slower block sizes are slower and generally give higher quality but more artifacts. With DCT=1, larger block sizes are extremely slower.

For presets Slow and Slower, perhaps it should use smaller block sizes.

EDIT: Not true. Larger block sizes are still faster with DCT=1.

MysteryX

3rd August 2017, 21:23

By running videos in several ST segments, I'm able to get ~50% CPU usage with 4 instances on a 8-core system ... but can't go higher, and adding more instances doesn't help. BUT, since the CPU is only at 50% usage, it runs at 1.18ghz instead of 2.4ghz, so really I'm still running at 25% of the CPU's capacity.

Any idea what is causing the bottleneck in this case?

Occasionally there's a CPU peak at 90% @ 2.4ghz and frame rate goes from 1.2 to 2.9, but this only lasts a few seconds before going back down.

burfadel

4th August 2017, 00:00

Wouldn't that be hitting the system cache pretty hard?

MysteryX

4th August 2017, 00:25

Wouldn't that be hitting the system cache pretty hard?
Which cache and how do I know? It hits memory hard: 6.4GB of 7.9GB taken. It says 1.5GB still available.

burfadel

4th August 2017, 01:23

Your CPU's L1, L2, and L3 cache. To oversimplify, the L1 cache is split for data and instructions and is there for the immediate execution, very small but extremely fast. It's 32 KB for instructions and 32 KB for data even the latest i7's. For Ryzen it is 64 KB for instructions and 32 KB for data. The L2 cache is a little bigger, is 256 KB per core for i5/i7 and 512 KB per core for Ryzen, and ideally contains data to be executed next. The L3 cache is the largest and slowest, being 6 MB for i5, 8 MB for i7, and 16 MB for 8 core Ryzen.

If you are doing data processing, SuperPi being a great example, you are transferring relatively small amounts of data, but doing a lot of processing. For encoding, you are transferring and processing a huge amount of data. This has to be copied into the cache. Generally it should be okay since the next lot of data needing transfer is known. There's a whole issue of cache misses and other factors, read though the article on Wikipedia https://en.wikipedia.org/wiki/CPU_cache. Yes, it is Wikipedia, but this page should be accurate enough :).

Anyways, processing one lot of data is fine because it knows what comes next. Processing lots of different segments in parallel, it has to load the different data into the cache ready to be executed. If you run many processes in parallel I can see how the cache can get saturated and simply can't feed the CPU. Setting affinity won't help much, because although the L2 cache is per core, the L3 cache is shared amongst all cores. Flog that with large amounts of data from multitasking processes, and you will have a bottleneck even with affinities set. This is my best guess as to what is happening anyway. The increase is power useage etc could be relating to huge amounts of cache misses.

MysteryX

4th August 2017, 02:28

Now this is good *AND* frustrating. I've encoded some videos. Performance is "OK", quality is great.

This is a strobe lights fest with tons of artifacts. It's coming out great with very little artifacts. The image is much better defined than with SVP.

This video requires Preset="slower" or shows considerable artifacts, and thus, can't run with MT. Running 4 ST instances gives me 1.2-1.6fps @ 45-50% CPU usage at half frequency. Processing a video takes 2.5h, compared to 40-60 min with SVP. Note that here I process in YV24 while SVP processes in YV12.

Source file (288p) (https://mega.nz/#!OBA0iTiI!D6_larToWlQUu8mPR9y4SrbS6noN8M9oo5PUNL5onQ0)
Output with SVP (https://mega.nz/#!zYIyRSJY!T4oDOTclNL8kO3rdpdx7Nx86Y-XIeG8z38o8Drrq774)
Output with FrameRateConverter (https://mega.nz/#!zJJ3AABS!bPpQfP3eAP7u5A91hYhSrvCaKD0hGr0JJfOqBe6HZyA)

Everything is perfect EXCEPT one detail. Near the end, frame 12939 is the wrong frame, and again at frame 13717, and frame 15992. The file got processed in 4 equal segments and this isn't on the merge locations so that's unrelated to the segments/merge code.

There is definitely yet another bug in MvTools2 that causes frames to occasionally mix up, even in ST. Nothing I can do about it. PINTERF COME BACK!!

We might want to know whether this is only with DCT=1 or whether DCT=0 is also affected by this bug. I'll do some more testing and report.

Script

file="Do-Ra-Me.mpg"
LWLibavVideoSource(file, cache=False)
AudioDub(LWLibavAudioSource(file, cache=False))
Crop(0, 0, -8, -0)
ConvertBits(16)
ConvertToYUV444(chromaresample="Spline36", ChromaInPlacement="MPEG1")
ConvertToStacked()
KNLMeansCL(D=2, A=2, h=1.8, channels="YUV", device_type="GPU", device_id=0, lsb_inout=true)
ConvertFromStacked()
SuperResXBR(5, 1, 0, XbrStr=2.7, XbrSharp=1.3, MatrixIn="Rec601", Engines=1)
ConvertBits(8, dither=1)
FrameRateConverter(NewNum=60, NewDen=1, Preset="slower")
SuperResXBR(3, 1, 0, XbrStr=2.7, XbrSharp=1.3, fWidth=1012, fHeight=778, fKernel="Bicubic", fB=0, fC=.75, FormatOut="YV12", Engines=1)
ResizeX(1004, 768, 0, 5, -8, -5)

StainlessS

4th August 2017, 02:58

On cache hits measurement type stuff,

In any case, to answer your question, yes, performance monitor counters are available on each core on any Intel processor after nehalem. But you probably want to know more than that...

Dont know if of interest:- https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/548988

EDIT: https://www.google.co.uk/search?q=l2+cpu+memory+cache+hit+miss+measurement&oq=l2+cpu+memory+cache+hit+miss+measurement&gs_l=psy-ab.3..33i160k1.23859.26422.0.26873.12.12.0.0.0.0.140.1294.3j9.12.0....0...1.1.64.psy-ab..0.12.1291...33i21k1.OOJIRj1KlF8

MysteryX

4th August 2017, 05:16

burfadel

4th August 2017, 10:55

On a 1080p video, running 4 ST instances (and no other filter) results in 79% CPU usage 1.87ghz with DCT=0, and 50-55% CPU usage 1.87ghz with preset="slower".

The first goes at 7fps, the latter goes at 0.6fps...

Oh that's different. The 79 percent usage is probably due to the cache, but since the data being processed is the same and the method different, 0.6 is slow :), so yes, looks like a problem! Because it is running so slow the cache should be a bit less of a bottleneck as well I would presume. so the usage and effectiveness of more processes is likely a cache bottleneck, but there looks like there is still a huge performance hit there.

Isn't is possible to compare the speed of each function? You know like with x264 and x265 where they compare he speed of the functions:

avx2: 'integral8h' asm code -> 3.69x faster than 'C' version

BIT_DEPTH = 8 : integral_init8h 3.69x 547.65 2020.36
BIT_DEPTH = 10|12 : integral_init8h 2.34x 868.53 2035.82https://bitbucket.org/multicoreware/x265/commits/324ee113f48943791ec295013012d30fcbe16338

Not so much the speed difference, but the number of cycles required for the function.

MysteryX

4th August 2017, 18:43

Measuring the speed of functions or even the cache wouldn't be very useful if I can't change anything about it.

Here's a 1080p video that is challenging for interpolation.
Source: Girls' Day - Female President (https://www.youtube.com/watch?v=v0f9ifrDSp8)

Encoded with x265 preset Medium
FrameRateConverter(Preset="Normal") (https://mega.nz/#!3IghTKyL!fb1VU8BM31JgllvaAkadZ2zInpiPLSPWI4BmDmhMAYw)
FrameRateConverter(Preset="Slower") (https://mega.nz/#!SYxiSa7A!GS1pYarcH9nSbmlTkAFRj2g2nxlyiEMI9oSeA4Q_McU)

Even with Preset="Normal", artifacts are minor. Slower looks better but both are totally acceptable.

Another problem though: playback is laggy! It seems the decoder isn't efficient enough to decode a 60fps x265 video.

Since I can't get a proper playback of the whole video, it's hard to see whether there is any misplaced frame.

Now I'm running only Interframe with x264 output medium preset to see more of how Avisynth behaves; MT=8. It started at 100% CPU, then 50-70% @ 1.87ghz, and now runs at 50% @ 1.18ghz. So there's a bottleneck that isn't specific to MvTools2.

Interesting to note that FrameRateConverter Normal Preset over 4 parallel instances runs faster than Interframe over 8 threads with GPU acceleration!! 7.6fps vs 6.5fps. Actually it peaks to 9.6fps during processing so it's considerably faster.

Videos encoded again with x264 preset medium
Interframe(Preset="Smooth") (https://mega.nz/#!aAB3gQaC!I-ZKY_xFhrD7oauiKxOqsOngcp1CNmWhliyiJsT62jo)
FrameRateConverter(Preset="Normal") (https://mega.nz/#!WZIDCbqQ!xfJB1j_SP4aRr3rMm2x3A6f2eh1K0u_yv6LP8P46EmU)
FrameRateConverter(Preset="Slower") (https://mega.nz/#!nERUwLIB!ebjGFO7Et4j0x8OzkazDe_H5wwDmgWCnb5UHmikHLLU)

burfadel

4th August 2017, 20:13

GPU acceleration is limited by the requirement of copying data to and from the GPU. Processing is likely much faster, but the copying is limitive. Even worse if you use multiple filters like knlmeanscl , avisynth shader etc. Ideally it would be done in one pass.

What other things have you in the script? What about just source filter and framerateconverter? Does prefetch(4) etc change speed? What about setting nice filter for removegrain (it is safe with rgtools)?

MysteryX

4th August 2017, 21:40

Here I have *only* FrameRateConverter. Prefetch wasn't working well, but perhaps running 4 instances with Prefetch(2) would give good results, I'll test that. I don't think RemoveGrain makes any difference at all, it's a very lightweight filter.

Groucho2004

4th August 2017, 21:49

As for KNLMeansCL, not using "MT_SERIALIZED" mode simply multiplies GPU memory usage by the number of threads set in "Prefetch()" and actually slows the filter (and therefore whole script) down. So, if I use MT, my script would look something like this:
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
SetFilterMTMode("knlmeanscl", MT_SERIALIZED)
DGSource("source")
Filter1
KNLMeansCL()
Filter3
Prefetch(4)

I could well imagine that other GPU filters behave similarly.

MysteryX

4th August 2017, 22:42

As of right now, MT_SERIALIZED doesn't work correctly with AVS+ and KNLMeans must run as MT_MULTI_INSTANCE -- otherwise you occasionally get some data mixed up between frames. But the 1080p is purely about FrameRateConverter and has no KNLMeansCL nor anything else.

Preset=Normal with 4 instances of Prefetch(2) gives 100% CPU usage @ 2.38ghz (haven't left it long enough to see whether it drops later but it did 10% of the video at 12fps).

Preset=Slower with 4 instances of Prefetch(2) doesn't go any faster. Still at 0.5fps, and it will freeze at some point since DCT=1 isn't stable with MT.

The video with Preset=Normal encoded with x264 plays perfectly. No misplaced frame. The one with Preset=Slower will take longer to process, I'll post in the morning.

burfadel

4th August 2017, 23:52

For MT to work on non-core filters, even with prefetch set, don't you have to set the filter mode? Instead of setting default mode, set each filter separately that you want to MT, and leave the ones you don't like KnlmeansCL. You can import this from an avs file, and have ask the filters set that you will likely use so you can use it with any encode. You have to state each function separately., for example removegrain and repair, even though they're from the same dll. Just don't blanket set them as multi instance etc, a whole heap of more modern filters are MT_Nice_Filter like removegrain, repair, all commands in recent Masktools, etc. I have MVtools functions not set, but everything else is (mostly nice filters) apart from the source filter, so MVTools functions for me is effectively single threaded.

MysteryX

5th August 2017, 00:16

Yes I have set all MT configurations in a file called AvisynthMT.avsi

burfadel

5th August 2017, 08:13

Have you tried other DCT modes, 3 or 4 could be beneficial:
3 - adaptive per-block switching from spatial to equal-weighted mixed mode (experimental, a little faster).
4 - adaptive per-block switching from spatial to mixed mode with more weight of DCT (experimental, a little faster).

MysteryX

5th August 2017, 15:54

The DCT=1 video (https://mega.nz/#!nERUwLIB!ebjGFO7Et4j0x8OzkazDe_H5wwDmgWCnb5UHmikHLLU) came out perfect, at 0.6fps over 4 instances. No misplaced frame here. Must identify the cause of misplaced frames...

DCT=3 goes at 1.7fps. DCT=4 also goes at 1.7fps. DCT=4 looks better than DCT=3 ... and honestly, the difference with DCT=1 is very minimal, at 3x the performance on 1080p content

MysteryX

5th August 2017, 20:10

DCT=4 gives nearly as good results as DCT=1. It however also has misplaced frames (consistently 1 to 3 frames per 4m video). With DCT=0, there is no misplaced frame. That's within a complex script with NO multi-threading.

With a plain script that only calls FrameRateConverter, the misplaced frame issue hasn't shown yet.

I'll test the older non-Pinterf version to see whether it has the same issue. Other than that, there's nothing else I can do about this.

burfadel

5th August 2017, 20:25

Thresholds would be different for DCT?

MysteryX

5th August 2017, 22:07

Considering the same performance between DCT 3 and 4, it seems to be using DCT in the same way, but interpreting the results differently

MysteryX

5th August 2017, 22:40

Just did an encoding with DCT=4 with preset=Slow (no diff), and it had no misplaced frame. It seems the issue isn't related to DCT=1, but rather, to the diff calling MvTools2 twice.

Btw, MvTools2 v2.6.0.5 works right out of the box with FrameRateConverter. You just need to specify BlkSize, and rename libfftw3f-3.dll as fftw3.dll.

Next experiments:
- old MvTools2 with preset="slower" (currently running)
- running my encodings with preset="slow" and DCT=4, they should come out fine

MysteryX

6th August 2017, 05:17

As expected.

- v2.6.0.5 with preset Slower did show the same misplaced frames. Note that one of the 4 instances crashed (in ST mode). My software then showed a failure, then it resumed that missing segment and processed it as 4 small segments and the output was fine.

- Preset=Slow does work fine. No misplaced frame.

The issue is thus with the Diff. Any idea what could be the culprit?

## For CalcDiff, calculate a 2nd version and create mask to restore from 2nd version the areas that look better
bak2 = CalcDiff ? MAnalyse(superfilt, isb=true, blksize=DiffBlkSize, blksizev=DiffBlkSizeV, overlap = DiffBlkSize>4?(DiffBlkSize/4+1)/2*2:0, overlapv = DiffBlkSizeV>4?(DiffBlkSizeV/4+1)/2*2:0, search=3, dct=0) : nop
fwd2 = CalcDiff ? MAnalyse(superfilt, isb=false, blksize=DiffBlkSize, blksizev=DiffBlkSizeV, overlap = DiffBlkSize>4?(DiffBlkSize/4+1)/2*2:0, search=3, dct=0) : nop
fwd2 = CalcDiff ? Recalculate ? MRecalculate(super, fwd2, blksize=DiffBlkSize/2, blksizev=DiffBlkSizeV/2, overlap = DiffBlkSize/2>4?(DiffBlkSize/8+1)/2*2:0, overlapv = DiffBlkSizeV/2>4?(DiffBlkSizeV/8+1)/2*2:0, thSAD=100) : fwd : nop
bak2 = CalcDiff ? Recalculate ? MRecalculate(super, bak2, blksize=DiffBlkSize/2, blksizev=DiffBlkSizeV/2, overlap = DiffBlkSize/2>4?(DiffBlkSize/8+1)/2*2:0, overlapv = DiffBlkSizeV/2>4?(DiffBlkSizeV/8+1)/2*2:0, thSAD=100) : bak : nop
Flow2 = CalcDiff ? MFlowFps(C, super, bak2, fwd2, num=NewNum, den=NewDen, blend=false, ml=200, mask=2, thSCD2=255) : nop
# Get raw mask again
EM2 = CalcDiff ? MaskTrh > 0 ? C.ConvertToY8().MMask(bak2, ml=255, kind=1, gamma=1/gam2, ysc=255, thSCD2=255) : Blank : nop
EMfwd2 = CalcDiff ? MaskTrh > 0 ? C.ConvertToY8().MMask(fwd2, ml=255, kind=1, gamma=1/gam2, thSCD2=255) : EM2 : nop
EM2 = CalcDiff ? MaskTrh > 0 ? EM2.Overlay(EMfwd2, opacity=.6, mode="lighten", pc_range=true) : EM2 : nop
EMocc2 = CalcDiff ? MaskOcc > 0 ? C.ConvertToY8().MMask(bak2, ml=MaskOcc, kind=2, gamma=1/gam2, ysc=255, thSCD2=255).mt_inpand() : Blank : nop
EM2 = CalcDiff ? MaskOcc > 0 ? EM2.Overlay(EMocc2, opacity=.4, mode="lighten", pc_range=true) : EM2 : nop
# Get difference mask between two versions
EMdiff = CalcDiff ? mt_lutxy(EM, EM2, "x y -")
\ .BicubicResize(Round(C.Width/BlkSize)*4, Round(C.Height/BlkSizeV)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=2))
\ .mt_binarize(60)
\ .FRC_GaussianBlur42(1.2)
\ .BicubicResize(C.Width, C.Height) : nop
# Apply mask to Flow / EM
EMdiff = CalcDiff ? OutFps ? EMdiff.ChangeFPS(NewNum, NewDen) : EMdiff : nop
Flow = CalcDiff ? mt_merge(Flow, Flow2, EMdiff, luma=true, chroma="process") : Flow
EM = CalcDiff ? mt_merge(EM, EM2, EMdiff, luma=true, chroma="process") : EM

Has anyone ever seen such an issue of randomly misplaced frame? Even without MT

MysteryX

6th August 2017, 16:53

Now THAT is interesting. I replaced the code above with this, essentially removing the 2nd MvTools call, and the first call was using DCT=0

bak2 = bak
fwd2 = fwd
Flow2 = Flow
# Get raw mask again
EM2 = CalcDiff ? MaskTrh > 0 ? C.ConvertToY8().MMask(bak2, ml=255, kind=1, gamma=1/gam2, ysc=255, thSCD2=255) : Blank : nop
EMfwd2 = CalcDiff ? MaskTrh > 0 ? C.ConvertToY8().MMask(fwd2, ml=255, kind=1, gamma=1/gam2, thSCD2=255) : EM2 : nop
EM2 = CalcDiff ? MaskTrh > 0 ? EM2.Overlay(EMfwd2, opacity=.6, mode="lighten", pc_range=true) : EM2 : nop
EMocc2 = CalcDiff ? MaskOcc > 0 ? C.ConvertToY8().MMask(bak2, ml=MaskOcc, kind=2, gamma=1/gam2, ysc=255, thSCD2=255).mt_inpand() : Blank : nop
EM2 = CalcDiff ? MaskOcc > 0 ? EM2.Overlay(EMocc2, opacity=.4, mode="lighten", pc_range=true) : EM2 : nop
# Get difference mask between two versions
EMdiff = CalcDiff ? mt_lutxy(EM, EM2, "x y -")
\ .BicubicResize(Round(C.Width/BlkSize)*4, Round(C.Height/BlkSizeV)*4)
\ .mt_expand(mode= mt_circle(zero=true, radius=2))
\ .mt_binarize(60)
\ .FRC_GaussianBlur42(1.2)
\ .BicubicResize(C.Width, C.Height) : nop
# Apply mask to Flow / EM
EMdiff = CalcDiff ? OutFps ? EMdiff.ChangeFPS(NewNum, NewDen) : EMdiff : nop
Flow = CalcDiff ? mt_merge(Flow, Flow2, EMdiff, luma=true, chroma="process") : Flow
EM = CalcDiff ? mt_merge(EM, EM2, EMdiff, luma=true, chroma="process") : EM

I still get misplaced frames with that code!! It seems the issue isn't with MvTools2 at all after all. Perhaps MaskTools2.

burfadel

6th August 2017, 20:06

Does having multiple commands and references to these, with the same label cause any issues? Like the Flow labels, EM labels etc?

MysteryX

7th August 2017, 00:49

Flow and EM are not labels. They are variables. Variables function in a very clearly defined and predictable way. Otherwise, every other script would produce random results.

I'm still running various tests to try to identify where it works and where it fails.

MysteryX

7th August 2017, 02:00

This thing will drive me crazy.

I replace EMdiff with this and it shows misplaced frames. I also tried replacing mt_lutxy with mt_makediff and it was the same.

EMdiff = mt_lutxy(EM, EM2, "x y -")

I replace EMdiff with this and it shows NO misplaced frames.

EMdiff = BlankClip(C.ConvertToY(), color_yuv=$888888)

!????

and if I just run FrameRateConverter without the full script, then the bug doesn't appear either

MysteryX

7th August 2017, 19:38

I got a freeze with DCT=0 and Prefetch(8), so the freezing issue isn't only with DCT=1

Here's another good one!
https://s2.postimg.org/fuyc16wkl/Bad_Frame.png (https://postimg.org/image/fuyc16wkl/)

Although I merged segments after the freeze, this error appeared way before the freeze occurred, and it didn't interrupt the encoding.

burfadel

7th August 2017, 20:05

I don't use multithreading with mvtools at all, it's the only filter that casuses issues if I have multithreading enabled for it. This is particularly true if I run more than one encoding instance to saturate the CPU, if I do this with multithreading enabled on mvtools after 12 hours or so the computer can become increasingly sluggish. It's not a physical memory issue, I believe it possibly uses up some of the allocated system resources; it's not as simple as a program showing to use 2500 KB of memory and it only using that amount.

It's one of the core important filters for avisynth, at least that's how I see it, so would be nice if someone cluey enough works out and resolves the issue :). Seriously, try setting the MT mode for every filter (MT_NICE_FILTER where possible). All masktools functions as well as RGTools are MT_NICE_FILTER capable. Thankfully you can import the script seeing as ou know, you have to state the MT for each function! However, don't set Default mode in Avisynth, nor set it for mvtools.

Once you do that and run with prefetch(8) etc, see if the encodes exhibit the same behaviour in the freezing and misplaced frames, I suspect it won't.

MysteryX

7th August 2017, 20:28

Here's my AviSynthMT.avsi file containing MT definitions. MvTools2 functions are set as MT_MULTI_INSTANCE
https://pastebin.com/Qh7np4Af

burfadel

7th August 2017, 20:33

Try without MT on MVtools at all, but leave it for the other functions. No MT mode worked ideally for me.

MysteryX

7th August 2017, 20:37

Another weird thing. If I run 4 instances, 3 instances have stable memory usage (551MB, 650MB, 761MB), and the 4th has memory usage bouncing up and down (990MB - 1040MB).

Other interesting observation: if I run a single instance in ST (with Preset="slower"), CPU runs at ~25% (1.0 fps), but frequency stays at 2.38ghz. Whereas when I run 4 instances, CPU runs at 50% (1.6 fps) but with frequency at 1.18ghz.

MysteryX

8th August 2017, 01:32

Some more hints. Running 1 instance in ST mode with preset="slower", it still has misplaced frames. It is thus not related to neither multi-instances nor MT modes.

Furthermore, misplaced frames are NOT random. Previous encoding had misplaced frame at 9050. This encoding has the exact same misplaced frame at 9049. The misplaced frames are always the same, whether I encode the clip as 4 parallel segments or the whole thing at once.

BUT, if I run a preview of the script in MPC-HC, those frames are OK.

Any more ideas? A bag of chips to whoever figures this one out.

Here's the script I'm using that causes the issue. You can use this source file (https://mega.nz/#!OBA0iTiI!D6_larToWlQUu8mPR9y4SrbS6noN8M9oo5PUNL5onQ0).

P="Encoder\"
LoadPlugin(P+"LSMASHSource.dll")
LoadPlugin(P+"masktools2.dll")
LoadPlugin(P+"KNLMeansCL.dll")
LoadPlugin(P+"ConvertStacked.dll")
LoadPlugin(P+"FrameRateConverter.dll")
Import(P+"FrameRateConverter.avsi")
LoadPlugin(P+"MvTools2.dll")
LoadPlugin(P+"RgTools.dll")
Import(P+"ResizeX.avsi")
LoadPlugin(P+"Shader.dll")
Import(P+"Shader.avsi")

file="Meu-Ayw-Tua-Lae-Tur.mpg"
LWLibavVideoSource(file, cache=False)
AudioDub(LWLibavAudioSource(file, cache=False))
Crop(0, 0, -8, -0)
ConvertBits(16)
ConvertToYUV444(chromaresample="Spline36", ChromaInPlacement="MPEG1")
ConvertToStacked()
KNLMeansCL(D=2, A=2, h=1.8, channels="YUV", device_type="GPU", device_id=0, lsb_inout=true)
ConvertFromStacked()
SuperResXBR(5, 1, 0, XbrStr=2.7, XbrSharp=1.3, MatrixIn="Rec601", Engines=1, FormatOut="YV24")
FrameRateConverter(NewNum=60, NewDen=1, Preset="slower")
SuperResXBR(3, 1, 0, XbrStr=2.7, XbrSharp=1.3, fWidth=1004, fHeight=768, fKernel="Bicubic", fB=0, fC=.75, FormatOut="YV12", Engines=1)
ResizeX(996, 768, 0, 0, -8, -0)

SpoCk0nd0pe

8th August 2017, 01:41

I tried to encode Game of thrones with your script using MeGUI on my 3.8 GHz I7 a few days ago. I had to cancel it because it would have taken more then 8 days for one episode :/

Encoding speed was 0.24 fps.

MysteryX

8th August 2017, 01:59

Using the whole script above or just FrameRateConverter? If using the script above you want to tweak it based on source and destination resolution.

MysteryX

8th August 2017, 06:28

poisondeathray

8th August 2017, 07:05

Did you rule out simple things like lsmash as a source filter ?

Take a break and revisit it when you are refreshed .

StainlessS

8th August 2017, 11:47

Might be interesting to see if CacheTest shows up anything strange.
CacheTest:- http://forum.doom9.org/showthread.php?p=1744085#post1744085

EDIT: And fed with AVI so as to take LSmash and ffmpegsource out of picture.

MysteryX

8th August 2017, 14:14

Just tried old MaskTools2 version. Problem persists.

burfadel

8th August 2017, 14:17

ARGG!!

I take out the last 2 lines of the script (after FRC) and the output is good. I instead take out the first SuperResXBR before FRC and the output is still good.

I move FRC to the very end: same misplaced frames again!!

The problem isn't SuperResXbr because it works fine when not using FRC preset=slower. I'm starting to think it might be a bug in the core that causes cache mix-up in very specific scenarios; but then I'm surprise it hasn't happened to anyone else. Or a bug in MaskTools2 perhaps.

I think I found the logic:
- Any full script giving a final outcome will be broken.
- Any test script will work.

Your logic is sound :).

You do have a lot of GPU copies. The data is sent to the GPU, KNLMeansCL, copied back, processed, back to GPU with SuperResXBR, back again, processed with framerateconverter, back to the GPU, SuperResXBR, back again and finalised processing.

Instead of KnlMeansCL, try the following cleaner. It uses mvtools for luma, and fft3dfilter for chroma.

For the latest mClean, please see the following thread:
https://forum.doom9.org/showthread.php?p=1815044#post1815044

MysteryX

8th August 2017, 18:18

Did you rule out simple things like lsmash as a source filter ?

Take a break and revisit it when you are refreshed .
You're either a mad genius or a lucky fool.

Removing lsmash solves the issue.

burfadel

8th August 2017, 18:30

Wow really? Surprisimg how things work!

MysteryX

8th August 2017, 18:44

Instead of KnlMeansCL, try the following cleaner. It uses mvtools for luma, and fft3dfilter for chroma.
Interesting. It supports neither 16-bit nor YV24, but nonetheless I gave it a try in YV12.

Original / KNLMeans(1.8) / MClean(450)
https://s1.postimg.org/cwmtfujbv/Denoise_Orig.png (http://postimg.org/image/cwmtfujbv/) https://s1.postimg.org/9n8e9dv8b/Denoise_KNL.png (http://postimg.org/image/9n8e9dv8b/) https://s1.postimg.org/rtvaguwkb/Denoise_MClean.png (http://postimg.org/image/rtvaguwkb/)

KNLMeans(1.8) gives much blurrier results. Yours does a good job at denoising, although I need slightly higher threshold of 450 instead of 350. It however doesn't help with blocking, so I may have to run a deblocker separately. Also shouldn't you adapt BlkSize based on the video size?

Edit: This updated version of FF3DFilter (https://forum.doom9.org/showthread.php?t=174347) should work with YV24 and 16-bit

burfadel

8th August 2017, 19:46

Interesting. It supports neither 16-bit nor YV24, but nonetheless I gave it a try in YV12.

Original / KNLMeans(1.8) / MClean(450)
https://s1.postimg.org/cwmtfujbv/Denoise_Orig.png (http://postimg.org/image/cwmtfujbv/) https://s1.postimg.org/9n8e9dv8b/Denoise_KNL.png (http://postimg.org/image/9n8e9dv8b/) https://s1.postimg.org/rtvaguwkb/Denoise_MClean.png (http://postimg.org/image/rtvaguwkb/)

KNLMeans(1.8) gives much blurrier results. Yours does a good job at denoising, although I need slightly higher threshold of 450 instead of 350. It however doesn't help with blocking, so I may have to run a deblocker separately. Also shouldn't you adapt BlkSize based on the video size?

Edit: This updated version of FF3DFilter (https://forum.doom9.org/showthread.php?t=174347) should work with YV24 and 16-bit

Yes, there are some improvements that could be made to suit different scenario's, it's there if anyone wants to do that! Ideally the updated Pinterf's fft3dfilter should be used, I didn't even consider to mention that.

It really is effective in removing noise though without the blurring, there are many instances where the clarity difference is more pronounced. The fact it distinguishes between blocking and noise is a good thing because you can apply your preferred deblocking separately and only if required. There are many videos that have noise but not blocking :). The screenshot of yours looks to be more heavy blocking than noise, try it on a clip with noise that doesn't have blocking and the difference between mclean and knlmeanscl is even greater.

Feel free to improve on the script! I'm sure there are changes that could be made to reduce the blocking without the blur that knlmeanscl seems to induce.

MysteryX

8th August 2017, 21:35

Agree. The fact that KNLMeans fixed the blocking issue is "nice", but ideally, we want to fix blocking only if blocking occurs -- otherwise we degrade the overall picture when it's not needed.

Are there cases where you found KNLMeans to still be better than your script?

Perhaps we could use FRC's BlkSize auto-selection for this script.

What is your favorite deblocker (http://avisynth.nl/index.php/External_filters#Deblocking)?

burfadel

8th August 2017, 22:32

I did some pretty heavy testing with it initially, I didn't come across any clip where knlmeanscl worked more effectively. It was also a consideration of gpu power use when also processing chroma, it didn't seem efficient. For blocking, I haven't come across any blocky stuff recently, but when I do I use the modified deblock_qed found here https://forum.doom9.org/showthread.php?p=1767112#post1767112

It's only luma blocking that needs processing, ideally the luma processing code would be borrowed from that deblock_qed modified script and applied, if enabled, to the luma part of mclean script. This would avoid unnecessary processing. The same applies for any other deblocker. Some small amount of blocking difference is probably unavoidable since knlmeanscl seems to blur which gives the illusion of deblocking on flat surfaces. In addition to a deblocker, a deband filter would help clean up the remaining inconsistencies whilst also improving the perceived image quality and avoiding the oversmoothing of those flat areas seen with knlmeanscl.

The auto blocksize for luma and chroma would be great. The chroma blocksize, cblksize could possibly be larger.

If you use a deinterlacer like nnedi3, a simple trick to half the required processing is to separate the fields, select the field that nnedi3 was going to work with, and then use half vertical block sizes. After that you apply nnedi3(dh=true). No point processing a field that you're discarding. Also you aren't interpolating via nnedi3() all that noise. It's why I've left the option for vertical blocks.

burfadel

8th August 2017, 22:58

A mask for the deblock could possibly be beneficial? The mask in the script may need tweaking.

MysteryX

8th August 2017, 23:36

burfadel

9th August 2017, 00:02

In my software, ideally I'd have a simple checkbox for Denoise, Deband, Dering and Degrain (is SMDegrain still useful over Denoise with your script? I found it useful on 1080p camera footage). Nothing wrong with having them as 4 separate calls. Or it could be 1 function that handles it all smoothly and we can turn each on/off with parameters. Last time I did tests for Dering I settled with HQDeringmod as HQDering was leaving flat edges and that one seemed to handle it better.

I'd love to know which functions are better for each type of problems for generic use, it takes a lot of testing. If one function handles it all with the best, that would be very useful.

btw burfadel you always come up with good stuff. I respect that.

Thankyou :)

I forgot about deringing consideration. Nothing wrong with a basically all-in-one cleaning script. The way framerateconverter works the cleaner and more retention of the source the better the results and you get less distortion with movement, so cleaning and framerateconverter work together. Think how mild removegrain as the prefilter helps. Still needs to be separate though, as cleaning is more ideal the earlier in the encode script and framerateconverter later.

EDIT: From memory SMDegrain is quite slow, however there may be elements of the script that could apply to improve the result. FFTFilter3D does an excellent job of chroma, it seems to be much better suited to chroma than luma. The sigma, dehalo, sharpen could be tweaked some more most likely. Any of the deringing, further noise filtering by borrowing from smdegrain, etc, really only needs to apply to luma. Ideally it should run pretty fast as well so it's readily usable, although no harm in having slower presets. I believe mclean is basically a temporal noise remover (chroma maybe temporal+spatial?), the deblocking and deringing additionals would be spatial components. I suspect it's clearer than knlmeansCL for this very reason, as the nature of spatial denoisers is to use surrounding pixels, which will inevitably cause some blurring and loss of very fine detail.

How about some additional masks for blocking and ringing? Ringing occurs next to lines mostly don't they? So if an edge mask is slightly expanded you could apply the dering filter only to those parts of the image. The same applies to the deblocking, if a mask can find the edges of the blocks, which would typically have a luma difference within a certain range, you could apply smoothing just to those edges. If the edge is larger than expected. it could be ignored because it is likely a low-contrast detail edge intead of a block edge. Once the mask is applied, it can be expanded to cover the centre of the blocks. You can then scale the luma of the centre of one block to the other.

Good in theory? Not sure whether it is actually doable or not!