SVP-like frame interpolation?

[chilledinsanity]

I'm almost certain this has been answered elsewhere, but I had difficulty finding an answer. The software SVP (Smooth Video Project) does a pretty impressive job of generating frames based on motion in order to increase the framerate of content, however its focus is for real-time playback. Is there some rough equivalent of this for Avisynth or some other software to generate the frames as a new video; not in real-time, but as some sort of script to be processed?

So say I had a 30fps clip that I wanted to convert to 60fps (or slow down more for slow motion), but instead of doubling frames, I wanted the computer to use motion analysis to take its best guess in generating new ones. How would I go about doing that? Please feel free to dumb this down for me (like posting a sample script), I often get hung up on simple syntax errors.

Thanks in advance.

I think this might be what you're looking for.

[manolito]

Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi

# Motion Protected FPS converter script by johnmeyer from Doom9
# Slightly modified interface by manolito
# Requires MVTools V2 and RemoveGrain
# Also needs fftw3.dll in the System32 or SysWOW64 folder for Dct values other than 0


function jm_fps(clip source, float "fps", int "BlkSize", int "Dct")
{
fps = default(fps, 25.000)
fps_num = int(fps * 1000)
fps_den = 1000
BlkSize = default(BlkSize, 16)
Dct = default(Dct, 0)

prefiltered = RemoveGrain(source, 22)
super = MSuper(source, hpad = 16, vpad = 16, levels = 1, sharp = 1, rfilter = 4) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16, sharp = 1, rfilter = 4) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward = MAnalyse(superfilt, isb = false, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

return out
}

Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.p...31#post1789031


Cheers
manolito

[amayra]

and there this :
https://github.com/gdiaz384/frameTools

[kolak]

Originally Posted by manolito

Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi


Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.p...31#post1789031


Cheers
manolito

DCT=1 will definitely help further, but slow down conversion a lot also

[johnmeyer]

Originally Posted by kolak

DCT=1 will definitely help further, but slow down conversion a lot also

Are you sure? I've never found that it did anything to reduce artifacts when doing slow motion. I find it useful to reduce flicker, if that is a problem.

You are absolutely correct, however, that it slows down the conversion (it is about 5x slower).

[kolak]

It always produced less artefacts for me when doing fps conversion.
DCT=0 is also very bad on fades. You can use other modes if DCT=1 is to slow (eg. DCT=3). DCT=0 is the simplest mode and it's for speed not quality.

[manolito]

This made me curious so I dug out my old fps conversion torture clip.

I converted it to half speed using dct values of 0, 1 and 3. Download the resulting clips here:
http://www23.zippyshare.com/v/L4HbbPzJ/file.html

Conversion speed was 1.3 fps for dct=0, 0.3 fps for dct=1 and 1.0 fps for dct=3.

Speed for dct=1 is forbiddingly slow, there is no way I will ever use this.

And watching the results I have to agree with johnmeyer: Using a dct value of 0 produces less artifacts than using values of 1 or 3, and it is faster.


Cheers
manolito

[CruNcher]

dct 1_test_speed=50%.mkv

most stable temporal result on the first view in many problematic areas, most viewers would rate it the highest MOS of the 3 results 0 and 3 it becomes harder to percept the temporal difference but 1 is pretty obvious.

I would predict everyone of the test viewers with ok visuals would rate dct 1 the most "annoying free result"

That you don't percept the difference let me wonder about your overall script results

Though overall all 3 have to much temporal problems and overall can only be rated BAD by everyone as such

For Marketing i would present it side by side that way the difference would be very fast visible to even a higher amount of viewers and maybe at another 50 slowmo to make it even more obvious


So overall you could also simplify it further in 2 presets

Slow = 1 / Fast = 3

if Slow,Medium,Fast would be usable maybe but for the the overall testcase presented here i would say nope useless

Which makes me wonder how AMDs,Intels and Nvidias FRC would compare here by now vs Mvtools or even Kronos

[kolak]

Originally Posted by manolito

This made me curious so I dug out my old fps conversion torture clip.

I converted it to half speed using dct values of 0, 1 and 3. Download the resulting clips here:
http://www23.zippyshare.com/v/L4HbbPzJ/file.html

Conversion speed was 1.3 fps for dct=0, 0.3 fps for dct=1 and 1.0 fps for dct=3.

Speed for dct=1 is forbiddingly slow, there is no way I will ever use this.

And watching the results I have to agree with johnmeyer: Using a dct value of 0 produces less artifacts than using values of 1 or 3, and it is faster. I know DCT=0 will fail me on fades.


Cheers
manolito

That's something odd if DCT=0 produces less artefacts (also- try fades with DCT=0).

This is definitely not a case for me, but I'm using vs (but this should not matter). I also used different/simpler script, so maybe this is the reason (again shouldn't be).

I've converted 500h worth of footage and done few days testing before this. Used DCT=3 at the end as a compromise.

I will try this script, maybe it has some magic I have to find average settings for 500h worth of footage not ones which will work for e.g. 1min clip. I know DCT=0 will fail me on fades.

[manolito]

Spent some more time tuning the script parameters for this particular test clip. Please keep in mind that this clip is extremely demanding, usually I would never recommend to use this script on anime.

What I found out is that a dct value of 3 does give better results on the control panel to the left, but it does not improve the vertical grille to the right. What does improve this grille is using a larger block size of 32 for MAnalyze. For MRecalculate I kept the original block size of 8, increasing it to 16 gave worse results.

So for this particular clip my preferred settings are dct =3 and blksize = 32 for MAnalyse. The resulting file is here:
http://www59.zippyshare.com/v/l6Ouh8fw/file.html

In my experience any settings which work for this clip will also give good results for other sources. I need to test if the increased block size will indeed work for HD sources...


Cheers
manolito

[kolak]

I quickly tested this script on one of my samples and it's not any better than my one. Some frames are better, others worse.
I use block 32 with overlap 8 as this is better (again- for me) on HD sources. My sources are real videos- TV stuff.
I also asked jackoneill to implement even bigger block sizes (for vs mvtools), but this seams to not improve quality as much as I expected. There are some other issues with this bigger block sizes which seams to be related to mvtools internals.

[chilledinsanity]

Thanks, I'll experiment with this.

[CruNcher]

Originally Posted by manolito

Spent some more time tuning the script parameters for this particular test clip. Please keep in mind that this clip is extremely demanding, usually I would never recommend to use this script on anime.

What I found out is that a dct value of 3 does give better results on the control panel to the left, but it does not improve the vertical grille to the right. What does improve this grille is using a larger block size of 32 for MAnalyze. For MRecalculate I kept the original block size of 8, increasing it to 16 gave worse results.

So for this particular clip my preferred settings are dct =3 and blksize = 32 for MAnalyse. The resulting file is here:
http://www59.zippyshare.com/v/l6Ouh8fw/file.html

In my experience any settings which work for this clip will also give good results for other sources. I need to test if the increased block size will indeed work for HD sources...


Cheers
manolito

This version introduced many new perceptable problems though you fought 1 problem and created a whole bunch of new ones which would most probably result now in a even lower mos score then before (predicted).

Though this indicates their must be a better result hiding in between

if you split all the best parts and put them back together you would have a overall better result, from the timing this could even work.


Temporal i can split this in 3 different parts that show different problems

Wherby the amount of problems are different from each case.

In this test case you could say you have 3 scene parts Beginning,Middle and the End part

and depending on your setup now they come out different the beginning part never completely fixed the middle and end part with some very good overall results except in your latest result the end part comes out perceptually much worse then in any result before, therfore though the middle part is perceptually better (no fast texture failure).

Logic now says take all the good parts and put them together and you have a pretty good overall result.

Which also could indicate you need something more adaptively clever to get this result in 1 try,without putting it manually together


My reasoning still stays dct 1 gives the overall best result except 1 difference now to the latest output therefore 1 part in the latest output is now totally ending up bad perceptually.

i would also weight the temporal failure of the latest result in the end part higher even critical now then the small problem in the middle part on the dct 1 result before (fast texture failure).

[manolito]

Thanks for looking into it...

I need to test if the increased block size will indeed work for HD sources...

Well, I found that it does not...

The best results for this clip came from using dct=1, but this is so slow that it is not really usable for me.

Of course you can optimize the results by finetuning the params for each and every source, maybe even split the source into several parts. But this is not what I'm after. I want a script which works well for the vast majority of sources without tweaking the params.

And I believe that johnmeyer's original params do an excellent job for most sources (maybe use dct=3 instead of 0). Way better than MotionProtectedFPS. And keep in mind that I am not interested in watching still frames. I want to just watch the movie, and it should look better than using ChangeFPS or ConvertFPS. I do not mind some motion artifacts as long as they do not get too annoying. And I use this only on film, not on anime. Plus I mostly use this type of fps conversion for PAL <-> NTSC conversion, not for slow motion.


Cheers
manolito

[CruNcher]

yes but overall this result could endup in a catastrophe now if it creates the same temporal issues for every source that heavy perceptual jitter is really annoying for moving objects like those angels it destroys everything and i really wonder how you can percept it now as better then anything before.

you fixed the texture problem yes but this is hyper annoying now in motion compared to that small glitch that was only visible mere milliseconds before.

Good thing in the end you at last saw the problems on the panel with the light scan

Also there are much much more problems to think about you using AVC currently as output you don't know yet how this will be perceived actually with HEVC/VP9 for example it could come out even worse (better perceptible).

It partly though could be also get fixed in the other direction as internaly a hevc encoder is smarter then mvtools and you might throw out extra resources for nothing

[manolito]

I don't know but I think you are being a little too critical. I showed the converted clips to some of my friends who are journalists working for Deutsche Welle TV. They do have trained eyes, but just like me they focus on the overall visual impression, and all of them agreed that the MVTools based conversions were quite pleasing to watch.

For this particular clip my ultimate conversion is here:
http://www82.zippyshare.com/v/7bRTiwej/file.html

dct=1 and blocksize for MAnalyse = 32. Way too slow, but I cannot find any annoying flaws.

My main problem with these fps conversion methods are hard coded subs or movie credits. The letters mostly get warped to a point where they are painful to watch. But this is the only occasion where I would consider splitting up the source and convert the different parts separately...


Cheers
manolito

[kolak]

MvTools are almost as good as Alchemist or Tachyon which are used in braodcast and cost small fortune.
It just would be nice to have it ported to GPU, like svp does. SVP made to many speed shortcuts so quality is not as good.

[CruNcher]

Originally Posted by manolito

I don't know but I think you are being a little too critical. I showed the converted clips to some of my friends who are journalists working for Deutsche Welle TV. They do have trained eyes, but just like me they focus on the overall visual impression, and all of them agreed that the MVTools based conversions were quite pleasing to watch.

For this particular clip my ultimate conversion is here:
http://www82.zippyshare.com/v/7bRTiwej/file.html

dct=1 and blocksize for MAnalyse = 32. Way too slow, but I cannot find any annoying flaws.

My main problem with these fps conversion methods are hard coded subs or movie credits. The letters mostly get warped to a point where they are painful to watch. But this is the only occasion where I would consider splitting up the source and convert the different parts separately...


Cheers
manolito

Slowly we getting there
beginning = nice
middle = nice (with a small new glitch instead of the complete texture fail we see a big shift now)
end = horrible (full of motion perceptible fails, heat haze effect)

[manolito]

end = horrible (full of motion perceptible fails, heat haze effect)

Are you talking about the three small angels in the background?

The human brain works differently. Everybody just sees the two large angels in the foreground, and they do not display motion artifacts. The information in the background is pretty much discarded because the brain determines that this information is not important. (This is gone by the "Formatio Reticularis" which works like a spotlight).


Cheers
manolito