Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 10th March 2017, 13:22   #1  |  Link
chilledinsanity
Registered User
 
Join Date: Jan 2002
Posts: 226
SVP-like frame interpolation?

I'm almost certain this has been answered elsewhere, but I had difficulty finding an answer. The software SVP (Smooth Video Project) does a pretty impressive job of generating frames based on motion in order to increase the framerate of content, however its focus is for real-time playback. Is there some rough equivalent of this for Avisynth or some other software to generate the frames as a new video; not in real-time, but as some sort of script to be processed?

So say I had a 30fps clip that I wanted to convert to 60fps (or slow down more for slow motion), but instead of doubling frames, I wanted the computer to use motion analysis to take its best guess in generating new ones. How would I go about doing that? Please feel free to dumb this down for me (like posting a sample script), I often get hung up on simple syntax errors.

Thanks in advance.
chilledinsanity is offline   Reply With Quote
Old 10th March 2017, 13:32   #2  |  Link
thecoreyburton
Guest
 
Posts: n/a
I think this might be what you're looking for.
  Reply With Quote
Old 10th March 2017, 14:48   #3  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi
Quote:
# Motion Protected FPS converter script by johnmeyer from Doom9
# Slightly modified interface by manolito
# Requires MVTools V2 and RemoveGrain
# Also needs fftw3.dll in the System32 or SysWOW64 folder for Dct values other than 0


function jm_fps(clip source, float "fps", int "BlkSize", int "Dct")
{
fps = default(fps, 25.000)
fps_num = int(fps * 1000)
fps_den = 1000
BlkSize = default(BlkSize, 16)
Dct = default(Dct, 0)

prefiltered = RemoveGrain(source, 22)
super = MSuper(source, hpad = 16, vpad = 16, levels = 1, sharp = 1, rfilter = 4) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16, sharp = 1, rfilter = 4) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward = MAnalyse(superfilt, isb = false, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

return out
}
Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.p...31#post1789031


Cheers
manolito

Last edited by manolito; 13th May 2018 at 21:34. Reason: improved params for MSuper
manolito is offline   Reply With Quote
Old 10th March 2017, 21:06   #4  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Quote:
Originally Posted by manolito View Post
Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi


Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.p...31#post1789031


Cheers
manolito
DCT=1 will definitely help further, but slow down conversion a lot also
kolak is offline   Reply With Quote
Old 10th March 2017, 22:07   #5  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Quote:
Originally Posted by kolak View Post
DCT=1 will definitely help further, but slow down conversion a lot also
Are you sure? I've never found that it did anything to reduce artifacts when doing slow motion. I find it useful to reduce flicker, if that is a problem.

You are absolutely correct, however, that it slows down the conversion (it is about 5x slower).
johnmeyer is offline   Reply With Quote
Old 24th March 2017, 17:46   #6  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,559
Quote:
Originally Posted by manolito View Post
Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi


Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.p...31#post1789031


Cheers
manolito
How does this script compare with InterFrame?
MysteryX is offline   Reply With Quote
Old 24th March 2017, 18:50   #7  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Quote:
Originally Posted by MysteryX View Post
How does this script compare with InterFrame?
For the answer, read my post here.

That post also explains why some in this thread thought my script provided some good results: I was able to tweak some things which are not "exposed" with SVP and its Interframe front end.

As for things said in this thread, I still am not convinced that any of the DCT settings are going to provide any substantial, real improvements, i.e., it won't produce differences that actually matter. Note I didn't say there wouldn't be differences, only that those differences won't really get at the reasons why motion estimation fails when doing frame rate changes or other operations which involve creating new frames from adjacent frames.

The real issue is how to define "objects" and how to track them. The motion estimation done by any of these MVTools-derived filters relies on nothing more than tracking pre-determined blocks of pixels rather than pre-identifying actual objects in the frame. This is why block size is the most important variable to change when trying to get good results. Depending on the video and the size of the things being tracked (like people's legs, vertical fence posts, and other difficult-to-track items), different block sizes will work on some videos better than others. I find a block size of 16 to be a good starting point, but sometimes find 8 or 32 works better. The block overlap can then provide some fine tuning.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.

So, motion estimation is not a "set it, and forget it" tool, and if you use it that way for creating new frames, you will get burned, and it will be sooner rather than later.
johnmeyer is offline   Reply With Quote
Old 25th March 2017, 07:24   #8  |  Link
CruNcher
Registered User
 
CruNcher's Avatar
 
Join Date: Apr 2002
Location: Germany
Posts: 4,926
Quote:
Originally Posted by johnmeyer View Post
For the answer, read my post here.

That post also explains why some in this thread thought my script provided some good results: I was able to tweak some things which are not "exposed" with SVP and its Interframe front end.

As for things said in this thread, I still am not convinced that any of the DCT settings are going to provide any substantial, real improvements, i.e., it won't produce differences that actually matter. Note I didn't say there wouldn't be differences, only that those differences won't really get at the reasons why motion estimation fails when doing frame rate changes or other operations which involve creating new frames from adjacent frames.

The real issue is how to define "objects" and how to track them. The motion estimation done by any of these MVTools-derived filters relies on nothing more than tracking pre-determined blocks of pixels rather than pre-identifying actual objects in the frame. This is why block size is the most important variable to change when trying to get good results. Depending on the video and the size of the things being tracked (like people's legs, vertical fence posts, and other difficult-to-track items), different block sizes will work on some videos better than others. I find a block size of 16 to be a good starting point, but sometimes find 8 or 32 works better. The block overlap can then provide some fine tuning.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.

So, motion estimation is not a "set it, and forget it" tool, and if you use it that way for creating new frames, you will get burned, and it will be sooner rather than later.
Exactly and in this case the Quantization noise from the lossy MPEG Input in manolitos test conversion case causes some of the tracking problems.
__________________
all my compares are riddles so please try to decipher them yourselves :)

It is about Time

Join the Revolution NOW before it is to Late !

http://forum.doom9.org/showthread.php?t=168004

Last edited by CruNcher; 25th March 2017 at 07:27.
CruNcher is offline   Reply With Quote
Old 28th March 2017, 03:32   #9  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,559
I have made a quick test to compare Interframe(preset="smooth") with jm_fps. This is part of a script that does more processing, and does an extra frame double which makes the changes more visible. This is a test on very bad quality content to see how it behaves.

http://screenshotcomparison.com/comparison/204832

http://screenshotcomparison.com/comparison/204833

The difference is huge. It's much better than I expected. Worth investigating some more. Plus, it allows processing in YV24.

Can the jm_fps script work in 16-bit, using either DitherTools hack or native AVS+ format?

I did further tests in YV24, and strangely, it generates a LOT of strong artifacts!! This seems more like a bug.
http://screenshotcomparison.com/comparison/204834

Last edited by MysteryX; 28th March 2017 at 03:46.
MysteryX is offline   Reply With Quote
Old 10th March 2017, 17:12   #10  |  Link
amayra
Quality Checker
 
amayra's Avatar
 
Join Date: Aug 2013
Posts: 284
and there this :
https://github.com/gdiaz384/frameTools
__________________
I love Doom9
amayra is offline   Reply With Quote
Old 11th March 2017, 00:17   #11  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
It always produced less artefacts for me when doing fps conversion.
DCT=0 is also very bad on fades. You can use other modes if DCT=1 is to slow (eg. DCT=3). DCT=0 is the simplest mode and it's for speed not quality.
kolak is offline   Reply With Quote
Old 11th March 2017, 09:26   #12  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
This made me curious so I dug out my old fps conversion torture clip.

I converted it to half speed using dct values of 0, 1 and 3. Download the resulting clips here:
http://www23.zippyshare.com/v/L4HbbPzJ/file.html

Conversion speed was 1.3 fps for dct=0, 0.3 fps for dct=1 and 1.0 fps for dct=3.

Speed for dct=1 is forbiddingly slow, there is no way I will ever use this.

And watching the results I have to agree with johnmeyer: Using a dct value of 0 produces less artifacts than using values of 1 or 3, and it is faster.


Cheers
manolito
manolito is offline   Reply With Quote
Old 11th March 2017, 14:35   #13  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Quote:
Originally Posted by manolito View Post
This made me curious so I dug out my old fps conversion torture clip.

I converted it to half speed using dct values of 0, 1 and 3. Download the resulting clips here:
http://www23.zippyshare.com/v/L4HbbPzJ/file.html

Conversion speed was 1.3 fps for dct=0, 0.3 fps for dct=1 and 1.0 fps for dct=3.

Speed for dct=1 is forbiddingly slow, there is no way I will ever use this.

And watching the results I have to agree with johnmeyer: Using a dct value of 0 produces less artifacts than using values of 1 or 3, and it is faster. I know DCT=0 will fail me on fades.


Cheers
manolito
That's something odd if DCT=0 produces less artefacts (also- try fades with DCT=0).

This is definitely not a case for me, but I'm using vs (but this should not matter). I also used different/simpler script, so maybe this is the reason (again shouldn't be).

I've converted 500h worth of footage and done few days testing before this. Used DCT=3 at the end as a compromise.

I will try this script, maybe it has some magic I have to find average settings for 500h worth of footage not ones which will work for e.g. 1min clip. I know DCT=0 will fail me on fades.

Last edited by kolak; 11th March 2017 at 15:27.
kolak is offline   Reply With Quote
Old 11th March 2017, 19:39   #14  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Spent some more time tuning the script parameters for this particular test clip. Please keep in mind that this clip is extremely demanding, usually I would never recommend to use this script on anime.

What I found out is that a dct value of 3 does give better results on the control panel to the left, but it does not improve the vertical grille to the right. What does improve this grille is using a larger block size of 32 for MAnalyze. For MRecalculate I kept the original block size of 8, increasing it to 16 gave worse results.

So for this particular clip my preferred settings are dct =3 and blksize = 32 for MAnalyse. The resulting file is here:
http://www59.zippyshare.com/v/l6Ouh8fw/file.html

In my experience any settings which work for this clip will also give good results for other sources. I need to test if the increased block size will indeed work for HD sources...


Cheers
manolito

Last edited by manolito; 11th March 2017 at 20:17.
manolito is offline   Reply With Quote
Old 11th March 2017, 20:29   #15  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
I quickly tested this script on one of my samples and it's not any better than my one. Some frames are better, others worse.
I use block 32 with overlap 8 as this is better (again- for me) on HD sources. My sources are real videos- TV stuff.
I also asked jackoneill to implement even bigger block sizes (for vs mvtools), but this seams to not improve quality as much as I expected. There are some other issues with this bigger block sizes which seams to be related to mvtools internals.

Last edited by kolak; 12th March 2017 at 16:11.
kolak is offline   Reply With Quote
Old 18th March 2017, 11:20   #16  |  Link
CruNcher
Registered User
 
CruNcher's Avatar
 
Join Date: Apr 2002
Location: Germany
Posts: 4,926
Quote:
Originally Posted by manolito View Post
Spent some more time tuning the script parameters for this particular test clip. Please keep in mind that this clip is extremely demanding, usually I would never recommend to use this script on anime.

What I found out is that a dct value of 3 does give better results on the control panel to the left, but it does not improve the vertical grille to the right. What does improve this grille is using a larger block size of 32 for MAnalyze. For MRecalculate I kept the original block size of 8, increasing it to 16 gave worse results.

So for this particular clip my preferred settings are dct =3 and blksize = 32 for MAnalyse. The resulting file is here:
http://www59.zippyshare.com/v/l6Ouh8fw/file.html

In my experience any settings which work for this clip will also give good results for other sources. I need to test if the increased block size will indeed work for HD sources...


Cheers
manolito
This version introduced many new perceptable problems though you fought 1 problem and created a whole bunch of new ones which would most probably result now in a even lower mos score then before (predicted).

Though this indicates their must be a better result hiding in between

if you split all the best parts and put them back together you would have a overall better result, from the timing this could even work.


Temporal i can split this in 3 different parts that show different problems

Wherby the amount of problems are different from each case.

In this test case you could say you have 3 scene parts Beginning,Middle and the End part

and depending on your setup now they come out different the beginning part never completely fixed the middle and end part with some very good overall results except in your latest result the end part comes out perceptually much worse then in any result before, therfore though the middle part is perceptually better (no fast texture failure).

Logic now says take all the good parts and put them together and you have a pretty good overall result.

Which also could indicate you need something more adaptively clever to get this result in 1 try,without putting it manually together


My reasoning still stays dct 1 gives the overall best result except 1 difference now to the latest output therefore 1 part in the latest output is now totally ending up bad perceptually.

i would also weight the temporal failure of the latest result in the end part higher even critical now then the small problem in the middle part on the dct 1 result before (fast texture failure).
__________________
all my compares are riddles so please try to decipher them yourselves :)

It is about Time

Join the Revolution NOW before it is to Late !

http://forum.doom9.org/showthread.php?t=168004

Last edited by CruNcher; 18th March 2017 at 12:17.
CruNcher is offline   Reply With Quote
Old 11th March 2017, 11:30   #17  |  Link
CruNcher
Registered User
 
CruNcher's Avatar
 
Join Date: Apr 2002
Location: Germany
Posts: 4,926
dct 1_test_speed=50%.mkv

most stable temporal result on the first view in many problematic areas, most viewers would rate it the highest MOS of the 3 results 0 and 3 it becomes harder to percept the temporal difference but 1 is pretty obvious.

I would predict everyone of the test viewers with ok visuals would rate dct 1 the most "annoying free result"

That you don't percept the difference let me wonder about your overall script results

Though overall all 3 have to much temporal problems and overall can only be rated BAD by everyone as such

For Marketing i would present it side by side that way the difference would be very fast visible to even a higher amount of viewers and maybe at another 50 slowmo to make it even more obvious


So overall you could also simplify it further in 2 presets

Slow = 1 / Fast = 3

if Slow,Medium,Fast would be usable maybe but for the the overall testcase presented here i would say nope useless

Which makes me wonder how AMDs,Intels and Nvidias FRC would compare here by now vs Mvtools or even Kronos
__________________
all my compares are riddles so please try to decipher them yourselves :)

It is about Time

Join the Revolution NOW before it is to Late !

http://forum.doom9.org/showthread.php?t=168004

Last edited by CruNcher; 11th March 2017 at 13:12.
CruNcher is offline   Reply With Quote
Old 18th March 2017, 10:24   #18  |  Link
chilledinsanity
Registered User
 
Join Date: Jan 2002
Posts: 226
Thanks, I'll experiment with this.
chilledinsanity is offline   Reply With Quote
Old 18th March 2017, 12:23   #19  |  Link
CruNcher
Registered User
 
CruNcher's Avatar
 
Join Date: Apr 2002
Location: Germany
Posts: 4,926
yes but overall this result could endup in a catastrophe now if it creates the same temporal issues for every source that heavy perceptual jitter is really annoying for moving objects like those angels it destroys everything and i really wonder how you can percept it now as better then anything before.

you fixed the texture problem yes but this is hyper annoying now in motion compared to that small glitch that was only visible mere milliseconds before.

Good thing in the end you at last saw the problems on the panel with the light scan

Also there are much much more problems to think about you using AVC currently as output you don't know yet how this will be perceived actually with HEVC/VP9 for example it could come out even worse (better perceptible).

It partly though could be also get fixed in the other direction as internaly a hevc encoder is smarter then mvtools and you might throw out extra resources for nothing
__________________
all my compares are riddles so please try to decipher them yourselves :)

It is about Time

Join the Revolution NOW before it is to Late !

http://forum.doom9.org/showthread.php?t=168004

Last edited by CruNcher; 18th March 2017 at 13:00.
CruNcher is offline   Reply With Quote
Old 19th March 2017, 15:06   #20  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
I don't know but I think you are being a little too critical. I showed the converted clips to some of my friends who are journalists working for Deutsche Welle TV. They do have trained eyes, but just like me they focus on the overall visual impression, and all of them agreed that the MVTools based conversions were quite pleasing to watch.

For this particular clip my ultimate conversion is here:
http://www82.zippyshare.com/v/7bRTiwej/file.html

dct=1 and blocksize for MAnalyse = 32. Way too slow, but I cannot find any annoying flaws.

My main problem with these fps conversion methods are hard coded subs or movie credits. The letters mostly get warped to a point where they are painful to watch. But this is the only occasion where I would consider splitting up the source and convert the different parts separately...


Cheers
manolito
manolito is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.