SVP-like frame interpolation?

[chilledinsanity]

I'm almost certain this has been answered elsewhere, but I had difficulty finding an answer. The software SVP (Smooth Video Project) does a pretty impressive job of generating frames based on motion in order to increase the framerate of content, however its focus is for real-time playback. Is there some rough equivalent of this for Avisynth or some other software to generate the frames as a new video; not in real-time, but as some sort of script to be processed?

So say I had a 30fps clip that I wanted to convert to 60fps (or slow down more for slow motion), but instead of doubling frames, I wanted the computer to use motion analysis to take its best guess in generating new ones. How would I go about doing that? Please feel free to dumb this down for me (like posting a sample script), I often get hung up on simple syntax errors.

Thanks in advance.

[thecoreyburton]

I think this (https://forum.doom9.org/showthread.php?t=160226) might be what you're looking for.

[manolito]

Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi
# Motion Protected FPS converter script by johnmeyer from Doom9
# Slightly modified interface by manolito
# Requires MVTools V2 and RemoveGrain
# Also needs fftw3.dll in the System32 or SysWOW64 folder for Dct values other than 0


function jm_fps(clip source, float "fps", int "BlkSize", int "Dct")
{
fps = default(fps, 25.000)
fps_num = int(fps * 1000)
fps_den = 1000
BlkSize = default(BlkSize, 16)
Dct = default(Dct, 0)

prefiltered = RemoveGrain(source, 22)
super = MSuper(source, hpad = 16, vpad = 16, levels = 1, sharp = 1, rfilter = 4) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16, sharp = 1, rfilter = 4) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward = MAnalyse(superfilt, isb = false, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

return out
}

Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.php?p=1789031#post1789031


Cheers
manolito

[amayra]

and there this :
https://github.com/gdiaz384/frameTools

[kolak]

Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi


Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.php?p=1789031#post1789031


Cheers
manolito

DCT=1 will definitely help further, but slow down conversion a lot also :)

[johnmeyer]

DCT=1 will definitely help further, but slow down conversion a lot also :)Are you sure? I've never found that it did anything to reduce artifacts when doing slow motion. I find it useful to reduce flicker, if that is a problem.

You are absolutely correct, however, that it slows down the conversion (it is about 5x slower).

[kolak]

It always produced less artefacts for me when doing fps conversion.
DCT=0 is also very bad on fades. You can use other modes if DCT=1 is to slow (eg. DCT=3). DCT=0 is the simplest mode and it's for speed not quality.

[manolito]

This made me curious so I dug out my old fps conversion torture clip.

I converted it to half speed using dct values of 0, 1 and 3. Download the resulting clips here:
http://www23.zippyshare.com/v/L4HbbPzJ/file.html

Conversion speed was 1.3 fps for dct=0, 0.3 fps for dct=1 and 1.0 fps for dct=3.

Speed for dct=1 is forbiddingly slow, there is no way I will ever use this.

And watching the results I have to agree with johnmeyer: Using a dct value of 0 produces less artifacts than using values of 1 or 3, and it is faster.


Cheers
manolito

[CruNcher]

dct 1_test_speed=50%.mkv

most stable temporal result on the first view in many problematic areas, most viewers would rate it the highest MOS of the 3 results 0 and 3 it becomes harder to percept the temporal difference but 1 is pretty obvious.

I would predict everyone of the test viewers with ok visuals would rate dct 1 the most "annoying free result"

That you don't percept the difference let me wonder about your overall script results ;)

Though overall all 3 have to much temporal problems and overall can only be rated BAD by everyone as such ;)

For Marketing i would present it side by side that way the difference would be very fast visible to even a higher amount of viewers and maybe at another 50 slowmo to make it even more obvious ;)


So overall you could also simplify it further in 2 presets

Slow = 1 / Fast = 3

if Slow,Medium,Fast would be usable maybe but for the the overall testcase presented here i would say nope useless

Which makes me wonder how AMDs,Intels and Nvidias FRC would compare here by now vs Mvtools or even Kronos :)

[kolak]

This made me curious so I dug out my old fps conversion torture clip.

I converted it to half speed using dct values of 0, 1 and 3. Download the resulting clips here:
http://www23.zippyshare.com/v/L4HbbPzJ/file.html

Conversion speed was 1.3 fps for dct=0, 0.3 fps for dct=1 and 1.0 fps for dct=3.

Speed for dct=1 is forbiddingly slow, there is no way I will ever use this.

And watching the results I have to agree with johnmeyer: Using a dct value of 0 produces less artifacts than using values of 1 or 3, and it is faster. I know DCT=0 will fail me on fades.


Cheers
manolito

That's something odd if DCT=0 produces less artefacts (also- try fades with DCT=0).

This is definitely not a case for me, but I'm using vs (but this should not matter). I also used different/simpler script, so maybe this is the reason (again shouldn't be).

I've converted 500h worth of footage and done few days testing before this. Used DCT=3 at the end as a compromise.

I will try this script, maybe it has some magic :) I have to find average settings for 500h worth of footage not ones which will work for e.g. 1min clip. I know DCT=0 will fail me on fades.

[manolito]

Spent some more time tuning the script parameters for this particular test clip. Please keep in mind that this clip is extremely demanding, usually I would never recommend to use this script on anime.

What I found out is that a dct value of 3 does give better results on the control panel to the left, but it does not improve the vertical grille to the right. What does improve this grille is using a larger block size of 32 for MAnalyze. For MRecalculate I kept the original block size of 8, increasing it to 16 gave worse results.

So for this particular clip my preferred settings are dct =3 and blksize = 32 for MAnalyse. The resulting file is here:
http://www59.zippyshare.com/v/l6Ouh8fw/file.html

In my experience any settings which work for this clip will also give good results for other sources. I need to test if the increased block size will indeed work for HD sources... ;)


Cheers
manolito

[kolak]

I quickly tested this script on one of my samples and it's not any better than my one. Some frames are better, others worse.
I use block 32 with overlap 8 as this is better (again- for me) on HD sources. My sources are real videos- TV stuff.
I also asked jackoneill to implement even bigger block sizes (for vs mvtools), but this seams to not improve quality as much as I expected. There are some other issues with this bigger block sizes which seams to be related to mvtools internals.

[chilledinsanity]

Thanks, I'll experiment with this.

[CruNcher]

Spent some more time tuning the script parameters for this particular test clip. Please keep in mind that this clip is extremely demanding, usually I would never recommend to use this script on anime.

What I found out is that a dct value of 3 does give better results on the control panel to the left, but it does not improve the vertical grille to the right. What does improve this grille is using a larger block size of 32 for MAnalyze. For MRecalculate I kept the original block size of 8, increasing it to 16 gave worse results.

So for this particular clip my preferred settings are dct =3 and blksize = 32 for MAnalyse. The resulting file is here:
http://www59.zippyshare.com/v/l6Ouh8fw/file.html

In my experience any settings which work for this clip will also give good results for other sources. I need to test if the increased block size will indeed work for HD sources... ;)


Cheers
manolito

This version introduced many new perceptable problems though you fought 1 problem and created a whole bunch of new ones which would most probably result now in a even lower mos score then before (predicted).

Though this indicates their must be a better result hiding in between :)

if you split all the best parts and put them back together you would have a overall better result, from the timing this could even work.


Temporal i can split this in 3 different parts that show different problems

Wherby the amount of problems are different from each case.

In this test case you could say you have 3 scene parts Beginning,Middle and the End part

and depending on your setup now they come out different the beginning part never completely fixed the middle and end part with some very good overall results except in your latest result the end part comes out perceptually much worse then in any result before, therfore though the middle part is perceptually better (no fast texture failure).

Logic now says take all the good parts and put them together and you have a pretty good overall result.

Which also could indicate you need something more adaptively clever to get this result in 1 try,without putting it manually together ;)


My reasoning still stays dct 1 gives the overall best result except 1 difference now to the latest output therefore 1 part in the latest output is now totally ending up bad perceptually.

i would also weight the temporal failure of the latest result in the end part higher even critical now then the small problem in the middle part on the dct 1 result before (fast texture failure).

[manolito]

Thanks for looking into it...

I need to test if the increased block size will indeed work for HD sources...

Well, I found that it does not... :devil:

The best results for this clip came from using dct=1, but this is so slow that it is not really usable for me.

Of course you can optimize the results by finetuning the params for each and every source, maybe even split the source into several parts. But this is not what I'm after. I want a script which works well for the vast majority of sources without tweaking the params.

And I believe that johnmeyer's original params do an excellent job for most sources (maybe use dct=3 instead of 0). Way better than MotionProtectedFPS. And keep in mind that I am not interested in watching still frames. I want to just watch the movie, and it should look better than using ChangeFPS or ConvertFPS. I do not mind some motion artifacts as long as they do not get too annoying. And I use this only on film, not on anime. Plus I mostly use this type of fps conversion for PAL <-> NTSC conversion, not for slow motion.


Cheers
manolito

[CruNcher]

yes but overall this result could endup in a catastrophe now if it creates the same temporal issues for every source that heavy perceptual jitter is really annoying for moving objects like those angels it destroys everything and i really wonder how you can percept it now as better then anything before.

you fixed the texture problem yes but this is hyper annoying now in motion compared to that small glitch that was only visible mere milliseconds before.

Good thing in the end you at last saw the problems on the panel with the light scan ;)

Also there are much much more problems to think about you using AVC currently as output you don't know yet how this will be perceived actually with HEVC/VP9 for example it could come out even worse (better perceptible).

It partly though could be also get fixed in the other direction as internaly a hevc encoder is smarter then mvtools and you might throw out extra resources for nothing ;)

[manolito]

I don't know but I think you are being a little too critical. I showed the converted clips to some of my friends who are journalists working for Deutsche Welle TV. They do have trained eyes, but just like me they focus on the overall visual impression, and all of them agreed that the MVTools based conversions were quite pleasing to watch.

For this particular clip my ultimate conversion is here:
http://www82.zippyshare.com/v/7bRTiwej/file.html

dct=1 and blocksize for MAnalyse = 32. Way too slow, but I cannot find any annoying flaws.

My main problem with these fps conversion methods are hard coded subs or movie credits. The letters mostly get warped to a point where they are painful to watch. But this is the only occasion where I would consider splitting up the source and convert the different parts separately...


Cheers
manolito

[kolak]

MvTools are almost as good as Alchemist or Tachyon which are used in braodcast and cost small fortune.
It just would be nice to have it ported to GPU, like svp does. SVP made to many speed shortcuts so quality is not as good.

[CruNcher]

I don't know but I think you are being a little too critical. I showed the converted clips to some of my friends who are journalists working for Deutsche Welle TV. They do have trained eyes, but just like me they focus on the overall visual impression, and all of them agreed that the MVTools based conversions were quite pleasing to watch.

For this particular clip my ultimate conversion is here:
http://www82.zippyshare.com/v/7bRTiwej/file.html

dct=1 and blocksize for MAnalyse = 32. Way too slow, but I cannot find any annoying flaws.

My main problem with these fps conversion methods are hard coded subs or movie credits. The letters mostly get warped to a point where they are painful to watch. But this is the only occasion where I would consider splitting up the source and convert the different parts separately...


Cheers
manolito

Slowly we getting there
beginning = nice
middle = nice (with a small new glitch instead of the complete texture fail we see a big shift now)
end = horrible (full of motion perceptible fails, heat haze effect)

:)

[manolito]

end = horrible (full of motion perceptible fails, heat haze effect)

Are you talking about the three small angels in the background?

The human brain works differently. Everybody just sees the two large angels in the foreground, and they do not display motion artifacts. The information in the background is pretty much discarded because the brain determines that this information is not important. (This is gone by the "Formatio Reticularis" which works like a spotlight).


Cheers
manolito

[CruNcher]

Are you talking about the three small angels in the background?

The human brain works differently. Everybody just sees the two large angels in the foreground, and they do not display motion artifacts. The information in the background is pretty much discarded because the brain determines that this information is not important. (This is gone by the "Formatio Reticularis" which works like a spotlight).


Cheers
manolito

Everything is connected the edge warping of the moving object 2 Angels is causing the problems and stays visible up until the end the edge of the moving object is "warping" everything it moves in front of gets distorted, that is pretty visible and was less visible some versions before these problems where better hidden.

Im pretty sure these warping distortions at the end got amplified i didn't percept them so crazy heavy before on the first sight.

just make a experiment glue both results together :)

the beginning and middle of your newest results and the older result of the end and voila a pretty stable overall result :)

And of course you concentrating on those 2 Angels if you would conduct a Eye tracker experiment you would see the ROI being the 2 Angels and everything in their close proximity is being percepted with highest priority.

[MysteryX]

Here is a "poor man's" fps conversion function by johnmeyer.

jm_fps.avsi


Motion based fps conversion will always introduce some artifacts, but with these parameters johnmeyer really had a golden hand. I did test many different scripts for this, but this one gave by far the best results on all the clips I used it for.

Maybe you are interested in a standalone tool for slow motion using different methods for changing frame rates. Have a look here:
https://forum.doom9.org/showthread.php?p=1789031#post1789031


Cheers
manolito

How does this script compare with InterFrame?

[johnmeyer]

How does this script compare with InterFrame?For the answer, read my post here (https://forum.doom9.org/showpost.php?p=1725099).

That post also explains why some in this thread thought my script provided some good results: I was able to tweak some things which are not "exposed" with SVP and its Interframe front end.

As for things said in this thread, I still am not convinced that any of the DCT settings are going to provide any substantial, real improvements, i.e., it won't produce differences that actually matter. Note I didn't say there wouldn't be differences, only that those differences won't really get at the reasons why motion estimation fails when doing frame rate changes or other operations which involve creating new frames from adjacent frames.

The real issue is how to define "objects" and how to track them. The motion estimation done by any of these MVTools-derived filters relies on nothing more than tracking pre-determined blocks of pixels rather than pre-identifying actual objects in the frame. This is why block size is the most important variable to change when trying to get good results. Depending on the video and the size of the things being tracked (like people's legs, vertical fence posts, and other difficult-to-track items), different block sizes will work on some videos better than others. I find a block size of 16 to be a good starting point, but sometimes find 8 or 32 works better. The block overlap can then provide some fine tuning.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.

So, motion estimation is not a "set it, and forget it" tool, and if you use it that way for creating new frames, you will get burned, and it will be sooner rather than later.

[MysteryX]

In the Natural Grounding Player / Yin Media Encoder (https://github.com/mysteryx93/NaturalGroundingPlayer/), I upscale videos from 288p 25fps into 768p 60fps. I get the best results by running Interframe between the 2 frame doubles, and it is one of the most important steps. For low quality videos with lots of artifacts, SVP is actually removing a lot of those artifacts by creating the interframe animations! Kind of too good to be true, but it works well.

I'm wondering whether the approach you suggest here would give better results, by combining 2 approaches to reduce the severity of artifacts when it fails. I'm looking for a generic script that will work most of the time, with a few simple tweakeable settings.

Do you have a specific script I could try to see the difference in my case?

[CruNcher]

For the answer, read my post here (https://forum.doom9.org/showpost.php?p=1725099).

That post also explains why some in this thread thought my script provided some good results: I was able to tweak some things which are not "exposed" with SVP and its Interframe front end.

As for things said in this thread, I still am not convinced that any of the DCT settings are going to provide any substantial, real improvements, i.e., it won't produce differences that actually matter. Note I didn't say there wouldn't be differences, only that those differences won't really get at the reasons why motion estimation fails when doing frame rate changes or other operations which involve creating new frames from adjacent frames.

The real issue is how to define "objects" and how to track them. The motion estimation done by any of these MVTools-derived filters relies on nothing more than tracking pre-determined blocks of pixels rather than pre-identifying actual objects in the frame. This is why block size is the most important variable to change when trying to get good results. Depending on the video and the size of the things being tracked (like people's legs, vertical fence posts, and other difficult-to-track items), different block sizes will work on some videos better than others. I find a block size of 16 to be a good starting point, but sometimes find 8 or 32 works better. The block overlap can then provide some fine tuning.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.

So, motion estimation is not a "set it, and forget it" tool, and if you use it that way for creating new frames, you will get burned, and it will be sooner rather than later.

Exactly and in this case the Quantization noise from the lossy MPEG Input in manolitos test conversion case causes some of the tracking problems.

[johnmeyer]

Do you have a specific script I could try to see the difference in my case?Definitely not. Like so many video issues, you have to do things manually. In this case, you have to watch the video, see where it fails, and then switch over to the other approach at those points.

I put both versions on two timelines in my NLE, with the motion estimated version on the dominant (i.e., default) track. I play the video, usually at 1.5x - 2x normal speed (to get through it in a hurry). When I see bad frame (or several bad frames), I simply cut to the other track until the problem goes away.

For somewhat critical work, I will sometimes crossfade from the ME version to the other version in order to make the switch less apparent. For really critical work, I will create a motion mask, feathered at the edges, to replace only the parts of the frame that are broken. This produces virtually perfect results, but it obviously takes quite a bit of time. When I do paid work (once in awhile some of my stuff ends up in movies or on TV), it is worth the time to do this.

[MysteryX]

I put both versions on two timelines in my NLE, with the motion estimated version on the dominant (i.e., default) track. I play the video, usually at 1.5x - 2x normal speed (to get through it in a hurry). When I see bad frame (or several bad frames), I simply cut to the other track until the problem goes away.
That's a perfect case where automation provided by a software would be most useful -- but this process would be somewhat complicated to implement.

[johnmeyer]

That's a perfect case where automation provided by a software would be most useful -- but this process would be somewhat complicated to implement.Complicated? No. Impossible? Yes.

You simply cannot anticipate the problems which get created. Also, things which "look good" to an algorithm look like heck to the human eye.

I've written over 100 AVISynth scripts, and have spent many thousands of hours editing video over the past twenty years. I understand both and, as an EE, programmer, and former software manager, I think I know what can and cannot be done (although I am often amazed and surprised what software people manage to create).

The best way to express my skepticism about finding an algorithmic solution to this problem is that if you could actually detect the anomaly created by motion estimation, then the motion estimation itself would be able to avoid creating it in the first place. Since even the commercial software cannot do this (i.e., even people with advanced programming skills and economic incentives), and since they've been working on this for a long time, I don't think it will happen anytime soon.

[MysteryX]

What I'm saying is that a software could allow you to manually go over the video at 50% speed to mark which frames are corrupt, and handle the rest. Exactly the same as you're doing, but without having to hack around with manual scripts.

The first part of your process could probably easily be done.

For somewhat critical work, I will sometimes crossfade from the ME version to the other version in order to make the switch less apparent. For really critical work, I will create a motion mask, feathered at the edges, to replace only the parts of the frame that are broken. This produces virtually perfect results, but it obviously takes quite a bit of time. When I do paid work (once in awhile some of my stuff ends up in movies or on TV), it is worth the time to do this.
Cross-fade maybe could maybe be semi-automated too. But that last part with motion masks, that can only be done manually by someone who really knows what he's doing.

Right now I'm implementing Deshaker (VirtualDub filter), which is difficult to use manually with its 2 passes, especially if you want to preview various settings -- and especially if you want to adjust settings for various segments.

I don't know if your process would be appropriate for my needs, but something that could be done is enter the frame ranges for which to use the alternate method, and handle everything else automatically. Ex: 100-115, 140-144, 160-180

[MysteryX]

Essentially, what you're doing is pretty simple if I understand correctly. You use 2 algorithms: a more aggressive frame interpolation (that causes more artifacts), and a safer method for when it fails. Then the idea is to identify which frames or area to use the alternative method.

If this approach was to be semi-automated, it would be best done as an Avisynth script than as a software. A script or plugin could be designed that takes a string with "100-115,140-144,160-180", and perhaps even allow specifying rectangles or zones for masks, and automatically do everything you're manually scripting.

The only challenge I'm seeing is that this would require conditional filters, but ScriptClip doesn't currently work with MT.

If you're doing a lot of it and are spending a lot of time on this process, perhaps it would be worth it to develop such an utility that systematizes your process.

[johnmeyer]

What I'm saying is that a software could allow you to manually go over the video at 50% speed to mark which frames are corrupt, and handle the rest. Exactly the same as you're doing, but without having to hack around with manual scripts.

The first part of your process could probably easily be done.

Cross-fade maybe could maybe be semi-automated too. But that last part with motion masks, that can only be done manually by someone who really knows what he's doing.

Right now I'm implementing Deshaker (VirtualDub filter), which is difficult to use manually with its 2 passes, especially if you want to preview various settings -- and especially if you want to adjust settings for various segments.1. If you Google my name and "Deshaker" you will find that I wrote a very complex set of scripts to completely automate the process. One set of scripts was written within Sony Vegas Pro, my NLE. It provides its own scripting language. I also wrote a script in VirtualDub that interacts with the Vegas script. The end result is that, when you want to stabilize a clip on the timeline, you press one button, and the entire process happens with no further interaction on your part, including both Deshaker passes.

This entire process is well-documented by posts that I did in the Vegas forum (https://www.vegascreativesoftware.info/us/forum/deshaker-vegas-script-back-again--57432/) many years ago.

2. Because Vegas provides scripting, I already do the automation that you suggest. If, for instance, I want to cut to the other clip, but do it via a two-frame cross-fade, I can simply press one key and it will do the entire cut and cross fade. What's more, if I wanted to, I could instead simply insert markers at each place that I want to fix, and then do all the cuts using a batch version of the same script.

Again, if you look at the scripting portion of the Vegas forum, you will find many of my posts describing some of my dozens of Vegas scripts.

Even though Sony pretty much abandoned Vegas, and the new owner, Magix, doesn't appear to be doing anything to make it better, it is still the most productive editing tool on the planet because of its scripting capability.

[MysteryX]

It still could be interesting to see how your process could be translated into pure Avisynth programming.

[TheFluff]

It still could be interesting to see how your process could be translated into pure Avisynth programming.

please don't

[kolak]

Tachyon (used in broadcast for fps conversion) uses fallback method with masking and fathering:
https://www.telestream.net/pdfs/app-notes/app_Vantage_Tachyon_VPL.pdf

page 5,6.

They find problematic areas and then replace them with frame blended or nearest frame using making and feathering. Looks like they use quality of vectors as deterministic process.
I'm just surprised that this doesn't break "local" motion coherency between frames.

Love to see this in mvtools :)

[poisondeathray]

I'm just surprised that this doesn't break "local" motion coherency between frames.



How do you know it doesn't ?

Have you tested it or seen samples ?

Manually doing it usually looks poor when you try to take care of occlusions in that manner (blending or nearest through accurate user defined rotoscoped masks), so I doubt an "automatic" method using a less accurate method would look any better

[johnmeyer]

It still could be interesting to see how your process could be translated into pure Avisynth programming.

please don't

Since there is no point, I am not even tempted.

[MysteryX]

Johnmeyer, the first part of your process is actually very simple to do.

Write a filter that takes 2 clips and a string as parameters. The filter returns either clip based on the frame number as configured in the string. Frame blending and making transitions transparent, that's a whole other story. Perhaps blending both in the transition frames. Is it like an artist's work where you have to draw it differently in each situation?

I'm just curious, what part of your process can't be systematized?

[videoFred]

A very simple solution is creating two clips from the same source: one with changeFPS() and one with interpolation (this can be done with MVTools2 or Interframe() ).

Then we can select scenes with clipclop():

source = Avisource("L:\VdP\VdP_Sp4_gekuist.avi").converttoYV12()
changed = source.ChangeFPS(25)


V0 = changed
V1 = InterFrame(source,Newnum=25, Newden=1, Cores=8, GPU = true)


NickNames =""" # Psuedonyms for clips

I = 1
"""


SCMD="""
I 0,20
I 836,1285
I 1723,2312
I 3032,3202
I 3986,4456
I 4682,4938
I 6060,6600

"""

SHOW= True

ClipClop(V0,V1,scmd=SCMD,nickname=NickNames,show=SHOW)


The result is a mixed clip: frame doubling on "difficult" scenes (fast moving objects) and frame interpolation on "easy" scenes (slow panning for example). Of cource this requires manual searching for the scenes who can be interpolated.

Fred.

[kolak]

How do you know it doesn't ?

Have you tested it or seen samples ?

Manually doing it usually looks poor when you try to take care of occlusions in that manner (blending or nearest through accurate user defined rotoscoped masks), so I doubt an "automatic" method using a less accurate method would look any better

Yes, I seen results. They are quite good.

[johnmeyer]

Of course you can use ClipClop, but to do that you already have to know frame numbers for your in/out points. To get those, you've already done all the work, presumably in your NLE. In Vegas (my NLE), once I arrive at the frame where the switch should be made, I just make it then and there. It takes less time to finish the job "on site" than actually type or write out the frame number, transfer those numbers to a script, and then execute the script. I see zero benefit to that workflow: it adds extra steps, takes more time, and doesn't let me nudge or make slight changes easily.

I have never understood why people insist on making AVISynth into an editing tool. It really is not well-suited to that job. But, if you want to do it, knock yourself out and have fun!

[StainlessS]

You can use Sawbones/FrameSurgeon Replace FXn range:- https://forum.doom9.org/showthread.php?t=173158&highlight=sawbones
to replace ranges using clipClop, with additional functionality included.


SawBones v1.02

SawBones/FrameSurgeon, a VirtualDub/Avisynth script, utility combo to edit bad frames.
Create Command file in VirtualDub with Sawbones, and use Command file in Avisynth script function FrameSurgeon().

SawBones, is a compiled AutoIt script utility, intended to assist in creating Avisynth command file for FrameSurgeon.avs script.

SawBones is used together with VirtualDub and NotePad. You must run the app, with BOTH VirtualDub and the
NotePad Editor VISIBLE, you can scroll through a video clip, and press eg CTRL/DELETE to insert a FrameSurgeon DELETE (DEL n) command
for the current frame into the NotePad Editor. You press the keys with VDub as the active window (not NotePad).
SawBones requires VirtualDub, VirtualDubMod will not be recognised (It does not provide marked ranges in Status Bar).
After creating Command file in NotePad, Save as eg Command.txt and provide it as eg FrameSurgeon(Cmd="Command.txt").

*** Current frame is shown in VirtualDub in the middle of the status bar. ***
*** Ranges are marked in VirtualDub via HOME and END keys, shown at Left of status bar (when marked). ***
*** FXd clips (where d is Digit 1-9) are user provided to FrameSurgeon.avs function and default to source clip if not user supplied. ***
*** Interpolation commands in FrameSurgeon are for YV12 and YUY2 only, others any colorspace. ***

It is important to NOT do any VirtualDub clip editing, if you eg delete a frame in VirtualDub, then all frames after that frame will
be off by 1, and so all SawBones edited ranges/frames inserted later into NotePad file will be also off by 1.
The VDub loaded clip can be either an AVI or AVS clip, it makes no difference, you could eg have a stacked multi-window frame
open in VDub so as to choose from your FXd clips.

All SawBones Keyboard commands that are inserted into NotePad Text file:-

CTRL/F1 CopyFromPrevious frame to current frame (ie replace current frame n with frame n - 1. (CP n)
CTRL/F2 CopyFromNext frame to current frame (ie replace current frame n with frame n + 1. (CN n)

CTRL/1 Replace current frame with same frame from FX1 clip. (FX1 n)
CTRL/2 Replace current frame with same frame from FX2 clip. (FX2 n)
CTRL/3 Replace current frame with same frame from FX3 clip. (FX3 n)
CTRL/4 Replace current frame with same frame from FX4 clip. (FX4 n)
CTRL/5 Replace current frame with same frame from FX5 clip. (FX5 n)
CTRL/6 Replace current frame with same frame from FX6 clip. (FX6 n)
CTRL/7 Replace current frame with same frame from FX7 clip. (FX7 n)
CTRL/8 Replace current frame with same frame from FX8 clip. (FX8 n)
CTRL/9 Replace current frame with same frame from FX9 clip. (FX9 n)

CTRL/SHIFT/1 Replace range with same range from FX1 clip. (FX1 s,e)
CTRL/SHIFT/2 Replace range with same range from FX2 clip. (FX2 s,e)
CTRL/SHIFT/3 Replace range with same range from FX3 clip. (FX3 s,e)
CTRL/SHIFT/4 Replace range with same range from FX4 clip. (FX4 s,e)
CTRL/SHIFT/5 Replace range with same range from FX5 clip. (FX5 s,e)
CTRL/SHIFT/6 Replace range with same range from FX6 clip. (FX6 s,e)
CTRL/SHIFT/7 Replace range with same range from FX7 clip. (FX7 s,e)
CTRL/SHIFT/8 Replace range with same range from FX8 clip. (FX8 s,e)
CTRL/SHIFT/9 Replace range with same range from FX9 clip. (FX9 s,e)

CTRL/SHIFT/ALT/1 Interpolate current frame n using n-1 and n+1 as source frames. (I1 n)
CTRL/SHIFT/ALT/2 Interpolate 2 frames starting at current frame n, using n-1 and n+2 as source frames. (I2 n)
CTRL/SHIFT/ALT/3 Interpolate 3 frames starting at current frame n, using n-1 and n+3 as source frames. (I3 n)
CTRL/SHIFT/ALT/4 Interpolate 4 frames starting at current frame n, using n-1 and n+4 as source frames. (I4 n)
CTRL/SHIFT/ALT/5 Interpolate 5 frames starting at current frame n, using n-1 and n+5 as source frames. (I5 n)
CTRL/SHIFT/ALT/6 Interpolate 6 frames starting at current frame n, using n-1 and n+6 as source frames. (I6 n)
CTRL/SHIFT/ALT/7 Interpolate 7 frames starting at current frame n, using n-1 and n+7 as source frames. (I7 n)
CTRL/SHIFT/ALT/8 Interpolate 8 frames starting at current frame n, using n-1 and n+8 as source frames. (I8 n)
CTRL/SHIFT/ALT/9 Interpolate 9 frames starting at current frame n, using n-1 and n+9 as source frames. (I9 n)

CTRL/SHIFT/ALT/? Interpolate Range. Ie in Avisynth Inclusive mode, where range is 100,101 [2 frames] then result = (I2 100)
The specified range start and end are the outer bad frames.

CTRL/DELETE Delete current frame. (DEL n)
CTRL/SHIFT/DELETE Delete range (DEL s,e)

CTRL+SHIFT+ALT+PAUSE is TERMINATE Program (or close via System Tray icon). PAUSE is also known as BREAK, usually next to Scroll Lock.


NOTE: A range shown in VDub status bar as eg "Selecting Frames 100-102(2 frames)" represents frames 100 and 101, frame 102 is exclusive and
does not count. By default, SawBones uses Inclusive END frame ranges (same as Avisynth where end frame DOES count).
You can change the default behaviour to behave the same as VDub by Running SawBones.Exe at least once and changing the auto created
SawBones.ini file contents from "RangeEndIsExclusive=0" to "RangeEndIsExclusive=1". When using VDub "RangeEndIsExclusive=1" mode,
we subtract 1 from the End Frame to convert to Avisynth End Frame Inclusive specification when writing range to NotePad command.
When sending a command using VDub exclusive mode with a status bar range of "100-100(frames=0)", it will beep and show a "No Frames"
type error message for a few seconds, in Avisynth Inclusive mode it will send a 1 frame range command to the NotePad window.

It is easy to see what is happening as the NotePad window will be visible and after each insertion into NotePad, an ENTER key
will also be sent to move the cursor down one line, each command is on its own line. If you make a mistake, it is easy to
just switch to NotePad window and delete the erroneous line.

Where same frames are flagged for replacment mulitple times, later one will take precidence.
All frame/range deletes will be done AFTER replacements, multiple deletes on same frame will only result on single frame deletion.
Already Interpolated frame/range CANNOT be replaced and will produce an error (In avs script), but they can be deleted.

FrameSurgeon.avs requires MvTools, GScript, RT_Stats, FrameSel, ClipClop and Prune Plugins.

AutoiIt compiled executable with source provided, just click Menu Tools/build to create executable (In Scite4AutoIt3 editor).
Requires AutoIt3 and Scite4AutoIt3 editor to re-build executable.


EDIT: You could use an input Avisynth Stacked clip for viewing multiple clips side by side to choose best FXn option.

[poisondeathray]

Yes, I seen results. They are quite good.

Can you post some examples ?







I have never understood why people insist on making AVISynth into an editing tool. It really is not well-suited to that job. But, if you want to do it, knock yourself out and have fun!

I completely agree . Once people actually use the other tools , they will never go back to using avisynth for those types of operations. Different tools better suited for different things

Along similar lines, a way to use NLE is a multicam edit where you have more than 2 layers. You have multiple iterations/settings of videos on different layers (e.g. large block size , smaller block size, dct=1, etc....) and just switch between them easily . It would take easily 20-30x longer to do this type of edit in avisynth

[MysteryX]

Of course you can use ClipClop, but to do that you already have to know frame numbers for your in/out points. To get those, you've already done all the work, presumably in your NLE. In Vegas (my NLE), once I arrive at the frame where the switch should be made, I just make it then and there. It takes less time to finish the job "on site" than actually type or write out the frame number, transfer those numbers to a script, and then execute the script. I see zero benefit to that workflow: it adds extra steps, takes more time, and doesn't let me nudge or make slight changes easily.

I have never understood why people insist on making AVISynth into an editing tool. It really is not well-suited to that job. But, if you want to do it, knock yourself out and have fun!
Not everybody uses Sony Vegas.

I also don't like writing Avisynth manually.

What I personally do is write a software that generates Avisynth code for my needs in a simple way, and then handles all the files and encoding steps.

Avisynth plugins must provide all the features, and then I can write software interface to make the job as simple as it can be. In this case, it's definitely possible to create a window where you can navigate the video frame by frame and mark the positions.

It shouldn't be that hard to do. The hardest part would be to open the Avisynth scripts directly in .NET without using Windows Media Player to do the preview; but others have already written that code.

It seems you know what you're doing. I'd just like to better understand the procedure you are using, and perhaps see some samples. If there is considerable improvement over Interframe, I might decide to program it. I see no reason why this couldn't be done. Plus, you say that the fade-in and fade-out between clip segments requires extra time. I see no reason why this wouldn't be automated.

[kolak]

Can you post some examples ?


Don't have them anymore, but could not do it anyway.
I've seen some sample where fallback occurred and it was better than massive morphing artefacts which typically appear in such a places.

[MysteryX]

Don't have them anymore, but could not do it anyway.
I've seen some sample where fallback occurred and it was better than massive morphing artefacts which typically appear in such a places.
I'd like to see it. Doing a blend of both algorithms on the transition frames should make it smooth; perhaps even gradual blending over 2 or 3 adjacent frames. All automated of course.

I'd just like to have more specific details and algorithms to give it a try.

I believe Interframe's anti-artifact results in disabling frame interpolation when it detects artifacts, which isn't ideal either.

Also, SVP doesn't support YV24, but does other libraries allow for YV24 processing? That's one area that would make a considerable difference in my case because I do a frame double after Interframe.

[MysteryX]

I re-read the whole thing to better understand. I have 2 questions.

I play the video, usually at 1.5x - 2x normal speed (to get through it in a hurry).
How do you play it at 2x speed? In order to see the result, all the frames must be calculated. You won't reach 2x speed unless the processing is faster than real-time, which it won't if it's slower than SVP.

In general, I only use this technology in conjunction with something else and then mix the two together. The reason is that other technologies, such as frame blending, never fail badly, but they also don't produce results that are as good as motion estimation, but only when motion estimation is behaving. Unfortunately, when motion estimation (including other tools like Twixtor) fails, it fails sepectacularly, ruining the viewing experience. You cannot rely on it.
How do I do frame blending interpolation in Avisynth? Are there other methods you use?

[StainlessS]

How do I do frame blending interpolation in Avisynth?

Well for blending, ConvertFPS :- http://avisynth.nl/index.php/FPS.

Play at 2x speed, presume he just does AssumeFPS(FrameRate*2.0). [or equivalent in Vegas]

[MysteryX]

I have made a quick test to compare Interframe(preset="smooth") with jm_fps. This is part of a script that does more processing, and does an extra frame double which makes the changes more visible. This is a test on very bad quality content to see how it behaves.

http://screenshotcomparison.com/comparison/204832

http://screenshotcomparison.com/comparison/204833

The difference is huge. It's much better than I expected. Worth investigating some more. Plus, it allows processing in YV24.

Can the jm_fps script work in 16-bit, using either DitherTools hack or native AVS+ format?

I did further tests in YV24, and strangely, it generates a LOT of strong artifacts!! This seems more like a bug.
http://screenshotcomparison.com/comparison/204834

[MysteryX]

Play at 2x speed, presume he just does AssumeFPS(FrameRate*2.0). [or equivalent in Vegas]
This won't show you the interpolation artifacts.

[StainlessS]

This won't show you the interpolation artifacts.

I never said that it did. John said "to get through it in a hurry". (got good high speed eyes has John).

EDIT: Note, John's script keeps every even output frame same as input, at double rate.