I need some help with interpolation videos, lots of detail in post

[FranceBB]

Originally Posted by lansing

Just use it and you'll see.

So far I see 3 approaches to deal with detected artifact.

1. Do nothing, rely solely on the ability of mvtools. You'll see artifact but get smoothness.

2. Don't change anything. Keep the object as to the original. You'll see smoothness on objects that aren't artifact detected and stutter on objects that are. No artifact.

3. Blend the frames. Since no new interpolated frame has been created, you basically ended up with what the original looks like on playback.

I'd also add 2.5:

2.5. Interpolate something, blend something else. You'll get smoothness on objects that aren't artifact detected and blending on objects that are. Still, no artifact, but a slightly better smoothness.


I very rarely rely on MVTools in Broadcast and I use it on very rare circumstances like slow-motions.
The reason why I use it is that it achieves better results than the built-in linear interpolation filter in AVID Media Composer (hats off to the open source community ).
Anyway, whenever we shoot something ourselves, we try to shoot at 50fps progressive in order to just use

assumeTFF()
separatefields()
selectevery(4,0,3)
weave()

When we have to make a slow-motion, we record at 200fps progressive so we can slow it down in post production and get it smooth.
Sometimes, the producer wants to make a slow-motion of a scene later on, on a second thought, when we are in studio and we already recorded the scene at 50fps progressive; in that case I use MVTools.

[lansing]

Originally Posted by bradwiggo

Which script would I want to use for option 2?

It's the sample script I posted. The snow was moving at 24 fps while the character was at 60 fps. Lowering the block size to 16x16 was able to detect the snow, but it wasn't able to interpolate any new snow object in between, so I think mvtools has reach its limit on this one. But you can try to tweak other parameters yourself to see if it helps.

[bradwiggo]

Originally Posted by lansing

It's the sample script I posted. The snow was moving at 24 fps while the character was at 60 fps. Lowering the block size to 16x16 was able to detect the snow, but it wasn't able to interpolate any new snow object in between, so I think mvtools has reach its limit on this one. But you can try to tweak other parameters yourself to see if it helps.

When I look at one of the previous attempt I had made, the snow does move position every frame, but it seems to be in position x in one frame, and then it will appear I both position x and y the next frame, and then just y the one after.

How did you tell it was detecting the snow if it wasn't changing it, is there a way to view all the moving objects it has detected?

[bradwiggo]

Originally Posted by lansing

It's the sample script I posted. The snow was moving at 24 fps while the character was at 60 fps. Lowering the block size to 16x16 was able to detect the snow, but it wasn't able to interpolate any new snow object in between, so I think mvtools has reach its limit on this one. But you can try to tweak other parameters yourself to see if it helps.

I tried using the sample script you posted, but megui is stuck at 99.98% completion.

That script seems doesn't seem to work very well, I press queue in megui and it it will start and immediately stop, no error displayed but megui has a red cross next to it on the icon which indicates an error.

The log tab in megui said this:

--[Error] [24/07/2018 19:14:11] Process exits with error: 0xC0000005 STATUS_ACCESS_VIOLATION (-1073741819)

[lansing]

Originally Posted by bradwiggo

I tried using the sample script you posted, but megui is stuck at 99.98% completion.

That script seems doesn't seem to work very well, I press queue in megui and it it will start and immediately stop, no error displayed but megui has a red cross next to it on the icon which indicates an error.

The log tab in megui said this:

--[Error] [24/07/2018 19:14:11] Process exits with error: 0xC0000005 STATUS_ACCESS_VIOLATION (-1073741819)

SVP is using Nvidia card to do the real time frame interpolation. Do you have a Nvidia card in the first place?

[bradwiggo]

Originally Posted by lansing

SVP is using Nvidia card to do the real time frame interpolation. Do you have a Nvidia card in the first place?

No, I have an AMD integrated card, SVP can use all types of care can't it? https://www.svp-team.com/wiki/GPU_Compatibility

[bradwiggo]

Originally Posted by johnmeyer

I suggest that anyone trying to help the OP read the following because this thread is going down the same path as the thread he started a month ago on Videohelp.com:

Why don't my interpolated videos look as good as examples I see on youtube?

The problem is that everyone there -- and it is now true of the posts so far in this thread -- didn't initially understand the real problem at the heart of what he is trying to do. Here's that problem:

He wants to create smoother motion for animation and, as most people reading this know, animation repeats some frames, but not others, and does so in a way that does not follow a regular pattern, like telecine patterns usually do. So, if you simply apply MVTools2, SVP, or Interframe motion estimation to create more frames, you end up with a real visual mess, and the motion doesn't look that much smoother.

What first needs to be done is to replace the dups in a way that takes into account the variable time gap between frames that are actually different. Thus, I don't think you can simply use FillDrops() (a function I've posted many times) to replace all duplicates, first because there are some situations where there is more than one dup in a row which will cause FillDrops() to fail, but also because some gaps in time between non-duplicate frame are going to be larger than others. THAT is where you want to insert a motion estimated frame, after you've deleted a duplicate, NOT at the place where the dup is removed.

One thing I would suggest to the OP, now that he is here on doom9, is to take a look at this thread I started several years ago:

Automatically fix dups followed (eventually) by drops

What I tried to do in that thread -- and with the help of some old code written by Didée I was able to accomplish -- was measure the temporal gaps between each non-dup frame. Then, after I deleted each duplicate, rather than insert an interpolated frame at the location of the deleted frame, I instead used my "gap logic" to insert an interpolated frame at a nearby location that had the biggest apparent jump in motion.

It wasn't perfect, but I think I was on the right track, and I believe that it might be a way to fix the OP's problem.

BTW, if someone can figure this out, what he wants is actually something that might be useful to other animation fans.

Is that the script you think the video might have been using, as that is my ultimate goal, to find a script that gets me as close to that video as possible.

[bradwiggo]

Originally Posted by StainlessS

Suggest LSMASHVideoSource(), and LSMASHAudioSource()
if one of these, mov, mp4, m4v, 3gp, 3g2, mj2, dvb, dcf, m21.

No idea what is best for mkv.

I tried the second script you posted in your first comment, and there is quite a lot of stutter, however I have just noticed I didn't change the fps bit to 24, could that be causing the stutter, as it is 24fps not 25?

[johnmeyer]

OK, even though I don't like animation, I've been reading these posts (at videohelp and now here) for over a month so I download the clip and played with it.

My conclusion?

Unfortunately I now have exactly the same conclusion as I did over in Videohelp: you are chasing a unicorn.

There are several problems. First, even though the animation has no dups (unlike anime, etc.) it is still only 24 fps, so there are big gaps in time between frames. The bigger the time gap, the tougher time motion estimation has in figuring out where to put everything for the intermediate frames.

Second, there are some really small objects to track (e.g., the snowflakes). There is no way in the world motion estimation can figure out what to do with these, especially since they, by design, are darting almost at random. From a technical standpoint, they are pretty much the same as noise.

Third, many of the objects are pretty murky. As one example, about 1/3 of the way through the clip her turquoise glove hand is moving in front of her purple dress. There is very little contrast between the hand and the dress, so the algorithms don't do what they should.

I tried those "magic" settings I referred to before, and which seem to have worked well for many people on other material, but the result was pretty bad. It isn't worth posting the result, but here's the script I used (I always frameserve from Vegas, so the video file is always "fs.avi", the frameserver signpost):

loadplugin("C:\Program Files\AviSynth 2.5\plugins\removegrain.dll")

film="e:\fs.avi"

#setmtmode(5,4)
source= Avisource(film).killaudio().converttoYV12()
#setmtmode(2)


prefiltered = RemoveGrain(source,22)
super = MSuper(source,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=4,search=3,dct=0)
forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=4,search=3,dct=0)
forward_re = MRecalculate(super, forward, blksize=8, overlap=2,thSAD=100)
backward_re = MRecalculate(super, backward, blksize=8, overlap=2,thSAD=100)
MFlowFps(source,super, backward_re, forward_re, num=60000, den=1001,ml=200,mask=2)

My advice is to simply watch and enjoy the film, although now that I know what film it is, my further advice is to turn down the volume whenever Idina Menzel is singing. She has the harshest, screechiest voice I've ever heard. Also, I've heard her sing live, and without autotune she can't hold a note.

Not my favorite singer, as you can tell.

[lansing]

Originally Posted by bradwiggo

No, I have an AMD integrated card, SVP can use all types of care can't it? https://www.svp-team.com/wiki/GPU_Compatibility

I don't know, you can try turning off gpu mode to see what happen, set "gpu:0" to turn it off.

[bradwiggo]

Originally Posted by johnmeyer

OK, even though I don't like animation, I've been reading these posts (at videohelp and now here) for over a month so I download the clip and played with it.

My conclusion?

Unfortunately I now have exactly the same conclusion as I did over in Videohelp: you are chasing a unicorn.

There are several problems. First, even though the animation has no dups (unlike anime, etc.) it is still only 24 fps, so there are big gaps in time between frames. The bigger the time gap, the tougher time motion estimation has in figuring out where to put everything for the intermediate frames.

Second, there are some really small objects to track (e.g., the snowflakes). There is no way in the world motion estimation can figure out what to do with these, especially since they, by design, are darting almost at random. From a technical standpoint, they are pretty much the same as noise.

Third, many of the objects are pretty murky. As one example, about 1/3 of the way through the clip her turquoise glove hand is moving in front of her purple dress. There is very little contrast between the hand and the dress, so the algorithms don't do what they should.

I tried those "magic" settings I referred to before, and which seem to have worked well for many people on other material, but the result was pretty bad. It isn't worth posting the result, but here's the script I used (I always frameserve from Vegas, so the video file is always "fs.avi", the frameserver signpost):

loadplugin("C:\Program Files\AviSynth 2.5\plugins\removegrain.dll")

film="e:\fs.avi"

#setmtmode(5,4)
source= Avisource(film).killaudio().converttoYV12()
#setmtmode(2)


prefiltered = RemoveGrain(source,22)
super = MSuper(source,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=4,search=3,dct=0)
forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=4,search=3,dct=0)
forward_re = MRecalculate(super, forward, blksize=8, overlap=2,thSAD=100)
backward_re = MRecalculate(super, backward, blksize=8, overlap=2,thSAD=100)
MFlowFps(source,super, backward_re, forward_re, num=60000, den=1001,ml=200,mask=2)

My advice is to simply watch and enjoy the film, although now that I know what film it is, my further advice is to turn down the volume whenever Idina Menzel is singing. She has the harshest, screechiest voice I've ever heard. Also, I've heard her sing live, and without autotune she can't hold a note.

Not my favorite singer, as you can tell.

I don't see how I am chasing the impossible though, as I have seen an interpolation that looks good (the youtube video), surely it must be possible to get it to look at least that good, as somebody has done it before.

[bradwiggo]

Originally Posted by lansing

I don't know, you can try turning off gpu mode to see what happen, set "gpu:0" to turn it off.

I tried that and got an error: cubic mode unsupported on CPU.

I have however just noticed that at the page I linked in my last comment, it says you need an older driver for AMD iGPUs, so I maybe should have a look at tracking that driver down.

[bradwiggo]

Originally Posted by StainlessS

See here,

https://forum.doom9.org/showthread.php?t=174793

Something smaller and simpler

# jm_fps.avs

Global G_DCT=1

function jm_fps(clip source) {
	fps_num = 60
	fps_den = 1

	prefiltered = RemoveGrain(source, 22)
	super = MSuper(source, hpad = 16, vpad = 16, levels = 1) # one level is enough for MRecalculate
	superfilt = MSuper(prefiltered, hpad = 16, vpad = 16) # all levels for MAnalyse
	backward = MAnalyse(superfilt, isb = true, blksize = 16, overlap = 4, search = 3, dct = G_DCT)
	forward = MAnalyse(superfilt, isb = false, blksize = 16, overlap = 4, search = 3, dct = G_DCT)
	forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
	backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
	out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

	return out
}

EDIT:

Manolito mod of above, from here:- https://forum.doom9.org/showthread.p...39#post1800439

# Motion Protected FPS converter script by johnmeyer from Doom9
# Slightly modified interface by manolito
# Requires MVTools V2 and RemoveGrain
# Also needs fftw3.dll in the System32 or SysWOW64 folder for Dct values other than 0


function jm_fps(clip source, float "fps", int "BlkSize", int "Dct")
{
fps = default(fps, 25.000)
fps_num = int(fps * 1000)
fps_den = 1000
BlkSize = default(BlkSize, 16)
Dct = default(Dct, 0)

prefiltered = RemoveGrain(source, 22)
super = MSuper(source, hpad = 16, vpad = 16, levels = 1, sharp = 1, rfilter = 4) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16, sharp = 1, rfilter = 4) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward = MAnalyse(superfilt, isb = false, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = false, ml = 200, mask = 2)

return out
}

I tried this script with the framerate at 25fps and at 23.978 fps (which is the source framerate), and both times the result had a lot of stutter: https://1drv.ms/u/s!AiOx2LWATSlvzj80scSHeGeZ30Ur (that is with the fps = 23.978)

[johnmeyer]

Originally Posted by bradwiggo

I don't see how I am chasing the impossible though, as I have seen an interpolation that looks good (the youtube video), surely it must be possible to get it to look at least that good, as somebody has done it before.

You don't seem to understand what I have been posting. This technology can work miracles on some scenes, and totally fail on others. I gave you a list of the types of things which cause it to fail. That is what you are seeing. There is no way around it except to do the film in sections and use frame blending for the frames where interpolation fails. It takes a lot of manual work, but if you are willing to put the work into it, you can get decent results.

You will not get those results by suddenly finding some magic settings, pushing a button, and having the results appear.

[bradwiggo]

Originally Posted by johnmeyer

You don't seem to understand what I have been posting. This technology can work miracles on some scenes, and totally fail on others. I gave you a list of the types of things which cause it to fail. That is what you are seeing. There is no way around it except to do the film in sections and use frame blending for the frames where interpolation fails. It takes a lot of manual work, but if you are willing to put the work into it, you can get decent results.

You will not get those results by suddenly finding some magic settings, pushing a button, and having the results appear.

I see what you mean, but the video I linked to I assume was all made with one script, as it is only 3 minutes long, so surely there must be a script which can produce that, even if it doesn't do as well for the rest of the film. I have looked a bit more at the youtube channel of the person who uploaded that video, and they have 3 similar videos, which I assume were done using the same script.

My goal at the moment isn't to try to find a script that can produce interpolation of the quality in the video for the whole movie, it is simply to find the script that was used to make that video. Even if that script makes the rest of the movie look rubbish.

[bradwiggo]

Originally Posted by lansing

I don't know, you can try turning off gpu mode to see what happen, set "gpu:0" to turn it off.

Does the quality of the interpolation depend on the type of graphics card you have?

[FranceBB]

Originally Posted by bradwiggo

Does the quality of the interpolation depend on the type of graphics card you have?

No, only speed is affected.
Anyway, the GPU implementation might be slightly different than the CPU one.

Anyway, johnmeyer is right, there are cases in which interpolation fails, no matter what filter you use. The best advice would be to use motion interpolation whenever you can and use blending on all the other scenes. Blending doesn't look as smooth as motion interpolation, but it doesn't make the video stutter as a repeated frame does.

[bradwiggo]

Originally Posted by FranceBB

No, only speed is affected.
Anyway, the GPU implementation might be slightly different than the CPU one.

Anyway, johnmeyer is right, there are cases in which interpolation fails, no matter what filter you use. The best advice would be to use motion interpolation whenever you can and use blending on all the other scenes. Blending doesn't look as smooth as motion interpolation, but it doesn't make the video stutter as a repeated frame does.

Was the youtube video done mainly with motion interpolation or blending? As at the moment I am more just trying to find the script that was used to make that video.

[poisondeathray]

One difference is the YT version you linked to uses scene blending . That could explain the "stutter" that you are seeing, because there would be some duplicate frames at the end of each scene. So in any of those generic interpolation scripts, you would use blend=true for the flow part . Usually it's set to false by default for most interpolation scripts for any generic scripts

In my experience, GPU quality is noticably worse for interpolation for SVP ; larger artifacts, and sometimes even duplicate frames. I posted examples and comparisons probably here and other forums as well. This might not be true for your system or card/hardware setup, so try different combinations . But it's usually much faster

Disable all the other options , things like like artifact masking if you want it to look like the YT version

If you do all that and still think the YT version is "smoother" ; the other possibility is some playback issue; there might be differences in HW acceleration in browser or local media playback.

[bxyhxyh]

Originally Posted by johnmeyer

as most people reading this know, animation repeats some frames, but not others, and does so in a way that does not follow a regular pattern

As johnmeyer said
This nature of animation leads to ugly results instead of getting smoother video.

For example,
Think that there are 2 objects move in the video.
They don't move together sometimes. They might even animated separately.
One moves at 12 fps and other one is 8 fps etc...
Not full 24 fps.

Any interpolation isn't smart enough to see this.
They will just add duplicate of that object since it can't see objects as 'moving'.
Result wouldn't be not much smoother than the original if not any smoother.

That's why he is saying it's impossible.
But you can try.