MVTools and gradual slow motion / time remapping

[lecapitan]

Hey everyone.

I was really bothered that I couldn't find a decent free/cheap alternative to Adobe's Time Remapping, so I decided to take a stab at it using Avisynth and MVTools. I'm posting my script here to share and also get some feedback on ways to improve it. As of right now I am happy with the results I get but I have to imagine it can be improved.

I make use of GScript as well since it's easier for me to use standard control statements and loops.

For the noob's: (myself included)

LoadPlugin("mvtools2.dll")
LoadPlugin("GScript.dll")


Here are the functions: (the good stuff)

# Gradually slows a clip down
function Slowmoe(clip source, float startFrame, float endFrame, float startRatio, float endRatio)
{
# Get the clip's original framerate
mfpsn = FramerateNumerator(source)
mfpsd = FramerateDenominator(source)

# Convert and trim clip
mclip = Trim(source, startFrame, endFrame)

# Flip frames
GScript("""
if (endFrame < startFrame) {
temp = endFrame
endFrame = startFrame
startFrame = temp
}
""")

# Do the MVTools SlowMotion
super = MSuper(mclip, pel=2)
backward_vec = MAnalyse(super, overlap=4, isb = true, truemotion=true, search=3)
forward_vec = MAnalyse(super, overlap=4, isb = false, truemotion=true, search=3)
slowdown = MFlowFPS(mclip, super, backward_vec, forward_vec, Round(mfpsn * 1.0/endRatio), mfpsd)
slowdown = AssumeFPS(slowdown, mfpsn, mfpsd)

# Initialize values
startFrames = (mfpsn * 1.0/startRatio)/mfpsd # Starting frames in a second
endFrames = (mfpsn * 1.0/endRatio)/mfpsd # Ending frames in a second
framesRange = endFrames - startFrames # Difference in frames
currentFrame = 0 # The current frame in the final clip
counter = 0 # The total frame counter
lastFrame = Framecount(slowdown) # The total frames in the slow mo clip
desiredFrames = 0 # The current desired frames
removeFrameCounter = 0 # The frame removal counter
removeFrameThreshold = Float(endFrames)/Float(startFrames) # The number of frames to remove
overflow = 0 # Overflow from the frame threshold

GScript("""
while (counter < lastFrame) {

# Increment the frame removal counter
removeFrameCounter = removeFrameCounter + 1

# Check if the frame removal counter is over the threshold
if (removeFrameCounter >= Floor(removeFrameThreshold + overflow)) {
# Store the overflow
if (overflow > 1) {
overflow = overflow - 1.0
}
else {
overflow = overflow + removeFrameThreshold - Floor(removeFrameThreshold)
}
# Calculate the desired frames. You could use a higher power to make the transition even smoother.
desiredFrames = startFrames + Pow(Float(counter)/Float(lastFrame), 4) * framesRange
removeFrameThreshold = Float(endFrames)/Float(desiredFrames)
# Stop the threshold if it gets too low
if (removeFrameThreshold < 1.1) {
removeFrameThreshold = 1.0
overflow = 0
}
# Reset the counter
removeFrameCounter = 0
}

# Delete or Keep frame
if (removeFrameCounter == 0) {
currentFrame = currentFrame + 1
}
else {
slowdown = DeleteFrame(slowdown, currentFrame)
}

# Increment the counter
counter = counter + 1
}
""")

# Delete the last frame
slowdown = DeleteFrame(slowdown, Framecount(slowdown))
# Return the clip
return slowdown
}

# Gradually speeds a clip up
function Speedmoe(clip source, float startFrame, float endFrame, float startRatio, float endRatio)
{
# Get the clip's original framerate
mfpsn = FramerateNumerator(source)
mfpsd = FramerateDenominator(source)

# Convert and trim clip
mclip = Trim(source, startFrame, endFrame)

# Flip frames
GScript("""
if (endFrame < startFrame) {
temp = endFrame
endFrame = startFrame
startFrame = temp
}
""")

# Do the MVTools SlowMotion
super = MSuper(mclip, pel=2)
backward_vec = MAnalyse(super, overlap=4, isb = true, truemotion=true, search=3)
forward_vec = MAnalyse(super, overlap=4, isb = false, truemotion=true, search=3)
slowdown = MFlowFPS(mclip, super, backward_vec, forward_vec, Round(mfpsn * 1.0/startRatio), mfpsd)
slowdown = AssumeFPS(slowdown, mfpsn, mfpsd)

# Initialize values
startFrames = (mfpsn * 1.0/startRatio)/mfpsd # Starting frames in a second
endFrames = (mfpsn * 1.0/endRatio)/mfpsd # Ending frames in a second
framesRange = startFrames - endFrames # Difference in frames
currentFrame = Framecount(slowdown) # The current frame in the final clip
counter = 0 # The total frame counter
lastFrame = Framecount(slowdown) # The total frames in the slow mo clip
desiredFrames = 0 # The current desired frames
removeFrameCounter = 0 # The frame removal counter
removeFrameThreshold = Float(startFrames)/Float(endFrames) # The number of frames to remove
overflow = 0 # Overflow from the frame threshold

GScript("""
while (counter < lastFrame) {

# Increment the frame removal counter
removeFrameCounter = removeFrameCounter + 1

# Check if the frame removal counter is over the threshold
if (removeFrameCounter >= Floor(removeFrameThreshold + overflow)) {
# Store the overflow
if (overflow > 1) {
overflow = overflow - 1.0
}
else {
overflow = overflow + removeFrameThreshold - Floor(removeFrameThreshold)
}
# Calculate the desired frames. You could use a higher root to make the transition even smoother.
desiredFrames = endFrames + Pow(Float(counter)/Float(lastFrame), 4) * framesRange
removeFrameThreshold = Float(startFrames)/Float(desiredFrames)
# Stop the threshold if it gets too low
if (removeFrameThreshold < 1.1) {
removeFrameThreshold = 1.0
overflow = 0
}
# Reset the counter
removeFrameCounter = 0
}

# Keep or Delete frame
if (removeFrameCounter == 0) {
currentFrame = currentFrame - 1
}
else {
slowdown = DeleteFrame(slowdown, currentFrame)
currentFrame = currentFrame - 1
}

# Increment the counter
counter = counter + 1
}
""")

# Return the clip
return slowdown
}


Here's an example of using them:

source = AVISource("stock.avi")
return Slowmoe(source, 0, 18, 1.0, 1.0) ++ Slowmoe(source, 18, 26, 1.0, 0.10) ++ Slowmoe(source, 26, 30, 0.10, 0.10) ++ Speedmoe(source, 30, Framecount(source), 0.10, 1.0)


I thought up the algorithm to do this on my own. I have no idea if it's even close to the real way to do it.

Anyways, it works like so:
First I take the slowest percentage you want the clip at and use MVTools to do the nice frame interpolation and slow the clip down to that speed. Then I loop through the clip and gradually remove frames to speed the clip back up. I start with looking at the clip's original fps. If the clip was originally 30 fps, and I've slowed it down to 120 fps, I would need to cut 75% of the frames out to get back to the original speed. In other words, I need to save every 4th frame and remove the rest. Then it's just a matter of ramping that down using some math.

Example:

I've got a 10 frame clip @ 30fps and I want to gradually slow it from 100% to 25%

Slow to 25%
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF (40 frames)

I start saving the first frame then removing the next 3
F---FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Eventually I get to a point where I save one frame and remove the next 2
F---F---F--FFFFFFFFFFFFFFFFFFFFFFFFFFFFF

And later, I get to a point where I save one frame and remove the adjacent frame
F---F---F--F--F--F--F-FFFFFFFFFFFFFFFFFF

Finally, I'm down to where all the frames are saved
F---F---F--F--F--F--F-F-F-F-F-F-FFFFFFFF

The overflow calculation helps to smooth the transitions out. It won't always be remove 3, then remove 2, then remove 1. It should look more like (removing frames) 33323232222121211110101010000.

That's all there is to it!

Thanks for looking :cool:

[Gavino]

Welcome to the forum, lecapitan.

I think your 'variable-rate frame decimation' algorithm is equivalent to animating ChangeFPS which I explained in this post.

I have been using this function in my own projects:
function VarSpeed(clip c, float v1, float v2) {
av = 0.5*(v1+v2)
count = round(c.FrameCount()/av)
Animate(c.Loop(), 0, count-1, "VarSpeed2", v1, av).Trim(0, count-1)
}

#helper function:
function VarSpeed2(clip c, float v) {
r = c.Framerate()
c.AssumeFPS(r*v)
ChangeFPS(r)
}

Your function uses MVTools for superior frame interpolation, while mine just uses ChangeFPS.
However, my function VarSpeed2 could easily be changed to use MFlowFPS instead (calculate the vectors once in VarSpeed and pass them as parameters to VarSpeed2).
Alternatively, you could use MFlowFPS once as you do now, and just use VarSpeed to do the frame decimation.

A few comments on your functions:
- YV12 input is always converted to YUY2, this is unnecessary (and undesirable);
- parameters are optional (names in quotes), but no defaults are provided;
- parameter names are confusing - startPercent and endPercent are actually ratios, not percentages;
- the final AssumeFPS is unnecessary (already done earlier);
- the last frame is Framecount-1, not Framecount;
- best to avoid 'last' as a user variable, since used specially by Avisynth;
- why two functions? could have single function and base operation on relation between start and end speed.
- I tried your example and it loses frame 18 from the output (perhaps related to Framecount issue).

I hope you find these comments helpful and constructive.

[lecapitan]

First, thanks for your insight. Here goes:


I think your 'variable-rate frame decimation' algorithm is equivalent to animating ChangeFPS which I explained in this post.

I have been using this function in my own projects:


I seemed to have trouble with Animate where it would not give me the correct amount of frames in the final clip. I would get a partial slowdown effect, but the framecount would be the same (or sometimes less) as the original clip, and a good chunk of the slow frames would be lost. It was probably my fault for not understanding Animate fully.


Your function uses MVTools for superior frame interpolation, while mine just uses ChangeFPS.
However, my function VarSpeed2 could easily be changed to use MFlowFPS instead (calculate the vectors once in VarSpeed and pass them as parameters to VarSpeed2).
Alternatively, you could use MFlowFPS once as you do now, and just use VarSpeed to do the frame decimation.


I've thought about it, and you are certainly right. The only rebuttal I have is, as far as I know, Animate is a linear function. I think to make a nice transition from fast to slow, a quadratic equation should be used. Using a quadratic equation makes for a nice ease-in and out effect.


A few comments on your functions:
- YV12 input is always converted to YUY2, this is unnecessary (and undesirable);


I admit, I don't fully understand color spaces, and I'm not sure what you would have me do to correct this. All I know is the MVTools functions complain if I do not convert the clip to YUY2.


- parameters are optional (names in quotes), but no defaults are provided;


Oops, they are not optional, all parameters are required. I should remove the quotes.


- parameter names are confusing - startPercent and endPercent are actually ratios, not percentages;


Percentages and ratios are the same things right? (lol) You're right since I'm expecting a number from 0-1 it should be named ratio instead of percent.


- the final AssumeFPS is unnecessary (already done earlier);


I found that the MVTools functions must play with the framerate, because without the AssumeFPS, Avisynth would complain that the clips were not the same format when I tried to concatenate them.


- the last frame is Framecount-1, not Framecount;


Oops.


- best to avoid 'last' as a user variable, since used specially by Avisynth;


Oops again.


- why two functions? could have single function and base operation on relation between start and end speed.


Just because of the extra checking I would have to do inside the loop to swap the functionality. Possibly lazy programming, possibly making the function easier to understand.


- I tried your example and it loses frame 18 from the output (perhaps related to Framecount issue).


How can you tell? If you mean that the 18th frame from the original never appears in the output clip, it might be because it was removed. As much as I'd like to ensure at least all the original frames are kept, I have no idea how to go about doing that.


Thanks again. Very useful information. I've used it to clean up the functions a little bit.

[Gavino]

I seemed to have trouble with Animate where it would not give me the correct amount of frames in the final clip. I would get a partial slowdown effect, but the framecount would be the same (or sometimes less) as the original clip, and a good chunk of the slow frames would be lost.
There are some subtle points that you need to be aware of.
Firstly, to get the right frames selected by ChangeFPS at each point during the animation, it needs to be given the average speed for the animation so far, not the instantaneous speed at that point. So the animation in my function runs (somewhat counter-intuitively) from starting speed v1 to final average speed (v1+v2)/2.
Secondly, Animate always returns the same number of frames as its input, so I work out the required number of output frames and Trim the Animate result to obtain them, while also looping the input to ensure (when slowing down, ie adding frames) that there will always be sufficient frames for the trim to succeed.

The only rebuttal I have is, as far as I know, Animate is a linear function. I think to make a nice transition from fast to slow, a quadratic equation should be used. Using a quadratic equation makes for a nice ease-in and out effect.
You may be right. It might be possible to amend my VarSpeed2 function to give a quadratic effect (by using some non-linear function of the parameter instead of the parameter directly), but I'd need to think about that.

I admit, I don't fully understand color spaces, and I'm not sure what you would have me do to correct this. All I know is the MVTools functions complain if I do not convert the clip to YUY2.
MVTools also accepts YV12 input, so it's only RGB that is a problem.
Most functions place the onus on the user to provide correct input. It's regarded as bad practice to convert the input behind the user's back, so to speak, especially if you don't convert back afterwards.

I found that the MVTools functions must play with the framerate, because without the AssumeFPS, Avisynth would complain that the clips were not the same format when I tried to concatenate them.
But after the MVTools functions, you call AssumeFPS twice with the same parameters and nothing is done to the framerate in between.

How can you tell? If you mean that the 18th frame from the original never appears in the output clip, it might be because it was removed.
Your example starts with Slowmoe(source, 0, 18, 1.0, 1.0), so I would expect that bit to produce frames 0-18 unchanged. I used ShowFrameNumber() on the input in my test, and could see that frame 18 was missing (and still is, using your latest version).

[lecapitan]

There are some subtle points that you need to be aware of.
Firstly, to get the right frames selected by ChangeFPS at each point during the animation, it needs to be given the average speed for the animation so far, not the instantaneous speed at that point. So the animation in my function runs (somewhat counter-intuitively) from starting speed v1 to final average speed (v1+v2)/2.
Secondly, Animate always returns the same number of frames as its input, so I work out the required number of output frames and Trim the Animate result to obtain them, while also looping the input to ensure (when slowing down, ie adding frames) that there will always be sufficient frames for the trim to succeed.


This confuses the heck out of me. I understand the part where you're trimming extra frames in for Animate, and this is probably where I failed in my attempts.


MVTools also accepts YV12 input, so it's only RGB that is a problem.
Most functions place the onus on the user to provide correct input. It's regarded as bad practice to convert the input behind the user's back, so to speak, especially if you don't convert back afterwards.


I had to read this a few times and think about what you originally said that threw me off course. Anyways, now I get it. You've explained this in such a roundabout way... but yes I can move that yuy2 conversion outside of my functions.


But after the MVTools functions, you call AssumeFPS twice with the same parameters and nothing is done to the framerate in between.


Aha, you're right. I was just making doubly sure :cool:


Your example starts with Slowmoe(source, 0, 18, 1.0, 1.0), so I would expect that bit to produce frames 0-18 unchanged. I used ShowFrameNumber() on the input in my test, and could see that frame 18 was missing (and still is, using your latest version).


This is something of a hack, and I haven't decided how to fix this yet. Basically, I wanted MVTools to interpolate in between all of the frames, but I didn't want to duplicate the frames when the Slowmoe function was used back to back. So, right now, the function deletes the last frame.

[hristoff2]

Only posting this so people coming from google can see this. I modified Gavino's code a bit to fit my needs.

- Added linear=false.
- Added version in which you can define three zones, for example deceleration, constant, acceleration.


Example: VarSpeedAdv(0,10000,10150,0,150,1,150)
Will result in frame 0 to 10000 being scaled linearly from speed 150 to speed 1 (normal), play constant at speed 1 in between frame 10000 and 10150 and then again speed up to factor 150 towards the last frame.



function VarSpeed(clip c, float v1, float v2) {
av = 0.5*(v1+v2)
count = round(c.FrameCount()/av)
Animate(c.Loop(), 0, count-1, "VarSpeedHelp", v1, av).Trim(0, count-1)
}

function VarSpeedAdv(clip clp, int startframe, int conststart, int constend, int endframe, float vin, float vconst, float vout) {

clp1 = clp.Trim(startframe,(conststart-1))
clp2 = clp.Trim(conststart,constend)
clp3 = clp.Trim((constend+1),endframe)

av1 = 0.5*(vin+vconst)
av2 = 0.5*(vconst+vconst)
av3 = 0.5*(vconst+vout)

count1 = round(clp1.FrameCount()/av1)
count2 = round(clp2.FrameCount()/av2)
count3 = round(clp3.FrameCount()/av3)

animate1 = Animate(clp1.Loop(), 0, count1, "VarSpeedHelp", vin, av1).Trim(0, count1-1)
animate2 = Animate(clp2.Loop(), 0, count2, "VarSpeedHelp", vconst, av2).Trim(0, count2-1)
animate3 = Animate(clp3.Loop(), 0, count3, "VarSpeedHelp", vconst, av3).Trim(0, count3-1)

splice = animate1+animate2+animate3
return splice
}

#helper function:
function VarSpeedHelp(clip c, float v) {
r = c.Framerate()
c.AssumeFPS(r*v)
ChangeFPS(r,linear=false)
}