Frame Interpolation using MVTools

[buletti]

Hi folks,

I'm trying to created a script to convert 24 fps to 30 fps. Instead of using a pulldown that is basically repeating existing frames to match the frame rate, I wanted to compute new, intermediate frames. Here's the script:

function convert24to30(clip clp) {
clp
AssumeFPS(24)      # treat 23.976 as 24.00

### segement the base clip in 6 clips of 4 frames
firstSet            = SelectEvery(24, 0,1,2,3)          # frame set of 4 frames
firstEndFrame       = SelectEvery(24, 3)                # remember last frame of the set

secondSet           = SelectEvery(24, 4,5,6,7)
secondEndFrame      = SelectEvery(24, 7)

thirdSet            = SelectEvery(24, 8,9,10,11)
thirdEndFrame       = SelectEvery(24, 11)

fourthSet           = SelectEvery(24, 12,13,14,15)
fourthEndFrame      = SelectEvery(24, 15)

fifthSet            = SelectEvery(24, 16,17,18,19)
fifthEndFrame       = SelectEvery(24, 19)

sixthSet            = SelectEvery(24, 20,21,22,23)
sixthEndFrame       = SelectEvery(24, 23)

### interpolate a fifth frame for the set of four frames 
newFrameFive        = generateFrame(firstEndFrame)
### create sequence 0,1,2,3,new frame
firstSetOfFive      = UnalignedSplice(firstSet, newFrameFive.AssumeFPS(firstSet))

newFrameTen         = generateFrame(secondEndFrame)
secondSetOfFive     = UnalignedSplice(secondSet, newFrameTen.AssumeFPS(secondSet))

newFrameFifteen     = generateFrame(thirdEndFrame)
thirdSetOfFive      = UnalignedSplice(thirdSet, newFrameFifteen.AssumeFPS(thirdSet))

newFrameTwenty      = generateFrame(fourthEndFrame)
fourthSetOfFive     = UnalignedSplice(fourthSet, newFrameTwenty.AssumeFPS(fourthSet))

newFrameTwentyfive  = generateFrame(fifthEndFrame)
fifthSetOfFive      = UnalignedSplice(fifthSet, newFrameTwentyfive.AssumeFPS(fifthSet))

newFrameThirty      = generateFrame(sixthEndFrame)
sixthSetOfFive      = UnalignedSplice(sixthSet, newFrameThirty.AssumeFPS(sixthSet))

### combine the 6 5-frame-sets
UnalignedSplice(firstSetOfFive, secondSetOfFive, thirdSetOfFive, fourthSetOfFive, fifthSetOfFive, sixthSetOfFive)

return last
}


function generateFrame(clip clp) {
global idx1         = 10 
idx1                = idx1 + 1
backward_vec        = clp.MVAnalyse(isb = true,  truemotion=true, blksize=8, overlap=4, pel=2, search=3,
\                                   searchparam=10, sharp=2, idx=idx1, dct=1)
forward_vec         = clp.MVAnalyse(isb = false, truemotion=true, blksize=8, overlap=4, pel=2, search=3,
\                                   searchparam=10, sharp=2, idx=idx1, dct=1)
newFrame            = clp.MVFlowInter(backward_vec, forward_vec, time=50.0, mL=80.0, thSAD=300, idx=idx1)

return newFrame
}

The script works fine in respect of increasing the frame rate. However, I'm wondering about the results of MVFlowInter. The generated frames look exactly the same as their originating frames. I cannot see that MVFlowInter creates a "picture at some intermediate time moment between current and next frame". Considering the "time" parameter I expected a frame that is temporally located in the middle between "current" and "next". Instead I just get another "current".
Any hints are appreciated.

[Didée]

function generateFrame(clip clp) {
# global idx1     = 10  ## <-- move this assignment to the very start your script! (NOT inside of any function{} block.)
global idx1       = idx1 + 1  ## <-- add this "global" verb
...etc...

Try like that. You original code did reset the global idx to 10 every time when generateFrame() was called. That caused the wrong subpel clip to be used ... there it went down the drain.

[Alex_ander]

@buletti
Why don't you use MVFlowFPS() instead? It will rebuild all frames and that's necessary here.

[buletti]

@Didée: Thanks for the hint about the global idx variable. However, I still can not get the expected result. Using the script in FFDShow seems to merely repeat every fourth frame. Using the script in AvsP or VirtualDub gives strange results likely due to the wild clip merging.

@Alex_ander: I wanted to use the script during playback in FFDShow. So, I assumed that interpolating 6 new frames could be feasible whilst interpolating 30 new frames with MVFlowFPS is not. However, why is it a bad idea to reuse the existing 24 frames?

[Sagekilla]

Why would interpolating 6 new frames with MVFlowFPS be impossible I've interpolated content from 30p to 60p before, I don't see why it can't do 24p to 30p.

[Gavino]

Originally Posted by buletti

Using the script in FFDShow seems to merely repeat every fourth frame.

Replace

UnalignedSplice(firstSetOfFive, secondSetOfFive, ..., sixthSetOfFive)

with

Interleave(firstSetOfFive, secondSetOfFive, ..., sixthSetOfFive)

The movement won't be as fluid as using MVFlowFPS but, as you say, the latter is too slow for real-time playback.

EDIT: No, Interleave done as above is still wrong because it takes one frame at a time from each group. What you need if you are going to use this approach is something like

c0 = SelectEvery(4, 0)
c1 = SelectEvery(4, 1)
c2 = SelectEvery(4, 2)
c3 = SelectEvery(4, 3)
new = ??? # what goes here?
Interleave(c0, c1, c2, c3, new)

where new is the new set of additional frames. But I don't think your GenerateFrame will give you the right frames anyway, so a rethink is required.

You're probably better off just using ChangeFPS or ConvertFPS.

[Sagekilla]

What do you mean? It's very possible to do RT MVFlowfps, if you're trying to do it for playback. Just use faster settings (pel=1, overlap=0, chroma=false, blksize=16) and it should work perfectly fine. You'll get more artifacts that way, but I don't think you'll actually notice them most of the time. Only if you're encoding would you avoid that..

[Alex_ander]

Originally Posted by buletti

... why is it a bad idea to reuse the existing 24 frames?

Because this would produce jerkyness due to local slowdown at each 5th frame (considering original motion with constant speed, e.g. panning) after setting all frames equidistantly. Speed variations would be more noticeable than with pulldown, which inserts a field (not full frames) into each pair of input frames.
It is quite possible to only insert interpolated frames e.g. first using ChangeFPS(30) for addition of repeated frames, then using this function:
http://forum.doom9.org/showthread.php?t=104294
which replaces repeating frames with ones created by MVFlowInt(), but I'd still prefer MVFlowFPS().