Using avisynth for special interpolation

[Kim Brandstetter]

I want to interpolate frames of a video that looks like the following. It's 30 fps according to specs, but in reality has this order:

1. Movie picture
2. Clone of previous movie picture
3. Movie picture
4. Clone of previous movie picture
5. Clone of previous movie picture
6. Repeat from 1

equals

AABBBCCDDDEEFFFGGHHH....

where

A = first frame
B = second frame
C = third frame
D = ...



Could start with an offset.

Goal is to create a good* transitionary image from 1 to 3 and 3 to 1 of the next round, so it looks like 30fps when it's in fact less that 15 fps.

Like this

Movie picture 1
Morph/other transition method* between Movie picture 1 and Movie picture 2
Movie picture 2
2 frames Morph/other transition method between Movie picture 2 and Movie picture 3
Movie picture 3
Morph/other transition method between Movie picture 3 and Movie picture 4
Movie picture 4
2 frames Morph/other transition method between Movie picture 4 and Movie picture 5
Movie picture 5
Morph/other transition method between Movie picture 5 and Movie picture 6
Movie picture 6
2 frames Morph/other transition method between Movie picture 6 and Movie picture 7
Movie picture 7
etc.


How do I do that in avisynth?

* I love SVP. Maybe it's transition method can be used?

[Guest]

SelectEvery(5,0,2). Then convert frame rate as needed using your favorite method.

The local Avisynth Gods will arrive shortly, I suppose. :)

[Kim Brandstetter]

Thanks. Doesn't that speed up or slow down the video? The Clones of previous movie pictures are there to keep the video in sync with the audio and in general to keep the video speed in sync with reality.

[Guest]

You either accept the duplicates or you remove them and do something with the decimated stream. You could try some motion adaptive frame interpolation, but it would have to start from the decimated stream. Your base stream is 11.988 fps. You can play it at that rate or you can make up some new frames.

Have you looked at the available motion-compensated interpolators. I don't know much about it but I have heard of mvflow, etc.

[colours]

SelectEvery(5,0,2) automatically adjusts the video frame rate from 30 fps to 12 fps so there shouldn't be any audio desync.

Assuming the AABBB pattern is fixed throughout the whole video, you can use neuron2's suggestion as is, but if the pattern changes throughout the source, one way out is to use TDecimate(cycleR=3,cycle=5) or the slightly more accurate SelectEven().TDecimate(cycleR=1,cycle=5). These two alternatives also adjust the frame rate automatically.

This is if you're okay with 12 fps output; if you want motion-interpolated 24/30/60/whatever fps output, you'll have to get rid of all the duplicates first (using any of the three methods mentioned above) prior to motion interpolation.

[Emulgator]

You may try filldrops123.avsi
Copy, paste, rename as filldrops123.avsi, put it into your plugins folder and call function as filldrops3.
---------------------------------------

# johnmeyer's filldropsI (I for Interlaced) from 16.April 2011 based on MugFunky's filldrops see below, extended by Emulgator on 25.Feb.2012.
# http://forum.doom9.org/showthread.php?p=1493082#post1493082
# Automatic dropping of duplicated frames/fields followed by generation of interpolated frames/fields and reinsertion.
# Needs MVTools2, is more forgiving when duplicates do differ a little.
# johnmeyers's Difference was 0.1, deviation now tunable.
function filldropsI (clip c, float "dev")
{
dev=default(dev, 2.0)
even = c.SeparateFields().SelectEven()
super_even=MSuper(even,pel=2)
vfe=manalyse(super_even,truemotion=true,isb=false,delta=1)
vbe=manalyse(super_even,truemotion=true,isb=true,delta=1)
filldrops_e = mflowinter(even,super_even,vbe,vfe,time=50)

odd = c.SeparateFields().SelectOdd()
super_odd=MSuper(odd,pel=2)
vfo=manalyse(super_odd,truemotion=true,isb=false,delta=1)
vbo=manalyse(super_odd,truemotion=true,isb=true,delta=1)
filldrops_o = mflowinter(odd,super_odd,vbo,vfo,time=50)

global devthresh=dev

evenfixed = ConditionalFilter(even, filldrops_e, even, "YDifferenceFromPrevious()", "lessthan", "devthresh", show=true)
oddfixed = ConditionalFilter(odd, filldrops_o, odd, "YDifferenceFromPrevious()", "lessthan", "devthresh", show=true)

Interleave(evenfixed,oddfixed)
Weave()
}

#[*
# MugFunky's filldrops from 19.December 2005
# http://forum.doom9.org/showthread.php?p=753779#post753779
# "...so i wrote a little function to replace all frames that are identical to the previous one with a motion-interpolation of the adjacent frames.
# it could be useful for captures that you can't get again, but are full of drops..."
# Needs MVTools, duplicates must match perfectly, (useful only for capture drop&repeats).
function filldrops (clip c, float "dev")
{
vf=c.mvanalyse(truemotion=true,pel=2,isb=false,delta=1,idx=1)
vb=c.mvanalyse(truemotion=true,pel=2,isb=true,delta=1,idx=1)
global filldrops_d = c.mvflowinter(vb,vf,time=50,idx=1)
global filldrops_c = c
global devthresh=dev

c.scriptclip("""ydifferencefromprevious()<devthresh? filldrops_d : filldrops_c""")
}
#*]

# MugFunky's filldrops from 19.December 2005, extended by Emulgator on 25.Feb.2012
# http://forum.doom9.org/showthread.php?p=753779#post753779
# "...so i wrote a little function to replace all frames that are identical to the previous one with a motion-interpolation of the adjacent frames.
# it could be useful for captures that you can't get again, but are full of drops..."
# Needs MVTools2, deviation now tunable.
function filldrops2 (clip c, float "dev")
{
dev=default(dev, 2.0)
super=MSuper(c,pel=2)
vf=manalyse(super,truemotion=true,isb=false,delta=1)
vb=manalyse(super,truemotion=true,isb=true,delta=1)

global filldrops_d = mflowinter(c,super,vb,vf,time=50)
global filldrops_c = c
global devthresh=dev

c.scriptclip("""ydifferencefromprevious()<devthresh? filldrops_d : filldrops_c""")
}


# filldrops3 based on MugFunky's filldrops from 19.December 2005, extended by Emulgator on 26.Feb.2012 to accommodate interpolation of Doupledrops
# http://forum.doom9.org/showthread.php?p=753779#post753779
# "...so i wrote a little function to replace all frames that are identical to the previous one with a motion-interpolation of the adjacent frames.
# it could be useful for captures that you can't get again, but are full of drops..."
# Needs MVTools2, deviation now tunable.
function filldrops3 (clip c, float "dev")
{
blksz=16 # Blocksize for Motion search. Default = 8. My Default 16: Delivers a smoother picture. 4: Gives too many artefacts with poorer source.
searchp=32 # Radius for Motion search. Default = 2, My Default = 32
bld=true # Decides what to do if a Scenechange is detected by the following parameters: true will blend, false will repeat previous frame.
SCDpb=800 # Default = 400. thSCD1. SAD (pixels*luma values) Threshold per Block. A Block exceeding this SAD is considered changed.
## SAD = Sum of Absolute Differences = Number of pixels per 8x8 block (=64) multiplied with their luma difference.
## SCDpb is always given related to 8x8 blocks and scaled internally to match other block sizes. SCDbp is thSCD1 and is evaluated before SCDpf=thSCD2
SCDpf=200 # 0-255, Default=130. thSCD2. Changed Blocks percentage threshold per frame. 0=0%, 127=50%, 255 = 100%.
## This Threshold sets which minimal percentage of changed blocks a frame has to contain to be considered as a scene change.

dev=default(dev, 2.0)
super=MSuper(c,pel=2)

back_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=true, delta=1, temporal=false, dct=0, divide=0, trymany=false)#delta=framedistance;isb=IS Backward
# Looks 1 frame ahead and calculates vectors backward to reference frame.
forw_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=false, delta=1, temporal=false, dct=0, divide=0, trymany=false)#search 3:=Exhaustive; 4:=Hex(default); 5:=UMH
# Looks 1 frame behind and calculates vectors forward to reference frame.
dblback_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=true, delta=2, temporal=false, dct=0, divide=0, trymany=false)#delta=framedistance;isb=IS Backward
# Looks 2 frames ahead and calculates vectors backward to reference frame. To be used when reference frame + 1 has to be excluded.
dblforw_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=false, delta=2, temporal=false, dct=0, divide=0, trymany=false)#search 3:=Exhaustive; 4:=Hex(default); 5:=UMH
# Looks 2 frames behind and calculates vectors forward to reference frame. To be used when reference frame -1 has to be excluded.

global fillsingdrops = MFlowInter(c,super,back_vec,forw_vec,time=50, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global filldoubdrops33 = MFlowInter(c,super,dblback_vec,dblforw_vec,time=33, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global filldoubdrops67 = MFlowInter(c,super,dblback_vec,dblforw_vec,time=67, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global oric=c
global devthresh=dev


#ConditionalFilter(c, c, fillsingdrops, "YDifferenceFromPrevious()", ">", "devthresh", show=true)##works for mending of single framerepeats
#ConditionalFilter(c, c, filldoubdrops33, "YDifferenceFromPrevious()>devthresh", "&&", "YDifferenceToNext()>devthresh", show=true)##wrong syntax

c.scriptclip("""(YDifferenceFromPrevious<devthresh) && (YDifferenceToNext<devthresh) ? (filldoubdrops33++filldoubdrops67.Trim(1,0)) : (YDifferenceFromPrevious<devthresh) ? fillsingdrops : oric""")

}

[Kim Brandstetter]

You may try filldrops123.avsi
Copy, paste, rename as filldrops123.avsi, put it into your plugins folder and call function as filldrops3.
---------------------------------------


super=MSuper(c,pel=2)



Thanks. MSuper is missing though.

PS: How to deactivate those annoying random questions?

[colours]

MSuper is part of MVTools; there's a patched build in Dither tools (http://forum.doom9.org/showthread.php?p=1386559#post1386559).

Be aware that unless you do some munging with variable frame rates (which is not recommended unless you really know what you're doing), there will be judder if you don't remove the duplicate frames prior to motion interpolation.

I assume the annoying questions go away after you hit some arbitrary post count; some other forum posters I asked said they didn't see the questions at all.

Edit: To make this work with one of the filldrops functions (without judder), you'd need to make the group of three identical frames a group of only two identical frames. (filldrops3 looks like it was designed to handle such groups of three, but that is not the right thing to do in this case.) TDecimate(mode=1) does exactly that, getting rid of the third frame. The script you use should be something like:

source
TDecimate(mode=1) # so instead of an AABBB pattern, you have AABB
filldrops3()

But then you might run into issues with multiple duplicate frames being detected etc., and the best option is just to decimate to 12 fps first.

source
SelectEven().TDecimate()
super = MSuper(last)
vecb = MAnalyse(super,isb=true)
vecf = MAnalyse(super,isb=false)
MFlowFps(last,super,vecb,vecf,num=30,den=1,blend=false) # replace num/den with the desired output frame rate

[Kim Brandstetter]

It works now. This is just amazing. Thanks to all of you for your quick support.

EDIT: See below.

[Kim Brandstetter]

I was wrong. filldrops3 does well at dropping cloned frames, but it does not interpolate between them. So what I got was 30fps while those 30fps are the previous pictures minus the cloned ones, so it is way faster.

Now I have this code

DirectShowSource("13. April 2014, 082538.AVI")
ConvertToYV12()
SelectEven().TDecimate()
super = MSuper(last)
vecb = MAnalyse(super,isb=true)
vecf = MAnalyse(super,isb=false)
MFlowFps(last,super,vecb,vecf,num=30,den=1,blend=false) # replace num/den with the desired output frame rate

But that creates a weirdly distorted video with artefacts all over the place. Also, I get 0.92 fps rendering time in virtualdub, which is bad.

[Richard1485]

but if the pattern changes throughout the source, one way out is to use TDecimate(cycleR=3,cycle=5) or the slightly more accurate SelectEven().TDecimate(cycleR=1,cycle=5).

I understand the first option but not the second. Might changes in the pattern not cause problems if at some point SelectEven() discards a frame that is not duplicated? I'm just curious. :)

[StainlessS]

DirectShowSource("13. April 2014, 082538.AVI")
ConvertToYV12()
SelectEven().TDecimate()
super = MSuper(last)
vecb = MAnalyse(super,isb=true)
vecf = MAnalyse(super,isb=false)
MFlowFps(last,super,vecb,vecf,num=30,den=1,blend=false) # replace num/den with the desired output frame rate


creates a weirdly distorted video

Perhaps try replace stuff in blue

super=MSuper(pel=1, hpad=0, vpad=0)
backward_1=MAnalyse(super, chroma=false, isb=true, blksize=16, blksizev=16, searchparam=3, plevel=0, search=3, badrange=(-24))
forward_1 =MAnalyse(super, chroma=false, isb=false, blksize=16, blksizev=16, searchparam=3, plevel=0, search=3, badrange=(-24))
backward_2 = MRecalculate(super, chroma=false, backward_1, blksize=8, blksizev=8, searchparam=0, search=3)
forward_2 = MRecalculate(super, chroma=false, forward_1, blksize=8, blksizev=8, searchparam=0, search=3)
MBlockFps(super, backward_2, forward_2, num=num, den=den, mode=0)


EDIT: replace num=30, den=1

should get rid of weird swirly stuff.

[colours]

First arrow is SelectEven, second is TDecimate (assuming it picks the correct frames).

AAABBCCCDD → AABCD → ABCD
ABBBCCDDDE → ABCDD → ABCD
AABBBCCDDD → ABBCD → ABCD
ABBCCCDDEE → ABCDE → ABCD or BCDE (note that A/E are duplicates across groups of five)
AABBCCCDDE → ABCCD → ABCD

The increased accuracy comes from nuking five sixths of the dupes in each ten-frame section without even needing to detect which frames are duplicates. The pattern change transitions might not be handled exactly, but stepping through the video to manually handle these is rather time-consuming.

Kim Brandstetter: You might also want to check if it's DirectShowSource causing the artifacts if StainlessS' suggestion doesn't help. Not likely, but it's not something to overlook.

[Richard1485]

The pattern change transitions might not be handled exactly, but stepping through the video to manually handle these is rather time-consuming.

Ah! So you're accepting a trade off: an overall increase in accuracy set against the risk of not handling some pattern changes exactly. Thanks for the explanation. Apart from that, I more or less saw what you were doing. :)

[StainlessS]

You might also want to check if it's DirectShowSource causing the artifacts

Yep, DirectshowSource can cause blending and other anomalies depening upon how it is set up (eg does AUTO deinterlace, inverse telecine + more), use AviSource or other, DirectShowSource only if nothing else works.

If the weird distortions you wrote of are sort of 'swirly', then MBlockFps instead of MFlowFps may get rid.

EDIT: The previously posted MBlockFPS script is product of some Avisynth wizard, not mine, dont recall where I got it but it
does generally produce less artifacts than the script snippet it replaces.

[Kim Brandstetter]

It is still juddering and disappointing.

So a general question: what can I expect from frame generation 12 fps -> 30 fps?

30 fps -> 60 fps is great but 12 fps -> 30 fps is not. Anybody can offer samples?

[StainlessS]

I think it likely that 12fps -> 24fps would be somewhat better than 12fps -> 30fps, the amount of movement
between frames at 12fps will be quite large and is asking rather a lot to smoothly generate flaw free video at 30fps.
The smaller movement differences between frames at 30fps make it much easier to 'guess' tweened frames to produce
60fps (it's also an exact multiple).

I've read good thinks about SVP but have never tried it.

Is 30fps an absolute requirement ?

[Kim Brandstetter]

I think it likely that 12fps -> 24fps would be somewhat better than 12fps -> 30fps, the amount of movement
between frames at 12fps will be quite large and is asking rather a lot to smoothly generate flaw free video at 30fps.
The smaller movement differences between frames at 30fps make it much easier to 'guess' tweened frames to produce
60fps (it's also an exact multiple).

I've read good thinks about SVP but have never tried it.

Is 30fps an absolute requirement ?

12 fps is almost unbearable.

[StainlessS]

Try 24fps, see if that is bearable.

[Kim Brandstetter]

Here's a testfile:http://www.mediafire.com/watch/durwstuwhk0b0dz/14._April_2014,_095208.AVI

Why don't you get the file and apply your ideas to it to tell me what works best?

[StainlessS]

It has already been suggested that you avoid Directshowsource, it perhaps would have been better to
upload the source as is, without any modification so that others have what you have.

[Kim Brandstetter]

It does not look different with DSS. But here you go, a fresh recording(the other one is 600MB):
http://www.mediafire.com/watch/durwstuwhk0b0dz/14._April_2014,_095208.AVI

[StainlessS]

Thanks, downloading sample (28MB), much nicer than 600MB :)

Incidentally, some time ago I had a user wanting to remove blending from a clip using Exblend(),
as it turned out, the blending originated in DirectShow and was absent when
Mpeg2Source() was used instead (mpeg 2).

[Kim Brandstetter]

So, what's your optimization idea?

[StainlessS]

Non I'm afraid.
I think I would just leave it at decimated rate of 12FPS (if all like provided sample).
Convert to 24fps is better than 30fps, but not good.
The MBlockFPS is both faster and better than MFlowFPS, but not good (faster probably due to
the bigger inital block size, and then MRecalculate at better precision).
Leaving at 12FPS retains the impression that is was taken from the back of a rodeo bull
and as they say, "a man on a galloping horse would not notice" (same goes for bull probably).

I just used Colours

SelectEven().TDecimate(cycleR=1,cycle=5)

for decimation, note you have 7 identical frames @ frame 46 of source.

I dont expect any miracles from Avisynth for the provided sample. (But perhaps others may prove me wrong)

EDIT: For other downloaders, the sample requires MJPG Motion Jpeg decoder.
EDIT: I used AviSource().

[colours]

Guess what, the source is VFR!

The average frame rate in the video is 11.8 fps, and DSS decides to convert it to 30 fps by adding in dupes (which are not in the file at all) because that was the frame rate it detected from the first frame. Looking at the timecodes reveals that the "true" frame rate is 12 fps with some frame drops (which explains why the average frame rate is slightly lower, and also the seven identical frames StainlessS pointed out).

FFVideoSource("file.avi",fpsnum=12,fpsden=1)

This works on the one sample you provided, but just in case the stars align in a specific manner and the "wrong" frames are selected, the following script works too.

FFVideoSource("file.avi",fpsnum=24,fpsden=1)
TDecimate(cycleR=2,cycle=4)

As TDecimate scans the whole video for metrics which is a bit overkill (since the duplicates in this case are exact duplicates), the video can be cropped first to speed things up a bit.

FFVideoSource("file.avi",fpsnum=24,fpsden=1)
Crop(0,0,64,64).TDecimate(cycleR=2,cycle=4,clip2=last)

Motion interpolation seems to be a lost cause here. Regardless of how bad you think 12 fps is, it's not feasible to upconvert your footage with minimal artifacts; the only real solution is to crank up the frame rate of your recording device (in which case the above snippets won't be relevant). You can try using SVP instead of MVTools, but I don't think the results would be much better.

One last option is to use blending instead of motion interpolation.

FFVideoSource("file.avi",fpsnum=24,fpsden=1)
Crop(0,0,64,64).TDecimate(cycleR=2,cycle=4,clip2=last)
ConvertFPS(60) # put desired frame rate here

[creaothceann]

Just a note, I've had far better results with DSS2 instead of DirectShowSource (when the source isn't VFR).

[foxyshadis]

Yeah, fcp's DSS2 mod is almost always better than DirectShowSource; thanks to the frame accuracy it has less timestamp weirdness for better VFR handling too.

[Kim Brandstetter]

FFVideoSource("file.avi",fpsnum=24,fpsden=1)
Crop(0,0,64,64).TDecimate(cycleR=2,cycle=4,clip2=last)
ConvertFPS(60) # put desired frame rate here

Thanks. I used it and the color scheme of windows changes in kmplayer. Why is that? Also, there is no audio and there seems to be no vsync. Apart from that, I like this solution the most.

[colours]

Replace

FFVideoSource("file.avi",fpsnum=24,fpsden=1)

with

FFVideoSource("file.avi",fpsnum=24,fpsden=1)
AudioDub(FFAudioSource("file.avi"))

to get audio. The colour scheme and lack of vsync aren't things I can help you with, sorry.

[Kim Brandstetter]

Replace

FFVideoSource("file.avi",fpsnum=24,fpsden=1)

with

FFVideoSource("file.avi",fpsnum=24,fpsden=1)
AudioDub(FFAudioSource("file.avi"))

Thanks.

[colours]

Thanks.


Actually, I noticed one of my earlier posts in this thread was kinda dumb and also bad advice, so let me retract that. (Specifically, the bit about using 24 fps instead of 12 fps; using 15 fps is better than either.) I just didn't bother editing the posts since I thought you had it settled already. Use this instead.

FFVideoSource("file.avi",fpsnum=15,fpsden=1)
AudioDub(FFAudioSource("file.avi"))
Crop(0,0,64,64).TDecimate(cycleR=1,cycle=5,clip2=last)
ConvertFPS(60) # put desired frame rate here

[Lirk]

You may try filldrops123.avsi
Emulgator, can you add option for blending mode as in ConvertFPS instead frame interpolation?

[StainlessS]

Further to Emulgator,

Note for your mugfunky/JohnMeyer mods, Time arg to MFlowInter is actually Float so might want to amend eg 33 and 67 to 1/3.0 and 2/3.0.

EDIT: Oops, I mean 100/3.0 and 200/3.0