AI EXTRApolation? (adding frames to the end of a video)

[wonkey_monkey]

I've been looking at the likes of DAINapp and FlowFrames that do pretty amazing things when it comes to interpolating new frames in a video, but is there anything that will extrapolate new frames?

I have a short, fast-moving, fairly motion-blurred clip (a camera being pulled back through a forest) and I want to add just two frames to the end of it that continue the motion. There's a lot of parallax because of all the trees at different distances to the camera.

Any suggestions?

[AVIL]

Hi:

A mad idea. With mvtools, get the motion vector of the previous to the last frame of clip and apply to mcompensate the last frame (e.g. by using a fake clip.trim(clip,1,0).duplicateframe(last)). Repeating the procedure with the newly created frame obtain a second "interpolated" frame (hopely). But no AI involved, Probably no Natural Intelligence involved too.

[StainlessS]

This of any use at all ? [only for single frame prediction]

You beat me to it (was busy with an edit).

Might be able to use these (manually),


Function PredictFromPrevious(clip Last)
{ # Predict i frame from Previous
yuy2=(isYUY2())?true:false
PreFilt=Last.DeGrainMedian() # prefiltered for better motion analysis
Last= (yuy2)?Interleaved2planar():Last
PreFilt= (yuy2)?PreFilt.Interleaved2planar():PreFilt
super=Last.MSuper(planar=yuy2)
superPreFilt=PreFilt.MSuper(planar=yuy2)
fv = superPreFilt.MAnalyse(isb = false, truemotion=true,delta = 1)
Last.MFlow(super,fv,planar=yuy2)
Last=(yuy2)?Planar2Interleaved():Last
Return Last
}


Function PredictFromNext(clip Last)
{
yuy2=(isYUY2())?true:false
PreFilt=Last.DeGrainMedian() # prefiltered for better motion analysis
Last= (yuy2)?Interleaved2planar():Last
PreFilt= (yuy2)?PreFilt.Interleaved2planar():PreFilt
super=Last.MSuper(planar=yuy2)
superPreFilt=PreFilt.MSuper(planar=yuy2)
bv = superPreFilt.MAnalyse(isb = true, truemotion=true,delta = 1)
Last.MFlow(super,bv,planar=yuy2)
Last=(yuy2)?Planar2Interleaved():Last
return Last
}


And use ClipClop() plug to select replacement frames from generated predicted clips.

EDIT: Returns clip where all frames predicted from previous/next frame.

EDIT: or bit more from elsewhere,

Here a modification of a script function posted by Gavino (Thankyou Maestro)
http://forum.videohelp.com/threads/322848-Scripts-and-filters-for-an-Opera-%28sample-videos-included%29?p=2089696&viewfull=1#post2089696
EDIT: Update version of below DoctorFrames included in zip, with frame interpolations up to 22 frames.
Function DoctorFrames(clip c, String "Scmd",String "Cmd",bool "Show",int "dv") {
# Replace damaged single frames using commands in a either command string or file.

Scmd = Default(Scmd,"") # User supplied list of newline or semicolon separated commands.
Cmd = Default(Cmd,"") # User supplied Filename containing newline separated commands.
Show = Default(Show,false) # Show Info on frame
dv = Default(dv,0) # ClipClop DebugView level (Need DebugView utility)

sup = c.MSuper()
PreFilt = c.DeGrainMedian() # prefiltered for better motion analysis
supPreFilt = PreFilt.MSuper()

ci_bv = sup.MAnalyse(isb=true, delta=2)
ci_fv = sup.MAnalyse(isb=false, delta=2)
ci = c.MFlowInter(sup,ci_bv, ci_fv, time=50.0, ml=100).DuplicateFrame(0).Trim(0,c.Framecount()-1)

cp = c.SelectEvery(1,-1).Trim(0,c.Framecount()-1) # chop off extra frame
cn = c.SelectEvery(1,1).DuplicateFrame(c.Framecount()-1) # Make same length as source

pp_fv = supPreFilt.MAnalyse(isb = false, truemotion=true,delta = 1)
pp=c.MFlow(sup,pp_fv)

pn_bv = supPreFilt.MAnalyse(isb = true, truemotion=true,delta = 1)
pn=c.MFlow(sup,pn_bv)

NickName=" # Define Command mnemonics allowed in command string and file.
CI=1 # CopyFromInterpolated
CP=2 # CopyFromPrevious (frame n replaced with frame n-1)
CN=3 # CopyFromNext (frame n replaced with frame n+1)
PP=4 # PredictFromPrevious (frame n replaced with frame predicted from frame n-1)
PN=5 # PredictFromNext (frame n replaced with frame predicted from frame n+1)
"
Return c.ClipClop(ci,cp,cn,pp,pn,scmd=Scmd,cmd=Cmd,nickname=Nickname,show=show,dv=dv)
}

Avisource("D:\avs\avi\TEST.AVI")
Trim(1000,-10) ++ trim(2000,-10) ++ trim(3000,-10) # Make two scene changes

SCMD="
CI 5 # CopyFromInterpolated clip frame 5
CP 9 # CopyFromPrevious at last frame before scene change
CN 10 # CopyFromNext at first frame of scene change
CI 15 # CopyFromInterpolated clip frame 15
PP 19 # PredictFromPrevious at last frame before scene change
PN 20 # PredictFromNext at first frame of scene change
CI 25 # CopyFromInterpolated clip frame 25
"

DoctorFrames(Scmd,show=true)

Function Could easily be extended.

Thread on fixing broken frames here:-
http://forum.doom9.org/showthread.php?t=152758

EDIT:- For single frame repair only.
EDIT:- Easy to mod for eg frame range denoising.
EDIT:- @IanB & Gavino, Thanx for impelling the array of clips, would never have gotten here otherwise.

EDIT: SCMD using mnemonics is equivalent to this (the mnemonic just an alternative means of giving clip index number)

SCMD="
1 5 # CopyFromInterpolated clip frame 5
2 9 # CopyFromPrevious at last frame before scene change
3 10 # CopyFromNext at first frame of scene change
1 15 # CopyFromInterpolated clip frame 15
4 19 # PredictFromPrevious at last frame before scene change
5 20 # PredictFromNext at first frame of scene change
1 25 # CopyFromInterpolated clip frame 25
"

[poisondeathray]

deblurring the motion blur beforehand (using various "AI" methods) will probably help improve the quality of fwd vector interpolation. You can add back blur after to match with original frames

[poisondeathray]

Here is an example of forward prediction . t+1 derived from t-1 and t0 - with a usable working demo

https://github.com/megvii-research/CVPR2023-DMVFN



(apngs, should animate in most browsers)
demo example - axial object rotation looks decently predicted

https://i.postimg.cc/xC7frRMz/demo1-apng.png (https://postimages.org/)

example on different data set
<edit> see post 11 below, added t+2 frame




I have a short, fast-moving, fairly motion-blurred clip (a camera being pulled back through a forest) and I want to add just two frames to the end of it that continue the motion. There's a lot of parallax because of all the trees at different distances to the camera.


Not sure how well it would work on motion blurred clip tree clip , but you would likely improve results by de-motionblurring, inference, then adding the blur back if desired

Another tool in the tool belt - other possible uses are fixing bad splices/ scene changes (maybe reverse the sequence after the scene change), extending a scene .

[Emulgator]

Wow. Even the torn ropes/cables look convincing.

[Frank62]

Very interesting!
How long does it take to predict let's say one frame in HD?

[poisondeathray]

How long does it take to predict let's say one frame in HD?

Maybe a few seconds . It's not as fast as mvtools2 , but not as slow in the order of something like ESRGAN


Another machine learning prediction project
https://github.com/YueWuHKUST/CVPR2022-Optimizing-Video-Prediction-via-Video-Frame-Interpolation

I haven't had a change to play with this one yet. It's linux only and I have to break out a VM

[Frank62]

Pretty fast. Did anyone ever try to predict more than one frame? If so: how does it look? Dramatically worse or acceptable?

[Emulgator]

Simple. Your video ends at n_old.
Extrapolate the n_old+1 frame.
Add it to your source video.
Now it became n_new.
Then again extrapolate n_new+1.
There it is: your n_old+2.

[poisondeathray]

It gets worse and errors accumulate (as expected)

Here is the above with t+2 (from t and interpolated t+1) , and I'll remove the one above

https://i.postimg.cc/3JB7t6pb/demo2b-apng.png (https://postimages.org/)

General quality is worse than something like RIFE where you had before/after frames.

It also "fails" under the usual "fail" situations like repeating patterns, picket fences etc...

[Frank62]

Yes, I thought so. Would have been too magical. Thanks for the example.

[kolak]

If you don't have next frame (even with big distance gap) then it's all guess :)

[Frank62]

What is meanwhile the best way to interpolate a bigger gap? Let's say about 10-20 frames missing with some movement?

[poisondeathray]

What is meanwhile the best way to interpolate a bigger gap? Let's say about 10-20 frames missing with some movement?

Too many frames to "tween" in most cases - motion will look unnatural / robotic . (Unless those 10 -20 frames was pure linear motion)

Normally you would photoshop a few frames in between adjusted with motion for reference points

[Frank62]

Thanks! So nothing new. I thought maybe with A. I. there would be better chances for this meanwhile.

[DTL]

What is meanwhile the best way to interpolate a bigger gap? Let's say about 10-20 frames missing with some movement?

Engine should be time-axis symmetrical so to decrease possible errors of forward-only motion interpolation 2 directions may be used:

1. Get forward interpolated frame sequence from past good frames.
2. Get backward interpolated frame sequence from next good frames
3. Interleave 2 sequences and pass to some majority-based temporal 'denoiser' and it should select samples or blocks values of higher-probability so it may decrease number of interpolation errors.

Though for best results of majority-based processing better to have odd-numbered data samples per each output sample (sample or block depending on the engine - the vsTTempSmooth is example of per-sample engine and mvtools (MDegrainN pmode=1) is block-based).

So to attempt to get better results - several examples of interpolation from different interpolation engines may be feeded to majority-selector engine and it will try to output most-probable looking result. Per-sample based majority-selector mode expected in vsTTempSmoth in some close future after feature-request transferring from MDegrainN or other may be faster algorithm implementation.

[Frank62]

Thank you very much for your hints. I did similar things already some years ago and tested a lot of such strategies, also with different engines including Alchemist. But - logically - the quality of synthesised frames falls of exponentially when there is motion, even of bigger objects, no matter what you try. By the way I also preferred block based approaches in most cases. I also mixed several things, don't remember exactly,and spent weeks for this,with no bigger success.
I had not much hope that A.I. could improve this a lot, but at least a bit. But nothing new under the sun.

[poisondeathray]

But - logically - the quality of synthesised frames falls of exponentially when there is motion, even of bigger objects, no matter what you try.


Yes, and I think a big problem for this approach is using "generic" model for "everything". This gives mediocre results, unless the test material is similar to the training material

You generally see the best results in machine learning with a custom model matching the input characteristics . The 3 provided models are trained on limited, specific testing sets. If you train your own model based on similar source material, you will get better results for any of the machine learning algorithms



I had not much hope that A.I. could improve this a lot, but at least a bit. But nothing new under the sun.

I would still consider it some improvement, because there is less manual work. Instead of photoshoping every frame, you only have to touch up parts of frames. I consider it a useful time saver when using these tools.

The forward only prediction (from previous frame(s) ) of mvtools2 is significantly worse using any variant of mflow,mcompensate, mrecalculate than DMVFN

[Frank62]

If you train your own model based on similar source material, you will get better results for any of the machine learning algorithms
That's what I would like to also with colorisation. F. e. I have a smurf (yes, the small blue creatures) short feature from 1959 in Black & White. If I had any idea of this whole model thing I would train it with colored scenes from the later series where the smurfs are blue...
But unfortunately I haven't, and would need too much time to learn all this I fear.

[Emulgator]

Now that RIFE runs so nicely here I would like to return to that DMVFN (or any other) t+1 frame extrapolation attempt.
poisondeathray (or anybody), can you guide me to a working model/dependencies folder structure for AviSynth+64 ?

[poisondeathray]

Now that RIFE runs so nicely here I would like to return to that DMVFN (or any other) t+1 frame extrapolation attempt.
poisondeathray (or anybody), can you guide me to a working model/dependencies folder structure for AviSynth+64 ?

It has not been ported to avs+ or vpy yet.