Replacing bad frames revisited [Archive]

View Full Version : Replacing bad frames revisited

zorr

7th April 2018, 15:31

There have been a couple of different scripts for replacing bad or duplicated frames. There's the filldrops script (original by Mug Funky, 2005 (https://forum.doom9.org/showthread.php?p=753779#post753779)) that works for the situation when one frame is duplicated. It uses MAnalyse with delta=1. MVTools2 documentation contains an example for replacing bad frames using MAnalyse with delta=2. Delta 2 is needed for the general case when the frame is bad and we cannot use any information from it.

Johnmeyer developed an improved function called replacebad (https://forum.doom9.org/showthread.php?p=1410106#post1410106) in 2010. In that thread Gavino gave an explanation why it's better: MFlowInter uses not only previous and next frame, but also frames next+delta and previous-delta when there is occlusion.

Let's say we have frames A-E and the frame C is bad:
0 1 2 3 4
A B C D E
X bad

These are the frames and the optimal forward / backward vectors we should use:

A B D E
---------------->
<----------------
--------> <--------

I tried to analyze what replacebad is doing and it goes like this: It first replaces the bad frame with a duplicate of the frame following it.

0 1 2 3 4
A B D D E

Then it creates a "previous frame"-clip by duplicating the first frame:

0 1 2 3 4 5
A A B D D E

The final phase is interpolation using that previous-frame clip using delta=1:

0 1 2 3 4 5
A B D D
-------->
<--------
--------> <--------

So to me it looks like it's using the frame D twice and not using the frame E at all. Am I correctly interpreting what is happening here?

To fix this, the frame 4 (D) should be replaced with frame 5 (E).

Here's a script which can be used to test the original behavior of replacebad (replaces frame C) and an improved version which replaces frames C and D. Set the BAD_FRAME variable to frame number you want replaced. To see the improved behaviour, comment / uncomment the first lines of replacebad_test accordingly. The results are different and to me the improved version indeed looks better. For comparison there's also recreateframes-function which is similar to the code used in MVTools2 documentation.

AVISource("d:\process2\1 deinterlaced.avi")

global BAD_FRAME = 11

return replacebad_test(last)
#return recreateframes(last)

function replacebad_test (clip c) {
# ORIGINAL This next statement replaces BAD_FRAME with a duplicate of the frame immediately after the BAD_FRAME
# goodframes = ConditionalFilter(c, trim(c,1,0), c, "current_frame", "equals", "BAD_FRAME")

# IMPROVED This next statement replaces BAD_FRAME with frame (BAD_FRAME+1) and the frame (BAD_FRAME+1) with frame (BAD_FRAME+2)
goodframes = ConditionalFilter(c, trim(c,1,0), c, "current_frame==BAD_FRAME || current_frame==BAD_FRAME+1", "equals", "true")

# This next statement gets the previous frame in the stream that now contains duplicates instead of bad frames
previousframe = Loop(goodframes,2,0,0)

super=MSuper(previousframe,pel=2)

vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)

replacement = mflowinter(previousframe,super,vbe,vfe,time=50)

fixed = ConditionalFilter(c, replacement, c, "current_frame", "equals", "BAD_FRAME")

return fixed
} # End function replacebad_test

function recreateframes(clip source)
{
super=source.MSuper(pel=2)
backward_vectors = MAnalyse(super, isb = true, truemotion=true, delta=2)
forward_vectors = MAnalyse(super, isb = false, truemotion=true, delta=2)
inter = source.MFlowInter(super, backward_vectors, forward_vectors, time=50)
return source.trim(0,BAD_FRAME-1) ++ inter.trim(BAD_FRAME-1,-1) ++ source.trim(BAD_FRAME+1,0)
}

Does anyone else see an improvement with this change?

johnmeyer

7th April 2018, 16:53

Oh, my head hurts reading that old thread where I was being taught by Gavino and Didée about how MflowInter() works. I almost figured it out then, but still don't fully get it.

I did, however, make another, much more robust replacement function and posted it in this thread:

Finding individual "bad" frames in video; save frame number; or repair (https://forum.doom9.org/showthread.php?t=174104)

You might want to take a look at both the code and the discussion in that thread to see if it helps you with what you are doing.

I'll have to actually try out your code to see what it is doing. I am not quite sure whether making additional duplicates to feed to the interpolation logic is going to produce better results, but I'll see what happens when I try it.

StainlessS

7th April 2018, 17:20

Well you beat me to it John, was looking for that very same thread.

I think this is probably a fair description:- http://forum.doom9.org/showthread.php?p=1789723#post1789723

But see entire thread.

johnmeyer

7th April 2018, 18:52

What I should do is to use AVISynth to insert a "bad"frame by simply replacing a frame with a black frame. However, I'll keep the original frame for reference. Then go through some of the same tests I did in that long-ago thread the OP referenced, but this time with the black frame as the reference frame. This will make obvious any issues that were "covered up" by replacing an exact duplicate because the metrics looking towards the duplicate will still produce a result that looks valid, but may not be optimal (which is what the OP is attempting to discover) and which may contain information from the bad frame which, of course, is not what we want.

I'll then compare the synthesized frame with the one that I removed and see how close I get. I guess I could create a metric for this difference (YDiff ...), but to keep it simple, I'll just look at it.

I did these tests before (as documented in that old post), but never did it using a bad frame which, now that I think about it, was stupid of me. How many times have I had to say that? (Don't answer.)

I think I have free time tomorrow, so I might get to it. I do know that the function I created, with your help StainlessS, that I linked to in my last post, does correctly replace bad frames, but whether it is optimal (trying hard not to say "best"), I don't yet know.

[edit]I am also still left wondering, in your explanation in that thread you just linked to (a portion of which is copied below) why the forward vector from G3 to G5 isn't simply the reciprocal of the backward vector from G5 back to G3. At the risk of "here we go again," I thought one of the vectors (with delta=2) should go from G5 to G3 and the other from G1 to G3.

<Sigh> I may never get this totally figured out. At least, like the bumblebee, my stuff may theoretically not be able to work, but the darn stuff still flies.

50%
|
<--------->
| |
v v
G1 G2 G3 B4 G5 G6
n n+1 n+2

zorr

7th April 2018, 21:19

I did, however, make another, much more robust replacement function and posted it in this thread:

Finding individual "bad" frames in video; save frame number; or repair (https://forum.doom9.org/showthread.php?t=174104)

You might want to take a look at both the code and the discussion in that thread to see if it helps you with what you are doing.

Thanks John. That script is certainly more advanced than the replacebad-function. Your script is solving (at least) two separate issues:
1) which frames are bad and thus should be replaced?
2) how to replace a bad frame?

What I'm mainly interested now is that second part. This newer script is actually doing the replacement using the "standard" way with delta=2. Are you saying this replacement method is working better than your earlier script? That's actually something I'm not sure about myself, I only noticed that my improved version was better than the original replacebad-method.

What I should do is to use AVISynth to insert a "bad"frame by simply replacing a frame with a black frame.
...
I'll then compare the synthesized frame with the one that I removed and see how close I get. I guess I could create a metric for this difference (YDiff ...), but to keep it simple, I'll just look at it.

That's a good idea. I have done something similar using LumaDifference, ChromaUDifference and ChromaVDifference inside FrameEvaluate. I have also used SSIM (https://en.wikipedia.org/wiki/Structural_similarity) which should give more realistic evaluation on how close those two frames are. If you're going to do that I recommend the v0.25.1 by mitsubishi (https://forum.doom9.org/showthread.php?p=1089303#post1089303) because it's the only version with SSIM_FRAME function and can be used like this:

FrameEvaluate(last, """
global ssim = SSIM_FRAME(orig, replaced)
global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)
""", args="orig,replaced")

That's maybe overkill if you're comparing one frame only.

[edit]I am also still left wondering, in your explanation in that thread you just linked to (a portion of which is copied below) why the forward vector from G3 to G5 isn't simply the reciprocal of the backward vector from G5 back to G3.

I think I can answer that. The forward vector from G3 to G5 is answering "how should pixels of G3 be moved in order to recreate frame G5?", whereas backward vector G5->G3 is answering "how should pixels of G5 be moved in order to recreate frame G3?". So the source pixels are different in the backward vector. Maybe someone more knowledgeable can confirm this?

StainlessS

7th April 2018, 21:45

isn't simply the reciprocal of the backward vector

Perhaps in a perfect world it would be, but just because the algo predicts a motion block to
move in a particular directions and distance, it dont automatically follow that the reverse situation
will produce the exact inverse result (mistakes will likely happen sometimes). Having two sets
of vectors will be a sort of self check, if the results dont match well then is to some degree, unreliable
(and some blurring will be probably be incorporated in the result, based on quality of match).

the other from G1 to G3

Not sure at all how that would help predict frame at B4, but anyways, the G3->G5 and G5->G3 allow
to move vector clip results to the n frame of each vector clip, so they coincide for creating the
final result frame, to do anything other than was done in the plugin, would be one helluva a nightmare
for the user [ you think that its tricky now :) ], the filter does it in the most sane way that I could
think of and removes most of the difficulties that would ensue had it been done pretty much any other way.

Never mind how crazy it all seems, I made that post with a little trepidation, but the fact that Gavino did not
squash me like a bug, makes me think that it were not so far off target :)

EDIT: Zorr, perhaps of interest

FrameSurgeon:- https://forum.doom9.org/showthread.php?p=1755860#post1755860

DoctorFrames:- https://forum.doom9.org/showthread.php?p=1764743#post1764743

MorphDupes_MI:- https://forum.doom9.org/showthread.php?p=1764867#post1764867

Snippet of script from MorphDupes_MI which creates the MC Clips.

thSCD1=(8*8)*255 thSCD2=255 BLEND=False bs=(In.width>960) ? 16 : 8
supFilt = In.Blur(0.6).MSuper(pel=2,sharp=sharp,rfilter=rfilter,hpad=16, vpad=16)
sup = In.MSuper(pel=2,sharp=sharp,rfilter=rfilter,hpad=16, vpad=16, levels=1)
SC_fv=supFilt.MAnalyse(isb=false, delta=1,blksize=bs,overlap=bs/2)
SC_fv=MRecalculate(sup,SC_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global SC@@@=In.MSCDetection(SC_fv,thSCD1=SC_thSCD1,thSCD2=SC_thSCD2)
For(Bad=1,MaxInterp) {
Eval(RT_String("I%0.2d_bv=supFilt.MAnalyse(isb=true, delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1))
Eval(RT_String("I%0.2d_fv=supFilt.MAnalyse(isb=false,delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1))
Eval(RT_String("I%0.2d_bv=MRecalculate(sup,I%0.2d_bv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad))
Eval(RT_String("I%0.2d_fv=MRecalculate(sup,I%0.2d_fv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad))
for(i=1,Bad) {
Eval(RT_String("Global I%0.2d_%0.2d@@@=In.MFlowInter(sup,I%0.2d_bv,I%0.2d_fv,time=100.0*%d/%d,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)",
\ Bad,i,Bad,Bad,i,Bad+1))
}
}

zorr

7th April 2018, 21:47

I think this is probably a fair description:- http://forum.doom9.org/showthread.php?p=1789723#post1789723

But see entire thread.

Thanks StainlessS, your explanation makes perfect sense to me. I did look at the whole thread (and other threads linked from it), and I saw that the replacement was done with delta=2 for one bad frame.

I couldn't find any mention of the two extra frames and vectors which are also used. That's the part that I'm focused on, because it's not clear to me which frames will be used when delta=2. Gavino said the extra frames are separated by delta, so with delta=2 that would mean frames n-3, n-1, n+1, and n+3 when replacing frame n. That doesn't seem optimal, n-2 and n+2 would be better than n-3 and n+3. And that's what replacebad-function was trying to do. Whether or not that actually improves quality is another question and should be tested.

StainlessS

7th April 2018, 22:24

From that thread
EDIT: Delta is always +ve, isb sets whether it is a forward or backward vector between n and n + delta.
(Worrying too much about what is in which particular vector clip, and at which frame number, will lead to madness, dont do it ,
but for the already insane, frame n vectors will contain vectors for n<->n+delta depending upon isb direction)

Showing what comes out of MorphDupes_MI for a simple "MaxInterp=5", ie maximum of 5 frames interpolated

Client

Colorbars.ConvertToYV12
MorphDupes_MI(MaxInterp=5)

Hacked part of previous snippet

ZZZ=""
For(Bad=1,MaxInterp) {
Q=RT_String("I%0.2d_bv=supFilt.MAnalyse(isb=true, delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1)
ZZZ=ZZZ+Chr(10)+Q
Eval(Q)
# Q=RT_String("I%0.2d_bv=supFilt.MAnalyse(isb=true, delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1)
# ZZZ=ZZZ+Chr(10)+Q
# Eval(Q)
Q=RT_String("I%0.2d_fv=supFilt.MAnalyse(isb=false,delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1)
ZZZ=ZZZ+Chr(10)+Q
Eval(Q)
Q=RT_String("I%0.2d_bv=MRecalculate(sup,I%0.2d_bv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad)
ZZZ=ZZZ+Chr(10)+Q
Eval(Q)
Q=RT_String("I%0.2d_fv=MRecalculate(sup,I%0.2d_fv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad)
ZZZ=ZZZ+Chr(10)+Q
Eval(Q)
for(i=1,Bad) {
Q=RT_String("Global I%0.2d_%0.2d@@@=In.MFlowInter(sup,I%0.2d_bv,I%0.2d_fv,time=100.0*%d/%d,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)",
\ Bad,i,Bad,Bad,i,Bad+1)
ZZZ=ZZZ+Chr(10)+Q
}
}
RT_DebugF("%s",ZZZ)

Result for only 5 frame interp (gets way bigger with eg up to Maxinterp=20 )

I01_bv=supFilt.MAnalyse(isb=true, delta=2,blksize=bs,overlap=bs/2)
#I01_bv=supFilt.MAnalyse(isb=true, delta=2,blksize=bs,overlap=bs/2)
I01_fv=supFilt.MAnalyse(isb=false,delta=2,blksize=bs,overlap=bs/2)
I01_bv=MRecalculate(sup,I01_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I01_fv=MRecalculate(sup,I01_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I01_01_MorphDupes_MI_1=In.MFlowInter(sup,I01_bv,I01_fv,time=100.0*1/2,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I02_bv=supFilt.MAnalyse(isb=true, delta=3,blksize=bs,overlap=bs/2)
#I02_bv=supFilt.MAnalyse(isb=true, delta=3,blksize=bs,overlap=bs/2)
I02_fv=supFilt.MAnalyse(isb=false,delta=3,blksize=bs,overlap=bs/2)
I02_bv=MRecalculate(sup,I02_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I02_fv=MRecalculate(sup,I02_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I02_01_MorphDupes_MI_1=In.MFlowInter(sup,I02_bv,I02_fv,time=100.0*1/3,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I02_02_MorphDupes_MI_1=In.MFlowInter(sup,I02_bv,I02_fv,time=100.0*2/3,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)
#I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)
I03_fv=supFilt.MAnalyse(isb=false,delta=4,blksize=bs,overlap=bs/2)
I03_bv=MRecalculate(sup,I03_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I03_fv=MRecalculate(sup,I03_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I03_01_MorphDupes_MI_1=In.MFlowInter(sup,I03_bv,I03_fv,time=100.0*1/4,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I03_02_MorphDupes_MI_1=In.MFlowInter(sup,I03_bv,I03_fv,time=100.0*2/4,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I03_03_MorphDupes_MI_1=In.MFlowInter(sup,I03_bv,I03_fv,time=100.0*3/4,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I04_bv=supFilt.MAnalyse(isb=true, delta=5,blksize=bs,overlap=bs/2)
#I04_bv=supFilt.MAnalyse(isb=true, delta=5,blksize=bs,overlap=bs/2)
I04_fv=supFilt.MAnalyse(isb=false,delta=5,blksize=bs,overlap=bs/2)
I04_bv=MRecalculate(sup,I04_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I04_fv=MRecalculate(sup,I04_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I04_01_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*1/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I04_02_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*2/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I04_03_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*3/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I04_04_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*4/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I05_bv=supFilt.MAnalyse(isb=true, delta=6,blksize=bs,overlap=bs/2)
#I05_bv=supFilt.MAnalyse(isb=true, delta=6,blksize=bs,overlap=bs/2)
I05_fv=supFilt.MAnalyse(isb=false,delta=6,blksize=bs,overlap=bs/2)
I05_bv=MRecalculate(sup,I05_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I05_fv=MRecalculate(sup,I05_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I05_01_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*1/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_02_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*2/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_03_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*3/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_04_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*4/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_05_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*5/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)

Have to get to pub before it shuts.

EDIT: If I recall correctly, Feisty2 said that Mrecalculate is not much use for framerate change and so expect to be same for
Interpolation, so maybe above MRecal stuff not necessary.

EDIT: Seems to be a cockup somewhere in the hack above, we got duplicates eg

I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)
I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)

Got to dash.
EDIT: Duplicate lines in red should be removed.

zorr

8th April 2018, 22:52

I decided to test these different replacement methods with the SSIM metric. Here's the script:

AVISource("d:\process2\1 deinterlaced.avi")

# SSIM needs YV12
ConvertToYV12()

orig = last

global ssim_total_orig = 0.0
global ssim_total_improved = 0.0
global ssim_total_standard = 0.0
global wins_orig = 0
global wins_improved = 0
global wins_standard = 0
frame_count = FrameCount()
ScriptClip(last, """
# replace frame
replaced_orig = replacebad_test(last, current_frame, false)
replaced_improved = replacebad_test(last, current_frame, true)
replaced_standard = recreateframes(last)
measure = (current_frame > 1 && current_frame < frame_count-2)

# compare to original with SSIM metric
global ssim_orig = (measure ? SSIM_FRAME(orig, replaced_orig) : 0.0)
global ssim_improved = (measure ? SSIM_FRAME(orig, replaced_improved) : 0.0)
global ssim_standard = (measure ? SSIM_FRAME(orig, replaced_standard) : 0.0)
global ssim_best = Max(ssim_orig, ssim_improved, ssim_standard)

# tag best SSIM result for each frame
global tag_orig = (ssim_best > 0.0) && (ssim_best == ssim_orig) ? "* " : " "
global tag_improved = (ssim_best > 0.0) && (ssim_best == ssim_improved) ? "* " : " "
global tag_standard = (ssim_best > 0.0) && (ssim_best == ssim_standard) ? "* " : " "

# calculate number of wins (best result for this frame)
global wins_orig = (ssim_best > 0.0) && (ssim_best == ssim_orig) ? wins_orig+1 : wins_orig
global wins_improved = (ssim_best > 0.0) && (ssim_best == ssim_improved) ? wins_improved+1 : wins_improved
global wins_standard = (ssim_best > 0.0) && (ssim_best == ssim_standard) ? wins_standard+1 : wins_standard

# calculate total SSIM
global ssim_total_orig = ssim_total_orig + (ssim_orig == 1.0 ? 0.0 : ssim_orig)
global ssim_total_improved = ssim_total_improved + (ssim_improved == 1.0 ? 0.0 : ssim_improved)
global ssim_total_standard = ssim_total_standard + (ssim_standard == 1.0 ? 0.0 : ssim_standard)

# tag best total SSIM result
global ssim_total_best = Max(ssim_total_orig, ssim_total_improved, ssim_total_standard)
global tag_total_orig = (ssim_total_best == ssim_total_orig) ? "* " : " "
global tag_total_improved = (ssim_total_best == ssim_total_improved) ? "* " : " "
global tag_total_standard = (ssim_total_best == ssim_total_standard) ? "* " : " "

return replaced_standard
""", args="orig, frame_count")

delimiter = "; "
resultFile = "ssimResults.txt"
WriteFileIf(resultFile, "current_frame == 0", """ "original replacebad" """, "delimiter", """ "improved replacebad" """,
\ "delimiter", """ "standard" """, "delimiter", append=false)
WriteFile(resultFile, "current_frame", "delimiter", "tag_orig", "ssim_orig", "delimiter", "tag_improved", "ssim_improved",
\ "delimiter", "tag_standard", "ssim_standard", "delimiter")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "wins " """, "wins_orig", "delimiter", "wins_improved",
\ "delimiter", "wins_standard", "delimiter", append=true)
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "total " """, "tag_total_orig", "ssim_total_orig",
\ "delimiter", "tag_total_improved", "ssim_total_improved", "delimiter", "tag_total_standard", "ssim_total_standard",
\ "delimiter", append=true)
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "average " """, "ssim_total_orig / (frame_count-4)",
\ "delimiter", "ssim_total_improved / (frame_count-4)", "delimiter", "ssim_total_standard / (frame_count-4)",
\ "delimiter", append=true)

return last

function replacebad_test(clip c, int frame, bool improved) {

# incoming frames: A B C D E F, where frame C is the current frame
# ORIGINAL replaces current frame with the next frame, other frames do not change: A B C D E F -> A B D D E F
# IMPROVED removes current frame: A B C D E F -> A B D E F
goodframes = improved ? (c.trim(0,frame-1) ++ c.trim(frame+1,0)) :
\ (c.trim(0,frame-1) ++ c.trim(frame+1,frame+1) ++ c.trim(frame+1, 0))

# This next statement gets the previous frame in the stream
previousframe = Loop(goodframes,2,0,0)

super=MSuper(previousframe,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
replacement = mflowinter(previousframe,super,vbe,vfe,time=50)

# clip has to be the same length as the original for SSIM to work
return replacement.trim(0,c.FrameCount()-1)
} # End function replacebad_test

# STANDARD replacement method with delta=2
function recreateframes(clip source)
{
super=source.MSuper(pel=2)
backward_vectors = MAnalyse(super, isb = true, truemotion=true, delta=2)
forward_vectors = MAnalyse(super, isb = false, truemotion=true, delta=2)
inter = source.MFlowInter(super, backward_vectors, forward_vectors, time=50)
previousframe = Loop(inter,2,0,0)
return previousframe.trim(0,source.FrameCount()-1)
}

It creates a text file ssimResults.txt and writes for each frame the SSIM metric for each of the three methods (original replacebad, my improved replacebad and the standard method with delta=2). It also calculates and writes the total and average SSIM and how many "wins" each method gets (ie how many times each method had the largest SSIM value for the frame). The winning value for each frame is also tagged with a "*" to make it easier to parse visually. The SSIM metric is not calculated for the first and last two frames because those frames don't have enough neighbour frames to do a proper interpolation.

Here's an example result for a short clip:
original replacebad; improved replacebad; standard;
0; 0.000000; 0.000000; 0.000000;
1; 0.000000; 0.000000; 0.000000;
2; 0.965214; * 0.965760; 0.964760;
3; * 0.920008; 0.919744; 0.919962;
4; 0.827575; 0.827842; * 0.829020;
5; 0.928627; 0.928224; * 0.929566;
6; 0.959854; 0.959874; * 0.960391;
7; 0.932269; 0.932424; * 0.933425;
8; * 0.906174; 0.905721; 0.905074;
9; 0.930301; 0.930339; * 0.931503;
10; 0.910099; 0.910200; * 0.911520;
11; 0.887747; * 0.888143; 0.888135;
12; 0.927442; * 0.928089; 0.928063;
13; 0.919194; 0.919065; * 0.920344;
14; 0.898007; 0.897815; * 0.898245;
15; 0.852154; 0.852494; * 0.853153;
16; 0.850728; * 0.850974; 0.850876;
17; 0.908898; * 0.909402; 0.908606;
18; 0.922686; 0.923325; * 0.924254;
19; 0.821544; 0.820911; * 0.821599;
20; 0.875906; * 0.876705; 0.876263;
21; 0.810928; 0.811540; * 0.813887;
22; 0.821736; 0.822203; * 0.823627;
23; 0.935320; * 0.935502; 0.935031;
24; 0.777279; 0.777546; * 0.779405;
25; 0.944397; * 0.944812; 0.944594;
26; 0.861217; 0.861562; * 0.863031;
27; 0.942816; * 0.943115; 0.941661;
28; 0.000000; 0.000000; 0.000000;
29; 0.000000; 0.000000; 0.000000;
wins 2; 9; 15;
total 23.238121; 23.243330; * 23.255993;
average 0.893774; 0.893974; 0.894461;

I tested it with about one minute long clip (about 3060 frames). This test video is streets of London shot hand-held in a moving car so it has a lot of interesting and challenging movement. The results are clear: standard method is clearly best (according to SSIM metric) and the worst is original replacebad.

original replacebad; improved replacebad; standard;
wins 389; 662; 2065;
total 2736.119385; 2736.775879; * 2739.429688;
average 0.894157; 0.894371; 0.895238;

The standard method won about 67,5% of the time, improved replacebad won 21,6% and original replacebad won 12,7%. The winning stats are revealing an important point: it's not enough to compare a single frame, with luck any of those methods could be a winner for that single frame.

It would be interesting to find out if these statistics are similar with other kind of videos.

It's still a bit of mystery how MVTools is using those extra frames, but at least I don't have to worry about it, it's doing a good job (at least for the case of single frame replacement).

StainlessS

10th April 2018, 11:57

It's still a bit of mystery how MVTools is using those extra frames

If its the below that you are talking about, [EDIT: Below from post #1]

A B D E
---------------->
<----------------
--------> <--------

It dont work like that, perhaps below better explains

A B C D E
<--------------->
<------------------------------>

To use [EDIT: sort of] like above (and predict C) then you would need,
B<->D with vectors result created @ same frame number as B (delta=2, shift forward 1 frame to bad position C),
and
A<->E with vectors result created @ same frame number as A (delta=4, shift forward 2 frames to bad position C),
and somehow mix both sets of results, the A<->E will nearly always produce worse result that B<->D, so I dont see the point.
MFlowInter only ever uses two frames to predict [EDIT: Interpolate] the result (delta is only the offset to select the 2nd src frame number used), it aint like eg MDegrainN where N pairs of vectors would be used for result.

EDIT: Both above assume arg ' time=50', ie 50.0% of the way between the source frames B,B+2 (ie @ B+1, where delta=2),
and 50.0% of the way between source frames A,A+4 (ie @ A+2, where delta=4).
EDIT: So, the forward shift in above cases are delta * 50.0% ie 2*0.5(=1) and 4*0.5(=2).

EDIT:
I couldn't find any mention of the two extra frames and vectors which are also used.
There are no extra frames nor vectors used. Below 2 source frames, 2 vectors. (modified from your source)

source=AviSource("...")

BAD_FRAME = 11

super=source.MSuper(pel=2)
backward_vectors = MAnalyse(super, isb = true, truemotion=true, delta=2)
forward_vectors = MAnalyse(super, isb = false, truemotion=true, delta=2)
inter = source.MFlowInter(super, backward_vectors, forward_vectors, time=50) # Interpolated, requires forward shift to correct position
BEFORE_BAD = source.trim(0,BAD_FRAME-1) # clip before bad frame
AFTER_BAD = source.trim(BAD_FRAME+1,0) # clip after bad frame
# Using Trim and Splice instead of shift, but exactly the same result.
# frame number where vectors coincide @ BAD_FRAME - 1 (equivalent to forward shift of 1 when spliced)
# Forward shift (where delta=2 & time=50.0) @ 2 * 0.5 = 1
# BAD_RELATIVE = -(2*0.5) = -1 == BAD_FRAME-1
FIXED_SHIFTED = inter.trim(BAD_FRAME-1,-1) # 2nd arg to trim of -1 means trim 1 frame
Return BEFORE_BAD ++ FIXED_SHIFTED ++ AFTER_BAD

2 vectors, backward_vectors, forward_vectors.
2 source frames, BAD_FRAME-1 and BAD_FRAME+1. (Each and every frame @ inter[n] is interpolated from source[n] and source[n+2], req fwd shift of 1)

zorr

10th April 2018, 21:20

MFlowInter only ever uses two frames to predict [EDIT: Interpolate] the result (delta is only the offset to select the 2nd src frame number used), it aint like eg MDegrainN where N pairs of vectors would be used for result.

...

There are no extra frames nor vectors used. Below 2 source frames, 2 vectors. (modified from your source)

Ahh, I didn't realize those extra frames / vectors weren't common knowlegde. Ok, let me make my case:

This message (https://forum.doom9.org/showpost.php?p=1409954&postcount=36) from Gavino is where I got that idea:

DidÃ©e's explanation is the same as mine (but clearer!), and would suggest that the delta=2 approach cannot be improved upon. Your results are therefore puzzling and invite further explanation, which I believe I have found.

By looking at the source code, I have discovered that for occlusion areas, MVFlowInter also uses information from the two further surrounding frames (current-delta and current+2*delta).
So the diagram becomes, when interpolating between C and C+d,
----> <----
C-d C C+d C+2d
----> <----
It is this additional information that will be more accurate when delta=1, explaining your findings and making your two-stage approach a better one. Well done.

I also looked at Pinterf's MVTools2 source and found this in MVFlowInter.cpp:

// Get motion info from more frames for occlusion areas
PVideoFrame mvFF = mvClipF.GetFrame(n, env);
mvClipF.Update(mvFF, env);// forward from prev to cur
mvFF = 0;
PVideoFrame mvBB = mvClipB.GetFrame(nref, env);
mvClipB.Update(mvBB, env);// backward from next next to next
mvBB = 0;

I tried to figure out which frames are referenced by that code but my C++ experience is pretty limited (and it doesn't help that for some reason the MVTools2 project doesn't open in Visual Studio 2017 Community edition).

Finally there's the experimental proof. The only difference in original and modified replacebad-function is that they have different frame at index 4 when the interpolation is done at index 2.

Original replacebad has frame D at index 4:
0 1 2 3 4 5
A B D D
? <-------> ?

Modified replacebad has frame E at index 4:
0 1 2 3 4 5
A B D E
? <-------> ?

I don't know how the extra vectors should be drawn, my original drawing of those was just an educated guess.

If MFlowInter would only use frames 2 and 3 then frame 4 should not have any effect on the result, but my SSIM test proved that it does (unless I made some horrific error in my code).

I should have been more clear in my first message, but this is why I strongly believe that MFlowInter is using more than two frames in the interpolation.

StainlessS

11th April 2018, 03:55

Zorr,
I thought I understood how it all worked, then you ruined it for me :)

Methoughts that forwards Vectors were at first badframe - 1, but are of course at first badframe - 1 + delta,
just when I had a nice little model in my head, of course they dont move about depending upon which mv func you intend to use,
what was I thinking. :(

Here a little folly I was playing with,
Below, SrcS and SrcE are the source frames for interpolation, XSTEP is the horizontal movement between frames.
Can go a bit wonky when XSTEP and BLKSZ are bigger than 8 or DELTA bigger than 2.
Forward and backwards vectors are aligned just to check them.

WID=640
HIT=64
XSTEP=8

DELTA=2
TIME=50

PAD=16
BLKSZ=8

MASK_MOTION = 0
MASK_SAD = 1
MASK_OCCLUSION = 2
MASK_HORIZONTAL = 3

#MASK_TYPE=MASK_MOTION
#MASK_TYPE=MASK_SAD
#MASK_TYPE=MASK_OCCLUSION
MASK_TYPE=MASK_HORIZONTAL

BlankClip(width=WID,height=HIT,Length=1,Color=$FFFFFF,Pixel_type="Y8")

A=Trim(0,-1)
B=A.BlankClip
c=A.BlankClip(Length=0)
For(i=1,WID/XSTEP-1) {
L=A.Crop(0,0,i*XSTEP,0)
R=B.Crop(0,0,WID-i*XSTEP,0)
Frm = StackHorizontal(L,R)
c= c ++ Frm
}

srcS=c.ConvertToY8
srcE=SrcS.Loop(0,0,DELTA-1) # Delete Delta frames, end interp src

super = srcS.MSuper(pel=2,hpad=PAD,vpad=PAD)
fvec = MAnalyse(super,blksize=BLKSZ, isb = false, truemotion=true, delta=DELTA)
bvec = MAnalyse(super,blksize=BLKSZ, isb = true, truemotion=true, delta=DELTA)
inter = srcS.MFlowInter(super, bvec,fvec, time=TIME)

MBV = srcS.MMask(bvec,kind=MASK_TYPE)
MFV = srcS.MMask(fvec,kind=MASK_TYPE)
MFVDEL= MFV.Loop(0,0,DELTA-1) # Delete DELTA frames from start, align n+delta with n

StackVertical(srcS,SrcE,MBV,MFVDEL,Inter)
SSS="""
Subtitle(String(current_Frame,"SRC[n=%.0f] (1st interp srcS)") , Y=0.5*HIT,Align=5)
Subtitle(String(current_Frame+DELTA,"SRC[n+Delta=%.0f] (2nd interp srcE)") , Y=1.5*HIT,Align=5)
Subtitle(String(current_Frame,"BVEC[n=%.0f]")+MTyp(MASK_TYPE) , Y=2.5*HIT,Align=5)
Subtitle(String(current_Frame+DELTA,"FVEC[n+DELTA=%.0f]")+MTyp(MASK_TYPE) , Y=3.5*HIT,Align=5)
Subtitle(string(current_frame+DELTA*TIME/100.0,"Predicted @ [%.2f]") , Y=4.5*HIT,Align=5)
"""
Last.ScriptClip(SSS)
TXT=SrcS.BlankClip(Height=20,Length=1,Color=$404040)
TXT=TXT.Subtitle(String(Delta,"Delta=%.0f")+String(Time," : Time=%.2f")+String(XSTEP," : BLKSZ=%.0f")+String(XSTEP," : XSTEP=%.0f"))
StackVertical(TXT,Last)

Return Last.ConvertToRGB32

Function MTyp(Int n) {Return " Type="+Select(n,"Motion","Sad","Occlusion","Horizontal","Vertical","ColorMap")}

https://s20.postimg.cc/wtaszh59p/Delta2_50.jpg (https://postimages.cc/)

https://s20.postimg.cc/fg0ikmhod/Delta3_33.jpg (https://postimages.cc/)

https://s20.postimg.cc/9f2tnk2rx/Delta3_66.jpg (https://postimages.cc/)

EDIT: And just for good measure.

https://s20.postimg.cc/pq2xjvmzh/Delta1_50.jpg (https://postimages.cc/)

EDIT: Oops, had vectors labels wrong, fixed.
Backward/forward vectors seem back to front to me, my head hurts :(
Somebody interested in figuring out if the script is correct ???

johnmeyer

11th April 2018, 04:54

Zorr,
I thought I understood how it all worked, then you ruined it for me :).Almost 15 years using MVTools and, like you, every time I am absolutely certain I have the MAnalyze logic totally figured out, I find myself back at square one.

zorr

11th April 2018, 23:08

Zorr,
I thought I understood how it all worked, then you ruined it for me :)

Sorry about that, I have a bad habit of ruining people's lives. ;)

Methoughts that forwards Vectors were at first badframe - 1, but are of course at first badframe - 1 + delta,
just when I had a nice little model in my head, of course they dont move about depending upon which mv func you intend to use,
what was I thinking. :(

I don't know if this is going to help anyone but here's the internal model I'm currently working with:

Forward vectors clip
0 1 2 3 4 5
--0-----> --2-----> --4----->
--1-----> --3----->

Backward vectors clip

0 1 2 3 4 5
<-0------ <-2------ <-4------
<-1------ <-3------

MAnalyze returns a clip with motion vector data. So here we have forward and backward vectors with delta=1. The arrow points from the frame whose pixels are being motion estimated to the frame which is the motion estimation target. The number in the arrow tells in which frame the motion vector is stored at. So for example forward vector 1->2 is stored at frame 1 whereas backward vector 2->1 is stored at frame 1 as well. I agree with you that this arrangement makes using the vectors easier than pretty much any other way. For a good measure here's forward vectors with delta=2:

0 1 2 3 4 5 6
--0-------------> --3------------->
--1-------------> --4------------->
--2------------->

When MFlowInter uses these vectors it takes the forward AND backward vector at the current frame. It will also use the pixels in the current frame and current frame + delta. Now this is already enough to do motion interpolation, but it can also use other frames (and it does) because it has a whole clip of vectors it can use.

Here a little folly I was playing with,
Below, SrcS and SrcE are the source frames for interpolation, XSTEP is the horizontal movement between frames.
Can go a bit wonky when XSTEP and BLKSZ are bigger than 8 or DELTA bigger than 2.
Forward and backwards vectors are aligned just to check them.

That's a really interesting folly (just a side note: I'm learning a lot from your scripts so thank you).

I played with it and made a few changes that made sense to me. I think it's better to show those forward/backward vectors which are actually used to generate the interpolated frame, so I removed the frame adjustment of forward vectors and I swapped the display order (forward, then backward). Also there was a minor bug, XSTEP was displayed for the value of BLKSZ.

WID=640
HIT=64
XSTEP=8

DELTA=2
TIME=50

PAD=16
BLKSZ=2

MASK_MOTION = 0
MASK_SAD = 1
MASK_OCCLUSION = 2
MASK_HORIZONTAL = 3

#MASK_TYPE=MASK_MOTION
#MASK_TYPE=MASK_SAD
#MASK_TYPE=MASK_OCCLUSION
MASK_TYPE=MASK_HORIZONTAL

BlankClip(width=WID,height=HIT,Length=1,Color=$FFFFFF,Pixel_type="Y8")

A=Trim(0,-1)
B=A.BlankClip
c=A.BlankClip(Length=0)
For(i=1,WID/XSTEP-1) {
L=A.Crop(0,0,i*XSTEP,0)
R=B.Crop(0,0,WID-i*XSTEP,0)
Frm = StackHorizontal(L,R)
c= c ++ Frm
}

srcS=c.ConvertToY8
srcE=SrcS.Loop(0,0,DELTA-1) # Delete Delta frames, end interp src

super = srcS.MSuper(pel=2,hpad=PAD,vpad=PAD)
fvec = MAnalyse(super,blksize=BLKSZ, isb = false, truemotion=true, delta=DELTA)
bvec = MAnalyse(super,blksize=BLKSZ, isb = true, truemotion=true, delta=DELTA)
inter = srcS.MFlowInter(super, bvec,fvec, time=TIME)

MBV = srcS.MMask(bvec,kind=MASK_TYPE)
MFV = srcS.MMask(fvec,kind=MASK_TYPE)

StackVertical(srcS,SrcE,MFV,MBV,Inter)
SSS="""
Subtitle(String(current_Frame,"SRC[n=%.0f] (1st interp srcS)") , Y=0.5*HIT,Align=5)
Subtitle(String(current_Frame+DELTA,"SRC[n+Delta=%.0f] (2nd interp srcE)") , Y=1.5*HIT,Align=5)
Subtitle(String(current_Frame,"FVEC[n=%.0f]")+MTyp(MASK_TYPE) , Y=2.5*HIT,Align=5)
Subtitle(String(current_Frame,"BVEC[n=%.0f]")+MTyp(MASK_TYPE) , Y=3.5*HIT,Align=5)
Subtitle(string(current_frame+DELTA*TIME/100.0,"Predicted @ [%.2f]") , Y=4.5*HIT,Align=5)
"""
Last.ScriptClip(SSS)
TXT=SrcS.BlankClip(Height=20,Length=1,Color=$404040)
TXT=TXT.Subtitle(String(Delta,"Delta=%.0f")+String(Time," : Time=%.2f")+String(BLKSZ," : BLKSZ=%.0f")+String(XSTEP," : XSTEP=%.0f"))
StackVertical(TXT,Last)

Return Last.ConvertToRGB32

Function MTyp(Int n) {Return " Type="+Select(n,"Motion","Sad","Occlusion","Horizontal","Vertical","ColorMap")}

By the way, I had to run this script with GImport because it has GScript syntax (the for loop), is that how you run it or is there some other trick?

StainlessS

12th April 2018, 03:00

GScript syntax (the for loop)

I wrote in AVS+, which dont need Gscript("") stuff, you can wrap in a string eg SSS="""...""" and GScript(SSS).
EDIT: or Eval(SSS) for avs+.

XSTEP was displayed for the value of BLKSZ.
I guess thats down to copy/paste and being in a hurry to try get some sleep,
which is what I'm gonna try to do now, I'll have a play with mod tomorrow.

ta ta.

StainlessS

12th April 2018, 18:40

The motion vector stuff is stored at the frame that the vector arrow is pointing at, (which is what Didee said in one of the linked threads I think).

Dont really have time at the moment to go over your post, but will maybe later.

The below script reverses direction of animated clip (by default), and in doing so is less confusing, I think (movement comes in from same direction as the frames).

Function VectorTest(Int "Delta",Float "Time",Int "MaskT",Int "BlkSz",Int "XStep",Bool "Align",Bool "HFlip",String "CS") {
/*
VectorTest(), An MvTools2::MFlowInter folly. by StainlessS @ Doom9 : https://forum.doom9.org/showthread.php?t=175373

Req AVS+ or GSCript, GRunt, MvTools2, RT_Stats v1.43+
Avs v2.58, Avs/+ v2.60.

Delta, Default 1. As for MvTools2
Time, Default 50.0, As for MvTools2
MaskT, Default 3, As for MvTools2::MMask(kind=MASK_TYPE), 0=Motion, 1=Sad, 2=Occlusion, 3=Horizontal, 4=Vertical, 5=ColorMap
BlkSz, Default 4, As for MvTools2
XStep, Default 8, Motion per frame of synthesized clip.
Align, Default True, Aligns forward vector frame n+Delta to n. False show frame n of forward vector.
HFlip, Default False, False, Animate from Right to Left, else Left to Right.
(Right to Left is less confusing, same direction that the frames come in from).
CS, Avs v2.5 defaults "YV12" else "Y8".

Returns RGB32 clip.
*/
Function MTyp(Int n) {Return " Type="+Select(n,"Motion","Sad","Occlusion","Horizontal","Vertical","ColorMap")}
myName="VectorTest: "
IsAvsPlus=(FindStr(UCase(versionString),"AVISYNTH+")!=0) HasGScript=RT_FunctionExist("GScript")
HasGrunt=RT_FunctionExist("GScriptclip") HasMvTools2=RT_FunctionExist("MSuper") Is26=VersionNumber>=2.6
Assert(IsAvsPlus||HasGscript,myName+"Essential AVS+ or GScript installed")
Assert(HasGrunt,myName+"Essential GRunt installed") Assert(HasMvTools2,myName+"Essential MvTools2 installed")
Delta=Default(Delta,1) Time=Default(Time,50.0) MaskT=Default(MaskT,3) BlkSz=DefaulT(BlkSz,4)
XStep=Default(XStep,8) Align=Default(Align,True) HFlip=Default(HFlip,False)
CS=Default(CS,Is26?"Y8":"YV12")
OLap=(BlkSz>=4)?BlkSz/2:RT_Undefined
FuncS="""
Function Fn(clip c,Int Delta,Float Time,Int XStep,Bool Align,Bool HFlip,String mType) {
c n=current_frame
Hit=(Height-20)/5
if(Align) {
Steps=640/XStep
x=((n+1)*XStep) + (Delta*XStep/2)
x=Min(x,(Steps-1)*XStep)
x=HFlip?x:639-x
cF=RT_YPlaneMin(n=n,x=x,y=Round(2.5*Hit)+20,w=1,h=1)-128
cB=RT_YPlaneMin(n=n,x=x,y=Round(3.5*Hit)+20,w=1,h=1)-128
Z=c.BlankClip(width=1,height=1,Color=$FFFFFF,Length=1)
OverLay(Z,x=x,y=Round(2.5*Hit)+20)
OverLay(Z,x=x,y=Round(3.5*Hit)+20)
Subtitle(RT_String("@x=%d FGrey=128%+d : BGrey=128%+d",x,cF,cB))
} else {
Subtitle("Align=False, Colors NOT shown")
}
Subtitle(String(n,"SRC[n=%.0f] (1st interp srcS)") , Y=0.5*Hit+20,Align=5)
Subtitle(String(n+Delta,"SRC[n+Delta=%.0f] (2nd interp srcE)") , Y=1.5*Hit+20,Align=5)
(Align)
\ ? Subtitle(String(n+Delta,"FVEC[n+DELTA=%.0f]")+mType+ " (Aligned)" , Y=2.5*Hit+20,Align=5)
\ : Subtitle(String(n,"FVEC[n=%.0f]")+mType , Y=2.5*Hit+20,Align=5)
Subtitle(String(n,"BVEC[n=%.0f]")+mType , Y=3.5*Hit+20,Align=5)
Subtitle(string(n+Delta*Time/100.0,"MFlowInter Predicted @ [%.2f]") , Y=4.5*Hit+20,Align=5)
Return Last
}
Wid=640 Hit=64 Steps=Wid/XStep
White=BlankClip(width=WID,height=HIT,Length=1,Color=$FFFFFF,Pixel_type=CS) Black=White.BlankClip srcS=Black.BlankClip(Length=0)
For(i=1,Steps-1) { W=White.Crop(0,0,i*XStep,0) K=Black.Crop(0,0,Wid-W.Width,0) Frm=StackHorizontal(K,W) srcS=srcS++Frm }
srcS=(HFlip)?srcS.FlipHorizontal:srcS
srcE=SrcS.Loop(0,0,Delta-1) # Delete Delta frames, end interp src
super=srcS.MSuper(pel=2,hpad=16,vpad=16)
fvec =MAnalyse(super, isb=false, blksize=BlkSz, overlap=OLap, delta=Delta, truemotion=true)
bvec =MAnalyse(super, isb=true, blksize=BlkSz, overlap=OLap, delta=Delta, truemotion=true)
inter=srcS.MFlowInter(super, bvec,fvec, time=Time)
mbv=srcS.MMask(bvec,kind=MaskT) mfv=srcS.MMask(fvec,kind=MaskT)
mfv=(Align)?mfv.Loop(0,0,Delta-1):mfv # Align, Delete DELTA frames from start, align n+delta with n
TXT=SrcS.BlankClip(Height=20,Length=1,Color=$404040)
StackVertical(TXT,srcS,SrcE,MFV,MBV,Inter)
mType=mTyp(MaskT)
ARGS = "Delta,Time,XStep,Align,HFlip,MType"
Last.GScriptClip("Fn(last, "+ARGS+")", local=true, args=ARGS)
DIR=(HFlip) ? " : ---->" : " : <----"
TXT=TXT.Subtitle(String(Delta,"Delta=%.0f")+String(Time," : Time=%.2f")+
\ String(BLKSZ," : BLKSZ=%.0f")+String(XSTEP," : XSTEP=%.0f")+" : ALIGN="+String(ALIGN)+" : HFlip="+String(HFlip)+DIR)
return StackVertical(TXT,Last)
"""
IsAvsPlus?Eval(FuncS):GScript(FuncS)
Return Last.ConvertToRGB32
}

EDIT: Above changed Align default to True.
EDIT: Added 2nd line to title bar, not show in below graphics (see post #23).

VectorTest(Delta=2,Align=False)

https://s20.postimg.cc/bn5ioqdcd/Vector_Test_2_False.jpg (https://postimages.org/)
EDIT: Above, white stepping in from the Right.

EDIT: I still prefer Align=True for FVec. [EDIT: Align is now true by default]

VectorTest(Delta=2) # EDIT: NOW, Align=True is default

https://s20.postimg.cc/lyhvh24y5/Vector_Test_2_True.jpg (https://postimages.org/)

VectorTest(Delta=1) # EDIT: NOW, Align=true is default

https://s20.postimg.cc/i24jl4p4d/Vector_Test_1_True.jpg (https://postimages.org/)

VectorTest(Delta=2,time=33.33)

https://s20.postimg.cc/9l9mo124d/Vector_Test_33.jpg (https://postimages.org/)

VectorTest(Delta=2,time=66.66)

https://s20.postimg.cc/5c4wm041p/Vector_Test_66.jpg (https://postimages.org/)

EDIT: Script and images updated.

zorr

12th April 2018, 21:58

The motion vector stuff is stored at the frame that the vector arrow is pointing at, (which is what Didee said in one of the linked threads I think).

You are right, he said it here (https://forum.doom9.org/showthread.php?p=1409835#post1409835).

The twist is just that when creating the interpolation between this-and-next-frame, MFlowInter does not use the forward and backward vector of the current frame. It uses the backward vector of the current frame, and the forward vector of the (current+delta) frame.

And in that case it's better to show the forward vector of frame n+delta like you did originally (Align=true in your latest version). So again we would see the vectors as they are used in creating the interpolated frame.

My initial thought process can be visualized like this: Overlay the forward vectors mask over the source frame n and then move the pixels under the mask area wherever the vector tells them to move but multiply the vector length by the time percentage (50% works with delta=2). Do the same with backwards mask, overlay it over frame n+delta and move the pixels under the mask area. Then combine these two frames somehow and that's the final result. That makes sense because I can see that the correct pixels would be moved.

But since the forward/backward vectors are aligned perhaps it works by first combining the forward / backward vectors into one and using those combined vectors to move pixels in both frames. :confused:

StainlessS

14th April 2018, 08:37

And in that case it's better to show the forward vector of frame n+delta like you did originally

Post #16 script and images updated, changed default Align=True (+ a few minor mods).

EDIT:
But since the forward/backward vectors are aligned perhaps it works by first combining the forward / backward vectors into one and using those combined vectors to move pixels in both frames.

Nope, dont think so.

My initial thought process can be visualized like this: Overlay the forward vectors mask over the source frame n and then move the pixels under the mask area wherever the vector tells them to move but multiply the vector length by the time percentage (50% works with delta=2). Do the same with backwards mask, overlay it over frame n+delta and move the pixels under the mask area. Then combine these two frames somehow and that's the final result. That makes sense because I can see that the correct pixels would be moved.

Think this is more like it. (The forward vector align is only for display in the VectorTest thing, of course vectors are not aligned).
I'm guessin' that both sets of results are produced and combined, if bad matches between the two then maybe blurred, degree of blurring depending upon badness of mismatch.

zorr

17th April 2018, 22:47

My initial thought process can be visualized like this: Overlay the forward vectors mask over the source frame n and then move the pixels under the mask area wherever the vector tells them to move but multiply the vector length by the time percentage (50% works with delta=2). Do the same with backwards mask, overlay it over frame n+delta and move the pixels under the mask area. Then combine these two frames somehow and that's the final result. That makes sense because I can see that the correct pixels would be moved.

Think this is more like it. (The forward vector align is only for display in the VectorTest thing, of course vectors are not aligned).

Ok bear with me because now I'm even more confused. :) I think we already established that the vectors used are the backward vector of frame n and the forward vector of frame n+delta. And those are the exact vectors shown in your folly when Aligned=true. But if you take those vectors and try use them like I described above it wouldn't really work because the vectors are aligned but the pixels in frames n and n+delta are not.

https://s20.postimg.cc/lyhvh24y5/Vector_Test_2_True.jpg

Of course there are other ways to use those vectors but I'm still a bit confused because it looks like the forward vector stored at n+delta is written to the location the arrow is pointing *at* whereas the backward vector stored at frame n is written to the location where the arrow is pointing *from*... perhaps this is by design to make the process easier or maybe my brain is malfunctioning.

StainlessS

22nd April 2018, 17:04

zorr

23rd April 2018, 22:36

n n+delta
| |
<---- BV Vectors used for eg MDegrain, vectors stored @ frame where arrowhead points (ie n).
FV ----> Pixels moved from frame at back end of arrow (n+/- delta), along vector to synth frame where arrowhead points.
|
n-delta

n n+delta
| |
<---- BV Vectors used for eg MFlowInter, Uses the next one along forward vector so that vectors used are from
| | either side of the frame that will be synthesized (reason, only part of the vector distances are
FVI ----> used, based on Time arg).
n n+delta
For MFlowInter, the created vectors are exactly the same as for eg MDegrain, its just that it uses the
next forward vector so that backward and forward vectors straddle the predicted frame, MFlowInter
just uses the vectors in a slightly different way to MDegrain.
The synthesized frame lies logically somewhere between n and n + delta (based on Time arg), and is
physically created at frame n, and so needs to be relocated to the required bad frame position in clip.

Hope the above makes sense.

That in itself makes sense. :) It's only when we also look at the screenshots of masks and make some reasonable assumptions that things start to get... weird. [EDIT] Added relevant screenshot below.

https://s20.postimg.cc/lyhvh24y5/Vector_Test_2_True.jpg

So, let's take a look at the masks. I don't know how MMask generates the colors from the vectors, but we can see that the forward mask is lighter than average and the backward mask is darker than average. Since we are showing horizontal component of motion vector and the movement is horizontal it's safe to assume that the lighter and darker colors represent horizontal motion vectors with opposite directions and the middle gray represents no motion. Since frame n+delta has the forward vectors (pointing from frame n to frame n+delta), the vector direction should be towards left and therefore lighter color means vectors towards left. Crudely depicted here:
frame N+delta FORWARD vectors shown along with pixel colors (0 = black, X = white)
vector direction is towards left (<)

0 0 0 X< X< X< X< X X X X X X X

0 0 0 X< X< X< X< X X X X X X X

0 0 0 X< X< X< X< X X X X X X X

One pixel and its motion vector
+-----+
| |
| <--------+
| |
+-----+

Now how can these vectors and pixels be used to generate an interpolated frame? The pixels at frame n+delta should be moved towards right, therefore we must move the pixel "backwards" from the arrowhead towards the back end of arrow.

Things get more interesting when we look at the backward vectors at frame n. The vector direction is now towards right
frame N BACKWARD vectors

0 0 0 0> 0> 0> 0> X X X X X X X

0 0 0 0> 0> 0> 0> X X X X X X X

0 0 0 0> 0> 0> 0> X X X X X X X

+-----+
| |
+--------> |
| |
+-----+

How can we use these vectors and pixels to generate the interpolated frame? The pixels should be moving towards left, so we take the pixel at the arrowhead and move it.... wait. There is no pixel at the arrowhead. :eek: Or more accurately, there is a pixel but it has the wrong color. We should move at least some of the white pixels in both frames but frame n doesn't have any motion over the white pixels.

Ok, this is not how it works so let's try again. This time we swap the pixels and the motion vectors.

frame N+delta FORWARD vectors shown along with pixel colors of frame N

0 0 0 0< 0< 0< 0< X X X X X X X

0 0 0 0< 0< 0< 0< X X X X X X X

0 0 0 0< 0< 0< 0< X X X X X X X

One pixel and its motion vector
+-----+
| |
| <--------+
| |
+-----+

What if we follow the arrow backwards and take the pixel located at the back end of the arrow and move it towards the arrowhead, but we take the pixel from frame N? It should work, there are white pixels there.

Does it also work with the backward vectors?

frame N BACKWARD vectors shown along with pixel colors of frame N+delta

0 0 0 X> X> X> X> X X X X X X X

0 0 0 X> X> X> X> X X X X X X X

0 0 0 X> X> X> X> X X X X X X X

+-----+
| |
+--------> |
| |
+-----+

Hmm... no, it doesn't look like this will work either. :confused: The vector points to the wrong direction.

So how can we make it work? There is at least one way, we must draw the arrow so that the back end is at the center of the pixel:

+-----+
| |
| +-------->
| |
+-----+

Now we can simply take the pixel at the back end and move it towards the arrowhead. But this is inconsistent with the way it was done with the forward vectors and the same method will not work for both. This implies that the backward vectors are stored at the back end of the arrow whereas the forward vectors are stored at the pixel pointed by the arrowhead. Or there's another, simpler explanation but then we would have to break some of the rules you laid out in your post. :D

StainlessS

24th April 2018, 02:12

I'm not allowing myself to get too confused (despite the assistance of your post) :) ,
the arrow only shows the frame where vectors are stored (arrow head), and the frame used as the pixel source (arrow flight/feathers/rear end),
the actual horizontal distance/direction is in the color (above or below Mid Grey, which might be 127, 128 or even 126 [rounded 125.5 mid point for TV levels], I'm guessin' that it might be 128).

I'm quite happy to accept that it works, now stop trying to confuse me https://www.cosgan.de/images/smilie/frech/o094.gif

EDIT: Although this kinda makes things a little more 'interesting' :)
From MFlowInter Docs, last arg tclip,
tclip
If set, the time parameter is ignored. Then the time for the motion interpolation is applied pixel-wise. Each component of each pixel of tclip gives the time to be applied to the corresponding source clip pixel. The time scale is 256, meaning that 0 corresponds to the current frame, and 255 is an almost the next frame (128 is exactly half-way). A single occlusion mask is calculated with the luma-time only, therefore it is recommended to keep the chroma-time synchronized with the luma.
Not sure, I think I'm confused again https://www.cosgan.de/images/smilie/konfus/a080.gif
(not really, it just allows specification of the time arg for every pixel individually,
the how does not really matter unless you want to use it tclip arg and if so just do as instructed in above doc).

The arrow in the title bar above the graphic[at RHS] shows the direction of the white block travelling across the frame, right to left (when HFlip=False).
For the forward vector, pixels at n moved left for the n to n+delta transition, so the color at '[B]FVEC[n+delta=62] aligned' is lighter than mid grey and so must mean move pixels at n left to match the pixel at n+delta. In the backwards vector is is just the other way around (from frame n+delta to n) and so is opposite and darker than mid grey.
So to recap, lighter than mid grey means Move_Left, darker, means Move_Right, absolute difference from mid grey is the distance to move.
If you change to HFlip=True, white block will move left to right, and n->n+Delta_Aligned will be darker than mid grey instead of lighter.
In a perfect world the results would always be exact opposite (relative mid grey), both vector sets are just a check on each other,
that both lots of calculations agree.

StainlessS

24th April 2018, 17:36

Post #16 script updated.

Shows colors of the vector greys, and little white (single pixel) dot overlaid where taken from (Align MUST be true otherwise not shown).

eg

CS="Y8"
#CS="YV24"
VectorTest(Delta=2,time=50.0,HFlip=False,CS=CS)

https://s20.postimg.cc/vnubrzazx/Vector_Test.jpg (https://postimages.org/)
Fwd/Bak, Is mostly opposite (relative mid grey, 128), but sometimes goes awry (usually at edge of grey, we try take mid pixel).

EDIT: With CS="YV24"
https://s20.postimg.cc/6vupkxh0d/Vector_Test_Col.jpg (https://postimages.org/)

EDIT: The movement in above graphics should be DELTA(2) * XSTEP(8)=16, but in MSuper, Pel=2, ie 2*8*2=32,
so for the vector Greys = +/- 32 , seems about right.
[XSTEP is the horizontal pixel movement per frame, Delta is the number of frames apart, and Pel=2 is at 1/2 pixel resolution].

zorr

24th April 2018, 23:23

I'm quite happy to accept that it works, now stop trying to confuse me https://www.cosgan.de/images/smilie/frech/o094.gif

Oh, I'm not quite done messing with your mind yet. :D

the arrow only shows the frame where vectors are stored (arrow head), and the frame used as the pixel source (arrow flight/feathers/rear end),
the actual horizontal distance/direction is in the color (above or below Mid Grey, which might be 127, 128 or even 126 [rounded 125.5 mid point for TV levels], I'm guessin' that it might be 128).

Yes, I agree completely and the arrows in my previous post were trying to show how the pixels should move based on the vectors, not about where they are stored. Also I found this in MVMask.cpp:

else if (kind == 3) // vector x mask
{
for (int j = 0; j < nBlkCount; j++)
smallMask[j] = std::max(int(0), std::min(255, int(mvClip.GetBlock(0, j).GetMV().x * fMaskNormFactor * 100 + 128))); // shited by 128 for signed support

I kinda like that it's 'shited' by 128. :) So yes, middle gray is hardcoded at value 128.

Some clarification for this explanation:
Since frame n+delta has the forward vectors (pointing from frame n to frame n+delta), the vector direction should be towards left and therefore lighter color means vectors towards left.

The justification for the direction (left) for the foward vector is that the forward vector really means forward in time and stepping forward in time the white/black edge is moving towards left. So it's simply the direction things are moving when time flows forward.

For the forward vector, pixels at n moved left for the n to n+delta transition, so the color at 'FVEC[n+delta=62] aligned' is lighter than mid grey and so must mean move pixels at n left to match the pixel at n+delta.

Great, we agree on this too!

FORWARD VECTORS (MMask lighter than average, positive x value)

0 1 2 3 4 5 6 7 8
+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|(8,0)| | | | | | | |(0,0)| motion vector (delta x, delta y)
| <-----------------------------------------------+ |
| | | | | | | | | |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+

+-----+ +-----+ +-----+
| | | | | |
| A | | X | | B |
| | | | | |
+-----+ +-----+ +-----+

So here's a row of pixels along with the forward vectors. There's a motion vector value displayed for pixel A and B. I used offset 8 here, 32 would have made this drawing too large...

Like we agreed the vector points towards left. The vector is stored at pixel A (there is a vector at pixel B too but it has delta zero). The goal here is to move PIXEL B along the vector towards PIXEL A into location of PIXEL X which is at 50% of the vector length.

This means we have to draw the vector so that the arrowhead is at PIXEL A and the rear end is at PIXEL B. So we have an algorithm like this:

1) place the arrowhead at the pixel which contains the motion vector (PIXEL A)
2) calculate position delta by adding motion vector to current pixel position (x=0+8, y=0+0) = (8,0)
3) place the rear end of arrow to this location (PIXEL B)
4) move the pixel at the rear end towards the arrowhead by 50% of arrow length (PIXEL X)
(in reality: render the pixel color of PIXEL B to the location of PIXEL X)

Ok, this totally works. So let's look at the backward vectors...

BACKWARD VECTORS (MMask darker than average, negative x value)

0 1 2 3 4 5 6 7 8
+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|(-8,0) | | | | | | |(0,0)| motion vector (delta x, delta y)
| | | | | | | | | |
| | | | | | | | | |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+

Now using the same algorithm we find that PIXEL B is at coordinate (0+(-8), 0+0) == (-8,0) which is completely wrong. Besides that we should be moving the PIXEL A with the backward vectors (we already moved PIXEL B so not useful to do it again). This means we need a different algorithm for the backward vectors.

[But if you figure it out 100.0%, and find different then do let us know]

I haven't really figured it out, just making observations which show that the current theory leads to some complications. I believe there is a simpler way to use the vectors. To really figure out what MVTools2 is doing would mean diving into the source code and I don't really have the time nor competence to do that. :)

StainlessS

25th April 2018, 13:00

Oh, I'm not quite done messing with your mind yet. :D
You are an evil and repulsive person who delights in the suffering of others.

Dont think I had my programming head on when I wrote the vector color direction stuff.
Light grey does indeed mean move pixel left for (forward) frames n to n+delta, but that is stating it the wrong way
around programmatically. If you iterate over frame n for every x, putting pixels in frame n+delta at x + Some_Offset,
you could end up eg setting only a single pixel many times (and all the rest would hold rubbish).

For forwards component pixel, You should actually iterate over the destination for every x, and use the vector grey
stuff to take the source pixel from frame n (the back end of the forward vector arrow) at x+((Fgrey-128)/pel), so
instead of moving pixel left in forward prediction, you actually get it from right (so light grey, get it from the
right, dark grey get it from the left).
[It also corrects the weird sign issue we had so (rel 128) +ve grey means take from the right, -ve from left,
and so it matches raster layout in memory].

For backwards component pixel, iterate over the destination for every x, and use the vector grey stuff to take the
source pixel from frame n+delta (the back end of the backward vector arrow) at x+((Bgrey-128)/pel),
so instead of moving pixel right in backward prediction, you actually get it from left. [when considering VectorTest(HFlip=False)]

The "x+((Fgrey-128)/pel)" and "x+((Bgrey-128)/pel)" above should of course be limited to valid pixel coords (min,max),
and also need to apply the Time arg to both.

# Something like (where Time=33.33%),

# EDIT: Added lines below
FGrey = FGrey_at_nPlusDelta[x] # @ head of Forward vector arrow, the horiz mask from fvec
BGrey = BGrey_at_n[x] # @ head of Backward vector arrow, the horiz mask from bvec
pel=2.0 # whatever pel is in use, as float

FCoord_X = x + Round(((Fgrey-128)/pel) * 33.33 / 100.0) # 33.33% of forward distance (from n)

BCoord_X = x + Round(((Bgrey-128)/pel) * (100.0 - 33.33) / 100.0) # 66.66% of backward distance (from n+delta)

FPixel = Frame_n[Min(Max(FCoord_X,0),Width-1)]
BPixel = Frame_nPlusDelta[Min(Max(BCoord_X,0),Width-1)]

# Self Check blurry stuff

ResultClipFrame_n[x] = Round((FPixel + BPixel)/2.0) # This frame needs shift to fix BAD frame @ Time=33.00%

So, how does that work for you ?

EDIT: I have not looked at the source, no idea how the self check blurry stuff works, just a wild guess.
(maybe [EDIT: likely] the other masks play a part)

zorr

25th April 2018, 23:09

You are an evil and repulsive person who delights in the suffering of others.

You understand me so well. This could be the start of a long and beautiful friendship.

If you iterate over frame n for every x, putting pixels in frame n+delta at x + Some_Offset, you could end up eg setting only a single pixel many times (and all the rest would hold rubbish).

For forwards component pixel, You should actually iterate over the destination for every x, and use the vector grey
stuff to take the source pixel from frame n ...

Yes that's true. I think what you described above works beautifully with MCompensate where you can read a pixel color pointed by the vector for every pixel in the destination. I'm not sure you can use that method when dealing with MFlowInter where the vectors are scaled by the time. But certainly this "missing pixels" problem is something the algorithm has to deal with.

[It also corrects the weird sign issue we had ...].

We will see about that. :devil:

# Something like (where Time=33.33%),

# EDIT: Added lines below
FGrey = FGrey_at_nPlusDelta[x] # @ head of Forward vector arrow, the horiz mask from fvec
BGrey = BGrey_at_n[x] # @ head of Backward vector arrow, the horiz mask from bvec
pel=2.0 # whatever pel is in use, as float

FCoord_X = x + Round(((Fgrey-128)/pel) * 33.33 / 100.0) # 33.33% of forward distance (from n)

BCoord_X = x + Round(((Bgrey-128)/pel) * (100.0 - 33.33) / 100.0) # 66.66% of backward distance (from n+delta)

FPixel = Frame_n[Min(Max(FCoord_X,0),Width-1)]
BPixel = Frame_nPlusDelta[Min(Max(BCoord_X,0),Width-1)]

# Self Check blurry stuff

ResultClipFrame_n[x] = Round((FPixel + BPixel)/2.0) # This frame needs shift to fix BAD frame @ Time=33.00%

So, how does that work for you ?

I just run your code though my ASCII SIMULATOR 6000 (tm) and this was the output:

PROCESSING...

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| | | | | | |(9,0)| | | | | | | | |(0,0)| forward vector
| <-----------------B (66.6%)---------+-----F (33.3%)---> | | | | | | |
| | | | | | |(-9,0) | | | | | | | |(0,0)| backward vector
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+

+-----+ +-----+ +-----+
| | | | | |
| A | | X | | B |
| | | | | |
+-----+ +-----+ +-----+
N+delta 33.3% N

...<<<<<<<<<......... backward vectors (-9,0) at frame N
...>>>>>>>>>......... forward vectors (+9,0) at frame N+delta
000000000000XXXXXXXXX frame N
000000000XXXXXXXXXXXX interpolated at 33.3%
000XXXXXXXXXXXXXXXXXX frame N+delta

--> forward offset (33.3%), read pixel from frame N
<----- backward offset (66.6%), read pixel from frame N+delta

000000000000XXXXXXXXX--> frame N with offset -3
<-----000XXXXXXXXXXXXXXXXXX frame N+delta with offset +6
000000000000XXXXXXXXXXXX average color

CONCLUSION: STAINLESSS IS A GENIUS

Oh wow, that actually works! So this example is using offset 9 to make the 33.3% and 66.6% nice integer locations. The vector length originally is 9 but it is scaled to 3 (forward vectors) and 6 (backward vectors) when time is 33.3%. To find the color for each pixel I moved the frame to the opposite direction of the vector so that the pixels to read from at each frame are aligned. And what do you know, they perfectly replicate the result we are hoping to achieve! :eek: Note that the rightmost portion of the frame has zero length vectors so that part doesn't need to be moved but it doesn't change the result. Well done mate.

StainlessS

25th April 2018, 23:45

You understand me so well. This could be the start of a long and beautiful friendship.
HeHeHehe

We will see about that. :devil:
That made me nervous, again.

CONCLUSION: STAINLESSS IS A GENIUS

Nah, I might be if I understood anything from your ASCII gram, tis a total puzzle. :)

zorr

26th April 2018, 23:31

We will see about that.
That made me nervous, again.

Actually when I wrote that I was confident that I could crush your hopes and dreams about that algorithm, but when I went through it I realized it works... but I decided to leave that comment there in order not to spoil the ending. :)

Nah, I might be if I understood anything from your ASCII gram, tis a total puzzle. :)

Fair enough, I was in a bit of hurry and didn't provide enough explanations. So let's go through it again but take it slow. Perhaps this will help someone else who is trying to figure out wft is going on with MaskTools2. Or in the year 3000 when the archeologists dig up the internet from the ruins they find this message and can perfectly understand our primitive technology.

Let's start with a screenshot of the two frames and the motion vector masks used when creating the interpolated frame. The interpolated frame is created at 33.3% between frames N and N+delta (like in the previous example).

https://s20.postimg.cc/9l9mo124d/Vector_Test_33.jpg

I'm changing the XSTEP to 4.5 in order to make the math simpler, this results in 9 pixels of movement in two frames (delta is 2). Looks like this in ASCII:

PIXEL COLOR: 0 = BLACK, X = WHITE
VECTOR DIRECTION: < = LEFT, > = RIGHT, . = ZERO LENGTH

000000000000000000XXXXXXXXXXXXXXXXXX FRAME N
000000000000000000XXXXXXXXXXXXXXXXXX
000000000000000000XXXXXXXXXXXXXXXXXX
000000000000000000XXXXXXXXXXXXXXXXXX

000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX FRAME N+delta
000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX
000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX
000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX

.........>>>>>>>>>.................. FRAME N+delta FORWARD VECTORS
.........>>>>>>>>>..................
.........>>>>>>>>>..................
.........>>>>>>>>>..................

.........<<<<<<<<<.................. FRAME N BACKWARD VECTORS
.........<<<<<<<<<..................
.........<<<<<<<<<..................
.........<<<<<<<<<..................

000000000000000XXXXXXXXXXXXXXXXXXXXX INTERPOLATED FRAME AT 33.33% BETWEEN N AND N+DELTA
000000000000000XXXXXXXXXXXXXXXXXXXXX
000000000000000XXXXXXXXXXXXXXXXXXXXX
000000000000000XXXXXXXXXXXXXXXXXXXXX

Let's take a closer look at the vectors. The magnitude (length) of the vectors is 9 (actually 9*pel which in this case is 18 but let's keep it simple), that's the distance the black/white edge is moving between frames N and N+delta in this example. The direction of the vectors can be determined by the brightness of the gray, and lighter than average means positive & pointing right, darker than average is negative & pointing left. The vectors are 2D so there is X and Y component but in this example there is only horizontal motion, therefore the Y component is zero.

FORWARD VECTORS CLOSEUP
... (0,0) (0,0) (0,0) (9,0) (9,0) (9,0) (9,0) ...

BACKWARD VECTORS CLOSEUP
... (0,0) (0,0) (0,0) (-9,0) (-9,0) (-9,0) (-9,0) ...

When the vectors are used in interpolation, their length is scaled by the time. Forward vectors are multiplied by time/100 (33.3/100 == 0.333) and the backward vectors are scaled by (100-time)/100 ((100-33.3)/100 == 0.666). Doing the calculations we get new forward vectors (3,0) and new backward vectors (-6,0).

These vectors can now be used to point the coordinates where colors should be read for the interpolated frame. We loop every pixel on the screen and read a pixel from the location pointed by the forward and backward vectors.

FORWARD VECTOR AT PIXEL (6,0)

0 1 2 3 4 5 6 7 8 9
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| | | | | | |(3,0)| | | | |
| | | | | | | +-----------------> | |
| | | | | | | | | | | |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+

BACKWARD VECTOR AT PIXEL (6,0)

0 1 2 3 4 5 6 7 8 9
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| | | | | | |(-6,0) | | | |
| <-----------------------------------+ | | | | |
| | | | | | | | | | | |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+

The pixel pointed by the forward vector is read from frame N and the pixel pointed by the backward vector is read from frame N+delta. These two color samples are then combined (a simple average will do but MVTools is perhaps using something more advanced).

This is all the information we need to create the interpolated frame, but it's not very straightforward to do manually. We would have to look at every pixel and its vectors, look at where the vectors are pointing at and what the colors are at those locations.

Luckily there's a shortcut we can use. Since all the forward vectors are identical we can think of the operation as shifting the frame N in the opposite direction the vector is pointing at. So in this case the vector (3,0) means we should shift the frame left by 3 pixels. Once we have done that operation we can read the pixel color at the same coordinate as the target pixel is at. We do the same thing with backward vectors, this time we shift frame N+delta in the opposite direction, in this case 6 pixels to the right.

000000000000000000XXXXXXXXXXXXXXXXXX FRAME N
000000000000000000XXXXXXXXXXXXXXXXXX
000000000000000000XXXXXXXXXXXXXXXXXX
000000000000000000XXXXXXXXXXXXXXXXXX

000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX FRAME N+delta
000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX
000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX
000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX

000000000000000000XXXXXXXXXXXXXXXXXX... FRAME N shifted left by 3 pixels
000000000000000000XXXXXXXXXXXXXXXXXX...
000000000000000000XXXXXXXXXXXXXXXXXX...
000000000000000000XXXXXXXXXXXXXXXXXX...

......000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX FRAME N+delta shifted right by 6 pixels
......000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX
......000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX
......000000000XXXXXXXXXXXXXXXXXXXXXXXXXXX

Note: new pixels not visible before marked with '.'

Already we can see that now the black/white edge is aligned in both frames. Now we take the average color from both frames (special case is those new pixels which became visible, let's leave the result blank for those).

000000000XXXXXXXXXXXXXXXXXX AVERAGE COLOR OF SHIFTED FRAMES N AND N+DELTA
000000000XXXXXXXXXXXXXXXXXX
000000000XXXXXXXXXXXXXXXXXX
000000000XXXXXXXXXXXXXXXXXX

000000000000000XXXXXXXXXXXXXXXXXXXXX INTERPOLATED FRAME AT 33.33% BETWEEN N AND N+DELTA (THE GOAL)
000000000000000XXXXXXXXXXXXXXXXXXXXX
000000000000000XXXXXXXXXXXXXXXXXXXXX
000000000000000XXXXXXXXXXXXXXXXXXXXX

And there's the result we were hoping to get, the black/white edge is in the correct position. One final note about that shifting trick we used: it should only be applied at those pixels where the vector was not zero. When the vector is zero we can read the source frame at the exact same location where the target pixel is. This would help fill the edges we now skipped.

Here's what it looks like if we also process the zero vectors (lowercase used where final pixel color was read with zero vector):

.........>>>>>>>>>.................. FRAME N+delta FORWARD VECTORS
.........>>>>>>>>>..................
.........>>>>>>>>>..................
.........>>>>>>>>>..................

ooooooooo000000XXXxxxxxxxxxxxxxxxxxx
ooooooooo000000XXXxxxxxxxxxxxxxxxxxx
ooooooooo000000XXXxxxxxxxxxxxxxxxxxx
ooooooooo000000XXXxxxxxxxxxxxxxxxxxx