Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 7th April 2018, 16:31   #1  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
Replacing bad frames revisited

There have been a couple of different scripts for replacing bad or duplicated frames. There's the filldrops script (original by Mug Funky, 2005) that works for the situation when one frame is duplicated. It uses MAnalyse with delta=1. MVTools2 documentation contains an example for replacing bad frames using MAnalyse with delta=2. Delta 2 is needed for the general case when the frame is bad and we cannot use any information from it.

Johnmeyer developed an improved function called replacebad in 2010. In that thread Gavino gave an explanation why it's better: MFlowInter uses not only previous and next frame, but also frames next+delta and previous-delta when there is occlusion.

Let's say we have frames A-E and the frame C is bad:
Code:
0	1	2	3	4
A	B	C	D	E
		X bad
These are the frames and the optimal forward / backward vectors we should use:

Code:
A	B		D	E
	---------------->
	<----------------
-------->		<--------
I tried to analyze what replacebad is doing and it goes like this: It first replaces the bad frame with a duplicate of the frame following it.

Code:
0	1	2	3	4
A	B	D	D	E
Then it creates a "previous frame"-clip by duplicating the first frame:

Code:
0	1	2	3	4	5
A	A	B	D	D	E
The final phase is interpolation using that previous-frame clip using delta=1:

Code:
0	1	2	3	4	5
	A	B	D	D			
		-------->
		<--------
	-------->	<--------
So to me it looks like it's using the frame D twice and not using the frame E at all. Am I correctly interpreting what is happening here?

To fix this, the frame 4 (D) should be replaced with frame 5 (E).

Here's a script which can be used to test the original behavior of replacebad (replaces frame C) and an improved version which replaces frames C and D. Set the BAD_FRAME variable to frame number you want replaced. To see the improved behaviour, comment / uncomment the first lines of replacebad_test accordingly. The results are different and to me the improved version indeed looks better. For comparison there's also recreateframes-function which is similar to the code used in MVTools2 documentation.

Code:
AVISource("d:\process2\1 deinterlaced.avi")

global BAD_FRAME = 11

return replacebad_test(last)
#return recreateframes(last)

function replacebad_test (clip c) {
	# ORIGINAL This next statement replaces BAD_FRAME with a duplicate of the frame immediately after the BAD_FRAME
#	goodframes = ConditionalFilter(c, trim(c,1,0), c, "current_frame", "equals", "BAD_FRAME")
  
	# IMPROVED This next statement replaces BAD_FRAME with frame (BAD_FRAME+1) and the frame (BAD_FRAME+1) with frame (BAD_FRAME+2)
	goodframes = ConditionalFilter(c, trim(c,1,0), c, "current_frame==BAD_FRAME || current_frame==BAD_FRAME+1", "equals", "true")
  
	# This next statement gets the previous frame in the stream that now contains duplicates instead of bad frames
	previousframe = Loop(goodframes,2,0,0)

	super=MSuper(previousframe,pel=2)

	vfe=manalyse(super,truemotion=true,isb=false,delta=1)
	vbe=manalyse(super,truemotion=true,isb=true,delta=1)

	replacement = mflowinter(previousframe,super,vbe,vfe,time=50)

	fixed = ConditionalFilter(c, replacement, c, "current_frame", "equals", "BAD_FRAME")

	return fixed
} # End function replacebad_test

function recreateframes(clip source)
{
	super=source.MSuper(pel=2)
	backward_vectors = MAnalyse(super, isb = true, truemotion=true, delta=2)
	forward_vectors = MAnalyse(super, isb = false, truemotion=true, delta=2)
	inter = source.MFlowInter(super, backward_vectors, forward_vectors, time=50)
	return source.trim(0,BAD_FRAME-1) ++ inter.trim(BAD_FRAME-1,-1) ++ source.trim(BAD_FRAME+1,0)	
}
Does anyone else see an improvement with this change?
zorr is offline   Reply With Quote
Old 7th April 2018, 17:53   #2  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,096
Oh, my head hurts reading that old thread where I was being taught by Gavino and Didée about how MflowInter() works. I almost figured it out then, but still don't fully get it.

I did, however, make another, much more robust replacement function and posted it in this thread:

Finding individual "bad" frames in video; save frame number; or repair

You might want to take a look at both the code and the discussion in that thread to see if it helps you with what you are doing.

I'll have to actually try out your code to see what it is doing. I am not quite sure whether making additional duplicates to feed to the interpolation logic is going to produce better results, but I'll see what happens when I try it.
johnmeyer is offline   Reply With Quote
Old 7th April 2018, 18:20   #3  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
Well you beat me to it John, was looking for that very same thread.

I think this is probably a fair description:- http://forum.doom9.org/showthread.ph...23#post1789723

But see entire thread.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 7th April 2018, 19:52   #4  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,096
What I should do is to use AVISynth to insert a "bad"frame by simply replacing a frame with a black frame. However, I'll keep the original frame for reference. Then go through some of the same tests I did in that long-ago thread the OP referenced, but this time with the black frame as the reference frame. This will make obvious any issues that were "covered up" by replacing an exact duplicate because the metrics looking towards the duplicate will still produce a result that looks valid, but may not be optimal (which is what the OP is attempting to discover) and which may contain information from the bad frame which, of course, is not what we want.

I'll then compare the synthesized frame with the one that I removed and see how close I get. I guess I could create a metric for this difference (YDiff ...), but to keep it simple, I'll just look at it.

I did these tests before (as documented in that old post), but never did it using a bad frame which, now that I think about it, was stupid of me. How many times have I had to say that? (Don't answer.)

I think I have free time tomorrow, so I might get to it. I do know that the function I created, with your help StainlessS, that I linked to in my last post, does correctly replace bad frames, but whether it is optimal (trying hard not to say "best"), I don't yet know.

[edit]I am also still left wondering, in your explanation in that thread you just linked to (a portion of which is copied below) why the forward vector from G3 to G5 isn't simply the reciprocal of the backward vector from G5 back to G3. At the risk of "here we go again," I thought one of the vectors (with delta=2) should go from G5 to G3 and the other from G1 to G3.

<Sigh> I may never get this totally figured out. At least, like the bumblebee, my stuff may theoretically not be able to work, but the darn stuff still flies.

Code:
              50%
               |
          <--------->
          |         |
          v         v
G1   G2   G3   B4   G5   G6
          n    n+1  n+2

Last edited by johnmeyer; 7th April 2018 at 19:58.
johnmeyer is offline   Reply With Quote
Old 7th April 2018, 22:19   #5  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
Quote:
Originally Posted by johnmeyer View Post
I did, however, make another, much more robust replacement function and posted it in this thread:

Finding individual "bad" frames in video; save frame number; or repair

You might want to take a look at both the code and the discussion in that thread to see if it helps you with what you are doing.
Thanks John. That script is certainly more advanced than the replacebad-function. Your script is solving (at least) two separate issues:
1) which frames are bad and thus should be replaced?
2) how to replace a bad frame?

What I'm mainly interested now is that second part. This newer script is actually doing the replacement using the "standard" way with delta=2. Are you saying this replacement method is working better than your earlier script? That's actually something I'm not sure about myself, I only noticed that my improved version was better than the original replacebad-method.

Quote:
Originally Posted by johnmeyer View Post
What I should do is to use AVISynth to insert a "bad"frame by simply replacing a frame with a black frame.
...
I'll then compare the synthesized frame with the one that I removed and see how close I get. I guess I could create a metric for this difference (YDiff ...), but to keep it simple, I'll just look at it.
That's a good idea. I have done something similar using LumaDifference, ChromaUDifference and ChromaVDifference inside FrameEvaluate. I have also used SSIM which should give more realistic evaluation on how close those two frames are. If you're going to do that I recommend the v0.25.1 by mitsubishi because it's the only version with SSIM_FRAME function and can be used like this:

Code:
FrameEvaluate(last, """
	global ssim = SSIM_FRAME(orig, replaced)
	global ssim_total = ssim_total + (ssim == 1.0 ? 0.0 : ssim)	
""", args="orig,replaced")
That's maybe overkill if you're comparing one frame only.

Quote:
Originally Posted by johnmeyer View Post
[edit]I am also still left wondering, in your explanation in that thread you just linked to (a portion of which is copied below) why the forward vector from G3 to G5 isn't simply the reciprocal of the backward vector from G5 back to G3.
I think I can answer that. The forward vector from G3 to G5 is answering "how should pixels of G3 be moved in order to recreate frame G5?", whereas backward vector G5->G3 is answering "how should pixels of G5 be moved in order to recreate frame G3?". So the source pixels are different in the backward vector. Maybe someone more knowledgeable can confirm this?
zorr is offline   Reply With Quote
Old 7th April 2018, 22:45   #6  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
Quote:
isn't simply the reciprocal of the backward vector
Perhaps in a perfect world it would be, but just because the algo predicts a motion block to
move in a particular directions and distance, it dont automatically follow that the reverse situation
will produce the exact inverse result (mistakes will likely happen sometimes). Having two sets
of vectors will be a sort of self check, if the results dont match well then is to some degree, unreliable
(and some blurring will be probably be incorporated in the result, based on quality of match).

Quote:
the other from G1 to G3
Not sure at all how that would help predict frame at B4, but anyways, the G3->G5 and G5->G3 allow
to move vector clip results to the n frame of each vector clip, so they coincide for creating the
final result frame, to do anything other than was done in the plugin, would be one helluva a nightmare
for the user [ you think that its tricky now ], the filter does it in the most sane way that I could
think of and removes most of the difficulties that would ensue had it been done pretty much any other way.

Never mind how crazy it all seems, I made that post with a little trepidation, but the fact that Gavino did not
squash me like a bug, makes me think that it were not so far off target

EDIT: Zorr, perhaps of interest

FrameSurgeon:- https://forum.doom9.org/showthread.p...60#post1755860

DoctorFrames:- https://forum.doom9.org/showthread.p...43#post1764743

MorphDupes_MI:- https://forum.doom9.org/showthread.p...67#post1764867

Snippet of script from MorphDupes_MI which creates the MC Clips.
Code:
         thSCD1=(8*8)*255  thSCD2=255  BLEND=False   bs=(In.width>960) ? 16 : 8
        supFilt  = In.Blur(0.6).MSuper(pel=2,sharp=sharp,rfilter=rfilter,hpad=16, vpad=16)
        sup = In.MSuper(pel=2,sharp=sharp,rfilter=rfilter,hpad=16, vpad=16, levels=1)
        SC_fv=supFilt.MAnalyse(isb=false, delta=1,blksize=bs,overlap=bs/2)
        SC_fv=MRecalculate(sup,SC_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
        Global SC@@@=In.MSCDetection(SC_fv,thSCD1=SC_thSCD1,thSCD2=SC_thSCD2)
        For(Bad=1,MaxInterp) {
            Eval(RT_String("I%0.2d_bv=supFilt.MAnalyse(isb=true, delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1))
            Eval(RT_String("I%0.2d_fv=supFilt.MAnalyse(isb=false,delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1))
            Eval(RT_String("I%0.2d_bv=MRecalculate(sup,I%0.2d_bv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad))
            Eval(RT_String("I%0.2d_fv=MRecalculate(sup,I%0.2d_fv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad))
            for(i=1,Bad) {
                Eval(RT_String("Global I%0.2d_%0.2d@@@=In.MFlowInter(sup,I%0.2d_bv,I%0.2d_fv,time=100.0*%d/%d,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)",
                    \ Bad,i,Bad,Bad,i,Bad+1))
            }
        }
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 7th April 2018 at 23:04.
StainlessS is offline   Reply With Quote
Old 7th April 2018, 22:47   #7  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
Quote:
Originally Posted by StainlessS View Post
I think this is probably a fair description:- http://forum.doom9.org/showthread.ph...23#post1789723

But see entire thread.
Thanks StainlessS, your explanation makes perfect sense to me. I did look at the whole thread (and other threads linked from it), and I saw that the replacement was done with delta=2 for one bad frame.

I couldn't find any mention of the two extra frames and vectors which are also used. That's the part that I'm focused on, because it's not clear to me which frames will be used when delta=2. Gavino said the extra frames are separated by delta, so with delta=2 that would mean frames n-3, n-1, n+1, and n+3 when replacing frame n. That doesn't seem optimal, n-2 and n+2 would be better than n-3 and n+3. And that's what replacebad-function was trying to do. Whether or not that actually improves quality is another question and should be tested.
zorr is offline   Reply With Quote
Old 7th April 2018, 23:24   #8  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
From that thread
Quote:
EDIT: Delta is always +ve, isb sets whether it is a forward or backward vector between n and n + delta.
(Worrying too much about what is in which particular vector clip, and at which frame number, will lead to madness, dont do it ,
but for the already insane, frame n vectors will contain vectors for n<->n+delta depending upon isb direction)
Showing what comes out of MorphDupes_MI for a simple "MaxInterp=5", ie maximum of 5 frames interpolated

Client
Code:
Colorbars.ConvertToYV12
MorphDupes_MI(MaxInterp=5)
Hacked part of previous snippet
Code:
ZZZ=""
        For(Bad=1,MaxInterp) {
            Q=RT_String("I%0.2d_bv=supFilt.MAnalyse(isb=true, delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1)
            ZZZ=ZZZ+Chr(10)+Q
            Eval(Q)
#            Q=RT_String("I%0.2d_bv=supFilt.MAnalyse(isb=true, delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1)
#            ZZZ=ZZZ+Chr(10)+Q
#            Eval(Q)
            Q=RT_String("I%0.2d_fv=supFilt.MAnalyse(isb=false,delta=%d,blksize=bs,overlap=bs/2)",Bad,Bad+1)
            ZZZ=ZZZ+Chr(10)+Q
            Eval(Q)
            Q=RT_String("I%0.2d_bv=MRecalculate(sup,I%0.2d_bv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad)
            ZZZ=ZZZ+Chr(10)+Q
            Eval(Q)
            Q=RT_String("I%0.2d_fv=MRecalculate(sup,I%0.2d_fv,blksize=bs/2,overlap=bs/4,thSAD=100)",Bad,Bad)
            ZZZ=ZZZ+Chr(10)+Q
            Eval(Q)
            for(i=1,Bad) {
                Q=RT_String("Global I%0.2d_%0.2d@@@=In.MFlowInter(sup,I%0.2d_bv,I%0.2d_fv,time=100.0*%d/%d,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)",
                    \ Bad,i,Bad,Bad,i,Bad+1)
                ZZZ=ZZZ+Chr(10)+Q
            }
        }
RT_DebugF("%s",ZZZ)
Result for only 5 frame interp (gets way bigger with eg up to Maxinterp=20 )
Code:
I01_bv=supFilt.MAnalyse(isb=true, delta=2,blksize=bs,overlap=bs/2)
#I01_bv=supFilt.MAnalyse(isb=true, delta=2,blksize=bs,overlap=bs/2)
I01_fv=supFilt.MAnalyse(isb=false,delta=2,blksize=bs,overlap=bs/2)
I01_bv=MRecalculate(sup,I01_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I01_fv=MRecalculate(sup,I01_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I01_01_MorphDupes_MI_1=In.MFlowInter(sup,I01_bv,I01_fv,time=100.0*1/2,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I02_bv=supFilt.MAnalyse(isb=true, delta=3,blksize=bs,overlap=bs/2)
#I02_bv=supFilt.MAnalyse(isb=true, delta=3,blksize=bs,overlap=bs/2)
I02_fv=supFilt.MAnalyse(isb=false,delta=3,blksize=bs,overlap=bs/2)
I02_bv=MRecalculate(sup,I02_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I02_fv=MRecalculate(sup,I02_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I02_01_MorphDupes_MI_1=In.MFlowInter(sup,I02_bv,I02_fv,time=100.0*1/3,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I02_02_MorphDupes_MI_1=In.MFlowInter(sup,I02_bv,I02_fv,time=100.0*2/3,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)
#I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)
I03_fv=supFilt.MAnalyse(isb=false,delta=4,blksize=bs,overlap=bs/2)
I03_bv=MRecalculate(sup,I03_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I03_fv=MRecalculate(sup,I03_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I03_01_MorphDupes_MI_1=In.MFlowInter(sup,I03_bv,I03_fv,time=100.0*1/4,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I03_02_MorphDupes_MI_1=In.MFlowInter(sup,I03_bv,I03_fv,time=100.0*2/4,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I03_03_MorphDupes_MI_1=In.MFlowInter(sup,I03_bv,I03_fv,time=100.0*3/4,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I04_bv=supFilt.MAnalyse(isb=true, delta=5,blksize=bs,overlap=bs/2)
#I04_bv=supFilt.MAnalyse(isb=true, delta=5,blksize=bs,overlap=bs/2)
I04_fv=supFilt.MAnalyse(isb=false,delta=5,blksize=bs,overlap=bs/2)
I04_bv=MRecalculate(sup,I04_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I04_fv=MRecalculate(sup,I04_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I04_01_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*1/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I04_02_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*2/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I04_03_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*3/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I04_04_MorphDupes_MI_1=In.MFlowInter(sup,I04_bv,I04_fv,time=100.0*4/5,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
I05_bv=supFilt.MAnalyse(isb=true, delta=6,blksize=bs,overlap=bs/2)
#I05_bv=supFilt.MAnalyse(isb=true, delta=6,blksize=bs,overlap=bs/2)
I05_fv=supFilt.MAnalyse(isb=false,delta=6,blksize=bs,overlap=bs/2)
I05_bv=MRecalculate(sup,I05_bv,blksize=bs/2,overlap=bs/4,thSAD=100)
I05_fv=MRecalculate(sup,I05_fv,blksize=bs/2,overlap=bs/4,thSAD=100)
Global I05_01_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*1/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_02_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*2/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_03_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*3/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_04_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*4/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Global I05_05_MorphDupes_MI_1=In.MFlowInter(sup,I05_bv,I05_fv,time=100.0*5/6,ml=ml,Blend=BLEND,thSCD1=thSCD1,thSCD2=thSCD2)
Have to get to pub before it shuts.

EDIT: If I recall correctly, Feisty2 said that Mrecalculate is not much use for framerate change and so expect to be same for
Interpolation, so maybe above MRecal stuff not necessary.

EDIT: Seems to be a cockup somewhere in the hack above, we got duplicates eg
Code:
I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)
I03_bv=supFilt.MAnalyse(isb=true, delta=4,blksize=bs,overlap=bs/2)
Got to dash.
EDIT: Duplicate lines in red should be removed.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 7th April 2018 at 23:41.
StainlessS is offline   Reply With Quote
Old 8th April 2018, 23:52   #9  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
I decided to test these different replacement methods with the SSIM metric. Here's the script:

Code:
AVISource("d:\process2\1 deinterlaced.avi")

# SSIM needs YV12
ConvertToYV12()

orig = last

global ssim_total_orig = 0.0
global ssim_total_improved = 0.0
global ssim_total_standard = 0.0
global wins_orig = 0
global wins_improved = 0
global wins_standard = 0
frame_count = FrameCount()
ScriptClip(last, """
	# replace frame
	replaced_orig = replacebad_test(last, current_frame, false)		
	replaced_improved = replacebad_test(last, current_frame, true)		
	replaced_standard = recreateframes(last)
	measure = (current_frame > 1 && current_frame < frame_count-2)
	
	# compare to original with SSIM metric
	global ssim_orig = (measure ? SSIM_FRAME(orig, replaced_orig) : 0.0) 
	global ssim_improved = (measure ? SSIM_FRAME(orig, replaced_improved) : 0.0) 
	global ssim_standard = (measure ? SSIM_FRAME(orig, replaced_standard) : 0.0) 
	global ssim_best = Max(ssim_orig, ssim_improved, ssim_standard)

	# tag best SSIM result for each frame
	global tag_orig = (ssim_best > 0.0) && (ssim_best == ssim_orig) ? "* " : "  "
	global tag_improved = (ssim_best > 0.0) && (ssim_best == ssim_improved) ? "* " : "  "
	global tag_standard = (ssim_best > 0.0) && (ssim_best == ssim_standard) ? "* " : "  "
	
	# calculate number of wins (best result for this frame)
	global wins_orig = (ssim_best > 0.0) && (ssim_best == ssim_orig) ? wins_orig+1 : wins_orig
	global wins_improved = (ssim_best > 0.0) && (ssim_best == ssim_improved) ? wins_improved+1 : wins_improved
	global wins_standard = (ssim_best > 0.0) && (ssim_best == ssim_standard) ? wins_standard+1 : wins_standard
	
	# calculate total SSIM
	global ssim_total_orig = ssim_total_orig + (ssim_orig == 1.0 ? 0.0 : ssim_orig)	
	global ssim_total_improved = ssim_total_improved + (ssim_improved == 1.0 ? 0.0 : ssim_improved)	
	global ssim_total_standard = ssim_total_standard + (ssim_standard == 1.0 ? 0.0 : ssim_standard)	

	# tag best total SSIM result
	global ssim_total_best = Max(ssim_total_orig, ssim_total_improved, ssim_total_standard)
	global tag_total_orig = (ssim_total_best == ssim_total_orig) ? "* " : "  "
	global tag_total_improved = (ssim_total_best == ssim_total_improved) ? "* " : "  "
	global tag_total_standard = (ssim_total_best == ssim_total_standard) ? "* " : "  "
	
	return replaced_standard
""", args="orig, frame_count")

delimiter = "; "
resultFile = "ssimResults.txt"
WriteFileIf(resultFile, "current_frame == 0", """ "original replacebad" """, "delimiter", """ "improved replacebad" """,
  \ "delimiter", """ "standard" """, "delimiter", append=false)
WriteFile(resultFile, "current_frame", "delimiter", "tag_orig", "ssim_orig", "delimiter", "tag_improved", "ssim_improved", 
  \ "delimiter", "tag_standard", "ssim_standard", "delimiter")
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "wins " """, "wins_orig", "delimiter", "wins_improved",
  \ "delimiter", "wins_standard", "delimiter", append=true)
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "total " """, "tag_total_orig", "ssim_total_orig",
  \ "delimiter", "tag_total_improved", "ssim_total_improved", "delimiter", "tag_total_standard", "ssim_total_standard",
  \ "delimiter", append=true)
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "average " """, "ssim_total_orig / (frame_count-4)",
  \ "delimiter", "ssim_total_improved / (frame_count-4)", "delimiter", "ssim_total_standard / (frame_count-4)",
  \ "delimiter", append=true)

return last


function replacebad_test(clip c, int frame, bool improved) {
	
	# incoming frames: A B C D E F, where frame C is the current frame
	# ORIGINAL replaces current frame with the next frame, other frames do not change: A B C D E F -> A B D D E F
	# IMPROVED removes current frame: A B C D E F -> A B D E F
	goodframes = improved ? (c.trim(0,frame-1) ++ c.trim(frame+1,0)) : 
	  \ (c.trim(0,frame-1) ++ c.trim(frame+1,frame+1) ++ c.trim(frame+1, 0))
  
	# This next statement gets the previous frame in the stream
	previousframe = Loop(goodframes,2,0,0)

	super=MSuper(previousframe,pel=2)
	vfe=manalyse(super,truemotion=true,isb=false,delta=1)
	vbe=manalyse(super,truemotion=true,isb=true,delta=1)
	replacement = mflowinter(previousframe,super,vbe,vfe,time=50)
	
	# clip has to be the same length as the original for SSIM to work
	return replacement.trim(0,c.FrameCount()-1)
} # End function replacebad_test

# STANDARD replacement method with delta=2
function recreateframes(clip source)
{
	super=source.MSuper(pel=2)
	backward_vectors = MAnalyse(super, isb = true, truemotion=true, delta=2)
	forward_vectors = MAnalyse(super, isb = false, truemotion=true, delta=2)
	inter = source.MFlowInter(super, backward_vectors, forward_vectors, time=50)
	previousframe = Loop(inter,2,0,0)
	return previousframe.trim(0,source.FrameCount()-1)
}
It creates a text file ssimResults.txt and writes for each frame the SSIM metric for each of the three methods (original replacebad, my improved replacebad and the standard method with delta=2). It also calculates and writes the total and average SSIM and how many "wins" each method gets (ie how many times each method had the largest SSIM value for the frame). The winning value for each frame is also tagged with a "*" to make it easier to parse visually. The SSIM metric is not calculated for the first and last two frames because those frames don't have enough neighbour frames to do a proper interpolation.

Here's an example result for a short clip:
Code:
original replacebad; improved replacebad; standard; 
0;   0.000000;   0.000000;   0.000000; 
1;   0.000000;   0.000000;   0.000000; 
2;   0.965214; * 0.965760;   0.964760; 
3; * 0.920008;   0.919744;   0.919962; 
4;   0.827575;   0.827842; * 0.829020; 
5;   0.928627;   0.928224; * 0.929566; 
6;   0.959854;   0.959874; * 0.960391; 
7;   0.932269;   0.932424; * 0.933425; 
8; * 0.906174;   0.905721;   0.905074; 
9;   0.930301;   0.930339; * 0.931503; 
10;   0.910099;   0.910200; * 0.911520; 
11;   0.887747; * 0.888143;   0.888135; 
12;   0.927442; * 0.928089;   0.928063; 
13;   0.919194;   0.919065; * 0.920344; 
14;   0.898007;   0.897815; * 0.898245; 
15;   0.852154;   0.852494; * 0.853153; 
16;   0.850728; * 0.850974;   0.850876; 
17;   0.908898; * 0.909402;   0.908606; 
18;   0.922686;   0.923325; * 0.924254; 
19;   0.821544;   0.820911; * 0.821599; 
20;   0.875906; * 0.876705;   0.876263; 
21;   0.810928;   0.811540; * 0.813887; 
22;   0.821736;   0.822203; * 0.823627; 
23;   0.935320; * 0.935502;   0.935031; 
24;   0.777279;   0.777546; * 0.779405; 
25;   0.944397; * 0.944812;   0.944594; 
26;   0.861217;   0.861562; * 0.863031; 
27;   0.942816; * 0.943115;   0.941661; 
28;   0.000000;   0.000000;   0.000000; 
29;   0.000000;   0.000000;   0.000000; 
wins 2; 9; 15; 
total   23.238121;   23.243330; * 23.255993; 
average 0.893774; 0.893974; 0.894461;
I tested it with about one minute long clip (about 3060 frames). This test video is streets of London shot hand-held in a moving car so it has a lot of interesting and challenging movement. The results are clear: standard method is clearly best (according to SSIM metric) and the worst is original replacebad.

Code:
original replacebad; improved replacebad; standard;
wins 389; 662; 2065;
total   2736.119385;   2736.775879; * 2739.429688;
average 0.894157; 0.894371; 0.895238;
The standard method won about 67,5% of the time, improved replacebad won 21,6% and original replacebad won 12,7%. The winning stats are revealing an important point: it's not enough to compare a single frame, with luck any of those methods could be a winner for that single frame.

It would be interesting to find out if these statistics are similar with other kind of videos.

It's still a bit of mystery how MVTools is using those extra frames, but at least I don't have to worry about it, it's doing a good job (at least for the case of single frame replacement).
zorr is offline   Reply With Quote
Old 10th April 2018, 12:57   #10  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
Quote:
It's still a bit of mystery how MVTools is using those extra frames
If its the below that you are talking about, [EDIT: Below from post #1]
Code:
A	B		D	E
	---------------->
	<----------------
-------->		<--------
It dont work like that, perhaps below better explains
Code:
A      B       C       D       E
       <--------------->
<------------------------------>
To use [EDIT: sort of] like above (and predict C) then you would need,
B<->D with vectors result created @ same frame number as B (delta=2, shift forward 1 frame to bad position C),
and
A<->E with vectors result created @ same frame number as A (delta=4, shift forward 2 frames to bad position C),
and somehow mix both sets of results, the A<->E will nearly always produce worse result that B<->D, so I dont see the point.
MFlowInter only ever uses two frames to predict [EDIT: Interpolate] the result (delta is only the offset to select the 2nd src frame number used), it aint like eg MDegrainN where N pairs of vectors would be used for result.

EDIT: Both above assume arg ' time=50', ie 50.0% of the way between the source frames B,B+2 (ie @ B+1, where delta=2),
and 50.0% of the way between source frames A,A+4 (ie @ A+2, where delta=4).
EDIT: So, the forward shift in above cases are delta * 50.0% ie 2*0.5(=1) and 4*0.5(=2).

EDIT:
Quote:
I couldn't find any mention of the two extra frames and vectors which are also used.
There are no extra frames nor vectors used. Below 2 source frames, 2 vectors. (modified from your source)
Code:
source=AviSource("...")

BAD_FRAME = 11

super=source.MSuper(pel=2)
backward_vectors = MAnalyse(super, isb = true, truemotion=true, delta=2)
forward_vectors = MAnalyse(super, isb = false, truemotion=true, delta=2)
inter = source.MFlowInter(super, backward_vectors, forward_vectors, time=50)    # Interpolated, requires forward shift to correct position
BEFORE_BAD    = source.trim(0,BAD_FRAME-1)  # clip before bad frame
AFTER_BAD     = source.trim(BAD_FRAME+1,0)  # clip after bad frame
# Using Trim and Splice instead of shift, but exactly the same result.
# frame number where vectors coincide @ BAD_FRAME - 1 (equivalent to forward shift of 1 when spliced)
# Forward shift (where delta=2 & time=50.0) @ 2 * 0.5 = 1
# BAD_RELATIVE = -(2*0.5) = -1 == BAD_FRAME-1
FIXED_SHIFTED = inter.trim(BAD_FRAME-1,-1)    # 2nd arg to trim of -1 means trim 1 frame
Return  BEFORE_BAD ++ FIXED_SHIFTED ++ AFTER_BAD
2 vectors, backward_vectors, forward_vectors.
2 source frames, BAD_FRAME-1 and BAD_FRAME+1. (Each and every frame @ inter[n] is interpolated from source[n] and source[n+2], req fwd shift of 1)
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 10th April 2018 at 17:46.
StainlessS is offline   Reply With Quote
Old 10th April 2018, 22:20   #11  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
Quote:
Originally Posted by StainlessS View Post
MFlowInter only ever uses two frames to predict [EDIT: Interpolate] the result (delta is only the offset to select the 2nd src frame number used), it aint like eg MDegrainN where N pairs of vectors would be used for result.

...

There are no extra frames nor vectors used. Below 2 source frames, 2 vectors. (modified from your source)
Ahh, I didn't realize those extra frames / vectors weren't common knowlegde. Ok, let me make my case:

This message from Gavino is where I got that idea:

Quote:
Originally Posted by Gavino View Post
Didée's explanation is the same as mine (but clearer!), and would suggest that the delta=2 approach cannot be improved upon. Your results are therefore puzzling and invite further explanation, which I believe I have found.

By looking at the source code, I have discovered that for occlusion areas, MVFlowInter also uses information from the two further surrounding frames (current-delta and current+2*delta).
So the diagram becomes, when interpolating between C and C+d,
Code:
       ----> <----
    C-d     C     C+d     C+2d
             ---->   <----
It is this additional information that will be more accurate when delta=1, explaining your findings and making your two-stage approach a better one. Well done.
I also looked at Pinterf's MVTools2 source and found this in MVFlowInter.cpp:

Code:
   
// Get motion info from more frames for occlusion areas
PVideoFrame mvFF = mvClipF.GetFrame(n, env);
mvClipF.Update(mvFF, env);// forward from prev to cur
mvFF = 0;
PVideoFrame mvBB = mvClipB.GetFrame(nref, env);
mvClipB.Update(mvBB, env);// backward from next next to next
mvBB = 0;
I tried to figure out which frames are referenced by that code but my C++ experience is pretty limited (and it doesn't help that for some reason the MVTools2 project doesn't open in Visual Studio 2017 Community edition).

Finally there's the experimental proof. The only difference in original and modified replacebad-function is that they have different frame at index 4 when the interpolation is done at index 2.

Code:
Original replacebad has frame D at index 4:		
0	1	2	3	4	5
	A	B	D	D
	?	<------->	?
	
Modified replacebad has frame E at index 4:
0	1	2	3	4	5
	A	B	D	E
	?	<------->	?
I don't know how the extra vectors should be drawn, my original drawing of those was just an educated guess.

If MFlowInter would only use frames 2 and 3 then frame 4 should not have any effect on the result, but my SSIM test proved that it does (unless I made some horrific error in my code).

I should have been more clear in my first message, but this is why I strongly believe that MFlowInter is using more than two frames in the interpolation.
zorr is offline   Reply With Quote
Old 11th April 2018, 04:55   #12  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
Zorr,
I thought I understood how it all worked, then you ruined it for me

Methoughts that forwards Vectors were at first badframe - 1, but are of course at first badframe - 1 + delta,
just when I had a nice little model in my head, of course they dont move about depending upon which mv func you intend to use,
what was I thinking.

Here a little folly I was playing with,
Below, SrcS and SrcE are the source frames for interpolation, XSTEP is the horizontal movement between frames.
Can go a bit wonky when XSTEP and BLKSZ are bigger than 8 or DELTA bigger than 2.
Forward and backwards vectors are aligned just to check them.

Code:
WID=640
HIT=64
XSTEP=8

DELTA=2
TIME=50

PAD=16
BLKSZ=8

MASK_MOTION     = 0
MASK_SAD        = 1
MASK_OCCLUSION  = 2
MASK_HORIZONTAL = 3

#MASK_TYPE=MASK_MOTION
#MASK_TYPE=MASK_SAD
#MASK_TYPE=MASK_OCCLUSION
MASK_TYPE=MASK_HORIZONTAL

BlankClip(width=WID,height=HIT,Length=1,Color=$FFFFFF,Pixel_type="Y8")

A=Trim(0,-1)
B=A.BlankClip
c=A.BlankClip(Length=0)
For(i=1,WID/XSTEP-1) {
    L=A.Crop(0,0,i*XSTEP,0)
    R=B.Crop(0,0,WID-i*XSTEP,0)
    Frm = StackHorizontal(L,R)
    c= c ++ Frm
}

srcS=c.ConvertToY8
srcE=SrcS.Loop(0,0,DELTA-1)   # Delete Delta frames, end interp src

super = srcS.MSuper(pel=2,hpad=PAD,vpad=PAD)
fvec  = MAnalyse(super,blksize=BLKSZ,  isb = false, truemotion=true, delta=DELTA)
bvec  = MAnalyse(super,blksize=BLKSZ,  isb = true,  truemotion=true, delta=DELTA)
inter = srcS.MFlowInter(super, bvec,fvec, time=TIME)

MBV   = srcS.MMask(bvec,kind=MASK_TYPE)
MFV   = srcS.MMask(fvec,kind=MASK_TYPE)
MFVDEL= MFV.Loop(0,0,DELTA-1)          # Delete DELTA frames from start, align n+delta with n

StackVertical(srcS,SrcE,MBV,MFVDEL,Inter)
SSS="""
    Subtitle(String(current_Frame,"SRC[n=%.0f] (1st interp srcS)")                     , Y=0.5*HIT,Align=5)
    Subtitle(String(current_Frame+DELTA,"SRC[n+Delta=%.0f] (2nd interp srcE)")         , Y=1.5*HIT,Align=5)
    Subtitle(String(current_Frame,"BVEC[n=%.0f]")+MTyp(MASK_TYPE)                     , Y=2.5*HIT,Align=5)
    Subtitle(String(current_Frame+DELTA,"FVEC[n+DELTA=%.0f]")+MTyp(MASK_TYPE)         , Y=3.5*HIT,Align=5)
    Subtitle(string(current_frame+DELTA*TIME/100.0,"Predicted @ [%.2f]")              , Y=4.5*HIT,Align=5)
"""
Last.ScriptClip(SSS)
TXT=SrcS.BlankClip(Height=20,Length=1,Color=$404040)
TXT=TXT.Subtitle(String(Delta,"Delta=%.0f")+String(Time," : Time=%.2f")+String(XSTEP," : BLKSZ=%.0f")+String(XSTEP," : XSTEP=%.0f"))
StackVertical(TXT,Last)

Return Last.ConvertToRGB32

Function MTyp(Int n) {Return " Type="+Select(n,"Motion","Sad","Occlusion","Horizontal","Vertical","ColorMap")}








EDIT: And just for good measure.



EDIT: Oops, had vectors labels wrong, fixed.
Backward/forward vectors seem back to front to me, my head hurts
Somebody interested in figuring out if the script is correct ???
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 8th August 2018 at 00:53.
StainlessS is offline   Reply With Quote
Old 11th April 2018, 05:54   #13  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,096
Quote:
Originally Posted by StainlessS View Post
Zorr,
I thought I understood how it all worked, then you ruined it for me .
Almost 15 years using MVTools and, like you, every time I am absolutely certain I have the MAnalyze logic totally figured out, I find myself back at square one.
johnmeyer is offline   Reply With Quote
Old 12th April 2018, 00:08   #14  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
Quote:
Originally Posted by StainlessS View Post
Zorr,
I thought I understood how it all worked, then you ruined it for me
Sorry about that, I have a bad habit of ruining people's lives.

Quote:
Originally Posted by StainlessS View Post
Methoughts that forwards Vectors were at first badframe - 1, but are of course at first badframe - 1 + delta,
just when I had a nice little model in my head, of course they dont move about depending upon which mv func you intend to use,
what was I thinking.
I don't know if this is going to help anyone but here's the internal model I'm currently working with:

Code:
Forward vectors clip	
0	1	2	3	4	5
--0----->	--2----->	--4----->		
	--1----->	--3----->
	
Backward vectors clip

0	1	2	3	4	5
<-0------	<-2------	<-4------
	<-1------	<-3------
MAnalyze returns a clip with motion vector data. So here we have forward and backward vectors with delta=1. The arrow points from the frame whose pixels are being motion estimated to the frame which is the motion estimation target. The number in the arrow tells in which frame the motion vector is stored at. So for example forward vector 1->2 is stored at frame 1 whereas backward vector 2->1 is stored at frame 1 as well. I agree with you that this arrangement makes using the vectors easier than pretty much any other way. For a good measure here's forward vectors with delta=2:

Code:
0	1	2	3	4	5	6
--0------------->	--3------------->
	--1------------->	--4------------->
		--2------------->
When MFlowInter uses these vectors it takes the forward AND backward vector at the current frame. It will also use the pixels in the current frame and current frame + delta. Now this is already enough to do motion interpolation, but it can also use other frames (and it does) because it has a whole clip of vectors it can use.

Quote:
Originally Posted by StainlessS View Post
Here a little folly I was playing with,
Below, SrcS and SrcE are the source frames for interpolation, XSTEP is the horizontal movement between frames.
Can go a bit wonky when XSTEP and BLKSZ are bigger than 8 or DELTA bigger than 2.
Forward and backwards vectors are aligned just to check them.
That's a really interesting folly (just a side note: I'm learning a lot from your scripts so thank you).

I played with it and made a few changes that made sense to me. I think it's better to show those forward/backward vectors which are actually used to generate the interpolated frame, so I removed the frame adjustment of forward vectors and I swapped the display order (forward, then backward). Also there was a minor bug, XSTEP was displayed for the value of BLKSZ.

Code:
WID=640
HIT=64
XSTEP=8

DELTA=2
TIME=50

PAD=16
BLKSZ=2

MASK_MOTION     = 0
MASK_SAD        = 1
MASK_OCCLUSION  = 2
MASK_HORIZONTAL = 3

#MASK_TYPE=MASK_MOTION
#MASK_TYPE=MASK_SAD
#MASK_TYPE=MASK_OCCLUSION
MASK_TYPE=MASK_HORIZONTAL

BlankClip(width=WID,height=HIT,Length=1,Color=$FFFFFF,Pixel_type="Y8")

A=Trim(0,-1)
B=A.BlankClip
c=A.BlankClip(Length=0)
For(i=1,WID/XSTEP-1) {
    L=A.Crop(0,0,i*XSTEP,0)
    R=B.Crop(0,0,WID-i*XSTEP,0)
    Frm = StackHorizontal(L,R)
    c= c ++ Frm
}

srcS=c.ConvertToY8
srcE=SrcS.Loop(0,0,DELTA-1)   # Delete Delta frames, end interp src

super = srcS.MSuper(pel=2,hpad=PAD,vpad=PAD)
fvec  = MAnalyse(super,blksize=BLKSZ,  isb = false, truemotion=true, delta=DELTA)
bvec  = MAnalyse(super,blksize=BLKSZ,  isb = true,  truemotion=true, delta=DELTA)
inter = srcS.MFlowInter(super, bvec,fvec, time=TIME)

MBV   = srcS.MMask(bvec,kind=MASK_TYPE)
MFV   = srcS.MMask(fvec,kind=MASK_TYPE)

StackVertical(srcS,SrcE,MFV,MBV,Inter)
SSS="""
    Subtitle(String(current_Frame,"SRC[n=%.0f] (1st interp srcS)")                     , Y=0.5*HIT,Align=5)
    Subtitle(String(current_Frame+DELTA,"SRC[n+Delta=%.0f] (2nd interp srcE)")         , Y=1.5*HIT,Align=5)
    Subtitle(String(current_Frame,"FVEC[n=%.0f]")+MTyp(MASK_TYPE) 			, Y=2.5*HIT,Align=5)
    Subtitle(String(current_Frame,"BVEC[n=%.0f]")+MTyp(MASK_TYPE)                     , Y=3.5*HIT,Align=5)
    Subtitle(string(current_frame+DELTA*TIME/100.0,"Predicted @ [%.2f]")              , Y=4.5*HIT,Align=5)
"""
Last.ScriptClip(SSS)
TXT=SrcS.BlankClip(Height=20,Length=1,Color=$404040)
TXT=TXT.Subtitle(String(Delta,"Delta=%.0f")+String(Time," : Time=%.2f")+String(BLKSZ," : BLKSZ=%.0f")+String(XSTEP," : XSTEP=%.0f"))
StackVertical(TXT,Last)

Return Last.ConvertToRGB32

Function MTyp(Int n) {Return " Type="+Select(n,"Motion","Sad","Occlusion","Horizontal","Vertical","ColorMap")}
By the way, I had to run this script with GImport because it has GScript syntax (the for loop), is that how you run it or is there some other trick?
zorr is offline   Reply With Quote
Old 12th April 2018, 04:00   #15  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
Quote:
GScript syntax (the for loop)
I wrote in AVS+, which dont need Gscript("") stuff, you can wrap in a string eg SSS="""...""" and GScript(SSS).
EDIT: or Eval(SSS) for avs+.

Quote:
XSTEP was displayed for the value of BLKSZ.
I guess thats down to copy/paste and being in a hurry to try get some sleep,
which is what I'm gonna try to do now, I'll have a play with mod tomorrow.

ta ta.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 12th April 2018 at 13:56.
StainlessS is offline   Reply With Quote
Old 12th April 2018, 19:40   #16  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
The motion vector stuff is stored at the frame that the vector arrow is pointing at, (which is what Didee said in one of the linked threads I think).

Dont really have time at the moment to go over your post, but will maybe later.

The below script reverses direction of animated clip (by default), and in doing so is less confusing, I think (movement comes in from same direction as the frames).

Code:
Function VectorTest(Int "Delta",Float "Time",Int "MaskT",Int "BlkSz",Int "XStep",Bool "Align",Bool "HFlip",String "CS") {
/*
    VectorTest(), An MvTools2::MFlowInter folly. by StainlessS @ Doom9 : https://forum.doom9.org/showthread.php?t=175373

    Req AVS+ or GSCript, GRunt, MvTools2, RT_Stats v1.43+
    Avs v2.58, Avs/+ v2.60.

    Delta, Default 1.     As for MvTools2
    Time,  Default 50.0,  As for MvTools2
    MaskT, Default 3,     As for MvTools2::MMask(kind=MASK_TYPE), 0=Motion, 1=Sad, 2=Occlusion, 3=Horizontal, 4=Vertical, 5=ColorMap
    BlkSz, Default 4,     As for MvTools2
    XStep, Default 8,     Motion per frame of synthesized clip.
    Align, Default True,  Aligns forward vector frame n+Delta to n. False show frame n of forward vector.
    HFlip, Default False, False, Animate from Right to Left, else Left to Right.
                          (Right to Left is less confusing, same direction that the frames come in from).
    CS,                   Avs v2.5 defaults "YV12" else "Y8".

    Returns RGB32 clip.
*/
    Function MTyp(Int n) {Return " Type="+Select(n,"Motion","Sad","Occlusion","Horizontal","Vertical","ColorMap")}
    myName="VectorTest: "
    IsAvsPlus=(FindStr(UCase(versionString),"AVISYNTH+")!=0)    HasGScript=RT_FunctionExist("GScript")
    HasGrunt=RT_FunctionExist("GScriptclip")                    HasMvTools2=RT_FunctionExist("MSuper")  Is26=VersionNumber>=2.6
    Assert(IsAvsPlus||HasGscript,myName+"Essential AVS+ or GScript installed")
    Assert(HasGrunt,myName+"Essential GRunt installed")         Assert(HasMvTools2,myName+"Essential MvTools2 installed")
    Delta=Default(Delta,1)  Time=Default(Time,50.0)             MaskT=Default(MaskT,3)    BlkSz=DefaulT(BlkSz,4)
    XStep=Default(XStep,8)  Align=Default(Align,True)           HFlip=Default(HFlip,False)
    CS=Default(CS,Is26?"Y8":"YV12")
    OLap=(BlkSz>=4)?BlkSz/2:RT_Undefined
    FuncS="""
        Function Fn(clip c,Int Delta,Float Time,Int XStep,Bool Align,Bool HFlip,String mType) {
            c    n=current_frame
            Hit=(Height-20)/5
            if(Align) {
                Steps=640/XStep
                x=((n+1)*XStep) + (Delta*XStep/2)
                x=Min(x,(Steps-1)*XStep)
                x=HFlip?x:639-x
                cF=RT_YPlaneMin(n=n,x=x,y=Round(2.5*Hit)+20,w=1,h=1)-128
                cB=RT_YPlaneMin(n=n,x=x,y=Round(3.5*Hit)+20,w=1,h=1)-128
                Z=c.BlankClip(width=1,height=1,Color=$FFFFFF,Length=1)
                OverLay(Z,x=x,y=Round(2.5*Hit)+20)
                OverLay(Z,x=x,y=Round(3.5*Hit)+20)
                Subtitle(RT_String("@x=%d FGrey=128%+d : BGrey=128%+d",x,cF,cB))
            } else {
                Subtitle("Align=False, Colors NOT shown")
            }
            Subtitle(String(n,"SRC[n=%.0f] (1st interp srcS)")                      , Y=0.5*Hit+20,Align=5)
            Subtitle(String(n+Delta,"SRC[n+Delta=%.0f] (2nd interp srcE)")          , Y=1.5*Hit+20,Align=5)
            (Align)
              \ ? Subtitle(String(n+Delta,"FVEC[n+DELTA=%.0f]")+mType+ " (Aligned)" , Y=2.5*Hit+20,Align=5)
              \ : Subtitle(String(n,"FVEC[n=%.0f]")+mType                           , Y=2.5*Hit+20,Align=5)
            Subtitle(String(n,"BVEC[n=%.0f]")+mType                                 , Y=3.5*Hit+20,Align=5)
            Subtitle(string(n+Delta*Time/100.0,"MFlowInter Predicted @ [%.2f]")     , Y=4.5*Hit+20,Align=5)
            Return Last
        }
        Wid=640 Hit=64  Steps=Wid/XStep
        White=BlankClip(width=WID,height=HIT,Length=1,Color=$FFFFFF,Pixel_type=CS)  Black=White.BlankClip  srcS=Black.BlankClip(Length=0)
        For(i=1,Steps-1) { W=White.Crop(0,0,i*XStep,0)  K=Black.Crop(0,0,Wid-W.Width,0)  Frm=StackHorizontal(K,W) srcS=srcS++Frm }
        srcS=(HFlip)?srcS.FlipHorizontal:srcS
        srcE=SrcS.Loop(0,0,Delta-1)                                # Delete Delta frames, end interp src
        super=srcS.MSuper(pel=2,hpad=16,vpad=16)
        fvec =MAnalyse(super, isb=false, blksize=BlkSz, overlap=OLap, delta=Delta, truemotion=true)
        bvec =MAnalyse(super, isb=true,  blksize=BlkSz, overlap=OLap, delta=Delta, truemotion=true)
        inter=srcS.MFlowInter(super, bvec,fvec, time=Time)
        mbv=srcS.MMask(bvec,kind=MaskT)    mfv=srcS.MMask(fvec,kind=MaskT)
        mfv=(Align)?mfv.Loop(0,0,Delta-1):mfv                      # Align, Delete DELTA frames from start, align n+delta with n
        TXT=SrcS.BlankClip(Height=20,Length=1,Color=$404040)
        StackVertical(TXT,srcS,SrcE,MFV,MBV,Inter)
        mType=mTyp(MaskT)
        ARGS = "Delta,Time,XStep,Align,HFlip,MType"
        Last.GScriptClip("Fn(last, "+ARGS+")", local=true, args=ARGS)
        DIR=(HFlip) ? " : ---->" : " : <----"
        TXT=TXT.Subtitle(String(Delta,"Delta=%.0f")+String(Time," : Time=%.2f")+
            \ String(BLKSZ," : BLKSZ=%.0f")+String(XSTEP," : XSTEP=%.0f")+" : ALIGN="+String(ALIGN)+" : HFlip="+String(HFlip)+DIR)
        return StackVertical(TXT,Last)
    """
    IsAvsPlus?Eval(FuncS):GScript(FuncS)
    Return Last.ConvertToRGB32
}
EDIT: Above changed Align default to True.
EDIT: Added 2nd line to title bar, not show in below graphics (see post #23).


Code:
VectorTest(Delta=2,Align=False)

EDIT: Above, white stepping in from the Right.

EDIT: I still prefer Align=True for FVec. [EDIT: Align is now true by default]
Code:
VectorTest(Delta=2) # EDIT: NOW, Align=True is default


Code:
VectorTest(Delta=1) # EDIT: NOW, Align=true is default


Code:
VectorTest(Delta=2,time=33.33)


Code:
VectorTest(Delta=2,time=66.66)


EDIT: Script and images updated.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 24th April 2018 at 19:43. Reason: update
StainlessS is offline   Reply With Quote
Old 12th April 2018, 22:58   #17  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
Quote:
Originally Posted by StainlessS View Post
The motion vector stuff is stored at the frame that the vector arrow is pointing at, (which is what Didee said in one of the linked threads I think).
You are right, he said it here.

Quote:
Originally Posted by Didée View Post
The twist is just that when creating the interpolation between this-and-next-frame, MFlowInter does not use the forward and backward vector of the current frame. It uses the backward vector of the current frame, and the forward vector of the (current+delta) frame.
And in that case it's better to show the forward vector of frame n+delta like you did originally (Align=true in your latest version). So again we would see the vectors as they are used in creating the interpolated frame.

My initial thought process can be visualized like this: Overlay the forward vectors mask over the source frame n and then move the pixels under the mask area wherever the vector tells them to move but multiply the vector length by the time percentage (50% works with delta=2). Do the same with backwards mask, overlay it over frame n+delta and move the pixels under the mask area. Then combine these two frames somehow and that's the final result. That makes sense because I can see that the correct pixels would be moved.

But since the forward/backward vectors are aligned perhaps it works by first combining the forward / backward vectors into one and using those combined vectors to move pixels in both frames.
zorr is offline   Reply With Quote
Old 14th April 2018, 09:37   #18  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
Quote:
And in that case it's better to show the forward vector of frame n+delta like you did originally
Post #16 script and images updated, changed default Align=True (+ a few minor mods).

EDIT:
Quote:
But since the forward/backward vectors are aligned perhaps it works by first combining the forward / backward vectors into one and using those combined vectors to move pixels in both frames.
Nope, dont think so.

Quote:
My initial thought process can be visualized like this: Overlay the forward vectors mask over the source frame n and then move the pixels under the mask area wherever the vector tells them to move but multiply the vector length by the time percentage (50% works with delta=2). Do the same with backwards mask, overlay it over frame n+delta and move the pixels under the mask area. Then combine these two frames somehow and that's the final result. That makes sense because I can see that the correct pixels would be moved.
Think this is more like it. (The forward vector align is only for display in the VectorTest thing, of course vectors are not aligned).
I'm guessin' that both sets of results are produced and combined, if bad matches between the two then maybe blurred, degree of blurring depending upon badness of mismatch.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 17th April 2018 at 10:42.
StainlessS is offline   Reply With Quote
Old 17th April 2018, 23:47   #19  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 208
Quote:
Originally Posted by StainlessS View Post

Quote:
Originally Posted by zorr View Post
My initial thought process can be visualized like this: Overlay the forward vectors mask over the source frame n and then move the pixels under the mask area wherever the vector tells them to move but multiply the vector length by the time percentage (50% works with delta=2). Do the same with backwards mask, overlay it over frame n+delta and move the pixels under the mask area. Then combine these two frames somehow and that's the final result. That makes sense because I can see that the correct pixels would be moved.
Think this is more like it. (The forward vector align is only for display in the VectorTest thing, of course vectors are not aligned).
Ok bear with me because now I'm even more confused. I think we already established that the vectors used are the backward vector of frame n and the forward vector of frame n+delta. And those are the exact vectors shown in your folly when Aligned=true. But if you take those vectors and try use them like I described above it wouldn't really work because the vectors are aligned but the pixels in frames n and n+delta are not.



Of course there are other ways to use those vectors but I'm still a bit confused because it looks like the forward vector stored at n+delta is written to the location the arrow is pointing *at* whereas the backward vector stored at frame n is written to the location where the arrow is pointing *from*... perhaps this is by design to make the process easier or maybe my brain is malfunctioning.

Last edited by zorr; 17th April 2018 at 23:48. Reason: Grammar fix
zorr is offline   Reply With Quote
Old 22nd April 2018, 18:04   #20  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,384
Code:
            n   n+delta
            |    |
            <---- BV        Vectors used for eg MDegrain, vectors stored @ frame where arrowhead points (ie n).
     FV ---->               Pixels moved from frame at back end of arrow (n+/- delta), along vector to synth frame where arrowhead points.
        |
        n-delta


            n   n+delta
            |    |
            <---- BV        Vectors used for eg MFlowInter, Uses the next one along forward vector so that vectors used are from
            |    |          either side of the frame that will be synthesized (reason, only part of the vector distances are
        FVI ---->           used, based on Time arg).
            n    n+delta
                            For MFlowInter, the created vectors are exactly the same as for eg MDegrain, its just that it uses the
                            next forward vector so that backward and forward vectors straddle the predicted frame, MFlowInter
                            just uses the vectors in a slightly different way to MDegrain.
                            The synthesized frame lies logically somewhere between n and n + delta (based on Time arg), and is
                            physically created at frame n, and so needs to be relocated to the required bad frame position in clip.
Hope the above makes sense.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 22nd April 2018 at 18:25.
StainlessS is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:41.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.