Generate broken frames from neighbour frames using MVFlow

[absence]

I have some video digitized from analogue tape. Due to bad tape occational frames are full of garbage/noise. Is it possible to use MVTools to estimate motion of nearby frames, and MVFlow to calculate a new frame that I can replaced a broken one with? How would such a script look? (Assume that I'll deinterlace the video first, since I imagine interlaced video will only cause trouble.) I've only used MVFlow with complete videos and have limited understanding of how to work on just a few frames in Avisynth.

[wonkey_monkey]

This is taken straight from the MVTools 2 doc page:

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
# Assume bad frames are 50 and 60
trim(0,49) ++ inter.trim(49,-1) \
 ++ trim(51,59) ++ inter.trim(59,-1) ++ trim(61,0)

If you have to use MVTools 1, it's this:

backward_vectors = source.MVAnalyse(isb = true, truemotion=true, pel=2, delta=2, idx=1)
forward_vectors = source.MVAnalyse(isb = false, truemotion=true, pel=2, delta=2, idx=1)
inter = source.MVFlowInter(backward_vectors, forward_vectors, time=50, ml=70, idx=1)
# Assume bad frames are 50 and 60
source.trim(0,49) ++ inter.trim(49,-1) \
 ++ source.trim(51,59) ++ inter.trim(59,-1) ++ source.trim(61,0)

[absence]

Originally Posted by davidhorman

This is taken straight from the MVTools 2 doc page:

Oh, you're right! And I've even visited that page... :P Sorry for the obvious question, and thanks for pointing it out.

[wonkey_monkey]

No worries, it's not very obvious - I often wish the MVTools pages were a little more navigable. I only went there to look at the individual filters myself, and was lucky to spot the examples.

David

[2Bdecided]

Didn't someone turn this into a simple function? (i.e. just provide clip and bad frame numbers) - or am I dreaming?!

Cheers,
David.

[absence]

Originally Posted by 2Bdecided

Didn't someone turn this into a simple function? (i.e. just provide clip and bad frame numbers) - or am I dreaming?!

No idea, but that would be nice.

I have another question too: What if there are two consecutive broken frames? I thought it would be logical to increase delta by 1, like this:

super = src.MSuper
bvec = super.MAnalyse(isb=true, delta=3)
fvec = super.MAnalyse(isb=false, delta=3)
inter = src.MFlowInter(super, bvec, fvec, time=50, ml=70)
src.trim(0,10264) ++ inter.trim(10264,-2) ++ src.trim(10267,0)

However, this only fixes frame 10265. 10266 is still broken, but looks different than the original. What do I need to write to get this right? And, why? (so I can understand how it works and help myself in the future)

[MatLz]

function rbf(clip rbfin,int ftr)
{
rbfin
sup=msuper()
bv=manalyse(sup,isb=true,delta=2)
fv=manalyse(sup,isb=false,delta=2)
inter=mflowinter(sup,bv,fv,time=50,ml=70)
trim(0,ftr-1)++inter.trim(ftr-1,-1)++trim(ftr+1,0)
}

I changed the time value, I find it more natural in motion.
When I will have more time, I will modify it because there are exeptions (start or end of scenes with or without motion/scrolling...)
Edit:I'm wrong, it is my insertion for missing function where the time value is 33.

[pbristow]

Hmmm... Thinking about Absence's query about how to regenerate 2 consecutive frames, I tried to unpick what the original example does... At first I thought I'd found an error, then realised I was getting confused because of the way manalyse is used twice to get both forward and backward vectors, both of which have to be passed to MFlowInter, even though for a simple linear interpolation only one set of vectors is needed. (I'd love to know what MFlowInter actually does with the extra vectors, if anything! Perhaps they're only needed for when "time" is negative (e.g. -50% instead of +50%)? )

In the case of recovering one bad frame between two good ones, the value of delta is two, because we're comparing a frame just before the glitch with the one two frames later. The target frame N is created by interpolating a point halfway between N-1 and N+1 (using time=50 - It's a percentage). A stream of candidate frames is created by interpolating the source clip 50% forward along the vectors, and then frame N of the source is replaced with frame N-1 of this stream of replacement candidates.

Now let's move on to the problem of recovering two consecutive frames, N and O, where O=N+1, from good frames N-1 (aka M) and N+2 (aka P):

M N O P
N-1 N N+1 N+2
good bad bad good

The data we want to use for frame N is an interpolation 33.3% of the way from M to P. The data we want for frame O (= N+1) is an interpolation 66.7% of the way from M to P.


Script, with explanatory comments:

function RecoverFramePair(clip Source, int N)
{
	# N is number of the first frame in Source that needs replacing. 
	# Frames N and N+1 will be replaced.

	Source
	Sup=msuper()

	bv=manalyse(Sup, isb=true, delta=3)
		# Looks 3 frames ahead and calculates vectors backward
		# to present frame.
	fv=manalyse(Sup, isb=false, delta=3)  
		# Looks 3 frames behind and calculates vectors forward 
		# to present frame.

	CandidatesForN = mflowinter(sup, bv, fv, time=33.3, ml=70)
		# Produces a stream of frames interpolated 1 frame 
		# ahead of the source, based on data from 0 and 3 frames 
		# ahead of the source. To find the correct candidate for
		# replacing frame N of the source, we want the N-1th 
		# frame of this stream. ( (N-1) + 1 = N )

	CandidatesForO = mflowinter(sup, bv, fv, time=66.7, ml=70)
		# Produces a stream of frames interpolated 2 frames 
		# ahead of the source, based on data from 0 and 3 frames 
		# ahead of the source. To find the correct candidate for
		# replacing frame N+1 of the source, we want the N-1th 
		# frame of *this* stream. ( (N-1) + 2 = (N+1) )

	Source.trim(0,FrameN-1) ++ CandidatesForN.trim(N-1,-1) \
	    ++ CandidatesForO.trim(N-1,-1) ++ Source.trim(FrameN+2,0)
}

Howzatt?

[Gavino]

Originally Posted by pbristow

Howzatt?

Fine, apart from the typos in the last line where you have written 'FrameN' instead of 'N'.

[absence]

Originally Posted by pbristow

Howzatt?

Perfect, thanks!

New problems just keep appearing though: I'm trying to clean up video from analogue tape that is slightly damaged in some places, causing broken scanlines to "roll" down the picture over time (e.g. in one frame scanlines 10-15 are broken, in the next 12-17 are broken, in the next 14-19, and so on). The entire sequence is too long to throw away and interpolate, and if I try to interpolate a shorter part of it, MVTools detects the broken scanlines as motion and puts them back in. Is this possible to handle?

[pbristow]

I can suggest an approach for that, but getting it to work will probably be a lot of work. It's a useful approach for all sorts of tricky problems, and it goes like this:

1. Work out how to detect when, and where, the problem is occuring.
(In this case:
- Are the broken scanlines partcularly bright, compared to the rest of the frame?
- One thing that should distinguish them is that they're horizontal, so some sort of horizontal edge-detection might do the trick. )
Using this you can create a mask to switch your "Problem solving magic" on and off in the right places.

2. Create a copy of the source where the effects of the problem are somehow reduced or obscured, so that they won't confuse your main processing. (In this case, motion estimation. E.g., use the trick of a blurred pre-filtered clip (as shown in the MVTools documentation), but with a lot more blurring keyed in (using the mask from step 1) wherever the break-up is detected on the original.)

3. Apply your main processing.
- i.e., use MVtools to create motion-compensated copies, from which you can patch-in replacement picture detail.
As to *how* to use MVTools, I would suggest using MDegrain3, with a large block size (16), "truemotion" on, and lots of overlap (8). As long as the blocksize is significantly bigger than the width of the damaged area, then the motion estimator has a better chance of locking onto real picture detail, rather than the movement of the problem. You might also gets some benefit from pushing he lambda higher than normal (thereby telling the algorithm "if one block seems to be behaving weirdly compared to its neighbours, then trust the neighbours, not the odd one!")

4. Look at the result, compare it with the original, and decide whether to use it as it is or refine it further (e.g. by keying in the MVDegrain output only where the problem was detected, using the mask from (1). You might get a more uniform and pleasing result, though, if your use it over the whole frame... I'm finding MVDegrain really useful for averaging out all sorts of temporary glitches and fluctuations, not just "noise".)

Good luck!

[pirej]

deleted by me, pirej

[Alex_ander]

You can apply MFlowFPS (to double framerate) to frames created from healthy fields only (either with NNEDI2 or simply by removing each other frame after separatefields.assumeframebased with further resizing), then re-interlace. Or simply leave it progressive (without using MFlowFPS) if your material give artifacts after MFlowFPS.

[um3k]

Is this what you're looking for?

##Insert MVFlow stuff here
Interleave(SelectEven(mvflowoutput), SelectOdd(premvflowclip))

Switch selecteven and selectodd if it gives the wrong frames.

[pirej]

Tried to do it me self, but.. no luck(knowledge)
i'm using a script like this

AVISource("D:\test.avi")
AssumeTFF()
separatefields()

super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)

trim(0,410) ++ inter.trim(410,-1) \
 ++ trim(412,412) ++ inter.trim(412,-1) \
 ++ trim(414,414) ++ inter.trim(414,-1) \
 ++ trim(416,416) ++ inter.trim(416,-1) ++ trim(418,0)

And it's doing a great job, but its not just couple of frames like in the example, its most part of the video..
So, how to adapt this script so that it does it's job (automatically) on every second field???
I tried MFlowFPS to double framerate from the upperfields, but it gives me a lot of artifact's.
And i couldn't follow um3k's instruction.. excuse me if i'm bordering you all.
By.

[Alex_ander]

AVISource("D:\test.avi")
AssumeTFF()
SeparateFields()
SelectEven()# or Odd to keep right fields
AssumeFrameBased()
LanczosResize(720,576)# if PAL
#__end__# enable this line for progressive output if you get artifacts
super = MSuper(pel=2)
backward_vec = MAnalyse(super, isb = true)
forward_vec = MAnalyse(super, isb = false)
MFlowFps(super, backward_vec, forward_vec, num=50, den=1, ml=100)# PAL, for NTSC num=60000, den=1001
AssumeTFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()

It even appeared to work with my random NTSC test video and converted it to true PAL.

[pirej]

Thank you Alex_ander, but let me tell you all how "smart" i am..
In meanwhile, i cleaned the (2)head's on my old vcr, and i did some scrubbing.. with a piece of white paper and 70% alcohol, waited for some 1/2 hour, and i did another test capture.
And guess what.. most of the error's disappeared, and there is some more color and... it's much better.


p.s. this is second time im cleaning the vcr-heads, but now i pressed the paper little harder while cleaning...

[pirej]

Hi, i was using pbristow's function example for "RecoverFramePair", and i did modified it to get another function for recovering ONE frame just to make it simple, and more practical for me.
So the script is something liek this:

function RecoverFramePair(clip Source, int N)
{
	# N is number of the first frame in Source that needs replacing. 
	# Frames N and N+1(O) will be replaced.

	Source
	Sup=msuper()
	bv=manalyse(Sup, isb=true, delta=3)
	fv=manalyse(Sup, isb=false, delta=3)  
	CandidatesForN = mflowinter(sup, bv, fv, time=33.3, ml=100)
	CandidatesForO = mflowinter(sup, bv, fv, time=66.7, ml=100)
	
	Source.trim(0,N-1) ++ CandidatesForN.trim(N-1,-1) \
    ++ CandidatesForO.trim(N-1,-1) ++ Source.trim(N+2,0)
}

function RecoverOneFrame(clip Source, int N)
{
	# N is number of the frame in Source that needs replacing. 
	# Frame N  will be replaced.
	Source
	Sup=msuper()
	bv=manalyse(Sup, isb=true, delta=3)
	fv=manalyse(Sup, isb=false, delta=3)  
	CandidatesForN = mflowinter(sup, bv, fv, time=50.0, ml=100)
	
	Source.trim(0,N-1) ++ CandidatesForN.trim(N-1,-1) ++ Source.trim(N+1,0)
	    
}

RecoverOneFrame(7)
RecoverFramePair(26)
RecoverOneFrame(30)
RecoverFramePair(50)

So, i'm using "manalyse" twice in the same script.. is it making any sense?
Does it slow down the script?
Does it generate some kind of problems?
I did not try to encode using the script yet, but the preview in avsp looks ok.
By.

[Didée]

It is very inefficient to make the MVTools-stuff inside such a function. Not a problem when you have to fix only a few random frames in a stream. But imagine a longer source, and there are, say, 100 frames to fix. Technically, you need one superclip, and one pair of vectorclips. But with these functions, the script would create one hundred superclips, and one hundred pairs of vectorclips.

In this scenario, the script most likely will crash because it runs out of memory.

A memory-efficient way is to do the MVTools-stuff one time at the top level, and use the functions only for replacing.

function RecoverFramePair(clip Source, int N)
{
	# N is number of the first frame in Source that needs replacing. 
	# Frames N and N+1(O) will be replaced.
	
	Source.trim(0,N-1) ++ CandidatesForN.trim(N-1,-1) \
    ++ CandidatesForO.trim(N-1,-1) ++ Source.trim(N+2,0)
}

function RecoverOneFrame(clip Source, int N)
{
	# N is number of the frame in Source that needs replacing. 
	# Frame N  will be replaced.
	
	Source.trim(0,N-1) ++ CandidatesForC.trim(N-1,-1) ++ Source.trim(N+1,0)
}

Source

	Sup = msuper()
	bv = manalyse(Sup, isb=true, delta=3)
	fv = manalyse(Sup, isb=false, delta=3)  
global 	CandidatesForN = mflowinter(sup, bv, fv, time=33.3, ml=100)
global 	CandidatesForO = mflowinter(sup, bv, fv, time=66.7, ml=100)
global 	CandidatesForC = mflowinter(sup, bv, fv, time=50.0, ml=100)

RecoverOneFrame(7)
RecoverFramePair(26)
RecoverOneFrame(30)
RecoverFramePair(50)

(Yes, usage of global variables is a bit hack-ish ... it would be more sane to hand the CandidateX clips as arguments.)

[Gavino]

Another problem with this code (also in original by pirej) is that the functions don't work for N=1 (ie when replacing frame 1 by interpolating between frame 0 and frame 2 or 3).

Originally Posted by Didée

function RecoverFramePair(clip Source, int N)
{
	# N is number of the first frame in Source that needs replacing. 
	# Frames N and N+1(O) will be replaced.
	
	Source.trim(0,N-1) ++ CandidatesForN.trim(N-1,-1) \
    ++ CandidatesForO.trim(N-1,-1) ++ Source.trim(N+2,0)
}

Because of the special meaning of 0 as the 2nd argument to Trim, this will give the wrong result when N=1.
It should be written instead as Source.Trim(0, -N), and similarly in RecoverOneFrame.

(See stickboy's classic Caveats of using Trim in functions.)