Inverse of Decimate() ...? [Archive]

pbristow

10th June 2010, 00:40

I have a video that has been wrongly decimated from 30fps to 24fps. My aim is to restore the missing frames using MVtools. The tricky bit is how to identify (automatically) the frames either side of the missing one, and get interpolation to only happen at that point.

My first thought was to just double the frame rate using MVTools2 (from 24 to 48fps), and then apply Decimate() (from the Decomb package) three times to remove the surplus frames (going from a cycle of 8 down to a cycle of 5)... But the problem is sometimes Decimate will pick the *original* frames to delete, and keep the interpolated ones, which is not ideal.

When going the other way (from 30 to 24fps), Decimate uses an algorithm to detect and delete the frame in each cycle which is most similar to its predecessor. Does anyone have a filter that can detect and [do something - ideally, create a new frame via MVTools] with the frame in each cycle that is *least* like its predecessor? :confused:

If not, I guess I can try canibalising the code from Decomb/Decimate to create my own... :)

Guest

10th June 2010, 00:52

Can't you just buy the DVD?

pbristow

10th June 2010, 01:01

Can't you just buy the DVD?

Heh. :) No, because there isn't one. It's not commercially released material.

Guest

10th June 2010, 01:18

What is it then and where did you get it?

pbristow

10th June 2010, 07:01

Hmmm... Looks like you're testing me for a possible rule 6 violation? :)

It's band footage, owned by the performer, shot by an inexperienced friend of theirs some years ago on a domestic, analog NTSC camcorder, and now passed to me to see if I can "make it look nicer". They plan to use clips and/or stills from it in their promotional material / website.

At some point before it got to me, someone decided to convert it to PAL by dropping frames and speeding it up. I dunno what software they used, and I don't have access to the original 30fps source. What I do know is that the pictures are reasonably clean (considering), but the motion jerks after every fourth frame, which is how I figured out what had been done to it. So, having slowed the thing down to 24fps, and I just need to fill in those missing frames to recover the original motion. After that, I'll try doing an MVTools-based conversion to PAL in case they ever want it for a DVD later, but for the web we'll probably stick with the 30fps version. (The reason we're talking in frames rather than fields per second is that for some reason, every 2nd field of the footage is much poorer quality than the first one, so I'm deinterlacing and throwing away the muddy fields first).

Feel free to Google the name Talis Kimberley. :)

pbristow

10th June 2010, 07:37

Ah! Just had a brainwave.

If I stick with the original "Decimate three times" idea, but change the motion interpolation to create the moment 55% past each frame, rather than 50%, then each interpolated frame will be more like its successor than its predecessor. That should bias Decimate towards dropping the interpolated frames, rather than the originals.

I'll give that a try later today.

pbristow

11th June 2010, 19:27

Hmmm... I had the logic backwards, of course. I settled in the end for a mocomp to the point 42% ahead of each source frame, (using MFlowInter) and then multiple Decimate calls.

It sort of works, but still isn't great. (E.g. static sequences get artificially shortened in favour of keeping all the frames with significant movement.)

Looks like I have another filter-hacking project to play with, then... :) (Maybe one day I'll get something working well enough to publish here!)

foxyshadis

11th June 2010, 21:51

What about MFlowFPS(.... fps*2 ....).selectevery(10,0,2,4,6,8,9).Assumefps(fps*5/4)? You already have a constant rhythm, so you can drop interpolated frames consistently.

IanB

11th June 2010, 23:24

Hint:- For Assumefps(fps*5/4) use AssumeScaledFPS(5, 4) to avoid rounding errors

pbristow

12th June 2010, 20:36

What about MFlowFPS(.... fps*2 ....).selectevery(10,0,2,4,6,8,9).Assumefps(fps*5/4)? You already have a constant rhythm, so you can drop interpolated frames consistently.

That would work if the pattern of missing frames was entirely consistent, but it isn't (probably because the source was decimated from 30*1000/1001 to exactly 24). As soon it slips by one frame from the pattern you're xpecting, you're then adding an extra frame to each cycle where it's not needed and still have a missing one elsewhere.

The idea was to exploit the cycle mechanism of Decimate to find the correct place to add an extra frame each time (there would still be the *occasional* extra frame or missing frame, but not often enough or regular enough to be so annoying).

At the moment I'm running a test using TDecimate, which can directly prune N from M, instead of using Decimate three times. I'll compare that with the file I got from using Decimate and see if it's good enough; If not, then its time to fire up my C compiler. :)

What I have in mind is a filter based on either Decimate or TDecimate (to do the detectionof "least similar pair of frames"), which accepts a second clip as "fill-in source". Wherever a big jump is found between frame n and frame n+1 of the input clip, then frame n of the fill-in source is inserted between them. That leaves the choice of how to create the fill-in frames entirely up to the user, which should make it a nice general purpose filter (you could use mvtools, or depan, or a simple blend of n with n+1, or even some completely unrelated source clip to create a freaky stobe/intercut effect!).

Whether I'll actually knuckle down and get it coded is another matter. I'm always big on ideas, slow on implementation... =:o{

Didée

12th June 2010, 20:46

It would definetly by nice to have such a filter.:) I've faced the problems a few times, too. And while it is more-or-less possible to solve that problem with currently available arms, it is quite some hassle to script that up.

One possible method was posted here (http://forum.doom9.org/showthread.php?p=879309#post879309), which was an even nastier case (decimated interlaced frames). Though I think the not-scriptclip based method worked more stable - but that one never has been posted.

Alex_ander

13th June 2010, 08:11

I think it is important here to protect original frames from decimation using the fact that they all belong to either Even or Odd frames in case of framerate duplication with mvtools. Probably this could be done within text processing engine of MultiDecimate plugin. During pass 1 it creates mfile.txt with frame difference values, then according to selected type of decimation cfile.txt is created where [in my understanding] the frames are evaluated in weighted numbers for decimation and finally the list of remaining frame numbers dfile.txt is created. Not sure but probably something like post-processing of cfile.txt could be done for protecting even frames (maybe by forced setting those values to above-threshold with re-addressing decimation flag to another candidate frame). Looks like this would need including some new 'protect Even/Odd' option in the plugin. Could neuron2 comment? Thanks.

johnmeyer

16th June 2010, 23:04

I have had to deal with all manner of badly done frame-rate conversions, usually dealing with NTSC -> PAL -> NTSC. I end up with video that has big jumps where too many frames (or fields) were deleted. Sometime, strangely, the video also has duplicates. So, you end up with video that stutters and hesitates, something I assume you are seeing in your video.

There is no universal solution, because the solution depends on the actual cadence of your footage. However, here is a code snippet I found a long time ago here in the forum that may come in handy:
function filldrops (clip c)
{
vf=c.mvanalyse(truemotion=true,pel=2,isb=false,delta=1,idx=1)
vb=c.mvanalyse(truemotion=true,pel=2,isb=true,delta=1,idx=1)
global filldrops_d = c.mvflowinter(vb,vf,time=50,idx=1)
global filldrops_c = c
c.scriptclip("""ydifferencefromprevious()==0? filldrops_d : filldrops_c""")

}
usage:

This code finds all duplicated frames and replaces the second instance of two duplicate frames with a motion-estimated frame. As you can see from the code, it only does this if the "ydifferencefromprevious" value is exactly zero. I think you might be able to adapt this, perhaps, to do exactly the opposite, namely to look for unusually large differences between frames, and synthesize a new frame from this. By itself, this code isn't going to solve your problem, but it may be a useful building block.

FWIW, here is the interlaced version I created which uses MVTools2 instead of MVTools. The above code was created by someone else, but only works on progressive. Also, I slightly increased the comparison parameter so the algorithm will synthesize a motion-estimated field even if the two adjacent fields are not perfectly identical.

This leads to an important concept for you to understand:

It is generally OK if your script occasionally unecessarily creates a synthesized frame (or field, for interlaced video).

As you already know, you'd rather use the original frames wherever possible, but if once in awhile your script goofs and substitutes an interpolated frame, as long as this doesn't happen often, it won't matter. After all, the video is going to have lots of interpolated frames, so if you have a few extra, it is usually not a big deal.

function filldropsI (clip c)
{
even = c.SeparateFields().SelectEven()
super_even=MSuper(even,pel=2)
vfe=manalyse(super_even,truemotion=true,isb=false,delta=1)
vbe=manalyse(super_even,truemotion=true,isb=true,delta=1)
filldrops_e = mflowinter(even,super_even,vbe,vfe,time=50)

odd = c.SeparateFields().SelectOdd()
super_odd=MSuper(odd,pel=2)
vfo=manalyse(super_odd,truemotion=true,isb=false,delta=1)
vbo=manalyse(super_odd,truemotion=true,isb=true,delta=1)
filldrops_o = mflowinter(odd,super_odd,vbo,vfo,time=50)

evenfixed = ConditionalFilter(even, filldrops_e, even, "YDifferenceFromPrevious()", "lessthan", "0.1")
oddfixed = ConditionalFilter(odd, filldrops_o, odd, "YDifferenceFromPrevious()", "lessthan", "0.1")

Interleave(evenfixed,oddfixed)
Weave()
# Following line removed after suggestion by Gavino
# AssumeFieldBased()
AssumeBFF()
}

Oh, if you want to replace the first frame rather than the second frame, use this code (this is for progressive frames using MVTools2):function filldropsnext (clip c)
{
previous = Loop(c,2,0,0)
super=MSuper(previous,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
filldrops = mflowinter(previous,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceToNext()", "lessthan", "0.1")
return fixed
}

P.S. Just after I posted I read the post suggesting using cfile, etc. I spent several years developing a technique to do exactly what he was suggesting. Once you do this as a two-pass operation, you can spend time with the parameter file and use a spreadsheet to figure out exactly which frames you want to decimate. So, you could use motion interpolation to increase the frame rate; run the software to get the cfile.txt file; put this in Excel and decide which pattern of frames or fields you want to decimate; create a new file that has the decimation frames; and then run the second pass. You can get exactly what you want with this approach, but it isn't exactly automatic.

Didée

16th June 2010, 23:56

Gavino

17th June 2010, 00:36

@johnmeyer
Perhaps I have misunderstood, but shouldn't all your functions be using motion vectors with delta=2 rather than delta=1 if you are trying to replace a duplicate frame by an interpolation between its neighbours?

Also, AssumeFieldBased() should not normally be used after Weave(), as this describes a field-separated clip.

foxyshadis

17th June 2010, 02:57

delta=1 means the two nearest neighbors. delta=2 is the next closest, ie, 2 back & 2 forward.

johnmeyer

17th June 2010, 03:03

Relying on YDifferenceToPrev/Next will get you pretty much nowhere. (At least not for an "automated" solution ... and spending hours or days on manual analysis of spreadsheet data is not what you usually want to do.) The valus for "typical" differences are both scene-dependent and motion-dependent, i.e. that value will change all the time.

You are correct that the values change all the time. However, in a local setting -- that is within a few frames on either side of the current frame -- they tend to not vary much unless there is a discontinuity of the type we are talking about. Thus, to make my idea work, you have to compile a moving average and then compare the current frame vectors with the changes between the last few frames and the changes in the next few frames. This is what I did with the spreadsheets I developed to analyze the cfile.txt vectors. This will, in my experience, generally give you a correct result. And, as I stated in my last post, if your script creates a few interpolated frames that aren't needed, all you will do is replace an original frame with an interpolated one. If you do everything correctly, this frame should be spatially and temporally very similar to the original frame. After all, the point of this is to create frames to replace those that are missing, and if those don't look pretty close to the missing original, then the whole "exercise" is pointless.

So, if the function isn't 100% perfect and a few extra "good" frames get replaced with interpolated frames, it shouldn't make much difference, especially compared to the horrible stutter that you generally get with what the OP described.

Perhaps I have misunderstood, but shouldn't all your functions be using motion vectors with delta=2 rather than delta=1 if you are trying to replace a duplicate frame by an interpolation between its neighbours?

Well, I always defer here to people who know more than I do. However, this filldrop function was originally created (not by me) for the situation where a capture card drops a frame and, instead of eliminating that frame and screwing up the audio sync, the card duplicates the previous frame. Thus, if you look at the second of the two duplicate frames, the frame before (which is a duplicate) is the correct frame for that time slot. The frame after is also correct. Therefore, you simply want to interpolate a new frame at the current location using the information from the two adjacent frames. Perhaps I don't understand correctly what the "delta" parameter does, but I thought it specified which frame to look at as the reference for MAnalyze, with delta=1 and isb=true looking at the last frame and delta=1 and isb=false looking at the next frame. If I use delta=2, I thought that meant the algorithm would look two frames ahead and two frames behind and therefore would not create the current frame as accurately. I thought the point of being able to look several frames away is to accumulate a more accurate estimate by averaging the vectors from the one frame away estimate with those from two frames away, etc, using a lesser weight for the estimates derived from frames which are further away.

But like I said, perhaps I don't understand correctly. I can tell you, for absolute certain, that the function I posted works fantastically well at replacing the second duplicate with a near-perfect motion-estimated synthesized frame. I have used it many, many times on dozens of hours of video (long story how I came into possession of so much lousy video ...)

Also, AssumeFieldBased() should not normally be used after Weave(), as this describes a field-separated clip.

Darn, I think you may have pointed this out in a post I made six months ago. I corrected that code, but somehow this snuck back in.

This is a mistake.

What is the proper etiquette on this board? Should I edit my original post to correct the mistake? Or, is this discussion in this post sufficient?

Didée

17th June 2010, 16:09

Gavino

17th June 2010, 18:43

delta=1 means the two nearest neighbors. delta=2 is the next closest, ie, 2 back & 2 forward.
That's true as far as the vectors are concerned, but read on...
... you simply want to interpolate a new frame at the current location using the information from the two adjacent frames. Perhaps I don't understand correctly what the "delta" parameter does, but I thought it specified which frame to look at as the reference for MAnalyze, with delta=1 and isb=true looking at the last frame and delta=1 and isb=false looking at the next frame. If I use delta=2, I thought that meant the algorithm would look two frames ahead and two frames behind and therefore would not create the current frame as accurately.
Per MVTools doc (http://avisynth.org.ru/mvtools/mvtools2.html), MFlowInter interpolates "between current and next (by delta) frame".
More precisely, what this means is that frame n of MFlowInter is an interpolation between input frames n and n+delta (using forward vector n->n+delta and backward vector n+delta->n).
So in general, if you want to replace a 'bad' frame n by interpolating between its immediate neighbours n-1 and n+1, you need to use delta=2 (and take the result from frame n-1 of MFlowInter).
See the example in the MVTools doc, "recreate bad frames by interpolation with MFlowInter".
I can tell you, for absolute certain, that the function I posted works fantastically well at replacing the second duplicate with a near-perfect motion-estimated synthesized frame.
You're right, the function certainly does work, and now I see why.
What happens is that if a duplicate is found at frame n, it interpolates between n and n+1, since delta=1.
At first sight, this is wrong, but since frame n is a duplicate of frame n-1, it is equivalent to the desired interpolation between n-1 and n+1, so the final result comes out OK. (Whether by accident or by design, I'm not sure. :))
Darn, I think you may have pointed this out in a post I made six months ago. I corrected that code, but somehow this snuck back in.
I'd forgotten about that myself. At least I'm consistent in my advice. :)
What is the proper etiquette on this board? Should I edit my original post to correct the mistake? Or, is this discussion in this post sufficient?
It varies. If it's a function that someone else might find later on and want to use, I think it's better to correct the original, with a note saying it's been changed (so that the subsequent discussion still makes sense).

johnmeyer

17th June 2010, 23:29

What happens is that if a duplicate is found at frame n, it interpolates between n and n+1, since delta=1.
At first sight, this is wrong, but since frame n is a duplicate of frame n-1, it is equivalent to the desired interpolation between n-1 and n+1, so the final result comes out OK. (Whether by accident or by design, I'm not sure. )
Thank you VERY much for taking the time to challenge my thinking. This has been very useful for me because it forced me to go back and look at several of my scripts that use this "filldrops" script code to replace bad frames. I also researched every post on this forum that uses the "mflowinter" call.

Unfortunately, this created a new source of confusion, and now makes me think that both the filldrops function and even the example given in the MVTools2 documentation are incorrect.

See if you agree.

Here is my thinking: MAnalyze is used to create motion vectors between the current frame and adjacent frames. However, if the current frame is totally screwed up (blank, duplicate, full of noise, etc.) then won't MAnalyze create vectors that are of absolutely no value, regardless of what number I assign to "delta?" In other words, if I have one frame that is completely, totally different than the frame before and the frame after, then motion vectors which use the pixels in this current, bad, reference frame will have no useful relationship to anything.

Thus, it would seem to me that even the example in the documentation is wrong, and that to make this work, I must actually use, as the input to MSuper and MFlowInter, the good frame immediately before the bad frame and generate all vectors with reference to that frame. I should then use delta=1 for the backward vector, and delta=2 for the forward vector (to skip over the bad frame). Finally, the call to MFlowinter should specify time=66.67.

Here is an attempt to explain the problem with a diagram:

F1 F2 F3 F4

F3 is the bad frame. What follows is pseudo code of what I think is actually the correct way to interpolate to replace a truly bad (garbage) frame:

super=MSuper(F2,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=2)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
F3 = mflowinter(F2,super,vbe,vfe,time=66.67)

So, what I am trying to understand is how the bad frame "F3" can be used to generate useful vectors and, if it cannot, how the demonstration code given in the documentation can be correct.

Now that I understand this, I can see how the filldrops code I quoted above (I think "MugFunky" invented it) does generate satisfactory results even though it actually isn't correct. It sort of works because the "bad frame" is a duplicate of a perfectly good frame, and therefore the backward vector created between it and a duplicate of itself will still be valid, and the vector created between it and the next frame will be valid. In other words, because the previous frame is a duplicate, the backward vectors will not contribute anything useful to the interpolation, but they won't cause the thing to blow up. Therefore, I get an interpolated frame that is useful, but which I now suspect is not as good as it might be.

So, I now believe that the original documentation and also the original filldrops code are both incorrect, and that the proper approach is the one I discovered when I had to replace the first of the two frames rather than the second. If you go back to my initial post in this thread, that is the code in the third code block that uses the Loop function to return the previous frame to the MSuper and MFlowInter function.

Gavino

18th June 2010, 00:42

Unfortunately, this created a new source of confusion, and now makes me think that both the filldrops function and even the example given in the MVTools2 documentation are incorrect.

See if you agree.
No, I believe the example in the docs is correct.

You're right that any vector derived from a 'bad' frame is of little value. But if you look at my explanation of MFlowInter again, you will see that with delta=2, neither the forward nor the backward vector used actually involves the 'bad' frame. Using your notation, the vectors used would be F2->F4 and F4->F2, and to replace F3 you would use the frame of MFlowInter corresponding to F2.
Your filldrops function written this way becomes:
function filldrops(clip c)
{
previous = Loop(c,2,0,0) # needed to select correct frame
super=MSuper(previous,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=2)
vbe=manalyse(super,truemotion=true,isb=true,delta=2)
filldrops = mflowinter(previous,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
return fixed
}
In fact, I remember now I posted similar code in response to a similar question in this thread.

Your original code does work correctly (possibly by accident) as long as the detected frame is an exact duplicate.
So does your filldropsnext function, and for the same reason. Because of the Loop function, it replaces frame n by interpolating this time between n-1 and n, and since n is a duplicate of n+1, this produces the required interpolation between n-1 and n+1.

johnmeyer

18th June 2010, 02:01

Well, I have now spent several hours on this and can say with a fairly high degree of confidence that the code example in the MVTools2 documentation does NOT work. I am not trying to be argumentative. Instead, I am just reporting what I have found from actually using the code on a real piece of video that has a totally bad frame.

I created a sample clip where I took one frame from a completely unrelated piece of video and replaced a good frame in a video where someone was moving their hand rapidly. The moving hand makes it easy to see what is going on with the interpolated video.

Here is that clip:

http://dl.dropbox.com/u/1561578/test%20bad%20frame%20clip.avi

I then copied the code from the MVTools2 documentation, and modified it to point to my video and to compare the original video with the synthesized video:

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

source=AVISource("E:\Disney Films\test bad frame clip.avi").ConvertToYV12
super = MSuper(source)
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70)
#return inter
stackvertical(source,inter)

This should be very easy for anyone to duplicate. Just download the 5 MB clip (uses HufYUV codec), modify the script above to point wherever you put the video, and modify (or delete) the line which loads the MVTools2 plugin.

What you will see is that when the "bad" frame appears in the top of the stacked video, the interpolated frame in the bottom actually contains part of the bad frame. This is obviously NOT what we want. We don't want to see any part of that bad frame.

I have tried many, many permutations to try to get this to work, including using delta=1, delta=2, and "mixed deltas" where I use a different delta for the forward vector than the backward vector. I also have tried using the previous frame as the reference and also the next frame as the reference. This actually gets me quite close and I think will be the way I ultimate solve this puzzle. This is what I was proposing in my previous post.

I am getting close to the correct solution to this problem and will post it when I get it working correctly.

BTW, the conditional code to only use the interpolated frame when needed works just fine (I haven't included that above), and as I posted earlier in this thread, if it occasionally uses an interpolated frame rather than the original, it doesn't matter much for most frames (because the interpolated version is usually pretty close to the original).

Quick P.S.: The code you posted does work, and it is exactly what I am working on. However, it is NOT the code in the documentation because it uses the previous frame for the vectors. I am trying to get it to work delta=1 for the backwards vector and delta=2 for the forward vector in order to get a more accurate estimation. It seems wrong to look at something two frames away in the backwards direction when the frame immediate preceding that frame is perfectly OK.

Gavino

18th June 2010, 02:43

I then copied the code from the MVTools2 documentation, and modified it to point to my video and to compare the original video with the synthesized video:
source=AVISource("E:\Disney Films\test bad frame clip.avi").ConvertToYV12
super = MSuper(source)
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70)
stackvertical(source,inter)

You are missing an important point from the docs example - to replace frame 50, it uses inter.Trim(49, -1). So the last line of your code should be
stackvertical(source, inter.Loop(2,0,0))
Quick P.S.: The code you posted does work, and it is exactly what I am working on. However, it is NOT the code in the documentation because it uses the previous frame for the vectors.
So (in effect) does the documentation, as I just explained above.
I am trying to get it to work delta=1 for the backwards vector and delta=2 for the forward vector in order to get a more accurate estimation. It seems wrong to look at something two frames away in the backwards direction when the frame immediate preceding that frame is perfectly OK.
It doesn't look two frames backwards from the bad frame, it looks two frames back from the next good frame to the current one. Remember I said it uses vectors F2->F4 and F4->F2.

To quote the documentation:
"It uses backward "mvbw" and forward "mvfw" motion vectors to create picture at some intermediate time moment between current and next (by delta) frame. It uses pixel-based (by MFlow method) motion compensation from both frames."

There are only two frames involved (current and current+delta) and it uses vectors in both directions between those frames.

johnmeyer

18th June 2010, 03:31

It doesn't look two frames backwards from the bad frame, it looks two frames back from the next good frame to the current one. Remember I said it uses vectors F2->F4 and F4->F2.

Well, I just read every one of your posts in other threads about how "delta" works, and I think my old, stupid brain finally "got it." My problem is that I thought the delta, when isb=true, was looking backwards from the current frame. Instead, now that I understand your explanation, everything is referenced to frames after the current frame. Put another way, the backwards vector refers to vectors that point backwards to the current frame, and not vectors backwards from the current frame.

Now everything makes sense.

I'll spend a little more time on this, just to make sure it all makes sense and make sure the resulting code works, and then I'll post the result, just in case someone stumbles on this thread later on. The only contribution to the world as we know it that will come from posting that code is that it integrates the conditional code with the interpolation code.

Thank you for helping me.

Gavino

18th June 2010, 10:47

Well, I just read every one of your posts in other threads about how "delta" works, and I think my old, stupid brain finally "got it."
...
Thank you for helping me.
You're welcome - it has also helped me to order my thoughts about what MFlowInter actually does in detail, which is not entirely clear from the documentation.
For anyone else still not sure, reread my posts above (also the relevant explanations in this thread).
I'll spend a little more time on this, just to make sure it all makes sense and make sure the resulting code works, and then I'll post the result, just in case someone stumbles on this thread later on. The only contribution to the world as we know it that will come from posting that code is that it integrates the conditional code with the interpolation code.
Isn't that just what I did in post #21? I guess you could also provide similarly revised versions of filldropsI and filldropsnext.

johnmeyer

18th June 2010, 18:36

johnmeyer said: I'll spend a little more time on this, just to make sure it all makes sense and make sure the resulting code works, and then I'll post the result ...

Gavino replied: Isn't that just what I did in post #21? I guess you could also provide similarly revised versions of filldropsI and filldropsnext.

Yes, you did post that in #21, but that conditional code isn't sufficient because it will trigger a frame replacement at most scene changes, which is definitely not what we want. Fortunately, a single bad frame is quite different from a scene change because it is different from BOTH of its neighbors. This also makes it different from other types of frame-to-frame variations that normally occur. Thus, it is actually pretty easy to identify with fairly high reliability, but it requires two tests, not one: you need to compare to both the previous and next frames and only if both differences are large for BOTH should the frame be replaced.

So, bottom line, without this double test, you don't get very good identification of a bad frame.

Doing two comparisons instead of one is easy and I'm sure you could do that in ten seconds (it took me ten minutes because I am stupid, and also because I didn't realize you have to declare a variable before it can be used inside of a conditional). But I like to always strive for perfection. So, what I'd like to do, especially given the nature of the original post and also given the number of other posts in this forum about similar problems, is to derive a "local threshold" using a moving average of the YDifference value. Ideally, that average should be reset after a scene change. Given how AVISynth works, I don't know if I can do that. It turns out that my script works pretty well without it, but I thought I'd take a little time an try to add this.

One way or the other, I'll post the result.

Gavino

18th June 2010, 19:23

Yes, sorry John, I was forgetting that replacing duplicate frames wasn't the original purpose of this thread.
I didn't realize you have to declare a variable before it can be used inside of a conditional
I don't want to hijack this thread even further, but I'm not sure what you mean by this - could you elaborate?

johnmeyer

18th June 2010, 20:36

I don't want to hijack this thread even further, but I'm not sure what you mean by this - could you elaborate?

OK. The problem is that this line won't work:

ConditionalFilter(clip, filldrops, clip, "YDifferenceFromPrevious(clip)>10 && YDifferenceToNext(clip)>10", "greaterthan", "0")

unless you make "clip" global:

global clip = c

I initially just used "c," which is declared in the function header, as the argument to YDifferenceToNext. This doesn't work. Once I stumbled on this, I remembered seeing it somewhere, either in the AVISynth documentation, the AVISynth Wiki, or maybe here in the forum.

Also, while we have posted a lot, I think what we've done should help the original poster, so we haven't completely taken this thread away from its original purpose (although we have concentrated on only one aspect of his problem).

Finally, I may have to take back everything I said about how MVTools "delta" works. Both the documentation as well as the tests I am doing today make it seem that my original understand of how MAnalyze uses Delta is correct, namely that it does look at the preceding frame. I think our confusion comes from the fact that the Mflowinter function DOES only look forward and DOES use the delta for determining how many frames ahead to interpolate. Thus delta is being used in one way by MAnalyse, but in a different way by MFlowInter. This is why you have to use delta=2 so it (MFlowInter) looks beyond the bad frame. And it is why you can't use a different delta for the forward and backward vectors in MAnalyze because this causes MFlowinter to interpolate to the wrong frame.

However, it seems increasingly clear to me that my initial understanding of MAnalyse is correct: the isb vectors look to the previous frame and the next frame rather than to the next frame and from the next frame. The more I think about that, the more I realize that it MUST be correct. After all, what would be the difference between a vector to a frame, and a vector back from that same frame, other than a negative sign?

I have now actually been able to prove this with the solution I am creating. This solution uses the motion vectors from the previous frame and from two frames ahead, and then feeds these to MFlowInter. This does produces output that is different, and better, than the code in the documentation which I now once again believe to be wrong, or at least less than optimal.

The solution I am developing uses the conditional filter to create a frame stream that replaces the bad frame with the next frame. This in essence creates the scenario that I've dealt with in the past, namely a perfect, exact duplicate in place of the bad frame. Once you have this, you can use delta=1 for both the forward and backward vectors, but the delta=1 for the forward vector is now actually pointing to a frame that is two frames ahead. MFlowInter is "happy" because it is being told to interpolate to the next frame, but since the "next" frame is now actually two frames ahead, I can use 50 for the time parameter and get the correct outcome.

Now, anticipating what you are thinking, no this is NOT the equivalent of what you've posted, with delta=2 because the vector which points to the previous frame (and I really do think it points to the previous frame) is now pointing to the previous frame, and not the frame prior to the previous frame. Thus, I get a much more accurate interpolation because the isb=true vector is constructed from the immediately adjacent previous frame rather than the frame before that.

The more I thought about it, and the more times I read the MVTools2 documentation, the more I have become convinced that your explanation of the isb delta vectors cannot be correct for MAnalyse. The MVTools2 documentation for MAnalyse states: delta : set the frame interval between the reference frame and the current frame. By default, it's 1, which means that the motion vectors are searched between the current frame and the previous ( or next ) frame.

More to come ...

Gavino

18th June 2010, 20:55

OK. The problem is that this line won't work:
ConditionalFilter(clip, filldrops, clip, "YDifferenceFromPrevious(clip)>10 && YDifferenceToNext(clip)>10", "greaterthan", "0")
unless you make "clip" global
No need to do that. Since 'clip' is the first parameter to ConditionalFilter, it becomes 'last' inside the run-time script.
ConditionalFilter(clip, filldrops, clip, "YDifferenceFromPrevious()>10 && YDifferenceToNext()>10", "greaterthan", "0")
I'll answer your other points when I've digested them fully.

Didée

18th June 2010, 21:54

the isb vectors look to the previous frame and the next frame rather than to the next frame and from the next frame.
Frankly, I'm not sure if I'd understand it from that explanation. But OK, it's a bit dry to explain with just words. Let's use a visual representation.

With "delta=1", the vectors produced by MAnalyse for frame "c" look like
(isb = false) (isb = true)
(c-1) --------------> (c) <-------------- (c+1)
(forward vec) (backward vec) And yes: "forward" and "backward" refer to the orientation of the vector.

I think our confusion comes from the fact that the Mflowinter function DOES only look forward and DOES use the delta for determining how many frames ahead to interpolate. Thus delta is being used in one way by MAnalyse, but in a different way by MFlowInter.
Partly incorrect. MFlowInter does look both forward and backward. However, it is using a little trick while doing so.

The "vector situation" for MFlowInter (with delta=2 vectors) :
<----------
A B C D E
---------->
The trick is: in order to interpolate between frame X and frame X+1 (or, more precisely, between frame X and frame X+[delta]), MFlowInter uses the "backward" vector of frame X, and the "forward" vector of frame X+(delta).
Well, call it a "trick", or rather a "technical necessity" ... it is done this way because it can't be done in any other way.

while we have posted a lot, I think what we've done should help the original poster, so we haven't completely taken this thread away from its original purpose

I reckon that the original problem already has been solved. :)

Look at post#18 (http://forum.doom9.org/showthread.php?p=1409324#post1409324). pbristow hasn't visited since then so I don't know if it works for him. For me, it certainly does work for exactly the posed problem.

johnmeyer

18th June 2010, 22:59

ConditionalFilter(clip, filldrops, clip, "YDifferenceFromPrevious(clip)>10 && YDifferenceToNext(clip)>10", "greaterthan", "0")
No need to do that. Since 'clip' is the first parameter to ConditionalFilter, it becomes 'last' inside the run-time script.

You are correct about not needing the first "clip" in this ConditionalFilter call, and also about not needing the "(clip)" arguments in the YdifferenceToNext calls ...

except ...

when clip is something other than "last." As my script has evolved, some versions used "last" and others used frames other than last, so I definitely did need it in earlier versions of the script.

I see that while I was preparing this post that Didée provided an excellent explanation of how vectors are created and used within MVTools2. His explanation reinforces what I always believed to be true prior to this thread and, after a temporary misunderstanding by me, was what I tried to explain in my most recent post. Even when Didée said I was partially incorrect, it was only because I used the word "look" to describe what Mflowinter was doing. Obviously it has to use the vectors from previous as well as future frames, so in that sense it is "looking" in both directions. However, the interpolation is only done in the forward direction (i.e., it does not interpolate a frame at a time prior to the current frame). So, I totally agree with Didée's explanation, and very much appreciate his post.

Now, to hopefully finish off my contributions here, the following is my final (for now) script for fixing bad frames. It uses a much better (in most ways) method of finding a bad frame (although it of course fails on the first and last two frames of the input video).

To detect bad frames the script looks at ratios of YDifference values between this frame and the previous frame, and the previous frame and the frame before that. It also does the same thing with the following frames. This achieves something similar to the moving average idea, but without having to compute a moving average. In other words, it normalizes the YDifference values to whatever is "normal" for this portion of this clip, and only takes action when there is a discontinuity from the norm in BOTH the positive and negative direction from the current frame. Thus, scene changes are ignored (because the frame AFTER the first frame of a scene change has little difference), and it doesn't really matter whether YDifference values are large are small, as long as they are relatively similar from frame to frame.

This code uses this logic to determine when to replace a "bad" frame with a duplicate of the frame following. This corrected stream is then fed to MAnalyse which then feeds its vectors to Mflowinter. By feeding the stream which now contains duplicates in place of the bad frames, the "filldrops" logic developed years ago by MugFunky now works just as it always has.

The advantage this approach has over what is shown in the MVTools documentation is that it avoids using a delta of 2 in the backwards direction. Using this backwards delta of 2 skips the previous frame which, being closer in time is going to provide a better starting point for the estimation vectors. Indeed, I was able to prove this by comparing the frame I got using the code shown in the MVTools2 documentation with what I got with this code.

This produces a better result.

So, I am finished here, and I hope this has helped someone.

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

global bad_thresh = 20
source=AVISource("E:\frameserver.avi").ConvertToYV12
corrected=replacebad(source).AssumeFPS(29.97, false)

return corrected

function replacebad (clip c)
{
global clip = c

# Replace bad frame with a duplicate of the frame immediately after the bad frame
goodframes = ConditionalFilter(clip, trim(clip,1,0), clip, \
"(YDifferenceFromPrevious(clip)/YDifferenceFromPrevious(loop(clip,2,0,0))) * \
(YDifferenceToNext(clip)/YDifferenceToNext(trim(clip,1,0)))", "greaterthan", "bad_thresh")

# This next line gets the previous frame in the stream that now contains duplicates instead of bad frames
previousframe = Loop(goodframes,2,0,0)

super=MSuper(previousframe,pel=2)

vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)

replacement = mflowinter(previousframe,super,vbe,vfe,time=50)

fixed = ConditionalFilter(clip, replacement, clip, "(YDifferenceFromPrevious(clip)/YDifferenceFromPrevious(loop(clip,2,0,0))) * \
(YDifferenceToNext(clip)/YDifferenceToNext(trim(clip,1,0)))", "greaterthan", "bad_thresh")

return fixed
}

Didée

18th June 2010, 23:14

Nitpick:
However, the interpolation [by MFlowInter] is only done in the forward direction (i.e., it does not interpolate a frame at a time prior to the current frame).
That's the point which I was referring to with "partially incorrect". Or at least, "misunderstandable".
It's correct the the final interpolation of MFLowInter is only "forward in time", in relation to the "current" frame. But in order to produce that interpolation, MFlowInter indees does both a forward interpolation and a backward interpolation, and mixes them acc. to the internal metrics. (After all, you are handing *two* vectors over to MFlowInter ... the 2nd vector isn't bogus. It is used.)
The twist is just that when creating the interpolation between this-and-next-frame, MFlowInter does not use the forward and backward vector of the current frame. It uses the backward vector of the current frame, and the forward vector of the (current+delta) frame.

Gavino

18th June 2010, 23:32

confusion comes from the fact that the Mflowinter function DOES only look forward
No, as I said (and Didée has since explained much more clearly), it looks both backwards and forwards, but from different starting points.
Remember "F2->F4 and F4->F2"? Note that these vectors are not simply the reverse of each other (unless the motion analysis is perfect and no object moves into or out of the picture).
However, it seems increasingly clear to me that my initial understanding of MAnalyse is correct: the isb vectors look to the previous frame and the next frame rather than to the next frame and from the next frame.
Actually (again as Didée explained), the vectors (for a given frame) look forward from the previous frame and back from the next frame.
But of course each vector is a clip, so the vector for current->next can be found in the next frame of the forward vector, and so on.
Now, anticipating what you are thinking, no this is NOT the equivalent of what you've posted, with delta=2 because the vector which points to the previous frame (and I really do think it points to the previous frame) is now pointing to the previous frame, and not the frame prior to the previous frame. Thus, I get a much more accurate interpolation because the isb=true vector is constructed from the immediately adjacent previous frame rather than the frame before that.
No, as I pointed out, MFlowInter only uses two frames and using delta=2 makes it use the frames immediately around the bad frame, which is the best you can get. The backward vector goes from the frame after the bad one to the one before the bad one (cf Didée's diagram).
The more I thought about it, and the more times I read the MVTools2 documentation, the more I have become convinced that your explanation of the isb delta vectors cannot be correct for MAnalyse.
My explanation may not have been very clear (Didée's is much better), but which part do you think was wrong?

EDIT: Just seen you have posted again in the interim. Perhaps more to come...

Gavino

19th June 2010, 00:20

You are correct about not needing the first "clip" in this ConditionalFilter call, and also about not needing the "(clip)" arguments in the YdifferenceToNext calls ... except ... when clip is something other than "last."
I didn't say anything about not needing 'clip' as the first argument.
I said that, whatever the first argument is, it is available as 'last' within the 'run-time' script (the bit inside the quotes). So there is no need to use 'clip' there and the global is not needed. This is independent of whether 'clip' itself is equal to 'last' in the 'outside' script.
This corrected stream is then fed to MAnalyse which then feeds its vectors to Mflowinter. By feeding the stream which now contains duplicates in place of the bad frames, the "filldrops" logic developed years ago by MugFunky now works just as it always has.
I thought we'd established that the "filldrops" logic is completely equivalent to the delta=2 approach (when you have an exact duplicate), but works in a less clear way. It seems to me you are doing twice as much work to arrive at the same result.
The advantage this approach has over what is shown in the MVTools documentation is that it avoids using a delta of 2 in the backwards direction. Using this backwards delta of 2 skips the previous frame which, being closer in time is going to provide a better starting point for the estimation vectors. Indeed, I was able to prove this by comparing the frame I got using the code shown in the MVTools2 documentation with what I got with this code.
Now that you have, I hope, a clearer understanding of how MFlowInter works, do you still believe this?
You can't get any 'closer in time' to the bad frame than the immediately surrounding frames, which is what MFlowInter (with delta=2) uses.

johnmeyer

19th June 2010, 01:23

johnmeyer said: The advantage this approach has over what is shown in the MVTools documentation is that it avoids using a delta of 2 in the backwards direction. Using this backwards delta of 2 skips the previous frame which, being closer in time is going to provide a better starting point for the estimation vectors. Indeed, I was able to prove this by comparing the frame I got using the code shown in the MVTools2 documentation with what I got with this code.

Gavino replied: Now that you have, I hope, a clearer understanding of how MFlowInter works, do you still believe this?

It is not a matter of belief; it is something that I was able to prove by doing many, many tests. There IS a difference in my approach, and it produces better frames, as it should.

As long as we both agree that Didee's explanation is correct, which I do, then I don't think we need to go further. Thank you VERY much for all your help.

Gavino

19th June 2010, 08:17

Didée's explanation is the same as mine (but clearer!), and would suggest that the delta=2 approach cannot be improved upon. Your results are therefore puzzling and invite further explanation, which I believe I have found.

By looking at the source code, I have discovered that for occlusion areas, MVFlowInter also uses information from the two further surrounding frames (current-delta and current+2*delta).
So the diagram becomes, when interpolating between C and C+d,
----> <----
C-d C C+d C+2d
----> <----
It is this additional information that will be more accurate when delta=1, explaining your findings and making your two-stage approach a better one. Well done.

Just to clarify further my remark about ConditionalFilter, what I am saying is this.
In ConditionalFilter(c1, c2, c3, "...", "...", "..."), 'last' is always set to the value of 'c1' before the expression strings are evaluated (on each frame). Therefore your code
goodframes = ConditionalFilter(clip, trim(clip,1,0), clip, \
"(YDifferenceFromPrevious(clip)/YDifferenceFromPrevious(loop(clip,2,0,0))) * \
(YDifferenceToNext(clip)/YDifferenceToNext(trim(clip,1,0)))", "greaterthan", "bad_thresh")
can be replaced by
goodframes = ConditionalFilter(c, trim(c,1,0), c, \
"(YDifferenceFromPrevious()/YDifferenceFromPrevious(loop(2,0,0))) * \
(YDifferenceToNext()/YDifferenceToNext(trim(1,0)))", "greaterthan", "bad_thresh")
and similarly for 'fixed', with the global variable being removed.
Using a global variable for this purpose in a function is in general a bad idea as the function will not work if called more than once.

johnmeyer

19th June 2010, 18:01

Therefore your code ... can be replaced by ... and similarly for 'fixed', with the global variable being removed.
Using a global variable for this purpose in a function is in general a bad idea as the function will not work if called more than once.Thanks for that tip! I don't like the idea of not being able to call the function more than once.

Interesting what you found in the source code. That confirms the behavior I have been observing.

I went ahead and modified the script to remove the global variable. This works for all ConditionalFilter calls that use "c", but I had a few debug calls (which I didn't post, and am not posting below) where I used ScriptClip to show the YDifference vectors so I could see what threshold value to use. This was done later in the script at a point where "last" was now referencing the stream where the bad frame had been replaced with a duplicate. Obviously the YDifference values for this stream no longer can identify the bad frame. So, at your suggestion, I removed the global variable for the normal working script, but I still have to use it for debugging because it IS needed for ScriptClip even though it isn't for ConditionalFilter.

Finally, I decided that my YDifference ratio approach wasn't such a good idea, because it can't differentiate between slightly bad frames and scene changes. So, I went back to the approach I've used for several years. It works great for finding single frame anomalies, but it isn't "automatic" so you have to spend a few minutes to determine the threshold value for each particular video stream.

Here's the script with these two revisions:

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

global bad_thresh = 20

source=AVISource("E:\frameserver.avi").ConvertToYV12.killaudio()
corrected=replacebad(source).AssumeFPS(29.97, false)

return corrected

function replacebad (clip c)
{
# This next statement replaces bad frame with a duplicate of the frame immediately after the bad frame
goodframes = ConditionalFilter(c, trim(c,1,0), c, \
"YDifferenceFromPrevious()>bad_thresh && YDifferenceToNext()>bad_thresh", "greaterthan", "0")

# This next statement gets the previous frame in the stream that now contains duplicates instead of bad frames
previousframe = Loop(goodframes,2,0,0)

super=MSuper(previousframe,pel=2)

vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)

replacement = mflowinter(previousframe,super,vbe,vfe,time=50)

fixed = ConditionalFilter(c, replacement, c, "YDifferenceFromPrevious()>bad_thresh && \
YDifferenceToNext()>bad_thresh", "greaterthan", "0")

return fixed
} # End function replacebad

Gavino

19th June 2010, 20:02

I removed the global variable for the normal working script, but I still have to use it for debugging because it IS needed for ScriptClip even though it isn't for ConditionalFilter.
Note that ScriptClip (indeed all 'run-time' filters) works in a similar way.

ScriptClip(clip, "
# here 'last' refers to 'clip', regardless of the value of 'last' outside
...
")

Or did you mean you needed to refer to a different clip 'inside', one not the same as (or derivable from) ScriptClip's input clip?
That's an example of what I call the 'variable binding problem' (alluded to here (http://avisynth.org/mediawiki/The_script_execution_model/Scope_and_lifetime_of_variables#Runtime_scripts)) and I developed the GRunT plugin specifically to solve this (and other usability issues with the standard run-time filters).

Incidentally, being a 'language lawyer' guy, I was suprised to see that this works in ConditionalFilter:
"YDifferenceFromPrevious()>bad_thresh && YDifferenceToNext()>bad_thresh", "greaterthan", "0"
It appears ConditionalFilter allows comparison of a boolean value with an integer, whereas the script language itself does not.
It obviously works, but it might be clearer to rewrite this as
"YDifferenceFromPrevious()>bad_thresh && YDifferenceToNext()>bad_thresh", "==", "true"

johnmeyer

19th June 2010, 22:13

Or did you mean you needed to refer to a different clip 'inside', one not the same as (or derivable from) ScriptClip's input clip?

I didn't provide, in my earlier posts, the code which causes the problem. So, here it is. In this code fragment:

global clip = c
fixed = ConditionalFilter(c, replacement.ScriptClip("""subtitle(string(YDifferenceFromPrevious(clip) ) \
+ ", " + string(YDifferenceToNext(clip) ) )"""), c, \
"YDifferenceFromPrevious()>bad_thresh && YDifferenceToNext()>bad_thresh", "greaterthan", "0")

If I do not include clip in the above, then the ScriptClip call will overlay the WRONG YDifference values onto the clip. The YDifference values used by the Conditional filter itself (i.e., the ones on the bottom line above) are the ones associated with "c," but the YDifference values in the ScriptClip call will be associated with "last" (which in the script I just posted is the video stream where the bad frames are replaced with duplicates) unless I pass the clip parameter explicitly, and to do that I must declare it as global. I didn't believe this until I experienced it myself, but it is true. In other words, if I eliminate "clip" from the first two lines so that all the YDifference calls look identical, those inside of the ScriptClip section return different results from those used by the conditional logic.

Incidentally, being a 'language lawyer' guy ... it might be clearer to rewrite this as ...

"YDifferenceFromPrevious()>bad_thresh && YDifferenceToNext()>bad_thresh", "==", "true"

Well, I accept that although, in the same vein, based on my early computer programming in the 1960s, I was taught to never compare using an equal operator unless absolutely necessary. I guess with Booleans this isn't necessary, but when comparing numbers, you can often end up with one variable equal to 4.000000 and the comparison equal to 4.0000001 due to roundoff error, and the equality will fail. So, it is that old (almost forty-year-old) habit that leads me to use the inequality operator.

Using "==" and ">" etc. is probably a good idea. I can't remember whether that was allowed in early versions of AVISynth. There must be some reason I used the non-symbolic version, but I can't remember. A lot of my code is "cut/pasted" from various places and, if it works, I don't mess with it.

Gavino

19th June 2010, 23:03

In other words, if I eliminate "clip" from the first two lines so that all the YDifference calls look identical, those inside of the ScriptClip section return different results from those used by the conditional logic.
That's right, since 'last' inside the ConditionalFilter refers to 'c', while inside the ScriptClip it refers to 'replacement'.

If you need to do something like this in the future, it can be written like this using GRunT:
fixed = ConditionalFilter(c, replacement.ScriptClip("""subtitle(string(YDifferenceFromPrevious(c) ) \
+ ", " + string(YDifferenceToNext(c) ) )""", args="c"), c, \
"YDifferenceFromPrevious()>bad_thresh && YDifferenceToNext()>bad_thresh", "greaterthan", "0")
Using "==" and ">" etc. is probably a good idea. I can't remember whether that was allowed in early versions of AVISynth. There must be some reason I used the non-symbolic version, but I can't remember.
It wasn't the use of "greaterthan" instead of ">" that struck me, it was the idea of comparing a boolean (true/false) with an integer (0), something the script language itself does not allow.
(Nor, if I recall, even did Fortran IV - yeah, I started back in the punched card era too :))
In fact, GRunT's ConditionalFilter make this simpler too, by allowing the three separate strings to be replaced by a single boolean expression.
ConditionalFilter(..., "YDifferenceFromPrevious()>bad_thresh && YDifferenceToNext()>bad_thresh")
Thanks for your replacebad function.

Dogway

13th July 2010, 09:52

A 4-years-old script, with some minor adjustments of today.

Didée: Is that aimed to interpolation between frames with higher differences? How can I do the opposite ((double)interpolate everything except where the differences are greater)

Didée

13th July 2010, 10:00

Didée: Is that aimed to interpolation between frames with higher differences?
Of course. When 29.97 has been false-decimated to 23.976, you need to insert frames where the frame-difference is the biggest.

How can I do the opposite ((double)interpolate everything except where the differences are greater)
Technically, that's easy: just invert the mask. (Precisely: in the "ismax = mt_lutxy(...)" line, replace "x y < 0 255 ?" with "x y < 255 0 ?". That's all, except that the var then should be rather named "isnotmax", but that's cosmetics.)

However, it is totally beyond me what that should be good for ... ?

Jenyok

28th March 2013, 14:24

A 4-years-old script, with some minor adjustments of today.

# mt_masktools-25.dll
# mvtools2.dll
# tivtc.dll

Source( "VIDEO_that_has_been_decimated_to_FILM" )

o = assumefps(1.0) ox=o.width() oy=o.height()

showdot = true # false # shows a "dot" in the interpolated frames of the result

super = showdot ? o.subtitle(".").MSuper(pel=2) : o.MSuper(pel=2)
bvec = MAnalyse(super, overlap=4, isb = true, search=4, dct=5)
fvec = MAnalyse(super, overlap=4, isb = false, search=4, dct=5)
double = o.MFlowFps(super, bvec, fvec, num=2, den=1, blend=false)

diff2next = mt_makediff(o,o.selectevery(1,1)).mt_lut("x 128 - abs 32 / 1 2.0 / ^ 128 *",U=-128,V=-128)
diff2next = mt_lutf(diff2next,diff2next,yexpr="x",mode="average").pointresize(32,32)
diff2next = interleave(diff2next.selectevery(4,0).tsg(2),diff2next.selectevery(4,1).tsg(2),
\ diff2next.selectevery(4,2).tsg(2),diff2next.selectevery(4,3).tsg(2))

max = diff2next.mt_logic(diff2next.selectevery(1,-3),"max")
\ .mt_logic(diff2next.selectevery(1,-2),"max")
\ .mt_logic(diff2next.selectevery(1,-1),"max")
\ .mt_logic(diff2next.selectevery(1, 1),"max")
\ .mt_logic(diff2next.selectevery(1, 2),"max")
\ .mt_logic(diff2next.selectevery(1, 3),"max")
ismax = mt_lutxy(diff2next,max,"x y < 0 255 ?",U=-128,V=-128).pointresize(ox,oy)
themask = interleave(o.mt_lut("0"),ismax)
interleave(o,o).mt_merge(double,themask,luma=true,U=3,V=3)

tdecimate(mode=1,cycleR=3,cycle=8)
assumefps(30000,1001)

return( last )

#===========================

function tsg(clip c, int t) { c
t<5?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<4?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<3?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<2?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25)
t<1?last:last.temporalsoften(1,255,0,255,2).merge(last,0.25) }

Worx.
.
.
.
This is a idea of Didee, see script code upper.
.
On this idea of Didee is besed new function
function DoubleFPS(clip clp, int "numerator", int "denominator", int "mode", bool "mopro", int "bias", bool "show")
.

function DoubleFPS(clip clp, int "numerator", int "denominator", \
int "mode", bool "mopro", int "bias", bool "show")
{
# Defaults
#
numerator = Default(numerator, 2)
denominator = Default(denominator, 1)
mode = Default(mode, 2)
mopro = Default(mopro, false)
bias = String(Default(bias, 128))
showdot = Default(show, false)

# Here is code of function...
#
v = addborders(clp, 16, 16, 16, 16)
o = v.assumefps(FramerateNumerator(clp))
ox = v.width()
oy = v.height()

S = (showdot == true) ? v.subtitle(".", x=32, y=24).MSuper(pel=4, hpad=16, vpad=16, rfilter=4) :
\ v.MSuper(pel=4, hpad=16, vpad=16, rfilter=4)

A = S.MAnalyse(isb=true, blksize=8, overlap=2, plevel=0, search=5, searchparam=8, badrange=(-24))
B = S.MAnalyse(isb=false, blksize=8, overlap=2, plevel=0, search=5, searchparam=8, badrange=(-24))

C = S.MRecalculate(A, blksize=8, search=5, searchparam=2)
D = S.MRecalculate(B, blksize=8, search=5, searchparam=2)
E = S.MRecalculate(C, blksize=4, search=5, searchparam=1)
F = S.MRecalculate(D, blksize=4, search=5, searchparam=1)

double = (mode == 1) ? v.MBlockFps(S, E, F,
\ num=FramerateNumerator(v) * numerator,
\ den=FramerateDenominator(v) * denominator,
\ mode=0, blend=false, ThSCD1=220)
\ : (mode == 2) ? v.MFlowFps(S, E, F,
\ num=FramerateNumerator(v) * numerator,
\ den=FramerateDenominator(v) * denominator,
\ mask=2, ml=10000, blend=false, ThSCD1=220)
\ : NOP()

diff2next = mt_makediff(o, o.selectevery(1, 1)).mt_lut("x "+bias+" - abs 32 / 1 2.0 / ^ 128 *", U=-128, V=-128)
diff2next = mt_lutf(diff2next, diff2next, yexpr="x", mode="average").pointresize(32, 32)
diff2next = interleave(diff2next.selectevery(4, 0).tsg(2), diff2next.selectevery(4, 1).tsg(2),
\ diff2next.selectevery(4, 2).tsg(2), diff2next.selectevery(4, 3).tsg(2))

max = diff2next.mt_logic(diff2next.selectevery(1, -3), "max")
\ .mt_logic(diff2next.selectevery(1, -2), "max")
\ .mt_logic(diff2next.selectevery(1, -1), "max")
\ .mt_logic(diff2next.selectevery(1, 1), "max")
\ .mt_logic(diff2next.selectevery(1, 2), "max")
\ .mt_logic(diff2next.selectevery(1, 3), "max")

ismax = mt_lutxy(diff2next, max, "x y < 255 0 ?", U=-128, V=-128).pointresize(ox, oy) # swap 255 0 to interpolate only on big changes.
themask = interleave(o.mt_lut("0"), ismax)
interleave(o, o).mt_merge(double, themask, luma=true, U=3, V=3)

(mopro == true) ? last.crop(16, 16, -16, -16) : double.crop(16, 16, -16, -16)

return (last)
}

function tsg(clip clp, int t)
{
last = clp

(t < 5) ? last : last.temporalsoften(1, 255, 0, 255, 2).merge(last, 0.25)
(t < 4) ? last : last.temporalsoften(1, 255, 0, 255, 2).merge(last, 0.25)
(t < 3) ? last : last.temporalsoften(1, 255, 0, 255, 2).merge(last, 0.25)
(t < 2) ? last : last.temporalsoften(1, 255, 0, 255, 2).merge(last, 0.25)
(t < 1) ? last : last.temporalsoften(1, 255, 0, 255, 2).merge(last, 0.25)

return (last)
}

.
Question to Didee.
.
How to correct this script function to do various frames conversion (not only doubling, multiply by various numerator and denominator) ?
when "mopro = true" .
.
Thanks.
.