Converting script from mvtools to mvtools2

[Floatingshed]

I regularly use this (nicked and cobbled together) script for removing film dirt:



_grey=false
limit = default(limit,60)
clap = last

bvec = clap.MVAnalyse(isb=false, blksize=8, delta=1, pel=2, truemotion=true, idx=4)
fvec = clap.MVAnalyse(isb=true, blksize=8, delta=1, pel=2, truemotion=true, idx=4)
backw = clap.MVFlow(bvec)
forw = clap.MVFlow(fvec)
clt = interleave(backw,clap,forw)
clt = clt.RemoveDirt(limit)

clt = clt.SelectEvery(3,1)



function RemoveDirt(clip input, int limit)
{
clensed=input.Clense(grey=false, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=false)
}

I'm finding that it is unstable with avisynth 2.6mt so I would like to update it to use mvtools2 in the hope of making it more reliable. Its all a bit above me, can anyone help please?

[feisty2]

sup = clap.msuper (pel=2)
bvec = sup.MAnalyse(isb=false, blksize=8, delta=1, pel=2, truemotion=true)
fvec = sup.MAnalyse(isb=true, blksize=8, delta=1, pel=2, truemotion=true)
backw = clap.MVFlow(sup,bvec)
forw = clap.MVFlow(sup,fvec)

[Floatingshed]

Thanks, but:

Invalid arguments to function "msuper"

[feisty2]

good now?

[johnmeyer]

Check out my post here:

RemoveDirtMC with MVTools2 (http://forum.doom9.org/showpost.php?p=1595608&postcount=17)

Scroll down to the bottom of the page to see the RemoveDirtMC function.

[Floatingshed]

Thanks John, that was very helpful. Looks likely that my original script was yours!

[StainlessS]

Seems to have forward and Backwards vectors back-to-front.
There was a big discussion about it somewhere (PBristow was involved I think).

bvec = sup.MAnalyse(isb=false, blksize=8, delta=1, pel=2, truemotion=true)


Above is forward vector (isb=false), ie prior frame shuffled about to look like current frame, opposite for the other one.

But, as also back-to-front here, does not matter, everything comes out OK in the wash

clt = interleave(backw,clap,forw)


Also, order does not actually matter in that script as only dealt with in 3 frame groups [SelectEvery(3,1)], but might if using same stuff for something else.

[johnmeyer]

Thanks John, that was very helpful. Looks likely that my original script was yours!I don't think so. I linked to my MVTools2 script, not the original MVTools. Thus, it already has all the conversions done. If you "nick" it directly, it should work, as is.

[johnmeyer]

Seems to have forward and Backwards vectors back-to-front ... Those code fragments don't look like they came from the script I linked to. Here is the code from that post:

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int limit, bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

As you can see, for bvec (backwards vector,) isb=true, where isb = "is backwards".

[StainlessS]

I see zero problems there JM.

[Floatingshed]

Thanks everyone, I really don't understand this very well but its doing exactly what I need.

[StainlessS]

I see zero problems there JM.

Missed a little anomaly [EDIT: two anomalies]

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

Of the interleaved 3 frame group, compensated frame derived from Next source frame is first in group(backw), then current frame, then compensated frame derived from Previous frame(forw), so if source Curr->Next is a scene change then will appear as 1st in group, and if Prev->Curr is scene change then will appear in 3rd of group. Not really a problem as handled in groups of 3 and then SelectEvery(3,1) throws the compensated frames away as only middle one matters, but if viewing interleaved clip (without SelectEvery) scene changes will appear in wrong order.
Have not checked this but I think is how it will pan out.
Dont know if this out of order grouping is done deliberately for some reason, but I would find it confusing at least.

EDIT: Is there some RemoveDirt() requirement for out-of-order grouping ?

EDIT: Checked for out-of-order and does as suggested above. Also Need to put quotes around limit in function definition, ie


Function RemoveDirtMC(clip,int "limit", bool "_grey") {


As Limit is supposed to be optional.

Comment out these two lines in script function to test out-of-order

#clp=clp.RemoveDirt(limit,_grey)
#clp=clp.SelectEvery(3,1)


And call with


Avisource("D:\avs\StarWars.avi")
RemoveDirtMC() # fail if Function arg Limit not quote enclosed. [needs RemoveDirtMC(6)]
return last


EDIT:
EDIT: Is there some RemoveDirt() requirement for out-of-order grouping ?

A number of examples have this temporal out-of-order grouping, including mvtools examples, why ?

EDIT: The order of below clips in blue makes zero difference to the output in MDegrain1, MDegrain2, MDegrain3,
but would seem to make more sense if, forward 3, forward 2, forward 1, backward 1, backward 2, backward 3 (for MDegrain3).

Function MCDegrain(clip c, int "frames")
{ # By Didee, http://forum.doom9.org/showthread.php?p=1508289#post1508289
frames = default(frames, 2)
bs = (c.width>960) ? 16 : 8
super = c.MSuper(pel=2, sharp=1)
backward_vec3 = MAnalyse(super, isb = true, delta = 3, blksize=bs, overlap=bs/2)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=bs, overlap=bs/2)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=bs, overlap=bs/2)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=bs, overlap=bs/2)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=bs, overlap=bs/2)
forward_vec3 = MAnalyse(super, isb = false, delta = 3, blksize=bs, overlap=bs/2)
(frames<=0) ? c :\
(frames==1) ? c.MDegrain1(super, backward_vec1,forward_vec1,thSAD=400) :\
(frames==2) ? c.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) :\
c.MDegrain3(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400)
return(last)
}

[johnmeyer]

StainlessS,

Thanks for catching those errors. The whole "isb" and "forward/backward matching" is extremely confusing. The author even apologizes for it in the documentation.

The source for the confusion (not only MY confusion, but apparently others, based on your finding similar errors elsewhere), can be found in this one sentence from the documentation:

isb : allows to choose between a forward search (motion from the previous frame to current one) ...

The problem I have -- and I think is shared by others -- is that I tend to think of the current frame as the reference point for everything, and therefore vectors from "the previous frame to the current one" seems like it should be a backward vector, not a forward one.

However, it is what it is, and I need to read the manual more carefully and more often.

I've made the corrections, and am now doing a scan of my other 100+ scripts to make sure the same error doesn't occur elsewhere.

Once again, thank you.

[johnmeyer]

BTW, I just did some searching of this forum (and others), and this error has been around a LONG time, and I'm not sure I was the one who was the original source of the error. It is shown correctly in the two MVTools2 examples given in the MVTools2 documentation, but I have found scripts in this forum from as far back as 2009 which have the wrong order.

[edit]The error appears in a LOT of other scripts, not just those that do motion compensated dirt removal. Yikes!

[edit2]Ah, the problem originated back in 2006, with the original author:

ein einfaches RemoveDirtMC (http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CCYQFjAB&url=http%3A%2F%2Fforum.gleitz.info%2Farchive%2Findex.php%2Ft-26963.html&ei=uhGcVZf7MIPhoATy3ZGYDA&usg=AFQjCNGiOUN117XySo5oFsKFnocK5O9y9g&sig2=a54FEySIDpUA2XW8wvO8kQ&bvm=bv.96952980,d.cGU&cad=rja)

[StainlessS]

To make things worse, yesterday I found a number of RemoveDirt() and RemoveDirtMC() functions
with differing args and even in differing order, arrrrrh !!!

[johnmeyer]

I went through over a hundred scripts today, many of them simply copied from other people's posts in this forum, and had to correct a lot of them. I'm going to take a second pass through them tomorrow.

I think I need to go back to the various threads where I've posted about RemoveDirtMC and correct those posts.

[Reel.Deel]

Another small thing I noticed is that when when using a prefiltered super clip, the unfiltered super usually gets levels=1, like so:
...
prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2,levels=1)
...

Not really sure if it makes any difference but I've seen it used this way (http://forum.doom9.org/showpost.php?p=1584689&postcount=4&highlight=levels) many times.

Another thing that I'm curious about, why is hpad and vpad set to 32? If the input clip has black borders then all is doing is padding the black borders with even more black borders, which is unnecessary in this case.

[johnmeyer]

Another thing that I'm curious about, why is hpad and vpad set to 32? If the input clip has black borders then all is doing is padding the black borders with even more black borders, which is unnecessary in this case.You might ask "poisondeathray" because I think he did the original conversion from MVTools to MVTools2, and this is when those variables got added. Here's the post:

RemoveDirtMC_NEW (http://forum.doom9.org/showthread.php?p=1639554#post1639554)

You will see that he included both the original MVTools version that was created back in 2006, and then the new version that used the new syntax.

He is still a regular member of the forum, so perhaps he can remember why he added the padding.

[StainlessS]

Another small thing I noticed is that when when using a prefiltered super clip, the unfiltered super usually gets levels=1, like so:


From Docs (MSuper)
levels

It is the number of hierarchical levels in super clip frames. MAnalyse is need in all levels, but for other client functions single finest level is enough (coarser levels are not used). 0 = auto, all possible levels are produced.


MAnalyse needs ALL levels (to make vectors), other client functions only require a single level, so, if not using prefiltered clip need all levels in MSuper() and so client functions also get all levels from that single super clip, but they only use the finest (highest rez) level.
If Prefiltered, then need all levels for prefiltered Super clip for MAnalyse, but only need single (highest rez) level for the Super clip used by client functions eg MFlow or MCompensate.
Single level for the standard Super clip when using prefiltered is probably somewhat faster than all levels, but in that case, the
Prefiltered Super is the one that takes most CPU. Always use Levels=1 for the Standard Super when using prefiltered and
when not using Prefiltered always default all levels (I think MAnalyse complains if it dont get all levels).

Hope that makes more sense. (I only recently got that sorted out in my head).

[johnmeyer]

A number of examples have this temporal out-of-order grouping, including mvtools examples, why ?

EDIT: The order of below clips in blue makes zero difference to the output in MDegrain1, MDegrain2, MDegrain3,
but would seem to make more sense if, forward 3, forward 2, forward 1, backward 1, backward 2, backward 3 (for MDegrain3) ...I continue to look into this, and I'm pretty certain that with MVTools2, if you incorrectly specify the vector order, MVTools2 will throw an error ("incorrect vector .." or something like that).

However, if you look at 99% of the MVTools2 scripts posted (I just looked at several dozen), almost none of them explicitly specify the vectors, e.g.:

mvflowfps(mvbw=vb,mvfw=vf, ...)Instead, that line is usually written (99% of the time) like this:

mvflowfps(vb,vf, ...)In this case, I think that MVTools2 gets the vector direction from the vector itself, and not from the order in which they are listed in the function call.

Thus, as long as you are working within the MVTools2 function calls, the "wrong" order won't get you into trouble. However, when passing the results to a function outside MVTools2 (like the Interleave statement you spotted), one can create some real problems.