MaskTools 2.0a48

[foxyshadis]

Actually, Deblock_QED and some one-off sharpener testing visualization script seem to be the only scripts on the forum using it without expr. As long as Deblock_QED is updated, it shouldn't be a major problem - and the error message should make it clear when it does happen that it's in Deblock_QED.

[Gavino]

Actually, Deblock_QED and some one-off sharpener testing visualization script seem to be the only scripts on the forum using it without expr.
Good to know, although the forum isn't the exclusive source for scripts. Some people actually write their own. ;)

[Manao]

New version (http://manao4.free.fr/masktools-v2.0a44.zip) available, with a new mode added to mt_luts / mt_lutsx. It will allow very flexible convolution with custom kernels.

Gavino : for mt_lutspa, the "evil" is done, I won't change it back. I'll try to keep backward compatibility in mind for later releases, but if it really matter to you, you can also take care of that on your side, by using named parameters (which I always do, thus why I didn't think adding a parameter in the middle of the parameter list would be a problem).

[jpsdr]

I'm agree with Gavino, and i'm thinking of people who use scripts made by others, and may don't know how to debug/correct it, or even how to contact the author of the script...

[hydra3333]

yes, please reconsider. just this once, maybe.

[kemuri-_9]

I'm in agreement with Manao, he's already made the change and so it is already propagating.
to change it would cause a re-propagation of changes.

besides if people seriously don't know how to fix the script they're using, then they should take that opportunity to actually learn about what they're doing.

[Gavino]

besides if people seriously don't know how to fix the script they're using, then they should take that opportunity to actually learn about what they're doing.
The issue is not whether people know how to fix their scripts, it's the basic software engineering principle that a new version of something should not impose unnecessary changes on existing users. IanB is very careful to adhere to this principle for Avisynth itself.

However, the damage is slight in this case and has already been done. It was probably a design 'mistake' in the first place to make 'relative' the first parameter of mt_lutspa, rather than the more frequently used 'expr'. But this is just nit-picking on what is ultimately a great piece of software.

[Manao]

Not really a mistake. Both "expr" and "relative" have to be provided. You can assume "relative" to have its default value, but it's a bad habit imho in this case.

Also, the logic behind the original positioning was that "expr" and "relative" should be the first two arguments (because they would be those used the most), and that "expr", "yExpr", "uExpr" and "vExpr" should be consecutive arguments (as they were in all the other filters using luts). Once those two predicates were in action, I didn't have a lot of choice in regard to their relative positioning. When I had to add "biased", once again, I decided it should be one of the first three, and that it shouldn't break the "expr" serie.

[Guest]

People that do not use named parameters deserve what they get, IMHO.

[hydra3333]

Oh ? They do ?

[Didée]

Well, it was me who "too quickly" fiddled that lutspa into that certain deblocking function ... Guilty! Blame me! Guilty!

(Will I be condemned to prison, or is there hope to get away with suspended sentence?)

A corrected script has been posted already ... but as usual, the problem is that once a malformed script has been spread in the wild, you can't make it disappear anymore.

[Bi11]

My 2 cents is that consistent relative positioning of the parameters is probably more important in this case (especially since mt_LutSpa is but one function in a much larger piece of software), as Manao pointed out.

I think most "ignorant" users (like me) get their scripts from the AviSynth Wiki. Hence the Wiki should be updated with the corrected version of broken scripts.
Edit: I see the AviSynth Wiki has already been updated with the corrected version of Deblock_QED. I would strongly advise those in charge of editing the Wiki to please link back to the source (post) of those scripts (just like a Wikipedia reference) for the benefit of everyone.

[IanB]

Manao, the new biased parameter of mt_lutspa has been introduced between the parameters relative and expr.
This causes problems for existing scripts which have called mt_lutspa with positional parameters (eg see here).

Rather than have everyone affected update their scripts/functions, it would be better to make biased the last parameter in the list. It is generally good practice to do this when introducing new options, to preserve backwards compatibility.Gavino : for mt_lutspa, the "evil" is done, I won't change it back. I'll try to keep backward compatibility in mind for later releases, but if it really matter to you, you can also take care of that on your side, by using named parameters (which I always do, thus why I didn't think adding a parameter in the middle of the parameter list would be a problem).Manao,

I must say I am very disappointed with your "evil is done, I won't change it back" attitude. You could have just said oops, my bad, here is a fixed v2.0a44a version and everybody would have been satisfied.

There are places in Avisynth where you do cannot use argument naming, Animate/ApplyRange being one.

You write excellent software, I hope you put away your black hat and start wearing your white hat again, please reconsider.

[Manao]

IanB: I didn't think of animate/applyrange, well spotted. But, luckily for me, since mt_lutspa's arguments are only bools and strings, it doesn't really apply.

As for my "evil" is done attitude (I should have said the genie is out of the box...), I would probably have made the change if the "problem" had been spotted right after 2.0a43, three weeks ago. Instead, something like that (http://forum.doom9.org/showthread.php?p=1402777#post1402777) happened in the meantime. Basically, making the change now would imho mess things up even more.

[Gavino]

IanB: I didn't think of animate/applyrange, well spotted. But, luckily for me, since mt_lutspa's arguments are only bools and strings, it doesn't really apply.
ApplyRange is not limited to numeric parameters, although in truth it's unlikely one would use it with mt_lutspa.

I don't think it's too late to make a change. The Deblock_QED fix uses named parameters, so will not be affected.

[jpsdr]

besides if people seriously don't know how to fix the script they're using, then they should take that opportunity to actually learn about what they're doing.

I don't agree. Why should the user of something (script or software) has to be able to fix it ?
A lot of people used a lot of video software, filters, plugins, scripts, they know what they are doing in video edition, but they don't have programming knowledge, or high video processing knowledge to do what they wanted. And they don't have to.

[Bi11]

You could have just said oops, my bad, here is a fixed v2.0a44a version and everybody would have been satisfied.
But what about the reasoning given for the positioning of the parameter in the first place.

I would probably have made the change if the "problem" had been spotted right after 2.0a43, three weeks ago.
It may have been spotted right after 2.0a43, but those who spotted it may have been too "dumb" to figure out the exact cause of the problem, or they may not be Doom9 members, or they didn't known how to submit a bug report, or didn't have time, or waited for someone else to post about it, or ...

Instead, something like that (http://forum.doom9.org/showthread.php?p=1402777#post1402777) happened in the meantime.
I can always update my post to state that the old script will continue to work with (the fixed) future versions of MaskTools 2.0.

Basically, making the change now would imho mess things up even more.
If you can live with slight internal inconsistency in parameter positioning then I see no reason not to apply the fix.

Scripts that worked before and after the change will continue to work after the fix. Scripts that didn't work after the change must be corrected, unless the fix is applied in which case they will continue to work as if the change never happened. Only scripts without named parameters that used the "biased" parameter after the change will no longer work after the fix, but those scripts would be an insignificant minority compared to scripts written before the change.

Thus, preserving backward compatibility is a win-win for everybody (except you, maybe :p).

Edit:
I just realized that the new "biased" parameter is a hack for the vagueness of "relative". The argument of consistency cannot be applied to hacks! Therefore, it is a software development error on your part, Manao, to break backward compatibility for a hack (esp. since it's usually done the other way around) in an otherwise flawless piece of software.
So that just leaves either keeping the hack but making sure to preserve backward compatibility, or fixing the hack properly (perhaps by changing bool relative to string mode, with values "absolute", "relative inclusive" and "relative exclusive"), but unavoidably breaking backward compatibility.

[Bi11]

You can probably get the best of all worlds by declaring the function as:

mt_lutspa : clip clip, bool relative("true"), string expr("x"), string yexpr("x"), string uexpr("x"), string vexpr("x"), string mode("relative exclusive"), bool biased("true")

Of course, you would have to declare "relative" and "biased" as being deprecated in favour of "mode" in the documentation.

Hence, the hack now serves the purpose of preserving backward compatibility, as it should be. :cool:

[JEEB]

Excuse me as a MaskTools beginner, but I happened to come upon something that begs me to question if this is intended behavior:


YourSourceFilter()

mask = last.mt_edge("cartoon",4,32,4,32)
mask

The (http://img27.imageshack.us/img27/8835/weirdcolors2.png) output (http://img27.imageshack.us/img27/4463/weirdcolors.png). What I would expect (http://img16.imageshack.us/img16/4586/13198mask.png).

This piece of script I got by combining rather random knowledge out of another script and my own scribbles, and used it here (http://forum.doom9.org/showthread.php?t=148395). Now, since the powerful white is there, and most of the other colors are rather slight, the actual script has worked. It's just that the other day I got asked about MaskTools2 usage on IRC, and got asked if random colors etc. were normal to have in the mask itself. So, as I couldn't give a proper answer to that question, here I am.

Tested on revisions 29, 34, 37, 39, 43 and 44 -- the output is quite identical. 32bit Avisynth 2.5.8 stable, no MT used. Source filter and resolution don't seem to matter, I have so far gotten identical results on several types of sources and several resolutions. Of course, if a source will be needed, I will be glad to provide.

[foxyshadis]

You forgot chroma="process" (or "copy" or -128). You can also follow it up with a .Greyscale() call.

Same thing tripped me and most others up the first time. :D

[JEEB]

Thank you, that indeed did help. I guess this is A) indeed intended behavior, and B) what I get from not RTFM'ing enough.

[foxyshadis]

I'd like to know whether I'm correct in believing that mt_makediff is supposed to be commutative, so that this will produce the original input:

s=Source()
dn=s.Denoise()
df=mt_makediff(s,dn,chroma="process")
rn=mt_makediff(dn,df,chroma="process")
s+rn+df

However, when I do this, s is very different from rn. It might be saturating instead of wrapping, but the change is never more than +-20. Is this intended?

[Gavino]

mt_makediff is not commutative - what makes you think it should be? (Subtraction is not a commutative operation.)

Loosely speaking, df = s-dn+128 and rn = dn-df+128, ie dn+dn-s.
To reconstitute s, use mt_adddiff(df, dn).
(Or mt_adddiff(dn, df), since mt_adddiff is commutative.)

[foxyshadis]

Ah, once I think through the math, it's obvious. Here I've been using mt_makediff as an equivalent to xor for ages now, because it looks close enough. That's my fault for not ever actually thinking all the way through it - I probably munged a few projects that way. Thanks.

Now of course there's mt_logic with a real xor.

[Didée]

Now of course there's mt_logic with a real xor.
Depending on what exactly you want to to. All the "AND", "OR" / "XOR" of mt_logic are only well-defined for clips' values of [0] and [255]. For values [1,254], the behaviour isn't well-defined. After all, those are binary operators.

For general-case usage (i.e: difference clips), the equivalent of "OR" is "max", the equivalent of "AND" is "min". There is no equivalent for "XOR", since that doesn't make sense if the argument is a field/range of possible values.

(Slight possibility that Manao was aware and made AND/OR also working that way, I never actually checked. But per documentation, IIRC, AND/OR are really laid out for binary logic.)

[Gavino]

All the "AND", "OR" / "XOR" of mt_logic are only well-defined for clips' values of [0] and [255]. For values [1,254], the behaviour isn't well-defined.
The documentation suggests they work bit-wise for all values [0,255].
It even uses pixel values of 11 and 5 as an example, eg for xor, 11 ^ 5 = 1011 ^ 101 = 1110 = 14.

Edit: I thought this conversation seemed familiar, and after some searching I found this (http://forum.doom9.org/showthread.php?p=1256289#post1256289). ;)

[Manao]

*Bump* (http://manao4.free.fr/masktools-v2.0a45.zip)

In store, a fix to a problem with mt_clamp and SSE2, pointed out by tp7 on IRC. I also implemented Bi11 proposal to mt_lutspa conundrum. There's now a "mode" parameter, which takes the positional place of the "relative" parameter, which is in turn, together with the "biased" parameter, moved further away and shouldn't be used anymore. Consequently, backward compatibility with named parameters is preserved, but is (yet again) broken with positional parameters.

[TurboPascal7]

Looks like my first post here. ^^
Since I work a lot with HD video, I have a realy stupid question about scripts optimization.
Is there any possibility to replace mt_**pand with predefined custom modes (i.e. mt_square) with multiple calls of mt_**pand internally in masktools?
For example, in gradfun2dbmod, one can replace mt_luts(input,input,mode="range",pixels=mt_square(radius),expr="y",u=1,v=1) with something like mt_lutxy(input.mt_expand(mode=mt_square(radius)),input.mt_inpand(mode=mt_square(radius)),"x y -",u=1,v=1) (gives ~same speed as original), wich is equal to mt_lutxy(input.mt_expand().mt_expand(),input.mt_inpand().mt_inpand(),"x y -",u=1,v=1) (radius=2, about 10 times faster).
Works good, but without *for* statement totaly unoptimal and nonuniversal.
It's still possible to optimize script for some often usable modes, but it looks too silly for me.
GFmask = radius==1 ? input.mt_edge(mode="min/max",thY1=0,thY2=255,u=1,v=1)
\ : radius==2? mt_lutxy(input.mt_expand().mt_expand(),input.mt_inpand().mt_inpand(),"x y -",u=1,v=1)
\ : radius==3? mt_lutxy(input.mt_expand().mt_expand().mt_expand(),input.mt_inpand().mt_inpand().mt_inpand(),"x y -",u=1,v=1)
\ : radius==4? mt_lutxy(input.mt_expand().mt_expand().mt_expand().mt_expand(),input.mt_inpand().mt_inpand().mt_inpand().mt_inpand(),"x y -",u=1,v=1)
\ : radius==5? mt_lutxy(input.mt_expand().mt_expand().mt_expand().mt_expand().mt_expand(),input.mt_inpand().mt_inpand().mt_inpand().mt_inpand().mt_inpand(),"x y -",u=1,v=1)
\ : mt_luts(input,input,mode="range",pixels=mt_square(radius),expr="y",u=1,v=1)
P.S. My english isn't so good as I want it to be, sorry about that.

[Manao]

There are no "for" statements, but you can still do recursive functions :function mt_inpand_rec(clip c, int radius)
{
return radius <= 0 ? c : radius <= 1 ? c.mt_inpand : mt_inpand_rec(c.mt_inpand, radius - 1)
}Also, mode="range" is equivalent to "x y - abs", not "x y -".

[TurboPascal7]

Oh, I haven't thought about recursion, my fault.
Also, mode="range" is equivalent to "x y - abs", not "x y -".
is there any case when maximum value can be smaller then minimal value and x y - gives negative result?

[Manao]

is there any case when maximum value can be smaller then minimal value and x y - gives negative result? Not really, unless it's very early in the morning and you just woke up :p

[cretindesalpes]

It made me think about two helper functions I wrote for this kind of task, and I thought that I could share them:

#=============================================================================
# mt_expand_multi
# mt_inpand_multi
#
# Calls mt_expand or mt_inpand multiple times in order to grow or shrink
# the mask from the desired width and height.
#
# Parameters:
# - sw : Growing/shrinking shape width. 0 is allowed. Default: 1
# - sh : Growing/shrinking shape height. 0 is allowed. Default: 1
# - mode : "rectangle" (default), "ellipse" or "losange". Replaces the
# mt_xxpand mode. Ellipses are actually combinations of
# rectangles and losanges and look more like octogons.
# Losanges are truncated (not scaled) when sw and sh are not
# equal.
# Other parameters are the same as mt_xxpand.
#=============================================================================

Function mt_expand_multi (clip src, int "thY", int "thC", string "mode",
\ int "offx", int "offy", int "w", int "h", int "y", int "u", int "v",
\ string "chroma", int "sw", int "sh")
{
sw = Default (sw, 1)
sh = Default (sh, 1)
mode = Default (mode, "rectangle")

mode_m =
\ (sw > 0 && sh > 0) ? (
\ (mode == "losange" || (mode == "ellipse" && (sw % 3) != 1))
\ ? "both" : "square"
\ )
\ : (sw > 0 ) ? "horizontal"
\ : ( sh > 0) ? "vertical"
\ : ""

(mode_m != "") ? src.mt_expand (
\ thY=thY, thC=thC, mode=mode_m,
\ offx=offx, offy=offy, w=w, h=h, y=y, u=u, v=v, chroma=chroma
\ ).mt_expand_multi (
\ thY=thY, thC=thC, mode=mode,
\ offx=offx, offy=offy, w=w, h=h, y=y, u=u, v=v, chroma=chroma,
\ sw=sw-1, sh=sh-1
\ ) : src
}

Function mt_inpand_multi (clip src, int "thY", int "thC", string "mode",
\ int "offx", int "offy", int "w", int "h", int "y", int "u", int "v",
\ string "chroma", int "sw", int "sh")
{
sw = Default (sw, 1)
sh = Default (sh, 1)
mode = Default (mode, "rectangle")

mode_m =
\ (sw > 0 && sh > 0) ? (
\ (mode == "losange" || (mode == "ellipse" && (sw % 3) != 1))
\ ? "both" : "square"
\ )
\ : (sw > 0 ) ? "horizontal"
\ : ( sh > 0) ? "vertical"
\ : ""

(mode_m != "") ? src.mt_inpand (
\ thY=thY, thC=thC, mode=mode_m,
\ offx=offx, offy=offy, w=w, h=h, y=y, u=u, v=v, chroma=chroma
\ ).mt_inpand_multi (
\ thY=thY, thC=thC, mode=mode,
\ offx=offx, offy=offy, w=w, h=h, y=y, u=u, v=v, chroma=chroma,
\ sw=sw-1, sh=sh-1
\ ) : src
}


Though I don't know if the "common" parameters are in the right order (I'm too lazy to check the mt source code).

[yup]

Hi all!
I want organize following calculation:
source data
prefiltered prefiltered clip
3 masks center, backward and forward,
3 filtered clip centerfiltered, backwardfiltered and forwardfiltered
output=prefiltered
if (center<backward)&(center<forward) then output=centerfiltered
if (backward<center)&(backward<forward) then output=backwardfiltered
if (forward<bacward)&(forward<center) then output=forwardfiltered
Can this make using masktools?
Please advice.
yup.

[Didée]

Sure, that can be done. But sorry, the description of the chain is not clear enough.

- you create "3 masks center, backward and forward", but don't explain what is done with those masks

- how exactly are centerfiltered, backwardfilterd and forwardfiltered created? (Probably with the three masks from above...)

- what means "(center<backward)" etc.? Do you mean "pixel value is smaller"? Or rather "difference to prefiltered", or "difference to source", or ...

For "pixel value is smaller", you can use mt_logic(mode="min"). If it's really the difference-to-something, then you can use the Corrector() plugin.

[yup]

@Didée:thanks:
It is very long story, hope you remember:)
http://forum.doom9.org/showthread.php?t=121197
3 masks center, backward and forward is min SAD value from both predictors (for center is backward1 and forward1, for backward backward1 and backward2, for forward forward1 and forward2 )
centerfiltered is median from 3 value center, backward and forward predictors.
backwardfiltered is median from 3 value center, backward and backward2 predictors.
forwardfiltered in a similar.
center<backward mean less by value and this try find better motion predictor from 3 variants.
yup.

[yup]

I think need use cascade mt_lutxyz

if (center<backward)&(center<forward) then output=centerfiltered

masktools equivalent

mt_lutxyz(center,backward,forward,'x y < x z < &')
output=mt_lutxy(last,centerfiltered,output, 'x y z?')

Please advice.
yup.

[Manao]

Yet another bugfix (http://manao4.free.fr/masktools-v2.0a46.zip), which concerns mt_lutf and "std" mode. Thanks again to TurboPascal7 for pointing out the bug.

[kypec]

Yet another bugfix (http://manao4.free.fr/masktools-v2.0a46.zip), which concerns mt_lutf and "std" mode. Thanks again to TurboPascal7 for pointing out the bug.
The download link in your first page still points to v2.0a45, please update it as well, thanks.

[ChrisBeringB]

How are the luts organized?
I'm asking because I'm worried about CPU cache locality.

Considering 'mask' is 99% zeros, which is better, speedwise:
a: MT_Lutxyz(mask, frame1, frame2, yexpr="some expression f(x,y,z)")
b: MT_Lutxyz(frame2, mask, frame1, yexpr="some expression f(y,z,x)")
c: MT_Lutxyz(frame1, frame2, mask, yexpr="some expression f(z,x,y)")

Same question could be asked for MT_Lutxy, but since the lut is small, it's not a big deal.

In the same vein, while trying to measure the speed difference, I noticed x264 didn't give exactly the same output for options a,b,c.
This suggests a fundamental bug.

Chris

[Gavino]

a: MT_Lutxyz(mask, frame1, frame2, yexpr="some expression f(x,y,z)")
b: MT_Lutxyz(frame2, mask, frame1, yexpr="some expression f(y,z,x)")
c: MT_Lutxyz(frame1, frame2, mask, yexpr="some expression f(z,x,y)")
...
I noticed x264 didn't give exactly the same output for options a,b,c.
This suggests a fundamental bug.

Perhaps just a typo, but you seem to have the expressions for b and c the wrong way round: b should be f(z,x,y) and c should be f(y,z,x). That might explain why you get different results.

[ChrisBeringB]

b and c are correct, but not that it matters - a,b and c are purely for illustrative purposes.
I imagined f(x,y,z) as a function that takes a mask as its first argument. So in b, when y=mask, all x'es are replaced with y's in f().

Chris

[ChrisBeringB]

MT_Adddiff and MT_Lutxy(yexpr="x y + 128 -")
doesn't yield the same results either.

Chris

[Manao]

Can you give precise examples ? Complete scripts ?

[ChrisBeringB]

q1=MT_Lutspa(BlankClip(width=256, height=256, pixel_type="YV12"), mode="absolute")
q2=q1.TurnRight
return Subtract(MT_Lutxy(q2, q1, yexpr="x y + 128 -"), MT_Adddiff(q2, q1)).levels(125, 1, 127, 0, 255, false).greyscale.converttorgb32

Shows that the two functions are indeed identical.

But in a complex script I have this line:
#mask2=MT_Adddiff(mask2, t)
mask2=MT_Lutxy(mask2, t, yexpr="x y + 128 -")

I get different, but consistent x264 encodings according to which line I use.
Very strange...

Chris

[Manao]

By default, both mt_adddiff and mt_lutxy (and actually, all filters from masktools, excepted mt_merge), create a random chroma channel (or, rather, U & V are malloc'd, but not memset'd). In order to have consistent encodings, you need either to add U=2, V=2 to both filters (that will copy the chroma channels from the source clip) or chroma="copy", or chroma = "128" (to set them to 128), or... Read the documentation, the part about common parameters (I know, it's easy to miss, everybody tends to jump directly to the filter list...)

My guess is that it also explains the behavior you're observing with mt_lutxyz.

[Manao]

As for mt_lutxyz cache coherency : the lut array is lut[x][y][z], so it's better if the first parameter is always constant. However, it's even better if you manage to avoid mt_lutxyz altogether...

[ChrisBeringB]

I've read your documentation carefully before using MT, and chroma isn't used at all in my script. Infact, after I first discovered the issue, I put greyscale on the output while doing further testing, just to be sure.

Thanks for the lut info - and yes, I try to use cascaded lutxy instead of lutxyz.

[ChrisBeringB]

mask2=MT_Adddiff(mask2, t, u=1, v=1)
mask2=MT_Lutxy(mask2, t, yexpr="x y + 128 -", u=1, v=1)
mask2=MT_Lutxy(mask2, t, yexpr="x y + 128 -")
Consistently gives the same result.
EDIT: also mask2=MT_Adddiff(mask2, t, u=0, v=0)

mask2=MT_Adddiff(mask2, t)
Consistently gives a different result.
EDIT: putting u=0, v=0 on every MT function in my script also yields this result.

In my script atleast.

Chris

[yup]

As for mt_lutxyz cache coherency : the lut array is lut[x][y][z], so it's better if the first parameter is always constant. However, it's even better if you manage to avoid mt_lutxyz altogether...
Yes when I replace one mt_lutxyz to two mt_lutxy my script load faster and VirtualDub consume less memory.
mt_lutxyz use very big table 256*256*256*4 by comparing to 256*256*3 for mt_lutxy and do not have chance place this at cache.
May be I wrong about size for table but I hope You understand.
yup.

[Amateur]

I tried using an older version, mftoon2 (mfToon-v0.54.avsi), but it keeps telling me that the "Unsharpmask" function cannot be found... I can't this function anywhere. Is it part of an older package I might be missing?

i too am getting this error and i didn't see an answer to his problem in this thread. i got the newest version of mask tools from manao's signature so where can i find unmask?