MaskTools2 - pfmod

[real.finder]

It's not a surprise I confirm my above proposal.
Reason is 'scaleb' and 'scalef' do not carry any special character, i.e. they look like constants, but in fact they are a combination of a resolution-varying constant and a bitshift- or multiplication operation. Admittedly, 'sin', etc. also are functions and have non-operator-names.

scaleb and scalef are quite long tokens too (as are the other recently introduced constants), which is, in a formula, often not the preferred concept for readability (as opposed to text, formulas often need to be looked at as the whole thing / recursively, cannot just be read from start to end).

I admit @ is equivocal with the XOR operator and 'do a bitshift to range' is maybe semantically closer to '<<R' when compared with '<<s'. That's why I mentioned alternatives '<<<' (triple < can be understood somehow as 'special left shift', can't it?) and '^^^' (special scale up), which might also be '***' :D

So what about '<<r' and '*r', (r for range or resolution) which closest follows the 'OperatorLetter' principle of all the earlier &u ... >>s two step operations .

I see << in vs py script, and there are another things like dup operator, but I will vote for SF and SB

[pinterf]

<< like operators would mean >>, depending on the defined base bitmap, the operator would do either scaling up or down.

[real.finder]

I see avs script from 2011 use << operator (YAHRmask), the guides and change logs of masktools didn't mention that << is supported, is there another hidden operators and functions that masktools2 support?

[pinterf]

They are in change log of 2.0.47/48?

[real.finder]

They are in change log of 2.0.47/48?

no, that why I ask, since you see the source code and edit it

even in the wiki http://avisynth.nl/index.php/MaskTools2#Reverse_polish_notation

[pinterf]

Then I'll refresh it in wiki and doc, but I'm sure I've seen them somewhere.

[raffriff42]

Then I'll refresh it in wiki and doc, but I'm sure I've seen them somewhere.They've been there for a while, but only in the source tree(s) AFAICT
<source root>/masktools/documentation/mt_masktools.html#reverse_polish_notationV) Reverse polish notation.
...Signed and unsigned unary and binary operators exists : "&u", "|u", "°u" ("@u"), "~u", "<<u", ">>u" round and clip their operands to the nearest integer in [0..2^64-1], then do a "and", "or", "xor", "complement to 1", "left shift", or "right shift". Accordingly, "&s", "|s", "°s" (@s), "~s", "<<s", ">>s" do the same operations, but round and clip to the nearest integer in [-2^63..2^63-1] beforehand. "<<" and ">>" are equivalent to "<<u" and ">>u". The second operand of shifts is always considered signed : a left shift by a negative value is equivalent to a right shift by the opposite of the value.

...
Changelog
Alpha 48 : added : signed and unsigned binary shifts to LUTs

[real.finder]

They've been there for a while, but only in the source tree(s) AFAICT
<source root>/masktools/documentation/mt_masktools.html#reverse_polish_notation

yes, I just note they are in masktools-v2.0a48, I usually see the wiki for documentation, and didn't interested in reverse polish notation before beta by tp7, and since he didn't put the documentation with the binaries I just see the wiki link as it should has the last update for beta changes


that aside, pinterf, can you add stack manipulation operators, swap and dup (as vapoursynth expr)?

[pinterf]

Updated MaskTools Wiki
http://avisynth.nl/index.php/MaskTools2

[real.finder]

Updated MaskTools Wiki
http://avisynth.nl/index.php/MaskTools2

:thanks:

and where is Reel.Deal? he interested in documentation and wiki

you already flooded porting plugins so I think someone else should did the documentation

[pinterf]

New release
masktools2 v2.2.6 (https://github.com/pinterf/masktools/releases/tag/2.2.6)
EDIT: new 2.2.6 link instead of v2.2.5 (hotfix)

The most important change in this release is the autoscaling of parameters (this is not the in-expression case!). You can use the same parameters as you do in 8 bit scripts.
E.g. mt_binarize threshold "80" will be autoscaled to 80*256 for a 16 bit clip. You can override this behaviour with a new parameter "paramscale" in order to have such parameters interpreted as native 10-16 bit or float values, or disable scaling with "none".

By introducing float support, filter signatures have been changed to allow floats for Y, U, V and other parameters. Unfortunately this resulted in a strange situation: when the new masktools2 was loaded later with LoadPlugin, and there was an old masktools2.dll autoloaded earlier, then Avisynth was finding the old filters from the 8 bit masktools2 version. This happened when the filter was invoked with integer parameters (best parameter type match).
Now the filters are defined with dual parameter list, one with the new float types, and another one with the old integer-type list, to prevent finding the old masktools2 filters rather than the new one.

The operator of the in-expression bit depth scaling (#F and #B) was changed to @B and @F

Then, after the request from real.finder, dup and swap keywords are supported now.


**v2.2.6 (20170401)
- fix: >>u operator AV error
**v2.2.5 (20170330)
- Change #F and #B operators to @B and @F
- Alias scaleb for @B
- Alias scalef for @F
- New: automatic scaling of parameters (threshold-like, sc_value) from the usual 8 bit range
Scripts need no extra measures to work for all bit depths with the same "command line"
- New parameter "paramscale" for filters working with threshold-like parameters
Filters: mt_binarize, mt_edge, mt_inpand, mt_expand, mt_inflate, mt_deflate, mt_motion, mt_logic, mt_clamp
paramscale can be "i8" (default), "i10", "i10", "i12", "i14", "i16", "f32" or "none" or ""
Using "paramscale" tells the filter that parameters are given at what bit depth range.
- dual function signatures (float and int), for backward compatibility with integer-type parameter list, prevent usage of earlier plugin-loaded masktools version
- keep old parameter ordering: parameters which are non-existant in 2.0b1 are inserted at the end of the parameter list, not before the common parameters Y, U, V
- new: "swap" keyword in expressions
- new: "dup" keyword in expressions
- a bit faster realtime lut calculation for 10+ bit depths

[Dreamland]

First:Thx a lot for great work
I have an issue with MaskTools2 v2.2.5


https://t9.pixhost.org/thumbs/224/38966724_appunti01.jpg (https://pixhost.org/show/224/38966724_appunti01.jpg)

my AVS works with MaskTools2 v2.2.4

:)

[pinterf]

First:Thx a lot for great work
I have an issue with MaskTools2 v2.2.5


https://t9.pixhost.org/thumbs/224/38966724_appunti01.jpg (https://pixhost.org/show/224/38966724_appunti01.jpg)

my AVS works with MaskTools2 v2.2.4

:)
Thanks. The operator >>u was called as a function with one parameter but it has two params. Hotfix is coming soon.

A lot of code has been changed in expression handling, in order to have dup, swap, and a general speedup in realtime calculations where lut is not avaliable (until VS-like Expr will appear - not very soon though). Malformed expressions can still cause troubles, but I think they weren't checked in earlier versions either.

EDIT: 2.2.6 is out.

[pinterf]

New version available with hotfix (>>u operator).
Masktools 2.2.6 (https://github.com/pinterf/masktools/releases/tag/2.2.6)

[Dreamland]

Perfect!
now it works ..many thanks! :)

[real.finder]

hi pinterf

remember this? https://forum.doom9.org/showthread.php?p=1794976#post1794976

some encoders use this formula in lutxyz
((clip1*(255-mask))+(clip2*mask))/255

ColorBars(pixel_type="yv12")
src = mt_lutspa(expr="x 255 *")
mask = mt_lut(y=-255)

smaskmerge (https://pastebin.com/YHM68SvX)(mt_lut(y=-0), src, mask,u=2,v=2) #will show all frame in zero, and that what should
#~ mt_merge(mt_lut(y=-0), src, mask,u=2,v=2) #will not show all frame in zero

mt_lutxy(last, src, "x y - abs 75 *").grayscale()

edit: and here smaskmerge with another formula https://pastebin.com/aLP9Mb3z

so, it's better replace the one in mt_merge or put some option to change the formula for compatible?

I vote for put option and the default will be the new formula in non float format

[pinterf]

Lut always works in 64 bit double precision, integer roundings are no problem there.

10-16 bit code path already has special treatment of min (0) and max (e.g. 1023,... 65535) masks
https://github.com/pinterf/masktools/blob/16bit/masktools/filters/merge/merge16.cpp#L15
https://github.com/pinterf/masktools/blob/16bit/masktools/filters/merge/merge16.cpp#L116

8 bit code kept the original implementation. Maybe it is reasonable to handle the situation in a similar way.
https://github.com/pinterf/masktools/blob/16bit/masktools/filters/merge/merge.cpp#L96

[pinterf]

New release

Masktools2 v2.2.7 (https://github.com/pinterf/masktools/releases/tag/2.2.7)

Changelog
**v2.2.7 (20170421)
- fix: mt_edge 10,12,14 bits: clamp mask value from 65535 to 1023 (10 bits), 4095 (12 bits) and 16383 (14 bits)
- fix: mt_merge 10-16 bits + non mod-16 width + luma=true + 4:2:2 colorspace, correct right side pixels
- fix: mt_merge 8 bit clips: keep original pixels from clip1/2 when mask is exactly 0 or 255
- YUVA, RGBAP support 8-32 bits
- "A" parameter like "Y", "U" and "V". Default value for "A" is 1 (do nothing, same as for "U" and "V")
- "alpha" parameter like "chroma" - overrides default plane mode
- aExpr parameter for lut-type filters like uExpr, and vExpr
- awExpr parameter for mt_luts like uwExpr, and vwExpr
- dual signature filters (both integer and float) are provided in separate binaries
In some cases specifying two different parameter lists with the same variables can cause troubles.
(dual signature version can be used to override an earlierly loaded different masktools version
(e.g. a 2.5 plugin) by defining the filters with both integer parameters _AND_ the new float parameter lists)
- Make "scaleparams" to work consistent with all filters and parameters:
parameters "Y","U","V" and "A" negative (memset) values are scaled automatically to the current bit depth from a default 8-bit value.
- New plane mode: 6 (copy from fourth clip) for "Y", "U", "V" and "A"
New "chroma" and "alpha" plane mode override: "copy fourth"
Use for mt_lutxyza which has four clips

[real.finder]

New release

Masktools2 v2.2.7 (https://github.com/pinterf/masktools/releases/tag/2.2.7)


:thanks:

but seems the dual filter signatures is same as the non dual filter signatures, at least for x86 xp (give me Unsupported color space as before)

[pinterf]

:thanks:

but seems the dual filter signatures is same as the non dual filter signatures, at least for x86 xp (give me Unsupported color space as before)
EDIT: Replaced those two DLLs (dual-xp x86/x64), sorry for the inconvenience.

[Motenai Yoda]

- fix: mt_merge 10-16 bits + non mod-16 width + luma=true + 4:2:2 colorspace, correct right side pixels
not 4:2:0 too?

[pinterf]

The original reason of 420 nonmod16 right edge garbage was the mt_edge thing.

[real.finder]

The original reason of 420 nonmod16 right edge garbage was the mt_edge thing.

mt_merge with luma=true in 4:2:0 use mpeg2 chroma place, right?

[pinterf]

mt_edge with luma=true in 4:2:0 use mpeg2 chroma place, right?
You mean mt_merge.
Yes, I can see no special clever subsampling shifts in the code.

[real.finder]

You mean mt_merge.
Yes, I can see no special clever subsampling shifts in the code.

yes, mt_merge

do you think mt_merge will need some Parameter for mpeg2 vs mpeg1 in 420 with luma=true?

[MysteryX]

M = mt_merge(M, BHard, Sc, luma=true)


This code applies ChangeFps (BHard) on scene changes (Sc).

If I instead write Sc.ConvertToY8(), even though luma=true, the BHard is only being applied to Luma and not to chroma planes.

Is this intentional or a bug?

(why should I process YV12 masks when only Y8 is needed?)

[StainlessS]

@MysteryX

u and v are defaulted to 2 (that way, the resulting clip contains the chroma of clip1, and looks right).

These three values describe the actual processing mode that is to be used on each plane / channel. Here is how the modes are coded :

x=-255..0 : all the pixels of the plane will be set to -x.
x=1 : the plane will not be processed. That means the content of the plane after the filter is pure garbage.
x=2 : the plane of the first input clip will be copied.
x=3 : the plane will be processed with the processing the filter is designed to do.
x=4 (when applicable) : the plane of the second input clip will be copied.
x=5 (when applicable) : the plane of the third input clip will be copied.

As you can see, defaults parameters are chosen to only process the luma, and not to care about the chroma. It's because most video processing doesn't touch the chroma when handling 4:2:0.
string "chroma" ("")

When defined, the value contained in this string will overwrite the u & v processing modes. This is a nice addition proposed by mg262 that makes the filter more user friendly. Allowed values for chroma are :

"process" : set u = v = 3.
"copy" or "copy first" : set u = v = 2.
"copy second" : set u = v = 4.
"copy third" : set u = v = 5.
"xxx", where xxx is a number : set u = v = -xxx.


Set u=3, v=3 or Chroma="Process".

[MysteryX]

Thanks, that works. There is one more issue.


file = "Motion Estimation Torture Clip.avi"
C = LWLibavVideoSource(file, cache=False).ConvertToYV12()
BHard = C.ChangeFPS(60)
B = C.ConvertFPS(60)
Sc = BlankClip(C, color=color_white).ConvertToY8()
mt_merge(B, BHard, Sc, luma=true, chroma="process")


This "should" apply ChangeFps to the clip and repeat frames. However, about 5% of ConvertFps is showing up so repeated frames aren't identical.

What is causing this?

[MysteryX]

Sc is in the 16-235 range which causes blending of every frame. This fixes the issue


Sc = Sc.mt_binarize(128)

Is there a better way to do it?

[Gavino]

Sc is in the 16-235 range which causes blending of every frame. This fixes the issue

Sc = Sc.mt_binarize(128)

Is there a better way to do it?
Instead of

Sc = BlankClip(C, color=color_white).ConvertToY8()

use
Sc = BlankClip(C, pixel_type="Y8", color_yuv=$ff0000)

[raffriff42]

In mt_binarize, the threshold argument is scaled from 0-255 to the current bit depth,
but the chroma argument assumes the current scale. I assume this is a bug?
Which one's "right?"

https://www.dropbox.com/s/z7gxg6uozubah5y/mt_bin14-1.jpg?raw=1

https://www.dropbox.com/s/ybyqasbe9b8weou/mt_bin14-2.jpg?raw=1

https://www.dropbox.com/s/vh58lykg4ztf6se/mt_bin14-3.jpg?raw=1

By the way, for the wiki, I'd like to how to describe a variable like threshold:
multi-bit-depth? deep-color-aware? autoscale? (asking for opinions)

[pinterf]

In mt_binarize, the threshold argument is scaled to the current bit depth,
but the chroma argument is always on a 0-255 scale. I assume this is a bug?

Yes. It should behave like Y,U,V and A in this case. I hope there are no more hidden variables. Thanks for the report.

[pinterf]

By the way, for the wiki, I'd like to how to describe a variable like threshold:
multi-bit-depth? deep-color-aware? autoscale? (asking for opinions)
Perhaps autoscale or autoscalable.

(just to refresh our minds: depending on "paramscale" such parameters can be autoscaled, default base is from 8 bits, paramscale="none" does nothing for users who like to keep the behaviour in their hands)

[pinterf]

New release
Masktools2 2.2.8 (https://github.com/pinterf/masktools/releases/tag/2.2.8)

Thanks to raffriff42 for the report.

Recent changes:

**v2.2.8 (20170427)
- Fix: "chroma" and "alpha" parameter should be scaled like "Y","U","V" and "A" when providing negative (memset) values

**v2.2.7 (20170421)
- fix: mt_edge 10,12,14 bits: clamp mask value from 65535 to 1023 (10 bits), 4095 (12 bits) and 16383 (14 bits)
- fix: mt_merge 10-16 bits + non mod-16 width + luma=true + 4:2:2 colorspace, correct right side pixels
- fix: mt_merge 8 bit clips: keep original pixels from clip1/2 when mask is exactly 0 or 255
- YUVA, RGBAP support 8-32 bits
- "A" parameter like "Y", "U" and "V". Default value for "A" is 1 (do nothing, same as for "U" and "V")
- "alpha" parameter like "chroma" - overrides default plane mode
- aExpr parameter for lut-type filters like uExpr, and vExpr
- awExpr parameter for mt_luts like uwExpr, and vwExpr
- dual signature filters (both integer and float) are provided in separate binaries
In some cases specifying two different parameter lists with the same variables can cause troubles.
(dual signature version can be used to override an earlierly loaded different masktools version
(e.g. a 2.5 plugin) by defining the filters with both integer parameters _AND_ the new float parameter lists)
- Make "scaleparams" to work consistent with all filters and parameters:
parameters "Y","U","V" and "A" negative (memset) values are scaled automatically to the current bit depth from a default 8-bit value.
- New plane mode: 6 (copy from fourth clip) for "Y", "U", "V" and "A"
New "chroma" and "alpha" plane mode override: "copy fourth"
Use for mt_lutxyza which has four clips

[hello_hello]

I accidentally posted this in the MVTools2 thread originally, so sorry about the cross-posting.

pinterf,
In case you'd care to take a look at it, there appears to be a problem with your flavour of MaskTools2 (any version) and the SeeSaw sharpening script.
https://forum.doom9.org/showthread.php?p=1808548#post1808548

Thanks.

Hi, I have tried the new version of masktools and there is a problem

Evaluate: Unhandled C++ exception!

(SeeSaw.avs, line 128)

(SeeSaw.avs, line 80)

[burfadel]

I accidentally posted this in the MVTools2 thread originally, so sorry about the cross-posting.

pinterf,
In case you'd care to take a look at it, there appears to be a problem with your flavour of MaskTools2 (any version) and the SeeSaw sharpening script.
https://forum.doom9.org/showthread.php?p=1808548#post1808548

Thanks.

As I posted in the other thread, does seesaw() work by itself, and if so what additional filter causes it not to work for you?

EDIT: Tried it. A while ago it only didn't work with using it with lsfmod etc, now it just doesn't work at all!

[manolito]

As outlined in the first post of the SeeSaw thread the required additional tools are RemoveGrain and MaskTools 1.5.8.
https://forum.doom9.org/showthread.php?p=758021#post758021

The current pinterf versions of MaskTools are not compatible, you need the old masktools.dll 1.5.8 from Manao.

Cheers
manolito

[hello_hello]

As I posted in the other thread, does seesaw() work by itself, and if so what additional filter causes it not to work for you?

Odd....
I wasn't thinking when you asked the question about LSFMod in the other thread, because obviously you feed SeeSaw a denoised clip, not a sharpened one....

but so far it's all working as expected with the non-pinterf versions of MaskTools2 I tried.

SeeSaw()

or

a = last
b = a.SMDegrain()
SeeSaw(a, b)

Or even

SeeSaw()
LFSMod()

All work, and it is sharpening. I checked in case that's what you meant by "not working".
Replacing MaskTools2 with a pinterf version results in the exception error, even when using just SeeSaw() on it's own.

manolito,
The only version of SeeSaw I've been able to find (and it's the one I'm using) specifies MaskTools2.
There's a doom9 link at the top of the script, but unfortunately it doesn't work so I don't know if it was part of the original SeeSaw thread or why the post/thread was deleted.
http://avisynth.nl/index.php/SeeSaw

[StainlessS]

manolito,
The only version of SeeSaw I've been able to find (and it's the one I'm using) specifies MaskTools2.
There's a doom9 link at the top of the script, but unfortunately it doesn't work so I don't know if it was part of the original SeeSaw thread or why the post/thread was deleted.
http://avisynth.nl/index.php/SeeSaw

The non working link in script is this one:- http://forum.doom9.org/showthread.php?p=1748710#post1748710

Which is I think a link to post where I modded SeeSaw as new Avs v2.6 change broke things:, Force int call arguments to user script function float params to be explicit floats.

I went looking for that post/thread, could not find it. (looks like thread was deleted [perhaps an InGoldie thread, he has habit of deletion]).

[Reel.Deel]

The non working link in script is this one:- http://forum.doom9.org/showthread.php?p=1748710#post1748710

Which is I think a link to post where I modded SeeSaw as new Avs v2.6 change broke things:, Force int call arguments to user script function float params to be explicit floats.

I went looking for that post/thread, could not find it. (looks like thread was deleted [perhaps an InGoldie thread, he has habit of deletion]).

I remember the thread, according to upload log on the wiki (http://avisynth.nl/index.php/File:SeeSaw.avs) you modified the script twice.


The only version of SeeSaw I've been able to find (and it's the one I'm using) specifies MaskTools2.
There's a doom9 link at the top of the script, but unfortunately it doesn't work so I don't know if it was part of the original SeeSaw thread or why the post/thread was deleted.
http://avisynth.nl/index.php/SeeSaw

SeeSaw using MaskTools2 has been on the wiki since early 2008.
https://web.archive.org/web/20080516150126/http://avisynth.org:80/mediawiki/SeeSaw

As outlined in the first post of the SeeSaw thread the required additional tools are RemoveGrain and MaskTools 1.5.8.
https://forum.doom9.org/showthread.php?p=758021#post758021

The current pinterf versions of MaskTools are not compatible, you need the old masktools.dll 1.5.8 from Manao.


Just a public service announcement. :)

Do not use the old SeeSaw script.
Do not use MaskTools 1.5.8.
Use RgTools over RemoveGrain.

[pinterf]

New version.

Download Masktools2 2.2.9 (https://github.com/pinterf/masktools/releases/tag/2.2.9)

Changes
**v2.2.9 (20170608)
- Add "none" and "ignore" to valid values for "chroma" and "alpha" parameters.
- Report error for invalid "chroma" or "alpha" parameter values instead of exception

[StainlessS]

Thank you oh wise one, we loves you !

[real.finder]

there are another bugs that hnn mention here https://forum.doom9.org/showthread.php?p=1808701#post1808701

the one in GradFun2DBmod reproduced, it's in mt_luts(input,input,mode="range",pixels=mt_square(radius),expr="y",u=1,v=1)

if I remove expr="y" then it will work

the other one didn't reproduced for me

[pinterf]

Thanks, found it, fix along with some serious speed improvements in 16 bit xxflate/xxpand will be coming soon

[pinterf]

New build with an important fix for mt_luts.
And a bonus performance increase in 10-16bits xxpand/xxflate for users having at least SSE4.

Download MaskTools2 2.2.10 (https://github.com/pinterf/masktools/releases/tag/2.2.10)

Changelog
**v2.2.10 (20170612)
- Fix: luts internal buffer overflow (crash)
- Speed: mt_inpand/mt_expand: 10-16 bits SSE4 (10-15x speed)
- Speed: mt_inflate/mt_deflate 10-16 bits SSE4 (4x speed)

[real.finder]

New build with an important fix for mt_luts.
And a bonus performance increase in 10-16bits xxpand/xxflate for users having at least SSE4.

Download MaskTools2 2.2.10 (https://github.com/pinterf/masktools/releases/tag/2.2.10)

Changelog
**v2.2.10 (20170612)
- Fix: luts internal buffer overflow (crash)
- Speed: mt_inpand/mt_expand: 10-16 bits SSE4 (10-15x speed)
- Speed: mt_inflate/mt_deflate 10-16 bits SSE4 (4x speed)


thanks, I can feel the speed up :thanks:

[Groucho2004]

thanks, I can feel the speed up :thanks:I decided to measure it and yes, your feeling was spot on. :D

Nice work Ferenc!

Edit: The same applies to MVTools.

[edcrfv94]

#invert PC range and invert back
mt_lut("x 255 - abs", y=3, u=3, v=3)
mt_lut("x 255 - abs", y=3, u=3, v=3)



#invert limited range and invert back
mt_lut(yexpr="x 235 - abs 16 +", uexpr="x 240 - abs 16 +", vexpr="x 240 - abs 16 +", y=3, u=3, v=3)
mt_lut(yexpr="x 235 - abs 16 +", uexpr="x 240 - abs 16 +", vexpr="x 240 - abs 16 +", y=3, u=3, v=3)



#There is no problem in Python
for x in range(16, 236):
x = abs(x -235) + 16
print (x)


limited range can not invert back to original image, it is bug?

[jpsdr]

Could it be that your limited range trig threshold, and so thresholded values cannot be reverted because information is lost ?

[edcrfv94]

Could it be that your limited range trig threshold, and so thresholded values cannot be reverted because information is lost ?

But limited range video use:

mt_lut("x 255 - abs", y=3, u=3, v=3)
mt_lut("x 255 - abs", y=3, u=3, v=3)
#e.g: 16 to 239 back to 16

can invert back to original image.
Where can disabled limited range threshold?