MaskTools2 - pfmod

[real.finder]

Why do you need that float input autoscaling? The usual #B and #F are working properly for the base float range.

yes, usual #B and #F work but some expr that use range_size (dehalo alpha) and range_max both return 1.0 in float and that will affect multiplication operations

[pinterf]

clamp_f sound good, and if it (clamp_f) make any x and y etc... from 1.0-0.0 to 255.0-0.0 for process like 8 bit then back to 1.0-0.0 will be better and we don't worry about do something special for float, in other word we will have one expr for both

and that also mean clamp_f will make #F and Range_size etc... keep or convert to like 8 bit expr (255.0-0.0)
I have to understand why multiplication is affected before we make changes in syntax that will probably held till the end of the XXIth century. The other way is more logical to me that we keep everything normalized to the 0..1.0 range. (but anyway, it's only the question of being documented)

[real.finder]

I have to understand why multiplication is affected before we make changes in syntax that will probably held till the end of the XXIth century. The other way is more logical to me that we keep everything normalized to the 0..1.0 range. (but anyway, it's only the question of being documented)

y x - y 0.001 + / range_max * "+LOS+" #F - y range_size + 512 #F / "+HIS+" + *

is ((((y-x)/y+0.001)*range_max)-los #F)*(((y+range_size)/512 #F)+his)

if we have x=0.4 y=0.5 (float)
then we have 0.3802586358762255 (96.96595214843751 in 255)

if we did it in 255.0-0.0 (x=102, y=127.5) then we have 0.0048961718903666 (1.248523832043474 in 255)

[martin53]

working example (is more complicated than I expected), will publish soon.

a bit late - link to example (https://forum.doom9.org/showthread.php?p=1799122#post1799122)

[pinterf]

y x - y 0.001 + / range_max * "+LOS+" #F - y range_size + 512 #F / "+HIS+" + *

is ((((y-x)/y+0.001)*range_max)-los #F)*(((y+range_size)/512 #F)+his)

if we have x=0.4 y=0.5 (float)
then we have 0.3802586358762255 (96.96595214843751 in 255)

if we did it in 255.0-0.0 (x=102, y=127.5) then we have 0.0048961718903666 (1.248523832043474 in 255)
I see. Why is 0.001 not scaled?

Meanwhile I had an idea that instead of having #B and #F operators inside the script, we would scale the inputs (not only floats) instead, to a desired bitdepth. But in this case we'd have to specify for each input, what method they are to be scaled with (e.g. bit-shifts or full-scale stretch.) if they are different from the logical default method.

For YUV the default would be the shift method (e.g. x << 8), for rgb inputs: full-stretch (e.g. x*65535/255), for greyscale:??? we don't know, they can be masks (full-stretch needed) or a YUV plane (bit-shift is needed).

Anyway, it takes one conversion at the input (per variable), and one at the output for the result. (if input's bitdepth is different that is specified).

[real.finder]

I see. Why is 0.001 not scaled?



I did scale it back then but it make 10-16 somehow wrong, and in float it not did anything right too cuz there are range_size and range_max problems

Meanwhile I had an idea that instead of having #B and #F operators inside the script, we would scale the inputs (not only floats) instead, to a desired bitdepth. But in this case we'd have to specify for each input, what method they are to be scaled with (e.g. bit-shifts or full-scale stretch.) if they are different from the logical default method.

For YUV the default would be the shift method (e.g. x << 8), for rgb inputs: full-stretch (e.g. x*65535/255), for greyscale:??? we don't know, they can be masks (full-stretch needed) or a YUV plane (bit-shift is needed).

Anyway, it takes one conversion at the input (per variable), and one at the output for the result. (if input's bitdepth is different that is specified).

I know, #B and #F better for most cases, so I ask for do 255.0-0.0 in float with clamp_f (special cases) only, so it will be special method for special cases

edit: #B and #F also will have effect if the expr is 10-16 bit (i10, i12 etc...)

[martin53]

y x - y 0.001 + / range_max * "+LOS+" #F - y range_size + 512 #F / "+HIS+" + *


Can you please provide LOS and HIS so I can review the whole calculation?
I transformed

y x - y 0.001 + / range_max * "+LOS+" #F - y range_size + 512 #F / "+HIS+" + *

manually to

(y-x) / (y+0.001) * range_max - #F(LOS) * ((y+range_size) / #F(512) + HIS)

and I'm wondering if HIS sholuld also be #F(HIS).

@pinterf: I used the helper function mt_infix() to check my transform. And I saw it transform the #F operator in a way that surprised me.
mt_infix("17 #F") is #F(17,133168,8). My question is: Where does the helper function get the 2nd and 3rd parameter from? It looks like ~2^17 and 8 bits. But the helper function works without any bit depth context; it is just a string rearranger.

Edit: mt_polish(mt_infix("17 #F")) should not produce an error, but probably be "17 #F", maybe with braces.
I.e. I think mt_infix("17 #F") should just be "#F(17)" and mt_polish("#F(17)") should be "17 #F"

EDIT 2: Please see the red braces I added after your quote. The * operand clearly affects the whole sum.

[real.finder]

Can you please provide LOS and HIS so I can review the whole calculation?
I transformed

y x - y 0.001 + / range_max * "+LOS+" #F - y range_size + 512 #F / "+HIS+" + *

manually to

(y-x) / (y+0.001) * range_max - #F(LOS) * (y+range_size) / #F(512) + HIS

and I'm wondering if HIS sholuld also be #F(HIS).

@pinterf: I used the helper function mt_infix() to check my transform. And I saw it transform the #F operator in a way that surprised me.
mt_infix("17 #F") is #F(17,133168,8). My question is: Where does the helper function get the 2nd and 3rd parameter from? It looks like ~2^17 and 8 bits. But the helper function works without any bit depth context; it is just a string rearranger.

see DeHalo_alpha function (it's by Didée since 2006, but there are port in post above too)

[martin53]

see DeHalo_alpha function (it's by Didée since 2006, but there are port in post above too)

Calculation in 8 bits, LOS=HIS=50 (the defaults),
omitting the coward denominator adder 0.001 for simplicity, #F is not there,
x=102, y=128, range_max=255, range_size=256.
(128 - 102) / 128 * 255 - 50 * ((128 + 256) / 512 + 50)
26/128*255 is 51.8, the term after the minus sign is 2538.

I think the braces I added in my 1st post with the formula could be incorrect. Then:
(128 - 102) / 128 * 255 - 50 * (128 + 256) / 512 + 50
is 51.8 - 37.5 + 50
= 64.3
This is far from your result (1.248523832043474 in 255).

[pinterf]

For infix you have to use parentheses like #F(value) as if it was a function like sin

[real.finder]

Calculation in 8 bits, LOS=HIS=50 (the defaults),
omitting the coward denominator adder 0.001 for simplicity, #F is not there,
x=102, y=128, range_max=255, range_size=256.
(128 - 102) / 128 * 255 - 50 * ((128 + 256) / 512 + 50)
26/128*255 is 51.8, the term after the minus sign is 2538.

I think the braces I added in my 1st post with the formula could be incorrect. Then:
(128 - 102) / 128 * 255 - 50 * (128 + 256) / 512 + 50
is 51.8 - 37.5 + 50
= 64.3
This is far from your result (1.248523832043474 in 255).

you forget HIS = string(highsens/100.0)

and it's ((((y-x)/y+0.001)*range_max)-#F(los))*(((y+range_size)/#F(512))+his)

so you must dealing with brackets First

anyway, why you did this? pinterf already get what I mean :)

[StainlessS]

braces like #F(value)

Small point but those are parenthesis (), these are braces {}. EDIT: and these are brackets <>.

[real.finder]

Small point but those are parenthesis (), these are braces {}. EDIT: and these are brackets <>.

US VS UK? :rolleyes: http://english.stackexchange.com/questions/3379/bracket-vs-brace

[pinterf]

Fresh, new build, test it. If the new things (autoconverting ducks to geese) are not working as expected, I can fix it later next week.

This release is also addressing some of martin53's report (mt_polish and probably mt_infix)

Masktools2 2.2.4 (https://github.com/pinterf/masktools/releases/tag/2.2.4)

**v2.2.4 (20170304)**
- mt_polish to recognize:
- new v2.2.x constants and variables
a, bitdepth, sbitdepth, range_half, range_max, range_size, ymin, ymax, cmin, cmax
- v2.2.x scaling functions (written as #B(expression) and #F(expression) for mt_polish)
#B() #F
- single operand unsigned and signed negate introduced in 2.0.48
~u and ~s (written as ~u(expression)and ~s(expression) for mt_polish)
- other operators introduced in 2.0.48:
@, &u, |u, °u, @u, >>, >>u, <<, <<u, &s, |s, °s, @s, >>s, <<s
- mt_infix: don't put extra parameter after #F( and #B(
- new expression syntax: auto scale modifiers for float clips (test for real.finder):
Keyword at the beginning of the expression:
- clamp_f_i8, clamp_f_i10, clamp_f_i12, clamp_f_i14 or clamp_f_i16 for scaling and clamping
- clamp_f_f32 or clamp_f: for clamping the result to 0..1

Input values 'x', 'y', 'z' and 'a' are autoscaled by 255.0, 1023.0, ... 65535.0 before the expression evaluation,
so the working range is similar to native 8, 10, ... 16 bits. The predefined constants 'range_max', etc. will behave for 8, 10,..16 bits accordingly.

The result is automatically scaled back to 0..1 _and_ is clamped to that range.
When using clamp_f_f32 (or clamp_f) the scale factor is 1.0 (so there is no scaling), but the final clamping will be done anyway.
No integer rounding occurs.

[real.finder]

Fresh, new build

mt_lutxy( ugly, are, "clamp_f_i8 y x - y 0.001 + / range_max * "+LOS+" #F - y range_size + 512 #F / "+HIS+" + *" )

quick test and work fine for both ducks and geese :thanks:

[real.finder]

funny bug in yv411

mt_edge("sobel",7,7,5,5) give me unsupported error (same old masktools2 in alpha and beta)

mt_edge("sobel") work fine!

[martin53]

anyway, why you did this? pinterf already get what I mean :)
I am a bit worried why a float-specific clamping problem might need expression syntax. I'd like to understand that and hope there will also be a solution without float-specific, complex syntax.

Rationale
When expressions need different syntax for float, aside from being harder to understand & manage and more time consuming when executed in realtime, it will be impossible forever to provide scripts that are equally usable for all clip types (restricted to those the plugin can process). It should be the highest plugin designer priority to have syntax pixel_type independent. Otherwise newbies will have an unneccessarily bad AviSynth experience.

[martin53]

Especially in the context of the masktools thread I'd like to point out a tiny new plugin StrResolve (https://forum.doom9.org/showthread.php?p=1799667#post1799667) that takes arbitrary strings, looks if they contain names of currently defined variables and silently replaces the names by the values in the output.

[pinterf]

Float is non-saturated, and as such, will need special attention regarding clamping. This syntax extension helps a bit for having consistent bit depth independent expressions without using extra parameters in the filters.
Float is an expert use case, and may not be identical to integer color spaces because of lacking the default clamped behaviour.

[martin53]

Float is non-saturated, and as such, will need special attention regarding clamping. This syntax extension helps a bit for having consistent bit depth independent expressions without using extra parameters in the filters.
Float is an expert use case, and may not be identical to integer color spaces because of lacking the default clamped behaviour.

Would the disadvantages be bigger, if clamped behaviour were default, and non clamping an expert use case than the advantages of identical expressions?

[real.finder]

Would the disadvantages be bigger, if clamped behaviour were default, and non clamping an expert use case than the advantages of identical expressions?

clamped behaviour is not default in float (whether now or before I asked for it)

people used to write expr that may give out of range values, lut is clamped it to 0-255, and so in 10-16 bit

but in float there are no clamped, so these kind of expr will be wrong in float, so I ask pinterf to add option to do that, so I used this option in dehalo alpha and Masked_DHA until now and it work well, and there are other scripts that need this option too I will port them soon

if you don't like it simply write new avs script function with new expr that don't give out of range :)

[martin53]

clamped behaviour is not default in float

Using the 'clip' operator at the end of the expression (for all pixel_types, since it's done anyway for fixed point) is not an option? Seems I don't get the difference.

[real.finder]

Using the 'clip' operator at the end of the expression (for all pixel_types, since it's done anyway for fixed point) is not an option? Seems I don't get the difference.

don't know, but I think yes

Clip[x,{min,max}]
gives x for min≤x≤max, min for x<min and max for x>max

but in this case you will need more things to write (the min and the max)

and it will not solve range_max and range_size problems in multiplication and division (cuz they give 1.0), unless you did two expr one for float and one for integer

[martin53]

two expr one for float and one for integer
Two expressions is clearly not the better option, I feel, because then everyone modifying it must understand/test/... both of them.

[real.finder]

hi pinterf

just start port FastLineDarkenMOD and note that

prot=5
mt_expand(thy=255/(prot+1)).mt_inpand()

give different results with different bit depth!

mt_edge (mode="prewitt", thY1=0, thY2=255) (from finedehalo)

this too

[real.finder]

this mt_binarize(70, mode="0 255") will not work in more than 8 bit, right?

[pinterf]

0 255 will work, mode names are unchanged. Input threshold parameter values however are not scaled

[real.finder]

mt_inflate(155) in more than 8 bit give

http://i.imgur.com/g8etO4R.png

[pinterf]

Isn't it 411?

[real.finder]

Isn't it 411?

it's 420

[pinterf]

it's 420
Could not reproduce, and the color space check is common for all mt_xxxx filters. Are you sure the right masktools plugin is loaded?
(I corrected the error message, anyway)

[real.finder]

Could not reproduce, and the color space check is common for all mt_xxxx filters. Are you sure the right masktools plugin is loaded?
(I corrected the error message, anyway)

did you try with xp one? cuz I am use xp in VirtualBox for testing

edit : even in x64 non xp

ColorBars
converttoyv12
convertbits(16)
mt_inflate(155)
convertbits(8)

[pinterf]

did you try with xp one? cuz I am use xp in VirtualBox for testing

edit : even in x64 non xp

ColorBars
converttoyv12
convertbits(16)
mt_inflate(155)
convertbits(8)
Works for me. xp, x64, all. What does avsmeter64 -avsinfo say?

[martin53]

Hi Ferenc,

operator #B scales from 8 bit to current bit depth using bit-shifts
operator #F scales from 8 bit to current bit depth using full range stretch


I suggest that you reconsider choosing #B and #F for these operators.

The point is that they collide with AviSynth's standard inline comment character '#'. David Horman says his RPN filters accept multiline strings with # comments (https://forum.doom9.org/showthread.php?p=1800428#post1800428) already, so he was the first to introduce this character for this purpose. Maybe you like alternatives like @B & @F (seems to me these can be fairly distinguished from @s, @u and the XOR @ operator itself), or '<<<' for bitshift, '^^^' for fullscale, or reserved words like you introduced them for the other new constants.

[pinterf]

I don't like them, either. I'd say yes on the changes before real.finder ports all the scripts in the world to the new syntax :)
real.finder?
Edit:
scaleb and scalef?

[real.finder]

Works for me. xp, x64, all. What does avsmeter64 -avsinfo say?

I don't think avsmeter64 -avsinfo will say something different in this case

I think I know were is the problem, I have mt_masktools-25.dll in plugins folder and made avs26.avsi to load the 2.6 one from subfolder if the avs is 2.6 or up, if I remove mt_masktools-25.dll it will work, but all my plugins work fine like this, why now there are some collision?

[real.finder]

I don't like them, either. I'd say yes on the changes before real.finder ports all the scripts in the world to the new syntax :)
real.finder?
Edit:
scaleb and scalef?

you can change it, no problem, I can use find and replace

[pinterf]

I don't think avsmeter64 -avsinfo will say something different in this case

I think I know were is the problem, I have mt_masktools-25.dll in plugins folder and made avs26.avsi to load the 2.6 one from subfolder if the avs is 2.6 or up, if I remove mt_masktools-25.dll it will work, but all my plugins work fine like this, why now there are some collision?
Using SetLogParams("log.txt", LOG_DEBUG) in the avs script (or may be a less restrictive setting) will list the possible collision.
Like for me:
"WARNING: VerticalCleaner() is defined by multiple plugins. Calls to this filter might be ambiguous and could result in the wrong function being called."

[real.finder]

Using SetLogParams("log.txt", LOG_DEBUG) in the avs script (or may be a less restrictive setting) will list the possible collision.
Like for me:
"WARNING: VerticalCleaner() is defined by multiple plugins. Calls to this filter might be ambiguous and could result in the wrong function being called."

http://pastebin.com/E1FALB7C

nothing special

[pinterf]

Nothing special?
WARNING: mt_inflate() is defined by multiple plugins.
Calls to this filter might be ambiguous and could result in the wrong function being called.
then
ERROR: mt_inflate : unsupported colorspace. masktools only support planar YUV colorspaces (YV12, YV16, YV24)

[real.finder]

Nothing special?
WARNING: mt_inflate() is defined by multiple plugins.
Calls to this filter might be ambiguous and could result in the wrong function being called.
then
ERROR: mt_inflate : unsupported colorspace. masktools only support planar YUV colorspaces (YV12, YV16, YV24)

defined by multiple plugins is something no special in my case cuz as I say in #86, the 2.6 one should being used cuz it load after 2.5 one, and that what happen except few cases in masktools2 in high bits

anyway, I can do something for this case by move the 2.5 one in another subfolder and call it if avs is 2.5

edit: just did this and note that the output of mt_inflate(155) in more than 8 bit is different

[pinterf]

Different, because filter parameters like threshold are not scaled, this was mentioned earlier. To scale or not to scale, that is the question, ideally this threshold is either a percent 0-100 or normalized to the 0.0-1.0 range. Does it have any drawback if we are not able to specify exactly e.g . 32765 in a 16 bit range, only a (full range scaled) 127/255*65535? Of course since this parameter is float, one can specify 127.6 if it is important.

You know the use cases better, I accept any logical proposal.

(And I have to check the plugin load behaviour, in particular the order of the filter function registrations)

[real.finder]

what about add bitdepth_scale parameter in the end of parameters?

and float 0.0-255.0 will be ok too (it will be compatible with old script in all bit depth since float parameter accept integer)

[pinterf]

The parameter type is float already because filters accept unscaled values for float clips

[real.finder]

The parameter type is float already because filters accept unscaled values for float clips

but it's not in 255.0 range with scale

anyway, bitdepth_scale (or if you have better name) can be in string to control scale method (like what you did in lut), can even be set to no scale like what is now, but the Default will be scale from 255 range (whether float or integer)

[pinterf]

The choice is @B @F. Scaling will assume parameter values on 8 bit scale but can be overridden to 10-16 bits or float or no scaling at all

[real.finder]

isn't better to use something away from @ and # etc...? like scaleb and scalef

[kgrabs]

Weird idea here, since mt_makediff and mt_adddiff now support working with 256< pixel values, would it be possible to make some sort of internal workaround for the limited range of diff clips by extending the range using a higher bitdepth? I figure if anything the math involved would be a nightmare, but I thought it'd be worth asking

Edit: nvm figured it out
bri=mt_lutxy(a,b,"x y < y x - 0 ?")
drk=mt_lutxy(a,b,"x y > x y - 0 ?")
mt_lutxyz(a,bri,drk,"x y + z -")

[pinterf]

isn't better to use something away from @ and # etc...? like scaleb and scalef
You and martin53 are a kind of powerusers of the new features, I appreciate your feedback, it is much easier to me to follow an advice than find things out of my mind without proper experience.

[martin53]

isn't better to use something away from @ and # etc...? like scaleb and scalef

It's not a surprise I confirm my above proposal.
Reason is 'scaleb' and 'scalef' do not carry any special character, i.e. they look like constants, but in fact they are a combination of a resolution-varying constant and a bitshift- or multiplication operation. Admittedly, 'sin', etc. also are functions and have non-operator-names.

scaleb and scalef are quite long tokens too (as are the other recently introduced constants), which is, in a formula, often not the preferred concept for readability (as opposed to text, formulas often need to be looked at as the whole thing / recursively, cannot just be read from start to end).

I admit @ is equivocal with the XOR operator and 'do a bitshift to range' is maybe semantically closer to '<<R' when compared with '<<s'. That's why I mentioned alternatives '<<<' (triple < can be understood somehow as 'special left shift', can't it?) and '^^^' (special scale up), which might also be '***' :D

So what about '<<r' and '*r', (r for range or resolution) which closest follows the 'OperatorLetter' principle of all the earlier &u ... >>s two step operations .