MaskTools2 - pfmod

[Andouille]

Microsoft will no longer provide security updates or technical support for the Windows XP operating system

And ? Video processing fails for that reason ?

[Morku]

And ? Video processing fails for that reason ?

Feel free to use outdated software for an outdated OS. No need for current software or fix compatibility yourself.

TheFluff already said everything.

[Andouille]

Feel free to use outdated software for an outdated OS or fix compatibility yourself.

TheFluff already said everything.

Define "outdated OS" for video processing.

[Groucho2004]

Not this again. :devil:

If one wants to keep using XP one has to live with the fact that more and more software won't work on it. It's as simple as that.

[Myrsloik]

This can help: According to statcounter's "Desktop Windows Version Market Share Worldwide (http://gs.statcounter.com/windows-version-market-share/desktop/worldwide#monthly-201701-201801)":

Windows XP market share dropped from almost 6% to about 3% during the last year.
Windows XP market share has consistently lost more than 3% market share yearly since January 2015, three years ago, when it had almost 14%.
Windows XP’s big brother, windows 2003, disappeared from those statistics in January 2015, three years ago.

Support for Windows XP has ended (https://www.microsoft.com/en-us/WindowsForBusiness/end-of-xp-support) almost 4 years ago:
After 12 years, support for Windows XP ended April 8, 2014. Microsoft will no longer provide security updates or technical support for the Windows XP operating system

All those 3% hang out on doom9 so your argument is invalid (locally) :D

[TheFluff]

Define "outdated OS" for video processing.

You know, this really is an exceptionally poorly thought out argument to make when a very annoying and hard-to-debug problem caused by attempting to use Windows XP's outdated multithreading support was described like three posts above yours.

More generally speaking though, it's not so much that the OS itself is badly suited to video processing (it definitely is though, but in indirect ways) - it's more that developing software for it is becoming harder because development tools are dropping support for it and some kernel-level API's (such as the aforementioned multithreading stuff) aren't available on it. Sure, you could go out of your way to implement replacement functionality for it yourself, but why would you go to all that effort for something used by like three people? It really isn't reasonable to demand that from a hobbyist developer. It's open source, fix it yourself if it's that important to you. Or use older versions.

[Andouille]

My point is who cares about "security" when processing audio/video ?
We are not speaking about internet browsing or piracy sites.

[Andouille]

@TheFluff
My MT avisynth is just as stable as a rock.
I don't even ask new plugins.
The only thing would be source plugin. (Thx Quyot)

[TheFluff]

@TheFluff
My MT avisynth is just as stable as a rock.
I don't even ask new plugins.
The only thing would be source plugin. (Thx Quyot)

why are you here hurfing durf about it then

[Andouille]

why are you here hurfing durf about it then

My hair is fluffy.

[real.finder]

That stupid crash took me weeks to catch until it turned out that a mysterious access violation is happening right in masktools2.

The original report was: ffdshow x64 crashed with a script using maa2. I have almost completely un-mmx'd ffdshow, and moved it to avs 2.6 interface from v2.5, changed things that existed in boost many years ago but now C++14/17 adopted them, so it finally could be compiled and debugged with VS2017 (with some features disabled). But the crash was still there, pointing to avisynth or one of its filters. Then I succeeded to reproduce the crash w/o ffdshow with a simple script:
c=BlankClip(width=620,height=600,length=1000,pixel_type="YV12")
mt_merge(c,c,c,y=-2,u=-2,v=-2,luma=true)
#Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
#Module: C:\Windows\System32\KernelBase.dll
#Address: 0x00007FF9A5313FB8
Prefetch(2)


It was the side effect of the /Zthreadsafeinit- switch. This option is used for keeping stupid XP compatibility and caused a static class to be initialized later in one thread than it was used from another thread. Specifically in MT environment GetFrame(0) started the initializing some variables but GetFrame(1) from another thread kicked in too early and couldn't yet see the changes that the first thread was doing.

Of course all this happened only in Release mode because it needed strict timing conditions, maybe sub-milliseconds so it could not be debugged. When will XP die????

so it was bug in masktools after all, about 1 or 2 month ryrynz told me that there crash happen with maa2 in avs+ 64 if using mt

if only x64 has this bug then you can let xp compatibility die for x86_64 and keep it only x86 in 32bit

people that use x64 pc should not use xp anyway since their pc can run another more modern os

[manolito]

For the market share stats it really depends where you look...
According to netmarketshare the numbers are a little different:
https://www.netmarketshare.com/operating-system-market-share.aspx?options=%7B%22filter%22%3A%7B%22%24and%22%3A%5B%7B%22deviceType%22%3A%7B%22%24in%22%3A%5B%22Desktop%2Flaptop%22%5D%7D%7D%5D%7D%2C%22dateLabel%22%3A%22Trend%22%2C%22attributes%22%3A%22share%22%2C%22group%22%3A%22platformVersion%22%2C%22sort%22%3A%7B%22share%22%3A-1%7D%2C%22id%22%3A%22platformsDesktopVersions%22%2C%22dateInterval%22%3A%22Monthly%22%2C%22dateStart%22%3A%222017-02%22%2C%22dateEnd%22%3A%222018-01%22%2C%22segments%22%3A%22-1000%22%7D

But whatever, who cares if XP has a market share of 3% or 7%? Fact is that it still is #3 (maybe #4) of all desktop OSs.


I would like to make a different point. As someone who always has 2 machines running in parallel, one ancient desktop on XP and a newer Thinkpad T530 on Win7-64, I do know the differences. And I do know why I prefer XP... :devil:

Having read all the stuff here at Doom9 about AVS+ and all the plugin modernization efforts, I can only say that most of this stuff is work in progress, it is far from stable. Mostly this refers to Pnterf's stuff (AVS+, MVTools2, MaskTools2 and more). I commend him for doing all the work he did, but the results often look like he updated stuff when he really did not understand the original code.

After playing a lot with the "modernized" plugins on my Win7 machine, I finally reverted to the old and stable tools (which also work under XP). Sure they are slower and do not offer "modern" features like high bit depth and advanced color spaces. But THEY JUST WORK. And I use these tools for "real" conversions, I do not have the time to chase down bugs.


Cheers
manolito

[VS_Fan]

After playing a lot with the "modernized" plugins on my Win7 machine, I finally reverted to the old and stable tools (which also work under XP). Sure they are slower and do not offer "modern" features like high bit depth and advanced color spaces. But THEY JUST WORK. And I use these tools for "real" conversions, I do not have the time to chase down bugs.
That statement makes one think you don't even use MT and, more importantly, you didn’t understand pinterf’s question:
New build ...
- Fix: rare crash in multithreading environment at the very first frames
(keeping XP compatibility with /Z:threadsafeinit- caused troubles!)...
That stupid crash took me weeks to catch until it turned out that a mysterious access violation is happening right in masktools2.
... When will XP die????

[manolito]

That statement makes one think you don't even use MT

You are absolutely correct:
I use plain vanilla AviSynth 2.60 which is 32bit and single threaded. And I plan to stick with it for the time being. My encoding bottle neck is not AVS filters, it is the encoder.


Cheers
manolito

[burfadel]

Can I just confirm the behaviour of mt_lut with different bit depths? I'll give some example screenshots of what is happening (including with 2.2.13). It's a random video frame.

Source, 8 bit:
https://i.imgur.com/8GxF68D.jpg

mt_lut with 8 bit:
https://i.imgur.com/obqEzeB.jpg

mt_lut 10 bit by:
convertbits(10)
mt_lut ( "x "+string(32)+" < 0 x "+string(45)+" > "+string(255)+" 0 x "+string(35)+" - "+string(255)+" "+string(32)+" "+string(65)+" - / * - ? ?" )
convertbits(8)
https://i.imgur.com/nihtUVR.jpg

Same as above regarding convertbits but with paramscale (slightly different frame of scame clip, but it's the same effect right through the video as expected:
mt_lut ( "x "+string(32)+" < 0 x "+string(45)+" > "+string(255)+" 0 x "+string(35)+" - "+string(255)+" "+string(32)+" "+string(65)+" - / * - ? ?", paramscale="i8")
https://i.imgur.com/M72wif0.jpg

Now, here's the same frame as the first images but using mt_binarize in 10 bit:
convertbits(10)
mt_binarize(64)
convertbits(8)
https://i.imgur.com/JlDJxcr.jpg

So, regardless of anything, mt_lut does not scale to the 8-bit values when not under 8 bit, so the mask wouldn't work. However, mt_binarize does automatically scale to the 8 bit values, so the 64 works.

Is that correct behaviour of mt_lut in comparison to mt_binarize?

[pinterf]

Can I just confirm the behaviour of mt_lut with different bit depths?
[...]
mt_lut ( "x "+string(32)+" < 0 x "+string(45)+" > "+string(255)+" 0 x "+string(35)+" - "+string(255)+" "+string(32)+" "+string(65)+" - / * - ? ?", paramscale="i8")

So, regardless of anything, mt_lut does not scale to the 8-bit values when not under 8 bit, so the mask wouldn't work. However, mt_binarize does automatically scale to the 8 bit values, so the 64 works.

Is that correct behaviour of mt_lut in comparison to mt_binarize?
Expression string is not an additional parameter which is affected by paramscale in any way. Unlike the threshold parameter in binarize. A filter can see nothing from your string contacenating, it will see only the assembled result. String(number) is not a parameter, just a part of a string expression which is resolved by Avisynth during script parsing.

Try this one:
mt_lut ( "x 32 scaleb < 0 x 45 scaleb > range_max 0 x 35 scaleb - range_max 32 scaleb 65 scaleb - / * - ? ?")

[FranceBB]

@VS_Fan... multi-threading in Avisynth totally depends on how the coder wrote the filter, 'cause Avisynth itself is single thread.
In other words: whoever is writing a plugin should implement multi-threading, otherwise it's gonna be single-thread.
MT mode it's a way to speed things up by running multiple instances of single-thread filters, but it's not perfect, it doesn't work with every single filter and should not be used with multi-threading filters, 'cause it would lower performances or worse (make AVS crash).

When will XP die?
I'm using Windows Server 2016 on my server and Linux on my home desktop, but I'm also running Windows XP Professional x86 on a VM and I'm planning to keep it 'till the end of extended support (April 2019).
Anyway, I understand that supporting XP might require some additional work, but I think it would be really useful to keep XP compatibility.
And, by the way, I really appreciate what you have done in these years; I really do. A big fat thank you ^_^

@manolito... I'm using standard Avisynth too and I admit that although it's not bullet proof, it's actually very reliable.

[StainlessS]

Whether tis 3% or 7% or some other, that is one helluva lot of machines, and will likely be slanted due to many XP m/c's being used offline,
just because you cannot see/count them, dont mean that they dont exist.

[feisty2]

I'm just curious, what's so special about windows xp? there're many windows operating systems predating windows xp, how come some people could upgrade from those even earlier operating systems to xp and then just stopped, why aren't they still using stuff like DOS, oh no, why are they using computers and electricity?

[manolito]

Because Windows did indeed improve up to XP (with the exception of the Millennium version which came out after Win98SE). After WinXP everything went downhill. Win8 and Win8.1 was a joke (at least for users without touchscreens), and Win10 is an absolute nightmare for me. Leaves Win7 which is not too bad, but only after tweaking dozens of things to make it resemble XP. This includes turning off this stupid UAC, tame this whole permissions thing, disable Virtual Store (I want to know where my files REALLY are stored). Also getting rid of the built-in ZIP support, installing Classic Startup Menu, make Windows use the DS source filters which I have installed instead of the crappy MS filters, and some more...

After applying all these tweaks I can live with Win7. But MS is actively working to put Win7 into the position WinXP is in now. Just read that their upcoming Office 2019 will need Win10. Once they succeed in making Win7 unusable I will certainly say goodbye to MS. I hope that ReactOS will be ready by then...

[feisty2]

It's ur personal preference, u personally can't let go of the older windows user interface and that doesn't mean the new windows user interface is any worse than the older one, a personal preference is not a valid point to judge the good and bad of an operating system, btw the default start menu in xp is also different from previous windows versions, and I bet user like u surely won't take that default start menu in xp, u have to make that 98 style :)

[manolito]

and I bet user like u surely won't take that default start menu in xp, u have to make that 98 style :)

How in the world did you know that? :devil:

I even have Win98SE still installed on a separate partition because I can't live without the old Delrina Winfax (newer fax software cannot touch it, and at least in Germany the legal situation requires oftentimes to send a fax instead of email).


Cheers
manolito

[TheFluff]

@VS_Fan... multi-threading in Avisynth totally depends on how the coder wrote the filter, 'cause Avisynth itself is single thread.
In other words: whoever is writing a plugin should implement multi-threading, otherwise it's gonna be single-thread.
MT mode it's a way to speed things up by running multiple instances of single-thread filters, but it's not perfect, it doesn't work with every single filter and should not be used with multi-threading filters, 'cause it would lower performances or worse (make AVS crash).
Explaining things you don't actually understand as if you were an authority on the subject to someone who does understand it is generally regarded as rather rude. Your understanding is roughly kinda sorta correct for how Avs-MT worked circa 2010, but not now.

[StainlessS]

Nuttin bettter than WXP with all of that crap turned off, ie in W95/98 mode,
None of them daft transluscent blue menus to appreciate, hit a menu, it apprears, we dont have to 'appreciate it.

Pinterf, if you choose to abandon, I can accept and understand that, so can all others that live in the past, if that is what you want, go for it,

Of course I would personally be disapointed, but, who cares, shit on me, and Well I dont blame you.

Do what you will, to us lowely scoudrels. :) [EDIT: I'm quite sure that we will get by.]

EDIT: My fav was w2K, now that was a blast.(I hated XP, my hate got bigger and shifted focus with appearance of Vista).

[real.finder]

I'm just curious, what's so special about windows xp? there're many windows operating systems predating windows xp, how come some people could upgrade from those even earlier operating systems to xp and then just stopped, why aren't they still using stuff like DOS, oh no, why are they using computers and electricity?

well, many people love xp, included me, in 2003 my dad Pentium 4 pc was have 2 os, 1st Windows ME and 2nd win xp, since back then some people still use win me for some reason, but I used to use xp, and still sometimes in VM nowadays, it's generally nice os

https://www.youtube.com/watch?v=UZas45Zn9zU

[sneaker_ger]

The wiki still says VC2015 but for the new release you need VC2017, correct? Can someone with access (and knowledge) update it?

http://avisynth.nl/index.php/MaskTools2#Download

[raffriff42]

readme for version 2.2.12 says "Visual Studio 2015 Update 3"
Confirmed with Dependency Walker (http://www.dependencywalker.com/).

EDIT sorry, somehow I missed the new release 2.2.13

[sneaker_ger]

New build
[...]
so it finally could be compiled and debugged with VS2017 (with some features disabled).
:confused:

[pinterf]

Visual Studio 2015 and 2017 redistributables are interchangeable, both versions are 14.xx.
For example when you install Visual Studio 2017 Redistributables, it will remove VS2015 version.
@raffriff42: I missed editing that section. At the top of the page I have modified the text to "Requires Visual Studio redistributables (14.xx family)"
https://docs.microsoft.com/en-us/cpp/porting/binary-compat-2015-2017

[ajp_anton]

function testing(clip c,bool info)
{ ... }

sourcefilter
testing(info=true)

error: invalid arguments.
Works if I make the argument "info" optional, or skip the argument name when calling the function.

[StainlessS]

function testing(clip c,bool info)
{ ... }

sourcefilter
testing(info=true)

error: invalid arguments.
Works if I make the argument "info" optional, or skip the argument name when calling the function.

It has always worked that way, non optionals have to be provided in given order and without naming, just giving in function prototype in quotes will accept as optional, but, must use eg x=Default(x,true) internally, else crash when not supplied.

[ajp_anton]

It has always worked that way, non optionals have to be provided in given order and without naming, just giving function prototype in quotes will accept as optional, but, must use eg x=Default(x,true) internally, else crash when not supplied.
May I ask why? Is it intentionally left this way, or can/will it be changed?

And I just realized I posted this in the wrong topic (thought I was in the avs+ topic...). A mod can move these posts if they want.

[StainlessS]

Is it intentionally left this way, or can/will it be changed

1) yes.
2) unlikely.

Non optionals are compulsory, and in compulsory order, so apart from anything else, names in those cases are superfluous.

EDIT: Names are used so as to accept optional defaults, and specify out-of-order optionals where is convienient.

[pinterf]

Fix mt_convolution 10-16 bit mode for SSE2/SSSE3-only processors

Download masktools2 v2.2.14 (https://github.com/pinterf/masktools/releases/tag/2.2.14)

Changelog
**v2.2.14 (20180225)
- Fix: mt_convolution invalid instruction on processors below SSE4.1


Thanks for zub35 for the report

[pinterf]

New release.
Download masktools2 v2.2.16 (https://github.com/pinterf/masktools/releases/tag/2.2.16)

This version is partly an update for handling 32 bit float the same way as Avisynth+ r2728 does.

And there are some quite significant new features.

Special thanks to real.finder for his ideas and testing.
I hope a couple of scripts will benefit from the changes, both quality and speedwise.

Changelog
**v2.2.16 (20180702)
- mt_merge new parameter hint for chroma placement when luma=true and 4:2:0/4:2:2
String 'cplace': possible values "mpeg1" or "mpeg2" (default)
Parameter is effective only for 420 and 422 formats, otherwise ignored.
Default "mpeg1" is using fast 2x2 pixel (1x2 for 4:2:2) averaging when converting a 4:4:4 mask to a 4:2:0 or 4:2:2 format (old behaviour)
420 schema:
+------+------+
| 0.25 | 0.25 |
|------+------|
| 0.25 | 0.25 |
+------+------+

"mpeg2" is using 2x3 (1x3 for 4:2:2) pixel weighted averaging when converting a 4:4:4 mask to a 4:2:0 or 4:2:2 format
420 schema:
------+------+-------+
0.125 | 0.25 | 0.125 |
------|------+-------|
0.125 | 0.25 | 0.125 |
------+------+-------+

- 32 bit float U and V chroma channels are now zero based (+/-0.5 for full scale). Was: 0..1, same as luma
Since internal format changed, use Avisynth+ r2724 or newer for this masktools2 2.2.16.
Affected predefined expression constants when plane is U or V:
cmin and cmax (limited range (16-128)/255 and (240-128)/255 instead of 16/255.0 and 240/255.0
range_max: 0.5 instead of 1.0
new: introduce range_min: -0.5 for float U/V chroma, 0 otherwise
range_half (0.0 instead of 0.5)
(range_size remained 1.0)
- New expression syntax for Lut expressions: autoscale any input (x,y,z,a) bit depths to 8-16 bits for internal
expression use. The primary reason of this feature is the "easy" usage of formerly written 8 bit optimized expressions.

New parameters for lut functions:
String "scale_inputs": "all","allf","int","intf","float","floatf","none", default "none"
and
Boolean "clamp_float": default false, but treated as always true (and thus ignored) when scale_inputs involves a float autoscale.
and
Boolean "use_expr": default 0, calls fast JIT-compiled "Expr" in Avisynth+ for mt_lut, lutxy, lutxyz, lutxyza
0: no Expr, use slow internal realtime calc if needed (as before)
1: call Expr for bits>8 or lutxyza
2: call Expr, when masktools would do its slow realtime calc (see 'realtime' column in the table above)

Extends and replaces experimental clamp_xxxx keywords.

[FranceBB]

Thank you for the new release! :D

[real.finder]

seems both use_expr=1 and use_expr=2 are same (both use_expr=1)

[pinterf]

Decisions for a single 1D mt_lut:
use_expr=1 (when bit depth is more than 8 bits): pass expression to Avs+ Expr filter for 10 or more bits
use_expr=2 (when no LUT is used) this is only for 32 bit float. 8-16 bit uses LUT, for 1D it has enough memory for the table.

Checked for a 16 bit clip.

For short expressions such as a simple "x 2 *" passing it to Expr is even faster than LUT.
For more complex ones 1D LUT is still faster, it depends.

[real.finder]

Decisions for a single 1D mt_lut:
use_expr=1 (when bit depth is more than 8 bits): pass expression to Avs+ Expr filter for 10 or more bits
use_expr=2 (when no LUT is used) this is only for 32 bit float. 8-16 bit uses LUT, for 1D it has enough memory for the table.

Checked for a 16 bit clip.

For short expressions such as a simple "x 2 *" passing it to Expr is even faster than LUT.
For more complex ones 1D LUT is still faster, it depends.

isn't use_expr=2 for use expr all times (8-32 bits) in avs+? even so I note use_expr=2 work with 16 bit now! (at least in the scripts I did edited) and option that use expr all times is needed if it not existing

[pinterf]

isn't use_expr=2 for use expr all times (8-32 bits) in avs+? even so I note use_expr=2 work with 16 bit now! (at least in the scripts I did edited) and option that use expr all times is needed if it not existing

Readme* says:
Possible values:
0: uses lut and internal realtime calculation
1: Expr, when bit depth>=10 or lutxyza
2: When masktools would use realtime calc

E.g. for use_expr=2:
8-16 bit lut is still LUT
8-12 bit lutxy is passed to Expr
14-16 and 32 bit lutxy is passed to Expr

E.g. for use_expr=1:
8 bit lut is still LUT
10-16 and 32 bit lut is Expr
8 bit lutxy is still LUT
10-16 and 32 bit lutxy is passed to Expr

The "feature matrix" in readme shows 'realcalc' bit depth limits for th different lut versions. When realcalc working mode kicks in, Expr is used instead for use_expr=2

*Masktools2 page on avisynth.nl is not yet updated

[real.finder]

I see

so what about use_expr 1 will be When masktools would use realtime calc

2 will be when more than 8 bit

3 always use expr in avs+

edit: or 1 mean always and 2 mean more than 8 and 3 mean if realtime

[pinterf]

What is the point of "always"? Use then Expr directly. It should be even a little faster because Expr is now Invoked and compiled from each GetFrame and not only once, during the filter construction.

[real.finder]

What is the point of "always"? Use then Expr directly. It should be even a little faster because Expr is now Invoked and compiled from each GetFrame and not only once, during the filter construction.

the point is make less lines in scripts, since there are no expr() in normal avs

[pinterf]

And pass even an 8 bit 1D lut to Expr? (I have started updating Expr in Avs+ to be able to work like LUT which is faster for some 1D or 2D cases, but had no time to continue and finish atm.)

[real.finder]

And pass even an 8 bit 1D lut to Expr? (I have started updating Expr in Avs+ to be able to work like LUT which is faster for some 1D or 2D cases, but had no time to continue and finish atm.)

if it faster (Especially with avx2) or even less ram then why not?

and it will make things easier for testing too

[pinterf]

if it faster (Especially with avx2) or even less ram then why not?

and it will make things easier for testing too
It's not perfect :)

[pinterf]

New version, download masktools2 2.2.17 (https://github.com/pinterf/masktools/releases/tag/2.2.17)

**v2.2.17 (20180710)
- mt_convolution: check plane dimensions to exceed convolution horizontal/vertical size
- lut functions: plane order to RGBA from BGRA like in Expr.
expr parameters y-u-v-a naming matches now to r-g-b-a
- for luts: use_expr=3: always send expression(s) to Expr

[pinterf]

New version, download masktools2 2.2.18 (https://github.com/pinterf/masktools/releases/tag/2.2.18)

Thanks tyaniw for the report

**v2.2.18 (20180905)
- mt_merge: fix right side artifacts for non-mod16 width, AVX2 and luma=false (regression in 2.2.16)
- mt_adddiff: fix 32 bit chroma (still used 0.5 centered chroma instead of 0.0)

[FranceBB]

Thank you for the new version.
Works fine on Windows XP. ^_^

[Tempter57]

pinterf
Hi. Where has disappeared in the last versions 2.2.17 & 2.2.18 mt_convolution? Some scripts, for example, BalanceBorders.avs, which use mt_convolution, produce an error