JPSDR Avisynth's plugins pack

[jpsdr]

Merge of Avisynth's plugins

Finaly, the All Might, One for All plugin is out, mergin all the avisynth's plugins i've made.

The main purpose of it is to reduce the number of threads created when you use more than one plugin, as in that case there is only one DLL file, so one threadpool created, instead of using several DLLs, creating several threadpools and so more threads.

Current version: 4.2.0

Merge of :
AutoYUY2: 4.1.12
NNEDI3 (http://forum.doom9.org/showthread.php?t=170083): 0.9.4.68
ResampleMT (http://forum.doom9.org/showthread.php?t=173772)/Desample (http://forum.doom9.org/showthread.php?p=1817097#post1817097): 2.10.0
aWarpSharpMT (http://forum.doom9.org/showthread.php?t=175064): 2.1.12
HDRTools (http://forum.doom9.org/showthread.php?t=175488): 1.2.3
JincResizeMT (http://forum.doom9.org/showthread.php?t=173772): 1.2.0

Sources are here (https://github.com/jpsdr/plugins_JPSDR).
Binaries are here (https://github.com/jpsdr/plugins_JPSDR/releases/download/4.2.0/plugins_JPSDR_v4_2_0.7z).

Version history
4.2.0 : Update of ResampleMT with AVX512 core code.
4.1.0 : Update of JincResizeMT, minor changes (more code refactory) on others filters.
4.0.0 : Add JincResizeMT, minor changes (more code refactory) on others filters.
3.9.0 : Revert added ASM on ResampleMT and some fixes, more ASM AVX512 on HDRTools.
3.8.0 : Add ASM AVX/AVX2/AVX512 on ResampleMT, add ASM AVX512 on HDRTools.
3.7.0 : Update NNEDI3 & HDRTools (add AVX512 code), update ResampleMT (update matrix class).
3.6.0 : Update resampler to new SSE code.
3.5.1 : Update resampler to new AVX2 code.
3.5.0 : Add resamplers last updates (chroma fix and filter parameters coeff new calcul).
3.4.0 : New AVS header and update on resampler and NNEDI3.
3.3.5 : Update to new AVS+ headers and fix awarp colorspace issue.
3.3.4 : Update of resample with DTL pull-request.
3.3.3 : Update to new AVS+ headers.
3.3.2 : Fix the value of a BT.1886 parameter in HDRTools.
3.3.1 : Update on threadpool, no user limit (except memory).
3.3.0 : Update of HDRTools with ACES tonemap.
3.2.8 : Fix on threadpool, using prefetch parameter created hang. Add negative prefetch for triming, read Multithreading.txt or Multithreading chapter here. Fix on resample.
3.2.7 : Fix on awarpsharp, fix on threadpool.
3.2.6 : Fix on resampler, fix on awarpsharp, new resamplers added in nnedi3_rpow2, update avs headers.
3.2.5 : Small fix on resampler and new kernel function.
3.2.4 : New resamplers and also added in nnedi3_rpow2 resizers.
3.2.3 : Fix for nnedi3 and new function in resample.
3.2.2 : Fix issue introduced in nnedi3.
3.2.1 : Fix on aWarpSharpMT, update to new avisynth headers.
3.2.0 : Update to HDRTools 0.6.0 (add of BT2446 A & C methods).
3.1.3 : Update HDRTools (add Crosstalk parameter and EOTF for SDR).
3.1.2 : Minor code change after threadpool update, fix in the number of threads,
fix in resampler to perfectly match avs+ output.
3.1.0 : Update in threadpool, add ThreadLevel parameter.
3.0.0 : Add of HDRTools.
2.2.0 : Update Matrix Class, fix bug in nnedi3 for YUY2, add 16 bits support on AutoYUY2.
2.1.1 : Optimized CPU placement if SetAffinity=true for prefetch>1, SetAffinity back to default false.
2.1.0 : 16 bits aWarpasharp, merge of new resample code, some fixes.
2.0.5 : Fix aWarp/aWarp4 default settings.
2.0.4 : Fix (good this once) crash on aBlur x64.
2.0.3 : Fix crash on aBlur and clarify some aWarp4 modes (aWarpSharpMT part).
2.0.2 : Fix on aWarpSharp and Resample.
2.0.1 : Fix on AutoYUY2.
2.0.0 : Add of aWarpSharpMT, small update on the others.
1.2.1 : Update to ResampleMT v2.0.1, fix in Desample functions.
1.2.0 : Update to ResampleMT v2.0.0 with Desample functions.
1.1.10 : Fix resample issue of doing nothing and fix possible deadlock in threadpool.
1.1.9 : Forgot to add AVX path on planarframe of NNEDI3.
1.1.8 : Fix threadpool, add AVX path in AutoYUY2 and NNEDI3.
1.1.7 : Same minor change an all filters.
1.1.6 : Minor update on threadpool and minor fix on NNEDI3.
1.1.5 : Minor update on threadpool.
1.1.4 : Fix YUYV planarframe NNEDI3 crash.
1.1.3 : Fix NNEDI3 x64 crash.
1.1.2 : Update of NNEDI3, and small fix.
1.1.1 : Update of all plugins (most significant is fix of range issue in ResampleMT).
1.1.0 : "Big" update of NNEDI3, and also update of AutoYUY2.
1.0.1 : Update of NNEDI3 and ResampleMT.
1.0.0 : First release.

For more informations on the changes, check each filter thread.

==================================================================

Multi-threading information

CPU example case : 4 cores with hyper-threading.

If you leave all the multi-threading parameters to their default value, it's set to be "optimal" when you're not using prefetch or if you are under standard avisynth, all the logical CPU will be used.
If you put SetAffinity to true it will allocate the threads on the CPU contiguously. Physical CPU 1 will have threads (0,1), ... physical CPU 4 will have threads (6,7), allowing optimal cache use. Make test to see what's best for you.

Now, if you are using prefetch on your script, things are different !
If you're using it with the max number of CPUs (8 in our exemple case), you still can make tests, but i would strongly advise to disable the internal multi-threading by using threads=1. In this case, there is no threadpool created, and all the other multi-threading related filter parameters have no effect, even prefetch.
If you're using prefetch on your script, with less than your CPU number, you may want to try to mix the external and internal mutli-threading, setting the internal multi-threading to a lower number of threads, and setting the prefetch parameter of the filter. This parameter will set the number of internal threadpool created, the best is to match the prefetch script value. If you don't set it (leave it to 1) or set a lower value than prefetch on your script, you'll have several instances (or GetFrame) created, but they'll not be running efficiently, because each instance (or GetFrame) will spend time waiting for a threadpool to be avaible, if not enough were created.
Unfortunately, as things are now, i have no way of knowing the prefetch value used in the avisynth script at the time i need the information, this is why you have to use the prefetch parameter in the filter.
In our CPU exemple case, you can have things like :

filter(...,threads=1)
prefetch(8)

or

filter(...,threads=2,prefetch=4)
prefetch(4)

or

filter(...,threads=4,prefetch=2)
prefetch(2)

or even

filter(...,threads=3,prefetch=4)
prefetch(4)

if you want to boost and go a little over your total CPU number.

Also, if your prefetch is not higher than your number of physical cores, you can try to put SetAffinity to true, but in that case, you have to set MaxPhysCore to false. The threads of each pool will be set on CPUs by steps.
For exemple, in our case :

filter(...,threads=2,prefetch=4,SetAffinity=true,MaxPhysCore=false)
prefetch(4)

Will create 4 pool of 2 threads, with the following :
pool[0] : threads(0 -> 1) on CPU 1.
pool[1] : threads(0 -> 1) on CPU 2.
pool[2] : threads(0 -> 1) on CPU 3.
pool[3] : threads(0 -> 1) on CPU 4.

filter(...,threads=4,prefetch=2,SetAffinity=true,MaxPhysCore=false)
prefetch(2)
Will create 2 pool of 4 threads, with the following :
pool[0] : threads(0 -> 1) on CPU 1.
pool[0] : threads(2 -> 3) on CPU 2.
pool[1] : threads(0 -> 1) on CPU 3.
pool[1] : threads(2 -> 3) on CPU 4.

Negative prefetch
The possibility to put negative prefecth to tune the prefetch parameter to optimal value has been added. The filter will throw an error if the number is not high enough to avoid waiting when requesting internal threadpool. For this to work properly, you have to put negative prefetch on ALL the filters of your script, and also ALL instances of the same filter.

Exemple :
filter(...,threads=2,prefetch=-2)
prefetch(2)
You'll see an error.

But with :
filter(...,threads=2,prefetch=-3)
prefetch(2)

You'll see no error, so the optimal is :
filter(...,threads=2,prefetch=3)
prefetch(2)

Once you've tune, put back a positive value.

[jpsdr]

New version.

[ChaosKing]

The main purpose of it is to reduce the number of threads created when you use more than one plugin, as in that case there is only one DLL file, so one threadpool created, instead of using several DLLs, creating several threadpools and so more threads.


That are the benefits, less ram consumption?

[jpsdr]

Exactly it's hard to say, but globaly it will use less resources for the same result.

[ChaosKing]

But only when I use (or load) all of the plugins above, right? Because I don't use the auto load feature, so I will not benefit from it at all?

[jpsdr]

If you're using more than one of my plugins in the same script (for exemple both nnedi3 and ResampleMT) you'll benefit.

[dipje]

thread pools? Really? I see 3 plugins merged in one.. So instead of 3 thread pools I now have one. A thread pool of like, 8 threads on a Core i7? Maybe twice that? so _maybe_ I save 16 to 32 threads or something? Of the 1800+ that are normally running on my system? :P.

That can't be the only reason / benefit of merging plugins, right?

[Groucho2004]

thread pools? Really? I see 3 plugins merged in one.. So instead of 3 thread pools I now have one. A thread pool of like, 8 threads on a Core i7? Maybe twice that? so _maybe_ I save 16 to 32 threads or something? Of the 1800+ that are normally running on my system? :P.

That can't be the only reason / benefit of merging plugins, right?
It would be interesting to see some comparisons, i.e. memory usage, speed, efficiency.

Generally, merging functions into one plugin is OK (see RGTools) but they should be in logical groups. jpsdr's merge seems a bit arbitrary.

[jpsdr]

That can't be the only reason / benefit of merging plugins, right?
In that case, it is. It's not a "function/theme merging" but a "resource merging".
If you have a 10 cores broadwell, each pool will have 20 threads, it can easely explode.
Maybe more noticeable when you're using a dedicated PC video where "nothing else" than avisynth is running... Having maybe less than 1800+ threads in that case.
Anyway, you don't want it, you don't have to take it. Isn't life beautifull ? ;)

[Groucho2004]

Of the 1800+ that are normally running on my system? :P.
Wow, 1800+ threads. I just checked on my system, 260 threads.
Task Manager:
https://s6.postimg.org/cfkdn5t35/Image2.png

[TheFluff]

there is nothing that stops you from sharing thread pools with other plugins over DLL borders, you know
actually doesn't avs+ have its own thread pool built in anyway?

[jpsdr]

there is nothing that stops you from sharing thread pools with other plugins over DLL borders,
Alreay tried, noticeable performance loss, little, but begin to be noticeable.


actually doesn't avs+ have its own thread pool built in anyway?
That will make the filter avs+ compatible only, not an option.

[jpsdr]

New version, see first post.

[jpsdr]

New version, see first post.

[jpsdr]

New version, see first post (nothing big this time).

[jpsdr]

New version (update of NNEDI3), see first post.

[jpsdr]

New version (update of NNEDI3), see first post.

[jpsdr]

New version, see first post (minor update).

[jpsdr]

New version, see first post (minor update).

[videoh]

Thank you, jp. Much appreciated!

[jpsdr]

New version, minor change, see first post.

[jpsdr]

New version, see first post.

[jpsdr]

New version, minor change, forgot to add AVX path on planarframe of NNEDI3.

[MysteryX]

I guess the main benefit is having the latest version of NNEDI3! It's hard to know what actually changed when the log says "big" and "small" changes.

btw what about fixing the issues corrected by the EDI_RPOW2 script (http://www.mediafire.com/file/pcmkxaauku375xd/edi_rpow2_v1.0.avsi)?

[burfadel]

NNEDI3 and the Resize MT are also updated and provided as separate packages. The real benefit of this package is relating to the threadpools when you are using both nnedi3 and the mt resize functions according to the first post.

[jpsdr]

btw what about fixing the issues corrected by the EDI_RPOW2 script (http://www.mediafire.com/file/pcmkxaauku375xd/edi_rpow2_v1.0.avsi)?

And what issues exactly this script is fixing ?
If it's the chroma shift issue of original nnedi3_rpow2, it's been fixed since the 0.9.4.11.

[hello_hello]

So specifying a resizer for the nnedi3_rpow2 cshift parameter hasn't achieved anything for quite a while?

[jpsdr]

No, it's just that it wasn't doing enough, it was missing a little something, almost unoticeable in 99% of the cases.

[jpsdr]

New version, see first post.

[FranceBB]

It scales perfectly on all my 4 cores 8 threads and works perfectly in XP x86.

Working flawlessly
Spline64ResizeMT(1280, 720, threads=8, logicalCores=true, MaxPhysCore=true, SetAffinity=true)

Working flawlessly
nnedi3_rpow2(cshift="Spline64ResizeMT", rfactor=2, fwidth=1920, fheight=1080, nsize=4, nns=4, qual=1, etype=0, pscrn=2, threads=0, csresize=true, mpeg2=true, threads_rs=0, logicalCores_rs=true, MaxPhysCore_rs=true, SetAffinity_rs=false, opt=3)

Working
Dither_convert_8_to_16()
nnedi3_resize16(target_width=1920, target_height=1080, mixed=true, thr=1.0, elast=1.5, nns=4, qual=2, etype=0, pscrn=4, threads=0, kernel_d="Spline", kernel_u="Spline", taps=6, f_d=1.0, f_u=2.0, sharp=0, lsb_in=true, lsb=true)
DitherPost()

Working flawlessly
AutoYUY2()


Release_XP SSE4 not working in XP as always (not a big deal).
Release_XP works like a charm.
It would be interesting to have AutoYUY2 working in other colour spaces, something like:
Converttoy8MT() #4:0:0
Converttoyv411MT() #4:1:1
Converttoyv12MT() #4:2:0
ConverttoYUY2MT() #4:2:2 interleaved (Actual AutoYUY2)
ConverttoYV16MT() #4:2:2 planar
ConverttoYV24MT() #4:4:4
ConverttoRGBMT() #RGB

[jpsdr]

AutoYUY2 is an old plugin from videoh, i was using an long time ago, and algorithm is realy dedicated to YV12 -> YV16 chroma upsampling. Even if there is a progressive mode, with progressive video, you'll get better results with actual resamplers. Also for the YV16->YV24, i would not be able to make better than actual resamplers.
Honestly, the only interest AutoYUY2 could have is YV12->YV16 on interlaced video, so, i personnaly don't intend to change anything on it.

[FranceBB]

Oh, ok, got it. Thank you for the explanation.

[jpsdr]

New version, significant update, see first post.

[jpsdr]

New version, fix in Desample.

[jpsdr]

New version, add of aWarpSharpMT.

[jpsdr]

New version, see first post.

[jpsdr]

New version, see first post.

[jpsdr]

New version, see first post.

[jpsdr]

New version, see first post.

[jpsdr]

New version, see first post.

[Atak_Snajpera]

This multithreaded resizing plugin is really good! Much better than plain Prefetch option in AviSynth+MT
CPU: E5-2690@2.9GHz (8C/16T)
Source: 3840x2160 YUV420P10

Crop(0,280,0,-280) + Spline36Resize(1920,800) + Prefetch(8)
http://i.cubeupload.com/Px4uU7.png

Crop(0,280,0,-280) + Spline36Resize(1920,800) + Prefetch(16)
http://i.cubeupload.com/Fr7VUV.png

Crop(0,280,0,-280) + Spline36ResizeMT(1920,800)
http://i.cubeupload.com/ZyL4dA.png

The fastest,lower memory consumption and cpu usage!

For comparison regular resizer.
Crop(0,280,0,-280) + Spline36Resize(1920,800)
http://i.cubeupload.com/i17UWO.png

[jpsdr]

New version, see first post, and i've also added on it a part about the multi-threading.

[Atak_Snajpera]

SetAffinity=true in latest version works terrible even without prefetch in script. Now it is slower than regular single threaded resizer!
http://i.cubeupload.com/FIc404.png

SetAffinity=false
http://i.cubeupload.com/Qa11Zu.png

BTW. I see that newer version is faster than old one (55 fps vs 52 fps)

[jpsdr]

What's the full script ?

[Atak_Snajpera]

#MT



#VideoSource
LoadPlugin("C:\Users\Dave\Documents\Delphi_Projects\RipBot264\_Compiled\Tools\AviSynth plugins\ffms\ffms_latest\x64\ffms2.dll")
video=FFVideoSource("E:\_Video_Samples\mkv\Passengers_2016_4K.mkv",cachefile = "C:\Temp\RipBot264temp\job1\Passengers_2016_4K.mkv.ffindex")
#Deinterlace



#Decimate



#Crop
video=Crop(video,0,280,-0,-280)



#Resize
LoadPlugin("C:\Users\Dave\Documents\Delphi_Projects\RipBot264\_Compiled\Tools\AviSynth plugins\Plugins_JPSDR\Plugins_JPSDR.dll")
video=Spline36ResizeMT(video,1920,800,SetAffinity=true).Sharpen(0.2)



#Levels



#Colours



#Denoise



#Custom



#Prefetch



#Subtitles



#AudioSource
Import("C:\Temp\RipBot264temp\job1\job1_a1.avs")


#Triming



#AVSameLength



#ColorSpace



#Return

[jpsdr]

What CPU do you have, more exactly how many logical cores do you have ? I just want to understand the 49 threads, but totaly expected if you have something like a 20 logical cores CPU.

[Atak_Snajpera]

https://forum.doom9.org/showthread.php?p=1831560#post1831560

[jpsdr]

Ok, there is something odd indeed, thanks reporting. All the other filters seems to behave properly, but the resampler runs only on one core with SetAffinity set to true, which is totaly unexpected. I can't investigate right now, but i will very shortly.
There is a bug somewhere...:sly:
aWarpsharp2 and nnedi3 give me 41 threads on my 20 cores CPU, in both cases true/false.
ResampleMT gives me 41 threads with true, 174 with false !!!
Yes, there is something realy wrong...

[jpsdr]

More fishy !! I'm on my break lunch and made some tests on my PC a work, and everything works fine, but i don't have the same CPU than i have at home (it's a simple 4 cores without HT). The only thing i can't check for now is Intel vs VS. Have you made your tests with VS or Intel version ? If Intel, can you make a test with the VS version ? I'll try also this when back home, but it will not be before several hours.

Edit :
Sometimes i'm very stupid, of course i can test, i just have to download them from my github... :p

Results :
The VS AVX and Intel AVX2 versions work fine with standard avisynth.

The VS AVX version works fine with avs+ (both x86 & x64).
The Intel AVX2 version is working... "fishy" with avs+ (both x86 & x64), but only for the resampler, the other filters work fine.

I'll update the release files on github, removing the Intel versions, and keeping only VS version, and adding an VS AVX2 version. Wait at least 24h to check/re-download the files.

[jpsdr]

... Before totaly removing, i'll check if with /O2 instead of /O3 with the Intel compiler, there is still the issue.