JPSDR Avisynth's plugins pack

[guest]

New version, see first post.

So which one do I use for Windows 10, with some cpu's that don't support AVX ?? (eg:- Xeon X5690's)

Also, using RipBot264.

Cheers

[kedautinh12]

If your card don't support avx, avx2. You can use w7 ver

[guest]

If your card don't support avx, avx2. You can use w7 ver

Thanks for the info...but don't you mean cpu, not card ??

Cheers

[kedautinh12]

Sr, it's my mistake

[guest]

Sr, it's my mistake

No worries.

I just wanted to say thanks, your suggestion of which build to use, has solved the problem I was having with RipBot264.

I must have used the wrong one, when I updated this plugin, but since then there's been a newer build :)

Cheers

[jpsdr]

New version, see first post.

[FranceBB]

What a sweet surprise! Thanks! :D

[FranceBB]

Can NNEDI3 use all the cores with its multithreading up to 56c/112th?
Has Jean Philippe made the plugin_JPSDR in a way that it's able to scale enough with its thread pool?
Yes, yes it can! And yes, yes he has!! :D

Image Link (https://i.imgur.com/7EZeXKm.png)

(disclaimer: it's running through a service, triggered by a remote API, which is why you don't see the command line prompt in the picture)

[jpsdr]

It can...????
Ah yes... I've defined 128 as maximum possibles threads in my threadpool code.
The issue is in the "processor mask" part for eventualy attach the threads to processor given by system function is a 32 bits data, so, no idea how it behaves with more than 32 cores...
So, stay with standard default (false) setting for SetAffinity.

Maybe not finaly, i don't remember the size of ULONG_PTR...

I'll have to update my threadpool code, i never thought i see so much core, and if there is too much core (>64), bad things will happend, so, an update is needed before someone begin to test with too much cores... :D

[FranceBB]

It can...????


Eheheh I was surprised too xD



Ah yes... I've defined 128 as maximum possibles threads in my threadpool code.


Cool :D



So, stay with standard default (false) setting for SetAffinity.


Roger that.


I'll have to update my threadpool code, i never thought i see so many cores, and if there are too many cores (>64), bad things will happen, so, an update is needed before someone begin to test with too much cores... :D

Well, I'm afraid I don't have a 64c/128th Xeon even if I work at Sky; the best I can do for tests is 56c/112th, which is pretty remarkable anyway I think xD

[Atak_Snajpera]

Do you plan to add AMD FSR upscaling algorithm. IT is now open source.

[DTL]

AMD FSR is looks for creating more sharper edges for 3D-rendered content. https://gpuopen.com/fidelityfx-superresolution/ It looks GPU-dependent and even more - exact 3D-rendering engine dependent. Can it shows some tests on typical video content and with open-source example on CPU only ? Currently it looks not like open source universal programming-language algorithm but some open API for 3D GPU and dependent on other 3D API.
At 3D samples on that site it looks it may be dependent on 3D mesh and textures data non-existent in video input data for upscaling.

[Atak_Snajpera]

FSR is just another spatial upscaler. It just takes 2D image and upscales to higher resolution just like neddi. It does not require anything from game engine like DLSS.

FidelityFX Super Resolution is a spatial upscaler: it works by taking the current anti-aliased frame and upscaling it to display resolution without relying on other data such as frame history or motion vectors.

https://www.purepc.pl/image/news/2021/07/16_amd_fidelityfx_super_resolution_zostalo_upublicznione_dostep_do_techniki_bedzie_jeszcze_latwiejszy_wkrotce_wsparcie_w_unity_1_b.jpg

[DTL]

Best candidate for adding to Avisynth is software-only platform-independent open-source with appropriate licensing. And inside frame MultiThreading-friendly. If it is based of hardware only shaders execution on some AMD GPU products - it is unlikely.

[Atak_Snajpera]

Best candidate for adding to Avisynth is software-only platform-independent open-source with appropriate licensing. And inside frame MultiThreading-friendly. If it is based of hardware only shaders execution on some AMD GPU products - it is unlikely.

What are you talking about? It works on any GPU with FP32 units.

https://www.purepc.pl/image/news/2021/07/16_amd_fidelityfx_super_resolution_zostalo_upublicznione_dostep_do_techniki_bedzie_jeszcze_latwiejszy_wkrotce_wsparcie_w_unity_3_b.jpg

Besides CAS was also designed for GPU but CPU port for avisynth is available from some time
https://github.com/Asd-g/AviSynth-CAS

[jpsdr]

... Can you create a specific topic discussion instead of here...?

[jpsdr]

New version, see first post.

[FranceBB]

Thanks for the new version. :)

[subterrestrial]

Thank you. Now the crash problem in AVS+ 3.70 cuda version which occured when "SetDeviceOpt(DEV_CUDA_PINNED_HOST)" was put in script is solved.

[jpsdr]

Good to know. Without someone with a proper configuration (who can produce the crash) to confirm the crash doesn't occur anymore, i wasn't 100% sure.

[DTL]

Can you look at idea of linear UV resampling at RGB<->YUV_subsampled processing in https://forum.doom9.org/showthread.php?p=1950287#post1950287 and may be add this as functions to HDRtools or ResampleMT ?

As I understand currently ConvertYUVtoLinearRGB and backward automatically uses chroma upsample if 4:2:0 or 4:2:2 provided as YUV source. So may be add method of linear UV scaling instead of OETFed for ConvertYUVtoLinearRGB and backward. As an option to process.

Current scripting form may work significally slower. The demonstrated in scripting workflow uses both ResampleMT and HDRtools plugins.

[jpsdr]

Doing something like NNEDI3 is a total pain in the ***, it will add dozen of parameters, and will not be faster, because it will just call external resampler functions the exact same way your script is doing.
And it you're using more than one of my DLL, instead of loading each DLL for each filter, use the plugin pack. The way you're using, each DLL will create a threapool, which is a waste of resources.
Replace LoadPlugin("ResampleMT.dll")
LoadPlugin("HDRTools.dll") with LoadPlugin("plugins_JPSDR.dll")

[DTL]

The idea of resampling UV components in linear domain is not to be faster but to be less distortive. It is for perfectionists mostly and definitely will be more slow. It require much more matrix and OETF<->Linear transformations to allow recreation of UV planes in full size (equal to size of Y component) for last YUV->RGB matrix with less distortions. Yes - the decoder of 4:2:0 and 4:2:2 in this method is much more complex in compare with typical 'fast' approach. Though it is full-deterministic 'linear' processing without content/scene dependent like NNEDI.

I think the compiled form will be faster because eliminating 'CombinePlane' planes shuffling as separate function. Or it is very fast in compare with ConvertTo/FromLinear ? So it is better to form this processing as a script functions and put to some script library ?

[Dogway]

I had a look at this 2 months ago, there are several approaches like improved YUV conversions like YUVr (YUV RCT) or sYUV (Sharp YUV). I actually tried the one in this paper (https://norkin.org/pdf/SPIE_2016_HDR_conversion_metrics.pdf) but it looks like geared towards HDR, your approach seems similar. I will check it back again when working with TransformsPack.

[jpsdr]

@DTL
I think you miss my point. By not being faster i mean compiled form will realy not be faster than a script.
In NNEDI3 you can specify the resampler, but doing this add a lot of parameters to the filter, that's just what i meant.
Either HDRTools calls resampling functions, or the other way round. Anyway, it will be done by combining planes and calling external functions with Invoke anyway.
The script is the perfect solution, you can change the resamplers and their parameters, have different resampler for /2 and x2 if you want, change things easely, etc... instead of one filter with 100+ parameters...!
The filters allready have on their own a lot of parameters.

[FranceBB]

Question: would it be possible to make plugins_JPSDR numa nodes aware too? In other words, instead of using 1 CPU only in a multi socket configuration, allow the filters and their thread pool to make use of both CPUs in a multi socket environment.
The reason why I'm asking you this is this:

https://i.imgur.com/34NR70s.png

[kedautinh12]

Question: would it be possible to make plugins_JPSDR numa nodes aware too? In other words, instead of using 1 CPU only in a multi socket configuration, allow the filters and their thread pool to make use of both CPUs in a multi socket environment.
The reason why I'm asking you this is this:

https://i.imgur.com/34NR70s.png

Wow, we have rich boy here :D

[FranceBB]

Wow, we have rich boy here :D

Please, it's not properly "mine". It's a Sky (i.e Comcast) Avisynth Server ehehehehehehe

(check the picture, they're next to me)

https://i.imgur.com/6bfR1Vg.png

[jpsdr]

Sorry, no idea... In fact, i don't even know what this numa node is... :(
If you don't change SetAffinity, the OS will choose itself where to run threads.
But... Once you show us an almost 100% on 112 logical cores with NNEDI3, what changed ?
Anyway, you can try to set threads to 112.

[FranceBB]

Sorry, no idea... In fact, i don't even know what this numa node is... :(


Ah... :(


But... Once you show us an almost 100% on 112 logical cores with NNEDI3, what changed ?


I was setting threads to 112 manually.


Anyway, you can try to set threads to 112.


Yeah, that's what I used to do, however the cache is not shared between the two CPUs, so what happened was that it was indeed using 100% but the performance could be much higher if the program was numa aware.
Have you ever worked with dual socket motherboards?
So basically you have two independent CPUs and the OS assigns tasks to either one or the other CPU, however if the program is numa aware, basically it can make use of both CPUs at the same time. If you force a program to spawn across the two CPUs (so you set more threads than CPU cores of a single CPU), it will make use of the other CPU, but everything will be handled by the OS and the performance will be sub-par, especially during the software syncing part done by the OS.
If you're willing to look into this I can let you play with one of those 5 server.

[DTL]

May be it better to run 2 separated processes with 112/2 threads each. And assign each process to each node. And process 2 files instead of 1.

[FranceBB]

May be it better to run 2 separated processes with 112/2 threads each. And assign each process to each node. And process 2 files instead of 1.

That's also a possibility, but I'd still love to have the filters numa node aware for those times in which I need to encode a single file and I have a short deadline.

[ChaosKing]

Split clip in 2 parts and encode to a lossless video format first. Easy peasy.

[wonkey_monkey]

https://i.imgur.com/6bfR1Vg.png

I had hair like that once...




:(

[FranceBB]

I had hair like that once...

:(

Sorry to hear that, David... but hey, Jimmy Carr had a transplant and still looks ok, so maybe nothing is lost for you either ehehehehe

[jpsdr]

Never worked with several sockets motherboards, and, honestly, i thought that having 2 CPU with 4 cores or one CPU with 8 cores will be the same from OS point of view : The OS report that you have 8 cores...
I don't know very well some system parts, and the only way i know to get informations is what's in Get_CPU_Info in ThreadPool.cpp file. If doing this doesn't allow to have all the informations, unfortunately for now, i don't know what to do.

[tormento]

I don't know very well some system parts, and the only way i know to get informations is what's in Get_CPU_Info in ThreadPool.cpp file. If doing this doesn't allow to have all the informations, unfortunately for now, i don't know what to do.
First result (https://docs.microsoft.com/en-us/windows/win32/procthread/numa-support) on Google :p

[jpsdr]

Thanks.
It seems issue is when there is more than 64 cores... Maybe i'll take a closer look one day in the future, not interested right now to spend time on this.

[hello_hello]

jpsdr,

Did you happen to see my post in the ResampleMT thread?
In case you missed it.
https://forum.doom9.org/showthread.php?p=1954534#post1954534
I did some more testing and so far the "wrong cropping" only seems to apply to RGB24/32.

Cheers.

[jpsdr]

Finaly sooner than expected, new version, see first post.

[Ceppo]

Thanks for the plugin pack! Downloaded!

[FranceBB]

Thanks, as always, Jean Philippe! ;)

[jpsdr]

New version, see first post.

[jpsdr]

You can either see the formula used in the code, or read the pdf, i described the formula used.

[jpsdr]

Color=0,HDRMode=2, first, means it's HLG not normalised.
You have nominal_luminance=800 for avsresize so maybe you should set in my plugin HLGLw=800 as the default 1000 don't match, and check the default value of this parameter for fmtconv.
Also, as i'm not realy an HLG mode master, try also with Color=0,HDRMode=1.
So, my advice would bet to test these 2 settings : Color=0,HDRMode=2,HLGLw=800 and Color=0,HDRMode=1,HLGLw=800 for avsresize compare.

EDIT
Re-read my pdf, and linear RGB is direct E <-> E' by a simple EOTF function, and not affected by the HDRMode or HLGLw, you need HLGLw only for displayed linear, defaut setting of my plugin output scene linear light and scaled by camera exposure, normalized to the range [0.0:1.0]. Maybe the others plugin output the displayed linear...
In that case, try with OOTF=false, it will ouput the displayed linear data.
But in that case you have to put also OOTF=false in ConvertLinearRGBToYUV.

Otherwise, only left, read the code to check formula used.
I don't know what the others plugins output, neither what they expect in input. "linear", but wich one ? scene linear light and scaled by camera exposure or displayed linear ?

EDIT2
Should move eventualy to HDRTools related thread, as it seems specific to it.

[jpsdr]

New version, see first post.

[StainlessS]

Cheers J.

[FranceBB]

Thank you, Jean Philippe! :)
I'll update my farm at work as I get back after Thanksgiving. ;)

[jpsdr]

New version, see first post.

[jpsdr]

New version, see first post.