Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
20th August 2016, 15:19 | #41 | Link |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
Pleeeaseee !!!! Help wanted....
I've tried to create an external ThreadPool in a DLL for plugin to use. I've searched and found how to create a simple DLL. I've been ... almost... successfull. I am for now doing tests on my "standard" PC, so it's Windows 7 x86 with Avisynth 2.6.0. You can get the source of the ThreadPool DLL and the filter. I have one minor and one critical issue. The minor : Even if i put the DLL in the same directory of the plugin, it's not working, i have to put it in the system32 directory. Strange, because i hadn't this issue when i've first made some quick very small little test, of an exe using 2 dll, both using a third dll... When everyone was in the same directory, everyone was happy... If anyone can tell me if there is something to do during the build process, on compiler option to add for this to be solved. The critical : Well... Indeed, everything works fine, i can open the avs script in VDub, etc... Until.... I want to close it It seems that in the FreeData called by the destructor, the program is stuck on the line Code:
WaitForSingleObject(thds[i],INFINITE); The strange things is that i'm doing exactly the same working thing i'm doing everywhere else. But, as it's the first time i'm trying to do a DLL, maybe the way i'm doing it is totaly wrong, it works by chance, and the fact that it doesn't end properly is caused by that. Maybe it's something else, compiler option, just a little trick, no idea. So, if there is someone with enough courage and powerwill to take a look at the sources code i've provided... In any case |
20th August 2016, 19:17 | #43 | Link |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
After some test, finaly, speed is slower. Another bad idea and almost 2 days wasted...
Well, it's when trying that you see. But... Instead of an external DLL, the same thing internaly... May keep up with speed, and reduce the number of threads... Another thing to try. |
20th August 2016, 21:33 | #45 | Link |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
I did some tests with avstp and plugins that use it. Unfortunately, it's not very efficient and does not scale well.
__________________
Groucho's Avisynth Stuff |
20th August 2016, 21:36 | #46 | Link |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Not really wasted, I'm sure you learned something. At least that's how I like to look at such experiences.
__________________
Groucho's Avisynth Stuff |
21st August 2016, 09:51 | #49 | Link |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
@jpsdr
In comparisons like this one, you can make use of this AVSMeter ini setting: "DisplayEfficiencyIndex" (0 or 1): If set to "1", the result of (FPS / CPU usage) is printed to the console. This ratio indicates the efficiency of a script (higher = better) and is useful for comparing different versions of a script or scripts that are supposed to do the same thing.
__________________
Groucho's Avisynth Stuff |
21st August 2016, 11:07 | #50 | Link |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
Ok, new version.
- Update to new avisynth header (don't know if it changes anything, but don't hurt). - Use an external thread pool class, but internaly within the DLL, not on an external DLL. Previous 1.0.1 version : Code:
[OS/Hardware info] OS version: Windows 7 (x64) Service Pack 1 (Build 7601) CPU (brand string): Intel(R) Core(TM) i7-6950X CPU @ 3.00GHz CPU (code name): Unknown Core 2 CPU clock (measured): 3624 MHz CPU cores / Logical cores: 10 / 20 [Avisynth info] Avisynth VersionString: AviSynth+ 0.1 (r2172, MT, x86_64) Avisynth VersionNumber: 2.60 File version: 0.1.0.0 Avisynth Interface Version: 6 Multi-threading support: Yes Linker/compiler version: 14.0 [Clip info] Number of frames: 1000 Length (hh:mm:ss.ms): 00:00:40.000 Frame width: 3200 Frame height: 1520 Framerate: 25.000 (25/1) Colorspace: YV12 [Runtime info] Frames processed: 1000 (0 - 999) FPS (min | max | average): 39.57 | 47.14 | 46.84 Memory usage (phys | virt): 91 | 87 MiB Thread count: 221 CPU usage (average): 47% Time (elapsed): 00:00:21.350 [Script] Colorbars(width=1920*2,height=1080*2,pixel_type="yv12").killaudio().assumefps(25,1).trim(0,999) BilinearResizeMT(width()-64,height()-64,threads=10) BicubicResizeMT(width()-64,height()-64,threads=10) LanczosResizeMT(width()-64,height()-64,threads=10) Lanczos4ResizeMT(width()-64,height()-64,threads=10) BlackmanResizeMT(width()-64,height()-64,threads=10) Spline16ResizeMT(width()-64,height()-64,threads=10) Spline36ResizeMT(width()-64,height()-64,threads=10) Spline64ResizeMT(width()-64,height()-64,threads=10) GaussResizeMT(width()-64,height()-64,threads=10) SincResizeMT(width()-64,height()-64,threads=10) Code:
[Clip info] Number of frames: 1000 Length (hh:mm:ss.ms): 00:00:40.000 Frame width: 3200 Frame height: 1520 Framerate: 25.000 (25/1) Colorspace: YV12 [Runtime info] Frames processed: 1000 (0 - 999) FPS (min | max | average): 27.91 | 47.02 | 46.56 Memory usage (phys | virt): 83 | 78 MiB Thread count: 31 CPU usage (average): 47% Time (elapsed): 00:00:21.477 [Script] Colorbars(width=1920*2,height=1080*2,pixel_type="yv12").killaudio().assumefps(25,1).trim(0,999) BilinearResizeMT(width()-64,height()-64,threads=10) BicubicResizeMT(width()-64,height()-64,threads=10) LanczosResizeMT(width()-64,height()-64,threads=10) Lanczos4ResizeMT(width()-64,height()-64,threads=10) BlackmanResizeMT(width()-64,height()-64,threads=10) Spline16ResizeMT(width()-64,height()-64,threads=10) Spline36ResizeMT(width()-64,height()-64,threads=10) Spline64ResizeMT(width()-64,height()-64,threads=10) GaussResizeMT(width()-64,height()-64,threads=10) SincResizeMT(width()-64,height()-64,threads=10) My next step : Doing the same thing on other plugins i'm working on (AutoYUY2 and NNEDI3). My next next step, creating a plugin package with my 3 filters, but the filters in standalone version will continue and will not be dropped. The point : None if you're using only one of them, reduce the number of threads created if you're using more than one of them, because they'll all share the same thread pool. If using an external DLL thread pool didn't produce a noticeable speed lost, it would have been the best, but it's, still from my point of view (of course you may not share it) the best compromise. For those who are still wondering the point of the whole subjet ? I personnaly don't use the MT mode because i don't want the memory explosion usage when you have a lot of cores. Last edited by jpsdr; 21st August 2016 at 14:18. |
23rd August 2016, 15:58 | #51 | Link |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
jpsdr, I sent you a PM.
__________________
Groucho's Avisynth Stuff |
23rd August 2016, 19:39 | #53 | Link | |
AVS+ Dev
Join Date: Aug 2013
Posts: 359
|
Quote:
The basic usage is: 1) You call IScriptEnv2->NewCompletion() in your plugin constructor. The capacity arg is the number of jobs you'll enqueue at once in parallel. 2) In your GetFrame() you call IJobCompletion->Reset(), then call IScriptEnv2->ParallelJob() for each task you want to execute in the threadpool. The 1st arg is your function to execute, the 2nd is data that'll be passed to your function, and the 3rd is the compleiton object you created in step 1. 3) After all tasks are queued for a frame, you call IJobCompletion->Get to receive the result of the i'th queued job. 4) In your plugin destructor, you call IJobCompletion->destroy(). Do not call delete on this object yourself. This will of course also reduce coding burden on you since proper thread-safe queueing, thread management, and in general everything associated with the thread pool will be managed and maintained by the core, and you won't have to maintain duplicate code for it in your plugin.
__________________
AviSynth+ Last edited by ultim; 23rd August 2016 at 20:01. |
|
23rd August 2016, 21:23 | #55 | Link |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
I'll try, out of curiosity to see if i see the performance drop i've noticed when i try to put the thread pool in an external DLL.
But it will complexify the code (a lot... not so much...?) to have in the same time my and your threadpool, because there is one thing not to forget : I want my plugins working multi-threaded an all avs+ and 2.6.x standard version, not only on the avs+ MT version. But, i'll give a try, i'm curious... |
23rd August 2016, 22:14 | #56 | Link |
AVS+ Dev
Join Date: Aug 2013
Posts: 359
|
Well, if you want to stay compatible with avs 2.6, than I can understand if you don't want to use avs+-specific APIs. May I ask the reasons for wanting to support 2.6? Is it a technical reason, like an area where avs+ doesn't work as great as 2.6 does, or is it simply because avs+ is not that commonly spread as 2.6 yet? I think the latter situation will improve once a new stable version of avs+ is released, which I'm planning on in the foreseeable future.
__________________
AviSynth+ |
24th August 2016, 08:43 | #57 | Link |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
It's just my way... to want to be compatible with the most possible (reasonable) versions, if it's not something if think too hard. In fact, i often hate the opposite : being obliged to stick with a specific version, when i think it could have been otherwise just with a little work. (By reasonable i mean i've dropped 2.5.x ... ). No realy technical reasons, just a less restriction philosophy...
And for those who still use 2.6.x for any reasons, they'll have something upgraded. After, as i said, it's just my way, for now. What future will be...? But, i'm still curious to give a try to your api, so, maybe not right now, but very soon, i'll made a specific version using it to make some tests. Last edited by jpsdr; 24th August 2016 at 08:52. |
24th August 2016, 20:48 | #59 | Link |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
Ooops...
Wait next release tomorrow, there is a tricky deadlock. It doens't show if you open a script with VDub and seek within it, it doesn't show if you run a script with avsmeter, but, it shoes if you're creating several jobs in batch in VDub and want to run them. Fix is done, but it's late, so new builds an push on github tomorrow. |
|
|