Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
10th February 2012, 10:16 | #1 | Link |
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
Sora's avs multi-process/multi-thread plugin package (2012-02-20)
Hello, today I have made two avisynth plugins to solve some problem.
the first one is SoraThread. It uses multiple threads to pipeline the filters. It aims at making full use of multi-core cpu. the other is SoraSMSource/SoraSMServer. It uses multiple processes to go through the filters. I heard some filter uses a huge number of memory, and process will crash during going through filters. Also shared memory is the fastest way to make inter-process communication so I have a try on this way. This one aims at break the limit of memory usage of a single 32-bit process. It's my first attempt to write an avisynth plugin, and my English is not that good. 2 pictures to explain what I am doing are in the 7z. I hope my work is of use. Please feel free to post comments and questions. Source code (in bsd license) is also in the 7z. http://ftp.sora.info.tm/sora_mtmp_package_20120220.7z <- some experimental codes are in this version old versions: http://ftp.sorayuki.net/sora_mtmp_package_20120219.7z <- so maybe you'll also like this. http://ftp.sorayuki.net/sora_mtmp_package_20120213.7z http://ftp.sorayuki.net/sora_mtmp_package_20120211.7z http://ftp.sorayuki.net/sora_mtmp_package_20120210.7z important: about the situation it crashes on: http://forum.doom9.org/showthread.php?p=1593287 Last edited by leiming2006; 6th April 2017 at 03:15. Reason: update download url |
11th February 2012, 01:38 | #2 | Link |
Registered User
Join Date: Dec 2005
Location: Germany
Posts: 1,795
|
Interesting plugin... just played a bit with it and got a speed gain of ~40% CPU from 25% -> 60%,
But I don't understand how to use it properly, perhaps you can give some examples? Can or should I use multiple SoraSMSource() ? I tried this little script Code:
1. called: sorasmserver aa myscript.avs Code:
SoraSMSource("aa") directshowsource("P4_02.m2ts") # FullHD Source SoraThread(30,60) fft3dfilter() SoraThread(30,60) fft3dfilter()
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK || VapourSynth Database |
11th February 2012, 07:36 | #4 | Link | ||
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
2012-2-11
+ sorathread: custom prefetch algorithm is supported via lua script f sorasmserver: the way to retrive single frame size is changed. the previous one causes overflow and crashes. (critical!) now, examples are also included Quote:
Quote:
if you are not in the problem of a single 32-bit process's memory limit, I think you don't need sorasmsource/sorasmserver. 2. be careful that sorathread with parameters 30,60 will eat 90 frames of memory. Last edited by leiming2006; 11th February 2012 at 07:45. |
||
11th February 2012, 12:44 | #5 | Link |
Registered User
Join Date: Sep 2008
Posts: 365
|
is SoraThread based on ThreadRequest? (http://forum.doom9.org/showthread.php?p=1405205)
|
11th February 2012, 14:55 | #6 | Link | |
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
Quote:
In fact I didn't know there is a plugin called ThreadRequest before... I have already make it work so I posted in the forum to see how people's attitude toward this. |
|
11th February 2012, 17:51 | #7 | Link | |
Registered User
Join Date: Dec 2005
Location: Germany
Posts: 1,795
|
Quote:
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK || VapourSynth Database |
|
13th February 2012, 06:40 | #8 | Link | |
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
updated.
2012-2-13 binaries are compiled with /LARGEADDRESSAWARE f sorathread: over use buffer and cache f sorasmserver: avs script without a empty line in the end of the file could be read correctly now. also, a new sorasmserver/sorasmsource with higher performance is planning. Quote:
And then, SoraSMServer/Source didn't use a prefetch buffer. I thought script running with SoraSMServer/Source would be slower then running in single thread, single process but the test result is SoraSMServer/Source is slightly faster. Last edited by leiming2006; 13th February 2012 at 10:02. |
|
16th February 2012, 09:35 | #10 | Link |
Registered User
Join Date: Apr 2009
Posts: 478
|
Not bad, I tried this yesterday and finished an encode that will normally crash with SetMTMode(2)
However I'm confused as to how to get more threads. Do I just call SoraThread multiple times, like for example: Code:
Sorathread(15,30) Sorathread(15,30) QTGMC() |
16th February 2012, 10:09 | #11 | Link | |
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
Quote:
It uses a stand-alone thread to have the filters before it work. It makes effect when used between filters, and (should) makes no effect or negative effect when arranged just in a row |
|
18th February 2012, 18:00 | #12 | Link |
Registered User
Join Date: Dec 2005
Location: Germany
Posts: 1,795
|
I get random crashes with the newest version 20120213. No Problems with the previous version.
And maybe it is a bug, I don't know ... If I set cache for example at 20, the encode will slow down and is even slower after a while then without sorathread I'm using now sorathread(2,1) and it seems to give the best performance on my encode.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK || VapourSynth Database |
19th February 2012, 07:18 | #13 | Link | |
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
updated.
2012-2-19 avisynth.h is now from 2.58 + sorathread: a totally cache-disabled version is offered (with cache=0) c sorathread: the parameter "cache" is now default 0 f sorasmserver: error message is given when avisynth script has errors. c sorathread (internal): some codes of mutex and conditional variable usage rollback to the version 20120211 Quote:
I have re-check the source code and the problem didn't appear in my computer so I can't get to know why it crashes. now I rollback the source code about inter-thread synchronization to previous version. and then, after thought twice I wonder if the cache is necessary. filters which need to refer to previous frames are likely maintain it's own cache. So the cache is changed to be disabled by default. (the cache algorithm will not execute and the mutex will not be created) |
|
20th February 2012, 05:35 | #15 | Link | |
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
Quote:
I checked the code and test the program. There is neither deadlock nor crashes happened in my computer but it seems they sometimes happened in others' computer. I don't know why it happens. I'll try different ways to fix, or at least work around it if I can. |
|
20th February 2012, 16:27 | #16 | Link |
Registered User
Join Date: Mar 2006
Location: Shanghai, China
Posts: 203
|
Someone in a Chinese forum said that sorathread hang or simply ate a whole cpu core with doing nothing.
Sometimes even crashes. In my computer I can't reproduce the bug. Even with minidump I can't find what is wrong. So I have to try some ways to work around the problem... A new mode called copy mode is added. (default: off) In this mode it will copy the prefetched frames' data into it's own memory. This mode is significantly slow. I wonder if it can go against the crash. Also, when the avisynth environment changes (I think it will not happen), the copy mode will be automatically enabled. Then, when wait for frame, there is a timeout of 2 seconds now. One person told me that sorathread hangs on his computer. But I can neither reproduce it on my computer, nor found where is wrong in my code. So... sorry, finally I comes up with such an idea...... 2012-2-20 + sorathread: copy mode is now added. frame data will copied into buffer instead of just save the smart pointer. + sorathread: timeout is added when wait for frames or empty slot of buffer. Trying to help my poor algorithm escape from deadlock. f sorathread: frame copying to that belongs to different IScriptEnvironment now will performance data copy Last edited by leiming2006; 20th February 2012 at 16:44. |
20th February 2012, 17:53 | #17 | Link | |
Registered User
Join Date: Dec 2005
Location: Germany
Posts: 1,795
|
Tested your newest version 2012-2-20
My System: Win x64 CPU Q9550 Avisynth 2.6 build May 25 2011 My test script: Quote:
With DataCopy= true it's slightly slower ~6.8 fps I also tested sorathread on "light" filters like deen or fastlinedarken. Here it is just not worth to run soratread, worst case: your encode becomes slower I love your filter, very easy to use with nice speed improvements!
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK || VapourSynth Database Last edited by ChaosKing; 20th February 2012 at 17:58. |
|
20th February 2012, 19:10 | #19 | Link |
Registered User
Join Date: Dec 2005
Location: Germany
Posts: 1,795
|
Well, I tested SetMTmode long time ago, it was ok, but had some problems with some filters + u need a moded avisynth dll for it.
With Soreathreat u can use it with the "official" avisynth versions, without to mod anything. I don't think it's better or worse than SetMTmode, it's just another way of using more CPU cores in avisynth.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK || VapourSynth Database |
Tags |
multi thread |
Thread Tools | Search this Thread |
Display Modes | |
|
|