Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 10th February 2012, 10:16   #1  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
Sora's avs multi-process/multi-thread plugin package (2012-02-20)

Hello, today I have made two avisynth plugins to solve some problem.

the first one is SoraThread.
It uses multiple threads to pipeline the filters.
It aims at making full use of multi-core cpu.

the other is SoraSMSource/SoraSMServer.
It uses multiple processes to go through the filters.
I heard some filter uses a huge number of memory, and process will crash during going through filters.
Also shared memory is the fastest way to make inter-process communication so I have a try on this way.
This one aims at break the limit of memory usage of a single 32-bit process.

It's my first attempt to write an avisynth plugin, and my English is not that good. 2 pictures to explain what I am doing are in the 7z.
I hope my work is of use.
Please feel free to post comments and questions.

Source code (in bsd license) is also in the 7z.

http://ftp.sora.info.tm/sora_mtmp_package_20120220.7z <- some experimental codes are in this version

old versions:
http://ftp.sorayuki.net/sora_mtmp_package_20120219.7z <- so maybe you'll also like this.
http://ftp.sorayuki.net/sora_mtmp_package_20120213.7z
http://ftp.sorayuki.net/sora_mtmp_package_20120211.7z
http://ftp.sorayuki.net/sora_mtmp_package_20120210.7z


important: about the situation it crashes on:
http://forum.doom9.org/showthread.php?p=1593287

Last edited by leiming2006; 6th April 2017 at 03:15. Reason: update download url
leiming2006 is offline   Reply With Quote
Old 11th February 2012, 01:38   #2  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 465
Interesting plugin... just played a bit with it and got a speed gain of ~40% CPU from 25% -> 60%,

But I don't understand how to use it properly, perhaps you can give some examples?
Can or should I use multiple SoraSMSource() ?

I tried this little script
Code:
1. called: sorasmserver aa myscript.avs
Code:
SoraSMSource("aa")
directshowsource("P4_02.m2ts") # FullHD Source

SoraThread(30,60)
fft3dfilter()
SoraThread(30,60)
fft3dfilter()
__________________
Search and denoise
ChaosKing is online now   Reply With Quote
Old 11th February 2012, 05:42   #3  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Hmmm will definetly give this a try
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 11th February 2012, 07:36   #4  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
2012-2-11
+ sorathread: custom prefetch algorithm is supported via lua script
f sorasmserver: the way to retrive single frame size is changed. the previous one causes overflow and crashes. (critical!)

now, examples are also included



Quote:
Originally Posted by ChaosKing View Post
Interesting plugin... just played a bit with it and got a speed gain of ~40% CPU from 25% -> 60%,

But I don't understand how to use it properly, perhaps you can give some examples?
Can or should I use multiple SoraSMSource() ?
examples are now included in the 7z.
Quote:
I tried this little script
Code:
1. called: sorasmserver aa myscript.avs
Code:
SoraSMSource("aa")
directshowsource("P4_02.m2ts") # FullHD Source

SoraThread(30,60)
fft3dfilter()
SoraThread(30,60)
fft3dfilter()
1. sorasmsource is a source filter, directshowsource will discard it.
if you are not in the problem of a single 32-bit process's memory limit, I think you don't need sorasmsource/sorasmserver.
2. be careful that sorathread with parameters 30,60 will eat 90 frames of memory.

Last edited by leiming2006; 11th February 2012 at 07:45.
leiming2006 is offline   Reply With Quote
Old 11th February 2012, 12:44   #5  |  Link
mastrboy
Registered User
 
Join Date: Sep 2008
Posts: 287
is SoraThread based on ThreadRequest? (http://forum.doom9.org/showthread.php?p=1405205)
mastrboy is offline   Reply With Quote
Old 11th February 2012, 14:55   #6  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
Quote:
Originally Posted by mastrboy View Post
is SoraThread based on ThreadRequest? (http://forum.doom9.org/showthread.php?p=1405205)
No, I was writing a new plugin.
In fact I didn't know there is a plugin called ThreadRequest before...
I have already make it work so I posted in the forum to see how people's attitude toward this.
leiming2006 is offline   Reply With Quote
Old 11th February 2012, 17:51   #7  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 465
Quote:
if you are not in the problem of a single 32-bit process's memory limit, I think you don't need sorasmsource/sorasmserver.
Ah okay, so basicly SoraSMSource/SoraSMServer and SoraThread do the same thing and I only need SoraSource/Server if I want to use more then 4GB Ram with 32Bit Software?
__________________
Search and denoise
ChaosKing is online now   Reply With Quote
Old 13th February 2012, 06:40   #8  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
updated.

2012-2-13
binaries are compiled with /LARGEADDRESSAWARE
f sorathread: over use buffer and cache
f sorasmserver: avs script without a empty line in the end
of the file could be read correctly now.


also, a new sorasmserver/sorasmsource with higher performance is planning.

Quote:
Originally Posted by ChaosKing View Post
Ah okay, so basicly SoraSMSource/SoraSMServer and SoraThread do the same thing and I only need SoraSource/Server if I want to use more then 4GB Ram with 32Bit Software?
inter-process communication is expansive, so SoraThread should has better performance then SoraSMServer/Source.

And then, SoraSMServer/Source didn't use a prefetch buffer. I thought script running with SoraSMServer/Source would be slower then running in single thread, single process but the test result is SoraSMServer/Source is slightly faster.

Last edited by leiming2006; 13th February 2012 at 10:02.
leiming2006 is offline   Reply With Quote
Old 15th February 2012, 05:20   #9  |  Link
Hiritsuki
Novice of AVS
 
Join Date: Oct 2009
Posts: 156
I had use this plugin to run my project, and it's stable than TCPServer/Source for use.

see that view this picture.
__________________
My PC
Hiritsuki is offline   Reply With Quote
Old 16th February 2012, 09:35   #10  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 452
Not bad, I tried this yesterday and finished an encode that will normally crash with SetMTMode(2)

However I'm confused as to how to get more threads. Do I just call SoraThread multiple times, like for example:

Code:
Sorathread(15,30)
Sorathread(15,30)
QTGMC()
aegisofrime is offline   Reply With Quote
Old 16th February 2012, 10:09   #11  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
Quote:
Originally Posted by aegisofrime View Post
Not bad, I tried this yesterday and finished an encode that will normally crash with SetMTMode(2)

However I'm confused as to how to get more threads. Do I just call SoraThread multiple times, like for example:

Code:
Sorathread(15,30)
Sorathread(15,30)
QTGMC()
SoraThread pipelines the processing.
It uses a stand-alone thread to have the filters before it work.
It makes effect when used between filters,
and (should) makes no effect or negative effect when arranged just in a row
leiming2006 is offline   Reply With Quote
Old 18th February 2012, 18:00   #12  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 465
I get random crashes with the newest version 20120213. No Problems with the previous version.

And maybe it is a bug, I don't know ... If I set cache for example at 20, the encode will slow down and is even slower after a while then without sorathread

I'm using now sorathread(2,1) and it seems to give the best performance on my encode.
__________________
Search and denoise
ChaosKing is online now   Reply With Quote
Old 19th February 2012, 07:18   #13  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
updated.

2012-2-19
avisynth.h is now from 2.58
+ sorathread: a totally cache-disabled version is offered (with cache=0)
c sorathread: the parameter "cache" is now default 0
f sorasmserver: error message is given when avisynth script has errors.
c sorathread (internal): some codes of mutex and conditional variable usage rollback to the version 20120211


Quote:
Originally Posted by ChaosKing View Post
I get random crashes with the newest version 20120213. No Problems with the previous version.

And maybe it is a bug, I don't know ... If I set cache for example at 20, the encode will slow down and is even slower after a while then without sorathread

I'm using now sorathread(2,1) and it seems to give the best performance on my encode.
I don't sure if it's the problem about working with multi-threaded avisynth. the test environment is avisynth 2.58 official build.

I have re-check the source code and the problem didn't appear in my computer so I can't get to know why it crashes. now I rollback the source code about inter-thread synchronization to previous version.

and then, after thought twice I wonder if the cache is necessary. filters which need to refer to previous frames are likely maintain it's own cache. So the cache is changed to be disabled by default. (the cache algorithm will not execute and the mutex will not be created)
leiming2006 is offline   Reply With Quote
Old 20th February 2012, 04:50   #14  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,100
Does this have negative effects?
Like filters may not work properly?
or is it, it either work or crash?
zerowalker is offline   Reply With Quote
Old 20th February 2012, 05:35   #15  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
Quote:
Originally Posted by zerowalker View Post
Does this have negative effects?
Like filters may not work properly?
or is it, it either work or crash?
Certainly it has overheads to maintain buffer.

I checked the code and test the program. There is neither deadlock nor crashes happened in my computer but it seems they sometimes happened in others' computer. I don't know why it happens. I'll try different ways to fix, or at least work around it if I can.
leiming2006 is offline   Reply With Quote
Old 20th February 2012, 16:27   #16  |  Link
leiming2006
Registered User
 
leiming2006's Avatar
 
Join Date: Mar 2006
Location: Shanghai, China
Posts: 202
Someone in a Chinese forum said that sorathread hang or simply ate a whole cpu core with doing nothing.
Sometimes even crashes.

In my computer I can't reproduce the bug.
Even with minidump I can't find what is wrong.

So I have to try some ways to work around the problem...

A new mode called copy mode is added. (default: off)
In this mode it will copy the prefetched frames' data into it's own memory.
This mode is significantly slow.
I wonder if it can go against the crash.

Also, when the avisynth environment changes (I think it will not happen),
the copy mode will be automatically enabled.

Then, when wait for frame, there is a timeout of 2 seconds now.
One person told me that sorathread hangs on his computer.
But I can neither reproduce it on my computer, nor found where is wrong in my code.
So... sorry, finally I comes up with such an idea......

2012-2-20
+ sorathread: copy mode is now added. frame data will copied into buffer instead of just save the smart pointer.
+ sorathread: timeout is added when wait for frames or empty slot of buffer. Trying to help my poor algorithm escape from deadlock.
f sorathread: frame copying to that belongs to different IScriptEnvironment now will performance data copy

Last edited by leiming2006; 20th February 2012 at 16:44.
leiming2006 is offline   Reply With Quote
Old 20th February 2012, 17:53   #17  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 465
Tested your newest version 2012-2-20
My System:
Win x64
CPU Q9550
Avisynth 2.6 build May 25 2011

My test script:
Quote:
deen("a3d", 3,2,3,3)
fastlinedarkenmod(72, thinning=0)
awarpsharp2(62)
sorathread() # 2,1# 2,0# datacopy=true
SantiagMod(strh=2,strv=2)# Antialiasing Filter with NNEDI3
New Version doesn't crash anymore, Speed seems to be the same as in previous versions with my script: ~4.5 fps (without sorathread) -> ~6.9 fps (with sorathread)

With DataCopy= true it's slightly slower ~6.8 fps

I also tested sorathread on "light" filters like deen or fastlinedarken. Here it is just not worth to run soratread, worst case: your encode becomes slower

I love your filter, very easy to use with nice speed improvements!
__________________
Search and denoise

Last edited by ChaosKing; 20th February 2012 at 17:58.
ChaosKing is online now   Reply With Quote
Old 20th February 2012, 18:53   #18  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,100
I still donīt get it, is it better than SetMTmode;S?

;D!
zerowalker is offline   Reply With Quote
Old 20th February 2012, 19:10   #19  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 465
Well, I tested SetMTmode long time ago, it was ok, but had some problems with some filters + u need a moded avisynth dll for it.

With Soreathreat u can use it with the "official" avisynth versions, without to mod anything.

I don't think it's better or worse than SetMTmode, it's just another way of using more CPU cores in avisynth.
__________________
Search and denoise
ChaosKing is online now   Reply With Quote
Old 20th February 2012, 19:12   #20  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,100
Okay, well i use the Avisynth 2.6 version that is posted in this forum with this;O

But am i supposed to write SoraThread(X,X) before every filter?

Thanks
zerowalker is offline   Reply With Quote
Reply

Tags
multi thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 13:15.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.