Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th July 2011, 18:48   #181  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Quote:
Originally Posted by SEt View Post
I suggest changing SetCacheHints return value to something like intptr_t for obvious reasons.
Does this mean that (most) plugins need to be recompiled?

I tried the new build but I get only 25% CPU utilization on my quad-core (upon playback in VDubMod). The previous build maxes out the CPU with the same script.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 14th July 2011, 19:29   #182  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Oops, I forgot to auto-add distributor to regular scripts. Redownload who got earlier version.

No recompilation unless they need to identify as caches, and it just underwent even greater change from void to int.
SEt is offline   Reply With Quote
Old 14th July 2011, 20:36   #183  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Thanks, the new build works just fine.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 14th July 2011, 22:37   #184  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
Quote:
Originally Posted by SEt View Post
No recompilation unless they need to identify as caches, and it just underwent even greater change from void to int.
Sorry if I'm a bit thick, but this does mean exactly ... what?
What's the scenario where a filter "needs to identify as cache"?
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 14th July 2011, 23:00   #185  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
For example ffdshow interface for avisynth is good candidate for such cache identification when "Buffer back/ahead" is turned on - no need to insert another cache after it and waste memory.
SEt is offline   Reply With Quote
Old 15th July 2011, 05:16   #186  |  Link
yup
Registered User
 
Join Date: Feb 2003
Location: Russia, Moscow
Posts: 854
SEt
Your build based on Avisynth 2.6 Alpha 3?
yup.
yup is offline   Reply With Quote
Old 15th July 2011, 05:45   #187  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
The build is from the latest CVS sources so the code is as fresh as possible.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 16th July 2011, 20:16   #188  |  Link
levi
Registered User
 
Join Date: Mar 2003
Posts: 116
Quote:
Originally Posted by SEt View Post
Another threading bugfix + updated to CVS version of Avisynth.
Thanks Set! This rocks! Any chance we could see a 64 bit compile?
levi is offline   Reply With Quote
Old 16th July 2011, 21:29   #189  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Maybe. I'm not very interested in 64-bit build now as I have only 6GB of ram and if you really want you can run some time-critical 64-bit code in 32-bit process too.
SEt is offline   Reply With Quote
Old 19th July 2011, 18:11   #190  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Updated to fix two ImageSource() errors. (Discussed here and here)
SEt is offline   Reply With Quote
Old 20th July 2011, 18:31   #191  |  Link
Büke
Registered User
 
Join Date: Jul 2010
Posts: 4
Parallelization across seperate processes

How about a plugin that loads any avs script into multiple processes, processes about 300 frames in each process, with 30 frames overlapping, sequentially combining the results in the master avs for final processing and encoding.

Additionally, the plugin can restart the processes to clear memory leaks.
Büke is offline   Reply With Quote
Old 20th July 2011, 22:00   #192  |  Link
Mini-Me
Registered User
 
Join Date: Jan 2011
Posts: 121
Quote:
Originally Posted by Büke View Post
How about a plugin that loads any avs script into multiple processes, processes about 300 frames in each process, with 30 frames overlapping, sequentially combining the results in the master avs for final processing and encoding.

Additionally, the plugin can restart the processes to clear memory leaks.
AFAIK, coding this as an Avisynth plugin would be exceedingly difficult or even impossible, given how Avisynth works. However, it's manageable for an outside piece of software...and while I haven't tested it myself, Chikuzen has written a Python script for AvsPMod that works very similarly to what you're describing.
Mini-Me is offline   Reply With Quote
Old 21st July 2011, 05:13   #193  |  Link
Büke
Registered User
 
Join Date: Jul 2010
Posts: 4
Quote:
Originally Posted by Mini-Me View Post
AFAIK, coding this as an Avisynth plugin would be exceedingly difficult or even impossible, given how Avisynth works. However, it's manageable for an outside piece of software...and while I haven't tested it myself, Chikuzen has written a Python script for AvsPMod that works very similarly to what you're describing.
Why would it be so difficult? Interprocess hosting and communication is often done nowadays in the audio plugin world (VST, etc.) for crossing 32-64 bit barriers, for extended memory and stability, and it is done with quiet the low latency, too.

The master plugin needs an exe host counterpart listening for commands from the scheduling master, loads and hosts the given avs, caches the requested frames and sends them back in raw format, it can even use portions of TCPServer() code.

Now I think about it, the overlapping frames are not even necessary, since Avisynth can deliver random access frames as requested by the script and plugins on demand. Runtime scripts often do not (and should not) use progressive global state variable manipulation, as most are made to be randomly seekable. This means, the processes can process less frames in parallel, such as 10-100, and maintain low memory overhead.

Macro parallelization via master controlled sub-processes is a real generic solution to this problem of utilizing all cores, it can even be built into Avisynth natively. The sub processes can even introduce additional compatibility for cross 32/64 bit hosting, and stability by restarting crashing job segments with alternate settings/scripts.

Having this done natively or via a a plugin such as ImportParallel(string AVSFile, int "ProcessCount", int "FrameCount", ...) all in memory will be a lot faster and transparent than using splitting scripts in other languages and using slow disk caches.

Last edited by Büke; 21st July 2011 at 05:26.
Büke is offline   Reply With Quote
Old 21st July 2011, 05:22   #194  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
This is unrelated to this thread. I request moderators to split the posts.

Regarding the suggestion, I think another hack is not what avisynth needs today. It needs clear and mandatory rules for plugins how to deal with multithreading.
SEt is offline   Reply With Quote
Old 21st July 2011, 05:41   #195  |  Link
Büke
Registered User
 
Join Date: Jul 2010
Posts: 4
Quote:
Originally Posted by SEt View Post
This is unrelated to this thread. I request moderators to split the posts.

Regarding the suggestion, I think another hack is not what avisynth needs today. It needs clear and mandatory rules for plugins how to deal with multithreading.
Why is this a hack? It is a real solution, that has been proven in other domains.

A host's mission is to make things easier for plugin authors, who would rather focus and specialize on the complexities of the algorithms, rather than their parallelization or optimization, or user interface, etc, etc.

Are you suggesting that new rules be put out and all plugins be rewritten, and new ones with additional complexities?

If the plugin authors were interested in multi-threading more than the core algorithm, they would have done this in majority of the plugins, including the native filters already..

More practical, and less dogmatic, is what I say.
Büke is offline   Reply With Quote
Old 21st July 2011, 22:41   #196  |  Link
SubJunk
Registered User
 
Join Date: Jun 2010
Posts: 443
Thanks a lot for the update, SEt!
SubJunk is offline   Reply With Quote
Old 22nd July 2011, 09:04   #197  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Quote:
Originally Posted by SEt View Post
Maybe. I'm not very interested in 64-bit build now as I have only 6GB of ram and if you really want you can run some time-critical 64-bit code in 32-bit process too.
Some 64bits pluggins can have sometime speed increase of 30%.
I've noticed, on QTMG for example, an increase speed of around 15% under 64bits.

My personnal thought, at the actual stage is :
The way MT is implemented on avisynth seems not to be very stable, so, the best way actualy to have a stable multithreading would be to not use MT version of avisynth, split your work and start several task. For exemple, launching 6 Vdub task on 6 files.
Using not MT version of avisynth keep garanty stability, and running several tasks make all your core work.
I think, it could be more interesting to focus on single threading 64bits port, because good 64bits port can provide good improvement in speed.
I've noticed it with the unfornutaly dead project of the 64 bits version begun by JoshyD.
jpsdr is offline   Reply With Quote
Old 29th July 2011, 03:51   #198  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Does anyone else suffer from I/O operations sometimes getting very slow when encoding something using SetMTMode? I noticed the behaviour already in the old build and just got it with the new build. Creating a d2v file was getting along very slowly until I paused x264 which was using 95-99% of the CPU. The x264 process was running at idle priority so it should not cause such things. The MT'd Avisynth v2.5.8 doesn't have this issue, or at least I never noticed it.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 29th July 2011, 10:49   #199  |  Link
leeperry
Kid for Today
 
Join Date: Aug 2004
Posts: 3,477
Quote:
Originally Posted by SubJunk View Post
Thanks a lot for the update, SEt!
X2,

but how can you MT w/ the new build? I'm currently using MT 0.70 w/ the 2009.09.19 2.6 build. MT 0.70 doesn't work w/ the new build, and neither does this version of MT: http://forum.doom9.org/showpost.php?...postcount=1093

I use avisynth in realtime in ffdshow, I do need MT.

Last edited by leeperry; 29th July 2011 at 11:02.
leeperry is offline   Reply With Quote
Old 31st July 2011, 19:49   #200  |  Link
levi
Registered User
 
Join Date: Mar 2003
Posts: 116
Quote:
Originally Posted by leeperry View Post
X2,

but how can you MT w/ the new build? I'm currently using MT 0.70 w/ the 2009.09.19 2.6 build. MT 0.70 doesn't work w/ the new build, and neither does this version of MT: http://forum.doom9.org/showpost.php?...postcount=1093

I use avisynth in realtime in ffdshow, I do need MT.
Did you try this one:

http://code.google.com/p/avisynth64/...0.rar&can=2&q=
levi is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:19.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.