Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

 
 
Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
Old 1st December 2011, 04:20   #1  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
MP_Pipeline 0.18 - run parts of avisynth script in external processes [2014-04-06]

This plugin is originally written for my friend to work-around the [del]2GB[/del] 4GB problem of 32-bit process. (Well, also for fun. ) Don't know whether it is useful for others, but I decided to post it here anyways.

As of 0.11, overhead of the plugin is much smaller, it may be possible to use it to speed up more scripts.

Change log:
Code:
0.18
* Fix deadlock when exported clip is consumed by multiple script block

0.17
* Properly terminate slave processes when initialization fails
* Fix "Not a clip" error when using ### inherit and the last block is empty

0.16
* Try to silent all error dialogs on exit of slave process
* Slave process shouldn't be stuck on exit anymore, it will terminate itself if it doesn't exit cleanly after 15 seconds
* Fix ### branch statement, previously it incorrectly rejects some input

0.15
* Properly clean script environment up on exit
* Allow using different avisynth dll to run script block (### dll)

0.14
* Fixed another crashing bug

0.13
* Fixed a bug that causes occasional crashing

0.12
* Fixed a problem that makes scripts unable to be loaded in some programs

0.11
* Greatly improved performance, maximum 80% overhead reduction
* New feature: Ability to lock threads to cores, may improve performance in some cases
* (0.10 is skipped to avoid confusion)

0.9
* New feature: Frame prefetching
* New feature: Exporting multiple clip variables in a single process
* New feature: Code block can be shared between processes

0.3
* Binaries in the x86 folder are in correct version now (In 0.2 the win64 slave is actually win32...)
* Integrated a patched TCPDeliver, no longer depend on the external one
* Fixed random crash when filter chain is destroyed
* Thunked branching

0.2
* x64 support (please copy TCPDeliver.dll in the package to respective plugin folder)
* x86/x64 mixed slave process (requires both x86/x64 version of AviSynth to be installed)
* Add a script variable in branch slave process, make it distinguishable in script
Limitations:
* Since each process have its own script environment, all script variables and loaded plugins won't be inherited, they must be re-initialized if needed
* Due to the limitation above, manually-loaded plugins and imported scripts need to be reloaded/re-imported before they can be used in new process (Or use inherited script snippet, please see MP_Pipeline_readme.avs for details)
* Clips before MP_Pipeline will be ignored
* Audio is not supported
* Every script block must return a clip (i.e. "last" must be a clip), otherwise MPP will raise this error: Invalid arguments to function "MPP_PrepareDownstreamClip"

Binary: http://nmm.me/z6
Source code: https://github.com/SAPikachu/MP_Pipeline/tree/0.18

Some example:

1. Basic usage:
Code:
MP_Pipeline("""
FFVideoSource("SomeVideo")
QTGMC()
### prefetch: 16, 0
### ###
""")
MCTD()

# MCTD and QTGMC will be run parallelly in 2 separate processes
2. Speed up MCTD at the cost of memory
Code:
# Must be 64bit system with at least 8GB memory to run this script
MP_Pipeline("""

# This may be smaller, but I only tested this number
SetMemoryMax(3072)

FFVideoSource("SomeVideo")
MCTD(settings="high")
### prefetch: 16, 0
### ###
""")

# Some time ago I used a script similar to this one for encoding, it is about 20% ~ 30% faster than plain MCTD.
3. Branching
Code:
MP_Pipeline("""
FFVideoSource("SomeVideo")
TNLMeans()
### prefetch: 16, 0
### branch: 4
### ###
""")

# TNLMeans will be run in 4 processes with branching (please see example script in the package for details)
4. Frame caching
Code:
MP_Pipeline("""
FFVideoSource("SomeUnseekableVideo", seekmode=-1)
TNLMeans()
### prefetch: 32, 24
# It is important to use a big backward cache since we can't seek

### ###

MCTD()

""")
Please see example script in the binary package for some other usage and setting explanations.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3

Last edited by SAPikachu; 6th April 2014 at 10:57.
SAPikachu is offline   Reply With Quote
 

Tags
avisynth, multi-process, pipeline

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:36.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.