MP_Pipeline 0.18 - run parts of avisynth script in external processes [2014-04-06]

[SAPikachu]

This plugin is originally written for my friend to work-around the [del]2GB[/del] 4GB problem of 32-bit process. (Well, also for fun. ) Don't know whether it is useful for others, but I decided to post it here anyways.

As of 0.11, overhead of the plugin is much smaller, it may be possible to use it to speed up more scripts.

Change log:

0.18
* Fix deadlock when exported clip is consumed by multiple script block

0.17
* Properly terminate slave processes when initialization fails
* Fix "Not a clip" error when using ### inherit and the last block is empty

0.16
* Try to silent all error dialogs on exit of slave process
* Slave process shouldn't be stuck on exit anymore, it will terminate itself if it doesn't exit cleanly after 15 seconds
* Fix ### branch statement, previously it incorrectly rejects some input

0.15
* Properly clean script environment up on exit
* Allow using different avisynth dll to run script block (### dll)

0.14
* Fixed another crashing bug

0.13
* Fixed a bug that causes occasional crashing

0.12
* Fixed a problem that makes scripts unable to be loaded in some programs

0.11
* Greatly improved performance, maximum 80% overhead reduction
* New feature: Ability to lock threads to cores, may improve performance in some cases
* (0.10 is skipped to avoid confusion)

0.9
* New feature: Frame prefetching
* New feature: Exporting multiple clip variables in a single process
* New feature: Code block can be shared between processes

0.3
* Binaries in the x86 folder are in correct version now (In 0.2 the win64 slave is actually win32...)
* Integrated a patched TCPDeliver, no longer depend on the external one
* Fixed random crash when filter chain is destroyed
* Thunked branching

0.2
* x64 support (please copy TCPDeliver.dll in the package to respective plugin folder)
* x86/x64 mixed slave process (requires both x86/x64 version of AviSynth to be installed)
* Add a script variable in branch slave process, make it distinguishable in script

Limitations:
* Since each process have its own script environment, all script variables and loaded plugins won't be inherited, they must be re-initialized if needed
* Due to the limitation above, manually-loaded plugins and imported scripts need to be reloaded/re-imported before they can be used in new process (Or use inherited script snippet, please see MP_Pipeline_readme.avs for details)
* Clips before MP_Pipeline will be ignored
* Audio is not supported
* Every script block must return a clip (i.e. "last" must be a clip), otherwise MPP will raise this error: Invalid arguments to function "MPP_PrepareDownstreamClip"

Binary: http://nmm.me/z6
Source code: https://github.com/SAPikachu/MP_Pipeline/tree/0.18

Some example:

1. Basic usage:

MP_Pipeline("""
FFVideoSource("SomeVideo")
QTGMC()
### prefetch: 16, 0
### ###
""")
MCTD()

# MCTD and QTGMC will be run parallelly in 2 separate processes

2. Speed up MCTD at the cost of memory

# Must be 64bit system with at least 8GB memory to run this script
MP_Pipeline("""

# This may be smaller, but I only tested this number
SetMemoryMax(3072)

FFVideoSource("SomeVideo")
MCTD(settings="high")
### prefetch: 16, 0
### ###
""")

# Some time ago I used a script similar to this one for encoding, it is about 20% ~ 30% faster than plain MCTD.

3. Branching

MP_Pipeline("""
FFVideoSource("SomeVideo")
TNLMeans()
### prefetch: 16, 0
### branch: 4
### ###
""")

# TNLMeans will be run in 4 processes with branching (please see example script in the package for details)

4. Frame caching

MP_Pipeline("""
FFVideoSource("SomeUnseekableVideo", seekmode=-1)
TNLMeans()
### prefetch: 32, 24
# It is important to use a big backward cache since we can't seek

### ###

MCTD()

""")

Please see example script in the binary package for some other usage and setting explanations.

[TheRyuu]

Originally Posted by SAPikachu

work-around the 4GB problem of 32-bit process.

ftfy.

[SAPikachu]

Originally Posted by TheRyuu

ftfy.

User processes can only use 2GB of full address space, don't they? (well... actually 3GB on some conditions, but that's a special case)

[SEt]

Actually 4GB on 64 bit OS.

[SAPikachu]

Originally Posted by SEt

Actually 4GB on 64 bit OS.

Didn't notice that until read this. Learned something today, thanks.

[kemuri-_9]

Originally Posted by SEt

Actually 4GB on 64 bit OS.

generally everything has to be compiled with large address awareness for the 32bit binaries (executable and dlls) to really allow addressing over 2GB of memory.

as this is also not usually a default build option iirc, most things don't have it enabled, preventing beyond 2GB of addressable memory.

[LoRd_MuldeR]

So this basically is an AVS2YUV clone, but not as a stand-alone application, but as an Avisynth plug-in?

I think this would be particularly useful to load 32-Bit plugins (that don't have 64-Bit equivalents) into a 64-Bit Avisynth environment. Or vice versa.

Is that supported/intended?

[Gavino]

Originally Posted by kemuri-_9

generally everything has to be compiled with large address awareness for the 32bit binaries (executable and dlls) to really allow addressing over 2GB of memory.

I thought it was just executables (not dlls). Thus Avisynth will benefit from increased memory if used by a client that has been built as 'large address aware'.

[LoRd_MuldeR]

Originally Posted by Gavino

I thought it was just executables (not dlls). Thus Avisynth will benefit from increased memory if used by a client that has been built as 'large address aware'.

Right:
http://blogs.msdn.com/b/oldnewthing/.../10065933.aspx

But then loading a DLL into some "LARGEADDRESSAWARE" process might break it, if the code in that DLL isn't prepared to deal with addresses beyond 2 GB.

[SAPikachu]

Originally Posted by LoRd_MuldeR

So this basically is an AVS2YUV clone, but not as a stand-alone application, but as an Avisynth plug-in?

I think this would be particularly useful to load 32-Bit plugins (that don't have 64-Bit equivalents) into a 64-Bit Avisynth environment. Or vice versa.

Is that supported/intended?

It is functionally similar to avs2yuv, but with some additional features like multiple levels of pipeline.

That is not my original intention, but it is interesting. It only supports x86 now, I will add x64 and mixed script environment support later when I have time.

[kolak]

Can we use this to divide file (using trim) to few parts and run on each one (in seperate process) QTGMC and put them together at the end?


Andrew

[06_taro]

Now add large memory aware flag to exceed 2GB limit in avs4x264mod.

[-Vit-]

Originally Posted by kolak

Can we use this to divide file (using trim) to few parts and run on each one (in seperate process) QTGMC and put them together at the end?

It's an interesting plugin, but doesn't seem to help for that unless I'm missing something. I tried this on some SD footage:

MP_Pipeline("""

WhateverSource("Some\Source")

### ###

QTGMC("Placebo")

### branch: 4

### ###

""")

Worked OK, ran five slave processes and produced the correct result. However, it was slower than single threaded (single threaded is 6fps, this script was 5fps). Used about 2.4Gb memory. Increasing branch slowed it down further, reducing branch to 2 speeded it up to just over 6fps.

By comparison, splitting the video and running many separate single threaded encoding processes, or just using SetMTMode gives 20-25fps. SetMTMode uses a lot less memory.

[kolak]

Hmmm- shame.

I'm forced to run few instances for HD- not a big deal, but if it could be automated than it would be easier.

[SAPikachu]

Originally Posted by -Vit-

It's an interesting plugin, but doesn't seem to help for that unless I'm missing something. I tried this on some SD footage:

MP_Pipeline("""

WhateverSource("Some\Source")

### ###

QTGMC("Placebo")

### branch: 4

### ###

""")

Worked OK, ran five slave processes and produced the correct result. However, it was slower than single threaded (single threaded is 6fps, this script was 5fps). Used about 2.4Gb memory. Increasing branch slowed it down further, reducing branch to 2 speeded it up to just over 6fps.

By comparison, splitting the video and running many separate single threaded encoding processes, or just using SetMTMode gives 20-25fps. SetMTMode uses a lot less memory.

The branch statement is actually not very useful, it is only suitable for spatial single-threaded plugins like TNLMeans, for temporal scripts/filters (especially complex script like QTGMC), the same frame will be repeatedly processed by multiple processes and cpu time will be wasted, decreasing speed. That's why I didn't mention it in OP.

[-Vit-]

Originally Posted by SAPikachu

The branch statement is actually not very useful, it is only suitable for spatial single-threaded plugins like TNLMeans, for temporal scripts/filters (especially complex script like QTGMC), the same frame will be repeatedly processed by multiple processes and cpu time will be wasted, decreasing speed. That's why I didn't mention it in OP.

Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

[06_taro]

Originally Posted by -Vit-

Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

Because it is much easier to use selectevery and interleave. You don't need to care exactly how many frames in total.

[SAPikachu]

Originally Posted by -Vit-

Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

Splitting the whole clip into big thunks doesn't make sense as we can't have parallelism in avs filter in this way. But I think we can split the clip into small thunks (32 frames each thunk for example), and use it with ThreadRequest. This may reduce duplicated processing, the speed may increase (but I'm afraid that this method will never be faster than SetMTMode since the overhead is much bigger) I need to make some new filters for that. Again, when I have time...

[pbristow]

Ooh!

So does this mean I can at last do:

LoadPlugin("MP_Pipeline.dll")
LoadPLugin("PB_3D_tools.dll")

AVIsource("some_anaglyph_3D_thing.avi")

MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)

...to create my cross-eye 3d versions from anaglyph 3D stuff in half the time?

I'm assuming "last" is passed though in the normal way. Have got the right idea about getting data back from the processes? Do LeftEye and RightEye need to be global variables, and if so, where should they be defined: Inside MP_Pipeline, outside, or both?

[Gavino]

Originally Posted by pbristow

...
MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)

I don't think that will work, since:

Originally Posted by SAPikachu

* Since each process have its own script environment, all script variables and loaded plugins won't be inherited, they must be re-initialized if needed

I assume this also applies the other way, so the outer script does not inherit any variables set by the processes.

I expect also that the script in quotes must return a clip, and yours doesn't.