MP_Pipeline 0.18 - run parts of avisynth script in external processes [2014-04-06]

[SAPikachu]

This plugin is originally written for my friend to work-around the 2GB 4GB problem of 32-bit process. (Well, also for fun. :) ) Don't know whether it is useful for others, but I decided to post it here anyways.

As of 0.11, overhead of the plugin is much smaller, it may be possible to use it to speed up more scripts.

Change log:

0.18
* Fix deadlock when exported clip is consumed by multiple script block

0.17
* Properly terminate slave processes when initialization fails
* Fix "Not a clip" error when using ### inherit and the last block is empty

0.16
* Try to silent all error dialogs on exit of slave process
* Slave process shouldn't be stuck on exit anymore, it will terminate itself if it doesn't exit cleanly after 15 seconds
* Fix ### branch statement, previously it incorrectly rejects some input

0.15
* Properly clean script environment up on exit
* Allow using different avisynth dll to run script block (### dll)

0.14
* Fixed another crashing bug

0.13
* Fixed a bug that causes occasional crashing

0.12
* Fixed a problem that makes scripts unable to be loaded in some programs

0.11
* Greatly improved performance, maximum 80% overhead reduction
* New feature: Ability to lock threads to cores, may improve performance in some cases
* (0.10 is skipped to avoid confusion)

0.9
* New feature: Frame prefetching
* New feature: Exporting multiple clip variables in a single process
* New feature: Code block can be shared between processes

0.3
* Binaries in the x86 folder are in correct version now (In 0.2 the win64 slave is actually win32...)
* Integrated a patched TCPDeliver, no longer depend on the external one
* Fixed random crash when filter chain is destroyed
* Thunked branching

0.2
* x64 support (please copy TCPDeliver.dll in the package to respective plugin folder)
* x86/x64 mixed slave process (requires both x86/x64 version of AviSynth to be installed)
* Add a script variable in branch slave process, make it distinguishable in script


Limitations:
* Since each process have its own script environment, all script variables and loaded plugins won't be inherited, they must be re-initialized if needed
* Due to the limitation above, manually-loaded plugins and imported scripts need to be reloaded/re-imported before they can be used in new process (Or use inherited script snippet, please see MP_Pipeline_readme.avs for details)
* Clips before MP_Pipeline will be ignored
* Audio is not supported
* Every script block must return a clip (i.e. "last" must be a clip), otherwise MPP will raise this error: Invalid arguments to function "MPP_PrepareDownstreamClip"

Binary: http://nmm.me/z6
Source code: https://github.com/SAPikachu/MP_Pipeline/tree/0.18

Some example:

1. Basic usage:

MP_Pipeline("""
FFVideoSource("SomeVideo")
QTGMC()
### prefetch: 16, 0
### ###
""")
MCTD()

# MCTD and QTGMC will be run parallelly in 2 separate processes


2. Speed up MCTD at the cost of memory

# Must be 64bit system with at least 8GB memory to run this script
MP_Pipeline("""

# This may be smaller, but I only tested this number
SetMemoryMax(3072)

FFVideoSource("SomeVideo")
MCTD(settings="high")
### prefetch: 16, 0
### ###
""")

# Some time ago I used a script similar to this one for encoding, it is about 20% ~ 30% faster than plain MCTD.


3. Branching

MP_Pipeline("""
FFVideoSource("SomeVideo")
TNLMeans()
### prefetch: 16, 0
### branch: 4
### ###
""")

# TNLMeans will be run in 4 processes with branching (please see example script in the package for details)


4. Frame caching

MP_Pipeline("""
FFVideoSource("SomeUnseekableVideo", seekmode=-1)
TNLMeans()
### prefetch: 32, 24
# It is important to use a big backward cache since we can't seek

### ###

MCTD()

""")


Please see example script in the binary package for some other usage and setting explanations.

[TheRyuu]

work-around the 4GB problem of 32-bit process.

ftfy.

[SAPikachu]

ftfy.

User processes can only use 2GB of full address space, don't they? (well... actually 3GB on some conditions, but that's a special case)

[SEt]

Actually 4GB on 64 bit OS.

[SAPikachu]

Actually 4GB on 64 bit OS.

Didn't notice that until read this. Learned something today, thanks. :)

[kemuri-_9]

Actually 4GB on 64 bit OS.

generally everything has to be compiled with large address awareness for the 32bit binaries (executable and dlls) to really allow addressing over 2GB of memory.

as this is also not usually a default build option iirc, most things don't have it enabled, preventing beyond 2GB of addressable memory.

[LoRd_MuldeR]

So this basically is an AVS2YUV clone, but not as a stand-alone application, but as an Avisynth plug-in?

I think this would be particularly useful to load 32-Bit plugins (that don't have 64-Bit equivalents) into a 64-Bit Avisynth environment. Or vice versa.

Is that supported/intended?

[Gavino]

generally everything has to be compiled with large address awareness for the 32bit binaries (executable and dlls) to really allow addressing over 2GB of memory.
I thought it was just executables (not dlls). Thus Avisynth will benefit from increased memory if used by a client that has been built as 'large address aware'.

[LoRd_MuldeR]

I thought it was just executables (not dlls). Thus Avisynth will benefit from increased memory if used by a client that has been built as 'large address aware'.

Right:
http://blogs.msdn.com/b/oldnewthing/archive/2010/09/22/10065933.aspx

But then loading a DLL into some "LARGEADDRESSAWARE" process might break it, if the code in that DLL isn't prepared to deal with addresses beyond 2 GB.

[SAPikachu]

So this basically is an AVS2YUV clone, but not as a stand-alone application, but as an Avisynth plug-in?

I think this would be particularly useful to load 32-Bit plugins (that don't have 64-Bit equivalents) into a 64-Bit Avisynth environment. Or vice versa.

Is that supported/intended?

It is functionally similar to avs2yuv, but with some additional features like multiple levels of pipeline.

That is not my original intention, but it is interesting. It only supports x86 now, I will add x64 and mixed script environment support later when I have time.

[kolak]

Can we use this to divide file (using trim) to few parts and run on each one (in seperate process) QTGMC and put them together at the end?


Andrew

[06_taro]

Now add large memory aware flag to exceed 2GB limit in avs4x264mod (http://forum.doom9.org/showthread.php?t=162656).

[-Vit-]

Can we use this to divide file (using trim) to few parts and run on each one (in seperate process) QTGMC and put them together at the end?
It's an interesting plugin, but doesn't seem to help for that unless I'm missing something. I tried this on some SD footage:

MP_Pipeline("""

WhateverSource("Some\Source")

### ###

QTGMC("Placebo")

### branch: 4

### ###

""")


Worked OK, ran five slave processes and produced the correct result. However, it was slower than single threaded (single threaded is 6fps, this script was 5fps). Used about 2.4Gb memory. Increasing branch slowed it down further, reducing branch to 2 speeded it up to just over 6fps.

By comparison, splitting the video and running many separate single threaded encoding processes, or just using SetMTMode gives 20-25fps. SetMTMode uses a lot less memory.

[kolak]

Hmmm- shame.

I'm forced to run few instances for HD- not a big deal, but if it could be automated than it would be easier.

[SAPikachu]

It's an interesting plugin, but doesn't seem to help for that unless I'm missing something. I tried this on some SD footage:

MP_Pipeline("""

WhateverSource("Some\Source")

### ###

QTGMC("Placebo")

### branch: 4

### ###

""")


Worked OK, ran five slave processes and produced the correct result. However, it was slower than single threaded (single threaded is 6fps, this script was 5fps). Used about 2.4Gb memory. Increasing branch slowed it down further, reducing branch to 2 speeded it up to just over 6fps.

By comparison, splitting the video and running many separate single threaded encoding processes, or just using SetMTMode gives 20-25fps. SetMTMode uses a lot less memory.

The branch statement is actually not very useful, it is only suitable for spatial single-threaded plugins like TNLMeans, for temporal scripts/filters (especially complex script like QTGMC), the same frame will be repeatedly processed by multiple processes and cpu time will be wasted, decreasing speed. That's why I didn't mention it in OP.

[-Vit-]

The branch statement is actually not very useful, it is only suitable for spatial single-threaded plugins like TNLMeans, for temporal scripts/filters (especially complex script like QTGMC), the same frame will be repeatedly processed by multiple processes and cpu time will be wasted, decreasing speed. That's why I didn't mention it in OP.
Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

[06_taro]

Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

Because it is much easier to use selectevery and interleave. You don't need to care exactly how many frames in total.

[SAPikachu]

Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

Splitting the whole clip into big thunks doesn't make sense as we can't have parallelism in avs filter in this way. But I think we can split the clip into small thunks (32 frames each thunk for example), and use it with ThreadRequest. This may reduce duplicated processing, the speed may increase (but I'm afraid that this method will never be faster than SetMTMode since the overhead is much bigger) I need to make some new filters for that. Again, when I have time...

[pbristow]

Ooh! :)

So does this mean I can at last do:


LoadPlugin("MP_Pipeline.dll")
LoadPLugin("PB_3D_tools.dll")

AVIsource("some_anaglyph_3D_thing.avi")

MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)


...to create my cross-eye 3d versions from anaglyph 3D stuff in half the time? :)

I'm assuming "last" is passed though in the normal way. Have got the right idea about getting data back from the processes? Do LeftEye and RightEye need to be global variables, and if so, where should they be defined: Inside MP_Pipeline, outside, or both?

[Gavino]

...
MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)

I don't think that will work, since:
* Since each process have its own script environment, all script variables and loaded plugins won't be inherited, they must be re-initialized if needed
I assume this also applies the other way, so the outer script does not inherit any variables set by the processes.

I expect also that the script in quotes must return a clip, and yours doesn't.

[SAPikachu]

Ooh! :)

So does this mean I can at last do:


LoadPlugin("MP_Pipeline.dll")
LoadPLugin("PB_3D_tools.dll")

AVIsource("some_anaglyph_3D_thing.avi")

MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)


...to create my cross-eye 3d versions from anaglyph 3D stuff in half the time? :)

I'm assuming "last" is passed though in the normal way. Have got the right idea about getting data back from the processes? Do LeftEye and RightEye need to be global variables, and if so, where should they be defined: Inside MP_Pipeline, outside, or both?

Like @Gavino said, it is not possible now and your script is invalid to use in MP_Pipeline. But I will add a feature in next version, that will make it possible to use a workaround for this situation.

[pbristow]

*SLAPS OWN FOREHEAD* Of course.

Can the StackHorizontal be placed inside MP_Pipeline call, perhaps as a third process? No, again, we'd still need some way to represent the output of each of the other two processes to feed them into StackHorizontal.

How about this:


# Let's assume that all plugins are auto-loaded, for simplicity.

AVIsource("some_anaglyph_3D_thing.avi")

StackHorizontal( \
MP_Pipeline("""ExtractOneSide(Eye="Right")""", \
MP_Pipeline("""ExtractOneSide(Eye="Left")""" \
)


When presented with a single line in the internal script, does MP_Pipeline launch that as a separate process from the calling script?
Is the use of the seperator (i.e. "### ###") mandatory to cause a new process to be created?
Will adding a separator to one-line script (or extra separators in the general case) confuse the plugin, or will it just disregard any surplus ones?

Can see I'm gonna need to have a play with this one, as soon as I get time. :)

[SAPikachu]

*SLAPS OWN FOREHEAD* Of course.

Can the StackHorizontal be placed inside MP_Pipeline call, perhaps as a third process? No, again, we'd still need some way to represent the output of each of the other two processes to feed them into StackHorizontal.

How about this:


# Let's assume that all plugins are auto-loaded, for simplicity.

AVIsource("some_anaglyph_3D_thing.avi")

StackHorizontal( \
MP_Pipeline("""ExtractOneSide(Eye="Right")""", \
MP_Pipeline("""ExtractOneSide(Eye="Left")""" \
)


When presented with a single line in the internal script, does MP_Pipeline launch that as a separate process from the calling script?
Is the use of the seperator (i.e. "### ###") mandatory to cause a new process to be created?
Will adding a separator to one-line script (or extra separators in the general case) confuse the plugin, or will it just disregard any surplus ones?

Can see I'm gonna need to have a play with this one, as soon as I get time. :)

That's inspiring, I didn't think of this model before. Maybe you can try this:



StackHorizontal( \

MP_Pipeline("""

AVIsource("some_anaglyph_3D_thing.avi")
ExtractOneSide(Eye="Right")
### ###

""", \
MP_Pipeline("""

AVIsource("some_anaglyph_3D_thing.avi")
ExtractOneSide(Eye="Left")
### ###

""" \
)



You may also need to add ThreadRequest to make it process at full speed.

Yes, the "### ###" splitter is required to tell the plugin to spawn a new process. Script body can be empty though.

[pbristow]

SAPikachu: Heh, we posted almost simultaneously. :) Thanks for doing this, and for looking at ways to improve it. I think, if it can spawn a single processing chain (a single "mini-script") as a process separate from the calling script, then that's enough for me (see my revised example above).

But it would be cool also to provide a generalised framework for capturing the outputs of several processes: Perhaps add an "and finally..." section to MP_Pipeline's script template (with a different separator from the "start a separate process" one, and use generalised variables such as "Result1" "Result2" etc. to represent the outputs of the various processes, and which combines them into a single clip to pass back to the calling script...?

Just brainstorming. Ignore me if I'm being thick. :)

[SAPikachu]

SAPikachu: Heh, we posted almost simultaneously. :) Thanks for doing this, and for looking at ways to improve it. I think, if it can spawn a single processing chain (a single "mini-script") as a process separate from the calling script, then that's enough for me (see my revised example above).

But it would be cool also to provide a generalised framework for capturing the outputs of several processes: Perhaps add an "and finally..." section to MP_Pipeline's script template (with a different separator from the "start a separate process" one, and use generalised variables such as "Result1" "Result2" etc. to represent the outputs of the various processes, and which combines them into a single clip to pass back to the calling script...?

Just brainstorming. Ignore me if I'm being thick. :)

Actually MP_Pipeline is mainly for serial filter chains, since its main job is extracting part of script into external processes and chaining them together. For parallelized processing, it is hard to transfer clip variables back to main process (thread-safety problem), so I think we should stick with multiple MP_Pipeline instances like the script in my post above.

[pbristow]

Actually MP_Pipeline is mainly for serial filter chains, since its main job is extracting part of script into external processes and chaining them together. For parallelized processing, it is hard to transfer clip variables back to main process (thread-safety problem), so I think we should stick with multiple MP_Pipeline instances like the script in my post above.


Yeah. Thinking about it earlier, I couldn't actually think of an example of how you'd use what I suggested that couldn't be done without it.

That said, I'm thinking my StackHorizontal example isn't going to achieve any speed up anyway: StackHorizontal is in charge of which frames it calls for and when, and presumably waits until it has a complete frame from its first argument before asking for one from the second. So unless MP_Pipeline pre-requests frames from its sub-scripts, *ahead* of them being called for by StackHorizontal/the parent script, then there won't be any speed up. Does MP_Pipeline do that?

[Gavino]

How about this:
AVIsource("some_anaglyph_3D_thing.avi")

StackHorizontal( \
MP_Pipeline("""ExtractOneSide(Eye="Right")""", \
MP_Pipeline("""ExtractOneSide(Eye="Left")""" \
)
I'm not sure that would have worked as it stands, and I see that SAPikachu's example has an AviSource call inside each MP_Pipeline.

Am I right in thinking that MP_Pipeline is essentially a source filter, and takes no clip input, or does it make use of 'last' in some way?

[pbristow]

Tried it out: Looks like you're right Gavino. MP_Pipeline.dll starts up each new process as an instance of MP_Pipeline.dll.slave.exe, which executes the relevant segment of the script via its own instance of AviSynth. Those processes don't appear (at present) to have any way of receiving input from the parent instance of AviSynth other than the script section itself. So, the AviSource (or equivalent) call has to be in the script section that's passed to MP_Pipeline.

A consequence of that is that if there's any common processing that needs to be done to the video before the processing paths diverge (e.g., in my usual cases, denoising and resizing the picture to fit half the screen width), that pre-processing will have to be run twice... Unless you prepare a mezzanine file first with a separate script.

Common *post*-processing, on the other hand is easier: Just put it after the calls to MP_Pipeline.

SAPikachu, can you confirm/critique that analysis? :)

It works though! I did a test using an MVTools-based frame-doubler, as a simplified proxy for my 3D processing, and a dummy "minimal load encoder" - i.e. just cropping off most of the picture in VirtualDub and saving small rectangle of it, uncompressed. Using MP_Pipeline, instead of the same script without, finished in 58s rather than 95s. During processing, two processor cores were nearly fully used, rather than one.

[SAPikachu]

Tried it out: Looks like you're right Gavino. MP_Pipeline.dll starts up each new process as an instance of MP_Pipeline.dll.slave.exe, which executes the relevant segment of the script via its own instance of AviSynth. Those processes don't appear (at present) to have any way of receiving input from the parent instance of AviSynth other than the script section itself. So, the AviSource (or equivalent) call has to be in the script section that's passed to MP_Pipeline.

A consequence of that is that if there's any common processing that needs to be done to the video before the processing paths diverge (e.g., in my usual cases, denoising and resizing the picture to fit half the screen width), that pre-processing will have to be run twice... Unless you prepare a mezzanine file first with a separate script.

Common *post*-processing, on the other hand is easier: Just put it after the calls to MP_Pipeline.

SAPikachu, can you confirm/critique that analysis? :)

It works though! I did a test using an MVTools-based frame-doubler, as a simplified proxy for my 3D processing, and a dummy "minimal load encoder" - i.e. just cropping off most of the picture in VirtualDub and saving small rectangle of it, uncompressed. Using MP_Pipeline, instead of the same script without, finished in 58s rather than 95s. During processing, two processor cores were nearly fully used, rather than one.

Yes, your analysis is right. I intentionally left out clip input in MP_Pipeline. Although it is possible to accept clip input, but it may cause thread-safety problems.

In next version, I will add a script variable to slave AviSynth environment so that different slave process can be distinguished in script. And then you can use BRANCH statement to workaround this problem. But I think you can try ThreadRequest (http://forum.doom9.org/showthread.php?t=154886), in theory it can give bigger performance boost than MP_Pipeline, since overhead of multiprocessing is big.

[SAPikachu]

Released 0.2, forgot to reply yesterday.

@pbristow, you can try the new version like the snippet below, this script can eliminate duplicated preprocessing filters:


MP_Pipeline("""

SomeSource()
SomePreprocess()

### ###

MP_PIPELINE_BRANCH_ID == 0 ? ExtractOneSide(Eye="Left") : ExtractOneSide(Eye="Right")

SomeOtherProcess()

Interleave(last, last) # important!

### branch: 2

### ###

LeftEye = SelectEvery(2, 0)
RightEye = SelectEvery(2, 1)

StackHorizontal(RightEye, LeftEye)

""")


Of course, if the preprocessing filter doesn't consume much CPU time, you can directly use multiple MP_Pipeline instance like my earlier posts since that will be easier to write.

[SAPikachu]

Released 0.3. Thunked branching is implemented, but in my test it is pretty useless, since it usually decreases speed.

[SAPikachu]

Released 0.9. I found a way to support multiple clip in thread-safe manner and implemented it in this version.

[SAPikachu]

Released 0.11, greatly improved performance.

[pbristow]

Released 0.11, greatly improved performance.

I've just been trying out this new version, and getting some astonishing results.

For example, I tried running an unmodified script and timeing the execution of a short job. Then I modified the script to split teh three main stages into separate threads within MP_Pipeline (i.e. I used three "### ###" separators). As expected, the CPU usage went up and execution time went dwn, but not by a huge amount as one of the three stages involves a lot more processing than the other two.

Then I tried moving the later stages of processing out of teh MP_Pipeline call into the main script. This again was quicker than the original version, but not by much.

...But then I tried putting teh later stages back inside the MP_Pipeline call, *WITHOUT* the extra "### ###" separators, so that MP_Pipline was basically executing teh entire script in a single process, but separete from teh main AVIsynth process...

...and got teh fastest times of all! By a big margin!

I've compared the output from the various techniques and they're all producing the exact same video, but for some reason invoking my entire script via MP_Pipeline is much quicker - *without* increasing the CPU usage noticably - than running it directly in AVisynth. :scared:

I had trouble believing this, so I re-rane everything and timed them all again: Same result. Running the whole script via MP_Pipeline - with no other multithreading or multiprocessing being done - is quicker than any other method I've tried, using little or no extra CPU load than running the script directly. (Don't worry, I am remembering to add together the CPU sued by Virtualdub with that used by the MP_Piepline slave processes. :) )

Since nothing is apparently being "missed out" in the processing when I use MP_Pipeline, it follows that the normal, default mode of execution under avisynth is somehow being very wasteful of CPU cycles. Could this be a breakthrough in overall AVIsynth performance about to happen?

N.B. I'm invoking AVIsynth via VirtualDub 1.10.0. It's *possible* the inefficiency is something to do with the communication between VirtualDub and Avisynth, but it seems unlikely.

I'll try some more tests later on (after my work shift) with some different scripts, just in case it's an oddity of the particular plugins/filters I'm using, but in the meantime...Can anyone else confirm these results?

[pbristow]

OK, I just tried something on a hunch: I ran a simple script with just version() in it, first directly, then by wrapping an MP_Pipeline call round it.

Directly: AviSynth 2.58 MT v/6 (SVP edition), build: Oct 5 2010 [13:43:59]

Via MP_Pipeline: AviSynth 2.58 tsp MT version 5(mod seraphy), build: Jul 12 2009 [07:46:21]

MP_Pipeline is picking up a different installation of Avisynth to Virtualdub (one which seems to be twice as efficient!). How is it doing that?!?

Turns out I have two avisynth.dll files active, one in system32 and one in sysWOW64. The one in sysWOW64 is version 2.5.8.6, is only sized 396KB and seems to be the one being picked up by VirtualDub. MP_Pipeline is presumably picking up the one in system 32 (version 2.5.8.5, size 3.44MB).

Let's try removing the big fella from system32 (where it shouldn't be anyway)...
Nope, that's not solved it. *WHERE* is MP_Pipeline getting its version of avisynth from?

[SAPikachu]

OK, I just tried something on a hunch: I ran a simple script with just version() in it, first directly, then by wrapping an MP_Pipeline call round it.

Directly: AviSynth 2.58 MT v/6 (SVP edition), build: Oct 5 2010 [13:43:59]

Via MP_Pipeline: AviSynth 2.58 tsp MT version 5(mod seraphy), build: Jul 12 2009 [07:46:21]

MP_Pipeline is picking up a different installation of Avisynth to Virtualdub (one which seems to be twice as efficient!). How is it doing that?!?

Turns out I have two avisynth.dll files active, one in system32 and one in sysWOW64. The one in sysWOW64 is version 2.5.8.6, is only sized 396KB and seems to be the one being picked up by VirtualDub. MP_Pipeline is presumably picking up the one in system 32 (version 2.5.8.5, size 3.44MB).

Let's try removing the big fella from system32 (where it shouldn't be anyway)...
Nope, that's not solved it. *WHERE* is MP_Pipeline getting its version of avisynth from?

Assuming you didn't specify "###platform: win64", the avisynth.dll in System32 shouldn't matter, since that's 64bit version and won't be loaded by 32bit process. Maybe there is another avisynth.dll in DLL search path. You can find out path of the DLL using Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653). Just click the slave process in the list, and select "View > Lower Pane View > DLLs" and there will be a list of loaded DLLs, including avisynth one.

About the performance issue, can you post your tested scripts? By the way did you activate ### prefetch? That will also affect the speed.

[pbristow]

It turns out I had a copy of avisynth.dll sitting in the live AVIsynth *plugins* directory! MP_Pipeline was finding and using that, rather than the one in SysWOW64. After removing/renaming the one in plugins, MP_Pipeline picks up the same version as Virtualdub, and when I replaced that with the (more efficient) version I'd found in plugins, both Virtualdub and MP_Pipeline pick up the new version. So, everything is consistent now, I just need to settle on which version I should be running.

I'd suggest adding a note to your documentation for MP_Pipeline (if it isn't one there already) about the mechanism it uses to locate avisynth.dll, and the possible pitfalls if multiple versions exist on the system.

Anyway, now I've got that sorted out, I'll start testing MP_Pipeline again. I have to say, so far it's been the least troublesome multi-processing technique I've tried, especially where a mix of temporal and spatial filters are being used and/or multiple sources files need parallel processing (e.g. to compare the results of multiple test runs). :)

(BTW, No I wasn't using "### prefetch". Might try that next, after I've re-run the last batch of tests with my new, 100% less insane AVIsynth installation! :) )

[pbristow]

Since you asked, here's my script, in the simplest version that uses MP_Pipeline. The other test scripts have the same content, just moving some of the later lines out of the "DoStuffToChunk" function to the main script.
The "Chunk" idea is a left-over from where I was trying to process the file in two parts. The idea is to use Interleave() to force the two chunks to be processed simultaneously store the result in a mezzanine file, and then witha second script separate the results out using SelectOdd and SelectEven and join the chunks together in the correct order. I might try that again now that I know my earlier results are contaminated...


SetMemoryMax(96)

#
# Preamble:
#

SourceName = "TestClip.avi"
AviSource(SourceName)
TotFrames = FrameCount()
Midpoint = TotFrames / 2


#
# The Business:
#

function DoStuffToChunk(sourcename, start, end)
{
MP_Pipeline("""
SetMemoryMax(96)
source = """" + sourcename + """"
AviSource(source)
Trim(""" + string(start) + """, """ + string(end) + """)

SmartDecimate()

ConvertToRGB24

Red = ShowRed()
Blue = ShowBlue()
Green = ShowGreen().GeneralConvolution(matrix = " 0 0 0 1 1 0 0 0 0 ")

MergeRGB(Red,Green,Blue).ConvertToYV12

### ###
""")
}


ChunkA = DoStuffToChunk(sourcename, 1, TotFrames)
Return ChunkA

[06_taro]

Normally your VirtualDub test is getting YV12 data in the same thread of running avisynth script, while by using MP_Pipeline these two CPU costing tasks are separated into two threads - they are not competing for CPU resources in a single thread any more, and thus faster. If VirtualDub is not just running as video analysis or stream copy but with video encoding in the same time, the improvement in speed should be even more significant, unless your encoding process has already eaten up all the CPU resources without MPP.

[pbristow]

06_taro, are you replying to me? If so, then yes, that's the idea of multiprocessing in general, and that's the result I was expecting to see: Shorter execution time with higher CPU use when using the NP_Pipeline plugin to split the workload between processes. The confusing results I got were due to my messed up multiple AVIsynth installations. Just one reason why it's always good to test these things rather than relying on the theory: Sometimes there's things going on that the theory didn't take into account. :)

[SAPikachu]

It turns out I had a copy of avisynth.dll sitting in the live AVIsynth *plugins* directory! MP_Pipeline was finding and using that, rather than the one in SysWOW64. After removing/renaming the one in plugins, MP_Pipeline picks up the same version as Virtualdub, and when I replaced that with the (more efficient) version I'd found in plugins, both Virtualdub and MP_Pipeline pick up the new version. So, everything is consistent now, I just need to settle on which version I should be running.

I'd suggest adding a note to your documentation for MP_Pipeline (if it isn't one there already) about the mechanism it uses to locate avisynth.dll, and the possible pitfalls if multiple versions exist on the system.

Anyway, now I've got that sorted out, I'll start testing MP_Pipeline again. I have to say, so far it's been the least troublesome multi-processing technique I've tried, especially where a mix of temporal and spatial filters are being used and/or multiple sources files need parallel processing (e.g. to compare the results of multiple test runs). :)

(BTW, No I wasn't using "### prefetch". Might try that next, after I've re-run the last batch of tests with my new, 100% less insane AVIsynth installation! :) )

MP_Pipeline depend on Windows to load avisynth.dll for it, it won't do anything special. You have a risk if you don't install avisynth.dll into system folder anyways...

So you have already found out that the performance issue was not due to the script. :) Just a suggestion, I think you can try using it together with ### branch and ### prefetch, that can make the script even more parallel and should have greater speed improvement.

[pbristow]

Hi SAPikachu,

can you explain a bit more about how branch and prefetch function, or how they should be used?

I deduce from the MP_Pipeline_readme.avs script in the package that branch splits the video stream temporally (rather than spatially), like the traditional MT modes do, but in larger chunks (i.e. multiple frames per processing chunk). That (together with pre-fetch) should help with some temporal filters, i.e. the ones that get bogged down in duplicated effort under Set_MT_Mode: The only duplication will be around the boundaries of each chunk, so the bigger the chunk the better, yes?

If you have several "### ###" process dividers, does it make any sense to use "### prefetch" in more than one place, or does one pre-fetch instruction apply to all processes?

Continuing to test and play... :)

[SAPikachu]

Hi SAPikachu,

can you explain a bit more about how branch and prefetch function, or how they should be used?

I deduce from the MP_Pipeline_readme.avs script in the package that branch splits the video stream temporally (rather than spatially), like the traditional MT modes do, but in larger chunks (i.e. multiple frames per processing chunk). That (together with pre-fetch) should help with some temporal filters, i.e. the ones that get bogged down in duplicated effort under Set_MT_Mode: The only duplication will be around the boundaries of each chunk, so the bigger the chunk the better, yes?

If you have several "### ###" process dividers, does it make any sense to use "### prefetch" in more than one place, or does one pre-fetch instruction apply to all processes?

Continuing to test and play... :)

branch basically does what you said, but normally bigger thunk may not be better, since you need to use prefetch to preload all parts parallelly, if thunk size is too big, it becomes impossible to prefetch a full thunk (due to memory constraint) before the active thunk fully consumed by downstream.

prefetch statement only applies to the containing script block, so you usually need to add it to all script blocks. Be very careful if downstream script blocks have temporal filters, if prefetch cache size is not big enough, it may cause frequent seeking and cache thrashing, the speed will be greatly decreased because of that.

Happy experimenting. :)

[pbristow]

Just checking my understanding again:


MP_Pipeline("""
AviSource("P:\Testfile.avi")
### ###
InterFrame (FlowPath="C:\Program Files (x86)\AviSynth\Plugins\Dependencies\", Preset="Fast", NewNum=60000, NewDen=1001)
### prefetch: 20, 4
### ###
TDecimate(mode=0,cycleR=2,cycle=10)
### ###
""")


In this example, prefetch is placed at the end of the code block containing Interframe. Am I correct in assuming this will pre-fetch frames from Interframe, so that they will be ready to supply later to TDecimate? Or (less intuitively), does it mean that Interframe will receive frames that were pre-fetched from Avisource?

(P.S. Before anyone pipes up, I know that I'm using Interframe and TDecimate "the wrong way round". It's actually the best way for the job I'm doing! :) )

Hmm... This is turning into a usage rather than development thread. Do you want to move this discussion to the other forum?

[SAPikachu]

Just checking my understanding again:


MP_Pipeline("""
AviSource("P:\Testfile.avi")
### ###
InterFrame (FlowPath="C:\Program Files (x86)\AviSynth\Plugins\Dependencies\", Preset="Fast", NewNum=60000, NewDen=1001)
### prefetch: 20, 4
### ###
TDecimate(mode=0,cycleR=2,cycle=10)
### ###
""")


In this example, prefetch is placed at the end of the code block containing Interframe. Am I correct in assuming this will pre-fetch frames from Interframe, so that they will be ready to supply later to TDecimate? Or (less intuitively), does it mean that Interframe will receive frames that were pre-fetched from Avisource?

(P.S. Before anyone pipes up, I know that I'm using Interframe and TDecimate "the wrong way round". It's actually the best way for the job I'm doing! :) )

Hmm... This is turning into a usage rather than development thread. Do you want to move this discussion to the other forum?

Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.

I think it's OK to keep the discussion here, since there is few development talk now. :)

[pbristow]

[QUOTE=SAPikachu;1567071]Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.
QUOTE]

I understand that it applies to the containing block; the question is, to which *end* of the block: The input, or the output? :)

Pre-fetch is something that happens at the boundary between things, y'see. So is it pre-fetching *for* the current block, from the previous, or *from* the current block, for the next?

[SAPikachu]

[QUOTE=SAPikachu;1567071]Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.
QUOTE]

I understand that it applies to the containing block; the question is, to which *end* of the block: The input, or the output? :)

Pre-fetch is something that happens at the boundary between things, y'see. So is it pre-fetching *for* the current block, from the previous, or *from* the current block, for the next?

Sorry, I meant prefetch *from* the current block, output from the current block is prefetched and cached by the filter, for next block to use.

[pbristow]

[QUOTE=pbristow;1567133]

Sorry, I meant prefetch *from* the current block, output from the current block is prefetched and cached by the filter, for next block to use.

Thanks. That helps with tuning the memory usage. :)

One thing: Audio doesn't seem to be carried through MP_Pipeline anymore. I'm having to re-dub the audio (using AudioDub(...) ) in the main script after the MP_Pipeline call, as none comes back from MP_Pipeline. Do you get the same, or am I doing something wrong? As far as I can see none of the filters I'm using should kill the audio, and if I move the AudioDub() call inside MP_Pipeline it doesn't work, but outside it does.

[SAPikachu]

Thanks. That helps with tuning the memory usage. :)

One thing: Audio doesn't seem to be carried through MP_Pipeline anymore. I'm having to re-dub the audio (using AudioDub(...) ) in the main script after the MP_Pipeline call, as none comes back from MP_Pipeline. Do you get the same, or am I doing something wrong? As far as I can see none of the filters I'm using should kill the audio, and if I move the AudioDub() call inside MP_Pipeline it doesn't work, but outside it does.

I forgot to note that in the readme, audio is intentionally unsupported, because it is complex to handle and may slow down the whole filter chain. Just AudioDub() after MP_Pipeline call. :P

[pbristow]

I forgot to note that in the readme, audio is intentionally unsupported, because it is complex to handle and may slow down the whole filter chain. Just AudioDub() after MP_Pipeline call. :P

OK, thanks. At least I know I'm not going deaf. :)

Overall impression: With the definite benefit from the prefetch facility (thanks for adding that! :) ), I'm finding it *most* useful in my comparison scripts, where I run the same basic pre-processing on several source files in parallel and then compare the results. Processing each source file via it's own slave process using MP_Pipeline and then combining the results in the main script works smoothly and efficiently every time, utilising up to 99% CPU and delivering speed gains to match. The only gotcha is memory: It's wise to use SetMemoryMax inside the MP_Pipleline call, with a parameter that won't overflow your free memory when all those parallel processes are demanding memory at once. (Formula: Take the average free memory (in Megabytes) you have when you first invoke VirtualDub (or whatever app calls AViSynth); divide that number by the number of parallel processes you're going to run; Shave off a few tens of megabytes for safety and/or uses that "SetMemoryMax" doesn't cover.)

With more linear AviSynth scripts, the picture is not so straightforward. Sometimes using MP_Pipeline gives the best speed-up; sometimes SetMTmode() does. Sometimes tweaking the prefetch and branch parameters makes all the difference; sometimes it makes virtually none. Like all multi-threading and multiprocessing techniques, it really depends on knowing where and how to use it - and that means knowing the characteristics of the filters in your script and thus how they will be affected by the various different ways of dividing up the work they do - to get the best results. And sometimes, no matter what combination of tricks (MT_Pipline, MT(), SetMTMode(), etc...) and parameters I try, I can't get above 70% CPU utilisation, and that just has to be accepted. Some bottleneck in my machine prevents enough data getting to the CPU to be processed any faster, I guess. (Time to start saving up for that new motherboard...?)

Big, big thanks to SAPikachu for a really useful plug-in. :)