MP_Pipeline 0.18 - run parts of avisynth script in external processes [2014-04-06]

[SAPikachu]

This plugin is originally written for my friend to work-around the 2GB 4GB problem of 32-bit process. (Well, also for fun. :) ) Don't know whether it is useful for others, but I decided to post it here anyways.

As of 0.11, overhead of the plugin is much smaller, it may be possible to use it to speed up more scripts.

Change log:

0.18
* Fix deadlock when exported clip is consumed by multiple script block

0.17
* Properly terminate slave processes when initialization fails
* Fix "Not a clip" error when using ### inherit and the last block is empty

0.16
* Try to silent all error dialogs on exit of slave process
* Slave process shouldn't be stuck on exit anymore, it will terminate itself if it doesn't exit cleanly after 15 seconds
* Fix ### branch statement, previously it incorrectly rejects some input

0.15
* Properly clean script environment up on exit
* Allow using different avisynth dll to run script block (### dll)

0.14
* Fixed another crashing bug

0.13
* Fixed a bug that causes occasional crashing

0.12
* Fixed a problem that makes scripts unable to be loaded in some programs

0.11
* Greatly improved performance, maximum 80% overhead reduction
* New feature: Ability to lock threads to cores, may improve performance in some cases
* (0.10 is skipped to avoid confusion)

0.9
* New feature: Frame prefetching
* New feature: Exporting multiple clip variables in a single process
* New feature: Code block can be shared between processes

0.3
* Binaries in the x86 folder are in correct version now (In 0.2 the win64 slave is actually win32...)
* Integrated a patched TCPDeliver, no longer depend on the external one
* Fixed random crash when filter chain is destroyed
* Thunked branching

0.2
* x64 support (please copy TCPDeliver.dll in the package to respective plugin folder)
* x86/x64 mixed slave process (requires both x86/x64 version of AviSynth to be installed)
* Add a script variable in branch slave process, make it distinguishable in script


Limitations:
* Since each process have its own script environment, all script variables and loaded plugins won't be inherited, they must be re-initialized if needed
* Due to the limitation above, manually-loaded plugins and imported scripts need to be reloaded/re-imported before they can be used in new process (Or use inherited script snippet, please see MP_Pipeline_readme.avs for details)
* Clips before MP_Pipeline will be ignored
* Audio is not supported
* Every script block must return a clip (i.e. "last" must be a clip), otherwise MPP will raise this error: Invalid arguments to function "MPP_PrepareDownstreamClip"

Binary: http://nmm.me/z6
Source code: https://github.com/SAPikachu/MP_Pipeline/tree/0.18

Some example:

1. Basic usage:

MP_Pipeline("""
FFVideoSource("SomeVideo")
QTGMC()
### prefetch: 16, 0
### ###
""")
MCTD()

# MCTD and QTGMC will be run parallelly in 2 separate processes


2. Speed up MCTD at the cost of memory

# Must be 64bit system with at least 8GB memory to run this script
MP_Pipeline("""

# This may be smaller, but I only tested this number
SetMemoryMax(3072)

FFVideoSource("SomeVideo")
MCTD(settings="high")
### prefetch: 16, 0
### ###
""")

# Some time ago I used a script similar to this one for encoding, it is about 20% ~ 30% faster than plain MCTD.


3. Branching

MP_Pipeline("""
FFVideoSource("SomeVideo")
TNLMeans()
### prefetch: 16, 0
### branch: 4
### ###
""")

# TNLMeans will be run in 4 processes with branching (please see example script in the package for details)


4. Frame caching

MP_Pipeline("""
FFVideoSource("SomeUnseekableVideo", seekmode=-1)
TNLMeans()
### prefetch: 32, 24
# It is important to use a big backward cache since we can't seek

### ###

MCTD()

""")


Please see example script in the binary package for some other usage and setting explanations.

[TheRyuu]

work-around the 4GB problem of 32-bit process.

ftfy.

[SAPikachu]

ftfy.

User processes can only use 2GB of full address space, don't they? (well... actually 3GB on some conditions, but that's a special case)

[SEt]

Actually 4GB on 64 bit OS.

[SAPikachu]

Actually 4GB on 64 bit OS.

Didn't notice that until read this. Learned something today, thanks. :)

[kemuri-_9]

Actually 4GB on 64 bit OS.

generally everything has to be compiled with large address awareness for the 32bit binaries (executable and dlls) to really allow addressing over 2GB of memory.

as this is also not usually a default build option iirc, most things don't have it enabled, preventing beyond 2GB of addressable memory.

[LoRd_MuldeR]

So this basically is an AVS2YUV clone, but not as a stand-alone application, but as an Avisynth plug-in?

I think this would be particularly useful to load 32-Bit plugins (that don't have 64-Bit equivalents) into a 64-Bit Avisynth environment. Or vice versa.

Is that supported/intended?

[Gavino]

generally everything has to be compiled with large address awareness for the 32bit binaries (executable and dlls) to really allow addressing over 2GB of memory.
I thought it was just executables (not dlls). Thus Avisynth will benefit from increased memory if used by a client that has been built as 'large address aware'.

[LoRd_MuldeR]

I thought it was just executables (not dlls). Thus Avisynth will benefit from increased memory if used by a client that has been built as 'large address aware'.

Right:
http://blogs.msdn.com/b/oldnewthing/archive/2010/09/22/10065933.aspx

But then loading a DLL into some "LARGEADDRESSAWARE" process might break it, if the code in that DLL isn't prepared to deal with addresses beyond 2 GB.

[SAPikachu]

So this basically is an AVS2YUV clone, but not as a stand-alone application, but as an Avisynth plug-in?

I think this would be particularly useful to load 32-Bit plugins (that don't have 64-Bit equivalents) into a 64-Bit Avisynth environment. Or vice versa.

Is that supported/intended?

It is functionally similar to avs2yuv, but with some additional features like multiple levels of pipeline.

That is not my original intention, but it is interesting. It only supports x86 now, I will add x64 and mixed script environment support later when I have time.

[kolak]

Can we use this to divide file (using trim) to few parts and run on each one (in seperate process) QTGMC and put them together at the end?


Andrew

[06_taro]

Now add large memory aware flag to exceed 2GB limit in avs4x264mod (http://forum.doom9.org/showthread.php?t=162656).

[-Vit-]

Can we use this to divide file (using trim) to few parts and run on each one (in seperate process) QTGMC and put them together at the end?
It's an interesting plugin, but doesn't seem to help for that unless I'm missing something. I tried this on some SD footage:

MP_Pipeline("""

WhateverSource("Some\Source")

### ###

QTGMC("Placebo")

### branch: 4

### ###

""")


Worked OK, ran five slave processes and produced the correct result. However, it was slower than single threaded (single threaded is 6fps, this script was 5fps). Used about 2.4Gb memory. Increasing branch slowed it down further, reducing branch to 2 speeded it up to just over 6fps.

By comparison, splitting the video and running many separate single threaded encoding processes, or just using SetMTMode gives 20-25fps. SetMTMode uses a lot less memory.

[kolak]

Hmmm- shame.

I'm forced to run few instances for HD- not a big deal, but if it could be automated than it would be easier.

[SAPikachu]

It's an interesting plugin, but doesn't seem to help for that unless I'm missing something. I tried this on some SD footage:

MP_Pipeline("""

WhateverSource("Some\Source")

### ###

QTGMC("Placebo")

### branch: 4

### ###

""")


Worked OK, ran five slave processes and produced the correct result. However, it was slower than single threaded (single threaded is 6fps, this script was 5fps). Used about 2.4Gb memory. Increasing branch slowed it down further, reducing branch to 2 speeded it up to just over 6fps.

By comparison, splitting the video and running many separate single threaded encoding processes, or just using SetMTMode gives 20-25fps. SetMTMode uses a lot less memory.

The branch statement is actually not very useful, it is only suitable for spatial single-threaded plugins like TNLMeans, for temporal scripts/filters (especially complex script like QTGMC), the same frame will be repeatedly processed by multiple processes and cpu time will be wasted, decreasing speed. That's why I didn't mention it in OP.

[-Vit-]

The branch statement is actually not very useful, it is only suitable for spatial single-threaded plugins like TNLMeans, for temporal scripts/filters (especially complex script like QTGMC), the same frame will be repeatedly processed by multiple processes and cpu time will be wasted, decreasing speed. That's why I didn't mention it in OP.
Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

[06_taro]

Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

Because it is much easier to use selectevery and interleave. You don't need to care exactly how many frames in total.

[SAPikachu]

Ah yes, because it splits into interleaved sequences... Would it be difficult to have it split into several contiguous chunks instead? Or is there some other reason not to do that?

Splitting the whole clip into big thunks doesn't make sense as we can't have parallelism in avs filter in this way. But I think we can split the clip into small thunks (32 frames each thunk for example), and use it with ThreadRequest. This may reduce duplicated processing, the speed may increase (but I'm afraid that this method will never be faster than SetMTMode since the overhead is much bigger) I need to make some new filters for that. Again, when I have time...

[pbristow]

Ooh! :)

So does this mean I can at last do:


LoadPlugin("MP_Pipeline.dll")
LoadPLugin("PB_3D_tools.dll")

AVIsource("some_anaglyph_3D_thing.avi")

MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)


...to create my cross-eye 3d versions from anaglyph 3D stuff in half the time? :)

I'm assuming "last" is passed though in the normal way. Have got the right idea about getting data back from the processes? Do LeftEye and RightEye need to be global variables, and if so, where should they be defined: Inside MP_Pipeline, outside, or both?

[Gavino]

...
MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)

I don't think that will work, since:
* Since each process have its own script environment, all script variables and loaded plugins won't be inherited, they must be re-initialized if needed
I assume this also applies the other way, so the outer script does not inherit any variables set by the processes.

I expect also that the script in quotes must return a clip, and yours doesn't.

[SAPikachu]

Ooh! :)

So does this mean I can at last do:


LoadPlugin("MP_Pipeline.dll")
LoadPLugin("PB_3D_tools.dll")

AVIsource("some_anaglyph_3D_thing.avi")

MP_Pipeline("""
global LeftEye = ExtractOneSide(Eye="Left")
### ###
global RightEye = ExtractOneSide(Eye="Right")
""")

StackHorizontal(RightEye,LeftEye)


...to create my cross-eye 3d versions from anaglyph 3D stuff in half the time? :)

I'm assuming "last" is passed though in the normal way. Have got the right idea about getting data back from the processes? Do LeftEye and RightEye need to be global variables, and if so, where should they be defined: Inside MP_Pipeline, outside, or both?

Like @Gavino said, it is not possible now and your script is invalid to use in MP_Pipeline. But I will add a feature in next version, that will make it possible to use a workaround for this situation.

[pbristow]

*SLAPS OWN FOREHEAD* Of course.

Can the StackHorizontal be placed inside MP_Pipeline call, perhaps as a third process? No, again, we'd still need some way to represent the output of each of the other two processes to feed them into StackHorizontal.

How about this:


# Let's assume that all plugins are auto-loaded, for simplicity.

AVIsource("some_anaglyph_3D_thing.avi")

StackHorizontal( \
MP_Pipeline("""ExtractOneSide(Eye="Right")""", \
MP_Pipeline("""ExtractOneSide(Eye="Left")""" \
)


When presented with a single line in the internal script, does MP_Pipeline launch that as a separate process from the calling script?
Is the use of the seperator (i.e. "### ###") mandatory to cause a new process to be created?
Will adding a separator to one-line script (or extra separators in the general case) confuse the plugin, or will it just disregard any surplus ones?

Can see I'm gonna need to have a play with this one, as soon as I get time. :)

[SAPikachu]

*SLAPS OWN FOREHEAD* Of course.

Can the StackHorizontal be placed inside MP_Pipeline call, perhaps as a third process? No, again, we'd still need some way to represent the output of each of the other two processes to feed them into StackHorizontal.

How about this:


# Let's assume that all plugins are auto-loaded, for simplicity.

AVIsource("some_anaglyph_3D_thing.avi")

StackHorizontal( \
MP_Pipeline("""ExtractOneSide(Eye="Right")""", \
MP_Pipeline("""ExtractOneSide(Eye="Left")""" \
)


When presented with a single line in the internal script, does MP_Pipeline launch that as a separate process from the calling script?
Is the use of the seperator (i.e. "### ###") mandatory to cause a new process to be created?
Will adding a separator to one-line script (or extra separators in the general case) confuse the plugin, or will it just disregard any surplus ones?

Can see I'm gonna need to have a play with this one, as soon as I get time. :)

That's inspiring, I didn't think of this model before. Maybe you can try this:



StackHorizontal( \

MP_Pipeline("""

AVIsource("some_anaglyph_3D_thing.avi")
ExtractOneSide(Eye="Right")
### ###

""", \
MP_Pipeline("""

AVIsource("some_anaglyph_3D_thing.avi")
ExtractOneSide(Eye="Left")
### ###

""" \
)



You may also need to add ThreadRequest to make it process at full speed.

Yes, the "### ###" splitter is required to tell the plugin to spawn a new process. Script body can be empty though.

[pbristow]

SAPikachu: Heh, we posted almost simultaneously. :) Thanks for doing this, and for looking at ways to improve it. I think, if it can spawn a single processing chain (a single "mini-script") as a process separate from the calling script, then that's enough for me (see my revised example above).

But it would be cool also to provide a generalised framework for capturing the outputs of several processes: Perhaps add an "and finally..." section to MP_Pipeline's script template (with a different separator from the "start a separate process" one, and use generalised variables such as "Result1" "Result2" etc. to represent the outputs of the various processes, and which combines them into a single clip to pass back to the calling script...?

Just brainstorming. Ignore me if I'm being thick. :)

[SAPikachu]

SAPikachu: Heh, we posted almost simultaneously. :) Thanks for doing this, and for looking at ways to improve it. I think, if it can spawn a single processing chain (a single "mini-script") as a process separate from the calling script, then that's enough for me (see my revised example above).

But it would be cool also to provide a generalised framework for capturing the outputs of several processes: Perhaps add an "and finally..." section to MP_Pipeline's script template (with a different separator from the "start a separate process" one, and use generalised variables such as "Result1" "Result2" etc. to represent the outputs of the various processes, and which combines them into a single clip to pass back to the calling script...?

Just brainstorming. Ignore me if I'm being thick. :)

Actually MP_Pipeline is mainly for serial filter chains, since its main job is extracting part of script into external processes and chaining them together. For parallelized processing, it is hard to transfer clip variables back to main process (thread-safety problem), so I think we should stick with multiple MP_Pipeline instances like the script in my post above.

[pbristow]

Actually MP_Pipeline is mainly for serial filter chains, since its main job is extracting part of script into external processes and chaining them together. For parallelized processing, it is hard to transfer clip variables back to main process (thread-safety problem), so I think we should stick with multiple MP_Pipeline instances like the script in my post above.


Yeah. Thinking about it earlier, I couldn't actually think of an example of how you'd use what I suggested that couldn't be done without it.

That said, I'm thinking my StackHorizontal example isn't going to achieve any speed up anyway: StackHorizontal is in charge of which frames it calls for and when, and presumably waits until it has a complete frame from its first argument before asking for one from the second. So unless MP_Pipeline pre-requests frames from its sub-scripts, *ahead* of them being called for by StackHorizontal/the parent script, then there won't be any speed up. Does MP_Pipeline do that?

[Gavino]

How about this:
AVIsource("some_anaglyph_3D_thing.avi")

StackHorizontal( \
MP_Pipeline("""ExtractOneSide(Eye="Right")""", \
MP_Pipeline("""ExtractOneSide(Eye="Left")""" \
)
I'm not sure that would have worked as it stands, and I see that SAPikachu's example has an AviSource call inside each MP_Pipeline.

Am I right in thinking that MP_Pipeline is essentially a source filter, and takes no clip input, or does it make use of 'last' in some way?

[pbristow]

Tried it out: Looks like you're right Gavino. MP_Pipeline.dll starts up each new process as an instance of MP_Pipeline.dll.slave.exe, which executes the relevant segment of the script via its own instance of AviSynth. Those processes don't appear (at present) to have any way of receiving input from the parent instance of AviSynth other than the script section itself. So, the AviSource (or equivalent) call has to be in the script section that's passed to MP_Pipeline.

A consequence of that is that if there's any common processing that needs to be done to the video before the processing paths diverge (e.g., in my usual cases, denoising and resizing the picture to fit half the screen width), that pre-processing will have to be run twice... Unless you prepare a mezzanine file first with a separate script.

Common *post*-processing, on the other hand is easier: Just put it after the calls to MP_Pipeline.

SAPikachu, can you confirm/critique that analysis? :)

It works though! I did a test using an MVTools-based frame-doubler, as a simplified proxy for my 3D processing, and a dummy "minimal load encoder" - i.e. just cropping off most of the picture in VirtualDub and saving small rectangle of it, uncompressed. Using MP_Pipeline, instead of the same script without, finished in 58s rather than 95s. During processing, two processor cores were nearly fully used, rather than one.

[SAPikachu]

Tried it out: Looks like you're right Gavino. MP_Pipeline.dll starts up each new process as an instance of MP_Pipeline.dll.slave.exe, which executes the relevant segment of the script via its own instance of AviSynth. Those processes don't appear (at present) to have any way of receiving input from the parent instance of AviSynth other than the script section itself. So, the AviSource (or equivalent) call has to be in the script section that's passed to MP_Pipeline.

A consequence of that is that if there's any common processing that needs to be done to the video before the processing paths diverge (e.g., in my usual cases, denoising and resizing the picture to fit half the screen width), that pre-processing will have to be run twice... Unless you prepare a mezzanine file first with a separate script.

Common *post*-processing, on the other hand is easier: Just put it after the calls to MP_Pipeline.

SAPikachu, can you confirm/critique that analysis? :)

It works though! I did a test using an MVTools-based frame-doubler, as a simplified proxy for my 3D processing, and a dummy "minimal load encoder" - i.e. just cropping off most of the picture in VirtualDub and saving small rectangle of it, uncompressed. Using MP_Pipeline, instead of the same script without, finished in 58s rather than 95s. During processing, two processor cores were nearly fully used, rather than one.

Yes, your analysis is right. I intentionally left out clip input in MP_Pipeline. Although it is possible to accept clip input, but it may cause thread-safety problems.

In next version, I will add a script variable to slave AviSynth environment so that different slave process can be distinguished in script. And then you can use BRANCH statement to workaround this problem. But I think you can try ThreadRequest (http://forum.doom9.org/showthread.php?t=154886), in theory it can give bigger performance boost than MP_Pipeline, since overhead of multiprocessing is big.

[SAPikachu]

Released 0.2, forgot to reply yesterday.

@pbristow, you can try the new version like the snippet below, this script can eliminate duplicated preprocessing filters:


MP_Pipeline("""

SomeSource()
SomePreprocess()

### ###

MP_PIPELINE_BRANCH_ID == 0 ? ExtractOneSide(Eye="Left") : ExtractOneSide(Eye="Right")

SomeOtherProcess()

Interleave(last, last) # important!

### branch: 2

### ###

LeftEye = SelectEvery(2, 0)
RightEye = SelectEvery(2, 1)

StackHorizontal(RightEye, LeftEye)

""")


Of course, if the preprocessing filter doesn't consume much CPU time, you can directly use multiple MP_Pipeline instance like my earlier posts since that will be easier to write.

[SAPikachu]

Released 0.3. Thunked branching is implemented, but in my test it is pretty useless, since it usually decreases speed.

[SAPikachu]

Released 0.9. I found a way to support multiple clip in thread-safe manner and implemented it in this version.

[SAPikachu]

Released 0.11, greatly improved performance.

[pbristow]

Released 0.11, greatly improved performance.

I've just been trying out this new version, and getting some astonishing results.

For example, I tried running an unmodified script and timeing the execution of a short job. Then I modified the script to split teh three main stages into separate threads within MP_Pipeline (i.e. I used three "### ###" separators). As expected, the CPU usage went up and execution time went dwn, but not by a huge amount as one of the three stages involves a lot more processing than the other two.

Then I tried moving the later stages of processing out of teh MP_Pipeline call into the main script. This again was quicker than the original version, but not by much.

...But then I tried putting teh later stages back inside the MP_Pipeline call, *WITHOUT* the extra "### ###" separators, so that MP_Pipline was basically executing teh entire script in a single process, but separete from teh main AVIsynth process...

...and got teh fastest times of all! By a big margin!

I've compared the output from the various techniques and they're all producing the exact same video, but for some reason invoking my entire script via MP_Pipeline is much quicker - *without* increasing the CPU usage noticably - than running it directly in AVisynth. :scared:

I had trouble believing this, so I re-rane everything and timed them all again: Same result. Running the whole script via MP_Pipeline - with no other multithreading or multiprocessing being done - is quicker than any other method I've tried, using little or no extra CPU load than running the script directly. (Don't worry, I am remembering to add together the CPU sued by Virtualdub with that used by the MP_Piepline slave processes. :) )

Since nothing is apparently being "missed out" in the processing when I use MP_Pipeline, it follows that the normal, default mode of execution under avisynth is somehow being very wasteful of CPU cycles. Could this be a breakthrough in overall AVIsynth performance about to happen?

N.B. I'm invoking AVIsynth via VirtualDub 1.10.0. It's *possible* the inefficiency is something to do with the communication between VirtualDub and Avisynth, but it seems unlikely.

I'll try some more tests later on (after my work shift) with some different scripts, just in case it's an oddity of the particular plugins/filters I'm using, but in the meantime...Can anyone else confirm these results?

[pbristow]

OK, I just tried something on a hunch: I ran a simple script with just version() in it, first directly, then by wrapping an MP_Pipeline call round it.

Directly: AviSynth 2.58 MT v/6 (SVP edition), build: Oct 5 2010 [13:43:59]

Via MP_Pipeline: AviSynth 2.58 tsp MT version 5(mod seraphy), build: Jul 12 2009 [07:46:21]

MP_Pipeline is picking up a different installation of Avisynth to Virtualdub (one which seems to be twice as efficient!). How is it doing that?!?

Turns out I have two avisynth.dll files active, one in system32 and one in sysWOW64. The one in sysWOW64 is version 2.5.8.6, is only sized 396KB and seems to be the one being picked up by VirtualDub. MP_Pipeline is presumably picking up the one in system 32 (version 2.5.8.5, size 3.44MB).

Let's try removing the big fella from system32 (where it shouldn't be anyway)...
Nope, that's not solved it. *WHERE* is MP_Pipeline getting its version of avisynth from?

[SAPikachu]

OK, I just tried something on a hunch: I ran a simple script with just version() in it, first directly, then by wrapping an MP_Pipeline call round it.

Directly: AviSynth 2.58 MT v/6 (SVP edition), build: Oct 5 2010 [13:43:59]

Via MP_Pipeline: AviSynth 2.58 tsp MT version 5(mod seraphy), build: Jul 12 2009 [07:46:21]

MP_Pipeline is picking up a different installation of Avisynth to Virtualdub (one which seems to be twice as efficient!). How is it doing that?!?

Turns out I have two avisynth.dll files active, one in system32 and one in sysWOW64. The one in sysWOW64 is version 2.5.8.6, is only sized 396KB and seems to be the one being picked up by VirtualDub. MP_Pipeline is presumably picking up the one in system 32 (version 2.5.8.5, size 3.44MB).

Let's try removing the big fella from system32 (where it shouldn't be anyway)...
Nope, that's not solved it. *WHERE* is MP_Pipeline getting its version of avisynth from?

Assuming you didn't specify "###platform: win64", the avisynth.dll in System32 shouldn't matter, since that's 64bit version and won't be loaded by 32bit process. Maybe there is another avisynth.dll in DLL search path. You can find out path of the DLL using Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653). Just click the slave process in the list, and select "View > Lower Pane View > DLLs" and there will be a list of loaded DLLs, including avisynth one.

About the performance issue, can you post your tested scripts? By the way did you activate ### prefetch? That will also affect the speed.

[pbristow]

It turns out I had a copy of avisynth.dll sitting in the live AVIsynth *plugins* directory! MP_Pipeline was finding and using that, rather than the one in SysWOW64. After removing/renaming the one in plugins, MP_Pipeline picks up the same version as Virtualdub, and when I replaced that with the (more efficient) version I'd found in plugins, both Virtualdub and MP_Pipeline pick up the new version. So, everything is consistent now, I just need to settle on which version I should be running.

I'd suggest adding a note to your documentation for MP_Pipeline (if it isn't one there already) about the mechanism it uses to locate avisynth.dll, and the possible pitfalls if multiple versions exist on the system.

Anyway, now I've got that sorted out, I'll start testing MP_Pipeline again. I have to say, so far it's been the least troublesome multi-processing technique I've tried, especially where a mix of temporal and spatial filters are being used and/or multiple sources files need parallel processing (e.g. to compare the results of multiple test runs). :)

(BTW, No I wasn't using "### prefetch". Might try that next, after I've re-run the last batch of tests with my new, 100% less insane AVIsynth installation! :) )

[pbristow]

Since you asked, here's my script, in the simplest version that uses MP_Pipeline. The other test scripts have the same content, just moving some of the later lines out of the "DoStuffToChunk" function to the main script.
The "Chunk" idea is a left-over from where I was trying to process the file in two parts. The idea is to use Interleave() to force the two chunks to be processed simultaneously store the result in a mezzanine file, and then witha second script separate the results out using SelectOdd and SelectEven and join the chunks together in the correct order. I might try that again now that I know my earlier results are contaminated...


SetMemoryMax(96)

#
# Preamble:
#

SourceName = "TestClip.avi"
AviSource(SourceName)
TotFrames = FrameCount()
Midpoint = TotFrames / 2


#
# The Business:
#

function DoStuffToChunk(sourcename, start, end)
{
MP_Pipeline("""
SetMemoryMax(96)
source = """" + sourcename + """"
AviSource(source)
Trim(""" + string(start) + """, """ + string(end) + """)

SmartDecimate()

ConvertToRGB24

Red = ShowRed()
Blue = ShowBlue()
Green = ShowGreen().GeneralConvolution(matrix = " 0 0 0 1 1 0 0 0 0 ")

MergeRGB(Red,Green,Blue).ConvertToYV12

### ###
""")
}


ChunkA = DoStuffToChunk(sourcename, 1, TotFrames)
Return ChunkA

[06_taro]

Normally your VirtualDub test is getting YV12 data in the same thread of running avisynth script, while by using MP_Pipeline these two CPU costing tasks are separated into two threads - they are not competing for CPU resources in a single thread any more, and thus faster. If VirtualDub is not just running as video analysis or stream copy but with video encoding in the same time, the improvement in speed should be even more significant, unless your encoding process has already eaten up all the CPU resources without MPP.

[pbristow]

06_taro, are you replying to me? If so, then yes, that's the idea of multiprocessing in general, and that's the result I was expecting to see: Shorter execution time with higher CPU use when using the NP_Pipeline plugin to split the workload between processes. The confusing results I got were due to my messed up multiple AVIsynth installations. Just one reason why it's always good to test these things rather than relying on the theory: Sometimes there's things going on that the theory didn't take into account. :)

[SAPikachu]

It turns out I had a copy of avisynth.dll sitting in the live AVIsynth *plugins* directory! MP_Pipeline was finding and using that, rather than the one in SysWOW64. After removing/renaming the one in plugins, MP_Pipeline picks up the same version as Virtualdub, and when I replaced that with the (more efficient) version I'd found in plugins, both Virtualdub and MP_Pipeline pick up the new version. So, everything is consistent now, I just need to settle on which version I should be running.

I'd suggest adding a note to your documentation for MP_Pipeline (if it isn't one there already) about the mechanism it uses to locate avisynth.dll, and the possible pitfalls if multiple versions exist on the system.

Anyway, now I've got that sorted out, I'll start testing MP_Pipeline again. I have to say, so far it's been the least troublesome multi-processing technique I've tried, especially where a mix of temporal and spatial filters are being used and/or multiple sources files need parallel processing (e.g. to compare the results of multiple test runs). :)

(BTW, No I wasn't using "### prefetch". Might try that next, after I've re-run the last batch of tests with my new, 100% less insane AVIsynth installation! :) )

MP_Pipeline depend on Windows to load avisynth.dll for it, it won't do anything special. You have a risk if you don't install avisynth.dll into system folder anyways...

So you have already found out that the performance issue was not due to the script. :) Just a suggestion, I think you can try using it together with ### branch and ### prefetch, that can make the script even more parallel and should have greater speed improvement.

[pbristow]

Hi SAPikachu,

can you explain a bit more about how branch and prefetch function, or how they should be used?

I deduce from the MP_Pipeline_readme.avs script in the package that branch splits the video stream temporally (rather than spatially), like the traditional MT modes do, but in larger chunks (i.e. multiple frames per processing chunk). That (together with pre-fetch) should help with some temporal filters, i.e. the ones that get bogged down in duplicated effort under Set_MT_Mode: The only duplication will be around the boundaries of each chunk, so the bigger the chunk the better, yes?

If you have several "### ###" process dividers, does it make any sense to use "### prefetch" in more than one place, or does one pre-fetch instruction apply to all processes?

Continuing to test and play... :)

[SAPikachu]

Hi SAPikachu,

can you explain a bit more about how branch and prefetch function, or how they should be used?

I deduce from the MP_Pipeline_readme.avs script in the package that branch splits the video stream temporally (rather than spatially), like the traditional MT modes do, but in larger chunks (i.e. multiple frames per processing chunk). That (together with pre-fetch) should help with some temporal filters, i.e. the ones that get bogged down in duplicated effort under Set_MT_Mode: The only duplication will be around the boundaries of each chunk, so the bigger the chunk the better, yes?

If you have several "### ###" process dividers, does it make any sense to use "### prefetch" in more than one place, or does one pre-fetch instruction apply to all processes?

Continuing to test and play... :)

branch basically does what you said, but normally bigger thunk may not be better, since you need to use prefetch to preload all parts parallelly, if thunk size is too big, it becomes impossible to prefetch a full thunk (due to memory constraint) before the active thunk fully consumed by downstream.

prefetch statement only applies to the containing script block, so you usually need to add it to all script blocks. Be very careful if downstream script blocks have temporal filters, if prefetch cache size is not big enough, it may cause frequent seeking and cache thrashing, the speed will be greatly decreased because of that.

Happy experimenting. :)

[pbristow]

Just checking my understanding again:


MP_Pipeline("""
AviSource("P:\Testfile.avi")
### ###
InterFrame (FlowPath="C:\Program Files (x86)\AviSynth\Plugins\Dependencies\", Preset="Fast", NewNum=60000, NewDen=1001)
### prefetch: 20, 4
### ###
TDecimate(mode=0,cycleR=2,cycle=10)
### ###
""")


In this example, prefetch is placed at the end of the code block containing Interframe. Am I correct in assuming this will pre-fetch frames from Interframe, so that they will be ready to supply later to TDecimate? Or (less intuitively), does it mean that Interframe will receive frames that were pre-fetched from Avisource?

(P.S. Before anyone pipes up, I know that I'm using Interframe and TDecimate "the wrong way round". It's actually the best way for the job I'm doing! :) )

Hmm... This is turning into a usage rather than development thread. Do you want to move this discussion to the other forum?

[SAPikachu]

Just checking my understanding again:


MP_Pipeline("""
AviSource("P:\Testfile.avi")
### ###
InterFrame (FlowPath="C:\Program Files (x86)\AviSynth\Plugins\Dependencies\", Preset="Fast", NewNum=60000, NewDen=1001)
### prefetch: 20, 4
### ###
TDecimate(mode=0,cycleR=2,cycle=10)
### ###
""")


In this example, prefetch is placed at the end of the code block containing Interframe. Am I correct in assuming this will pre-fetch frames from Interframe, so that they will be ready to supply later to TDecimate? Or (less intuitively), does it mean that Interframe will receive frames that were pre-fetched from Avisource?

(P.S. Before anyone pipes up, I know that I'm using Interframe and TDecimate "the wrong way round". It's actually the best way for the job I'm doing! :) )

Hmm... This is turning into a usage rather than development thread. Do you want to move this discussion to the other forum?

Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.

I think it's OK to keep the discussion here, since there is few development talk now. :)

[pbristow]

[QUOTE=SAPikachu;1567071]Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.
QUOTE]

I understand that it applies to the containing block; the question is, to which *end* of the block: The input, or the output? :)

Pre-fetch is something that happens at the boundary between things, y'see. So is it pre-fetching *for* the current block, from the previous, or *from* the current block, for the next?

[SAPikachu]

[QUOTE=SAPikachu;1567071]Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.
QUOTE]

I understand that it applies to the containing block; the question is, to which *end* of the block: The input, or the output? :)

Pre-fetch is something that happens at the boundary between things, y'see. So is it pre-fetching *for* the current block, from the previous, or *from* the current block, for the next?

Sorry, I meant prefetch *from* the current block, output from the current block is prefetched and cached by the filter, for next block to use.

[pbristow]

[QUOTE=pbristow;1567133]

Sorry, I meant prefetch *from* the current block, output from the current block is prefetched and cached by the filter, for next block to use.

Thanks. That helps with tuning the memory usage. :)

One thing: Audio doesn't seem to be carried through MP_Pipeline anymore. I'm having to re-dub the audio (using AudioDub(...) ) in the main script after the MP_Pipeline call, as none comes back from MP_Pipeline. Do you get the same, or am I doing something wrong? As far as I can see none of the filters I'm using should kill the audio, and if I move the AudioDub() call inside MP_Pipeline it doesn't work, but outside it does.

[SAPikachu]

Thanks. That helps with tuning the memory usage. :)

One thing: Audio doesn't seem to be carried through MP_Pipeline anymore. I'm having to re-dub the audio (using AudioDub(...) ) in the main script after the MP_Pipeline call, as none comes back from MP_Pipeline. Do you get the same, or am I doing something wrong? As far as I can see none of the filters I'm using should kill the audio, and if I move the AudioDub() call inside MP_Pipeline it doesn't work, but outside it does.

I forgot to note that in the readme, audio is intentionally unsupported, because it is complex to handle and may slow down the whole filter chain. Just AudioDub() after MP_Pipeline call. :P

[pbristow]

I forgot to note that in the readme, audio is intentionally unsupported, because it is complex to handle and may slow down the whole filter chain. Just AudioDub() after MP_Pipeline call. :P

OK, thanks. At least I know I'm not going deaf. :)

Overall impression: With the definite benefit from the prefetch facility (thanks for adding that! :) ), I'm finding it *most* useful in my comparison scripts, where I run the same basic pre-processing on several source files in parallel and then compare the results. Processing each source file via it's own slave process using MP_Pipeline and then combining the results in the main script works smoothly and efficiently every time, utilising up to 99% CPU and delivering speed gains to match. The only gotcha is memory: It's wise to use SetMemoryMax inside the MP_Pipleline call, with a parameter that won't overflow your free memory when all those parallel processes are demanding memory at once. (Formula: Take the average free memory (in Megabytes) you have when you first invoke VirtualDub (or whatever app calls AViSynth); divide that number by the number of parallel processes you're going to run; Shave off a few tens of megabytes for safety and/or uses that "SetMemoryMax" doesn't cover.)

With more linear AviSynth scripts, the picture is not so straightforward. Sometimes using MP_Pipeline gives the best speed-up; sometimes SetMTmode() does. Sometimes tweaking the prefetch and branch parameters makes all the difference; sometimes it makes virtually none. Like all multi-threading and multiprocessing techniques, it really depends on knowing where and how to use it - and that means knowing the characteristics of the filters in your script and thus how they will be affected by the various different ways of dividing up the work they do - to get the best results. And sometimes, no matter what combination of tricks (MT_Pipline, MT(), SetMTMode(), etc...) and parameters I try, I can't get above 70% CPU utilisation, and that just has to be accepted. Some bottleneck in my machine prevents enough data getting to the CPU to be processed any faster, I guess. (Time to start saving up for that new motherboard...?)

Big, big thanks to SAPikachu for a really useful plug-in. :)

[SAPikachu]

OK, thanks. At least I know I'm not going deaf. :)

Overall impression: With the definite benefit from the prefetch facility (thanks for adding that! :) ), I'm finding it *most* useful in my comparison scripts, where I run the same basic pre-processing on several source files in parallel and then compare the results. Processing each source file via it's own slave process using MP_Pipeline and then combining the results in the main script works smoothly and efficiently every time, utilising up to 99% CPU and delivering speed gains to match. The only gotcha is memory: It's wise to use SetMemoryMax inside the MP_Pipleline call, with a parameter that won't overflow your free memory when all those parallel processes are demanding memory at once. (Formula: Take the average free memory (in Megabytes) you have when you first invoke VirtualDub (or whatever app calls AViSynth); divide that number by the number of parallel processes you're going to run; Shave off a few tens of megabytes for safety and/or uses that "SetMemoryMax" doesn't cover.)

With more linear AviSynth scripts, the picture is not so straightforward. Sometimes using MP_Pipeline gives the best speed-up; sometimes SetMTmode() does. Sometimes tweaking the prefetch and branch parameters makes all the difference; sometimes it makes virtually none. Like all multi-threading and multiprocessing techniques, it really depends on knowing where and how to use it - and that means knowing the characteristics of the filters in your script and thus how they will be affected by the various different ways of dividing up the work they do - to get the best results. And sometimes, no matter what combination of tricks (MT_Pipline, MT(), SetMTMode(), etc...) and parameters I try, I can't get above 70% CPU utilisation, and that just has to be accepted. Some bottleneck in my machine prevents enough data getting to the CPU to be processed any faster, I guess. (Time to start saving up for that new motherboard...?)

Big, big thanks to SAPikachu for a really useful plug-in. :)

Thanks for your great review. :)

[SAPikachu]

Released 0.13. Fixed a crashing bug so it is recommended to update to this version.

[real.finder]

I have avs 2.6 mt by SEt and avs64 2.5 (http://code.google.com/p/avisynth64/downloads/detail?name=avisynth64_8-29-10.rar&can=2&q=)

and I use this script

MP_Pipeline("""
### platform: win64
DGDecode_MPEG2Source("x:\xx.d2v").ThreadRequest()
SetMTMode(2)
ColorMatrix(d2v="x:\xx.d2v", threads=0)
Trim(x, xx) ++ Trim(xx, xxx) ++ Trim(xxxx, xxxxxx)
SetMTMode(6)
#~ tfm(output="matches.txt")
#~ tdecimate(mode=4,output="metrics.txt")
tfm(input="matches.txt")
tdecimate(mode=5, hybrid=2, vfrDec=1, input="metrics.txt", tfmIn="matches.txt", mkvOut="mkv-timecodesfile.txt", tcfv1=false)
Crop(x, x, -x, x)
xxxResize(1280, 720)
### ###
### platform: win32
LoadPlugin(AviSynthPluginsDir + "EEDI2 mt/EEDI2_imp.dll")
LoadPlugin(AviSynthPluginsDir + "avs26/mt_masktools-26.dll")
SetMemoryMax(1000)
SetMTMode(2)
filter1
filter2
SoraThread()
filter3
filter4
### lock threads to cores
### ###
""")
AssumeFrameBased

it gave me very good speed

MP_Pipeline is cool plugin, I can use the script in x264 64 and x264 32, and I can use it in avspmod

thank you for it

----------

edit 1: don't use MP_Pipeline in 1st pass (analysis) of tfm and tdecimate vfr (or animeivtc mode 4 with omode=2) Because it will generate empty files

[kolak]

SAPikachu I have a problem with latest mp_pipeline. Avisynth 2.6mt and Vdub 32bit on Win 7 64bit.
When I load script to Vdub and than close it all slave processes stay active and take memory. I also think older version did not need loadPlugin....latest one does (or is it my imagination :) )

[SAPikachu]

SAPikachu I have a problem with latest mp_pipeline. Avisynth 2.6mt and Vdub 32bit on Win 7 64bit.
When I load script to Vdub and than close it all slave processes stay active and take memory. I also think older version did not need loadPlugin....latest one does (or is it my imagination :) )

Do you mean MP_Pipeline needs to be manually loaded? If you put it into plugin autoload folder it should be automatically loaded on start. Or maybe you put it into wrong place?

About the slave process problem, I suspect avs-mt is the culprit. Can you try a regular avs build?

[kolak]

I fixed autoload by re-installing avisynth.

I tried "normal" avisynth and it's the same- processes don't close.
I tried loading script to MP Classic and also the same :(

I think older version was fine.

[SAPikachu]

That's strange, it is fine on my machine. Can you try the following steps to capture a memory dump for me to debug?

1. Use this version of MP_Pipeline (http://nmm.me/vz), and download and extract Procdump package (http://nmm.me/w3)
2. Load your script into vdub/mpc/whatever player, and close it
3. If the slave process is stuck, double-click "dump_immediately.cmd" in the Procdump package, and accept the agreement
4. A dump file (maybe very big) will be generated, please compress and upload it to any sharing site (Mediafire for example) and send it to me

[kolak]

I can do it.
I also found that it happens on my laptop only- on some other PC it works fine.

Check your PM.

[SAPikachu]

I just checked the dump, seems the slave process was stuck inside AppleProResDecoder, can you try disabling it and try again?

[kolak]

I tried reading ProRes file with ffvideo and it was fine.
Than I tried on other machine and yes- with qtinput and ProRes file it's also a problem.
I think it's qtinput+ProRes decoder- any way to solve it? Other mov formats seams to be fine and of course ProRes is the one which I'm interested the most :)

Thanks for looking into it.

[SAPikachu]

I can't really handle it in my code (except for forcibly terminating the slave process itself on exit, but I don't want to do that since some plugin can't properly clean up in that case). Maybe you can try another version of QuickTime, or just use ffms2?

[kolak]

Already tried some other ProRes decoder, will try older QT, but for now ffvideo is the only one solution (I don't like it that much- bit unpredictable also)

[aldix]

Hello,

latest adopter of MP_Pipeline here.

As I was suggested, I copied all the x86 files into avs plugins directory (using SET's 2.6 MT), then made the following script:

MP_Pipeline("""
f3kdb(sample_mode=2,dynamic_grain=false,keep_tv_range=false,dither_algo=3,y=48,cb=24,cr=24,grainY=48,grainC=24)
### ###
BlindDehalo3(rx=2.5, ry=2.5, strength=125)
### lock threads to cores
""")

But for the life of me, I can't get it to work. All it gives me is this error msg:

MP_Pipeline: Unable to create slave process. Message: Script Error: Script Error: Invalid arguments to function 'f3kdb'

But everything works when I remove MP stuff. What am I doing wrong?

Thanks!

[SAPikachu]

Hello,

latest adopter of MP_Pipeline here.

As I was suggested, I copied all the x86 files into avs plugins directory (using SET's 2.6 MT), then made the following script:

MP_Pipeline("""
f3kdb(sample_mode=2,dynamic_grain=false,keep_tv_range=false,dither_algo=3,y=48,cb=24,cr=24,grainY=48,grainC=24)
### ###
BlindDehalo3(rx=2.5, ry=2.5, strength=125)
### lock threads to cores
""")

But for the life of me, I can't get it to work. All it gives me is this error msg:

MP_Pipeline: Unable to create slave process. Message: Script Error: Script Error: Invalid arguments to function 'f3kdb'

But everything works when I remove MP stuff. What am I doing wrong?

Thanks!

Did you put your source filter before MP_Pipeline? MPP can't use video source from the outside, so you need to put all source filter inside it. Like this:



MP_Pipeline("""

FFVideoSource("abcde.mkv")

f3kdb(sample_mode=2,dynamic_grain=false,keep_tv_range=false,dither_algo=3,y=48,cb=24,cr=24,grainY=48,grainC=24)
### ###
BlindDehalo3(rx=2.5, ry=2.5, strength=125)
### lock threads to cores

# By the way, you need to put another block splitter here,
# because all special statement won't be effective for code
# after the last splitter.
### ###

""")

[aldix]

Yup, figured out the source filter bit meanwhile.

But now it's saying that there's no function named BlindDehalo3 even if I'd call it explicitly via import after source filter.

Interestingly, f3kdb now works, though.

Thanks a lot for the reply btw, not only directly from the author him/herself, but so promptly indeed :)

[SAPikachu]

Yup, figured out the source filter bit meanwhile.

But now it's saying that there's no function named BlindDehalo3 even if I'd call it explicitly via import after source filter.

Interestingly, f3kdb now works, though.

Thanks a lot for the reply btw, not only directly from the author him/herself, but so promptly indeed :)

Manually-loaded plugins and imported scripts need to be reloaded/re-imported before they can be used in new process. (Or use inherited script snippet, please see MP_Pipeline_readme.avs for details)

Actually, it will be much easier to put all plugins/imported scripts into your avisynth plugins folder, so that they will be loaded automatically.

[aldix]

Well, they all are in the plugins folder, but I still have to load them manually...


I don't know what to say. Even if I wrap the MP_Pipeline around all the script, with breaks (### ###) between parts,
I still get the same msg about invalid arguments.

For instance, let's take this one:


import("c:\program files (x86)\avisynth 2.5\plugins\gradfun2dbmod.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\ylevels.avs")
import("C:\Program Files (x86)\AviSynth 2.5\plugins\ContraSharpen.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\lsfmod1.9.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\sbr.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\dfttestmc.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\HQDering.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\BlindDeHalo3_mt2.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\Minblur.avs")

source = last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5

preNR = source.degrainmedian(mode=3,limity=8,limituv=10).fft3dfilter(wintype=1,degrid=1,bw=32,bh=32,ow=16,oh=16,bt=3,sigma=2.5,sigma2=2.2,sigma3=1.8,sigma4=0.5,plane=4,ncpu=1).GradFun2DB(1.01)
preNR = source.degrainmedian(mode=3,limity=8,limituv=10).dfttest(sigma=6, ftype=1, tbsize=1, threads=1).GradFun2DB(1.01)
preNR_super = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma)
source_super = source.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, chroma=chroma, levels=1)
Recalculate = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma, levels=1)
vb2 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=2, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr2 = MRecalculate(Recalculate, vb2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vb1 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr1 = MRecalculate(Recalculate, vb1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf1 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr1 = MRecalculate(Recalculate, vf1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf2 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr2 = MRecalculate(Recalculate, vf2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
maskp1 = MMask(vfr1, kind=1, ysc=255).UtoY()
maskp2 = MMask(vfr2, kind=1).UtoY()
maskp3 = MMask(vbr1, kind=1, ysc=255).UtoY()
maskp4 = MMask(vbr2, kind=1).UtoY()
tmask = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)
source2 = mt_merge(source,preNR,tmask,Y=3,U=3,V=3)
KEEP = "0.23"
den = source2.MDegrain2(source_super,vbr1,vfr1,vbr2,vfr2,thSAD=thSAD,thSCD1=256,thSCD2=92)
\.mt_adddiff(mt_makediff(source,preNR,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)

# PROTECTING
threshold = 16
cutoff = 64
maxdiff = 4
rg17 = den.removegrain(17,-1)
iOB = source.mt_lut("x "+string(cutoff)+" >= x 0 ?",U=1,V=1)
mB = mt_makediff(iOB,rg17,U=1,V=1).mt_binarize(128+threshold,upper=false,U=1,V=1).removegrain(5,-1)
lB = mt_lutxy(den,source,"x y - abs "+string(maxdiff)+" <= x x y - 0 < y "+string(maxdiff)+" - x ? ?",U=1,V=1)
smB = mt_merge(den,lB,mB,U=2,V=2)

# EDGECLEANING
mP = mt_edge(smB,"prewitt",0,255,0,0,V=1,U=1)
mS = mP.mt_expand(mode=mt_square(radius=2),U=1,V=1).mt_inflate(U=1,V=1)
mD = mt_lutxy(mS,mP.mt_inflate(U=1,V=1),"x y - "+string(32)+" <= 0 x y - ?",U=1,V=1).mt_inflate(U=1,V=1).removegrain(20,-1)
smE = mt_merge(smB,Eval("smB." + "Removegrain(2,0)"),mD,luma=true,U=3,V=3)


# MASKING
mE = mt_edge(smE,"prewitt",0,255,0,0,V=1,U=1).mt_lut(expr="x 1.8 ^",U=1,V=1).removegrain(4,-1).mt_inflate(U=1,V=1)
mL = mt_logic(tmask.invert(),mE,"min",U=1,V=1).removegrain(20,-1)
mF = mt_logic(tmask,mE,"max",U=1,V=1).removegrain(20,-1)


# SHARPENING
b1c = source.MCompensate(source_super,vb1)
f1c = source.MCompensate(source_super,vf1)
#Sclp = smE.LSFmod(defaults="slow", preblur="ON", strength=100)
Sclp = ContraSharpen(smE,source)
Tmax = source.mt_logic(f1c,"max",U=1,V=1).mt_logic(b1c,"max",U=1,V=1)
Tmin = source.mt_logic(f1c,"min",U=1,V=1).mt_logic(b1c,"min",U=1,V=1)
shrp = Sclp.mt_clamp(Tmax, Tmin, 2, 2, U=1, V=1)
sL = mt_merge(smE,shrp,mL,U=2,V=2)

# ENHANCING
#GFc = sL.f3kdb(sample_mode=2,precision_mode=3)
GFc = sL.GradFun2DBmod(thr=1.4,thrC=1.8,mode=2,str=0.8,strC=0.0,temp=50,adapt=64)
Frs = mt_merge(GFc,sL,mF,luma=true,U=3,V=3).BlindDehalo3(rx=1.25, ry=1.25, strength=90, sharpness=1, ppmode=2)
Frs#.mergechroma(den)
YlevelsS(0,1.0,255,0,255,false)


In the above script, if I'd wrap MP_Pipeline around it (before imports and after ylevels), breaks on diff parts of the script,
it comes back with "invalid arguments" on degrainmedian.

I don't know what else to try. I'm sure it's something minor I'm missing as usual.

[real.finder]

the .avs is not auto load, change it to .avsi for each Script

[aldix]

the .avs is not auto load, change it to .avsi for each Script


Excellent information, real.finder! Greatly obliged. Learn something new every day :)

But this still doesn't change my primary concern - invalid arguments.

[real.finder]

Excellent information, real.finder! Greatly obliged. Learn something new every day :)

But this still doesn't change my primary concern - invalid arguments.

put the full script (with MP_Pipeline) to see what the problem

[aldix]

Apologies for late reply, too busy.

It's basically just the script above, with MP stuff inserted. Anyway, however I place the MP calls, it comes back with some filter having 'invalid arguments'.


MP_Pipeline("""
MPEG2Source("C:\Users\redacted\smileys people ep3\VTS_02_1.d2v", cpu=0)
Crop(2, 0, -2, -0)
source = last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5

preNR = source.degrainmedian(mode=3,limity=8,limituv=10).fft3dfilter(wintype=1,degrid=1,bw=32,bh=32,ow=16,oh=16,bt=3,sigma=2.5,sigma2=2.2,sigma3=1.8,sigma4=0.5,plane=4,ncpu=1).GradFun2DB(1.01)
### ###
preNR = source.degrainmedian(mode=3,limity=8,limituv=10).dfttest(sigma=6, ftype=1, tbsize=1, threads=1).GradFun2DB(1.01)
### ###
preNR_super = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma)
source_super = source.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, chroma=chroma, levels=1)
Recalculate = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma, levels=1)
vb2 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=2, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr2 = MRecalculate(Recalculate, vb2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vb1 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr1 = MRecalculate(Recalculate, vb1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf1 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr1 = MRecalculate(Recalculate, vf1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf2 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr2 = MRecalculate(Recalculate, vf2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
maskp1 = MMask(vfr1, kind=1, ysc=255).UtoY()
maskp2 = MMask(vfr2, kind=1).UtoY()
maskp3 = MMask(vbr1, kind=1, ysc=255).UtoY()
maskp4 = MMask(vbr2, kind=1).UtoY()
tmask = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)
source2 = mt_merge(source,preNR,tmask,Y=3,U=3,V=3)
KEEP = "0.23"
den = source2.MDegrain2(source_super,vbr1,vfr1,vbr2,vfr2,thSAD=thSAD,thSCD1=256,thSCD2=92)
\.mt_adddiff(mt_makediff(source,preNR,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)
### ###
# PROTECTING
threshold = 16
cutoff = 64
maxdiff = 4
rg17 = den.removegrain(17,-1)
iOB = source.mt_lut("x "+string(cutoff)+" >= x 0 ?",U=1,V=1)
mB = mt_makediff(iOB,rg17,U=1,V=1).mt_binarize(128+threshold,upper=false,U=1,V=1).removegrain(5,-1)
lB = mt_lutxy(den,source,"x y - abs "+string(maxdiff)+" <= x x y - 0 < y "+string(maxdiff)+" - x ? ?",U=1,V=1)
smB = mt_merge(den,lB,mB,U=2,V=2)
### ###
# EDGECLEANING
mP = mt_edge(smB,"prewitt",0,255,0,0,V=1,U=1)
mS = mP.mt_expand(mode=mt_square(radius=2),U=1,V=1).mt_inflate(U=1,V=1)
mD = mt_lutxy(mS,mP.mt_inflate(U=1,V=1),"x y - "+string(32)+" <= 0 x y - ?",U=1,V=1).mt_inflate(U=1,V=1).removegrain(20,-1)
smE = mt_merge(smB,Eval("smB." + "Removegrain(2,0)"),mD,luma=true,U=3,V=3)

### ###
# MASKING
mE = mt_edge(smE,"prewitt",0,255,0,0,V=1,U=1).mt_lut(expr="x 1.8 ^",U=1,V=1).removegrain(4,-1).mt_inflate(U=1,V=1)
mL = mt_logic(tmask.invert(),mE,"min",U=1,V=1).removegrain(20,-1)
mF = mt_logic(tmask,mE,"max",U=1,V=1).removegrain(20,-1)

### ###
# SHARPENING
b1c = source.MCompensate(source_super,vb1)
f1c = source.MCompensate(source_super,vf1)
#Sclp = smE.LSFmod(defaults="slow", preblur="ON", strength=100)
Sclp = ContraSharpen(smE,source)
Tmax = source.mt_logic(f1c,"max",U=1,V=1).mt_logic(b1c,"max",U=1,V=1)
Tmin = source.mt_logic(f1c,"min",U=1,V=1).mt_logic(b1c,"min",U=1,V=1)
shrp = Sclp.mt_clamp(Tmax, Tmin, 2, 2, U=1, V=1)
sL = mt_merge(smE,shrp,mL,U=2,V=2)
### ###
# ENHANCING
#GFc = sL.f3kdb(sample_mode=2,precision_mode=3)
GFc = sL.GradFun2DBmod(thr=1.4,thrC=1.8,mode=2,str=0.8,strC=0.0,temp=50,adapt=64)
Frs = mt_merge(GFc,sL,mF,luma=true,U=3,V=3).BlindDehalo3(rx=1.25, ry=1.25, strength=90, sharpness=1, ppmode=2)
Frs#.mergechroma(den)
### ###
YlevelsS(0,1.0,255,0,255,false)
### ###
""")



edit: Sorry, forgot to add source filter at first, fixed.

[real.finder]

what is the source and video source filter?

I don't see one in your script

[SAPikachu]

Variables won't be automatically transferred to next script block. For clip variables you need "### export clip" to use it in next block, for others you need "### inherit". Please see the example script in the package for details, because I am on vacation now, I can't provide another example for you.

[aldix]

MP_Pipeline("""
MPEG2Source("C:\VTS_02_1.d2v", cpu=0)
Crop(2, 0, -2, -0)
### export clip: source,overlap,blksize,hpad,vpad,thSAD,halfblksize,halfoverlap,halfthSAD,chroma,search
source=last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5
....


Now I'm getting 'Unable to create slave process ... Invalid arguments to function 'MPP_PrepareDownStreamClip'.

I really, really appreciate all the help You've given so far SAPIkachu, I just can't seem to wrap my head around it,
scarce script example isn't much use to me, I'm afraid.


Thank You again.

[SAPikachu]

Actually, you should use "export clip" like this:


source = ...
### export clip: source


Edit: I was wrong about what error the script have. See my new post below. Sorry.

[aldix]

Actually, you should use "export clip" like this:


source = ...
### export clip: source



Well, for this I followed the example script/tutorial. There it was said to list 'em separated by commas.

And regardless, now it whines for other variables, like hpad.

I don't know. Perhaps I just shouldn't try to use it or something.

[real.finder]

you can use the mp_pipleline without split by now, until you learn all its settings

like:


MP_Pipeline("""
### platform: win32
SetMemoryMax(1500)
MPEG2Source("C:\Users\redacted\smileys people ep3\VTS_02_1.d2v", cpu=0)
Crop(2, 0, -2, -0)
source = last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5

preNR = source.degrainmedian(mode=3,limity=8,limituv=10).fft3dfilter(wintype=1,degrid=1,bw=32,bh=32,ow=16,oh=16,bt=3,sigma=2.5,sigma2=2.2,sigma3=1.8,sigma4=0.5,plane=4,ncpu=1).GradFun2DB(1.01)
preNR = source.degrainmedian(mode=3,limity=8,limituv=10).dfttest(sigma=6, ftype=1, tbsize=1, threads=1).GradFun2DB(1.01)
preNR_super = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma)
source_super = source.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, chroma=chroma, levels=1)
Recalculate = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma, levels=1)
vb2 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=2, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr2 = MRecalculate(Recalculate, vb2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vb1 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr1 = MRecalculate(Recalculate, vb1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf1 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr1 = MRecalculate(Recalculate, vf1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf2 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr2 = MRecalculate(Recalculate, vf2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
maskp1 = MMask(vfr1, kind=1, ysc=255).UtoY()
maskp2 = MMask(vfr2, kind=1).UtoY()
maskp3 = MMask(vbr1, kind=1, ysc=255).UtoY()
maskp4 = MMask(vbr2, kind=1).UtoY()
tmask = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)
source2 = mt_merge(source,preNR,tmask,Y=3,U=3,V=3)
KEEP = "0.23"
den = source2.MDegrain2(source_super,vbr1,vfr1,vbr2,vfr2,thSAD=thSAD,thSCD1=256,thSCD2=92)
\.mt_adddiff(mt_makediff(source,preNR,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)

# PROTECTING
threshold = 16
cutoff = 64
maxdiff = 4
rg17 = den.removegrain(17,-1)
iOB = source.mt_lut("x "+string(cutoff)+" >= x 0 ?",U=1,V=1)
mB = mt_makediff(iOB,rg17,U=1,V=1).mt_binarize(128+threshold,upper=false,U=1,V=1).removegrain(5,-1)
lB = mt_lutxy(den,source,"x y - abs "+string(maxdiff)+" <= x x y - 0 < y "+string(maxdiff)+" - x ? ?",U=1,V=1)
smB = mt_merge(den,lB,mB,U=2,V=2)

# EDGECLEANING
mP = mt_edge(smB,"prewitt",0,255,0,0,V=1,U=1)
mS = mP.mt_expand(mode=mt_square(radius=2),U=1,V=1).mt_inflate(U=1,V=1)
mD = mt_lutxy(mS,mP.mt_inflate(U=1,V=1),"x y - "+string(32)+" <= 0 x y - ?",U=1,V=1).mt_inflate(U=1,V=1).removegrain(20,-1)
smE = mt_merge(smB,Eval("smB." + "Removegrain(2,0)"),mD,luma=true,U=3,V=3)


# MASKING
mE = mt_edge(smE,"prewitt",0,255,0,0,V=1,U=1).mt_lut(expr="x 1.8 ^",U=1,V=1).removegrain(4,-1).mt_inflate(U=1,V=1)
mL = mt_logic(tmask.invert(),mE,"min",U=1,V=1).removegrain(20,-1)
mF = mt_logic(tmask,mE,"max",U=1,V=1).removegrain(20,-1)


# SHARPENING
b1c = source.MCompensate(source_super,vb1)
f1c = source.MCompensate(source_super,vf1)
#Sclp = smE.LSFmod(defaults="slow", preblur="ON", strength=100)
Sclp = ContraSharpen(smE,source)
Tmax = source.mt_logic(f1c,"max",U=1,V=1).mt_logic(b1c,"max",U=1,V=1)
Tmin = source.mt_logic(f1c,"min",U=1,V=1).mt_logic(b1c,"min",U=1,V=1)
shrp = Sclp.mt_clamp(Tmax, Tmin, 2, 2, U=1, V=1)
sL = mt_merge(smE,shrp,mL,U=2,V=2)

# ENHANCING
#GFc = sL.f3kdb(sample_mode=2,precision_mode=3)
GFc = sL.GradFun2DBmod(thr=1.4,thrC=1.8,mode=2,str=0.8,strC=0.0,temp=50,adapt=64)
Frs = mt_merge(GFc,sL,mF,luma=true,U=3,V=3).BlindDehalo3(rx=1.25, ry=1.25, strength=90, sharpness=1, ppmode=2)
Frs#.mergechroma(den)
YlevelsS(0,1.0,255,0,255,false)
### lock threads to cores
### prefetch: 16, 12
### ###
""")
AssumeFrameBased


as I say here http://forum.doom9.org/showthread.php?p=1629295#post1629295

[aldix]

Well, thanks for this. Now I got it to work and going through the frames in AvspMod is very fast. However, encoding in x264 (with Simple Launcher GUI) first
hangs with 'potential deadlock', then nonetheless starts but very slowly and without using 100% cpu. Just sits at 40-50%. Memory meanwhile is 70%+.
What's up with that?

I did some investigating on my own and discovered that by removing 'lock threads to cores' statement I can get some speed back - 0,5fps and about 10% cpu load,
but that's it.

At least I'm glad that there was some little thing I missed and everything works for me. It's just not what I expected...

[SAPikachu]

Well, for this I followed the example script/tutorial. There it was said to list 'em separated by commas.

And regardless, now it whines for other variables, like hpad.

I don't know. Perhaps I just shouldn't try to use it or something.

Sorry I misread your script. Your syntax was right, but ### export clip only works for clip variables. For other variables, you need to use ### inherit start / ### inherit end. (Or just copy all the variables to all script blocks)

"lock threads to cores" is an advanced function, don't use it unless you really know how it works. Actually nearly no one told me that it is useful.

[aldix]

Well, I don't know what to tell you guys. I've tried both with and without ### inherit start/end wrapping the variables, and with lowering/rising the setMemoryMax.
I'm only getting ~1 fps on a crf encode (once it finally starts to encode after the deadlock msg) on a script that used to give 1.6 fps with just a single
SetMTMode(2)/SetMemoryMax(640) pair inserted in the middle. Now that doesn't work (x264 deadlocks eventually and cpu load doesn't achieve 100%)
and this pipeline doesn't appear to increase any speed at all.

If such set-up is just to make big scripts run with less load on cpu (haven't yet seen cpu at 100% with this) while increasing the memory load, it's fine. Currently I just need to script to run, period, don't really
care how long it takes. But I was getting giddy under the (false?) impression that this set-up actually speeds things along? Was I wrong or still doing something incorrectly?

I really, really appreciate all the replies and help. It's just frustrating.

[real.finder]

for me using "lock threads to cores" make the process faster

and mp_pipeline very useful if you know how to use it correctly, Each script has a special case to deal with, and must be conduct experiments, and measurement speed by avs meter (http://forum.doom9.org/showthread.php?t=165528)

some time I use analysis pass in avspmod to measurement speed

Good luck

[aldix]

and mp_pipeline very useful if you know how to use it correctly


Yes, thank you. That's a really useful statement given the current matters.

edit: Evidently cpu's non-100% load is the problem here. I'm just trying a different script which went at 11fps on
1st pass and now it runs at 3.5fps with 40% cpu load.

[SAPikachu]

Well, I don't know what to tell you guys. I've tried both with and without ### inherit start/end wrapping the variables, and with lowering/rising the setMemoryMax.
I'm only getting ~1 fps on a crf encode (once it finally starts to encode after the deadlock msg) on a script that used to give 1.6 fps with just a single
SetMTMode(2)/SetMemoryMax(640) pair inserted in the middle. Now that doesn't work (x264 deadlocks eventually and cpu load doesn't achieve 100%)
and this pipeline doesn't appear to increase any speed at all.

If such set-up is just to make big scripts run with less load on cpu (haven't yet seen cpu at 100% with this) while increasing the memory load, it's fine. Currently I just need to script to run, period, don't really
care how long it takes. But I was getting giddy under the (false?) impression that this set-up actually speeds things along? Was I wrong or still doing something incorrectly?

I really, really appreciate all the replies and help. It's just frustrating.

Well, it was a good thing that your script could run without error, this meant you got syntax correct and you can start to tweak the script.

I noticed you put many splitters (### ###) into your script, if you don't use prefetching it would be better to reduce count of splitters to roughly number of cpu cores you have. If you add too many splitters it may slow down your script. Then you can try moving splitters around to see if it can improve your speed.

[pbristow]

I don't seem to be able to utilize my CPU fully


[SIGH] This is false goal. Forget it.

The purpose of multi-tasking is to *speed up* the task you're working on, not to make your CPU as busy as possible. The only way you will ever get 100% utilisation on a multicore CPU is with extremely unusual (and probably artificial) workloads. Instead, the question to focus on is "how much time am I saving?"

Here's a rough guide as to when to say "Woot!" rather than "Hmph! Well, that's better than nothing I suppose..." when using multi-tasking, versus the number of cores/threads you've used:

Using 2: Execution time drops 30%. (fps increases 40%)
Using 3: Execution time drops 42%. (fps increases 73%)
Using 4: Execution time drops 50%. (fps increases 100%)
Using 6: Execution time drops 59%. (fps increases 144%)
Using 8: Execution time drops 66%. (fps increases 183%)


...In other words, don't expect a performance improvement much beyond the square root of the number of cores/processors/threads employed.

:)

[pbristow]

...and, the post I'm replying to has already been deleted. [FACEPALM]

Perhaps I should make a couple of new rules for myself:
(1) Don't reply to anything until it's at least a day old;
(2) don't reply to anything without first updating the page six times to make sure no one else has already covered it! :stupid:.

[aldix]

The only use for me that I can see is that big scripts wrapped into MP don't deadlock x264 anymore, even if not on full load at 2nd pass. Though that also was the case *before* all my recent MT troubles began (after updating to latest v).

So I don't really know what the take-away here is. Everything is just as slow for me as it previously was :)

[zerowalker]

pbristow, i solved it, that's why i deleted it:)

[zerowalker]

I am trying to get MP Pipeline to work with MCTD and also work with audio.

So that it can work with Megui, making the audio bypass the MT Pipeline and be directly encoded.
And the Video to be worked and processed in MCTD with MT Pipeline.

But sadly, it doesn´t seem to work:(

[SAPikachu]

I am trying to get MP Pipeline to work with MCTD and also work with audio.

So that it can work with Megui, making the audio bypass the MT Pipeline and be directly encoded.
And the Video to be worked and processed in MCTD with MT Pipeline.

But sadly, it doesn´t seem to work:(

Sorry, I forgot to note that MP_Pipeline doesn't support audio, you need to encode audio without using MP_Pipeline. OP is just updated to include this limitation.

Maybe you can demux the audio and directly encode it, or simply comment out MP_Pipeline for your audio script. If you want to use the same script for both audio and video encoding, you can try AudioDub after MP_Pipeline.

[zerowalker]

Audiodub seems to work, for example:

u=Avisource("")
MP_Pipeline("""
SetMemoryMax(3072)
Avisource("")
MCTD(settings="low")
### prefetch: 16, 0

### ###
""")
AudioDub(u)


So Thanks:)


Though, i wonder is there a way to prevent MP to load when audio decoding?
Cause currently, it seems to load MP which takes a very long time with more branches, and then they just close, and after all that, it starts processing.

[SAPikachu]

Audiodub seems to work, for example:

u=Avisource("")
MP_Pipeline("""
SetMemoryMax(3072)
Avisource("")
MCTD(settings="low")
### prefetch: 16, 0

### ###
""")
AudioDub(u)


So Thanks:)


Though, i wonder is there a way to prevent MP to load when audio decoding?
Cause currently, it seems to load MP which takes a very long time with more branches, and then they just close, and after all that, it starts processing.

I think it is not possible with single script, you can only do that with separated script file.

I still think that you should demux your source audio and directly encode or mux it to the output, there is no reason to pass it through AviSynth unless you want to do some post-processing to it.

[aldix]

From what does it depend whether the script used inside MP_ gets 100% cpu load or not? Some long scripts of mine get 35-40% cpu load whereas short ones get 100%, I see no rhyme or reason to this. Can you explain, Sapi?

[SAPikachu]

From what does it depend whether the script used inside MP_ gets 100% cpu load or not? Some long scripts of mine get 35-40% cpu load whereas short ones get 100%, I see no rhyme or reason to this. Can you explain, Sapi?

Please remember that CPU usage != speed as what @pbristow said, you need to compare your encoding speed, not CPU utilization.

CPU utilization mainly depends on whether works are fairly distributed across all script blocks, and how good is your prefetch setting. In other words, you need to make all the script blocks consume roughly same amount of CPU.

[aldix]

Well, I like smaller cpu load very much, it enables me to do other tasks. That's why I asked. I don't really understand prefetch, so I'm pretty much setting it at random, sometimes with what's suggested in the example script, sometimes way higher. I don't know...

[zerowalker]

Okay, i think the audio goes pretty well now. Not sure why i went so slow before, must have done something wrong.

But i have another question.


If i want to use MCTD, and run it in parallel in 4 threads on the same clip, how can i do that?
I am thinking, that i should have 4 separate script which are identical, Except for the duration, meaning i have Trimmed the clip differently.
And than later i can just suit them up.

Though i have never done something like that so i am not absolutely sure of it.

Or is it possible to have 1 script, with 4 branches, that each process a different "trim()" and MCTD?

[SAPikachu]

Well, I like smaller cpu load very much, it enables me to do other tasks. That's why I asked. I don't really understand prefetch, so I'm pretty much setting it at random, sometimes with what's suggested in the example script, sometimes way higher. I don't know...

Well you can set priority of your encoding-related process to low, then your normal tasks won't be (much) affected.

### prefetch controls how much frames ahead should be preloaded in background, and how many frames behind should be cached.

Here is a simplified illustration of effect of prefetching:

Without prefetching:

T+0:
+------------------+
| Upstream block | -> Producing frame #0
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #0 from upstream
+------------------+

T+1:
+------------------+
| Upstream block | -> Wait for downstream block to complete processing frame #0
+------------------+

+------------------+
| Downstream block | -> Processing frame #0
+------------------+

T+2:
+------------------+
| Upstream block | -> Producing frame #1
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #1 from upstream
+------------------+

T+3:
+------------------+
| Upstream block | -> Wait for downstream block to complete processing frame #1
+------------------+

+------------------+
| Downstream block | -> Processing frame #1
+------------------+

---------------------------------------------------------------------------------

With prefetching:

T+0:
+------------------+
| Upstream block | -> Producing frame #0
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #0 from upstream
+------------------+

T+1:
+------------------+
| Upstream block | -> Producing frame #1
+------------------+

+------------------+
| Downstream block | -> Processing frame #0
+------------------+

T+2:
+------------------+
| Upstream block | -> Producing frame #2
+------------------+

+------------------+
| Downstream block | -> Processing frame #1
+------------------+

T+3:
+------------------+
| Upstream block | -> Producing frame #3
+------------------+

+------------------+
| Downstream block | -> Processing frame #2
+------------------+



Of course, if upstream block goes too far away from downstream block, it still needs waiting. This really need to tweaked individually for different scripts.

[SAPikachu]

Okay, i think the audio goes pretty well now. Not sure why i went so slow before, must have done something wrong.

But i have another question.


If i want to use MCTD, and run it in parallel in 4 threads on the same clip, how can i do that?
I am thinking, that i should have 4 separate script which are identical, Except for the duration, meaning i have Trimmed the clip differently.
And than later i can just suit them up.

Though i have never done something like that so i am not absolutely sure of it.

Or is it possible to have 1 script, with 4 branches, that each process a different "trim()" and MCTD?

If you have much memory, you may try branch with big block size and large prefetch setting (at least 2x bigger than branch block size). Not sure whether this works since I haven't tried that.

Splitting the script into parts and joining them after encoding may have better performance, but it may also affect rate-controlling of x264, making bitrate distribution worse.

[zerowalker]

Don´t really get how i am supposed to set it up?
And i don´t have an enormous amount of memory. Got 8gb.

[SAPikachu]

Don´t really get how i am supposed to set it up?
And i don´t have an enormous amount of memory. Got 8gb.

Something like this:

MP_Pipeline("""

XXXSource("...")
### prefetch: 512, 256

### ###

MCTD()
### branch: 4, 32
### prefetch: 64, 0

### ###

### prefetch: 64, 0

### ###

""")


Note again, I haven't tested these parameters in any ways, not sure how they perform.

Actually, if you have only 8GB of memory, I would suggest trying this:


MP_Pipeline("""

XXXSource("...")
### prefetch: 64, 32

### ###
SetMemoryMax(3072) # Set this to smaller value if your memory runs out or the slave process crashes
MCTD()
# Don't put anything else in this block
### prefetch: 32, 0

### ###

""")

[zerowalker]

Okay, will be trying the second one.
And it seems to run at 9fps with my settings, which i guess is pretty good.

Though i will have to Trim it still, as the Clip is 7 hours, the chance of it crashing is extremely high, it has happened so many times, that i have given up doing it in one run.

[aldix]

Someone above wrote that 4GB RAM is good enough for this and now 8GB is 'only'? That being said, I'd really like to know how am I supposed to calculate prefetch amounts? Based on what? I've loathed to go over 40 for first # and now you're citing amounts like 512.

As a point of reference, here's a script for 1080p HD source encoding. Currently it's running @60-70% CPU/RAM w/ ~1.2 fps (2/4 i5 650 cpu), but if I'd remove gradfun2dbmod, it gains around 0.5 fps and goes 100% cpu. Why is beyond me.

MP_Pipeline("""
### platform: win32
SetMemoryMax(750)
AVCSource("W:\00250.dga")
Crop(0, 60, -0, -60)
source = last
super = source.MSuper(pel=1, sharp=2, rfilter=2)
b1v = MAnalyse(super,isb=true, delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
f1v = MAnalyse(super,isb=false,delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
cf1 = MCompensate(super, f1v, thSCD1=200)
cb1 = MCompensate(super, b1v, thSCD1=200)
interleave(cf1, source.MDegrain1(super,b1v,f1v,thSAD=130), cb1)
Temporalsoften(1,3,3,6,2)
selectevery(3,1)
S4Mod(strength=4.25,hthr=200,hbias=-100)
Range8to16()
DitherPost(mode=7, ampo=1, ampn=0)
gradfun2dbmod(str=0.3, strC=0.0, adapt=64, temp=80)
YlevelsS(5,1.0,250,0,255,false)
### prefetch: 7, 0
### ###
""")



I also tried to divide this by using ### ### and a few SetMemoryMax calls. That only resulted in many MP processes running (as seen in task manager) and almost full memory use, but not speed increase.

What's the most perfect for me is if I could piggy-back some of the encoding load to RAM from CPU (meaning that I can live with about 80% RAM load, but I'd rather see around 50-60% CPU load when encoding.

I'm sorry for being tense but it seems like a perfect tool for me which I just don't know how to use properly...

[SAPikachu]

Okay, will be trying the second one.
And it seems to run at 9fps with my settings, which i guess is pretty good.

Though i will have to Trim it still, as the Clip is 7 hours, the chance of it crashing is extremely high, it has happened so many times, that i have given up doing it in one run.

A better way (quality-wise) is export the script to lossless video (for example, use avs2avi with HuffYUV encoder in ffdshow). If it crashes, just use trim to start from the crashed point and output to a new file. After the whole clip is exported, use AviSource to load and join all parts, then feed the resulting clip to x264. It needs plenty of disk space though (compression rate is only about 1:2 IIRC). If you don't have much space, trim and encoding in parts may be the only way.

By the way, just curious, why MCTD crashes frequently on your computer? As far as I know it is very stable and seldom crashes on my rig. Are you using AviSynth-mt?

[SAPikachu]

Someone above wrote that 4GB RAM is good enough for this and now 8GB is 'only'? That being said, I'd really like to know how am I supposed to calculate prefetch amounts? Based on what? I've loathed to go over 40 for first # and now you're citing amounts like 512.

As a point of reference, here's a script for 1080p HD source encoding. Currently it's running @60-70% CPU/RAM w/ ~1.2 fps (2/4 i5 650 cpu), but if I'd remove gradfun2dbmod, it gains around 0.5 fps and goes 100% cpu. Why is beyond me.



I also tried to divide this by using ### ### and a few SetMemoryMax calls. That only resulted in many MP processes running (as seen in task manager) and almost full memory use, but not speed increase.

What's the most perfect for me is if I could piggy-back some of the encoding load to RAM from CPU (meaning that I can live with about 80% RAM load, but I'd rather see around 50-60% CPU load when encoding.

I'm sorry for being tense but it seems like a perfect tool for me which I just don't know how to use properly...

Well, 512 is indeed not necessary for regular uses, it was there to prevent branched MCTD in downstream block from thrashing the prefetch cache. For 4-branch MCTD, 8GB RAM is not really enough, because to get best performance each MCTD instance need to have >3GB memory.

prefetch setting is largely based on experience... As a starting point, I will use 16, 0 or 8, 0 for the last script block, and doubled value for each upstream block. It look like this:


MP_Pipeline("""
...
# prefetch: 64, 0
### ###
...
# prefetch: 32, 0
### ###
...
# prefetch: 16, 0
### ###

""")

(Note this only applies to spatial filters, for temporal filters, backwards cache must also be tweaked)

Then, I will start to mess with the values to see which parameter set is the fastest.

For your script, I will use this configuration as beginning:


MP_Pipeline("""
### platform: win32
AVCSource("W:\00250.dga")
Crop(0, 60, -0, -60)

### prefetch: 64, 32
### ###
SetMemoryMax(750)
source = last
super = source.MSuper(pel=1, sharp=2, rfilter=2)
b1v = MAnalyse(super,isb=true, delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
f1v = MAnalyse(super,isb=false,delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
cf1 = MCompensate(super, f1v, thSCD1=200)
cb1 = MCompensate(super, b1v, thSCD1=200)
source.MDegrain1(super,b1v,f1v,thSAD=130)
### prefetch: 32, 16
### export clip: cf1, cb1
### ###

interleave(cf1, last, cb1)
Temporalsoften(1,3,3,6,2)
selectevery(3,1)
S4Mod(strength=4.25,hthr=200,hbias=-100)
Range8to16()
DitherPost(mode=7, ampo=1, ampn=0)
gradfun2dbmod(str=0.3, strC=0.0, adapt=64, temp=80)
YlevelsS(5,1.0,250,0,255,false)
### prefetch: 16, 0
### ###
""")


But you can't get decent speed if you want to limit its CPU usage, in fact the goal of optimizing script with MPP is make CPU busy processing the clip at all time, so that it won't waste time on idling (note that this doesn't mean 100% CPU utilization is always the best, the final indicator is still encoding fps).

[zerowalker]

A better way (quality-wise) is export the script to lossless video (for example, use avs2avi with HuffYUV encoder in ffdshow). If it crashes, just use trim to start from the crashed point and output to a new file. After the whole clip is exported, use AviSource to load and join all parts, then feed the resulting clip to x264. It needs plenty of disk space though (compression rate is only about 1:2 IIRC). If you don't have much space, trim and encoding in parts may be the only way.

By the way, just curious, why MCTD crashes frequently on your computer? As far as I know it is very stable and seldom crashes on my rig. Are you using AviSynth-mt?

That is whay i am doing, though using Lagarith, same thing though:)

Why it crashes, well i don´t know. It doesn´t really crash normally. Except, if i enable the GPU part of MCTD, it will crash if i start a game. Not always, but i think it crashed if it changed resolution or something, and that´s pretty irritating, as GPU adds about 10-20% on speed.

But then again, it´s probably not common for someone to play games while using MCTD.

Then why it crashed without GPU. I am not entirely sure i have used it without GPU, But i think at least it has crashed normally, meaning avs2avi has crashed like a normal application crash. But it has only happened 1 time and that was after, 15+ hours i think?
And that video file went mad, it had no search point, and took years to resave etc.

However, that may have been the GPU fault also, not entirely sure.

And i use the Latest Avisynth, NOT the MT versions, so i can´t do MT, except with MP Pipeline.

[aldix]

prefetch setting is largely based on experience... As a starting point, I will use 16, 0 or 8, 0 for the last script block, and doubled value for each upstream block. It look like this:


MP_Pipeline("""
...
# prefetch: 64, 0
### ###
...
# prefetch: 32, 0
### ###
...
# prefetch: 16, 0
### ###

""")

(Note this only applies to spatial filters, for temporal filters, backwards cache must also be tweaked)

Then, I will start to mess with the values to see which parameter set is the fastest.

For your script, I will use this configuration as beginning:


MP_Pipeline("""
### platform: win32
AVCSource("W:\00250.dga")
Crop(0, 60, -0, -60)

### prefetch: 64, 32
### ###
SetMemoryMax(750)
source = last
super = source.MSuper(pel=1, sharp=2, rfilter=2)
b1v = MAnalyse(super,isb=true, delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
f1v = MAnalyse(super,isb=false,delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
cf1 = MCompensate(super, f1v, thSCD1=200)
cb1 = MCompensate(super, b1v, thSCD1=200)
source.MDegrain1(super,b1v,f1v,thSAD=130)
### prefetch: 32, 16
### export clip: cf1, cb1
### ###

interleave(cf1, last, cb1)
Temporalsoften(1,3,3,6,2)
selectevery(3,1)
S4Mod(strength=4.25,hthr=200,hbias=-100)
Range8to16()
DitherPost(mode=7, ampo=1, ampn=0)
gradfun2dbmod(str=0.3, strC=0.0, adapt=64, temp=80)
YlevelsS(5,1.0,250,0,255,false)
### prefetch: 16, 0
### ###
""")


But you can't get decent speed if you want to limit its CPU usage, in fact the goal of optimizing script with MPP is make CPU busy processing the clip at all time, so that it won't waste time on idling (note that this doesn't mean 100% CPU utilization is always the best, the final indicator is still encoding fps).

Thanks a lot for these clarifications. I now have a somewhat better understanding how it all works. I suppose I'll experiment different setups with my scripts and see what works the best.
So 3 separate prefetch settings is good, right?

Yeah, I understand that in most cases 100% cpu load might be for the best. It's just that I'd love to encode while doing other stuff and for now, I'm just limited to encode while I sleep (in that sense, I'd love to see MPP cut down on HD encoding time - that one always goes longer than my sleep circle, heh).

[SAPikachu]

That is whay i am doing, though using Lagarith, same thing though:)

Why it crashes, well i don´t know. It doesn´t really crash normally. Except, if i enable the GPU part of MCTD, it will crash if i start a game. Not always, but i think it crashed if it changed resolution or something, and that´s pretty irritating, as GPU adds about 10-20% on speed.

But then again, it´s probably not common for someone to play games while using MCTD.

Then why it crashed without GPU. I am not entirely sure i have used it without GPU, But i think at least it has crashed normally, meaning avs2avi has crashed like a normal application crash. But it has only happened 1 time and that was after, 15+ hours i think?
And that video file went mad, it had no search point, and took years to resave etc.

However, that may have been the GPU fault also, not entirely sure.

And i use the Latest Avisynth, NOT the MT versions, so i can´t do MT, except with MP Pipeline.

Yeah using GPU is indeed instable that may be the reason. I never use GPU in MCTD that I totally forgot about that..

[SAPikachu]

Thanks a lot for these clarifications. I now have a somewhat better understanding how it all works. I suppose I'll experiment different setups with my scripts and see what works the best.
So 3 separate prefetch settings is good, right?

Yeah, I understand that in most cases 100% cpu load might be for the best. It's just that I'd love to encode while doing other stuff and for now, I'm just limited to encode while I sleep (in that sense, I'd love to see MPP cut down on HD encoding time - that one always goes longer than my sleep circle, heh).

Don't quite understand your question, prefetch setting is supposed to be different for different script block. Do you mean how many script block should be used? I think 3 should be good for your script, but I may be wrong and you should tweak it yourself.

[Stereodude]

Something like this:

MP_Pipeline("""

XXXSource("...")
### prefetch: 512, 256

### ###

MCTD()
### branch: 4, 32
### prefetch: 64, 0

### ###

### prefetch: 64, 0

### ###

""")


Note again, I haven't tested these parameters in any ways, not sure how they perform.Can you elaborate on how you chose those parameters? If you're encoding the file in 32 frame chunks what's the purpose of prefetching more than 32 frames for those chunks?

Also, if you have 4 32 frame chunks being encoded at once it seems that you wouldn't need more then 128 frames of video cached at any given time, so why the 512/256 prefetch of the source?

[SAPikachu]

Can you elaborate on how you chose those parameters? If you're encoding the file in 32 frame chunks what's the purpose of prefetching more than 32 frames for those chunks?

Also, if you have 4 32 frame chunks being encoded at once it seems that you wouldn't need more then 128 frames of video cached at any given time, so why the 512/256 prefetch of the source?

32 frames per chunk is just an arbitrary value, as what I said before, I just made this up and haven't tested it.

Prefetching 64 frames per chunk is intentional though. If you only prefetch 32 frames, the block won't start preparing next chunk before current chunk is consumed, and if your downstream block is fast enough, the pipeline may stall without consuming all the CPU time.

512/256 in source is just a safe setting, since MCTD is a temporal filter, we'd better use some more memory to ensure it won't fetch behind the cache, as that will seriously impact performance or even crash the pipeline in some cases.

[Stereodude]

32 frames per chunk is just an arbitrary value, as what I said before, I just made this up and haven't tested it.

Prefetching 64 frames per chunk is intentional though. If you only prefetch 32 frames, the block won't start preparing next chunk before current chunk is consumed, and if your downstream block is fast enough, the pipeline may stall without consuming all the CPU time.

512/256 in source is just a safe setting, since MCTD is a temporal filter, we'd better use some more memory to ensure it won't fetch behind the cache, as that will seriously impact performance or even crash the pipeline in some cases.
Thanks for the answers. I tried this, and some variations. On my system, Core i7-4770k @ 4.2gHz Win 7 x64 SP1 w/ 16GB, processing 1920x1080 with MCTemporalDenoise(settings="medium") I was able to get ~5.33FPS with 6 thunks. With 5 threads using MT I'm capped at ~3.75FPS. (Trying to increase the threads any further using MT results in a loss of performance.)

So, it's faster than using MT, but it's also very unstable. For some reason the first thread / pipeline with the source and prefetch will stall at a full "core" worth of CPU usage. The downstream threads / pipelines end sit at 0% CPU and no frames come out. I tried a variety of things like changing the prefetch parameters and thunk size, number of thunks, but nothing eliminated it. The point where it hangs moves around (without changing any settings just running the batch file again). On one run it might happen 200 frames in, another 900+ frames, and another several thousand frames in, etc. It doesn't make a whole lot of sense since the source is DGdecNV which uses the graphics card (ie: no CPU usage).

Edit: Frankly speaking MCTD seems like it's ripe for "improvements". Its output is impressive, but seems like a big fat inefficient kludge. Maybe it's a huge undertaking, but it seems like combining all the source for the various plugins the script uses into a single dll would be a good start. From there (admittedly not having looked at any of the source code) it seems like multi-threading it to at least a basic level shouldn't be that hard. The basic analysis of multiple input frames can be done simultaneously across multiple threads, once the processing decisions have been made, you can process multiple output frames simultaneously across multiple threads. Multi-threading the intermediate processing decisions that are purely sequential is obviously harder.

[SAPikachu]

Thanks for the answers. I tried this, and some variations. On my system, Core i7-4770k @ 4.2gHz Win 7 x64 SP1 w/ 16GB, processing 1920x1080 with MCTemporalDenoise(settings="medium") I was able to get ~5.33FPS with 6 thunks. With 5 threads using MT I'm capped at ~3.75FPS. (Trying to increase the threads any further using MT results in a loss of performance.)

So, it's faster than using MT, but it's also very unstable. For some reason the first thread / pipeline with the source and prefetch will stall at a full "core" worth of CPU usage. The downstream threads / pipelines end sit at 0% CPU and no frames come out. I tried a variety of things like changing the prefetch parameters and thunk size, number of thunks, but nothing eliminated it. The point where it hangs moves around (without changing any settings just running the batch file again). On one run it might happen 200 frames in, another 900+ frames, and another several thousand frames in, etc. It doesn't make a whole lot of sense since the source is DGdecNV which uses the graphics card (ie: no CPU usage).


Can you post your full script here? Not sure whether this is because a problem in your script, or a bug in MPP.


Edit: Frankly speaking MCTD seems like it's ripe for "improvements". Its output is impressive, but seems like a big fat inefficient kludge. Maybe it's a huge undertaking, but it seems like combining all the source for the various plugins the script uses into a single dll would be a good start. From there (admittedly not having looked at any of the source code) it seems like multi-threading it to at least a basic level shouldn't be that hard. The basic analysis of multiple input frames can be done simultaneously across multiple threads, once the processing decisions have been made, you can process multiple output frames simultaneously across multiple threads. Multi-threading the intermediate processing decisions that are purely sequential is obviously harder.

Well, the main problem is that many plugins are not thread-safe, so even you managed to merge the plugins, it is still very hard to parallelize it IMO.

You can also try using MVTools in dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559), it is natively MT-enabled and may increase speed a bit.

[Stereodude]

Can you post your full script here? Not sure whether this is because a problem in your script, or a bug in MPP.Sure. Here's the final variation I tried.

LoadPlugin("MP_Pipeline.dll")
LoadPlugin("flash3kyuu_deband.dll")

MP_Pipeline("""

### platform: win32
# this part of script will be run in a 32 bit process

LoadPlugin("DGDecodeNVx86.dll")
DGSource("Video_2.dgi").crop(0,18,-0,-18)

### prefetch: 512, 384

### ###

### platform: win32
# ### lock threads to cores

Import("MCTemporalDenoise.v1.4.20.avsi")

SetMemoryMax(1280)
MCTemporalDenoise(settings="medium")


### branch: 6, 32
### prefetch: 64, 0

### ###

# ### prefetch: 48, 0

# ### ###

""")
flash3kyuu_deband(range=15, grainY=32, grainC=32, sample_mode=2, dither_algo=3, mt=false).addborders(0,18,0,18)

[SAPikachu]

Sure. Here's the final variation I tried.

LoadPlugin("MP_Pipeline.dll")
LoadPlugin("flash3kyuu_deband.dll")

MP_Pipeline("""

### platform: win32
# this part of script will be run in a 32 bit process

LoadPlugin("DGDecodeNVx86.dll")
DGSource("Video_2.dgi").crop(0,18,-0,-18)

### prefetch: 512, 384

### ###

### platform: win32
# ### lock threads to cores

Import("MCTemporalDenoise.v1.4.20.avsi")

SetMemoryMax(1280)
MCTemporalDenoise(settings="medium")


### branch: 6, 32
### prefetch: 64, 0

### ###

# ### prefetch: 48, 0

# ### ###

""")
flash3kyuu_deband(range=15, grainY=32, grainC=32, sample_mode=2, dither_algo=3, mt=false).addborders(0,18,0,18)

Can you try 768,512 in the DGSource block? If that still has problem, it is likely a bug in MPP and I will look into it when I get some free time...

[Stereodude]

You can also try using MVTools in dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559), it is natively MT-enabled and may increase speed a bit.This increased CPU usage from 12.5% (1 logical core on my i7) to 20-25%. FPS went from ~1.5 to ~2.0.

Can you try 768,512 in the DGSource block? If that still has problem, it is likely a bug in MPP and I will look into it when I get some free time...I'll give it a shot later today when the machine is done with the current jobs. Right now I'm experimenting with intermediate lossless (Lagarith) AVI files. I split the source into 4 slightly overlapping chunks and output them to 4 AVI files. With the older single threaded MVTools I can process all 4 chunks simultaneously for a combined ~4.5FPS. It also avoids any temporal implications that AVIsynth MT or thunking the source can cause and I don't have the process the source twice for a 2-pass encode either.

[aldix]

Sorry much for OT, but I just have to ask.

Is DGDecodeNV worth it? I mean, does it make script/encoding faster when used instead of AVCSource/FFVideosource etc? I have GTX 460 GPU which has VP4.

Thanks!

[Stereodude]

With the older single threaded MVTools I can process all 4 chunks simultaneously for a combined ~4.5FPS.FWIW, the MT version of MVTools ups the combined throughput to lossless Lagarith AVIs for 4 simultaneous chunks to ~5.3FPS.
Is DGDecodeNV worth it? I mean, does it make script/encoding faster when used instead of AVCSource/FFVideosource etc? I have GTX 460 GPU which has VP4.I think so. CPU usage for decoding is effectively 0 which means more of your CPU's resources can encode.

[aldix]

Thanks a lot for the reply. Very intriguing. I'm going to think about it some more and then perhaps indeed purchase it. :)

[Stereodude]

Can you try 768,512 in the DGSource block? If that still has problem, it is likely a bug in MPP and I will look into it when I get some free time...Sorry, I've been meaning to post back with the results of my findings. But, I couldn't get it to lock up / crash no matter the settings I tried yesterday though I didn't let it run for hours on end. I'm not sure what's going on. I'll mess with it again later.

[SAPikachu]

Sorry, I've been meaning to post back with the results of my findings. But, I couldn't get it to lock up / crash no matter the settings I tried yesterday though I didn't let it run for hours on end. I'm not sure what's going on. I'll mess with it again later.

Thanks and no problem, just take your time. :)

[real.finder]

hi SAPikachu :)

as it is known, a lot of programs and tools (like x264 itself) can use the avs .dll that in the same folder with it and ignore the main version in syswow64 or system32

and there are programs can use internal avs .dll like megui (in tools\avs folder), or external one like avspmod

I wonder if it possible to make every Block in mp_pipeline works with a different versions of avisynth by choosing the .dll in the block

something like:-

### platform: win32(".dll path here")

or

### AVS: (".dll path here")

it will be a wonderful thing, and easy to test different versions of the AVS, such as avisynth+ and MTs ver., in both 64 or 32

and the more importantly thing is to take advantage of the features of each one of them according to need in the Blocks

----------

The second thing (is not that important) is that mp_pipeline having the same problem here (http://forum.doom9.org/showthread.php?t=145912)

the last ver. of fixed avs2yuv was here (http://forum.doom9.org/showthread.php?p=1527698#post1527698)

thanks :)

[SAPikachu]

hi SAPikachu :)

as it is known, a lot of programs and tools (like x264 itself) can use the avs .dll that in the same folder with it and ignore the main version in syswow64 or system32

and there are programs can use internal avs .dll like megui (in tools\avs folder), or external one like avspmod

I wonder if it possible to make every Block in mp_pipeline works with a different versions of avisynth by choosing the .dll in the block

something like:-

### platform: win32(".dll path here")

or

### AVS: (".dll path here")

it will be a wonderful thing, and easy to test different versions of the AVS, such as avisynth+ and MTs ver., in both 64 or 32

and the more importantly thing is to take advantage of the features of each one of them according to need in the Blocks

----------

The second thing (is not that important) is that mp_pipeline having the same problem here (http://forum.doom9.org/showthread.php?t=145912)

the last ver. of fixed avs2yuv was here (http://forum.doom9.org/showthread.php?p=1527698#post1527698)

thanks :)

1. This shouldn't be very hard, will do it in a few days.

2. Will come along with #1 :)

[real.finder]

1. This shouldn't be very hard, will do it in a few days.

2. Will come along with #1 :)

Thank you. I'm looking forward to it :)

[SAPikachu]

Thank you. I'm looking forward to it :)

0.15 released with the 2 changes.

[real.finder]

0.15 released with the 2 changes.

Finally :)

thank you so much :)

[turbojet]

Thanks for adding avisynth dll function, it works great, and surprisingly very useful currently.

[turbojet]

Is there any way to split a source into 2 parts and ivtc them in separate processes then join them at the end?

Trying something like this:
MP_Pipeline("""
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
### pass a
### ###
b=dss2("source").trim(framecount/2+1,0).tfm().tdecimate()
### pass a, b
### ###
a+b
lanczosresize(1280,720)
""")


Throws 'I don't know what "a" means'

TDecimate is the bottleneck here, with it there's only 40-50% cpu usage. If this can be done, I'll need to figure out how to leave some extra frames (50?) when possible and for motion estimation and exclude them while joining. Also make the split divisible by 5 for ivtc.

[real.finder]

try

MP_Pipeline("""
### platform: win32
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
### export clip: a
### ###
### platform: win32
b=dss2("source").trim(framecount/2+1,0).tfm().tdecimate()
### pass clip: a
### export clip: b
### ###
### platform: win32
a+b
lanczosresize(1280,720)
### ###
""")


Should work

--------


I think that mp_pipeline need a good guide to be used by more people easily :)

[turbojet]

Doesn't know what framecount means, strange.

If I change trim to numbers it throws: Invalid arguments to function "MPP_PrepareDownstreamClip"

[Gavino]

Doesn't know what framecount means, strange.
That's because there is no 'last' defined at that point.
Replace
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
by
a=dss2("source")
a=trim(0,a.framecount/2).tfm().tdecimate()
and similarly for 'b'.

[SAPikachu]

Is there any way to split a source into 2 parts and ivtc them in separate processes then join them at the end?

Trying something like this:
MP_Pipeline("""
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
### pass a
### ###
b=dss2("source").trim(framecount/2+1,0).tfm().tdecimate()
### pass a, b
### ###
a+b
lanczosresize(1280,720)
""")


Throws 'I don't know what "a" means'

TDecimate is the bottleneck here, with it there's only 40-50% cpu usage. If this can be done, I'll need to figure out how to leave some extra frames (50?) when possible and for motion estimation and exclude them while joining. Also make the split divisible by 5 for ivtc.

It doesn't make sense to split the clip in this way, since final output frame is requested sequentially, the second process will be idle before first half of the clip is processed, then the first process will become idle.

If you don't want to run 2 encoding processes, you can try ### branch, like this:


MP_Pipeline("""
# Note: I am not sure whether dss2 guarantees frame-accurate,
# if not, you need to use another source filter that is frame-accurate.
dss2("source")
tfm()
tdecimate()

# 2 branch processes, 50 frames per thunk
### branch: 2, 50
# Prefetch 100 frames (2 thunks). Reduce this and thunk size if you don't have enough memory
### prefetch: 100, 0
### ###
lanczosresize(1280,720)
""")

[turbojet]

With that script I get:

MP_Pipeline: Error while creating last part of the filter chain:
ThunkedInterleave: Frame count of all clips except clip with last thunk must be
multiples of thunk_size. (Incorrect clip: #0)

According to some things I've read DSS2 is frame-accurate which is the advantage over DSS but even in the DSS2 mod (https://code.google.com/p/xvid4psp/downloads/detail?name=DSS2%20mod%20%2B%20LAVFilters.7z&can=2&q=) readme it's written 'Since DirectShow seeking isn't frame-accurate (or time-accurate,
as there is no frames, only time)' I found a bug with DSS2 and interlaced mpeg2 videos that are over 36 minutes long. FFMS2 showed very inaccurate fps and currently trying LWLibavsource, which seems to be the fastest overall after setting dr=true.

2 sequential encodes might be what I end up doing but it's not a very big speed increase, 44-45 fps at 40-50% CPU to 54-55 fps at 100% cpu. I was expecting more considering cpu usage.

[SAPikachu]

With that script I get:

MP_Pipeline: Error while creating last part of the filter chain:
ThunkedInterleave: Frame count of all clips except clip with last thunk must be
multiples of thunk_size. (Incorrect clip: #0)


Can you post your full script? I just tried on my machine and didn't get that error.


According to some things I've read DSS2 is frame-accurate which is the advantage over DSS but even in the DSS2 mod (https://code.google.com/p/xvid4psp/downloads/detail?name=DSS2%20mod%20%2B%20LAVFilters.7z&can=2&q=) readme it's written 'Since DirectShow seeking isn't frame-accurate (or time-accurate,
as there is no frames, only time)' I found a bug with DSS2 and interlaced mpeg2 videos that are over 36 minutes long. FFMS2 showed very inaccurate fps and currently trying LWLibavsource, which seems to be the fastest overall after setting dr=true.

2 sequential encodes might be what I end up doing but it's not a very big speed increase, 44-45 fps at 40-50% CPU to 54-55 fps at 100% cpu. I was expecting more considering cpu usage.


Yes I think LWLibavsource is a better choice.

[turbojet]

MP_Pipeline("""
#lwlibavvideosource("mppthunk.mpg")
blankclip(163,1920,1080,"YV12",29.97).assumefps(30000,1001) #same as source, also errors
tfm()
tdecimate()

### branch: 2, 50
### prefetch: 100, 0
### ###
lanczosresize(1280,720)
""")

source (http://www.sendspace.com/file/0y0rx5) if you need it. Can try blankclip first.

[SAPikachu]

MP_Pipeline("""
#lwlibavvideosource("mppthunk.mpg")
blankclip(163,1920,1080,"YV12",29.97).assumefps(30000,1001) #same as source, also errors
tfm()
tdecimate()

### branch: 2, 50
### prefetch: 100, 0
### ###
lanczosresize(1280,720)
""")

source (http://www.sendspace.com/file/0y0rx5) if you need it. Can try blankclip first.

OK, I have found source of the error, it is related to length of the clip. I will fix it later when I get around, for a temporary workaround you may pad some frames after tdecimate, like:

MP_Pipeline("""
#lwlibavvideosource("mppthunk.mpg")
blankclip(163,1920,1080,"YV12",29.97).assumefps(30000,1001) #same as source, also errors
tfm()
tdecimate()
last + blankclip(last, 50)

### branch: 2, 50
### prefetch: 100, 0
### ###
trim(0, framecount-1-50)
lanczosresize(1280,720)
""")

[turbojet]

Thanks that increased the speed from 49 to 67 fps. I found out tfm is the bottleneck. Telecide().tdecimate() mpp goes from 60 to 84 fps. Mpp slows down uncomb().tdecimate() from 113 to 94 fps.

[real.finder]

hi :)

I have some reports

It seems that the avs dll function not work with old versions of avs

test it with 257 mt (http://www.avisynth.nl/users/tsp/MT_07.zip) and 257 official
will show

1 (http://i.imgur.com/thApjtc.png) then 2 (http://i.imgur.com/Sjummwq.png) and last 3 (http://i.imgur.com/DY6Z9zQ.png)

edit: the problem in plugins, it seems not compatible with older versions

edit 2: for who cares about that, the problems were in some avsi files (like dither.avsi) , because 257 does not support /* comment */

and in 258 it was because Srestore.avsi 2.7g :-

Function StrReplace(string s, string find, string replace) # Repeated, string replacements
# Original:- http://forum.doom9.org/showthread.php?t=147846&highlight=gscript By Vampiredom, Gavino, IanB


the correct for 258 is :-


# Original:- http://forum.doom9.org/showthread.php?t=147846&highlight=gscript By Vampiredom, Gavino, IanB
# Repeated, string replacements
Function StrReplace(string s, string find, string replace)
{etc...}



thanks

[steptoe]

I'm still trying to get my head around how to run 32bit plugins on a 64bit avisynth install, or just to start with running a combination of 32bit and 64bit plugins on a 32bit avisynth install if 64bit plugins are available for the filters/scripts I use. Once I get that working then I'll look into trying 64bit avisynth

It may sound dumb to some, but what simple steps are needed to get mp_pipeline running. I've tried but struggling with the basic setup


I have 32bit avisynth+ installed, running windows 7 64bit on my Intel i5-2500k at 4.5ghz stable, with 16GB memory

The scripts/plugins I use work perfectly with either AVStoDVD or DVD-RB Pro. Depending what I'm backing up

I'm looking to try and squeeze a bit more out of my system, which isn't exactly slow, but can't justify dropping in an Intel i7 just for video work

I'm mainly using RemoveDirtMC, TomsMoComp, Unsharp, AutoGain or AutoAdjust and FluxSmooth with Toon/awarpsharp2 for animation. No complex all-in-one noise scripts just to clean the source a bit with some mild sharpening of usually 70's/80's DVD sources of childrens cartoons such as DangerMouse, The Clangers, Trapdoor, BagPuss, Paddington Bear, Mr Benn, netc. That era

Thanks

[SAPikachu]

I'm still trying to get my head around how to run 32bit plugins on a 64bit avisynth install, or just to start with running a combination of 32bit and 64bit plugins on a 32bit avisynth install if 64bit plugins are available for the filters/scripts I use. Once I get that working then I'll look into trying 64bit avisynth

It may sound dumb to some, but what simple steps are needed to get mp_pipeline running. I've tried but struggling with the basic setup


I have 32bit avisynth+ installed, running windows 7 64bit on my Intel i5-2500k at 4.5ghz stable, with 16GB memory

The scripts/plugins I use work perfectly with either AVStoDVD or DVD-RB Pro. Depending what I'm backing up

I'm looking to try and squeeze a bit more out of my system, which isn't exactly slow, but can't justify dropping in an Intel i7 just for video work

I'm mainly using RemoveDirtMC, TomsMoComp, Unsharp, AutoGain or AutoAdjust and FluxSmooth with Toon/awarpsharp2 for animation. No complex all-in-one noise scripts just to clean the source a bit with some mild sharpening of usually 70's/80's DVD sources of childrens cartoons such as DangerMouse, The Clangers, Trapdoor, BagPuss, Paddington Bear, Mr Benn, netc. That era

Thanks

Even using MP_Pipeline, you need to install both x86/x64 versions of Avisynth to to able to run all x86/x64 plugins. Here is an example of script:


MP_Pipeline("""

### platform: win64

LoadPlugin("Some_x64_plugin.dll")

SomeSource()
SomeX64Filter()

### ###

### platform: win32

LoadPlugin("Some_x86_plugin.dll")

SomeX86Filter()

### ###

""")

[real.finder]

hi :)

in the 0.15 ver. when I load script in MPC or x264 (or anything) and than close it, some slave processes stay active in task manger and take memory, with 0.14 this not happened

and sometimes if used some complex script with dvd sources (480) some slave processes crashed (http://i.imgur.com/mXqSyQh.png) in closing (or end), it does not affect, but it's annoying, and that in both 0.14 or 0.15, and this sometimes not happened even with the same script

I will PM you a dump file for 0.15 now

[aldix]

Yeah, I can also report hanging slave processes.

[steptoe]

Many thanks for the reply trying to use MP-Pipeline, I didn't know I needed 64bit/32bit avisynth installed at the same time. I'll have another try at some point


Thanks again for making it much more obvious how to get it to work

[SAPikachu]

hi :)

in the 0.15 ver. when I load script in MPC or x264 (or anything) and than close it, some slave processes stay active in task manger and take memory, with 0.14 this not happened

and sometimes if used some complex script with dvd sources (480) some slave processes crashed (http://i.imgur.com/mXqSyQh.png) in closing (or end), it does not affect, but it's annoying, and that in both 0.14 or 0.15, and this sometimes not happened even with the same script

I will PM you a dump file for 0.15 now

Thanks for the dump, I guess there are some plugins have troubles when destructing in slave process. I will try to fix it next week.

[Dogway]

Hello, great plugin, very stable!

Is this right?

MP_Pipeline("""
DGSource("1080i_source.dgi")
AssumeTFF()
QTGMC(preset="faster",border=false,edithreads=1,showsettings=false)
### prefetch: 64, 32
### ###

bicubicResize(1024,576,b=-.5,c=.25)
ColorMatrix(mode="Rec.709->Rec.601",threads=1)
limiter()
### branch: 4, 8
### prefetch: 32, 16
### ###
""")

I get an average of 40% on my 4 cores, is there a way to say, limit the process to 2 cores (I noticed how load decreases when core number increases...) and use the other 2 for another encode (ie. second half of source)? For RAM's sake ideally I would like to reuse loaded calls, but I guess the only way is to open another instance of vdub?

Also on another script I had to remove the last block limiter, before closing MP_Pipeline, how does this work?

edit: also to add I can't make dither_resize16() work inside mp_pipeline, I wonder if there's a multithreading conflict there.

[SAPikachu]

(Sorry for the late reply, recently I am too busy to check here)

Hello, great plugin, very stable!

Is this right?

MP_Pipeline("""
DGSource("1080i_source.dgi")
AssumeTFF()
QTGMC(preset="faster",border=false,edithreads=1,showsettings=false)
### prefetch: 64, 32
### ###

bicubicResize(1024,576,b=-.5,c=.25)
ColorMatrix(mode="Rec.709->Rec.601",threads=1)
limiter()
### branch: 4, 8
### prefetch: 32, 16
### ###
""")

I get an average of 40% on my 4 cores, is there a way to say, limit the process to 2 cores (I noticed how load decreases when core number increases...) and use the other 2 for another encode (ie. second half of source)? For RAM's sake ideally I would like to reuse loaded calls, but I guess the only way is to open another instance of vdub?


You may need to set thread affinity in task manger for each slave process, "### lock threads to cores" didn't work well for multiple encoding pipeline at the same time.

Also I think "### branch" in the second block should be removed, since the first block has QTGMC, the second block won't be slower than first block, it doesn't make sense to branch it


Also on another script I had to remove the last block limiter, before closing MP_Pipeline, how does this work?


Sorry I don't understand this, what did you want to do?


edit: also to add I can't make dither_resize16() work inside mp_pipeline, I wonder if there's a multithreading conflict there.

Any error message?

[Dogway]

Thanks for reply.

With dither_resize16() the script is just unable to load, freezes vdub. Could you make it work before?

Thanks for scripting help, while the above script was succesfully encoded, I think I get RAM issues with more complex scripts, so I tend to avoid prefetch of 64 frames, etc. I'm currently on XP x86, and while I plan to switch to 7 x64 soon I wonder what are my options RAM wise for the below script, halving buffer and threading at the same time just doesn't seem to come along. I'm kinda stuck on what could be optimal in this scenario (using quad core i5 here), my current script is as follows:

MP_Pipeline("""
setmemorymax(1024)
DGSource("1080p.dgi",crop_t=132,crop_b=140)
### prefetch: 16, 0
### ###

bicubicResize(1280,544,b=-.5,c=.25)
# ### branch: 2, 4 # not sure about this
### prefetch: 16, 16
### ###

pre=fluxsmoothT(3)
### export clip: pre
### prefetch: 16, 8
### ###

smdegrain(tr=2,thSAD=170,prefilter=pre,lsb_out=true,refinemotion=true,contrasharp=30)
DitherPost(stacked=true,prot=false,mode=6,ampn=1,staticnoise=true)
Limiter()
### ###
""")

[SAPikachu]

Thanks for reply.

With dither_resize16() the script is just unable to load, freezes vdub. Could you make it work before?


Just tried and it works without problem in AvsP and vdub.


Thanks for scripting help, while the above script was succesfully encoded, I think I get RAM issues with more complex scripts, so I tend to avoid prefetch of 64 frames, etc. I'm currently on XP x86, and while I plan to switch to 7 x64 soon I wonder what are my options RAM wise for the below script, halving buffer and threading at the same time just doesn't seem to come along. I'm kinda stuck on what could be optimal in this scenario (using quad core i5 here), my current script is as follows:

MP_Pipeline("""
setmemorymax(1024)
DGSource("1080p.dgi",crop_t=132,crop_b=140)
### prefetch: 16, 0
### ###

bicubicResize(1280,544,b=-.5,c=.25)
# ### branch: 2, 4 # not sure about this
### prefetch: 16, 16
### ###

pre=fluxsmoothT(3)
### export clip: pre
### prefetch: 16, 8
### ###

smdegrain(tr=2,thSAD=170,prefilter=pre,lsb_out=true,refinemotion=true,contrasharp=30)
DitherPost(stacked=true,prot=false,mode=6,ampn=1,staticnoise=true)
Limiter()
### ###
""")

4GB memory is really not enough for MPP to speed up complex script, so I think you don't have many choices except for switching to x64 with more memory...

[SAPikachu]

Released 0.16. Thanks @real.finder, @aldix and @turbojet reporting the bugs!

[real.finder]

Released 0.16. Thanks @real.finder, @aldix and @turbojet reporting the bugs!

Finally :)

:thanks:

[Stereodude]

FWIW, you might want to make it clear in the readme / documentation that each slave process must return something. IE:

Will not work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

Will work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
clip1
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

I was banging my head on the wall for quite some time trying to figure out why I couldn't get export to work and kept getting this message:
Script error: Invalid arguments to function
"MPP_PrepareDownstreamClip"

BTW, thanks for the very useful plugin!

[Dogway]

Just tried and it works without problem in AvsP and vdub.
I will look into that.


4GB memory is really not enough for MPP to speed up complex script, so I think you don't have many choices except for switching to x64 with more memory...

I was hoping something more elaborated, I guess that with that I can read that my script is correct, right? despite the 16, 16, 16 prefetch values.

@Stereodude: "last" should get passed through variables, if not I wonder if "### export clip: last" could work, otherwise explicitly calling last and exporting to next core could help, since on a normal workflow you wouldn't want to output clip1 or clip2 at that early stage.

[Stereodude]

@Stereodude: "last" should get passed through variables, if not I wonder if "### export clip: last" could work, otherwise explicitly calling last and exporting to next core could help, since on a normal workflow you wouldn't want to output clip1 or clip2 at that early stage.
You can also return last like this:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
last
### export clip: clip1, clip2
### prefetch: 8, 4
### ###
Heck, you could return ColorBars(), but unless you have some explicit action being done in the slave process outside of a variable you get the error I mentioned.

[Dogway]

You can also return last like this:


Didn't run onto that bug before, but thanks, nice to know.

Just tried and it works without problem in AvsP and vdub.

Problem is avstp.dll (by cretindesalpes), put this in your plugin folder and try.
I don't know whose fault. It's just a combination of dither_resize16(), MP_Pipeline() and avstp.dll which is needed for MT with cretindesalpes' mvtools. I don't know if it's wise to drop avstp.dll when dealing with mvtools based functions or whether it helps in conjunction to MP_Pipeline().

[SAPikachu]

FWIW, you might want to make it clear in the readme / documentation that each slave process must return something. IE:

Will not work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

Will work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
clip1
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

I was banging my head on the wall for quite some time trying to figure out why I couldn't get export to work and kept getting this message:
Script error: Invalid arguments to function
"MPP_PrepareDownstreamClip"

BTW, thanks for the very useful plugin!

Thanks, I just added a note about this to the OP.

[SAPikachu]

I was hoping something more elaborated, I guess that with that I can read that my script is correct, right? despite the 16, 16, 16 prefetch values.


Well, your script doesn't have obvious error, but I would suggest trying this:


MP_Pipeline("""
DGSource("1080p.dgi",crop_t=132,crop_b=140)
bicubicResize(1280,544,b=-.5,c=.25)
pre=fluxsmoothT(3)
### export clip: pre
### prefetch: 16, 8
### ###

setmemorymax(1024)
# Put smdegrain into separate process, since it is the most complex part
smdegrain(tr=2,thSAD=170,prefilter=pre,lsb_out=true,refinemotion=true,contrasharp=30)
### prefetch: 16, 0

### ###
DitherPost(stacked=true,prot=false,mode=6,ampn=1,staticnoise=true)
Limiter()
### ###
""")



Problem is avstp.dll (by cretindesalpes), put this in your plugin folder and try.
I don't know whose fault. It's just a combination of dither_resize16(), MP_Pipeline() and avstp.dll which is needed for MT with cretindesalpes' mvtools. I don't know if it's wise to drop avstp.dll when dealing with mvtools based functions or whether it helps in conjunction to MP_Pipeline().

I have avstp.dll in my plugins folder, and it didn't cause any problem..

[real.finder]

I have avstp.dll in my plugins folder, and it didn't cause any problem..

I test it, ok in win7 but in winxp will happen like dogway said

Does not occur if you do not put avsp.dll in autoload folder, and if you load avsp.dll manually in script (in xp) will not happen and will work normally :eek:

[SAPikachu]

OK I reproduced the problem in an XP VM, looks like a conflict between different versions of C runtime. You can try Avisynth+ (http://www.avs-plus.net/), seems it doesn't have this problem.

[Dogway]

Thanks real.finder I was starting to feel paranoid. I will load on demand, and switch to Avs+ on the long term along Win7, which is the next natural step on avisynth development. I just feel it's a bit too early with lack of MT, etc.

As for the script improvement, I wonder if fluxsmoothT doesn't need some prefetch too since it's looking for and backwards?

[aldix]

Released 0.16. Thanks @real.finder, @aldix and @turbojet reporting the bugs!


Excellent news, thanks so much :)

[SAPikachu]

As for the script improvement, I wonder if fluxsmoothT doesn't need some prefetch too since it's looking for and backwards?

Actually fluxsmoothT already has prefetch. I think what you meant is whether upstream of fluxsmoothT needs prefetching, right? IMO it won't make much difference because there are only 2 upstream filters, but of course I may be wrong and you can try it to see what happens. :)

[Dogway]

Sure, I guess I messed up with my wording. I thought that bicubicresize (or DitherResize16() for instance) would drag a bit too much for fluxsmoothT to keep going, specially resizes of such gap. The above is already filtered and got about 6.4fps and ~60% load on a i5-4670k, I think it's ok since I was looking for >=5fps rate. Thanks for help!

[SAPikachu]

Sure, I guess I messed up with my wording. I thought that bicubicresize (or DitherResize16() for instance) would drag a bit too much for fluxsmoothT to keep going, specially resizes of such gap. The above is already filtered and got about 6.4fps and ~60% load on a i5-4670k, I think it's ok since I was looking for >=5fps rate. Thanks for help!

Well, bicubicresize isn't very slow (at least much faster than smdegrain, not sure about DitherResize16 though), the bottleneck is smdegrain (correct me if I am wrong), so it shouldn't matter much. Anyways glad to hear you are happy with the speed. :)

[Dogway]

I'm having serious issues with a script, export clip doesn't seem to work:
MP_Pipeline("""
setmemorymax(1024)
ffvideoSource("E:\Part A.avi")

### inherit start ###
source=last
### inherit end ###
### prefetch: 16, 8
### ###

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
In this way the rain variable is getting "last" as source, instead of inherited "source" variable. Why is this? If I move the dfttest line also above the splitter, the "clean" variable also gets "last" (the denoised dfttest) instead of "source". I tested with a simplified code using mt_lut() and that worked so I'm not sure why it occurs.

[Stereodude]

I'm having serious issues with a script, export clip doesn't seem to work:
MP_Pipeline("""
setmemorymax(1024)
ffvideoSource("E:\Part A.avi")

### inherit start ###
source=last
### inherit end ###
### prefetch: 16, 8
### ###

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
In this way the rain variable is getting "last" as source, instead of inherited "source" variable. Why is this?Because you're not using it correctly. The inherit block with source = last is the same as putting source = last in every thread / segment. But, based on your description that's not what you're trying to do (redefine source in each thread / segment to be the implied output from the previous thread / segment. It needs to be like this:

MP_Pipeline("""
setmemorymax(1024)
source=ffvideoSource("E:\Part A.avi")
source
### export clip: source
### prefetch: 16, 8
### ###

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean
### pass clip: source
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
or:
MP_Pipeline("""
setmemorymax(1024)
ffvideoSource("E:\Part A.avi")
### prefetch: 16, 8
### ###
source = last

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean, source
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
However, I think there still might be another mistake or two in there. The output from Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true) becomes last for the next thread / segment. Is this what you want? Also, in my two modifications in this line clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true) the source and last are the same footage (I think).

[Dogway]

Thanks a bunch Stereodude, this was driving me mad.
I guess that's what I wanted; "pass clip".

I don't know what errors you refer to. Dfttest is "last" until ReplaceFramesSimple comes in and outputs either "clean" or "rain". "rain" also needs dfttest because that's what mfilter needs, and also "source" (before dfttest) as the source of smdegrain. I'm going to check, because I would swear I tried your second variation and got as you said the same clip for "last" and "source" but for the "rain" variable. edit: worked using 2nd variation too.

[Stereodude]

I don't know what errors you refer to. Dfttest is "last" until ReplaceFramesSimple comes in and outputs either "clean" or "rain". "rain" also needs dfttest because that's what mfilter needs, and also "source" (before dfttest) as the source of smdegrain. I'm going to check, because I would swear I tried your second variation and got as you said the same clip for "last" and "source" but for the "rain" variable. edit: worked using 2nd variation too.
I will admit to not completely understanding how last works in AVIsynth. I don't typically use it. I had never used it prior to using MP_Pipeline. If it is updated after every filter not explicitly assigned to a variable as the implied return variable, then there's no error. I was thinking last was only the implied return from the thread / segment (or entire script if not using MP_Pipeline), not an implied variable that got updated after each line not assigned to a variable.

[Stereodude]

hi :)

in the 0.15 ver. when I load script in MPC or x264 (or anything) and than close it, some slave processes stay active in task manger and take memory, with 0.14 this not happened

and sometimes if used some complex script with dvd sources (480) some slave processes crashed (http://i.imgur.com/mXqSyQh.png) in closing (or end), it does not affect, but it's annoying, and that in both 0.14 or 0.15, and this sometimes not happened even with the same script.

Yeah, I can also report hanging slave processes.I'm still seeing a variation of this with .16. When working with scripts in Virtualdub the x86 version is pretty good about closing them, but the x64 version of VD with x64 MP_Pipeline seems to just leave them sit open. When you close VD they all disappear, but reloading the file (F2) or closing it and re-opening it causes threads to stay open.

[SAPikachu]

I'm still seeing a variation of this with .16. When working with scripts in Virtualdub the x86 version is pretty good about closing them, but the x64 version of VD with x64 MP_Pipeline seems to just leave them sit open. When you close VD they all disappear, but reloading the file (F2) or closing it and re-opening it causes threads to stay open.

Thanks for the report, will have a look on it later this week.

[Stereodude]

Thanks for the report, will have a look on it later this week.I found a specific repeatable case where the 32-bit version leaves threads open. If a script fails to open successfully in VirtualDub, generating some error during the opening process, the threads seem to end up orphaned and will stay open until you close VirtualDub.

[steptoe]

I still can't get my head around how MP_Pipeline works, basically I'm trying to get this to work in 32-bit. I use avisynth+

Once I can get that to work, I'll then try the 64-bit versions of the filters. Both 32-bit and 64-bit of avisynth+ are installed


Thanks again


SetMemoryMax(1024)
dss2("D:\PugWash.mkv")

Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\mvtools.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirt.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removegrain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\autogain\autogain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\tdeint\tdeint.dll")
Import("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirtmc.avs")

tdeint()
removedirtmc(75,false)
autogain()

This works perfectly in AVSPMod, but trying to get that to work with MP_Pipeline I just can't get my head around the commands

[SAPikachu]

I found a specific repeatable case where the 32-bit version leaves threads open. If a script fails to open successfully in VirtualDub, generating some error during the opening process, the threads seem to end up orphaned and will stay open until you close VirtualDub.

Finally got around to look on this. I can't reproduce the problem on latest (1.10.4) x64 version of VirtualDub if the script doesn't contain error, every time I hit F2 old slave processes are properly killed. Do you have a script that can reliably trigger this problem?

On the other hand, if the script has error, slave processes can't indeed be killed until host process exits. I will release a fix for this a bit later.

[SAPikachu]

I still can't get my head around how MP_Pipeline works, basically I'm trying to get this to work in 32-bit. I use avisynth+

Once I can get that to work, I'll then try the 64-bit versions of the filters. Both 32-bit and 64-bit of avisynth+ are installed


Thanks again


SetMemoryMax(1024)
dss2("D:\PugWash.mkv")

Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\mvtools.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirt.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removegrain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\autogain\autogain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\tdeint\tdeint.dll")
Import("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirtmc.avs")

tdeint()
removedirtmc(75,false)
autogain()

This works perfectly in AVSPMod, but trying to get that to work with MP_Pipeline I just can't get my head around the commands

Here is a starting point for you, you need to tweak the numbers yourself to fit your system, and I am not sure whether the speed will increase because RemoveDirtMC doesn't look very complex to me.

MP_Pipeline("""

### inherit start ###

Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\mvtools.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirt.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removegrain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\autogain\autogain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\tdeint\tdeint.dll")
Import("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirtmc.avs")

### inherit end ###

dss2("D:\PugWash.mkv")
tdeint()
### prefetch: 32, 16
### ###

SetMemoryMax(1500) # Reduce this if you don't have enough memory
removedirtmc(75,false)
### prefetch: 32, 0
### ###

autogain()

""")

[Stereodude]

Finally got around to look on this. I can't reproduce the problem on latest (1.10.4) x64 version of VirtualDub if the script doesn't contain error, every time I hit F2 old slave processes are properly killed. Do you have a script that can reliably trigger this problem?I'll see if I can come up with one that repeats it reliably. My initial report of an issue on x64 may have been caused by the 2nd issue, that slave processes hang around if the script has errors when opening. I'll do more testing.

[BloC]

how MP_Pipeline works and What are the best settings for these orders ؟؟؟
I am tired of slow Encoding in the my computer Intel Corei5
I useing avs32x and avs64 ram 8 Gib



LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", showyuv=false, analyze=false, autowhite=false, autogain=false)

[aldix]

Well, i5 is rather slow ...

[Stereodude]

how MP_Pipeline works and What are the best settings for these orders ؟؟؟
I am tired of slow Encoding in the my computer Intel Corei5
I useing avs32x and avs64 ram 8 Gib



LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", showyuv=false, analyze=false, autowhite=false, autogain=false)

What x264 settings are you using, and what is the CPU usage of your i5 while you're encoding?

Well, i5 is rather slow ...What a helpful reply. I wouldn't consider an i5 slow. Depending on which one he has and how fast it's running it could be anywhere from decent to fast.

[BloC]

What x264 settings are you using, and what is the CPU usage of your i5 while you're encoding?

What a helpful reply. I wouldn't consider an i5 slow. Depending on which one he has and how fast it's running it could be anywhere from decent to fast.

My X264 settings

cabac=1 / ref=4 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=umh / subme=6 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=1 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=abr / mbtree=1 / bitrate=2200 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=50000 / vbv_bufsize=50000 / nal_hrd=none / ip_ratio=1.40 / aq=1:1.00

I am encoding 5 fps When using these filters

This is because it bothered me I want to learn to use MP_Pipeline 0.16

[Stereodude]

I am encoding 5 fps When using these filters

This is because it bothered me I want to learn to use MP_Pipeline 0.16What does your CPU usage look like when encoding?

I'm not familiar with most of the filters you're using, so I don't know which of the filters are slow. What resolution is your input source?

Edit: Your script might look something like this:
LoadPlugin("MP_Pipeline.dll")

MP_Pipeline("""
### inherit start ###
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")
### inherit end ###
LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

### prefetch: 12, 6
### ###
Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)
### prefetch: 16, 8
### ###

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise
### prefetch: 18, 9
### ###

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, \
gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", \
showyuv=false, analyze=false, autowhite=false, autogain=false)
### prefetch: 20, 10
""")
It may have more threads than it needs, but I don't know how CPU intensive some of them are.

[SAPikachu]

Released 0.17 (finally..). @Stereodude can you try this version to see whether it fixes the hanging process problem?

[nekosama]

I'm using avisynth 2.6A5 and I'm trying to split up my script because it keeps crashing because of high memory usage.
But the problem is that pipeline just doesn't want to work.
http://i.imgur.com/TMlSqgq.png
here's my avs script.
MP_Pipeline("""op=import("other part of source.avs")
ed=import("other part of source.avs")
LWLibavVideoSource("etc.mkv",threads=1)
a=trim(0,1628)
b=trim(3861,30447)
c=trim(32608,34119)
a ++ op ++ b ++ ed ++ c
aa=ediaa()
softsmooth=vaguedenoiser(1,nsteps=8,method=3)
strange(24774,24917,ediaa().eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(568,1628,eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(4240,5376,aa)
strange(5377,5448,ediaa())
strange(5497,5952,ediaa())
strange(6061,30447,aa)
strange(32737,34119,aa)
### prefetch: 16, 0
### ###
""")
strange(33059,34119,softsmooth.gradfun3())

trim(30457,34120)
btw in the x64 folder there are files for win32 and in x86 folder files for win64, why? I'm using the 32 version (also I have 64 system but I thought you would need to run avisynth in 64 to use 64 bit pipeline, correct me if wrong)

[SAPikachu]

I'm using avisynth 2.6A5 and I'm trying to split up my script because it keeps crashing because of high memory usage.
But the problem is that pipeline just doesn't want to work.
http://i.imgur.com/TMlSqgq.png
here's my avs script.
MP_Pipeline("""op=import("other part of source.avs")
ed=import("other part of source.avs")
LWLibavVideoSource("etc.mkv",threads=1)
a=trim(0,1628)
b=trim(3861,30447)
c=trim(32608,34119)
a ++ op ++ b ++ ed ++ c
aa=ediaa()
softsmooth=vaguedenoiser(1,nsteps=8,method=3)
strange(24774,24917,ediaa().eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(568,1628,eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(4240,5376,aa)
strange(5377,5448,ediaa())
strange(5497,5952,ediaa())
strange(6061,30447,aa)
strange(32737,34119,aa)
### prefetch: 16, 0
### ###
""")
strange(33059,34119,softsmooth.gradfun3())

trim(30457,34120)


This is usually because the slave process crashed, can you try removing one part of your script at a time to see which line cause the problem?


btw in the x64 folder there are files for win32 and in x86 folder files for win64, why? I'm using the 32 version (also I have 64 system but I thought you would need to run avisynth in 64 to use 64 bit pipeline, correct me if wrong)

They are used for running scripts with mixing x86/x64 slaves, so that you can use x86 filters with x64 x264 for example. You need to install both x64 and x86 avisynth to use that.

[MeteorRain]

I am encoding 5 fps When using these filters


I won't say 5fps is slow on an i5 for QTGMC level script.

For example doing QTGMC on 1920x1080i source can easily put the speed into ~3fps level on my i7 4770.

If you don't want to put too much time on MP_Pipeline I'd rather suggest you do parallel encoding, 2 a time in your case, if the process doesn't consume more than 50% of overall resource.

If you have a 6 physical cores you can do 3 at a time.

[BloC]

What does your CPU usage look like when encoding?

I'm not familiar with most of the filters you're using, so I don't know which of the filters are slow. What resolution is your input source?

Edit: Your script might look something like this:
LoadPlugin("MP_Pipeline.dll")

MP_Pipeline("""
### inherit start ###
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")
### inherit end ###
LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

### prefetch: 12, 6
### ###
Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)
### prefetch: 16, 8
### ###

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise
### prefetch: 18, 9
### ###

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, \
gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", \
showyuv=false, analyze=false, autowhite=false, autogain=false)
### prefetch: 20, 10
""")
It may have more threads than it needs, but I don't know how CPU intensive some of them are.

wooow wooow this scripts very good

thanks man for helping me ;)

[Dogway]

Is it possible to pass variables to next thread? They are not clip variables though, I'm trying to do something like this:

smdegrain(globals=2)
### ###
smdegrain(globals=1)

The second instance tries to read all the motion vectors exported from the first smdegrain, that is, Super, bv2, bv1, fv1, fv2, etc. I tried with export clip, but obviously didn't work.


Also out of curiosity, on a quad core what is an optimal division, 4, or 3 code chunks (leaving one free for the encoder)?

[SAPikachu]

Is it possible to pass variables to next thread? They are not clip variables though, I'm trying to do something like this:

smdegrain(globals=2)
### ###
smdegrain(globals=1)

The second instance tries to read all the motion vectors exported from the first smdegrain, that is, Super, bv2, bv1, fv1, fv2, etc. I tried with export clip, but obviously didn't work.


They are actually clip variables, but MVTools stores some additional data in another obscure place, so it is not possible to pass these clips to downstream (at least for now).


Also out of curiosity, on a quad core what is an optimal division, 4, or 3 code chunks (leaving one free for the encoder)?

Well, I think there is no solution suited for every case, it depends on what filters are used. I usually disregard number of cores and just place heavy filters in separate chunks when possible. Even then my CPU cannot be fully utilized.

[Dogway]

I see, cretindesalpes worked in mvtools mod, from the few I know are active and might know the internals of the plugin.

I probably wanted to say threads instead of cores, mine is a i5 so 4 cores 4 threads, but I guess your answer still applies here (making 5 or 6 chunks no problem). Thanks for help.

[real.finder]

I discovered fun fact

I have two different editions of Srestore.avsi in two autoload folder, one in plugins64 and the other in plugins

the one in plugins is the newest and accept call Srestore more than once in the script


The script was like:-

MP_Pipeline("""
### platform: win32
some filters
Srestore
Srestore
### ###
""")

When runs it in x264 32-bit works normally, but in x264 64-bit gives ERROR: SRESTORE CAN ONLY CALLED ONCE!!!

Which means that autoload avsi from out of MP_Pipeline which is 64-bit was affect in the processes, and if I import the new Srestore.avsi from plugins manually after ### platform: win32 problem will be solved

[SAPikachu]

I discovered fun fact

I have two different editions of Srestore.avsi in two autoload folder, one in plugins64 and the other in plugins

the one in plugins is the newest and accept call Srestore more than once in the script


The script was like:-

MP_Pipeline("""
### platform: win32
some filters
Srestore
Srestore
### ###
""")

When runs it in x264 32-bit works normally, but in x264 64-bit gives ERROR: SRESTORE CAN ONLY CALLED ONCE!!!

Which means that autoload avsi from out of MP_Pipeline which is 64-bit was affect in the processes, and if I import the new Srestore.avsi from plugins manually after ### platform: win32 problem will be solved

This is interesting, but I just had a quick test and can't get this behavior, 32bit and 64bit x264 don't affect slave's environment. Not sure what happened in your system though..

[real.finder]

This is interesting, but I just had a quick test and can't get this behavior, 32bit and 64bit x264 don't affect slave's environment. Not sure what happened in your system though..

My experience was only in Srestore and call it twice in the script

Didn't try something else

the summary of what happened, if I run the script in x264 64-bit, the Srestore in plugins64 will be adopted in 32 slave's environment in mp_pipeline, Ignoring that one in 32 autoload folder

and by the way, I using avs 64 (http://code.google.com/p/avisynth64/downloads/detail?name=avisynth64_8-29-10.rar&can=2&q=) and SEt mt (https://www.dropbox.com/s/xhqggxamegia420/avisynth_20130309.7z)

in windows server 2008 r2

[Boulder]

As SRestore is being discussed, is this a proper way to use MP_Pipeline:

MP_Pipeline("""
avstp_set_threads(2)
DGSource("who_coliseum.dgi")
Trim(0,43685)
BicubicResize(1280,1080)
QTGMC("slower",dct=5,tr2=0,chromamotion=true,chromanoise=false,SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1,edithreads=2)
BicubicResize(1280,720)
### prefetch 3,3
### ###
""")
SRestore(frate=25,mode=-2,speed=-1,dclip=Reduceby2(),cache=10)

AFAIK, SRestore needs linear access which should be ensured by the cache parameter.

[SAPikachu]

As SRestore is being discussed, is this a proper way to use MP_Pipeline:

MP_Pipeline("""
avstp_set_threads(2)
DGSource("who_coliseum.dgi")
Trim(0,43685)
BicubicResize(1280,1080)
QTGMC("slower",dct=5,tr2=0,chromamotion=true,chromanoise=false,SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1,edithreads=2)
BicubicResize(1280,720)
### prefetch 3,3
### ###
""")
SRestore(frate=25,mode=-2,speed=-1,dclip=Reduceby2(),cache=10)

AFAIK, SRestore needs linear access which should be ensured by the cache parameter.

1. There should be a colon after ### prefetch .
2. You can omit cache parameter in SRestore if prefetch is used, MPP will cache frames itself.
3. 3,3 may be not enough for SRestore AFAIK, and if the two numbers equal, MPP will only cache frames behind the current frame and won't prefetch frames.

I suggest trying this:
MP_Pipeline("""
avstp_set_threads(2)
DGSource("who_coliseum.dgi")
Trim(0,43685)
BicubicResize(1280,1080)
QTGMC("slower",dct=5,tr2=0,chromamotion=true,chromanoise=false,SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1,edithreads=2)
BicubicResize(1280,720)
### prefetch: 20, 10
### ###
""")
SRestore(frate=25,mode=-2,speed=-1,dclip=Reduceby2())

[Boulder]

Thanks, it seems to be running very smoothly now :)

[real.finder]

AFAIK, SRestore needs linear access which should be ensured by the cache parameter.

I used it with mt mode 2 naturally

and if the two numbers equal, MPP will only cache frames behind the current frame and won't prefetch frames.

so, with ### prefetch: 20, 10 will be 10 behind and 10 prefetch?

[SAPikachu]

so, with ### prefetch: 20, 10 will be 10 behind and 10 prefetch?

Correct.

[SAPikachu]

Released 0.18.

[turbojet]

I used to be able to load different avisynth.dll's fine but now having a problem. Any ideas?

MP_Pipeline("""###
### dll: "F:\avisynth+.dll"
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

MP_Pipeline: Unable to create slave process. Message: Unable to load "F:\avisynt
h+.dll", code = 126

[SAPikachu]

I used to be able to load different avisynth.dll's fine but now having a problem. Any ideas?

MP_Pipeline("""###
### dll: "F:\avisynth+.dll"
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

MP_Pipeline: Unable to create slave process. Message: Unable to load "F:\avisynt
h+.dll", code = 126


Try this:

MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

MP_Pipeline: Unable to create slave process. Message: Unable to load "F:\avisynt
h+.dll", code = 126


Note that we shouldn't put quotes around the file name.

[turbojet]

Thanks that's what I initially tried but was getting confused by avsmeter's avs version. version() inside the mp block confirms it's working.

[Groucho2004]

Thanks that's what I initially tried but was getting confused by avsmeter's avs version. version() inside the mp block confirms it's working.
Can you elaborate on that? What was confusing?

[turbojet]

AVSmeter reports the system's avisynth dll version rather than what's in the mp-pipeline block. Some examples, with avisynth 2.6 system dll:

version()
MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

#version() shows avisynth 2.6

MP_Pipeline("""###
### dll: F:\avisynth+.dll
version()
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

#version() shows avisynth+

MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")
version()

#version() shows avisynth 2.6

I'm not sure avsmeter could or should do anything about it, really up to you.

[Groucho2004]

AVSmeter reports the system's avisynth dll version rather than what's in the mp-pipeline block.
The alternate avisynth dll is loaded by an external process of which any application that loads the script is not and cannot be aware.
So, determining the version of that avisynth.dll can only happen on the script level unless MP_Pipeline exports a function for that purpose.

[SAPikachu]

The alternate avisynth dll is loaded by an external process of which any application that loads the script is not and cannot be aware.
So, determining the version of that avisynth.dll can only happen on the script level unless MP_Pipeline exports a function for that purpose.

It will take non-trivial effort to (cleanly) implement this function, so I think I won't work on it just for displaying the version..

[real.finder]

AVSmeter reports the system's avisynth dll version rather than what's in the mp-pipeline block. Some examples, with avisynth 2.6 system dll:

version()
MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

#version() shows avisynth 2.6

MP_Pipeline("""###
### dll: F:\avisynth+.dll
version()
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

#version() shows avisynth+

MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")
version()

#version() shows avisynth 2.6

I'm not sure avsmeter could or should do anything about it, really up to you.

If you want, you can put avs dll with avsmeter in the same folder

this method work in any thing load avs script, like x264

[Groucho2004]

If you want, you can put avs dll with avsmeter in the same folder

this method work in any thing load avs script, like x264
That statement is not true. If you load avisynth.dll like this:
LoadLibrary("z:\\whateverpath\\avisynth.dll")
it will load that specific DLL.

However, most applications will load avisynth.dll without specifying a path and in that case the search order would be:

The directory from which the application loaded.
The current directory.
The System32/SysWoW64 directory.
The Windows directory.
The directories that are listed in the PATH environment variable.

[aldix]

### dll: doesn't seem to work with modified Avisynth for MT.dll

It gives

Unable to create slave process. Unable to load ... avisynth.dll, code=126

Any help or it's hopeless to try?

[SAPikachu]

### dll: doesn't seem to work with modified Avisynth for MT.dll

It gives

Unable to create slave process. Unable to load ... avisynth.dll, code=126

Any help or it's hopeless to try?

Code 126 is ERROR_MOD_NOT_FOUND, maybe try using full path to the DLL? If it doesn't work either, can you post your full script here for me to check?

[aldix]

Oops, sorry for the delayed reply. Recalled about this just now ;)

I did use the full path. It wasn't a "system path," though, just from a folder from desktop. But that shouldn't matter, right?

All the other avisynth versions work fine.

Relevant script snip (retracted):


loadplugin("C:\...\MP_Pipeline.dll")
MP_Pipeline("""

### platform: win32


loadplugin("C:\....\ffms2.dll")
ffvideosource("C:\...\video.mkv").ConvertToYV12()

Crop(0,140,0,-140)
### export clip: last
### prefetch: 96, 64
### ###

...
loadplugin calls/imports
...

last.nnedi3_resize16(1280,532,noring=true,threads=4,lsb=true).ditherpost()


### dll: C:\...\Desktop\MT_07\avisynth.dll

last.denoise()

last.sharpen()

...

[SAPikachu]

Oops, sorry for the delayed reply. Recalled about this just now ;)

I did use the full path. It wasn't a "system path," though, just from a folder from desktop. But that shouldn't matter, right?

All the other avisynth versions work fine.

Relevant script snip (retracted):


loadplugin("C:\...\MP_Pipeline.dll")
MP_Pipeline("""

### platform: win32


loadplugin("C:\....\ffms2.dll")
ffvideosource("C:\...\video.mkv").ConvertToYV12()

Crop(0,140,0,-140)
### export clip: last
### prefetch: 96, 64
### ###

...
loadplugin calls/imports
...

last.nnedi3_resize16(1280,532,noring=true,threads=4,lsb=true).ditherpost()


### dll: C:\...\Desktop\MT_07\avisynth.dll

last.denoise()

last.sharpen()

...


Does your dll path contain non-ASCII character? This may be a problem.

If this is not the case, can you upload the problematic version of avisynth.dll and another one that worked correctly to somewhere and send me the link? I suspect it is due to a dependency problem.

[aldix]

Hiya and merry X-mas :)

Could the case be that dll path contains spaces? I.e. if a path is \user\bla bla bla\else, it might choke?

If that's not it, sure, I'll post them somewhere. Just let me know.

I hope I won't forget about this thread again, sorry about this...

[SAPikachu]

Hiya and merry X-mas :)

Could the case be that dll path contains spaces? I.e. if a path is \user\bla bla bla\else, it might choke?

If that's not it, sure, I'll post them somewhere. Just let me know.

I hope I won't forget about this thread again, sorry about this...

Merry Christmas (late) and Happy New Year (early). :)

In theory space shouldn't matter, but to be certain can you also try a path without space?

[aldix]

Happy rest of the running year to you too :)

Sadly, there's no difference. There is still the error code = 126

Avisynth.dll for MT.dll (they're packed together, as per here: http://xhmikosr.1f0.de/_old/avisynth/plugins/ - the original doom9 link in the relevant thread is defunct, but it's the modded 2.5.something version which still supports MT.dll)
http://s000.tinyupload.com/?file_id=86671118413725137164

As for other DLLs that worked, here's 2.6 MT:
http://s000.tinyupload.com/?file_id=18443819082695553005

I'll be very intrigued as to what you'll be able to figure out :)


Thanks!

[SAPikachu]

This DLL loaded on my system (although I have to rename my plugins folder first, seems there are some plugins that break in this version.). This is likely a VC runtime problem, try installing this: http://thehotfixshare.net/board/index.php?autocom=downloads&showfile=10061

[aldix]

Thanks for the suggestion.

However, both in trying to install the upgrade and the thing itself (I presume), i.e. Microsoft Visual Studio .NET 2003 (http://www.microsoft.com/en-us/download/confirmation.aspx?id=703),
it fails with a message of the program required being not installed.

I'm on a Windows Server 2008 (I *think*). Might that be the problem to begin with?

[SAPikachu]

OK, then maybe we need to do a manual installation. You can download the 2 core DLL from https://www.dropbox.com/sh/fxxttnvcrtkzueg/AADS-NBg8-QGHUnA3UxvxwmSa?dl=0 (copied from my system), copy them to C:\Windows\SysWOW64 and try the script again?

[aldix]

Sorry to say, but exact same problem remains, Error 126 :(

[SAPikachu]

Sorry to say, but exact same problem remains, Error 126 :(

Well, I am out of guess now.. Maybe we have to use (one of) the last hand, can you use Process Monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645) to capture a log during loading of the script and send me the log? Hope the log can reveal the cause for us..

[Groucho2004]

If I understand this correctly, an explicitly loaded avisynth.dll will still use the plugins from the directory that is referenced in the registry, right?

If so, I suggest the following:
- Rename the avisynth.dll in SysWOW64 to avisynth.dl (for example)
- Copy the 2.5.7 DLL to SysWOW64

- Run this (http://forum.doom9.org/showthread.php?t=170647) tool
1. File -> Save info
2. Tools -> Plugin Info -> Save plugin info report
3. Post both logs and any error messages you might get.

[real.finder]

Sorry to say, but exact same problem remains, Error 126 :(

http://forum.doom9.org/showpost.php?p=1657662&postcount=136

[mohamedh]

Hi,

At first I would like to thank you for this great tool

I'm not sure if I did really understand how it works
so here's my scripte before mp-pipeline


MPEG2Source("C:\Users\bassiouny\Desktop\bleach disk1\VIDEO_TS\VIDEO_TS.d2v").ThreadRequest()
trim(14580,55640)#=41060
trim(2699,0)
import(AviSynthPluginsDir + "AnimeIVTC 2 mod.avs")
AnimeIVTC(mode=1, omode=1, credconv="mocomp", nnedi3pel=true, e1=36123, i1=38361)
vid=last
op=Import("C:\Users\bassiouny\Desktop\op.avs")

op+vid


GradFun3(thr=0.28) #or any debandning filter
import("C:\Users\bassiouny\Desktop\fansubs\AvsPmod\plugins\DeHaloH.avsi")
DeHaloHmod(radius=4)
vmToon()
mcDAA3()

Crop(2, 0, -2, -0) #not imporatant
spline36resize(720,480)



And in order to increase the encoding speed I'm using Mp-pipeline like this:

MP_Pipeline("""

### platform: win64
SetMemoryMax(2548)
setMTmode(2)
MPEG2Source("C:\Users\bassiouny\Desktop\bleach disk1\VIDEO_TS\VIDEO_TS.d2v").ThreadRequest()
trim(14580,55640)#=41060
trim(2699,0)
### ###
### platform: win32
SetMemoryMax(948)
import(AviSynthPluginsDir + "AnimeIVTC 2 mod.avs")
setMTmode(2)
AnimeIVTC(mode=1, omode=1, credconv="mocomp", nnedi3pel=true, e1=36123, i1=38361)
vid=last
op=Import("C:\Users\bassiouny\Desktop\op.avs")

op+vid

SoraThread()

GradFun3(thr=0.28) #or any debandning filter
import("C:\Users\bassiouny\Desktop\fansubs\AvsPmod\plugins\DeHaloH.avsi")
DeHaloHmod(radius=4)
vmToon()
mcDAA3()
### ###

### platform: win64
setMTmode(2)
Crop(2, 0, -2, -0) #not imporatant
spline36resize(720,480)
### lock threads to cores

### ###
""")


and here's the op.avs that I called in my script

MPEG2Source("C:\Users\bassiouny\Desktop\bleach disk1\VIDEO_TS\VIDEO_TS.d2v").ThreadRequest()
trim(14580,55640)

trim(0,2698)
import(AviSynthPluginsDir + "AnimeIVTC 2 mod.avs")
AnimeIVTC(4, omode=1)


does my script contain mistakes? can someone suggest some improvements or tips ?

And thanks in advance

P.S: I'm using a 32 GB of ram, quad core processor, 64bit avisynth and the 64 bit X264 encoder

[Dogway]

I just came to say thanks for this plugin. The other day I was encoding while I had to do some Photoshop work, it looks like I got out of RAM (got 8Gb though) and quite surprisingly MP_Pipeline paused the encoding and warned that I had to free up some memory. It really shows some professionalism, the only thing I miss for this is better documentation, I always feel like dealing with some kind of voodoo magic when setting it up.

Again thank you.

[SAPikachu]

Hi,

At first I would like to thank you for this great tool

I'm not sure if I did really understand how it works
so here's my scripte before mp-pipeline


MPEG2Source("C:\Users\bassiouny\Desktop\bleach disk1\VIDEO_TS\VIDEO_TS.d2v").ThreadRequest()
trim(14580,55640)#=41060
trim(2699,0)
import(AviSynthPluginsDir + "AnimeIVTC 2 mod.avs")
AnimeIVTC(mode=1, omode=1, credconv="mocomp", nnedi3pel=true, e1=36123, i1=38361)
vid=last
op=Import("C:\Users\bassiouny\Desktop\op.avs")

op+vid


GradFun3(thr=0.28) #or any debandning filter
import("C:\Users\bassiouny\Desktop\fansubs\AvsPmod\plugins\DeHaloH.avsi")
DeHaloHmod(radius=4)
vmToon()
mcDAA3()

Crop(2, 0, -2, -0) #not imporatant
spline36resize(720,480)



And in order to increase the encoding speed I'm using Mp-pipeline like this:

MP_Pipeline("""

### platform: win64
SetMemoryMax(2548)
setMTmode(2)
MPEG2Source("C:\Users\bassiouny\Desktop\bleach disk1\VIDEO_TS\VIDEO_TS.d2v").ThreadRequest()
trim(14580,55640)#=41060
trim(2699,0)
### ###
### platform: win32
SetMemoryMax(948)
import(AviSynthPluginsDir + "AnimeIVTC 2 mod.avs")
setMTmode(2)
AnimeIVTC(mode=1, omode=1, credconv="mocomp", nnedi3pel=true, e1=36123, i1=38361)
vid=last
op=Import("C:\Users\bassiouny\Desktop\op.avs")

op+vid

SoraThread()

GradFun3(thr=0.28) #or any debandning filter
import("C:\Users\bassiouny\Desktop\fansubs\AvsPmod\plugins\DeHaloH.avsi")
DeHaloHmod(radius=4)
vmToon()
mcDAA3()
### ###

### platform: win64
setMTmode(2)
Crop(2, 0, -2, -0) #not imporatant
spline36resize(720,480)
### lock threads to cores

### ###
""")


and here's the op.avs that I called in my script

MPEG2Source("C:\Users\bassiouny\Desktop\bleach disk1\VIDEO_TS\VIDEO_TS.d2v").ThreadRequest()
trim(14580,55640)

trim(0,2698)
import(AviSynthPluginsDir + "AnimeIVTC 2 mod.avs")
AnimeIVTC(4, omode=1)


does my script contain mistakes? can someone suggest some improvements or tips ?

And thanks in advance

P.S: I'm using a 32 GB of ram, quad core processor, 64bit avisynth and the 64 bit X264 encoder

Here are some tips:
1. You may try setting memory limit of all block to around 2.5G, especially the one with AnimeIVTC, since that's a quite complex filter.
2. Use ### prefetch in each block. 32, 16 is a good start, but you may want to tweak the values for better performance.
3. Not quite sure how setmtmode/ThreadRequest/SoraThread work in MPP but I think I will remove them and use more blocks and prefetch instead, since that will be more stable, and maybe even faster.

[SAPikachu]

I just came to say thanks for this plugin. The other day I was encoding while I had to do some Photoshop work, it looks like I got out of RAM (got 8Gb though) and quite surprisingly MP_Pipeline paused the encoding and warned that I had to free up some memory. It really shows some professionalism, the only thing I miss for this is better documentation, I always feel like dealing with some kind of voodoo magic when setting it up.

Again thank you.

Well.. if I am not mistaken this is actually message from Windows, I don't remember that I coded this kind of function into MPP. :)

Regarding documentation, I am really not good at that..

[surgical]

Greetings to all:
First of all, thank you and congratulations for this tool.
It's really useful, and has relieved me of some problems such as crashes and hangs due to some demanding MT scripts, going to encode to x264 in Megui.
My problem, for which I ask your assistance is due to a script, more complex than usual, that I'm unable to transcribe MP pipeline properly.
Setmemorymax(1024)
SetMTMode(3,6)
LoadPlugin("C:\CODIFICACION VIDEO BD\MeGUI 2525\tools\dgindexnv\DGDecodeNV.dll")
DGSource("E:\CARPETA CODIFICACION\*******.dgi")
SelectRangeEvery(10000,500)

SetMTMode(2,0)
l_width = last.width
l_height = last.height
sss=1.5
dispWidth = round(sss*l_width/8)*8
dispHeight = round(sss*l_height/8)*8
mWidth = float(l_width)
mHeight = float(l_height)
ratio = (mWidth/mHeight)
newHeight= round((dispWidth/ratio)/8)*8
newHeight > dispHeight ? Eval("""
newHeight=dispHeight
newWidth=round((newHeight*ratio)/8)*8
""" ) : Eval("""
newWidth=dispWidth
""" )
LanczosResize(newWidth,newHeight)

Repair(Gaussresize(newWidth,newHeight,p=100),1)
source=last
s=60
w=12
w=w*(128/s)
subtract(source.binomialBlur(varY=0.4515, varC=0, Y=3, U=2, V=2, useMMX=true),source)
levels(0+s,1,255-s,0,255)
levels(127-w,1,128+w,127-w,128+w)
subtract(source,last)

LimitedSharpenFasterHC(strength=80)

UnsharpHQ(THRESHOLD=80,SHARPSTR=2.6,SMOOTH=0.0, SHOW=false)
aWarpSharp2(thresh=100, blur=2, type=0, depth=16, chroma=4)

source.mt_makediff(mt_makediff(source.binomialBlur(varY=0.0001, varC=0, Y=3, U=2, V=2, useMMX=true),last),U=2,V=2)

SMDegrain(tr=2,thSAD=180,prefilter=2,contrasharp=30,refinemotion=true,lsb=true,chroma=false,plane=0)
GradFun3()
FastLineDarkenMod()
LinearResize(1920,1080, kernel="spline36", mode=0)

As much as I look at the readme.avs and this thread, performing multiple tests, I didn't do it correctly.
The first problem, and perhaps more importantly, is with a section of the script, which uses a similar syntax MP own code, and that causes problems like "expected to, or)".
The rest, I guess I would find out where and how it should be used, as appropriate, the correct syntax (splitts, export clip, etc ....) to adapt in the best possible way to use in MP; or if I had to even make changes to the syntax of the script itself for adaptation to MP
In normal use, encoding it in Megui, only I managed to make it run more or less stable at 0.5 FPS on my i7 5930x with 16 GB RAM
It is certainly not my intention seems comfortable and simply wait to be others who do the work. I want to learn the proper use of this tool but with this script I'm totally lost.
Thank you all in advance

[colours]

>binomialBlur(varY=0.4515, varC=0, Y=3, U=2, V=2, useMMX=true)
>binomialBlur(varY=0.0001, varC=0, Y=3, U=2, V=2, useMMX=true)

These probably don't do what you think they do. Unless you think they're supposed to give you black frames instead of blurring, in which case, yes, they do exactly what you think they do. The variance parameters get rounded down to the nearest multiple multiple of 0.5, and var*=0 has special treatment. Either way, this is useless.

Heck, I might as well clean up the whole script while I'm at it. All comments and significant changes are blue.

DGSource("E:\CARPETA CODIFICACION\*******.dgi")
SelectRangeEvery(10000,500)

sss = 1.5
newWidth = round(sss*width(last)/8)*8
newHeight = round(sss*height(last)/8)*8
LanczosResize(newWidth,newHeight)

source = last
s = 60
w = 24
mt_makediff(source.RemoveGrain(11),source)
mt_lut("x 128 - "+string(128.0/(128.0-s))+" * 128 + "+string(128-w)+" max "+string(128+w)+" min")
mt_makediff(source,last,u=2,v=2)

LimitedSharpenFasterHC(strength=80)

UnsharpHQ(THRESHOLD=80,SHARPSTR=2.6,SMOOTH=0.0, SHOW=false)
aWarpSharp2(thresh=100, blur=2, type=0, depth=16, chroma=4)
# do you really need four sharpening filters in a row?

last.MergeChroma(source)

FastLineDarkenMod() # moved this before denoising/debanding, but is this even necessary?

Spline36Resize(1920,1080)
# don't do the denoising/debanding with supersampling because
# that doesn't help with quality and makes things slower

SMDegrain(tr=2,thSAD=180,prefilter=2,contrasharp=30,refinemotion=true,lsb_in=false,lsb_out=true,chroma=false,plane=0)
GradFun3(lsb_in=true)

As usual, I didn't bother testing this, so while it should be correct, I have no idea whether it actually is.

With all the cruft cleaned up, it should be fairly clear how to use MP_Pipeline with it. Oh, and the reason you were seeing syntax errors is most likely that you didn't appropriately escape the triple quotes used with Eval in the original script.

[surgical]

Thanks, colours:
The truth is that I'm aware that this script needs to be optimized (in this also I've to learn a lot), but I wanted to reflect as it had it because, mainly, I was interested to learn how to use MP appropriately with a complex script that was valid, although this was not optimized, you know , like a learning script.
As for the quotes of the VAL function, I suppose, if I'm not mistaken, they must be written without spaces; I don't know if there will be any impropriety in the syntax; anyway, the script works.
Anyway I appreciate very much the ideas and corrections that you've reflected in the script; any other that you think will be welcome. I would like to see as well optimized by one skilled as you, but I don't think that this is the right thread for this topic .

[BakaProxy]

I seem to have an issue as well.

MP_Pipeline("""
LWLibavVideoSource("C:\Users\Yours Truly\Desktop\avs\00002.m2ts")
tfm(pp=0)

### prefetch: 32, 8
### ###
""")

http://i.imgur.com/gn4NxNs.png

http://i.imgur.com/n7TGY1q.png

Problem signature:
Problem Event Name: APPCRASH
Application Name: MP_Pipeline.dll.slave.exe
Application Version: 0.0.0.0
Application Timestamp: 52fde7be
Fault Module Name: KERNELBASE.dll
Fault Module Version: 6.1.7601.18933
Fault Module Timestamp: 55a69ec4
Exception Code: e06d7363
Exception Offset: 0000c42d
OS Version: 6.1.7601.2.1.0.256.1
Locale ID: 1033
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

[BakaProxy]

nevermind, putting the plugin in "plugins+" instead of plugin fixed it somehow.

using avisynth+.

[real.finder]

hi SAPikachu

setmtmode(2,12)
ColorBars(width=720, height=480, pixel_type="RGB32")
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()

cpu 75% 34 fps


MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### ###
setmtmode(2,12)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
""")


cpu 10% 7.5 fps


MP_Pipeline("""
### platform: win32
setmtmode(2,12)
ColorBars(width=720, height=480, pixel_type="RGB32")
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
distributor()
### ###
""")


cpu 95% 25 fps


MP_Pipeline("""
### platform: win32
setmtmode(2,12)
ColorBars(width=720, height=480, pixel_type="RGB32")
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()
### ###
""")


cpu 78% 38 fps


setmtmode(2,12)
MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### ###
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
""")


https://i.imgur.com/O05E3EZ.png
and if it work it will be very slow


MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### ###
setmtmode(2,12)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()
""")


and


ColorBars(width=720, height=480, pixel_type="RGB32")
setmtmode(2,12)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()


crash

seems that avs mt need setmtmode in the beginning of script!

and seems mp_pipeline has a hiding call in the beginning of every block except the first one

so I think we need something like "### setmtmode" to use it between script blocks

[SAPikachu]

hi SAPikachu

setmtmode(2,12)
ColorBars(width=720, height=480, pixel_type="RGB32")
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()

cpu 75% 34 fps


MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### ###
setmtmode(2,12)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
""")


cpu 10% 7.5 fps


MP_Pipeline("""
### platform: win32
setmtmode(2,12)
ColorBars(width=720, height=480, pixel_type="RGB32")
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
distributor()
### ###
""")


cpu 95% 25 fps


MP_Pipeline("""
### platform: win32
setmtmode(2,12)
ColorBars(width=720, height=480, pixel_type="RGB32")
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()
### ###
""")


cpu 78% 38 fps


setmtmode(2,12)
MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### ###
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
""")


https://i.imgur.com/O05E3EZ.png
and if it work it will be very slow


MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### ###
setmtmode(2,12)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()
""")


and


ColorBars(width=720, height=480, pixel_type="RGB32")
setmtmode(2,12)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()


crash

seems that avs mt need setmtmode in the beginning of script!

and seems mp_pipeline has a hiding call in the beginning of every block except the first one

so I think we need something like "### setmtmode" to use it between script blocks

SetMTMode outside MPP affects host AVS process and MPP itself, as you can see, generally they are not compatible and will crash. I don't really recommend using SetMTMode, even inside child processes, but if you want, you can either put SetMTMode inside every script block, or use ### inherit start/end to include it automatically.

[real.finder]

SetMTMode outside MPP affects host AVS process and MPP itself, as you can see, generally they are not compatible and will crash. I don't really recommend using SetMTMode, even inside child processes, but if you want, you can either put SetMTMode inside every script block, or use ### inherit start/end to include it automatically.


MP_Pipeline("""
### platform: win32
### inherit start ###
setmtmode(3,12)
### inherit end ###
ColorBars(width=720, height=480, pixel_type="RGB32")
setmtmode(1)
distributor()
### ###
setmtmode(2)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
""")


7 fps


MP_Pipeline("""
### platform: win32
### inherit start ###
setmtmode(3,12)
### inherit end ###
ColorBars(width=720, height=480, pixel_type="RGB32")
setmtmode(1)
distributor()
### ###
setmtmode(2)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()
### ###
""")


and


MP_Pipeline("""
### platform: win32
### inherit start ###
setmtmode(3,12)
### inherit end ###
ColorBars(width=720, height=480, pixel_type="RGB32")
setmtmode(1)
distributor()
### ###
setmtmode(2)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
setmtmode(1)
distributor()
### ###
""")
gradfun3()


crash

My suggestion is to add something like "### setmtmode: x, y" to add setmtmode(x,y) in the top of script block and setmtmode(1) then distributor() in the bottom of block

and with this we can use mt in certain blocks only

thanks :)

[SAPikachu]

My suggestion is to add something like "### setmtmode: x, y" to add setmtmode(x,y) in the top of script block and setmtmode(1) then distributor() in the bottom of block

and with this we can use mt in certain blocks only

thanks :)

That's only one line less than current.. You can already do this by manually adding these lines to individual blocks.

[real.finder]

That's only one line less than current.. You can already do this by manually adding these lines to individual blocks.

no, I can't, I will get crash as I mention before, even with using ### inherit start/end, because the setmtmode will be after "MPP_GetUpstreamClip()"

I can do this in first block only because it doesn't have MPP_GetUpstreamClip()

[SAPikachu]

no, I can't, I will get crash as I mention before, even with using ### inherit start/end, because the setmtmode will be after "MPP_PrepareDownstreamClip()"

I can use do this in first block only because it doesn't have MPP_PrepareDownstreamClip()

MPP_PrepareDownstreamClip is used to pull video from upstream process, it is not thread-safe IIRC, so even you can place SetMTMode before that, it is unlikely to work.

EDIT: Sorry, my comment above is incorrect, I didn't pay attention while writing that.. But anyways, using SetMTMode inside MPP is not supported, it causes many issues and I don't want to add code to support it.

[real.finder]

MPP_PrepareDownstreamClip is used to pull video from upstream process, it is not thread-safe IIRC, so even you can place SetMTMode before that, it is unlikely to work.

even if it not thread-safe it should work at lest with mode 6 or 5 in setmtmode

and you can make it thread-safe alone for that, right?

[real.finder]

EDIT: Sorry, my comment above is incorrect, I didn't pay attention while writing that.. But anyways, using SetMTMode inside MPP is not supported, it causes many issues and I don't want to add code to support it.

I make a SetMTMode.avsi and putted setmtmode(2) in it and put SetMTMode.avsi in autoload folder and load


MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### ###
### platform: win32
setmtmode(2)
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
distributor()
### ###
""")


in x264 64bit and everything ok, cpu 75% and fps 38

now I installed VS 2012 and successful compile MP_Pipeline and tried to add ### top start/end but my knowledge in C++ zero as I not a programmer in first place

the main problem is I don't now where the top of block

after some tried I could understand a little how MP_Pipeline work

the block start with MPP_SharedMemoryClient() ((plugin)) and MPP_GetUpstreamClip() ((script I think))

[real.finder]

ok, I did it in a roundabout way

http://www.mediafire.com/download/kvy2wy6k15uj0k1/MP_Pipeline+0.18.1.rar

edit: build with xp support

or with vc10 and no static http://www.mediafire.com/file/wj1v0ydkfhirl5e/MP_Pipeline_0.18.1_vc10_no_static.rar

http://www.mediafire.com/download/3szchn46e0ul6cy/MP_Pipeline-0.18.1+src.rar

test script


MP_Pipeline("""
### platform: win32
ColorBars(width=720, height=480, pixel_type="RGB32")
### setmtmode: 2,0
# setmtmode here for the next block
### ###
### platform: win32
ConvertToYV12()
QTGMC(InputType=1, Ezdenoise=2)
LSFMod(strength=66)
gradfun3()
setmtmode(1)
# enable MT
distributor()
### ###
""")


edit: new one https://github.com/realfinder/MP_Pipeline/releases

[real.finder]

edit: see here https://github.com/SAPikachu/MP_Pipeline/issues/1

[vinnytx]

Hi,

I use this script to resize movies during playback
Import("C:\Program Files (x86)\AviSynth\plugins\dither.avsi")
Import("C:\Program Files (x86)\AviSynth\plugins\mt_xxpand_multi.avsi")
Import("C:\Program Files (x86)\AviSynth\plugins\nnedi3_resize16_v3.3.avsi")
LoadPlugin("C:\Program Files (x86)\AviSynth\plugins\nnedi3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth\plugins\dither.dll")
SetMemoryMax(512)
SetMTMode(3,8)
ffdshow_source()
SetMTMode(3)
dispWidth = 1920
dispHeight = 1080
mWidth = float(last.width)
mHeight = float(last.height)
ratio = (mWidth/mHeight)
newHeight= round((dispWidth/ratio)/2)*2
newHeight > dispHeight ? Eval("""
newHeight=dispHeight
newWidth=round((newHeight*ratio)/2)*2
""") : Eval("""
newWidth=dispWidth
""")
nnedi3_resize16(newWidth,newHeight)
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last

It has a huge cpu usage
Please, someone can tell me how speed up my script with this plugin?

[vinnytx]

I installed Avisynth+ and I tried to simplify my script thanks to a previous post of colours

LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins+\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
sss = 1.5
newWidth = round(sss*width(last)/8)*8
newHeight = round(sss*height(last)/8)*8
LanczosResize(newWidth,newHeight)
distributor()
### ###
""")


But now I have this error message
MP_Pipeline: Unable to create slave process. Message: Script error: Script error: Invalid arguments to function 'width'. (ffdshow_filter_avisynth_script, line 10)

[Groucho2004]

I installed Avisynth+ and I tried to simplify my script thanks to a previous post of colours

LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins+\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
sss = 1.5
newWidth = round(sss*width(last)/8)*8
newHeight = round(sss*height(last)/8)*8
LanczosResize(newWidth,newHeight)
distributor()
### ###
""")


But now I have this error message
MP_Pipeline: Unable to create slave process. Message: Script error: Script error: Invalid arguments to function 'width'. (ffdshow_filter_avisynth_script, line 10)
1. There is no source filter in your script and therefore the "last" variable is undefined
2. Avisynth+ does not have a function "distributor()"

[vinnytx]

This script works well
ffdshow_source()
SetMemoryMax(512)
sss = 1.5
newWidth = round(sss*width(last)/8)*8
newHeight = round(sss*height(last)/8)*8
LanczosResize(newWidth,newHeight)


This doesn't work

LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins+\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
ffdshow_source()
SetMemoryMax(512)
sss = 1.5
newWidth = round(sss*width(last)/8)*8
newHeight = round(sss*height(last)/8)*8
LanczosResize(newWidth,newHeight)
### ###
""")

I have this error message
MP_Pipeline: Unable to create slave process. Message: Script error: Script error: There is no function named 'ffdshow_source'. (ffdshow_filter_avisynth_script, line 10)

[StainlessS]

No idea, but try eg


LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins+\MP_Pipeline.dll")
ffdshow_source() # NOT INSIDE MP_PipeLine script
MP_Pipeline("""
### platform: win32
SetMemoryMax(512)
sss = 1.5
newWidth = round(sss*width(last)/8)*8
newHeight = round(sss*height(last)/8)*8
LanczosResize(newWidth,newHeight)
### ###
""")

[vinnytx]

Ok, now I have another error message
MP_Pipeline: Unable to create slave process. Message: Script error: Script error: Invalid arguments to function 'width' (ffdshow_filter_avisynth_script, line 11)

[Groucho2004]

Ok, now I have another error message
MP_Pipeline: Unable to create slave process. Message: Script error: Script error: Invalid arguments to function 'width' (ffdshow_filter_avisynth_script, line 11)
The source filter statement has to be within the "MP_Pipeline" construct. This combination may not work at all since ffdshow_source() is not a conventional source filter. My knowledge of Avisynth scripts in ffdshow is very limited though...

[DJATOM]

No idea, but try eg


LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins+\MP_Pipeline.dll")
ffdshow_source() # NOT INSIDE MP_PipeLine script
MP_Pipeline("""
### platform: win32
SetMemoryMax(512)
sss = 1.5
newWidth = round(sss*width(last)/8)*8
newHeight = round(sss*height(last)/8)*8
LanczosResize(newWidth,newHeight)
### ###
""")


As I know, MP_Pipeline can't pass clips that way. So it will not work.

[vinnytx]

As I know, MP_Pipeline can't pass clips that way. So it will not work.

Any advice?

[Stereodude]

I know jack about ffdshow_source. Is it a built in function of AVIsynth/AVIsynth+ otherwise don't you need to load a plugin? Is it auto-loading for 64-bit, but not 32-bit?

[StainlessS]

Never used it (ffdshow_source), but see here:- http://ffdshow-tryout.sourceforge.net/wiki/video:avisynth

Is installed by ffdshow-tryout.

EDIT: you have to select tick box on install, see below
https://s20.postimg.cc/3tnzz7zi5/ffdshow-tryouts_zpsh2dtglea.jpg (https://postimg.cc/image/qv4l4yz5l/)

EDIT: I dont currently have ffdshow installed, but I think it may install the dll into it's own directory,
perhaps a copy into autoload plugins dir would work for general usage.

[vinnytx]

ffavisynth.avsi
ffavisynth.dll

They are already in the Avisynth+ plugins+ directory

[Chyrka]

Hello.
Is it possible to use MP_Pipeline and ScriptClip inside it?
They both started from """ tripple quote marks. ScriptClip does not want to work with a single quotation mark as it is stated in the documentation. :(

[real.finder]

Hello.
Is it possible to use MP_Pipeline and ScriptClip inside it?
They both started from """ tripple quote marks. ScriptClip does not want to work with a single quotation mark as it is stated in the documentation. :(

made a simple function like this

function ScriptClip_lines(clip c) {
c
*ScriptClip here*
}


and put it in .avs and import that .avs inside MP_Pipeline
then put the "ScriptClip_lines()" line in the place you want

[Chyrka]

real.finder, that hack must work. Thanks.

[bxyhxyh]

Hello, I wanted to speed up xsharpen with this. With MT version, xsharpen goes at nice 2x-4x speed. But sometimes it gives me corrupted frame.

Can I call it like this to prevent from denoising being inside of it?


mp_pipeline("""
Import("C:\Users\BallGNM\Desktop\denoised.avs")
s16=dither_resize16(6840,4860)
s16.ditherpost(mode=-1).xsharpen(strength=255,threshold=255).Dither_convert_8_to_16()
s16.Dither_limit_dif16 (last, thr=0.7, elast=2.0)
dither_resize16(1120,840).gradfun3(smode=2,lsb_in=true,lsb=false,mode=6,thr=0.5)
### branch 2
### ###
""")
dehalo_alpha_mt()

[tuanden0]

Can someone help me? I'm using avs+.

MP_Pipeline's great, my speed increase 30 ~ 45% and encode time decrease 30% with my script:

MP_Pipeline("""
### platform: win64
LWLibavVideoSource("E:\Test Zone\test.mkv")
AssumeFPS(24000, 1001)
FFT3DFilter(sigma=1.7, bt=1, ncpu=4)
flash3kyuu_deband()
Prefetch(4)
### ###
### platform: win32
Toon(0.25)
TextSub("E:\Test Zone\test.ass", 1)
### ###
""")

Then, I tried this script and my encode time decrease 2/3 with my speed increase 50 ~ 60%. But, sometime it's crashed :confused::confused::

MP_Pipeline("""
### platform: win64
LWLibavVideoSource("E:\Test Zone\test.mkv")
AssumeFPS(24000, 1001)
FFT3DFilter(sigma=1.7, bt=1, ncpu=4)
flash3kyuu_deband()
Prefetch(4)
### ###
### platform: win32
Toon(0.25)
TextSub("E:\Test Zone\test.ass", 1)
Prefetch(4)
### ###
""")

I tried to put Prefetch(4) outside MP_Pipeline but they slower than my 1st script :(:(

[pinterf]

Can someone help me? I'm using avs+.

I tried to put Prefetch(4) outside MP_Pipeline but they slower than my 1st script :(:(
This is a known issue in avs+ since r206x, workaround in progress.

[tuanden0]

@pinterf: :thanks: for the information

[FranceBB]

I'd like to split my script into multiple process in order to avoid the 2 GB limit, 'cause it crashes.

I'm trying to split my script like this:


MP_Pipeline("""

FFVideoSource("op.mxf", fpsnum=24000, fpsden=1001)
Spline64ResizeMT(1920, 1080, threads=4, logicalCores=true, MaxPhysCore=true, SetAffinity=true)
#AmplifyDB(-12.0)
opening=last

### platform: win32
### export clip: opening
### ###

FFVideoSource("ep.mxf", fpsnum=24000, fpsden=1001)
tdeint(mode=2, order=-1, field=-1, mthreshL=6, mthreshC=6, map=0, type=2, debug=false, mtnmode=1, sharp=true, cthresh=6, blockx=16, blocky=16, chroma=true, MI=64, tryWeave=true, link=1, denoise=true, slow=2, opt=4)
Converttoyv12(interlaced=false, chromaresample="Spline64", matrix="Rec709")
opening++trim(2157, 30327)
episode=last

### platform: win32
### export clip: episode
### ###

FFVideoSource("ed.mxf", fpsnum=24000, fpsden=1001)
Spline64ResizeMT(1920, 1080, threads=4, logicalCores=true, MaxPhysCore=true, SetAffinity=true)
#AmplifyDB(-14.0)
episode++trim(0,0)
final=last

### platform: win32
### export clip: final
### ###

#Normalize(0.89, show=false)
#ResampleAudio(48000)
tweak(sat=1.53 , dither=true)

### platform: win32
### ###

dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=3, dither=0)

### platform: win32
### ###

Spline64ResizeMT(3840, 2160, threads=4, logicalCores=true, MaxPhysCore=true, SetAffinity=true)

### platform: win32
### ###

aWarpSharp2(thresh=180, blur=3, type=0, depth=35, depthC=10, chroma=4)

### platform: win32
### ###

f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, opt=3, mt=true, keep_tv_range=true, input_depth=8, output_depth=8)

### platform: win32
### ###

TextSubmod("sub.ass")

### platform: win32
### ###
""")


I'll let you know if the encode finishes successfully or not.
If it works, MPP it's gonna be really useful to my daily tasks! :D

[FranceBB]

It works, but I don't know how to handle audio now, 'cause MPP doesn't support it.

[real.finder]

It works, but I don't know how to handle audio now, 'cause MPP doesn't support it.

maybe this (http://forum.doom9.org/showthread.php?t=169961) can help you, or you must wait someone can added audio to mpp (https://github.com/SAPikachu/MP_Pipeline/issues/1#issuecomment-269108878), or made another script just for audio

[FranceBB]

@real.finder... uh... I see... I just replied on github as well. As a "workaround", I'm using audio created in a separate avisynth script, created to make a low-res version that was sent out to our subtitling team. Anyway, audio support would be a really useful feature. Unfortunately, I'm a .NET guy and my lazy ass made me forget almost everything about C++ I studied at university and embrace C# time ago. Perhaps, JPSDR / StainlessS / Groucho2004 could take a look at the code and, if interested, add audio support.

[ChaosKing]

Or you could use avisynth 64 bit to avoid the 2gb limit... as far as I can see all your filters are also available for x64.

[Stereodude]

Anyone have a working download link for .18?

[poisondeathray]

Anyone have a working download link for .18?

MP_Pipeline-0.18 mirror

http://www.mediafire.com/file/vsmamxey9llxgxc/MP_Pipeline-0.18.rar

[Stereodude]

MP_Pipeline-0.18 mirror

http://www.mediafire.com/file/vsmamxey9llxgxc/MP_Pipeline-0.18.rar
Thanks!

[Dogway]

I'm trying to load 32-bit plugins in avs+ x64 latest build 2772 without success, I put the content of MP_Pipeline 0.18 64 folder into plugins+64 and try to preview the next script with AvsPmod.v2.5.1_x64 (also tried vdub2).

Did more tests with other code explicitly loading x86 avisynth dll.
MP_Pipeline("""
### platform: win32
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins+\JpegSource.dll")
jpegSource("D:\Profiles\Usuarios\Administrador\Desktop\147.jpg",length=1)

### ###
""")
But I get an error as if I still was using 64-bit avisynth.

MP_Pipeline: Unable to create slave process. Message: Script error: Cannot load a 64 bit DLL in 32 bit Avisynth: 'C:/Program Files (x86)/AviSynth+/plugins64+/firstfoundplugin_x64.dll'.

[Dogway]

I have doing more tests and I'm still unable to mix 32 and 64 bit plugins.
Here's my clean scenario.
Installed avs+ r2772 both x86 and x64, x86 through Avisynth Repository

Current registry paths:
[HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth]
@="D:\\Rip\\workshop\\Front_Ends\\AvisynthRepository\\AVSPLUS_x64"
"PluginDir2_5"="D:\\Rip\\workshop\\Front_Ends\\AvisynthRepository\\AVSPLUS_x64\\plugins"
"PluginDir+"="C:\\Program Files (x86)\\AviSynth+\\plugins64+"

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Avisynth]
@="D:\\Rip\\workshop\\Front_Ends\\AvisynthRepository\\AVSPLUS_x86"
"PluginDir2_5"="D:\\Rip\\workshop\\Front_Ends\\AvisynthRepository\\AVSPLUS_x86\\plugins"
"PluginDir+"="C:\\Program Files (x86)\\AviSynth+\\plugins+"

MP_Pipeline x64 is 318Kb v0.18 version:

plugins+ folder:
ConvertStacked.dll
DirectShowSource.dll
ImageSeq.dll
MtModes.avsi
Shibatch.dll
TimeStretch.dll
VDubFilter.dll

plugins64+ folder:
ConvertStacked.dll
DirectShowSource.dll
ImageSeq.dll
MP_Pipeline.dll
MP_Pipeline.dll.slave.exe
MP_Pipeline.dll.win32
MP_Pipeline.dll.win32.slave.exe
MP_Pipeline_readme.avs
MtModes.avsi
Shibatch.dll
SmoothAdjust.dll
TimeStretch.dll
VDubFilter.dll


This is my script loaded in AvsPmod 2.5.1 x64 from this link (https://forum.doom9.org/showthread.php?p=1801816#post1801816).
Plugin Autoload Folder in AvsPmod is set to: C:\Program Files (x86)\AviSynth+\plugins64+

LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
imageSource("D:\Profiles\Usuarios\Administrador\Desktop\147.jpg")

### ###
""")

and error:

MP_Pipeline: Unable to create slave process. Message: Script error: Cannot load a 64 bit DLL in 32 bit Avisynth: 'C:/Program Files (x86)/AviSynth+/plugins64+/ConvertStacked.dll'.

Trying to explicitly load x86 avisynth:
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
### dll: "C:\Windows\SysWOW64\AviSynth.dll"
imageSource("D:\Profiles\Usuarios\Administrador\Desktop\147.jpg")

### ###
""")

MP_Pipeline: Unable to create slave process. Message: Unable to load "C:\Windows\SysWOW64\AviSynth.dll", code = 126

[Stereodude]

The combo works find for me (r2772 and MP 0.18). What program are you trying to load the script with? It looks like you're trying to load your script with a x86 (32-bit) program. This causes AVIsynth+ to try to load your 64-bit script in the x86 version and it fails.

Have you tried loading the script with the 64-bit version of VD2 or AVSmeter?

[Dogway]

My setup is posted above. r2772 yes, MP 0.18 yes 64bit AvsPmod, VD2, avsmeter64... and so on.
I had to fix the registries because AvisynthRepository was messing with them bad time but still no luck.
Now they are:
[HKEY_LOCAL_MACHINE\SOFTWARE\AviSynth]
@="C:\\Program Files (x86)\\AviSynth+\\plugins64+"
"plugindir2_5"="C:\\Program Files (x86)\\AviSynth+\\plugins64"
"plugindir+"="C:\\Program Files (x86)\\AviSynth+\\plugins64+"

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\AviSynth]
@="C:\\Program Files (x86)\\AviSynth+\\plugins+"
"plugindir2_5"="C:\\Program Files (x86)\\AviSynth+\\plugins"
"plugindir+"="C:\\Program Files (x86)\\AviSynth+\\plugins+"
In my opinion MP_Pipeline is failing to load avisynth x86 for some reason.
It can't even load an avisynth.dll
### dll: C:\Windows\SysWOW64\AviSynth.dll
MP_Pipeline: Unable to create slave process. Message: Unable to load "C:\Windows\SysWOW64\AviSynth.dll", code = 193
By the way I'm on Win7 SP1 x64 if that makes a difference.

[Yanak]

Hello,

Not sure if it will help anything but I have both x86 and x64 version of avisynth, also on Win7 SP1 x64 , installed via the AviSynthPlus-MT-r2772.exe installer provided by pinterf here (https://github.com/pinterf/AviSynthPlus/releases)

One thing that ticks me in your script is " LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\MP_Pipeline.dll") "
Why program files x86 ?, the x64 version should be in "Program Files", the x86 version of avisynth is the one that should be in Program Files (x86),

according to your reg entries the "plugins+" & " plugins64+ " are located at the same place for both x86 and x64 version, both in Program Files (x86), seems strange, unless i'm not seeing clear tonight it looks like both your avisynth installs are locate at the same place in program files x86

I checked and i do not have any "plugins64+" folder in the x86 version at "C:\Program Files (x86)\AviSynth+", only a "plugin+" folder is present there.

My reg entries :
[HKEY_LOCAL_MACHINE\SOFTWARE\AviSynth]
@="C:\\Program Files\\AviSynth+"
"plugindir2_5"="C:\\Program Files\\AviSynth+\\plugins64"
"plugindir+"="C:\\Program Files\\AviSynth+\\plugins64+"
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\AviSynth]
@="C:\\Program Files\\AviSynth+"
"plugindir2_5"="C:\\Program Files\\AviSynth+\\plugins"
"plugindir+"="C:\\Program Files (x86)\\AviSynth+\\plugins+"

MP-pipeline files :
MP_Pipeline.dll
MP_Pipeline.dll.slave.exe
MP_Pipeline.dll.win32
MP_Pipeline.dll.win32.slave.exe
MP_Pipeline_readme.avs
All located inside the x64 version of avisynth, here : "C:\Program Files\AviSynth+\plugins64+"

My x64 version of avisynth is installed at C:\Program Files\AviSynth+\
My x86 version of avisynth is nstalled at C:\Program Files (x86)\AviSynth+\



also with avisynth+ you don't need to call the Mp_pipeline dll at the start of the script , it is auto loaded so this line should not even be needed in the script.
Tested also within the 2.5.1 x64 version of AvsPmod that is, unless i make an error, the same as yours, there again no need to use the " LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\MP_Pipeline.dll") ", only this to load a pic .

MP_Pipeline("""
### platform: win32
imageSource("F:\My Pic.png")
### ###
""")

Like i said not sure it it will help with anything, maybe you can compare and find something that is odd.

Good luck, took me a while before getting MP_pipeline to work, hopefully the forum good souls helped me at this time :)

[Dogway]

Thanks for the help. I just preferred to have all my plugins in the same location for practical purposes, I investigated if this made a difference (like not taking advantage of 64bit features) and it seems it doesn't. The location of 64-bit executables is unrelated to their performance (despite some saying otherwise). BUT it could be MP_Pipeline wasn't designed in this particular way, I should have a look at what line 193 says in the source. I use MP_Pipeline since years ago, but recently just switched to avs+ x64 and after some hurdles this issue is pretty much the only thing irking me.
I'm finishing a project so I will have a deeper look in a few days. Maybe installing though the official installer this time.

[Yanak]

I see, maybe it is not liking it this way indeed, i have not looked into the code plus I am not a coder so i don't know much more than this sadly.

The very last thing that come to my mind and might help figure out thing is in avspmod to only do this as script, without using the LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\MP_Pipeline.dll") that is not needed anymore with avs+ ( it is needed inside the win32 or win64 loops of mp_pipeline as this nice tool can't auto-load plugins sadly ), anyways :

MP_Pipeline("""
### platform: win32
Version()
### ###
""")

And after the same with win64 variant

MP_Pipeline("""
### platform: win64
Version()
### ###
""")

and see what the preview window returns you for the versions of avisynth+ it is using, this is if it managed to load anything at all with mp_pipeline.

good luck

[Groucho2004]

I had to fix the registries because AvisynthRepository was messing with them bad time but still no luck.I'm assuming you mean that the Universal Installer messed with your registry entries. That shouldn't happen if you set up the batch file correctly. Can you describe the problem in more detail?

[real.finder]

try put those in plugins64 folder not the + one

MP_Pipeline.dll
MP_Pipeline.dll.slave.exe
MP_Pipeline.dll.win32
MP_Pipeline.dll.win32.slave.exe


and those in plugins

MP_Pipeline.dll
MP_Pipeline.dll.slave.exe
MP_Pipeline.dll.win64
MP_Pipeline.dll.win64.slave.exe


and you can try my edit too https://forum.doom9.org/showthread.php?p=1746479#post1746479

and make sure that you dont use teracopy or whatever since it did some problems back then for my friend, to check that try to load avs dll using ### dll: from other location

[FranceBB]

I just wanna report that Planar RGB 16bit (RGBP16) produces a green clip. Converting it back to yuv in the same block "solves" the issue.

MP_Pipeline("""

FFVideoSource("I:\Production\RAW\test8K.mkv")

### ###

Crop(0, 0, -0, -630)

### ###

AddBorders(0, 314, 0, 314)

### ###

Spline64Resize(848, 480)

### ###

trim(1081, 5220)

### ###

ConvertBits(16)

### ###

ConvertToPlanarRGB()

### ###

Cube("C:\Programmi\AviSynth+\LUTs\void_null.cube", cpu=1, fullrange=false)

Converttoyuv420()

""")

https://i.imgur.com/qhCegup.png

MP_Pipeline("""

FFVideoSource("I:\Production\RAW\test8K.mkv")

### ###

Crop(0, 0, -0, -630)

### ###

AddBorders(0, 314, 0, 314)

### ###

Spline64Resize(848, 480)

### ###

trim(1081, 5220)

### ###

ConvertBits(16)

### ###

ConvertToPlanarRGB()

Cube("C:\Programmi\AviSynth+\LUTs\void_null.cube", cpu=1, fullrange=false)

Converttoyuv420()

### ###

""")

https://i.imgur.com/o85cRBo.png

[Stereodude]

I just wanna report that Planar RGB 16bit (RGBP16) produces a green clip. Converting it back to yuv in the same block "solves" the issue.
I've also noticed it doesn't handle passing at least some of the new AVS+ formats between "blocks" correctly. Not all of them fail in such an obvious manner though.

[FranceBB]

Any chance for someone to pickup the project, work on it, make it Avisynth+ compatible (modern high bit depth formats) and support audio?
It would be a very good thing to have it working properly.
It's been years since the last release...
Even just audio support would be a big plus.

[Myrsloik]

Any chance for someone to pickup the project, work on it, make it Avisynth+ compatible (modern high bit depth formats) and support audio?
It would be a very good thing to have it working properly.
It's been years since the last release...
Even just audio support would be a big plus.

Why is it still needed when you have 64bit binaries and threading? I'm curious...

[FranceBB]

Why is it still needed when you have 64bit binaries and threading? I'm curious...

X86 only Avisynth is still very much popular so are hybrid workflows.
It could be because of plugins that have not been ported yet or perhaps because of people having dual boot OS which includes a relatively modern x64 OS and an x86 one. I've seen many people having Windows 7 x64 and XP x86. I myself have Windows 10 Pro x64, however I still have my Win XP x86 machine up and running and having completely separate workflows (x86 and x64) would be a compatibility nightmare 'cause I know that x64 variants may be slightly different. Besides, it's still very common to switch between Avisynth versions, that's why Groucho's Universal Avisynth Installer was very welcome when he released it. In other words, having x86 filters loading inside x64 Avisynth or Avisynth x86 being able to allocate more than 2 GB of memory it's still very useful, especially in a world where high bit depth and ultra high resolutions are common. An example? Satellite feeds in FULL HD are still interlaced, but you would almost definitely run out of RAM if you use QTGMC on a x86 system. And... sure, there are other deinterlacers like tdeint, yadif, Bob() itself and so on, but they're beyond what I'm trying to say here. What I'm trying to say is that we're not living in an utopia in which everything is x64, perfectly multi-threaded, 16bit planar and we're living in a nirvana. Just look at 16 bit stacked and interleaved: years and years after the planar regular high bit depth support inside avisynth, stacked and interleaved are still very much alive due to the lack of compatibility with some older plugins that have never been ported.

Anyway, that's my opinion on why it's relevant. :)

[goorawin]

Why is it still needed when you have 64bit binaries and threading? I'm curious...
It is still anywhere from 10% to 40% quicker than Avisynth 64bit with multi treading.
It just takes sometime to set it up for your system and your script.
And yes you can still include audio outside the pipeline script, which works well.
Not ideal but better than nothing.

[real.finder]

It is still anywhere from 10% to 40% quicker than Avisynth 64bit with multi treading.
It just takes sometime to set it up for your system and your script.
And yes you can still include audio outside the pipeline script, which works well.
Not ideal but better than nothing.

audio inside is needed for cases like this https://forum.doom9.org/showthread.php?p=1671402#post1671402 (mvtools2)

aside from that adding avs+ high bit depth support should be in C api (like x264, ffmpeg and avspmod) to avoid losing older avs support since mpp can load different avs in every block by ### dll: path

[poisondeathray]

It is still anywhere from 10% to 40% quicker than Avisynth 64bit with multi treading.
It just takes sometime to set it up for your system and your script.
And yes you can still include audio outside the pipeline script, which works well.
Not ideal but better than nothing.

Depends on the script,

I've had 80-90% slower with mp_pipeline

[tormento]

Now that pinterf has released MP_Pipeline with Avisynth+ colorspace support, would somebody explain me what are the differences between MP_Pipeline and Prefetch?

[gpower2]

Now that pinterf has released MP_Pipeline with Avisynth+ colorspace support, would somebody explain me what are the differences between MP_Pipeline and Prefetch?

I think that Prefetch uses new threads inside the same AviSynth process, while MP_Pipeline creates new processes all together.

But someone with more experience on the subject could give more details.

[MeteorRain]

Assume simplest scenario:

source().deinterlace().denoise().deband()

If you run that with MT prefetch(4), you might get:

Thread1: source().deinterlace().denoise().deband() → for frames 0, 4, 8, 12...
Thread2: source().deinterlace().denoise().deband() → for frames 1, 5, 9, 13...
Thread3: source().deinterlace().denoise().deband() → for frames 2, 6, 10, 14...
Thread4: source().deinterlace().denoise().deband() → for frames 3, 7, 11, 15...

If you run that with MP and put each filter into its own process, you get:

Process1: source() ↓
Process2: deinterlace() ↓
Process3: denoise() ↓
Process4: deband() →

Multi threading has restrictions, such as (1) the filter must be thread-safe to properly function (2) the filter must be non-temporal or have internal historical data cache to run fast (3) you have enough memory to hold the historical cache.

Multi process however, doesn't require filters to be thread-safe, and does not break request order (unless you branch, which is not generally recommended).

[MeteorRain]

For "historical data" part, fortunately I was rewriting fft3dfilter and it requires historical data so I can share some experience.

In fft3dfilter, for the parameter bt = X, a total of X frames will be examined for each output frame. If bt=5, then [prev2 prev cur next next2] will be examined to produce cur_output.

The original caching mechanism simply store the last (bt+2) frames and replace the oldest frame with the newest one.

Now, if you run it in multi thread, it's possible that prefetch would fetch frame #5 #6 #7 #8 at the same time.
The filter would then fetch and process its 4 neighbors for each request.
#5 ← [3 4 6 7]
#6 ← [4 5 7 8]
#7 ← [5 6 8 9]
#8 ← [6 7 9 10]

If you simply keep the cache within threads, obviously you'll have to waste time re-computing frames. To increase the efficiency we'd use a shared caching system, where it caches a few historical data and shares with all requests.

#existing_cache = [1 2 3 4 5 6]
#5 ← [7]
#existing_cache = [2 3 4 5 6 7]
#6 ← [8]
#existing_cache = [3 4 5 6 7 8]
#7 ← [9]
#existing_cache = [4 5 6 7 8 9]
#8 ← [10]
#existing_cache = [5 6 7 8 9 10]

That works great until prefetch issues frame requests in random order:

#existing_cache = [1 2 3 4 5 6]
#8 ← [7 9 10]
#existing_cache = [5 6 7 8 9 10]
#5 ← [3 4]
#existing_cache = [9 10 3 4 5 6 7]
#7 ← [8]
#existing_cache = [4 5 6 7 8 9]
#6 ← []
#existing_cache = [9 4 5 6 7 8]

Note the cache miss for #3 #4 and #8, unless we increase the cache size to 8.

The more thread you create, the more cache space you need to prevent a cache miss. Thus increasing the memory usage (possibly by a lot).

[tormento]

At the end of the day, a simple script with SMDegrain would have no speed benefit, as everything is inside the avsi, right?

[MeteorRain]

Theoretically you can split the image into half and put them into multiple processes. I run QTGMC in 2 processes by cutting it half and let each process do half the work.

[Stereodude]

Theoretically you can split the image into half and put them into multiple processes. I run QTGMC in 2 processes by cutting it half and let each process do half the work.
I've split UHD footage into 4 ~FHD segments run them concurrently with MCTD (in 4 different MPP segments) and put them back together into a single UHD image. I made them each slightly larger than FHD to make them overlap at the segment edges for things that move through the frame and cropped them.

like:LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")

MP_Pipeline("""

### platform: win64
source=LWLibavVideoSource("UHD_source.mkv").crop(0,280,-0,-280)

source

### export clip: source
### prefetch: 16, 8
### ###
### platform: win32
SetMemoryMax(1280)

lefttop=source.crop(0,0,-1856,-736).MCTemporalDenoise(settings="low", radius=3)

lefttop

### export clip: lefttop
### pass clip: source
### prefetch: 16, 8
### ###
### platform: win32
SetMemoryMax(1280)

leftbot=source.crop(0,736,-1856,-0).MCTemporalDenoise(settings="low", radius=3)

leftbot

### export clip: leftbot
### pass clip: source, lefttop
### prefetch: 16, 8
### ###
### platform: win32
SetMemoryMax(1280)

righttop=source.crop(1856,0,-0,-736).MCTemporalDenoise(settings="low", radius=3)

righttop

### export clip: righttop
### pass clip: source, lefttop, leftbot
### prefetch: 16, 8
### ###
### platform: win32
SetMemoryMax(1280)

source.crop(1856,736,-0,-0).MCTemporalDenoise(settings="low", radius=3)

### pass clip: lefttop, leftbot, righttop
### prefetch: 16, 8
### ###
### platform: win64

rightbot=last

left=stackvertical(lefttop.crop(0,0,1920,-64),leftbot.crop(0,64,1920,-0))
right=stackvertical(righttop.crop(64,0,-0,-64),rightbot.crop(64,64,-0,-0))

stackhorizontal(left,right)

### prefetch: 20, 10
### ###

""")

MPP also can speed up heavy serial processing. Like QTGMC followed by srestore followed by MCTD. You put each one in it's own MPP segment and it is faster than when done in a conventional script.

like:LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")

MP_Pipeline("""

### platform: win64
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNV.dll")

DGSource("Video_2.dgi")


### prefetch: 16, 8
### ###
### platform: win64
SetMemoryMax(1536)

QTGMC( Preset="Slower", InputType=0, SourceMatch=3, Lossless=2, Sharpness=0.2, EZKeepGrain=0.0, ShowSettings=false )

### prefetch: 16, 8
### ###
### platform: win64

srestore()

### prefetch: 16, 8
### ###
### platform: win32
SetMemoryMax(1280)

MCTemporalDenoise(settings="low", radius=3)

### prefetch: 16, 8
### ###

""")

MPP also supports adjustable thunking (threaded chunking). You can set the chunk size (in frames) and the number of simultaneous chunks. I've rarely used this.

[MeteorRain]

I've split UHD footage into 4 ~FHD segments run them concurrently with MCTD (in 4 different MPP segments) and put them back together into a single UHD image. I made them each slightly larger than FHD to make them overlap at the segment edges for things that move through the frame and cropped them.

Yup, that's exactly how I use MCTD on my server CPU (low freq slow single core). Except that I use it on HD content, splitting images into 2 976x1080 including 16px overscan, and crop and stick them together.

[Stereodude]

Yup, that's exactly how I use MCTD on my server CPU (low freq slow single core). Except that I use it on HD content, splitting images into 2 976x1080 including 16px overscan, and crop and stick them together.
Maybe I'm too generous, but I've used 64 pixels of overlap. I've not done it much. I've subsequently gotten MCTD working in 64-bit, but I have yet not checked to see if it can handle UHD footage well or not. The 32-bit side will eventually pass a UHD image though MCTD, but it's REALLY slow.

I suspect MCTD could be heavily optimized by someone who really understood what it's doing in each of the many plugins it's using and consolidating it into a single plugin with only the essential bits and pieces, but that's so far beyond me...

It's quite slow and also stuck at 8-bit color only (and will probably always be). :( So far, I've not found a better noise reduction plugin though.

[gispos]

So far, I've not found a better noise reduction plugin though.
+1
The only filters that I still use and that are comparable are the denoise filters in Resolve 16

[tormento]

So far, I've not found a better noise reduction plugin though.
It's a long time, since I jumped to x64, that I don't give it a try. Is latest version still 1.4.20 from Lato? Are all the x64 plugins available?

[Stereodude]

It's a long time, since I jumped to x64, that I don't give it a try. Is latest version still 1.4.20 from Lato?
Yes

Are all the x64 plugins available?
More or less...

MysteryX put together this list of plugins looking at high bit depth support.
Supports high-bit-depth

- MVTools: yes
- MaskTools2: yes
- RgTools: yes
- FFT3Dfilter: yes
- AddGrainC: yes (?)
- DeBlock: yes
- DctFilter: yes

- TTempSmooth: NO
- EEDI2: NO
- SangNom: NO -- Note: VapourSynth version doesn't use SangNom
- GradFun2db: NO
Using this list I couldn't find a x64 build of Sangnom, but I found all the rest in x64. However, I haven't actually had MCTD complain about SangNom being missing. I'm not sure when it uses it, but it doesn't with the NR presets I'm using within MCTD.

[real.finder]

same story for awarpsharp and awarpsharp2 there are SangNom2 but it need to add SangNom in the same dll like awarpsharp2 https://forum.doom9.org/showthread.php?p=1655648#post1655648

[Stereodude]

same story for awarpsharp and awarpsharp2 there are SangNom2 but it need to add SangNom in the same dll like awarpsharp2 https://forum.doom9.org/showthread.php?p=1655648#post1655648
From a quick review of the script, it looks like SangNom2 could be substituted in place of SangNom in MCTD. Just change the two calls on line 1024 of the MCTD .AVSI to call SangNom2 instead.

Edit: I updated the script to use SangNom2 and posted it here (https://forum.doom9.org/showthread.php?p=1906491#post1906491).

[tormento]

MysteryX put together this list of plugins looking at high bit depth support.

To sum up x64 releases:

- MVTools: found
- MaskTools2: found
- RgTools: found
- FFT3Dfilter: found. What about replacing with Neo_FFT3D (https://forum.doom9.org/showthread.php?p=1905291#post1905291), with proper script changes?

- SangNom: Sangnom2, this (https://github.com/tp7/SangNom2/releases) the latest?
- AddGrainC: is this (https://forum.doom9.org/showthread.php?p=1655346#post1655346) the latest?
- DeBlock: this (https://github.com/mysteryx93/Avisynth-Deblock/releases)?
- DctFilter: this (https://github.com/chikuzen/DCTFilter/releases)?

- TTempSmooth: where?
- EEDI2: where?
- GradFun2db: where?

[real.finder]

even if there are MaskTools2 with x64 and HBD, scripts need update to make sure it can work with HBD, Especially the mt_lut* things

[MeteorRain]

tormento: Please allow some time before using neo_fft3d in production. Well it has been through some tests but I'm still working on it during the days.

(Good news is I managed to write a new dual synth wrapper, inspired by feisty2, and have just moved fft3d onto this new platform. Looking good so far, so I may release a new version in a few days.)

[Stereodude]

- SangNom: Sangnom2, this (https://github.com/tp7/SangNom2/releases) the latest?
- AddGrainC: is this (https://forum.doom9.org/showthread.php?p=1655346#post1655346) the latest?
- DeBlock: this (https://github.com/mysteryx93/Avisynth-Deblock/releases)?
- DctFilter: this (https://github.com/chikuzen/DCTFilter/releases)?
Sure, those will work.

- TTempSmooth: where?
- EEDI2: where?
- GradFun2db: where?
You can find them here (http://avisynth.nl/index.php/AviSynth%2B_x64_plugins).

[tormento]

tormento: Please allow some time before using neo_fft3d in production. Well it has been through some tests but I'm still working on it during the days.
I always hope someone will write a OpenCL version :o

[tormento]

You can find them
:thanks:

[tormento]

Sure, those will work.
Did you notice this (https://forum.doom9.org/showthread.php?p=1905669#post1905669)release of AddGrain?

[Stereodude]

Did you notice this (https://forum.doom9.org/showthread.php?p=1905669#post1905669)release of AddGrain?
I guess I saw it, but didn't connect it to MCTD in my head. Have you tried using it with MCTD? It seems there is some discussion if the output is correct or not.

[tormento]

I guess I saw it, but didn't connect it to MCTD in my head. Have you tried using it with MCTD? It seems there is some discussion if the output is correct or not.



Not yet. I have a long restoration process that will finish in a day or two.

[Stereodude]

MP_Pipeline doesn't work with L-SMASH-Works until the .lwi file has already been successfully created. MPP .18 and .20 both do it.

This fails:
LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")
SetMemoryMax(1)
MP_Pipeline("""

### platform: win64
LoadPlugin("C:\HDTV Tools\L-SMASH-Works\x64\LSMASHSource.dll")

LWLibavVideoSource("source.mkv")

### prefetch: 16, 8
### ###
""")

VD2 returns this:
https://i.imgur.com/ss03pra.png

It also fails to load from AVSmeter and x264. The .lwi file appears to only be partially created. It's smaller than when the source fully loads and the .lwi doesn't have a proper "footer" in it.

This works:
LoadPlugin("C:\HDTV Tools\L-SMASH-Works\x64\LSMASHSource.dll")

LWLibavVideoSource("source.mkv")

After using the version of the script without MPP and it returns video (the full .lwi is created) then the MP_Pipeline version of the script will work fine. Is there some sort of timeout internal to MPP that is getting tripped?

[Stereodude]

Also, MPP .20 is incompatible with the old x64 build of DGMPGDec from 2010 (I'm not aware of a newer x64 build). You basically just get a green screen as the output if used in a MPP segment. It works with MPP .18. The x64 build works outside of MPP .20.


FWIW, the 32bit version of DGMPGDec works with MPP .20.

[real.finder]

Also, MPP .20 is incompatible with the old x64 build of DGMPGDec from 2010 (I'm not aware of a newer x64 build). You basically just get a green screen as the output if used in a MPP segment. It works with MPP .18. The x64 build works outside of MPP .20.


FWIW, the 32bit version of DGMPGDec works with MPP .20.

https://kuroko.fushizen.eu/bin/

[Stereodude]

https://kuroko.fushizen.eu/bin/
I can't read Japanese, but it's a stripped down version isn't it? The cpu2 argument is ignored / not used right?

[pinterf]

MP_Pipeline doesn't work with L-SMASH-Works until the .lwi file has already been successfully created. MPP .18 and .20 both do it.

This fails:
LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")
SetMemoryMax(1)
MP_Pipeline("""

### platform: win64
LoadPlugin("C:\HDTV Tools\L-SMASH-Works\x64\LSMASHSource.dll")

LWLibavVideoSource("source.mkv")

### prefetch: 16, 8
### ###
""")

VD2 returns this:
[lot of progress inditator messages]

It also fails to load from AVSmeter and x264. The .lwi file appears to only be partially created. It's smaller than when the source fully loads and the .lwi doesn't have a proper "footer" in it.

This works:
LoadPlugin("C:\HDTV Tools\L-SMASH-Works\x64\LSMASHSource.dll")

LWLibavVideoSource("source.mkv")

After using the version of the script without MPP and it returns video (the full .lwi is created) then the MP_Pipeline version of the script will work fine. Is there some sort of timeout internal to MPP that is getting tripped?
Could you somehow disable those message flow that appears on stdout during indexing? Seems like those "creating lwi" messages somehow confuse the internal pipe logic which is using stdout as well.

[real.finder]

I can't read Japanese, but it's a stripped down version isn't it? The cpu2 argument is ignored / not used right?

it seems https://i.imgur.com/njIyNGu.png

[Stereodude]

Could you somehow disable those message flow that appears on stdout during indexing? Seems like those "creating lwi" messages somehow confuse the internal pipe logic which is using stdout as well.
It doesn't look like there is an argument that can suppress them.

[pinterf]

It doesn't look like there is an argument that can suppress them.
I see. During an initial handshake the host process can detect a "SLAVE_OK" message sent over stdout. If no "SLAVE_OK" is received then it assumes that something bad happened and starts collecting data sent over the stdout pipe and waits until the internal buffer size (1024 bytes) is full (or timeout). Then it exits and is showing us the text which is intended to be an Error text, but it is only the indexing log. This 1024 bytes is that you posted from VD log screen.

[FranceBB]

I'm probably doing something wrong, but why this doesn't work?


MP_Pipeline("""
video1=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1220_001.MP4")
### export clip: video1
### ###
video2=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1226_001.MP4")
### export clip: video2
### pass clip: video1
### ###

video1++video2
### ###
""")


https://i.imgur.com/epJffbV.png


nor does this:


MP_Pipeline("""
video1=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1220_001.MP4")
### export clip: video1
### ###
video2=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1226_001.MP4")
### export clip: video1, video2
### ###

video1++video2
### ###
""")


https://i.imgur.com/epJffbV.png

nor does this


MP_Pipeline("""
FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1220_001.MP4")
### ###
video2=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1226_001.MP4")
myvideo=last++video2
return myvideo
### ###
""")


https://i.imgur.com/Nz03xaB.png

while this clearly does:


FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1220_001.MP4")
video2=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1226_001.MP4")
myvideo=last++video2
return myvideo

[gispos]

I think it's because you don't pass the variable 'last'.
That's how it works for me.

MP_Pipeline("""
### inherit start ###
SourceFile = String(ScriptDir()) + "IMG_2563.MOV_x64.avs.mkv"
SourceFile = Exist(SourceFile) ? SourceFile : "E:\Ablage\IMG_2563.MOV_x64.avs.mkv"
### inherit end ###
LWLibavVideoSource(SourceFile, cache=False)
### ###

v2=LWLibavVideoSource(SourceFile, cache=False)

### export clip: v2
### ###
v3=Tweak(v2, hue=0.0, sat=1.5, bright=0, cont=1.0, coring=True, sse=False, startHue=0, endHue=360, maxSat=150, minSat=0, interp=16)
last+v3

### ###
return last
""")


or

MP_Pipeline("""
### inherit start ###
SourceFile = String(ScriptDir()) + "IMG_2563.MOV_x64.avs.mkv"
SourceFile = Exist(SourceFile) ? SourceFile : "E:\Ablage\IMG_2563.MOV_x64.avs.mkv"
### inherit end ###
LWLibavVideoSource(SourceFile, cache=False)
### ###
video2=LWLibavVideoSource(SourceFile, cache=False)
last++video2
### ###
return last
""")

[StainlessS]

Think there were several posts about that requirement for "return last", fairly recently, maybe within last 6 months. [not sure which thread]

EDIT: Here:- https://forum.doom9.org/showthread.php?p=1914707#post1914707
See several posts ahead of it too.

[FranceBB]

Right... right...
I forgot about return last, but it didn't help as this failed:


MP_Pipeline("""
video1=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1220_001.MP4")
### export clip: video1
### ###
video2=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1226_001.MP4")
### export clip: video1, video2
### ###

video1++video2++video3

return last

""")


So I tried with Gispos's method and indeed it worked:


MP_Pipeline("""
### inherit start ###
SourceFile = String(ScriptDir()) + "20190806_1220_001.MP4"
SourceFile = Exist(SourceFile) ? SourceFile : "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
### inherit end ###
FFVideoSource(SourceFile, cache=False)
### ###
video2=FFVideoSource(SourceFile, cache=False)
last++video2
### ###
return last
""")



but here is the thing, as I tried adding more and more SourceFile, it broke.
This code is still working:


MP_Pipeline("""
### inherit start ###
SourceFile = String(ScriptDir()) + "20190806_1220_001.MP4"
SourceFile = Exist(SourceFile) ? SourceFile : "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
SourceFile2 = String(ScriptDir()) + "20190806_1226_001.MP4"
SourceFile2 = Exist(SourceFile2) ? SourceFile2 : "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
SourceFile3 = String(ScriptDir()) + "20190807_2005_001.MP4"
SourceFile3 = Exist(SourceFile3) ? SourceFile3 : "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
### inherit end ###
FFVideoSource(SourceFile, cache=False)
### ###
video2=FFVideoSource(SourceFile2, cache=False)
last++video2
### ###
video3=FFVideoSource(SourceFile3, cache=False)
last++video3
### ###

return last
""")


but this one is not:


MP_Pipeline("""
### inherit start ###
SourceFile = String(ScriptDir()) + "20190806_1220_001.MP4"
SourceFile = Exist(SourceFile) ? SourceFile : "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
SourceFile2 = String(ScriptDir()) + "20190806_1226_001.MP4"
SourceFile2 = Exist(SourceFile2) ? SourceFile2 : "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
SourceFile3 = String(ScriptDir()) + "20190807_2005_001.MP4"
SourceFile3 = Exist(SourceFile3) ? SourceFile3 : "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
SourceFile4 = String(ScriptDir()) + "20190807_2011_001.MP4"
SourceFile4 = Exist(SourceFile4) ? SourceFile4 : "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
### inherit end ###
FFVideoSource(SourceFile, cache=False)
### ###
video2=FFVideoSource(SourceFile2, cache=False)
last++video2
### ###
video3=FFVideoSource(SourceFile3, cache=False)
last++video3
### ###
video4=FFVideoSource(SourceFile4, cache=False)
last++video4
### ###

return last
""")


however this one is:


MP_Pipeline("""
### inherit start ###
SourceFile = String(ScriptDir()) + "20190806_1220_001.MP4"
SourceFile = Exist(SourceFile) ? SourceFile : "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
SourceFile4 = String(ScriptDir()) + "20190807_2011_001.MP4"
SourceFile4 = Exist(SourceFile4) ? SourceFile4 : "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
### inherit end ###
FFVideoSource(SourceFile, cache=False)
### ###
### ###
video4=FFVideoSource(SourceFile4, cache=False)
last++video4
### ###

return last
""")


Why?

I also tried with:


MP_Pipeline("""
### inherit start ###
video1 = "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
video2 = "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
video3 = "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
video4 = "I:\temp\Raw Canon EOS R\20190807_2008_001.MP4"
video5 = "I:\temp\Raw Canon EOS R\20190807_2010_001.MP4"
video6 = "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
video7 = "I:\temp\Raw Canon EOS R\20190807_2013_001.MP4"
video8 = "I:\temp\Raw Canon EOS R\20190808_1014_001.MP4"
video9 = "I:\temp\Raw Canon EOS R\20190808_1015_001.MP4"
video10 = "I:\temp\Raw Canon EOS R\20190808_1016_001.MP4"
video11 = "I:\temp\Raw Canon EOS R\20190808_1531_002.MP4"
video12 = "I:\temp\Raw Canon EOS R\20190808_1619_001.MP4"
video13 = "I:\temp\Raw Canon EOS R\20190808_1620_001.MP4"
video14 = "I:\temp\Raw Canon EOS R\20190808_1621_001.MP4"
video15 = "I:\temp\Raw Canon EOS R\20190812_1400_001.MP4"
video16 = "I:\temp\Raw Canon EOS R\20190812_1947_001.MP4"
video17 = "I:\temp\Raw Canon EOS R\20190812_1948_001.MP4"
video18 = "I:\temp\Raw Canon EOS R\20190812_2006_001.MP4"
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
v2=FFVideoSource(video2, cache=False)
last=last++v2
return last
### ###
v3=FFVideoSource(video3, cache=False)
last=last++v3

return last
""")


https://i.imgur.com/4zJsCTZ.png

same goes for this one:


MP_Pipeline("""
### inherit start ###
video1 = "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
video2 = "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
video3 = "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
video4 = "I:\temp\Raw Canon EOS R\20190807_2008_001.MP4"
video5 = "I:\temp\Raw Canon EOS R\20190807_2010_001.MP4"
video6 = "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
video7 = "I:\temp\Raw Canon EOS R\20190807_2013_001.MP4"
video8 = "I:\temp\Raw Canon EOS R\20190808_1014_001.MP4"
video9 = "I:\temp\Raw Canon EOS R\20190808_1015_001.MP4"
video10 = "I:\temp\Raw Canon EOS R\20190808_1016_001.MP4"
video11 = "I:\temp\Raw Canon EOS R\20190808_1531_002.MP4"
video12 = "I:\temp\Raw Canon EOS R\20190808_1619_001.MP4"
video13 = "I:\temp\Raw Canon EOS R\20190808_1620_001.MP4"
video14 = "I:\temp\Raw Canon EOS R\20190808_1621_001.MP4"
video15 = "I:\temp\Raw Canon EOS R\20190812_1400_001.MP4"
video16 = "I:\temp\Raw Canon EOS R\20190812_1947_001.MP4"
video17 = "I:\temp\Raw Canon EOS R\20190812_1948_001.MP4"
video18 = "I:\temp\Raw Canon EOS R\20190812_2006_001.MP4"
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
mysource2=FFVideoSource(video2, cache=False)
last++mysource2
### ###
mysource3=FFVideoSource(video3, cache=False)
last++mysource3

return last
""")


https://i.imgur.com/4zJsCTZ.png

I know that the compiler is always right, but... what is it trying to tell me this time? :confused:

and... by the way, I'm not mad, files are there and are not corrupted and have been indexed correctly, so... why?

https://i.imgur.com/eRjtAza.png

[StainlessS]

Unhandled C++ execption is a dll error, not your fault.

No idea bout the rest of it, tis a mystery to me [mp_pipeline thingy].

[pinterf]

@FranceBB: which Avisynth version are you using?

[gispos]

MP_Pipeline("""
### inherit start ###
video1 = "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
video2 = "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
video3 = "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
video4 = "I:\temp\Raw Canon EOS R\20190807_2008_001.MP4"
video5 = "I:\temp\Raw Canon EOS R\20190807_2010_001.MP4"
video6 = "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
video7 = "I:\temp\Raw Canon EOS R\20190807_2013_001.MP4"
video8 = "I:\temp\Raw Canon EOS R\20190808_1014_001.MP4"
video9 = "I:\temp\Raw Canon EOS R\20190808_1015_001.MP4"
video10 = "I:\temp\Raw Canon EOS R\20190808_1016_001.MP4"
video11 = "I:\temp\Raw Canon EOS R\20190808_1531_002.MP4"
video12 = "I:\temp\Raw Canon EOS R\20190808_1619_001.MP4"
video13 = "I:\temp\Raw Canon EOS R\20190808_1620_001.MP4"
video14 = "I:\temp\Raw Canon EOS R\20190808_1621_001.MP4"
video15 = "I:\temp\Raw Canon EOS R\20190812_1400_001.MP4"
video16 = "I:\temp\Raw Canon EOS R\20190812_1947_001.MP4"
video17 = "I:\temp\Raw Canon EOS R\20190812_1948_001.MP4"
video18 = "I:\temp\Raw Canon EOS R\20190812_2006_001.MP4"
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
mysource2=FFVideoSource(video2, cache=False)
last++mysource2
### ###
mysource3=FFVideoSource(video3, cache=False)
last++mysource3

### ###
return last
""")



It is not due to 'inherit start' and 'inherit end', this is just one method I like to use.

You make a mistake at the end of MP_Pipeline. Please compare.

MP_Pipeline("""
### inherit start ###
video1 = "E:\Ablage\IMG_2563.MOV_x64.avs.mkv"
### inherit end ###
LWLibavVideoSource(video1, cache=False)
### ###
mysource2=LWLibavVideoSource(video1, cache=False)
last++mysource2

### ###
mysource3=LWLibavVideoSource(video1, cache=False)
last++mysource3
### ### (note this line)
return last
""")


After the last ### ### the process is in the main thread and MP_Pipeline no longer has access to the clip variables.
And MP_Pipeline can only access what is written in 'inherit start'.
For me it usually looks like this.

### ###
audio=LWLibavAudioSource(SourceFile, cache=False)
audioDub(last, audio)
return last
""")


Edit: I added the missing one to your code. Try it.

[FranceBB]

After the last ### ### the process is in the main thread and MP_Pipeline no longer has access to the clip variables.
And MP_Pipeline can only access what is written in 'inherit
Edit: I added the missing one to your code. Try it.

Same.

This returns unhandled C++ exception:


MP_Pipeline("""
### inherit start ###
video1 = "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
video2 = "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
video3 = "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
video4 = "I:\temp\Raw Canon EOS R\20190807_2008_001.MP4"
video5 = "I:\temp\Raw Canon EOS R\20190807_2010_001.MP4"
video6 = "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
video7 = "I:\temp\Raw Canon EOS R\20190807_2013_001.MP4"
video8 = "I:\temp\Raw Canon EOS R\20190808_1014_001.MP4"
video9 = "I:\temp\Raw Canon EOS R\20190808_1015_001.MP4"
video10 = "I:\temp\Raw Canon EOS R\20190808_1016_001.MP4"
video11 = "I:\temp\Raw Canon EOS R\20190808_1531_002.MP4"
video12 = "I:\temp\Raw Canon EOS R\20190808_1619_001.MP4"
video13 = "I:\temp\Raw Canon EOS R\20190808_1620_001.MP4"
video14 = "I:\temp\Raw Canon EOS R\20190808_1621_001.MP4"
video15 = "I:\temp\Raw Canon EOS R\20190812_1400_001.MP4"
video16 = "I:\temp\Raw Canon EOS R\20190812_1947_001.MP4"
video17 = "I:\temp\Raw Canon EOS R\20190812_1948_001.MP4"
video18 = "I:\temp\Raw Canon EOS R\20190812_2006_001.MP4"
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
mysource2=FFVideoSource(video2, cache=False)
last++mysource2
### ###
mysource3=FFVideoSource(video3, cache=False)
last++mysource3


return last
""")


Same goes for this:


MP_Pipeline("""
### inherit start ###
video1 = "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
video2 = "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
video3 = "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
video4 = "I:\temp\Raw Canon EOS R\20190807_2008_001.MP4"
video5 = "I:\temp\Raw Canon EOS R\20190807_2010_001.MP4"
video6 = "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
video7 = "I:\temp\Raw Canon EOS R\20190807_2013_001.MP4"
video8 = "I:\temp\Raw Canon EOS R\20190808_1014_001.MP4"
video9 = "I:\temp\Raw Canon EOS R\20190808_1015_001.MP4"
video10 = "I:\temp\Raw Canon EOS R\20190808_1016_001.MP4"
video11 = "I:\temp\Raw Canon EOS R\20190808_1531_002.MP4"
video12 = "I:\temp\Raw Canon EOS R\20190808_1619_001.MP4"
video13 = "I:\temp\Raw Canon EOS R\20190808_1620_001.MP4"
video14 = "I:\temp\Raw Canon EOS R\20190808_1621_001.MP4"
video15 = "I:\temp\Raw Canon EOS R\20190812_1400_001.MP4"
video16 = "I:\temp\Raw Canon EOS R\20190812_1947_001.MP4"
video17 = "I:\temp\Raw Canon EOS R\20190812_1948_001.MP4"
video18 = "I:\temp\Raw Canon EOS R\20190812_2006_001.MP4"
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
mysource2=FFVideoSource(video2, cache=False)
last++mysource2
### ###
mysource3=FFVideoSource(video3, cache=False)
last++mysource3

### ###
return last
""")


So putting the ### ### at the end didn't really help and removing them didn't help either.

Unhandled C++ execption is a dll error, not your fault.

No idea bout the rest of it, tis a mystery to me [mp_pipeline thingy].

I see...

@FranceBB: which Avisynth version are you using?

Ah, Ferenc is here. :D
I'm running Avisynth+ 3.6.1 r3300 on x86 under Windows XP with PAE turned on and 32 GB of RAM.

Output from AVSMeter:

Operating system: Windows XP (x86) Service Pack 3.0 (Build 2600)

CPU: Intel(R) Xeon(R) CPU X5482 @ 3.20GHz / Xeon (Harpertown)
MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1
4 physical cores / 4 logical cores
[Avisynth info]
VersionString: AviSynth+ 3.6.1 (r3300, 3.6, i386)
VersionNumber: 2.60
File / Product version: 3.6.1.0 / 3.6.1.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\WINDOWS\system32\avisynth.dll
Avisynth.dll time stamp: 2020-06-19, 20:44:24 (UTC)
PluginDir2_5 (HKLM, x86): C:\Programmi\AviSynth\plugins
PluginDir+ (HKLM, x86): C:\Programmi\AviSynth+\plugins+


So it's the last stable release.
The output of AVSMeter doesn't really show anything either:

https://i.imgur.com/Mhrvrif.png

[gispos]

Same.
So putting the ### ### at the end didn't really help and removing them didn't help either.

You're right, the ### ### doesn't do anything.
The filename comes from 'inherit' and the last clip variable (mysource3) is created in the main thread.

[gispos]

Same.

This returns unhandled C++ exception:

Ah, Ferenc is here. :D
I'm running Avisynth+ 3.6.1 r3300 on x86 under Windows XP with PAE turned on and 32 GB of RAM.

Does that work for you now with MP_Pipeline?
If not then my posting was a bump ;)

[FranceBB]

Not really...
I'm still stuck... :(

EDIT: For the sake of avoid a potential Windows-related thing, I tried the very same script on Windows 10 x64 Enterprise with the very same version of MPP and AVS+; same result, unhandled C++ exception...
Let's see if great master Ferenc is gonna jump on this one and come to rescue...

[pinterf]

Not really...
I'm still stuck... :(

EDIT: For the sake of avoid a potential Windows-related thing, I tried the very same script on Windows 10 x64 Enterprise with the very same version of MPP and AVS+; same result, unhandled C++ exception...
Let's see if great master Ferenc is gonna jump on this one and come to rescue...
You know well if a problem is unexplainable than after hesitating a bit, sooner or later I can't resist and dig into the mistery. :)

So I have put some debugging into MP_Pipeline.
The error comes from parsing the inherit block (my new error message, much better than unknown exception).
"Error in regex library while copying 'inherit block':
regex_error(error_stack): There was insufficient memory to determine whether the regular expression could match the specified character sequence"

Parsing is done with the standard std::regex C++ library.

Pattern is
^\s*### inherit start ###\s*$(?:.|\s)*?^\s*### inherit end ###\s*$

Probably this pattern is too complex for the engine to analyze against the source. Right now I have not any clue about the workaround.

This is what I have found about the topic.
https://stackoverflow.com/questions/27331047/c-std-regex-crashes-in-msvc-during-long-multiline-match

edit: of course one can do it w/o involving the regex library, though this present method is more elegant (but fails)

[gispos]

Parsing is done with the standard std::regex C++ library.

Pattern is
^\s*### inherit start ###\s*$(?:.|\s)*?^\s*### inherit end ###\s*$

Probably this pattern is too complex for the engine to analyze against the source.

Could that have something to do with the fact that I get strange errors when I pass an Ansi String with 'Eval'.
I think I didn't have any problems with older Avisynth versions without 'Plus', but I can also be wrong.

[pinterf]

No, those long inherit sections are not affecting Eval. Nevertheless I have already fixed this error (release later, patience please)

[gispos]

No, those long inherit sections are not affecting Eval. Nevertheless I have already fixed this error (release later, patience please)
Too bad. :)
I am sure that I opened Avisynth scripts with 'Eval' back then. After an update to a newer version (I think it was one of the first MT versions at that time),
I could only open scripts with AviSource, depending on the content. The current Avisynth versions are at war with Ansi Strings. :(

Edit:
...Nevertheless I have already fixed this error (release later, patience please)
I don't know if I got it right. Have you already fixed something in this regard?

[FranceBB]

Just for the sake of testing, I tested the very same script with the new Avisynth Test 4 version and it still behaves the same.

By the way and totally unrelated, ConvertBits() no longer throws an error for 8-16 when the input = output bit depth correctly! :)

[pinterf]

Yes, fixed the crash by not using regex. It is not avisynth related. (On my machine at the moment, I have to cleanup the source before commiting it and release)

[pinterf]

New version, download v0.22 (https://github.com/pinterf/MP_Pipeline/releases/tag/0.22)
Fix crash when there were too many characters (over some hundred) in script between
inherit start and inherit end markers (regex library limitation)

[FranceBB]

Thanks! :D
Now I no longer have an unhandled C++ exception and I get an error message from all the scripts.


MP_Pipeline("""
video1=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1220_001.MP4")
### export clip: video1
### ###
video2=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1226_001.MP4")
### export clip: video2
### pass clip: video1
### ###
video3=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2005_001.MP4")
### export clip: video3
### pass clip: video1, video2
### ###
video4=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2008_001.MP4")
### export clip: video4
### pass clip: video1, video2, video3
### ###
video5=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2010_001.MP4")
### export clip: video5
### pass clip: video1, video2, video3, video4
### ###
video6=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2011_001.MP4")
### export clip: video6
### pass clip: video1, video2, video3, video4, video5
### ###
video7=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2013_001.MP4")
### export clip: video7
### pass clip: video1, video2, video3, video4, video5, video6
### ###
video8=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1014_001.MP4")
### export clip: video8
### pass clip: video1, video2, video3, video4, video5, video6, video7
### ###
video9=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1015_001.MP4")
### export clip: video9
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8
### ###
video10=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1016_001.MP4")
### export clip: video10
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9
### ###
video11=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1531_002.MP4")
### export clip: video11
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10
### ###
video12=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1619_001.MP4")
### export clip: video12
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11
### ###
video13=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1620_001.MP4")
### export clip: video13
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12
### ###
video14=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1621_001.MP4")
### export clip: video14
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13
### ###
video15=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_1400_001.MP4")
### export clip: video15
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14
### ###
video16=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_1947_001.MP4")
### export clip: video16
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14, video15
### ###
video17=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_1948_001.MP4")
### export clip: video17
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14, video15, video16
### ###
video18=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_2006_001.MP4")
### export clip: video18
### pass clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14, video15, video16, video17
### ###

video1++video2++video3++video4++video5++video6++video7++video8++video9++video10++video11++video12++video13++video14++video15++video16++video17++video18

return last

""")


https://i.imgur.com/stNMfcp.png


MP_Pipeline("""
video1=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1220_001.MP4")
### export clip: video1
### ###
video2=FFVideoSource("I:\temp\Raw Canon EOS R\20190806_1226_001.MP4")
### export clip: video1, video2
### ###
video3=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2005_001.MP4")
### export clip: video1, video2, video3
### ###
video4=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2008_001.MP4")
### export clip: video1, video2, video3, video4
### ###
video5=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2010_001.MP4")
### export clip: video1, video2, video3, video4, video5
### ###
video6=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2011_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6
### ###
video7=FFVideoSource("I:\temp\Raw Canon EOS R\20190807_2013_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7
### ###
video8=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1014_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8
### ###
video9=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1015_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9
### ###
video10=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1016_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10
### ###
video11=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1531_002.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11
### ###
video12=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1619_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12
### ###
video13=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1620_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13
### ###
video14=FFVideoSource("I:\temp\Raw Canon EOS R\20190808_1621_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14
### ###
video15=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_1400_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14, video15
### ###
video16=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_1947_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14, video15, video16
### ###
video17=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_1948_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14, video15, video16, video17
### ###
video18=FFVideoSource("I:\temp\Raw Canon EOS R\20190812_2006_001.MP4")
### export clip: video1, video2, video3, video4, video5, video6, video7, video8, video9, video10, video11, video12, video13, video14, video15, video16, video17, video18
### ###

video1++video2++video3++video4++video5++video6++video7++video8++video9++video10++video11++video12++video13++video14++video15++video16++video17++video18

return last

""")


https://i.imgur.com/srfzOTO.png


MP_Pipeline("""
### inherit start ###
video1 = String(ScriptDir()) + "20190806_1220_001.MP4"
video1 = Exist(video1) ? video1 : "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
video2 = String(ScriptDir()) + "20190806_1226_001.MP4"
video2 = Exist(video2) ? video2 : "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
video3 = String(ScriptDir()) + "20190807_2005_001.MP4"
video3 = Exist(video3) ? video3 : "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
video4 = String(ScriptDir()) + "20190807_2008_001.MP4"
video4 = Exist(video4) ? video4 : "I:\temp\Raw Canon EOS R\20190807_2008_001.MP4"
video5 = String(ScriptDir()) + "20190807_2010_001.MP4"
video5 = Exist(video5) ? video5 : "I:\temp\Raw Canon EOS R\20190807_2010_001.MP4"
video6 = String(ScriptDir()) + "20190807_2011_001.MP4"
video6 = Exist(video6) ? video6 : "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
video7 = String(ScriptDir()) + "20190807_2013_001.MP4"
video7 = Exist(video7) ? video7 : "I:\temp\Raw Canon EOS R\20190807_2013_001.MP4"
video8 = String(ScriptDir()) + "20190808_1014_001.MP4"
video8 = Exist(video8) ? video8 : "I:\temp\Raw Canon EOS R\20190808_1014_001.MP4"
video9 = String(ScriptDir()) + "20190808_1015_001.MP4"
video9 = Exist(video9) ? video9 : "I:\temp\Raw Canon EOS R\20190808_1015_001.MP4"
video10 = String(ScriptDir()) + "20190808_1016_001.MP4"
video10 = Exist(video10) ? video10 : "I:\temp\Raw Canon EOS R\20190808_1016_001.MP4"
video11 = String(ScriptDir()) + "20190808_1531_002.MP4"
video11 = Exist(video11) ? video11 : "I:\temp\Raw Canon EOS R\20190808_1531_002.MP4"
video12 = String(ScriptDir()) + "20190808_1619_001.MP4"
video12 = Exist(video12) ? video12 : "I:\temp\Raw Canon EOS R\20190808_1619_001.MP4"
video13 = String(ScriptDir()) + "20190808_1620_001.MP4"
video13 = Exist(video13) ? video13 : "I:\temp\Raw Canon EOS R\20190808_1620_001.MP4"
video14 = String(ScriptDir()) + "20190808_1621_001.MP4"
video14 = Exist(video14) ? video14 : "I:\temp\Raw Canon EOS R\20190808_1621_001.MP4"
video15 = String(ScriptDir()) + "20190812_1400_001.MP4"
video15 = Exist(video15) ? video15 : "I:\temp\Raw Canon EOS R\20190812_1400_001.MP4"
video16 = String(ScriptDir()) + "20190812_1947_001.MP4"
video16 = Exist(video16) ? video16 : "I:\temp\Raw Canon EOS R\20190812_1947_001.MP4"
video17 = String(ScriptDir()) + "20190812_1948_001.MP4"
video17 = Exist(video17) ? video17 : "I:\temp\Raw Canon EOS R\20190812_1948_001.MP4"
video18 = String(ScriptDir()) + "20190812_2006_001.MP4"
video18 = Exist(video18) ? video18 : "I:\temp\Raw Canon EOS R\20190812_2006_001.MP4"
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
v2=FFVideoSource(video2, cache=False)
last=last++v2
return last
### ###
v3=FFVideoSource(video3, cache=False)
last=last++v3

return last
""")


https://i.imgur.com/iYTn8lT.png


Perhaps these messages will make something pop in your mind, 'cause as things stands, I'm glad that I don't get "unhandled C++ expression" anymore, but I still have no idea about what's wrong...

[pinterf]

Thanks! :D
Now I no longer have an unhandled C++ exception and I get an error message from all the scripts.

Thanks! v0.22 Removed until I check what had happened.
Edit: release re-enabled (w/o modification)

[pinterf]

Thanks! v0.22 Removed until I check what had happened.
Your new errors fortunately are not related to any new bugs.

Your first sample does not work because there must be at least one "last" in the first section. When I insert a dummy "BlankClip" there, it will work* (*: see later)

It even fails if I skelenonize it to a single video1:
MP_Pipeline("""
video1=FFVideoSource("V01.m2v")
### export clip: video1
### ###

video1

last

""")


But this one is O.K.

MP_Pipeline("""
BlankClip() ## providing a "last" clip variable
video1=FFVideoSource("V01.m2v")
### export clip: video1
### ###

video1

last

""")


*... until clip count reached 8.
Then we have another error message.
"MP_Pipeline: Unable to create slave process. Message: Script error: SharedMemoryServer: Each process can only export no more than 8 clips."

[pinterf]

The 3rd example is probably wrong as well.

I think you mustn't use "return last" there. MP_Pipeline inserts some "hidden" lines before and after the script blocks, when you put there a return last, this "epilog" will not be reached.

### inherit end ###
FFVideoSource(video1, cache=False)
### ###
v2=FFVideoSource(video2, cache=False)
last=last++v2
return last
### ###
v3=FFVideoSource(video3, cache=False)
last=last++v3

return last
""")

Works:
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
v2=FFVideoSource(video2, cache=False)
last=last++v2
### ###
v3=FFVideoSource(video3, cache=False)
last=last++v3

return last
""")

[FranceBB]

I tested again with your suggestion and this indeed works! :D


MP_Pipeline("""
### inherit start ###
video1 = String(ScriptDir()) + "20190806_1220_001.MP4"
video1 = Exist(video1) ? video1 : "I:\temp\Raw Canon EOS R\20190806_1220_001.MP4"
video2 = String(ScriptDir()) + "20190806_1226_001.MP4"
video2 = Exist(video2) ? video2 : "I:\temp\Raw Canon EOS R\20190806_1226_001.MP4"
video3 = String(ScriptDir()) + "20190807_2005_001.MP4"
video3 = Exist(video3) ? video3 : "I:\temp\Raw Canon EOS R\20190807_2005_001.MP4"
video4 = String(ScriptDir()) + "20190807_2008_001.MP4"
video4 = Exist(video4) ? video4 : "I:\temp\Raw Canon EOS R\20190807_2008_001.MP4"
video5 = String(ScriptDir()) + "20190807_2010_001.MP4"
video5 = Exist(video5) ? video5 : "I:\temp\Raw Canon EOS R\20190807_2010_001.MP4"
video6 = String(ScriptDir()) + "20190807_2011_001.MP4"
video6 = Exist(video6) ? video6 : "I:\temp\Raw Canon EOS R\20190807_2011_001.MP4"
video7 = String(ScriptDir()) + "20190807_2013_001.MP4"
video7 = Exist(video7) ? video7 : "I:\temp\Raw Canon EOS R\20190807_2013_001.MP4"
video8 = String(ScriptDir()) + "20190808_1014_001.MP4"
video8 = Exist(video8) ? video8 : "I:\temp\Raw Canon EOS R\20190808_1014_001.MP4"
video9 = String(ScriptDir()) + "20190808_1015_001.MP4"
video9 = Exist(video9) ? video9 : "I:\temp\Raw Canon EOS R\20190808_1015_001.MP4"
video10 = String(ScriptDir()) + "20190808_1016_001.MP4"
video10 = Exist(video10) ? video10 : "I:\temp\Raw Canon EOS R\20190808_1016_001.MP4"
video11 = String(ScriptDir()) + "20190808_1531_002.MP4"
video11 = Exist(video11) ? video11 : "I:\temp\Raw Canon EOS R\20190808_1531_002.MP4"
video12 = String(ScriptDir()) + "20190808_1619_001.MP4"
video12 = Exist(video12) ? video12 : "I:\temp\Raw Canon EOS R\20190808_1619_001.MP4"
video13 = String(ScriptDir()) + "20190808_1620_001.MP4"
video13 = Exist(video13) ? video13 : "I:\temp\Raw Canon EOS R\20190808_1620_001.MP4"
video14 = String(ScriptDir()) + "20190808_1621_001.MP4"
video14 = Exist(video14) ? video14 : "I:\temp\Raw Canon EOS R\20190808_1621_001.MP4"
video15 = String(ScriptDir()) + "20190812_1400_001.MP4"
video15 = Exist(video15) ? video15 : "I:\temp\Raw Canon EOS R\20190812_1400_001.MP4"
video16 = String(ScriptDir()) + "20190812_1947_001.MP4"
video16 = Exist(video16) ? video16 : "I:\temp\Raw Canon EOS R\20190812_1947_001.MP4"
video17 = String(ScriptDir()) + "20190812_1948_001.MP4"
video17 = Exist(video17) ? video17 : "I:\temp\Raw Canon EOS R\20190812_1948_001.MP4"
video18 = String(ScriptDir()) + "20190812_2006_001.MP4"
video18 = Exist(video18) ? video18 : "I:\temp\Raw Canon EOS R\20190812_2006_001.MP4"
### inherit end ###
FFVideoSource(video1, cache=False)
### ###
v2=FFVideoSource(video2, cache=False)
last=last++v2
### ###
v3=FFVideoSource(video3, cache=False)
last=last++v3
### ###
v4=FFVideoSource(video4, cache=False)
last=last++v4
### ###
v5=FFVideoSource(video5, cache=False)
last=last++v5
### ###
v6=FFVideoSource(video6, cache=False)
last=last++v6
### ###
v7=FFVideoSource(video7, cache=False)
last=last++v7
### ###
v8=FFVideoSource(video8, cache=False)
last=last++v8
### ###
v9=FFVideoSource(video9, cache=False)
last=last++v9
### ###
v10=FFVideoSource(video10, cache=False)
last=last++v10
### ###
v11=FFVideoSource(video11, cache=False)
last=last++v11
### ###
v12=FFVideoSource(video12, cache=False)
last=last++v12
### ###
v13=FFVideoSource(video13, cache=False)
last=last++v13
### ###
v14=FFVideoSource(video14, cache=False)
last=last++v14
### ###
v15=FFVideoSource(video15, cache=False)
last=last++v15
### ###
v16=FFVideoSource(video16, cache=False)
last=last++v16
### ###
v17=FFVideoSource(video17, cache=False)
last=last++v17
### ###
v18=FFVideoSource(video18, cache=False)
last=last++v18

return last
""")


Thank you!! ;)

[Music Fan]

Hi,

I didn't use Avisynth for 1 or 2 years and would like to re-install it. As I'm quite confused with 32/64 bit mixed plugins, I have some questions.
If I understand correctly, Avisynth+ 64 bit is not supposed to work with 32 bit plugins but it can be achieved with MP_Pipeline.
I use Windows 10 pro 64 and would like to install Avisynth+ 64 bit.

1) Is it enough to make work in a single script 32 and 64 bit plugins (thanks to MP_Pipeline) or Avisynth+ 32 bit has to be installed anyway to make work 32 bit plugins, even when used in a script using Avisynth+ 64 bit ?

2) Do I have to put only the 4 MP_Pipeline files of the x64 folder (dll, dll.slave.exe, ...) in Avisynth+ 64 bit's plugin folder to make the whole stuff work with 32 and 64 bit plugins ?

3) In which folder should I place the 32 bit plugins ?

Thanks ;)

[mastrboy]

Is there a known bug with using Internal String functions with MP_Pipeline?

As soon as I include a Internal Avisynth function like "findstr" or "leftstr" I get a "Unable to create slave process" exception.

Example script not working:
MP_Pipeline("""
### inherit start ###

script_folder = LeftStr(ScriptDir(),StrLen(ScriptDir()) - 1)
filename_no_ext = ReplaceStr(scriptfile(),".avs",".mkv")
src_file = script_folder + "\" + filename_no_ext

### inherit end ###
LWLibavVideoSource(src_file)

### ###

# crop borders
crop(240,0,-240,0)

""")

If I just set the script_folder and filename_no_ext manually it works as expected, but defeats the idea of loading a source file based on the Avisynth script name...

[gispos]

For ScriptDir() you have to use string(ScriptDir()), the other things I haven't tested yet.

[flossy_cake]

Hello, is there any possibility of audio support?

I want to do audio compression to make audio louder without clipping ("soft clipping") but there are only 32-bit plugins for that.

I would have used Soxfilter 64-bit compander but it's broken & linear access only.

[FranceBB]

Hello, is there any possibility of audio support?


+1 for audio support.

I asked for it long time ago (I guess last time I begged was probably 2015) but the original author was gone.
Heck, MPPipelins would have also been broken with newer Avisynth+ colorspaces if it wasn't for the always available and irreplaceable C++ & ASM Supreme Grandmaster Ferenc Pinter who picked it up and updated it to version 0.22.
I mean, honestly, pinterf has been picking up almost every project the community has left behind and single handedly updated everything to modern standard. If it wasn't for him I don't know where we would have been...


I would have used Soxfilter 64-bit compander but it's broken & linear access only.

Sox you said, uh?
Indeed the last Avisynth compatible version was 2.6.1 from 2016, however I guess we're gonna be in for a 2023 miracle after it being broken since 2017, 'cause Grandmaster Ferenc has picked up the "challenge":

https://forum.doom9.org/showthread.php?t=181351&page=115

[JamesJohnston]

I'm working on getting MP_Pipeline functional with a simple "hello world" type of pipeline. The process has been frustrating because this filter does not really give helpful error messages.

For example:


LoadPlugin("C:\VideoProject\Software\AviSynth-plugin\downloaded\MP_Pipeline_v0.23\x64\MP_Pipeline.dll")

MP_Pipeline("""
### platform: win64
### dll: C:\VideoProject\Software\AviSynthPlus_3.7.3_20230715-filesonly\x64\Output\avisynth.dll

xxColorBars()

### ###
""")


will yield an error frame that reads this when I try to preview it in AvsPmod:


MP_Pipeline: Unable to create slave process. Message:
(New File (1), line 10)


If I change "xxColorBars" to "ColorBars" in the example, then it will work. Easy to spot here, but more complex scripts are more likely to have errors that are harder to spot - getting the exact error in the embedded script would be really helpful!

Any chance this plugin could show the actual errors from the embedded script?

My environment:


AvsPmod 2.7.7.8 64-bit
Avisynth+ 3.7.3_20230715 64-bit
MP_Pipeline release from https://github.com/pinterf/MP_Pipeline/releases/tag/0.23 --- 64-bit plugin
Windows 10 64-bit

[spoRv]

What I do is to create the script I'd like to use within MP_Pipeline, test it, and if there are no errors just put it inside the MP_Pipeline

OR

just comment MP_Pipeline and """) lines with a # in the front of them, to see actual error.