MP_Pipeline 0.18 - run parts of avisynth script in external processes [2014-04-06]

[Stereodude]

@Stereodude: "last" should get passed through variables, if not I wonder if "### export clip: last" could work, otherwise explicitly calling last and exporting to next core could help, since on a normal workflow you wouldn't want to output clip1 or clip2 at that early stage.
You can also return last like this:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
last
### export clip: clip1, clip2
### prefetch: 8, 4
### ###
Heck, you could return ColorBars(), but unless you have some explicit action being done in the slave process outside of a variable you get the error I mentioned.

[Dogway]

You can also return last like this:


Didn't run onto that bug before, but thanks, nice to know.

Just tried and it works without problem in AvsP and vdub.

Problem is avstp.dll (by cretindesalpes), put this in your plugin folder and try.
I don't know whose fault. It's just a combination of dither_resize16(), MP_Pipeline() and avstp.dll which is needed for MT with cretindesalpes' mvtools. I don't know if it's wise to drop avstp.dll when dealing with mvtools based functions or whether it helps in conjunction to MP_Pipeline().

[SAPikachu]

FWIW, you might want to make it clear in the readme / documentation that each slave process must return something. IE:

Will not work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

Will work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
clip1
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

I was banging my head on the wall for quite some time trying to figure out why I couldn't get export to work and kept getting this message:
Script error: Invalid arguments to function
"MPP_PrepareDownstreamClip"

BTW, thanks for the very useful plugin!

Thanks, I just added a note about this to the OP.

[SAPikachu]

I was hoping something more elaborated, I guess that with that I can read that my script is correct, right? despite the 16, 16, 16 prefetch values.


Well, your script doesn't have obvious error, but I would suggest trying this:


MP_Pipeline("""
DGSource("1080p.dgi",crop_t=132,crop_b=140)
bicubicResize(1280,544,b=-.5,c=.25)
pre=fluxsmoothT(3)
### export clip: pre
### prefetch: 16, 8
### ###

setmemorymax(1024)
# Put smdegrain into separate process, since it is the most complex part
smdegrain(tr=2,thSAD=170,prefilter=pre,lsb_out=true,refinemotion=true,contrasharp=30)
### prefetch: 16, 0

### ###
DitherPost(stacked=true,prot=false,mode=6,ampn=1,staticnoise=true)
Limiter()
### ###
""")



Problem is avstp.dll (by cretindesalpes), put this in your plugin folder and try.
I don't know whose fault. It's just a combination of dither_resize16(), MP_Pipeline() and avstp.dll which is needed for MT with cretindesalpes' mvtools. I don't know if it's wise to drop avstp.dll when dealing with mvtools based functions or whether it helps in conjunction to MP_Pipeline().

I have avstp.dll in my plugins folder, and it didn't cause any problem..

[real.finder]

I have avstp.dll in my plugins folder, and it didn't cause any problem..

I test it, ok in win7 but in winxp will happen like dogway said

Does not occur if you do not put avsp.dll in autoload folder, and if you load avsp.dll manually in script (in xp) will not happen and will work normally :eek:

[SAPikachu]

OK I reproduced the problem in an XP VM, looks like a conflict between different versions of C runtime. You can try Avisynth+ (http://www.avs-plus.net/), seems it doesn't have this problem.

[Dogway]

Thanks real.finder I was starting to feel paranoid. I will load on demand, and switch to Avs+ on the long term along Win7, which is the next natural step on avisynth development. I just feel it's a bit too early with lack of MT, etc.

As for the script improvement, I wonder if fluxsmoothT doesn't need some prefetch too since it's looking for and backwards?

[aldix]

Released 0.16. Thanks @real.finder, @aldix and @turbojet reporting the bugs!


Excellent news, thanks so much :)

[SAPikachu]

As for the script improvement, I wonder if fluxsmoothT doesn't need some prefetch too since it's looking for and backwards?

Actually fluxsmoothT already has prefetch. I think what you meant is whether upstream of fluxsmoothT needs prefetching, right? IMO it won't make much difference because there are only 2 upstream filters, but of course I may be wrong and you can try it to see what happens. :)

[Dogway]

Sure, I guess I messed up with my wording. I thought that bicubicresize (or DitherResize16() for instance) would drag a bit too much for fluxsmoothT to keep going, specially resizes of such gap. The above is already filtered and got about 6.4fps and ~60% load on a i5-4670k, I think it's ok since I was looking for >=5fps rate. Thanks for help!

[SAPikachu]

Sure, I guess I messed up with my wording. I thought that bicubicresize (or DitherResize16() for instance) would drag a bit too much for fluxsmoothT to keep going, specially resizes of such gap. The above is already filtered and got about 6.4fps and ~60% load on a i5-4670k, I think it's ok since I was looking for >=5fps rate. Thanks for help!

Well, bicubicresize isn't very slow (at least much faster than smdegrain, not sure about DitherResize16 though), the bottleneck is smdegrain (correct me if I am wrong), so it shouldn't matter much. Anyways glad to hear you are happy with the speed. :)

[Dogway]

I'm having serious issues with a script, export clip doesn't seem to work:
MP_Pipeline("""
setmemorymax(1024)
ffvideoSource("E:\Part A.avi")

### inherit start ###
source=last
### inherit end ###
### prefetch: 16, 8
### ###

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
In this way the rain variable is getting "last" as source, instead of inherited "source" variable. Why is this? If I move the dfttest line also above the splitter, the "clean" variable also gets "last" (the denoised dfttest) instead of "source". I tested with a simplified code using mt_lut() and that worked so I'm not sure why it occurs.

[Stereodude]

I'm having serious issues with a script, export clip doesn't seem to work:
MP_Pipeline("""
setmemorymax(1024)
ffvideoSource("E:\Part A.avi")

### inherit start ###
source=last
### inherit end ###
### prefetch: 16, 8
### ###

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
In this way the rain variable is getting "last" as source, instead of inherited "source" variable. Why is this?Because you're not using it correctly. The inherit block with source = last is the same as putting source = last in every thread / segment. But, based on your description that's not what you're trying to do (redefine source in each thread / segment to be the implied output from the previous thread / segment. It needs to be like this:

MP_Pipeline("""
setmemorymax(1024)
source=ffvideoSource("E:\Part A.avi")
source
### export clip: source
### prefetch: 16, 8
### ###

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean
### pass clip: source
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
or:
MP_Pipeline("""
setmemorymax(1024)
ffvideoSource("E:\Part A.avi")
### prefetch: 16, 8
### ###
source = last

Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true)
clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true)
### export clip: clean, source
### prefetch: 8, 4
### ###

rain=smdegrain(source,tr=1,mfilter=last,lsb_out=true)
ReplaceFramesSimple (clean, rain, mappings="
[5952 6177] [6555 6989] [7376 8549] [8613 8934] [52373 52729]
[52838 54213] [54416 58738] [58873 59729] [60051 61030]")
### ###

SmoothLevels16(preset="tv2pc",interp=40,debug=false,dither=100,limiter=0,HQ=true)
ditherpost(mode=6)
""")
However, I think there still might be another mistake or two in there. The output from Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,u=true,v=true) becomes last for the next thread / segment. Is this what you want? Also, in my two modifications in this line clean=smdegrain(source,tr=2,mfilter=last,lsb_out=true) the source and last are the same footage (I think).

[Dogway]

Thanks a bunch Stereodude, this was driving me mad.
I guess that's what I wanted; "pass clip".

I don't know what errors you refer to. Dfttest is "last" until ReplaceFramesSimple comes in and outputs either "clean" or "rain". "rain" also needs dfttest because that's what mfilter needs, and also "source" (before dfttest) as the source of smdegrain. I'm going to check, because I would swear I tried your second variation and got as you said the same clip for "last" and "source" but for the "rain" variable. edit: worked using 2nd variation too.

[Stereodude]

I don't know what errors you refer to. Dfttest is "last" until ReplaceFramesSimple comes in and outputs either "clean" or "rain". "rain" also needs dfttest because that's what mfilter needs, and also "source" (before dfttest) as the source of smdegrain. I'm going to check, because I would swear I tried your second variation and got as you said the same clip for "last" and "source" but for the "rain" variable. edit: worked using 2nd variation too.
I will admit to not completely understanding how last works in AVIsynth. I don't typically use it. I had never used it prior to using MP_Pipeline. If it is updated after every filter not explicitly assigned to a variable as the implied return variable, then there's no error. I was thinking last was only the implied return from the thread / segment (or entire script if not using MP_Pipeline), not an implied variable that got updated after each line not assigned to a variable.

[Stereodude]

hi :)

in the 0.15 ver. when I load script in MPC or x264 (or anything) and than close it, some slave processes stay active in task manger and take memory, with 0.14 this not happened

and sometimes if used some complex script with dvd sources (480) some slave processes crashed (http://i.imgur.com/mXqSyQh.png) in closing (or end), it does not affect, but it's annoying, and that in both 0.14 or 0.15, and this sometimes not happened even with the same script.

Yeah, I can also report hanging slave processes.I'm still seeing a variation of this with .16. When working with scripts in Virtualdub the x86 version is pretty good about closing them, but the x64 version of VD with x64 MP_Pipeline seems to just leave them sit open. When you close VD they all disappear, but reloading the file (F2) or closing it and re-opening it causes threads to stay open.

[SAPikachu]

I'm still seeing a variation of this with .16. When working with scripts in Virtualdub the x86 version is pretty good about closing them, but the x64 version of VD with x64 MP_Pipeline seems to just leave them sit open. When you close VD they all disappear, but reloading the file (F2) or closing it and re-opening it causes threads to stay open.

Thanks for the report, will have a look on it later this week.

[Stereodude]

Thanks for the report, will have a look on it later this week.I found a specific repeatable case where the 32-bit version leaves threads open. If a script fails to open successfully in VirtualDub, generating some error during the opening process, the threads seem to end up orphaned and will stay open until you close VirtualDub.

[steptoe]

I still can't get my head around how MP_Pipeline works, basically I'm trying to get this to work in 32-bit. I use avisynth+

Once I can get that to work, I'll then try the 64-bit versions of the filters. Both 32-bit and 64-bit of avisynth+ are installed


Thanks again


SetMemoryMax(1024)
dss2("D:\PugWash.mkv")

Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\mvtools.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirt.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removegrain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\autogain\autogain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\tdeint\tdeint.dll")
Import("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirtmc.avs")

tdeint()
removedirtmc(75,false)
autogain()

This works perfectly in AVSPMod, but trying to get that to work with MP_Pipeline I just can't get my head around the commands

[SAPikachu]

I found a specific repeatable case where the 32-bit version leaves threads open. If a script fails to open successfully in VirtualDub, generating some error during the opening process, the threads seem to end up orphaned and will stay open until you close VirtualDub.

Finally got around to look on this. I can't reproduce the problem on latest (1.10.4) x64 version of VirtualDub if the script doesn't contain error, every time I hit F2 old slave processes are properly killed. Do you have a script that can reliably trigger this problem?

On the other hand, if the script has error, slave processes can't indeed be killed until host process exits. I will release a fix for this a bit later.

[SAPikachu]

I still can't get my head around how MP_Pipeline works, basically I'm trying to get this to work in 32-bit. I use avisynth+

Once I can get that to work, I'll then try the 64-bit versions of the filters. Both 32-bit and 64-bit of avisynth+ are installed


Thanks again


SetMemoryMax(1024)
dss2("D:\PugWash.mkv")

Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\mvtools.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirt.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removegrain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\autogain\autogain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\tdeint\tdeint.dll")
Import("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirtmc.avs")

tdeint()
removedirtmc(75,false)
autogain()

This works perfectly in AVSPMod, but trying to get that to work with MP_Pipeline I just can't get my head around the commands

Here is a starting point for you, you need to tweak the numbers yourself to fit your system, and I am not sure whether the speed will increase because RemoveDirtMC doesn't look very complex to me.

MP_Pipeline("""

### inherit start ###

Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\mvtools.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirt.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removegrain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\autogain\autogain.dll")
Loadplugin("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\tdeint\tdeint.dll")
Import("C:\Program Files (x86)\Applications\Video\AviSynth\plugins\removedirtmc\removedirtmc.avs")

### inherit end ###

dss2("D:\PugWash.mkv")
tdeint()
### prefetch: 32, 16
### ###

SetMemoryMax(1500) # Reduce this if you don't have enough memory
removedirtmc(75,false)
### prefetch: 32, 0
### ###

autogain()

""")

[Stereodude]

Finally got around to look on this. I can't reproduce the problem on latest (1.10.4) x64 version of VirtualDub if the script doesn't contain error, every time I hit F2 old slave processes are properly killed. Do you have a script that can reliably trigger this problem?I'll see if I can come up with one that repeats it reliably. My initial report of an issue on x64 may have been caused by the 2nd issue, that slave processes hang around if the script has errors when opening. I'll do more testing.

[BloC]

how MP_Pipeline works and What are the best settings for these orders ؟؟؟
I am tired of slow Encoding in the my computer Intel Corei5
I useing avs32x and avs64 ram 8 Gib



LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", showyuv=false, analyze=false, autowhite=false, autogain=false)

[aldix]

Well, i5 is rather slow ...

[Stereodude]

how MP_Pipeline works and What are the best settings for these orders ؟؟؟
I am tired of slow Encoding in the my computer Intel Corei5
I useing avs32x and avs64 ram 8 Gib



LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", showyuv=false, analyze=false, autowhite=false, autogain=false)

What x264 settings are you using, and what is the CPU usage of your i5 while you're encoding?

Well, i5 is rather slow ...What a helpful reply. I wouldn't consider an i5 slow. Depending on which one he has and how fast it's running it could be anywhere from decent to fast.

[BloC]

What x264 settings are you using, and what is the CPU usage of your i5 while you're encoding?

What a helpful reply. I wouldn't consider an i5 slow. Depending on which one he has and how fast it's running it could be anywhere from decent to fast.

My X264 settings

cabac=1 / ref=4 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=umh / subme=6 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=1 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=abr / mbtree=1 / bitrate=2200 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=50000 / vbv_bufsize=50000 / nal_hrd=none / ip_ratio=1.40 / aq=1:1.00

I am encoding 5 fps When using these filters

This is because it bothered me I want to learn to use MP_Pipeline 0.16

[Stereodude]

I am encoding 5 fps When using these filters

This is because it bothered me I want to learn to use MP_Pipeline 0.16What does your CPU usage look like when encoding?

I'm not familiar with most of the filters you're using, so I don't know which of the filters are slow. What resolution is your input source?

Edit: Your script might look something like this:
LoadPlugin("MP_Pipeline.dll")

MP_Pipeline("""
### inherit start ###
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")
### inherit end ###
LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

### prefetch: 12, 6
### ###
Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)
### prefetch: 16, 8
### ###

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise
### prefetch: 18, 9
### ###

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, \
gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", \
showyuv=false, analyze=false, autowhite=false, autogain=false)
### prefetch: 20, 10
""")
It may have more threads than it needs, but I don't know how CPU intensive some of them are.

[SAPikachu]

Released 0.17 (finally..). @Stereodude can you try this version to see whether it fixes the hanging process problem?

[nekosama]

I'm using avisynth 2.6A5 and I'm trying to split up my script because it keeps crashing because of high memory usage.
But the problem is that pipeline just doesn't want to work.
http://i.imgur.com/TMlSqgq.png
here's my avs script.
MP_Pipeline("""op=import("other part of source.avs")
ed=import("other part of source.avs")
LWLibavVideoSource("etc.mkv",threads=1)
a=trim(0,1628)
b=trim(3861,30447)
c=trim(32608,34119)
a ++ op ++ b ++ ed ++ c
aa=ediaa()
softsmooth=vaguedenoiser(1,nsteps=8,method=3)
strange(24774,24917,ediaa().eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(568,1628,eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(4240,5376,aa)
strange(5377,5448,ediaa())
strange(5497,5952,ediaa())
strange(6061,30447,aa)
strange(32737,34119,aa)
### prefetch: 16, 0
### ###
""")
strange(33059,34119,softsmooth.gradfun3())

trim(30457,34120)
btw in the x64 folder there are files for win32 and in x86 folder files for win64, why? I'm using the 32 version (also I have 64 system but I thought you would need to run avisynth in 64 to use 64 bit pipeline, correct me if wrong)

[SAPikachu]

I'm using avisynth 2.6A5 and I'm trying to split up my script because it keeps crashing because of high memory usage.
But the problem is that pipeline just doesn't want to work.
http://i.imgur.com/TMlSqgq.png
here's my avs script.
MP_Pipeline("""op=import("other part of source.avs")
ed=import("other part of source.avs")
LWLibavVideoSource("etc.mkv",threads=1)
a=trim(0,1628)
b=trim(3861,30447)
c=trim(32608,34119)
a ++ op ++ b ++ ed ++ c
aa=ediaa()
softsmooth=vaguedenoiser(1,nsteps=8,method=3)
strange(24774,24917,ediaa().eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(568,1628,eedi3_rpow2(2).Dither_convert_8_to_16().dither_resize16(1920,1080,kernel="spline36").ditherpost())
strange(4240,5376,aa)
strange(5377,5448,ediaa())
strange(5497,5952,ediaa())
strange(6061,30447,aa)
strange(32737,34119,aa)
### prefetch: 16, 0
### ###
""")
strange(33059,34119,softsmooth.gradfun3())

trim(30457,34120)


This is usually because the slave process crashed, can you try removing one part of your script at a time to see which line cause the problem?


btw in the x64 folder there are files for win32 and in x86 folder files for win64, why? I'm using the 32 version (also I have 64 system but I thought you would need to run avisynth in 64 to use 64 bit pipeline, correct me if wrong)

They are used for running scripts with mixing x86/x64 slaves, so that you can use x86 filters with x64 x264 for example. You need to install both x64 and x86 avisynth to use that.

[MeteorRain]

I am encoding 5 fps When using these filters


I won't say 5fps is slow on an i5 for QTGMC level script.

For example doing QTGMC on 1920x1080i source can easily put the speed into ~3fps level on my i7 4770.

If you don't want to put too much time on MP_Pipeline I'd rather suggest you do parallel encoding, 2 a time in your case, if the process doesn't consume more than 50% of overall resource.

If you have a 6 physical cores you can do 3 at a time.

[BloC]

What does your CPU usage look like when encoding?

I'm not familiar with most of the filters you're using, so I don't know which of the filters are slow. What resolution is your input source?

Edit: Your script might look something like this:
LoadPlugin("MP_Pipeline.dll")

MP_Pipeline("""
### inherit start ###
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mt_masktools-25.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\RemoveGrainSSE2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\mvtools2.dll")
LoadPlugin("D:\Needs help in Videos\filter vedio\plugins\nnedi3.dll")
### inherit end ###
LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Users\1\Desktop\115.d2v", cpu=4, info=3)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

### prefetch: 12, 6
### ###
Import("D:\Needs help in Videos\filter vedio\plugins\QTGMC.avs")
QTGMC(EZDenoise=5.0,Sharpness=1.0)
### prefetch: 16, 8
### ###

LanczosResize(704,384) # Lanczos (Sharp)

LoadPlugin("D:\Needs help in Videos\Transfer programs to mkv\MeGUI\MeGUI_2028_x86\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise
### prefetch: 18, 9
### ###

Tweak(sat=1.5,hue=-2.5,bright=8,cont=1.0,coring=false)
ConvertToYV12()ColorYUV(gain_y=0, off_y=0, gamma_y=0, cont_y=0, gain_u=0, off_u=0, \
gamma_u=0, cont_u=0, gain_v=0, off_v=0, gamma_v=0, cont_v=0, levels="TV->PC", opt="", \
showyuv=false, analyze=false, autowhite=false, autogain=false)
### prefetch: 20, 10
""")
It may have more threads than it needs, but I don't know how CPU intensive some of them are.

wooow wooow this scripts very good

thanks man for helping me ;)

[Dogway]

Is it possible to pass variables to next thread? They are not clip variables though, I'm trying to do something like this:

smdegrain(globals=2)
### ###
smdegrain(globals=1)

The second instance tries to read all the motion vectors exported from the first smdegrain, that is, Super, bv2, bv1, fv1, fv2, etc. I tried with export clip, but obviously didn't work.


Also out of curiosity, on a quad core what is an optimal division, 4, or 3 code chunks (leaving one free for the encoder)?

[SAPikachu]

Is it possible to pass variables to next thread? They are not clip variables though, I'm trying to do something like this:

smdegrain(globals=2)
### ###
smdegrain(globals=1)

The second instance tries to read all the motion vectors exported from the first smdegrain, that is, Super, bv2, bv1, fv1, fv2, etc. I tried with export clip, but obviously didn't work.


They are actually clip variables, but MVTools stores some additional data in another obscure place, so it is not possible to pass these clips to downstream (at least for now).


Also out of curiosity, on a quad core what is an optimal division, 4, or 3 code chunks (leaving one free for the encoder)?

Well, I think there is no solution suited for every case, it depends on what filters are used. I usually disregard number of cores and just place heavy filters in separate chunks when possible. Even then my CPU cannot be fully utilized.

[Dogway]

I see, cretindesalpes worked in mvtools mod, from the few I know are active and might know the internals of the plugin.

I probably wanted to say threads instead of cores, mine is a i5 so 4 cores 4 threads, but I guess your answer still applies here (making 5 or 6 chunks no problem). Thanks for help.

[real.finder]

I discovered fun fact

I have two different editions of Srestore.avsi in two autoload folder, one in plugins64 and the other in plugins

the one in plugins is the newest and accept call Srestore more than once in the script


The script was like:-

MP_Pipeline("""
### platform: win32
some filters
Srestore
Srestore
### ###
""")

When runs it in x264 32-bit works normally, but in x264 64-bit gives ERROR: SRESTORE CAN ONLY CALLED ONCE!!!

Which means that autoload avsi from out of MP_Pipeline which is 64-bit was affect in the processes, and if I import the new Srestore.avsi from plugins manually after ### platform: win32 problem will be solved

[SAPikachu]

I discovered fun fact

I have two different editions of Srestore.avsi in two autoload folder, one in plugins64 and the other in plugins

the one in plugins is the newest and accept call Srestore more than once in the script


The script was like:-

MP_Pipeline("""
### platform: win32
some filters
Srestore
Srestore
### ###
""")

When runs it in x264 32-bit works normally, but in x264 64-bit gives ERROR: SRESTORE CAN ONLY CALLED ONCE!!!

Which means that autoload avsi from out of MP_Pipeline which is 64-bit was affect in the processes, and if I import the new Srestore.avsi from plugins manually after ### platform: win32 problem will be solved

This is interesting, but I just had a quick test and can't get this behavior, 32bit and 64bit x264 don't affect slave's environment. Not sure what happened in your system though..

[real.finder]

This is interesting, but I just had a quick test and can't get this behavior, 32bit and 64bit x264 don't affect slave's environment. Not sure what happened in your system though..

My experience was only in Srestore and call it twice in the script

Didn't try something else

the summary of what happened, if I run the script in x264 64-bit, the Srestore in plugins64 will be adopted in 32 slave's environment in mp_pipeline, Ignoring that one in 32 autoload folder

and by the way, I using avs 64 (http://code.google.com/p/avisynth64/downloads/detail?name=avisynth64_8-29-10.rar&can=2&q=) and SEt mt (https://www.dropbox.com/s/xhqggxamegia420/avisynth_20130309.7z)

in windows server 2008 r2

[Boulder]

As SRestore is being discussed, is this a proper way to use MP_Pipeline:

MP_Pipeline("""
avstp_set_threads(2)
DGSource("who_coliseum.dgi")
Trim(0,43685)
BicubicResize(1280,1080)
QTGMC("slower",dct=5,tr2=0,chromamotion=true,chromanoise=false,SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1,edithreads=2)
BicubicResize(1280,720)
### prefetch 3,3
### ###
""")
SRestore(frate=25,mode=-2,speed=-1,dclip=Reduceby2(),cache=10)

AFAIK, SRestore needs linear access which should be ensured by the cache parameter.

[SAPikachu]

As SRestore is being discussed, is this a proper way to use MP_Pipeline:

MP_Pipeline("""
avstp_set_threads(2)
DGSource("who_coliseum.dgi")
Trim(0,43685)
BicubicResize(1280,1080)
QTGMC("slower",dct=5,tr2=0,chromamotion=true,chromanoise=false,SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1,edithreads=2)
BicubicResize(1280,720)
### prefetch 3,3
### ###
""")
SRestore(frate=25,mode=-2,speed=-1,dclip=Reduceby2(),cache=10)

AFAIK, SRestore needs linear access which should be ensured by the cache parameter.

1. There should be a colon after ### prefetch .
2. You can omit cache parameter in SRestore if prefetch is used, MPP will cache frames itself.
3. 3,3 may be not enough for SRestore AFAIK, and if the two numbers equal, MPP will only cache frames behind the current frame and won't prefetch frames.

I suggest trying this:
MP_Pipeline("""
avstp_set_threads(2)
DGSource("who_coliseum.dgi")
Trim(0,43685)
BicubicResize(1280,1080)
QTGMC("slower",dct=5,tr2=0,chromamotion=true,chromanoise=false,SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1,edithreads=2)
BicubicResize(1280,720)
### prefetch: 20, 10
### ###
""")
SRestore(frate=25,mode=-2,speed=-1,dclip=Reduceby2())

[Boulder]

Thanks, it seems to be running very smoothly now :)

[real.finder]

AFAIK, SRestore needs linear access which should be ensured by the cache parameter.

I used it with mt mode 2 naturally

and if the two numbers equal, MPP will only cache frames behind the current frame and won't prefetch frames.

so, with ### prefetch: 20, 10 will be 10 behind and 10 prefetch?

[SAPikachu]

so, with ### prefetch: 20, 10 will be 10 behind and 10 prefetch?

Correct.

[SAPikachu]

Released 0.18.

[turbojet]

I used to be able to load different avisynth.dll's fine but now having a problem. Any ideas?

MP_Pipeline("""###
### dll: "F:\avisynth+.dll"
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

MP_Pipeline: Unable to create slave process. Message: Unable to load "F:\avisynt
h+.dll", code = 126

[SAPikachu]

I used to be able to load different avisynth.dll's fine but now having a problem. Any ideas?

MP_Pipeline("""###
### dll: "F:\avisynth+.dll"
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

MP_Pipeline: Unable to create slave process. Message: Unable to load "F:\avisynt
h+.dll", code = 126


Try this:

MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

MP_Pipeline: Unable to create slave process. Message: Unable to load "F:\avisynt
h+.dll", code = 126


Note that we shouldn't put quotes around the file name.

[turbojet]

Thanks that's what I initially tried but was getting confused by avsmeter's avs version. version() inside the mp block confirms it's working.

[Groucho2004]

Thanks that's what I initially tried but was getting confused by avsmeter's avs version. version() inside the mp block confirms it's working.
Can you elaborate on that? What was confusing?

[turbojet]

AVSmeter reports the system's avisynth dll version rather than what's in the mp-pipeline block. Some examples, with avisynth 2.6 system dll:

version()
MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

#version() shows avisynth 2.6

MP_Pipeline("""###
### dll: F:\avisynth+.dll
version()
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")

#version() shows avisynth+

MP_Pipeline("""###
### dll: F:\avisynth+.dll
blankclip(1630,1920,1080,"YV12",23.976)
### ###
""")
version()

#version() shows avisynth 2.6

I'm not sure avsmeter could or should do anything about it, really up to you.

[Groucho2004]

AVSmeter reports the system's avisynth dll version rather than what's in the mp-pipeline block.
The alternate avisynth dll is loaded by an external process of which any application that loads the script is not and cannot be aware.
So, determining the version of that avisynth.dll can only happen on the script level unless MP_Pipeline exports a function for that purpose.