MP_Pipeline 0.18 - run parts of avisynth script in external processes [2014-04-06]

[SAPikachu]

OK, thanks. At least I know I'm not going deaf. :)

Overall impression: With the definite benefit from the prefetch facility (thanks for adding that! :) ), I'm finding it *most* useful in my comparison scripts, where I run the same basic pre-processing on several source files in parallel and then compare the results. Processing each source file via it's own slave process using MP_Pipeline and then combining the results in the main script works smoothly and efficiently every time, utilising up to 99% CPU and delivering speed gains to match. The only gotcha is memory: It's wise to use SetMemoryMax inside the MP_Pipleline call, with a parameter that won't overflow your free memory when all those parallel processes are demanding memory at once. (Formula: Take the average free memory (in Megabytes) you have when you first invoke VirtualDub (or whatever app calls AViSynth); divide that number by the number of parallel processes you're going to run; Shave off a few tens of megabytes for safety and/or uses that "SetMemoryMax" doesn't cover.)

With more linear AviSynth scripts, the picture is not so straightforward. Sometimes using MP_Pipeline gives the best speed-up; sometimes SetMTmode() does. Sometimes tweaking the prefetch and branch parameters makes all the difference; sometimes it makes virtually none. Like all multi-threading and multiprocessing techniques, it really depends on knowing where and how to use it - and that means knowing the characteristics of the filters in your script and thus how they will be affected by the various different ways of dividing up the work they do - to get the best results. And sometimes, no matter what combination of tricks (MT_Pipline, MT(), SetMTMode(), etc...) and parameters I try, I can't get above 70% CPU utilisation, and that just has to be accepted. Some bottleneck in my machine prevents enough data getting to the CPU to be processed any faster, I guess. (Time to start saving up for that new motherboard...?)

Big, big thanks to SAPikachu for a really useful plug-in. :)

Thanks for your great review. :)

[SAPikachu]

Released 0.13. Fixed a crashing bug so it is recommended to update to this version.

[real.finder]

I have avs 2.6 mt by SEt and avs64 2.5 (http://code.google.com/p/avisynth64/downloads/detail?name=avisynth64_8-29-10.rar&can=2&q=)

and I use this script

MP_Pipeline("""
### platform: win64
DGDecode_MPEG2Source("x:\xx.d2v").ThreadRequest()
SetMTMode(2)
ColorMatrix(d2v="x:\xx.d2v", threads=0)
Trim(x, xx) ++ Trim(xx, xxx) ++ Trim(xxxx, xxxxxx)
SetMTMode(6)
#~ tfm(output="matches.txt")
#~ tdecimate(mode=4,output="metrics.txt")
tfm(input="matches.txt")
tdecimate(mode=5, hybrid=2, vfrDec=1, input="metrics.txt", tfmIn="matches.txt", mkvOut="mkv-timecodesfile.txt", tcfv1=false)
Crop(x, x, -x, x)
xxxResize(1280, 720)
### ###
### platform: win32
LoadPlugin(AviSynthPluginsDir + "EEDI2 mt/EEDI2_imp.dll")
LoadPlugin(AviSynthPluginsDir + "avs26/mt_masktools-26.dll")
SetMemoryMax(1000)
SetMTMode(2)
filter1
filter2
SoraThread()
filter3
filter4
### lock threads to cores
### ###
""")
AssumeFrameBased

it gave me very good speed

MP_Pipeline is cool plugin, I can use the script in x264 64 and x264 32, and I can use it in avspmod

thank you for it

----------

edit 1: don't use MP_Pipeline in 1st pass (analysis) of tfm and tdecimate vfr (or animeivtc mode 4 with omode=2) Because it will generate empty files

[kolak]

SAPikachu I have a problem with latest mp_pipeline. Avisynth 2.6mt and Vdub 32bit on Win 7 64bit.
When I load script to Vdub and than close it all slave processes stay active and take memory. I also think older version did not need loadPlugin....latest one does (or is it my imagination :) )

[SAPikachu]

SAPikachu I have a problem with latest mp_pipeline. Avisynth 2.6mt and Vdub 32bit on Win 7 64bit.
When I load script to Vdub and than close it all slave processes stay active and take memory. I also think older version did not need loadPlugin....latest one does (or is it my imagination :) )

Do you mean MP_Pipeline needs to be manually loaded? If you put it into plugin autoload folder it should be automatically loaded on start. Or maybe you put it into wrong place?

About the slave process problem, I suspect avs-mt is the culprit. Can you try a regular avs build?

[kolak]

I fixed autoload by re-installing avisynth.

I tried "normal" avisynth and it's the same- processes don't close.
I tried loading script to MP Classic and also the same :(

I think older version was fine.

[SAPikachu]

That's strange, it is fine on my machine. Can you try the following steps to capture a memory dump for me to debug?

1. Use this version of MP_Pipeline (http://nmm.me/vz), and download and extract Procdump package (http://nmm.me/w3)
2. Load your script into vdub/mpc/whatever player, and close it
3. If the slave process is stuck, double-click "dump_immediately.cmd" in the Procdump package, and accept the agreement
4. A dump file (maybe very big) will be generated, please compress and upload it to any sharing site (Mediafire for example) and send it to me

[kolak]

I can do it.
I also found that it happens on my laptop only- on some other PC it works fine.

Check your PM.

[SAPikachu]

I just checked the dump, seems the slave process was stuck inside AppleProResDecoder, can you try disabling it and try again?

[kolak]

I tried reading ProRes file with ffvideo and it was fine.
Than I tried on other machine and yes- with qtinput and ProRes file it's also a problem.
I think it's qtinput+ProRes decoder- any way to solve it? Other mov formats seams to be fine and of course ProRes is the one which I'm interested the most :)

Thanks for looking into it.

[SAPikachu]

I can't really handle it in my code (except for forcibly terminating the slave process itself on exit, but I don't want to do that since some plugin can't properly clean up in that case). Maybe you can try another version of QuickTime, or just use ffms2?

[kolak]

Already tried some other ProRes decoder, will try older QT, but for now ffvideo is the only one solution (I don't like it that much- bit unpredictable also)

[aldix]

Hello,

latest adopter of MP_Pipeline here.

As I was suggested, I copied all the x86 files into avs plugins directory (using SET's 2.6 MT), then made the following script:

MP_Pipeline("""
f3kdb(sample_mode=2,dynamic_grain=false,keep_tv_range=false,dither_algo=3,y=48,cb=24,cr=24,grainY=48,grainC=24)
### ###
BlindDehalo3(rx=2.5, ry=2.5, strength=125)
### lock threads to cores
""")

But for the life of me, I can't get it to work. All it gives me is this error msg:

MP_Pipeline: Unable to create slave process. Message: Script Error: Script Error: Invalid arguments to function 'f3kdb'

But everything works when I remove MP stuff. What am I doing wrong?

Thanks!

[SAPikachu]

Hello,

latest adopter of MP_Pipeline here.

As I was suggested, I copied all the x86 files into avs plugins directory (using SET's 2.6 MT), then made the following script:

MP_Pipeline("""
f3kdb(sample_mode=2,dynamic_grain=false,keep_tv_range=false,dither_algo=3,y=48,cb=24,cr=24,grainY=48,grainC=24)
### ###
BlindDehalo3(rx=2.5, ry=2.5, strength=125)
### lock threads to cores
""")

But for the life of me, I can't get it to work. All it gives me is this error msg:

MP_Pipeline: Unable to create slave process. Message: Script Error: Script Error: Invalid arguments to function 'f3kdb'

But everything works when I remove MP stuff. What am I doing wrong?

Thanks!

Did you put your source filter before MP_Pipeline? MPP can't use video source from the outside, so you need to put all source filter inside it. Like this:



MP_Pipeline("""

FFVideoSource("abcde.mkv")

f3kdb(sample_mode=2,dynamic_grain=false,keep_tv_range=false,dither_algo=3,y=48,cb=24,cr=24,grainY=48,grainC=24)
### ###
BlindDehalo3(rx=2.5, ry=2.5, strength=125)
### lock threads to cores

# By the way, you need to put another block splitter here,
# because all special statement won't be effective for code
# after the last splitter.
### ###

""")

[aldix]

Yup, figured out the source filter bit meanwhile.

But now it's saying that there's no function named BlindDehalo3 even if I'd call it explicitly via import after source filter.

Interestingly, f3kdb now works, though.

Thanks a lot for the reply btw, not only directly from the author him/herself, but so promptly indeed :)

[SAPikachu]

Yup, figured out the source filter bit meanwhile.

But now it's saying that there's no function named BlindDehalo3 even if I'd call it explicitly via import after source filter.

Interestingly, f3kdb now works, though.

Thanks a lot for the reply btw, not only directly from the author him/herself, but so promptly indeed :)

Manually-loaded plugins and imported scripts need to be reloaded/re-imported before they can be used in new process. (Or use inherited script snippet, please see MP_Pipeline_readme.avs for details)

Actually, it will be much easier to put all plugins/imported scripts into your avisynth plugins folder, so that they will be loaded automatically.

[aldix]

Well, they all are in the plugins folder, but I still have to load them manually...


I don't know what to say. Even if I wrap the MP_Pipeline around all the script, with breaks (### ###) between parts,
I still get the same msg about invalid arguments.

For instance, let's take this one:


import("c:\program files (x86)\avisynth 2.5\plugins\gradfun2dbmod.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\ylevels.avs")
import("C:\Program Files (x86)\AviSynth 2.5\plugins\ContraSharpen.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\lsfmod1.9.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\sbr.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\dfttestmc.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\HQDering.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\BlindDeHalo3_mt2.avs")
import("c:\program files (x86)\avisynth 2.5\plugins\Minblur.avs")

source = last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5

preNR = source.degrainmedian(mode=3,limity=8,limituv=10).fft3dfilter(wintype=1,degrid=1,bw=32,bh=32,ow=16,oh=16,bt=3,sigma=2.5,sigma2=2.2,sigma3=1.8,sigma4=0.5,plane=4,ncpu=1).GradFun2DB(1.01)
preNR = source.degrainmedian(mode=3,limity=8,limituv=10).dfttest(sigma=6, ftype=1, tbsize=1, threads=1).GradFun2DB(1.01)
preNR_super = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma)
source_super = source.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, chroma=chroma, levels=1)
Recalculate = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma, levels=1)
vb2 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=2, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr2 = MRecalculate(Recalculate, vb2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vb1 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr1 = MRecalculate(Recalculate, vb1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf1 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr1 = MRecalculate(Recalculate, vf1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf2 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr2 = MRecalculate(Recalculate, vf2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
maskp1 = MMask(vfr1, kind=1, ysc=255).UtoY()
maskp2 = MMask(vfr2, kind=1).UtoY()
maskp3 = MMask(vbr1, kind=1, ysc=255).UtoY()
maskp4 = MMask(vbr2, kind=1).UtoY()
tmask = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)
source2 = mt_merge(source,preNR,tmask,Y=3,U=3,V=3)
KEEP = "0.23"
den = source2.MDegrain2(source_super,vbr1,vfr1,vbr2,vfr2,thSAD=thSAD,thSCD1=256,thSCD2=92)
\.mt_adddiff(mt_makediff(source,preNR,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)

# PROTECTING
threshold = 16
cutoff = 64
maxdiff = 4
rg17 = den.removegrain(17,-1)
iOB = source.mt_lut("x "+string(cutoff)+" >= x 0 ?",U=1,V=1)
mB = mt_makediff(iOB,rg17,U=1,V=1).mt_binarize(128+threshold,upper=false,U=1,V=1).removegrain(5,-1)
lB = mt_lutxy(den,source,"x y - abs "+string(maxdiff)+" <= x x y - 0 < y "+string(maxdiff)+" - x ? ?",U=1,V=1)
smB = mt_merge(den,lB,mB,U=2,V=2)

# EDGECLEANING
mP = mt_edge(smB,"prewitt",0,255,0,0,V=1,U=1)
mS = mP.mt_expand(mode=mt_square(radius=2),U=1,V=1).mt_inflate(U=1,V=1)
mD = mt_lutxy(mS,mP.mt_inflate(U=1,V=1),"x y - "+string(32)+" <= 0 x y - ?",U=1,V=1).mt_inflate(U=1,V=1).removegrain(20,-1)
smE = mt_merge(smB,Eval("smB." + "Removegrain(2,0)"),mD,luma=true,U=3,V=3)


# MASKING
mE = mt_edge(smE,"prewitt",0,255,0,0,V=1,U=1).mt_lut(expr="x 1.8 ^",U=1,V=1).removegrain(4,-1).mt_inflate(U=1,V=1)
mL = mt_logic(tmask.invert(),mE,"min",U=1,V=1).removegrain(20,-1)
mF = mt_logic(tmask,mE,"max",U=1,V=1).removegrain(20,-1)


# SHARPENING
b1c = source.MCompensate(source_super,vb1)
f1c = source.MCompensate(source_super,vf1)
#Sclp = smE.LSFmod(defaults="slow", preblur="ON", strength=100)
Sclp = ContraSharpen(smE,source)
Tmax = source.mt_logic(f1c,"max",U=1,V=1).mt_logic(b1c,"max",U=1,V=1)
Tmin = source.mt_logic(f1c,"min",U=1,V=1).mt_logic(b1c,"min",U=1,V=1)
shrp = Sclp.mt_clamp(Tmax, Tmin, 2, 2, U=1, V=1)
sL = mt_merge(smE,shrp,mL,U=2,V=2)

# ENHANCING
#GFc = sL.f3kdb(sample_mode=2,precision_mode=3)
GFc = sL.GradFun2DBmod(thr=1.4,thrC=1.8,mode=2,str=0.8,strC=0.0,temp=50,adapt=64)
Frs = mt_merge(GFc,sL,mF,luma=true,U=3,V=3).BlindDehalo3(rx=1.25, ry=1.25, strength=90, sharpness=1, ppmode=2)
Frs#.mergechroma(den)
YlevelsS(0,1.0,255,0,255,false)


In the above script, if I'd wrap MP_Pipeline around it (before imports and after ylevels), breaks on diff parts of the script,
it comes back with "invalid arguments" on degrainmedian.

I don't know what else to try. I'm sure it's something minor I'm missing as usual.

[real.finder]

the .avs is not auto load, change it to .avsi for each Script

[aldix]

the .avs is not auto load, change it to .avsi for each Script


Excellent information, real.finder! Greatly obliged. Learn something new every day :)

But this still doesn't change my primary concern - invalid arguments.

[real.finder]

Excellent information, real.finder! Greatly obliged. Learn something new every day :)

But this still doesn't change my primary concern - invalid arguments.

put the full script (with MP_Pipeline) to see what the problem

[aldix]

Apologies for late reply, too busy.

It's basically just the script above, with MP stuff inserted. Anyway, however I place the MP calls, it comes back with some filter having 'invalid arguments'.


MP_Pipeline("""
MPEG2Source("C:\Users\redacted\smileys people ep3\VTS_02_1.d2v", cpu=0)
Crop(2, 0, -2, -0)
source = last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5

preNR = source.degrainmedian(mode=3,limity=8,limituv=10).fft3dfilter(wintype=1,degrid=1,bw=32,bh=32,ow=16,oh=16,bt=3,sigma=2.5,sigma2=2.2,sigma3=1.8,sigma4=0.5,plane=4,ncpu=1).GradFun2DB(1.01)
### ###
preNR = source.degrainmedian(mode=3,limity=8,limituv=10).dfttest(sigma=6, ftype=1, tbsize=1, threads=1).GradFun2DB(1.01)
### ###
preNR_super = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma)
source_super = source.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, chroma=chroma, levels=1)
Recalculate = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma, levels=1)
vb2 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=2, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr2 = MRecalculate(Recalculate, vb2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vb1 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr1 = MRecalculate(Recalculate, vb1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf1 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr1 = MRecalculate(Recalculate, vf1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf2 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr2 = MRecalculate(Recalculate, vf2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
maskp1 = MMask(vfr1, kind=1, ysc=255).UtoY()
maskp2 = MMask(vfr2, kind=1).UtoY()
maskp3 = MMask(vbr1, kind=1, ysc=255).UtoY()
maskp4 = MMask(vbr2, kind=1).UtoY()
tmask = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)
source2 = mt_merge(source,preNR,tmask,Y=3,U=3,V=3)
KEEP = "0.23"
den = source2.MDegrain2(source_super,vbr1,vfr1,vbr2,vfr2,thSAD=thSAD,thSCD1=256,thSCD2=92)
\.mt_adddiff(mt_makediff(source,preNR,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)
### ###
# PROTECTING
threshold = 16
cutoff = 64
maxdiff = 4
rg17 = den.removegrain(17,-1)
iOB = source.mt_lut("x "+string(cutoff)+" >= x 0 ?",U=1,V=1)
mB = mt_makediff(iOB,rg17,U=1,V=1).mt_binarize(128+threshold,upper=false,U=1,V=1).removegrain(5,-1)
lB = mt_lutxy(den,source,"x y - abs "+string(maxdiff)+" <= x x y - 0 < y "+string(maxdiff)+" - x ? ?",U=1,V=1)
smB = mt_merge(den,lB,mB,U=2,V=2)
### ###
# EDGECLEANING
mP = mt_edge(smB,"prewitt",0,255,0,0,V=1,U=1)
mS = mP.mt_expand(mode=mt_square(radius=2),U=1,V=1).mt_inflate(U=1,V=1)
mD = mt_lutxy(mS,mP.mt_inflate(U=1,V=1),"x y - "+string(32)+" <= 0 x y - ?",U=1,V=1).mt_inflate(U=1,V=1).removegrain(20,-1)
smE = mt_merge(smB,Eval("smB." + "Removegrain(2,0)"),mD,luma=true,U=3,V=3)

### ###
# MASKING
mE = mt_edge(smE,"prewitt",0,255,0,0,V=1,U=1).mt_lut(expr="x 1.8 ^",U=1,V=1).removegrain(4,-1).mt_inflate(U=1,V=1)
mL = mt_logic(tmask.invert(),mE,"min",U=1,V=1).removegrain(20,-1)
mF = mt_logic(tmask,mE,"max",U=1,V=1).removegrain(20,-1)

### ###
# SHARPENING
b1c = source.MCompensate(source_super,vb1)
f1c = source.MCompensate(source_super,vf1)
#Sclp = smE.LSFmod(defaults="slow", preblur="ON", strength=100)
Sclp = ContraSharpen(smE,source)
Tmax = source.mt_logic(f1c,"max",U=1,V=1).mt_logic(b1c,"max",U=1,V=1)
Tmin = source.mt_logic(f1c,"min",U=1,V=1).mt_logic(b1c,"min",U=1,V=1)
shrp = Sclp.mt_clamp(Tmax, Tmin, 2, 2, U=1, V=1)
sL = mt_merge(smE,shrp,mL,U=2,V=2)
### ###
# ENHANCING
#GFc = sL.f3kdb(sample_mode=2,precision_mode=3)
GFc = sL.GradFun2DBmod(thr=1.4,thrC=1.8,mode=2,str=0.8,strC=0.0,temp=50,adapt=64)
Frs = mt_merge(GFc,sL,mF,luma=true,U=3,V=3).BlindDehalo3(rx=1.25, ry=1.25, strength=90, sharpness=1, ppmode=2)
Frs#.mergechroma(den)
### ###
YlevelsS(0,1.0,255,0,255,false)
### ###
""")



edit: Sorry, forgot to add source filter at first, fixed.

[real.finder]

what is the source and video source filter?

I don't see one in your script

[SAPikachu]

Variables won't be automatically transferred to next script block. For clip variables you need "### export clip" to use it in next block, for others you need "### inherit". Please see the example script in the package for details, because I am on vacation now, I can't provide another example for you.

[aldix]

MP_Pipeline("""
MPEG2Source("C:\VTS_02_1.d2v", cpu=0)
Crop(2, 0, -2, -0)
### export clip: source,overlap,blksize,hpad,vpad,thSAD,halfblksize,halfoverlap,halfthSAD,chroma,search
source=last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5
....


Now I'm getting 'Unable to create slave process ... Invalid arguments to function 'MPP_PrepareDownStreamClip'.

I really, really appreciate all the help You've given so far SAPIkachu, I just can't seem to wrap my head around it,
scarce script example isn't much use to me, I'm afraid.


Thank You again.

[SAPikachu]

Actually, you should use "export clip" like this:


source = ...
### export clip: source


Edit: I was wrong about what error the script have. See my new post below. Sorry.

[aldix]

Actually, you should use "export clip" like this:


source = ...
### export clip: source



Well, for this I followed the example script/tutorial. There it was said to list 'em separated by commas.

And regardless, now it whines for other variables, like hpad.

I don't know. Perhaps I just shouldn't try to use it or something.

[real.finder]

you can use the mp_pipleline without split by now, until you learn all its settings

like:


MP_Pipeline("""
### platform: win32
SetMemoryMax(1500)
MPEG2Source("C:\Users\redacted\smileys people ep3\VTS_02_1.d2v", cpu=0)
Crop(2, 0, -2, -0)
source = last
blksize = 16
overlap = blksize/2
hpad = blksize
vpad = blksize
thSAD = 300
halfblksize = blksize/2
halfoverlap = overlap/2
halfthSAD = thSAD/2
chroma = true
search = 5

preNR = source.degrainmedian(mode=3,limity=8,limituv=10).fft3dfilter(wintype=1,degrid=1,bw=32,bh=32,ow=16,oh=16,bt=3,sigma=2.5,sigma2=2.2,sigma3=1.8,sigma4=0.5,plane=4,ncpu=1).GradFun2DB(1.01)
preNR = source.degrainmedian(mode=3,limity=8,limituv=10).dfttest(sigma=6, ftype=1, tbsize=1, threads=1).GradFun2DB(1.01)
preNR_super = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma)
source_super = source.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, chroma=chroma, levels=1)
Recalculate = preNR.MSuper(hpad=hpad, vpad=vpad, pel=2, sharp=2, rfilter=2, chroma=chroma, levels=1)
vb2 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=2, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr2 = MRecalculate(Recalculate, vb2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vb1 = MAnalyse(preNR_super, isb=true, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vbr1 = MRecalculate(Recalculate, vb1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf1 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr1 = MRecalculate(Recalculate, vf1, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
vf2 = MAnalyse(preNR_super,isb=false, truemotion=false, delta=1, blksize=blksize, overlap=overlap, search=search, chroma=chroma)
vfr2 = MRecalculate(Recalculate, vf2, overlap=halfoverlap, blksize=halfblksize, thSAD=halfthSAD, search=search, chroma=chroma)
maskp1 = MMask(vfr1, kind=1, ysc=255).UtoY()
maskp2 = MMask(vfr2, kind=1).UtoY()
maskp3 = MMask(vbr1, kind=1, ysc=255).UtoY()
maskp4 = MMask(vbr2, kind=1).UtoY()
tmask = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)
source2 = mt_merge(source,preNR,tmask,Y=3,U=3,V=3)
KEEP = "0.23"
den = source2.MDegrain2(source_super,vbr1,vfr1,vbr2,vfr2,thSAD=thSAD,thSCD1=256,thSCD2=92)
\.mt_adddiff(mt_makediff(source,preNR,U=3,V=3).mt_lut("x 128 - abs 1 < x x 128 - abs 1 - "+KEEP+" * x 128 - x 128 - abs 0.001 + / * 128 + ?",U=2,V=2),U=3,V=3)

# PROTECTING
threshold = 16
cutoff = 64
maxdiff = 4
rg17 = den.removegrain(17,-1)
iOB = source.mt_lut("x "+string(cutoff)+" >= x 0 ?",U=1,V=1)
mB = mt_makediff(iOB,rg17,U=1,V=1).mt_binarize(128+threshold,upper=false,U=1,V=1).removegrain(5,-1)
lB = mt_lutxy(den,source,"x y - abs "+string(maxdiff)+" <= x x y - 0 < y "+string(maxdiff)+" - x ? ?",U=1,V=1)
smB = mt_merge(den,lB,mB,U=2,V=2)

# EDGECLEANING
mP = mt_edge(smB,"prewitt",0,255,0,0,V=1,U=1)
mS = mP.mt_expand(mode=mt_square(radius=2),U=1,V=1).mt_inflate(U=1,V=1)
mD = mt_lutxy(mS,mP.mt_inflate(U=1,V=1),"x y - "+string(32)+" <= 0 x y - ?",U=1,V=1).mt_inflate(U=1,V=1).removegrain(20,-1)
smE = mt_merge(smB,Eval("smB." + "Removegrain(2,0)"),mD,luma=true,U=3,V=3)


# MASKING
mE = mt_edge(smE,"prewitt",0,255,0,0,V=1,U=1).mt_lut(expr="x 1.8 ^",U=1,V=1).removegrain(4,-1).mt_inflate(U=1,V=1)
mL = mt_logic(tmask.invert(),mE,"min",U=1,V=1).removegrain(20,-1)
mF = mt_logic(tmask,mE,"max",U=1,V=1).removegrain(20,-1)


# SHARPENING
b1c = source.MCompensate(source_super,vb1)
f1c = source.MCompensate(source_super,vf1)
#Sclp = smE.LSFmod(defaults="slow", preblur="ON", strength=100)
Sclp = ContraSharpen(smE,source)
Tmax = source.mt_logic(f1c,"max",U=1,V=1).mt_logic(b1c,"max",U=1,V=1)
Tmin = source.mt_logic(f1c,"min",U=1,V=1).mt_logic(b1c,"min",U=1,V=1)
shrp = Sclp.mt_clamp(Tmax, Tmin, 2, 2, U=1, V=1)
sL = mt_merge(smE,shrp,mL,U=2,V=2)

# ENHANCING
#GFc = sL.f3kdb(sample_mode=2,precision_mode=3)
GFc = sL.GradFun2DBmod(thr=1.4,thrC=1.8,mode=2,str=0.8,strC=0.0,temp=50,adapt=64)
Frs = mt_merge(GFc,sL,mF,luma=true,U=3,V=3).BlindDehalo3(rx=1.25, ry=1.25, strength=90, sharpness=1, ppmode=2)
Frs#.mergechroma(den)
YlevelsS(0,1.0,255,0,255,false)
### lock threads to cores
### prefetch: 16, 12
### ###
""")
AssumeFrameBased


as I say here http://forum.doom9.org/showthread.php?p=1629295#post1629295

[aldix]

Well, thanks for this. Now I got it to work and going through the frames in AvspMod is very fast. However, encoding in x264 (with Simple Launcher GUI) first
hangs with 'potential deadlock', then nonetheless starts but very slowly and without using 100% cpu. Just sits at 40-50%. Memory meanwhile is 70%+.
What's up with that?

I did some investigating on my own and discovered that by removing 'lock threads to cores' statement I can get some speed back - 0,5fps and about 10% cpu load,
but that's it.

At least I'm glad that there was some little thing I missed and everything works for me. It's just not what I expected...

[SAPikachu]

Well, for this I followed the example script/tutorial. There it was said to list 'em separated by commas.

And regardless, now it whines for other variables, like hpad.

I don't know. Perhaps I just shouldn't try to use it or something.

Sorry I misread your script. Your syntax was right, but ### export clip only works for clip variables. For other variables, you need to use ### inherit start / ### inherit end. (Or just copy all the variables to all script blocks)

"lock threads to cores" is an advanced function, don't use it unless you really know how it works. Actually nearly no one told me that it is useful.

[aldix]

Well, I don't know what to tell you guys. I've tried both with and without ### inherit start/end wrapping the variables, and with lowering/rising the setMemoryMax.
I'm only getting ~1 fps on a crf encode (once it finally starts to encode after the deadlock msg) on a script that used to give 1.6 fps with just a single
SetMTMode(2)/SetMemoryMax(640) pair inserted in the middle. Now that doesn't work (x264 deadlocks eventually and cpu load doesn't achieve 100%)
and this pipeline doesn't appear to increase any speed at all.

If such set-up is just to make big scripts run with less load on cpu (haven't yet seen cpu at 100% with this) while increasing the memory load, it's fine. Currently I just need to script to run, period, don't really
care how long it takes. But I was getting giddy under the (false?) impression that this set-up actually speeds things along? Was I wrong or still doing something incorrectly?

I really, really appreciate all the replies and help. It's just frustrating.

[real.finder]

for me using "lock threads to cores" make the process faster

and mp_pipeline very useful if you know how to use it correctly, Each script has a special case to deal with, and must be conduct experiments, and measurement speed by avs meter (http://forum.doom9.org/showthread.php?t=165528)

some time I use analysis pass in avspmod to measurement speed

Good luck

[aldix]

and mp_pipeline very useful if you know how to use it correctly


Yes, thank you. That's a really useful statement given the current matters.

edit: Evidently cpu's non-100% load is the problem here. I'm just trying a different script which went at 11fps on
1st pass and now it runs at 3.5fps with 40% cpu load.

[SAPikachu]

Well, I don't know what to tell you guys. I've tried both with and without ### inherit start/end wrapping the variables, and with lowering/rising the setMemoryMax.
I'm only getting ~1 fps on a crf encode (once it finally starts to encode after the deadlock msg) on a script that used to give 1.6 fps with just a single
SetMTMode(2)/SetMemoryMax(640) pair inserted in the middle. Now that doesn't work (x264 deadlocks eventually and cpu load doesn't achieve 100%)
and this pipeline doesn't appear to increase any speed at all.

If such set-up is just to make big scripts run with less load on cpu (haven't yet seen cpu at 100% with this) while increasing the memory load, it's fine. Currently I just need to script to run, period, don't really
care how long it takes. But I was getting giddy under the (false?) impression that this set-up actually speeds things along? Was I wrong or still doing something incorrectly?

I really, really appreciate all the replies and help. It's just frustrating.

Well, it was a good thing that your script could run without error, this meant you got syntax correct and you can start to tweak the script.

I noticed you put many splitters (### ###) into your script, if you don't use prefetching it would be better to reduce count of splitters to roughly number of cpu cores you have. If you add too many splitters it may slow down your script. Then you can try moving splitters around to see if it can improve your speed.

[pbristow]

I don't seem to be able to utilize my CPU fully


[SIGH] This is false goal. Forget it.

The purpose of multi-tasking is to *speed up* the task you're working on, not to make your CPU as busy as possible. The only way you will ever get 100% utilisation on a multicore CPU is with extremely unusual (and probably artificial) workloads. Instead, the question to focus on is "how much time am I saving?"

Here's a rough guide as to when to say "Woot!" rather than "Hmph! Well, that's better than nothing I suppose..." when using multi-tasking, versus the number of cores/threads you've used:

Using 2: Execution time drops 30%. (fps increases 40%)
Using 3: Execution time drops 42%. (fps increases 73%)
Using 4: Execution time drops 50%. (fps increases 100%)
Using 6: Execution time drops 59%. (fps increases 144%)
Using 8: Execution time drops 66%. (fps increases 183%)


...In other words, don't expect a performance improvement much beyond the square root of the number of cores/processors/threads employed.

:)

[pbristow]

...and, the post I'm replying to has already been deleted. [FACEPALM]

Perhaps I should make a couple of new rules for myself:
(1) Don't reply to anything until it's at least a day old;
(2) don't reply to anything without first updating the page six times to make sure no one else has already covered it! :stupid:.

[aldix]

The only use for me that I can see is that big scripts wrapped into MP don't deadlock x264 anymore, even if not on full load at 2nd pass. Though that also was the case *before* all my recent MT troubles began (after updating to latest v).

So I don't really know what the take-away here is. Everything is just as slow for me as it previously was :)

[zerowalker]

pbristow, i solved it, that's why i deleted it:)

[zerowalker]

I am trying to get MP Pipeline to work with MCTD and also work with audio.

So that it can work with Megui, making the audio bypass the MT Pipeline and be directly encoded.
And the Video to be worked and processed in MCTD with MT Pipeline.

But sadly, it doesn´t seem to work:(

[SAPikachu]

I am trying to get MP Pipeline to work with MCTD and also work with audio.

So that it can work with Megui, making the audio bypass the MT Pipeline and be directly encoded.
And the Video to be worked and processed in MCTD with MT Pipeline.

But sadly, it doesn´t seem to work:(

Sorry, I forgot to note that MP_Pipeline doesn't support audio, you need to encode audio without using MP_Pipeline. OP is just updated to include this limitation.

Maybe you can demux the audio and directly encode it, or simply comment out MP_Pipeline for your audio script. If you want to use the same script for both audio and video encoding, you can try AudioDub after MP_Pipeline.

[zerowalker]

Audiodub seems to work, for example:

u=Avisource("")
MP_Pipeline("""
SetMemoryMax(3072)
Avisource("")
MCTD(settings="low")
### prefetch: 16, 0

### ###
""")
AudioDub(u)


So Thanks:)


Though, i wonder is there a way to prevent MP to load when audio decoding?
Cause currently, it seems to load MP which takes a very long time with more branches, and then they just close, and after all that, it starts processing.

[SAPikachu]

Audiodub seems to work, for example:

u=Avisource("")
MP_Pipeline("""
SetMemoryMax(3072)
Avisource("")
MCTD(settings="low")
### prefetch: 16, 0

### ###
""")
AudioDub(u)


So Thanks:)


Though, i wonder is there a way to prevent MP to load when audio decoding?
Cause currently, it seems to load MP which takes a very long time with more branches, and then they just close, and after all that, it starts processing.

I think it is not possible with single script, you can only do that with separated script file.

I still think that you should demux your source audio and directly encode or mux it to the output, there is no reason to pass it through AviSynth unless you want to do some post-processing to it.

[aldix]

From what does it depend whether the script used inside MP_ gets 100% cpu load or not? Some long scripts of mine get 35-40% cpu load whereas short ones get 100%, I see no rhyme or reason to this. Can you explain, Sapi?

[SAPikachu]

From what does it depend whether the script used inside MP_ gets 100% cpu load or not? Some long scripts of mine get 35-40% cpu load whereas short ones get 100%, I see no rhyme or reason to this. Can you explain, Sapi?

Please remember that CPU usage != speed as what @pbristow said, you need to compare your encoding speed, not CPU utilization.

CPU utilization mainly depends on whether works are fairly distributed across all script blocks, and how good is your prefetch setting. In other words, you need to make all the script blocks consume roughly same amount of CPU.

[aldix]

Well, I like smaller cpu load very much, it enables me to do other tasks. That's why I asked. I don't really understand prefetch, so I'm pretty much setting it at random, sometimes with what's suggested in the example script, sometimes way higher. I don't know...

[zerowalker]

Okay, i think the audio goes pretty well now. Not sure why i went so slow before, must have done something wrong.

But i have another question.


If i want to use MCTD, and run it in parallel in 4 threads on the same clip, how can i do that?
I am thinking, that i should have 4 separate script which are identical, Except for the duration, meaning i have Trimmed the clip differently.
And than later i can just suit them up.

Though i have never done something like that so i am not absolutely sure of it.

Or is it possible to have 1 script, with 4 branches, that each process a different "trim()" and MCTD?

[SAPikachu]

Well, I like smaller cpu load very much, it enables me to do other tasks. That's why I asked. I don't really understand prefetch, so I'm pretty much setting it at random, sometimes with what's suggested in the example script, sometimes way higher. I don't know...

Well you can set priority of your encoding-related process to low, then your normal tasks won't be (much) affected.

### prefetch controls how much frames ahead should be preloaded in background, and how many frames behind should be cached.

Here is a simplified illustration of effect of prefetching:

Without prefetching:

T+0:
+------------------+
| Upstream block | -> Producing frame #0
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #0 from upstream
+------------------+

T+1:
+------------------+
| Upstream block | -> Wait for downstream block to complete processing frame #0
+------------------+

+------------------+
| Downstream block | -> Processing frame #0
+------------------+

T+2:
+------------------+
| Upstream block | -> Producing frame #1
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #1 from upstream
+------------------+

T+3:
+------------------+
| Upstream block | -> Wait for downstream block to complete processing frame #1
+------------------+

+------------------+
| Downstream block | -> Processing frame #1
+------------------+

---------------------------------------------------------------------------------

With prefetching:

T+0:
+------------------+
| Upstream block | -> Producing frame #0
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #0 from upstream
+------------------+

T+1:
+------------------+
| Upstream block | -> Producing frame #1
+------------------+

+------------------+
| Downstream block | -> Processing frame #0
+------------------+

T+2:
+------------------+
| Upstream block | -> Producing frame #2
+------------------+

+------------------+
| Downstream block | -> Processing frame #1
+------------------+

T+3:
+------------------+
| Upstream block | -> Producing frame #3
+------------------+

+------------------+
| Downstream block | -> Processing frame #2
+------------------+



Of course, if upstream block goes too far away from downstream block, it still needs waiting. This really need to tweaked individually for different scripts.

[SAPikachu]

Okay, i think the audio goes pretty well now. Not sure why i went so slow before, must have done something wrong.

But i have another question.


If i want to use MCTD, and run it in parallel in 4 threads on the same clip, how can i do that?
I am thinking, that i should have 4 separate script which are identical, Except for the duration, meaning i have Trimmed the clip differently.
And than later i can just suit them up.

Though i have never done something like that so i am not absolutely sure of it.

Or is it possible to have 1 script, with 4 branches, that each process a different "trim()" and MCTD?

If you have much memory, you may try branch with big block size and large prefetch setting (at least 2x bigger than branch block size). Not sure whether this works since I haven't tried that.

Splitting the script into parts and joining them after encoding may have better performance, but it may also affect rate-controlling of x264, making bitrate distribution worse.

[zerowalker]

Don´t really get how i am supposed to set it up?
And i don´t have an enormous amount of memory. Got 8gb.

[SAPikachu]

Don´t really get how i am supposed to set it up?
And i don´t have an enormous amount of memory. Got 8gb.

Something like this:

MP_Pipeline("""

XXXSource("...")
### prefetch: 512, 256

### ###

MCTD()
### branch: 4, 32
### prefetch: 64, 0

### ###

### prefetch: 64, 0

### ###

""")


Note again, I haven't tested these parameters in any ways, not sure how they perform.

Actually, if you have only 8GB of memory, I would suggest trying this:


MP_Pipeline("""

XXXSource("...")
### prefetch: 64, 32

### ###
SetMemoryMax(3072) # Set this to smaller value if your memory runs out or the slave process crashes
MCTD()
# Don't put anything else in this block
### prefetch: 32, 0

### ###

""")

[zerowalker]

Okay, will be trying the second one.
And it seems to run at 9fps with my settings, which i guess is pretty good.

Though i will have to Trim it still, as the Clip is 7 hours, the chance of it crashing is extremely high, it has happened so many times, that i have given up doing it in one run.