MP_Pipeline 0.18 - run parts of avisynth script in external processes [2014-04-06]

[aldix]

Someone above wrote that 4GB RAM is good enough for this and now 8GB is 'only'? That being said, I'd really like to know how am I supposed to calculate prefetch amounts? Based on what? I've loathed to go over 40 for first # and now you're citing amounts like 512.

As a point of reference, here's a script for 1080p HD source encoding. Currently it's running @60-70% CPU/RAM w/ ~1.2 fps (2/4 i5 650 cpu), but if I'd remove gradfun2dbmod, it gains around 0.5 fps and goes 100% cpu. Why is beyond me.

MP_Pipeline("""
### platform: win32
SetMemoryMax(750)
AVCSource("W:\00250.dga")
Crop(0, 60, -0, -60)
source = last
super = source.MSuper(pel=1, sharp=2, rfilter=2)
b1v = MAnalyse(super,isb=true, delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
f1v = MAnalyse(super,isb=false,delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
cf1 = MCompensate(super, f1v, thSCD1=200)
cb1 = MCompensate(super, b1v, thSCD1=200)
interleave(cf1, source.MDegrain1(super,b1v,f1v,thSAD=130), cb1)
Temporalsoften(1,3,3,6,2)
selectevery(3,1)
S4Mod(strength=4.25,hthr=200,hbias=-100)
Range8to16()
DitherPost(mode=7, ampo=1, ampn=0)
gradfun2dbmod(str=0.3, strC=0.0, adapt=64, temp=80)
YlevelsS(5,1.0,250,0,255,false)
### prefetch: 7, 0
### ###
""")



I also tried to divide this by using ### ### and a few SetMemoryMax calls. That only resulted in many MP processes running (as seen in task manager) and almost full memory use, but not speed increase.

What's the most perfect for me is if I could piggy-back some of the encoding load to RAM from CPU (meaning that I can live with about 80% RAM load, but I'd rather see around 50-60% CPU load when encoding.

I'm sorry for being tense but it seems like a perfect tool for me which I just don't know how to use properly...

[SAPikachu]

Okay, will be trying the second one.
And it seems to run at 9fps with my settings, which i guess is pretty good.

Though i will have to Trim it still, as the Clip is 7 hours, the chance of it crashing is extremely high, it has happened so many times, that i have given up doing it in one run.

A better way (quality-wise) is export the script to lossless video (for example, use avs2avi with HuffYUV encoder in ffdshow). If it crashes, just use trim to start from the crashed point and output to a new file. After the whole clip is exported, use AviSource to load and join all parts, then feed the resulting clip to x264. It needs plenty of disk space though (compression rate is only about 1:2 IIRC). If you don't have much space, trim and encoding in parts may be the only way.

By the way, just curious, why MCTD crashes frequently on your computer? As far as I know it is very stable and seldom crashes on my rig. Are you using AviSynth-mt?

[SAPikachu]

Someone above wrote that 4GB RAM is good enough for this and now 8GB is 'only'? That being said, I'd really like to know how am I supposed to calculate prefetch amounts? Based on what? I've loathed to go over 40 for first # and now you're citing amounts like 512.

As a point of reference, here's a script for 1080p HD source encoding. Currently it's running @60-70% CPU/RAM w/ ~1.2 fps (2/4 i5 650 cpu), but if I'd remove gradfun2dbmod, it gains around 0.5 fps and goes 100% cpu. Why is beyond me.



I also tried to divide this by using ### ### and a few SetMemoryMax calls. That only resulted in many MP processes running (as seen in task manager) and almost full memory use, but not speed increase.

What's the most perfect for me is if I could piggy-back some of the encoding load to RAM from CPU (meaning that I can live with about 80% RAM load, but I'd rather see around 50-60% CPU load when encoding.

I'm sorry for being tense but it seems like a perfect tool for me which I just don't know how to use properly...

Well, 512 is indeed not necessary for regular uses, it was there to prevent branched MCTD in downstream block from thrashing the prefetch cache. For 4-branch MCTD, 8GB RAM is not really enough, because to get best performance each MCTD instance need to have >3GB memory.

prefetch setting is largely based on experience... As a starting point, I will use 16, 0 or 8, 0 for the last script block, and doubled value for each upstream block. It look like this:


MP_Pipeline("""
...
# prefetch: 64, 0
### ###
...
# prefetch: 32, 0
### ###
...
# prefetch: 16, 0
### ###

""")

(Note this only applies to spatial filters, for temporal filters, backwards cache must also be tweaked)

Then, I will start to mess with the values to see which parameter set is the fastest.

For your script, I will use this configuration as beginning:


MP_Pipeline("""
### platform: win32
AVCSource("W:\00250.dga")
Crop(0, 60, -0, -60)

### prefetch: 64, 32
### ###
SetMemoryMax(750)
source = last
super = source.MSuper(pel=1, sharp=2, rfilter=2)
b1v = MAnalyse(super,isb=true, delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
f1v = MAnalyse(super,isb=false,delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
cf1 = MCompensate(super, f1v, thSCD1=200)
cb1 = MCompensate(super, b1v, thSCD1=200)
source.MDegrain1(super,b1v,f1v,thSAD=130)
### prefetch: 32, 16
### export clip: cf1, cb1
### ###

interleave(cf1, last, cb1)
Temporalsoften(1,3,3,6,2)
selectevery(3,1)
S4Mod(strength=4.25,hthr=200,hbias=-100)
Range8to16()
DitherPost(mode=7, ampo=1, ampn=0)
gradfun2dbmod(str=0.3, strC=0.0, adapt=64, temp=80)
YlevelsS(5,1.0,250,0,255,false)
### prefetch: 16, 0
### ###
""")


But you can't get decent speed if you want to limit its CPU usage, in fact the goal of optimizing script with MPP is make CPU busy processing the clip at all time, so that it won't waste time on idling (note that this doesn't mean 100% CPU utilization is always the best, the final indicator is still encoding fps).

[zerowalker]

A better way (quality-wise) is export the script to lossless video (for example, use avs2avi with HuffYUV encoder in ffdshow). If it crashes, just use trim to start from the crashed point and output to a new file. After the whole clip is exported, use AviSource to load and join all parts, then feed the resulting clip to x264. It needs plenty of disk space though (compression rate is only about 1:2 IIRC). If you don't have much space, trim and encoding in parts may be the only way.

By the way, just curious, why MCTD crashes frequently on your computer? As far as I know it is very stable and seldom crashes on my rig. Are you using AviSynth-mt?

That is whay i am doing, though using Lagarith, same thing though:)

Why it crashes, well i don´t know. It doesn´t really crash normally. Except, if i enable the GPU part of MCTD, it will crash if i start a game. Not always, but i think it crashed if it changed resolution or something, and that´s pretty irritating, as GPU adds about 10-20% on speed.

But then again, it´s probably not common for someone to play games while using MCTD.

Then why it crashed without GPU. I am not entirely sure i have used it without GPU, But i think at least it has crashed normally, meaning avs2avi has crashed like a normal application crash. But it has only happened 1 time and that was after, 15+ hours i think?
And that video file went mad, it had no search point, and took years to resave etc.

However, that may have been the GPU fault also, not entirely sure.

And i use the Latest Avisynth, NOT the MT versions, so i can´t do MT, except with MP Pipeline.

[aldix]

prefetch setting is largely based on experience... As a starting point, I will use 16, 0 or 8, 0 for the last script block, and doubled value for each upstream block. It look like this:


MP_Pipeline("""
...
# prefetch: 64, 0
### ###
...
# prefetch: 32, 0
### ###
...
# prefetch: 16, 0
### ###

""")

(Note this only applies to spatial filters, for temporal filters, backwards cache must also be tweaked)

Then, I will start to mess with the values to see which parameter set is the fastest.

For your script, I will use this configuration as beginning:


MP_Pipeline("""
### platform: win32
AVCSource("W:\00250.dga")
Crop(0, 60, -0, -60)

### prefetch: 64, 32
### ###
SetMemoryMax(750)
source = last
super = source.MSuper(pel=1, sharp=2, rfilter=2)
b1v = MAnalyse(super,isb=true, delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
f1v = MAnalyse(super,isb=false,delta= 1, blksize=16, blksizeV=16, overlap=8, truemotion=false, search=5, sadx264=4)
cf1 = MCompensate(super, f1v, thSCD1=200)
cb1 = MCompensate(super, b1v, thSCD1=200)
source.MDegrain1(super,b1v,f1v,thSAD=130)
### prefetch: 32, 16
### export clip: cf1, cb1
### ###

interleave(cf1, last, cb1)
Temporalsoften(1,3,3,6,2)
selectevery(3,1)
S4Mod(strength=4.25,hthr=200,hbias=-100)
Range8to16()
DitherPost(mode=7, ampo=1, ampn=0)
gradfun2dbmod(str=0.3, strC=0.0, adapt=64, temp=80)
YlevelsS(5,1.0,250,0,255,false)
### prefetch: 16, 0
### ###
""")


But you can't get decent speed if you want to limit its CPU usage, in fact the goal of optimizing script with MPP is make CPU busy processing the clip at all time, so that it won't waste time on idling (note that this doesn't mean 100% CPU utilization is always the best, the final indicator is still encoding fps).

Thanks a lot for these clarifications. I now have a somewhat better understanding how it all works. I suppose I'll experiment different setups with my scripts and see what works the best.
So 3 separate prefetch settings is good, right?

Yeah, I understand that in most cases 100% cpu load might be for the best. It's just that I'd love to encode while doing other stuff and for now, I'm just limited to encode while I sleep (in that sense, I'd love to see MPP cut down on HD encoding time - that one always goes longer than my sleep circle, heh).

[SAPikachu]

That is whay i am doing, though using Lagarith, same thing though:)

Why it crashes, well i don´t know. It doesn´t really crash normally. Except, if i enable the GPU part of MCTD, it will crash if i start a game. Not always, but i think it crashed if it changed resolution or something, and that´s pretty irritating, as GPU adds about 10-20% on speed.

But then again, it´s probably not common for someone to play games while using MCTD.

Then why it crashed without GPU. I am not entirely sure i have used it without GPU, But i think at least it has crashed normally, meaning avs2avi has crashed like a normal application crash. But it has only happened 1 time and that was after, 15+ hours i think?
And that video file went mad, it had no search point, and took years to resave etc.

However, that may have been the GPU fault also, not entirely sure.

And i use the Latest Avisynth, NOT the MT versions, so i can´t do MT, except with MP Pipeline.

Yeah using GPU is indeed instable that may be the reason. I never use GPU in MCTD that I totally forgot about that..

[SAPikachu]

Thanks a lot for these clarifications. I now have a somewhat better understanding how it all works. I suppose I'll experiment different setups with my scripts and see what works the best.
So 3 separate prefetch settings is good, right?

Yeah, I understand that in most cases 100% cpu load might be for the best. It's just that I'd love to encode while doing other stuff and for now, I'm just limited to encode while I sleep (in that sense, I'd love to see MPP cut down on HD encoding time - that one always goes longer than my sleep circle, heh).

Don't quite understand your question, prefetch setting is supposed to be different for different script block. Do you mean how many script block should be used? I think 3 should be good for your script, but I may be wrong and you should tweak it yourself.

[Stereodude]

Something like this:

MP_Pipeline("""

XXXSource("...")
### prefetch: 512, 256

### ###

MCTD()
### branch: 4, 32
### prefetch: 64, 0

### ###

### prefetch: 64, 0

### ###

""")


Note again, I haven't tested these parameters in any ways, not sure how they perform.Can you elaborate on how you chose those parameters? If you're encoding the file in 32 frame chunks what's the purpose of prefetching more than 32 frames for those chunks?

Also, if you have 4 32 frame chunks being encoded at once it seems that you wouldn't need more then 128 frames of video cached at any given time, so why the 512/256 prefetch of the source?

[SAPikachu]

Can you elaborate on how you chose those parameters? If you're encoding the file in 32 frame chunks what's the purpose of prefetching more than 32 frames for those chunks?

Also, if you have 4 32 frame chunks being encoded at once it seems that you wouldn't need more then 128 frames of video cached at any given time, so why the 512/256 prefetch of the source?

32 frames per chunk is just an arbitrary value, as what I said before, I just made this up and haven't tested it.

Prefetching 64 frames per chunk is intentional though. If you only prefetch 32 frames, the block won't start preparing next chunk before current chunk is consumed, and if your downstream block is fast enough, the pipeline may stall without consuming all the CPU time.

512/256 in source is just a safe setting, since MCTD is a temporal filter, we'd better use some more memory to ensure it won't fetch behind the cache, as that will seriously impact performance or even crash the pipeline in some cases.

[Stereodude]

32 frames per chunk is just an arbitrary value, as what I said before, I just made this up and haven't tested it.

Prefetching 64 frames per chunk is intentional though. If you only prefetch 32 frames, the block won't start preparing next chunk before current chunk is consumed, and if your downstream block is fast enough, the pipeline may stall without consuming all the CPU time.

512/256 in source is just a safe setting, since MCTD is a temporal filter, we'd better use some more memory to ensure it won't fetch behind the cache, as that will seriously impact performance or even crash the pipeline in some cases.
Thanks for the answers. I tried this, and some variations. On my system, Core i7-4770k @ 4.2gHz Win 7 x64 SP1 w/ 16GB, processing 1920x1080 with MCTemporalDenoise(settings="medium") I was able to get ~5.33FPS with 6 thunks. With 5 threads using MT I'm capped at ~3.75FPS. (Trying to increase the threads any further using MT results in a loss of performance.)

So, it's faster than using MT, but it's also very unstable. For some reason the first thread / pipeline with the source and prefetch will stall at a full "core" worth of CPU usage. The downstream threads / pipelines end sit at 0% CPU and no frames come out. I tried a variety of things like changing the prefetch parameters and thunk size, number of thunks, but nothing eliminated it. The point where it hangs moves around (without changing any settings just running the batch file again). On one run it might happen 200 frames in, another 900+ frames, and another several thousand frames in, etc. It doesn't make a whole lot of sense since the source is DGdecNV which uses the graphics card (ie: no CPU usage).

Edit: Frankly speaking MCTD seems like it's ripe for "improvements". Its output is impressive, but seems like a big fat inefficient kludge. Maybe it's a huge undertaking, but it seems like combining all the source for the various plugins the script uses into a single dll would be a good start. From there (admittedly not having looked at any of the source code) it seems like multi-threading it to at least a basic level shouldn't be that hard. The basic analysis of multiple input frames can be done simultaneously across multiple threads, once the processing decisions have been made, you can process multiple output frames simultaneously across multiple threads. Multi-threading the intermediate processing decisions that are purely sequential is obviously harder.

[SAPikachu]

Thanks for the answers. I tried this, and some variations. On my system, Core i7-4770k @ 4.2gHz Win 7 x64 SP1 w/ 16GB, processing 1920x1080 with MCTemporalDenoise(settings="medium") I was able to get ~5.33FPS with 6 thunks. With 5 threads using MT I'm capped at ~3.75FPS. (Trying to increase the threads any further using MT results in a loss of performance.)

So, it's faster than using MT, but it's also very unstable. For some reason the first thread / pipeline with the source and prefetch will stall at a full "core" worth of CPU usage. The downstream threads / pipelines end sit at 0% CPU and no frames come out. I tried a variety of things like changing the prefetch parameters and thunk size, number of thunks, but nothing eliminated it. The point where it hangs moves around (without changing any settings just running the batch file again). On one run it might happen 200 frames in, another 900+ frames, and another several thousand frames in, etc. It doesn't make a whole lot of sense since the source is DGdecNV which uses the graphics card (ie: no CPU usage).


Can you post your full script here? Not sure whether this is because a problem in your script, or a bug in MPP.


Edit: Frankly speaking MCTD seems like it's ripe for "improvements". Its output is impressive, but seems like a big fat inefficient kludge. Maybe it's a huge undertaking, but it seems like combining all the source for the various plugins the script uses into a single dll would be a good start. From there (admittedly not having looked at any of the source code) it seems like multi-threading it to at least a basic level shouldn't be that hard. The basic analysis of multiple input frames can be done simultaneously across multiple threads, once the processing decisions have been made, you can process multiple output frames simultaneously across multiple threads. Multi-threading the intermediate processing decisions that are purely sequential is obviously harder.

Well, the main problem is that many plugins are not thread-safe, so even you managed to merge the plugins, it is still very hard to parallelize it IMO.

You can also try using MVTools in dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559), it is natively MT-enabled and may increase speed a bit.

[Stereodude]

Can you post your full script here? Not sure whether this is because a problem in your script, or a bug in MPP.Sure. Here's the final variation I tried.

LoadPlugin("MP_Pipeline.dll")
LoadPlugin("flash3kyuu_deband.dll")

MP_Pipeline("""

### platform: win32
# this part of script will be run in a 32 bit process

LoadPlugin("DGDecodeNVx86.dll")
DGSource("Video_2.dgi").crop(0,18,-0,-18)

### prefetch: 512, 384

### ###

### platform: win32
# ### lock threads to cores

Import("MCTemporalDenoise.v1.4.20.avsi")

SetMemoryMax(1280)
MCTemporalDenoise(settings="medium")


### branch: 6, 32
### prefetch: 64, 0

### ###

# ### prefetch: 48, 0

# ### ###

""")
flash3kyuu_deband(range=15, grainY=32, grainC=32, sample_mode=2, dither_algo=3, mt=false).addborders(0,18,0,18)

[SAPikachu]

Sure. Here's the final variation I tried.

LoadPlugin("MP_Pipeline.dll")
LoadPlugin("flash3kyuu_deband.dll")

MP_Pipeline("""

### platform: win32
# this part of script will be run in a 32 bit process

LoadPlugin("DGDecodeNVx86.dll")
DGSource("Video_2.dgi").crop(0,18,-0,-18)

### prefetch: 512, 384

### ###

### platform: win32
# ### lock threads to cores

Import("MCTemporalDenoise.v1.4.20.avsi")

SetMemoryMax(1280)
MCTemporalDenoise(settings="medium")


### branch: 6, 32
### prefetch: 64, 0

### ###

# ### prefetch: 48, 0

# ### ###

""")
flash3kyuu_deband(range=15, grainY=32, grainC=32, sample_mode=2, dither_algo=3, mt=false).addborders(0,18,0,18)

Can you try 768,512 in the DGSource block? If that still has problem, it is likely a bug in MPP and I will look into it when I get some free time...

[Stereodude]

You can also try using MVTools in dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559), it is natively MT-enabled and may increase speed a bit.This increased CPU usage from 12.5% (1 logical core on my i7) to 20-25%. FPS went from ~1.5 to ~2.0.

Can you try 768,512 in the DGSource block? If that still has problem, it is likely a bug in MPP and I will look into it when I get some free time...I'll give it a shot later today when the machine is done with the current jobs. Right now I'm experimenting with intermediate lossless (Lagarith) AVI files. I split the source into 4 slightly overlapping chunks and output them to 4 AVI files. With the older single threaded MVTools I can process all 4 chunks simultaneously for a combined ~4.5FPS. It also avoids any temporal implications that AVIsynth MT or thunking the source can cause and I don't have the process the source twice for a 2-pass encode either.

[aldix]

Sorry much for OT, but I just have to ask.

Is DGDecodeNV worth it? I mean, does it make script/encoding faster when used instead of AVCSource/FFVideosource etc? I have GTX 460 GPU which has VP4.

Thanks!

[Stereodude]

With the older single threaded MVTools I can process all 4 chunks simultaneously for a combined ~4.5FPS.FWIW, the MT version of MVTools ups the combined throughput to lossless Lagarith AVIs for 4 simultaneous chunks to ~5.3FPS.
Is DGDecodeNV worth it? I mean, does it make script/encoding faster when used instead of AVCSource/FFVideosource etc? I have GTX 460 GPU which has VP4.I think so. CPU usage for decoding is effectively 0 which means more of your CPU's resources can encode.

[aldix]

Thanks a lot for the reply. Very intriguing. I'm going to think about it some more and then perhaps indeed purchase it. :)

[Stereodude]

Can you try 768,512 in the DGSource block? If that still has problem, it is likely a bug in MPP and I will look into it when I get some free time...Sorry, I've been meaning to post back with the results of my findings. But, I couldn't get it to lock up / crash no matter the settings I tried yesterday though I didn't let it run for hours on end. I'm not sure what's going on. I'll mess with it again later.

[SAPikachu]

Sorry, I've been meaning to post back with the results of my findings. But, I couldn't get it to lock up / crash no matter the settings I tried yesterday though I didn't let it run for hours on end. I'm not sure what's going on. I'll mess with it again later.

Thanks and no problem, just take your time. :)

[real.finder]

hi SAPikachu :)

as it is known, a lot of programs and tools (like x264 itself) can use the avs .dll that in the same folder with it and ignore the main version in syswow64 or system32

and there are programs can use internal avs .dll like megui (in tools\avs folder), or external one like avspmod

I wonder if it possible to make every Block in mp_pipeline works with a different versions of avisynth by choosing the .dll in the block

something like:-

### platform: win32(".dll path here")

or

### AVS: (".dll path here")

it will be a wonderful thing, and easy to test different versions of the AVS, such as avisynth+ and MTs ver., in both 64 or 32

and the more importantly thing is to take advantage of the features of each one of them according to need in the Blocks

----------

The second thing (is not that important) is that mp_pipeline having the same problem here (http://forum.doom9.org/showthread.php?t=145912)

the last ver. of fixed avs2yuv was here (http://forum.doom9.org/showthread.php?p=1527698#post1527698)

thanks :)

[SAPikachu]

hi SAPikachu :)

as it is known, a lot of programs and tools (like x264 itself) can use the avs .dll that in the same folder with it and ignore the main version in syswow64 or system32

and there are programs can use internal avs .dll like megui (in tools\avs folder), or external one like avspmod

I wonder if it possible to make every Block in mp_pipeline works with a different versions of avisynth by choosing the .dll in the block

something like:-

### platform: win32(".dll path here")

or

### AVS: (".dll path here")

it will be a wonderful thing, and easy to test different versions of the AVS, such as avisynth+ and MTs ver., in both 64 or 32

and the more importantly thing is to take advantage of the features of each one of them according to need in the Blocks

----------

The second thing (is not that important) is that mp_pipeline having the same problem here (http://forum.doom9.org/showthread.php?t=145912)

the last ver. of fixed avs2yuv was here (http://forum.doom9.org/showthread.php?p=1527698#post1527698)

thanks :)

1. This shouldn't be very hard, will do it in a few days.

2. Will come along with #1 :)

[real.finder]

1. This shouldn't be very hard, will do it in a few days.

2. Will come along with #1 :)

Thank you. I'm looking forward to it :)

[SAPikachu]

Thank you. I'm looking forward to it :)

0.15 released with the 2 changes.

[real.finder]

0.15 released with the 2 changes.

Finally :)

thank you so much :)

[turbojet]

Thanks for adding avisynth dll function, it works great, and surprisingly very useful currently.

[turbojet]

Is there any way to split a source into 2 parts and ivtc them in separate processes then join them at the end?

Trying something like this:
MP_Pipeline("""
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
### pass a
### ###
b=dss2("source").trim(framecount/2+1,0).tfm().tdecimate()
### pass a, b
### ###
a+b
lanczosresize(1280,720)
""")


Throws 'I don't know what "a" means'

TDecimate is the bottleneck here, with it there's only 40-50% cpu usage. If this can be done, I'll need to figure out how to leave some extra frames (50?) when possible and for motion estimation and exclude them while joining. Also make the split divisible by 5 for ivtc.

[real.finder]

try

MP_Pipeline("""
### platform: win32
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
### export clip: a
### ###
### platform: win32
b=dss2("source").trim(framecount/2+1,0).tfm().tdecimate()
### pass clip: a
### export clip: b
### ###
### platform: win32
a+b
lanczosresize(1280,720)
### ###
""")


Should work

--------


I think that mp_pipeline need a good guide to be used by more people easily :)

[turbojet]

Doesn't know what framecount means, strange.

If I change trim to numbers it throws: Invalid arguments to function "MPP_PrepareDownstreamClip"

[Gavino]

Doesn't know what framecount means, strange.
That's because there is no 'last' defined at that point.
Replace
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
by
a=dss2("source")
a=trim(0,a.framecount/2).tfm().tdecimate()
and similarly for 'b'.

[SAPikachu]

Is there any way to split a source into 2 parts and ivtc them in separate processes then join them at the end?

Trying something like this:
MP_Pipeline("""
a=dss2("source").trim(0,framecount/2).tfm().tdecimate()
### pass a
### ###
b=dss2("source").trim(framecount/2+1,0).tfm().tdecimate()
### pass a, b
### ###
a+b
lanczosresize(1280,720)
""")


Throws 'I don't know what "a" means'

TDecimate is the bottleneck here, with it there's only 40-50% cpu usage. If this can be done, I'll need to figure out how to leave some extra frames (50?) when possible and for motion estimation and exclude them while joining. Also make the split divisible by 5 for ivtc.

It doesn't make sense to split the clip in this way, since final output frame is requested sequentially, the second process will be idle before first half of the clip is processed, then the first process will become idle.

If you don't want to run 2 encoding processes, you can try ### branch, like this:


MP_Pipeline("""
# Note: I am not sure whether dss2 guarantees frame-accurate,
# if not, you need to use another source filter that is frame-accurate.
dss2("source")
tfm()
tdecimate()

# 2 branch processes, 50 frames per thunk
### branch: 2, 50
# Prefetch 100 frames (2 thunks). Reduce this and thunk size if you don't have enough memory
### prefetch: 100, 0
### ###
lanczosresize(1280,720)
""")

[turbojet]

With that script I get:

MP_Pipeline: Error while creating last part of the filter chain:
ThunkedInterleave: Frame count of all clips except clip with last thunk must be
multiples of thunk_size. (Incorrect clip: #0)

According to some things I've read DSS2 is frame-accurate which is the advantage over DSS but even in the DSS2 mod (https://code.google.com/p/xvid4psp/downloads/detail?name=DSS2%20mod%20%2B%20LAVFilters.7z&can=2&q=) readme it's written 'Since DirectShow seeking isn't frame-accurate (or time-accurate,
as there is no frames, only time)' I found a bug with DSS2 and interlaced mpeg2 videos that are over 36 minutes long. FFMS2 showed very inaccurate fps and currently trying LWLibavsource, which seems to be the fastest overall after setting dr=true.

2 sequential encodes might be what I end up doing but it's not a very big speed increase, 44-45 fps at 40-50% CPU to 54-55 fps at 100% cpu. I was expecting more considering cpu usage.

[SAPikachu]

With that script I get:

MP_Pipeline: Error while creating last part of the filter chain:
ThunkedInterleave: Frame count of all clips except clip with last thunk must be
multiples of thunk_size. (Incorrect clip: #0)


Can you post your full script? I just tried on my machine and didn't get that error.


According to some things I've read DSS2 is frame-accurate which is the advantage over DSS but even in the DSS2 mod (https://code.google.com/p/xvid4psp/downloads/detail?name=DSS2%20mod%20%2B%20LAVFilters.7z&can=2&q=) readme it's written 'Since DirectShow seeking isn't frame-accurate (or time-accurate,
as there is no frames, only time)' I found a bug with DSS2 and interlaced mpeg2 videos that are over 36 minutes long. FFMS2 showed very inaccurate fps and currently trying LWLibavsource, which seems to be the fastest overall after setting dr=true.

2 sequential encodes might be what I end up doing but it's not a very big speed increase, 44-45 fps at 40-50% CPU to 54-55 fps at 100% cpu. I was expecting more considering cpu usage.


Yes I think LWLibavsource is a better choice.

[turbojet]

MP_Pipeline("""
#lwlibavvideosource("mppthunk.mpg")
blankclip(163,1920,1080,"YV12",29.97).assumefps(30000,1001) #same as source, also errors
tfm()
tdecimate()

### branch: 2, 50
### prefetch: 100, 0
### ###
lanczosresize(1280,720)
""")

source (http://www.sendspace.com/file/0y0rx5) if you need it. Can try blankclip first.

[SAPikachu]

MP_Pipeline("""
#lwlibavvideosource("mppthunk.mpg")
blankclip(163,1920,1080,"YV12",29.97).assumefps(30000,1001) #same as source, also errors
tfm()
tdecimate()

### branch: 2, 50
### prefetch: 100, 0
### ###
lanczosresize(1280,720)
""")

source (http://www.sendspace.com/file/0y0rx5) if you need it. Can try blankclip first.

OK, I have found source of the error, it is related to length of the clip. I will fix it later when I get around, for a temporary workaround you may pad some frames after tdecimate, like:

MP_Pipeline("""
#lwlibavvideosource("mppthunk.mpg")
blankclip(163,1920,1080,"YV12",29.97).assumefps(30000,1001) #same as source, also errors
tfm()
tdecimate()
last + blankclip(last, 50)

### branch: 2, 50
### prefetch: 100, 0
### ###
trim(0, framecount-1-50)
lanczosresize(1280,720)
""")

[turbojet]

Thanks that increased the speed from 49 to 67 fps. I found out tfm is the bottleneck. Telecide().tdecimate() mpp goes from 60 to 84 fps. Mpp slows down uncomb().tdecimate() from 113 to 94 fps.

[real.finder]

hi :)

I have some reports

It seems that the avs dll function not work with old versions of avs

test it with 257 mt (http://www.avisynth.nl/users/tsp/MT_07.zip) and 257 official
will show

1 (http://i.imgur.com/thApjtc.png) then 2 (http://i.imgur.com/Sjummwq.png) and last 3 (http://i.imgur.com/DY6Z9zQ.png)

edit: the problem in plugins, it seems not compatible with older versions

edit 2: for who cares about that, the problems were in some avsi files (like dither.avsi) , because 257 does not support /* comment */

and in 258 it was because Srestore.avsi 2.7g :-

Function StrReplace(string s, string find, string replace) # Repeated, string replacements
# Original:- http://forum.doom9.org/showthread.php?t=147846&highlight=gscript By Vampiredom, Gavino, IanB


the correct for 258 is :-


# Original:- http://forum.doom9.org/showthread.php?t=147846&highlight=gscript By Vampiredom, Gavino, IanB
# Repeated, string replacements
Function StrReplace(string s, string find, string replace)
{etc...}



thanks

[steptoe]

I'm still trying to get my head around how to run 32bit plugins on a 64bit avisynth install, or just to start with running a combination of 32bit and 64bit plugins on a 32bit avisynth install if 64bit plugins are available for the filters/scripts I use. Once I get that working then I'll look into trying 64bit avisynth

It may sound dumb to some, but what simple steps are needed to get mp_pipeline running. I've tried but struggling with the basic setup


I have 32bit avisynth+ installed, running windows 7 64bit on my Intel i5-2500k at 4.5ghz stable, with 16GB memory

The scripts/plugins I use work perfectly with either AVStoDVD or DVD-RB Pro. Depending what I'm backing up

I'm looking to try and squeeze a bit more out of my system, which isn't exactly slow, but can't justify dropping in an Intel i7 just for video work

I'm mainly using RemoveDirtMC, TomsMoComp, Unsharp, AutoGain or AutoAdjust and FluxSmooth with Toon/awarpsharp2 for animation. No complex all-in-one noise scripts just to clean the source a bit with some mild sharpening of usually 70's/80's DVD sources of childrens cartoons such as DangerMouse, The Clangers, Trapdoor, BagPuss, Paddington Bear, Mr Benn, netc. That era

Thanks

[SAPikachu]

I'm still trying to get my head around how to run 32bit plugins on a 64bit avisynth install, or just to start with running a combination of 32bit and 64bit plugins on a 32bit avisynth install if 64bit plugins are available for the filters/scripts I use. Once I get that working then I'll look into trying 64bit avisynth

It may sound dumb to some, but what simple steps are needed to get mp_pipeline running. I've tried but struggling with the basic setup


I have 32bit avisynth+ installed, running windows 7 64bit on my Intel i5-2500k at 4.5ghz stable, with 16GB memory

The scripts/plugins I use work perfectly with either AVStoDVD or DVD-RB Pro. Depending what I'm backing up

I'm looking to try and squeeze a bit more out of my system, which isn't exactly slow, but can't justify dropping in an Intel i7 just for video work

I'm mainly using RemoveDirtMC, TomsMoComp, Unsharp, AutoGain or AutoAdjust and FluxSmooth with Toon/awarpsharp2 for animation. No complex all-in-one noise scripts just to clean the source a bit with some mild sharpening of usually 70's/80's DVD sources of childrens cartoons such as DangerMouse, The Clangers, Trapdoor, BagPuss, Paddington Bear, Mr Benn, netc. That era

Thanks

Even using MP_Pipeline, you need to install both x86/x64 versions of Avisynth to to able to run all x86/x64 plugins. Here is an example of script:


MP_Pipeline("""

### platform: win64

LoadPlugin("Some_x64_plugin.dll")

SomeSource()
SomeX64Filter()

### ###

### platform: win32

LoadPlugin("Some_x86_plugin.dll")

SomeX86Filter()

### ###

""")

[real.finder]

hi :)

in the 0.15 ver. when I load script in MPC or x264 (or anything) and than close it, some slave processes stay active in task manger and take memory, with 0.14 this not happened

and sometimes if used some complex script with dvd sources (480) some slave processes crashed (http://i.imgur.com/mXqSyQh.png) in closing (or end), it does not affect, but it's annoying, and that in both 0.14 or 0.15, and this sometimes not happened even with the same script

I will PM you a dump file for 0.15 now

[aldix]

Yeah, I can also report hanging slave processes.

[steptoe]

Many thanks for the reply trying to use MP-Pipeline, I didn't know I needed 64bit/32bit avisynth installed at the same time. I'll have another try at some point


Thanks again for making it much more obvious how to get it to work

[SAPikachu]

hi :)

in the 0.15 ver. when I load script in MPC or x264 (or anything) and than close it, some slave processes stay active in task manger and take memory, with 0.14 this not happened

and sometimes if used some complex script with dvd sources (480) some slave processes crashed (http://i.imgur.com/mXqSyQh.png) in closing (or end), it does not affect, but it's annoying, and that in both 0.14 or 0.15, and this sometimes not happened even with the same script

I will PM you a dump file for 0.15 now

Thanks for the dump, I guess there are some plugins have troubles when destructing in slave process. I will try to fix it next week.

[Dogway]

Hello, great plugin, very stable!

Is this right?

MP_Pipeline("""
DGSource("1080i_source.dgi")
AssumeTFF()
QTGMC(preset="faster",border=false,edithreads=1,showsettings=false)
### prefetch: 64, 32
### ###

bicubicResize(1024,576,b=-.5,c=.25)
ColorMatrix(mode="Rec.709->Rec.601",threads=1)
limiter()
### branch: 4, 8
### prefetch: 32, 16
### ###
""")

I get an average of 40% on my 4 cores, is there a way to say, limit the process to 2 cores (I noticed how load decreases when core number increases...) and use the other 2 for another encode (ie. second half of source)? For RAM's sake ideally I would like to reuse loaded calls, but I guess the only way is to open another instance of vdub?

Also on another script I had to remove the last block limiter, before closing MP_Pipeline, how does this work?

edit: also to add I can't make dither_resize16() work inside mp_pipeline, I wonder if there's a multithreading conflict there.

[SAPikachu]

(Sorry for the late reply, recently I am too busy to check here)

Hello, great plugin, very stable!

Is this right?

MP_Pipeline("""
DGSource("1080i_source.dgi")
AssumeTFF()
QTGMC(preset="faster",border=false,edithreads=1,showsettings=false)
### prefetch: 64, 32
### ###

bicubicResize(1024,576,b=-.5,c=.25)
ColorMatrix(mode="Rec.709->Rec.601",threads=1)
limiter()
### branch: 4, 8
### prefetch: 32, 16
### ###
""")

I get an average of 40% on my 4 cores, is there a way to say, limit the process to 2 cores (I noticed how load decreases when core number increases...) and use the other 2 for another encode (ie. second half of source)? For RAM's sake ideally I would like to reuse loaded calls, but I guess the only way is to open another instance of vdub?


You may need to set thread affinity in task manger for each slave process, "### lock threads to cores" didn't work well for multiple encoding pipeline at the same time.

Also I think "### branch" in the second block should be removed, since the first block has QTGMC, the second block won't be slower than first block, it doesn't make sense to branch it


Also on another script I had to remove the last block limiter, before closing MP_Pipeline, how does this work?


Sorry I don't understand this, what did you want to do?


edit: also to add I can't make dither_resize16() work inside mp_pipeline, I wonder if there's a multithreading conflict there.

Any error message?

[Dogway]

Thanks for reply.

With dither_resize16() the script is just unable to load, freezes vdub. Could you make it work before?

Thanks for scripting help, while the above script was succesfully encoded, I think I get RAM issues with more complex scripts, so I tend to avoid prefetch of 64 frames, etc. I'm currently on XP x86, and while I plan to switch to 7 x64 soon I wonder what are my options RAM wise for the below script, halving buffer and threading at the same time just doesn't seem to come along. I'm kinda stuck on what could be optimal in this scenario (using quad core i5 here), my current script is as follows:

MP_Pipeline("""
setmemorymax(1024)
DGSource("1080p.dgi",crop_t=132,crop_b=140)
### prefetch: 16, 0
### ###

bicubicResize(1280,544,b=-.5,c=.25)
# ### branch: 2, 4 # not sure about this
### prefetch: 16, 16
### ###

pre=fluxsmoothT(3)
### export clip: pre
### prefetch: 16, 8
### ###

smdegrain(tr=2,thSAD=170,prefilter=pre,lsb_out=true,refinemotion=true,contrasharp=30)
DitherPost(stacked=true,prot=false,mode=6,ampn=1,staticnoise=true)
Limiter()
### ###
""")

[SAPikachu]

Thanks for reply.

With dither_resize16() the script is just unable to load, freezes vdub. Could you make it work before?


Just tried and it works without problem in AvsP and vdub.


Thanks for scripting help, while the above script was succesfully encoded, I think I get RAM issues with more complex scripts, so I tend to avoid prefetch of 64 frames, etc. I'm currently on XP x86, and while I plan to switch to 7 x64 soon I wonder what are my options RAM wise for the below script, halving buffer and threading at the same time just doesn't seem to come along. I'm kinda stuck on what could be optimal in this scenario (using quad core i5 here), my current script is as follows:

MP_Pipeline("""
setmemorymax(1024)
DGSource("1080p.dgi",crop_t=132,crop_b=140)
### prefetch: 16, 0
### ###

bicubicResize(1280,544,b=-.5,c=.25)
# ### branch: 2, 4 # not sure about this
### prefetch: 16, 16
### ###

pre=fluxsmoothT(3)
### export clip: pre
### prefetch: 16, 8
### ###

smdegrain(tr=2,thSAD=170,prefilter=pre,lsb_out=true,refinemotion=true,contrasharp=30)
DitherPost(stacked=true,prot=false,mode=6,ampn=1,staticnoise=true)
Limiter()
### ###
""")

4GB memory is really not enough for MPP to speed up complex script, so I think you don't have many choices except for switching to x64 with more memory...

[SAPikachu]

Released 0.16. Thanks @real.finder, @aldix and @turbojet reporting the bugs!

[real.finder]

Released 0.16. Thanks @real.finder, @aldix and @turbojet reporting the bugs!

Finally :)

:thanks:

[Stereodude]

FWIW, you might want to make it clear in the readme / documentation that each slave process must return something. IE:

Will not work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

Will work:
### platform: win32
Import("QTGMC-3.32.avsi")
SetMemoryMax(2048)

clip1 = last.QTGMC( Preset="Medium", InputType=2, Sharpness=0.2, ProgSADMask=4, ShowSettings=false )
clip2 = last.QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
clip1
### export clip: clip1, clip2
### prefetch: 8, 4
### ###

I was banging my head on the wall for quite some time trying to figure out why I couldn't get export to work and kept getting this message:
Script error: Invalid arguments to function
"MPP_PrepareDownstreamClip"

BTW, thanks for the very useful plugin!

[Dogway]

Just tried and it works without problem in AvsP and vdub.
I will look into that.


4GB memory is really not enough for MPP to speed up complex script, so I think you don't have many choices except for switching to x64 with more memory...

I was hoping something more elaborated, I guess that with that I can read that my script is correct, right? despite the 16, 16, 16 prefetch values.

@Stereodude: "last" should get passed through variables, if not I wonder if "### export clip: last" could work, otherwise explicitly calling last and exporting to next core could help, since on a normal workflow you wouldn't want to output clip1 or clip2 at that early stage.