Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 21st March 2012, 02:10   #41  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by pbristow View Post
It turns out I had a copy of avisynth.dll sitting in the live AVIsynth *plugins* directory! MP_Pipeline was finding and using that, rather than the one in SysWOW64. After removing/renaming the one in plugins, MP_Pipeline picks up the same version as Virtualdub, and when I replaced that with the (more efficient) version I'd found in plugins, both Virtualdub and MP_Pipeline pick up the new version. So, everything is consistent now, I just need to settle on which version I should be running.

I'd suggest adding a note to your documentation for MP_Pipeline (if it isn't one there already) about the mechanism it uses to locate avisynth.dll, and the possible pitfalls if multiple versions exist on the system.

Anyway, now I've got that sorted out, I'll start testing MP_Pipeline again. I have to say, so far it's been the least troublesome multi-processing technique I've tried, especially where a mix of temporal and spatial filters are being used and/or multiple sources files need parallel processing (e.g. to compare the results of multiple test runs).

(BTW, No I wasn't using "### prefetch". Might try that next, after I've re-run the last batch of tests with my new, 100% less insane AVIsynth installation! )
MP_Pipeline depend on Windows to load avisynth.dll for it, it won't do anything special. You have a risk if you don't install avisynth.dll into system folder anyways...

So you have already found out that the performance issue was not due to the script. Just a suggestion, I think you can try using it together with ### branch and ### prefetch, that can make the script even more parallel and should have greater speed improvement.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 24th March 2012, 22:35   #42  |  Link
pbristow
Registered User
 
pbristow's Avatar
 
Join Date: Jun 2009
Location: UK
Posts: 262
Hi SAPikachu,

can you explain a bit more about how branch and prefetch function, or how they should be used?

I deduce from the MP_Pipeline_readme.avs script in the package that branch splits the video stream temporally (rather than spatially), like the traditional MT modes do, but in larger chunks (i.e. multiple frames per processing chunk). That (together with pre-fetch) should help with some temporal filters, i.e. the ones that get bogged down in duplicated effort under Set_MT_Mode: The only duplication will be around the boundaries of each chunk, so the bigger the chunk the better, yes?

If you have several "### ###" process dividers, does it make any sense to use "### prefetch" in more than one place, or does one pre-fetch instruction apply to all processes?

Continuing to test and play...
pbristow is offline   Reply With Quote
Old 25th March 2012, 00:33   #43  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by pbristow View Post
Hi SAPikachu,

can you explain a bit more about how branch and prefetch function, or how they should be used?

I deduce from the MP_Pipeline_readme.avs script in the package that branch splits the video stream temporally (rather than spatially), like the traditional MT modes do, but in larger chunks (i.e. multiple frames per processing chunk). That (together with pre-fetch) should help with some temporal filters, i.e. the ones that get bogged down in duplicated effort under Set_MT_Mode: The only duplication will be around the boundaries of each chunk, so the bigger the chunk the better, yes?

If you have several "### ###" process dividers, does it make any sense to use "### prefetch" in more than one place, or does one pre-fetch instruction apply to all processes?

Continuing to test and play...
branch basically does what you said, but normally bigger thunk may not be better, since you need to use prefetch to preload all parts parallelly, if thunk size is too big, it becomes impossible to prefetch a full thunk (due to memory constraint) before the active thunk fully consumed by downstream.

prefetch statement only applies to the containing script block, so you usually need to add it to all script blocks. Be very careful if downstream script blocks have temporal filters, if prefetch cache size is not big enough, it may cause frequent seeking and cache thrashing, the speed will be greatly decreased because of that.

Happy experimenting.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 25th March 2012, 13:54   #44  |  Link
pbristow
Registered User
 
pbristow's Avatar
 
Join Date: Jun 2009
Location: UK
Posts: 262
Just checking my understanding again:

Code:
MP_Pipeline("""
    	AviSource("P:\Testfile.avi")
	### ###
	InterFrame (FlowPath="C:\Program Files (x86)\AviSynth\Plugins\Dependencies\", Preset="Fast", NewNum=60000, NewDen=1001)
	### prefetch: 20, 4
	### ###
	TDecimate(mode=0,cycleR=2,cycle=10)
	### ###
""")
In this example, prefetch is placed at the end of the code block containing Interframe. Am I correct in assuming this will pre-fetch frames from Interframe, so that they will be ready to supply later to TDecimate? Or (less intuitively), does it mean that Interframe will receive frames that were pre-fetched from Avisource?

(P.S. Before anyone pipes up, I know that I'm using Interframe and TDecimate "the wrong way round". It's actually the best way for the job I'm doing! )

Hmm... This is turning into a usage rather than development thread. Do you want to move this discussion to the other forum?

Last edited by pbristow; 25th March 2012 at 13:57.
pbristow is offline   Reply With Quote
Old 26th March 2012, 05:53   #45  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by pbristow View Post
Just checking my understanding again:

Code:
MP_Pipeline("""
    	AviSource("P:\Testfile.avi")
	### ###
	InterFrame (FlowPath="C:\Program Files (x86)\AviSynth\Plugins\Dependencies\", Preset="Fast", NewNum=60000, NewDen=1001)
	### prefetch: 20, 4
	### ###
	TDecimate(mode=0,cycleR=2,cycle=10)
	### ###
""")
In this example, prefetch is placed at the end of the code block containing Interframe. Am I correct in assuming this will pre-fetch frames from Interframe, so that they will be ready to supply later to TDecimate? Or (less intuitively), does it mean that Interframe will receive frames that were pre-fetched from Avisource?

(P.S. Before anyone pipes up, I know that I'm using Interframe and TDecimate "the wrong way round". It's actually the best way for the job I'm doing! )

Hmm... This is turning into a usage rather than development thread. Do you want to move this discussion to the other forum?
Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.

I think it's OK to keep the discussion here, since there is few development talk now.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 26th March 2012, 16:22   #46  |  Link
pbristow
Registered User
 
pbristow's Avatar
 
Join Date: Jun 2009
Location: UK
Posts: 262
[QUOTE=SAPikachu;1567071]Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.
QUOTE]

I understand that it applies to the containing block; the question is, to which *end* of the block: The input, or the output?

Pre-fetch is something that happens at the boundary between things, y'see. So is it pre-fetching *for* the current block, from the previous, or *from* the current block, for the next?
pbristow is offline   Reply With Quote
Old 27th March 2012, 13:52   #47  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
[QUOTE=pbristow;1567133]
Quote:
Originally Posted by SAPikachu View Post
Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.
QUOTE]

I understand that it applies to the containing block; the question is, to which *end* of the block: The input, or the output?

Pre-fetch is something that happens at the boundary between things, y'see. So is it pre-fetching *for* the current block, from the previous, or *from* the current block, for the next?
Sorry, I meant prefetch *from* the current block, output from the current block is prefetched and cached by the filter, for next block to use.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 28th March 2012, 15:53   #48  |  Link
pbristow
Registered User
 
pbristow's Avatar
 
Join Date: Jun 2009
Location: UK
Posts: 262
[QUOTE=SAPikachu;1567266]
Quote:
Originally Posted by pbristow View Post

Sorry, I meant prefetch *from* the current block, output from the current block is prefetched and cached by the filter, for next block to use.
Thanks. That helps with tuning the memory usage.

One thing: Audio doesn't seem to be carried through MP_Pipeline anymore. I'm having to re-dub the audio (using AudioDub(...) ) in the main script after the MP_Pipeline call, as none comes back from MP_Pipeline. Do you get the same, or am I doing something wrong? As far as I can see none of the filters I'm using should kill the audio, and if I move the AudioDub() call inside MP_Pipeline it doesn't work, but outside it does.
pbristow is offline   Reply With Quote
Old 29th March 2012, 01:22   #49  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by pbristow View Post

Thanks. That helps with tuning the memory usage.

One thing: Audio doesn't seem to be carried through MP_Pipeline anymore. I'm having to re-dub the audio (using AudioDub(...) ) in the main script after the MP_Pipeline call, as none comes back from MP_Pipeline. Do you get the same, or am I doing something wrong? As far as I can see none of the filters I'm using should kill the audio, and if I move the AudioDub() call inside MP_Pipeline it doesn't work, but outside it does.
I forgot to note that in the readme, audio is intentionally unsupported, because it is complex to handle and may slow down the whole filter chain. Just AudioDub() after MP_Pipeline call. :P
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 30th March 2012, 00:25   #50  |  Link
pbristow
Registered User
 
pbristow's Avatar
 
Join Date: Jun 2009
Location: UK
Posts: 262
Quote:
Originally Posted by SAPikachu View Post
I forgot to note that in the readme, audio is intentionally unsupported, because it is complex to handle and may slow down the whole filter chain. Just AudioDub() after MP_Pipeline call. :P
OK, thanks. At least I know I'm not going deaf.

Overall impression: With the definite benefit from the prefetch facility (thanks for adding that! ), I'm finding it *most* useful in my comparison scripts, where I run the same basic pre-processing on several source files in parallel and then compare the results. Processing each source file via it's own slave process using MP_Pipeline and then combining the results in the main script works smoothly and efficiently every time, utilising up to 99% CPU and delivering speed gains to match. The only gotcha is memory: It's wise to use SetMemoryMax inside the MP_Pipleline call, with a parameter that won't overflow your free memory when all those parallel processes are demanding memory at once. (Formula: Take the average free memory (in Megabytes) you have when you first invoke VirtualDub (or whatever app calls AViSynth); divide that number by the number of parallel processes you're going to run; Shave off a few tens of megabytes for safety and/or uses that "SetMemoryMax" doesn't cover.)

With more linear AviSynth scripts, the picture is not so straightforward. Sometimes using MP_Pipeline gives the best speed-up; sometimes SetMTmode() does. Sometimes tweaking the prefetch and branch parameters makes all the difference; sometimes it makes virtually none. Like all multi-threading and multiprocessing techniques, it really depends on knowing where and how to use it - and that means knowing the characteristics of the filters in your script and thus how they will be affected by the various different ways of dividing up the work they do - to get the best results. And sometimes, no matter what combination of tricks (MT_Pipline, MT(), SetMTMode(), etc...) and parameters I try, I can't get above 70% CPU utilisation, and that just has to be accepted. Some bottleneck in my machine prevents enough data getting to the CPU to be processed any faster, I guess. (Time to start saving up for that new motherboard...?)

Big, big thanks to SAPikachu for a really useful plug-in.

Last edited by pbristow; 30th March 2012 at 00:29.
pbristow is offline   Reply With Quote
Old 30th March 2012, 02:25   #51  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by pbristow View Post
OK, thanks. At least I know I'm not going deaf.

Overall impression: With the definite benefit from the prefetch facility (thanks for adding that! ), I'm finding it *most* useful in my comparison scripts, where I run the same basic pre-processing on several source files in parallel and then compare the results. Processing each source file via it's own slave process using MP_Pipeline and then combining the results in the main script works smoothly and efficiently every time, utilising up to 99% CPU and delivering speed gains to match. The only gotcha is memory: It's wise to use SetMemoryMax inside the MP_Pipleline call, with a parameter that won't overflow your free memory when all those parallel processes are demanding memory at once. (Formula: Take the average free memory (in Megabytes) you have when you first invoke VirtualDub (or whatever app calls AViSynth); divide that number by the number of parallel processes you're going to run; Shave off a few tens of megabytes for safety and/or uses that "SetMemoryMax" doesn't cover.)

With more linear AviSynth scripts, the picture is not so straightforward. Sometimes using MP_Pipeline gives the best speed-up; sometimes SetMTmode() does. Sometimes tweaking the prefetch and branch parameters makes all the difference; sometimes it makes virtually none. Like all multi-threading and multiprocessing techniques, it really depends on knowing where and how to use it - and that means knowing the characteristics of the filters in your script and thus how they will be affected by the various different ways of dividing up the work they do - to get the best results. And sometimes, no matter what combination of tricks (MT_Pipline, MT(), SetMTMode(), etc...) and parameters I try, I can't get above 70% CPU utilisation, and that just has to be accepted. Some bottleneck in my machine prevents enough data getting to the CPU to be processed any faster, I guess. (Time to start saving up for that new motherboard...?)

Big, big thanks to SAPikachu for a really useful plug-in.
Thanks for your great review.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 16th May 2012, 05:28   #52  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Released 0.13. Fixed a crashing bug so it is recommended to update to this version.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 23rd December 2012, 02:26   #53  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 1,235
I have avs 2.6 mt by SEt and avs64 2.5

and I use this script

Code:
MP_Pipeline("""
### platform: win64
DGDecode_MPEG2Source("x:\xx.d2v").ThreadRequest()
SetMTMode(2)
ColorMatrix(d2v="x:\xx.d2v", threads=0)
Trim(x, xx) ++ Trim(xx, xxx) ++ Trim(xxxx, xxxxxx)
SetMTMode(6)
#~ tfm(output="matches.txt")
#~ tdecimate(mode=4,output="metrics.txt")
tfm(input="matches.txt")
tdecimate(mode=5, hybrid=2, vfrDec=1, input="metrics.txt", tfmIn="matches.txt", mkvOut="mkv-timecodesfile.txt", tcfv1=false)
Crop(x, x, -x, x)
xxxResize(1280, 720)
### ###
### platform: win32
LoadPlugin(AviSynthPluginsDir + "EEDI2 mt/EEDI2_imp.dll")
LoadPlugin(AviSynthPluginsDir + "avs26/mt_masktools-26.dll")
SetMemoryMax(1000)
SetMTMode(2)
filter1
filter2
SoraThread()
filter3
filter4
### lock threads to cores
### ###
""")
AssumeFrameBased
it gave me very good speed

MP_Pipeline is cool plugin, I can use the script in x264 64 and x264 32, and I can use it in avspmod

thank you for it

----------

edit 1: don't use MP_Pipeline in 1st pass (analysis) of tfm and tdecimate vfr (or animeivtc mode 4 with omode=2) Because it will generate empty files

Last edited by real.finder; 14th May 2013 at 18:13.
real.finder is offline   Reply With Quote
Old 8th February 2013, 23:32   #54  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,393
SAPikachu I have a problem with latest mp_pipeline. Avisynth 2.6mt and Vdub 32bit on Win 7 64bit.
When I load script to Vdub and than close it all slave processes stay active and take memory. I also think older version did not need loadPlugin....latest one does (or is it my imagination )
kolak is offline   Reply With Quote
Old 9th February 2013, 03:11   #55  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by kolak View Post
SAPikachu I have a problem with latest mp_pipeline. Avisynth 2.6mt and Vdub 32bit on Win 7 64bit.
When I load script to Vdub and than close it all slave processes stay active and take memory. I also think older version did not need loadPlugin....latest one does (or is it my imagination )
Do you mean MP_Pipeline needs to be manually loaded? If you put it into plugin autoload folder it should be automatically loaded on start. Or maybe you put it into wrong place?

About the slave process problem, I suspect avs-mt is the culprit. Can you try a regular avs build?
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 9th February 2013, 12:28   #56  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,393
I fixed autoload by re-installing avisynth.

I tried "normal" avisynth and it's the same- processes don't close.
I tried loading script to MP Classic and also the same

I think older version was fine.
kolak is offline   Reply With Quote
Old 11th February 2013, 11:28   #57  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
That's strange, it is fine on my machine. Can you try the following steps to capture a memory dump for me to debug?

1. Use this version of MP_Pipeline, and download and extract Procdump package
2. Load your script into vdub/mpc/whatever player, and close it
3. If the slave process is stuck, double-click "dump_immediately.cmd" in the Procdump package, and accept the agreement
4. A dump file (maybe very big) will be generated, please compress and upload it to any sharing site (Mediafire for example) and send it to me
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 11th February 2013, 13:32   #58  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,393
I can do it.
I also found that it happens on my laptop only- on some other PC it works fine.

Check your PM.

Last edited by kolak; 11th February 2013 at 14:15.
kolak is offline   Reply With Quote
Old 13th February 2013, 07:45   #59  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
I just checked the dump, seems the slave process was stuck inside AppleProResDecoder, can you try disabling it and try again?
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 13th February 2013, 12:53   #60  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,393
I tried reading ProRes file with ffvideo and it was fine.
Than I tried on other machine and yes- with qtinput and ProRes file it's also a problem.
I think it's qtinput+ProRes decoder- any way to solve it? Other mov formats seams to be fine and of course ProRes is the one which I'm interested the most

Thanks for looking into it.
kolak is offline   Reply With Quote
Reply

Tags
avisynth, multi-process, pipeline

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:58.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.