Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 25th May 2013, 09:25   #81  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
for me using "lock threads to cores" make the process faster

and mp_pipeline very useful if you know how to use it correctly, Each script has a special case to deal with, and must be conduct experiments, and measurement speed by avs meter

some time I use analysis pass in avspmod to measurement speed

Good luck
real.finder is offline   Reply With Quote
Old 25th May 2013, 16:36   #82  |  Link
aldix
Registered User
 
Join Date: Sep 2012
Posts: 156
Quote:
Originally Posted by real.finder View Post
and mp_pipeline very useful if you know how to use it correctly

Yes, thank you. That's a really useful statement given the current matters.

edit: Evidently cpu's non-100% load is the problem here. I'm just trying a different script which went at 11fps on
1st pass and now it runs at 3.5fps with 40% cpu load.

Last edited by aldix; 25th May 2013 at 16:42.
aldix is offline   Reply With Quote
Old 27th May 2013, 02:54   #83  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by aldix View Post
Well, I don't know what to tell you guys. I've tried both with and without ### inherit start/end wrapping the variables, and with lowering/rising the setMemoryMax.
I'm only getting ~1 fps on a crf encode (once it finally starts to encode after the deadlock msg) on a script that used to give 1.6 fps with just a single
SetMTMode(2)/SetMemoryMax(640) pair inserted in the middle. Now that doesn't work (x264 deadlocks eventually and cpu load doesn't achieve 100%)
and this pipeline doesn't appear to increase any speed at all.

If such set-up is just to make big scripts run with less load on cpu (haven't yet seen cpu at 100% with this) while increasing the memory load, it's fine. Currently I just need to script to run, period, don't really
care how long it takes. But I was getting giddy under the (false?) impression that this set-up actually speeds things along? Was I wrong or still doing something incorrectly?

I really, really appreciate all the replies and help. It's just frustrating.
Well, it was a good thing that your script could run without error, this meant you got syntax correct and you can start to tweak the script.

I noticed you put many splitters (### ###) into your script, if you don't use prefetching it would be better to reduce count of splitters to roughly number of cpu cores you have. If you add too many splitters it may slow down your script. Then you can try moving splitters around to see if it can improve your speed.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 27th May 2013, 18:01   #84  |  Link
pbristow
Registered User
 
pbristow's Avatar
 
Join Date: Jun 2009
Location: UK
Posts: 263
Quote:
Originally Posted by zerowalker View Post
I don't seem to be able to utilize my CPU fully

[SIGH] This is false goal. Forget it.

The purpose of multi-tasking is to *speed up* the task you're working on, not to make your CPU as busy as possible. The only way you will ever get 100% utilisation on a multicore CPU is with extremely unusual (and probably artificial) workloads. Instead, the question to focus on is "how much time am I saving?"

Here's a rough guide as to when to say "Woot!" rather than "Hmph! Well, that's better than nothing I suppose..." when using multi-tasking, versus the number of cores/threads you've used:

Using 2: Execution time drops 30%. (fps increases 40%)
Using 3: Execution time drops 42%. (fps increases 73%)
Using 4: Execution time drops 50%. (fps increases 100%)
Using 6: Execution time drops 59%. (fps increases 144%)
Using 8: Execution time drops 66%. (fps increases 183%)


...In other words, don't expect a performance improvement much beyond the square root of the number of cores/processors/threads employed.

pbristow is offline   Reply With Quote
Old 27th May 2013, 18:22   #85  |  Link
pbristow
Registered User
 
pbristow's Avatar
 
Join Date: Jun 2009
Location: UK
Posts: 263
...and, the post I'm replying to has already been deleted. [FACEPALM]

Perhaps I should make a couple of new rules for myself:
(1) Don't reply to anything until it's at least a day old;
(2) don't reply to anything without first updating the page six times to make sure no one else has already covered it! .
pbristow is offline   Reply With Quote
Old 27th May 2013, 21:18   #86  |  Link
aldix
Registered User
 
Join Date: Sep 2012
Posts: 156
The only use for me that I can see is that big scripts wrapped into MP don't deadlock x264 anymore, even if not on full load at 2nd pass. Though that also was the case *before* all my recent MT troubles began (after updating to latest v).

So I don't really know what the take-away here is. Everything is just as slow for me as it previously was
aldix is offline   Reply With Quote
Old 2nd June 2013, 20:53   #87  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,121
pbristow, i solved it, that's why i deleted it
zerowalker is offline   Reply With Quote
Old 11th June 2013, 03:37   #88  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,121
I am trying to get MP Pipeline to work with MCTD and also work with audio.

So that it can work with Megui, making the audio bypass the MT Pipeline and be directly encoded.
And the Video to be worked and processed in MCTD with MT Pipeline.

But sadly, it doesnīt seem to work
zerowalker is offline   Reply With Quote
Old 12th June 2013, 13:10   #89  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by zerowalker View Post
I am trying to get MP Pipeline to work with MCTD and also work with audio.

So that it can work with Megui, making the audio bypass the MT Pipeline and be directly encoded.
And the Video to be worked and processed in MCTD with MT Pipeline.

But sadly, it doesnīt seem to work
Sorry, I forgot to note that MP_Pipeline doesn't support audio, you need to encode audio without using MP_Pipeline. OP is just updated to include this limitation.

Maybe you can demux the audio and directly encode it, or simply comment out MP_Pipeline for your audio script. If you want to use the same script for both audio and video encoding, you can try AudioDub after MP_Pipeline.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 13th June 2013, 13:12   #90  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,121
Audiodub seems to work, for example:

u=Avisource("")
MP_Pipeline("""
SetMemoryMax(3072)
Avisource("")
MCTD(settings="low")
### prefetch: 16, 0

### ###
""")
AudioDub(u)


So Thanks


Though, i wonder is there a way to prevent MP to load when audio decoding?
Cause currently, it seems to load MP which takes a very long time with more branches, and then they just close, and after all that, it starts processing.

Last edited by zerowalker; 13th June 2013 at 13:16.
zerowalker is offline   Reply With Quote
Old 14th June 2013, 01:29   #91  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by zerowalker View Post
Audiodub seems to work, for example:

u=Avisource("")
MP_Pipeline("""
SetMemoryMax(3072)
Avisource("")
MCTD(settings="low")
### prefetch: 16, 0

### ###
""")
AudioDub(u)


So Thanks


Though, i wonder is there a way to prevent MP to load when audio decoding?
Cause currently, it seems to load MP which takes a very long time with more branches, and then they just close, and after all that, it starts processing.
I think it is not possible with single script, you can only do that with separated script file.

I still think that you should demux your source audio and directly encode or mux it to the output, there is no reason to pass it through AviSynth unless you want to do some post-processing to it.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 14th June 2013, 20:38   #92  |  Link
aldix
Registered User
 
Join Date: Sep 2012
Posts: 156
From what does it depend whether the script used inside MP_ gets 100% cpu load or not? Some long scripts of mine get 35-40% cpu load whereas short ones get 100%, I see no rhyme or reason to this. Can you explain, Sapi?
aldix is offline   Reply With Quote
Old 15th June 2013, 02:08   #93  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by aldix View Post
From what does it depend whether the script used inside MP_ gets 100% cpu load or not? Some long scripts of mine get 35-40% cpu load whereas short ones get 100%, I see no rhyme or reason to this. Can you explain, Sapi?
Please remember that CPU usage != speed as what @pbristow said, you need to compare your encoding speed, not CPU utilization.

CPU utilization mainly depends on whether works are fairly distributed across all script blocks, and how good is your prefetch setting. In other words, you need to make all the script blocks consume roughly same amount of CPU.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 15th June 2013, 18:15   #94  |  Link
aldix
Registered User
 
Join Date: Sep 2012
Posts: 156
Well, I like smaller cpu load very much, it enables me to do other tasks. That's why I asked. I don't really understand prefetch, so I'm pretty much setting it at random, sometimes with what's suggested in the example script, sometimes way higher. I don't know...
aldix is offline   Reply With Quote
Old 15th June 2013, 21:59   #95  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,121
Okay, i think the audio goes pretty well now. Not sure why i went so slow before, must have done something wrong.

But i have another question.


If i want to use MCTD, and run it in parallel in 4 threads on the same clip, how can i do that?
I am thinking, that i should have 4 separate script which are identical, Except for the duration, meaning i have Trimmed the clip differently.
And than later i can just suit them up.

Though i have never done something like that so i am not absolutely sure of it.

Or is it possible to have 1 script, with 4 branches, that each process a different "trim()" and MCTD?
zerowalker is offline   Reply With Quote
Old 16th June 2013, 13:02   #96  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by aldix View Post
Well, I like smaller cpu load very much, it enables me to do other tasks. That's why I asked. I don't really understand prefetch, so I'm pretty much setting it at random, sometimes with what's suggested in the example script, sometimes way higher. I don't know...
Well you can set priority of your encoding-related process to low, then your normal tasks won't be (much) affected.

### prefetch controls how much frames ahead should be preloaded in background, and how many frames behind should be cached.

Here is a simplified illustration of effect of prefetching:
Code:
Without prefetching:

T+0:
+------------------+
|  Upstream block  | -> Producing frame #0
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #0 from upstream
+------------------+

T+1:
+------------------+
|  Upstream block  | -> Wait for downstream block to complete processing frame #0
+------------------+

+------------------+
| Downstream block | -> Processing frame #0
+------------------+

T+2:
+------------------+
|  Upstream block  | -> Producing frame #1
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #1 from upstream
+------------------+

T+3:
+------------------+
|  Upstream block  | -> Wait for downstream block to complete processing frame #1
+------------------+

+------------------+
| Downstream block | -> Processing frame #1
+------------------+

---------------------------------------------------------------------------------

With prefetching:

T+0:
+------------------+
|  Upstream block  | -> Producing frame #0
+------------------+

+------------------+
| Downstream block | -> Waiting for frame #0 from upstream
+------------------+

T+1:
+------------------+
|  Upstream block  | -> Producing frame #1
+------------------+

+------------------+
| Downstream block | -> Processing frame #0
+------------------+

T+2:
+------------------+
|  Upstream block  | -> Producing frame #2
+------------------+

+------------------+
| Downstream block | -> Processing frame #1
+------------------+

T+3:
+------------------+
|  Upstream block  | -> Producing frame #3
+------------------+

+------------------+
| Downstream block | -> Processing frame #2
+------------------+
Of course, if upstream block goes too far away from downstream block, it still needs waiting. This really need to tweaked individually for different scripts.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 16th June 2013, 13:11   #97  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by zerowalker View Post
Okay, i think the audio goes pretty well now. Not sure why i went so slow before, must have done something wrong.

But i have another question.


If i want to use MCTD, and run it in parallel in 4 threads on the same clip, how can i do that?
I am thinking, that i should have 4 separate script which are identical, Except for the duration, meaning i have Trimmed the clip differently.
And than later i can just suit them up.

Though i have never done something like that so i am not absolutely sure of it.

Or is it possible to have 1 script, with 4 branches, that each process a different "trim()" and MCTD?
If you have much memory, you may try branch with big block size and large prefetch setting (at least 2x bigger than branch block size). Not sure whether this works since I haven't tried that.

Splitting the script into parts and joining them after encoding may have better performance, but it may also affect rate-controlling of x264, making bitrate distribution worse.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th June 2013, 13:35   #98  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,121
Donīt really get how i am supposed to set it up?
And i donīt have an enormous amount of memory. Got 8gb.
zerowalker is offline   Reply With Quote
Old 17th June 2013, 14:04   #99  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by zerowalker View Post
Donīt really get how i am supposed to set it up?
And i donīt have an enormous amount of memory. Got 8gb.
Something like this:
Code:
MP_Pipeline("""

XXXSource("...")
### prefetch: 512, 256

### ###

MCTD()
### branch: 4, 32
### prefetch: 64, 0

### ###

### prefetch: 64, 0

### ###

""")
Note again, I haven't tested these parameters in any ways, not sure how they perform.

Actually, if you have only 8GB of memory, I would suggest trying this:

Code:
MP_Pipeline("""

XXXSource("...")
### prefetch: 64, 32

### ###
SetMemoryMax(3072) # Set this to smaller value if your memory runs out or the slave process crashes
MCTD()
# Don't put anything else in this block
### prefetch: 32, 0

### ###

""")
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th June 2013, 14:17   #100  |  Link
zerowalker
Registered User
 
Join Date: Jul 2011
Posts: 1,121
Okay, will be trying the second one.
And it seems to run at 9fps with my settings, which i guess is pretty good.

Though i will have to Trim it still, as the Clip is 7 hours, the chance of it crashing is extremely high, it has happened so many times, that i have given up doing it in one run.
zerowalker is offline   Reply With Quote
Reply

Tags
avisynth, multi-process, pipeline

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:30.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.