Running 4 AA filters at once, why CPU not 100%? [Archive]

View Full Version : Running 4 AA filters at once, why CPU not 100%?

Chengbin

26th August 2009, 13:05

Since I quit using multithreading avisynth, I decided to run 4 separate encodings each with AA to multithread.

Here's my problem. My CPU is not running at 100% like it should. Sometimes each encoding runs at 25% (1/4 of my quad core), sometimes it runs at 0-5%. Therefore my CPU is never running at 100%, more like 60%.

I'm running 4 of this on a lossless source, 3 of them use the one on my external hard drive (through USB 2.0), and 1 use a copy on my main hard drive.

Each encoding has its own filters (yes, I made 4 copies of the same filter)

Here are the only problems I can think of:

1) Some encodings are using virtual RAM, and my hard drive can't move fast enough. But I don't see my hard drive light being steady.

2) My external hard drive's USB 2.0 is the bottleneck (running at 30MB/s). My external's light is blinking good, but I don't know how it can be a bottleneck. Yes, the lossless source is 100Mbps, but each encoding is running from 0.01-0.05fps, a 30MB/s connection isn't good enough for 3 encodes?

Anybody got a clue as to what is preventing my computer from using 100% of my CPU?

Adub

26th August 2009, 22:53

You didn't tell use what you are using to encode. Xvid or X264?

Chengbin

26th August 2009, 23:02

You didn't tell use what you are using to encode. Xvid or X264?

x264, lossless mode, -preset ultrafast to save RAM.

Didée

26th August 2009, 23:17

And, what format is the lossless source in? Uncompressed RGB, x264-lossless, Huffyuv, FFV1, Snow, Lagarith, CorePNG, WARP, ... ?

Chengbin

27th August 2009, 00:10

x264 lossless

henryho_hk

27th August 2009, 11:34

Yes, your external HDD and the USB connection are the bottlenecks. Even with a local harddisk (XviD TGEDIT preset ~25Mbps), I notice some slowdown with 4 simultaneous encodes.

Do make sure that you have enough memory though. Run taskmgr to verify your memory load.

Also, you may want to ensure that each encode is using 1 thread only. The Windows kernel is not particularly efficient for multiple multi-thread CPU-intensive programs. From my experience, 4 single-threaded encodes run faster than 4 quad-threaded encodes in my Q6600 machine.

Chengbin

27th August 2009, 13:43

I confirmed hard drive bottleneck as well, but why would 4 encodings running at 0.02fps cause a hard drive bottleneck, even if the source is x264 lossless and I'm encoding to x264 lossless?

tedkunich

27th August 2009, 17:46

I confirmed hard drive bottleneck as well, but why would 4 encodings running at 0.02fps cause a hard drive bottleneck, even if the source is x264 lossless and I'm encoding to x264 lossless?

Are you that naive? You have 8 concurrent 100Mbps streams (4 sources and 4 outputs) trying to cram through a 30Mbps pipe and you wonder why the computer dog slow????? We wont even address the fact that the drive heads are probably thrashing about so much that your actual throughput is significantly less than 30Mbps...

Chengbin

27th August 2009, 17:49

Are you that naive? You have 8 concurrent 100Mbps streams (4 sources and 4 outputs) trying to cram through a 30Mbps pipe and you wonder why the computer dog slow????? We wont even address the fact that the drive heads are probably thrashing about so much that your actual throughput is significantly less than 30Mbps...

100mbps=12.5 megabytes per second.

Didn't I mention multiple times that the encoding speed is 0.02fps? That means it takes around 50 seconds to decode and encode 8 frames (4 encodings).

kemuri-_9

27th August 2009, 20:48

you have 3 simultaneous reads of 100 mbps going on your external.
and since it appears you're also writing back to it, that's 3 simultaneous writes.

you have 6 simultaneous I/O operations working on your external.
no HDD can realistically handle all the seeking and transferring such large amounts of data in this fashion well.

TheRyuu

27th August 2009, 21:27

I would suggest to you to just use the MT avisynth.

There should be nothing wrong with MT("AA()",4,8) (4 threads, 8 overlap, insert your appropriate AA filter as well)

Chengbin

27th August 2009, 21:51

@kemuri 9

It would still be a problem when the hard drive is supposed to read 1 frame per minute for each encode?

@the guru

There would be something wrong whwn you are on a 32 bit OS.

I tried MT, but it would only use 2 cores + x264 or 3 cores in virtualdub before running out of memory.

TheRyuu

27th August 2009, 23:40

I tried MT, but it would only use 2 cores + x264 or 3 cores in virtualdub before running out of memory.

Don't judge it based on the cpu usage. Look at the speed (hopefully improvement).

Paste the avs script that you're using and a sample which causes your problems of running out of memory.

Furthermore, what are the specs of your computer?

Chengbin

28th August 2009, 01:23

I have a Q6600 + 3GB RAM on a 32bit Vista.

The memory problem is because of multithreading, because all I did was write setmtmode (2,x) before everything else. I tried (2,0), ran out of memory. (2,3), barely allowed, no encoding. (2,2), can encode, but speed improvement not as great as running 2 separate encodings.

BigDid

28th August 2009, 01:41

Hi,

I second TheRyuu demand for posting your script.
I made some (promising) MT tests on a slow 2 cores but would like to test your script or an evolution of your script (apple to oranges once again) ;)
Thanks.

Did

Chengbin

28th August 2009, 01:48

TheRyuu

28th August 2009, 03:44

LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\DirectShowSource.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\MSharpen.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\sangnom.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\masktools.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\mt_masktools-26.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\mvtools.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\mvtools2.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\Toon-v1.0.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\awarpsharp.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\warpsharp.dll")
LoadPlugin("G:\HDConvertToX_2.1.458.3528 - Copy\filters\removegrain.dll")

Import("G:\HDConvertToX_2.1.458.3528 - Copy\filters\LimitedSharpenFaster.avsi")
Import("G:\HDConvertToX_2.1.458.3528 - Copy\filters\AnimeIVTC - v1.06.avsi")

setmemorymax(64)
setmtmode(2,2)
directshowsource("G:\conan\520\lossless.mkv")
sharpaamcmod(aastr=5, ds=0.2, shpre=80, shpost=300, smpost=100, tradius=2, mt=true, pel=1, ov=4, blk=8, aatype="sangnom")
trim(7251,7500)

Well first of all you're using mvtoolsmulti which is threaded (to a certain degree (with mt=true)). So using setmtmode on top of that can probably cause slowdown or too much memory usage or both. So either use sharpaamcmod without setmtmode and mvtoolsmulti or regular mvtools + setmtmode (probably faster) or MT() (probably broken, see below).

I tried the following:
MT("""
sharpaamcmod(aastr=5, ds=0.2, shpre=80, shpost=300, smpost=100, tradius=2, mt=false, pel=1, ov=4, blk=8, aatype="sangnom")
""",3,8)

It worked however it led to artifacts on the video (i have no idea why, probably mvtools breakage with MT()).

As for SetMTMode:

SetMTMode(2,4) #mode=2, threads=4
#you can try using 6 threads and see if that increases speed
#remember, don't just speed based on cpu usage

Source()
sharpaamcmod(aastr=5, ds=0.2, shpre=80, shpost=300, smpost=100, tradius=2, mt=false, pel=1, ov=4, blk=8, aatype="sangnom")

Worked fine for me, and is probably the fastest of all possible combination's of threading this script.

Note: I tried this on 720x480 dvd video (because i have no idea what you're using. I got about 8fps on a low motion test scene encoding with x264 to lossless (on Q6600 @ 3ghz).

A few more notes about your script:
-Don't use setmemorymax, you have 3gb, let avisynth do it's thing regarding memory usage (using such a low value may be slowing down your script unnecessarily).
-trim() will be inaccurate because directshowsource is not frame accurate.
I suggest you use FFMS2 (http://code.google.com/p/ffmpegsource/downloads/list) when dealing with h264 predictive lossless video. If you want full threaded decoding make sure to use the ffmpeg-mt compiled version of FFMS2. Or at least use DSS2 if you want something (probably) frame accurate with directshow.

henryho_hk

28th August 2009, 06:12

Also, don't forget the efforts for four h.264 lossless decompression and four h.264 lossless compression. If you are just doing some intermediate processing tasks, I believe space is not much a concern and you may consider other less CPU-intensive compression such as huffyv12.

The kernel is spending considerable effort in the USB connection too. Opt for an e-SATA harddisk if external enclosure is a must (still worse than internal ones though, coz the e-SATA chipset such as JMicron can't really compete with the ICHx controllers)

And, as Ryuu and I mentioned, don't run overload your Windows box with busy threads. The Windows kernel is not particularly efficient managing multiple multi-thread CPU-intensive programs. From my own experience on Q6600, 4 single-threaded encodes run faster than 4 quadruple-threaded encodes.

BigDid

28th August 2009, 07:19

Hi,

@chengbin, thanks for the script.
@theRyuu, thanks for testing I could have been stuck with the mt=true thingie

- Gear is listed in my sig, with 2*1Gb Ram;

- Source is a Mpeg2 capture movie, 4500Kbp/s;

- Outputs are:
1/ Xvid 1.22 vfw/Vdub 1 pass, Quant 2.67 (usual)
2/ x264 r1145 vfw/Vdub 1 pass lossless (not usual)

- Script is related to the original except:
setmemorymax(512) # 1st line
autocrop() #720*400
Setmtmode(1,0) # eventually
MT("""sharpaamcmod(...)""",2,8) # same as TheRyuu, eventually or sharpaamcmod(...) without MT()
MT("SelectRangeEvery(300,10)") # gives 4940 frames

- Results, from slower to quicker, estimation from 3' encode each time:
1/Setmtmode()
xvid. 23'07" gives 3.56fps
x264 23'

2/ no MT or Setmt, just plain calls
xvid. 22'55" gives 3.59fps
x264 22'52"

3/ MT() alone without ,2,8) in the end
not working !

4/ MT(...,2,8)
xvid. 13'42" gives 6fps
x264 15'38" gives 5.26fps

... and for comparison purposes MT("Maa()")
xvid. 3'40" gives 22.45fps
x264 4'45"

Without same source, script and parameters, speed may vary but it gives some clues.

Did

Chengbin

28th August 2009, 13:21

Thank you for your testings.

Some questions:

Is directshowsource still frame inaccurate when it loads ffmpeg-mt to decode the video?

So encoding a 1920x1080 video with this script, running single thread, at 0.06fps, doesn't that sound a bit too slow to your results?

Didée

28th August 2009, 16:44

At first glance: perhaps. Looking closer: no.

Realize the facts mentioned in #18. You're running 12 highly demanding tasks in parallel.

What you have is simply a case of "system overload".

TheRyuu

28th August 2009, 20:11

Is directshowsource still frame inaccurate when it loads ffmpeg-mt to decode the video??

Directshowsource is always inaccurate no matter what the underlying decoder is.

Chengbin

28th August 2009, 23:59

Directshowsource must be a very broken filter.

Directshowsource2 was INFINITELY better. My hard drive stopped thrashing (so it was the hard drive bottleneck), and therefore the speed went up SIGNIFICANTLY, around 5 times.

Thank you guys for telling me how bad directshowsource is.

Chengbin

29th August 2009, 03:18

I just tried setmtmode(2,2) with dss2. It was MUCH faster than single thread avisynth despite only using 2 cores. Single thread was 0.1fps, with setmtmode(2,2), the speed reached 0.65, and increasing before it hit the 2GB limit. Why is it so much faster?

EDIT: I just tried using setmtmode(2,1), it is much faster as well, hitting 0.52fps! WHY??????? Is MT avisynth this magical?

I'm really pissed and ecstatic at the same time...

EDIT 2: Now the speed dropped to 0.3fps. Very good nonetheless. I was encoding at 0.02fps 24 hours ago.