x264 cpu usage Question

[gav1577]

Hi i have noticed that x264 does not take full advantge of my cpu's
power on the first pass but does on the second pass is there a way to make the first pass use my cpu as much as the second pass ?
i have a quad core q6600

Thanks

[LoRd_MuldeR]

If you already use the optimal number of threads (threads = 3/2 * number_of_cores) then there is not much you can do.
Most likely the bottleneck is not x264 itself, but your source. Do you use heavy filters in your AVS script ???

[gav1577]

No i dont use any filters at all and all my sources are from original bluray movies it does use all cores but not more than 75/80% load
on all cores on the first pass. However on the second pass all cores see 95/100% any ideas :confused:

[LoRd_MuldeR]

I assume you are using "fast" settings in the first pass, so your HDD and/or the decoder can easily become the bottleneck...

[Manao]

The bottleneck is the frametype decision in x264, which isn't threaded (yet)

[gav1577]

Thanks for the help. i cant see the hdd being the bottleneck i have two fast hdds running in raid 0. But yes i am using a fast first pass

First pass
--level 4.1 --bframes 3 --direct auto --subme 2 --partitions none --chroma-qp-offset -4 --vbv-bufsize 30000 --vbv-maxrate 30000 --me dia --threads auto --thread-input --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --mvrange 511 --aud --output

Second Pass
--level 4.1 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-rdo --bime --weightb --direct auto --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --chroma-qp-offset -4 --vbv-bufsize 30000 --vbv-maxrate 30000 --me umh --threads auto --thread-input --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --mvrange 511 --aud --output

[gav1577]

The bottleneck is the frametype decision in x264, which isn't threaded (yet)
Thanks manao. i dont think i had this problem with a dual core though is it just quad cores that have this problem ?

[LoRd_MuldeR]

Thanks manao. i dont think i had this problem with a dual core though is it just quad cores that have this problem ?

Well, your QuadCore obviously has much more power then the DualCore. But an application needs very good multi-threading capabilities to use four cores at 100%.
So the "frametype decision" bottleneck simply didn't exhibit on your old machine, because it was busy with other stuff...

[gav1577]

Ok guys that cleared things up for me thanks for the help much appreciated :)

[JetJock_fl]

I'm having the same problem as others that previously posted. However I've noticed that a benchmarking program that uses the x264.exe encoder loads up all my cores but that Xvid4psp that uses x264.exe exhibits the same problem as above and only uses about 33% and maxes out at maybe 40% occasionally. That's alot of unused encoding cycles. I have a Sata 300 drive and motherboard so I don't think it's a data access problem.

The benchmarking program is here:

http://www.techarp.com/showarticle.aspx?artno=520

and it uses versions 747 of x264 and version 819 of x264
the script lines go like this:

start /high /b x264-747 --quiet --progress --pass 1 --bitrate 3959 --stats "1.stats" --level 4.1 --keyint 24 --min-keyint 1 --bframes 3 --direct auto --subme 1 --analyse none --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 30000 --vbv-maxrate 38000 --qcomp 0.5 --me dia --threads auto --thread-input --sar 1:1 --progress --no-psnr --no-ssim --output NUL "test-720p.avs" --mvrange 511 --aud --nal-hrd 2>&1 | tee run1pass1(v747).log

That script above maxes out all four of my cores.

[Shinigami-Sama]

I
and it uses versions 747 of x264 and version 819 of x264


x264 win32 daily builds: rev. 928
:rolleyes:

[Snowknight26]

Try mt() in your avs script?

[Sharktooth]

x264 win32 daily builds: rev. 928
:rolleyes:
it's a benchmark. it uses the same (old) version of x264 for results consistency.

[JetJock_fl]

Try mt() in your avs script?

Here is the script:

import("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\functions\AudioFunctions.avs")
import("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\functions\VideoFunctions.avs")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\apps\DGMPGDec\DGDecode.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\NicAudio.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\TIVTC.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\ColorMatrix.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\soxfilter.dll")

video = MPEG2Source("C:\Media\HDMedia\Spiderman.index\Spiderman.d2v",cpu=0,info=3)
audio = NicAC3Source("C:\Media\HDMedia\Spiderman.index\Spiderman T80 3_2ch 448Kbps DELAY 2256ms.ac3")
AudioDub(video, audio)

ConvertToDolbyProLogicII()
DelayAudio(2.256)
AutoYV12()
ColorMatrix(hints=true,interlaced=true)
TFM(order=-1).TDecimate(hybrid=1)
Crop(4, 2, -4, -2)
Lanczos4Resize(480, 272)


What would the sentax be for a quad core q6600 cpu and where would I put it.

[JetJock_fl]

Ok, I thought it was a little rude to just ask how to use MT and not do any reading on my own. So I read a little and think I have a small grasp of useage, obviously I can't be considered at novice at it with only 30 mins of study. But I am now 'familar' with it's usage. Here it goes:


setMTMode(2,0)
import("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\functions\AudioFunctions.avs")
import("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\functions\VideoFunctions.avs")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\apps\DGMPGDec\DGDecode.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\NicAudio.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\TIVTC.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\ColorMatrix.dll")
loadplugin("C:\Program Files\Winnydows\XviD4PSP5\dlls\AviSynth\plugins\soxfilter.dll")

video = MPEG2Source("C:\Media\HDMedia\Spiderman.index\Spiderman.d2v",cpu=0,info=3)
audio = NicAC3Source("C:\Media\HDMedia\Spiderman.index\Spiderman T80 3_2ch 448Kbps DELAY 2256ms.ac3")
AudioDub(video, audio)

ConvertToDolbyProLogicII()
DelayAudio(2.256)
MT("AutoYV12()")
ColorMatrix(hints=true,interlaced=true)
TFM(order=-1).TDecimate(hybrid=1)
Crop(4, 2, -4, -2)
MT("Lanczos4Resize(480, 272)")


How did I do?

[JetJock_fl]

Well, I guess I could find out for myself and just try it out...here it goes:

[Sagekilla]

Well, as a heads up, resizing is a very fast operation and I doubt it would benefit greatly from MT().

Also, are you sure that spiderman rip is interlaced and doesn't require pulldown to 23.976 fps?

[JetJock_fl]

Nah, Nah nah nah, hey hey hey GOODBYE!!!!!

Wow GUYS!!! It worked....a little. I'm happier anyway! The above script didn't do exactly right, kept getting some height and width issues. With some more research I found the correct syntax Is:

MT("Lanczos4Resize(480,last.height)")
MT("Lanczos4Resize(last.width,272)",splitvertical=true)

MY CPU USEAGE is now up to 60% FPS shot up to 18fps and total encode time dropped an hour or more. WhaHOOO!!!! Now if I could just eeek out that final 40% on the CPU useage I would be king of the world!!!

[JetJock_fl]

Well, as a heads up, resizing is a very fast operation and I doubt it would benefit greatly from MT().

Also, are you sure that spiderman rip is interlaced and doesn't require pulldown to 23.976 fps?

Not real sure. I recorded it from ABC or TNT on my Comcast DVR. Then saved it. I think the DVR puts out 1080i, not 1080p. That's What I have it set to output anyway.

[Sharktooth]

try using MT with other filters. however keep in mind avisynth_mt is alpha... and highly unstable.
i got several bug reports on megui that were fixed by reinstalling the normal avisynth.

[JetJock_fl]

As far as Lanczos4resize being fast....it ain't so fast on my system. UGH! But thanks for the slap! ;)

Before when I would look at the TaskManager it would show the x264.exe using 33% of cpu useage, now it's shows 58% cpu useage. System Idle Process is up at 42%. Sure would like to grab that LAST massive amount of idle cycles.

[Warpman]

As far as Lanczos4resize being fast....it ain't so fast on my system. UGH! But thanks for the slap! ;)

Before when I would look at the TaskManager it would show the x264.exe using 33% of cpu useage, now it's shows 58% cpu useage. System Idle Process is up at 42%. Sure would like to grab that LAST massive amount of idle cycles.

well lanczos4resize is far away from i would call "fast" ... there are much faster ones... like bilinear for example...

also if your task manager says x264 does not use 100% of all cores that does NOT mean that x264 is the bottleneck.

Most likely your input is (stil) limiting x264...

[Sagekilla]

I mean fast relatively speaking to other filters. Within the resizers, yes lanczos might not be the fastest, but compared to most denoisers and deinterlacers, the speed hit is minimal.

[Sagekilla]

Of course, if you're bottlenecked by avisynth this is always a great time to crank up the settings on x264 and get even better compression ;)

[JetJock_fl]

True Sage, but I can't find out where this program passes the paramters to x264.exe

I can't find where starts x264.exe

[JetJock_fl]

I can't get this line to thread:

Crop(4, 2, -4, -2) ;this one results in a resize error

Nor this one:

ColorMatrix(hints=true,interlaced=true) ;and this one says something about hints and nostream

[Sagekilla]

Don't bother threading Crop.. And ColorMatrix won't thread properly because it needs the full frame to perform it's operations I believe, and MT() will split the frame in half or vertically. IMO, you're better off using SetMTMode() since that does frame based multithreading, since you're having issues with MT().

Edit: In fact, I believe that's where most of your issues are stemming from: MT's split frame based multithreading. If you were cropping in one direction then you could tell MT to split in the other direction (crop horizontal, split vertical)

[JetJock_fl]

Sage,
If I use the SetMTMode() do I have to go back and change the Lanczos4resize script line or leave it as is?

[Ranguvar]

If you use SetMTMode(), the frames are split up temporally and handed to each core, so you can do a normal resize/crop as you would if you were doing no multithreading.

[JetJock_fl]

Well then, I guess it's just a matter of trying each way and seeing which one gives me the faster FPS or better quality. I would hope neither would have an effect on quality though or playability. Why is one way suggested over the other? Seems like the SetMTMode() provides for the more universal solution.

[Sagekilla]

You have to remove your MT() if you use SetMTMode() There shoudl be no affect quality wise for which one you use, as long as you have overlapping with MT(). The only issue is that SetMTMode will not work in certain modes with all filters. Look up the documentation for MT for specifics.

[Ranguvar]

@Sagekilla: Really? I was under the impression that functions which used masks across the entire frame or near to it would be affected by MT()'s spatial cutting. Or do you mean for this case only?

[JetJock_fl]

Ok, I changed it to just SetMTMode() and my cpu finally saw 95-100%!! Thanks guys. I will encode a movie and see if it is indeed faster. Seems like the fps are lower than the previous method. Odd?!?! Let me run it for 3 or 4 hours and see how it does.

[Sagekilla]

Well, the thing with MT is, like I said, you'll have overlapping between the two halves (if you enable it) so if you use masks it -should- be fine. There may be some oddities at the split point, but you can easily alleviate that by increasing the overlap I believe. Also, while your CPU may be at near 100% utilization, it's possible some loss in efficiency was introduced due to threading with Avisynth. If your encodes are running much faster than before, I wouldn't worry about it.

[Ranguvar]

We're going a little off topic, my fault, but since the frame is divided spatially with MT(), functions that need information from the whole frame like masks will experience a quality drop, because they can now only access their little piece of a frame. Or so I've heard.