Bottleneck where?

[ilovejedd]

First, I apologize if this is the wrong section. Please feel free to move it somewhere more appropriate.

I'm encoding a bunch of videos my mom bought (foreign service similar to Amazon Unbox, they're TV shows and not available on DVD) to Xvid so she can watch them on her CRT TV with a Philips DVP5960. Afaik, the videos were encoded in WMV9.

I'm used to my computer being unusable (100% CPU utilization) whenever I encode something. However, on my new build, CPU utilization was only around 55% at ~175fps. I'm trying to figure out where the bottleneck might be. If it's the hard drive, I'm resigned to that. I don't plan on switching to RAID0. If it's something else, though, there might be some changes I can make to maximize performance and speed up encoding time. Any thoughts?

Pentium DC E2160 @ 2.2GHz (245x9.0)
WD7500AAKS x3 (source HDD diff from target HDD)
WINTEC AMPX 2x2GB

Avisynth 2.56
ffmpeg rev 11870 (http://ffdshow.faireal.net/mirror/ffmpeg/)

ffmpeg -y -i "input.avs" -vcodec libxvid -vtag XVID -g 300 -bf 2 -r 29.97 -b 468k -acodec libmp3lame -ac 1 -ab 32k -f avi "output.avi"

input.avs:

DirectShowSource("video.wmv",fps=30000/1001,convertfps=true)
BilinearResize(320,240)
Amplify(3)


Thanks!

[sysKin]

You have a dual core now, that's all. A single core fully utilized is 50% usage.

Use Xvid 1.2.x to get multithreading, but never expect Xvid to occupy both cores at 100% anymore.

[ilovejedd]

Both cores were being used. Codec used was libxvid within ffmpeg.

http://img187.imageshack.us/img187/5918/usagepg5.png (http://imageshack.us)

[Didée]

The bottleneck probably is the decoding of the source. After all, your conversion did run at 175fps .... you can not encode faster than you can decode the source material.

[Tagert]

If you want further speed improvement, then try outputting the finished movie to another harddrive if you have such.

[ilovejedd]

If you want further speed improvement, then try outputting the finished movie to another harddrive if you have such.
It already is:
Pentium DC E2160 @ 2.2GHz (245x9.0)
WD7500AAKS x3 (source HDD diff from target HDD)
WINTEC AMPX 2x2GB

The bottleneck probably is the decoding of the source. After all, your conversion did run at 175fps .... you can not encode faster than you can decode the source material.
Yeah... So is there any way the Avisynth script can be optimized to make decoding faster or are there any options I should add to the ffmpeg commandline?

ffmpeg -y -i "input.avs" -vcodec libxvid -vtag XVID -g 300 -bf 2 -r 29.97 -b 468k -acodec libmp3lame -ac 1 -ab 32k -f avi "output.avi"

input.avs:

DirectShowSource("video.wmv",fps=30000/1001,convertfps=true)
BilinearResize(320,240)
Amplify(3)


I kinda had a feeling that it was my hard drives that're causing the slowdown (either reading or writing) but I was kinda hoping it wasn't and that there was some other way I could speed up encoding...

[Ranguvar]

Try setting the # of threads for Xvid manually, also try using MT() or SetMTMode() for AviSynth...

[Zep]

It already is:



Yeah... So is there any way the Avisynth script can be optimized to make decoding faster or are there any options I should add to the ffmpeg commandline?



Do a decode and nothing else to see how fast you can truly decode. You can use AssumeFPS(1000) then run the avs in VDub without preview turned on as one way to test. 175 is decent so you may simply be memory bound.

[ilovejedd]

Do a decode and nothing else to see how fast you can truly decode. You can use AssumeFPS(1000) then run the avs in VDub without preview turned on as one way to test. 175 is decent so you may simply be memory bound.

Thanks for the tip. Playing around with settings right now.
ffmpeg -y -i "input.avs" -vcodec copy -an -f avi "NULL"

ffmpeg -y -i "input.avs" -vcodec copy -acodec libmp3lame -f avi "NULL"

The first command line gave me decode speed of 740fps. The second one 550fps so I would assume the hard drive isn't the bottleneck yet. Will try to find a revision of ffmpeg that doesn't crash on my machine when using libx264. If fps for x264 is higher than xvid, do you think it might be a good idea to run two instances of ffmpeg for encoding?

[Ranguvar]

Yeah, I can do easily 3000fps with just a stream copy of ~1700kb/s Xvid DVD-res material (500GB SATA 7.2Krpm 16MB cache) :P You got plenty of headroom on that area, even if you have a lesser HDD.

[ilovejedd]

Thanks for the info. Is that RAIDed by any chance? If not, then I should get a close enough performance with my HDDs (Western Digital Caviar SE16 750GB SATA 7200 RPM 16MB Cache).

I guess I'm getting a performance hit, too, decoding the video via Avisynth. Is there a specific version I should use or is Avisynth 2.57 sufficient?

[Ranguvar]

Nope, no raid :)

AviSynth 2.57 is fine. I would use MT and multithread those filters though. Don't know how much of a boost you'll get.

[Zep]

Yeah, I can do easily 3000fps with just a stream copy of ~1700kb/s Xvid DVD-res material (500GB SATA 7.2Krpm 16MB cache) :P You got plenty of headroom on that area, even if you have a lesser HDD.

a stream copy is not decoding so that is a fake FPS rate since it is just how many frames were
copied per second, not rendered.

[Zep]

Thanks for the tip. Playing around with settings right now.
ffmpeg -y -i "input.avs" -vcodec copy -an -f avi "NULL"

ffmpeg -y -i "input.avs" -vcodec copy -acodec libmp3lame -f avi "NULL"

The first command line gave me decode speed of 740fps. The second one 550fps so I would assume the hard drive isn't the bottleneck yet. Will try to find a revision of ffmpeg that doesn't crash on my machine when using libx264. If fps for x264 is higher than xvid, do you think it might be a good idea to run two instances of ffmpeg for encoding?

IMHO better to use more threads if possible. 2 instances would be butting heads for HD I/O access of the same file and it is kludgy anyway :)

Anyway now you know how fast frames can get to your encode app. So the bottleneck is there in that app. be it encoding or some unknown conversion it may be doing etc... and it could still be memory since when you add more functions then more data has to be moved around. heck it could even just be hitting a latency wall and not a throughput one.

good luck with that :D

[Ranguvar]

a stream copy is not decoding so that is a fake FPS rate since it is just how many frames were
copied per second, not rendered.
Yes, I know...

I was just saying that a hard drive can handle that many frames a second, so his hard drive definitely wasn't a bottleneck.

[sysKin]

This thread is silly.

A single-threaded Xvid will use only half of a dual-core, where's the magic here?

[ilovejedd]

This thread is silly.

A single-threaded Xvid will use only half of a dual-core, where's the magic here?
Afaik, the xvid implementation within ffmpeg (libxvid) is not single-threaded so I don't think my question was silly at all. Feel free to correct me if you know it's to the contrary.

[sysKin]

Afaik, the xvid implementation within ffmpeg (libxvid) is not single-threaded so I don't think my question was silly at all. Feel free to correct me if you know it's to the contrary.

All I know is that I wrote multithreading support for 1.2.x branch and not 1.1 release. So unless you are *sure* you got 1.2.x, and you're sure your ffmpeg extended its API to set number of threads, you're getting a single thread.

your CPU graphs are definitely classic examples of a single-threaded application.

[Ranguvar]

And then multithread your script ;)

Install MT as per instructions on its thread, then try typing

SetMTMode(1,2)

at the very top of your script. If that doesn't work, try

SetMTMode(2,2)