Low CPU Usage with Xvid

[DVDHack]

Any suggestions why Xvid only uses 55% or so of CPU ?? I'm running a 2.7GHz Pentium 4 and on second pass only get about 8fps on average.

[JohnMK]

Hyperthreading will frequently show your CPU as only being half-used. It just means the program using the CPU isn't appreciably multi-threaded. This is normal.

[Sharktooth]

Originally posted by JohnMK
Hyperthreading will frequently show your CPU as only being half-used.TRUE!
Originally posted by JohnMK
It just means the program using the CPU isn't appreciably multi-threaded. This is normal. Uhm... not exactly.
There are many, many, many reasons that could make Hyperthreading a complete failure... Its not just the "program".
Heres a list of some possible situations where hyperthreading screw things up:
1) - As johnMK said the running program is not coded using multiple threads or it is but in a wrong way (that definatly wont be good on any multi processor system).
2) - The program works on a large data set that just doesnt fits in the CPU cache (a constant data flow cant be kept to the CPU execution pipeline - and since the P4s have a very long pipeline everytime it happens the pipe should be emptied - wasting CPU cycles - before restarting the normal execution operations, HT "helps" emptying the pipeline faster ...).
3) - The program uses some kind of code that requires a perfect sync between threads (and since the P4 reorders the instructions before execution, all syncs gets screwed - result: apparently low CPU usage coz threads are still executed in pairs but one unit has to wait the other one to finish... many wasted CPU cycles)
4) - Some external thread/task interferes with the running program. It may happen when executing more than a program simultaneously (for example converting audio and video at the same time with different softwares). In those cases there are a lot of cache miss and the CPU is forced to empty the pipeline to continue the execution (read point 2).
5) - Excess of DMA requests. This is the case when you're ripping and compressing a DVD simultaneously with DVDShrink. Working on a large data set in conjunction with an excess of DMA requests may interrupt the data flow between memory and CPU cache in the compression thread(s) creating the situation described in point 4).

There are a whole lot of situation where HyperThreading does more bad than good.
So watch out of those "pro-HT" smelly benchmarks. Try it for yourself and draw your conclusions.

[DVDHack]

Well the performance I'm getting in fps metrics are certainly disturbing. Is there an easy wasy to turn it off to verify the problem?

[Danzel]

Hyper Threading can be easily disabled in the bios (usually found by pressing del when the computer first starts up)

Danzel.

[jared1999]

Originally posted by DVDHack
Any suggestions why Xvid only uses 55% or so of CPU ?? I'm running a 2.7GHz Pentium 4 and on second pass only get about 8fps on average.
Something is messed up. I have a P4 2.8 with HT and usually get 20-25 fps with RC3 on the second pass, more on the first. It depends on settings of course, but for me time is not an issue so I use highest possible. Are you sure Xvid is using CPU optimizations?

Originally posted by Sharktooth
3) - The program uses some kind of code that requires a perfect sync between threads (and since the P4 reorders the instructions before execution, all syncs gets screwed - result: apparently low CPU usage coz threads are still executed in pairs but one unit has to wait the other one to finish... many wasted CPU cycles)
What do you mean by perfect sync? AFAIK thread synchronization usually is nothing more than waiting for and achieving certain states. Mutexes, monitors and barriers all do this. I don't see what a P4 can do that causes threads to actually go out of synch. Please elaborate.

Originally posted by Sharktooth
4) - Some external thread/task interferes with the running program. It may happen when executing more than a program simultaneously (for example converting audio and video at the same time with different softwares). In those cases there are a lot of cache miss and the CPU is forced to empty the pipeline to continue the execution (read point 2).
This is quite contrary to my experience. From my testing there isn't much of a difference when running single threaded apps alone, but multi-threaded apps and running many programs at the same time do get a (relative) performance boost. It just shows we need more documentation and/or testing on this matter I guess :)

Edit: Clarity.

[Selur]

Something is messed up. I have a P4 2.8 with HT and usually get 20-25 fps with RC3 on the second pass, more on the first. It depends on settings of course, but for me time is not an issue so I use highest possible.
Can someone confirm this, that one get's 20fps+ for a normal DVD encode in second pass with VHQ4+Chroma motion, chroma optimizer, qpel, trellis, etc. turned on, on a p4 2.8?

Personally I would have thought 8fps are okay if everything is turned on. ;)

Cu Selur

[Koepi]

GMC is known to be very slow and unoptimized, it can go down to 8fps even on a p4 2.8ghz HT.

Regards
Koepi

[DVDHack]

I don't use Qpel but do use GMC. I also don't resize. I just archive DVD to Xvid using no resize so I get 720x576. I set image size to 2.35GB so 2 DVDs go on one DVD-R.

I use AS@L5 2 pass with GMC. Motion precision is 6, VHQ is 1. I have turned off packed bitstream to allow ffdshow playback to resize to 1280x720.

Any suggestions?

Incidentally I get high 40s for first pass encode.

[sysKin]

Originally posted by DVDHack
I use AS@L5 2 pass with GMC. Motion precision is 6, VHQ is 1. I have turned off packed bitstream to allow ffdshow playback to resize to 1280x720.

Any suggestions?Disable GMC and use VHQ2, 3 or 4 instead. You will get better quality and it should also be faster (especially at 2 or 3).

GMC is the last speed/quality switch you should use, as it gives you very small quality gain at huge speed cost.

[jared1999]

Originally posted by Selur
Can someone confirm this, that one get's 20fps+ for a normal DVD encode in second pass with VHQ4+Chroma motion, chroma optimizer, qpel, trellis, etc. turned on, on a p4 2.8?
My bad, too little information in that post. I don't use GMC, Qpel and Adaptive Quants. My cpu is also slightly overclocked to 2.95 GHz (memory at 210 MHz), but the performance gain from that is small. I haven't had the time to do lots of encoding with RC3 yet, but what I have done so far have ended up with an average 20+ fps.

[Sharktooth]

Originally posted by jared1999

What do you mean by perfect sync? AFAIK thread synchronization usually is nothing more than waiting for and achieving certain states. Mutexes, monitors and barriers all do this. I don't see what a P4 can do that causes threads to actually go out of synch. Please elaborate.
Some coders may want to balance workload for each thread to have equal execution times obtaining virtually no "waits" during the execution.
However those kind of optimizations may screw up the P4 instruction execution order cause the P4 has a dedicated unit to "reorder" the execution queue to optimize the execution times.

Originally posted by jared1999
This is quite contrary to my experience. From my testing there isn't much of a difference when running single threaded apps alone, but multi-threaded apps and running many programs at the same time do get a (relative) performance boost.[/B]
except if those apps/threads work on a large data set (just like video/audio codecs) that will produce a lot of cache miss.
It is well known how p4 behaves when a cache miss is triggered (i have described it in my previous post) and when it happens HT only helps emptying the pipeline faster causing more troubles.

[jared1999]

@Sharktooth:
Interesting, I'll look some more into this. Thanks for explaining.

[Sharktooth]

If you find something interesting let me know, thanks.

[DVDHack]

OK I disabled GMC set VHQ to 4 and got about 13fps. This is with no resize ie 720x576 PAL. I still get 55-60% CPU utilisation on second pass and nearly 80% on first pass.

Any further thoughts?

[Lagoon]

With every single option used and no turbo, I get about 5fps on second pass with a P4 2.8 for a 720x388 file.

So it really depends what setting do you use.

Though I also notice the first pass *seems* to use hyperthreading as its getting higher than ~50% use, around 85, a single thread app would never use more than 55-60 at most. But don't ask me why it doesn't go as high on the 2nd pass.

[Sharktooth]

always remember that windows is still not 100% accurate in reporting CPU usage.

[Longinus]

I just finished putting together my P4 3ghz system, and at my first compression test, I ran into this '50% processor use' bug.. Since my Hyper-Threading is enabled, I imagined that the codec, was using only one 'processor power'... But that is NOT the case..

If you encode video AND audio, virtualdub will use more than one thread (one for video and the other for audio codec), the CPU usage will be higher than 50% (but sometimes not 100%). So I just encoded the video to see what I would get..

With only one instance of VDUB running, with processor affinity set only to 'CPU 0' on taskmanager, I'm getting about 120 fps with this mpeg video. The graph show only the first CPU at 50% usage, and the other as a flat line in 0%.. So I thought that the second "processor" was not in use at all..

But that turned out to be wrong....

If I open another instance of vdub, set the processor affinity to the other processor, CPU 1, and encode the same video with the same settings, I get about 60fps, on both vdubs.. and, of course, 100% CPU usage...

So, in the end, the same 120fps that only one vdub was making...

[jared1999]

I'm guessing it has to do with HT using virtual processors. The set of resources available is virtually divided in two, giving this effect while in fact the running task is using all of the "actual" cpu. But this is pure speculation :p

[Sharktooth]

As usual intel does the wrong things in the wrongest ways...
The "logical processor" thing is ... well i cant find the right words... You'll never know what will be the exact usage of those "logical processors".
In short HT does not add another CPU core but only a second pipeline that's depending on the first.
On a dual processor system those pipelines are completely separated so it is possible to measure the CPU usage without problems.
HT uses a workaround to keep track of the cycle timings. It makes the system believe it's running on a dual CPU.
Thats needed to use the OS multiprocessor optimizations but its the whole concept that is completely wrong.
HT is faulty, sometimes unnecessary, sometimes slower and only usefull in few circustamcies.
A "dual core on a single die" approach would have been much better, faster, more compatible (no hangs) and more robust. But since Intel was loosing ground and popularity they decided to go for the "cheap" way in the hope to regain some performance at about the same price of regular p4s. They did it again with IA32e (an extension for IA32 to cope with AMD 64 bit instructions and data sets).
Do not expect your next P4 IA32extended to have the same performance with 32bit and 64bit code... It haven't any dedicated logic to execute 64bit instructions.
So for example a 64bit divx codec will have major benefit on Athlon 64s but not on P4s with IA32e.
Whats the point on that? Have the ability to run 64bit apps (with no tangible advantage). Why? No one knows but Intel.
End of Rant.

[JnZ]

Originally posted by DVDHack
Any suggestions why Xvid only uses 55% or so of CPU ?? I'm running a 2.7GHz Pentium 4 and on second pass only get about 8fps on average.

For all people there: I have the same problem, but a little worse.

I noticed, when I activate QPel due encoding, movies are 2-2,5x more consuming CPU power during playback (decoding).

For example: I encoded Kubricks last movie Eyes Wide Shut, it's 4:3 movie with full screen video. I want to go to 3CD rip with AC3.
XviD settings: H.263,2xBvops-1.50/1.00,Chroma motion,QPel,VHQ-4, MSP-6.
I used strong noise filter in AVS script (movie was incredible noised), and go to 704x512 res. After encode, rip looks very good (maybe better than DVD:D ).
BUT: When I play this movie, it consumes about 80%! of my CPU power :scared: WOW! (AMD Barton 2500+@3200+).

Other 16:9 movies: With QPel: 40-50%; without QPel: 15-20% CPU usage.

SO: Is this thing with QPel right, or only on my PC?:scared:

[Sharktooth]

It's ok. Thats why standalones have playback problems with movies encoded with Q-pel.
The CPU usage is so high coz its a 4:3 movie. The decoder has to process more pixels than a normal 16:9 one.

[Rash]

Originally posted by sysKin
GMC is the last speed/quality switch you should use, as it gives you very small quality gain at huge speed cost.
If it weren't you saying this, I would have problems believing it hehe. Doesn't GMC help like hell on camera pan, for example? I thought if I disabled GMC, XviD would have to allocate a lot more bits to camera pans, thus lowering the overall image quality.