Multithreaded (SMP) XviD - Cpu usage

[Hironimo]

Hi.

I tried out the 1.2.127 beta build (i'snt there any later one? - this is about a year old!) from koepi which supports SMP.

First question: Default number of threads is 0. Does that mean multithreading is off and would be the same as 1 thread?
For a dual-core cpu the right setting would be 2, right?

The main issue:
When I encode the CPU is not used 100%, but only 60 to 70%. Why? I would expect 100, but would be ok with 90.

Btw, I use VirtualDubMod to encode from an avisynth script.
My System: AMD Athlon 64 x2 3800+ and 2gig of Ram on Windows XP.

Can you tell me why? Is this normal? How to make XviD use it's full potential of the CPU?

[Manao]

What is your avs script ? The script may be the bottleneck if it's slower than the actual encoding.

[foxyshadis]

Newer Xvid: http://ffdshow.faireal.net/mirror/XviD/

0 threads = # of cpus. Maybe with some multiplier, don't have time to look right now, but 0 is the best default.

[Hironimo]

What is your avs script ? The script may be the bottleneck if it's slower than the actual encoding.

Hi. You may be right that the script is the bottleneck:

LoadPlugin("F:\Program Files\AviSynth 2.5\plugins\DGDecode.dll") # dgdecode from dgindex (http://neuron2.net/dgmpgdec/dgmpgdec.html)
LoadPlugin("F:\Program Files\AviSynth 2.5\plugins\TDeint.dll") # TDeint 1.0 RC 2 (http://bengal.missouri.edu/~kes25c/)
LoadPlugin("F:\Program Files\AviSynth 2.5\plugins\aWarpSharp.dll") # http://www.avisynth.org.ru/docs/english/externalfilters/awarpsharp.htm

# scripts
Import("F:\Program Files\AviSynth 2.5\plugins\scripts\Mrestore.avs")

# load mpeg2
MPEG2Source("G:\capture\cut\03-05_Wolf's Rain.d2v",cpu=4,idct=5)

# bob the source (to 50fps, required for Mrestore)
# and restore to the original 23.976 fps (which eliminates interlacing)
#tdeint(mode=1)
tdeint(mode=1,type=2)
Mrestore(rate=23.976)

However, I get the same cpu usage on both cores. Should then not at least one core be used to the maximum?

[Reefer]

According to http://www.divx-digest.com/articles/xvid_setup_page3.html

Number of threads: - This options was enabled in version 1.2.x of the XviD codec. This allows you to run XviD encodinds in multi-thread mode, to take advantage of dual-core, hyperthreaded or multi-CPU setups. Leaving this at "0" disables multi-threading, and this is the recommended setting. Setting it to 2 might speed up encoding in a dual-core or hyperthreaded system, but certain options (like GMC) are not multi-threaded, so the speed up might not be evident.


I'm not sure if this really answers your question as I'm still pretty new to xvid encoding myself, but saw your thread and remembered reading this today and had set mine to two.

[HarryM]

Can you tell me why? Is this normal? How to make XviD use it's full potential of the CPU?

I think, XviD SMP isn't fully optimized yet(?), only motion vector search. You must use Virtualdub (instead of meGUI). Virtualdub uses 4 independent thread. 1st for I/O operation, 2nd for encoding, 3th for preview, 4th for program controlling.
With Virtualdub, you have CPU usage >90%. Very nice.:)

[foxyshadis]

According to http://www.divx-digest.com/articles/xvid_setup_page3.html



I'm not sure if this really answers your question as I'm still pretty new to xvid encoding myself, but saw your thread and remembered reading this today and had set mine to two.

I thought this was changed in CVS after I brought it up to syskin, but that might not be the case. It's always safest to explicitly define your threadcount, I suppose.

50% cpu on both cores, or any variation, is exactly the same as 100% on one core. Thread scheduling on windows and most other operating systems doesn't make an extra special effort to keep a thread on one CPU when there are lots of other OS threads running occasionally.

MRestore is probably your biggest bottleneck, but it's probably possible to thread everything. Look for the avisynth MT version in the avisynth forum, and follow its instructions on how to use SetMTMode.

I have no idea if xvid_encraw has a separate thread for input, like x264, at this point. If it does, it'd be the same speed as VDub, and it's probably a recent addition. But Hironimo did say that he uses VDMod, which has an input thread.

[dragongodz]

When I encode the CPU is not used 100%, but only 60 to 70%. Why? I would expect 100, but would be ok with 90.
oh boy.

please read the following link for a VERY basic explanation
http://forum.doom9.org/showthread.php?p=895927#post895927

[Hironimo]

oh boy.

please read the following link for a VERY basic explanation
http://forum.doom9.org/showthread.php?p=895927#post895927

I am aware that I will not get 100% more speed, i'm no newbie on how things work in a computer, being a programmer.
But you probably did not read this topic accurately, as this thread does not talk about speed improvement but how much of the cpu is actually used.

[dragongodz]

But you probably did not read this topic accurately, as this thread does not talk about speed improvement but how much of the cpu is actually used.
i am afraid it is you who needs to learn to read accurately. if you bothered to read the specific post i linked to(which was by me) i explain why you will not get 100% cpu use all the time.

[Ice =A=]

Well, Hironimo is right in a way. If he get's not much more then 60% cpu usage there can hardly be any real speed improvement, since a few percent cpu usage are from background programs and a few more would be multi thread overhead...
Cpu usage is kind of an upper limit to real speed (if you don't consider further codec optimizations).

[dragongodz]

hardly be any real speed improvement
that wasnt the question asked. infact he has replied

this thread does not talk about speed improvement but how much of the cpu is actually used.

and going back to the very first post

When I encode the CPU is not used 100%, but only 60 to 70%. Why? I would expect 100, but would be ok with 90.
expect 100% but ok with 90% , why ? because he thinks thats what it should be ?

in the link i gave i DO explain why both cpus may not always be used. i even went to the trouble to make it as simple as possible so it shouldnt confuse anyone.

so either people understand why they are not likely to see 100% dual core cpu usage, or necessarily anywhere near it depending on the program, or they dont. the question has been answered either way.

[Hironimo]

@dragongodz
You are right, I should have read more closely. I have stumbled over your post previously and thought I remembered what it was about, but I was wrong.

Your explanation is correct I think so:
- I do not expect to have double the performance with 2 cores.
- I do not expect the two cores to run at 100% if the 2 process are/need to be syncronized (which is the case here)

However:
60-70% cpu use (overall for both cores) is really low. Too low I think. This could be because the SMP code of XviD is not mature yet, but we can only guess - right?

I have run SMP encodes with x264 using the exact same configuration (same source and avisynth script) and got a CPU usage of 75-95% (so I guess 80-85 average) - this is much more like it and what I would like to see for XviD too!

Don't you agree?

[dragongodz]

60-70% cpu use (overall for both cores) is really low. Too low I think. This could be because the SMP code of XviD is not mature yet, but we can only guess - right?
depends. depends on whats been parelised etc. its a pity more active work doesnt appear to be going on with this for xvid. of course x264 appears to have had more work done in this area aswell so it shouldnt be too surprising if it is more efficient.

x264 using the exact same configuration (same source and avisynth script) and got a CPU usage of 75-95% (so I guess 80-85 average) - this is much more like it and what I would like to see for XviD too!

Don't you agree?
only if there is an actual speed benefit.

this is the problem with just looking at cpu usage, it doesnt actually tell you of any speed increase.
example, try increasing threads with xvid to say 3 for dual core cpu. you may get a bit more cpu usage and may even get a little more speed. however now try increasing it to say 5 and again you may see more cpu usage but may actually see it being slower than 2 or 3 threads. it really depends on how the threadings done etc.

read this post in the original xvid smp thread
http://forum.doom9.org/showthread.php?p=791030#post791030

actually have a read of the whole thread. theres lots of titbits in there.

[Hironimo]

Hi.

I ran a compare between XvidD and X264 again, this time without the timeconsuming Mrestore.

It turns out that then both use the Cpu at about 95% average. XviD estimated time of for the encode 46,5 minutes, and X264 about 64 minutes (factor of 1,376)

When you comare that with the times to run it with Mrestore: 121 minutes for XviD and 135 for x264 (factor of 1,115).

So indeed the bottleneck is Mrestore, and XviD encoding is not as fast compared to x264 as it should be, if the whole process ran optimally.

Why XviD uses that much less cpu is I think both because it is faster in encoding and so the bottleneck stands out more, but probably also because it is not as good in running multiple threads.

Still I wonder:
If Mencoder is the bottleneck, why would not at least this be running at full speed, causing one of the cores cause to be running at much higher usage then the other?

@dragongodz
Sure, using more of the cpu's power does not always mean faster encoding. In the example of more threads then the cpu has cores this is obvious because the overhead needed to syncronize all the threads uses up the time that would else have been used to do the actual encoding.

I did not say that cpu usage is the indicator for speed, but It can be used to see if the conversion and encoding process is done optimally or not. If you see that the cpu is not by fare used to It's potential this means something may be wrong, such as a bottleneck is slowing down everything.