Updated: CCE SP Trial v2.70.02.00 & AMD Athlon64 X2 - CPU Utilization

[samuelal]

Hello everyone

I would like to know if there is a way to make the above CCE version utilize the entire 100% out of each core in the Athlon64 X2 processor?

Currently, I get around 70-75% CPU utilization out of each core while encoding with CCE (I am using the Windows' Task Manager application for monitoring the CPU utilization for that matter).

The following is the AVS file which is being created for each cell in each backup project I run for Rebuilder:
LoadPlugin("C:\Program Files\DVD-RB\DGDecode.dll")
mpeg2source("C:\RB\DVDTEST\D2VAVS\V01.D2V",idct=5)
trim(0,3206)
ConvertToYUY2()
AudioDub(BlankClip())
// IDCT=5 is for SSE2/MMX enabled CPUs which applies to my processor.

* Trying something silly like changing the priority to High doesn't have any effect on this (naturally).

* Furthermore - As I understand, version 2.70 and above are infact dual-core/dual-processor aware in contrast to previous versions.

UPDATE Dec 29th 17:46 GMT +01:00: http://forum.doom9.org/showthread.php?p=758908#post758908

Thank you for looking into this.

Samuel.

[Fishman0919]

What ver of windows are you runnig... Win XP... Home or Pro?

For MS web site..

IOW, you can set up Windows 2003 Server on a dual core, dual
CPU Opteron system and only have to pay for a single 2003
license.

But both XP Pro and Home are licensed on a per installation
basis, not per processor. XP Pro supports SMP and can use
up to two physical CPUs, and they can be dual core (for a
total of 4 processors).

XP Home is not SMP capable at all, and as such only supports
a single core, single CPU configuration. It will work with
dual core CPUs, but the second core is effectively useless.

[samuelal]

What ver of windows are you runnig... Win XP... Home or Pro?
Hello Fishman0919!

I am using the Windows XP Pro w/ SP2.

I am aware of the fact that this OS is in fact Dual-Core/CPU aware but that doesn't answer the question.

But thank you anyways!


P.S. I'm getting abit bummed down since this post has over 175 viewing hits, yet it seems that no one has found a solution for it or interested in investigating the issue with me.

[Just Me]

Hi!

Well I got the same problem using my AMD X2 3800+ 2.0@2.5 GHz CPU

If I run the CCE 2.70.02.01 I get around 65-70% CPU load using a simple AVS script without any filters.


LoadPlugin("C:\Programme\AviSynth 2.5\plugins\DGDecode.dll")
Mpeg2Source("E:\test.d2v")
LanczosResize(720,432,1,73,716,430)
AddBorders(0,72,0,72)
ConvertToYUY2()


The CCE speed moves within the range of 2,5 up to 3 and that is not good for a X2 CPU.

I am using Win XP Prof. SP2 and all following updates.

Greeting
Just Me

p.s. Excuse my bad English

[samuelal]

Hi!

Well I got the same problem using my AMD X2 3800+ 2.0@2.5 GHz CPU

If I run the CCE 2.70.02.01 I get around 65-70% CPU load using a simple AVS script without any filters.
......
Hello Just Me!

Thanks for adding your situation to the thread.

I've also added here (and at the begining of the thread) my part of the AVS file being used each time Rebuilder runs:

LoadPlugin("C:\Program Files\DVD-RB\DGDecode.dll")
mpeg2source("C:\RB\DVDTEST\D2VAVS\V01.D2V",idct=5)
trim(0,3206)
ConvertToYUY2()
AudioDub(BlankClip())
// IDCT=5 is for SSE2/MMX enabled CPUs which applies to my processor.

I am hoping that sooner than later someone could shed some light on this somewhat strange (and CPU-cycle-wastful) situation.

Thanks again.

Samuel.

[Doom9]

and CPU-cycle-wastfulWho says anybody is wasting CPU cycles? Do you understand what multithreaded programming really involves? Try this metaphor: you have a task at work, a task designed for a single person. You have a colleague that has some time on his hands and he offers his assistance. Since the task is essentially a single person task, there are certain things that need to be done in a certain order, other parts of the task can be done in an arbitrary order and are not interdependent. So you can give your colleague things to do that depend on what you already completed and those that are independent of one another, but not things that depend on the completion of your current subtask. Similarly, at some point, you and your colleague need to wait for one another before the next subtask can be started. So, in the end even though two people are working on the task, it will not be completed in half the time, but somewhere in between half the time and the time it would've taken you had you completed the task on your own.
Depending on how you can divide tasks into subtasks and schedule their execution, having two people work on the same problem can be more or less efficient, and the only way you can reach full utilization of both people is if you give them completely separate tasks so they can work on their own. That would be the situation of having two PCs doing something separate. In case of dual core, you only have two brains, but you share your arms and legs. If a job requires only the use of one hand, that's no problem, but if both hands are required, that'll slow down the other job if it also requires the use of a hand.

It's no different in multithreaded programming. Video compression is inheritently interdependent because the interdependence of frames (P frames depending on previous frames as an example), and you have a shared bus, memory and IO devices. But if you have an idea on how to attein 100% efficiency (not use of CPU cycles.. waiting for another thread to do something can also be made to use CPU cycles but that would be an actual waste), I'm sure you have a good future in the software business..

[samuelal]

Who says anybody is wasting CPU cycles?...

Hello Doom9.

I appreciate the time and thought you've taken into writing your response, but to my opinion it has nothing to do with what I seek for - a solution for the program to utilize the entire 100% out of each core - which is easily possible with other programs such as 3D-rendering, CPU-stress and what have you.

Something is preventing this particular app from using the entire 100% out of each CPU-Core and not just on my machine - it could be a bug or maybe a feature inherent in its code or whatever.

And those 25-30% out of each core not being used by CCE is a waste of computing power since all the rest of the processes running on my machine (see the sig) don't amount to even 1-2% total while its running.

In the mean time, I am hoping that someone could post in this thread an answer to my question as to why myself and others are seeing 75% on each core while encoding with CCE 2.70.xx.xx .

Thanks.

Samuel.

[samuelal]

Ok, I've decided to take the time and show to whoever is interested - Doom9, that could apply to you as well - that there is in fact a way to utilize both cores at 100% while encoding.

I took the same test project, and below you will see what happens on both the HC Batch Encoder and the CCE:

HC Batch v05-12-2005 | Rumbah's DVDDualRB RC1 (http://forum.doom9.org/showthread.php?p=692474) | IDCT=5 | NAN's DGDecode.dll v1.1.0+ SSE2 (http://forum.doom9.org/showthread.php?t=101353) | AviSynth v2.5.5 | PAL Source | 25fps | Movie Only:
LoadPlugin("C:\Program Files\DVD-RB\DGDecode.dll")
mpeg2source("C:\RB\DVD-TEST-WORK\D2VAVS\V01.D2V",idct=5)
trim(0,3206)
ConvertToYV12()http://img517.imageshack.us/img517/7997/hcbatchdualcoretest5wn.th.jpg (http://img517.imageshack.us/my.php?image=hcbatchdualcoretest5wn.jpg)

CCE SP Trial v2.70.02.00 | IDCT=5 | NAN's DGDecode.dll v1.1.0+ SSE2 (http://forum.doom9.org/showthread.php?t=101353) | AviSynth v2.5.5 | PAL Source | 25fps | Movie Only:
LoadPlugin("C:\Program Files\DVD-RB\DGDecode.dll")
mpeg2source("C:\RB\DVD-TEST-WORK\D2VAVS\V01.D2V",idct=5)
trim(0,3206)
ConvertToYV12()http://img517.imageshack.us/img517/4936/ccedualcoretest0zk.th.jpg (http://img517.imageshack.us/my.php?image=ccedualcoretest0zk.jpg)


I hope this helps to whoever can shed some light on this.

Samuel.

[Just Me]

@ samuelal

If I try the HC Encoder 0.16A I get around 50% CPU utilization :confused: :(

Edit: Oh you are using two HC encoders at the same time.

[samuelal]

@ samuelal

If I try the HC Encoder 0.16A I get around 50% CPU utilization :confused: :(

Edit: Oh you are using two HC encoders at the same time.
Yup, when using the DVDDualRB by Rumbah (it's linked in my post above), you can actually encode two sessions at once in conjunction with DVD-Rebuilder (it was actually made for it and very easy to setup), and it keeps feeding both sessions of HC Encoders the correct avs each time the previous once has finished.

Great stuff.

But still waiting for an answer for CCE if anyone knows.

Samuel.

[Doom9]

Actually, may I make the humble request that you read my answer again.. it explains exactly why things happen the way they do.

Your second scenario with two HC's running concurrently is included in my metaphor.. it's the case that you have two times the same job and have two people working on it.. one person per job. They do not have to share hands (memory/IO), and they do not have to wait for one another (frame interdependence). Just because there are SMP optimized programs that can make use of both cores to the fullest extent, doesn't mean this applies to video encoding, different scenarios result in different ways to utilize cores, and you'd need two threads running concurrently without requiring any synchronization and waiting and with as little IO and memory sharing so that they do not have any negative influence upon each other. This is for instance no problem for a 3d rendering application, because each frame of the output is rendered completely separate and independent of each other.

I have a feeling you still won't believe me, so I challenge you to find me a video encoder that is 100% efficient.. I have used plenty and they never use both cores to the fullest extent, but if you restrict them to using one core only, they use the one to the fullest extent. This is because if the app is single threaded (it isn't really but that's beside the point.. all GUI apps have at least two thread unless the GUI blocks while the application is working), the program spends all CPU time on reading the frame, then on encoding it, etc. Or an even better challenge: write a 100% efficient video encoder so you can prove the world you're the only one who knows better...

[samuelal]

Doom9

Thank you for explaining the situation to me.

I now understand it better than before in conjunction with your latest post.

This explains clearly why I got the results I did with the two instances of HC Encoder in contrast to the single-dual-core-optimize-instance of CCE.

Thanks again!

Samuel.

[chilledoutuk]

The real question is that is there any way possible that you could run two instances of cce. If there was a way to run two instances of cce then because of the way rebuilder encodes on a chapter by chapter basis each cpu and do its own chapter which would be the best way to encode.
there surley must be a way of fooling cce into letting it run multiple instances as i can run 2.67 and 2.7 at the same time.

[Zep]

The real question is that is there any way possible that you could run two instances of cce. If there was a way to run two instances of cce then because of the way rebuilder encodes on a chapter by chapter basis each cpu and do its own chapter which would be the best way to encode.
there surley must be a way of fooling cce into letting it run multiple instances as i can run 2.67 and 2.7 at the same time.

well not sure about cce but i max at 100% on both cores
when i encode using avisynth (without MT version) with Vdub and
syskin's SMP Xvid. No need for multiple instances of anything.

IMHO his bottle neck is else where. Odds are SLOW RAM
and/or slow read from hard drive like say if you are reading the
source and writing the encode to the same drive and if that drive
is also the boot drive then you have paging and other
read/writes fighting for time as well. (just launch an app
from same drive as the source video and watch your FPS tank)

anyway it appears both the avs and the encoder are sitting there
waiting for frames that are not coming in fast enough so the
CPU load is low.

he could try avisynth256MT3 and see if that helps but
if it is a ram BW bottle neck not much you can do other than
get faster ram.

[Ice =A=]

@chilledoutuk: You could indeed split the file/movie in two parts and encode the two in two different threads and (given that there is no other bottleneck like Zep mentioned) you would get about 100$% cpu usage. However, the possiblity that the two parts would be finished in exactly the same time is very unlikely, so measured over the time of the whole encoding you would have NO 100% cpu usage. So back at point 1.
Doom9 just is right with what he wrote! Which is somehow a tragedy, since otherwise those huge supercomputers with thousands of processors would be much, much faster then they are now, but computing speed just does not rise linearly with cpus, not even close to that...

[Blue_MiSfit]

well not sure about cce but i max at 100% on both cores
when i encode using avisynth (without MT version) with Vdub and
syskin's SMP Xvid. No need for multiple instances of anything.

IMHO his bottle neck is else where. Odds are SLOW RAM
and/or slow read from hard drive like say if you are reading the
source and writing the encode to the same drive and if that drive
is also the boot drive then you have paging and other
read/writes fighting for time as well. (just launch an app
from same drive as the source video and watch your FPS tank)

anyway it appears both the avs and the encoder are sitting there
waiting for frames that are not coming in fast enough so the
CPU load is low.

he could try avisynth256MT3 and see if that helps but
if it is a ram BW bottle neck not much you can do other than
get faster ram.

Um... no. DDR400 (no matter what the timings) is not "SLOW RAM". It is the industry standard. Memory timings make at most a 5% difference in overall performance. He's overclocked to DDR500 speeds anyway. I dont see how you could remotely consider this to be "SLOW RAM". PC100 is "SLOW RAM".

There is no possible way that "SLOW RAM" could be the cause of not getting 100% CPU utalization. If he had "SLOW RAM" (DDR266 for example) he would see slower FPS overall, but the CPU utalization would change very little. You should carefully read Doom9's post on why video encoders typically do not exhibit 100% utalization on SMP systems. It really is an elegant explanation that is quite simple to understand.

That said, you are are correct in stating that having that many processes running - all presumably heavily using the swap file on the boot drive would significantly reduce performance. This is very much the case, and explains why I never do encoding work on my boot drive - always on the SATA II 16mb cache NCQ drive :)

Your assumptions were logical, but I think you would benefit from a slightly deeper understanding of how video encoding works, and how to identify system performance bottlenecks.

Listen to the guys that write the code. They know a lot more than we do :)

~misfit

[NaN]

The problem is that avisynth and the decoder isn't multithreaded. Especially the decoder needs cpu-time that is delivered by 1 core. If this core has - say 75% - cpu-power left for cce, only 75% of the 2nd core can be used since the work load gets distributed evenly on both cores.

@samuelal: you can try my recent release (it's DGDecode 1.1.1 at http://forum.doom9.org/showthread.php?t=105484), it is not faster itself, but brings the new faster version of Skals idct (idct=6) that should account for 5%+ encoding speed.

[Just Me]

But why do I get only 65-75% CPU utalization using DVD-RB 1.05.2 while a friend of mine getting 100% CPU utalization using the same version and encoder?

I never encode on my boot drive.

p.s. Of course he is also using a X2 DualCore CPU.