Quadcore or server processor for x264?

[Dust Signs]

Hi,

I hope that this topic fits here (I put it here because it's hardware related).
I plan to save some money to buy a new system next spring to speed up x264 encoding (significantly). I currently encode on a So939 Athlon 64 X2 4200+. Due to a huge data loss I have to encode a lot of movies again which costs me a lot of time - that's one of the reasons why a speedup would be nice. I currently encode with about 50fps in the first and 10-14fps (depending) in the second pass (using HQ Slower in StaxRip with 2 threads) when encoding a 720x288 movie.
The big question is now: buy a core quadro or a server processor? I don't have any experience at all with server processors, but I heard that x264 could benefit of the huge cache sizes. On the other hand, I read benchmarks of the new Intel Core 2 QX (http://hardware.thgweb.de/2006/11/01/intel_core_2_extreme_qx6700/page23.html) which consisted of a performance test with the Mainconcept H.264 encoder - and compared to my current processor it's 2,4 times faster which would be quite nice. I think that these results can be adapted to x264.
Now the question: what would be more advantageous? I don't want to spend a lot more than 1500€ for the new PC, but I planned to make it an encoding PC only - so a cheaper mainboard, an older harddisk etc. will do quite well. Server processor or quadcore (which is in fact no real quadcore)? The only thing that really matters here is performance - the maximum price I'm willing to pay for a CPU is between 700 and 800€ (absolute max). What would you say? Any suggestions?

Dust Signs

[Sharktooth]

Core2 Quad is not a big performer (i mean it's not up to the hype). It's the fastest CPU available but the performance are not scaled to what you could expect from a quad-core.
http://www.tweaktown.com/articles/974/
I would wait for a true Quad-Core before spending 999$ for a CPU... they should be available next spring/summer...

[foxyshadis]

Yes, a set of benchmarks revolving around games and synthetics (with one single solitary encoding test that shows a fair gain) is entirely applicable to this situation. The Register did the same thing, I wonder if they've ever even heard of the mystical "photoshop" "after effects" and "compile linux" tests. You know, those things where threading helps.

But I still agree, wait to see what the prices turn out to be before you drop a wad like that, and perhaps wait until K8L and Clovertown match up in the next big clash (and prices of Duos and X2s drop even further).

[Blue_MiSfit]

The [H]ard-OCP tests seemed to indicate that the quad delivered ~ 35% improved performance in DivX encoding over the X6800. That seems pretty good to me!

[Dust Signs]

Ok, so the Intel "quadcore" would now be a waste of money, you think? What about server processors?
As I already said: the only purpose of my new PC will be x264 encoding; I'll keep my current one for all the other things I have to do.

Dust Signs

[Blue_MiSfit]

if by server processors you mean the new Core 2 Duo based Xeon processors, they are a pretty sweet deal. But they aren't much faster last time I checked, and they need a socket 771 board, which usually means ECC RAM etc... Not worth it. If you have that much $ just get a freaking MacPro and run boot camp :)

[Dust Signs]

I mean every server processor available/affordable ;). I read this 5 minutes ago: http://www.amd.com/us-en/Corporate/VirtualPressRoom/0,,51_104_609,00.html
I mean if I can be below the 1500€ total it doesn't matter how much the mainboard or the RAM costs... if I get someting for my money which is visibly faster (ideally double or more) than my current system.

Dust Signs

[bkman]

Core2 Quad is not a big performer (i mean it's not up to the hype). It's the fastest CPU available but the performance are not scaled to what you could expect from a quad-core.
http://www.tweaktown.com/articles/974/
I would wait for a true Quad-Core before spending 999$ for a CPU... they should be available next spring/summer...

What do you expect from a quad core on apps that aren't coded to take advantage of that many cores? Not much, that's what.

So basically, you can't conclude much from Kentsfield tests compared to to other quad-cores unless you have some other quad-cores to compare with. Oops, your bias is showing again.

[quake74]

Let me contribute some benchmark on *interesting* applications like DivX, XviD, DVDShrink: http://www.hardcoreware.net/reviews/review-347-4.htm

[Dust Signs]

DVDShrink is very interesting. Unfortunately none of the bigger hardware sites benchmarks x264 :(

Dust Signs

[Sharktooth]

What do you expect from a quad core on apps that aren't coded to take advantage of that many cores? Not much, that's what.

So basically, you can't conclude much from Kentsfield tests compared to to other quad-cores unless you have some other quad-cores to compare with. Oops, your bias is showing again.
Ah yeah... and what you expected from pentium D? kentsfield is the same... 2 dual core connected thru (a even lower than Core2 CPUs) FSB.
Performance CANT be on par with true Quad-Cores and it costs really too much.
I would prefer a dual socket Core2Duo solution... higher FSB, more bandwidth etc.
I see kentsfield like an intent to find a "cheap" (moneywise) solution to put 4 cores in a system, but IMHO they failed coz they sell that behemot at 999$...
Also since Dust Signs wants something for spring, i suggested him to wait for TRUE quad-cores (Intel or AMD) that will surely perform much better than kentsfield.
However, bkman, read the rules again... your offenses are not welcome...

[bkman]

That wasn't an insult, just an observation. I wish you would be more objective when advising someone, and not state things which cannot be backed up empirically.

Edit: Btw topic creator, according to this (http://www.ixbt.com/cpu/images/intel-core2-extreme-qx6700/results.xls), from here (http://www.digit-life.com/articles2/cpu/intel-core2-extreme-qx6700.html), x264 encoding on a Core 2 Quad is about 65%* faster compared to a dual-core C2D at the same clocks. This may or may not improve with a so-called "true" quad-core design, so you might have to try something like ELDER x264 to make more effective use of the 4 cores. It seems the Mainconcepts encoder scales a little more evenly, reaching about 70% improvement per clock vs dual-core.

* I think that's right. I suck at math.

[Dust Signs]

Elder sounds nice. I hope that I can integrate it somehow in Staxrip, because I currently encode with it :)

Dust Signs

[Romario]

I don't like how new Intel Core 2 Quad encode video? It's shame, I will wait true quad core from AMD.

[Inventive Software]

AMD scales better with more cores than Intel does, but I'm waiting for AMD to produce a response to Core 2 Duo before I even think about 4 core CPUs.

[Doom9]

I would prefer a dual socket Core2Duo solution... higher FSB, more bandwidth etc.And an eccentric and expensive solution at that. Dual slot boards have been available since 2000.. they never took off. There was HT which helped about as much as it hurt.. it took two cores in one socket to make the whole thing mainstream compatible.
Intel's quad core is continuing with intel's quick fix strategy.. put two separate processory in the same socket.. but by now, they're no longer trying to hide it.

The Pentium D came long before the X2.. now as AMD doesn't want to wait this time over they bring back dual socket boards, but that's no innovation either.. dual socket solutions have supported dual core chips for a while now.. but the cost ensures they'll never become a mainstream solution. 4x4 is just AMD's "me too" to intel's "we're first again" (but not quite doing it right) approach.
We'll have to wait and see if a proper quad core design really does so much better.. adding a second core didn't increase encoding performance by a 100%, so I'm not expecting any wonders from going from two to four cores, regardless of the design.

[dragongodz]

adding a second core didn't increase encoding performance by a 100%, so I'm not expecting any wonders from going from two to four cores, regardless of the design.
and of course people shouldnt. the simple fact is not all functions take the same amount of time. so for a very dumbed down example

-process a takes 5 cycles
-process b takes 2 cycles and can be done at the same time as process a
-process c requires both processes a and b to be finished(needs the final data from both) before it can be done.

will that give 100% cpus all the time ? no , because process b will be finished before process a and process c can not start until it has all the data from both. so the second cpu can sit idle while process a finishes. also if process c has no other process that can be performed at the same time then that also means the second cpu could be idle during thats operation aswell.

now of course the ideal is to always have something for all cpus to do however this is unrealistic. meaning it simply can not always be done. so any speed increase is dependant upon what processes can be done at the same time and how much wait time there is when not.

[Doom9]

very good explanation :)

To further illustrate upon that point, there are jobs that can be parallelized more efficiently than others.. take xbitlab's chess benchmark for instance: http://www.xbitlabs.com/articles/cpu/display/core2extreme-qx6700_9.html - it scales almost linearly.

[Sharktooth]

Well, having all data in cache and running indipendent threads is always an ideal situation.
However, "real life" situations are a bit different... and the kentsfield limits will show up.
For example, having linked threads executed in cores 1 and 3 (or 2 and 4) will surely kill the kentsfield performances and there is no way (in hardware) to prevent it. So basically, software must be optimized for kentsfield (and not just for multiple threads) at the risk it will run slower on future multi-core processors.

[Inventive Software]

Surely the best way to optimise specifically for Kentsfield, purely based on its core architecture, is to have threads 1 and 2 operate on cores 1 and 2 together, but have threads 3 and 4 operate together on 3 and 4, and only get the result when the 4 threads are complete? Or split the process into 2 groups of 2 threads...

I may be completely wrong on this, but that's how I see it at the moment.

I still think AMD's 4-core option will be better, cause the memory controller's on the CPU, and they link together without the FSB. But we're waiting till mid-2007 for pure 4-core options from Intel and AMD. (Pure being linked cache and all cores being able to talk to each other without FSB.)

[Sharktooth]

Well, as i said, kentsfield is a "cheap" solution...

[Doom9]

However, "real life" situations are a bit different.And speaking of real life, we'll see how the current quad core chips stack up when native quad core chips are out... at that point we'll also know if, due to the non linear performance scaling of most real world applications, the difference between a "two chips on one die", and "one chip" will be as significant as it was when going from one to two cores.

[foxyshadis]

Surely the best way to optimise specifically for Kentsfield, purely based on its core architecture, is to have threads 1 and 2 operate on cores 1 and 2 together, but have threads 3 and 4 operate together on 3 and 4, and only get the result when the 4 threads are complete? Or split the process into 2 groups of 2 threads...
The scheduler has no idea which threads use the same cache data. However, the best design is for all threads to use independant data anyway; after all, you can never assume that two linked threads will run at exactly the same time in order to cooperatively read and modify each other's data, unless you enjoy race conditions. No matter which core each thread's running on, no matter how many dies/cpus, anytime two threads need to read & write data from each other performance suffers considerably. Only reading from the same data w/o writing at all will show minor performance benefits to a single large cache over two separate caches.

The real problem with two-die is heat, and threads getting scheduled randomly and constantly having to refill their caches (both Windows and Linux/Unix schedulers attempt to avoid this), if they use very little data. Video processing churns through so much data that it rarely holds any cache beyond a full quantum, and the smarter prefetch in Core and K8L makes more difference.

[Romario]

I may be completely wrong on this, but that's how I see it at the moment.

I still think AMD's 4-core option will be better, cause the memory controller's on the CPU, and they link together without the FSB. But we're waiting till mid-2007 for pure 4-core options from Intel and AMD. (Pure being linked cache and all cores being able to talk to each other without FSB.)

Absolutely, you got the point, Inventive Software. I think that AMD true quadcore (Barselona) will be much better solution for x264 codec then Core 2 Quad.

[soresu]

Seeing as the AMD Barcelona cores optimisations/enhancements are mostly at the FPU end... does anyone know roughly how much the x264 code depends on integer and floating point calculations separately (ratio).

[Romario]

Well, I don't know that, soresu, but I think that it's not a problem to optimize x264 for that.

Perhaps, devs should think about two separated versions of x264: one for Core 2 architecture (for Quad Cores), and one for Athlon 64/AMD Barselona architecture.

[soresu]

As far as i'm aware Romario its not so much optimising for floating point or integer as certain operations probably are suited for one or the other, my point was that i was wondering which specific parts of the x264 codebase would get the biggest boost from coming FPU beast that will be Barcelona (assuming of course that IPC enhancements in the core wont do much for integer operations specifically).

[Inventive Software]

AMD need good SSE and SSE2 performance IMO, as that's what most codecs use, x264, Xvid and DivX included.

[Sharktooth]

Will get it with K8L architecture. Dont forget AMD is adding the same 128bit SSE execution as it was introduced in Core2.
However both Intel and AMD "true" Quad-cores will surely be faster than kentsfield due to a better cores management/data exchange that will lead to a better performance scaling.
They will also have a much better TDP so they will eat less current and be cooler...

[708145]

Elder sounds nice. I hope that I can integrate it somehow in Staxrip, because I currently encode with it :)


ELDER has a easy to use CLI, so inclusion to other GUIs is very welcome :)

bis besser,
T0B1A5

[Sagittaire]

Will get it with K8L architecture. Dont forget AMD is adding the same 128bit SSE execution as it was introduced in Core2.
However both Intel and AMD "true" Quad-cores will surely be faster than kentsfield due to a better cores management/data exchange that will lead to a better performance scaling.
They will also have a much better TDP so they will eat less current and be cooler...

We will see that but at this time ...
http://www.tomshardware.com/2006/07/14/core2_duo_knocks_out_athlon_64/page18.html

- For same clock Intel's solution are by far better than AMD's solution. You can read that in all hardware testing from the web.
- Intel's solution is by far better than AMD's solution (price, performance, overclocking potential ...). You can read that in all hardware testing from the web.

... it's the reality and anything else. Intel's quad processor is by far better than AMD's quad processor simply because at this time AMD's quad processor don't exist. AMD quad core CPU performance is pure speculation. At this time Intel dual core is 25% better than AMD dual core for same clock with better max speed limit. IMO "cheap" Intel's quad core will be always better than "new" AMD quad core.


PS: I'am not a Intel fan ... my CPU is a AMD at this time.

[Mutant_Fruit]

AMD scales better with more cores than Intel does, but I'm waiting for AMD to produce a response to Core 2 Duo before I even think about 4 core CPUs.
Care to prove that? Everything i've seen begs to differ.

[Sharktooth]

We will see that but at this time ...
http://www.tomshardware.com/2006/07/14/core2_duo_knocks_out_athlon_64/page18.html

- For same clock Intel's solution are by far better than AMD's solution. You can read that in all hardware testing from the web.
- Intel's solution is by far better than AMD's solution (price, performance, overclocking potential ...). You can read that in all hardware testing from the web.

... it's the reality and anything else. Intel's quad processor is by far better than AMD's quad processor simply because at this time AMD's quad processor don't exist. AMD quad core CPU performance is pure speculation. At this time Intel dual core is 25% better than AMD dual core for same clock with better max speed limit. IMO "cheap" Intel's quad core will be always better than "new" AMD quad core.


PS: I'am not a Intel fan ... my CPU is a AMD at this time.
Well, you also should read twice before posting.
I said both AMD and Intel TRUE quad-core solutions will be BETTER than kentsfield and that AMD is adding 128Bit SSE execution (and not only that) to their next core.
So what's up with all "this is better than that", "you can read it here and there"...?
Also your "IMO "cheap" Intel's quad core will be always better than "new" AMD quad core" statement cant be proved and it's pure speculation, while mine is quite obvious: newer CPUs will always perform better than older ones...

[swaaye]

LOL. Well, just how far off do you guys think AMD's real quad core is? I think it's a LONG ways off. That 4X4 nonsense is not going to compete with Kentsfield just as a X2 can't compete with a C2D; the computational resources of Conroe way overshadow K8.

So, by the time AMD's "true" quad core arrives, Intel will likely have its answer to that in the form of a 45 nm C2Q and, well, that will probably be the end of that trumping attempt. AMD is a year behind in process tech, and Intel no longer has a problem with their CPU core tech. That spells tough times for AMD.

IMO most of Kentsfields problems are just as have been explained in this thread: the incredible challenge of making good multithreaded apps. Sure multithreading has been around for decades, but not in most apps that desktop systems run. It's a real issue right now and so these new dual cores and quad cores aren't being used optimally at all. And I have no doubt that 4 cores means even less efficiency right now because even multithreaded apps aren't necessarily designed for a 4 CPU system.

[Sharktooth]

Well, only time will tell. In the last period AMD focused his research on power saving technologies, while intel focused on performance.
The difference is Intel gone thru 3 core techs before regaining the performace crown, in the meanwhile AMD kept a single core tech and is perfectioning the K9 core. However the K8 core was born with the multi-core in mind and it's far from dead, it will still give some headaces to Intel (with the K8L additions).
As i said only time will tell...
For what concerns the multicore efficiency, most things depends on softwares. Im sure kentsfield is underperforming due to software limitations but i believe true multicore solutions will perform a bit better.
However the chip giants are all walking the multi-core way, so software makers will soon adapt to the situation.

[EagleDM]

I already posted a graph showing, not a 30% difference, but a very HUGE one against dual core processors on this thread

http://forum.doom9.org/showthread.php?p=897887#post897887

THe review should be up soon and I will give you the link when is up (probably tomorrow)

the reason maybe a lot of hardware pages are not showing big increases is probably because they use automated encoding with default settings (not recommended)

As I found out, the more cores you add, the more INTENSIVE the calculations have to be in order to use more portion of the cores..

Example. DivX configured to balanced gives about 60% of CPU ussage (each core) witch in total gives about a 60% total CPU ussage (all cores sum up)

with Extreme Quality configured, the encoder gives a harder work to the cores and you have about 80% cpu ussage on each core and a total CPU ussage of about 80-85%, which of course, improves the result.

As a matter of fact, the encoding on "Balanced" end up at the same time as the "Extreme" preset on the QuadCore because of the better use of each core.

Same applies to x264.

[708145]

As I found out, the more cores you add, the more INTENSIVE the calculations have to be in order to use more portion of the cores..


That's true for most video codecs. A possible workaround is to use GOP level parallelism (as in ELDER, my little pet) and get almost linear speedup _regardless_ of the codec options.

bis besser,
T0B1A5

[foxyshadis]

As a matter of fact, the encoding on "Balanced" end up at the same time as the "Extreme" preset on the QuadCore because of the better use of each core.

Same applies to x264.

That means it's simply decoder-bound. (Or preprocessing if any.) If you were to thread the decoding and any processing filters, it would probably peg near 100%.

But although multithreading is more interesting, multiprocess (eg, ELDER) is probably the future of video coding, since it's so much more scalable and independant of input bottlenecks. That includes thread-processes, where two essentially independant threads work on different pieces of video, just like two processes. (Most servers work that way, though some are true multiprocess.)

[zzuser]

I already posted a graph showing, not a 30% difference, but a very HUGE one against dual core processors on this thread

Could you please run four instances of mencoder on four different files to see how much faster parallel encoding is for batch processing? If you have access to a two core CPU it would be really great if you then could run the same four files on that to compare.

- Zed

[soresu]

While this has little to do with multicore CPU's, I saw this about a new C/C++ compiler for nVidia GPU's...

http://www.betanews.com/article/nVidia_DirectX_10_Graphics_Cards_Do_CPU_Work/1163022823

Seems as if nVidia is finally pulling their thumbs out of their collective behinds and getting GPGPU for the masses running.

[bkman]

I already posted a graph showing, not a 30% difference, but a very HUGE one against dual core processors on this thread

http://forum.doom9.org/showthread.php?p=897887#post897887.

Based on that graph you posted, the quad-core cpu shows an approximately 60% increase in speed over dual-core by my calculations. Remember that taking 0.5 the time is a 100% speed increase. The speed difference is easier to calculate when represented by frames per second, rather than time.

Edit: Oh sorry, I skimmed your post and though you were the guy that was saying it was only a 30% difference. My mistake >_<

[EagleDM]

Review is up, you can read the Encoding results here

http://www.maximopc.org/articulos/intel_core2quad_qx6700_parte_2.html

[Sharktooth]

Your review shows also an overclocked a64 x2 3800+ is faster at encoding with x264VFW than a non-overclocked c2d e6600.
Also it seems the OCed 3800+ is giving hard times to e6600 even in multitasking...
Well, e6600 can be overclocked wildly as well though...

[EagleDM]

Yes, that's right, (nice observation!!!)

That's because the X2 3800+ has a divided Cache, whereas the Conroe has a unified Cache architectura, I talked about this on other forums, as much as Intel want us to believe this is all good and no bad, it certainly has it's downsides, one of them is this.

If multiple processes are running (multitasking) with no compatible code that can be shared between Cache, the processor and Cache incurrs in a penalty, having to dynamically resize and divide the Cache for each process each core is running, under this condition, the divided cache (each core has it's own cache of X2 3800+) has the definitive advantage, not incurring in any penalty and handling each process with its own memory.

Also remember that the C2D architecture still don't have a Integrated Memory Controller that it will be out in the next Socket revision of Intel. But, as you well said in your post, the C2D is overclockeable to hell, and with that minor detail in the calculation, the result should catapult in favor of the C2D.

[Sharktooth]

Well, the intel next socket revision will be out in 2007 in about the same timeframe when AMD will be ready with their new K8L architecture.
As i previously said K8L integrates some new stuff (L3 cache, 128bit SSE execution, HyperTransport 3.0, etc...) that should heat up the performance battle again.
Well... only time will tell...

[Inventive Software]

And I was just about to upgrade to C2D too. :( Means it'll be obsolete by spring.

[foxyshadis]

You'll always be at best a year from obselescence, if you're always aiming for the top performance. That's why you shouldn't lay out the big bucks on the top model unless you just can't afford to wait another 4-6 months. (The most spectacular recent flameout has to be the P-EE 840, going from $1200 to $400 to $250 in just over a year.)

[dvd_maniac]

I was about to lay down some cash for a new X264 encoding rig with a Core 2 Duo and I'm glad I read this thread first...
It seems from all the benchmarks that X264 works better with Amd's X2?

[bkman]

It seems from all the benchmarks that X264 works better with Amd's X2?

No...

[Sharktooth]

No... c2d is faster.
I just pointed out an overclocked x2 3800+ can be faster at encoding with x264 than a non-overclocked e6600.
However the 6700 and x6800 are faster than the e6600 and ALL c2d CPUs can be overclocked to hell.
My point was if you dont want to spend too much money the AMD platform seems to be more "future proof" and cheaper keeping the compatibility with new sockets (AM2+ and AMD3) and still on track with low and mid-range CPUs performance at low prices (both motherboards and CPUs).