Xvid on Quadcore (Faster than Dualcore?) [Archive]

ilhyfe

19th July 2007, 22:21

I can max out both cores here so it should be possible to get faster speeds on a quadcore even if you can't max out all 4 cores.

Well I tried using avisynth MT and I ONLY get about 120 fps on encoding a DVD to XviD. I had 120 fps before on a c2d @ 3 GHz.
The strange thing is that I get these 120 fps only if I use 2 threads in the XviD configuration. If I enter 4, I get much less speed. Do you have any idea what I may doing wrong? I was expecting 160-200 fps.

celtic_druid

20th July 2007, 06:17

Could always try encoding two DVD's at a time.

ilhyfe

20th July 2007, 13:04

Could always try encoding two DVD's at a time.

Then the framerate drops to ~70fps each.

Romario

20th July 2007, 18:25

Seems that optimizations for quadcore isn't good.

708145

20th July 2007, 19:16

Seems that optimizations for quadcore isn't good.

Intels current quad cores have a memory bandwidth problem. They will fix it when the Phenoms appear I guess.

Romario

20th July 2007, 21:20

I hope so. Currently, quadcore is huge waste of our money.

themostestultimategenius

21st July 2007, 02:30

SGD$400 for the Q6600 after 22nd July. Isn't that expensive.

Blue_MiSfit

21st July 2007, 03:16

Seriously. $220 USD for a Q6600? Sign me up please..

Speed is same then on dualcore. Every CPU is on 50%.

Well then you're not utilizing the extra cores. Isn't ELDER supposed to be good at this kind of stuff? :)

~MiSfit

ilhyfe

22nd July 2007, 17:58

Seriously. $220 USD for a Q6600? Sign me up please..

Well then you're not utilizing the extra cores. Isn't ELDER supposed to be good at this kind of stuff? :)

~MiSfit

I tried it, but it doesn't start the jobs. I'll read the docs when I got some time, then test it again :)
Anyway, I don't think xvid on quadcore is that slow because of memory bandwidth problems.

Sharktooth

23rd July 2007, 18:26

i think so... it's due to the architectural problems i explained in HW forum a while ago.
The intel CPUs are not true quadcores and have no real interconnection between cores.
The situation gets more critical when you need to synch threads between cores 1/2 and 3/4.
The Core 2 xx50 series have a higher FSB but performance are not so "improved" coz the problem still lies INSIDE the CPU. The processor design is a "cheap" solution to glue 2 Core 2 Duo CPUs thru the FSB in one package and tell the world "We have the quad!" (marketing)... Also the Core 2 Duo is made up of 2 cores that talk sharing a cache...
So you can just imagine what could happen if you need to tranfer some data from, for example, core 1 to core 4 while performing other operations or what could happen when moving large data from memory... The whole thing will work at half speed if not lower... ;)

Manao

23rd July 2007, 19:25

Just to add the input : go see that (http://forum.doom9.org/showthread.php?p=1023900#post1023900)

You'll see that a Q6600 goes twice as fast as a E6600 with x264.

That means XviD isn't well adapted to multithreading, not that a Q6600 sucks.

Sharktooth : don't take that wrong, but whenever you speak of Intel and AMD, you seem completely biased.

Sharktooth

23rd July 2007, 19:50

c2q seems to handle x264 a bit better however xvid is not the unique multithreaded application having performance problems with c2q.
Here's what happens with large data blocks:
http://www.techspot.com/articles-info/36/bench/CPU_04.png
and
http://www.techspot.com/articles-info/36/bench/Appz_01.png

As you can see, there's little to no gain and even lower results!... the CPU is limited by its poor architecture.
Obviously there are OTHER tests that show more speed gain, but i chosen those 2 coz they use large memory blocks and are ideal to show what i was referring to...

Manao

23rd July 2007, 20:00

But x264 being better than xvid, threading wise, shows that what matters most is the algorithm behind the threading. I know how x264 and xvid are threaded, and even on a true quadcore, x264 would scale as well while xvid wouldn't.

Now, for the charts you giving, I don't know how the threading was done for that application, so how can I judge them ? Is there a true quadcore going twice as fast as its dualcore counterpart on this test ?

Sharktooth

23rd July 2007, 20:11

Theoretically a "true quad" should scale almost linearly and should not have those slowdowns.
That's why i always said the actual "fake" quads are a waste of money.
Obviously, if you need to run x264 24h/day then it will probably be a good choice.

Manao

23rd July 2007, 20:22

Theoretically a "true quad" should scale almost linearly and should not have those slowdownsNo, it's simply not true, it depends completely on the algorithm used for threading.

XviD, with 4 threads, cuts a frame 4 parts, and encodes those parts independently. Since the encoding speed varies on all macroblocks, the need for synchronization costs more than it seems : it will make some CPUs unused while waiting for the others. So the speed on the frame depends on the speed of the slowest part, hence not a perfect scaling ( whatever the CPUs )

Furthermore, the actual encoding isn't threaded, which reduces even more the scaling. It scales well on 2 cores only because the non threaded parts are threded alongside avisynth/the input. On 4 cores, the 2 additionnal cores aren't masked anymore.

Sharktooth

24th July 2007, 10:08

708145

24th July 2007, 19:21

I tried it, but it doesn't start the jobs. I'll read the docs when I got some time, then test it again :)
Anyway, I don't think xvid on quadcore is that slow because of memory bandwidth problems.

ELDER works for some folks and doesn't for others. Since I have holidays starting this thursday I'll have a look at it and publish a new revision.

Mutant_Fruit

25th July 2007, 00:42

i was talking in general. i meant, the true quads do not have the problems of FSB interconnection between the 2 groups of cores and those apps that show problems coz of that architecture limitation, should scale linearly unless the threading model is unoptimal.

In general, a 'true quadcore' will probably only scale marginally better than a 'fake quadcore'. The scaling is nearly entirely determined by the algorithms used by the program. In the real world it's often impossible to have an optimal threading model, or the complexity wouldn't make it worthwhile as compared to a less efficient 'sub-optimal' version.

The optimal threading model for video would be to encode each frame completely independently of any other frame. However that'd kill compression.

EDIT: Also, the only thing your graph shows is that xvid has a terrible multi-threading algorithm. Xvid just does not scale. It seems that whenever you talk about cpu models, intel == shit, amd == great. If i remember correctly you were advising people against buying C2D's several months ago because they weren't worth it and opterons were much better value, despite nearly every benchmark proving that to be false.

Manao

25th July 2007, 05:42

The optimal threading model for video would be to encode each frame completely independently of any other frame. However that'd kill compression.That's what x264 does ( the only dependency between frames is that enough of the reference frame must be encoded for the current one to start ), and it doesn't kill the compression. Elder does that too ( for x264 & xvid ), and it doesn't kill the compression either.

Mutant_Fruit

25th July 2007, 05:56

I can assure you, no video encoder encodes video frames completely independently of each other. I think you may have misunderstood what i meant. If a video frame was encoded 'independently' then it would not reference data from another frame and no other frames would reference it. Basically, it'd be like encoding a series of jpg's one after the other.

Every video encoder uses redundancy between different frames as a main form of compression. This limits the potential of any threading algorithm as frame 2 can't be started until the required info from frame1 has been calculated.

Elder works by splitting up one long video sequence into a series of shorter sequences. It doesn't encode individual frames as still images completely independent from each other and sew them back up into a video.

Sharktooth

25th July 2007, 17:31

at that time the c2d CPUs were much less cheap than now and AMD is still cheaper at present days, so if you had a limited budget, buying an e6700 was not properly the best idea... but that's not what we're discussing now.
The main fact is xvid doesnt scale well on an actual quadcore.
Here's what comes to my mind:
Xvid divides the frames and encode the parts separately on their own thread.
Those parts are dependant and some data must be "shared" by the cores. The c2q architectural limit kicks in while moving data thru FSB to make it available to each core...
Furthermore synching threads make the situation even worse.
Those factors in conjunction with a suboptimal threading model is the cause of xvid's speed scaling problems.

Romario

25th July 2007, 18:21

Hmm, it's possible that you are right, Sharktooth.

But, in your opinion, can Xvid be modded for OPTIMAL use of quadcore, because 3D programs (3D Studio MAX 9, Maya 8.5) can use quadcore in optimal way.

Mutant_Fruit

25th July 2007, 21:50

at that time the c2d CPUs were much less cheap than now
The price has dropped by about 20 euro since i bought my 6300 a year ago.

The main fact is xvid doesnt scale well on an actual quadcore.
Here's what comes to my mind:
Xvid divides the frames and encode the parts separately on their own thread.
Those parts are dependant and some data must be "shared" by the cores. The c2q architectural limit kicks in while moving data thru FSB to make it available to each core...
Furthermore synching threads make the situation even worse.
Those factors in conjunction with a suboptimal threading model is the cause of xvid's speed scaling problems.

Well, as you so kindly pointed out, there are plenty of other benchmarks which support the fact that the 'fake' quadcore scales pretty well. As further proof that the quadcore scales really well for video encoding, look at x264. It performs the exact same task as xvid, yet it can get about 350% processor usage on a quadcore (400% being the max). That's pretty damn good and also a very reasonable efficiency to achieve.

Thing is you bashed the C2D's months ago for being a 'cheap hack', now you're bashing the quadcores for being a 'cheap hack'. The truth is, they perform exceedingly well in software capable of taking advantage of the extra cores.

celtic_druid

26th July 2007, 05:21

I've seen at least one report of ~100% usage across all cores on a quad. They were getting much lower usage until they started using AVISynth MT.

Shinigami-Sama

26th July 2007, 06:19

Rule 4 guys...

anyways I have to agree with shark, the quadcores right now are hackish, next gen will be fixed just like the c2ds were fixed

also I have to say if you want to get them all over 80% you might have to play with thread priority a bit, I had to do that a few times back when I played with Vdub more than I do now( <3 ripitforme )

Sharktooth

26th July 2007, 12:45

To sum it up, im telling actual quadcores from intel have some bandwidth and cores interconnection problems.
When a software needs to share data between the cores or elaborate some huge data sets there is a consistent performance loss.
Yes, it's avoidable, but the coder must think about it when writing the code or you'll get a marginal gain or even a performance loss.
That's called CPU specific optimizations (or CPU specific code path) and will force the coder to add more CPU specific code to take advantage of that particular processor.
That wouldnt be necessary if the CPU was planned using brain instead of a$$... ;)

Manao

26th July 2007, 12:57

That's called CPU specific optimizations (or CPU specific code path) and will force the coder to add more CPU specific code to take advantage of that particular processor.No, it's not CPU specific. Even "true" multicores benefit from x264's threading model in comparison to xvid's. So it only forces the coder to do the algorithm only once, in the proper way.

Stop spinning facts, you won't be able to hide the "current" truth : A Q6600 costs 250€, and it's the most powerful stuff you can get for that price, for video encoding purposes.

Sharktooth

26th July 2007, 13:09

Manao, the x264 multithreading code is more efficient than xvid's one and doesnt require much dependancy between threads. That's why c2q performs so good with it.
However im not talking about xvid or x264. I also said if you plan to use x264 on a regular basis then you would be happy with a "fake" quad too, but if you also plan to use other apps which may expose the c2q limits (for example photoshop), then wait for true quads (they're just behind the corner) or at least get the cheap c2qs versions (q6600 or similar). Spending 500+ euros on a fake quad is just wasted money.
Penryn and Barcelona (and their derivations) will have a significant performance boost, much lower power consumption and new technologies support (including DDR3). First samples will be out on august...

Mutant_Fruit

26th July 2007, 15:05

Manao, the x264 multithreading code is more efficient than xvid's one and doesnt require much dependancy between threads. That's why c2q performs so good with it.
And thats the exact reason why xvid will not scale even on a 'true' multicore cpu as you so like to call them.The more interdependence, the more waiting around, the less work you can actually get done per second. Fact of life. It's true on both 'true' and 'fake' quadcores.

Fine, you name Photoshop as an application that doesn't have any speedup when run with a quadcore as opposed to dual core. I say that's because the program itself has been programmed to only run on 2 cores. It would be interesting to see if running complex operations in photoshop only results in 2 cores maxing out and 2 cores remaining idle. That'd indicate photoshop doesn't support 4 cores.

To be honest, Sharktooth is just a fanboi for AMD. The consistent Intel bashing and consistent touting of AMD as the only manufacturer of chips worth buying just proves the point. The truth is AMD are horribly late with their answer to the C2D and C2Q. Intel are mopping the floor with their chips and have been for the last year.

Unfortunately we're going to have to have to wait for AMD to release their quadcores so we can compare benchmarks between it and intels quadcore before he'll realise that a 'true' quadcore still hits limits far below it's potential, just like the 'fake' quadcore does.

EDIT: http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3038&p=6 <-- Look at all the benchmarks.

EDIT2: I'd also like to add that AMD's are significantly cheaper than C2D's. Yuo're absolutely right. I believe AMD have one chip over $200, with only 2 others above the $150 mark. That's because they are just not competitive. If AMD priced their chips any higher, then just wouldn't sell because they'd be vastly outperformed by similarly priced C2D chips. The architecture is old and crap at this stage and just can't stand up to the new designs from intel. The source of that info is from the link above and should be easily verified at the AMD website if you care to check it out.

Shinigami-Sama

26th July 2007, 17:05

No, it's not CPU specific. Even "true" multicores benefit from x264's threading model in comparison to xvid's. So it only forces the coder to do the algorithm only once, in the proper way.

Stop spinning facts, you won't be able to hide the "current" truth : A Q6600 costs 250€, and it's the most powerful stuff you can get for that price, for video encoding purposes.

I'd like to add based on what I know about x264, the way AVC works ALLOWs for its thread model to be more threadable than Xvid. That said Xvid also has an older threading model that could PROBABLY get a speed up, but the quadcores currently need to have things programed specifically for them right now, the next ones will have the cores all talking without the FSB(I hope) which will significantly speed up encoding. AMDs just don't that limitation right now due to their design.

Manao

26th July 2007, 17:43

I'd like to add based on what I know about x264, the way AVC works ALLOWs for its thread model to be more threadable than XvidNo.

Guys, please, don't affirm stuff when you're not sure of you. Multireference increases temporal constraints, it doesn't reduce it. Intra prediction in AVC ( in comparison to ASP )increases spatial constraints ( which isn't used anyway with x264's threading model ). Finally BReference increases the number of reference frames and thus decreases the inherent threadability of bframes.

There is nothing in AVC that makes it inherently more threadable that ASP.

XviD can be threaded a la x264, there's just nobody wanting to do it.

And quadcores doesn't need to be specifically programmed - I know Mutant_Fruits and me are repeating ourselves over and over, but you keep adding incorrect/false statements to the mess.

AMDs just don't that limitation right now due to their designDo you mean that, by design, AMD not having quadcores make them more threadable ?

Mutant_Fruit

26th July 2007, 19:44

The way AVC works ALLOWs for its thread model to be more threadable than Xvid.
Even if that were true (which it ain't), the point is this is *not* the fault of a 'fake' quadcore which is what Sharktooth is trying to say. It's purely the fault of a bad threading model in Xvid which doesn't scale.

quadcores currently need to have things programed specifically for them right now
Nope, they don't. X264 doesn't have 'quadcore specific' optimisations. It just threads as needed. However the threading algorithm limits at about 500-550% i believe, but once again, that's not the fault of quadcores or 8-core chips. It's a fault of the algorithm and the inherent serial nature of video encoding.

the next ones will have the cores all talking without the FSB(I hope) which will significantly speed up encoding. AMDs just don't that limitation right now due to their design.
Nope, when AMD finally get around to releasing their quadcores they'll have (at most) a few % in the difference as compared to 'fake' quadcores. I'd expect about 1-3% increase at the most. That's my prediction.

ilhyfe

27th July 2007, 09:32

Lalelu...

I have a 320% speedup on Cinebench 9.5 with 4 core's instead of one. Thats near the theoretical maximum. Unfortunately there's no cheap 2 x dualcore or "real" quadcore solution (for 250 Euro).

Mutant_Fruit

27th July 2007, 15:13

Theoretical maximum is 400%, so you're still a fair bit from it. Generally speaking i'd say an efficiency of 340-360% is achieveable.[quote]

[quote]Unfortunately there's no cheap 2 x dualcore or "real" quadcore solution (for 250 Euro).
The Q6600 is dropping to a mere $266 in the very near future. At that price, the only reason you could say no is if you're a hardcore camera. Games still benefit from a single really fast core, or at least a dualcore with higher clocked cores. Still, i'm fairly tempted to get one, AMD has nothing to compete with it really, nothing comes close.

Sharktooth

27th July 2007, 17:27

Just to be clear i didnt talk at all about AMD in this thread except in my last post where i talked about 2 codenames of 2 true quads from both Intel (Penryn) and AMD (Barcelona)...
It's just your pregiudice M_F... and no, photoshop use all the 4 cores, it scales well on dual cores coz they have a shared cache and bad on actual fake quads due to the huge bandwidth limit of the FSB (cache is many many many times faster...)
If only Intel made all 4 cores share the same L2 cache the actual quads would be much faster in "particular situations"... the same situations that made the Pentium D a poor CPU...

Mutant_Fruit

27th July 2007, 18:16

It's just your pregiudice M_F... and no, photoshop use all the 4 cores, it scales well on dual cores coz they have a shared cache and bad on actual fake quads due to the huge bandwidth limit of the FSB (cache is many many many times faster...)
Or it could just not scale well because it's not really able to take advantage of more than 2 cores effectively. What you're blaming on 'huge bandwidth limits' is probably more likely to be a bad multithreading algorithm.

On the other hand, some applications showed hardly any improvement at all for Toms testers, such as Photoshop CS2 and Quake 4, for which there is a quad-core patch forthcoming. However, as soon as software developers wrap their minds around the quad-core way of doing things, serious speedup will ensue. AMD is not going to be caught napping; expect that company's quad-core processor in a couple of weeks. Toms Hardware's Patrick Schmid's opinion, and lots more including Tech Spot's intense quad-core testing, after the jump.

Toms hardware are usually Intel biased, so take their review with a grain of salt, but teh 3D studio benchmark and Adobe Premiere benchmark show the power of the new core.

http://www.tomshardware.com/2006/11/02/kentsfield_released/

To be honest, lets leave this drop until AMD gets around to releasing their 'true' quadcore. Then give me a ping and we'll continue the discussion. I'm not budging from my position, you're still not convinced that it's simply bad algorithms in certain programs that make the 'fake' quad underperform.

So. When AMD releases their 'true' quadcore, i predict that for the same clockspeed there will be a minor performance increase due to the 'true quadcore' architecture. Performance in photoshop will still be as terrible with AMD's quad as it is with intels quad.

JnZ

6th August 2007, 12:39

Hi gyus,
a few weeks ago I get twos AMD s940 Opterons 285 for a good price and build a small encoding machine. Today I'm making some tests but at this point, it looks that x264 can use all cores better, than xvid and with xvid, it depends, which build I use (i have some own ICL MT CVS builds, and 1.2-127 SMP from koepi).

Speed for x264:
1 x X2@2500MHz (dual core) ~ 15fps
2 x Opt.285@2600MhZ (quad cores) ~ 32fps
---------------------------------------------
Both machines almost on 100% cpu usage.

Speed for XviD (Q2 one pass encoding):
1 x X2@2500MHz (dual core) ~ 20fps (CPU usage about 80%)
2 x Opt.285@2600MhZ (quad cores) ~ 27fps (MSVC build) (CPU usage only 50%)

I can't get over 50% with XviD. I set up 4 threads. x264 encoding is litlle big speedup, but Xvid not. :-(

Sharktooth

6th August 2007, 13:22

x264 MT model scales better than the xvid's one. however you got a 7fps boost. consider that as a bonus over intel quadcore cpus owners that gets no speedup at all... ;)

JnZ

6th August 2007, 13:32

x264 MT model scales better than the xvid's one. however you got a 7fps boost. consider that as a bonus over intel quadcore cpus owners that gets no speedup at all... ;)

What a pitty, that someone don't develop XviD and optimize it for multicore.I encode most DVD to XviD. Maybe it's time to go for x264.

Manao

6th August 2007, 13:39

however you got a 7fps boost. consider that as a bonus over intel quadcore cpus owners that gets no speedup at all...It's not the same CPU, nor the same frequency, and we don't know anything about the avs script / the encoding settings. So you can strike that remark as unfounded...

Sharktooth

6th August 2007, 13:58

uhm... there's only 100mhz of difference...
as x264 shows the opt285 scales at 2x (+1 FPS due to the 100Mhz bonus) the performance with a good MT algo...
you may also not know that socket 940 opterons are practically identical to socket 939 athlons (and even AM2's) except for the pinout and the multiprocessor support (which is disabled in 1xx series and by consequence it has 939 pins instead of 940 and it's socket compatible with athlon cpus).

JnZ

6th August 2007, 14:27

It's not the same CPU, nor the same frequency, and we don't know anything about the avs script / the encoding settings. So you can strike that remark as unfounded...

Yes, as sharktooth says: Opt285 and X2 differs only in 100MHz and 1MB (opt285) vs 512kb (x2) per core cache. Anyway speed of x264 are boosted more than twice. (some extra fps gives 100MHz and 1Mb cache, but not much). Considering almost double speedup, it's not bad at all.

But if you compare Xvid, quad core give only 7fps boost (80% cpu usage on X2 vs 50% usage on opt285). In x264 encoding all cpus wotk at almost 100%.

So only one explanation is bad MT support in XviD, which is pitty. XviD is very usefull for DVD backups etc.