XviD and Multi-Processor Systems (SMP)

[Buzz Lightyear]

I searched the forum, but couldn't find any recent threads on the SMP capabilities of XviD.
So here I go:
Does XviD support Multiprocessor Systems? (It doesn't look like it, right now...)
If not, is it planned for future releases?

(Btw, I'm not talking about HT. My brand-new Dual Xeon needs something to do, until I finished my FE models...)

[708145]

Well, I'm currently developing ELDER.
A parallEL encoDER for XviD. Look here (http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER) for current status.

edit 2005-07-08: ELDERchap V1.0RC1 released. http://www.iti.uni-stuttgart.de/~bergmats/dl/ELDERchap1.0RC1.tar.gz

bis besser,
T0B1A5

[mikeX]

AFAIK XviD does not take advantage of SMP (or HT) systems. The only benefit you can have from what I've heard so far, is that you can leave decoding/filtering etc for one CPU and encoding for the other...
The only way you can split encoding on more than one CPU is the way TOB1A5 mentioned, which is btw very similar to transcode's 'cluster mode' (I don't know how that one handles scene changes/chunk borders however)

[Buzz Lightyear]

Alright. Thanks for the answers!
I guess, SMP for XviD isn't really trivial.... ;)

[Buzz Lightyear]

Yesterday I discovered something strange.
I was encoding a movie with Koepis XviD-1.0.3-20122004 build and to my great surprise both (!) CPUs were used! Both at 100%! (Vdubmod had 100% CPU usage)
Is it possible that XviD supports SMP in the end?
I don't undestand that. I like it, but I don't understand it....
:confused:

[708145]

Well, unfortunately SMP code does not materialize randomly and by itself :p

So my guess is that avisynth used one CPU and xvid the other. If you didn't use avisynth (or other filters for that matter) then I don't know :confused:

bis besser,
Tobias

[BigDid]

Originally posted by 708145
Well, I'm currently developing ELDER.
A parallEL encoDER for XviD. Look here (http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER) for current status.

bis besser,
T0B1A5
Hello 708145,

Link down ?

Did

[708145]

Originally posted by BigDid
Hello 708145,

Link down ?

Did

The server was quite busy. It's resolved now.

bis besser,
Tobias

[Buzz Lightyear]

Originally posted by 708145
Well, unfortunately SMP code does not materialize randomly and by itself :p

Well... that's too bad! ;)

So my guess is that avisynth used one CPU and xvid the other. If you didn't use avisynth (or other filters for that matter) then I don't know :confused:


My mistake. I should have written that I use GordianKnot to setup my encodes. So that means that you're probably right with XviD + Avysynth. I'll check that...

EDIT1:
...looks like you're right. I've set up a new encode with different setting, and now CPU1 is at 100% and CPU2 is at 40%. So my previous results (both CPUs at ~100%) was purely coincidental.

EDIT2:
What the h***? At around 85% of pass 1, suddenly CPU2 jumped to ~100%, too! And for pass 2, both CPUs are constantly at ~80-100%! I don't understand that.... :confused:

[tdsfan]

How far along are you with ELDER? Is there anything to download?

[tdsfan]

I just found this page, it mentions you can compile xvid for smp, is this true??

http://www.discdude.net/xvid/compile.html

[708145]

Originally posted by tdsfan
How far along are you with ELDER? Is there anything to download?

Well, I've been in holidays for 5 weeks and just returned yesterday. Lots of work was piling up here, so => It will take some time till I can have a look at it.

Just bug me often enough, it helps for motivation and "finding" time. :p

bis besser,
T0B1A5

[Sharktooth]

im bugging you... ;)

[Ice =A=]

I'm bugging you, too... :)

@Buzz Lightyear:
Any new insights on your case?

[Shinigami-Sama]

I'll ginf hte heard I guess
BAAAAAA~
I wish I knew how to help I realy do..

[708145]

Just a small question for all the ELDER-lovers in here:

Which platforms do you use? I guess mainly Win32(maybe HT or SMP).
If so, I'll implement the Win32 version first.
*hint* I have some free time tomorrow. *hint*

bis besser,
Tobias

[Sharktooth]

Win32, linux and BSD... but mainly Win32.

[Shinigami-Sama]

win 32
lunix when I get a second pc maybe

[Buzz Lightyear]

@Ice =A=:
I haven't had time to look closer into that, but I think that 708145 is right and that XviD uses one CPU and Avisynth the other.
That's not "real SMP" but I don't care. There are 2 processes (XviD and Avisynth) and 2 CPUs and it's damn fast, so why bother? :)

@708145:
auch ein www.nichtlustig.de Fan, wie ich sehe :)

[movmasty]

Originally posted by Buzz Lightyear
EDIT1:
...looks like you're right. I've set up a new encode with different setting, and now CPU1 is at 100% and CPU2 is at 40%. So my previous results (both CPUs at ~100%) was purely coincidental.

EDIT2:
What the h***? At around 85% of pass 1, suddenly CPU2 jumped to ~100%, too! And for pass 2, both CPUs are constantly at ~80-100%! I don't understand that.... :confused:
Tell us about frame rate please...

[movmasty]

Ok, and what about the coming dual core cpu's?
will be the same thing?

Another question, differences among parallel and serial use of more cpu,
if its obviously difficult to split a task, you could always let the first cpu to do half job and then send to the other,

i mean decode and half the filters by the 1st cpu, then send the frame to the 2nd for the last filters and encoding, while the 1st cpu will decode the succesive frame...

In this way speed will really double....almost...no:confused:

[708145]

Originally posted by movmasty
Ok, and what about the coming dual core cpu's?
will be the same thing?

Another question, differences among parallel and serial use of more cpu,
if its obviously difficult to split a task, you coulsd alvays let the first cpu to do half job and then send to the other,

i mean decode and half the filters by the 1st cpu, then send the frame to the 2nd for the last filters and encoding, while the 1st cpu will decode the succesive frame...

In this way speed will really double....almost...no:confused:

Hi movmasty,

The method you mention is a kind of pipelining. The interface between the different pipiline stages might be a bit bandwidth consuming but will be no issue if all data-to-be-transferred fits into a shared cache in a multi-core setup.

IMHO this should be implemented in Avisynth 3.

And now a small note regarding the development status: The first release will definitely be Win32 SMP/Multicore since it is the most advanced version currently. I just lack a dual *windows* machine to test it on :( (And no, hyperthreading doesn't count!)

bis besser,
Tobias

[Buzz Lightyear]

@movmasty and all the other curious people:

I made a litte test today with a ~5min video clip (7485 frames, 25fps, Bitrate 1261 k(=1000)Bits/s, PAL Progressive) from a Harry Potter DVD.
I setup the encode with GKnot and used XviD 1.1.0 beta 2 (Koepi). Hypertherading was diabled, unless otherwise noted. CPUs are Xeons with 3.2 Ghz.

Run 1: "superquality" ;)
GKnot: litte noise, put noise before resize, Lanzos resize
XviD: profile AS@L5, adaptive quant., QPel, GMC, chroma optimizer, chroma motion, motion search 6, VHQ 4, _no_ Turbo, Trellis

Run 2: "fast"
GKnot: _no_ noise filter, Lanzos resize
XviD: profile AS@L5, adaptive quant., QPel, _no_ GMC, chroma optimizer, chroma motion, motion search 6, VHQ 1, Turbo, Trellis

Run 1: 12:38min - 13:13min - 16:02min (2CPUs - 2CPUs with HT - 1CPU)
Run 2: 5:01min - 5:05 - 7:36min (2CPUs - 2CPUs with HT - 1CPU)

This shows several things:
- 2CPUs are by far not twice as fast as 1 CPU
- using 1 CPU, the load was always 100%, using 2 CPUs the load on both CPUs was ~50..90% most of the time
- it depends on the options you use in GKnot/Avisynth and XviD if you get an (more or less) even load distribution between the 2 CPUs (Run1: 2CPUs are 27% faster than 1 CPU, Run2: 51% faster)
- in the past, I also had cases where both CPUs were running at ~95% nearly all the time. so that should give probably a ~60-80% advantage over 1 CPU
- using HT (with 2 CPUs) makes things a little bit slower (HT with 1 CPU not tested)

[movmasty]

Originally posted by Buzz Lightyear
@movmasty and all the other curious people:

This shows several things:
- 2CPUs are by far not twice as fast as 1 CPU
We knew this.

BUT

Are the program you tested optimized for a "serial use" of 2 cpus????

Pipelined in 708145 language.

[708145]

Originally posted by movmasty
Are the program you tested optimized for a "serial use" of 2 cpus????
Pipelined in 708145 language.

Yes. It is pipelined ("serial use" is misleading) but the pipeline stages are not well balanced in most of the cases.

The first stage is the filtering and scaling and the second is xvid. In those cases were both are equally computation intensive you get close to 100% speedup.
If you use only a resizer + insane xvid settings the speedup will be closer to 10%.
edit: decoding will be done in the first stage as well. /edit

bis besser,
Tobias

[Buzz Lightyear]

<trollmode>

@movmasty:
What EXACTLY do you want to know?
Originally posted by movmasty
Tell us about frame rate please...
First you ask for framerates (btw, that would be 25fps for PAL!), and then I do some benchmarks and submit the results, only to hear from you that:
Originally posted by movmasty
We knew this.
Well. Thanks for that. And then you ask
Originally posted by movmasty
Are the program you tested optimized for a "serial use" of 2 cpus????
Do you even read what I wrote before. I wrote that I used GKnot & XviD. What do you ask for the programs I use?
And if they are "optimized" for "serial use". How should I know? Did I write them?
And what's there to optimize in a serial or pipelined process? I don't get what you mean!

</trollmode>

[movmasty]

Originally posted by Buzz Lightyear
<trollmode>

@movmasty:
What EXACTLY do you want to know?

I don't get what you mean!

</trollmode>
i meant, what difference between ENCODING framerate(not pal framerate!) with 1 or 2 cpu??????????????????

[708145]

Hi,

I just want to tell you that I made quite some progress on the linux client which works quite OK now: 3.2x speedup on a 4 CPU SMP system.

Linux cluster mode gained 11.6x on 22 nodes so plenty of room for improvement :)

The current problem is that porting to windows is not as easy as I thought since I can't convince mencoder to map a directory of VOBs as a DVD. Current workaround would be to really have the DVD in a drive which supposedly kills the drive faster than normally. :(
Hopefully I find a nice solution.

bis besser,
Tobias

[movmasty]

Originally posted by 708145
Hi,

I can't convince mencoder to map a directory of VOBs as a DVD.
Tobias
Daemon Tools ?

[708145]

Originally posted by movmasty
Daemon Tools ?

I'd like to port it with the least amount of windows specific tools.
And I never used those Tools so I'll just test with a DVD in my drive and leave it to the users to fake the DVD in whichever way they want :p

bis besser,
Tobias

[Joe Fenton]

If you check the thread here (http://forum.doom9.org/showthread.php?s=&threadid=95097), you'll see that there has been a modification made to x264 that allows SMP systems to encode faster. My dual Opteron encodes about 50% faster! I was wondering if one of the xvid folks could take a peek and see if something similar was feasible for xvid. It would really be nice to get 50% more speed from xvid. :sly:

[708145]

Originally posted by Joe Fenton
If you check the thread here (http://forum.doom9.org/showthread.php?s=&threadid=95097), you'll see that there has been a modification made to x264 that allows SMP systems to encode faster. My dual Opteron encodes about 50% faster! I was wondering if one of the xvid folks could take a peek and see if something similar was feasible for xvid. It would really be nice to get 50% more speed from xvid. :sly:

Yes I read it too.
Unfortunately multi-slice is not defined in MPEG4 part2 IIRC.

There was some talking about encoding the b-frames in parallel in Xvid though. Don't know what came out of it.

As a sidenote: I got a nice speedup on a dual opteron using Handbrake (1 vs. 2 threads): speed almost doubled for 2nd pass. As I didn't control the filtering options, I can't say if it was Xvid or not that was responsible for that.

Second note: You should really switch to Linux alltogether :p For me it's already working there :) Just have to clean up the code a bit and release :rolleyes:
A windows port is really a big hassle and (at least for me) not worth it. But ELDER is OS so everybody is invited to port it :p

bis besser,
Tobias

[Joe Fenton]

Uh - I AM running linux. In fact, I currently can't run Windows on this system as Windows has no SATA support, and the SATA driver disk from VIA just gives an install error. So I just use Fedora Core 3 AMD64. It's nice. It's fast. It plays DOOM3. WINE plays American McGee's Alice. That's good enough for me. :D

However, I'm still working on the encoding part, so any suggestions on encoding in linux are greatly appreciated. Especially if they're 64bit. I've got x264 compiled as 64bit. I've got xvid as 64bit. I've got FFMPEG compiled as 64bit. Nearly all my AV stuff is all 64bit. The Windows codecs are still 32bit. :mad: Oh well, I'm lucky I can still use them.

[akupenguin]

Originally posted by 708145
Unfortunately multi-slice is not defined in MPEG4 part2 IIRC.

It is. It's called "Group Of Blocks" (Annex E.1.1). While the standard says it's specifically for error resilience, that really has exactly the same requirements as threaded encoding.
Multithreading via GOB is supported by libavcodec.

[708145]

It is. It's called "Group Of Blocks" (Annex E.1.1). While the standard says it's specifically for error resilience, that really has exactly the same requirements as threaded encoding.
Multithreading via GOB is supported by libavcodec.

Ah I remember. I think you told me once already :)

That means the speedup I saw in Handbrake was probably due to GOB.

bis besser,
Tobias

[Ice =A=]

Well, one month has passed, any updates? :)

[708145]

Well, one month has passed, any updates? :)

Still not gotten around to test RC1 of ELDERchap. Once I did I'll post it here.

bis besser,
Tobias

[Ice =A=]

Well, considering how common Xvid is with computer enthusiasts and how those will love multicore processors, you really would make yourself a name with a working ELDER!!! :) Just a little motivation, since I just got my new Athlon X2, which really wants something to do right now... ;)
I sure wish I was good enough to help with ELDER's developement!

P.S.: Jetzt nur nich nachlassen! ;)

[Doom9]

It is. It's called "Group Of Blocks" (Annex E.1.1). While the standard says it's specifically for error resilience, that really has exactly the same requirements as threaded encoding.
Multithreading via GOB is supported by libavcodec.Is there any noticeable drawback with this? Normally error resilience tools blow up the stream with redudant data, which would naturally lead to lower compression efficiency. But done in an acceptable manner like AVC slices, it certainly would make XviD more attractive as other codecs are catching up now speed-wise as they are being SMP optimized.

[708145]

Well, considering how common Xvid is with computer enthusiasts and how those will love multicore processors, you really would make yourself a name with a working ELDER!!! :) Just a little motivation, since I just got my new Athlon X2, which really wants something to do right now... ;)
I sure wish I was good enough to help with ELDER's developement!

P.S.: Jetzt nur nich nachlassen! ;)

Finally ELDERchap is approaching! :D

It's the little brother of ELDER as it doesn't support .avs input and does not balance the load very good. :p

I'm currently doing some speed tests with RC1 and I'll post the results here.

Oh, and last but not least: get it here: http://www.iti.uni-stuttgart.de/~bergmats/dl/ELDERchap1.0RC1.tar.gz
Info is here: http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER

I'm curious how it performs on Linux64 and SMP systems.
If someone wants to help me port this to Windows just speak up (if not I'll do it anyways but it will take much longer).

bis besser,
Tobias

[sysKin]

Is there any noticeable drawback with this? Normally error resilience tools blow up the stream with redudant data, which would naturally lead to lower compression efficiency. But done in an acceptable manner like AVC slices, it certainly would make XviD more attractive as other codecs are catching up now speed-wise as they are being SMP optimized.
It should have the same drawbacks as in x264, and they are not big (did someone exactly measure the psnr loss in x264?). As far as I can see, there will be a loss of 4..5 bytes for the GOB startcode and then some unspecified loss for motion and intra prediction, in one row of macroblocks. If this becomes 16 bytes per average P/I frame, and there are 12 P/I frames per second, that's a loss of 1.5kbps. Sounds doable.

What about decoder support for this?

If I find some time, I'd like to try(1) parallel B-frame encoding first - either by encoding many b-frames in parallel or one b-frame in mutiple slices, which is always possible without any changes to the stream.

(1) I've never coded SMP and it sounds like a useful/fun skill to have

Radek

[708145]

Syskin! :D

Nice to have you working on SMP support for XviD. :)

The RC1 undersized 2% in my experiments and gave near linear speedup.
2CPUs: 1.9x
4CPUs: 3.9x
8CPUs: 7.7x

bis besser,
Tobias

[sysKin]

Syskin! :D

Nice to have you working on SMP support for XviD. :)
Let's not go that far PLEASE :)

The RC1 undersized 2% in my experiments and gave near linear speedup.
2CPUs: 1.9x
4CPUs: 3.9x
8CPUs: 7.7x

bis besser,
Tobias
You rock! Do I really need to do anything, then?

[Doom9]

What about decoder support for this?I suppose since lavc already has that feature, if you use lavc for decoding xvid it should work out. I have yet to try the threads option in lavc though but they've just fixed my box (bad ram again.. that's the 2nd time I got a new box with Kingston HyperX RAM and the second time it's faulty.. next time I'll buy something else) so I'll be able to try over the week-end.

You rock! Do I really need to do anything, then?Well, it sounds like an elder port for Windows has become somewhat unlikely and it's an app meant for clusters and is too heavy for the regular end user.. most people use XviD VfW. Though perhaps they could live with a souped up encraw that has all features with AVS input and AVI/MP4 output (sorta like x264.exe) and using a GUI like MeGUI.. but then again most people use x264 VfW regardless of the fact that the commandline version is better.

[708145]

You rock! Do I really need to do anything, then?

Hehe, yes. There are a lot of things to do (volunteers welcome):
* I'm not sure if the bit distribution is good currently.
* use avisynth: it doesn't work for non-DVD source yet
* port to Windows: it only works in Linux so far
* target x264 and snow
* optional: integrate in MeGUI if there is some interest for it

The best way I see is a combination of vertical and horizontal parallelism, i.e. encoding per scene and parallelism by frame type or slice/GOB.

bis besser,
Tobias

[708145]

Well, it sounds like an elder port for Windows has become somewhat unlikely and it's an app meant for clusters and is too heavy for the regular end user.. most people use XviD VfW. Though perhaps they could live with a souped up encraw that has all features with AVS input and AVI/MP4 output (sorta like x264.exe) and using a GUI like MeGUI.. but then again most people use x264 VfW regardless of the fact that the commandline version is better.

Well, once it works reliably the goal is to do a usable windows port. But I fear it'll be SMP only for Windows.

Which reminds me: concatenating the chunks together and sound are still missing :p

bis besser,
Tobias

[sysKin]

Hehe, yes. There are a lot of things to do (volunteers welcome)With my parallel-bframes / group-of-block-parallel pframes, I meant...

But Doom9 answered that and he's right of course.

[708145]

Some more results from the speedup department:
1 CPU: 1.000
2 CPU: 1.985
4 CPU: 3.947
8 CPU: 7.830
16CPU: 9.330

all values compare to a run of ELDER with one CPU. I know I should compare to a single CPU run without ELDER. :rolleyes:

Has anyone tried it yet? The README is incomplete so just complain when something is missing.

Further improvements have to be done to allow good scaling beyond 8x. ;)

bis besser,
Tobias

[Shinigami-Sama]

dam 7081455
good work, nearly a full cpu worth of speed up by using this :)

[708145]

dam 7081455
good work, nearly a full cpu worth of speed up by using this :)

Are you talking about speedup on a HT CPU?
That would be great! :D

Please report some details.

bis besser,
Tobias