Multithreaded XviD - official thread

[GodofaGap]

I know VirtualDub uses separate threads, but it could be it is introduced in the 1.16.x branch. (Of which there is no VDM version)

[DaForce]

ahh i see.
Right might stick with avisynth and virtualdub then.

Still i would expect alot faster than my friend p4-d than what im getting. x264 is much better.

ahh well.

Thanks Mate

[Zep]

ahh i see.
Right might stick with avisynth and virtualdub then.

Still i would expect alot faster than my friend p4-d than what im getting. x264 is much better.

ahh well.

Thanks Mate

Something is wrong on your box. I have the same CPU as you and I get 70+ FPS on 720x576 MPEG2 source video. You are not even reaching real time of 25FPS which is just crazy. If i shut off 1 core I get about 40 FPS. I use avisynth to feed to Vdub.

[DaForce]

Hey Zep, thanks for your reply man.

Well exactly it should be running faster in xvid. However are you converting from 720x576 to a smaller size ?
As in the test we are doing we are just converting from 720x576 mpeg2 to 720x576 XVID.

I while ago i convert some DVD vob files to a smaller xvid file and was getting about 90FPS on the final pass.

In rendering test (3D) its time is spot on with other x2 4400's so its not the box as a whole but maybe some of the codecs or something.

Would you be interesting in testing the file that we are using? Would certainly let me know if there problem is with my machine.

I just tried converting to 320x256 and it was getting over 100fps on the first pass and about 85 average on the pass.
But for the test we were not resizing the footage.

[Dreassica]

I'm seeing no speedup on using smp 1.2 version compared to old singlethreaded xvid, using same avs and settings. I do notice 2nd core having considerable less load, 28% against 96 for core 1. I have xp patched etc, so thats not it.

[foxyshadis]

Framerate's only going to be as fast as the slowest part of it. If you open a new virtualdub, load the script, and use analyse video, you'll get the maximum speed any encoder could possibly run at. To get more you'd have to change your script.

To really test xvid mt's speed, try loading a plain avi file instead of a script.

[Zep]

Hey Zep, thanks for your reply man.

Well exactly it should be running faster in xvid. However are you converting from 720x576 to a smaller size ?
As in the test we are doing we are just converting from 720x576 mpeg2 to 720x576 XVID.

I while ago i convert some DVD vob files to a smaller xvid file and was getting about 90FPS on the final pass.

In rendering test (3D) its time is spot on with other x2 4400's so its not the box as a whole but maybe some of the codecs or something.

Would you be interesting in testing the file that we are using? Would certainly let me know if there problem is with my machine.

I just tried converting to 320x256 and it was getting over 100fps on the first pass and about 85 average on the pass.
But for the test we were not resizing the footage.

makes no huge difference really. The CPU eaten via the resize is gotten back because Xvid now encodes a lot less. Basically i get about the same FPS if i do not resize or if i do to 320 x 256. The CPU usage is just in a different area. In this case avisynth more than Xvid when resizing down. Note i use multi threaded avisynth so the resize is fast compared to the avisynth most are using.

let me give you an example. I JUST encoded a HDTV show 720p to 624 x 352 and got a steady 68 FPS. Both source rez and file size i/o is much greater than DVD input. In this case if I do NOT resize down I get only about 56 FPS because Xvid has to encode a lot more and the resize for me is much faster than the Xvid settings I'm using (Max quality)


Oh BTW I use PC4000 and run it @ 267MHz and it makes a huge difference also since avisynth/vdub/xvid/encoding are mega memory read/writes etc...

[DaForce]

cool thanks for your info.. really helps.

When i last tried converting.. without resize it was about 30fps i think.. with resize down to 320x240 (or so ) it was up to 100fps.

And last night i noticed my memory settings were POV.. bloody CAS3.. (dont know why ) so they are not back to cas2 and tighter timings will try again later tonight.

Thanks again.

[shpitz]

If you open a new virtualdub, load the script, and use analyse video, you'll get the maximum speed any encoder could possibly run at.

can you explain how to do that? i don't any analyze option in the vdub menus.

Note i use multi threaded avisynth so the resize is fast compared to the avisynth most are using.

let me give you an example. I JUST encoded a HDTV show 720p to 624 x 352 and got a steady 68 FPS. Both source rez and file size i/o is much greater than DVD input.

can you point me to the avisynth version you're using?

can you post your script that you get 68fps on HD material?
which OS are you using? you use vdub or vdubmod?

thanks

[Selur]

can you explain how to do that? i don't any analyze option in the vdub menus.
File->Run video analysis pass
(using Virtual Dub 1.6.16)

[shpitz]

thanks Selur

[_xxl]

checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking for pthread_join in -lpthread... yes

No problem here.
Latest xvid 20070101.I don't know why
"checking for pthread_create in -lpthread" is disabled.
:confused:
http://i11.tinypic.com/3ywgrc9.jpg

[ChronoCross]

Latest xvid 20070101.I don't know why
"checking for pthread_create in -lpthread" is disabled.
:confused:
http://i11.tinypic.com/3ywgrc9.jpg

same here

[celtic_druid]

checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking for pthread_join in -lpthread... yes

Check your config.log.

[_xxl]

Can you share MinGW and Msys dir?

[Yong]

Could someone please test this?
Xvid vfw and encraw(original) compiled with pthreads support
http://www.mytempdir.com/1145230
I only have p4 presscot so i cant test it :p

@drevil_xxl:
You could try this http://sourceware.org/pthreads-win32/
Donwload the source, then
make GC-inlined
copy the libpthreadGC2.a to msys/lib
rename the libpthreadGC2.a to libpthread.a
Configure xvid again see if its work.

[celtic_druid]

I wouldn't rename since some apps still expect lpthreadGC2. Windows equiv of ln -s is just to copy it though.

My mingw dir is over 1GB.

As I said, check your config.log. If something isn't detected you can generally see why and fix it.

[_xxl]

copy the libpthreadGC2.a to msys/lib
rename the libpthreadGC2.a to libpthread.a
Configure xvid again see if its work.
Works!

I have tested xvid with AMD X2.
1).xvid 1.2-127 multithreaded:
http://i16.tinypic.com/4733s4i.jpg
Total time 2:23s

2).xvid 1.1.2 no multithread:
http://i12.tinypic.com/33m3fja.jpg
Total time 3:12s

You're seeing is the kernel's habit of tossing threads around in strange fashions. It doesn't really affect performance to execute half-and-half on two cpus instead of one.

[pixelk]

As sysKin website seems down, A firend compiled the latest 1.2.x source for me, you can get the binaires ( core + wfv ) here :
http://www.knackes.com/blog/index.php?2007/09/16/151-xvid-multithread

[Lenny_Nero]

I have said quite a few times before that like for like testing needs to have the same hard drives and data on them, I get about a 30% speed change depending on which way I send to and from my 7200 rpm drives (SATA I only) and even more if going to and from the 10k drives.

Same for a big clean empty space, and again even more speed if the cluster size is tuned to the OS cache working set size, only then can you get the full use of the CPU[s].

[MacAddict]

(snip) and again even more speed if the cluster size is tuned to the OS cache working set size, only then can you get the full use of the CPU[s].

Any guides or sites that help out on these recommendations?

[Mutant_Fruit]

The fastest way to 'improve' performance (multithreaded and singlethreaded) would be to run the reading of the data from the disk in a separate thread and buffer 1-5 frames of data in memory so that when xvid itself requires data, it's already in memory and so doesn't stall waiting for the data to be read from disk. x264 does this.

If that's already done in xvid, then ignore my suggestion. For someone competent in C, it'd probably be less than a 1 hour hack (assuming xvid does all it's reading from a specific function which can be altered and doesn't require the shotgun approach of changing dozens of areas) .

[Lenny_Nero]

Any guides or sites that help out on these recommendations?
Most of my own work is just that, I have always mucked about with Hard drive cluster sizes because it was the first things my dad used to have to setup with stuff. Hdd were a bit bigger/smaller then (10 x 12 inch platters got you 40 MB) but the data is still read on and off in the same way

I have been racking my brain for the keys in the registry I will post it in this thread ASA I remember but [HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\FileSystem]
has quite a few under it, or get to the other computer with the text file with my workings. There is a sysinternal's tool (IIRC) that can sort of do the same in a very simple sort of way.

I dont use XP or Vista (spit) for windows I use NT4 and NT5 AS (win2000) and 64bit 2003.

As for link's these can go some of the way to help.
http://www-128.ibm.com/developerworks/lotus/library/ls-Rules_WinNT2000/#A%20few%20tuning%20tips
http://www-128.ibm.com/developerworks/lotus/library/ls-Rules_WinNT2000/
http://www.ntcompatible.com/thread29591-1.html
http://www.dfwsug.org/cookbook.html
http://www.teamapproach.ca/trouble/CacheSize.htm
http://www.virtualdub.org/blog/pivot/entry.php?id=71

[squid_80]

The fastest way to 'improve' performance (multithreaded and singlethreaded) would be to run the reading of the data from the disk in a separate thread and buffer 1-5 frames of data in memory so that when xvid itself requires data, it's already in memory and so doesn't stall waiting for the data to be read from disk. x264 does this.

If that's already done in xvid, then ignore my suggestion. For someone competent in C, it'd probably be less than a 1 hour hack (assuming xvid does all it's reading from a specific function which can be altered and doesn't require the shotgun approach of changing dozens of areas) .

I added it to xvid_encraw just over a month ago. Virtualdub has been doing it forever.

[HarryM]

I added it to xvid_encraw just over a month ago. Virtualdub has been doing it forever.

It's good idea! But better is "customizable -read ahead-".Using the 1st pass statistics from the 2pass encoding process (in this case you know future frame type in the encoded data - significant informations, of course).

When you set e.g. 30 frames ahead reading (input buffer = 30 frames), you can theoretically start one new internal-encoding thread, if I-frames are closer than 30 frames.
But you need output buffer (=30 frames) too, for encoded frames sorting. Minor improvement, but still improvement.

If you set input buffer=(I-frames distance), you can ALWAYS use two internal encoding threads. But you need gigant input buffer at size ~100 to 200MB for distance bigger than 250.
What is 100MB of memory, today? Nothing.:)

It is idea for three- and quad-core cpu, mainly.

[Mutant_Fruit]

When you set e.g. 30 frames ahead reading (input buffer = 30 frames), you can theoretically start one new internal-encoding thread, if I-frames are closer than 30 frames.
The benefits seem marginal at best. If the encoder wants 30 frames, it can read them in, and thereafter it will only request 1 at a time (as once it finishes with 1 frame out of that 30, it'll request one more). So there's no need for the disc buffer to retain 30 (or more) frames itself.

I hope that makes sense.

[Zach]

After reading through all seven pages, I feel I'm crashing a private party here, but I'll ask anyway. :)

I take it the only way to get and test this SMP version of xvid is to download the source and compile it one's self? (The original link is now dead, but its over a year old, so that's understable.)

[imcold]

There are CVS compiles from Celtic Druid at http://mirror.celticdruid.info/XviD/ .

[Zach]

Sure, but those aren't the multi-threaded variants, are they?

You mean v.1.1.3 on Celtic Druid's site is the multi-threaded counterpart to the "official" one (found here (http://www.xvid.org/Downloads.43.0.html))?

[foxyshadis]

Anything that says "cvs head" means the latest & greatest up-to-the-minute ("nightly") builds of the development code, which right now means 1.2.

Of course they're a few months old now, I wonder if any big fixes have been introduced in the meantime.

[pixelk]

Sure, but those aren't the multi-threaded variants, are they?

You mean v.1.1.3 on Celtic Druid's site is the multi-threaded counterpart to the "official" one (found here (http://www.xvid.org/Downloads.43.0.html))?

I compile the last CVS every few days. Get the binary here : http://www.knackes.com/blog/index.php?2007/09/16/151-xvid-multithread&link1

[foxyshadis]

Thanks for the heads up! Current builds are always appreciated.

[Zach]

I compile the last CVS every few days. Get the binary here : http://www.knackes.com/blog/index.php?2007/09/16/151-xvid-multithread&link1

Thanks!

[MacAddict]

I compile the last CVS every few days. Get the binary here : http://www.knackes.com/blog/index.php?2007/09/16/151-xvid-multithread&link1

Thanks for your builds, much appreciated. I seem to be having more frequent crashes using xvid_encraw via megui with the 10-14 build compared to the September builds. Anyone else?

[pixelk]

xvidcore_20071015 binary should work much better.

[|sawo|]

I have q6600 @ 3GHz and i got 55fps with 4 threads (mpeg2 source, 1500kbps 1 pass xvid)
The fps is pretty much the same as the regular xvid and the cpu usage is around 50%
With the same settings im able to get ~111fps using divx 6 (also i set crop,resize and deinterlace directly from the divx panel almost without fps drop!)

[Ice =A=]

The usual answer: What are "same settings"?:devil::sly::confused:

[|sawo|]

1500kbps 1pass balanced profile for divx.
Even with the Insane Quality preset activated in divx6 im still able to get around ~68fps which is far more than xvid.
Also dont forget that i have many additional options such as resize,deinterlace,crop etc activated in the divx6 panel!

[Zach]

I have q6600 @ 3GHz and i got 55fps with 4 threads (mpeg2 source, 1500kbps 1 pass xvid)
The fps is pretty much the same as the regular xvid and the cpu usage is around 50%
With the same settings im able to get ~111fps using divx 6 (also i set crop,resize and deinterlace directly from the divx panel almost without fps drop!)

But I'll wager that the final xvid file is significantly smaller than the final divx file, though, right? Have you checked? I'm curious.

I've been doing a lot of Divx vs. Xvid(MT) benchmarks over the last week, and as hard as I try to match up the profiles (so that I'm comparing apples to apples), Divx 6.7 always beats the snot out of Celtic Druid's Xvid MT build (found here (http://ffdshow.faireal.net/mirror/XviD/)) in terms of encoding speed. (Xvid's 'Threads' = 2, not 4, since I only have a dual core.)

On the other hand, the Xvid encode always produces a significantly smaller file (edit: without any loss to picture quality as far as my eyes can tell).

Well, logic dictates that this is probably why DivX performs so much faster: it's spending a lot less time compressing.

But, alas, I'm admittedly a n00b when it comes to the low-level profile settings, so maybe I'm not setting up correctly. But even when just comparing the built-in "Home Theater" to "Home Theater" profiles, DivX is faster, but Xvid produces smaller files. <shrug>

[foxyshadis]

I would certainly hope that xvid wouldn't be significantly smaller than divx if both were set to 1500kbps.

But yes, xvid's threading capability isn't as refined as divx's.

However, sawo, you need to remember not to let the avisynth be a limiting factor - use setmtmode(2) at the beginning of the script (with the latest MT avisynth), and use leakkerneldeint as your deinterlacer (fastest decent-quality avisynth deint). That might boost your xvid fps. If you are feeding it a preprocessed source vid this doesn't apply.

[|sawo|]

Zach actually the difference between divx/xvid was 100-200kb with 80mb mpeg2 source so i think the size in this case is not big deal considering the fps difference.
foxyshadis im currently using the latest virtualdub with the vfw versions of the codecs(without avisynth), because it saves time for me.When i try the MT avisynth version ill post the results

[Zach]

I would certainly hope that xvid wouldn't be significantly smaller than divx if both were set to 1500kbps.

Yes, I should clarify that all my benchmarking was with using a default "Target quantizer: 4.00" rather than a "Target bitrate (kbps)" which, I guess you are saying, is basically VBR vs. CBR, right?

I don't like the concept of hard-coding a bitrate. It just seems wasteful. :p

Alright, well, sorry to intrude. Carry on.

[JCDenton]

Hi everyone,

I'm sorry to bother you with this, supposedly stupid, question but I really don't know what to do or where else to look for advice.

First off, my system:

A64 X2
WindowsXP SP2
VDub Mod 1.5.10.1

I can't get the XviD codec (tried the "offizial" one and Koepi's 28062007 build linked at doom9.net) to use both of my cores - the funny thing is, just a couple of days ago it worked perfectly fine, CPU-load was near 100% almost every time, even when I switched the priority to "idle".
I also tried pixelk's 20071031 build, but here I only get up to 70% CPU load with 4 threads and "higher" priority.

I don't remember having changed anything, even reinstalled the codecs an VDub over and over again - does anyone have an idea what might be wrong here ?

Oh, and please pardon my clumsy English, as you've obviously figured out by now, it's not my mother tounge ;)

Regards

denton

[foxyshadis]

Official and 28062007 are both part of the 1.1 branch, which explains why those won't thread. Don't use 4 threads if you only have 2 cores, though; 2 is best and 3 is the max, may help or may hurt in different situations, but 4 will always hurt by uselessly increasing the cpu meter without improving performance.

The other half of the problem is that the input is probably slowing the whole process down now: You have to use MT avisynth if you have an even remotely complex script to get full utilization.

[JCDenton]

Stupid me, could have figured for myself that ">2 Threads@2 Cores = bad" :p
Well, thanks for your reply - sadly, I've never used avisynth and don't have any experience with scripting, so I'm just going to stick with pixelk's build, at least that way I can use the remaining CPU cycles to transcode the sound stream ...

Thanks again.

Regards

denton

[humax]

i have a Q6600 Quadcore . Do I have to use 4 Threads ??
I looked at pixel k homepage but it is in french language . There was a 4 in Number of Threads . In other threads i read that a maximum of 3 Threads has to be chosen .

Thx for Help

[Adub]

You don't "have" to use 4 threads. But it should help. Actually I think it is 6 threads for you, but don't quote me on that.

[squid_80]

Actually testing on my Q6600 shows 2 threads to be optimal. 3 or 4 may improve performance very slightly, but only if you're not doing anything with the pc; otherwise they're worse than 2.

[sysKin]

You don't "have" to use 4 threads. But it should help. Actually I think it is 6 threads for you, but don't quote me on that.

Any basis for that? I'd say 3 threads. Definitely no more threads than cores, what made you think that!

[foxyshadis]

Any basis for that? I'd say 3 threads. Definitely no more threads than cores, what made you think that!

Confusion with x264's behavior. ^.~