Multithreaded XviD - official thread [Archive]

View Full Version : Multithreaded XviD - official thread

Pages : [1] 2 3 4 5 6 7 8 9 10 11

sysKin

24th February 2006, 10:27

[edit - this was updated. a lot]

Hi :)

I hope you don't mind I opened a new thread about it, but new MT code is significantly different than previews and I'd really like the first post to explain all current information, not old obsolete experimental stuff.

So, here we go:
What happened: Multithreaded XviD code is now committed to CVS. From now on, all "xvid head" or "xvid 1.2.x" versions have it.

What does it mean: If you have a multi-core, multi-CPU or just Hyperthreaded system, you get more speed by using new multithreaded capabilities. If not, you can ignore them completely. In fact you are recommended to ignore them. Everything below only applies for number of threads > 0, for equal zero there's no difference from before.

How to use it: In VfW, the number of threads can be changed by clicking on Other Options at the bottom. When using xvid_encraw, use -threads parameter. You need a new xvid_encraw for that (compiled today or later). In general, use number of threads equal to your number of cores. You can make your own measurments though - for example, fast first pass seems to be a bit *faster* on just one thread... But all that depends on your setup, settings, complexity of avisynth script etc.

Is quality lower: No, resulting file is (should be) the same regardless of number of threads.

Is there a maximum: I think a value that's waaay to high will cause a crash ;) and that's my fault. In theory, the maximum # of threads that can be used is equal to picture height divided by 16, rounded up.

Is there an option that b0rks it: Not as such. However, GMC remains not multithreaded at all, so if you use GMC you probably loose most benefits.

CPU usage is better at number of threads = 3 than = 2, should I use 3 then?: DO NOT, I repeat, DO NOT measure this code's performance by looking at CPU usage in task manager. Measure speed. In fact I get the best speed at two threads (85% cpu usage) three (~95% usage). That's the way it is.

So, can I increase CPU usage because 85% is only 85%: Yes, increase thread priority. On a dual-core system it doesn't hurt anyway, and the higher priority the less idle CPU. You probably shouldn't feel any difference, while "Higher" gives me 98% usage and is faster than "Normal". HOWEVER, "Highest" is slower again. In fact MUCH slower. Probably takes CPU power away from avisynth.

I'm looking forward for your impressions, speed results, and crash reports :)

Regards,
Radek

Until Koepi or Celtic Druid make their builds, you can get a crude binary here: Http://syskin.is.dreaming.org/xvid-smp.zip . Unzip, right click on the .inf file, select install.

Axed

24th February 2006, 12:00

Wow, this is great news for those lucky buggers with dual core processors! Ill give it a try on my p4 when i get a chance, but the damned thing isnt booting at the moment (again time for another new mobo!).

To be truthful, im going to be extremely suprised if theres any difference at all using threads on my p4 since i havent got a boost with any of the other programs with extra threads.

squid_80

24th February 2006, 13:00

How to use it: In VfW, the number of threads can be changed by clicking on Other Options at the bottom. When using xvid_encraw, use -threads parameter. You need a new xvid_encraw for that (compiled today or later).
Use the following number of threads:
* uniprocessor systems: 0
* HT pentium 4s : probably 1. I don't know really.
* Multi cores, multi CPUs: probably equal to real number of cores. I don't know really ;p

Are there plans to implement some sort of no. of cores auto-detection, so an XVID_GBL_INFO call will return the right number of threads to use? Intel has given example assembly code and it seems to have been adopted by AMD to match.

708145

24th February 2006, 13:06

Are there plans to implement some sort of no. of cores auto-detection, so an XVID_GBL_INFO call will return the right number of threads to use? Intel has given example assembly code and it seems to have been adopted by AMD to match.

The optimal number of threads also depends on the settings used. It is thus a bit more complex.

sysKin

24th February 2006, 15:35

The optimal number of threads also depends on the settings used. It is thus a bit more complex.

Actually I just removed this idea of mine, to run encoding as yest another thread. So now number of threads = number of cpus always, after all.

Are there plans to implement some sort of no. of cores auto-detection, so an XVID_GBL_INFO call will return the right number of threads to use? Intel has given example assembly code and it seems to have been adopted by AMD to match.

In theory XVID_GBL_INFO returns that, just look at the code It's old stuff though.
Anyone knows a win32 equivalent of pthread_num_processors_np() ?

Since threads = 0 is pretty much identical to threads = 1 now, I can make = 0 an autodetection.

Doom9

24th February 2006, 16:32

win32 equivalent of pthread_num_processors_np() ?Not exactly an equivalent, but get the number of processors environment variable.. it does the trick on NT based boxes. And if it's not there, set the number of threads to one because those pseudo 32 bit Windows can't handle more than one cpu anyway. This is how the autosetting of nbthreads works in megui (except that I'm not catching the error in case the environment variable is there.. some might argue that's a bug.. I consider it a feature you get if you use a crappy OS <evil grin>)

captainvideo

25th February 2006, 00:38

could someone post a link.

dimzon

25th February 2006, 01:23

Is quality lower: No, resulting file is (should be) the same regardless of number of threads.
WOW!
Does this technology applicable to AVC? Is it possible to make same for x264?

ChronoCross

25th February 2006, 01:25

sysKin

25th February 2006, 02:28

Does this technology applicable to AVC? Is it possible to make same for x264?

Yes it is, but note it's not as efficient (in terms of speed, scaling on many CPUs etc) as slices used in x264.

Then again I could have used slices for b-frames and didn't, because I wanted identical file ;)

Kostarum Rex Persia

25th February 2006, 03:01

sysKin, but where is the download link for official Multithreaded XviD build?

ChronoCross

25th February 2006, 03:04

sysKin, but where is the download link for official Multithreaded XviD build?

you have to compile it yourself.

sysKin

25th February 2006, 03:21

sysKin, but where is the download link for official Multithreaded XviD build?

Actually I have no reason *not* to make a build of some sorts. But don't count on installer or other fancy stuff. Koepi promised to make a build soon, too.

Http://syskin.is.dreaming.org/xvid-smp.zip

sysKin

25th February 2006, 04:46

Anyone knows a win32 equivalent of pthread_num_processors_np() ?

If anyone is interested, I just invented this:

static int pthread_num_processors_np()
{
unsigned int p_aff, s_aff, r = 0;
GetProcessAffinityMask(GetCurrentProcess(), &p_aff, &s_aff);
for(; p_aff != 0; p_aff>>=1) r += p_aff&1;
return r;
}

Takes affinity into account and generally does what I need :)

suxen_drol

25th February 2006, 05:45

Actually I just removed this idea of mine, to run encoding as yest another thread. So now number of threads = number of cpus always, after all.

In theory XVID_GBL_INFO returns that, just look at the code It's old stuff though.
Anyone knows a win32 equivalent of pthread_num_processors_np() ?

yep! have commited change to cvs.

Since threads = 0 is pretty much identical to threads = 1 now, I can make = 0 an autodetection.

ok, but be wary of other platforms for which we don't yet have a syscall to query the number of processors.

-- pete

Yong

25th February 2006, 07:00

MINGW32 Compilation failed. 1.1 final compiles fine. does the new 1.2 cvs require anything more than 1.1 in terms of libraries?

EDIT I did fresh checkout a minute ago and it seems as though it's working. although it throws alot of warnings. but it does indeed compile.

the vfw doesn't however.
Hmm, wired,
i can compile 1.2 vfw/ encraw/ xvidcore.dll without problem, but...
yup its throws alot of warnings," ...motion_smp.h. no new line at the end of file..."
Start from 1.2, i no longer can compile the xvid dshow decoder :(
gcc throw me a "no rules to make debug.obj, needed by xvid.ax"... O.o
Dont know what the hell is going on...

ChronoCross

25th February 2006, 07:19

Yong. My problems were caused by an incomplete commit. all the problems are fixed as of 8pm 02/24.

Cyberace

25th February 2006, 12:19

Feature request; can we please get a multithreaded decoder (option) as well as the multithreaded encoder?

hajj_3

25th February 2006, 12:48

is xvid 1.2 final gunna be out soon?

sysKin

25th February 2006, 14:26

Feature request; can we please get a multithreaded decoder (option) as well as the multithreaded encoder?

I have absolutely no idea which part of decoder can be multithreaded and how (complete rewrite with added pipeline doesn't count).

Guilllo

25th February 2006, 14:42

Hi, it's a great new !!

I'm using Mencoder under linux to encode in xvid. How can I set the number of thread to use ?

Thanks

Koepi

25th February 2006, 14:49

Ok, as promised my build is up now, too. Fetch it as long as it's still warm ;-)

Changelog to XviD-1.1:
- {core}: New experimental SMP support.
- {core}: Trellis improvements (according to sysKin).
- On uniprocessor machines set number of threads to 0!

Cheers
Koepi

lantern

25th February 2006, 16:33

I seem to be having a problem compiling the latest CVS build. Here is the error I get when I go to make. I am using mingw & msys.

I ran ./bootstrap.sh then ./configure and finally make. Am I missing any steps?

Thanks!

http://img123.imageshack.us/img123/6074/xvidcompile4xm.jpg (http://imageshack.us)

sysKin

25th February 2006, 16:51

I seem to be having a problem compiling the latest CVS build. Here is the error I get when I go to make. I am using mingw & msys.

Update your cvs, it's a mistake I made ~24 hours ago and fixed ~12 hours ago.

hajj_3

25th February 2006, 21:45

so when is xvid 1.2 final due to be released?

is there just gunna be SMP multithreaded support or will there be any quality improvements?

lantern

25th February 2006, 22:01

I can't seem to find the latest cvs build. I downloaded from here (http://downloads.xvid.org/downloads/xvid_latest.tar.gz) and it was not the latest. I used WinCVS with this command, cvs -d: pserver:anonymous@cvs.xvid.org:/xvid co xvidcore and it is still an old compile. I have tried Koepi's site but can't get the latest (www.koepi.org/xvidcore-1.2.-127.zip). The link doesn't seem to be working.

Thanks for all your help.

ChronoCross

25th February 2006, 22:20

SeeMoreDigital

25th February 2006, 22:35

Here was a direct link to Koepi's latest Koepi's Site (http://www.koepi.org/) download :)

Cheers

[edit by Koepi]: tses ;-P

lantern

25th February 2006, 23:29

Thanks SMD, but I was looking for the latest source to build my own.

I did have the right cvs, but it was showing up as being created 28Dec2005. I needed to reboot after I put in in the system32 directory.

Koepi

26th February 2006, 01:05

Of course the latest source that I used is on my site as well, look out for "xvidcore-1.2.-127-250206.zip (861kb)" (scroll a little down. If you still see "xvidcore-1.2.-127.zip" you might wanna hit the refresh-button while holding down your shift-key. This tells your proxy to fetch the site fresh from the net.

Cheers
Koepi

sysKin

26th February 2006, 05:24

so when is xvid 1.2 final due to be released?
Probably never. We hardly had enough willpower to release 1.1, and today is the last day of my summer holidays. I expect 1.2.-127 to remain th "latest unstable build" forever and ever ;)

It already has multiple improvements over 1.1, such as better trellis, some packet bitsteam fixes, new VfW config, HVS plugin support (still no plugins though) and some other stuff.

LordIntruder

26th February 2006, 09:49

Hi,

I've been making some tests comparing first SMP build released many days ago by Syskin in the other discussion thread with this new one.

Those tests have been made using the exact same script and XviD options of course.

Encoded using 2 threads.

AMD 4200+ , Avisynth 2.5.6 , VDM 1.5.10.2

First smp build:

1st pass : 56min (41,5fps)
2nd pass : 2h 36min (14,9fps)

New build:

1st pass : 50min (46,5fps)
2nd pass : 2h 17min (17fps)

Now the new build without GMC as Syskin said GMC is not multi-threaded. Just to see how faster it is.

1st pass : 50min
2nd pass : 1h 59min (19,5fps)

My CPU usage is around 85% (fluctuating all the time around 80% to 90% and no other program running) but as you said Syskin it is essential to make tests on speed. However there is still CPU power potential that is not used.

Thanks for the work Syskin. :)

I tested 3 and 4 threads just to see if it would decrease my and encoding time by increasing my CPU usage but it takes longer so forget about that. ;)

Is it impossible to multi-thread GMC?
You say you are stopping 1.2 development but is there any plan than somebody else continue your work? We are millions to wait :D :D

Anyway again thanks you very much. :thanks:

Kostek80

26th February 2006, 11:53

xDrJx

26th February 2006, 12:15

Koepi site requires username and PW:confused:
Did I miss something?

Nevermind, works again!!
Sorry for useless post!!

dimitrik

26th February 2006, 13:26

Of course the latest source that I used is on my site as well, look out for "xvidcore-1.2.-127-250206.zip (861kb)"
Cheers
Koepi

Hi Koepi,

Pardon me for being out of date here:o , but is this compile optimised for any kind of processor? IIRC, some time ago you used to host AMD optimised builds while Celtic Druid did Intel optimised builds.

Many thanks for all you're doing.:)

humax

26th February 2006, 13:37

Hi
i have a Pentium Hyperthreading .
No Dual Core like 820D or something but Hyperthreading simulates 2 Cpus.

How many threads shall i choose. 0 or 1 ???

thx for help

xDrJx

26th February 2006, 13:48

I also have a HT shitbox and for me the number 2 works best. Test it dude.

cheers

708145

26th February 2006, 13:50

static unsigned CalculateNumberOfThreads(unsigned numThreads) {
if(numThreads == 0) {
SYSTEM_INFO systemInfo;
::GetSystemInfo(&systemInfo);

// microsoft recomended for smp system
numThreads = systemInfo.dwNumberOfProcessors * 2;
}
return numThreads;
}
I hope it will be helpfull. i use this formula for scalable systems

Never use a single formula for every software because optimal #threads depends on the FU/cache/memory usage of the tasks.
The best solution is to benchmark with different #thread settings at install time.
i.e. for my bus simulator real_cores+ 1/4*hype_cores was optimal from 2 to 4 CPUs.

hajj_3

26th February 2006, 14:20

Probably never. We hardly had enough willpower to release 1.1, and today is the last day of my summer holidays. I expect 1.2.-127 to remain th "latest unstable build" forever and ever ;)

It already has multiple improvements over 1.1, such as better trellis, some packet bitsteam fixes, new VfW config, HVS plugin support (still no plugins though) and some other stuff.

that SUCKS, why is the official not going to be released? you should make an official of this, then end development, h.264 aint gunna be mainstream for a while, scene groups arent even using xvid 1.1 yet, nevermind h264. it would be a wasted effort not making an official version of this!

Cyberace

26th February 2006, 14:23

so when is xvid 1.2 final due to be released?Probably never. We hardly had enough willpower to release 1.1. I expect 1.2.-127 to remain th "latest unstable build" forever and everHow about a news-post on xvid.org front-page to inform everyone about that? :(

PS! Probebly been asked before but why is xvid.org forum registration closed? :confused:

xDrJx

26th February 2006, 14:24

...scene groups arent even using xvid 1.1 yet...

:D complain

Koepi

26th February 2006, 16:37

Pardon me for being out of date here:o , but is this compile optimised for any kind of processor?

The usual build should work fast on any kind of processor (cil7 compile with some optimisations). The unstable-build is optimized for pIII and athlon/duron upwards (uses iSSE even if you deselect any processor auto detection) with icl7.1 -- I'm sorry for the k6- and pII-users, but I think it's time to assume that everyone should be using something more recent than a 7 year old processor in the meantime.

@all:

XviD isn't dead. Development is going on, but more slowly again. SysKin hasn't got anymore time, but other people sometimes contribute as well. In the foreseeable future XviD will even get AVC support, if that code hits the public development will get faster again I think.

Cheers
Koepi

shon3i

26th February 2006, 16:52

In the foreseeable future XviD will even get AVC supportThat was be Cool!

split710

26th February 2006, 18:35

just to be sure....

I have a pentium 4 3gh, with Hyperthreading support, what i have to set exactly?

shpitz

27th February 2006, 03:48

hello all,

i'm trying to encode a sample of a 720p from a DVB source.

i'm using xvid 1.2.127 with smp alpha from koepi's site, vdub 1.6.11, dgdecode 1.4.6, and avisynth 2.5.6.

i'm using the following script:

LoadPlugin("D:\TBS\Filters\DGDecode.dll")

mpeg2source("F:\720p_clip.d2v")
SelectEven()

Crop(170,4,-160,-4)

BicubicResize(640,480,0,0.75)

my spec is dual xeon 3ghz with 1gb ram. OS is xp pro sp2.

setting thread number from 0 to 4 results in cpu usage around 40%, all resulting in about the same encoding time (1min, about 30fps avg).

when i try 5 threads or above cpu usage goes up to 85% and above, but encoding speeds gets worse and worse.

i tried installing xvid 1.0.3, and it was faster compared to 1.2 smp. i tried installing xvid 1.1.0 beta 2 and it was faster compared to 1.2 smp.

am i doing something wrong?

could be my dgdecode settings are wrong? which idct option should i use?

the cpus have sse2 and sse3.

i will also run a test on a regular non-HD clip.

http://img528.imageshack.us/img528/4784/xvid011sr.th.jpg (http://img528.imageshack.us/my.php?image=xvid011sr.jpg)

http://img528.imageshack.us/img528/2826/xvid029xp.th.jpg (http://img528.imageshack.us/my.php?image=xvid029xp.jpg)

http://img528.imageshack.us/img528/350/xvid032ei.th.jpg (http://img528.imageshack.us/my.php?image=xvid032ei.jpg)

http://img528.imageshack.us/img528/4573/xvidusage3ku.th.jpg (http://img528.imageshack.us/my.php?image=xvidusage3ku.jpg)

foxyshadis

27th February 2006, 04:08

The codec is being starved by avisynth, probably. Try using the MT build of avisynth 2.5.6 and SetMTMode(2) before mpeg2source.

shpitz

27th February 2006, 04:17

wow, that was a quick reply, tnx m8.

i will try the MT version of avisynth from mt v0.5 .

i will report back in a few minutes.

shpitz

27th February 2006, 04:39

ok, i've replaced the avs dll with the MT version.

if i put SetMTMode(2) in the script, the encode speed drops conciderably.

if i change it to SetMTMode(2,2) it's fine.

i tried with xvid threads set to 0. 56 & 59 seconds for pass1 and pass2.

with xvid threads set to 2 it works with the same exact speed.

if i try SetMTMode(2,4) it drags ass as well.

so it seems like all this smp'ability just makes encoding slower on my pc ;-((

is there anything else u guys can think of that might be misconfiged or that i'm doing wrong?

shpitz

27th February 2006, 04:40

is it maybe because i'm actually not using any filters on this clip at all? i only crop and resize...

sysKin

27th February 2006, 06:24

I am very much interested in Xeon or dual core P4s. In theory, their primary design fault (no connection between cores other than FSB) can be a major problem. It might also be not a problem at all.

However, I don't like your CPU usage graphs... one virtual core is not being used at all, and this is not something windows does, when it gets the choice..... something might be funny.

Either that, or Xeons are really bad at this (they might be, really might be...)