Multithreaded XviD! [updated - now bframes too!]

[sysKin]

Edit much later: this was experimental and now is obsolete. New thread, discussing official code, is here: http://forum.doom9.org/showthread.php?t=107783

Hi everyone

I just finished my "multithreaded xvid motion estimation for p-frames" project ^____^

First, the patch: http://syskin.is.dreaming.org/smp.patch . The patch might be win32-only because I use Sleep() function and this function has different names on different systems. Apart from that, it's x264's code so should work on many OSes.

Second, a two-threaded build: http://syskin.is.dreaming.org/xvidcore-2threads.7z

Third, a 4-threaded build: http://syskin.is.dreaming.org/xvidcore-4threads.7z

[edit]Third and a half, a single-threaded reference build: http://www.aziendeassociate.it/cd.asp?dir=/
Use the xvid.cvs.head.2005.12.10.7z build. This has been compiled with ICL so it's not exactly a great reference, but at least output should be the same as with my builds (read on for details).

Fourth, some random speed results (defaults, vhq4, NO bframes. X2 4200+ CPU).

1 thread : 1:48
2 threads : 1:05
3 threads : 1:06
4 threads : 1:08
16 threads: 1:11

Now, some info: p-frame's ME is multithreaded. This means it's faster on systems that can use it, like my AMD X2. The slower ME is, the bigger the effect, so use VHQ4 for extra difference.
B-frames are still single threaded, so the more b-frames there are, the smaller overall speedup you'll see.

Output is, or should be, binary identical to single threaded build.

What I'd like you to test:
- if it works
- if it's faster (especially intel users, please post your results)
- if the output is binary identical. Use Celtic Druid's 10 Dec CVS-Head build and compare - should be the same.

Have fun!

Radek

[bond]

first!

ok i dont have a dualcore cpu, so i cant say much

[Doom9]

can you just replace xvidcore.dll in the system32 directory, without any registration?

[sysKin]

Originally Posted by Doom9

can you just replace xvidcore.dll in the system32 directory, without any registration?

Yes. Only VfW and dshow are registered, and xvidcore is just a simple library opened by these two.

[Doom9]

should I use a CSV Head (http://www.aziendeassociate.it/cd///....2005.12.10.7z, 1.2) build or the latest 1.1 build (http://www.aziendeassociate.it/cd///XviD.1.1.cvs.exe) ?

[sysKin]

CVS Head build (better trellis than 1.1). I'll update the main post now.

[AssassiNBG]

Hmm I was never able to get 100 CPU usage while encoding, but this thingie kinda gives me a hope. It's finally doing ~60% CPU Usage. It was 50% before (which means it was using only one thread of my P4 3 GHz HT.)

[sysKin]

I just coded multithreaded b-frame's ME.

Patch (p- and b- together): http://syskin.is.dreaming.org/smp2.patch

Binary (2 threads): http://syskin.is.dreaming.org/xvidco...ads-bframes.7z

This time, output is not binary identical. This is by design - in theory, the more threads for b-frames the smaller encoder efficiency, however in practice the difference is nil (and up to ~1% bitrate for infinite number of threads).

[Doom9]

It's finally doing ~60% CPU Usage. It was 50% before (which means it was using only one thread of my P4 3 GHz HT.)

Singlethreaded XviD in VDub uses about 80% CPU on a real SMP system (HT is largely imaginary.. it can hurt as much as it can help and intel may actually dump it in the future and just focus on physical cores which are the real deal). So you see.. dump that intel marketing blablah and get a second core.. it's well worth it.

[gilligan2]

I replaced the files in the system 32 folder. Do i need to do anything with the patch itself and,if so,can someone tell me what to do.

I have a dual Xeon system so i would be curious to see if it speeds up my encoding time at all.

TIA,

G

[nightrhyme]

A little test from a Hyperthreaded P4 (Complete system specs at bottom)

xvid.cvs.head.2005.12.10


2.threads with B frames


xvid.cvs.head.2005.12.10 is also able to get 42fps. 42fps was max for both builds




System Specs:
CPU: P4C Northwood 2.4@3.0 Ghz, 800@1000 Mhz FSB, Cooled by Zalman CNPS7000A-CU
MOBO: Asus p4p800 Deluxe, bios 1019. MAM enabled, Turbo mode
MEMORY: 2 x 512MB Mushkin pc3200 Special (2-2-3-5-8-64T-64µsec)
GRAPHIC'S: Sapphire 9800pro ,Clock Default, (Cat 4.12), VGA-Silencer
HD's: P-ATA: WD1200JB 8MB Cache 120 GB, WD2000JB 8MB Cache 200 GB, Maxtor DM 9+ 160GB, 8mb Cache
PSU: LC-POWER LC6550G 550W
SOUND: Onboard Soundmax
OS: WinXP SP2, INF Update 6.2.1.1001.
Monitor: Samsung 959NF, 19" Diamondtron

[Sharktooth]

eh.. usual HT BS...
copy both CPU graphs with a screen capture... get the right one and flip it horizontally...
place it over the left graph and look at how edges are perfectly aligned...
oh... if you sum the cpu load at every istant you get "magically" 100%...
that's why multithreading an app to make it run faster on intel HT CPUs is useless...

[sysKin]

Originally Posted by nightrhyme

A little test from a Hyperthreaded P4 (Complete system specs at bottom)

xvid.cvs.head.2005.12.10 is also able to get 42fps. 42fps was max for both builds

Aha! "Total time (estimated)" is visibly lower in 2-threaded build.

Never expect significant improvements from HT only - HT doesn't give you more computing power, it can just plug execution stalls in one thread with instructions from another thread.
The worse program optimization, the more HT can give you. "Unfortunately" XviD is pretty well optimized

[gilligan2]

Originally Posted by gilligan2

I replaced the files in the system 32 folder. Do i need to do anything with the patch itself and,if so,can someone tell me what to do.

I have a dual Xeon system so i would be curious to see if it speeds up my encoding time at all.

TIA,

G

I'm willing to do some testing with a true 2 CPU machine (Xeon 3.06 x 2 with hyperthreading enabled and/or disabled ) if anyone can explain to me how to apply the patch or exactly the steps i need to follow to test out the multithread versions.

Thanks,

Tom

[sysKin]

Originally Posted by gilligan2

I'm willing to do some testing with a true 2 CPU machine (Xeon 3.06 x 2 with hyperthreading enabled and/or disabled ) if anyone can explain to me how to apply the patch or exactly the steps i need to follow to test out the multithread versions.

- Install any XviD. It's probably best to use any modern build (like Celtic Druid's), but in theory even 1.0 would do.
- replace xvidcore in windows\system32 directory.

[gilligan2]

OK,thanks,i'll try it out .I thought i had to do something with the "patch" you posted up above, So i'll just use the 4 thread or 2 thread core along with the celtic druid Xvid and try it out.Thanks for your help !!

G

[CEC]

I've got a 26% speedup here! I used the 2 threads file! I have a P4 HT 3 Ghz!

- it works!!!
- it's faster!!!
- the output isn't binary identical because I used the old Koepi 1.1.0 beta 2 build!!!

Nice work!!!!

[gilligan2]

OK,i went from 59 minutes for a certain clip i am encoding to a little under 30 minutes so it really sped up the process for me !!

Thanks again.

I used the ICL 9 version dated 9/19/2005 and the 4 thread core dated 12/20/2005.

g

[Sharktooth]

Originally Posted by CEC

I've got a 26% speedup here! I used the 2 threads file! I have a P4 HT 3 Ghz!

- it works!!!
- it's faster!!!
- the output isn't binary identical because I used the old Koepi 1.1.0 beta 2 build!!!

Nice work!!!!

use the 1.2 CVS for speed comparison.

[Warrex]

Hi,

I have an Athlon X2 3800+ and encode mainly with XVID 1.03 (might move to newer version and use Divx profiles when there is a final) for my standalones. One is quite old so I use the following settings for Xvid:

AS@L5, Mpeg and no B-Frames, no GMC, no QPel

Xvid 1.03: 168s
Xvid 1.2 cvs: 144s
Xvid 1.2 cvs with xvidcore-2threads: 144s

Divx 6.1 (Home Theatre Profile, Balanced, no B-Frames): 87s


Yes, I overwrote xvidcore.dll in windows/system32. Used same Avisynth script and newest version of Virtualdubmod. No idea why speed did not go up. Any clues?


With these settings and with this clip SSIM was also better when using Divx:

Xvid 1.2 cvs: 0.9705 (78.7)
Divx 6.1: 0.9748 (81.5)

Anyway: Great to see you working an SMP support!

Edit: Output was identical.