Multithreaded XviD! [updated - now bframes too!]

[KornX]

yeah like i said,
by disabeling "display decompressed output" it was gone...

KornX

[guada 2]

@reygen

" My X2 4600+ doesn't have any problems with Vdub like his X2 3800+ is having. My CPU is not overclocked yet I run with a huge heatsink (Thermaltake Sonic Tower) and fan that keeps my CPU cool at 40 degrees Celsius at full load on both cores."

Wow!!!
You dont have a motherhead, but a car.... :D

[Revgen]

@reygen

" My X2 4600+ doesn't have any problems with Vdub like his X2 3800+ is having. My CPU is not overclocked yet I run with a huge heatsink (Thermaltake Sonic Tower) and fan that keeps my CPU cool at 40 degrees Celsius at full load on both cores."

Wow!!!
You dont have a motherhead, but a car.... :D

Yep.;)

This CPU cost me mucho money. I was not going to give it a chance to fry. Especially since I was going to be doing full load encodes.

[Zep]

Originally Posted by guada 2
@reygen

" My X2 4600+ doesn't have any problems with Vdub like his X2 3800+ is having. My CPU is not overclocked yet I run with a huge heatsink (Thermaltake Sonic Tower) and fan that keeps my CPU cool at 40 degrees Celsius at full load on both cores."

Wow!!!
You dont have a motherhead, but a car....


Yep.;)

This CPU cost me mucho money. I was not going to give it a chance to fry. Especially since I was going to be doing full load encodes.

haha same here. I have the X2 4400+ OC to 2.6Ghz with PC4000
running at 263Mhz 2.5 3 3 4 and it rips through encodes :)

Anyway i bought this heatsink to keep from frying eggs

http://www.newegg.com/Product/ShowImage.asp?image=35-109-119-07.jpg,35-109-119-06.jpg,35-109-119-02.jpg,35-109-119-08.jpg,35-109-119-03.jpg,35-109-119-09.jpg&CurImage=35-109-119-07.jpg&Description=THERMALRIGHT%20XP-90%20Multiple%20Heatpipes%20Cpu%20Heatsink%20-%20Retail

which uses a 90mm 3000RPM fan and does well (but not as well as that
monster you have LOL ) I sit around 50c on full load. (65c is the max rate temp for the CPU)

BTW - i found going from PC3200 to PC4000 was a HUGE speed help.
avs scripts are so memory bound it isn't funny. (well i do all HDTV
encodes so not sure how bad it is for lower rez stuff not needing
to move as much data around in memory)

now with avisynth2.6MTa and XvidSMP my speeds are just wow
and I see syskin is working on a even better SMP build!

Life is good indeed! :D

[DVD_GR]

Saw no speed improvement on my dual core Intel running 64bit XP when replacing xvidcore.dll using Celtic Druid's 1.1 Final (approaching 40 fps ~60% CPU load).

Tried to install Koepi's XviD-1.2.-127-07012006.exe, but it gives an error on xvid.ax (see attached) & it won't complete the install. I believe that the problem is because of 64bit Windows sytem paths - xvid.ax registers in syswow64


I need to mention for all Intel cpu users at the hardware part of coding that intel dual core architecture is like a greek tragedy,so dont be ready to see some real improvements from the 1st generation of intels dual-cores,their implementation is really bad.And of course as mentioned before hyperthreading is just a <cheat>...2-4% of average performance gain from 50 to 100% cpu usage change as measured on many hardware sites..dont even bother to test...

[hajj_3]

got my opteron 146 clocked @ 2.85ghz from 2.0, 43c under full load using xp-90 heatsink :). ram is @ 285 (1:1) @ 2.5-4-4-7. stable under 25 hours of prime95 torture test :)

havent tried encoding anything yet, will see how fast i can encode a movie with xvid 1.1. £105 oem for an opteron 146 at scan.co.uk, you cant beat it!

[chilledoutuk]

i have run this on my bros dual athlon xp setup but only get 75% utilisation is this normal and is there any more room for further optomising the code for multithread encoding?

[Doom9]

i have run this on my bros dual athlon xp setup but only get 75% utilisation is this normal and is there any more room for further optomising the code for multithread encoding?Do a search on my nick and keywords like thread, interdependence or smp....

here's the link: http://forum.doom9.org/showthread.php?t=104535

[Zep]

got my opteron 146 clocked @ 2.85ghz from 2.0, 43c under full load using xp-90 heatsink :). ram is @ 285 (1:1) @ 2.5-4-4-7. stable under 25 hours of prime95 torture test :)

havent tried encoding anything yet, will see how fast i can encode a movie with xvid 1.1. £105 oem for an opteron 146 at scan.co.uk, you cant beat it!

single core right? that is why your temp is so good.
2 cores means more heat. If i run just 1 core (turn one off)
my temps are much lower and of course there is double the L2 cache
(in my case 2 megs worth, 1 meg per core) It is a great heatsink
but for dual core that are way OC i would get something even
better if i had to do it again.

why are you running the ram 1 to 1? (Just curious as i run 2 to 1
and my effective rate is 526Mhz with a 6.8 gigs a second using
sandra bench score though the true read is 6 gigs
and the true write just over 2 gigs a second and latency is 48ns)

LOL i ran *2* prime95's torture tests at once. one for each core for
3 days without any errors a month ago at my current settings. I figured
after 3 days that should be proof enough my OC settings are good.
(note i had to up vlink voltage and ram voltage too)

So far super rock solid and i have encoded many many things since :)

[hajj_3]

yeah its single core, i run 1:1 285 which is 570mhz effictively, stock is 200. ram does 310 (620), but would have to use divider and set cpu to 2.8ghz with 9/10 divider. dual core aint worth the £ at the moment, when you can get an opteron 146 for £105 that does 2.85ghz you just cant beat it for value. if i had the £ i would get a 4400+

[swaaye]

Koepi's site is asking for a password... :confused:

[SCIF]

What version of xvid core i should use for 2 threads? Celtic Druid's xvidcore-dualthread.7z(26.12.05 ) (http://www.aziendeassociate.it/cd///xvidcore-dualthread.7z) or sysKin's dualthread(from first post) (http://syskin.is.dreaming.org/xvidcore-2threads.7z) or anything else?

[KornX]

take koepis

www.koepi.org

the PW shouldn't pop up...
Am i right, Koepi?


KornX

[SCIF]

You are not authorized to view this page.
Is it only for me???

[Revgen]

I have Koepi's multithreaded build. Is there a place that I could upload it to?

[celtic_druid]

Shouldn't make all that much difference. They are all using the same codec, just difference compilers. Mine was ICL9, Koepi's is probably ICL7 and Syskin's gcc?

[SCIF]

Mine was ICL9, Koepi's is probably ICL7 and Syskin's gcc?
Are sources identical? From your page download
xvidcore-dualthread.7z? What your latest build?

[celtic_druid]

Think that the cvs may have been updated between Syskin and my builds. It wasn't between mine and Koepi's though, so the source should be the same, unless he added other patches.

cvs was updated recently, but that was just xvidencraw.
- Removed the 9999 frames encode limit from xvid_encraw

[Koepi]

celtic:

can you make sure to use BS_VERSION 42 for the multithreaded test build? 43 is for 1.2 vanilla cvs. (I didn't check which BS_VERSION you use, I just wanted to make sure.)

@all:

I already wrote my webhoster, let's hope the site gets back to normal soon.

Cheers
Koepi

[celtic_druid]

Can't remember what it was using.

Ok, there was one thing in the CVS that did change. My build is using 40.

I'll put a new build up with 42.

[swaaye]

I just put together a Opteron 165 dual core for my encoding joy. Running at 2.5GHz with 2GB PC4000 heh heh heh. This patch absolutely hauls on it, sitting around 95% CPU or higher. Amazingly fast. First pass ran over 100fps with what I'm doing, and that is like 3x faster than the AthlonXP Barton @ 2.3GHz I was using a month ago for the same avisynth/xvid setup.

I used Xvid 1.1 as a base and then installed the SMP DLL. seems to be working great. Tried 1.2 but it didn't get along with StaxRip.

[stax76]

I used Xvid 1.1 as a base and then installed the SMP DLL. seems to be working great. Tried 1.2 but it didn't get along with StaxRip.

It's mandatory to 'Load Default' in the XviD config dialog to ensure you load 1.2 settings into StaxRip and overwrite projects and profiles with fresh 1.2 settings, did you try that?

[JustChecking]

OK,i went from 59 minutes for a certain clip i am encoding to a little under 30 minutes so it really sped up the process for me !!

Thanks again.

I used the ICL 9 version dated 9/19/2005 and the 4 thread core dated 12/20/2005.

g:) :thanks:
The url to the 4 threaded core is broken :(

Someone (maybe you gilligan2) that have it and can put it up somewhere? Me wanna test a dual, dualcore opteron. 2 threads did half job, now all 4 cpucores works at 50% (before just 2 cpucores at 50%) so i think the 4 threded build should do the trick.. or well.. i hope it will :) If someone have it.

[swaaye]

It's mandatory to 'Load Default' in the XviD config dialog to ensure you load 1.2 settings into StaxRip and overwrite projects and profiles with fresh 1.2 settings, did you try that?

Yeah I always hit load defaults when I first set up a default profile. When I started a job, VDubmod would just load up and quick quit.

XVid 1.1 plus the SMP dll is working great tho. Am I missing anything this way? I just installed XVid1.1 final and the xvidcore-dualthread.7z DLL

[swaaye]

Koepi I've tried your build but it just seems to crash when Staxrip starts the encode process. VDubmod pops up and promptly closes, it continues to 2nd pass, closes, encoding ends. Dunno what's up.

[HookedOnTV]

Should this work with GordianKnot? Using Koepi's build VirtuaDubMod flashes open then closes and GordianKnot says finished.

[shpitz]

hey guys,

i've tried the 'smp' versions and none of them gave any boost, they actually were a little slower the celtic's 1.1 or the 1.2 cvs head compilations.

when i tried koepi's 1.2 smp compilation vdub just crashes as soon as i start the encode.

what is that smp.dll swaaye was referring to?

i was running the following script:

mpeg2source("F:\snap.d2v",idct=7)
trim(49,5673)

# DEINTERLACING
LeakKernelDeint(order=1, sharp=true, forceCPU=5)

# Cropping
Crop(24,6,-16,-4)

bicubicResize(640,480,0,0.75)


a = last

b=a.RemoveGrain(mode=8)

SeeSaw(a,b, NRlimit=6, NRlimit2=7, Sstr=1.5, Slimit=5, Spower=5, SdampLo=6, Szp=16)


so i need to use any special version of avisynth? i'm using 2.5.6a

[HookedOnTV]

Finally able to seem some improvement. I had to do the encode from directly within VirtualDubMod. From there I could see cpu (x2 3800) utilitization jump to the 80% area. From MeGUI it tops out at 54%.

[celtic_druid]

I could do an mencoder compile linked against a multithreaded libxvidcore if anyone is interested?

[blubberbirne]

Is it sure, that the multithread encoder only works in first pass?

[HookedOnTV]

I could do an mencoder compile linked against a multithreaded libxvidcore if anyone is interested?

That would be great!

[captainvideo]

I am running an AMD 4200 x2, NForce4 motherboard, 2Gig ram, SATA HD.

I tried out the multithreaded dll. It did seem to be running an additional thread in total, but it didnt improve encoding speed. (working from a dvd rip, creating a 400x300 29.97 at about 950 bitrate.) I had the same fps when running with affinity set to one core or two, or running the original dll.

I suppose it could mean I am system/hardware limited somewhere outside of the cpu, but I would appreciate if anyone could say how to check for something like that. Is there a way to monitor the percentage of the memory bus that is being used? I have to wonder about it though, since I can open another instance of vdub/xvid and simultaneously do a second encoding session at the same frame rate as a single instance (doubling my overall cpu usage/overall movie frames converted), so I have to think that I am not being limited by memory or harddrive access rates.

So maybe the extra thread is running on the same cpu? or maybe it just isnt doing anything useful? or maybe it is not working at all? anybody?

btw I d/l the xvid beta version from koepi's site (the one that was supposed to be pre-patched.,- and it just crashed with a division by zero error.

Ugh. hope I said all I meant to say. I've had to wait 5 days to be allowed to post on this forum.

Anybody have any definite luck with this new dll?

[foxyshadis]

Most likely filtering (avisynth or vdub) is starving the encoder. You have to give you entire encoding chain; is it (A)GK? Try opening an avi directly and re-encoding with fast recompress, it should go nearly twice as fast with two threads as with one.

[Nrmf]

am i understanding this correctly theres a xvid build for dual core cpus...i have an amd 3800x2 i would like to test it out using avi.net is this possible.

[Alain_French]

Hi,

Is the optimization done for single pass ? Could someone test on dual core or tell me this is the same thing for single/double pass encoding.

Thanks

[celtic_druid]

Should be whenever bframe and or pframes are used.

[Alain_French]

ok thank you.

[captainvideo]

Most likely filtering (avisynth or vdub) is starving the encoder. You have to give you entire encoding chain; is it (A)GK? Try opening an avi directly and re-encoding with fast recompress, it should go nearly twice as fast with two threads as with one.

OK, this is my current test result.
started with Xvid 1.1.0-30122005
added the xvidcore.dll from beginning of thread.

Ran vdub-mpeg2 1.6.11 build23858 release mon dec 5 2005

opened vob from a dvd rip (720x480 29.97)

selected fast recompress and xvid compression
selected no audio

no resizing/no cropping/no deinterlacing

Xvid target bitrate 1018, H.263, singlepass, motionsearch#5(veryhigh),VHQ-1(mode decision),VHQ forB-No,
use chromamotion-yes, turbo-yes, trellis yes.

Running original dll for 9000 frames took 4:37 32.4fps
running smp dll for same 9000 frames took 4:04 36.9fps
For this particular instance I saw about 13% improvement in speed.


Ran test with same 9000 frames but turned on filtering
audio compress as mp3 160k
video
added internal deinterlace filter (blend)
added resize to 400x300 (nearest neighbor)

running smp dll for 9000 frames took 4:04 (244sec)36.9fps
running orig dll took 4:17 (257sec)35fps
about6% improvement

AMD4200X2,NFORCE4,2G mem,SATA HD, WinXPproSP2-32bit(amd dualcore driver/patch installed,MS-KB896256 not installed)

I never learned/used autoGK or avisynth, I sometimes use DVDx2.3, and I use several versions of virtualdub. Are we thinking that vdub is holding it back even in the fast recompress mode? or, is this about what i should expect?

[squid_80]

Are we thinking that vdub is holding it back even in the fast recompress mode?
I would say yes. Vdub-mpeg2 has a rather slow mpeg2 decoder.

[captainvideo]

ok for my last post I was using as an smp core dll,
a version with an md5 hash of
EE357239965A155ADDA0556C0ED19757

634,946 bytes modified 12/18/05, I think this was from first entry in thread.



I downloaded a different version that mentions bframes in title of zipfile.
634,946 bytes
modified 12/19


I did another test on same vob 9000 frames

I changed the xvid specs to (raising spec from prior test)

motion search 6 ultra
vhq mode 4
vhq for b frame = yes
use chroma=yes
turbo off
trellis yes


with filtering
audio to mp3 160
video deinterlace
video resize to 400x300


using original xvidcore.dll time=7:02 422sec 21.3fps

using 2thread withbframes xvidcore.dll time=5:18 318sec 28.3fps

looks like 30% faster


md5 sums on coredll's

my original xvidcore.dll md5=81CCA8C60DD2EDAF394B6E75FF8E325F filesize 761,856 bytes 12/30/05

my 2thread with b xvidcore.dll md5=D4531E2C2EA4D281A18855DC8F79E28C

my sys spec in earlier post.

[woah!]

cant really give any fps differences as i didnt do the single version test before i changed over.

but this screenshot should say all i want it to.. it sits at 50% cpu with the old xvid version, and its at 80-95% now... Nice 1


http://images.dr3vil.com/files/default/nice1.jpg

[Doom9]

Try opening an avi directly and re-encoding with fast recompress, it should go nearly twice as fast with two threads as with one.That still depends a lot on the output. If your source were a HD x264 stream, decoding would take a lot of CPU cycles. And on top of that, do not forget that VDub uses separate threads for reading and encoding.. thus even a single threaded XviD build would run faster on an SMP capable machine. By how much largely depends on the decoding complexity and your settings.. if encoding takes about as much cpu power as decoding, we have two threads using the same amount of cpu power which can be nicely placed on a core each, delivering an almost optimal CPU usage. If decoding or encoding is significantly more complex, then things would slow down.
If smp capable builds enter the picture, it gets more complex as suddenly we have three threads to be shuffled around on two cores...

[vipera]

Around 95% load on the first pass, 100% load on the second pass.
I have an AMD X2 4200+, using AutoGK to encode from DVD to 720x416 resolution.

[shpitz]

cant really give any fps differences as i didnt do the single version test before i changed over.

but this screenshot should say all i want it to.. it sits at 50% cpu with the old xvid version, and its at 80-95% now... Nice 1



HUGE misconception !

CPU usage has NOTHING to do with encoding speed... there's a lot more to it than just cpu usage...

you need to check encoding speed in terms of either fps or time it takes, not by cpu usage...

[shpitz]

my original xvidcore.dll md5=81CCA8C60DD2EDAF394B6E75FF8E325F filesize 761,856 bytes 12/30/05

my 2thread with b xvidcore.dll md5=D4531E2C2EA4D281A18855DC8F79E28C

my sys spec in earlier post.

can you post links to the binaries?

[sysKin]

Okay people, could you please don't post in this thread anymore? It's about old experimental code which is not used anymore anywhere, and I'm only confused if you still use that old code or new, offcial one.

@moderators, could you just close this thread please :)

[Koepi]

The currently "official" multithreading-thread is here: http://forum.doom9.org/showthread.php?t=107783

Please use that one instead; I'll close this one now.

Cheers
Koepi