Multithreaded XviD - official thread

[shpitz]

yeah, it is well known that xeons are chokers when it comes to memory-cpu bandwidth. opterons and dual-cores should do a better job in that respect.

i might disable HT and try again, from my tests it appears that 2 threads is optimal as with 4 it really chokes.

i will also disable PAT and see if it makes a difference.

expect more tomorrow ;-)

[Revgen]

@shpitz

The SetMTmode filter needs to be set before you're source like this:

SetMTmode(2,2)
mpeg2source("yourdrive:\yourd2v.d2v")
Yourfilter()

Also, are you actually using a real dual-core or dual CPU solution? HT will not gain as much as the real thing.

HT in some cases can even slow down performance.

[sysKin]

yeah, it is well known that xeons are chokers when it comes to memory-cpu bandwidth.

Yes, and with an application that works on two cpus, *additional* data transfers happen - between the cpus. They add up to the memory bandwidth because they all use FSB as the data pipe.

This code doesn't use a lot of data to communicate, but it does depend on this communication happening fast. If FSB is delaying the information, one of the threads suddenly has nothing to do (because it's not informed how much another thread has done) and must wait...


Actually, I'll make some tests how this code's speed depends on Hypertransport speed. Good idea :)

[Yong]

I failed to compile the lastest cvs code(libxvidcore)...
Here is the part of msys output:
D: =build
C: ./decoder.c
In file included from ../../src/bitstream/../motion/motion_smp.h:34,
from ../../src/bitstream/../encoder.h:158,
from ../../src/bitstream/bitstream.h:31,
from ../../src/decoder.c:40:
c:/msys/mingw/bin/../lib/gcc/mingw32/3.4.5/../../../../include/winbase.h:552: error: syntax error before "DWORD"
c:/msys/mingw/bin/../lib/gcc/mingw32/3.4.5/../../../../include/winbase.h:556: error: syntax error before "DWORD"
c:/msys/mingw/bin/../lib/gcc/mingw32/3.4.5/../../../../include/winbase.h:558: error: syntax error before "ftLastAccessTime"
c:/msys/mingw/bin/../lib/gcc/mingw32/3.4.5/../../../../include/winbase.h:559: error: syntax error before "ftLastWriteTime"
[...]
alot of erros, "make distclean && make -s" doesnt help...
But dshow decoder compiling work again, thx xvid devs :)

EDIT: may be there's something wrong with motion_smp.h,
i revert it to older revision then compling works again :p

[shpitz]

@shpitz

The SetMTmode filter needs to be set before you're source like this:

SetMTmode(2,2)
mpeg2source("yourdrive:\yourd2v.d2v")
Yourfilter()

Also, are you actually using a real dual-core or dual CPU solution? HT will not gain as much as the real thing.

HT in some cases can even slow down performance.

dual xeon 3ghz, and yes, i've added the setmtmode before mpeg2source.

i will need to thoroughly investigate it, the computer seems to choke completely in terms of smp encoding.

[Zep]

dual xeon 3ghz, and yes, i've added the setmtmode before mpeg2source.

i will need to thoroughly investigate it, the computer seems to choke completely in terms of smp encoding.


It depends on how low level syskin went. Dual CPU sucks when the code is lower level like say each CPU working on part of the same marcobock VS each CPU working on half of a video frame and never accessing the same data so no need to sync much or sync fast and thus each CPU is always crunching with very few wait states. Higher level makes scaling much better also like when throwing a render farm at a project.

[woah!]

i am getting nearly double the framerate in encoding by using Vdub with SetMTMode(2) and the lastest xvidcore.dll

it sits at about 85% cpu and from 10fps for 1 thread to 18-19fps with 2 threads:


going from 1920x1088i 30fps to 720x304 23.976fps :

SetMTMode(2)
mpeg2source("G:\INDEXED.d2v")
crop(0,144,-0,-148)
Telecide(order=1,guide=1,post=2).Decimate(mode=0)
#TomsMoComp(1,30,1)
#FDecimate(rate=23.976,threshold=3.5)
#Kerneldeint(order=1,sharp=true)
#Kernelbob(order=1,sharp=true) # double framerate 60fps
#ColorMatrix("Rec.709->Rec.601",mmx=true,hints=false)
#LanczosResize(848,480)
#BicubicResize(704, 400, 0.33, 0.33, 8, 8, 1904, 1072)
#bicubicResize(704,400)
LanczosResize(720,304)
asharp(2,4,hqbf=true)
#trim(61000, 61720)


thx for this nice upgrade which i could never have got with any single-core cpu upgrade.

http://images.dr3vil.com/files/default/2core.jpg

heres a clip of the output file:

http://s19.yousendit.com/d.aspx?id=051E4GRBM6JOI18W03XLOVCLQH

[shpitz]

woah!,

can you post a link to the xvid version you are using?

also, which avisynth version are you using?

did you install anything else (such as MT 0.5 filter) ?

can you also post your xvid encoding settings you used and which matrix? the encode looks great.

[AmazingRando]

:D I just wanted to thank you guys for your effort in making XviD SMP capable. I've used XviD for years now on hundreds of encodes and it's been great.

Last fall I reluctantly decided to move to Divx 6 since XviD wasn't SMP capable. I found that I needed to encode at "extreme" or "insane" quality to get the same quality as with XviD. But now I'm thrilled to be using XviD once again :D I did several test encodes last night to compare quality and performance between XviD 1.2 and Divx 6.1.1. On a dual-core 2.8Ghz Pentium D I was able to do a two-pass encode of a 44 min DVD ripped TV episode (720x480x24fps) in 47 mins including audio encoding and muxing. Compare that to 2 hours 2 mins for Divx 6.1.1 on insane quality. I judge the quality to be comparable between the two. BTW, the Pentium D 820 doesn't support HT, just two physical processors (cores). I have it set to 2 threads and get about 85-90% CPU utilization.

So, I'm one happy guy :D . Keep up the great work.

AR

PS. Future AVC support would be great!

[Koepi]

Igor Levicki asked me to post this (which I'll gladly do as it might be helpful):
In case you folks still haven't figured out the way to detect number of CPUs in the system let me help a bit:



#include <stdio.h>

typedef unsigned long u32;

u32 GetLogicalCPUCount(void)
{
u32 logical_cores = 0;

__asm {
xor eax, eax
cpuid
cmp eax, 1
jb no_logical_cores
mov eax, 1
cpuid
shr ebx, 16
and ebx, 0xFF
mov dword ptr [logical_cores], ebx
no_logical_cores:
}

return logical_cores;
}

u32 GetPhysicalCPUCount(void)
{
u32 physical_cores = 1;

__asm {
xor eax, eax
cpuid
cmp eax, 4
jb no_physical_cores
mov eax, 4
xor ecx, ecx
cpuid
shr eax, 26
and eax, 0x1F
add eax, 1
mov dword ptr [physical_cores], eax
no_physical_cores:
}

return physical_cores;
}

u32 HasHyperThreading(void)
{
u32 has_htt = 0;

__asm {
xor eax, eax
cpuid
cmp eax, 1
jb no_htt
mov eax, 1
cpuid
test edx, 0x10000000
jz no_htt
mov dword ptr [has_htt], 1
no_htt:
}

return has_htt;
}

int main(int argc, char* argv[])
{
u32 pcores, lcores, htt;

lcores = GetLogicalCPUCount();
pcores = GetPhysicalCPUCount();
htt = HasHyperThreading();

printf("Number of logical cores = %ld\n", lcores);
printf("Number of physical cores = %ld\n", pcores);

if (htt && ((lcores / pcores) > 1)) {
printf("CPU has HyperThreading = YES\n");
} else if (htt && (lcores == 1)) {
printf("CPU has HyperThreading = YES, DISABLED\n");
}

return 0;
}


If you have any doubts, check AP-485 (http://developer.intel.com/design/xeon/applnots/241618.htm) document. Same should apply to AMD.

Take note that logical cpu count == physical cpu count on current dual-core CPUs and that the presence of HyperThreading needs to be detected in a slightly different way than before -- having HTT bit set and 2 logical cores doesn't mean you have HTT. Pentium D 955 reports 4 logical and 2 physical cores. I understand that physical cores are counted as logical for compatibility reasons.

Perhaps it wouldn't hurt to add more checks (CPU familiy/model, etc) but I believe that this code should be safe to execute even on older CPUs, at least those which have CPUID instruction so you might at least want to add the check for that.

I would like to comment on FSB and bandwidth issues, I have done some threading of median filtering on my Pentium D 930 recently and I managed to get 2x speedup. With careful threading I bet you can do it too -- just make the threads work independently. I did it by making a queue from which two threads dequeue "packets" for processing which are queued by the third, main thread.

[squid_80]

Now that's what I was talking about.;) Would fit nicely into cpuid.asm, no?

[seehowyouare]

On a dual-core 2.8Ghz Pentium D I was able to do a two-pass encode of a 44 min DVD ripped TV episode (720x480x24fps) in 47 mins including audio encoding and muxing. Compare that to 2 hours 2 mins for Divx 6.1.1 on insane quality.

Sorry to be a party pooper but my x2 4400 Windows x64 system doesn't agree, regardless of number of threads I set :eek:
And I tried using GK, AutoGK and RealAnime. Fastest is DivX 6.11 at 75fps encoding speed, then XviD-1.1.0-30122005.exe and XviD-1.2.-127-25022006.exe is the slowest.
FWIW - the x264 standard codec uses only 1 core at 65% CPU @ 8fps. woot ! Yet to try Sharktooth's build.

Maybe my testing is wrong so if someone has an "agreed" test setup/procedure/script I can follow to fairly test the 1 thread vs 2 thread codec speeds on my PC I'd appreciate it.

Thanks

[shpitz]

anyone that experiences a speed-up, can you please post the exact versions (direct links would help a lot) of the components you are using (codec, avisynth, etc...).

seehowyouare's experience is kinda encouraging so-to-speak, maybe it's not a hardware issue on my end after all...

i did disable PAT but speed was not affected in almost any way, next thing i will try is disable HT in bios.

expect more to come...

[celtic_druid]

Well if you want an alternative to try:
http://ffdshow.faireal.net/mirror/XviD/xvid.cvs.head.2006.02.28.7z
http://ffdshow.faireal.net/mirror/XviD/XviD.cvs.head.exe
Same thing just with/without installer.

[AmazingRando]

Maybe my testing is wrong so if someone has an "agreed" test setup/procedure/script I can follow to fairly test the 1 thread vs 2 thread codec speeds on my PC I'd appreciate it.


:confused: That's strange. Is it me or does it seem like the people who are having trouble are AMD X2 users? I'll post my software, versions, etc. tonight. I have a dedicated system for encoding so it's a very clean install. Moreover, I'm using either GK codec pack 1.9 and GK rip pack .35 pack2. The only changes are keopi's latest xvid 1.2 binary and then the Divx stuff. For Divx I'm using Divx Pro 6.1.1 and Dr. Divx OSS 2.0 beta 7 with the current DrFFMPEG. Pretty stock really. More later...

AR

[shpitz]

:confused: That's strange. Is it me or does it seem like the people who are having trouble are AMD X2 users?

not only, i'm a xeon (intel) user...

[seehowyouare]

FWIW - the x264 standard codec uses only 1 core at 65% CPU @ 8fps. woot ! Yet to try Sharktooth's build.

I tried Sharktooth's x264 build set at 4 threads and encoding speed in VirtualDubMod increased from 8fps using the standard single thread x264 codec to 15fps on identical encoding job. CPU usage is much higher on both cores so I know multi thread stuff does work on my system.
Screenshots of Task Manager using single thread (http://img312.imageshack.us/img312/488/x264stdr4088gh.png) x264-Std_r408exe and multi thread (http://img401.imageshack.us/img401/8928/x264445installexe7wv.png)x264-445-install.exe

[Nic]

Yesterday I got a dual cpu box with 3 gig of ram dropped on my desk at work. Each cpu is a dual core opteron 280. Gives me a good excuse to do some XviD multithreaded encoding :)

I'm not really noticing any speed up. But at present i've got a few tasks in the background, so I need to clean things and do some proper testing.

I'll let you know.

-Nic

[dimzon]

Yesterday I got a dual cpu box with 3 gig of ram dropped on my desk at work. Each cpu is a dual core opteron 280. Gives me a good excuse to do some XviD multithreaded encoding :)

I'm not really noticing any speed up. But at present i've got a few tasks in the background, so I need to clean things and do some proper testing.

I'll let you know.

-Nic
Can You try elder too?
http://forum.doom9.org/showthread.php?t=100766

Thanx!

[Doom9]

one thing not to be forgotten (and I think I'm repeating myself here already) is that even the single threaded XviD build makes use of SMP. Not actually the codec, but since almost all people use XviD from within Virtualdub, the multithreaded architecture of VirtualDub comes to the rescue. You have one thread reading the input (and thus decoding the source), and one doing the actual encoding.. and so the thread scheduler puts the encoding thread on one core, and the decoding thread on another, which in turn means the encoder will max out one core, and the decoding thread will run as fast as the encoder thread can process the data, resulting in not such a shabby speedup. If you really want to see the raw difference, encode with a singlethreaded and a multithreaded encraw build..
At the settings used for the codec comparison, the CPU usage of XviD was in the +70% (no other tasks running but whatever windows is running) which is a lot closer to the cpu usage I'm getting when running a commandline smp capable encoder (x264, elecard, nerodigital).
If you really want to see the difference even in VDub, you can temporarily switch off one core (add /onecp uto boot.ini in the appropriate line, then reboot) ,make a test (nbthreads set to 0), remove the line, reboot, make the test again with nbthreads set to 2, and compare the encoding time.

[celtic_druid]

mencoder just got updated to support threads with XviD now to.

[seehowyouare]

and I think I'm repeating myself here already) is that even the single threaded XviD build makes use of SMP.

I am sure you are because of people like me :-)

Ok, I'm an encoding noob and just got my first dual core CPU.
Can someone confirm or deny if I got this threading/dual core right in my head.

1) Decoding only uses 1 thread
2) Encoding can use multiple threads (dep. on build etc)
3) VDM uses threads to seperate encoding and decoding tasks
4) VDM can allocates threads to CPUs.
5) Having 2 x CPU or 2 x dual core means VDM can decode on 1 x CPU and encode on the 2nd CPU.
6) Setting 2 x XviD threads on my dual core PC will make no difference to speed as VDM controls thread and CPU allocation.

PS - I will receive a dual core 2.8 Intel to keep the x2 4400 compnay next week.

[Ice =A=]

@seehowyouare:
I can at least help in some regards:
1. No, luckily there are also more and more SMP DEcoders available, like Nero, DivX6.1.1 or Quicktime 7 (which is nevertheless very slow on PCs), and some more coming (like CoreAVC).
2. yes
6. There is a difference between the two: Most encoding programs can only separate threads like decoding of source material and encoding or audio and sound. Since the encoding itself needs most of the computing time thge most gains come from multithreaded encoding, which is where the SMP optimized XviD comes handy.
There is only one encoding program I know of which can really designate seperate threads to a non SMP XviD, namely ELDER...

[sysKin]

1. No, luckily there are also more and more SMP DEcoders available, like Nero, DivX6.1.1 or Quicktime 7 (which is nevertheless very slow on PCs), and some more coming (like CoreAVC).

I consider that VERY unlikely. In fact, I consider that almost impossible. Other than postprocessing, I can almost guarantee that there's no multithreaded ASP decoders at all.


1) Decoding only uses 1 thread
2) Encoding can use multiple threads (dep. on build etc)
3) VDM uses threads to seperate encoding and decoding tasks
Yes, yes, yes.
4) VDM can allocates threads to CPUs.No, windows kernel does that.
5) Having 2 x CPU or 2 x dual core means VDM can decode on 1 x CPU and encode on the 2nd CPU.Yup.
6) Setting 2 x XviD threads on my dual core PC will make no difference to speed as VDM controls thread and CPU allocation.No, because encoding and source decoding take different processing power. The more threads there are, the more balance between CPUs.
HOWEVER if filtering and encoding take similar power, such as more complex avs script combined with fast-1st-pass, then indeed just one encoding thread seems to be faster.

[dimzon]

I consider that VERY unlikely. In fact, I consider that almost impossible. Other than postprocessing, I can almost guarantee that there's no multithreaded ASP decoders at all.
theoretically it's possible to split source bitstream by I-frame:
IpppIpbbp and decode each piece in separate thread
unfortunally this scheme is unusable for normal playback but can be used for speedup during transcoding (on more than 2 CPU machines with huge RAM avaluable)

[dimzon]

Or, maybe, it's posible to decode B-frames @ separate thread...

[Ice =A=]

I consider that VERY unlikely. In fact, I consider that almost impossible. Other than postprocessing, I can almost guarantee that there's no multithreaded ASP decoders at all. Now that I'm thinking of it technically, it really seems difficult. But I know that with Quicktime there is a big speed difference between one and two cores, and regarding Nero, I at least know it's using (whatever that means) more then one core.
Maybe they can do different threads for luminiscense and chroma channels or compute different parts of the image in parallel, no idea... :o

[foxyshadis]

I know Elecard/Nero and CoreAVC at least do Cabac decoding on a separate thread, that's a huge processor sink for dvd video and up and the most obvious and easiest speedup. I'm sure they both have a smart scheduler to go with it that tries to partition the work to be done across cpus as much as possible.

[sysKin]

I know Elecard/Nero and CoreAVC at least do Cabac decoding on a separate thread, that's a huge processor sink for dvd video and up and the most obvious and easiest speedup.
Yeah this is why I said ASP decoder. For AVC, there are at least several possibilities to either pipeline (like cabac->everything else ->deblocking) or work in parallel (deblocking itself, or decode slices in parallel if file was encoded with slices).

But ASP... yeah I suppose you can reinvent entire decoder to decode entire frames ahead (either on GOP boundary or b-frames) but that sounds weird. Within one frame, this sounds impossible.
Postprocessing can be split into slices. In fact I might do that if you say that's useful.

As for quicktime, are you sure it's not audio decoder running on another CPU that makes this difference?

[lantern]

It was built with mingw/msys. I have left the thread count to 0 or 1 or 2. I have a P4 w/Hyperthreading.

Crash report from VDub:

VirtualDub-MPEG2 crash report -- build 23843 (release)
--------------------------------------

Disassembly:
0d0495e0: ffc7 inc edi
0d0495e2: 43 inc ebx
0d0495e3: 1800 sbb [eax], al
0d0495e5: 0000 add [eax], al
0d0495e7: 00c7 add bh, al
0d0495e9: 43 inc ebx
0d0495ea: 1c00 sbb al, 00h
0d0495ec: 0000 add [eax], al
0d0495ee: 00c7 add bh, al
0d0495f0: 43 inc ebx
0d0495f1: 1000 adc [eax], al
0d0495f3: 0000 add [eax], al
0d0495f5: 00c7 add bh, al
0d0495f7: 43 inc ebx
0d0495f8: 1400 adc al, 00h
0d0495fa: 0000 add [eax], al
0d0495fc: 00c7 add bh, al
0d0495fe: 43 inc ebx
0d0495ff: 0800 or [eax], al
0d049601: 0000 add [eax], al
0d049603: 00c7 add bh, al
0d049605: 43 inc ebx
0d049606: 0c00 or al, 00h
0d049608: 0000 add [eax], al
0d04960a: 00c7 add bh, al
0d04960c: 0300 add eax, [eax]
0d04960e: 0000 add [eax], al
0d049610: 00c7 add bh, al
0d049612: 43 inc ebx
0d049613: 0400 add al, 00h
0d049615: 0000 add [eax], al
0d049617: 00c7 add bh, al
0d049619: 83900100000000 adc dword ptr [eax+01], 00h
0d049620: 0000 add [eax], al
0d049622: c7839401000000 mov dword ptr [ebx+194], 00000000
000000
0d04962c: c7838801000000 mov dword ptr [ebx+188], 00000000
000000
0d049636: c7838c01000000 mov dword ptr [ebx+18c], 00000000
000000
0d049640: c7838001000000 mov dword ptr [ebx+180], 00000000
000000
0d04964a: c7838401000000 mov dword ptr [ebx+184], 00000000
000000
0d049654: c7837801000000 mov dword ptr [ebx+178], 00000000
000000
0d04965e: c7837c01000000 mov dword ptr [ebx+17c], 00000000
000000
0d049668: c78570ffffff00 mov dword ptr [ebp-90], 00000000
000000
0d049672: c78574ffffff00 mov dword ptr [ebp-8c], 00000000
000000
0d04967c: 83b8ec00000010 cmp dword ptr [eax+ec], 10h <-- FAULT
0d049683: 89bbf0000000 mov [ebx+f0], edi
0d049689: 0f8463040000 jz 0d049af2
0d04968f: 8b7dd4 mov edi, [ebp-2ch]
0d049692: 8d47e1 lea eax, [edi-1fh]
0d049695: 85c0 test eax, eax
0d049697: 0f8e2d070000 jle 0d049dca
0d04969d: 89f9 mov ecx, edi
0d04969f: bbffffffff mov ebx, ffffffff
0d0496a4: 8b75c8 mov esi, [ebp-38h]
0d0496a7: d3eb shr ebx, cl
0d0496a9: 89c1 mov ecx, eax
0d0496ab: 89b59cfeffff mov [ebp-164], esi
0d0496b1: 21f3 and ebx, esi
0d0496b3: d3e3 shl ebx, cl
0d0496b5: b920000000 mov ecx, 00000020
0d0496ba: 29c1 sub ecx, eax
0d0496bc: 8b45cc mov eax, [ebp-34h]
0d0496bf: d3e8 shr eax, cl
0d0496c1: 09c3 or ebx, eax
0d0496c3: 8d4701 lea eax, [edi+01h]
0d0496c6: 83f81f cmp eax, 1fh
0d0496c9: 89c6 mov esi, eax
0d0496cb: 7625 jbe 0d0496f2
0d0496cd: 8b55cc mov edx, [ebp-34h]
0d0496d0: 8b75d8 mov esi, [ebp-28h]
0d0496d3: 8945d4 mov [ebp-2ch], eax
0d0496d6: 8955c8 mov [ebp-38h], edx
0d0496d9: 89959cfeffff mov [ebp-164], edx
0d0496df: 8b db 8bh

Windows 5.1 (Windows XP build 2600) [Service Pack 2]

EAX = 0d3dc5ac
EBX = 0d3baa2c
ECX = 00000000
EDX = 0000007e
EBP = 0d95ee60
ESI = 00000004
EDI = 0000001a
ESP = 0d95eca8
EIP = 0d04967c
EFLAGS = 00010246
FPUCW = ffff027f
FPUTW = ffffaaaa

Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'xvidcore'... ...while running thread "Processing" (thread.cpp:150).

Pointer dumps:

EBX 0d3baa28: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ESP 0d95eca8: 0d95ee28 00000003 00000000 0d95ee28 00000000 00000001 00000001 00000010
0d95ecc8: 0000000e 0d95ee28 00000001 00000000 00000000 00000000 00000000 003283d8
0d95ece8: 00000000 0d95ed4c 0d95ede0 00000001 0000000b 337f0000 0d95ed18 0d95ed18
0d95ed08: 0d215d50 80000200 000001b6 00acfd68 00000000 00000003 00000040 0000001f
EBP 0d95ee60: 0d95ef40 04602e03 0d215c80 0d95eee8 00000000 00000000 00000000 00000000
0d95ee80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0d95eea0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0d95eec0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Thread call stack:
0d04967c: xvidcore!0000967c
0460c295: xvidvfw!DriverProc [04600000+bd80+515]
7c910833: ntdll!RtlAllocateHeap [7c900000+105d4+25f]
7c910833: ntdll!RtlAllocateHeap [7c900000+105d4+25f]
7c80b5b8: kernel32!GetModuleHandleA [7c800000+b529+8f]
7c80b58c: kernel32!GetModuleHandleA [7c800000+b529+63]
7c80b5a1: kernel32!GetModuleHandleA [7c800000+b529+78]
7c80b4b6: kernel32!GetModuleFileNameA [7c800000+b357+15f]
7c80b4cb: kernel32!GetModuleFileNameA [7c800000+b357+174]
7c910732: ntdll!RtlAllocateHeap [7c900000+105d4+15e]
7c910732: ntdll!RtlAllocateHeap [7c900000+105d4+15e]
7c9106ab: ntdll!RtlAllocateHeap [7c900000+105d4+d7]
7c9106eb: ntdll!RtlAllocateHeap [7c900000+105d4+117]
7c910732: ntdll!RtlAllocateHeap [7c900000+105d4+15e]
7c910732: ntdll!RtlAllocateHeap [7c900000+105d4+15e]
7c9106ab: ntdll!RtlAllocateHeap [7c900000+105d4+d7]
7c9106eb: ntdll!RtlAllocateHeap [7c900000+105d4+117]
7c910732: ntdll!RtlAllocateHeap [7c900000+105d4+15e]
7c911538: ntdll!wcsncpy [7c900000+10a8f+aa9]
7c911596: ntdll!wcsncpy [7c900000+10a8f+b07]
7c9106eb: ntdll!RtlAllocateHeap [7c900000+105d4+117]
7c910833: ntdll!RtlAllocateHeap [7c900000+105d4+25f]
7c910895: ntdll!RtlImageDirectoryEntryToData [7c900000+10856+3f]
7c910833: ntdll!RtlAllocateHeap [7c900000+105d4+25f]
7c910895: ntdll!RtlImageDirectoryEntryToData [7c900000+10856+3f]
7c9037bf: ntdll!RtlConvertUlongToLargeInteger [7c900000+3745+7a]
7c90da54: ntdll!NtFreeVirtualMemory [7c900000+da48+c]
7c918331: ntdll!RtlReAllocateHeap [7c900000+179fd+934]
7c90d4ea: ntdll!NtAllocateVirtualMemory [7c900000+d4de+c]
7c9180ff: ntdll!RtlReAllocateHeap [7c900000+179fd+702]
7c911bff: ntdll!RtlInitializeCriticalSection [7c900000+11b2d+d2]
7c91825d: ntdll!RtlReAllocateHeap [7c900000+179fd+860]
0050ad37: VDResamplerSeparablePointRowStageMMX::Process()
00509f9c: VDResamplerSeparableStage::ProcessRow()
00509a6a: VDResamplerSeparableStage::ProcessPoint()
75a718a8: MSVFW32!ICSendMessage [75a70000+187d+2b]
75a74c09: MSVFW32!ICCompress [75a70000+4ba6+63]
004ae882: VideoSequenceCompressor::PackFrameInternal()
004ae536: VideoSequenceCompressor::packFrame()
75a718a8: MSVFW32!ICSendMessage [75a70000+187d+2b]
75a74c4d: MSVFW32!ICDecompress [75a70000+4c10+3d]
0047f4ea: Dubber::WriteVideoFrame()
0047ee18: Dubber::WriteVideoFrame()
0047fa63: Dubber::ThreadRun()
004df5fe: VDThread::StaticThreadStart()
005285bf: _threadstartex@4()
7c80b50b: kernel32!GetModuleFileNameA [7c800000+b357+1b4]

-- End of report

[woah!]

mencoder just got updated to support threads with XviD now to.

where would this version be as i looked around the places i have linked and havent seen it?

[celtic_druid]

http://www1.mplayerhq.hu/cgi-bin/cvsweb.cgi/main/libmpcodecs/ve_xvid4.c.diff?r1=1.24&r2=1.25

[devaster]

mencoder just got updated to support threads with XviD now to.

how i activate it ? i cant find it in manual ... or i am blind ?!?

[celtic_druid]

Probably the manual you are looking at hasn't been updated yet. According to the above it is just threads=#. So -xvidencopts pass=1:threads=2

[devaster]

:thanks: ooh thanx

[DiJayy]

From what I can tell, when encoding MPEG2 transport streams (~18mbps) to XviD using avisynth and vdubmod or avs2avi, using 2 threads is actually a bit slower than 0, maybe because the decoder's doing so much work on one thread that the second xvid thread crowds it, but I don't know enough about this stuff to have a fair estimate. On 0 threads cpu usage is ~80%, one 2 threads its 100%

AMD Athlon 64 X2 3800+ using Windows XP x64

I also tried playing with the process priority but it doesn't change much, usually just makes the fps less stable and more liquid, but doesn't actually speed up anything.

[shpitz]

On 0 threads cpu usage is ~80%, one 2 threads its 100%

again, cpu usage has NOTHING to do with encoding speed.

if you want to assess if you had a speed-up or not you should only look at the TIME it took to encode...

only cause the cpu is working harder doesn't mean it is working more efficient...

[ilhyfe]

Hi,
Encoded using 2 threads.

AMD 4200+ , Avisynth 2.5.6 , VDM 1.5.10.2


First smp build:

1st pass : 56min (41,5fps)
2nd pass : 2h 36min (14,9fps)

New build:

1st pass : 50min (46,5fps)
2nd pass : 2h 17min (17fps)


I'd be interested in some more details. What kind of source? Any filters? Which destination resolution?
A friend of mine gets more than 100 fps while encoding a DVB stream to xvid (512*384) while I get max 50. He's using a X2 4400+, I a X2 3800+. Both of my cores are working @ ~90% while his are both on 100%.
We are both using koepis smp build.

so long...

[Wam7]

:thanks: for the Xvid SMP, much appreciated. Doing a quick calculation I get about a 12% improvement in 2nd pass speeds using 2 threads.

[ChronoCross]

Is there a way to get the latest cvs co to recognize that I have pthread.h? I can't build the SMP version without it. Configure says I don't have it.

Building x264 SMP works fine (doesn't search for pthread as far as I know)

btw I know this thread is quite old however it is the OFFICIAL thread and therefore compilation problems might be best suited here.

[sysKin]

Is there a way to get the latest cvs co to recognize that I have pthread.h? I can't build the SMP version without it. Configure says I don't have it.
Unfortunately all pthread logic, including makefile, was put in blindly. I don't know about others but from what I know, this was only ever tested with win32 and VC++.

Test results and patches appreciated.

[celtic_druid]

checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking for pthread_join in -lpthread... yes


No problem here.

[ChronoCross]

checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking for pthread_join in -lpthread... yes


No problem here.


where is pthread located for you in you mingw?

[akupenguin]

Yeah this is why I said ASP decoder. For AVC, there are at least several possibilities to either pipeline (like cabac->everything else ->deblocking) or work in parallel (deblocking itself, or decode slices in parallel if file was encoded with slices).

But ASP... yeah I suppose you can reinvent entire decoder to decode entire frames ahead (either on GOP boundary or b-frames) but that sounds weird. Within one frame, this sounds impossible.
Postprocessing can be split into slices. In fact I might do that if you say that's useful.
GOPs or b-frames would only work for certain restricted bitstreams... GOP parallelism would require a ridiculously large buffer with the standard 10 second keyframe limit, and b-frame parallelism fails with adaptive b-frames.

So I propose a different method of frame-level parallelism (which, btw, is codec-agnostic and would work in encoding too):
Decode N consecutive frames with N threads. Whenever a thread tries to decode a motion vector that points into a region of the reference frame that hasn't been decoded yet, it stalls until the thread responsible for that frame has decoded enough. If the movie has enough b-frames that only one thread at a time is in an i/p-frame, then there will be no stalls and it's just like b-frame parallelism. If not, it's slightly less efficient but still works.

Alternate plan: Use dxva/xvmc except with software emulation of the video card. (probably harder to implement, and definitely uses more bus bandwidth)

[DaForce]

Hey chaps and chapettes,

My friend finally got his Dell machine up and running, it has a Pentium D 2.8ghz.
So we wanted to do some speed comparisons between my machine (x2 4400) and his.

So using Virtualdubmod and the latest Koepi Xvid SMP build (Love your work Koepi) we set off converting a sample piece of footage.

Here are my results for the tests and the details of the file used for encoding

Virtualdubmod + xvid - 2pass
2m34s thread @ 0
2m38s thread @ 1
2m 8s thread @ 2
2m10s thread @ 3

mewig + xvid - 2pass
2m11s

Original file details:
720x576 MPEG2 25FPS
1111frames (duration 0:44s)

Now his machine did it in 1m58s at the time i didnt note what his thread was set to but it could very well have been 0 (I believe it was).
So my machine is some 25% slower than his.

Next we used mewig to convert using x264 and my machine was almost 30% quicker (which is more like what i expected in xvid as well).
NOTE: we had identical settings in both VDM and Mewig

So what were we doing wrong? does xvid (smp build) not like AMDs as much as it likes Intels.

Any suggestions or ideas would be great. I really did expect my machine to beat him considerably in xvid.

[sysKin]

I really did expect my machine to beat him considerably in xvid.
And you should, that's the slowest P-D of them all.

Very few people tested with multi-core pentiums (or at least, very few people admitted it), but their results were quite bad so far. This is what I'd expect, given P4's lack of communication between cores (like hypertransport).

Perhaps you made some silly mistake, like a different setting somewhere? Or Full Recompress in vdub?

[DaForce]

Well yeah exactly.. i expected a result the same as the x264 thru mewig... where i was 30% faster.

I have retested again today and got the same problem. Even when using mewig which uses its own xvidcore.dll i get about the same time ..about 2m10s which is slower than the P4D but about 15s

in VDM we are doing a full recompress of the MPEG2 stream to an XVID video, leaving the audio as is (direct stream copy) and not much else really. I have double check the settings many times.
Im kinda stumped. So basically 2 different programs using 2 different xvidcore.dll's produce the same miserable time for me.

I did a quick render test in 3dsmax using vray and my resulting time on a benchmark scene was as expect.. so my puter isnt running funky or anything.

Any ideas mate?

p.s. Im in Australia as well... Canberra :D

[DaForce]

hmmm i can get the encoding time down down to 1m43s (instead of 2m08s) using Virtualdub (not virtualdubmod) and using avisynth and dvd2avi to be able to load the mpeg2 into virtualdub.

Which is weird. So its 25s quicker when using the above method than when using virtualdubmod. Doesnt make much sense.

Havnt tried the above on my friends P4-D tho.

[GodofaGap]

VirtualDub does encoding and decoding in separate threads, but it could also be that the MPEG2 decoder in VDM is just slower than DGIndex.

[DaForce]

I tried using the avisynth route in VDM as well and got basically the same time.

So Virtualdub uses seperate threads and VDM doesnt?