XviD 1.2 -127 Realtime encoding issue

[mod]

I've tried capturing some TV with VDub 1.6.12 and latest 1.2 -127 unstable build by Koepi (nice work! :)) but the codec seems to kill my machine if used for capturing.
I've tried various codec settings (starting from default ones), the issue remains (If anyone wants I'll post)
I've tried also various VDub(s) settings.. I'll post if asked :)

Other tries before starting this thread:

Stable 1.1.0

VDub 1.6.12 ------------------- Bitrate control problems
VDub 1.6.11 ------------------- Works
VDub MPEG2 1.6.11 ------------ Works
VDubMod 1.5.10.2 ------------- Works

UnStable 1.2 -127

VDub 1.6.12 ------------------ 99% cpu usage WITHOUT REAL ENCODING
VDub 1.6.11 ------------------ 99% cpu usage WITHOUT REAL ENCODING

others: 99%...

I know that latest VDub 1.6.12 is a "ok let's do it" release for the realtime encoding (works perfectly with other codecs), but the issue appears only with XviD, so I posted it here.

NOTE: VDub doesn't crash during "encoding", simply uses all cpu WITHOUT compressing any frame. This takes me about 5 mins to take back my Task Manager and kill process.

OS: WinXP Pro Sp2
DUT: AthlonXp 2700 Thoroughbred, 1024 MB DDR PC3200, Asus A7V880, Asus 9600 XT Vivo 128 MB
SIGNAL: PALB

[squid_80]

I'm seeing the same thing if I set virtualdub's priority to anything below normal. Seems I'm not the only one, (http://forum.doom9.org/showthread.php?p=764598#post764598) either. I'm thinking maybe the second thread doesn't inherit the priority of the first, causing a deadlock.

Edit: It's not just Koepi's build, sysKin's build behaves the same.

[Koepi]

That buld is marked unstable and experimental for a good reason ;)

Also, it has two threads hardcoded. This seems to make things bad on single processor machines. I was starting to code a CPU counter yesterday but I didn't make much progress (until now it only works for: AMD dualcore and Intel with HT; it detects everything else as single core.) If someone has a reliable 10-liner of usable inline assembly for processor count I'd plug it in and set the numbers of threads accordingly.

Unfortunately I'm at work the whole day so I could start looking into that this earliest this evening.

Cheers
Koepi

[squid_80]

Might have something here: http://www.virtualdub.org/blog/pivot/entry.php?id=62#body
Every once in a while, a poor soul will reinvent the really bad version of this by using Sleep(0), which consumes a lot of CPU time. Sleep(0) on Win32 also only yields to threads of equal or higher priority. I always know when someone has used one of these because all the fans suddenly turn on in my computer. What's really fun is when the zero-wait loop ends up starving a lower-priority thread that is supposed to set the flag, and what's supposed to be a 10ms wait becomes 400ms or even seconds. This situation is known as priority inversion.

[Koepi]

That sounds exactly like the problem :)

Cheers
Koepi

[Doom9]

@Koepi: why not simply read the environment variable that specifies the number of processors? I guess the problem is that linux doesn't have that variable, right?
There's an other option of course, make the whole thing user-settable. Then at least for the VfW frontend you can simply read out the environment variable that gives you the number of CPUs and pre-set this automatically.

[amango]

I installed 1.2 but it killed all my available Codecs. Only XVID is listed in Virtual Dub? What can I do?

[mod]

I don't know the difficulty of coding what I'm going to say in the XviD core (not my area), but what about having in the encoder the possibility of setting the number of threads? If the detection is difficult let the user decide, maybe with a default n° = 1.

[Sagittaire]

no problem with celtic duid build ...

[squid_80]

It's do-able... I've got xvidcore set up to support an arbitrary number of threads. The VFW actually calls the core to find out how many threads should run (using xvid_global with XVID_GBL_INFO) then passes that number back when starting the encoder. Unfortunately xvidcore doesn't actually do a real check (I guess that's what Koepi's working on). But it's possible to add an override to the vfw interface.

Might be an idea to decide on a maximum number of threads, at the moment I'm doing malloc (and subsequent free) calls to allocate memory based on the number of threads and that's sloooow.

[squid_80]

no problem with celtic duid build ...
Celtic_druid's dual-threaded build shows exactly the same problem. (Do people with real dual-cores see this problem? Probably not if they've got a core for each thread...)

[Koepi]

Doom9: I think for the majority of users this would suffice...

@all: sysKin seems to be back and has new code handy soon. I think that I'll try Doom9's suggestion first though and extend the API temporarily. That will most certainly break compatibility with some apps -- but then, it's a developer, unstable, alpha build :)

Cheers
Koepi

[squid_80]

I think the only thing really missing is a line like pEnc->mbParam.num_threads = create->num_threads; somewhere in enc_create (from encoder.c). I think the xvid_enc_create_t struct has had num_threads for a while so any app should know not to leave it uninitialized. Enabling the threads option in the vfw works once the SMP functions are changed slightly.

[Koepi]

squid_80: you're right, the parameter is there. I must have been sleeping. Ok, that'll be a simple addition :)

Cheers
Koepi

[mod]

Happy to hear that :)

[sysKin]

There's a new code coming soon, which will definitely have user-settable number of threads. It will also use a new approach in which encoding thread (MC, quantization, bitstream writing) is done in parallel with ME, and you can only specify the number of ME threads ("extra" threads) and that should be zero for uniprocessor systems.

In fact, uniprocessor systems will most likely get best results with non-SMP build, simple as that.

I haven't got a solution for the Sleep(0) issue. WaitForObject() is definitely not it, if you read Avery's blog carefully it's a solution for a completely different issue. Anyway, with new code, *lowering* thread priority should work fine while *rising* it should cause a temporary freeze - all that on uniprocessor systems with non-zero number of ME threads.

I'll probably just ignore the Sleep(0) thingy because it does exactly what I want, with the possibly temporary freeze if under certain conditions. Doesn't sound very bad.

I'll release new improved SMP code soon enough.

[JustChecking]

I'll release new improved SMP code soon enough.
Any ~ ETA for it? *waiting* *waiting* :)

[mod]

While we wait.. I insert a little question inside the thread:

I still have bitrate control problems during capture, I'm the only one?
I know that a few people directly use mpeg4 (I use it only a few times too), but a feedback would be nice.
The bitrate is about 9x the set one (set 850 Kbps got 950KBps).
The issue only interest XviD encodings, and seems that it doesn't depend by capture settings.

[Koepi]

Are you sure you took the _sound bitrate_ right with into your calculation?

[mod]

The bitrate I reported is only for the video component: reopening the file with VDub (or other info progs) the video bitrate is ALWAYS > 7500 Kps.
During capture, the value starts from about 400 KBps and quickly "grows" till about 900 KBps.

EDIT: I made again some other captures with VDubMod 1.5.10.2, and XviD works fine.
With VDub 1.6.12 it seems that the issue is NOT only with XviD, but also with DivX. I tried reinstalling and loading defaults, but it's the same. If I won't get it solved I'll wait before opening another thread in the VDub area, VDub 1.6.12 is still marked as "experimental" :)

BTW, I won't post here more about this (unless asked), I didn't start this thread for this issue. :)

[squid_80]

While we wait.. I insert a little question inside the thread:

I still have bitrate control problems during capture, I'm the only one?
I know that a few people directly use mpeg4 (I use it only a few times too), but a feedback would be nice.
The bitrate is about 9x the set one (set 850 Kbps got 950KBps).
The issue only interest XviD encodings, and seems that it doesn't depend by capture settings.
I think I found your answer: http://forums.virtualdub.org/index.php?act=ST&f=15&t=11643&st=0&#entry47303
Video codecs that used bitrate regulation required 10x the bitrate because they received the wrong frame rate value. (regression from 1.6.11)

EDIT: Note that this is only in capture mode.

[mod]

Thanks squid_80, this solves all my doubts :)