Encoding XviD in Virtualdub is so SLOW!

[mbeatle]

I'm not sure exactly why things encode so slowly on my machine. I'm capturing TV @ 640x480 using HuffyUV, then editing and encoding 2 passes into Xvid with Virtualdub. I get 4-5 fps out of my comp and I can't help but think it should be much higher. Compression is set to fast, motion search precision is set to 6 - ultra high, and VHQ mode is set to 4 - wide search. Speeds like this remind me of my Celeron 450 encoding movies into VCD.

I'm not applying any filters, btw.

[JarrettH]

Set VHQ to 1. I saw a thread a while back and it apparently is no benefit using the higher settings. VHQ1 offers best quality and size.
Sorry if I'm misunderstanding, but isn't "Fast" an option in DivX?

[mbeatle]

Not sure about DivX, I'm using XviD...

I'll try with a VHQ of 1 and see if it makes a difference.

[Taurus]

Originally posted by mbeatle
I'm not sure exactly why things encode so slowly on my machine. I'm capturing TV @ 640x480 using HuffyUV, then editing and encoding 2 passes into Xvid with Virtualdub. I get 4-5 fps out of my comp and I can't help but think it should be much higher. Compression is set to fast, motion search precision is set to 6 - ultra high, and VHQ mode is set to 4 - wide search. Speeds like this remind me of my Celeron 450 encoding movies into VCD.

I'm not applying any filters, btw.

Some more info needed.
How fast is your machine?
What kind of editing?
You wrote that you dont use any filters!?!
Fast recompress on?
Avisynth for frameserving?
Which XVID settings?

Not enough info to help out.
Post a logfile.

Cheers

Taurus

P.S.: I get almost 25 frames with Huffy@640x480 with xvid slightly tuned settings on an AthlonXP 2600+

[Mug Funky]

man, there's something WRONG with your setup...

i'm on a p3-733 and i get higher framerates for 640x480 WITH heavy avisynthing.

do you have GMC turned on? i always leave it off because sod-all mpeg-4 standalones support 3 warp points ("divx" certification has a LOT to answer for as far as mpeg-4 compliance goes). apparently it slows things down a lot.

[mbeatle]

Sorry, I copied this post from another forum where I have my rig posted in my signature. Here it is (http://www.mdbbox.com/P4C.txt).

No, I do not apply any filters. I was put off by the speed so much that I didn't want to make it any worse. I'm not using AviSynth or a frameserver, just Virtualdub.

Basically my editing consists of a TV show where I select the sections of program and leave the commercials out. I then use Virtualdub to merge the 4 avi files that I get.

I'm not sure what GMC is. Where is the option for that?

I'll look for a logfile when I get home.

[mbeatle]

Here's the log I got from encoding one segment using 2 passes:

[i] Starting up: VirtualDub build 18160/release

AVI: Opening file "E:\PVR\goodeats0903-1859(S-Video).avi"

[i] AVI: OpenDML hierarchical index detected on stream 0.

[i] AVI: OpenDML hierarchical index detected on stream 1.

AVI: Opening file "E:\PVR\goodeats0903-1859(S-Video).avi"

[i] AVI: OpenDML hierarchical index detected on stream 0.

[i] AVI: OpenDML hierarchical index detected on stream 1.

Beginning dub operation.

Ending operation.

AVI: Opening file "E:\PVR\goodeats0903-1859(S-Video).avi"

[i] AVI: OpenDML hierarchical index detected on stream 0.

[i] AVI: OpenDML hierarchical index detected on stream 1.

Beginning dub operation.

Ending operation.

[Didée]

1. Be sure to set Vdub to "fast recompress".

2. In XviD's config window, hit the "default" button.

3. Start the encoding

4. Then report back what speed you're getting ...

[PlazzTT]

Try opening the task manager while it's encoding to make sure that VirtualDub is using most of the CPU.

[mbeatle]

Setting everything to default (VHQ mode went back to 0) I got 22-28 fps, depending on motion. With VHQ back @ 4, I'm clocking 6-7 fps. That's an improvement, but shouldn't I expect better?

CPU usage is pegged @ 50-55% (this cpu has hyperthreading) which means virtualdub is not using more than one thread, at least not very efficiently.

[gldblade]

VHQ 4 isn't really worth it if you're hoping for speed. It doesn't bring too much additional quality either. Most people just stick to VHQ 1.

[Manao]

It's not vdub's fault, it's rather the codec itself that doesn't support hyperthreading. With only 50 % of your processing power used, your encoding fps are what you should expect them to be. Disable hyperthreading if you want to double these numbers.

[/\o/\]

i think you shouldn't use overclockled system

[Didée]

Originally posted by mbeatle
Setting everything to default (VHQ mode went back to 0)
Now, what's that. VHQ default should be 1, not 0. At least for the 1.x builds I have tried.

What is reported (build version and date) if you click "about" instead of "configure" in Virtualdub?

[mbeatle]

Build 18160, Dec 2, 2003.

[nexus]

You should _really_ use a newer build (v1.0.2)

[Mug Funky]

also, if you're hyperthreading, you could save time by doing 2 encodes at once... just set 2 encodes going in separate VD windows, and then set their affinities to the different fake CPUs.

even at 50% of you CPU you should be getting some serious speed. definitely use one of the latest builds... Xvid has moved on greatly since 2003 (wish i could say the same for divx, nudge nudge)

[Dirk99]

I´m coding my first movie with GKnot 0.28.8 and the Xvid 1.0 Codec on my 2,8GHz hyperthreading System.

It is interessting, that the CPU usage is @ ~100% on the 1-pass and only @ ~65% on the 2-pass.
Does anybody know why the CPU usage is so different between the two passes?

BR,
Dirk

[Manao]

At the first pass, I guess that the avisynth script is as fast as XviD. Since it runs in another thread, it uses all the remaining processing power.

At the second pass, XviD is by far slower than avisynth, hence avisynth doesn't half the processing power to do all its computation, hence you still have 35 % of processing power left.

[Dirk99]

Perhaps a naively question: Why doesn´t XviD take the unused 35% to do its work faster?

Edit:
The Credits of the Movie were encodet with ~95% at the second pass...

[lordadmira]

Disable hyperthreading for Virtual Dub in Task Manager. Then see what kind of speed u get. There is a tool out there that automatically sets the HT setting for an app but I can't remember the name of it.

[moon1234]

You do this by going to task manager, click processes tab, right click on the process and then select set affinity. But if you need to disable hyperthreading you have to do it in the BIOS of your computer. Task manager will only show the virtualized processors and you can choose to disable the application from running on several of them.

The problem is that an application needs to be hyperthreading aware or it will only use one of the virrtualized processors. This means that only 50% of your CPU speed will be utilized becuase it sees the two virtualized processors as two independent ones.

Does that make sense?

[Dirk99]

That sounds logical.

I will try a little bit and post the results here.

Thanks a lot!

[lordadmira]

Originally posted by moon1234
...Does that make sense? Sort of. The problem isn't so much HT as it is Windows's stupidity in using it. Theoretically it should be indistinguishable from a normal multi processor system. The more total threads u have on ur system the more useful it is. It allows 2 threads to execute simultaneously on 1 CPU. In Task Manager the 2nd CPU represents the virtual CPU and the 1st is the physical. Disabling affinity for the virtual CPU is the only thing u can really do. If that doesn't work then disable HT totally in BIOS. If u ask me the whole idea of HT is completely stupid.

LA

[Leak]

Originally posted by lordadmira
If u ask me the whole idea of HT is completely stupid.

Well, Hyperthreading is the difference between UT2k4 making Foobar2000 skip or not when running both in parallel, so it's hardly "completely stupid" - it really helps if an IRQ handler takes it's time and another, different IRQ has to be handled in the meantime. Also, a single process running amok doesn't stall the whole system.

np: Reinhard Voigt - Track 2 (Premiere World)

[Dirk99]

I´ve made a little test with the beginning of a 2h 11min film. The beginning mean, I encoded only the first 12`000 frames of the film, because I want a fast result.


______________HT on______HT off (only task man.)___HT off (Bios)
12k frames____11m 30s____11m 1s____________________11m 8s
CPU usage_____65%(both)__50%(1st: 100%/2nd: 0%)____100%
Temperature___58°C_______58°C______________________57°C
Temp. Idle____44°C_______44°C______________________53°C


This result is very interessting, I think. The HT technology doesn´t effect the encoding time very much. The best should be to disable it in the task manager.
The most interessting result on my system is, that a CPU usage of 50% when HT is disabled in the task manager results in the same time as a CPU usage of 100% when HT is disabled in the bios. I don´t understand that.
Another interessting point are the temperatures in my system. A idle system with disabled HT technology is warmer than the configuration where HT is enabled. The second thing I could not explain.

I hope someone else can reproduce these results. Explanations for my results are interessting for me, too.

Thank you,
Dirk

P.S.: Sorry about my english. I think it isn´t very good. :(

[Leak]

Originally posted by Dirk99
The most interessting result on my system is, that a CPU usage of 50% when HT is disabled in the task manager results in the same time as a CPU usage of 100% when HT is disabled in the bios. I don´t understand that.

Well, Windows treats a hyperthreading CPU the same way it treats a dual-processor machine - it keeps track of the usage of each CPU and reports the average use of all CPUs.

Since nailing the encoding process to one CPU makes one "CPU" use 100% and the other 0% you'll get an average of 50%. Without setting the processor affinity the encoding process is bounced between both "CPUs" and what little parallelism is there in the encoding process is taken advantage of, so you'll get a result greater than 50%, but the extra overhead needed to handle that parallelism (since you don't have 2 full CPUs, but 2 "wannabe" CPUs that share quite a bit of hardware) makes everything take a bit longer than without it.

In the end, with hyperthreading the system is more responsive even if one CPU has 100% load, so I normally don't change the processor affinity.

np: Jürgen Paape - Triumph (Kompakt Total 1)

[lordadmira]

Those are interesting results. The temperature thing. The temp is directly related to the number of transistors in use on the chip. Even when the processor is doing "nothing" every circuit on there is being pumped with current. So the temp difference on idle would seem to indicate that the HT is using the chip hardware resources more efficiently than normal. "More idle than idle." The only reason I can think of that ur #2 went the fastest is that setting the affinity eliminated any thread switching between virtual CPU's while at the same time keeping the HT benefits for all other processes. So I think it would be beneficial to always disable HT affinity for video encodes. Atleast until HT aware video apps come out.

LA

[lordadmira]

Originally posted by Dirk99
Perhaps a naively question: Why doesn´t XviD take the unused 35% to do its work faster? Because that extra 35% isn't "really there". Actually the whole second 50% isn't really there. :p It's CPU power that doesn't exist independantly but can "ride the coattails" of the first 50% of power. That's what the HT does, it lets u put 2 people in 1 roller coaster car. And obviously, u can't put the same person in the same car twice.

LA

[ChronoCross]

Here are some things I do before i start encoding.

1) set the swap file size to Minimum the size of your ram and maximum to 2x your ram. I have seen 2fps speedup because of more efficient usage of memory

2) shutdown all non esseitial programs. I recommend searching your machine for virus's spyware, adware, etc.

3) reboot and freshly load all my encoding software.

4) press the start button. with intense avs filtering and tweak xvid settings I can usually push 4-7fps out of my 1.1ghz P3.

Does xvid plan on making itself HT compliant in the future? specail supposrt for dual processors?

[Dirk99]

I also think the most interessting result is the temperature thing.
And with the explantation of the HT-Technologie I unterstand now a little bit more what my PC does with this feature.
There are two virtual CPU on one physical CPU, not two physical CPU´s in my computer. Therefore one of the virtual CPU's can also use 100% of the physical CPU and Windows thinks that the system is balanced only to 50% usage.

Thx you!

[-IRnoob-]

I've had the same problem with my 3.2GHz HT capable P4. My advice is to disable HT in the BIOS. You should see big gains after that.

[Ramromulus]

I have a P4 3.0C and i found with HT enabled my encoding time went faster. With it disabled I lost 2fps and the system became consderibly slower.

[Dirk99]

The test I´ve done gives the hint to enable HT in the Bios and to disable it in the task manager for VirtualDub.
But it is a result on only one computer with a P4 2,8E GHz 800MHz FSB and 1 MB Cache...