Xvid encoding low CPU usage question

[madhatter300871]

Hi

Encoding with xvid 1.3.2, using xvid_encraw, does not utilise 100% cpu usage.

On my quad core machine, it uses about 25% in total at about 5fps, on my dual core machine it uses about 50-60% total usage at about 8fps.

In contrast, encoding using x264 always uses 100% cpu usage and encodes 1080p at about 30fps.

The source is the same source, using an avisynth script that delivers a directshowsource, no other filters involved, no audio involved.

Any ideas as to where the bottle neck is thats causing low cpu usage ?

[hello_hello]

As far as I know it's XviD. It doesn't multi thread very well, or something like that. Mind you, your speed is actually very slow now I think about it properly (depending on the resolution). Encoding DVDs I think I get about 80 to 90% CPU usage or more using my dual core PC and about 40 to 50% using the quad core. I can't tell you what that translates to in FPS off the top of my head but I did just finish encoding a 720p source to a SD AVI which tends to be a lot slower than encoding DVDs, and using the dual core I got 25fps for the first pass and 45fps for the second.

I'm using AutoGK to encode and I'm not using xvid_encraw so I don't know where your bottleneck would be. Have you tried a standard XivD version? Which version of AVISynth are you running?

[jsquare]

I was also concerned since I'm getting 70-80% usage on my AMD X6 at around 50fps and x264 uses 100% of the CPU, but I mainly use XviD for DVD or SD sources. Don't know why but my DVD/SD encodes using x264 look terrible compared to XviD.

[hello_hello]

I was also concerned since I'm getting 70-80% usage on my AMD X6 at around 50fps and x264 uses 100% of the CPU, but I mainly use XviD for DVD or SD sources. Don't know why but my DVD/SD encodes using x264 look terrible compared to XviD.

Are you using a GUI for your Xvid and x264 encodes? If so, which ones? All else being equal (same resizing and filters etc) x264 encodes would probably retain a little more detail, but it also depends on each encoder's quality setting and the target file sizes used etc.
At worst, and all else being equal, the two encodes should look fairly identical. I don't know what's going wrong with your x264 encodes but it's not "normal".

[jsquare]

Are you using a GUI for your Xvid and x264 encodes? If so, which ones? All else being equal (same resizing and filters etc) x264 encodes would probably retain a little more detail, but it also depends on each encoder's quality setting and the target file sizes used etc.
At worst, and all else being equal, the two encodes should look fairly identical. I don't know what's going wrong with your x264 encodes but it's not "normal".

Using StaxRip, no filters just crop and resize, also I do most of my XviD encodes in 1-pass Q=2.5.
Just tried my Xvid Q2.5 and x264 CRF20 encodes from same source on my Samsung Plasma and the x264 looks better there, on my PC is where it looks bad, maybe MPC-HC is not doing a good job.

[Jawor]

As far as I know it's XviD. It doesn't multi thread very well, or something like that.
Yes, Xvid's multithreading isn't as efficient as (for example) x264's. The Independent Slice Coding option uses a better multithreading algorithm, but it breaks compatibility with standalone players (that's why it is allowed only in the “unrestricted” profile).

Besides, my builds of version 1.3.2 from 01.06.2011 and 02.06.2011 were broken in that regard (multithreading was disabled in them). Builds from 17.06.2011 and 24.06.2011 have fixed this, so please update if you have 01.06.2011 or 02.06.2011 installed.

[madhatter300871]

I am using :-

avisynth 2.5.8
xvid_encraw
xvid 1.3.2, but not sure from which date so I will check.

Tonight I will make sure to download one of the latest builds from Jawor with multithreading enabled.

What I have done is install xvid 1.3.2, then copy xvid_encraw from "program files\xvid" and xvidcore.dll from "windows\system32" into a folder of my own. I am presuming the xvidcore.dll in windows\system32 will be the latest version as installed by the xvid 1.3.2 installer and I am also presuming that when I run xvid_encraw it will look for xvidcore.dll in its own folder ? Am I right ?

((Jawor ..... on a side note ..... do you think you might be able to enable mkv output support in a build of xvid_encraw ? I use this feature in my own gui, can't for the life of me find the build I had with mkv enabled !!))

Is it relevant wether a GUI or xvid_encraw is used to encode ? Do some GUIs offer speed increases ?

[Jawor]

I am presuming the xvidcore.dll in windows\system32 will be the latest version as installed by the xvid 1.3.2 installer and I am also presuming that when I run xvid_encraw it will look for xvidcore.dll in its own folder ? Am I right ?
Yes.

((Jawor ..... on a side note ..... do you think you might be able to enable mkv output support in a build of xvid_encraw ? I use this feature in my own gui, can't for the life of me find the build I had with mkv enabled !!))
Building xvid_encraw with Matroska output requires a file named matroska.cpp which was AFAIK never really finished for the Xvid CVS/SVN (see here (http://forum.doom9.org/showthread.php?p=1007729#post1007729)).

Only squid_80's xvid_encraw (which was rewritten almost from scratch) can use Matroska output. Unfortunately this version is almost 4 years old and doesn't work well with xvidcore.dll versions 1.3.x. Apparently nobody knows how to fix squid_80's source code...

But why use Matroska for MPEG-4 ASP? ;) Standalone players always perferred the AVI container for ASP (some also support ASP in MP4). And if we don't care about old standalone DVD players, we may as well use x264 in MKV (it is supported by many Blu-Ray players). Encoding with x264 doesn't have to be slower - even “fast” settings can provide better quality than Xvid.
Is it relevant wether a GUI or xvid_encraw is used to encode ? Do some GUIs offer speed increases ?
Calling xvid_encraw from a GUI (instead of CLI) shouldn't make any difference.

[madhatter300871]

Jawor

Ahhhhh yes it was squid_80's version that I used, comes back to me now. The only reason I used mkv output from xvid_encraw was that my own gui needed it. It's no big deal, I'll re-write it to use raw output instead. And your right, I have no idea why I used mkv output, I think I just did because I did !

I mainly encode to x264 in a mkv container but I am having difficulty with one particular standalone playing back my x264 encodes ... it plays my xvid encodes perfectly. It does support mkv playback, it does support avi playback and I have seen it playback my x264 video but with a lot of dropped frames. The easy solution was to go back to xvid, which I don't mind doing.

Does xvid_encraw output open dml avi files ?

[Jawor]

Does xvid_encraw output open dml avi files ?
No. The standard xvid_encraw (the one from Xvid.org's CVS/SVN) writes AVI files using Microsoft's old vfw32.lib (AVIFile) interface which doesn't support OpenDML.

On the other hand, squid_80's “advanced” xvid_encraw can write OpenDML files.

[henryho_hk]

The "Independent Slice Coding" feature shines with VirtualDub, reaching about 80% in my quad-core Q6600. If squid80's threaded input feature can be added to xvid_encraw again, I think we can expect similar performance.

As for vfw32.lib, two worse problems are that it produces invalid AVI file (rejected by VD) when over 2GB and it actually wraps around when over 4GB.

[Jawor]

As for vfw32.lib, two worse problems are that it produces invalid AVI file (rejected by VD) when over 2GB and it actually wraps around when over 4GB.
It's the old “signed/unsigned 32-bit int” dilemma ;) If a value is described as an integer in the specs, some people will inevitably interpret it as a “signed int” and others as an “unsigned int”. And even if the specs will explicitly say whether it's signed or unsigned, some people would probably tend to simply ignore them or re-interpret them... The same issue plagues the ISO9660 filesystem.

[madhatter300871]

So ... I updated to XviD from 24.6.2011 and CPU usage is higher, I am getting between 80%-85% now which is much better.

On a side note if I may .... Jawor, when I redirect your console output (in order to just display your console messages in my own front end) I only seem to get output every few hundred frames. I have tried different values for '-progress' and have also tried without using '-progress'. When not re-directed your console outputs messages every frame as expected.

My code is fine, very very simple re-direction, code I use on a regular basis that works perfectly.

Presuming my code is fine (I understand I am just asking you to trust me on this) is there anything in your code that causes this to happen ?

Thanks if you manage to get back to me on this.

[Jawor]

when I redirect your console output (in order to just display your console messages in my own front end) I only seem to get output every few hundred frames
Does the same thing happen with other builds of version 1.3.2 (Koepi, xvidvideo.ru etc.)?

is there anything in your code that causes this to happen ?


Actually, it's not my code ;) I'm only providing (slightly customized) binaries. Except for those small customizations (like additional checkboxes in the VfW GUI, smaller and more flexible installers etc.), the source code comes from Xvid.org (meaning it's written by the Xvid Team with some contributions from freelance outsiders). Some of my previous patches (like additional and lowercase FourCCs) were deemed useful enough to be committed to the official CVS/SVN, so right now the only differences between my builds and vanilla are in the VfW GUI and installers.

[madhatter300871]

Jawor, thanks for getting back to me. I have been so busy at work lately I haven't had time to do the things I like ........

Anyway ......

I'll try other builds, but I think I already have tried and it was the same. I won't contact the XviD team for a response, I would think they will be way to busy to address this and I don't suppose it's overly important, would just be nice if my GUI updated it's XviD output "live" and not only every few minutes.

[henryho_hk]

Would it be stdout buffering or line separator problem?

[madhatter300871]

Henryho_hk ...

I don't know, but all the other command line programs I use, use my exact same code for redirecting stdout and stderr.

How could I check if it is stdout buffering or line separator problem ?

I know this is now totally off topic, but hope we can exchange just 1 or 2 messages on it now that we are chatting ....

Thanks.

[madhatter300871]

A few months on and I fixed the problem.

I have made a change to xvidencraw, I have set it to flush stderr after writing progress to it. Now my redirected stderr is updated on the fly in real time.

The problem did indeed to be that while xvidencraw did write to the console every nth frame (as defined by -progress n) I only got data in my redirected stderr after the stderr buffer was full (I presume). I think stderr has a buffer size of 1024, I am not sure.

Anyway, I included fflush(stderr) after writing to stderr in xvidencraw.c, recompiled and hey presto !

[MilesAhead]

Anyway, I included fflush(stderr) after writing to stderr in xvidencraw.c, recompiled and hey presto !

Any chance of downloading a version with this fix?

[madhatter300871]

Any chance of downloading a version with this fix?

Yes, no problem. I can email it to you if you want. I think you can send me your email address in a private message (not used the private messages before).

[madhatter300871]

I have emailed it to you.

So may I ask, how come you are interested in this also, do you redirect the output for your own purposes ?