xvid encoding too slow in linux

[lam_das]

This is not to start a flame war. I have a genuine problem here, either with my setup or with the code itself.

I used a 23 sec mpeg2 sample to encode into xvid using ffmpeg and resultant fps was about 22 in pass 1, and 19 in pass 2.

I then ran virtualdub under wine and encoded the same sample with identical (well almost, because I couldn't find an equivalent in ffmpeg for chroma_me) xvid settings and got about 42fps in pass 1 and about 36fps in pass 2. The resultant avi looks way better (much less blocky and more clear) than the ffmpeg and was smaller than ffmpeg ((vdb)1672 kbps vs. (ffmpeg)1760 kbps shown by mplayer for a target of 1800kbps).

vdb used 99% cpu in pass2 and 70% in pass1. ffmpeg used 52% in both passes.

So, what gives people? Is it known that vdb is faster than ffmpeg by a factor of 2?

I ran the same sample with mencoder. I got identical looking, if not better looking, avi from it but the pass2 fps was about 8.6 and pass1 was about 17.6. That is almost 1/4th performance.

And native apps are 64-bit while vdb is 32-bit running under wine.

Some config:

gentoo linux: amd64, mplayer 1.0_rc1-r1, xvid 1.1.2, ffmpeg svn
wine: 0.9.29, vdb 1.6.17, koepi xvid 1.2.-127.
h/w: amd 3800 X2, 2gb ram.

xvid settings taken from doom9 and translated exactly one-by-one under linux. mencoder exact match with vdb. ffmpeg missing chroma_me, which should slow it down further.

[nm]

You are using XviD 1.2 (the CVS version) with VirtualDub and XviD 1.1.2 with ffmpeg/MEncoder. Only 1.2 is multi-threaded and can therefore use both cores of your CPU. If you get XviD 1.2 running with MEncoder, and it's still not faster, play with the threads parameter.

The significantly worse quality of ffmpeg is probably due to the parameters used. Post your ffmpeg command-line so that people can take a look at it (I haven't used ffmpeg that much, so I don't know what to try).

[lam_das]

You are using XviD 1.2 (the CVS version) with VirtualDub and XviD 1.1.2 with ffmpeg/MEncoder. Only 1.2 is multi-threaded and can therefore use both cores of your CPU. If you get XviD 1.2 running with MEncoder, and it's still not faster, play with the threads parameter.
darn! I thought 1.1.2 had the threads support. Do you think it can cover that much ground (9fps to 36fps for pass2) with threads support though? I am not sure. Only building the cvs version will tell....I will post back the results.

The significantly worse quality of ffmpeg is probably due to the parameters used. Post your ffmpeg command-line so that people can take a look at it (I haven't used ffmpeg that much, so I don't know what to try).
I will post the parameters once I am home, but I tried to match them as closely as possible.

PS: do you know if xvid 1.2 breaks ABI?

[lam_das]

I did a quick test here. xvid 1.1.2 vs xvid 1.2.0-127 thru mencoder. The threads=3 gives the best performance on a dual core. But it betters 1.1.2 stable release by only 5% in speed of the second pass, which doesn't even scratch the surface for 9 to 36 jump. There is something more than the threaded version involved here. I will post the details on the actual cmdline args later.

[lam_das]

I upgraded xvid to cvs and I got mencoder fps jump from 17.6 to 66 fps for pass1 and from 8.6 to 45fps in pass2. No threading involved. Threading is just 5-10% improvment. That beats vdb by a good measure.

I never thought it was possible to quadruple the perf from 1.1.2 to 1.2.0-dev.

I have another question: why is ffmpeg generated xvid file 24bpp while mencoder generated one is 12bpp? What is the difference between the two?

[nm]

I have another question: why is ffmpeg generated xvid file 24bpp while mencoder generated one is 12bpp? What is the difference between the two?
It has probably got something to do with the colorspace conversions applied to the source, or other preprocessing. 24bpp means it's encoded in RGB (or less likely some large YUV colorspace) and 12bpp means YV12, the latter being what you usually want. That's actually what I was supposed to say earlier: check the other ffmpeg parameters, not just XviD encoding parameters. But since you got MEncoder working, there's probably no reason to tweak ffmpeg...

I upgraded xvid to cvs and I got mencoder fps jump from 17.6 to 66 fps for pass1 and from 8.6 to 45fps in pass2. No threading involved. Threading is just 5-10% improvment.
I think that in MEncoder the default threads=0 is actually automatic thread setting that tries to guess the optimal number. If MEncoder uses > 50% of the dual core CPU, it is threaded. By setting threads=1, you could force it to a single thread and probably halve the speed.

[lam_das]

I think that in MEncoder the default threads=0 is actually automatic thread setting that tries to guess the optimal number. If MEncoder uses > 50% of the dual core CPU, it is threaded. By setting threads=1, you could force it to a single thread and probably halve the speed.
When I compiled the cvs version, the first thing I did was my experiment with without threads, and threads=1,2,3,4,8 just for fun, and repeated same encode 3 times (23secs sample, so an encode would end in a minute or less) to get a good average. The maximum difference was between 1 and 3, and was about 6%. Although, now I use threads=2 in my scripts just to make sure I can run two encodes in parallel in a nightly batch job.

only legible explanation for this boost is that gentoo linux sets CFLAGS before compiling xvid and these differ from the manual install from cvs tarball. e.g. from xvid build directory


$ make list-cflags

---------------------------------------------------------------
Using CFLAGS
---------------------------------------------------------------

CFLAGS=-Wall -O2 -fstrength-reduce -finline-functions -ffast-math -fomit-frame-pointer

But my CFLAGS in /etc/make.conf don't include "-finline-functions -ffast-math" because of the potential breakage they can cause. -fstrength-reduce is part of -O2. So, it looks like -ffast-math is what did the trick and may be use of pthread library added to it, to give it a huge boost. I can verify it by compiling 1.1.2 with -ffast-math but I am rejoicing the speed at the moment and don't wanna meddle with it anymore.

[rvm]

It has probably got something to do with the colorspace conversions applied to the source, or other preprocessing. 24bpp means it's encoded in RGB (or less likely some large YUV colorspace) and 12bpp means YV12, the latter being what you usually want. That's actually what I was supposed to say earlier: check the other ffmpeg parameters, not just XviD encoding parameters. But since you got MEncoder working, there's probably no reason to tweak ffmpeg...

According to mencoder-users list, the bpp field is only used for uncompressed video. In xvid doesn't mean anything.

http://archives.free.net.ph/message/20061125.065441.c94e51df.en.html

[lam_das]

thanks. That clears up the confusion about whether mencoder is doing something wrong compared to ffmpeg.