xvid and multi-core cpu?

[molngab]

Hello!

If I use the xvid codec to transcode a dvd rip to xvid avi format (with a well-known converter, eg. Autogk or other)-so, the xvid codec can fully support a modern, multi-core CPU? x4, X6 amd, core ix series 4 core etc?
My experience: I have a new, x6 amd cpu-six core. And the transcoding speed isn't so fast. a short time ago I used an x2 two core cpu, and now the transcoding isn't significantly faster. Fast, fast, but not too fast.


Thanks

[nurbs]

Xvid doesn't scale well on multicore CPUs. Your best bet is to do several encodes concurrently.

[Sharktooth]

or to trim your source in several parts and encode them concurrently

[henryho_hk]

Even for 2-pass encodes, you can use a compressibility test to estimate the bitrates required for each parts (not necessarily identical in length). It works fairly well, except that the part with the most number of bits (bitrate * length) will usually take the longest encoding time.

[yetanotherid]

My experience: I have a new, x6 amd cpu-six core. And the transcoding speed isn't so fast. a short time ago I used an x2 two core cpu, and now the transcoding isn't significantly faster. Fast, fast, but not too fast.

Same here. My dual core PC is only marginally slower encoding to xvid than my quad core. When using the x264 encoder though, the quad core is twice as fast.

I think it's just something you have to live with.

[billa]

yea what about those i7 :)

[IgorC]

New Xvid 1.3 RC1 has a new multicore support. It's slice based but the loss in quality is nothing (<0.1%)
Now Xvid scales good with high number of cores.
http://list.xvid.org/pipermail/xvid-devel/2010-December/006232.html

For quad core:
Threads Row-based MT (FPS) Slice-based MT (FPS)

-1 21.0 21.0
1 20.9 20.9
2 29.7 35.6
3 35.3 48.2
4 38.3 56.0
6 34.8 49.0
8 34.0 54.7

Download http://www.free-codecs.com/download/xvid_codec.htm

[Didée]

That download actually is the "Suppose me to crash" build by Jawor, isn't it?

Edit - (immediate) crashing with VfW codec seems to occur when: qpel AND (VHQ>1)

[henryho_hk]

And that compile (SSE2, SSE3 and SSSE3 versions) is producing corrupted outputs with VAQ.

[Jawor]

(immediate) crashing with VfW codec seems to occur when: qpel AND (VHQ>1)
Damn GCC 4.x and its optimizations... damn them all to hell...

Thank you, I'm already looking into this.
And that compile (SSE2, SSE3 and SSSE3 versions) is producing corrupted outputs with VAQ.

Even a vanilla 1.3.0 RC1 build (made with MSVC 2008, no changes to any compile settings) has this problem. Old xvid_encraw (especially as old as Squid's 31.08.2007) is incompatible with 1.3.0 RC1. I haven't noticed such problems when encoding through the VfW interface.

<EDIT>
I reproduced the problem with Qpel + VHQ. s/-O3/-O2/g fixes the crash. New builds uploaded.

Thanks again for reporting this.

Another lesson learned: “Trust GCC even less than before.”
</EDIT>

[Didée]

Did several test runs with several feature combinations -- no crashes anymore, and the results don't show any anomalies (by just eyeballing).

Apparently, the build is good now. (Speaking of VfW. I didn't test encraw.)
Thanks for the fix!

[Jawor]

Apparently, the build is good now. (Speaking of VfW. I didn't test encraw.)
encraw doesn't crash either with the latest build. The problem was in the old xvidcore.dll (both encraw and VfW were crashing with it).
Thanks for the fix!
No problem.

[halpern]

There is a thread setting in MeGUI in the Xvid configuration section. It is currently set to 1 on my machine. I am running an AMD 64 X2 (Dual Core) pc. Should I change this to 2 to allow multi-core processing?

[Jawor]

Yes. This value should be equal to the number of cores.

[halpern]

Will I need to use Xvid 1.3 RC1 to get multithread/core support or can I use the version that comes standard with MeGUI?

[Jawor]

Versions 1.2.x also support multithreading.

[halpern]

Changing that thread option did not increase the fps for me. The only way for me to increase the fps was to use the MT plugin, and it only doubles the fps when I'm using the Yadif plugin. All my sources need to be deinterlaced.

[LoRd_MuldeR]

Changing that thread option did not increase the fps for me. The only way for me to increase the fps was to use the MT plugin

Which means that the bottleneck which is limiting your throughput isn't the Xvid encoder, but the pre-processing in Avisynth.

(Actually that's kind of surprising, as Yadif is pretty damn fast and Xvid's multi-threading is known to not scale very well)

[Jawor]

Xvid's multithreading is less efficient than (for example) x264's (unless we use slice-based multithreading, but that would break standalone player compatibility). AviSynth filtering can also become a bottleneck, so using multithreading in AviSynth is certainly a good idea if we want to utilize a multi-core CPU better.

[halpern]

I'm getting ~23fps on my AMD X2 64 6000+ dual core CPU when using Yadif and the SetMTMode from the MT plugin. When using TIVTC, its still like 12fps which I think is horrible. Is there anything else I can do to help speed up the process? I apologize but I'm just trying to get my PC to perform as well as possible. Thanks.

[henryho_hk]

so using multithreading in AviSynth is certainly a good idea if we want to utilize a multi-core CPU better.

I am still using brute-force multi-encode-processes method. :devil:

[HarryM]

I am using Avidemux with its XviD plug-in 1.2. Multi-threading is good, but it could be better. Only audiostream compression is independently threaded. I think that if each running process (in, preprocessing, coding, audio, mux, out...) in a pure independent thread, than would be to get 100% cpu usage for quad-core easily.

[LoRd_MuldeR]

I am using Avidemux with its XviD plug-in 1.2. Multi-threading is good, but it could be better. Only audiostream compression is independently threaded. I think that if each running process (in, preprocessing, coding, audio, mux, out...) in a pure independent thread, than would be to get 100% cpu usage for quad-core easily.

You are wrong about that. Usually you will be bottlenecked by the video encoder. Or by the pre-processing filters, if you are using slow filters. Decoding the input can only become a bottleneck when using an extremely fast filters+encoder combo. In any case: No matter where your bottleneck is, you will only be able to improve the throughput by multi-threading in that part of the tool-chain which currently is your bottleneck! For example making the decoder multi-threaded won't improve the throughput at all, as long as the decoder is idle (waiting for encoder to process the decoded frames) most of the time anyway. So unless you use very slow filters, you are most likely bottlenecked by the Xvid encoder. And thus the one and only way to further improve the overall throughput from there would be improving Xvid's throughput. Unfortunately it's a well known fact that Xvid's multi-threading isn't very efficient and doesn't scale very well...

[Tuik]

Xvid's multithreading is less efficient than (for example) x264's (unless we use slice-based multithreading, but that would break standalone player compatibility). AviSynth filtering can also become a bottleneck, so using multithreading in AviSynth is certainly a good idea if we want to utilize a multi-core CPU better.

What multithreading method do you suggest with a simple script like:


SetMemoryMax(1024)
DSS2()
Spline16Rezise()
Sharpen()


Would be right this way, or is there a better way?

SetMemoryMax(1024)
SetMTMode(5)
DSS2()
SetMTMode(2)
Spline16Rezise()
Sharpen()
Distributor()


The system is an i7 950 (8 threads) 8GB ram

[Blue_MiSfit]

That's about the best possible case. You may have issues - AviSynth MT is kind of hit or miss unless everything is dialed in perfectly.

SetMTMode is the only way to do MT'd resizing AFAIK, so it's basically your only option in this case :)

If AviSynth is bottlenecking you, use a faster resizer like bicubic or bilinear.

Derek

[Tuik]

I dunno if the Distributor() in the end is needed or not...

[TSoft]

Hi,
I often use MeGUI to convert 720p mkv to avi. Currently MeGUI is using xvid_encraw 1.3.0 by Jawor (I think lol), I have an i7 950 and I can't make use all four cores (or 8 virtual). I tried the filter MT (MT.dll) as described in this forum, but xvid_encraw crashes.
My question is, is there another way to use all 8 cores while encoding from mkv to avi ?

Sorry for my english