x264 on a high-end box

[agressiv]

Not a question, but I figured I'd put MeGUI/x264 on one of our 4-way opteron 2.8ghz Rev F evals we got in. Arguably one of the fastest 4-ways on the market, getting 55 FPS encoding a 2nd pass (HQ-Slow profile, 8 threads) is very impressive :)

And even then, x264 was hovering 35-45% the whole time.

Its not from a manufacturer we purchase from, so I figured I'd at least let it do some work before we sent it back :)

agressiv

[Thunderbolt8]

since I got like 2 frames on anything im encoding with either hq-slower or -crf mode, Ill come over and bring some of my stuff that needs to be done~~

[Taxidermista]

Not a question, but I figured I'd put MeGUI/x264 on one of our 4-way opteron 2.8ghz Rev F evals we got in. Arguably one of the fastest 4-ways on the market, getting 55 FPS encoding a 2nd pass (HQ-Slow profile, 8 threads) is very impressive :)

And even then, x264 was hovering 35-45% the whole time.

Its not from a manufacturer we purchase from, so I figured I'd at least let it do some work before we sent it back :)

agressiv

High definition?

[agressiv]

I only had access to a few DVDs at work, so it was one of those.

I ran it again at HQ-Insane, and still got about 15fps.

agressiv

[ArcticFox]

55fps!!!!!!

Just wow, id be lucky to get 5.

[Kostarum Rex Persia]

agressiv, are you saying that on TWO Opteron Socket F processors, you got ONLY 55 FPS? That's very strange, because you tried MeGUI on normal DVD source, 720x576 PAL, right?

CPU usage were only 35-45 %? That's the reason why you got only 55 FPS.

[woah!]

55fps is pretty good for the "2nd" pass, my E6600 at 3200mhz does 100fps on the first pass and 40fps on the 2nd pass with the same profile used:


Log for job job1-1

avis [info]: 448x336 @ 29.97 fps (40947 frames)
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
x264 [info]: slice I:167 Avg QP:22.31 size: 16996
x264 [info]: slice P:16664 Avg QP:24.39 size: 6278
x264 [info]: slice B:24116 Avg QP:25.99 size: 2550
x264 [info]: mb I I16..4: 31.4% 0.0% 68.6%
x264 [info]: mb P I16..4: 13.9% 0.0% 0.0% P16..4: 58.6% 0.0% 0.0% 0.0% 0.0% skip:27.4%
x264 [info]: mb B I16..4: 1.1% 0.0% 0.0% B16..8: 26.1% 0.0% 0.0% direct:14.9% skip:57.9%
x264 [info]: direct mvs spatial:92.4% temporal:7.6%
x264 [info]: SSIM Mean Y:0.9768366
x264 [info]: kb/s:989.2

encoded 40947 frames, 99.21 fps, 990.45 kb/s




Log for job job1-2

avis [info]: 448x336 @ 29.97 fps (40947 frames)
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
mp4 [info]: initial delay 2002 (scale 30000)
x264 [info]: slice I:167 Avg QP:21.20 size: 17934
x264 [info]: slice P:16664 Avg QP:23.73 size: 6201
x264 [info]: slice B:24116 Avg QP:25.03 size: 2673
x264 [info]: mb I I16..4: 23.5% 42.6% 33.9%
x264 [info]: mb P I16..4: 7.1% 8.1% 4.5% P16..4: 35.4% 13.9% 4.0% 1.1% 0.6% skip:25.3%
x264 [info]: mb B I16..4: 0.5% 0.4% 0.5% B16..8: 23.4% 3.1% 8.3% direct: 6.0% skip:57.8%
x264 [info]: 8x8 transform intra:39.9% inter:36.5%
x264 [info]: direct mvs spatial:63.9% temporal:36.1%
x264 [info]: ref P 77.3% 16.4% 6.3%
x264 [info]: ref B 84.7% 12.1% 3.3%
x264 [info]: SSIM Mean Y:0.9823452
x264 [info]: kb/s:1000.0

encoded 40947 frames, 39.43 fps, 1001.25 kb/s

desired video bitrate of this job: 1000 kbit/s - obtained video bitrate (approximate): 1004 kbit/s

[foxyshadis]

agressiv, are you saying that on TWO Opteron Socket F processors, you got ONLY 55 FPS? That's very strange, because you tried MeGUI on normal DVD source, 720x576 PAL, right?

CPU usage were only 35-45 %? That's the reason why you got only 55 FPS.
No, 4 dual-core opterons, 8 cores. I wonder if this shows more avisynth's or x264's threading inefficiencies, or both. Still pretty cool.

[akupenguin]

x264 is arbitrarily limited to 4 threads + input. Which makes 45% pretty good (insofar as threading efficiency can even be measured by % cpu), considering that 50% would be perfect.
I suppose there's no real reason to limit it, as long as you know that low resolution + lots of slices => lots of wasted bits. I just have to get to work on that sliceless threading model...

[Kostarum Rex Persia]

Yes, you really should do that, akupenguin. Please try to optimize x264 for more threads.

[ChronoCross]

Yes, you really should do that, akupenguin. Please try to optimize x264 for more threads.

Boy that sounded pretty assholeish. Could you please try not to make yourself look like an ass, seriously you should learn that he doesn't need to do anything.

[bond]

x264 is arbitrarily limited to 4 threads + input. Which makes 45% pretty good (insofar as threading efficiency can even be measured by % cpu), considering that 50% would be perfect.
I suppose there's no real reason to limit it, as long as you know that low resolution + lots of slices => lots of wasted bits. I just have to get to work on that sliceless threading model...as slices are also an error resilience tool, being useable for other things than multithreading, it might also make sense to remove the limit

[check]

are the inefficiencies that creep in when encoding with slices based on absolute dimensions of the slice, number of slices or a mixture of both?

[akupenguin]

There is a constant overhead of a few bytes per slice per frame, to write the slice header.
There is an variable overhead due to resetting cabac contexts, which converges asymptotically to some constant (* number of slices) once each slice is above some threshold bitsize (I have no idea what the threshold is).
There is an overhead due to the lack of intra prediction and motion vector prediction across the edge of the slice, whose magnitude is proportional to the number of macroblocks on the border (width of the frame * number of slices) and also depends on bitrate.
Optionally (not implemented in x264) you can limit deblocking to within a slice, which slightly improves parallelism but lets some block artifacts go unfiltered. This is likewise proportional to (width of the frame * number of slices).

[lexor]

Why the hell isn't KRP banned yet? his every utterance sounds so arrogant and prickish. Does this vB has a feature to ignore posts of a member?

[akupenguin]

Yes (User CP -> Buddy / Ignore Lists), but it only ignores the content of a post. I can't set it hide the whole post, make threads started by KRP invisible, and not bump threads when KRP posts.

[Thunderbolt8]

upporting more threads would at least be a step into the future, since intel already anounced quad core cpus to come for the next year. but i remember someone saying the quality would decrease slighly when using multiple threads or something like that, is this correct ?

btw. why are so many people using amd processors right now ? atm all the intel cpus are quite superior compared to the amd ones, even regarding the price. with a cheap, overlocked 6300 or 805 D you can surpass almost any current amd cpu and also reach close to intel's best ones.

[lexor]

upporting more threads would at least be a step into the future, since intel already anounced quad core cpus to come for the next year. but i remember someone saying the quality would decrease slighly when using multiple threads or something like that, is this correct ?
I don't see how quality could possibly decrese in multithreaded environment, if anything it may go up with additional cpu power. Quality loss can only come from a bug, not from the fact that it's multithreaded.


btw. why are so many people using amd processors right now ? atm all the intel cpus are quite superior compared to the amd ones, even regarding the price. with a cheap, overlocked 6300 or 805 D you can surpass almost any current amd cpu and also reach close to intel's best ones.

because it was not always the case for both price and performance, in fact it is quite recent occurance that preceded Core2 release. And for the past few years AMD had upper hand in both respects. You can't expect people to buy new hardware everytime new thing comes out.

[Manao]

Quality loss can only come from a bug, not from the fact that it's multithreaded.No. See akupenguin's explanation.

[foxyshadis]

but i remember someone saying the quality would decrease slighly when using multiple threads or something like that, is this correct ?
You mean, the 3 posts above yours, by the x264 author?

The "sliceless multithreading" mentioned earlier won't cause that degradation, though it's some way off. It's not too bad as it is, especially for HD, but if you compare closely enough you'll see a difference on SD.

btw. why are so many people using amd processors right now ? atm all the intel cpus are quite superior compared to the amd ones, even regarding the price. with a cheap, overlocked 6300 or 805 D you can surpass almost any current amd cpu and also reach close to intel's best ones.
Haha, I never understod how anyone could recommend or desire a jet engine Pentium D of any sort, when X2s were plentiful and cheaper (let alone now), but I suppose I've been using silent laptops too long. [/OT]

[lexor]

No. See akupenguin's explanation.
I don't see him saying anything about quality loss in this thread. Reagardless, quality loss cannot come from the fact that it is multithreaded. It either has a bug, or it's threaded wrong (i.e. design flaw or a conscious descision to cut a corner). Multithreading doesn't decrease or increase quality of computation.

[Isochroma]

Since the slice method causes increasing loss of compression efficiency as greater numbers of slices are used, it is most useful as an interim solution until alternative multithreaded code is created.

Just a few days ago I was reading a fascinating article about Intel's plans to ramp up the number of cores on a die to some fabulous numbers (http://www.tomshardware.com/2006/09/27/idf_fall_2006/) in the very near future: 88 cores!

This is really exciting news, and of course x264 along with lots of other compute-intensive software will be revolutionized as its internals are completely rewritten to support arbitrary thread scaling without loss of compression or compute efficiency.

Perhaps most interesting will be the change in mindspace as new programmers are raised in a parallel environment, along with tools which both work optimally and enforce design criteria of this modality on users and systems, respectively.

I'm optimistic about the fundamental ability of x264 and other video codecs to make completely scalable use of massively multicore processors, due to the inherently parallel nature of the material being processed. As with raytracing and physics simulations, the source data is derived from or mimics operations in realspace, which is of course parallel in nature.

[emmel]

There is no bug. The model/algorithm depends on the number of slices/threads as explained before.

Just try --threads 1,2,3,4 with a small sample. With constant quality, the bitrate depends on the number of threads. With constant bitrate, the quality is thread-dependent. Wasted bits = loss in quality.

[foxyshadis]

I don't see him saying anything about quality loss in this thread. Reagardless, quality loss cannot come from the fact that it is multithreaded. It either has a bug, or it's threaded wrong (i.e. design flaw or a conscious descision to cut a corner). Multithreading doesn't decrease or increase quality of computation.
In that case, it's the MPEG committee's "design flaw". But please understand that it isn't a design flaw or a cut corner, it's an implementation of a feature of the h.264 spec (slices), which causes a slight quality loss. As a consequence of implementing that feature, fairly simple multithreading is possible. That and error resilience was the point of including it in the spec. It's not the threading that causes quality loss; it could just as easily be sliced single-threadedly, with the same loss, but that would be rather pointless even on single-cores. Or it can be unsliced multithreadedly, but that option does not yet exist in x264.

[akupenguin]

Let me put it another way:
While video filtering is perfectly parallelizable, video compression is only almost embarrasingly parallel. A compressed video stream presents the information in some order (defined by the standard), and the point of compression is to minimize the number of bits needed to represent a given piece of information, by exploiting any redundancy it shares with previous pieces of the stream. In lossy compression, you can't evaluate that relationship until you have encoded said previous pieces. So video compression is naturally serial, and you have to make some approximations if you want to split it into many threads.

The "approximation" I refer to is that the encoder (either with support from the standard, or just as an implementation decision) must choose to ignore some possible redundancies, in order to independently process some pieces of the encoding task that could have depended on eachother. If you don't exploit that redundancy, then your compression algorithm doesn't compress quite as well as the original serial one. And it is no coincidence that the same features help both error resilience and threading: Error resilience is also about adding a controlled amount of redundancy back into the compressed stream.

Different threading models use different pieces of redundancy:

x264's slice-based threading fails to exploit some spatial redundancy between blocks within one frame.
xvid's mb-row threading is lossless, but only works because mpeg4asp's entropy coder fails to adapt its probability model to the observed data.
my proposed sliceless threading method depends on the fact that a given macroblock is usually predicted only from a nearby block in the previous frame, or a neighbor in the current frame. If typical motion vectors were large, then sliceless threading would also waste bits. It also depends on the fact that h264's entropy coder doesn't adapt to anything beyond the current frame.
B-frame threading depends on the fact that (most) B-frames are not used in inter-prediction. It only works in sections of the video that use B-frames, and doesn't scale beyond about 2 threads.
GOP-level threading is lossless, but depends on the fact that you already have to spend bits on I-frames in order to get fast seeking and some minimal error resilience. It also takes a rediculously large amount of RAM if you want to fit it into a standard stream-like API.
ELDER's threading has a small impact on the precision of ratecontrol, and has to choose between extra I-frames (wasted bits), or a little duplicate work (wasted time). It also doesn't fit at all into a stream-like API.
Sub-macroblock threading might be possible if thread synchronization were fast enough. But it would duplicate quite a bit of work, since it would bypass some of x264's early termination conditions. And it wouldn't scale beyond 2-4 threads.

[lexor]

From above I gather that the problem arises due to desire to separate the work into units that can be run separately, and it isn't possible to make that division perfectly. What about the parallelization of loops offered by OpenMP? From what I see the code is full of loops. Having vague understanding of the standard, I'm not quite sure what the loops are doing, but pretty much everything seems to sit in some kind of loop (either a loop directly present in a function, or a function with no loops being called from a loop).

Will that hit the same problems?

[Tagert]

Just to add some numbers into the existing discussion.
Got a dual core rig myself.
intel core2 duo 6600 cpu

Been trying a few settings with the megui and x264 today.
Some numbers:

720x576 @25fps, dv avi clip with a few fading titles and such (deinterlaced in sony vegas before opened in megui)
HQ-slow profile.
2 threads.
1st pass - 26fps - load 50-60%
2nd pass - 21fps - load 90 -97%

If I tried deinterlacing this with avisynth and some of its plugins instead of in vegas the fps dropped considerably to about 3fps on the first pass.

[foxyshadis]

Don't use the slowest avisynth deinterlacer, then. If you don't mind vegas's crappy deinterlacing, there are several extremely fast plugins for avisynth as well.

[Blue_MiSfit]

Boy that sounded pretty assholeish. Could you please try not to make yourself look like an ass, seriously you should learn that he doesn't need to do anything.

Dude relax, he's a troll yes but there's a language barrier here. A lot of the time we all misinterpret what he's saying. Be nice :) He basically said "yeah a better multithreading system would be good", just not in a way that we would consider "nice".

~MiSfit

[squid_80]

He's been banned since last week anyway :p

[ChronoCross]

He's been banned since last week anyway :p

hmm wow I hadn't noticed. but I love getting lectured by a regular who posted 2 weeks after I said it.....oh well.

[Tagert]

Don't use the slowest avisynth deinterlacer, then. If you don't mind vegas's crappy deinterlacing, there are several extremely fast plugins for avisynth as well.
Well im pretty new at this avisynth thing so I can't know every good script yet so bear with me ;)

Think it was called Mike Crash' De-interlace filter that I grabbed to use in Vegas.

In Avisynth I used Tdeint and EDII.

[arfster]

Not a question, but I figured I'd put MeGUI/x264 on one of our 4-way opteron 2.8ghz Rev F evals we got in. Arguably one of the fastest 4-ways on the market, getting 55 FPS encoding a 2nd pass (HQ-Slow profile, 8 threads) is very impressive :)

Just for comparison, my 3.6ghz Pentium D does 19fps. NTSC DVD, HQslow.

You've got twice the cores running but at 25% lower clock speed, and still get 3x the fps :-)

[JamPS]

Well im pretty new at this avisynth thing so I can't know every good script yet so bear with me ;)

Think it was called Mike Crash' De-interlace filter that I grabbed to use in Vegas.

In Avisynth I used Tdeint and EDII.

Have you tried "StaxRip" application?
It's opensource and works best for me.:D
http://www.planetdvb.net/staxrip/