x264 feature test: multithreaded encoding for dual-cpus

[tomos]

may give this a go tomorrow then. have something on the go at the moment that will take another 14-15 hrs to finish (1080p)

would you like dvd res, or hdtv res as the test?

[HardwareGeek]

I have a Pentium 4 with hypertheading technology and my speed of encoding doesn't change with one or two threads. ...
Also, a program needs to be written to take advantage of hyperthreading, in order to see significant improvement. A multithreaded program may see a little improvement, but I don't think always. Single-threaded programs frequently see less performance with hyperthreading.

[ndkamal]

For TOMOS,I m talking about a DVD Backup, like Matrix or another one, we can put resolution, the lentgh of the movie, the speed of encoding with one or two Cores, with one of the latest versions of X264. Thanks for advance.

[Doom9]

I put some performance data in the hardware forum when I first got my X2.

[tomos]

ah ok. have matrix at home so use that - although i havent done a dvd backup in ages.

is using dgindex, and avs to encode in vdub ok? aiming at 700mb for just the film? 2-passes etc.

-Doom9--

i'm kinda curious now as well anyway so will give this a go for my own curiosity :D

[ndkamal]

You can use CLI version, MP4 output, a bitrate of 2000 with CE Highprofile without turbo, Automated 2 pass,resolution 672*272, the others parameters don't change, not all the movie but one or two chapters like Chapters 10 and 11. The number of fps and time encoding.

[tomos]

never used CLI ver before but will give it a go

[Doom9]

I think you can set the number of threads in VfW as well.. but if you want to use the cli, we have a whole forum with 3 tools that use the x264 commandline encoder.
@ndkamal: with a little searching, you'd have found this: http://forum.doom9.org/showthread.php?t=94226&highlight=doom9+x2

[ndkamal]

Thanks at moderator I haven't see this topic before.

[tomos]

ok, doing this now (with megui) oddly - 2 threads 85-90% cpu usage?? never noticed it before

may try with 3 thread to see if it picks up the slack

[tomos]

ok. just did chapter 10+11 of matrix now with latest release in megui - no resizing (couldnt see how in megui), and with the parameters that ndkamal specced

1st - 2 threads, both cores used

start - 17.27.42 end - 17.43.32 FPS - 11.8495
start - 17.54.04 end - 18.08.10 FPS - 13.3062

2nd - 1 threads, one core used (affinity set process - above normal)

start - 18.10.13 end - 18.37.41 FPS - 6.83
start - 18.37.41 end - 19.03.01 FPS - 7.40

so 1st avg = 12.5778
and 2nd avg = 7.115

i figure 77% performance incrase in using dual core over single. although to be fair thats single core without any OS processes on it. maybe 80-85 % increase in performance?

[ndkamal]

Thanks TOMOS. I Have made some tests on my Pentium 4, and I have noticed that the multithreaded encoding works in VFW version and not in CLI version, perharps a bug, I have a increase of 20 % of speed.

[ndkamal]

Some questions for TOMOS about your encoding, what resolution, Do you use CLI or VFW ?

[tomos]

in that one i just did, it was using megui which afaik is just a front end for the CLI.

normally, i use virtualdubmod though

[Doom9]

and I have noticed that the multithreaded encoding works in VFW version and not in CLI version, perharps a bug,It's VirtualDub using two threads by default.. seems virtual SMP likes that more than the traditional read->write approach.

[Revgen]

CLI works fine with my AMD Athlon X2. Perhaps virtual threads don't work as well for CLI as true multi-threading.

[ndkamal]

I would have the opinion of others P4 HT owners. So I have noticed a increase of performance in the latest version of x264 about 80% and 60 % in build 270.

[tomos]

just as a sidenote. i tried adding a 3rd thread, but that made it a little slower than 2 threads.

[Doom9]

thread synchronization takes its time, and the thread scheduler is likely more efficient if it has the same number of performance hungry threads as cores it can distribute the threads on. It's like breaking down a two man job into three bits and let two people work at it.. they can do just fine with jobs one and two, but if each job takes the same amount of time, they'll finish both at the same time and if job 3 cannot be further divided, the two people can not work as efficiently as they could if they both had a separate job. I know that's grossly simplified but I think trying to imagine multithreading in terms of tasks at work and people to do it makes understanding quite easy.

[tomos]

just testing tbh since my CPU wasnt running @ 100%

if it just used 5% more then over a long encoding time it would have been worth it

live and learn tho :)

[sjchmura]

So is anyone compiling(I understand non-standard) dual core SMP build like the new 408 one???

[Doom9]

@sjchmura: you know, a little research cannot hurt. Ever since I don't know how many builds there's an option to select the number of threads you want. The fact that you're asking shows that you have not done the research we expect our members to do before they post a question.

[sjchmura]

@sjchmura: you know, a little research cannot hurt. Ever since I don't know how many builds there's an option to select the number of threads you want. The fact that you're asking shows that you have not done the research we expect our members to do before they post a question.


I am not sure why you want to insult me but ... as someone that has helped alot of people here this attack was not waranted. If my confusion over numbering - fine. But there ARE different builds and how they are counted. For example, sharktooth build is 408 (recently held due teo bugs). I was assuming the 270 dual CPU build was older but if this is a new numbering system then that is all you needed to say.

Anyone trying to move from Nero AVC to x264 needs to do ALOT of research and it is not easy at time to keep up with build numbers and differences.

I am sorry I offended you or the authors of the test build.

Best
STeve

[madoka]

No. The movie is sliced into N equal parts within each frame, so ratecontrol is not affected. The only detrimental effects are: cabac contexts are reset, and macroblocks on the top edge of a slice don't get to predict MVs from the row above (both slightly increase bitrate)...
What is the advantage of this approach over a more simplistic one, where one divides the entire movie into N equal parts? Then one just runs N different processes while setting their processor/core affinity accordinly.

[Doom9]

where one divides the entire movie into N equal parts? Then one just runs N different processes while setting their processor/core affinity accordinly.Consider this (and it's a pretty obvious thing if you ask me): part 1 is a slow motion scene, part 2 a high motion scene. If you divide the target size by N, part 1 will look very good, part 2 will look bad because they both get the same bitrate.

[708145]

Consider this (and it's a pretty obvious thing if you ask me): part 1 is a slow motion scene, part 2 a high motion scene. If you divide the target size by N, part 1 will look very good, part 2 will look bad because they both get the same bitrate.

well splitting in the temporal domain does work but needs a bit of thinking and development to get it right.
ELDER does keyframe exact splitting (actually frame types are the same for each frame in xvid default vs. ELDER) and uses a bitrate distribution _very_ close to xvid's default.
The main drawback I can see is that I lack some spare time to dedicate for ELDER development... there are too many things with higher priority :)
Anyway I will get out beta4 some time soon but probably not with all the features I planned for this version.

[madoka]

Consider this (and it's a pretty obvious thing if you ask me): part 1 is a slow motion scene, part 2 a high motion scene. If you divide the target size by N, part 1 will look very good, part 2 will look bad because they both get the same bitrate.
Forgive my naivete, but I'm not quite following. Suppose there are three parts: part A is slow motion, part B has moderate motion, and part C is high motion. So ideally fewer bits should be used for part A, and the savings used on part C. But how would a single-pass rate control algorithm know this? I imagine there's some sort of look-ahead window which the algorithm uses to tweak the bit rate, but in general the window will be small. Hence, while in part A it can't "see" far enough to realize that part C needs more bits.

Now, a 2-pass rate control algorithm wouldn't be bound by the restriction, since the statistics gathered in the 1st pass would indicate that part2 is high motion. However, since the 1st pass is essentially encoding with a constant quantizer, the amount of motion doesn't matter, so dividing the movie into N parts also doesn't matter. Then, in the 2nd pass, the process encoding part 1 will know, based on the 1st pass statistics, to use fewer bits. Similarly, the process encoding part 2 will also know it can use more bits.

The procedure above requires some changes in x264, but it seems less complicated than threading.

Lastly, the examples we came up with are both quite artificial. Theories and models aside, isn't it safe to assume that real videos distribute their high and low motion scenes over time roughly uniformly? If not, then what about dividing the movie into k*N pieces, and each processor/core encodes k random pieces?

[akupenguin]

Forgive my naivete, but I'm not quite following. Suppose there are three parts: part A is slow motion, part B has moderate motion, and part C is high motion. So ideally fewer bits should be used for part A, and the savings used on part C. But how would a single-pass rate control algorithm know this?
That's where x264 wins. 1pass ABR can do an approximate distribution without any lookahead. Strictly speaking, ABR could still work if you split the movie into pieces... but since each piece could encode at different speeds, the result would be non-deterministic, which sucks for a developer.

Lastly, the examples we came up with are both quite artificial. Theories and models aside, isn't it safe to assume that real videos distribute their high and low motion scenes over time roughly uniformly?
Random distribution, maybe. I'm not even sure of that; there may be a bias for more action towards the end or something. But when local bitrate varies by a factor of >10, a 2 hour movie is not nearly long enough to make a random distribution appear even.

If not, then what about dividing the movie into k*N pieces, and each processor/core encodes k random pieces?
Doesn't that just make the distribution problem even worse?

[Doom9]

isn't it safe to assume that real videos distribute their high and low motion scenes over time roughly uniformly?I'm sure if you ask a film student he'd have to disagree. Movies storylines generally follow a certain pattern with a bunch of climaxes. For an action oriented movie, those climaxes would generally signify a high amount of action, and the climaxes are not uniformly distributed. I can't get into more details though since I just don't recall them.

[Ice =A=]

Just one more thing: Splitting the whole movie in two parts would also increase the encoding time on multicore systems, since the two parts would likely not be finished simultaneously.

[madoka]

That's where x264 wins. 1pass ABR can do an approximate distribution without any lookahead.
Interesting. Can you give me a brief explanation on how the algorithm works?

If not, then what about dividing the movie into k*N pieces, and each processor/core encodes k random pieces?Doesn't that just make the distribution problem even worse?
I'll concede Doom9's point that a typical movie doesn't uniformly distribute its low/high motion scenes over time. So the idea is to sample the movie randomly; if the sample is large enough it should have roughly the same low/high motion distribution as the movie, no? Of course, if the pieces gets too small the discontinuities at the boundaries will become problematic.

However, if I understand how 2-pass rate control works, then none of the above would matter. Besides scene complexity, are there other information that requires coordination across all processes?

On the other hand, synchronization is a problem that needs to be addressed. But the same problem exists for threads, no? There's no guarantee that each thread finish processing their respective frame slice at the same time, either.

I'm curious about the feasibility/flaws/drawbacks to this approach, because it seems more easily extensible to cluster encoding. I was thinking of doing a project on this, but 708145 (http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER) beat me to it. I guess I'll just have to think of something else to do...:(

[akupenguin]

Interesting. Can you give me a brief explanation on how the algorithm works?:search: and/or look in x264's "doc" directory.

I'm curious about the feasibility/flaws/drawbacks to this approach, because it seems more easily extensible to cluster encoding.

Slices:
+ Really easy to implement. (x264's multithreading code is a total of 50 lines of C in one function)
- Doesn't perfectly fill even 2 CPUs.
- Each slice boundary costs a few bits.

GOP threading:
+ Scales perfectly to decent-sized clusters.
- Requires application support / restricts API (seekable input, heavily buffered output.) Not usable for realtime streaming.
- Requires CQ or 2pass (no ABR).

[JamPS]

Report:
When using --threads 2 I get around 95% CPU utilization when encoding with StaxRip(x264 gets to work alone from commandline).
So I'd say multithreading support is nearly perfect on my computer :)
I don't think it will get/should be any better...

[Sharktooth]

JamPS, well... it all depends on the CPU you have...

[Zarxrax]

I did a search and can't seem to find an answer, so I figure I'll ask.
Does --thread-input create an additional thread for avisynth aside from the other threads that have been created?

If I use --threads 2 --thread-input, would this make a total of 3 threads, or just 2?

[akupenguin]

--threads 2 creates a total of 3 threads, one of which does avisynth.
--thread-input matters only if you're not otherwise using threaded encoding.

[ChronoCross]

so it would be mor efficient when using a SMP to do:

x264.exe --threads 2

and for single processor

x264.exe --threads 1 --thread-input

Would the first one be faster on single processor than the second one or is there a reason the second one would be faster?

[akupenguin]

for 2x SMP, do
x264.exe --threads 2
(which automatically enables thread-input also)

for a single processor with avisynth input, do
x264.exe --threads 1

for a single processor with rawyuv file input (i.e. the input is limited by harddrive speed, not CPU), do
x264.exe --threads 1 --thread-input

[Sharktooth]

so --thread-input could be useful even in the input is limited by the network, right?
lets say, we have 8 PCs encoding simultaneously and getting data from a single fileserver. the server network card gets easily overloaded and packets speed would be subdivided for the 8 PCs.
--thread-input would help as in the 3rd example you made... is my assumption correct?