x264farm: distributed encoding

[GrandAdmiralThrawn]

Hello!

For my own x264 Benchmark project (have already tested Itanium² shared memory cluster machines) I am looking into testing x264 with a distributed computation cluster.

I have set up a Torque/Maui cluster and I will also get access to another cluster based on the IBM LoadLeveler.

My question now is: These systems use PBS style job control systems with MPI support. In the case of my own setup thats either LAM/MPI or OpenMPI, job control by Torque with the Maui scheduler.

Is there any guide on how to run x264farm with a PBS-style job submission system? Is that even possible? How about other systems like the IBM LoadLeveler, where I might not be allowed to access any computation nodes directly? Any information very much appreciated.

Operating systems are CentOS Linux and (afaik) SuSE Linux Enterprise Server. Hardware is i386 and x86_64. No Windows stuff available in that environment.

Thanks for any input.

[Mounir]

I am getting started or rather trying to gather infos for distributed encoding accross computers and apparently there is little to no benefit to my big surprise (!).

It seems to me the current concept is flawed and that it would be better this way (i'm not a programmer mind you):
- say you have two computers
- For 1st pass(x264) devide the video in two segments: cpu1 deal with segment 1, cpu2 segment2
- Merge the stats files to make one (not sure if that's possible)

For the second pass i'm not sure but what if cpu1 encode the beginning of the video and the cpu2 encode start by the end (backwards encoding)
- The point being to make one file only not two and merge them later (= less space used)

Also since the two cpu's try to decode the video at the same time, if you decode a HD stream a good GFX card + DgindexNv would be required though i'm not even sure if dgindexNV could read the video backwards correctly
For lossless (huffyuv, lagarith etc..) sources things would be easier i think

I'm just throwin in ideas...

[Atak_Snajpera]

Distributed encoding is now ultra simple
see here -> http://forum.doom9.org/showthread.php?t=127611

[GrandAdmiralThrawn]

Actually, I solved the Problem in a very simple (albeit suboptimal) way. It's not even necessary to merge stats files.

I just split my source video into as many slices as I had nodes in the cluster to be tested (which was actually a Sun GRID Engine, very similar to OpenPBS/torque, running on OpenSuSE Linux). Then I let every node do its pass 1 and pass 2 on its respective slice, and at the end I just merged the final output video. To do this I used the job array feature of the PBS "qsub" command. Quite simple actually, the job script to achieve this was very minimalistic.

If you do this intelligently (which I did not, but doesn't matter) you should be able to avoid artefacts on the joints by making sure that the parts overlap each other a bit, so you can cut the overlapping parts and join the output video smoothly.

With that method you can use pretty much any Linux/Unix cluster that you can compile x264 (+libav/ffmpeg) on. All you need is shared storage for the cluster nodes and a decent enough job submission / portable batch system.

Performance scaling is linear!

[Yiu]

I found this thread in my bookmarks and laughed.

My system specs are a bit more grounded now, and I've been doing test with new builds of x264 as well as re-optimizing settings and I still get the same quality encoding @ 2~5 FPS.

My signature is updated too.