x264farm: distributed encoding

[Allbestmessage]

There is a command line encoder profile you could use.

[Breakthrough]

Hello, I have a few questions...

What build of x264 do you all reccommend to use with x264farm? Or at least, what was the final build before the revision to the .stats file was committed?

Also, does the server computer also have the ability to render? Could I theoretically run a controller AND an agent on the same PC?

Thanks!

[nst6563]

I'm itching to get this working as designed...however it seems I've either got something setup wrong, or something is setup wrong

To start, I've got the controller running on a 3Ghz Core2Quad with 6GB ram. I have another machine that is connected via 100mb switch which is a core2duo w/4gb ram.

I've got the config setup for adhoc, and the controller and client see each other - that's not the problem.

The problem I think I've run into is the speed. I was hoping it would be faster than standard encoding with something like ripbot or handbrake...however I get about half the speed compared to those gui-based encoders.

I've tried the controller-based encoding as well as the client-based encoding. I don't remember if I got the client based encoding to work correctly tho... the way I read and figured out was that each client needs access to the sources being encoded right? I made a mapped drive on the machines, and pointed the config to that and it seemed to work ok.

I have 5 machines with a total of 13 cores that I could put towards something like this...I'd love to see it scream though an encode...what could be my limiting factor other than the 100mb switch (trying to get a 1000 but my current 1000 switch shot craps...thanks Dlink)

any plans of something like a gui for the controller setup? Drag a bunch of files to encode to the window (or provide a path with the files to encode), answer some questions and let it go. maybe launch the agents as a windows service?

[iplmedia]

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.

[creamyhorror]

Originally Posted by iplmedia

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.

Don't forget the single-pass variant: x264farm-sp - single-pass distributed encoding, which was last modified in 2008 I think.

[creamyhorror]

Originally Posted by iplmedia

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.

Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264

[bnshrdr]

Originally Posted by creamyhorror

Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264

I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264. Sure I use it for all of my encodes, but I don't know enough about it to write pretty much a wrapper program to utilize it.

Currently I have 2 computers here at my home encoding the same video file, but some of the ways I am achieving this are either ugly or impractical. Also I don't really know how to truly handle a multi-pass scenario.

I just need info on how to do faster seeking, like using an index. I know of DGAVCDec but I don't know how to put it to good use.

I also need a way to get the frame count of the source before processing the job, but my knowledge of the AVISynth API is 0.

I am also welcoming any ideas about the whole design of this program.

[iplmedia]

Originally Posted by bnshrdr

I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264.

If am very familiar with x264 just not how to wrap it! x264farm has many benefits to it by dividing the video into GOP, however, it decompresses it to ray YUV before compression (From what I understand) and this is done in the first pass that only 1 machine performs. Thus greatly slowing down the encoding process. I would love to discuss the possibility of working on this project if you like. Send me an email and we can discuss this.

[creamyhorror]

Great, hope you guys get together to start a project. You might want to hang out in #x264dev on Freenode IRC to ask the x264 devs questions.

[iplmedia]

I have worked with and am working with one of the x264 developers already on a different project. They are a great team of developers!

[bnshrdr]

As much as I'd like to start a little project, I really don't have time. During the school year I always nibble little bits away from my project goals. I'm kind of just a hobby programmer, but being a full time student limits the time I can contribute to any sort of project. Thanks though for the offer and I appreciate your support. I know where I'll come when I have problems :P.

[popper]

the Very first thing you should perhaps do is totally do away with a fixed x264 version requirement in any code so as not to run up against the code updating being dropped as happened with farm for SO long...

it might be a very good idea to start by parsing a current x264 --fullhelp at runtime and populating your input/output variables from that parse, so that as fullhelp grows over weeks,months or years your code can take advantage of any new options that come along later and still work fine when things change.


as it is today the general feeling is that always using the constant quality option CRF= as your base is the default way to go.

x264farm while a good idea always bugged me in the way it doesnt seperate out all the different parts of the operations code into self contained stand alone scripts, so making it really hard to add new or just different ways of doing the same thing for a given situation,as a plug-in type way to work, so that could be made far cleaner if YOUR really serious about this rework.

you might want to add a new better client/server part but keep all the other bits as they were for instance, or you might want to add DHT for the auto what Hw's available to our IP cluster at this time, and who's ready to take another job, send a DHT message and add and/or act to the 'message based' actions que(s) etc... the options are endless with a little thought and an open self contained fully commented script for one action, or were needed for simpicity a set of related fine grained actions etc.

it should also be considered that current x264 also takes many input file types stand alone now OC so thats a good thing, and sooner or later it will also allow audio parsing/re-encoding to a given container too, so making the whole updated x264farm framework far easier and better later IF YOU JUST PARSE a current x264 fullhelp output or ask DS to provide a better way perhaps if your serious about distributed encoding.

for instance how might a future x264 farm make easiest use of things like this guys http://forum.doom9.org/showthread.php?t=152232 with HPC cluster (750 cores)...

people will always want to use the latest x264 so you had better find an easy way to always auto include/use it with a new x264farm framework without braking the farm....if you want people to use it and give lots of thanks and perhaps even get involved for years to come ;0)

and preferably cross platform usable too, put it infact the whole turnkey package on a generic slax liveCD http://www.slax.org (WITH DEFAULT ready setup pxe server LAN booting AS STANDARD BTW) AND boot any hardware thats in the LAN ANYTIME.

just a few thoughts to consider anyway.

[Karyudo]

Good timing! I just started researching how to build some kind of x264 render farm, and here I see new interest after what looks to be a year or more of downtime and benign neglect. Nice!

I can offer almost zip in the technical department, but I do have a home network with four (soon to be five?) computers that could conceivably all run DGDecNV on at least Core2Duo processors, so maybe I'd be able to help with beta testing (when things progress that far, I mean).

Or, if that's no big help, then at least let me add my vote for this idea as a project that I think would be powerful and useful!

[iplmedia]

Thank you for your thoughts on this project. I am currently planning out the best architecture for this project trying to utilize the best possible means to accomplish the best possible product. I agree with future compatibility as well as cross-platform. I currently work solely on a mac and would like to see it work on that but understand their are many other users in the *nix and windows world. The concept of a liveCD for the cluster is very familiar to the Linux world (Clusterknoppix etc).

I am considering using OpenMPI and dividing the video into GOP's for distribution so regardless of how many videos you have in queue the entire cluster will be utilized. I realize that their are inherent flaws and considerations but my goal would be very fast cluster detection, minimal bottle-necking and intelligent distribution amongst all available resources with fault-detection and load balancing.

It's a tall order but in my opinion much needed and sought after.

Please continue to post your thoughts/ideas/suggestions.

Thanks

[kemuri-_9]

nitpick: OpenMPI is only an implementation of MPI-2, you would be using the MPI-2 API if using OpenMPI as a test implementation.

I had worked somewhat on an MPI-2 patch (was using the MPICH2 implementation myself for windows) nearly a year ago for x264 but stopped eventually when i realized that it would require a fairly large restructure.
I had also planned on GOP distribution among the cluster and was coding along that guideline.
it would probably be easier to implement now utilizing threaded lookahead that was committed some moths ago.

[magnatique]

when this is up to a testing phase, I would be happy to give the farm some testing grounds as we have core2, quads i7s at the office on our gigabit lan, with some on XP / win7 so it should give a good testing ground!

Phil

[iplmedia]

Hello all,

Thank you for your input. I am continuing to research and gather the best possible method to accomplish this and your input helps eliminate dead-ends and un-needed wastes of time etc. So please keep them coming. In regards to the MPI-2 implementation I still see it as the best method as well as the GOP. What were some road-blocks you ran into with using MPI and would you suggest a different method or is it still be best method just that it requires a lot of work. I am willing to invest the work/time/investment that it would take to accomplish this I just want to make sure I'm heading in the right direction. Would you be willing to send me the work that you have started already on the x264 MPI implementation? Any thoughts and information you can pass my way would be greatly beneficial. Cheers

Aaron

[kemuri-_9]

hmm.. having to pick my brain as to the thought processes i had back when i was working on it...

the MPI-2 API has some functionality to determine rank in the operational cluster (along with the size of the cluster) and i used this to define the following:
a node with a rank of zero would indicate the master node - that is the node that is controlling source decoding to send and receive frame data back from slave nodes for writing the final encoded video
otherwise a node has a rank of nonzero and would be considered a slave node.

A) Where to decide where to transfer frames between master and slave nodes
When i was working on this i had issues on where I was going to do this and was a major factor in stopping work.

A1) from working on the threaded lookahead patch,
I would think it would be easiest location for transfer frames from the master node to the slave node.

A2) as for the reverse.... don't recall thinking about that much... thoughts now would have it either in encoder_frame_end or in x264cli

B) what requires transferring:
originally when i was working on this there was only 1 frame type, but now with there being 2, this changes up a little bit.

B1) fdec x264_frame_t
frames that have been slicetype decided, would be transferred from master to slave for encoding.

B2) fenc x264_frame_t?
frames that have been encoded and QP decided, would be transferred from slaves to master.
this is a bit tricky as x264_frame_ts have a lot of data that would be unncessary at the master slave as all it would really want is the resulting x264_picture_t.
though if you only do decide to send back the x264_picture_t you would have to special case the --dump-yuv option in some fashion.

B3) stats
information acquired while processing frames,
involving bit costs and other information used at encoder_close and in stat file writing,
would need to be synced between master and slave node (i didn't get to this part so not sure what the best way of doing it would be).

B4) x264_param_t
the parameters to init the encoding engine with,
I had ambiguous thought patterns on how to handle this,
but i think my last choice was to init the master node with the param_t given on the cli and then send a possibly altered param_t to slaves to init with.

C) Gotchas i remember
C1) data sending.
transferring structures is eloquent in MPI-2 especially when they have pointed to data.
this required creating an extra send buffer and `packing` along data in a struct referred by pointers along with the struct to be unpacked at the receiving side.

C2) GOP distributed encoding.
using a GOP distributed encoding style would allow slave nodes to process on different GOPs with requiring relatively no information from each other.
however, this does have some down sides:
-- a) this could potentially cause the master node to allocate a large amount of data to buffer depending on how transferring of data is done.
that is if choosing to transfer all the GOP in bulk then this would require a buffer large enough to send and recv it all. naturally this could be alleviated by transferring in smaller pieces (e.g. mini GOPs).
-- b) requires unique GOPs to exist. that is the GOP distributed encoding fails when there are not unique IDR initiated GOPs, like in the case of Periodic Intra Refresh (PIR) or Open GOP.

C3) different CPUs and related issues
as x264 works on several different CPUs you would think "oh i could have this cluster over all the different cpus it works on easily", which is sadly untrue.
x264 uses the CPU's cacheline to determine how to pad frame data in x264_frame_t's for optimal performance on that cpu.
there's a few ways you could handle this when clustering between unique cpus:

-- a) alter all the node CPUs to have the same cacheline values as the cpu within the cluster that has the largest cacheline performance value
(this would require using more memory for the cpus that had the cacheline padding increased)

-- b) alter all the node CPUs to have the smallest cacheline values of the cpu within the cluster
(this would require using less memory for the cpus that had a cacheline padding decrease at the cost of some performance)

-- c) repad the data to the receiving cpu's cacheline at send/recv ing of data
(but seriously this would be more effort than it's worth so i don't recommend it)

tl;dr:
lots of things to worry about and the finesse required to do it was a major factor in me stopping work on it.
(didn't help i was in my last semester of college while working on this and wanted to graduate more than work on it)

overall i didn't see a particular issue in using MPI-2, as the surrounding issues are not transfer protocol specific but sync specific.
MPI-2 does offer a variety of sync/async API to help out on the issue of synchronizing, so i would continue recommend using it as the implementation to do this cluster support work.

I would be willing to share the code i have, but as it is incredibly old and imo bleh now,
I would only use it now as reference code on how to do something rather than actually trying to apply it to current repository and working from there.

[popper]

for more input you might also look to spending some idle time on http://gogloom.com/client2/index2?ma...hannel=%23x264 to.

i still think using a generic slax is a good way to go , incase i didnt make it clear you download a generic slax livecd, boot it, select the setup pxe server LAN booting option, and instantly you have a lan bootable slax, go over to any other PC on your LAN AND boot it into pxe mode and it will be running that exact same livecd you just booted with every module that you happen to have put on the slax livecd....

slax is the only generic liveCD/liveUSB distro i know of that comes with this pre-setup pxe setup and ready to use out the box, anywere.

OC its werth also mentioning that slax creator Tomas M is also the guy that gave you the liveCD/liveUSB scripts that everyone uses today OC

http://www.slax.org/forum.php

you could boot a native liveCD/liveUSB or just use a Vbox and liveCD/liveUSB that for background client/server LAN and Vlan use etc with all the farm and related apps in module form you require on there for on the fly mouting/loading , no messing about.

[popper]

"I currently work solely on a mac and would like to see it work on that but understand there are many other users in the *nix and windows world."

thats not such a big problem though,(PPC G4/5 OR X86 BTW?) all it takes is picking the right tools for your potential client/server tunnel apps to the cluster you might use....

using 'rebol view' and 'rebol core' for the GUI scripting and client/server tcp:ip udp and multicast cross platform apps today. http://re-bol.com/rebol.html http://www.rebol.com/download.html is one option to consider if your into cross platform scripting control and related apps for instance.

heres a self contained multicast rebol view script that shows you how to make and use a simple multicast whiteboard for realtime data drawing and passing ,and more.
http://www.rebolfrance.info/org/arti...st/sources.zip
google translate his french page if you like
http://www.rebolfrance.info/org/arti...multicast.html

you want a tiny self contained pro webserver at the heart of this cluster(s) then include The Cheyenne Web Server
http://www.rebol.com/news/cheyenne.html written in rebol, or want a cache proxy for the potential message parsing etc then
use this as your basic outline and change it as required.
http://www.rebol.org/view-script.r?script=proxy.r

perhaps you want to use java as well for part of the new farm, then theres a DHT and building a multicast protocol on top of Bamboo paper by Marcel Dischinger outlining how you might include that http://bamboo-dht.org/tutorial.html

want a basic self contained multicast ipv4 overlay tunnel for connecting remote farm sections then consider the java 'mtunnel' , alas he took the page down now but its still available...

as i said the cross platform options are many if you just consider your requirements and can figure out the best and various ways to chop up the core HD content.

how about you draw up a basic visual flow chart jpg outlining your current thinking as to all the different sections that need laying out and posting that as a start for people to see and comment on any obvious potential problems ?, nothing fancy, just a quick visual aid.