Running tons of encodes in parallel on multiple servers

[Blue_MiSfit]

Hey guys,

SO... I'm building out a new workflow for doing tons and tons (tens of thousands) of x264 encodes per month, and am looking to automate this as much as possible :)

I'm probably going to use StaxRip, due to its wonderful flexibility. However, this only solves gives me control over one machine.

Is there an app out there for managing a central job queue, so I can send out encoding orders to say 40 other machines?

Rhozet Carbon Coder is an example, but it does a lot more than I need, and uses Mainconcept :@. WTB x264...

~MiSfit

[kieranrk]

You could write a shell/perl/python script to do that pretty trivially.

[Blue_MiSfit]

On Windows?

[Underground78]

On Windows?

Python does exist on Windows (and is pretty cool by the way ! :devil:).

[RunningSkittle]

http://en.wikipedia.org/wiki/BOINC

easy to use and LGPL :)

[RunningSkittle]

or as suggested earlier you could use python to push commands to the systems. probably easier than setting up BOINC

Or you could have the clients pull from a queue or something

[Blue_MiSfit]

Thanks for the suggestions, guys!

I'll be reading up (again) on python. Can anyone suggest some good tutorials etc? I'm looking on docs.python.org but insider tips are always welcome :)

~MiSfit

[kolak]

Hey guys,

SO... I'm building out a new workflow for doing tons and tons (tens of thousands) of x264 encodes per month, and am looking to automate this as much as possible :)

I'm probably going to use StaxRip, due to its wonderful flexibility. However, this only solves gives me control over one machine.

Is there an app out there for managing a central job queue, so I can send out encoding orders to say 40 other machines?

Rhozet Carbon Coder is an example, but it does a lot more than I need, and uses Mainconcept :@. WTB x264...

~MiSfit

Latest Carbon Coder has ability to access external encoder :)

[Blue_MiSfit]

I heard some noise about this but I've never seen it in action.

Are you saying that you've personally used Rhozet to feed video to an external EXE? I.e. acting as Avisynth or something along those lines?

Please tell me more!!!! :)

~MiSfit

[Nazgul]

I'm more comfortable on a DOS command-line, so I set up my own system of dividing the workload. I split the encoding workload between my two machines by putting the source files in a shared directory, then I use looping batch files on each machine to grab the next file out of that directory, generate the .avs file, encode it, and then move the encoded file into another shared directory. I can even drop new source files into the directory while one or both machines are working, and they just get added to the queue. The files I do this with will vary in the parameters used (resizing, framerate conversion, bitrate), so I just append codes to the end of the filenames to indicate which bitrate, resolution, framerate, etc to use. The batch files I use are ugly, but they work. :)

[RunningSkittle]

I'm more comfortable on a DOS command-line, so I set up my own system of dividing the workload. I split the encoding workload between my two machines by putting the source files in a shared directory, then I use looping batch files on each machine to grab the next file out of that directory, generate the .avs file, encode it, and then move the encoded file into another shared directory. I can even drop new source files into the directory while one or both machines are working, and they just get added to the queue. The files I do this with will vary in the parameters used (resizing, framerate conversion, bitrate), so I just append codes to the end of the filenames to indicate which bitrate, resolution, framerate, etc to use. The batch files I use are ugly, but they work. :)

I dont think this would expand well to 10+ machines, how do you ensure that two machines are not assigned the same encode? I mean using nt batch to do this... as i would be inclined to implement a similar (albeit more sophisticated) method in python.

[Blue_MiSfit]

Indeed. I am starting with 40 blade servers, and will probably grow a lot. To facilitate that, I'm building the network side with 10 GbE or InfiniBand connections to a huge scale-out NAS system (2-3 PB initially) - either Isilon or homebrew :)

So yeah, easy scalability is basically my #1 priority. That, and using x264 for absolutely everything ;)

~MiSfit

[kolak]

I heard some noise about this but I've never seen it in action.

Are you saying that you've personally used Rhozet to feed video to an external EXE? I.e. acting as Avisynth or something along those lines?

Please tell me more!!!! :)

~MiSfit

There is possibility to show path to external programme and write a commnad line, so it could work :)

Use Ateme Kyrion software encoder- not very expensive (very good AVC for low bitrates) and support distributed encoding.

[Nazgul]

I dont think this would expand well to 10+ machines, how do you ensure that two machines are not assigned the same encode? I mean using nt batch to do this... as i would be inclined to implement a similar (albeit more sophisticated) method in python.

When each machine grabs a source file to work with, it moves it out of the "source" shared folder. So when another machine goes to grab the next file, it grabs a different one. Because large files can take a while to move out of a folder, I set up a temporary folder for each remote machine on the same drive, so that the source file can be moved quickly out of the sourcedir, then it moves from that temp folder to the remote machine. Adding another machine to the pool would just involve adding another temp folder.

The commands on the remote machine would look something like this:

move x:\queue\nextfile.mkv x:\queue\machinename
move x:\queue\machinename\nextfile.mkv c:\encode

And then when it's done working with that sourcefile:

move c:\encode\nextfile.mkv x:\queue\done

I've had this setup running for a long time now and I've never had a conflict. Granted, I only encode with two computers at a time, but considering you'd need two computers both finishing their encodes at the exact same time and then trying to move the same source file into two different folders on the same drive as the source folder, an operation which is near instantaneous, I'm guessing it would take a lot of machines running in this setup before that became a real concern.

In order to make sure I could add new files into x:\queue at will, I actually have each machine loop through x:\queue\*.*, but I break the loop after the first iteration and then start it again. I do this because I add numbers to the front of the filenames indicating priority, so all the files starting with [1] get worked before [2]*.*, etc. setting up the loop the way I did, if the machines have worked their way down to the [3] files, I can add a new [1] file and it'll cut to the front of the line.

As I said before, my way isn't pretty, but it gets the job done for me. :)

[audyovydeo]

Indeed. I am starting with 40 blade servers, and will probably grow a lot. To facilitate that, I'm building the network side with 10 GbE or InfiniBand connections to a huge scale-out NAS system (2-3 PB initially) - either Isilon or homebrew :)

So yeah, easy scalability is basically my #1 priority. That, and using x264 for absolutely everything ;)

~MiSfit


Again, I'm curious about audio. If you split files, audio demux/encode/remuxing (+resync ?) can become a nighmare.

how have you planned it ?

cheers
a/v

[Blue_MiSfit]

A/V:

Audio has been fairly simple for smaller workflow based on open-source software.

Basically, we demux MPG / TS files with eac3to, and read MOVs directly with QTInput, and process them with SoundOut, encoding an AC3 at 640kbps. The resulting audio streams can then be stored with the transcoded video.

~MiSfit

[kieranrk]

encoding an AC3 at 640kbps.

A little bit OT but what AC3 encoder do you use? The one that comes with SoundOut (libaften) or another one?

[Blue_MiSfit]

For MOVs - yes I use libaften that comes with SoundOut.

For MPG/TS/WAV sources, I use eac3to's version of libaften.

The quality is good, though it doesn't do as nice a job as our Dolby DP600 program optimizer, which does some really deep analysis of dynamics and sets the dialog normalization correctly. The latter is also extremely slow (~ 3x realtime), whereas eac3to+libaften is insanely fast :)

~MiSfit