I'm creating a custom render farm system, anyone interested in contributing?

[docmattman]

I'm building a custom render farm system. The basic idea is that it will be a collection of machines that can take audio/video and distribute the encoding across the machines. I know this has been done already. However, I've investigated several other systems. I'm not interested in any commercial software/systems, and the free ones that I've found leave a lot to be desired. Anyway, if I can get it to a point that I'm happy with, I'd love to release it as an open source system and maybe even get some other people involved to help with it (since many other folks on here know a LOT more than I do about media encoding).

I have the system started already. I've setup some virtual machines locally to use as testing platforms. So far, they all run Ubuntu Server 64bit using FFMPEG as the main encoding system. I have one machine setup as the "brain", which handles all incoming requests, then stores them in a queue and distributes the jobs out to the "worker nodes". The brain machine doesn't ever do any of the media encoding, it's just there to do all the intelligent logic and queueing while the "dumb" workers do all the encoding work and nothing else.

I've got the basics in place already. The brain machine can accept incoming jobs and distribute them out to all available machines. It's pretty neat, I just finished throwing 150 jobs at it and they all went through without any hiccups.

A few caveats... I'm a Windows user. My experience with Linux systems isn't super extensive. I've just barely been able to accomplish the start of this system after many hours of researching lots of different things. Also, I'm a web programmer mainly, so much of the code written for this system is done in PHP. The brain machine is setup as a LAMP machine and all data is stored in a MySQL database. Since I'm mostly a web programming, I have built a web "monitor" for the system. It list all the jobs from the queue and updates their progress in realtime using AJAX commands to query the brain database.

If anyone is interested in getting on board and building this into something better than it is, let me know. I'd love to have some help with this, as lots of it is outside my realm of expertise. The main issue that I see for bringing on help right now is that I have no idea how to package this system up into an installable product. The best I can do for right now is upload my virtual machine hard disks to one of my web sites and have helpers grab it that way. Any interested helpers would need to download two VirtualBox hard drives (one for the brain and the other for the worker machines). The drive images would probably be somewhere around 2-3gigs each.

Here are a couple of the main goals I would love this system to accomplish that I haven't been able to do yet.

1. Be platform independent. It would be great to have installers for Windows, Mac, and Linux machines so it can run on any hardware.

2. Be able to split up large media files and distribute pieces of it to multiple worker machines for parallel encoding of a single file, then join all the pieces back together after all then chunks are encoded separately.

3. Get a lot more advanced and efficient with FFMPEG commands and options. So far, I've just been doing basic H264 encoding jobs that are suitable for web playback. However, I'm no expert in all the options available to optimize encodings, so I'd need assistance there.

4. Release the system as free open source to invite more contributers and users. I spent a lot of time looking for a suitable solution to my needs and it would be nice to provide that solution as a free product to help others too.

5. Host it on GitHub for easier source management and contributions.


There are probably several more goals that I could think of, but this is getting too long already. Anyway, if anyone is interested in possibly joining my effort here you are more than welcome. I've got a good start already but it's still got a long way to go. Also, if people are interested, maybe I'll create a screencast video of the system in action. Let me know if anyone is interested...

[Atak_Snajpera]

looks like ripbot264 distributed encoding mode will have competition ;)

[docmattman]

looks like ripbot264 distributed encoding mode will have competition ;)

That looks like a pretty nice piece of software. I don't think I ever came across it when I was researching existing solutions. Can it accept incoming jobs through the command line or API? Or do you have to use the GUI to create them?

The main initial goal of my system was to be able to run along side other servers and run submitted jobs without a GUI. My main use is to accept user uploaded videos through a website, then have them send to the render farm for encoding for web playback. Kinda like YouTube but on a MUCH smaller scale.

[Atak_Snajpera]

not now but i could implement some basic command line functionality in next version

[docmattman]

I added a few screenshots of the cluster monitor system to the first post. It's just the basics for now, but hopefully I'll have time to keep improving it.

[smok3]

My main use is to accept user uploaded videos through a website

how is that working for huge files? (i have been thinkering with porting my ffdrop scripts into some linux vbox engine, but not sure how to go about that.)

[docmattman]

It works pretty well so far. I've sent some full length HD movie files through the encoding cluster and it worked just fine. It took a while to do the encoding obviously, but it works well.

As far as accepting large uploaded files on the website... The server just needs to be configured to accept large files. But after that, it's just like any other upload except that it takes longer to transfer.

[smok3]

The server just needs to be configured to accept large files.
How big did you test? (really interested in some real-world numbers)

[docmattman]

How big did you test? (really interested in some real-world numbers)

I still have the system setup in virutal machines. Right now I'm running a quad-core Windows 7 system. I have 5 VirtualBox machines (1 for the Brain and 4 worker nodes). So all the encoding processes for the workers are all sharing the same processor as the host machine, so I can't give any really good benchmarks for this type of situation.

However, I do plan on moving these virtual machines to physical hardware in the next week or so. I'll report back on some numbers when I do that.

As far as what I've tested so far... I've thrown many various size "small" videos at it. These are mostly 720p videos somewhere between 1-5 mins in length. I've tested many of these, since that will be typical for my personal use of the system. But I've also thrown a full-length movie (1080p mp4 file about an hour and a half in length, around 4.5GB). The encodings were successfully for all the different versions I requested (ipod, dvd, 480p, 720p, 1080p). The encodings took longer to complete on the virtual machines sharing the same hardware then they will on separate physical hardware. When I have it all setup on real hardware next week, I'll run it through and post encoding times.

Times and benchmarks are also a bit tough to take too seriously because of the HUGE number of options and filters that can be put into the encoding commands. Right now, my encoding profiles for the versions mentioned above are VERY basic. They could probably use some tweaking for optimizing time and file size. I'm still researching to find the best "generic" encoding settings I can for these jobs.

Anyway, long story short, I'll post some real number after next week when I have real hardware to run it. Sorry I can't provide much more information yet.

[bnshrdr]

I have a general question about distributed encoding. Is the strategy to give each "worker" n frames to encode, and then the brain combines them all together? If so, is there a significant amount of quality loss when encoding many different lengths of the video separately? If you know where to look, can you see where the different clips were spliced together?

[Atak_Snajpera]

my tests show that quality drop is invisible for humans

[docmattman]

Splitting a single file and using multiple machines to work on pieces of it is not something this system is able to handle yet. I don't quite have enough knowledge about how/where to split files for the workers to encode. This is something I plan on putting in, but until I do more research and find some answers about how/where to split files, it is not supported.

The system I have created is mostly designed to handle MANY files at a time. The general use-case for my needs will be that someone will upload 50-100 video clips and they will need several version of all of those clips (ipod, dvd, 480p, 720p, 1080p). So, if someone uploads 100 clips and needs 5 versions of them all, it will be 500 total encoding jobs that need to be done. This would take quite a while for a single machine to handle. So this system is designed to distribute the jobs to the various machines. It's also "plug and play" friendly. So, all you would need to do is install the system on a new machine and plug it in and it will be available for encoding. Scaling is a big concern for me, since my needs can require several hundred encoding jobs coming in at once. Theoretically you could install the worker software on 100+ machines, to make this package of jobs run through much quicker.