Any interest in a Hadoop based scalable transcoding system?

[rkuo]

I'm working on open sourcing some transcoding infrastructure I've built leveraging Hadoop as the distributed computing piece. Just wanted to get a feel for the interest level around this?

Currently, it simply distributes files out of a queue to multiple machines ... one per machine. However, it wouldn't take much more work at this point to execute parallel chunked encodes or generate multiple streams from a single input file, etc ... given the right recipes. I mainly leverage Handbrake right now for encoding (aka I doubt my personal workflows that I've tested with are going to generalize that well), but I'm going for a plugin architecture here so that you could really plug in anything.

This wouldn't be intended as tan off the shelf solution for the average consumer. You'd have to be comfortable setting up a Hadoop cluster, which means you have to be technical and have more than a few computers running Linux. So this is more prosumer and/or enterprise class stuff.

The plus side to this is you get a lot of stuff for free from Hadoop, including automatic retries, a very basic web interface for monitoring jobs and tasks, reasonably smart handling of nodes appearing and disappearing, a very active open source community, etc. Stacking on workflow management and better UI's for monitoring task progress is not out of the question either. Distributed computing is a hard problem and building "on the shoulders of giants", so to speak, seems to be the right way to go about this.

[Atak_Snajpera]

Are you going to do something similar to my distributed encoding in ripbot264?

[rkuo]

I suppose in the distributed sense, but I'm more interested in developing something that can be replicated and scaled across general purpose clusters like hadoop. That way it can scale freely and take advantage of advances in that distributed computing technology for free.

Plus this all has to be Linux based. It's not really a problem as you can basically convert any machine that can run VirtualBox into an extra node ... but this is definitely more prosumer than end user.

[Atak_Snajpera]

are you sure that handbreak guarantes frame acurate seeking? I'm asking because chunk mode will be more efficient than file mode. Imagine that you have one movie in 4k resolution and you have to compress it with x265 with preset veryslow. File mode won't give you any speed up only chunk mode.

[rkuo]

I don't plan to do any chunked encoding with Handbrake ... that's just where I started to prove out using Hadoop as the distributed computing framework.

I think what I'm going to do just to get a proof of concept running is use MKV as a base format, let mkvmerge handle the chunking, then use x264 to encode the individual chunks before merging them back together. Will revisit frame serving and supporting more formats after that.

[Atak_Snajpera]

I don't think that chunking via mkvmerge will give 100 percent correct results. Currently the best method is to use avisynth script with ffmpegsource and cut always at key frames. This way you do not have to physically divide source file.

[rkuo]

According to mkvmerge, it cuts at keyframe boundaries for video. I tried a local splitting of the files and it seemed to work.

There are pros and cons to any approach. I'd certainly prefer to not be limited to MKV inputs in the future.