Questions on distributed encoding...

[Klinky]

I don't really like to wait around with my computer tied up doing video encoding & recently I finally got around to encoding quite a few MPEG2 files to H264. Converting them on a single computer takes a few hours each. This is slow for someone who is low on patience. So, I needed a solution that would easily use all of my computers appropriately. I thought "why not have a program chop each file into smaller parts & have each computer run a server that will encode and return the part when it's done?". So I made a program that did this. After reviewing the .stats file from a local encode vs a chopped encode, there are differences, they don't look huge, but smalls changes over large periods of time could make a difference. I have not done PSNR or SSIM on them as, I haven't had time to really play with it a whole lot plus I am not that great working with x264. Obviously it's probably not going to be as good as doing a local two-pass, though I don't know how "bad" it is either, quite yet. My gut suggests it'd probably better than a single pass, but perhaps I am wrong, maybe my method is like running it through a dumpster. Thus my first questions:

How badly do you feel dividing a video into multiple sections, individually doing a two-pass on each section, then merging each file back into a single file would affect video quality?

What would be ways to mitigate loss of quality using this setup?

This brings up the idea of a second solution. Monitor the .stats file while the local computer does the first pass. In realtime, create subjobs by dividing on I-frames & having the remote computers only handle the 2nd pass. This is more complicated as we're dealing with faking .stats files and faking the .mbtree file as well, but it appears it is possible. I have parts of this working, though it is much more complex than the simple chopping method. It would appear that this method might provide identical quality to a local two-pass, if x264 follows the stats file on the remote computer and the subjob has the exact frames it's looking for, I can't think of why this wouldn't work. Though some questions arise again:

How would this not work? &

Would this be scalable?

I ask about scalability because obviously subjobs could only be sent out as quickly as the local computer could produce first pass frames. Where as the chop method could scale to total number of frames(though that wouldn't be a good idea). Avoiding a slow first pass, my computer is doing about 200fps, 2nd pass can do about 50fps on an equally powerful computer. So it would top out at four computers. That might be of benefit. However if people use a slow first pass, that isn't a whole lot faster than a 2nd pass, then it'll only scale to a second computer of equal value.

Thoughts or ideas welcome...

[linyx]

How badly do you feel dividing a video into multiple sections, individually doing a two-pass on each section, then merging each file back into a single file would affect video quality?
The more sections the worse it will be (more like, it will hurt x264's bit-distribution efficiency), I would strongly recommend using CRF mode for a project like that.

It should be relatively easy to scale in CRF mode, as for 2-pass, that sounds near impossible without a lot of effort.

A better solution though would be to encode a single file on each computer if that were possibly.

[nm]

Have you looked at previous projects such as x264farm (http://omion.dyndns.org/x264farm/x264farm.html) (also here at Doom9: 1 (http://forum.doom9.org/showthread.php?t=117889), 2 (http://forum.doom9.org/showthread.php?t=136053)) and ELDER (http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER)? They might answer some of your questions.

[Selur]

A better solution though would be to encode a single file on each computer if that were possibly.
Agree. (+ is easier to implement)

Little side note: if you reencode the audio and do some filtering (e.g. autogain, normalization), splitting the source and then simply rejoining could be a problem.

Some time back (2+ years) I helped setting up something like this for a company I worked for.

System looked like this:
A A single distributor (per Fileserver cluster) which checks the Fileservers for space and runs a apache server for the InputClients to connect to.
X Fileservers which each hold a Encoding server, which can send encoding jobs to clients.
Y EncodingClients (Xubuntu on a USB Stick) which can collect the available Encoding servers and ask them for jobs to process. You can configure each client to process up to Z jobs parallel, etc. through a .ini entry, so setting up a new client is just copying the usb image to a usb stick and setting some config parameters.
W InputClients (some computer with a web browser) where someone can upload a source with some additional infos (needed resolution, formats, tagging, etc.) to a specific file server (you connect to the distributor which tells you where to upload the stuff to)
The whole thing wasn't really hard to implement, the trickier parts were a. the Clients (Input client needs to analyze the source and create the processing jobs; EncodingClients which need to process the jobs - consisting of a bunch of subjobs), b. get the whole EncodingClients-On-An-USB-Stick thing working and automate it as far as possible and c. finding a good timing for a client to grab the source material from the fileserver so that on one hand no client had to wait for to long and on the other hand the network connection wasn't to slow when the number of clients rose.
-> took us around 5 days to get the whole thing working with two men
(one web developer for the php and networking stuff, one encoding guy for the analyzing, job creation etc.; 2 days planning, 2 days implementing the main thing, 1 day getting it all working together and fiddling with linux-on-a-stick)

Cu Selur

Ps.: After some more thought about it: the main challenge was getting the analysis working with all their input formats.

[Klinky]

The more sections the worse it will be (more like, it will hurt x264's bit-distribution efficiency),

This is what I thought would be an issue.
I would strongly recommend using CRF mode for a project like that.

CRF is an idea, but then file size restrictions go out the window and if you miss your target you have to rencode with a higher/lower settings.


A better solution though would be to encode a single file on each computer if that were possibly.


I've got some older computers that are on the slow side (10fps for x264). They might save 10 - 15 minutes on a job while working on a small chunk from a distributed encode, however if they were to do it all themselves it'd probably take goodness knows how long. Babysitting the computer isn't very fun, but my program can adapt to distributing single jobs instead of splitting them. However, if it takes 24hrs for the computer to churn through one video & I have to set reminders for when it's done so I can merge audio, name it, etc... It seems tedious. Also if the power goes out or I need to reboot the machine or do something else that is processor intensive with it, bye bye progress.


Have you looked at previous projects such as x264farm (also here at Doom9: 1, 2) and ELDER? They might answer some of your questions.

Yes I saw those, but they both seem discontinued. I think I read ELDER was broken with newer versions of x264 + their website seems to not be functional to even ready about configuring it & x264farm seems to rely on sending out uncompressed video over the network, which would be a bottleneck at some point.

Currently I have it setup where servers mount the clients drive and then they are just reading compressed data and writing back compressed data. Using the extra computers I increased fps from 50fps to 80fps, only using a few MB/sec data over the network. If there was ever a point where the network/storage became a bottleneck, then maybe a multicast file transfer so the servers have a local copy could be implemented.

Ultimately perhaps someone would need to know more about the internals of x264 & how to split the tasks up on a lower level, though there are problems with that again as you'd probably be sending uncompressed video data which would eventually bottleneck again. Also internal operations are probably sensitive to latency.

For now I guess the "true" two-pass idea will be put on hold and perhaps I'll tweak around with crf modes or do some comparisons to see how big the difference is between true two-pass and chunked two-pass.

[nm]

I've got some older computers that are on the slow side (10fps for x264). They might save 10 - 15 minutes on a job while working on a small chunk from a distributed encode, however if they were to do it all themselves it'd probably take goodness knows how long. Babysitting the computer isn't very fun, but my program can adapt to distributing single jobs instead of splitting them. However, if it takes 24hrs for the computer to churn through one video & I have to set reminders for when it's done so I can merge audio, name it, etc... It seems tedious.
You could make a frontend/script that allows you to set the encode up in the beginning and the system would handle all those tasks automatically. There could also be a job queue where encodes get distributed to free machines once they have finished with the previous job.

Also if the power goes out or I need to reboot the machine or do something else that is processor intensive with it, bye bye progress.
Yep, that's a downside. If you need a single encode to finish as fast as possible or minimize the penalty of stopping in the middle of an encode, smaller work-units is the way to go.


Have you looked at previous projects such as x264farm (also here at Doom9: 1, 2) and ELDER? They might answer some of your questions.
Yes I saw those, but they both seem discontinued.
Yes, but you can pick up ideas from them. x264farm is pretty though-out overall and most of the stuff you were wondering about has been implemented and tested there.

x264farm seems to rely on sending out uncompressed video over the network, which would be a bottleneck at some point.
That's the default, but it can be configured to use compressed inputs. See "Agent-based encoding" at the homepage.

[popper]

why don't you just make a minimal virtual-Box image with all your distributed encoding apps and support scripts etc included.

hell you could even make a slax live-CD/usb2/Vbox with your required app modules with its included generic PXE boot option to boot any other lan connected pc http://www.youtube.com/watch?v=KRi5J8NkzH0&feature=related .

http://www.slax.org/modules.php

build your New working distributed encoding framework around that, and be up and running on any number of LAN booted [Virtualbox]pc's within minutes.

make something that actually works with a current and future x264 codebase rather than those old apps/scripts above that dont....

OC adding new internal patches to x264 so it could in the future message pass data and many client/server type messages to other x264 available on the LAN would be a better long term option if you were able to do that it would seem