x264farm: distributed encoding

[Atak_Snajpera]

AMD FX also doesn't have real 8 cores. So it's not HT win, it's AMD fail
AMD FX has 8 integer units and only 4 FPU. In fact it looks more like 4 core cpu with doubled number of integer units.

[nm]

A near lossless setting. No filters. Straight BluRay/H264 to x264 compressed video files. Retains approximately 99.8 per cent visual quality for animation and approximately 95 per cent visual quality for live action. It's fine-tuned for anime essentially.
Can take up to 8 days to compress a legal back-up of one of my average BluRays. I'm running out of space, so spending days getting near lossless quality AND rather small file sizes is worth it.

0.5 fps on an i5 750 sounds like a lot of time wasted for very minimal gain. You are using placebo-level settings which probably give you 1-5 % more compression than --preset slower would, while taking 5 times longer to encode.

Or put in a different way, you can get the same quality by using:

A. --preset placebo settings that give you a 5000 MB file in 48 hours

B. --preset slow/slower settings that give you a 5250 MB file in 10 hours.

Is that 250 MB really worth it? At current HDD prices it costs less than 2 cents!


I seriously suggest spending the money on HDD space instead of CPU power. With the amount of money you'd spend on server hardware, you can get enough HDD space that you can make 1:1 copies of all the Blu-rays you'll ever own!


Well when one whole film, lets use Transformers 2 as an example, can take up 16 days

Ok, you're not using placebo but insane settings?! What did you put in that command line?

If I could get encoding time for film down to 1 day (12 hours per pass, excluding audio)

Use CRF instead of 2-pass. Or if you absolutely need to control the file size (why?), use fast first pass.

Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?

2008 R2 standard edition supports 4 physical CPUs: http://en.wikipedia.org/wiki/Windows_Server_2008_R2#Editions

[Yiu]

Is that 250 MB really worth it? At current HDD prices it costs less than 2 cents!Yeah it's really worth it. Esspecially when I'm already running low with 6TB of data already. I need another 2TB Caviar Black but the prices over in the UK just spiked. Prices went up nearly 50% on HDDs, but SSDs haven't gone up. About $225 USD for 1 WD CB 2TB.

I have a lot of data, and it's going to be expensive when I decide to raid all my drives.

Ok, you're not using placebo but insane settings?! What did you put in that command line?I spent months looking into the x264 settings. Anything to reduce size without reducing quality I put in. If it ever compromised quality, I'd bump something else up that bumped up analysis and vector encoding.

Use CRF instead of 2-pass. Or if you absolutely need to control the file size (why?), use fast first pass.
Well I might as well be setting CQ and setting it to 2-4. But they're really selective settings which I've looked at and used, CQ doesn't give as good visual benefits, in my opinion.

2008 R2 standard edition supports 4 physical CPUs.
Yeah as I thought, my information is kinda behind. Since x64 x264 can run on WinServer on 4 CPUs on just standard I'll have no problem starting out. By the time I can afford more racks I'll be able to afford Ultimate.

EDIT:
I'll probably need HPC/Advanced for the RAM headroom. Unless x264 doesn't actually use that much memory? I'd assume it'd use more to speed up encoding time but I don't know.

[Atak_Snajpera]

did you calculate how much money you waste for electricity? under havy load your pc probably eats about 250-300 W. i would rather buy extra hdd. if you compare average salary from uk and from eastern europe you will quickly realize that prices are not that bad. we from east earn four times less than you from uk. please stop wining.

[Yiu]

did you calculate how much money you waste for electricity? under havy load your pc probably eats about 250-300 W. i would rather buy extra hdd. if you compare average salary from uk and from eastern europe you will quickly realize that prices are not that bad. we from east earn four times less than you from uk. please stop wining.

I'm from Michigan USA. I don't qualify for a lot of free shit other people do over here. Look at average salary all you want, I'm an 19 year old student. And in the end, I'm using less HDD space, buying less HDDs and thus saving the amount of power I use in the long run. (Keep reading it gets better.)
That's why I'm looking to buy a workhorse, so that the 1000+ watts I'm using* can be used to finish the video compression, rather than on my regular desktop system which has disk drives, graphics cards, external sound cards, and a whole lot more than the workhorse will ever have. More energy can go straight to RAM and more processing power, I actually save a lot of money getting a video compressed with a workhorse.

*The main point is, I can only have 1 of the computers on at a time to actually break even. But, if the workhorse can do the work faster than the desktop and either computer would be more or less useless while the CPUs maxed out, then I'm saving time. Swapping the money I save for more time.

It's extremely useful for distributing my own content as well. In the UK your upload speed is limited to 1/8th your download speed on most connections. I only get a 1MB/s download rate (8Mbits) so a 128KB/s upload rate can restrict deadlines and file transfers for me. In all honesty if I had a monster internet connection and more money to throw at HDDs, I'd be far less worried about it but with the rest of the world still on slower internet connections it saves time not only for me but thousands of other people.

[Atak_Snajpera]

Still simple remuxing from HD-DVD/BD/AVCHD is better because:

1) Quality is 100% the same with source
2) Speed is only limited by your HDD/SSD
3) lower electricity bills

and now what you get:
1) Smaller size
2) Quality will never be at 100% with source
3) Ridiculous encoding speed even at good system (0.45 FPS!?!!)
4) A lot higher electricity bills due to have load on PC.

I see no logic here.

[Yiu]

You have noooo idea how much media I have. With minimal quality loss and most BluRays getting shrunk to 10GB including specials and extras, it's worth it.

I would need about 6TB of space just for videos if I did direct copying. Then I've got to factor in my other 4TB of ever expanding media types. As I said, I'm gagging on 6TB of total space already. If I could, I'd get a media server but that would eat up tons of electricity, and all for what? The ability to access all my data at the once? Plus, how much more data will I eat up just over the next 12 months?
I only started digitizing my BluRay and CD and other mediums in February. I've not even been doing this for 12 months and I'm at 6TB.
I was looking to not buy anymore HDDs until next year last month when I bought my Caviar Black but things aren't going as well as I planned in terms of space.

I'm already buying the Desktop, Workhorse and a camera for college all at the same time with my school loan and grant. I have no money to buy any HDDs. I'll need to find a job or get my business up to a decent pay-grade to be able to keep up with my current HDD purchase intervals.

[jasonwc]

Your arguments really don't make any sense. I'm going to prove it by breaking down the cost of your alternatives. First let's start with your hardware choices.

You purchased the $1,000 Core-i7-3960K processor and then overclocked it to 5 Ghz despite the fact that the Core-i7-3930K could have been overclocked to the same rate and would have performed within 1-2% for a savings of $400. And I'm betting you spent at least $300 on the motherboard. So that's $1,300 on two components.

Compare that to a Core i7 2600k ($280 at Microcenter) and a decent 1155 board ($150) for $430. You could then take the extra $870 and put it into drive space. Even at the outrageous prices you can buy FIVE 2 TB drives providing 10 TB of storage for that price (Best Buy has a 2 TB WD20EARX for $170).

Or, if you had reacted to the impending price hike, you could have bought a bunch of drives at $80 or less in October like I did. . .

I have a 4U Norco RPC-4224 file server with 24 hotswap bays (currently using 9 bays - 16 TB useable, 2 TB parity) and I only spent $700 to put together the server. I buy drives as necessary. I acquired two Hitachi 5K3000 2 TB drives for $60/each before the hard drive price surge, and then two Western Digital WD20EARX drives from Best Buy for $80 afterwards. That's an additional 10 TB (9.1 TiB) of storage for $280. Even at around 35-40 GB/movie and accounting for TB --> TiB conversion, you could store 250+ movies on those drives.

But if you really insist on encoding rather than remuxing, you should be getting 5-6 fps in second pass or more with your 5 Ghz hexa-core setup.

You're spending 5-10x more encoding time than if you had used placebo settings for a ~1% efficiency improvement, and in so doing are wasting MANY Kwh of electricity. You're likely burning 300 watts/hr. At 10c/ KwH or more, you would be much better off simply using a lower quality setting and slightly increasing bitrate.

For example, you say that your encodes take 8 days. With a 5 Ghz OC on a 130 TDP processor and a demanding new platform, you have to be using at least 300 watts/hr (remember your PSU is only 80-90% efficient at best).

So that's .3kW x 8 x 24 hours/day = 57.6 KwH

Assume an average price of 10 cents/kwH and you're paying $5.76 to encode a movie. You also are spending money to store that movie. You said each encode was 10 GB. Assume a price of $200/2TB hard drive. That's 10 cents/GB.

So, 10 GB x $0.10/GB = $1.00 for HDD space
$1.00 (HDD space) + $5.76 for electricity = $6.76 total

Now, consider a "placebo" encode which on your system should be at least 5x faster (1.6 days).

.3 kW x 1.6 x 24 = 11.52 KwH ($1.15)

So, let's say by altering your settings from insane to placebo you had to store an extra 1 GB per movie (probably more like 100-250 MB). That's only an extra $0.10.

$1.10 for storage (11 GB) + $1.15 for electricity = $2.25 total

In contrast, if you remuxed you would use perhaps 40 GB MAX. Even at a price of $200 for a 2 TB drive, you're only spending 10 cents/ GB. So, the remux would only cost $4.00 to store (and I'm greatly exaggerating the remux size to be conservative. Most of my remuxes are 20-30 GB - so $2-3).

So, you're looking at $2-4 for a remux depending on whether you include extras and the size of the movie, even with absurd hard drive prices.

Since I bought a bunch of drives before the prices spiked, I'm only paying 3-4 cents/GB so my remuxes cost $0.60-$1.20 to store (20-30 GB MKVs, no extras).

So, here's the cost we get per movie from highest to lowest:

1) Encoding with Insane settings: $6.76/movie
2) Remux with inflated drive prices: $4.00/movie MAX; more realistically $3.00 or less
3) Encoding with Placebo settings: $2.25/movie

BUT those prices assume your current hardware as a sunk cost. Had you purchased a Core i7 2600K system instead you would have had an additional $870 to play with. Say you took the $870 and purchased five of the 2 TB Western Digital drives I suggested. That would provide 9.1 TiB or 9,313 GiB of storage.

So, you could have had a Core i7 2600K system AND 9.1 TiB of additional storage enough for 846 slightly larger encodes or ~250 remuxes.

And with a 2600K overclocked to 5 Ghz, you would lose perhaps 30% in encoding speed. So... you could spend 2 days encoding a movie at placebo settings or 12 hours for equivalent quality with a 10% increase in bitrate.

So, if you went back and made different hardware choices, choice #3 (encoding with placebo settings) would likely cost you about $1/movie (2600k has better performance per watt) while choice #2 (remux) would effectively be FREE for the first year or two (not sure how fast you collect digital media).

The choices you have made make no economic sense. Now, if you just really enjoy having a ridiculously overpowered top-of-the-line hexacore Core i7 system, that's fine. But don't pretend that you're getting "value" from overclocking a $1,000 extreme processor or spending 8 days to encode a movie. You're not. Also, you don't need a 7200 RPM drive for bulk media storage. Get the cheapest 5400 RPM drive that's reliable.

EDIT: I misread your prior post. I thought you actually live in Michigan. It appears you live in Scotland, so I'm not sure what drive prices are there.

[nibus]

If your main problem is hard drive space, may I suggest a temporary solution until hard drive prices go down: BluRay BD-R's. You can get them for less than $1 a disc (25gb) which is by far the cheapest source of space right now. With that much space you could probably remux most of your blurays, especially if you compress the audio to AAC with eac3to.

But seriously, I can't imagine the settings you are using. I just built an i7 2600k and overclocked it to 4.5ghz on air - and I can encode 1080p Placebo setting and get over 2fps on most film sources.

[Adub]

Nice breakdown, jasonwc, very nice breakdown. I actually needed to look at something similar for my own setup.

[GrandAdmiralThrawn]

Hello!

For my own x264 Benchmark project (have already tested Itanium² shared memory cluster machines) I am looking into testing x264 with a distributed computation cluster.

I have set up a Torque/Maui cluster and I will also get access to another cluster based on the IBM LoadLeveler.

My question now is: These systems use PBS style job control systems with MPI support. In the case of my own setup thats either LAM/MPI or OpenMPI, job control by Torque with the Maui scheduler.

Is there any guide on how to run x264farm with a PBS-style job submission system? Is that even possible? How about other systems like the IBM LoadLeveler, where I might not be allowed to access any computation nodes directly? Any information very much appreciated.

Operating systems are CentOS Linux and (afaik) SuSE Linux Enterprise Server. Hardware is i386 and x86_64. No Windows stuff available in that environment.

Thanks for any input.

[Mounir]

I am getting started or rather trying to gather infos for distributed encoding accross computers and apparently there is little to no benefit to my big surprise (!).

It seems to me the current concept is flawed and that it would be better this way (i'm not a programmer mind you):
- say you have two computers
- For 1st pass(x264) devide the video in two segments: cpu1 deal with segment 1, cpu2 segment2
- Merge the stats files to make one (not sure if that's possible)

For the second pass i'm not sure but what if cpu1 encode the beginning of the video and the cpu2 encode start by the end (backwards encoding)
- The point being to make one file only not two and merge them later (= less space used)

Also since the two cpu's try to decode the video at the same time, if you decode a HD stream a good GFX card + DgindexNv would be required though i'm not even sure if dgindexNV could read the video backwards correctly
For lossless (huffyuv, lagarith etc..) sources things would be easier i think

I'm just throwin in ideas...

[Atak_Snajpera]

Distributed encoding is now ultra simple
see here -> http://forum.doom9.org/showthread.php?t=127611

[GrandAdmiralThrawn]

Actually, I solved the Problem in a very simple (albeit suboptimal) way. It's not even necessary to merge stats files.

I just split my source video into as many slices as I had nodes in the cluster to be tested (which was actually a Sun GRID Engine, very similar to OpenPBS/torque, running on OpenSuSE Linux). Then I let every node do its pass 1 and pass 2 on its respective slice, and at the end I just merged the final output video. To do this I used the job array feature of the PBS "qsub" command. Quite simple actually, the job script to achieve this was very minimalistic.

If you do this intelligently (which I did not, but doesn't matter) you should be able to avoid artefacts on the joints by making sure that the parts overlap each other a bit, so you can cut the overlapping parts and join the output video smoothly.

With that method you can use pretty much any Linux/Unix cluster that you can compile x264 (+libav/ffmpeg) on. All you need is shared storage for the cluster nodes and a decent enough job submission / portable batch system.

Performance scaling is linear!

[Yiu]

I found this thread in my bookmarks and laughed.

My system specs are a bit more grounded now, and I've been doing test with new builds of x264 as well as re-optimizing settings and I still get the same quality encoding @ 2~5 FPS.

My signature is updated too.