Encoding using networked PC's

[oddball]

I often wonder why there is no encoder out there that makes use of more than 1 PC on a network. I have 3 and soon to be 4 where sharing the load of an encoding job would be ideal. I wonder how hard this could be to implement. x264 or XviD HD video takes a lot of processing time even on a powerful system. It would be great if some of it could be offloaded to more than 1 PC at the same time.

[foxyshadis]

Check out x264farm and ELDER (x264 and xvid) in the AVC forum, as well as almost all of the professional ($$$$) encoders. Someone worked on a multi-PC frontend to HC and Quenc as well, but it didn't get too far iirc.

[oddball]

Thanks! I see there may be a tradeoff in quality but worth a look.

[oddball]

AFAICS Elder only works on a multi processor system not on multiple computers networked. Looking into x264farm now.

[jkcnet]

Vegas 7 will do that, but you need a gige network

[Mug Funky]

i do it manually, but it's not good for 2-pass encodes.

ideally what could be done is a common stats file that all nodes write to (but at different locations... sort of like those multi-threaded download programs), and then the same deal in the second pass, writing to the file at different byte-offsets (and maybe a final pass to clean it up).

or possibly the final pass will write to separate files which get joined later on... that'd be less headachey

[foxyshadis]

ELDER is definitely designed for clustering. In fact it's a lot more scalable that x264farm, which relies on a central avisynth frameserver, although it doesn't need avisynth on peers. (Imho using a specialized data transfer port would be even more scalable than godawful CIFS. Right now you have to put the source files on a network share for ELDER.) I've used both over the wired/wireless here, though obviously the network bottleneck of wireless leaves x264farm lacking.

[Mug Funky]

got gigabit here :) the bottleneck isn't so bad.

and hopefully there'll be a SAN in the future which will be even faster. i wont hold my breath though.

however, i haven't tried elder. i might have to, because right now i'm just splitting the file into x files, and running one on each machine. good if you need a 3 hour movie ready within the hour...

[quake74]

To me x264farm is of no use because I am on wireless. I tried elder but I had problems setting it up and I don't want to install perl on my machines (I already have cygwin and seems to conflict, and I suspect it messed up my env variables). I ended up making my own simple bat file which takes an avs and produces a bunch of bat files (kinda like elder) which I start by hand from the computers I want. Not efficient, but very light. It works well with 1pass crf backups, but I'm thinking about going to 2 pass. The problem I see is that the bitrate distribution is not going to be uniform, so maybe I could do a small crf compression test and then assign the bitrate weighting on the crf test results...

[wonkey_monkey]

I'm working on a little something to do distributed encoding of AVS files which will work like quake74's batch file - not much use for 2-pass encoding but if you're trying to run a complicated script (like Didee's mcbob, which runs at ~1/40th realtime) it would greatly help.

Could this be useful to anyone?

David

[wonkey_monkey]

Well, to be honest I'm a bit surprised that no-one at all was interested in my idea, but I got it working in any case, and have managed to get a 16x speed improvement (on an mcbob script) over using a single computer:

http://www.channeltv.co.uk/avsfactory.png

David

[oddball]

If you can get it working for multiple pass it would be cool.

[wonkey_monkey]

Is it possible to get avs2avi to do two passes and use different codec settings for each pass? Otherwise I'd have to manually keep track of first and second passes - not difficult, but easier if I can send a single command to each PC.

ETA: Although actually, having said that, this method really isn't suited to two passes - each computer is only concerned with a small fragment of the video, so the bits won't be distributed as they would in a full two pass encode. Each fragment will have the same average bitrate as the others.

David

[WinMedia]

Encoding of AVI to WMV using networked PC's is now possible.
It works on plain old 100Mb network and Windows XP/2000 with Winodws Media Encoder 9 installed.

http://www.winmediatech.com/

[oddball]

WMV meh. Give us an x264 version please. :)

[WinMedia]

Here is partial list of organizations using WMV and related technologies.

Cap Gemini Ernst & Young Group kick-started an internal sales campaign using Microsoft Producer to deliver a "talk show" style, rich-media presentation that is viewed online and feels like a live event.

General Mills deployed Windows Media 9 Series and Windows Server 2003 as a major enhancement to Champions TV, its popular intranet video streaming tool. Now, more than 20,000 General Mills employees receive high quality rich-media programming more quickly and efficiently.

IDT Media has successfully deployed a product that delivers live licensed television content to the corporate desktop using Windows Media 9 Series. IDT Media provides secure, high-quality content while reducing deployment costs and allowing a revenue stream from commercial advertisements.

Marathon Ashland Petroleum LLC switched from providing VHS-based employee training to a streaming media–based solution using Windows Media. Both live and on-demand content is served to more than 12,000 employees using unicast and multicast network connections

Meijer, a private grocery and general merchandise retailer, implemented a pilot program using Windows Media to stream training programs to its 80,000 employees.

Mercedes-Benz USA Technical Training group developed information access and e-learning programs using Windows Media. Rich, streaming media–based programs communicate information efficiently to enhance customer satisfaction and provide substantial cost savings in terms of employee travel, salaries, and lost technician productivity.

Misys has deployed a rich-media solution that uses Windows Media and Microsoft Producer to quickly and easily deliver education and training content to more than 1,200 hospitals. The company has dramatically reduced travel costs, tape duplication fees, and technical support calls since deployment.

Morrison & Foerster LLP, one of the world's largest law firms, selected Microsoft Producer and Windows Media to deliver training and communication content to 2,700 desktops. The new system cost-effectively trains legal staff and provides efficient access to professional development courses for attorneys in all 18 offices worldwide.

In order to reduce capital and operational costs and prepare for the move to digital TV, PBS is replacing expensive broadcast equipment with IT building blocks based on the Microsoft .NET Framework and Windows Server system. The project is expected to save PBS up to $100 million U.S. annually in operational costs.

.... and the list goes on and on and on ...


http://www.microsoft.com/windows/windowsmedia/forpros/enterprise/casestudies.aspx

[PatchWorKs]

Well, to be honest I'm a bit surprised that no-one at all was interested in my idea

Dav, is it open source ?

I hope to see a distributed MediaCoder (http://mediacoder.sourceforge.net/) !

:helpful:

[wonkey_monkey]

Dav, is it open source ?

I hope to see a distributed MediaCoder (http://mediacoder.sourceforge.net/) !

:helpful:

You're welcome to the source if you think you could make use of it, but it's PowerBASIC, not C, and it's not very well commented (it also has a couple of hardcoded passwords and things which I'd have to remove before sending it anywhere).

If you're interested, here's how it works...

Each computer in the "factory" (I'm currently using about 25) has AviSynth installed and the necessary codecs (so far I'm using Xvid) and has the telnet service enabled. The manager breaks the input script into chunks (four times the number of available workers) with trim() (placing it either at the end of the script or at a point you must specify with a particular comment), then telnets to each worker, updates its AviSynth plugins, and issues the command (using avs2avi.exe) to process a chunk. When a chunk completes, the result is copied from the C:\ drive to a network drive. When all chunks are complete, the AVI files are joined (still to be done).

One caveat is that the network share can't be on a standard Windows computer - it must be Windows Server or Linux/Samba (which I use). This is because more than 10 clients are connecting to the share.

David

[Wilbert]

@WinMedia, you are very close to breaking rule 5. You have made six posts now and all of them are about NetEncoder.

[WinMedia]

Sorry, i was not aware of such a rule. I will go and look at the FAQ again to see what the rules are.

My only reason for posting about NetEncoder is that I really think the software can help people get their work done faster and just want to get the word out.

I'm not asking for any compensation or registration or licensing fees or anything. Just some feedback, that's all.

But if i've broken any rules of this fine community, i do apologize!

[Inventive Software]

http://forum.doom9.org/forum-rules.htm
You won't find them in the FAQ, but there's links at the top of the page. ;) In your case, rule 5 is applicable.