x264farm: distributed encoding

[legoman666]

Originally Posted by SpAwN_gUy

hehe.. use first of my tutorials in my signature.

currently i'm stuggling on the field of relative-paths in .d2v, so we had no need to modify it by hands... (you can join me here )
!!!! UPD: Nevermind. this feature will be in the next beta release of DGIndex .... i assume DGMPGDec 1.5.0 beta 3

well,... not exactly this one.. it will not solve slow starts, but it WILL solve Overall ControllerPC's CPU load. (you will not have to decode and compress video for each agent)
but you can look to "--firstavs and --fastavs" params

thanks, Ill give it a look-see when I get home.

[Finar]

Hi!

My idea is to make my home server (100% uptime) coding my content all the time, and distributing it with other computers, when they are online. But I'm not sure if it is possible to shutdown Client PCs safely while main encoding still going... What will happen when x264Farm Server will suddenly lose the connection with one of the Clients?

Thanks!

[SpAwN_gUy]

Originally Posted by Finar

Hi!

My idea is to make my home server (100% uptime) coding my content all the time, and distributing it with other computers, when they are online. But I'm not sure if it is possible to shutdown Client PCs safely while main encoding still going... What will happen when x264Farm Server will suddenly lose the connection with one of the Clients?

simply nothing bad .... even if you shutdown controller (after restart encoding continues just before crush)

well.. you can also start agent on controllerPC.. if there will be any agents available it will use them.. otherwise.. it will encode on it's own...

[omion]

Originally Posted by Finar

Hi!

My idea is to make my home server (100% uptime) coding my content all the time, and distributing it with other computers, when they are online. But I'm not sure if it is possible to shutdown Client PCs safely while main encoding still going... What will happen when x264Farm Server will suddenly lose the connection with one of the Clients?

Thanks!

Yeah, that's no problem. If the controller sees the agent disconnect it will simply give out that job to another agent. When I first got my Mac Mini it would crash every hour or so when encoding, so I made sure the encode would survive! (turned out I had some bad RAM - replaced it and all was good)

The whole setup is designed to be able to withstand all sorts of network-related issues, but you will lose all the progress you had on whatever you disconnected. Disconnecting the controller will throw out all the frames that the agents are currently set to encode, which on the first pass will be a fairly sizable chunk of the frames.


There is one issue that still may exist... if the controller does not get the signal that the agent has disconnected, it may wait forever to receive the job even if you re-connect the network. The only way around that is to guess when the connection has died, which I have done in the areas where this bug is most likely. I am fairly certain, however, that this issue is still around somewhere.

[Inventive Software]

Hey hey! Spent the last couple of hours catching up on a lost relic I forgot about after moving back from uni for the summer.

Questions.

1) What network speed is ideally required? I've read that 1 Gbit kicks ass (to quote both Spawn_guy and omion ), but I'm limited in my home setup to 100 Mbit (laptop is 100 Mbit, desktop has 100 Mbit card). Will that suffice?

2) I see there are 2 modes of encoding: controller based and agent based. Which is best for just 2 computers with XP on them? One's a laptop that's dual-core and the fastest computer I have. The other's a Celeron 800 that's quite slow, but reasonably steady.

Another note: I will inquire as to whether I can use a networked room at my university to test this out. It would be very interesting to get this working at uni.

[SpAwN_gUy]

Originally Posted by Inventive Software

1) What network speed is ideally required? I've read that 1 Gbit kicks ass (to quote both Spawn_guy and omion ), but I'm limited in my home setup to 100 Mbit (laptop is 100 Mbit, desktop has 100 Mbit card). Will that suffice?

yep.. it REALLY KICKS ASS... but as i've found, our new Taiwan-NoName-1Gb-SWiTCH has gone.. 'cause it was causing lots of troubles and 1Gb-Cisco costs lots of money.. so now i'm back to 100Mb..
well. it keeps spinnin' ... and everything goes wery nice.. i miss the spees i had that time, but i think i can handle it with more new dualCore agents

Originally Posted by Inventive Software

2) I see there are 2 modes of encoding: controller based and agent based. Which is best for just 2 computers with XP on them? One's a laptop that's dual-core and the fastest computer I have. The other's a Celeron 800 that's quite slow, but reasonably steady.

hm.. i really dunno.. for just two PCs.. and 100Mb.. maybe in this condition you should better just use your dualCore PC.. as standalone..
but if there will be more agents then it's up to you what to choose ... i prefer agent-based and no agent on the controllerPC ..

[SpAwN_gUy]

Originally Posted by omion

There is one issue that still may exist... if the controller does not get the signal that the agent has disconnected, it may wait forever to receive the job even if you re-connect the network. The only way around that is to guess when the connection has died, which I have done in the areas where this bug is most likely. I am fairly certain, however, that this issue is still around somewhere.

the "Dead?" issue is still on the second-pass if agent suddenly dies.. encode cannot be finished automatically... 'cause "Dead?" sign is still waiting for something..

[legoman666]

Originally Posted by SpAwN_gUy

yep.. it REALLY KICKS ASS... but as i've found, our new Taiwan-NoName-1Gb-SWiTCH has gone.. 'cause it was causing lots of troubles and 1Gb-Cisco costs lots of money.. so now i'm back to 100Mb..
well. it keeps spinnin' ... and everything goes wery nice.. i miss the spees i had that time, but i think i can handle it with more new dualCore agents

hm.. i really dunno.. for just two PCs.. and 100Mb.. maybe in this condition you should better just use your dualCore PC.. as standalone..
but if there will be more agents then it's up to you what to choose ... i prefer agent-based and no agent on the controllerPC ..

It also depends on the source and what quality you're encoding at. If you have a dvd source and are encoding to something like 1500kbps, (depending on the speed of the clients) they'll probably use ~40% of a 100mb connection each. on the other hand, if your source is 1080p content and your converting to 720p @ high quality, each client will use less, since they're converting so much slower.

Example. converting 1080i content to 720p with rediculous settings.
controller is AMD x2 @ 2.8ghz
client 1 is Intel C2D @ 2.0ghz
client 2 is AMD x2 @ 2.4ghz
client 3 is AMD 3000+ @ 1.8ghz
client 4 is Intel CD @ 1.8ghz
client 5 is dual p3 500mhz

Even though I had 4 reasonably fast clients and 1 slow one, I was only using 10-12% of my gigabit connection during the second pass since it was encoding so slowly (12fps total or so)

On the other hand, I converted a dvd to 1400kbps (with medium quality) using the same clients, and it was using >30% of the gigabit connection during the second pass.

[SpAwN_gUy]

'kay.. i'm not doing nothing...

i just decided to build some wierd and heavily patched x264 build..

this one is r680 of Cef's sources + some new pathces from Dark Shikari

Cef (http://mirror05.x264.nl/Cef/)
-----
is the regular one, with msvc/win64 asm fixes, and the 2 patches aq and thread pool.
-----

Dark Shikari (http://forum.doom9.org/member.php?u=83421)
-----
The SATD patch is "worthwhile" but only with --me esa, as the other modes are too slow, ironically.
The --me-prepass patch is overall generally useful.
Also note the --subme 7 and DIA patches, having no negatives at all, are perfectly fine no matter what.
-----

x264_2pass_vbv.diff
x264_faster-dia.diff
x264_me-prepass.diff
x264_satd_fpel.11.diff
x264_subme7_vc8.diff

both x64 and x86 versions built with VS2005 are here: http://rapidshare.com/files/58338767...0-MSVS2005.rar

[cogman]

haven't read the entire thread, so sorry if any of these questions have been answered.

What is this written in? I see the ml tag but I don't know what language it is. (reading some of the code has not made it any clearer).

The AVS script is processed on the controller, correct? What would constitute a speed increase? If you are using a slow network but encoding for a handheld device (IE a PDA) would it be best to resize then distribute? On the other hand, if you are just applying filters but keeping the default resolutions (IE re-encoding a dvd) would it be possible to send the raw video and just filter on the other end? How about when converting an AVI? would it be better to break it apart (is that possible?) and send over the pieces to be processed to save bandwidth?

If any of this is already done, sorry for asking .

One last question. Would it be possible to use a fast compression algorithm (LZMA possibly) on outgoing video to save a bit more on bandwidth? Provided it is not going to provide huge benifits, but would it bog things down?


Thanks, I'm pretty interested in this project and like the Idea.

[akupenguin]

Originally Posted by cogman

Would it be possible to use a fast compression algorithm (LZMA possibly) on outgoing video to save a bit more on bandwidth? Provided it is not going to provide huge benifits, but would it bog things down?

Already supported: <compression>
But LZMA is really slow when you're dealing with the amount of data involved in raw video: 2MB/s on my core2 e6600, which would be 4fps @ 480p. x264farm uses huffyuv. (Ok, I don't know if it's bitstream compatible with huffyuv, but the same algorithms.)

[SpAwN_gUy]

Originally Posted by cogman

haven't read the entire thread, so sorry if any of these questions have been answered.

Shame on you ... most of answers are written in the read-me.html, provided with package..

but okay.. once again..

Originally Posted by cogman

What is this written in? I see the ml tag but I don't know what language it is. (reading some of the code has not made it any clearer).

(first post) - OCaml ... it is easy to "read" the code to me, but i can't code on(in?) it.. devenv could be founded on sourceforge.net .

Originally Posted by cogman

The AVS script is processed on the controller, correct? What would constitute a speed increase? If you are using a slow network but encoding for a handheld device (IE a PDA) would it be best to resize then distribute? On the other hand, if you are just applying filters but keeping the default resolutions (IE re-encoding a dvd) would it be possible to send the raw video and just filter on the other end?

Yes and NO.. Yes - for controller-based encoding.. NO - for agent-base encoding.. (try the readme)

Originally Posted by cogman

How about when converting an AVI? would it be better to break it apart (is that possible?) and send over the pieces to be processed to save bandwidth?

it does not matter in agent based.. well.. there is no need in doing that..
but for "splitting" and "reEncoding" the "some king of avi" you need (you are talking about agent-based encoding) the video-decompression decoder to be installed on evry PC.. or you "don't" when using RAW input Video (i'm currently doing that,..) or using "controller-based"

Originally Posted by cogman

One last question. Would it be possible to use a fast compression algorithm (LZMA possibly) on outgoing video to save a bit more on bandwidth? Provided it is not going to provide huge benifits, but would it bog things down?

yes, as aku said there is a switch in controller's config.xml to enable huffyuv compression (you can even spcify on which pass (first or second or both) to use it)..

[SpAwN_gUy]

BTW guys... if anyone interested.. the new beta of DGIndex is out...
and Yes.. it has "Relative paths" feature ...
DGIndex 1.5.0 beta 3

[squid_80]

Originally Posted by SpAwN_gUy

BTW guys... if anyone interested.. the new beta of DGIndex is out...
and Yes.. it has "Relative paths" feature ...
DGIndex 1.5.0 beta 3

You know, if you map the drive to the same letter on each pc you could just use absolute paths.

[rhodges]

Controller: Vista 64bit

Running the controller and 2 agents.

Agents:
X264: 32bit
AviSynth: 32bit

The config for the agent has agentpipe set to 0, but I'm still seeing avs2yuv.exe running, and it typically chews up 300+ MB of memory.

Looking at it, it's never consuming any CPU, just a ton of memory.

I searched the dump files for "avs2yuv" but I didn't see it.

What's using it and how can I stop it? (That is, if I can.)

Thanks!

[omion]

Originally Posted by akupenguin

Already supported: <compression>
But LZMA is really slow when you're dealing with the amount of data involved in raw video: 2MB/s on my core2 e6600, which would be 4fps @ 480p. x264farm uses huffyuv. (Ok, I don't know if it's bitstream compatible with huffyuv, but the same algorithms.)

It's very similar, but I precomputed the code tables. It turns out, with Paeth pre-processing, all the movies I tested (even anime) had almost the same symbol probabilities. It gets better compression than standard huffyuv since I don't have to send the code tables.


@rhodges:
avs2yuv is used by the controller. If you have <agentpipe> set to 0, the controller uses avs2yuv to get the raw data which is then sent to the agent.
Also, the controller uses it in the first pass to find good split points. This is probably the one that you're seeing. It only does things when the controller needs more splits, which isn't too often.

If avs2yuv is taking up too much memory, add "SetMemoryMax(64)" to the top of you AVS file to limit the frame cache to 64MB (I've noticed that avs2yuv generally uses twice this number in total memory).
If that line makes the whole encode go slower then you can make a copy of the AVS file, add "SetMemoryMax(64)" to the top of the copy, then use "--fastavs copyname.avs" in the controller settings. This will make the splitter use a memory-friendly version of the AVS script, while still using the regular one for the actual encoding.


@all:
As always, I've had way too much stuff to do lately, but I should have the next version of x264farm out very soon (like tomorrow). Also, I was going to release r680 of the controller-friendly x264, but it looks like Cef's builds that SpAwN_gUy posted work fine, so I'm just going to save time and use his builds (And he's got a 64-bit one. Yay!)

[SpAwN_gUy]

Originally Posted by squid_80

You know, if you map the drive to the same letter on each pc you could just use absolute paths.

hehe.. i did so.. but.. it is sometimes not possible.. and it give more trouble to people on those PCs where agents are strted..
i've wrote a simple GUI for agent, that starts agent, restarts it when agent exits, and minimizes to tray. and stores all the info and configs on my share..
and encode goes jst fine when bases are set to Network shares...

so.. Relativity - is a bery nice thing to have

[SpAwN_gUy]

Originally Posted by omion

Also, I was going to release r680 of the controller-friendly x264, but it looks like Cef's builds that SpAwN_gUy posted work fine, so I'm just going to save time and use his builds (And he's got a 64-bit one. Yay!)

yes.. i've also mastered x264 compilation .. thanks to Cef.. it's a cool thing to know ... like you can easyly patch sources for you needs.. ...

btw.. my build is not the exact r680 .. and not the exact "dark" build.. it's a mix.. and i do have both 64 and 32... so newermind..

looking forward to a new version of x264farm

[Inventive Software]

@omion: a quick compression algorithm I could suggest is QuickLZ. Not sure how this compares to HuffYUV, but I can't see any harm in comparing, if you have the time to of course.

[omion]

OK, after 3 months, 1.14 is finally out. It has most of the patches which have been discussed recently.

SpAwN_gUy's "Dead?" issue should be fixed.
Also, the spacing issue legoman666 had with x264 should be fixed too.
More details in the changelog.

I was going to have this out last night, but I accidentally did something weird with my version-control system which merged the past two revisions instead of my current working version. I think I fixed it, but there may still be some odd issues that sneaked in.

The agent was changed, but only to fix the issue with x264 having spaces. If that issue does not affect you, you don't need to replace the agent.

Have fun!