x264farm-sp - single-pass distributed encoding

[omion]

I just finished up the single-pass version of x264farm - a distributed video encoder. Since the new single-pass version uses completely different code than the original (and therefore probably has its own bugs), I'd like to start up a thread for just the single-pass version (which I cleverly call x264farm-sp :rolleyes:)

Download version 1.07:
Windows binary (http://omion.dyndns.org/x264farm-sp/x264farm-sp-1.07_win32.rar)
Source (http://omion.dyndns.org/x264farm-sp/x264farm-sp-1.07_src.zip) (agent compiles nicely on Linux and OS-X -- all you need is OCaml and Xml-Light (http://tech.motion-twin.com/xmllight.html), then run "make" in the agent/ directory)


What it does:
Distributes a single-pass x264 encode across multiple computers on a network. Every encoding computer runs an "agent" program which controls the operation of x264 on that computer. The computers are linked together by a "controller" program running on one of them. The controller controls the settings of the current encode.

How it's different from x264farm:

Only allows 1-pass encoding
Handles agents which exit silently
Much easier to set up
Slightly smaller executable size and processor overhead


How to run:
Basic setup should be fairly simple:
Start up the agent on each computer you want encoding
Start up the controller with the encoding parameters, like this:
controller -i input.avs -o output.mkv -x "1-pass x264 parameters"
Assuming everything works, all the computers should find each other on the network and encode the video together.
If it doesn't work, then you may need to change some settings around. See x264farm-sp.html (http://omion.dyndns.org/x264farm-sp/x264farm-sp.html) for the details.

Bugs discovered since last release:

Nothing yet...
Both the controller and agent will output a log to a file called (by default) "out-dump.txt". Linking to that file (or the relevant portion of it) when making a bug report is very helpful.
Other known limitations are posted in the "current issues" section of the HTML file (http://omion.dyndns.org/x264farm-sp/x264farm-sp.html) included in the release.

The changelog may be found here (http://omion.dyndns.org/x264farm-sp/x264farm-sp.html#changelog).

[audyovydeo]

Applause !
With special mention for the original name. You're a natural branding manager, I can tell ;-)

Jokes apart, I'll set aside some time this weekend to have a go.
Does it still need a "special" x264 executable or takes any off of svn ?

thanks for the effort
cheers
audyovydeo

[omion]

Applause !
With special mention for the original name. You're a natural branding manager, I can tell ;-)
Suggestions are welcome :cool:

Jokes apart, I'll set aside some time this weekend to have a go.
Does it still need a "special" x264 executable or takes any off of svn ?

thanks for the effort
cheers
audyovydeo
Thanks for reminding me - I think there may still be a problem in Windows for non-MSVS builds of x264. I actually forgot all about that. If you use Windows, try to get a build that was compiled with MSVS (not cygwin or minGW). Other than that, anything will work OK.

I'll get an update out sometime that will work around the issue.

(Last time I checked, Cef's builds (http://mirror05.x264.nl/Cef/) work, although they aren't updated too fast)


One other issue that I remembered is that this program outputs Matroska version 2 files, which I suppose could have problems running on older players. If they do, you can put it through MMG (http://www.bunkus.org/videotools/mkvtoolnix/index.html) and output a version 1 file. In reality, though, everybody is going to re-mux it anyway, so it shouldn't be an issue.

[SpAwN_gUy]

Cef told me, that he has some troubles with sources 700+ ... and he had made a non-patched build...

might be, that i'll re-open my comilations.. 'cause for now i've got a fresh set-up, so no YASM in my VS...

i'm not sooo much an single-pass fan,.. but.. if i'll have time..

[audyovydeo]

Suggestions are welcome :cool:



_ _ _____ __ _ _
(_) | | / /__ \ / /| || |
__ _ _ __ _ __| | / / ) |/ /_| || |_
/ _` | '__| |/ _` | / / / /| '_ \__ _|
| (_| | | | | (_| |/ / / /_| (_) | | |
\__, |_| |_|\__,_/_/ |____|\___/ |_|
__/ |
|___/



???

cheers
audyovydeo

[omion]

To all my adoring fans: (hello? is anybody using this? :p)

I released 1.01, which will dynamically determine if x264 can't support normal controller-based encoding and will work around it if necessary.

The actual problem is that most x264 builds for Windows open the input stream in text mode. In text mode, ASCII character 26 is encountered the program will close the file. When piping binary data into x264 (like I do with controller-based encoding) this character comes up quite a lot. The result is that almost all of the encodes are cut short and cannot progress past a certain frame.

In 1.01 I added a test that will check x264 for this issue. If it has the problem, then any controller-based encodes will be scanned for the character 26. Wherever it is found, it will be replaced with either 25 or 27, determined by a random number generator. This is fairly CPU-intensive and slightly degrades quality, but it allows the encode to actually finish.

Most of my other changes are internal, code cleanup-type things.

@audyovydeo:
Grid/264 sounds great, and I wish I had thought of it when I released the 2-pass x264farm... but I think any name for this should convey the one-pass-ness of the program, which is difficult to turn into a catchy name.

[RickA]

Thanks a bunch for your work on this omion. I am really interested in network encoding potential. Consider me another member amongst your growing, adoring fan base. :-)


Cheers,
Rick

[smok3]

i will test this as soon as i get back to some multi-comp environment, cheers.

from http://omion.dyndns.org/x264farm-sp/x264farm-sp.html
Access to x264
why? wouldnt it be better to get the file from controler? (so i don't have to manually update all the nodes)

[omion]

why? wouldnt it be better to get the file from controler? (so i don't have to manually update all the nodes)

That is a possibility, and I've thought about doing a remote-update of the files needed by the agents, but the problem with that is portability.

My personal setup has 1 64-bit Windows computer, 2 62-bit Linux computers, 1 mac, and 1 32-bit Windows virtual machine. Each of those types of computers takes a different x264 binary, and (especially for Linux) would be difficult to find out exactly what is needed.

One thing that some people did with x264farm (the 2-pass version) when all of the computers are Windows is to set up a shared folder with the agent executable, x264, and perhaps also the the avs/d2v files. Once all agent computers are set up to use that same shared folder, then updating them all is easy.

[smok3]

ok, got it, wasn't enough multiplatformaware when i wrote that post i guess.

[SpAwN_gUy]

ok, got it, wasn't enough multiplatformaware when i wrote that post i guess.

yep... 1 agent win binary + 1 local x264-Win binary + 1 config.xml = works great with 3 to 10 agents :) ... and all this is on 1 shared folder :)

well, i'm thinking about few shared folder, each of those for a proper platform :) ...

[peter100m]

Cool work omion!

Is there a reason to only support mkv as output file or is there a chance that we can get raw .h264 files or .mp4 files as output?

[SpAwN_gUy]

let's just wait until it gets more-or-less stable :) ...
and then.. start implement new features.. :) ...

[omion]

Cool work omion!

Is there a reason to only support mkv as output file or is there a chance that we can get raw .h264 files or .mp4 files as output?
Yes, there is a reason. It would be more work to implement more features :p

I have been looking into outputting mp4 files, and with the setup of x264farm-sp it shouldn't be too hard. It will probably just take a bit of time.

[peter100m]

@omion: alright, I will be waiting patiently :)

[omion]

Oops. There's still a major bug in the text-mode detection. The problem that I mentioned here (http://forum.doom9.org/showthread.php?p=1116025#post1116025) has been fixed, but that let through another, worse bug. I forgot to fix the line-ending problem in text-mode. What this means is that every once in a while a byte will go "missing" in the encode, causing every frame encoded from then on to be shifted left one pixel.

The problem is only encountered when you are using controller-based encoding on an agent which gives a warning about controller-based encoding. Nothing else is affected.

I'll have a fix out soon.

[SpAwN_gUy]

The problem that I mentioned here (http://forum.doom9.org/showthread.php?p=1116025#post1116025) has been fixed, but that let through another, worse bug.
btw... i've seen something here is the patch fo x264 (http://mailman.videolan.org/pipermail/x264-devel/2008-March/004325.html)..
and compiled binary is here (http://forum.doom9.org/showthread.php?p=1124129#post1124129)

is this build fixes broblems mentioned earlier (i'm doing agent-based..)

[omion]

btw... i've seen something here is the patch fo x264 (http://mailman.videolan.org/pipermail/x264-devel/2008-March/004325.html)..
and compiled binary is here (http://forum.doom9.org/showthread.php?p=1124129#post1124129)

is this build fixes broblems mentioned earlier (i'm doing agent-based..)
Thanks!

BTW, the next version will be out soon. It will have a fix for the previous issue, and it won't require avs2yuv. I'm still testing it for now.

(I was about to release it on Tuesday, but I found another bug regarding text-mode conversion, so it got delayed... again)

[omion]

1.02 is out. Get it here (http://omion.dyndns.org/x264farm-sp/x264farm-sp-1.02_win32.rar). I'm only going to make the 32-bit build from now on, since the controller accesses the AVI functions directly, which is not easily workable in 64-bit.

I fixed the text-mode bug in the controller-based encoding. This should FINALLY be gone. If the video kind of shifts slowly to the left, or the colors get detached, let me know.

I also removed the dependency on avs2yuv. This was the hard part. Basically, I built AVI handling into the controller. During controller-based encoding, the controller starts up multiple other copies of itself which are told to simply send raw data to the agents.
This decreases overhead, too. I used to have to pipe the data from avs2yuv to the controller, then over to the agent. Now I just send the data directly from the "child" controller, and the whole encode is much less choppy (on my setup, anyway)
This also means you will see multiple "controller.exe" processes running on the computer. Every one that is running at below normal priority is one of these child controllers.

There is a possibility that the new agent will use up more memory than the old one, but it shouldn't be much of a problem.

[SpAwN_gUy]

can anyone test... this build? for usage with controller-based encoding?

x264 revision... em.. from 20080410. modified.
built unde MSVS 2005 SP1
gPACK, pthreads - enabled :)

applied patches:
x264_me-prepass_DeathTheSheep.01.diff
x264_2pass_vbv.7.diff
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing

grab it here (http://rapidshare.com/files/107421121/x264.20080410.MSVC2005.modified.exe)

[puffpio]

this looks very promising,

how does the controller split the video feed to the workers? does it have some built in scene cut detection? in all the examples i've seen is that it outputs to a matroska container, can it output to an mp4 container?

[omion]

this looks very promising,

how does the controller split the video feed to the workers? does it have some built in scene cut detection?
It actually doesn't have any scene-cut detection. My 2-pass version does, but it required so much processing that it wasn't worth it. Also, the old version needed to restart an agent's encoding whenever a scene was mis-detected, whereas this version doesn't have that same problem, so re-encoding some frames is faster than figuring out which frames don't need to be re-encoded.

in all the examples i've seen is that it outputs to a matroska container, can it output to an mp4 container?
Currently, it only supports the Matroska container. I just started working on MP4 support, but it's not going too fast. MP4 is quite a bit different from Matroska, but I'll figure it out sometime.

[puffpio]

do you think it would saturate the network to store the source video on a network share that all the workers feed from?

or should each worker have a local copy of the source video?

[puffpio]

would it be simpler to write out the final video output as raw avc? ie .264 extension

[omion]

do you think it would saturate the network to store the source video on a network share that all the workers feed from?

or should each worker have a local copy of the source video?
Going through a network share actually works quite well. If your source is a DVD, a 100base-T network will saturate at about 500 frames per second, which is a lot...

Giving each worker a local copy tends to take longer, actually. If each worker copies the files off the network, you will need to make that many complete copies of it. By using network shares, only the data that each worker needs will be sent to it.

would it be simpler to write out the final video output as raw avc? ie .264 extension
It would be easier to write to .264, but most muxers can't handle a straight AVC stream. Outputting MP4 will be much more useful.

[puffpio]

have you looked into / read about mapreduce or hadoop?

i just had to make a presentation on mapreduce a couple months ago and this seems to fit the problem set that a mapreduce type of problem w/ a distributed file system would excel at

[omion]

have you looked into / read about mapreduce or hadoop?

i just had to make a presentation on mapreduce a couple months ago and this seems to fit the problem set that a mapreduce type of problem w/ a distributed file system would excel at

I have taken a look at mapreduce, and indeed it would be perfect for it if each frame were encoded separately. However, since the optimal split points are not known until after much data is received, the encoding cannot be done entirely parallel. That is, the data from one agent is dependent on the data from any number of other agents. I don't know how this is handled in mapreduce, but since I already had a working distributed renderer in place I didn't bother finding out.

[puffpio]

yeah i think a first pass would need to be done to determine split points, then the chunks can be processed via mapreduce...in that case there could be two distributed mapreduce jobs: (1) find scenecuts, (2) encode

anyways, thats enough of a digression on mapreduce....i hope to test out this stuff as soon as i get some more computers. (at one point i used to work in a computer lab in college..100 PC's...that i had running distributed.net at night :D)

[puffpio]

I setup an encode w/ 2 computers
1 controller/agent and another pure agent

i encoded in agent based mode w/ both computers accessing the same network share for the video file

when i woke up this morning my controller/agent computer had frozen

the logfiles from both the controll and agents look completely normal

[omion]

I setup an encode w/ 2 computers
1 controller/agent and another pure agent

i encoded in agent based mode w/ both computers accessing the same network share for the video file

when i woke up this morning my controller/agent computer had frozen

the logfiles from both the controll and agents look completely normal
When you say frozen, do you mean that the entire computer was unresponsive, or was it just x264farm?

If it was the entire computer, then it sounds like you may have a hardware problem. x264farm is incapable of taking down an entire computer directly. In my experience, the computer crashing is usually due to overheating or memory problems which manifests itself when x264farm is run.

My Mac Mini always used to freeze when I would run x264farm. No other program would do it - x264, Mathematica, Parallels, and Photoshop ran fine. But as soon as I swapped out the RAM with another set, everything was fixed. Sometimes things like that happen.

[puffpio]

the controller/agent that completely froze up (totally unresponsive)
is a core 2 duo @ 2ghz w/ 2 gb of ram, laptop

I've stressed it w/ an x264 HD encode that took 2 days to fully encode, and it didn't lock up.
but i agree it could be other things in the system.

while encoding this, i used a newer build of x264 as well as some deinterlacing filters that i didn't use during my last 2 day encode.

if there is no way the controller or agent app could freeze up the entire comp, i'll try it again but not use the deinterlacer and see what happens then.

[omion]

Now that I think about it, there is one thing that may make the computer lock up. I just ran into it while writing the new version of x264farm. If something goes wrong, then the agent is supposed to kill off the x264 process. If it doesn't, then x264 will continue to encode until it hits the end of the movie. If it keeps having the same problem then more and more x264 processes will start up, and it will eventually make the computer so slow as to be unusable.

The reason I didn't mention this is that I haven't seen it occur in any of my released builds, and I've only just run across it in the development build.

When you do your next encode, keep the task manager open to make sure there are never too many x264 processes.

the controller/agent that completely froze up (totally unresponsive)
is a core 2 duo @ 2ghz w/ 2 gb of ram, laptop

I've stressed it w/ an x264 HD encode that took 2 days to fully encode, and it didn't lock up.
but i agree it could be other things in the system.

while encoding this, i used a newer build of x264 as well as some deinterlacing filters that i didn't use during my last 2 day encode.

if there is no way the controller or agent app could freeze up the entire comp, i'll try it again but not use the deinterlacer and see what happens then.

[puffpio]

im on my next build right now..
currently the controller threw an exception
i dont have symbols for the exe but in the dissassembly it hit an exception at:

0045AF60 mov ecx,dword ptr [esi+edi*4]

interestingly, at the same time one of my agents has locked up and i need to hard reboot it.

[omion]

1.03 is out.

I made a few changes to make the agent a bit more efficient.

The biggest change is that the agents now encode to either MP4 or raw AVC bitstream. NOTE: The controller is still only capable of outputting MKV files, but MP4 output is next on my list.
Basically, I noticed after I implemented the MKV reader that it would only write to the file after about 1MB of frame data had been accumulated. This means that my file reader will be up to 1MB behind what x264 is doing. If a good ending frame is in there somewhere it has to wait until the rest of that 1MB is filled up before it can tell x264 to stop. This was relatively inefficient.
After a few tests, I found that MP4 had no delay and bitstream files have a 4KB buffer, making them both more responsive than MKV.
The result is that there are fewer unused frames at the end of each encode. This will make everything run a little bit smoother.

I also added some fairly fancy process management for agent-based encodes. If there are more x264 programs running agent-based encodes than there are physical processors, the leftovers will be delegated to idle priority. Also, if there are not fewer x264 programs than processors, then all x264 processes will be allocated a CPU (with the exception that one idle-priority x264 process will still be allowed to encode on any processor. Here's a little example:

let's say you have a 4-core processor (like my new one:D). Affinity is the processors that it can run on (0-3)

With 3 agent-based x264 programs, nothing changes:
Program Priority Affinity
1 Normal 0,1,2,3
2 Normal 0,1,2,3
3 Normal 0,1,2,3

With 4, each one gets its own processor:
Program Priority Affinity
1 Normal 0
2 Normal 1
3 Normal 2
4 Normal 3

With 5, the last one gets pushed down to idle, but it can run on any processor:
Program Priority Affinity
1 Normal 0
2 Normal 1
3 Normal 2
4 Normal 3
5 Idle 0,1,2,3

With more than 5, each additional x264 program gets allocated to one processor again, but at idle priority:
Program Priority Affinity
1 Normal 0
2 Normal 1
3 Normal 2
4 Normal 3
5 Idle 0,1,2,3
6 Idle 0
7 Idle 1
8 Idle 2
9 Idle 3
10 Idle 0
etc...

Program #5 in the tables is treated differently so that there is always something that can be run in case any one of the first four stalls.

Only agent-based encodes are counted, and only agent-based encodes are affected.

If this doesn't work for whatever reason, you can use the --nosched argument to the agent, and it will act like it always used to.

[peter100m]

Great job Omion. Looking forward to the addition of mp4 output with controller.

[omion]

Great job Omion. Looking forward to the addition of mp4 output with controller.

Ask and ye shall receive! Version 1.04 (http://omion.dyndns.org/x264farm-sp/x264farm-sp-1.04_win32.rar) now supports direct output to MP4. Just give the output file a ".mp4" filename and it will create a (hopefully valid:rolleyes:) MP4 file. (MAN, that file format is bizarre. I don't know what Apple was thinking when they invented it, or what MPEG was thinking when they used it to be the "standard")

The x264 executables need not support MP4 output - everything is handled by the controller.
Also, if you run into a problem with the resultant MP4 file (for whatever reason) run the same settings with an MKV extension and see if the problem goes away. The encode should get done very quickly since the temp files are the same.

Other changes include a much cleaned-up controller output, as well as FPS estimates and removing dead agents from the output.

I did actually notice a bug that crept into version 1.03, which is still there in 1.04. If you use agent-based encoding, you may see some x264 processes that are not killed off when they should be. These processes will continue encoding until they hit the end of the video (which may be a long time if they were working on the beginning). If this is a problem, use the --nosched command-line option to the agent. It should fix the problem.

[omion]

Not sure if anybody still cares, but...

I just released 1.05. Get the Windows binaries here (http://omion.dyndns.org/x264farm-sp/x264farm-sp-1.05_win32.rar).
I did a few bugfixes, but the biggest news in this version is that you can now encode multiple videos at the same time by specifying more than one of each option. It's a little confusing, so here's an example:

Let's say you wanted to encode a high-quality and a low-quality version of a single video. You might use the following line:
controller -i infile.avs -x "--crf 30" -o lowquality.mp4 -x "--crf 20" -o highquality.mp4
This will make 2 encodes of the same file: lowquality.mp4 will use CRF 30, highquality.mp4 will use CRF 20. The order of each option determines the output file it is paired with, so the first "-x" and the first "-o" will be the first encode, the second "-x" and the second "-o" will be the second encode. Since there is no second "-i" option in this case, the first one is for both.

If you want to use the same encoding settings on multiple input files, use something like this:
controller -x "x264_options..." -i input1.avs -o output1.mkv -i input2.avs -o output2.mp4 -i input3.avs -o output3.mkv
This will encode input1.avs to output1.mkv using the "-x" options given, and will encode input2.avs to output2.mp4 using the same "-x" options. Same for input3.avs and output3.mkv (note that you can mix MKV and MP4 output files)
If you want to use different options, just specify more "-x" lists.

BIG CAVEAT:
The input file or encoding options (or both) MUST be different between every two encodes that you are doing. The encodes will probably finish properly, but anything else you do based on the results of the encodes (like change around the zones) may start deleting useful temporary files.

How it REALLY works:
The controller keeps a list of each "-i", "-x", "-o", "--seek", "--frames", and "--zones" options, each of them arranged in the order they appear. If one option is specified fewer times than another, the last instance in that list will be used for all the rest of the encodes. So "-i a.avs -i b.avs" will use "b.avs" for every encode after the first one.
The controller then will be able to tell agents to do part of any of those videos, which should increase the parallelizability(:confused:) of the encoding process, accompanied by a stylish text-based progress bar for each video.

In conclusion, some things to remember:

Either the input file or encoding options must be different between any two encodes on the command line
There must be as many "-o" options as there are of any other option (that is, every encode must have a distinct output file)
To use zones with one encode but not the next, use --zones "" for the second one.
Similarly, for --frames, just specify a very large number (no larger than 1,073,741,823:cool:. Movies have around 150,000 frames on average)


(I should probably mention some of this in the HTML file...)

[peter100m]

omion: Thanks for this! Sounds like a great addition, will definately try it out!

[Slabrdix]

Hi,

I've programmed an Wrapper for MeGui to start the x264farm-sp Contoller as a replacement for x264.

It's simple

you have to make a directory in megui/tools ex x264Farm
put the controller and it's config.xml in it + x264FarmWrapper.exe
use X264FarmWrapper as a replacement for x264 in the settings !! That's it. :)

You must be aware that you should only have one Worker Que !!

Unfortunately I can not really give precise status information to the megui, just rough estimations as i have to calculate the processed frames from output parsed fps and time running, but it's better than nothing.
Perhaps controller.exe can be updated so that it outputs the actual number of frames processed, as this is the key information Megui uses to output status info.

Have Fun

[Slabrdix]

Fixed problem with spaces in filenames

Regards

[Slabrdix]

Hi
The controller seems to have problems with spaces in filenames

controller.exe --clean --restart -i "V:\Film 1\Film 1.avs" -o "V:\Film1\Film 1.mp4" -x" --crf 23"

is giving

ERROR: AVS file "V:\Film 1\Film 1.avs" does not seem to be valid

as an error

Regards

[ankurs]

distributed encoding will only be accepted by masses when 2pass encoding is added

i hope the issues get resolved sometime in future :sly:

[professor_desty_nova]

distributed encoding will only be accepted by masses when 2pass encoding is added


Did you not see the thread about the original x264farm? (started in 2006;)) http://forum.doom9.org/showthread.php?t=117889

The original x264farm suports 2 passes. This thread is a later development for people who wants to do single pass.

[Dagger-]

Any chance of a version that works on Windows 2000? It calls GetProcessId(), which requires Vista/XP SP1, and so refuses to run (with "The procedure entry point GetProcessId could not be located in the dynamic link library KERNEL32.dll").

As far as I can see, GetProcessId is only actually called from opttest.ml -- which I'm guessing is test code -- so this should be fairly trivial to do. (And, indeed, it works fine if I hexedit the import table to remove that function.)

[omion]

Any chance of a version that works on Windows 2000? It calls GetProcessId(), which requires Vista/XP SP1, and so refuses to run (with "The procedure entry point GetProcessId could not be located in the dynamic link library KERNEL32.dll").

As far as I can see, GetProcessId is only actually called from opttest.ml -- which I'm guessing is test code -- so this should be fairly trivial to do. (And, indeed, it works fine if I hexedit the import table to remove that function.)

That should be easy to do. Even if GetProcessId is used elsewhere, it certainly shouldn't be. I don't have a Windows 2000 computer, but I'm glad you zeroed in on the problem. I'll make an updated version and get it out soon.

@Slabrdix:
About the spaces problem - I'm not at my main computer now so I can't test it, but I probably just forgot to add quotes to something... Just to make sure, it works when you remove the spaces from the file and path?

(sorry it's taken so long to reply, I didn't get a notification for anything until Dagger-'s reply...)

[omion]

1.06 is out. Windows binaries are here (http://omion.dyndns.org/x264farm-sp/x264farm-sp-1.06_win32.rar).

I removed the references to GetProcessId, so it should now work in Windows 2000. I did add a few more Win32 functions, but I think they're all defined in Windows 2000.

I also fixed a problem where the agents wouldn't encode files with spaces in the filename. This appears to be different from the problem that Slabrdix posted, which I can't reproduce. I check the AVS script with Win32 function calls, not an external process, so spaces wouldn't matter at all. I'm not sure what the posted problem is...

The agent now tells the controller if x264 failed. If it has, the controller will put a counter next to the agent's name indicating how many times it has failed. If many agents have failed it's probably a problem in the AVS file, but if only one fails it's probably a problem with the agent's configuration (like it doesn't have all the AVS filters).

There are a few other changes, mainly improving the efficiency of controller-based encoding.

[Dagger-]

Thanks for that -- works perfectly.

And now that I've finally had a chance to test properly, your "Made controller-based data transmission faster" seems to have helped quite a bit too: no more needing to run two encodes per CPU to max them out.

(As for my excuse for taking ages to reply, I shall offer "assassinated by a cold".)

[omion]

1.07 is out. Windows binaries are here (http://omion.dyndns.org/x264farm-sp/x264farm-sp-1.07_win32.rar).

I made a few fixes regarding performance, but I didn't notice any actual improvement. (I did the same thing to the agent that I did in 1.06 to the controller)

The biggest improvement is that calling x264 from another process no longer breaks things.
Originally, if the agent was set up to use an intermediate program which runs x264 (like a batch file, for example) everything would work fine until the agent wanted to stop the encoding. Then it would close the batch file but keep x264 running. Now it runs the encoder in a job and deletes the job at the end, which cleans up any leftover processes.


Now, just thinking out loud:
I've been looking into intercepting x264's file calls so that x264farm has better control over the location of the files. My idea was that if x264 read from a file that didn't exist in the encode directory, the agent would tell the controller to get that piece from its computer. This would enable agent-based encoding without copying the files over or setting up a shared folder. The source files would just be read directly over the network.
The problem is that I can't seem to tell the process to redirect functions. The only way I see that's remotely possible is to rewrite the program's function table to point to different places, but that's way over my head (and it sounds a lot like something a virus would do)... If anybody has made it this far through this paragraph, is there an easier way to use my function instead of system ones for a child process?

[Quark.Fusion]

You can write custom file system :) like AVFS (http://www.turtlewar.org/avfs/). But how reading over network differs from reading from shared folder?
Other way is to change x264 source code to support network transfer.
And the simpliest way is to use AviSynth's TCPServer/TCPSource, but they not stable enough and require very fast link.

[omion]

You can write custom file system :) like AVFS (http://www.turtlewar.org/avfs/). But how reading over network differs from reading from shared folder?
Other way is to change x264 source code to support network transfer.
And the simpliest way is to use AviSynth's TCPServer/TCPSource, but they not stable enough and require very fast link.

Well, it looks like TCPserver does basically what controller-based encoding does, so I don't think it would help too much. AVFS is similar too -- I want to avoid transferring the uncompressed data if at all possible.

What I really want is when the script says something like
SomeInputFunction("c:\file.d2v")
I want to capture the actual file lookup and send it to the controller's system. But to do that I would need to get something in between the AVS DLL and the Windows system call.
Doing it this way has the networking advantages of agent-based encoding with shared folders, but you wouldn't have to set up any sharing or even change the AVS file. The source files wouldn't have to use absolute paths anymore either, since it only does a lookup in the controller's computer.

I've been looking into Google Chrome, since it looks like the sandbox uses DLL function redirection. Chrome is where I got the idea to use jobs to contain the child processes, and it looks like there are a lot of other clever things in it. The biggest problem is that I don't really know how to read C++.