x264farm: distributed encoding

[aicjofs]

When you say "bring it to it's knees", do you mean that all the avs2yuv instances max out the CPU, or is it a memory issue?

More of a memory issue. It is gone when I use SetMemoryMax(64). Of course in hindsight I see you already recommended that. Still hits the CPU to 100% but system is still responsive. It was the page file usage that was bogging it, all the avs2yuv instances chewed up all 2GB but now not an issue with SetMem. Going to experiment with nice.exe next, I don't need the computers for anything while encoding that's what a laptop is for, but I might see what different CPU priorities do.

Whats your thoughts about 2 agents on a Hyperthreaded CPU? Can't imagine any gain as it's not another physical CPU.....If I have time I'll give it a try tomorrow to decide for myself.

In Megui, you can enter the codec configuration, and check the "Show commandline" box,

*sigh*, boy am I dumb. :D Amazing what is right under your nose all along. Appreciate the tip. That's exactly what I needed.

[havix]

Nice job!!! I can't believe I even got it to work to be honest. The only slow part to getting this working is copying and pasting different commands from megui and removing the incompatible parameters. I can't wait to play around with this some more this weekend and expand it from my current 2 machine setup to a total of 4 machines. Has anyone experimented what gives the best speed pertaining to the controller machine and if you should run only one agent so you don't choke the avisynth processing on a dual-core machine? Right now I'm running two agents on the controller but I'm not sure if this could be slowing down the delivery.

[havix]

Sorry guys I should have read through the rest of the instructions. I will play with task priorities in the config files.

[aicjofs]

More testing. All of this is during the 1st pass, I'll wait for the 2nd pass before I add more.

When running one instance of agent.exe on each of the P4 machines, I am getting around 20-30% CPU usage. I thought it would be interesting to try --threads 2 in x264, but x264 says it's not compiled with ptthread and I see no speed difference. I added 2 instances of agent to the 2 P4's and cpu usage is close to 40-50% now. According to the new fps indicator I am getting about 20fps on each P4(10fps + 10fps per agent) and about 20fps each agent on the Opteron. So total fps is around 80fps per the indicator for the 1st pass.

So 2 questions now are:

The x264 ptthread issue. Is this something you need to add to your build of x264? or am I completely off base?

On the SetMemoryMax issue. I still have available RAM on the controller computer is there any benefit from setting this higher then 64?

I'll add more once 2nd pass is complete.

EDIT: One other thing. Splitter list too long splitter spinning? What does this mean?

[omion]

More testing. All of this is during the 1st pass, I'll wait for the 2nd pass before I add more.

When running one instance of agent.exe on each of the P4 machines, I am getting around 20-30% CPU usage. I thought it would be interesting to try --threads 2 in x264, but x264 says it's not compiled with ptthread and I see no speed difference. I added 2 instances of agent to the 2 P4's and cpu usage is close to 40-50% now. According to the new fps indicator I am getting about 20fps on each P4(10fps + 10fps per agent) and about 20fps each agent on the Opteron. So total fps is around 80fps per the indicator for the 1st pass.

So 2 questions now are:

The x264 ptthread issue. Is this something you need to add to your build of x264? or am I completely off base?

Yeah. I don't use it in my builds. In fact, I don't know if MSVS uses the same style threading as what x264 uses (it was really designed for GCC)


On the SetMemoryMax issue. I still have available RAM on the controller computer is there any benefit from setting this higher then 64?

Maybe. The difference probably won't be noticable, but you can set it as high as you want without running out of memory.


EDIT: One other thing. Splitter list too long splitter spinning? What does this mean?
That's perfectly OK. That means that there are more jobs in the queue than agents. In this case the splitter thread is paused in order to devote more resources to avs2yuv and the agents. The splitter thread starts up automatically when a job is checked out.


BTW: How is the CPU utilization laid out on the controller computer? Is it mostly avs2yuv, or does the agent (x264) take up the majority of the CPU? It looks a bit like your remote agents are starved for work.

[aicjofs]

2nd pass kicked out broken pipe error, and invalid arguement on the agents.

2006-11-10~17:24:16.85 Got a connection from 127.0.0.1:3451
22006-11-10~17:27:07.25 1st pass
2006-11-10~17:27:07.25 Doing the following:
x264 --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr --fps 24000/1001 --pass 1 --stats "G:\x264farm\temp\cpu1\165835 166209 05dd43.txt" -o NUL - 718x358
2006-11-10~17:27:18.57 Exited 0
2006-11-10~17:27:18.60 Finished OK; shutting down socket
2006-11-10~17:27:18.60 DOING!
2006-11-10~17:27:18.60 Got a connection from 127.0.0.1:3461
2006-11-10~17:27:25.59 Agent croaked with exception "End_of_file"; restarting
2006-11-10~17:27:25.59 DOING!
2006-11-10~17:27:36.78 Got a connection from 127.0.0.1:3463
2006-11-10~17:27:36.81 2nd pass
2006-11-10~17:27:37.42 Done recieving stats
2006-11-10~17:27:37.42 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats 02c6b7.txt" -o "G:\x264farm\temp\cpu1\output c4ab32.mkv" - 718x358
2006-11-10~17:27:37.48 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:37.48 DOING!
2006-11-10~17:27:37.59 Got a connection from 127.0.0.1:3469
2006-11-10~17:27:37.70 2nd pass
2006-11-10~17:27:38.07 Done recieving stats
2006-11-10~17:27:38.10 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats 5cf6b5.txt" -o "G:\x264farm\temp\cpu1\output a12837.mkv" - 718x358
2006-11-10~17:27:38.45 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:38.45 DOING!
2006-11-10~17:27:38.57 Got a connection from 127.0.0.1:3475
2006-11-10~17:27:38.89 2nd pass
2006-11-10~17:27:39.03 Done recieving stats
2006-11-10~17:27:39.03 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats c66430.txt" -o "G:\x264farm\temp\cpu1\output a8a9cd.mkv" - 718x358
2006-11-10~17:27:39.18 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:39.18 DOING!

2006-11-10~17:25:11.96 Got a connection from 192.168.1.101:3454
2006-11-10~17:25:11.98 1st pass
2006-11-10~17:25:11.98 Doing the following:
x264 --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr --fps 24000/1001 --pass 1 --stats "d:\x264farm\temp\cpu1\164335 164709 c1d557.txt" -o NUL - 718x358
2006-11-10~17:25:32.31 Exited 0
2006-11-10~17:25:32.39 Finished OK; shutting down socket
2006-11-10~17:25:32.39 DOING!
2006-11-10~17:25:32.39 Got a connection from 192.168.1.101:3455
2006-11-10~17:27:17.37 Agent croaked with exception "End_of_file"; restarting
2006-11-10~17:27:17.39 DOING!
2006-11-10~17:27:28.57 Got a connection from 192.168.1.101:3467
2006-11-10~17:27:29.31 2nd pass
2006-11-10~17:27:29.42 Done recieving stats
2006-11-10~17:27:29.42 Doing the following:
x264 --bitrate 5 --fps 24000/1001 --pass 3 --stats "d:\x264farm\temp\cpu1\stats 6281e0.txt" -o "d:\x264farm\temp\cpu1\output 58bba6.mkv" - 718x358
2006-11-10~17:27:29.54 Agent croaked with exception "Sys_error(\"Invalid argument\")"; restarting
2006-11-10~17:27:29.54 DOING!
2006-11-10~17:27:29.84 Got a connection from 192.168.1.101:3473
2006-11-10~17:27:29.96 2nd pass
2006-11-10~17:27:30.31 Done recieving stats
2006-11-10~17:27:30.31 Doing the following:
x264 --bitrate 841 --fps 24000/1001 --pass 3 --stats "d:\x264farm\temp\cpu1\stats 5226cf.txt" -o "d:\x264farm\temp\cpu1\output 969611.mkv" - 718x358
2006-11-10~17:27:30.35 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:30.35 DOING!


The AVS

SetMemoryMax(64)
SetMTMode(2)
DGDecode_mpeg2source("H:\CRZ0NNW7\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)
#blank deinterlace line
crop( 0, 60, -2, -62)

#resize
#denoise

Command

-b 1000 --avs "file.avs" --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr"

--second "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr" -o "test.mkv"

Since I am a noob at x264 command line it's possible my command is the issue. I just copied from meGUI like foxy recommended.

EDIT:

BTW: How is the CPU utilization laid out on the controller computer? Is it mostly avs2yuv, or does the agent (x264) take up the majority of the CPU? It looks a bit like your remote agents are starved for work.

Each x264 is using about 20% so 40% total, and the rest for controller, and avs2yuv.

[omion]

2nd pass kicked out broken pipe error, and invalid arguement on the agents.

[...]

Since I am a noob at x264 command line it's possible my command is the issue. I just copied from meGUI like foxy recommended.

Your command line looks OK (*) (except your -b argument needs to have "%" or "kbps" after the number, but that throws an error at the beginning, so I assume it's OK). What does the controller output say when the agents were putting up the broken pipe errors? If you closed the window or something, the log is saved in the "out-dump.txt" file in the directory where you ran the controller.

Actually, there's something else odd. When the agents say "Doing the following: ..." it doesn't look like they actually get the second pass line that you specified. Make sure that the controller got the right options by opening up the controller's "out-dump.txt". Near the top, there should be a line that gives the second-pass parameters that the controller is actually using.

It's possible that the "broken pipe" that the agents are complaining about are actually the pipe to x264, indicating that it died prematurely.
[ edit: This is probably the case. I seem to remember that x264 will croak if the first pass uses more B-frames than the second pass specifies, and since the second pass options didn't make it to the agent, it defaults to 0. ]



Each x264 is using about 20% so 40% total, and the rest for controller, and avs2yuv.
Hmm. It's odd that the remote computers aren't using any more than 20% per agent. And it's probably not a network issue, or adding more agents wouldn't help. Are you using version 1.01?

(*) The option vbv-maxrate may throw off the second pass. The reason is that I had to do ratecontrol in the controller, but I hardcoded a lot of the values to the defaults (qpmin, qpmax, ipratio, etc.) I'm pretty sure vbv-maxrate is one of those that would need to be supported in the controller, but isn't. The result is that x264 will make some GOPs with a lower bitrate than the controller expects, so they are marked to be re-encoded and the controller will inflate the bitrate of the entire GOP to accommodate that. (I can't really explain it too well)


NOTE: If you stop the controller, then restart it, it will pick up where it left off. You have to give it the same arguments, or it will think it's doing a different encode.

[aicjofs]

Looks like you nailed it. The 2nd pass options were blank in the dump log for the controller, I'm getting much better at understand the format of your dumps. I have started it 3 times in a row now and the 2 pass options are always there. I have run it the whole time with a .bat file so it's not like the options weren't there to begin with. If I look deeper through the logs I can see a x264 error about not the same b frames as the 1st pass, which of course makes since when no 2nd pass options were sent. Since I can't duplicate this it must be a non issue.

It's also complainging about no buffer set when the vbv-maxrate was set. I'm leaving out that option next go around. If you don't think we should be using it(vbv-max) maybe another update to the help file?

About to start another round. Next I'd like to see about getting more work for the clients.

Pretty exciting stuff actually. I'll post a screenshot in a few.


EDIT:

Controller

http://home.comcast.net/~aicjofs/264farm.jpg

One of the agents. Looks Identical to the other.

http://home.comcast.net/~aicjofs/264farm1.jpg

[omion]

Ooh. Screenshots. Yummy.

The shot of the network utilization is a bit odd. It looks kind of like it's maxing out at ~10%, which would be right where 100Mbps would be. Are your other computers connected with gigabit also?

I made a little network test setup for debugging, if you want to see if it's a network issue. The "sender" is here (http://omion.dyndns.org/other/from3.exe), and the "receiver" is here (http://omion.dyndns.org/other/to3.exe).

As with x264farm, start up the receiver first:
to3 45678 4096
(45678 is the port to use, 4096 is the string length)

Then start up the sender:
from3 192.168.1.100:45678 4096
(192.168.1.100:45678 is where you started up the receiver, 4096 is the string length)

I can get my jumbo-frame gigabit network up to ~25% with this test. See if you get any more than 10%.

It's also complainging about no buffer set when the vbv-maxrate was set. I'm leaving out that option next go around. If you don't think we should be using it(vbv-max) maybe another update to the help file?
OK. I changed the working copy of the help file to include all the ratecontrol things. I should be able to get it out soon, or at the latest when 1.02 comes out.

[Inventive Software]

Do you plan to reduce the transmission size across networks omion? If what he has is average 12.5% usage across 1 Gbit network, that's more than 100 Mbit can handle, so it'd hinder it. Is the transmission size (i.e the stuff sent through the cables) affected by the size of the content?

[omion]

It is definitely a hinderance, but you can get around 20 DVD-size frames per second out of a 100mbit connection. With my settings, anyway, that's about all I can pull out of my remote computers.

The data will have to be transmitted somehow. The only ways around this is to transmit it before rendering (which would need avisynth installed on the remote computer), or compress the raw data before sending.

I gave up on doing remote avisynth, and the compression would have to compress 10MB per second while using as little CPU as possible. I suppose it would be easy to try out some compressions, but I have a feeling it would take up too much CPU power to be worthwhile.


I am actually thinking about making avisynth servers, where the "controller" would also be divided amongst several computers, so the agents would be able to distribute the network load.

I haven't been able to figure out how to do the load-balancing, but it should be possible.

[708145]

I gave up on doing remote avisynth, and the compression would have to compress 10MB per second while using as little CPU as possible. I suppose it would be easy to try out some compressions, but I have a feeling it would take up too much CPU power to be worthwhile.


<advertisement>ELDER does the avisynth processing on each compute node. If someone wants to try it.</advertisement>

http://forum.doom9.org/showthread.php?p=718465#post718465

bis besser,
T0B1A5

[aicjofs]

All machines are gigabit. No jumbo frames the switch doesn't support it. I get 33% utilization on all 3 machines. Picture below is connecting one client at a time. Seems the bandwidth is available. If you want to give me the avs and your controller command line I can mimmick it with my own input file.

http://home.comcast.net/~aicjofs/264farm2.jpg

Yes 708145 I have used Elder but last time I looked you were freezing x264 development we need to continuing pushing forward in the x264 arena. :D

[708145]

Yes 708145 I have used Elder but last time I looked you were freezing x264 development we need to continuing pushing forward in the x264 arena. :D

I froze the _GPL_ branch of x264 2pass encoding. Beta5 includes an x264 crf mode :)
There is a development version of x264 2pass as well.

[omion]

1.02 is out (at last)

It took so long because I tried to do too many things and ended up basically rewriting the controller. It should run a bit less buggy now.

You will only need to run one agent program per computer now. It does all the threading in the program, so that you don't have to.

aicjofs's issue with the remote machines not using all the CPU should be fixed now. It should be quite a bit faster (at the expense of ~8MB more memory usage, and maybe slower x264 startup time).

Note that the format of the config.xml files has changed. See my examples in the package for what to do.

There should be fewer bizarre bugs, too. I updated my version of OCaml, and it seems to have fixed a lot of the "operation completed successfully" errors.


I'm not including the Mac or Linux builds with the main download, because I don't think enough people use them to warrant having everybody download them. I am still planning to have the Mac build available as a separate download soon.

[aicjofs]

Major improvements in this version.

Easier to setup(less config.xml's to edit) for one.

So I tried running the agents on the clients with "number 2". This loaded the P4's to about 40-50% on 1st and 2nd pass. Network utilization about 17% and peaked to 20% on controller. Fps (reported in controller) of about 77fps 1st pass 75fps 2nd pass.

I then tried "number 4" and "+1" (for 2nd pass) for clients. That ran 4 to 5 instances of x264 on each client. Then on both passes the clients were loaded from 85-100% average around 90%. This boosted fps to 89fps 1st pass, and 83fps 2nd pass, also increased network util on controller to about 25% and peaked as high as 30%.

Normally I would get around 50fps if it was just the controller rig using MeGUI or Nero with the simple parameters. Obviously this is for light encoding and I can't assume it scales because I haven't tried it. But it seems like it would if I added a more difficult profile that normally drags along at 10fps on a single machine. With x264farm I might just get realtime speed around 20-30fps.

I really like this build. Only real waste of CPU time seems to be when the ranger has passed out all frames and you are near the end of the 1st pass and the controller is waiting for all the clients to send back their frames, this causes the other clients to go idle waiting for all the frames to return so the 2nd pass can start. Only cosmetic gripe(and this is only while testing it matters) is the agents toggles the "frames -fps" display to fast. When I had 4 "CPU's
" on the agent it was hard to read where each CPU was on what frame. Not a big deal but while testing it was happening to fast to read the bottom line. Maybe slow the switching delay or add all CPU's to the bottom line? No need to really see once x264farm matures, but it would be nice for testing purposes.

[CruNcher]

@ omion

what does this error mean ?


2006-11-20~19:30:40.00 x264farm version 1.02-151
2006-11-20~19:30:40.00 OPTIONS:
2006-11-20~19:30:40.00 -b 8000%
2006-11-20~19:30:40.00 --first "--no-fast-pskip --no-dct-decimate --bframes 3 --weightb --direct auto --no-cabac --subme 1 --analyse none --me dia --no-psnr"
2006-11-20~19:30:40.00 --second "--no-fast-pskip --no-dct-decimate --bframes 3 --weightb --direct auto --no-cabac --subme 2 --analyse p8x8,b8x8,i4x4 --me dia --no-psnr"
2006-11-20~19:30:40.00 --avs "sylver.avs"
2006-11-20~19:30:40.00 --fastavs ""
2006-11-20~19:30:40.00 --zones 2006-11-20~19:30:40.00
2006-11-20~19:30:40.00 --seek 0
2006-11-20~19:30:40.00 --frames 0
2006-11-20~19:30:40.00 --batch 5000
2006-11-20~19:30:40.00 --split 250
2006-11-20~19:30:40.00 --thresh 20.000000
2006-11-20~19:30:40.00 --preseek 0
2006-11-20~19:30:40.00 --force false
2006-11-20~19:30:40.00 --restart false
2006-11-20~19:30:40.00 --config "config.xml"
2006-11-20~19:30:40.01 --3thresh 0.800000
2006-11-20~19:30:40.01 --3gops 1073741823
2006-11-20~19:30:40.01 --3ratio 0.050000
2006-11-20~19:30:40.01 -o "out.mkv"
2006-11-20~19:30:40.01 Using config file ".\config.xml"
2006-11-20~19:30:40.01 CONFIG:
2006-11-20~19:30:40.01 temp_dir: "H:\\x264farm\\temp"
2006-11-20~19:30:40.01 agent_list 1:
2006-11-20~19:30:40.01 "Local CPU 1" = 127.0.0.1 50700-50703
2006-11-20~19:30:40.01 "Laptop 1" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.01 agent_list 2:
2006-11-20~19:30:40.01 "Local CPU 1" = 127.0.0.1 50700-50703
2006-11-20~19:30:40.01 "Laptop 1" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.01 "Laptop 2" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.48 GOT NORMAL INFO "sylver.avs: 1920x1080, 25 fps, 10816 frames\013"
2006-11-20~19:30:40.48 ERROR: AVS file 'sylver.avs' does not seem to be valid

[omion]

@ omion

what does this error mean ?


2006-11-20~19:30:40.48 GOT NORMAL INFO "sylver.avs: 1920x1080, 25 fps, 10816 frames\013"
2006-11-20~19:30:40.48 ERROR: AVS file 'sylver.avs' does not seem to be valid


That normally means that the AVS file has an error and the video information could not be read. Except that (as you can see) the information could be read.

The problem is that I assumed the FPS is always a rational number (i.e. x/y), but over in PAL land it looks like it's just given as an integer. I'll make a quick fix and put up a new version as soon as I can.

[omion]

1.03 is out. CruNcher's problem should be fixed. That's the only change since 1.02.

[ChrisBensch]

I'm using one of my machines as a controller and the other as an agent with v1.03. Both are Gigabit and are using about 7% on average. I have 4 total ports for the agent machine but only one is being used and I only see one instance of x264.exe (I'm still encoding the first pass). I don't see any errors, but I'm sure it could be faster based on other people's results. I've even tried taking out my processing settings. I guess I have one question...is anyone else doing HDTV 1920x1080i ---> HDTV 1280x720p encoding? Or are these results posted from DVD backups?

I used the to3 and from3 network tests and I peaked my Gigabit setup at 17% using the settings from page 3. I guess I'll try with DVD stuff later...

[omion]

I'm using one of my machines as a controller and the other as an agent with v1.03. Both are Gigabit and are using about 7% on average.7% CPU, or 7% network usage? If it's network usage, how's the CPU usage?

I have 4 total ports for the agent machine but only one is being used and I only see one instance of x264.exe (I'm still encoding the first pass). I don't see any errors, but I'm sure it could be faster based on other people's results. I've even tried taking out my processing settings. I guess I have one question...is anyone else doing HDTV 1920x1080i ---> HDTV 1280x720p encoding? Or are these results posted from DVD backups?I must admit, I haven't done any HDTV tests with it yet. I'll try it out once I un-break what is going to become 1.04.

[ChrisBensch]

Well, I think I was jumping the gun a bit, it seems that during the first pass I was only getting one instance of x264, but during the second I was getting two. The 7% was bandwidth, cpu was around 45%. Now that on the second pass, the cpu is at 100% and bandwidth is around 1%...my agent/encoder is the slower machine...so we'll see what happens when I add the rest of my boxes. Keep up the good work!

[omion]

Well, I think I was jumping the gun a bit, it seems that during the first pass I was only getting one instance of x264, but during the second I was getting two. The 7% was bandwidth, cpu was around 45%. Now that on the second pass, the cpu is at 100% and bandwidth is around 1%...my agent/encoder is the slower machine...so we'll see what happens when I add the rest of my boxes. Keep up the good work!
Great!

It looks like I still need to work on the network code to keep x264 fed, though. The agent should use 100% of each core, even for the first pass. As aicjofs noted, one way around this is to increase the number of agents running with the <number> line in config.xml, but that's still just a workaround. I'll see what I can tweak in the next release.

BTW: The reason you get one x264 during the first pass and two during the second pass is due to the pad="1" attribute in the <number> line. Basically, during the second pass, the jobs are small enough that a a relatively large percentage of the time is x264 setting up. Therefore it's a good idea to have something running in the background for doing something useful during that time.

[ChrisBensch]

So then only a single instance of x264 is normal during 1st pass? I thought that on a dual cpu system I'd get one per core? I have 4 ports setup for use (50700-50703)

[havix]

Just a suggestion, maybe you could integrate something like "psexec" to launch agents on remote machines when the controller launches. This would make things easier to jump right into encoding.

[aicjofs]

So then only a single instance of x264 is normal during 1st pass? I thought that on a dual cpu system I'd get one per core? I have 4 ports setup for use (50700-50703)

You are changing the "number" tags in the controller config.xml right? Not only do you add ports but you need to change that number to get more x264 instances.


<number pad="1">change this number</number>

[omion]

You are changing the "number" tags in the controller config.xml right? Not only do you add ports but you need to change that number to get more x264 instances.


<number pad="1">change this number</number>

Right... mostly. You actually don't need to increase the number of ports for more encodes.

Before 1.02, each agent listened on its own port and were completely separate. When I released 1.02, two things changed:
* I implemented a more "server" style for the agents. All of the jobs go through one port, and the agent program splits off threads to handle each one.
* I let the agent select one port from a range of ports, in order to squash an uncommon but annoying problem when a port is not available right after an agent is stopped.

So basically, the number of available ports was expanded, but the number of used ports was reduced to 1.

It is also worth noting that the agents also have a <number> line in their XML file. This <number> does nothing. Only <number> on the controller side will have any effect.
I had originally had the agents reject connections if there were already <number> connections, but figuring out what was connected proved difficult. I removed the effect, but I kept the config line just in case I re-enable it.

Just a suggestion, maybe you could integrate something like "psexec" to launch agents on remote machines when the controller launches. This would make things easier to jump right into encoding.That does sound like a good idea. I don't know what the best way to only start up the agent if it has not already started, but I think it would be possible.
I've got a lot of ideas backed up, so it will probably take a while to implement.

[CruNcher]

Hi omion it's working now but something doesn't feels quite right with the cpu usage on the client see screenshot bellow

http://cruncher.mufflastig.com/x264/x264farm.png

Network utilization is @ 50%
Cpu usage on the client is fluctuating very heavy

and i have a improvement wish :)
could you please make the controller delete the created files on the fly after he concentate them this way you don't need 2x the diskspace :)

[omion]

Network utilization is @ 50%
Cpu usage on the client is fluctuating very heavy

It's possible that the network is saturated at 50%. My gigabit network never gets more than about 35-40%. The next version will have a user-definable agent buffer, which may make things better (or at least different :D ) I still have some work to do on it, but it shouldn't take too long.


and i have a improvement wish :)
could you please make the controller delete the created files on the fly after he concentate them this way you don't need 2x the diskspace :)
It would be easy to add a switch that would do that. I'm hesitant to do it by default because if there was an error opening one of the files, the controller would have to recreate the deleted files when it was restarted. That hasn't happened to me yet, but I don't like the default behavior leaving the files in a state where it could happen. A switch would be good though. I'll add it into 1.04.

[omion]

OK. 1.04 is out, along with a very big addition that I knew needed to happen some time.

A few minor changes first:
* The "-b" bitrate option has been renamed "-B" to better match x264's parameters
* I also added the --savedisk command line option, to implement CruNcher's on-the-fly temp file deletion. It will not use up more disk space than the final file, but if something goes wrong when merging, the deleted parts have to be re-rendered


1.04 introduces "agent-based" encoding, when the agent has access to the AVS file and calls x264 directly, rather than "controller-based" where the controller does everything.

In order to set it up, you need to use the <base></base> line in the agent's config.xml. This is the base directory to search for the AVS file, in case the agent's computer has a different directory structure.

Here's an example:
Agent's config.xml: <base>F:\temp\x264farm</base>
Controller's AVS file: D:\movies\working\somewhere\file.avs

The following files will be searched for on the agent's computer:
F:\temp\x264farm\D:\movies\working\somewhere\file.avs
F:\temp\x264farm\movies\working\somewhere\file.avs
F:\temp\x264farm\working\somewhere\file.avs
F:\temp\x264farm\somewhere\file.avs
F:\temp\x264farm\file.avs

The first file that is found whose contents match the controller's AVS file will be used as input to x264, rather than getting the frame data through the controller.

Also, if you use <base></base> or <base/> in the config file, that indicates that the only place to look for the file is exactly where the controller found it. This is good if both computers have the same directory structure, or if the agent is running on the controller's computer.

If a matching file is not found anywhere, the agent will default to receiving data from the controller, just like before.

To turn off the agent-based encoding, either remove or comment out the <base> line in config.xml.


Get it at the beginning of the thread. Have fun!

[akupenguin]

The first file that is found whose contents match the controller's AVS file will be used as input to x264
If you check that the contents match, why not just send the avs to the agent and check whether it can be opened?
If you want to allow that the agent has the source files (not just the avs) in a different place, then you don't want to enforce that the two avs are the same. Or does your check for "match" ignore any string constants within the avs or something like that?

[omion]

You're right. I suppose I didn't mention one important thing, which I knew but didn't bother to say ;) :
All the AVS source files should be relative to the AVS, or, as akupenguin says, they won't work in a different directory. This includes the .D2V file: in DGindex, you should DEselect "Options > Use Full Paths" when making the D2V.

And I check for exact equality with MD5, so there's not a whole lot of room for tweaking the file.

[ChrisBensch]

I'm using the new 1.04 with "agent based encoding" and I have to say I like it. My network util is at about .25-.30% on a gigabit network. My normal MeGui fps is about 8-9fps on this particular movie and with two machines I'm getting about 15fps. I expect that when I add the other two, I could get around 27-29fps total...faster than realtime for my HDTV encodes. I can't wait to see this sort of thing in MeGui!

[thetrueavatar]

Hello,
First thank you for this good intiative because distributed encoding is really cool.
However my main problem is that I would like to distribute my encoding on friends computer instead of my owns computers.
So the high network requirement seems too high.

So I have a suggestion to reduce a bit the network bottleneck. Is it possible to do a quick lossless compression on the controller (with hyufhyuv for example) that could reduce DVD size by 2 instead of sending raw video on the network? I have heard that Huffyuv was really fast and I suppose that time lost for lossless encoding/decodingto will be compensate by the gain over the network connection. This seems not too complicate to implement if we use agent base encoding and could provide a real performance improvement.
What do you think about it? Does it make sense?

[omion]

I'm using the new 1.04 with "agent based encoding" and I have to say I like it. My network util is at about .25-.30% on a gigabit network. My normal MeGui fps is about 8-9fps on this particular movie and with two machines I'm getting about 15fps. I expect that when I add the other two, I could get around 27-29fps total...faster than realtime for my HDTV encodes. I can't wait to see this sort of thing in MeGui!
Great! Thanks for the feedback!

Hello,
First thank you for this good intiative because distributed encoding is really cool.
However my main problem is that I would like to distribute my encoding on friends computer instead of my owns computers.
So the high network requirement seems too high.

So I have a suggestion to reduce a bit the network bottleneck. Is it possible to do a quick lossless compression on the controller (with hyufhyuv for example) that could reduce DVD size by 2 instead of sending raw video on the network? I have heard that Huffyuv was really fast and I suppose that time lost for lossless encoding/decodingto will be compensate by the gain over the network connection. This seems not too complicate to implement if we use agent base encoding and could provide a real performance improvement.
What do you think about it? Does it make sense?
It makes a lot of sense, actually. Although I'd probably make my own compressor, since it's about as much work to do that as it is to link to C code from OCaml. I've been toying with various compression settings, but it may be a while before I get it working.

The issue may be that the network connection is still too slow. My cheapo DSL connection can upload ~30KB/s. Uncompressed DVD-res video is ~500KB/frame. This means, on my connection, I could send 1 frame every 16 seconds (that's 0.06fps!) Even with 2:1 compression, that would still be extremely slow.

With computers that are so different in apparent speed (yours vs. whatever is on the end of the 0.06fps connection), it will probably slow down the encoding when compared to just using your computer.

Probably the optimal solution would be to use agent-based encoding and somehow share the DVD files with your friend (like with a shared folder in Windows) In this case, you would be transmitting the original compressed frames, so he might get ~1fps (a big improvement!) I don't know how well shared folders hold up over the internet, though.

[Inventive Software]

Forgive me if I've missed something, but if all the computers used for encoding have the source files that the agent has, would that not reduce the load over the network, i.e not transmitting the uncompressed AviSynth output? Just my $0.02, as I think anything to reduce the network dependancy to something justifiable like D1 over 100 Mbit networks (not possible at the moment as the data rate is something like 124 Mbits/sec) would be a good thing. :)

[omion]

Forgive me if I've missed something, but if all the computers used for encoding have the source files that the agent has, would that not reduce the load over the network, i.e not transmitting the uncompressed AviSynth output?

Yup. That's what agent-based encoding does. The controller tells each of the agents to do some range of frames, then the agents send back the compressed MKV files.

The original method of encoding, controller-based, acts like you described, but I added the agent-based encoding in 1.04.

Just my $0.02, as I think anything to reduce the network dependancy to something justifiable like D1 over 100 Mbit networks (not possible at the moment as the data rate is something like 124 Mbits/sec) would be a good thing. :)
I don't think I understand what you mean. You can't send D1 over 100mbps networks real-time, but if x264 can only encode 1/2 real-time, then the encoder is the bottleneck anyway.

I'm also writing a program which will analyze different compression methods to decrease network usage in case the source files are not accessible to the agent, but I don't know when that will be added.

[Inventive Software]

If you have enough computers for the encoding, then it'll be possible to encode more than realtime (we're talking people with access to university networks and suchlike), which is why reducing the raw transfer data over the network becomes critical, otherwise the network becomes the bottleneck.

Besides, it's fairly well known that 100 Mbit is the absolute maximum transfer in peak conditions with everything perfect (based on people stating dialup @ 56K never reaches 56K), and it's more likely to be around 90 Mbit tops (average ;)). Those lucky people with relatively new motherboards with Gigabit ethernet don't have to worry so much. ;)

[TriHFH]

Hey, thanks a lot for working on this project, I think it has a ton of potential. I tried using x264 farm but I am getting this error:

ERROR: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error

but i opened it in virtualdubmod and it opened fine w/o any errors. What can I do to solve this/ what is causing these problems?

[omion]

Hmm... for whatever reason avs2yuv didn't like the AVS file. Post the output of the command:
avs2yuv -frames 1 -o NUL pb213.avs
Oh, and :script: ;)

[TriHFH]

Here's the script:

DGDecode_mpeg2source("F:\pb213\pb213.d2v",info=3)
ColorMatrix(hints=true)
TDecimate(cycleR=3)
TTempSmooth(lmdiff=1,cmdiff=2,strength=1,scthresh=8)
RemoveGrain(5)
#crop
#resize
#denoise

And the output of that command is:

c:\x264farm\controller\pb213.avs: 1280x720, 24000/1001 fps, 61376 frames

[omion]

OK, that's really weird. The command I gave you is almost exactly what my program tests for the data. The only thing I can think of is that the controller can't find avs2yuv. Where did you put avs2yuv?

Are there any other warnings / errors / suspicious lines before the error you posted?

(my program actually tests the command avs2yuv -frames 1 -raw -o NUL "avsfile" 2>&1
Try that, too, just in case)

[TriHFH]

Nope, I did not notice any weird lines/errors before my error. Here is a screensot of it:


That new command outputs this:

C:\x264farm\controller\pb213.avs: 1280x720, 24000/1001 fps, 61376 frames

http://yogurtrat.com/files/1/error_thumb.png (http://yogurtrat.com/files/1/error.PNG)

I wasnt sure where to put avs2yuv, so i put it in the controller folder and the main folder :P

[omion]

I wasnt sure where to put avs2yuv, so i put it in the controller folder and the main folder :P
Well, the "right" folder is anywhere on the path. Putting it in the controller's directory is great as long as you call the controller from that directory.

Unfortunately, everything looks fine. There are a few issues with your command line, but nothing that would cause it to die there (*)

I'll try to reproduce the error on my computer and see what could be happening. In the mean time, try out a braindead-simple AVS script and see if x264farm will take it. Something like this:colorbars()

(*) Since I don't know what's wrong, I'll just nitpick everything else:p :

The -B option needs to be "-B 4353kbps", to distinguish it from 4353% of the first pass size. It looks like x264fram will default to percent, which is probably not what you want. I should make a warning for that...
leave out --stats from both the first and second passes. x264farm keeps track of the stats file internally.
The --zones option should be outside the -1 and -2 options. That is, it should be an option of x264farm itself, not part of the string to pass to x264. Otherwise the x264farm ratecontroller will not be too happy (it gets easily confused;) )
"--no psnr" should be "--no-psnr"

[thetrueavatar]

why don't you transmit compressed source(mpeg2 for example) instead of raw?

[foxyshadis]

The -B option needs to be "-B 4353kbps", to distinguish it from 4353% of the first pass size. It looks like x264fram will default to percent, which is probably not what you want. I should make a warning for that...


Perhaps % below 100, and bitrate above 100, if no modifier is given?

[TriHFH]

I tried that avs and it seems to have worked (it is processing something) when i load it directly from avs2yuv, but when i load the avs into the controller, it gives me that same error, even with that simple avs.

And thanks so much for your help so far... I'm really excited to get this working :D

[omion]

why don't you transmit compressed source(mpeg2 for example) instead of raw?
It would be difficult to do through x264farm, since I would need to intercept the AVIsynth calls to the source files. You can, however, do it the roundabout way:
Share the folder with the source files on the main computer
Then use the "Map network drive" feature on each of the agent's computers to use the network folder as a separate hard drive (it does not need to have write access)
Then use "agent-based" encoding and use the new drive letter as the base directory

Perhaps % below 100, and bitrate above 100, if no modifier is given?
Yeah, that's probably what I'll do.

[omion]

I tried that avs and it seems to have worked (it is processing something) when i load it directly from avs2yuv, but when i load the avs into the controller, it gives me that same error, even with that simple avs.

And thanks so much for your help so far... I'm really excited to get this working :D

One more possibility, then I'm stumped. Are you using Windows 2000/XP? If not, many of the functions that I use (possibly including the avs2yuv call) won't work. If you aren't using 2000/XP, neither the controller nor the agent will work at all so you should focus on upgrading instead of video encoding ;)

If you have Windows 2000 or XP, then try out this controller (http://omion.dyndns.org/x264farm/x264farm-1.04+_controller-only.rar). It will die before anything gets encoded even without your error, but post all the lines that start with "DEBUG:" That should narrow down the problem.

[Kripsy]

Is x264farm capable of encoding a h264 source file, or is it only mpeg stuff? I ask because I am encoding some rather large 1080p caps to something a bit more cpu friendly with megui currently, but it takes me upward two days to do it!