x264farm: distributed encoding [Archive]

omion

5th November 2006, 09:19

Well, after about a year of working on this, let's see how it works in the real world! :D

I have been making a parallel encoder for x264 which can be distributed across many computers in a network. I have not done any serious benchmarking for it, but it is faster than running it on 1 computer. It was designed to be extremely portable; the only things you need on most of the computers is the agent program (supplied) and x264. All the AVIsynth processing is done by a central computer. I have personally tested it running on a Windows x64 computer, a 64-bit Linux box, and an Intel Mac Mini.

The idea is similar to tobias's ELDER, but designed from the ground up to be used on the network. It also supports fun stuff like resuming encodes (sometimes with a big penalty, but it's better than starting over)

A few notes:

It is still a work in progress, so it will make a file called out_dump.txt in the encode directory containing a bunch of debugging stuff.
You really need a fast network, and at least one decent computer. I sacrificed network efficiency and scalability for ease-of-use and portability.
I find it very easy to use, but I'm a bit biased. There are probably many usability issues that still need to be worked out.
All the "config.xml" files in the downloads are for my setup only. You want to change all the options around to reflect your setup before you start it up.

Eh... I guess that's all for the announcement. Have fun!

Readme (http://omion.dyndns.org/x264farm/x264farm.html)

DOWNLOAD VERSION 1.15:

WINDOWS BINARY (http://omion.dyndns.org/x264farm/x264farm-1.15_win32.rar) (controller and agent)
SOURCE (http://omion.dyndns.org/x264farm/x264farm-1.15_src.zip)
(Agent binaries for OSX (intel) and 64-bit Linux are available on request. I can't build 32-bit Linux binaries, but it's pretty easy to do yourself)

You also need the following:

x264 (http://mirror05.x264.nl/Cef/). NOTE: Currently, only builds done with MSVS will work with x264farm's controller-based encoding in Windows. I used to build my own, but I've found that Cef's builds work great so I've decided to stop making my own.
avs2yuv (http://akuvian.org/src/avisynth/avs2yuv/) on the controller computer.
nice.exe (http://omion.dyndns.org/x264farm/nice-20070415.rar) for encoding with different priorities on a Windows machine. If you just want to use normal priority, just set <nice> to 0 and it will be ignored. (source (http://omion.dyndns.org/x264farm/nice-20070415_src.rar). updated 2007-04-15)

when building from the source:

You need OCaml, extlib (http://sourceforge.net/projects/ocaml-lib/), and xml-light (http://tech.motion-twin.com/xmllight.html)
Run "make" in the /common directory FIRST
Run "make" in the /controller and /agent directories.

It's harder than it should be, but I haven't spent a long time polishing the build process

CHANGELOG:
1.15 (2007-10-26):
Added ad-hoc agent discovery (although disabled by default). Agents may be added while the encoding is running, even if they are not specified in the controller's config.xml file.
Ability to specify log file location on command line for both agent and controller.
Faster controller-based compression.
Changed the bucket-based agent cache to a single circular buffer.
Changed the default port from 50700 to 40700. This will only affect you if you re-use one of my example config files.
1.14 (2007-10-03):
Fixed an inconsistency with running x264 with spaces in the path.
Added the capability to change the avs2yuv executable in the config.xml file.
Add timestamps to the error messages and when the display was last updated (counting from when the program was started).
Show "~" for errors if there are aren't enough to display, and made the default number of errors displayed 4.
Doesn't update the FPS on the second pass if the encoding failed.
Properly handle dead agents on the second pass.
1.13 (2007-07-03):
Added the --batchmult option to help with one agent checking out a big job at the end and making everything else wait for it.
Added previous split points to the 1st-pass temp file. This means that encodes will start up faster after having been stopped.
Optimized the splitter to try to get the jobs out faster in the beginning.
1st-pass agents will now try to avoid working on jobs which may have incorrect start points, in order to minimize the number of re-encoded frames.
Put ETA display back for both passes. Note that the ETA given in the first pass measures the end of the first pass only, and is incredibly inaccurate.
Second pass GOPs are no longer encoded chronologically.
Completely redesigned first pass to be easier to work on.
1.12 (2007-05-30):
Added a heartbeat thread to each agent when agent-based encoding starts. Agents will send a signal to the controller every 10 seconds. If the controller does not receive a signal within 30 seconds, that agent is given up for dead. This should greatly reduce on the amount of stalled encodes.
Controller output is much more organized, and no longer prints huge amounts of scrolling garbage. The out-dump.txt files are just as garbage-filled as before, though.
1.11 (2007-04-28):
Fixed an overflow in the Matroska timecode calculation.
Second-pass equality check takes bitrate into account.
Changed the GOP layout from B-trees to red-black trees (not that anybody would notice the difference).
1.10 (2007-04-25):
Implemented per-pass compression, so that the first and second passes have different compression priorities
More precise file size (tunable with the --sizeprec option)
Ratecontrol handles extreme bitrates better
The --keeptemp option actually works now...
Minor sanity checks when writing the files
1.09 (2007-03-10):
Agent now deletes temp files which are older than a specifiied amount of time (by default 1 week).
Agent now uses niceness when both nice and piped input are being used.
2nd pass bitrate should be more accurate.
1.08 (2007-02-07):
Redesigned first pass. It should pick jobs more intelligently. The occasional problem with credits taking a very long time to encode is minimized.
More verbose garbage printed to the screen regarding the new first pass mode. It will disappear shortly.
Second pass should recover from x264 complaining about the minimum bitrate. Some jobs may fail a few times, but they will encode eventually.
The agent and config.xml format are unchanged.
1.07 (2007-01-05):
Fixed an issue with the controller computer rejecting network connections after an hour or so
Made some of the more pointless messages log-only
1.06 (2006-12-25):
2nd pass resuming works again (sorry, guys! :o )
Plugged the hole in the first pass which could cause frames which are being worked on to be re-encoded
Compression should be faster. There are no more checksums, and the compression loop is faster.
Fixed up the re-encode part to ask for confirmation instead of just quitting, and will just automatically restart if the old output file does not exist anymore.
Fixed a potentially major bug when the output files aren't written if the requested bitrate is below what x264 thinks is the minimum bitrate.
1.05 (2006-12-12):
Added optional compression for controller-based encodes. Currently extremely slow, though, in order to check for validity.
Added --nocomp switch to force compression off.
Added <pipe> setting to agent's config.xml file to locally pipe data to x264 in agent-based encodings.
Added tables in 2nd pass to show the performance of the various agents.
Controller now searches its own directory for avs2yuv.
Hid a bug involving the 1st pass splitter re-splitting ranges which are already allocated to agents not using controller-based encoding.
1.04 (2006-11-24):
Made agent-based encoding.
User-selectable number of agent cache buckets.
Added many ratecontrol parameters to the controller, and pass them to x264.
Lowered latency and memory usage due to sending data across the network.
1.03-152 (2006-11-20):
Fixed an error when the number of frames per second was an integer.
1.02 (2006-11-17):
MAJOR rewrite that honestly shouldn't have needed to happen.
Made the agent per-computer instead of per-core and fixed a bug with remote computers not getting enough data.
Controller only connects once per session, instead of once per job.
The code itself is much better-written.
Config file syntax completely changed.
1.01 (2006-11-09):
Made the --rerc option in order to change the re-ratecontrol frequency during the third pass.
Made the network more resilient to delayed-ACK problems.
Removed some of the more verbose printing (although everything is still in out-dump.txt).
Added simple FPS display (although it's buried in the debug info).
Fixed the batch length (now 5000, was 25000 but claimed to be 2500).
1.00 (2006-11-04):
Initial release. Rampant bugs and programmer hacks predominate.

WHEN REPLYING:
If you post really long chunks of text (like from out_dump.txt), use the [ code ] tag and it will make it scrollable. That will prevent the page from overflowing with a bunch of verbose garbage that x264farm likes to output ;)

crypto

5th November 2006, 11:02

What a cool idea. How do you schedule the tasks? Do you run a scene detection and distribute scenes for encoding?

Disabled

5th November 2006, 15:26

DDogg

5th November 2006, 15:52

Sounds exciting. Suggest you, or someone else consider running through a hypothetical setup using 2 or 3 windows XP PCs. PC-master, PC-slave1, PC-slave 2 in a step 1,2,3 format

Once I can get it working, perhaps I could do a how-to that the average Joe-tooluser like me could follow.

My set up is 7 XP machine on a gig backbone, with the master being a OC'ed 4.5 Gh Intel D930, slave1 - hyper-threaded 3.2 P4, slave 2 - Athlon XP 3200, plus various others along those lines. I'm happy to help others if you can help me get going.

omion

5th November 2006, 18:01

What a cool idea. How do you schedule the tasks? Do you run a scene detection and distribute scenes for encoding?

@Disabled:
I forgot to mention one thing: x264 compiled for Windows with gcc won't work currently. I made a build with MSVS that should solve your problem.

There's nothing special about my build, it just uses straight SVN. You can get build 598 here (http://omion.dyndns.org/x264farm/x264-598_omion.rar)

@crypto:
Basically, that's how it works. For the first pass, the video is broken up into large segments, where the last frame is determined by a simple scene-detection algorithm. The second pass is split up by scene, which is already known since they're in the stats file.

@DDogg:
I made a test with my one computer, and x264farm was actually around 15% slower than not using it, due to all the overhead. Unfortunately, all of my computers are different, so I can't make a scalability test, but I have noticed that even adding my wimpy Core Solo Mac Mini makes it faster than just one computer.

It would be nice if somebody did some tests with it. However, note that, by default, x264farm will re-encode 5% of the scenes that came the farthest from the ratecontrol prediction. Anybody interested in a comparison should turn that off with "--3ratio 0"

In terms of setting up, it is a bit difficult. Here's what I would do:
1. Copy the agent directory to any computer that you want doing the encoding.
2. On each computer, make as many copies of the config.xml file as you have cores (dual-core computers should have 2 different xml files, etc...)
3. Edit the config files to point to the right x264 executable, and each one must have a different port.
4. Run the agent on each computer, once for each config file, with "agent --config <some_file.xml>"
5. Extract the controller directory to the main computer
6. Edit the controller's config.xml file to point at all the agents you have running. You need to know the computer's IP address and the port that you assigned each one.

After that, the setup is done. Now you can start encoding. The basic command line is:
controller --first "<first_pass_options>" --second "<second_pass_options>" -b <target_bitrate>kbps --avs <input_file.avs> -o <output_file.mkv>

Look at the HTML file included in the release for what you should and shouldn't include in --first and --second.
Also note that the -b needs to be in the form "-b 123kbps" or "-b 123%", where the second form indicates a percentage of the first pass bitrate to use.

Disabled

5th November 2006, 18:30

omion

5th November 2006, 18:53

Many thanks for the x264 build, its working now.
I tried encoding a file on the host and on my laptop with your framework and it kinda worked. The laptop only got used about 20% of the encoding time, but its slow as hell anyway. I sometimes get the error 'Agent croaked with exception "End_of_file"; restarting' on my laptop, that might be the reason for it not being used that much.

That error's actually OK... sometimes. I think it happens when the controller connects, realizes there's nothing to do, and disconnects. It sounds like you have something else going on, though.

And I can't start a second encode of the same file without deleting the tmp files manually.

If you want to do the same encoding on the same file, it should reuse the temp files (in order to properly resume). If you really want to start from scratch (like for testing), use the --restart switch

I will not go out and benchmark this tool, as I don't have that powerfull machines here.
if you provide a linux 32bit build, I might try it at the university some time. But then I realise it still needs avisynth I do not have there, so I need to host it on a laptop...

*edit*
I just tried to encode with my laptop being the only client, and I get the same error as before and its not encoding the whole time, I would say the laptop is idle about 30% of the encoding time now.

I don't have an easy way to make 32-bit binaries with my Linux box, so I think you'd have to see if someone else can build it.

Now, when you say it's idle, does it do a job for 30% of the time and wait for the other 70%, or does it say it's doing something but it's only using 30% of the CPU?

Disabled

5th November 2006, 20:19

That error's actually OK... sometimes. I think it happens when the controller connects, realizes there's nothing to do, and disconnects. It sounds like you have something else going on, though.
[...]
Now, when you say it's idle, does it do a job for 30% of the time and wait for the other 70%, or does it say it's doing something but it's only using 30% of the CPU?
I'm refering to the "laptop is the only agent" situation now. The CPU on the laptop (on the controller as well) is idle for about 30% of the time and completely busy the rest.
I'm not sure if I read the (messy) log correct, but I think after an idle phase the above mentioned error is shown and then it continues to encode.
I just started a second run to verify the results.
I did two tests with both machines running an agent, the first time the laptop was used just once, the second time I noticed four phases the CPU was maxed out on the laptop.

I don't have an easy way to make 32-bit binaries with my Linux box, so I think you'd have to see if someone else can build it.

Well perhaps I will look into compiling the source myself. My Linux box is actually faster than the laptop I'm using :)

*edit*
Now it happened again. Both machines were idle for about 2 minutes, then the "End_of_file" Error and after a few secs its encoding again...

CruNcher

5th November 2006, 20:47

@omion
great work, but this wouldn't work with ABR 1pass encodes would it ?

omion

5th November 2006, 21:28

I'm refering to the "laptop is the only agent" situation now. The CPU on the laptop (on the controller as well) is idle for about 30% of the time and completely busy the rest.
I'm not sure if I read the (messy) log correct, but I think after an idle phase the above mentioned error is shown and then it continues to encode.
I just started a second run to verify the results.
I did two tests with both machines running an agent, the first time the laptop was used just once, the second time I noticed four phases the CPU was maxed out on the laptop.

Well perhaps I will look into compiling the source myself. My Linux box is actually faster than the laptop I'm using :)

*edit*
Now it happened again. Both machines were idle for about 2 minutes, then the "End_of_file" Error and after a few secs its encoding again...
2 minutes... weird. That sounds like a network issue. Are they connected wirelessly?

@omion

is this only working for a 2pass encode ?
Yup. It's only 2-pass because that's all I use :p It would be quite difficult to turn it back into a 1-pass, as both passes are set up quite differently.

Disabled

5th November 2006, 21:39

2 minutes... weird. That sounds like a network issue. Are they connected wirelessly?

Nope, 100% pure old fashioned 100Mbit Local Area Network
In addition to that I would say there was no traffic at all during that time... (doing a third run...)

*edit*
The controller sends Data to the agent. The Laptop starts encoding and it finishes ("Exited 0") on the Laptop. The Agent then shows "lap recieving stats" for a little over a minute. And then contines with "lap done recieving stats", but the Agent then shows the EOF exception and continues to encode...
I guess I'll send you the whole log. I have the feeling the "revieving stats" message is just once in the log file, while "done receiving stats" is seven times, and I counted five idle times...
Encoding time: 20 Mins, with about 5 mins idle

pyrates

5th November 2006, 22:49

Is there a way to distribute the avisynth script as well or does that still need to be worked on just one pc only for now? Because the one I use is quite processor intensive but gives awesome results.

omion

6th November 2006, 01:39

pyrates

6th November 2006, 01:51

I had initially made the program to distribute everything, but that requires being able to have the input files available to all computers, and also to have AVIsynth and all the filters available to all computers.

Synchronizing all the computers and especially making it work with non-Windows computers proved to be just about impossible. It was far too much work for its usefulness, so I basically got rid of it as soon as I got it working.

Well that would be exactly what I'd need. Plus avisynth is something that's only available for computers that run windows on it. And without it, I just can't use this form of distributed computing. I'd have no problem having the source files copied onto all the computers anyways.

ChrisBensch

6th November 2006, 05:17

Wow, thanks for your effort in distributed encoding!
I gave it a quick try and didn't get it working.
The agent throws an error:
2006-11-05~15:12:13.01 Doing the following:
x264.exe --fps 10000000/333667 --pass 1 --stats "C:\agent-win32\tmp\0 893 969666.txt" -o NUL - 720x480
2006-11-05~15:12:14.34 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-05~15:12:14.35 DOING!

This is the controller config.xml
<config>
<temp>C:\controller-win32\tmp</temp>
<agents>
<agent name="local">
<ip>192.168.0.1:50722</ip>
</agent>
</agents>
</config>

And the agent config.xml:
<config>
<temp>C:\agent-win32\tmp</temp>

<port>50722</port>
<x264>x264.exe</x264>
<nice>00</nice>
</config>

starting the controller with this cmd
controller.exe -b 1500kbps --avs test.avs -o test.mkv

I'm using the latest x264 build r598A
If you need more information, just tell me.

I'm having the same problem, and I've replaced x.264 with the correct version from this thread. Here is my output:
---------------------
D:\TEMP\agent-win32>agent --config config.xml
Started up the print mutex
Using config file ".\config.xml"
Temp dir: "D:\\TEMP\\cpu1"
Affinity: ""
Port: 50722
x264: "x264.exe"
nice: 10
Connecting...
Listening...
DOING!
Got a connection from 127.0.0.1:1287
Doing stuff
Doing the following:
nice -n 10 x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --ana
lyse none --vbv-maxrate 25000 --me dia --merange 12 --progress --no-psnr --fps 24000/1001
--pass 1 --stats "D:\TEMP\cpu1\0 1298 6414bb.txt" -o NUL - 1280x608
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2

Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
DOING!
Got a connection from 127.0.0.1:1290
Doing stuff
Doing the following:
nice -n 10 x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --ana
lyse none --vbv-maxrate 25000 --me dia --merange 12 --progress --no-psnr --fps 24000/1001
--pass 1 --stats "D:\TEMP\cpu1\0 1298 5a6db2.txt" -o NUL - 1280x608
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2

Agent croaked with exception "Unix.Unix_error(An existing connection was forcibly closed b
y the remote host.
, "recv", "")"; restarting
DOING!
---------------------

I don't think it's an x264 setting as it looks as though it's starting. I'll try without the VBV maxrate and see what happens.

omion

6th November 2006, 05:44

That sounds sort of like something went wrong on the controller side. Does the controller output say anything about an error or exception?

ChrisBensch

6th November 2006, 05:44

By removing the --vbv-maxrange stuff it got much further, I have two machines, one with dual cpu and one with dual core (controller running two agents also). It kept sending all frames to one machine, so I added "--batch 2500 --split 250" and it sends data to the other machine, but my two local agents both die with:

DOING!
Got a connection from 127.0.0.1:1453
Doing stuff
Doing the following:
nice -n 10 x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --ana
lyse none --me dia --merange 12 --progress --no-psnr --fps 24000/1001 --pass 1 --stats "D:
\TEMP\cpu1\0 12720 31d72a.txt" -o NUL - 1280x608
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2

Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting

So now I'm kinda stuck...any ideas?

Here is my avs:

LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\DGDecode.dll")
LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\TIVTC.dll")
MPEG2Source("Hero.d2v")
Crop(16,134,-16,-142)
TFM(d2v="Hero.d2v")
#TDecimate(cycleR=3)
TDecimate()
LanczosResize(1280,608)

Here is my controller startup line:

controller -b 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --merange 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weightb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --merange 12 --progress --no-psnr" --avs "D:\$VIDEO_WORK$\WORKING\Hero.avs" -o "D:\$VIDEO_WORK$\WORKING\Hero-farm.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

ChrisBensch

6th November 2006, 05:48

Here is the controller output:

edited...

Notice the error:
> LOCAL CPU 2 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION "Unix.Unix_error(_, \"send\", \"\")"

omion

6th November 2006, 06:38

So the local machine croaks, but the agents on the other machine work as expected? That is strange.

Try out this (http://omion.dyndns.org:8080/x264farm-net_test.rar) controller, and see if that helps. If it doesn't, see if the line Local CPU 2 sent 4096 bytes occurs between the lines Local CPU 2 sent first pass options
> LOCAL CPU 2 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION "Unix.Unix_error(_, \"send\", \"\")" Also, the error line should now give a number where the underscore is. If the error still occurs, write back with that number.

I must admit, I don't really know why it's dying, but I think I can figure it out given enough time (and patience!)

PS. You can delete the controller output now. I copied it to my computer, and I don't think anybody else will be able to comprehend it ;)

ChrisBensch

6th November 2006, 06:44

omion

6th November 2006, 07:41

With the new controller, here's what I got:

Local CPU 1 sent zone string ""
Local CPU 1 sent range (0,2582)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522890,recv,)
Local CPU 1 added (0,2582) back to the split list, since it caught an exception
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX

2522890?! That's... very big. That doesn't look like a normal error code. Maybe they aren't as portable as I thought.

Well, it's odd that you now get the error in a different place (from a "recv" instead of a "send") That really makes it sound like something else in the system is getting in the way of transferring the data.

Just a few questions: what version of Windows do you have, and do you use a firewall / antivirus?

I have to get to bed soon, but I updated the test controller (http://omion.dyndns.org:8080/x264farm-net_test2.rar) to give me a bit better info about what's going on. Try it out and post the log like you did in post #20. I should be able to look at it again tomorrow morning.

Disabled

6th November 2006, 08:32

I ran a test with the test2 controller and the problem is still there:
2006-11-06~08:04:58.32 lap receiving stats
2006-11-06~08:04:58.51 lap recieving stats
2006-11-06~08:07:58.56 lap done recieving stats

By the way, the "recieving stats" is printed out twice the first time and never again...
If I should have tested the prior build just tell me.

ChrisBensch

6th November 2006, 08:43

controller v.test2

Here's the newest output:

> LOCAL CPU 1 GOING ON SPLITTER_LIST_CONDITION
Local CPU 1 getting a split
Local CPU 1 got (2582,5175)
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX
> HOME-MAIN 10001 GOING ON SPLITTER_LIST_CONDITION
> LOCAL CPU 1 BROADCASTING SPLITTER_LIST_CONDITION
> HOME-MAIN 10001 WAITING ON SPLITTER_LIST_CONDITION
Local CPU 1 do_job 2582-5175
Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522900="The opera
tion completed successfully.\013\n",recv,)
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX
Local CPU 1 connect_and_do
Local CPU 1 connected to 127.0.0.1:50721
Local CPU 1 connected!
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX
Local CPU 1 getting a split
Local CPU 1 got (2582,5175)
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX
> LOCAL CPU 1 BROADCASTING SPLITTER_LIST_CONDITION
> HOME-MAIN 10001 GOING ON SPLITTER_LIST_CONDITION
Local CPU 1 do_job 2582-5175
> HOME-MAIN 10001 WAITING ON SPLITTER_LIST_CONDITION
Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522848="The opera
tion completed successfully.\013\n",recv,)
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX

I started with two computers, one with just two agents, one with a controller and two agents. I then removed one of the agents from the controller machine for a total of 3 agents and one controller. The weird part is that the agents on the other machine (remote agents only) starts to work fine, but the local encoder machine is the one that borks. Hope this helps.

708145

6th November 2006, 08:45

@omion
great work, but this wouldn't work with ABR 1pass encodes would it ?

I agree: Great work omion :)

And maybe it is useful for you guys that I throw a windows cluster mode for ELDER into the ring as well :p
It's just that I leave 2pass x264 encoding to ominon for the time being :) But ELDER beta5b will feature a distributed x264 crf mode.

edit: ELDER beta5b is out now -> http://forum.doom9.org/showthread.php?p=718465#post718465

Distributed abr is not more work so I can do it for beta5c if you insist, Cruncher :D

bis besser,
T0B1A5

ChrisBensch

6th November 2006, 08:48

I removed all agents on the controller box and just used a remote machine, so far so good. I have two other computers that I'll be adding this week...I'm excited about getting this weird thing fixed. I have three machines with Pentium D 2.8GHz and one Dual AMD Athlon MP @ 1.6GHz (this may become my controller and let the other three work).

ChrisBensch

6th November 2006, 16:24

I'm also going to try the new ELDER...I have 5 machines at home just waiting for some distributed encoding!

omion

6th November 2006, 18:26

ChrisBensch

6th November 2006, 18:44

Here's the output from test3 controller:

Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522470="The opera
tion completed successfully.\013\n",recv,)
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception

omion

6th November 2006, 18:48

@ChrisBensch:
Fun. It looks like the error was "no error". That's why the number was so big. I'll have to look into that...

ChrisBensch

6th November 2006, 18:55

yeah that's weird, the encoding still works on remote machines, just not the local machine...

omion

7th November 2006, 01:48

yeah that's weird, the encoding still works on remote machines, just not the local machine...

OK. Try out this one (http://omion.dyndns.org:8080/x264farm-net_test4.rar). It may give a bunch of warnings, but it should keep going. I don't know if the end result will be watchable, though. Test out the final file and make sure it plays correctly.

I just made the programs ignore that particular error, which would help if it's a superfluous error, but of course it still won't work if the error is there for a reason.

ChrisBensch

7th November 2006, 20:54

Here's the latest...

Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
Got a strange error while receiving. Check output file!
Got a strange error while receiving. Check output file!
Got a strange error while receiving. Check output file!
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION "Failure(\"Tag \\\"\\\\000\\\\000\
\\\000\\\\000\\\" (length 1048576) was not expected here\")"
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception

ChrisBensch

8th November 2006, 02:14

Maybe this is a dumb question...but should I be updating the agent.exe too? It's included in the build but so far we've only been working on the controller...

omion

8th November 2006, 02:20

@ChrisBensch:
Updating agent.exe probably wouldn't help anything. I've just been putting them both together because I've been updating them both at the same time, but the agent has stayed basically the same.

Unfortunately, I'm out of ideas. Since the last build didn't work, that means the error is actually an error and not something to be ignored.

Actually, do you have Windows 98 / Me? I've been told that 9x doesn't work for transferring data the way I do...

ChrisBensch

8th November 2006, 03:06

All of my PC's are XP SP2, I have Symantec AV Corp running on all boxes. It's been about a year since any of them have been reloaded so it could be something on my side. I'll try to find the time to reload them and get back to you with the results. Regardless of what happens, I should still be able to use my other 4 boxes for agents and the error one for controller only (which seems to work fine). Thanks for all the hard work. This is a project which I have been working on for some time and have been using it to learn software dev skills.

omion

8th November 2006, 04:24

Regardless of what happens, I should still be able to use my other 4 boxes for agents and the error one for controller only (which seems to work fine).

That sounds good. The agents on the controller's computer usually don't add up to much, since the computer is busy rendering the AVS script (especially with 4 other computers!)

Before you do a complete Windows reinstall, it would be good to try a different computer as the controller. You could probably get away with installing AVIsynth, using a simple colorBars() AVS file, and see what happens to the problem. Even though that may not be viable with normal encodes, it would be interesting to see which one of the following happens:
1) No errors anywhere
2) Errors with the old controller
3) Errors with the new controller

ChrisBensch

8th November 2006, 07:45

I'll give it a shot for ya

pimpMyHD

8th November 2006, 16:41

elguaxo

8th November 2006, 17:12

* You really need a fast network

This project sounds very interesting! Now my first question before I start. My network is now 54Mbps wireless, with really good reception. Does it make sense to build a "x264farm" with that network speed?

Thanks.

omion

8th November 2006, 17:50

considering the low speed encoding with HD resolution ( around 1 to 10 fps max ) could your system can be used over internet ?
I have several server over the web, and it will be fine if i could use their ressources to improve me encoding times.

regards,
Probably not. Remember that the low speed for encoding is because there is more data to encode. That means HD will actually be worse than smaller frame sizes.

Let's say you have a 2-hour movie @ 1920x1080x24, and half of it being sent out on the internet. That means you will need to upload 1920 * 1080 * 1.5 * 24 * 3600 bytes, or 250GB! On my internet connection, that would take around 3 months...

This project sounds very interesting! Now my first question before I start. My network is now 54Mbps wireless, with really good reception. Does it make sense to build a "x264farm" with that network speed?

Thanks.
54Mbps is probably OK. My experience with wireless is that the actual transmission speed is about 1/3 of the connection speed. For a 54Mbps connection, you can transmit around 4.5 million pixels per second, which is around 13 DVD-size frames per second.

What that means is that if all your remote computers can encode faster than 13 frames per second, they will be limited by the network connection. If they can't make 13 fps, wireless will be more than enough.

One more note: When I had my computers on a wireless connection, I actually saw one frame corrupted in such a way that the frame checksum said it was OK. Luckily, I had Ethereal running on each computer, so I could verify that the frame data was different. I realize that is an extremely unlikely occurrence, but I haven't trusted wireless ever since.

PS: My build (http://omion.dyndns.org/x264farm/x264-600_omion.rar) of x264 r600 is out.

elguaxo

8th November 2006, 19:18

Great! I will start my x264farm :cool:

foxyshadis

9th November 2006, 02:39

54Mbps is probably OK. My experience with wireless is that the actual transmission speed is about 1/3 of the connection speed. For a 54Mbps connection, you can transmit around 4.5 million pixels per second, which is around 13 DVD-size frames per second.

What that means is that if all your remote computers can encode faster than 13 frames per second, they will be limited by the network connection. If they can't make 13 fps, wireless will be more than enough.

One more note: When I had my computers on a wireless connection, I actually saw one frame corrupted in such a way that the frame checksum said it was OK. Luckily, I had Ethereal running on each computer, so I could verify that the frame data was different. I realize that is an extremely unlikely occurrence, but I haven't trusted wireless ever since.

If you use WPA encryption, that can't happen. (Although it might break the connection or cause a long reset instead.) Still, I have plenty of experience with wireless and ethernet drivers getting wedged opening and closing connections rapidly, and cheap routers rebooting during transfers - driver bugs or overheating - especially with WPA. Wireless reliability is around 12 years behind; fortunately, '95 was around when ethernet started getting reliable. :p

Maybe you should use md5 & sha1 for hashes. ;)

Omion, what do you think about using zlib for optional mild compression, to help with the slow network/fast system disparity? You can probably rip some code out of TCPDeliver for doing it efficiently.

pimpMyHD, you'll probably need to look at ELDER, which directly uses the lossy original and performs filtering on the slaves. Even that's bound to be horribly slow, especially since it works best with the full source available, not chunks.

ChrisBensch

9th November 2006, 03:38

Alright, I've got one controller and two slaves, one a dual athlon mp (2 @ 1.66GHz and another a dual core 2.8GHz. I'm going to encode "The Fog" in HD @ 1280x608x23.976fps. I'll be adding another dual core 2.8GHz soon.

Here's my script:

LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\DGDecode.dll")
LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\TIVTC.dll")
MPEG2Source("The Fog-HDTV.d2v")
Crop(0,0,-2,-8)
TFM(d2v="The Fog-HDTV.d2v")
#TDecimate(cycleR=3)
TDecimate()
LanczosResize(1280,608)

Normally on one machine I get about 8-10fps on the first pass and 2-3fps on the second...so let's see what we get!

aicjofs

9th November 2006, 06:47

You say probably need "Avs2yuv" installed.

Does WinXP machines need this? If so where do I install the .exe (the path, controller directory)? I get an "avs2yuv" exited with error 1 on the controller upon startup.

edit: my bad, seems the d2v I had made was made with the a different version of DGIndex then the plugin dll.

omion

9th November 2006, 06:55

Yup, every controller computer needs it. You can put it either somewhere on the path, or wherever you are running the controller from (the controller's directory will not work if you start the controller from a different directory)

The "avs2yuv exited with error 1" error generally means you have an error in your AVS script. When this happens to me, I open the script in VirtualDubMod and see what it says about the script.

[ edit: I see where the confusion comes from. What I meant was that you do need avs2yuv installed, so you probably need to download it (since I don't think most people have it). I'll make that clearer in the first post. ]

aicjofs

10th November 2006, 06:44

omion

10th November 2006, 07:06

Working nice on a P4 3.2Ghz 1GB RAM, P4 2.8 Ghz 512MB RAM and a Dual Core Opteron at 3.0Ghz, 2GB RAM and Gigabit ethernet network, leaving the other rigs out for the time being. Although the controller with 2 agents brings the Opteron system to it's knees. I never intalled nice.exe and just configured the xmls to 0 on all systems.

When you say "bring it to it's knees", do you mean that all the avs2yuv instances max out the CPU, or is it a memory issue? If the problem is with the memory, use SetMemoryMax(64) at the top of your AVS script. x264farm actually doesn't benefit from a huge frame cache, so the only reason to have more than that is if you have other wide-reaching filters.

Also, if you use many post-processing filters, you can save CPU time by using the --fastavs option. Make a copy of the AVS script and delete all the spatial post-processing filters from it. Then pass it with the --fastavs option. It's not much of a savings, but it should help.

My current setup is like this
farm.avs:
SetMemoryMax(64)

mpeg2source("d2v.d2v",cpu=6,iPP=false)
Telecide(order=1,guide=0,post=2,back=1,bthresh=40,nt=20,ovr="over.txt",show=false).Decimate(mode=0,show=false)

coloryuv(levels="TV->PC")
aWarpSharp(8.0, 2, 0.5, 2)
Undot()

crop(8,0,704,480)
farm-fast.avs:
SetMemoryMax(64)

mpeg2source("d2v.d2v",cpu=0,iPP=false)
Telecide(order=1,guide=0,post=2,back=1,bthresh=40,nt=20,ovr="over.txt",show=false).Decimate(mode=0,show=false)

crop(8,0,704,480)
Note that the two scripts have to output the same size frame, and you still have to use IVTC in both.

I like where this is headed Omion, reminds me of Vidomi 4 years back or so with MPEG4 ASP and I really like it. Thanks for taking the time to do something that has really been needed to be tackled for quite a while.

And thank you for taking your time to test it out. It is still very much in its infancy, and I still have a lot of things I want to add to it. Stay tuned! (My next release should be soon...)

Maybe I missed it somewhere but have you implemented a que function to batch encode?
No queue yet. I've been thinking about that ever since I rebuilt x264farm from scratch, but I was never quite sure what the best way to implement it would be. It'll get in there sometime, I just don't know when.

foxyshadis

10th November 2006, 07:50

omion

10th November 2006, 08:03

Fast-avs is a great option for first pass! I should try to get that in MeGUI somehow (since it sometimes chooses a slooooooooow deinterlacer).
Well... it's not really used the way you probably think it is. The fast-avs is only used for the split-point-finding thread during the first pass. All the actual data that's sent to the agents come out of the regular AVS script.

But actually, a first-pass AVS sounds like a great idea! I'm glad you thought I thought of it ;)

Also note that 1.01 (which should be coming out tonight) will print out FPS stats. They'll be kind of buried in the debug data, but they're there.

omion

10th November 2006, 08:41

Well, that was faster than I thought. 1.01 is out. Get it in the first post.

The main change is that the combined FPS is now displayed for each pass.

The FPS is determined by the number of frames calculated, not the number of frames output. During both the first and second passes, there may be some frames that are thrown out, so this may not quite reflect the actual number of frames output.

You may have to look a bit to find the FPS line through all the debug garbage. One of the other things changes was weeding out some of the more verbose debug lines.

I also added the --rerc option. Basically, during the second pass, ratecontrol is done at the beginning of the pass, but it is also done when all the frames have been processed. x264farm will then re-encode the GOPs which are the farthest from what the ratecontrol says. The ratecontrol will then be processed every --rerc GOPs processed. Basically, set this number lower for more accurate bitrate and complexity targets, while giving up a bit of parellelism while the ratecontrol is being calculated.

The default for --rerc is 0, which means that all the GOPs have to be processed a second time before the ratecontrol is run a third time.

It's a bit complex without knowing the inner details of x264farm, but feel free to ask questions!

aicjofs

10th November 2006, 10:05

When you say "bring it to it's knees", do you mean that all the avs2yuv instances max out the CPU, or is it a memory issue?

More of a memory issue. It is gone when I use SetMemoryMax(64). Of course in hindsight I see you already recommended that. Still hits the CPU to 100% but system is still responsive. It was the page file usage that was bogging it, all the avs2yuv instances chewed up all 2GB but now not an issue with SetMem. Going to experiment with nice.exe next, I don't need the computers for anything while encoding that's what a laptop is for, but I might see what different CPU priorities do.

Whats your thoughts about 2 agents on a Hyperthreaded CPU? Can't imagine any gain as it's not another physical CPU.....If I have time I'll give it a try tomorrow to decide for myself.

In Megui, you can enter the codec configuration, and check the "Show commandline" box,

*sigh*, boy am I dumb. :D Amazing what is right under your nose all along. Appreciate the tip. That's exactly what I needed.

havix

10th November 2006, 18:04

Nice job!!! I can't believe I even got it to work to be honest. The only slow part to getting this working is copying and pasting different commands from megui and removing the incompatible parameters. I can't wait to play around with this some more this weekend and expand it from my current 2 machine setup to a total of 4 machines. Has anyone experimented what gives the best speed pertaining to the controller machine and if you should run only one agent so you don't choke the avisynth processing on a dual-core machine? Right now I'm running two agents on the controller but I'm not sure if this could be slowing down the delivery.

havix

10th November 2006, 18:15

Sorry guys I should have read through the rest of the instructions. I will play with task priorities in the config files.

aicjofs

11th November 2006, 01:57

omion

11th November 2006, 03:10

More testing. All of this is during the 1st pass, I'll wait for the 2nd pass before I add more.

When running one instance of agent.exe on each of the P4 machines, I am getting around 20-30% CPU usage. I thought it would be interesting to try --threads 2 in x264, but x264 says it's not compiled with ptthread and I see no speed difference. I added 2 instances of agent to the 2 P4's and cpu usage is close to 40-50% now. According to the new fps indicator I am getting about 20fps on each P4(10fps + 10fps per agent) and about 20fps each agent on the Opteron. So total fps is around 80fps per the indicator for the 1st pass.

So 2 questions now are:

The x264 ptthread issue. Is this something you need to add to your build of x264? or am I completely off base?

Yeah. I don't use it in my builds. In fact, I don't know if MSVS uses the same style threading as what x264 uses (it was really designed for GCC)

On the SetMemoryMax issue. I still have available RAM on the controller computer is there any benefit from setting this higher then 64?

Maybe. The difference probably won't be noticable, but you can set it as high as you want without running out of memory.

EDIT: One other thing. Splitter list too long splitter spinning? What does this mean?
That's perfectly OK. That means that there are more jobs in the queue than agents. In this case the splitter thread is paused in order to devote more resources to avs2yuv and the agents. The splitter thread starts up automatically when a job is checked out.

BTW: How is the CPU utilization laid out on the controller computer? Is it mostly avs2yuv, or does the agent (x264) take up the majority of the CPU? It looks a bit like your remote agents are starved for work.

aicjofs

11th November 2006, 03:38

2nd pass kicked out broken pipe error, and invalid arguement on the agents.

2006-11-10~17:24:16.85 Got a connection from 127.0.0.1:3451
22006-11-10~17:27:07.25 1st pass
2006-11-10~17:27:07.25 Doing the following:
x264 --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr --fps 24000/1001 --pass 1 --stats "G:\x264farm\temp\cpu1\165835 166209 05dd43.txt" -o NUL - 718x358
2006-11-10~17:27:18.57 Exited 0
2006-11-10~17:27:18.60 Finished OK; shutting down socket
2006-11-10~17:27:18.60 DOING!
2006-11-10~17:27:18.60 Got a connection from 127.0.0.1:3461
2006-11-10~17:27:25.59 Agent croaked with exception "End_of_file"; restarting
2006-11-10~17:27:25.59 DOING!
2006-11-10~17:27:36.78 Got a connection from 127.0.0.1:3463
2006-11-10~17:27:36.81 2nd pass
2006-11-10~17:27:37.42 Done recieving stats
2006-11-10~17:27:37.42 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats 02c6b7.txt" -o "G:\x264farm\temp\cpu1\output c4ab32.mkv" - 718x358
2006-11-10~17:27:37.48 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:37.48 DOING!
2006-11-10~17:27:37.59 Got a connection from 127.0.0.1:3469
2006-11-10~17:27:37.70 2nd pass
2006-11-10~17:27:38.07 Done recieving stats
2006-11-10~17:27:38.10 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats 5cf6b5.txt" -o "G:\x264farm\temp\cpu1\output a12837.mkv" - 718x358
2006-11-10~17:27:38.45 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:38.45 DOING!
2006-11-10~17:27:38.57 Got a connection from 127.0.0.1:3475
2006-11-10~17:27:38.89 2nd pass
2006-11-10~17:27:39.03 Done recieving stats
2006-11-10~17:27:39.03 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats c66430.txt" -o "G:\x264farm\temp\cpu1\output a8a9cd.mkv" - 718x358
2006-11-10~17:27:39.18 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:39.18 DOING!

2006-11-10~17:25:11.96 Got a connection from 192.168.1.101:3454
2006-11-10~17:25:11.98 1st pass
2006-11-10~17:25:11.98 Doing the following:
x264 --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr --fps 24000/1001 --pass 1 --stats "d:\x264farm\temp\cpu1\164335 164709 c1d557.txt" -o NUL - 718x358
2006-11-10~17:25:32.31 Exited 0
2006-11-10~17:25:32.39 Finished OK; shutting down socket
2006-11-10~17:25:32.39 DOING!
2006-11-10~17:25:32.39 Got a connection from 192.168.1.101:3455
2006-11-10~17:27:17.37 Agent croaked with exception "End_of_file"; restarting
2006-11-10~17:27:17.39 DOING!
2006-11-10~17:27:28.57 Got a connection from 192.168.1.101:3467
2006-11-10~17:27:29.31 2nd pass
2006-11-10~17:27:29.42 Done recieving stats
2006-11-10~17:27:29.42 Doing the following:
x264 --bitrate 5 --fps 24000/1001 --pass 3 --stats "d:\x264farm\temp\cpu1\stats 6281e0.txt" -o "d:\x264farm\temp\cpu1\output 58bba6.mkv" - 718x358
2006-11-10~17:27:29.54 Agent croaked with exception "Sys_error(\"Invalid argument\")"; restarting
2006-11-10~17:27:29.54 DOING!
2006-11-10~17:27:29.84 Got a connection from 192.168.1.101:3473
2006-11-10~17:27:29.96 2nd pass
2006-11-10~17:27:30.31 Done recieving stats
2006-11-10~17:27:30.31 Doing the following:
x264 --bitrate 841 --fps 24000/1001 --pass 3 --stats "d:\x264farm\temp\cpu1\stats 5226cf.txt" -o "d:\x264farm\temp\cpu1\output 969611.mkv" - 718x358
2006-11-10~17:27:30.35 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:30.35 DOING!

The AVS

SetMemoryMax(64)
SetMTMode(2)
DGDecode_mpeg2source("H:\CRZ0NNW7\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)
#blank deinterlace line
crop( 0, 60, -2, -62)

#resize
#denoise

Command

-b 1000 --avs "file.avs" --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr"

--second "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr" -o "test.mkv"

Since I am a noob at x264 command line it's possible my command is the issue. I just copied from meGUI like foxy recommended.

EDIT:

BTW: How is the CPU utilization laid out on the controller computer? Is it mostly avs2yuv, or does the agent (x264) take up the majority of the CPU? It looks a bit like your remote agents are starved for work.

Each x264 is using about 20% so 40% total, and the rest for controller, and avs2yuv.

omion

11th November 2006, 05:49

2nd pass kicked out broken pipe error, and invalid arguement on the agents.

[...]

Since I am a noob at x264 command line it's possible my command is the issue. I just copied from meGUI like foxy recommended.

Your command line looks OK (*) (except your -b argument needs to have "%" or "kbps" after the number, but that throws an error at the beginning, so I assume it's OK). What does the controller output say when the agents were putting up the broken pipe errors? If you closed the window or something, the log is saved in the "out-dump.txt" file in the directory where you ran the controller.

Actually, there's something else odd. When the agents say "Doing the following: ..." it doesn't look like they actually get the second pass line that you specified. Make sure that the controller got the right options by opening up the controller's "out-dump.txt". Near the top, there should be a line that gives the second-pass parameters that the controller is actually using.

It's possible that the "broken pipe" that the agents are complaining about are actually the pipe to x264, indicating that it died prematurely.
[ edit: This is probably the case. I seem to remember that x264 will croak if the first pass uses more B-frames than the second pass specifies, and since the second pass options didn't make it to the agent, it defaults to 0. ]

Each x264 is using about 20% so 40% total, and the rest for controller, and avs2yuv.
Hmm. It's odd that the remote computers aren't using any more than 20% per agent. And it's probably not a network issue, or adding more agents wouldn't help. Are you using version 1.01?

(*) The option vbv-maxrate may throw off the second pass. The reason is that I had to do ratecontrol in the controller, but I hardcoded a lot of the values to the defaults (qpmin, qpmax, ipratio, etc.) I'm pretty sure vbv-maxrate is one of those that would need to be supported in the controller, but isn't. The result is that x264 will make some GOPs with a lower bitrate than the controller expects, so they are marked to be re-encoded and the controller will inflate the bitrate of the entire GOP to accommodate that. (I can't really explain it too well)

NOTE: If you stop the controller, then restart it, it will pick up where it left off. You have to give it the same arguments, or it will think it's doing a different encode.

aicjofs

11th November 2006, 07:09

Looks like you nailed it. The 2nd pass options were blank in the dump log for the controller, I'm getting much better at understand the format of your dumps. I have started it 3 times in a row now and the 2 pass options are always there. I have run it the whole time with a .bat file so it's not like the options weren't there to begin with. If I look deeper through the logs I can see a x264 error about not the same b frames as the 1st pass, which of course makes since when no 2nd pass options were sent. Since I can't duplicate this it must be a non issue.

It's also complainging about no buffer set when the vbv-maxrate was set. I'm leaving out that option next go around. If you don't think we should be using it(vbv-max) maybe another update to the help file?

About to start another round. Next I'd like to see about getting more work for the clients.

Pretty exciting stuff actually. I'll post a screenshot in a few.

EDIT:

Controller

http://home.comcast.net/~aicjofs/264farm.jpg

One of the agents. Looks Identical to the other.

http://home.comcast.net/~aicjofs/264farm1.jpg

omion

11th November 2006, 08:59

Ooh. Screenshots. Yummy.

The shot of the network utilization is a bit odd. It looks kind of like it's maxing out at ~10%, which would be right where 100Mbps would be. Are your other computers connected with gigabit also?

I made a little network test setup for debugging, if you want to see if it's a network issue. The "sender" is here (http://omion.dyndns.org/other/from3.exe), and the "receiver" is here (http://omion.dyndns.org/other/to3.exe).

As with x264farm, start up the receiver first:
to3 45678 4096
(45678 is the port to use, 4096 is the string length)

Then start up the sender:
from3 192.168.1.100:45678 4096
(192.168.1.100:45678 is where you started up the receiver, 4096 is the string length)

I can get my jumbo-frame gigabit network up to ~25% with this test. See if you get any more than 10%.

It's also complainging about no buffer set when the vbv-maxrate was set. I'm leaving out that option next go around. If you don't think we should be using it(vbv-max) maybe another update to the help file?
OK. I changed the working copy of the help file to include all the ratecontrol things. I should be able to get it out soon, or at the latest when 1.02 comes out.

Inventive Software

11th November 2006, 19:18

Do you plan to reduce the transmission size across networks omion? If what he has is average 12.5% usage across 1 Gbit network, that's more than 100 Mbit can handle, so it'd hinder it. Is the transmission size (i.e the stuff sent through the cables) affected by the size of the content?

omion

11th November 2006, 20:42

It is definitely a hinderance, but you can get around 20 DVD-size frames per second out of a 100mbit connection. With my settings, anyway, that's about all I can pull out of my remote computers.

The data will have to be transmitted somehow. The only ways around this is to transmit it before rendering (which would need avisynth installed on the remote computer), or compress the raw data before sending.

I gave up on doing remote avisynth, and the compression would have to compress 10MB per second while using as little CPU as possible. I suppose it would be easy to try out some compressions, but I have a feeling it would take up too much CPU power to be worthwhile.

I am actually thinking about making avisynth servers, where the "controller" would also be divided amongst several computers, so the agents would be able to distribute the network load.

I haven't been able to figure out how to do the load-balancing, but it should be possible.

708145

11th November 2006, 21:14

I gave up on doing remote avisynth, and the compression would have to compress 10MB per second while using as little CPU as possible. I suppose it would be easy to try out some compressions, but I have a feeling it would take up too much CPU power to be worthwhile.

<advertisement>ELDER does the avisynth processing on each compute node. If someone wants to try it.</advertisement>

http://forum.doom9.org/showthread.php?p=718465#post718465

bis besser,
T0B1A5

aicjofs

11th November 2006, 21:51

All machines are gigabit. No jumbo frames the switch doesn't support it. I get 33% utilization on all 3 machines. Picture below is connecting one client at a time. Seems the bandwidth is available. If you want to give me the avs and your controller command line I can mimmick it with my own input file.

http://home.comcast.net/~aicjofs/264farm2.jpg

Yes 708145 I have used Elder but last time I looked you were freezing x264 development we need to continuing pushing forward in the x264 arena. :D

708145

11th November 2006, 23:11

Yes 708145 I have used Elder but last time I looked you were freezing x264 development we need to continuing pushing forward in the x264 arena. :D

I froze the _GPL_ branch of x264 2pass encoding. Beta5 includes an x264 crf mode :)
There is a development version of x264 2pass as well.

omion

18th November 2006, 08:59

1.02 is out (at last)

It took so long because I tried to do too many things and ended up basically rewriting the controller. It should run a bit less buggy now.

You will only need to run one agent program per computer now. It does all the threading in the program, so that you don't have to.

aicjofs's issue with the remote machines not using all the CPU should be fixed now. It should be quite a bit faster (at the expense of ~8MB more memory usage, and maybe slower x264 startup time).

Note that the format of the config.xml files has changed. See my examples in the package for what to do.

There should be fewer bizarre bugs, too. I updated my version of OCaml, and it seems to have fixed a lot of the "operation completed successfully" errors.

I'm not including the Mac or Linux builds with the main download, because I don't think enough people use them to warrant having everybody download them. I am still planning to have the Mac build available as a separate download soon.

aicjofs

20th November 2006, 07:27

Major improvements in this version.

Easier to setup(less config.xml's to edit) for one.

So I tried running the agents on the clients with "number 2". This loaded the P4's to about 40-50% on 1st and 2nd pass. Network utilization about 17% and peaked to 20% on controller. Fps (reported in controller) of about 77fps 1st pass 75fps 2nd pass.

I then tried "number 4" and "+1" (for 2nd pass) for clients. That ran 4 to 5 instances of x264 on each client. Then on both passes the clients were loaded from 85-100% average around 90%. This boosted fps to 89fps 1st pass, and 83fps 2nd pass, also increased network util on controller to about 25% and peaked as high as 30%.

Normally I would get around 50fps if it was just the controller rig using MeGUI or Nero with the simple parameters. Obviously this is for light encoding and I can't assume it scales because I haven't tried it. But it seems like it would if I added a more difficult profile that normally drags along at 10fps on a single machine. With x264farm I might just get realtime speed around 20-30fps.

I really like this build. Only real waste of CPU time seems to be when the ranger has passed out all frames and you are near the end of the 1st pass and the controller is waiting for all the clients to send back their frames, this causes the other clients to go idle waiting for all the frames to return so the 2nd pass can start. Only cosmetic gripe(and this is only while testing it matters) is the agents toggles the "frames -fps" display to fast. When I had 4 "CPU's
" on the agent it was hard to read where each CPU was on what frame. Not a big deal but while testing it was happening to fast to read the bottom line. Maybe slow the switching delay or add all CPU's to the bottom line? No need to really see once x264farm matures, but it would be nice for testing purposes.

CruNcher

20th November 2006, 19:56

@ omion

what does this error mean ?

2006-11-20~19:30:40.00 x264farm version 1.02-151
2006-11-20~19:30:40.00 OPTIONS:
2006-11-20~19:30:40.00 -b 8000%
2006-11-20~19:30:40.00 --first "--no-fast-pskip --no-dct-decimate --bframes 3 --weightb --direct auto --no-cabac --subme 1 --analyse none --me dia --no-psnr"
2006-11-20~19:30:40.00 --second "--no-fast-pskip --no-dct-decimate --bframes 3 --weightb --direct auto --no-cabac --subme 2 --analyse p8x8,b8x8,i4x4 --me dia --no-psnr"
2006-11-20~19:30:40.00 --avs "sylver.avs"
2006-11-20~19:30:40.00 --fastavs ""
2006-11-20~19:30:40.00 --zones 2006-11-20~19:30:40.00
2006-11-20~19:30:40.00 --seek 0
2006-11-20~19:30:40.00 --frames 0
2006-11-20~19:30:40.00 --batch 5000
2006-11-20~19:30:40.00 --split 250
2006-11-20~19:30:40.00 --thresh 20.000000
2006-11-20~19:30:40.00 --preseek 0
2006-11-20~19:30:40.00 --force false
2006-11-20~19:30:40.00 --restart false
2006-11-20~19:30:40.00 --config "config.xml"
2006-11-20~19:30:40.01 --3thresh 0.800000
2006-11-20~19:30:40.01 --3gops 1073741823
2006-11-20~19:30:40.01 --3ratio 0.050000
2006-11-20~19:30:40.01 -o "out.mkv"
2006-11-20~19:30:40.01 Using config file ".\config.xml"
2006-11-20~19:30:40.01 CONFIG:
2006-11-20~19:30:40.01 temp_dir: "H:\\x264farm\\temp"
2006-11-20~19:30:40.01 agent_list 1:
2006-11-20~19:30:40.01 "Local CPU 1" = 127.0.0.1 50700-50703
2006-11-20~19:30:40.01 "Laptop 1" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.01 agent_list 2:
2006-11-20~19:30:40.01 "Local CPU 1" = 127.0.0.1 50700-50703
2006-11-20~19:30:40.01 "Laptop 1" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.01 "Laptop 2" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.48 GOT NORMAL INFO "sylver.avs: 1920x1080, 25 fps, 10816 frames\013"
2006-11-20~19:30:40.48 ERROR: AVS file 'sylver.avs' does not seem to be valid

omion

21st November 2006, 05:46

@ omion

what does this error mean ?

2006-11-20~19:30:40.48 GOT NORMAL INFO "sylver.avs: 1920x1080, 25 fps, 10816 frames\013"
2006-11-20~19:30:40.48 ERROR: AVS file 'sylver.avs' does not seem to be valid

That normally means that the AVS file has an error and the video information could not be read. Except that (as you can see) the information could be read.

The problem is that I assumed the FPS is always a rational number (i.e. x/y), but over in PAL land it looks like it's just given as an integer. I'll make a quick fix and put up a new version as soon as I can.

omion

21st November 2006, 07:45

1.03 is out. CruNcher's problem should be fixed. That's the only change since 1.02.

ChrisBensch

21st November 2006, 10:19

I'm using one of my machines as a controller and the other as an agent with v1.03. Both are Gigabit and are using about 7% on average. I have 4 total ports for the agent machine but only one is being used and I only see one instance of x264.exe (I'm still encoding the first pass). I don't see any errors, but I'm sure it could be faster based on other people's results. I've even tried taking out my processing settings. I guess I have one question...is anyone else doing HDTV 1920x1080i ---> HDTV 1280x720p encoding? Or are these results posted from DVD backups?

I used the to3 and from3 network tests and I peaked my Gigabit setup at 17% using the settings from page 3. I guess I'll try with DVD stuff later...

omion

21st November 2006, 16:59

I'm using one of my machines as a controller and the other as an agent with v1.03. Both are Gigabit and are using about 7% on average.7% CPU, or 7% network usage? If it's network usage, how's the CPU usage?

I have 4 total ports for the agent machine but only one is being used and I only see one instance of x264.exe (I'm still encoding the first pass). I don't see any errors, but I'm sure it could be faster based on other people's results. I've even tried taking out my processing settings. I guess I have one question...is anyone else doing HDTV 1920x1080i ---> HDTV 1280x720p encoding? Or are these results posted from DVD backups?I must admit, I haven't done any HDTV tests with it yet. I'll try it out once I un-break what is going to become 1.04.

ChrisBensch

21st November 2006, 17:35

omion

21st November 2006, 17:43

Well, I think I was jumping the gun a bit, it seems that during the first pass I was only getting one instance of x264, but during the second I was getting two. The 7% was bandwidth, cpu was around 45%. Now that on the second pass, the cpu is at 100% and bandwidth is around 1%...my agent/encoder is the slower machine...so we'll see what happens when I add the rest of my boxes. Keep up the good work!
Great!

It looks like I still need to work on the network code to keep x264 fed, though. The agent should use 100% of each core, even for the first pass. As aicjofs noted, one way around this is to increase the number of agents running with the <number> line in config.xml, but that's still just a workaround. I'll see what I can tweak in the next release.

BTW: The reason you get one x264 during the first pass and two during the second pass is due to the pad="1" attribute in the <number> line. Basically, during the second pass, the jobs are small enough that a a relatively large percentage of the time is x264 setting up. Therefore it's a good idea to have something running in the background for doing something useful during that time.

ChrisBensch

21st November 2006, 20:59

So then only a single instance of x264 is normal during 1st pass? I thought that on a dual cpu system I'd get one per core? I have 4 ports setup for use (50700-50703)

havix

21st November 2006, 21:24

Just a suggestion, maybe you could integrate something like "psexec" to launch agents on remote machines when the controller launches. This would make things easier to jump right into encoding.

aicjofs

22nd November 2006, 05:34

So then only a single instance of x264 is normal during 1st pass? I thought that on a dual cpu system I'd get one per core? I have 4 ports setup for use (50700-50703)

You are changing the "number" tags in the controller config.xml right? Not only do you add ports but you need to change that number to get more x264 instances.

<number pad="1">change this number</number>

omion

22nd November 2006, 06:22

You are changing the "number" tags in the controller config.xml right? Not only do you add ports but you need to change that number to get more x264 instances.

<number pad="1">change this number</number>

Right... mostly. You actually don't need to increase the number of ports for more encodes.

Before 1.02, each agent listened on its own port and were completely separate. When I released 1.02, two things changed:
* I implemented a more "server" style for the agents. All of the jobs go through one port, and the agent program splits off threads to handle each one.
* I let the agent select one port from a range of ports, in order to squash an uncommon but annoying problem when a port is not available right after an agent is stopped.

So basically, the number of available ports was expanded, but the number of used ports was reduced to 1.

It is also worth noting that the agents also have a <number> line in their XML file. This <number> does nothing. Only <number> on the controller side will have any effect.
I had originally had the agents reject connections if there were already <number> connections, but figuring out what was connected proved difficult. I removed the effect, but I kept the config line just in case I re-enable it.

Just a suggestion, maybe you could integrate something like "psexec" to launch agents on remote machines when the controller launches. This would make things easier to jump right into encoding.That does sound like a good idea. I don't know what the best way to only start up the agent if it has not already started, but I think it would be possible.
I've got a lot of ideas backed up, so it will probably take a while to implement.

CruNcher

22nd November 2006, 09:36

Hi omion it's working now but something doesn't feels quite right with the cpu usage on the client see screenshot bellow

http://cruncher.mufflastig.com/x264/x264farm.png

Network utilization is @ 50%
Cpu usage on the client is fluctuating very heavy

and i have a improvement wish :)
could you please make the controller delete the created files on the fly after he concentate them this way you don't need 2x the diskspace :)

omion

22nd November 2006, 17:43

Network utilization is @ 50%
Cpu usage on the client is fluctuating very heavy

It's possible that the network is saturated at 50%. My gigabit network never gets more than about 35-40%. The next version will have a user-definable agent buffer, which may make things better (or at least different :D ) I still have some work to do on it, but it shouldn't take too long.

and i have a improvement wish :)
could you please make the controller delete the created files on the fly after he concentate them this way you don't need 2x the diskspace :)
It would be easy to add a switch that would do that. I'm hesitant to do it by default because if there was an error opening one of the files, the controller would have to recreate the deleted files when it was restarted. That hasn't happened to me yet, but I don't like the default behavior leaving the files in a state where it could happen. A switch would be good though. I'll add it into 1.04.

omion

25th November 2006, 01:14

OK. 1.04 is out, along with a very big addition that I knew needed to happen some time.

A few minor changes first:
* The "-b" bitrate option has been renamed "-B" to better match x264's parameters
* I also added the --savedisk command line option, to implement CruNcher's on-the-fly temp file deletion. It will not use up more disk space than the final file, but if something goes wrong when merging, the deleted parts have to be re-rendered

1.04 introduces "agent-based" encoding, when the agent has access to the AVS file and calls x264 directly, rather than "controller-based" where the controller does everything.

In order to set it up, you need to use the <base></base> line in the agent's config.xml. This is the base directory to search for the AVS file, in case the agent's computer has a different directory structure.

Here's an example:
Agent's config.xml: <base>F:\temp\x264farm</base>
Controller's AVS file: D:\movies\working\somewhere\file.avs

The following files will be searched for on the agent's computer:
F:\temp\x264farm\D:\movies\working\somewhere\file.avs
F:\temp\x264farm\movies\working\somewhere\file.avs
F:\temp\x264farm\working\somewhere\file.avs
F:\temp\x264farm\somewhere\file.avs
F:\temp\x264farm\file.avs

The first file that is found whose contents match the controller's AVS file will be used as input to x264, rather than getting the frame data through the controller.

Also, if you use <base></base> or <base/> in the config file, that indicates that the only place to look for the file is exactly where the controller found it. This is good if both computers have the same directory structure, or if the agent is running on the controller's computer.

If a matching file is not found anywhere, the agent will default to receiving data from the controller, just like before.

To turn off the agent-based encoding, either remove or comment out the <base> line in config.xml.

Get it at the beginning of the thread. Have fun!

akupenguin

25th November 2006, 03:27

The first file that is found whose contents match the controller's AVS file will be used as input to x264
If you check that the contents match, why not just send the avs to the agent and check whether it can be opened?
If you want to allow that the agent has the source files (not just the avs) in a different place, then you don't want to enforce that the two avs are the same. Or does your check for "match" ignore any string constants within the avs or something like that?

omion

25th November 2006, 06:44

You're right. I suppose I didn't mention one important thing, which I knew but didn't bother to say ;) :
All the AVS source files should be relative to the AVS, or, as akupenguin says, they won't work in a different directory. This includes the .D2V file: in DGindex, you should DEselect "Options > Use Full Paths" when making the D2V.

And I check for exact equality with MD5, so there's not a whole lot of room for tweaking the file.

ChrisBensch

27th November 2006, 06:54

thetrueavatar

29th November 2006, 10:45

Hello,
First thank you for this good intiative because distributed encoding is really cool.
However my main problem is that I would like to distribute my encoding on friends computer instead of my owns computers.
So the high network requirement seems too high.

So I have a suggestion to reduce a bit the network bottleneck. Is it possible to do a quick lossless compression on the controller (with hyufhyuv for example) that could reduce DVD size by 2 instead of sending raw video on the network? I have heard that Huffyuv was really fast and I suppose that time lost for lossless encoding/decodingto will be compensate by the gain over the network connection. This seems not too complicate to implement if we use agent base encoding and could provide a real performance improvement.
What do you think about it? Does it make sense?

omion

29th November 2006, 19:57

I'm using the new 1.04 with "agent based encoding" and I have to say I like it. My network util is at about .25-.30% on a gigabit network. My normal MeGui fps is about 8-9fps on this particular movie and with two machines I'm getting about 15fps. I expect that when I add the other two, I could get around 27-29fps total...faster than realtime for my HDTV encodes. I can't wait to see this sort of thing in MeGui!
Great! Thanks for the feedback!

Hello,
First thank you for this good intiative because distributed encoding is really cool.
However my main problem is that I would like to distribute my encoding on friends computer instead of my owns computers.
So the high network requirement seems too high.

So I have a suggestion to reduce a bit the network bottleneck. Is it possible to do a quick lossless compression on the controller (with hyufhyuv for example) that could reduce DVD size by 2 instead of sending raw video on the network? I have heard that Huffyuv was really fast and I suppose that time lost for lossless encoding/decodingto will be compensate by the gain over the network connection. This seems not too complicate to implement if we use agent base encoding and could provide a real performance improvement.
What do you think about it? Does it make sense?
It makes a lot of sense, actually. Although I'd probably make my own compressor, since it's about as much work to do that as it is to link to C code from OCaml. I've been toying with various compression settings, but it may be a while before I get it working.

The issue may be that the network connection is still too slow. My cheapo DSL connection can upload ~30KB/s. Uncompressed DVD-res video is ~500KB/frame. This means, on my connection, I could send 1 frame every 16 seconds (that's 0.06fps!) Even with 2:1 compression, that would still be extremely slow.

With computers that are so different in apparent speed (yours vs. whatever is on the end of the 0.06fps connection), it will probably slow down the encoding when compared to just using your computer.

Probably the optimal solution would be to use agent-based encoding and somehow share the DVD files with your friend (like with a shared folder in Windows) In this case, you would be transmitting the original compressed frames, so he might get ~1fps (a big improvement!) I don't know how well shared folders hold up over the internet, though.

Inventive Software

29th November 2006, 20:03

Forgive me if I've missed something, but if all the computers used for encoding have the source files that the agent has, would that not reduce the load over the network, i.e not transmitting the uncompressed AviSynth output? Just my $0.02, as I think anything to reduce the network dependancy to something justifiable like D1 over 100 Mbit networks (not possible at the moment as the data rate is something like 124 Mbits/sec) would be a good thing. :)

omion

30th November 2006, 00:49

Forgive me if I've missed something, but if all the computers used for encoding have the source files that the agent has, would that not reduce the load over the network, i.e not transmitting the uncompressed AviSynth output?

Yup. That's what agent-based encoding does. The controller tells each of the agents to do some range of frames, then the agents send back the compressed MKV files.

The original method of encoding, controller-based, acts like you described, but I added the agent-based encoding in 1.04.

Just my $0.02, as I think anything to reduce the network dependancy to something justifiable like D1 over 100 Mbit networks (not possible at the moment as the data rate is something like 124 Mbits/sec) would be a good thing. :)
I don't think I understand what you mean. You can't send D1 over 100mbps networks real-time, but if x264 can only encode 1/2 real-time, then the encoder is the bottleneck anyway.

I'm also writing a program which will analyze different compression methods to decrease network usage in case the source files are not accessible to the agent, but I don't know when that will be added.

Inventive Software

30th November 2006, 01:39

If you have enough computers for the encoding, then it'll be possible to encode more than realtime (we're talking people with access to university networks and suchlike), which is why reducing the raw transfer data over the network becomes critical, otherwise the network becomes the bottleneck.

Besides, it's fairly well known that 100 Mbit is the absolute maximum transfer in peak conditions with everything perfect (based on people stating dialup @ 56K never reaches 56K), and it's more likely to be around 90 Mbit tops (average ;)). Those lucky people with relatively new motherboards with Gigabit ethernet don't have to worry so much. ;)

TriHFH

30th November 2006, 04:50

Hey, thanks a lot for working on this project, I think it has a ton of potential. I tried using x264 farm but I am getting this error:

ERROR: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error

but i opened it in virtualdubmod and it opened fine w/o any errors. What can I do to solve this/ what is causing these problems?

omion

30th November 2006, 04:59

Hmm... for whatever reason avs2yuv didn't like the AVS file. Post the output of the command:
avs2yuv -frames 1 -o NUL pb213.avs
Oh, and :script: ;)

TriHFH

30th November 2006, 05:06

Here's the script:

DGDecode_mpeg2source("F:\pb213\pb213.d2v",info=3)
ColorMatrix(hints=true)
TDecimate(cycleR=3)
TTempSmooth(lmdiff=1,cmdiff=2,strength=1,scthresh=8)
RemoveGrain(5)
#crop
#resize
#denoise

And the output of that command is:

c:\x264farm\controller\pb213.avs: 1280x720, 24000/1001 fps, 61376 frames

omion

30th November 2006, 05:22

OK, that's really weird. The command I gave you is almost exactly what my program tests for the data. The only thing I can think of is that the controller can't find avs2yuv. Where did you put avs2yuv?

Are there any other warnings / errors / suspicious lines before the error you posted?

(my program actually tests the command avs2yuv -frames 1 -raw -o NUL "avsfile" 2>&1
Try that, too, just in case)

TriHFH

30th November 2006, 05:50

Nope, I did not notice any weird lines/errors before my error. Here is a screensot of it:

That new command outputs this:

C:\x264farm\controller\pb213.avs: 1280x720, 24000/1001 fps, 61376 frames

http://yogurtrat.com/files/1/error_thumb.png (http://yogurtrat.com/files/1/error.PNG)

I wasnt sure where to put avs2yuv, so i put it in the controller folder and the main folder :P

omion

30th November 2006, 06:17

I wasnt sure where to put avs2yuv, so i put it in the controller folder and the main folder :P
Well, the "right" folder is anywhere on the path. Putting it in the controller's directory is great as long as you call the controller from that directory.

Unfortunately, everything looks fine. There are a few issues with your command line, but nothing that would cause it to die there (*)

I'll try to reproduce the error on my computer and see what could be happening. In the mean time, try out a braindead-simple AVS script and see if x264farm will take it. Something like this:colorbars()

(*) Since I don't know what's wrong, I'll just nitpick everything else:p :

The -B option needs to be "-B 4353kbps", to distinguish it from 4353% of the first pass size. It looks like x264fram will default to percent, which is probably not what you want. I should make a warning for that...
leave out --stats from both the first and second passes. x264farm keeps track of the stats file internally.
The --zones option should be outside the -1 and -2 options. That is, it should be an option of x264farm itself, not part of the string to pass to x264. Otherwise the x264farm ratecontroller will not be too happy (it gets easily confused;) )
"--no psnr" should be "--no-psnr"

thetrueavatar

30th November 2006, 08:26

why don't you transmit compressed source(mpeg2 for example) instead of raw?

foxyshadis

30th November 2006, 08:49

The -B option needs to be "-B 4353kbps", to distinguish it from 4353% of the first pass size. It looks like x264fram will default to percent, which is probably not what you want. I should make a warning for that...

Perhaps % below 100, and bitrate above 100, if no modifier is given?

TriHFH

1st December 2006, 02:54

omion

1st December 2006, 03:05

why don't you transmit compressed source(mpeg2 for example) instead of raw?
It would be difficult to do through x264farm, since I would need to intercept the AVIsynth calls to the source files. You can, however, do it the roundabout way:
Share the folder with the source files on the main computer
Then use the "Map network drive" feature on each of the agent's computers to use the network folder as a separate hard drive (it does not need to have write access)
Then use "agent-based" encoding and use the new drive letter as the base directory

Perhaps % below 100, and bitrate above 100, if no modifier is given?
Yeah, that's probably what I'll do.

omion

1st December 2006, 06:43

I tried that avs and it seems to have worked (it is processing something) when i load it directly from avs2yuv, but when i load the avs into the controller, it gives me that same error, even with that simple avs.

And thanks so much for your help so far... I'm really excited to get this working :D

One more possibility, then I'm stumped. Are you using Windows 2000/XP? If not, many of the functions that I use (possibly including the avs2yuv call) won't work. If you aren't using 2000/XP, neither the controller nor the agent will work at all so you should focus on upgrading instead of video encoding ;)

If you have Windows 2000 or XP, then try out this controller (http://omion.dyndns.org/x264farm/x264farm-1.04+_controller-only.rar). It will die before anything gets encoded even without your error, but post all the lines that start with "DEBUG:" That should narrow down the problem.

Kripsy

1st December 2006, 08:35

Is x264farm capable of encoding a h264 source file, or is it only mpeg stuff? I ask because I am encoding some rather large 1080p caps to something a bit more cpu friendly with megui currently, but it takes me upward two days to do it!

foxyshadis

1st December 2006, 11:41

Anything you can do through avisynth, x264farm can work with. It only cares about the output, not the input. You'd probably want the most recent DirectShowSource (see avisynth development forum) if you go that route, lots of bugfixes.

TriHFH

2nd December 2006, 20:36

Here is the results of the debug lines.

2006-12-01~06:33:05.03 DEBUG: Running command "avs2yuv -frames 1 -raw -o NUL "pb213.avs" 2>&1"
2006-12-01~06:33:05.04 DEBUG: avs2yuv.exe in current directory (C:\x264farm\controller)? true
2006-12-01~06:33:05.04 DEBUG: opening avs2yuv
2006-12-01~06:33:07.42 DEBUG: avs2yuv responded: "pb213.avs: 1280x720, 24000/1001 fps, 61376 frames\013\n"
2006-12-01~06:33:07.43 DEBUG: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error
2006-12-01~06:33:09.03 ERROR: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error

Now I tried the controller on another pc, and it got a little farther this time, but it still stopped. I tried this with a different avs file along with the simple one. The simple one gave the same error and stopped.

2006-12-01~23:46:05.00 x264farm version 1.04-153
2006-12-01~23:46:05.00 Using config file ".\config.xml"
2006-12-01~23:46:05.00 CONFIG:
2006-12-01~23:46:05.00 temp_dir: "C:\\x264farm\\temp\\controller"
2006-12-01~23:46:05.00 agent_list 1:
2006-12-01~23:46:05.00 "Local CPU 1" = 127.0.0.1 50700-50703
2006-12-01~23:46:05.00 "Laptop 1" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Katie 1" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.00 agent_list 2:
2006-12-01~23:46:05.00 "Local CPU 1" = 127.0.0.1 50700-50703
2006-12-01~23:46:05.00 "Laptop 1" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Laptop 2" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Katie 1" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.00 "Katie 2" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.01 DEBUG: Running command "avs2yuv -frames 1 -raw -o NUL "seoul.avs" 2>&1"
2006-12-01~23:46:05.01 DEBUG: avs2yuv.exe in current directory (C:\x264farm\controller)? true
2006-12-01~23:46:05.01 DEBUG: opening avs2yuv
2006-12-01~23:46:10.71 DEBUG: avs2yuv responded: "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013\n"
2006-12-01~23:46:16.40 GOT NORMAL INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:22.15 GOT FIRST INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:28.20 GOT FAST INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:28.21 INFO:
2006-12-01~23:46:28.23 i_res_x = 1280
2006-12-01~23:46:28.29 i_res_y = 720
2006-12-01~23:46:28.57 i_fps_n = 30000
2006-12-01~23:46:28.95 i_fps_d = 1001
2006-12-01~23:46:29.14 i_fps_f = 29.970030
2006-12-01~23:46:29.14 i_num_frames = 208097
2006-12-01~23:46:29.14 i_bytes_y = 921600
2006-12-01~23:46:29.14 i_bytes_uv = 230400
2006-12-01~23:46:29.14 i_bytes_per_frame = 1382400
2006-12-01~23:46:29.14 multiplier for kbps to fbits = 0.029970
2006-12-01~23:46:29.14 fast_res_x = 1280
2006-12-01~23:46:29.14 fast_res_y = 720
2006-12-01~23:46:29.14 IF THIS WAS RELEASED, YELL AT OMION!

Here is the seoul.avs

DGDecode_mpeg2source("G:\captured\Metallica\seoul.d2v",info=3)
ColorMatrix(hints=true)
edeintted = last.AssumeBFF().SeparateFields().SelectEven().EEDI2(field=-1)
TDeint(order=0,edeint=edeintted)
TTempSmooth(lmdiff=1,cmdiff=2,strength=1,scthresh=8)
RemoveGrain(5)
crop( 0, 0, 0, -8)

LanczosResize(1280,720) # Lanczos (Sharp)
#denoise

omion

2nd December 2006, 21:51

Here is the results of the debug lines.
That's odd. It looks like everything works fine, but avs2yuv exits with an error anyway. I'll see if ignoring the output will work...

Now I tried the controller on another pc, and it got a little farther this time, but it still stopped. I tried this with a different avs file along with the simple one. The simple one gave the same error and stopped.

Use the release version (at the beginning of the thread) of x264farm. The debug one I posted was based on my current programming version, which didn't work properly. I just made it exit before it got to the part that broke.

TriHFH

3rd December 2006, 01:53

Thank you :)
Thank you for spending all this time working on it and troubleshooting my problem :)

omion

3rd December 2006, 05:35

Try out this controller (http://omion.dyndns.org/x264farm/x264farm-1.04+2_controller-only.rar); it should actually encode. If it works, I'll put it into the next release. Or maybe I'll just make it the next release.

BTW: what are your system specs, anyway?

OverlordQ

3rd December 2006, 11:31

There a guide for figuring out command line switches for idiots like me :D

omion

3rd December 2006, 21:43

There a guide for figuring out command line switches for idiots like me :D

Well, there's the HTML file (http://omion.dyndns.org/x264farm/x264farm.html), which is included with the releases. However, it's probably a bit too much, so here's a crash-course in command syntax:

The agents don't need any switches, and get all of their options through the config files.
The basic controller command-line is:
conrtoller --first "<first_pass_settings>" --second "<second_pass_settings>" -B <BITRATE>kbps -i "input.avs" -o "output.mkv"

There is, however, a whole slew of x264 options that you shouldn't use in the --first and --second strings. Don't include the stats file, input or output files, or anything which has an x264farm equivalent. If you have any questions about whether or not an option can be included in the --first or --second strings, just post here.

(I just noticed that I don't exactly make clear in the HTML file which options to leave out. I'll have to change that...)

AVmaniac

5th December 2006, 14:09

You really did a great job with creating x264farm.
I am actually loving it.
It is giving my x264 encoding jobs the boost i was waiting for.

But I've one question ... is it possible to use it with Sharktooth's
builds of x264? Because i'm missing the aq-strength flag in your builds.

:thanks: again

omion

5th December 2006, 18:42

You really did a great job with creating x264farm.
I am actually loving it.
It is giving my x264 encoding jobs the boost i was waiting for.
Thanks!

But I've one question ... is it possible to use it with Sharktooth's
builds of x264? Because i'm missing the aq-strength flag in your builds.
Sharktooth's build will not work with controller-based encoding (the default) However, it should work just fine if you give all the computers access to the source files and use agent-based encoding.

The problem with everybody else's builds is that they pipe input in text-mode, whereas I'm sending binary data to x264. Using text-mode basically makes piping data useless for x264 because it will stop encoding if it encounters a certain byte (0x1A).

I would also like Sharktooth's builds to work, because then I wouldn't have to spend time building it :D . I pointed out the problem in this thread (http://forum.doom9.org/showthread.php?t=116700), but as far as I know it hasn't been fixed. If you test out the code I have in the first post on that page and it encodes more than 8 frames, then the build should work with x264farm.

kenji123

6th December 2006, 16:38

love the work you've done omion. just tried it out this morning, and it worked flawlessly. hopefully the issue with Sharktooth's build will be resolved soon.

kenji123

9th December 2006, 15:29

That's odd. It looks like everything works fine, but avs2yuv exits with an error anyway. I'll see if ignoring the output will work...

ERROR: avs2yuv exited with error 1; AVS file "Track1.avs" probably has an error

During the debugging of the x264farm GUI i'm creating, just like TriHFH, i also got this error. it seems avs2yuv must be in the same directory as the AVS file being used.

In the next release could you add a parameter so that the avs2yuv location could be specified, omion.

omion

9th December 2006, 17:26

It will find avs2yuv anywhere on the path (I, personally, don't have it in the AVS directory). But I'll add something to check in the controller's directory, too.

The next version may take a while. I'm adding compression to the controller-based encoding, and it's going to need a lot of debugging.

SpAwN_gUy

11th December 2006, 16:23

So, i've Downloaded everything.. and installed
now i have about 10 PCs Under my control.. COOL.. Yeppie...
After making some preperations.. the Installation of Agent takes about... 30Seconds %) ... and this is WITH agent-based encoding..

at first i've tried to encode 5mins clip.. just for test..
1st went GOOD .. and really Fast..
the second file became Broken :(( (second encode of the same file)
and the third went great again...

and the i've started 4,5Hours encode.. and went Home(that was Friday)..

...Skip...
2006-12-08~23:20:28.87 pc17 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-08~23:20:28.87 pc17 2 close_connection
2006-12-08~23:20:28.89 pc17 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-08~23:20:28.89 pc17 2 close_connection

...SKIP...

2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection
2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection
2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection

....SKiP... and then Again
2006-12-11~08:48:23.14 pc6 3 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-11~08:48:23.14 pc6 1 close_connection
2006-12-11~08:48:23.14 pc6 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-11~08:48:23.14 pc6 3 close_connection

and then My CONTROLLER went OUT of Memory(i think).. and Explorer made some exceptions..
and today i've made a reboot.

BUT.. Agents are STILL working.. and STILL encoding something..

ok.. few Questions:
Can i RESUME Encoding? (i still have batch file.. and still have TEMP files on controller) and i'm doing agent-based encoding
i'm using controller from your last post(where you posted it ;) )

How can i start AGENT in "TRAY-Mode"... or just as Service(it would be nice.. really) ... well that is not a question..

and i'm currently thinking of writing 2 DB clients (i'm a Delphi programmer) .. so it would automate Agent Installation and Controller-Starter with GUI ... (it's not that hard.. to me.. i just need to finish my Job..)

omion

12th December 2006, 01:13

So, i've Downloaded everything.. and installed
now i have about 10 PCs Under my control.. COOL.. Yeppie...
After making some preperations.. the Installation of Agent takes about... 30Seconds %) ... and this is WITH agent-based encoding..

at first i've tried to encode 5mins clip.. just for test..
1st went GOOD .. and really Fast..
the second file became Broken :(( (second encode of the same file)

How did the file break?

and the third went great again...

and the i've started 4,5Hours encode.. and went Home(that was Friday)..

<CODE/>
and then My CONTROLLER went OUT of Memory(i think).. and Explorer made some exceptions..
and today i've made a reboot.

Hmm. Are all of the computers running agent-based encoding? They all have access to the source files, x264, and avisynth? It's odd that the controller would run out of memory with agent-based encoding.

(it's also a bit odd that "socket" was throwing the errors. Normally "send" or "recv" throw them. Check your controller config.xml file for any incorrect IP addresses. It could be that one of them is specified incorrectly, but the short encode didn't distribute anything to them so it worked fine...)

BUT.. Agents are STILL working.. and STILL encoding something..

One of the issues with agent-based encoding is that the controller can't stop the agent from encoding if it dies. The agents will stop after the current encode.

ok.. few Questions:
Can i RESUME Encoding? (i still have batch file.. and still have TEMP files on controller) and i'm doing agent-based encoding
i'm using controller from your last post(where you posted it ;) )

Yup. As long as you use the exact same settings, it will start up where it left off.

If you don't remember the settings you used, look for some files called "settings.txt" in the temp directory. There is one file in the AVS's temp directory, and one file in each of the pass sub-directories. The ones in the pass directories give the settings of that pass. (I hope that makes sense)

How can i start AGENT in "TRAY-Mode"... or just as Service(it would be nice.. really) ... well that is not a question..

Firedaemon (http://www.firedaemon.com/)

and i'm currently thinking of writing 2 DB clients (i'm a Delphi programmer) .. so it would automate Agent Installation and Controller-Starter with GUI ... (it's not that hard.. to me.. i just need to finish my Job..)
Sounds interesting!

One more question:
Are any of your computers running Windows 95/98/Me? They probably won't work with x264farm at all.

SpAwN_gUy

12th December 2006, 09:37

How did the file break?
well. i'm not sure.. i haven't saved it :( but the MPC ended with "can not render file"
Hmm. Are all of the computers running agent-based encoding? They all have access to the source files, x264, and avisynth? It's odd that the controller would run out of memory with agent-based encoding.
Well.. it did it again this Night.. Controller was "Alive" till Half an Hour Ago.. and the same thing started at:
>> 2006-12-12~00:32:47.54 pc661 3 process_job failed with "End_of_file"; putting job back
and it was working since:
2006-12-11~18:16:25.00 x264farm version 1.04-153

today i was using a "Final build" instead of "maybe the next release"

and yeah the IPs - are good(made it from PING). xml-file has some "not started agens IPs" .. so when encode starts not all the agents are working.. and then controller hops all-new agents in line :) ..

Yup. As long as you use the exact same settings, it will start up where it left off.
yeah i've remembered it.. i read it somewhere here.. and it did actuallty continued...
Firedaemon (http://www.firedaemon.com/)
thanks :) .. i was thinking about some "AppToService" programm
One more question:
Are any of your computers running Windows 95/98/Me? They probably won't work with x264farm at all.
Of Cource :) .. a first i was thinking to run an Agent on Vista_Final.. but then i couldn't figure out the IP, because PING didn't answered anything.

omion

13th December 2006, 02:48

Released 1.05.

Firstly, note that you need not overwrite your config.xml files for this update, although I did add a few features which need an updated config.xml file in order to use them.

FPS chart
I added a little chart that prints out occasionally during the 2nd pass, indicating the speed of each agent. I could have done the same for the first pass, but it would have been wildly inaccurate. It looks like this:
Agent | Now | GOPs | Frames | FPS |
Local 1 | 340 | 69 | 7749 | 2.69 |
Local 2 | 342 | 67 | 7850 | 2.82 |
Linux 1 | 341 | 22 | 2184 | 2.48 |
Linux 2 | 339 | 17 | 2236 | 2.52 |
Mac 1 | 329 | 52 | 6456 | 2.55 |
Mac 2 | x | 54 | 6334 | 2.39 |
Mac 3 | 330 | 56 | 6453 | 2.57 |
16.63 FPS total on 39262 / 40492 frames (96%)
ETA: 1:13"Agent" is the name of the agent
"Now" is the current job number (from 0 to the number of scenes) "x" means the agent is not currently working. Since the chart is printed out when an agent checks out a job, at least one agent will always have an "x".
"GOPs" is how many scenes the agent has completed
"Frames" is how many frames the agent has completed
"FPS" is the speed of the agent (based on the times when the agent was actually working, not the total time the controller was working)
The final FPS line is the actual frames per second, and will probably be less than the sum of the previous FPS columns.

Compression
The main change is a beta-quality compression setting, primarily for testing purposes. In order to turn it on, add the following to you agent's config.xml:
<compression>
<type>1</type>
<type>0</type>
</compression>
If you are using the supplied config.xml file, make sure type 1 is above type 0, in order to use compression.

If you do use compression, I have a few requests:

After every encode, check the agent's directory for files with names like "MD5 error 01 (ABCDEF).txt". If there are any there, tell me!
On the controller's side, there should be a bunch of lines like this:01: 34.6% compression (11272631295 / 32551191552 bytes) Post the last line in the encode that looks like this, along with the type of movie it is (animated, computer animated, live-action)

Note that compression is currently DEAD SLOW. This is because I check the integrity of the uncompressed data in case I did something wrong. I'll remove the integrity checks in the next version, which will make it much more usable.

I also added the --nocomp option to easily turn off compression on the controller side without having to change the config files or restart the agents.

Based on my limited testing, the compression is usually ~3:1 for live-action stuff, and ~4:1 for anime.

Agent-based piping
If the agent's config.xml file has a line like this:
<agentpipe>1</agentpipe>that means that, in agent-based encoding, the agent will pipe data to x264 just like the controller does in controller-based encoding. It will keep all the benefits of agent-based encoding, and it lets you use a 64-bit x264 with 32-bit avisynth. See this thread (http://forum.doom9.org/showthread.php?p=766596) for more info about it.

Bugs
There was a bug which caused frames to be given out to multiple agents at the same time, if agent-based encoding was used. 1.05 makes it very unlikely to happen, but it still may. This bug does not affect the integrity of the file at all, just the speed.
The controller now searches for avs2yuv in the controller's directory, which would seem like a logical place to search.

SpAwN_gUy

13th December 2006, 13:17

foxyshadis

13th December 2006, 13:56

32-bit versions run fine on 64-bit windows, they just don't get the small additional gains of pure 64-bit code. If it's 64-bit *nix, it shouldn't be any harder to compile it than a 32-bit version.

omion

13th December 2006, 17:36

Nice.. will give it a try :)

well... i'm not sure about Compression.. only, maybe will give it a try.. for a TestEncode of PotC2 last Titles ~5mins

so.. another question (you are the Guru, i'm only the Encoder :) )
>>>Under my control i have 32bit and 64bit PCs ... so do i really need to find somewhere 64bit version of x264?
okay.. will try.. will it work under x32 windows with x64 processors?

x264farm is only 32-bit, but it will work perfectly on 64-bit operating systems (I have 64-bit Windows here).
If you have 32-bit Windows, then you can only use 32-bit software (the 64-bit CPU is "hidden")
If you have 64-bit Windows, then you may use my 64-bit x264 build. It's about 10% faster than 32-bit. I didn't make a 64-bit x264farm build because that would just waste more memory than the 32-bit build.

Can we use SharkTooth(damn, i can't remember his Nick.. again)'s builds or only Yours?
in Agent-BasedEncodings... i think it is the most convenient way.. 'cause i don't have a Server .. well. i do have some under my controll, but not me :)

In agent-based encodes, yes. For controller-based encodes, you have to use mine (for now... I'll bug somebody into making a patch. I don't like to do that, but a lot of people are asking for Sharktooth :D )

as soon as the Console program will run smoothly.. i'll make a starterGUI :) .. honestly ;)
Great!

SpAwN_gUy

15th December 2006, 10:01

oaky,... well it seems.. that i do have some problems..

controller still hangs when Exiting all the Jobs..("... agent .. DEAD" or something). then it made some kind of an error when it was exactly 50%. and with version 1.05 i'm looking at some strange behaiour: somehow wery strange it continues the Encoding.. currently i've started it for the third time and it is still 1% ... it was more before :(

P.S.: GUI Developement already started (Delphi, Standart components(mostly.. i'm not sure about our XMLParser.. but others are stndrt), PlainText-Files(no DB.. i was advised to do a WebSevice,.. but it is simplier to write PlainText :) ..))

BTW: Agent x64 runs Great on Vista Ultimate :) ...

today or tomorrow i'l borrow DualCore p4 Duo :) ...

SpAwN_gUy

15th December 2006, 11:29

You've requested Compression levels :)
01: 19.9% compression (9147744 / 45909504 bytes)
01: 21.2% compression (14221274 / 67184640 bytes)
01: 23.5% compression (1822839306 / 7761692160 bytes)
01: 23.5% compression (1821968890 / 7758332928 bytes)
TYPE: Soup Opera (X-Files Season 1 first DVD :) )

and i still don't have access to look for Errors.. so.. later.

I have Only 1PC with controller-based encoding... but it REALLY Slows My Controller.. damn.. the Prcentage of CPU Usage is somehow LOW,.. but the Speed of reacting - is terrible.

AVmaniac

15th December 2006, 15:21

omion

15th December 2006, 16:52

after making some tests and having some movies successfully encoded i recognized a strange behaviour:
every winXP based machine running the controller is seperated from the network after doing some work.
I can ping addresses but can not browse any websites or getting my mails ... tried over telnet .. but nothing works ...
the locally installed ftpserver is not accessible...
and no new(preconfigured) agents can connect to the controller..
but after stopping the controller everything works fine again ..
until it was running again for some time....

That's very strange... How long would you say it works before everything gets disconnected?

When you run into the problem next, run "netstat" on the controller computer in a separate command window. There should be a few lines that look like this:
TCP LOCAL-COMPUTER:2170 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2171 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2172 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2173 AGENT-COMPUTER:50700 ESTABLISHED
If there are lots of them that show up, then it could be an issue with running out of usable ports. I ran into this with early versions of x264farm. (It should be fixed, though...)

What kind of firewalls / antivirus do you have on the computers?
Also, what kind of router do you use for your internet connection?

You've requested Compression levels
...
Wow. Those are really good. 4:1 lossless compression.
I have Only 1PC with controller-based encoding... but it REALLY Slows My ControllerYes, it makes encoding go really slow... the next version should be much better.
the Prcentage of CPU Usage is somehow LOW,.. but the Speed of reacting - is terrible.I don't know exactly why this is, but I noticed the same thing on my computer. For some reason compression prevents the controller from using all of the CPU. I'll look into it if it persists after I remove all the debug code.

AVmaniac

17th December 2006, 23:09

That's very strange... How long would you say it works before everything gets disconnected?

I think about 45mins.

When you run into the problem next, run "netstat" on the controller computer in a separate command window. There should be a few lines...
If there are lots of them that show up, then it could be an issue with running out of usable ports.

No, seems as everything is normal .. there are only a few entries listed.

What kind of firewalls / antivirus do you have on the computers?

First also thought of that as problem source and deactivated everything for testing...
I've also reinstalled WinXP (had to for other reasons)

Also, what kind of router do you use for your internet connection?

I am connected to a campus LAN with a 100Mbps port.
It is connected to a 5port GBit switch providing the connection
for my two PC's.

I also have the problem that everything (agent and controller) , don't know if agent or controller is responsible, simply stops working without crashing.

Thanks for your reply omion

SpAwN_gUy

18th December 2006, 10:18

Okay, i've Figured out why my PC Slows down(controller).. it's bacause of te Huge Local Shares Usage(agent-based)

It seems like i'm having the same problem as AVmaniac... 'cause my Miranda stops responding and connecting after a while saying something about sokets..

and , man, there is something strange about 1.05 ... em.. it DOES NOT CONTINUES the ENCODES.. REally... and it hang again.. (on 93%!!!!! damn)..("the job compleded successfully" thing)
well, the compression - is a good thing,... but i think you've made something wrong since 1.04.

and the 1.04 controller does not work with 1.05-Agents :( ...

okay,.. i am stopping for some time.. and waiting for a new version...

AVmaniac

18th December 2006, 11:09

Last night i startet to test x264farm with normal DVB-S content using dgdecode and until now everything went fine, no network problems .. no hanging (curently 63% of the 2nd pass).
Seems as if it has something to do with the DVB-S2(1080i) content i wanted to encode before.
Perhaps it's the decoders fault (i'm using CoreAVC) or a mix of things.

that's all for now but will keep on testing.

Status Update:

the DVBs capture farm encode finished successfully, so it had to be a decoder bug.
To figure it out I installed version 1.2 of corecodec's fine software(I'm a legal buyer) and resumed the HDTV encode
and until now everything runs fine!!!
network ok ... no hanging .. nothing .. just like omion had it in mind to work like !!! :D

Status Update Nr.2:

I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
my useable network connectivity is gone.
But the controller is running and running and ... like the Duracell Rabbit ..
so i've to wait for the controller to finish before being able to use my WinXP machine again for network access.
And here is what putty is telling me when trying to connect to my gentoo machine:

Unable to open connection to
mygentoobox
Network error: No buffer space available

Status Update Nr.3:

everything stopped working again, and because there seems to be no resuming mode for 2nd pass
I've to choices a) waiting for a new version of x264farm
b) keep on trying...

so far, AVmaniac

SpAwN_gUy

19th December 2006, 17:31

and resumed the HDTV encode
and until now everything runs fine!!!
network ok ... no hanging .. nothing .. just like omion had it in mind to work like !!! :D
you know.. after a downgrade to 1.04 - yesterday it made the same thing :) ... too bad tha i've made a wrong Script.. and the DVD is not well made... :(
I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
you know :) .. the same here.. when i'm sitting in front of the PC - everything is Fine :) .. and when i'm leaving my workplace... 3 hours and "Bang - he's dead" :)
everything stopped working again, and because there seems to be no resuming mode for 2nd pass
I've to choices a) waiting for a new version of x264farm
b) keep on trying...
as i've reported i think this is the bug of 1.05 :( ... the major bug... i think. with 1.04 it just continued the encode.. well the percentage was from the beginning, but moved REALLY fast...(normal ~35(-45)fps.. continue ~80fps)
look in your temp-dir .. how many copies of files "100 100-130 ????.mkv" you have? ... i had the number of restarts..

AVmaniac

21st December 2006, 01:04

i've just downloaded it and windows told me that it is no
valid win32 executable.

omion

21st December 2006, 03:36

i've just downloaded it and windows told me that it is no
valid win32 executable.

Yeah. I think I put the 64-bit version in there accidently

[edit: should be fixed now]

omion

21st December 2006, 05:07

I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
my useable network connectivity is gone.
But the controller is running and running and ... like the Duracell Rabbit ..
so i've to wait for the controller to finish before being able to use my WinXP machine again for network access.
And here is what putty is telling me when trying to connect to my gentoo machine:

Unable to open connection to
mygentoobox
Network error: No buffer space available

This looks good (an a bad way... but it shines some light on the issue) The "no buffer space available" error means that you ran out of memory somewhere important. The buffer memory is taken out of the non-paged pool, which means that it must be physical memory, and 32-bit Windows has a hard-coded maximum non-paged pool of 256MB.

If you don't mind trying to repeat the problem:
when the problem occurs, go to the task manager > Performance > Kernel memory > Nonpaged. If it's near 256000 then that could easily be the problem. If not, then it may be a more general memory issue.

I've also been told that the error can also be cased if some internal TCP queue is full, and there are rumors that it may be related to XP's limit on the number of connections per second. Neither of those are likely, as x264farm only makes one connection per client.

What kind of network cards do you have in the computers?

omion

21st December 2006, 05:19

as i've reported i think this is the bug of 1.05 :( ... the major bug... i think. with 1.04 it just continued the encode.. well the percentage was from the beginning, but moved REALLY fast...(normal ~35(-45)fps.. continue ~80fps)
look in your temp-dir .. how many copies of files "100 100-130 ????.mkv" you have? ... i had the number of restarts..

I just got around to verifying this, and it looks like you're right. The 2nd pass does not restart correctly.

I don't have much time to fix it now, as working in retail at Christmas doesn't leave me much time or energy. I'll get to it as soon as I can, though.

SpAwN_gUy

21st December 2006, 15:22

I don't have much time to fix it now, as working in retail at Christmas doesn't leave me much time or energy. I'll get to it as soon as I can, though.
well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

i'm still using 1.04..
If the 1st pass folder is not deleted(some error) it restarts 1st pass again(when restarted) :( ..
can you tell me which files should be in the Temp folder after the first pass?
or what files are deleted after the first pass? :)

and sometimes not exits(Killing at the end) and hangs there(i guess some TTL would be nice)
2006-12-18~23:50:30.25 pc56 2 exiting...
2006-12-18~23:50:30.25 pc661 2 close_connection
2006-12-18~23:50:30.28 pc17 2 attempt_connect sees no jobs left; DEAD

BTW.. if agent is run as Service.. it can only encode in controller-based encoding :(

708145

21st December 2006, 15:37

well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

I have to step in here ;)
There is Handbrake on Linux and ELDER on windows which are rather similar.

omion

22nd December 2006, 05:45

well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

i'm still using 1.04..
If the 1st pass folder is not deleted(some error) it restarts 1st pass again(when restarted) :( ..
can you tell me which files should be in the Temp folder after the first pass?
or what files are deleted after the first pass? :)

After the first pass completes successfully, the final stats file is written to the AVS directory, then the 1st pass subdirectory is deleted.
If the subdirectory is not deleted, the encoding will assume that the file is not written properly, and will continue on the first pass. If all goes well, it will realize that the first pass is in fact done, and will write the stats file and try to delete the temp directory.

Here's how it works, in case anybody was wondering:
The controller will make a directory like:
"temp_root\0123456789ABCDEF0123456789ABCDEF avsfile.avs"
for all the info related to the AVS.

Inside that directory, the first pass will make a directory like:
"1-0123456789ABCDEF0123456789ABCDEF"
which will hold all the files specific to the first pass string that you used.

In the first pass directory, there should be exactly three files:
"settings.txt" is a human-readable file which contains the settings used for this pass
"working_stats1.txt" and "working_stats2.txt" for storing the frames which have already been calculated. In each encoding, one of them is read from and the other is written to. This ensures that there is never a point when there is nothing in the stats file when re-encoding.

Once the first pass is done, the final file written to the AVS directory is:
"1-0123456789ABCDEF0123456789ABCDEF.txt"
where the numbers are the same as the first pass directory. If this file exists and the directory of the same name does not, then the controller assumes that the first pass is done. BTW: This file should be a drop-in replacement for x264's stats file, so you can refer to it if you want to do the second pass without x264farm.

When the second pass starts, you will get another directory in the AVS directory:
"2-0123456789ABCDEF0123456789ABCDEF"
where the numbers are based off the second-pass settings. The first and second pass settings are completely separate, so if you use only change the second pass settings, the controller will see that the first pass is done.

The second pass directory has some number of files in it:
"working_stats1.txt" and "working_stats2.txt" are similar to the first pass, but have a slightly different layout.
"1 - 123 456 abcdef.mkv" are the rendered scenes. Merging together all these files in numerical order will result in the complete video. There should never be more than one file with the same "1 - 123 456" part, but as SpAwN_gUy pointed out, the second pass is doing something wrong when resuming.

Once the second pass finishes, the output file is merged, the second pass directory is deleted, and another file is placed in the AVS directory:
"DONE-0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF.txt"
This file contains the name of the output file. The first 32 letters are the same as the first pass directory name, and the last 32 letters are the same as the second pass directory name. This is so that x264farm can put up that annoying little "This job was already encoded..." message. I really need to put an "encode anyway!" option in there...

and sometimes not exits(Killing at the end) and hangs there(i guess some TTL would be nice)
2006-12-18~23:50:30.25 pc56 2 exiting...
2006-12-18~23:50:30.25 pc661 2 close_connection
2006-12-18~23:50:30.28 pc17 2 attempt_connect sees no jobs left; DEAD

If that's the last line that it printed, then it does look like the entire thing just hung. Normally it would say "pc17 2 exiting..." right after "pc17 2 attempt_connect sees no jobs left; DEAD" I'll try to reproduce it.

BTW.. if agent is run as Service.. it can only encode in controller-based encoding :(
Bummer. That's odd though, as there is very little difference externally between agent- and controller-based encoding, other than the command line that the agent runs. Maybe x264 doesn't like running as a service.

I have to step in here ;)
There is Handbrake on Linux and ELDER on windows which are rather similar.

Yup. ELDER is similar to x264farm, but does Handbrake do distributed encoding? I thought it was just a one-click DVD encoding thing.

omion

22nd December 2006, 05:57

BLEH. I finally found the reason that the second pass is not resuming: I commented out all the code which handles resuming :rolleyes:
I now remember what happened: I was about to re-write the code that handles resuming to be more functional and less imperative. Imperative code tends to be harder to debug, and the second-pass resuming code was nasty. So I commented it all out and was planning to put something in its place when I completely forgot about it...
:stupid:

I try to look through the diffs before I commit the changes, specifically to prevent this kind of thing. I guess I just missed it this time...

1.06 will work! :p

foxyshadis

22nd December 2006, 09:15

Bummer. That's odd though, as there is very little difference externally between agent- and controller-based encoding, other than the command line that the agent runs. Maybe x264 doesn't like running as a service.

Might be a write permissions thing, unless it's running as local system or the user. Network service has very few write rights. Local service has no network rights. And there's always a few programs that don't like non-interactive, but I can't figure out how a bunch of command-line programs would be.

omion

25th December 2006, 20:52

Sorry about the wait. 1.06 is out (Merry Christmas!)

You don't need to update any of your config.xml files if you have 1.05. Just replace the controller.exe and agent.exe files.

Primarily, it fixes the "not resuming on 2nd pass" issue.

There are a few minor fixes, including asking for a confirmation to re-encode the file if it sees that it was encoded before.

The main fix, though, is one I just stumbled across right before I was going to release it on Saturday. This is probably the problem SpAwN_gUy was having with broken files (here (http://forum.doom9.org/showthread.php?p=915266#post915266)).
Apparently, x264 will exit if the requested bitrate is less than the lowest bitrate possible (instead of just overriding the requested bitrate) On long amounts of blank frames, the controller may request a smaller bitrate than possible, and the agent won't recognize the error. This results in an empty file being returned.

The main problem with this is that the MKV header is taken from the first encoded file. If the error occurred in the first file, then the output file won't have an MKV header. This means the entire file will be unreadable.

I now have the controller try not to send bitrates that are too low, and the agent will recognize when x264 exits weirdly.

The bizarre network-based issues (probably) haven't been fixed, but I'll see what I can tweak for 1.07.

deets

25th December 2006, 22:31

does this work with the new threading of x264? could i for example set the threads to auto on each machine?

this would really speed things up. sorry if its an obvious question :)

omion

25th December 2006, 23:27

It will work just fine, as long as you set it up right...

Using the new threading will not give you any performance boost with my x264 build, so if you use controller-based encoding it won't help.

Using agent-based encoding with e.g. Sharktooth's build will work nicely, and it means you would only need to have each computer run one concurrent encode.

There are some performance issues with this, though. Especially during the second pass, the x264 threading won't scale as well as having multiple x264 processes. This is due to the time between jobs when x264 is setting up or sending back the rendered data. It is always a good idea to have another x264 running in the background to fill in the time.

I, personally, would use either x264farm or x264's native threads, not both at the same time.

SpAwN_gUy

28th December 2006, 13:11

Okay.. i've send some LOGS.. to your eMail... and.. i think i'll not have any HTTP Traffic till next Year i guess...

1.06 hangs at closing_Session 's with One file(my current encode,.. previous small went just fine..)
and i'm having something strange with the second file..
on the second pass it can not encode it in the end...
>> 4.81 FPS total on 127459 / 127584 frames (99%)
and i'm getting
>>pc418 2 process_job failed with "End_of_file"; putting job back
or
>>pc6 1 process_job failed with ("The operation completed successfully.\013\n",recv,); putting job back

and then suddenly controller trys to stop(close connections) it hangs.

*offtop* yeah, i've seen "Ani-x264" with x264FARM support.. and i didn't liked it :( .... not so many options.. and no Agent-based encodes.. and no Compression...
and my programmings are not going so well.. i'm not good with free time .. :(

omion

1st January 2007, 07:05

I'm doing a few more tests, and I can confirm a few bugs:

I recreated AVmaniac's problem with the controller computer disconnecting from the network ("No buffer space available"), and I realized that it had happened to me a few times before. I don't know exactly why it happens, but I'll try to tweak some settings to make it not happen anymore.

Also, I think I know what's wrong with some of SpAwN_gUy's files... I'll keep looking into it.

omion

1st January 2007, 22:05

every winXP based machine running the controller is seperated from the network after doing some work.

I think I found the problem (or at least a problem). For some reason there is some kind of file handle leak in the controller. I don't know exactly where it comes from, but on my computer the number of handles increases about 5/second.

This normally wouldn't cause the whole computer to lose connectivity, but it looks like there is a corresponding increase in the number of handles in the "system" process, which could easily cause bizarre issues if they get too large. I think the maximum number of handles for a 32-bit process is 10000...

I'll see if I can fix that somehow.

[edit: more info...]
If you can recreate the bug, fire up the task manager, go to the Processes tab, then turn on View > Select Columns > Handles and see what it says for the "System" and "controller" processes.

Note: there looks to be a workaround here (http://support.microsoft.com/kb/327699), which raises the handle limit. The controller shouldn't need that many handles anyway, so I'll fix it as soon as I can.

omion

5th January 2007, 07:10

Well, I think I found the cause of the controller computer disconnecting from the internet. It's partially Windows' fault, but I am probably the most to blame.

It's my fault because I don't know how to program sockets correctly.
It's Windows' fault because it lets me do it anyway. :p

When each thread from the controller tries to connect to its agent, it starts up a socket. If the connection fails, it shuts down the socket and tries again.
That's apparently the wrong way to do it. I guess after the socket is shut down, it's still a valid handle that has just been thrown away. And the OCaml garbage collector doesn't clean up handles...

This basically means that every 5 seconds one socket is used up per disconnected agent port.

On a Unix-based system, the sockets are filehandles, and the program will die with "too many filehandles open". But on Windows, it just keep on chugging until the OS runs out of... whatever sockets are made out of.

I'll see what I can change around to make it stop doing that. It shouldn't be too hard (assuming that's the only thing wrong... :D )

SpAwN_gUy

5th January 2007, 10:02

:) ... i was pointing on it :) ..... netstst stats - rock :)

omion

5th January 2007, 18:07

:) ... i was pointing on it :) ..... netstst stats - rock :)

Actually, netstat doesn't show the sockets that are causing the problem (which is why it took so long to fix!) The problem sockets are already closed, so they don't show up on any network monitor. But the socket descriptor is still around, which makes a problem for the OS when it runs out of them. The only place I found them listed is in the task manager under "handles".

I actually don't know what caused the large amounts of half-closed sockets that you sent me. Do you remember which encoding you were doing when you used netstat? Was it associated with one of the log files you sent?

BTW: some good news (at last)
I found that you actually don't need to map a network drive in order to get agent-based encoding without copying anything.

If you give a line like one of the following into the <base> part:
\\192.168.1.100\shareddir\movies
\\COMPNAME\shareddir\movies
\\COMPNAME\shareddir\
it will connect to the directory which is shared as "shareddir" on 192.168.1.100 (or whatever computer is named COMPNAME). If you want the shared directory itself to be the base, you MUST add a trailing backslash. If you want a subdirectory of the share to be the base, you MUST NOT use a trailing backslash. (like I did above)
You can see if it works by looking at the output right after the agent starts up.

omion

6th January 2007, 03:31

1.07's out. The disconnecting problem should be solved.

Note that you only need to update the executables, not the config.xml files.

I had planned a couple additional features for this release, but they ended up being WAY too hard, so it's basically just a bug-fix now.
I was going to put in a way of giving out a job to multiple agents at the end of the encode, just to make sure a dead agent doesn't stop the pass from finishing. But that turned out to be immensely difficult the way the program's currently structured.

PS. I found out that you can actually run entire programs through shared directories in the command prompt. This means you can type something like this:
\\192.168.1.100\x264farm\agent --config \\192.168.1.100\x264farm\config-agent.xml
and just have to update one agent and one config.xml.

SpAwN_gUy

29th January 2007, 12:11

I'm Back Again :) ...
new version - COOL.. i'll start testing it now..
with those advices(about netshares) - COOL Again :) ..

Okay.
so.. those Advices with netshares.. it works :) ... but..

//for the future generations
Methods with netShares works ONLY when:
agentpipe = 1
Nice(with .dll), avs2yuv, x264.exe ARE in the Folder of .bat-File(netStart I guess)..
well.. x264.exe is no necessarily(written probably with mistakes ;) ) must be in the folder of .bat-File..
<x264>\\PC201\x264farm\!AGENT\x264.exe</x264>
still works :) ...

so.. now No need to UpDate agents .exe's with new versions.. and lots of problems are gone :) ... and no x264.exe updates ... cool :) ...

omion

30th January 2007, 06:42

Hmm... that's a bit odd. Here's my setup (which I tested everything on)
Agent and config file on remote computer. Started with:
\\192.168.1.100\path\to\agent --config \\192.168.1.100\path\to\config-remote.xml
Config file contains the following:
<config>
<temp>C:\Documents and Settings\Omion\My Documents\x264farm_temp</temp>
<port from="50700" to="50703"/>
<x264>x264-614_omion.exe</x264>
<nice>0</nice>
<bases>
<base>\\192.168.1.100\DVD\MKV</base>
<base/>
</bases>
<agentpipe>0</agentpipe>
</config>

avs2yuv doesn't exist on the agent computer (not needed since agentpipe=0)

x264 is somewhere on the path (not quite sure where anymore... but it's not in the agent directory)

It works just fine with agent-based encoding, too.

BTW: There haven't been any updates in a while because I've been completely redesigning the first pass.

The old implementation would just split the movie into a bunch of chunks and send them sequentially to the agents.
The new one will attempt to keep the agents as separate as possible, and will try to start agents right after they left off. The agents will also try to avoid anything that could keep everything waiting on one agent to finish (although that's impossible to do all the time)

The improvements should make the very end of the first pass much faster. Depending on the file it could decrease first pass encoding time by ~5%.

I suppose I should get around to updating my x264 builds, too... I was hoping that they wouldn't be needed by now, but they still are. I'll build them soon.

SpAwN_gUy

5th February 2007, 11:56

hmm...
i've noticed <bases> part.. does it work in 1.07? ;)

there are still some problems with minimum bitrate sended to x264(i've tried the same encode on different bitrates and different firstpass settings(noticed that during controllerGUI Developement in past few days... it can now import jobs from MeGUI)).

and.. my people noticed(me too), that when agent is run through the network it sometimes crashes.. (i haven't found the reason), 'cause even when hostPC 've been rebooted agent were still up and running.. dunno why this happened, really.

omion

6th February 2007, 06:08

hmm...
i've noticed <bases> part.. does it work in 1.07? ;) I haven't had any problems with it. Try it out!

there are still some problems with minimum bitrate sended to x264(i've tried the same encode on different bitrates and different firstpass settings(noticed that during controllerGUI Developement in past few days... it can now import jobs from MeGUI)).Yeah. I've been busy on another program lately, but I'm probably going to have another version of x264farm out by Wednesday (California time :D ). I think I can fix the minimum bitrate thing fairly easily, but...
and.. my people noticed(me too), that when agent is run through the network it sometimes crashes.. (i haven't found the reason), 'cause even when hostPC 've been rebooted agent were still up and running.. dunno why this happened, really.What does it do when it crashes? Does the whole program exit, or just refuse to encode anything?

SpAwN_gUy

6th February 2007, 08:52

I haven't had any problems with it. Try it out! i think i will :) 'cause i'm a bit short in HDD-Space..
Yeah. I've been busy on another program lately, but I'm probably going to have another version of x264farm out by Wednesday (California time :D ). I think I can fix the minimum bitrate thing fairly easily, but...
Looking forward to it :)
What does it do when it crashes? Does the whole program exit, or just refuse to encode anything?well, it simply exits,.. and that's all..after doing some job.. (i'm not really sure).. sometimes suddenly after 2 hours of working,.. sometimes after 6 hours idle.. i don't know the reason...

omion

6th February 2007, 09:04

well, it simply exits,.. and that's all..after doing some job.. (i'm not really sure).. sometimes suddenly after 2 hours of working,.. sometimes after 6 hours idle.. i don't know the reason...
Hm. Weird. I'll test it out and see if I can reproduce that. It's possible that a network hiccup (like a quick disconnect) would make it panic, but I don't know.

SpAwN_gUy

6th February 2007, 10:49

Hm. Weird. I'll test it out and see if I can reproduce that. It's possible that a network hiccup (like a quick disconnect) would make it panic, but I don't know. the THING is that.. I ReBOOTed my PC (for 3 mins or so) and NONE of 2 working agents EXiTED... so, it may be problematic...
i'll try to find out the reason here.. but..

omion

6th February 2007, 20:46

Oh... THAT'S why my calculations for the minimum bitrate don't line up with x264. From ratecontrol.c:
/* FIXME: ugly padding because VfW drops delayed B-frames */
rc->num_entries += h->param.i_bframe;
x264 adds a couple of frames, and initializes them to a relatively large number of constant bits, which raises the minimum bitrate, which causes x264 to flip out for blank scenes.

I'm going to see if removing that line does any harm...

omion

8th February 2007, 06:47

1.08 is out! At last!

I completely redesigned the first pass to hopefully make it more efficient. The controller program should use less CPU power, less RAM, and will get finished faster than before. (How much faster depends on the movie and encoding settings, though. Very slow, dark movies will improve the most)

The second pass is a bit more error-resilient. Before 1.08 there was a problem with scenes that were very short and entirely black, which I mentioned in my previous post. 1.08 may still report some failed encodes, but everything should get done eventually. I wrote akupenguin about the possibility of removing the line I mentioned before from x264.

The syntax and config.xml files have not changed; you only need to replace the binaries.

SpAwN_gUy

9th February 2007, 10:30

Nice... TESTiNG in Progress :)

whatever-ever

13th February 2007, 20:08

I need to configure my setup a little better (possibly it's my avs that's causing problems).

Currently the avs is located on controller only, and loading plugins via network (even though it's on the local machine), and d2v is linked to locally (not via network). It is a complex as it sounds, but it was the editing in the avs to that, which made the external agent catch on.

wierd thing is it reports a lot about d2v file not being found (on external) which is obviously because of the pathing you might say, but no... i tried both \\network\dir\ and C:\local\dir\, same error -- HOWEVER, the controller reported an increasing boost (going from 1 hour to 30min to 4 (!) min ETA).
And get this: the boost came when i connected the external agent on wifi!!

Since I got it working kinda "half-ass", changing avs while encoding (when restarting the external agent), I was wondering if someone could help me set up the avs properly.

Current setup:

controller config:
<config>
<temp>C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller\temp</temp>
<agents>
<agent name="LOCAL">
<ip>127.0.0.1</ip>
<port from="50702" to="50702" />
<number pad="1">1</number>
</agent>
<agent name="LAPTOP">
<ip>10.0.0.7</ip>
<port from="50701" to="50701" />
<number pad="1">1</number>
</agent>
</agents>
</config>
LOCAL agent config:
<config>
<temp>C:\!RAW\__x264farm\x264farm-1.08-161_win32\agent\temp</temp>
<port from="50702" to="50702" />
<x264>x264.exe</x264>
<nice>5</nice>
<base>C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
<agentpipe>1</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>
LAPTOP agent config (external):
<config>
<temp>C:\Documents and Settings\****\Desktop\x264farm\agent\temp</temp>
<port from="50701" to="50701" />
<x264>x264-622_omion.exe</x264>
<nice>5</nice>
<base>C:\Documents and Settings\****\Desktop\x264farm\agent</base>
<base>\\CONTROLLERPCNETWORKNAME\\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
<agentpipe>1</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

AVS (C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller\test.avs):
SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("C:\!RAW\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

The controller cmd line:
encode.bat:
controller.exe --avs %1 --bitrate %2kbps --first "--crf 19 --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 5 --ref 8 --me umh --bframes 7 --weightb --b-pyramid --analyse all" --second "--trellis 2 --bime --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 7 --ref 12 --me umh --bframes 7 --weightb --b-pyramid --analyse all --8x8dct" --output "output.mkv" --nocomp --batch 2500 --split 250
executed as:
encode test.avs 1224

The AVS is only located in the controller folder.

The goal: to encode agent-based.

I know it's a bit wrongly set up (the avs part, right?) but it kind'a worked.

First pass took about 50 mins to encode (don't know what the ??? went wrong there..) and second pass about 5 minutes.

Thanks in advance for any help.

PS: Thumbs up, omion :) I've been looking for a project as such for a long time. Thanks!

omion

13th February 2007, 20:34

ETA: 1:51:34How long is the movie? How long would it have taken with the same settings without using x264farm?

and that's on 6%, oh my.. GOD it's slow. ETA is rising for every pct. the encode is increasing. And this is the second pass. The first pass was not that fast either.

The setup is one controller + one agent on SAME machine (x264 running 50%)

I need to work a bit on getting my laptops working in the setup too, but even though it's only 50% locally it shouldnt be THIS slow. I must have done something wrong. I'll post configs and dump.txt later

That would definitely help.

Using x264farm on one computer is always going to be slower than not, simply because of the extra overhead. Also, x264farm will re-process any scenes that varied wildly from what was expected, which will result in a higher quality but which may take 10% longer than not using it.

A few more questions:
Are you using agent-based encoding? If the agent says something like this then you are:
Testing file "file.avs": FOUND!
2 agent-based encoding

What is your CPU usage like? Is it maxed out? You said x264 is using 50% of your CPU, but is something using the rest? (also, is it a dual-core?)

whatever-ever

13th February 2007, 20:55

How long is the movie? How long would it have taken with the same settings without using x264farm?

11min clip and normally it's 25fps for first pass, 11fps for second pass. so about 25 min.

Are you using agent-based encoding? If the agent says something like this then you are:
Testing file "file.avs": FOUND!
2 agent-based encoding
yes I am, I've updated my post to make it more specific, the first was embaressing, sorry.

What is your CPU usage like? Is it maxed out? You said x264 is using 50% of your CPU, but is something using the rest? (also, is it a dual-core?)
It's hyperthread, but not dualcore per say. Yes, 100% cpu usage during most of the first pass (not sure during second pass, it went very fast so I didnt get to check). 50% x264, 50% avs2yuv, BUT, x264 doesnt exceed 50%, and avs2yuv didnt run all the time (for quite some time cpu usage was only at 50% total).

I've updated with specs on my setup if it helps,

EDIT: laptop cpu usage is roughly 20-35%

thanks for the quick reply

omion

13th February 2007, 22:11

I need to configure my setup a little better (possibly it's my avs that's causing problems).

Currently the avs is located on controller only, and loading plugins via network (even though it's on the local machine), and d2v is linked to locally (not via network). It is a complex as it sounds, but it was the editing in the avs to that, which made the external agent catch on.
That is a bit odd, but sometimes it's necessary. However, I don't think that the avs will work on remote computers. It's best to use relative paths wherever possible.

For example:
SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("..\..\..\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

And make sure the .d2v file references its source files relatively too; when you create the file, turn OFF "Use Full Paths". If you open the .d2v file with a text editor, the VOB files should be referenced relatively.

wierd thing is it reports a lot about d2v file not being found (on external) which is obviously because of the pathing you might say, but no... i tried both \\network\dir\ and C:\local\dir\, same error -- HOWEVER, the controller reported an increasing boost (going from 1 hour to 30min to 4 (!) min ETA).
And get this: the boost came when i connected the external agent on wifi!!
Uhh... so it can't find the d2v file, but it works anyway? I must say, this doesn't make a whole lot of sense. Care to explain this a bit more? Also, were those times that you posted the controller's ETA or the actual time?

Since I got it working kinda "half-ass", changing avs while encoding (when restarting the external agent), I was wondering if someone could help me set up the avs properly.

If you can get any encodes out of an agent, then you have set up everything correctly. The relative paths I mentioned before will ensure that the avs file will work on remote agents.

The goal: to encode agent-based.

I know it's a bit wrongly set up (the avs part, right?) but it kind'a worked.

First pass took about 50 mins to encode (don't know what the ??? went wrong there..) and second pass about 5 minutes.

Thanks in advance for any help.

PS: Thumbs up, omion :) I've been looking for a project as such for a long time. Thanks!

Looks pretty good. However, I noticed that you have two <base> lines in your laptop config. If you wanted to have the option of encoding from either of them, you need to use the <bases> element, like so:
<bases>
<base>C:\Documents and Settings\****\Desktop\x264farm\agent</base>
<base>\\CONTROLLERPCNETWORKNAME\\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
</bases>

However, judging by your paths, the first one (to x264farm\agent) probably won't be used to store any avs files.

whatever-ever

13th February 2007, 22:58

That is a bit odd, but sometimes it's SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("..\..\..\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?

[...] make sure the .d2v file references its source files relatively too

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

Uhh... so it can't find the d2v file, but it works anyway?
it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.

judging by your paths, the first one (to x264farm\agent) probably won't be used to store any avs files. Thanks! It also loads the network path each time (only place there is a avs), so you're right, I should just delete the other.

PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! :))

omion

14th February 2007, 00:58

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?
All those ..\..\.. mean to go from the .avs directory (C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller) to the directory of the .d2v file (C:\!RAW).

If you have the C:\!RAW directory shared somewhere, then the relative path will still make it point to the right directory.

Think of it this way: the way you had it, your laptop is seeing the .avs file refer to "C:\!RAW\test2.d2v" Does that path exist on the laptop? If not, then you have to make sure you share the proper path.

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

If you are sharing the directory "C:\!RAW" as "\\NETWORKNAME" then yes.

it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.
I thought so. ;) Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.

PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! :))
Yup. I noticed that you had the <agentpipe>1</agentpipe> line in your config.xml files, which needs avs2yuv to function. <agentpipe> is only needed if you have a 64-bit x264, but 32-bit avisynth. If you have 32-bit everything then <agentpipe> will only add a layer of complexity.

Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore :cool:

whatever-ever

14th February 2007, 08:17

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.
[...]
I thought so. ;) Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.
[...]
Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore :cool:

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).

I'll shut up and run a test :)

THANKS!

SpAwN_gUy

14th February 2007, 11:40

:) .. maybe i'll write a tutorial.... or maybe not ;) ...

so,.. i've come back with the tests of 1.08..

AWESOME!!!!!!!!!!!!!
i think you've modified x264... 'cause it RUNS pretty well,.. :)) ..
i've encoded that problemous file... but i've messed up with bitrate, so the file came oversized.. now i'm ReEncoding it..

minuses: agents are suddenly exiting(started over network). I have DUMP...
controller suddenly hanged yesterday.. on exiting first pass.. i dunno why. it was running smoothly, and when i left - something happened(i don't have dump..)

and i've made a some kind of contest.. calculated sum of fpses of agents.. (2x and 3x).. 2x =( 1, pad1) 3x = (2, pad1)
yeppie... now i have one CoreDuo2 !!!!!.... it's freaking awesome.. (and two or three are coming)
Agent | FPS
p4 Core Duo2 E6400 2.13GHz/2Gb/W32bit 3x | 7.05
p4 3000/1Gb/W32bit 2x | 3.05
AMD 3000+/1Gb/W64bit 2x | 2.86
AMD 3500+/1Gb/W32bit 2x | 2.83
AMD 3700?/1Gb/W32bit 2x | 2.67
p4 3000/1Gb/W32bit 2x | 2.23
p4 3200/1Gb/W32bit 2x | 1.48
p4 1800/512/W32bit 2x | 0.89
13.09 FPS total
now it' running ~16fps

now.. adout gui..
i'm a bit confused and lost in translation... what ARE the Assorted params??? I should never use them in passParams, or controller DELETES them from pass params?? .. and do i need "--threads 2" and "--thread-input" in passParams?

SpAwN_gUy

14th February 2007, 12:11

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).
the simpliest way to test will agent-based encoding run or not ("is everything OK? and will it run??") is to Open desired AVS from the Agent Computer(at least one computer) though the network.. if it runs - then OK.. it will..(well, about 30% for sure).

on each agent PC you will need AviSynth Installed.

ok, ok,... the Tutorial ;) :

dir structure:
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\file.d2v
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs
.vob are in:
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\iN\VIDEO_TS\
AviSynth plugz are in
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\filters

in the network paths it looks like:
\\controllerpc\x264farm\file.d2v
\\controllerpc\x264farm\fileAVS.avs
.vob are in:
\\controllerpc\x264farm\iN\VIDEO_TS\
AviSynth plugz are in
\\controllerpc\x264farm\filters\

.d2v internals (the head of..) should look like this
UPD..(2007-10-02): Don't forget to download DGIndex 1.5.0 beta 3 .. it has "Relative paths saving"
here (http://forum.doom9.org/showthread.php?p=1050990#post1050990)
DGIndexProjectFile13
5
iN\VIDEO_TS\VTS_01_1.VOB
iN\VIDEO_TS\VTS_01_2.VOB
iN\VIDEO_TS\VTS_01_3.VOB
iN\VIDEO_TS\VTS_01_4.VOB
iN\VIDEO_TS\VTS_01_5.VOB

Stream_Type=1
......

.avs is
SetMemoryMax(32)
LoadPlugin(".\filters\DGDecode.dll")
# SOURCE
mpeg2source("file.d2v", info=3)
...... etc

agent Config bases part lokks like
<bases>
<base>\\controllerpc\x264farm\</base>
<base></base>
</bases>

and i start the controller with
controller.exe --first "--bframes 3 --b-pyramid --direct auto --filter 1,1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-psnr --no-ssim" --second "--ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-psnr --no-ssim" --avs "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs" --output "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileOutput.mkv" --bitrate 1375kbps

all agents are encoding agent-based :) ...

BTW... just now 2nd pass finished..(first pass was about ~20fps)
START: 1000AM
END: 1307AM
this encode took about 20 hours to encode on my PC..

now i'm encoding one more time.. and i think most of agents - exited :( ... first pass is only using 3 of manyPC's.. :(

whatever-ever

14th February 2007, 15:01

Thanks for the tutotiral Spawn!

I'm not sure what did the trick, but LOCALLY it's going great now. Encoding with 20.7 + 13.16fps on the two local agents, VERY fast (same as when encoding without farm).

And the output is "better" on the local agents.
Reporting "encoded frames: 103/187" and updating continuously.

It might be the controller cmd line that's doing the improvements, but I'm not sure.

However I can't get the external agent to follow.
Getting a lot of Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255\")

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Now it's encoding at 0.44 and 0.57 locally and 3-4fps externally, okay... This is getting wierder and wierder!

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1 :)

SpAwN_gUy

14th February 2007, 16:08

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1 :)
it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw... What is "vdvmax"??

and... temp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

whatever-ever

14th February 2007, 16:23

it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw. twmp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

PS: exiting doesn't seem to be a problem - anymore. Don't know why, but all jobs are running smoothly, just not very fast.

whatever-ever

14th February 2007, 18:05

btw... What is "vdvmax"??

vbv ;)

omion

14th February 2007, 20:31

When you guys say it's exiting, does it put up any warnings or just die?

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

There is no way to tell x264farm to not do the second pass, but you can stop it manually and use the stats file it creates to do a separate second pass.

The created stats file is here:
<controller_temp_dir>\<lots_of_numbers> <name_of_avs>\1-<lots_of_numbers>.txt

For example,
E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs\1-4E2586CB8D8815243D6B8F08CE7FEA77.txt

Exactly which string of numbers goes with which encoding is displayed at the beginning of x264farm:
AVS temp dir is E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs
First string is "...", with MD5 "4E2586CB8D8815243D6B8F08CE7FEA77"

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Uhh... I think it's not encoding fine. Without avs2yuv, there is no chance that agentpipe=1 is going to encode at all. With agentpipe=0, the existence of avs2yuv won't matter.

I'll have to look at the dump file to say more, though.

whatever-ever

14th February 2007, 21:41

Setting up a mac, where do I get the x264 encoder for that?

whatever-ever

14th February 2007, 21:45

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

Yes all machines are 100pct. avs2yuv is taking a lot of cpu usage sometimes though (up to 50 pct.), however agents (not running avs2yuv) are slow too (not as slow, but still.. not so impressive)

I've saved the out-dump, but only from the controller, I'll check if it's usable and mail it.

omion

15th February 2007, 02:33

Setting up a mac, where do I get the x264 encoder for that?

To build it yourself, you need x-code (on the DVD that came with the mac) and SVN (over here (http://metissian.com/projects/macosx/subversion/))

Check out x264 from the repository:
svn co svn://svn.videolan.org/x264/trunk x264
Go to the x264 dir and type:
./configure
make
sudo make install
Assuming everything is set up right, that will get you the x264 binary.

You'll also need the agent for the mac. You can get 1.08 here (http://omion.dyndns.org/x264farm/x264farm-1.08-161_agent_intelmac.zip). I didn't put a config file in the archive, but it's set up the same way as in Windows.

SpAwN_gUy

15th February 2007, 09:31

When you guys say it's exiting, does it put up any warnings or just die?
well.. few people said that it shows something, but they didn't remembered that.. :(

and.. x264 takes about 30% of CPUtime..
i'll do some testings about agentpipe=0.. and will send logs to you..

...here we go
firstpass
pipe=1 nice=20 - avs=80% x264=27% 15.5fps
pipe=1 nice=0 - avs=50% x264=50% 16fps
pipe=0 nice=20 - error
pipe=0 nice=0 - x264=98% 17fps

secondpass
pipe=1 nice=20 - avs=10% x264=84% 4.3fps
pipe=1 nice=0 - avs=10% x264=85% 4.11fps
pipe=0 nice=20 - error
pipe=0 nice=0 - x264=98% 4.05fps

fistpass-local priority: normal
x264=72% 22fps

secondpass-local priority: normal
x264=80% 4.5fps
agent is p4 1800/512/W32bit
local(controller) is p4 2400/512/W32bit

... he-he... dump does not provide anything :) ... here are the screen dumps(theese were not provided in eMail):
Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt" --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

omion

16th February 2007, 04:00

Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt" --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

Oh! It looks like nice.exe does not appreciate running things from network paths. I'll see what needs to be done.

@whatever-ever:
I'll take a look at those logs you sent me. There should be something I can do about the speed.

SpAwN_gUy

16th February 2007, 09:41

yeah,.. and..
it seems, that x264farm has something wrong with Oversize :( .. i had 2 encodes of the same source(with different x264profiles), with same bitrate, and the files were
a) different in size (~30Mb)
b) they were oversized.. about ~60Mb to ~93Mb...
why?

omion

18th February 2007, 03:51

@SpAwN_gUy:

Well, the files are going to be different, since the scenes never finish encoding in the same order. They should not be that different, though.

It also looks like my bitrate-multiplier calculation is not quite right... There is not enough feedback to correct for any bitrate discrepancy, so x264farm is always trying to pull the bitrate up. I'll try to fix it in the next release (or at least make it better)

To fix it in the short-term, you can increase the number of 3rd-pass scenes.
The settings that control the third pass:
--3thresh How far off the bitrate must be in order to consider for 3rd pass. 0 = completely wrong, 1 = exactly right. Default = 0.8
--3gops The maximum number of scenes to consider for the 3rd pass. Default = a lot :p
--3ratio Same as --3gops, but as a percentage of the total scenes (well, not really percentage, but from 0 to 1) Default = 0.05.

The default parameters will re-encode up to 5% of the total scenes, but only if the scene's ratio expected_bitrate / actual_bitrate is less than 0.8.

To make the bitrate more exact, raise both --3thresh and --3ratio. Something like --3thresh 0.95 --3ratio 0.2

Meanwhile, I'll try to make the ratecontrol smarter.

whatever-ever

18th February 2007, 11:54

My speed was way up on a test I ran this friday. I'm trying to run another, but I'm having a little trouble with the avs somehow. Looking into it now

SpAwN_gUy

19th February 2007, 16:54

so.. how the loop goes(or a hang)
controller hangs on close_connection ... of agents at the end of second pass..
and after the restart.. it starts(or continues) to reEncode something.. it is 99% of an encode...(with modified 3tresh and.. as omion advised).. it happened 3 times for now.. (yeah i know... I have Lots of those errorous sources and everything is going crazzyy.. but i think it is good in terms of testing.. ;) )... so.. and keep going again and again... i'll modify .bat file.. and post some results tomorrow

//me sorry.. i'm a bit drunk right now.. but i'm allmost Russian(Belarussian.. telling the truth), so i can work and programm..
soory my girl Friend waits for me near work entrance.. so.... ;) ..

SpAwN_gUy

21st February 2007, 09:11

Yeah... and Is it possible, to run NOT Only the x264.exe with nice, but avs2yuv also?

'cause if agentpipe=1 and nice=20 then x264 is running in lowest priority, but avs2yuv is running Normal, and it slows down overall system performance,.. so my people can't work :( ...

and.. that option,.. that asks for "Do you really want to ReEncode the same AVS?"(or something)..
can you check for th SAME Bitrate and Same Filename? .. 'cause i had an encode with different bitrate and different output filename... it was nice, that i didn't need to make firstpass again,.. but..
/*played with nice..
well.. nice REALLy does not like NetPaths.. i've played with different variations of comLine.. and nice does not want to start avs2yuv with netPath param..
error is: "AviSynth can not find the file D:\controllerpc\x264farm\...."
and batch file was: nice -n 20 avs2yuv ... "\\controllerpc\x246farm\.avs" | nice.. x264...

theturtle

25th February 2007, 09:44

hrm, I can get this working on local worker but the 2nd machine I have doesn't appear to want to encode anything. i'm still plugging away at it. the network paths and setup was a tad difficult, but I think i've got that all worked out.

for some reason right now, the remote agent i have, receives a command to encode. yet it doesn't appear to be encoding.

I did have a question. regarding agent based encoding. If you are doing this, can the input video in the avs be on a network share for remote agents? Both my agents are using the same .avs and I just have the AVS looking for the input video on a Network drive I mapped(which is actually a local share on the controll/local agent machine). I guess thats not really a good question, I guess since they both use the same avs, they both have to have the same access to the input video and the path has to be the same, otherwise it won't work.

Anyhow, not sure whats going on with this remote agent, I can't get it to actually encode anything for nothing. The second pass will be up in an hour or 2, perhaps it will have better luck on that pass. :s

*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.

theturtle

25th February 2007, 15:14

omion

25th February 2007, 18:55

i thought i'd post this separately, i just finished an encode with 2 computers local agent and remote agent. the resulting encode has flashes of grey every few seconds or so. I thought at first this was during scene changes but its not. Its like a blank spot in the video or something. Perhaps made during merging process? :s

Are there entire frames which are gray, or does it seem to start in the middle of one frame and end in the middle of another?

Also, you can find out where the gray occurs by doing the following:
Turn off all the agents except for one (in the config file, or just don't start the other agents up)
Run the controller with the --keeptemp option (and probably also --frames xyz and --restart to make it fast and repeatable)
If the resultant file does not have the gray frames, then the problem is with another agent
If the resultant file does have the gray frames, then you can check the temp directory and see if the raw pieces have gray frames. If they don't then it's probably the merger that's doing it.

Also, just to make sure, the initial AVS file doesn't have the gray frames, right?

@SpAwN_gUy:
Sorry about the delay in fixing nice. I don't really have a lot of time lately, but I should get it done by Wednesday this week

AVmaniac

25th February 2007, 20:54

theturtle

26th February 2007, 02:24

*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.

^^^^ regarding this. For some reason if I set the <base> to my network drive that I mapped, my remote agent executes x264.exe but yet cpu is at 0%. If I copy the avs to the remote agent's agent-win32 directory, it appears to encode fine. Although I've just noticed, my source is h.264 mkv. I didn't have either ffdshow or haali media splitter installed on my agent. Should I need these? I figured the input to the agent was yuv from the controller.

*update* I just installed both of these and it appears its encoding, the network utilization skyrocketed from my encode last night. it was hopping around 20-50%, now its about 99% steady. :) perhaps that was my problem. hopefully it doesn't produce the same gray frames. we'll see. mooore testing ;)

I don't think it's a bug of the merging part.
Try to play the script with the WMP.
The grey frames should also appear here!
Did you try to reinstall your avisynth or tried a different version?
That's what helped me getting rid of this bug...

greetings to all x264farmers %-)

just talked with someone about this, they seem to be pretty confident its an issue with directshowsource. my source is h.264, i'm going to try the new avc/h.264 avs source plugin. I was getting blank gray frames in the source with directshowsource btw, so its definitely not a x264farm issue.

mpioner

26th February 2007, 12:08

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.

theturtle

26th February 2007, 14:10

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.

tried this?
http://forum.doom9.org/showthread.php?t=117428&highlight=x264farm+gui

dunno how well it works, the cli version took a bit of figuring and alot of mistakes on my part, but wasn't too difficult.

SpAwN_gUy

5th March 2007, 09:54

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.i'm playing around with Delphi... for like some months.. and it seems like a testing version will be shown in few weeks.. i guess..(well it's all unofficial)..

agentGUI has an UnKILLable option (if agent crashes it is auto-Restarted) and can minimize to tray.
controllerGUI has "Import MeGUI Jobs" button, and actually imports them,.. and generates command lide for starting controller.. :) ... that is for now,.. lots of ideas, and no time at all..

@ALL
is the Tutorial i wrote usefull? or maybe...??

omion

11th March 2007, 08:32

1.09 is out (or will be shortly, when my server reboots...)

It should be more accurate with the bitrate target, and the agents now use nice for avs2yuv (if both niceness and piped input are used)
I didn't add bitrate in the same-file detection (because I only remembered it just now)

Also, the agents will by default delete any temp file which is older than 1 week. I noticed my temp directory filling up with a bunch of junk after testing x264farm, and I decided to do something about it.
By default, a cleanup thread will wake up once a day and delete all the temp files that are over 1 week old. The amount of time is adjustable with the --stale <x> command-line option. <x> is the maximum age that a temp file can be, in seconds.

The release took so long because I've been trying to rewrite the nice.exe program. I think 1.1MB is way too big for a program like that, and the fact that it can't run network things is not so good either. However. in order to easily access process priorities I have had to use C/C++ (which I don't really know) and the whole thing ended up being extraordinarily frustrating and a huge waste of time. So now, instead of a new, slimmed-down nice.exe program, I have a reminder of why I program with OCaml in the first place :rolleyes:

SpAwN_gUy

12th March 2007, 14:36

Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't :( ... well.. it's not working now at all..

agentpipe=1
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
2 agent-based encoding
2 doing the following:
nice -n 20 avs2yuv -raw -seek 0 -frames 5223 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats 41425d.txt" -o NUL - 576x432
Avisynth error:
Import: couldn't open "D:\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"

agentpipe=0
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
7 agent-based encoding
7 doing the following:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --directauto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\PC201\x264farm-2\xFiles\S1p2\S1p2.avs'

7 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

i've downloaded archive woth nice once again,.. and i think it have not being changed..

i guess.. i'm going back to controller-1.09 and agent-1.08... waiting for nice, only :) ...

maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?

yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...

Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising :)
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool :) ...
37fps on start of second pass.. :) and rising :) ...

i think. today is a good day,.. i'm happy :) really :) ..

omion

12th March 2007, 19:54

Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't :( ... well.. it's not working now at all..

Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. :angry: I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.

I'll see what I can do about that program.

And just in case you were wondering:

maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?

I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.

yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...
Once everything starts up there isn't a difference, but the topmost agents will get the first splits, resulting in a faster startup.

With a lot of computers (like your setup) I can believe that that makes a big difference.

Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising :)
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool :) ...
37fps on start of second pass.. :) and rising :) ...

i think. today is a good day,.. i'm happy :) really :) ..
Good to hear! Thanks!

SpAwN_gUy

13th March 2007, 12:33

YEPPIE.... i've figured all out :) ... i just remembered, when i was trying to run agent as service, there was an advice..(at the end of the post,.. add some mystery ;) )
And just in case you were wondering:
I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.
and i just played a bit with delphi and i could writa a programm, that starts something(i dunno how will it work and what do you need from that starter).. it is now 40Kb .. i'll play a bit later and will post some results..
it is ConsoleApp and uses WinAPI (CreateProcess with Priority flag set).. and i've found in old winSDK only four priorities.. IDLE, NORMAL, HIGH and REALTIME... and i dunno wich are usefull :) ..

anyway results -> later.

--and-now--------------------------

Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. :angry: I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.
testing testing,.. AND:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"WORKS!!!!
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"Does NOT

and then after:
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432WORKS!!!!
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432
Does not

maybe.. it is the best way.. to change those slashes back? ;)

omion

14th March 2007, 00:17

@SpAwN_gUy:

So it looks like everything works when using forward slashes rather than backslashes? That's good to know. It's looking like it won't matter for much longer, though.

I was just contacted by foxius (at least, that will be his name when his 5 days are up ;) ) who is making a nice:rolleyes: alternative to the nice.exe program I am currently distributing. It will be a drop-in replacement, and should weigh in at ~50KB.

I can't say for sure, but it looks like it should be done soon (depending on how many more unreasonable requests I make :p )

omion

15th March 2007, 07:29

All right! I finally updated nice.exe!
(or rather, foxius updated it and I took credit for it ;) )

Get it here (http://omion.dyndns.org/x264farm/nice-20070314_win32.rar) (source (http://omion.dyndns.org/x264farm/nice-20070314_src.rar)). I also updated the first post's link.

Awesome new features:

If the program's extension is "exe", "bat", or "com", you no longer need to specifically write the extension. (This approximates the shell's handling of extensions)
Network paths work without nasty hacks
It's MUCH smaller than the old version, which had to have all the cygwin libraries dragged along with it.

MAJOR props to foxius! Thanks a bunch! :D

SpAwN_gUy

15th March 2007, 10:35

:)) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually :) ...
winAPI:
CreateProcess(with parameters)
okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ;) ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.

btw. feature request :) ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)

and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"

okay,.. i've done "pause" in .bat file... so..
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...

omion

16th March 2007, 07:41

:)) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually :) ...

Yeah. I had the basic idea of how to do it (involving CreateProcess), but my programming skills are basically limited to Perl and OCaml, neither of which are particularly good at hooking into the OS. And C just drove me nuts.

And about me using foxius's code rather than yours, well... he contacted me first :cool: Don't worry; I'll tell you the next time I need a program that I don't want to write :p

okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ;) ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.

That looks like something's wrong in the file itself...

btw. feature request :) ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)

Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... :rolleyes: )

That comment about finding GOPs (around line 450 in pass2.ml) was actually referring to the fact that there could be many more GOPs done than are in the temp stats file, and how it is quite difficult to determine where to restart counting when the program is re-run.

However, the only thing deleting the unneeded GOPs will get you is extra disk space. Everything is calculated based on the working_stats#.txt files. The result is that, once the reference to the old GOP is overwritten by the new GOP file name, the old GOP file might as well not exist any more. Nothing in the program references old GOP files, so they don't have any effect on anything.
This means that there is no reason not to delete them, but it won't help the program if they are deleted.
In either case, the whole temp directory is cleaned up at the end of the second pass, so the final result is the same no matter what files are in there.

and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"

Hmm... I'll have to look into it a bit more. Ratecontrol is quite a subtle thing, and it's fairly easy to do wrong.

I probably have a plus when I should have a minus or something. Due to the self-correcting nature of ratecontrol, just about every bug in it will result in something that is a bit off, but not by much.

Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.

okay,.. i've done "pause" in .bat file... so..
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)

(apologies for any horrible grammar or anything in this post. My brain is not working properly right now. Too much caffeine... need more sleep...)

SpAwN_gUy

16th March 2007, 09:11

And about me using foxius's code rather than yours, well... he contacted me first :cool: Don't worry; I'll tell you the next time I need a program that I don't want to write :p don't worry, i'm beginning to be more opened to peaple,.. but not all the time :) ... i gust fing your progam very interesting, and i'm trying to help.. thats all :)

That looks like something's wrong in the file itself...
....
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)
well,.. it shows right after writind to log file line "Merging 91 / 302"
and, i've muxed with my bare hands (mkvmerged) all those .mkv-files.. some of them were "unKnown format" (i guess one of those zero-bytes file, i have in _temp), and (i guess i forgot some of files, but.. ~godd mkv.. (it became 1 second less)) the end file is good,.. (thise 91,92,93 - were good in the end).
so i'm sending you those files.. via e-mail..

Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... :rolleyes: ) waiting :) ..

Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.
GordianKnot-calculator (aac-sound goes as vorbis in "Interleaving & AVI-Overhead") or Calculator in meGUI.. (as far as meGUI has "GUI" and Profiles,.. and controllerGUI can import meGUI Jobs... so, generating a comLine is not a problem to me.. ;) ..)

btw. minasan, how would you like to start controller? invisible and all "usefull info is displayed in GUI"(as meGUI does) or just run a pregenerated comLine, while showing DOSBox(like GKnot does)?

yeah,.. and something like a BUG,.. i think.. after "resuming" on second-pass,.. the info(that periodically is shown..) does not show "Current GOP" (column "Now") just "x"..

foxius

19th March 2007, 09:22

omion

19th March 2007, 19:09

Finally, I have a voice =)

x264farm is working like a charm here, the only bug I've found so far is that sometimes I get a DELFIRST from an agent and it doesn't resolve until I kill the agent, clean his temp folder and restart. I think, it would be more convenient if the agent could clean the broken files itself.

Well, DELFIRST is not really a problem in itself. It comes about when an agent finishes a job right before another job was being worked on, and it discovered that the first frame on the second job won't be an I frame. DELFIRST really just means "delete first GOP in the job, when it gets returned" and it will be taken care of when the agent returns the job.
It turns out that dealing with this case correctly is quite difficult, so I added it to the output so I could properly debug it.

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.

It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.

@SpAwN_gUy:
I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is. Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message.

And sorry about the delay. I thought I would get all this stuff done last week, but I got sick. And I play too much Oblivion. (Maybe I got sick from playing Oblivion...?)

foxius

19th March 2007, 20:06

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.

Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.

It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.

Perhaps it would work even without removing the temp files. I'll try not to delete them next time.

I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else.

SpAwN_gUy

20th March 2007, 09:36

Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.
let me guess.. maybe i'm having the sam sort of a problem..
it goes after al agents are ("agentname close_connection") and than controller just waits.. for something.
and all connection to source files are closed, and controller just waits..
it's not happening all the time, but some times..

this happens to me on both passes.. fist and second,.. in the end.. just before the MUXing starts :( ...

so my 5crnts: i think this loop occurs when agent ended encoding sended job, but something happened during sending of MKV, or Controlled didn't got it, or agent could not send it..

So a small solution,.. after a "timer" (~2 minutes) controller could ReCheck Working(as C thinks) Agents if they are actually working... timer could start after beginning of closing connections and disconnecting agents.. (well, theese are just my thoughts)

SpAwN_gUy

20th March 2007, 09:50

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem. i have now end of second-pass with 60% bitrate .. and it just reencodes something again and again..
2007-03-20~10:43:59.65 #1 7 < #2 8 (read #2)
and
2007-03-20~10:44:03.46 Number of GOPs done: 719 / 719
2007-03-20~10:44:03.46 Looks like the second pass was already done; using 0 for the number of GOPs computed and :) now i've found 770 copies of GOP #658 in tempDir.. so i deleted them.. and it encodes again something.. on 100%total and 60% of Quality..

... ADD(few mins later): Yeppie.. it did.. :) ... two deleted GOPs(manually) and retcontrol(i guess this is the proper naming) works again.

I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is. yes, mkvMerge did that also.. but not the farm :(
Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message. yes i do,.. i thought about it,.. that you would like to recreate it again :) ...

... testings... so i am... it is still..
in the DOSBox
#0299: "299 - 56204 56453 5654cd.mkv"
#0300: "300 - 56454 56559 148fc1.mkv"
#0301: "301 - 56560 56617 befbc3.mkv"
Fatal error: exception Invalid_argument("Char.chr")

in the out-dump.txt
2007-03-20~11:08:53.92 Merging 88 / 302
2007-03-20~11:08:54.03 Merging 89 / 302
2007-03-20~11:08:54.17 Merging 90 / 302
2007-03-20~11:08:54.29 Merging 91 / 302
2007-03-20~11:08:54.62 Merging 92 / 302

omion

20th March 2007, 19:01

[edit: Oops. It looks like SpAwN_gUy wrote a bunch of stuff after I started writing this. I'll get to those responses later]

First off, I'd like to mention that I updated my x264 builds to r634. links: 32-bit (http://omion.dyndns.org/x264/32/x264-634_omion.rar) 64-bit (http://omion.dyndns.org/x264/64/x264x64-634_omion.rar)

Also, I don't know if I mentioned this, but you only need my builds if you are using controller-based encoding or piped input on the agent. If you are using non-piped input to an agent-based encoder, you may use any x264 build you want.

Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.Does the agent show any errors? What's the CPU usage like when this happens?

One problem that I have had is that if an agent stops doing anything without informing the controller that something's wrong, then the encoding will never finish. If you restart either the controller or agent it should complete, but there's not really a good answer right now. I have been trying to think of a nice, clean way to handle this since long before I released x264farm...

Perhaps it would work even without removing the temp files. I'll try not to delete them next time.

I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else.

Well... I really don't think DELFIRST is related to the problem you're having. The DELFIRST condition is a product of the controller so doing something with the agents won't get anywhere.

Also, if it gets into a loop, the agents will still be encoding. The loops occur when the controller throws out everything that an agent does and tells the agent to encode the same thing again. As long as --batch is set to larger than x264's --keyint option, I don't think it's possible to get in a loop.

As to what is causing the problem you're having, I don't really know. Is it doing it a lot, or was it just that one time? If multiple times, was it always the same agent that was having the problem?

foxius

21st March 2007, 11:36

Does the agent show any errors? What's the CPU usage like when this happens?

I don't really remember. I think there was no error displayed on the agent computer. I'll check that next time.

One problem that I have had is that if an agent stops doing anything without informing the controller that something's wrong, then the encoding will never finish. If you restart either the controller or agent it should complete, but there's not really a good answer right now. I have been trying to think of a nice, clean way to handle this since long before I released x264farm...

I think the controller has to "ping" the agents to get their status (I think you already do that to get the work progress). If it's stuck then you send a "restart" signal. The agents do their work in worker threads, so the main thread is always responsive.

Well... I really don't think DELFIRST is related to the problem you're having. The DELFIRST condition is a product of the controller so doing something with the agents won't get anywhere.

But whenever that agent is stuck in a loop I see the DELFIRST message on the controller. I guess they're related in some way.

As to what is causing the problem you're having, I don't really know. Is it doing it a lot, or was it just that one time? If multiple times, was it always the same agent that was having the problem?

I had it at least for each movie i recoded. For the last movie (harry potter, part 1) i had several DELFIRST & dead loops. It wasn't always the same agent.

foxius

28th March 2007, 15:21

omion,

does x264farm have any kind of IPC that lets third party applications to get some feedback from it? it'd make easier to develop a GUI or even may be integrate with MeGUI.

fox

SpAwN_gUy

29th March 2007, 09:18

does x264farm have any kind of IPC that lets third party applications to get some feedback from it? it'd make easier to develop a GUI or even may be integrate with MeGUI.
i think not.. well, sources are opened..(in first post)..
as omion is not familiar with C i'm not familiar with OCaml :) .. i can read through code, but.. i can't programm(i've never tried in OCaml) and i don' have tools.. 'cause i'm short in traffic..

BTW... if something goes wrong.. i'm offline. :( ... for some time.. my boss told me that i'm not working.. and disabled iNet for me.. (eMail still works)

foxius

29th March 2007, 11:45

i think not.. well, sources are opened..(in first post)..
as omion is not familiar with C i'm not familiar with OCaml :) .. i can read through code, but.. i can't programm(i've never tried in OCaml) and i don' have tools.. 'cause i'm short in traffic..

knowledge of C programming is not required for implementing IPC (interprocess communication).

x264farm will allow monitoring of the encoding process via IPC (named pipes, shared memory or, ideally, RPC). Then anyone could write a third party software in almost any language to communicate with x264farm.

I've googled for "ocaml RPC" and found some implementations. I'm not sure which is more suitable because I'm not familiar with the language. RPC (remote procedure calls) will allow you to define a set of functions in IDL (interface description language). IDL then can be compiled into ocaml, c++, delphi, etc. x264farn will act as an RPC server and third party software will act as a client which will be able to call functions defined in IDL.

Given that x264farm already has functions for monitoring the agents, imlementing RPC will take a few simple steps:

1. Define a set of published functions (an interface) via IDL
2. Compile IDL into ocaml
3. Map the interface to the implementation (i.e. the function defined in IDL will call the function inside x264farm)
4. Distribute the IDL file so anyone can compile it to other programming languages and use it.

RPC server and clients will be able to communicate over various protocols, such as tcp/ip, ipx, named pipes, etc.

Wishbringer

5th April 2007, 20:42

Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.

SpAwN_gUy

6th April 2007, 09:17

Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.
yeah,.. me too... i've reported that.. i have sources of newNice.exe ... and i think i'll look though a bit later, i think..
and now i'm f___ing around of HDDVD-BackUp,.. and i think AgentBased encoding will not work :( ...
currently i'm short on 40mins from a movie, but everythig seems fine,..

foxius

8th April 2007, 14:23

Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.

Could you please copy/paste the windows error message you're getting? Unfortunately, I don't have Vista and can't test it myself. Once I have the error message I'll try to fix it.

SpAwN_gUy

9th April 2007, 09:10

@foxius ... my old problem shows:
Леша-К7, 16.03.2007 13:24 :
Problem signature:
Problem Event Name: BEX
Application Name: nice.exe
Application Version: 0.0.0.0
Application Timestamp: 45f8df57
Fault Module Name: nice.exe
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 45f8df57
Exception Offset: 00005183
Exception Code: c000000d
Exception Data: 00000000
OS Version: 6.0.6000.2.0.0.256.1
Locale ID: 1033
Additional Information 1: 5fd9
Additional Information 2: de9895fb4090bedd1a27a6ff8cfedad1
Additional Information 3: e9b8
Additional Information 4: 8079ce0cad7bc845eec58faba3ac0031
on Vista-Ultimate 64bit PC.
after doing this:
nice -n 20 avs2yuv -raw -seek 1250 -frames 250 -o - "\\controllerpc\x264farm-2\CHECH-Small\nazdravicko.avs" | nice -n 20 \\controllerpc\x264farm\!AGENT\x264x64.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 292 --fps 25/1 --pass 3 --stats "temp\stats d40faa.txt" -o "temp\output 83d1d8.mkv" - 688x400

Wishbringer

10th April 2007, 17:50

@foxius

Curiously it's only a window that opens, that says that nice.exe works anymore, then another one that asks if i want to send infos to Microsoft.

But no further details what infos are available. So I can't give you a hint, sorry.

out-dump.txt from agent doesn't contain useable infos too.

omion

11th April 2007, 05:01

legoman666

12th April 2007, 02:18

is there a way to set the amount of frames sent to each node for each job? I have 5 computers in my network, 1 is a 1.8ghz laptop, 1 is a 1.6ghz amd xp, 1 is a p2 350mhz, 1 is an AMD x2 2.6ghz (the controller), and the last is a 1.6ghz amd 2800+.

Now of course, the p2 is a slow piece of crap. Looking at the command line output during encoding, it looks like it sends a minimum of 5250 frames for each job. After all my clients finish their work, my p2 350 is still chugging away at its job. Is there a way to set it so that each job is smaller? like 250 frames?

I somehow bugged it out so that its doing 250 frames per job, but i have no idea what I did to make it do that. any ideas?

legoman666

12th April 2007, 02:52

ah nevermind, i see. for the first pass its doing ~5000 frames per job, but for the second pass its a much lower number.

:D silly me.

Wishbringer

12th April 2007, 21:22

nice -n 10 help
does it work?

@omion:
no crash, help page was displayed.

I don't really know why it crashes when starting x264.exe

legoman666

12th April 2007, 22:39

this little program is amazing, thanks for the hard work!

although I do have a little question, what options do i use only for the first pass and which do i use only for the second? Or do you use them all for both. I'm using megui to get the command line instructions and just copying them for use with controller.exe. At the moment, my megui command line says (for the HQ-slowest profile):

--pass 2 --bitrate 918 --stats ".stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "I:\example.mp4" "I:\example.avs"

and so for controller.exe im doing

controller.exe -B 918kbps --first "--ref 10 --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --analyse all --merange 12 --no-psnr --trellis 2 --progress " --second "--ref 10 --no-fast-pskip --bframes 3 --b-rdo --bime --b-pyramid --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --me umh --merange 12 --no-psnr --no-dct-decimate --no-ssim --no-psnr --vbv-maxrate 25000 --progress" --avs "I:\examlpe.avs" -o C:\example.mkv

is there anything im missing/doing wrong? with these setting and 8 physical cores working on the encoding, it took ~6 hours to encode a dvd thats about 1.5 hours long, which seems reasonable considering the quality setting. Something seems to be wrong with the first pass settings, because when I used megui alone to do the encoding, the first pass was the quickest (~30fps) and the second pass was ~3.5. With my current settings for controller.exe, the first pass took ~4 hours and the 2nd ~2hours.

list of computers workin on encoding (all over 100mb/s lan)
AMD x2 3800+ @2.6ghz (controller) 2 cores (local)
AMD 1800+ (catherine)
P2 350mhz (server)
AMD A64 2800+ (basement)
Centrino 1.6ghz (mom's laptop)
2xP3 500mhz Xeon (staircase)

Speeds at which they work

Agent | Now | GOPs | Frames | FPS |
Local 1 | 584 | 191 | 35515 | 9.11 |
Local 2 | 395 | 176 | 35362 | 9.61 |
basemnet 1 | 610 | 139 | 24938 | 6.18 |
server 1 | 644 | 7 | 1448 | 0.33 |
mom's laptop 1 | x | 40 | 7095 | 3.87 |
catherine 1 | x | 106 | 18011 | 4.59 |
staircase 1 | 659 | 32 | 6050 | 1.43 |
staircase 2 | 583 | 26 | 5110 | 1.13 |

Wishbringer

13th April 2007, 09:48

@legoman666

could it be that megui uses lesser quality settings for the first pass?
normally you don't need all options in first pass (like -b-rdo).
best is to look at megui's log to see, what commandline it used for first pass.

legoman666

13th April 2007, 14:47

ah thanks for the suggestion, that helps :D

controller.exe -B 8079kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --progress --no-dct-decimate --no-psnr --no-ssim" --second "--ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --progress --no-dct-decimate --no-psnr --no-ssim" --avs "I:\movie.avs" -o C:\movie.mkv

theres my next project, 720p uber quality :D. should take.... 12-14 hours for my 6 computers.

SpAwN_gUy

13th April 2007, 15:15

although I do have a little question, what options do i use only for the first pass and which do i use only for the second? Or do you use them all for both. I'm using megui to get the command line instructions and just copying them for use with controller.exe. At the moment, my megui command line says (for the HQ-slowest profile):
okay.. i'm back,.. and i don't have iNet.. :) ...
so,.. i have few suggestions for you ..
1) i've noticed that most powerfull PCs sould be first in controller's config (i have something like 10PCs Farm ;) ade powers vary alot)
2) .. well.. if you add Job in MeGUI and then close it.. you'll get 2 job.xml files .. look through them and you'll find something like <Commandline> .... copy the values from both files and you'll see the difference..
2b) if someone wants i can send a beta-controllerGUI prog (and agentGUI) ... main thing is that controllerGUI can generate commandline for the controller (but don't even try to start it for now ;) ... it'll give you a headake)..

peace to all, i'm waiting for something new ... ;)

legoman666

13th April 2007, 16:18

hmm, im getting an error with my next encoding project,

2007-04-13~10:05:50.00 x264farm version 1.09-163
2007-04-13~10:05:50.00 Using config file ".\config.xml"
2007-04-13~10:05:50.03 CONFIG:
2007-04-13~10:05:50.03 temp_dir: "I:\\hdtv\\x264farm\\temp\\controller"
2007-04-13~10:05:50.03 agent_list 1:
2007-04-13~10:05:50.03 "Local 1" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "Local 2" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "basemnet 1" = 192.168.1.13 50700-50703
2007-04-13~10:05:50.03 "server 1" = 192.168.1.4 50700-50703
2007-04-13~10:05:50.03 "mom's laptop 1" = 192.168.1.167 50700-50703
2007-04-13~10:05:50.03 "catherine 1" = 192.168.1.215 50700-50703
2007-04-13~10:05:50.03 "staircase 1" = 192.168.1.9 50700-50703
2007-04-13~10:05:50.03 "staircase 2" = 192.168.1.9 50700-50703
2007-04-13~10:05:50.03 agent_list 2:
2007-04-13~10:05:50.03 "Local 1" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "Local 2" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "basemnet 1" = 192.168.1.13 50700-50703
2007-04-13~10:05:50.03 "server 1" = 192.168.1.4 50700-50703
2007-04-13~10:05:50.03 "mom's laptop 1" = 192.168.1.167 50700-50703
2007-04-13~10:05:50.03 "catherine 1" = 192.168.1.215 50700-50703
2007-04-13~10:05:50.03 "staircase 1" = 192.168.1.9 50700-50703
2007-04-13~10:05:50.03 "staircase 2" = 192.168.1.9 50700-50703
2007-04-13~10:05:51.54
WARNING: avs2yuv exited with error 1; avs2yuv responded:

2007-04-13~10:05:51.54 GOT NORMAL INFO "\013"
2007-04-13~10:05:51.54
ERROR: AVS file 'I:\CASINO\casino.avs' does not seem to be valid

my avs file is pretty simple and worked just fine in megui when I encoded the first 200 frames just to see if it would work. heres the contents of it:

LoadPlugin("C:\PROGRA~2\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~2\GORDIA~1\AviSynthPlugins\UnDot.dll")
DirectShowSource("I:\CASINO\CASINO\BDMV\STREAM\00000.m2ts")
crop(0,144,1920,800)
LanczosResize(1280,528)
Undot()
ConvertToYV12()

my guess is the ConvertToYV12() at the bottom? megui told me to add it so I'm not really sure. any ideas?

foxius

13th April 2007, 22:53

I found that you get the best speed when using agent-based encoding. avs2yuv happens to consume too much CPU & bandwidth, but obviously useful when you can't use agent based encoding.

My setup is something like this:

1. All source files are located in network shares.
2. D2V files refer to network shares.
3. AVS files also refer to sources on network shares.
4. I also put AVS plugins on the network share
5. Thus, the agents will read the source without requiring avs2yuv to decode the source and send raw frames over your network.
6. It is easier to find(and eventually solve) problems with AVS scripts
7. I keep the sources on one HDD, write the encoded file to another.

Sample AVS script:

LoadPlugin("\\foxxp\_recoding\DGMPGDec\DGDecode.dll")
LoadPlugin("\\foxxp\_recoding\DGMPGDec\Undot.dll")

MPEG2Source("\\foxxp\_recoding\ka\ka.d2v")

# CROPPING
crop(0,0,720,576)

# RESIZING
LanczosResize(720,400)

Undot()

I also have a batch file (.bat) based on the MeGUI "Slower" profile. I just substitute

controller.exe --bitrate 1200kbps --avs "\\foxxp\_recoding\ka\ka.avs" --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --no-psnr --no-ssim" --second "--ref 5 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --no-psnr --no-ssim" -o "c:\_recoding\ka\ka.mkv"

I currently have 4 agents (including local) on a 100mbit network, 2 more over 100mbit uplink. I enable the second core on the local for the second pass (each core gives 5-7fps), and put one more job for HT processors on the network (gives 1-2 more fps per CPU). It takes 3-4 hours to encode an average movie.

I also found that the second pass never goes beyond 13-14fps, no matter how many agents/cores I add. I think it should have something to do with the bandwidth.

Hope this would be useful.

foxius

13th April 2007, 23:00

my guess is the ConvertToYV12() at the bottom? megui told me to add it so I'm not really sure. any ideas?

try removing it :) i think you only need that filter when you have some unusual (RGB?) source (i might be wrong). if it doesn't help:

1. try opening the .avs file in VirtualDub
2. try agent-based encoding

omion

14th April 2007, 04:28

WARNING: avs2yuv exited with error 1; avs2yuv responded:

2007-04-13~10:05:51.54 GOT NORMAL INFO "\013"

Try opening it up with avs2yuv. That's what x264farm uses to check the info. Run the following:
avs2yuv -frames 1 -o NUL AVSFILE.avs and see what it returns.

@all:
Again, sorry about the lack of programming. I'm pretty swamped right now :(

legoman666

14th April 2007, 05:31

apparently avs2yuv doesn't like the directshowsource? Which is weird because megui is converting it right now with no problems (at a appalingly slow 1.4fps I might add, wish i could use x264farm :P )

I:\hdtv\x264farm\controller>avs2yuv.exe -frames 1 -o NUL "I:\CASINO\casino.avs"

Avisynth error:
DirectShowSource: Could not open as video or audio.

Video returned: "DirectShowSource: couldn't create filter graph:
CoInitialize has not been called. "

Audio returned: "DirectShowSource: couldn't create filter graph:
CoInitialize has not been called. "

(I:\CASINO\casino.avs, line 18)

I:\hdtv\x264farm\controller>

its just the m2ts file from a blu ray backup (Casin Royale)

chipzoller

15th April 2007, 05:18

Might this project (sometime in the near or far future) see a GUI at some point? This being considered or worked on? Great tool, btw...

rhodges

15th April 2007, 17:00

@all who have a problem with nice vs vista:
Does it give the error message when you try to use the nice command without x264farm? Like if you type this into the terminal:
nice -n 10 help
does it work?

@all:
Sorry about not responding in a while. I got sick again, and all the students just got back to school (I work at the UCSD bookstore), and I've been doing a major overhaul of my other program (mp3packer (http://www.hydrogenaudio.org/forums/index.php?showtopic=32379)) so I haven't had much time for x264farm. I'll get to work on it as soon as I can, though.

I downloaded the source and ran the debug version and it was crashing due to a buffer size problem. The problem is the IsWinXPorLater function returns false, so max_command_size gets set to MAX_PATH which is too small.

I commented out the IsWinXPorLater check and it works like a champ on Vista 64 Ultimate.

[Edit]
I didn't try this, but I found on microsoft's site how you'd change the function:
http://msdn2.microsoft.com/en-us/library/ms724451.aspx

from:
return (version.dwMajorVersion >= 5 && version.dwMinorVersion >= 1);

to:
return ((version.dwMajorVersion > 5) || (version.dwMajorVersion == 5 && version.dwMinorVersion >= 1));

omion

15th April 2007, 18:25

[Edit]
I didn't try this, but I found on microsoft's site how you'd change the function:
http://msdn2.microsoft.com/en-us/library/ms724451.aspx

from:
return (version.dwMajorVersion >= 5 && version.dwMinorVersion >= 1);

to:
return ((version.dwMajorVersion > 5) || (version.dwMajorVersion == 5 && version.dwMinorVersion >= 1));

Ah. That would be a problem. Vista is version 6.0, so dwMinorVersion < 1...

I put a new nice build up. Download it here (http://omion.dyndns.org/x264farm/nice-20070415.rar) (src (http://omion.dyndns.org/x264farm/nice-20070415_src.rar))

harissa

15th April 2007, 22:55

I spent 24 h trying to make this work and I couldn't
Here s my setup

my avs
loadplugin("DGDecode.dll")
Mpeg2Source("carnavale.d2v")
I dont't know where to put nice.exe , avs2yuv.exe and x264-634_omion.exe
both controller and agent scrollin like hell , and showing this

E:\x264farm\temp\controller
2007-04-15~16:30:46.00 x264farm version 1.09-163
2007-04-15~16:30:46.00 Oops! File " G:\9\ka.mkv" does not exist
2007-04-15~16:30:46.00 WARNING: Second pass options are not set; this will use the default x264 settings
2007-04-15~16:30:46.00 Using config file ".\config.xml"
2007-04-15~16:30:46.00 CONFIG:
2007-04-15~16:30:46.00 temp_dir: "E:\\x264farm\\temp\\controller"
2007-04-15~16:30:46.00 agent_list 1:
2007-04-15~16:30:46.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.00 "Linux 1" = 169.254.23.92 50700-50703
2007-04-15~16:30:46.00 "Apple 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.00 "Apple 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 agent_list 2:
2007-04-15~16:30:46.01 "Local 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Local 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Linux 1" = 169.254.23.92 50700-50703
2007-04-15~16:30:46.01 "Linux 2" = 169.254.23.92 50700-50703
2007-04-15~16:30:46.01 "Apple 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Apple 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Apple 3" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.15 GOT NORMAL INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-15~16:30:46.26 GOT FIRST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-15~16:30:46.37 GOT FAST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-15~16:30:46.39 INFO:
2007-04-15~16:30:46.39 i_res_x = 1280
2007-04-15~16:30:46.39 i_res_y = 720
2007-04-15~16:30:46.39 i_fps_n = 60000
2007-04-15~16:30:46.39 i_fps_d = 1001
2007-04-15~16:30:46.39 i_fps_f = 59.940060
2007-04-15~16:30:46.39 i_num_frames = 23247
2007-04-15~16:30:46.39 i_bytes_y = 921600
2007-04-15~16:30:46.39 i_bytes_uv = 230400
2007-04-15~16:30:46.39 i_bytes_per_frame = 1382400
2007-04-15~16:30:46.39 multiplier for kbps to fbits = 0.059940
2007-04-15~16:30:46.39 fast_res_x = 1280
2007-04-15~16:30:46.39 fast_res_y = 720
2007-04-15~16:30:46.39 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\option_log.txt.temp: No such file or directory")2007-04-15~16:30:46.39 AVS temp dir is E:\x264farm\temp\controller\132FA1D98D68B80231C0CECCD0F61FAF x264.avs
2007-04-15~16:30:46.40 First string is "-2\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n3hœ)Z\013ûø‚¾\127Üºx\027\020 = G:\\dvb\\x264.avs\n", with MD5 "F51A9C869FA15B80F7D0F2202ABBDF98"
2007-04-15~16:30:46.60 Second string is "\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n", with MD5 "A25B8F0CCB96F56266CFA33B07D0F828"
2007-04-15~16:30:46.81 not (not (is_something first_dir) && is_file first_stats) [first pass after making dir]
2007-04-15~16:30:46.81 not (is_dir first_dir)
2007-04-15~16:30:46.81 11111111 FIRST PASS 11111111
2007-04-15~16:30:46.81 From file:
2007-04-15~16:30:46.81 ( 0,23246) Empty
2007-04-15~16:30:46.81 Local 1 starting on 169.254.25.121 (id 1)
2007-04-15~16:30:46.81 Local 2 starting on 169.254.25.121 (id 2)
2007-04-15~16:30:46.81 Linux 1 starting on 169.254.23.92 (id 3)
2007-04-15~16:30:46.81 Apple 1 starting on 169.254.25.121 (id 4)
2007-04-15~16:30:46.81 Apple 2 starting on 169.254.25.121 (id 5)
2007-04-15~16:30:46.81 Splitter starting
2007-04-15~16:30:46.82 Split points at 5,3,1,1
2007-04-15~16:30:49.87 queue_beginning found the beginning; split to Q(0,5062) :: (5063,23246)
2007-04-15~16:30:49.87 Ranges at beginning:
2007-04-15~16:30:49.87 ( 0, 5062) Queued
2007-04-15~16:30:49.87 ( 5063,23246) Empty
2007-04-15~16:30:49.87 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:30:49.87 Splitter: fewer queues than max_queues_split
2007-04-15~16:30:49.87 split_largest_range found the biggest empty range to be 18184 frames long
2007-04-15~16:30:53.29 split_largest_range found start I frame 11460 in (11405,11655)
2007-04-15~16:30:56.14 split_largest_range split to (5063,11460) :: (11461,16617) :: (16618,23246)
2007-04-15~16:30:56.14 Splitter: split_largest_range did something; re-run
2007-04-15~16:30:56.14 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:30:56.14 Splitter: fewer queues than max_queues_split
2007-04-15~16:30:56.14 split_largest_range found the biggest empty range to be 6629 frames long
2007-04-15~16:30:58.62 split_largest_range found start I frame 19947 in (19807,20057)
2007-04-15~16:30:58.62 Splitter: split_largest_range did something; re-run
2007-04-15~16:30:58.62 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:30:58.62 Splitter: fewer queues than max_queues_split
2007-04-15~16:30:58.62 split_largest_range found the biggest empty range to be 6398 frames long
2007-04-15~16:31:02.35 split_largest_range found start I frame 8307 in (8137,8387)
2007-04-15~16:31:02.35 Splitter: split_largest_range did something; re-run
2007-04-15~16:31:02.35 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:31:02.35 Splitter: fewer queues than max_queues_split
2007-04-15~16:31:02.35 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-15~16:31:02.35 but that's not long enough! (5250)
2007-04-15~16:31:02.35 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-15~16:31:02.35 ( 0, 5062) Queued
2007-04-15~16:31:02.35 ( 5063, 8307) Empty
2007-04-15~16:31:02.35 ( 8308,11460) Queued
2007-04-15~16:31:02.35 (11461,16617) Queued
2007-04-15~16:31:02.35 (16618,19947) Empty
2007-04-15~16:31:02.35 (19948,23246) Queued
2007-04-15~16:31:54.68 using compression type 0 (no compression)
2007-04-15~16:31:54.68 Local 2 found no ranges after full
2007-04-15~16:31:54.68 Local 2 check_out_job got frames 0 - 5062
2007-04-15~16:31:54.68 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:31:54.68 using compression type 0 (no compression)
2007-04-15~16:31:54.68 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-15~16:31:54.68 Splitter: fewer queues than max_queues_split
2007-04-15~16:31:54.68 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-15~16:31:54.68 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-15~16:31:54.68 Local 2 sent zone string ""
2007-04-15~16:31:54.68 but that's not long enough! (5250)
2007-04-15~16:31:54.68 Local 2 sent range (0,5062)
2007-04-15~16:31:54.68 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-15~16:31:54.68 ( 0, 5062) Working (Local 2)
2007-04-15~16:31:54.68 ( 5063, 8307) Empty
2007-04-15~16:31:54.68 ( 8308,11460) Queued
2007-04-15~16:31:54.68 (11461,16617) Queued
2007-04-15~16:31:54.68 (16618,19947) Empty
2007-04-15~16:31:54.68 (19948,23246) Queued
2007-04-15~16:31:54.68 Local 2 agent-based encoding; waiting for agent to finish
2007-04-15~16:31:54.68 Apple 1 found no ranges after full
2007-04-15~16:31:54.68 Apple 1 check_out_job got frames 8308 - 11460
2007-04-15~16:31:54.68 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:31:54.70 Apple 1 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-15~16:31:54.70 Splitter: fewer queues than max_queues_split
2007-04-15~16:31:54.70 Apple 1 sent video info 1280x720 @ 60000/1001
2007-04-15~16:31:54.70 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-15~16:31:54.70 Apple 1 sent zone string ""
2007-04-15~16:31:54.70 but that's not long enough! (5250)
2007-04-15~16:31:54.70 Apple 1 sent range (8307,11460)
2007-04-15~16:31:54.70 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-15~16:31:54.70 ( 0, 5062) Working (Local 2)
2007-04-15~16:31:54.70 ( 5063, 8307) Empty
2007-04-15~16:31:54.70 ( 8308,11460) Working (Apple 1)
2007-04-15~16:31:54.70 (11461,16617) Queued
2007-04-15~16:31:54.70 (16618,19947) Empty
2007-04-15~16:31:54.70 (19948,23246) Queued
2007-04-15~16:31:54.70 Apple 1 agent-based encoding; waiting for agent to finish
2007-04-15~16:31:54.73 Local 2 process_job failed with ("The operation completed successfully.\013\n",recv,); putting job (0,5062) back
2007-04-15~16:31:54.73 Apple 1 process_job failed with "End_of_file"; putting job (8308,11460) back
2007-04-15~16:31:54.73 put back range
2007-04-15~16:31:54.73 ( 0, 5062) Queued
2007-04-15~16:31:54.73 ( 5063, 8307) Empty
2007-04-15~16:31:54.73 ( 8308,11460) Working (Apple 1)
2007-04-15~16:31:54.73 (11461,16617) Queued
2007-04-15~16:31:54.73 (16618,19947) Empty
2007-04-15~16:31:54.73 (19948,23246) Queued
2007-04-15~16:31:54.73 put back range
2007-04-15~16:31:54.73 ( 0, 5062) Queued
2007-04-15~16:31:54.73 ( 5063, 8307) Empty
2007-04-15~16:31:54.73 ( 8308,11460) Queued
2007-04-15~16:31:54.73 (11461,16617) Queued
2007-04-15~16:31:54.75 (16618,19947) Empty
2007-04-15~16:31:54.75 (19948,23246) Queued
2007-04-15~16:31:54.73 Local 2 close connection
2007-04-15~16:31:54.75 Apple 1 close connection

E:\x264farm\temp\agent
2007-04-15~16:31:54.00 x264farm version 1.09-163
2007-04-15~16:31:54.00 Using config file ".\config.xml"
2007-04-15~16:31:54.00 Temp dir: "E:\x264farm\temp\agent"
2007-04-15~16:31:54.00 Ports: 50700 - 50703
2007-04-15~16:31:54.00 x264: "x264.exe"
2007-04-15~16:31:54.00 nice: 10
2007-04-15~16:31:54.00 agents: 1
2007-04-15~16:31:54.00 agent bases:
2007-04-15~16:31:54.00 [exact]
2007-04-15~16:31:54.00 pipe: false
2007-04-15~16:31:54.00 compressions: 0001
2007-04-15~16:31:54.00 deleting old files: after 604800 seconds
2007-04-15~16:31:54.00 trying port 50700
2007-04-15~16:31:54.00 STALE ignoring file "agent.exe"
2007-04-15~16:31:54.01 bound to port 50700
2007-04-15~16:31:54.01 STALE ignoring file "config.xml"
2007-04-15~16:31:54.01 listening...
2007-04-15~16:31:54.01 STALE not time for "out-dump.txt"
2007-04-15~16:31:54.01 working!
2007-04-15~16:31:54.01 STALE ignoring file "x264.exe"
2007-04-15~16:31:54.07 Got a connection from 169.254.25.121:4519
2007-04-15~16:31:54.07 working!
2007-04-15~16:31:54.07 Got a connection from 169.254.25.121:4522
2007-04-15~16:31:54.07 working!
2007-04-15~16:31:54.07 2 first pass frames 0 to 5062
2007-04-15~16:31:54.07 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.07 2 agent-based encoding
2007-04-15~16:31:54.07 2 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats b4efcd.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.09 3 first pass frames 8307 to 11460
2007-04-15~16:31:54.09 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.09 3 agent-based encoding
2007-04-15~16:31:54.09 3 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats a79e8d.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.12 2 exited 1
2007-04-15~16:31:54.12 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.12 3 exited 1
2007-04-15~16:31:54.12 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.15 Got a connection from 169.254.25.121:4527
2007-04-15~16:31:54.17 working!
2007-04-15~16:31:54.17 4 first pass frames 0 to 5062
2007-04-15~16:31:54.17 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.17 4 agent-based encoding
2007-04-15~16:31:54.17 4 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 71045f.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.20 4 exited 1
2007-04-15~16:31:54.20 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.20 Got a connection from 169.254.25.121:4529
2007-04-15~16:31:54.20 working!
2007-04-15~16:31:54.21 5 first pass frames 0 to 5062
2007-04-15~16:31:54.21 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.21 5 agent-based encoding
2007-04-15~16:31:54.21 5 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 1a9ec3.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.25 5 exited 1
2007-04-15~16:31:54.25 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.25 Got a connection from 169.254.25.121:4530
2007-04-15~16:31:54.25 working

omion

16th April 2007, 01:03

E:\x264farm\temp\controller
2007-04-15~16:30:46.00 x264farm version 1.09-163
2007-04-15~16:30:46.00 Oops! File " G:\9\ka.mkv" does not exist

This looks a bit fishy. What was the command line that you ran the controller with?

2007-04-15~16:30:46.39 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\option_log.txt.temp: No such file or directory")This is sort of odd, too, but I don't think it matters.

E:\x264farm\temp\agent
2007-04-15~16:31:54.17 4 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 71045f.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.20 4 exited 1
2007-04-15~16:31:54.20 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"

This probably means that the agent can't find x264. If you just type "x264.exe" in the agent directory, what does it say?

If it can't find x264, you need to make sure that it's on the path (in the agent's dir is a good place) and set the <x264> line in the agent's config.xml file to the right executable (that is, if you are using "x264-634_omion.exe", make sure config.xml has the line <x264>x264-634_omion.exe</x264> in it)

harissa

16th April 2007, 03:19

foxius

16th April 2007, 07:57

I run controller
controller.exe --bitrate 3000kbps --avs "G:\dvb\x264.avs" --first --second -o "G:\ka.mkv"
after that on the same pc , even on another pc , i run agent.exe , and it go mad scrollin

i ve tried to put the x264.exe in both dir , dosen t change anything ,

is there any other setting except the config.xlm
I dont't know where to put nice.exe , avs2yuv.exe and x264-634_omion.exe
cauz I can t find how to make this run

1. nice.exe goes to agent's folder
2. avs2yuv.exe goes to controller's folder
3. x264-634_omion.exe should be renamed to x264.exe and put into agent's folder (unless you've changed x264 path in agent's config.xml)

Wishbringer

16th April 2007, 10:27

@omion:
is your x264 build a plain compile from source with MS Visual Studio,
or did you some changes in source?
I just tried to compile build 650 with Studio 2005, no errors, but i can't get controller based encode to run.

harissa

16th April 2007, 14:45

i fixed evrything still not workin i m postin all info this time
E:\x264farm\temp\controller
has avs2yuv.exe controller.exe config.xml
<config>
<temp>E:\x264farm\temp\controller</temp> 
<agents>
<agent name="Local"> 
<ip>169.254.25.121</ip> 
<port from="50700" to="50703"/> 
<number>2</number> 
</agent>
<agent name="Linux">
<ip>169.254.23.92</ip>
<port from="50700" to="50703"/>
<number pad="1">1</number>

</agent>
<agent name="Apple">
<ip>169.254.25.121</ip>
<port from="50700" to="50703"/>
<number pad="1">2</number>
</agent>

</agents>

</config>

here the log from the controller
controller.exe --bitrate 3000kbps --avs "G:\dvb\x264.avs" --first --second -o "G:\ka.mkv"

after this point it start scrollin non stop once the agent open
2007-04-16~08:33:18.00 x264farm version 1.09-163
2007-04-16~08:33:18.00 WARNING: Second pass options are not set; this will use the default x264 settings
2007-04-16~08:33:18.00 Using config file ".\config.xml"
2007-04-16~08:33:18.00 CONFIG:
2007-04-16~08:33:18.00 temp_dir: "E:\\x264farm\\temp\\controller"
2007-04-16~08:33:18.00 agent_list 1:
2007-04-16~08:33:18.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.00 "Linux 1" = 169.254.23.92 50700-50703
2007-04-16~08:33:18.00 "Apple 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.00 "Apple 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 agent_list 2:
2007-04-16~08:33:18.01 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Linux 1" = 169.254.23.92 50700-50703
2007-04-16~08:33:18.01 "Linux 2" = 169.254.23.92 50700-50703
2007-04-16~08:33:18.01 "Apple 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Apple 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Apple 3" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.39 GOT NORMAL INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~08:33:18.50 GOT FIRST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~08:33:18.62 GOT FAST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~08:33:18.62 INFO:
2007-04-16~08:33:18.62 i_res_x = 1280
2007-04-16~08:33:18.62 i_res_y = 720
2007-04-16~08:33:18.62 i_fps_n = 60000
2007-04-16~08:33:18.62 i_fps_d = 1001
2007-04-16~08:33:18.62 i_fps_f = 59.940060
2007-04-16~08:33:18.62 i_num_frames = 23247
2007-04-16~08:33:18.62 i_bytes_y = 921600
2007-04-16~08:33:18.62 i_bytes_uv = 230400
2007-04-16~08:33:18.62 i_bytes_per_frame = 1382400
2007-04-16~08:33:18.62 multiplier for kbps to fbits = 0.059940
2007-04-16~08:33:18.62 fast_res_x = 1280
2007-04-16~08:33:18.62 fast_res_y = 720
2007-04-16~08:33:18.62 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\option_log.txt.temp: No such file or directory")2007-04-16~08:33:18.64 AVS temp dir is E:\x264farm\temp\controller\132FA1D98D68B80231C0CECCD0F61FAF x264.avs
2007-04-16~08:33:18.64 First string is "--second\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n3hœ)Z\013ûø‚¾\127Üºx\027\020 = G:\\dvb\\x264.avs\n", with MD5 "83936EFFC345B4F1C55FA600066A3830"
2007-04-16~08:33:18.84 Second string is "\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n", with MD5 "A25B8F0CCB96F56266CFA33B07D0F828"
2007-04-16~08:33:19.04 not (not (is_something first_dir) && is_file first_stats) [first pass after making dir]
2007-04-16~08:33:19.04 11111111 FIRST PASS 11111111
2007-04-16~08:33:19.04 #1 3 > #2 2 (read #1)
2007-04-16~08:33:19.04 From file:
2007-04-16~08:33:19.04 ( 0,23246) Empty
2007-04-16~08:33:19.04 Local 1 starting on 169.254.25.121 (id 1)
2007-04-16~08:33:19.04 Local 2 starting on 169.254.25.121 (id 2)
2007-04-16~08:33:19.04 Apple 1 starting on 169.254.25.121 (id 4)
2007-04-16~08:33:19.04 Linux 1 starting on 169.254.23.92 (id 3)
2007-04-16~08:33:19.04 Apple 2 starting on 169.254.25.121 (id 5)
2007-04-16~08:33:19.04 Splitter starting
2007-04-16~08:33:19.06 Split points at 5,3,1,1
2007-04-16~08:33:21.73 queue_beginning found the beginning; split to Q(0,5062) :: (5063,23246)
2007-04-16~08:33:21.73 Ranges at beginning:
2007-04-16~08:33:21.75 ( 0, 5062) Queued
2007-04-16~08:33:21.75 ( 5063,23246) Empty
2007-04-16~08:33:21.75 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:21.75 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:21.75 split_largest_range found the biggest empty range to be 18184 frames long
2007-04-16~08:33:24.60 split_largest_range found start I frame 11460 in (11405,11655)
2007-04-16~08:33:27.07 split_largest_range split to (5063,11460) :: (11461,16617) :: (16618,23246)
2007-04-16~08:33:27.09 Splitter: split_largest_range did something; re-run
2007-04-16~08:33:27.09 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:27.09 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:27.09 split_largest_range found the biggest empty range to be 6629 frames long
2007-04-16~08:33:29.15 split_largest_range found start I frame 19947 in (19807,20057)
2007-04-16~08:33:29.15 Splitter: split_largest_range did something; re-run
2007-04-16~08:33:29.15 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:29.15 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:29.15 split_largest_range found the biggest empty range to be 6398 frames long
2007-04-16~08:33:32.35 split_largest_range found start I frame 8307 in (8137,8387)
2007-04-16~08:33:32.35 Splitter: split_largest_range did something; re-run
2007-04-16~08:33:32.35 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:32.35 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:32.35 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~08:33:32.35 but that's not long enough! (5250)
2007-04-16~08:33:32.35 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~08:33:32.35 ( 0, 5062) Queued
2007-04-16~08:33:32.35 ( 5063, 8307) Empty
2007-04-16~08:33:32.35 ( 8308,11460) Queued
2007-04-16~08:33:32.35 (11461,16617) Queued
2007-04-16~08:33:32.35 (16618,19947) Empty
2007-04-16~08:33:32.35 (19948,23246) Queued
2007-04-16~08:33:55.40 using compression type 0 (no compression)
2007-04-16~08:33:55.40 Local 2 found no ranges after full
2007-04-16~08:33:55.40 Local 2 check_out_job got frames 0 - 5062
2007-04-16~08:33:55.40 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:55.40 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~08:33:55.40 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:55.40 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-16~08:33:55.40 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~08:33:55.40 Local 2 sent zone string ""
2007-04-16~08:33:55.40 but that's not long enough! (5250)
2007-04-16~08:33:55.40 Local 2 sent range (0,5062)
2007-04-16~08:33:55.40 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~08:33:55.40 ( 0, 5062) Working (Local 2)
2007-04-16~08:33:55.40 ( 5063, 8307) Empty
2007-04-16~08:33:55.40 ( 8308,11460) Queued
2007-04-16~08:33:55.40 (11461,16617) Queued
2007-04-16~08:33:55.42 (16618,19947) Empty
2007-04-16~08:33:55.42 (19948,23246) Queued
2007-04-16~08:33:55.40 Local 2 agent-based encoding; waiting for agent to finish
2007-04-16~08:33:55.60 using compression type 0 (no compression)
2007-04-16~08:33:55.60 Apple 2 found no ranges after full
2007-04-16~08:33:55.60 Apple 2 check_out_job got frames 8308 - 11460
2007-04-16~08:33:55.60 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:55.60 Apple 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~08:33:55.60 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:55.60 Apple 2 sent video info 1280x720 @ 60000/1001
2007-04-16~08:33:55.60 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~08:33:55.60 Apple 2 sent zone string ""
2007-04-16~08:33:55.60 but that's not long enough! (5250)
2007-04-16~08:33:55.60 Apple 2 sent range (8307,11460)
2007-04-16~08:33:55.60 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~08:33:55.60 ( 0, 5062) Working (Local 2)
2007-04-16~08:33:55.60 ( 5063, 8307) Empty
2007-04-16~08:33:55.60 ( 8308,11460) Working (Apple 2)
2007-04-16~08:33:55.60 (11461,16617) Queued
2007-04-16~08:33:55.60 (16618,19947) Empty
2007-04-16~08:33:55.60 (19948,23246) Queued

harissa

16th April 2007, 14:46

E:\x264farm\temp\agent , on the same computer
agent.exe x264.exe nice.exe config.xml
<config>
<temp>E:\x264farm\temp\agent</temp>

2007-04-16~08:33:49.00 x264farm version 1.09-163
2007-04-16~08:33:49.00 Using config file ".\config.xml"
2007-04-16~08:33:49.00 Temp dir: "E:\x264farm\temp\agent"
2007-04-16~08:33:49.00 Ports: 50700 - 50703
2007-04-16~08:33:49.00 x264: "x264.exe"
2007-04-16~08:33:49.00 nice: 10
2007-04-16~08:33:49.00 agents: 1
2007-04-16~08:33:49.00 agent bases:
2007-04-16~08:33:49.00 [exact]
2007-04-16~08:33:49.00 pipe: false
2007-04-16~08:33:49.00 compressions: 0001
2007-04-16~08:33:49.00 deleting old files: after 604800 seconds
2007-04-16~08:33:49.00 trying port 50700
2007-04-16~08:33:49.00 STALE ignoring file "agent.exe"
2007-04-16~08:33:49.01 bound to port 50700
2007-04-16~08:33:49.01 STALE ignoring file "config.xml"
2007-04-16~08:33:49.01 listening...
2007-04-16~08:33:49.01 STALE ignoring file "nice.exe"
2007-04-16~08:33:49.01 working!
2007-04-16~08:33:49.01 STALE not time for "out-dump.txt"
2007-04-16~08:33:49.01 STALE ignoring file "x264.exe"
2007-04-16~08:33:55.57 Got a connection from 169.254.25.121:1624
2007-04-16~08:33:55.57 working!
2007-04-16~08:33:55.57 2 first pass frames 0 to 5062
2007-04-16~08:33:55.57 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:55.57 2 agent-based encoding
2007-04-16~08:33:55.57 2 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 45b9ad.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:55.78 Got a connection from 169.254.25.121:1627
2007-04-16~08:33:55.78 working!
2007-04-16~08:33:55.78 3 first pass frames 8307 to 11460
2007-04-16~08:33:55.78 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:55.78 3 agent-based encoding
2007-04-16~08:33:55.78 3 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 5d3a0f.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.00 Got a connection from 169.254.25.121:1625
2007-04-16~08:33:56.00 working!
2007-04-16~08:33:56.01 4 first pass frames 19947 to 23246
2007-04-16~08:33:56.01 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.01 4 agent-based encoding
2007-04-16~08:33:56.01 4 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 9f4a62.txt" --seek 19947 --frames 3300 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.17 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 45b9ad.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.18 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 5d3a0f.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.18 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 9f4a62.txt\\\" --seek 19947 --frames 3300 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.20 Got a connection from 169.254.25.121:1628
2007-04-16~08:33:56.20 working!
2007-04-16~08:33:56.21 Got a connection from 169.254.25.121:1629
2007-04-16~08:33:56.21 working!
2007-04-16~08:33:56.21 Got a connection from 169.254.25.121:1630
2007-04-16~08:33:56.21 working!
2007-04-16~08:33:56.23 5 first pass frames 0 to 5062
2007-04-16~08:33:56.23 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.23 5 agent-based encoding
2007-04-16~08:33:56.23 5 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 5484d5.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.25 7 first pass frames 19947 to 23246
2007-04-16~08:33:56.25 6 first pass frames 8307 to 11460
2007-04-16~08:33:56.25 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.25 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.25 7 agent-based encoding
2007-04-16~08:33:56.25 6 agent-based encoding
2007-04-16~08:33:56.25 7 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats b883ee.txt" --seek 19947 --frames 3300 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.25 6 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats e89baf.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.29 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 5484d5.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.31 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats b883ee.txt\\\" --seek 19947 --frames 3300 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.32 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats e89baf.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.34 Got a connection from 169.254.25.121:1631
2007-04-16~08:33:56.34 working!
2007-04-16~08:33:56.34 Got a connection from 169.254.25.121:1632
2007-04-16~08:33:56.34 working!
2007-04-16~08:33:56.34 Got a connection from 169.254.25.121:1633
2007-04-16~08:33:56.34 working!
2007-04-16~08:33:56.35 8 first pass frames 0 to 5062
2007-04-16~08:33:56.35 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.37 8 agent-based encoding
2007-04-16~08:33:56.37 8 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats b989cd.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.39 9 first pass frames 8307 to 11460
2007-04-16~08:33:56.39 10 first pass frames 19947 to 23246
2007-04-16~08:33:56.39 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.39 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.39 9 agent-based encoding
2007-04-16~08:33:56.39 10 agent-based encoding
2007-04-16~08:33:56.39 9 doing the following:

foxius

16th April 2007, 15:01

foxius

16th April 2007, 15:09

harissa, please ensure the following folders exist prior to running the controller and agent:

E:\x264farm\temp\controller
E:\x264farm\temp\agent

it seems to me that the controller is not able to write to the temp folder.

here's my x264 folder tree for reference.

x264farm
│ gpl.txt
│ x264farm.html
│
├───agent-win32
│ │ agent.exe
│ │ config.xml
│ │ gpl.txt
│ │ nice.exe
│ │ out-dump.txt
│ │ x264.exe
│ │
│ └───temp
└───controller
│ avs2yuv.exe
│ config.xml
│ controller.exe
│ enc.bat
│ out-dump.txt
│
└───temp

and here's my agent config:

<config>
<temp>.\temp</temp>

<port from="50700" to="50703"/>

<x264>x264.exe</x264>

<nice>20</nice>

<base>d:\</base>

<agentpipe>0</agentpipe>

<compression>
<type>0</type>
<type>1</type>
</compression>

</config>

and here's controller config:

<config>
<temp>.\temp</temp> 

<agents>
<agent name="Local">
<ip>127.0.0.1</ip>
<port from="50700" to="50703"/>
<number>1</number>
</agent>

<agent name="agent1">
<ip>192.168.13.1</ip>
<port from="50700" to="50703"/>
<number>1</number>
</agent>

<agent name="agent2">
<ip>192.168.13.15</ip>
<port from="50700" to="50703"/>
<number>1</number>
</agent>
</agents>
</config>

as you can see, i keep temp folders inside agent's and controller's folder and use a relative path in the config. this allows me to copy them to any path without modifying the config at all.

harissa

16th April 2007, 15:13

i ve tried that
i have E:\x264farm\temp
4 dir inside
controller and agent are temp dir ,
and controller2 dir, contain all exe
and agent2 dir contain all exe

harissa

16th April 2007, 15:25

I ve done the same setting as u do , i have some improvement less error msg , but both windows are scrollin like hell , can i post again the log for 2 agent so u can take a look

foxius

16th April 2007, 15:40

I ve done the same setting as u do , i have some improvement less error msg , but both windows are scrollin like hell , can i post again the log for 2 agent so u can take a look

sure. do you get the errors only in agents' windows now? controller is ok?

harissa

16th April 2007, 16:08

same dir as you same commande line
2007-04-16~10:09:46.00 x264farm version 1.09-163
2007-04-16~10:09:46.00 WARNING: Second pass options are not set; this will use the default x264 settings
2007-04-16~10:09:46.00 Using config file ".\config.xml"
2007-04-16~10:09:46.00 CONFIG:
2007-04-16~10:09:46.00 temp_dir: "E:\\x264farm\\temp\\controller\\temp"
2007-04-16~10:09:46.00 agent_list 1:
2007-04-16~10:09:46.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "pc2 1" = 169.254.23.92 50700-50703
2007-04-16~10:09:46.00 "pc3 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "pc3 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 agent_list 2:
2007-04-16~10:09:46.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.01 "pc2 1" = 169.254.23.92 50700-50703
2007-04-16~10:09:46.01 "pc2 2" = 169.254.23.92 50700-50703
2007-04-16~10:09:46.01 "pc3 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.01 "pc3 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.01 "pc3 3" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.12 GOT NORMAL INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~10:09:46.23 GOT FIRST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~10:09:46.32 GOT FAST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~10:09:46.34 INFO:
2007-04-16~10:09:46.34 i_res_x = 1280
2007-04-16~10:09:46.34 i_res_y = 720
2007-04-16~10:09:46.34 i_fps_n = 60000
2007-04-16~10:09:46.34 i_fps_d = 1001
2007-04-16~10:09:46.34 i_fps_f = 59.940060
2007-04-16~10:09:46.34 i_num_frames = 23247
2007-04-16~10:09:46.34 i_bytes_y = 921600
2007-04-16~10:09:46.34 i_bytes_uv = 230400
2007-04-16~10:09:46.34 i_bytes_per_frame = 1382400
2007-04-16~10:09:46.34 multiplier for kbps to fbits = 0.059940
2007-04-16~10:09:46.34 fast_res_x = 1280
2007-04-16~10:09:46.34 fast_res_y = 720
2007-04-16~10:09:46.34 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\temp\\option_log.txt.temp: No such file or directory")2007-04-16~10:09:46.34 AVS temp dir is E:\x264farm\temp\controller\temp\132FA1D98D68B80231C0CECCD0F61FAF x264.avs
2007-04-16~10:09:46.34 First string is "--second\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n3hœ)Z\013ûø‚¾\127Üºx\027\020 = G:\\dvb\\x264.avs\n", with MD5 "83936EFFC345B4F1C55FA600066A3830"
2007-04-16~10:09:46.54 Second string is "\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n", with MD5 "A25B8F0CCB96F56266CFA33B07D0F828"
2007-04-16~10:09:46.75 not (not (is_something first_dir) && is_file first_stats) [first pass after making dir]
2007-04-16~10:09:46.76 11111111 FIRST PASS 11111111
2007-04-16~10:09:46.76 #1 1 < #2 2 (read #2)
2007-04-16~10:09:46.76 From file:
2007-04-16~10:09:46.76 ( 0,23246) Empty
2007-04-16~10:09:46.76 Local 1 starting on 169.254.25.121 (id 1)
2007-04-16~10:09:46.76 Local 2 starting on 169.254.25.121 (id 2)
2007-04-16~10:09:46.76 pc2 1 starting on 169.254.23.92 (id 3)
2007-04-16~10:09:46.76 pc3 1 starting on 169.254.25.121 (id 4)
2007-04-16~10:09:46.76 pc3 2 starting on 169.254.25.121 (id 5)
2007-04-16~10:09:46.76 Splitter starting
2007-04-16~10:09:46.76 Split points at 5,3,1,1
2007-04-16~10:09:49.90 queue_beginning found the beginning; split to Q(0,5062) :: (5063,23246)
2007-04-16~10:09:49.90 Ranges at beginning:
2007-04-16~10:09:49.90 ( 0, 5062) Queued
2007-04-16~10:09:49.90 ( 5063,23246) Empty
2007-04-16~10:09:49.90 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:09:49.90 Splitter: fewer queues than max_queues_split
2007-04-16~10:09:49.90 split_largest_range found the biggest empty range to be 18184 frames long
2007-04-16~10:09:53.43 split_largest_range found start I frame 11460 in (11405,11655)
2007-04-16~10:09:56.29 split_largest_range split to (5063,11460) :: (11461,16617) :: (16618,23246)
2007-04-16~10:09:56.29 Splitter: split_largest_range did something; re-run
2007-04-16~10:09:56.29 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:09:56.29 Splitter: fewer queues than max_queues_split
2007-04-16~10:09:56.29 split_largest_range found the biggest empty range to be 6629 frames long
2007-04-16~10:09:58.78 split_largest_range found start I frame 19947 in (19807,20057)
2007-04-16~10:09:58.78 Splitter: split_largest_range did something; re-run
2007-04-16~10:09:58.78 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:09:58.78 Splitter: fewer queues than max_queues_split
2007-04-16~10:09:58.78 split_largest_range found the biggest empty range to be 6398 frames long
2007-04-16~10:10:02.56 split_largest_range found start I frame 8307 in (8137,8387)
2007-04-16~10:10:02.57 Splitter: split_largest_range did something; re-run
2007-04-16~10:10:02.57 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:10:02.57 Splitter: fewer queues than max_queues_split
2007-04-16~10:10:02.57 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~10:10:02.57 but that's not long enough! (5250)
2007-04-16~10:10:02.57 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~10:10:02.57 ( 0, 5062) Queued
2007-04-16~10:10:02.57 ( 5063, 8307) Empty
2007-04-16~10:10:02.57 ( 8308,11460) Queued
2007-04-16~10:10:02.57 (11461,16617) Queued
2007-04-16~10:10:02.57 (16618,19947) Empty
2007-04-16~10:10:02.57 (19948,23246) Queued
2007-04-16~10:10:07.57 using compression type 0 (no compression)
2007-04-16~10:10:07.57 Local 2 found no ranges after full
2007-04-16~10:10:07.57 Local 2 check_out_job got frames 0 - 5062
2007-04-16~10:10:07.57 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:10:07.57 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~10:10:07.57 Splitter: fewer queues than max_queues_split
2007-04-16~10:10:07.57 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-16~10:10:07.57 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~10:10:07.57 Local 2 sent zone string ""
2007-04-16~10:10:07.57 but that's not long enough! (5250)
2007-04-16~10:10:07.57 Local 2 sent range (0,5062)
2007-04-16~10:10:07.57 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~10:10:07.57 ( 0, 5062) Working (Local 2)
2007-04-16~10:10:07.57 ( 5063, 8307) Empty
2007-04-16~10:10:07.57 ( 8308,11460) Queued
2007-04-16~10:10:07.57 (11461,16617) Queued
2007-04-16~10:10:07.57 (16618,19947) Empty
2007-04-16~10:10:07.57 (19948,23246) Queued
2007-04-16~10:10:07.57 Local 2 agent-based encoding; waiting for agent to finish
2007-04-16~10:10:07.96 Local 2 process_job failed with "End_of_file"; putting job (0,5062) back
2007-04-16~10:10:07.96 put back range
2007-04-16~10:10:07.96 ( 0, 5062) Queued
2007-04-16~10:10:07.96 ( 5063, 8307) Empty
2007-04-16~10:10:07.96 ( 8308,11460) Queued
2007-04-16~10:10:07.96 (11461,16617) Queued
2007-04-16~10:10:07.96 (16618,19947) Empty
2007-04-16~10:10:07.96 (19948,23246) Queued
2007-04-16~10:10:07.96 Local 2 close connection
2007-04-16~10:10:07.98 using compression type 0 (no compression)
2007-04-16~10:10:07.98 Local 2 found no ranges after full
2007-04-16~10:10:07.98 Local 2 check_out_job got frames 0 - 5062
2007-04-16~10:10:07.98 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:10:07.98 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~10:10:07.98 Splitter: fewer queues than max_queues_split
2007-04-16~10:10:07.98 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-16~10:10:07.98 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~10:10:07.98 Local 2 sent zone string ""
2007-04-16~10:10:07.98 but that's not long enough! (5250)
2007-04-16~10:10:07.98 Local 2 sent range (0,5062)
2007-04-16~10:10:07.98 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~10:10:07.98 ( 0, 5062) Working (Local 2)
2007-04-16~10:10:07.98 ( 5063, 8307) Empty
2007-04-16~10:10:07.98 ( 8308,11460) Queued
2007-04-16~10:10:07.98 (11461,16617) Queued
2007-04-16~10:10:07.98 (16618,19947) Empty
2007-04-16~10:10:07.98 (19948,23246) Queued
2007-04-16~10:10:07.98 Local 2 agent-based encoding; waiting for agent to finish

and agent log
2007-04-16~10:10:02.00 x264farm version 1.09-163
2007-04-16~10:10:02.03 Using config file ".\config.xml"
2007-04-16~10:10:02.03 Temp dir: "E:\x264farm\temp\agent\temp"
2007-04-16~10:10:02.03 Ports: 50700 - 50703
2007-04-16~10:10:02.03 x264: "x264.exe"
2007-04-16~10:10:02.03 nice: 10
2007-04-16~10:10:02.03 agents: 1
2007-04-16~10:10:02.03 agent bases:
2007-04-16~10:10:02.03 [exact]
2007-04-16~10:10:02.03 pipe: false
2007-04-16~10:10:02.03 compressions: 0001
2007-04-16~10:10:02.03 deleting old files: after 604800 seconds
2007-04-16~10:10:02.03 trying port 50700
2007-04-16~10:10:02.04 bound to port 50700
2007-04-16~10:10:02.04 listening...
2007-04-16~10:10:02.04 working!
2007-04-16~10:10:07.10 Got a connection from 169.254.25.121:1751
2007-04-16~10:10:07.10 working!
2007-04-16~10:10:07.10 2 first pass frames 0 to 5062
2007-04-16~10:10:07.10 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.10 2 agent-based encoding
2007-04-16~10:10:07.10 2 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 40ebd3.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.50 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 40ebd3.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.51 Got a connection from 169.254.25.121:1756
2007-04-16~10:10:07.51 working!
2007-04-16~10:10:07.51 3 first pass frames 0 to 5062
2007-04-16~10:10:07.51 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.51 3 agent-based encoding
2007-04-16~10:10:07.51 3 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 26cabf.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.54 Got a connection from 169.254.25.121:1750
2007-04-16~10:10:07.54 working!
2007-04-16~10:10:07.54 4 first pass frames 8307 to 11460
2007-04-16~10:10:07.54 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.54 4 agent-based encoding
2007-04-16~10:10:07.54 4 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 124d23.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.59 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 26cabf.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.60 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 124d23.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.60 Got a connection from 169.254.25.121:1757
2007-04-16~10:10:07.62 working!
2007-04-16~10:10:07.62 Got a connection from 169.254.25.121:1758
2007-04-16~10:10:07.62 working!
2007-04-16~10:10:07.62 5 first pass frames 0 to 5062
2007-04-16~10:10:07.62 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.64 5 agent-based encoding
2007-04-16~10:10:07.64 5 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats a7e4e6.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.64 6 first pass frames 8307 to 11460
2007-04-16~10:10:07.65 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.65 6 agent-based encoding
2007-04-16~10:10:07.65 6 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 37aaac.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.70 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats a7e4e6.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.70 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 37aaac.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.71 Got a connection from 169.254.25.121:1759
2007-04-16~10:10:07.71 working!
2007-04-16~10:10:07.71 Got a connection from 169.254.25.121:1760
2007-04-16~10:10:07.71 working!
2007-04-16~10:10:07.73 7 first pass frames 0 to 5062
2007-04-16~10:10:07.73 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.73 7 agent-based encoding
2007-04-16~10:10:07.73 7 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 0c36c2.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.75 8 first pass frames 8307 to 11460
2007-04-16~10:10:07.75 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.75 8 agent-based encoding
2007-04-16~10:10:07.75 8 doing the following: