x264farm: distributed encoding [Archive]

omion

5th November 2006, 09:19

Well, after about a year of working on this, let's see how it works in the real world! :D

I have been making a parallel encoder for x264 which can be distributed across many computers in a network. I have not done any serious benchmarking for it, but it is faster than running it on 1 computer. It was designed to be extremely portable; the only things you need on most of the computers is the agent program (supplied) and x264. All the AVIsynth processing is done by a central computer. I have personally tested it running on a Windows x64 computer, a 64-bit Linux box, and an Intel Mac Mini.

The idea is similar to tobias's ELDER, but designed from the ground up to be used on the network. It also supports fun stuff like resuming encodes (sometimes with a big penalty, but it's better than starting over)

A few notes:

It is still a work in progress, so it will make a file called out_dump.txt in the encode directory containing a bunch of debugging stuff.
You really need a fast network, and at least one decent computer. I sacrificed network efficiency and scalability for ease-of-use and portability.
I find it very easy to use, but I'm a bit biased. There are probably many usability issues that still need to be worked out.
All the "config.xml" files in the downloads are for my setup only. You want to change all the options around to reflect your setup before you start it up.

Eh... I guess that's all for the announcement. Have fun!

Readme (http://omion.dyndns.org/x264farm/x264farm.html)

DOWNLOAD VERSION 1.15:

WINDOWS BINARY (http://omion.dyndns.org/x264farm/x264farm-1.15_win32.rar) (controller and agent)
SOURCE (http://omion.dyndns.org/x264farm/x264farm-1.15_src.zip)
(Agent binaries for OSX (intel) and 64-bit Linux are available on request. I can't build 32-bit Linux binaries, but it's pretty easy to do yourself)

You also need the following:

x264 (http://mirror05.x264.nl/Cef/). NOTE: Currently, only builds done with MSVS will work with x264farm's controller-based encoding in Windows. I used to build my own, but I've found that Cef's builds work great so I've decided to stop making my own.
avs2yuv (http://akuvian.org/src/avisynth/avs2yuv/) on the controller computer.
nice.exe (http://omion.dyndns.org/x264farm/nice-20070415.rar) for encoding with different priorities on a Windows machine. If you just want to use normal priority, just set <nice> to 0 and it will be ignored. (source (http://omion.dyndns.org/x264farm/nice-20070415_src.rar). updated 2007-04-15)

when building from the source:

You need OCaml, extlib (http://sourceforge.net/projects/ocaml-lib/), and xml-light (http://tech.motion-twin.com/xmllight.html)
Run "make" in the /common directory FIRST
Run "make" in the /controller and /agent directories.

It's harder than it should be, but I haven't spent a long time polishing the build process

CHANGELOG:
1.15 (2007-10-26):
Added ad-hoc agent discovery (although disabled by default). Agents may be added while the encoding is running, even if they are not specified in the controller's config.xml file.
Ability to specify log file location on command line for both agent and controller.
Faster controller-based compression.
Changed the bucket-based agent cache to a single circular buffer.
Changed the default port from 50700 to 40700. This will only affect you if you re-use one of my example config files.
1.14 (2007-10-03):
Fixed an inconsistency with running x264 with spaces in the path.
Added the capability to change the avs2yuv executable in the config.xml file.
Add timestamps to the error messages and when the display was last updated (counting from when the program was started).
Show "~" for errors if there are aren't enough to display, and made the default number of errors displayed 4.
Doesn't update the FPS on the second pass if the encoding failed.
Properly handle dead agents on the second pass.
1.13 (2007-07-03):
Added the --batchmult option to help with one agent checking out a big job at the end and making everything else wait for it.
Added previous split points to the 1st-pass temp file. This means that encodes will start up faster after having been stopped.
Optimized the splitter to try to get the jobs out faster in the beginning.
1st-pass agents will now try to avoid working on jobs which may have incorrect start points, in order to minimize the number of re-encoded frames.
Put ETA display back for both passes. Note that the ETA given in the first pass measures the end of the first pass only, and is incredibly inaccurate.
Second pass GOPs are no longer encoded chronologically.
Completely redesigned first pass to be easier to work on.
1.12 (2007-05-30):
Added a heartbeat thread to each agent when agent-based encoding starts. Agents will send a signal to the controller every 10 seconds. If the controller does not receive a signal within 30 seconds, that agent is given up for dead. This should greatly reduce on the amount of stalled encodes.
Controller output is much more organized, and no longer prints huge amounts of scrolling garbage. The out-dump.txt files are just as garbage-filled as before, though.
1.11 (2007-04-28):
Fixed an overflow in the Matroska timecode calculation.
Second-pass equality check takes bitrate into account.
Changed the GOP layout from B-trees to red-black trees (not that anybody would notice the difference).
1.10 (2007-04-25):
Implemented per-pass compression, so that the first and second passes have different compression priorities
More precise file size (tunable with the --sizeprec option)
Ratecontrol handles extreme bitrates better
The --keeptemp option actually works now...
Minor sanity checks when writing the files
1.09 (2007-03-10):
Agent now deletes temp files which are older than a specifiied amount of time (by default 1 week).
Agent now uses niceness when both nice and piped input are being used.
2nd pass bitrate should be more accurate.
1.08 (2007-02-07):
Redesigned first pass. It should pick jobs more intelligently. The occasional problem with credits taking a very long time to encode is minimized.
More verbose garbage printed to the screen regarding the new first pass mode. It will disappear shortly.
Second pass should recover from x264 complaining about the minimum bitrate. Some jobs may fail a few times, but they will encode eventually.
The agent and config.xml format are unchanged.
1.07 (2007-01-05):
Fixed an issue with the controller computer rejecting network connections after an hour or so
Made some of the more pointless messages log-only
1.06 (2006-12-25):
2nd pass resuming works again (sorry, guys! :o )
Plugged the hole in the first pass which could cause frames which are being worked on to be re-encoded
Compression should be faster. There are no more checksums, and the compression loop is faster.
Fixed up the re-encode part to ask for confirmation instead of just quitting, and will just automatically restart if the old output file does not exist anymore.
Fixed a potentially major bug when the output files aren't written if the requested bitrate is below what x264 thinks is the minimum bitrate.
1.05 (2006-12-12):
Added optional compression for controller-based encodes. Currently extremely slow, though, in order to check for validity.
Added --nocomp switch to force compression off.
Added <pipe> setting to agent's config.xml file to locally pipe data to x264 in agent-based encodings.
Added tables in 2nd pass to show the performance of the various agents.
Controller now searches its own directory for avs2yuv.
Hid a bug involving the 1st pass splitter re-splitting ranges which are already allocated to agents not using controller-based encoding.
1.04 (2006-11-24):
Made agent-based encoding.
User-selectable number of agent cache buckets.
Added many ratecontrol parameters to the controller, and pass them to x264.
Lowered latency and memory usage due to sending data across the network.
1.03-152 (2006-11-20):
Fixed an error when the number of frames per second was an integer.
1.02 (2006-11-17):
MAJOR rewrite that honestly shouldn't have needed to happen.
Made the agent per-computer instead of per-core and fixed a bug with remote computers not getting enough data.
Controller only connects once per session, instead of once per job.
The code itself is much better-written.
Config file syntax completely changed.
1.01 (2006-11-09):
Made the --rerc option in order to change the re-ratecontrol frequency during the third pass.
Made the network more resilient to delayed-ACK problems.
Removed some of the more verbose printing (although everything is still in out-dump.txt).
Added simple FPS display (although it's buried in the debug info).
Fixed the batch length (now 5000, was 25000 but claimed to be 2500).
1.00 (2006-11-04):
Initial release. Rampant bugs and programmer hacks predominate.

WHEN REPLYING:
If you post really long chunks of text (like from out_dump.txt), use the [ code ] tag and it will make it scrollable. That will prevent the page from overflowing with a bunch of verbose garbage that x264farm likes to output ;)

crypto

5th November 2006, 11:02

What a cool idea. How do you schedule the tasks? Do you run a scene detection and distribute scenes for encoding?

Disabled

5th November 2006, 15:26

DDogg

5th November 2006, 15:52

Sounds exciting. Suggest you, or someone else consider running through a hypothetical setup using 2 or 3 windows XP PCs. PC-master, PC-slave1, PC-slave 2 in a step 1,2,3 format

Once I can get it working, perhaps I could do a how-to that the average Joe-tooluser like me could follow.

My set up is 7 XP machine on a gig backbone, with the master being a OC'ed 4.5 Gh Intel D930, slave1 - hyper-threaded 3.2 P4, slave 2 - Athlon XP 3200, plus various others along those lines. I'm happy to help others if you can help me get going.

omion

5th November 2006, 18:01

What a cool idea. How do you schedule the tasks? Do you run a scene detection and distribute scenes for encoding?

@Disabled:
I forgot to mention one thing: x264 compiled for Windows with gcc won't work currently. I made a build with MSVS that should solve your problem.

There's nothing special about my build, it just uses straight SVN. You can get build 598 here (http://omion.dyndns.org/x264farm/x264-598_omion.rar)

@crypto:
Basically, that's how it works. For the first pass, the video is broken up into large segments, where the last frame is determined by a simple scene-detection algorithm. The second pass is split up by scene, which is already known since they're in the stats file.

@DDogg:
I made a test with my one computer, and x264farm was actually around 15% slower than not using it, due to all the overhead. Unfortunately, all of my computers are different, so I can't make a scalability test, but I have noticed that even adding my wimpy Core Solo Mac Mini makes it faster than just one computer.

It would be nice if somebody did some tests with it. However, note that, by default, x264farm will re-encode 5% of the scenes that came the farthest from the ratecontrol prediction. Anybody interested in a comparison should turn that off with "--3ratio 0"

In terms of setting up, it is a bit difficult. Here's what I would do:
1. Copy the agent directory to any computer that you want doing the encoding.
2. On each computer, make as many copies of the config.xml file as you have cores (dual-core computers should have 2 different xml files, etc...)
3. Edit the config files to point to the right x264 executable, and each one must have a different port.
4. Run the agent on each computer, once for each config file, with "agent --config <some_file.xml>"
5. Extract the controller directory to the main computer
6. Edit the controller's config.xml file to point at all the agents you have running. You need to know the computer's IP address and the port that you assigned each one.

After that, the setup is done. Now you can start encoding. The basic command line is:
controller --first "<first_pass_options>" --second "<second_pass_options>" -b <target_bitrate>kbps --avs <input_file.avs> -o <output_file.mkv>

Look at the HTML file included in the release for what you should and shouldn't include in --first and --second.
Also note that the -b needs to be in the form "-b 123kbps" or "-b 123%", where the second form indicates a percentage of the first pass bitrate to use.

Disabled

5th November 2006, 18:30

omion

5th November 2006, 18:53

Many thanks for the x264 build, its working now.
I tried encoding a file on the host and on my laptop with your framework and it kinda worked. The laptop only got used about 20% of the encoding time, but its slow as hell anyway. I sometimes get the error 'Agent croaked with exception "End_of_file"; restarting' on my laptop, that might be the reason for it not being used that much.

That error's actually OK... sometimes. I think it happens when the controller connects, realizes there's nothing to do, and disconnects. It sounds like you have something else going on, though.

And I can't start a second encode of the same file without deleting the tmp files manually.

If you want to do the same encoding on the same file, it should reuse the temp files (in order to properly resume). If you really want to start from scratch (like for testing), use the --restart switch

I will not go out and benchmark this tool, as I don't have that powerfull machines here.
if you provide a linux 32bit build, I might try it at the university some time. But then I realise it still needs avisynth I do not have there, so I need to host it on a laptop...

*edit*
I just tried to encode with my laptop being the only client, and I get the same error as before and its not encoding the whole time, I would say the laptop is idle about 30% of the encoding time now.

I don't have an easy way to make 32-bit binaries with my Linux box, so I think you'd have to see if someone else can build it.

Now, when you say it's idle, does it do a job for 30% of the time and wait for the other 70%, or does it say it's doing something but it's only using 30% of the CPU?

Disabled

5th November 2006, 20:19

That error's actually OK... sometimes. I think it happens when the controller connects, realizes there's nothing to do, and disconnects. It sounds like you have something else going on, though.
[...]
Now, when you say it's idle, does it do a job for 30% of the time and wait for the other 70%, or does it say it's doing something but it's only using 30% of the CPU?
I'm refering to the "laptop is the only agent" situation now. The CPU on the laptop (on the controller as well) is idle for about 30% of the time and completely busy the rest.
I'm not sure if I read the (messy) log correct, but I think after an idle phase the above mentioned error is shown and then it continues to encode.
I just started a second run to verify the results.
I did two tests with both machines running an agent, the first time the laptop was used just once, the second time I noticed four phases the CPU was maxed out on the laptop.

I don't have an easy way to make 32-bit binaries with my Linux box, so I think you'd have to see if someone else can build it.

Well perhaps I will look into compiling the source myself. My Linux box is actually faster than the laptop I'm using :)

*edit*
Now it happened again. Both machines were idle for about 2 minutes, then the "End_of_file" Error and after a few secs its encoding again...

CruNcher

5th November 2006, 20:47

@omion
great work, but this wouldn't work with ABR 1pass encodes would it ?

omion

5th November 2006, 21:28

I'm refering to the "laptop is the only agent" situation now. The CPU on the laptop (on the controller as well) is idle for about 30% of the time and completely busy the rest.
I'm not sure if I read the (messy) log correct, but I think after an idle phase the above mentioned error is shown and then it continues to encode.
I just started a second run to verify the results.
I did two tests with both machines running an agent, the first time the laptop was used just once, the second time I noticed four phases the CPU was maxed out on the laptop.

Well perhaps I will look into compiling the source myself. My Linux box is actually faster than the laptop I'm using :)

*edit*
Now it happened again. Both machines were idle for about 2 minutes, then the "End_of_file" Error and after a few secs its encoding again...
2 minutes... weird. That sounds like a network issue. Are they connected wirelessly?

@omion

is this only working for a 2pass encode ?
Yup. It's only 2-pass because that's all I use :p It would be quite difficult to turn it back into a 1-pass, as both passes are set up quite differently.

Disabled

5th November 2006, 21:39

2 minutes... weird. That sounds like a network issue. Are they connected wirelessly?

Nope, 100% pure old fashioned 100Mbit Local Area Network
In addition to that I would say there was no traffic at all during that time... (doing a third run...)

*edit*
The controller sends Data to the agent. The Laptop starts encoding and it finishes ("Exited 0") on the Laptop. The Agent then shows "lap recieving stats" for a little over a minute. And then contines with "lap done recieving stats", but the Agent then shows the EOF exception and continues to encode...
I guess I'll send you the whole log. I have the feeling the "revieving stats" message is just once in the log file, while "done receiving stats" is seven times, and I counted five idle times...
Encoding time: 20 Mins, with about 5 mins idle

pyrates

5th November 2006, 22:49

Is there a way to distribute the avisynth script as well or does that still need to be worked on just one pc only for now? Because the one I use is quite processor intensive but gives awesome results.

omion

6th November 2006, 01:39

pyrates

6th November 2006, 01:51

I had initially made the program to distribute everything, but that requires being able to have the input files available to all computers, and also to have AVIsynth and all the filters available to all computers.

Synchronizing all the computers and especially making it work with non-Windows computers proved to be just about impossible. It was far too much work for its usefulness, so I basically got rid of it as soon as I got it working.

Well that would be exactly what I'd need. Plus avisynth is something that's only available for computers that run windows on it. And without it, I just can't use this form of distributed computing. I'd have no problem having the source files copied onto all the computers anyways.

ChrisBensch

6th November 2006, 05:17

Wow, thanks for your effort in distributed encoding!
I gave it a quick try and didn't get it working.
The agent throws an error:
2006-11-05~15:12:13.01 Doing the following:
x264.exe --fps 10000000/333667 --pass 1 --stats "C:\agent-win32\tmp\0 893 969666.txt" -o NUL - 720x480
2006-11-05~15:12:14.34 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-05~15:12:14.35 DOING!

This is the controller config.xml
<config>
<temp>C:\controller-win32\tmp</temp>
<agents>
<agent name="local">
<ip>192.168.0.1:50722</ip>
</agent>
</agents>
</config>

And the agent config.xml:
<config>
<temp>C:\agent-win32\tmp</temp>

<port>50722</port>
<x264>x264.exe</x264>
<nice>00</nice>
</config>

starting the controller with this cmd
controller.exe -b 1500kbps --avs test.avs -o test.mkv

I'm using the latest x264 build r598A
If you need more information, just tell me.

I'm having the same problem, and I've replaced x.264 with the correct version from this thread. Here is my output:
---------------------
D:\TEMP\agent-win32>agent --config config.xml
Started up the print mutex
Using config file ".\config.xml"
Temp dir: "D:\\TEMP\\cpu1"
Affinity: ""
Port: 50722
x264: "x264.exe"
nice: 10
Connecting...
Listening...
DOING!
Got a connection from 127.0.0.1:1287
Doing stuff
Doing the following:
nice -n 10 x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --ana
lyse none --vbv-maxrate 25000 --me dia --merange 12 --progress --no-psnr --fps 24000/1001
--pass 1 --stats "D:\TEMP\cpu1\0 1298 6414bb.txt" -o NUL - 1280x608
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2

Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
DOING!
Got a connection from 127.0.0.1:1290
Doing stuff
Doing the following:
nice -n 10 x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --ana
lyse none --vbv-maxrate 25000 --me dia --merange 12 --progress --no-psnr --fps 24000/1001
--pass 1 --stats "D:\TEMP\cpu1\0 1298 5a6db2.txt" -o NUL - 1280x608
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2

Agent croaked with exception "Unix.Unix_error(An existing connection was forcibly closed b
y the remote host.
, "recv", "")"; restarting
DOING!
---------------------

I don't think it's an x264 setting as it looks as though it's starting. I'll try without the VBV maxrate and see what happens.

omion

6th November 2006, 05:44

That sounds sort of like something went wrong on the controller side. Does the controller output say anything about an error or exception?

ChrisBensch

6th November 2006, 05:44

By removing the --vbv-maxrange stuff it got much further, I have two machines, one with dual cpu and one with dual core (controller running two agents also). It kept sending all frames to one machine, so I added "--batch 2500 --split 250" and it sends data to the other machine, but my two local agents both die with:

DOING!
Got a connection from 127.0.0.1:1453
Doing stuff
Doing the following:
nice -n 10 x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --ana
lyse none --me dia --merange 12 --progress --no-psnr --fps 24000/1001 --pass 1 --stats "D:
\TEMP\cpu1\0 12720 31d72a.txt" -o NUL - 1280x608
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2

Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting

So now I'm kinda stuck...any ideas?

Here is my avs:

LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\DGDecode.dll")
LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\TIVTC.dll")
MPEG2Source("Hero.d2v")
Crop(16,134,-16,-142)
TFM(d2v="Hero.d2v")
#TDecimate(cycleR=3)
TDecimate()
LanczosResize(1280,608)

Here is my controller startup line:

controller -b 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --merange 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weightb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --merange 12 --progress --no-psnr" --avs "D:\$VIDEO_WORK$\WORKING\Hero.avs" -o "D:\$VIDEO_WORK$\WORKING\Hero-farm.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

ChrisBensch

6th November 2006, 05:48

Here is the controller output:

edited...

Notice the error:
> LOCAL CPU 2 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION "Unix.Unix_error(_, \"send\", \"\")"

omion

6th November 2006, 06:38

So the local machine croaks, but the agents on the other machine work as expected? That is strange.

Try out this (http://omion.dyndns.org:8080/x264farm-net_test.rar) controller, and see if that helps. If it doesn't, see if the line Local CPU 2 sent 4096 bytes occurs between the lines Local CPU 2 sent first pass options
> LOCAL CPU 2 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION "Unix.Unix_error(_, \"send\", \"\")" Also, the error line should now give a number where the underscore is. If the error still occurs, write back with that number.

I must admit, I don't really know why it's dying, but I think I can figure it out given enough time (and patience!)

PS. You can delete the controller output now. I copied it to my computer, and I don't think anybody else will be able to comprehend it ;)

ChrisBensch

6th November 2006, 06:44

omion

6th November 2006, 07:41

With the new controller, here's what I got:

Local CPU 1 sent zone string ""
Local CPU 1 sent range (0,2582)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522890,recv,)
Local CPU 1 added (0,2582) back to the split list, since it caught an exception
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX

2522890?! That's... very big. That doesn't look like a normal error code. Maybe they aren't as portable as I thought.

Well, it's odd that you now get the error in a different place (from a "recv" instead of a "send") That really makes it sound like something else in the system is getting in the way of transferring the data.

Just a few questions: what version of Windows do you have, and do you use a firewall / antivirus?

I have to get to bed soon, but I updated the test controller (http://omion.dyndns.org:8080/x264farm-net_test2.rar) to give me a bit better info about what's going on. Try it out and post the log like you did in post #20. I should be able to look at it again tomorrow morning.

Disabled

6th November 2006, 08:32

I ran a test with the test2 controller and the problem is still there:
2006-11-06~08:04:58.32 lap receiving stats
2006-11-06~08:04:58.51 lap recieving stats
2006-11-06~08:07:58.56 lap done recieving stats

By the way, the "recieving stats" is printed out twice the first time and never again...
If I should have tested the prior build just tell me.

ChrisBensch

6th November 2006, 08:43

controller v.test2

Here's the newest output:

> LOCAL CPU 1 GOING ON SPLITTER_LIST_CONDITION
Local CPU 1 getting a split
Local CPU 1 got (2582,5175)
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX
> HOME-MAIN 10001 GOING ON SPLITTER_LIST_CONDITION
> LOCAL CPU 1 BROADCASTING SPLITTER_LIST_CONDITION
> HOME-MAIN 10001 WAITING ON SPLITTER_LIST_CONDITION
Local CPU 1 do_job 2582-5175
Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522900="The opera
tion completed successfully.\013\n",recv,)
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX
Local CPU 1 connect_and_do
Local CPU 1 connected to 127.0.0.1:50721
Local CPU 1 connected!
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX
Local CPU 1 getting a split
Local CPU 1 got (2582,5175)
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX
> LOCAL CPU 1 BROADCASTING SPLITTER_LIST_CONDITION
> HOME-MAIN 10001 GOING ON SPLITTER_LIST_CONDITION
Local CPU 1 do_job 2582-5175
> HOME-MAIN 10001 WAITING ON SPLITTER_LIST_CONDITION
Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522848="The opera
tion completed successfully.\013\n",recv,)
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception
> LOCAL CPU 1 UNLOCKED SPLITTER_LIST_MUTEX

I started with two computers, one with just two agents, one with a controller and two agents. I then removed one of the agents from the controller machine for a total of 3 agents and one controller. The weird part is that the agents on the other machine (remote agents only) starts to work fine, but the local encoder machine is the one that borks. Hope this helps.

708145

6th November 2006, 08:45

@omion
great work, but this wouldn't work with ABR 1pass encodes would it ?

I agree: Great work omion :)

And maybe it is useful for you guys that I throw a windows cluster mode for ELDER into the ring as well :p
It's just that I leave 2pass x264 encoding to ominon for the time being :) But ELDER beta5b will feature a distributed x264 crf mode.

edit: ELDER beta5b is out now -> http://forum.doom9.org/showthread.php?p=718465#post718465

Distributed abr is not more work so I can do it for beta5c if you insist, Cruncher :D

bis besser,
T0B1A5

ChrisBensch

6th November 2006, 08:48

I removed all agents on the controller box and just used a remote machine, so far so good. I have two other computers that I'll be adding this week...I'm excited about getting this weird thing fixed. I have three machines with Pentium D 2.8GHz and one Dual AMD Athlon MP @ 1.6GHz (this may become my controller and let the other three work).

ChrisBensch

6th November 2006, 16:24

I'm also going to try the new ELDER...I have 5 machines at home just waiting for some distributed encoding!

omion

6th November 2006, 18:26

ChrisBensch

6th November 2006, 18:44

Here's the output from test3 controller:

Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION Unix.Unix_error(2522470="The opera
tion completed successfully.\013\n",recv,)
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception

omion

6th November 2006, 18:48

@ChrisBensch:
Fun. It looks like the error was "no error". That's why the number was so big. I'll have to look into that...

ChrisBensch

6th November 2006, 18:55

yeah that's weird, the encoding still works on remote machines, just not the local machine...

omion

7th November 2006, 01:48

yeah that's weird, the encoding still works on remote machines, just not the local machine...

OK. Try out this one (http://omion.dyndns.org:8080/x264farm-net_test4.rar). It may give a bunch of warnings, but it should keep going. I don't know if the end result will be watchable, though. Test out the final file and make sure it plays correctly.

I just made the programs ignore that particular error, which would help if it's a superfluous error, but of course it still won't work if the error is there for a reason.

ChrisBensch

7th November 2006, 20:54

Here's the latest...

Local CPU 1 using zones ""
Local CPU 1 sent HELO
Local CPU 1 sent video info 1280x608 @ 24000/1001
Local CPU 1 sent zone string ""
Local CPU 1 sent range (2582,5175)
Local CPU 1 sent first pass options
Local CPU 1 sent 4096 bytes
Got a strange error while receiving. Check output file!
Got a strange error while receiving. Check output file!
Got a strange error while receiving. Check output file!
> LOCAL CPU 1 LOCKED SPLITTER_LIST_MUTEX FROM EXCEPTION "Failure(\"Tag \\\"\\\\000\\\\000\
\\\000\\\\000\\\" (length 1048576) was not expected here\")"
Local CPU 1 added (2582,5175) back to the split list, since it caught an exception

ChrisBensch

8th November 2006, 02:14

Maybe this is a dumb question...but should I be updating the agent.exe too? It's included in the build but so far we've only been working on the controller...

omion

8th November 2006, 02:20

@ChrisBensch:
Updating agent.exe probably wouldn't help anything. I've just been putting them both together because I've been updating them both at the same time, but the agent has stayed basically the same.

Unfortunately, I'm out of ideas. Since the last build didn't work, that means the error is actually an error and not something to be ignored.

Actually, do you have Windows 98 / Me? I've been told that 9x doesn't work for transferring data the way I do...

ChrisBensch

8th November 2006, 03:06

All of my PC's are XP SP2, I have Symantec AV Corp running on all boxes. It's been about a year since any of them have been reloaded so it could be something on my side. I'll try to find the time to reload them and get back to you with the results. Regardless of what happens, I should still be able to use my other 4 boxes for agents and the error one for controller only (which seems to work fine). Thanks for all the hard work. This is a project which I have been working on for some time and have been using it to learn software dev skills.

omion

8th November 2006, 04:24

Regardless of what happens, I should still be able to use my other 4 boxes for agents and the error one for controller only (which seems to work fine).

That sounds good. The agents on the controller's computer usually don't add up to much, since the computer is busy rendering the AVS script (especially with 4 other computers!)

Before you do a complete Windows reinstall, it would be good to try a different computer as the controller. You could probably get away with installing AVIsynth, using a simple colorBars() AVS file, and see what happens to the problem. Even though that may not be viable with normal encodes, it would be interesting to see which one of the following happens:
1) No errors anywhere
2) Errors with the old controller
3) Errors with the new controller

ChrisBensch

8th November 2006, 07:45

I'll give it a shot for ya

pimpMyHD

8th November 2006, 16:41

elguaxo

8th November 2006, 17:12

* You really need a fast network

This project sounds very interesting! Now my first question before I start. My network is now 54Mbps wireless, with really good reception. Does it make sense to build a "x264farm" with that network speed?

Thanks.

omion

8th November 2006, 17:50

considering the low speed encoding with HD resolution ( around 1 to 10 fps max ) could your system can be used over internet ?
I have several server over the web, and it will be fine if i could use their ressources to improve me encoding times.

regards,
Probably not. Remember that the low speed for encoding is because there is more data to encode. That means HD will actually be worse than smaller frame sizes.

Let's say you have a 2-hour movie @ 1920x1080x24, and half of it being sent out on the internet. That means you will need to upload 1920 * 1080 * 1.5 * 24 * 3600 bytes, or 250GB! On my internet connection, that would take around 3 months...

This project sounds very interesting! Now my first question before I start. My network is now 54Mbps wireless, with really good reception. Does it make sense to build a "x264farm" with that network speed?

Thanks.
54Mbps is probably OK. My experience with wireless is that the actual transmission speed is about 1/3 of the connection speed. For a 54Mbps connection, you can transmit around 4.5 million pixels per second, which is around 13 DVD-size frames per second.

What that means is that if all your remote computers can encode faster than 13 frames per second, they will be limited by the network connection. If they can't make 13 fps, wireless will be more than enough.

One more note: When I had my computers on a wireless connection, I actually saw one frame corrupted in such a way that the frame checksum said it was OK. Luckily, I had Ethereal running on each computer, so I could verify that the frame data was different. I realize that is an extremely unlikely occurrence, but I haven't trusted wireless ever since.

PS: My build (http://omion.dyndns.org/x264farm/x264-600_omion.rar) of x264 r600 is out.

elguaxo

8th November 2006, 19:18

Great! I will start my x264farm :cool:

foxyshadis

9th November 2006, 02:39

54Mbps is probably OK. My experience with wireless is that the actual transmission speed is about 1/3 of the connection speed. For a 54Mbps connection, you can transmit around 4.5 million pixels per second, which is around 13 DVD-size frames per second.

What that means is that if all your remote computers can encode faster than 13 frames per second, they will be limited by the network connection. If they can't make 13 fps, wireless will be more than enough.

One more note: When I had my computers on a wireless connection, I actually saw one frame corrupted in such a way that the frame checksum said it was OK. Luckily, I had Ethereal running on each computer, so I could verify that the frame data was different. I realize that is an extremely unlikely occurrence, but I haven't trusted wireless ever since.

If you use WPA encryption, that can't happen. (Although it might break the connection or cause a long reset instead.) Still, I have plenty of experience with wireless and ethernet drivers getting wedged opening and closing connections rapidly, and cheap routers rebooting during transfers - driver bugs or overheating - especially with WPA. Wireless reliability is around 12 years behind; fortunately, '95 was around when ethernet started getting reliable. :p

Maybe you should use md5 & sha1 for hashes. ;)

Omion, what do you think about using zlib for optional mild compression, to help with the slow network/fast system disparity? You can probably rip some code out of TCPDeliver for doing it efficiently.

pimpMyHD, you'll probably need to look at ELDER, which directly uses the lossy original and performs filtering on the slaves. Even that's bound to be horribly slow, especially since it works best with the full source available, not chunks.

ChrisBensch

9th November 2006, 03:38

Alright, I've got one controller and two slaves, one a dual athlon mp (2 @ 1.66GHz and another a dual core 2.8GHz. I'm going to encode "The Fog" in HD @ 1280x608x23.976fps. I'll be adding another dual core 2.8GHz soon.

Here's my script:

LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\DGDecode.dll")
LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\TIVTC.dll")
MPEG2Source("The Fog-HDTV.d2v")
Crop(0,0,-2,-8)
TFM(d2v="The Fog-HDTV.d2v")
#TDecimate(cycleR=3)
TDecimate()
LanczosResize(1280,608)

Normally on one machine I get about 8-10fps on the first pass and 2-3fps on the second...so let's see what we get!

aicjofs

9th November 2006, 06:47

You say probably need "Avs2yuv" installed.

Does WinXP machines need this? If so where do I install the .exe (the path, controller directory)? I get an "avs2yuv" exited with error 1 on the controller upon startup.

edit: my bad, seems the d2v I had made was made with the a different version of DGIndex then the plugin dll.

omion

9th November 2006, 06:55

Yup, every controller computer needs it. You can put it either somewhere on the path, or wherever you are running the controller from (the controller's directory will not work if you start the controller from a different directory)

The "avs2yuv exited with error 1" error generally means you have an error in your AVS script. When this happens to me, I open the script in VirtualDubMod and see what it says about the script.

[ edit: I see where the confusion comes from. What I meant was that you do need avs2yuv installed, so you probably need to download it (since I don't think most people have it). I'll make that clearer in the first post. ]

aicjofs

10th November 2006, 06:44

omion

10th November 2006, 07:06

Working nice on a P4 3.2Ghz 1GB RAM, P4 2.8 Ghz 512MB RAM and a Dual Core Opteron at 3.0Ghz, 2GB RAM and Gigabit ethernet network, leaving the other rigs out for the time being. Although the controller with 2 agents brings the Opteron system to it's knees. I never intalled nice.exe and just configured the xmls to 0 on all systems.

When you say "bring it to it's knees", do you mean that all the avs2yuv instances max out the CPU, or is it a memory issue? If the problem is with the memory, use SetMemoryMax(64) at the top of your AVS script. x264farm actually doesn't benefit from a huge frame cache, so the only reason to have more than that is if you have other wide-reaching filters.

Also, if you use many post-processing filters, you can save CPU time by using the --fastavs option. Make a copy of the AVS script and delete all the spatial post-processing filters from it. Then pass it with the --fastavs option. It's not much of a savings, but it should help.

My current setup is like this
farm.avs:
SetMemoryMax(64)

mpeg2source("d2v.d2v",cpu=6,iPP=false)
Telecide(order=1,guide=0,post=2,back=1,bthresh=40,nt=20,ovr="over.txt",show=false).Decimate(mode=0,show=false)

coloryuv(levels="TV->PC")
aWarpSharp(8.0, 2, 0.5, 2)
Undot()

crop(8,0,704,480)
farm-fast.avs:
SetMemoryMax(64)

mpeg2source("d2v.d2v",cpu=0,iPP=false)
Telecide(order=1,guide=0,post=2,back=1,bthresh=40,nt=20,ovr="over.txt",show=false).Decimate(mode=0,show=false)

crop(8,0,704,480)
Note that the two scripts have to output the same size frame, and you still have to use IVTC in both.

I like where this is headed Omion, reminds me of Vidomi 4 years back or so with MPEG4 ASP and I really like it. Thanks for taking the time to do something that has really been needed to be tackled for quite a while.

And thank you for taking your time to test it out. It is still very much in its infancy, and I still have a lot of things I want to add to it. Stay tuned! (My next release should be soon...)

Maybe I missed it somewhere but have you implemented a que function to batch encode?
No queue yet. I've been thinking about that ever since I rebuilt x264farm from scratch, but I was never quite sure what the best way to implement it would be. It'll get in there sometime, I just don't know when.

foxyshadis

10th November 2006, 07:50

omion

10th November 2006, 08:03

Fast-avs is a great option for first pass! I should try to get that in MeGUI somehow (since it sometimes chooses a slooooooooow deinterlacer).
Well... it's not really used the way you probably think it is. The fast-avs is only used for the split-point-finding thread during the first pass. All the actual data that's sent to the agents come out of the regular AVS script.

But actually, a first-pass AVS sounds like a great idea! I'm glad you thought I thought of it ;)

Also note that 1.01 (which should be coming out tonight) will print out FPS stats. They'll be kind of buried in the debug data, but they're there.

omion

10th November 2006, 08:41

Well, that was faster than I thought. 1.01 is out. Get it in the first post.

The main change is that the combined FPS is now displayed for each pass.

The FPS is determined by the number of frames calculated, not the number of frames output. During both the first and second passes, there may be some frames that are thrown out, so this may not quite reflect the actual number of frames output.

You may have to look a bit to find the FPS line through all the debug garbage. One of the other things changes was weeding out some of the more verbose debug lines.

I also added the --rerc option. Basically, during the second pass, ratecontrol is done at the beginning of the pass, but it is also done when all the frames have been processed. x264farm will then re-encode the GOPs which are the farthest from what the ratecontrol says. The ratecontrol will then be processed every --rerc GOPs processed. Basically, set this number lower for more accurate bitrate and complexity targets, while giving up a bit of parellelism while the ratecontrol is being calculated.

The default for --rerc is 0, which means that all the GOPs have to be processed a second time before the ratecontrol is run a third time.

It's a bit complex without knowing the inner details of x264farm, but feel free to ask questions!

aicjofs

10th November 2006, 10:05

When you say "bring it to it's knees", do you mean that all the avs2yuv instances max out the CPU, or is it a memory issue?

More of a memory issue. It is gone when I use SetMemoryMax(64). Of course in hindsight I see you already recommended that. Still hits the CPU to 100% but system is still responsive. It was the page file usage that was bogging it, all the avs2yuv instances chewed up all 2GB but now not an issue with SetMem. Going to experiment with nice.exe next, I don't need the computers for anything while encoding that's what a laptop is for, but I might see what different CPU priorities do.

Whats your thoughts about 2 agents on a Hyperthreaded CPU? Can't imagine any gain as it's not another physical CPU.....If I have time I'll give it a try tomorrow to decide for myself.

In Megui, you can enter the codec configuration, and check the "Show commandline" box,

*sigh*, boy am I dumb. :D Amazing what is right under your nose all along. Appreciate the tip. That's exactly what I needed.

havix

10th November 2006, 18:04

Nice job!!! I can't believe I even got it to work to be honest. The only slow part to getting this working is copying and pasting different commands from megui and removing the incompatible parameters. I can't wait to play around with this some more this weekend and expand it from my current 2 machine setup to a total of 4 machines. Has anyone experimented what gives the best speed pertaining to the controller machine and if you should run only one agent so you don't choke the avisynth processing on a dual-core machine? Right now I'm running two agents on the controller but I'm not sure if this could be slowing down the delivery.

havix

10th November 2006, 18:15

Sorry guys I should have read through the rest of the instructions. I will play with task priorities in the config files.

aicjofs

11th November 2006, 01:57

omion

11th November 2006, 03:10

More testing. All of this is during the 1st pass, I'll wait for the 2nd pass before I add more.

When running one instance of agent.exe on each of the P4 machines, I am getting around 20-30% CPU usage. I thought it would be interesting to try --threads 2 in x264, but x264 says it's not compiled with ptthread and I see no speed difference. I added 2 instances of agent to the 2 P4's and cpu usage is close to 40-50% now. According to the new fps indicator I am getting about 20fps on each P4(10fps + 10fps per agent) and about 20fps each agent on the Opteron. So total fps is around 80fps per the indicator for the 1st pass.

So 2 questions now are:

The x264 ptthread issue. Is this something you need to add to your build of x264? or am I completely off base?

Yeah. I don't use it in my builds. In fact, I don't know if MSVS uses the same style threading as what x264 uses (it was really designed for GCC)

On the SetMemoryMax issue. I still have available RAM on the controller computer is there any benefit from setting this higher then 64?

Maybe. The difference probably won't be noticable, but you can set it as high as you want without running out of memory.

EDIT: One other thing. Splitter list too long splitter spinning? What does this mean?
That's perfectly OK. That means that there are more jobs in the queue than agents. In this case the splitter thread is paused in order to devote more resources to avs2yuv and the agents. The splitter thread starts up automatically when a job is checked out.

BTW: How is the CPU utilization laid out on the controller computer? Is it mostly avs2yuv, or does the agent (x264) take up the majority of the CPU? It looks a bit like your remote agents are starved for work.

aicjofs

11th November 2006, 03:38

2nd pass kicked out broken pipe error, and invalid arguement on the agents.

2006-11-10~17:24:16.85 Got a connection from 127.0.0.1:3451
22006-11-10~17:27:07.25 1st pass
2006-11-10~17:27:07.25 Doing the following:
x264 --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr --fps 24000/1001 --pass 1 --stats "G:\x264farm\temp\cpu1\165835 166209 05dd43.txt" -o NUL - 718x358
2006-11-10~17:27:18.57 Exited 0
2006-11-10~17:27:18.60 Finished OK; shutting down socket
2006-11-10~17:27:18.60 DOING!
2006-11-10~17:27:18.60 Got a connection from 127.0.0.1:3461
2006-11-10~17:27:25.59 Agent croaked with exception "End_of_file"; restarting
2006-11-10~17:27:25.59 DOING!
2006-11-10~17:27:36.78 Got a connection from 127.0.0.1:3463
2006-11-10~17:27:36.81 2nd pass
2006-11-10~17:27:37.42 Done recieving stats
2006-11-10~17:27:37.42 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats 02c6b7.txt" -o "G:\x264farm\temp\cpu1\output c4ab32.mkv" - 718x358
2006-11-10~17:27:37.48 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:37.48 DOING!
2006-11-10~17:27:37.59 Got a connection from 127.0.0.1:3469
2006-11-10~17:27:37.70 2nd pass
2006-11-10~17:27:38.07 Done recieving stats
2006-11-10~17:27:38.10 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats 5cf6b5.txt" -o "G:\x264farm\temp\cpu1\output a12837.mkv" - 718x358
2006-11-10~17:27:38.45 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:38.45 DOING!
2006-11-10~17:27:38.57 Got a connection from 127.0.0.1:3475
2006-11-10~17:27:38.89 2nd pass
2006-11-10~17:27:39.03 Done recieving stats
2006-11-10~17:27:39.03 Doing the following:
x264 --bitrate 1706 --fps 24000/1001 --pass 3 --stats "G:\x264farm\temp\cpu1\stats c66430.txt" -o "G:\x264farm\temp\cpu1\output a8a9cd.mkv" - 718x358
2006-11-10~17:27:39.18 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:39.18 DOING!

2006-11-10~17:25:11.96 Got a connection from 192.168.1.101:3454
2006-11-10~17:25:11.98 1st pass
2006-11-10~17:25:11.98 Doing the following:
x264 --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr --fps 24000/1001 --pass 1 --stats "d:\x264farm\temp\cpu1\164335 164709 c1d557.txt" -o NUL - 718x358
2006-11-10~17:25:32.31 Exited 0
2006-11-10~17:25:32.39 Finished OK; shutting down socket
2006-11-10~17:25:32.39 DOING!
2006-11-10~17:25:32.39 Got a connection from 192.168.1.101:3455
2006-11-10~17:27:17.37 Agent croaked with exception "End_of_file"; restarting
2006-11-10~17:27:17.39 DOING!
2006-11-10~17:27:28.57 Got a connection from 192.168.1.101:3467
2006-11-10~17:27:29.31 2nd pass
2006-11-10~17:27:29.42 Done recieving stats
2006-11-10~17:27:29.42 Doing the following:
x264 --bitrate 5 --fps 24000/1001 --pass 3 --stats "d:\x264farm\temp\cpu1\stats 6281e0.txt" -o "d:\x264farm\temp\cpu1\output 58bba6.mkv" - 718x358
2006-11-10~17:27:29.54 Agent croaked with exception "Sys_error(\"Invalid argument\")"; restarting
2006-11-10~17:27:29.54 DOING!
2006-11-10~17:27:29.84 Got a connection from 192.168.1.101:3473
2006-11-10~17:27:29.96 2nd pass
2006-11-10~17:27:30.31 Done recieving stats
2006-11-10~17:27:30.31 Doing the following:
x264 --bitrate 841 --fps 24000/1001 --pass 3 --stats "d:\x264farm\temp\cpu1\stats 5226cf.txt" -o "d:\x264farm\temp\cpu1\output 969611.mkv" - 718x358
2006-11-10~17:27:30.35 Agent croaked with exception "Sys_error(\"Broken pipe\")"; restarting
2006-11-10~17:27:30.35 DOING!

The AVS

SetMemoryMax(64)
SetMTMode(2)
DGDecode_mpeg2source("H:\CRZ0NNW7\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)
#blank deinterlace line
crop( 0, 60, -2, -62)

#resize
#denoise

Command

-b 1000 --avs "file.avs" --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr"

--second "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --thread-input --progress --no-psnr" -o "test.mkv"

Since I am a noob at x264 command line it's possible my command is the issue. I just copied from meGUI like foxy recommended.

EDIT:

BTW: How is the CPU utilization laid out on the controller computer? Is it mostly avs2yuv, or does the agent (x264) take up the majority of the CPU? It looks a bit like your remote agents are starved for work.

Each x264 is using about 20% so 40% total, and the rest for controller, and avs2yuv.

omion

11th November 2006, 05:49

2nd pass kicked out broken pipe error, and invalid arguement on the agents.

[...]

Since I am a noob at x264 command line it's possible my command is the issue. I just copied from meGUI like foxy recommended.

Your command line looks OK (*) (except your -b argument needs to have "%" or "kbps" after the number, but that throws an error at the beginning, so I assume it's OK). What does the controller output say when the agents were putting up the broken pipe errors? If you closed the window or something, the log is saved in the "out-dump.txt" file in the directory where you ran the controller.

Actually, there's something else odd. When the agents say "Doing the following: ..." it doesn't look like they actually get the second pass line that you specified. Make sure that the controller got the right options by opening up the controller's "out-dump.txt". Near the top, there should be a line that gives the second-pass parameters that the controller is actually using.

It's possible that the "broken pipe" that the agents are complaining about are actually the pipe to x264, indicating that it died prematurely.
[ edit: This is probably the case. I seem to remember that x264 will croak if the first pass uses more B-frames than the second pass specifies, and since the second pass options didn't make it to the agent, it defaults to 0. ]

Each x264 is using about 20% so 40% total, and the rest for controller, and avs2yuv.
Hmm. It's odd that the remote computers aren't using any more than 20% per agent. And it's probably not a network issue, or adding more agents wouldn't help. Are you using version 1.01?

(*) The option vbv-maxrate may throw off the second pass. The reason is that I had to do ratecontrol in the controller, but I hardcoded a lot of the values to the defaults (qpmin, qpmax, ipratio, etc.) I'm pretty sure vbv-maxrate is one of those that would need to be supported in the controller, but isn't. The result is that x264 will make some GOPs with a lower bitrate than the controller expects, so they are marked to be re-encoded and the controller will inflate the bitrate of the entire GOP to accommodate that. (I can't really explain it too well)

NOTE: If you stop the controller, then restart it, it will pick up where it left off. You have to give it the same arguments, or it will think it's doing a different encode.

aicjofs

11th November 2006, 07:09

Looks like you nailed it. The 2nd pass options were blank in the dump log for the controller, I'm getting much better at understand the format of your dumps. I have started it 3 times in a row now and the 2 pass options are always there. I have run it the whole time with a .bat file so it's not like the options weren't there to begin with. If I look deeper through the logs I can see a x264 error about not the same b frames as the 1st pass, which of course makes since when no 2nd pass options were sent. Since I can't duplicate this it must be a non issue.

It's also complainging about no buffer set when the vbv-maxrate was set. I'm leaving out that option next go around. If you don't think we should be using it(vbv-max) maybe another update to the help file?

About to start another round. Next I'd like to see about getting more work for the clients.

Pretty exciting stuff actually. I'll post a screenshot in a few.

EDIT:

Controller

http://home.comcast.net/~aicjofs/264farm.jpg

One of the agents. Looks Identical to the other.

http://home.comcast.net/~aicjofs/264farm1.jpg

omion

11th November 2006, 08:59

Ooh. Screenshots. Yummy.

The shot of the network utilization is a bit odd. It looks kind of like it's maxing out at ~10%, which would be right where 100Mbps would be. Are your other computers connected with gigabit also?

I made a little network test setup for debugging, if you want to see if it's a network issue. The "sender" is here (http://omion.dyndns.org/other/from3.exe), and the "receiver" is here (http://omion.dyndns.org/other/to3.exe).

As with x264farm, start up the receiver first:
to3 45678 4096
(45678 is the port to use, 4096 is the string length)

Then start up the sender:
from3 192.168.1.100:45678 4096
(192.168.1.100:45678 is where you started up the receiver, 4096 is the string length)

I can get my jumbo-frame gigabit network up to ~25% with this test. See if you get any more than 10%.

It's also complainging about no buffer set when the vbv-maxrate was set. I'm leaving out that option next go around. If you don't think we should be using it(vbv-max) maybe another update to the help file?
OK. I changed the working copy of the help file to include all the ratecontrol things. I should be able to get it out soon, or at the latest when 1.02 comes out.

Inventive Software

11th November 2006, 19:18

Do you plan to reduce the transmission size across networks omion? If what he has is average 12.5% usage across 1 Gbit network, that's more than 100 Mbit can handle, so it'd hinder it. Is the transmission size (i.e the stuff sent through the cables) affected by the size of the content?

omion

11th November 2006, 20:42

It is definitely a hinderance, but you can get around 20 DVD-size frames per second out of a 100mbit connection. With my settings, anyway, that's about all I can pull out of my remote computers.

The data will have to be transmitted somehow. The only ways around this is to transmit it before rendering (which would need avisynth installed on the remote computer), or compress the raw data before sending.

I gave up on doing remote avisynth, and the compression would have to compress 10MB per second while using as little CPU as possible. I suppose it would be easy to try out some compressions, but I have a feeling it would take up too much CPU power to be worthwhile.

I am actually thinking about making avisynth servers, where the "controller" would also be divided amongst several computers, so the agents would be able to distribute the network load.

I haven't been able to figure out how to do the load-balancing, but it should be possible.

708145

11th November 2006, 21:14

I gave up on doing remote avisynth, and the compression would have to compress 10MB per second while using as little CPU as possible. I suppose it would be easy to try out some compressions, but I have a feeling it would take up too much CPU power to be worthwhile.

<advertisement>ELDER does the avisynth processing on each compute node. If someone wants to try it.</advertisement>

http://forum.doom9.org/showthread.php?p=718465#post718465

bis besser,
T0B1A5

aicjofs

11th November 2006, 21:51

All machines are gigabit. No jumbo frames the switch doesn't support it. I get 33% utilization on all 3 machines. Picture below is connecting one client at a time. Seems the bandwidth is available. If you want to give me the avs and your controller command line I can mimmick it with my own input file.

http://home.comcast.net/~aicjofs/264farm2.jpg

Yes 708145 I have used Elder but last time I looked you were freezing x264 development we need to continuing pushing forward in the x264 arena. :D

708145

11th November 2006, 23:11

Yes 708145 I have used Elder but last time I looked you were freezing x264 development we need to continuing pushing forward in the x264 arena. :D

I froze the _GPL_ branch of x264 2pass encoding. Beta5 includes an x264 crf mode :)
There is a development version of x264 2pass as well.

omion

18th November 2006, 08:59

1.02 is out (at last)

It took so long because I tried to do too many things and ended up basically rewriting the controller. It should run a bit less buggy now.

You will only need to run one agent program per computer now. It does all the threading in the program, so that you don't have to.

aicjofs's issue with the remote machines not using all the CPU should be fixed now. It should be quite a bit faster (at the expense of ~8MB more memory usage, and maybe slower x264 startup time).

Note that the format of the config.xml files has changed. See my examples in the package for what to do.

There should be fewer bizarre bugs, too. I updated my version of OCaml, and it seems to have fixed a lot of the "operation completed successfully" errors.

I'm not including the Mac or Linux builds with the main download, because I don't think enough people use them to warrant having everybody download them. I am still planning to have the Mac build available as a separate download soon.

aicjofs

20th November 2006, 07:27

Major improvements in this version.

Easier to setup(less config.xml's to edit) for one.

So I tried running the agents on the clients with "number 2". This loaded the P4's to about 40-50% on 1st and 2nd pass. Network utilization about 17% and peaked to 20% on controller. Fps (reported in controller) of about 77fps 1st pass 75fps 2nd pass.

I then tried "number 4" and "+1" (for 2nd pass) for clients. That ran 4 to 5 instances of x264 on each client. Then on both passes the clients were loaded from 85-100% average around 90%. This boosted fps to 89fps 1st pass, and 83fps 2nd pass, also increased network util on controller to about 25% and peaked as high as 30%.

Normally I would get around 50fps if it was just the controller rig using MeGUI or Nero with the simple parameters. Obviously this is for light encoding and I can't assume it scales because I haven't tried it. But it seems like it would if I added a more difficult profile that normally drags along at 10fps on a single machine. With x264farm I might just get realtime speed around 20-30fps.

I really like this build. Only real waste of CPU time seems to be when the ranger has passed out all frames and you are near the end of the 1st pass and the controller is waiting for all the clients to send back their frames, this causes the other clients to go idle waiting for all the frames to return so the 2nd pass can start. Only cosmetic gripe(and this is only while testing it matters) is the agents toggles the "frames -fps" display to fast. When I had 4 "CPU's
" on the agent it was hard to read where each CPU was on what frame. Not a big deal but while testing it was happening to fast to read the bottom line. Maybe slow the switching delay or add all CPU's to the bottom line? No need to really see once x264farm matures, but it would be nice for testing purposes.

CruNcher

20th November 2006, 19:56

@ omion

what does this error mean ?

2006-11-20~19:30:40.00 x264farm version 1.02-151
2006-11-20~19:30:40.00 OPTIONS:
2006-11-20~19:30:40.00 -b 8000%
2006-11-20~19:30:40.00 --first "--no-fast-pskip --no-dct-decimate --bframes 3 --weightb --direct auto --no-cabac --subme 1 --analyse none --me dia --no-psnr"
2006-11-20~19:30:40.00 --second "--no-fast-pskip --no-dct-decimate --bframes 3 --weightb --direct auto --no-cabac --subme 2 --analyse p8x8,b8x8,i4x4 --me dia --no-psnr"
2006-11-20~19:30:40.00 --avs "sylver.avs"
2006-11-20~19:30:40.00 --fastavs ""
2006-11-20~19:30:40.00 --zones 2006-11-20~19:30:40.00
2006-11-20~19:30:40.00 --seek 0
2006-11-20~19:30:40.00 --frames 0
2006-11-20~19:30:40.00 --batch 5000
2006-11-20~19:30:40.00 --split 250
2006-11-20~19:30:40.00 --thresh 20.000000
2006-11-20~19:30:40.00 --preseek 0
2006-11-20~19:30:40.00 --force false
2006-11-20~19:30:40.00 --restart false
2006-11-20~19:30:40.00 --config "config.xml"
2006-11-20~19:30:40.01 --3thresh 0.800000
2006-11-20~19:30:40.01 --3gops 1073741823
2006-11-20~19:30:40.01 --3ratio 0.050000
2006-11-20~19:30:40.01 -o "out.mkv"
2006-11-20~19:30:40.01 Using config file ".\config.xml"
2006-11-20~19:30:40.01 CONFIG:
2006-11-20~19:30:40.01 temp_dir: "H:\\x264farm\\temp"
2006-11-20~19:30:40.01 agent_list 1:
2006-11-20~19:30:40.01 "Local CPU 1" = 127.0.0.1 50700-50703
2006-11-20~19:30:40.01 "Laptop 1" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.01 agent_list 2:
2006-11-20~19:30:40.01 "Local CPU 1" = 127.0.0.1 50700-50703
2006-11-20~19:30:40.01 "Laptop 1" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.01 "Laptop 2" = 192.168.1.3 50700-50703
2006-11-20~19:30:40.48 GOT NORMAL INFO "sylver.avs: 1920x1080, 25 fps, 10816 frames\013"
2006-11-20~19:30:40.48 ERROR: AVS file 'sylver.avs' does not seem to be valid

omion

21st November 2006, 05:46

@ omion

what does this error mean ?

2006-11-20~19:30:40.48 GOT NORMAL INFO "sylver.avs: 1920x1080, 25 fps, 10816 frames\013"
2006-11-20~19:30:40.48 ERROR: AVS file 'sylver.avs' does not seem to be valid

That normally means that the AVS file has an error and the video information could not be read. Except that (as you can see) the information could be read.

The problem is that I assumed the FPS is always a rational number (i.e. x/y), but over in PAL land it looks like it's just given as an integer. I'll make a quick fix and put up a new version as soon as I can.

omion

21st November 2006, 07:45

1.03 is out. CruNcher's problem should be fixed. That's the only change since 1.02.

ChrisBensch

21st November 2006, 10:19

I'm using one of my machines as a controller and the other as an agent with v1.03. Both are Gigabit and are using about 7% on average. I have 4 total ports for the agent machine but only one is being used and I only see one instance of x264.exe (I'm still encoding the first pass). I don't see any errors, but I'm sure it could be faster based on other people's results. I've even tried taking out my processing settings. I guess I have one question...is anyone else doing HDTV 1920x1080i ---> HDTV 1280x720p encoding? Or are these results posted from DVD backups?

I used the to3 and from3 network tests and I peaked my Gigabit setup at 17% using the settings from page 3. I guess I'll try with DVD stuff later...

omion

21st November 2006, 16:59

I'm using one of my machines as a controller and the other as an agent with v1.03. Both are Gigabit and are using about 7% on average.7% CPU, or 7% network usage? If it's network usage, how's the CPU usage?

I have 4 total ports for the agent machine but only one is being used and I only see one instance of x264.exe (I'm still encoding the first pass). I don't see any errors, but I'm sure it could be faster based on other people's results. I've even tried taking out my processing settings. I guess I have one question...is anyone else doing HDTV 1920x1080i ---> HDTV 1280x720p encoding? Or are these results posted from DVD backups?I must admit, I haven't done any HDTV tests with it yet. I'll try it out once I un-break what is going to become 1.04.

ChrisBensch

21st November 2006, 17:35

omion

21st November 2006, 17:43

Well, I think I was jumping the gun a bit, it seems that during the first pass I was only getting one instance of x264, but during the second I was getting two. The 7% was bandwidth, cpu was around 45%. Now that on the second pass, the cpu is at 100% and bandwidth is around 1%...my agent/encoder is the slower machine...so we'll see what happens when I add the rest of my boxes. Keep up the good work!
Great!

It looks like I still need to work on the network code to keep x264 fed, though. The agent should use 100% of each core, even for the first pass. As aicjofs noted, one way around this is to increase the number of agents running with the <number> line in config.xml, but that's still just a workaround. I'll see what I can tweak in the next release.

BTW: The reason you get one x264 during the first pass and two during the second pass is due to the pad="1" attribute in the <number> line. Basically, during the second pass, the jobs are small enough that a a relatively large percentage of the time is x264 setting up. Therefore it's a good idea to have something running in the background for doing something useful during that time.

ChrisBensch

21st November 2006, 20:59

So then only a single instance of x264 is normal during 1st pass? I thought that on a dual cpu system I'd get one per core? I have 4 ports setup for use (50700-50703)

havix

21st November 2006, 21:24

Just a suggestion, maybe you could integrate something like "psexec" to launch agents on remote machines when the controller launches. This would make things easier to jump right into encoding.

aicjofs

22nd November 2006, 05:34

So then only a single instance of x264 is normal during 1st pass? I thought that on a dual cpu system I'd get one per core? I have 4 ports setup for use (50700-50703)

You are changing the "number" tags in the controller config.xml right? Not only do you add ports but you need to change that number to get more x264 instances.

<number pad="1">change this number</number>

omion

22nd November 2006, 06:22

You are changing the "number" tags in the controller config.xml right? Not only do you add ports but you need to change that number to get more x264 instances.

<number pad="1">change this number</number>

Right... mostly. You actually don't need to increase the number of ports for more encodes.

Before 1.02, each agent listened on its own port and were completely separate. When I released 1.02, two things changed:
* I implemented a more "server" style for the agents. All of the jobs go through one port, and the agent program splits off threads to handle each one.
* I let the agent select one port from a range of ports, in order to squash an uncommon but annoying problem when a port is not available right after an agent is stopped.

So basically, the number of available ports was expanded, but the number of used ports was reduced to 1.

It is also worth noting that the agents also have a <number> line in their XML file. This <number> does nothing. Only <number> on the controller side will have any effect.
I had originally had the agents reject connections if there were already <number> connections, but figuring out what was connected proved difficult. I removed the effect, but I kept the config line just in case I re-enable it.

Just a suggestion, maybe you could integrate something like "psexec" to launch agents on remote machines when the controller launches. This would make things easier to jump right into encoding.That does sound like a good idea. I don't know what the best way to only start up the agent if it has not already started, but I think it would be possible.
I've got a lot of ideas backed up, so it will probably take a while to implement.

CruNcher

22nd November 2006, 09:36

Hi omion it's working now but something doesn't feels quite right with the cpu usage on the client see screenshot bellow

http://cruncher.mufflastig.com/x264/x264farm.png

Network utilization is @ 50%
Cpu usage on the client is fluctuating very heavy

and i have a improvement wish :)
could you please make the controller delete the created files on the fly after he concentate them this way you don't need 2x the diskspace :)

omion

22nd November 2006, 17:43

Network utilization is @ 50%
Cpu usage on the client is fluctuating very heavy

It's possible that the network is saturated at 50%. My gigabit network never gets more than about 35-40%. The next version will have a user-definable agent buffer, which may make things better (or at least different :D ) I still have some work to do on it, but it shouldn't take too long.

and i have a improvement wish :)
could you please make the controller delete the created files on the fly after he concentate them this way you don't need 2x the diskspace :)
It would be easy to add a switch that would do that. I'm hesitant to do it by default because if there was an error opening one of the files, the controller would have to recreate the deleted files when it was restarted. That hasn't happened to me yet, but I don't like the default behavior leaving the files in a state where it could happen. A switch would be good though. I'll add it into 1.04.

omion

25th November 2006, 01:14

OK. 1.04 is out, along with a very big addition that I knew needed to happen some time.

A few minor changes first:
* The "-b" bitrate option has been renamed "-B" to better match x264's parameters
* I also added the --savedisk command line option, to implement CruNcher's on-the-fly temp file deletion. It will not use up more disk space than the final file, but if something goes wrong when merging, the deleted parts have to be re-rendered

1.04 introduces "agent-based" encoding, when the agent has access to the AVS file and calls x264 directly, rather than "controller-based" where the controller does everything.

In order to set it up, you need to use the <base></base> line in the agent's config.xml. This is the base directory to search for the AVS file, in case the agent's computer has a different directory structure.

Here's an example:
Agent's config.xml: <base>F:\temp\x264farm</base>
Controller's AVS file: D:\movies\working\somewhere\file.avs

The following files will be searched for on the agent's computer:
F:\temp\x264farm\D:\movies\working\somewhere\file.avs
F:\temp\x264farm\movies\working\somewhere\file.avs
F:\temp\x264farm\working\somewhere\file.avs
F:\temp\x264farm\somewhere\file.avs
F:\temp\x264farm\file.avs

The first file that is found whose contents match the controller's AVS file will be used as input to x264, rather than getting the frame data through the controller.

Also, if you use <base></base> or <base/> in the config file, that indicates that the only place to look for the file is exactly where the controller found it. This is good if both computers have the same directory structure, or if the agent is running on the controller's computer.

If a matching file is not found anywhere, the agent will default to receiving data from the controller, just like before.

To turn off the agent-based encoding, either remove or comment out the <base> line in config.xml.

Get it at the beginning of the thread. Have fun!

akupenguin

25th November 2006, 03:27

The first file that is found whose contents match the controller's AVS file will be used as input to x264
If you check that the contents match, why not just send the avs to the agent and check whether it can be opened?
If you want to allow that the agent has the source files (not just the avs) in a different place, then you don't want to enforce that the two avs are the same. Or does your check for "match" ignore any string constants within the avs or something like that?

omion

25th November 2006, 06:44

You're right. I suppose I didn't mention one important thing, which I knew but didn't bother to say ;) :
All the AVS source files should be relative to the AVS, or, as akupenguin says, they won't work in a different directory. This includes the .D2V file: in DGindex, you should DEselect "Options > Use Full Paths" when making the D2V.

And I check for exact equality with MD5, so there's not a whole lot of room for tweaking the file.

ChrisBensch

27th November 2006, 06:54

thetrueavatar

29th November 2006, 10:45

Hello,
First thank you for this good intiative because distributed encoding is really cool.
However my main problem is that I would like to distribute my encoding on friends computer instead of my owns computers.
So the high network requirement seems too high.

So I have a suggestion to reduce a bit the network bottleneck. Is it possible to do a quick lossless compression on the controller (with hyufhyuv for example) that could reduce DVD size by 2 instead of sending raw video on the network? I have heard that Huffyuv was really fast and I suppose that time lost for lossless encoding/decodingto will be compensate by the gain over the network connection. This seems not too complicate to implement if we use agent base encoding and could provide a real performance improvement.
What do you think about it? Does it make sense?

omion

29th November 2006, 19:57

I'm using the new 1.04 with "agent based encoding" and I have to say I like it. My network util is at about .25-.30% on a gigabit network. My normal MeGui fps is about 8-9fps on this particular movie and with two machines I'm getting about 15fps. I expect that when I add the other two, I could get around 27-29fps total...faster than realtime for my HDTV encodes. I can't wait to see this sort of thing in MeGui!
Great! Thanks for the feedback!

Hello,
First thank you for this good intiative because distributed encoding is really cool.
However my main problem is that I would like to distribute my encoding on friends computer instead of my owns computers.
So the high network requirement seems too high.

So I have a suggestion to reduce a bit the network bottleneck. Is it possible to do a quick lossless compression on the controller (with hyufhyuv for example) that could reduce DVD size by 2 instead of sending raw video on the network? I have heard that Huffyuv was really fast and I suppose that time lost for lossless encoding/decodingto will be compensate by the gain over the network connection. This seems not too complicate to implement if we use agent base encoding and could provide a real performance improvement.
What do you think about it? Does it make sense?
It makes a lot of sense, actually. Although I'd probably make my own compressor, since it's about as much work to do that as it is to link to C code from OCaml. I've been toying with various compression settings, but it may be a while before I get it working.

The issue may be that the network connection is still too slow. My cheapo DSL connection can upload ~30KB/s. Uncompressed DVD-res video is ~500KB/frame. This means, on my connection, I could send 1 frame every 16 seconds (that's 0.06fps!) Even with 2:1 compression, that would still be extremely slow.

With computers that are so different in apparent speed (yours vs. whatever is on the end of the 0.06fps connection), it will probably slow down the encoding when compared to just using your computer.

Probably the optimal solution would be to use agent-based encoding and somehow share the DVD files with your friend (like with a shared folder in Windows) In this case, you would be transmitting the original compressed frames, so he might get ~1fps (a big improvement!) I don't know how well shared folders hold up over the internet, though.

Inventive Software

29th November 2006, 20:03

Forgive me if I've missed something, but if all the computers used for encoding have the source files that the agent has, would that not reduce the load over the network, i.e not transmitting the uncompressed AviSynth output? Just my $0.02, as I think anything to reduce the network dependancy to something justifiable like D1 over 100 Mbit networks (not possible at the moment as the data rate is something like 124 Mbits/sec) would be a good thing. :)

omion

30th November 2006, 00:49

Forgive me if I've missed something, but if all the computers used for encoding have the source files that the agent has, would that not reduce the load over the network, i.e not transmitting the uncompressed AviSynth output?

Yup. That's what agent-based encoding does. The controller tells each of the agents to do some range of frames, then the agents send back the compressed MKV files.

The original method of encoding, controller-based, acts like you described, but I added the agent-based encoding in 1.04.

Just my $0.02, as I think anything to reduce the network dependancy to something justifiable like D1 over 100 Mbit networks (not possible at the moment as the data rate is something like 124 Mbits/sec) would be a good thing. :)
I don't think I understand what you mean. You can't send D1 over 100mbps networks real-time, but if x264 can only encode 1/2 real-time, then the encoder is the bottleneck anyway.

I'm also writing a program which will analyze different compression methods to decrease network usage in case the source files are not accessible to the agent, but I don't know when that will be added.

Inventive Software

30th November 2006, 01:39

If you have enough computers for the encoding, then it'll be possible to encode more than realtime (we're talking people with access to university networks and suchlike), which is why reducing the raw transfer data over the network becomes critical, otherwise the network becomes the bottleneck.

Besides, it's fairly well known that 100 Mbit is the absolute maximum transfer in peak conditions with everything perfect (based on people stating dialup @ 56K never reaches 56K), and it's more likely to be around 90 Mbit tops (average ;)). Those lucky people with relatively new motherboards with Gigabit ethernet don't have to worry so much. ;)

TriHFH

30th November 2006, 04:50

Hey, thanks a lot for working on this project, I think it has a ton of potential. I tried using x264 farm but I am getting this error:

ERROR: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error

but i opened it in virtualdubmod and it opened fine w/o any errors. What can I do to solve this/ what is causing these problems?

omion

30th November 2006, 04:59

Hmm... for whatever reason avs2yuv didn't like the AVS file. Post the output of the command:
avs2yuv -frames 1 -o NUL pb213.avs
Oh, and :script: ;)

TriHFH

30th November 2006, 05:06

Here's the script:

DGDecode_mpeg2source("F:\pb213\pb213.d2v",info=3)
ColorMatrix(hints=true)
TDecimate(cycleR=3)
TTempSmooth(lmdiff=1,cmdiff=2,strength=1,scthresh=8)
RemoveGrain(5)
#crop
#resize
#denoise

And the output of that command is:

c:\x264farm\controller\pb213.avs: 1280x720, 24000/1001 fps, 61376 frames

omion

30th November 2006, 05:22

OK, that's really weird. The command I gave you is almost exactly what my program tests for the data. The only thing I can think of is that the controller can't find avs2yuv. Where did you put avs2yuv?

Are there any other warnings / errors / suspicious lines before the error you posted?

(my program actually tests the command avs2yuv -frames 1 -raw -o NUL "avsfile" 2>&1
Try that, too, just in case)

TriHFH

30th November 2006, 05:50

Nope, I did not notice any weird lines/errors before my error. Here is a screensot of it:

That new command outputs this:

C:\x264farm\controller\pb213.avs: 1280x720, 24000/1001 fps, 61376 frames

http://yogurtrat.com/files/1/error_thumb.png (http://yogurtrat.com/files/1/error.PNG)

I wasnt sure where to put avs2yuv, so i put it in the controller folder and the main folder :P

omion

30th November 2006, 06:17

I wasnt sure where to put avs2yuv, so i put it in the controller folder and the main folder :P
Well, the "right" folder is anywhere on the path. Putting it in the controller's directory is great as long as you call the controller from that directory.

Unfortunately, everything looks fine. There are a few issues with your command line, but nothing that would cause it to die there (*)

I'll try to reproduce the error on my computer and see what could be happening. In the mean time, try out a braindead-simple AVS script and see if x264farm will take it. Something like this:colorbars()

(*) Since I don't know what's wrong, I'll just nitpick everything else:p :

The -B option needs to be "-B 4353kbps", to distinguish it from 4353% of the first pass size. It looks like x264fram will default to percent, which is probably not what you want. I should make a warning for that...
leave out --stats from both the first and second passes. x264farm keeps track of the stats file internally.
The --zones option should be outside the -1 and -2 options. That is, it should be an option of x264farm itself, not part of the string to pass to x264. Otherwise the x264farm ratecontroller will not be too happy (it gets easily confused;) )
"--no psnr" should be "--no-psnr"

thetrueavatar

30th November 2006, 08:26

why don't you transmit compressed source(mpeg2 for example) instead of raw?

foxyshadis

30th November 2006, 08:49

The -B option needs to be "-B 4353kbps", to distinguish it from 4353% of the first pass size. It looks like x264fram will default to percent, which is probably not what you want. I should make a warning for that...

Perhaps % below 100, and bitrate above 100, if no modifier is given?

TriHFH

1st December 2006, 02:54

omion

1st December 2006, 03:05

why don't you transmit compressed source(mpeg2 for example) instead of raw?
It would be difficult to do through x264farm, since I would need to intercept the AVIsynth calls to the source files. You can, however, do it the roundabout way:
Share the folder with the source files on the main computer
Then use the "Map network drive" feature on each of the agent's computers to use the network folder as a separate hard drive (it does not need to have write access)
Then use "agent-based" encoding and use the new drive letter as the base directory

Perhaps % below 100, and bitrate above 100, if no modifier is given?
Yeah, that's probably what I'll do.

omion

1st December 2006, 06:43

I tried that avs and it seems to have worked (it is processing something) when i load it directly from avs2yuv, but when i load the avs into the controller, it gives me that same error, even with that simple avs.

And thanks so much for your help so far... I'm really excited to get this working :D

One more possibility, then I'm stumped. Are you using Windows 2000/XP? If not, many of the functions that I use (possibly including the avs2yuv call) won't work. If you aren't using 2000/XP, neither the controller nor the agent will work at all so you should focus on upgrading instead of video encoding ;)

If you have Windows 2000 or XP, then try out this controller (http://omion.dyndns.org/x264farm/x264farm-1.04+_controller-only.rar). It will die before anything gets encoded even without your error, but post all the lines that start with "DEBUG:" That should narrow down the problem.

Kripsy

1st December 2006, 08:35

Is x264farm capable of encoding a h264 source file, or is it only mpeg stuff? I ask because I am encoding some rather large 1080p caps to something a bit more cpu friendly with megui currently, but it takes me upward two days to do it!

foxyshadis

1st December 2006, 11:41

Anything you can do through avisynth, x264farm can work with. It only cares about the output, not the input. You'd probably want the most recent DirectShowSource (see avisynth development forum) if you go that route, lots of bugfixes.

TriHFH

2nd December 2006, 20:36

Here is the results of the debug lines.

2006-12-01~06:33:05.03 DEBUG: Running command "avs2yuv -frames 1 -raw -o NUL "pb213.avs" 2>&1"
2006-12-01~06:33:05.04 DEBUG: avs2yuv.exe in current directory (C:\x264farm\controller)? true
2006-12-01~06:33:05.04 DEBUG: opening avs2yuv
2006-12-01~06:33:07.42 DEBUG: avs2yuv responded: "pb213.avs: 1280x720, 24000/1001 fps, 61376 frames\013\n"
2006-12-01~06:33:07.43 DEBUG: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error
2006-12-01~06:33:09.03 ERROR: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error

Now I tried the controller on another pc, and it got a little farther this time, but it still stopped. I tried this with a different avs file along with the simple one. The simple one gave the same error and stopped.

2006-12-01~23:46:05.00 x264farm version 1.04-153
2006-12-01~23:46:05.00 Using config file ".\config.xml"
2006-12-01~23:46:05.00 CONFIG:
2006-12-01~23:46:05.00 temp_dir: "C:\\x264farm\\temp\\controller"
2006-12-01~23:46:05.00 agent_list 1:
2006-12-01~23:46:05.00 "Local CPU 1" = 127.0.0.1 50700-50703
2006-12-01~23:46:05.00 "Laptop 1" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Katie 1" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.00 agent_list 2:
2006-12-01~23:46:05.00 "Local CPU 1" = 127.0.0.1 50700-50703
2006-12-01~23:46:05.00 "Laptop 1" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Laptop 2" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Katie 1" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.00 "Katie 2" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.01 DEBUG: Running command "avs2yuv -frames 1 -raw -o NUL "seoul.avs" 2>&1"
2006-12-01~23:46:05.01 DEBUG: avs2yuv.exe in current directory (C:\x264farm\controller)? true
2006-12-01~23:46:05.01 DEBUG: opening avs2yuv
2006-12-01~23:46:10.71 DEBUG: avs2yuv responded: "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013\n"
2006-12-01~23:46:16.40 GOT NORMAL INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:22.15 GOT FIRST INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:28.20 GOT FAST INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:28.21 INFO:
2006-12-01~23:46:28.23 i_res_x = 1280
2006-12-01~23:46:28.29 i_res_y = 720
2006-12-01~23:46:28.57 i_fps_n = 30000
2006-12-01~23:46:28.95 i_fps_d = 1001
2006-12-01~23:46:29.14 i_fps_f = 29.970030
2006-12-01~23:46:29.14 i_num_frames = 208097
2006-12-01~23:46:29.14 i_bytes_y = 921600
2006-12-01~23:46:29.14 i_bytes_uv = 230400
2006-12-01~23:46:29.14 i_bytes_per_frame = 1382400
2006-12-01~23:46:29.14 multiplier for kbps to fbits = 0.029970
2006-12-01~23:46:29.14 fast_res_x = 1280
2006-12-01~23:46:29.14 fast_res_y = 720
2006-12-01~23:46:29.14 IF THIS WAS RELEASED, YELL AT OMION!

Here is the seoul.avs

DGDecode_mpeg2source("G:\captured\Metallica\seoul.d2v",info=3)
ColorMatrix(hints=true)
edeintted = last.AssumeBFF().SeparateFields().SelectEven().EEDI2(field=-1)
TDeint(order=0,edeint=edeintted)
TTempSmooth(lmdiff=1,cmdiff=2,strength=1,scthresh=8)
RemoveGrain(5)
crop( 0, 0, 0, -8)

LanczosResize(1280,720) # Lanczos (Sharp)
#denoise

omion

2nd December 2006, 21:51

Here is the results of the debug lines.
That's odd. It looks like everything works fine, but avs2yuv exits with an error anyway. I'll see if ignoring the output will work...

Now I tried the controller on another pc, and it got a little farther this time, but it still stopped. I tried this with a different avs file along with the simple one. The simple one gave the same error and stopped.

Use the release version (at the beginning of the thread) of x264farm. The debug one I posted was based on my current programming version, which didn't work properly. I just made it exit before it got to the part that broke.

TriHFH

3rd December 2006, 01:53

Thank you :)
Thank you for spending all this time working on it and troubleshooting my problem :)

omion

3rd December 2006, 05:35

Try out this controller (http://omion.dyndns.org/x264farm/x264farm-1.04+2_controller-only.rar); it should actually encode. If it works, I'll put it into the next release. Or maybe I'll just make it the next release.

BTW: what are your system specs, anyway?

OverlordQ

3rd December 2006, 11:31

There a guide for figuring out command line switches for idiots like me :D

omion

3rd December 2006, 21:43

There a guide for figuring out command line switches for idiots like me :D

Well, there's the HTML file (http://omion.dyndns.org/x264farm/x264farm.html), which is included with the releases. However, it's probably a bit too much, so here's a crash-course in command syntax:

The agents don't need any switches, and get all of their options through the config files.
The basic controller command-line is:
conrtoller --first "<first_pass_settings>" --second "<second_pass_settings>" -B <BITRATE>kbps -i "input.avs" -o "output.mkv"

There is, however, a whole slew of x264 options that you shouldn't use in the --first and --second strings. Don't include the stats file, input or output files, or anything which has an x264farm equivalent. If you have any questions about whether or not an option can be included in the --first or --second strings, just post here.

(I just noticed that I don't exactly make clear in the HTML file which options to leave out. I'll have to change that...)

AVmaniac

5th December 2006, 14:09

You really did a great job with creating x264farm.
I am actually loving it.
It is giving my x264 encoding jobs the boost i was waiting for.

But I've one question ... is it possible to use it with Sharktooth's
builds of x264? Because i'm missing the aq-strength flag in your builds.

:thanks: again

omion

5th December 2006, 18:42

You really did a great job with creating x264farm.
I am actually loving it.
It is giving my x264 encoding jobs the boost i was waiting for.
Thanks!

But I've one question ... is it possible to use it with Sharktooth's
builds of x264? Because i'm missing the aq-strength flag in your builds.
Sharktooth's build will not work with controller-based encoding (the default) However, it should work just fine if you give all the computers access to the source files and use agent-based encoding.

The problem with everybody else's builds is that they pipe input in text-mode, whereas I'm sending binary data to x264. Using text-mode basically makes piping data useless for x264 because it will stop encoding if it encounters a certain byte (0x1A).

I would also like Sharktooth's builds to work, because then I wouldn't have to spend time building it :D . I pointed out the problem in this thread (http://forum.doom9.org/showthread.php?t=116700), but as far as I know it hasn't been fixed. If you test out the code I have in the first post on that page and it encodes more than 8 frames, then the build should work with x264farm.

kenji123

6th December 2006, 16:38

love the work you've done omion. just tried it out this morning, and it worked flawlessly. hopefully the issue with Sharktooth's build will be resolved soon.

kenji123

9th December 2006, 15:29

That's odd. It looks like everything works fine, but avs2yuv exits with an error anyway. I'll see if ignoring the output will work...

ERROR: avs2yuv exited with error 1; AVS file "Track1.avs" probably has an error

During the debugging of the x264farm GUI i'm creating, just like TriHFH, i also got this error. it seems avs2yuv must be in the same directory as the AVS file being used.

In the next release could you add a parameter so that the avs2yuv location could be specified, omion.

omion

9th December 2006, 17:26

It will find avs2yuv anywhere on the path (I, personally, don't have it in the AVS directory). But I'll add something to check in the controller's directory, too.

The next version may take a while. I'm adding compression to the controller-based encoding, and it's going to need a lot of debugging.

SpAwN_gUy

11th December 2006, 16:23

So, i've Downloaded everything.. and installed
now i have about 10 PCs Under my control.. COOL.. Yeppie...
After making some preperations.. the Installation of Agent takes about... 30Seconds %) ... and this is WITH agent-based encoding..

at first i've tried to encode 5mins clip.. just for test..
1st went GOOD .. and really Fast..
the second file became Broken :(( (second encode of the same file)
and the third went great again...

and the i've started 4,5Hours encode.. and went Home(that was Friday)..

...Skip...
2006-12-08~23:20:28.87 pc17 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-08~23:20:28.87 pc17 2 close_connection
2006-12-08~23:20:28.89 pc17 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-08~23:20:28.89 pc17 2 close_connection

...SKIP...

2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection
2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection
2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection

....SKiP... and then Again
2006-12-11~08:48:23.14 pc6 3 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-11~08:48:23.14 pc6 1 close_connection
2006-12-11~08:48:23.14 pc6 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-11~08:48:23.14 pc6 3 close_connection

and then My CONTROLLER went OUT of Memory(i think).. and Explorer made some exceptions..
and today i've made a reboot.

BUT.. Agents are STILL working.. and STILL encoding something..

ok.. few Questions:
Can i RESUME Encoding? (i still have batch file.. and still have TEMP files on controller) and i'm doing agent-based encoding
i'm using controller from your last post(where you posted it ;) )

How can i start AGENT in "TRAY-Mode"... or just as Service(it would be nice.. really) ... well that is not a question..

and i'm currently thinking of writing 2 DB clients (i'm a Delphi programmer) .. so it would automate Agent Installation and Controller-Starter with GUI ... (it's not that hard.. to me.. i just need to finish my Job..)

omion

12th December 2006, 01:13

So, i've Downloaded everything.. and installed
now i have about 10 PCs Under my control.. COOL.. Yeppie...
After making some preperations.. the Installation of Agent takes about... 30Seconds %) ... and this is WITH agent-based encoding..

at first i've tried to encode 5mins clip.. just for test..
1st went GOOD .. and really Fast..
the second file became Broken :(( (second encode of the same file)

How did the file break?

and the third went great again...

and the i've started 4,5Hours encode.. and went Home(that was Friday)..

<CODE/>
and then My CONTROLLER went OUT of Memory(i think).. and Explorer made some exceptions..
and today i've made a reboot.

Hmm. Are all of the computers running agent-based encoding? They all have access to the source files, x264, and avisynth? It's odd that the controller would run out of memory with agent-based encoding.

(it's also a bit odd that "socket" was throwing the errors. Normally "send" or "recv" throw them. Check your controller config.xml file for any incorrect IP addresses. It could be that one of them is specified incorrectly, but the short encode didn't distribute anything to them so it worked fine...)

BUT.. Agents are STILL working.. and STILL encoding something..

One of the issues with agent-based encoding is that the controller can't stop the agent from encoding if it dies. The agents will stop after the current encode.

ok.. few Questions:
Can i RESUME Encoding? (i still have batch file.. and still have TEMP files on controller) and i'm doing agent-based encoding
i'm using controller from your last post(where you posted it ;) )

Yup. As long as you use the exact same settings, it will start up where it left off.

If you don't remember the settings you used, look for some files called "settings.txt" in the temp directory. There is one file in the AVS's temp directory, and one file in each of the pass sub-directories. The ones in the pass directories give the settings of that pass. (I hope that makes sense)

How can i start AGENT in "TRAY-Mode"... or just as Service(it would be nice.. really) ... well that is not a question..

Firedaemon (http://www.firedaemon.com/)

and i'm currently thinking of writing 2 DB clients (i'm a Delphi programmer) .. so it would automate Agent Installation and Controller-Starter with GUI ... (it's not that hard.. to me.. i just need to finish my Job..)
Sounds interesting!

One more question:
Are any of your computers running Windows 95/98/Me? They probably won't work with x264farm at all.

SpAwN_gUy

12th December 2006, 09:37

How did the file break?
well. i'm not sure.. i haven't saved it :( but the MPC ended with "can not render file"
Hmm. Are all of the computers running agent-based encoding? They all have access to the source files, x264, and avisynth? It's odd that the controller would run out of memory with agent-based encoding.
Well.. it did it again this Night.. Controller was "Alive" till Half an Hour Ago.. and the same thing started at:
>> 2006-12-12~00:32:47.54 pc661 3 process_job failed with "End_of_file"; putting job back
and it was working since:
2006-12-11~18:16:25.00 x264farm version 1.04-153

today i was using a "Final build" instead of "maybe the next release"

and yeah the IPs - are good(made it from PING). xml-file has some "not started agens IPs" .. so when encode starts not all the agents are working.. and then controller hops all-new agents in line :) ..

Yup. As long as you use the exact same settings, it will start up where it left off.
yeah i've remembered it.. i read it somewhere here.. and it did actuallty continued...
Firedaemon (http://www.firedaemon.com/)
thanks :) .. i was thinking about some "AppToService" programm
One more question:
Are any of your computers running Windows 95/98/Me? They probably won't work with x264farm at all.
Of Cource :) .. a first i was thinking to run an Agent on Vista_Final.. but then i couldn't figure out the IP, because PING didn't answered anything.

omion

13th December 2006, 02:48

Released 1.05.

Firstly, note that you need not overwrite your config.xml files for this update, although I did add a few features which need an updated config.xml file in order to use them.

FPS chart
I added a little chart that prints out occasionally during the 2nd pass, indicating the speed of each agent. I could have done the same for the first pass, but it would have been wildly inaccurate. It looks like this:
Agent | Now | GOPs | Frames | FPS |
Local 1 | 340 | 69 | 7749 | 2.69 |
Local 2 | 342 | 67 | 7850 | 2.82 |
Linux 1 | 341 | 22 | 2184 | 2.48 |
Linux 2 | 339 | 17 | 2236 | 2.52 |
Mac 1 | 329 | 52 | 6456 | 2.55 |
Mac 2 | x | 54 | 6334 | 2.39 |
Mac 3 | 330 | 56 | 6453 | 2.57 |
16.63 FPS total on 39262 / 40492 frames (96%)
ETA: 1:13"Agent" is the name of the agent
"Now" is the current job number (from 0 to the number of scenes) "x" means the agent is not currently working. Since the chart is printed out when an agent checks out a job, at least one agent will always have an "x".
"GOPs" is how many scenes the agent has completed
"Frames" is how many frames the agent has completed
"FPS" is the speed of the agent (based on the times when the agent was actually working, not the total time the controller was working)
The final FPS line is the actual frames per second, and will probably be less than the sum of the previous FPS columns.

Compression
The main change is a beta-quality compression setting, primarily for testing purposes. In order to turn it on, add the following to you agent's config.xml:
<compression>
<type>1</type>
<type>0</type>
</compression>
If you are using the supplied config.xml file, make sure type 1 is above type 0, in order to use compression.

If you do use compression, I have a few requests:

After every encode, check the agent's directory for files with names like "MD5 error 01 (ABCDEF).txt". If there are any there, tell me!
On the controller's side, there should be a bunch of lines like this:01: 34.6% compression (11272631295 / 32551191552 bytes) Post the last line in the encode that looks like this, along with the type of movie it is (animated, computer animated, live-action)

Note that compression is currently DEAD SLOW. This is because I check the integrity of the uncompressed data in case I did something wrong. I'll remove the integrity checks in the next version, which will make it much more usable.

I also added the --nocomp option to easily turn off compression on the controller side without having to change the config files or restart the agents.

Based on my limited testing, the compression is usually ~3:1 for live-action stuff, and ~4:1 for anime.

Agent-based piping
If the agent's config.xml file has a line like this:
<agentpipe>1</agentpipe>that means that, in agent-based encoding, the agent will pipe data to x264 just like the controller does in controller-based encoding. It will keep all the benefits of agent-based encoding, and it lets you use a 64-bit x264 with 32-bit avisynth. See this thread (http://forum.doom9.org/showthread.php?p=766596) for more info about it.

Bugs
There was a bug which caused frames to be given out to multiple agents at the same time, if agent-based encoding was used. 1.05 makes it very unlikely to happen, but it still may. This bug does not affect the integrity of the file at all, just the speed.
The controller now searches for avs2yuv in the controller's directory, which would seem like a logical place to search.

SpAwN_gUy

13th December 2006, 13:17

foxyshadis

13th December 2006, 13:56

32-bit versions run fine on 64-bit windows, they just don't get the small additional gains of pure 64-bit code. If it's 64-bit *nix, it shouldn't be any harder to compile it than a 32-bit version.

omion

13th December 2006, 17:36

Nice.. will give it a try :)

well... i'm not sure about Compression.. only, maybe will give it a try.. for a TestEncode of PotC2 last Titles ~5mins

so.. another question (you are the Guru, i'm only the Encoder :) )
>>>Under my control i have 32bit and 64bit PCs ... so do i really need to find somewhere 64bit version of x264?
okay.. will try.. will it work under x32 windows with x64 processors?

x264farm is only 32-bit, but it will work perfectly on 64-bit operating systems (I have 64-bit Windows here).
If you have 32-bit Windows, then you can only use 32-bit software (the 64-bit CPU is "hidden")
If you have 64-bit Windows, then you may use my 64-bit x264 build. It's about 10% faster than 32-bit. I didn't make a 64-bit x264farm build because that would just waste more memory than the 32-bit build.

Can we use SharkTooth(damn, i can't remember his Nick.. again)'s builds or only Yours?
in Agent-BasedEncodings... i think it is the most convenient way.. 'cause i don't have a Server .. well. i do have some under my controll, but not me :)

In agent-based encodes, yes. For controller-based encodes, you have to use mine (for now... I'll bug somebody into making a patch. I don't like to do that, but a lot of people are asking for Sharktooth :D )

as soon as the Console program will run smoothly.. i'll make a starterGUI :) .. honestly ;)
Great!

SpAwN_gUy

15th December 2006, 10:01

oaky,... well it seems.. that i do have some problems..

controller still hangs when Exiting all the Jobs..("... agent .. DEAD" or something). then it made some kind of an error when it was exactly 50%. and with version 1.05 i'm looking at some strange behaiour: somehow wery strange it continues the Encoding.. currently i've started it for the third time and it is still 1% ... it was more before :(

P.S.: GUI Developement already started (Delphi, Standart components(mostly.. i'm not sure about our XMLParser.. but others are stndrt), PlainText-Files(no DB.. i was advised to do a WebSevice,.. but it is simplier to write PlainText :) ..))

BTW: Agent x64 runs Great on Vista Ultimate :) ...

today or tomorrow i'l borrow DualCore p4 Duo :) ...

SpAwN_gUy

15th December 2006, 11:29

You've requested Compression levels :)
01: 19.9% compression (9147744 / 45909504 bytes)
01: 21.2% compression (14221274 / 67184640 bytes)
01: 23.5% compression (1822839306 / 7761692160 bytes)
01: 23.5% compression (1821968890 / 7758332928 bytes)
TYPE: Soup Opera (X-Files Season 1 first DVD :) )

and i still don't have access to look for Errors.. so.. later.

I have Only 1PC with controller-based encoding... but it REALLY Slows My Controller.. damn.. the Prcentage of CPU Usage is somehow LOW,.. but the Speed of reacting - is terrible.

AVmaniac

15th December 2006, 15:21

omion

15th December 2006, 16:52

after making some tests and having some movies successfully encoded i recognized a strange behaviour:
every winXP based machine running the controller is seperated from the network after doing some work.
I can ping addresses but can not browse any websites or getting my mails ... tried over telnet .. but nothing works ...
the locally installed ftpserver is not accessible...
and no new(preconfigured) agents can connect to the controller..
but after stopping the controller everything works fine again ..
until it was running again for some time....

That's very strange... How long would you say it works before everything gets disconnected?

When you run into the problem next, run "netstat" on the controller computer in a separate command window. There should be a few lines that look like this:
TCP LOCAL-COMPUTER:2170 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2171 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2172 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2173 AGENT-COMPUTER:50700 ESTABLISHED
If there are lots of them that show up, then it could be an issue with running out of usable ports. I ran into this with early versions of x264farm. (It should be fixed, though...)

What kind of firewalls / antivirus do you have on the computers?
Also, what kind of router do you use for your internet connection?

You've requested Compression levels
...
Wow. Those are really good. 4:1 lossless compression.
I have Only 1PC with controller-based encoding... but it REALLY Slows My ControllerYes, it makes encoding go really slow... the next version should be much better.
the Prcentage of CPU Usage is somehow LOW,.. but the Speed of reacting - is terrible.I don't know exactly why this is, but I noticed the same thing on my computer. For some reason compression prevents the controller from using all of the CPU. I'll look into it if it persists after I remove all the debug code.

AVmaniac

17th December 2006, 23:09

That's very strange... How long would you say it works before everything gets disconnected?

I think about 45mins.

When you run into the problem next, run "netstat" on the controller computer in a separate command window. There should be a few lines...
If there are lots of them that show up, then it could be an issue with running out of usable ports.

No, seems as everything is normal .. there are only a few entries listed.

What kind of firewalls / antivirus do you have on the computers?

First also thought of that as problem source and deactivated everything for testing...
I've also reinstalled WinXP (had to for other reasons)

Also, what kind of router do you use for your internet connection?

I am connected to a campus LAN with a 100Mbps port.
It is connected to a 5port GBit switch providing the connection
for my two PC's.

I also have the problem that everything (agent and controller) , don't know if agent or controller is responsible, simply stops working without crashing.

Thanks for your reply omion

SpAwN_gUy

18th December 2006, 10:18

Okay, i've Figured out why my PC Slows down(controller).. it's bacause of te Huge Local Shares Usage(agent-based)

It seems like i'm having the same problem as AVmaniac... 'cause my Miranda stops responding and connecting after a while saying something about sokets..

and , man, there is something strange about 1.05 ... em.. it DOES NOT CONTINUES the ENCODES.. REally... and it hang again.. (on 93%!!!!! damn)..("the job compleded successfully" thing)
well, the compression - is a good thing,... but i think you've made something wrong since 1.04.

and the 1.04 controller does not work with 1.05-Agents :( ...

okay,.. i am stopping for some time.. and waiting for a new version...

AVmaniac

18th December 2006, 11:09

Last night i startet to test x264farm with normal DVB-S content using dgdecode and until now everything went fine, no network problems .. no hanging (curently 63% of the 2nd pass).
Seems as if it has something to do with the DVB-S2(1080i) content i wanted to encode before.
Perhaps it's the decoders fault (i'm using CoreAVC) or a mix of things.

that's all for now but will keep on testing.

Status Update:

the DVBs capture farm encode finished successfully, so it had to be a decoder bug.
To figure it out I installed version 1.2 of corecodec's fine software(I'm a legal buyer) and resumed the HDTV encode
and until now everything runs fine!!!
network ok ... no hanging .. nothing .. just like omion had it in mind to work like !!! :D

Status Update Nr.2:

I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
my useable network connectivity is gone.
But the controller is running and running and ... like the Duracell Rabbit ..
so i've to wait for the controller to finish before being able to use my WinXP machine again for network access.
And here is what putty is telling me when trying to connect to my gentoo machine:

Unable to open connection to
mygentoobox
Network error: No buffer space available

Status Update Nr.3:

everything stopped working again, and because there seems to be no resuming mode for 2nd pass
I've to choices a) waiting for a new version of x264farm
b) keep on trying...

so far, AVmaniac

SpAwN_gUy

19th December 2006, 17:31

and resumed the HDTV encode
and until now everything runs fine!!!
network ok ... no hanging .. nothing .. just like omion had it in mind to work like !!! :D
you know.. after a downgrade to 1.04 - yesterday it made the same thing :) ... too bad tha i've made a wrong Script.. and the DVD is not well made... :(
I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
you know :) .. the same here.. when i'm sitting in front of the PC - everything is Fine :) .. and when i'm leaving my workplace... 3 hours and "Bang - he's dead" :)
everything stopped working again, and because there seems to be no resuming mode for 2nd pass
I've to choices a) waiting for a new version of x264farm
b) keep on trying...
as i've reported i think this is the bug of 1.05 :( ... the major bug... i think. with 1.04 it just continued the encode.. well the percentage was from the beginning, but moved REALLY fast...(normal ~35(-45)fps.. continue ~80fps)
look in your temp-dir .. how many copies of files "100 100-130 ????.mkv" you have? ... i had the number of restarts..

AVmaniac

21st December 2006, 01:04

i've just downloaded it and windows told me that it is no
valid win32 executable.

omion

21st December 2006, 03:36

i've just downloaded it and windows told me that it is no
valid win32 executable.

Yeah. I think I put the 64-bit version in there accidently

[edit: should be fixed now]

omion

21st December 2006, 05:07

I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
my useable network connectivity is gone.
But the controller is running and running and ... like the Duracell Rabbit ..
so i've to wait for the controller to finish before being able to use my WinXP machine again for network access.
And here is what putty is telling me when trying to connect to my gentoo machine:

Unable to open connection to
mygentoobox
Network error: No buffer space available

This looks good (an a bad way... but it shines some light on the issue) The "no buffer space available" error means that you ran out of memory somewhere important. The buffer memory is taken out of the non-paged pool, which means that it must be physical memory, and 32-bit Windows has a hard-coded maximum non-paged pool of 256MB.

If you don't mind trying to repeat the problem:
when the problem occurs, go to the task manager > Performance > Kernel memory > Nonpaged. If it's near 256000 then that could easily be the problem. If not, then it may be a more general memory issue.

I've also been told that the error can also be cased if some internal TCP queue is full, and there are rumors that it may be related to XP's limit on the number of connections per second. Neither of those are likely, as x264farm only makes one connection per client.

What kind of network cards do you have in the computers?

omion

21st December 2006, 05:19

as i've reported i think this is the bug of 1.05 :( ... the major bug... i think. with 1.04 it just continued the encode.. well the percentage was from the beginning, but moved REALLY fast...(normal ~35(-45)fps.. continue ~80fps)
look in your temp-dir .. how many copies of files "100 100-130 ????.mkv" you have? ... i had the number of restarts..

I just got around to verifying this, and it looks like you're right. The 2nd pass does not restart correctly.

I don't have much time to fix it now, as working in retail at Christmas doesn't leave me much time or energy. I'll get to it as soon as I can, though.

SpAwN_gUy

21st December 2006, 15:22

I don't have much time to fix it now, as working in retail at Christmas doesn't leave me much time or energy. I'll get to it as soon as I can, though.
well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

i'm still using 1.04..
If the 1st pass folder is not deleted(some error) it restarts 1st pass again(when restarted) :( ..
can you tell me which files should be in the Temp folder after the first pass?
or what files are deleted after the first pass? :)

and sometimes not exits(Killing at the end) and hangs there(i guess some TTL would be nice)
2006-12-18~23:50:30.25 pc56 2 exiting...
2006-12-18~23:50:30.25 pc661 2 close_connection
2006-12-18~23:50:30.28 pc17 2 attempt_connect sees no jobs left; DEAD

BTW.. if agent is run as Service.. it can only encode in controller-based encoding :(

708145

21st December 2006, 15:37

well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

I have to step in here ;)
There is Handbrake on Linux and ELDER on windows which are rather similar.

omion

22nd December 2006, 05:45

well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

i'm still using 1.04..
If the 1st pass folder is not deleted(some error) it restarts 1st pass again(when restarted) :( ..
can you tell me which files should be in the Temp folder after the first pass?
or what files are deleted after the first pass? :)

After the first pass completes successfully, the final stats file is written to the AVS directory, then the 1st pass subdirectory is deleted.
If the subdirectory is not deleted, the encoding will assume that the file is not written properly, and will continue on the first pass. If all goes well, it will realize that the first pass is in fact done, and will write the stats file and try to delete the temp directory.

Here's how it works, in case anybody was wondering:
The controller will make a directory like:
"temp_root\0123456789ABCDEF0123456789ABCDEF avsfile.avs"
for all the info related to the AVS.

Inside that directory, the first pass will make a directory like:
"1-0123456789ABCDEF0123456789ABCDEF"
which will hold all the files specific to the first pass string that you used.

In the first pass directory, there should be exactly three files:
"settings.txt" is a human-readable file which contains the settings used for this pass
"working_stats1.txt" and "working_stats2.txt" for storing the frames which have already been calculated. In each encoding, one of them is read from and the other is written to. This ensures that there is never a point when there is nothing in the stats file when re-encoding.

Once the first pass is done, the final file written to the AVS directory is:
"1-0123456789ABCDEF0123456789ABCDEF.txt"
where the numbers are the same as the first pass directory. If this file exists and the directory of the same name does not, then the controller assumes that the first pass is done. BTW: This file should be a drop-in replacement for x264's stats file, so you can refer to it if you want to do the second pass without x264farm.

When the second pass starts, you will get another directory in the AVS directory:
"2-0123456789ABCDEF0123456789ABCDEF"
where the numbers are based off the second-pass settings. The first and second pass settings are completely separate, so if you use only change the second pass settings, the controller will see that the first pass is done.

The second pass directory has some number of files in it:
"working_stats1.txt" and "working_stats2.txt" are similar to the first pass, but have a slightly different layout.
"1 - 123 456 abcdef.mkv" are the rendered scenes. Merging together all these files in numerical order will result in the complete video. There should never be more than one file with the same "1 - 123 456" part, but as SpAwN_gUy pointed out, the second pass is doing something wrong when resuming.

Once the second pass finishes, the output file is merged, the second pass directory is deleted, and another file is placed in the AVS directory:
"DONE-0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF.txt"
This file contains the name of the output file. The first 32 letters are the same as the first pass directory name, and the last 32 letters are the same as the second pass directory name. This is so that x264farm can put up that annoying little "This job was already encoded..." message. I really need to put an "encode anyway!" option in there...

and sometimes not exits(Killing at the end) and hangs there(i guess some TTL would be nice)
2006-12-18~23:50:30.25 pc56 2 exiting...
2006-12-18~23:50:30.25 pc661 2 close_connection
2006-12-18~23:50:30.28 pc17 2 attempt_connect sees no jobs left; DEAD

If that's the last line that it printed, then it does look like the entire thing just hung. Normally it would say "pc17 2 exiting..." right after "pc17 2 attempt_connect sees no jobs left; DEAD" I'll try to reproduce it.

BTW.. if agent is run as Service.. it can only encode in controller-based encoding :(
Bummer. That's odd though, as there is very little difference externally between agent- and controller-based encoding, other than the command line that the agent runs. Maybe x264 doesn't like running as a service.

I have to step in here ;)
There is Handbrake on Linux and ELDER on windows which are rather similar.

Yup. ELDER is similar to x264farm, but does Handbrake do distributed encoding? I thought it was just a one-click DVD encoding thing.

omion

22nd December 2006, 05:57

BLEH. I finally found the reason that the second pass is not resuming: I commented out all the code which handles resuming :rolleyes:
I now remember what happened: I was about to re-write the code that handles resuming to be more functional and less imperative. Imperative code tends to be harder to debug, and the second-pass resuming code was nasty. So I commented it all out and was planning to put something in its place when I completely forgot about it...
:stupid:

I try to look through the diffs before I commit the changes, specifically to prevent this kind of thing. I guess I just missed it this time...

1.06 will work! :p

foxyshadis

22nd December 2006, 09:15

Bummer. That's odd though, as there is very little difference externally between agent- and controller-based encoding, other than the command line that the agent runs. Maybe x264 doesn't like running as a service.

Might be a write permissions thing, unless it's running as local system or the user. Network service has very few write rights. Local service has no network rights. And there's always a few programs that don't like non-interactive, but I can't figure out how a bunch of command-line programs would be.

omion

25th December 2006, 20:52

Sorry about the wait. 1.06 is out (Merry Christmas!)

You don't need to update any of your config.xml files if you have 1.05. Just replace the controller.exe and agent.exe files.

Primarily, it fixes the "not resuming on 2nd pass" issue.

There are a few minor fixes, including asking for a confirmation to re-encode the file if it sees that it was encoded before.

The main fix, though, is one I just stumbled across right before I was going to release it on Saturday. This is probably the problem SpAwN_gUy was having with broken files (here (http://forum.doom9.org/showthread.php?p=915266#post915266)).
Apparently, x264 will exit if the requested bitrate is less than the lowest bitrate possible (instead of just overriding the requested bitrate) On long amounts of blank frames, the controller may request a smaller bitrate than possible, and the agent won't recognize the error. This results in an empty file being returned.

The main problem with this is that the MKV header is taken from the first encoded file. If the error occurred in the first file, then the output file won't have an MKV header. This means the entire file will be unreadable.

I now have the controller try not to send bitrates that are too low, and the agent will recognize when x264 exits weirdly.

The bizarre network-based issues (probably) haven't been fixed, but I'll see what I can tweak for 1.07.

deets

25th December 2006, 22:31

does this work with the new threading of x264? could i for example set the threads to auto on each machine?

this would really speed things up. sorry if its an obvious question :)

omion

25th December 2006, 23:27

It will work just fine, as long as you set it up right...

Using the new threading will not give you any performance boost with my x264 build, so if you use controller-based encoding it won't help.

Using agent-based encoding with e.g. Sharktooth's build will work nicely, and it means you would only need to have each computer run one concurrent encode.

There are some performance issues with this, though. Especially during the second pass, the x264 threading won't scale as well as having multiple x264 processes. This is due to the time between jobs when x264 is setting up or sending back the rendered data. It is always a good idea to have another x264 running in the background to fill in the time.

I, personally, would use either x264farm or x264's native threads, not both at the same time.

SpAwN_gUy

28th December 2006, 13:11

Okay.. i've send some LOGS.. to your eMail... and.. i think i'll not have any HTTP Traffic till next Year i guess...

1.06 hangs at closing_Session 's with One file(my current encode,.. previous small went just fine..)
and i'm having something strange with the second file..
on the second pass it can not encode it in the end...
>> 4.81 FPS total on 127459 / 127584 frames (99%)
and i'm getting
>>pc418 2 process_job failed with "End_of_file"; putting job back
or
>>pc6 1 process_job failed with ("The operation completed successfully.\013\n",recv,); putting job back

and then suddenly controller trys to stop(close connections) it hangs.

*offtop* yeah, i've seen "Ani-x264" with x264FARM support.. and i didn't liked it :( .... not so many options.. and no Agent-based encodes.. and no Compression...
and my programmings are not going so well.. i'm not good with free time .. :(

omion

1st January 2007, 07:05

I'm doing a few more tests, and I can confirm a few bugs:

I recreated AVmaniac's problem with the controller computer disconnecting from the network ("No buffer space available"), and I realized that it had happened to me a few times before. I don't know exactly why it happens, but I'll try to tweak some settings to make it not happen anymore.

Also, I think I know what's wrong with some of SpAwN_gUy's files... I'll keep looking into it.

omion

1st January 2007, 22:05

every winXP based machine running the controller is seperated from the network after doing some work.

I think I found the problem (or at least a problem). For some reason there is some kind of file handle leak in the controller. I don't know exactly where it comes from, but on my computer the number of handles increases about 5/second.

This normally wouldn't cause the whole computer to lose connectivity, but it looks like there is a corresponding increase in the number of handles in the "system" process, which could easily cause bizarre issues if they get too large. I think the maximum number of handles for a 32-bit process is 10000...

I'll see if I can fix that somehow.

[edit: more info...]
If you can recreate the bug, fire up the task manager, go to the Processes tab, then turn on View > Select Columns > Handles and see what it says for the "System" and "controller" processes.

Note: there looks to be a workaround here (http://support.microsoft.com/kb/327699), which raises the handle limit. The controller shouldn't need that many handles anyway, so I'll fix it as soon as I can.

omion

5th January 2007, 07:10

Well, I think I found the cause of the controller computer disconnecting from the internet. It's partially Windows' fault, but I am probably the most to blame.

It's my fault because I don't know how to program sockets correctly.
It's Windows' fault because it lets me do it anyway. :p

When each thread from the controller tries to connect to its agent, it starts up a socket. If the connection fails, it shuts down the socket and tries again.
That's apparently the wrong way to do it. I guess after the socket is shut down, it's still a valid handle that has just been thrown away. And the OCaml garbage collector doesn't clean up handles...

This basically means that every 5 seconds one socket is used up per disconnected agent port.

On a Unix-based system, the sockets are filehandles, and the program will die with "too many filehandles open". But on Windows, it just keep on chugging until the OS runs out of... whatever sockets are made out of.

I'll see what I can change around to make it stop doing that. It shouldn't be too hard (assuming that's the only thing wrong... :D )

SpAwN_gUy

5th January 2007, 10:02

:) ... i was pointing on it :) ..... netstst stats - rock :)

omion

5th January 2007, 18:07

:) ... i was pointing on it :) ..... netstst stats - rock :)

Actually, netstat doesn't show the sockets that are causing the problem (which is why it took so long to fix!) The problem sockets are already closed, so they don't show up on any network monitor. But the socket descriptor is still around, which makes a problem for the OS when it runs out of them. The only place I found them listed is in the task manager under "handles".

I actually don't know what caused the large amounts of half-closed sockets that you sent me. Do you remember which encoding you were doing when you used netstat? Was it associated with one of the log files you sent?

BTW: some good news (at last)
I found that you actually don't need to map a network drive in order to get agent-based encoding without copying anything.

If you give a line like one of the following into the <base> part:
\\192.168.1.100\shareddir\movies
\\COMPNAME\shareddir\movies
\\COMPNAME\shareddir\
it will connect to the directory which is shared as "shareddir" on 192.168.1.100 (or whatever computer is named COMPNAME). If you want the shared directory itself to be the base, you MUST add a trailing backslash. If you want a subdirectory of the share to be the base, you MUST NOT use a trailing backslash. (like I did above)
You can see if it works by looking at the output right after the agent starts up.

omion

6th January 2007, 03:31

1.07's out. The disconnecting problem should be solved.

Note that you only need to update the executables, not the config.xml files.

I had planned a couple additional features for this release, but they ended up being WAY too hard, so it's basically just a bug-fix now.
I was going to put in a way of giving out a job to multiple agents at the end of the encode, just to make sure a dead agent doesn't stop the pass from finishing. But that turned out to be immensely difficult the way the program's currently structured.

PS. I found out that you can actually run entire programs through shared directories in the command prompt. This means you can type something like this:
\\192.168.1.100\x264farm\agent --config \\192.168.1.100\x264farm\config-agent.xml
and just have to update one agent and one config.xml.

SpAwN_gUy

29th January 2007, 12:11

I'm Back Again :) ...
new version - COOL.. i'll start testing it now..
with those advices(about netshares) - COOL Again :) ..

Okay.
so.. those Advices with netshares.. it works :) ... but..

//for the future generations
Methods with netShares works ONLY when:
agentpipe = 1
Nice(with .dll), avs2yuv, x264.exe ARE in the Folder of .bat-File(netStart I guess)..
well.. x264.exe is no necessarily(written probably with mistakes ;) ) must be in the folder of .bat-File..
<x264>\\PC201\x264farm\!AGENT\x264.exe</x264>
still works :) ...

so.. now No need to UpDate agents .exe's with new versions.. and lots of problems are gone :) ... and no x264.exe updates ... cool :) ...

omion

30th January 2007, 06:42

Hmm... that's a bit odd. Here's my setup (which I tested everything on)
Agent and config file on remote computer. Started with:
\\192.168.1.100\path\to\agent --config \\192.168.1.100\path\to\config-remote.xml
Config file contains the following:
<config>
<temp>C:\Documents and Settings\Omion\My Documents\x264farm_temp</temp>
<port from="50700" to="50703"/>
<x264>x264-614_omion.exe</x264>
<nice>0</nice>
<bases>
<base>\\192.168.1.100\DVD\MKV</base>
<base/>
</bases>
<agentpipe>0</agentpipe>
</config>

avs2yuv doesn't exist on the agent computer (not needed since agentpipe=0)

x264 is somewhere on the path (not quite sure where anymore... but it's not in the agent directory)

It works just fine with agent-based encoding, too.

BTW: There haven't been any updates in a while because I've been completely redesigning the first pass.

The old implementation would just split the movie into a bunch of chunks and send them sequentially to the agents.
The new one will attempt to keep the agents as separate as possible, and will try to start agents right after they left off. The agents will also try to avoid anything that could keep everything waiting on one agent to finish (although that's impossible to do all the time)

The improvements should make the very end of the first pass much faster. Depending on the file it could decrease first pass encoding time by ~5%.

I suppose I should get around to updating my x264 builds, too... I was hoping that they wouldn't be needed by now, but they still are. I'll build them soon.

SpAwN_gUy

5th February 2007, 11:56

hmm...
i've noticed <bases> part.. does it work in 1.07? ;)

there are still some problems with minimum bitrate sended to x264(i've tried the same encode on different bitrates and different firstpass settings(noticed that during controllerGUI Developement in past few days... it can now import jobs from MeGUI)).

and.. my people noticed(me too), that when agent is run through the network it sometimes crashes.. (i haven't found the reason), 'cause even when hostPC 've been rebooted agent were still up and running.. dunno why this happened, really.

omion

6th February 2007, 06:08

hmm...
i've noticed <bases> part.. does it work in 1.07? ;) I haven't had any problems with it. Try it out!

there are still some problems with minimum bitrate sended to x264(i've tried the same encode on different bitrates and different firstpass settings(noticed that during controllerGUI Developement in past few days... it can now import jobs from MeGUI)).Yeah. I've been busy on another program lately, but I'm probably going to have another version of x264farm out by Wednesday (California time :D ). I think I can fix the minimum bitrate thing fairly easily, but...
and.. my people noticed(me too), that when agent is run through the network it sometimes crashes.. (i haven't found the reason), 'cause even when hostPC 've been rebooted agent were still up and running.. dunno why this happened, really.What does it do when it crashes? Does the whole program exit, or just refuse to encode anything?

SpAwN_gUy

6th February 2007, 08:52

I haven't had any problems with it. Try it out! i think i will :) 'cause i'm a bit short in HDD-Space..
Yeah. I've been busy on another program lately, but I'm probably going to have another version of x264farm out by Wednesday (California time :D ). I think I can fix the minimum bitrate thing fairly easily, but...
Looking forward to it :)
What does it do when it crashes? Does the whole program exit, or just refuse to encode anything?well, it simply exits,.. and that's all..after doing some job.. (i'm not really sure).. sometimes suddenly after 2 hours of working,.. sometimes after 6 hours idle.. i don't know the reason...

omion

6th February 2007, 09:04

well, it simply exits,.. and that's all..after doing some job.. (i'm not really sure).. sometimes suddenly after 2 hours of working,.. sometimes after 6 hours idle.. i don't know the reason...
Hm. Weird. I'll test it out and see if I can reproduce that. It's possible that a network hiccup (like a quick disconnect) would make it panic, but I don't know.

SpAwN_gUy

6th February 2007, 10:49

Hm. Weird. I'll test it out and see if I can reproduce that. It's possible that a network hiccup (like a quick disconnect) would make it panic, but I don't know. the THING is that.. I ReBOOTed my PC (for 3 mins or so) and NONE of 2 working agents EXiTED... so, it may be problematic...
i'll try to find out the reason here.. but..

omion

6th February 2007, 20:46

Oh... THAT'S why my calculations for the minimum bitrate don't line up with x264. From ratecontrol.c:
/* FIXME: ugly padding because VfW drops delayed B-frames */
rc->num_entries += h->param.i_bframe;
x264 adds a couple of frames, and initializes them to a relatively large number of constant bits, which raises the minimum bitrate, which causes x264 to flip out for blank scenes.

I'm going to see if removing that line does any harm...

omion

8th February 2007, 06:47

1.08 is out! At last!

I completely redesigned the first pass to hopefully make it more efficient. The controller program should use less CPU power, less RAM, and will get finished faster than before. (How much faster depends on the movie and encoding settings, though. Very slow, dark movies will improve the most)

The second pass is a bit more error-resilient. Before 1.08 there was a problem with scenes that were very short and entirely black, which I mentioned in my previous post. 1.08 may still report some failed encodes, but everything should get done eventually. I wrote akupenguin about the possibility of removing the line I mentioned before from x264.

The syntax and config.xml files have not changed; you only need to replace the binaries.

SpAwN_gUy

9th February 2007, 10:30

Nice... TESTiNG in Progress :)

whatever-ever

13th February 2007, 20:08

I need to configure my setup a little better (possibly it's my avs that's causing problems).

Currently the avs is located on controller only, and loading plugins via network (even though it's on the local machine), and d2v is linked to locally (not via network). It is a complex as it sounds, but it was the editing in the avs to that, which made the external agent catch on.

wierd thing is it reports a lot about d2v file not being found (on external) which is obviously because of the pathing you might say, but no... i tried both \\network\dir\ and C:\local\dir\, same error -- HOWEVER, the controller reported an increasing boost (going from 1 hour to 30min to 4 (!) min ETA).
And get this: the boost came when i connected the external agent on wifi!!

Since I got it working kinda "half-ass", changing avs while encoding (when restarting the external agent), I was wondering if someone could help me set up the avs properly.

Current setup:

controller config:
<config>
<temp>C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller\temp</temp>
<agents>
<agent name="LOCAL">
<ip>127.0.0.1</ip>
<port from="50702" to="50702" />
<number pad="1">1</number>
</agent>
<agent name="LAPTOP">
<ip>10.0.0.7</ip>
<port from="50701" to="50701" />
<number pad="1">1</number>
</agent>
</agents>
</config>
LOCAL agent config:
<config>
<temp>C:\!RAW\__x264farm\x264farm-1.08-161_win32\agent\temp</temp>
<port from="50702" to="50702" />
<x264>x264.exe</x264>
<nice>5</nice>
<base>C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
<agentpipe>1</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>
LAPTOP agent config (external):
<config>
<temp>C:\Documents and Settings\****\Desktop\x264farm\agent\temp</temp>
<port from="50701" to="50701" />
<x264>x264-622_omion.exe</x264>
<nice>5</nice>
<base>C:\Documents and Settings\****\Desktop\x264farm\agent</base>
<base>\\CONTROLLERPCNETWORKNAME\\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
<agentpipe>1</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

AVS (C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller\test.avs):
SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("C:\!RAW\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

The controller cmd line:
encode.bat:
controller.exe --avs %1 --bitrate %2kbps --first "--crf 19 --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 5 --ref 8 --me umh --bframes 7 --weightb --b-pyramid --analyse all" --second "--trellis 2 --bime --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 7 --ref 12 --me umh --bframes 7 --weightb --b-pyramid --analyse all --8x8dct" --output "output.mkv" --nocomp --batch 2500 --split 250
executed as:
encode test.avs 1224

The AVS is only located in the controller folder.

The goal: to encode agent-based.

I know it's a bit wrongly set up (the avs part, right?) but it kind'a worked.

First pass took about 50 mins to encode (don't know what the ??? went wrong there..) and second pass about 5 minutes.

Thanks in advance for any help.

PS: Thumbs up, omion :) I've been looking for a project as such for a long time. Thanks!

omion

13th February 2007, 20:34

ETA: 1:51:34How long is the movie? How long would it have taken with the same settings without using x264farm?

and that's on 6%, oh my.. GOD it's slow. ETA is rising for every pct. the encode is increasing. And this is the second pass. The first pass was not that fast either.

The setup is one controller + one agent on SAME machine (x264 running 50%)

I need to work a bit on getting my laptops working in the setup too, but even though it's only 50% locally it shouldnt be THIS slow. I must have done something wrong. I'll post configs and dump.txt later

That would definitely help.

Using x264farm on one computer is always going to be slower than not, simply because of the extra overhead. Also, x264farm will re-process any scenes that varied wildly from what was expected, which will result in a higher quality but which may take 10% longer than not using it.

A few more questions:
Are you using agent-based encoding? If the agent says something like this then you are:
Testing file "file.avs": FOUND!
2 agent-based encoding

What is your CPU usage like? Is it maxed out? You said x264 is using 50% of your CPU, but is something using the rest? (also, is it a dual-core?)

whatever-ever

13th February 2007, 20:55

How long is the movie? How long would it have taken with the same settings without using x264farm?

11min clip and normally it's 25fps for first pass, 11fps for second pass. so about 25 min.

Are you using agent-based encoding? If the agent says something like this then you are:
Testing file "file.avs": FOUND!
2 agent-based encoding
yes I am, I've updated my post to make it more specific, the first was embaressing, sorry.

What is your CPU usage like? Is it maxed out? You said x264 is using 50% of your CPU, but is something using the rest? (also, is it a dual-core?)
It's hyperthread, but not dualcore per say. Yes, 100% cpu usage during most of the first pass (not sure during second pass, it went very fast so I didnt get to check). 50% x264, 50% avs2yuv, BUT, x264 doesnt exceed 50%, and avs2yuv didnt run all the time (for quite some time cpu usage was only at 50% total).

I've updated with specs on my setup if it helps,

EDIT: laptop cpu usage is roughly 20-35%

thanks for the quick reply

omion

13th February 2007, 22:11

I need to configure my setup a little better (possibly it's my avs that's causing problems).

Currently the avs is located on controller only, and loading plugins via network (even though it's on the local machine), and d2v is linked to locally (not via network). It is a complex as it sounds, but it was the editing in the avs to that, which made the external agent catch on.
That is a bit odd, but sometimes it's necessary. However, I don't think that the avs will work on remote computers. It's best to use relative paths wherever possible.

For example:
SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("..\..\..\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

And make sure the .d2v file references its source files relatively too; when you create the file, turn OFF "Use Full Paths". If you open the .d2v file with a text editor, the VOB files should be referenced relatively.

wierd thing is it reports a lot about d2v file not being found (on external) which is obviously because of the pathing you might say, but no... i tried both \\network\dir\ and C:\local\dir\, same error -- HOWEVER, the controller reported an increasing boost (going from 1 hour to 30min to 4 (!) min ETA).
And get this: the boost came when i connected the external agent on wifi!!
Uhh... so it can't find the d2v file, but it works anyway? I must say, this doesn't make a whole lot of sense. Care to explain this a bit more? Also, were those times that you posted the controller's ETA or the actual time?

Since I got it working kinda "half-ass", changing avs while encoding (when restarting the external agent), I was wondering if someone could help me set up the avs properly.

If you can get any encodes out of an agent, then you have set up everything correctly. The relative paths I mentioned before will ensure that the avs file will work on remote agents.

The goal: to encode agent-based.

I know it's a bit wrongly set up (the avs part, right?) but it kind'a worked.

First pass took about 50 mins to encode (don't know what the ??? went wrong there..) and second pass about 5 minutes.

Thanks in advance for any help.

PS: Thumbs up, omion :) I've been looking for a project as such for a long time. Thanks!

Looks pretty good. However, I noticed that you have two <base> lines in your laptop config. If you wanted to have the option of encoding from either of them, you need to use the <bases> element, like so:
<bases>
<base>C:\Documents and Settings\****\Desktop\x264farm\agent</base>
<base>\\CONTROLLERPCNETWORKNAME\\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
</bases>

However, judging by your paths, the first one (to x264farm\agent) probably won't be used to store any avs files.

whatever-ever

13th February 2007, 22:58

That is a bit odd, but sometimes it's SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("..\..\..\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?

[...] make sure the .d2v file references its source files relatively too

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

Uhh... so it can't find the d2v file, but it works anyway?
it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.

judging by your paths, the first one (to x264farm\agent) probably won't be used to store any avs files. Thanks! It also loads the network path each time (only place there is a avs), so you're right, I should just delete the other.

PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! :))

omion

14th February 2007, 00:58

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?
All those ..\..\.. mean to go from the .avs directory (C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller) to the directory of the .d2v file (C:\!RAW).

If you have the C:\!RAW directory shared somewhere, then the relative path will still make it point to the right directory.

Think of it this way: the way you had it, your laptop is seeing the .avs file refer to "C:\!RAW\test2.d2v" Does that path exist on the laptop? If not, then you have to make sure you share the proper path.

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

If you are sharing the directory "C:\!RAW" as "\\NETWORKNAME" then yes.

it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.
I thought so. ;) Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.

PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! :))
Yup. I noticed that you had the <agentpipe>1</agentpipe> line in your config.xml files, which needs avs2yuv to function. <agentpipe> is only needed if you have a 64-bit x264, but 32-bit avisynth. If you have 32-bit everything then <agentpipe> will only add a layer of complexity.

Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore :cool:

whatever-ever

14th February 2007, 08:17

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.
[...]
I thought so. ;) Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.
[...]
Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore :cool:

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).

I'll shut up and run a test :)

THANKS!

SpAwN_gUy

14th February 2007, 11:40

:) .. maybe i'll write a tutorial.... or maybe not ;) ...

so,.. i've come back with the tests of 1.08..

AWESOME!!!!!!!!!!!!!
i think you've modified x264... 'cause it RUNS pretty well,.. :)) ..
i've encoded that problemous file... but i've messed up with bitrate, so the file came oversized.. now i'm ReEncoding it..

minuses: agents are suddenly exiting(started over network). I have DUMP...
controller suddenly hanged yesterday.. on exiting first pass.. i dunno why. it was running smoothly, and when i left - something happened(i don't have dump..)

and i've made a some kind of contest.. calculated sum of fpses of agents.. (2x and 3x).. 2x =( 1, pad1) 3x = (2, pad1)
yeppie... now i have one CoreDuo2 !!!!!.... it's freaking awesome.. (and two or three are coming)
Agent | FPS
p4 Core Duo2 E6400 2.13GHz/2Gb/W32bit 3x | 7.05
p4 3000/1Gb/W32bit 2x | 3.05
AMD 3000+/1Gb/W64bit 2x | 2.86
AMD 3500+/1Gb/W32bit 2x | 2.83
AMD 3700?/1Gb/W32bit 2x | 2.67
p4 3000/1Gb/W32bit 2x | 2.23
p4 3200/1Gb/W32bit 2x | 1.48
p4 1800/512/W32bit 2x | 0.89
13.09 FPS total
now it' running ~16fps

now.. adout gui..
i'm a bit confused and lost in translation... what ARE the Assorted params??? I should never use them in passParams, or controller DELETES them from pass params?? .. and do i need "--threads 2" and "--thread-input" in passParams?

SpAwN_gUy

14th February 2007, 12:11

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).
the simpliest way to test will agent-based encoding run or not ("is everything OK? and will it run??") is to Open desired AVS from the Agent Computer(at least one computer) though the network.. if it runs - then OK.. it will..(well, about 30% for sure).

on each agent PC you will need AviSynth Installed.

ok, ok,... the Tutorial ;) :

dir structure:
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\file.d2v
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs
.vob are in:
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\iN\VIDEO_TS\
AviSynth plugz are in
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\filters

in the network paths it looks like:
\\controllerpc\x264farm\file.d2v
\\controllerpc\x264farm\fileAVS.avs
.vob are in:
\\controllerpc\x264farm\iN\VIDEO_TS\
AviSynth plugz are in
\\controllerpc\x264farm\filters\

.d2v internals (the head of..) should look like this
UPD..(2007-10-02): Don't forget to download DGIndex 1.5.0 beta 3 .. it has "Relative paths saving"
here (http://forum.doom9.org/showthread.php?p=1050990#post1050990)
DGIndexProjectFile13
5
iN\VIDEO_TS\VTS_01_1.VOB
iN\VIDEO_TS\VTS_01_2.VOB
iN\VIDEO_TS\VTS_01_3.VOB
iN\VIDEO_TS\VTS_01_4.VOB
iN\VIDEO_TS\VTS_01_5.VOB

Stream_Type=1
......

.avs is
SetMemoryMax(32)
LoadPlugin(".\filters\DGDecode.dll")
# SOURCE
mpeg2source("file.d2v", info=3)
...... etc

agent Config bases part lokks like
<bases>
<base>\\controllerpc\x264farm\</base>
<base></base>
</bases>

and i start the controller with
controller.exe --first "--bframes 3 --b-pyramid --direct auto --filter 1,1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-psnr --no-ssim" --second "--ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-psnr --no-ssim" --avs "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs" --output "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileOutput.mkv" --bitrate 1375kbps

all agents are encoding agent-based :) ...

BTW... just now 2nd pass finished..(first pass was about ~20fps)
START: 1000AM
END: 1307AM
this encode took about 20 hours to encode on my PC..

now i'm encoding one more time.. and i think most of agents - exited :( ... first pass is only using 3 of manyPC's.. :(

whatever-ever

14th February 2007, 15:01

Thanks for the tutotiral Spawn!

I'm not sure what did the trick, but LOCALLY it's going great now. Encoding with 20.7 + 13.16fps on the two local agents, VERY fast (same as when encoding without farm).

And the output is "better" on the local agents.
Reporting "encoded frames: 103/187" and updating continuously.

It might be the controller cmd line that's doing the improvements, but I'm not sure.

However I can't get the external agent to follow.
Getting a lot of Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255\")

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Now it's encoding at 0.44 and 0.57 locally and 3-4fps externally, okay... This is getting wierder and wierder!

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1 :)

SpAwN_gUy

14th February 2007, 16:08

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1 :)
it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw... What is "vdvmax"??

and... temp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

whatever-ever

14th February 2007, 16:23

it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw. twmp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

PS: exiting doesn't seem to be a problem - anymore. Don't know why, but all jobs are running smoothly, just not very fast.

whatever-ever

14th February 2007, 18:05

btw... What is "vdvmax"??

vbv ;)

omion

14th February 2007, 20:31

When you guys say it's exiting, does it put up any warnings or just die?

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

There is no way to tell x264farm to not do the second pass, but you can stop it manually and use the stats file it creates to do a separate second pass.

The created stats file is here:
<controller_temp_dir>\<lots_of_numbers> <name_of_avs>\1-<lots_of_numbers>.txt

For example,
E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs\1-4E2586CB8D8815243D6B8F08CE7FEA77.txt

Exactly which string of numbers goes with which encoding is displayed at the beginning of x264farm:
AVS temp dir is E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs
First string is "...", with MD5 "4E2586CB8D8815243D6B8F08CE7FEA77"

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Uhh... I think it's not encoding fine. Without avs2yuv, there is no chance that agentpipe=1 is going to encode at all. With agentpipe=0, the existence of avs2yuv won't matter.

I'll have to look at the dump file to say more, though.

whatever-ever

14th February 2007, 21:41

Setting up a mac, where do I get the x264 encoder for that?

whatever-ever

14th February 2007, 21:45

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

Yes all machines are 100pct. avs2yuv is taking a lot of cpu usage sometimes though (up to 50 pct.), however agents (not running avs2yuv) are slow too (not as slow, but still.. not so impressive)

I've saved the out-dump, but only from the controller, I'll check if it's usable and mail it.

omion

15th February 2007, 02:33

Setting up a mac, where do I get the x264 encoder for that?

To build it yourself, you need x-code (on the DVD that came with the mac) and SVN (over here (http://metissian.com/projects/macosx/subversion/))

Check out x264 from the repository:
svn co svn://svn.videolan.org/x264/trunk x264
Go to the x264 dir and type:
./configure
make
sudo make install
Assuming everything is set up right, that will get you the x264 binary.

You'll also need the agent for the mac. You can get 1.08 here (http://omion.dyndns.org/x264farm/x264farm-1.08-161_agent_intelmac.zip). I didn't put a config file in the archive, but it's set up the same way as in Windows.

SpAwN_gUy

15th February 2007, 09:31

When you guys say it's exiting, does it put up any warnings or just die?
well.. few people said that it shows something, but they didn't remembered that.. :(

and.. x264 takes about 30% of CPUtime..
i'll do some testings about agentpipe=0.. and will send logs to you..

...here we go
firstpass
pipe=1 nice=20 - avs=80% x264=27% 15.5fps
pipe=1 nice=0 - avs=50% x264=50% 16fps
pipe=0 nice=20 - error
pipe=0 nice=0 - x264=98% 17fps

secondpass
pipe=1 nice=20 - avs=10% x264=84% 4.3fps
pipe=1 nice=0 - avs=10% x264=85% 4.11fps
pipe=0 nice=20 - error
pipe=0 nice=0 - x264=98% 4.05fps

fistpass-local priority: normal
x264=72% 22fps

secondpass-local priority: normal
x264=80% 4.5fps
agent is p4 1800/512/W32bit
local(controller) is p4 2400/512/W32bit

... he-he... dump does not provide anything :) ... here are the screen dumps(theese were not provided in eMail):
Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt" --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

omion

16th February 2007, 04:00

Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt" --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

Oh! It looks like nice.exe does not appreciate running things from network paths. I'll see what needs to be done.

@whatever-ever:
I'll take a look at those logs you sent me. There should be something I can do about the speed.

SpAwN_gUy

16th February 2007, 09:41

yeah,.. and..
it seems, that x264farm has something wrong with Oversize :( .. i had 2 encodes of the same source(with different x264profiles), with same bitrate, and the files were
a) different in size (~30Mb)
b) they were oversized.. about ~60Mb to ~93Mb...
why?

omion

18th February 2007, 03:51

@SpAwN_gUy:

Well, the files are going to be different, since the scenes never finish encoding in the same order. They should not be that different, though.

It also looks like my bitrate-multiplier calculation is not quite right... There is not enough feedback to correct for any bitrate discrepancy, so x264farm is always trying to pull the bitrate up. I'll try to fix it in the next release (or at least make it better)

To fix it in the short-term, you can increase the number of 3rd-pass scenes.
The settings that control the third pass:
--3thresh How far off the bitrate must be in order to consider for 3rd pass. 0 = completely wrong, 1 = exactly right. Default = 0.8
--3gops The maximum number of scenes to consider for the 3rd pass. Default = a lot :p
--3ratio Same as --3gops, but as a percentage of the total scenes (well, not really percentage, but from 0 to 1) Default = 0.05.

The default parameters will re-encode up to 5% of the total scenes, but only if the scene's ratio expected_bitrate / actual_bitrate is less than 0.8.

To make the bitrate more exact, raise both --3thresh and --3ratio. Something like --3thresh 0.95 --3ratio 0.2

Meanwhile, I'll try to make the ratecontrol smarter.

whatever-ever

18th February 2007, 11:54

My speed was way up on a test I ran this friday. I'm trying to run another, but I'm having a little trouble with the avs somehow. Looking into it now

SpAwN_gUy

19th February 2007, 16:54

so.. how the loop goes(or a hang)
controller hangs on close_connection ... of agents at the end of second pass..
and after the restart.. it starts(or continues) to reEncode something.. it is 99% of an encode...(with modified 3tresh and.. as omion advised).. it happened 3 times for now.. (yeah i know... I have Lots of those errorous sources and everything is going crazzyy.. but i think it is good in terms of testing.. ;) )... so.. and keep going again and again... i'll modify .bat file.. and post some results tomorrow

//me sorry.. i'm a bit drunk right now.. but i'm allmost Russian(Belarussian.. telling the truth), so i can work and programm..
soory my girl Friend waits for me near work entrance.. so.... ;) ..

SpAwN_gUy

21st February 2007, 09:11

Yeah... and Is it possible, to run NOT Only the x264.exe with nice, but avs2yuv also?

'cause if agentpipe=1 and nice=20 then x264 is running in lowest priority, but avs2yuv is running Normal, and it slows down overall system performance,.. so my people can't work :( ...

and.. that option,.. that asks for "Do you really want to ReEncode the same AVS?"(or something)..
can you check for th SAME Bitrate and Same Filename? .. 'cause i had an encode with different bitrate and different output filename... it was nice, that i didn't need to make firstpass again,.. but..
/*played with nice..
well.. nice REALLy does not like NetPaths.. i've played with different variations of comLine.. and nice does not want to start avs2yuv with netPath param..
error is: "AviSynth can not find the file D:\controllerpc\x264farm\...."
and batch file was: nice -n 20 avs2yuv ... "\\controllerpc\x246farm\.avs" | nice.. x264...

theturtle

25th February 2007, 09:44

hrm, I can get this working on local worker but the 2nd machine I have doesn't appear to want to encode anything. i'm still plugging away at it. the network paths and setup was a tad difficult, but I think i've got that all worked out.

for some reason right now, the remote agent i have, receives a command to encode. yet it doesn't appear to be encoding.

I did have a question. regarding agent based encoding. If you are doing this, can the input video in the avs be on a network share for remote agents? Both my agents are using the same .avs and I just have the AVS looking for the input video on a Network drive I mapped(which is actually a local share on the controll/local agent machine). I guess thats not really a good question, I guess since they both use the same avs, they both have to have the same access to the input video and the path has to be the same, otherwise it won't work.

Anyhow, not sure whats going on with this remote agent, I can't get it to actually encode anything for nothing. The second pass will be up in an hour or 2, perhaps it will have better luck on that pass. :s

*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.

theturtle

25th February 2007, 15:14

omion

25th February 2007, 18:55

i thought i'd post this separately, i just finished an encode with 2 computers local agent and remote agent. the resulting encode has flashes of grey every few seconds or so. I thought at first this was during scene changes but its not. Its like a blank spot in the video or something. Perhaps made during merging process? :s

Are there entire frames which are gray, or does it seem to start in the middle of one frame and end in the middle of another?

Also, you can find out where the gray occurs by doing the following:
Turn off all the agents except for one (in the config file, or just don't start the other agents up)
Run the controller with the --keeptemp option (and probably also --frames xyz and --restart to make it fast and repeatable)
If the resultant file does not have the gray frames, then the problem is with another agent
If the resultant file does have the gray frames, then you can check the temp directory and see if the raw pieces have gray frames. If they don't then it's probably the merger that's doing it.

Also, just to make sure, the initial AVS file doesn't have the gray frames, right?

@SpAwN_gUy:
Sorry about the delay in fixing nice. I don't really have a lot of time lately, but I should get it done by Wednesday this week

AVmaniac

25th February 2007, 20:54

theturtle

26th February 2007, 02:24

*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.

^^^^ regarding this. For some reason if I set the <base> to my network drive that I mapped, my remote agent executes x264.exe but yet cpu is at 0%. If I copy the avs to the remote agent's agent-win32 directory, it appears to encode fine. Although I've just noticed, my source is h.264 mkv. I didn't have either ffdshow or haali media splitter installed on my agent. Should I need these? I figured the input to the agent was yuv from the controller.

*update* I just installed both of these and it appears its encoding, the network utilization skyrocketed from my encode last night. it was hopping around 20-50%, now its about 99% steady. :) perhaps that was my problem. hopefully it doesn't produce the same gray frames. we'll see. mooore testing ;)

I don't think it's a bug of the merging part.
Try to play the script with the WMP.
The grey frames should also appear here!
Did you try to reinstall your avisynth or tried a different version?
That's what helped me getting rid of this bug...

greetings to all x264farmers %-)

just talked with someone about this, they seem to be pretty confident its an issue with directshowsource. my source is h.264, i'm going to try the new avc/h.264 avs source plugin. I was getting blank gray frames in the source with directshowsource btw, so its definitely not a x264farm issue.

mpioner

26th February 2007, 12:08

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.

theturtle

26th February 2007, 14:10

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.

tried this?
http://forum.doom9.org/showthread.php?t=117428&highlight=x264farm+gui

dunno how well it works, the cli version took a bit of figuring and alot of mistakes on my part, but wasn't too difficult.

SpAwN_gUy

5th March 2007, 09:54

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.i'm playing around with Delphi... for like some months.. and it seems like a testing version will be shown in few weeks.. i guess..(well it's all unofficial)..

agentGUI has an UnKILLable option (if agent crashes it is auto-Restarted) and can minimize to tray.
controllerGUI has "Import MeGUI Jobs" button, and actually imports them,.. and generates command lide for starting controller.. :) ... that is for now,.. lots of ideas, and no time at all..

@ALL
is the Tutorial i wrote usefull? or maybe...??

omion

11th March 2007, 08:32

1.09 is out (or will be shortly, when my server reboots...)

It should be more accurate with the bitrate target, and the agents now use nice for avs2yuv (if both niceness and piped input are used)
I didn't add bitrate in the same-file detection (because I only remembered it just now)

Also, the agents will by default delete any temp file which is older than 1 week. I noticed my temp directory filling up with a bunch of junk after testing x264farm, and I decided to do something about it.
By default, a cleanup thread will wake up once a day and delete all the temp files that are over 1 week old. The amount of time is adjustable with the --stale <x> command-line option. <x> is the maximum age that a temp file can be, in seconds.

The release took so long because I've been trying to rewrite the nice.exe program. I think 1.1MB is way too big for a program like that, and the fact that it can't run network things is not so good either. However. in order to easily access process priorities I have had to use C/C++ (which I don't really know) and the whole thing ended up being extraordinarily frustrating and a huge waste of time. So now, instead of a new, slimmed-down nice.exe program, I have a reminder of why I program with OCaml in the first place :rolleyes:

SpAwN_gUy

12th March 2007, 14:36

Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't :( ... well.. it's not working now at all..

agentpipe=1
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
2 agent-based encoding
2 doing the following:
nice -n 20 avs2yuv -raw -seek 0 -frames 5223 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats 41425d.txt" -o NUL - 576x432
Avisynth error:
Import: couldn't open "D:\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"

agentpipe=0
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
7 agent-based encoding
7 doing the following:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --directauto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\PC201\x264farm-2\xFiles\S1p2\S1p2.avs'

7 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

i've downloaded archive woth nice once again,.. and i think it have not being changed..

i guess.. i'm going back to controller-1.09 and agent-1.08... waiting for nice, only :) ...

maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?

yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...

Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising :)
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool :) ...
37fps on start of second pass.. :) and rising :) ...

i think. today is a good day,.. i'm happy :) really :) ..

omion

12th March 2007, 19:54

Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't :( ... well.. it's not working now at all..

Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. :angry: I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.

I'll see what I can do about that program.

And just in case you were wondering:

maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?

I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.

yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...
Once everything starts up there isn't a difference, but the topmost agents will get the first splits, resulting in a faster startup.

With a lot of computers (like your setup) I can believe that that makes a big difference.

Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising :)
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool :) ...
37fps on start of second pass.. :) and rising :) ...

i think. today is a good day,.. i'm happy :) really :) ..
Good to hear! Thanks!

SpAwN_gUy

13th March 2007, 12:33

YEPPIE.... i've figured all out :) ... i just remembered, when i was trying to run agent as service, there was an advice..(at the end of the post,.. add some mystery ;) )
And just in case you were wondering:
I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.
and i just played a bit with delphi and i could writa a programm, that starts something(i dunno how will it work and what do you need from that starter).. it is now 40Kb .. i'll play a bit later and will post some results..
it is ConsoleApp and uses WinAPI (CreateProcess with Priority flag set).. and i've found in old winSDK only four priorities.. IDLE, NORMAL, HIGH and REALTIME... and i dunno wich are usefull :) ..

anyway results -> later.

--and-now--------------------------

Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. :angry: I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.
testing testing,.. AND:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"WORKS!!!!
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"Does NOT

and then after:
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432WORKS!!!!
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432
Does not

maybe.. it is the best way.. to change those slashes back? ;)

omion

14th March 2007, 00:17

@SpAwN_gUy:

So it looks like everything works when using forward slashes rather than backslashes? That's good to know. It's looking like it won't matter for much longer, though.

I was just contacted by foxius (at least, that will be his name when his 5 days are up ;) ) who is making a nice:rolleyes: alternative to the nice.exe program I am currently distributing. It will be a drop-in replacement, and should weigh in at ~50KB.

I can't say for sure, but it looks like it should be done soon (depending on how many more unreasonable requests I make :p )

omion

15th March 2007, 07:29

All right! I finally updated nice.exe!
(or rather, foxius updated it and I took credit for it ;) )

Get it here (http://omion.dyndns.org/x264farm/nice-20070314_win32.rar) (source (http://omion.dyndns.org/x264farm/nice-20070314_src.rar)). I also updated the first post's link.

Awesome new features:

If the program's extension is "exe", "bat", or "com", you no longer need to specifically write the extension. (This approximates the shell's handling of extensions)
Network paths work without nasty hacks
It's MUCH smaller than the old version, which had to have all the cygwin libraries dragged along with it.

MAJOR props to foxius! Thanks a bunch! :D

SpAwN_gUy

15th March 2007, 10:35

:)) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually :) ...
winAPI:
CreateProcess(with parameters)
okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ;) ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.

btw. feature request :) ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)

and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"

okay,.. i've done "pause" in .bat file... so..
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...

omion

16th March 2007, 07:41

:)) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually :) ...

Yeah. I had the basic idea of how to do it (involving CreateProcess), but my programming skills are basically limited to Perl and OCaml, neither of which are particularly good at hooking into the OS. And C just drove me nuts.

And about me using foxius's code rather than yours, well... he contacted me first :cool: Don't worry; I'll tell you the next time I need a program that I don't want to write :p

okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ;) ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.

That looks like something's wrong in the file itself...

btw. feature request :) ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)

Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... :rolleyes: )

That comment about finding GOPs (around line 450 in pass2.ml) was actually referring to the fact that there could be many more GOPs done than are in the temp stats file, and how it is quite difficult to determine where to restart counting when the program is re-run.

However, the only thing deleting the unneeded GOPs will get you is extra disk space. Everything is calculated based on the working_stats#.txt files. The result is that, once the reference to the old GOP is overwritten by the new GOP file name, the old GOP file might as well not exist any more. Nothing in the program references old GOP files, so they don't have any effect on anything.
This means that there is no reason not to delete them, but it won't help the program if they are deleted.
In either case, the whole temp directory is cleaned up at the end of the second pass, so the final result is the same no matter what files are in there.

and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"

Hmm... I'll have to look into it a bit more. Ratecontrol is quite a subtle thing, and it's fairly easy to do wrong.

I probably have a plus when I should have a minus or something. Due to the self-correcting nature of ratecontrol, just about every bug in it will result in something that is a bit off, but not by much.

Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.

okay,.. i've done "pause" in .bat file... so..
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)

(apologies for any horrible grammar or anything in this post. My brain is not working properly right now. Too much caffeine... need more sleep...)

SpAwN_gUy

16th March 2007, 09:11

And about me using foxius's code rather than yours, well... he contacted me first :cool: Don't worry; I'll tell you the next time I need a program that I don't want to write :p don't worry, i'm beginning to be more opened to peaple,.. but not all the time :) ... i gust fing your progam very interesting, and i'm trying to help.. thats all :)

That looks like something's wrong in the file itself...
....
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)
well,.. it shows right after writind to log file line "Merging 91 / 302"
and, i've muxed with my bare hands (mkvmerged) all those .mkv-files.. some of them were "unKnown format" (i guess one of those zero-bytes file, i have in _temp), and (i guess i forgot some of files, but.. ~godd mkv.. (it became 1 second less)) the end file is good,.. (thise 91,92,93 - were good in the end).
so i'm sending you those files.. via e-mail..

Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... :rolleyes: ) waiting :) ..

Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.
GordianKnot-calculator (aac-sound goes as vorbis in "Interleaving & AVI-Overhead") or Calculator in meGUI.. (as far as meGUI has "GUI" and Profiles,.. and controllerGUI can import meGUI Jobs... so, generating a comLine is not a problem to me.. ;) ..)

btw. minasan, how would you like to start controller? invisible and all "usefull info is displayed in GUI"(as meGUI does) or just run a pregenerated comLine, while showing DOSBox(like GKnot does)?

yeah,.. and something like a BUG,.. i think.. after "resuming" on second-pass,.. the info(that periodically is shown..) does not show "Current GOP" (column "Now") just "x"..

foxius

19th March 2007, 09:22

omion

19th March 2007, 19:09

Finally, I have a voice =)

x264farm is working like a charm here, the only bug I've found so far is that sometimes I get a DELFIRST from an agent and it doesn't resolve until I kill the agent, clean his temp folder and restart. I think, it would be more convenient if the agent could clean the broken files itself.

Well, DELFIRST is not really a problem in itself. It comes about when an agent finishes a job right before another job was being worked on, and it discovered that the first frame on the second job won't be an I frame. DELFIRST really just means "delete first GOP in the job, when it gets returned" and it will be taken care of when the agent returns the job.
It turns out that dealing with this case correctly is quite difficult, so I added it to the output so I could properly debug it.

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.

It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.

@SpAwN_gUy:
I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is. Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message.

And sorry about the delay. I thought I would get all this stuff done last week, but I got sick. And I play too much Oblivion. (Maybe I got sick from playing Oblivion...?)

foxius

19th March 2007, 20:06

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.

Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.

It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.

Perhaps it would work even without removing the temp files. I'll try not to delete them next time.

I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else.

SpAwN_gUy

20th March 2007, 09:36

Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.
let me guess.. maybe i'm having the sam sort of a problem..
it goes after al agents are ("agentname close_connection") and than controller just waits.. for something.
and all connection to source files are closed, and controller just waits..
it's not happening all the time, but some times..

this happens to me on both passes.. fist and second,.. in the end.. just before the MUXing starts :( ...

so my 5crnts: i think this loop occurs when agent ended encoding sended job, but something happened during sending of MKV, or Controlled didn't got it, or agent could not send it..

So a small solution,.. after a "timer" (~2 minutes) controller could ReCheck Working(as C thinks) Agents if they are actually working... timer could start after beginning of closing connections and disconnecting agents.. (well, theese are just my thoughts)

SpAwN_gUy

20th March 2007, 09:50

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem. i have now end of second-pass with 60% bitrate .. and it just reencodes something again and again..
2007-03-20~10:43:59.65 #1 7 < #2 8 (read #2)
and
2007-03-20~10:44:03.46 Number of GOPs done: 719 / 719
2007-03-20~10:44:03.46 Looks like the second pass was already done; using 0 for the number of GOPs computed and :) now i've found 770 copies of GOP #658 in tempDir.. so i deleted them.. and it encodes again something.. on 100%total and 60% of Quality..

... ADD(few mins later): Yeppie.. it did.. :) ... two deleted GOPs(manually) and retcontrol(i guess this is the proper naming) works again.

I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is. yes, mkvMerge did that also.. but not the farm :(
Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message. yes i do,.. i thought about it,.. that you would like to recreate it again :) ...

... testings... so i am... it is still..
in the DOSBox
#0299: "299 - 56204 56453 5654cd.mkv"
#0300: "300 - 56454 56559 148fc1.mkv"
#0301: "301 - 56560 56617 befbc3.mkv"
Fatal error: exception Invalid_argument("Char.chr")

in the out-dump.txt
2007-03-20~11:08:53.92 Merging 88 / 302
2007-03-20~11:08:54.03 Merging 89 / 302
2007-03-20~11:08:54.17 Merging 90 / 302
2007-03-20~11:08:54.29 Merging 91 / 302
2007-03-20~11:08:54.62 Merging 92 / 302

omion

20th March 2007, 19:01

[edit: Oops. It looks like SpAwN_gUy wrote a bunch of stuff after I started writing this. I'll get to those responses later]

First off, I'd like to mention that I updated my x264 builds to r634. links: 32-bit (http://omion.dyndns.org/x264/32/x264-634_omion.rar) 64-bit (http://omion.dyndns.org/x264/64/x264x64-634_omion.rar)

Also, I don't know if I mentioned this, but you only need my builds if you are using controller-based encoding or piped input on the agent. If you are using non-piped input to an agent-based encoder, you may use any x264 build you want.

Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.Does the agent show any errors? What's the CPU usage like when this happens?

One problem that I have had is that if an agent stops doing anything without informing the controller that something's wrong, then the encoding will never finish. If you restart either the controller or agent it should complete, but there's not really a good answer right now. I have been trying to think of a nice, clean way to handle this since long before I released x264farm...

Perhaps it would work even without removing the temp files. I'll try not to delete them next time.

I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else.

Well... I really don't think DELFIRST is related to the problem you're having. The DELFIRST condition is a product of the controller so doing something with the agents won't get anywhere.

Also, if it gets into a loop, the agents will still be encoding. The loops occur when the controller throws out everything that an agent does and tells the agent to encode the same thing again. As long as --batch is set to larger than x264's --keyint option, I don't think it's possible to get in a loop.

As to what is causing the problem you're having, I don't really know. Is it doing it a lot, or was it just that one time? If multiple times, was it always the same agent that was having the problem?

foxius

21st March 2007, 11:36

Does the agent show any errors? What's the CPU usage like when this happens?

I don't really remember. I think there was no error displayed on the agent computer. I'll check that next time.

One problem that I have had is that if an agent stops doing anything without informing the controller that something's wrong, then the encoding will never finish. If you restart either the controller or agent it should complete, but there's not really a good answer right now. I have been trying to think of a nice, clean way to handle this since long before I released x264farm...

I think the controller has to "ping" the agents to get their status (I think you already do that to get the work progress). If it's stuck then you send a "restart" signal. The agents do their work in worker threads, so the main thread is always responsive.

Well... I really don't think DELFIRST is related to the problem you're having. The DELFIRST condition is a product of the controller so doing something with the agents won't get anywhere.

But whenever that agent is stuck in a loop I see the DELFIRST message on the controller. I guess they're related in some way.

As to what is causing the problem you're having, I don't really know. Is it doing it a lot, or was it just that one time? If multiple times, was it always the same agent that was having the problem?

I had it at least for each movie i recoded. For the last movie (harry potter, part 1) i had several DELFIRST & dead loops. It wasn't always the same agent.

foxius

28th March 2007, 14:21

omion,

does x264farm have any kind of IPC that lets third party applications to get some feedback from it? it'd make easier to develop a GUI or even may be integrate with MeGUI.

fox

SpAwN_gUy

29th March 2007, 08:18

does x264farm have any kind of IPC that lets third party applications to get some feedback from it? it'd make easier to develop a GUI or even may be integrate with MeGUI.
i think not.. well, sources are opened..(in first post)..
as omion is not familiar with C i'm not familiar with OCaml :) .. i can read through code, but.. i can't programm(i've never tried in OCaml) and i don' have tools.. 'cause i'm short in traffic..

BTW... if something goes wrong.. i'm offline. :( ... for some time.. my boss told me that i'm not working.. and disabled iNet for me.. (eMail still works)

foxius

29th March 2007, 10:45

i think not.. well, sources are opened..(in first post)..
as omion is not familiar with C i'm not familiar with OCaml :) .. i can read through code, but.. i can't programm(i've never tried in OCaml) and i don' have tools.. 'cause i'm short in traffic..

knowledge of C programming is not required for implementing IPC (interprocess communication).

x264farm will allow monitoring of the encoding process via IPC (named pipes, shared memory or, ideally, RPC). Then anyone could write a third party software in almost any language to communicate with x264farm.

I've googled for "ocaml RPC" and found some implementations. I'm not sure which is more suitable because I'm not familiar with the language. RPC (remote procedure calls) will allow you to define a set of functions in IDL (interface description language). IDL then can be compiled into ocaml, c++, delphi, etc. x264farn will act as an RPC server and third party software will act as a client which will be able to call functions defined in IDL.

Given that x264farm already has functions for monitoring the agents, imlementing RPC will take a few simple steps:

1. Define a set of published functions (an interface) via IDL
2. Compile IDL into ocaml
3. Map the interface to the implementation (i.e. the function defined in IDL will call the function inside x264farm)
4. Distribute the IDL file so anyone can compile it to other programming languages and use it.

RPC server and clients will be able to communicate over various protocols, such as tcp/ip, ipx, named pipes, etc.

Wishbringer

5th April 2007, 19:42

Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.

SpAwN_gUy

6th April 2007, 08:17

Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.
yeah,.. me too... i've reported that.. i have sources of newNice.exe ... and i think i'll look though a bit later, i think..
and now i'm f___ing around of HDDVD-BackUp,.. and i think AgentBased encoding will not work :( ...
currently i'm short on 40mins from a movie, but everythig seems fine,..

foxius

8th April 2007, 13:23

Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.

Could you please copy/paste the windows error message you're getting? Unfortunately, I don't have Vista and can't test it myself. Once I have the error message I'll try to fix it.

SpAwN_gUy

9th April 2007, 08:10

@foxius ... my old problem shows:
Леша-К7, 16.03.2007 13:24 :
Problem signature:
Problem Event Name: BEX
Application Name: nice.exe
Application Version: 0.0.0.0
Application Timestamp: 45f8df57
Fault Module Name: nice.exe
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 45f8df57
Exception Offset: 00005183
Exception Code: c000000d
Exception Data: 00000000
OS Version: 6.0.6000.2.0.0.256.1
Locale ID: 1033
Additional Information 1: 5fd9
Additional Information 2: de9895fb4090bedd1a27a6ff8cfedad1
Additional Information 3: e9b8
Additional Information 4: 8079ce0cad7bc845eec58faba3ac0031
on Vista-Ultimate 64bit PC.
after doing this:
nice -n 20 avs2yuv -raw -seek 1250 -frames 250 -o - "\\controllerpc\x264farm-2\CHECH-Small\nazdravicko.avs" | nice -n 20 \\controllerpc\x264farm\!AGENT\x264x64.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 292 --fps 25/1 --pass 3 --stats "temp\stats d40faa.txt" -o "temp\output 83d1d8.mkv" - 688x400

Wishbringer

10th April 2007, 16:50

@foxius

Curiously it's only a window that opens, that says that nice.exe works anymore, then another one that asks if i want to send infos to Microsoft.

But no further details what infos are available. So I can't give you a hint, sorry.

out-dump.txt from agent doesn't contain useable infos too.

omion

11th April 2007, 04:01

legoman666

12th April 2007, 01:18

is there a way to set the amount of frames sent to each node for each job? I have 5 computers in my network, 1 is a 1.8ghz laptop, 1 is a 1.6ghz amd xp, 1 is a p2 350mhz, 1 is an AMD x2 2.6ghz (the controller), and the last is a 1.6ghz amd 2800+.

Now of course, the p2 is a slow piece of crap. Looking at the command line output during encoding, it looks like it sends a minimum of 5250 frames for each job. After all my clients finish their work, my p2 350 is still chugging away at its job. Is there a way to set it so that each job is smaller? like 250 frames?

I somehow bugged it out so that its doing 250 frames per job, but i have no idea what I did to make it do that. any ideas?

legoman666

12th April 2007, 01:52

ah nevermind, i see. for the first pass its doing ~5000 frames per job, but for the second pass its a much lower number.

:D silly me.

Wishbringer

12th April 2007, 20:22

nice -n 10 help
does it work?

@omion:
no crash, help page was displayed.

I don't really know why it crashes when starting x264.exe

legoman666

12th April 2007, 21:39

this little program is amazing, thanks for the hard work!

although I do have a little question, what options do i use only for the first pass and which do i use only for the second? Or do you use them all for both. I'm using megui to get the command line instructions and just copying them for use with controller.exe. At the moment, my megui command line says (for the HQ-slowest profile):

--pass 2 --bitrate 918 --stats ".stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "I:\example.mp4" "I:\example.avs"

and so for controller.exe im doing

controller.exe -B 918kbps --first "--ref 10 --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --analyse all --merange 12 --no-psnr --trellis 2 --progress " --second "--ref 10 --no-fast-pskip --bframes 3 --b-rdo --bime --b-pyramid --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --me umh --merange 12 --no-psnr --no-dct-decimate --no-ssim --no-psnr --vbv-maxrate 25000 --progress" --avs "I:\examlpe.avs" -o C:\example.mkv

is there anything im missing/doing wrong? with these setting and 8 physical cores working on the encoding, it took ~6 hours to encode a dvd thats about 1.5 hours long, which seems reasonable considering the quality setting. Something seems to be wrong with the first pass settings, because when I used megui alone to do the encoding, the first pass was the quickest (~30fps) and the second pass was ~3.5. With my current settings for controller.exe, the first pass took ~4 hours and the 2nd ~2hours.

list of computers workin on encoding (all over 100mb/s lan)
AMD x2 3800+ @2.6ghz (controller) 2 cores (local)
AMD 1800+ (catherine)
P2 350mhz (server)
AMD A64 2800+ (basement)
Centrino 1.6ghz (mom's laptop)
2xP3 500mhz Xeon (staircase)

Speeds at which they work

Agent | Now | GOPs | Frames | FPS |
Local 1 | 584 | 191 | 35515 | 9.11 |
Local 2 | 395 | 176 | 35362 | 9.61 |
basemnet 1 | 610 | 139 | 24938 | 6.18 |
server 1 | 644 | 7 | 1448 | 0.33 |
mom's laptop 1 | x | 40 | 7095 | 3.87 |
catherine 1 | x | 106 | 18011 | 4.59 |
staircase 1 | 659 | 32 | 6050 | 1.43 |
staircase 2 | 583 | 26 | 5110 | 1.13 |

Wishbringer

13th April 2007, 08:48

@legoman666

could it be that megui uses lesser quality settings for the first pass?
normally you don't need all options in first pass (like -b-rdo).
best is to look at megui's log to see, what commandline it used for first pass.

legoman666

13th April 2007, 13:47

ah thanks for the suggestion, that helps :D

controller.exe -B 8079kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --progress --no-dct-decimate --no-psnr --no-ssim" --second "--ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --progress --no-dct-decimate --no-psnr --no-ssim" --avs "I:\movie.avs" -o C:\movie.mkv

theres my next project, 720p uber quality :D. should take.... 12-14 hours for my 6 computers.

SpAwN_gUy

13th April 2007, 14:15

although I do have a little question, what options do i use only for the first pass and which do i use only for the second? Or do you use them all for both. I'm using megui to get the command line instructions and just copying them for use with controller.exe. At the moment, my megui command line says (for the HQ-slowest profile):
okay.. i'm back,.. and i don't have iNet.. :) ...
so,.. i have few suggestions for you ..
1) i've noticed that most powerfull PCs sould be first in controller's config (i have something like 10PCs Farm ;) ade powers vary alot)
2) .. well.. if you add Job in MeGUI and then close it.. you'll get 2 job.xml files .. look through them and you'll find something like <Commandline> .... copy the values from both files and you'll see the difference..
2b) if someone wants i can send a beta-controllerGUI prog (and agentGUI) ... main thing is that controllerGUI can generate commandline for the controller (but don't even try to start it for now ;) ... it'll give you a headake)..

peace to all, i'm waiting for something new ... ;)

legoman666

13th April 2007, 15:18

hmm, im getting an error with my next encoding project,

2007-04-13~10:05:50.00 x264farm version 1.09-163
2007-04-13~10:05:50.00 Using config file ".\config.xml"
2007-04-13~10:05:50.03 CONFIG:
2007-04-13~10:05:50.03 temp_dir: "I:\\hdtv\\x264farm\\temp\\controller"
2007-04-13~10:05:50.03 agent_list 1:
2007-04-13~10:05:50.03 "Local 1" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "Local 2" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "basemnet 1" = 192.168.1.13 50700-50703
2007-04-13~10:05:50.03 "server 1" = 192.168.1.4 50700-50703
2007-04-13~10:05:50.03 "mom's laptop 1" = 192.168.1.167 50700-50703
2007-04-13~10:05:50.03 "catherine 1" = 192.168.1.215 50700-50703
2007-04-13~10:05:50.03 "staircase 1" = 192.168.1.9 50700-50703
2007-04-13~10:05:50.03 "staircase 2" = 192.168.1.9 50700-50703
2007-04-13~10:05:50.03 agent_list 2:
2007-04-13~10:05:50.03 "Local 1" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "Local 2" = 127.0.0.1 50700-50703
2007-04-13~10:05:50.03 "basemnet 1" = 192.168.1.13 50700-50703
2007-04-13~10:05:50.03 "server 1" = 192.168.1.4 50700-50703
2007-04-13~10:05:50.03 "mom's laptop 1" = 192.168.1.167 50700-50703
2007-04-13~10:05:50.03 "catherine 1" = 192.168.1.215 50700-50703
2007-04-13~10:05:50.03 "staircase 1" = 192.168.1.9 50700-50703
2007-04-13~10:05:50.03 "staircase 2" = 192.168.1.9 50700-50703
2007-04-13~10:05:51.54
WARNING: avs2yuv exited with error 1; avs2yuv responded:

2007-04-13~10:05:51.54 GOT NORMAL INFO "\013"
2007-04-13~10:05:51.54
ERROR: AVS file 'I:\CASINO\casino.avs' does not seem to be valid

my avs file is pretty simple and worked just fine in megui when I encoded the first 200 frames just to see if it would work. heres the contents of it:

LoadPlugin("C:\PROGRA~2\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~2\GORDIA~1\AviSynthPlugins\UnDot.dll")
DirectShowSource("I:\CASINO\CASINO\BDMV\STREAM\00000.m2ts")
crop(0,144,1920,800)
LanczosResize(1280,528)
Undot()
ConvertToYV12()

my guess is the ConvertToYV12() at the bottom? megui told me to add it so I'm not really sure. any ideas?

foxius

13th April 2007, 21:53

I found that you get the best speed when using agent-based encoding. avs2yuv happens to consume too much CPU & bandwidth, but obviously useful when you can't use agent based encoding.

My setup is something like this:

1. All source files are located in network shares.
2. D2V files refer to network shares.
3. AVS files also refer to sources on network shares.
4. I also put AVS plugins on the network share
5. Thus, the agents will read the source without requiring avs2yuv to decode the source and send raw frames over your network.
6. It is easier to find(and eventually solve) problems with AVS scripts
7. I keep the sources on one HDD, write the encoded file to another.

Sample AVS script:

LoadPlugin("\\foxxp\_recoding\DGMPGDec\DGDecode.dll")
LoadPlugin("\\foxxp\_recoding\DGMPGDec\Undot.dll")

MPEG2Source("\\foxxp\_recoding\ka\ka.d2v")

# CROPPING
crop(0,0,720,576)

# RESIZING
LanczosResize(720,400)

Undot()

I also have a batch file (.bat) based on the MeGUI "Slower" profile. I just substitute

controller.exe --bitrate 1200kbps --avs "\\foxxp\_recoding\ka\ka.avs" --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --no-psnr --no-ssim" --second "--ref 5 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --no-psnr --no-ssim" -o "c:\_recoding\ka\ka.mkv"

I currently have 4 agents (including local) on a 100mbit network, 2 more over 100mbit uplink. I enable the second core on the local for the second pass (each core gives 5-7fps), and put one more job for HT processors on the network (gives 1-2 more fps per CPU). It takes 3-4 hours to encode an average movie.

I also found that the second pass never goes beyond 13-14fps, no matter how many agents/cores I add. I think it should have something to do with the bandwidth.

Hope this would be useful.

foxius

13th April 2007, 22:00

my guess is the ConvertToYV12() at the bottom? megui told me to add it so I'm not really sure. any ideas?

try removing it :) i think you only need that filter when you have some unusual (RGB?) source (i might be wrong). if it doesn't help:

1. try opening the .avs file in VirtualDub
2. try agent-based encoding

omion

14th April 2007, 03:28

WARNING: avs2yuv exited with error 1; avs2yuv responded:

2007-04-13~10:05:51.54 GOT NORMAL INFO "\013"

Try opening it up with avs2yuv. That's what x264farm uses to check the info. Run the following:
avs2yuv -frames 1 -o NUL AVSFILE.avs and see what it returns.

@all:
Again, sorry about the lack of programming. I'm pretty swamped right now :(

legoman666

14th April 2007, 04:31

apparently avs2yuv doesn't like the directshowsource? Which is weird because megui is converting it right now with no problems (at a appalingly slow 1.4fps I might add, wish i could use x264farm :P )

I:\hdtv\x264farm\controller>avs2yuv.exe -frames 1 -o NUL "I:\CASINO\casino.avs"

Avisynth error:
DirectShowSource: Could not open as video or audio.

Video returned: "DirectShowSource: couldn't create filter graph:
CoInitialize has not been called. "

Audio returned: "DirectShowSource: couldn't create filter graph:
CoInitialize has not been called. "

(I:\CASINO\casino.avs, line 18)

I:\hdtv\x264farm\controller>

its just the m2ts file from a blu ray backup (Casin Royale)

chipzoller

15th April 2007, 04:18

Might this project (sometime in the near or far future) see a GUI at some point? This being considered or worked on? Great tool, btw...

rhodges

15th April 2007, 16:00

@all who have a problem with nice vs vista:
Does it give the error message when you try to use the nice command without x264farm? Like if you type this into the terminal:
nice -n 10 help
does it work?

@all:
Sorry about not responding in a while. I got sick again, and all the students just got back to school (I work at the UCSD bookstore), and I've been doing a major overhaul of my other program (mp3packer (http://www.hydrogenaudio.org/forums/index.php?showtopic=32379)) so I haven't had much time for x264farm. I'll get to work on it as soon as I can, though.

I downloaded the source and ran the debug version and it was crashing due to a buffer size problem. The problem is the IsWinXPorLater function returns false, so max_command_size gets set to MAX_PATH which is too small.

I commented out the IsWinXPorLater check and it works like a champ on Vista 64 Ultimate.

[Edit]
I didn't try this, but I found on microsoft's site how you'd change the function:
http://msdn2.microsoft.com/en-us/library/ms724451.aspx

from:
return (version.dwMajorVersion >= 5 && version.dwMinorVersion >= 1);

to:
return ((version.dwMajorVersion > 5) || (version.dwMajorVersion == 5 && version.dwMinorVersion >= 1));

omion

15th April 2007, 17:25

[Edit]
I didn't try this, but I found on microsoft's site how you'd change the function:
http://msdn2.microsoft.com/en-us/library/ms724451.aspx

from:
return (version.dwMajorVersion >= 5 && version.dwMinorVersion >= 1);

to:
return ((version.dwMajorVersion > 5) || (version.dwMajorVersion == 5 && version.dwMinorVersion >= 1));

Ah. That would be a problem. Vista is version 6.0, so dwMinorVersion < 1...

I put a new nice build up. Download it here (http://omion.dyndns.org/x264farm/nice-20070415.rar) (src (http://omion.dyndns.org/x264farm/nice-20070415_src.rar))

harissa

15th April 2007, 21:55

I spent 24 h trying to make this work and I couldn't
Here s my setup

my avs
loadplugin("DGDecode.dll")
Mpeg2Source("carnavale.d2v")
I dont't know where to put nice.exe , avs2yuv.exe and x264-634_omion.exe
both controller and agent scrollin like hell , and showing this

E:\x264farm\temp\controller
2007-04-15~16:30:46.00 x264farm version 1.09-163
2007-04-15~16:30:46.00 Oops! File " G:\9\ka.mkv" does not exist
2007-04-15~16:30:46.00 WARNING: Second pass options are not set; this will use the default x264 settings
2007-04-15~16:30:46.00 Using config file ".\config.xml"
2007-04-15~16:30:46.00 CONFIG:
2007-04-15~16:30:46.00 temp_dir: "E:\\x264farm\\temp\\controller"
2007-04-15~16:30:46.00 agent_list 1:
2007-04-15~16:30:46.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.00 "Linux 1" = 169.254.23.92 50700-50703
2007-04-15~16:30:46.00 "Apple 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.00 "Apple 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 agent_list 2:
2007-04-15~16:30:46.01 "Local 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Local 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Linux 1" = 169.254.23.92 50700-50703
2007-04-15~16:30:46.01 "Linux 2" = 169.254.23.92 50700-50703
2007-04-15~16:30:46.01 "Apple 1" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Apple 2" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.01 "Apple 3" = 169.254.25.121 50700-50703
2007-04-15~16:30:46.15 GOT NORMAL INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-15~16:30:46.26 GOT FIRST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-15~16:30:46.37 GOT FAST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-15~16:30:46.39 INFO:
2007-04-15~16:30:46.39 i_res_x = 1280
2007-04-15~16:30:46.39 i_res_y = 720
2007-04-15~16:30:46.39 i_fps_n = 60000
2007-04-15~16:30:46.39 i_fps_d = 1001
2007-04-15~16:30:46.39 i_fps_f = 59.940060
2007-04-15~16:30:46.39 i_num_frames = 23247
2007-04-15~16:30:46.39 i_bytes_y = 921600
2007-04-15~16:30:46.39 i_bytes_uv = 230400
2007-04-15~16:30:46.39 i_bytes_per_frame = 1382400
2007-04-15~16:30:46.39 multiplier for kbps to fbits = 0.059940
2007-04-15~16:30:46.39 fast_res_x = 1280
2007-04-15~16:30:46.39 fast_res_y = 720
2007-04-15~16:30:46.39 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\option_log.txt.temp: No such file or directory")2007-04-15~16:30:46.39 AVS temp dir is E:\x264farm\temp\controller\132FA1D98D68B80231C0CECCD0F61FAF x264.avs
2007-04-15~16:30:46.40 First string is "-2\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n3hœ)Z\013ûø‚¾\127Üºx\027\020 = G:\\dvb\\x264.avs\n", with MD5 "F51A9C869FA15B80F7D0F2202ABBDF98"
2007-04-15~16:30:46.60 Second string is "\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n", with MD5 "A25B8F0CCB96F56266CFA33B07D0F828"
2007-04-15~16:30:46.81 not (not (is_something first_dir) && is_file first_stats) [first pass after making dir]
2007-04-15~16:30:46.81 not (is_dir first_dir)
2007-04-15~16:30:46.81 11111111 FIRST PASS 11111111
2007-04-15~16:30:46.81 From file:
2007-04-15~16:30:46.81 ( 0,23246) Empty
2007-04-15~16:30:46.81 Local 1 starting on 169.254.25.121 (id 1)
2007-04-15~16:30:46.81 Local 2 starting on 169.254.25.121 (id 2)
2007-04-15~16:30:46.81 Linux 1 starting on 169.254.23.92 (id 3)
2007-04-15~16:30:46.81 Apple 1 starting on 169.254.25.121 (id 4)
2007-04-15~16:30:46.81 Apple 2 starting on 169.254.25.121 (id 5)
2007-04-15~16:30:46.81 Splitter starting
2007-04-15~16:30:46.82 Split points at 5,3,1,1
2007-04-15~16:30:49.87 queue_beginning found the beginning; split to Q(0,5062) :: (5063,23246)
2007-04-15~16:30:49.87 Ranges at beginning:
2007-04-15~16:30:49.87 ( 0, 5062) Queued
2007-04-15~16:30:49.87 ( 5063,23246) Empty
2007-04-15~16:30:49.87 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:30:49.87 Splitter: fewer queues than max_queues_split
2007-04-15~16:30:49.87 split_largest_range found the biggest empty range to be 18184 frames long
2007-04-15~16:30:53.29 split_largest_range found start I frame 11460 in (11405,11655)
2007-04-15~16:30:56.14 split_largest_range split to (5063,11460) :: (11461,16617) :: (16618,23246)
2007-04-15~16:30:56.14 Splitter: split_largest_range did something; re-run
2007-04-15~16:30:56.14 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:30:56.14 Splitter: fewer queues than max_queues_split
2007-04-15~16:30:56.14 split_largest_range found the biggest empty range to be 6629 frames long
2007-04-15~16:30:58.62 split_largest_range found start I frame 19947 in (19807,20057)
2007-04-15~16:30:58.62 Splitter: split_largest_range did something; re-run
2007-04-15~16:30:58.62 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:30:58.62 Splitter: fewer queues than max_queues_split
2007-04-15~16:30:58.62 split_largest_range found the biggest empty range to be 6398 frames long
2007-04-15~16:31:02.35 split_largest_range found start I frame 8307 in (8137,8387)
2007-04-15~16:31:02.35 Splitter: split_largest_range did something; re-run
2007-04-15~16:31:02.35 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:31:02.35 Splitter: fewer queues than max_queues_split
2007-04-15~16:31:02.35 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-15~16:31:02.35 but that's not long enough! (5250)
2007-04-15~16:31:02.35 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-15~16:31:02.35 ( 0, 5062) Queued
2007-04-15~16:31:02.35 ( 5063, 8307) Empty
2007-04-15~16:31:02.35 ( 8308,11460) Queued
2007-04-15~16:31:02.35 (11461,16617) Queued
2007-04-15~16:31:02.35 (16618,19947) Empty
2007-04-15~16:31:02.35 (19948,23246) Queued
2007-04-15~16:31:54.68 using compression type 0 (no compression)
2007-04-15~16:31:54.68 Local 2 found no ranges after full
2007-04-15~16:31:54.68 Local 2 check_out_job got frames 0 - 5062
2007-04-15~16:31:54.68 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:31:54.68 using compression type 0 (no compression)
2007-04-15~16:31:54.68 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-15~16:31:54.68 Splitter: fewer queues than max_queues_split
2007-04-15~16:31:54.68 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-15~16:31:54.68 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-15~16:31:54.68 Local 2 sent zone string ""
2007-04-15~16:31:54.68 but that's not long enough! (5250)
2007-04-15~16:31:54.68 Local 2 sent range (0,5062)
2007-04-15~16:31:54.68 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-15~16:31:54.68 ( 0, 5062) Working (Local 2)
2007-04-15~16:31:54.68 ( 5063, 8307) Empty
2007-04-15~16:31:54.68 ( 8308,11460) Queued
2007-04-15~16:31:54.68 (11461,16617) Queued
2007-04-15~16:31:54.68 (16618,19947) Empty
2007-04-15~16:31:54.68 (19948,23246) Queued
2007-04-15~16:31:54.68 Local 2 agent-based encoding; waiting for agent to finish
2007-04-15~16:31:54.68 Apple 1 found no ranges after full
2007-04-15~16:31:54.68 Apple 1 check_out_job got frames 8308 - 11460
2007-04-15~16:31:54.68 Splitter: fewer queues than max_queues_after_full
2007-04-15~16:31:54.70 Apple 1 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-15~16:31:54.70 Splitter: fewer queues than max_queues_split
2007-04-15~16:31:54.70 Apple 1 sent video info 1280x720 @ 60000/1001
2007-04-15~16:31:54.70 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-15~16:31:54.70 Apple 1 sent zone string ""
2007-04-15~16:31:54.70 but that's not long enough! (5250)
2007-04-15~16:31:54.70 Apple 1 sent range (8307,11460)
2007-04-15~16:31:54.70 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-15~16:31:54.70 ( 0, 5062) Working (Local 2)
2007-04-15~16:31:54.70 ( 5063, 8307) Empty
2007-04-15~16:31:54.70 ( 8308,11460) Working (Apple 1)
2007-04-15~16:31:54.70 (11461,16617) Queued
2007-04-15~16:31:54.70 (16618,19947) Empty
2007-04-15~16:31:54.70 (19948,23246) Queued
2007-04-15~16:31:54.70 Apple 1 agent-based encoding; waiting for agent to finish
2007-04-15~16:31:54.73 Local 2 process_job failed with ("The operation completed successfully.\013\n",recv,); putting job (0,5062) back
2007-04-15~16:31:54.73 Apple 1 process_job failed with "End_of_file"; putting job (8308,11460) back
2007-04-15~16:31:54.73 put back range
2007-04-15~16:31:54.73 ( 0, 5062) Queued
2007-04-15~16:31:54.73 ( 5063, 8307) Empty
2007-04-15~16:31:54.73 ( 8308,11460) Working (Apple 1)
2007-04-15~16:31:54.73 (11461,16617) Queued
2007-04-15~16:31:54.73 (16618,19947) Empty
2007-04-15~16:31:54.73 (19948,23246) Queued
2007-04-15~16:31:54.73 put back range
2007-04-15~16:31:54.73 ( 0, 5062) Queued
2007-04-15~16:31:54.73 ( 5063, 8307) Empty
2007-04-15~16:31:54.73 ( 8308,11460) Queued
2007-04-15~16:31:54.73 (11461,16617) Queued
2007-04-15~16:31:54.75 (16618,19947) Empty
2007-04-15~16:31:54.75 (19948,23246) Queued
2007-04-15~16:31:54.73 Local 2 close connection
2007-04-15~16:31:54.75 Apple 1 close connection

E:\x264farm\temp\agent
2007-04-15~16:31:54.00 x264farm version 1.09-163
2007-04-15~16:31:54.00 Using config file ".\config.xml"
2007-04-15~16:31:54.00 Temp dir: "E:\x264farm\temp\agent"
2007-04-15~16:31:54.00 Ports: 50700 - 50703
2007-04-15~16:31:54.00 x264: "x264.exe"
2007-04-15~16:31:54.00 nice: 10
2007-04-15~16:31:54.00 agents: 1
2007-04-15~16:31:54.00 agent bases:
2007-04-15~16:31:54.00 [exact]
2007-04-15~16:31:54.00 pipe: false
2007-04-15~16:31:54.00 compressions: 0001
2007-04-15~16:31:54.00 deleting old files: after 604800 seconds
2007-04-15~16:31:54.00 trying port 50700
2007-04-15~16:31:54.00 STALE ignoring file "agent.exe"
2007-04-15~16:31:54.01 bound to port 50700
2007-04-15~16:31:54.01 STALE ignoring file "config.xml"
2007-04-15~16:31:54.01 listening...
2007-04-15~16:31:54.01 STALE not time for "out-dump.txt"
2007-04-15~16:31:54.01 working!
2007-04-15~16:31:54.01 STALE ignoring file "x264.exe"
2007-04-15~16:31:54.07 Got a connection from 169.254.25.121:4519
2007-04-15~16:31:54.07 working!
2007-04-15~16:31:54.07 Got a connection from 169.254.25.121:4522
2007-04-15~16:31:54.07 working!
2007-04-15~16:31:54.07 2 first pass frames 0 to 5062
2007-04-15~16:31:54.07 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.07 2 agent-based encoding
2007-04-15~16:31:54.07 2 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats b4efcd.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.09 3 first pass frames 8307 to 11460
2007-04-15~16:31:54.09 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.09 3 agent-based encoding
2007-04-15~16:31:54.09 3 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats a79e8d.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.12 2 exited 1
2007-04-15~16:31:54.12 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.12 3 exited 1
2007-04-15~16:31:54.12 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.15 Got a connection from 169.254.25.121:4527
2007-04-15~16:31:54.17 working!
2007-04-15~16:31:54.17 4 first pass frames 0 to 5062
2007-04-15~16:31:54.17 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.17 4 agent-based encoding
2007-04-15~16:31:54.17 4 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 71045f.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.20 4 exited 1
2007-04-15~16:31:54.20 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.20 Got a connection from 169.254.25.121:4529
2007-04-15~16:31:54.20 working!
2007-04-15~16:31:54.21 5 first pass frames 0 to 5062
2007-04-15~16:31:54.21 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-15~16:31:54.21 5 agent-based encoding
2007-04-15~16:31:54.21 5 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 1a9ec3.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.25 5 exited 1
2007-04-15~16:31:54.25 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"
2007-04-15~16:31:54.25 Got a connection from 169.254.25.121:4530
2007-04-15~16:31:54.25 working

omion

16th April 2007, 00:03

E:\x264farm\temp\controller
2007-04-15~16:30:46.00 x264farm version 1.09-163
2007-04-15~16:30:46.00 Oops! File " G:\9\ka.mkv" does not exist

This looks a bit fishy. What was the command line that you ran the controller with?

2007-04-15~16:30:46.39 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\option_log.txt.temp: No such file or directory")This is sort of odd, too, but I don't think it matters.

E:\x264farm\temp\agent
2007-04-15~16:31:54.17 4 doing the following:
nice -n 10 x264.exe -2 --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 71045f.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-15~16:31:54.20 4 exited 1
2007-04-15~16:31:54.20 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")"

This probably means that the agent can't find x264. If you just type "x264.exe" in the agent directory, what does it say?

If it can't find x264, you need to make sure that it's on the path (in the agent's dir is a good place) and set the <x264> line in the agent's config.xml file to the right executable (that is, if you are using "x264-634_omion.exe", make sure config.xml has the line <x264>x264-634_omion.exe</x264> in it)

harissa

16th April 2007, 02:19

foxius

16th April 2007, 06:57

I run controller
controller.exe --bitrate 3000kbps --avs "G:\dvb\x264.avs" --first --second -o "G:\ka.mkv"
after that on the same pc , even on another pc , i run agent.exe , and it go mad scrollin

i ve tried to put the x264.exe in both dir , dosen t change anything ,

is there any other setting except the config.xlm
I dont't know where to put nice.exe , avs2yuv.exe and x264-634_omion.exe
cauz I can t find how to make this run

1. nice.exe goes to agent's folder
2. avs2yuv.exe goes to controller's folder
3. x264-634_omion.exe should be renamed to x264.exe and put into agent's folder (unless you've changed x264 path in agent's config.xml)

Wishbringer

16th April 2007, 09:27

@omion:
is your x264 build a plain compile from source with MS Visual Studio,
or did you some changes in source?
I just tried to compile build 650 with Studio 2005, no errors, but i can't get controller based encode to run.

harissa

16th April 2007, 13:45

i fixed evrything still not workin i m postin all info this time
E:\x264farm\temp\controller
has avs2yuv.exe controller.exe config.xml
<config>
<temp>E:\x264farm\temp\controller</temp> 
<agents>
<agent name="Local"> 
<ip>169.254.25.121</ip> 
<port from="50700" to="50703"/> 
<number>2</number> 
</agent>
<agent name="Linux">
<ip>169.254.23.92</ip>
<port from="50700" to="50703"/>
<number pad="1">1</number>

</agent>
<agent name="Apple">
<ip>169.254.25.121</ip>
<port from="50700" to="50703"/>
<number pad="1">2</number>
</agent>

</agents>

</config>

here the log from the controller
controller.exe --bitrate 3000kbps --avs "G:\dvb\x264.avs" --first --second -o "G:\ka.mkv"

after this point it start scrollin non stop once the agent open
2007-04-16~08:33:18.00 x264farm version 1.09-163
2007-04-16~08:33:18.00 WARNING: Second pass options are not set; this will use the default x264 settings
2007-04-16~08:33:18.00 Using config file ".\config.xml"
2007-04-16~08:33:18.00 CONFIG:
2007-04-16~08:33:18.00 temp_dir: "E:\\x264farm\\temp\\controller"
2007-04-16~08:33:18.00 agent_list 1:
2007-04-16~08:33:18.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.00 "Linux 1" = 169.254.23.92 50700-50703
2007-04-16~08:33:18.00 "Apple 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.00 "Apple 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 agent_list 2:
2007-04-16~08:33:18.01 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Linux 1" = 169.254.23.92 50700-50703
2007-04-16~08:33:18.01 "Linux 2" = 169.254.23.92 50700-50703
2007-04-16~08:33:18.01 "Apple 1" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Apple 2" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.01 "Apple 3" = 169.254.25.121 50700-50703
2007-04-16~08:33:18.39 GOT NORMAL INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~08:33:18.50 GOT FIRST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~08:33:18.62 GOT FAST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~08:33:18.62 INFO:
2007-04-16~08:33:18.62 i_res_x = 1280
2007-04-16~08:33:18.62 i_res_y = 720
2007-04-16~08:33:18.62 i_fps_n = 60000
2007-04-16~08:33:18.62 i_fps_d = 1001
2007-04-16~08:33:18.62 i_fps_f = 59.940060
2007-04-16~08:33:18.62 i_num_frames = 23247
2007-04-16~08:33:18.62 i_bytes_y = 921600
2007-04-16~08:33:18.62 i_bytes_uv = 230400
2007-04-16~08:33:18.62 i_bytes_per_frame = 1382400
2007-04-16~08:33:18.62 multiplier for kbps to fbits = 0.059940
2007-04-16~08:33:18.62 fast_res_x = 1280
2007-04-16~08:33:18.62 fast_res_y = 720
2007-04-16~08:33:18.62 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\option_log.txt.temp: No such file or directory")2007-04-16~08:33:18.64 AVS temp dir is E:\x264farm\temp\controller\132FA1D98D68B80231C0CECCD0F61FAF x264.avs
2007-04-16~08:33:18.64 First string is "--second\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n3hœ)Z\013ûø‚¾\127Üºx\027\020 = G:\\dvb\\x264.avs\n", with MD5 "83936EFFC345B4F1C55FA600066A3830"
2007-04-16~08:33:18.84 Second string is "\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n", with MD5 "A25B8F0CCB96F56266CFA33B07D0F828"
2007-04-16~08:33:19.04 not (not (is_something first_dir) && is_file first_stats) [first pass after making dir]
2007-04-16~08:33:19.04 11111111 FIRST PASS 11111111
2007-04-16~08:33:19.04 #1 3 > #2 2 (read #1)
2007-04-16~08:33:19.04 From file:
2007-04-16~08:33:19.04 ( 0,23246) Empty
2007-04-16~08:33:19.04 Local 1 starting on 169.254.25.121 (id 1)
2007-04-16~08:33:19.04 Local 2 starting on 169.254.25.121 (id 2)
2007-04-16~08:33:19.04 Apple 1 starting on 169.254.25.121 (id 4)
2007-04-16~08:33:19.04 Linux 1 starting on 169.254.23.92 (id 3)
2007-04-16~08:33:19.04 Apple 2 starting on 169.254.25.121 (id 5)
2007-04-16~08:33:19.04 Splitter starting
2007-04-16~08:33:19.06 Split points at 5,3,1,1
2007-04-16~08:33:21.73 queue_beginning found the beginning; split to Q(0,5062) :: (5063,23246)
2007-04-16~08:33:21.73 Ranges at beginning:
2007-04-16~08:33:21.75 ( 0, 5062) Queued
2007-04-16~08:33:21.75 ( 5063,23246) Empty
2007-04-16~08:33:21.75 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:21.75 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:21.75 split_largest_range found the biggest empty range to be 18184 frames long
2007-04-16~08:33:24.60 split_largest_range found start I frame 11460 in (11405,11655)
2007-04-16~08:33:27.07 split_largest_range split to (5063,11460) :: (11461,16617) :: (16618,23246)
2007-04-16~08:33:27.09 Splitter: split_largest_range did something; re-run
2007-04-16~08:33:27.09 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:27.09 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:27.09 split_largest_range found the biggest empty range to be 6629 frames long
2007-04-16~08:33:29.15 split_largest_range found start I frame 19947 in (19807,20057)
2007-04-16~08:33:29.15 Splitter: split_largest_range did something; re-run
2007-04-16~08:33:29.15 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:29.15 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:29.15 split_largest_range found the biggest empty range to be 6398 frames long
2007-04-16~08:33:32.35 split_largest_range found start I frame 8307 in (8137,8387)
2007-04-16~08:33:32.35 Splitter: split_largest_range did something; re-run
2007-04-16~08:33:32.35 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:32.35 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:32.35 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~08:33:32.35 but that's not long enough! (5250)
2007-04-16~08:33:32.35 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~08:33:32.35 ( 0, 5062) Queued
2007-04-16~08:33:32.35 ( 5063, 8307) Empty
2007-04-16~08:33:32.35 ( 8308,11460) Queued
2007-04-16~08:33:32.35 (11461,16617) Queued
2007-04-16~08:33:32.35 (16618,19947) Empty
2007-04-16~08:33:32.35 (19948,23246) Queued
2007-04-16~08:33:55.40 using compression type 0 (no compression)
2007-04-16~08:33:55.40 Local 2 found no ranges after full
2007-04-16~08:33:55.40 Local 2 check_out_job got frames 0 - 5062
2007-04-16~08:33:55.40 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:55.40 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~08:33:55.40 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:55.40 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-16~08:33:55.40 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~08:33:55.40 Local 2 sent zone string ""
2007-04-16~08:33:55.40 but that's not long enough! (5250)
2007-04-16~08:33:55.40 Local 2 sent range (0,5062)
2007-04-16~08:33:55.40 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~08:33:55.40 ( 0, 5062) Working (Local 2)
2007-04-16~08:33:55.40 ( 5063, 8307) Empty
2007-04-16~08:33:55.40 ( 8308,11460) Queued
2007-04-16~08:33:55.40 (11461,16617) Queued
2007-04-16~08:33:55.42 (16618,19947) Empty
2007-04-16~08:33:55.42 (19948,23246) Queued
2007-04-16~08:33:55.40 Local 2 agent-based encoding; waiting for agent to finish
2007-04-16~08:33:55.60 using compression type 0 (no compression)
2007-04-16~08:33:55.60 Apple 2 found no ranges after full
2007-04-16~08:33:55.60 Apple 2 check_out_job got frames 8308 - 11460
2007-04-16~08:33:55.60 Splitter: fewer queues than max_queues_after_full
2007-04-16~08:33:55.60 Apple 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~08:33:55.60 Splitter: fewer queues than max_queues_split
2007-04-16~08:33:55.60 Apple 2 sent video info 1280x720 @ 60000/1001
2007-04-16~08:33:55.60 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~08:33:55.60 Apple 2 sent zone string ""
2007-04-16~08:33:55.60 but that's not long enough! (5250)
2007-04-16~08:33:55.60 Apple 2 sent range (8307,11460)
2007-04-16~08:33:55.60 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~08:33:55.60 ( 0, 5062) Working (Local 2)
2007-04-16~08:33:55.60 ( 5063, 8307) Empty
2007-04-16~08:33:55.60 ( 8308,11460) Working (Apple 2)
2007-04-16~08:33:55.60 (11461,16617) Queued
2007-04-16~08:33:55.60 (16618,19947) Empty
2007-04-16~08:33:55.60 (19948,23246) Queued

harissa

16th April 2007, 13:46

E:\x264farm\temp\agent , on the same computer
agent.exe x264.exe nice.exe config.xml
<config>
<temp>E:\x264farm\temp\agent</temp>

2007-04-16~08:33:49.00 x264farm version 1.09-163
2007-04-16~08:33:49.00 Using config file ".\config.xml"
2007-04-16~08:33:49.00 Temp dir: "E:\x264farm\temp\agent"
2007-04-16~08:33:49.00 Ports: 50700 - 50703
2007-04-16~08:33:49.00 x264: "x264.exe"
2007-04-16~08:33:49.00 nice: 10
2007-04-16~08:33:49.00 agents: 1
2007-04-16~08:33:49.00 agent bases:
2007-04-16~08:33:49.00 [exact]
2007-04-16~08:33:49.00 pipe: false
2007-04-16~08:33:49.00 compressions: 0001
2007-04-16~08:33:49.00 deleting old files: after 604800 seconds
2007-04-16~08:33:49.00 trying port 50700
2007-04-16~08:33:49.00 STALE ignoring file "agent.exe"
2007-04-16~08:33:49.01 bound to port 50700
2007-04-16~08:33:49.01 STALE ignoring file "config.xml"
2007-04-16~08:33:49.01 listening...
2007-04-16~08:33:49.01 STALE ignoring file "nice.exe"
2007-04-16~08:33:49.01 working!
2007-04-16~08:33:49.01 STALE not time for "out-dump.txt"
2007-04-16~08:33:49.01 STALE ignoring file "x264.exe"
2007-04-16~08:33:55.57 Got a connection from 169.254.25.121:1624
2007-04-16~08:33:55.57 working!
2007-04-16~08:33:55.57 2 first pass frames 0 to 5062
2007-04-16~08:33:55.57 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:55.57 2 agent-based encoding
2007-04-16~08:33:55.57 2 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 45b9ad.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:55.78 Got a connection from 169.254.25.121:1627
2007-04-16~08:33:55.78 working!
2007-04-16~08:33:55.78 3 first pass frames 8307 to 11460
2007-04-16~08:33:55.78 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:55.78 3 agent-based encoding
2007-04-16~08:33:55.78 3 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 5d3a0f.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.00 Got a connection from 169.254.25.121:1625
2007-04-16~08:33:56.00 working!
2007-04-16~08:33:56.01 4 first pass frames 19947 to 23246
2007-04-16~08:33:56.01 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.01 4 agent-based encoding
2007-04-16~08:33:56.01 4 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 9f4a62.txt" --seek 19947 --frames 3300 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.17 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 45b9ad.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.18 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 5d3a0f.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.18 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 9f4a62.txt\\\" --seek 19947 --frames 3300 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.20 Got a connection from 169.254.25.121:1628
2007-04-16~08:33:56.20 working!
2007-04-16~08:33:56.21 Got a connection from 169.254.25.121:1629
2007-04-16~08:33:56.21 working!
2007-04-16~08:33:56.21 Got a connection from 169.254.25.121:1630
2007-04-16~08:33:56.21 working!
2007-04-16~08:33:56.23 5 first pass frames 0 to 5062
2007-04-16~08:33:56.23 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.23 5 agent-based encoding
2007-04-16~08:33:56.23 5 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats 5484d5.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.25 7 first pass frames 19947 to 23246
2007-04-16~08:33:56.25 6 first pass frames 8307 to 11460
2007-04-16~08:33:56.25 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.25 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.25 7 agent-based encoding
2007-04-16~08:33:56.25 6 agent-based encoding
2007-04-16~08:33:56.25 7 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats b883ee.txt" --seek 19947 --frames 3300 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.25 6 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats e89baf.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.29 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats 5484d5.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.31 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats b883ee.txt\\\" --seek 19947 --frames 3300 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.32 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\stats e89baf.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~08:33:56.34 Got a connection from 169.254.25.121:1631
2007-04-16~08:33:56.34 working!
2007-04-16~08:33:56.34 Got a connection from 169.254.25.121:1632
2007-04-16~08:33:56.34 working!
2007-04-16~08:33:56.34 Got a connection from 169.254.25.121:1633
2007-04-16~08:33:56.34 working!
2007-04-16~08:33:56.35 8 first pass frames 0 to 5062
2007-04-16~08:33:56.35 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.37 8 agent-based encoding
2007-04-16~08:33:56.37 8 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\stats b989cd.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~08:33:56.39 9 first pass frames 8307 to 11460
2007-04-16~08:33:56.39 10 first pass frames 19947 to 23246
2007-04-16~08:33:56.39 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.39 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~08:33:56.39 9 agent-based encoding
2007-04-16~08:33:56.39 10 agent-based encoding
2007-04-16~08:33:56.39 9 doing the following:

foxius

16th April 2007, 14:01

foxius

16th April 2007, 14:09

harissa, please ensure the following folders exist prior to running the controller and agent:

E:\x264farm\temp\controller
E:\x264farm\temp\agent

it seems to me that the controller is not able to write to the temp folder.

here's my x264 folder tree for reference.

x264farm
│ gpl.txt
│ x264farm.html
│
├───agent-win32
│ │ agent.exe
│ │ config.xml
│ │ gpl.txt
│ │ nice.exe
│ │ out-dump.txt
│ │ x264.exe
│ │
│ └───temp
└───controller
│ avs2yuv.exe
│ config.xml
│ controller.exe
│ enc.bat
│ out-dump.txt
│
└───temp

and here's my agent config:

<config>
<temp>.\temp</temp>

<port from="50700" to="50703"/>

<x264>x264.exe</x264>

<nice>20</nice>

<base>d:\</base>

<agentpipe>0</agentpipe>

<compression>
<type>0</type>
<type>1</type>
</compression>

</config>

and here's controller config:

<config>
<temp>.\temp</temp> 

<agents>
<agent name="Local">
<ip>127.0.0.1</ip>
<port from="50700" to="50703"/>
<number>1</number>
</agent>

<agent name="agent1">
<ip>192.168.13.1</ip>
<port from="50700" to="50703"/>
<number>1</number>
</agent>

<agent name="agent2">
<ip>192.168.13.15</ip>
<port from="50700" to="50703"/>
<number>1</number>
</agent>
</agents>
</config>

as you can see, i keep temp folders inside agent's and controller's folder and use a relative path in the config. this allows me to copy them to any path without modifying the config at all.

harissa

16th April 2007, 14:13

i ve tried that
i have E:\x264farm\temp
4 dir inside
controller and agent are temp dir ,
and controller2 dir, contain all exe
and agent2 dir contain all exe

harissa

16th April 2007, 14:25

I ve done the same setting as u do , i have some improvement less error msg , but both windows are scrollin like hell , can i post again the log for 2 agent so u can take a look

foxius

16th April 2007, 14:40

I ve done the same setting as u do , i have some improvement less error msg , but both windows are scrollin like hell , can i post again the log for 2 agent so u can take a look

sure. do you get the errors only in agents' windows now? controller is ok?

harissa

16th April 2007, 15:08

same dir as you same commande line
2007-04-16~10:09:46.00 x264farm version 1.09-163
2007-04-16~10:09:46.00 WARNING: Second pass options are not set; this will use the default x264 settings
2007-04-16~10:09:46.00 Using config file ".\config.xml"
2007-04-16~10:09:46.00 CONFIG:
2007-04-16~10:09:46.00 temp_dir: "E:\\x264farm\\temp\\controller\\temp"
2007-04-16~10:09:46.00 agent_list 1:
2007-04-16~10:09:46.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "pc2 1" = 169.254.23.92 50700-50703
2007-04-16~10:09:46.00 "pc3 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "pc3 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 agent_list 2:
2007-04-16~10:09:46.00 "Local 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.00 "Local 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.01 "pc2 1" = 169.254.23.92 50700-50703
2007-04-16~10:09:46.01 "pc2 2" = 169.254.23.92 50700-50703
2007-04-16~10:09:46.01 "pc3 1" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.01 "pc3 2" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.01 "pc3 3" = 169.254.25.121 50700-50703
2007-04-16~10:09:46.12 GOT NORMAL INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~10:09:46.23 GOT FIRST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~10:09:46.32 GOT FAST INFO "G:\\dvb\\x264.avs: 1280x720, 60000/1001 fps, 23247 frames\013"
2007-04-16~10:09:46.34 INFO:
2007-04-16~10:09:46.34 i_res_x = 1280
2007-04-16~10:09:46.34 i_res_y = 720
2007-04-16~10:09:46.34 i_fps_n = 60000
2007-04-16~10:09:46.34 i_fps_d = 1001
2007-04-16~10:09:46.34 i_fps_f = 59.940060
2007-04-16~10:09:46.34 i_num_frames = 23247
2007-04-16~10:09:46.34 i_bytes_y = 921600
2007-04-16~10:09:46.34 i_bytes_uv = 230400
2007-04-16~10:09:46.34 i_bytes_per_frame = 1382400
2007-04-16~10:09:46.34 multiplier for kbps to fbits = 0.059940
2007-04-16~10:09:46.34 fast_res_x = 1280
2007-04-16~10:09:46.34 fast_res_y = 720
2007-04-16~10:09:46.34 Oops. Writing the args failed with: Sys_error("E:\\x264farm\\temp\\controller\\temp\\option_log.txt.temp: No such file or directory")2007-04-16~10:09:46.34 AVS temp dir is E:\x264farm\temp\controller\temp\132FA1D98D68B80231C0CECCD0F61FAF x264.avs
2007-04-16~10:09:46.34 First string is "--second\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n3hœ)Z\013ûø‚¾\127Üºx\027\020 = G:\\dvb\\x264.avs\n", with MD5 "83936EFFC345B4F1C55FA600066A3830"
2007-04-16~10:09:46.54 Second string is "\n„•¦¾\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\n0+0\n0\n", with MD5 "A25B8F0CCB96F56266CFA33B07D0F828"
2007-04-16~10:09:46.75 not (not (is_something first_dir) && is_file first_stats) [first pass after making dir]
2007-04-16~10:09:46.76 11111111 FIRST PASS 11111111
2007-04-16~10:09:46.76 #1 1 < #2 2 (read #2)
2007-04-16~10:09:46.76 From file:
2007-04-16~10:09:46.76 ( 0,23246) Empty
2007-04-16~10:09:46.76 Local 1 starting on 169.254.25.121 (id 1)
2007-04-16~10:09:46.76 Local 2 starting on 169.254.25.121 (id 2)
2007-04-16~10:09:46.76 pc2 1 starting on 169.254.23.92 (id 3)
2007-04-16~10:09:46.76 pc3 1 starting on 169.254.25.121 (id 4)
2007-04-16~10:09:46.76 pc3 2 starting on 169.254.25.121 (id 5)
2007-04-16~10:09:46.76 Splitter starting
2007-04-16~10:09:46.76 Split points at 5,3,1,1
2007-04-16~10:09:49.90 queue_beginning found the beginning; split to Q(0,5062) :: (5063,23246)
2007-04-16~10:09:49.90 Ranges at beginning:
2007-04-16~10:09:49.90 ( 0, 5062) Queued
2007-04-16~10:09:49.90 ( 5063,23246) Empty
2007-04-16~10:09:49.90 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:09:49.90 Splitter: fewer queues than max_queues_split
2007-04-16~10:09:49.90 split_largest_range found the biggest empty range to be 18184 frames long
2007-04-16~10:09:53.43 split_largest_range found start I frame 11460 in (11405,11655)
2007-04-16~10:09:56.29 split_largest_range split to (5063,11460) :: (11461,16617) :: (16618,23246)
2007-04-16~10:09:56.29 Splitter: split_largest_range did something; re-run
2007-04-16~10:09:56.29 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:09:56.29 Splitter: fewer queues than max_queues_split
2007-04-16~10:09:56.29 split_largest_range found the biggest empty range to be 6629 frames long
2007-04-16~10:09:58.78 split_largest_range found start I frame 19947 in (19807,20057)
2007-04-16~10:09:58.78 Splitter: split_largest_range did something; re-run
2007-04-16~10:09:58.78 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:09:58.78 Splitter: fewer queues than max_queues_split
2007-04-16~10:09:58.78 split_largest_range found the biggest empty range to be 6398 frames long
2007-04-16~10:10:02.56 split_largest_range found start I frame 8307 in (8137,8387)
2007-04-16~10:10:02.57 Splitter: split_largest_range did something; re-run
2007-04-16~10:10:02.57 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:10:02.57 Splitter: fewer queues than max_queues_split
2007-04-16~10:10:02.57 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~10:10:02.57 but that's not long enough! (5250)
2007-04-16~10:10:02.57 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~10:10:02.57 ( 0, 5062) Queued
2007-04-16~10:10:02.57 ( 5063, 8307) Empty
2007-04-16~10:10:02.57 ( 8308,11460) Queued
2007-04-16~10:10:02.57 (11461,16617) Queued
2007-04-16~10:10:02.57 (16618,19947) Empty
2007-04-16~10:10:02.57 (19948,23246) Queued
2007-04-16~10:10:07.57 using compression type 0 (no compression)
2007-04-16~10:10:07.57 Local 2 found no ranges after full
2007-04-16~10:10:07.57 Local 2 check_out_job got frames 0 - 5062
2007-04-16~10:10:07.57 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:10:07.57 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~10:10:07.57 Splitter: fewer queues than max_queues_split
2007-04-16~10:10:07.57 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-16~10:10:07.57 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~10:10:07.57 Local 2 sent zone string ""
2007-04-16~10:10:07.57 but that's not long enough! (5250)
2007-04-16~10:10:07.57 Local 2 sent range (0,5062)
2007-04-16~10:10:07.57 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~10:10:07.57 ( 0, 5062) Working (Local 2)
2007-04-16~10:10:07.57 ( 5063, 8307) Empty
2007-04-16~10:10:07.57 ( 8308,11460) Queued
2007-04-16~10:10:07.57 (11461,16617) Queued
2007-04-16~10:10:07.57 (16618,19947) Empty
2007-04-16~10:10:07.57 (19948,23246) Queued
2007-04-16~10:10:07.57 Local 2 agent-based encoding; waiting for agent to finish
2007-04-16~10:10:07.96 Local 2 process_job failed with "End_of_file"; putting job (0,5062) back
2007-04-16~10:10:07.96 put back range
2007-04-16~10:10:07.96 ( 0, 5062) Queued
2007-04-16~10:10:07.96 ( 5063, 8307) Empty
2007-04-16~10:10:07.96 ( 8308,11460) Queued
2007-04-16~10:10:07.96 (11461,16617) Queued
2007-04-16~10:10:07.96 (16618,19947) Empty
2007-04-16~10:10:07.96 (19948,23246) Queued
2007-04-16~10:10:07.96 Local 2 close connection
2007-04-16~10:10:07.98 using compression type 0 (no compression)
2007-04-16~10:10:07.98 Local 2 found no ranges after full
2007-04-16~10:10:07.98 Local 2 check_out_job got frames 0 - 5062
2007-04-16~10:10:07.98 Splitter: fewer queues than max_queues_after_full
2007-04-16~10:10:07.98 Local 2 sent full AVS info 33689C295A0DFBF882BE7FDCBA781B14 = "G:\/dvb/x264.avs"
2007-04-16~10:10:07.98 Splitter: fewer queues than max_queues_split
2007-04-16~10:10:07.98 Local 2 sent video info 1280x720 @ 60000/1001
2007-04-16~10:10:07.98 split_largest_range found the biggest empty range to be 3330 frames long
2007-04-16~10:10:07.98 Local 2 sent zone string ""
2007-04-16~10:10:07.98 but that's not long enough! (5250)
2007-04-16~10:10:07.98 Local 2 sent range (0,5062)
2007-04-16~10:10:07.98 Splitter: split_largest_range did nothing, but there are enough queues
2007-04-16~10:10:07.98 ( 0, 5062) Working (Local 2)
2007-04-16~10:10:07.98 ( 5063, 8307) Empty
2007-04-16~10:10:07.98 ( 8308,11460) Queued
2007-04-16~10:10:07.98 (11461,16617) Queued
2007-04-16~10:10:07.98 (16618,19947) Empty
2007-04-16~10:10:07.98 (19948,23246) Queued
2007-04-16~10:10:07.98 Local 2 agent-based encoding; waiting for agent to finish

and agent log
2007-04-16~10:10:02.00 x264farm version 1.09-163
2007-04-16~10:10:02.03 Using config file ".\config.xml"
2007-04-16~10:10:02.03 Temp dir: "E:\x264farm\temp\agent\temp"
2007-04-16~10:10:02.03 Ports: 50700 - 50703
2007-04-16~10:10:02.03 x264: "x264.exe"
2007-04-16~10:10:02.03 nice: 10
2007-04-16~10:10:02.03 agents: 1
2007-04-16~10:10:02.03 agent bases:
2007-04-16~10:10:02.03 [exact]
2007-04-16~10:10:02.03 pipe: false
2007-04-16~10:10:02.03 compressions: 0001
2007-04-16~10:10:02.03 deleting old files: after 604800 seconds
2007-04-16~10:10:02.03 trying port 50700
2007-04-16~10:10:02.04 bound to port 50700
2007-04-16~10:10:02.04 listening...
2007-04-16~10:10:02.04 working!
2007-04-16~10:10:07.10 Got a connection from 169.254.25.121:1751
2007-04-16~10:10:07.10 working!
2007-04-16~10:10:07.10 2 first pass frames 0 to 5062
2007-04-16~10:10:07.10 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.10 2 agent-based encoding
2007-04-16~10:10:07.10 2 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 40ebd3.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.50 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 40ebd3.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.51 Got a connection from 169.254.25.121:1756
2007-04-16~10:10:07.51 working!
2007-04-16~10:10:07.51 3 first pass frames 0 to 5062
2007-04-16~10:10:07.51 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.51 3 agent-based encoding
2007-04-16~10:10:07.51 3 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 26cabf.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.54 Got a connection from 169.254.25.121:1750
2007-04-16~10:10:07.54 working!
2007-04-16~10:10:07.54 4 first pass frames 8307 to 11460
2007-04-16~10:10:07.54 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.54 4 agent-based encoding
2007-04-16~10:10:07.54 4 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 124d23.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.59 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 26cabf.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.60 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 124d23.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.60 Got a connection from 169.254.25.121:1757
2007-04-16~10:10:07.62 working!
2007-04-16~10:10:07.62 Got a connection from 169.254.25.121:1758
2007-04-16~10:10:07.62 working!
2007-04-16~10:10:07.62 5 first pass frames 0 to 5062
2007-04-16~10:10:07.62 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.64 5 agent-based encoding
2007-04-16~10:10:07.64 5 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats a7e4e6.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.64 6 first pass frames 8307 to 11460
2007-04-16~10:10:07.65 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.65 6 agent-based encoding
2007-04-16~10:10:07.65 6 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 37aaac.txt" --seek 8307 --frames 3154 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.70 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats a7e4e6.txt\\\" --seek 0 --frames 5063 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.70 Thread received exception "Sys_error(\"nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats \\\"E:\\\\x264farm\\\\temp\\\\agent\\\\temp\\\\stats 37aaac.txt\\\" --seek 8307 --frames 3154 -o NUL \\\"G:\\\\dvb\\\\x264.avs\\\": No error\")"
2007-04-16~10:10:07.71 Got a connection from 169.254.25.121:1759
2007-04-16~10:10:07.71 working!
2007-04-16~10:10:07.71 Got a connection from 169.254.25.121:1760
2007-04-16~10:10:07.71 working!
2007-04-16~10:10:07.73 7 first pass frames 0 to 5062
2007-04-16~10:10:07.73 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.73 7 agent-based encoding
2007-04-16~10:10:07.73 7 doing the following:
nice -n 10 x264.exe --second --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 60000/1001 --pass 1 --stats "E:\x264farm\temp\agent\temp\stats 0c36c2.txt" --seek 0 --frames 5063 -o NUL "G:\dvb\x264.avs"
2007-04-16~10:10:07.75 8 first pass frames 8307 to 11460
2007-04-16~10:10:07.75 Testing file "G:\dvb\x264.avs": FOUND!
2007-04-16~10:10:07.75 8 agent-based encoding
2007-04-16~10:10:07.75 8 doing the following:

foxius

16th April 2007, 17:41

harissa

Assuming you're using the following command line, you're not providing options for the "--first" argument. I think you should remove it if you don't want to pass any first pass options (i might be wrong though)

controller.exe --bitrate 3000kbps --avs "G:\dvb\x264.avs" --first --second -o "G:\ka.mkv"

harissa

16th April 2007, 18:42

great its working
but with a simple avs . i got 4 therad running at 25% each = 100%
i have a quad core
once i input some dll in the avs , i got only 2 thread of x264 runnin at 25% each = 50% , how can i push it to use 4 thread

harissa

16th April 2007, 19:02

nevermind i found it
but i wonder if there s an option like in megui called turbo , that really boost second pass speed ?

foxius

16th April 2007, 19:31

nevermind i found it
but i wonder if there s an option like in megui called turbo , that really boost second pass speed ?

afaik it boosts only the first pass.

just select your options in MeGUI and use the command line it generates with x264farm. But you should remove some of the options - consult the manual or this thread, i've seen an advice regarding this subject.

harissa

16th April 2007, 21:11

with a quad core, the setting in config.xlm , says
<number>4</number> <!-- How many jobs to send to the agent at one time.

so it s supposed to run 4 x264.exe
but when i use this avs :
loadplugin("DGDecode.dll")
Mpeg2Source("carnavale.d2v")
he really load 4 x264 at once and the cpu run at 100%
, but when i add
loadplugin("Select2.dll")
it drop to 2 thread of x264.exe , and it run the first pass at 50% ,
only the second pass load all 4 x264.exe

with selecteven() it run only 3 x264.exe , instad of 4

legoman666

19th April 2007, 22:51

Hate to keep bugging you all, but i have a small feature request :D

Im on a 10/100MB lan, using a farm of 6 computers. During the first pass, my bandwidth usage is at 99-100%, if i enable compressions on the 2 clients that are the fastest (and thus demand the most data), my lan usage drops to ~30-50%.

but during the second pass, which is much more cpu intensive pre frame, my lan usage is never more than 10%, and spikes whenever a client finishes a job. But the controller is still compressing the data it sends to the 2 fastest clients. So the controller is wasting cpu time compressing the data during the second pass that need not be compressed.

put simply, can you add an option to only enable compression during the first pass? It'd be extremely useful for people in my position :D Thanks :D

SpAwN_gUy

20th April 2007, 09:42

put simply, can you add an option to only enable compression during the first pass? It'd be extremely useful for people in my position :D Thanks :Dwell, you can workaround that...
1) finish First pass WITH compression
2) on the start of Second pass terminate "controller"
3) add "--nocomp" option to the controller cmdLine
4) start controller withOUT the compression :) ...

for me.. agent-based encodings are the greatest :)

omion

20th April 2007, 22:47

@legoman666:
That'd be an easy addition for the next version. It makes sense too. SpAwN_gUy's workaround will work in the meantime.
[edit] By the way, how does setting compression affect the overall encoding speed?

@harissa:
That's a bit odd. How long is the video, and large is the --batch parameter? Also, does the controller think it's sending out 4 jobs?

@all:
Sorry about the complete lack of upgrades. I've been working full-tilt on my other program, so I haven't had any time to do anything with x264farm... I should be getting back to x264farm sometime next week.

legoman666

21st April 2007, 00:14

@legoman666:
That'd be an easy addition for the next version. It makes sense too. SpAwN_gUy's workaround will work in the meantime.
[edit] By the way, how does setting compression affect the overall encoding speed?

Using compression on 2/6 clients, controller.exe cpu usage jumps from ~10% to ~30%, but lan usage drops way down and all the clients that were being choked by lack of bandwidth all get a nice speed increase.

Anyway, I got a bug report :D check the thread here: http://forum.doom9.org/showthread.php?t=124950

Im getting green block artifacts on keyframes (I think they're keyframes, hard to say) when I use x264farm but not when I use megui.

omion

22nd April 2007, 04:47

Well, I updated x264 to revision 651, which should be chock full of tasty speed improvements since r634. Get it here (http://omion.dyndns.org/x264/32/x264-651_omion.rar) or on the first post. I don't have any time to test it, though...

[edit: actually, that's revision 652... SVN must have been updated between the time I checked it and the time I downloaded it. Since the only change is a compiler tweak, I don't think there's any need to update the numbers...]

rhodges

24th April 2007, 03:27

I am having a problem where sometimes a thread in agent.exe dies and it does not notify the controller and this causes the controller to hang.

From the controller log

2007-04-23~20:50:44.00 DESKTOP 1 sent full AVS info F10F4738AA6B93F016C4E3DD695756AB = "\/192.168.1.5/z$/NetworkEncoder/Source/Lost Series 02/Lost S02e20.avs"
2007-04-23~20:50:44.00 DESKTOP 1 sent video info 720x400 @ 24000/1001
2007-04-23~20:50:44.00 DESKTOP 1 sent zone string ""
2007-04-23~20:50:44.00 DESKTOP 1 sent range (62239,62281)
2007-04-23~20:50:44.01 DESKTOP 1 no attempted bits found; using 3187909153.107259 / 3605544280.000000 (+4696)
2007-04-23~20:50:44.01 DESKTOP 1 optimal bitrate is 7; minimum is 7
2007-04-23~20:50:44.01 DESKTOP 1 sent target bitrate 7 (really want 4.470565)
2007-04-23~20:50:44.03 DESKTOP 1 agent-based encoding; waiting for agent to finish

From the agent log:

2007-04-23~20:50:43.17 3 agent-based encoding
2007-04-23~20:50:43.17 3 doing the following:
nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 7 --fps 24000/1001 --pass 3 --stats "c:\temp\agent\stats 8409bf.txt" --seek 62239 --frames 43 -o "c:\temp\agent\output 256df9.mkv" "\\192.168.1.5\Z$\NetworkEncoder\Source\Lost Series 02\Lost S02e20.avs"
2007-04-23~20:50:43.46 Thread received exception "Sys_error(\"nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 7 --fps 24000/1001 --pass 3 --stats \\\"c:\\\\temp\\\\agent\\\\stats 8409bf.txt\\\" --seek 62239 --frames 43 -o \\\"c:\\\\temp\\\\agent\\\\output 256df9.mkv\\\" \\\"\\\\\\\\192.168.1.5\\\\Z$\\\\NetworkEncoder\\\\Source\\\\Lost Series 02\\\\Lost S02e20.avs\\\": No error\")"

It's strange because the error is "No Error".

The computer is a Vista x64, E6600 with 2GB RAM, running both the controller and agent. I believe I've run into this before on a couple of my other computers but I haven't gotten it lately and never looked at the log files.

It doesn't always happen, it just happens randomly. I am going to see if I get the error if I disable "nice" to see if maybe the error is there? (I was using the latest nice that works on Vista). Again, I thought I was encountering this error on my XP and 2003 machines as well (but I haven't seen the problem lately).

Any ideas?

omion

24th April 2007, 04:50

@rhodges:

I'm pretty sure the "no error" thing is just because Windows doesn't have the same quality of thread / signal handling that Unix does, or maybe it's just the Windows port of OCaml. Either way, "no error" is actually an error.

I don't know exactly what's causing the problem, but since the agent can report it, then it should be able to tell the controller. I'll see if I just missed part of the exception-handling code somewhere.

Is that the end of the agent's log, or is there anything more?

[edit: dang, that's a lot of backslashes... I'll see if I can fix that too ;) ]

rhodges

24th April 2007, 12:19

Is that the end of the agent's log, or is there anything more?

There is more, but it's just the other the other thread doing its thing. (It's configured to run 2 jobs).

I tried it w/o nice and I still got the same error.

kdf

24th April 2007, 13:52

Wow, 5 days before I could post this !

First off thanks for your hard work.. it is appreciated.

You have a small error in the controller make file, something which probably wouldnt show up when compiled in windows. In the makefile you refer to "extlib.cmxa" on my system extlib was compiled as "extLib.cmxa"

Nothing serious but worth pointing out for linux newbies that might not know where to look.

Now, onto my experiences.. my setup

System 1 - WIndows 2003 Server (Controller)
System 2 - Windows XP (Agent1)
System 3 - Linux Dual Core (Agent2)
System 4 - Linux (Agent3)
System 5 - Linux (Agent4)
System 6 - Linux (Agent5)

When encoding the first pass all systems check out a job and begin working. However each agent only uses between 5-15% CPU. Network utilisation on the agents is also low (10%).

The problem seems to lie in the controller, the CPU is bouncing off 100% most of the time with network being about 70% utilisation. I figured that this was due to the non agent based encoding. So going back to basics I set it up for one controller, one agent with agent based encoding.

No matter what I do all I get is an error thrown back by x264 stating "no valid output file" the command being sent to the agent indicates an output of

"-o /dev/null"

Which would be correct for the first pass ! the agent has network access to the .avs, .d2v and its own local copy of avs2yuv and x264. When running the agent finds the remote .avs file correctly so the <base> must be correclty set. All the temp storage areas are also correctly set.

any ideas what could be causing this ?

foxius

24th April 2007, 14:09

foxius

24th April 2007, 14:18

Which would be correct for the first pass ! the agent has network access to the .avs, .d2v and its own local copy of avs2yuv and x264. When running the agent finds the remote .avs file correctly so the <base> must be correclty set. All the temp storage areas are also correctly set.

any ideas what could be causing this ?

Make sure that:

1. .avs, .d2v and source files (e.g. .vob files) are accessible by agent PCs, which is true in your case. They should be accessible either at network share (mounted at base path) or locally (located at base path or via absolute path in .avs/.d2v files)

2. all avisynth plugins (.dll) listed in your .avs script are accessible either locally or at network share.

legoman666

24th April 2007, 14:44

I'm converting my Ghost in the Shell 2: Innocence blu ray to h264 and for some reason, x264farm isn't encoding at the specified bitrate. Here's the command line options I used:
controller.exe -B 10654kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --ratetol 2.0 --me dia --progress --no-dct-decimate --no-psnr --no-ssim" --second "--ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --ratetol 2.0 --me umh --progress --no-dct-decimate --no-psnr --no-ssim" --avs "C:\innocence.avs" -o "C:\innocence.mkv"

The movie is ~1.5 hours long and as you can see, I told it 10654kbps. The resulting .mkv file x264 outputs is about 100mb and looks like 100kbps instead of 10000. The source is just fine. Don't know what the porblem here could be.
http://img129.imageshack.us/img129/5359/vlcsnap5516607sp5.jpg

708145

24th April 2007, 15:18

I'm converting my Ghost in the Shell 2: Innocence blu ray to h264 and for some reason, x264farm isn't encoding at the specified bitrate. Here's the command line options I used:
controller.exe -B 10654kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --ratetol 2.0 --me dia --progress --no-dct-decimate --no-psnr --no-ssim" --second "--ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --ratetol 2.0 --me umh --progress --no-dct-decimate --no-psnr --no-ssim" --avs "C:\innocence.avs" -o "C:\innocence.mkv"

I think you should omit the bitrate in the first pass.

legoman666

24th April 2007, 18:50

I think you should omit the bitrate in the first pass.

the bitrate is not in the first pass, it is outside of it, as you can see. My command for this encode is no different than my other encodes with x264farm, which have been successful in this format.

omion

24th April 2007, 21:15

The movie is ~1.5 hours long and as you can see, I told it 10654kbps. The resulting .mkv file x264 outputs is about 100mb and looks like 100kbps instead of 10000. The source is just fine. Don't know what the porblem here could be.

Yuck. It looks like my ratecontrol equation exploded. That's just too high a bitrate for it to handle!

If you look in the log, you'll probably see something like:
2007-04-24~11:00:11.07 Expect XYZ bits with ratefactor 0.0000
2007-04-24~11:00:11.09 WARNING: 2pass curve failed to converge

Which means it's trying to encode to as low a bitrate as possible. I'll see if I can tweak it to handle that correctly...

rhodges

24th April 2007, 22:37

I've seen the warning:

WARNING: 2pass curve failed to converge

with 1500kbps encodes. Is it only bad when seen with:

Expect XYZ bits with ratefactor 0.0000

?

omion

24th April 2007, 23:22

@rhodges:

Well... it's always "bad" in the sense that you're not going to get the bitrate you specified. However, it will only look bad if the final ratefactor is 0.

Generally, this means that you are trying to encode to a bitrate that is very far off from the first pass bitrate, and the predictions say that it can't be done (either too big or too small)

It looks like the problem with legoman666's encode is that two ratefactors have the same bitrate, so the algorithm that changes the ratefactor can't fit to a certain bitrate without using infinity. OCaml seems to actually use negative infinity, which then gets turned into 0.

It should be an easy fix, but I'll have to do a bit of regression testing too.

legoman666

24th April 2007, 23:50

ah, thanks for the clarification, I eagerly await the update :D

What's the max bitrate x264farm can handle? 10654 is kind of overkill, its just the max bitrate that would make the final file with the audio fit onto a dual layer DVD.

omion

25th April 2007, 00:12

What's the max bitrate x264farm can handle? 10654 is kind of overkill, its just the max bitrate that would make the final file with the audio fit onto a dual layer DVD.

Depends on the source. The max bitrate is however many bits you get out of an encode where all the frames are at --qpmin.
It's more likely that the bitrate will be hit if it is near the first pass bitrate. So doing a first pass with "--qp 10" will be more likely to actually encode to an insane bitrate like 10654 :p

OK. Try out this controller (http://omion.dyndns.org:8080/controller-1.09+2_ratecontrol_tweak.rar) and post the part of the out-dump.txt file where it says something like:
2007-04-24~15:35:09.60 YOU GOT TO THE RATECONTROL! 0.000000 bits / 0.000000 bits, or mult -1.#IND00
2007-04-24~15:35:09.60 4443605833.333333 bits for usage
2007-04-24~15:35:09.71 Avail: 4443605833.333333
2007-04-24~15:35:09.71 SP exp: 19324850.135399
2007-04-24~15:35:09.73 Expect 1919500010.512299 bits with ratefactor 229.9426
2007-04-24~15:35:09.75 Expect 37065704.310011 bits with ratefactor 0.0000
2007-04-24~15:35:09.75 Chose 538.2664 from (0.0000,37065704) to (229.9426,1919500011) (32 steps left)
2007-04-24~15:35:09.78 Expect 2051434512.139841 bits with ratefactor 538.2664
2007-04-24~15:35:09.78 Chose 6128.6425 from (229.9426,1919500011) to (538.2664,2051434512) (31 steps left)
2007-04-24~15:35:09.79 Expect 2051434512.139841 bits with ratefactor 6128.6425
2007-04-24~15:35:09.79 BITRATE FLATLINE AT 2051434512. Decrease --qpmin or the target bitrate to get an accurate result

Note that you only need to start encoding the second pass to see this block of text. As long as your first pass settings are the same, you should see it right away, then you can cancel the encode.

legoman666

25th April 2007, 01:03

I'm not sure if it matters, but since I already deleted all the temp files from the last attempt at the encode, I had to redo the first pass. To save time, i only had it do 1000 frames. Anyway, here's the text from the log:

2007-04-24~20:04:13.28 Number of GOPs done: 0 / 7
2007-04-24~20:04:13.28 Looks like the second pass has not finished; using 0 for the number of GOPs computed
2007-04-24~20:04:13.28 Max GOP threshold is 7 GOPs (out of 7) or a ratio of 0.800000
2007-04-24~20:04:13.28 YOU GOT TO THE RATECONTROL! 0.000000 bits / 0.000000 bits, or mult -1.#IND00
2007-04-24~20:04:13.29 444804943.916667 bits for usage
2007-04-24~20:04:13.29 Avail: 444804943.916667
2007-04-24~20:04:13.29 SP exp: 6127739.005974
2007-04-24~20:04:13.29 Expect 132361430.293441 bits with ratefactor 72.5888
2007-04-24~20:04:13.31 Expect 9872760.918776 bits with ratefactor 0.0000
2007-04-24~20:04:13.31 Chose 257.7478 from (0.0000,9872761) to (72.5888,132361430) (32 steps left)
2007-04-24~20:04:13.31 Expect 398101841.677236 bits with ratefactor 257.7478
2007-04-24~20:04:13.31 Chose 290.2890 from (72.5888,132361430) to (257.7478,398101842) (31 steps left)
2007-04-24~20:04:13.31 Expect 410522678.328337 bits with ratefactor 290.2890
2007-04-24~20:04:13.31 Chose 380.1046 from (257.7478,398101842) to (290.2890,410522678) (30 steps left)
2007-04-24~20:04:13.31 Expect 418102594.118023 bits with ratefactor 380.1046
2007-04-24~20:04:13.32 Chose 696.5048 from (290.2890,410522678) to (380.1046,418102594) (29 steps left)
2007-04-24~20:04:13.32 Expect 418102594.118023 bits with ratefactor 696.5048
2007-04-24~20:04:13.32 BITRATE FLATLINE AT 418102594. Decrease --qpmin or the target bitrate to get an accurate result
2007-04-24~20:04:13.32 GOP 0 (109 frames) @ 0.000000 = 18316332.59 bits, or 4028.925040 kbps, min 189827 (41.755016 kbps)
2007-04-24~20:04:13.32 GOP 1 (117 frames) @ 0.000000 = 93085444.08 bits, or 19075.374693 kbps, min 222308 (45.556085 kbps)
2007-04-24~20:04:13.32 GOP 2 (212 frames) @ 0.000000 = 103591369.69 bits, or 11715.609261 kbps, min 423690 (47.916989 kbps)
2007-04-24~20:04:13.32 GOP 3 (133 frames) @ 0.000000 = 60512358.90 bits, or 10908.614796 kbps, min 319945 (57.676759 kbps)
2007-04-24~20:04:13.32 GOP 4 (250 frames) @ 0.000000 = 69884615.42 bits, or 6702.220860 kbps, min 522869 (50.145279 kbps)
2007-04-24~20:04:13.34 GOP 5 (179 frames) @ 0.000000 = 72209766.72 bits, or 9672.084348 kbps, min 398920 (53.433047 kbps)
2007-04-24~20:04:13.34 GOP 6 ( 1 frames) @ 0.000000 = 502706.71 bits, or 12052.908102 kbps, min 420 (10.069930 kbps)

edit: my p2 agent running your 651 build of x264.exe crashed as soon as it tried to start up x264-651_omion.exe. "x264-651_omion.exe has encountered a problem and needs to close.. blah blah"

omion

25th April 2007, 01:31

I'm not sure if it matters, but since I already deleted all the temp files from the last attempt at the encode, I had to redo the first pass. To save time, i only had it do 1000 frames. Anyway, here's the text from the log:
[ code... ]

Great! That's exactly what it should do. I'll do a bit more testing, and I should have a new version out soon.

Also note: all agents running your 651 build of x264.exe crashed as soon as they try to start up x264-651_omion.exe. "x264-651_omion.exe has encountered a problem and needs to close.. blah blah"
Err... did 634 do that? And what OS do you have? Does it crash if you run it without x264farm?

legoman666

25th April 2007, 01:36

Err... did 634 do that? And what OS do you have? Does it crash if you run it without x264farm?

Looks like I was mistaken, only 1 of the computers out of 6 running the 651 build crashed. And no, it didn't crash before when it was using 634.

its a p2 350mhz, windows xp sp2, voodoo 3 2000, 320mb ram

Awesome, the encoded file is now the proper bitrate :D 55mb for the 1000frames I tested instead of 92mb for 150000 frames it was before.

Is there a way to get x264farm to use the .stats file from the first pass created by megui? Since x264farm wasn't encoding the video correctly, I had megui do it instead. problem is, its been running for 10 hours and is only 23% done with the second pass. So i want to cancel it and have x264farm start over at the second pass. Is this possible?

[P]ako

25th April 2007, 01:39

Awesome! Thank you for it.

I just tried it and I love it. I used only 3 computers and I encoded a file in half of the time it'd have taken me with my fastest computer.

Question, can one do only a second pass with it? If so, how do we tell the program what stat file to use?

omion

25th April 2007, 02:17

ako;993309']Awesome! Thank you for it.

I just tried it and I love it. I used only 3 computers and I encoded a file in half of the time it've have taken me with my fastest computer.

Question, can one do only a second pass with it? If so, how do we tell the program what stat file to use?

x264farm will do both the first pass and the second pass. In fact, that's the only thing it can do. Use the -1 switch to set the first pass settings, and the -2 switch to set the second pass. Read the HTML file for info on which x264 options to leave out of the -1 and -2 strings.

@legoman:
Something may have changed in the source that doesn't work with Pentium 2s any more... (or it's MSVS screwing things up again)

If the computer is using agent-based encoding and <agentpipe> is 0, try out Sharktooth's newest build and see if it works. If it does, then I may need to tweak some settings in the compiler.

[P]ako

25th April 2007, 02:30

So, if I want to encode the same video again, I have to start from the 1st pass, right?

omion

25th April 2007, 02:43

Oops. Just noticed your edit.

Awesome, the encoded file is now the proper bitrate :D 55mb for the 1000frames I tested instead of 92mb for 150000 frames it was before.
Hooray!

Is there a way to get x264farm to use the .stats file from the first pass created by megui? Since x264farm wasn't encoding the video correctly, I had megui do it instead. problem is, its been running for 10 hours and is only 23% done with the second pass. So i want to cancel it and have x264farm start over at the second pass. Is this possible?

Kind of. If you know the filename of the first pass temp directory, you can insert the stats file in it and make x264farm start from there.

So, if you start up x264farm and the temp dir is:
somewhere\ABC.avs\1-XYZ
then you can delete the "1-XYZ" directory and save the stats file as
somewhere\ABC.avs\1-XYZ.txt
Then start up x264farm. It should start from the second pass.

The best way to find out what the first pass directory is, is to start up x264farm and it will say:
AVS temp dir is somewhere\ABC.avs
First string is "...", with MD5 "XYZ"

then you know where it's looking.

Note that you must delete the "1-XYZ" directory in order for this to work. If not, it will probably refuse to encode.

omion

25th April 2007, 02:56

ako;993322']So, if I want to encode the same video again, I have to start from the 1st pass, right?

x264farm keep all the 1st pass stats files, so if it notices you use the exact same settings for the first pass, it will start up from the beginning of the second pass. This means you can do the following:

x264farm -B 1000kbps -1 "abc" -2 "pqr"
x264farm -B 300kbps -1 "abc" -2 "xyz"

The second command will see that "abc" is the same for both, and re-use the first pass to make a completely different second pass.

[P]ako

25th April 2007, 03:12

Cool, thank you. I guess I will have to use --force to make it encode the second pass again since I will be using the same settings as before (just changes in the filter chain or subs).

omion

25th April 2007, 03:59

ako;993335']Cool, thank you. I guess I will have to use --force to make it encode the second pass again since I will be using the same settings as before (just changes in the filter chain or subs).

Ehh... actually, if you change the AVS file, it is considered a new file to encode. In that case, the first pass is not reused between the passes. Unfortunately, there is no way to make it re-encode between two different AVS files without moving around the stats files like I told legoman666.

Wishbringer

25th April 2007, 09:42

When using x264farm controller with options like:
controller.exe -B 100% -1 "whatever" -2 "whatever"
or
controller.exe -B "100%" -1 "whatever" -2 "whatever"

it tolds me that it is assuming 100 is percent.
Seems it doesnt recognize the percent-sign.
Maybe because % is used on other places as a placeholder for parameters:
tool.exe %1 %2 %3

Why not use "p" instead:
-B 99p and -B 1000kbps

SpAwN_gUy

25th April 2007, 10:17

omion

25th April 2007, 18:03

Wishbringer:
It should only do that inside a batch file. In that case, use two percents (like 100%%) to make a literal percent sign.

any of 64bit builds for me? ;)
Nope :( The 64-bit builds got pretty trashed after the last update. There's not a whole lot I can do, since I don't have any clue how to edit the assembly to fit Windows' style, or how to program any C to work around the other problems with it...

[P]ako

25th April 2007, 20:31

Ehh... actually, if you change the AVS file, it is considered a new file to encode. In that case, the first pass is not reused between the passes. Unfortunately, there is no way to make it re-encode between two different AVS files without moving around the stats files like I told legoman666.

Correct me if I am wrong. I was planning to trick "x264farm" by using an avs like this:

directshowsource("filtered_feed.avs")

Then I thought that filtered_feed.avs can be altered all you want, except the name. The controller would see no difference and start second pass. Does my theory make sense? I haven't tested it yet since my computers are encoding in an orgy of bytes.

omion

26th April 2007, 01:40

ako;993658']Correct me if I am wrong. I was planning to trick "x264farm" by using an avs like this:

directshowsource("filtered_feed.avs")

Then I thought that filtered_feed.avs can be altered all you want, except the name. The controller would see no difference and start second pass. Does my theory make sense? I haven't tested it yet since my computers are encoding in an orgy of bytes.

Heh :D That'll work. x264farm computes an MD5 on the AVS you feed it, but ignores anything else.

Note, however, that this is technically breaking the system. You will have to make sure to use the "--restart" option if you don't want to re-use the first pass. Failing to do so will result in Pain And Suffering™ and probably really bad quality.

(the "--force" option you mentioned before will only force encoding to start from the beginning of the second pass)

@all:
I'm almost done with the next version of x264farm. It should have a better filesize accuracy, as well as pass-dependent compression settings on the agent side. I just need to do a few more tests to make sure it works...

legoman666

26th April 2007, 03:58

I finished that encode at 10654kbps with x264farm. The quality is great, however, there is no way in hell that it actually encoded my source at 10654kbps. The final file size was ~4.2gb for a 1hour 40 minute movie. At 10654kbps, the file shoule have been ~7.8gb. 4.2gb means the actual bitrate was closer to 5700kbps.

Of course, the final file's quality looked just fine, but it's still not what I told it to do :D

Heh, I thought this might be amusing. I had 7 agents running on the first pass of an encode. Only one had compression enabled. I added another with compression enabled and the extra cpu usage actually lowered the total encoding speed.
http://img47.imageshack.us/img47/1139/hahaol6.jpg
The part of the graph where it goes from ~80% to 65% is when I added the extra agent. The total usage should've gone up, not down. Hehe, compression isnt always a good thing.

omion

26th April 2007, 06:25

I finished that encode at 10654kbps with x264farm. The quality is great, however, there is no way in hell that it actually encoded my source at 10654kbps. The final file size was ~4.2gb for a 1hour 40 minute movie. At 10654kbps, the file shoule have been ~7.8gb. 4.2gb means the actual bitrate was closer to 5700kbps.

Of course, the final file's quality looked just fine, but it's still not what I told it to do :D

Yeah. That's because x264 couldn't figure out how to use up all those bits. That's what the original problem was with the ratecontrol screwing up. If you really want that bitrate, lower the --qpmin setting (10 by default) However, --qpmin is set to 10 because there really isn't any quality gain below that.

Hehe, compression isnt always a good thing.
Definitely. In fact, it's generally a bad thing.

legoman666

26th April 2007, 06:57

omion

26th April 2007, 07:27

All righty, guys. 1.10 is out.

First, you should know that the config.xml files haven't changed (unless you want to add pass-specific compression) but you must use both the 1.10 versions of the controller and agent (no mixing 1.10 controller with 1.09 agents)

One of the big changes is that it should hit the bitrate target much more accurately now. I just finished a full-length movie test, and the resultant file size was off by ~0.2%, which should be much better than before. I introduced a new switch, --sizeprec, which will determine how much this is taken into account:

--sizeprec 0.0 will scale the local bitrate in order to maintain a consistent quality throughout the movie. This is the same as 1.09.
--sizeprec 1.0 will scale the local bitrate to hit the desired filesize without regard to fluctuating bitrate
A number between 0 and 1 will fade between the two
Anything below 0 or above 1 makes no sense. (although I just realized I didn't clamp it to that range... oops) :rolleyes:

Basically, if the first scenes are too small, using 0 will correct the following scenes to be the "right size". Using 1 will overcorrect the following scenes to make up for the missing bits from the first scenes.
The new default is 1, since the quality impact is extremely minor (some scenes don't get quite enough bits, other scenes get a bit too many) and it hits the overall bitrate much more accurately.

I also added pass-specific compression options, so legoman666 should replace the <compression> block in the agent's config.xml file to be like this:<passcompression>
<first>
<type>1</type>
</first>
<second>
<type>0</type>
</second>
</passcompression>
This will enable compression type 1 on the first pass, and only use type 0 (no compression) on the second.

I also kept in the change to the ratecontrol that will prevent it from panicking when the bitrate is too high. As legoman666 noted, even this change will not result in the desired bitrate, but it will look really good ;)

omion

26th April 2007, 07:42

What does it mean when one of the ranges has DELFIRST next to it? Whenever there is a range like that, the controller and the agent encoding that range exibit weird behavior..

( 0,107511) Full
(107512,107711) Queued
(107712,112877) Working (staircase 2) DELFIRST
(112878,156845) Full

DELFIRST is weird. Basically, when the controller notices a condition that may cause an infinite loop, the offending range is marked DELFIRST and treated very... carefully. So really, the strange behavior of the controller is to avoid the much worse behavior of an infinite loop.

You may have noticed that, in your example, nothing will work on the queued range until the DELFIRST range has been returned by "starecase 2". In the middle of an encode this doesn't matter, because there are other ranges to work on instead. At the end, though, it can stall like this since there aren't enough frames left to avoid the DELFIRST range.

There are still some optimizations that I can do that will minimize the stalling that DELFIRSTs cause, but nothing that would have helped in the example case. The holdup is always minor, and never more than 2*keyint (in your example it's 200 frames) so there's not really a big problem.

SpAwN_gUy

26th April 2007, 08:35

1.10 is out.. nice.. testing testing :)
i have two new sources :) ...

SpAwN_gUy

26th April 2007, 10:02

foxius

26th April 2007, 14:03

omion

27th April 2007, 05:04

SpAwN_gUy

27th April 2007, 08:31

Clearing the controller's temp folder helped helped me once, though it had to reencode from scratch... You may try it =)well. i already tried to delete SOME of the temp-Files,... but ;) this is funny: theese files, that were ReNcoded(same gops) the problem was there ;) ... and might be, that it IS good to report the continous problem... to fix it for future generations :) ..

SpAwN_gUy

27th April 2007, 09:45

and one more thing, i've asked about cheking of "Same bitrate AND same Filename" on questioning "it seems like this encode was already made. encode again?".. cause.. i'm playing around wit encoding quality-based and bitrate-Based.. so input AVSes are the same, and outputFiles and Bitrate Are different..
controller.exe --first "--bframes 3 --b-pyramid --direct auto --filter -2,-2 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --progress --no-psnr --no-ssim" --second "--ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-2 --subme 7 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-psnr --no-ssim --deadzone-inter 6 --deadzone-intra 6" --avs "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm.avs" --output "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm_80.mkv" --bitrate 80
controller.exe --first "--bframes 3 --b-pyramid --direct auto --filter -2,-2 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --progress --no-psnr --no-ssim" --second "--ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-2 --subme 7 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-psnr --no-ssim --deadzone-inter 6 --deadzone-intra 6" --avs "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm.avs" --output "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm_1440.mkv" --bitrate 1440kbps

driftr

28th April 2007, 05:21

hehe.... i'm back :) .... remmember me? ;) .. i had a problem with not muxed final MKV-File.. i still do ;) ... now it shows more interesting things...
Fatal error: exception Matroska.Matroska_error("A signed integer -14535679 was passed to write_uint_to_string") same place same guy :)

I'm having the same problem. I have a 42 minute tv ep I purchased off Amazons unbox. It's in some weird ASF wmv format so I can't clip out an intro bit w/o converting it to another format first, so I am choosing lagarith. It creates a 14gig file. megui handles this fine but when I use xfarm I get the same problem at the end. (by the way it'll nice to use 6 computers + 1 dedicated controller to take my encode time down to about 90 minutes from 4 hours)

The out-dump.txt file doesn't seem to end properly, the file reaches just over two megs and here's the last few lines
2007-04-27~10:22:37.21 Merging 630 / 864
2007-04-27~10:22:37.32 Merging 631 / 864
2007-04-27~10:22:37.43 Merging 632 / 864
2007-04-27~10:22:37.67 Merging 633 / 864

but in the command prompt with the controller command I get this
#0862: "862 - 74979 75228 f5b794.mkv"
#0863: "863 - 75229 75286 d1f21f.mkv"
Fatal error: exception Matroska.Matroska_error("A signed integer -36848815 was p
assed to write_uint_to_string")

Think the large source file size is losing sync during joining? I'm encoding it down to 300 megs but it always dies at a 200 meg joined mkv. The other thought is the lagarith file has audio which I'm not using...maybe I should try with no audio stream...

omion

28th April 2007, 06:33

omion

28th April 2007, 07:05

I FOUND IT!

It's a stupid (64-bit!) integer overflow error. If you are having this problem, look at the top of out-dump.txt, and it will probably say something like:
2007-04-27~10:41:30.34 INFO:
2007-04-27~10:41:30.35 i_fps_n = 12570329
2007-04-27~10:41:30.35 i_fps_d = 524288
2007-04-27~10:41:30.35 i_fps_f = 23.976000

i_fps_n / i_fps_d = actual framerate. Note that both i_fps_n and i_fps_d are fairly large numbers (much larger than the "exact" 24000 / 1001)

The problem is that in the Matroska file, the numbers get too large to even represent with a 64-bit integer. Part of the calculation is:
frame_num * i_fps_d * 1000000000

Using SpAwN_gUy's values for i_fps_d, this overflows the maximum 64-bit integer value on frame 17593.

I'll see if switching to floats gives an accurate result, but in the meantime, you should use the "assumefps" AVS filter with integers for both numerator and denominator. So, for 23.976fps stuff, put this at the end of the AVS file:
assumefps(24000,1001)
That gives small enough numbers for just about any movie. For any other frame rates, see the filter's page (http://avisynth.org/mediawiki/AssumeFPS).

If you do add the assumefps filter and you don't want to lose all your previously-computed work, you may copy the contents of the old AVS temp directory to the new one (x264farm will see the AVS file as a different one since it was changed)

harissa

28th April 2007, 18:24

is it possible to use a hub instad of a router ,
doses the x264farm support the ip like 169.254.25.121
and without gatway

omion

29th April 2007, 02:23

is it possible to use a hub instad of a router ,
doses the x264farm support the ip like 169.254.25.121
and without gatway

If you mean what I think you mean, then yes. ;) Basically, if you don't have a router, then a hub won't work. However, a switch will work great. Generally, if you go to a store and ask for a hub you get a switch anyway.

As for the IP addresses: if your computer can see something at that IP address, then x264farm can encode with it. These addresses can be local (192.168.*.*) or anywhere on the internet.

It sounds like you are interested in setting up a network without an internet connection. In this case, you need to connect every computer up to a switch. You should probably give every computer a manual IP address, then just tell x264farm to use those addresses.

driftr

29th April 2007, 05:15

I'll see if switching to floats gives an accurate result, but in the meantime, you should use the "assumefps" AVS filter with integers for both numerator and denominator. So, for 23.976fps stuff, put this at the end of the AVS file:
assumefps(24000,1001)

Worked like a charm, well I used 30000,1001 since I had 29.976 source, thanks! BTW I'm using a dual 3.6ghz xeon as my controller with a mix of 6 agents, took 109 min to convert a 42 min 640x480 avi to x264. Locally on the controller it would have taken over 4 hours! Now adding two more agents to test out the next conversion.

driftr

2nd May 2007, 20:24

omion

3rd May 2007, 00:37

Still testing it out, I'm loving the agent based encoding. At home with a P4 3.4GHz HT, a 800MHz athlon, a P4 2.4GHz HT, and a Dual Core 2Ghz laptop I was able to encode a 43 minute TV rip in under two hour with agent based encoding; was going just under realtime encoding. The file was just under 2 gigs so it was easy to copy to each computer, unlike the 12+ gig ones I've been trying at work.

I think I read something on this before but is there any way to use agent based encoding with the avs and video files being in unique locations? Not all the computers I use can say use the C drive and since it seems I have to have a relative path somewhere that limits the agent based encoding, which is so much happier. Anyway, thanks again!

You can use the <base> command in the agent's config.xml files to specify an alternate AVS search location. The exact semantics of the <base> line are a bit odd, but the example agent config.xml file goes over exactly how to use it.

Note that you have to make sure the paths from the AVS file to its source files are valid. That is, the AVS must be able to find its source files in the new location.

@all:
I'm currently updating my server, so the file availability will be a bit splotchy for the next day or two. Sorry!

omion

3rd May 2007, 06:33

1.11 is out!

Main improvements are the lack of crashing with strange fps values, and basing the second-pass equality tests on the output bitrate.

I also changed the second pass GOP storage from a B+tree to a red-black tree. There's a good chance that nobody would notice this change. However, I can rest easier knowing that the tree actually balances itself when a GOP is taken out of it. My B-tree implementation didn't have a very good deletion function...

Neither the agent nor the config.xml files have changed, so there is no need to update them. The only update to the agent was the version number.

My server update is mostly complete, so there shouldn't be too many 404s. However, I added a mirror to 1.11 on the first post, just in case.

SpAwN_gUy

3rd May 2007, 08:35

In case someone has forgotten.. i've updated my Signature with Link to
my TUTORIAL: How to Setup agent-based encoding with x264farm (http://forum.doom9.org/showthread.php?p=954207#post954207)

and.. @omion.. thanks for the tip with AssumeFPS... worked great.. and i'm going to 1.11 testings :) ... thanks..

rhodges

4th May 2007, 04:31

I'm trying the latest, 1.11 and I'm running into a problem. During the first pass, none of the agent's actually do anything.

In the dump I see

2007-05-03~22:14:13.00 4 doing the following:
nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "c:\temp\agent\stats b29e1a.txt" --seek 0 --frames 5128 -o NUL "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e01.avs"
2007-05-03~22:14:13.52 Thread received exception "Sys_error(\"nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats \\\"c:\\\\temp\\\\agent\\\\stats b29e1a.txt\\\" --seek 0 --frames 5128 -o NUL \\\"\\\\\\\\192.168.1.5\\\\Z$\\\\NetworkEncoder\\\\Source\\\\Black Adder Series 01\\\\Black Adder S01e01.avs\\\": No error\")"
2007-05-03~22:14:13.55 closing socket
2007-05-03~22:14:13.55 socket closed

If I take the command

nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "c:\temp\agent\stats b29e1a.txt" --seek 0 --frames 5128 -o NUL "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e01.avs"

and run it, I get

x264 [error]: no ratecontrol method specified
x264 [error]: x264_encoder_open failed

I didn't change any of my config files, I just dropped the new binaries over the old.

Should it perhaps pass the bitrate? I'm using Cef's 655 build. Maybe that's the problem?

Any help will be appreciated. Thanks!

Edit: Yeah, I guess I should read the changelog, it looks like 655 was changed to require a rate control
"require a ratecontrol method to be specified, it no longer defaults to cqp=26."

rhodges

4th May 2007, 15:36

A while back I posed about getting some random "No error" errors. I think I've narrowed down the problem.

2007-05-04~07:56:40.75 23 doing the following:
nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 22 --fps 30000/1001 --pass 3 --stats "c:\temp\agent\stats 1eacca.txt" --seek 49484 --frames 33 -o "c:\temp\agent\output 85bcd3.mkv" "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e04.avs"
2007-05-04~07:56:41.58 Thread received exception "Sys_error(\"nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 22 --fps 30000/1001 --pass 3 --stats \\\"c:\\\\temp\\\\agent\\\\stats 1eacca.txt\\\" --seek 49484 --frames 33 -o \\\"c:\\\\temp\\\\agent\\\\output 85bcd3.mkv\\\" \\\"\\\\\\\\192.168.1.5\\\\Z$\\\\NetworkEncoder\\\\Source\\\\Black Adder Series 01\\\\Black Adder S01e04.avs\\\": No error\")"
2007-05-04~07:56:41.59 closing socket
2007-05-04~07:56:41.59 socket closed

When the above happens, it seems the controller/agent loses track of this and never does anything with this thread again.

I wanted to see what the error was, so I ran the command line

nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 22 --fps 30000/1001 --pass 3 --stats "c:\temp\agent\stats 1eacca.txt" --seek 49484 --frames 33 -o "c:\temp\agent\output 85bcd3.mkv" "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e04.avs"

and I got back

x264 [error]: requested bitrate is too low. estimated minimum is 34 kbps
x264 [error]: x264_encoder_open failed

I'm not too sure why, but again, it seems like it enters this error condition and then the agent just forgets about the thread and the controller still thinks it's being worked on.

omion

5th May 2007, 03:55

and I got back

x264 [error]: requested bitrate is too low. estimated minimum is 34 kbps
x264 [error]: x264_encoder_open failed

I'm not too sure why, but again, it seems like it enters this error condition and then the agent just forgets about the thread and the controller still thinks it's being worked on.

That's interesting! I have seen that error quite a bit, but every time it's happened to me the agent sees that the thread croaked and starts up another one at a higher bitrate (like it's supposed to). I'll have to look at it a bit more.

Does it always happen when you encode something with a very low bitrate? That is, if you send "-B 1kbps" to the controller, will it lock up the agents fairly reliably? (and what OS are you running on your computers?)

Also, I've been thinking about being able to test if the agent - controller connection is dead, but until a few days ago I haven't been able to accurately reproduce that issue. However, I now found a way to artificially create the dead connection, so I'll start finding a way around it... the fix should may some light on the agent - x264 connection.

rhodges

5th May 2007, 05:16

Does it always happen when you encode something with a very low bitrate? That is, if you send "-B 1kbps" to the controller, will it lock up the agents fairly reliably? (and what OS are you running on your computers?)

I'm running four computers:

Agent x2 - XP - AMD X2 3800
Agent x2 - XP - Core2Duo E6400
Agent x2 - 2003 Advanced Server - AMD X2 4600
Agent x2 and Controller - Vista 64bit - Core2Duo E6600

(The Vista box is running 32bit x264, Cef's r654 build)

All the source files, avs, etc is hosted on a network share (the 2k3 machine), access via UNC paths with each machine using a local temp directory. The source is on an 8 drive RAID 5 array with all machines connected via gigabit.

I've seen the issue a couple of times on the XP and 2k3 machines, but I haven't had a problem in a long time. I can't even be 100% certain that was the problem.

I get the problem on the Vista machine. I've been doing my encodes at 1500kbps (just recently went to 1000kbps with the Black Adder, the other was Season 2 of Lost and Season 1/2 of Stargate Atlantis.)

Lost and Stargate encountered an error on nearly every episode. I've done 10 episodes of Black Adder now and I've only had the problem twice.

I'll give a shot with a low bitrate tomorrow, I'm going to let it run tonight and see if I can get thru my queue.

Thanks for all your work on this. Even with it hanging it has cut down my encoding time significantly. I would not be surprised if the hanging thread is due to Vista. I've found Vista to be an overall stable operating system, but it does have a few rough edges they really should have worked out before releasing it.

[P]ako

5th May 2007, 19:21

Omion, could you compile an agent for a 10.3.9/PPC Mac?

omion

6th May 2007, 00:06

@[P]ako:
I'll see what I can do. None of my current systems are PPC, but I can get access to one.

[P]ako

6th May 2007, 01:12

@[P]ako:
I'll see what I can do. None of my current systems are PPC, but I can get access to one.

Thank you! :)

rhodges

6th May 2007, 05:15

I confirmed I'm getting the error and hang with my XP machine in addition to the Vista machine. I am giving it a shot with -B 1kbps to see if it happens reliably.

-----
Edit:

None of the agents ran the second pass, seems the controller freaked out when I attempted -B 1kbps as you suggested.

2007-05-06~01:04:17.72 YOU GOT TO THE RATECONTROL! 0.000000 bits / 0.000000 bits, or mult -1.#IND00
2007-05-06~01:04:17.72 2012110.100000 bits for usage
2007-05-06~01:04:18.22 Avail: 2012110.100000
2007-05-06~01:04:18.22 SP exp: 156431871.819221
2007-05-06~01:04:18.30 Expect 248654180.006108 bits with ratefactor 0.0129
2007-05-06~01:04:18.38 Expect 248652798.442261 bits with ratefactor 0.0000
2007-05-06~01:04:18.38 Chose 0.0000 from (0.0000,248652798) to (0.0129,248654180) (32 steps left)
2007-05-06~01:04:18.47 Expect 248652798.442261 bits with ratefactor 0.0000

The last line kept repeating until I manually killed the controller (woke up to 35MB dump file).

Xitsa

8th May 2007, 18:34

I tried this beautiful stuff —it's amazing!

While I seek what to feed to x264farm I encounter strange thing:
on a simple AVS (let it be input.avs) like AVISource("...\some.avi") it stucks.
Controller gives only
agent_list 1:
///agents info
agent_list 2:
///agents info
and its all…

inputs.avs plays normally. avs2yuv shows frame numbers when asked to.

The thing strange because when I feed x264farm that AVS: DirectShowSource("input.avs") everything was OK.

May be it help somebody :)

And omion, greate thanks!!

SpAwN_gUy

10th May 2007, 09:09

AVISource("...\some.avi")
....
DirectShowSource("input.avs") everything was OK. the difference ... wait..
at first i thought... that your agents don't have a proper video decoder(agent-based encoding)..
then.. if you DDS another AVS - this will result.. input.avs will be decoded on your local system, so there will be no problem,..

but then.. if i open DDS.avs on another computer.. it would decode input.avs on the samePC... so.. i'm a bit confused :( ...
sorry for my lame answer..

the_corona

11th May 2007, 15:00

burfadel

11th May 2007, 16:25

I'm a bit of a newbie, so sorry if this is clear to everyone.

Is there a way I can use this to do only one pass encodes? Specifically I'd just like to split this command:
x264 --crf 21 --ref 6 --mixed-refs --bframes 6 --direct auto --b-pyramid --subme 6 --keyint 600 --min-keyint 25 --b-rdo --bime --weightb --filter -1,-1 --trellis 0 --analyse all --8x8dct --me umh --threads 1 --progress --output "F:\mkv\temp\output.mp4" "F:\mkv\temp\with_processing.avs"
but just spread across my 3 pc on the lan.

I noticed that when I do the following:
controller.exe --avs "F:\mkv\temp\with_processing.avs" -1 "--bitrate 800 --ref 6 --mixed-refs --bframes 6 --direct auto --b-pyramid --subme 6 --keyint 600 --min-keyint 25 --b-rdo --bime --weightb --filter -1,-1 --trellis 0 --analyse all --8x8dct --me umh --threads 1 --progress" --output "F:\mkv\temp\output_farm.mp4"
on the agents it turns out as:
nice -n 10 x264 --bitrate 800 --ref 6 --mixed-refs --bframes 6 --direct auto --b-pyramid --subme 6 --keyint 600 --min-keyint 25 --b-rdo --bime --weightb --filter -1,-1 --trellis 0 --analyse all --8x8dct --me umh --threads 1 --progress --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "F:\x264enc\agent-vista\tmp\stats ca2af6.txt" --seek 8183 --frames 3343 -o NUL "F:\mkv\temp\with_processing.avs"
notice the -o NUL which means no output file.

Is there a way then if only first pass is specified it will just output the result to an mkv file, return that and the controller joins it (which is the way I think 2nd pass works)?

Thanks for your help.
Corona

Nice settings, although I'd omit the threads options. Normally setting it to auto seems to be best... or at least set it to more than 1 for more efficient encoding speed. This is particularly true if you are using a dual core system (or more) threads set to 1 would be a speed killer.

Secondly, trellis should ultimately be set to 2 (the settings 'always'). This is from akupenguin, the author of x264 so he knows what he's talking about. There is a lot of misinformation out there about trellis and h264/x264 because it works different to Xvid. In xvid, setting trellis when on crf mode apparently leads to unpredictable results, but with x264 it improves the final result and leads to a slightly smaller file size (generally) :)

the_corona

11th May 2007, 16:39

Thanks burfadel, glad the settings are reasonable, I'm just beginning and am just experimenting.

I set threads to 1 because I have a single core System (A64 3200+@2200 - 2 GB - Vista64) so I figured I'd avoid the overhead of thread management and joining/syncinc threads and what not. I'll guess I'll return it to auto and I'll also try trellis, thanks for the tipp!

But what I really posted this for is to see if there is a way to split up one pass encodes with x264Farm :-)

Bit more info perhaps, I'm encoding dvb captures in 4:3 PAL (non-anime/cartoons) and am getting about 4,5 fps with the system above (avscript is simple deinterlace and some removegrain, no resize so 720x576). The source is relatively clean. Does that speed sound reasonable?

I have two more machines I'd like to ultimatly use to encode all those recordings I have to x264. The other machines are 1 Sempron64 3000+ - 1 GB XP SP2 and 1 P4 2,2 - 512 mb XP SP2).

The sempron machines gives me about 3,5 or so fps with x264 and the P4 seems really slow, don't really know whats going on, but I get <2 most of the time, but that would add up to (optimally, I know thats not what I would get really) about 9 fps, or at least twice as fast :-)

Thnx guys

0gg

11th May 2007, 20:38

is it possible to use MT.dll for multicore with the renderfarm ?

http://forum.doom9.org/showthread.php?p=975017#post975017

foxyshadis

13th May 2007, 09:05

MT/SetMTMode can be used in the avs with no detrimental effects to x264farm, as long as it works with the avisynth filters you're using.

You can probably fake out x264farm by using a --crf xx at the end of the second pass command; I don't know if that would totally freak out its ratecontrol though. (If it's the first pass avs speed you're worried about, you should specify an alternate, much faster one, with firstavs and fastavs.) I could see the usefulness of a one-pass option.

SpAwN_gUy

14th May 2007, 08:09

You can probably fake out x264farm by using a --crf xx at the end of the second pass command; I don't know if that would totally freak out its ratecontrol though. (If it's the first pass avs speed you're worried about, you should specify an alternate, much faster one, with firstavs and fastavs.) I could see the usefulness of a one-pass option.Yes, indeed.. but for now.. farm is only usefull for two-pass encodings...

and there are bitrate settings for "100%" and "1000kbps" ... well,.. for my uses.. 80% of quality ... is like... wa-a-ay too low bitrates.. i think i need to try setting it to something near 100% ....

and trick with output and -crf will not work.. i think,.. 'cause during first pass only .stats file is transferred to controller..
my thoughts lead me to ... some weird easy settings for the first pass.. and then "-crf" for the secondpass settings..
but.. ratecontroll will die ;) ... i think :) ...

the_corona

14th May 2007, 10:48

So I can just fake a first pass with very fast settings, maybe even use --seek and --noframes (is that what its called) to create an empty stats file, and then proceed to pass 2 with crf settings?

Or do I have to run the first past over all frames? If so, can you suggest a very fast setting that has the minimum to allow for correct stats?

I kinda want to use CRF as I'm not really concerned for a fixed file size. I just want "near" transparent quality at a reasonably low filesize. Plus a two pass just seems extremely slow, at 4fps avg each 20 min show that I want to encoded will take 5/6 hours or so.

Thnx alot!

Wishbringer

14th May 2007, 11:24

and there are bitrate settings for "100%" and "1000kbps" ... well,.. for my uses.. 80% of quality ... is like... wa-a-ay too low bitrates.. i think i need to try setting it to something near 100% ....

80% bitrate from 1st pass depends on settings of 1st pass:
Till rev.655 x264 defaulted to "-crf 26" and that created quite low bitrates.
So if you set for example "-crf 15" or "-crf 18" in 1st pass settings you get a higher bitrate in 2nd pass with "80%".

Wishbringer

14th May 2007, 11:30

So I can just fake a first pass with very fast settings, maybe even use --seek and --noframes (is that what its called) to create an empty stats file, and then proceed to pass 2 with crf settings?

Or do I have to run the first past over all frames? If so, can you suggest a very fast setting that has the minimum to allow for correct stats?

You must use 1st pass, because this pass is needed to find gops...
And in 2nd pass jobs are send to clients in gops.

But for that you can set in 1st pass quite low quality settings (diamond, no trellis, no rdo etc.)
But same refs and bFrames i think, to have same iFrames.

the_corona

16th May 2007, 10:07

Thnx for the help guys.

Can I ask just one more question. Would it be possible to enable one pass encodes in the future? And why in two pass modes does the one pass mode keep reencoding the same frames over and over (I think this has to do with keyframe intervals not being hit or something)?

But why go through all that trouble, the way I see it if a file is split into n parts where n = number of clients (or agent threads I guess) as long as that number is relatively low compared to the no. of frames of the encode, you have a couple of suboptimal I-frames. Is it really worth it reencoding again and again till you make them optimal?

I'd rather it just split the file into n-parts (in my case 3) and let each agent process their part and then just join the result. In CRF mode it shouldn't really matter, should it?

Thnx again

Wishbringer

16th May 2007, 12:32

But why go through all that trouble, the way I see it if a file is split into n parts where n = number of clients (or agent threads I guess) as long as that number is relatively low compared to the no. of frames of the encode, you have a couple of suboptimal I-frames. Is it really worth it reencoding again and again till you make them optimal?

It's suboptimal in time, except you have 3 identical clients.
Example: client1: P3 600MHz
client2: A64 3000+
client3: C2D QX6700
Result: client3 is finished after one hour, waiting for client2 four further hours, waiting for client1 the next week.

Suboptimal i-Frames: lets guess, you take a source b frame and encode it as an iFrame in your file. bframe have lesser quality than pframes and even lesser quality than iframes, so new resulting iframe has that lower quality too. All appending p and b frames would have lower quality too!!, because they depends on that bad iframe. So it's better to find scene changes and good gops and encode iframes from source iframes.
A gop could be of 250 frames, around 5 seconds, so you would have that 5 seconds of bad quality per wrong decission.

SpAwN_gUy

16th May 2007, 13:47

i can see onepass enodes in the future... $)
this should look like "Compressibility check" in GKnot..
it's like make a simple
..
# COMPRESSIBILITY CHECK
SelectRangeEvery(400,20)
...
it should depend on some controller params .. on exactly how to split.. and those scene scene changes in the middle of those ranges could be the split points...
this would happen in controller-first pass.. and in the controller-second pass we could split on found keyFrames..

/me .. dropped my Delphi GUI and downloaded MeGUI sources.. it seems like i'm a very poor C# programmer ;) .. i guess i need to find someone who actually participated in MeGUI_dev...

SpAwN_gUy

16th May 2007, 13:49

80% bitrate from 1st pass depends on settings of 1st pass:
Till rev.655 x264 defaulted to "-crf 26" and that created quite low bitrates.
So if you set for example "-crf 15" or "-crf 18" in 1st pass settings you get a higher bitrate in 2nd pass with "80%".thanks for the tip :) ... will do.. i think..

EDIT: BWAHAHAHAHA.... this one is my 100th post here ... for like..4 years.. damn.. exactly 4 years :) ... happy anniversary me ... :)

AVmaniac

19th May 2007, 11:24

.... over and over and over again my x264farm displays this message

x264 [warning]: Error: 2pass curve failed to converge
x264 [warning]: target: 39004.00 kbit/s, expected: 3781.41 kbit/s, avg QP: 10.00
11
x264 [warning]: try reducing target bitrate or reducing qp_min (currently 10)

and repeats the same ranges for more than a day now.
my target bitrate for the encode is 4Mbit for a 1920x832 DVB-S2
capture.

Disabled

19th May 2007, 11:38

my target bitrate for the encode is 4Mbit for a 1920x832 DVB-S2
capture.
Looks more like 40Mbit to me:

target: 39004.00 kbit/s

AVmaniac

19th May 2007, 11:55

Looks more like 40Mbit to me:

yep ... but only looks like

-B 4000kbps

also line2

x264 [warning]: target: xxxx kbit/s ....

differs from my post for the single involved ranges.

omion

25th May 2007, 07:14

All right. I'm back from the dead. :p I spent all my time working on my other program, but I'm out of ideas for it, so I'm back here!

@AVmaniac:
Check your PM

@the_corona:
I definitely err on the side of optimality, but the only time it should be chasing around I-frames is with very low-motion scenes like ending credits. Increasing the --batch length will help this problem a bit, but I'm working on a way to try to serialize those sequences to avoid the splitting problems. It won't be done for a long time, though, as it's not easy to do.

I think I've also figured out a way to check if the connection has died. It's annoyingly difficult to do without non-blocking IO, which I don't have access to...

All these ideas are in the pre-alpha stage, and actually most haven't left my head yet, so don't hold your breath. :rolleyes:

omion

31st May 2007, 01:28

1.12 is out! The config.xml files didn't change, nor did the switches, but the new controller won't work with 1.11 agents.

The most noticeable change is the output. It is now actually well-organized rather than spouting out a whole bunch of random garbage. (if you want the random garbage, check in out-dump.txt - it's still the same:p) Also, the line that says "THIS SHOULD NOT BE OVERWRITTEN!" will be taken out soon. It means just about nothing.

I also made a test to see if an agent-based computer has died silently. Each agent sends out a heartbeat signal every 10 seconds. If the controller does not receive a signal for 30 seconds, that agent is considered dead and its job put back. This should vastly help problems when the agents dying randomly and the controller waiting forever for the agent to finish. There is still no change to controller-based encodes, but those tend not to have the same problems.

I should go over the first-pass progress line. You'll see a line in the output that looks something like this:
##########_____###|||||||||||||....._______......####|||||||####...._____||||||||....._______.....
This is a representation of the video, with the entire video squashed into the width of the screen. Here's what the symbols mean:
. = empty
_ = queued
| = working ( ! = working with DELFIRST )
# = filled
It's quite likely that part of it will change from ".||." to "....", implying that no frames were done, but this is only due to the mapping from frames to characters. It's most likely that something was done, but filled frames are covered up by empty frames, so the filled frames just don't show.
And have fun watching it fill up!

SpAwN_gUy

31st May 2007, 08:39

SpAwN_gUy

31st May 2007, 12:59

hmm.. third encode.. and..
99% done (33812 / 34062) at 16.20 FPS
Agent | FPS | Description
pc79 1 | 3.12 | No jobs left; exited
pc79 2 | 3.04 | No jobs left; exited
pc206 1 | 2.80 | No jobs left; exited
pc206 2 | 2.75 | No jobs left; exited
pc419 1 | ? | No jobs left; exited
pc661 1 | 3.18 | No jobs left; exited
pc6 1 | ? | No jobs left; exited
pc2 1 | 2.86 | Dead?
pc418 1 | 2.05 | No jobs left; exited
pc82 1 | ? | No jobs left; exited
Recent errors:
and? .. yep, it seems to be dead.. there is no one connected to my controllerPC... and that's all.. no Jobs redoing.. just nothing :( ...

omion

1st June 2007, 05:32

Yeppie... testing testing :) ...

Okay... testencode of Credits of "Pirates of the Carrebean 2" - went nice :) ... but as far as it is 5000framse long i didnt' see any difference in the first-pass :) ...
but then my next encode.. hm.. Nice :) ... one episode of "anime".. first-pass 70fps, second-pass(currently in progress) ~ 22fps.. with 3 agents.. and "nice -20"...

//BTW,.. omion, have you tried that "meGUI x264farm Mod", i've sent you few days ago?
//what about those "zones" and "assorted params" in CommandLine generation?
//is it correct?
Nevermind. i've noticed -h param .. and now everything is clear to me :) ... (my thoughts were right..) thanks..
Ah! I completely forgot about that e-mail. Sorry. :o I'll test out the mod ASAP.

But yes, the zones are parsed the same way as in x264: zone1start,zone1end,b=1234/zone2start,zone2end,q=30/...

All the assorted params look good, too. And nicely bounded!

One minor nit-pick:
The 1073741823 in "scenes to re-encode" looks a bit odd. Even though that is the default value, it looks completely random. Maybe have a checkbox for "any" default selected, and when de-selected, have the user type in a number in the box. Also, it looks like that number is not bounded. I use 1073741823 because that is the largest integer that OCaml programs can deal with. (plus, typing in a much larger number in that box will result in a nasty message)

and what about that --logfile option to save "out-dump.txt" somewhere outside deafult location?
That will actually be a bit difficult. Right now the log file is opened way before the options are parsed. The way the controller is set up, it wouldn't be possible to delay opening the dump file at all. I may be able to do some magic to get the dump file to the right place afterwards (open dump -> parse options -> close dump -> move to correct location -> open dump again) I'll look into the best way to do this.

hmm.. third encode.. and..
99% done (33812 / 34062) at 16.20 FPS
Agent | FPS | Description
pc79 1 | 3.12 | No jobs left; exited
pc79 2 | 3.04 | No jobs left; exited
pc206 1 | 2.80 | No jobs left; exited
pc206 2 | 2.75 | No jobs left; exited
pc419 1 | ? | No jobs left; exited
pc661 1 | 3.18 | No jobs left; exited
pc6 1 | ? | No jobs left; exited
pc2 1 | 2.86 | Dead?
pc418 1 | 2.05 | No jobs left; exited
pc82 1 | ? | No jobs left; exited
Recent errors:
and? .. yep, it seems to be dead.. there is no one connected to my controllerPC... and that's all.. no Jobs redoing.. just nothing :( ...

Grr.:mad: I thought I fixed that. At least it knows it's dead now.:p I'm not quite out of ideas for this problem, so it could see a fix in the next version. Unfortunately I started on a great big overhaul of the first pass job distribution code, so half of the controller is a complete mess. It may take a bit of time.

(it's also odd that the agents have exited when it there are still some unencoded frames. The agents should only exit when all of the frames are complete. I'll look into that too)

ATT

1st June 2007, 09:02

is there an easy way to translate megui parameters for x264farm i can figure out --B and --avs but the other options i'm kinda lost

today i did my first encode using x264farm using General x264 parameters x264farm.html page i only edited the --B --avs <= (i created a script using megui script creator and then i just map it) used a 1080p 20MB mkv sample for my testing.
controller -B 2500Kbps --first "--crf 19 --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 5 --ref 8 --me umh --bframes 7 --weightb --b-pyramid --analyse all" --second "--trellis 2 --bime --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 7 --ref 12 --me umh --bframes 7 --weightb --b-pyramid --analyse all --8x8dct" --avs "X:\ xxx.avs" --output "output.mkv"

i wasn't sure if my PC #1 controller should also run a agent cause on my first test i saw controller only using 50% cpu it's a dual core machine so i made run an agent as well not sure if was ok but it run. memory jump to 2.34GB few times then it settle down to 1.2 with 75% to 90% cpu usage when encoding controller/agent did most the work PC#2 only did about 800 frames don't know about the settings in General x264 parameters page but using the same bitrate on megui with 1 PC it was definitely faster using megui using the PS3 profile.

i also tried pasting parts of this profile to x264farm controller but it keep giving me errors can't remember really well but something about it did not understand for example: --level 4.1

--pass 2 --bitrate XXX --stats ".stats" --level 4.1 --ref 16 --mixed-refs --bframes 3 --no-b-adapt --b-pyramid --bime --weightb --direct auto --trellis 1 --analyse p8x8,b8x8,i4x4 --threads auto --thread-input --progress --no-psnr --no-ssim --output "X:\XXX.mp4" "X:\XXXX.avs"

PS; does the controller need to run a agent too ?
or does the controller not help in encoding or is it secretly helping :) aside from splitting the work ?

infernix

2nd June 2007, 01:22

I've set up x264farm and gave it a shot. My setup:

- Debian Etch 64bit on a c2d 3ghz (gbit)
- On that c2d 3ghz box, winXP in qemu (realtek nic + driver with tcp offloading, max ~8000kb/sec so like 80mbit)
- Debian Etch 64bit on an A64 3000 (100mbit)
- Debian Etch 32bit on a Celeron D 3.2ghz (gbit, fileserver)

The source avs was only an avisource() for a 77gb huffyuv (1280x720, ~43mins) which I ran through avs2yuv on the winxp VM beforehand (needed directshow filters).

I've tried the controller-based method and it is way too slow. On the c2d I got only about 20% cpu usage on x264, 100% on the WinXP Qemu which was running the controller (so only 120 of 200% on the dual core box), and 30% on the other two boxes. It was a bit slower without compression. Obviously, this framesize is not suitable for controller based frameserving.

Which brings me to my next point: agent based encoding on linux. I was looking at the code and I didn't see why it wouldn't work. With wine you can set up a complete working environment with avisynth filters and VFW decoders in such a way that avs2yuv works exactly like on winwoes, with the one difference that you can't use DirectshowSource (at least I haven't had that much luck so far with wine cvs, maybe that could work too).

I've set this up on my boxes and this seems to work just fine. I had to alter agent.ml and replace 'avs2yuv' with 'wine avs2yuv' and had to set <base>./</base> because searching does not play nice; agents on linux are searching for windows paths (e.g. Z:\/some/path), but otherwise this seems fine. The fileserver reads and writes locally; the c2d reads with about 8-12mb/sec, the a64 with 3-5mb/sec, so this seems to work fine with my mixed 100/1000mbit setup.

The fundamental issues with this approach right now are:

avs2yuv eats a lot of cpu to decode to raw; this avs2yuv+x264 combo would be better suited for just mencoder, but unfortunately it doesn't support frame exact seeking (or does it? i'll look into this)
controller running on the qemu VM; this slows things a bit down as the encoded results are being sent here, so given these facts, I humbly beg you to write it so that it can be compiled on linux if at all possible (why wouldn't it, anyway?)

I am also getting an error that I can't quite comprehend:

Z:\series\BSG\tmp>x264farmer
Using config file "x:\avisynth\config.xml"
CONFIG:
temp_dir: "c:\\temp\\x264farm"
agent_list 1:
"believe 1" = 192.168.0.163 50700-50703
"whitebox 1" = 192.168.0.5 50704-50707
"spirit 1" = 192.168.0.1 50708-50711
agent_list 2:
"believe 1" = 192.168.0.163 50700-50703
"believe 2" = 192.168.0.163 50700-50703
"whitebox 1" = 192.168.0.5 50704-50707
"whitebox 2" = 192.168.0.5 50704-50707
"spirit 1" = 192.168.0.1 50708-50711
"spirit 2" = 192.168.0.1 50708-50711
Parsing bitrate 6000kbps
GOT NORMAL INFO "e05farm.avs: 1280x720, 25 fps, 62737 frames\013"
GOT FIRST INFO "e05farm.avs: 1280x720, 25 fps, 62737 frames\013"
GOT FAST INFO "e05farm.avs: 1280x720, 25 fps, 62737 frames\013"
INFO:
i_res_x = 1280
i_res_y = 720
i_fps_n = 25
i_fps_d = 1
i_fps_f = 25.000000
i_num_frames = 62737
i_bytes_y = 921600
i_bytes_uv = 230400
i_bytes_per_frame = 1382400
multiplier for kbps to fbits = 0.025000
fast_res_x = 1280
fast_res_y = 720
Oops. Writing the args failed with: Sys_error("c:\\temp\\x264farm\\option_log.tx
t.temp: No such file or directory")
FIRST PASS
THIS SHOULD NOT BE OVERWRITTEN!
83% done (52359 / 62737) at 24.28 FPS
###############################||||||||#########################|||||||#######
Agent | FPS | Description
believe 1 | 22.19 | Waiting for job
whitebox 1 | 6.87 | Doing frames 51941-57081
spirit 1 | 3.85 | Doing frames 25662-30900
Recent errors:

Notice the "Oops. Writing the args failed with: Sys_error("c:\\temp\\x264farm\\option_log.txt.temp: No such file or directory"). What exactly is the function of this, and is this a serious problem?

I hope to test this on ~20 core 2 duos (all linux) on a dual gbit lan with a 4TB iSCSI storage backend soon, so I would really appreciate any work you could do on the linux side; I'll happily test this. Replicating this test enviroinment really is quite easy as it's only a matter of installing x264, mounting ~/.wine and copying some files to ~/bin on all 20 boxes.

:thanks:

omion

2nd June 2007, 07:14

I've set up x264farm and gave it a shot. My setup:

- Debian Etch 64bit on a c2d 3ghz (gbit)
- On that c2d 3ghz box, winXP in qemu (realtek nic + driver with tcp offloading, max ~8000kb/sec so like 80mbit)
- Debian Etch 64bit on an A64 3000 (100mbit)
- Debian Etch 32bit on a Celeron D 3.2ghz (gbit, fileserver)

The source avs was only an avisource() for a 77gb huffyuv (1280x720, ~43mins) which I ran through avs2yuv on the winxp VM beforehand (needed directshow filters).

I've tried the controller-based method and it is way too slow. On the c2d I got only about 20% cpu usage on x264, 100% on the WinXP Qemu which was running the controller (so only 120 of 200% on the dual core box), and 30% on the other two boxes. It was a bit slower without compression. Obviously, this framesize is not suitable for controller based frameserving.

I'm not too framiliar with Qemu, but do you have it set up as dual-core? And were you also running an agent in the Qemu emulation, or was just the controller using 100%?

And yes, controller-based encoding can choke fairly easily, especially with that much data being passed around.

Which brings me to my next point: agent based encoding on linux. I was looking at the code and I didn't see why it wouldn't work. With wine you can set up a complete working environment with avisynth filters and VFW decoders in such a way that avs2yuv works exactly like on winwoes, with the one difference that you can't use DirectshowSource (at least I haven't had that much luck so far with wine cvs, maybe that could work too).

I've set this up on my boxes and this seems to work just fine. I had to alter agent.ml and replace 'avs2yuv' with 'wine avs2yuv' and had to set <base>./</base> because searching does not play nice; agents on linux are searching for windows paths (e.g. Z:\/some/path), but otherwise this seems fine.
What did you have <base> set to before that it would search in "Z:\/some/path"? The search function is not so smart, so it will look in Windows-style dirs first, but then it should sort itself out and look in the right place. For example, if the controller is getting the file from the Windows dir "Z:/some/path", and the agent has <base> set to "/temp/x264farm" the directories searched will be:
/temp/x264farm/Z:/some/path
/temp/x264farm/some/path
/temp/x264farm/path
/temp/x264farm
If not, then something's wrong.

The fileserver reads and writes locally; the c2d reads with about 8-12mb/sec, the a64 with 3-5mb/sec, so this seems to work fine with my mixed 100/1000mbit setup.

The fundamental issues with this approach right now are:

avs2yuv eats a lot of cpu to decode to raw; this avs2yuv+x264 combo would be better suited for just mencoder, but unfortunately it doesn't support frame exact seeking (or does it? i'll look into this)
Well, the only time avs2yuv is used for agent-based encoding is if you have the <agentpipe> setting on. Otherwise the agent simply calls x264 with the proper -seek and -frames settings. Also, I ran some tests on my Windows box and piping avs2yuv > x264 ran almost as fast as just having x264 do it. I don't remember the exact number, but it was something like 1% slower.

controller running on the qemu VM; this slows things a bit down as the encoded results are being sent here, so given these facts, I humbly beg you to write it so that it can be compiled on linux if at all possible (why wouldn't it, anyway?)

You're right that the controller should be able to be compiled on Linux. I try to make my code as portable as possible, although I haven't tried compiling the controller on anything but Windows so far.

I am also getting an error that I can't quite comprehend:

[code omitted]
Notice the "Oops. Writing the args failed with: Sys_error("c:\\temp\\x264farm\\option_log.txt.temp: No such file or directory"). What exactly is the function of this, and is this a serious problem?

That error means basically nothing. The controller will write its options to a file in the temp directory so that you (well, mainly me, since it's not really documented) can go back and see what you had last encoded. This is very helpful if the controller crashes and you want to pick up where you left off - you have to use the exact same settings or it will start from the beginning.

Actually, now that I look at the code, there's no way that file could be created on its own... hmm...

Try creating an empty file named "c:\temp\x264farm\option_log.txt" and see if that fixes the problem.

I hope to test this on ~20 core 2 duos (all linux) on a dual gbit lan with a 4TB iSCSI storage backend soon, so I would really appreciate any work you could do on the linux side; I'll happily test this. Replicating this test enviroinment really is quite easy as it's only a matter of installing x264, mounting ~/.wine and copying some files to ~/bin on all 20 boxes.

:thanks:

If you'd like, you can send all those computers to me and I'll test them for free :D

In the meantime, I'll see if I can get the controller compiled on my Linux box. I don't have Wine on it, so it's no good for encoding, but I should be able to fix something.

infernix

3rd June 2007, 17:49

I'm not too framiliar with Qemu, but do you have it set up as dual-core? And were you also running an agent in the Qemu emulation, or was just the controller using 100%?

Qemu-kvm doesn't do SMP yet, so no, it's just using one core. No agent inside the winxp VM, that would be really really slow :) So the controller, and thus the WinXP VM, is idle 90% of the time; the only time it really does anything is when it starts sectioning for the 1stpass, when it receives data, and in the very end when it merges all the mkvs.

What did you have <base> set to before that it would search in "Z:\/some/path"? The search function is not so smart, so it will look in Windows-style dirs first, but then it should sort itself out and look in the right place. For example, if the controller is getting the file from the Windows dir "Z:/some/path", and the agent has <base> set to "/temp/x264farm" the directories searched will be:
/temp/x264farm/Z:/some/path
/temp/x264farm/some/path
/temp/x264farm/path
/temp/x264farm
If not, then something's wrong.

I didn't use <base>, i used <bases> and that failed. the thing is, the agent is running on linux (so uses unix style paths), but the controller runs on windows (using windows style paths) and I think the agent validates a windows style path on linux (Z:\series\blah\tmp is Z:seriesblahtmp on linux), which fails for obvious reasons. The added twist here is that avs2yuv is in the middle; Wine will autoconvert unix style paths to windows style paths if it can find them. But it's not much of an issue, with <base>./</base> i just chdir to the shared directory with the .avi and avs on the NFS mountpoint and all is well.

Well, the only time avs2yuv is used for agent-based encoding is if you have the <agentpipe> setting on. Otherwise the agent simply calls x264 with the proper -seek and -frames settings. Also, I ran some tests on my Windows box and piping avs2yuv > x264 ran almost as fast as just having x264 do it. I don't remember the exact number, but it was something like 1% slower.

I have to use <agentpipe> because if I don't, it will try to run the linux x264 executable with an avs script. That fails horribly. Also, because Wine isn't SMP aware (http://bugs.winehq.org/show_bug.cgi?id=6383), avs2yuv and any other windows executable ran through Wine will only use 1 cpu. And yes, you're probably right that the difference between x264.exe versus avs2yuv.exe + x264 is negligible, but otoh I am quite sure that if we could only use mencoder, that'd be faster since there's no avisynth processing done in my case. But until I figure out how to do exact frame seeking with it, I'll leave it be; I'll take this up with the mplayer/ffmpeg devs.

That error means basically nothing. The controller will write its options to a file in the temp directory so that you (well, mainly me, since it's not really documented) can go back and see what you had last encoded. *snip* Try creating an empty file named "c:\temp\x264farm\option_log.txt" and see if that fixes the problem.

That does seem to resolve the issue; maybe add this to the code?

If you'd like, you can send all those computers to me and I'll test them for free :D In the meantime, I'll see if I can get the controller compiled on my Linux box. I don't have Wine on it, so it's no good for encoding, but I should be able to fix something.

I don't think my client would be very happy with that ;)

Some more questions i ran into:

Use <number pad="1"> to send one additional job to the agent during the second pass. Since the second pass jobs are so short (generally a few seconds) it is a good idea to have something else working in the background when a job gets done. This setting has no bearing on the first pass. I watched the x264 processes on the agent boxes, and generally it seems that when i set <number pad="1">1</number>, there's two processes started, but whenever one's finished, there's only one process left. It seems that only after that last one is finished, the data gets sent to the controller, which doesn't make it any faster than without pad="1". Is this a bug or am I missing the point here?
First pass statistics; the way I normally encode is to first run a 1stpass and then use the statistics from that to decide whether it needs a higher bitrate/avs tweaking. I am guessing that with this method it isn't possible to generate any sort of FRF or QP numbers? is it at all possible to combine the 1stpass data for the ranges, or to use a 1stpass stats file from a normal x264 1stpass?
It would be nice if the controller could spit out some numbers regarding performance, such as total time taken for 1stpass/2ndpass, total average fps for 1stpass and 2ndpass, and perhaps even time spent transferring data from agent <--> controller (because that time is spent doing nothing, at least the way i see it now)

I'm eagerly awaiting a linux buildable controller; no matter that it doesn't encode, right now the controller in my WinXP QEMU VM doesn't encode either.

:thanks:

driftr

6th June 2007, 16:57

Thanks for 1.12! Small request... can we get the estimated time to finish back? The new output it great and the percentage is nice but if it's not a big deal it'd be nice to see that displayed also. Small request, don't mind me :)

infernix

7th June 2007, 17:27

I'm running into a quality difference issue I can't figure out.

Source vs x264farm vs x264 standalone:
http://dx.infernix.net/images/thumbs/brb1137b-1-avssource.png (http://dx.infernix.net/images/viewer.php?id=brb1137b-1-avssource.png)http://dx.infernix.net/images/thumbs/thx0097h-1-farm.png (http://dx.infernix.net/images/viewer.php?id=thx0097h-1-farm.png)http://dx.infernix.net/images/thumbs/bbq0106y-1-nonfarm.png (http://dx.infernix.net/images/viewer.php?id=bbq0106y-1-nonfarm.png)

Things i've tried:

comparing the options inside the .264 header. they were the same except for bitrate=, which is obvious
doing the 1stpass with --batch 500000 so it runs it completely on one agent, in one go
excluding agent specific issues; I ran a 500frame encode on all agents without the others though x264farm, and the 64bit agents produced the exact same frame. the 32bit agent didn't though, but not nearly as much of a difference as in the screens.

I'm wondering what is causing this. I'll run the encode without the 32bit agent to exclude that, and will post results in a few hours when that's done.

My script:

set BITRATE=6947
set SCRIPT=e06farm.avs
set DEST=e06farmspirit.mkv
x264farmc -B 100 -1 "--bitrate %BITRATE% --threads auto --thread-input --deblock -3:-3 --bframes 3 --me umh --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --direct auto --cqmfile /etc/prestige_cqm.cfg" -2 "--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --aq-strength 0.3 --me umh --trellis 1 --subme 7 --threads auto --thread-input --progress --no-psnr --cqmfile /etc/prestige_cqm.cfg" -i %SCRIPT% -o %DEST% --config x:\avisynth\config.xml

Any ideas? :thanks:

[P]ako

8th June 2007, 06:19

Are you using the x264 that Omion provides in the first post?

infernix

8th June 2007, 09:27

ako;1011745']Are you using the x264 that Omion provides in the first post?

Can't, as I'm only running linux agents. I'd try to build it on linux if I knew what patches are used.

I did try fiddling with the threshold parameters. I tried ' --3thresh 0.95 --3ratio 0.2 --split 1500 --thresh 100' and that does look better already. I'll try some more later tonight. Oh, and the pure 64bit agent encode exhibited the same issue.

[P]ako

8th June 2007, 16:23

Omion said or implied that he built his version of x264 straight from the svn.

I asked because of this:
--aq-strength 0.3

If I am not mistaken, that's a parameter to call a deblocking patch, a patch he didn't use to build his version. Anyway, I don't think that's the problem. But, It will not hurt to try just plain x264.

SpAwN_gUy

8th June 2007, 16:37

well,... i thought about that... last week..
so, omion made a Build with aq-Patch... testings showed that it can be used :) ...
(plain SVN r656 + aq-patch .. no pthreads and no gPACK... 'cause we don't need them ;) ...)

okay... i've uploaded it here:
URL-Updated: http://rapidshare.com/files/36891146/x264-656_aq5_omion.rar

infernix

8th June 2007, 17:02

ako;1011863']If I am not mistaken, that's a parameter to call a deblocking patch, a patch he didn't use to build his version. Anyway, I don't think that's the problem. But, It will not hurt to try just plain x264.

Yeah, well I do build from SVN + apply the AQ patch, but I can't try any precompiled binaries unless they're for Debian Etch 64bit.

The resulting encode with AQ patch is what you see in the standalone screenshot. I don't have any use for the plain version as the AQ patch + prestige matrix is a clear winner for this specific encode; the grain is preserved as much as possible.

[P]ako

9th June 2007, 20:31

@Infernix

Just out of curiosity, do all of your computers' CPUs have the same instruction sets?

infernix

9th June 2007, 21:15

In these tests they didnt. One Celeron D 3.2ghz in 32bit mode, one Core 2 Duo E6600 in 64bit and one Athlon 64 3k in 64bit.

Today I've been testing on 12 Core 2 Duos. I was a bit disappointed; they all turned out to be 4300s, i though they would've been faster. I had some problems because I had to run qemu (without KVM which is hardware virtualisation) on one of them for the controller and it was really slow for 1stpass and network traffic (2mb/sec only). I just finished setting up Virtualbox which seems much faster so far.

I got about 15fps on the 2ndpass but am sure the controller was holding things back too. All these boxes are completely identical, except for memory (between 2 and 8GB).

A controller that runs on Linux would help alot here. I'll run some more tests with Virtualbox now :)

infernix

12th June 2007, 02:21

Okay, i've ran some tests on 14 E4300s now:

X:\enctest\data>x264farmer
Using config file "x:\avisynth\config.xml"
CONFIG:
temp_dir: "X:\\enctest\\cache\\controller"
agent_list 1:
"one 1" = x.x.x.18 50700-50705
"two 1" = x.x.x.9 50700-50705
"three 1" = x.x.x.12 50700-50705
"four 1" = x.x.x.10 50700-50705
"five 1" = x.x.x.8 50700-50705
"six 1" = x.x.x.7 50700-50705
"seven 1" = x.x.x.5 50700-50705
"eight 1" = x.x.x.6 50700-50705
"nine 1" = x.x.x.16 50700-50705
"ten 1" = x.x.x.15 50700-50705
"eleven 1" = x.x.x.14 50700-50705
"twelve 1" = x.x.x.13 50700-50705
"thirteen 1" = x.x.x.11 50700-50705
"fourteen 1" = x.x.x.4 50700-50705
agent_list 2:
"one 1" = x.x.x.18 50700-50705
"two 1" = x.x.x.9 50700-50705
"three 1" = x.x.x.12 50700-50705
"four 1" = x.x.x.10 50700-50705
"five 1" = x.x.x.8 50700-50705
"six 1" = x.x.x.7 50700-50705
"seven 1" = x.x.x.5 50700-50705
"eight 1" = x.x.x.6 50700-50705
"nine 1" = x.x.x.16 50700-50705
"ten 1" = x.x.x.15 50700-50705
"eleven 1" = x.x.x.14 50700-50705
"twelve 1" = x.x.x.13 50700-50705
"thirteen 1" = x.x.x.11 50700-50705
"fourteen 1" = x.x.x.4 50700-50705
Parsing bitrate 100
WARNING: No units given to bitrate "100"; assuming percent
GOT NORMAL INFO "e06farm2.avs: 1280x720, 25 fps, 62901 frames\013"
GOT FIRST INFO "e06farm2.avs: 1280x720, 25 fps, 62901 frames\013"
GOT FAST INFO "e06farm2.avs: 1280x720, 25 fps, 62901 frames\013"
INFO:
i_res_x = 1280
i_res_y = 720
i_fps_n = 25
i_fps_d = 1
i_fps_f = 25.000000
i_num_frames = 62901
i_bytes_y = 921600
i_bytes_uv = 230400
i_bytes_per_frame = 1382400
multiplier for kbps to fbits = 0.025000
fast_res_x = 1280
fast_res_y = 720
Oops. Writing the args failed with: Sys_error("X:\\enctest\\cache\\controller
ption_log.txt.temp: No such file or directory")
FIRST PASS
THIS SHOULD NOT BE OVERWRITTEN!
100% done (62901 / 62901) at 56.52 FPS
############################################################################
Agent | FPS | Description
one 1 | 12.84 | No jobs left; exited
two 1 | 10.04 | No jobs left; exited
three 1 | ? | No jobs left; exited
four 1 | ? | No jobs left; exited
five 1 | ? | No jobs left; exited
six 1 | ? | No jobs left; exited
seven 1 | 11.78 | No jobs left; exited
eight 1 | 13.25 | No jobs left; exited
nine 1 | 14.11 | No jobs left; exited
ten 1 | 13.08 | No jobs left; exited
eleven 1 | 15.10 | No jobs left; exited
twelve 1 | 12.84 | No jobs left; exited
thirteen 1 | 12.80 | No jobs left; exited
fourteen 1 | 13.53 | No jobs left; exited
Recent errors:

SECOND PASS
THIS SHOULD NOT BE OVERWRITTEN!
45% done (28410 / 62901) at 24.69 FPS
Agent | FPS | Description
one 1 | 2.44 | Doing frames 30145-30228
two 1 | 1.81 | Doing frames 29418-29565
three 1 | 2.23 | Doing frames 29072-29193
four 1 | 2.00 | Doing frames 28751-28962
five 1 | 2.11 | Doing frames 29566-29704
six 1 | 1.95 | Done with frames 29705-29804
seven 1 | 2.07 | Doing frames 28194-28415
eight 1 | 2.02 | Doing frames 30082-30144
nine 1 | 2.01 | Doing frames 30007-30081
ten 1 | 2.17 | Doing frames 29932-30006
eleven 1 | 2.10 | Doing frames 27944-28193
twelve 1 | 2.04 | Doing frames 29194-29417
thirteen 1 | 2.33 | Doing frames 28416-28620
fourteen 1 | 1.97 | Done with frames 28963-29071
Recent errors:

**** note: I marked the stats here to copy/paste and it now shows the difference in FPS that 3rdpass makes up for with the settings I've used ****

100% done (62901 / 62901) at 20.93 FPS
Agent | FPS | Description
one 1 | 2.07 | No jobs left; exited
two 1 | 1.83 | No jobs left; exited
three 1 | 2.27 | No jobs left; exited
four 1 | 2.01 | No jobs left; exited
five 1 | 2.05 | No jobs left; exited
six 1 | 1.90 | No jobs left; exited
seven 1 | 2.15 | No jobs left; exited
eight 1 | 2.01 | No jobs left; exited
nine 1 | 1.99 | No jobs left; exited
ten 1 | 2.01 | No jobs left; exited
eleven 1 | 2.02 | No jobs left; exited
twelve 1 | 2.17 | No jobs left; exited
thirteen 1 | 2.04 | No jobs left; exited
fourteen 1 | 1.94 | No jobs left; exited
Recent errors:

Merging 594 / 594
TIME 4760.817000

X:\enctest\data>

Nice and speedy :) but the firstpass is limited by the fact that it's maxed out in CPU. Virtualbox proved to be faster because the E4300 doesn't have VMX but it's still slow. However, second pass is at max speed so far. This was 1280x720 content; i'm going to try some 1920x1080 by the end of the week.

Settings used:

set BITRATE=6947
set SCRIPT=e06farm2.avs
set DEST=e06farm3tresh1-3ratio0.3-split500-thresh40.mkv
x264farmc -B 100 -1 "--bitrate %BITRATE% --threads auto --thread-input --deblock -3:-3 --bframes 3 --me umh --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --direct auto --cqmfile /etc/prestige_cqm.cfg" -2 "--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --aq-strength 0.3 --me umh --trellis 1 --subme 7 --threads auto --thread-input --progress --no-psnr --cqmfile /etc/prestige_cqm.cfg" -i %SCRIPT% -o %DEST% --config x:\avisynth\config.xml --3thresh 1 --3ratio 0.3 --split 500 --thresh 40

Will try a bunch of threshold parameters and make some visual comparisons when I'm done, so if anyone has suggestions on what to try - fire away, it's speedy enough :D

jimmydu3

12th June 2007, 16:31

well,... i thought about that... last week..
so, omion made a Build with aq-Patch... testings showed that it can be used :) ...
(plain SVN r656 + aq-patch .. no pthreads and no gPACK... 'cause we don't need them ;) ...)

okay... i've uploaded it here:
http://rapidshare.com/files/35973730/x264-656_aq5_omion.rar.html

could you repost it? it's been downloaded too many times now on rapidshare

SpAwN_gUy

13th June 2007, 08:21

could you repost it? it's been downloaded too many times now on rapidshare
try this one...
http://rapidshare.com/files/36891146/x264-656_aq5_omion.rar

SpAwN_gUy

14th June 2007, 09:57

Okay, i've ran some tests on 14 E4300s

Settings used:

set BITRATE=6947
set SCRIPT=e06farm2.avs
set DEST=e06farm3tresh1-3ratio0.3-split500-thresh40.mkv
x264farmc -B 100 -1 "--bitrate %BITRATE% --threads auto --thread-input --deblock -3:-3 --bframes 3 --me umh --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --direct auto --cqmfile /etc/prestige_cqm.cfg" -2 "--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --aq-strength 0.3 --me umh --trellis 1 --subme 7 --threads auto --thread-input --progress --no-psnr --cqmfile /etc/prestige_cqm.cfg" -i %SCRIPT% -o %DEST% --config x:\avisynth\config.xml --3thresh 1 --3ratio 0.3 --split 500 --thresh 40

Will try a bunch of threshold parameters and make some visual comparisons when I'm done, so if anyone has suggestions on what to try - fire away, it's speedy enough :DI'm sorry to ask, but what are you encoding?
and What are your testResults(with those thresholds experiments)? ('cause i've only tryed the defaul ones.. and one another "--3thresh 0.95 --3ratio 0.2" (that was for totalSize prescision))

infernix

14th June 2007, 12:11

I'm sorry to ask, but what are you encoding?
and What are your testResults(with those thresholds experiments)? ('cause i've only tryed the defaul ones.. and one another "--3thresh 0.95 --3ratio 0.2" (that was for totalSize prescision))

I'm trying various things. Most tests have been on a BSG episode, I also have some HD disc sources to test now. But I can't actually view the results in realtime so you'll have to be a bit more patient before I can show various results between the different things i'm trying :)

infernix

14th June 2007, 21:07

New test result for the sameframe:

http://dx.infernix.net/images/thumbs/omg0749x-1-farmnewhq.png (http://dx.infernix.net/images/viewer.php?id=omg0749x-1-farmnewhq.png)

This can be compared to the previous frames. Commandline:

x264farmc -B 100 -1 "--bitrate %BITRATE% --threads auto --thread-input --deblock -3:-3 --bframes 3 --me umh --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --direct auto --cqmfile /etc/prestige_cqm.cfg" -2 "--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --aq-strength 0.3 --me umh --trellis 1 --subme 7 --threads auto --thread-input --progress --no-psnr --cqmfile /etc/prestige_cqm.cfg" -i %SCRIPT% -o %DEST% --config x:\avisynth\config.xml --3thresh 1 --3ratio 0.3 --split 500 --thresh 40

The big change here is that I'm using the exact same high quality parameters on bost 1st and 2nd pass. The threshold parameters didn't help; I did a test with just the threshold parameters and splitpoint and it wasn't nearly as good.

For honest comparison I should run a standalone encode with these parameters, but I am going to continue testing on other sources for now. However, after some more reviewing, the farm encode with equal quality setting for both passes looks even better (although marginally) than the standalone with a fast 1stpass and high quality 2nd pass. I'm not sure whether --3thresh 1 influences this much.

SpAwN_gUy

18th June 2007, 08:35

just now i've got PMessage from James
hi, I saw you were working on adding x264farm support to megui. Do you have a test version? How is it coming along? Thanks, Jamesyep. it's quite usefull,.. usefull for commandLine Generation.. ;) ...

What's functioning:
- full commandLine generation for controller
- assorted params are NOT assorted, but they could be modified.
- profiles saving - works fine..

and what's NOT functioning:
- Job procession (you CAN add Job to jobList, but don't even try to start it ;) ... 'cause i dunno what will happen and custom progress-window is in to-do list)
- i'm trying to implement FULL x264Settings Support, but i can't find any "usefull" programmer of meGUI ... to help me with 3 lines of code :( ...

http://rapidshare.com/files/37876588/meGUI_x264farm-MOD_20070618_VS2005.rar

just in case someone interested.. in this "should look like this"-test build :) ...

infernix

18th June 2007, 12:46

A test with 1080p content (1920x1080):

X:\enctest\data\hollowsleep>farmboi.bat
Using config file "x:\avisynth\config.xml"
CONFIG:
temp_dir: "X:\\enctest\\cache\\controller"
agent_list 1:
"one 1" = 10.70.0.18 50700-50705
"two 1" = 10.70.0.9 50700-50705
"three 1" = 10.70.0.12 50700-50705
"four 1" = 10.70.0.10 50700-50705
"five 1" = 10.70.0.8 50700-50705
"six 1" = 10.70.0.7 50700-50705
"seven 1" = 10.70.0.5 50700-50705
"eight 1" = 10.70.0.6 50700-50705
"nine 1" = 10.60.0.16 50700-50705
"ten 1" = 10.60.0.15 50700-50705
"eleven 1" = 10.60.0.14 50700-50705
"twelve 1" = 10.60.0.13 50700-50705
"thirteen 1" = 10.50.0.11 50700-50705
"fourteen 1" = 10.50.0.4 50700-50705
"fifteen 1" = 10.60.1.1 50700-50705
"sixteen 1" = 10.70.1.2 50700-50705
"seventeen 1" = 10.60.1.3 50700-50705
"eightteen 1" = 10.60.1.4 50700-50705
"nineteen 1" = 10.60.1.5 50700-50705
"twenty 1" = 10.70.1.6 50700-50705
"twentyone 1" = 10.70.1.7 50700-50705
"twentytwo 1" = 10.60.1.8 50700-50705
"twentythree 1" = 10.60.1.9 50700-50705
"twentyfour 1" = 10.60.1.10 50700-50705
agent_list 2:
"one 1" = 10.70.0.18 50700-50705
"two 1" = 10.70.0.9 50700-50705
"three 1" = 10.70.0.12 50700-50705
"four 1" = 10.70.0.10 50700-50705
"five 1" = 10.70.0.8 50700-50705
"six 1" = 10.70.0.7 50700-50705
"seven 1" = 10.70.0.5 50700-50705
"eight 1" = 10.70.0.6 50700-50705
"nine 1" = 10.60.0.16 50700-50705
"ten 1" = 10.60.0.15 50700-50705
"eleven 1" = 10.60.0.14 50700-50705
"twelve 1" = 10.60.0.13 50700-50705
"thirteen 1" = 10.50.0.11 50700-50705
"fourteen 1" = 10.50.0.4 50700-50705
"fifteen 1" = 10.60.1.1 50700-50705
"sixteen 1" = 10.70.1.2 50700-50705
"seventeen 1" = 10.60.1.3 50700-50705
"eightteen 1" = 10.60.1.4 50700-50705
"nineteen 1" = 10.60.1.5 50700-50705
"twenty 1" = 10.70.1.6 50700-50705
"twentyone 1" = 10.70.1.7 50700-50705
"twentytwo 1" = 10.60.1.8 50700-50705
"twentythree 1" = 10.60.1.9 50700-50705
"twentyfour 1" = 10.60.1.10 50700-50705
Parsing bitrate 100
WARNING: No units given to bitrate "100"; assuming percent
GOT NORMAL INFO "hollowsleep.avs: 1920x1080, 13978/583 fps, 151925 frames\013"
GOT FIRST INFO "hollowsleep.avs: 1920x1080, 13978/583 fps, 151925 frames\013"
GOT FAST INFO "hollowsleep.avs: 1920x1080, 13978/583 fps, 151925 frames\013"
INFO:
i_res_x = 1920
i_res_y = 1080
i_fps_n = 13978
i_fps_d = 583
i_fps_f = 23.975986
i_num_frames = 151925
i_bytes_y = 2073600
i_bytes_uv = 518400
i_bytes_per_frame = 3110400
multiplier for kbps to fbits = 0.023976
fast_res_x = 1920
fast_res_y = 1080
Oops. Writing the args failed with: Sys_error("X:\\enctest\\cache\\controller\\o
ption_log.txt.temp: No such file or directory")
FIRST PASS
THIS SHOULD NOT BE OVERWRITTEN!
100% done (151925 / 151925) at 5.45 FPS
##############################################################################
Agent | FPS | Description
one 1 | 0.63 | No jobs left; exited
two 1 | 0.45 | No jobs left; exited
three 1 | 0.69 | No jobs left; exited
four 1 | 0.89 | No jobs left; exited
five 1 | 0.65 | No jobs left; exited
six 1 | 0.63 | No jobs left; exited
seven 1 | 0.70 | No jobs left; exited
eight 1 | 0.59 | No jobs left; exited
nine 1 | 0.81 | No jobs left; exited
ten 1 | 0.58 | No jobs left; exited
eleven 1 | 0.90 | No jobs left; exited
twelve 1 | 0.70 | No jobs left; exited
thirteen 1 | 0.62 | No jobs left; exited
fourteen 1 | 0.81 | No jobs left; exited
fifteen 1 | 0.61 | No jobs left; exited
sixteen 1 | 0.84 | No jobs left; exited
seventeen 1 | 0.56 | No jobs left; exited
eightteen 1 | 0.64 | No jobs left; exited
nineteen 1 | 0.61 | No jobs left; exited
twenty 1 | 0.80 | No jobs left; exited
twentyone 1 | 0.61 | No jobs left; exited
twentytwo 1 | 0.61 | No jobs left; exited
twentythree 1 | 0.91 | No jobs left; exited
twentyfour 1 | 0.53 | No jobs left; exited
Recent errors:

SECOND PASS
THIS SHOULD NOT BE OVERWRITTEN!
100% done (151925 / 151925) at 11.73 FPS
Agent | FPS | Description
one 1 | 0.68 | No jobs left; exited
two 1 | 0.51 | No jobs left; exited
three 1 | 0.64 | No jobs left; exited
four 1 | 0.71 | No jobs left; exited
five 1 | 0.69 | No jobs left; exited
six 1 | 0.70 | No jobs left; exited
seven 1 | 0.67 | No jobs left; exited
eight 1 | 0.68 | No jobs left; exited
nine 1 | 0.67 | No jobs left; exited
ten 1 | 0.68 | No jobs left; exited
eleven 1 | 0.67 | No jobs left; exited
twelve 1 | 0.56 | No jobs left; exited
thirteen 1 | 0.62 | No jobs left; exited
fourteen 1 | 0.59 | No jobs left; exited
fifteen 1 | 0.61 | No jobs left; exited
sixteen 1 | 0.67 | No jobs left; exited
seventeen 1 | 0.66 | No jobs left; exited
eightteen 1 | 0.68 | No jobs left; exited
nineteen 1 | 0.58 | No jobs left; exited
twenty 1 | 0.63 | No jobs left; exited
twentyone 1 | 0.63 | No jobs left; exited
twentytwo 1 | 0.66 | No jobs left; exited
twentythree 1 | 0.62 | No jobs left; exited
twentyfour 1 | 0.69 | No jobs left; exited
Recent errors:
one 1 failed with "End_of_file"
twentyfour 1 encoding failed at 21kbps. Raising bitrate
twentyfour 1 failed with "End_of_file"
Merging 2193 / 2193
TIME 43075.285000

That's right; 24 E4300s at work :)

First pass performance is severely limited here because it takes really long for the GOP split points to complete inside a VirtualBox environment. Second pass performance is what's to be expected with these parameters I guess; but 1stpass shouldn't lag behind more than one or two FPS.

I'll post some visual comparisons (source v.s. encode) later when I get around to it.

Omion, any progress with a linux compilable controller?

SpAwN_gUy

20th June 2007, 16:16

okay,.. i guess... i've made something more or less usefull (i've been waiting for this for a long time :) )

meGUI-x264farm_mod-beta3-iB-SpAwN_gUy

this one is, allmost the same meGUI-mod, but.. i guess.. status-Window can be used not only for watching "zeros" ;) ...

so:
- full commandLine generation for controller
- assorted params are NOT assorted, but they could be modified.
- profiles saving - works fine..
- Processing - works(i've tried only one encode for now) and the Window is just the same as it was... (original, i mean) and i guess percents will be doing from 0% to 100% Two times(lasy)

and what is in "TODO" list:
- new(custom) ProgressWindow
- x264 params (still waiting for any-one.. thanks foxyshadis for PM, but no PM back from you :( )
- Error log is, kinda, not working(i'm too lazy)

and.. i can't find out-dump.txt on my HDD (need to find where meGUI starts controller.. and look in there, i guess)
UPD: found it in output-file dir ... or input ??? .... hm.. somewhere there :)

http://rapidshare.com/files/38338586/meGUI-x264farm_mod-beta3-iB-SpAwN_gUy.rar

field-testings are welcome :)

jimmydu3

22nd June 2007, 16:22

aaron330i

22nd June 2007, 18:30

I've tried the new build of megui w/ x264farm, but is there a reason why MKV is the only container available? I'd really like to encode to MP4. Thanks for the great work! This could really speed up the conversion of my collection.

I am guessing, but it is probably because MKV files have either very mature or most reliable code libraries available to merge the pieces together into the final video. And x264 also supports this as an output option.

SpAwN_gUy

23rd June 2007, 11:50

I've tried the new build of megui w/ x264farm, but is there a reason why MKV is the only container available? I'd really like to encode to MP4. Thanks for the great work! This could really speed up the conversion of my collection.This is just because
-o / --output "output.mkv"
The output file. Only Matroska (MKV) files are created. Specifying a different file extension will result in an MKV file with the wrong extension. as written on x264farm-readme or at http://omion.dyndns.org/x264farm/x264farm.html

and, yeah, i also think that MKV's are easyer to mux at the end

and you can manually mux that MKV with Additional sound in some other prog, other then meGUI.. YAMB or mp4box (i guess)

jimmydu3

24th June 2007, 23:04

everything loads, but in my agent log i'm getting "not a file (no base)" when testing the file. is this a avisynth script problem?

SpAwN_gUy

26th June 2007, 08:28

everything loads, but in my agent log i'm getting "not a file (no base)" when testing the file. is this a avisynth script problem?check one of mine HowTo's in mine signature,.. maybe that is the problem..

and.. CHECK <bases /> part in your agen't config.
everything seems to work allmost fine here :) ... with Agent-based encoding...

/me.. hm.. it seems that allo of mine agents are closed,... damn, i need those guys to start them again :)

dchard

26th June 2007, 13:41

I'am newbie with x264fram, but I like to try it out.

For first time testing I choosed a simple 1920x1080 sample from w6rtz's page (http://www.w6rz.net/parkrun1920_23mbps.ts).

I like to compress it with degrain, limitedsharpen, Ylevels, tweak and spline36resize at the end to 1280x720 to x264.

I want to use a dualcore notebook for controlling, and a single core a64 for agent.

For simplicity I want to use controller-based encoding.

But there is some points that I don't understand.

1. The controller computer is just controlling the agent(s), or it also encodes some part of the video?

2. In controller config where and how can I set:
- the avs script's location
- the x264 paramteres (first and second pass, other parameters etc.).

Thank you!

Dchard

SpAwN_gUy

27th June 2007, 13:28

I'am newbie with x264fram, but I like to try it out.Welcome :)
I like to compress it with degrain, limitedsharpen, Ylevels, tweak and spline36resize at the end to 1280x720 to x264. i suppose,.. No Problems here..
I want to use a dualcore notebook for controlling, and a single core a64 for agent. as soon as you find InputVideo Decoder plugin (DGDecode, DGAVCSource, RAWVideo.. (i'm currently looking for proper lossless)) you can try to set up agent-based.. but in your situation... i guess controller-based is way too situable(iz zis korekt in inglish? (joke))

1. The controller computer is just controlling the agent(s), or it also encodes some part of the video? nope,.. it just manages jobs.
BUT... if you LAUNCH an Agent On controllerPC - it surely can :) as one of the agents

2. In controller config where and how can I set:
- the avs script's location
- the x264 paramteres (first and second pass, other parameters etc.).this MUST NOT be (and not) present in config.xml..
you specify it in controller.exe's commandline..
you can try meGUI mod.. in one of the previous posts of mine here...
or read x264farm.html provided in pakage

dchard

28th June 2007, 18:21

Oh, now I understand.

Just one question:

The controller will run on a coreduo as is said. How much resource used for just controlling the clients? It is worth to run an agent on the controller computer?

Thank you!

Dchard

SpAwN_gUy

29th June 2007, 08:40

The controller will run on a coreduo as is said. How much resource used for just controlling the clients? It is worth to run an agent on the controller computer?in your configuration..(just two PC's one - faster, second - slower)
IF you will use "controller-based"(when AVS is Decoded by controller and then fed to just an encoder (agent)) then you may not start an agent on controllerPC..
IF you will set up agent-based encoding - then you probably should run an agent on the controllerPC.

SpAwN_gUy

29th June 2007, 16:20

okay,... i confirm one of bugs found here...
this line in the .bat-File
controller.exe --first "--crf 18 --deadzone-inter 6 --deadzone-intra 6 --bframes 3 --b-pyramid --direct auto --filter -3,0 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --progress --no-psnr --no-ssim --aq-strength 0.3" --second "--deadzone-inter 6 --deadzone-intra 6 --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,0 --subme 7 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --thread-input --progress --no-psnr --no-ssim --aq-strength 0.6 --aq-sensitivity 10" --avs "G:\DVDRiP\AFRO\DVD1\AS01\AS01.CM_LS_VD_full.avs" --output "G:\DVDRiP\AFRO\DVD1\AS01\AS01.CM_VD_80of18.mkv" --bitrate "80%" --3thresh 1 --firstavs "G:\DVDRiP\AFRO\DVD1\AS01\AS01.original.avs" --fastavs "G:\DVDRiP\AFRO\DVD1\AS01\AS01.original.avs"
gives me
controller.exe --first "--crf 18 --deadzone-inter 6 --deadzone-intra 6 --bframes
3 --b-pyramid --direct auto --filter -3,0 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --p
rogress --no-psnr --no-ssim --aq-strength 0.3" --second "--deadzone-inter 6 --deadzone-intra 6 --ref 16 --mixed-re
fs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,0 --subme 7 --analys
e all --8x8dct --vbv-maxrate 25000 --me umh --thread-input --progress --no-psnr --no-ssim --aq-strength 0.6 --aq-
sensitivity 10" --avs "G:\DVDRiP\AFRO\DVD1\AS01\AS01.CM_LS_VD_full.avs" --output "G:\DVDRiP\AFRO\DVD1\AS01\AS01.CM
_VD_80of18.mkv" --bitrate "80\DVDRiP\AFRO\DVD1\AS01\AS01.original.avs" --fastavs "G:\DVDRiP\AFRO\DVD1\AS01\AS01.or
iginal.avs"
Using config file ".\config.xml"
CONFIG:
temp_dir: "G:\\DVDRiP\\_temp"

Fatal error: exception Types.Unsupported_argument("Bitrate needs to be in the form "123%" or "123.4kbps"")
and when i put "%" with bitrate to the end - seems like working fine.. (i guess i need to update meGUIMod)

omion

4th July 2007, 06:30

Hi, guys! Remember me? :D

I'm back from the dead to announce that x264farm version 1.13 is out. I completely re-designed the first pass, which is why it took so long.

There were minor changes to the agent, but only in the code. Nothing external has changed in the agent or the config.xml files, so you don't need to replace them.

Hopefully, the first pass should be more intelligent when it comes to checking out jobs. It will recognize any low-motion areas and try to start working at the beginning of them (effectively serializing part of the encode) This will make re-encoded frames very rare now. One interesting side-effect is that if the only region left to be encoded is one of these zones, no agent will work on it. It makes it look like most of the agents are stalled, but doing it this way will actually decrease encoding time.

Controlling this new behavior is the --rethresh option, which specifies the maximum split threshold to ignore. It's a bit complicated, but increasing this number will make x264farm more likely to "serialize" larger portions of the video. Units are the same as for the --thresh option, and it will default to (--thresh + 1) / 2. If you think this option is causing weird behavior, set it to 1 to utrn it off.

This also changes around the 1st pass symbol line:
. = empty
_ = queued, but the threshold is too low. Not a good place to start encoding
- = any other queued range
| = working
! = working, but the first GOP(s) is/are incorrect (has been called DELFIRST)
# = full

The new controller will also try to make the jobs smaller as more frames get encoded. This is to lower the probability that everything is done except that your slowest computer just checked out 10000 frames, and everything has to wait for it before it can begin the second pass.

The option for controlling this is --batchmult, by default 0.5. This means that at the end of the first pass, the splitter will be making jobs half the size as at the beginning. The agents may still be getting jobs that were created at the beginning of the encode, so it's not foolproof, but it should help in the long run. Set --batchmult to 1 to turn off.

There are a few other improvements, too:
The ETA is back by popular demand
Agents should start getting jobs faster at the beginning of the first pass
Split points have been added to the temp file. This means restarting an encode will start encoding faster.
If the split length is smaller than the max GOP size, it will now figure that out and eventually encode everything. Previous behavior was to go into an infinite loop.
The second pass no longer checks out jobs in chronological order (don't ask why)

Note that I don't think any bugs were fixed in this release. My copy of x264farm spent the majority of the time broken, and I didn't have too much time to look through this thread.
[ edit: Actually, I may have fixed the issue SpAwN_gUy had in this post (http://forum.doom9.org/showthread.php?p=1009320#post1009320). Some testing would be appreciated :) ]

omion

4th July 2007, 06:57

I've tried the new build of megui w/ x264farm, but is there a reason why MKV is the only container available? I'd really like to encode to MP4. Thanks for the great work! This could really speed up the conversion of my collection.
The main reason is that I didn't want to bother with any other container. There is actually a lot of custom code that I had to write to support even one container.

When a second-pass encode gets sent back to the controller, it will actually parse the file and remove the settings string that x264 adds to each encode. This requires re-muxing the video in the container, so I have to be able to read and write all supported containers.

I suppose it would have made more sense to use MP4, as it seems to be easier to convert from MP4 to MKV than the other way around. But all I use is MKV, so that's what stuck. :)

infernix

4th July 2007, 10:36

SpAwN_gUy

4th July 2007, 11:33

thanks for an UpDate, omion.
i'll test it asap.

and i think i'll come soon back with some BUGz :) ... 'cause lately i've been using --firstavs and --fastavs options.. and i have some troubles with it... :) ...

firts of all i want to know.. do i really need to specify BOTH first and fast avs'es if they are the same? (wich one default to wich one?)

omion

4th July 2007, 18:09

Nice to see an updated version :)

I hadn't update my posts yet but I did run into a serious problem, on scenes with rapid brightness/contrast changes (like a dark scene with lightning, a bunch of flashing cameras etc), there's extreme frame duplication of up to several seconds.

My farm is currently dead but if I get the chance to rebuild it, I'll post some more details about this strange issue. But perhaps the new 1stpass code fixed it.

Either way: :thanks:!
That's bad! I'm looking forward to any details, and in the meantime I'll see if I can duplicate it.

thanks for an UpDate, omion.
i'll test it asap.

and i think i'll come soon back with some BUGz :) ... 'cause lately i've been using --firstavs and --fastavs options.. and i have some troubles with it... :) ...

firts of all i want to know.. do i really need to specify BOTH first and fast avs'es if they are the same? (wich one default to wich one?)

--fastavs will default to whatever you have --firstavs set to. So if you want them to both be the same you only need to specify the --firstavs option.

AVmaniac

5th July 2007, 07:58

omion is back and with him version 1.13 of his exellent software. :thanks:

sorry for not answering the privat message but i was not running any encodes since then.

now that my farm is running again and the if the error..
(remember .. the controller seemed to loose control in second pass and scenes were encoded over and over again)
...still occurs, i will let you know.

btw. i also recognized scene duplications in some of my encodes
like infernix mentioned.
I'll take an eye on that in my actuell encode.

btw x246farm runs nice on my new Q6600 system.

AVmaniac

8th July 2007, 10:50

And again there were several frame duplications in my encodes.
It's seems to be either a problem with the h.264 decoder(CoreAVC) or sourcefilter (matroska for *.ts) of the input file because DVD or normal DVB-S captures (over dgindex) are not concerned.
The avs-script playback with the wmplayer also runs without any mistake.
there were also no

...rapid brightness/contrast changes (like a dark scene with lightning, a bunch of flashing cameras etc)

like interfix described that could result in the frameduplications.

What source material are you encoding infernix?

infernix

8th July 2007, 17:36

interfix? :)

I'm encoding huffyuv avi files (~100gb) with just avisource() in the avs. These huffs were generated off HD/BD disc sources. I've also done some broadcast sources. I've done two succesful encodes and three unsuccesful (frame duplication). The succesful farm encodes were simply movies which did not have such rapid contrast/brightness changes so those were fine.

AVmaniac

8th July 2007, 18:47

interfix? :)

sorry buddy .. i will have that fixed.

levi

12th July 2007, 02:12

this is so very cool. I always wanted to do distributed encoding. I'm getting interested in x264 largely because of what you are doing here. I would like to encourage you to keep up the good work.

I want to ask a couple of questions:

Could you have agent & controller screen entries begin with date-time stamp so it can be seen how long since the last update was?

Could you make it possible to instruct the controller to re-read the config.xml agents section on a regular basis in order to easily add servers to the farm?

Thanks again!

SpAwN_gUy

12th July 2007, 10:10

Could you have agent & controller screen entries begin with date-time stamp so it can be seen how long since the last update was? you can access theese timestamps in out-dump.txt (near controller.exe)

Could you make it possible to instruct the controller to re-read the config.xml agents section on a regular basis in order to easily add servers to the farm?well, i was asking that.. but eventually i've come to writing Client-Server based x264farm configurer and agentGUI wich will be used to start/stop agents, collect IPs and update everything we need.. (agents, config's, paths,.. etc)

levi

12th July 2007, 16:35

Thanks for your response. Your additions sound awesome. Is this part of your megui mod? I can hardly wait. Let me know if you need a tester. =)

One other thing, I much prefer staxrip to megui =) any chance your code would work with staxrip?

SpAwN_gUy

14th July 2007, 09:32

Thanks for your response. Your additions sound awesome. Is this part of your megui mod? I can hardly wait. Let me know if you need a tester. =) this will be.. another prog,.. 'cause not every one needs to install .Net framework, and maybe not everyone wants meGUI :) ...

One other thing, I much prefer staxrip to megui =) any chance your code would work with staxrip? hm.. guess not :) ... two different progs.. but controller of agents - probably :) ... i think you need to ask staxrip developer :)

driftr

15th July 2007, 18:52

Is anyone else having an issue with 1.13 where agents aren't getting more than one job? I have <number>2</number> set in the controller config but the agents seem to only get one job at a time. Sometimes the first instance will get it, sometimes the second, but never the two together.

I just reverted to 1.11 (don't have 1.12 but it was working on 1.12) and all's encoding fine. Any help would be appreciated.

omion

15th July 2007, 23:39

@driftr:
Does this happen on both passes or just the first? Also, what does the controller's status display for the agents? "Waiting for job"?

I did change around the first-pass job-handling code in 1.13. The goal is to make it less likely to re-encode large chunks of the file, but it will occasionally not send out enough jobs. This is actually a good thing: if a job has a very weak starting frame, if it is encoded right away there is a very good chance that it would need to be re-encoded when the actual starting frame is found later on. So the encoding is delayed until more information is found out.

There may be something else preventing jobs from getting to the agent, though. Could you send me a copy of the controller's out-dump.txt file when this happens? Thanks.

SpAwN_gUy

16th July 2007, 10:01

omion

17th July 2007, 06:45

'kay i'm back with those bugs i've found :) ...
1) --firstpass ".avs"
controller.exe --first "custom_params_pass_1" --second "custom_params_pass_2" --avs "G:\DVDRiP\AFRO\DVD1\AS05\AS05.LS_VD.avs" --output "G:\DVDRiP\AFRO\DVD1\AS05\AS05.LS_VD_60of18.mkv" --3thresh 1 --firstavs "G:\DVDRiP\AFRO\DVD1\AS05\AS05.proper_original.avs" --bitrate "60"
controller.exe --first "custom_params_pass_1" --second "custom_params_pass_2" --avs "G:\DVDRiP\AFRO\DVD1\AS05\AS05.LS_VD.avs" --output "G:\DVDRiP\AFRO\DVD1\AS05\AS05.LS_VD_1000.mkv" --3thresh 1 --firstavs "G:\DVDRiP\AFRO\DVD1\AS05\AS05.proper_original.avs" --bitrate "1000kbps"notice the difference.. only bitrate and output filename.
okay.. the first-pass encoding goes two times.. is it normal?
when no firstavs entered - no first pass encoding in second encode..

Uh... it shouldn't happen twice. Does the out-dump.txt file say why that happened? I added in code to re-check the output of the first pass to make sure that all the frames are there before going to the second pass, so it could be that. But it shouldn't have to re-encode the whole thing.

2) "Dead?" apears sometimes..
SECOND PASS
THIS SHOULD NOT BE OVERWRITTEN!
99% done (37911 / 38002) at 9.75 FPS
ETA: 0:09
Agent | FPS | Description
pc79 1 | 1.65 | No jobs left; exited
pc79 2 | 1.58 | No jobs left; exited
pc206 1 | 1.46 | No jobs left; exited
pc206 2 | 1.59 | No jobs left; exited
pc82 1 | 15.26 | No jobs left; exited
pc82 2 | ? | No jobs left; exited
pc422 1 | ? | No jobs left; exited
pc419 1 | ? | No jobs left; exited
pc661 1 | 1.86 | Dead?
pc6 1 | ? | No jobs left; exited
pc2 1 | 1.73 | No jobs left; exited
pc418 1 | 1.29 | No jobs left; exited
pc17 1 | ? | No jobs left; exited
Recent errors: no errors btw.. and it stalled..

Ah. The second pass. I fixed this on the first pass (right??), but I don't remember adding a fix for the second pass. I'll look into it. It was actually annoyingly easy to fix on the first pass (I just ignore the threads at the end) so it shouldn't be too hard for the second pass.

One question: when the controller marks an agent as "Dead?", is the agent actually dead, or is it still running?

and then..
3) when encoding is reloaded (after "crash") it continus normally.. from the place it ended
and..SECOND PASS
THIS SHOULD NOT BE OVERWRITTEN!
98% done (37581 / 38002) at -6.32 FPS
ETA: -1:-6
Agent | FPS | Description
pc79 1 | ? | Doing frames 37976-38001 (26)
pc79 2 | ? | Doing frames 37877-37975 (99)
pc206 1 | 0.84 | Doing frames 37781-37876 (96)
pc206 2 | ? | Doing frames 34900-34998 (99)
pc82 1 | ? | Disconnected
pc82 2 | ? | Disconnected
pc422 1 | ? | Disconnected
pc419 1 | ? | Disconnected
pc661 1 | ? | Disconnected
pc6 1 | ? | Disconnected
pc2 1 | 2.05 | Doing frames 17862-17887 (26)
pc418 1 | ? | Doing frames 19369-19443 (75)
pc17 1 | ? | Disconnected well, there is NO "-EE.EE" (or something like that) and it IS good,... but notice highlighted strings.

Getting rid of the "-1.#INF" (or whatever) has been on my mind for a while, but it's not a big deal. The big deal is why they're created (generally when an agent is behaving badly)

The negative FPS occurs when the controller doesn't know when the encoding will end or how many frames have been encoded. Due to the way I coded it, the ending of the second pass is actually quite hard to guess. I'll see if there's a way to make it look nicer, anyway.

The negative FPS happens on the first pass when x frames are completed, but the controller then realizes that y frames are bad and need to be re-encoded. If y>x, then the number of encoded frames actually decreases. This shouldn't happen very often or for too long.

I'm still working on x264farm as fast as I can, even though I don't post as often as I'd like. It usually takes more effort to write responses than to write code. :p

SpAwN_gUy

17th July 2007, 08:31

Uh... it shouldn't happen twice. Does the out-dump.txt file say why that happened? I added in code to re-check the output of the first pass to make sure that all the frames are there before going to the second pass, so it could be that. But it shouldn't have to re-encode the whole thing.i'll re-check it.. 'cause out-dump.txt.. hm.. is overwritten :) ..
Ah. The second pass. I fixed this on the first pass (right??), but I don't remember adding a fix for the second pass. I'll look into it. It was actually annoyingly easy to fix on the first pass (I just ignore the threads at the end) so it shouldn't be too hard for the second pass.No problems with first-pass.. actally, after i posted yesterday,.. this "Dead?" thing happened for 4 agents.. and.. 3 times.. i guess it is because of my net problems, but i think the behaviour is.. kinda,.. not good :)
One question: when the controller marks an agent as "Dead?", is the agent actually dead, or is it still running?it IS running.. when restarted, controller hops all the agents (that were running).. but.. well .. i dunno, 'cause if agent crushes - mine agentGUI restarts it :) ... but i can't catch x264.exe errors(or for now i dunno how :) ), aswell as it's output.. i can only parse agent's output.. and it's errors.. so sometimes if x264 crashes and window shows "Send the error" i need to wait until user presses the button..
The negative FPS occurs when the controller doesn't know when the encoding will end or how many frames have been encoded. Due to the way I coded it, the ending of the second pass is actually quite hard to guess. I'll see if there's a way to make it look nicer, anyway. well,.. i would like to see progress-bar, as in first pass :) ...so like,.. what gops are encoded, what will need to be re-encoded, bad,.. and so on.. :) ..
I'm still working on x264farm as fast as I can, even though I don't post as often as I'd like. It usually takes more effort to write responses than to write code. :p keep it up :) .. i'm trying to be a support-unit as much as i can :)

SpAwN_gUy

17th July 2007, 08:42

and i just noticed...
pc206 1 | 1.28 | Doing frames 18459-18510 (52)
pc206 2 | 1.44 | Doing frames 15086-15119 (34)
pc82 1 | 3.52 | Doing frames 10785-10839 (55)
pc82 2 | 3.92 | Doing frames 32006-32086 (81)
two allmost the same PCs.. intel Core2 Duo e6300,...
both have no processes running (just agent),
both encoding "agent-based"
both in one room with mine (mine is waay too slow.. but the controller ;) )
but first one is on 100mbit-LAN
and the second one is on 1GiGAbit-LAN ....
damn... i need all other PCs to gen GIGa aswell ;)

SpAwN_gUy

2nd August 2007, 13:51

Hi guys... we are partially back ;) ...

me - made new meGUI-MOD.
omion - made new x264-plainSVN build ...
but i've also managed to build cef's sources ;) ... (unfortunately with external pthreads.dll ,.. but it works.. i've tested, for real)

'kay... one after another:
meGUI-rev138.x264farm_mod-beta4.by.iB-SpAwN_gUy
this one is
core version: 0.2.6.1001
x264farm-mod ver: 0.7
Changelog:
Updated to x264farm 1.13 commandline parameters.
-- help also..
Fixed "job status - error" after successfull encode
--sorry no errors additions for now.. (lazy again) next release
Fixed - "completeness" runs from 0 to 100% two times
--now percentage goes from 0 to 100% ...
--from 0 to 50% - first-pass
--from 50 to 100% - second-pass
--current-pass detection - is present :) ...

currently NOT working:
- x264 parameters.. (actually it IS opening (in dev-builds) but i can't manage to get output parameters :( ...) so it is disabled..
- no x264farm errors handling...

and i can't find meGUIs behaviour.. when it starts controller in it's own dir, and when in "output" dir ,... so controller's out-dump.txt will be placed in one of those dirs..

tha LiNK: http://rapidshare.com/files/46519420/meGUI-rev138.x264farm_mod-beta4.by.iB-SpAwN_gUy.rar

and now for x264-667b-MSVS2005 builds..
omion made, on my request, plainSVN build..
and i figured out how to build my builds :) ...
so,.. my build is MSVS2005 + AQ-PATCH (updated..btw) and requires pthread.dll(included) to be placed near it (i'm working on it..)
and it is NOT capable to output .mp4 ('cause no GPAC)
theese could be found here: http://rapidshare.com/files/46519549/x264-667b-MSVS2005.rar

elpierco

18th August 2007, 20:31

Suchy

2nd September 2007, 13:14

Anyone tries it in virtual LANs (like Hamachi)?

SpAwN_gUy

3rd September 2007, 10:46

Anyone tries it in virtual LANs (like Hamachi)?(i'm not sure what it really is.. so)
i suppose.. there will be no problem.. IF:
you can obtain IP-Adresses of all agents :) ... for controller-based encodings.. and access to shares(they must be seen from agentPCs) on filestoragePC.

'kay guys i'm back :) from my vacations..
before i went to germany omion wrote to me, that he has something new to show to us.. and work is in progress... aswell as i got some working sources from berringham(i hope this is correct ;) ) of MeGUI developers..

SpAwN_gUy

3rd September 2007, 10:54

I have been ripping dvd's from netflix using dvd decrypter and now have almost 2 terabytes of vobs amased on my fileserver. I have started using autogordian knot to compress these ~8gb files to avi's with my new quad core. I have about 10 assorted computers laying around that I have tried running dvd::rip on but had terrible audio sync problems. Gordian works well but takes about 1.5 hours to generate an avi. I am wondering if I can use x264 parallel encoder to re encode these vobs to avi or similar compressed ~700MB format? so, as i think, you are encoding to XviD or DivX with AutoGK..
so maybe, if you still want to encode with DivX and XviD, you should use so other prog (ELDER, i think.. you can find it in theese forums).
But if you want to take a step into the Future ;) ... you can start to read throug "HowTo"'s on doom9 MainSite...

So far I installed controller and agent on my quad and generated an .avs file for a collection of vobs. I am wondering about the command line arguments to use to achieve my desired compression. I could be totally off in my assumptions of what this software can do but any help/advice would be great.
for the controller commandline arguents you can try downloading my meGUI-Mod.. commandLine arguments are listed in lower box..

but before that i recomment reading tutorials on how to properly encode video and all those Audio/video sync tutorials.. and so on..

this one is not the easiest way to encode.. but seems to be faster anyway :) ..

Disabled

3rd September 2007, 15:38

There should be no problem using this over a VPN like Hamachi, I guess you could even use it over the plain internet, the thing is that you need to transfer huge amounts of data, so unless all your computers are connected very fast to the net (and in both directions, up/down) it will most probably be faster to encode it on just one machine.

Anyone tries it in virtual LANs (like Hamachi)?

infernix

11th September 2007, 18:16

Omion, have you given up on a linux buildable controller?

omion

12th September 2007, 05:53

Omion, have you given up on a linux buildable controller?

I haven't given up at all. I just completely forgot...

I got another computer a month ago, and I decided to switch around the roles of all my others (my Mac Mini is now my server, my old server is now for backups) and in the middle of all that I decided to reinstall Windows on my only computer I wasn't moving around. So for a while none of my 4 computers worked.

I actually just got around to testing out my encoding setup yesterday, and I'm currently trying to get a new build of x264farm for controller-based encoding out.

But I should get around to the controller soon. Sorry about the (unusually long) delay...

legoman666

18th September 2007, 00:40

legoman666

18th September 2007, 02:47

Came across an annoying bug, my first pass got 75% done (did it correctly). Then the controller started to only give jobs to 1 of the clients; and naturally it was the slowest one (p3 450). So i turned off the agent on that machine. Now its only giving jobs to the 1 agent i'm running on the same machine as the controller.

confuzed.

omion

18th September 2007, 07:59

I found a small bug. Maybe?

in the agent config.xml, it requires quotes around the path of the location of x264.exe but not for the location of the temp dir. If you put quotes around the temp dir location, it gives an error.

As a side note: gigabit network kicks ass.
Fixed. Next build won't require quotes around x264. [edit: but based on my current amount of free time, I have no idea when that will be released...]

And yes, gigabit networks are awesome! I got mine up to 105MB/s through Samba once :cool: That's faster than most people's hard drives!

Came across an annoying bug, my first pass got 75% done (did it correctly). Then the controller started to only give jobs to 1 of the clients; and naturally it was the slowest one (p3 450). So i turned off the agent on that machine. Now its only giving jobs to the 1 agent i'm running on the same machine as the controller.

confuzed.
Uhh... that will happen sometimes when the controller thinks that starting a new encode will damage some of the frames that are already encoded. It should only happen toward the very end, though. Did you happen to set the batch size to be very large? Like around (num_frames/num_agents)? If not, send me an email of the out-dump.txt file and I'll see what happened. (my e-mail address is at the top of x264farm.html in my downloads)

If you want to disable this "feature" (since it seems like it isn't working) you can use "--rethresh 1" on the controller's command-line.

Omion, have you given up on a linux buildable controller?

This is harder than I thought it would be... the only way I know of to parse .avs files is with avs2yuv through Wine (this is actually why avs2yuv exists) and I keep running into path problems. avs2yuv can't use a Linux-style path, but the controller (if built for Linux) can't use a Windows-style path.

Since the controller only needs the avs file to send to avs2yuv, I thought I could avoid this issue. I thought I could just set a Windows-style path on the command line and send it directly to avs2yuv. However, there is one part of the controller that needs access to the avs file: the part that makes a hash of the file to send to the agent for agent-based encoding. Without access to the avs file, there is no way to check if the agent is encoding the correct file, so I'd have to turn off agent-based encoding.

I am now coming to the conclusion that you may have EITHER agent-based encoding OR a native Linux build, and I think being able to use agent-based encoding would be faster in almost all cases.

smok3

18th September 2007, 08:22

iam just curious, do we have sort of a diagram on how the nodes are communicating between each other (like via master, or what...?)
:thanks:

SpAwN_gUy

18th September 2007, 12:39

iam just curious, do we have sort of a diagram on how the nodes are communicating between each other (like via master, or what...?)
:thanks:
agents are not comunicating each other..
the only one who has all IPs is controller. it rules the agents. and calculates what needs to be encoded.

omion

18th September 2007, 23:16

iam just curious, do we have sort of a diagram on how the nodes are communicating between each other (like via master, or what...?)
:thanks:

The agents only communicate with the controller; they have no idea if any other agents are working. Each concurrent encode by one agent is treated as a completely separate encode - this means that two controllers could connect to a single agent and work on two different movies at the same time.

I had an idea to allow multiple "servers" that each agent could connect to, instead of the controller. This would even out the encoding load without using agent-based encoding, but it looks like most people use agent-based, so I gave up on that idea.

legoman666

19th September 2007, 00:03

Fixed. Next build won't require quotes around x264. [edit: but based on my current amount of free time, I have no idea when that will be released...]

And yes, gigabit networks are awesome! I got mine up to 105MB/s through Samba once :cool: That's faster than most people's hard drives!

Uhh... that will happen sometimes when the controller thinks that starting a new encode will damage some of the frames that are already encoded. It should only happen toward the very end, though. Did you happen to set the batch size to be very large? Like around (num_frames/num_agents)? If not, send me an email of the out-dump.txt file and I'll see what happened. (my e-mail address is at the top of x264farm.html in my downloads)

If you want to disable this "feature" (since it seems like it isn't working) you can use "--rethresh 1" on the controller's command-line.

thanks for the advice. I was using the default (i think it is) batch size of 5000. For this next encode I'm about to do, im using 3000 so I'll hopefully avoid this issue.

Anyway, you mentioned that most people use agent based encoding. I don't because I have no f'ing idea how it works, I read the little explanation in the agents config file, but I am still clueless. From my understanding, the avs, the source, avisynth, and all the filters need to be installed in identical places on all of the agents. I don't want to copy my 40gb source to all of the clients.

edit: I suppose this is because all filtering in the avs is done on the controller, but with this as my avs
LoadPlugin("C:\PROGRA~2\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~2\GORDIA~1\AviSynthPlugins\KernelDeInt.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\EEDI2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\TDeint.dll")

SetMemoryMax(128)
# SOURCE
mpeg2source("H:\-=HD Movies=-\Akira (1988)\Akira - 1080i - RAS.d2v")

interp = separatefields().selecteven().eedi2()
tdeint(edeint=interp)

#KernelDeInt(order=1,sharp=true)

# CROPPING
crop(0,0,1920,1080)

# RESIZING
LanczosResize(1280,592)

The first pass took several minutes to get started, then several more minutes for anything to actually happen. I suppose its because of the deinterlacing method I was using. I switched it back to the kernel deinterlace and it going its "normal" speed now. Would using agent based encoding solve this problem?

edit2: the source is 1080i content.

SpAwN_gUy

19th September 2007, 08:24

Anyway, you mentioned that most people use agent based encoding. I don't because I have no f'ing idea how it works hehe.. use first of my tutorials in my signature.

currently i'm stuggling on the field of relative-paths in .d2v, so we had no need to modify it by hands... :) (you can join me here (http://forum.doom9.org/showthread.php?t=129510&goto=newpost) )
!!!! UPD: Nevermind. this feature will be in the next beta release of DGIndex ;) .... i assume DGMPGDec 1.5.0 beta 3

The first pass took several minutes to get started, then several more minutes for anything to actually happen. I suppose its because of the deinterlacing method I was using. I switched it back to the kernel deinterlace and it going its "normal" speed now. Would using agent based encoding solve this problem? well,... not exactly this one.. it will not solve slow starts, but it WILL solve Overall ControllerPC's CPU load. (you will not have to decode and compress video for each agent)
but you can look to "--firstavs and --fastavs" params ;)

smok3

19th September 2007, 08:32

The agents only communicate with the controller; they have no idea if any other agents are working. Each concurrent encode by one agent is treated as a completely separate encode - this means that two controllers could connect to a single agent and work on two different movies at the same time.

I had an idea to allow multiple "servers" that each agent could connect to, instead of the controller. This would even out the encoding load without using agent-based encoding, but it looks like most people use agent-based, so I gave up on that idea.

ok, tnx.

legoman666

19th September 2007, 18:37

hehe.. use first of my tutorials in my signature.

currently i'm stuggling on the field of relative-paths in .d2v, so we had no need to modify it by hands... :) (you can join me here (http://forum.doom9.org/showthread.php?t=129510&goto=newpost) )
!!!! UPD: Nevermind. this feature will be in the next beta release of DGIndex ;) .... i assume DGMPGDec 1.5.0 beta 3

well,... not exactly this one.. it will not solve slow starts, but it WILL solve Overall ControllerPC's CPU load. (you will not have to decode and compress video for each agent)
but you can look to "--firstavs and --fastavs" params ;)

thanks, Ill give it a look-see when I get home.

Finar

19th September 2007, 21:23

SpAwN_gUy

20th September 2007, 08:12

omion

21st September 2007, 04:58

Hi!

My idea is to make my home server (100% uptime) coding my content all the time, and distributing it with other computers, when they are online. But I'm not sure if it is possible to shutdown Client PCs safely while main encoding still going... What will happen when x264Farm Server will suddenly lose the connection with one of the Clients?

Thanks!

Yeah, that's no problem. If the controller sees the agent disconnect it will simply give out that job to another agent. When I first got my Mac Mini it would crash every hour or so when encoding, so I made sure the encode would survive! :D (turned out I had some bad RAM - replaced it and all was good)

The whole setup is designed to be able to withstand all sorts of network-related issues, but you will lose all the progress you had on whatever you disconnected. Disconnecting the controller will throw out all the frames that the agents are currently set to encode, which on the first pass will be a fairly sizable chunk of the frames.

There is one issue that still may exist... if the controller does not get the signal that the agent has disconnected, it may wait forever to receive the job even if you re-connect the network. The only way around that is to guess when the connection has died, which I have done in the areas where this bug is most likely. I am fairly certain, however, that this issue is still around somewhere.

Inventive Software

24th September 2007, 13:54

Hey hey! Spent the last couple of hours catching up on a lost relic I forgot about after moving back from uni for the summer. :)

Questions. :D

1) What network speed is ideally required? I've read that 1 Gbit kicks ass (to quote both Spawn_guy and omion ;)), but I'm limited in my home setup to 100 Mbit (laptop is 100 Mbit, desktop has 100 Mbit card). Will that suffice?

2) I see there are 2 modes of encoding: controller based and agent based. Which is best for just 2 computers with XP on them? One's a laptop that's dual-core and the fastest computer I have. The other's a Celeron 800 that's quite slow, but reasonably steady. :)

Another note: I will inquire as to whether I can use a networked room at my university to test this out. It would be very interesting to get this working at uni. :D

SpAwN_gUy

24th September 2007, 15:50

1) What network speed is ideally required? I've read that 1 Gbit kicks ass (to quote both Spawn_guy and omion ;)), but I'm limited in my home setup to 100 Mbit (laptop is 100 Mbit, desktop has 100 Mbit card). Will that suffice? yep.. it REALLY KICKS ASS... but as i've found, our new Taiwan-NoName-1Gb-SWiTCH has gone.. 'cause it was causing lots of troubles and 1Gb-Cisco costs lots of money.. so now i'm back to 100Mb..
well. it keeps spinnin' ;) ... and everything goes wery nice.. i miss the spees i had that time, but i think i can handle it with more new dualCore agents ;)

2) I see there are 2 modes of encoding: controller based and agent based. Which is best for just 2 computers with XP on them? One's a laptop that's dual-core and the fastest computer I have. The other's a Celeron 800 that's quite slow, but reasonably steady. :) hm.. i really dunno.. for just two PCs.. and 100Mb.. maybe in this condition you should better just use your dualCore PC.. as standalone..
but if there will be more agents then it's up to you what to choose :) ... i prefer agent-based and no agent on the controllerPC :) ..

SpAwN_gUy

24th September 2007, 15:54

There is one issue that still may exist... if the controller does not get the signal that the agent has disconnected, it may wait forever to receive the job even if you re-connect the network. The only way around that is to guess when the connection has died, which I have done in the areas where this bug is most likely. I am fairly certain, however, that this issue is still around somewhere.
the "Dead?" issue is still on the second-pass if agent suddenly dies.. encode cannot be finished automatically... 'cause "Dead?" sign is still waiting for something..

legoman666

25th September 2007, 02:06

yep.. it REALLY KICKS ASS... but as i've found, our new Taiwan-NoName-1Gb-SWiTCH has gone.. 'cause it was causing lots of troubles and 1Gb-Cisco costs lots of money.. so now i'm back to 100Mb..
well. it keeps spinnin' ;) ... and everything goes wery nice.. i miss the spees i had that time, but i think i can handle it with more new dualCore agents ;)

hm.. i really dunno.. for just two PCs.. and 100Mb.. maybe in this condition you should better just use your dualCore PC.. as standalone..
but if there will be more agents then it's up to you what to choose :) ... i prefer agent-based and no agent on the controllerPC :) ..

It also depends on the source and what quality you're encoding at. If you have a dvd source and are encoding to something like 1500kbps, (depending on the speed of the clients) they'll probably use ~40% of a 100mb connection each. on the other hand, if your source is 1080p content and your converting to 720p @ high quality, each client will use less, since they're converting so much slower.

Example. converting 1080i content to 720p with rediculous settings.
controller is AMD x2 @ 2.8ghz
client 1 is Intel C2D @ 2.0ghz
client 2 is AMD x2 @ 2.4ghz
client 3 is AMD 3000+ @ 1.8ghz
client 4 is Intel CD @ 1.8ghz
client 5 is dual p3 500mhz

Even though I had 4 reasonably fast clients and 1 slow one, I was only using 10-12% of my gigabit connection during the second pass since it was encoding so slowly (12fps total or so)

On the other hand, I converted a dvd to 1400kbps (with medium quality) using the same clients, and it was using >30% of the gigabit connection during the second pass.

SpAwN_gUy

26th September 2007, 10:15

'kay.. i'm not doing nothing...

i just decided to build some wierd and heavily patched x264 build..

this one is r680 of Cef's sources + some new pathces from Dark Shikari
Cef (http://mirror05.x264.nl/Cef/)
-----
is the regular one, with msvc/win64 asm fixes, and the 2 patches aq and thread pool.
-----

Dark Shikari (http://forum.doom9.org/member.php?u=83421)
-----
The SATD patch is "worthwhile" but only with --me esa, as the other modes are too slow, ironically.
The --me-prepass patch is overall generally useful.
Also note the --subme 7 and DIA patches, having no negatives at all, are perfectly fine no matter what.
-----

x264_2pass_vbv.diff
x264_faster-dia.diff
x264_me-prepass.diff
x264_satd_fpel.11.diff
x264_subme7_vc8.diff
both x64 and x86 versions built with VS2005 are here: http://rapidshare.com/files/58338767/x264-680-MSVS2005.rar

cogman

30th September 2007, 23:39

haven't read the entire thread, so sorry if any of these questions have been answered.

What is this written in? I see the ml tag but I don't know what language it is. (reading some of the code has not made it any clearer).

The AVS script is processed on the controller, correct? What would constitute a speed increase? If you are using a slow network but encoding for a handheld device (IE a PDA) would it be best to resize then distribute? On the other hand, if you are just applying filters but keeping the default resolutions (IE re-encoding a dvd) would it be possible to send the raw video and just filter on the other end? How about when converting an AVI? would it be better to break it apart (is that possible?) and send over the pieces to be processed to save bandwidth?

If any of this is already done, sorry for asking :).

One last question. Would it be possible to use a fast compression algorithm (LZMA possibly) on outgoing video to save a bit more on bandwidth? Provided it is not going to provide huge benifits, but would it bog things down?

Thanks, I'm pretty interested in this project and like the Idea.

akupenguin

1st October 2007, 00:51

Would it be possible to use a fast compression algorithm (LZMA possibly) on outgoing video to save a bit more on bandwidth? Provided it is not going to provide huge benifits, but would it bog things down?
Already supported: <compression>
But LZMA is really slow when you're dealing with the amount of data involved in raw video: 2MB/s on my core2 e6600, which would be 4fps @ 480p. x264farm uses huffyuv. (Ok, I don't know if it's bitstream compatible with huffyuv, but the same algorithms.)

SpAwN_gUy

1st October 2007, 08:21

haven't read the entire thread, so sorry if any of these questions have been answered. Shame on you ;) ... most of answers are written in the read-me.html, provided with package..

but okay.. once again..
What is this written in? I see the ml tag but I don't know what language it is. (reading some of the code has not made it any clearer). (first post) - OCaml ... it is easy to "read" the code to me, but i can't code on(in?) it.. devenv could be founded on sourceforge.net .
The AVS script is processed on the controller, correct? What would constitute a speed increase? If you are using a slow network but encoding for a handheld device (IE a PDA) would it be best to resize then distribute? On the other hand, if you are just applying filters but keeping the default resolutions (IE re-encoding a dvd) would it be possible to send the raw video and just filter on the other end?Yes and NO.. Yes - for controller-based encoding.. NO - for agent-base encoding.. (try the readme)
How about when converting an AVI? would it be better to break it apart (is that possible?) and send over the pieces to be processed to save bandwidth?it does not matter in agent based.. well.. there is no need in doing that..
but for "splitting" and "reEncoding" the "some king of avi" you need (you are talking about agent-based encoding) the video-decompression decoder to be installed on evry PC.. or you "don't" when using RAW input Video (i'm currently doing that,..) or using "controller-based"
One last question. Would it be possible to use a fast compression algorithm (LZMA possibly) on outgoing video to save a bit more on bandwidth? Provided it is not going to provide huge benifits, but would it bog things down? yes, as aku said there is a switch in controller's config.xml to enable huffyuv compression (you can even spcify on which pass (first or second or both) to use it)..

SpAwN_gUy

2nd October 2007, 08:14

BTW guys... if anyone interested.. the new beta of DGIndex is out...
and Yes.. it has "Relative paths" feature ...
DGIndex 1.5.0 beta 3 (http://forum.doom9.org/showthread.php?p=1050990#post1050990)

squid_80

2nd October 2007, 11:36

BTW guys... if anyone interested.. the new beta of DGIndex is out...
and Yes.. it has "Relative paths" feature ...
DGIndex 1.5.0 beta 3 (http://forum.doom9.org/showthread.php?p=1050990#post1050990)
You know, if you map the drive to the same letter on each pc you could just use absolute paths.

rhodges

3rd October 2007, 02:55

Controller: Vista 64bit

Running the controller and 2 agents.

Agents:
X264: 32bit
AviSynth: 32bit

The config for the agent has agentpipe set to 0, but I'm still seeing avs2yuv.exe running, and it typically chews up 300+ MB of memory.

Looking at it, it's never consuming any CPU, just a ton of memory.

I searched the dump files for "avs2yuv" but I didn't see it.

What's using it and how can I stop it? (That is, if I can.)

Thanks!

omion

3rd October 2007, 05:50

Already supported: <compression>
But LZMA is really slow when you're dealing with the amount of data involved in raw video: 2MB/s on my core2 e6600, which would be 4fps @ 480p. x264farm uses huffyuv. (Ok, I don't know if it's bitstream compatible with huffyuv, but the same algorithms.)
It's very similar, but I precomputed the code tables. It turns out, with Paeth pre-processing, all the movies I tested (even anime) had almost the same symbol probabilities. It gets better compression than standard huffyuv since I don't have to send the code tables.

@rhodges:
avs2yuv is used by the controller. If you have <agentpipe> set to 0, the controller uses avs2yuv to get the raw data which is then sent to the agent.
Also, the controller uses it in the first pass to find good split points. This is probably the one that you're seeing. It only does things when the controller needs more splits, which isn't too often.

If avs2yuv is taking up too much memory, add "SetMemoryMax(64)" to the top of you AVS file to limit the frame cache to 64MB (I've noticed that avs2yuv generally uses twice this number in total memory).
If that line makes the whole encode go slower then you can make a copy of the AVS file, add "SetMemoryMax(64)" to the top of the copy, then use "--fastavs copyname.avs" in the controller settings. This will make the splitter use a memory-friendly version of the AVS script, while still using the regular one for the actual encoding.

@all:
As always, I've had way too much stuff to do lately, but I should have the next version of x264farm out very soon (like tomorrow). Also, I was going to release r680 of the controller-friendly x264, but it looks like Cef's builds (http://mirror05.x264.nl/Cef/) that SpAwN_gUy posted work fine, so I'm just going to save time and use his builds :p (And he's got a 64-bit one. Yay!)

SpAwN_gUy

3rd October 2007, 10:10

You know, if you map the drive to the same letter on each pc you could just use absolute paths. hehe.. i did so.. but.. it is sometimes not possible.. and it give more trouble to people on those PCs where agents are strted..
i've wrote a simple GUI for agent, that starts agent, restarts it when agent exits, and minimizes to tray. and stores all the info and configs on my share..
and encode goes jst fine when bases are set to Network shares...

so.. Relativity - is a bery nice thing to have :)

SpAwN_gUy

3rd October 2007, 10:31

Also, I was going to release r680 of the controller-friendly x264, but it looks like Cef's builds (http://mirror05.x264.nl/Cef/) that SpAwN_gUy posted work fine, so I'm just going to save time and use his builds :p (And he's got a 64-bit one. Yay!)
yes.. i've also mastered x264 compilation :) .. thanks to Cef.. it's a cool thing to know :) ... like you can easyly patch sources for you needs.. :) ...

btw.. my build is not the exact r680 .. and not the exact "dark" build.. it's a mix.. and i do have both 64 and 32... so newermind.. :)

looking forward to a new version of x264farm :)

Inventive Software

3rd October 2007, 17:40

@omion: a quick compression algorithm I could suggest is QuickLZ (http://www.quicklz.com/). Not sure how this compares to HuffYUV, but I can't see any harm in comparing, if you have the time to of course. ;)

omion

3rd October 2007, 22:19

OK, after 3 months, 1.14 is finally out. It has most of the patches which have been discussed recently.

SpAwN_gUy's "Dead?" issue should be fixed.
Also, the spacing issue legoman666 had with x264 should be fixed too.
More details in the changelog.

I was going to have this out last night, but I accidentally did something weird with my version-control system which merged the past two revisions instead of my current working version. I think I fixed it, but there may still be some odd issues that sneaked in.

The agent was changed, but only to fix the issue with x264 having spaces. If that issue does not affect you, you don't need to replace the agent.

Have fun!

SpAwN_gUy

4th October 2007, 15:23

nice.. 'll test :)

Wishbringer

5th October 2007, 16:38

first error:
using agent 1.14-181 under Vista Ultimate - 64bit and get following error:
Fatal error: exception Unix.Unix_error(_, "socket", "")

jumped back to agent 1.13-176 and all is working smooth.

second error:
even with simplest avs script I get following message from controller:
ERROR: AVS file [xxx] does not seem to be valid
and something with avs2yuv not correct.

again, with 1.13 controller works fine under Vista 64

thoughts/question:
is it still sane to use "number">1 in controller config-file since x264 can use threads=auto by itself.
I would only see an increase in memory usage.

testing:
I did some encodes with a quadcore controller and a dualcore agent
controller.exe -B "100%" -1 "--crf 18 --keyint 999 --level 4.1 --ref 8 --mixed-refs --bframes 16 --direct auto --analyse none --me dia --subme 1 --threads auto" -2 "--keyint 999 --level 4.1 --ref 8 --mixed-refs --bframes 16 --direct auto --analyse p8x8,b8x8,i4x4,p4x4 --subme 7 --me umh --b-rdo --bime --weightb --trellis 2 --threads auto" -i LotR1.avs -o LotR1.mkv

1.trial: with local "number"=4 and client "number"=2 resulting in 88fps for 1st pass and 27fps for 2nd pass
2.trial: with both "number"=1 resulting in 82fps and 25fps
not far away, but with less memory usage (1.trial was at 4GB at 90%, 2.trial was at 45%)

I think, difference in fps results in loading time between encodes where cpu usage drops to nearly 0%

omion

7th October 2007, 06:22

first error:
using agent 1.14-181 under Vista Ultimate - 64bit and get following error:

jumped back to agent 1.13-176 and all is working smooth.

This is very odd. The only thing I changed in the agent is the quoting for the x264 executable. What do you have set for <x264> in the agent's config.xml?

second error:
even with simplest avs script I get following message from controller:

and something with avs2yuv not correct.

again, with 1.13 controller works fine under Vista 64

I also changed the quoting for avs2yuv on the controller side, but it could be a lot of things.
What happens when you run this in the controller directory:
avs2yuv -frames 1 -raw -o NUL "xxx.avs" 2>&1

thoughts/question:
is it still sane to use "number">1 in controller config-file since x264 can use threads=auto by itself.
I would only see an increase in memory usage.

testing:
I did some encodes with a quadcore controller and a dualcore agent

1.trial: with local "number"=4 and client "number"=2 resulting in 88fps for 1st pass and 27fps for 2nd pass
2.trial: with both "number"=1 resulting in 82fps and 25fps
not far away, but with less memory usage (1.trial was at 4GB at 90%, 2.trial was at 45%)

I think, difference in fps results in loading time between encodes where cpu usage drops to nearly 0%

Changing the <number> in the agent's config.xml file actually does nothing. I had it limit the number of encodes, but it caused too much trouble, so I just use the <number> from the controller.

But even then, since x264 does multi-threaded encodes, <number> is not too important anymore. Assuming that multi-threaded x264 pegs your CPU, it should be fine.

However, as you noted, there is a relatively large delay between encode jobs, so you might want to try out the following line in the proper section of the controller config.xml file:
<number pad="1">1</number>

This line will use 1 encode during the first pass and 2 during the second pass. The first pass doesn't switch jobs too much, so there aren't many times when x264 is starting up. However, on the second pass there are many jobs, so it's a good idea have one extra running in the background when the other finishes.

One more thing: you noted that memory usage was 4GB with the first trial? That is quite excessive... What was using all that memory, avs2yuv or x264?

Thasp

7th October 2007, 08:17

Regardless of what I set in the XMLs, and I have gone over it a few times, I keep getting this.

Fatal error: exception Sys_error(": No such file or directory")

It's possible I keep overlooking this one small thing each time I set it up. Has anyone else made the same stupid mistake, and found out what it was? Or is it a bug for certain setups?

here's the full error.

E:\tools\x264farm\controller>controller
x264farm version 1.14-181
WARNING: First pass options are not set; this will use the default x264 settings

WARNING: Second pass options are not set; this will use the default x264 setting
s
Using config file ".\config.xml"
CONFIG:
temp_dir: "E:\\x264farm\\temp\\controller"
agent_list 1:
"Local 1" = 127.0.0.1 50700-50703
"Local 2" = 127.0.0.1 50700-50703
"Linux 1" = 192.168.0.106 50700-50703
agent_list 2:
"Local 1" = 127.0.0.1 50700-50703
"Local 2" = 127.0.0.1 50700-50703
"Linux 1" = 192.168.0.106 50700-50703
"Linux 2" = 192.168.0.106 50700-50703
Fatal error: exception Sys_error(": No such file or directory")

It gives the same error message with anything, asking it to encode. Everything but --help.

omion

7th October 2007, 08:30

Wishbringer

7th October 2007, 09:14

What do you have set for <x264> in the agent's config.xml?
is:
<x264>x264</x264>

What happens when you run this in the controller directory:
avs2yuv -frames 1 -raw -o NUL "xxx.avs" 2>&1
shows:
xxx.avs: 720x304, xxxfps, xxxframes

Changing the <number> in the agent's config.xml file actually does nothing.
I did only change <number> in the controller config.xml, I didn't even know that there is <number> in agent's config.xml too.

... there is a relatively large delay between encode jobs, so you might want to try out the following line in the proper section of the controller config.xml file:
<number pad="1">1</number>
Thanks! Didn't know that it is possible for controller too.

... you noted that memory usage was 4GB with the first trial?... What was using all that memory, avs2yuv or x264?

I think x264, because it uses for it searchs several frames in buffer (even more with high settings for ref-frames and bframes) and that multiplied by 4, because x264 was started four times).
But next time I will look deeper into it.
Maybe because i use "--keyint 999"?!

Thasp

7th October 2007, 09:40

@Thasp:
That means you need to tell it what to encode. ;) Use the "-i file.avs" option to tell it which file to use. Unlike the agents, the controller needs something to do before it can start up.

I probably should have a better error message for that, though...

Doh.. I was also putting the first pass, before specifying the avs. When I put the AVS behind specifying what pass, it "works"

Now there's new fun stuff.

I can get one of the machines to actually encode the first pass, at 25% CPU, and the second won't encode at all.

24 doing the following:
nice -n 10 "x264" --crf 19 --sar 10:11 --no-psnr --direct auto --subme 5 --ref
3 --me hex --bframes 3 --weightb --b-pyramid --analyse all --qcomp 0.6 --cplxbl
ur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3
--fps 24000/1001 --pass 1 --stats "f:\x264farm\stats 2f0ce0.txt" -o NUL - 1280x
720
x264 [info]: using SAR=10/11
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSSE3

Thread received exception "Sys_error(\"Broken pipe\")"
closing socket
socket closed
29 recv thread got exception; killing listener thread

Setting more jobs to send to the other machine, 6, which is the threads I use on a quad core to ge tmost speed sped it up. But it's been over a half hour, of consistent 100% CPU, and it's just now almost done. On a dual core with the settings I specified on this first pass I could have been finished already.. :(

I'll see if second pass can be split across both machines. If the same error as first pass, maybe do the first pass on one machine, and the second pass with beefier settings on both, but I'm not sure quite how since there doesn't seem to be a way to specify a stats file to use.

rhodges

8th October 2007, 00:53

Vista 64, experiencing the following error as well:

WARNING: avs2yuv exited with error 1; avs2yuv responded:
The system cannot find the path specified.
2007-10-07~17:06:21.07
ERROR: AVS file '\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e16.avs' does not seem to be valid

The only thing I did was drop in the new binaries and fired up the encoder.

Edit:
I can paste the path into start/run and open the AVS file and I can even play the AVS in Media Player Classic.

omion

8th October 2007, 04:34

Doh.. I was also putting the first pass, before specifying the avs. When I put the AVS behind specifying what pass, it "works"

Now there's new fun stuff.

I can get one of the machines to actually encode the first pass, at 25% CPU, and the second won't encode at all.

[error!]

Setting more jobs to send to the other machine, 6, which is the threads I use on a quad core to ge tmost speed sped it up. But it's been over a half hour, of consistent 100% CPU, and it's just now almost done. On a dual core with the settings I specified on this first pass I could have been finished already.. :(

The "broken pipe" error makes it sound like there is some network issue, as does the fact that a computer will only encode at 25% CPU.

A few questions:
1. What speed is your network running at? 10mbps will be far too slow, and even 100mbps may not be enough to max out a quad-core.
2. If you turn off the agent running on the controller's computer, what is the CPU usage? If it's maxed out, then it means the controller can't render the AVS file fast enough.
3. What is your AVS file like? Are there a lot of filters in it?

Using agent-based encoding will probably be the best, and it will solve all those problems, but it is somewhat difficult to set up. (I think SpAwN_gUy gave a little guide on how to do it somewhere in this thread...)

I'll see if second pass can be split across both machines. If the same error as first pass, maybe do the first pass on one machine, and the second pass with beefier settings on both, but I'm not sure quite how since there doesn't seem to be a way to specify a stats file to use.

I don't have anything set up to specify a first-pass file to use, other than stuffing it into the temp directory. I have an idea to output a stats file if only the first pass is specified, and input a stats file if only the second pass is specified. It is, however, unimplemented. I'll work on that for the next version.

@all: I just finished a C-optimized version of the Paeth encoder which compression uses. It's now 3 times faster than the OCaml version, which should give a nice performance boost to compression. I'm also working on an improved buffer scheme to make agents run a bit better with controller-based encoding.

ChrisBensch

8th October 2007, 06:44

Using agent based encoding, I get this error:
Oops. Writing the args failed with: Sys_error("D:\\TEMP\\option_log.txt.temp: No such file
or directory")

The encode still runs and all passes finish. My question is, why is this happening and what options may not be in use for x264?

omion

8th October 2007, 07:31

omion

8th October 2007, 07:37

@ChrisBensch:

The controller keeps a log of all the options you've used in the past. This is to help you restart the encode if the controller's computer crashes and you don't remember the exact settings you used. That error just means that the log could not be written for some reason. The actual encoding output was not compromised in any way.

To stop the message, go into the directory "D:\TEMP" and see if there's a file called "option_log.txt". If not, create it (blank file) and restart the controller. It shouldn't give the message any more.

It's kind of annoying, though. I thought I killed that bug...

rhodges

8th October 2007, 11:17

@rhodges:
Try running it through the command I gave Wishbringer, i.e.
avs2yuv -frames 1 -raw -o NUL "\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e16.avs" 2>&1
If and only if this succeeds will x264farm understand it. I've actually never used the default admin shares "c$", "d$", etc, and I've actually turned them off my computer (don't remember how), but I bet it has something to do with it.

If it doesn't work, make a user share in a sub-directory and try to access it from there.

The command works:

Z:\NetworkEncoder\Bin\controller>avs2yuv -frames 1 -raw -o NUL "\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1Series 01\SG1 S01e16.avs" 2>&1
\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e16.avs: 640x352, 24000/1001 fps, 63600 frames

Here is the command I'm using to start the controller:

Z:\NetworkEncoder\Bin\controller>..\controller.exe --config .\config-controller-desktop.xml settings-sd.txt -i "\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e01.avs" -o "SG1 S01e01.mkv"

Like I mentioned, these worked (I encoded these once but didn't like the output) with 1.13. I upgrade to 1.14 to see if it would help with the occassional 2nd pass hanging (an agent would die and the controller wouldn't always notice)

omion

8th October 2007, 16:58

@rhodges:
I probably screwed up the avs2yuv quoting when I changed everything around. See if adding the following line to the controller's config.xml file:
<avs2yuv>avs2yuv</avs2yuv>
add it to the same level as the <temp> and <agents> parts (the top level, basically)

rhodges

9th October 2007, 02:53

omion

9th October 2007, 08:46

That fixed the problem, thanks!

Well, that just means that my automatic avs2yuv-finder is broken somehow. Download this file (http://omion.dyndns.org/x264farm/controllerinfo-1.14ish.rar), and run it exactly as you would run the controller (put in the same directory as the controller) However, only use the -i option. Based on your previous post, it should look something like this:
Z:\NetworkEncoder\Bin\controller>..\controllerinfo.exe -i "\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e01.avs"
Then post the output here. Thanks!

rhodges

10th October 2007, 01:44

Here are the results:

Z:\NetworkEncoder\Bin\controller>..\controllerinfo.exe -i "\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e01.avs"
x264farm version 1.14-181
WARNING: THIS CONTROLLER DOES NOT WORK!
AVS: "\".\\avs2yuv.exe\""
Running "\".\\avs2yuv.exe\" -frames 1 -raw -o NUL \"\\\\192.168.1.5\\z$\\NetworkEncoder\\Source\\SG1\\SG1 Series 01\\SG1 S01e01.avs\" 2>&1"

WARNING: avs2yuv exited with error 1; avs2yuv responded:
GOT NORMAL INFO "The system cannot find the path specified.\013"

ERROR: AVS file '\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e01.avs' does not seem to be valid

Z:\NetworkEncoder\Bin\controller>

Looks like relative pathing back up a directory is the problem?

Feature Request:

The ability to specify the log file name as a command line argument to agent.exe and controller.exe.

omion

10th October 2007, 02:02

Looks like relative pathing back up a directory is the problem?
One more question: where exactly do you have avs2yuv?

Feature Request:

The ability to specify the log file name as a command line argument to agent.exe and controller.exe.
Somebody (I think it was SpAwN_gUy) requested that before, but I was hesitant to do it. The problem (back then) was that it implied that it had to be written after the command-line arguments are specified. That means that if something goes wrong before the command-line is parsed, there is no log of it.

However, I think now the programs are stable enough to delay writing until the args are parsed, so I think I'll add it in.

rhodges

10th October 2007, 02:32

One more question: where exactly do you have avs2yuv?

It's in both the controller directory and up a directory. I wanted all the agents to run from a network share, so I setup the structure below and then relative pathed backup a directory to run the .exe so the log file would say in the original folder (hence my FR to specify the log file).

I had some problems, I think with nice.exe or maybe even avs2yuv, so I ended up just writing a batch file to push the .exe into each folder (hence the duplicates you see below)

It does however, seem that in some places I'm using the .exe in the folder and sometimes the .exe in the folder above.

Bin\
charlene\
charlene.bat
charlene.xml
nice.exe
x264.exe
agent.exe
avs2yuv.exe
controller.exe
controller\
... (same binary files above and similar .bat, .xml file)
desktop\
...
htlink\
...
htserver\
...
nice.exe
x264.exe
agent.exe
avs2yuv.exe
controller.exe
controllerinfo.exe

what I'd ultimately like to do is have something like:

NetworkEncoder\
logs\
agent.charlene.log
agent.desktop.log
agent.htlink.log
agent.htserver.log
controller.controller.log
settings\
agent.charlene.xml
agent.desktop.xml
agent.htlink.xml
agent.htserver.xml
controller.controller.xml
bin\
nice.exe
x264.exe
agent.exe
avs2yuv.exe
controller.exe
charlene.bat
controller.bat
desktop.bat
htlink.bat
htserver.bat

What performance impact would there be with having the agents put there temp files on the network? Everyone is connected via gigabit and the network share is actually an 8 drive raid 5 array.

On a side note, I think 1.14 is now catching the occasional agent dieing on the 2nd pass causing the encode to hang issue I was having. :)

omion

10th October 2007, 04:12

OK. I finally repeated the problem, so it'll be fixed in the next version. In the meantime, keep using the config.xml workaround I posted before.

I had forgotten that Windows will remove the first and last quote from the command it's executing. So the line it was trying to run:
".\avs2yuv.exe" -frames 1 -raw -o NUL "\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e01.avs" 2>&1
actually turns out running:
.\avs2yuv.exe" -frames 1 -raw -o NUL "\\192.168.1.5\z$\NetworkEncoder\Source\SG1\SG1 Series 01\SG1 S01e01.avs 2>&1
which is wrong... I guess I removed my previous workaround when I added the ability to specify your own avs2yuv file. Well, it'll be fixed in the next update!

What performance impact would there be with having the agents put there temp files on the network? Everyone is connected via gigabit and the network share is actually an 8 drive raid 5 array.
Not much. The temp directory is only for holding highly compressed data - the stats file on the first pass and an output GOP on the second pass. I would be surprised if there was any difference with a gigabit network.

On a side note, I think 1.14 is now catching the occasional agent dieing on the 2nd pass causing the encode to hang issue I was having. :)
Great!

SpAwN_gUy

19th October 2007, 15:02

freakin' awesome..

my meGUI mod is allmost complete (if anyone asks..)..
i only need to implement error handling (something about those 3 lines (or is it 4.. ??) in the end..) and that's all..

you can choose x264 profiles and edit them.. and all unNeeded (and need) options are properly handled ;) ...

i guess next week or so.. and meybe it will be in the "official" meGUI (i wish..)

Finar

19th October 2007, 18:59

SpAwN_gUy
my meGUI mod is allmost complete (if anyone asks..)..
Yes, thank you! I hardly wait for this MOD. And I promise to publish the material about x264farm & your meGUI MOD in Russian IT Magazine "computerra" ( www.computerra.ru ), so all this will get some more popularity I hope.

SatansChild

22nd October 2007, 03:20

I can't seem to get this to compile on 64 bit kubuntu system, from the output I think it's a dependency I"m missing but I'm not sure what. per omion's first post tine the thread I have both extlib and xml-light installed ( the libextlib-ocaml-dev and libxml-light-ocaml-dev packages under kubuntu). The common portion seems to compile fine but gives an error when I try to compile the agent portion after. Anybosy have any ideaas what I"m missing? Compiler output with the pertinent error is below.

root@core:/workspace/x264farm/agent# make
ocamlopt -c -thread -I ../common unix.cmxa threads.cmxa prodcons.ml
ocamlopt -c -thread unix.cmxa threads.cmxa agentinclude.ml
ocamlopt -o agent -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx agentinclude.cmx list2.cmx huff.cmx net.cmx matroska.cmx prodcons.cmx agent.ml
File "agent.ml", line 610, characters 10-19:
Warning Y: unused variable heartbeat.
File "agent.ml", line 794, characters 10-19:
Warning Y: unused variable heartbeat.
File "agent.ml", line 561, characters 6-11:
Warning Y: unused variable fps_f.
File "agent.ml", line 87, characters 4-43:
Unbound constructor Xml.Element
make: *** [agent] Error 2

omion

22nd October 2007, 04:11

I can't seem to get this to compile on 64 bit kubuntu system, from the output I think it's a dependency I"m missing but I'm not sure what. per omion's first post tine the thread I have both extlib and xml-light installed ( the libextlib-ocaml-dev and libxml-light-ocaml-dev packages under kubuntu). The common portion seems to compile fine but gives an error when I try to compile the agent portion after. Anybosy have any ideaas what I"m missing? Compiler output with the pertinent error is below.
Hmm... That's an xml-light error. The Ubuntu repositories seem to have xml-light in the wrong place (or at least not in the place that the standalone installer uses).

Probably the easiest thing to do would be to copy all the xml-light files from their directory to the ocaml directory. The following should do the trick:
cd /usr/lib/ocaml/3.09.1/xml-light
sudo cp xml-light.cmxa xml-light.lib xml-light.cma xml.mli xmlParser.mli dtd.mli xml.cmi xmlParser.cmi dtd.cmi xml.cmx dtd.cmx xmlParser.cmx `ocamlc -where`
If the computer complains about the first command, use 3.09.2 instead of 3.09.1.

It should work after that.

@all: I'm almost done with the next version of x264farm. Look forward to all sorts of Extra Awesomeness sometime this week (probably) :cool:

SatansChild

22nd October 2007, 05:50

It looks like I'm missing the xml-light.lib file.

root@core:/usr/lib/ocaml/3.09.2/xml-light# cp xml-light.cmxa xml-light.lib xml-light.cma xml.mli xmlParser.mli dtd.mli xml.cmi xmlParser.cmi dtd.cmi xml.cmx dtd.cmx xmlParser.cmx `ocamlc -where`
cp: cannot stat `xml-light.lib': No such file or directory
root@core:/usr/lib/ocaml/3.09.2/xml-light# ls
dtd.cmi dtd.mli xml.cmx xml-light.cma xml.mli xmlParser.cmx
dtd.cmx xml.cmi xml-light.a xml-light.cmxa xmlParser.cmi xmlParser.mli

At which point the compile chokes looking for a xml-light.a file.
gcc: /usr/lib/ocaml/3.09.2/xml-light.a: No such file or directory
Error during linking

Once I coppied that file over too it seems to compile ok, I only get warnings (I assume these are normal?) and the executable is created.

ocamlopt -o agent -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx agentinclude.cmx list2.cmx huff.cmx net.cmx matroska.cmx prodcons.cmx agent.ml
File "agent.ml", line 610, characters 10-19:
Warning Y: unused variable heartbeat.
File "agent.ml", line 794, characters 10-19:
Warning Y: unused variable heartbeat.
File "agent.ml", line 561, characters 6-11:
Warning Y: unused variable fps_f.
File "agent.ml", line 302, characters 25-7338:
Warning P: this pattern-matching is not exhaustive.
Here is an example of a value that is not matched:
'a'

Unless ubuntu changes thier version of xml-light to be more standard it may be easier for people to just copy everything in the xml-light dir up one dir.

sudo cp /usr/lib/ocaml/3.09.2/xml-light/* /usr/lib/ocaml/3.09.2/

omion

22nd October 2007, 06:33

@SatansChild:
Oh yeah. The .lib is Windows-only. Linux version has an .a extension. I just copied that line from the makefile for xml-light.

And yes, those warnings are normal.

omion

26th October 2007, 09:13

OK! Since the local fires (http://www.nytimes.com/2007/10/26/us/26calif.html) have given me a week off, I decided to work on 1.15. So here it is! (http://omion.dyndns.org/x264farm/x264farm-1.15_win32.rar)

The biggest improvement is something that I hope you can all help me out on. I have implemented a configuration-less "ad-hoc" networking mode. Basically, when enabled, the controller will be able to see all the agents on the local network. Agents can also be dynamically added to a running encode simply by starting them up - the controller should recognize it immediately.

Due to the fact that every computer on the network will receive this signal, and not every computer may be yours, this feature is not enabled by default(*). However, I would appreciate any feedback regarding it. To enable it, do the following:

1. Add the following section to the first level of your agent's config.xml file (I put mine right above the <port> element):
<name>Agent Name</name>
<adhoc controller="40700" agent="40701"/>
<number pad="1">2</number><name> specifies the name for the agent to broadcast as.
<adhoc ... /> specifies the ports to broadcast to (probably shouldn't change this line)
<number> is set up the same was as in the controller's <agent> section.

2. In the controller's config file, add the following section:
<adhoc controller="40700" agent="40701"/>
This should match the line from the agent's file.

3. Run the encode as normal. If you want, comment out the agents in the controller's config file to make sure they are detected automatically. If you don't specify a name in the agent's config file, the controller will use the agent's IP address.

Once any agents are found, encoding will commence normally.

Please test this out and reply with any issues. If all goes well, I am hoping to have this feature enabled by default, and de-emphasize the config.xml-based agent setup.

Other things that I changed:

The display looks different. It now gives one line per computer, not per parallel agent. It should make it more obvious which computers are doing what.
Log file location may now be specified on the command line (use the --logfile option for either the agent and controller) (so stop nagging me about it:devil:)
Compression should be more efficient. I haven't benchmarked the entire compression, but the Paeth transformation is now 2.5x-3x faster than before. (The compression is composed of Paeth transformation followed by Huffman encoding)
The agent's frame cache is now based on a circular string buffer rather than a variable-length string-array. (Hooray!:p)
I changed the port in my example XML files from 50700 to 40700. 50700 is in the "ephemeral port" range, which may cause problems on certain OSes. Never ran into any, though...

WARNING: There has been one regression since the last version. The controller will now only look at the first port in the agent's port range. That means if you have a controller set up like this:
<agent name="Somewhere">
<ip>192.168.1.1</ip>
<port from="40700" to="40703"/>
<number pad="1">1</number>
</agent>then the controller will only look at port 40700. If the agent did not connect to that port, others will not be tried.
If you are using the new ad-hoc agent discovery, this will not matter since the agent will tell the controller the correct port to use. However, if you're not, the agent will not receive any work.
This is an unfortunate side-effect of (oddly enough) my new printing scheme. I'm thinking of a way to put it back in, but it didn't make the 1.15 release.

(*) And yes, I am aware of the irony of having to configure a configuration-less setup, but this should only be temporary.:p

SpAwN_gUy

26th October 2007, 09:40

WTF... i've missed 1.15.... damn :(

btw.. do we need (in error handling) backward compatibility with 1.13? where "Recent errors" were empty strings.. and in 1.14 they are "~"

i'll test with 1.15..

(for now.. errors are "properly" handled when controller is 1.13 and agents are 1.14.. and i can test "WARNING"s and "ERROR"s with 1.14 and inproper avs2yuv ... thanks for that ;) )

UPD:
ggg.. so it was not me :) ... i was too quick ;) .. 'kay.. i'll try those NEW things.. and post some results...

SpAwN_gUy

26th October 2007, 10:02

Due to the fact that every computer on the network will receive this signal, and not every computer may be yours, this feature is not enabled by default(*).
(*) And yes, I am aware of the irony of having to configure a configuration-less setup, but this should only be temporary.:p
'kay... the problem with this mode.. is.. hm..

you can actually be banned for pinging ports on evry mashine.. and if you have network with.. 400PCs.. ??? ...

IMO you'd better change this feature to Opposite..

i have the controller.. I'm spreading pre-configured agents.. and.. i think.. agents, most likely, should ping the controller (and i will pre config mine IP(the controller's) in the agent's config)

but.. i just came to that: we'll loose "multi-controller" compatibility (partially).. and agents will allways (when started, not only when encoding) ping my PC.. and tell me "Gimmie something to encode"... (BTW.. this will encourage me to give them something to encode ;) )

SpAwN_gUy

26th October 2007, 10:36

test-results:
with no "adhoc" on the controller and "yes and no" on the agent side ... and good old ports 50700 - 50703..
it's NOT working at all.. :( ..
config is properly loaded.. all agents are in the list(the sart-up one).. and no agents are doing something.. (well.. they are not listed, not a single one..)..
FIRST PASS
0% done (0 / 4197) at 0.00 FPS
Last updated: 0:07
--------------------------------
Agent | FPS | Description
Recent errors:
~
~
~
~ for 3 minutes now..

i'll try "adhoc" and post..

UPD: WITH adhoc in the controller - it all works "fine" (as it was)... but it starts more slowly..

.... okay.. i give up.. i've tried my older params lines.. and generated by meGUImod..
the Whole thing is NOT working... when in the --first param of the controller "--crf 12" or "--pass 1 --bitrate 1000" is NOT specified..
i'm not sure where the legs came from.. i'll try another x264build(this was mine..)

UPD3: tried Both...

Agent gets one the first-pass:
nice -n 20 "x264.exe" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\stats 600b48.txt" --seek 0 --frames 4197 -o NUL
when --first option is:
--first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim"
there is NO "--crf" or something specified... and when i add "--bitrate 1000" it works..
i dunno what's wrong with everything..

i suppose "--pass 1" gives .stats output.. and "--crf" - does not.. but it could be overriden..
adding "--bitrate 1000" - starts the encode... but i suppose we loose quality-based mode
adding "-crf" - does not fix the problem
adding "-crf 18" - fixes it
adding "-crf 18 --bitrate 1000" - fixes the problem,.. but.. i'm not sure about "quality mode" :(

foxyshadis

26th October 2007, 10:55

It's a broadcast, not pings, so it'll only work on a local subnet, and they're so common (smb browse messages, dhcp, ARP, printer lookups...) that it's not going to hurt to have one more sent out now and then. I agree the agents need that capability as well, though it could get quite sticky if there are two controllers on the subnet. And you might need to create a new functionality of agents that do get manually tapped to relay broadcasts back to the controller - if you expect this to ever be used in large routed networks, that is, which might not be the case. (Find out how long it takes to encode HD LotR with the whole Sears Tower working on it!)

This automatic topology building is totally hot, omion. That's almost half the setup effort reduced right there.

omion

26th October 2007, 19:52

'kay... the problem with this mode.. is.. hm..

you can actually be banned for pinging ports on evry mashine.. and if you have network with.. 400PCs.. ??? ...

IMO you'd better change this feature to Opposite..
I was fairly worried about this, but as foxyshadis pointed out, it's a broadcast message, not a port scan or anything. Nothing can be banned from sending out a few of these (they're actually required for routing sometimes) but I'm mainly worried about sending out many of them.

i have the controller.. I'm spreading pre-configured agents.. and.. i think.. agents, most likely, should ping the controller (and i will pre config mine IP(the controller's) in the agent's config)The way it actually works is the controllers will send out a broadcast every minute, and any connected agents will send a response to that controller. Also, the agents will send one broadcast when they start up, then stay quiet for the rest of the time. The total network utilization for this is around 400 bytes per minute (depending on the number of agents)

but.. i just came to that: we'll loose "multi-controller" compatibility (partially).. and agents will allways (when started, not only when encoding) ping my PC.. and tell me "Gimmie something to encode"... (BTW.. this will encourage me to give them something to encode ;) )

Yeah. The agents still don't know when to stop encoding, so this is mainly good for doing only one encode per network at a time. I'm really hoping to add functionality to actually limit the number of encodes that an agent does, but it's quite difficult to do correctly in Windows.

UPD: WITH adhoc in the controller - it all works "fine" (as it was)... but it starts more slowly..How long does it take to recognize all the agents?

with no "adhoc" on the controller and "yes and no" on the agent side ... and good old ports 50700 - 50703..
it's NOT working at all..Sorry, but what is "yes and no"? The <adhoc> section must (currently) be present in both the agent and controller config files for it to work.

there is NO "--crf" or something specified... and when i add "--bitrate 1000" it worksWell... x264 needs some sort of bitrate information for the first pass. Without --bitrate or --crf or --qp x264 will refuse to encode.

SpAwN_gUy

29th October 2007, 08:49

How long does it take to recognize all the agents? as far as i have 10agents written in config.. and adhoc option enabled (everywhere) it is only.. hm.. 2 seconds :) ... (400PCs in the LAN)
Sorry, but what is "yes and no"? The <adhoc> section must (currently) be present in both the agent and controller config files for it to work.well,.. if there is NO such option in controllers config.. then controller - does not work..

if it is there - everything works fine..

Well... x264 needs some sort of bitrate information for the first pass. Without --bitrate or --crf or --qp x264 will refuse to encode.but evrything worked fine... some time ago.. (i guess .. x264farm 1.09) ... maybe the x264 had different behaviour ??? ...

'coz.. i dont know what to do in meGUI .. i need to add "--crf 19" or 21(the default) ??? ... so we would not loose the functionality and "quality-mode" would properly work...

omion

29th October 2007, 18:09

as far as i have 10agents written in config.. and adhoc option enabled (everywhere) it is only.. hm.. 2 seconds :) ... (400PCs in the LAN)
That's fast enough ;) It does take a bit of time to set up the agents, so that's to be expected
well,.. if there is NO such option in controllers config.. then controller - does not work..

if it is there - everything works fine..
Ah. Yes, it does have to be enabled in both config files. If the controller doesn't have it, it won't bother to search. If the agents don't have it, they won't reply

but evrything worked fine... some time ago.. (i guess .. x264farm 1.09) ... maybe the x264 had different behaviour ??? ...

'coz.. i dont know what to do in meGUI .. i need to add "--crf 19" or 21(the default) ??? ... so we would not loose the functionality and "quality-mode" would properly work...
Yeah, x264 changed a while back. It used to assume (I think) "--qp 26". I don't remember exactly when it changed, but it caught me off guard too.

Wishbringer

29th October 2007, 19:07

Yeah, x264 changed a while back. It used to assume (I think) "--qp 26". I don't remember exactly when it changed, but it caught me off guard too.

x264 rev.655 (mid may 07)
http://forum.doom9.org/showpost.php?p=1003219&postcount=319

And I found another (forgotten) link:
http://forum.doom9.org/showpost.php?p=989251&postcount=228
which one is better version of nice?
first post link from march or your later posted link from april?

omion

29th October 2007, 23:40

x264 rev.655 (mid may 07)
http://forum.doom9.org/showpost.php?p=1003219&postcount=319That was the one. Thanks! :)

And I found another (forgotten) link:
http://forum.doom9.org/showpost.php?p=989251&postcount=228
which one is better version of nice?
first post link from march or your later posted link from april?

Oops. The April one is better, although the only thing I changed was the Vista compatibility. I'll update the link on the main page ASAP.

SpAwN_gUy

30th October 2007, 12:04

Ah. Yes, it does have to be enabled in both config files. If the controller doesn't have it, it won't bother to search. If the agents don't have it, they won't replysorry... maybe i'm writing somehow veri bad ;) ..

but.. i am trying to point on:
when i use my good old config.xml:

and start an encode.. (the test one.. that works fine.. btw.. i need to get some other testfile..)
i only get:
x264farm controller version 1.15-182
Using config file ".\config.xml"
WARNING: Output file 'E:\DVDRiP\!!x264farm\TEST\X\test-meGUI-HQInsane.mkv' already exists and will be overwr
itten!
CONFIG:
temp_dir: "G:\\DVDRiP\\_temp"
agent list:
pc79 = 010.00.00.000:50700 (1,2)
pc206 = 200.00.00.000:50700 (1,2)
...omitted, security purposes... there are 8 of them.......
pc17 = 100.000.000.00:50700 (1,1)

FIRST PASS
0% done (0 / 4197) at 0.00 FPS
Last updated: 0:15
----------------------------------------------------------------------------------------------------------
Agent | FPS | Description
Recent errors:
~
~
~
~

and.. THAT'S ALL... not even updated (for 5 mins now..)

but, when i UNcomment the adhoc in config - Everything works fine..

i only want to clarify, that (now) this feature is not Optional, but mandatory... ;)

Yeah, x264 changed a while back. It used to assume (I think) "--qp 26". I don't remember exactly when it changed, but it caught me off guard too.so.. i suppose when comLine generation for controller is done by hand... one can fix that..

but for meGUI ??? ... (i already omitted "--pass" beginnings.. and output trailings...)..
well.. i can add "--crf 19" with no problem.. so first pass will work ... or should i rise or lower this value?

SatansChild

1st November 2007, 05:06

The new version won't compile for me, it seems to have some issues with the c-optimizations.c (in the common dir). Make output is posted below.

root@core:/workspace/x264farm/1.15/common# make
ocamlopt -c -thread unix.cmxa threads.cmxa pack.ml
ocamlopt -c -thread unix.cmxa threads.cmxa pack.cmx types.ml
ocamlopt -c -thread unix.cmxa threads.cmxa types.cmx pack.cmx net.ml
ocamlopt -c -thread rbtree.mli
ocamlopt -c -thread unix.cmxa threads.cmxa rbtree.ml
ocamlopt -c -thread list2.mli
ocamlopt -c -thread unix.cmxa threads.cmxa list2.ml
ocamlopt -c -thread unix.cmxa threads.cmxa matroska.ml
ocamlopt -c -thread buffer2.mli
ocamlopt -c -thread unix.cmxa threads.cmxa buffer2.ml
ocamlopt -c -thread unix.cmxa threads.cmxa pack.cmx types.cmx huff.ml
ocamlopt -c c-optimizations.c
c-optimizations.c: In function ‘do_paeth’:
c-optimizations.c:30: warning: pointer targets in initialization differ in signedness
c-optimizations.c:37: warning: unused variable ‘p’
c-optimizations.c: In function ‘undo_paeth’:
c-optimizations.c:100: warning: pointer targets in initialization differ in signedness
c-optimizations.c:107: warning: unused variable ‘p’
ocamlopt -c c-optimizations.obj optimizations.ml
/usr/bin/ocamlopt: don't know what to do with c-optimizations.obj.

omion

2nd November 2007, 00:59

sorry... maybe i'm writing somehow veri bad ;) ..

but.. i am trying to point on:
when i use my good old config.xml:

and start an encode.. (the test one.. that works fine.. btw.. i need to get some other testfile..)
i only get:
...
and.. THAT'S ALL... not even updated (for 5 mins now..)CR@P! I screwed up. I had wanted to have ad-hoc enabled by default for this build, but I backed out of it at the last second. So I turned off the thread that receives the broadcast responses from the agents... the problem is the agents are now started by that thread too. So basically none of the agents are started unless ad-hoc is enabled.

The way around this is, as SpAwN_gUy noted, to uncomment the <adhoc> element in the controller's config file.

If you think your IT guys will complain about the broadcast packets, you can turn off the broadcast with the --noadhoc command line option. This will keep the controller listening for agent responses, but it will stop it from actively searching for agents.

@SatansChild:
For Linux, you have to run "make OBJ_EXT=.o" instead of just "make". It's a pain, and I should really document that better... I think there's a way to make OCaml keep consistent object extensions, but I haven't looked into it too far.

SatansChild

2nd November 2007, 03:12

Adding OBJ_EXT=.o seems to help it get passed the c-optimizations.c file but then it runs into issues with the controller.ml file / module OptParse.
root@core:/workspace/x264farm/1.15/controller# make OBJ_EXT=.o
ocamlopt -c c-console.o console.ml
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.ml
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx rbtree.cmx ratecontrol.ml
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx huff.cmx pass1.ml
File "pass1.ml", line 1641, characters 9-18:
Warning Y: unused variable new_range.
File "pass1.ml", line 2052, characters 8-20:
Warning Y: unused variable start_offset.
File "pass1.ml", line 2051, characters 8-27:
Warning Y: unused variable found_some_i_frames.
File "pass1.ml", line 1401, characters 6-11:
Warning Y: unused variable print.
File "pass1.ml", line 430, characters 5-17:
Warning Y: unused variable print_ranges.
File "pass1.ml", line 378, characters 5-14:
Warning Y: unused variable batch_now.
File "pass1.ml", line 42, characters 5-16:
Warning Y: unused variable first_batch.
File "pass1.ml", line 36, characters 5-31:
Warning Y: unused variable first_pass_frames_done_ref.
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx huff.cmx rbtree.cmx matroska.cmx ratecontrol.cmx pass2.ml
File "pass2.ml", line 1462, characters 6-11:
Warning Y: unused variable print.
File "pass2.ml", line 680, characters 5-21:
Warning Y: unused variable print_agent_hash.
File "pass2.ml", line 46, characters 5-15:
Warning Y: unused variable version_rx.
ocamlopt -o controller -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa extLib.cmxa pack.cmx rbtree.cmx types.cmx ../common/c-optimizations.o optimizations.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx huff.cmx matroska.cmx ratecontrol.cmx pass1.cmx pass2.cmx controller.ml
File "controller.ml", line 25, characters 0-13:
Unbound module OptParse
make: *** [controller] Error 2

omion

2nd November 2007, 06:19

Adding OBJ_EXT=.o seems to help it get passed the c-optimizations.c file but then it runs into issues with the controller.ml file / module OptParse.
Hmm. Where is the file "optparse.cmx"? It's very likely that it is not where I expected it to be. That's the most likely explanation.

The makefile expects the libraries to be in the directory specified by "ocamlc -where", but I've seen some configurations which put all the extlib modules in their own subdirectory.

SpAwN_gUy

9th November 2007, 13:08

okay... some minor Update ;) ...

now i have my own (not so own.. but) Branch in meGUI developement on SF.net .. i'm not so familiar with SF so far.. so..

this one i kinda Release Candidate of meGUI x264farm mod :) ... i've made this build to get some feedback ;)

changelog:
- updated to latest meGUI
- moved to SF.net (sources are HERE (http://megui.svn.sourceforge.net/svnroot/megui/branches/x264farm) (use TortoiseSVN to checkout and MSVS C# edition to build))
- x264 profiles selection IS working, but Profile-Name is NOT stored
- ".log" and other stuff is present.. (if not, just tell me)

currently thougths are about:
"do we really need manual editing for first and second pass settings?"
"what do we need?" and "any suggestions?"

http://rapidshare.com/files/68496094/megui-x264farm_mod-2k7.11.09.exe

i guess i'll make something with SF.net file storage of future versions.. (if there will be any)

and as soon as x264farm itself will be more-less stable ;) ... and "mod" will meet all the needs.. this will move to main thread and x264farm itself will take it's place on meGUI's autoUpdate ;) ... omion? are you there? ;)

Adub

9th November 2007, 22:46

Super cool!
Super lame I can't test though. Or even use. :(

SpAwN_gUy

12th November 2007, 08:52

Super cool!
Super lame I can't test though. Or even use. :(
tell me what is NOT working, and what do you do?

'cause my extraSense's team is drinking alot currently, so the can't really work meanwhile.. ;)

Adub

12th November 2007, 10:14

SpAwN_gUy

12th November 2007, 11:31

NO, no, no. I meant that I can't use it. Personally. I am a poor college student living in a dorm at CalPoly San Luis Obispo with 2 other guys. I don't have the first thing that even looks like a network.

So, I am sure it works fine, it is just that I can't use it. Not that I don't want to.
i guess.. (in our prices)
switch 8-ports: 10-12$
cable (5-10m): 5$
so you will have a network ;) ...

'kay... 10 downloads of meGUI-Mod.. and NO Feedback?? :(

Adub

12th November 2007, 12:24

I would still need computers on the ends of those cables. and I don't think my roommates laptops count.

To everyone else, FEEDBACK DAMMIT!!!! I want to know if this works so that I have something to look forward too, as classes are kicking my ass!

SpAwN_gUy

12th November 2007, 12:54

I would still need computers on the ends of those cables. and I don't think my roommates laptops count. why not?
omion is using his macBook laptop as one of his encoding mashines.. (as seen in "config.xml" shipped with controller)

Yoshiyuki Blade

12th November 2007, 13:25

I just downloaded it. It'll be a while before I can learn how to use this thing, and it'll probably be longer for me to set things up :D. I'll provide feedback asap.

Jabroni

16th November 2007, 18:38

Great software.. i use this on 2 computers i have.. both quads, and it works wonderfull.. i just have one question.. Is there a way to compute the x264 output stats of all parts encoded?? :confused:

AVmaniac

19th November 2007, 01:47

Really nice work!!!

Now everybody can use the performance of x264farm with the control through megui .. thanks, works really good so far!

okay... some minor Update ;) ...

now i have my own (not so own.. but) Branch in meGUI developement on SF.net .. i'm not so familiar with SF so far.. so..

this one i kinda Release Candidate of meGUI x264farm mod :) ... i've made this build to get some feedback ;)

Jabroni

19th November 2007, 03:38

i guess.. (in our prices)
switch 8-ports: 10-12$
cable (5-10m): 5$
so you will have a network ;) ...

'kay... 10 downloads of meGUI-Mod.. and NO Feedback?? :(

Ok i started playing with it.. but i have several questions.. where do i specify the ips of the 'workers' ?? (lets say I already have the agent client working on my other computer). Right now I have x264farm run as client based (that would max both of my quad on each computer).. or what is a 'worker' used for? (for me a 'worker' would be another agent client.. but not quite sure what is under your nomenclature)...

SpAwN_gUy

19th November 2007, 10:33

Ok i started playing with it.. but i have several questions.. support-unit-mode.. OFF:
RTFM ;) ... "x264farm.html" shipped with binary..
where do i specify the ips of the 'workers' ?? (lets say I already have the agent client working on my other computer). support-unit-mode.. ON:
"config.xml"
Right now I have x264farm run as client based (that would max both of my quad on each computer).. or what is a 'worker' used for? (for me a 'worker' would be another agent client.. but not quite sure what is under your nomenclature)...
i assume: worker = agent
"not" "worker" = controller..

try proper tutorial in my signature..

when you'll set up agent-based - youll have both avs(decoded) and x264(encoded) buy each agent.. .. and just a bit by controller..

if you have few agents - then i guess it does not really matter controller- or agen-based method to use..
but if you have "heavy" avs-script.. then agent-based is preferrable.. i think..

SpAwN_gUy

19th November 2007, 10:36

Great software.. i use this on 2 computers i have.. both quads, and it works wonderfull.. i just have one question.. Is there a way to compute the x264 output stats of all parts encoded?? :confused: x264.stats are stored in the "TEMP" folder for each encode.. try looking for .txt's there

Jabroni

20th November 2007, 02:51

support-unit-mode.. OFF:
RTFM ;) ... "x264farm.html" shipped with binary..
support-unit-mode.. ON:
"config.xml"

i assume: worker = agent
"not" "worker" = controller..

try proper tutorial in my signature..

when you'll set up agent-based - youll have both avs(decoded) and x264(encoded) buy each agent.. .. and just a bit by controller..

if you have few agents - then i guess it does not really matter controller- or agen-based method to use..
but if you have "heavy" avs-script.. then agent-based is preferrable.. i think..

Actually what i was trying to point out.. is that I already have a working x264farm setup.. via just command prompts, now the real question was apart the configuration of your GUI implementation, I pointed in the settings to the controller.exe, but when I assign a job to a worker, it justs launched a plain x264.exe thread (no controller.exe that would fire up all the agents to send the work).
I tired to check all the .txt on your release, but it seems like its from a regular meGUI install...

Might be some newbie questions.. im new onto the encoding job, plus i started with something that is relative 'new' or 'unexplored' by most people... but still.. i like it :devil: :D

SpAwN_gUy

21st November 2007, 11:08

I pointed in the settings to the controller.exe, but when I assign a job to a worker, it justs launched a plain x264.exe thread (no controller.exe that would fire up all the agents to send the work).
I tired to check all the .txt on your release, but it seems like its from a regular meGUI install...
video codec needs to be set to "x264farm"..
then create ("config") a new profile,
then from the config screen Choose an x264-profile (top comboBox), then press YES. after that - tune commandline and params as you like (or just keep them as generated)..
and then save a NEW (or update the profile, if you are editing some) profile FOR x264farm...

in BitrateCalculator - you can specify "x264farm" as codec... and apply bitrate values..

i guess this is... like that.. ;) ..

Pitbull_Raven

21st November 2007, 17:32

video codec needs to be set to "x264farm"..
I use megui to recode and downsize files so I can watch them on my xbox360. Files have to be smaller than 4gb or the xbox will not play them. Will your mod allow me to encode to mp4 using my custom profiles?

Jabroni

22nd November 2007, 08:25

video codec needs to be set to "x264farm"..
then create ("config") a new profile,
then from the config screen Choose an x264-profile (top comboBox), then press YES. after that - tune commandline and params as you like (or just keep them as generated)..
and then save a NEW (or update the profile, if you are editing some) profile FOR x264farm...

in BitrateCalculator - you can specify "x264farm" as codec... and apply bitrate values..

i guess this is... like that.. ;) ..

Ok after some playing around I can now give a lil' feedback on my experience. I like how it was integrated on a GUI, but, it would be nice if it would give you the option to spawn and show the dialog of the controller.exe process (since its a bit more informative) So far ive done 2 encodes, im doing my third one which should take a bit more.... still overcall its really good... but what it would be a bit plus, is that it could give a processed stats of the encoding, so far ive come with this utility http://forum.doom9.org/showthread.php?t=118559 , i use that to parse a txt it creates on the controller temp folder, with all the stats output of the agents to generate some nice looking stats of the encode (QP and such).

Pitbull_Raven

22nd November 2007, 12:20

I think this needs a better How To. The one I read is a bit outdated, doesnt even mention a GUI. Also, I haven't been able to get it to run on 2 machines. Could someone please take the time to explain a step by step?

I downloaded the gui version from rapidshare. What else is needed? I have all the files installed on my Z: drive. This drive is a share on my server and accessable to both the controler and the second PC I am going to use for tests. Both pc's are using that share as Z. My sample file is default.avs which is pointing to default.avi. I can play the avs file on both the controller and agent pc. After that I'm pretty lost as to what to do to get it all up and running.

Jabroni

22nd November 2007, 22:06

Uhm.. i have this encode.. for some reason controller.exe wont split more parts when after like 80% on the first pass, when they are other agents available.... (it even just send 1 thread on the current agent, instead of the 3 i have configured)

Heres an output of my controller console:

FIRST PASS
92% done (344335 / 371705) at 6.15 FPS
Last updated: 15:33:47
########################################################################||____
Agent | FPS | Description
Local | 3.89 | Doing [344335-349115]
SageTV | 5.65 | Waiting for job
Local | 1.50 | Waiting for job

tre31

23rd November 2007, 04:20

Ok I've had agent-based encoding going quite well between windows only machines, however I need too add one linux machine too the configuration, but how do you maintain the same configuration (linux doesn't use d:\x264farm .. there is no such thing as d:\ in linux).

So far I've managed too get the agent compiled and going on x86-32 linux, x264 the same, its all good, but I don't know how too structure the .avs and shares too include the linux machine.

Currently I have z:\ shared over the network (root of drive k:\), that has a directory x264farm with all thats required in there, I put the .avs in that directory, so the question is now, too get get the .avs too work for linux, previously I used z:\ in both the .d2v and .avs too get it working as agent's, but how do you get linux too function as an agent using the same .avs?

Is this even possible?

-------
edit:
Just realised that without using the beta avisynth for linux it wouldn't work anyway, and even then i'd still have that same problem with dgmpegdec and non-local .m2v file... looks like my cunning plan was somewhat shortsighted. Onto the backup plan, winpe/bartpe/reatogo usbkey and figuring out howto get avisynth working on that (yes its an extra layer too the problem, but its a bare pc, no cd, no hdd, but I have a usbkey, slowly bits & pieces have been dying but the core of it works .. lol).

omion

23rd November 2007, 09:34

Uhm.. i have this encode.. for some reason controller.exe wont split more parts when after like 80% on the first pass, when they are other agents available.... (it even just send 1 thread on the current agent, instead of the 3 i have configured)

Yup. x264farm does that when it sees that the current splits are "risky". If the controller does not know if a split ends on an I frame, it will not start an agent on the job directly after that. The reasoning behind this is that if the I frame is calculated incorrectly the next split will have to be done over again, which may make it take longer.

Imagine a video with 10000 frames where no frame is complex enough to trigger an I frame. In this video, all I frames will be a multiple of 250 (the default max I frame interval) If x264farm decides to start encoding at, say, frame 4999 then the entire second half of the video will have its I frames off by 1 from what they should be. When the encoder which started at the beginning finally gets to frame 4999, it will see that the frame should not be an I frame, and will then throw out all the encoding done in the second half.

This example may seem a bit strange, but there is one area where it makes a big difference: credits. Credits are usually encoded at the maximum I frame interval, so they are encoded serially to avoid re-encoding. Older versions of x264farm were quite a bit slower at the end since every frame in the credits were encoded multiple times. (and I noticed your controller output is waiting on the end of the encode, exactly where the credits probably are)

That being said, if you want to change the behavior, you may adjust the --rethresh parameter of the controller. "1" means that it will not do any of this, larger numbers will tell it to be more "careful".

@tre31:
Don't feel too bad: I haven't gotten agent-based encoding to work on my Linux computers either. :p Does controller-based work for you at least?

About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.

audyovydeo

23rd November 2007, 10:20

About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.

Hello,

I started playing with x/farm just yesterday (1.15).
Everything on the controller so far, I'll try with 2nd PC this weekend.
I'm pretty sure my config files are fine.
I think I got a path problem, because when i run the controller, it says :

x264farm controller version 1.15-182
Using config file ".\config.xml"
CONFIG:
temp_dir: "C:\\bin\\x264farm115\\controller\\temp"
agent list:
Local = 127.0.0.1:50001 (1,1)

FIRST PASS
0% done (0 / 1000) at 0.00 FPS
Last updated: 0:02
------------------------------------------------------------------------------
Agent | FPS | Description
Recent errors:
~
~
~
~

ie : nothing happens, it just waits.
The agent runs correctly.

my dir structure is
\bin \x264farm \agent \temp
\bin \x264farm \controller \temp
and it makes no difference whether the .avs and .avi are in \controller or \controller\temp.
Paths in the avs are absolute. The avs is as simple as it can be :

DirectShowSource("C:\bin\x264farm115\controller\cube.avi")
converttoyv12()

note : I am using svn x264 700, not cefs.
is cef's version mandatory ?

thanks
audyovydeo

omion

23rd November 2007, 10:40

Hello,

I started playing with x/farm just yesterday (1.15).
Everything on the controller so far, I'll try with 2nd PC this weekend.
I'm pretty sure my config files are fine.
I think I got a path problem, because when i run the controller, it says :

...

note : I am using svn x264 700, not cefs.
is cef's version mandatory ?

thanks
audyovydeo

Cef's version is not mandatory, but in order for controller-based encoding to work with Windows x264farm must be compiled with MSVS, not GCC.

The problem is actually something that SpAwN_gUy pointed out before. In version 1.15 you need to set the adhoc element in the controller's config.xml file, even if you don't plan on using that feature. It's a stupid bug that made its way into the program because I got cold feet from broadcasting stuff by default.

So, add the following line to the controller's config file:
<adhoc controller="40700" agent="40701"/>

Then, if you don't want to use ad-hoc agent discovery (look back a few pages for more info) use the --noadhoc argument to the controller.

audyovydeo

23rd November 2007, 11:01

So, add the following line to the controller's config file:
<adhoc controller="40700" agent="40701"/>

Thanks : now controller sends agent the job.
I have bumped x264 to cef's 699 version.
Agent gets the job, but throws exceptions :

"s not recognized as an internal or external command,
operable program or batch file.
436 exited 1
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")
"
closing socket
socket closed
Got a connection from 127.0.0.1:2143"

I'll go back and reread the thread for insights.

cheers
audyovydeo

omion

23rd November 2007, 11:11

"s not recognized as an internal or external command,
operable program or batch file.
436 exited 1
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")
"
closing socket
socket closed
Got a connection from 127.0.0.1:2143"

That's usually from either using the config option <nice> without nice.exe available, or not using the right x264 executable.

SpAwN_gUy

23rd November 2007, 14:47

New people... nice :) ...

About "new" Progress Window for x264farm - i thought about that.. and, i guess, this will not be the problem.. only... some time..

'kay.. and i guess. the new tutorial and config.xml generator would be also a nice thing to do....

audyovydeo

23rd November 2007, 15:04

That's usually from either using the config option <nice> without nice.exe available, or not using the right x264 executable.

mmmh, I'd tried with and without nice even before posting here.

Anyhow, with your <adhoc> suggestion agent & controller speak.
Problem remains :

Got a connection from 127.0.0.1:3920
working!
920 first pass frames 0 to 999
Testing file "C:\bin\x264farm115\controller\temp\z_input.avs": FOUND!
920 agent-based encoding
920 doing the following:
"x264_667_omion" --crf 25 -r 1 -b 3 -A none -m 1 --me hex --qcomp 0.6 --cplxbl
ur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3
--fps 5000000/166833 --pass 1 --stats "C:\bin\x264farm115\agent-win32\temp\stats
e9903a.txt" --seek 0 --frames 1000 -o NUL "C:\bin\x264farm115\controller\temp\
z_input.avs"
Thread 659 killed on uncaught exception Unix.Unix_error(40, "send", "")
'x264_667_omion" --crf 25 -r 1 -b 3 -A none -m 1 --me hex --qcomp 0.6 --cplxblur
20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --
fps 5000000/166833 --pass 1 --stats "C:\bin\x264farm115\agent-win32\temp\stats'
is not recognized as an internal or external command,
operable program or batch file.
920 exited 1
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")
"
Thread 661 killed on uncaught exception Unix.Unix_error(40, "send", "")
closing socket
socket closed
Got a connection from 127.0.0.1:3921
... and so ad infinitum.

In bold are the different errors I get for each request cycle. There are obiously many things at work here.

Tried with :
snv 700
cef's 699
dark shikari's 697
omion's 667

so it's probably not the compiler issue here.
Again this is with agent & controller running on same machine.
I'll try w/different this weekend, but any input is welcome.

thanks
audyovydeo

audyovydeo

23rd November 2007, 15:08

New people... nice :) ...

Well it always boils down to SMP vs grid computing doesnt it

Besides, having just bought a C2D I'm nowhere near buying a dual quad-core ;-)

cheers
audyovydeo

Jabroni

23rd November 2007, 18:09

Ok now im stuck with this encode on an endless loop
Controller:

SECOND PASS
99% done (370955 / 371705) at 6.81 FPS
Last updated: 1d 11:39:16
ETA: 1:50
Agent | FPS | Description
Local | 3.40 | Doing [235385-235634],[254484-254733]
SageTV | 6.11 | Disconnected

Agent

nice -n 10 "x264" --ref 3 --bframes 16 --b-pyramid --weightb --direct auto --f
ilter -2,-1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me um
h --merange 12 --threads auto --thread-input --cqmfile C:\Encodes\prestige.cfg -
-progress --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10
--qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 24 --fps 30000/1001
--pass 3 --stats "c:\temp\agent\stats 8d3520.txt" --seek 234635 --frames 250 -
o "c:\temp\agent\output 9e8809.mkv" "d:\encodes\encode1080.avs"
avis [info]: 1280x720 @ 29.97 fps (371705 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSSE3
x264 [error]: requested bitrate is too low. estimated minimum is 62 kbps
x264 [error]: x264_encoder_open failed
Thread received exception "Sys_error(\"nice -n 10 \\\"x264\\\" --ref 3 --bframes
16 --b-pyramid --weightb --direct auto --filter -2,-1 --subme 6 --trellis 1 --a
nalyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 12 --threads auto --threa
d-input --cqmfile C:\\\\Encodes\\\\prestige.cfg --progress --no-psnr --no-ssim
--qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --iprati
o 1.4 --pbratio 1.3 --bitrate 24 --fps 30000/1001 --pass 3 --stats \\\"c:\\\\tem
p\\\\agent\\\\stats 8d3520.txt\\\" --seek 234635 --frames 250 -o \\\"c:\\\\temp
\\\\agent\\\\output 9e8809.mkv\\\" \\\"d:\\\\encodes\\\\encode1080.avs\\\": No e
rror\")"
closing socket
socket closed

it seems that the x264.exe isnt letting use a low bitrate... and the controller doesnt catch this and increase the bitrate (not quite sure if the controller has to do that).

omion

23rd November 2007, 21:41

mmmh, I'd tried with and without nice even before posting here.

Anyhow, with your <adhoc> suggestion agent & controller speak.
Problem remains :
...
... and so ad infinitum.

In bold are the different errors I get for each request cycle. There are obiously many things at work here.

Well, those all look like the same error. One of the issues I'm cleaning up with the agent is that errors will sort of "bubble up". First cmd.exe reports that it can't find something, then the agent reports that x264 failed, then it reports that, due to the failure, the entire thread died.

Tried with :
snv 700
cef's 699
dark shikari's 697
omion's 667

so it's probably not the compiler issue here.
Again this is with agent & controller running on same machine.
I'll try w/different this weekend, but any input is welcome.

thanks
audyovydeo

That issue looks like a quoting problem. I had an issue with it in the controller. The problem is that the command prompt will remove the first and last quotes from a line it's running, which looks like what it's doing here. The problem is that I have no idea why that's happening for you and not me. I'll look into it (although I think I may have already fixed it for the next version)

In the meantime, I think this will work to stop the problem: try using the <nice> parameter in the agent's config file. Get the nice.exe (http://omion.dyndns.org/x264farm/nice-20070415.rar) program, and use the following line in the agent's config.xml file:
<nice>1</nice>

That will set the encodes to slightly below normal priority, but it should be just enough to throw off the console's stupid quote-removal "feature".

@Jabroni:
Yes, that is due to the controller using a bitrate which is too low. I have it automatically increase the bitrate when it thinks this is the problem, but it will only ever ask for 3 times what it thinks is the optimal. If this value is less than the actual lowest bitrate, then it will stall like that.

If you haven't done any more encodes since then (or if your latest encode has the same problem) could you send me the controller's out-dump.txt file? I'll PM you my e-mail address.

tre31

26th November 2007, 04:47

I'm not entirely sure whether this is bug, but I have seen it replicated multiple times using 1.13, I have only done one encode with 1.15 and I let it go untouched (because I was getting over having failed encodes .. heh).

Basically what happens is I get an error (don't have logs sorry), that states that the 2-pass bitrate curve is wrong (pretty sure it was related too Quality calculations being off), however that is not the actual cause of the error, the error is caused (and you could probably replicate this yourself using 1.13) when agents fail during the second pass (multiple reasons, crashes, hardware fail, etc), and loose connection too the controller, or if the encode stops during the 2nd pass for some reason.

This happened multiple times too me, so it is definately there in 1.13, I'm not sure about 1.15 (one of the reasons I updated actually).

adhoc agent discovery works quite well, now my controller config is just these lines (much easier)...
<config>
<temp>z:\x264farm</temp>
<adhoc controller="40700" agent="40701"/>

</config>

It's also probably a smarter way too do things as each agent should know its own config, rather than relying on the controller too always know the capabilities of the agent (ie. ip's can change in a dynamic dns setting - I ended up setting static dhcp entries too make sure each machine really was the machine I expected it too be).

@omion:
'About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.'

Relative paths could just work you know for all agents if the path is the same, ie. on linux create directory in root dir 'shared' or something (and use smbmount \\machine\shareddrivename /shared -o workgroup=WORKGROUP,username=user,password=pass), and in root dir of drive on windows shared drive create same directory 'shared', so .. what is created is the same.
sample .avs (so you get my drift):
DGDecode_mpeg2source("\shared\x264farm\clip.d2v")

corresponding .d2v would obviously have too be similar.

It could work - if x264farm on linux converts the slashes in the .avs too the correct way for linux, otherwise it won't (but that wouldn't be too hard too implement I don't think - food for thought).
---------
please note: the above is just a theory, and may have no basis in reality. I think its probably a bit too soon anyway, wait for avisynth linux too actually be out of beta (at least one version) maybe, who says there is even a DGDecode function in avisynth linux, too many variables.

audyovydeo

26th November 2007, 10:46

In the meantime, I think this will work to stop the problem: try using the <nice> parameter in the agent's config file. Get the nice.exe (http://omion.dyndns.org/x264farm/nice-20070415.rar) program, and use the following line in the agent's config.xml file:
<nice>1</nice>

That will set the encodes to slightly below normal priority, but it should be just enough to throw off the console's stupid quote-removal "feature".

Tested this weekend, this does work around the issue.

Now that I got it working, can I some feature requests :
- adding support for mp4 (the standard container)
- supporting 1pass encoding

reason is, I think 2pass encoding is bound to disappear over time. Personally I haven't done a single 2-pass encode since I got a good grip on crf.

cheers
audyovydeo

omion

27th November 2007, 01:13

I'm not entirely sure whether this is bug, but I have seen it replicated multiple times using 1.13, I have only done one encode with 1.15 and I let it go untouched (because I was getting over having failed encodes .. heh).

Basically what happens is I get an error (don't have logs sorry), that states that the 2-pass bitrate curve is wrong (pretty sure it was related too Quality calculations being off), however that is not the actual cause of the error, the error is caused (and you could probably replicate this yourself using 1.13) when agents fail during the second pass (multiple reasons, crashes, hardware fail, etc), and loose connection too the controller, or if the encode stops during the 2nd pass for some reason.

This happened multiple times too me, so it is definately there in 1.13, I'm not sure about 1.15 (one of the reasons I updated actually).

I did change a few things which affected the stability of the program after 1.13, especially when the remote agent dies and hangs the whole encode. So there's a very good chance that it has been fixed, but without logs I wouldn't know for sure. If it happens again, send me the logs for both the controller and any agents.

adhoc agent discovery works quite well, now my controller config is just these lines (much easier)...
<config>
<temp>z:\x264farm</temp>
<adhoc controller="40700" agent="40701"/>

</config>

It's also probably a smarter way too do things as each agent should know its own config, rather than relying on the controller too always know the capabilities of the agent (ie. ip's can change in a dynamic dns setting - I ended up setting static dhcp entries too make sure each machine really was the machine I expected it too be).

I thought it was a strange (and cumbersome) way to set up the agents too, which is why I added the ad-hoc mode. Glad to see it works! :)

@omion:
'About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.'

Relative paths could just work you know for all agents if the path is the same, ie. on linux create directory in root dir 'shared' or something (and use smbmount \\machine\shareddrivename /shared -o workgroup=WORKGROUP,username=user,password=pass), and in root dir of drive on windows shared drive create same directory 'shared', so .. what is created is the same.
sample .avs (so you get my drift):
DGDecode_mpeg2source("\shared\x264farm\clip.d2v")

corresponding .d2v would obviously have too be similar.

It could work - if x264farm on linux converts the slashes in the .avs too the correct way for linux, otherwise it won't (but that wouldn't be too hard too implement I don't think - food for thought).
---------
please note: the above is just a theory, and may have no basis in reality. I think its probably a bit too soon anyway, wait for avisynth linux too actually be out of beta (at least one version) maybe, who says there is even a DGDecode function in avisynth linux, too many variables.

That should work fine. One thing to note: neither x264farm nor x264 actually parse the avs file. It's all done through AVIsynth. As long as AVIsynth for Linux converts the paths correctly, this should work.

The problem is with Windows AVIsynth running through Wine to a Linux build of x264farm. I haven't gotten a good way to do this properly, but when the Linux AVIsynth is stabilized it should be much easier.

omion

27th November 2007, 08:53

Now that I got it working, can I some feature requests :
- adding support for mp4 (the standard container)
- supporting 1pass encoding

reason is, I think 2pass encoding is bound to disappear over time. Personally I haven't done a single 2-pass encode since I got a good grip on crf.

cheers
audyovydeo1 pass encoding has been on my "next version" list for a very long time now. I'm still planning to put it in, but it will need a lot of work, so I'll probably put it off some more.

About .mp4:

There are 2 reasons I support mkv:
1. It's the container I use
2. I already wrote an mkv porser

The reason I don't yet support mp4:
Writing parsers is hard, and I don't want te do it again ;)

I have to actually recreate every temp file since x264 will add a bunch of encoding info to each encode. This is normally a good thing, but it adds a lot of space if it is included in every scene. So I have to take apart each file, strip out the encoder info, and put it back together. That's hard to code for, so I only support one container.
It should be a fairly straightforward conversion from mkv to mp4, and I'm sure that somebody's written one. If not, I may make a standalone app that x264farm might use... maybe. I'm kind of burnt out on containers now, though, and I still have a lot of things to throw into the next version of x264farm, so again, maybe sometime but not now.

audyovydeo

27th November 2007, 09:45

I'm kind of burnt out on containers now, though, and I still have a lot of things to throw into the next version of x264farm, so again, maybe sometime but not now.

I understand burnout perfectly, no pbm.
Also I agree 1pass is higher piority than mp4.

While your tool is not yet industrial strength, it's on the right track. I dont know of any other distributed encoder for h264.
This may well become the poor man's solution to heavy-duty encoding, vs professional hardware encoders. So any minute you spend on it, in my view, is well spent.

cheers
audyovydeo

SpAwN_gUy

27th November 2007, 10:27

'kay....
*offtop("dot" is working extremely bad on my keyboard this pisses me of.. sorry)*
'kay.
i have a some kind of feature request.. this one is more fr then a BUG...
it would be nice, when:
i'm doing agent-based mostly... so if i set something NOT right.. and agent sees the .avs.. but can not open it (well. none of them can.. this is due to "not relative paths", "not inserted plugin-strings" and so on..)..
i'm getting "End_of_file" from all of the agents...

is it possible to swithch from agent-based to controller-based encoding?
so when i'm at home... an have 2 encodes in the list.. so the first one went fine, but the second - stalled in infinite loop.. (i dont use controllerPC as agent)

and for.. mp4.. hm.. i'll try to search for muxer... i guess i've seen it somewhere, with .mkv input, append-file.. and so on...

btw... for final MKV muxing yopu could try to use libmatroska muxer.. it has append feature... (mmg - has..)

i'll be back ;)

SpAwN_gUy

27th November 2007, 11:52

so..
as i've read in iNet....

there is NO direct way to reMUX mkv to mp4 :( ...
at first one have to extract RAW h264 stream and then with mp4box mux it to mp4..

but.. i dunno about the possibility to stream raw h264stream directrly to mp4box input.. (if this will work.. then libmatroska -> mp4box input in the end - and you've got mp4-file ;) )...
ok then...
gpac lib is the one that could help with integrating mp4... i guess searching throug the code will give the idea to connect mkv-raw output with mp4-input...

basically so..

hehehe. i just thought... if omion will be able to produce raw AVC stream then, i guess.. we'll have all three "major" output formats available :) (mkv, mp4, raw)

Pitbull_Raven

27th November 2007, 13:20

So here is the thing about all this... It's too much technical speak for the average Joe. I've been trying to make something out of it for some time now but still can't get it to work. Is there an updated HowTo or something? Could someone please take the time to write a short summary of what needs be done to get this to work on 2 or more Win machines? Please guys. I am sure there are more and more people out there giving up on this beautiful piece of software just because they can't understand how to get it to work. A short "step-by-step" would really be nice.

foxyshadis

28th November 2007, 03:10

spawn, use gdsmux from haali.

SpAwN_gUy

29th November 2007, 15:45

spawn, use gdsmux from haali.this one is DirectShow muxer... so, no cross-platform support i guess..
it simply uses DirectShow codecs/parsers/dumpers to demux input and to produce final file..

St Devious

17th December 2007, 19:44

OK , I', a newbie at this . But i want to do this as i have 4-5 Computer in my house on my network and i have quite a lot of HDV captured footage to encode , so if i can use this the process will be faster .

Could someone help me out ?

Another question , is there any kind of gui for this , or is this completely command-line ?

Is there any chance of this getting integrated into megui in future or a different program ?

SpAwN_gUy

18th December 2007, 09:19

OK , I', a newbie at this . But i want to do this as i have 4-5 Computer in my house on my network and i have quite a lot of HDV captured footage to encode , so if i can use this the process will be faster . it seems like it IS faster.. not so "stable" but faster..
and Welcome... :)

Could someone help me out ?Sure, why not.. Share your questions with us ;) ...

Another question , is there any kind of gui for this , or is this completely command-line ?
Is there any chance of this getting integrated into megui in future or a different program ?
try... hm... this one ;) (http://forum.doom9.org/showthread.php?p=1063839#post1063839)
work is still in progress..
but at first you need to configure controller and agents.. and then look through one of a bit old tutorials..

St Devious

18th December 2007, 18:00

oh man this is just too complicated for me , dont know wat to do with modded megui .

waiting for someone tocome up with a gui for this .

Sharktooth

18th December 2007, 18:10

:readguid:

SpAwN_gUy

19th December 2007, 09:06

oh man this is just too complicated for me , dont know wat to do with modded megui well.. you could at least try to read the topic.. a few pages back.. and few posts after the link i gave you...
if you would... then you would read this (http://forum.doom9.org/showthread.php?p=1067736#post1067736)

waiting for someone tocome up with a gui for this . write your own... this is a free world... i guess..

audyovydeo

19th December 2007, 10:19

oh man this is just too complicated for me , dont know wat to do with modded megui .

waiting for someone tocome up with a gui for this .

And while we are in a philosophical mood, GUI is not a synonym of EASY, generally speaking.

I managed to set up x264farm cmdline in, let's say, 2 hours and 3 posts+replies. Which isn't much, for an alpha-level application with alpha-level docs.

Just get it up & running on your main machine, then stem off to the others. Reread through the help doc and this thread, and you'll get it.

cheers
audyovydeo

St Devious

20th December 2007, 01:54

ok i'm giving this one more try . I selected x264 farm codec in megui and i see these options which r different from x264 .

I selected HQ-slowest profile , and left all other parameters at default . So wat do i do now , copy the collective command line parameters which r written in box near bottom or can i encode to x264farm using megui .

If yes , how do i set up the agents ? run megui too on them ?

Another question : I want to have only one agent . So in the config.xml file of the controller , do i have 2 agents ( 1 my pc and 2nd another pc ) or only the agent's ip address and other info .

Some more questions : Where do i put nice.exe and avs2yuv.exe ?

I tried running the controller and agent . agent is running fine , but controller exits . here r the config.xml files for both .

CONTROLLER :

<config>
<temp>C:\x264farm\controller\temp</temp> 

<agents>

<agent name="Matrix"> 
<ip>192.168.111.147</ip> 
<port from="40700" to="40703"/> 
<number>2</number> 
</agent>

<agent name="Agent Smith">
<ip>192.168.111.66</ip>
<port from="40700" to="40703"/>
<number pad="1">1</number>

</agent>

</agents>
<adhoc controller="12345" agent="23456"/>
</config>

AGENT :

<config>

<temp>C:\x264farm\agent\temp</temp>


<port from="40700" to="40703"/>



<name>Local</name>
<number pad="0">2</number>


<x264>x264</x264>


<nice>20</nice>


<base></base>


<agentpipe>0</agentpipe>


<compression>
<type>0</type>
<type>1</type>
</compression>




<adhoc controller="12345" agent="23456"/>
</config>

And the error i get is :

http://i16.tinypic.com/72ktsvs.jpg

audyovydeo

20th December 2007, 10:42

Again, I haven't tried the megui option. But what is your controller command-line ?

to resume :

0. read through the doc and this thread once again.

1. deploy the whole cast of characters (directories, paths, all executables, source avi, avs scripts, etc). Nice and avi2yuv should, as always with any windows program, somewhere in your path. just like the correct version of x264.exe

2. edit the xml files accordingly (temp paths are important and finicky to configure)

3. launch the agent(s)

4. run the controller
here is my cmdline :

controller.exe -B 500kbps --avs temp\input.avs -o test.mkv -1 "--crf 25 -r 1 -b 3 -A none -m 1 --me hex" -2 "-r 1 -b 3 -A none -m 1 --me hex"

and ... (how do they say ?) ... et voila' !

cheers
audyovydeo

SpAwN_gUy

20th December 2007, 10:50

I selected HQ-slowest profile , and left all other parameters at default . So wat do i do now , copy the collective command line parameters which r written in box near bottom or can i encode to x264farm using megui . and why do you think (and asked about) this Mod is made for?
and why x264farm is "Codec" in meGUI ? ...

If yes , how do i set up the agents ? run megui too on them ? NO. you don't need to setUp meGUI on them..
just run agent.exe on agentPC and let it run..

Another question : I want to have only one agent . So in the config.xml file of the controller , do i have 2 agents ( 1 my pc and 2nd another pc ) or only the agent's ip address and other info . well.. since we have adhoc feature (and it IS working for me at least).. you dont even need to specify agents in controller's config.. but you can leave it like that...

so.. the only mashines that will be Encoding are the PCs where agent.exe is started.. and RUNNING

Some more questions : Where do i put nice.exe and avs2yuv.exe ? agent's and controller's folder

I tried running the controller and agent . agent is running fine , but controller exits . here r the config.xml files for both. nice picture you have here.. :) ..
this picture means that you haven't read anything.. i'm sorry.. but "WHY???"
READ the manual.. on how to use x264farm...
controller-based encoding is the simpliest thing to set up.. :readfaq:

SpAwN_gUy

20th December 2007, 11:37

But what is your controller command-line ?according to the picture he showed.. it's "nothing" :) ..

BWAHAHAHA.... the previous post was my 200th ;) ... nice using :)

St Devious

20th December 2007, 15:50

@ Spawn_guy : u r not exactly helping me here .

I gave u my agent and controller config , tell me wats wrong with that if u really want to help rather than make smart remarks . I need help , thats y i posted here.

So i select megui mod on the controller pc , and run agent.exe on each of the agent pc's and start encoding in megui ?

Look at the picture and my agent , controller config and tell y the controller cant find agent pc .

The ip's of boh the pc's r the ip's on lan , is that correct ?

Sharktooth

20th December 2007, 16:10

im sorry to say this, but your problems (as well as in the other threads) are all related to your ignorance.
those kind of things (video encoding) are not simple and you need to read and understand how things work.
do as spawn guy said: READ.
If you find it too much complicated, then start with the basic things.

audyovydeo

20th December 2007, 16:12

Hard to help someone who's not putting effort into the task. We've practically spoon-fed you all the info.

Specifically :

1. in your agent XML file, the fact that you got < nice> value of 20 only shows that you have not read this post :

http://forum.doom9.org/showthread.php?p=1068890#post1068890

2. in your controller xml there is no trace of the ADHOC directive, which means you havent read this post :

http://forum.doom9.org/showthread.php?p=1068730#post1068730

3. your controller command-line is non-existent, which means you haven't read the documentation.

Hard to figure what else we can do, short of drilling for oil.

cheers
audyovydeo

SpAwN_gUy

20th December 2007, 17:26

2. in your controller xml there is no trace of the ADHOC directive, which means you havent read this post : well.. i guess.. he has "adhoc" enabled.. in both configs..

'kay.. my give up..
-1) setUP configs.. i guess everything seems to be fine
0) run agent.exe on agentPC's
1) configure in meGUI path to the controller in options..
-----------------READTHEGUIDEONmeGUIUSAGE--------
2) create an AVS and Choose it as an input
3) choose x264farm as Codec
4) configure params as in the post i gave you
5) press OK.
6) push Enque
7) press start on the second Tab in meGUI..
wait

i guess i'll need some money for such kind of things :) ..
do we have to think for you?

i guess there will be no posts from me in 4 days.. i'm going to Geneva... for 2 weeks.. anyone want's to meet me there? :)

UPD: ah.. BTW.. there is one more tool for x264farm ... i wonder what it's name.. i dunno is it upDated or not.. you can search for it in here.. i tried it a long time ago.. so search fo x264farm support

Sharktooth

20th December 2007, 17:34

Spawn_guy, Geneva? No... for now, my best would be Genova (Italy).

SpAwN_gUy

21st December 2007, 09:27

Spawn_guy, Geneva? No... for now, my best would be Genova (Italy).
hm.. it's quite far from Geneva (Switzerland) .. :) .. i think.. but who knows :) ...

antiOFFTOP: my next steps on x264farm.. are:
1) organize storing normal x264settings in meGUI for x264farm (now they are only "strings")
2) complete my agentGUI (Delphi).. (i'm waiting for a new updated version of x264farm.. as omion said.. he's trying to release it soon)
2b) write an "aget as service starter".. not all of my people are satisfied with using it as a gui... (they are working in Terminals)
3) write and agentGUI/Service installer.. so everyone could change sources and compile the installer.. and then to share it with others.. who wishes to participate in encoding..

agentGUI is allmost complete.. i'm reWriting it now.. and most of my people use an old one..

but.. as tomorrow is my last workDay this year.. i suppose.. neXt year..

ChrisBensch

3rd January 2008, 06:29

SpAwN_gUy

6th January 2008, 21:17

Is it possible, since we don't really have single-pass yet, to have a REALLY fast first pass, then use our second pass as the "real" pass? Maybe just dumping the first pass data to /dev/null? I use the crf preset of MeGUI in my cli so I'm not sure if a first pass even benefits me at all. Right now it's just running the same single pass, just twice. I wish there were a better way.
Yep.. i'm allmost back from... where i was :) ...

omion is currently working on "single-pass" encoding... (as i've heard.. well.. that was last year... so, maybe, soon)
and "yes" for now.. using CRF(for x264) and PERCENTAGE(for x264farm) will do Two passes... but during the first one it will only write .stats..

ChrisBensch

11th January 2008, 03:56

omion

11th January 2008, 10:14

Hi guys! I'm back!

Is it possible, since we don't really have single-pass yet, to have a REALLY fast first pass, then use our second pass as the "real" pass? Maybe just dumping the first pass data to /dev/null? I use the crf preset of MeGUI in my cli so I'm not sure if a first pass even benefits me at all. Right now it's just running the same single pass, just twice. I wish there were a better way.

I don't really know what you are saying here. The first pass video data is actually sent to /dev/null (check the code!); it only keeps the stats file. The stats file is basically what sets the second pass in the right direction. I could make a one-pass option (and am still thinking about it, as spawn_guy says) but it would require an entire rewrite of the encoding loop.

This might be a dumb question, but is there a sort of "rule of thumb" for percentages and how they correlate to a CRF value? I usually encode with a CRF of 18...is there a percentage that would match that?
Yup. 100% gets you the same file size that the first pass would have gotten, had you just done a one-pass encode. So if you use --crf 18 on the first pass, 100% will get you around crf 18 on the second pass (but hopefully with higher quality, which is the whole point of a two-pass encode).

Now, on an unrelated note:
<rant>
Networking is HORRIBLE.
I've spent the larger part of two months working on the stupid networking code. I've made about 4 different implementations of it, and each one had some sort of horrible problem with it that would make it completely unacceptable. My current code actually crashes the program. Fairly reliably, too. OCaml code is not supposed to crash. Exit with an error, maybe, but not crash. I don't know how I managed it. It was probably due to some workaround that I added a month ago because Windows can't seem to read and write to the same socket at the same time. So now I have to think of another workaround. The main problem is probably just Windows, which really bugs me because I know those problems aren't going to be fixed. Should I tell the Windows devs to implement timeouts on send/receive functions? The best that can happen is they add it to "Windows 7" when it comes out in 5 years.

I've been working on this code for quite a while now. I think I can do it with two uni-directional sockets between the agent and the controller. But they can't be totally uni-directional, or else the TCP "delayed ack" algorithm will kick in and bring everything down to 10kbps. WHY is that stupid "congestion control" enabled on so many computers? Is an extra 60 bytes really going to take down the network?

I think the code is getting to be too much for me to handle. In order to do all of these workarounds I have to make a whole bunch of threads to shuffle around the data, and the networking makes it break sometimes. I'll get errors like "Thread 37 failed with ENOTSOCK" Not a socket? A half second ago it was a socket, and it was working just fine. And what on earth is thread 37? Is that the thread that is sending data? Or the thread that's receiving data? Or maybe it's the thread that sends dummy data on the loopback socket that I had to set up in order to get Windows unstuck from the select statement that works just fine on every other operating system. So I track it down, and it turns out it's on the thread that sends a ping signal to the controller as long as the 3 threads that communicate with x264 are still working... so I have to workaround that too.

All this extra junk to try and get computers to work together. It's really aggrivating.
</rant>

So, how was your Christmas? :rolleyes:

SpAwN_gUy

11th January 2008, 10:22

Alright, so percentage is the way for farm. This might be a dumb question, but is there a sort of "rule of thumb" for percentages and how they correlate to a CRF value? I usually encode with a CRF of 18...is there a percentage that would match that?
the History Chanel:
at first.. first-pass settings for x264farm needed NO "bitrate" or "crf" options.. 'cause in older x264 builds the default option, when no parameter was specified was "--crf 26" .. or 25..
but after.. the default setting was removed.. and thats why i had to add it manually to comLine Generation in meGUI-Mod... but i never tried such low bitrates so.. my value is "--crf 18" for the first pass..
so.. basically, when you specify in first-pass settings "--crf 18" and for the x264farm bitrate "100" (or "100%") then you will have an TwoPass Encoding with bitrate of CRF 18 ...

on how "percentage" of bitrate is assumed you can read somewhere here or i guess it was somewhere in the doc's, provided with binaries

------------
UPD: glad you're back omion :) ...
So, how was your Christmas? :rolleyes: very nice indeed.. Geneva seems fine.. Berlin is nice at night.. and Prague is extremely Cheap.. and Beer is beautiful :)

Elvis'es and mine Birthsday went just fine :) .. i got drunk.. again.. aswell as on celebration of NY in my Corporation :)

and Testings of DGAVCDec (neuron2's) with x264farm.. just ended with beautiful results :) ... (yeah, i had to modify internals of ".dga" file aswell :) ...)

30th January 2008, 17:02

Hello guys!
I'm trying to encode tga-sequence (1500frames long), so it has no GOPs by default :)
Which settings should I use to speed-up fist-pass calculations (I mean 1st pass split parameters)?

Poopoo

30th January 2008, 17:56

very nice indeed.. Geneva seems fine.. Berlin is nice at night.. and Prague is extremely Cheap.. and Beer is beautiful :)

You drank beer in Prague ? How long have you been sick from it ?

omion

2nd February 2008, 00:21

Hello guys!
I'm trying to encode tga-sequence (1500frames long), so it has no GOPs by default :)
Which settings should I use to speed-up fist-pass calculations (I mean 1st pass split parameters)?

If it's all smooth motion (no clear GOPs) and fairly long then you can change the --fastavs parameter to something like a small black clip (must be the same # of frames and framerate).

HOWEVER (and this is a big however) if there are no clear GOP boundaries in the entire clip, x264farm will NOT split up the file at all. This means that the first pass will encode a bit slower than if you didn't use x264farm. The problem is that x264 will encode a bunch of GOPs of length 250 (by default) and so any encode that does not start on a multiple of 250 will be thrown out. x264farm decides to "save" computing resources until it can find a reasonable split point.

So in your case there is not much you can do to speed up the first pass. Actually, the default job length is longer than 1500 frames, so x264farm won't even try to split it.

@all:
I suppose this is as good a time as any to give a little update. I've decided to concentrate my time on making a single-pass x264farm, rather than trying to hack my way through the mess that is the current 2-pass code. It will be highly simplified (no wacky split options, config.xml no longer required, smaller executables, etc.) but it should be just as powerful as - and hopefully more efficient than - the current program. (And I'm making a 64-bit version, not that it would make any speed difference at all:p)

I'll probably end up applying the algorithms to a new 2-pass x264farm, but that will be a while away.

ChrisBensch

11th February 2008, 07:16

Any news on the MeGui-mod for x264farm? How about a client config gui? I dabble in Java/C# and I'd be happy to contribute if I can.

SpAwN_gUy

11th February 2008, 12:16

Any news on the MeGui-mod for x264farm? How about a client config gui? I dabble in Java/C# and I'd be happy to contribute if I can.
i'm busy with my work.. i'm hacking linux.. ;) ..
so, no updates for now :( ... contact me via PM... for more instructions :) ...

konzerten_nr9

10th March 2008, 05:55

I am runnig an x264farm consists of a controller and 6 agents, 5 agents running on Windows using agent-based encoding, 1 on Linux using controller-based encoding.

Now I am runnig a test with a simple test.avs as input ike:
directshowsource("test.wmv", audio=false)
which test.wmv has 9440 frames.

My command line is:
controller -B 3500kbps --qcomp 0.75 -1 "--crf 19 --keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --analyse all --threads auto --progress" -2 "--keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --subme 5 --ref 1 --trellis 1 --me hex --merange 16 --threads auto --progress" --avs ..\test.avs -o ..\test.mkv

The problem is, the footage is splitted into several parts during the first pass, but it is not being encoded by multiple agents simultaneously. Instead, the splitted parts are encoded by a single agent sequentially.

This results in the throughput of first pass varying between 2 to 15fps depending on the agent picked up at the beginning of the first pass, while the throughput of the second pass is around 30fps.

Is there any way to have the first pass encoded by multiple agents simultaneously?

Thank you in advance for your kind suggestions.

Jabroni

10th March 2008, 06:00

konzerten_nr9

10th March 2008, 07:12

Hello Jabroni,

If you scroll on this thread I asked the same thing.. the problem is that its not able to do more splitting because it cant find more I-Frames (or whatever type of frame it is used to split).

Actually the first pass is splitted into 2 parts:
Frames 0-5159 Frames 5160-9439
but all the jobs are done on 1 single agent serially.
Thus I thought it is a new situation here...

driftr

13th March 2008, 16:15

So here's my latest problem. Large source files. Like over 10 gig high bitrate source files. I often give up on x264farm because the first pass takes forever. The process to split the file into batches and send the jobs out maxes out at 100% proc usage (or 25% on my quad core system) and goes extremely slow and might get a second job out before the first one finishes encoding. Overall the first pass is so slow just doing command line x264 encoding on the single computer alone is faster. Any thoughts on how to get the process multithreaded or better handled?

SpAwN_gUy

13th March 2008, 17:04

i'm here again...

'kay.. mine things are doing fine.. even with 10Gb 1080p x264 encoded files.. did that a while ago... first pass - multi threaded..

the reason of 100% CPU on the Controller might be on Split-points determination... (i have single core.. when starting - yeah it's a bit annoying, but when working - fine..)

driftr, post you AVS script..

driftr

13th March 2008, 18:48

I wrote that rather slopily... the avs script is a simple directshowsource opening the file with no filtering or framerate change or anything. The problem is source material having sizes like 720x576 or say blu-ray with 1920x1080.

What's happening is avs2yuv can't keep up with the how fast the computers are encoding the jobs. So I suppose I could bump up the batch size but then the initial set of jobs would take really long esp. with say 15+ agents. What I'd really like to see is some kind of multiple controller setup where multiple instances of avs2yuv can create the jobs.

Right now on this simple hour long 720x576 wmv I'm just converting to x264 for fun and there's a point where agents are just idle waiting for jobs because the controller can only push out so many at a time.

SpAwN_gUy

14th March 2008, 11:38

well.. i just don't know what to say...

well.. you can run multiple controllers :) ...

and... try out setting "--batchmult 0" and/or "--rethresh 1.0"
i can't remember which one is "the one".

driftr

14th March 2008, 15:21

Humm I forgot about those settings. I can see them helping marginally but I'll play with them.

I'm not reading anything about how to run multiple controllers... got an example?

SpAwN_gUy

14th March 2008, 16:19

I'm not reading anything about how to run multiple controllers... got an example?
try to run.. 2 (two) cintroller.exe's at the same time ;) ...
like.. prepare two .bat files.. and the run them..

or make 2 Workers in meGUI-x264farm-mod ... and the run them :) ... (the old one(working) could be still found somwhere here.. my version is still in unworking state :( )

omion

17th March 2008, 23:44

Hi guys! Just to let you know, I'm not dead yet... :p

I've finally got my 1-pass x264farm in a usable state. I'll be releasing it in a few days unless some other major bugs appear.

It should be a bit more efficient than the current code, and driftr's problem should not exist (the splitter does not examine the avs file)

@driftr:
I assume that you are seeing one avs2yuv instance using 100% of one core, correct? Or is it controller.exe? If it's avs2yuv then setting "--thresh 1.0" will minimize the usage of the splitter thread at the expense of some extra time taken re-rendering frames around the split point.

Note: --rethresh is a different setting from --thresh. It's a bit complicated, but "--rethresh 1.0" will not do what you want (in fact, it will basically do all of the same calculations, then ignore them :scared: )

@SpAwN_gUy:
Do multiple controllers work? I didn't think they'd play nice with the shared stats file. I'm on my laptop right now (and lunch break is almost over) so I can't check, but did you find a way to get this to work with the current iteration of x264farm?

SpAwN_gUy

18th March 2008, 09:24

I've finally got my 1-pass x264farm in a usable state. I'll be releasing it in a few days unless some other major bugs appear.
so'.. i guess i need to stick to meGUI thread.. 'cause berrinam is now missin' ... :) :( ...

Do multiple controllers work? I didn't think they'd play nice with the shared stats file. I'm on my laptop right now (and lunch break is almost over) so I can't check, but did you find a way to get this to work with the current iteration of x264farm?Never tried that.. but i can :) .. and i will...

i was trying to say, that multiple controllers will work on different jobs (but never tried that either..)

so.. i guess.. more testing will be done :)

omion

19th March 2008, 21:10

Single-pass version is out. (Yay! A new release!)
It has a new thread (http://forum.doom9.org/showthread.php?p=1114551) to keep the bugs separate from the 2-pass version. Have fun!

vwpassion

31st March 2008, 09:24

First of all: Nice job omion!

Now my concern: Is there a way to make x264farm work with current builds of x264 from Sharktooth or older versions from Cef above 655?

Roswellian

1st April 2008, 08:26

Hello, I am newbie here and just went through the whole threads. There are too many stuff and almost totally confused me. Right now I set up the agent-base mode and my problem is that there is no response on my agents after controller started. Here is some codes on my agent side.

<config>
<temp>.\temp</temp>
<port from="40700" to="40703"/>
<name>Local</name>
<number pad="1">1</number>
<x264>x264.exe</x264>
<nice>0</nice>
<base>d:\Sample</base>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

Controller side
<config>
<temp>.\temp</temp> 

<agents>
<agent name="Local"> 
<ip>127.0.0.1</ip> 
<port from="40700" to="40703"/> 
<number>1</number> 
</agent>

</agents>

</config>

As you see there, I tried to encode files locally by connection (One controller + One agent). My sample file is small m2ts file (5000frames) and here is my avs file.

DirectShowSource("1.GRF",video=true,fps=23.9759856527702,audio=false)
#deinterlace
crop( 0, 22, 0, -22)
Spline16Resize(1280,688) # Spline16 (Neutral)
#denoise

My controller command is like this..
controller.exe --first "--level 4.1 --threads 1 --thread-input --bframes 3 --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --no-ssim" --second "--level 4.1 --threads 1 --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --8x8dct --no-fast-pskip --progress --no-psnr --no-ssim" --avs "D:\Sample\default.avs" --output "D:\Sample\farm.mkv" --bitrate 1375kbps --batch 1000 --split 250 --thresh 20.0

after the controller is started..and log shows
.................
2008-04-01~02:15:52.78 0 @ INFINITY
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 1009 @ 13.58
2008-04-01~02:15:52.78 Empty
2008-04-01~02:15:52.78 2720 @ 48.69
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 3716 @ 17.03
2008-04-01~02:15:52.78 Empty
2008-04-01~02:15:52.78 4539 @ 13.07
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 5599 @ INFINITY
2008-04-01~02:15:52.78 0% done (0 / 5599) at 0.00 FPS
2008-04-01~02:15:52.78 Last updated: 0:18
2008-04-01~02:15:52.78 ---------------......................---------------...........---------------
2008-04-01~02:15:52.78 Agent | FPS | Description
2008-04-01~02:15:52.78 Recent errors:
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

The controller sort of hangs there and there is also no any response on the agent window. I checked the agent port bonding by telnet, and it is fine. Since I am a newbie, it is pretty hard for me to nail down this problem. Could someone give me a little help? Will be really appreciated. :):thanks:

omion

1st April 2008, 08:41

Hello, I am newbie here and just went through the whole threads. There are too many stuff and almost totally confused me.
Me too.:eek: I'm trying to get rid of the complicated stuff in my single-pass version, but it may take a while to get into the 2-pass one...

after the controller is started..and log shows
.................
2008-04-01~02:15:52.78 0% done (0 / 5599) at 0.00 FPS
2008-04-01~02:15:52.78 Last updated: 0:18
2008-04-01~02:15:52.78 ---------------......................---------------...........---------------
2008-04-01~02:15:52.78 Agent | FPS | Description
2008-04-01~02:15:52.78 Recent errors:
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~

It looks like the controller never gets the agent's info from the config file. It should say something about the agent right after the "Agent | FPS | Description" line. Most likely it's not using the config file you made. What is the location/name of the controller's config file?

Try adding "--config config-file-name.xml" to the controller's command line (replacing config-file-name.xml with the actual name, of course ;)) If that doesn't work, then I'll have to think of something else...

Roswellian

1st April 2008, 08:59

Omion, thanks for the quick reply. Actually I did not change config file name and config.xml is located in the same folder with agent/controller, but i still make a quick try as you mentioned. It is still not working. :-(

Edit: Just went through the whole threads again, and it was adhoc parameter problem. Now it is solved. :P

kenji123

9th April 2008, 22:15

I've integrated x264farm into my gui, i'm hopeing if someone could try it out so that i could get some feedback. its setup as controller based encoding. i've done tests using 3 pcs and it works ok. but would like to know how it works on a broader range of pcs.

file 1 - this runs on the controller pc (http://kenji123.uni.cc/ani-x264/Ani-x2641500.7z)
file 2 - copy this to all the agent pcs (http://kenji123.uni.cc/ani-x264/Ani-x264FARMClient2000.7z)

if needed, view this small tutorial to get it started; here (http://kenji123.uni.cc/ani-x264/guides/Encoding with x264Farm/index.html)

RickA

9th April 2008, 23:08

Greets,

Thank you so much for the GUI. Will give it a try as soon as my current encode finishes. I love the idea of being able to use multiple computers to work together on the same encode. :-)

Cheers,
Rick

Additional:
Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means? Will go back and read more before I start asking router questions. :-)

SpAwN_gUy

10th April 2008, 15:55

Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means?em.. how does AC3-Audio cooperates with VideoEncoding?

SpAwN_gUy

10th April 2008, 15:57

I've integrated x264farm into my gui, i'm hopeing if someone could try it out so that i could get some feedback. its setup as controller based encoding. i've done tests using 3 pcs and it works ok. but would like to know how it works on a broader range of pcs.

file 1 - this runs on the controller pc (http://kenji321.uni.cc/Ani-x2641500.7z)
file 2 - copy this to all the agent pcs (http://kenji321.uni.cc/Ani-x264FARMClient2000.7z)

if needed, view this small tutorial to get it started; here (http://kenji123.uni.cc/ani-x264/guides/Encoding with x264Farm/index.html)i've seen your GUI .. like in the start of the thread :) ... and just "before" meGUI-mod developement start..
i guess i'll take another shot.. and try it .. again :)

[P]ako

11th April 2008, 02:52

Greets,
Additional:
Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means? Will go back and read more before I start asking router questions. :-)

Well, x264 is just for video encoding. You still need to encode the audio with the program(s) of your liking to any of the formats that MKV supports and finally mux it with the video.

Razorholt

11th April 2008, 06:22

Date: #Apr.11.2008#
Time: [12:14 a.m.]

Stack Trace: at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at ani_x264farmclient.ani_x264farm.listener() in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 423
Message: Value cannot be null.
Parameter name: path

---------------------------------------------------------------------

Listening for server connection

Did I miss a config step somewhere? I followed your guide tho :)
Also, I got an error message when I want to minimize the window on the client's machine.
************** Exception Text **************
System.IO.FileNotFoundException: Could not find file 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'.
File name: 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Drawing.Icon..ctor(String fileName, Int32 width, Int32 height)
at System.Drawing.Icon..ctor(String fileName)
at ani_x264farmclient.ani_x264farm.OnResize(EventArgs e) in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 562
at System.Windows.Forms.Control.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Thanks,
- Dan

kenji123

11th April 2008, 14:49

Date: #Apr.11.2008#
Time: [12:14 a.m.]

Stack Trace: at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at ani_x264farmclient.ani_x264farm.listener() in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 423
Message: Value cannot be null.
Parameter name: path

---------------------------------------------------------------------

Listening for server connection

Did I miss a config step somewhere? I followed your guide tho :)

No you didn't.:) This one is my fault, a required directory wasn't created by the program.

Also, I got an error message when I want to minimize the window on the client's machine.
************** Exception Text **************
System.IO.FileNotFoundException: Could not find file 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'.
File name: 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Drawing.Icon..ctor(String fileName, Int32 width, Int32 height)
at System.Drawing.Icon..ctor(String fileName)
at ani_x264farmclient.ani_x264farm.OnResize(EventArgs e) in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 562
at System.Windows.Forms.Control.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Thanks,
- Dan

Again my fault. When the program minimizes, it loads an icon for the system tray, which i failed to include.:) This error would of also occurred with the controller program. (i've fixed it so that you don't need the icon file.)

anyways;
i've fixed the bugs, so you can replace the old exes with the ones in this file here (http://kenji123.uni.cc/ani-x264/Ani-x2641500FARM2000_exeonly.7z)

the links in my previous post have also be updated.

thanks for trying it out and providing feedback :)

Razorholt

11th April 2008, 15:37

Ok, I'm trying now... Always glad to help ;)

Razorholt

11th April 2008, 15:55

log start: #Apr.11.2008_9.45.a.m_Married_Life_1080p.mov#

---------------------------------------------------------------------
copying input file to tmpfiles directory

Source: C:\Videos\trailers\Married_Life_1080p.mov
Destination: C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov
Buffer Size: 1,000,000 Bytes/s

---------------------------------------------------------------------
remuxing to a mkv container

command line:
-o "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv" --engage allow_avc_in_vfw_mode "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov"

lines returned by the mkvmerge cli:

mkvmerge v2.0.2 ('You're My Flame') built on Feb 21 2007 23:40:43
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov': Using the Quicktime/MP4 demultiplexer.
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov' track 1: Using the MPEG-4 part 10 (AVC) video output module.
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov' track 2: Using the AAC output module.
The file 'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv' has been opened for writing.
The cue entries (the index) are being written...
Muxing took 5 seconds.

---------------------------------------------------------------------
creating avisynth script

script generated:

LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Undot.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\FluxSmooth.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Convolution3DYV12.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\AutoCrop.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Decomb.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\VSFilter.dll")

function logo (clip orig, string image, int start, int length, int fadein, int fadeout, int trans, int x, int y, int threshold)
{
trans = Default(trans, 0)
fadein = Default(fadein, 0)
fadeout = Default(fadeout, 0)
x = Default(x, 0)
y = Default(y, 0)
threshold = Default(threshold, 0)
framerate = orig.framerate()
overlay = ImageReader(image, 0, length-1, framerate, false)
mask= overlay.ConvertToRGB32().ColorKeyMask(trans, threshold).ShowAlpha().FadeIO(fadein,fadeout)
numframes = overlay.framecount()
origclip = orig.Trim(start, start+length-1)
newclip = Overlay(origclip,overlay,x=x, y=y, mask=mask)
return orig.Trim(0, start - 1) + newclip + orig.Trim(start + numframes, 0)
}

DirectShowSource("C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv",convertfps=true,audio=false) # Source
Lanczos4Resize(704,400) # Lanczos (Sharp)

logo("F:\[6]Graphics\My Graphics\otakuleg_textlogov1 copy.png",0,300,25,25,0,5,5,0)

Telecide() # Inverse Telecine
Decimate(5) # Inverse Telecine

FadeIO2(25) # FadeIn/FadeOut Frames
ConvertToYV12()

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:

Connecting to client no. 1
Connection established with client no. 1
Sending job info to client no. 1
Job info finished sending; sending agent configuration
Sending agent configuration to client no. 1
Configuration finished sending; starting x264Farm agent
x264Farm agent is running on client no. 1
Disconnected from client no. 1
Working...
---------------------------------------------------------------------
starting x264Farm controller

command line:
--first "--qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress" --second "--merange 12 --me umh --subme 6 --filter 1,1 --trellis 1 --ref 8 --mixed-refs --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --bframes 16 --b-pyramid --b-rdo --weightb --bime --direct auto --progress" --bitrate 280kbps --avs "InputFile.avs" --output "OutputFile.mkv" --config "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"

lines returned by the x264Farm Controller, Agent(s) and x264 cli:

x264farm agent version 1.15-182
x264farm controller version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_agent_config.xml"
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"
Temp dir: "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186"
CONFIG:
Ports: 6060-6070
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\MarriedLifemov1182793186"
x264: "x264.exe"
agent list:
nice: 10
Local = 127.0.0.1:6060 (1,1)
agents: (1,1)
bobby4 = 192.168.0.1:6060 (1,2)
agent bases: [None]
pipe: false
compressions: 0001 - 0001
deleting old files: after 604800 seconds
UDP ping port: 40700
UDP listening port: 40701
UDP name: "Local"
trying port 6060
Removing stale files from temp dir...
bound to port 6060
listening...
working!
Listener set up
WARNING: avs2yuv exited with error 1; avs2yuv responded:

x264Farm did not end as expected
Trying to recoverx264farm controller version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"
CONFIG:
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\MarriedLifemov1182793186"
agent list:
Local = 127.0.0.1:6060 (1,1)
bobby4 = 192.168.0.1:6060 (1,2)
WARNING: avs2yuv exited with error 1; avs2yuv responded:

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:
Connection established with client no. 1
Connecting to client no. 1
Stopping x264Farm agent on client no. 1
x264Farm agent has been stopped on client no. 1
Disconnecting from client no. 1
Disconnected from client no. 1
Working...

---------------------------------------------------------------------

log end: #Apr.11.2008_9.46.a.m_Married_Life_1080p.mov#

I also noticed that I can't import AVS files. Is it correct?

kenji123

11th April 2008, 16:19

You will need to select or create a new profile. the profile it is set at isusing a custom AVS script i was using for personal usage. you can easily create or edit profiles by going to the settings tab.

It checks the script first for errors, also any files used in the script must have an absolute path (eg. not "AVISource("test.avi")" but "AVISource("D:\folder\test.avi")").

If thats not the case, can you post the AVS script?

Razorholt

11th April 2008, 17:27

I have updated my avs file. It seems to be working at that level but then, I got this:
Recent errors:
4:07 bobby4 failed with "End_of_file"
173 recv thread got exception; killing listener thread
4:09 bobby4 failed with "End_of_file"
172 first pass frames 0 to 3432
4:11 Local failed with "End_of_file"
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 Cache64
172 controller-based encoding
4:13 Local failed with "End_of_file"
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 Cache64
172 doing the following:
nice -n 10 "x264.exe" --qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 2997/125 --pass 1 --stats "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov915614466\stats 6702cb.txt" -o NUL - 528x288
Last updated: 4:15
Got a connection from 192.168.0.150:1311
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
working!
Agent | FPS | Description
174 recv thread got exception; killing listener thread
Local | 0.00 | Sending [0-3432]
Thread received exception "End_of_file"
bobby4 | 0.00 | Waiting for job
closing socket
Local | 0.00 | Connected to 192.168.0.150:6060
socket closed
bobby4 | 0.00 | Disconnected
Thread received exception "End_of_file"
Recent errors:
closing socket
4:07 bobby4 failed with "End_of_file"
socket closed
4:09 bobby4 failed with "End_of_file"
4:11 Local failed with "End_of_file"
4:13 Local failed with "End_of_file"
Thread received exception ("An existing connection was forcibly closed by the remote host.\013\n",recv,)
Last updated: 4:15
closing socket
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
socket closed
Agent | FPS | Description

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:
Connection established with client no. 1
Connecting to client no. 1
Stopping x264Farm agent on client no. 1
x264Farm agent has been stopped on client no. 1
Disconnecting from client no. 1
Disconnected from client no. 1
Working...

---------------------------------------------------------------------

log end: #Apr.11.2008_11.24.a.m_Married_Life_1080p.mov#

gnzrg25

11th April 2008, 18:34

Hi. Im newbie...
I'm trying to compile x264 with visual c++ 2005.
Get these errors.

1>------ Build started: Project: x264, Configuration: Release Win32 ------
1>Linking...
1>x264.obj : error LNK2019: unresolved external symbol _strncasecmp referenced in function _Parse
1>libx264.lib(encoder.obj) : error LNK2019: unresolved external symbol _x264_cqm_init referenced in function _x264_encoder_open
1>libx264.lib(encoder.obj) : error LNK2019: unresolved external symbol _x264_cqm_parse_file referenced in function _x264_encoder_open
1>bin/x264.exe : fatal error LNK1120: 3 unresolved externals
1>Build log was saved at "file://c:\Documents and Settings\Administrator\Desktop\X264_rev270_src\x264\build\win32\obj\x264_Release\BuildLog.htm"
1>x264 - 4 error(s), 0 warning(s)
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

tnx..

Razorholt

11th April 2008, 19:59

My customized AVS:
DirectShowSource("{source}",convertfps=true,audio=false) # Source
Lanczos4Resize(704,400) # Lanczos (Sharp)
ConvertToYV12()

Log:
starting x264Farm controller

command line:
--first "--qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress" --second "--merange 12 --me umh --subme 6 --filter 1,1 --trellis 1 --ref 8 --mixed-refs --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --bframes 16 --b-pyramid --b-rdo --weightb --bime --direct auto --progress" --bitrate 280kbps --avs "InputFile.avs" --output "OutputFile.mkv" --config "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_controller_config.xml"

lines returned by the x264Farm Controller, Agent(s) and x264 cli:

x264farm agent version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_agent_config.xml"
Temp dir: "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371"
Ports: 6060-6065
x264: "x264.exe"
nice: 10
agents: (1,1)
agent bases: [None]
pipe: false
compressions: 0001 - 0001
deleting old files: after 604800 seconds
UDP ping port: 40700
x264farm controller version 1.15-182
UDP listening port: 40701
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_controller_config.xml"
UDP name: "Local"
CONFIG:
trying port 6060
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\test2avi2147434371"
Removing stale files from temp dir...
agent list:
bound to port 6060
Local = 127.0.0.1:6060 (1,1)
listening...
bobby4 = 192.168.0.1:6060 (1,2)
Listener set up
working!

I see Job 1 - analyzing - [working...] at the status bar and that's it. Nothing is happening :(

RickA

11th April 2008, 21:52

To those not able to follow the thread where I was talking about the nice GUI for x264farm created by kenji123 - thanks again btw. Um, the one that he posted right above mine... Which I was referring to in my post giving the option of using the MKV container with AAC, but no option for AC3 audio in the MKV container.

Then gave a good reason why it would be nice of him to implement said option. Since just about every DVD and Hi-Def disc under the sun has AC3 audio by default. As well as the vast majority of home sound systems having the support for directly decoding AC3 Dolby Digital more so than AAC.

I know x264 is only for encoding video content. How many of you seriously watch / encode video without putting an audio in the container with it to listen to? That is why I was addressing the GUI. Have you even tried his GUI to have an idea of what I was talking about before jumping in with comments?

Cheers,
R

[P]ako

12th April 2008, 23:35

I know x264 is only for encoding video content. How many of you seriously watch / encode video without putting an audio in the container with it to listen to? That is why I was addressing the GUI. Have you even tried his GUI to have an idea of what I was talking about before jumping in with comments?

Cheers,
R

Please read your post carefully, in your addendum you mentioned x264farm and AC3, not the GUI for x264farm and AC3. And because you were speaking of x264farm I assumed you were addressing the thread poster (Omion). There is no reason to get pissy over the internet when someone seems not to understand you.

SpAwN_gUy

14th April 2008, 15:15

Hi. Im newbie...
I'm trying to compile x264 with visual c++ 2005.
Get these errors.
1st) this should go to x264 compilation thread..
2nd) I've just mastered it (again.. yes.. za sikret tikniq, 2 build x264 under Win MSVS 2005 ... even with pathces,.. and gpac and pthreads.. bwahahahaha)
ok.. i'll help..
1) convert .sln from 2003version to 2005 (as you did, as i can see)
2) add to project: "cabac-a.asm" to Core/x86/ folder AND configure it to compile with nasm as OTHER .asm's

ok. here is compilation of Mine:
upd. removed.... due to producing lotsofsh_t...

SpAwN_gUy

15th April 2008, 15:08

ok.. i've made another one... this one is a bit MORE proper :) ..

x264 rev.819, modified, MSVC2005 build
General thread:
http://forum.doom9.org/showthread.php?t=130364

x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.7.diff
http://thread.gmane.org/gmane.comp.video.x264.devel/3093/focus=3748
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing
http://forum.doom9.org/showthread.php?p=1047919#post1047919

grab it here (http://rapidshare.com/files/107703602/x264.819.msvc2005.modified.exe)

nvubert

22nd April 2008, 16:35

your x264farm is what i needed for some long time for my HD encodes. I'm using it about 5 months and it's really a piece of good job.
Here's my setup:
Controller:Q6600@4.0GHz (wc) 4GBDDR2-1066 2xRaptor36GB raid0 and 2xF1 320GB raid0, selfmodded XPSP2
Agents:my HTPC with E6600@3.2GHz 2GB-DDR2-800, E4300@3.2FGHz 2GB-DDR2-800 and 3xA64 X2@2.66GHz 2GBDDR2-800 ( network 1Gbps )

My question is:
Is possible to configure controller that only second pass will be processed over the agents and first pass only on my local Q6600? ( i see that v1.15 have recognize local and auto configure with agent encoding that not use av2yuv.exe )
The reason is that on 720p encodes at first pass with no avs filters speed is very slow 8-30fps, the same source on megui on local only i have 20-50fps ( depending on source )
Maybe is some trick that allow encode only 2pass on x264farm with first pass encoded on standalone x264.exe
Thanks for help

SpAwN_gUy

22nd April 2008, 16:51

The reason is that on 720p encodes at first pass with no avs filters speed is very slow 8-30fps, the same source on megui on local only i have 20-50fps ( depending on source ) you are doing controller-based or agent-based encodings?
Maybe is some trick that allow encode only 2pass on x264farm with first pass encoded on standalone x264.exe there is a trick.
you make first-pass on standalone, then start x264farm.. and after it starts doing first pass.... quit it (terminate). then locate folder (i.e. "193949954353 your.avs") in controller's temp folder, then (i guess.. i haven't done it myself) delete dir (in the dir metioned before) named like "1-34534...." and replace(or create...) file "1-BC95ECB7B64E98DEB......txt" with your .stats file form first-pass standalone encode...

but, doing agent-based is alot faster(on my side here)

nvubert

22nd April 2008, 17:11

I do controller based encoding because i can't configure every machine for share network,avs filters,etc. However since v1.15 agent on my controller self-configure for agent based encoding ( detecting avs file and says agent encoding - there's no avs2yuv.exe in taskmanager for it ) and other computers doing controller-based.

This "trick" I already knew, but i need some automate this ( if this is only solution )
It is very complicate to do this "hack" on every encode

SpAwN_gUy

23rd April 2008, 08:12

I do controller based encoding because i can't configure every machine for share network,avs filters,etc. i can... teach you how to setup agent-based... (well, there is a tutorial for mpeg2 encoding in my signature)... and i can explain everything... 'cause now i do AVC reEncodings agent-based...

nvubert

23rd April 2008, 10:40

There's no need to teach things i already know :)

Problem is that some computers can't host avs filters ( sometimes i use lots of them and scripts are heavy ) because they are HTPC's for TV 24/7 and others media in houses my neighbors, but anyway thanks for good intention :thanks:

[P]ako

27th April 2008, 19:39

Omion, would it be possible to have a new build of x264farm that includes the AQ patch (already in cvs) and the experimental film grain optimization patch?

FGO patch: http://forum.doom9.org/showthread.php?t=137117&highlight=patch

SpAwN_gUy

2nd May 2008, 13:31

ako;1129577']Omion, would it be possible to have a new build of x264farm that includes the AQ patch (already in cvs) and the experimental film grain optimization patch?

FGO patch: http://forum.doom9.org/showthread.php?t=137117&highlight=patchthere is no reason to reCompile x264farm... just replace x264.exe itself.

if you wish.. i've made r839 MSVC2005 build of x264... (but without patches)... and i'm struggling with git-apply ('cause it would be much more simplyer.. then to applay them by hands..(as i did before..))

[P]ako

2nd May 2008, 15:41

there is no reason to reCompile x264farm... just replace x264.exe itself.

Did I miss something? If I remember correctly Omion said that x264farm works only with his MSVS build of x264.

Dustin013

2nd May 2008, 20:27

I have been playing around with x264 and I am loving the codec. I have found a nice setup to encode all my Divx backups to work on my PC and IPhone which is amazing. However, I am using dual Intel 3.2ghz Dual Core Xeon processors and I can still only encode around a 3:1 ratio (about 1/3 the play time length to encode a file) when encoding AVI into a x264 format in a MP4 wrapper.

x264Farm seems like the perfect solution to speed this whole process up. However, I am converting AVIs encoded in Xvid mainly and some Divx and it would seem x264farm doesn't support AVI encoding?

Is it possible to modify x264farm to allow encoding of AVI files and possibly setup a way to batch encode?

I currently have access to some servers at work I was given permission to test on. I have 11 servers I can use to try this out. The controller server is a mammoth with 16GB ram and top of the line dual core processors. Then I have 10 dell 1U units that have 2GB ram in them each and dual 2.8ghz DC Xeon processors. If I could get the big server to work as the controller and the other 10 as agents I think I could encode my entire 3TB collection of backups into Mp4 format fairly quickly.

Any suggestions?

driftr

4th May 2008, 20:21

Dustin013: you can convert any file to x264. I've done mpeg2, avi (xvid, divx, wmv), etc etc. with x264rfarm. x264farm is just a program that utilizes the x264 codec so there's virtually no difference other than the supported build of the codec.

You just have to feed x264farm an AVS script. If you haven't used avisynth that's what you need to read up on and there are many guides at doom9 to help with that.

And one thing to keep in mind for the controller/agent computers is you'll have to install avisynth and any codecs on the computers. So you might want to be sure the powers that be are aware the process isn't a light weight application (thou it shouldn't interfere with any usual processes).

SpAwN_gUy

7th May 2008, 08:27

ako;1133563']Did I miss something? If I remember correctly Omion said that x264farm works only with his MSVS build of x264. one page back there is rev819..
i have unpatched 839... (i can post, and upgrade to current rev.)
i have linux and will try to use it for patching sources.. 'cause my conditions do not allow me to set up git properly..

[P]ako

7th May 2008, 16:05

I see. When I have time, I will try with Shikari's build of x264 with the FGO patch.

Humbula

25th May 2008, 21:36

elo,

can you tell me how you calculate the bitrate needed for a part of the second pass? is it possible to calculate is from the passfile?

Adub

26th May 2008, 02:47

Use a bitrate calculator. Just punch it into google, use MeGUI, or any other variety of programs. Again, if you google it, you'll find hundreds. And this is probably the wrong thread.

Humbula

26th May 2008, 05:22

Use a bitrate calculator. Just punch it into google, use MeGUI, or any other variety of programs. Again, if you google it, you'll find hundreds. And this is probably the wrong thread.

No, you didn't understand my posting. Let me try again: x264 farm "cuts" the movie into parts, which will be encoded by the next machine, requesting a job. Lets say one of this parts has about 250 frames.
How does x264 farm calculate the bitrate (needed) for this part?

Adub

26th May 2008, 05:54

Humbula

26th May 2008, 06:15

The bitrate shouldn't change. When you specify a bitrate in the commandline, you are saying to the encoder "aim for this". It doesn't matter if you split up the file or not, the bitrate will stay the same, as you want uniform quality throughout your encode.

x264 farm's aim in NOT to keep the bitrate for every part the same.

"The ratecontrol used is designed to be as close to the ratecontrol done by x264 itself"

I want to know how this ratecontrol works...

Adub

26th May 2008, 20:35

Oh, okay. I am sorry, I didn't fully comprehend what you are asking. And frankly, I don't know the answer. Probably the only way to find out is to look at the source or talk to Omion.

Humbula

26th May 2008, 21:32

to be honest i dont understand the code. strange language... :)

omion, help me...

foxyshadis

27th May 2008, 00:17

x264farm just processes the first pass stats files to come up with each segment's relative sizes.

Humbula

27th May 2008, 06:44

x264farm just processes the first pass stats files to come up with each segment's relative sizes.

And how can I calculate a segments size using the statsfile?

omion

27th May 2008, 06:49

to be honest i dont understand the code. strange language... :)

omion, help me...
The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it :D

And how can I calculate a segments size using the statsfile?
Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused. :)

Humbula

27th May 2008, 23:00

The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it :D

:goodpost:

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused. :)

Thanks, I'll have a look :)

akupenguin

28th May 2008, 02:40

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)
Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

omion

28th May 2008, 06:05

Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

Ah. Yes. I just realized that I actually answered a different question than what was asked. The size of the first pass frames is in itex+ptex+mv+misc (which was the question I was answering), but that does not directly correspond to the second pass frame size (which was the question asked). The stats file is processed through the functions in ratecontrol.c before the optimal second pass frame sizes are known.

Even after processing by the ratecontrol functions, there is no guarantee that the frames will actually be that size. In fact, since x264 can't directly set the number of bits used in a frame, it will always be a little off.

taemun

28th May 2008, 11:42

Adub

28th May 2008, 19:56

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent.

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

foxyshadis

28th May 2008, 23:27

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

Depends on if the input is bottlenecked or not. (Combine MT+x264 threading and you get better results.)

On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Avisynth caches. You can use setmemorymax to control that, since avisynth caching ignores multiple instances. (This is even worse on 2.58, where default is half the available system memory, not one quarter. If they all initialize at once, they all see the same available memory.) Even if frames are never needed again, they'll sit in cache until their number comes up when the maximum memory limit is reached. Then it just stays at that limit until you're done encoding, mostly.

omion

28th May 2008, 23:49

Hey all.

I'm new to x264 encoding in general, (have worked out what most of the buttons in MeGUI etc do), and given that I have several computers lying around here, I thought that there should be a relatively easy way to get x264 to run over all of them at once.

Hence, found this project.

I've had it running, although I'd have to admit that a few things aren't clear to me:

1> Unless I specify a rate control method in the -1 "blah blah", it doesn't do anything at all.... was this intended? What value should I be using in here for a good quality 720p rip (is --qp 26 sufficient?).
That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

I generally use a lower setting (--qp 22 or something) but do a few tests to see what looks good for you.

2> I'd like a little more control over the IP that the controller and agents bind to ... I have multiple IP addresses, that will go differing routes to get to the clients. One of those interfaces is gig, one is 100M. I'd rather be able to force it to go the gig path. This may be related to my inability to work out the networking parameters in the controller - the first time I tried I couldn't get anything aside from adhoc working - this suits my purposes fine, as you can kill the processes remotely, but adhoc can't specify the controller IP to be bound to.

Currently it's set to bind to any IP address, since I didn't need the ability to change it. That feature would be nice for some people, though, so I'll add it in. However, it's probably not going to get into the 2-pass version of x264farm any time soon, as I'm mainly working on the 1-pass version now.

3> I'd like to be able to change the agents list/the number of x264 instances per agent whilst the controller is running. You can sort of do this by running in adhoc, and just starting/closing/restarting the agent whenever you want to change something, but that only works (for me) if you can force the IP that the adhoc controller is running on.
I have (pretty buggy) support for that in the 1-pass version. It does still need you to restart the agent every time you want to change the settings, but it generally works. In non-adhoc mode for the 2-pass version, the agent list is defined immutably in the beginning of the encode. Changing the way that works would be fairly difficult.

4> What is AVS2YUV doing here? I presently have three instances of x264 running over two computers (one on controller, ~500MB/ram, two remote, ~230MB/ram each), and the controller also has three avs2yuv's running:
One of 766MB (no CPU-time, unless an agent recently finished a block).
Two of 550MB (3~13% CPU-time). Presumably these are frameserving across the network (controller based encoding).
[as I've been typing this, both of the 550MB processes have died, and two new ones of around 350MB have come up]

Reason I don't understand, is because x264 can handle an .avs input directly, why is there a need for a frame server here? (proviso: during agent-based encoding)
On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Any controller-based encode needs to start up its own instance of avs2yuv to render the AVS file. There is also one extra avs2yuv which the controller consults for finding scene cuts. The memory usage is always a bit large by default (since Avisynth is set up to cache everything it gets its hands on) but you can limit that by using SetMemoryMax(...) in the AVS file. You are correct that frameserving should use very little RAM - every output frame is used exactly once, and in order. Therefore you should see no performance decrease by limiting the amount of memory Avisynth uses.

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$

I can't really help you there. I don't know the inner workings of graphedit or MeGUI... Generally errors like that are due to some path being hard-coded in the AVS (or an included file), but since you have the agent's U:\ mapped to the controller's U:\, I don't really know.

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent. Aside from the small delay that occurs when dumping back the stats file, or the resulting .mkv files in the second pass. IF you're worried about that delay in the second pass, I can see why you might use pad=1; number=1. (btw, why not simplify that into, numberFirstPass=1; numberSecondPass=2?)
I honestly don't know how they compare. When x264 first added threading support I tested it out, and using multiple single-threaded x264 processes worked a little faster. But then the threading model was completely changed around and it looks like it's quite a bit better now than it used to be. Test it out and report back! (oh, and use "--threads auto", or it will force all agents to use the same number of threads, which is probably not what you want;))

foxyshadis

29th May 2008, 06:25

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$
Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

taemun

29th May 2008, 07:02

Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

:D
Yeah I came to that conclusion late last night. CoreAVC was only installed on the controller, so of course it failed. Pity I didn't think about that at the time (after trying to dig through the .grf with notepad :P). I'd forgotten that I had used something a little more exotic when I made the graph.

That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

Yeah I did see that helpful warning, was more wondering whether I was doing something wrong, because the FAQ (http://omion.dyndns.org/x264farm/x264farm.html) didn't mention it.

That feature would be nice for some people, though, so I'll add it in.

Thanks, that'd be great.

There is also one extra avs2yuv which the controller consults for finding scene cuts.

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end :)

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t

omion

29th May 2008, 07:43

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end :)

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

That would certainly make the most sense, wouldn't it?:p The only real reason I didn't do that is that I had abstracted away the avs2yuv process so far that it hid all the parts that I could have closed. Rather than attempting to do that, I decided instead to try and get rid of the splitter altogether. After much hacking away, I finally decided to do a ground-up rewrite to the entire program, which turned into x264farm-sp (which doesn't have a splitter). The 2-pass x264farm has since fallen by the wayside.

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

That is due to an annoying bug that x264 will open the stdin handle in text mode, meaning that it will interpret ASCII character 0x1A as an EOF marker and stop encoding. (it also turns 0x0D0A into 0x0D) MSVC builds don't have this issue, though, as there are a few MSVC-specific lines in x264 to switch to binary.

This is only a problem in controller-based mode, though. You may use anything you want if you ensure that the computer is using agent-based encoding.

(Note that it's not really a bug in x264; I think stdin and stdout are usually opened in text-mode in Windows. It's just a bit odd that x264 expects binary data to pass through them...)

It should be an easy fix for making it work in cygwin/msys, but I really don't know enough C to say for sure.

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t
I'm not really sure what the most current version of MSVC x264 is. I gave up making my own builds a long time ago, since x264 is really only tested against GCC, and some of the changes would break MSVC for long periods of time. Not knowing a lot of C, I was clueless to fix it in these cases. Since then, I have had no need for controller-based encoding (on Windows boxes), so I haven't kept up with the progress. I'll have to defer to SpAwN_gUy to answer that question.

If you can get agent-based encoding up and running, then you can bypass this restriction. That would probably be the best course of action right now. (if the CoreAVC fix made agent-based encoding work, then you can use anything you want)

SpAwN_gUy

2nd June 2008, 11:58

i've just moved to the new PCbox ... so i've lost the tracks of the threads...

okay, since b0bor started to post patched sources (damn git..) it will not be too hard to make MSVC builds ...

good news are: meGUI-x264farm-mod (ma-a-n... this is quite a huge name...) is still indev.. i've moved 80% of code to the latest (like two weeks ago) meGUI revision... so.. some may wait for a new version soon..

/*offtop... as far as i'm learning bash and perl (and doing some fancy scripts..).. C# is my "back to the reality" thing... so more to a hobby.. but i'll catchUP :) ... and i'm on Vista x64 now..

Adub

2nd June 2008, 22:03

Good to hear from you again. As you know, looking forward to the mod!! Thanks again.

thebigfatgeek

7th June 2008, 23:19

Hi, I have just installed the agents and the controllers, modified th config.xml files and I keep on getting the same result. I cannot see the error, or that anything is happening (except some processor activity on controller.exe and avs2yuv, but after 0:46 seconds, it stops, it does not complain (which is nice) but it also does not work (which is not nice)

This is the output from the out-dump.txt file:

2008-06-07~23:09:23.53 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.53 Last updated: 0:45
2008-06-07~23:09:23.53 ----...----________________________________...............-----...............
2008-06-07~23:09:23.53 Agent | FPS | Description
2008-06-07~23:09:23.53 Recent errors:
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 SPLITTER MUTEX unlocking range_list_mutex and yielding
2008-06-07~23:09:23.53 SPLITTER MUTEX sleeping for 1/10 second
2008-06-07~23:09:23.64 SPLITTER MUTEX locked range_list_mutex after yielding
2008-06-07~23:09:23.64 e 0
2008-06-07~23:09:23.64 f 0
2008-06-07~23:09:23.64 q 3
2008-06-07~23:09:23.64 w 0
2008-06-07~23:09:23.64 es 3
2008-06-07~23:09:23.64 fs 0
2008-06-07~23:09:23.64 qs 8
2008-06-07~23:09:23.64 ws 0
2008-06-07~23:09:23.64 Splitter: current batch length is 2116
2008-06-07~23:09:23.64 Splitter: try_queue_after_full_or_working
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 Splitter: try_extend_small_thresh_queue
2008-06-07~23:09:23.64 extend_small_thresh_queue found no ranges of sufficient patheticosity
2008-06-07~23:09:23.64 Splitter: try_split_largest_range
2008-06-07~23:09:23.64 Splitter: wait_on_full
2008-06-07~23:09:23.64 SPLITTER MUTEX broadcasting range_list_empty_condition
2008-06-07~23:09:23.64 0 @ INFINITY
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 2502 @ 7.56
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 4607 @ 10.63
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 6857 @ 4.69
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 9282 @ 2.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 11709 @ 7.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 14176 @ 2.42
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 16791 @ 6.33
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 19347 @ 7.58
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 22022 @ 9.34
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 24696 @ 4.79
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 27246 @ 3.65
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 37171 @ 11.85
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 39525 @ 6.16
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 49544 @ INFINITY
2008-06-07~23:09:23.64 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

and my controller was started with:

Controller>controller -B 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --me range 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weigh tb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --m e umh --merange 12 --progress --no-psnr" --avs "24.avs" -o "24.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

Any ideas or comments would be appreciated

Cheers

ajp_anton

16th July 2008, 15:33

Bug!

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network.

Apparently I can get around this by creating a .avs file that finds its inputs via the network, and copy that file onto every computer, even the local one (since it seems intelligent enough to check if the files are the same... =)

Otherwise this thing is really the best thing ever. Too bad it's not until now that I really needed it as it took only 10 mins to figure everything out and get it working.

SpAwN_gUy

17th July 2008, 09:08

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network. i have no problem with that ...
which nice are you using? (the one in the first post) jr the original? .. 'cause we had some problems using the original one.. and some problems using the tiny one(1st post)

currently i'm using the one in the first post... and i'm doing agent-based..

SpAwN_gUy

17th July 2008, 09:12

2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex
controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

SpAwN_gUy

17th July 2008, 09:18

oh.. jeah.. right.. the mod.. i need to start MSVC... and stop doing nothing...

btw.. about builds of x264... it is reported, that MSVC builds (with current revision of x264) are TOTALLY BROKEN.. and must not be used :(( ...
but the good nwes are.. that x264_mingw_stdin_patch is IN the TRUNK :) .. (yeah, yeah,.. thanks to me.. but you are welcome..) so all the builds, that are currently made.. (like on x264.nl and all the patched ones) need to be checked(at first).. and then.. well.. the must be working and compatible with controller-based x264farm encoding process..

thebigfatgeek

19th July 2008, 02:19

controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

I can get it to see the agent, but I just cannot get a controller startup command that will work:

nice -n 20 "x264.exe" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --s
bme 1 --analyse none --progress --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.
--qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass
1 --stats ".\temp\stats b26912.txt" -o NUL - 640x496
264 [error]: no ratecontrol method specified
264 [error]: x264_encoder_open failed

Wishbringer

19th July 2008, 10:44

Controller>controller -B 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --me range 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weigh tb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --m e umh --merange 12 --progress --no-psnr" --avs "24.avs" -o "24.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

You did set up the final bitrate for the second pass, but

264 [error]: no ratecontrol method specified

look at first pass string , there is no ratecontrol set. I would add something like --crf 18 into first pass string

see: http://forum.doom9.org/showpost.php?p=1143312&postcount=589 some posts before.

SpAwN_gUy

5th August 2008, 10:51

hi, guys, i've been asked to help with x264farm setup...
and i'm on the new pcBox ... so i had some time.. to just test everything from the scratch or something like this...
http://www.ljplus.ru/img4/s/p/spawn_guy/th_x264farm-vista-x64-doubleAgent-gcc_928.jpg
fullPicture is here (http://www.ljplus.ru/img4/s/p/spawn_guy/x264farm-vista-x64-doubleAgent-gcc_928.jpg)
the Picture shows win Vista SP1 x64 ... with two running agents.
one from commandline, - this one is doing controller-based
second started over the network through my agentGUI. - this one is agent-based
they both use nice at 20,
and mingw compiled x264 x86 r928 from b0bor's site .. (plain git)
no x264x64 (i havent found one x64 for now)
aviSynth 258.. but x86

test encode uses mpeg2-progressive source, mpeg2-Telecined with YATTA, x264-source form HDTVstream using DGAVCDec..

to get all thing working i needed to (this is imporatnt) set nice's compatibility mode to winXP-SP2 ... otherwise the whole thig was not working.. (but only for agents...)

and.. UNBLOK controller and agents in winFirewall !!!

and that's just it :) ...
any questions are welcome ....
(i guess, i should better show the meGUI-Mod.. but it is still unusable... (well, the new version))

controller's config.xml
<config>
<temp>controller-temp</temp>

<adhoc controller="40700" agent="40701"/>
<agents>
</agents>
</config>
agent's config (the command-line controller-based one):
<config>
<temp>temp</temp>
<port from="40700" to="40701"/>
<x264>x264.exe</x264>
<nice>20</nice>
<adhoc controller="40700" agent="40701"/>
<number>1</number>
<agentpipe>0</agentpipe>
<compression>
<type>1</type>
<type>0</type>
</compression>
</config>
angent's config (the one that is common and is started over the network with my agentGUI (it can be started the same way using "\\myPC\x264farm\!AGENT\agent.exe --config \\myPC\x264farm\!AGENT\config.xml")):
<config>

<temp>temp</temp>
<port from="40700" to="40701"/>
<x264>\\myPC\x264farm\!AGENT\x264.exe</x264>
<nice>20</nice>
<bases>
<base>\\myPC\x264farm-2\</base>
<base>\\myPC\x264farm\</base>
</bases>
<adhoc controller="40700" agent="40701"/>
<number>1</number>
<agentpipe>0</agentpipe>
<compression>
<type>1</type>
<type>0</type>
</compression>
</config>

theese are the internals of test.avs used:
SetMemoryMax(64)

# PLUGINS
LoadPlugin("\\myPC\x264farm\TEST\filters\DGDecode.dll")
LoadPlugin("\\myPC\x264farm\TEST\filters\FluxSmooth.dll")
LoadPlugin("\\myPC\x264farm\TEST\filters\DGAVCDecode.dll")
LoadPlugin("\\myPC\x264farm\!filters\TDeint.dll")
LoadPlugin("\\myPC\x264farm\!filters\TIVTC.dll")

function mpeg2(){
clip1=mpeg2source(".\mpeg2\test.d2v", idct=0)

clip1=crop(clip1,14,82,696,412)

return LanczosResize(clip1,640,480)
}

function mpeg2anime(){

d2vpath = ".\mpeg2-anime\yatta-test.d2v"
clip1=mpeg2source(d2vpath, idct=0)

deint = tdeint(clip1)
clip1 = tfm(clip1, d2v=d2vpath, clip2=deint)

return LanczosResize(clip1,640,480).AssumeFPS(25).Trim(400, 3400)
}

function ColBars() {
return ColorBars(640,480).AssumeFPS(25).ConvertToYV12().KillAudio.Trim(0,1000)
}

function h264() {
clip2=AVCSource(".\h264\hgttg-test.dga", deblock=false)

return clip2.AssumeFPS(25).ConvertToYV12().KillAudio.LanczosResize(640,480)
}

mpeg2().Info.Overlay(Version, x=200, y=400) + ColBars().Info.Overlay(Version, x=200, y=400) + h264().Info.Overlay(Version, x=200, y=400) + mpeg2anime().Info.Overlay(Version)

the is: E:\DVDRIP\x264farm\TEST\mpeg2\test.d2v internals (the .vob is located just near E:\DVDRIP\x264farm\TEST\mpeg2\VTS_01_8.VOB)
DGIndexProjectFile16
1
VTS_01_8.VOB

Stream_Type=1
MPEG_Type=2
iDCT_Algorithm=5
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=16:9
Picture_Size=720x576
Field_Operation=0
Frame_Rate=25000 (25/1)
Location=0,0,0,b9de

900 5 0 20480 0 7 18 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 e2
d00 5 0 258048 0 7 18 d2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
.......
this is E:\DVDRIP\x264farm\TEST\h264\hgttg-test.dga
the only part i har to change manually is path to libavcodec (this is not implemented to be relative-path .. i had to change it to network-path)
DGAVCIndexFile5

hgttg-test.h264
\\myPC\x264farm\TEST\filters\libavcodec.dll

FO 2

LPOC 0
SPS 0
PPS 28

this is E:\DVDRIP\x264farm\TEST\mpeg2-anime\yatta-test.d2v (the one that is manually inverse-Telecined with YATTA ( ivtc.org ))
DGIndexProjectFile16
1
VTS_01_5.VOB

Stream_Type=1
MPEG_Type=2
iDCT_Algorithm=5
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=4:3
Picture_Size=720x480
Field_Operation=0
Frame_Rate=29970 (30000/1001)
Location=0,0,0,44837

900 5 0 2048 0 5 2 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 b2 a2
...
aviSynth filters are located here
localPath: E:\DVDRIP\x264farm\TEST\filters\
networkPath: \\myPC\x264farm\TEST\filters
basically.. my setUP is to use network-share:
localPath: E:\DVDRIP\x264farm
networkPath: \\myPC\x264farm
for my agentGUI i use common agent setUP located here:
local: E:\DVDRIP\x264farm\!AGENT
network: \\myPC\x264farm\!AGENT

that's i folk :) ... it's pretty simple...

the some kind of minimal setUP can be obtained form here (http://rapidshare.com/files/134974974/x264farm-minimal.rar)

winAMP plays: [Apex - Same Old Blues]:[Drum & Bass-2008]:-:[5:40s]:[320-44KHz-stereo]:[MP3]

BTW: if you find any confidential data in things, that i wrote here .. please.. DON'T USE IT.. and tell me about it..
some mistakes can be.. that "myPC" real network name is "SC0151" remember that...

Jabroni

15th September 2008, 23:05

omion

15th September 2008, 23:24

Im trying to get this working with a newer X264 build (techouse builds), however Im getting some probs. I have been able to make it process the frames, the agent doesnt display any errors at all, neither on the controller, but the % of what is done never increases (it seems like the agent isnt returning the encoded info)

Any place what I should be looking into ??

:thanks:

I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.

Jabroni

15th September 2008, 23:30

I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.

Ok good.. thought this was just me being stupid on setting the new build :p Well for now i'll just stick with an old x264.exe build (the one that I had working previously)

This is the url of the custom build that I tried hxxp://techouse.project357.com/ (latest svn release) If you need any more info for debug let me know and I'll happly try to help :)

kemuri-_9

16th September 2008, 00:54

I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.

if you really need to get back to the old stats format, you can just make a build that has that revision which affected it reversed.
keep all the other improvements and such...

SpAwN_gUy

16th September 2008, 14:42

so... we are back in bizzznisss? ;) ... cool :) ...

'cause re-encoding avchd from my sony hdr-sr12 to something reasonable at 720p... is really painful.. just "standalone"...
megui-mod update is really close :)

ajp_anton

28th September 2008, 19:29

So are we going to see a version of x264farm that supports the new stats files?

omion

29th September 2008, 16:57

So are we going to see a version of x264farm that supports the new stats files?

It'll be out eventually. The problem that I've had is that after I fixed the stats problem I noticed that the encode does not progress, so I'll have to fix that as well, and I haven't looked at the x264farm code in a long time. Part of the reason it's taken so long is that I also updated the regular-expression parser because I no longer have PCRE on my system. So it's still being worked on...

Nakaleen

8th October 2008, 02:22

Hi I am new to x264farm. Arfter some playing around I got x264farm to work over a network, also for it to work inside Staxrip.

My problem is that x264 keeps crashing on all my computers on the last section of the second pass. All it says is 'x264 as stopped working' and waits for me to close that error window, which is a pain when I leave it overnight only to find that is hasen't even finished one encode hehehe.

I have tried diffrent x264 builds, diffrent os (also crashes in the same spot on XP) a later version of Nice (ever tried to find the program 'Nice' via google, not easy hehehe) but x264 still reports an error

Here is an example of my setting (it is a modified version of Staxrip's Highprofile settings)

Controller.exe --first "--bitrate 907 --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate 907 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "R:\Movies\Ah My Goddess\Ah My Goddess_EncoderOutput.mkv" --avs "R:\Movies\Ah My Goddess\Ah My Goddess.avs"

I am running with Agent Encoding...

I think that all I can add, I've tried sifting through the out-dump.txt files but can't see anything that says what caused the errors.

I would LOVE for this to work as it has increased my encoding speeds from 12fps to about 30-35fps (if I keep an eye on all my computers to close down all the error boxes hehehe)

Sagekilla

8th October 2008, 02:28

Is there any chance you could actually use x264farm for crf 1-pass? I can imagine it wouldn't be much more complex, since it would be more or less identical to 1-pass.

professor_desty_nova

8th October 2008, 08:31

Is there any chance you could actually use x264farm for crf 1-pass? I can imagine it wouldn't be much more complex, since it would be more or less identical to 1-pass.

Are you talking of x264farm-sp - single-pass distributed encoding (http://forum.doom9.org/showthread.php?t=136053) ;)

SpAwN_gUy

8th October 2008, 09:21

My problem is that x264 keeps crashing on all my computers on the last section of the second pass. All it says is 'x264 as stopped working' and waits for me to close that error window, which is a pain when I leave it overnight only to find that is hasen't even finished one encode hehehe.how are you running agents?
from console or from "staxrip-agent"?

if from console the you could see x264's error.. if from satxrip - then you will not see the message...

i've had some "crashings" also... and (as i remember) i've implemented part of code to mine agentGUI to suppress those "XP-Fault-Windows" ... and i think i did the same thing to my delphi-nice-implementation (or not mine.. ).

but i still have "new-vista-error-windows" (haven't checked for new solution yet)

so.. or you kick staxrip-writer to implement "suppressors" (it's just 1 line of code on delphi) .. or you wait for new x264farm (i've told omin this issue.. long time ago).. or i'll look for my "nice.exe" somewhere here ...
"suppressors" must be implemented in a higher level application..
like staxrip-agent -then-> x264farm-agent -then-> nice.exe -then-> x264.exe

Nakaleen

11th October 2008, 10:34

Sorry for the delay. First week back from work so I haven't had time to play around with this again.

how are you running agents? from console or from "staxrip-agent"?

I don't know what you mean by 'Staxrip-agent'. Agent is running on every computer (3) from a console (started from a shortcut).

Each agent has the following 'config.xml' (execept for the <name> and <port> which is diffrent)

<config>
<temp>C:\Agent</temp>
<port from="40720" to="40729"/>
<adhoc controller="50700" agent="50701"/>
<name>Main</name>
<number>2</number>
<x264>x264.exe</x264>
<nice>10</nice>
<base></base>
<agentpipe>0</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

I am using Staxrip with it's Commandline encoding option with the following....

C:\Controller\Controller.exe" --first "--bitrate %video_bitrate% --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate %video_bitrate% --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "%encoder_out_file%" --avs "%source_dir%%target_name%.avs"

Which makes for an example....

"C:\Controller\Controller.exe" --first "--bitrate 920 --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate 920 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "R:\Vandread\Vandread s01-e01\Vandread s01-e01_EncoderOutput.mkv" --avs "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs"

Drive R:\ is a shared drive mapped at every computer to R

if from console the you could see x264's error.. if from satxrip - then you will not see the message...

Looking at the out-dump.txt this is the error reported.

2008-10-11~19:08:13.10 218 2nd pass frames 244 to 273
2008-10-11~19:08:13.18 218 done recieving stats
2008-10-11~19:08:13.20 Testing file "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs": FOUND!
2008-10-11~19:08:13.20 218 agent-based encoding
2008-10-11~19:08:13.21 218 doing the following:
nice -n 10 "x264.exe" --bitrate 890 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 11 --fps 25/1 --pass 3 --stats "C:\Agent\stats 94473f.txt" --seek 244 --frames 30 -o "C:\Agent\output f8b890.mkv" "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs"
2008-10-11~19:08:28.65 218 exited -1073741819
2008-10-11~19:08:28.67 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (-1073741819)\")"
2008-10-11~19:08:28.75 closing socket
2008-10-11~19:08:28.75 socket closed

so.. or you kick staxrip-writer to implement "suppressors" (it's just 1 line of code on delphi) .. or you wait for new x264farm (i've told omin this issue.. long time ago).. or i'll look for my "nice.exe" somewhere here ...
"suppressors" must be implemented in a higher level application..
like staxrip-agent -then-> x264farm-agent -then-> nice.exe -then-> x264.exe

delphi is WAY out of my leage, as well as most programing (haven't done any since C++ 10 years ago hehehe) so there is nothing I can do in at area hehehehe

SpAwN_gUy

13th October 2008, 09:31

Agent is running on every computer (3) from a console (started from a shortcut).

Looking at the out-dump.txt this is the error reported. out-dump.txt will not give you the resuls i mean..
you should look on the console-window, where agent.exe runs.
'cause x264.exe outputs information direct to console. and out-dump.txt does not have this information. while "console window" still has the information.

Nakaleen

2nd December 2008, 23:16

out-dump.txt will not give you the resuls i mean..
you should look on the console-window, where agent.exe runs.
'cause x264.exe outputs information direct to console. and out-dump.txt does not have this information. while "console window" still has the information.

Desided to give this program a try again....

And again it is giving me a head ache from all the keyboard slamming.

3 computers, 3 diffrent setups, a mixture of XP, vista, Intel, amd, laptops etc... no matter what setup they all crashes at some stage. This is very annoying as I set up to encode 5 videos overnight and wake up to see it crashed at the end of the first one.

I copied this from the last render off one of the agents, that caused a crash.

815 2nd pass frames 35531 to 35531
815 done recieving stats Testing file "R:\Read Or Die\Read Or Die S01E02\Read Or Die S01E02.avs": FOUND!
815 agent-based encoding
815 doing the following:
nice -n 10 "x264" --bitrate 917 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 1853 --fps 25/1 --pass 3 --stats "C:\Agent\stats 24b0cf.txt" --seek 35531 --frames 1 -o "C:\Agent\output b6babc.mkv" "R:\Read Or Die\Read Or Die S01E02\Read Or Die S01E02.avs"
avis [info]: 640x480 @ 25.00 fps (35532 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 Cache64
encoded frames: 1/1 (100.0%), 4.02 fps, eta 0:00:00
815 exited -1073741819
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (-1073741819)\")"
closing socket
socket closed
Thread 822 killed on uncaught exception Unix.Unix_error(40, "send", "")

It is not much diffrent then what is in the log files. So I am stumped that is causing the error. Searching for error code -1073741819 for x264 found nothing. Maybe it is one of my x264 setting... but I can't find any info on what setting could crash all my systems.

Ichling

3rd December 2008, 00:00

Hi Nakaleen,
can you give me the exact options that you used to start the controller with? I was running in similar problems with my first tries at x264farm(-sp) and they, in the end, where all related to the x264 settings i've used.

Jabroni

1st February 2009, 19:14

So has anyone done any custom build of x264 thats a bit newer to current SVN release ? I want to make DXVA compatible encodes, and I think I need some parameters that the old x264s doesnt have (or at least the one that works with x264farm).

SpaWnGuY: Are you still working on your megui-mod? I accidently 'upgraded' the megui install, and now it removed all the x264farm stuff, and I cant find the installer of it =\ Would you mind posting it again please?

:thanks:

Pulp Catalyst

12th February 2009, 16:28

hopefully when BD Re-Builder is released someone will design a Farm program like this one
http://www.afterdawn.com/guides/archive/dvd_rebuilder_tutorial_advanced_page_8.cfm

RB-Farm was brilliant and worked great, why you ask, because the way DVD Re-builder pulls apart the DVD,
hopefully BD Re-builder will do the same,

Jabroni

12th April 2009, 19:38

Was wondering if there is going to be any update to this project and/or MeGUI. Would love to distribute all my new encodes with x264farm (im using parameters used on newer x264.exe build, and for some reason they make the x264farm clients crash)

:thanks:

Allbestmessage

15th April 2009, 10:30

There is a command line encoder profile you could use.

Breakthrough

24th July 2009, 18:47

Hello, I have a few questions...

What build of x264 do you all reccommend to use with x264farm? Or at least, what was the final build before the revision to the .stats file was committed?

Also, does the server computer also have the ability to render? Could I theoretically run a controller AND an agent on the same PC?

Thanks!

nst6563

28th July 2009, 19:50

I'm itching to get this working as designed...however it seems I've either got something setup wrong, or something is setup wrong :p

To start, I've got the controller running on a 3Ghz Core2Quad with 6GB ram. I have another machine that is connected via 100mb switch which is a core2duo w/4gb ram.

I've got the config setup for adhoc, and the controller and client see each other - that's not the problem.

The problem I think I've run into is the speed. I was hoping it would be faster than standard encoding with something like ripbot or handbrake...however I get about half the speed compared to those gui-based encoders.

I've tried the controller-based encoding as well as the client-based encoding. I don't remember if I got the client based encoding to work correctly tho... the way I read and figured out was that each client needs access to the sources being encoded right? I made a mapped drive on the machines, and pointed the config to that and it seemed to work ok.

I have 5 machines with a total of 13 cores that I could put towards something like this...I'd love to see it scream though an encode...what could be my limiting factor other than the 100mb switch (trying to get a 1000 but my current 1000 switch shot craps...thanks Dlink)

any plans of something like a gui for the controller setup? Drag a bunch of files to encode to the window (or provide a path with the files to encode), answer some questions and let it go. maybe launch the agents as a windows service?

iplmedia

15th January 2010, 07:17

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.

creamyhorror

15th January 2010, 17:10

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.
Don't forget the single-pass variant: x264farm-sp - single-pass distributed encoding (http://forum.doom9.org/showthread.php?t=136053), which was last modified in 2008 I think.

creamyhorror

18th January 2010, 16:54

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.
Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264 (http://forum.doom9.org/showthread.php?p=1364155#post1364155)

bnshrdr

18th January 2010, 21:18

Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264 (http://forum.doom9.org/showthread.php?p=1364155#post1364155)

I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264. Sure I use it for all of my encodes, but I don't know enough about it to write pretty much a wrapper program to utilize it.

Currently I have 2 computers here at my home encoding the same video file, but some of the ways I am achieving this are either ugly or impractical. Also I don't really know how to truly handle a multi-pass scenario.

I just need info on how to do faster seeking, like using an index. I know of DGAVCDec but I don't know how to put it to good use.

I also need a way to get the frame count of the source before processing the job, but my knowledge of the AVISynth API is 0.

I am also welcoming any ideas about the whole design of this program.

iplmedia

19th January 2010, 22:17

I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264.

If am very familiar with x264 just not how to wrap it! x264farm has many benefits to it by dividing the video into GOP, however, it decompresses it to ray YUV before compression (From what I understand) and this is done in the first pass that only 1 machine performs. Thus greatly slowing down the encoding process. I would love to discuss the possibility of working on this project if you like. Send me an email and we can discuss this.

creamyhorror

20th January 2010, 00:58

Great, hope you guys get together to start a project. You might want to hang out in #x264dev on Freenode IRC to ask the x264 devs questions.

iplmedia

20th January 2010, 21:47

I have worked with and am working with one of the x264 developers already on a different project. They are a great team of developers!

bnshrdr

21st January 2010, 05:36

As much as I'd like to start a little project, I really don't have time. During the school year I always nibble little bits away from my project goals. I'm kind of just a hobby programmer, but being a full time student limits the time I can contribute to any sort of project. Thanks though for the offer and I appreciate your support. I know where I'll come when I have problems :P.

popper

21st January 2010, 14:39

the Very first thing you should perhaps do is totally do away with a fixed x264 version requirement in any code so as not to run up against the code updating being dropped as happened with farm for SO long...

it might be a very good idea to start by parsing a current x264 --fullhelp at runtime and populating your input/output variables from that parse, so that as fullhelp grows over weeks,months or years your code can take advantage of any new options that come along later and still work fine when things change.

as it is today the general feeling is that always using the constant quality option CRF= as your base is the default way to go.

x264farm while a good idea always bugged me in the way it doesnt seperate out all the different parts of the operations code into self contained stand alone scripts, so making it really hard to add new or just different ways of doing the same thing for a given situation,as a plug-in type way to work, so that could be made far cleaner if YOUR really serious about this rework.

you might want to add a new better client/server part but keep all the other bits as they were for instance, or you might want to add DHT for the auto what Hw's available to our IP cluster at this time, and who's ready to take another job, send a DHT message and add and/or act to the 'message based' actions que(s) etc... the options are endless with a little thought and an open self contained fully commented script for one action, or were needed for simpicity a set of related fine grained actions etc.

it should also be considered that current x264 also takes many input file types stand alone now OC so thats a good thing, and sooner or later it will also allow audio parsing/re-encoding to a given container too, so making the whole updated x264farm framework far easier and better later IF YOU JUST PARSE a current x264 fullhelp output or ask DS to provide a better way perhaps if your serious about distributed encoding.

for instance how might a future x264 farm make easiest use of things like this guys http://forum.doom9.org/showthread.php?t=152232 with HPC cluster (750 cores)...

people will always want to use the latest x264 so you had better find an easy way to always auto include/use it with a new x264farm framework without braking the farm....if you want people to use it and give lots of thanks and perhaps even get involved for years to come ;0)

and preferably cross platform usable too, put it infact the whole turnkey package on a generic slax liveCD http://www.slax.org (WITH DEFAULT ready setup pxe server LAN booting AS STANDARD BTW) AND boot any hardware thats in the LAN ANYTIME.

just a few thoughts to consider anyway.

Karyudo

22nd January 2010, 17:53

Good timing! I just started researching how to build some kind of x264 render farm, and here I see new interest after what looks to be a year or more of downtime and benign neglect. Nice!

I can offer almost zip in the technical department, but I do have a home network with four (soon to be five?) computers that could conceivably all run DGDecNV on at least Core2Duo processors, so maybe I'd be able to help with beta testing (when things progress that far, I mean).

Or, if that's no big help, then at least let me add my vote for this idea as a project that I think would be powerful and useful!

iplmedia

22nd January 2010, 22:05

Thank you for your thoughts on this project. I am currently planning out the best architecture for this project trying to utilize the best possible means to accomplish the best possible product. I agree with future compatibility as well as cross-platform. I currently work solely on a mac and would like to see it work on that but understand their are many other users in the *nix and windows world. The concept of a liveCD for the cluster is very familiar to the Linux world (Clusterknoppix etc).

I am considering using OpenMPI and dividing the video into GOP's for distribution so regardless of how many videos you have in queue the entire cluster will be utilized. I realize that their are inherent flaws and considerations but my goal would be very fast cluster detection, minimal bottle-necking and intelligent distribution amongst all available resources with fault-detection and load balancing.

It's a tall order but in my opinion much needed and sought after.

Please continue to post your thoughts/ideas/suggestions.

Thanks

kemuri-_9

23rd January 2010, 03:37

nitpick: OpenMPI is only an implementation of MPI-2, you would be using the MPI-2 API if using OpenMPI as a test implementation.

I had worked somewhat on an MPI-2 patch (was using the MPICH2 implementation myself for windows) nearly a year ago for x264 but stopped eventually when i realized that it would require a fairly large restructure.
I had also planned on GOP distribution among the cluster and was coding along that guideline.
it would probably be easier to implement now utilizing threaded lookahead that was committed some moths ago.

magnatique

24th January 2010, 19:05

when this is up to a testing phase, I would be happy to give the farm some testing grounds as we have core2, quads i7s at the office on our gigabit lan, with some on XP / win7 so it should give a good testing ground!

Phil

iplmedia

24th January 2010, 23:14

Hello all,

Thank you for your input. I am continuing to research and gather the best possible method to accomplish this and your input helps eliminate dead-ends and un-needed wastes of time etc. So please keep them coming. In regards to the MPI-2 implementation I still see it as the best method as well as the GOP. What were some road-blocks you ran into with using MPI and would you suggest a different method or is it still be best method just that it requires a lot of work. I am willing to invest the work/time/investment that it would take to accomplish this I just want to make sure I'm heading in the right direction. Would you be willing to send me the work that you have started already on the x264 MPI implementation? Any thoughts and information you can pass my way would be greatly beneficial. Cheers

Aaron

kemuri-_9

25th January 2010, 01:56

hmm.. having to pick my brain as to the thought processes i had back when i was working on it...

the MPI-2 API has some functionality to determine rank in the operational cluster (along with the size of the cluster) and i used this to define the following:
a node with a rank of zero would indicate the master node - that is the node that is controlling source decoding to send and receive frame data back from slave nodes for writing the final encoded video
otherwise a node has a rank of nonzero and would be considered a slave node.

A) Where to decide where to transfer frames between master and slave nodes
When i was working on this i had issues on where I was going to do this and was a major factor in stopping work.

A1) from working on the threaded lookahead patch,
I would think it would be easiest location for transfer frames from the master node to the slave node.

A2) as for the reverse.... don't recall thinking about that much... thoughts now would have it either in encoder_frame_end or in x264cli

B) what requires transferring:
originally when i was working on this there was only 1 frame type, but now with there being 2, this changes up a little bit.

B1) fdec x264_frame_t
frames that have been slicetype decided, would be transferred from master to slave for encoding.

B2) fenc x264_frame_t?
frames that have been encoded and QP decided, would be transferred from slaves to master.
this is a bit tricky as x264_frame_ts have a lot of data that would be unncessary at the master slave as all it would really want is the resulting x264_picture_t.
though if you only do decide to send back the x264_picture_t you would have to special case the --dump-yuv option in some fashion.

B3) stats
information acquired while processing frames,
involving bit costs and other information used at encoder_close and in stat file writing,
would need to be synced between master and slave node (i didn't get to this part so not sure what the best way of doing it would be).

B4) x264_param_t
the parameters to init the encoding engine with,
I had ambiguous thought patterns on how to handle this,
but i think my last choice was to init the master node with the param_t given on the cli and then send a possibly altered param_t to slaves to init with.

C) Gotchas i remember
C1) data sending.
transferring structures is eloquent in MPI-2 especially when they have pointed to data.
this required creating an extra send buffer and `packing` along data in a struct referred by pointers along with the struct to be unpacked at the receiving side.

C2) GOP distributed encoding.
using a GOP distributed encoding style would allow slave nodes to process on different GOPs with requiring relatively no information from each other.
however, this does have some down sides:
-- a) this could potentially cause the master node to allocate a large amount of data to buffer depending on how transferring of data is done.
that is if choosing to transfer all the GOP in bulk then this would require a buffer large enough to send and recv it all. naturally this could be alleviated by transferring in smaller pieces (e.g. mini GOPs).
-- b) requires unique GOPs to exist. that is the GOP distributed encoding fails when there are not unique IDR initiated GOPs, like in the case of Periodic Intra Refresh (PIR) or Open GOP.

C3) different CPUs and related issues
as x264 works on several different CPUs you would think "oh i could have this cluster over all the different cpus it works on easily", which is sadly untrue.
x264 uses the CPU's cacheline to determine how to pad frame data in x264_frame_t's for optimal performance on that cpu.
there's a few ways you could handle this when clustering between unique cpus:

-- a) alter all the node CPUs to have the same cacheline values as the cpu within the cluster that has the largest cacheline performance value
(this would require using more memory for the cpus that had the cacheline padding increased)

-- b) alter all the node CPUs to have the smallest cacheline values of the cpu within the cluster
(this would require using less memory for the cpus that had a cacheline padding decrease at the cost of some performance)

-- c) repad the data to the receiving cpu's cacheline at send/recv ing of data
(but seriously this would be more effort than it's worth so i don't recommend it)

tl;dr:
lots of things to worry about and the finesse required to do it was a major factor in me stopping work on it.
(didn't help i was in my last semester of college while working on this and wanted to graduate more than work on it)

overall i didn't see a particular issue in using MPI-2, as the surrounding issues are not transfer protocol specific but sync specific.
MPI-2 does offer a variety of sync/async API to help out on the issue of synchronizing, so i would continue recommend using it as the implementation to do this cluster support work.

I would be willing to share the code i have, but as it is incredibly old and imo bleh now,
I would only use it now as reference code on how to do something rather than actually trying to apply it to current repository and working from there.

popper

25th January 2010, 02:02

for more input you might also look to spending some idle time on http://gogloom.com/client2/index2?mainCHAT=1&network=FreeNode&channel=%23x264 to.

i still think using a generic slax is a good way to go , incase i didnt make it clear you download a generic slax livecd, boot it, select the setup pxe server LAN booting option, and instantly you have a lan bootable slax, go over to any other PC on your LAN AND boot it into pxe mode and it will be running that exact same livecd you just booted with every module that you happen to have put on the slax livecd....

slax is the only generic liveCD/liveUSB distro i know of that comes with this pre-setup pxe setup and ready to use out the box, anywere.

OC its werth also mentioning that slax creator Tomas M is also the guy that gave you the liveCD/liveUSB scripts that everyone uses today OC

http://www.slax.org/forum.php

you could boot a native liveCD/liveUSB or just use a Vbox and liveCD/liveUSB that for background client/server LAN and Vlan use etc with all the farm and related apps in module form you require on there for on the fly mouting/loading , no messing about.

popper

25th January 2010, 04:34

"I currently work solely on a mac and would like to see it work on that but understand there are many other users in the *nix and windows world."

thats not such a big problem though,(PPC G4/5 OR X86 BTW?) all it takes is picking the right tools for your potential client/server tunnel apps to the cluster you might use....

using 'rebol view' and 'rebol core' for the GUI scripting and client/server tcp:ip udp and multicast cross platform apps today. http://re-bol.com/rebol.html http://www.rebol.com/download.html is one option to consider if your into cross platform scripting control and related apps for instance.

heres a self contained multicast rebol view script that shows you how to make and use a simple multicast whiteboard for realtime data drawing and passing ,and more.
http://www.rebolfrance.info/org/articles/multicast/sources.zip
google translate his french page if you like
http://www.rebolfrance.info/org/articles/multicast/multicast.html

you want a tiny self contained pro webserver at the heart of this cluster(s) then include The Cheyenne Web Server
http://www.rebol.com/news/cheyenne.html written in rebol, or want a cache proxy for the potential message parsing etc then
use this as your basic outline and change it as required.
http://www.rebol.org/view-script.r?script=proxy.r

perhaps you want to use java as well for part of the new farm, then theres a DHT and building a multicast protocol on top of Bamboo paper by Marcel Dischinger outlining how you might include that http://bamboo-dht.org/tutorial.html

want a basic self contained multicast ipv4 overlay tunnel for connecting remote farm sections then consider the java 'mtunnel' , alas he took the page down now but its still available...

as i said the cross platform options are many if you just consider your requirements and can figure out the best and various ways to chop up the core HD content.

how about you draw up a basic visual flow chart jpg outlining your current thinking as to all the different sections that need laying out and posting that as a start for people to see and comment on any obvious potential problems ?, nothing fancy, just a quick visual aid.

iplmedia

25th January 2010, 04:49

Thank you all for your excellent input and interest in this thread. I will consider compiling a more detail brain mapping of my thoughts for critique here. Of course I need to find the free time. At any rate, Great suggestions. Keep them coming.

Aaron

popper

25th January 2010, 05:54

OC allowing for the inclusion of a cheap freeNAS http://freenas.org/freenas or 3 :) and simple Ethernet iSCSI setup etc is always a good thing to take into account for these types of mass data storage LAN Processing Farm jobs too as its quicker than other protocols over ethernet.

http://www.microsoft.com/downloads/details.aspx?familyid=12CB3C1A-15D6-4585-B385-BEFD1319F825&displaylang=en

http://freenas.org/documentation:setup_and_user_guide:iscsi_target

http://frrl.wordpress.com/2009/08/25/fun-with-freenas-iscsi-when-a-local-disk-is-not-local/
http://www.windowsnetworking.com/articles_tutorials/Options-creating-software-based-Windows-iSCSI-Server.html

SpAwN_gUy

26th January 2010, 10:42

hi guys :) .... i'm not totally back, but anyway :)

i just thought about "how's it going on my fav x264farm?" and, viola, the thing are moving :) (i thought about "fixing" farm to be able to work with newer x264 builds. but that was just a thought :( )
i'm (as a x264farm support-unit) currently switched my job(i'm a php-dev now), so now i don't have access to a variety of HW, but i do have some ideas (if someone)... and i'll keep monitoring.. 'cause i do have alot of AVC-HD-fromMyCam-Footage to encode.. and his is a real pain in the cpu.

greets from Belarus.

popper

23rd February 2010, 13:53

well that interest didnt last very long it seems, given the TOTAL LACK of any code updates or even progress posts so far...

the intent to make something better is only as good as the coding work you actually put in and release.

Yiu

4th November 2011, 05:59

Hey Spawn Guy.
I'm thinking of getting a farm server with a number of AMD Opterons running Ubuntu/WinServer2008 and controlling it through my Win7 (x64) desktop. The problem is, I have no idea where to start when it comes to setting up the server, x264farm, and all that.
I've been using x264 for a few years now, and I've perfected a rather nifty configuration but I only get 0.45FPS on my Intel i5 750 @ 3GHz setup so I pretty much NEED the server otherwise I spend days on a single conversion.
I know I should get the system built, have a Linux/WinS version of x264 on the machine, then figure out how to set up your x264farm/MeGUI mod to work with it but I'm clueless and to be honest, scared of spending thousands on a server which may go to waste if I don't figure things out. :L

Can you give any pointers as to where to start?

SpAwN_gUy

4th November 2011, 11:17

Yiu

4th November 2011, 11:25

the good thing is, that it's not so hard to set things up on a winPC.
the bad thing is... x264farm is INCOMPATIBLE with current x264 builds :(
i think the latest version supported is.. ~r900 or so..
so until someone update x264farm code to support parsing of current .stats file... i guess you'd have to stick to 1PC encoding :(

Thanks for getting back so quickly. I don't know how stupid this sounds right now but would it be possible to run x64 x264 in Win Server?
I'm probably getting the i7 3960X and OCing to ~5GHz but that's still not good enough for the amount of time it'll take.

If not,does the *nix variant of x264 support the same command-line arguments? I'm pretty sure 48 logical cores running at 2GHz will be worth getting the workhorse anyway. If possible, I'd like to raise the clock speed on the Opterons too.

48 cores X 2 GHz = 96 operations /s
V.S.
6 cores X 5 GHz X 2 (HyperThreading) = 60 operations /s

Then again, the i5 2500 beats the AMD 8 Core bulldozers so it's iffy anyway...

Sorry to hear about the compatibility problems. :(
I had been looking forward to setting up x264farm for a few months but I'm not sure if my settings file will work with the versions of x264 your application supports. If it does then it shouldn't matter, but if it's not then I'll have to build a multi-rack system instead of networking lol.

4th November 2011, 12:39

Thanks for getting back so quickly. I don't know how stupid this sounds right now but would it be possible to run x64 x264 in Win Server?

Yes.

I'm probably getting the i7 3960X and OCing to ~5GHz but that's still not good enough for the amount of time it'll take.

What kind of encoding settings and filters are you tormenting yourself with?

does the *nix variant of x264 support the same command-line arguments?

Yes. But there's no native AviSynth support on other operating systems than Windows. You need Wine and avs2yuv for that.

48 cores X 2 GHz = 96 operations /s
V.S.
6 cores X 5 GHz X 2 (HyperThreading) = 60 operations /s

HyperThreading gain is x1.5 at best.

Sorry to hear about the compatibility problems. :(
I had been looking forward to setting up x264farm for a few months but I'm not sure if my settings file will work with the versions of x264 your application supports.

I'd advice against using such an old x264 version even if it supported all your settings.

Do you need to finish one encode as fast as possible, or could you simply encode separate works in parallel?

Yiu

4th November 2011, 13:01

HyperThreading gain is x1.5 at best.
Yet the 4 core i5 2500k thwarts the AMD FX 8 core processor, let alone the 2600k, 2700k, or the awaited 6 core 3960X.

What kind of encoding settings and filters are you tormenting yourself with?
A near lossless setting. No filters. Straight BluRay/H264 to x264 compressed video files. Retains approximately 99.8 per cent visual quality for animation and approximately 95 per cent visual quality for live action. It's fine-tuned for anime essentially.
Can take up to 8 days to compress a legal back-up of one of my average BluRays. I'm running out of space, so spending days getting near lossless quality AND rather small file sizes is worth it.

Do you need to finish one encode as fast as possible, or could you simply encode separate works in parallel?
Well when one whole film, lets use Transformers 2 as an example, can take up 16 days and still comes out 12GB (with DTS 5.1 in 3 languages and many other audio streams), I'd love to split it up on several machines. It's not about NEEDING speed (I.E.: Realtime), it's about not waiting for days with 100% CPU usage. If I could get encoding time for film down to 1 day (12 hours per pass, excluding audio), I'd be happy.

With this same settings file, I edited a little clip here (http://www.mediafire.com/?monpic74ip7hn9a) which is completely original (well, aside from the actual DTS Logo). It's 3.27MB and is in 1920x1080 @ 23.976FPS with FLAC 16 bit 48KHz 5.1 surround audio.

The video data for that file is only 516KB. Granted there's only 1 Keyframe and limited chroma motion, but the point is for cel and digi-cel animation it's a distribution foothold. I, by the way, am working on an animated series which is what I plan to use this for.

It's starting to look like there's no good solution.
Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?

vivan

4th November 2011, 13:37

Yet the 4 core i5 2500k thwarts the AMD FX 8 core processorAMD FX also doesn't have real 8 cores. So it's not HT win, it's AMD fail ;)

Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?http://en.wikipedia.org/wiki/Windows_Server_2008_R2#Editions - it says that it support from 1 to 64 physical CPUs depending on edition.

Atak_Snajpera

4th November 2011, 13:44

AMD FX also doesn't have real 8 cores. So it's not HT win, it's AMD fail
AMD FX has 8 integer units and only 4 FPU. In fact it looks more like 4 core cpu with doubled number of integer units.

4th November 2011, 14:22

A near lossless setting. No filters. Straight BluRay/H264 to x264 compressed video files. Retains approximately 99.8 per cent visual quality for animation and approximately 95 per cent visual quality for live action. It's fine-tuned for anime essentially.
Can take up to 8 days to compress a legal back-up of one of my average BluRays. I'm running out of space, so spending days getting near lossless quality AND rather small file sizes is worth it.

0.5 fps on an i5 750 sounds like a lot of time wasted for very minimal gain. You are using placebo-level settings which probably give you 1-5 % more compression than --preset slower would, while taking 5 times longer to encode.

Or put in a different way, you can get the same quality by using:

A. --preset placebo settings that give you a 5000 MB file in 48 hours

B. --preset slow/slower settings that give you a 5250 MB file in 10 hours.

Is that 250 MB really worth it? At current HDD prices it costs less than 2 cents!

I seriously suggest spending the money on HDD space instead of CPU power. With the amount of money you'd spend on server hardware, you can get enough HDD space that you can make 1:1 copies of all the Blu-rays you'll ever own!

Well when one whole film, lets use Transformers 2 as an example, can take up 16 days

Ok, you're not using placebo but insane settings?! What did you put in that command line?

If I could get encoding time for film down to 1 day (12 hours per pass, excluding audio)

Use CRF instead of 2-pass. Or if you absolutely need to control the file size (why?), use fast first pass.

Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?

2008 R2 standard edition supports 4 physical CPUs: http://en.wikipedia.org/wiki/Windows_Server_2008_R2#Editions

Yiu

6th November 2011, 04:40

Is that 250 MB really worth it? At current HDD prices it costs less than 2 cents!Yeah it's really worth it. Esspecially when I'm already running low with 6TB of data already. I need another 2TB Caviar Black but the prices over in the UK just spiked. Prices went up nearly 50% on HDDs, but SSDs haven't gone up. About $225 USD for 1 WD CB 2TB.

I have a lot of data, and it's going to be expensive when I decide to raid all my drives.

Ok, you're not using placebo but insane settings?! What did you put in that command line?I spent months looking into the x264 settings. Anything to reduce size without reducing quality I put in. If it ever compromised quality, I'd bump something else up that bumped up analysis and vector encoding.

Use CRF instead of 2-pass. Or if you absolutely need to control the file size (why?), use fast first pass.
Well I might as well be setting CQ and setting it to 2-4. But they're really selective settings which I've looked at and used, CQ doesn't give as good visual benefits, in my opinion.

2008 R2 standard edition supports 4 physical CPUs.
Yeah as I thought, my information is kinda behind. Since x64 x264 can run on WinServer on 4 CPUs on just standard I'll have no problem starting out. By the time I can afford more racks I'll be able to afford Ultimate.

EDIT:
I'll probably need HPC/Advanced for the RAM headroom. Unless x264 doesn't actually use that much memory? I'd assume it'd use more to speed up encoding time but I don't know.

Atak_Snajpera

6th November 2011, 11:26

Yiu

6th November 2011, 15:26

did you calculate how much money you waste for electricity? under havy load your pc probably eats about 250-300 W. i would rather buy extra hdd. if you compare average salary from uk and from eastern europe you will quickly realize that prices are not that bad. we from east earn four times less than you from uk. please stop wining.

I'm from Michigan USA. I don't qualify for a lot of free shit other people do over here. Look at average salary all you want, I'm an 19 year old student. And in the end, I'm using less HDD space, buying less HDDs and thus saving the amount of power I use in the long run. (Keep reading it gets better.)
That's why I'm looking to buy a workhorse, so that the 1000+ watts I'm using* can be used to finish the video compression, rather than on my regular desktop system which has disk drives, graphics cards, external sound cards, and a whole lot more than the workhorse will ever have. More energy can go straight to RAM and more processing power, I actually save a lot of money getting a video compressed with a workhorse.

*The main point is, I can only have 1 of the computers on at a time to actually break even. But, if the workhorse can do the work faster than the desktop and either computer would be more or less useless while the CPUs maxed out, then I'm saving time. Swapping the money I save for more time.

It's extremely useful for distributing my own content as well. In the UK your upload speed is limited to 1/8th your download speed on most connections. I only get a 1MB/s download rate (8Mbits) so a 128KB/s upload rate can restrict deadlines and file transfers for me. In all honesty if I had a monster internet connection and more money to throw at HDDs, I'd be far less worried about it but with the rest of the world still on slower internet connections it saves time not only for me but thousands of other people.

Atak_Snajpera

6th November 2011, 15:56

Still simple remuxing from HD-DVD/BD/AVCHD is better because:

1) Quality is 100% the same with source
2) Speed is only limited by your HDD/SSD
3) lower electricity bills

and now what you get:
1) Smaller size
2) Quality will never be at 100% with source
3) Ridiculous encoding speed even at good system (0.45 FPS!?!!)
4) A lot higher electricity bills due to have load on PC.

I see no logic here.

Yiu

6th November 2011, 16:58

You have noooo idea how much media I have. With minimal quality loss and most BluRays getting shrunk to 10GB including specials and extras, it's worth it.

I would need about 6TB of space just for videos if I did direct copying. Then I've got to factor in my other 4TB of ever expanding media types. As I said, I'm gagging on 6TB of total space already. If I could, I'd get a media server but that would eat up tons of electricity, and all for what? The ability to access all my data at the once? Plus, how much more data will I eat up just over the next 12 months?
I only started digitizing my BluRay and CD and other mediums in February. I've not even been doing this for 12 months and I'm at 6TB.
I was looking to not buy anymore HDDs until next year last month when I bought my Caviar Black but things aren't going as well as I planned in terms of space.

I'm already buying the Desktop, Workhorse and a camera for college all at the same time with my school loan and grant. I have no money to buy any HDDs. I'll need to find a job or get my business up to a decent pay-grade to be able to keep up with my current HDD purchase intervals.

jasonwc

4th December 2011, 18:16

Your arguments really don't make any sense. I'm going to prove it by breaking down the cost of your alternatives. First let's start with your hardware choices.

You purchased the $1,000 Core-i7-3960K processor and then overclocked it to 5 Ghz despite the fact that the Core-i7-3930K could have been overclocked to the same rate and would have performed within 1-2% for a savings of $400. And I'm betting you spent at least $300 on the motherboard. So that's $1,300 on two components.

Compare that to a Core i7 2600k ($280 at Microcenter) and a decent 1155 board ($150) for $430. You could then take the extra $870 and put it into drive space. Even at the outrageous prices you can buy FIVE 2 TB drives providing 10 TB of storage for that price (Best Buy has a 2 TB WD20EARX for $170).

Or, if you had reacted to the impending price hike, you could have bought a bunch of drives at $80 or less in October like I did. . .

I have a 4U Norco RPC-4224 file server with 24 hotswap bays (currently using 9 bays - 16 TB useable, 2 TB parity) and I only spent $700 to put together the server. I buy drives as necessary. I acquired two Hitachi 5K3000 2 TB drives for $60/each before the hard drive price surge, and then two Western Digital WD20EARX drives from Best Buy for $80 afterwards. That's an additional 10 TB (9.1 TiB) of storage for $280. Even at around 35-40 GB/movie and accounting for TB --> TiB conversion, you could store 250+ movies on those drives.

But if you really insist on encoding rather than remuxing, you should be getting 5-6 fps in second pass or more with your 5 Ghz hexa-core setup.

You're spending 5-10x more encoding time than if you had used placebo settings for a ~1% efficiency improvement, and in so doing are wasting MANY Kwh of electricity. You're likely burning 300 watts/hr. At 10c/ KwH or more, you would be much better off simply using a lower quality setting and slightly increasing bitrate.

For example, you say that your encodes take 8 days. With a 5 Ghz OC on a 130 TDP processor and a demanding new platform, you have to be using at least 300 watts/hr (remember your PSU is only 80-90% efficient at best).

So that's .3kW x 8 x 24 hours/day = 57.6 KwH

Assume an average price of 10 cents/kwH and you're paying $5.76 to encode a movie. You also are spending money to store that movie. You said each encode was 10 GB. Assume a price of $200/2TB hard drive. That's 10 cents/GB.

So, 10 GB x $0.10/GB = $1.00 for HDD space
$1.00 (HDD space) + $5.76 for electricity = $6.76 total

Now, consider a "placebo" encode which on your system should be at least 5x faster (1.6 days).

.3 kW x 1.6 x 24 = 11.52 KwH ($1.15)

So, let's say by altering your settings from insane to placebo you had to store an extra 1 GB per movie (probably more like 100-250 MB). That's only an extra $0.10.

$1.10 for storage (11 GB) + $1.15 for electricity = $2.25 total

In contrast, if you remuxed you would use perhaps 40 GB MAX. Even at a price of $200 for a 2 TB drive, you're only spending 10 cents/ GB. So, the remux would only cost $4.00 to store (and I'm greatly exaggerating the remux size to be conservative. Most of my remuxes are 20-30 GB - so $2-3).

So, you're looking at $2-4 for a remux depending on whether you include extras and the size of the movie, even with absurd hard drive prices.

Since I bought a bunch of drives before the prices spiked, I'm only paying 3-4 cents/GB so my remuxes cost $0.60-$1.20 to store (20-30 GB MKVs, no extras).

So, here's the cost we get per movie from highest to lowest:

1) Encoding with Insane settings: $6.76/movie
2) Remux with inflated drive prices: $4.00/movie MAX; more realistically $3.00 or less
3) Encoding with Placebo settings: $2.25/movie

BUT those prices assume your current hardware as a sunk cost. Had you purchased a Core i7 2600K system instead you would have had an additional $870 to play with. Say you took the $870 and purchased five of the 2 TB Western Digital drives I suggested. That would provide 9.1 TiB or 9,313 GiB of storage.

So, you could have had a Core i7 2600K system AND 9.1 TiB of additional storage enough for 846 slightly larger encodes or ~250 remuxes.

And with a 2600K overclocked to 5 Ghz, you would lose perhaps 30% in encoding speed. So... you could spend 2 days encoding a movie at placebo settings or 12 hours for equivalent quality with a 10% increase in bitrate.

So, if you went back and made different hardware choices, choice #3 (encoding with placebo settings) would likely cost you about $1/movie (2600k has better performance per watt) while choice #2 (remux) would effectively be FREE for the first year or two (not sure how fast you collect digital media).

The choices you have made make no economic sense. Now, if you just really enjoy having a ridiculously overpowered top-of-the-line hexacore Core i7 system, that's fine. But don't pretend that you're getting "value" from overclocking a $1,000 extreme processor or spending 8 days to encode a movie. You're not. Also, you don't need a 7200 RPM drive for bulk media storage. Get the cheapest 5400 RPM drive that's reliable.

EDIT: I misread your prior post. I thought you actually live in Michigan. It appears you live in Scotland, so I'm not sure what drive prices are there.