x264farm: distributed encoding [Archive] - Page 3

TriHFH

2nd December 2006, 20:36

Here is the results of the debug lines.

2006-12-01~06:33:05.03 DEBUG: Running command "avs2yuv -frames 1 -raw -o NUL "pb213.avs" 2>&1"
2006-12-01~06:33:05.04 DEBUG: avs2yuv.exe in current directory (C:\x264farm\controller)? true
2006-12-01~06:33:05.04 DEBUG: opening avs2yuv
2006-12-01~06:33:07.42 DEBUG: avs2yuv responded: "pb213.avs: 1280x720, 24000/1001 fps, 61376 frames\013\n"
2006-12-01~06:33:07.43 DEBUG: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error
2006-12-01~06:33:09.03 ERROR: avs2yuv exited with error 1; AVS file "pb213.avs" probably has an error

Now I tried the controller on another pc, and it got a little farther this time, but it still stopped. I tried this with a different avs file along with the simple one. The simple one gave the same error and stopped.

2006-12-01~23:46:05.00 x264farm version 1.04-153
2006-12-01~23:46:05.00 Using config file ".\config.xml"
2006-12-01~23:46:05.00 CONFIG:
2006-12-01~23:46:05.00 temp_dir: "C:\\x264farm\\temp\\controller"
2006-12-01~23:46:05.00 agent_list 1:
2006-12-01~23:46:05.00 "Local CPU 1" = 127.0.0.1 50700-50703
2006-12-01~23:46:05.00 "Laptop 1" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Katie 1" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.00 agent_list 2:
2006-12-01~23:46:05.00 "Local CPU 1" = 127.0.0.1 50700-50703
2006-12-01~23:46:05.00 "Laptop 1" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Laptop 2" = 192.168.0.104 50700-50705
2006-12-01~23:46:05.00 "Katie 1" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.00 "Katie 2" = 192.168.0.103 50700-50705
2006-12-01~23:46:05.01 DEBUG: Running command "avs2yuv -frames 1 -raw -o NUL "seoul.avs" 2>&1"
2006-12-01~23:46:05.01 DEBUG: avs2yuv.exe in current directory (C:\x264farm\controller)? true
2006-12-01~23:46:05.01 DEBUG: opening avs2yuv
2006-12-01~23:46:10.71 DEBUG: avs2yuv responded: "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013\n"
2006-12-01~23:46:16.40 GOT NORMAL INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:22.15 GOT FIRST INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:28.20 GOT FAST INFO "seoul.avs: 1280x720, 30000/1001 fps, 208097 frames\013"
2006-12-01~23:46:28.21 INFO:
2006-12-01~23:46:28.23 i_res_x = 1280
2006-12-01~23:46:28.29 i_res_y = 720
2006-12-01~23:46:28.57 i_fps_n = 30000
2006-12-01~23:46:28.95 i_fps_d = 1001
2006-12-01~23:46:29.14 i_fps_f = 29.970030
2006-12-01~23:46:29.14 i_num_frames = 208097
2006-12-01~23:46:29.14 i_bytes_y = 921600
2006-12-01~23:46:29.14 i_bytes_uv = 230400
2006-12-01~23:46:29.14 i_bytes_per_frame = 1382400
2006-12-01~23:46:29.14 multiplier for kbps to fbits = 0.029970
2006-12-01~23:46:29.14 fast_res_x = 1280
2006-12-01~23:46:29.14 fast_res_y = 720
2006-12-01~23:46:29.14 IF THIS WAS RELEASED, YELL AT OMION!

Here is the seoul.avs

DGDecode_mpeg2source("G:\captured\Metallica\seoul.d2v",info=3)
ColorMatrix(hints=true)
edeintted = last.AssumeBFF().SeparateFields().SelectEven().EEDI2(field=-1)
TDeint(order=0,edeint=edeintted)
TTempSmooth(lmdiff=1,cmdiff=2,strength=1,scthresh=8)
RemoveGrain(5)
crop( 0, 0, 0, -8)

LanczosResize(1280,720) # Lanczos (Sharp)
#denoise

omion

2nd December 2006, 21:51

Here is the results of the debug lines.
That's odd. It looks like everything works fine, but avs2yuv exits with an error anyway. I'll see if ignoring the output will work...

Now I tried the controller on another pc, and it got a little farther this time, but it still stopped. I tried this with a different avs file along with the simple one. The simple one gave the same error and stopped.

Use the release version (at the beginning of the thread) of x264farm. The debug one I posted was based on my current programming version, which didn't work properly. I just made it exit before it got to the part that broke.

TriHFH

3rd December 2006, 01:53

Thank you :)
Thank you for spending all this time working on it and troubleshooting my problem :)

omion

3rd December 2006, 05:35

Try out this controller (http://omion.dyndns.org/x264farm/x264farm-1.04+2_controller-only.rar); it should actually encode. If it works, I'll put it into the next release. Or maybe I'll just make it the next release.

BTW: what are your system specs, anyway?

OverlordQ

3rd December 2006, 11:31

There a guide for figuring out command line switches for idiots like me :D

omion

3rd December 2006, 21:43

There a guide for figuring out command line switches for idiots like me :D

Well, there's the HTML file (http://omion.dyndns.org/x264farm/x264farm.html), which is included with the releases. However, it's probably a bit too much, so here's a crash-course in command syntax:

The agents don't need any switches, and get all of their options through the config files.
The basic controller command-line is:
conrtoller --first "<first_pass_settings>" --second "<second_pass_settings>" -B <BITRATE>kbps -i "input.avs" -o "output.mkv"

There is, however, a whole slew of x264 options that you shouldn't use in the --first and --second strings. Don't include the stats file, input or output files, or anything which has an x264farm equivalent. If you have any questions about whether or not an option can be included in the --first or --second strings, just post here.

(I just noticed that I don't exactly make clear in the HTML file which options to leave out. I'll have to change that...)

AVmaniac

5th December 2006, 14:09

You really did a great job with creating x264farm.
I am actually loving it.
It is giving my x264 encoding jobs the boost i was waiting for.

But I've one question ... is it possible to use it with Sharktooth's
builds of x264? Because i'm missing the aq-strength flag in your builds.

:thanks: again

omion

5th December 2006, 18:42

You really did a great job with creating x264farm.
I am actually loving it.
It is giving my x264 encoding jobs the boost i was waiting for.
Thanks!

But I've one question ... is it possible to use it with Sharktooth's
builds of x264? Because i'm missing the aq-strength flag in your builds.
Sharktooth's build will not work with controller-based encoding (the default) However, it should work just fine if you give all the computers access to the source files and use agent-based encoding.

The problem with everybody else's builds is that they pipe input in text-mode, whereas I'm sending binary data to x264. Using text-mode basically makes piping data useless for x264 because it will stop encoding if it encounters a certain byte (0x1A).

I would also like Sharktooth's builds to work, because then I wouldn't have to spend time building it :D . I pointed out the problem in this thread (http://forum.doom9.org/showthread.php?t=116700), but as far as I know it hasn't been fixed. If you test out the code I have in the first post on that page and it encodes more than 8 frames, then the build should work with x264farm.

kenji123

6th December 2006, 16:38

love the work you've done omion. just tried it out this morning, and it worked flawlessly. hopefully the issue with Sharktooth's build will be resolved soon.

kenji123

9th December 2006, 15:29

That's odd. It looks like everything works fine, but avs2yuv exits with an error anyway. I'll see if ignoring the output will work...

ERROR: avs2yuv exited with error 1; AVS file "Track1.avs" probably has an error

During the debugging of the x264farm GUI i'm creating, just like TriHFH, i also got this error. it seems avs2yuv must be in the same directory as the AVS file being used.

In the next release could you add a parameter so that the avs2yuv location could be specified, omion.

omion

9th December 2006, 17:26

It will find avs2yuv anywhere on the path (I, personally, don't have it in the AVS directory). But I'll add something to check in the controller's directory, too.

The next version may take a while. I'm adding compression to the controller-based encoding, and it's going to need a lot of debugging.

SpAwN_gUy

11th December 2006, 16:23

So, i've Downloaded everything.. and installed
now i have about 10 PCs Under my control.. COOL.. Yeppie...
After making some preperations.. the Installation of Agent takes about... 30Seconds %) ... and this is WITH agent-based encoding..

at first i've tried to encode 5mins clip.. just for test..
1st went GOOD .. and really Fast..
the second file became Broken :(( (second encode of the same file)
and the third went great again...

and the i've started 4,5Hours encode.. and went Home(that was Friday)..

...Skip...
2006-12-08~23:20:28.87 pc17 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-08~23:20:28.87 pc17 2 close_connection
2006-12-08~23:20:28.89 pc17 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-08~23:20:28.89 pc17 2 close_connection

...SKIP...

2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection
2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection
2006-12-08~23:20:28.90 pc17 2 process_job failed with ("unknown error #0",socket,); putting job back
2006-12-08~23:20:28.90 pc17 2 close_connection

....SKiP... and then Again
2006-12-11~08:48:23.14 pc6 3 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-11~08:48:23.14 pc6 1 close_connection
2006-12-11~08:48:23.14 pc6 2 process_job failed with ("The operation completed successfully.\013\n",socket,); putting job back
2006-12-11~08:48:23.14 pc6 3 close_connection

and then My CONTROLLER went OUT of Memory(i think).. and Explorer made some exceptions..
and today i've made a reboot.

BUT.. Agents are STILL working.. and STILL encoding something..

ok.. few Questions:
Can i RESUME Encoding? (i still have batch file.. and still have TEMP files on controller) and i'm doing agent-based encoding
i'm using controller from your last post(where you posted it ;) )

How can i start AGENT in "TRAY-Mode"... or just as Service(it would be nice.. really) ... well that is not a question..

and i'm currently thinking of writing 2 DB clients (i'm a Delphi programmer) .. so it would automate Agent Installation and Controller-Starter with GUI ... (it's not that hard.. to me.. i just need to finish my Job..)

omion

12th December 2006, 01:13

So, i've Downloaded everything.. and installed
now i have about 10 PCs Under my control.. COOL.. Yeppie...
After making some preperations.. the Installation of Agent takes about... 30Seconds %) ... and this is WITH agent-based encoding..

at first i've tried to encode 5mins clip.. just for test..
1st went GOOD .. and really Fast..
the second file became Broken :(( (second encode of the same file)

How did the file break?

and the third went great again...

and the i've started 4,5Hours encode.. and went Home(that was Friday)..

<CODE/>
and then My CONTROLLER went OUT of Memory(i think).. and Explorer made some exceptions..
and today i've made a reboot.

Hmm. Are all of the computers running agent-based encoding? They all have access to the source files, x264, and avisynth? It's odd that the controller would run out of memory with agent-based encoding.

(it's also a bit odd that "socket" was throwing the errors. Normally "send" or "recv" throw them. Check your controller config.xml file for any incorrect IP addresses. It could be that one of them is specified incorrectly, but the short encode didn't distribute anything to them so it worked fine...)

BUT.. Agents are STILL working.. and STILL encoding something..

One of the issues with agent-based encoding is that the controller can't stop the agent from encoding if it dies. The agents will stop after the current encode.

ok.. few Questions:
Can i RESUME Encoding? (i still have batch file.. and still have TEMP files on controller) and i'm doing agent-based encoding
i'm using controller from your last post(where you posted it ;) )

Yup. As long as you use the exact same settings, it will start up where it left off.

If you don't remember the settings you used, look for some files called "settings.txt" in the temp directory. There is one file in the AVS's temp directory, and one file in each of the pass sub-directories. The ones in the pass directories give the settings of that pass. (I hope that makes sense)

How can i start AGENT in "TRAY-Mode"... or just as Service(it would be nice.. really) ... well that is not a question..

Firedaemon (http://www.firedaemon.com/)

and i'm currently thinking of writing 2 DB clients (i'm a Delphi programmer) .. so it would automate Agent Installation and Controller-Starter with GUI ... (it's not that hard.. to me.. i just need to finish my Job..)
Sounds interesting!

One more question:
Are any of your computers running Windows 95/98/Me? They probably won't work with x264farm at all.

SpAwN_gUy

12th December 2006, 09:37

How did the file break?
well. i'm not sure.. i haven't saved it :( but the MPC ended with "can not render file"
Hmm. Are all of the computers running agent-based encoding? They all have access to the source files, x264, and avisynth? It's odd that the controller would run out of memory with agent-based encoding.
Well.. it did it again this Night.. Controller was "Alive" till Half an Hour Ago.. and the same thing started at:
>> 2006-12-12~00:32:47.54 pc661 3 process_job failed with "End_of_file"; putting job back
and it was working since:
2006-12-11~18:16:25.00 x264farm version 1.04-153

today i was using a "Final build" instead of "maybe the next release"

and yeah the IPs - are good(made it from PING). xml-file has some "not started agens IPs" .. so when encode starts not all the agents are working.. and then controller hops all-new agents in line :) ..

Yup. As long as you use the exact same settings, it will start up where it left off.
yeah i've remembered it.. i read it somewhere here.. and it did actuallty continued...
Firedaemon (http://www.firedaemon.com/)
thanks :) .. i was thinking about some "AppToService" programm
One more question:
Are any of your computers running Windows 95/98/Me? They probably won't work with x264farm at all.
Of Cource :) .. a first i was thinking to run an Agent on Vista_Final.. but then i couldn't figure out the IP, because PING didn't answered anything.

omion

13th December 2006, 02:48

Released 1.05.

Firstly, note that you need not overwrite your config.xml files for this update, although I did add a few features which need an updated config.xml file in order to use them.

FPS chart
I added a little chart that prints out occasionally during the 2nd pass, indicating the speed of each agent. I could have done the same for the first pass, but it would have been wildly inaccurate. It looks like this:
Agent | Now | GOPs | Frames | FPS |
Local 1 | 340 | 69 | 7749 | 2.69 |
Local 2 | 342 | 67 | 7850 | 2.82 |
Linux 1 | 341 | 22 | 2184 | 2.48 |
Linux 2 | 339 | 17 | 2236 | 2.52 |
Mac 1 | 329 | 52 | 6456 | 2.55 |
Mac 2 | x | 54 | 6334 | 2.39 |
Mac 3 | 330 | 56 | 6453 | 2.57 |
16.63 FPS total on 39262 / 40492 frames (96%)
ETA: 1:13"Agent" is the name of the agent
"Now" is the current job number (from 0 to the number of scenes) "x" means the agent is not currently working. Since the chart is printed out when an agent checks out a job, at least one agent will always have an "x".
"GOPs" is how many scenes the agent has completed
"Frames" is how many frames the agent has completed
"FPS" is the speed of the agent (based on the times when the agent was actually working, not the total time the controller was working)
The final FPS line is the actual frames per second, and will probably be less than the sum of the previous FPS columns.

Compression
The main change is a beta-quality compression setting, primarily for testing purposes. In order to turn it on, add the following to you agent's config.xml:
<compression>
<type>1</type>
<type>0</type>
</compression>
If you are using the supplied config.xml file, make sure type 1 is above type 0, in order to use compression.

If you do use compression, I have a few requests:

After every encode, check the agent's directory for files with names like "MD5 error 01 (ABCDEF).txt". If there are any there, tell me!
On the controller's side, there should be a bunch of lines like this:01: 34.6% compression (11272631295 / 32551191552 bytes) Post the last line in the encode that looks like this, along with the type of movie it is (animated, computer animated, live-action)

Note that compression is currently DEAD SLOW. This is because I check the integrity of the uncompressed data in case I did something wrong. I'll remove the integrity checks in the next version, which will make it much more usable.

I also added the --nocomp option to easily turn off compression on the controller side without having to change the config files or restart the agents.

Based on my limited testing, the compression is usually ~3:1 for live-action stuff, and ~4:1 for anime.

Agent-based piping
If the agent's config.xml file has a line like this:
<agentpipe>1</agentpipe>that means that, in agent-based encoding, the agent will pipe data to x264 just like the controller does in controller-based encoding. It will keep all the benefits of agent-based encoding, and it lets you use a 64-bit x264 with 32-bit avisynth. See this thread (http://forum.doom9.org/showthread.php?p=766596) for more info about it.

Bugs
There was a bug which caused frames to be given out to multiple agents at the same time, if agent-based encoding was used. 1.05 makes it very unlikely to happen, but it still may. This bug does not affect the integrity of the file at all, just the speed.
The controller now searches for avs2yuv in the controller's directory, which would seem like a logical place to search.

SpAwN_gUy

13th December 2006, 13:17

foxyshadis

13th December 2006, 13:56

32-bit versions run fine on 64-bit windows, they just don't get the small additional gains of pure 64-bit code. If it's 64-bit *nix, it shouldn't be any harder to compile it than a 32-bit version.

omion

13th December 2006, 17:36

Nice.. will give it a try :)

well... i'm not sure about Compression.. only, maybe will give it a try.. for a TestEncode of PotC2 last Titles ~5mins

so.. another question (you are the Guru, i'm only the Encoder :) )
>>>Under my control i have 32bit and 64bit PCs ... so do i really need to find somewhere 64bit version of x264?
okay.. will try.. will it work under x32 windows with x64 processors?

x264farm is only 32-bit, but it will work perfectly on 64-bit operating systems (I have 64-bit Windows here).
If you have 32-bit Windows, then you can only use 32-bit software (the 64-bit CPU is "hidden")
If you have 64-bit Windows, then you may use my 64-bit x264 build. It's about 10% faster than 32-bit. I didn't make a 64-bit x264farm build because that would just waste more memory than the 32-bit build.

Can we use SharkTooth(damn, i can't remember his Nick.. again)'s builds or only Yours?
in Agent-BasedEncodings... i think it is the most convenient way.. 'cause i don't have a Server .. well. i do have some under my controll, but not me :)

In agent-based encodes, yes. For controller-based encodes, you have to use mine (for now... I'll bug somebody into making a patch. I don't like to do that, but a lot of people are asking for Sharktooth :D )

as soon as the Console program will run smoothly.. i'll make a starterGUI :) .. honestly ;)
Great!

SpAwN_gUy

15th December 2006, 10:01

oaky,... well it seems.. that i do have some problems..

controller still hangs when Exiting all the Jobs..("... agent .. DEAD" or something). then it made some kind of an error when it was exactly 50%. and with version 1.05 i'm looking at some strange behaiour: somehow wery strange it continues the Encoding.. currently i've started it for the third time and it is still 1% ... it was more before :(

P.S.: GUI Developement already started (Delphi, Standart components(mostly.. i'm not sure about our XMLParser.. but others are stndrt), PlainText-Files(no DB.. i was advised to do a WebSevice,.. but it is simplier to write PlainText :) ..))

BTW: Agent x64 runs Great on Vista Ultimate :) ...

today or tomorrow i'l borrow DualCore p4 Duo :) ...

SpAwN_gUy

15th December 2006, 11:29

You've requested Compression levels :)
01: 19.9% compression (9147744 / 45909504 bytes)
01: 21.2% compression (14221274 / 67184640 bytes)
01: 23.5% compression (1822839306 / 7761692160 bytes)
01: 23.5% compression (1821968890 / 7758332928 bytes)
TYPE: Soup Opera (X-Files Season 1 first DVD :) )

and i still don't have access to look for Errors.. so.. later.

I have Only 1PC with controller-based encoding... but it REALLY Slows My Controller.. damn.. the Prcentage of CPU Usage is somehow LOW,.. but the Speed of reacting - is terrible.

AVmaniac

15th December 2006, 15:21

omion

15th December 2006, 16:52

after making some tests and having some movies successfully encoded i recognized a strange behaviour:
every winXP based machine running the controller is seperated from the network after doing some work.
I can ping addresses but can not browse any websites or getting my mails ... tried over telnet .. but nothing works ...
the locally installed ftpserver is not accessible...
and no new(preconfigured) agents can connect to the controller..
but after stopping the controller everything works fine again ..
until it was running again for some time....

That's very strange... How long would you say it works before everything gets disconnected?

When you run into the problem next, run "netstat" on the controller computer in a separate command window. There should be a few lines that look like this:
TCP LOCAL-COMPUTER:2170 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2171 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2172 AGENT-COMPUTER:50700 ESTABLISHED
TCP LOCAL-COMPUTER:2173 AGENT-COMPUTER:50700 ESTABLISHED
If there are lots of them that show up, then it could be an issue with running out of usable ports. I ran into this with early versions of x264farm. (It should be fixed, though...)

What kind of firewalls / antivirus do you have on the computers?
Also, what kind of router do you use for your internet connection?

You've requested Compression levels
...
Wow. Those are really good. 4:1 lossless compression.
I have Only 1PC with controller-based encoding... but it REALLY Slows My ControllerYes, it makes encoding go really slow... the next version should be much better.
the Prcentage of CPU Usage is somehow LOW,.. but the Speed of reacting - is terrible.I don't know exactly why this is, but I noticed the same thing on my computer. For some reason compression prevents the controller from using all of the CPU. I'll look into it if it persists after I remove all the debug code.

AVmaniac

17th December 2006, 23:09

That's very strange... How long would you say it works before everything gets disconnected?

I think about 45mins.

When you run into the problem next, run "netstat" on the controller computer in a separate command window. There should be a few lines...
If there are lots of them that show up, then it could be an issue with running out of usable ports.

No, seems as everything is normal .. there are only a few entries listed.

What kind of firewalls / antivirus do you have on the computers?

First also thought of that as problem source and deactivated everything for testing...
I've also reinstalled WinXP (had to for other reasons)

Also, what kind of router do you use for your internet connection?

I am connected to a campus LAN with a 100Mbps port.
It is connected to a 5port GBit switch providing the connection
for my two PC's.

I also have the problem that everything (agent and controller) , don't know if agent or controller is responsible, simply stops working without crashing.

Thanks for your reply omion

SpAwN_gUy

18th December 2006, 10:18

Okay, i've Figured out why my PC Slows down(controller).. it's bacause of te Huge Local Shares Usage(agent-based)

It seems like i'm having the same problem as AVmaniac... 'cause my Miranda stops responding and connecting after a while saying something about sokets..

and , man, there is something strange about 1.05 ... em.. it DOES NOT CONTINUES the ENCODES.. REally... and it hang again.. (on 93%!!!!! damn)..("the job compleded successfully" thing)
well, the compression - is a good thing,... but i think you've made something wrong since 1.04.

and the 1.04 controller does not work with 1.05-Agents :( ...

okay,.. i am stopping for some time.. and waiting for a new version...

AVmaniac

18th December 2006, 11:09

Last night i startet to test x264farm with normal DVB-S content using dgdecode and until now everything went fine, no network problems .. no hanging (curently 63% of the 2nd pass).
Seems as if it has something to do with the DVB-S2(1080i) content i wanted to encode before.
Perhaps it's the decoders fault (i'm using CoreAVC) or a mix of things.

that's all for now but will keep on testing.

Status Update:

the DVBs capture farm encode finished successfully, so it had to be a decoder bug.
To figure it out I installed version 1.2 of corecodec's fine software(I'm a legal buyer) and resumed the HDTV encode
and until now everything runs fine!!!
network ok ... no hanging .. nothing .. just like omion had it in mind to work like !!! :D

Status Update Nr.2:

I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
my useable network connectivity is gone.
But the controller is running and running and ... like the Duracell Rabbit ..
so i've to wait for the controller to finish before being able to use my WinXP machine again for network access.
And here is what putty is telling me when trying to connect to my gentoo machine:

Unable to open connection to
mygentoobox
Network error: No buffer space available

Status Update Nr.3:

everything stopped working again, and because there seems to be no resuming mode for 2nd pass
I've to choices a) waiting for a new version of x264farm
b) keep on trying...

so far, AVmaniac

SpAwN_gUy

19th December 2006, 17:31

and resumed the HDTV encode
and until now everything runs fine!!!
network ok ... no hanging .. nothing .. just like omion had it in mind to work like !!! :D
you know.. after a downgrade to 1.04 - yesterday it made the same thing :) ... too bad tha i've made a wrong Script.. and the DVD is not well made... :(
I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
you know :) .. the same here.. when i'm sitting in front of the PC - everything is Fine :) .. and when i'm leaving my workplace... 3 hours and "Bang - he's dead" :)
everything stopped working again, and because there seems to be no resuming mode for 2nd pass
I've to choices a) waiting for a new version of x264farm
b) keep on trying...
as i've reported i think this is the bug of 1.05 :( ... the major bug... i think. with 1.04 it just continued the encode.. well the percentage was from the beginning, but moved REALLY fast...(normal ~35(-45)fps.. continue ~80fps)
look in your temp-dir .. how many copies of files "100 100-130 ????.mkv" you have? ... i had the number of restarts..

AVmaniac

21st December 2006, 01:04

i've just downloaded it and windows told me that it is no
valid win32 executable.

omion

21st December 2006, 03:36

i've just downloaded it and windows told me that it is no
valid win32 executable.

Yeah. I think I put the 64-bit version in there accidently

[edit: should be fixed now]

omion

21st December 2006, 05:07

I thought everything would be nice and easy now but i was wrong.
I was away from PC for a while so i don't know exactly when it happened, but with frame 33820/190494
my useable network connectivity is gone.
But the controller is running and running and ... like the Duracell Rabbit ..
so i've to wait for the controller to finish before being able to use my WinXP machine again for network access.
And here is what putty is telling me when trying to connect to my gentoo machine:

Unable to open connection to
mygentoobox
Network error: No buffer space available

This looks good (an a bad way... but it shines some light on the issue) The "no buffer space available" error means that you ran out of memory somewhere important. The buffer memory is taken out of the non-paged pool, which means that it must be physical memory, and 32-bit Windows has a hard-coded maximum non-paged pool of 256MB.

If you don't mind trying to repeat the problem:
when the problem occurs, go to the task manager > Performance > Kernel memory > Nonpaged. If it's near 256000 then that could easily be the problem. If not, then it may be a more general memory issue.

I've also been told that the error can also be cased if some internal TCP queue is full, and there are rumors that it may be related to XP's limit on the number of connections per second. Neither of those are likely, as x264farm only makes one connection per client.

What kind of network cards do you have in the computers?

omion

21st December 2006, 05:19

as i've reported i think this is the bug of 1.05 :( ... the major bug... i think. with 1.04 it just continued the encode.. well the percentage was from the beginning, but moved REALLY fast...(normal ~35(-45)fps.. continue ~80fps)
look in your temp-dir .. how many copies of files "100 100-130 ????.mkv" you have? ... i had the number of restarts..

I just got around to verifying this, and it looks like you're right. The 2nd pass does not restart correctly.

I don't have much time to fix it now, as working in retail at Christmas doesn't leave me much time or energy. I'll get to it as soon as I can, though.

SpAwN_gUy

21st December 2006, 15:22

I don't have much time to fix it now, as working in retail at Christmas doesn't leave me much time or energy. I'll get to it as soon as I can, though.
well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

i'm still using 1.04..
If the 1st pass folder is not deleted(some error) it restarts 1st pass again(when restarted) :( ..
can you tell me which files should be in the Temp folder after the first pass?
or what files are deleted after the first pass? :)

and sometimes not exits(Killing at the end) and hangs there(i guess some TTL would be nice)
2006-12-18~23:50:30.25 pc56 2 exiting...
2006-12-18~23:50:30.25 pc661 2 close_connection
2006-12-18~23:50:30.28 pc17 2 attempt_connect sees no jobs left; DEAD

BTW.. if agent is run as Service.. it can only encode in controller-based encoding :(

708145

21st December 2006, 15:37

well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

I have to step in here ;)
There is Handbrake on Linux and ELDER on windows which are rather similar.

omion

22nd December 2006, 05:45

well take your time :) ...
as far as(i think) x263farm is the only one Prog like this one :) ...

i'm still using 1.04..
If the 1st pass folder is not deleted(some error) it restarts 1st pass again(when restarted) :( ..
can you tell me which files should be in the Temp folder after the first pass?
or what files are deleted after the first pass? :)

After the first pass completes successfully, the final stats file is written to the AVS directory, then the 1st pass subdirectory is deleted.
If the subdirectory is not deleted, the encoding will assume that the file is not written properly, and will continue on the first pass. If all goes well, it will realize that the first pass is in fact done, and will write the stats file and try to delete the temp directory.

Here's how it works, in case anybody was wondering:
The controller will make a directory like:
"temp_root\0123456789ABCDEF0123456789ABCDEF avsfile.avs"
for all the info related to the AVS.

Inside that directory, the first pass will make a directory like:
"1-0123456789ABCDEF0123456789ABCDEF"
which will hold all the files specific to the first pass string that you used.

In the first pass directory, there should be exactly three files:
"settings.txt" is a human-readable file which contains the settings used for this pass
"working_stats1.txt" and "working_stats2.txt" for storing the frames which have already been calculated. In each encoding, one of them is read from and the other is written to. This ensures that there is never a point when there is nothing in the stats file when re-encoding.

Once the first pass is done, the final file written to the AVS directory is:
"1-0123456789ABCDEF0123456789ABCDEF.txt"
where the numbers are the same as the first pass directory. If this file exists and the directory of the same name does not, then the controller assumes that the first pass is done. BTW: This file should be a drop-in replacement for x264's stats file, so you can refer to it if you want to do the second pass without x264farm.

When the second pass starts, you will get another directory in the AVS directory:
"2-0123456789ABCDEF0123456789ABCDEF"
where the numbers are based off the second-pass settings. The first and second pass settings are completely separate, so if you use only change the second pass settings, the controller will see that the first pass is done.

The second pass directory has some number of files in it:
"working_stats1.txt" and "working_stats2.txt" are similar to the first pass, but have a slightly different layout.
"1 - 123 456 abcdef.mkv" are the rendered scenes. Merging together all these files in numerical order will result in the complete video. There should never be more than one file with the same "1 - 123 456" part, but as SpAwN_gUy pointed out, the second pass is doing something wrong when resuming.

Once the second pass finishes, the output file is merged, the second pass directory is deleted, and another file is placed in the AVS directory:
"DONE-0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF.txt"
This file contains the name of the output file. The first 32 letters are the same as the first pass directory name, and the last 32 letters are the same as the second pass directory name. This is so that x264farm can put up that annoying little "This job was already encoded..." message. I really need to put an "encode anyway!" option in there...

and sometimes not exits(Killing at the end) and hangs there(i guess some TTL would be nice)
2006-12-18~23:50:30.25 pc56 2 exiting...
2006-12-18~23:50:30.25 pc661 2 close_connection
2006-12-18~23:50:30.28 pc17 2 attempt_connect sees no jobs left; DEAD

If that's the last line that it printed, then it does look like the entire thing just hung. Normally it would say "pc17 2 exiting..." right after "pc17 2 attempt_connect sees no jobs left; DEAD" I'll try to reproduce it.

BTW.. if agent is run as Service.. it can only encode in controller-based encoding :(
Bummer. That's odd though, as there is very little difference externally between agent- and controller-based encoding, other than the command line that the agent runs. Maybe x264 doesn't like running as a service.

I have to step in here ;)
There is Handbrake on Linux and ELDER on windows which are rather similar.

Yup. ELDER is similar to x264farm, but does Handbrake do distributed encoding? I thought it was just a one-click DVD encoding thing.

omion

22nd December 2006, 05:57

BLEH. I finally found the reason that the second pass is not resuming: I commented out all the code which handles resuming :rolleyes:
I now remember what happened: I was about to re-write the code that handles resuming to be more functional and less imperative. Imperative code tends to be harder to debug, and the second-pass resuming code was nasty. So I commented it all out and was planning to put something in its place when I completely forgot about it...
:stupid:

I try to look through the diffs before I commit the changes, specifically to prevent this kind of thing. I guess I just missed it this time...

1.06 will work! :p

foxyshadis

22nd December 2006, 09:15

Bummer. That's odd though, as there is very little difference externally between agent- and controller-based encoding, other than the command line that the agent runs. Maybe x264 doesn't like running as a service.

Might be a write permissions thing, unless it's running as local system or the user. Network service has very few write rights. Local service has no network rights. And there's always a few programs that don't like non-interactive, but I can't figure out how a bunch of command-line programs would be.

omion

25th December 2006, 20:52

Sorry about the wait. 1.06 is out (Merry Christmas!)

You don't need to update any of your config.xml files if you have 1.05. Just replace the controller.exe and agent.exe files.

Primarily, it fixes the "not resuming on 2nd pass" issue.

There are a few minor fixes, including asking for a confirmation to re-encode the file if it sees that it was encoded before.

The main fix, though, is one I just stumbled across right before I was going to release it on Saturday. This is probably the problem SpAwN_gUy was having with broken files (here (http://forum.doom9.org/showthread.php?p=915266#post915266)).
Apparently, x264 will exit if the requested bitrate is less than the lowest bitrate possible (instead of just overriding the requested bitrate) On long amounts of blank frames, the controller may request a smaller bitrate than possible, and the agent won't recognize the error. This results in an empty file being returned.

The main problem with this is that the MKV header is taken from the first encoded file. If the error occurred in the first file, then the output file won't have an MKV header. This means the entire file will be unreadable.

I now have the controller try not to send bitrates that are too low, and the agent will recognize when x264 exits weirdly.

The bizarre network-based issues (probably) haven't been fixed, but I'll see what I can tweak for 1.07.

deets

25th December 2006, 22:31

does this work with the new threading of x264? could i for example set the threads to auto on each machine?

this would really speed things up. sorry if its an obvious question :)

omion

25th December 2006, 23:27

It will work just fine, as long as you set it up right...

Using the new threading will not give you any performance boost with my x264 build, so if you use controller-based encoding it won't help.

Using agent-based encoding with e.g. Sharktooth's build will work nicely, and it means you would only need to have each computer run one concurrent encode.

There are some performance issues with this, though. Especially during the second pass, the x264 threading won't scale as well as having multiple x264 processes. This is due to the time between jobs when x264 is setting up or sending back the rendered data. It is always a good idea to have another x264 running in the background to fill in the time.

I, personally, would use either x264farm or x264's native threads, not both at the same time.

SpAwN_gUy

28th December 2006, 13:11

Okay.. i've send some LOGS.. to your eMail... and.. i think i'll not have any HTTP Traffic till next Year i guess...

1.06 hangs at closing_Session 's with One file(my current encode,.. previous small went just fine..)
and i'm having something strange with the second file..
on the second pass it can not encode it in the end...
>> 4.81 FPS total on 127459 / 127584 frames (99%)
and i'm getting
>>pc418 2 process_job failed with "End_of_file"; putting job back
or
>>pc6 1 process_job failed with ("The operation completed successfully.\013\n",recv,); putting job back

and then suddenly controller trys to stop(close connections) it hangs.

*offtop* yeah, i've seen "Ani-x264" with x264FARM support.. and i didn't liked it :( .... not so many options.. and no Agent-based encodes.. and no Compression...
and my programmings are not going so well.. i'm not good with free time .. :(

omion

1st January 2007, 07:05

I'm doing a few more tests, and I can confirm a few bugs:

I recreated AVmaniac's problem with the controller computer disconnecting from the network ("No buffer space available"), and I realized that it had happened to me a few times before. I don't know exactly why it happens, but I'll try to tweak some settings to make it not happen anymore.

Also, I think I know what's wrong with some of SpAwN_gUy's files... I'll keep looking into it.

omion

1st January 2007, 22:05

every winXP based machine running the controller is seperated from the network after doing some work.

I think I found the problem (or at least a problem). For some reason there is some kind of file handle leak in the controller. I don't know exactly where it comes from, but on my computer the number of handles increases about 5/second.

This normally wouldn't cause the whole computer to lose connectivity, but it looks like there is a corresponding increase in the number of handles in the "system" process, which could easily cause bizarre issues if they get too large. I think the maximum number of handles for a 32-bit process is 10000...

I'll see if I can fix that somehow.

[edit: more info...]
If you can recreate the bug, fire up the task manager, go to the Processes tab, then turn on View > Select Columns > Handles and see what it says for the "System" and "controller" processes.

Note: there looks to be a workaround here (http://support.microsoft.com/kb/327699), which raises the handle limit. The controller shouldn't need that many handles anyway, so I'll fix it as soon as I can.

omion

5th January 2007, 07:10

Well, I think I found the cause of the controller computer disconnecting from the internet. It's partially Windows' fault, but I am probably the most to blame.

It's my fault because I don't know how to program sockets correctly.
It's Windows' fault because it lets me do it anyway. :p

When each thread from the controller tries to connect to its agent, it starts up a socket. If the connection fails, it shuts down the socket and tries again.
That's apparently the wrong way to do it. I guess after the socket is shut down, it's still a valid handle that has just been thrown away. And the OCaml garbage collector doesn't clean up handles...

This basically means that every 5 seconds one socket is used up per disconnected agent port.

On a Unix-based system, the sockets are filehandles, and the program will die with "too many filehandles open". But on Windows, it just keep on chugging until the OS runs out of... whatever sockets are made out of.

I'll see what I can change around to make it stop doing that. It shouldn't be too hard (assuming that's the only thing wrong... :D )

SpAwN_gUy

5th January 2007, 10:02

:) ... i was pointing on it :) ..... netstst stats - rock :)

omion

5th January 2007, 18:07

:) ... i was pointing on it :) ..... netstst stats - rock :)

Actually, netstat doesn't show the sockets that are causing the problem (which is why it took so long to fix!) The problem sockets are already closed, so they don't show up on any network monitor. But the socket descriptor is still around, which makes a problem for the OS when it runs out of them. The only place I found them listed is in the task manager under "handles".

I actually don't know what caused the large amounts of half-closed sockets that you sent me. Do you remember which encoding you were doing when you used netstat? Was it associated with one of the log files you sent?

BTW: some good news (at last)
I found that you actually don't need to map a network drive in order to get agent-based encoding without copying anything.

If you give a line like one of the following into the <base> part:
\\192.168.1.100\shareddir\movies
\\COMPNAME\shareddir\movies
\\COMPNAME\shareddir\
it will connect to the directory which is shared as "shareddir" on 192.168.1.100 (or whatever computer is named COMPNAME). If you want the shared directory itself to be the base, you MUST add a trailing backslash. If you want a subdirectory of the share to be the base, you MUST NOT use a trailing backslash. (like I did above)
You can see if it works by looking at the output right after the agent starts up.

omion

6th January 2007, 03:31

1.07's out. The disconnecting problem should be solved.

Note that you only need to update the executables, not the config.xml files.

I had planned a couple additional features for this release, but they ended up being WAY too hard, so it's basically just a bug-fix now.
I was going to put in a way of giving out a job to multiple agents at the end of the encode, just to make sure a dead agent doesn't stop the pass from finishing. But that turned out to be immensely difficult the way the program's currently structured.

PS. I found out that you can actually run entire programs through shared directories in the command prompt. This means you can type something like this:
\\192.168.1.100\x264farm\agent --config \\192.168.1.100\x264farm\config-agent.xml
and just have to update one agent and one config.xml.

SpAwN_gUy

29th January 2007, 12:11

I'm Back Again :) ...
new version - COOL.. i'll start testing it now..
with those advices(about netshares) - COOL Again :) ..

Okay.
so.. those Advices with netshares.. it works :) ... but..

//for the future generations
Methods with netShares works ONLY when:
agentpipe = 1
Nice(with .dll), avs2yuv, x264.exe ARE in the Folder of .bat-File(netStart I guess)..
well.. x264.exe is no necessarily(written probably with mistakes ;) ) must be in the folder of .bat-File..
<x264>\\PC201\x264farm\!AGENT\x264.exe</x264>
still works :) ...

so.. now No need to UpDate agents .exe's with new versions.. and lots of problems are gone :) ... and no x264.exe updates ... cool :) ...

omion

30th January 2007, 06:42

Hmm... that's a bit odd. Here's my setup (which I tested everything on)
Agent and config file on remote computer. Started with:
\\192.168.1.100\path\to\agent --config \\192.168.1.100\path\to\config-remote.xml
Config file contains the following:
<config>
<temp>C:\Documents and Settings\Omion\My Documents\x264farm_temp</temp>
<port from="50700" to="50703"/>
<x264>x264-614_omion.exe</x264>
<nice>0</nice>
<bases>
<base>\\192.168.1.100\DVD\MKV</base>
<base/>
</bases>
<agentpipe>0</agentpipe>
</config>

avs2yuv doesn't exist on the agent computer (not needed since agentpipe=0)

x264 is somewhere on the path (not quite sure where anymore... but it's not in the agent directory)

It works just fine with agent-based encoding, too.

BTW: There haven't been any updates in a while because I've been completely redesigning the first pass.

The old implementation would just split the movie into a bunch of chunks and send them sequentially to the agents.
The new one will attempt to keep the agents as separate as possible, and will try to start agents right after they left off. The agents will also try to avoid anything that could keep everything waiting on one agent to finish (although that's impossible to do all the time)

The improvements should make the very end of the first pass much faster. Depending on the file it could decrease first pass encoding time by ~5%.

I suppose I should get around to updating my x264 builds, too... I was hoping that they wouldn't be needed by now, but they still are. I'll build them soon.

SpAwN_gUy

5th February 2007, 11:56

hmm...
i've noticed <bases> part.. does it work in 1.07? ;)

there are still some problems with minimum bitrate sended to x264(i've tried the same encode on different bitrates and different firstpass settings(noticed that during controllerGUI Developement in past few days... it can now import jobs from MeGUI)).

and.. my people noticed(me too), that when agent is run through the network it sometimes crashes.. (i haven't found the reason), 'cause even when hostPC 've been rebooted agent were still up and running.. dunno why this happened, really.

omion

6th February 2007, 06:08

hmm...
i've noticed <bases> part.. does it work in 1.07? ;) I haven't had any problems with it. Try it out!

there are still some problems with minimum bitrate sended to x264(i've tried the same encode on different bitrates and different firstpass settings(noticed that during controllerGUI Developement in past few days... it can now import jobs from MeGUI)).Yeah. I've been busy on another program lately, but I'm probably going to have another version of x264farm out by Wednesday (California time :D ). I think I can fix the minimum bitrate thing fairly easily, but...
and.. my people noticed(me too), that when agent is run through the network it sometimes crashes.. (i haven't found the reason), 'cause even when hostPC 've been rebooted agent were still up and running.. dunno why this happened, really.What does it do when it crashes? Does the whole program exit, or just refuse to encode anything?