x264farm: distributed encoding [Archive] - Page 10

foxyshadis

26th October 2007, 10:55

It's a broadcast, not pings, so it'll only work on a local subnet, and they're so common (smb browse messages, dhcp, ARP, printer lookups...) that it's not going to hurt to have one more sent out now and then. I agree the agents need that capability as well, though it could get quite sticky if there are two controllers on the subnet. And you might need to create a new functionality of agents that do get manually tapped to relay broadcasts back to the controller - if you expect this to ever be used in large routed networks, that is, which might not be the case. (Find out how long it takes to encode HD LotR with the whole Sears Tower working on it!)

This automatic topology building is totally hot, omion. That's almost half the setup effort reduced right there.

omion

26th October 2007, 19:52

'kay... the problem with this mode.. is.. hm..

you can actually be banned for pinging ports on evry mashine.. and if you have network with.. 400PCs.. ??? ...

IMO you'd better change this feature to Opposite..
I was fairly worried about this, but as foxyshadis pointed out, it's a broadcast message, not a port scan or anything. Nothing can be banned from sending out a few of these (they're actually required for routing sometimes) but I'm mainly worried about sending out many of them.

i have the controller.. I'm spreading pre-configured agents.. and.. i think.. agents, most likely, should ping the controller (and i will pre config mine IP(the controller's) in the agent's config)The way it actually works is the controllers will send out a broadcast every minute, and any connected agents will send a response to that controller. Also, the agents will send one broadcast when they start up, then stay quiet for the rest of the time. The total network utilization for this is around 400 bytes per minute (depending on the number of agents)

but.. i just came to that: we'll loose "multi-controller" compatibility (partially).. and agents will allways (when started, not only when encoding) ping my PC.. and tell me "Gimmie something to encode"... (BTW.. this will encourage me to give them something to encode ;) )

Yeah. The agents still don't know when to stop encoding, so this is mainly good for doing only one encode per network at a time. I'm really hoping to add functionality to actually limit the number of encodes that an agent does, but it's quite difficult to do correctly in Windows.

UPD: WITH adhoc in the controller - it all works "fine" (as it was)... but it starts more slowly..How long does it take to recognize all the agents?

with no "adhoc" on the controller and "yes and no" on the agent side ... and good old ports 50700 - 50703..
it's NOT working at all..Sorry, but what is "yes and no"? The <adhoc> section must (currently) be present in both the agent and controller config files for it to work.

there is NO "--crf" or something specified... and when i add "--bitrate 1000" it worksWell... x264 needs some sort of bitrate information for the first pass. Without --bitrate or --crf or --qp x264 will refuse to encode.

SpAwN_gUy

29th October 2007, 08:49

How long does it take to recognize all the agents? as far as i have 10agents written in config.. and adhoc option enabled (everywhere) it is only.. hm.. 2 seconds :) ... (400PCs in the LAN)
Sorry, but what is "yes and no"? The <adhoc> section must (currently) be present in both the agent and controller config files for it to work.well,.. if there is NO such option in controllers config.. then controller - does not work..

if it is there - everything works fine..

Well... x264 needs some sort of bitrate information for the first pass. Without --bitrate or --crf or --qp x264 will refuse to encode.but evrything worked fine... some time ago.. (i guess .. x264farm 1.09) ... maybe the x264 had different behaviour ??? ...

'coz.. i dont know what to do in meGUI .. i need to add "--crf 19" or 21(the default) ??? ... so we would not loose the functionality and "quality-mode" would properly work...

omion

29th October 2007, 18:09

as far as i have 10agents written in config.. and adhoc option enabled (everywhere) it is only.. hm.. 2 seconds :) ... (400PCs in the LAN)
That's fast enough ;) It does take a bit of time to set up the agents, so that's to be expected
well,.. if there is NO such option in controllers config.. then controller - does not work..

if it is there - everything works fine..
Ah. Yes, it does have to be enabled in both config files. If the controller doesn't have it, it won't bother to search. If the agents don't have it, they won't reply

but evrything worked fine... some time ago.. (i guess .. x264farm 1.09) ... maybe the x264 had different behaviour ??? ...

'coz.. i dont know what to do in meGUI .. i need to add "--crf 19" or 21(the default) ??? ... so we would not loose the functionality and "quality-mode" would properly work...
Yeah, x264 changed a while back. It used to assume (I think) "--qp 26". I don't remember exactly when it changed, but it caught me off guard too.

Wishbringer

29th October 2007, 19:07

Yeah, x264 changed a while back. It used to assume (I think) "--qp 26". I don't remember exactly when it changed, but it caught me off guard too.

x264 rev.655 (mid may 07)
http://forum.doom9.org/showpost.php?p=1003219&postcount=319

And I found another (forgotten) link:
http://forum.doom9.org/showpost.php?p=989251&postcount=228
which one is better version of nice?
first post link from march or your later posted link from april?

omion

29th October 2007, 23:40

x264 rev.655 (mid may 07)
http://forum.doom9.org/showpost.php?p=1003219&postcount=319That was the one. Thanks! :)

And I found another (forgotten) link:
http://forum.doom9.org/showpost.php?p=989251&postcount=228
which one is better version of nice?
first post link from march or your later posted link from april?

Oops. The April one is better, although the only thing I changed was the Vista compatibility. I'll update the link on the main page ASAP.

SpAwN_gUy

30th October 2007, 12:04

Ah. Yes, it does have to be enabled in both config files. If the controller doesn't have it, it won't bother to search. If the agents don't have it, they won't replysorry... maybe i'm writing somehow veri bad ;) ..

but.. i am trying to point on:
when i use my good old config.xml:

and start an encode.. (the test one.. that works fine.. btw.. i need to get some other testfile..)
i only get:
x264farm controller version 1.15-182
Using config file ".\config.xml"
WARNING: Output file 'E:\DVDRiP\!!x264farm\TEST\X\test-meGUI-HQInsane.mkv' already exists and will be overwr
itten!
CONFIG:
temp_dir: "G:\\DVDRiP\\_temp"
agent list:
pc79 = 010.00.00.000:50700 (1,2)
pc206 = 200.00.00.000:50700 (1,2)
...omitted, security purposes... there are 8 of them.......
pc17 = 100.000.000.00:50700 (1,1)

FIRST PASS
0% done (0 / 4197) at 0.00 FPS
Last updated: 0:15
----------------------------------------------------------------------------------------------------------
Agent | FPS | Description
Recent errors:
~
~
~
~

and.. THAT'S ALL... not even updated (for 5 mins now..)

but, when i UNcomment the adhoc in config - Everything works fine..

i only want to clarify, that (now) this feature is not Optional, but mandatory... ;)

Yeah, x264 changed a while back. It used to assume (I think) "--qp 26". I don't remember exactly when it changed, but it caught me off guard too.so.. i suppose when comLine generation for controller is done by hand... one can fix that..

but for meGUI ??? ... (i already omitted "--pass" beginnings.. and output trailings...)..
well.. i can add "--crf 19" with no problem.. so first pass will work ... or should i rise or lower this value?

SatansChild

1st November 2007, 05:06

The new version won't compile for me, it seems to have some issues with the c-optimizations.c (in the common dir). Make output is posted below.

root@core:/workspace/x264farm/1.15/common# make
ocamlopt -c -thread unix.cmxa threads.cmxa pack.ml
ocamlopt -c -thread unix.cmxa threads.cmxa pack.cmx types.ml
ocamlopt -c -thread unix.cmxa threads.cmxa types.cmx pack.cmx net.ml
ocamlopt -c -thread rbtree.mli
ocamlopt -c -thread unix.cmxa threads.cmxa rbtree.ml
ocamlopt -c -thread list2.mli
ocamlopt -c -thread unix.cmxa threads.cmxa list2.ml
ocamlopt -c -thread unix.cmxa threads.cmxa matroska.ml
ocamlopt -c -thread buffer2.mli
ocamlopt -c -thread unix.cmxa threads.cmxa buffer2.ml
ocamlopt -c -thread unix.cmxa threads.cmxa pack.cmx types.cmx huff.ml
ocamlopt -c c-optimizations.c
c-optimizations.c: In function ‘do_paeth’:
c-optimizations.c:30: warning: pointer targets in initialization differ in signedness
c-optimizations.c:37: warning: unused variable ‘p’
c-optimizations.c: In function ‘undo_paeth’:
c-optimizations.c:100: warning: pointer targets in initialization differ in signedness
c-optimizations.c:107: warning: unused variable ‘p’
ocamlopt -c c-optimizations.obj optimizations.ml
/usr/bin/ocamlopt: don't know what to do with c-optimizations.obj.

omion

2nd November 2007, 00:59

sorry... maybe i'm writing somehow veri bad ;) ..

but.. i am trying to point on:
when i use my good old config.xml:

and start an encode.. (the test one.. that works fine.. btw.. i need to get some other testfile..)
i only get:
...
and.. THAT'S ALL... not even updated (for 5 mins now..)CR@P! I screwed up. I had wanted to have ad-hoc enabled by default for this build, but I backed out of it at the last second. So I turned off the thread that receives the broadcast responses from the agents... the problem is the agents are now started by that thread too. So basically none of the agents are started unless ad-hoc is enabled.

The way around this is, as SpAwN_gUy noted, to uncomment the <adhoc> element in the controller's config file.

If you think your IT guys will complain about the broadcast packets, you can turn off the broadcast with the --noadhoc command line option. This will keep the controller listening for agent responses, but it will stop it from actively searching for agents.

@SatansChild:
For Linux, you have to run "make OBJ_EXT=.o" instead of just "make". It's a pain, and I should really document that better... I think there's a way to make OCaml keep consistent object extensions, but I haven't looked into it too far.

SatansChild

2nd November 2007, 03:12

Adding OBJ_EXT=.o seems to help it get passed the c-optimizations.c file but then it runs into issues with the controller.ml file / module OptParse.
root@core:/workspace/x264farm/1.15/controller# make OBJ_EXT=.o
ocamlopt -c c-console.o console.ml
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.ml
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx rbtree.cmx ratecontrol.ml
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx huff.cmx pass1.ml
File "pass1.ml", line 1641, characters 9-18:
Warning Y: unused variable new_range.
File "pass1.ml", line 2052, characters 8-20:
Warning Y: unused variable start_offset.
File "pass1.ml", line 2051, characters 8-27:
Warning Y: unused variable found_some_i_frames.
File "pass1.ml", line 1401, characters 6-11:
Warning Y: unused variable print.
File "pass1.ml", line 430, characters 5-17:
Warning Y: unused variable print_ranges.
File "pass1.ml", line 378, characters 5-14:
Warning Y: unused variable batch_now.
File "pass1.ml", line 42, characters 5-16:
Warning Y: unused variable first_batch.
File "pass1.ml", line 36, characters 5-31:
Warning Y: unused variable first_pass_frames_done_ref.
ocamlopt -c -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa pack.cmx types.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx huff.cmx rbtree.cmx matroska.cmx ratecontrol.cmx pass2.ml
File "pass2.ml", line 1462, characters 6-11:
Warning Y: unused variable print.
File "pass2.ml", line 680, characters 5-21:
Warning Y: unused variable print_agent_hash.
File "pass2.ml", line 46, characters 5-15:
Warning Y: unused variable version_rx.
ocamlopt -o controller -thread -I ../common unix.cmxa str.cmxa threads.cmxa xml-light.cmxa extLib.cmxa pack.cmx rbtree.cmx types.cmx ../common/c-optimizations.o optimizations.cmx c-console.o console.cmx controllerinclude.cmx list2.cmx net.cmx huff.cmx matroska.cmx ratecontrol.cmx pass1.cmx pass2.cmx controller.ml
File "controller.ml", line 25, characters 0-13:
Unbound module OptParse
make: *** [controller] Error 2

omion

2nd November 2007, 06:19

Adding OBJ_EXT=.o seems to help it get passed the c-optimizations.c file but then it runs into issues with the controller.ml file / module OptParse.
Hmm. Where is the file "optparse.cmx"? It's very likely that it is not where I expected it to be. That's the most likely explanation.

The makefile expects the libraries to be in the directory specified by "ocamlc -where", but I've seen some configurations which put all the extlib modules in their own subdirectory.

SpAwN_gUy

9th November 2007, 13:08

okay... some minor Update ;) ...

now i have my own (not so own.. but) Branch in meGUI developement on SF.net .. i'm not so familiar with SF so far.. so..

this one i kinda Release Candidate of meGUI x264farm mod :) ... i've made this build to get some feedback ;)

changelog:
- updated to latest meGUI
- moved to SF.net (sources are HERE (http://megui.svn.sourceforge.net/svnroot/megui/branches/x264farm) (use TortoiseSVN to checkout and MSVS C# edition to build))
- x264 profiles selection IS working, but Profile-Name is NOT stored
- ".log" and other stuff is present.. (if not, just tell me)

currently thougths are about:
"do we really need manual editing for first and second pass settings?"
"what do we need?" and "any suggestions?"

http://rapidshare.com/files/68496094/megui-x264farm_mod-2k7.11.09.exe

i guess i'll make something with SF.net file storage of future versions.. (if there will be any)

and as soon as x264farm itself will be more-less stable ;) ... and "mod" will meet all the needs.. this will move to main thread and x264farm itself will take it's place on meGUI's autoUpdate ;) ... omion? are you there? ;)

Adub

9th November 2007, 22:46

Super cool!
Super lame I can't test though. Or even use. :(

SpAwN_gUy

12th November 2007, 08:52

Super cool!
Super lame I can't test though. Or even use. :(
tell me what is NOT working, and what do you do?

'cause my extraSense's team is drinking alot currently, so the can't really work meanwhile.. ;)

Adub

12th November 2007, 10:14

SpAwN_gUy

12th November 2007, 11:31

NO, no, no. I meant that I can't use it. Personally. I am a poor college student living in a dorm at CalPoly San Luis Obispo with 2 other guys. I don't have the first thing that even looks like a network.

So, I am sure it works fine, it is just that I can't use it. Not that I don't want to.
i guess.. (in our prices)
switch 8-ports: 10-12$
cable (5-10m): 5$
so you will have a network ;) ...

'kay... 10 downloads of meGUI-Mod.. and NO Feedback?? :(

Adub

12th November 2007, 12:24

I would still need computers on the ends of those cables. and I don't think my roommates laptops count.

To everyone else, FEEDBACK DAMMIT!!!! I want to know if this works so that I have something to look forward too, as classes are kicking my ass!

SpAwN_gUy

12th November 2007, 12:54

I would still need computers on the ends of those cables. and I don't think my roommates laptops count. why not?
omion is using his macBook laptop as one of his encoding mashines.. (as seen in "config.xml" shipped with controller)

Yoshiyuki Blade

12th November 2007, 13:25

I just downloaded it. It'll be a while before I can learn how to use this thing, and it'll probably be longer for me to set things up :D. I'll provide feedback asap.

Jabroni

16th November 2007, 18:38

Great software.. i use this on 2 computers i have.. both quads, and it works wonderfull.. i just have one question.. Is there a way to compute the x264 output stats of all parts encoded?? :confused:

AVmaniac

19th November 2007, 01:47

Really nice work!!!

Now everybody can use the performance of x264farm with the control through megui .. thanks, works really good so far!

okay... some minor Update ;) ...

now i have my own (not so own.. but) Branch in meGUI developement on SF.net .. i'm not so familiar with SF so far.. so..

this one i kinda Release Candidate of meGUI x264farm mod :) ... i've made this build to get some feedback ;)

Jabroni

19th November 2007, 03:38

i guess.. (in our prices)
switch 8-ports: 10-12$
cable (5-10m): 5$
so you will have a network ;) ...

'kay... 10 downloads of meGUI-Mod.. and NO Feedback?? :(

Ok i started playing with it.. but i have several questions.. where do i specify the ips of the 'workers' ?? (lets say I already have the agent client working on my other computer). Right now I have x264farm run as client based (that would max both of my quad on each computer).. or what is a 'worker' used for? (for me a 'worker' would be another agent client.. but not quite sure what is under your nomenclature)...

SpAwN_gUy

19th November 2007, 10:33

Ok i started playing with it.. but i have several questions.. support-unit-mode.. OFF:
RTFM ;) ... "x264farm.html" shipped with binary..
where do i specify the ips of the 'workers' ?? (lets say I already have the agent client working on my other computer). support-unit-mode.. ON:
"config.xml"
Right now I have x264farm run as client based (that would max both of my quad on each computer).. or what is a 'worker' used for? (for me a 'worker' would be another agent client.. but not quite sure what is under your nomenclature)...
i assume: worker = agent
"not" "worker" = controller..

try proper tutorial in my signature..

when you'll set up agent-based - youll have both avs(decoded) and x264(encoded) buy each agent.. .. and just a bit by controller..

if you have few agents - then i guess it does not really matter controller- or agen-based method to use..
but if you have "heavy" avs-script.. then agent-based is preferrable.. i think..

SpAwN_gUy

19th November 2007, 10:36

Great software.. i use this on 2 computers i have.. both quads, and it works wonderfull.. i just have one question.. Is there a way to compute the x264 output stats of all parts encoded?? :confused: x264.stats are stored in the "TEMP" folder for each encode.. try looking for .txt's there

Jabroni

20th November 2007, 02:51

support-unit-mode.. OFF:
RTFM ;) ... "x264farm.html" shipped with binary..
support-unit-mode.. ON:
"config.xml"

i assume: worker = agent
"not" "worker" = controller..

try proper tutorial in my signature..

when you'll set up agent-based - youll have both avs(decoded) and x264(encoded) buy each agent.. .. and just a bit by controller..

if you have few agents - then i guess it does not really matter controller- or agen-based method to use..
but if you have "heavy" avs-script.. then agent-based is preferrable.. i think..

Actually what i was trying to point out.. is that I already have a working x264farm setup.. via just command prompts, now the real question was apart the configuration of your GUI implementation, I pointed in the settings to the controller.exe, but when I assign a job to a worker, it justs launched a plain x264.exe thread (no controller.exe that would fire up all the agents to send the work).
I tired to check all the .txt on your release, but it seems like its from a regular meGUI install...

Might be some newbie questions.. im new onto the encoding job, plus i started with something that is relative 'new' or 'unexplored' by most people... but still.. i like it :devil: :D

SpAwN_gUy

21st November 2007, 11:08

I pointed in the settings to the controller.exe, but when I assign a job to a worker, it justs launched a plain x264.exe thread (no controller.exe that would fire up all the agents to send the work).
I tired to check all the .txt on your release, but it seems like its from a regular meGUI install...
video codec needs to be set to "x264farm"..
then create ("config") a new profile,
then from the config screen Choose an x264-profile (top comboBox), then press YES. after that - tune commandline and params as you like (or just keep them as generated)..
and then save a NEW (or update the profile, if you are editing some) profile FOR x264farm...

in BitrateCalculator - you can specify "x264farm" as codec... and apply bitrate values..

i guess this is... like that.. ;) ..

Pitbull_Raven

21st November 2007, 17:32

video codec needs to be set to "x264farm"..
I use megui to recode and downsize files so I can watch them on my xbox360. Files have to be smaller than 4gb or the xbox will not play them. Will your mod allow me to encode to mp4 using my custom profiles?

Jabroni

22nd November 2007, 08:25

video codec needs to be set to "x264farm"..
then create ("config") a new profile,
then from the config screen Choose an x264-profile (top comboBox), then press YES. after that - tune commandline and params as you like (or just keep them as generated)..
and then save a NEW (or update the profile, if you are editing some) profile FOR x264farm...

in BitrateCalculator - you can specify "x264farm" as codec... and apply bitrate values..

i guess this is... like that.. ;) ..

Ok after some playing around I can now give a lil' feedback on my experience. I like how it was integrated on a GUI, but, it would be nice if it would give you the option to spawn and show the dialog of the controller.exe process (since its a bit more informative) So far ive done 2 encodes, im doing my third one which should take a bit more.... still overcall its really good... but what it would be a bit plus, is that it could give a processed stats of the encoding, so far ive come with this utility http://forum.doom9.org/showthread.php?t=118559 , i use that to parse a txt it creates on the controller temp folder, with all the stats output of the agents to generate some nice looking stats of the encode (QP and such).

Pitbull_Raven

22nd November 2007, 12:20

I think this needs a better How To. The one I read is a bit outdated, doesnt even mention a GUI. Also, I haven't been able to get it to run on 2 machines. Could someone please take the time to explain a step by step?

I downloaded the gui version from rapidshare. What else is needed? I have all the files installed on my Z: drive. This drive is a share on my server and accessable to both the controler and the second PC I am going to use for tests. Both pc's are using that share as Z. My sample file is default.avs which is pointing to default.avi. I can play the avs file on both the controller and agent pc. After that I'm pretty lost as to what to do to get it all up and running.

Jabroni

22nd November 2007, 22:06

Uhm.. i have this encode.. for some reason controller.exe wont split more parts when after like 80% on the first pass, when they are other agents available.... (it even just send 1 thread on the current agent, instead of the 3 i have configured)

Heres an output of my controller console:

FIRST PASS
92% done (344335 / 371705) at 6.15 FPS
Last updated: 15:33:47
########################################################################||____
Agent | FPS | Description
Local | 3.89 | Doing [344335-349115]
SageTV | 5.65 | Waiting for job
Local | 1.50 | Waiting for job

tre31

23rd November 2007, 04:20

Ok I've had agent-based encoding going quite well between windows only machines, however I need too add one linux machine too the configuration, but how do you maintain the same configuration (linux doesn't use d:\x264farm .. there is no such thing as d:\ in linux).

So far I've managed too get the agent compiled and going on x86-32 linux, x264 the same, its all good, but I don't know how too structure the .avs and shares too include the linux machine.

Currently I have z:\ shared over the network (root of drive k:\), that has a directory x264farm with all thats required in there, I put the .avs in that directory, so the question is now, too get get the .avs too work for linux, previously I used z:\ in both the .d2v and .avs too get it working as agent's, but how do you get linux too function as an agent using the same .avs?

Is this even possible?

-------
edit:
Just realised that without using the beta avisynth for linux it wouldn't work anyway, and even then i'd still have that same problem with dgmpegdec and non-local .m2v file... looks like my cunning plan was somewhat shortsighted. Onto the backup plan, winpe/bartpe/reatogo usbkey and figuring out howto get avisynth working on that (yes its an extra layer too the problem, but its a bare pc, no cd, no hdd, but I have a usbkey, slowly bits & pieces have been dying but the core of it works .. lol).

omion

23rd November 2007, 09:34

Uhm.. i have this encode.. for some reason controller.exe wont split more parts when after like 80% on the first pass, when they are other agents available.... (it even just send 1 thread on the current agent, instead of the 3 i have configured)

Yup. x264farm does that when it sees that the current splits are "risky". If the controller does not know if a split ends on an I frame, it will not start an agent on the job directly after that. The reasoning behind this is that if the I frame is calculated incorrectly the next split will have to be done over again, which may make it take longer.

Imagine a video with 10000 frames where no frame is complex enough to trigger an I frame. In this video, all I frames will be a multiple of 250 (the default max I frame interval) If x264farm decides to start encoding at, say, frame 4999 then the entire second half of the video will have its I frames off by 1 from what they should be. When the encoder which started at the beginning finally gets to frame 4999, it will see that the frame should not be an I frame, and will then throw out all the encoding done in the second half.

This example may seem a bit strange, but there is one area where it makes a big difference: credits. Credits are usually encoded at the maximum I frame interval, so they are encoded serially to avoid re-encoding. Older versions of x264farm were quite a bit slower at the end since every frame in the credits were encoded multiple times. (and I noticed your controller output is waiting on the end of the encode, exactly where the credits probably are)

That being said, if you want to change the behavior, you may adjust the --rethresh parameter of the controller. "1" means that it will not do any of this, larger numbers will tell it to be more "careful".

@tre31:
Don't feel too bad: I haven't gotten agent-based encoding to work on my Linux computers either. :p Does controller-based work for you at least?

About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.

audyovydeo

23rd November 2007, 10:20

About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.

Hello,

I started playing with x/farm just yesterday (1.15).
Everything on the controller so far, I'll try with 2nd PC this weekend.
I'm pretty sure my config files are fine.
I think I got a path problem, because when i run the controller, it says :

x264farm controller version 1.15-182
Using config file ".\config.xml"
CONFIG:
temp_dir: "C:\\bin\\x264farm115\\controller\\temp"
agent list:
Local = 127.0.0.1:50001 (1,1)

FIRST PASS
0% done (0 / 1000) at 0.00 FPS
Last updated: 0:02
------------------------------------------------------------------------------
Agent | FPS | Description
Recent errors:
~
~
~
~

ie : nothing happens, it just waits.
The agent runs correctly.

my dir structure is
\bin \x264farm \agent \temp
\bin \x264farm \controller \temp
and it makes no difference whether the .avs and .avi are in \controller or \controller\temp.
Paths in the avs are absolute. The avs is as simple as it can be :

DirectShowSource("C:\bin\x264farm115\controller\cube.avi")
converttoyv12()

note : I am using svn x264 700, not cefs.
is cef's version mandatory ?

thanks
audyovydeo

omion

23rd November 2007, 10:40

Hello,

I started playing with x/farm just yesterday (1.15).
Everything on the controller so far, I'll try with 2nd PC this weekend.
I'm pretty sure my config files are fine.
I think I got a path problem, because when i run the controller, it says :

...

note : I am using svn x264 700, not cefs.
is cef's version mandatory ?

thanks
audyovydeo

Cef's version is not mandatory, but in order for controller-based encoding to work with Windows x264farm must be compiled with MSVS, not GCC.

The problem is actually something that SpAwN_gUy pointed out before. In version 1.15 you need to set the adhoc element in the controller's config.xml file, even if you don't plan on using that feature. It's a stupid bug that made its way into the program because I got cold feet from broadcasting stuff by default.

So, add the following line to the controller's config file:
<adhoc controller="40700" agent="40701"/>

Then, if you don't want to use ad-hoc agent discovery (look back a few pages for more info) use the --noadhoc argument to the controller.

audyovydeo

23rd November 2007, 11:01

So, add the following line to the controller's config file:
<adhoc controller="40700" agent="40701"/>

Thanks : now controller sends agent the job.
I have bumped x264 to cef's 699 version.
Agent gets the job, but throws exceptions :

"s not recognized as an internal or external command,
operable program or batch file.
436 exited 1
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")
"
closing socket
socket closed
Got a connection from 127.0.0.1:2143"

I'll go back and reread the thread for insights.

cheers
audyovydeo

omion

23rd November 2007, 11:11

"s not recognized as an internal or external command,
operable program or batch file.
436 exited 1
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")
"
closing socket
socket closed
Got a connection from 127.0.0.1:2143"

That's usually from either using the config option <nice> without nice.exe available, or not using the right x264 executable.

SpAwN_gUy

23rd November 2007, 14:47

New people... nice :) ...

About "new" Progress Window for x264farm - i thought about that.. and, i guess, this will not be the problem.. only... some time..

'kay.. and i guess. the new tutorial and config.xml generator would be also a nice thing to do....

audyovydeo

23rd November 2007, 15:04

That's usually from either using the config option <nice> without nice.exe available, or not using the right x264 executable.

mmmh, I'd tried with and without nice even before posting here.

Anyhow, with your <adhoc> suggestion agent & controller speak.
Problem remains :

Got a connection from 127.0.0.1:3920
working!
920 first pass frames 0 to 999
Testing file "C:\bin\x264farm115\controller\temp\z_input.avs": FOUND!
920 agent-based encoding
920 doing the following:
"x264_667_omion" --crf 25 -r 1 -b 3 -A none -m 1 --me hex --qcomp 0.6 --cplxbl
ur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3
--fps 5000000/166833 --pass 1 --stats "C:\bin\x264farm115\agent-win32\temp\stats
e9903a.txt" --seek 0 --frames 1000 -o NUL "C:\bin\x264farm115\controller\temp\
z_input.avs"
Thread 659 killed on uncaught exception Unix.Unix_error(40, "send", "")
'x264_667_omion" --crf 25 -r 1 -b 3 -A none -m 1 --me hex --qcomp 0.6 --cplxblur
20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --
fps 5000000/166833 --pass 1 --stats "C:\bin\x264farm115\agent-win32\temp\stats'
is not recognized as an internal or external command,
operable program or batch file.
920 exited 1
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (1)\")
"
Thread 661 killed on uncaught exception Unix.Unix_error(40, "send", "")
closing socket
socket closed
Got a connection from 127.0.0.1:3921
... and so ad infinitum.

In bold are the different errors I get for each request cycle. There are obiously many things at work here.

Tried with :
snv 700
cef's 699
dark shikari's 697
omion's 667

so it's probably not the compiler issue here.
Again this is with agent & controller running on same machine.
I'll try w/different this weekend, but any input is welcome.

thanks
audyovydeo

audyovydeo

23rd November 2007, 15:08

New people... nice :) ...

Well it always boils down to SMP vs grid computing doesnt it

Besides, having just bought a C2D I'm nowhere near buying a dual quad-core ;-)

cheers
audyovydeo

Jabroni

23rd November 2007, 18:09

Ok now im stuck with this encode on an endless loop
Controller:

SECOND PASS
99% done (370955 / 371705) at 6.81 FPS
Last updated: 1d 11:39:16
ETA: 1:50
Agent | FPS | Description
Local | 3.40 | Doing [235385-235634],[254484-254733]
SageTV | 6.11 | Disconnected

Agent

nice -n 10 "x264" --ref 3 --bframes 16 --b-pyramid --weightb --direct auto --f
ilter -2,-1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me um
h --merange 12 --threads auto --thread-input --cqmfile C:\Encodes\prestige.cfg -
-progress --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10
--qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 24 --fps 30000/1001
--pass 3 --stats "c:\temp\agent\stats 8d3520.txt" --seek 234635 --frames 250 -
o "c:\temp\agent\output 9e8809.mkv" "d:\encodes\encode1080.avs"
avis [info]: 1280x720 @ 29.97 fps (371705 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSSE3
x264 [error]: requested bitrate is too low. estimated minimum is 62 kbps
x264 [error]: x264_encoder_open failed
Thread received exception "Sys_error(\"nice -n 10 \\\"x264\\\" --ref 3 --bframes
16 --b-pyramid --weightb --direct auto --filter -2,-1 --subme 6 --trellis 1 --a
nalyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 12 --threads auto --threa
d-input --cqmfile C:\\\\Encodes\\\\prestige.cfg --progress --no-psnr --no-ssim
--qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --iprati
o 1.4 --pbratio 1.3 --bitrate 24 --fps 30000/1001 --pass 3 --stats \\\"c:\\\\tem
p\\\\agent\\\\stats 8d3520.txt\\\" --seek 234635 --frames 250 -o \\\"c:\\\\temp
\\\\agent\\\\output 9e8809.mkv\\\" \\\"d:\\\\encodes\\\\encode1080.avs\\\": No e
rror\")"
closing socket
socket closed

it seems that the x264.exe isnt letting use a low bitrate... and the controller doesnt catch this and increase the bitrate (not quite sure if the controller has to do that).

omion

23rd November 2007, 21:41

mmmh, I'd tried with and without nice even before posting here.

Anyhow, with your <adhoc> suggestion agent & controller speak.
Problem remains :
...
... and so ad infinitum.

In bold are the different errors I get for each request cycle. There are obiously many things at work here.

Well, those all look like the same error. One of the issues I'm cleaning up with the agent is that errors will sort of "bubble up". First cmd.exe reports that it can't find something, then the agent reports that x264 failed, then it reports that, due to the failure, the entire thread died.

Tried with :
snv 700
cef's 699
dark shikari's 697
omion's 667

so it's probably not the compiler issue here.
Again this is with agent & controller running on same machine.
I'll try w/different this weekend, but any input is welcome.

thanks
audyovydeo

That issue looks like a quoting problem. I had an issue with it in the controller. The problem is that the command prompt will remove the first and last quotes from a line it's running, which looks like what it's doing here. The problem is that I have no idea why that's happening for you and not me. I'll look into it (although I think I may have already fixed it for the next version)

In the meantime, I think this will work to stop the problem: try using the <nice> parameter in the agent's config file. Get the nice.exe (http://omion.dyndns.org/x264farm/nice-20070415.rar) program, and use the following line in the agent's config.xml file:
<nice>1</nice>

That will set the encodes to slightly below normal priority, but it should be just enough to throw off the console's stupid quote-removal "feature".

@Jabroni:
Yes, that is due to the controller using a bitrate which is too low. I have it automatically increase the bitrate when it thinks this is the problem, but it will only ever ask for 3 times what it thinks is the optimal. If this value is less than the actual lowest bitrate, then it will stall like that.

If you haven't done any more encodes since then (or if your latest encode has the same problem) could you send me the controller's out-dump.txt file? I'll PM you my e-mail address.

tre31

26th November 2007, 04:47

I'm not entirely sure whether this is bug, but I have seen it replicated multiple times using 1.13, I have only done one encode with 1.15 and I let it go untouched (because I was getting over having failed encodes .. heh).

Basically what happens is I get an error (don't have logs sorry), that states that the 2-pass bitrate curve is wrong (pretty sure it was related too Quality calculations being off), however that is not the actual cause of the error, the error is caused (and you could probably replicate this yourself using 1.13) when agents fail during the second pass (multiple reasons, crashes, hardware fail, etc), and loose connection too the controller, or if the encode stops during the 2nd pass for some reason.

This happened multiple times too me, so it is definately there in 1.13, I'm not sure about 1.15 (one of the reasons I updated actually).

adhoc agent discovery works quite well, now my controller config is just these lines (much easier)...
<config>
<temp>z:\x264farm</temp>
<adhoc controller="40700" agent="40701"/>

</config>

It's also probably a smarter way too do things as each agent should know its own config, rather than relying on the controller too always know the capabilities of the agent (ie. ip's can change in a dynamic dns setting - I ended up setting static dhcp entries too make sure each machine really was the machine I expected it too be).

@omion:
'About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.'

Relative paths could just work you know for all agents if the path is the same, ie. on linux create directory in root dir 'shared' or something (and use smbmount \\machine\shareddrivename /shared -o workgroup=WORKGROUP,username=user,password=pass), and in root dir of drive on windows shared drive create same directory 'shared', so .. what is created is the same.
sample .avs (so you get my drift):
DGDecode_mpeg2source("\shared\x264farm\clip.d2v")

corresponding .d2v would obviously have too be similar.

It could work - if x264farm on linux converts the slashes in the .avs too the correct way for linux, otherwise it won't (but that wouldn't be too hard too implement I don't think - food for thought).
---------
please note: the above is just a theory, and may have no basis in reality. I think its probably a bit too soon anyway, wait for avisynth linux too actually be out of beta (at least one version) maybe, who says there is even a DGDecode function in avisynth linux, too many variables.

audyovydeo

26th November 2007, 10:46

In the meantime, I think this will work to stop the problem: try using the <nice> parameter in the agent's config file. Get the nice.exe (http://omion.dyndns.org/x264farm/nice-20070415.rar) program, and use the following line in the agent's config.xml file:
<nice>1</nice>

That will set the encodes to slightly below normal priority, but it should be just enough to throw off the console's stupid quote-removal "feature".

Tested this weekend, this does work around the issue.

Now that I got it working, can I some feature requests :
- adding support for mp4 (the standard container)
- supporting 1pass encoding

reason is, I think 2pass encoding is bound to disappear over time. Personally I haven't done a single 2-pass encode since I got a good grip on crf.

cheers
audyovydeo

omion

27th November 2007, 01:13

I'm not entirely sure whether this is bug, but I have seen it replicated multiple times using 1.13, I have only done one encode with 1.15 and I let it go untouched (because I was getting over having failed encodes .. heh).

Basically what happens is I get an error (don't have logs sorry), that states that the 2-pass bitrate curve is wrong (pretty sure it was related too Quality calculations being off), however that is not the actual cause of the error, the error is caused (and you could probably replicate this yourself using 1.13) when agents fail during the second pass (multiple reasons, crashes, hardware fail, etc), and loose connection too the controller, or if the encode stops during the 2nd pass for some reason.

This happened multiple times too me, so it is definately there in 1.13, I'm not sure about 1.15 (one of the reasons I updated actually).

I did change a few things which affected the stability of the program after 1.13, especially when the remote agent dies and hangs the whole encode. So there's a very good chance that it has been fixed, but without logs I wouldn't know for sure. If it happens again, send me the logs for both the controller and any agents.

adhoc agent discovery works quite well, now my controller config is just these lines (much easier)...
<config>
<temp>z:\x264farm</temp>
<adhoc controller="40700" agent="40701"/>

</config>

It's also probably a smarter way too do things as each agent should know its own config, rather than relying on the controller too always know the capabilities of the agent (ie. ip's can change in a dynamic dns setting - I ended up setting static dhcp entries too make sure each machine really was the machine I expected it too be).

I thought it was a strange (and cumbersome) way to set up the agents too, which is why I added the ad-hoc mode. Glad to see it works! :)

@omion:
'About the path problem: you can specify the paths in both the .avs and .d2v files to be relative. That is, if everything is in the same directory you don't need to specify any paths at all.'

Relative paths could just work you know for all agents if the path is the same, ie. on linux create directory in root dir 'shared' or something (and use smbmount \\machine\shareddrivename /shared -o workgroup=WORKGROUP,username=user,password=pass), and in root dir of drive on windows shared drive create same directory 'shared', so .. what is created is the same.
sample .avs (so you get my drift):
DGDecode_mpeg2source("\shared\x264farm\clip.d2v")

corresponding .d2v would obviously have too be similar.

It could work - if x264farm on linux converts the slashes in the .avs too the correct way for linux, otherwise it won't (but that wouldn't be too hard too implement I don't think - food for thought).
---------
please note: the above is just a theory, and may have no basis in reality. I think its probably a bit too soon anyway, wait for avisynth linux too actually be out of beta (at least one version) maybe, who says there is even a DGDecode function in avisynth linux, too many variables.

That should work fine. One thing to note: neither x264farm nor x264 actually parse the avs file. It's all done through AVIsynth. As long as AVIsynth for Linux converts the paths correctly, this should work.

The problem is with Windows AVIsynth running through Wine to a Linux build of x264farm. I haven't gotten a good way to do this properly, but when the Linux AVIsynth is stabilized it should be much easier.

omion

27th November 2007, 08:53

Now that I got it working, can I some feature requests :
- adding support for mp4 (the standard container)
- supporting 1pass encoding

reason is, I think 2pass encoding is bound to disappear over time. Personally I haven't done a single 2-pass encode since I got a good grip on crf.

cheers
audyovydeo1 pass encoding has been on my "next version" list for a very long time now. I'm still planning to put it in, but it will need a lot of work, so I'll probably put it off some more.

About .mp4:

There are 2 reasons I support mkv:
1. It's the container I use
2. I already wrote an mkv porser

The reason I don't yet support mp4:
Writing parsers is hard, and I don't want te do it again ;)

I have to actually recreate every temp file since x264 will add a bunch of encoding info to each encode. This is normally a good thing, but it adds a lot of space if it is included in every scene. So I have to take apart each file, strip out the encoder info, and put it back together. That's hard to code for, so I only support one container.
It should be a fairly straightforward conversion from mkv to mp4, and I'm sure that somebody's written one. If not, I may make a standalone app that x264farm might use... maybe. I'm kind of burnt out on containers now, though, and I still have a lot of things to throw into the next version of x264farm, so again, maybe sometime but not now.

audyovydeo

27th November 2007, 09:45

I'm kind of burnt out on containers now, though, and I still have a lot of things to throw into the next version of x264farm, so again, maybe sometime but not now.

I understand burnout perfectly, no pbm.
Also I agree 1pass is higher piority than mp4.

While your tool is not yet industrial strength, it's on the right track. I dont know of any other distributed encoder for h264.
This may well become the poor man's solution to heavy-duty encoding, vs professional hardware encoders. So any minute you spend on it, in my view, is well spent.

cheers
audyovydeo

SpAwN_gUy

27th November 2007, 10:27

'kay....
*offtop("dot" is working extremely bad on my keyboard this pisses me of.. sorry)*
'kay.
i have a some kind of feature request.. this one is more fr then a BUG...
it would be nice, when:
i'm doing agent-based mostly... so if i set something NOT right.. and agent sees the .avs.. but can not open it (well. none of them can.. this is due to "not relative paths", "not inserted plugin-strings" and so on..)..
i'm getting "End_of_file" from all of the agents...

is it possible to swithch from agent-based to controller-based encoding?
so when i'm at home... an have 2 encodes in the list.. so the first one went fine, but the second - stalled in infinite loop.. (i dont use controllerPC as agent)

and for.. mp4.. hm.. i'll try to search for muxer... i guess i've seen it somewhere, with .mkv input, append-file.. and so on...

btw... for final MKV muxing yopu could try to use libmatroska muxer.. it has append feature... (mmg - has..)

i'll be back ;)

SpAwN_gUy

27th November 2007, 11:52

so..
as i've read in iNet....

there is NO direct way to reMUX mkv to mp4 :( ...
at first one have to extract RAW h264 stream and then with mp4box mux it to mp4..

but.. i dunno about the possibility to stream raw h264stream directrly to mp4box input.. (if this will work.. then libmatroska -> mp4box input in the end - and you've got mp4-file ;) )...
ok then...
gpac lib is the one that could help with integrating mp4... i guess searching throug the code will give the idea to connect mkv-raw output with mp4-input...

basically so..

hehehe. i just thought... if omion will be able to produce raw AVC stream then, i guess.. we'll have all three "major" output formats available :) (mkv, mp4, raw)