Log in

View Full Version : x264farm: distributed encoding


Pages : 1 2 3 4 5 6 7 8 9 10 [11] 12 13 14

SpAwN_gUy
29th November 2007, 15:45
spawn, use gdsmux from haali.this one is DirectShow muxer... so, no cross-platform support i guess..
it simply uses DirectShow codecs/parsers/dumpers to demux input and to produce final file..

St Devious
17th December 2007, 19:44
OK , I', a newbie at this . But i want to do this as i have 4-5 Computer in my house on my network and i have quite a lot of HDV captured footage to encode , so if i can use this the process will be faster .

Could someone help me out ?

Another question , is there any kind of gui for this , or is this completely command-line ?

Is there any chance of this getting integrated into megui in future or a different program ?

SpAwN_gUy
18th December 2007, 09:19
OK , I', a newbie at this . But i want to do this as i have 4-5 Computer in my house on my network and i have quite a lot of HDV captured footage to encode , so if i can use this the process will be faster . it seems like it IS faster.. not so "stable" but faster..
and Welcome... :)

Could someone help me out ?Sure, why not.. Share your questions with us ;) ...

Another question , is there any kind of gui for this , or is this completely command-line ?
Is there any chance of this getting integrated into megui in future or a different program ?
try... hm... this one ;) (http://forum.doom9.org/showthread.php?p=1063839#post1063839)
work is still in progress..
but at first you need to configure controller and agents.. and then look through one of a bit old tutorials..

St Devious
18th December 2007, 18:00
oh man this is just too complicated for me , dont know wat to do with modded megui .

waiting for someone tocome up with a gui for this .

Sharktooth
18th December 2007, 18:10
:readguid:

SpAwN_gUy
19th December 2007, 09:06
oh man this is just too complicated for me , dont know wat to do with modded megui well.. you could at least try to read the topic.. a few pages back.. and few posts after the link i gave you...
if you would... then you would read this (http://forum.doom9.org/showthread.php?p=1067736#post1067736)

waiting for someone tocome up with a gui for this . write your own... this is a free world... i guess..

audyovydeo
19th December 2007, 10:19
oh man this is just too complicated for me , dont know wat to do with modded megui .

waiting for someone tocome up with a gui for this .

And while we are in a philosophical mood, GUI is not a synonym of EASY, generally speaking.

I managed to set up x264farm cmdline in, let's say, 2 hours and 3 posts+replies. Which isn't much, for an alpha-level application with alpha-level docs.

Just get it up & running on your main machine, then stem off to the others. Reread through the help doc and this thread, and you'll get it.

cheers
audyovydeo

St Devious
20th December 2007, 01:54
ok i'm giving this one more try . I selected x264 farm codec in megui and i see these options which r different from x264 .

I selected HQ-slowest profile , and left all other parameters at default . So wat do i do now , copy the collective command line parameters which r written in box near bottom or can i encode to x264farm using megui .

If yes , how do i set up the agents ? run megui too on them ?

Another question : I want to have only one agent . So in the config.xml file of the controller , do i have 2 agents ( 1 my pc and 2nd another pc ) or only the agent's ip address and other info .

Some more questions : Where do i put nice.exe and avs2yuv.exe ?

I tried running the controller and agent . agent is running fine , but controller exits . here r the config.xml files for both .

CONTROLLER :

<config>
<temp>C:\x264farm\controller\temp</temp> <!-- Some directory for storing temp data -->

<agents>

<agent name="Matrix"> <!-- The name of the agent. This is only informational -->
<ip>192.168.111.147</ip> <!-- IP address of the agent (127.0.0.1 is the computer the controller is running on) -->
<port from="40700" to="40703"/> <!-- A port range. This MUST be identical to the setup of the agent -->
<number>2</number> <!-- How many jobs to send to the agent at one time. Should generally be equal to the number of cores on the target machine -->
</agent>

<agent name="Agent Smith">
<ip>192.168.111.66</ip>
<port from="40700" to="40703"/>
<number pad="1">1</number>
<!--
The pad="1" attribute tells the controller how many additional jobs to send for the second pass.
It's good to set this to "1" on all non-controller computers, when there should be a backup
agent running when the others are sending back jobs.
-->
</agent>

</agents>
<adhoc controller="12345" agent="23456"/>
</config>

AGENT :

<config>

<temp>C:\x264farm\agent\temp</temp>

<!-- Usable port range. Note that only one of these ports will be used at a time. -->
<port from="40700" to="40703"/>

<!--
These options control the ad-hoc networking support. Uncomment the <adhoc> element
to enable it. <name> is the name that the agent will tell the controller, and
<number> is the number of simultaneous encodes to tell the controller (same
style as in the controller's config.xml file
-->
<!--<adhoc controller="40700" agent="40701"/>-->
<name>Local</name>
<number pad="0">2</number>

<!--
Use this to change the executable that is run. It will default to "x264"
if it is not specified here
-->
<x264>x264</x264>

<!--
If you want to set your encoding to a lower priority, set <nice> to a larger number.
0 means normal priority, 20 means idle. If you are running Windows, you need to
have the "nice" program installed, either in the directory you run the agent from,
or somewhere on the path.
Unix-ish operating systems already have this program.
-->
<nice>20</nice>

<!--
<base> is for agent-based encoding.
If it is set to "<base></base>" or "<base/>" then the AVS file is searched for
in the exact directory that the controller found it.
If it is set to a directory, that directory is taken as the base encode directory.
For example, if the controller is encoding file "D:\movies\working\somewhere\file.avs"
and you have <base>F:\temp\x264farm</base> here, then the AVS file is searched for in this order:

F:\temp\x264farm\D:\movies\working\somewhere\file.avs
F:\temp\x264farm\movies\working\somewhere\file.avs
F:\temp\x264farm\working\somewhere\file.avs
F:\temp\x264farm\somewhere\file.avs
F:\temp\x264farm\file.avs

If none of those files are found, the default controller-based encoding is used.

If <base> is not specified or if it is commented out, controller-based encoding is always used.

You may specify multiple base directories with the <bases> line. For example:
<bases>
<base>base1</base>
<base>base2</base>
<base/>
</bases>
Each of these bases is treated exactly as described above.
The first directory listed which has an identical file is used as the encoding base


NOTE: Thanks to a few quirks in the way OCaml handles Windows directories,
if you use only a drive letter you MUST have a trailing backslash;
if you use some subdirectory you MUST NOT have a trailing backslash.
For example:
<base>c:</base> *WRONG*
<base>c:\</base> *RIGHT*
<base>c:\documents and settings</base> *RIGHT*
<base>c:\documents and settings\</base> *WRONG*
You can see if a directory was accepted by the "agent bases:" line displayed
when the agent is first run.

See x264farm.html for more information
-->
<base></base>

<!--
Set <agentpipe> to 1 in order to pipe data through to x264 in agent mode.

If this is off the following basic command is executed:
x264 <options> -seek a -frames b input.avs

If this is on the command is more like this:
avs2yuv -raw -seek a -frames b -o - input.avs | x264 <options> - 720x480

Really the only reason to use this is if you have a 64-bit version of x264,
but you have a 32-bit version of avisynth. In this case, x264 is incapable of
simply linking to avisynth, and therefore needs the data fed to it by avs2yuv.
Obviously, this requires that the agent have access to avs2yuv.
-->
<agentpipe>0</agentpipe>

<!--
The <compression> section is where you can select which types of compression to
accept from the controller for controller-based encoding. These settings do nothing
when using agent-based encoding.
These are the currently supported compression types:
0: Uncompressed. This is the fallback compression type.
1: Paeth / Huffman. This tends to get 3:1 - 4:1 compression.
The first one in this list which the controller supports is used by the controller.
Every type under type 0, therefore, is disabled, since "no compression" is always supported
-->
<compression>
<type>0</type>
<type>1</type>
</compression>

<!--
<passcompression> sets different compression settings for the first and second passes.
Both the <first> and <second> sections have the same format as the above <compression>
element, but they only affect one of the two passes.
This is useful if the network is the bottleneck during the first pass,
but the encoder is the bottleneck during the second.
-->

<!--
<passcompression>
<first>
<type>1</type>
</first>
<second>
<type>0</type>
</second>
</passcompression>
-->
<adhoc controller="12345" agent="23456"/>
</config>

And the error i get is :

http://i16.tinypic.com/72ktsvs.jpg

audyovydeo
20th December 2007, 10:42
Again, I haven't tried the megui option. But what is your controller command-line ?

to resume :

0. read through the doc and this thread once again.

1. deploy the whole cast of characters (directories, paths, all executables, source avi, avs scripts, etc). Nice and avi2yuv should, as always with any windows program, somewhere in your path. just like the correct version of x264.exe

2. edit the xml files accordingly (temp paths are important and finicky to configure)

3. launch the agent(s)

4. run the controller
here is my cmdline :

controller.exe -B 500kbps --avs temp\input.avs -o test.mkv -1 "--crf 25 -r 1 -b 3 -A none -m 1 --me hex" -2 "-r 1 -b 3 -A none -m 1 --me hex"


and ... (how do they say ?) ... et voila' !



cheers
audyovydeo

SpAwN_gUy
20th December 2007, 10:50
I selected HQ-slowest profile , and left all other parameters at default . So wat do i do now , copy the collective command line parameters which r written in box near bottom or can i encode to x264farm using megui . and why do you think (and asked about) this Mod is made for?
and why x264farm is "Codec" in meGUI ? ...

If yes , how do i set up the agents ? run megui too on them ? NO. you don't need to setUp meGUI on them..
just run agent.exe on agentPC and let it run..

Another question : I want to have only one agent . So in the config.xml file of the controller , do i have 2 agents ( 1 my pc and 2nd another pc ) or only the agent's ip address and other info . well.. since we have adhoc feature (and it IS working for me at least).. you dont even need to specify agents in controller's config.. but you can leave it like that...

so.. the only mashines that will be Encoding are the PCs where agent.exe is started.. and RUNNING

Some more questions : Where do i put nice.exe and avs2yuv.exe ? agent's and controller's folder

I tried running the controller and agent . agent is running fine , but controller exits . here r the config.xml files for both. nice picture you have here.. :) ..
this picture means that you haven't read anything.. i'm sorry.. but "WHY???"
READ the manual.. on how to use x264farm...
controller-based encoding is the simpliest thing to set up.. :readfaq:

SpAwN_gUy
20th December 2007, 11:37
But what is your controller command-line ?according to the picture he showed.. it's "nothing" :) ..


BWAHAHAHA.... the previous post was my 200th ;) ... nice using :)

St Devious
20th December 2007, 15:50
@ Spawn_guy : u r not exactly helping me here .

I gave u my agent and controller config , tell me wats wrong with that if u really want to help rather than make smart remarks . I need help , thats y i posted here.

So i select megui mod on the controller pc , and run agent.exe on each of the agent pc's and start encoding in megui ?

Look at the picture and my agent , controller config and tell y the controller cant find agent pc .

The ip's of boh the pc's r the ip's on lan , is that correct ?

Sharktooth
20th December 2007, 16:10
im sorry to say this, but your problems (as well as in the other threads) are all related to your ignorance.
those kind of things (video encoding) are not simple and you need to read and understand how things work.
do as spawn guy said: READ.
If you find it too much complicated, then start with the basic things.

audyovydeo
20th December 2007, 16:12
Hard to help someone who's not putting effort into the task. We've practically spoon-fed you all the info.

Specifically :

1. in your agent XML file, the fact that you got < nice> value of 20 only shows that you have not read this post :

http://forum.doom9.org/showthread.php?p=1068890#post1068890


2. in your controller xml there is no trace of the ADHOC directive, which means you havent read this post :

http://forum.doom9.org/showthread.php?p=1068730#post1068730

3. your controller command-line is non-existent, which means you haven't read the documentation.


Hard to figure what else we can do, short of drilling for oil.

cheers
audyovydeo

SpAwN_gUy
20th December 2007, 17:26
2. in your controller xml there is no trace of the ADHOC directive, which means you havent read this post : well.. i guess.. he has "adhoc" enabled.. in both configs..

'kay.. my give up..
-1) setUP configs.. i guess everything seems to be fine
0) run agent.exe on agentPC's
1) configure in meGUI path to the controller in options..
-----------------READTHEGUIDEONmeGUIUSAGE--------
2) create an AVS and Choose it as an input
3) choose x264farm as Codec
4) configure params as in the post i gave you
5) press OK.
6) push Enque
7) press start on the second Tab in meGUI..
wait

i guess i'll need some money for such kind of things :) ..
do we have to think for you?

i guess there will be no posts from me in 4 days.. i'm going to Geneva... for 2 weeks.. anyone want's to meet me there? :)



UPD: ah.. BTW.. there is one more tool for x264farm ... i wonder what it's name.. i dunno is it upDated or not.. you can search for it in here.. i tried it a long time ago.. so search fo x264farm support

Sharktooth
20th December 2007, 17:34
Spawn_guy, Geneva? No... for now, my best would be Genova (Italy).

SpAwN_gUy
21st December 2007, 09:27
Spawn_guy, Geneva? No... for now, my best would be Genova (Italy).
hm.. it's quite far from Geneva (Switzerland) .. :) .. i think.. but who knows :) ...

antiOFFTOP: my next steps on x264farm.. are:
1) organize storing normal x264settings in meGUI for x264farm (now they are only "strings")
2) complete my agentGUI (Delphi).. (i'm waiting for a new updated version of x264farm.. as omion said.. he's trying to release it soon)
2b) write an "aget as service starter".. not all of my people are satisfied with using it as a gui... (they are working in Terminals)
3) write and agentGUI/Service installer.. so everyone could change sources and compile the installer.. and then to share it with others.. who wishes to participate in encoding..

agentGUI is allmost complete.. i'm reWriting it now.. and most of my people use an old one..

but.. as tomorrow is my last workDay this year.. i suppose.. neXt year..

ChrisBensch
3rd January 2008, 06:29
Is it possible, since we don't really have single-pass yet, to have a REALLY fast first pass, then use our second pass as the "real" pass? Maybe just dumping the first pass data to /dev/null? I use the crf preset of MeGUI in my cli so I'm not sure if a first pass even benefits me at all. Right now it's just running the same single pass, just twice. I wish there were a better way.

SpAwN_gUy
6th January 2008, 21:17
Is it possible, since we don't really have single-pass yet, to have a REALLY fast first pass, then use our second pass as the "real" pass? Maybe just dumping the first pass data to /dev/null? I use the crf preset of MeGUI in my cli so I'm not sure if a first pass even benefits me at all. Right now it's just running the same single pass, just twice. I wish there were a better way.
Yep.. i'm allmost back from... where i was :) ...

omion is currently working on "single-pass" encoding... (as i've heard.. well.. that was last year... so, maybe, soon)
and "yes" for now.. using CRF(for x264) and PERCENTAGE(for x264farm) will do Two passes... but during the first one it will only write .stats..

ChrisBensch
11th January 2008, 03:56
Alright, so percentage is the way for farm. This might be a dumb question, but is there a sort of "rule of thumb" for percentages and how they correlate to a CRF value? I usually encode with a CRF of 18...is there a percentage that would match that?

omion
11th January 2008, 10:14
Hi guys! I'm back!

Is it possible, since we don't really have single-pass yet, to have a REALLY fast first pass, then use our second pass as the "real" pass? Maybe just dumping the first pass data to /dev/null? I use the crf preset of MeGUI in my cli so I'm not sure if a first pass even benefits me at all. Right now it's just running the same single pass, just twice. I wish there were a better way.

I don't really know what you are saying here. The first pass video data is actually sent to /dev/null (check the code!); it only keeps the stats file. The stats file is basically what sets the second pass in the right direction. I could make a one-pass option (and am still thinking about it, as spawn_guy says) but it would require an entire rewrite of the encoding loop.

This might be a dumb question, but is there a sort of "rule of thumb" for percentages and how they correlate to a CRF value? I usually encode with a CRF of 18...is there a percentage that would match that?
Yup. 100% gets you the same file size that the first pass would have gotten, had you just done a one-pass encode. So if you use --crf 18 on the first pass, 100% will get you around crf 18 on the second pass (but hopefully with higher quality, which is the whole point of a two-pass encode).


Now, on an unrelated note:
<rant>
Networking is HORRIBLE.
I've spent the larger part of two months working on the stupid networking code. I've made about 4 different implementations of it, and each one had some sort of horrible problem with it that would make it completely unacceptable. My current code actually crashes the program. Fairly reliably, too. OCaml code is not supposed to crash. Exit with an error, maybe, but not crash. I don't know how I managed it. It was probably due to some workaround that I added a month ago because Windows can't seem to read and write to the same socket at the same time. So now I have to think of another workaround. The main problem is probably just Windows, which really bugs me because I know those problems aren't going to be fixed. Should I tell the Windows devs to implement timeouts on send/receive functions? The best that can happen is they add it to "Windows 7" when it comes out in 5 years.

I've been working on this code for quite a while now. I think I can do it with two uni-directional sockets between the agent and the controller. But they can't be totally uni-directional, or else the TCP "delayed ack" algorithm will kick in and bring everything down to 10kbps. WHY is that stupid "congestion control" enabled on so many computers? Is an extra 60 bytes really going to take down the network?

I think the code is getting to be too much for me to handle. In order to do all of these workarounds I have to make a whole bunch of threads to shuffle around the data, and the networking makes it break sometimes. I'll get errors like "Thread 37 failed with ENOTSOCK" Not a socket? A half second ago it was a socket, and it was working just fine. And what on earth is thread 37? Is that the thread that is sending data? Or the thread that's receiving data? Or maybe it's the thread that sends dummy data on the loopback socket that I had to set up in order to get Windows unstuck from the select statement that works just fine on every other operating system. So I track it down, and it turns out it's on the thread that sends a ping signal to the controller as long as the 3 threads that communicate with x264 are still working... so I have to workaround that too.

All this extra junk to try and get computers to work together. It's really aggrivating.
</rant>

So, how was your Christmas? :rolleyes:

SpAwN_gUy
11th January 2008, 10:22
Alright, so percentage is the way for farm. This might be a dumb question, but is there a sort of "rule of thumb" for percentages and how they correlate to a CRF value? I usually encode with a CRF of 18...is there a percentage that would match that?
the History Chanel:
at first.. first-pass settings for x264farm needed NO "bitrate" or "crf" options.. 'cause in older x264 builds the default option, when no parameter was specified was "--crf 26" .. or 25..
but after.. the default setting was removed.. and thats why i had to add it manually to comLine Generation in meGUI-Mod... but i never tried such low bitrates so.. my value is "--crf 18" for the first pass..
so.. basically, when you specify in first-pass settings "--crf 18" and for the x264farm bitrate "100" (or "100%") then you will have an TwoPass Encoding with bitrate of CRF 18 ...

on how "percentage" of bitrate is assumed you can read somewhere here or i guess it was somewhere in the doc's, provided with binaries

------------
UPD: glad you're back omion :) ...
So, how was your Christmas? :rolleyes: very nice indeed.. Geneva seems fine.. Berlin is nice at night.. and Prague is extremely Cheap.. and Beer is beautiful :)

Elvis'es and mine Birthsday went just fine :) .. i got drunk.. again.. aswell as on celebration of NY in my Corporation :)

and Testings of DGAVCDec (neuron2's) with x264farm.. just ended with beautiful results :) ... (yeah, i had to modify internals of ".dga" file aswell :) ...)

au
30th January 2008, 17:02
Hello guys!
I'm trying to encode tga-sequence (1500frames long), so it has no GOPs by default :)
Which settings should I use to speed-up fist-pass calculations (I mean 1st pass split parameters)?

Poopoo
30th January 2008, 17:56
very nice indeed.. Geneva seems fine.. Berlin is nice at night.. and Prague is extremely Cheap.. and Beer is beautiful :)


You drank beer in Prague ? How long have you been sick from it ?

omion
2nd February 2008, 00:21
Hello guys!
I'm trying to encode tga-sequence (1500frames long), so it has no GOPs by default :)
Which settings should I use to speed-up fist-pass calculations (I mean 1st pass split parameters)?

If it's all smooth motion (no clear GOPs) and fairly long then you can change the --fastavs parameter to something like a small black clip (must be the same # of frames and framerate).

HOWEVER (and this is a big however) if there are no clear GOP boundaries in the entire clip, x264farm will NOT split up the file at all. This means that the first pass will encode a bit slower than if you didn't use x264farm. The problem is that x264 will encode a bunch of GOPs of length 250 (by default) and so any encode that does not start on a multiple of 250 will be thrown out. x264farm decides to "save" computing resources until it can find a reasonable split point.

So in your case there is not much you can do to speed up the first pass. Actually, the default job length is longer than 1500 frames, so x264farm won't even try to split it.


@all:
I suppose this is as good a time as any to give a little update. I've decided to concentrate my time on making a single-pass x264farm, rather than trying to hack my way through the mess that is the current 2-pass code. It will be highly simplified (no wacky split options, config.xml no longer required, smaller executables, etc.) but it should be just as powerful as - and hopefully more efficient than - the current program. (And I'm making a 64-bit version, not that it would make any speed difference at all:p)

I'll probably end up applying the algorithms to a new 2-pass x264farm, but that will be a while away.

ChrisBensch
11th February 2008, 07:16
Any news on the MeGui-mod for x264farm? How about a client config gui? I dabble in Java/C# and I'd be happy to contribute if I can.

SpAwN_gUy
11th February 2008, 12:16
Any news on the MeGui-mod for x264farm? How about a client config gui? I dabble in Java/C# and I'd be happy to contribute if I can.
i'm busy with my work.. i'm hacking linux.. ;) ..
so, no updates for now :( ... contact me via PM... for more instructions :) ...

konzerten_nr9
10th March 2008, 05:55
I am runnig an x264farm consists of a controller and 6 agents, 5 agents running on Windows using agent-based encoding, 1 on Linux using controller-based encoding.

Now I am runnig a test with a simple test.avs as input ike:
directshowsource("test.wmv", audio=false)
which test.wmv has 9440 frames.

My command line is:
controller -B 3500kbps --qcomp 0.75 -1 "--crf 19 --keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --analyse all --threads auto --progress" -2 "--keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --subme 5 --ref 1 --trellis 1 --me hex --merange 16 --threads auto --progress" --avs ..\test.avs -o ..\test.mkv

The problem is, the footage is splitted into several parts during the first pass, but it is not being encoded by multiple agents simultaneously. Instead, the splitted parts are encoded by a single agent sequentially.

This results in the throughput of first pass varying between 2 to 15fps depending on the agent picked up at the beginning of the first pass, while the throughput of the second pass is around 30fps.

Is there any way to have the first pass encoded by multiple agents simultaneously?

Thank you in advance for your kind suggestions.

Jabroni
10th March 2008, 06:00
I am runnig an x264farm consists of a controller and 6 agents, 5 agents running on Windows using agent-based encoding, 1 on Linux using controller-based encoding.

Now I am runnig a test with a simple test.avs as input ike:
directshowsource("test.wmv", audio=false)
which test.wmv has 9440 frames.

My command line is:
controller -B 3500kbps --qcomp 0.75 -1 "--crf 19 --keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --analyse all --threads auto --progress" -2 "--keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --subme 5 --ref 1 --trellis 1 --me hex --merange 16 --threads auto --progress" --avs ..\test.avs -o ..\test.mkv

The problem is, the footage is splitted into several parts during the first pass, but it is not being encoded by multiple agents simultaneously. Instead, the splitted parts are encoded by a single agent sequentially.

This results in the throughput of first pass varying between 2 to 15fps depending on the agent picked up at the beginning of the first pass, while the throughput of the second pass is around 30fps.

Is there any way to have the first pass encoded by multiple agents simultaneously?

Thank you in advance for your kind suggestions.


If you scroll on this thread I asked the same thing.. the problem is that its not able to do more splitting because it cant find more I-Frames (or whatever type of frame it is used to split). So you have to wait till the first pass is over so it can start sending again work to all agents on the 2nd pass

konzerten_nr9
10th March 2008, 07:12
Hello Jabroni,

If you scroll on this thread I asked the same thing.. the problem is that its not able to do more splitting because it cant find more I-Frames (or whatever type of frame it is used to split).

Actually the first pass is splitted into 2 parts:
Frames 0-5159 Frames 5160-9439
but all the jobs are done on 1 single agent serially.
Thus I thought it is a new situation here...

driftr
13th March 2008, 16:15
So here's my latest problem. Large source files. Like over 10 gig high bitrate source files. I often give up on x264farm because the first pass takes forever. The process to split the file into batches and send the jobs out maxes out at 100% proc usage (or 25% on my quad core system) and goes extremely slow and might get a second job out before the first one finishes encoding. Overall the first pass is so slow just doing command line x264 encoding on the single computer alone is faster. Any thoughts on how to get the process multithreaded or better handled?

SpAwN_gUy
13th March 2008, 17:04
i'm here again...

'kay.. mine things are doing fine.. even with 10Gb 1080p x264 encoded files.. did that a while ago... first pass - multi threaded..

the reason of 100% CPU on the Controller might be on Split-points determination... (i have single core.. when starting - yeah it's a bit annoying, but when working - fine..)

driftr, post you AVS script..

driftr
13th March 2008, 18:48
I wrote that rather slopily... the avs script is a simple directshowsource opening the file with no filtering or framerate change or anything. The problem is source material having sizes like 720x576 or say blu-ray with 1920x1080.

What's happening is avs2yuv can't keep up with the how fast the computers are encoding the jobs. So I suppose I could bump up the batch size but then the initial set of jobs would take really long esp. with say 15+ agents. What I'd really like to see is some kind of multiple controller setup where multiple instances of avs2yuv can create the jobs.

Right now on this simple hour long 720x576 wmv I'm just converting to x264 for fun and there's a point where agents are just idle waiting for jobs because the controller can only push out so many at a time.

SpAwN_gUy
14th March 2008, 11:38
well.. i just don't know what to say...

well.. you can run multiple controllers :) ...

and... try out setting "--batchmult 0" and/or "--rethresh 1.0"
i can't remember which one is "the one".

driftr
14th March 2008, 15:21
Humm I forgot about those settings. I can see them helping marginally but I'll play with them.

I'm not reading anything about how to run multiple controllers... got an example?

SpAwN_gUy
14th March 2008, 16:19
I'm not reading anything about how to run multiple controllers... got an example?
try to run.. 2 (two) cintroller.exe's at the same time ;) ...
like.. prepare two .bat files.. and the run them..

or make 2 Workers in meGUI-x264farm-mod ... and the run them :) ... (the old one(working) could be still found somwhere here.. my version is still in unworking state :( )

omion
17th March 2008, 23:44
Hi guys! Just to let you know, I'm not dead yet... :p

I've finally got my 1-pass x264farm in a usable state. I'll be releasing it in a few days unless some other major bugs appear.

It should be a bit more efficient than the current code, and driftr's problem should not exist (the splitter does not examine the avs file)

@driftr:
I assume that you are seeing one avs2yuv instance using 100% of one core, correct? Or is it controller.exe? If it's avs2yuv then setting "--thresh 1.0" will minimize the usage of the splitter thread at the expense of some extra time taken re-rendering frames around the split point.

Note: --rethresh is a different setting from --thresh. It's a bit complicated, but "--rethresh 1.0" will not do what you want (in fact, it will basically do all of the same calculations, then ignore them :scared: )

@SpAwN_gUy:
Do multiple controllers work? I didn't think they'd play nice with the shared stats file. I'm on my laptop right now (and lunch break is almost over) so I can't check, but did you find a way to get this to work with the current iteration of x264farm?

SpAwN_gUy
18th March 2008, 09:24
I've finally got my 1-pass x264farm in a usable state. I'll be releasing it in a few days unless some other major bugs appear.
so'.. i guess i need to stick to meGUI thread.. 'cause berrinam is now missin' ... :) :( ...

Do multiple controllers work? I didn't think they'd play nice with the shared stats file. I'm on my laptop right now (and lunch break is almost over) so I can't check, but did you find a way to get this to work with the current iteration of x264farm?Never tried that.. but i can :) .. and i will...

i was trying to say, that multiple controllers will work on different jobs (but never tried that either..)

so.. i guess.. more testing will be done :)

omion
19th March 2008, 21:10
Single-pass version is out. (Yay! A new release!)
It has a new thread (http://forum.doom9.org/showthread.php?p=1114551) to keep the bugs separate from the 2-pass version. Have fun!

vwpassion
31st March 2008, 09:24
First of all: Nice job omion!

Now my concern: Is there a way to make x264farm work with current builds of x264 from Sharktooth or older versions from Cef above 655?

Roswellian
1st April 2008, 08:26
Hello, I am newbie here and just went through the whole threads. There are too many stuff and almost totally confused me. Right now I set up the agent-base mode and my problem is that there is no response on my agents after controller started. Here is some codes on my agent side.

<config>
<temp>.\temp</temp>
<port from="40700" to="40703"/>
<name>Local</name>
<number pad="1">1</number>
<x264>x264.exe</x264>
<nice>0</nice>
<base>d:\Sample</base>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

Controller side
<config>
<temp>.\temp</temp> <!-- Some directory for storing temp data -->

<agents>
<agent name="Local"> <!-- The name of the agent. This is only informational -->
<ip>127.0.0.1</ip> <!-- IP address of the agent (127.0.0.1 is the computer the controller is running on) -->
<port from="40700" to="40703"/> <!-- A port range. This MUST be identical to the setup of the agent -->
<number>1</number> <!-- How many jobs to send to the agent at one time. Should generally be equal to the number of cores on the target machine -->
</agent>

</agents>

</config>


As you see there, I tried to encode files locally by connection (One controller + One agent). My sample file is small m2ts file (5000frames) and here is my avs file.

DirectShowSource("1.GRF",video=true,fps=23.9759856527702,audio=false)
#deinterlace
crop( 0, 22, 0, -22)
Spline16Resize(1280,688) # Spline16 (Neutral)
#denoise


My controller command is like this..
controller.exe --first "--level 4.1 --threads 1 --thread-input --bframes 3 --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --no-ssim" --second "--level 4.1 --threads 1 --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --8x8dct --no-fast-pskip --progress --no-psnr --no-ssim" --avs "D:\Sample\default.avs" --output "D:\Sample\farm.mkv" --bitrate 1375kbps --batch 1000 --split 250 --thresh 20.0

after the controller is started..and log shows
.................
2008-04-01~02:15:52.78 0 @ INFINITY
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 1009 @ 13.58
2008-04-01~02:15:52.78 Empty
2008-04-01~02:15:52.78 2720 @ 48.69
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 3716 @ 17.03
2008-04-01~02:15:52.78 Empty
2008-04-01~02:15:52.78 4539 @ 13.07
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 5599 @ INFINITY
2008-04-01~02:15:52.78 0% done (0 / 5599) at 0.00 FPS
2008-04-01~02:15:52.78 Last updated: 0:18
2008-04-01~02:15:52.78 ---------------......................---------------...........---------------
2008-04-01~02:15:52.78 Agent | FPS | Description
2008-04-01~02:15:52.78 Recent errors:
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

The controller sort of hangs there and there is also no any response on the agent window. I checked the agent port bonding by telnet, and it is fine. Since I am a newbie, it is pretty hard for me to nail down this problem. Could someone give me a little help? Will be really appreciated. :):thanks:

omion
1st April 2008, 08:41
Hello, I am newbie here and just went through the whole threads. There are too many stuff and almost totally confused me.
Me too.:eek: I'm trying to get rid of the complicated stuff in my single-pass version, but it may take a while to get into the 2-pass one...



after the controller is started..and log shows
.................
2008-04-01~02:15:52.78 0% done (0 / 5599) at 0.00 FPS
2008-04-01~02:15:52.78 Last updated: 0:18
2008-04-01~02:15:52.78 ---------------......................---------------...........---------------
2008-04-01~02:15:52.78 Agent | FPS | Description
2008-04-01~02:15:52.78 Recent errors:
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~

It looks like the controller never gets the agent's info from the config file. It should say something about the agent right after the "Agent | FPS | Description" line. Most likely it's not using the config file you made. What is the location/name of the controller's config file?

Try adding "--config config-file-name.xml" to the controller's command line (replacing config-file-name.xml with the actual name, of course ;)) If that doesn't work, then I'll have to think of something else...

Roswellian
1st April 2008, 08:59
Omion, thanks for the quick reply. Actually I did not change config file name and config.xml is located in the same folder with agent/controller, but i still make a quick try as you mentioned. It is still not working. :-(

Edit: Just went through the whole threads again, and it was adhoc parameter problem. Now it is solved. :P

kenji123
9th April 2008, 22:15
I've integrated x264farm into my gui, i'm hopeing if someone could try it out so that i could get some feedback. its setup as controller based encoding. i've done tests using 3 pcs and it works ok. but would like to know how it works on a broader range of pcs.

file 1 - this runs on the controller pc (http://kenji123.uni.cc/ani-x264/Ani-x2641500.7z)
file 2 - copy this to all the agent pcs (http://kenji123.uni.cc/ani-x264/Ani-x264FARMClient2000.7z)

if needed, view this small tutorial to get it started; here (http://kenji123.uni.cc/ani-x264/guides/Encoding with x264Farm/index.html)

RickA
9th April 2008, 23:08
Greets,

Thank you so much for the GUI. Will give it a try as soon as my current encode finishes. I love the idea of being able to use multiple computers to work together on the same encode. :-)

Cheers,
Rick

Additional:
Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means? Will go back and read more before I start asking router questions. :-)

SpAwN_gUy
10th April 2008, 15:55
Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means?em.. how does AC3-Audio cooperates with VideoEncoding?

SpAwN_gUy
10th April 2008, 15:57
I've integrated x264farm into my gui, i'm hopeing if someone could try it out so that i could get some feedback. its setup as controller based encoding. i've done tests using 3 pcs and it works ok. but would like to know how it works on a broader range of pcs.

file 1 - this runs on the controller pc (http://kenji321.uni.cc/Ani-x2641500.7z)
file 2 - copy this to all the agent pcs (http://kenji321.uni.cc/Ani-x264FARMClient2000.7z)

if needed, view this small tutorial to get it started; here (http://kenji123.uni.cc/ani-x264/guides/Encoding with x264Farm/index.html)i've seen your GUI .. like in the start of the thread :) ... and just "before" meGUI-mod developement start..
i guess i'll take another shot.. and try it .. again :)

[P]ako
11th April 2008, 02:52
Greets,
Additional:
Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means? Will go back and read more before I start asking router questions. :-)

Well, x264 is just for video encoding. You still need to encode the audio with the program(s) of your liking to any of the formats that MKV supports and finally mux it with the video.

Razorholt
11th April 2008, 06:22
Date: #Apr.11.2008#
Time: [12:14 a.m.]

Stack Trace: at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at ani_x264farmclient.ani_x264farm.listener() in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 423
Message: Value cannot be null.
Parameter name: path

---------------------------------------------------------------------

Listening for server connection

Did I miss a config step somewhere? I followed your guide tho :)
Also, I got an error message when I want to minimize the window on the client's machine.
************** Exception Text **************
System.IO.FileNotFoundException: Could not find file 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'.
File name: 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Drawing.Icon..ctor(String fileName, Int32 width, Int32 height)
at System.Drawing.Icon..ctor(String fileName)
at ani_x264farmclient.ani_x264farm.OnResize(EventArgs e) in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 562
at System.Windows.Forms.Control.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Thanks,
- Dan

kenji123
11th April 2008, 14:49
Date: #Apr.11.2008#
Time: [12:14 a.m.]

Stack Trace: at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at ani_x264farmclient.ani_x264farm.listener() in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 423
Message: Value cannot be null.
Parameter name: path

---------------------------------------------------------------------

Listening for server connection

Did I miss a config step somewhere? I followed your guide tho :)


No you didn't.:) This one is my fault, a required directory wasn't created by the program.




Also, I got an error message when I want to minimize the window on the client's machine.
************** Exception Text **************
System.IO.FileNotFoundException: Could not find file 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'.
File name: 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Drawing.Icon..ctor(String fileName, Int32 width, Int32 height)
at System.Drawing.Icon..ctor(String fileName)
at ani_x264farmclient.ani_x264farm.OnResize(EventArgs e) in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 562
at System.Windows.Forms.Control.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Thanks,
- Dan

Again my fault. When the program minimizes, it loads an icon for the system tray, which i failed to include.:) This error would of also occurred with the controller program. (i've fixed it so that you don't need the icon file.)

anyways;
i've fixed the bugs, so you can replace the old exes with the ones in this file here (http://kenji123.uni.cc/ani-x264/Ani-x2641500FARM2000_exeonly.7z)

the links in my previous post have also be updated.

thanks for trying it out and providing feedback :)