x264farm: distributed encoding

[Humbula]

Originally Posted by foxyshadis

x264farm just processes the first pass stats files to come up with each segment's relative sizes.

And how can I calculate a segments size using the statsfile?

[omion]

Originally Posted by Humbula

to be honest i dont understand the code. strange language...

omion, help me...

The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it

And how can I calculate a segments size using the statsfile?

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused.

[Humbula]

Originally Posted by omion

The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it

Originally Posted by omion

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused.

Thanks, I'll have a look

[akupenguin]

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

[omion]

Originally Posted by akupenguin

Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

Ah. Yes. I just realized that I actually answered a different question than what was asked. The size of the first pass frames is in itex+ptex+mv+misc (which was the question I was answering), but that does not directly correspond to the second pass frame size (which was the question asked). The stats file is processed through the functions in ratecontrol.c before the optimal second pass frame sizes are known.

Even after processing by the ratecontrol functions, there is no guarantee that the frames will actually be that size. In fact, since x264 can't directly set the number of bits used in a frame, it will always be a little off.

[taemun]

Hey all.

I'm new to x264 encoding in general, (have worked out what most of the buttons in MeGUI etc do), and given that I have several computers lying around here, I thought that there should be a relatively easy way to get x264 to run over all of them at once.

Hence, found this project.

I've had it running, although I'd have to admit that a few things aren't clear to me:

1> Unless I specify a rate control method in the -1 "blah blah", it doesn't do anything at all.... was this intended? What value should I be using in here for a good quality 720p rip (is --qp 26 sufficient?).

2> I'd like a little more control over the IP that the controller and agents bind to ... I have multiple IP addresses, that will go differing routes to get to the clients. One of those interfaces is gig, one is 100M. I'd rather be able to force it to go the gig path. This may be related to my inability to work out the networking parameters in the controller - the first time I tried I couldn't get anything aside from adhoc working - this suits my purposes fine, as you can kill the processes remotely, but adhoc can't specify the controller IP to be bound to.

3> I'd like to be able to change the agents list/the number of x264 instances per agent whilst the controller is running. You can sort of do this by running in adhoc, and just starting/closing/restarting the agent whenever you want to change something, but that only works (for me) if you can force the IP that the adhoc controller is running on.

4> What is AVS2YUV doing here? I presently have three instances of x264 running over two computers (one on controller, ~500MB/ram, two remote, ~230MB/ram each), and the controller also has three avs2yuv's running:
One of 766MB (no CPU-time, unless an agent recently finished a block).
Two of 550MB (3~13% CPU-time). Presumably these are frameserving across the network (controller based encoding).
[as I've been typing this, both of the 550MB processes have died, and two new ones of around 350MB have come up]

Reason I don't understand, is because x264 can handle an .avs input directly, why is there a need for a frame server here? (proviso: during agent-based encoding)
On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

5> I can't seem to get agent-based encoding working, the agent throws the error:

Avisynth error:
DirectShowSource: Couldn't open GRF file.U:\mraw\00011.m2ts.grf:
Class not registered

Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent. Aside from the small delay that occurs when dumping back the stats file, or the resulting .mkv files in the second pass. IF you're worried about that delay in the second pass, I can see why you might use pad=1; number=1. (btw, why not simplify that into, numberFirstPass=1; numberSecondPass=2?)

Sorry about the length of this, and I hope someone can help me understand. Thanks.

t

[Adub]

Originally Posted by taemun

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent.

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

[foxyshadis]

Originally Posted by Merlin7777

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

Depends on if the input is bottlenecked or not. (Combine MT+x264 threading and you get better results.)

Originally Posted by taemun

On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Avisynth caches. You can use setmemorymax to control that, since avisynth caching ignores multiple instances. (This is even worse on 2.58, where default is half the available system memory, not one quarter. If they all initialize at once, they all see the same available memory.) Even if frames are never needed again, they'll sit in cache until their number comes up when the maximum memory limit is reached. Then it just stays at that limit until you're done encoding, mostly.

[omion]

Originally Posted by taemun

Hey all.

I'm new to x264 encoding in general, (have worked out what most of the buttons in MeGUI etc do), and given that I have several computers lying around here, I thought that there should be a relatively easy way to get x264 to run over all of them at once.

Hence, found this project.

I've had it running, although I'd have to admit that a few things aren't clear to me:

1> Unless I specify a rate control method in the -1 "blah blah", it doesn't do anything at all.... was this intended? What value should I be using in here for a good quality 720p rip (is --qp 26 sufficient?).

That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

I generally use a lower setting (--qp 22 or something) but do a few tests to see what looks good for you.

2> I'd like a little more control over the IP that the controller and agents bind to ... I have multiple IP addresses, that will go differing routes to get to the clients. One of those interfaces is gig, one is 100M. I'd rather be able to force it to go the gig path. This may be related to my inability to work out the networking parameters in the controller - the first time I tried I couldn't get anything aside from adhoc working - this suits my purposes fine, as you can kill the processes remotely, but adhoc can't specify the controller IP to be bound to.

Currently it's set to bind to any IP address, since I didn't need the ability to change it. That feature would be nice for some people, though, so I'll add it in. However, it's probably not going to get into the 2-pass version of x264farm any time soon, as I'm mainly working on the 1-pass version now.

3> I'd like to be able to change the agents list/the number of x264 instances per agent whilst the controller is running. You can sort of do this by running in adhoc, and just starting/closing/restarting the agent whenever you want to change something, but that only works (for me) if you can force the IP that the adhoc controller is running on.

I have (pretty buggy) support for that in the 1-pass version. It does still need you to restart the agent every time you want to change the settings, but it generally works. In non-adhoc mode for the 2-pass version, the agent list is defined immutably in the beginning of the encode. Changing the way that works would be fairly difficult.

4> What is AVS2YUV doing here? I presently have three instances of x264 running over two computers (one on controller, ~500MB/ram, two remote, ~230MB/ram each), and the controller also has three avs2yuv's running:
One of 766MB (no CPU-time, unless an agent recently finished a block).
Two of 550MB (3~13% CPU-time). Presumably these are frameserving across the network (controller based encoding).
[as I've been typing this, both of the 550MB processes have died, and two new ones of around 350MB have come up]

Reason I don't understand, is because x264 can handle an .avs input directly, why is there a need for a frame server here? (proviso: during agent-based encoding)
On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Any controller-based encode needs to start up its own instance of avs2yuv to render the AVS file. There is also one extra avs2yuv which the controller consults for finding scene cuts. The memory usage is always a bit large by default (since Avisynth is set up to cache everything it gets its hands on) but you can limit that by using SetMemoryMax(...) in the AVS file. You are correct that frameserving should use very little RAM - every output frame is used exactly once, and in order. Therefore you should see no performance decrease by limiting the amount of memory Avisynth uses.

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$

I can't really help you there. I don't know the inner workings of graphedit or MeGUI... Generally errors like that are due to some path being hard-coded in the AVS (or an included file), but since you have the agent's U:\ mapped to the controller's U:\, I don't really know.

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent. Aside from the small delay that occurs when dumping back the stats file, or the resulting .mkv files in the second pass. IF you're worried about that delay in the second pass, I can see why you might use pad=1; number=1. (btw, why not simplify that into, numberFirstPass=1; numberSecondPass=2?)

I honestly don't know how they compare. When x264 first added threading support I tested it out, and using multiple single-threaded x264 processes worked a little faster. But then the threading model was completely changed around and it looks like it's quite a bit better now than it used to be. Test it out and report back! (oh, and use "--threads auto", or it will force all agents to use the same number of threads, which is probably not what you want)

[foxyshadis]

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$

Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

[taemun]

Originally Posted by foxyshadis

Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

Yeah I came to that conclusion late last night. CoreAVC was only installed on the controller, so of course it failed. Pity I didn't think about that at the time (after trying to dig through the .grf with notepad :P). I'd forgotten that I had used something a little more exotic when I made the graph.

Originally Posted by omion

That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

Yeah I did see that helpful warning, was more wondering whether I was doing something wrong, because the FAQ didn't mention it.

Originally Posted by omion

That feature would be nice for some people, though, so I'll add it in.

Thanks, that'd be great.

Originally Posted by omion

There is also one extra avs2yuv which the controller consults for finding scene cuts.

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t

[omion]

Originally Posted by taemun

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

That would certainly make the most sense, wouldn't it? The only real reason I didn't do that is that I had abstracted away the avs2yuv process so far that it hid all the parts that I could have closed. Rather than attempting to do that, I decided instead to try and get rid of the splitter altogether. After much hacking away, I finally decided to do a ground-up rewrite to the entire program, which turned into x264farm-sp (which doesn't have a splitter). The 2-pass x264farm has since fallen by the wayside.

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

That is due to an annoying bug that x264 will open the stdin handle in text mode, meaning that it will interpret ASCII character 0x1A as an EOF marker and stop encoding. (it also turns 0x0D0A into 0x0D) MSVC builds don't have this issue, though, as there are a few MSVC-specific lines in x264 to switch to binary.

This is only a problem in controller-based mode, though. You may use anything you want if you ensure that the computer is using agent-based encoding.

(Note that it's not really a bug in x264; I think stdin and stdout are usually opened in text-mode in Windows. It's just a bit odd that x264 expects binary data to pass through them...)

It should be an easy fix for making it work in cygwin/msys, but I really don't know enough C to say for sure.

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t

I'm not really sure what the most current version of MSVC x264 is. I gave up making my own builds a long time ago, since x264 is really only tested against GCC, and some of the changes would break MSVC for long periods of time. Not knowing a lot of C, I was clueless to fix it in these cases. Since then, I have had no need for controller-based encoding (on Windows boxes), so I haven't kept up with the progress. I'll have to defer to SpAwN_gUy to answer that question.

If you can get agent-based encoding up and running, then you can bypass this restriction. That would probably be the best course of action right now. (if the CoreAVC fix made agent-based encoding work, then you can use anything you want)

[SpAwN_gUy]

i've just moved to the new PCbox ... so i've lost the tracks of the threads...

okay, since b0bor started to post patched sources (damn git..) it will not be too hard to make MSVC builds ...

good news are: meGUI-x264farm-mod (ma-a-n... this is quite a huge name...) is still indev.. i've moved 80% of code to the latest (like two weeks ago) meGUI revision... so.. some may wait for a new version soon..

/*offtop... as far as i'm learning bash and perl (and doing some fancy scripts..).. C# is my "back to the reality" thing... so more to a hobby.. but i'll catchUP ... and i'm on Vista x64 now..

[Adub]

Good to hear from you again. As you know, looking forward to the mod!! Thanks again.

[thebigfatgeek]

Hi, I have just installed the agents and the controllers, modified th config.xml files and I keep on getting the same result. I cannot see the error, or that anything is happening (except some processor activity on controller.exe and avs2yuv, but after 0:46 seconds, it stops, it does not complain (which is nice) but it also does not work (which is not nice)

This is the output from the out-dump.txt file:

2008-06-07~23:09:23.53 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.53 Last updated: 0:45
2008-06-07~23:09:23.53 ----...----________________________________...............-----...............
2008-06-07~23:09:23.53 Agent | FPS | Description
2008-06-07~23:09:23.53 Recent errors:
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 SPLITTER MUTEX unlocking range_list_mutex and yielding
2008-06-07~23:09:23.53 SPLITTER MUTEX sleeping for 1/10 second
2008-06-07~23:09:23.64 SPLITTER MUTEX locked range_list_mutex after yielding
2008-06-07~23:09:23.64 e 0
2008-06-07~23:09:23.64 f 0
2008-06-07~23:09:23.64 q 3
2008-06-07~23:09:23.64 w 0
2008-06-07~23:09:23.64 es 3
2008-06-07~23:09:23.64 fs 0
2008-06-07~23:09:23.64 qs 8
2008-06-07~23:09:23.64 ws 0
2008-06-07~23:09:23.64 Splitter: current batch length is 2116
2008-06-07~23:09:23.64 Splitter: try_queue_after_full_or_working
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 Splitter: try_extend_small_thresh_queue
2008-06-07~23:09:23.64 extend_small_thresh_queue found no ranges of sufficient patheticosity
2008-06-07~23:09:23.64 Splitter: try_split_largest_range
2008-06-07~23:09:23.64 Splitter: wait_on_full
2008-06-07~23:09:23.64 SPLITTER MUTEX broadcasting range_list_empty_condition
2008-06-07~23:09:23.64 0 @ INFINITY
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 2502 @ 7.56
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 4607 @ 10.63
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 6857 @ 4.69
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 9282 @ 2.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 11709 @ 7.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 14176 @ 2.42
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 16791 @ 6.33
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 19347 @ 7.58
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 22022 @ 9.34
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 24696 @ 4.79
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 27246 @ 3.65
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 37171 @ 11.85
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 39525 @ 6.16
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 49544 @ INFINITY
2008-06-07~23:09:23.64 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

and my controller was started with:

Controller>controller -B 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --me range 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weigh tb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --m e umh --merange 12 --progress --no-psnr" --avs "24.avs" -o "24.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

Any ideas or comments would be appreciated

Cheers

[ajp_anton]

Bug!

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network.

Apparently I can get around this by creating a .avs file that finds its inputs via the network, and copy that file onto every computer, even the local one (since it seems intelligent enough to check if the files are the same... =)

Otherwise this thing is really the best thing ever. Too bad it's not until now that I really needed it as it took only 10 mins to figure everything out and get it working.

[SpAwN_gUy]

Originally Posted by ajp_anton

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network.

i have no problem with that ...
which nice are you using? (the one in the first post) jr the original? .. 'cause we had some problems using the original one.. and some problems using the tiny one(1st post)

currently i'm using the one in the first post... and i'm doing agent-based..

[SpAwN_gUy]

Originally Posted by thebigfatgeek

2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

[SpAwN_gUy]

oh.. jeah.. right.. the mod.. i need to start MSVC... and stop doing nothing...

btw.. about builds of x264... it is reported, that MSVC builds (with current revision of x264) are TOTALLY BROKEN.. and must not be used ( ...
but the good nwes are.. that x264_mingw_stdin_patch is IN the TRUNK .. (yeah, yeah,.. thanks to me.. but you are welcome..) so all the builds, that are currently made.. (like on x264.nl and all the patched ones) need to be checked(at first).. and then.. well.. the must be working and compatible with controller-based x264farm encoding process..

[thebigfatgeek]

Originally Posted by SpAwN_gUy

controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

I can get it to see the agent, but I just cannot get a controller startup command that will work:

nice -n 20 "x264.exe" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --s
bme 1 --analyse none --progress --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.
--qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass
1 --stats ".\temp\stats b26912.txt" -o NUL - 640x496
264 [error]: no ratecontrol method specified
264 [error]: x264_encoder_open failed