x264farm: distributed encoding

[whatever-ever]

Originally Posted by omion

That is a bit odd, but sometimes it's SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("..\..\..\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?

Originally Posted by omion

[...] make sure the .d2v file references its source files relatively too

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

Originally Posted by omion

Uhh... so it can't find the d2v file, but it works anyway?

it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.

Originally Posted by omion

judging by your paths, the first one (to x264farm\agent) probably won't be used to store any avs files.

Thanks! It also loads the network path each time (only place there is a avs), so you're right, I should just delete the other.


PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! )

[omion]

Originally Posted by whatever-ever

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?

All those ..\..\.. mean to go from the .avs directory (C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller) to the directory of the .d2v file (C:\!RAW).

If you have the C:\!RAW directory shared somewhere, then the relative path will still make it point to the right directory.

Think of it this way: the way you had it, your laptop is seeing the .avs file refer to "C:\!RAW\test2.d2v" Does that path exist on the laptop? If not, then you have to make sure you share the proper path.

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

If you are sharing the directory "C:\!RAW" as "\\NETWORKNAME" then yes.

it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.

I thought so. Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.

PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! )

Yup. I noticed that you had the <agentpipe>1</agentpipe> line in your config.xml files, which needs avs2yuv to function. <agentpipe> is only needed if you have a 64-bit x264, but 32-bit avisynth. If you have 32-bit everything then <agentpipe> will only add a layer of complexity.

Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore

[whatever-ever]

Originally Posted by omion

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.
[...]
I thought so. Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.
[...]
Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).

I'll shut up and run a test

THANKS!

[SpAwN_gUy]

.. maybe i'll write a tutorial.... or maybe not ...

so,.. i've come back with the tests of 1.08..

AWESOME!!!!!!!!!!!!!
i think you've modified x264... 'cause it RUNS pretty well,.. ) ..
i've encoded that problemous file... but i've messed up with bitrate, so the file came oversized.. now i'm ReEncoding it..

minuses: agents are suddenly exiting(started over network). I have DUMP...
controller suddenly hanged yesterday.. on exiting first pass.. i dunno why. it was running smoothly, and when i left - something happened(i don't have dump..)

and i've made a some kind of contest.. calculated sum of fpses of agents.. (2x and 3x).. 2x =( 1, pad1) 3x = (2, pad1)
yeppie... now i have one CoreDuo2 !!!!!.... it's freaking awesome.. (and two or three are coming)

                                   Agent | FPS 
p4 Core Duo2 E6400 2.13GHz/2Gb/W32bit 3x | 7.05
p4 3000/1Gb/W32bit                    2x | 3.05
AMD 3000+/1Gb/W64bit                  2x | 2.86
AMD 3500+/1Gb/W32bit                  2x | 2.83
AMD 3700?/1Gb/W32bit                  2x | 2.67
p4 3000/1Gb/W32bit                    2x | 2.23
p4 3200/1Gb/W32bit                    2x | 1.48
p4 1800/512/W32bit                    2x | 0.89
13.09 FPS total 
now it' running ~16fps

now.. adout gui..
i'm a bit confused and lost in translation... what ARE the Assorted params??? I should never use them in passParams, or controller DELETES them from pass params?? .. and do i need "--threads 2" and "--thread-input" in passParams?

[SpAwN_gUy]

Originally Posted by whatever-ever

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).

the simpliest way to test will agent-based encoding run or not ("is everything OK? and will it run??") is to Open desired AVS from the Agent Computer(at least one computer) though the network.. if it runs - then OK.. it will..(well, about 30% for sure).

on each agent PC you will need AviSynth Installed.

ok, ok,... the Tutorial :

dir structure:

D:\DVD-RiP\!!!x264farm-LOCAL-DiR\
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\file.d2v 
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs
.vob are in:
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\iN\VIDEO_TS\
AviSynth plugz are in
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\filters

in the network paths it looks like:

\\controllerpc\x264farm\file.d2v
\\controllerpc\x264farm\fileAVS.avs
.vob are in:
\\controllerpc\x264farm\iN\VIDEO_TS\
AviSynth plugz are in
\\controllerpc\x264farm\filters\

.d2v internals (the head of..) should look like this
UPD..(2007-10-02): Don't forget to download DGIndex 1.5.0 beta 3 .. it has "Relative paths saving"
here

DGIndexProjectFile13
5
iN\VIDEO_TS\VTS_01_1.VOB
iN\VIDEO_TS\VTS_01_2.VOB
iN\VIDEO_TS\VTS_01_3.VOB
iN\VIDEO_TS\VTS_01_4.VOB
iN\VIDEO_TS\VTS_01_5.VOB

Stream_Type=1           
......

.avs is

SetMemoryMax(32)
LoadPlugin(".\filters\DGDecode.dll")
#  SOURCE
mpeg2source("file.d2v", info=3)
...... etc

agent Config bases part lokks like

        <bases>
                <base>\\controllerpc\x264farm\</base>
                <base></base>
        </bases>

and i start the controller with

controller.exe --first "--bframes 3 --b-pyramid  --direct auto --filter 1,1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-psnr  --no-ssim" --second "--ref 16 --mixed-refs  --no-fast-pskip  --bframes 3 --b-pyramid  --b-rdo  --bime  --weightb  --direct auto --filter 1,1 --subme 7 --trellis 2 --analyse all --8x8dct  --vbv-maxrate 25000 --me umh --threads 2 --thread-input  --progress  --no-psnr  --no-ssim" --avs "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs" --output "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileOutput.mkv" --bitrate 1375kbps

all agents are encoding agent-based ...

BTW... just now 2nd pass finished..(first pass was about ~20fps)
START: 1000AM
END: 1307AM
this encode took about 20 hours to encode on my PC..

now i'm encoding one more time.. and i think most of agents - exited ... first pass is only using 3 of manyPC's..

[whatever-ever]

Thanks for the tutotiral Spawn!

I'm not sure what did the trick, but LOCALLY it's going great now. Encoding with 20.7 + 13.16fps on the two local agents, VERY fast (same as when encoding without farm).

And the output is "better" on the local agents.
Reporting "encoded frames: 103/187" and updating continuously.

It might be the controller cmd line that's doing the improvements, but I'm not sure.

However I can't get the external agent to follow.
Getting a lot of

Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255\")

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Now it's encoding at 0.44 and 0.57 locally and 3-4fps externally, okay... This is getting wierder and wierder!

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1

[SpAwN_gUy]

Originally Posted by whatever-ever

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1

it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw... What is "vdvmax"??

and... temp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

[whatever-ever]

Originally Posted by SpAwN_gUy

it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw. twmp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

PS: exiting doesn't seem to be a problem - anymore. Don't know why, but all jobs are running smoothly, just not very fast.

[whatever-ever]

Originally Posted by SpAwN_gUy

btw... What is "vdvmax"??

vbv

[omion]

When you guys say it's exiting, does it put up any warnings or just die?

Originally Posted by whatever-ever

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

There is no way to tell x264farm to not do the second pass, but you can stop it manually and use the stats file it creates to do a separate second pass.

The created stats file is here:
<controller_temp_dir>\<lots_of_numbers> <name_of_avs>\1-<lots_of_numbers>.txt

For example,
E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs\1-4E2586CB8D8815243D6B8F08CE7FEA77.txt

Exactly which string of numbers goes with which encoding is displayed at the beginning of x264farm:

AVS temp dir is E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs
First string is "...", with MD5 "4E2586CB8D8815243D6B8F08CE7FEA77"

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Uhh... I think it's not encoding fine. Without avs2yuv, there is no chance that agentpipe=1 is going to encode at all. With agentpipe=0, the existence of avs2yuv won't matter.

I'll have to look at the dump file to say more, though.

[whatever-ever]

Setting up a mac, where do I get the x264 encoder for that?

[whatever-ever]

Originally Posted by omion

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

Yes all machines are 100pct. avs2yuv is taking a lot of cpu usage sometimes though (up to 50 pct.), however agents (not running avs2yuv) are slow too (not as slow, but still.. not so impressive)

I've saved the out-dump, but only from the controller, I'll check if it's usable and mail it.

[omion]

Originally Posted by whatever-ever

Setting up a mac, where do I get the x264 encoder for that?

To build it yourself, you need x-code (on the DVD that came with the mac) and SVN (over here)

Check out x264 from the repository:

svn co svn://svn.videolan.org/x264/trunk x264

Go to the x264 dir and type:

./configure
make
sudo make install

Assuming everything is set up right, that will get you the x264 binary.

You'll also need the agent for the mac. You can get 1.08 here. I didn't put a config file in the archive, but it's set up the same way as in Windows.

[SpAwN_gUy]

Originally Posted by omion

When you guys say it's exiting, does it put up any warnings or just die?

well.. few people said that it shows something, but they didn't remembered that..

and.. x264 takes about 30% of CPUtime..
i'll do some testings about agentpipe=0.. and will send logs to you..

...here we go

firstpass
pipe=1 nice=20 - avs=80% x264=27% 15.5fps
pipe=1 nice=0  - avs=50% x264=50% 16fps
pipe=0 nice=20 - error
pipe=0 nice=0  - x264=98% 17fps

secondpass
pipe=1 nice=20 - avs=10% x264=84% 4.3fps
pipe=1 nice=0  - avs=10% x264=85% 4.11fps
pipe=0 nice=20 - error
pipe=0 nice=0  - x264=98% 4.05fps

fistpass-local priority: normal
x264=72% 22fps

secondpass-local priority: normal
x264=80% 4.5fps

agent is p4 1800/512/W32bit
local(controller) is p4 2400/512/W32bit

... he-he... dump does not provide anything ... here are the screen dumps(theese were not provided in eMail):

Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
  nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt"  --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

[omion]

Originally Posted by SpAwN_gUy

Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
  nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt"  --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

Oh! It looks like nice.exe does not appreciate running things from network paths. I'll see what needs to be done.

@whatever-ever:
I'll take a look at those logs you sent me. There should be something I can do about the speed.

[SpAwN_gUy]

yeah,.. and..
it seems, that x264farm has something wrong with Oversize .. i had 2 encodes of the same source(with different x264profiles), with same bitrate, and the files were
a) different in size (~30Mb)
b) they were oversized.. about ~60Mb to ~93Mb...
why?

[omion]

@SpAwN_gUy:

Well, the files are going to be different, since the scenes never finish encoding in the same order. They should not be that different, though.

It also looks like my bitrate-multiplier calculation is not quite right... There is not enough feedback to correct for any bitrate discrepancy, so x264farm is always trying to pull the bitrate up. I'll try to fix it in the next release (or at least make it better)

To fix it in the short-term, you can increase the number of 3rd-pass scenes.
The settings that control the third pass:
--3thresh How far off the bitrate must be in order to consider for 3rd pass. 0 = completely wrong, 1 = exactly right. Default = 0.8
--3gops The maximum number of scenes to consider for the 3rd pass. Default = a lot
--3ratio Same as --3gops, but as a percentage of the total scenes (well, not really percentage, but from 0 to 1) Default = 0.05.

The default parameters will re-encode up to 5% of the total scenes, but only if the scene's ratio expected_bitrate / actual_bitrate is less than 0.8.

To make the bitrate more exact, raise both --3thresh and --3ratio. Something like --3thresh 0.95 --3ratio 0.2

Meanwhile, I'll try to make the ratecontrol smarter.

[whatever-ever]

My speed was way up on a test I ran this friday. I'm trying to run another, but I'm having a little trouble with the avs somehow. Looking into it now

[SpAwN_gUy]

so.. how the loop goes(or a hang)
controller hangs on close_connection ... of agents at the end of second pass..
and after the restart.. it starts(or continues) to reEncode something.. it is 99% of an encode...(with modified 3tresh and.. as omion advised).. it happened 3 times for now.. (yeah i know... I have Lots of those errorous sources and everything is going crazzyy.. but i think it is good in terms of testing.. )... so.. and keep going again and again... i'll modify .bat file.. and post some results tomorrow

//me sorry.. i'm a bit drunk right now.. but i'm allmost Russian(Belarussian.. telling the truth), so i can work and programm..
soory my girl Friend waits for me near work entrance.. so.... ..

[SpAwN_gUy]

Yeah... and Is it possible, to run NOT Only the x264.exe with nice, but avs2yuv also?

'cause if agentpipe=1 and nice=20 then x264 is running in lowest priority, but avs2yuv is running Normal, and it slows down overall system performance,.. so my people can't work ...

and.. that option,.. that asks for "Do you really want to ReEncode the same AVS?"(or something)..
can you check for th SAME Bitrate and Same Filename? .. 'cause i had an encode with different bitrate and different output filename... it was nice, that i didn't need to make firstpass again,.. but..
/*played with nice..
well.. nice REALLy does not like NetPaths.. i've played with different variations of comLine.. and nice does not want to start avs2yuv with netPath param..
error is: "AviSynth can not find the file D:\controllerpc\x264farm\...."
and batch file was: nice -n 20 avs2yuv ... "\\controllerpc\x246farm\.avs" | nice.. x264...