x264farm: distributed encoding [Archive] - Page 4

whatever-ever

13th February 2007, 20:08

I need to configure my setup a little better (possibly it's my avs that's causing problems).

Currently the avs is located on controller only, and loading plugins via network (even though it's on the local machine), and d2v is linked to locally (not via network). It is a complex as it sounds, but it was the editing in the avs to that, which made the external agent catch on.

wierd thing is it reports a lot about d2v file not being found (on external) which is obviously because of the pathing you might say, but no... i tried both \\network\dir\ and C:\local\dir\, same error -- HOWEVER, the controller reported an increasing boost (going from 1 hour to 30min to 4 (!) min ETA).
And get this: the boost came when i connected the external agent on wifi!!

Since I got it working kinda "half-ass", changing avs while encoding (when restarting the external agent), I was wondering if someone could help me set up the avs properly.

Current setup:

controller config:
<config>
<temp>C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller\temp</temp>
<agents>
<agent name="LOCAL">
<ip>127.0.0.1</ip>
<port from="50702" to="50702" />
<number pad="1">1</number>
</agent>
<agent name="LAPTOP">
<ip>10.0.0.7</ip>
<port from="50701" to="50701" />
<number pad="1">1</number>
</agent>
</agents>
</config>
LOCAL agent config:
<config>
<temp>C:\!RAW\__x264farm\x264farm-1.08-161_win32\agent\temp</temp>
<port from="50702" to="50702" />
<x264>x264.exe</x264>
<nice>5</nice>
<base>C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
<agentpipe>1</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>
LAPTOP agent config (external):
<config>
<temp>C:\Documents and Settings\****\Desktop\x264farm\agent\temp</temp>
<port from="50701" to="50701" />
<x264>x264-622_omion.exe</x264>
<nice>5</nice>
<base>C:\Documents and Settings\****\Desktop\x264farm\agent</base>
<base>\\CONTROLLERPCNETWORKNAME\\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
<agentpipe>1</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

AVS (C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller\test.avs):
SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("C:\!RAW\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

The controller cmd line:
encode.bat:
controller.exe --avs %1 --bitrate %2kbps --first "--crf 19 --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 5 --ref 8 --me umh --bframes 7 --weightb --b-pyramid --analyse all" --second "--trellis 2 --bime --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 7 --ref 12 --me umh --bframes 7 --weightb --b-pyramid --analyse all --8x8dct" --output "output.mkv" --nocomp --batch 2500 --split 250
executed as:
encode test.avs 1224

The AVS is only located in the controller folder.

The goal: to encode agent-based.

I know it's a bit wrongly set up (the avs part, right?) but it kind'a worked.

First pass took about 50 mins to encode (don't know what the ??? went wrong there..) and second pass about 5 minutes.

Thanks in advance for any help.

PS: Thumbs up, omion :) I've been looking for a project as such for a long time. Thanks!

omion

13th February 2007, 20:34

ETA: 1:51:34How long is the movie? How long would it have taken with the same settings without using x264farm?

and that's on 6%, oh my.. GOD it's slow. ETA is rising for every pct. the encode is increasing. And this is the second pass. The first pass was not that fast either.

The setup is one controller + one agent on SAME machine (x264 running 50%)

I need to work a bit on getting my laptops working in the setup too, but even though it's only 50% locally it shouldnt be THIS slow. I must have done something wrong. I'll post configs and dump.txt later

That would definitely help.

Using x264farm on one computer is always going to be slower than not, simply because of the extra overhead. Also, x264farm will re-process any scenes that varied wildly from what was expected, which will result in a higher quality but which may take 10% longer than not using it.

A few more questions:
Are you using agent-based encoding? If the agent says something like this then you are:
Testing file "file.avs": FOUND!
2 agent-based encoding

What is your CPU usage like? Is it maxed out? You said x264 is using 50% of your CPU, but is something using the rest? (also, is it a dual-core?)

whatever-ever

13th February 2007, 20:55

How long is the movie? How long would it have taken with the same settings without using x264farm?

11min clip and normally it's 25fps for first pass, 11fps for second pass. so about 25 min.

Are you using agent-based encoding? If the agent says something like this then you are:
Testing file "file.avs": FOUND!
2 agent-based encoding
yes I am, I've updated my post to make it more specific, the first was embaressing, sorry.

What is your CPU usage like? Is it maxed out? You said x264 is using 50% of your CPU, but is something using the rest? (also, is it a dual-core?)
It's hyperthread, but not dualcore per say. Yes, 100% cpu usage during most of the first pass (not sure during second pass, it went very fast so I didnt get to check). 50% x264, 50% avs2yuv, BUT, x264 doesnt exceed 50%, and avs2yuv didnt run all the time (for quite some time cpu usage was only at 50% total).

I've updated with specs on my setup if it helps,

EDIT: laptop cpu usage is roughly 20-35%

thanks for the quick reply

omion

13th February 2007, 22:11

I need to configure my setup a little better (possibly it's my avs that's causing problems).

Currently the avs is located on controller only, and loading plugins via network (even though it's on the local machine), and d2v is linked to locally (not via network). It is a complex as it sounds, but it was the editing in the avs to that, which made the external agent catch on.
That is a bit odd, but sometimes it's necessary. However, I don't think that the avs will work on remote computers. It's best to use relative paths wherever possible.

For example:
SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("..\..\..\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

And make sure the .d2v file references its source files relatively too; when you create the file, turn OFF "Use Full Paths". If you open the .d2v file with a text editor, the VOB files should be referenced relatively.

wierd thing is it reports a lot about d2v file not being found (on external) which is obviously because of the pathing you might say, but no... i tried both \\network\dir\ and C:\local\dir\, same error -- HOWEVER, the controller reported an increasing boost (going from 1 hour to 30min to 4 (!) min ETA).
And get this: the boost came when i connected the external agent on wifi!!
Uhh... so it can't find the d2v file, but it works anyway? I must say, this doesn't make a whole lot of sense. Care to explain this a bit more? Also, were those times that you posted the controller's ETA or the actual time?

Since I got it working kinda "half-ass", changing avs while encoding (when restarting the external agent), I was wondering if someone could help me set up the avs properly.

If you can get any encodes out of an agent, then you have set up everything correctly. The relative paths I mentioned before will ensure that the avs file will work on remote agents.

The goal: to encode agent-based.

I know it's a bit wrongly set up (the avs part, right?) but it kind'a worked.

First pass took about 50 mins to encode (don't know what the ??? went wrong there..) and second pass about 5 minutes.

Thanks in advance for any help.

PS: Thumbs up, omion :) I've been looking for a project as such for a long time. Thanks!

Looks pretty good. However, I noticed that you have two <base> lines in your laptop config. If you wanted to have the option of encoding from either of them, you need to use the <bases> element, like so:
<bases>
<base>C:\Documents and Settings\****\Desktop\x264farm\agent</base>
<base>\\CONTROLLERPCNETWORKNAME\\!RAW\__x264farm\x264farm-1.08-161_win32\controller</base>
</bases>

However, judging by your paths, the first one (to x264farm\agent) probably won't be used to store any avs files.

whatever-ever

13th February 2007, 22:58

That is a bit odd, but sometimes it's SetMemoryMax(64)
LoadPlugin("\\CONTROLLERPCNETWORKNAME\AviSynthPlugins\DGDecode.dll")
mpeg2source("..\..\..\test2.d2v")
crop(10,2,702,572)
LanczosResize(576,432)

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?

[...] make sure the .d2v file references its source files relatively too

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

Uhh... so it can't find the d2v file, but it works anyway?
it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.

judging by your paths, the first one (to x264farm\agent) probably won't be used to store any avs files. Thanks! It also loads the network path each time (only place there is a avs), so you're right, I should just delete the other.

PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! :))

omion

14th February 2007, 00:58

the ..\..\..\ is relative to the network path or..? ..\test2.d2v would mean the test2.d2v is refered to as \\CONTROLLERPCNETWORKNAME\test2.d2v, right?
All those ..\..\.. mean to go from the .avs directory (C:\!RAW\__x264farm\x264farm-1.08-161_win32\controller) to the directory of the .d2v file (C:\!RAW).

If you have the C:\!RAW directory shared somewhere, then the relative path will still make it point to the right directory.

Think of it this way: the way you had it, your laptop is seeing the .avs file refer to "C:\!RAW\test2.d2v" Does that path exist on the laptop? If not, then you have to make sure you share the proper path.

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.

So it would be the same if I wrote \\NETWORKNAME\test2.d2v in the AVS?

If you are sharing the directory "C:\!RAW" as "\\NETWORKNAME" then yes.

it found the avs fine, but it kept saying the d2v wasnt found or wasn't "valid" (don't remember the exact words, but that was basicly the saying). The ETA's as reported by the controller yes.
I thought so. ;) Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.

PS: I had to set up avs2yuv on the laptop too! It kept saying avs2yuv no such internal/external command (no, on the agent log, NOT controller. controller works fine with avs2yuv). Not that big a deal since it's working, I just thought maybe you had some comment about that? I can't see why the agent would need it as it ran agent-encode the entire time (and well... it definately wouldn't make controller encode on the agent! :))
Yup. I noticed that you had the <agentpipe>1</agentpipe> line in your config.xml files, which needs avs2yuv to function. <agentpipe> is only needed if you have a 64-bit x264, but 32-bit avisynth. If you have 32-bit everything then <agentpipe> will only add a layer of complexity.

Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore :cool:

whatever-ever

14th February 2007, 08:17

The easiest way to make it all work is to have the .avs, .d2v, and .vob files in the same (shared!) directory, then just omit all the paths when one references another.
[...]
I thought so. ;) Remember that there are really 3 source files that all computers need access to: the .avs file, the .d2v file that it refers to, and the raw .vob files that the .d2v refers to.

The error you are getting means that the laptop reads the .avs and tries to load "C:\!RAW\test2.d2v", which probably doesn't exist on the laptop.
[...]
Set the <agentpipe> to the number 0 or comment out / delete the line altogether. The agents won't need avs2yuv anymore :cool:

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).

I'll shut up and run a test :)

THANKS!

SpAwN_gUy

14th February 2007, 11:40

:) .. maybe i'll write a tutorial.... or maybe not ;) ...

so,.. i've come back with the tests of 1.08..

AWESOME!!!!!!!!!!!!!
i think you've modified x264... 'cause it RUNS pretty well,.. :)) ..
i've encoded that problemous file... but i've messed up with bitrate, so the file came oversized.. now i'm ReEncoding it..

minuses: agents are suddenly exiting(started over network). I have DUMP...
controller suddenly hanged yesterday.. on exiting first pass.. i dunno why. it was running smoothly, and when i left - something happened(i don't have dump..)

and i've made a some kind of contest.. calculated sum of fpses of agents.. (2x and 3x).. 2x =( 1, pad1) 3x = (2, pad1)
yeppie... now i have one CoreDuo2 !!!!!.... it's freaking awesome.. (and two or three are coming)
Agent | FPS
p4 Core Duo2 E6400 2.13GHz/2Gb/W32bit 3x | 7.05
p4 3000/1Gb/W32bit 2x | 3.05
AMD 3000+/1Gb/W64bit 2x | 2.86
AMD 3500+/1Gb/W32bit 2x | 2.83
AMD 3700?/1Gb/W32bit 2x | 2.67
p4 3000/1Gb/W32bit 2x | 2.23
p4 3200/1Gb/W32bit 2x | 1.48
p4 1800/512/W32bit 2x | 0.89
13.09 FPS total
now it' running ~16fps

now.. adout gui..
i'm a bit confused and lost in translation... what ARE the Assorted params??? I should never use them in passParams, or controller DELETES them from pass params?? .. and do i need "--threads 2" and "--thread-input" in passParams?

SpAwN_gUy

14th February 2007, 12:11

Thanks for the tips.

It's so obvious now, damn I've been blindsighted! My d2v wasn't relative (the d2v file if viewed).
the simpliest way to test will agent-based encoding run or not ("is everything OK? and will it run??") is to Open desired AVS from the Agent Computer(at least one computer) though the network.. if it runs - then OK.. it will..(well, about 30% for sure).

on each agent PC you will need AviSynth Installed.

ok, ok,... the Tutorial ;) :

dir structure:
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\file.d2v
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs
.vob are in:
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\iN\VIDEO_TS\
AviSynth plugz are in
D:\DVD-RiP\!!!x264farm-LOCAL-DiR\filters

in the network paths it looks like:
\\controllerpc\x264farm\file.d2v
\\controllerpc\x264farm\fileAVS.avs
.vob are in:
\\controllerpc\x264farm\iN\VIDEO_TS\
AviSynth plugz are in
\\controllerpc\x264farm\filters\

.d2v internals (the head of..) should look like this
UPD..(2007-10-02): Don't forget to download DGIndex 1.5.0 beta 3 .. it has "Relative paths saving"
here (http://forum.doom9.org/showthread.php?p=1050990#post1050990)
DGIndexProjectFile13
5
iN\VIDEO_TS\VTS_01_1.VOB
iN\VIDEO_TS\VTS_01_2.VOB
iN\VIDEO_TS\VTS_01_3.VOB
iN\VIDEO_TS\VTS_01_4.VOB
iN\VIDEO_TS\VTS_01_5.VOB

Stream_Type=1
......

.avs is
SetMemoryMax(32)
LoadPlugin(".\filters\DGDecode.dll")
# SOURCE
mpeg2source("file.d2v", info=3)
...... etc

agent Config bases part lokks like
<bases>
<base>\\controllerpc\x264farm\</base>
<base></base>
</bases>

and i start the controller with
controller.exe --first "--bframes 3 --b-pyramid --direct auto --filter 1,1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-psnr --no-ssim" --second "--ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-psnr --no-ssim" --avs "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileAVS.avs" --output "D:\DVD-RiP\!!!x264farm-LOCAL-DiR\fileOutput.mkv" --bitrate 1375kbps

all agents are encoding agent-based :) ...

BTW... just now 2nd pass finished..(first pass was about ~20fps)
START: 1000AM
END: 1307AM
this encode took about 20 hours to encode on my PC..

now i'm encoding one more time.. and i think most of agents - exited :( ... first pass is only using 3 of manyPC's.. :(

whatever-ever

14th February 2007, 15:01

Thanks for the tutotiral Spawn!

I'm not sure what did the trick, but LOCALLY it's going great now. Encoding with 20.7 + 13.16fps on the two local agents, VERY fast (same as when encoding without farm).

And the output is "better" on the local agents.
Reporting "encoded frames: 103/187" and updating continuously.

It might be the controller cmd line that's doing the improvements, but I'm not sure.

However I can't get the external agent to follow.
Getting a lot of Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255\")

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Now it's encoding at 0.44 and 0.57 locally and 3-4fps externally, okay... This is getting wierder and wierder!

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1 :)

SpAwN_gUy

14th February 2007, 16:08

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

EDIT: probably NICE ***. I shut down one of the local agents and removed the vdvmax from the controller cmd line, and now local agent is at 24.77 fps and external agent at 24.1 :)
it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw... What is "vdvmax"??

and... temp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

whatever-ever

14th February 2007, 16:23

it seems like this here too... few pages before i had same troubles.. and as far as nice and avs2yuv are not updating so much, i've decided to pack them back so it would run good..

i can't reject nice.exe, 'cause my people are working during the enCode.. so agentpipe=1, avs2yuv and nice are in the folder(on each agentPC) of .bat file, that stars agent..

btw. twmp workaround about those exits of agents: multiple starting lines in the .bat file,.. so after one crash it would start again..

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

PS: exiting doesn't seem to be a problem - anymore. Don't know why, but all jobs are running smoothly, just not very fast.

whatever-ever

14th February 2007, 18:05

btw... What is "vdvmax"??

vbv ;)

omion

14th February 2007, 20:31

When you guys say it's exiting, does it put up any warnings or just die?

Clearly somethings up with the second pass. First pass is encoded at a nice speed of ~50fps, second pass around 3-5fps!!

The optimal solution would possibly be to just encode first pass in farm mode, then finish 2nd pass on controller w/o farm. Sounds stupid? It is!

Is it possible to encode only first pass in x264farm? Setting the --second parametres to nothing doesn't help, it just results in warnings and no encode.

It's VERY odd to me that the first pass goes so smoothly and fully uses the cpu's, then up comes second pass where cpu load still is 100pct on all machines, but between 0-1 fps on local agents and 1-3 on external agent. I know second pass takes longer, but... not more than 10 times as much! i'd say double the time. When encoding without farm I get 25fps first pass (as the agents each get) and 11fps second pass (ten times as much as agents gets).

Something MUST be wrong

There is no way to tell x264farm to not do the second pass, but you can stop it manually and use the stats file it creates to do a separate second pass.

The created stats file is here:
<controller_temp_dir>\<lots_of_numbers> <name_of_avs>\1-<lots_of_numbers>.txt

For example,
E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs\1-4E2586CB8D8815243D6B8F08CE7FEA77.txt

Exactly which string of numbers goes with which encoding is displayed at the beginning of x264farm:
AVS temp dir is E:\x264farm\temp\controller\2E4CD50BE0ACD87071BE02B4F72B3C54 file.avs
First string is "...", with MD5 "4E2586CB8D8815243D6B8F08CE7FEA77"

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

But that seems to be because I deleted the avs2yuv.exe from the EXTERNAL AGENT's directory (I put it back and changed piped to 1 and now it's encoding fine...). This doesn't make sense though... It WAS set to 0, therefore it should check for avs2yuv. It might have been the nice.exe which exited with that message, but... I'm not sure.

Uhh... I think it's not encoding fine. Without avs2yuv, there is no chance that agentpipe=1 is going to encode at all. With agentpipe=0, the existence of avs2yuv won't matter.

I'll have to look at the dump file to say more, though.

whatever-ever

14th February 2007, 21:41

Setting up a mac, where do I get the x264 encoder for that?

whatever-ever

14th February 2007, 21:45

It is a bit odd that your numbers are so low. Is the controller taking up all the CPU time?

Oh, and it would be the most helpful if you sent me the dumps from when the second pass was encoding. I'd be able to see if x264farm decided to do something stupid (it's happened before!)

Yes all machines are 100pct. avs2yuv is taking a lot of cpu usage sometimes though (up to 50 pct.), however agents (not running avs2yuv) are slow too (not as slow, but still.. not so impressive)

I've saved the out-dump, but only from the controller, I'll check if it's usable and mail it.

omion

15th February 2007, 02:33

Setting up a mac, where do I get the x264 encoder for that?

To build it yourself, you need x-code (on the DVD that came with the mac) and SVN (over here (http://metissian.com/projects/macosx/subversion/))

Check out x264 from the repository:
svn co svn://svn.videolan.org/x264/trunk x264
Go to the x264 dir and type:
./configure
make
sudo make install
Assuming everything is set up right, that will get you the x264 binary.

You'll also need the agent for the mac. You can get 1.08 here (http://omion.dyndns.org/x264farm/x264farm-1.08-161_agent_intelmac.zip). I didn't put a config file in the archive, but it's set up the same way as in Windows.

SpAwN_gUy

15th February 2007, 09:31

When you guys say it's exiting, does it put up any warnings or just die?
well.. few people said that it shows something, but they didn't remembered that.. :(

and.. x264 takes about 30% of CPUtime..
i'll do some testings about agentpipe=0.. and will send logs to you..

...here we go
firstpass
pipe=1 nice=20 - avs=80% x264=27% 15.5fps
pipe=1 nice=0 - avs=50% x264=50% 16fps
pipe=0 nice=20 - error
pipe=0 nice=0 - x264=98% 17fps

secondpass
pipe=1 nice=20 - avs=10% x264=84% 4.3fps
pipe=1 nice=0 - avs=10% x264=85% 4.11fps
pipe=0 nice=20 - error
pipe=0 nice=0 - x264=98% 4.05fps

fistpass-local priority: normal
x264=72% 22fps

secondpass-local priority: normal
x264=80% 4.5fps
agent is p4 1800/512/W32bit
local(controller) is p4 2400/512/W32bit

... he-he... dump does not provide anything :) ... here are the screen dumps(theese were not provided in eMail):
Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt" --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

omion

16th February 2007, 04:00

Testing file "\\controllerpc\x264farm\TEST\X\test.avs": FOUND!
6 agent-based encoding
6 doing the following:
nice -n 20 \\controllerpc\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "temp\0 4196 582a6d.txt" --seek 0 --frames 4197 -o NUL "\\controllerpc\x264farm\TEST\X\test.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\controllerpc\x264farm\TEST\X\test.avs'
6 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

Oh! It looks like nice.exe does not appreciate running things from network paths. I'll see what needs to be done.

@whatever-ever:
I'll take a look at those logs you sent me. There should be something I can do about the speed.

SpAwN_gUy

16th February 2007, 09:41

yeah,.. and..
it seems, that x264farm has something wrong with Oversize :( .. i had 2 encodes of the same source(with different x264profiles), with same bitrate, and the files were
a) different in size (~30Mb)
b) they were oversized.. about ~60Mb to ~93Mb...
why?

omion

18th February 2007, 03:51

@SpAwN_gUy:

Well, the files are going to be different, since the scenes never finish encoding in the same order. They should not be that different, though.

It also looks like my bitrate-multiplier calculation is not quite right... There is not enough feedback to correct for any bitrate discrepancy, so x264farm is always trying to pull the bitrate up. I'll try to fix it in the next release (or at least make it better)

To fix it in the short-term, you can increase the number of 3rd-pass scenes.
The settings that control the third pass:
--3thresh How far off the bitrate must be in order to consider for 3rd pass. 0 = completely wrong, 1 = exactly right. Default = 0.8
--3gops The maximum number of scenes to consider for the 3rd pass. Default = a lot :p
--3ratio Same as --3gops, but as a percentage of the total scenes (well, not really percentage, but from 0 to 1) Default = 0.05.

The default parameters will re-encode up to 5% of the total scenes, but only if the scene's ratio expected_bitrate / actual_bitrate is less than 0.8.

To make the bitrate more exact, raise both --3thresh and --3ratio. Something like --3thresh 0.95 --3ratio 0.2

Meanwhile, I'll try to make the ratecontrol smarter.

whatever-ever

18th February 2007, 11:54

My speed was way up on a test I ran this friday. I'm trying to run another, but I'm having a little trouble with the avs somehow. Looking into it now

SpAwN_gUy

19th February 2007, 16:54

so.. how the loop goes(or a hang)
controller hangs on close_connection ... of agents at the end of second pass..
and after the restart.. it starts(or continues) to reEncode something.. it is 99% of an encode...(with modified 3tresh and.. as omion advised).. it happened 3 times for now.. (yeah i know... I have Lots of those errorous sources and everything is going crazzyy.. but i think it is good in terms of testing.. ;) )... so.. and keep going again and again... i'll modify .bat file.. and post some results tomorrow

//me sorry.. i'm a bit drunk right now.. but i'm allmost Russian(Belarussian.. telling the truth), so i can work and programm..
soory my girl Friend waits for me near work entrance.. so.... ;) ..

SpAwN_gUy

21st February 2007, 09:11

Yeah... and Is it possible, to run NOT Only the x264.exe with nice, but avs2yuv also?

'cause if agentpipe=1 and nice=20 then x264 is running in lowest priority, but avs2yuv is running Normal, and it slows down overall system performance,.. so my people can't work :( ...

and.. that option,.. that asks for "Do you really want to ReEncode the same AVS?"(or something)..
can you check for th SAME Bitrate and Same Filename? .. 'cause i had an encode with different bitrate and different output filename... it was nice, that i didn't need to make firstpass again,.. but..
/*played with nice..
well.. nice REALLy does not like NetPaths.. i've played with different variations of comLine.. and nice does not want to start avs2yuv with netPath param..
error is: "AviSynth can not find the file D:\controllerpc\x264farm\...."
and batch file was: nice -n 20 avs2yuv ... "\\controllerpc\x246farm\.avs" | nice.. x264...

theturtle

25th February 2007, 09:44

hrm, I can get this working on local worker but the 2nd machine I have doesn't appear to want to encode anything. i'm still plugging away at it. the network paths and setup was a tad difficult, but I think i've got that all worked out.

for some reason right now, the remote agent i have, receives a command to encode. yet it doesn't appear to be encoding.

I did have a question. regarding agent based encoding. If you are doing this, can the input video in the avs be on a network share for remote agents? Both my agents are using the same .avs and I just have the AVS looking for the input video on a Network drive I mapped(which is actually a local share on the controll/local agent machine). I guess thats not really a good question, I guess since they both use the same avs, they both have to have the same access to the input video and the path has to be the same, otherwise it won't work.

Anyhow, not sure whats going on with this remote agent, I can't get it to actually encode anything for nothing. The second pass will be up in an hour or 2, perhaps it will have better luck on that pass. :s

*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.

theturtle

25th February 2007, 15:14

omion

25th February 2007, 18:55

i thought i'd post this separately, i just finished an encode with 2 computers local agent and remote agent. the resulting encode has flashes of grey every few seconds or so. I thought at first this was during scene changes but its not. Its like a blank spot in the video or something. Perhaps made during merging process? :s

Are there entire frames which are gray, or does it seem to start in the middle of one frame and end in the middle of another?

Also, you can find out where the gray occurs by doing the following:
Turn off all the agents except for one (in the config file, or just don't start the other agents up)
Run the controller with the --keeptemp option (and probably also --frames xyz and --restart to make it fast and repeatable)
If the resultant file does not have the gray frames, then the problem is with another agent
If the resultant file does have the gray frames, then you can check the temp directory and see if the raw pieces have gray frames. If they don't then it's probably the merger that's doing it.

Also, just to make sure, the initial AVS file doesn't have the gray frames, right?

@SpAwN_gUy:
Sorry about the delay in fixing nice. I don't really have a lot of time lately, but I should get it done by Wednesday this week

AVmaniac

25th February 2007, 20:54

theturtle

26th February 2007, 02:24

*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.

^^^^ regarding this. For some reason if I set the <base> to my network drive that I mapped, my remote agent executes x264.exe but yet cpu is at 0%. If I copy the avs to the remote agent's agent-win32 directory, it appears to encode fine. Although I've just noticed, my source is h.264 mkv. I didn't have either ffdshow or haali media splitter installed on my agent. Should I need these? I figured the input to the agent was yuv from the controller.

*update* I just installed both of these and it appears its encoding, the network utilization skyrocketed from my encode last night. it was hopping around 20-50%, now its about 99% steady. :) perhaps that was my problem. hopefully it doesn't produce the same gray frames. we'll see. mooore testing ;)

I don't think it's a bug of the merging part.
Try to play the script with the WMP.
The grey frames should also appear here!
Did you try to reinstall your avisynth or tried a different version?
That's what helped me getting rid of this bug...

greetings to all x264farmers %-)

just talked with someone about this, they seem to be pretty confident its an issue with directshowsource. my source is h.264, i'm going to try the new avc/h.264 avs source plugin. I was getting blank gray frames in the source with directshowsource btw, so its definitely not a x264farm issue.

mpioner

26th February 2007, 12:08

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.

theturtle

26th February 2007, 14:10

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.

tried this?
http://forum.doom9.org/showthread.php?t=117428&highlight=x264farm+gui

dunno how well it works, the cli version took a bit of figuring and alot of mistakes on my part, but wasn't too difficult.

SpAwN_gUy

5th March 2007, 09:54

omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.i'm playing around with Delphi... for like some months.. and it seems like a testing version will be shown in few weeks.. i guess..(well it's all unofficial)..

agentGUI has an UnKILLable option (if agent crashes it is auto-Restarted) and can minimize to tray.
controllerGUI has "Import MeGUI Jobs" button, and actually imports them,.. and generates command lide for starting controller.. :) ... that is for now,.. lots of ideas, and no time at all..

@ALL
is the Tutorial i wrote usefull? or maybe...??

omion

11th March 2007, 08:32

1.09 is out (or will be shortly, when my server reboots...)

It should be more accurate with the bitrate target, and the agents now use nice for avs2yuv (if both niceness and piped input are used)
I didn't add bitrate in the same-file detection (because I only remembered it just now)

Also, the agents will by default delete any temp file which is older than 1 week. I noticed my temp directory filling up with a bunch of junk after testing x264farm, and I decided to do something about it.
By default, a cleanup thread will wake up once a day and delete all the temp files that are over 1 week old. The amount of time is adjustable with the --stale <x> command-line option. <x> is the maximum age that a temp file can be, in seconds.

The release took so long because I've been trying to rewrite the nice.exe program. I think 1.1MB is way too big for a program like that, and the fact that it can't run network things is not so good either. However. in order to easily access process priorities I have had to use C/C++ (which I don't really know) and the whole thing ended up being extraordinarily frustrating and a huge waste of time. So now, instead of a new, slimmed-down nice.exe program, I have a reminder of why I program with OCaml in the first place :rolleyes:

SpAwN_gUy

12th March 2007, 14:36

Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't :( ... well.. it's not working now at all..

agentpipe=1
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
2 agent-based encoding
2 doing the following:
nice -n 20 avs2yuv -raw -seek 0 -frames 5223 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats 41425d.txt" -o NUL - 576x432
Avisynth error:
Import: couldn't open "D:\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"

agentpipe=0
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
7 agent-based encoding
7 doing the following:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --directauto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\PC201\x264farm-2\xFiles\S1p2\S1p2.avs'

7 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"

i've downloaded archive woth nice once again,.. and i think it have not being changed..

i guess.. i'm going back to controller-1.09 and agent-1.08... waiting for nice, only :) ...

maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?

yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...

Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising :)
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool :) ...
37fps on start of second pass.. :) and rising :) ...

i think. today is a good day,.. i'm happy :) really :) ..

omion

12th March 2007, 19:54

Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't :( ... well.. it's not working now at all..

Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. :angry: I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.

I'll see what I can do about that program.

And just in case you were wondering:

maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?

I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.

yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...
Once everything starts up there isn't a difference, but the topmost agents will get the first splits, resulting in a faster startup.

With a lot of computers (like your setup) I can believe that that makes a big difference.

Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising :)
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool :) ...
37fps on start of second pass.. :) and rising :) ...

i think. today is a good day,.. i'm happy :) really :) ..
Good to hear! Thanks!

SpAwN_gUy

13th March 2007, 12:33

YEPPIE.... i've figured all out :) ... i just remembered, when i was trying to run agent as service, there was an advice..(at the end of the post,.. add some mystery ;) )
And just in case you were wondering:
I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.
and i just played a bit with delphi and i could writa a programm, that starts something(i dunno how will it work and what do you need from that starter).. it is now 40Kb .. i'll play a bit later and will post some results..
it is ConsoleApp and uses WinAPI (CreateProcess with Priority flag set).. and i've found in old winSDK only four priorities.. IDLE, NORMAL, HIGH and REALTIME... and i dunno wich are usefull :) ..

anyway results -> later.

--and-now--------------------------

Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. :angry: I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.
testing testing,.. AND:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"WORKS!!!!
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"Does NOT

and then after:
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432WORKS!!!!
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432
Does not

maybe.. it is the best way.. to change those slashes back? ;)

omion

14th March 2007, 00:17

@SpAwN_gUy:

So it looks like everything works when using forward slashes rather than backslashes? That's good to know. It's looking like it won't matter for much longer, though.

I was just contacted by foxius (at least, that will be his name when his 5 days are up ;) ) who is making a nice:rolleyes: alternative to the nice.exe program I am currently distributing. It will be a drop-in replacement, and should weigh in at ~50KB.

I can't say for sure, but it looks like it should be done soon (depending on how many more unreasonable requests I make :p )

omion

15th March 2007, 07:29

All right! I finally updated nice.exe!
(or rather, foxius updated it and I took credit for it ;) )

Get it here (http://omion.dyndns.org/x264farm/nice-20070314_win32.rar) (source (http://omion.dyndns.org/x264farm/nice-20070314_src.rar)). I also updated the first post's link.

Awesome new features:

If the program's extension is "exe", "bat", or "com", you no longer need to specifically write the extension. (This approximates the shell's handling of extensions)
Network paths work without nasty hacks
It's MUCH smaller than the old version, which had to have all the cygwin libraries dragged along with it.

MAJOR props to foxius! Thanks a bunch! :D

SpAwN_gUy

15th March 2007, 10:35

:)) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually :) ...
winAPI:
CreateProcess(with parameters)
okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ;) ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.

btw. feature request :) ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)

and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"

okay,.. i've done "pause" in .bat file... so..
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...

omion

16th March 2007, 07:41

:)) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually :) ...

Yeah. I had the basic idea of how to do it (involving CreateProcess), but my programming skills are basically limited to Perl and OCaml, neither of which are particularly good at hooking into the OS. And C just drove me nuts.

And about me using foxius's code rather than yours, well... he contacted me first :cool: Don't worry; I'll tell you the next time I need a program that I don't want to write :p

okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ;) ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.

That looks like something's wrong in the file itself...

btw. feature request :) ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)

Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... :rolleyes: )

That comment about finding GOPs (around line 450 in pass2.ml) was actually referring to the fact that there could be many more GOPs done than are in the temp stats file, and how it is quite difficult to determine where to restart counting when the program is re-run.

However, the only thing deleting the unneeded GOPs will get you is extra disk space. Everything is calculated based on the working_stats#.txt files. The result is that, once the reference to the old GOP is overwritten by the new GOP file name, the old GOP file might as well not exist any more. Nothing in the program references old GOP files, so they don't have any effect on anything.
This means that there is no reason not to delete them, but it won't help the program if they are deleted.
In either case, the whole temp directory is cleaned up at the end of the second pass, so the final result is the same no matter what files are in there.

and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"

Hmm... I'll have to look into it a bit more. Ratecontrol is quite a subtle thing, and it's fairly easy to do wrong.

I probably have a plus when I should have a minus or something. Due to the self-correcting nature of ratecontrol, just about every bug in it will result in something that is a bit off, but not by much.

Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.

okay,.. i've done "pause" in .bat file... so..
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)

(apologies for any horrible grammar or anything in this post. My brain is not working properly right now. Too much caffeine... need more sleep...)

SpAwN_gUy

16th March 2007, 09:11

And about me using foxius's code rather than yours, well... he contacted me first :cool: Don't worry; I'll tell you the next time I need a program that I don't want to write :p don't worry, i'm beginning to be more opened to peaple,.. but not all the time :) ... i gust fing your progam very interesting, and i'm trying to help.. thats all :)

That looks like something's wrong in the file itself...
....
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)
well,.. it shows right after writind to log file line "Merging 91 / 302"
and, i've muxed with my bare hands (mkvmerged) all those .mkv-files.. some of them were "unKnown format" (i guess one of those zero-bytes file, i have in _temp), and (i guess i forgot some of files, but.. ~godd mkv.. (it became 1 second less)) the end file is good,.. (thise 91,92,93 - were good in the end).
so i'm sending you those files.. via e-mail..

Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... :rolleyes: ) waiting :) ..

Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.
GordianKnot-calculator (aac-sound goes as vorbis in "Interleaving & AVI-Overhead") or Calculator in meGUI.. (as far as meGUI has "GUI" and Profiles,.. and controllerGUI can import meGUI Jobs... so, generating a comLine is not a problem to me.. ;) ..)

btw. minasan, how would you like to start controller? invisible and all "usefull info is displayed in GUI"(as meGUI does) or just run a pregenerated comLine, while showing DOSBox(like GKnot does)?

yeah,.. and something like a BUG,.. i think.. after "resuming" on second-pass,.. the info(that periodically is shown..) does not show "Current GOP" (column "Now") just "x"..

foxius

19th March 2007, 09:22

omion

19th March 2007, 19:09

Finally, I have a voice =)

x264farm is working like a charm here, the only bug I've found so far is that sometimes I get a DELFIRST from an agent and it doesn't resolve until I kill the agent, clean his temp folder and restart. I think, it would be more convenient if the agent could clean the broken files itself.

Well, DELFIRST is not really a problem in itself. It comes about when an agent finishes a job right before another job was being worked on, and it discovered that the first frame on the second job won't be an I frame. DELFIRST really just means "delete first GOP in the job, when it gets returned" and it will be taken care of when the agent returns the job.
It turns out that dealing with this case correctly is quite difficult, so I added it to the output so I could properly debug it.

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.

It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.

@SpAwN_gUy:
I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is. Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message.

And sorry about the delay. I thought I would get all this stuff done last week, but I got sick. And I play too much Oblivion. (Maybe I got sick from playing Oblivion...?)

foxius

19th March 2007, 20:06

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.

Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.

It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.

Perhaps it would work even without removing the temp files. I'll try not to delete them next time.

I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else.