x264farm: distributed encoding [Archive] - Page 7

rhodges

4th May 2007, 04:31

I'm trying the latest, 1.11 and I'm running into a problem. During the first pass, none of the agent's actually do anything.

In the dump I see

2007-05-03~22:14:13.00 4 doing the following:
nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "c:\temp\agent\stats b29e1a.txt" --seek 0 --frames 5128 -o NUL "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e01.avs"
2007-05-03~22:14:13.52 Thread received exception "Sys_error(\"nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats \\\"c:\\\\temp\\\\agent\\\\stats b29e1a.txt\\\" --seek 0 --frames 5128 -o NUL \\\"\\\\\\\\192.168.1.5\\\\Z$\\\\NetworkEncoder\\\\Source\\\\Black Adder Series 01\\\\Black Adder S01e01.avs\\\": No error\")"
2007-05-03~22:14:13.55 closing socket
2007-05-03~22:14:13.55 socket closed

If I take the command

nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "c:\temp\agent\stats b29e1a.txt" --seek 0 --frames 5128 -o NUL "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e01.avs"

and run it, I get

x264 [error]: no ratecontrol method specified
x264 [error]: x264_encoder_open failed

I didn't change any of my config files, I just dropped the new binaries over the old.

Should it perhaps pass the bitrate? I'm using Cef's 655 build. Maybe that's the problem?

Any help will be appreciated. Thanks!

Edit: Yeah, I guess I should read the changelog, it looks like 655 was changed to require a rate control
"require a ratecontrol method to be specified, it no longer defaults to cqp=26."

rhodges

4th May 2007, 15:36

A while back I posed about getting some random "No error" errors. I think I've narrowed down the problem.

2007-05-04~07:56:40.75 23 doing the following:
nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 22 --fps 30000/1001 --pass 3 --stats "c:\temp\agent\stats 1eacca.txt" --seek 49484 --frames 33 -o "c:\temp\agent\output 85bcd3.mkv" "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e04.avs"
2007-05-04~07:56:41.58 Thread received exception "Sys_error(\"nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 22 --fps 30000/1001 --pass 3 --stats \\\"c:\\\\temp\\\\agent\\\\stats 1eacca.txt\\\" --seek 49484 --frames 33 -o \\\"c:\\\\temp\\\\agent\\\\output 85bcd3.mkv\\\" \\\"\\\\\\\\192.168.1.5\\\\Z$\\\\NetworkEncoder\\\\Source\\\\Black Adder Series 01\\\\Black Adder S01e04.avs\\\": No error\")"
2007-05-04~07:56:41.59 closing socket
2007-05-04~07:56:41.59 socket closed

When the above happens, it seems the controller/agent loses track of this and never does anything with this thread again.

I wanted to see what the error was, so I ran the command line

nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 22 --fps 30000/1001 --pass 3 --stats "c:\temp\agent\stats 1eacca.txt" --seek 49484 --frames 33 -o "c:\temp\agent\output 85bcd3.mkv" "\\192.168.1.5\Z$\NetworkEncoder\Source\Black Adder Series 01\Black Adder S01e04.avs"

and I got back

x264 [error]: requested bitrate is too low. estimated minimum is 34 kbps
x264 [error]: x264_encoder_open failed

I'm not too sure why, but again, it seems like it enters this error condition and then the agent just forgets about the thread and the controller still thinks it's being worked on.

omion

5th May 2007, 03:55

and I got back

x264 [error]: requested bitrate is too low. estimated minimum is 34 kbps
x264 [error]: x264_encoder_open failed

I'm not too sure why, but again, it seems like it enters this error condition and then the agent just forgets about the thread and the controller still thinks it's being worked on.

That's interesting! I have seen that error quite a bit, but every time it's happened to me the agent sees that the thread croaked and starts up another one at a higher bitrate (like it's supposed to). I'll have to look at it a bit more.

Does it always happen when you encode something with a very low bitrate? That is, if you send "-B 1kbps" to the controller, will it lock up the agents fairly reliably? (and what OS are you running on your computers?)

Also, I've been thinking about being able to test if the agent - controller connection is dead, but until a few days ago I haven't been able to accurately reproduce that issue. However, I now found a way to artificially create the dead connection, so I'll start finding a way around it... the fix should may some light on the agent - x264 connection.

rhodges

5th May 2007, 05:16

Does it always happen when you encode something with a very low bitrate? That is, if you send "-B 1kbps" to the controller, will it lock up the agents fairly reliably? (and what OS are you running on your computers?)

I'm running four computers:

Agent x2 - XP - AMD X2 3800
Agent x2 - XP - Core2Duo E6400
Agent x2 - 2003 Advanced Server - AMD X2 4600
Agent x2 and Controller - Vista 64bit - Core2Duo E6600

(The Vista box is running 32bit x264, Cef's r654 build)

All the source files, avs, etc is hosted on a network share (the 2k3 machine), access via UNC paths with each machine using a local temp directory. The source is on an 8 drive RAID 5 array with all machines connected via gigabit.

I've seen the issue a couple of times on the XP and 2k3 machines, but I haven't had a problem in a long time. I can't even be 100% certain that was the problem.

I get the problem on the Vista machine. I've been doing my encodes at 1500kbps (just recently went to 1000kbps with the Black Adder, the other was Season 2 of Lost and Season 1/2 of Stargate Atlantis.)

Lost and Stargate encountered an error on nearly every episode. I've done 10 episodes of Black Adder now and I've only had the problem twice.

I'll give a shot with a low bitrate tomorrow, I'm going to let it run tonight and see if I can get thru my queue.

Thanks for all your work on this. Even with it hanging it has cut down my encoding time significantly. I would not be surprised if the hanging thread is due to Vista. I've found Vista to be an overall stable operating system, but it does have a few rough edges they really should have worked out before releasing it.

[P]ako

5th May 2007, 19:21

Omion, could you compile an agent for a 10.3.9/PPC Mac?

omion

6th May 2007, 00:06

@[P]ako:
I'll see what I can do. None of my current systems are PPC, but I can get access to one.

[P]ako

6th May 2007, 01:12

@[P]ako:
I'll see what I can do. None of my current systems are PPC, but I can get access to one.

Thank you! :)

rhodges

6th May 2007, 05:15

I confirmed I'm getting the error and hang with my XP machine in addition to the Vista machine. I am giving it a shot with -B 1kbps to see if it happens reliably.

-----
Edit:

None of the agents ran the second pass, seems the controller freaked out when I attempted -B 1kbps as you suggested.

2007-05-06~01:04:17.72 YOU GOT TO THE RATECONTROL! 0.000000 bits / 0.000000 bits, or mult -1.#IND00
2007-05-06~01:04:17.72 2012110.100000 bits for usage
2007-05-06~01:04:18.22 Avail: 2012110.100000
2007-05-06~01:04:18.22 SP exp: 156431871.819221
2007-05-06~01:04:18.30 Expect 248654180.006108 bits with ratefactor 0.0129
2007-05-06~01:04:18.38 Expect 248652798.442261 bits with ratefactor 0.0000
2007-05-06~01:04:18.38 Chose 0.0000 from (0.0000,248652798) to (0.0129,248654180) (32 steps left)
2007-05-06~01:04:18.47 Expect 248652798.442261 bits with ratefactor 0.0000

The last line kept repeating until I manually killed the controller (woke up to 35MB dump file).

Xitsa

8th May 2007, 18:34

I tried this beautiful stuff —it's amazing!

While I seek what to feed to x264farm I encounter strange thing:
on a simple AVS (let it be input.avs) like AVISource("...\some.avi") it stucks.
Controller gives only
agent_list 1:
///agents info
agent_list 2:
///agents info
and its all…

inputs.avs plays normally. avs2yuv shows frame numbers when asked to.

The thing strange because when I feed x264farm that AVS: DirectShowSource("input.avs") everything was OK.

May be it help somebody :)

And omion, greate thanks!!

SpAwN_gUy

10th May 2007, 09:09

AVISource("...\some.avi")
....
DirectShowSource("input.avs") everything was OK. the difference ... wait..
at first i thought... that your agents don't have a proper video decoder(agent-based encoding)..
then.. if you DDS another AVS - this will result.. input.avs will be decoded on your local system, so there will be no problem,..

but then.. if i open DDS.avs on another computer.. it would decode input.avs on the samePC... so.. i'm a bit confused :( ...
sorry for my lame answer..

the_corona

11th May 2007, 15:00

burfadel

11th May 2007, 16:25

I'm a bit of a newbie, so sorry if this is clear to everyone.

Is there a way I can use this to do only one pass encodes? Specifically I'd just like to split this command:
x264 --crf 21 --ref 6 --mixed-refs --bframes 6 --direct auto --b-pyramid --subme 6 --keyint 600 --min-keyint 25 --b-rdo --bime --weightb --filter -1,-1 --trellis 0 --analyse all --8x8dct --me umh --threads 1 --progress --output "F:\mkv\temp\output.mp4" "F:\mkv\temp\with_processing.avs"
but just spread across my 3 pc on the lan.

I noticed that when I do the following:
controller.exe --avs "F:\mkv\temp\with_processing.avs" -1 "--bitrate 800 --ref 6 --mixed-refs --bframes 6 --direct auto --b-pyramid --subme 6 --keyint 600 --min-keyint 25 --b-rdo --bime --weightb --filter -1,-1 --trellis 0 --analyse all --8x8dct --me umh --threads 1 --progress" --output "F:\mkv\temp\output_farm.mp4"
on the agents it turns out as:
nice -n 10 x264 --bitrate 800 --ref 6 --mixed-refs --bframes 6 --direct auto --b-pyramid --subme 6 --keyint 600 --min-keyint 25 --b-rdo --bime --weightb --filter -1,-1 --trellis 0 --analyse all --8x8dct --me umh --threads 1 --progress --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass 1 --stats "F:\x264enc\agent-vista\tmp\stats ca2af6.txt" --seek 8183 --frames 3343 -o NUL "F:\mkv\temp\with_processing.avs"
notice the -o NUL which means no output file.

Is there a way then if only first pass is specified it will just output the result to an mkv file, return that and the controller joins it (which is the way I think 2nd pass works)?

Thanks for your help.
Corona

Nice settings, although I'd omit the threads options. Normally setting it to auto seems to be best... or at least set it to more than 1 for more efficient encoding speed. This is particularly true if you are using a dual core system (or more) threads set to 1 would be a speed killer.

Secondly, trellis should ultimately be set to 2 (the settings 'always'). This is from akupenguin, the author of x264 so he knows what he's talking about. There is a lot of misinformation out there about trellis and h264/x264 because it works different to Xvid. In xvid, setting trellis when on crf mode apparently leads to unpredictable results, but with x264 it improves the final result and leads to a slightly smaller file size (generally) :)

the_corona

11th May 2007, 16:39

Thanks burfadel, glad the settings are reasonable, I'm just beginning and am just experimenting.

I set threads to 1 because I have a single core System (A64 3200+@2200 - 2 GB - Vista64) so I figured I'd avoid the overhead of thread management and joining/syncinc threads and what not. I'll guess I'll return it to auto and I'll also try trellis, thanks for the tipp!

But what I really posted this for is to see if there is a way to split up one pass encodes with x264Farm :-)

Bit more info perhaps, I'm encoding dvb captures in 4:3 PAL (non-anime/cartoons) and am getting about 4,5 fps with the system above (avscript is simple deinterlace and some removegrain, no resize so 720x576). The source is relatively clean. Does that speed sound reasonable?

I have two more machines I'd like to ultimatly use to encode all those recordings I have to x264. The other machines are 1 Sempron64 3000+ - 1 GB XP SP2 and 1 P4 2,2 - 512 mb XP SP2).

The sempron machines gives me about 3,5 or so fps with x264 and the P4 seems really slow, don't really know whats going on, but I get <2 most of the time, but that would add up to (optimally, I know thats not what I would get really) about 9 fps, or at least twice as fast :-)

Thnx guys

0gg

11th May 2007, 20:38

is it possible to use MT.dll for multicore with the renderfarm ?

http://forum.doom9.org/showthread.php?p=975017#post975017

foxyshadis

13th May 2007, 09:05

MT/SetMTMode can be used in the avs with no detrimental effects to x264farm, as long as it works with the avisynth filters you're using.

You can probably fake out x264farm by using a --crf xx at the end of the second pass command; I don't know if that would totally freak out its ratecontrol though. (If it's the first pass avs speed you're worried about, you should specify an alternate, much faster one, with firstavs and fastavs.) I could see the usefulness of a one-pass option.

SpAwN_gUy

14th May 2007, 08:09

You can probably fake out x264farm by using a --crf xx at the end of the second pass command; I don't know if that would totally freak out its ratecontrol though. (If it's the first pass avs speed you're worried about, you should specify an alternate, much faster one, with firstavs and fastavs.) I could see the usefulness of a one-pass option.Yes, indeed.. but for now.. farm is only usefull for two-pass encodings...

and there are bitrate settings for "100%" and "1000kbps" ... well,.. for my uses.. 80% of quality ... is like... wa-a-ay too low bitrates.. i think i need to try setting it to something near 100% ....

and trick with output and -crf will not work.. i think,.. 'cause during first pass only .stats file is transferred to controller..
my thoughts lead me to ... some weird easy settings for the first pass.. and then "-crf" for the secondpass settings..
but.. ratecontroll will die ;) ... i think :) ...

the_corona

14th May 2007, 10:48

So I can just fake a first pass with very fast settings, maybe even use --seek and --noframes (is that what its called) to create an empty stats file, and then proceed to pass 2 with crf settings?

Or do I have to run the first past over all frames? If so, can you suggest a very fast setting that has the minimum to allow for correct stats?

I kinda want to use CRF as I'm not really concerned for a fixed file size. I just want "near" transparent quality at a reasonably low filesize. Plus a two pass just seems extremely slow, at 4fps avg each 20 min show that I want to encoded will take 5/6 hours or so.

Thnx alot!

Wishbringer

14th May 2007, 11:24

and there are bitrate settings for "100%" and "1000kbps" ... well,.. for my uses.. 80% of quality ... is like... wa-a-ay too low bitrates.. i think i need to try setting it to something near 100% ....

80% bitrate from 1st pass depends on settings of 1st pass:
Till rev.655 x264 defaulted to "-crf 26" and that created quite low bitrates.
So if you set for example "-crf 15" or "-crf 18" in 1st pass settings you get a higher bitrate in 2nd pass with "80%".

Wishbringer

14th May 2007, 11:30

So I can just fake a first pass with very fast settings, maybe even use --seek and --noframes (is that what its called) to create an empty stats file, and then proceed to pass 2 with crf settings?

Or do I have to run the first past over all frames? If so, can you suggest a very fast setting that has the minimum to allow for correct stats?

You must use 1st pass, because this pass is needed to find gops...
And in 2nd pass jobs are send to clients in gops.

But for that you can set in 1st pass quite low quality settings (diamond, no trellis, no rdo etc.)
But same refs and bFrames i think, to have same iFrames.

the_corona

16th May 2007, 10:07

Thnx for the help guys.

Can I ask just one more question. Would it be possible to enable one pass encodes in the future? And why in two pass modes does the one pass mode keep reencoding the same frames over and over (I think this has to do with keyframe intervals not being hit or something)?

But why go through all that trouble, the way I see it if a file is split into n parts where n = number of clients (or agent threads I guess) as long as that number is relatively low compared to the no. of frames of the encode, you have a couple of suboptimal I-frames. Is it really worth it reencoding again and again till you make them optimal?

I'd rather it just split the file into n-parts (in my case 3) and let each agent process their part and then just join the result. In CRF mode it shouldn't really matter, should it?

Thnx again

Wishbringer

16th May 2007, 12:32

But why go through all that trouble, the way I see it if a file is split into n parts where n = number of clients (or agent threads I guess) as long as that number is relatively low compared to the no. of frames of the encode, you have a couple of suboptimal I-frames. Is it really worth it reencoding again and again till you make them optimal?

It's suboptimal in time, except you have 3 identical clients.
Example: client1: P3 600MHz
client2: A64 3000+
client3: C2D QX6700
Result: client3 is finished after one hour, waiting for client2 four further hours, waiting for client1 the next week.

Suboptimal i-Frames: lets guess, you take a source b frame and encode it as an iFrame in your file. bframe have lesser quality than pframes and even lesser quality than iframes, so new resulting iframe has that lower quality too. All appending p and b frames would have lower quality too!!, because they depends on that bad iframe. So it's better to find scene changes and good gops and encode iframes from source iframes.
A gop could be of 250 frames, around 5 seconds, so you would have that 5 seconds of bad quality per wrong decission.

SpAwN_gUy

16th May 2007, 13:47

i can see onepass enodes in the future... $)
this should look like "Compressibility check" in GKnot..
it's like make a simple
..
# COMPRESSIBILITY CHECK
SelectRangeEvery(400,20)
...
it should depend on some controller params .. on exactly how to split.. and those scene scene changes in the middle of those ranges could be the split points...
this would happen in controller-first pass.. and in the controller-second pass we could split on found keyFrames..

/me .. dropped my Delphi GUI and downloaded MeGUI sources.. it seems like i'm a very poor C# programmer ;) .. i guess i need to find someone who actually participated in MeGUI_dev...

SpAwN_gUy

16th May 2007, 13:49

80% bitrate from 1st pass depends on settings of 1st pass:
Till rev.655 x264 defaulted to "-crf 26" and that created quite low bitrates.
So if you set for example "-crf 15" or "-crf 18" in 1st pass settings you get a higher bitrate in 2nd pass with "80%".thanks for the tip :) ... will do.. i think..

EDIT: BWAHAHAHAHA.... this one is my 100th post here ... for like..4 years.. damn.. exactly 4 years :) ... happy anniversary me ... :)

AVmaniac

19th May 2007, 11:24

.... over and over and over again my x264farm displays this message

x264 [warning]: Error: 2pass curve failed to converge
x264 [warning]: target: 39004.00 kbit/s, expected: 3781.41 kbit/s, avg QP: 10.00
11
x264 [warning]: try reducing target bitrate or reducing qp_min (currently 10)

and repeats the same ranges for more than a day now.
my target bitrate for the encode is 4Mbit for a 1920x832 DVB-S2
capture.

Disabled

19th May 2007, 11:38

my target bitrate for the encode is 4Mbit for a 1920x832 DVB-S2
capture.
Looks more like 40Mbit to me:

target: 39004.00 kbit/s

AVmaniac

19th May 2007, 11:55

Looks more like 40Mbit to me:

yep ... but only looks like

-B 4000kbps

also line2

x264 [warning]: target: xxxx kbit/s ....

differs from my post for the single involved ranges.

omion

25th May 2007, 07:14

All right. I'm back from the dead. :p I spent all my time working on my other program, but I'm out of ideas for it, so I'm back here!

@AVmaniac:
Check your PM

@the_corona:
I definitely err on the side of optimality, but the only time it should be chasing around I-frames is with very low-motion scenes like ending credits. Increasing the --batch length will help this problem a bit, but I'm working on a way to try to serialize those sequences to avoid the splitting problems. It won't be done for a long time, though, as it's not easy to do.

I think I've also figured out a way to check if the connection has died. It's annoyingly difficult to do without non-blocking IO, which I don't have access to...

All these ideas are in the pre-alpha stage, and actually most haven't left my head yet, so don't hold your breath. :rolleyes:

omion

31st May 2007, 01:28

1.12 is out! The config.xml files didn't change, nor did the switches, but the new controller won't work with 1.11 agents.

The most noticeable change is the output. It is now actually well-organized rather than spouting out a whole bunch of random garbage. (if you want the random garbage, check in out-dump.txt - it's still the same:p) Also, the line that says "THIS SHOULD NOT BE OVERWRITTEN!" will be taken out soon. It means just about nothing.

I also made a test to see if an agent-based computer has died silently. Each agent sends out a heartbeat signal every 10 seconds. If the controller does not receive a signal for 30 seconds, that agent is considered dead and its job put back. This should vastly help problems when the agents dying randomly and the controller waiting forever for the agent to finish. There is still no change to controller-based encodes, but those tend not to have the same problems.

I should go over the first-pass progress line. You'll see a line in the output that looks something like this:
##########_____###|||||||||||||....._______......####|||||||####...._____||||||||....._______.....
This is a representation of the video, with the entire video squashed into the width of the screen. Here's what the symbols mean:
. = empty
_ = queued
| = working ( ! = working with DELFIRST )
# = filled
It's quite likely that part of it will change from ".||." to "....", implying that no frames were done, but this is only due to the mapping from frames to characters. It's most likely that something was done, but filled frames are covered up by empty frames, so the filled frames just don't show.
And have fun watching it fill up!

SpAwN_gUy

31st May 2007, 08:39

SpAwN_gUy

31st May 2007, 12:59

hmm.. third encode.. and..
99% done (33812 / 34062) at 16.20 FPS
Agent | FPS | Description
pc79 1 | 3.12 | No jobs left; exited
pc79 2 | 3.04 | No jobs left; exited
pc206 1 | 2.80 | No jobs left; exited
pc206 2 | 2.75 | No jobs left; exited
pc419 1 | ? | No jobs left; exited
pc661 1 | 3.18 | No jobs left; exited
pc6 1 | ? | No jobs left; exited
pc2 1 | 2.86 | Dead?
pc418 1 | 2.05 | No jobs left; exited
pc82 1 | ? | No jobs left; exited
Recent errors:
and? .. yep, it seems to be dead.. there is no one connected to my controllerPC... and that's all.. no Jobs redoing.. just nothing :( ...

omion

1st June 2007, 05:32

Yeppie... testing testing :) ...

Okay... testencode of Credits of "Pirates of the Carrebean 2" - went nice :) ... but as far as it is 5000framse long i didnt' see any difference in the first-pass :) ...
but then my next encode.. hm.. Nice :) ... one episode of "anime".. first-pass 70fps, second-pass(currently in progress) ~ 22fps.. with 3 agents.. and "nice -20"...

//BTW,.. omion, have you tried that "meGUI x264farm Mod", i've sent you few days ago?
//what about those "zones" and "assorted params" in CommandLine generation?
//is it correct?
Nevermind. i've noticed -h param .. and now everything is clear to me :) ... (my thoughts were right..) thanks..
Ah! I completely forgot about that e-mail. Sorry. :o I'll test out the mod ASAP.

But yes, the zones are parsed the same way as in x264: zone1start,zone1end,b=1234/zone2start,zone2end,q=30/...

All the assorted params look good, too. And nicely bounded!

One minor nit-pick:
The 1073741823 in "scenes to re-encode" looks a bit odd. Even though that is the default value, it looks completely random. Maybe have a checkbox for "any" default selected, and when de-selected, have the user type in a number in the box. Also, it looks like that number is not bounded. I use 1073741823 because that is the largest integer that OCaml programs can deal with. (plus, typing in a much larger number in that box will result in a nasty message)

and what about that --logfile option to save "out-dump.txt" somewhere outside deafult location?
That will actually be a bit difficult. Right now the log file is opened way before the options are parsed. The way the controller is set up, it wouldn't be possible to delay opening the dump file at all. I may be able to do some magic to get the dump file to the right place afterwards (open dump -> parse options -> close dump -> move to correct location -> open dump again) I'll look into the best way to do this.

hmm.. third encode.. and..
99% done (33812 / 34062) at 16.20 FPS
Agent | FPS | Description
pc79 1 | 3.12 | No jobs left; exited
pc79 2 | 3.04 | No jobs left; exited
pc206 1 | 2.80 | No jobs left; exited
pc206 2 | 2.75 | No jobs left; exited
pc419 1 | ? | No jobs left; exited
pc661 1 | 3.18 | No jobs left; exited
pc6 1 | ? | No jobs left; exited
pc2 1 | 2.86 | Dead?
pc418 1 | 2.05 | No jobs left; exited
pc82 1 | ? | No jobs left; exited
Recent errors:
and? .. yep, it seems to be dead.. there is no one connected to my controllerPC... and that's all.. no Jobs redoing.. just nothing :( ...

Grr.:mad: I thought I fixed that. At least it knows it's dead now.:p I'm not quite out of ideas for this problem, so it could see a fix in the next version. Unfortunately I started on a great big overhaul of the first pass job distribution code, so half of the controller is a complete mess. It may take a bit of time.

(it's also odd that the agents have exited when it there are still some unencoded frames. The agents should only exit when all of the frames are complete. I'll look into that too)

ATT

1st June 2007, 09:02

is there an easy way to translate megui parameters for x264farm i can figure out --B and --avs but the other options i'm kinda lost

today i did my first encode using x264farm using General x264 parameters x264farm.html page i only edited the --B --avs <= (i created a script using megui script creator and then i just map it) used a 1080p 20MB mkv sample for my testing.
controller -B 2500Kbps --first "--crf 19 --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 5 --ref 8 --me umh --bframes 7 --weightb --b-pyramid --analyse all" --second "--trellis 2 --bime --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 7 --ref 12 --me umh --bframes 7 --weightb --b-pyramid --analyse all --8x8dct" --avs "X:\ xxx.avs" --output "output.mkv"

i wasn't sure if my PC #1 controller should also run a agent cause on my first test i saw controller only using 50% cpu it's a dual core machine so i made run an agent as well not sure if was ok but it run. memory jump to 2.34GB few times then it settle down to 1.2 with 75% to 90% cpu usage when encoding controller/agent did most the work PC#2 only did about 800 frames don't know about the settings in General x264 parameters page but using the same bitrate on megui with 1 PC it was definitely faster using megui using the PS3 profile.

i also tried pasting parts of this profile to x264farm controller but it keep giving me errors can't remember really well but something about it did not understand for example: --level 4.1

--pass 2 --bitrate XXX --stats ".stats" --level 4.1 --ref 16 --mixed-refs --bframes 3 --no-b-adapt --b-pyramid --bime --weightb --direct auto --trellis 1 --analyse p8x8,b8x8,i4x4 --threads auto --thread-input --progress --no-psnr --no-ssim --output "X:\XXX.mp4" "X:\XXXX.avs"

PS; does the controller need to run a agent too ?
or does the controller not help in encoding or is it secretly helping :) aside from splitting the work ?

infernix

2nd June 2007, 01:22

I've set up x264farm and gave it a shot. My setup:

- Debian Etch 64bit on a c2d 3ghz (gbit)
- On that c2d 3ghz box, winXP in qemu (realtek nic + driver with tcp offloading, max ~8000kb/sec so like 80mbit)
- Debian Etch 64bit on an A64 3000 (100mbit)
- Debian Etch 32bit on a Celeron D 3.2ghz (gbit, fileserver)

The source avs was only an avisource() for a 77gb huffyuv (1280x720, ~43mins) which I ran through avs2yuv on the winxp VM beforehand (needed directshow filters).

I've tried the controller-based method and it is way too slow. On the c2d I got only about 20% cpu usage on x264, 100% on the WinXP Qemu which was running the controller (so only 120 of 200% on the dual core box), and 30% on the other two boxes. It was a bit slower without compression. Obviously, this framesize is not suitable for controller based frameserving.

Which brings me to my next point: agent based encoding on linux. I was looking at the code and I didn't see why it wouldn't work. With wine you can set up a complete working environment with avisynth filters and VFW decoders in such a way that avs2yuv works exactly like on winwoes, with the one difference that you can't use DirectshowSource (at least I haven't had that much luck so far with wine cvs, maybe that could work too).

I've set this up on my boxes and this seems to work just fine. I had to alter agent.ml and replace 'avs2yuv' with 'wine avs2yuv' and had to set <base>./</base> because searching does not play nice; agents on linux are searching for windows paths (e.g. Z:\/some/path), but otherwise this seems fine. The fileserver reads and writes locally; the c2d reads with about 8-12mb/sec, the a64 with 3-5mb/sec, so this seems to work fine with my mixed 100/1000mbit setup.

The fundamental issues with this approach right now are:

avs2yuv eats a lot of cpu to decode to raw; this avs2yuv+x264 combo would be better suited for just mencoder, but unfortunately it doesn't support frame exact seeking (or does it? i'll look into this)
controller running on the qemu VM; this slows things a bit down as the encoded results are being sent here, so given these facts, I humbly beg you to write it so that it can be compiled on linux if at all possible (why wouldn't it, anyway?)

I am also getting an error that I can't quite comprehend:

Z:\series\BSG\tmp>x264farmer
Using config file "x:\avisynth\config.xml"
CONFIG:
temp_dir: "c:\\temp\\x264farm"
agent_list 1:
"believe 1" = 192.168.0.163 50700-50703
"whitebox 1" = 192.168.0.5 50704-50707
"spirit 1" = 192.168.0.1 50708-50711
agent_list 2:
"believe 1" = 192.168.0.163 50700-50703
"believe 2" = 192.168.0.163 50700-50703
"whitebox 1" = 192.168.0.5 50704-50707
"whitebox 2" = 192.168.0.5 50704-50707
"spirit 1" = 192.168.0.1 50708-50711
"spirit 2" = 192.168.0.1 50708-50711
Parsing bitrate 6000kbps
GOT NORMAL INFO "e05farm.avs: 1280x720, 25 fps, 62737 frames\013"
GOT FIRST INFO "e05farm.avs: 1280x720, 25 fps, 62737 frames\013"
GOT FAST INFO "e05farm.avs: 1280x720, 25 fps, 62737 frames\013"
INFO:
i_res_x = 1280
i_res_y = 720
i_fps_n = 25
i_fps_d = 1
i_fps_f = 25.000000
i_num_frames = 62737
i_bytes_y = 921600
i_bytes_uv = 230400
i_bytes_per_frame = 1382400
multiplier for kbps to fbits = 0.025000
fast_res_x = 1280
fast_res_y = 720
Oops. Writing the args failed with: Sys_error("c:\\temp\\x264farm\\option_log.tx
t.temp: No such file or directory")
FIRST PASS
THIS SHOULD NOT BE OVERWRITTEN!
83% done (52359 / 62737) at 24.28 FPS
###############################||||||||#########################|||||||#######
Agent | FPS | Description
believe 1 | 22.19 | Waiting for job
whitebox 1 | 6.87 | Doing frames 51941-57081
spirit 1 | 3.85 | Doing frames 25662-30900
Recent errors:

Notice the "Oops. Writing the args failed with: Sys_error("c:\\temp\\x264farm\\option_log.txt.temp: No such file or directory"). What exactly is the function of this, and is this a serious problem?

I hope to test this on ~20 core 2 duos (all linux) on a dual gbit lan with a 4TB iSCSI storage backend soon, so I would really appreciate any work you could do on the linux side; I'll happily test this. Replicating this test enviroinment really is quite easy as it's only a matter of installing x264, mounting ~/.wine and copying some files to ~/bin on all 20 boxes.

:thanks:

omion

2nd June 2007, 07:14

I've set up x264farm and gave it a shot. My setup:

- Debian Etch 64bit on a c2d 3ghz (gbit)
- On that c2d 3ghz box, winXP in qemu (realtek nic + driver with tcp offloading, max ~8000kb/sec so like 80mbit)
- Debian Etch 64bit on an A64 3000 (100mbit)
- Debian Etch 32bit on a Celeron D 3.2ghz (gbit, fileserver)

The source avs was only an avisource() for a 77gb huffyuv (1280x720, ~43mins) which I ran through avs2yuv on the winxp VM beforehand (needed directshow filters).

I've tried the controller-based method and it is way too slow. On the c2d I got only about 20% cpu usage on x264, 100% on the WinXP Qemu which was running the controller (so only 120 of 200% on the dual core box), and 30% on the other two boxes. It was a bit slower without compression. Obviously, this framesize is not suitable for controller based frameserving.

I'm not too framiliar with Qemu, but do you have it set up as dual-core? And were you also running an agent in the Qemu emulation, or was just the controller using 100%?

And yes, controller-based encoding can choke fairly easily, especially with that much data being passed around.

Which brings me to my next point: agent based encoding on linux. I was looking at the code and I didn't see why it wouldn't work. With wine you can set up a complete working environment with avisynth filters and VFW decoders in such a way that avs2yuv works exactly like on winwoes, with the one difference that you can't use DirectshowSource (at least I haven't had that much luck so far with wine cvs, maybe that could work too).

I've set this up on my boxes and this seems to work just fine. I had to alter agent.ml and replace 'avs2yuv' with 'wine avs2yuv' and had to set <base>./</base> because searching does not play nice; agents on linux are searching for windows paths (e.g. Z:\/some/path), but otherwise this seems fine.
What did you have <base> set to before that it would search in "Z:\/some/path"? The search function is not so smart, so it will look in Windows-style dirs first, but then it should sort itself out and look in the right place. For example, if the controller is getting the file from the Windows dir "Z:/some/path", and the agent has <base> set to "/temp/x264farm" the directories searched will be:
/temp/x264farm/Z:/some/path
/temp/x264farm/some/path
/temp/x264farm/path
/temp/x264farm
If not, then something's wrong.

The fileserver reads and writes locally; the c2d reads with about 8-12mb/sec, the a64 with 3-5mb/sec, so this seems to work fine with my mixed 100/1000mbit setup.

The fundamental issues with this approach right now are:

avs2yuv eats a lot of cpu to decode to raw; this avs2yuv+x264 combo would be better suited for just mencoder, but unfortunately it doesn't support frame exact seeking (or does it? i'll look into this)
Well, the only time avs2yuv is used for agent-based encoding is if you have the <agentpipe> setting on. Otherwise the agent simply calls x264 with the proper -seek and -frames settings. Also, I ran some tests on my Windows box and piping avs2yuv > x264 ran almost as fast as just having x264 do it. I don't remember the exact number, but it was something like 1% slower.

controller running on the qemu VM; this slows things a bit down as the encoded results are being sent here, so given these facts, I humbly beg you to write it so that it can be compiled on linux if at all possible (why wouldn't it, anyway?)

You're right that the controller should be able to be compiled on Linux. I try to make my code as portable as possible, although I haven't tried compiling the controller on anything but Windows so far.

I am also getting an error that I can't quite comprehend:

[code omitted]
Notice the "Oops. Writing the args failed with: Sys_error("c:\\temp\\x264farm\\option_log.txt.temp: No such file or directory"). What exactly is the function of this, and is this a serious problem?

That error means basically nothing. The controller will write its options to a file in the temp directory so that you (well, mainly me, since it's not really documented) can go back and see what you had last encoded. This is very helpful if the controller crashes and you want to pick up where you left off - you have to use the exact same settings or it will start from the beginning.

Actually, now that I look at the code, there's no way that file could be created on its own... hmm...

Try creating an empty file named "c:\temp\x264farm\option_log.txt" and see if that fixes the problem.

I hope to test this on ~20 core 2 duos (all linux) on a dual gbit lan with a 4TB iSCSI storage backend soon, so I would really appreciate any work you could do on the linux side; I'll happily test this. Replicating this test enviroinment really is quite easy as it's only a matter of installing x264, mounting ~/.wine and copying some files to ~/bin on all 20 boxes.

:thanks:

If you'd like, you can send all those computers to me and I'll test them for free :D

In the meantime, I'll see if I can get the controller compiled on my Linux box. I don't have Wine on it, so it's no good for encoding, but I should be able to fix something.

infernix

3rd June 2007, 17:49

I'm not too framiliar with Qemu, but do you have it set up as dual-core? And were you also running an agent in the Qemu emulation, or was just the controller using 100%?

Qemu-kvm doesn't do SMP yet, so no, it's just using one core. No agent inside the winxp VM, that would be really really slow :) So the controller, and thus the WinXP VM, is idle 90% of the time; the only time it really does anything is when it starts sectioning for the 1stpass, when it receives data, and in the very end when it merges all the mkvs.

What did you have <base> set to before that it would search in "Z:\/some/path"? The search function is not so smart, so it will look in Windows-style dirs first, but then it should sort itself out and look in the right place. For example, if the controller is getting the file from the Windows dir "Z:/some/path", and the agent has <base> set to "/temp/x264farm" the directories searched will be:
/temp/x264farm/Z:/some/path
/temp/x264farm/some/path
/temp/x264farm/path
/temp/x264farm
If not, then something's wrong.

I didn't use <base>, i used <bases> and that failed. the thing is, the agent is running on linux (so uses unix style paths), but the controller runs on windows (using windows style paths) and I think the agent validates a windows style path on linux (Z:\series\blah\tmp is Z:seriesblahtmp on linux), which fails for obvious reasons. The added twist here is that avs2yuv is in the middle; Wine will autoconvert unix style paths to windows style paths if it can find them. But it's not much of an issue, with <base>./</base> i just chdir to the shared directory with the .avi and avs on the NFS mountpoint and all is well.

Well, the only time avs2yuv is used for agent-based encoding is if you have the <agentpipe> setting on. Otherwise the agent simply calls x264 with the proper -seek and -frames settings. Also, I ran some tests on my Windows box and piping avs2yuv > x264 ran almost as fast as just having x264 do it. I don't remember the exact number, but it was something like 1% slower.

I have to use <agentpipe> because if I don't, it will try to run the linux x264 executable with an avs script. That fails horribly. Also, because Wine isn't SMP aware (http://bugs.winehq.org/show_bug.cgi?id=6383), avs2yuv and any other windows executable ran through Wine will only use 1 cpu. And yes, you're probably right that the difference between x264.exe versus avs2yuv.exe + x264 is negligible, but otoh I am quite sure that if we could only use mencoder, that'd be faster since there's no avisynth processing done in my case. But until I figure out how to do exact frame seeking with it, I'll leave it be; I'll take this up with the mplayer/ffmpeg devs.

That error means basically nothing. The controller will write its options to a file in the temp directory so that you (well, mainly me, since it's not really documented) can go back and see what you had last encoded. *snip* Try creating an empty file named "c:\temp\x264farm\option_log.txt" and see if that fixes the problem.

That does seem to resolve the issue; maybe add this to the code?

If you'd like, you can send all those computers to me and I'll test them for free :D In the meantime, I'll see if I can get the controller compiled on my Linux box. I don't have Wine on it, so it's no good for encoding, but I should be able to fix something.

I don't think my client would be very happy with that ;)

Some more questions i ran into:

Use <number pad="1"> to send one additional job to the agent during the second pass. Since the second pass jobs are so short (generally a few seconds) it is a good idea to have something else working in the background when a job gets done. This setting has no bearing on the first pass. I watched the x264 processes on the agent boxes, and generally it seems that when i set <number pad="1">1</number>, there's two processes started, but whenever one's finished, there's only one process left. It seems that only after that last one is finished, the data gets sent to the controller, which doesn't make it any faster than without pad="1". Is this a bug or am I missing the point here?
First pass statistics; the way I normally encode is to first run a 1stpass and then use the statistics from that to decide whether it needs a higher bitrate/avs tweaking. I am guessing that with this method it isn't possible to generate any sort of FRF or QP numbers? is it at all possible to combine the 1stpass data for the ranges, or to use a 1stpass stats file from a normal x264 1stpass?
It would be nice if the controller could spit out some numbers regarding performance, such as total time taken for 1stpass/2ndpass, total average fps for 1stpass and 2ndpass, and perhaps even time spent transferring data from agent <--> controller (because that time is spent doing nothing, at least the way i see it now)

I'm eagerly awaiting a linux buildable controller; no matter that it doesn't encode, right now the controller in my WinXP QEMU VM doesn't encode either.

:thanks:

driftr

6th June 2007, 16:57

Thanks for 1.12! Small request... can we get the estimated time to finish back? The new output it great and the percentage is nice but if it's not a big deal it'd be nice to see that displayed also. Small request, don't mind me :)

infernix

7th June 2007, 17:27

I'm running into a quality difference issue I can't figure out.

Source vs x264farm vs x264 standalone:
http://dx.infernix.net/images/thumbs/brb1137b-1-avssource.png (http://dx.infernix.net/images/viewer.php?id=brb1137b-1-avssource.png)http://dx.infernix.net/images/thumbs/thx0097h-1-farm.png (http://dx.infernix.net/images/viewer.php?id=thx0097h-1-farm.png)http://dx.infernix.net/images/thumbs/bbq0106y-1-nonfarm.png (http://dx.infernix.net/images/viewer.php?id=bbq0106y-1-nonfarm.png)

Things i've tried:

comparing the options inside the .264 header. they were the same except for bitrate=, which is obvious
doing the 1stpass with --batch 500000 so it runs it completely on one agent, in one go
excluding agent specific issues; I ran a 500frame encode on all agents without the others though x264farm, and the 64bit agents produced the exact same frame. the 32bit agent didn't though, but not nearly as much of a difference as in the screens.

I'm wondering what is causing this. I'll run the encode without the 32bit agent to exclude that, and will post results in a few hours when that's done.

My script:

set BITRATE=6947
set SCRIPT=e06farm.avs
set DEST=e06farmspirit.mkv
x264farmc -B 100 -1 "--bitrate %BITRATE% --threads auto --thread-input --deblock -3:-3 --bframes 3 --me umh --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --direct auto --cqmfile /etc/prestige_cqm.cfg" -2 "--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --aq-strength 0.3 --me umh --trellis 1 --subme 7 --threads auto --thread-input --progress --no-psnr --cqmfile /etc/prestige_cqm.cfg" -i %SCRIPT% -o %DEST% --config x:\avisynth\config.xml

Any ideas? :thanks:

[P]ako

8th June 2007, 06:19

Are you using the x264 that Omion provides in the first post?

infernix

8th June 2007, 09:27

ako;1011745']Are you using the x264 that Omion provides in the first post?

Can't, as I'm only running linux agents. I'd try to build it on linux if I knew what patches are used.

I did try fiddling with the threshold parameters. I tried ' --3thresh 0.95 --3ratio 0.2 --split 1500 --thresh 100' and that does look better already. I'll try some more later tonight. Oh, and the pure 64bit agent encode exhibited the same issue.

[P]ako

8th June 2007, 16:23

Omion said or implied that he built his version of x264 straight from the svn.

I asked because of this:
--aq-strength 0.3

If I am not mistaken, that's a parameter to call a deblocking patch, a patch he didn't use to build his version. Anyway, I don't think that's the problem. But, It will not hurt to try just plain x264.

SpAwN_gUy

8th June 2007, 16:37

well,... i thought about that... last week..
so, omion made a Build with aq-Patch... testings showed that it can be used :) ...
(plain SVN r656 + aq-patch .. no pthreads and no gPACK... 'cause we don't need them ;) ...)

okay... i've uploaded it here:
URL-Updated: http://rapidshare.com/files/36891146/x264-656_aq5_omion.rar

infernix

8th June 2007, 17:02

ako;1011863']If I am not mistaken, that's a parameter to call a deblocking patch, a patch he didn't use to build his version. Anyway, I don't think that's the problem. But, It will not hurt to try just plain x264.

Yeah, well I do build from SVN + apply the AQ patch, but I can't try any precompiled binaries unless they're for Debian Etch 64bit.

The resulting encode with AQ patch is what you see in the standalone screenshot. I don't have any use for the plain version as the AQ patch + prestige matrix is a clear winner for this specific encode; the grain is preserved as much as possible.

[P]ako

9th June 2007, 20:31

@Infernix

Just out of curiosity, do all of your computers' CPUs have the same instruction sets?

infernix

9th June 2007, 21:15

In these tests they didnt. One Celeron D 3.2ghz in 32bit mode, one Core 2 Duo E6600 in 64bit and one Athlon 64 3k in 64bit.

Today I've been testing on 12 Core 2 Duos. I was a bit disappointed; they all turned out to be 4300s, i though they would've been faster. I had some problems because I had to run qemu (without KVM which is hardware virtualisation) on one of them for the controller and it was really slow for 1stpass and network traffic (2mb/sec only). I just finished setting up Virtualbox which seems much faster so far.

I got about 15fps on the 2ndpass but am sure the controller was holding things back too. All these boxes are completely identical, except for memory (between 2 and 8GB).

A controller that runs on Linux would help alot here. I'll run some more tests with Virtualbox now :)

infernix

12th June 2007, 02:21

Okay, i've ran some tests on 14 E4300s now:

X:\enctest\data>x264farmer
Using config file "x:\avisynth\config.xml"
CONFIG:
temp_dir: "X:\\enctest\\cache\\controller"
agent_list 1:
"one 1" = x.x.x.18 50700-50705
"two 1" = x.x.x.9 50700-50705
"three 1" = x.x.x.12 50700-50705
"four 1" = x.x.x.10 50700-50705
"five 1" = x.x.x.8 50700-50705
"six 1" = x.x.x.7 50700-50705
"seven 1" = x.x.x.5 50700-50705
"eight 1" = x.x.x.6 50700-50705
"nine 1" = x.x.x.16 50700-50705
"ten 1" = x.x.x.15 50700-50705
"eleven 1" = x.x.x.14 50700-50705
"twelve 1" = x.x.x.13 50700-50705
"thirteen 1" = x.x.x.11 50700-50705
"fourteen 1" = x.x.x.4 50700-50705
agent_list 2:
"one 1" = x.x.x.18 50700-50705
"two 1" = x.x.x.9 50700-50705
"three 1" = x.x.x.12 50700-50705
"four 1" = x.x.x.10 50700-50705
"five 1" = x.x.x.8 50700-50705
"six 1" = x.x.x.7 50700-50705
"seven 1" = x.x.x.5 50700-50705
"eight 1" = x.x.x.6 50700-50705
"nine 1" = x.x.x.16 50700-50705
"ten 1" = x.x.x.15 50700-50705
"eleven 1" = x.x.x.14 50700-50705
"twelve 1" = x.x.x.13 50700-50705
"thirteen 1" = x.x.x.11 50700-50705
"fourteen 1" = x.x.x.4 50700-50705
Parsing bitrate 100
WARNING: No units given to bitrate "100"; assuming percent
GOT NORMAL INFO "e06farm2.avs: 1280x720, 25 fps, 62901 frames\013"
GOT FIRST INFO "e06farm2.avs: 1280x720, 25 fps, 62901 frames\013"
GOT FAST INFO "e06farm2.avs: 1280x720, 25 fps, 62901 frames\013"
INFO:
i_res_x = 1280
i_res_y = 720
i_fps_n = 25
i_fps_d = 1
i_fps_f = 25.000000
i_num_frames = 62901
i_bytes_y = 921600
i_bytes_uv = 230400
i_bytes_per_frame = 1382400
multiplier for kbps to fbits = 0.025000
fast_res_x = 1280
fast_res_y = 720
Oops. Writing the args failed with: Sys_error("X:\\enctest\\cache\\controller
ption_log.txt.temp: No such file or directory")
FIRST PASS
THIS SHOULD NOT BE OVERWRITTEN!
100% done (62901 / 62901) at 56.52 FPS
############################################################################
Agent | FPS | Description
one 1 | 12.84 | No jobs left; exited
two 1 | 10.04 | No jobs left; exited
three 1 | ? | No jobs left; exited
four 1 | ? | No jobs left; exited
five 1 | ? | No jobs left; exited
six 1 | ? | No jobs left; exited
seven 1 | 11.78 | No jobs left; exited
eight 1 | 13.25 | No jobs left; exited
nine 1 | 14.11 | No jobs left; exited
ten 1 | 13.08 | No jobs left; exited
eleven 1 | 15.10 | No jobs left; exited
twelve 1 | 12.84 | No jobs left; exited
thirteen 1 | 12.80 | No jobs left; exited
fourteen 1 | 13.53 | No jobs left; exited
Recent errors:

SECOND PASS
THIS SHOULD NOT BE OVERWRITTEN!
45% done (28410 / 62901) at 24.69 FPS
Agent | FPS | Description
one 1 | 2.44 | Doing frames 30145-30228
two 1 | 1.81 | Doing frames 29418-29565
three 1 | 2.23 | Doing frames 29072-29193
four 1 | 2.00 | Doing frames 28751-28962
five 1 | 2.11 | Doing frames 29566-29704
six 1 | 1.95 | Done with frames 29705-29804
seven 1 | 2.07 | Doing frames 28194-28415
eight 1 | 2.02 | Doing frames 30082-30144
nine 1 | 2.01 | Doing frames 30007-30081
ten 1 | 2.17 | Doing frames 29932-30006
eleven 1 | 2.10 | Doing frames 27944-28193
twelve 1 | 2.04 | Doing frames 29194-29417
thirteen 1 | 2.33 | Doing frames 28416-28620
fourteen 1 | 1.97 | Done with frames 28963-29071
Recent errors:

**** note: I marked the stats here to copy/paste and it now shows the difference in FPS that 3rdpass makes up for with the settings I've used ****

100% done (62901 / 62901) at 20.93 FPS
Agent | FPS | Description
one 1 | 2.07 | No jobs left; exited
two 1 | 1.83 | No jobs left; exited
three 1 | 2.27 | No jobs left; exited
four 1 | 2.01 | No jobs left; exited
five 1 | 2.05 | No jobs left; exited
six 1 | 1.90 | No jobs left; exited
seven 1 | 2.15 | No jobs left; exited
eight 1 | 2.01 | No jobs left; exited
nine 1 | 1.99 | No jobs left; exited
ten 1 | 2.01 | No jobs left; exited
eleven 1 | 2.02 | No jobs left; exited
twelve 1 | 2.17 | No jobs left; exited
thirteen 1 | 2.04 | No jobs left; exited
fourteen 1 | 1.94 | No jobs left; exited
Recent errors:

Merging 594 / 594
TIME 4760.817000

X:\enctest\data>

Nice and speedy :) but the firstpass is limited by the fact that it's maxed out in CPU. Virtualbox proved to be faster because the E4300 doesn't have VMX but it's still slow. However, second pass is at max speed so far. This was 1280x720 content; i'm going to try some 1920x1080 by the end of the week.

Settings used:

set BITRATE=6947
set SCRIPT=e06farm2.avs
set DEST=e06farm3tresh1-3ratio0.3-split500-thresh40.mkv
x264farmc -B 100 -1 "--bitrate %BITRATE% --threads auto --thread-input --deblock -3:-3 --bframes 3 --me umh --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --direct auto --cqmfile /etc/prestige_cqm.cfg" -2 "--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --aq-strength 0.3 --me umh --trellis 1 --subme 7 --threads auto --thread-input --progress --no-psnr --cqmfile /etc/prestige_cqm.cfg" -i %SCRIPT% -o %DEST% --config x:\avisynth\config.xml --3thresh 1 --3ratio 0.3 --split 500 --thresh 40

Will try a bunch of threshold parameters and make some visual comparisons when I'm done, so if anyone has suggestions on what to try - fire away, it's speedy enough :D

jimmydu3

12th June 2007, 16:31

well,... i thought about that... last week..
so, omion made a Build with aq-Patch... testings showed that it can be used :) ...
(plain SVN r656 + aq-patch .. no pthreads and no gPACK... 'cause we don't need them ;) ...)

okay... i've uploaded it here:
http://rapidshare.com/files/35973730/x264-656_aq5_omion.rar.html

could you repost it? it's been downloaded too many times now on rapidshare

SpAwN_gUy

13th June 2007, 08:21

could you repost it? it's been downloaded too many times now on rapidshare
try this one...
http://rapidshare.com/files/36891146/x264-656_aq5_omion.rar

SpAwN_gUy

14th June 2007, 09:57

Okay, i've ran some tests on 14 E4300s

Settings used:

set BITRATE=6947
set SCRIPT=e06farm2.avs
set DEST=e06farm3tresh1-3ratio0.3-split500-thresh40.mkv
x264farmc -B 100 -1 "--bitrate %BITRATE% --threads auto --thread-input --deblock -3:-3 --bframes 3 --me umh --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --direct auto --cqmfile /etc/prestige_cqm.cfg" -2 "--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --aq-strength 0.3 --me umh --trellis 1 --subme 7 --threads auto --thread-input --progress --no-psnr --cqmfile /etc/prestige_cqm.cfg" -i %SCRIPT% -o %DEST% --config x:\avisynth\config.xml --3thresh 1 --3ratio 0.3 --split 500 --thresh 40

Will try a bunch of threshold parameters and make some visual comparisons when I'm done, so if anyone has suggestions on what to try - fire away, it's speedy enough :DI'm sorry to ask, but what are you encoding?
and What are your testResults(with those thresholds experiments)? ('cause i've only tryed the defaul ones.. and one another "--3thresh 0.95 --3ratio 0.2" (that was for totalSize prescision))