x264farm: distributed encoding

[omion]

Well, I updated x264 to revision 651, which should be chock full of tasty speed improvements since r634. Get it here (http://omion.dyndns.org/x264/32/x264-651_omion.rar) or on the first post. I don't have any time to test it, though...

[edit: actually, that's revision 652... SVN must have been updated between the time I checked it and the time I downloaded it. Since the only change is a compiler tweak, I don't think there's any need to update the numbers...]

[rhodges]

I am having a problem where sometimes a thread in agent.exe dies and it does not notify the controller and this causes the controller to hang.

From the controller log



2007-04-23~20:50:44.00 DESKTOP 1 sent full AVS info F10F4738AA6B93F016C4E3DD695756AB = "\/192.168.1.5/z$/NetworkEncoder/Source/Lost Series 02/Lost S02e20.avs"
2007-04-23~20:50:44.00 DESKTOP 1 sent video info 720x400 @ 24000/1001
2007-04-23~20:50:44.00 DESKTOP 1 sent zone string ""
2007-04-23~20:50:44.00 DESKTOP 1 sent range (62239,62281)
2007-04-23~20:50:44.01 DESKTOP 1 no attempted bits found; using 3187909153.107259 / 3605544280.000000 (+4696)
2007-04-23~20:50:44.01 DESKTOP 1 optimal bitrate is 7; minimum is 7
2007-04-23~20:50:44.01 DESKTOP 1 sent target bitrate 7 (really want 4.470565)
2007-04-23~20:50:44.03 DESKTOP 1 agent-based encoding; waiting for agent to finish



From the agent log:



2007-04-23~20:50:43.17 3 agent-based encoding
2007-04-23~20:50:43.17 3 doing the following:
nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 7 --fps 24000/1001 --pass 3 --stats "c:\temp\agent\stats 8409bf.txt" --seek 62239 --frames 43 -o "c:\temp\agent\output 256df9.mkv" "\\192.168.1.5\Z$\NetworkEncoder\Source\Lost Series 02\Lost S02e20.avs"
2007-04-23~20:50:43.46 Thread received exception "Sys_error(\"nice -n 20 x264.exe --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 7 --fps 24000/1001 --pass 3 --stats \\\"c:\\\\temp\\\\agent\\\\stats 8409bf.txt\\\" --seek 62239 --frames 43 -o \\\"c:\\\\temp\\\\agent\\\\output 256df9.mkv\\\" \\\"\\\\\\\\192.168.1.5\\\\Z$\\\\NetworkEncoder\\\\Source\\\\Lost Series 02\\\\Lost S02e20.avs\\\": No error\")"



It's strange because the error is "No Error".

The computer is a Vista x64, E6600 with 2GB RAM, running both the controller and agent. I believe I've run into this before on a couple of my other computers but I haven't gotten it lately and never looked at the log files.

It doesn't always happen, it just happens randomly. I am going to see if I get the error if I disable "nice" to see if maybe the error is there? (I was using the latest nice that works on Vista). Again, I thought I was encountering this error on my XP and 2003 machines as well (but I haven't seen the problem lately).

Any ideas?

[omion]

@rhodges:

I'm pretty sure the "no error" thing is just because Windows doesn't have the same quality of thread / signal handling that Unix does, or maybe it's just the Windows port of OCaml. Either way, "no error" is actually an error.

I don't know exactly what's causing the problem, but since the agent can report it, then it should be able to tell the controller. I'll see if I just missed part of the exception-handling code somewhere.

Is that the end of the agent's log, or is there anything more?

[edit: dang, that's a lot of backslashes... I'll see if I can fix that too ;) ]

[rhodges]

Is that the end of the agent's log, or is there anything more?



There is more, but it's just the other the other thread doing its thing. (It's configured to run 2 jobs).

I tried it w/o nice and I still got the same error.

[kdf]

Wow, 5 days before I could post this !

First off thanks for your hard work.. it is appreciated.

You have a small error in the controller make file, something which probably wouldnt show up when compiled in windows. In the makefile you refer to "extlib.cmxa" on my system extlib was compiled as "extLib.cmxa"

Nothing serious but worth pointing out for linux newbies that might not know where to look.

Now, onto my experiences.. my setup

System 1 - WIndows 2003 Server (Controller)
System 2 - Windows XP (Agent1)
System 3 - Linux Dual Core (Agent2)
System 4 - Linux (Agent3)
System 5 - Linux (Agent4)
System 6 - Linux (Agent5)

When encoding the first pass all systems check out a job and begin working. However each agent only uses between 5-15% CPU. Network utilisation on the agents is also low (10%).

The problem seems to lie in the controller, the CPU is bouncing off 100% most of the time with network being about 70% utilisation. I figured that this was due to the non agent based encoding. So going back to basics I set it up for one controller, one agent with agent based encoding.

No matter what I do all I get is an error thrown back by x264 stating "no valid output file" the command being sent to the agent indicates an output of

"-o /dev/null"

Which would be correct for the first pass ! the agent has network access to the .avs, .d2v and its own local copy of avs2yuv and x264. When running the agent finds the remote .avs file correctly so the <base> must be correclty set. All the temp storage areas are also correctly set.

any ideas what could be causing this ?

[foxius]

Well, I updated x264 to revision 651, which should be chock full of tasty speed improvements since r634. Get it here (http://omion.dyndns.org/x264/32/x264-651_omion.rar) or on the first post. I don't have any time to test it, though...

[edit: actually, that's revision 652... SVN must have been updated between the time I checked it and the time I downloaded it. Since the only change is a compiler tweak, I don't think there's any need to update the numbers...]

Wow. I'm getting 12fps per each core on Pentium D 820 2.8Ghz. Before it was 7fps per core. Other agents also show 1.5-1.7x speed up with the new build.

[foxius]

Which would be correct for the first pass ! the agent has network access to the .avs, .d2v and its own local copy of avs2yuv and x264. When running the agent finds the remote .avs file correctly so the <base> must be correclty set. All the temp storage areas are also correctly set.

any ideas what could be causing this ?

Make sure that:

1. .avs, .d2v and source files (e.g. .vob files) are accessible by agent PCs, which is true in your case. They should be accessible either at network share (mounted at base path) or locally (located at base path or via absolute path in .avs/.d2v files)

2. all avisynth plugins (.dll) listed in your .avs script are accessible either locally or at network share.

[legoman666]

I'm converting my Ghost in the Shell 2: Innocence blu ray to h264 and for some reason, x264farm isn't encoding at the specified bitrate. Here's the command line options I used:
controller.exe -B 10654kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --ratetol 2.0 --me dia --progress --no-dct-decimate --no-psnr --no-ssim" --second "--ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --ratetol 2.0 --me umh --progress --no-dct-decimate --no-psnr --no-ssim" --avs "C:\innocence.avs" -o "C:\innocence.mkv"

The movie is ~1.5 hours long and as you can see, I told it 10654kbps. The resulting .mkv file x264 outputs is about 100mb and looks like 100kbps instead of 10000. The source is just fine. Don't know what the porblem here could be.
http://img129.imageshack.us/img129/5359/vlcsnap5516607sp5.jpg

[708145]

I'm converting my Ghost in the Shell 2: Innocence blu ray to h264 and for some reason, x264farm isn't encoding at the specified bitrate. Here's the command line options I used:
controller.exe -B 10654kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --ratetol 2.0 --me dia --progress --no-dct-decimate --no-psnr --no-ssim" --second "--ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --ratetol 2.0 --me umh --progress --no-dct-decimate --no-psnr --no-ssim" --avs "C:\innocence.avs" -o "C:\innocence.mkv"


I think you should omit the bitrate in the first pass.

[legoman666]

I think you should omit the bitrate in the first pass.

the bitrate is not in the first pass, it is outside of it, as you can see. My command for this encode is no different than my other encodes with x264farm, which have been successful in this format.

[omion]

The movie is ~1.5 hours long and as you can see, I told it 10654kbps. The resulting .mkv file x264 outputs is about 100mb and looks like 100kbps instead of 10000. The source is just fine. Don't know what the porblem here could be.


Yuck. It looks like my ratecontrol equation exploded. That's just too high a bitrate for it to handle!

If you look in the log, you'll probably see something like:
2007-04-24~11:00:11.07 Expect XYZ bits with ratefactor 0.0000
2007-04-24~11:00:11.09 WARNING: 2pass curve failed to converge

Which means it's trying to encode to as low a bitrate as possible. I'll see if I can tweak it to handle that correctly...

[rhodges]

I've seen the warning:

WARNING: 2pass curve failed to converge

with 1500kbps encodes. Is it only bad when seen with:

Expect XYZ bits with ratefactor 0.0000

?

[omion]

@rhodges:

Well... it's always "bad" in the sense that you're not going to get the bitrate you specified. However, it will only look bad if the final ratefactor is 0.

Generally, this means that you are trying to encode to a bitrate that is very far off from the first pass bitrate, and the predictions say that it can't be done (either too big or too small)


It looks like the problem with legoman666's encode is that two ratefactors have the same bitrate, so the algorithm that changes the ratefactor can't fit to a certain bitrate without using infinity. OCaml seems to actually use negative infinity, which then gets turned into 0.

It should be an easy fix, but I'll have to do a bit of regression testing too.

[legoman666]

ah, thanks for the clarification, I eagerly await the update :D

What's the max bitrate x264farm can handle? 10654 is kind of overkill, its just the max bitrate that would make the final file with the audio fit onto a dual layer DVD.

[omion]

What's the max bitrate x264farm can handle? 10654 is kind of overkill, its just the max bitrate that would make the final file with the audio fit onto a dual layer DVD.

Depends on the source. The max bitrate is however many bits you get out of an encode where all the frames are at --qpmin.
It's more likely that the bitrate will be hit if it is near the first pass bitrate. So doing a first pass with "--qp 10" will be more likely to actually encode to an insane bitrate like 10654 :p

OK. Try out this controller (http://omion.dyndns.org:8080/controller-1.09+2_ratecontrol_tweak.rar) and post the part of the out-dump.txt file where it says something like:
2007-04-24~15:35:09.60 YOU GOT TO THE RATECONTROL! 0.000000 bits / 0.000000 bits, or mult -1.#IND00
2007-04-24~15:35:09.60 4443605833.333333 bits for usage
2007-04-24~15:35:09.71 Avail: 4443605833.333333
2007-04-24~15:35:09.71 SP exp: 19324850.135399
2007-04-24~15:35:09.73 Expect 1919500010.512299 bits with ratefactor 229.9426
2007-04-24~15:35:09.75 Expect 37065704.310011 bits with ratefactor 0.0000
2007-04-24~15:35:09.75 Chose 538.2664 from (0.0000,37065704) to (229.9426,1919500011) (32 steps left)
2007-04-24~15:35:09.78 Expect 2051434512.139841 bits with ratefactor 538.2664
2007-04-24~15:35:09.78 Chose 6128.6425 from (229.9426,1919500011) to (538.2664,2051434512) (31 steps left)
2007-04-24~15:35:09.79 Expect 2051434512.139841 bits with ratefactor 6128.6425
2007-04-24~15:35:09.79 BITRATE FLATLINE AT 2051434512. Decrease --qpmin or the target bitrate to get an accurate result


Note that you only need to start encoding the second pass to see this block of text. As long as your first pass settings are the same, you should see it right away, then you can cancel the encode.

[legoman666]

I'm not sure if it matters, but since I already deleted all the temp files from the last attempt at the encode, I had to redo the first pass. To save time, i only had it do 1000 frames. Anyway, here's the text from the log:

2007-04-24~20:04:13.28 Number of GOPs done: 0 / 7
2007-04-24~20:04:13.28 Looks like the second pass has not finished; using 0 for the number of GOPs computed
2007-04-24~20:04:13.28 Max GOP threshold is 7 GOPs (out of 7) or a ratio of 0.800000
2007-04-24~20:04:13.28 YOU GOT TO THE RATECONTROL! 0.000000 bits / 0.000000 bits, or mult -1.#IND00
2007-04-24~20:04:13.29 444804943.916667 bits for usage
2007-04-24~20:04:13.29 Avail: 444804943.916667
2007-04-24~20:04:13.29 SP exp: 6127739.005974
2007-04-24~20:04:13.29 Expect 132361430.293441 bits with ratefactor 72.5888
2007-04-24~20:04:13.31 Expect 9872760.918776 bits with ratefactor 0.0000
2007-04-24~20:04:13.31 Chose 257.7478 from (0.0000,9872761) to (72.5888,132361430) (32 steps left)
2007-04-24~20:04:13.31 Expect 398101841.677236 bits with ratefactor 257.7478
2007-04-24~20:04:13.31 Chose 290.2890 from (72.5888,132361430) to (257.7478,398101842) (31 steps left)
2007-04-24~20:04:13.31 Expect 410522678.328337 bits with ratefactor 290.2890
2007-04-24~20:04:13.31 Chose 380.1046 from (257.7478,398101842) to (290.2890,410522678) (30 steps left)
2007-04-24~20:04:13.31 Expect 418102594.118023 bits with ratefactor 380.1046
2007-04-24~20:04:13.32 Chose 696.5048 from (290.2890,410522678) to (380.1046,418102594) (29 steps left)
2007-04-24~20:04:13.32 Expect 418102594.118023 bits with ratefactor 696.5048
2007-04-24~20:04:13.32 BITRATE FLATLINE AT 418102594. Decrease --qpmin or the target bitrate to get an accurate result
2007-04-24~20:04:13.32 GOP 0 (109 frames) @ 0.000000 = 18316332.59 bits, or 4028.925040 kbps, min 189827 (41.755016 kbps)
2007-04-24~20:04:13.32 GOP 1 (117 frames) @ 0.000000 = 93085444.08 bits, or 19075.374693 kbps, min 222308 (45.556085 kbps)
2007-04-24~20:04:13.32 GOP 2 (212 frames) @ 0.000000 = 103591369.69 bits, or 11715.609261 kbps, min 423690 (47.916989 kbps)
2007-04-24~20:04:13.32 GOP 3 (133 frames) @ 0.000000 = 60512358.90 bits, or 10908.614796 kbps, min 319945 (57.676759 kbps)
2007-04-24~20:04:13.32 GOP 4 (250 frames) @ 0.000000 = 69884615.42 bits, or 6702.220860 kbps, min 522869 (50.145279 kbps)
2007-04-24~20:04:13.34 GOP 5 (179 frames) @ 0.000000 = 72209766.72 bits, or 9672.084348 kbps, min 398920 (53.433047 kbps)
2007-04-24~20:04:13.34 GOP 6 ( 1 frames) @ 0.000000 = 502706.71 bits, or 12052.908102 kbps, min 420 (10.069930 kbps)

edit: my p2 agent running your 651 build of x264.exe crashed as soon as it tried to start up x264-651_omion.exe. "x264-651_omion.exe has encountered a problem and needs to close.. blah blah"

[omion]

I'm not sure if it matters, but since I already deleted all the temp files from the last attempt at the encode, I had to redo the first pass. To save time, i only had it do 1000 frames. Anyway, here's the text from the log:
[ code... ]

Great! That's exactly what it should do. I'll do a bit more testing, and I should have a new version out soon.

Also note: all agents running your 651 build of x264.exe crashed as soon as they try to start up x264-651_omion.exe. "x264-651_omion.exe has encountered a problem and needs to close.. blah blah"
Err... did 634 do that? And what OS do you have? Does it crash if you run it without x264farm?

[legoman666]

Err... did 634 do that? And what OS do you have? Does it crash if you run it without x264farm?

Looks like I was mistaken, only 1 of the computers out of 6 running the 651 build crashed. And no, it didn't crash before when it was using 634.

its a p2 350mhz, windows xp sp2, voodoo 3 2000, 320mb ram

Awesome, the encoded file is now the proper bitrate :D 55mb for the 1000frames I tested instead of 92mb for 150000 frames it was before.

Is there a way to get x264farm to use the .stats file from the first pass created by megui? Since x264farm wasn't encoding the video correctly, I had megui do it instead. problem is, its been running for 10 hours and is only 23% done with the second pass. So i want to cancel it and have x264farm start over at the second pass. Is this possible?

[[P]ako]

Awesome! Thank you for it.

I just tried it and I love it. I used only 3 computers and I encoded a file in half of the time it'd have taken me with my fastest computer.

Question, can one do only a second pass with it? If so, how do we tell the program what stat file to use?

[omion]

ako;993309']Awesome! Thank you for it.

I just tried it and I love it. I used only 3 computers and I encoded a file in half of the time it've have taken me with my fastest computer.

Question, can one do only a second pass with it? If so, how do we tell the program what stat file to use?

x264farm will do both the first pass and the second pass. In fact, that's the only thing it can do. Use the -1 switch to set the first pass settings, and the -2 switch to set the second pass. Read the HTML file for info on which x264 options to leave out of the -1 and -2 strings.


@legoman:
Something may have changed in the source that doesn't work with Pentium 2s any more... (or it's MSVS screwing things up again)

If the computer is using agent-based encoding and <agentpipe> is 0, try out Sharktooth's newest build and see if it works. If it does, then I may need to tweak some settings in the compiler.

[[P]ako]

So, if I want to encode the same video again, I have to start from the 1st pass, right?

[omion]

Oops. Just noticed your edit.

Awesome, the encoded file is now the proper bitrate :D 55mb for the 1000frames I tested instead of 92mb for 150000 frames it was before.
Hooray!

Is there a way to get x264farm to use the .stats file from the first pass created by megui? Since x264farm wasn't encoding the video correctly, I had megui do it instead. problem is, its been running for 10 hours and is only 23% done with the second pass. So i want to cancel it and have x264farm start over at the second pass. Is this possible?

Kind of. If you know the filename of the first pass temp directory, you can insert the stats file in it and make x264farm start from there.

So, if you start up x264farm and the temp dir is:
somewhere\ABC.avs\1-XYZ
then you can delete the "1-XYZ" directory and save the stats file as
somewhere\ABC.avs\1-XYZ.txt
Then start up x264farm. It should start from the second pass.

The best way to find out what the first pass directory is, is to start up x264farm and it will say:
AVS temp dir is somewhere\ABC.avs
First string is "...", with MD5 "XYZ"

then you know where it's looking.

Note that you must delete the "1-XYZ" directory in order for this to work. If not, it will probably refuse to encode.

[omion]

ako;993322']So, if I want to encode the same video again, I have to start from the 1st pass, right?

x264farm keep all the 1st pass stats files, so if it notices you use the exact same settings for the first pass, it will start up from the beginning of the second pass. This means you can do the following:


x264farm -B 1000kbps -1 "abc" -2 "pqr"
x264farm -B 300kbps -1 "abc" -2 "xyz"

The second command will see that "abc" is the same for both, and re-use the first pass to make a completely different second pass.

[[P]ako]

Cool, thank you. I guess I will have to use --force to make it encode the second pass again since I will be using the same settings as before (just changes in the filter chain or subs).

[omion]

ako;993335']Cool, thank you. I guess I will have to use --force to make it encode the second pass again since I will be using the same settings as before (just changes in the filter chain or subs).

Ehh... actually, if you change the AVS file, it is considered a new file to encode. In that case, the first pass is not reused between the passes. Unfortunately, there is no way to make it re-encode between two different AVS files without moving around the stats files like I told legoman666.

[Wishbringer]

When using x264farm controller with options like:
controller.exe -B 100% -1 "whatever" -2 "whatever"
or
controller.exe -B "100%" -1 "whatever" -2 "whatever"

it tolds me that it is assuming 100 is percent.
Seems it doesnt recognize the percent-sign.
Maybe because % is used on other places as a placeholder for parameters:
tool.exe %1 %2 %3

Why not use "p" instead:
-B 99p and -B 1000kbps

[SpAwN_gUy]

Well, I updated x264 to revision 651, which should be chock full of tasty speed improvements since r634. Get it here (http://omion.dyndns.org/x264/32/x264-651_omion.rar) or on the first post. I don't have any time to test it, though...

[edit: actually, that's revision 652... SVN must have been updated between the time I checked it and the time I downloaded it. Since the only change is a compiler tweak, I don't think there's any need to update the numbers...]any of 64bit builds for me? ;)

[omion]

Wishbringer:
It should only do that inside a batch file. In that case, use two percents (like 100%%) to make a literal percent sign.

any of 64bit builds for me? ;)
Nope :( The 64-bit builds got pretty trashed after the last update. There's not a whole lot I can do, since I don't have any clue how to edit the assembly to fit Windows' style, or how to program any C to work around the other problems with it...

[[P]ako]

Ehh... actually, if you change the AVS file, it is considered a new file to encode. In that case, the first pass is not reused between the passes. Unfortunately, there is no way to make it re-encode between two different AVS files without moving around the stats files like I told legoman666.

Correct me if I am wrong. I was planning to trick "x264farm" by using an avs like this:

directshowsource("filtered_feed.avs")

Then I thought that filtered_feed.avs can be altered all you want, except the name. The controller would see no difference and start second pass. Does my theory make sense? I haven't tested it yet since my computers are encoding in an orgy of bytes.

[omion]

ako;993658']Correct me if I am wrong. I was planning to trick "x264farm" by using an avs like this:

directshowsource("filtered_feed.avs")

Then I thought that filtered_feed.avs can be altered all you want, except the name. The controller would see no difference and start second pass. Does my theory make sense? I haven't tested it yet since my computers are encoding in an orgy of bytes.

Heh :D That'll work. x264farm computes an MD5 on the AVS you feed it, but ignores anything else.

Note, however, that this is technically breaking the system. You will have to make sure to use the "--restart" option if you don't want to re-use the first pass. Failing to do so will result in Pain And Suffering™ and probably really bad quality.

(the "--force" option you mentioned before will only force encoding to start from the beginning of the second pass)


@all:
I'm almost done with the next version of x264farm. It should have a better filesize accuracy, as well as pass-dependent compression settings on the agent side. I just need to do a few more tests to make sure it works...

[legoman666]

I finished that encode at 10654kbps with x264farm. The quality is great, however, there is no way in hell that it actually encoded my source at 10654kbps. The final file size was ~4.2gb for a 1hour 40 minute movie. At 10654kbps, the file shoule have been ~7.8gb. 4.2gb means the actual bitrate was closer to 5700kbps.

Of course, the final file's quality looked just fine, but it's still not what I told it to do :D

Heh, I thought this might be amusing. I had 7 agents running on the first pass of an encode. Only one had compression enabled. I added another with compression enabled and the extra cpu usage actually lowered the total encoding speed.
http://img47.imageshack.us/img47/1139/hahaol6.jpg
The part of the graph where it goes from ~80% to 65% is when I added the extra agent. The total usage should've gone up, not down. Hehe, compression isnt always a good thing.

[omion]

I finished that encode at 10654kbps with x264farm. The quality is great, however, there is no way in hell that it actually encoded my source at 10654kbps. The final file size was ~4.2gb for a 1hour 40 minute movie. At 10654kbps, the file shoule have been ~7.8gb. 4.2gb means the actual bitrate was closer to 5700kbps.

Of course, the final file's quality looked just fine, but it's still not what I told it to do :D

Yeah. That's because x264 couldn't figure out how to use up all those bits. That's what the original problem was with the ratecontrol screwing up. If you really want that bitrate, lower the --qpmin setting (10 by default) However, --qpmin is set to 10 because there really isn't any quality gain below that.

Hehe, compression isnt always a good thing.
Definitely. In fact, it's generally a bad thing.

[legoman666]

What does it mean when one of the ranges has DELFIRST next to it? Whenever there is a range like that, the controller and the agent encoding that range exibit weird behavior..

( 0,107511) Full
(107512,107711) Queued
(107712,112877) Working (staircase 2) DELFIRST
(112878,156845) Full

[omion]

All righty, guys. 1.10 is out.

First, you should know that the config.xml files haven't changed (unless you want to add pass-specific compression) but you must use both the 1.10 versions of the controller and agent (no mixing 1.10 controller with 1.09 agents)

One of the big changes is that it should hit the bitrate target much more accurately now. I just finished a full-length movie test, and the resultant file size was off by ~0.2%, which should be much better than before. I introduced a new switch, --sizeprec, which will determine how much this is taken into account:

--sizeprec 0.0 will scale the local bitrate in order to maintain a consistent quality throughout the movie. This is the same as 1.09.
--sizeprec 1.0 will scale the local bitrate to hit the desired filesize without regard to fluctuating bitrate
A number between 0 and 1 will fade between the two
Anything below 0 or above 1 makes no sense. (although I just realized I didn't clamp it to that range... oops) :rolleyes:

Basically, if the first scenes are too small, using 0 will correct the following scenes to be the "right size". Using 1 will overcorrect the following scenes to make up for the missing bits from the first scenes.
The new default is 1, since the quality impact is extremely minor (some scenes don't get quite enough bits, other scenes get a bit too many) and it hits the overall bitrate much more accurately.

I also added pass-specific compression options, so legoman666 should replace the <compression> block in the agent's config.xml file to be like this:<passcompression>
<first>
<type>1</type>
</first>
<second>
<type>0</type>
</second>
</passcompression>
This will enable compression type 1 on the first pass, and only use type 0 (no compression) on the second.

I also kept in the change to the ratecontrol that will prevent it from panicking when the bitrate is too high. As legoman666 noted, even this change will not result in the desired bitrate, but it will look really good ;)

[omion]

What does it mean when one of the ranges has DELFIRST next to it? Whenever there is a range like that, the controller and the agent encoding that range exibit weird behavior..

( 0,107511) Full
(107512,107711) Queued
(107712,112877) Working (staircase 2) DELFIRST
(112878,156845) Full

DELFIRST is weird. Basically, when the controller notices a condition that may cause an infinite loop, the offending range is marked DELFIRST and treated very... carefully. So really, the strange behavior of the controller is to avoid the much worse behavior of an infinite loop.

You may have noticed that, in your example, nothing will work on the queued range until the DELFIRST range has been returned by "starecase 2". In the middle of an encode this doesn't matter, because there are other ranges to work on instead. At the end, though, it can stall like this since there aren't enough frames left to avoid the DELFIRST range.

There are still some optimizations that I can do that will minimize the stalling that DELFIRSTs cause, but nothing that would have helped in the example case. The holdup is always minor, and never more than 2*keyint (in your example it's 200 frames) so there's not really a big problem.

[SpAwN_gUy]

1.10 is out.. nice.. testing testing :)
i have two new sources :) ...

[SpAwN_gUy]

hehe.... i'm back :) .... remmember me? ;) .. i had a problem with not muxed final MKV-File.. i still do ;) ... now it shows more interesting things...
Fatal error: exception Matroska.Matroska_error("A signed integer -14535679 was passed to write_uint_to_string") same place same guy :)

[foxius]

hehe.... i'm back :) .... remmember me? ;) .. i had a problem with not muxed final MKV-File.. i still do ;) ... now it shows more interesting things...
Fatal error: exception Matroska.Matroska_error("A signed integer -14535679 was passed to write_uint_to_string") same place same guy :)

Clearing the controller's temp folder helped helped me once, though it had to reencode from scratch... You may try it =)

[omion]

hehe.... i'm back :) .... remmember me? ;) .. i had a problem with not muxed final MKV-File.. i still do ;) ... now it shows more interesting things...
Fatal error: exception Matroska.Matroska_error("A signed integer -14535679 was passed to write_uint_to_string") same place same guy :)

Ah-ha! I was wondering if that would happen. After the bizarre problems you had here (http://forum.doom9.org/showthread.php?p=970730#post970730), I added a few checks to the Matroska writer to make sure that everything was valid. I guess it isn't... The next step is to find out why a negative number is being passed to a function that only takes positive numbers.

[SpAwN_gUy]

Clearing the controller's temp folder helped helped me once, though it had to reencode from scratch... You may try it =)well. i already tried to delete SOME of the temp-Files,... but ;) this is funny: theese files, that were ReNcoded(same gops) the problem was there ;) ... and might be, that it IS good to report the continous problem... to fix it for future generations :) ..

[SpAwN_gUy]

and one more thing, i've asked about cheking of "Same bitrate AND same Filename" on questioning "it seems like this encode was already made. encode again?".. cause.. i'm playing around wit encoding quality-based and bitrate-Based.. so input AVSes are the same, and outputFiles and Bitrate Are different..
controller.exe --first "--bframes 3 --b-pyramid --direct auto --filter -2,-2 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --progress --no-psnr --no-ssim" --second "--ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-2 --subme 7 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-psnr --no-ssim --deadzone-inter 6 --deadzone-intra 6" --avs "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm.avs" --output "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm_80.mkv" --bitrate 80
controller.exe --first "--bframes 3 --b-pyramid --direct auto --filter -2,-2 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --progress --no-psnr --no-ssim" --second "--ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-2 --subme 7 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads 2 --thread-input --progress --no-psnr --no-ssim --deadzone-inter 6 --deadzone-intra 6" --avs "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm.avs" --output "G:\DVDRiP\Bakumatsu\DVD1\B01_FFilm_1440.mkv" --bitrate 1440kbps

[driftr]

hehe.... i'm back :) .... remmember me? ;) .. i had a problem with not muxed final MKV-File.. i still do ;) ... now it shows more interesting things...
Fatal error: exception Matroska.Matroska_error("A signed integer -14535679 was passed to write_uint_to_string") same place same guy :)

I'm having the same problem. I have a 42 minute tv ep I purchased off Amazons unbox. It's in some weird ASF wmv format so I can't clip out an intro bit w/o converting it to another format first, so I am choosing lagarith. It creates a 14gig file. megui handles this fine but when I use xfarm I get the same problem at the end. (by the way it'll nice to use 6 computers + 1 dedicated controller to take my encode time down to about 90 minutes from 4 hours)

The out-dump.txt file doesn't seem to end properly, the file reaches just over two megs and here's the last few lines
2007-04-27~10:22:37.21 Merging 630 / 864
2007-04-27~10:22:37.32 Merging 631 / 864
2007-04-27~10:22:37.43 Merging 632 / 864
2007-04-27~10:22:37.67 Merging 633 / 864

but in the command prompt with the controller command I get this
#0862: "862 - 74979 75228 f5b794.mkv"
#0863: "863 - 75229 75286 d1f21f.mkv"
Fatal error: exception Matroska.Matroska_error("A signed integer -36848815 was p
assed to write_uint_to_string")

Think the large source file size is losing sync during joining? I'm encoding it down to 300 megs but it always dies at a 200 meg joined mkv. The other thought is the lagarith file has audio which I'm not using...maybe I should try with no audio stream...

[omion]

and one more thing, i've asked about cheking of "Same bitrate AND same Filename" on questioning "it seems like this encode was already made. encode again?".. cause.. i'm playing around wit encoding quality-based and bitrate-Based.. so input AVSes are the same, and outputFiles and Bitrate Are different..

Oh yeah. Forgot about that. :p It'll be in the next version. The second pass equality will be based off the bitrate, whereas the first pass can still be reused.

@driftr:
I don't think it's an audio problem. It may be a filesize issue, but I've encoded things much larger than 200MB... It may be an integer overflow error, but I use 64-bit integers for everything Matroska-related, so I'm not sure. Whatever it is, I'm working on it.

[omion]

I FOUND IT!

It's a stupid (64-bit!) integer overflow error. If you are having this problem, look at the top of out-dump.txt, and it will probably say something like:
2007-04-27~10:41:30.34 INFO:
2007-04-27~10:41:30.35 i_fps_n = 12570329
2007-04-27~10:41:30.35 i_fps_d = 524288
2007-04-27~10:41:30.35 i_fps_f = 23.976000

i_fps_n / i_fps_d = actual framerate. Note that both i_fps_n and i_fps_d are fairly large numbers (much larger than the "exact" 24000 / 1001)

The problem is that in the Matroska file, the numbers get too large to even represent with a 64-bit integer. Part of the calculation is:
frame_num * i_fps_d * 1000000000

Using SpAwN_gUy's values for i_fps_d, this overflows the maximum 64-bit integer value on frame 17593.


I'll see if switching to floats gives an accurate result, but in the meantime, you should use the "assumefps" AVS filter with integers for both numerator and denominator. So, for 23.976fps stuff, put this at the end of the AVS file:
assumefps(24000,1001)
That gives small enough numbers for just about any movie. For any other frame rates, see the filter's page (http://avisynth.org/mediawiki/AssumeFPS).

If you do add the assumefps filter and you don't want to lose all your previously-computed work, you may copy the contents of the old AVS temp directory to the new one (x264farm will see the AVS file as a different one since it was changed)

[harissa]

is it possible to use a hub instad of a router ,
doses the x264farm support the ip like 169.254.25.121
and without gatway

[omion]

is it possible to use a hub instad of a router ,
doses the x264farm support the ip like 169.254.25.121
and without gatway

If you mean what I think you mean, then yes. ;) Basically, if you don't have a router, then a hub won't work. However, a switch will work great. Generally, if you go to a store and ask for a hub you get a switch anyway.

As for the IP addresses: if your computer can see something at that IP address, then x264farm can encode with it. These addresses can be local (192.168.*.*) or anywhere on the internet.

It sounds like you are interested in setting up a network without an internet connection. In this case, you need to connect every computer up to a switch. You should probably give every computer a manual IP address, then just tell x264farm to use those addresses.

[driftr]

I'll see if switching to floats gives an accurate result, but in the meantime, you should use the "assumefps" AVS filter with integers for both numerator and denominator. So, for 23.976fps stuff, put this at the end of the AVS file:
assumefps(24000,1001)

Worked like a charm, well I used 30000,1001 since I had 29.976 source, thanks! BTW I'm using a dual 3.6ghz xeon as my controller with a mix of 6 agents, took 109 min to convert a 42 min 640x480 avi to x264. Locally on the controller it would have taken over 4 hours! Now adding two more agents to test out the next conversion.

[driftr]

Still testing it out, I'm loving the agent based encoding. At home with a P4 3.4GHz HT, a 800MHz athlon, a P4 2.4GHz HT, and a Dual Core 2Ghz laptop I was able to encode a 43 minute TV rip in under two hour with agent based encoding; was going just under realtime encoding. The file was just under 2 gigs so it was easy to copy to each computer, unlike the 12+ gig ones I've been trying at work.

I think I read something on this before but is there any way to use agent based encoding with the avs and video files being in unique locations? Not all the computers I use can say use the C drive and since it seems I have to have a relative path somewhere that limits the agent based encoding, which is so much happier. Anyway, thanks again!

[omion]

Still testing it out, I'm loving the agent based encoding. At home with a P4 3.4GHz HT, a 800MHz athlon, a P4 2.4GHz HT, and a Dual Core 2Ghz laptop I was able to encode a 43 minute TV rip in under two hour with agent based encoding; was going just under realtime encoding. The file was just under 2 gigs so it was easy to copy to each computer, unlike the 12+ gig ones I've been trying at work.

I think I read something on this before but is there any way to use agent based encoding with the avs and video files being in unique locations? Not all the computers I use can say use the C drive and since it seems I have to have a relative path somewhere that limits the agent based encoding, which is so much happier. Anyway, thanks again!

You can use the <base> command in the agent's config.xml files to specify an alternate AVS search location. The exact semantics of the <base> line are a bit odd, but the example agent config.xml file goes over exactly how to use it.

Note that you have to make sure the paths from the AVS file to its source files are valid. That is, the AVS must be able to find its source files in the new location.


@all:
I'm currently updating my server, so the file availability will be a bit splotchy for the next day or two. Sorry!

[omion]

1.11 is out!

Main improvements are the lack of crashing with strange fps values, and basing the second-pass equality tests on the output bitrate.

I also changed the second pass GOP storage from a B+tree to a red-black tree. There's a good chance that nobody would notice this change. However, I can rest easier knowing that the tree actually balances itself when a GOP is taken out of it. My B-tree implementation didn't have a very good deletion function...

Neither the agent nor the config.xml files have changed, so there is no need to update them. The only update to the agent was the version number.

My server update is mostly complete, so there shouldn't be too many 404s. However, I added a mirror to 1.11 on the first post, just in case.